- developers - lists.mariadb.org

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Kristian Nielsen 30 Aug '16

30 Aug '16

Rich Prohaska <prohaska7(a)gmail.com> writes: > I rearranged the tokudb lock request wait function a little bit, and got > the lock tree unit tests to compile again (since the API changed). commit > on this tree: https://github.com/prohaska7/mariadb-server/tree/toku_opr3 Thanks! So I took a look at the complete patch we have so far, thinking what remains to do to get this included in main MariaDB. Do you have any suggestions for how/whether to upstream the TokuDB parts? I assume the new kill_query functionality would make sense to upstream. The callback to report waits could be upstreamed, but would not be used in non-mariadb builds, probably. And the actual tokudb_lock_wait_needed_callback() would probably not be upstreamed (or would go in #ifdef MARIADB_BASE_VERSION perhaps). Any thoughts? I still need to look into the should_retry_lock_requests, currently I get hangs if it is enabled. And I want to see if the wait_needed_callback interface can be cleaned up a bit. Otherwise the patch looks fairly complete to me, do you agree? I think this could go in 10.1, to fix optimistic parallel replication with TokuDB (which does not currently work at all). - Kristian.

2 1

Re: [Maria-developers] [Commits] 2a4ed5f: MDEV-10017: Get unexpected `Empty Set` for correlated subquery with aggregate functions (part 1)
by Sergey Petrunia 29 Aug '16

29 Aug '16

Hi Sanja, Table t2 is the testcase is redundant. Ok to push after this is addressed. On Mon, Aug 29, 2016 at 01:50:09PM +0200, Oleksandr Byelkin wrote: > revision-id: 2a4ed5f7a74860608cb36b761eb139a57cbd4463 (mariadb-10.1.16-15-g2a4ed5f) > parent(s): 76bd9f655a42dc1e21acfa26650a0cb76c70a22e > committer: Oleksandr Byelkin > timestamp: 2016-08-29 13:50:09 +0200 > message: > > MDEV-10017: Get unexpected `Empty Set` for correlated subquery with aggregate functions (part 1) > > Make aggregate function dependency visible. > > --- > mysql-test/r/func_group.result | 19 +++++++++++++++++++ > mysql-test/r/subselect3.result | 1 + > mysql-test/r/subselect3_jcl6.result | 1 + > mysql-test/r/subselect_mat.result | 1 + > mysql-test/r/subselect_sj_mat.result | 1 + > mysql-test/t/func_group.test | 11 +++++++++++ > sql/item_sum.cc | 10 ++++++++++ > sql/share/errmsg-utf8.txt | 3 +++ > 8 files changed, 47 insertions(+) > > diff --git a/mysql-test/r/func_group.result b/mysql-test/r/func_group.result > index 0d9d4ff..4bc5ad1 100644 > --- a/mysql-test/r/func_group.result > +++ b/mysql-test/r/func_group.result > @@ -2340,5 +2340,24 @@ companynr AVG(fld1) avg1 avg2 > 37 9223372036854775805.0000 9223372036854775805 9223372036854775805 > DROP TABLE t1; > # > +# case where aggregate resolved in the local SELECT > +# but outer ones are checked > +# > +create table t10 (a int , b int, c int); > +insert into t10 values (0,0,0),(1,1,1); > +create table t11 as select * from t10; > +create table t12 as select * from t10; > +create table t2 as select * from t10; > +explain extended select a from t10 where c<3 or a in (select c from t12 union select max(t10.b) from t11 group by t11.c); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY t10 ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DEPENDENT SUBQUERY t12 ALL NULL NULL NULL NULL 2 100.00 Using where > +3 DEPENDENT UNION t11 ALL NULL NULL NULL NULL 2 100.00 Using temporary > +NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL > +Warnings: > +Note 1276 Field or reference 'test.t10.b' of SELECT #3 was resolved in SELECT #1 > +Note 1003 select `test`.`t10`.`a` AS `a` from `test`.`t10` where ((`test`.`t10`.`c` < 3) or <expr_cache><`test`.`t10`.`a`,`test`.`t10`.`b`>(<in_optimizer>(`test`.`t10`.`a`,<exists>(select `test`.`t12`.`c` from `test`.`t12` where (<cache>(`test`.`t10`.`a`) = `test`.`t12`.`c`) union select max(`test`.`t10`.`b`) from `test`.`t11` group by `test`.`t11`.`c` having (<cache>(`test`.`t10`.`a`) = <ref_null_helper>(max(`test`.`t10`.`b`))))))) > +drop table t10,t11,t12,t2; > +# > # End of 10.1 tests > # > diff --git a/mysql-test/r/subselect3.result b/mysql-test/r/subselect3.result > index 24d9f0d..052b46d 100644 > --- a/mysql-test/r/subselect3.result > +++ b/mysql-test/r/subselect3.result > @@ -889,6 +889,7 @@ ERROR 42S22: Unknown column 'c' in 'field list' > SHOW WARNINGS; > Level Code Message > Note 1276 Field or reference 'test.t1.a' of SELECT #3 was resolved in SELECT #2 > +Note 1981 Aggregate function 'count()' of SELECT #3 belongs to SELECT #2 > Note 1276 Field or reference 'test.t1.c' of SELECT #3 was resolved in SELECT #2 > Error 1054 Unknown column 'c' in 'field list' > DROP TABLE t1; > diff --git a/mysql-test/r/subselect3_jcl6.result b/mysql-test/r/subselect3_jcl6.result > index 19d3d25..28f4e40 100644 > --- a/mysql-test/r/subselect3_jcl6.result > +++ b/mysql-test/r/subselect3_jcl6.result > @@ -899,6 +899,7 @@ ERROR 42S22: Unknown column 'c' in 'field list' > SHOW WARNINGS; > Level Code Message > Note 1276 Field or reference 'test.t1.a' of SELECT #3 was resolved in SELECT #2 > +Note 1981 Aggregate function 'count()' of SELECT #3 belongs to SELECT #2 > Note 1276 Field or reference 'test.t1.c' of SELECT #3 was resolved in SELECT #2 > Error 1054 Unknown column 'c' in 'field list' > DROP TABLE t1; > diff --git a/mysql-test/r/subselect_mat.result b/mysql-test/r/subselect_mat.result > index d0c5c2e..bfb40b9 100644 > --- a/mysql-test/r/subselect_mat.result > +++ b/mysql-test/r/subselect_mat.result > @@ -1139,6 +1139,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra > 3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where > Warnings: > Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 > +Note 1981 Aggregate function 'max()' of SELECT #3 belongs to SELECT #1 > Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` group by `test`.`t1`.`a` having <expr_cache><`test`.`t1`.`a`,`test`.`t1`.`b`,max(`test`.`t1`.`b`),max(`test`.`t1`.`b`)>(<in_optimizer>(`test`.`t1`.`a`,<exists>(select `test`.`t2`.`c` from `test`.`t2` where (<nop>(<expr_cache><`test`.`t2`.`d`,`test`.`t1`.`b`,max(`test`.`t1`.`b`),max(`test`.`t1`.`b`)>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select `test`.`t3`.`e` from `test`.`t3` where (max(`test`.`t1`.`b`) = `test`.`t3`.`e`) having (<cache>(`test`.`t2`.`d`) >= <ref_null_helper>(`test`.`t3`.`e`)))))) and (<cache>(`test`.`t1`.`a`) = `test`.`t2`.`c`))))) > select a from t1 group by a > having a in (select c from t2 where d >= some(select e from t3 where max(b)=e)); > diff --git a/mysql-test/r/subselect_sj_mat.result b/mysql-test/r/subselect_sj_mat.result > index e846549..366a03b 100644 > --- a/mysql-test/r/subselect_sj_mat.result > +++ b/mysql-test/r/subselect_sj_mat.result > @@ -1173,6 +1173,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra > 3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where > Warnings: > Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 > +Note 1981 Aggregate function 'max()' of SELECT #3 belongs to SELECT #1 > Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` group by `test`.`t1`.`a` having <expr_cache><`test`.`t1`.`a`,`test`.`t1`.`b`,max(`test`.`t1`.`b`),max(`test`.`t1`.`b`)>(<in_optimizer>(`test`.`t1`.`a`,<exists>(select `test`.`t2`.`c` from `test`.`t2` where (<nop>(<expr_cache><`test`.`t2`.`d`,`test`.`t1`.`b`,max(`test`.`t1`.`b`),max(`test`.`t1`.`b`)>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select `test`.`t3`.`e` from `test`.`t3` where (max(`test`.`t1`.`b`) = `test`.`t3`.`e`) having (<cache>(`test`.`t2`.`d`) >= <ref_null_helper>(`test`.`t3`.`e`)))))) and (<cache>(`test`.`t1`.`a`) = `test`.`t2`.`c`))))) > select a from t1 group by a > having a in (select c from t2 where d >= some(select e from t3 where max(b)=e)); > diff --git a/mysql-test/t/func_group.test b/mysql-test/t/func_group.test > index 47477c7..4a8b76b 100644 > --- a/mysql-test/t/func_group.test > +++ b/mysql-test/t/func_group.test > @@ -1617,5 +1617,16 @@ SELECT companynr, AVG(fld1), AVG(fld1)<<0 AS avg1, CAST(AVG(fld1) AS UNSIGNED)<< > DROP TABLE t1; > > --echo # > +--echo # case where aggregate resolved in the local SELECT > +--echo # but outer ones are checked > +--echo # > +create table t10 (a int , b int, c int); > +insert into t10 values (0,0,0),(1,1,1); > +create table t11 as select * from t10; > +create table t12 as select * from t10; > +create table t2 as select * from t10; > +explain extended select a from t10 where c<3 or a in (select c from t12 union select max(t10.b) from t11 group by t11.c); > +drop table t10,t11,t12,t2; > +--echo # > --echo # End of 10.1 tests > --echo # > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index 1cfee1a..2a2b7f9 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -380,6 +380,16 @@ bool Item_sum::register_sum_func(THD *thd, Item **ref) > sl->master_unit()->item->with_sum_func= 1; > } > thd->lex->current_select->mark_as_dependent(thd, aggr_sel, NULL); > + > + if ((thd->lex->describe & DESCRIBE_EXTENDED) && aggr_sel) > + { > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_NOTE, > + ER_WARN_AGGFUNC_DEPENDENCE, > + ER_THD(thd, ER_WARN_AGGFUNC_DEPENDENCE), > + func_name(), > + thd->lex->current_select->select_number, > + aggr_sel->select_number); > + } > return FALSE; > } > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 709ac55..a6eaf2b 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7139,3 +7139,6 @@ ER_KILL_QUERY_DENIED_ERROR > ER_NO_EIS_FOR_FIELD > eng "Engine-independent statistics are not collected for column '%s'" > ukr "Незалежна від типу таблиці статистика не збирається для стовбця '%s'" > +ER_WARN_AGGFUNC_DEPENDENCE > + eng "Aggregate function '%-.192s)' of SELECT #%d belongs to SELECT #%d" > + ukr "Агрегатна функція '%-.192s)' з SELECTу #%d належить до SELECTу #%d" > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] c584f16: MDEV-10017: Get unexpected `Empty Set` for correlated subquery with aggregate functions (part 1)
by Sergey Petrunia 29 Aug '16

29 Aug '16

Hi Sanja, On Thu, Jun 23, 2016 at 05:50:08PM +0200, Oleksandr Byelkin wrote: > revision-id: c584f1655ef45dafd2b5d0eb55ca21421de9a266 (mariadb-10.1.14-25-gc584f16) > parent(s): 773ce408762b5f8256d4053b6d0d418d15657b92 > committer: Oleksandr Byelkin > timestamp: 2016-06-23 17:50:07 +0200 > message: > > MDEV-10017: Get unexpected `Empty Set` for correlated subquery with aggregate functions (part 1) > > Make aggregate function dependency visible. > * I was able to get a crash with the patch. The details are below. * The wording of the warning feels odd: > Field or reference 'test.t10.b' of SELECT #3 was resolved in SELECT #1 > ... > Aggregate function 'max()' of SELECT #3 belongs to SELECT #1 + I like how the wording of the new warning is similar to the old one. - It is not immediately clear what does "belong to" mean. I would change it to " Aggregate function 'max()' of SELECT #3 is aggregated in SELECT #1 " This looks much clearer. OTOH, "Aggregate .. is aggregated" doesn't sound very good. We probably need to consult a native speaker (of both English and SQL :). == Details about the crash == create table ten(a int); insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); create table t10 (a int , b int, c int); insert into t10 select a,a,a from ten; create table t11 as select * from t10; create table t2 as select * from t10; explain extended select a from t10 where c<3 or a in (select c from t12 union select max(t10.b) from t11 group by t11.c); Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff7f1fb00 (LWP 4805)] 0x0000555555d0dd3f in Item_sum::register_sum_func (this=0x7fff89845680, thd=0x7fff9cbf6070, ref=0x7fff898457d0) at /home/psergey/dev-git/10.1/sql/item_sum.cc:391 (gdb) wher #0 0x0000555555d0dd3f in Item_sum::register_sum_func (this=0x7fff89845680, thd=0x7fff9cbf6070, ref=0x7fff898457d0) at /home/psergey/dev-git/10.1/sql/item_sum.cc:391 #1 0x0000555555d0d6f2 in Item_sum::check_sum_func (this=0x7fff89845680, thd=0x7fff9cbf6070, ref=0x7fff898457d0) at /home/psergey/dev-git/10.1/sql/item_sum.cc:177 #2 0x0000555555d1041e in Item_sum_hybrid::fix_fields (this=0x7fff89845680, thd=0x7fff9cbf6070, ref=0x7fff898457d0) at /home/psergey/dev-git/10.1/sql/item_sum.cc:1222 #3 0x00005555559c3cf2 in setup_fields (thd=0x7fff9cbf6070, ref_pointer_array=0x7fff898480f0, fields=..., mark_used_columns=MARK_COLUMNS_READ, sum_func_list=0x7fff89847ec8, allow_sum_func=true) at /home/psergey/dev-git/10.1/sql/sql_base.cc:7899 #4 0x0000555555a5b886 in JOIN::prepare (this=0x7fff89847b68, rref_pointer_array=0x7fff89845138, tables_init=0x7fff89845830, wild_num=0, conds_init=0x0, og_num=1, order_init=0x0, skip_order_by=true, group_init=0x7fff89845f38, having_init=0x0, proc_param_init=0x0, select_lex_arg=0x7fff89844ec0, unit_arg=0x7fff89844078) at /home/psergey/dev-git/10.1/sql/sql_select.cc:795 #5 0x0000555555af7082 in st_select_lex_unit::prepare (this=0x7fff89844078, thd_arg=0x7fff9cbf6070, sel_result=0x7fff89846118, additional_options=268435456) at /home/psergey/dev-git/10.1/sql/sql_union.cc:454 #6 0x0000555555d04d2c in subselect_union_engine::prepare (this=0x7fff89846140, thd_arg=0x7fff9cbf6070) at /home/psergey/dev-git/10.1/sql/item_subselect.cc:3590 #7 0x0000555555cfabe4 in Item_subselect::fix_fields (this=0x7fff89845f88, thd_param=0x7fff9cbf6070, ref=0x7fff89846270) at /home/psergey/dev-git/10.1/sql/item_subselect.cc:262 #8 0x0000555555d03f3d in Item_in_subselect::fix_fields (this=0x7fff89845f88, thd_arg=0x7fff9cbf6070, ref=0x7fff89846270) at /home/psergey/dev-git/10.1/sql/item_subselect.cc:3253 #9 0x0000555555c999cf in Item_cond::fix_fields (this=0x7fff89846178, thd=0x7fff9cbf6070, ref=0x7fff89846e98) at /home/psergey/dev-git/10.1/sql/item_cmpfunc.cc:4523 #10 0x00005555559c5bf1 in setup_conds (thd=0x7fff9cbf6070, tables=0x7fff89843330, leaves=..., conds=0x7fff89846e98) at /home/psergey/dev-git/10.1/sql/sql_base.cc:8627 #11 0x0000555555aa3eab in setup_without_group (thd=0x7fff9cbf6070, ref_pointer_array=0x7fff89846fd8, tables=0x7fff89843330, leaves=..., fields=..., all_fields=..., conds=0x7fff89846e98, order=0x0, group=0x0, hidden_group_fields=0x7fff89846d78, reserved=0x7fff9cbfa454) at /home/psergey/dev-git/10.1/sql/sql_select.cc:645 #12 0x0000555555a5b94b in JOIN::prepare (this=0x7fff89846a50, rref_pointer_array=0x7fff9cbfa430, tables_init=0x7fff89843330, wild_num=0, conds_init=0x7fff89846178, og_num=0, order_init=0x0, skip_order_by=false, group_init=0x0, having_init=0x0, proc_param_init=0x0, select_lex_arg=0x7fff9cbfa1b8, unit_arg=0x7fff9cbf9ab8) at /home/psergey/dev-git/10.1/sql/sql_select.cc:799 #13 0x0000555555a64f91 in mysql_select (thd=0x7fff9cbf6070, rref_pointer_array=0x7fff9cbfa430, tables=0x7fff89843330, wild_num=0, fields=..., conds=0x7fff89846178, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748612, result=0x7fff89846378, unit=0x7fff9cbf9ab8, select_lex=0x7fff9cbfa1b8) at /home/psergey/dev-git/10.1/sql/sql_select.cc:3415 #14 0x0000555555a9a6f5 in mysql_explain_union (thd=0x7fff9cbf6070, unit=0x7fff9cbf9ab8, result=0x7fff89846378) at /home/psergey/dev-git/10.1/sql/sql_select.cc:24737 #15 0x0000555555a2a9aa in execute_sqlcom_select (thd=0x7fff9cbf6070, all_tables=0x7fff89843330) at /home/psergey/dev-git/10.1/sql/sql_parse.cc:5834 #16 0x0000555555a20c72 in mysql_execute_command (thd=0x7fff9cbf6070) at /home/psergey/dev-git/10.1/sql/sql_parse.cc:2961 #17 0x0000555555a2e4be in mysql_parse (thd=0x7fff9cbf6070, rawbuf=0x7fff89843088 "explain extended select a from t10 where c<3 or a in (select c from t12 union select max(t10.b) from t11 group by t11.c)", length=120, parser_state=0x7ffff7f1e630) at /home/psergey/dev-git/10.1/sql/sql_parse.cc:7319 BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Aggregate Functions
by Varun Gupta 26 Aug '16

26 Aug '16

Hi, As GSOC is nearing its end, I have written a blog which contains how can aggregate functions be written and executed to get the desired results.In the blog I also have added the technical details of this project on aggregate functions. Here are the links: Jira issue: https://jira.mariadb.org/browse/MDEV-7773 Blog Link: http://varunraiko.github.io/aggregate-functions/ Disclaimer: I hereby declare that all my work done so far (and future) by me on the Aggregate Functions’ project, is released under the NEW BSD licence. Regards, Varun

2 1

Re: [Maria-developers] Sachin weekly report
by Sachin Setia 24 Aug '16

24 Aug '16

Hi Sergei! Actually I completed the work on update and delete. Now they will use index for looking up records. But I am thinking I have done a lot of changes in optimizer which may break it , and also there are lots of queries where my code does not work, fixing this might take a long amount of time. I am thinking of a change in my existing code :- Suppose a table t1 create table t1 (a blob, b blob, c blob, unique(a,b,c)); In current code , for query like there will a KEY with only one keypart which points to field DB_ROW_HASH_1. It was okay for normal updates , insert and delete , but in the case of where optimization I have do a lot of stuff , first to match field (like in add_key_part), then see whether all the fields in hash_str are present in where or not, then create keys by calculating hash. I do this by checking the HA_UNIQUE_HASH flag in KEY , but this also makes (I think) optimizer code bad because of too much dependence. Also I need to patch get_mm_parts and get_mm_leaf function , which I think should not be patched. I am thinking of a another approach to this problem at server level instead of having just one keypart we can have 1+3 keypart. Last three keypart will be for field a, b, c and first one for DB_ROW_HASH_1 .These will be only at server level not at storage level. key_info->key_part will point at keypart containing field a , while key_part having field DB_ROW_HASH_1 will -1 index. By this way I do not have to patch more of optimizer code. But there is one problem , what should be the length of key_part? I am thinking of it equal to field->pack_length(), this would not work because while creating keys optimizer calls get_key_image() (which is real data so can exceed pack_lenght() in case of blob), so to get this work I have to patch optimizer where it calls get_key_image() and see if key is HA_UNIQUE_HASH . If yes then instead of get_key_image just use memcpy(key, field->ptr(), field->pack_length()); this wont copy the actual data, but we do not need actual data. I will patch handler methods like ha_index_read, ha_index_idx_read , multi_range_read_info_const basically handler methods which are related to index or range search. In these methods i need to calculate hash , which I can calculate from key_ptr but key_ptr doe not have actual data(in case of blobs etc).So to get the date for hash , I will make a field clone of (a,b,c etc) but there ptr will point in key_ptr. Then field->val_str() method will work simply and i can calculate hash. And also I can compare returned result with actual key in handler method itself. What do you think of this approach ? Regards sachin On Sat, Aug 20, 2016 at 11:16 PM, Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, Sachin! > > On Aug 19, Sachin Setia wrote: > > On Fri, Aug 19, 2016 at 2:42 PM, Sergei Golubchik <serg(a)mariadb.org> > wrote: > > > > > First. I believe you'll need to do your final evaluation soon, and > > > it will need to have a link to the code. Did you check google > > > guidelines about it? Is everything clear there? Do you need help > > > publishing your work in a format that google requires? > > > > > > They don't accept delays for any reasons, so even if your code is > > > not 100% complete and ready, you'd better still publish it and > > > submit the evaluation, because otherwise google will fail you and > > > that'd be too sad. > > > > > > If you'd like you can publish the google-way only the > > > unique-constraint part without further optimizer work. Or at least > > > please mention that you'd completed the original project and went > > > working on extensions. I mean, it's better than saying "the code is > > > not 100% complete" :) > > > > > Okay I am thinking of writing a blog post with a link to my github > > repository. > > Blog Link <http://sachin1001gsoc.blogspot.in/2016/08/gsoc-2016.html> > > Please check this. > > I think that'll do, yes. > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org >

2 3

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Kristian Nielsen 24 Aug '16

24 Aug '16

Rich Prohaska <prohaska7(a)gmail.com> writes: > On Tue, Aug 23, 2016 at 1:45 PM, Kristian Nielsen <knielsen(a)knielsen-hq.org> > wrote: >> In the original parallel replication patch, thd_report_wait_for() did not >> call back directly into tokudb_kill_query(). The kill happened >> asynchronously, in a separate background thread. Then there is no problem >> with TokuDB (or InnoDB) holding locks over the call to >> thd_report_wait_for(). >> >> I am considering re-introducing that orginal code - this might simplify >> TokuDB implementation (and would also simplify InnoDB implementation). >> I was never very happy about the way thd_report_wait_for() works currently. Ok, so I implemented this, available in this branch: https://github.com/knielsen/server/commits/toku_opr3 With this code, I am no longer able to reproduce any hangs with the tests I've been running so far. > BTW, the current_lock_tree_mutex logic is broken. The underlying tokuft > lock manager has one manager object (and its mutex) and many lock tree > objects (each with its own mutex). Since the thd_report_wait_for function > is called when holding the lock tree mutex (not the manager mutex), it can > be called in parallel; thus there is a race on the current_lock_tree_mutex > logic. Yes, and this is the main problem solved by doing the kill asynchronously. And from my tests, it looks like this was actually what was causing the problems - crash, assertion, and hangs. (I haven't determined this conclusively, but it seems at least plausible). It really was always an ugly hack around the locking problem with thd_report_wait_for() (InnoDB had a similar issue), it seems good to get rid of it. I still have the should_retry_lock_requests disabled in retry_all_lock_requests() - otherwise I get hangs. I haven't investigated this deeply yet. I also want to go back over the entire set of patches and see what needs cleaning up, and think about how this could go into 10.2 and possibly 10.1. - Kristian.

1 0

[Maria-developers] Commit emails reply-to maria-developers@lists.launchpad.net
by Sergey Vojtovich 24 Aug '16

24 Aug '16

Hi! Why do commit emails have "Reply-To: maria-developers(a)lists.launchpad.net"? I believe it is wrong at least for the following reasons: - I must be careful if I want to reply privately, e.g. something like "hold on, I'll review it in a few hours" doesn't deserve to hit maria-developers@ - I want a person, I'm replying to, to be primary recepient, but Reply-To substitutes his email with maria-devlopers@. Makes it easy to miss reviews. Sergei mentioned that one of reasons might be so that we want to keep commits(a)mariadb.org. I believe it doesn't work because: - if I do group reply, I end up with email that has CC: commits(a)mariadb.org Thanks, Sergey

1 0

Re: [Maria-developers] [Commits] 294961c: MDEV-10538: MariaDB fails to start without galera_recovery in systemd mode
by Sergey Vojtovich 24 Aug '16

24 Aug '16

Hi Nirbhay, Ok to push. I suggest that we avoid such overcomplicated constructs in our default systemd configuration. Consider integrating recovery into the server (I know it's not trivial) or creating "included" config for galera. Thanks, Sergey On Mon, Aug 22, 2016 at 06:38:07PM -0400, Nirbhay Choubey wrote: > revision-id: 294961cc4d4d55730e807b3f0a7f93a3dd4d4f7b (mariadb-10.1.16-14-g294961c) > parent(s): d0d99dec914d409d7889fcec2efd126edad10bde > author: Nirbhay Choubey > committer: Nirbhay Choubey > timestamp: 2016-08-22 18:38:06 -0400 > message: > > MDEV-10538: MariaDB fails to start without galera_recovery in systemd mode > > Update ExecStartPre scripts to not fail if 'galera_recovery' > script is not available. > > --- > support-files/mariadb.service.in | 6 ++++-- > support-files/mariadb@.service.in | 15 +++++++++------ > 2 files changed, 13 insertions(+), 8 deletions(-) > > diff --git a/support-files/mariadb.service.in b/support-files/mariadb.service.in > index 6b8b2ba..9b97172 100644 > --- a/support-files/mariadb.service.in > +++ b/support-files/mariadb.service.in > @@ -63,9 +63,11 @@ PermissionsStartOnly=true > # galera_recovery simply returns an empty string. In any case, however, > # the script is not expected to return with a non-zero status. > # It is always safe to unset _WSREP_START_POSITION environment variable. > +# Do not panic if galera_recovery script is not available. (MDEV-10538) > ExecStartPre=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION" > -ExecStartPre=/bin/sh -c "VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && \ > - systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1" > +ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \ > + VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] \ > + && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1" > > # Needed to create system tables etc. > # ExecStartPre=/usr/bin/mysql_install_db -u mysql > diff --git a/support-files/mariadb@.service.in b/support-files/mariadb@.service.in > index 965e852..e941f21 100644 > --- a/support-files/mariadb@.service.in > +++ b/support-files/mariadb@.service.in > @@ -70,16 +70,19 @@ PermissionsStartOnly=true > # galera_recovery simply returns an empty string. In any case, however, > # the script is not expected to return with a non-zero status. > # It is always safe to unset _WSREP_START_POSITION%I environment variable. > +# Do not panic if galera_recovery script is not available. (MDEV-10538) > ExecStartPre=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION%I" > -ExecStartPre=/bin/sh -c "VAR=`/usr/bin/galera_recovery \ > - --defaults-file=@INSTALL_SYSCONF2DIR@/my%I.cnf`; [ $? -eq 0 ] && \ > - systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > + > +ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \ > + VAR=`/usr/bin/galera_recovery --defaults-file=@INSTALL_SYSCONF2DIR@/my%I.cnf`; [ $? -eq 0 ] \ > + && systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > + > # Alternate: (remove ConditionPathExists above) > # use [mysqld.INSTANCENAME] as sections in my.cnf > # > -#ExecStartPre=/bin/sh -c "VAR=`/usr/bin/galera_recovery \ > -# --defaults-group-suffix=%I`; [ $? -eq 0 ] && \ > -# systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > +#ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \ > +# VAR=`/usr/bin/galera_recovery --defaults-group-suffix=%I`; [ $? -eq 0 ] \ > +# && systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > > # Needed to create system tables etc. > # ExecStartPre=/usr/bin/mysql_install_db -u mysql > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Please review MDEV-10604 Disable unstable MTR tests by default
by Elena Stepanova 24 Aug '16

24 Aug '16

Hi Sergey (Vojtovich), Please review the patch that we've discussed at length. Serg, FYI. The length of the "unstable" list shouldn't scare anyone. At the end, this task should cause no change in our current procedure, the tests will still be run in buildbot and hopefully on developers' local machines (and every external user who wants to run them can do it easily). The change should mainly help those who run tests as a part of their build process. The lists can be tuned later, now they have been created generously, to solve the most urgent issue in shortest time possible. Please note that it's still not a 100% guarantee that nothing will fail sporadically, I've already got an absolutely new random failure after disabling all known ones, so it will still require some iterations. Regards, Elena -------- Forwarded Message -------- Subject: [Commits] 5ac5050: MDEV-10604 Disable unstable MTR tests by default Date: Mon, 22 Aug 2016 14:44:30 +0300 (EEST) From: elenst(a)montyprogram.com Reply-To: maria-developers(a)lists.launchpad.net To: commits(a)mariadb.org revision-id: 5ac505064a75508213d4dbce388ff5ee62d1ad56 (mariadb-10.0.26-92-g5ac5050) parent(s): 05f61ba46046ca835071a73b9255e787dcce9255 author: Elena Stepanova committer: Elena Stepanova timestamp: 2016-08-22 14:38:27 +0300 message: MDEV-10604 Disable unstable MTR tests by default - add new syntax to disabled.def in MTR to distinguish "unstable" tests from disabled tests; - allow wildcards in test names in disabled.def; - add a new option enable-unstable and env option MTR_ENABLE_DISABLED; - mark "unstable" tests which have failed since the beginning of the year in buildbot on main 10.0 branch; - mark "unstable" tests which fail in Debian builds; - mark "unstable" tests which have been changed during the last two releases --- mysql-test/disabled.def | 67 +++++++++++++++++++- mysql-test/lib/mtr_cases.pm | 47 ++++++++++++-- mysql-test/mysql-test-run.pl | 12 +++- mysql-test/suite/binlog/disabled.def | 14 ++++ mysql-test/suite/federated/disabled.def | 15 +++++ mysql-test/suite/funcs_1/disabled.def | 16 ++++- mysql-test/suite/innodb/disabled.def | 21 ++++++ mysql-test/suite/parts/disabled.def | 13 ++++ mysql-test/suite/perfschema/disabled.def | 24 +++++++ mysql-test/suite/rpl/disabled.def | 39 ++++++++++++ mysql-test/suite/sys_vars/disabled.def | 14 ++++ storage/connect/mysql-test/connect/disabled.def | 12 ++++ .../mysql-test/mroonga/storage/disabled.def | 30 +++++++++ storage/tokudb/mysql-test/rpl/disabled.def | 26 ++++++++ storage/tokudb/mysql-test/tokudb/disabled.def | 29 +++++++++ storage/tokudb/mysql-test/tokudb_bugs/disabled.def | 27 ++++++++ .../tokudb/mysql-test/tokudb_parts/disabled.def | 26 ++++++++ 17 files changed, 421 insertions(+), 11 deletions(-) diff --git a/mysql-test/disabled.def b/mysql-test/disabled.def index 5c4b742..f0cb6ec 100644 --- a/mysql-test/disabled.def +++ b/mysql-test/disabled.def @@ -6,17 +6,82 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# + <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## + tablespace : disabled in MariaDB (no TABLESPACE table attribute) events_time_zone : Test is not predictable as it depends on precise timing. read_many_rows_innodb : Bug#11748886 2010-11-15 mattiasj report already exists log_tables-big : Bug#11756699 2010-11-15 mattiasj report already exists mysql_embedded : Bug#12561297 2011-05-14 Anitha Dependent on PB2 changes - eventum#41836 -#show_explain : Psergey: random timeout in range-checked-for-each record query. ssl_crl_clients_valid : broken upstream ssl_crl : broken upstream ssl_crl_clrpath : broken upstream file_contents : MDEV-6526 these files are not installed anymore lowercase_fs_on : lower_case_table_names=0 is not an error until 10.1 + +########################## + ++ bootstrap : Modified on 2016-06-18 (MDEV-9969) ++ create_delayed : MDEV-10605 - failed with timeout ++ create_or_replace : Modified on 2016-06-23 (MDEV-9728) ++ ctype_recoding : Modified on 2016-06-10 (MDEV-10181) ++ ctype_utf8 : Modified on 2016-06-21 (merge) ++ ctype_utf8mb4 : Modified on 2016-06-21 (merge) ++ events_1 : Modified on 2016-06-21 (MDEV-9524) ++ func_group : Modified on 2016-08-08 (MDEV-10468) ++ func_in : Modified on 2016-06-20 (MDEV-10020) ++ func_math : Modified on 2016-08-10 (merge) ++ func_misc : Modified on 2016-08-10 (merge) ++ grant2 : Modified on 2016-07-18 (MDEV-8569) ++ help : Modified on 2016-06-21 (MDEV-9524) ++ host_cache_size_functionality : MDEV-10606 - sporadic failure on shutdown ++ index_intersect_innodb : MDEV-10643 - failed with timeout ++ index_merge_innodb : MDEV-7142 - sporadic wrong execution plan ++ information_schema_stats : Modified on 2016-07-25 (MDEV-10428) ++ innodb_mysql_lock : MDEV-7861 - sporadic lock detection failure ++ insert_innodb : Modified on 2016-06-14 (merge from upstream) ++ loaddata : Modified on 2016-08-10 (merge) ++ locale : Modified on 2016-06-21 (merge) ++ mdev-504 : MDEV-10607 - sporadic "can't connect" ++ mdev375 : MDEV-10607 - sporadic "can't connect" ++ merge : MDEV-10607 - sporadic "can't connect" ++ multi_update : Modified on 2016-06-20 (MDEV-5973) ++ myisam_enable_keys-10506 : Added on 2016-08-10 (MDEV-10506) ++ mysqlcheck : Modified on 2016-08-10 (merge) ++ mysqldump : MDEV-10512 - sporadic assertion failure ++ mysqltest : MDEV-9269 - fails on Alpha ++ named_pipe : Modified on 2016-08-02 (MDEV-10383) ++ openssl_1 : Modified on 2016-07-11 (MDEV-10211) ++ parser : Modified on 2016-06-21 (merge) ++ pool_of_threads : MDEV-10100 - sporadic error on detecting max connections ++ ps_1general : Modified on 2016-07-12 (merge) ++ range : Modified on 2016-08-10 (merge) ++ range_mrr_icp : Modified on 2016-08-10 (merge) ++ query_cache : MDEV-10611 - sporadic mutex problem ++ shutdown : MDEV-10612 - sporadic crashes ++ sp-prelocking : Modified on 2016-08-10 (merge) ++ sp-security : MDEV-10607 - sporadic "can't connect" ++ ssl : MDEV-10211 - different ciphers on some platforms ++ ssl_ca : Modified on 2016-07-11 (MDEV-10211) ++ ssl_compress : Modified on 2016-07-11 (MDEV-10211) ++ ssl_timeout : Modified on 2016-07-11 (MDEV-10211) ++ stat_tables_par_innodb : MDEV-10515 - sporadic wrong results ++ status_user : Modified on 2016-06-20 (MDEV-8633) ++ subselect_innodb : MDEV-10614 - sporadic wrong results ++ temp_table : Modified on 2016-06-18 (MDEV-8569) ++ type_date : Modified on 2016-08-10 (merge) ++ type_datetime : Modified on 2016-06-16 (MDEV-9374) ++ view : Modified on 2016-08-10 (merge) ++ xtradb_mrr : Modified on 2016-08-04 (MDEV-9946) diff --git a/mysql-test/lib/mtr_cases.pm b/mysql-test/lib/mtr_cases.pm index b76b10d..886e5a6 100644 --- a/mysql-test/lib/mtr_cases.pm +++ b/mysql-test/lib/mtr_cases.pm @@ -36,6 +36,7 @@ our $do_test; our $skip_test; our $binlog_format; our $enable_disabled; +our $enable_unstable; our $default_storage_engine; our $opt_with_ndbcluster_only; @@ -313,6 +314,9 @@ sub combinations_from_file($$) } our %disabled; +our %disabled_wildcards; +our %unstable; +our %unstable_wildcards; sub parse_disabled { my ($filename, $suitename) = @_; @@ -321,10 +325,21 @@ sub parse_disabled { chomp; next if /^\s*#/ or /^\s*$/; mtr_error("Syntax error in $filename line $.") - unless /^\s*(?:([-0-9A-Za-z_\/]+)\.)?([-0-9A-Za-z_]+)\s*:\s*(.*?)\s*$/; + unless /^\s*(\+)?\s*(?:([-0-9A-Za-z_\/]+)\.)?([-0-9A-Za-z_\*]+)\s*:\s*(.*?)\s*$/; mtr_error("Wrong suite name in $filename line $.") - if defined $1 and defined $suitename and $1 ne $suitename; - $disabled{($1 || $suitename || '') . ".$2"} = $3; + if defined $2 and defined $suitename and $2 ne $suitename; + $suitename= $2 || $suitename || ''; + my ($casename, $unstable_flag, $text)= ($3, $1, $4); + + if ($casename =~ /\*/) { + # Wildcard + $disabled_wildcards{"$casename"}= $text; + $unstable_wildcards{"$casename"}= 1 if defined $unstable_flag; + } + else { + $disabled{$suitename . ".$casename"}= $text; + $unstable{$suitename . ".$casename"}= 1 if defined $unstable_flag; + } } close DISABLED; } @@ -721,16 +736,36 @@ sub collect_one_test_case { # Check for disabled tests # ---------------------------------------------------------------------- my $disable = $disabled{".$tname"} || $disabled{$name}; + if (not $disable) { + foreach my $w (keys %disabled_wildcards) { + if ($tname =~ /^$w/) { + $disable= $disabled_wildcards{$w}; + last; + } + } + } + my $unstable = $unstable{".$tname"} || $unstable{$name}; + if (not $unstable) { + foreach my $w (keys %unstable_wildcards) { + if ($tname =~ /^$w/) { + $unstable= $unstable_wildcards{$w}; + last; + } + } + } if (not defined $disable and $suite->{parent}) { $disable = $disabled{$suite->{parent}->{name} . ".$tname"}; } + if (not defined $unstable and $suite->{parent}) { + $unstable = $unstable{$suite->{parent}->{name} . ".$tname"}; + } if (defined $disable) { $tinfo->{comment}= $disable; - if ( $enable_disabled ) + if ( $enable_disabled or ( $enable_unstable and $unstable ) ) { - # User has selected to run all disabled tests - mtr_report(" - $tinfo->{name} wil be run although it's been disabled\n", + # User has selected to run disabled tests + mtr_report(" - $tinfo->{name} will be run although it's been disabled" . ($unstable ? " as unstable\n" : "\n"), " due to '$disable'"); } else diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl index 5af19c8..82e84ec 100755 --- a/mysql-test/mysql-test-run.pl +++ b/mysql-test/mysql-test-run.pl @@ -1203,6 +1203,7 @@ sub command_line_setup { 'force-restart' => \$opt_force_restart, 'reorder!' => \$opt_reorder, 'enable-disabled' => \&collect_option, + 'enable-unstable' => \&collect_option, 'verbose+' => \$opt_verbose, 'verbose-restart' => \&report_option, 'sleep=i' => \$opt_sleep, @@ -1597,6 +1598,14 @@ sub command_line_setup { $ENV{'STAGING_RUN'}= $opt_staging_run; # -------------------------------------------------------------------------- + # Unstable tests + # -------------------------------------------------------------------------- + if ( $ENV{'MTR_ENABLE_UNSTABLE'} ) + { + collect_option('enable-unstable',1); + } + + # -------------------------------------------------------------------------- # Gcov flag # -------------------------------------------------------------------------- if ( ($opt_gcov or $opt_gprof) and ! $source_dist ) @@ -6455,7 +6464,8 @@ Options to control what test suites or cases to run twice to run only "big" tests. staging-run Run a limited number of tests (no slow tests). Used for running staging trees with valgrind. - enable-disabled Run also tests marked as disabled + enable-disabled Run also tests marked as disabled or unstable + enable-unstable Run also tests marked as unstable print-testcases Don't run the tests but print details about all the selected tests, in the order they would be run. skip-test-list=FILE Skip the tests listed in FILE. Each line in the file diff --git a/mysql-test/suite/binlog/disabled.def b/mysql-test/suite/binlog/disabled.def index 424e554..7a42685 100644 --- a/mysql-test/suite/binlog/disabled.def +++ b/mysql-test/suite/binlog/disabled.def @@ -6,8 +6,22 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## binlog_truncate_innodb : BUG#11764459 2010-10-20 anitha Originally disabled due to BUG#42643. Product bug fixed, but test changes needed binlog_spurious_ddl_errors : BUG#11761680 2013-01-18 astha Fixed on mysql-5.6 and trunk + +########################## + ++ binlog_commit_wait : sporadic failures: MDEV-10150 ++ binlog_dmls_on_tmp_tables_readonly : Added on 2016-05-04 (upstream) ++ binlog_xa_recover : sporadic failures: MDEV-8517 diff --git a/mysql-test/suite/federated/disabled.def b/mysql-test/suite/federated/disabled.def index bb2e4dd..279947b 100644 --- a/mysql-test/suite/federated/disabled.def +++ b/mysql-test/suite/federated/disabled.def @@ -6,6 +6,21 @@ # # <testcasename> : BUG#<xxxx> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## + +########################## + ++ federatedx : sporadic failures: MDEV-10617 ++ federated_innodb : sporadic failures: MDEV-10617, MDEV-8691 ++ federated_transactions : sporadic failures: MDEV-10617, MDEV-8691 ++ federated_innodb : sporadic failures: MDEV-8691 diff --git a/mysql-test/suite/funcs_1/disabled.def b/mysql-test/suite/funcs_1/disabled.def index 9bfe956..3cdc1c6 100644 --- a/mysql-test/suite/funcs_1/disabled.def +++ b/mysql-test/suite/funcs_1/disabled.def @@ -1,12 +1,22 @@ ############################################################################## # -# List the test cases that are to be disabled temporarely. +# List the test cases that are to be disabled temporarily. # # Separate the test case name and the comment with ':'. # -# <testcasename> : Comment test +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # -# Don't use any TAB characters for whitespace. +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. # ############################################################################## ++ processlist_priv_no_prot : Include file modified on 2016-07-12 (merge) ++ processlist_priv_ps : Include file modified on 2016-07-12 (merge) diff --git a/mysql-test/suite/innodb/disabled.def b/mysql-test/suite/innodb/disabled.def index 8cae44a..49e87c4 100644 --- a/mysql-test/suite/innodb/disabled.def +++ b/mysql-test/suite/innodb/disabled.def @@ -6,7 +6,28 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## +########################## + ++ innodb.binlog_consistent : sporadic failures: MDEV-10618 ++ innodb.innodb-alter-table : sporadic failures: MDEV-10619 ++ innodb.innodb-wl5522-debug-zip : sporadic failures: MDEV-10427 ++ innodb.innodb_bug30423 : sporadic failures: MDEV-7311 ++ innodb-alter-table : Modified on 2016-08-11 (MDEV-10535) ++ innodb-alter-tempfile : Modified on 2016-08-09 (MDEV-10469) ++ innodb-fk-warnings : Modified on 2016-07-18 (MDEV-8569) ++ innodb-fkcheck : Modified on 2016-06-13 (MDEV-10083) ++ innodb-wl5522 : rdiff file modified on 2016-08-10 (merge) ++ innodb_corrupt_bit : Modified on 2016-06-21 (merge) + diff --git a/mysql-test/suite/parts/disabled.def b/mysql-test/suite/parts/disabled.def index abbdbf7..17e2855 100644 --- a/mysql-test/suite/parts/disabled.def +++ b/mysql-test/suite/parts/disabled.def @@ -6,6 +6,14 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -15,3 +23,8 @@ parts.partition_mgm_lc1_archive : MDEV-5077 2013-09-27 svoj Cannot exchange part parts.partition_exchange_archive: MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table parts.partition_mgm_lc0_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table parts.partition_mgm_lc2_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table + +########################## + ++ partition_float_myisam : sporadic failures: MDEV-10621 ++ partition_int_myisam : sporadic failures: MDEV-10621 diff --git a/mysql-test/suite/perfschema/disabled.def b/mysql-test/suite/perfschema/disabled.def index d9e9e5b..8b4e222 100644 --- a/mysql-test/suite/perfschema/disabled.def +++ b/mysql-test/suite/perfschema/disabled.def @@ -6,6 +6,16 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -22,3 +32,17 @@ binlog_edge_stmt: needs slave_relay_log_info binlog_ok_mix: needs slave_relay_log_info binlog_ok_row: needs slave_relay_log_info binlog_ok_stmt: needs slave_relay_log_info + +########################## + ++ digest_table_full : Modified on 2016-06-21 (merge) ++ func_file_io : MDEV-5708 - fails for s390x ++ func_mutex : MDEV-5708 - fails for s390x ++ rpl_gtid_func : Modified on 2016-06-21 (merge) ++ sizing_low : Modified on 2016-04-26 (5.6.30 merge) ++ socket_summary_by_event_name_func : sporadic failures: MDEV-10622 ++ start_server_low_digest : Modified on 2016-06-21 (merge) ++ statement_digest : Modified on 2016-06-21 (merge) ++ statement_digest_consumers : Modified on 2016-06-21 (merge) ++ statement_digest_long_query : Modified on 2016-06-21 (merge) ++ table_name : Added on 2016-04-26 (5.6.30 merge) diff --git a/mysql-test/suite/rpl/disabled.def b/mysql-test/suite/rpl/disabled.def index de3091a..1d68be9 100644 --- a/mysql-test/suite/rpl/disabled.def +++ b/mysql-test/suite/rpl/disabled.def @@ -6,6 +6,14 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -13,3 +21,34 @@ rpl_spec_variables : BUG#11755836 2009-10-27 jasonh rpl_spec_variables fails on PB2 hpux rpl_get_master_version_and_clock : Bug#11766137 Jan 05 2011 joro Valgrind warnings rpl_get_master_version_and_clock rpl_partition_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table + +########################## + ++ last_insert_id : sporadic failures: MDEV-10625 ++ rpl_auto_increment : sporadic failures: MDEV-8691 ++ rpl_auto_increment_bug45679 : sporadic failures: MDEV-8691 ++ rpl_auto_increment_update_failure : sporadic failures: MDEV-10625 ++ rpl_binlog_index : sporadic failures: MDEV-9501 ++ rpl_checksum_cache : sporadic failures: MDEV-10626 ++ rpl_circular_for_4_hosts : sporadic failures: MDEV-10627 ++ rpl_gtid_crash : sporadic failures: MDEV-9501 ++ rpl_gtid_master_promote : sporadic failures: MDEV-10628 ++ rpl_gtid_stop_start : sporadic failures: MDEV-10629 ++ rpl_gtid_until : sporadic failures: MDEV-10625 ++ rpl_ignore_table : Modified on 2016-06-22 ++ rpl_insert : sporadic failures: MDEV-9329 ++ rpl_insert_delayed : sporadic failures: MDEV-9329 ++ rpl_invoked_features : sporadic failures: MDEV-8691 ++ rpl_mdev6020 : sporadic failures: MDEV-1630 ++ rpl_mdev6386 : sporadic failures: MDEV-10631 ++ rpl_parallel : sporadic failures: MDEV-10632 ++ rpl_parallel_temptable : sporadic failures: MDEV-10356 ++ rpl_row_drop_create_temp_table : sporadic failures: MDEV-10626 ++ rpl_row_sp001 : sporadic failures: MDEV-9329 ++ rpl_semi_sync_uninstall_plugin : sporadic failures: MDEV-7140 ++ rpl_slave_grp_exec : sporadic failures: MDEV-10514 ++ rpl_switch_stm_row_mixed : sporadic failures: MDEV-10611 ++ rpl_sync : sporadic failures: MDEV-10633 ++ rpl_temporary_error2 : sporadic failures: MDEV-10634 ++ sec_behind_master-5114 : sporadic failures: MDEV-8518 ++ rpl_skip_replication : sporadic failures: MDEV-9268 diff --git a/mysql-test/suite/sys_vars/disabled.def b/mysql-test/suite/sys_vars/disabled.def index 7582152..06c394c 100644 --- a/mysql-test/suite/sys_vars/disabled.def +++ b/mysql-test/suite/sys_vars/disabled.def @@ -6,9 +6,23 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## innodb_flush_checkpoint_debug_basic: removed from XtraDB-26.0 table_open_cache_instances_basic: no such variable in MariaDB + +########################## + ++ autocommit_func2 : sporadic failures: MDEV-9329 ++ general_log_file_basic : Modified on 2016-08-09 (MDEV-10465) ++ slow_query_log_file_basic : Modified on 2016-08-09 (MDEV-10465) diff --git a/storage/connect/mysql-test/connect/disabled.def b/storage/connect/mysql-test/connect/disabled.def index 9b45709..0ce6169 100644 --- a/storage/connect/mysql-test/connect/disabled.def +++ b/storage/connect/mysql-test/connect/disabled.def @@ -6,6 +6,14 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -14,3 +22,7 @@ jdbc : Variable settings depend on machine configuration jdbc_new : Variable settings depend on machine configuration jdbc_oracle : Variable settings depend on machine configuration jdbc_postgresql : Variable settings depend on machine configuration + +########################## + ++ tbl : MDEV-9844, MDEV-10179 - sporadic crashes, valgrind warnings, wrong results diff --git a/storage/mroonga/mysql-test/mroonga/storage/disabled.def b/storage/mroonga/mysql-test/mroonga/storage/disabled.def index 6866adc..c5c44db 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/disabled.def +++ b/storage/mroonga/mysql-test/mroonga/storage/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + alter_table_add_index_token_filters_one_token_filter : Bundled Mroonga does not support token filter yet. alter_table_change_token_filter : Bundled Mroonga does not support token filter yet. fulltext_token_filters_skip : Bundled Mroonga does not support token filter yet. @@ -8,3 +30,11 @@ create_table_token_filters_table_comment_one_token_filter : Bundled Mroonga does foreign_key_create : Bundled Mroonga does not support this test yet. partition_insert : Bundled Mroonga does not support this test yet. +########################## + ++ column_datetime_32bit_2038 : sporadic failures: MDEV-9270 ++ column_datetime_32bit_before_unix_epoch : sporadic failures: MDEV-9270 ++ column_datetime_32bit_max : sporadic failures: MDEV-9270 ++ column_datetime_32bit_out_of_range : sporadic failures: MDEV-9270 ++ index_multiple_column_unique_date_32bit_equal : sporadic failures: MDEV-9270 ++ index_multiple_column_unique_date_order_32bit_desc : sporadic failures: MDEV-9270 diff --git a/storage/tokudb/mysql-test/rpl/disabled.def b/storage/tokudb/mysql-test/rpl/disabled.def index 4c1a9a3..c58408e 100644 --- a/storage/tokudb/mysql-test/rpl/disabled.def +++ b/storage/tokudb/mysql-test/rpl/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# + <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + rpl_tokudb_delete_pk: unreliable, uses timestamp differences rpl_tokudb_delete_pk_lookup1: unreliable, uses timestamp differences rpl_tokudb_update_pk_uc0_lookup0: unreliable, uses timestamp differences @@ -13,3 +35,7 @@ rpl_tokudb_write_unique_uc1: unreliable, uses timestamp differences rpl_tokudb_read_only_ff: unreliable, uses timestamp differences rpl_tokudb_read_only_tf: unreliable, uses timestamp differences rpl_tokudb_read_only_tt: unreliable, uses timestamp differences + +########################## + ++ * : MDEV-9891 - massive crashes on shutdown diff --git a/storage/tokudb/mysql-test/tokudb/disabled.def b/storage/tokudb/mysql-test/tokudb/disabled.def index c98a8aa..3545544 100644 --- a/storage/tokudb/mysql-test/tokudb/disabled.def +++ b/storage/tokudb/mysql-test/tokudb/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + fast_update_binlog_mixed: No UPSERT in MariaDB 5.5 fast_update_binlog_row: No UPSERT in MariaDB 5.5 fast_update_binlog_statement: No UPSERT in MariaDB 5.5 @@ -28,3 +50,10 @@ type_timestamp_explicit: cluster_key_part: engine options on partitioned tables i_s_tokudb_lock_waits_released: unstable, race conditions i_s_tokudb_locks_released: unstable, race conditions + +########################## + ++ background_job_manager : sporadic failures: MDEV-10327 ++ cluster_filter_unpack_varchar : sporadic failures: MDEV-10636 + ++ * : MDEV-9891 - massive crashes on shutdown diff --git a/storage/tokudb/mysql-test/tokudb_bugs/disabled.def b/storage/tokudb/mysql-test/tokudb_bugs/disabled.def index d0d1a47..6554b9a 100644 --- a/storage/tokudb/mysql-test/tokudb_bugs/disabled.def +++ b/storage/tokudb/mysql-test/tokudb_bugs/disabled.def @@ -1,2 +1,29 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + 5585: times out, too many huge insert...selects db233: different execution path in mariadb, debug_sync point is not hit + +########################## + ++ checkpoint_lock : sporadic failures: MDEV-10637 ++ checkpoint_lock_3 : sporadic failures: MDEV-10637 + ++ * : MDEV-9891 - massive crashes on shutdown diff --git a/storage/tokudb/mysql-test/tokudb_parts/disabled.def b/storage/tokudb/mysql-test/tokudb_parts/disabled.def index 3252a46..7df2ac7 100644 --- a/storage/tokudb/mysql-test/tokudb_parts/disabled.def +++ b/storage/tokudb/mysql-test/tokudb_parts/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# + <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + partition_max_parts_hash_tokudb: 5.6 test not merged yet partition_max_parts_inv_tokudb: 5.6 test not merged yet partition_max_parts_key_tokudb: 5.6 test not merged yet @@ -7,3 +29,7 @@ partition_max_sub_parts_key_list_tokudb: 5.6 test not merged yet partition_max_sub_parts_key_range_tokudb: 5.6 test not merged yet partition_max_sub_parts_list_tokudb: 5.6 test not merged yet partition_max_sub_parts_range_tokudb: 5.6 test not merged yet + +########################## + ++ * : MDEV-9891 - massive crashes on shutdown _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 1

Re: [Maria-developers] Patch for parallel replication DBUG_ASSERT check on DROP TABLE et al
by Kristian Nielsen 20 Aug '16

20 Aug '16

Monty writes: > Can you please check if this looks correct? > diff --git a/sql/mdl.cc b/sql/mdl.cc > index 28d2006..3b1f9f2 100644 > --- a/sql/mdl.cc > +++ b/sql/mdl.cc > @@ -17,6 +17,7 @@ > #include "sql_class.h" > #include "debug_sync.h" > #include "sql_array.h" > +#include "rpl_rli.h" > #include <hash.h> > #include <mysqld_error.h> > #include <mysql/plugin.h> > @@ -442,6 +443,7 @@ class MDL_lock > virtual void notify_conflicting_locks(MDL_context *ctx) = 0; > > virtual bitmap_t hog_lock_types_bitmap() const = 0; > + bool check_if_conflicting_replication_locks(MDL_context *ctx); This should probably be #ifdef NDEBUG, since you're only using it in debug build? > > /** List of granted tickets for this lock. */ > Ticket_list m_granted; > @@ -2290,6 +2292,44 @@ void MDL_scoped_lock::notify_conflicting_locks(MDL_context *ctx) > } > } > > +/** > + Check if there is any conflicting lock that could cause this thread > + to wait for another thread which is not ready to commit. > + This is always an error, as the upper level of parallel replication > + should not allow a scheduling of a conflicting DDL until all earlier > + transactions has commited. > + > + This function is only called for a slave using parallel replication > + and trying to get an exclusive lock for the table. > +*/ > + > +bool MDL_lock::check_if_conflicting_replication_locks(MDL_context *ctx) > +{ > + Ticket_iterator it(m_granted); > + MDL_ticket *conflicting_ticket; > + > + while ((conflicting_ticket= it++)) > + { > + if (conflicting_ticket->get_ctx() != ctx) > + { > + MDL_context *conflicting_ctx= conflicting_ticket->get_ctx(); > + > + /* > + If the conflicting thread is another parallel replication > + thread for the same master and it's not in commit stage, then > + the current transaction has started too early and something is > + seriously wrong. > + */ > + if (conflicting_ctx->get_thd()->rgi_slave && > + conflicting_ctx->get_thd()->rgi_slave->rli == > + ctx->get_thd()->rgi_slave->rli && > + !conflicting_ctx->get_thd()->rgi_slave->did_mark_start_commit) > + return 1; // Fatal error No, this is not correct. For example, the threads might be in different domains, or using different multi-source connections. You need a check like in thd_report_wait_for(). _If_ T1 and T2 are in the same parallel replication domain, _and_ T1 goes before T2, _and_ there is a DDL lock conflict but T1 has not started to commit, then it indicates a potential problem. Also, the comments are too vague, "Something is seriously wrong" is not a good description. What is wrong? Please describe the issue being checked more precisely. What is the role of each of the two threads involved? What is the situation being checked? Why is it wrong? Also, use some temporaries for conflicting_ctx->get_thd()->rgi_slave etc, so the condition becomes more readable. > + } > + } > + return 0; > +} > + > > /** > Acquire one lock with waiting for conflicting locks to go away if needed. > @@ -2355,6 +2395,11 @@ MDL_context::acquire_lock(MDL_request *mdl_request, ulong lock_wait_timeout) > if (lock->needs_notification(ticket) && lock_wait_timeout) > lock->notify_conflicting_locks(this); > > + DBUG_ASSERT(mdl_request->type != MDL_INTENTION_EXCLUSIVE || > + !(get_thd()->rgi_slave && > + get_thd()->rgi_slave->is_parallel_exec && > + lock->check_if_conflicting_replication_locks(this))); > + > mysql_prlock_unlock(&lock->m_rwlock); > > will_wait_for(ticket); > diff --git a/sql/mdl.h b/sql/mdl.h > index c4d792a..13de602 100644 > --- a/sql/mdl.h > +++ b/sql/mdl.h > @@ -910,7 +910,6 @@ class MDL_context > */ > MDL_wait_for_subgraph *m_waiting_for; > private: > - THD *get_thd() const { return m_owner->get_thd(); } > MDL_ticket *find_ticket(MDL_request *mdl_req, > enum_mdl_duration *duration); > void release_locks_stored_before(enum_mdl_duration duration, MDL_ticket *sentinel); > @@ -919,6 +918,7 @@ class MDL_context > MDL_ticket **out_ticket); > > public: > + THD *get_thd() const { return m_owner->get_thd(); } > void find_deadlock(); > > ulong get_thread_id() const { return thd_get_thread_id(get_thd()); } - Kristian.

2 4