- developers - lists.mariadb.org

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Kristian Nielsen 24 Aug '16

24 Aug '16

Rich Prohaska <prohaska7(a)gmail.com> writes: > On Tue, Aug 23, 2016 at 1:45 PM, Kristian Nielsen <knielsen(a)knielsen-hq.org> > wrote: >> In the original parallel replication patch, thd_report_wait_for() did not >> call back directly into tokudb_kill_query(). The kill happened >> asynchronously, in a separate background thread. Then there is no problem >> with TokuDB (or InnoDB) holding locks over the call to >> thd_report_wait_for(). >> >> I am considering re-introducing that orginal code - this might simplify >> TokuDB implementation (and would also simplify InnoDB implementation). >> I was never very happy about the way thd_report_wait_for() works currently. Ok, so I implemented this, available in this branch: https://github.com/knielsen/server/commits/toku_opr3 With this code, I am no longer able to reproduce any hangs with the tests I've been running so far. > BTW, the current_lock_tree_mutex logic is broken. The underlying tokuft > lock manager has one manager object (and its mutex) and many lock tree > objects (each with its own mutex). Since the thd_report_wait_for function > is called when holding the lock tree mutex (not the manager mutex), it can > be called in parallel; thus there is a race on the current_lock_tree_mutex > logic. Yes, and this is the main problem solved by doing the kill asynchronously. And from my tests, it looks like this was actually what was causing the problems - crash, assertion, and hangs. (I haven't determined this conclusively, but it seems at least plausible). It really was always an ugly hack around the locking problem with thd_report_wait_for() (InnoDB had a similar issue), it seems good to get rid of it. I still have the should_retry_lock_requests disabled in retry_all_lock_requests() - otherwise I get hangs. I haven't investigated this deeply yet. I also want to go back over the entire set of patches and see what needs cleaning up, and think about how this could go into 10.2 and possibly 10.1. - Kristian.

1 0

[Maria-developers] Commit emails reply-to maria-developers@lists.launchpad.net
by Sergey Vojtovich 24 Aug '16

24 Aug '16

Hi! Why do commit emails have "Reply-To: maria-developers(a)lists.launchpad.net"? I believe it is wrong at least for the following reasons: - I must be careful if I want to reply privately, e.g. something like "hold on, I'll review it in a few hours" doesn't deserve to hit maria-developers@ - I want a person, I'm replying to, to be primary recepient, but Reply-To substitutes his email with maria-devlopers@. Makes it easy to miss reviews. Sergei mentioned that one of reasons might be so that we want to keep commits(a)mariadb.org. I believe it doesn't work because: - if I do group reply, I end up with email that has CC: commits(a)mariadb.org Thanks, Sergey

1 0

Re: [Maria-developers] [Commits] 294961c: MDEV-10538: MariaDB fails to start without galera_recovery in systemd mode
by Sergey Vojtovich 24 Aug '16

24 Aug '16

Hi Nirbhay, Ok to push. I suggest that we avoid such overcomplicated constructs in our default systemd configuration. Consider integrating recovery into the server (I know it's not trivial) or creating "included" config for galera. Thanks, Sergey On Mon, Aug 22, 2016 at 06:38:07PM -0400, Nirbhay Choubey wrote: > revision-id: 294961cc4d4d55730e807b3f0a7f93a3dd4d4f7b (mariadb-10.1.16-14-g294961c) > parent(s): d0d99dec914d409d7889fcec2efd126edad10bde > author: Nirbhay Choubey > committer: Nirbhay Choubey > timestamp: 2016-08-22 18:38:06 -0400 > message: > > MDEV-10538: MariaDB fails to start without galera_recovery in systemd mode > > Update ExecStartPre scripts to not fail if 'galera_recovery' > script is not available. > > --- > support-files/mariadb.service.in | 6 ++++-- > support-files/mariadb@.service.in | 15 +++++++++------ > 2 files changed, 13 insertions(+), 8 deletions(-) > > diff --git a/support-files/mariadb.service.in b/support-files/mariadb.service.in > index 6b8b2ba..9b97172 100644 > --- a/support-files/mariadb.service.in > +++ b/support-files/mariadb.service.in > @@ -63,9 +63,11 @@ PermissionsStartOnly=true > # galera_recovery simply returns an empty string. In any case, however, > # the script is not expected to return with a non-zero status. > # It is always safe to unset _WSREP_START_POSITION environment variable. > +# Do not panic if galera_recovery script is not available. (MDEV-10538) > ExecStartPre=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION" > -ExecStartPre=/bin/sh -c "VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && \ > - systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1" > +ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \ > + VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] \ > + && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1" > > # Needed to create system tables etc. > # ExecStartPre=/usr/bin/mysql_install_db -u mysql > diff --git a/support-files/mariadb@.service.in b/support-files/mariadb@.service.in > index 965e852..e941f21 100644 > --- a/support-files/mariadb@.service.in > +++ b/support-files/mariadb@.service.in > @@ -70,16 +70,19 @@ PermissionsStartOnly=true > # galera_recovery simply returns an empty string. In any case, however, > # the script is not expected to return with a non-zero status. > # It is always safe to unset _WSREP_START_POSITION%I environment variable. > +# Do not panic if galera_recovery script is not available. (MDEV-10538) > ExecStartPre=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION%I" > -ExecStartPre=/bin/sh -c "VAR=`/usr/bin/galera_recovery \ > - --defaults-file=@INSTALL_SYSCONF2DIR@/my%I.cnf`; [ $? -eq 0 ] && \ > - systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > + > +ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \ > + VAR=`/usr/bin/galera_recovery --defaults-file=@INSTALL_SYSCONF2DIR@/my%I.cnf`; [ $? -eq 0 ] \ > + && systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > + > # Alternate: (remove ConditionPathExists above) > # use [mysqld.INSTANCENAME] as sections in my.cnf > # > -#ExecStartPre=/bin/sh -c "VAR=`/usr/bin/galera_recovery \ > -# --defaults-group-suffix=%I`; [ $? -eq 0 ] && \ > -# systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > +#ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \ > +# VAR=`/usr/bin/galera_recovery --defaults-group-suffix=%I`; [ $? -eq 0 ] \ > +# && systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > > # Needed to create system tables etc. > # ExecStartPre=/usr/bin/mysql_install_db -u mysql > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Please review MDEV-10604 Disable unstable MTR tests by default
by Elena Stepanova 24 Aug '16

24 Aug '16

Hi Sergey (Vojtovich), Please review the patch that we've discussed at length. Serg, FYI. The length of the "unstable" list shouldn't scare anyone. At the end, this task should cause no change in our current procedure, the tests will still be run in buildbot and hopefully on developers' local machines (and every external user who wants to run them can do it easily). The change should mainly help those who run tests as a part of their build process. The lists can be tuned later, now they have been created generously, to solve the most urgent issue in shortest time possible. Please note that it's still not a 100% guarantee that nothing will fail sporadically, I've already got an absolutely new random failure after disabling all known ones, so it will still require some iterations. Regards, Elena -------- Forwarded Message -------- Subject: [Commits] 5ac5050: MDEV-10604 Disable unstable MTR tests by default Date: Mon, 22 Aug 2016 14:44:30 +0300 (EEST) From: elenst(a)montyprogram.com Reply-To: maria-developers(a)lists.launchpad.net To: commits(a)mariadb.org revision-id: 5ac505064a75508213d4dbce388ff5ee62d1ad56 (mariadb-10.0.26-92-g5ac5050) parent(s): 05f61ba46046ca835071a73b9255e787dcce9255 author: Elena Stepanova committer: Elena Stepanova timestamp: 2016-08-22 14:38:27 +0300 message: MDEV-10604 Disable unstable MTR tests by default - add new syntax to disabled.def in MTR to distinguish "unstable" tests from disabled tests; - allow wildcards in test names in disabled.def; - add a new option enable-unstable and env option MTR_ENABLE_DISABLED; - mark "unstable" tests which have failed since the beginning of the year in buildbot on main 10.0 branch; - mark "unstable" tests which fail in Debian builds; - mark "unstable" tests which have been changed during the last two releases --- mysql-test/disabled.def | 67 +++++++++++++++++++- mysql-test/lib/mtr_cases.pm | 47 ++++++++++++-- mysql-test/mysql-test-run.pl | 12 +++- mysql-test/suite/binlog/disabled.def | 14 ++++ mysql-test/suite/federated/disabled.def | 15 +++++ mysql-test/suite/funcs_1/disabled.def | 16 ++++- mysql-test/suite/innodb/disabled.def | 21 ++++++ mysql-test/suite/parts/disabled.def | 13 ++++ mysql-test/suite/perfschema/disabled.def | 24 +++++++ mysql-test/suite/rpl/disabled.def | 39 ++++++++++++ mysql-test/suite/sys_vars/disabled.def | 14 ++++ storage/connect/mysql-test/connect/disabled.def | 12 ++++ .../mysql-test/mroonga/storage/disabled.def | 30 +++++++++ storage/tokudb/mysql-test/rpl/disabled.def | 26 ++++++++ storage/tokudb/mysql-test/tokudb/disabled.def | 29 +++++++++ storage/tokudb/mysql-test/tokudb_bugs/disabled.def | 27 ++++++++ .../tokudb/mysql-test/tokudb_parts/disabled.def | 26 ++++++++ 17 files changed, 421 insertions(+), 11 deletions(-) diff --git a/mysql-test/disabled.def b/mysql-test/disabled.def index 5c4b742..f0cb6ec 100644 --- a/mysql-test/disabled.def +++ b/mysql-test/disabled.def @@ -6,17 +6,82 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# + <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## + tablespace : disabled in MariaDB (no TABLESPACE table attribute) events_time_zone : Test is not predictable as it depends on precise timing. read_many_rows_innodb : Bug#11748886 2010-11-15 mattiasj report already exists log_tables-big : Bug#11756699 2010-11-15 mattiasj report already exists mysql_embedded : Bug#12561297 2011-05-14 Anitha Dependent on PB2 changes - eventum#41836 -#show_explain : Psergey: random timeout in range-checked-for-each record query. ssl_crl_clients_valid : broken upstream ssl_crl : broken upstream ssl_crl_clrpath : broken upstream file_contents : MDEV-6526 these files are not installed anymore lowercase_fs_on : lower_case_table_names=0 is not an error until 10.1 + +########################## + ++ bootstrap : Modified on 2016-06-18 (MDEV-9969) ++ create_delayed : MDEV-10605 - failed with timeout ++ create_or_replace : Modified on 2016-06-23 (MDEV-9728) ++ ctype_recoding : Modified on 2016-06-10 (MDEV-10181) ++ ctype_utf8 : Modified on 2016-06-21 (merge) ++ ctype_utf8mb4 : Modified on 2016-06-21 (merge) ++ events_1 : Modified on 2016-06-21 (MDEV-9524) ++ func_group : Modified on 2016-08-08 (MDEV-10468) ++ func_in : Modified on 2016-06-20 (MDEV-10020) ++ func_math : Modified on 2016-08-10 (merge) ++ func_misc : Modified on 2016-08-10 (merge) ++ grant2 : Modified on 2016-07-18 (MDEV-8569) ++ help : Modified on 2016-06-21 (MDEV-9524) ++ host_cache_size_functionality : MDEV-10606 - sporadic failure on shutdown ++ index_intersect_innodb : MDEV-10643 - failed with timeout ++ index_merge_innodb : MDEV-7142 - sporadic wrong execution plan ++ information_schema_stats : Modified on 2016-07-25 (MDEV-10428) ++ innodb_mysql_lock : MDEV-7861 - sporadic lock detection failure ++ insert_innodb : Modified on 2016-06-14 (merge from upstream) ++ loaddata : Modified on 2016-08-10 (merge) ++ locale : Modified on 2016-06-21 (merge) ++ mdev-504 : MDEV-10607 - sporadic "can't connect" ++ mdev375 : MDEV-10607 - sporadic "can't connect" ++ merge : MDEV-10607 - sporadic "can't connect" ++ multi_update : Modified on 2016-06-20 (MDEV-5973) ++ myisam_enable_keys-10506 : Added on 2016-08-10 (MDEV-10506) ++ mysqlcheck : Modified on 2016-08-10 (merge) ++ mysqldump : MDEV-10512 - sporadic assertion failure ++ mysqltest : MDEV-9269 - fails on Alpha ++ named_pipe : Modified on 2016-08-02 (MDEV-10383) ++ openssl_1 : Modified on 2016-07-11 (MDEV-10211) ++ parser : Modified on 2016-06-21 (merge) ++ pool_of_threads : MDEV-10100 - sporadic error on detecting max connections ++ ps_1general : Modified on 2016-07-12 (merge) ++ range : Modified on 2016-08-10 (merge) ++ range_mrr_icp : Modified on 2016-08-10 (merge) ++ query_cache : MDEV-10611 - sporadic mutex problem ++ shutdown : MDEV-10612 - sporadic crashes ++ sp-prelocking : Modified on 2016-08-10 (merge) ++ sp-security : MDEV-10607 - sporadic "can't connect" ++ ssl : MDEV-10211 - different ciphers on some platforms ++ ssl_ca : Modified on 2016-07-11 (MDEV-10211) ++ ssl_compress : Modified on 2016-07-11 (MDEV-10211) ++ ssl_timeout : Modified on 2016-07-11 (MDEV-10211) ++ stat_tables_par_innodb : MDEV-10515 - sporadic wrong results ++ status_user : Modified on 2016-06-20 (MDEV-8633) ++ subselect_innodb : MDEV-10614 - sporadic wrong results ++ temp_table : Modified on 2016-06-18 (MDEV-8569) ++ type_date : Modified on 2016-08-10 (merge) ++ type_datetime : Modified on 2016-06-16 (MDEV-9374) ++ view : Modified on 2016-08-10 (merge) ++ xtradb_mrr : Modified on 2016-08-04 (MDEV-9946) diff --git a/mysql-test/lib/mtr_cases.pm b/mysql-test/lib/mtr_cases.pm index b76b10d..886e5a6 100644 --- a/mysql-test/lib/mtr_cases.pm +++ b/mysql-test/lib/mtr_cases.pm @@ -36,6 +36,7 @@ our $do_test; our $skip_test; our $binlog_format; our $enable_disabled; +our $enable_unstable; our $default_storage_engine; our $opt_with_ndbcluster_only; @@ -313,6 +314,9 @@ sub combinations_from_file($$) } our %disabled; +our %disabled_wildcards; +our %unstable; +our %unstable_wildcards; sub parse_disabled { my ($filename, $suitename) = @_; @@ -321,10 +325,21 @@ sub parse_disabled { chomp; next if /^\s*#/ or /^\s*$/; mtr_error("Syntax error in $filename line $.") - unless /^\s*(?:([-0-9A-Za-z_\/]+)\.)?([-0-9A-Za-z_]+)\s*:\s*(.*?)\s*$/; + unless /^\s*(\+)?\s*(?:([-0-9A-Za-z_\/]+)\.)?([-0-9A-Za-z_\*]+)\s*:\s*(.*?)\s*$/; mtr_error("Wrong suite name in $filename line $.") - if defined $1 and defined $suitename and $1 ne $suitename; - $disabled{($1 || $suitename || '') . ".$2"} = $3; + if defined $2 and defined $suitename and $2 ne $suitename; + $suitename= $2 || $suitename || ''; + my ($casename, $unstable_flag, $text)= ($3, $1, $4); + + if ($casename =~ /\*/) { + # Wildcard + $disabled_wildcards{"$casename"}= $text; + $unstable_wildcards{"$casename"}= 1 if defined $unstable_flag; + } + else { + $disabled{$suitename . ".$casename"}= $text; + $unstable{$suitename . ".$casename"}= 1 if defined $unstable_flag; + } } close DISABLED; } @@ -721,16 +736,36 @@ sub collect_one_test_case { # Check for disabled tests # ---------------------------------------------------------------------- my $disable = $disabled{".$tname"} || $disabled{$name}; + if (not $disable) { + foreach my $w (keys %disabled_wildcards) { + if ($tname =~ /^$w/) { + $disable= $disabled_wildcards{$w}; + last; + } + } + } + my $unstable = $unstable{".$tname"} || $unstable{$name}; + if (not $unstable) { + foreach my $w (keys %unstable_wildcards) { + if ($tname =~ /^$w/) { + $unstable= $unstable_wildcards{$w}; + last; + } + } + } if (not defined $disable and $suite->{parent}) { $disable = $disabled{$suite->{parent}->{name} . ".$tname"}; } + if (not defined $unstable and $suite->{parent}) { + $unstable = $unstable{$suite->{parent}->{name} . ".$tname"}; + } if (defined $disable) { $tinfo->{comment}= $disable; - if ( $enable_disabled ) + if ( $enable_disabled or ( $enable_unstable and $unstable ) ) { - # User has selected to run all disabled tests - mtr_report(" - $tinfo->{name} wil be run although it's been disabled\n", + # User has selected to run disabled tests + mtr_report(" - $tinfo->{name} will be run although it's been disabled" . ($unstable ? " as unstable\n" : "\n"), " due to '$disable'"); } else diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl index 5af19c8..82e84ec 100755 --- a/mysql-test/mysql-test-run.pl +++ b/mysql-test/mysql-test-run.pl @@ -1203,6 +1203,7 @@ sub command_line_setup { 'force-restart' => \$opt_force_restart, 'reorder!' => \$opt_reorder, 'enable-disabled' => \&collect_option, + 'enable-unstable' => \&collect_option, 'verbose+' => \$opt_verbose, 'verbose-restart' => \&report_option, 'sleep=i' => \$opt_sleep, @@ -1597,6 +1598,14 @@ sub command_line_setup { $ENV{'STAGING_RUN'}= $opt_staging_run; # -------------------------------------------------------------------------- + # Unstable tests + # -------------------------------------------------------------------------- + if ( $ENV{'MTR_ENABLE_UNSTABLE'} ) + { + collect_option('enable-unstable',1); + } + + # -------------------------------------------------------------------------- # Gcov flag # -------------------------------------------------------------------------- if ( ($opt_gcov or $opt_gprof) and ! $source_dist ) @@ -6455,7 +6464,8 @@ Options to control what test suites or cases to run twice to run only "big" tests. staging-run Run a limited number of tests (no slow tests). Used for running staging trees with valgrind. - enable-disabled Run also tests marked as disabled + enable-disabled Run also tests marked as disabled or unstable + enable-unstable Run also tests marked as unstable print-testcases Don't run the tests but print details about all the selected tests, in the order they would be run. skip-test-list=FILE Skip the tests listed in FILE. Each line in the file diff --git a/mysql-test/suite/binlog/disabled.def b/mysql-test/suite/binlog/disabled.def index 424e554..7a42685 100644 --- a/mysql-test/suite/binlog/disabled.def +++ b/mysql-test/suite/binlog/disabled.def @@ -6,8 +6,22 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## binlog_truncate_innodb : BUG#11764459 2010-10-20 anitha Originally disabled due to BUG#42643. Product bug fixed, but test changes needed binlog_spurious_ddl_errors : BUG#11761680 2013-01-18 astha Fixed on mysql-5.6 and trunk + +########################## + ++ binlog_commit_wait : sporadic failures: MDEV-10150 ++ binlog_dmls_on_tmp_tables_readonly : Added on 2016-05-04 (upstream) ++ binlog_xa_recover : sporadic failures: MDEV-8517 diff --git a/mysql-test/suite/federated/disabled.def b/mysql-test/suite/federated/disabled.def index bb2e4dd..279947b 100644 --- a/mysql-test/suite/federated/disabled.def +++ b/mysql-test/suite/federated/disabled.def @@ -6,6 +6,21 @@ # # <testcasename> : BUG#<xxxx> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## + +########################## + ++ federatedx : sporadic failures: MDEV-10617 ++ federated_innodb : sporadic failures: MDEV-10617, MDEV-8691 ++ federated_transactions : sporadic failures: MDEV-10617, MDEV-8691 ++ federated_innodb : sporadic failures: MDEV-8691 diff --git a/mysql-test/suite/funcs_1/disabled.def b/mysql-test/suite/funcs_1/disabled.def index 9bfe956..3cdc1c6 100644 --- a/mysql-test/suite/funcs_1/disabled.def +++ b/mysql-test/suite/funcs_1/disabled.def @@ -1,12 +1,22 @@ ############################################################################## # -# List the test cases that are to be disabled temporarely. +# List the test cases that are to be disabled temporarily. # # Separate the test case name and the comment with ':'. # -# <testcasename> : Comment test +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # -# Don't use any TAB characters for whitespace. +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. # ############################################################################## ++ processlist_priv_no_prot : Include file modified on 2016-07-12 (merge) ++ processlist_priv_ps : Include file modified on 2016-07-12 (merge) diff --git a/mysql-test/suite/innodb/disabled.def b/mysql-test/suite/innodb/disabled.def index 8cae44a..49e87c4 100644 --- a/mysql-test/suite/innodb/disabled.def +++ b/mysql-test/suite/innodb/disabled.def @@ -6,7 +6,28 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## +########################## + ++ innodb.binlog_consistent : sporadic failures: MDEV-10618 ++ innodb.innodb-alter-table : sporadic failures: MDEV-10619 ++ innodb.innodb-wl5522-debug-zip : sporadic failures: MDEV-10427 ++ innodb.innodb_bug30423 : sporadic failures: MDEV-7311 ++ innodb-alter-table : Modified on 2016-08-11 (MDEV-10535) ++ innodb-alter-tempfile : Modified on 2016-08-09 (MDEV-10469) ++ innodb-fk-warnings : Modified on 2016-07-18 (MDEV-8569) ++ innodb-fkcheck : Modified on 2016-06-13 (MDEV-10083) ++ innodb-wl5522 : rdiff file modified on 2016-08-10 (merge) ++ innodb_corrupt_bit : Modified on 2016-06-21 (merge) + diff --git a/mysql-test/suite/parts/disabled.def b/mysql-test/suite/parts/disabled.def index abbdbf7..17e2855 100644 --- a/mysql-test/suite/parts/disabled.def +++ b/mysql-test/suite/parts/disabled.def @@ -6,6 +6,14 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -15,3 +23,8 @@ parts.partition_mgm_lc1_archive : MDEV-5077 2013-09-27 svoj Cannot exchange part parts.partition_exchange_archive: MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table parts.partition_mgm_lc0_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table parts.partition_mgm_lc2_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table + +########################## + ++ partition_float_myisam : sporadic failures: MDEV-10621 ++ partition_int_myisam : sporadic failures: MDEV-10621 diff --git a/mysql-test/suite/perfschema/disabled.def b/mysql-test/suite/perfschema/disabled.def index d9e9e5b..8b4e222 100644 --- a/mysql-test/suite/perfschema/disabled.def +++ b/mysql-test/suite/perfschema/disabled.def @@ -6,6 +6,16 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -22,3 +32,17 @@ binlog_edge_stmt: needs slave_relay_log_info binlog_ok_mix: needs slave_relay_log_info binlog_ok_row: needs slave_relay_log_info binlog_ok_stmt: needs slave_relay_log_info + +########################## + ++ digest_table_full : Modified on 2016-06-21 (merge) ++ func_file_io : MDEV-5708 - fails for s390x ++ func_mutex : MDEV-5708 - fails for s390x ++ rpl_gtid_func : Modified on 2016-06-21 (merge) ++ sizing_low : Modified on 2016-04-26 (5.6.30 merge) ++ socket_summary_by_event_name_func : sporadic failures: MDEV-10622 ++ start_server_low_digest : Modified on 2016-06-21 (merge) ++ statement_digest : Modified on 2016-06-21 (merge) ++ statement_digest_consumers : Modified on 2016-06-21 (merge) ++ statement_digest_long_query : Modified on 2016-06-21 (merge) ++ table_name : Added on 2016-04-26 (5.6.30 merge) diff --git a/mysql-test/suite/rpl/disabled.def b/mysql-test/suite/rpl/disabled.def index de3091a..1d68be9 100644 --- a/mysql-test/suite/rpl/disabled.def +++ b/mysql-test/suite/rpl/disabled.def @@ -6,6 +6,14 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -13,3 +21,34 @@ rpl_spec_variables : BUG#11755836 2009-10-27 jasonh rpl_spec_variables fails on PB2 hpux rpl_get_master_version_and_clock : Bug#11766137 Jan 05 2011 joro Valgrind warnings rpl_get_master_version_and_clock rpl_partition_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table + +########################## + ++ last_insert_id : sporadic failures: MDEV-10625 ++ rpl_auto_increment : sporadic failures: MDEV-8691 ++ rpl_auto_increment_bug45679 : sporadic failures: MDEV-8691 ++ rpl_auto_increment_update_failure : sporadic failures: MDEV-10625 ++ rpl_binlog_index : sporadic failures: MDEV-9501 ++ rpl_checksum_cache : sporadic failures: MDEV-10626 ++ rpl_circular_for_4_hosts : sporadic failures: MDEV-10627 ++ rpl_gtid_crash : sporadic failures: MDEV-9501 ++ rpl_gtid_master_promote : sporadic failures: MDEV-10628 ++ rpl_gtid_stop_start : sporadic failures: MDEV-10629 ++ rpl_gtid_until : sporadic failures: MDEV-10625 ++ rpl_ignore_table : Modified on 2016-06-22 ++ rpl_insert : sporadic failures: MDEV-9329 ++ rpl_insert_delayed : sporadic failures: MDEV-9329 ++ rpl_invoked_features : sporadic failures: MDEV-8691 ++ rpl_mdev6020 : sporadic failures: MDEV-1630 ++ rpl_mdev6386 : sporadic failures: MDEV-10631 ++ rpl_parallel : sporadic failures: MDEV-10632 ++ rpl_parallel_temptable : sporadic failures: MDEV-10356 ++ rpl_row_drop_create_temp_table : sporadic failures: MDEV-10626 ++ rpl_row_sp001 : sporadic failures: MDEV-9329 ++ rpl_semi_sync_uninstall_plugin : sporadic failures: MDEV-7140 ++ rpl_slave_grp_exec : sporadic failures: MDEV-10514 ++ rpl_switch_stm_row_mixed : sporadic failures: MDEV-10611 ++ rpl_sync : sporadic failures: MDEV-10633 ++ rpl_temporary_error2 : sporadic failures: MDEV-10634 ++ sec_behind_master-5114 : sporadic failures: MDEV-8518 ++ rpl_skip_replication : sporadic failures: MDEV-9268 diff --git a/mysql-test/suite/sys_vars/disabled.def b/mysql-test/suite/sys_vars/disabled.def index 7582152..06c394c 100644 --- a/mysql-test/suite/sys_vars/disabled.def +++ b/mysql-test/suite/sys_vars/disabled.def @@ -6,9 +6,23 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## innodb_flush_checkpoint_debug_basic: removed from XtraDB-26.0 table_open_cache_instances_basic: no such variable in MariaDB + +########################## + ++ autocommit_func2 : sporadic failures: MDEV-9329 ++ general_log_file_basic : Modified on 2016-08-09 (MDEV-10465) ++ slow_query_log_file_basic : Modified on 2016-08-09 (MDEV-10465) diff --git a/storage/connect/mysql-test/connect/disabled.def b/storage/connect/mysql-test/connect/disabled.def index 9b45709..0ce6169 100644 --- a/storage/connect/mysql-test/connect/disabled.def +++ b/storage/connect/mysql-test/connect/disabled.def @@ -6,6 +6,14 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -14,3 +22,7 @@ jdbc : Variable settings depend on machine configuration jdbc_new : Variable settings depend on machine configuration jdbc_oracle : Variable settings depend on machine configuration jdbc_postgresql : Variable settings depend on machine configuration + +########################## + ++ tbl : MDEV-9844, MDEV-10179 - sporadic crashes, valgrind warnings, wrong results diff --git a/storage/mroonga/mysql-test/mroonga/storage/disabled.def b/storage/mroonga/mysql-test/mroonga/storage/disabled.def index 6866adc..c5c44db 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/disabled.def +++ b/storage/mroonga/mysql-test/mroonga/storage/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + alter_table_add_index_token_filters_one_token_filter : Bundled Mroonga does not support token filter yet. alter_table_change_token_filter : Bundled Mroonga does not support token filter yet. fulltext_token_filters_skip : Bundled Mroonga does not support token filter yet. @@ -8,3 +30,11 @@ create_table_token_filters_table_comment_one_token_filter : Bundled Mroonga does foreign_key_create : Bundled Mroonga does not support this test yet. partition_insert : Bundled Mroonga does not support this test yet. +########################## + ++ column_datetime_32bit_2038 : sporadic failures: MDEV-9270 ++ column_datetime_32bit_before_unix_epoch : sporadic failures: MDEV-9270 ++ column_datetime_32bit_max : sporadic failures: MDEV-9270 ++ column_datetime_32bit_out_of_range : sporadic failures: MDEV-9270 ++ index_multiple_column_unique_date_32bit_equal : sporadic failures: MDEV-9270 ++ index_multiple_column_unique_date_order_32bit_desc : sporadic failures: MDEV-9270 diff --git a/storage/tokudb/mysql-test/rpl/disabled.def b/storage/tokudb/mysql-test/rpl/disabled.def index 4c1a9a3..c58408e 100644 --- a/storage/tokudb/mysql-test/rpl/disabled.def +++ b/storage/tokudb/mysql-test/rpl/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# + <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + rpl_tokudb_delete_pk: unreliable, uses timestamp differences rpl_tokudb_delete_pk_lookup1: unreliable, uses timestamp differences rpl_tokudb_update_pk_uc0_lookup0: unreliable, uses timestamp differences @@ -13,3 +35,7 @@ rpl_tokudb_write_unique_uc1: unreliable, uses timestamp differences rpl_tokudb_read_only_ff: unreliable, uses timestamp differences rpl_tokudb_read_only_tf: unreliable, uses timestamp differences rpl_tokudb_read_only_tt: unreliable, uses timestamp differences + +########################## + ++ * : MDEV-9891 - massive crashes on shutdown diff --git a/storage/tokudb/mysql-test/tokudb/disabled.def b/storage/tokudb/mysql-test/tokudb/disabled.def index c98a8aa..3545544 100644 --- a/storage/tokudb/mysql-test/tokudb/disabled.def +++ b/storage/tokudb/mysql-test/tokudb/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + fast_update_binlog_mixed: No UPSERT in MariaDB 5.5 fast_update_binlog_row: No UPSERT in MariaDB 5.5 fast_update_binlog_statement: No UPSERT in MariaDB 5.5 @@ -28,3 +50,10 @@ type_timestamp_explicit: cluster_key_part: engine options on partitioned tables i_s_tokudb_lock_waits_released: unstable, race conditions i_s_tokudb_locks_released: unstable, race conditions + +########################## + ++ background_job_manager : sporadic failures: MDEV-10327 ++ cluster_filter_unpack_varchar : sporadic failures: MDEV-10636 + ++ * : MDEV-9891 - massive crashes on shutdown diff --git a/storage/tokudb/mysql-test/tokudb_bugs/disabled.def b/storage/tokudb/mysql-test/tokudb_bugs/disabled.def index d0d1a47..6554b9a 100644 --- a/storage/tokudb/mysql-test/tokudb_bugs/disabled.def +++ b/storage/tokudb/mysql-test/tokudb_bugs/disabled.def @@ -1,2 +1,29 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + 5585: times out, too many huge insert...selects db233: different execution path in mariadb, debug_sync point is not hit + +########################## + ++ checkpoint_lock : sporadic failures: MDEV-10637 ++ checkpoint_lock_3 : sporadic failures: MDEV-10637 + ++ * : MDEV-9891 - massive crashes on shutdown diff --git a/storage/tokudb/mysql-test/tokudb_parts/disabled.def b/storage/tokudb/mysql-test/tokudb_parts/disabled.def index 3252a46..7df2ac7 100644 --- a/storage/tokudb/mysql-test/tokudb_parts/disabled.def +++ b/storage/tokudb/mysql-test/tokudb_parts/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# + <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + partition_max_parts_hash_tokudb: 5.6 test not merged yet partition_max_parts_inv_tokudb: 5.6 test not merged yet partition_max_parts_key_tokudb: 5.6 test not merged yet @@ -7,3 +29,7 @@ partition_max_sub_parts_key_list_tokudb: 5.6 test not merged yet partition_max_sub_parts_key_range_tokudb: 5.6 test not merged yet partition_max_sub_parts_list_tokudb: 5.6 test not merged yet partition_max_sub_parts_range_tokudb: 5.6 test not merged yet + +########################## + ++ * : MDEV-9891 - massive crashes on shutdown _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 1

Re: [Maria-developers] Patch for parallel replication DBUG_ASSERT check on DROP TABLE et al
by Kristian Nielsen 20 Aug '16

20 Aug '16

Monty writes: > Can you please check if this looks correct? > diff --git a/sql/mdl.cc b/sql/mdl.cc > index 28d2006..3b1f9f2 100644 > --- a/sql/mdl.cc > +++ b/sql/mdl.cc > @@ -17,6 +17,7 @@ > #include "sql_class.h" > #include "debug_sync.h" > #include "sql_array.h" > +#include "rpl_rli.h" > #include <hash.h> > #include <mysqld_error.h> > #include <mysql/plugin.h> > @@ -442,6 +443,7 @@ class MDL_lock > virtual void notify_conflicting_locks(MDL_context *ctx) = 0; > > virtual bitmap_t hog_lock_types_bitmap() const = 0; > + bool check_if_conflicting_replication_locks(MDL_context *ctx); This should probably be #ifdef NDEBUG, since you're only using it in debug build? > > /** List of granted tickets for this lock. */ > Ticket_list m_granted; > @@ -2290,6 +2292,44 @@ void MDL_scoped_lock::notify_conflicting_locks(MDL_context *ctx) > } > } > > +/** > + Check if there is any conflicting lock that could cause this thread > + to wait for another thread which is not ready to commit. > + This is always an error, as the upper level of parallel replication > + should not allow a scheduling of a conflicting DDL until all earlier > + transactions has commited. > + > + This function is only called for a slave using parallel replication > + and trying to get an exclusive lock for the table. > +*/ > + > +bool MDL_lock::check_if_conflicting_replication_locks(MDL_context *ctx) > +{ > + Ticket_iterator it(m_granted); > + MDL_ticket *conflicting_ticket; > + > + while ((conflicting_ticket= it++)) > + { > + if (conflicting_ticket->get_ctx() != ctx) > + { > + MDL_context *conflicting_ctx= conflicting_ticket->get_ctx(); > + > + /* > + If the conflicting thread is another parallel replication > + thread for the same master and it's not in commit stage, then > + the current transaction has started too early and something is > + seriously wrong. > + */ > + if (conflicting_ctx->get_thd()->rgi_slave && > + conflicting_ctx->get_thd()->rgi_slave->rli == > + ctx->get_thd()->rgi_slave->rli && > + !conflicting_ctx->get_thd()->rgi_slave->did_mark_start_commit) > + return 1; // Fatal error No, this is not correct. For example, the threads might be in different domains, or using different multi-source connections. You need a check like in thd_report_wait_for(). _If_ T1 and T2 are in the same parallel replication domain, _and_ T1 goes before T2, _and_ there is a DDL lock conflict but T1 has not started to commit, then it indicates a potential problem. Also, the comments are too vague, "Something is seriously wrong" is not a good description. What is wrong? Please describe the issue being checked more precisely. What is the role of each of the two threads involved? What is the situation being checked? Why is it wrong? Also, use some temporaries for conflicting_ctx->get_thd()->rgi_slave etc, so the condition becomes more readable. > + } > + } > + return 0; > +} > + > > /** > Acquire one lock with waiting for conflicting locks to go away if needed. > @@ -2355,6 +2395,11 @@ MDL_context::acquire_lock(MDL_request *mdl_request, ulong lock_wait_timeout) > if (lock->needs_notification(ticket) && lock_wait_timeout) > lock->notify_conflicting_locks(this); > > + DBUG_ASSERT(mdl_request->type != MDL_INTENTION_EXCLUSIVE || > + !(get_thd()->rgi_slave && > + get_thd()->rgi_slave->is_parallel_exec && > + lock->check_if_conflicting_replication_locks(this))); > + > mysql_prlock_unlock(&lock->m_rwlock); > > will_wait_for(ticket); > diff --git a/sql/mdl.h b/sql/mdl.h > index c4d792a..13de602 100644 > --- a/sql/mdl.h > +++ b/sql/mdl.h > @@ -910,7 +910,6 @@ class MDL_context > */ > MDL_wait_for_subgraph *m_waiting_for; > private: > - THD *get_thd() const { return m_owner->get_thd(); } > MDL_ticket *find_ticket(MDL_request *mdl_req, > enum_mdl_duration *duration); > void release_locks_stored_before(enum_mdl_duration duration, MDL_ticket *sentinel); > @@ -919,6 +918,7 @@ class MDL_context > MDL_ticket **out_ticket); > > public: > + THD *get_thd() const { return m_owner->get_thd(); } > void find_deadlock(); > > ulong get_thread_id() const { return thd_get_thread_id(get_thd()); } - Kristian.

2 4

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Rich Prohaska 18 Aug '16

18 Aug '16

Hello Kristian, I am working on a second variant of the kill query design that will only kill the pending lock request if any for the thd being killed. The previous design had a problem when killing the query that triggered the wait for call On Aug 15, 2016 7:26 PM, "Rich Prohaska" <prohaska7(a)gmail.com> wrote: Hello Kristian, I have a prototype of the TokuFT code that will cause ALL lock waiters to call their killed callback here: https://github.com/ prohaska7/tokuft/tree/kill_lockers On Mon, Aug 15, 2016 at 11:51 AM, Rich Prohaska <prohaska7(a)gmail.com> wrote: > Hello Kristian, > See attached snapshot of slave threads and tokudb locks. Thread 16 is > waiting for a tokudb lock held by thread 16, which is waiting for a tokudb > lock held by thread 14. Thread 14 is waiting for a prior transaction to > complete, presumably either thread 15 or 16. So, we have a deadlock that > tokudb can not detect because the ordering constraint is not available to > tokudb. I assume that the optimistic scheduler killed thread 16, but since > tokudb does not implement the kill_query function, the deadlock is only > resolved when the tokudb lock timer pops. > > On Mon, Aug 15, 2016 at 8:16 AM, Rich Prohaska <prohaska7(a)gmail.com> > wrote: > >> Hello Kristian, >> The simplest kill_query implementation for tokudb would just signal all >> of the pending lock request's condition variables. This would cause the >> killed callback to be called. A performance refinement, if necessary, >> would allow thread A (executing the kill_query function) to identify and >> signal a condition variable for a blocked thread B. >> >> On Mon, Aug 15, 2016 at 5:42 AM, Kristian Nielsen < >> knielsen(a)knielsen-hq.org> wrote: >> >>> Rich Prohaska <prohaska7(a)gmail.com> writes: >>> >>> > tokudb lock timeouts are resolving the replication stall. >>> unfortunately, >>> > the tokudb lock timeout is 4 seconds, so the throughput is almost zero. >>> >>> Yes. Sorry for not making it clear that my proof-of-concept patch was >>> incomplete... >>> >>> >> > I suspect that the poor slave replication performance for optimistic >>> >> > replication occurs because TokuDB does not implement the kill_query >>> >> > handlerton function. kill_handlerton gets called to resolve lock >>> wait >>> >>> >> Possibly, but I'm not sure it's that important. The kill will be >>> effective >>> >> as soon as the wait is over. >>> >>> No, you're absolutely right, after testing (and thinking) some more, I >>> realise that indeed the kill_query functionality is important. >>> >>> A possible scenario is, given transactions T1, T2, and T3 in that order: >>> >>> T3 acquires a lock on row R3, T2 similarly acquires R2. >>> Now T3 tries to acquire R2, but has to wait for T2 to release it. >>> Later T1 tries to acquire R3, also has to wait. >>> >>> At this point, we kill T3, since it is holding a lock (R3) needed by an >>> earlier transaction T1. However, T3 will not notice the kill until its >>> own >>> wait (on R2 held by T2) times out. T2 cannot release the lock because it >>> is >>> waiting for T1 to commit first. So we have a deadlock :-/ >>> >>> With InnoDB, the kill causes T3 to wake up immediately and roll back, so >>> that T1 can proceed without much delay. >>> >>> Ok, so something more is needed here. I see there is a killed_callback() >>> which seems to check for the kill, so I'm hoping that can be used with a >>> suitable wakeup of the offending lock_request (or all requests, >>> perhaps). But as I'm completely new to TokuDB, I still need some more >>> time >>> to read the code and try to understand how everything fits together... >>> >>> > TokuFT implements pessimistic locking and 2 phase locking algorithms. >>> This >>> > wiki describes locking and concurrency in a little more detail: >>> > https://github.com/percona/tokudb-engine/wiki/Transactions-a >>> nd-Concurrency. >>> >>> Thanks, this was quite helpful. >>> >>> > Yes, I think they are false positives since the thd_report_wait_for >>> API is >>> > called but it does NOT call the THD::awake function. >>> >>> Ah. Then it's probably normal, caused by the group-commit optimisation. >>> In >>> conservative mode, if two transactions T1 and T2 did not group commit on >>> the >>> master, then cannot be started in parallel on the slave. But T2 can >>> start as >>> soon as T1 has reached COMMIT. Thus, if T2 happens to conflict with T1, >>> there is a small window where T2 can need to wait on T1 until T1 has >>> completed its commit. >>> >>> Thanks, >>> >>> - Kristian. >>> >> >> >

2 2

Re: [Maria-developers] Freeing options & wrappers.
by Kristian Nielsen 16 Aug '16

16 Aug '16

Oleksandr Byelkin <sanja(a)montyprogram.com> writes: >>> Making something more I think do not worth it because: >>> 1) in 10.2 C/C will replace the library and it has no the problem >> But Connector-C should have exactly the same problem. If it frees the >> options on failed connect, it will break async API. If it does not, code >> that skips mysql_close() will leak. How does C/C solve this problem? >> If we are making any changes in 10.0, we should preferably make things work >> the same as they will work in 10.2. > Don't it has rewritten async operation? (at least I got such impression). My understanding is that Georg implemented all the async API also in connector-c. It should be using much the same code (mysql_async.c and such). But this is not about the implementation. This is about the semantics of how libmysqlclient works. My understanding is that connector-c is supposed to be binary compatible with libmysqlclient.so? So either connector-c removes any mysql_options() set when mysql_real_connect() fails without CLIENT_REMEMBER_OPTIONS. Or it does not. In the former case, there is the problem with MYSQL_OPT_NONBLOCK. In the latter case, the problem with leak when mysql_close() is not called. I really think you need to understand the problem fully, including how this is going to work in 10.2, before you make any changes to how the client library works... - Kristian.

2 3

Re: [Maria-developers] c56c966: MDEV-10563 Crash during shutdown in Master_info_index::any_slave_sql_running
by Kristian Nielsen 16 Aug '16

16 Aug '16

Vicențiu Ciorbaru <vicentiu(a)mariadb.org> writes: > This is undefined behavior: > // In GCC v6 this function _always_ returns 1. > ClassA::member_function() { > if (!this) > return 0; > return 1; > } Yeah, we shouldn't do this, agree. > make use of this construct. I have not changed the logic of the code, > however I am not sure whether "give_error_if_slave_running()" is supposed > to return TRUE if master_info_index is deleted (aka, we are performing a > shutdown). Probably there is no good reason to return TRUE. Unfortunately, there is a lot of code like this in replication, where it is not clear what the purpose is, but also very hard to be sure it's not needed. My guess would be - without knowing for sure - that the return value doesn't matter. This should only happen during shutdown anyway, and it seems to be used only for changing the values of some system-variables. If server is shutting down, then values of replication-related system-variables do not really matter. As to the patch, you're basically moving the check for NULL into all the callers. I'm not sure if the check is even needed for all callers, but then, I also do not think it's worth it to try and determine where it is needed and where not. But wouldn't it be nicer to instead change the give_error_if_slave_running() to be a normal function that takes the Master_info_index pointer as an argument (or a static method, if you want)? This way, you can keep the null-pointer check in the function. Also helps avoid a new bug if another caller is added. - Kristian.

2 1

[Maria-developers] Freeing options & wrappers.
by Oleksandr Byelkin 16 Aug '16

16 Aug '16

Hi! (sorry I forgot mailing list so resending the e-mail) So it is about https://jira.mariadb.org/browse/MDEV-10559 Original problem was that database drivers in Python & Perl (probably other) do not call mysql_close for MYSQL object if there was no connection happing (memory leak). MySQL has solution for this - freeing option on failed connection if user did not asked something else (we even has the same flag, but it did not work). So I took MySQL solution. But it appeared not so compatible with our async wrappers. Frankly speaking the code is difficult to read and debug because everything made over macros nor I have found decent description/overview of how it is done (thank you if somebody point to it) so I can not say that understand what it is doing, and all my suggestion based on how I understood it. So what we have: we allocate memory for some options set between mysql_init and mysql_real_connect, instead of "reall connect" called some wrapper which store some pointers to the options then call mysql_real_connect and if "real_connect" free them it fail on exit. I think that best solution is that wrapper 1) store current state of CLIENT_REMEMBER_OPTIONS 2) set the flag 3) call mysql_real_connect 4) if connect fail and there was not "remember options" then wrapper free options by itself Again, I am not 100% sure that it is possible nor really know the place were this should be added (if idea above ok, I probably find ho to make it).

2 4

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Kristian Nielsen 15 Aug '16

15 Aug '16

[Moving the discussion to maria-developers@, hope that is ok/makes sense...] Ok, so here is a proof-of-concept patch for this, which seems to make TokuDB work with optimistic parallel replication. The core of the patch is this line in lock_request.cc lock_wait_callback(callback_data, m_txnid, conflicts.get(i)); which ends up doing this: thd_report_wait_for (requesting_thd, blocking_thd); All the rest of the patch is just getting the right information around between the different parts of the code. I put this on top of Jocelyn Fournier's tokudb_rpl.rpl_parallel_optimistic patches, and pushed it on my github: https://github.com/knielsen/server/tree/toku_opr2 With this patch, the test case passes! So that's promising. Some things still left to do for this to be a good patch: - I think the callback needs to trigger also for an already waiting transaction, in case another transaction arrives later to contend for the same lock, but happens to get the lock earlier. I can look into this. - This patch needs linear time (in number of active transactions) per callback to find the THD from the TXNID, maybe that could be optimised. - Probably the new callback etc. needs some cleanup to better match TokuDB code organisation and style. - And testing, of course. I'll definitely need some help there, as I'm not familiar with how to run TokuDB efficiently. Any thoughts or comments? - Kristian.

2 8