- developers - lists.mariadb.org

Re: [Maria-developers] [Commits] 294961c: MDEV-10538: MariaDB fails to start without galera_recovery in systemd mode
by Sergey Vojtovich 24 Aug '16

24 Aug '16

Hi Nirbhay, Ok to push. I suggest that we avoid such overcomplicated constructs in our default systemd configuration. Consider integrating recovery into the server (I know it's not trivial) or creating "included" config for galera. Thanks, Sergey On Mon, Aug 22, 2016 at 06:38:07PM -0400, Nirbhay Choubey wrote: > revision-id: 294961cc4d4d55730e807b3f0a7f93a3dd4d4f7b (mariadb-10.1.16-14-g294961c) > parent(s): d0d99dec914d409d7889fcec2efd126edad10bde > author: Nirbhay Choubey > committer: Nirbhay Choubey > timestamp: 2016-08-22 18:38:06 -0400 > message: > > MDEV-10538: MariaDB fails to start without galera_recovery in systemd mode > > Update ExecStartPre scripts to not fail if 'galera_recovery' > script is not available. > > --- > support-files/mariadb.service.in | 6 ++++-- > support-files/mariadb@.service.in | 15 +++++++++------ > 2 files changed, 13 insertions(+), 8 deletions(-) > > diff --git a/support-files/mariadb.service.in b/support-files/mariadb.service.in > index 6b8b2ba..9b97172 100644 > --- a/support-files/mariadb.service.in > +++ b/support-files/mariadb.service.in > @@ -63,9 +63,11 @@ PermissionsStartOnly=true > # galera_recovery simply returns an empty string. In any case, however, > # the script is not expected to return with a non-zero status. > # It is always safe to unset _WSREP_START_POSITION environment variable. > +# Do not panic if galera_recovery script is not available. (MDEV-10538) > ExecStartPre=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION" > -ExecStartPre=/bin/sh -c "VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && \ > - systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1" > +ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \ > + VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] \ > + && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1" > > # Needed to create system tables etc. > # ExecStartPre=/usr/bin/mysql_install_db -u mysql > diff --git a/support-files/mariadb@.service.in b/support-files/mariadb@.service.in > index 965e852..e941f21 100644 > --- a/support-files/mariadb@.service.in > +++ b/support-files/mariadb@.service.in > @@ -70,16 +70,19 @@ PermissionsStartOnly=true > # galera_recovery simply returns an empty string. In any case, however, > # the script is not expected to return with a non-zero status. > # It is always safe to unset _WSREP_START_POSITION%I environment variable. > +# Do not panic if galera_recovery script is not available. (MDEV-10538) > ExecStartPre=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION%I" > -ExecStartPre=/bin/sh -c "VAR=`/usr/bin/galera_recovery \ > - --defaults-file=@INSTALL_SYSCONF2DIR@/my%I.cnf`; [ $? -eq 0 ] && \ > - systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > + > +ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \ > + VAR=`/usr/bin/galera_recovery --defaults-file=@INSTALL_SYSCONF2DIR@/my%I.cnf`; [ $? -eq 0 ] \ > + && systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > + > # Alternate: (remove ConditionPathExists above) > # use [mysqld.INSTANCENAME] as sections in my.cnf > # > -#ExecStartPre=/bin/sh -c "VAR=`/usr/bin/galera_recovery \ > -# --defaults-group-suffix=%I`; [ $? -eq 0 ] && \ > -# systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > +#ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \ > +# VAR=`/usr/bin/galera_recovery --defaults-group-suffix=%I`; [ $? -eq 0 ] \ > +# && systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > > # Needed to create system tables etc. > # ExecStartPre=/usr/bin/mysql_install_db -u mysql > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Please review MDEV-10604 Disable unstable MTR tests by default
by Elena Stepanova 24 Aug '16

24 Aug '16

Hi Sergey (Vojtovich), Please review the patch that we've discussed at length. Serg, FYI. The length of the "unstable" list shouldn't scare anyone. At the end, this task should cause no change in our current procedure, the tests will still be run in buildbot and hopefully on developers' local machines (and every external user who wants to run them can do it easily). The change should mainly help those who run tests as a part of their build process. The lists can be tuned later, now they have been created generously, to solve the most urgent issue in shortest time possible. Please note that it's still not a 100% guarantee that nothing will fail sporadically, I've already got an absolutely new random failure after disabling all known ones, so it will still require some iterations. Regards, Elena -------- Forwarded Message -------- Subject: [Commits] 5ac5050: MDEV-10604 Disable unstable MTR tests by default Date: Mon, 22 Aug 2016 14:44:30 +0300 (EEST) From: elenst(a)montyprogram.com Reply-To: maria-developers(a)lists.launchpad.net To: commits(a)mariadb.org revision-id: 5ac505064a75508213d4dbce388ff5ee62d1ad56 (mariadb-10.0.26-92-g5ac5050) parent(s): 05f61ba46046ca835071a73b9255e787dcce9255 author: Elena Stepanova committer: Elena Stepanova timestamp: 2016-08-22 14:38:27 +0300 message: MDEV-10604 Disable unstable MTR tests by default - add new syntax to disabled.def in MTR to distinguish "unstable" tests from disabled tests; - allow wildcards in test names in disabled.def; - add a new option enable-unstable and env option MTR_ENABLE_DISABLED; - mark "unstable" tests which have failed since the beginning of the year in buildbot on main 10.0 branch; - mark "unstable" tests which fail in Debian builds; - mark "unstable" tests which have been changed during the last two releases --- mysql-test/disabled.def | 67 +++++++++++++++++++- mysql-test/lib/mtr_cases.pm | 47 ++++++++++++-- mysql-test/mysql-test-run.pl | 12 +++- mysql-test/suite/binlog/disabled.def | 14 ++++ mysql-test/suite/federated/disabled.def | 15 +++++ mysql-test/suite/funcs_1/disabled.def | 16 ++++- mysql-test/suite/innodb/disabled.def | 21 ++++++ mysql-test/suite/parts/disabled.def | 13 ++++ mysql-test/suite/perfschema/disabled.def | 24 +++++++ mysql-test/suite/rpl/disabled.def | 39 ++++++++++++ mysql-test/suite/sys_vars/disabled.def | 14 ++++ storage/connect/mysql-test/connect/disabled.def | 12 ++++ .../mysql-test/mroonga/storage/disabled.def | 30 +++++++++ storage/tokudb/mysql-test/rpl/disabled.def | 26 ++++++++ storage/tokudb/mysql-test/tokudb/disabled.def | 29 +++++++++ storage/tokudb/mysql-test/tokudb_bugs/disabled.def | 27 ++++++++ .../tokudb/mysql-test/tokudb_parts/disabled.def | 26 ++++++++ 17 files changed, 421 insertions(+), 11 deletions(-) diff --git a/mysql-test/disabled.def b/mysql-test/disabled.def index 5c4b742..f0cb6ec 100644 --- a/mysql-test/disabled.def +++ b/mysql-test/disabled.def @@ -6,17 +6,82 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# + <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## + tablespace : disabled in MariaDB (no TABLESPACE table attribute) events_time_zone : Test is not predictable as it depends on precise timing. read_many_rows_innodb : Bug#11748886 2010-11-15 mattiasj report already exists log_tables-big : Bug#11756699 2010-11-15 mattiasj report already exists mysql_embedded : Bug#12561297 2011-05-14 Anitha Dependent on PB2 changes - eventum#41836 -#show_explain : Psergey: random timeout in range-checked-for-each record query. ssl_crl_clients_valid : broken upstream ssl_crl : broken upstream ssl_crl_clrpath : broken upstream file_contents : MDEV-6526 these files are not installed anymore lowercase_fs_on : lower_case_table_names=0 is not an error until 10.1 + +########################## + ++ bootstrap : Modified on 2016-06-18 (MDEV-9969) ++ create_delayed : MDEV-10605 - failed with timeout ++ create_or_replace : Modified on 2016-06-23 (MDEV-9728) ++ ctype_recoding : Modified on 2016-06-10 (MDEV-10181) ++ ctype_utf8 : Modified on 2016-06-21 (merge) ++ ctype_utf8mb4 : Modified on 2016-06-21 (merge) ++ events_1 : Modified on 2016-06-21 (MDEV-9524) ++ func_group : Modified on 2016-08-08 (MDEV-10468) ++ func_in : Modified on 2016-06-20 (MDEV-10020) ++ func_math : Modified on 2016-08-10 (merge) ++ func_misc : Modified on 2016-08-10 (merge) ++ grant2 : Modified on 2016-07-18 (MDEV-8569) ++ help : Modified on 2016-06-21 (MDEV-9524) ++ host_cache_size_functionality : MDEV-10606 - sporadic failure on shutdown ++ index_intersect_innodb : MDEV-10643 - failed with timeout ++ index_merge_innodb : MDEV-7142 - sporadic wrong execution plan ++ information_schema_stats : Modified on 2016-07-25 (MDEV-10428) ++ innodb_mysql_lock : MDEV-7861 - sporadic lock detection failure ++ insert_innodb : Modified on 2016-06-14 (merge from upstream) ++ loaddata : Modified on 2016-08-10 (merge) ++ locale : Modified on 2016-06-21 (merge) ++ mdev-504 : MDEV-10607 - sporadic "can't connect" ++ mdev375 : MDEV-10607 - sporadic "can't connect" ++ merge : MDEV-10607 - sporadic "can't connect" ++ multi_update : Modified on 2016-06-20 (MDEV-5973) ++ myisam_enable_keys-10506 : Added on 2016-08-10 (MDEV-10506) ++ mysqlcheck : Modified on 2016-08-10 (merge) ++ mysqldump : MDEV-10512 - sporadic assertion failure ++ mysqltest : MDEV-9269 - fails on Alpha ++ named_pipe : Modified on 2016-08-02 (MDEV-10383) ++ openssl_1 : Modified on 2016-07-11 (MDEV-10211) ++ parser : Modified on 2016-06-21 (merge) ++ pool_of_threads : MDEV-10100 - sporadic error on detecting max connections ++ ps_1general : Modified on 2016-07-12 (merge) ++ range : Modified on 2016-08-10 (merge) ++ range_mrr_icp : Modified on 2016-08-10 (merge) ++ query_cache : MDEV-10611 - sporadic mutex problem ++ shutdown : MDEV-10612 - sporadic crashes ++ sp-prelocking : Modified on 2016-08-10 (merge) ++ sp-security : MDEV-10607 - sporadic "can't connect" ++ ssl : MDEV-10211 - different ciphers on some platforms ++ ssl_ca : Modified on 2016-07-11 (MDEV-10211) ++ ssl_compress : Modified on 2016-07-11 (MDEV-10211) ++ ssl_timeout : Modified on 2016-07-11 (MDEV-10211) ++ stat_tables_par_innodb : MDEV-10515 - sporadic wrong results ++ status_user : Modified on 2016-06-20 (MDEV-8633) ++ subselect_innodb : MDEV-10614 - sporadic wrong results ++ temp_table : Modified on 2016-06-18 (MDEV-8569) ++ type_date : Modified on 2016-08-10 (merge) ++ type_datetime : Modified on 2016-06-16 (MDEV-9374) ++ view : Modified on 2016-08-10 (merge) ++ xtradb_mrr : Modified on 2016-08-04 (MDEV-9946) diff --git a/mysql-test/lib/mtr_cases.pm b/mysql-test/lib/mtr_cases.pm index b76b10d..886e5a6 100644 --- a/mysql-test/lib/mtr_cases.pm +++ b/mysql-test/lib/mtr_cases.pm @@ -36,6 +36,7 @@ our $do_test; our $skip_test; our $binlog_format; our $enable_disabled; +our $enable_unstable; our $default_storage_engine; our $opt_with_ndbcluster_only; @@ -313,6 +314,9 @@ sub combinations_from_file($$) } our %disabled; +our %disabled_wildcards; +our %unstable; +our %unstable_wildcards; sub parse_disabled { my ($filename, $suitename) = @_; @@ -321,10 +325,21 @@ sub parse_disabled { chomp; next if /^\s*#/ or /^\s*$/; mtr_error("Syntax error in $filename line $.") - unless /^\s*(?:([-0-9A-Za-z_\/]+)\.)?([-0-9A-Za-z_]+)\s*:\s*(.*?)\s*$/; + unless /^\s*(\+)?\s*(?:([-0-9A-Za-z_\/]+)\.)?([-0-9A-Za-z_\*]+)\s*:\s*(.*?)\s*$/; mtr_error("Wrong suite name in $filename line $.") - if defined $1 and defined $suitename and $1 ne $suitename; - $disabled{($1 || $suitename || '') . ".$2"} = $3; + if defined $2 and defined $suitename and $2 ne $suitename; + $suitename= $2 || $suitename || ''; + my ($casename, $unstable_flag, $text)= ($3, $1, $4); + + if ($casename =~ /\*/) { + # Wildcard + $disabled_wildcards{"$casename"}= $text; + $unstable_wildcards{"$casename"}= 1 if defined $unstable_flag; + } + else { + $disabled{$suitename . ".$casename"}= $text; + $unstable{$suitename . ".$casename"}= 1 if defined $unstable_flag; + } } close DISABLED; } @@ -721,16 +736,36 @@ sub collect_one_test_case { # Check for disabled tests # ---------------------------------------------------------------------- my $disable = $disabled{".$tname"} || $disabled{$name}; + if (not $disable) { + foreach my $w (keys %disabled_wildcards) { + if ($tname =~ /^$w/) { + $disable= $disabled_wildcards{$w}; + last; + } + } + } + my $unstable = $unstable{".$tname"} || $unstable{$name}; + if (not $unstable) { + foreach my $w (keys %unstable_wildcards) { + if ($tname =~ /^$w/) { + $unstable= $unstable_wildcards{$w}; + last; + } + } + } if (not defined $disable and $suite->{parent}) { $disable = $disabled{$suite->{parent}->{name} . ".$tname"}; } + if (not defined $unstable and $suite->{parent}) { + $unstable = $unstable{$suite->{parent}->{name} . ".$tname"}; + } if (defined $disable) { $tinfo->{comment}= $disable; - if ( $enable_disabled ) + if ( $enable_disabled or ( $enable_unstable and $unstable ) ) { - # User has selected to run all disabled tests - mtr_report(" - $tinfo->{name} wil be run although it's been disabled\n", + # User has selected to run disabled tests + mtr_report(" - $tinfo->{name} will be run although it's been disabled" . ($unstable ? " as unstable\n" : "\n"), " due to '$disable'"); } else diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl index 5af19c8..82e84ec 100755 --- a/mysql-test/mysql-test-run.pl +++ b/mysql-test/mysql-test-run.pl @@ -1203,6 +1203,7 @@ sub command_line_setup { 'force-restart' => \$opt_force_restart, 'reorder!' => \$opt_reorder, 'enable-disabled' => \&collect_option, + 'enable-unstable' => \&collect_option, 'verbose+' => \$opt_verbose, 'verbose-restart' => \&report_option, 'sleep=i' => \$opt_sleep, @@ -1597,6 +1598,14 @@ sub command_line_setup { $ENV{'STAGING_RUN'}= $opt_staging_run; # -------------------------------------------------------------------------- + # Unstable tests + # -------------------------------------------------------------------------- + if ( $ENV{'MTR_ENABLE_UNSTABLE'} ) + { + collect_option('enable-unstable',1); + } + + # -------------------------------------------------------------------------- # Gcov flag # -------------------------------------------------------------------------- if ( ($opt_gcov or $opt_gprof) and ! $source_dist ) @@ -6455,7 +6464,8 @@ Options to control what test suites or cases to run twice to run only "big" tests. staging-run Run a limited number of tests (no slow tests). Used for running staging trees with valgrind. - enable-disabled Run also tests marked as disabled + enable-disabled Run also tests marked as disabled or unstable + enable-unstable Run also tests marked as unstable print-testcases Don't run the tests but print details about all the selected tests, in the order they would be run. skip-test-list=FILE Skip the tests listed in FILE. Each line in the file diff --git a/mysql-test/suite/binlog/disabled.def b/mysql-test/suite/binlog/disabled.def index 424e554..7a42685 100644 --- a/mysql-test/suite/binlog/disabled.def +++ b/mysql-test/suite/binlog/disabled.def @@ -6,8 +6,22 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## binlog_truncate_innodb : BUG#11764459 2010-10-20 anitha Originally disabled due to BUG#42643. Product bug fixed, but test changes needed binlog_spurious_ddl_errors : BUG#11761680 2013-01-18 astha Fixed on mysql-5.6 and trunk + +########################## + ++ binlog_commit_wait : sporadic failures: MDEV-10150 ++ binlog_dmls_on_tmp_tables_readonly : Added on 2016-05-04 (upstream) ++ binlog_xa_recover : sporadic failures: MDEV-8517 diff --git a/mysql-test/suite/federated/disabled.def b/mysql-test/suite/federated/disabled.def index bb2e4dd..279947b 100644 --- a/mysql-test/suite/federated/disabled.def +++ b/mysql-test/suite/federated/disabled.def @@ -6,6 +6,21 @@ # # <testcasename> : BUG#<xxxx> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## + +########################## + ++ federatedx : sporadic failures: MDEV-10617 ++ federated_innodb : sporadic failures: MDEV-10617, MDEV-8691 ++ federated_transactions : sporadic failures: MDEV-10617, MDEV-8691 ++ federated_innodb : sporadic failures: MDEV-8691 diff --git a/mysql-test/suite/funcs_1/disabled.def b/mysql-test/suite/funcs_1/disabled.def index 9bfe956..3cdc1c6 100644 --- a/mysql-test/suite/funcs_1/disabled.def +++ b/mysql-test/suite/funcs_1/disabled.def @@ -1,12 +1,22 @@ ############################################################################## # -# List the test cases that are to be disabled temporarely. +# List the test cases that are to be disabled temporarily. # # Separate the test case name and the comment with ':'. # -# <testcasename> : Comment test +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # -# Don't use any TAB characters for whitespace. +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. # ############################################################################## ++ processlist_priv_no_prot : Include file modified on 2016-07-12 (merge) ++ processlist_priv_ps : Include file modified on 2016-07-12 (merge) diff --git a/mysql-test/suite/innodb/disabled.def b/mysql-test/suite/innodb/disabled.def index 8cae44a..49e87c4 100644 --- a/mysql-test/suite/innodb/disabled.def +++ b/mysql-test/suite/innodb/disabled.def @@ -6,7 +6,28 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## +########################## + ++ innodb.binlog_consistent : sporadic failures: MDEV-10618 ++ innodb.innodb-alter-table : sporadic failures: MDEV-10619 ++ innodb.innodb-wl5522-debug-zip : sporadic failures: MDEV-10427 ++ innodb.innodb_bug30423 : sporadic failures: MDEV-7311 ++ innodb-alter-table : Modified on 2016-08-11 (MDEV-10535) ++ innodb-alter-tempfile : Modified on 2016-08-09 (MDEV-10469) ++ innodb-fk-warnings : Modified on 2016-07-18 (MDEV-8569) ++ innodb-fkcheck : Modified on 2016-06-13 (MDEV-10083) ++ innodb-wl5522 : rdiff file modified on 2016-08-10 (merge) ++ innodb_corrupt_bit : Modified on 2016-06-21 (merge) + diff --git a/mysql-test/suite/parts/disabled.def b/mysql-test/suite/parts/disabled.def index abbdbf7..17e2855 100644 --- a/mysql-test/suite/parts/disabled.def +++ b/mysql-test/suite/parts/disabled.def @@ -6,6 +6,14 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -15,3 +23,8 @@ parts.partition_mgm_lc1_archive : MDEV-5077 2013-09-27 svoj Cannot exchange part parts.partition_exchange_archive: MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table parts.partition_mgm_lc0_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table parts.partition_mgm_lc2_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table + +########################## + ++ partition_float_myisam : sporadic failures: MDEV-10621 ++ partition_int_myisam : sporadic failures: MDEV-10621 diff --git a/mysql-test/suite/perfschema/disabled.def b/mysql-test/suite/perfschema/disabled.def index d9e9e5b..8b4e222 100644 --- a/mysql-test/suite/perfschema/disabled.def +++ b/mysql-test/suite/perfschema/disabled.def @@ -6,6 +6,16 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -22,3 +32,17 @@ binlog_edge_stmt: needs slave_relay_log_info binlog_ok_mix: needs slave_relay_log_info binlog_ok_row: needs slave_relay_log_info binlog_ok_stmt: needs slave_relay_log_info + +########################## + ++ digest_table_full : Modified on 2016-06-21 (merge) ++ func_file_io : MDEV-5708 - fails for s390x ++ func_mutex : MDEV-5708 - fails for s390x ++ rpl_gtid_func : Modified on 2016-06-21 (merge) ++ sizing_low : Modified on 2016-04-26 (5.6.30 merge) ++ socket_summary_by_event_name_func : sporadic failures: MDEV-10622 ++ start_server_low_digest : Modified on 2016-06-21 (merge) ++ statement_digest : Modified on 2016-06-21 (merge) ++ statement_digest_consumers : Modified on 2016-06-21 (merge) ++ statement_digest_long_query : Modified on 2016-06-21 (merge) ++ table_name : Added on 2016-04-26 (5.6.30 merge) diff --git a/mysql-test/suite/rpl/disabled.def b/mysql-test/suite/rpl/disabled.def index de3091a..1d68be9 100644 --- a/mysql-test/suite/rpl/disabled.def +++ b/mysql-test/suite/rpl/disabled.def @@ -6,6 +6,14 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -13,3 +21,34 @@ rpl_spec_variables : BUG#11755836 2009-10-27 jasonh rpl_spec_variables fails on PB2 hpux rpl_get_master_version_and_clock : Bug#11766137 Jan 05 2011 joro Valgrind warnings rpl_get_master_version_and_clock rpl_partition_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table + +########################## + ++ last_insert_id : sporadic failures: MDEV-10625 ++ rpl_auto_increment : sporadic failures: MDEV-8691 ++ rpl_auto_increment_bug45679 : sporadic failures: MDEV-8691 ++ rpl_auto_increment_update_failure : sporadic failures: MDEV-10625 ++ rpl_binlog_index : sporadic failures: MDEV-9501 ++ rpl_checksum_cache : sporadic failures: MDEV-10626 ++ rpl_circular_for_4_hosts : sporadic failures: MDEV-10627 ++ rpl_gtid_crash : sporadic failures: MDEV-9501 ++ rpl_gtid_master_promote : sporadic failures: MDEV-10628 ++ rpl_gtid_stop_start : sporadic failures: MDEV-10629 ++ rpl_gtid_until : sporadic failures: MDEV-10625 ++ rpl_ignore_table : Modified on 2016-06-22 ++ rpl_insert : sporadic failures: MDEV-9329 ++ rpl_insert_delayed : sporadic failures: MDEV-9329 ++ rpl_invoked_features : sporadic failures: MDEV-8691 ++ rpl_mdev6020 : sporadic failures: MDEV-1630 ++ rpl_mdev6386 : sporadic failures: MDEV-10631 ++ rpl_parallel : sporadic failures: MDEV-10632 ++ rpl_parallel_temptable : sporadic failures: MDEV-10356 ++ rpl_row_drop_create_temp_table : sporadic failures: MDEV-10626 ++ rpl_row_sp001 : sporadic failures: MDEV-9329 ++ rpl_semi_sync_uninstall_plugin : sporadic failures: MDEV-7140 ++ rpl_slave_grp_exec : sporadic failures: MDEV-10514 ++ rpl_switch_stm_row_mixed : sporadic failures: MDEV-10611 ++ rpl_sync : sporadic failures: MDEV-10633 ++ rpl_temporary_error2 : sporadic failures: MDEV-10634 ++ sec_behind_master-5114 : sporadic failures: MDEV-8518 ++ rpl_skip_replication : sporadic failures: MDEV-9268 diff --git a/mysql-test/suite/sys_vars/disabled.def b/mysql-test/suite/sys_vars/disabled.def index 7582152..06c394c 100644 --- a/mysql-test/suite/sys_vars/disabled.def +++ b/mysql-test/suite/sys_vars/disabled.def @@ -6,9 +6,23 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## innodb_flush_checkpoint_debug_basic: removed from XtraDB-26.0 table_open_cache_instances_basic: no such variable in MariaDB + +########################## + ++ autocommit_func2 : sporadic failures: MDEV-9329 ++ general_log_file_basic : Modified on 2016-08-09 (MDEV-10465) ++ slow_query_log_file_basic : Modified on 2016-08-09 (MDEV-10465) diff --git a/storage/connect/mysql-test/connect/disabled.def b/storage/connect/mysql-test/connect/disabled.def index 9b45709..0ce6169 100644 --- a/storage/connect/mysql-test/connect/disabled.def +++ b/storage/connect/mysql-test/connect/disabled.def @@ -6,6 +6,14 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -14,3 +22,7 @@ jdbc : Variable settings depend on machine configuration jdbc_new : Variable settings depend on machine configuration jdbc_oracle : Variable settings depend on machine configuration jdbc_postgresql : Variable settings depend on machine configuration + +########################## + ++ tbl : MDEV-9844, MDEV-10179 - sporadic crashes, valgrind warnings, wrong results diff --git a/storage/mroonga/mysql-test/mroonga/storage/disabled.def b/storage/mroonga/mysql-test/mroonga/storage/disabled.def index 6866adc..c5c44db 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/disabled.def +++ b/storage/mroonga/mysql-test/mroonga/storage/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + alter_table_add_index_token_filters_one_token_filter : Bundled Mroonga does not support token filter yet. alter_table_change_token_filter : Bundled Mroonga does not support token filter yet. fulltext_token_filters_skip : Bundled Mroonga does not support token filter yet. @@ -8,3 +30,11 @@ create_table_token_filters_table_comment_one_token_filter : Bundled Mroonga does foreign_key_create : Bundled Mroonga does not support this test yet. partition_insert : Bundled Mroonga does not support this test yet. +########################## + ++ column_datetime_32bit_2038 : sporadic failures: MDEV-9270 ++ column_datetime_32bit_before_unix_epoch : sporadic failures: MDEV-9270 ++ column_datetime_32bit_max : sporadic failures: MDEV-9270 ++ column_datetime_32bit_out_of_range : sporadic failures: MDEV-9270 ++ index_multiple_column_unique_date_32bit_equal : sporadic failures: MDEV-9270 ++ index_multiple_column_unique_date_order_32bit_desc : sporadic failures: MDEV-9270 diff --git a/storage/tokudb/mysql-test/rpl/disabled.def b/storage/tokudb/mysql-test/rpl/disabled.def index 4c1a9a3..c58408e 100644 --- a/storage/tokudb/mysql-test/rpl/disabled.def +++ b/storage/tokudb/mysql-test/rpl/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# + <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + rpl_tokudb_delete_pk: unreliable, uses timestamp differences rpl_tokudb_delete_pk_lookup1: unreliable, uses timestamp differences rpl_tokudb_update_pk_uc0_lookup0: unreliable, uses timestamp differences @@ -13,3 +35,7 @@ rpl_tokudb_write_unique_uc1: unreliable, uses timestamp differences rpl_tokudb_read_only_ff: unreliable, uses timestamp differences rpl_tokudb_read_only_tf: unreliable, uses timestamp differences rpl_tokudb_read_only_tt: unreliable, uses timestamp differences + +########################## + ++ * : MDEV-9891 - massive crashes on shutdown diff --git a/storage/tokudb/mysql-test/tokudb/disabled.def b/storage/tokudb/mysql-test/tokudb/disabled.def index c98a8aa..3545544 100644 --- a/storage/tokudb/mysql-test/tokudb/disabled.def +++ b/storage/tokudb/mysql-test/tokudb/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + fast_update_binlog_mixed: No UPSERT in MariaDB 5.5 fast_update_binlog_row: No UPSERT in MariaDB 5.5 fast_update_binlog_statement: No UPSERT in MariaDB 5.5 @@ -28,3 +50,10 @@ type_timestamp_explicit: cluster_key_part: engine options on partitioned tables i_s_tokudb_lock_waits_released: unstable, race conditions i_s_tokudb_locks_released: unstable, race conditions + +########################## + ++ background_job_manager : sporadic failures: MDEV-10327 ++ cluster_filter_unpack_varchar : sporadic failures: MDEV-10636 + ++ * : MDEV-9891 - massive crashes on shutdown diff --git a/storage/tokudb/mysql-test/tokudb_bugs/disabled.def b/storage/tokudb/mysql-test/tokudb_bugs/disabled.def index d0d1a47..6554b9a 100644 --- a/storage/tokudb/mysql-test/tokudb_bugs/disabled.def +++ b/storage/tokudb/mysql-test/tokudb_bugs/disabled.def @@ -1,2 +1,29 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + 5585: times out, too many huge insert...selects db233: different execution path in mariadb, debug_sync point is not hit + +########################## + ++ checkpoint_lock : sporadic failures: MDEV-10637 ++ checkpoint_lock_3 : sporadic failures: MDEV-10637 + ++ * : MDEV-9891 - massive crashes on shutdown diff --git a/storage/tokudb/mysql-test/tokudb_parts/disabled.def b/storage/tokudb/mysql-test/tokudb_parts/disabled.def index 3252a46..7df2ac7 100644 --- a/storage/tokudb/mysql-test/tokudb_parts/disabled.def +++ b/storage/tokudb/mysql-test/tokudb_parts/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# + <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + partition_max_parts_hash_tokudb: 5.6 test not merged yet partition_max_parts_inv_tokudb: 5.6 test not merged yet partition_max_parts_key_tokudb: 5.6 test not merged yet @@ -7,3 +29,7 @@ partition_max_sub_parts_key_list_tokudb: 5.6 test not merged yet partition_max_sub_parts_key_range_tokudb: 5.6 test not merged yet partition_max_sub_parts_list_tokudb: 5.6 test not merged yet partition_max_sub_parts_range_tokudb: 5.6 test not merged yet + +########################## + ++ * : MDEV-9891 - massive crashes on shutdown _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 1

Re: [Maria-developers] Patch for parallel replication DBUG_ASSERT check on DROP TABLE et al
by Kristian Nielsen 20 Aug '16

20 Aug '16

Monty writes: > Can you please check if this looks correct? > diff --git a/sql/mdl.cc b/sql/mdl.cc > index 28d2006..3b1f9f2 100644 > --- a/sql/mdl.cc > +++ b/sql/mdl.cc > @@ -17,6 +17,7 @@ > #include "sql_class.h" > #include "debug_sync.h" > #include "sql_array.h" > +#include "rpl_rli.h" > #include <hash.h> > #include <mysqld_error.h> > #include <mysql/plugin.h> > @@ -442,6 +443,7 @@ class MDL_lock > virtual void notify_conflicting_locks(MDL_context *ctx) = 0; > > virtual bitmap_t hog_lock_types_bitmap() const = 0; > + bool check_if_conflicting_replication_locks(MDL_context *ctx); This should probably be #ifdef NDEBUG, since you're only using it in debug build? > > /** List of granted tickets for this lock. */ > Ticket_list m_granted; > @@ -2290,6 +2292,44 @@ void MDL_scoped_lock::notify_conflicting_locks(MDL_context *ctx) > } > } > > +/** > + Check if there is any conflicting lock that could cause this thread > + to wait for another thread which is not ready to commit. > + This is always an error, as the upper level of parallel replication > + should not allow a scheduling of a conflicting DDL until all earlier > + transactions has commited. > + > + This function is only called for a slave using parallel replication > + and trying to get an exclusive lock for the table. > +*/ > + > +bool MDL_lock::check_if_conflicting_replication_locks(MDL_context *ctx) > +{ > + Ticket_iterator it(m_granted); > + MDL_ticket *conflicting_ticket; > + > + while ((conflicting_ticket= it++)) > + { > + if (conflicting_ticket->get_ctx() != ctx) > + { > + MDL_context *conflicting_ctx= conflicting_ticket->get_ctx(); > + > + /* > + If the conflicting thread is another parallel replication > + thread for the same master and it's not in commit stage, then > + the current transaction has started too early and something is > + seriously wrong. > + */ > + if (conflicting_ctx->get_thd()->rgi_slave && > + conflicting_ctx->get_thd()->rgi_slave->rli == > + ctx->get_thd()->rgi_slave->rli && > + !conflicting_ctx->get_thd()->rgi_slave->did_mark_start_commit) > + return 1; // Fatal error No, this is not correct. For example, the threads might be in different domains, or using different multi-source connections. You need a check like in thd_report_wait_for(). _If_ T1 and T2 are in the same parallel replication domain, _and_ T1 goes before T2, _and_ there is a DDL lock conflict but T1 has not started to commit, then it indicates a potential problem. Also, the comments are too vague, "Something is seriously wrong" is not a good description. What is wrong? Please describe the issue being checked more precisely. What is the role of each of the two threads involved? What is the situation being checked? Why is it wrong? Also, use some temporaries for conflicting_ctx->get_thd()->rgi_slave etc, so the condition becomes more readable. > + } > + } > + return 0; > +} > + > > /** > Acquire one lock with waiting for conflicting locks to go away if needed. > @@ -2355,6 +2395,11 @@ MDL_context::acquire_lock(MDL_request *mdl_request, ulong lock_wait_timeout) > if (lock->needs_notification(ticket) && lock_wait_timeout) > lock->notify_conflicting_locks(this); > > + DBUG_ASSERT(mdl_request->type != MDL_INTENTION_EXCLUSIVE || > + !(get_thd()->rgi_slave && > + get_thd()->rgi_slave->is_parallel_exec && > + lock->check_if_conflicting_replication_locks(this))); > + > mysql_prlock_unlock(&lock->m_rwlock); > > will_wait_for(ticket); > diff --git a/sql/mdl.h b/sql/mdl.h > index c4d792a..13de602 100644 > --- a/sql/mdl.h > +++ b/sql/mdl.h > @@ -910,7 +910,6 @@ class MDL_context > */ > MDL_wait_for_subgraph *m_waiting_for; > private: > - THD *get_thd() const { return m_owner->get_thd(); } > MDL_ticket *find_ticket(MDL_request *mdl_req, > enum_mdl_duration *duration); > void release_locks_stored_before(enum_mdl_duration duration, MDL_ticket *sentinel); > @@ -919,6 +918,7 @@ class MDL_context > MDL_ticket **out_ticket); > > public: > + THD *get_thd() const { return m_owner->get_thd(); } > void find_deadlock(); > > ulong get_thread_id() const { return thd_get_thread_id(get_thd()); } - Kristian.

2 4

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Rich Prohaska 18 Aug '16

18 Aug '16

Hello Kristian, I am working on a second variant of the kill query design that will only kill the pending lock request if any for the thd being killed. The previous design had a problem when killing the query that triggered the wait for call On Aug 15, 2016 7:26 PM, "Rich Prohaska" <prohaska7(a)gmail.com> wrote: Hello Kristian, I have a prototype of the TokuFT code that will cause ALL lock waiters to call their killed callback here: https://github.com/ prohaska7/tokuft/tree/kill_lockers On Mon, Aug 15, 2016 at 11:51 AM, Rich Prohaska <prohaska7(a)gmail.com> wrote: > Hello Kristian, > See attached snapshot of slave threads and tokudb locks. Thread 16 is > waiting for a tokudb lock held by thread 16, which is waiting for a tokudb > lock held by thread 14. Thread 14 is waiting for a prior transaction to > complete, presumably either thread 15 or 16. So, we have a deadlock that > tokudb can not detect because the ordering constraint is not available to > tokudb. I assume that the optimistic scheduler killed thread 16, but since > tokudb does not implement the kill_query function, the deadlock is only > resolved when the tokudb lock timer pops. > > On Mon, Aug 15, 2016 at 8:16 AM, Rich Prohaska <prohaska7(a)gmail.com> > wrote: > >> Hello Kristian, >> The simplest kill_query implementation for tokudb would just signal all >> of the pending lock request's condition variables. This would cause the >> killed callback to be called. A performance refinement, if necessary, >> would allow thread A (executing the kill_query function) to identify and >> signal a condition variable for a blocked thread B. >> >> On Mon, Aug 15, 2016 at 5:42 AM, Kristian Nielsen < >> knielsen(a)knielsen-hq.org> wrote: >> >>> Rich Prohaska <prohaska7(a)gmail.com> writes: >>> >>> > tokudb lock timeouts are resolving the replication stall. >>> unfortunately, >>> > the tokudb lock timeout is 4 seconds, so the throughput is almost zero. >>> >>> Yes. Sorry for not making it clear that my proof-of-concept patch was >>> incomplete... >>> >>> >> > I suspect that the poor slave replication performance for optimistic >>> >> > replication occurs because TokuDB does not implement the kill_query >>> >> > handlerton function. kill_handlerton gets called to resolve lock >>> wait >>> >>> >> Possibly, but I'm not sure it's that important. The kill will be >>> effective >>> >> as soon as the wait is over. >>> >>> No, you're absolutely right, after testing (and thinking) some more, I >>> realise that indeed the kill_query functionality is important. >>> >>> A possible scenario is, given transactions T1, T2, and T3 in that order: >>> >>> T3 acquires a lock on row R3, T2 similarly acquires R2. >>> Now T3 tries to acquire R2, but has to wait for T2 to release it. >>> Later T1 tries to acquire R3, also has to wait. >>> >>> At this point, we kill T3, since it is holding a lock (R3) needed by an >>> earlier transaction T1. However, T3 will not notice the kill until its >>> own >>> wait (on R2 held by T2) times out. T2 cannot release the lock because it >>> is >>> waiting for T1 to commit first. So we have a deadlock :-/ >>> >>> With InnoDB, the kill causes T3 to wake up immediately and roll back, so >>> that T1 can proceed without much delay. >>> >>> Ok, so something more is needed here. I see there is a killed_callback() >>> which seems to check for the kill, so I'm hoping that can be used with a >>> suitable wakeup of the offending lock_request (or all requests, >>> perhaps). But as I'm completely new to TokuDB, I still need some more >>> time >>> to read the code and try to understand how everything fits together... >>> >>> > TokuFT implements pessimistic locking and 2 phase locking algorithms. >>> This >>> > wiki describes locking and concurrency in a little more detail: >>> > https://github.com/percona/tokudb-engine/wiki/Transactions-a >>> nd-Concurrency. >>> >>> Thanks, this was quite helpful. >>> >>> > Yes, I think they are false positives since the thd_report_wait_for >>> API is >>> > called but it does NOT call the THD::awake function. >>> >>> Ah. Then it's probably normal, caused by the group-commit optimisation. >>> In >>> conservative mode, if two transactions T1 and T2 did not group commit on >>> the >>> master, then cannot be started in parallel on the slave. But T2 can >>> start as >>> soon as T1 has reached COMMIT. Thus, if T2 happens to conflict with T1, >>> there is a small window where T2 can need to wait on T1 until T1 has >>> completed its commit. >>> >>> Thanks, >>> >>> - Kristian. >>> >> >> >

2 2

Re: [Maria-developers] Freeing options & wrappers.
by Kristian Nielsen 16 Aug '16

16 Aug '16

Oleksandr Byelkin <sanja(a)montyprogram.com> writes: >>> Making something more I think do not worth it because: >>> 1) in 10.2 C/C will replace the library and it has no the problem >> But Connector-C should have exactly the same problem. If it frees the >> options on failed connect, it will break async API. If it does not, code >> that skips mysql_close() will leak. How does C/C solve this problem? >> If we are making any changes in 10.0, we should preferably make things work >> the same as they will work in 10.2. > Don't it has rewritten async operation? (at least I got such impression). My understanding is that Georg implemented all the async API also in connector-c. It should be using much the same code (mysql_async.c and such). But this is not about the implementation. This is about the semantics of how libmysqlclient works. My understanding is that connector-c is supposed to be binary compatible with libmysqlclient.so? So either connector-c removes any mysql_options() set when mysql_real_connect() fails without CLIENT_REMEMBER_OPTIONS. Or it does not. In the former case, there is the problem with MYSQL_OPT_NONBLOCK. In the latter case, the problem with leak when mysql_close() is not called. I really think you need to understand the problem fully, including how this is going to work in 10.2, before you make any changes to how the client library works... - Kristian.

2 3

Re: [Maria-developers] c56c966: MDEV-10563 Crash during shutdown in Master_info_index::any_slave_sql_running
by Kristian Nielsen 16 Aug '16

16 Aug '16

Vicențiu Ciorbaru <vicentiu(a)mariadb.org> writes: > This is undefined behavior: > // In GCC v6 this function _always_ returns 1. > ClassA::member_function() { > if (!this) > return 0; > return 1; > } Yeah, we shouldn't do this, agree. > make use of this construct. I have not changed the logic of the code, > however I am not sure whether "give_error_if_slave_running()" is supposed > to return TRUE if master_info_index is deleted (aka, we are performing a > shutdown). Probably there is no good reason to return TRUE. Unfortunately, there is a lot of code like this in replication, where it is not clear what the purpose is, but also very hard to be sure it's not needed. My guess would be - without knowing for sure - that the return value doesn't matter. This should only happen during shutdown anyway, and it seems to be used only for changing the values of some system-variables. If server is shutting down, then values of replication-related system-variables do not really matter. As to the patch, you're basically moving the check for NULL into all the callers. I'm not sure if the check is even needed for all callers, but then, I also do not think it's worth it to try and determine where it is needed and where not. But wouldn't it be nicer to instead change the give_error_if_slave_running() to be a normal function that takes the Master_info_index pointer as an argument (or a static method, if you want)? This way, you can keep the null-pointer check in the function. Also helps avoid a new bug if another caller is added. - Kristian.

2 1

[Maria-developers] Freeing options & wrappers.
by Oleksandr Byelkin 16 Aug '16

16 Aug '16

Hi! (sorry I forgot mailing list so resending the e-mail) So it is about https://jira.mariadb.org/browse/MDEV-10559 Original problem was that database drivers in Python & Perl (probably other) do not call mysql_close for MYSQL object if there was no connection happing (memory leak). MySQL has solution for this - freeing option on failed connection if user did not asked something else (we even has the same flag, but it did not work). So I took MySQL solution. But it appeared not so compatible with our async wrappers. Frankly speaking the code is difficult to read and debug because everything made over macros nor I have found decent description/overview of how it is done (thank you if somebody point to it) so I can not say that understand what it is doing, and all my suggestion based on how I understood it. So what we have: we allocate memory for some options set between mysql_init and mysql_real_connect, instead of "reall connect" called some wrapper which store some pointers to the options then call mysql_real_connect and if "real_connect" free them it fail on exit. I think that best solution is that wrapper 1) store current state of CLIENT_REMEMBER_OPTIONS 2) set the flag 3) call mysql_real_connect 4) if connect fail and there was not "remember options" then wrapper free options by itself Again, I am not 100% sure that it is possible nor really know the place were this should be added (if idea above ok, I probably find ho to make it).

2 4

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Kristian Nielsen 16 Aug '16

16 Aug '16

[Moving the discussion to maria-developers@, hope that is ok/makes sense...] Ok, so here is a proof-of-concept patch for this, which seems to make TokuDB work with optimistic parallel replication. The core of the patch is this line in lock_request.cc lock_wait_callback(callback_data, m_txnid, conflicts.get(i)); which ends up doing this: thd_report_wait_for (requesting_thd, blocking_thd); All the rest of the patch is just getting the right information around between the different parts of the code. I put this on top of Jocelyn Fournier's tokudb_rpl.rpl_parallel_optimistic patches, and pushed it on my github: https://github.com/knielsen/server/tree/toku_opr2 With this patch, the test case passes! So that's promising. Some things still left to do for this to be a good patch: - I think the callback needs to trigger also for an already waiting transaction, in case another transaction arrives later to contend for the same lock, but happens to get the lock earlier. I can look into this. - This patch needs linear time (in number of active transactions) per callback to find the THD from the TXNID, maybe that could be optimised. - Probably the new callback etc. needs some cleanup to better match TokuDB code organisation and style. - And testing, of course. I'll definitely need some help there, as I'm not familiar with how to run TokuDB efficiently. Any thoughts or comments? - Kristian.

2 8

Re: [Maria-developers] Status of InnoDB 5.7 merge to 10.2
by Kristian Nielsen 13 Aug '16

13 Aug '16

Jan Lindström <jan.lindstrom(a)mariadb.com> writes: > I have reached to situation where I need help fixing remaining test > failures and mentoring is current status good enough for 10.2 Beta. I need > help on test failures on parts where I do not follow why test is failing or > server crashes I do not know how to fix. > > First of all current merge result is on bb-10.2-jan with > commit ca4f194460da469e29624d930e6d0027349d047c > > Higher level of status is on https://jira.mariadb.org/browse/MDEV-10024 > > Note that mtr thinks currently that innodb_plugin compiled using static > linking is xtradb (but not have_xtradb as xtradb is not compiled). > Furthermore galera is not compiled (waiting for codership). > > Here are current issues where I need help and possible persons who could > help: > > (1) Parallel replication lock wait/deadlock handling on InnoDB 5.7 with > refactored lock0lock.cc does not work > https://jira.mariadb.org/browse/MDEV-10550 (Monty, Kristian) >From a quick look at the patch, this seems to be because of code removed from innobase_kill_query(). Specifically: > @@ -4799,65 +5511,11 @@ innobase_kill_query( > DBUG_ENTER("innobase_kill_query"); > DBUG_ASSERT(hton == innodb_hton_ptr); > - if (!wsrep_thd_is_BF(trx->mysql_thd, FALSE) && > - trx->abort_type == TRX_SERVER_ABORT) { > - ut_ad(!lock_mutex_own()); > - lock_mutex_enter(); When parallel replication discovers a case where a transaction T1 goes to wait on a row lock held by a later (in the replication stream) transaction T2, T2 will be killed to avoid a deadlock. In this situation, we are already holding lock mutex. So this code is there in 10.1 to prevent re-taking the mutex, code that is removed here in your patch. > + if (trx != NULL) { > + /* Cancel a pending lock request if there are any */ > + lock_trx_handle_wait(trx); > +lock_trx_handle_wait( > +/*=================*/ > + trx_t* trx) /*!< in/out: trx lock state */ > { > dberr_t err; > - ulint heap_no; > > - ut_ad(!dict_index_is_clust(index)); > - ut_ad(!dict_index_is_online_ddl(index)); > - ut_ad(block->frame == page_align(rec)); > - ut_ad(page_rec_is_user_rec(rec) || page_rec_is_supremum(rec)); > - ut_ad(rec_offs_validate(rec, index, offsets)); > - ut_ad(mode == LOCK_X || mode == LOCK_S); > + lock_mutex_enter(); But in the new code here, the protection on lock_mutex_enter is no longer there, lock_mutex_enter() is done unconditionally. The code which sets the flag TRX_SERVER_ABORT seems to be still there, though: > +void > +lock_report_waiters_to_mysql( > +/*=======================*/ > + struct thd_wait_reports* waitee_buf_ptr, /*!< in: set of trxs */ > + THD* mysql_thd, /*!< in: THD */ > + trx_id_t victim_trx_id) /*!< in: Trx selected > + as deadlock victim, if > + any */ > +{ > + struct thd_wait_reports* p; > + struct thd_wait_reports* q; > + ulint i; > + > + p = waitee_buf_ptr; > + while (p) { > + i = 0; > + while (i < p->used) { > + trx_t *w_trx = p->waitees[i]; > + /* There is no need to report waits to a trx already > + selected as a victim. */ > + if (w_trx->id != victim_trx_id) { > + /* If thd_report_wait_for() decides to kill the > + transaction, then we will get a call back into > + innobase_kill_query. We mark this by setting > + current_lock_mutex_owner, so we can avoid trying > + to recursively take lock_sys->mutex. */ > + w_trx->abort_type = TRX_REPLICATION_ABORT; > + thd_report_wait_for(mysql_thd, w_trx->mysql_thd); > + w_trx->abort_type = TRX_SERVER_ABORT; abort_type is set to TRX_REPLICATION_ABORT around thd_report_wait_for(), during which innobase_kill_query() may be invoked with lock mutex already held. So it would appear that a first attempt should be to restore the protection around lock_mutex_enter()... Note that, if w_trx->abort_type is not functional in your current patch, the parallel replication code was originally working without this, you can see in the snippet above the comment about current_lock_mutex_owner, which was (mistakenly) not removed when this code was changed to use Galera stuff. So you can check git history and go back to the original code (using current_lock_mutex_owner) if necessary, perhaps. Or maybe w_trx->abort_type is fine, I'm not familiar with that change. > (2) mysqld: /home/jan/mysql/10.2/sql/handler.cc:2692: int > handler::ha_index_first(uchar*): Assertion `table_share->tmp_table != > NO_TMP_TABLE || m_lock_type != 2' failed. > https://jira.mariadb.org/browse/MDEV-10549 (Serg) > > (3) Some of the debug sync waits do not work InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10548 (Monty, Serg) > > (4) Test multi_update_innodb fails with InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10550 (Petrunia) > > (5) Test innodb.defrag_mdl-9155 hangs on InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10551 (Jan, Serg) > > (6) IS tables are not found on 10.2 InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10200 (Jan) Hope this helps, - Kristian.

2 1

[Maria-developers] MariaDB Developers Meetup
by Ian Gilfillan 10 Aug '16

10 Aug '16

The MariaDB Developers Meetup will be taking place from the 6th to the 8th of October in Amsterdam. https://mariadb.org/2016-mariadb-developers-meetup/ Details of where to sign up, and where to suggest topics for the meetup are on that page. Please feel free to suggest topic you're interested in presenting, as well as topics you'd like to see covered. The meetup is scheduled to be just after Percona Live, which also takes place in Amsterdam from the 3rd to the 5th. The Foundation has arranged a discount so that anyone signing up for the MariaDB Developers Meetup and that also wants to attend Percona Live will receive a discount for that event (the Developers Meetup has no charge). Percona Live are also looking for talks or sessions, so if you have a topic that would be of interest to the broader community, feel free to submit it at https://www.percona.com/live/plam16/program

1 1