Re: [Maria-developers] d8e3cfacebc: MDEV-14616: WSREP has not yet prepared node for application use error

15 Mar 2018

Hi, Jan!

On Feb 13, jan wrote:
...
revision-id: d8e3cfacebc47849532ef3e7bbf18f9da53bab88 (mariadb-10.1.31-20-gd8e3cfacebc)
parent(s): 474f3df092cc73342b2da03aa03ba5c66930f596
author: Jan Lindström
committer: Jan Lindström
timestamp: 2018-02-13 10:04:48 +0200
message:
MDEV-14616: WSREP has not yet prepared node for application use error
Contains following Galera fixes:
MW-405 Make sure wsrep is ready in wait_until_connected_again.inc
What is "MW-405", can I see the original bug report somewhere?
...
wait_until_connected_again issues 'SHOW STATUS' query repeatedly
       until mysqld replies without errors. However, SHOW STATUS is
       treated specially by wsrep in that it is allowed to proceed
       even if wsrep is not yet in ready state. As a consequence,
       after returning from wait_until_connected_again, wsrep may
       not be ready yet and subsequent queries may fail with error
       "1047 WSREP has not yet prepared node for application use".
       To avoid those errors, the patch includes
       wait_wsrep_ready.inc at the end of the wait_until_connected_again.inc
I don't quite understand the issue. Old code did wait_wsrep_ready.inc
after wait_until_connected_again.inc. Now wait_wsrep_ready.inc is
included in wait_until_connected_again.inc.

This seems to imply that the bug was "there were some cases where
wait_wsrep_ready.inc was not included after
wait_until_connected_again.inc".

There are, of course, two ways to fix such a bug. Either include
wait_wsrep_ready.inc after wait_until_connected_again.inc, or include
wait_wsrep_ready.inc in wait_until_connected_again.inc.

The latter fix implies that there are no possible cases when one would
want to wait for a server to be connected, but before wsrep is ready.
Are there? For the purpose of testing, that could be useful, could it
not?
...
kill_galera.inc can no longer rely on wait_until_connected_again.inc.
       This is because wait_until_connected_again now tries to make sure
       that the server it is connected eventually transition to ready state.
       Whereas some tests may need to kill galera while the server is in a
       non-primary view.
Which means that there are cases when a test needs the server connected,
but before wsrep is ready, right?
...
MW-408 Fix 'WSREP error while trying to determine node state'
mysql-test-run.pl sporadically reports 'WSREP error while
       trying to determine node state' right after starting servers
       for test execution. This happens because we try to execute a
       SELECT statement that queries the current value of status variable
       wsrep_ready. If this statement fails, the above message is reported.
       The failure is due to fact that wsrep may return error
       ER_LOCK_WAIT_TIMEOUT (on any SELECT statement) if it is not ready
       and wsrep_sync_wait enabled for SELECTs. The fix is to disable
       wsrep_sync_wait for the session that issues those SELECT statements.
1. Could you put this fix in a separate commit please?

2. Sorry, I don't understand. Where do "we try to execute a SELECT
statement that queries the current value of status variable wsrep_ready" ?
...
---
 mysql-test/include/restart_mysqld.inc             |   3 -
 mysql-test/include/start_mysqld.inc               |   3 -
 mysql-test/include/wait_until_connected_again.inc |  11 ++-
 mysql-test/mysql-test-run.pl                      | 104 ++++++++++++++++++++++
 mysql-test/suite/galera/include/kill_galera.inc   |   1 -
 5 files changed, 111 insertions(+), 11 deletions(-)

diff --git a/mysql-test/include/restart_mysqld.inc b/mysql-test/include/restart_mysqld.inc
index a0447280ff5..940e081c431 100644
--- a/mysql-test/include/restart_mysqld.inc
+++ b/mysql-test/include/restart_mysqld.inc
@@ -50,9 +50,6 @@ if (!$restart_parameters)
 # Call script that will poll the server waiting for it to be back online again
 --source include/wait_until_connected_again.inc
-# Wait for wsrep
---source include/wait_wsrep_ready.inc
-
 # Turn off reconnect again
 --disable_reconnect
diff --git a/mysql-test/include/start_mysqld.inc b/mysql-test/include/start_mysqld.inc
index 04dff714d49..e31f26aad8c 100644
--- a/mysql-test/include/start_mysqld.inc
+++ b/mysql-test/include/start_mysqld.inc
@@ -16,9 +16,6 @@ if (!$restart_parameters)
 # Call script that will poll the server waiting for it to be back online again
 --source include/wait_until_connected_again.inc
-# Wait for wsrep
---source include/wait_wsrep_ready.inc
-
 # Turn off reconnect again
 --disable_reconnect
diff --git a/mysql-test/include/wait_until_connected_again.inc b/mysql-test/include/wait_until_connected_again.inc
index 6f64ef45440..4958d276ebd 100644
--- a/mysql-test/include/wait_until_connected_again.inc
+++ b/mysql-test/include/wait_until_connected_again.inc
@@ -11,10 +11,7 @@ let $counter= 5000;
 let $mysql_errno= 9999;
 while ($mysql_errno)
 {
-  # Strangely enough, the server might return "Too many connections"
-  # while being shutdown, thus 1040 is an "allowed" error
-  # See BUG#36228
-  --error 0,1040,1053,2002,2003,2005,2006,2013,1927
+  --error 0,ER_SERVER_SHUTDOWN,ER_CONNECTION_KILLED,2002,2003,2006,2013
   show status;
dec $counter;
@@ -26,3 +23,9 @@ while ($mysql_errno)
 }
 --enable_query_log
 --enable_result_log
+
+# WSREP: SHOW STATUS queries are allowed even if wsrep
+#        is not ready. Make sure wsrep is ready before
+#        returning from this script
+
+--source include/wait_wsrep_ready.inc
diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl
index eaec51b82b4..324f2e98ebd 100755
--- a/mysql-test/mysql-test-run.pl
+++ b/mysql-test/mysql-test-run.pl
@@ -2883,6 +2883,49 @@ sub mysql_server_wait {
                                       $warn_seconds);
 }
+sub have_wsrep() {
+  my $wsrep_on= $mysqld_variables{'wsrep-on'};
+  return defined $wsrep_on
+}
+
+
+sub check_wsrep_support() {
+  if (have_wsrep())
+  {
+    mtr_report(" - binaries built with wsrep patch");
+
+    # ADD scripts to $PATH to that wsrep_sst_* can be found
+    my ($path) = grep { -f "$_/wsrep_sst_rsync"; } "$::bindir/scripts", $::path_client_bindir;
+    mtr_error("No SST scripts") unless $path;
+    $ENV{PATH}="$path:$ENV{PATH}";
+
+    # Check whether WSREP_PROVIDER environment variable is set.
+    if (defined $ENV{'WSREP_PROVIDER'}) {
+      if ((mtr_file_exists($ENV{'WSREP_PROVIDER'}) eq "")  &&
+          ($ENV{'WSREP_PROVIDER'} ne "none")) {
+        mtr_error("WSREP_PROVIDER env set to an invalid path");
+      }
+      # WSREP_PROVIDER is valid; set to a valid path or "none").
+      mtr_verbose("WSREP_PROVIDER env set to $ENV{'WSREP_PROVIDER'}");
+    } else {
+      # WSREP_PROVIDER env not defined. Lets try to locate the wsrep provider
+      # library.
+      my $file_wsrep_provider=
+        mtr_file_exists("/usr/lib/galera/libgalera_smm.so",
+                        "/usr/lib64/galera/libgalera_smm.so");
+
+      if ($file_wsrep_provider ne "") {
+        # wsrep provider library found !
+        mtr_verbose("wsrep provider library found : $file_wsrep_provider");
+        $ENV{'WSREP_PROVIDER'}= $file_wsrep_provider;
+      } else {
+        mtr_verbose("Could not find wsrep provider library, setting it to 'none'");
+        $ENV{'WSREP_PROVIDER'}= "none";
+      }
+    }
+  }
+}
+
 sub create_config_file_for_extern {
   my %opts=
     (
@@ -3341,6 +3384,62 @@ sub run_query {
   return $res
 }
+sub run_query_output {
+  my ($mysqld, $query, $outfile)= @_;
+
+  my $args;
+  mtr_init_args(\$args);
+  mtr_add_arg($args, "--defaults-file=%s", $path_config_file);
+  mtr_add_arg($args, "--defaults-group-suffix=%s", $mysqld->after('mysqld'));
+
+  mtr_add_arg($args, "--silent");
+  mtr_add_arg($args, "--execute=%s", $query);
+
+  my $res= My::SafeProcess->run
+    (
+     name          => "run_query_output -> ".$mysqld->name(),
+     path          => $exe_mysql,
+     args          => \$args,
+     output        => $outfile,
+     error         => $outfile
+    );
+
+  return $res
+}
+
+sub wait_wsrep_ready($$) {
+  my ($tinfo, $mysqld)= @_;
+
+  my $sleeptime= 100; # Milliseconds
+  my $loops= ($opt_start_timeout * 1000) / $sleeptime;
+
+  my $name= $mysqld->name();
+  my $outfile= "$opt_vardir/tmp/$name.wsrep_ready";
+  my $query= "SET SESSION wsrep_sync_wait = 0;
+              SELECT VARIABLE_VALUE
+              FROM INFORMATION_SCHEMA.GLOBAL_STATUS
+              WHERE VARIABLE_NAME = 'wsrep_ready'";
+
+  for (my $loop= 1; $loop <= $loops; $loop++)
+  {
+    if (run_query_output($mysqld, $query, $outfile) != 0)
+    {
+      $tinfo->{logfile}= "WSREP error while trying to determine node state";
+      return 0;
+    }
+
+    if (mtr_grab_file($outfile) =~ /^ON/)
+    {
+      unlink($outfile);
+      return 1;
+    }
+
+    mtr_milli_sleep($sleeptime);
+  }
+
+  $tinfo->{logfile}= "WSREP did not transition to state READY";
+  return 0;
+}
sub do_before_run_mysqltest($)
 {
@@ -5380,6 +5479,11 @@ sub start_servers($) {
       $tinfo->{comment}= "Failed to start ".$_->name() . "\n";
       return 1;
     }
+
+    if (have_wsrep() && !wait_wsrep_ready($tinfo, $_))
+    {
+      return 1;
+    }
   }
   return 0;
 }
diff --git a/mysql-test/suite/galera/include/kill_galera.inc b/mysql-test/suite/galera/include/kill_galera.inc
index c61bad8e19d..f95dccf2185 100644
--- a/mysql-test/suite/galera/include/kill_galera.inc
+++ b/mysql-test/suite/galera/include/kill_galera.inc
@@ -18,4 +18,3 @@
         exit(0);
 EOF
---source include/wait_until_disconnected.inc
_______________________________________________
commits mailing list
commits@mariadb.org
https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits
Regards,
Sergei
Chief Architect MariaDB
and security@mariadb.org

    

Re: [Maria-developers] d8e3cfacebc: MDEV-14616: WSREP has not yet prepared node for application use error

Sergei Golubchik