Re: [Maria-developers] Crash during shutdown (need help from PBXT team)
[I Cc'ed Paul, let me know if there is another prefered way of communicating, like using Launchpad bugs or some mailing list] Vladimir Kolesnikov <vladimir@primebase.org> writes:
"Crash during shutdown (need help from PBXT team)"
can you provide any details?
Yes, of course! Just been busy with other stuff, but very happy that you mention it on your own initiative! Basically, after merging PBXT I see crashes in the main.mysqld_option_err test case. Here is an example build from Buildbot showing this: http://askmonty.org/buildbot/builders/debian5-i386-fulltest/builds/52 http://askmonty.org/buildbot/builders/debian5-i386-fulltest/builds/52/steps/... The failure appears to be somewhat random, but I was able to repeat it easily enough locally using a release build. Debug build did not seem to show the problem. The issue is with the fix that Paul pushed for Bug#489088: https://bugs.launchpad.net/pbxt/+bug/489088 === modified file 'storage/pbxt/src/myxt_xt.cc' --- storage/pbxt/src/myxt_xt.cc 2009-11-27 15:37:02 +0000 +++ storage/pbxt/src/myxt_xt.cc 2009-12-01 09:50:46 +0000 @@ -3041,6 +3041,14 @@ xtPublic MX_CHARSET_INFO *myxt_getcharse return (MX_CHARSET_INFO *)&my_charset_utf8_general_ci; } +#ifdef DBUG_OFF +//typedef struct st_plugin_int *plugin_ref; +#define REF_MYSQL_PLUGIN(x) (x) +#else +//typedef struct st_plugin_int **plugin_ref; +#define REF_MYSQL_PLUGIN(x) (*(x)) +#endif + xtPublic void *myxt_create_thread() { #ifdef DRIZZLED + LEX_STRING& plugin_name = REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->name; + if ((plugin_name.length == 4) && (strncmp(plugin_name.str, "PBXT", plugin_name.length) == 0)) { + REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->ref_count--; + } new_thd->thread_stack = (char *) &new_thd; new_thd->store_globals(); lex_start(new_thd); This code crashes because new_thd->variables.table_plugin is NULL at this point in the code (or at least sometimes is). So this problem is similar to the previous problem handled just above in the code with global_system_variables.table_plugin being NULL. I tried a patch like this: === modified file 'storage/pbxt/src/myxt_xt.cc' --- storage/pbxt/src/myxt_xt.cc 2009-12-01 09:50:46 +0000 +++ storage/pbxt/src/myxt_xt.cc 2009-12-04 21:41:25 +0000 @@ -3112,10 +3112,15 @@ xtPublic void *myxt_create_thread() * references to the PBXT plugin object and will effectively deadlock the plugin so * that server will have to force plugin shutdown. To avoid deadlocking and forced shutdown * we must dereference the plugin after creating THD objects. + * Similarly to global_system_variables.table_plugin as described above, + * new_thd->valriables.table_plugin can also become NULL due to shutdown. */ - LEX_STRING& plugin_name = REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->name; - if ((plugin_name.length == 4) && (strncmp(plugin_name.str, "PBXT", plugin_name.length) == 0)) { - REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->ref_count--; + plugin_ref table_plugin = new_thd->variables.table_plugin; + if (table_plugin) { + LEX_STRING& plugin_name = REF_MYSQL_PLUGIN(table_plugin)->name; + if ((plugin_name.length == 4) && (strncmp(plugin_name.str, "PBXT", plugin_name.length) == 0)) { + REF_MYSQL_PLUGIN(table_plugin)->ref_count--; + } } new_thd->thread_stack = (char *) &new_thd; new_thd->store_globals(); So this fixes the crashes, but of course re-introduces the original problem in Bug#489088 with warnings about forced shutdown. I think you should be able to repeat the problem easily enough on Linux with a non-debug build running the test main.mysqld_option_err, but if not let me know and I will help set something up. - Kristian.
Hi Kristian, Unfortunately I cannot repeat the problem on my machine. Can you please put the branch to the debian box that you gave earlier access to? Thanks, Vladimir Kristian Nielsen wrote:
[I Cc'ed Paul, let me know if there is another prefered way of communicating, like using Launchpad bugs or some mailing list]
Vladimir Kolesnikov <vladimir@primebase.org> writes:
"Crash during shutdown (need help from PBXT team)"
can you provide any details?
Yes, of course! Just been busy with other stuff, but very happy that you mention it on your own initiative!
Basically, after merging PBXT I see crashes in the main.mysqld_option_err test case. Here is an example build from Buildbot showing this:
http://askmonty.org/buildbot/builders/debian5-i386-fulltest/builds/52 http://askmonty.org/buildbot/builders/debian5-i386-fulltest/builds/52/steps/...
The failure appears to be somewhat random, but I was able to repeat it easily enough locally using a release build. Debug build did not seem to show the problem.
The issue is with the fix that Paul pushed for Bug#489088:
https://bugs.launchpad.net/pbxt/+bug/489088
=== modified file 'storage/pbxt/src/myxt_xt.cc' --- storage/pbxt/src/myxt_xt.cc 2009-11-27 15:37:02 +0000 +++ storage/pbxt/src/myxt_xt.cc 2009-12-01 09:50:46 +0000 @@ -3041,6 +3041,14 @@ xtPublic MX_CHARSET_INFO *myxt_getcharse return (MX_CHARSET_INFO *)&my_charset_utf8_general_ci; }
+#ifdef DBUG_OFF +//typedef struct st_plugin_int *plugin_ref; +#define REF_MYSQL_PLUGIN(x) (x) +#else +//typedef struct st_plugin_int **plugin_ref; +#define REF_MYSQL_PLUGIN(x) (*(x)) +#endif + xtPublic void *myxt_create_thread() { #ifdef DRIZZLED
+ LEX_STRING& plugin_name = REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->name; + if ((plugin_name.length == 4) && (strncmp(plugin_name.str, "PBXT", plugin_name.length) == 0)) { + REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->ref_count--; + } new_thd->thread_stack = (char *) &new_thd; new_thd->store_globals(); lex_start(new_thd);
This code crashes because new_thd->variables.table_plugin is NULL at this point in the code (or at least sometimes is).
So this problem is similar to the previous problem handled just above in the code with global_system_variables.table_plugin being NULL.
I tried a patch like this:
=== modified file 'storage/pbxt/src/myxt_xt.cc' --- storage/pbxt/src/myxt_xt.cc 2009-12-01 09:50:46 +0000 +++ storage/pbxt/src/myxt_xt.cc 2009-12-04 21:41:25 +0000 @@ -3112,10 +3112,15 @@ xtPublic void *myxt_create_thread() * references to the PBXT plugin object and will effectively deadlock the plugin so * that server will have to force plugin shutdown. To avoid deadlocking and forced shutdown * we must dereference the plugin after creating THD objects. + * Similarly to global_system_variables.table_plugin as described above, + * new_thd->valriables.table_plugin can also become NULL due to shutdown. */ - LEX_STRING& plugin_name = REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->name; - if ((plugin_name.length == 4) && (strncmp(plugin_name.str, "PBXT", plugin_name.length) == 0)) { - REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->ref_count--; + plugin_ref table_plugin = new_thd->variables.table_plugin; + if (table_plugin) { + LEX_STRING& plugin_name = REF_MYSQL_PLUGIN(table_plugin)->name; + if ((plugin_name.length == 4) && (strncmp(plugin_name.str, "PBXT", plugin_name.length) == 0)) { + REF_MYSQL_PLUGIN(table_plugin)->ref_count--; + } } new_thd->thread_stack = (char *) &new_thd; new_thd->store_globals();
So this fixes the crashes, but of course re-introduces the original problem in Bug#489088 with warnings about forced shutdown.
I think you should be able to repeat the problem easily enough on Linux with a non-debug build running the test main.mysqld_option_err, but if not let me know and I will help set something up.
- Kristian.
Vladimir Kolesnikov <vladimir@primebase.org> writes:
Unfortunately I cannot repeat the problem on my machine. Can you please put the branch to the debian box that you gave earlier access to?
Yes. There is now a build directory on the box: /home/buildbot/shutdown-crash/mysql-5.1.41-MariaDB-beta/ (I pushed the corresponding bzr version to lp:~knielsen/maria/maria-pbxt-shutdown-crash. This is a merge of PBXT into MariaDB). I built using BUILD/compile-pentium-max I get the test failure repeatably with this: (cd mysql-test && perl mysql-test-run.pl mysqld_option_err) It does not seem to fail the same way every time, but it does seem to fail everytime I run it. Let me know if you need anything else. Thanks again for looking into this! - Kristian.
Kristian Nielsen wrote:
[I Cc'ed Paul, let me know if there is another prefered way of communicating, like using Launchpad bugs or some mailing list]
Vladimir Kolesnikov <vladimir@primebase.org> writes:
"Crash during shutdown (need help from PBXT team)"
can you provide any details?
Yes, of course! Just been busy with other stuff, but very happy that you mention it on your own initiative!
Basically, after merging PBXT I see crashes in the main.mysqld_option_err test case. Here is an example build from Buildbot showing this:
http://askmonty.org/buildbot/builders/debian5-i386-fulltest/builds/52 http://askmonty.org/buildbot/builders/debian5-i386-fulltest/builds/52/steps/...
The failure appears to be somewhat random, but I was able to repeat it easily enough locally using a release build. Debug build did not seem to show the problem.
The issue is with the fix that Paul pushed for Bug#489088:
https://bugs.launchpad.net/pbxt/+bug/489088
=== modified file 'storage/pbxt/src/myxt_xt.cc' --- storage/pbxt/src/myxt_xt.cc 2009-11-27 15:37:02 +0000 +++ storage/pbxt/src/myxt_xt.cc 2009-12-01 09:50:46 +0000 @@ -3041,6 +3041,14 @@ xtPublic MX_CHARSET_INFO *myxt_getcharse return (MX_CHARSET_INFO *)&my_charset_utf8_general_ci; } +#ifdef DBUG_OFF +//typedef struct st_plugin_int *plugin_ref; +#define REF_MYSQL_PLUGIN(x) (x) +#else +//typedef struct st_plugin_int **plugin_ref; +#define REF_MYSQL_PLUGIN(x) (*(x)) +#endif + xtPublic void *myxt_create_thread() { #ifdef DRIZZLED
+ LEX_STRING& plugin_name = REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->name; + if ((plugin_name.length == 4) && (strncmp(plugin_name.str, "PBXT", plugin_name.length) == 0)) { + REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->ref_count--; + } new_thd->thread_stack = (char *) &new_thd; new_thd->store_globals(); lex_start(new_thd);
This code crashes because new_thd->variables.table_plugin is NULL at this point in the code (or at least sometimes is).
So this problem is similar to the previous problem handled just above in the code with global_system_variables.table_plugin being NULL.
I tried a patch like this:
=== modified file 'storage/pbxt/src/myxt_xt.cc' --- storage/pbxt/src/myxt_xt.cc 2009-12-01 09:50:46 +0000 +++ storage/pbxt/src/myxt_xt.cc 2009-12-04 21:41:25 +0000 @@ -3112,10 +3112,15 @@ xtPublic void *myxt_create_thread() * references to the PBXT plugin object and will effectively deadlock the plugin so * that server will have to force plugin shutdown. To avoid deadlocking and forced shutdown * we must dereference the plugin after creating THD objects. + * Similarly to global_system_variables.table_plugin as described above, + * new_thd->valriables.table_plugin can also become NULL due to shutdown. */ - LEX_STRING& plugin_name = REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->name; - if ((plugin_name.length == 4) && (strncmp(plugin_name.str, "PBXT", plugin_name.length) == 0)) { - REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->ref_count--; + plugin_ref table_plugin = new_thd->variables.table_plugin; + if (table_plugin) { + LEX_STRING& plugin_name = REF_MYSQL_PLUGIN(table_plugin)->name; + if ((plugin_name.length == 4) && (strncmp(plugin_name.str, "PBXT", plugin_name.length) == 0)) { + REF_MYSQL_PLUGIN(table_plugin)->ref_count--; + } } new_thd->thread_stack = (char *) &new_thd; new_thd->store_globals();
So this fixes the crashes, but of course re-introduces the original problem in Bug#489088 with warnings about forced shutdown.
I think you should be able to repeat the problem easily enough on Linux with a non-debug build running the test main.mysqld_option_err, but if not let me know and I will help set something up.
- Kristian.
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Hi Kristian & Vlad, The crash also occurs immediately if MariaDB is started with an unknown command line parameter. In this case the server shuts down on startup, and new_thd- >variables.table_plugin is NULL as described below. On Dec 7, 2009, at 1:34 PM, Kristian Nielsen wrote: > Vladimir Kolesnikov <vladimir@primebase.org> writes: > >> Unfortunately I cannot repeat the problem on my machine. Can you >> please put the branch to the debian box that you gave earlier access >> to? > > Yes. > > There is now a build directory on the box: > > /home/buildbot/shutdown-crash/mysql-5.1.41-MariaDB-beta/ > > (I pushed the corresponding bzr version to > lp:~knielsen/maria/maria-pbxt-shutdown-crash. This is a merge of > PBXT into > MariaDB). > > I built using > > BUILD/compile-pentium-max > > I get the test failure repeatably with this: > > (cd mysql-test && perl mysql-test-run.pl mysqld_option_err) > > It does not seem to fail the same way every time, but it does seem > to fail > everytime I run it. > > Let me know if you need anything else. > > Thanks again for looking into this! > > - Kristian. > >> Kristian Nielsen wrote: >>> [I Cc'ed Paul, let me know if there is another prefered way of >>> communicating, >>> like using Launchpad bugs or some mailing list] >>> >>> Vladimir Kolesnikov <vladimir@primebase.org> writes: >>> >>> >>>> "Crash during shutdown (need help from PBXT team)" >>>> >>>> can you provide any details? >>>> >>> >>> Yes, of course! Just been busy with other stuff, but very happy >>> that you >>> mention it on your own initiative! >>> >>> Basically, after merging PBXT I see crashes in the >>> main.mysqld_option_err test >>> case. Here is an example build from Buildbot showing this: >>> >>> http://askmonty.org/buildbot/builders/debian5-i386-fulltest/builds/52 >>> http://askmonty.org/buildbot/builders/debian5-i386-fulltest/builds/52/steps/test_4/logs/stdio >>> >>> The failure appears to be somewhat random, but I was able to >>> repeat it easily >>> enough locally using a release build. Debug build did not seem to >>> show the >>> problem. >>> >>> The issue is with the fix that Paul pushed for Bug#489088: >>> >>> https://bugs.launchpad.net/pbxt/+bug/489088 >>> >>> === modified file 'storage/pbxt/src/myxt_xt.cc' >>> --- storage/pbxt/src/myxt_xt.cc 2009-11-27 15:37:02 +0000 >>> +++ storage/pbxt/src/myxt_xt.cc 2009-12-01 09:50:46 +0000 >>> @@ -3041,6 +3041,14 @@ xtPublic MX_CHARSET_INFO *myxt_getcharse >>> return (MX_CHARSET_INFO *)&my_charset_utf8_general_ci; >>> } >>> +#ifdef DBUG_OFF >>> +//typedef struct st_plugin_int *plugin_ref; >>> +#define REF_MYSQL_PLUGIN(x) (x) >>> +#else >>> +//typedef struct st_plugin_int **plugin_ref; >>> +#define REF_MYSQL_PLUGIN(x) (*(x)) >>> +#endif >>> + >>> xtPublic void *myxt_create_thread() >>> { >>> #ifdef DRIZZLED >>> >>> + LEX_STRING& plugin_name = REF_MYSQL_PLUGIN(new_thd- >>> >variables.table_plugin)->name; >>> + if ((plugin_name.length == 4) && (strncmp(plugin_name.str, >>> "PBXT", plugin_name.length) == 0)) { >>> + REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->ref_count--; >>> + } >>> new_thd->thread_stack = (char *) &new_thd; >>> new_thd->store_globals(); >>> lex_start(new_thd); >>> >>> This code crashes because new_thd->variables.table_plugin is NULL >>> at this >>> point in the code (or at least sometimes is). >>> >>> So this problem is similar to the previous problem handled just >>> above in the >>> code with global_system_variables.table_plugin being NULL. >>> >>> I tried a patch like this: >>> >>> === modified file 'storage/pbxt/src/myxt_xt.cc' >>> --- storage/pbxt/src/myxt_xt.cc 2009-12-01 09:50:46 +0000 >>> +++ storage/pbxt/src/myxt_xt.cc 2009-12-04 21:41:25 +0000 >>> @@ -3112,10 +3112,15 @@ xtPublic void *myxt_create_thread() >>> * references to the PBXT plugin object and will effectively >>> deadlock the plugin so * that server will have to force >>> plugin shutdown. To avoid deadlocking and forced shutdown * we >>> must dereference the plugin after creating THD objects. >>> + * Similarly to global_system_variables.table_plugin as >>> described above, >>> + * new_thd->valriables.table_plugin can also become NULL due to >>> shutdown. >>> */ >>> - LEX_STRING& plugin_name = REF_MYSQL_PLUGIN(new_thd- >>> >variables.table_plugin)->name; >>> - if ((plugin_name.length == 4) && (strncmp(plugin_name.str, >>> "PBXT", plugin_name.length) == 0)) { >>> - REF_MYSQL_PLUGIN(new_thd->variables.table_plugin)->ref_count--; >>> + plugin_ref table_plugin = new_thd->variables.table_plugin; >>> + if (table_plugin) { >>> + LEX_STRING& plugin_name = REF_MYSQL_PLUGIN(table_plugin)->name; >>> + if ((plugin_name.length == 4) && (strncmp(plugin_name.str, >>> "PBXT", plugin_name.length) == 0)) { >>> + REF_MYSQL_PLUGIN(table_plugin)->ref_count--; >>> + } >>> } >>> new_thd->thread_stack = (char *) &new_thd; >>> new_thd->store_globals(); >>> >>> So this fixes the crashes, but of course re-introduces the >>> original problem in >>> Bug#489088 with warnings about forced shutdown. >>> >>> I think you should be able to repeat the problem easily enough on >>> Linux with a >>> non-debug build running the test main.mysqld_option_err, but if >>> not let me >>> know and I will help set something up. >>> >>> - Kristian. >>> >> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~maria-developers >> Post to : maria-developers@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~maria-developers >> More help : https://help.launchpad.net/ListHelp > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-developers > Post to : maria-developers@lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-developers > More help : https://help.launchpad.net/ListHelp -- Paul McCullagh PrimeBase Technologies www.primebase.org www.blobstreaming.org pbxt.blogspot.com
participants (3)
-
Kristian Nielsen
-
Paul McCullagh
-
Vladimir Kolesnikov