[Maria-developers] TokuDB test suite failures when building at Launchpad
Hello! Are here any TokuDB devs who could help me debug why TokuDB does repeatedly fails to pass build suite tests when built on Launchpad.net? I repeatedly get these test failures when running MariaDB builds on Launchpad: rpl-tokudb.tokudb_innodb_xa_crash tokudb_bugs.mdev4533 tokudb.simple_join_tokudb_innodb tokudb.truncate_txn_rollback_innodb tokudb_bugs.5733_innodb The build run log can be viewed at https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.0/+builds?build_text=&build_state=all and it links to the individual build logs. All amd64 builds fail while all i386 (where TokuDB is disabled) pass OK. Sounds very much like a TokuDB-related issue. Or could the problem be something else and the TokuDB tests are just the first victim? Example of a recent log: https://launchpadlibrarian.net/196425361/buildlog_ubuntu-vivid-amd64.mariadb... One example of the test outputs: *** rpl-tokudb.tokudb_innodb_xa_crash 'mix,xtradb' [ fail ] Test ended at 2015-02-02 22:29:06 CURRENT_TEST: rpl-tokudb.tokudb_innodb_xa_crash Failed to start mysqld.1 mysqltest failed but provided no output - saving '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/' to '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/' - found 'core' (1/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace Compressed file /build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/mysqld.1/data/core ***Warnings generated in error logs during shutdown after running tests: rpl-tokudb.tokudb_innodb_xa_crash 150202 22:29:05 [ERROR] mysqld got signal 11 ; Attempting backtrace. You can use the following information to find out 150202 22:29:05 [ERROR] mysqld got signal 11 ; Attempting backtrace. You can use the following information to find out ***
mysqld crashed and left a core file. posting the stack traces from the core file would help. On Tue, Feb 3, 2015 at 6:08 AM, Otto Kekäläinen <otto@seravo.fi> wrote:
Hello!
Are here any TokuDB devs who could help me debug why TokuDB does repeatedly fails to pass build suite tests when built on Launchpad.net?
I repeatedly get these test failures when running MariaDB builds on Launchpad: rpl-tokudb.tokudb_innodb_xa_crash tokudb_bugs.mdev4533 tokudb.simple_join_tokudb_innodb tokudb.truncate_txn_rollback_innodb tokudb_bugs.5733_innodb
The build run log can be viewed at
https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.0/+builds?build_text=&build_state=all and it links to the individual build logs. All amd64 builds fail while all i386 (where TokuDB is disabled) pass OK. Sounds very much like a TokuDB-related issue. Or could the problem be something else and the TokuDB tests are just the first victim?
Example of a recent log:
https://launchpadlibrarian.net/196425361/buildlog_ubuntu-vivid-amd64.mariadb...
One example of the test outputs:
*** rpl-tokudb.tokudb_innodb_xa_crash 'mix,xtradb' [ fail ] Test ended at 2015-02-02 22:29:06
CURRENT_TEST: rpl-tokudb.tokudb_innodb_xa_crash
Failed to start mysqld.1 mysqltest failed but provided no output
- saving '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/' to '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/' - found 'core' (1/5)
Trying 'dbx' to get a backtrace
Trying 'gdb' to get a backtrace Compressed file
/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/mysqld.1/data/core ***Warnings generated in error logs during shutdown after running tests: rpl-tokudb.tokudb_innodb_xa_crash
150202 22:29:05 [ERROR] mysqld got signal 11 ; Attempting backtrace. You can use the following information to find out 150202 22:29:05 [ERROR] mysqld got signal 11 ; Attempting backtrace. You can use the following information to find out ***
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
2015-02-03 16:50 GMT+02:00 Rich Prohaska <prohaska7@gmail.com>:
mysqld crashed and left a core file. posting the stack traces from the core file would help.
Posting them is not possible, because the Launchpad build system does not leave any files behind, only the log. I've even asked Canonical staff if they can manually help us here, but they refused. Rick, is there anything else we can do? Does the failures give any hints on what is going on?
Hi, Otto! I can give you a patch that you apply and run another build in the Launchpad build system. Would that be ok? It'll be a purely debugging patch, not something that should end up in the final packages, so don't forget to remove it afterwards. On Feb 03, Otto Kekäläinen wrote:
2015-02-03 16:50 GMT+02:00 Rich Prohaska <prohaska7@gmail.com>:
mysqld crashed and left a core file. posting the stack traces from the core file would help.
Posting them is not possible, because the Launchpad build system does not leave any files behind, only the log. I've even asked Canonical staff if they can manually help us here, but they refused. Rick, is there anything else we can do? Does the failures give any hints on what is going on?
Regards, Sergei
Failing TokuDB tests are now stopping MariaDB 10.0 from entering Ubuntu, as the official builds fail too: https://launchpad.net/ubuntu/+source/mariadb-10.0/10.0.16-1 https://launchpadlibrarian.net/196473592/buildlog_ubuntu-vivid-amd64.mariadb... This time the failures are different: Completed: Failed 7/4551 tests, 99.85% were successful. Failing test(s): tokudb.rows-32m-0 tokudb.rows-32m-1 tokudb.savepoint-5 tokudb_bugs.frm_store tokudb_bugs.frm_store2 tokudb_bugs.frm_store3 plugins.show_all_plugins Others yield warnings. I have no idea what to do about these. Only fix I can do at the moment if to disable TokuDB completely. I remember that TokuDB has created a lot of work for me during the last 2 years or so. Could TokuDB devs consider setting up their own Launchpad PPA (it's free) and run development builds there every now and then, so that these issues would have a chance to be caught before release? Thanks
3 of the tests fail when they hit the test case time limit. either increase the limit, mark these tests as big tests, or disable these tests. these 3 tests are tokudb.rows-32m-0, tokudb.rows-32m-1 and tokudb.savepoint-5. On Wed, Feb 4, 2015 at 7:08 AM, Otto Kekäläinen <otto@seravo.fi> wrote:
Failing TokuDB tests are now stopping MariaDB 10.0 from entering Ubuntu, as the official builds fail too:
https://launchpad.net/ubuntu/+source/mariadb-10.0/10.0.16-1
https://launchpadlibrarian.net/196473592/buildlog_ubuntu-vivid-amd64.mariadb...
This time the failures are different:
Completed: Failed 7/4551 tests, 99.85% were successful. Failing test(s): tokudb.rows-32m-0 tokudb.rows-32m-1 tokudb.savepoint-5 tokudb_bugs.frm_store tokudb_bugs.frm_store2 tokudb_bugs.frm_store3 plugins.show_all_plugins
Others yield warnings. I have no idea what to do about these. Only fix I can do at the moment if to disable TokuDB completely.
I remember that TokuDB has created a lot of work for me during the last 2 years or so. Could TokuDB devs consider setting up their own Launchpad PPA (it's free) and run development builds there every now and then, so that these issues would have a chance to be caught before release?
Thanks
Hi, Otto! On Feb 04, Otto Kekäläinen wrote:
2015-02-04 1:21 GMT+02:00 Sergei Golubchik <serg@mariadb.org>:
I can give you a patch that you apply and run another build in the Launchpad build system. Would that be ok?
Yes, please :)
Try this one: === modified file 'mysql-test/mysql-test-run.pl' --- mysql-test/mysql-test-run.pl +++ mysql-test/mysql-test-run.pl @@ -4886,6 +4886,10 @@ sub extract_warning_lines ($$) { } $Fwarn = undef; # Close file + if (@$matched_lines) { + $matched_lines = [ @lines ]; + } + if (scalar(@$matched_lines) > 0 && defined($last_warning_position->{$error_log}{test_names})) { return ($last_warning_position->{$error_log}{test_names}, $matched_lines); Regards, Sergei
Thanks! Added it and ran new builds: https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.0/+builds?build_text=&build_state=all log -> https://launchpadlibrarian.net/196929183/buildlog_ubuntu-vivid-amd64.mariadb... 2015-02-06 18:51 GMT+02:00 Sergei Golubchik <serg@mariadb.org>:
Hi, Otto!
On Feb 04, Otto Kekäläinen wrote:
2015-02-04 1:21 GMT+02:00 Sergei Golubchik <serg@mariadb.org>:
I can give you a patch that you apply and run another build in the Launchpad build system. Would that be ok?
Yes, please :)
Try this one:
=== modified file 'mysql-test/mysql-test-run.pl' --- mysql-test/mysql-test-run.pl +++ mysql-test/mysql-test-run.pl @@ -4886,6 +4886,10 @@ sub extract_warning_lines ($$) { } $Fwarn = undef; # Close file
+ if (@$matched_lines) { + $matched_lines = [ @lines ]; + } + if (scalar(@$matched_lines) > 0 && defined($last_warning_position->{$error_log}{test_names})) { return ($last_warning_position->{$error_log}{test_names}, $matched_lines);
Regards, Sergei
-- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen
Hello, tokudb uses getline to retrieve lines from various linux system files and tokudb frees the buffer that getline allocates. perhaps getline has a problem, or tokudb's portability function that computes the system's processor frequency is broken. i dont see a bug in tokudb related to this problem. On Sat, Feb 7, 2015 at 1:33 PM, Otto Kekäläinen <otto@seravo.fi> wrote:
Thanks!
Added it and ran new builds:
https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.0/+builds?build_text=&build_state=all
log -> https://launchpadlibrarian.net/196929183/buildlog_ubuntu-vivid-amd64.mariadb...
2015-02-06 18:51 GMT+02:00 Sergei Golubchik <serg@mariadb.org>:
Hi, Otto!
On Feb 04, Otto Kekäläinen wrote:
2015-02-04 1:21 GMT+02:00 Sergei Golubchik <serg@mariadb.org>:
I can give you a patch that you apply and run another build in the Launchpad build system. Would that be ok?
Yes, please :)
Try this one:
=== modified file 'mysql-test/mysql-test-run.pl' --- mysql-test/mysql-test-run.pl +++ mysql-test/mysql-test-run.pl @@ -4886,6 +4886,10 @@ sub extract_warning_lines ($$) { } $Fwarn = undef; # Close file
+ if (@$matched_lines) { + $matched_lines = [ @lines ]; + } + if (scalar(@$matched_lines) > 0 && defined($last_warning_position->{$error_log}{test_names})) { return ($last_warning_position->{$error_log}{test_names}, $matched_lines);
Regards, Sergei
-- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Hi, Rich! On Feb 08, Rich Prohaska wrote:
Hello, tokudb uses getline to retrieve lines from various linux system files and tokudb frees the buffer that getline allocates. perhaps getline has a problem, or tokudb's portability function that computes the system's processor frequency is broken. i dont see a bug in tokudb related to this problem.
Could that happen that getline() allocates using system malloc, and you free() with jemalloc? ... sql/signal_handler.cc:153(handle_fatal_signal)[0x737022] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc90)[0x7f243f447c90] plugins/ha_tokudb.so(+0x14597c)[0x7f2437d8d97c] plugins/ha_tokudb.so(+0x146bb8)[0x7f2437d8ebb8] plugins/ha_tokudb.so(free+0x2f5)[0x7f2437d797c5] plugins/ha_tokudb.so(_Z31toku_os_get_processor_frequencyPm+0x15e)[0x7f2437d5ff6e] ... Regards, Sergei
Rich, Sergei: just checking - are you preparing a debug patch or something else that would help us get these rpl-tokudb.tokudb_innodb_xa_crash fails fixed in near future?
Hi Otto Ok forgive my ignorance but is Tokudb being enabled, you compile successfully but does the config enable tokudb to allow it to be tested Cheers Peter On Tue, Feb 3, 2015 at 9:08 PM, Otto Kekäläinen <otto@seravo.fi> wrote:
Hello!
Are here any TokuDB devs who could help me debug why TokuDB does repeatedly fails to pass build suite tests when built on Launchpad.net?
I repeatedly get these test failures when running MariaDB builds on Launchpad: rpl-tokudb.tokudb_innodb_xa_crash tokudb_bugs.mdev4533 tokudb.simple_join_tokudb_innodb tokudb.truncate_txn_rollback_innodb tokudb_bugs.5733_innodb
The build run log can be viewed at
https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.0/+builds?build_text=&build_state=all and it links to the individual build logs. All amd64 builds fail while all i386 (where TokuDB is disabled) pass OK. Sounds very much like a TokuDB-related issue. Or could the problem be something else and the TokuDB tests are just the first victim?
Example of a recent log:
https://launchpadlibrarian.net/196425361/buildlog_ubuntu-vivid-amd64.mariadb...
One example of the test outputs:
*** rpl-tokudb.tokudb_innodb_xa_crash 'mix,xtradb' [ fail ] Test ended at 2015-02-02 22:29:06
CURRENT_TEST: rpl-tokudb.tokudb_innodb_xa_crash
Failed to start mysqld.1 mysqltest failed but provided no output
- saving '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/' to '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/' - found 'core' (1/5)
Trying 'dbx' to get a backtrace
Trying 'gdb' to get a backtrace Compressed file
/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/mysqld.1/data/core ***Warnings generated in error logs during shutdown after running tests: rpl-tokudb.tokudb_innodb_xa_crash
150202 22:29:05 [ERROR] mysqld got signal 11 ; Attempting backtrace. You can use the following information to find out 150202 22:29:05 [ERROR] mysqld got signal 11 ; Attempting backtrace. You can use the following information to find out ***
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
-- *Peter McLarty* *Data Analyst DBA* Compare The Market P: +61 7 3377 8952 M: +61 4 0209 4238 http://www.comparethemarket.com.au <http://t.senalcinco.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs5vfP7RW7dSsk865364gW2zhrDH56dLb5f7JHPR002?t=http%3A%2F%2Fwww.comparethemarket.com.au%2F&si=5940338295308288&pi=b2c7e93c-f4d5-4d7e-d9e0-0e90eae5c025> -- Compare The Market is a brand and trading name of Compare The Market (Pty) Ltd (CTM). This email is for the intended addressee and is confidential and subject to copyright. If you are not the intended addressee, confidentiality has not been waived and any use, interference with, or disclosure of this email is unauthorised. If you are not the intended addressee please immediately notify CTM and then delete the email. CTM does not warrant that this email is error or virus free.
Hello! I am not sure what you mean, but the builds at https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.0/+builds?build_text=&build_state=all are built from the sources at https://github.com/ottok/mariadb-10.0. Make options are visible in https://github.com/ottok/mariadb-10.0/blob/master/debian/rules AFAIK the tests check if TokuDB is available, and run only then. For example on i386 the TokuDB tests are skipped automatically. I don't know if this answered your question, but the source link is anyway above. 2015-02-04 0:49 GMT+02:00 Peter Mclarty <peter.mclarty@comparethemarket.com.au>:
Hi Otto
Ok forgive my ignorance but is Tokudb being enabled, you compile successfully but does the config enable tokudb to allow it to be tested
Cheers
Peter
On Tue, Feb 3, 2015 at 9:08 PM, Otto Kekäläinen <otto@seravo.fi> wrote:
Hello!
Are here any TokuDB devs who could help me debug why TokuDB does repeatedly fails to pass build suite tests when built on Launchpad.net?
I repeatedly get these test failures when running MariaDB builds on Launchpad: rpl-tokudb.tokudb_innodb_xa_crash tokudb_bugs.mdev4533 tokudb.simple_join_tokudb_innodb tokudb.truncate_txn_rollback_innodb tokudb_bugs.5733_innodb
The build run log can be viewed at https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.0/+builds?build_text=&build_state=all and it links to the individual build logs. All amd64 builds fail while all i386 (where TokuDB is disabled) pass OK. Sounds very much like a TokuDB-related issue. Or could the problem be something else and the TokuDB tests are just the first victim?
Example of a recent log: https://launchpadlibrarian.net/196425361/buildlog_ubuntu-vivid-amd64.mariadb...
One example of the test outputs:
*** rpl-tokudb.tokudb_innodb_xa_crash 'mix,xtradb' [ fail ] Test ended at 2015-02-02 22:29:06
CURRENT_TEST: rpl-tokudb.tokudb_innodb_xa_crash
Failed to start mysqld.1 mysqltest failed but provided no output
- saving '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/' to '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/' - found 'core' (1/5)
Trying 'dbx' to get a backtrace
Trying 'gdb' to get a backtrace Compressed file /build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/mysqld.1/data/core ***Warnings generated in error logs during shutdown after running tests: rpl-tokudb.tokudb_innodb_xa_crash
150202 22:29:05 [ERROR] mysqld got signal 11 ; Attempting backtrace. You can use the following information to find out 150202 22:29:05 [ERROR] mysqld got signal 11 ; Attempting backtrace. You can use the following information to find out ***
Hi, Peter! On Feb 04, Peter Mclarty wrote:
Ok forgive my ignorance but is Tokudb being enabled, you compile successfully but does the config enable tokudb to allow it to be tested
Yes, it must've been enabled. If TokuDB would've been disabled, the test would've been skipped automatically, it could've never failed. I've checked the log - it looks like all tokudb tests failed, and all failed identically - the server crashed on startup.
https://launchpadlibrarian.net/196425361/buildlog_ubuntu-vivid-amd64.mariadb...
*** rpl-tokudb.tokudb_innodb_xa_crash 'mix,xtradb' [ fail ] Test ended at 2015-02-02 22:29:06
Regards, Sergei
Hello! Just a reminder to any TokuDB people on the list here: the thread that I started in February about this is still without an conclusion. In Ubuntu 15.04 at least for now TokuDB in MariaDB will be disabled completely due to this bug. Here is a link to the thread in Feb: https://lists.launchpad.net/maria-developers/msg08111.html 2015-02-03 13:08 GMT+02:00 Otto Kekäläinen <otto@seravo.fi>:
Hello!
Are here any TokuDB devs who could help me debug why TokuDB does repeatedly fails to pass build suite tests when built on Launchpad.net?
I repeatedly get these test failures when running MariaDB builds on Launchpad: rpl-tokudb.tokudb_innodb_xa_crash tokudb_bugs.mdev4533 tokudb.simple_join_tokudb_innodb tokudb.truncate_txn_rollback_innodb tokudb_bugs.5733_innodb
The build run log can be viewed at https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.0/+builds?build_text=&build_state=all and it links to the individual build logs. All amd64 builds fail while all i386 (where TokuDB is disabled) pass OK. Sounds very much like a TokuDB-related issue. Or could the problem be something else and the TokuDB tests are just the first victim?
Example of a recent log: https://launchpadlibrarian.net/196425361/buildlog_ubuntu-vivid-amd64.mariadb...
One example of the test outputs:
*** rpl-tokudb.tokudb_innodb_xa_crash 'mix,xtradb' [ fail ] Test ended at 2015-02-02 22:29:06
CURRENT_TEST: rpl-tokudb.tokudb_innodb_xa_crash
Failed to start mysqld.1 mysqltest failed but provided no output
- saving '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/' to '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/' - found 'core' (1/5)
Trying 'dbx' to get a backtrace
Trying 'gdb' to get a backtrace Compressed file /build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/mysqld.1/data/core ***Warnings generated in error logs during shutdown after running tests: rpl-tokudb.tokudb_innodb_xa_crash
150202 22:29:05 [ERROR] mysqld got signal 11 ; Attempting backtrace. You can use the following information to find out 150202 22:29:05 [ERROR] mysqld got signal 11 ; Attempting backtrace. You can use the following information to find out ***
-- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen
participants (4)
-
Otto Kekäläinen -
Peter Mclarty -
Rich Prohaska -
Sergei Golubchik