- developers - lists.mariadb.org

[Maria-developers] WL#3859: Plug-in Service API discussion
by Kristian Nielsen 12 Mar '09

12 Mar '09

While we were discussing a generic I/O layer in MariaDB, Sergei brought up the issue of WL#3859. This worklog carries a principal issue that I wanted to mention and discuss. Sergei, if I understood you correctly, you said that Sun MySQL would never take a new internal API that does not follow the WL#3859 framework. Correct? My understanding (which may or may not be correct) is that at least part of MySQL/Sun wants to develop the storage engine plugin layer (and all other plugin layers) to be a complete ABI, that is complete binary compatibility. So that the user can obtain a pre-compiled plugin independently and load it into the server without risk of crashing if plugin is compiled against a different source version of the server (but may get an error message if versions are incompatible). This basically means that all data and code needs to be exported via the WL#3859 framework, not using the system linker as is currently done. Advantages include - Avoiding crash on incompatible version is good. - Supports selling proprietary (non Open Source) plugins. - Works on Windows where the system linker cannot export symbols from executables (though there is a standard workaround of just compiling the server into a library that is called from a stub main executable). Disadvantages include - Supports selling proprietary (non Open Source) plugins (yes, can be an advantage or disadvantage depending on point-of-view ;-). - Will tend to discourage making desirable changes, as the procedure becomes more complicated (more work to track versioned API, less flexibitily in what C++ language constructs can be used); and there will be the desire to minimize binary incompatibility. It would be great if some of you still inside Sun could confirm or reject that this is the way things are. This raises at least two important questions for MariaDB: 1. Do we want to adopt this direction, moving all existing exposed server internals into this framework, and implementing any new ones within it as well? Or reserve the right to use old-style direct linking where it makes more sense? (We can of course in any case use the WL#3859 framework where it makes sense, the issue is the principle of using it for everything eventually). 2. It seems impossible that we can keep the Sun/MySQL API versions and the MariaDB API versions 100% synchronised. So how will we handle version compatibility (ie. Sun/MySQL version 0x0109 may be different from Maria version 0x0109 due to different paces of development)? Will all plugins need to track two different versions, one for Sun/MySQL and one for MariaDB? Or will we somehow coordinate between the two teams to avoid version conflicts? Any opinions? My personal opinion is that I am sceptical if this worklog is a good idea; most of the big projects of this kind that have been successful have used the old MySQL approach (Apache, Perl, the Linux kernel). Though I'm not 100% decided one way or the other yet. But more importantly I think we need to have a conscious decision on how to deal with this in MariaDB, as it is a fundamental design issue that will impact many other desicions along the way. - Kristian.

2 3

[Maria-developers] bzr commit into mariadb 5.1, with Maria 1.5:maria branch (knielsen:2679) Bug#43418
by knielsen＠knielsen-hq.org 12 Mar '09

12 Mar '09

#At lp:maria 2679 knielsen(a)knielsen-hq.org 2009-03-12 BUG#43418: MTR2: does not notice a memory leak occuring at shutdown of mysqld w/ --valgrind Partial fix. Do a graceful shutdown at end of testsuite instead of kill -9. Do a simplified parsing for log errors after final server shutdown. modified: mysql-test/mysql-test-run.pl === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2009-02-19 09:01:25 +0000 +++ b/mysql-test/mysql-test-run.pl 2009-03-12 11:39:00 +0000 @@ -357,7 +357,7 @@ sub main { mtr_print_thick_line(); mtr_print_header(); - my $completed= run_test_server($server, $tests, $opt_parallel); + my ($completed, $fail)= run_test_server($server, $tests, $opt_parallel); # Send Ctrl-C to any children still running kill("INT", keys(%children)); @@ -393,6 +393,10 @@ sub main { mtr_error("Not all tests completed"); } + if ($fail) { + mtr_error("Test suite failure."); + } + mtr_print_line(); if ( $opt_gcov ) { @@ -412,6 +416,7 @@ sub run_test_server ($$$) { my $num_saved_cores= 0; # Number of core files saved in vardir/log/ so far. my $num_saved_datadir= 0; # Number of datadirs saved in vardir/log/ so far. my $num_failed_test= 0; # Number of tests failed so far + my $test_failure= 0; # Scheduler variables my $max_ndb= $childs / 2; @@ -445,7 +450,7 @@ sub run_test_server ($$$) { $s->remove($sock); if (--$childs == 0){ $suite_timeout_proc->kill(); - return $completed; + return ($completed, $test_failure); } next; } @@ -509,18 +514,19 @@ sub run_test_server ($$$) { } $num_saved_datadir++; + $test_failure= 1; if ( !$opt_force ) { # Test has failed, force is off $suite_timeout_proc->kill(); push(@$completed, $result); - return $completed; + return ($completed, 1); } elsif ($opt_max_test_fail > 0 and $num_failed_test >= $opt_max_test_fail) { $suite_timeout_proc->kill(); mtr_report("Too many tests($num_failed_test) failed!", "Terminating..."); - return undef; + return (undef, 1); } $num_failed_test++; } @@ -571,6 +577,10 @@ sub run_test_server ($$$) { elsif ($line eq 'START'){ ; # Send first test } + elsif ($line eq 'WARNINGS'){ + mtr_report("***Warnings found in log after final shutdown."); + $test_failure= 1; + } else { mtr_error("Unknown response: '$line' from client"); } @@ -649,7 +659,7 @@ sub run_test_server ($$$) { if ( ! $suite_timeout_proc->wait_one(0) ) { mtr_report("Test suite timeout! Terminating..."); - return undef; + return (undef, 1); } } } @@ -706,10 +716,11 @@ sub run_worker ($) { # Ask server for first test print $server "START\n"; + my $test= undef; while(my $line= <$server>){ chomp($line); if ($line eq 'TESTCASE'){ - my $test= My::Test::read_test($server); + $test= My::Test::read_test($server); #$test->print_test(); # Clear comment and logfile, to avoid @@ -725,6 +736,22 @@ sub run_worker ($) { } elsif ($line eq 'BYE'){ mtr_report("Server said BYE"); + # We need to gracefully shut down the servers to see any + # Valgrind memory leak errors etc. since last server restart. + if ($opt_warnings && defined($test)) { + # Set marker so we only consider log messages generated during shutdown + foreach my $mysqld ( mysqlds() ) { + if (defined $mysqld->value('log-error')) { + mark_log($mysqld->value('log-error'), "<final shutdown>"); + } + } + stop_all_servers(1); + if (check_warnings($test)) { + # Warnings appeared in log file(s) during final server shutdown. + print $server "WARNINGS\n"; + exit(1); + } + } exit(0); } else { @@ -3095,8 +3122,8 @@ sub run_on_all($$) sub mark_log { - my ($log, $tinfo)= @_; - my $log_msg= "CURRENT_TEST: $tinfo->{name}\n"; + my ($log, $name)= @_; + my $log_msg= "CURRENT_TEST: $name\n"; mtr_tofile($log, $log_msg); } @@ -3247,7 +3274,7 @@ sub run_testcase ($) { } # Write start of testcase to log - mark_log($path_current_testlog, $tinfo); + mark_log($path_current_testlog, $tinfo->{name}); if (start_servers($tinfo)) { @@ -3526,6 +3553,7 @@ sub extract_warning_lines ($) { qr/Assertion .* failed/, ); + my $match_count= 0; foreach my $line ( @lines ) { foreach my $pat ( @patterns ) @@ -3533,12 +3561,14 @@ sub extract_warning_lines ($) { if ( $line =~ /$pat/ ) { print $Fwarn $line; + ++$match_count; last; } } } $Fwarn = undef; # Close file + return $match_count; } @@ -3554,8 +3584,6 @@ sub start_check_warnings ($$) { my $name= "warnings-".$mysqld->name(); - extract_warning_lines($mysqld->value('log-error')); - my $args; mtr_init_args(\$args); @@ -3614,17 +3642,37 @@ sub check_warnings ($) { # Start the mysqltest processes in parallel to save time # also makes it possible to wait for any process to exit during the check my %started; + my $total_found= 0; foreach my $mysqld ( mysqlds() ) { - if ( defined $mysqld->{'proc'} ) + my $found= 0; + if (defined $mysqld->value('log-error')) { + $found= extract_warning_lines($mysqld->value('log-error')); + } + + if ( defined $mysqld->{'proc'}) { my $proc= start_check_warnings($tinfo, $mysqld); $started{$proc->pid()}= $proc; } + else + { + $total_found+= $found; + } } # Return immediately if no check proceess was started - return 0 unless ( keys %started ); + unless ( keys %started ) { + # When we check warnings after shutdown of mysqld, we cannot use the + # SQL implementation of warning checking, so we just use the Perl + # checking instead. + # (eventually, it would be better to do all the parsing here in Perl). + if ($total_found > 0) { + return 1; + } else { + return 0; + } + } my $timeout_proc= My::SafeProcess->timer(check_timeout()); @@ -4161,12 +4209,14 @@ sub mysqld_start ($$) { sub stop_all_servers () { + my ($graceful)= @_; mtr_verbose("Stopping all servers..."); # Kill all started servers - My::SafeProcess::shutdown(0, # shutdown timeout 0 => kill - started(all_servers())); + my $timeout = $graceful ? + $opt_shutdown_timeout : 0; # shutdown timeout 0 => kill + My::SafeProcess::shutdown($timeout, started(all_servers())); # Remove pidfiles foreach my $server ( all_servers() ) @@ -4426,7 +4476,7 @@ sub start_servers($) { # Already started # Write start of testcase to log file - mark_log($mysqld->value('log-error'), $tinfo); + mark_log($mysqld->value('log-error'), $tinfo->{name}); next; } @@ -4485,7 +4535,7 @@ sub start_servers($) { mkpath($tmpdir) unless -d $tmpdir; # Write start of testcase to log file - mark_log($mysqld->value('log-error'), $tinfo); + mark_log($mysqld->value('log-error'), $tinfo->{name}); # Run <tname>-master.sh if ($mysqld->option('#!run-master-sh') and

1 0

[Maria-developers] what triggers row-header compaction for Maria
by MARK CALLAGHAN 09 Mar '09

09 Mar '09

What triggers row-header compaction for Maria? There are fields in the row-header that can be removed under certain conditions (transaction has committed, no other transactions can see that row). Is this done when OPTIMIZE TABLE is run? Is this done in other cases? -- Mark Callaghan mdcallag(a)gmail.com

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2679)
by Michael Widenius 09 Mar '09

09 Mar '09

#At lp:maria based on revid:cjac@dev0-20090305222554-cxws3ke1ktvhhwku 2679 Michael Widenius 2009-03-09 Merge with lp:maria modified: .bzr-mysql/default.conf === modified file '.bzr-mysql/default.conf' --- a/.bzr-mysql/default.conf 2009-03-04 11:31:27 +0000 +++ b/.bzr-mysql/default.conf 2009-03-09 14:13:44 +0000 @@ -3,4 +3,4 @@ tree_location = lp:maria post_commit_to = maria-developers(a)lists.launchpad.net post_commit_url = lp:maria tree_name = maria -project_name = "mariadb 5.1, with Maria 1.5" +project_name = "MariaDB 5.1, with Maria 1.5"

1 0

[Maria-developers] safe patches to backport -- pool-of-threads
by MARK CALLAGHAN 09 Mar '09

09 Mar '09

One of the features listed here http://askmonty.org/wiki/index.php/MariaDB_versus_MySQL as a potential backport is support for pool-of-threads. I am not sure how safe this feature is today. http://mysqlha.blogspot.com/2009/01/no-new-global-mutexes-and-how-to-make.h… describes some of the problems that might cause a server to hang when pool-of-threads is used. http://bugs.mysql.com/bug.php?id=42288 describes the performance overhead of pool-of-threads. The overhead is significant for workloads with fast queries such as sysbench. I think that pool-of-threads needs a lot of work before it can be considered safe. We backported this to our 5.0 branch at work and have begun to search for fixes. -- Mark Callaghan mdcallag(a)gmail.com

3 2

[Maria-developers] the _BOOK_ in http://forge.mysql.com/worklog/task.php?id=3134
by MARK CALLAGHAN 09 Mar '09

09 Mar '09

Is the _BOOK_ the Gray and Reuter book? http://www.informatik.uni-trier.de/~ley/db/books/dbtext/Gray93.html -- Mark Callaghan mdcallag(a)gmail.com

2 1

[Maria-developers] GCOV / LCOV data for storage/maria
by Patrick Crews 05 Mar '09

05 Mar '09

Hello all, Here are some lcov files for code coverage of storage/maria. I did 3 runs: unit only (make test in storage/maria), mtr-only, and both (mtr then make test). Included in the tarball are 3 directories, labelled for each run. You can open the index.html file to take a look at code coverage. I have some reservations about these numbers as OSX 10.5 was a little iffy on gcov a while back. I'll try to put together some numbers on OpenSolaris or Ubuntu as time permits (or someone else can post their numbers for comparison if they are feeling particularly industrious) Thanks, Patrick

1 0

[Maria-developers] bzr commit into mariadb 6.0, with Maria 1.5:maria/6.0 branch (sanja:3039)
by sanja＠askmonty.org 05 Mar '09

05 Mar '09

#At file:///home/bell/mysql/bzr/maria-6.0/ 3039 sanja(a)askmonty.org 2009-03-05 We need only commit e-mail (not push one). modified: .bzr-mysql/default.conf per-file messages: .bzr-mysql/default.conf We need only commit e-mail (not push one). === modified file '.bzr-mysql/default.conf' --- a/.bzr-mysql/default.conf 2009-03-04 22:15:38 +0000 +++ b/.bzr-mysql/default.conf 2009-03-05 09:19:00 +0000 @@ -1,6 +1,5 @@ [MYSQL] tree_location = lp:maria/6.0 post_commit_to = "maria-developers(a)lists.launchpad.net" -post_push_to = "maria-developers(a)lists.launchpad.net" tree_name = maria/6.0 project_name = "mariadb 6.0, with Maria 1.5"

1 0

[Maria-developers] max #rows per page -- DIR_COUNT_SIZE
by MARK CALLAGHAN 05 Mar '09

05 Mar '09

DIR_COUNT_SIZE is 1. Doesn't that mean that the max #rows on a page is 256? Do you want that limit when Maria pages can be large? -- Mark Callaghan mdcallag(a)gmail.com

1 0

[Maria-developers] bzr push into mariadb 6.0, with Maria 1.5:maria/6.0 branch (sanja:3037 to 3038)
by sanja＠askmonty.org 04 Mar '09

04 Mar '09

3038 sanja(a)askmonty.org 2009-03-05 The tree config modified: .bzr-mysql/default.conf 3037 Michael Widenius 2009-02-15 (empty) merge with main branch === modified file '.bzr-mysql/default.conf' --- a/.bzr-mysql/default.conf 2009-02-12 17:56:03 +0000 +++ b/.bzr-mysql/default.conf 2009-03-04 22:15:38 +0000 @@ -1,5 +1,6 @@ [MYSQL] -tree_location = bzr+ssh://bk-internal.mysql.com/bzrroot/server/mysql-6.0-bugteam -post_commit_to = "commits(a)lists.mysql.com" -post_push_to = "commits(a)lists.mysql.com" -tree_name = "mysql-6.0-bugteam" +tree_location = lp:maria/6.0 +post_commit_to = "maria-developers(a)lists.launchpad.net" +post_push_to = "maria-developers(a)lists.launchpad.net" +tree_name = maria/6.0 +project_name = "mariadb 6.0, with Maria 1.5"

1 0