- developers - lists.mariadb.org

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2740) Bug#46010
by knielsen＠knielsen-hq.org 30 Sep '09

30 Sep '09

#At lp:maria 2740 knielsen(a)knielsen-hq.org 2009-09-30 Apply patch from Bug#46010. This is needed to fix failures seen in Buildbot (main.ctype_gbk_binlog). Problem is that some tests did not RESET MASTER, causing them to pick up random binlog files depending on prior tests. modified: mysql-test/r/ctype_gbk_binlog.result mysql-test/r/mysqlbinlog-cp932.result mysql-test/suite/binlog/r/binlog_incident.result mysql-test/suite/binlog/r/binlog_tmp_table.result mysql-test/suite/binlog/t/binlog_incident.test mysql-test/suite/binlog/t/binlog_tmp_table.test mysql-test/t/ctype_gbk_binlog.test mysql-test/t/mysqlbinlog-cp932.test === modified file 'mysql-test/r/ctype_gbk_binlog.result' --- a/mysql-test/r/ctype_gbk_binlog.result 2009-07-07 14:18:44 +0000 +++ b/mysql-test/r/ctype_gbk_binlog.result 2009-09-30 06:35:31 +0000 @@ -1,3 +1,4 @@ +RESET MASTER; SET NAMES gbk; CREATE TABLE t1 ( f1 BLOB === modified file 'mysql-test/r/mysqlbinlog-cp932.result' --- a/mysql-test/r/mysqlbinlog-cp932.result 2007-02-19 10:57:06 +0000 +++ b/mysql-test/r/mysqlbinlog-cp932.result 2009-09-30 06:35:31 +0000 @@ -1,4 +1,4 @@ -flush logs; +RESET MASTER; create table t3 (f text character set utf8); create table t4 (f text character set cp932); flush logs; === modified file 'mysql-test/suite/binlog/r/binlog_incident.result' --- a/mysql-test/suite/binlog/r/binlog_incident.result 2009-05-11 11:32:38 +0000 +++ b/mysql-test/suite/binlog/r/binlog_incident.result 2009-09-30 06:35:31 +0000 @@ -1,3 +1,4 @@ +RESET MASTER; CREATE TABLE t1 (a INT); INSERT INTO t1 VALUES (1),(2),(3); SELECT * FROM t1; === modified file 'mysql-test/suite/binlog/r/binlog_tmp_table.result' --- a/mysql-test/suite/binlog/r/binlog_tmp_table.result 2009-01-28 14:35:12 +0000 +++ b/mysql-test/suite/binlog/r/binlog_tmp_table.result 2009-09-30 06:35:31 +0000 @@ -1,3 +1,4 @@ +RESET MASTER; create table foo (a int); flush logs; create temporary table tmp1_foo like foo; === modified file 'mysql-test/suite/binlog/t/binlog_incident.test' --- a/mysql-test/suite/binlog/t/binlog_incident.test 2009-05-11 11:32:38 +0000 +++ b/mysql-test/suite/binlog/t/binlog_incident.test 2009-09-30 06:35:31 +0000 @@ -6,6 +6,7 @@ source include/have_log_bin.inc; source include/have_debug.inc; let $MYSQLD_DATADIR= `select @@datadir`; +RESET MASTER; CREATE TABLE t1 (a INT); @@ -24,4 +25,4 @@ exec $MYSQL_BINLOG --start-position=106 eval SELECT cont LIKE '%RELOAD DATABASE; # Shall generate syntax error%' AS `Contain RELOAD DATABASE` FROM (SELECT load_file('$MYSQLTEST_VARDIR/tmp/binlog_incident-bug44442.sql') AS cont) AS tbl; --enable_query_log -remove_file $MYSQLTEST_VARDIR/tmp/binlog_incident-bug44442.sql; \ No newline at end of file +remove_file $MYSQLTEST_VARDIR/tmp/binlog_incident-bug44442.sql; === modified file 'mysql-test/suite/binlog/t/binlog_tmp_table.test' --- a/mysql-test/suite/binlog/t/binlog_tmp_table.test 2009-01-31 21:47:50 +0000 +++ b/mysql-test/suite/binlog/t/binlog_tmp_table.test 2009-09-30 06:35:31 +0000 @@ -30,6 +30,7 @@ source include/have_binlog_format_mixed_ connect (master,127.0.0.1,root,,test,$MASTER_MYPORT,); connect (master1,127.0.0.1,root,,test,$MASTER_MYPORT,); +RESET MASTER; create table foo (a int); === modified file 'mysql-test/t/ctype_gbk_binlog.test' --- a/mysql-test/t/ctype_gbk_binlog.test 2009-07-07 14:18:44 +0000 +++ b/mysql-test/t/ctype_gbk_binlog.test 2009-09-30 06:35:31 +0000 @@ -1,6 +1,7 @@ -- source include/have_binlog_format_mixed_or_statement.inc -- source include/have_gbk.inc +RESET MASTER; SET NAMES gbk; --character_set gbk === modified file 'mysql-test/t/mysqlbinlog-cp932.test' --- a/mysql-test/t/mysqlbinlog-cp932.test 2007-12-12 17:19:24 +0000 +++ b/mysql-test/t/mysqlbinlog-cp932.test 2009-09-30 06:35:31 +0000 @@ -5,8 +5,9 @@ -- source include/have_cp932.inc -- source include/have_log_bin.inc +RESET MASTER; + # Bug#16217 (mysql client did not know how not switch its internal charset) -flush logs; create table t3 (f text character set utf8); create table t4 (f text character set cp932); --exec $MYSQL --default-character-set=utf8 test -e "insert into t3 values(_utf8'ソ')" @@ -14,7 +15,7 @@ create table t4 (f text character set cp flush logs; rename table t3 to t03, t4 to t04; let $MYSQLD_DATADIR= `select @@datadir`; ---exec $MYSQL_BINLOG --short-form $MYSQLD_DATADIR/master-bin.000002 | $MYSQL --default-character-set=utf8 +--exec $MYSQL_BINLOG --short-form $MYSQLD_DATADIR/master-bin.000001 | $MYSQL --default-character-set=utf8 # original and recovered data must be equal select HEX(f) from t03; select HEX(f) from t3;

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2739)
by knielsen＠knielsen-hq.org 29 Sep '09

29 Sep '09

#At lp:maria 2739 knielsen(a)knielsen-hq.org 2009-09-29 Fix race in test of times shown in SHOW FULL PROCESSLIST. (These are by their nature difficult to test reliably due to differences in timing). modified: mysql-test/r/information_schema.result mysql-test/t/information_schema.test === modified file 'mysql-test/r/information_schema.result' --- a/mysql-test/r/information_schema.result 2009-09-23 11:03:47 +0000 +++ b/mysql-test/r/information_schema.result 2009-09-29 20:19:43 +0000 @@ -1385,7 +1385,7 @@ who other connection here SELECT IF(`time` > 0, 'OK', `time`) AS time_low, IF(`time` < 1000, 'OK', `time`) AS time_high, -IF(time_ms > 900, 'OK', time_ms) AS time_ms_low, +IF(time_ms > 1500, 'OK', time_ms) AS time_ms_low, IF(time_ms < 1000000, 'OK', time_ms) AS time_ms_high FROM INFORMATION_SCHEMA.PROCESSLIST WHERE ID=@tid; === modified file 'mysql-test/t/information_schema.test' --- a/mysql-test/t/information_schema.test 2009-09-23 11:03:47 +0000 +++ b/mysql-test/t/information_schema.test 2009-09-29 20:19:43 +0000 @@ -1105,13 +1105,13 @@ connect (pslistcon,localhost,root,,test) let $ID= `select connection_id()`; SELECT 'other connection here' AS who; connection default; -sleep 1; +sleep 2; --disable_query_log eval SET @tid=$ID; --enable_query_log SELECT IF(`time` > 0, 'OK', `time`) AS time_low, IF(`time` < 1000, 'OK', `time`) AS time_high, - IF(time_ms > 900, 'OK', time_ms) AS time_ms_low, + IF(time_ms > 1500, 'OK', time_ms) AS time_ms_low, IF(time_ms < 1000000, 'OK', time_ms) AS time_ms_high FROM INFORMATION_SCHEMA.PROCESSLIST WHERE ID=@tid;

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2738)
by knielsen＠knielsen-hq.org 29 Sep '09

29 Sep '09

#At lp:maria 2738 knielsen(a)knielsen-hq.org 2009-09-29 Fix running test suite from installed directory rather than source directory. Fix trivial typo in BUILD/* scripts. modified: BUILD/SETUP.sh mysql-test/mysql-test-run.pl per-file messages: BUILD/SETUP.sh Fix typo. mysql-test/mysql-test-run.pl Fix mysql-test-run.pl to be able to find plugins used in testing also when running the test suite from an installed mysqld package. === modified file 'BUILD/SETUP.sh' --- a/BUILD/SETUP.sh 2009-09-10 08:11:23 +0000 +++ b/BUILD/SETUP.sh 2009-09-29 19:02:48 +0000 @@ -173,7 +173,7 @@ local_infile_configs="--enable-local-inf max_no_embedded_configs="$SSL_LIBRARY --with-plugins=max" max_no_ndb_configs="$SSL_LIBRARY --with-plugins=max-no-ndb --with-embedded-server --with-libevent" -max_configs="$SSL_LIBRARY --with-plugins=max --with-embedded-server -with-libevent" +max_configs="$SSL_LIBRARY --with-plugins=max --with-embedded-server --with-libevent" # Disable NDB in maria max builds max_configs=$max_no_ndb_configs === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2009-09-26 02:25:56 +0000 +++ b/mysql-test/mysql-test-run.pl 2009-09-29 19:02:48 +0000 @@ -1886,7 +1886,8 @@ sub environment_setup { # -------------------------------------------------------------------------- my $lib_udf_example= mtr_file_exists(vs_config_dirs('sql', 'udf_example.dll'), - "$basedir/sql/.libs/udf_example.so",); + "$basedir/sql/.libs/udf_example.so", + "$basedir/lib/mysql/plugin/udf_example.so",); if ( $lib_udf_example ) { @@ -1913,7 +1914,8 @@ sub environment_setup { } my $lib_example_plugin= mtr_file_exists(vs_config_dirs('storage/example',$plugin_filename), - "$basedir/storage/example/.libs/".$plugin_filename); + "$basedir/storage/example/.libs/".$plugin_filename, + "$basedir/lib/mysql/plugin/".$plugin_filename); $ENV{'EXAMPLE_PLUGIN'}= ($lib_example_plugin ? basename($lib_example_plugin) : ""); $ENV{'EXAMPLE_PLUGIN_OPT'}= "--plugin-dir=". @@ -1928,7 +1930,8 @@ sub environment_setup { # ---------------------------------------------------- my $lib_simple_parser= mtr_file_exists(vs_config_dirs('plugin/fulltext', 'mypluglib.dll'), - "$basedir/plugin/fulltext/.libs/mypluglib.so",); + "$basedir/plugin/fulltext/.libs/mypluglib.so", + "$basedir/lib/mysql/plugin/mypluglib.so",); $ENV{'SIMPLE_PARSER'}= ($lib_simple_parser ? basename($lib_simple_parser) : "");

1 0

[Maria-developers] Notes from todays Maria call
by Henrik Ingo 29 Sep '09

29 Sep '09

Hi Since I for once took good notes (less stressed today :-) I'm sharing them here. Sergey, maybe you can reuse some of this to update todo page. Maria - win and valgrind problems - race conditions only seen on slow machines - problem likely in the test, not the server - this is not blocking release - arjen working on builds, henrik spoke with him today. - arjen happy with cooperation with kristian and says mariadb is easier to build than mysql. - bo and kristian set up windows dev environment for bo today - will provide another buildbot slave for win - kristian testing make dist - which can be used to produce a tarball - bo and kristian working on windows binary - release tree is maria-5.1-merge - Windows VM needed by Kristian, and also as a build environment - Daniel now setting this up on is own dev server - Sanja's QC bug fix still waiting for review. This is also not blocking release. (Current behavior is as in MySQL.) - Hakan adds smoke test to MariaDB test plan - We have a couple of mirrors for downloads. Ourdelta will probably play a role here too. - Ourdelta provides also apt and yum repos! - Henrik will talk with Bryan about tracking clicks of downloads - Henrik has not had time to look at documentation stuff Daniel is working on, will talk about that separately later. Also talked about Virtual Columns, Create table options etc, not important for release. henrik -- email: henrik.ingo(a)avoinelama.fi tel: +358-40-5697354 www: www.avoinelama.fi/~hingo book: www.openlife.cc

1 0

[Maria-developers] Updated (by Knielsen): mysqlbinlog: remove undesired effects of format description binlog statement (50)
by worklog-noreply＠askmonty.org 29 Sep '09

29 Sep '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: mysqlbinlog: remove undesired effects of format description binlog statement CREATION DATE..: Mon, 17 Aug 2009, 14:12 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 50 (http://askmonty.org/worklog/?tid=50) VERSION........: Server-9.x STATUS.........: Code-Review PRIORITY.......: 60 WORKED HOURS...: 14 ESTIMATE.......: 6 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: -=-=(Knielsen - Tue, 29 Sep 2009, 10:12)=-=- Category updated. --- /tmp/wklog.50.old.3293 2009-09-29 10:12:26.000000000 +0300 +++ /tmp/wklog.50.new.3293 2009-09-29 10:12:26.000000000 +0300 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Knielsen - Tue, 29 Sep 2009, 10:12)=-=- Status updated. --- /tmp/wklog.50.old.3293 2009-09-29 10:12:26.000000000 +0300 +++ /tmp/wklog.50.new.3293 2009-09-29 10:12:26.000000000 +0300 @@ -1 +1 @@ -Un-Assigned +Code-Review -=-=(Knielsen - Mon, 28 Sep 2009, 16:00)=-=- Updated patch against MySQL 5.1.39 Fixes a couple of issues with the patch. Discussions with MySQL developer. Worked 4 hours and estimate 6 hours remain (original estimate unchanged). -=-=(Knielsen - Wed, 09 Sep 2009, 18:31)=-=- Discussed with MySQL developers on Bug#46640. Committed a suggested fix: https://lists.launchpad.net/maria-developers/msg00926.html Worked 8 hours and estimate 10 hours remain (original estimate increased by 18 hours). -=-=(Guest - Wed, 09 Sep 2009, 11:50)=-=- High Level Description modified. --- /tmp/wklog.50.old.11584 2009-09-09 11:50:12.000000000 +0300 +++ /tmp/wklog.50.new.11584 2009-09-09 11:50:12.000000000 +0300 @@ -5,7 +5,11 @@ This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. -2. When one applies "format description binlog statement" at the slave, it -will change the slave's server_id when applied. +2. When one executes a BINLOG statement (containing "format description binlog +statement" or other events), this will cause subsequent statements run in +the same session to be binlogged with the server id of the last event +executed, not with the real server id. This can cause problems like infinite +recursive application in a circular replication topology. This WL is to fix these issues. + -=-=(Knielsen - Tue, 08 Sep 2009, 15:07)=-=- Low Level Design modified. --- /tmp/wklog.50.old.19208 2009-09-08 15:07:09.000000000 +0300 +++ /tmp/wklog.50.new.19208 2009-09-08 15:07:09.000000000 +0300 @@ -1 +1,34 @@ +I think the fix for point 2 is to replace the call to +apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct +call to ev->apply_event(). However, I need to check two things to be sure this +is correct: + +1. The existing code does + + if (!ev->when) ev->when= my_time(0) + +Need to understand if this is needed/correct or not. + +2. The existing code does ev->update_pos(). I think this is redundant for +BINLOG statement (as it uses a fake rli structure), but I need to check to +make sure. + +Once this is done, point 1 may no longer be needed. The user can use +--base64-output=never when applying the output to a 5.0 server, and omit that +option when applying the output to a 5.1 server. There should be no need to +omit the format description event in the output when other BINLOG statements +for row events are present, as it no longer changes the server id. In fact the +format description event is required to be able to execute other BINLOG +statements, as its purpose is to define the binary format of the events +contained in these statements. + +Alternatively, we could implement that if the format description event of the +source binlog has server version < 5.1, and --base64-output=auto (default), +then the format description event is omitted (and should any BINLOG statement +be needed, unlikely as it is from a 5.0 server, we will need to throw an +error). + +The binlog format version for 5.0 and 5.1 is actually the same, hence the need +to look at server version to guess if the format description event can be +omitted. -=-=(Knielsen - Tue, 08 Sep 2009, 14:29)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.17531 2009-09-08 14:29:30.000000000 +0300 +++ /tmp/wklog.50.new.17531 2009-09-08 14:29:30.000000000 +0300 @@ -12,3 +12,37 @@ One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? +First, the problem is not the format description log event. In fact all log +events have their own value of server_id, and every event in the BINLOG +statement is executed with its own id as server_id. + +It seems it was introduced deliberately in the patch for Bug#32407. This patch +makes sql thread and BINLOG statement share code path for executing the binary +event, even though the bug is really about something different (outputting +format description event to allow proper execution of other BINLOG +statements). + +Nevertheless, I think using the server_id in the events of a BINLOG statement +is wrong: + +1. It is different behaviour from executing normal statement-based events not +written using BINLOG. + +2. It causes the possibility for infinite cycle of replication in a cyclic +replication setup, if the server ids in BINLOG are different from all other +server ids in the cycle. + +3. The functionality of applying events with original server id from the event +is still available by pointing the slave thread to the binlog. The +functionality of applying row-based binlog events with server id of the server +executing the BINLOG statements is not otherwise available. + +In fact most of the code from the slave thread that is now also run when +executing BINLOG statements is redundant, or does the wrong thing (like +updating binlog position). + +Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be +able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is +it to be able to simulate the effect of the slave sql thread? I think the +former is the most important one. + -=-=(Knielsen - Mon, 07 Sep 2009, 10:06)=-=- Research worklog, bugs, and existing server code. Worked 2 hours and estimate 0 hours remain (original estimate increased by 2 hours). -=-=(Psergey - Mon, 17 Aug 2009, 14:13)=-=- Dependency created: 39 now depends on 50 -=-=(Psergey - Mon, 17 Aug 2009, 14:13)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.11389 2009-08-17 14:13:05.000000000 +0300 +++ /tmp/wklog.50.new.11389 2009-08-17 14:13:05.000000000 +0300 @@ -1 +1,14 @@ +First item +---------- +AFAIU what needs to be done is: +1. record a source server version (it is in the first binlog event). +2. don't emit a BINLOG statement if the recorded version number is 5.0.x or + below. + +and we'll get a 5.0-applicable binlog. + +Second item +----------- +One question that one needs to sort out before disabling server_id change is +why it was put there in the first place? Should it be always removed? DESCRIPTION: According to complaints in BUG#46640: 1. mysqlbinlog will emit a 5.0-incompatible "format description binlog statement" even when reading a binary log that was produced by 5.0. This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. 2. When one executes a BINLOG statement (containing "format description binlog statement" or other events), this will cause subsequent statements run in the same session to be binlogged with the server id of the last event executed, not with the real server id. This can cause problems like infinite recursive application in a circular replication topology. This WL is to fix these issues. HIGH-LEVEL SPECIFICATION: First item ---------- AFAIU what needs to be done is: 1. record a source server version (it is in the first binlog event). 2. don't emit a BINLOG statement if the recorded version number is 5.0.x or below. and we'll get a 5.0-applicable binlog. Second item ----------- One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? First, the problem is not the format description log event. In fact all log events have their own value of server_id, and every event in the BINLOG statement is executed with its own id as server_id. It seems it was introduced deliberately in the patch for Bug#32407. This patch makes sql thread and BINLOG statement share code path for executing the binary event, even though the bug is really about something different (outputting format description event to allow proper execution of other BINLOG statements). Nevertheless, I think using the server_id in the events of a BINLOG statement is wrong: 1. It is different behaviour from executing normal statement-based events not written using BINLOG. 2. It causes the possibility for infinite cycle of replication in a cyclic replication setup, if the server ids in BINLOG are different from all other server ids in the cycle. 3. The functionality of applying events with original server id from the event is still available by pointing the slave thread to the binlog. The functionality of applying row-based binlog events with server id of the server executing the BINLOG statements is not otherwise available. In fact most of the code from the slave thread that is now also run when executing BINLOG statements is redundant, or does the wrong thing (like updating binlog position). Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is it to be able to simulate the effect of the slave sql thread? I think the former is the most important one. LOW-LEVEL DESIGN: I think the fix for point 2 is to replace the call to apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct call to ev->apply_event(). However, I need to check two things to be sure this is correct: 1. The existing code does if (!ev->when) ev->when= my_time(0) Need to understand if this is needed/correct or not. 2. The existing code does ev->update_pos(). I think this is redundant for BINLOG statement (as it uses a fake rli structure), but I need to check to make sure. Once this is done, point 1 may no longer be needed. The user can use --base64-output=never when applying the output to a 5.0 server, and omit that option when applying the output to a 5.1 server. There should be no need to omit the format description event in the output when other BINLOG statements for row events are present, as it no longer changes the server id. In fact the format description event is required to be able to execute other BINLOG statements, as its purpose is to define the binary format of the events contained in these statements. Alternatively, we could implement that if the format description event of the source binlog has server version < 5.1, and --base64-output=auto (default), then the format description event is omitted (and should any BINLOG statement be needed, unlikely as it is from a 5.0 server, we will need to throw an error). The binlog format version for 5.0 and 5.1 is actually the same, hence the need to look at server version to guess if the format description event can be omitted. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): mysqlbinlog: remove undesired effects of format description binlog statement (50)
by worklog-noreply＠askmonty.org 29 Sep '09

29 Sep '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: mysqlbinlog: remove undesired effects of format description binlog statement CREATION DATE..: Mon, 17 Aug 2009, 14:12 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 50 (http://askmonty.org/worklog/?tid=50) VERSION........: Server-9.x STATUS.........: Code-Review PRIORITY.......: 60 WORKED HOURS...: 14 ESTIMATE.......: 6 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: -=-=(Knielsen - Tue, 29 Sep 2009, 10:12)=-=- Category updated. --- /tmp/wklog.50.old.3293 2009-09-29 10:12:26.000000000 +0300 +++ /tmp/wklog.50.new.3293 2009-09-29 10:12:26.000000000 +0300 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Knielsen - Tue, 29 Sep 2009, 10:12)=-=- Status updated. --- /tmp/wklog.50.old.3293 2009-09-29 10:12:26.000000000 +0300 +++ /tmp/wklog.50.new.3293 2009-09-29 10:12:26.000000000 +0300 @@ -1 +1 @@ -Un-Assigned +Code-Review -=-=(Knielsen - Mon, 28 Sep 2009, 16:00)=-=- Updated patch against MySQL 5.1.39 Fixes a couple of issues with the patch. Discussions with MySQL developer. Worked 4 hours and estimate 6 hours remain (original estimate unchanged). -=-=(Knielsen - Wed, 09 Sep 2009, 18:31)=-=- Discussed with MySQL developers on Bug#46640. Committed a suggested fix: https://lists.launchpad.net/maria-developers/msg00926.html Worked 8 hours and estimate 10 hours remain (original estimate increased by 18 hours). -=-=(Guest - Wed, 09 Sep 2009, 11:50)=-=- High Level Description modified. --- /tmp/wklog.50.old.11584 2009-09-09 11:50:12.000000000 +0300 +++ /tmp/wklog.50.new.11584 2009-09-09 11:50:12.000000000 +0300 @@ -5,7 +5,11 @@ This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. -2. When one applies "format description binlog statement" at the slave, it -will change the slave's server_id when applied. +2. When one executes a BINLOG statement (containing "format description binlog +statement" or other events), this will cause subsequent statements run in +the same session to be binlogged with the server id of the last event +executed, not with the real server id. This can cause problems like infinite +recursive application in a circular replication topology. This WL is to fix these issues. + -=-=(Knielsen - Tue, 08 Sep 2009, 15:07)=-=- Low Level Design modified. --- /tmp/wklog.50.old.19208 2009-09-08 15:07:09.000000000 +0300 +++ /tmp/wklog.50.new.19208 2009-09-08 15:07:09.000000000 +0300 @@ -1 +1,34 @@ +I think the fix for point 2 is to replace the call to +apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct +call to ev->apply_event(). However, I need to check two things to be sure this +is correct: + +1. The existing code does + + if (!ev->when) ev->when= my_time(0) + +Need to understand if this is needed/correct or not. + +2. The existing code does ev->update_pos(). I think this is redundant for +BINLOG statement (as it uses a fake rli structure), but I need to check to +make sure. + +Once this is done, point 1 may no longer be needed. The user can use +--base64-output=never when applying the output to a 5.0 server, and omit that +option when applying the output to a 5.1 server. There should be no need to +omit the format description event in the output when other BINLOG statements +for row events are present, as it no longer changes the server id. In fact the +format description event is required to be able to execute other BINLOG +statements, as its purpose is to define the binary format of the events +contained in these statements. + +Alternatively, we could implement that if the format description event of the +source binlog has server version < 5.1, and --base64-output=auto (default), +then the format description event is omitted (and should any BINLOG statement +be needed, unlikely as it is from a 5.0 server, we will need to throw an +error). + +The binlog format version for 5.0 and 5.1 is actually the same, hence the need +to look at server version to guess if the format description event can be +omitted. -=-=(Knielsen - Tue, 08 Sep 2009, 14:29)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.17531 2009-09-08 14:29:30.000000000 +0300 +++ /tmp/wklog.50.new.17531 2009-09-08 14:29:30.000000000 +0300 @@ -12,3 +12,37 @@ One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? +First, the problem is not the format description log event. In fact all log +events have their own value of server_id, and every event in the BINLOG +statement is executed with its own id as server_id. + +It seems it was introduced deliberately in the patch for Bug#32407. This patch +makes sql thread and BINLOG statement share code path for executing the binary +event, even though the bug is really about something different (outputting +format description event to allow proper execution of other BINLOG +statements). + +Nevertheless, I think using the server_id in the events of a BINLOG statement +is wrong: + +1. It is different behaviour from executing normal statement-based events not +written using BINLOG. + +2. It causes the possibility for infinite cycle of replication in a cyclic +replication setup, if the server ids in BINLOG are different from all other +server ids in the cycle. + +3. The functionality of applying events with original server id from the event +is still available by pointing the slave thread to the binlog. The +functionality of applying row-based binlog events with server id of the server +executing the BINLOG statements is not otherwise available. + +In fact most of the code from the slave thread that is now also run when +executing BINLOG statements is redundant, or does the wrong thing (like +updating binlog position). + +Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be +able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is +it to be able to simulate the effect of the slave sql thread? I think the +former is the most important one. + -=-=(Knielsen - Mon, 07 Sep 2009, 10:06)=-=- Research worklog, bugs, and existing server code. Worked 2 hours and estimate 0 hours remain (original estimate increased by 2 hours). -=-=(Psergey - Mon, 17 Aug 2009, 14:13)=-=- Dependency created: 39 now depends on 50 -=-=(Psergey - Mon, 17 Aug 2009, 14:13)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.11389 2009-08-17 14:13:05.000000000 +0300 +++ /tmp/wklog.50.new.11389 2009-08-17 14:13:05.000000000 +0300 @@ -1 +1,14 @@ +First item +---------- +AFAIU what needs to be done is: +1. record a source server version (it is in the first binlog event). +2. don't emit a BINLOG statement if the recorded version number is 5.0.x or + below. + +and we'll get a 5.0-applicable binlog. + +Second item +----------- +One question that one needs to sort out before disabling server_id change is +why it was put there in the first place? Should it be always removed? DESCRIPTION: According to complaints in BUG#46640: 1. mysqlbinlog will emit a 5.0-incompatible "format description binlog statement" even when reading a binary log that was produced by 5.0. This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. 2. When one executes a BINLOG statement (containing "format description binlog statement" or other events), this will cause subsequent statements run in the same session to be binlogged with the server id of the last event executed, not with the real server id. This can cause problems like infinite recursive application in a circular replication topology. This WL is to fix these issues. HIGH-LEVEL SPECIFICATION: First item ---------- AFAIU what needs to be done is: 1. record a source server version (it is in the first binlog event). 2. don't emit a BINLOG statement if the recorded version number is 5.0.x or below. and we'll get a 5.0-applicable binlog. Second item ----------- One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? First, the problem is not the format description log event. In fact all log events have their own value of server_id, and every event in the BINLOG statement is executed with its own id as server_id. It seems it was introduced deliberately in the patch for Bug#32407. This patch makes sql thread and BINLOG statement share code path for executing the binary event, even though the bug is really about something different (outputting format description event to allow proper execution of other BINLOG statements). Nevertheless, I think using the server_id in the events of a BINLOG statement is wrong: 1. It is different behaviour from executing normal statement-based events not written using BINLOG. 2. It causes the possibility for infinite cycle of replication in a cyclic replication setup, if the server ids in BINLOG are different from all other server ids in the cycle. 3. The functionality of applying events with original server id from the event is still available by pointing the slave thread to the binlog. The functionality of applying row-based binlog events with server id of the server executing the BINLOG statements is not otherwise available. In fact most of the code from the slave thread that is now also run when executing BINLOG statements is redundant, or does the wrong thing (like updating binlog position). Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is it to be able to simulate the effect of the slave sql thread? I think the former is the most important one. LOW-LEVEL DESIGN: I think the fix for point 2 is to replace the call to apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct call to ev->apply_event(). However, I need to check two things to be sure this is correct: 1. The existing code does if (!ev->when) ev->when= my_time(0) Need to understand if this is needed/correct or not. 2. The existing code does ev->update_pos(). I think this is redundant for BINLOG statement (as it uses a fake rli structure), but I need to check to make sure. Once this is done, point 1 may no longer be needed. The user can use --base64-output=never when applying the output to a 5.0 server, and omit that option when applying the output to a 5.1 server. There should be no need to omit the format description event in the output when other BINLOG statements for row events are present, as it no longer changes the server id. In fact the format description event is required to be able to execute other BINLOG statements, as its purpose is to define the binary format of the events contained in these statements. Alternatively, we could implement that if the format description event of the source binlog has server version < 5.1, and --base64-output=auto (default), then the format description event is omitted (and should any BINLOG statement be needed, unlikely as it is from a 5.0 server, we will need to throw an error). The binlog format version for 5.0 and 5.1 is actually the same, hence the need to look at server version to guess if the format description event can be omitted. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): mysqlbinlog: remove undesired effects of format description binlog statement (50)
by worklog-noreply＠askmonty.org 29 Sep '09

29 Sep '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: mysqlbinlog: remove undesired effects of format description binlog statement CREATION DATE..: Mon, 17 Aug 2009, 14:12 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 50 (http://askmonty.org/worklog/?tid=50) VERSION........: Server-9.x STATUS.........: Code-Review PRIORITY.......: 60 WORKED HOURS...: 14 ESTIMATE.......: 6 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: -=-=(Knielsen - Tue, 29 Sep 2009, 10:12)=-=- Category updated. --- /tmp/wklog.50.old.3293 2009-09-29 10:12:26.000000000 +0300 +++ /tmp/wklog.50.new.3293 2009-09-29 10:12:26.000000000 +0300 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Knielsen - Tue, 29 Sep 2009, 10:12)=-=- Status updated. --- /tmp/wklog.50.old.3293 2009-09-29 10:12:26.000000000 +0300 +++ /tmp/wklog.50.new.3293 2009-09-29 10:12:26.000000000 +0300 @@ -1 +1 @@ -Un-Assigned +Code-Review -=-=(Knielsen - Mon, 28 Sep 2009, 16:00)=-=- Updated patch against MySQL 5.1.39 Fixes a couple of issues with the patch. Discussions with MySQL developer. Worked 4 hours and estimate 6 hours remain (original estimate unchanged). -=-=(Knielsen - Wed, 09 Sep 2009, 18:31)=-=- Discussed with MySQL developers on Bug#46640. Committed a suggested fix: https://lists.launchpad.net/maria-developers/msg00926.html Worked 8 hours and estimate 10 hours remain (original estimate increased by 18 hours). -=-=(Guest - Wed, 09 Sep 2009, 11:50)=-=- High Level Description modified. --- /tmp/wklog.50.old.11584 2009-09-09 11:50:12.000000000 +0300 +++ /tmp/wklog.50.new.11584 2009-09-09 11:50:12.000000000 +0300 @@ -5,7 +5,11 @@ This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. -2. When one applies "format description binlog statement" at the slave, it -will change the slave's server_id when applied. +2. When one executes a BINLOG statement (containing "format description binlog +statement" or other events), this will cause subsequent statements run in +the same session to be binlogged with the server id of the last event +executed, not with the real server id. This can cause problems like infinite +recursive application in a circular replication topology. This WL is to fix these issues. + -=-=(Knielsen - Tue, 08 Sep 2009, 15:07)=-=- Low Level Design modified. --- /tmp/wklog.50.old.19208 2009-09-08 15:07:09.000000000 +0300 +++ /tmp/wklog.50.new.19208 2009-09-08 15:07:09.000000000 +0300 @@ -1 +1,34 @@ +I think the fix for point 2 is to replace the call to +apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct +call to ev->apply_event(). However, I need to check two things to be sure this +is correct: + +1. The existing code does + + if (!ev->when) ev->when= my_time(0) + +Need to understand if this is needed/correct or not. + +2. The existing code does ev->update_pos(). I think this is redundant for +BINLOG statement (as it uses a fake rli structure), but I need to check to +make sure. + +Once this is done, point 1 may no longer be needed. The user can use +--base64-output=never when applying the output to a 5.0 server, and omit that +option when applying the output to a 5.1 server. There should be no need to +omit the format description event in the output when other BINLOG statements +for row events are present, as it no longer changes the server id. In fact the +format description event is required to be able to execute other BINLOG +statements, as its purpose is to define the binary format of the events +contained in these statements. + +Alternatively, we could implement that if the format description event of the +source binlog has server version < 5.1, and --base64-output=auto (default), +then the format description event is omitted (and should any BINLOG statement +be needed, unlikely as it is from a 5.0 server, we will need to throw an +error). + +The binlog format version for 5.0 and 5.1 is actually the same, hence the need +to look at server version to guess if the format description event can be +omitted. -=-=(Knielsen - Tue, 08 Sep 2009, 14:29)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.17531 2009-09-08 14:29:30.000000000 +0300 +++ /tmp/wklog.50.new.17531 2009-09-08 14:29:30.000000000 +0300 @@ -12,3 +12,37 @@ One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? +First, the problem is not the format description log event. In fact all log +events have their own value of server_id, and every event in the BINLOG +statement is executed with its own id as server_id. + +It seems it was introduced deliberately in the patch for Bug#32407. This patch +makes sql thread and BINLOG statement share code path for executing the binary +event, even though the bug is really about something different (outputting +format description event to allow proper execution of other BINLOG +statements). + +Nevertheless, I think using the server_id in the events of a BINLOG statement +is wrong: + +1. It is different behaviour from executing normal statement-based events not +written using BINLOG. + +2. It causes the possibility for infinite cycle of replication in a cyclic +replication setup, if the server ids in BINLOG are different from all other +server ids in the cycle. + +3. The functionality of applying events with original server id from the event +is still available by pointing the slave thread to the binlog. The +functionality of applying row-based binlog events with server id of the server +executing the BINLOG statements is not otherwise available. + +In fact most of the code from the slave thread that is now also run when +executing BINLOG statements is redundant, or does the wrong thing (like +updating binlog position). + +Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be +able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is +it to be able to simulate the effect of the slave sql thread? I think the +former is the most important one. + -=-=(Knielsen - Mon, 07 Sep 2009, 10:06)=-=- Research worklog, bugs, and existing server code. Worked 2 hours and estimate 0 hours remain (original estimate increased by 2 hours). -=-=(Psergey - Mon, 17 Aug 2009, 14:13)=-=- Dependency created: 39 now depends on 50 -=-=(Psergey - Mon, 17 Aug 2009, 14:13)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.11389 2009-08-17 14:13:05.000000000 +0300 +++ /tmp/wklog.50.new.11389 2009-08-17 14:13:05.000000000 +0300 @@ -1 +1,14 @@ +First item +---------- +AFAIU what needs to be done is: +1. record a source server version (it is in the first binlog event). +2. don't emit a BINLOG statement if the recorded version number is 5.0.x or + below. + +and we'll get a 5.0-applicable binlog. + +Second item +----------- +One question that one needs to sort out before disabling server_id change is +why it was put there in the first place? Should it be always removed? DESCRIPTION: According to complaints in BUG#46640: 1. mysqlbinlog will emit a 5.0-incompatible "format description binlog statement" even when reading a binary log that was produced by 5.0. This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. 2. When one executes a BINLOG statement (containing "format description binlog statement" or other events), this will cause subsequent statements run in the same session to be binlogged with the server id of the last event executed, not with the real server id. This can cause problems like infinite recursive application in a circular replication topology. This WL is to fix these issues. HIGH-LEVEL SPECIFICATION: First item ---------- AFAIU what needs to be done is: 1. record a source server version (it is in the first binlog event). 2. don't emit a BINLOG statement if the recorded version number is 5.0.x or below. and we'll get a 5.0-applicable binlog. Second item ----------- One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? First, the problem is not the format description log event. In fact all log events have their own value of server_id, and every event in the BINLOG statement is executed with its own id as server_id. It seems it was introduced deliberately in the patch for Bug#32407. This patch makes sql thread and BINLOG statement share code path for executing the binary event, even though the bug is really about something different (outputting format description event to allow proper execution of other BINLOG statements). Nevertheless, I think using the server_id in the events of a BINLOG statement is wrong: 1. It is different behaviour from executing normal statement-based events not written using BINLOG. 2. It causes the possibility for infinite cycle of replication in a cyclic replication setup, if the server ids in BINLOG are different from all other server ids in the cycle. 3. The functionality of applying events with original server id from the event is still available by pointing the slave thread to the binlog. The functionality of applying row-based binlog events with server id of the server executing the BINLOG statements is not otherwise available. In fact most of the code from the slave thread that is now also run when executing BINLOG statements is redundant, or does the wrong thing (like updating binlog position). Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is it to be able to simulate the effect of the slave sql thread? I think the former is the most important one. LOW-LEVEL DESIGN: I think the fix for point 2 is to replace the call to apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct call to ev->apply_event(). However, I need to check two things to be sure this is correct: 1. The existing code does if (!ev->when) ev->when= my_time(0) Need to understand if this is needed/correct or not. 2. The existing code does ev->update_pos(). I think this is redundant for BINLOG statement (as it uses a fake rli structure), but I need to check to make sure. Once this is done, point 1 may no longer be needed. The user can use --base64-output=never when applying the output to a 5.0 server, and omit that option when applying the output to a 5.1 server. There should be no need to omit the format description event in the output when other BINLOG statements for row events are present, as it no longer changes the server id. In fact the format description event is required to be able to execute other BINLOG statements, as its purpose is to define the binary format of the events contained in these statements. Alternatively, we could implement that if the format description event of the source binlog has server version < 5.1, and --base64-output=auto (default), then the format description event is omitted (and should any BINLOG statement be needed, unlikely as it is from a 5.0 server, we will need to throw an error). The binlog format version for 5.0 and 5.1 is actually the same, hence the need to look at server version to guess if the format description event can be omitted. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Progress (by Knielsen): mysqlbinlog: remove undesired effects of format description binlog statement (50)
by worklog-noreply＠askmonty.org 28 Sep '09

28 Sep '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: mysqlbinlog: remove undesired effects of format description binlog statement CREATION DATE..: Mon, 17 Aug 2009, 14:12 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 50 (http://askmonty.org/worklog/?tid=50) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 14 ESTIMATE.......: 6 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: -=-=(Knielsen - Mon, 28 Sep 2009, 16:00)=-=- Updated patch against MySQL 5.1.39 Fixes a couple of issues with the patch. Discussions with MySQL developer. Worked 4 hours and estimate 6 hours remain (original estimate unchanged). -=-=(Knielsen - Wed, 09 Sep 2009, 18:31)=-=- Discussed with MySQL developers on Bug#46640. Committed a suggested fix: https://lists.launchpad.net/maria-developers/msg00926.html Worked 8 hours and estimate 10 hours remain (original estimate increased by 18 hours). -=-=(Guest - Wed, 09 Sep 2009, 11:50)=-=- High Level Description modified. --- /tmp/wklog.50.old.11584 2009-09-09 11:50:12.000000000 +0300 +++ /tmp/wklog.50.new.11584 2009-09-09 11:50:12.000000000 +0300 @@ -5,7 +5,11 @@ This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. -2. When one applies "format description binlog statement" at the slave, it -will change the slave's server_id when applied. +2. When one executes a BINLOG statement (containing "format description binlog +statement" or other events), this will cause subsequent statements run in +the same session to be binlogged with the server id of the last event +executed, not with the real server id. This can cause problems like infinite +recursive application in a circular replication topology. This WL is to fix these issues. + -=-=(Knielsen - Tue, 08 Sep 2009, 15:07)=-=- Low Level Design modified. --- /tmp/wklog.50.old.19208 2009-09-08 15:07:09.000000000 +0300 +++ /tmp/wklog.50.new.19208 2009-09-08 15:07:09.000000000 +0300 @@ -1 +1,34 @@ +I think the fix for point 2 is to replace the call to +apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct +call to ev->apply_event(). However, I need to check two things to be sure this +is correct: + +1. The existing code does + + if (!ev->when) ev->when= my_time(0) + +Need to understand if this is needed/correct or not. + +2. The existing code does ev->update_pos(). I think this is redundant for +BINLOG statement (as it uses a fake rli structure), but I need to check to +make sure. + +Once this is done, point 1 may no longer be needed. The user can use +--base64-output=never when applying the output to a 5.0 server, and omit that +option when applying the output to a 5.1 server. There should be no need to +omit the format description event in the output when other BINLOG statements +for row events are present, as it no longer changes the server id. In fact the +format description event is required to be able to execute other BINLOG +statements, as its purpose is to define the binary format of the events +contained in these statements. + +Alternatively, we could implement that if the format description event of the +source binlog has server version < 5.1, and --base64-output=auto (default), +then the format description event is omitted (and should any BINLOG statement +be needed, unlikely as it is from a 5.0 server, we will need to throw an +error). + +The binlog format version for 5.0 and 5.1 is actually the same, hence the need +to look at server version to guess if the format description event can be +omitted. -=-=(Knielsen - Tue, 08 Sep 2009, 14:29)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.17531 2009-09-08 14:29:30.000000000 +0300 +++ /tmp/wklog.50.new.17531 2009-09-08 14:29:30.000000000 +0300 @@ -12,3 +12,37 @@ One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? +First, the problem is not the format description log event. In fact all log +events have their own value of server_id, and every event in the BINLOG +statement is executed with its own id as server_id. + +It seems it was introduced deliberately in the patch for Bug#32407. This patch +makes sql thread and BINLOG statement share code path for executing the binary +event, even though the bug is really about something different (outputting +format description event to allow proper execution of other BINLOG +statements). + +Nevertheless, I think using the server_id in the events of a BINLOG statement +is wrong: + +1. It is different behaviour from executing normal statement-based events not +written using BINLOG. + +2. It causes the possibility for infinite cycle of replication in a cyclic +replication setup, if the server ids in BINLOG are different from all other +server ids in the cycle. + +3. The functionality of applying events with original server id from the event +is still available by pointing the slave thread to the binlog. The +functionality of applying row-based binlog events with server id of the server +executing the BINLOG statements is not otherwise available. + +In fact most of the code from the slave thread that is now also run when +executing BINLOG statements is redundant, or does the wrong thing (like +updating binlog position). + +Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be +able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is +it to be able to simulate the effect of the slave sql thread? I think the +former is the most important one. + -=-=(Knielsen - Mon, 07 Sep 2009, 10:06)=-=- Research worklog, bugs, and existing server code. Worked 2 hours and estimate 0 hours remain (original estimate increased by 2 hours). -=-=(Psergey - Mon, 17 Aug 2009, 14:13)=-=- Dependency created: 39 now depends on 50 -=-=(Psergey - Mon, 17 Aug 2009, 14:13)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.11389 2009-08-17 14:13:05.000000000 +0300 +++ /tmp/wklog.50.new.11389 2009-08-17 14:13:05.000000000 +0300 @@ -1 +1,14 @@ +First item +---------- +AFAIU what needs to be done is: +1. record a source server version (it is in the first binlog event). +2. don't emit a BINLOG statement if the recorded version number is 5.0.x or + below. + +and we'll get a 5.0-applicable binlog. + +Second item +----------- +One question that one needs to sort out before disabling server_id change is +why it was put there in the first place? Should it be always removed? DESCRIPTION: According to complaints in BUG#46640: 1. mysqlbinlog will emit a 5.0-incompatible "format description binlog statement" even when reading a binary log that was produced by 5.0. This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. 2. When one executes a BINLOG statement (containing "format description binlog statement" or other events), this will cause subsequent statements run in the same session to be binlogged with the server id of the last event executed, not with the real server id. This can cause problems like infinite recursive application in a circular replication topology. This WL is to fix these issues. HIGH-LEVEL SPECIFICATION: First item ---------- AFAIU what needs to be done is: 1. record a source server version (it is in the first binlog event). 2. don't emit a BINLOG statement if the recorded version number is 5.0.x or below. and we'll get a 5.0-applicable binlog. Second item ----------- One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? First, the problem is not the format description log event. In fact all log events have their own value of server_id, and every event in the BINLOG statement is executed with its own id as server_id. It seems it was introduced deliberately in the patch for Bug#32407. This patch makes sql thread and BINLOG statement share code path for executing the binary event, even though the bug is really about something different (outputting format description event to allow proper execution of other BINLOG statements). Nevertheless, I think using the server_id in the events of a BINLOG statement is wrong: 1. It is different behaviour from executing normal statement-based events not written using BINLOG. 2. It causes the possibility for infinite cycle of replication in a cyclic replication setup, if the server ids in BINLOG are different from all other server ids in the cycle. 3. The functionality of applying events with original server id from the event is still available by pointing the slave thread to the binlog. The functionality of applying row-based binlog events with server id of the server executing the BINLOG statements is not otherwise available. In fact most of the code from the slave thread that is now also run when executing BINLOG statements is redundant, or does the wrong thing (like updating binlog position). Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is it to be able to simulate the effect of the slave sql thread? I think the former is the most important one. LOW-LEVEL DESIGN: I think the fix for point 2 is to replace the call to apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct call to ev->apply_event(). However, I need to check two things to be sure this is correct: 1. The existing code does if (!ev->when) ev->when= my_time(0) Need to understand if this is needed/correct or not. 2. The existing code does ev->update_pos(). I think this is redundant for BINLOG statement (as it uses a fake rli structure), but I need to check to make sure. Once this is done, point 1 may no longer be needed. The user can use --base64-output=never when applying the output to a 5.0 server, and omit that option when applying the output to a 5.1 server. There should be no need to omit the format description event in the output when other BINLOG statements for row events are present, as it no longer changes the server id. In fact the format description event is required to be able to execute other BINLOG statements, as its purpose is to define the binary format of the events contained in these statements. Alternatively, we could implement that if the format description event of the source binlog has server version < 5.1, and --base64-output=auto (default), then the format description event is omitted (and should any BINLOG statement be needed, unlikely as it is from a 5.0 server, we will need to throw an error). The binlog format version for 5.0 and 5.1 is actually the same, hence the need to look at server version to guess if the format description event can be omitted. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Progress (by Knielsen): mysqlbinlog: remove undesired effects of format description binlog statement (50)
by worklog-noreply＠askmonty.org 28 Sep '09

28 Sep '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: mysqlbinlog: remove undesired effects of format description binlog statement CREATION DATE..: Mon, 17 Aug 2009, 14:12 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 50 (http://askmonty.org/worklog/?tid=50) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 14 ESTIMATE.......: 6 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: -=-=(Knielsen - Mon, 28 Sep 2009, 16:00)=-=- Updated patch against MySQL 5.1.39 Fixes a couple of issues with the patch. Discussions with MySQL developer. Worked 4 hours and estimate 6 hours remain (original estimate unchanged). -=-=(Knielsen - Wed, 09 Sep 2009, 18:31)=-=- Discussed with MySQL developers on Bug#46640. Committed a suggested fix: https://lists.launchpad.net/maria-developers/msg00926.html Worked 8 hours and estimate 10 hours remain (original estimate increased by 18 hours). -=-=(Guest - Wed, 09 Sep 2009, 11:50)=-=- High Level Description modified. --- /tmp/wklog.50.old.11584 2009-09-09 11:50:12.000000000 +0300 +++ /tmp/wklog.50.new.11584 2009-09-09 11:50:12.000000000 +0300 @@ -5,7 +5,11 @@ This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. -2. When one applies "format description binlog statement" at the slave, it -will change the slave's server_id when applied. +2. When one executes a BINLOG statement (containing "format description binlog +statement" or other events), this will cause subsequent statements run in +the same session to be binlogged with the server id of the last event +executed, not with the real server id. This can cause problems like infinite +recursive application in a circular replication topology. This WL is to fix these issues. + -=-=(Knielsen - Tue, 08 Sep 2009, 15:07)=-=- Low Level Design modified. --- /tmp/wklog.50.old.19208 2009-09-08 15:07:09.000000000 +0300 +++ /tmp/wklog.50.new.19208 2009-09-08 15:07:09.000000000 +0300 @@ -1 +1,34 @@ +I think the fix for point 2 is to replace the call to +apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct +call to ev->apply_event(). However, I need to check two things to be sure this +is correct: + +1. The existing code does + + if (!ev->when) ev->when= my_time(0) + +Need to understand if this is needed/correct or not. + +2. The existing code does ev->update_pos(). I think this is redundant for +BINLOG statement (as it uses a fake rli structure), but I need to check to +make sure. + +Once this is done, point 1 may no longer be needed. The user can use +--base64-output=never when applying the output to a 5.0 server, and omit that +option when applying the output to a 5.1 server. There should be no need to +omit the format description event in the output when other BINLOG statements +for row events are present, as it no longer changes the server id. In fact the +format description event is required to be able to execute other BINLOG +statements, as its purpose is to define the binary format of the events +contained in these statements. + +Alternatively, we could implement that if the format description event of the +source binlog has server version < 5.1, and --base64-output=auto (default), +then the format description event is omitted (and should any BINLOG statement +be needed, unlikely as it is from a 5.0 server, we will need to throw an +error). + +The binlog format version for 5.0 and 5.1 is actually the same, hence the need +to look at server version to guess if the format description event can be +omitted. -=-=(Knielsen - Tue, 08 Sep 2009, 14:29)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.17531 2009-09-08 14:29:30.000000000 +0300 +++ /tmp/wklog.50.new.17531 2009-09-08 14:29:30.000000000 +0300 @@ -12,3 +12,37 @@ One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? +First, the problem is not the format description log event. In fact all log +events have their own value of server_id, and every event in the BINLOG +statement is executed with its own id as server_id. + +It seems it was introduced deliberately in the patch for Bug#32407. This patch +makes sql thread and BINLOG statement share code path for executing the binary +event, even though the bug is really about something different (outputting +format description event to allow proper execution of other BINLOG +statements). + +Nevertheless, I think using the server_id in the events of a BINLOG statement +is wrong: + +1. It is different behaviour from executing normal statement-based events not +written using BINLOG. + +2. It causes the possibility for infinite cycle of replication in a cyclic +replication setup, if the server ids in BINLOG are different from all other +server ids in the cycle. + +3. The functionality of applying events with original server id from the event +is still available by pointing the slave thread to the binlog. The +functionality of applying row-based binlog events with server id of the server +executing the BINLOG statements is not otherwise available. + +In fact most of the code from the slave thread that is now also run when +executing BINLOG statements is redundant, or does the wrong thing (like +updating binlog position). + +Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be +able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is +it to be able to simulate the effect of the slave sql thread? I think the +former is the most important one. + -=-=(Knielsen - Mon, 07 Sep 2009, 10:06)=-=- Research worklog, bugs, and existing server code. Worked 2 hours and estimate 0 hours remain (original estimate increased by 2 hours). -=-=(Psergey - Mon, 17 Aug 2009, 14:13)=-=- Dependency created: 39 now depends on 50 -=-=(Psergey - Mon, 17 Aug 2009, 14:13)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.11389 2009-08-17 14:13:05.000000000 +0300 +++ /tmp/wklog.50.new.11389 2009-08-17 14:13:05.000000000 +0300 @@ -1 +1,14 @@ +First item +---------- +AFAIU what needs to be done is: +1. record a source server version (it is in the first binlog event). +2. don't emit a BINLOG statement if the recorded version number is 5.0.x or + below. + +and we'll get a 5.0-applicable binlog. + +Second item +----------- +One question that one needs to sort out before disabling server_id change is +why it was put there in the first place? Should it be always removed? DESCRIPTION: According to complaints in BUG#46640: 1. mysqlbinlog will emit a 5.0-incompatible "format description binlog statement" even when reading a binary log that was produced by 5.0. This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. 2. When one executes a BINLOG statement (containing "format description binlog statement" or other events), this will cause subsequent statements run in the same session to be binlogged with the server id of the last event executed, not with the real server id. This can cause problems like infinite recursive application in a circular replication topology. This WL is to fix these issues. HIGH-LEVEL SPECIFICATION: First item ---------- AFAIU what needs to be done is: 1. record a source server version (it is in the first binlog event). 2. don't emit a BINLOG statement if the recorded version number is 5.0.x or below. and we'll get a 5.0-applicable binlog. Second item ----------- One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? First, the problem is not the format description log event. In fact all log events have their own value of server_id, and every event in the BINLOG statement is executed with its own id as server_id. It seems it was introduced deliberately in the patch for Bug#32407. This patch makes sql thread and BINLOG statement share code path for executing the binary event, even though the bug is really about something different (outputting format description event to allow proper execution of other BINLOG statements). Nevertheless, I think using the server_id in the events of a BINLOG statement is wrong: 1. It is different behaviour from executing normal statement-based events not written using BINLOG. 2. It causes the possibility for infinite cycle of replication in a cyclic replication setup, if the server ids in BINLOG are different from all other server ids in the cycle. 3. The functionality of applying events with original server id from the event is still available by pointing the slave thread to the binlog. The functionality of applying row-based binlog events with server id of the server executing the BINLOG statements is not otherwise available. In fact most of the code from the slave thread that is now also run when executing BINLOG statements is redundant, or does the wrong thing (like updating binlog position). Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is it to be able to simulate the effect of the slave sql thread? I think the former is the most important one. LOW-LEVEL DESIGN: I think the fix for point 2 is to replace the call to apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct call to ev->apply_event(). However, I need to check two things to be sure this is correct: 1. The existing code does if (!ev->when) ev->when= my_time(0) Need to understand if this is needed/correct or not. 2. The existing code does ev->update_pos(). I think this is redundant for BINLOG statement (as it uses a fake rli structure), but I need to check to make sure. Once this is done, point 1 may no longer be needed. The user can use --base64-output=never when applying the output to a 5.0 server, and omit that option when applying the output to a 5.1 server. There should be no need to omit the format description event in the output when other BINLOG statements for row events are present, as it no longer changes the server id. In fact the format description event is required to be able to execute other BINLOG statements, as its purpose is to define the binary format of the events contained in these statements. Alternatively, we could implement that if the format description event of the source binlog has server version < 5.1, and --base64-output=auto (default), then the format description event is omitted (and should any BINLOG statement be needed, unlikely as it is from a 5.0 server, we will need to throw an error). The binlog format version for 5.0 and 5.1 is actually the same, hence the need to look at server version to guess if the format description event can be omitted. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Information for release notes
by Kristian Nielsen 27 Sep '09

27 Sep '09

Hi Daniel, Here is a rough changelog written from reading through bzr history. I took a dump of every commit in MariaDB that is not in MySQL 5.1.38, and wrote this summary of the changes. I don't have sufficient knowledge to parse through all of the Maria changes from before MariaDB started. Instead I wrote a generic entry for the Maria storage engine, and also added entries for what is already written (by Monty I think) on http://askmonty.org/wiki/index.php/MariaDB_versus_MySQL So this is probably the best shot we have at a complete list of relevant differences between MySQL and MariaDB. Do let me know if anyone finds anything missing or any errors. Daniel, I hope you can use this as the basis for the release notes. Maybe this list will also be useful for you to fill out the Manual on the Wiki. I am thinking you could check that all new features are documented, and add pages for those that are not. If not trivial, you could just make a stub page and ask on maria-developers@ for someone to fill them out? - Kristian. ----------------------------------------------------------------------- Generally: MariaDB-5.1.38-beta1 is based on MySQL-5.1.38, but has these additional features and bug fixes. Feature: mysql-test-run now has options --stop-file and --stop-keep-alive (also accessible with environment variables MTR_STOP_FILE and MTR_STOP_KEEP_ALIVE). With these, it is possible to pause a running test temporarily and let it continue later. Bug fix: mysqlslap: setting --engine does not get replicated http://bugs.mysql.com/bug.php?id=46967 mysqlslap: specifying --engine and --create does not work with --engine=<storage_engine>:<option> https://bugs.launchpad.net/maria/+bug/429773 Feature: Improvements to Gcov support in testing. Feature: New column TIME_MS in SHOW FULL PROCESSLIST and INFORMATION_SCHEMA.PROCESSLIST, similar to the old TIME column, but with microsecond resolution. Also fixes old TIME column to not be influenced by @TIMESTAMP. Patch by Percona. Bug fix: rpl_do_grant fails on PB-2 with a failing connect http://bugs.mysql.com/bug.php?id=47016 Bug fix: Windows: mysql-test-run --log-error fixed to not add --console. Bug fix: The myisam_crash_before_flush_keys test fails on Windows http://bugs.mysql.com/bug.php?id=47455 Bug fix: rpl.rpl_get_master_version_and_clock fails on hpux11.31 http://bugs.mysql.com/bug.php?id=46931 Bug fix: safe_process: FATAL ERROR, Unknown option: --nocore http://bugs.mysql.com/bug.php?id=46212 Feature: Optimizer improvement: Table elimination. http://askmonty.org/worklog/Server-Sprint/?tid=17 Bug fix: XtraDB crashes on startup on windows https://bugs.launchpad.net/maria/+bug/417751 Bug fix: Eliminate compiler warnings. Bug fix: Fix parsing of enum-valued options for plugins. https://bugs.launchpad.net/maria/+bug/423035 Feature: Enhancements to slow log. Includes details of execution plan and microsecond-precision resolution. Based on microslow patch by Percona. Feature: PBXT storage engine. PBXT is developed by PrimeBase Technologies, http://www.primebase.org/ Bug fix: Solaris build fixes. Feature: XtraDB storage engine as a replacement for the MySQL built-in InnoDB. XtraDB is based on the Oracle/Innobase InnoDB plugin version 1.0.3, with enhancements. XtraDB is developed by Percona. Feature: Performance improvements for common cases of character set conversion. Feature: Pool-of-threads, allowing to map a high number of simultaneous connections onto a lower number of operating system treads, to reduce overhead with using large number of threads. Feature: New handler call prepare_index_scan() and other small improvements to the internal storage engine handler API. Bug fix: query_cache_debug.test fails http://bugs.mysql.com/bug.php?id=45632 Bug fix: enum-style command-line options are not honoured (maria.maria-recover fails) http://bugs.mysql.com/bug.php?id=41010 Bug fix: mysql-test-run sometimes terminated mysqld early, causing loss of memory leak error reports from Valgrind and GCov test coverage output. http://bugs.mysql.com/bug.php?id=43418 Bug fix: Several Valgrind reported bugs in the source code fixed. Bug fix: Save SAFE_MUTEX configure #define in config.h, to facilitate correct build options for plugins. Feature: ./configure now outputs a summary section at the end of the output. (By C.J. Adams-Collier). Bug fix: test maria.maria fails if server built without '--with-partition' https://bugs.launchpad.net/maria/+bug/330611 Feature: NDB storage engine disabled in MariaDB builds. (NDB is not supported in MariaDB). Feature: Added --abort-source-on-error to the mysql client. Feature: Faster CHECKSUM TABLE. Feature: Debugging code to check for taking mutexes in the wrong order, which enables to catch potential deadlocks in the server code. Feature: Maria storage engine. The Maria storage engine is an enhanment over the MyISAM storage engine which provides journaling and crash recovery.

4 3