- developers - lists.mariadb.org

[Maria-developers] Rev 2740: MWL#67: MRR backport in file:///home/psergey/dev/maria-5.2-dsmrr/
by Sergey Petrunya 22 Dec '09

22 Dec '09

At file:///home/psergey/dev/maria-5.2-dsmrr/ ------------------------------------------------------------ revno: 2740 revision-id: psergey(a)askmonty.org-20091222123321-i4t0owvlu0d6hb08 parent: psergey(a)askmonty.org-20091219195454-yzk3mszr3jbn3fib committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.2-dsmrr timestamp: Tue 2009-12-22 15:33:21 +0300 message: MWL#67: MRR backport - Make index condition pushdown be controlled by an @@optimizer_switch flag, not by @@engine_condition_pushdown - Make MRR buffer size be controlled by @@mrr_buffer_size, not by @@read_rnd_buffer_size - Move parts of code to separate files - Code cleanup - Add --sorted_result to some SELECTs in tests. Diff too large for email (4499 lines, the limit is 1000).

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2779)
by knielsen＠knielsen-hq.org 22 Dec '09

22 Dec '09

#At lp:maria 2779 knielsen(a)knielsen-hq.org 2009-12-22 Slightly better way to handle max_connections for embedded server. modified: storage/pbxt/src/ha_pbxt.cc === modified file 'storage/pbxt/src/ha_pbxt.cc' --- a/storage/pbxt/src/ha_pbxt.cc 2009-12-22 10:33:20 +0000 +++ b/storage/pbxt/src/ha_pbxt.cc 2009-12-22 11:52:44 +0000 @@ -1175,12 +1175,13 @@ static int pbxt_init(void *p) * +1 Temporary thread (e.g. TempForClose, TempForEnd) */ #ifndef DRIZZLED -#ifdef EMBEDDED_LIBRARY - pbxt_max_threads = 100; -#else - if (pbxt_max_threads == 0) - pbxt_max_threads = max_connections + 7; -#endif + if (pbxt_max_threads == 0) { + // Embedded server sets max_connections=1 + if (max_connections > 1) + pbxt_max_threads = max_connections + 7; + else + pbxt_max_threads = 100; + } #endif self = xt_init_threading(pbxt_max_threads); /* Create the main self: */ if (!self)

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2778)
by knielsen＠knielsen-hq.org 22 Dec '09

22 Dec '09

#At lp:maria 2778 knielsen(a)knielsen-hq.org 2009-12-22 [merge] Merge fix for PBXT running inside embedded server (MBug#439889). Also some small fixes to make the PBXT testsuite work in --embedded. renamed: mysql-test/suite/pbxt/t/load_unique_error1.inc => mysql-test/std_data/pbxt_load_unique_error1.inc modified: config/ac-macros/plugins.m4 mysql-test/suite/pbxt/r/pbxt_bugs.result mysql-test/suite/pbxt/t/pbxt_bugs.test mysql-test/suite/pbxt/t/pbxt_locking.test mysql-test/suite/pbxt/t/pbxt_transactions.test mysql-test/suite/pbxt/t/ps_1general.test sql/sql_plugin.cc storage/pbxt/plug.in storage/pbxt/src/Makefile.am storage/pbxt/src/ha_pbxt.cc storage/pbxt/src/table_xt.cc storage/pbxt/src/thread_xt.cc storage/pbxt/src/trace_xt.cc per-file messages: config/ac-macros/plugins.m4 MYSQL_PLUGIN_DEPENDS_ON_MYSQL_INTERNALS macro extended to support multiple files. mysql-test/std_data/pbxt_load_unique_error1.inc Move file to be accessible also for testing embedded server. mysql-test/suite/pbxt/r/pbxt_bugs.result Fix LOAD DATA LOCAL INFILE path so it works also for testing embedded server. mysql-test/suite/pbxt/t/pbxt_bugs.test Fix LOAD DATA LOCAL INFILE path so it works also for testing embedded server. mysql-test/suite/pbxt/t/pbxt_locking.test Disable for embedded, as it needs SHOW PROCESSLIST functionality not available there. mysql-test/suite/pbxt/t/pbxt_transactions.test Disable test for embedded, as it needs ability to connect from outside (mysqldump). mysql-test/suite/pbxt/t/ps_1general.test Fix replace_result for new mysql-test-run.pl. sql/sql_plugin.cc Remove hack that disables PBXT in embedded. storage/pbxt/plug.in Fix crash in PBXT in embedded server. storage/pbxt/src/Makefile.am Remove not needed CFLAGS/CXXFLAGS (they cause autotools to generate different object names, which in turn cause the MYSQL_PLUGIN_DEPENDS_ON_MYSQL_INTERNALS replacement of object files inside library files not to work). storage/pbxt/src/ha_pbxt.cc Ugly hack to allow more threads in embedded server (need a better fix I think). storage/pbxt/src/table_xt.cc Use stderr for logging not stdout (prevent spamming --embedded test suite output with stray messages). storage/pbxt/src/thread_xt.cc Use stderr for logging not stdout (prevent spamming --embedded test suite output with stray messages). storage/pbxt/src/trace_xt.cc Use stderr for logging not stdout (prevent spamming --embedded test suite output with stray messages). === modified file 'config/ac-macros/plugins.m4' --- a/config/ac-macros/plugins.m4 2009-04-25 10:05:32 +0000 +++ b/config/ac-macros/plugins.m4 2009-12-22 10:33:20 +0000 @@ -267,7 +267,6 @@ dnl we have to recompile these modules dnl to compile server parts with the different #defines dnl Normally it happens when we compile the embedded server dnl Thus one should mark such files in his handler using this macro -dnl (currently only one such a file per plugin is supported) dnl dnl --------------------------------------------------------------------------- @@ -463,11 +462,13 @@ dnl Although this is "pretty", it breaks mysql_plugin_defs="$mysql_plugin_defs, [builtin_]$2[_plugin]" [with_plugin_]$2=yes AC_MSG_RESULT([yes]) - m4_ifdef([$11],[ - condition_dependent_plugin_modules="$condition_dependent_plugin_modules m4_bregexp($11, [[^/]+$], [\&])" - condition_dependent_plugin_objects="$condition_dependent_plugin_objects m4_bregexp($11, [[^/]+\.], [\&o])" - condition_dependent_plugin_links="$condition_dependent_plugin_links $6/$11" - condition_dependent_plugin_includes="$condition_dependent_plugin_includes -I[\$(top_srcdir)]/$6/m4_bregexp($11, [^.+[/$]], [\&])" + m4_ifdef([$11], [ + m4_foreach([plugin], [$11], [ + condition_dependent_plugin_modules="$condition_dependent_plugin_modules m4_bregexp(plugin, [[^/]+$], [\&])" + condition_dependent_plugin_objects="$condition_dependent_plugin_objects m4_bregexp(plugin, [[^/]+\.], [\&o])" + condition_dependent_plugin_links="$condition_dependent_plugin_links $6/plugin" + condition_dependent_plugin_includes="$condition_dependent_plugin_includes -I[\$(top_srcdir)]/$6/m4_bregexp(plugin, [^.+[/$]], [\&])" + ]) ]) fi fi === renamed file 'mysql-test/suite/pbxt/t/load_unique_error1.inc' => 'mysql-test/std_data/pbxt_load_unique_error1.inc' === modified file 'mysql-test/suite/pbxt/r/pbxt_bugs.result' --- a/mysql-test/suite/pbxt/r/pbxt_bugs.result 2009-08-17 15:57:58 +0000 +++ b/mysql-test/suite/pbxt/r/pbxt_bugs.result 2009-12-22 10:33:20 +0000 @@ -1212,7 +1212,7 @@ c1 2147483647 DROP TABLE IF EXISTS t1; CREATE TABLE t1 (c1 INTEGER NOT NULL PRIMARY KEY, c2 VARCHAR(255)); -LOAD DATA LOCAL INFILE 'suite/pbxt/t/load_unique_error1.inc' REPLACE INTO TABLE t1 FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (@c1,c2) SET c1 = @c1 % 2; +LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/std_data/pbxt_load_unique_error1.inc' REPLACE INTO TABLE t1 FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (@c1,c2) SET c1 = @c1 % 2; SELECT * FROM t1 ORDER BY c1; c1 c2 0 opq === modified file 'mysql-test/suite/pbxt/t/pbxt_bugs.test' --- a/mysql-test/suite/pbxt/t/pbxt_bugs.test 2009-08-17 15:57:58 +0000 +++ b/mysql-test/suite/pbxt/t/pbxt_bugs.test 2009-12-22 10:33:20 +0000 @@ -921,7 +921,8 @@ SELECT c1 FROM t2; DROP TABLE IF EXISTS t1; --enable_warnings CREATE TABLE t1 (c1 INTEGER NOT NULL PRIMARY KEY, c2 VARCHAR(255)); -LOAD DATA LOCAL INFILE 'suite/pbxt/t/load_unique_error1.inc' REPLACE INTO TABLE t1 FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (@c1,c2) SET c1 = @c1 % 2; +--replace_result $MYSQLTEST_VARDIR MYSQLTEST_VARDIR +eval LOAD DATA LOCAL INFILE '$MYSQLTEST_VARDIR/std_data/pbxt_load_unique_error1.inc' REPLACE INTO TABLE t1 FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (@c1,c2) SET c1 = @c1 % 2; --sorted_result SELECT * FROM t1 ORDER BY c1; DROP TABLE t1; === modified file 'mysql-test/suite/pbxt/t/pbxt_locking.test' --- a/mysql-test/suite/pbxt/t/pbxt_locking.test 2009-04-02 10:03:14 +0000 +++ b/mysql-test/suite/pbxt/t/pbxt_locking.test 2009-12-22 10:33:20 +0000 @@ -1,6 +1,9 @@ # This test covers various aspects of PBXT locking mechanism, including # internal permanent/temporary row locking and MySQL locking +# SHOW PROCESSLIST has hardcoded "Writing to net" as state. +-- source include/not_embedded.inc + # TEST: select for update test drop table if exists t1; === modified file 'mysql-test/suite/pbxt/t/pbxt_transactions.test' --- a/mysql-test/suite/pbxt/t/pbxt_transactions.test 2009-04-02 10:03:14 +0000 +++ b/mysql-test/suite/pbxt/t/pbxt_transactions.test 2009-12-22 10:33:20 +0000 @@ -1,3 +1,6 @@ +# We cannot run mysqldump against embedded server. +-- source include/not_embedded.inc + --disable_warnings drop table if exists t1, t2, t3; --enable_warnings === modified file 'mysql-test/suite/pbxt/t/ps_1general.test' --- a/mysql-test/suite/pbxt/t/ps_1general.test 2009-04-02 10:03:14 +0000 +++ b/mysql-test/suite/pbxt/t/ps_1general.test 2009-12-22 10:33:20 +0000 @@ -582,7 +582,7 @@ prepare stmt1 from ' rename table t5 to create table t5 (a int) ; # rename must fail, t7 does not exist # Clean up the filename here because embedded server reports whole path ---replace_result $MYSQLTEST_VARDIR . master-data/ '' t7.frm t7 +--replace_result $MYSQLTEST_VARDIR . mysqld.1/data/ '' t7.frm t7 --error 1017 execute stmt1 ; create table t7 (a int) ; === modified file 'sql/sql_plugin.cc' --- a/sql/sql_plugin.cc 2009-12-03 11:19:05 +0000 +++ b/sql/sql_plugin.cc 2009-12-22 10:33:20 +0000 @@ -1168,22 +1168,7 @@ int plugin_init(int *argc, char **argv, !my_strnncoll(&my_charset_latin1, (const uchar*) plugin->name, 6, (const uchar*) "InnoDB", 6)) continue; -#ifdef EMBEDDED_LIBRARY - /* - MariaDB: disable PBXT in embedded server. We do this for two reasons - - PBXT currently doesn't work in embedded server (see - https://bugs.launchpad.net/maria/+bug/439889) - - Embedded server is supposed to be "leaner" and our current - understanding of that is "without PBXT". At the same time, we want - regular server to be with PBXT, and since we don't support compiling - embedded server with different options than the regular server, - the only way was to disable PBXT from here. - */ - if (!my_strnncoll(&my_charset_latin1, (const uchar*) plugin->name, - 4, (const uchar*) "PBXT", 4)) - continue; -#endif bzero(&tmp, sizeof(tmp)); tmp.plugin= plugin; tmp.name.str= (char *)plugin->name; === modified file 'storage/pbxt/plug.in' --- a/storage/pbxt/plug.in 2009-05-12 06:44:01 +0000 +++ b/storage/pbxt/plug.in 2009-12-09 21:39:23 +0000 @@ -5,3 +5,4 @@ MYSQL_PLUGIN_STATIC(pbxt, [src/libpbx MYSQL_PLUGIN_ACTIONS(pbxt, [ # AC_CONFIG_FILES(storage/pbxt/src/Makefile) ]) +MYSQL_PLUGIN_DEPENDS_ON_MYSQL_INTERNALS(pbxt, [[src/ha_pbxt.cc],[src/myxt_xt.cc],[src/discover_xt.cc]]) === modified file 'storage/pbxt/src/Makefile.am' --- a/storage/pbxt/src/Makefile.am 2009-11-24 10:55:06 +0000 +++ b/storage/pbxt/src/Makefile.am 2009-12-22 10:33:20 +0000 @@ -46,7 +46,5 @@ libpbxt_la_CFLAGS = $(AM_CFLAGS) -DMYSQ EXTRA_LIBRARIES = libpbxt.a noinst_LIBRARIES = libpbxt.a libpbxt_a_SOURCES = $(libpbxt_la_SOURCES) -libpbxt_a_CXXFLAGS = $(AM_CXXFLAGS) -libpbxt_a_CFLAGS = $(AM_CFLAGS) -std=c99 EXTRA_DIST = pbms_enabled.cc win_inttypes.h === modified file 'storage/pbxt/src/ha_pbxt.cc' --- a/storage/pbxt/src/ha_pbxt.cc 2009-11-27 15:37:02 +0000 +++ b/storage/pbxt/src/ha_pbxt.cc 2009-12-22 10:33:20 +0000 @@ -1175,9 +1175,13 @@ static int pbxt_init(void *p) * +1 Temporary thread (e.g. TempForClose, TempForEnd) */ #ifndef DRIZZLED +#ifdef EMBEDDED_LIBRARY + pbxt_max_threads = 100; +#else if (pbxt_max_threads == 0) pbxt_max_threads = max_connections + 7; #endif +#endif self = xt_init_threading(pbxt_max_threads); /* Create the main self: */ if (!self) goto error_3; === modified file 'storage/pbxt/src/table_xt.cc' --- a/storage/pbxt/src/table_xt.cc 2009-11-25 15:40:51 +0000 +++ b/storage/pbxt/src/table_xt.cc 2009-12-22 10:33:20 +0000 @@ -1297,7 +1297,7 @@ xtPublic void xt_create_table(XTThreadPt XTSortedListInfoRec li_undo; #ifdef TRACE_CREATE_TABLES - printf("CREATE %s\n", name->ps_path); + fprintf(stderr, "CREATE %s\n", name->ps_path); #endif enter_(); if (strlen(xt_last_name_of_path(name->ps_path)) > XT_TABLE_NAME_SIZE-1) @@ -1619,7 +1619,7 @@ xtPublic void xt_drop_table(XTThreadPtr enter_(); #ifdef TRACE_CREATE_TABLES - printf("DROP %s\n", tab_name->ps_path); + fprintf(stderr, "DROP %s\n", tab_name->ps_path); #endif table_pool = tab_lock_table(self, tab_name, FALSE, TRUE, TRUE, &tab); @@ -1777,7 +1777,7 @@ xtPublic void xt_check_table(XTThreadPtr u_llong ext_data_len = 0; #if defined(DUMP_CHECK_TABLE) || defined(CHECK_TABLE_STATS) - printf("\nCHECK TABLE: %s\n", tab->tab_name->ps_path); + fprintf(stderr, "\nCHECK TABLE: %s\n", tab->tab_name->ps_path); #endif xt_lock_mutex(self, &tab->tab_db->db_co_ext_lock); @@ -1787,38 +1787,38 @@ xtPublic void xt_check_table(XTThreadPtr pushr_(xt_unlock_mutex, &tab->tab_rec_lock); #ifdef CHECK_TABLE_STATS - printf("Record buffer size = %lu\n", (u_long) tab->tab_dic.dic_mysql_buf_size); - printf("Fixed length rec. len. = %lu\n", (u_long) tab->tab_dic.dic_mysql_rec_size); - printf("Handle data record size = %lu\n", (u_long) tab->tab_dic.dic_rec_size); - printf("Min/max header size = %d/%d\n", (int) offsetof(XTTabRecFix, rf_data), tab->tab_dic.dic_rec_fixed ? (int) offsetof(XTTabRecFix, rf_data) : (int) offsetof(XTTabRecExtDRec, re_data)); - printf("Min/avg/max record size = %llu/%llu/%llu\n", (u_llong) tab->tab_dic.dic_min_row_size, (u_llong) tab->tab_dic.dic_ave_row_size, (u_llong) tab->tab_dic.dic_max_row_size); + fprintf(stderr, "Record buffer size = %lu\n", (u_long) tab->tab_dic.dic_mysql_buf_size); + fprintf(stderr, "Fixed length rec. len. = %lu\n", (u_long) tab->tab_dic.dic_mysql_rec_size); + fprintf(stderr, "Handle data record size = %lu\n", (u_long) tab->tab_dic.dic_rec_size); + fprintf(stderr, "Min/max header size = %d/%d\n", (int) offsetof(XTTabRecFix, rf_data), tab->tab_dic.dic_rec_fixed ? (int) offsetof(XTTabRecFix, rf_data) : (int) offsetof(XTTabRecExtDRec, re_data)); + fprintf(stderr, "Min/avg/max record size = %llu/%llu/%llu\n", (u_llong) tab->tab_dic.dic_min_row_size, (u_llong) tab->tab_dic.dic_ave_row_size, (u_llong) tab->tab_dic.dic_max_row_size); if (tab->tab_dic.dic_def_ave_row_size) - printf("Avg row len set for tab = %lu\n", (u_long) tab->tab_dic.dic_def_ave_row_size); + fprintf(stderr, "Avg row len set for tab = %lu\n", (u_long) tab->tab_dic.dic_def_ave_row_size); else - printf("Avg row len set for tab = not specified\n"); - printf("Rows fixed length = %s\n", tab->tab_dic.dic_rec_fixed ? "YES" : "NO"); + fprintf(stderr, "Avg row len set for tab = not specified\n"); + fprintf(stderr, "Rows fixed length = %s\n", tab->tab_dic.dic_rec_fixed ? "YES" : "NO"); if (tab->tab_dic.dic_tab_flags & XT_TAB_FLAGS_TEMP_TAB) - printf("Table type = TEMP\n"); + fprintf(stderr, "Table type = TEMP\n"); if (tab->tab_dic.dic_def_ave_row_size) - printf("Maximum fixed size = %lu\n", (u_long) XT_TAB_MAX_FIX_REC_LENGTH_SPEC); + fprintf(stderr, "Maximum fixed size = %lu\n", (u_long) XT_TAB_MAX_FIX_REC_LENGTH_SPEC); else - printf("Maximum fixed size = %lu\n", (u_long) XT_TAB_MAX_FIX_REC_LENGTH); - printf("Minimum variable size = %lu\n", (u_long) XT_TAB_MIN_VAR_REC_LENGTH); - printf("Minimum auto-increment = %llu\n", (u_llong) tab->tab_dic.dic_min_auto_inc); - printf("Number of columns = %lu\n", (u_long) tab->tab_dic.dic_no_of_cols); - printf("Number of fixed columns = %lu\n", (u_long) tab->tab_dic.dic_fix_col_count); - printf("Columns req. for index = %lu\n", (u_long) tab->tab_dic.dic_ind_cols_req); + fprintf(stderr, "Maximum fixed size = %lu\n", (u_long) XT_TAB_MAX_FIX_REC_LENGTH); + fprintf(stderr, "Minimum variable size = %lu\n", (u_long) XT_TAB_MIN_VAR_REC_LENGTH); + fprintf(stderr, "Minimum auto-increment = %llu\n", (u_llong) tab->tab_dic.dic_min_auto_inc); + fprintf(stderr, "Number of columns = %lu\n", (u_long) tab->tab_dic.dic_no_of_cols); + fprintf(stderr, "Number of fixed columns = %lu\n", (u_long) tab->tab_dic.dic_fix_col_count); + fprintf(stderr, "Columns req. for index = %lu\n", (u_long) tab->tab_dic.dic_ind_cols_req); if (tab->tab_dic.dic_ind_rec_len) - printf("Rec len req. for index = %llu\n", (u_llong) tab->tab_dic.dic_ind_rec_len); - printf("Columns req. for blobs = %lu\n", (u_long) tab->tab_dic.dic_blob_cols_req); - printf("Number of blob columns = %lu\n", (u_long) tab->tab_dic.dic_blob_count); - printf("Number of indices = %lu\n", (u_long) tab->tab_dic.dic_key_count); + fprintf(stderr, "Rec len req. for index = %llu\n", (u_llong) tab->tab_dic.dic_ind_rec_len); + fprintf(stderr, "Columns req. for blobs = %lu\n", (u_long) tab->tab_dic.dic_blob_cols_req); + fprintf(stderr, "Number of blob columns = %lu\n", (u_long) tab->tab_dic.dic_blob_count); + fprintf(stderr, "Number of indices = %lu\n", (u_long) tab->tab_dic.dic_key_count); #endif #ifdef DUMP_CHECK_TABLE - printf("Records:-\n"); - printf("Free list: %llu (%llu)\n", (u_llong) tab->tab_rec_free_id, (u_llong) tab->tab_rec_fnum); - printf("EOF: %llu\n", (u_llong) tab->tab_rec_eof_id); + fprintf(stderr, "Records:-\n"); + fprintf(stderr, "Free list: %llu (%llu)\n", (u_llong) tab->tab_rec_free_id, (u_llong) tab->tab_rec_fnum); + fprintf(stderr, "EOF: %llu\n", (u_llong) tab->tab_rec_eof_id); #endif rec_size = XT_REC_EXT_HEADER_SIZE; @@ -1830,24 +1830,24 @@ xtPublic void xt_check_table(XTThreadPtr xt_throw(self); #ifdef DUMP_CHECK_TABLE - printf("%-4llu ", (u_llong) rec_id); + fprintf(stderr, "%-4llu ", (u_llong) rec_id); #endif switch (rec_buf->tr_rec_type_1 & XT_TAB_STATUS_MASK) { case XT_TAB_STATUS_FREED: #ifdef DUMP_CHECK_TABLE - printf("======== "); + fprintf(stderr, "======== "); #endif free_rec_count++; break; case XT_TAB_STATUS_DELETE: #ifdef DUMP_CHECK_TABLE - printf("delete "); + fprintf(stderr, "delete "); #endif delete_rec_count++; break; case XT_TAB_STATUS_FIXED: #ifdef DUMP_CHECK_TABLE - printf("record-F "); + fprintf(stderr, "record-F "); #endif alloc_rec_count++; row_size = myxt_store_row_length(ot, (char *) ot->ot_row_rbuffer + XT_REC_FIX_HEADER_SIZE); @@ -1859,7 +1859,7 @@ xtPublic void xt_check_table(XTThreadPtr break; case XT_TAB_STATUS_VARIABLE: #ifdef DUMP_CHECK_TABLE - printf("record-V "); + fprintf(stderr, "record-V "); #endif alloc_rec_count++; row_size = myxt_load_row_length(ot, tab->tab_dic.dic_rec_size, ot->ot_row_rbuffer + XT_REC_FIX_HEADER_SIZE, NULL); @@ -1871,7 +1871,7 @@ xtPublic void xt_check_table(XTThreadPtr break; case XT_TAB_STATUS_EXT_DLOG: #ifdef DUMP_CHECK_TABLE - printf("record-X "); + fprintf(stderr, "record-X "); #endif alloc_rec_count++; ext_data_len += XT_GET_DISK_4(rec_buf->re_log_dat_siz_4); @@ -1885,9 +1885,9 @@ xtPublic void xt_check_table(XTThreadPtr } #ifdef DUMP_CHECK_TABLE if (rec_buf->tr_rec_type_1 & XT_TAB_STATUS_CLEANED_BIT) - printf("C"); + fprintf(stderr, "C"); else - printf(" "); + fprintf(stderr, " "); #endif prev_rec_id = XT_GET_DISK_4(rec_buf->tr_prev_rec_id_4); xn_id = XT_GET_DISK_4(rec_buf->tr_xact_id_4); @@ -1895,12 +1895,12 @@ xtPublic void xt_check_table(XTThreadPtr switch (rec_buf->tr_rec_type_1 & XT_TAB_STATUS_MASK) { case XT_TAB_STATUS_FREED: #ifdef DUMP_CHECK_TABLE - printf(" prev=%-3llu (xact=%-3llu row=%lu)\n", (u_llong) prev_rec_id, (u_llong) xn_id, (u_long) row_id); + fprintf(stderr, " prev=%-3llu (xact=%-3llu row=%lu)\n", (u_llong) prev_rec_id, (u_llong) xn_id, (u_long) row_id); #endif break; case XT_TAB_STATUS_EXT_DLOG: #ifdef DUMP_CHECK_TABLE - printf(" prev=%-3llu xact=%-3llu row=%lu Xlog=%lu Xoff=%llu Xsiz=%lu\n", (u_llong) prev_rec_id, (u_llong) xn_id, (u_long) row_id, (u_long) XT_GET_DISK_2(rec_buf->re_log_id_2), (u_llong) XT_GET_DISK_6(rec_buf->re_log_offs_6), (u_long) XT_GET_DISK_4(rec_buf->re_log_dat_siz_4)); + fprintf(stderr, " prev=%-3llu xact=%-3llu row=%lu Xlog=%lu Xoff=%llu Xsiz=%lu\n", (u_llong) prev_rec_id, (u_llong) xn_id, (u_long) row_id, (u_long) XT_GET_DISK_2(rec_buf->re_log_id_2), (u_llong) XT_GET_DISK_6(rec_buf->re_log_offs_6), (u_long) XT_GET_DISK_4(rec_buf->re_log_dat_siz_4)); #endif log_size = XT_GET_DISK_4(rec_buf->re_log_dat_siz_4); @@ -1922,7 +1922,7 @@ xtPublic void xt_check_table(XTThreadPtr break; default: #ifdef DUMP_CHECK_TABLE - printf(" prev=%-3llu xact=%-3llu row=%lu\n", (u_llong) prev_rec_id, (u_llong) xn_id, (u_long) row_id); + fprintf(stderr, " prev=%-3llu xact=%-3llu row=%lu\n", (u_llong) prev_rec_id, (u_llong) xn_id, (u_long) row_id); #endif break; } @@ -1931,16 +1931,16 @@ xtPublic void xt_check_table(XTThreadPtr #ifdef CHECK_TABLE_STATS if (!tab->tab_dic.dic_rec_fixed) - printf("Extendend data length = %llu\n", ext_data_len); + fprintf(stderr, "Extendend data length = %llu\n", ext_data_len); if (alloc_rec_count) { - printf("Minumum comp. rec. len. = %llu\n", (u_llong) min_comp_rec_len); - printf("Average comp. rec. len. = %llu\n", (u_llong) ((double) alloc_rec_bytes / (double) alloc_rec_count + (double) 0.5)); - printf("Maximum comp. rec. len. = %llu\n", (u_llong) max_comp_rec_len); - } - printf("Free record count = %llu\n", (u_llong) free_rec_count); - printf("Deleted record count = %llu\n", (u_llong) delete_rec_count); - printf("Allocated record count = %llu\n", (u_llong) alloc_rec_count); + fprintf(stderr, "Minumum comp. rec. len. = %llu\n", (u_llong) min_comp_rec_len); + fprintf(stderr, "Average comp. rec. len. = %llu\n", (u_llong) ((double) alloc_rec_bytes / (double) alloc_rec_count + (double) 0.5)); + fprintf(stderr, "Maximum comp. rec. len. = %llu\n", (u_llong) max_comp_rec_len); + } + fprintf(stderr, "Free record count = %llu\n", (u_llong) free_rec_count); + fprintf(stderr, "Deleted record count = %llu\n", (u_llong) delete_rec_count); + fprintf(stderr, "Allocated record count = %llu\n", (u_llong) alloc_rec_count); #endif if (tab->tab_rec_fnum != free_rec_count) xt_logf(XT_INFO, "Table %s: incorrect number of free blocks, %llu, should be: %llu\n", tab->tab_name, (u_llong) free_rec_count, (u_llong) tab->tab_rec_fnum); @@ -1978,9 +1978,9 @@ xtPublic void xt_check_table(XTThreadPtr pushr_(xt_unlock_mutex, &tab->tab_row_lock); #ifdef DUMP_CHECK_TABLE - printf("Rows:-\n"); - printf("Free list: %llu (%llu)\n", (u_llong) tab->tab_row_free_id, (u_llong) tab->tab_row_fnum); - printf("EOF: %llu\n", (u_llong) tab->tab_row_eof_id); + fprintf(stderr, "Rows:-\n"); + fprintf(stderr, "Free list: %llu (%llu)\n", (u_llong) tab->tab_row_free_id, (u_llong) tab->tab_row_fnum); + fprintf(stderr, "EOF: %llu\n", (u_llong) tab->tab_row_eof_id); #endif rec_id = 1; @@ -1988,13 +1988,13 @@ xtPublic void xt_check_table(XTThreadPtr if (!tab->tab_rows.xt_tc_read_4(ot->ot_row_file, rec_id, &ref_id, self)) xt_throw(self); #ifdef DUMP_CHECK_TABLE - printf("%-3llu ", (u_llong) rec_id); + fprintf(stderr, "%-3llu ", (u_llong) rec_id); #endif #ifdef DUMP_CHECK_TABLE if (ref_id == 0) - printf("====== 0\n"); + fprintf(stderr, "====== 0\n"); else - printf("in use %llu\n", (u_llong) ref_id); + fprintf(stderr, "in use %llu\n", (u_llong) ref_id); #endif rec_id++; } @@ -2026,7 +2026,7 @@ xtPublic void xt_rename_table(XTThreadPt memset(&dic, 0, sizeof(dic)); #ifdef TRACE_CREATE_TABLES - printf("RENAME %s --> %s\n", old_name->ps_path, new_name->ps_path); + fprintf(stderr, "RENAME %s --> %s\n", old_name->ps_path, new_name->ps_path); #endif if (strlen(xt_last_name_of_path(new_name->ps_path)) > XT_TABLE_NAME_SIZE-1) xt_throw_taberr(XT_CONTEXT, XT_ERR_NAME_TOO_LONG, new_name); @@ -2221,7 +2221,7 @@ xtPublic xtBool xt_flush_record_row(XTOp xt_tab_store_header(ot, &rec_head); #ifdef TRACE_FLUSH - printf("FLUSH rec/row %d %s\n", (int) tab->tab_bytes_to_flush, tab->tab_name->ps_path); + fprintf(stderr, "FLUSH rec/row %d %s\n", (int) tab->tab_bytes_to_flush, tab->tab_name->ps_path); fflush(stdout); #endif /* Write the table header: */ @@ -2276,7 +2276,7 @@ xtPublic xtBool xt_flush_record_row(XTOp xt_unlock_mutex_ns(&cp->cp_state_lock); #ifdef TRACE_FLUSH - printf("FLUSH --end-- %s\n", tab->tab_name->ps_path); + fprintf(stderr, "FLUSH --end-- %s\n", tab->tab_name->ps_path); fflush(stdout); #endif xt_unlock_mutex_ns(&tab->tab_rec_flush_lock); === modified file 'storage/pbxt/src/thread_xt.cc' --- a/storage/pbxt/src/thread_xt.cc 2009-11-24 10:55:06 +0000 +++ b/storage/pbxt/src/thread_xt.cc 2009-12-22 10:33:20 +0000 @@ -96,7 +96,7 @@ xtPublic xtBool xt_init_logging(void) { int err; - log_file = stdout; + log_file = stderr; log_level = XT_LOG_TRACE; err = xt_p_mutex_init_with_autoname(&log_mutex, NULL); if (err) { === modified file 'storage/pbxt/src/trace_xt.cc' --- a/storage/pbxt/src/trace_xt.cc 2009-08-17 11:12:36 +0000 +++ b/storage/pbxt/src/trace_xt.cc 2009-12-22 10:33:20 +0000 @@ -109,10 +109,10 @@ xtPublic void xt_print_trace(void) xt_lock_mutex_ns(&trace_mutex); if (trace_log_end > trace_log_offset+1) { trace_log_buffer[trace_log_end] = 0; - printf("%s", trace_log_buffer + trace_log_offset + 1); + fprintf(stderr, "%s", trace_log_buffer + trace_log_offset + 1); } trace_log_buffer[trace_log_offset] = 0; - printf("%s", trace_log_buffer); + fprintf(stderr, "%s", trace_log_buffer); trace_log_offset = 0; trace_log_end = 0; xt_unlock_mutex_ns(&trace_mutex); @@ -379,9 +379,9 @@ xtPublic void xt_dump_conn_tracking(void ptr = conn_info; for (int i=0; i<XT_TRACK_MAX_CONNS; i++) { if (ptr->ci_curr_xact_id || ptr->ci_prev_xact_id) { - printf("%3d curr=%d prev=%d prev-time=%ld\n", (int) ptr->cu_t_id, (int) ptr->ci_curr_xact_id, (int) ptr->ci_prev_xact_id, (long) ptr->ci_prev_xact_time); + fprintf(stderr, "%3d curr=%d prev=%d prev-time=%ld\n", (int) ptr->cu_t_id, (int) ptr->ci_curr_xact_id, (int) ptr->ci_prev_xact_id, (long) ptr->ci_prev_xact_time); if (i+1<XT_TRACK_MAX_CONNS) { - printf(" diff=%d\n", (int) (ptr+1)->ci_curr_xact_id - (int) ptr->ci_curr_xact_id); + fprintf(stderr, " diff=%d\n", (int) (ptr+1)->ci_curr_xact_id - (int) ptr->ci_curr_xact_id); } } ptr++;

1 0

[Maria-developers] bzr commit into Mariadb 5.2, with Maria 2.0:maria/5.2 branch (igor:2740)
by Igor Babaev 22 Dec '09

22 Dec '09

#At lp:maria/5.2 based on revid:igor@askmonty.org-20091221022615-kx5ieiu0okmiupuc 2740 Igor Babaev 2009-12-21 [merge] Merge from 5.2-dsmrr modified: mysql-test/r/innodb_mrr.result mysql-test/r/type_date.result === modified file 'mysql-test/r/innodb_mrr.result' --- a/mysql-test/r/innodb_mrr.result 2009-12-15 07:16:46 +0000 +++ b/mysql-test/r/innodb_mrr.result 2009-12-19 19:54:54 +0000 @@ -318,7 +318,7 @@ filler char(10), key(d), primary key (a, insert into t2 select A.a, B.a, B.a, A.a, 'filler' from t1 A, t1 B; explain select * from t2 force index (d) where d < 10; id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t2 range d d 5 NULL 47 Using index condition; Using MRR +1 SIMPLE t2 range d d 5 NULL 53 Using index condition; Using MRR drop table t2; drop table t1; set @@read_rnd_buffer_size= @read_rnd_buffer_size_save; === modified file 'mysql-test/r/type_date.result' --- a/mysql-test/r/type_date.result 2009-01-13 14:04:28 +0000 +++ b/mysql-test/r/type_date.result 2009-12-19 19:54:54 +0000 @@ -207,6 +207,7 @@ id select_type table type possible_keys Warnings: Warning 1292 Incorrect date value: '0000-00-00' for column 'a' at row 1 Warning 1292 Incorrect date value: '0000-00-00' for column 'a' at row 1 +Warning 1265 Data truncated for column 'a' at row 1 SELECT * FROM t1 WHERE a = '0000-00-00'; a 0000-00-00 @@ -214,6 +215,7 @@ a Warnings: Warning 1292 Incorrect date value: '0000-00-00' for column 'a' at row 1 Warning 1292 Incorrect date value: '0000-00-00' for column 'a' at row 1 +Warning 1265 Data truncated for column 'a' at row 1 SELECT * FROM t2 WHERE a = '0000-00-00'; a 0000-00-00 @@ -245,6 +247,7 @@ id select_type table type possible_keys Warnings: Warning 1292 Incorrect date value: '1000-00-00' for column 'a' at row 1 Warning 1292 Incorrect date value: '1000-00-00' for column 'a' at row 1 +Warning 1265 Data truncated for column 'a' at row 1 SELECT * FROM t1 WHERE a = '1000-00-00'; a 1000-00-00 @@ -252,6 +255,7 @@ a Warnings: Warning 1292 Incorrect date value: '1000-00-00' for column 'a' at row 1 Warning 1292 Incorrect date value: '1000-00-00' for column 'a' at row 1 +Warning 1265 Data truncated for column 'a' at row 1 SELECT * FROM t2 WHERE a = '1000-00-00'; a 1000-00-00

2 1

Re: [Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2785)
by Adam M. Dutko 21 Dec '09

21 Dec '09

Kristian, Sorry for making it seem like I wasn't adressing the group but I thought it was Arjen posing the question about build hosts. On a related note, I just got a netra 1u system I can put opensolaris on and configure. Do you have notes on configuring buildbot in addition to those found on your live journal site that you want me to follow? I'll start working on getting it running. -Adam On Dec 21, 2009 11:29 AM, <knielsen(a)knielsen-hq.org> wrote: #At lp:maria 2785 knielsen(a)knielsen-hq.org 2009-12-21 Don't include the `innodb' suite in the list of default test suites. This suite is for the innodb plugin, which we do not use in MariaDB (we instead have XtraDB as default storage engine, which includes the innodb plugin). We have included the same tests in the main suite, so no point in running them twice (and most of them are disabled anyway due to the absense of the innodb plugin). modified: mysql-test/mysql-test-run.pl === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2009-12-06 17:34:54 +0000 +++ b/mysql-test/mysql-test-run.pl 2009-12-21 16:26:36 +0000 @@ -126,7 +126,7 @@ my $path_config_file; # The ge # executables will be used by the test suite. our $opt_vs_config = $ENV{'MTR_VS_CONFIG'}; -my $DEFAULT_SUITES= "main,binlog,federated,rpl,innodb,maria,parts"; +my $DEFAULT_SUITES= "main,binlog,federated,rpl,maria,parts"; my $opt_suites; our $opt_verbose= 0; # Verbose output, enable with --verbose _______________________________________________ Mailing list: https://launchpad.net/~maria-develop...

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2785: Don't include the `innodb' suite in the list of default test suites.
by noreply＠launchpad.net 21 Dec '09

21 Dec '09

------------------------------------------------------------ revno: 2785 committer: knielsen(a)knielsen-hq.org branch nick: mariadb-5.1 timestamp: Mon 2009-12-21 17:26:36 +0100 message: Don't include the `innodb' suite in the list of default test suites. This suite is for the innodb plugin, which we do not use in MariaDB (we instead have XtraDB as default storage engine, which includes the innodb plugin). We have included the same tests in the main suite, so no point in running them twice (and most of them are disabled anyway due to the absense of the innodb plugin). modified: mysql-test/mysql-test-run.pl -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2785)
by knielsen＠knielsen-hq.org 21 Dec '09

21 Dec '09

#At lp:maria 2785 knielsen(a)knielsen-hq.org 2009-12-21 Don't include the `innodb' suite in the list of default test suites. This suite is for the innodb plugin, which we do not use in MariaDB (we instead have XtraDB as default storage engine, which includes the innodb plugin). We have included the same tests in the main suite, so no point in running them twice (and most of them are disabled anyway due to the absense of the innodb plugin). modified: mysql-test/mysql-test-run.pl === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2009-12-06 17:34:54 +0000 +++ b/mysql-test/mysql-test-run.pl 2009-12-21 16:26:36 +0000 @@ -126,7 +126,7 @@ my $path_config_file; # The ge # executables will be used by the test suite. our $opt_vs_config = $ENV{'MTR_VS_CONFIG'}; -my $DEFAULT_SUITES= "main,binlog,federated,rpl,innodb,maria,parts"; +my $DEFAULT_SUITES= "main,binlog,federated,rpl,maria,parts"; my $opt_suites; our $opt_verbose= 0; # Verbose output, enable with --verbose

1 0

Re: [Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2785)
by Adam M. Dutko 21 Dec '09

21 Dec '09

Arjen, Do you have a matrix of distros/platforms you'd like to see added to the build pool? I have some more compute power that I might be able to add to the build pool. -Adam On Dec 18, 2009 6:02 PM, "Michael Widenius" <monty(a)askmonty.org> wrote: Hi! >>>>> "Kristian" == Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: Kristian> Arjen Lentz <arjen(a)openquery.com> writes: >> Which other external libraries are currently... Kristian> These are the ones I know of: Kristian> zlib Kristian> libedit Kristian> libreadline Kristian> yassl Kristian> libevent Kristian> I think most (all?) of them have a ./configure option to use the system one Kristian> instead of the bundled one. Kristian> I guess the main problem with boost is the sheer size. The largest of the Kristian> above is yassl, with 182 files and <2 MB total size. Of the above, the following should be easy to remove: zlib libedit libreadline yassl is patched (or will be patched) libevent is probably patched; If not, it shouldn't be here. Regards, Monty _______________________________________________ Mailing list: https://launchpad.net/~maria-develope...

2 1

[Maria-developers] Updated (by Guest): mysqlbinlog: remove undesired effects of format description binlog statement (50)
by worklog-noreply＠askmonty.org 21 Dec '09

21 Dec '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: mysqlbinlog: remove undesired effects of format description binlog statement CREATION DATE..: Mon, 17 Aug 2009, 14:12 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 50 (http://askmonty.org/worklog/?tid=50) VERSION........: Server-9.x STATUS.........: Complete PRIORITY.......: 60 WORKED HOURS...: 16 ESTIMATE.......: 4 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: -=-=(Guest - Mon, 21 Dec 2009, 15:40)=-=- Status updated. --- /tmp/wklog.50.old.26984 2009-12-21 13:40:53.000000000 +0000 +++ /tmp/wklog.50.new.26984 2009-12-21 13:40:53.000000000 +0000 @@ -1 +1 @@ -Code-Review +Complete -=-=(Knielsen - Mon, 05 Oct 2009, 14:38)=-=- Follow-up with MySQL developers on patch Worked 2 hours and estimate 4 hours remain (original estimate unchanged). -=-=(Knielsen - Tue, 29 Sep 2009, 10:12)=-=- Category updated. --- /tmp/wklog.50.old.3293 2009-09-29 10:12:26.000000000 +0300 +++ /tmp/wklog.50.new.3293 2009-09-29 10:12:26.000000000 +0300 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Knielsen - Tue, 29 Sep 2009, 10:12)=-=- Status updated. --- /tmp/wklog.50.old.3293 2009-09-29 10:12:26.000000000 +0300 +++ /tmp/wklog.50.new.3293 2009-09-29 10:12:26.000000000 +0300 @@ -1 +1 @@ -Un-Assigned +Code-Review -=-=(Knielsen - Mon, 28 Sep 2009, 16:00)=-=- Updated patch against MySQL 5.1.39 Fixes a couple of issues with the patch. Discussions with MySQL developer. Worked 4 hours and estimate 6 hours remain (original estimate unchanged). -=-=(Knielsen - Wed, 09 Sep 2009, 18:31)=-=- Discussed with MySQL developers on Bug#46640. Committed a suggested fix: https://lists.launchpad.net/maria-developers/msg00926.html Worked 8 hours and estimate 10 hours remain (original estimate increased by 18 hours). -=-=(Guest - Wed, 09 Sep 2009, 11:50)=-=- High Level Description modified. --- /tmp/wklog.50.old.11584 2009-09-09 11:50:12.000000000 +0300 +++ /tmp/wklog.50.new.11584 2009-09-09 11:50:12.000000000 +0300 @@ -5,7 +5,11 @@ This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. -2. When one applies "format description binlog statement" at the slave, it -will change the slave's server_id when applied. +2. When one executes a BINLOG statement (containing "format description binlog +statement" or other events), this will cause subsequent statements run in +the same session to be binlogged with the server id of the last event +executed, not with the real server id. This can cause problems like infinite +recursive application in a circular replication topology. This WL is to fix these issues. + -=-=(Knielsen - Tue, 08 Sep 2009, 15:07)=-=- Low Level Design modified. --- /tmp/wklog.50.old.19208 2009-09-08 15:07:09.000000000 +0300 +++ /tmp/wklog.50.new.19208 2009-09-08 15:07:09.000000000 +0300 @@ -1 +1,34 @@ +I think the fix for point 2 is to replace the call to +apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct +call to ev->apply_event(). However, I need to check two things to be sure this +is correct: + +1. The existing code does + + if (!ev->when) ev->when= my_time(0) + +Need to understand if this is needed/correct or not. + +2. The existing code does ev->update_pos(). I think this is redundant for +BINLOG statement (as it uses a fake rli structure), but I need to check to +make sure. + +Once this is done, point 1 may no longer be needed. The user can use +--base64-output=never when applying the output to a 5.0 server, and omit that +option when applying the output to a 5.1 server. There should be no need to +omit the format description event in the output when other BINLOG statements +for row events are present, as it no longer changes the server id. In fact the +format description event is required to be able to execute other BINLOG +statements, as its purpose is to define the binary format of the events +contained in these statements. + +Alternatively, we could implement that if the format description event of the +source binlog has server version < 5.1, and --base64-output=auto (default), +then the format description event is omitted (and should any BINLOG statement +be needed, unlikely as it is from a 5.0 server, we will need to throw an +error). + +The binlog format version for 5.0 and 5.1 is actually the same, hence the need +to look at server version to guess if the format description event can be +omitted. -=-=(Knielsen - Tue, 08 Sep 2009, 14:29)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.17531 2009-09-08 14:29:30.000000000 +0300 +++ /tmp/wklog.50.new.17531 2009-09-08 14:29:30.000000000 +0300 @@ -12,3 +12,37 @@ One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? +First, the problem is not the format description log event. In fact all log +events have their own value of server_id, and every event in the BINLOG +statement is executed with its own id as server_id. + +It seems it was introduced deliberately in the patch for Bug#32407. This patch +makes sql thread and BINLOG statement share code path for executing the binary +event, even though the bug is really about something different (outputting +format description event to allow proper execution of other BINLOG +statements). + +Nevertheless, I think using the server_id in the events of a BINLOG statement +is wrong: + +1. It is different behaviour from executing normal statement-based events not +written using BINLOG. + +2. It causes the possibility for infinite cycle of replication in a cyclic +replication setup, if the server ids in BINLOG are different from all other +server ids in the cycle. + +3. The functionality of applying events with original server id from the event +is still available by pointing the slave thread to the binlog. The +functionality of applying row-based binlog events with server id of the server +executing the BINLOG statements is not otherwise available. + +In fact most of the code from the slave thread that is now also run when +executing BINLOG statements is redundant, or does the wrong thing (like +updating binlog position). + +Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be +able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is +it to be able to simulate the effect of the slave sql thread? I think the +former is the most important one. + -=-=(Knielsen - Mon, 07 Sep 2009, 10:06)=-=- Research worklog, bugs, and existing server code. Worked 2 hours and estimate 0 hours remain (original estimate increased by 2 hours). ------------------------------------------------------------ -=-=(View All Progress Notes, 12 total)=-=- http://askmonty.org/worklog/index.pl?tid=50&nolimit=1 DESCRIPTION: According to complaints in BUG#46640: 1. mysqlbinlog will emit a 5.0-incompatible "format description binlog statement" even when reading a binary log that was produced by 5.0. This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. 2. When one executes a BINLOG statement (containing "format description binlog statement" or other events), this will cause subsequent statements run in the same session to be binlogged with the server id of the last event executed, not with the real server id. This can cause problems like infinite recursive application in a circular replication topology. This WL is to fix these issues. HIGH-LEVEL SPECIFICATION: First item ---------- AFAIU what needs to be done is: 1. record a source server version (it is in the first binlog event). 2. don't emit a BINLOG statement if the recorded version number is 5.0.x or below. and we'll get a 5.0-applicable binlog. Second item ----------- One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? First, the problem is not the format description log event. In fact all log events have their own value of server_id, and every event in the BINLOG statement is executed with its own id as server_id. It seems it was introduced deliberately in the patch for Bug#32407. This patch makes sql thread and BINLOG statement share code path for executing the binary event, even though the bug is really about something different (outputting format description event to allow proper execution of other BINLOG statements). Nevertheless, I think using the server_id in the events of a BINLOG statement is wrong: 1. It is different behaviour from executing normal statement-based events not written using BINLOG. 2. It causes the possibility for infinite cycle of replication in a cyclic replication setup, if the server ids in BINLOG are different from all other server ids in the cycle. 3. The functionality of applying events with original server id from the event is still available by pointing the slave thread to the binlog. The functionality of applying row-based binlog events with server id of the server executing the BINLOG statements is not otherwise available. In fact most of the code from the slave thread that is now also run when executing BINLOG statements is redundant, or does the wrong thing (like updating binlog position). Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is it to be able to simulate the effect of the slave sql thread? I think the former is the most important one. LOW-LEVEL DESIGN: I think the fix for point 2 is to replace the call to apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct call to ev->apply_event(). However, I need to check two things to be sure this is correct: 1. The existing code does if (!ev->when) ev->when= my_time(0) Need to understand if this is needed/correct or not. 2. The existing code does ev->update_pos(). I think this is redundant for BINLOG statement (as it uses a fake rli structure), but I need to check to make sure. Once this is done, point 1 may no longer be needed. The user can use --base64-output=never when applying the output to a 5.0 server, and omit that option when applying the output to a 5.1 server. There should be no need to omit the format description event in the output when other BINLOG statements for row events are present, as it no longer changes the server id. In fact the format description event is required to be able to execute other BINLOG statements, as its purpose is to define the binary format of the events contained in these statements. Alternatively, we could implement that if the format description event of the source binlog has server version < 5.1, and --base64-output=auto (default), then the format description event is omitted (and should any BINLOG statement be needed, unlikely as it is from a 5.0 server, we will need to throw an error). The binlog format version for 5.0 and 5.1 is actually the same, hence the need to look at server version to guess if the format description event can be omitted. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Guest): mysqlbinlog: remove undesired effects of format description binlog statement (50)
by worklog-noreply＠askmonty.org 21 Dec '09

21 Dec '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: mysqlbinlog: remove undesired effects of format description binlog statement CREATION DATE..: Mon, 17 Aug 2009, 14:12 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 50 (http://askmonty.org/worklog/?tid=50) VERSION........: Server-9.x STATUS.........: Complete PRIORITY.......: 60 WORKED HOURS...: 16 ESTIMATE.......: 4 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: -=-=(Guest - Mon, 21 Dec 2009, 15:40)=-=- Status updated. --- /tmp/wklog.50.old.26984 2009-12-21 13:40:53.000000000 +0000 +++ /tmp/wklog.50.new.26984 2009-12-21 13:40:53.000000000 +0000 @@ -1 +1 @@ -Code-Review +Complete -=-=(Knielsen - Mon, 05 Oct 2009, 14:38)=-=- Follow-up with MySQL developers on patch Worked 2 hours and estimate 4 hours remain (original estimate unchanged). -=-=(Knielsen - Tue, 29 Sep 2009, 10:12)=-=- Category updated. --- /tmp/wklog.50.old.3293 2009-09-29 10:12:26.000000000 +0300 +++ /tmp/wklog.50.new.3293 2009-09-29 10:12:26.000000000 +0300 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Knielsen - Tue, 29 Sep 2009, 10:12)=-=- Status updated. --- /tmp/wklog.50.old.3293 2009-09-29 10:12:26.000000000 +0300 +++ /tmp/wklog.50.new.3293 2009-09-29 10:12:26.000000000 +0300 @@ -1 +1 @@ -Un-Assigned +Code-Review -=-=(Knielsen - Mon, 28 Sep 2009, 16:00)=-=- Updated patch against MySQL 5.1.39 Fixes a couple of issues with the patch. Discussions with MySQL developer. Worked 4 hours and estimate 6 hours remain (original estimate unchanged). -=-=(Knielsen - Wed, 09 Sep 2009, 18:31)=-=- Discussed with MySQL developers on Bug#46640. Committed a suggested fix: https://lists.launchpad.net/maria-developers/msg00926.html Worked 8 hours and estimate 10 hours remain (original estimate increased by 18 hours). -=-=(Guest - Wed, 09 Sep 2009, 11:50)=-=- High Level Description modified. --- /tmp/wklog.50.old.11584 2009-09-09 11:50:12.000000000 +0300 +++ /tmp/wklog.50.new.11584 2009-09-09 11:50:12.000000000 +0300 @@ -5,7 +5,11 @@ This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. -2. When one applies "format description binlog statement" at the slave, it -will change the slave's server_id when applied. +2. When one executes a BINLOG statement (containing "format description binlog +statement" or other events), this will cause subsequent statements run in +the same session to be binlogged with the server id of the last event +executed, not with the real server id. This can cause problems like infinite +recursive application in a circular replication topology. This WL is to fix these issues. + -=-=(Knielsen - Tue, 08 Sep 2009, 15:07)=-=- Low Level Design modified. --- /tmp/wklog.50.old.19208 2009-09-08 15:07:09.000000000 +0300 +++ /tmp/wklog.50.new.19208 2009-09-08 15:07:09.000000000 +0300 @@ -1 +1,34 @@ +I think the fix for point 2 is to replace the call to +apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct +call to ev->apply_event(). However, I need to check two things to be sure this +is correct: + +1. The existing code does + + if (!ev->when) ev->when= my_time(0) + +Need to understand if this is needed/correct or not. + +2. The existing code does ev->update_pos(). I think this is redundant for +BINLOG statement (as it uses a fake rli structure), but I need to check to +make sure. + +Once this is done, point 1 may no longer be needed. The user can use +--base64-output=never when applying the output to a 5.0 server, and omit that +option when applying the output to a 5.1 server. There should be no need to +omit the format description event in the output when other BINLOG statements +for row events are present, as it no longer changes the server id. In fact the +format description event is required to be able to execute other BINLOG +statements, as its purpose is to define the binary format of the events +contained in these statements. + +Alternatively, we could implement that if the format description event of the +source binlog has server version < 5.1, and --base64-output=auto (default), +then the format description event is omitted (and should any BINLOG statement +be needed, unlikely as it is from a 5.0 server, we will need to throw an +error). + +The binlog format version for 5.0 and 5.1 is actually the same, hence the need +to look at server version to guess if the format description event can be +omitted. -=-=(Knielsen - Tue, 08 Sep 2009, 14:29)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.17531 2009-09-08 14:29:30.000000000 +0300 +++ /tmp/wklog.50.new.17531 2009-09-08 14:29:30.000000000 +0300 @@ -12,3 +12,37 @@ One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? +First, the problem is not the format description log event. In fact all log +events have their own value of server_id, and every event in the BINLOG +statement is executed with its own id as server_id. + +It seems it was introduced deliberately in the patch for Bug#32407. This patch +makes sql thread and BINLOG statement share code path for executing the binary +event, even though the bug is really about something different (outputting +format description event to allow proper execution of other BINLOG +statements). + +Nevertheless, I think using the server_id in the events of a BINLOG statement +is wrong: + +1. It is different behaviour from executing normal statement-based events not +written using BINLOG. + +2. It causes the possibility for infinite cycle of replication in a cyclic +replication setup, if the server ids in BINLOG are different from all other +server ids in the cycle. + +3. The functionality of applying events with original server id from the event +is still available by pointing the slave thread to the binlog. The +functionality of applying row-based binlog events with server id of the server +executing the BINLOG statements is not otherwise available. + +In fact most of the code from the slave thread that is now also run when +executing BINLOG statements is redundant, or does the wrong thing (like +updating binlog position). + +Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be +able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is +it to be able to simulate the effect of the slave sql thread? I think the +former is the most important one. + -=-=(Knielsen - Mon, 07 Sep 2009, 10:06)=-=- Research worklog, bugs, and existing server code. Worked 2 hours and estimate 0 hours remain (original estimate increased by 2 hours). ------------------------------------------------------------ -=-=(View All Progress Notes, 12 total)=-=- http://askmonty.org/worklog/index.pl?tid=50&nolimit=1 DESCRIPTION: According to complaints in BUG#46640: 1. mysqlbinlog will emit a 5.0-incompatible "format description binlog statement" even when reading a binary log that was produced by 5.0. This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. 2. When one executes a BINLOG statement (containing "format description binlog statement" or other events), this will cause subsequent statements run in the same session to be binlogged with the server id of the last event executed, not with the real server id. This can cause problems like infinite recursive application in a circular replication topology. This WL is to fix these issues. HIGH-LEVEL SPECIFICATION: First item ---------- AFAIU what needs to be done is: 1. record a source server version (it is in the first binlog event). 2. don't emit a BINLOG statement if the recorded version number is 5.0.x or below. and we'll get a 5.0-applicable binlog. Second item ----------- One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? First, the problem is not the format description log event. In fact all log events have their own value of server_id, and every event in the BINLOG statement is executed with its own id as server_id. It seems it was introduced deliberately in the patch for Bug#32407. This patch makes sql thread and BINLOG statement share code path for executing the binary event, even though the bug is really about something different (outputting format description event to allow proper execution of other BINLOG statements). Nevertheless, I think using the server_id in the events of a BINLOG statement is wrong: 1. It is different behaviour from executing normal statement-based events not written using BINLOG. 2. It causes the possibility for infinite cycle of replication in a cyclic replication setup, if the server ids in BINLOG are different from all other server ids in the cycle. 3. The functionality of applying events with original server id from the event is still available by pointing the slave thread to the binlog. The functionality of applying row-based binlog events with server id of the server executing the BINLOG statements is not otherwise available. In fact most of the code from the slave thread that is now also run when executing BINLOG statements is redundant, or does the wrong thing (like updating binlog position). Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is it to be able to simulate the effect of the slave sql thread? I think the former is the most important one. LOW-LEVEL DESIGN: I think the fix for point 2 is to replace the call to apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct call to ev->apply_event(). However, I need to check two things to be sure this is correct: 1. The existing code does if (!ev->when) ev->when= my_time(0) Need to understand if this is needed/correct or not. 2. The existing code does ev->update_pos(). I think this is redundant for BINLOG statement (as it uses a fake rli structure), but I need to check to make sure. Once this is done, point 1 may no longer be needed. The user can use --base64-output=never when applying the output to a 5.0 server, and omit that option when applying the output to a 5.1 server. There should be no need to omit the format description event in the output when other BINLOG statements for row events are present, as it no longer changes the server id. In fact the format description event is required to be able to execute other BINLOG statements, as its purpose is to define the binary format of the events contained in these statements. Alternatively, we could implement that if the format description event of the source binlog has server version < 5.1, and --base64-output=auto (default), then the format description event is omitted (and should any BINLOG statement be needed, unlikely as it is from a 5.0 server, we will need to throw an error). The binlog format version for 5.0 and 5.1 is actually the same, hence the need to look at server version to guess if the format description event can be omitted. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0