- developers - lists.mariadb.org

[Maria-developers] Rev 2746: Fix mysql-test-run --embedded: in file:///home/psergey/bzr-new/mysql-5.1-maria-contd4/
by Sergey Petrunya 03 Oct '09

03 Oct '09

At file:///home/psergey/bzr-new/mysql-5.1-maria-contd4/ ------------------------------------------------------------ revno: 2746 revision-id: psergey(a)askmonty.org-20091003185654-ibglul7pqli0mei9 parent: psergey(a)askmonty.org-20091001212739-x4mzj5e7u8pig71f committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: mysql-5.1-maria-contd4 timestamp: Sat 2009-10-03 22:56:54 +0400 message: Fix mysql-test-run --embedded: - Don't run testcase for BUG#45632 with embedded server. === modified file 'mysql-test/r/variables_debug.result' --- a/mysql-test/r/variables_debug.result 2009-06-24 22:22:20 +0000 +++ b/mysql-test/r/variables_debug.result 2009-10-03 18:56:54 +0000 @@ -10,18 +10,3 @@ select @@debug; @@debug T -set session debug="t"; -show session variables like 'debug'; -Variable_name Value -debug t -set session debug="t"; -show session variables like 'debug'; -Variable_name Value -debug t -set session debug="d:t"; -show session variables like 'debug'; -Variable_name Value -debug d:t -show session variables like 'debug'; -Variable_name Value -debug t === added file 'mysql-test/r/variables_debug_notembedded.result' --- a/mysql-test/r/variables_debug_notembedded.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/r/variables_debug_notembedded.result 2009-10-03 18:56:54 +0000 @@ -0,0 +1,15 @@ +set session debug="t"; +show session variables like 'debug'; +Variable_name Value +debug t +set session debug="t"; +show session variables like 'debug'; +Variable_name Value +debug t +set session debug="d:t"; +show session variables like 'debug'; +Variable_name Value +debug d:t +show session variables like 'debug'; +Variable_name Value +debug t === modified file 'mysql-test/t/variables_debug.test' --- a/mysql-test/t/variables_debug.test 2009-06-24 22:22:20 +0000 +++ b/mysql-test/t/variables_debug.test 2009-10-03 18:56:54 +0000 @@ -10,31 +10,3 @@ select @@debug; set debug= '-P'; select @@debug; - -# -# Checks that assigning variable 'debug' in one session has no influence on -# other session. (BUG#45632 of bugs.mysql.com) -# -connect(con1,localhost,root,,test,,); -connect(con2,localhost,root,,test,,); - -# makes output independant of current debug status -connection con1; -set session debug="t"; -show session variables like 'debug'; -connection con2; -set session debug="t"; -show session variables like 'debug'; - -# checks influence one session debug variable on another -connection con1; -set session debug="d:t"; -show session variables like 'debug'; -connection con2; -show session variables like 'debug'; - -disconnect con1; -disconnect con2; - -connection default; - === added file 'mysql-test/t/variables_debug_notembedded.test' --- a/mysql-test/t/variables_debug_notembedded.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/variables_debug_notembedded.test 2009-10-03 18:56:54 +0000 @@ -0,0 +1,34 @@ +# +# Checks that assigning variable 'debug' in one session has no influence on +# other session. (BUG#45632 of bugs.mysql.com) +# +# (this will not work with embedded server because mysqltest_embedded uses the +# same thread for all connections) +# +--source include/have_debug.inc +--source include/not_embedded.inc + +connect(con1,localhost,root,,test,,); +connect(con2,localhost,root,,test,,); + +# makes output independant of current debug status +connection con1; +set session debug="t"; +show session variables like 'debug'; +connection con2; +set session debug="t"; +show session variables like 'debug'; + +# checks influence one session debug variable on another +connection con1; +set session debug="d:t"; +show session variables like 'debug'; +connection con2; +show session variables like 'debug'; + +disconnect con1; +disconnect con2; + +connection default; + +

1 0

[Maria-developers] Updated (by Guest): Add a mysqlbinlog option to change the used database (36)
by worklog-noreply＠askmonty.org 03 Oct '09

03 Oct '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to change the used database CREATION DATE..: Fri, 07 Aug 2009, 14:57 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 36 (http://askmonty.org/worklog/?tid=36) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Guest - Sat, 03 Oct 2009, 02:27)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.23366 2009-10-03 02:27:45.000000000 +0300 +++ /tmp/wklog.36.new.23366 2009-10-03 02:27:45.000000000 +0300 @@ -1,77 +1 @@ -Context -------- -(See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global -overview) -At the moment, the server has a replication slave option - - --replicate-rewrite-db="from->to" - -the option affects -- Table_map_log_event (all RBR events) -- Load_log_event (LOAD DATA) -- Query_log_event (SBR-based updates, with the usual assumption that the - statement refers to tables in current database, so that changing the current - database will make the statement to work on a table in a different database). - -See also MySQL BUG#42941. Note this bug is fixed in MySQL 5.1.37, which is not -merged into MariaDB at the time of writing, but planned to be merged before -release. - -What we could do ----------------- - -Option1: make mysqlbinlog accept --replicate-rewrite-db option -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Make mysqlbinlog accept --replicate-rewrite-db options and process them to the -same extent as replication slave would process --replicate-rewrite-db option. - - -Option2: Add database-agnostic RBR events and --strip-db option -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Right now RBR events require a databasename. It is not possible to have RBR -event stream that won't mention which database the events are for. When I -tried to use debugger and specify empty database name, attempt to apply the -binlog resulted in this error: - -090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on -opening tables, - -We could do as follows: -- Make the server interpret empty database name in RBR event (i.e. in a - Table_map_log_event) as "use current database". Binlog slave thread - probably should not allow such events as it doesn't have a natural current - database. -- Add a mysqlbinlog --strip-db option that would - = not produce any "USE dbname" statements - = change databasename for all RBR events to be empty - -That way, mysqlbinlog output will be database-agnostic and apply to the -current database. -(this will have the usual limitations that we assume that all statements in -the binlog refer to the current database). - -Option3: Enhance database rewrite -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -If there is a need to support database change for statements that use -dbname.tablename notation and are replicated as statements (i.e. are DDL -statements and/or DML statements that are binlogged as statements), -then that could be supported as follows: - -- Make the server's parser recognize special form of comments - - /* !database-alias(oldname,newname) */ - - and save the mapping somewhere - -- Put the hooks in table open and name resolution code to use the saved - mapping. - - -Once we've done the above, it will be easy to perform a complete, -no-compromise or restrictions database name change in binary log. - -It will be possible to do the rewrites either on the slave ( ---replicate-rewrite-db will work for all kinds of statements), or in -mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to -parse the statement). - +G9m7yq <a href="http://ijfmyyjtveuu.com/">ijfmyyjtveuu</a>, [url=http://jeczeaqoxbpt.com/]jeczeaqoxbpt[/url] [link=http://nrisgrrvcrkm.com/]nrisgrrvcrkm[/link] http://edmnozsmotmt.com/ -=-=(Guest - Tue, 15 Sep 2009, 18:04)=-=- Low Level Design modified. --- /tmp/wklog.36.old.19322 2009-09-15 18:04:49.000000000 +0300 +++ /tmp/wklog.36.new.19322 2009-09-15 18:04:49.000000000 +0300 @@ -191,7 +191,7 @@ - In process_event() function add switch case for Load_log_event and add print_use_stmt() invocations where needed (according to the - events lis above), e.g.: + events list above), e.g.: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, -=-=(Guest - Tue, 15 Sep 2009, 15:53)=-=- Low Level Design modified. --- /tmp/wklog.36.old.13421 2009-09-15 15:53:31.000000000 +0300 +++ /tmp/wklog.36.new.13421 2009-09-15 15:53:31.000000000 +0300 @@ -150,10 +150,17 @@ following events (see process_event() function): - Query_log_event -- Execute_load_query_log_event -- Create_file_log_event - -TODO. Needed to check this list requires carefully !!! +- Load_log_event +- Execute_load_query_log_event [ :public Query_log_event ] +- Create_file_log_event [ :public Load_log_event ] + +TODO. Needed to check this list carefully (not sure for Create_file_log_event) + Notes. + - In replication, only Query_log_event and Load_log_event uses + rpl_filter->get_rewrite_db(); + - In mysqlbinlog (process_event), Execute_load_query_log_event + and Create_file_log_event are processed in separate switch + cases. And Load_log_event is processed in the default switch case. Conditions for emiting use-statement: - LOG_EVENT_SUPPRESS_USE_F is OFF for the event @@ -182,8 +189,9 @@ */ } -- In process_event() function add print_use_stmt() invocations where - needed (according to the events lis above), e.g.: +- In process_event() function add switch case for Load_log_event and + add print_use_stmt() invocations where needed (according to the + events lis above), e.g.: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, @@ -207,6 +215,11 @@ } break; ... + case LOAD_EVENT: + print_use_stmt((Load_log_event*)ev, print_event_info); + break; + default: + ... } ... } -=-=(Guest - Tue, 15 Sep 2009, 12:12)=-=- Low Level Design modified. --- /tmp/wklog.36.old.3961 2009-09-15 12:12:26.000000000 +0300 +++ /tmp/wklog.36.new.3961 2009-09-15 12:12:26.000000000 +0300 @@ -144,6 +144,8 @@ 3. Supporting rewrite-db for SBR events --------------------------------------- +Limited to emiting USE <db_to> instead of USE <db_from>. + USE statements can be emited by mysqlbinlog as a result of processing the following events (see process_event() function): -=-=(Guest - Tue, 15 Sep 2009, 12:08)=-=- Low Level Design modified. --- /tmp/wklog.36.old.3794 2009-09-15 12:08:54.000000000 +0300 +++ /tmp/wklog.36.new.3794 2009-09-15 12:08:54.000000000 +0300 @@ -1 +1,229 @@ +Content +------- +1. Adding rewrite-db option +2. Supporting rewrite-db option for RBR events +3. Supporting rewrite-db option for SBR events + (Limited to affecting only USE statements) +4. Current status + +1. Adding rewrite-db option +--------------------------- + +1.1. Syntax: + --rewrite-db='db_from->db_to' + +1.2. Add 'OPT_REWRITE_DB' to 'options_client' (in client_priv.h). + +1.3. In mysqlbinlog.cc: + +- Add { "rewrite-db", OPT_REWRITE_DB, ...} record to my_long_options: +- Add Rpl_filter object to mysqlbinlog.cc + + Rpl_filter* binlog_filter; + +- Add corresponding switch case to get_one_option(): + + case OPT_REWRITE_DB: + <extract db-from and db-to strings> + binlog_filter->add_db_rewrite(db_from, db_to); + break; + . +Note. To make Rpl_filter usable in a MYSQL_CLIENT context, few small +additional changes are required: + +- In sql_list.cc/h, Sql_alloc::new(size_t) and Sql_alloc::new[](size_t) + uses sql_alloc() which is THD dependent. These are to be modified + as follows: + + #ifdef MYSQL_CLIENT + extern MEM_ROOT sql_list_client_mem_root; // defined in sql_list.cc + #endif + + class Sql_alloc + { ... + static void *operator new(size_t size) throw () + { + #ifndef MYSQL_CLIENT + return sql_alloc(size); + #else + return alloc_root(&sql_list_client_mem_root, size); + #endif + } + static void *operator new[](size_t size) throw () + { + #ifndef MYSQL_CLIENT + return sql_alloc(size); + #else + return alloc_root(&sql_list_client_mem_root, size); + #endif + } + ... + } + +- In rpl_filter.cc: + + Rpl_filter::Rpl_filter() : + ... + { + #ifdef MYSQL_CLIENT + init_alloc_root(&sql_list_client_mem_root, ...); + #endif + ... + } + + Rpl_filter::~Rpl_filter() + { ... + #ifdef MYSQL_CLIENT + free_root(&sql_list_client_mem_root, ...); + #endif + } + +2. Supporting rewrite-db for RBR events +--------------------------------------- + +In binlog, each row operation event is preceded by Table map event(s) which maps +table id(s) to database and table names. So, it's enough to support rewriting +database name in a Table map. + +2.1. Add rewrite_db() member to Table_map_log_event: + + int Table_map_log_event::rewrite_db( + const char* new_db, + size_t new_db_len, + const Format_description_log_event* desc) + { + /* 1. In temp_buf member (possibly reallocating it) rewrite + event length, db length, and db parts + 2. Change m_dblen and m_dbnam members + */ + } + +Comment. This function assumes that temp_buf member contains Table map +binlog representaion (temp_buf is used for creating corresponding +BINLOG statement). + +2.2. In mysqlbinlog modify corresponding switch case in the +process_event() function: + + Exit_status process_event( + PRINT_EVENT_INFO *print_event_info, + Log_event *ev, ...) + { + ... + switch (ev_type) { + ... + case TABLE_MAP_EVENT: + { + Table_map_log_event *map= ((Table_map_log_event *)ev); + if (shall_skip_database(map->get_db_name())) + { ... + } + // WL36 + size_t new_len= 0; + const char* new_db= binlog_filter->get_rewrite_db( + map->get_db_name(), &new_len); + if (new_len && map->rewrite_db(new_db, new_len, + glob_description_event)) + { error("Could not rewrite database name"); + goto err; + } + } + case WRITE_ROWS_EVENT: + case DELETE_ROWS_EVENT: + case UPDATE_ROWS_EVENT: + ... + } + ... + } + +Comment. Rpl_filter::get_rewrite_db(db_from, &len): if filter contains +a (db_from, db_to) pair, this function returns pointer to db_to and +sets len = db_to length; otherwise, it returns db_from and does not +change len value. + +3. Supporting rewrite-db for SBR events +--------------------------------------- + +USE statements can be emited by mysqlbinlog as a result of processing the +following events (see process_event() function): + +- Query_log_event +- Execute_load_query_log_event +- Create_file_log_event + +TODO. Needed to check this list requires carefully !!! + +Conditions for emiting use-statement: +- LOG_EVENT_SUPPRESS_USE_F is OFF for the event + (e.g. it is ON for 'create database' statement) +- event's db name differs from db_name in PRINT_EVENT_INFO + (PRINT_EVENT_INFO keeps db name of the last issued USE statement; + initially, this db name is empty). + +3.1. In mysqlbinlog.cc + +- Add the following function: + + void print_use_stmt(Log_event* event, PRINT_EVENT_INFO* pinfo) + { + if (event->flags & LOG_EVENT_SUPPRESS_USE_F) + return; + /* + - For events listed above get db_from = event->db; + - If db_from is the same as pinfo->db then return; + - If there is rewrite-db rule db_from->db_to, + set db = db_to. Else set db = db_from; + - Print "use <db>" to mysqlbinlog output + - Set pinfo->db = db_from + (this suppresses emiting use-statements by corresponding + log_event's print-function) + */ + } + +- In process_event() function add print_use_stmt() invocations where + needed (according to the events lis above), e.g.: + + Exit_status process_event( + PRINT_EVENT_INFO *print_event_info, + Log_event *ev, ...) + { + ... + switch (ev_type) { + case QUERY_EVENT: + if (shall_skip_database(((Query_log_event*)ev)->db)) + goto end; + if (opt_base64_output_mode == BASE64_OUTPUT_ALWAYS) + { + // Possibly in case of rewite-db rule for ev->db + // a warning should be emited here (see note below) + ... write_event_header_and_base64(ev, ...) ... + } + else + { + print_use_stmt((Query_log_event*)ev, print_event_info); + ev->print(result_file, print_event_info); + } + break; + ... + } + ... + } + +Note. write_event_header_and_base64() does not print use-statement. It +produces BINLOG statement using ev->temp_buf content (i.e. the binary +log representation of the event). We don't rewrite temp_buf here with +db_to name (as we do it for Table map event) - this implies the +limitation 3 mentioned above. +Question: Is supporting of rewite_db + --base64-output really needed +currently? + +4. Current status +----------------- + +The outlined design (implemented for mysql-5.1.37) is tested for +simple test-cases. + +TODO. 1. Check list of events which can emit use-statement. + 2. Supporting of rewite_db + --base64-output ? + -=-=(Guest - Mon, 14 Sep 2009, 11:51)=-=- Low Level Design modified. --- /tmp/wklog.36.old.9711 2009-09-14 11:51:43.000000000 +0300 +++ /tmp/wklog.36.new.9711 2009-09-14 11:51:43.000000000 +0300 @@ -1 +1 @@ -Pay no attention: just check for having access + -=-=(Guest - Mon, 14 Sep 2009, 11:51)=-=- Low Level Design modified. --- /tmp/wklog.36.old.9678 2009-09-14 11:51:28.000000000 +0300 +++ /tmp/wklog.36.new.9678 2009-09-14 11:51:28.000000000 +0300 @@ -1 +1 @@ - +Pay no attention: just check for having access -=-=(Knielsen - Mon, 17 Aug 2009, 12:44)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.7834 2009-08-17 12:44:17.000000000 +0300 +++ /tmp/wklog.36.new.7834 2009-08-17 12:44:17.000000000 +0300 @@ -13,7 +13,9 @@ statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). -See also MySQL BUG#42941. +See also MySQL BUG#42941. Note this bug is fixed in MySQL 5.1.37, which is not +merged into MariaDB at the time of writing, but planned to be merged before +release. What we could do ---------------- -=-=(Guest - Sun, 16 Aug 2009, 17:11)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.27162 2009-08-16 17:11:12.000000000 +0300 +++ /tmp/wklog.36.new.27162 2009-08-16 17:11:12.000000000 +0300 @@ -13,6 +13,8 @@ statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). +See also MySQL BUG#42941. + What we could do ---------------- -=-=(Psergey - Mon, 10 Aug 2009, 15:41)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.13035 2009-08-10 15:41:51.000000000 +0300 +++ /tmp/wklog.36.new.13035 2009-08-10 15:41:51.000000000 +0300 @@ -1,5 +1,7 @@ Context ------- +(See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global +overview) At the moment, the server has a replication slave option --replicate-rewrite-db="from->to" ------------------------------------------------------------ -=-=(View All Progress Notes, 14 total)=-=- http://askmonty.org/worklog/index.pl?tid=36&nolimit=1 DESCRIPTION: Sometimes there is a need to take a binary log and apply it to a database with a different name than the original name of the database on binlog producer. If one is using statement-based replication, he can achieve this by grepping out "USE dbname" statements out of the output of mysqlbinlog(*). With row-based replication this is no longer possible, as database name is encoded within the the BINLOG '....' statement. This task is about adding an option to mysqlbinlog that would allow to change the names of used databases in both RBR and SBR events. (*) this implies that all statements refer to tables in the current database, doesn't catch updates made inside stored functions and so forth, but still works for a practially-important subset of cases. HIGH-LEVEL SPECIFICATION: G9m7yq <a href="http://ijfmyyjtveuu.com/">ijfmyyjtveuu</a>, [url=http://jeczeaqoxbpt.com/]jeczeaqoxbpt[/url] [link=http://nrisgrrvcrkm.com/]nrisgrrvcrkm[/link] http://edmnozsmotmt.com/ LOW-LEVEL DESIGN: Content ------- 1. Adding rewrite-db option 2. Supporting rewrite-db option for RBR events 3. Supporting rewrite-db option for SBR events (Limited to affecting only USE statements) 4. Current status 1. Adding rewrite-db option --------------------------- 1.1. Syntax: --rewrite-db='db_from->db_to' 1.2. Add 'OPT_REWRITE_DB' to 'options_client' (in client_priv.h). 1.3. In mysqlbinlog.cc: - Add { "rewrite-db", OPT_REWRITE_DB, ...} record to my_long_options: - Add Rpl_filter object to mysqlbinlog.cc Rpl_filter* binlog_filter; - Add corresponding switch case to get_one_option(): case OPT_REWRITE_DB: <extract db-from and db-to strings> binlog_filter->add_db_rewrite(db_from, db_to); break; . Note. To make Rpl_filter usable in a MYSQL_CLIENT context, few small additional changes are required: - In sql_list.cc/h, Sql_alloc::new(size_t) and Sql_alloc::new[](size_t) uses sql_alloc() which is THD dependent. These are to be modified as follows: #ifdef MYSQL_CLIENT extern MEM_ROOT sql_list_client_mem_root; // defined in sql_list.cc #endif class Sql_alloc { ... static void *operator new(size_t size) throw () { #ifndef MYSQL_CLIENT return sql_alloc(size); #else return alloc_root(&sql_list_client_mem_root, size); #endif } static void *operator new[](size_t size) throw () { #ifndef MYSQL_CLIENT return sql_alloc(size); #else return alloc_root(&sql_list_client_mem_root, size); #endif } ... } - In rpl_filter.cc: Rpl_filter::Rpl_filter() : ... { #ifdef MYSQL_CLIENT init_alloc_root(&sql_list_client_mem_root, ...); #endif ... } Rpl_filter::~Rpl_filter() { ... #ifdef MYSQL_CLIENT free_root(&sql_list_client_mem_root, ...); #endif } 2. Supporting rewrite-db for RBR events --------------------------------------- In binlog, each row operation event is preceded by Table map event(s) which maps table id(s) to database and table names. So, it's enough to support rewriting database name in a Table map. 2.1. Add rewrite_db() member to Table_map_log_event: int Table_map_log_event::rewrite_db( const char* new_db, size_t new_db_len, const Format_description_log_event* desc) { /* 1. In temp_buf member (possibly reallocating it) rewrite event length, db length, and db parts 2. Change m_dblen and m_dbnam members */ } Comment. This function assumes that temp_buf member contains Table map binlog representaion (temp_buf is used for creating corresponding BINLOG statement). 2.2. In mysqlbinlog modify corresponding switch case in the process_event() function: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, Log_event *ev, ...) { ... switch (ev_type) { ... case TABLE_MAP_EVENT: { Table_map_log_event *map= ((Table_map_log_event *)ev); if (shall_skip_database(map->get_db_name())) { ... } // WL36 size_t new_len= 0; const char* new_db= binlog_filter->get_rewrite_db( map->get_db_name(), &new_len); if (new_len && map->rewrite_db(new_db, new_len, glob_description_event)) { error("Could not rewrite database name"); goto err; } } case WRITE_ROWS_EVENT: case DELETE_ROWS_EVENT: case UPDATE_ROWS_EVENT: ... } ... } Comment. Rpl_filter::get_rewrite_db(db_from, &len): if filter contains a (db_from, db_to) pair, this function returns pointer to db_to and sets len = db_to length; otherwise, it returns db_from and does not change len value. 3. Supporting rewrite-db for SBR events --------------------------------------- Limited to emiting USE <db_to> instead of USE <db_from>. USE statements can be emited by mysqlbinlog as a result of processing the following events (see process_event() function): - Query_log_event - Load_log_event - Execute_load_query_log_event [ :public Query_log_event ] - Create_file_log_event [ :public Load_log_event ] TODO. Needed to check this list carefully (not sure for Create_file_log_event) Notes. - In replication, only Query_log_event and Load_log_event uses rpl_filter->get_rewrite_db(); - In mysqlbinlog (process_event), Execute_load_query_log_event and Create_file_log_event are processed in separate switch cases. And Load_log_event is processed in the default switch case. Conditions for emiting use-statement: - LOG_EVENT_SUPPRESS_USE_F is OFF for the event (e.g. it is ON for 'create database' statement) - event's db name differs from db_name in PRINT_EVENT_INFO (PRINT_EVENT_INFO keeps db name of the last issued USE statement; initially, this db name is empty). 3.1. In mysqlbinlog.cc - Add the following function: void print_use_stmt(Log_event* event, PRINT_EVENT_INFO* pinfo) { if (event->flags & LOG_EVENT_SUPPRESS_USE_F) return; /* - For events listed above get db_from = event->db; - If db_from is the same as pinfo->db then return; - If there is rewrite-db rule db_from->db_to, set db = db_to. Else set db = db_from; - Print "use <db>" to mysqlbinlog output - Set pinfo->db = db_from (this suppresses emiting use-statements by corresponding log_event's print-function) */ } - In process_event() function add switch case for Load_log_event and add print_use_stmt() invocations where needed (according to the events list above), e.g.: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, Log_event *ev, ...) { ... switch (ev_type) { case QUERY_EVENT: if (shall_skip_database(((Query_log_event*)ev)->db)) goto end; if (opt_base64_output_mode == BASE64_OUTPUT_ALWAYS) { // Possibly in case of rewite-db rule for ev->db // a warning should be emited here (see note below) ... write_event_header_and_base64(ev, ...) ... } else { print_use_stmt((Query_log_event*)ev, print_event_info); ev->print(result_file, print_event_info); } break; ... case LOAD_EVENT: print_use_stmt((Load_log_event*)ev, print_event_info); break; default: ... } ... } Note. write_event_header_and_base64() does not print use-statement. It produces BINLOG statement using ev->temp_buf content (i.e. the binary log representation of the event). We don't rewrite temp_buf here with db_to name (as we do it for Table map event) - this implies the limitation 3 mentioned above. Question: Is supporting of rewite_db + --base64-output really needed currently? 4. Current status ----------------- The outlined design (implemented for mysql-5.1.37) is tested for simple test-cases. TODO. 1. Check list of events which can emit use-statement. 2. Supporting of rewite_db + --base64-output ? ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

2 1

[Maria-developers] Updated (by Knielsen): Add a mysqlbinlog option to change the used database (36)
by worklog-noreply＠askmonty.org 03 Oct '09

03 Oct '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to change the used database CREATION DATE..: Fri, 07 Aug 2009, 14:57 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 36 (http://askmonty.org/worklog/?tid=36) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Knielsen - Sat, 03 Oct 2009, 08:34)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.5353 2009-10-03 08:34:43.000000000 +0300 +++ /tmp/wklog.36.new.5353 2009-10-03 08:34:43.000000000 +0300 @@ -1 +1,77 @@ -G9m7yq <a href="http://ijfmyyjtveuu.com/">ijfmyyjtveuu</a>, [url=http://jeczeaqoxbpt.com/]jeczeaqoxbpt[/url] [link=http://nrisgrrvcrkm.com/]nrisgrrvcrkm[/link] http://edmnozsmotmt.com/ +Context +------- +(See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global +overview) +At the moment, the server has a replication slave option + + --replicate-rewrite-db="from->to" + +the option affects +- Table_map_log_event (all RBR events) +- Load_log_event (LOAD DATA) +- Query_log_event (SBR-based updates, with the usual assumption that the + statement refers to tables in current database, so that changing the current + database will make the statement to work on a table in a different database). + +See also MySQL BUG#42941. Note this bug is fixed in MySQL 5.1.37, which is not +merged into MariaDB at the time of writing, but planned to be merged before +release. + +What we could do +---------------- + +Option1: make mysqlbinlog accept --replicate-rewrite-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make mysqlbinlog accept --replicate-rewrite-db options and process them to the +same extent as replication slave would process --replicate-rewrite-db option. + + +Option2: Add database-agnostic RBR events and --strip-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Right now RBR events require a databasename. It is not possible to have RBR +event stream that won't mention which database the events are for. When I +tried to use debugger and specify empty database name, attempt to apply the +binlog resulted in this error: + +090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on +opening tables, + +We could do as follows: +- Make the server interpret empty database name in RBR event (i.e. in a + Table_map_log_event) as "use current database". Binlog slave thread + probably should not allow such events as it doesn't have a natural current + database. +- Add a mysqlbinlog --strip-db option that would + = not produce any "USE dbname" statements + = change databasename for all RBR events to be empty + +That way, mysqlbinlog output will be database-agnostic and apply to the +current database. +(this will have the usual limitations that we assume that all statements in +the binlog refer to the current database). + +Option3: Enhance database rewrite +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +If there is a need to support database change for statements that use +dbname.tablename notation and are replicated as statements (i.e. are DDL +statements and/or DML statements that are binlogged as statements), +then that could be supported as follows: + +- Make the server's parser recognize special form of comments + + /* !database-alias(oldname,newname) */ + + and save the mapping somewhere + +- Put the hooks in table open and name resolution code to use the saved + mapping. + + +Once we've done the above, it will be easy to perform a complete, +no-compromise or restrictions database name change in binary log. + +It will be possible to do the rewrites either on the slave ( +--replicate-rewrite-db will work for all kinds of statements), or in +mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to +parse the statement). + -=-=(Guest - Sat, 03 Oct 2009, 02:27)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.23366 2009-10-03 02:27:45.000000000 +0300 +++ /tmp/wklog.36.new.23366 2009-10-03 02:27:45.000000000 +0300 @@ -1,77 +1 @@ -Context -------- -(See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global -overview) -At the moment, the server has a replication slave option - - --replicate-rewrite-db="from->to" - -the option affects -- Table_map_log_event (all RBR events) -- Load_log_event (LOAD DATA) -- Query_log_event (SBR-based updates, with the usual assumption that the - statement refers to tables in current database, so that changing the current - database will make the statement to work on a table in a different database). - -See also MySQL BUG#42941. Note this bug is fixed in MySQL 5.1.37, which is not -merged into MariaDB at the time of writing, but planned to be merged before -release. - -What we could do ----------------- - -Option1: make mysqlbinlog accept --replicate-rewrite-db option -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Make mysqlbinlog accept --replicate-rewrite-db options and process them to the -same extent as replication slave would process --replicate-rewrite-db option. - - -Option2: Add database-agnostic RBR events and --strip-db option -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Right now RBR events require a databasename. It is not possible to have RBR -event stream that won't mention which database the events are for. When I -tried to use debugger and specify empty database name, attempt to apply the -binlog resulted in this error: - -090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on -opening tables, - -We could do as follows: -- Make the server interpret empty database name in RBR event (i.e. in a - Table_map_log_event) as "use current database". Binlog slave thread - probably should not allow such events as it doesn't have a natural current - database. -- Add a mysqlbinlog --strip-db option that would - = not produce any "USE dbname" statements - = change databasename for all RBR events to be empty - -That way, mysqlbinlog output will be database-agnostic and apply to the -current database. -(this will have the usual limitations that we assume that all statements in -the binlog refer to the current database). - -Option3: Enhance database rewrite -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -If there is a need to support database change for statements that use -dbname.tablename notation and are replicated as statements (i.e. are DDL -statements and/or DML statements that are binlogged as statements), -then that could be supported as follows: - -- Make the server's parser recognize special form of comments - - /* !database-alias(oldname,newname) */ - - and save the mapping somewhere - -- Put the hooks in table open and name resolution code to use the saved - mapping. - - -Once we've done the above, it will be easy to perform a complete, -no-compromise or restrictions database name change in binary log. - -It will be possible to do the rewrites either on the slave ( ---replicate-rewrite-db will work for all kinds of statements), or in -mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to -parse the statement). - +G9m7yq <a href="http://ijfmyyjtveuu.com/">ijfmyyjtveuu</a>, [url=http://jeczeaqoxbpt.com/]jeczeaqoxbpt[/url] [link=http://nrisgrrvcrkm.com/]nrisgrrvcrkm[/link] http://edmnozsmotmt.com/ -=-=(Guest - Tue, 15 Sep 2009, 18:04)=-=- Low Level Design modified. --- /tmp/wklog.36.old.19322 2009-09-15 18:04:49.000000000 +0300 +++ /tmp/wklog.36.new.19322 2009-09-15 18:04:49.000000000 +0300 @@ -191,7 +191,7 @@ - In process_event() function add switch case for Load_log_event and add print_use_stmt() invocations where needed (according to the - events lis above), e.g.: + events list above), e.g.: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, -=-=(Guest - Tue, 15 Sep 2009, 15:53)=-=- Low Level Design modified. --- /tmp/wklog.36.old.13421 2009-09-15 15:53:31.000000000 +0300 +++ /tmp/wklog.36.new.13421 2009-09-15 15:53:31.000000000 +0300 @@ -150,10 +150,17 @@ following events (see process_event() function): - Query_log_event -- Execute_load_query_log_event -- Create_file_log_event - -TODO. Needed to check this list requires carefully !!! +- Load_log_event +- Execute_load_query_log_event [ :public Query_log_event ] +- Create_file_log_event [ :public Load_log_event ] + +TODO. Needed to check this list carefully (not sure for Create_file_log_event) + Notes. + - In replication, only Query_log_event and Load_log_event uses + rpl_filter->get_rewrite_db(); + - In mysqlbinlog (process_event), Execute_load_query_log_event + and Create_file_log_event are processed in separate switch + cases. And Load_log_event is processed in the default switch case. Conditions for emiting use-statement: - LOG_EVENT_SUPPRESS_USE_F is OFF for the event @@ -182,8 +189,9 @@ */ } -- In process_event() function add print_use_stmt() invocations where - needed (according to the events lis above), e.g.: +- In process_event() function add switch case for Load_log_event and + add print_use_stmt() invocations where needed (according to the + events lis above), e.g.: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, @@ -207,6 +215,11 @@ } break; ... + case LOAD_EVENT: + print_use_stmt((Load_log_event*)ev, print_event_info); + break; + default: + ... } ... } -=-=(Guest - Tue, 15 Sep 2009, 12:12)=-=- Low Level Design modified. --- /tmp/wklog.36.old.3961 2009-09-15 12:12:26.000000000 +0300 +++ /tmp/wklog.36.new.3961 2009-09-15 12:12:26.000000000 +0300 @@ -144,6 +144,8 @@ 3. Supporting rewrite-db for SBR events --------------------------------------- +Limited to emiting USE <db_to> instead of USE <db_from>. + USE statements can be emited by mysqlbinlog as a result of processing the following events (see process_event() function): -=-=(Guest - Tue, 15 Sep 2009, 12:08)=-=- Low Level Design modified. --- /tmp/wklog.36.old.3794 2009-09-15 12:08:54.000000000 +0300 +++ /tmp/wklog.36.new.3794 2009-09-15 12:08:54.000000000 +0300 @@ -1 +1,229 @@ +Content +------- +1. Adding rewrite-db option +2. Supporting rewrite-db option for RBR events +3. Supporting rewrite-db option for SBR events + (Limited to affecting only USE statements) +4. Current status + +1. Adding rewrite-db option +--------------------------- + +1.1. Syntax: + --rewrite-db='db_from->db_to' + +1.2. Add 'OPT_REWRITE_DB' to 'options_client' (in client_priv.h). + +1.3. In mysqlbinlog.cc: + +- Add { "rewrite-db", OPT_REWRITE_DB, ...} record to my_long_options: +- Add Rpl_filter object to mysqlbinlog.cc + + Rpl_filter* binlog_filter; + +- Add corresponding switch case to get_one_option(): + + case OPT_REWRITE_DB: + <extract db-from and db-to strings> + binlog_filter->add_db_rewrite(db_from, db_to); + break; + . +Note. To make Rpl_filter usable in a MYSQL_CLIENT context, few small +additional changes are required: + +- In sql_list.cc/h, Sql_alloc::new(size_t) and Sql_alloc::new[](size_t) + uses sql_alloc() which is THD dependent. These are to be modified + as follows: + + #ifdef MYSQL_CLIENT + extern MEM_ROOT sql_list_client_mem_root; // defined in sql_list.cc + #endif + + class Sql_alloc + { ... + static void *operator new(size_t size) throw () + { + #ifndef MYSQL_CLIENT + return sql_alloc(size); + #else + return alloc_root(&sql_list_client_mem_root, size); + #endif + } + static void *operator new[](size_t size) throw () + { + #ifndef MYSQL_CLIENT + return sql_alloc(size); + #else + return alloc_root(&sql_list_client_mem_root, size); + #endif + } + ... + } + +- In rpl_filter.cc: + + Rpl_filter::Rpl_filter() : + ... + { + #ifdef MYSQL_CLIENT + init_alloc_root(&sql_list_client_mem_root, ...); + #endif + ... + } + + Rpl_filter::~Rpl_filter() + { ... + #ifdef MYSQL_CLIENT + free_root(&sql_list_client_mem_root, ...); + #endif + } + +2. Supporting rewrite-db for RBR events +--------------------------------------- + +In binlog, each row operation event is preceded by Table map event(s) which maps +table id(s) to database and table names. So, it's enough to support rewriting +database name in a Table map. + +2.1. Add rewrite_db() member to Table_map_log_event: + + int Table_map_log_event::rewrite_db( + const char* new_db, + size_t new_db_len, + const Format_description_log_event* desc) + { + /* 1. In temp_buf member (possibly reallocating it) rewrite + event length, db length, and db parts + 2. Change m_dblen and m_dbnam members + */ + } + +Comment. This function assumes that temp_buf member contains Table map +binlog representaion (temp_buf is used for creating corresponding +BINLOG statement). + +2.2. In mysqlbinlog modify corresponding switch case in the +process_event() function: + + Exit_status process_event( + PRINT_EVENT_INFO *print_event_info, + Log_event *ev, ...) + { + ... + switch (ev_type) { + ... + case TABLE_MAP_EVENT: + { + Table_map_log_event *map= ((Table_map_log_event *)ev); + if (shall_skip_database(map->get_db_name())) + { ... + } + // WL36 + size_t new_len= 0; + const char* new_db= binlog_filter->get_rewrite_db( + map->get_db_name(), &new_len); + if (new_len && map->rewrite_db(new_db, new_len, + glob_description_event)) + { error("Could not rewrite database name"); + goto err; + } + } + case WRITE_ROWS_EVENT: + case DELETE_ROWS_EVENT: + case UPDATE_ROWS_EVENT: + ... + } + ... + } + +Comment. Rpl_filter::get_rewrite_db(db_from, &len): if filter contains +a (db_from, db_to) pair, this function returns pointer to db_to and +sets len = db_to length; otherwise, it returns db_from and does not +change len value. + +3. Supporting rewrite-db for SBR events +--------------------------------------- + +USE statements can be emited by mysqlbinlog as a result of processing the +following events (see process_event() function): + +- Query_log_event +- Execute_load_query_log_event +- Create_file_log_event + +TODO. Needed to check this list requires carefully !!! + +Conditions for emiting use-statement: +- LOG_EVENT_SUPPRESS_USE_F is OFF for the event + (e.g. it is ON for 'create database' statement) +- event's db name differs from db_name in PRINT_EVENT_INFO + (PRINT_EVENT_INFO keeps db name of the last issued USE statement; + initially, this db name is empty). + +3.1. In mysqlbinlog.cc + +- Add the following function: + + void print_use_stmt(Log_event* event, PRINT_EVENT_INFO* pinfo) + { + if (event->flags & LOG_EVENT_SUPPRESS_USE_F) + return; + /* + - For events listed above get db_from = event->db; + - If db_from is the same as pinfo->db then return; + - If there is rewrite-db rule db_from->db_to, + set db = db_to. Else set db = db_from; + - Print "use <db>" to mysqlbinlog output + - Set pinfo->db = db_from + (this suppresses emiting use-statements by corresponding + log_event's print-function) + */ + } + +- In process_event() function add print_use_stmt() invocations where + needed (according to the events lis above), e.g.: + + Exit_status process_event( + PRINT_EVENT_INFO *print_event_info, + Log_event *ev, ...) + { + ... + switch (ev_type) { + case QUERY_EVENT: + if (shall_skip_database(((Query_log_event*)ev)->db)) + goto end; + if (opt_base64_output_mode == BASE64_OUTPUT_ALWAYS) + { + // Possibly in case of rewite-db rule for ev->db + // a warning should be emited here (see note below) + ... write_event_header_and_base64(ev, ...) ... + } + else + { + print_use_stmt((Query_log_event*)ev, print_event_info); + ev->print(result_file, print_event_info); + } + break; + ... + } + ... + } + +Note. write_event_header_and_base64() does not print use-statement. It +produces BINLOG statement using ev->temp_buf content (i.e. the binary +log representation of the event). We don't rewrite temp_buf here with +db_to name (as we do it for Table map event) - this implies the +limitation 3 mentioned above. +Question: Is supporting of rewite_db + --base64-output really needed +currently? + +4. Current status +----------------- + +The outlined design (implemented for mysql-5.1.37) is tested for +simple test-cases. + +TODO. 1. Check list of events which can emit use-statement. + 2. Supporting of rewite_db + --base64-output ? + -=-=(Guest - Mon, 14 Sep 2009, 11:51)=-=- Low Level Design modified. --- /tmp/wklog.36.old.9711 2009-09-14 11:51:43.000000000 +0300 +++ /tmp/wklog.36.new.9711 2009-09-14 11:51:43.000000000 +0300 @@ -1 +1 @@ -Pay no attention: just check for having access + -=-=(Guest - Mon, 14 Sep 2009, 11:51)=-=- Low Level Design modified. --- /tmp/wklog.36.old.9678 2009-09-14 11:51:28.000000000 +0300 +++ /tmp/wklog.36.new.9678 2009-09-14 11:51:28.000000000 +0300 @@ -1 +1 @@ - +Pay no attention: just check for having access -=-=(Knielsen - Mon, 17 Aug 2009, 12:44)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.7834 2009-08-17 12:44:17.000000000 +0300 +++ /tmp/wklog.36.new.7834 2009-08-17 12:44:17.000000000 +0300 @@ -13,7 +13,9 @@ statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). -See also MySQL BUG#42941. +See also MySQL BUG#42941. Note this bug is fixed in MySQL 5.1.37, which is not +merged into MariaDB at the time of writing, but planned to be merged before +release. What we could do ---------------- -=-=(Guest - Sun, 16 Aug 2009, 17:11)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.27162 2009-08-16 17:11:12.000000000 +0300 +++ /tmp/wklog.36.new.27162 2009-08-16 17:11:12.000000000 +0300 @@ -13,6 +13,8 @@ statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). +See also MySQL BUG#42941. + What we could do ---------------- ------------------------------------------------------------ -=-=(View All Progress Notes, 15 total)=-=- http://askmonty.org/worklog/index.pl?tid=36&nolimit=1 DESCRIPTION: Sometimes there is a need to take a binary log and apply it to a database with a different name than the original name of the database on binlog producer. If one is using statement-based replication, he can achieve this by grepping out "USE dbname" statements out of the output of mysqlbinlog(*). With row-based replication this is no longer possible, as database name is encoded within the the BINLOG '....' statement. This task is about adding an option to mysqlbinlog that would allow to change the names of used databases in both RBR and SBR events. (*) this implies that all statements refer to tables in the current database, doesn't catch updates made inside stored functions and so forth, but still works for a practially-important subset of cases. HIGH-LEVEL SPECIFICATION: Context ------- (See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global overview) At the moment, the server has a replication slave option --replicate-rewrite-db="from->to" the option affects - Table_map_log_event (all RBR events) - Load_log_event (LOAD DATA) - Query_log_event (SBR-based updates, with the usual assumption that the statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). See also MySQL BUG#42941. Note this bug is fixed in MySQL 5.1.37, which is not merged into MariaDB at the time of writing, but planned to be merged before release. What we could do ---------------- Option1: make mysqlbinlog accept --replicate-rewrite-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Make mysqlbinlog accept --replicate-rewrite-db options and process them to the same extent as replication slave would process --replicate-rewrite-db option. Option2: Add database-agnostic RBR events and --strip-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Right now RBR events require a databasename. It is not possible to have RBR event stream that won't mention which database the events are for. When I tried to use debugger and specify empty database name, attempt to apply the binlog resulted in this error: 090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on opening tables, We could do as follows: - Make the server interpret empty database name in RBR event (i.e. in a Table_map_log_event) as "use current database". Binlog slave thread probably should not allow such events as it doesn't have a natural current database. - Add a mysqlbinlog --strip-db option that would = not produce any "USE dbname" statements = change databasename for all RBR events to be empty That way, mysqlbinlog output will be database-agnostic and apply to the current database. (this will have the usual limitations that we assume that all statements in the binlog refer to the current database). Option3: Enhance database rewrite ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If there is a need to support database change for statements that use dbname.tablename notation and are replicated as statements (i.e. are DDL statements and/or DML statements that are binlogged as statements), then that could be supported as follows: - Make the server's parser recognize special form of comments /* !database-alias(oldname,newname) */ and save the mapping somewhere - Put the hooks in table open and name resolution code to use the saved mapping. Once we've done the above, it will be easy to perform a complete, no-compromise or restrictions database name change in binary log. It will be possible to do the rewrites either on the slave ( --replicate-rewrite-db will work for all kinds of statements), or in mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to parse the statement). LOW-LEVEL DESIGN: Content ------- 1. Adding rewrite-db option 2. Supporting rewrite-db option for RBR events 3. Supporting rewrite-db option for SBR events (Limited to affecting only USE statements) 4. Current status 1. Adding rewrite-db option --------------------------- 1.1. Syntax: --rewrite-db='db_from->db_to' 1.2. Add 'OPT_REWRITE_DB' to 'options_client' (in client_priv.h). 1.3. In mysqlbinlog.cc: - Add { "rewrite-db", OPT_REWRITE_DB, ...} record to my_long_options: - Add Rpl_filter object to mysqlbinlog.cc Rpl_filter* binlog_filter; - Add corresponding switch case to get_one_option(): case OPT_REWRITE_DB: <extract db-from and db-to strings> binlog_filter->add_db_rewrite(db_from, db_to); break; . Note. To make Rpl_filter usable in a MYSQL_CLIENT context, few small additional changes are required: - In sql_list.cc/h, Sql_alloc::new(size_t) and Sql_alloc::new[](size_t) uses sql_alloc() which is THD dependent. These are to be modified as follows: #ifdef MYSQL_CLIENT extern MEM_ROOT sql_list_client_mem_root; // defined in sql_list.cc #endif class Sql_alloc { ... static void *operator new(size_t size) throw () { #ifndef MYSQL_CLIENT return sql_alloc(size); #else return alloc_root(&sql_list_client_mem_root, size); #endif } static void *operator new[](size_t size) throw () { #ifndef MYSQL_CLIENT return sql_alloc(size); #else return alloc_root(&sql_list_client_mem_root, size); #endif } ... } - In rpl_filter.cc: Rpl_filter::Rpl_filter() : ... { #ifdef MYSQL_CLIENT init_alloc_root(&sql_list_client_mem_root, ...); #endif ... } Rpl_filter::~Rpl_filter() { ... #ifdef MYSQL_CLIENT free_root(&sql_list_client_mem_root, ...); #endif } 2. Supporting rewrite-db for RBR events --------------------------------------- In binlog, each row operation event is preceded by Table map event(s) which maps table id(s) to database and table names. So, it's enough to support rewriting database name in a Table map. 2.1. Add rewrite_db() member to Table_map_log_event: int Table_map_log_event::rewrite_db( const char* new_db, size_t new_db_len, const Format_description_log_event* desc) { /* 1. In temp_buf member (possibly reallocating it) rewrite event length, db length, and db parts 2. Change m_dblen and m_dbnam members */ } Comment. This function assumes that temp_buf member contains Table map binlog representaion (temp_buf is used for creating corresponding BINLOG statement). 2.2. In mysqlbinlog modify corresponding switch case in the process_event() function: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, Log_event *ev, ...) { ... switch (ev_type) { ... case TABLE_MAP_EVENT: { Table_map_log_event *map= ((Table_map_log_event *)ev); if (shall_skip_database(map->get_db_name())) { ... } // WL36 size_t new_len= 0; const char* new_db= binlog_filter->get_rewrite_db( map->get_db_name(), &new_len); if (new_len && map->rewrite_db(new_db, new_len, glob_description_event)) { error("Could not rewrite database name"); goto err; } } case WRITE_ROWS_EVENT: case DELETE_ROWS_EVENT: case UPDATE_ROWS_EVENT: ... } ... } Comment. Rpl_filter::get_rewrite_db(db_from, &len): if filter contains a (db_from, db_to) pair, this function returns pointer to db_to and sets len = db_to length; otherwise, it returns db_from and does not change len value. 3. Supporting rewrite-db for SBR events --------------------------------------- Limited to emiting USE <db_to> instead of USE <db_from>. USE statements can be emited by mysqlbinlog as a result of processing the following events (see process_event() function): - Query_log_event - Load_log_event - Execute_load_query_log_event [ :public Query_log_event ] - Create_file_log_event [ :public Load_log_event ] TODO. Needed to check this list carefully (not sure for Create_file_log_event) Notes. - In replication, only Query_log_event and Load_log_event uses rpl_filter->get_rewrite_db(); - In mysqlbinlog (process_event), Execute_load_query_log_event and Create_file_log_event are processed in separate switch cases. And Load_log_event is processed in the default switch case. Conditions for emiting use-statement: - LOG_EVENT_SUPPRESS_USE_F is OFF for the event (e.g. it is ON for 'create database' statement) - event's db name differs from db_name in PRINT_EVENT_INFO (PRINT_EVENT_INFO keeps db name of the last issued USE statement; initially, this db name is empty). 3.1. In mysqlbinlog.cc - Add the following function: void print_use_stmt(Log_event* event, PRINT_EVENT_INFO* pinfo) { if (event->flags & LOG_EVENT_SUPPRESS_USE_F) return; /* - For events listed above get db_from = event->db; - If db_from is the same as pinfo->db then return; - If there is rewrite-db rule db_from->db_to, set db = db_to. Else set db = db_from; - Print "use <db>" to mysqlbinlog output - Set pinfo->db = db_from (this suppresses emiting use-statements by corresponding log_event's print-function) */ } - In process_event() function add switch case for Load_log_event and add print_use_stmt() invocations where needed (according to the events list above), e.g.: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, Log_event *ev, ...) { ... switch (ev_type) { case QUERY_EVENT: if (shall_skip_database(((Query_log_event*)ev)->db)) goto end; if (opt_base64_output_mode == BASE64_OUTPUT_ALWAYS) { // Possibly in case of rewite-db rule for ev->db // a warning should be emited here (see note below) ... write_event_header_and_base64(ev, ...) ... } else { print_use_stmt((Query_log_event*)ev, print_event_info); ev->print(result_file, print_event_info); } break; ... case LOAD_EVENT: print_use_stmt((Load_log_event*)ev, print_event_info); break; default: ... } ... } Note. write_event_header_and_base64() does not print use-statement. It produces BINLOG statement using ev->temp_buf content (i.e. the binary log representation of the event). We don't rewrite temp_buf here with db_to name (as we do it for Table map event) - this implies the limitation 3 mentioned above. Question: Is supporting of rewite_db + --base64-output really needed currently? 4. Current status ----------------- The outlined design (implemented for mysql-5.1.37) is tested for simple test-cases. TODO. 1. Check list of events which can emit use-statement. 2. Supporting of rewite_db + --base64-output ? ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): Add a mysqlbinlog option to change the used database (36)
by worklog-noreply＠askmonty.org 03 Oct '09

03 Oct '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to change the used database CREATION DATE..: Fri, 07 Aug 2009, 14:57 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 36 (http://askmonty.org/worklog/?tid=36) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Knielsen - Sat, 03 Oct 2009, 08:34)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.5353 2009-10-03 08:34:43.000000000 +0300 +++ /tmp/wklog.36.new.5353 2009-10-03 08:34:43.000000000 +0300 @@ -1 +1,77 @@ -G9m7yq <a href="http://ijfmyyjtveuu.com/">ijfmyyjtveuu</a>, [url=http://jeczeaqoxbpt.com/]jeczeaqoxbpt[/url] [link=http://nrisgrrvcrkm.com/]nrisgrrvcrkm[/link] http://edmnozsmotmt.com/ +Context +------- +(See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global +overview) +At the moment, the server has a replication slave option + + --replicate-rewrite-db="from->to" + +the option affects +- Table_map_log_event (all RBR events) +- Load_log_event (LOAD DATA) +- Query_log_event (SBR-based updates, with the usual assumption that the + statement refers to tables in current database, so that changing the current + database will make the statement to work on a table in a different database). + +See also MySQL BUG#42941. Note this bug is fixed in MySQL 5.1.37, which is not +merged into MariaDB at the time of writing, but planned to be merged before +release. + +What we could do +---------------- + +Option1: make mysqlbinlog accept --replicate-rewrite-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make mysqlbinlog accept --replicate-rewrite-db options and process them to the +same extent as replication slave would process --replicate-rewrite-db option. + + +Option2: Add database-agnostic RBR events and --strip-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Right now RBR events require a databasename. It is not possible to have RBR +event stream that won't mention which database the events are for. When I +tried to use debugger and specify empty database name, attempt to apply the +binlog resulted in this error: + +090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on +opening tables, + +We could do as follows: +- Make the server interpret empty database name in RBR event (i.e. in a + Table_map_log_event) as "use current database". Binlog slave thread + probably should not allow such events as it doesn't have a natural current + database. +- Add a mysqlbinlog --strip-db option that would + = not produce any "USE dbname" statements + = change databasename for all RBR events to be empty + +That way, mysqlbinlog output will be database-agnostic and apply to the +current database. +(this will have the usual limitations that we assume that all statements in +the binlog refer to the current database). + +Option3: Enhance database rewrite +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +If there is a need to support database change for statements that use +dbname.tablename notation and are replicated as statements (i.e. are DDL +statements and/or DML statements that are binlogged as statements), +then that could be supported as follows: + +- Make the server's parser recognize special form of comments + + /* !database-alias(oldname,newname) */ + + and save the mapping somewhere + +- Put the hooks in table open and name resolution code to use the saved + mapping. + + +Once we've done the above, it will be easy to perform a complete, +no-compromise or restrictions database name change in binary log. + +It will be possible to do the rewrites either on the slave ( +--replicate-rewrite-db will work for all kinds of statements), or in +mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to +parse the statement). + -=-=(Guest - Sat, 03 Oct 2009, 02:27)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.23366 2009-10-03 02:27:45.000000000 +0300 +++ /tmp/wklog.36.new.23366 2009-10-03 02:27:45.000000000 +0300 @@ -1,77 +1 @@ -Context -------- -(See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global -overview) -At the moment, the server has a replication slave option - - --replicate-rewrite-db="from->to" - -the option affects -- Table_map_log_event (all RBR events) -- Load_log_event (LOAD DATA) -- Query_log_event (SBR-based updates, with the usual assumption that the - statement refers to tables in current database, so that changing the current - database will make the statement to work on a table in a different database). - -See also MySQL BUG#42941. Note this bug is fixed in MySQL 5.1.37, which is not -merged into MariaDB at the time of writing, but planned to be merged before -release. - -What we could do ----------------- - -Option1: make mysqlbinlog accept --replicate-rewrite-db option -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Make mysqlbinlog accept --replicate-rewrite-db options and process them to the -same extent as replication slave would process --replicate-rewrite-db option. - - -Option2: Add database-agnostic RBR events and --strip-db option -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Right now RBR events require a databasename. It is not possible to have RBR -event stream that won't mention which database the events are for. When I -tried to use debugger and specify empty database name, attempt to apply the -binlog resulted in this error: - -090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on -opening tables, - -We could do as follows: -- Make the server interpret empty database name in RBR event (i.e. in a - Table_map_log_event) as "use current database". Binlog slave thread - probably should not allow such events as it doesn't have a natural current - database. -- Add a mysqlbinlog --strip-db option that would - = not produce any "USE dbname" statements - = change databasename for all RBR events to be empty - -That way, mysqlbinlog output will be database-agnostic and apply to the -current database. -(this will have the usual limitations that we assume that all statements in -the binlog refer to the current database). - -Option3: Enhance database rewrite -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -If there is a need to support database change for statements that use -dbname.tablename notation and are replicated as statements (i.e. are DDL -statements and/or DML statements that are binlogged as statements), -then that could be supported as follows: - -- Make the server's parser recognize special form of comments - - /* !database-alias(oldname,newname) */ - - and save the mapping somewhere - -- Put the hooks in table open and name resolution code to use the saved - mapping. - - -Once we've done the above, it will be easy to perform a complete, -no-compromise or restrictions database name change in binary log. - -It will be possible to do the rewrites either on the slave ( ---replicate-rewrite-db will work for all kinds of statements), or in -mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to -parse the statement). - +G9m7yq <a href="http://ijfmyyjtveuu.com/">ijfmyyjtveuu</a>, [url=http://jeczeaqoxbpt.com/]jeczeaqoxbpt[/url] [link=http://nrisgrrvcrkm.com/]nrisgrrvcrkm[/link] http://edmnozsmotmt.com/ -=-=(Guest - Tue, 15 Sep 2009, 18:04)=-=- Low Level Design modified. --- /tmp/wklog.36.old.19322 2009-09-15 18:04:49.000000000 +0300 +++ /tmp/wklog.36.new.19322 2009-09-15 18:04:49.000000000 +0300 @@ -191,7 +191,7 @@ - In process_event() function add switch case for Load_log_event and add print_use_stmt() invocations where needed (according to the - events lis above), e.g.: + events list above), e.g.: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, -=-=(Guest - Tue, 15 Sep 2009, 15:53)=-=- Low Level Design modified. --- /tmp/wklog.36.old.13421 2009-09-15 15:53:31.000000000 +0300 +++ /tmp/wklog.36.new.13421 2009-09-15 15:53:31.000000000 +0300 @@ -150,10 +150,17 @@ following events (see process_event() function): - Query_log_event -- Execute_load_query_log_event -- Create_file_log_event - -TODO. Needed to check this list requires carefully !!! +- Load_log_event +- Execute_load_query_log_event [ :public Query_log_event ] +- Create_file_log_event [ :public Load_log_event ] + +TODO. Needed to check this list carefully (not sure for Create_file_log_event) + Notes. + - In replication, only Query_log_event and Load_log_event uses + rpl_filter->get_rewrite_db(); + - In mysqlbinlog (process_event), Execute_load_query_log_event + and Create_file_log_event are processed in separate switch + cases. And Load_log_event is processed in the default switch case. Conditions for emiting use-statement: - LOG_EVENT_SUPPRESS_USE_F is OFF for the event @@ -182,8 +189,9 @@ */ } -- In process_event() function add print_use_stmt() invocations where - needed (according to the events lis above), e.g.: +- In process_event() function add switch case for Load_log_event and + add print_use_stmt() invocations where needed (according to the + events lis above), e.g.: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, @@ -207,6 +215,11 @@ } break; ... + case LOAD_EVENT: + print_use_stmt((Load_log_event*)ev, print_event_info); + break; + default: + ... } ... } -=-=(Guest - Tue, 15 Sep 2009, 12:12)=-=- Low Level Design modified. --- /tmp/wklog.36.old.3961 2009-09-15 12:12:26.000000000 +0300 +++ /tmp/wklog.36.new.3961 2009-09-15 12:12:26.000000000 +0300 @@ -144,6 +144,8 @@ 3. Supporting rewrite-db for SBR events --------------------------------------- +Limited to emiting USE <db_to> instead of USE <db_from>. + USE statements can be emited by mysqlbinlog as a result of processing the following events (see process_event() function): -=-=(Guest - Tue, 15 Sep 2009, 12:08)=-=- Low Level Design modified. --- /tmp/wklog.36.old.3794 2009-09-15 12:08:54.000000000 +0300 +++ /tmp/wklog.36.new.3794 2009-09-15 12:08:54.000000000 +0300 @@ -1 +1,229 @@ +Content +------- +1. Adding rewrite-db option +2. Supporting rewrite-db option for RBR events +3. Supporting rewrite-db option for SBR events + (Limited to affecting only USE statements) +4. Current status + +1. Adding rewrite-db option +--------------------------- + +1.1. Syntax: + --rewrite-db='db_from->db_to' + +1.2. Add 'OPT_REWRITE_DB' to 'options_client' (in client_priv.h). + +1.3. In mysqlbinlog.cc: + +- Add { "rewrite-db", OPT_REWRITE_DB, ...} record to my_long_options: +- Add Rpl_filter object to mysqlbinlog.cc + + Rpl_filter* binlog_filter; + +- Add corresponding switch case to get_one_option(): + + case OPT_REWRITE_DB: + <extract db-from and db-to strings> + binlog_filter->add_db_rewrite(db_from, db_to); + break; + . +Note. To make Rpl_filter usable in a MYSQL_CLIENT context, few small +additional changes are required: + +- In sql_list.cc/h, Sql_alloc::new(size_t) and Sql_alloc::new[](size_t) + uses sql_alloc() which is THD dependent. These are to be modified + as follows: + + #ifdef MYSQL_CLIENT + extern MEM_ROOT sql_list_client_mem_root; // defined in sql_list.cc + #endif + + class Sql_alloc + { ... + static void *operator new(size_t size) throw () + { + #ifndef MYSQL_CLIENT + return sql_alloc(size); + #else + return alloc_root(&sql_list_client_mem_root, size); + #endif + } + static void *operator new[](size_t size) throw () + { + #ifndef MYSQL_CLIENT + return sql_alloc(size); + #else + return alloc_root(&sql_list_client_mem_root, size); + #endif + } + ... + } + +- In rpl_filter.cc: + + Rpl_filter::Rpl_filter() : + ... + { + #ifdef MYSQL_CLIENT + init_alloc_root(&sql_list_client_mem_root, ...); + #endif + ... + } + + Rpl_filter::~Rpl_filter() + { ... + #ifdef MYSQL_CLIENT + free_root(&sql_list_client_mem_root, ...); + #endif + } + +2. Supporting rewrite-db for RBR events +--------------------------------------- + +In binlog, each row operation event is preceded by Table map event(s) which maps +table id(s) to database and table names. So, it's enough to support rewriting +database name in a Table map. + +2.1. Add rewrite_db() member to Table_map_log_event: + + int Table_map_log_event::rewrite_db( + const char* new_db, + size_t new_db_len, + const Format_description_log_event* desc) + { + /* 1. In temp_buf member (possibly reallocating it) rewrite + event length, db length, and db parts + 2. Change m_dblen and m_dbnam members + */ + } + +Comment. This function assumes that temp_buf member contains Table map +binlog representaion (temp_buf is used for creating corresponding +BINLOG statement). + +2.2. In mysqlbinlog modify corresponding switch case in the +process_event() function: + + Exit_status process_event( + PRINT_EVENT_INFO *print_event_info, + Log_event *ev, ...) + { + ... + switch (ev_type) { + ... + case TABLE_MAP_EVENT: + { + Table_map_log_event *map= ((Table_map_log_event *)ev); + if (shall_skip_database(map->get_db_name())) + { ... + } + // WL36 + size_t new_len= 0; + const char* new_db= binlog_filter->get_rewrite_db( + map->get_db_name(), &new_len); + if (new_len && map->rewrite_db(new_db, new_len, + glob_description_event)) + { error("Could not rewrite database name"); + goto err; + } + } + case WRITE_ROWS_EVENT: + case DELETE_ROWS_EVENT: + case UPDATE_ROWS_EVENT: + ... + } + ... + } + +Comment. Rpl_filter::get_rewrite_db(db_from, &len): if filter contains +a (db_from, db_to) pair, this function returns pointer to db_to and +sets len = db_to length; otherwise, it returns db_from and does not +change len value. + +3. Supporting rewrite-db for SBR events +--------------------------------------- + +USE statements can be emited by mysqlbinlog as a result of processing the +following events (see process_event() function): + +- Query_log_event +- Execute_load_query_log_event +- Create_file_log_event + +TODO. Needed to check this list requires carefully !!! + +Conditions for emiting use-statement: +- LOG_EVENT_SUPPRESS_USE_F is OFF for the event + (e.g. it is ON for 'create database' statement) +- event's db name differs from db_name in PRINT_EVENT_INFO + (PRINT_EVENT_INFO keeps db name of the last issued USE statement; + initially, this db name is empty). + +3.1. In mysqlbinlog.cc + +- Add the following function: + + void print_use_stmt(Log_event* event, PRINT_EVENT_INFO* pinfo) + { + if (event->flags & LOG_EVENT_SUPPRESS_USE_F) + return; + /* + - For events listed above get db_from = event->db; + - If db_from is the same as pinfo->db then return; + - If there is rewrite-db rule db_from->db_to, + set db = db_to. Else set db = db_from; + - Print "use <db>" to mysqlbinlog output + - Set pinfo->db = db_from + (this suppresses emiting use-statements by corresponding + log_event's print-function) + */ + } + +- In process_event() function add print_use_stmt() invocations where + needed (according to the events lis above), e.g.: + + Exit_status process_event( + PRINT_EVENT_INFO *print_event_info, + Log_event *ev, ...) + { + ... + switch (ev_type) { + case QUERY_EVENT: + if (shall_skip_database(((Query_log_event*)ev)->db)) + goto end; + if (opt_base64_output_mode == BASE64_OUTPUT_ALWAYS) + { + // Possibly in case of rewite-db rule for ev->db + // a warning should be emited here (see note below) + ... write_event_header_and_base64(ev, ...) ... + } + else + { + print_use_stmt((Query_log_event*)ev, print_event_info); + ev->print(result_file, print_event_info); + } + break; + ... + } + ... + } + +Note. write_event_header_and_base64() does not print use-statement. It +produces BINLOG statement using ev->temp_buf content (i.e. the binary +log representation of the event). We don't rewrite temp_buf here with +db_to name (as we do it for Table map event) - this implies the +limitation 3 mentioned above. +Question: Is supporting of rewite_db + --base64-output really needed +currently? + +4. Current status +----------------- + +The outlined design (implemented for mysql-5.1.37) is tested for +simple test-cases. + +TODO. 1. Check list of events which can emit use-statement. + 2. Supporting of rewite_db + --base64-output ? + -=-=(Guest - Mon, 14 Sep 2009, 11:51)=-=- Low Level Design modified. --- /tmp/wklog.36.old.9711 2009-09-14 11:51:43.000000000 +0300 +++ /tmp/wklog.36.new.9711 2009-09-14 11:51:43.000000000 +0300 @@ -1 +1 @@ -Pay no attention: just check for having access + -=-=(Guest - Mon, 14 Sep 2009, 11:51)=-=- Low Level Design modified. --- /tmp/wklog.36.old.9678 2009-09-14 11:51:28.000000000 +0300 +++ /tmp/wklog.36.new.9678 2009-09-14 11:51:28.000000000 +0300 @@ -1 +1 @@ - +Pay no attention: just check for having access -=-=(Knielsen - Mon, 17 Aug 2009, 12:44)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.7834 2009-08-17 12:44:17.000000000 +0300 +++ /tmp/wklog.36.new.7834 2009-08-17 12:44:17.000000000 +0300 @@ -13,7 +13,9 @@ statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). -See also MySQL BUG#42941. +See also MySQL BUG#42941. Note this bug is fixed in MySQL 5.1.37, which is not +merged into MariaDB at the time of writing, but planned to be merged before +release. What we could do ---------------- -=-=(Guest - Sun, 16 Aug 2009, 17:11)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.27162 2009-08-16 17:11:12.000000000 +0300 +++ /tmp/wklog.36.new.27162 2009-08-16 17:11:12.000000000 +0300 @@ -13,6 +13,8 @@ statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). +See also MySQL BUG#42941. + What we could do ---------------- ------------------------------------------------------------ -=-=(View All Progress Notes, 15 total)=-=- http://askmonty.org/worklog/index.pl?tid=36&nolimit=1 DESCRIPTION: Sometimes there is a need to take a binary log and apply it to a database with a different name than the original name of the database on binlog producer. If one is using statement-based replication, he can achieve this by grepping out "USE dbname" statements out of the output of mysqlbinlog(*). With row-based replication this is no longer possible, as database name is encoded within the the BINLOG '....' statement. This task is about adding an option to mysqlbinlog that would allow to change the names of used databases in both RBR and SBR events. (*) this implies that all statements refer to tables in the current database, doesn't catch updates made inside stored functions and so forth, but still works for a practially-important subset of cases. HIGH-LEVEL SPECIFICATION: Context ------- (See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global overview) At the moment, the server has a replication slave option --replicate-rewrite-db="from->to" the option affects - Table_map_log_event (all RBR events) - Load_log_event (LOAD DATA) - Query_log_event (SBR-based updates, with the usual assumption that the statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). See also MySQL BUG#42941. Note this bug is fixed in MySQL 5.1.37, which is not merged into MariaDB at the time of writing, but planned to be merged before release. What we could do ---------------- Option1: make mysqlbinlog accept --replicate-rewrite-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Make mysqlbinlog accept --replicate-rewrite-db options and process them to the same extent as replication slave would process --replicate-rewrite-db option. Option2: Add database-agnostic RBR events and --strip-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Right now RBR events require a databasename. It is not possible to have RBR event stream that won't mention which database the events are for. When I tried to use debugger and specify empty database name, attempt to apply the binlog resulted in this error: 090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on opening tables, We could do as follows: - Make the server interpret empty database name in RBR event (i.e. in a Table_map_log_event) as "use current database". Binlog slave thread probably should not allow such events as it doesn't have a natural current database. - Add a mysqlbinlog --strip-db option that would = not produce any "USE dbname" statements = change databasename for all RBR events to be empty That way, mysqlbinlog output will be database-agnostic and apply to the current database. (this will have the usual limitations that we assume that all statements in the binlog refer to the current database). Option3: Enhance database rewrite ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If there is a need to support database change for statements that use dbname.tablename notation and are replicated as statements (i.e. are DDL statements and/or DML statements that are binlogged as statements), then that could be supported as follows: - Make the server's parser recognize special form of comments /* !database-alias(oldname,newname) */ and save the mapping somewhere - Put the hooks in table open and name resolution code to use the saved mapping. Once we've done the above, it will be easy to perform a complete, no-compromise or restrictions database name change in binary log. It will be possible to do the rewrites either on the slave ( --replicate-rewrite-db will work for all kinds of statements), or in mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to parse the statement). LOW-LEVEL DESIGN: Content ------- 1. Adding rewrite-db option 2. Supporting rewrite-db option for RBR events 3. Supporting rewrite-db option for SBR events (Limited to affecting only USE statements) 4. Current status 1. Adding rewrite-db option --------------------------- 1.1. Syntax: --rewrite-db='db_from->db_to' 1.2. Add 'OPT_REWRITE_DB' to 'options_client' (in client_priv.h). 1.3. In mysqlbinlog.cc: - Add { "rewrite-db", OPT_REWRITE_DB, ...} record to my_long_options: - Add Rpl_filter object to mysqlbinlog.cc Rpl_filter* binlog_filter; - Add corresponding switch case to get_one_option(): case OPT_REWRITE_DB: <extract db-from and db-to strings> binlog_filter->add_db_rewrite(db_from, db_to); break; . Note. To make Rpl_filter usable in a MYSQL_CLIENT context, few small additional changes are required: - In sql_list.cc/h, Sql_alloc::new(size_t) and Sql_alloc::new[](size_t) uses sql_alloc() which is THD dependent. These are to be modified as follows: #ifdef MYSQL_CLIENT extern MEM_ROOT sql_list_client_mem_root; // defined in sql_list.cc #endif class Sql_alloc { ... static void *operator new(size_t size) throw () { #ifndef MYSQL_CLIENT return sql_alloc(size); #else return alloc_root(&sql_list_client_mem_root, size); #endif } static void *operator new[](size_t size) throw () { #ifndef MYSQL_CLIENT return sql_alloc(size); #else return alloc_root(&sql_list_client_mem_root, size); #endif } ... } - In rpl_filter.cc: Rpl_filter::Rpl_filter() : ... { #ifdef MYSQL_CLIENT init_alloc_root(&sql_list_client_mem_root, ...); #endif ... } Rpl_filter::~Rpl_filter() { ... #ifdef MYSQL_CLIENT free_root(&sql_list_client_mem_root, ...); #endif } 2. Supporting rewrite-db for RBR events --------------------------------------- In binlog, each row operation event is preceded by Table map event(s) which maps table id(s) to database and table names. So, it's enough to support rewriting database name in a Table map. 2.1. Add rewrite_db() member to Table_map_log_event: int Table_map_log_event::rewrite_db( const char* new_db, size_t new_db_len, const Format_description_log_event* desc) { /* 1. In temp_buf member (possibly reallocating it) rewrite event length, db length, and db parts 2. Change m_dblen and m_dbnam members */ } Comment. This function assumes that temp_buf member contains Table map binlog representaion (temp_buf is used for creating corresponding BINLOG statement). 2.2. In mysqlbinlog modify corresponding switch case in the process_event() function: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, Log_event *ev, ...) { ... switch (ev_type) { ... case TABLE_MAP_EVENT: { Table_map_log_event *map= ((Table_map_log_event *)ev); if (shall_skip_database(map->get_db_name())) { ... } // WL36 size_t new_len= 0; const char* new_db= binlog_filter->get_rewrite_db( map->get_db_name(), &new_len); if (new_len && map->rewrite_db(new_db, new_len, glob_description_event)) { error("Could not rewrite database name"); goto err; } } case WRITE_ROWS_EVENT: case DELETE_ROWS_EVENT: case UPDATE_ROWS_EVENT: ... } ... } Comment. Rpl_filter::get_rewrite_db(db_from, &len): if filter contains a (db_from, db_to) pair, this function returns pointer to db_to and sets len = db_to length; otherwise, it returns db_from and does not change len value. 3. Supporting rewrite-db for SBR events --------------------------------------- Limited to emiting USE <db_to> instead of USE <db_from>. USE statements can be emited by mysqlbinlog as a result of processing the following events (see process_event() function): - Query_log_event - Load_log_event - Execute_load_query_log_event [ :public Query_log_event ] - Create_file_log_event [ :public Load_log_event ] TODO. Needed to check this list carefully (not sure for Create_file_log_event) Notes. - In replication, only Query_log_event and Load_log_event uses rpl_filter->get_rewrite_db(); - In mysqlbinlog (process_event), Execute_load_query_log_event and Create_file_log_event are processed in separate switch cases. And Load_log_event is processed in the default switch case. Conditions for emiting use-statement: - LOG_EVENT_SUPPRESS_USE_F is OFF for the event (e.g. it is ON for 'create database' statement) - event's db name differs from db_name in PRINT_EVENT_INFO (PRINT_EVENT_INFO keeps db name of the last issued USE statement; initially, this db name is empty). 3.1. In mysqlbinlog.cc - Add the following function: void print_use_stmt(Log_event* event, PRINT_EVENT_INFO* pinfo) { if (event->flags & LOG_EVENT_SUPPRESS_USE_F) return; /* - For events listed above get db_from = event->db; - If db_from is the same as pinfo->db then return; - If there is rewrite-db rule db_from->db_to, set db = db_to. Else set db = db_from; - Print "use <db>" to mysqlbinlog output - Set pinfo->db = db_from (this suppresses emiting use-statements by corresponding log_event's print-function) */ } - In process_event() function add switch case for Load_log_event and add print_use_stmt() invocations where needed (according to the events list above), e.g.: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, Log_event *ev, ...) { ... switch (ev_type) { case QUERY_EVENT: if (shall_skip_database(((Query_log_event*)ev)->db)) goto end; if (opt_base64_output_mode == BASE64_OUTPUT_ALWAYS) { // Possibly in case of rewite-db rule for ev->db // a warning should be emited here (see note below) ... write_event_header_and_base64(ev, ...) ... } else { print_use_stmt((Query_log_event*)ev, print_event_info); ev->print(result_file, print_event_info); } break; ... case LOAD_EVENT: print_use_stmt((Load_log_event*)ev, print_event_info); break; default: ... } ... } Note. write_event_header_and_base64() does not print use-statement. It produces BINLOG statement using ev->temp_buf content (i.e. the binary log representation of the event). We don't rewrite temp_buf here with db_to name (as we do it for Table map event) - this implies the limitation 3 mentioned above. Question: Is supporting of rewite_db + --base64-output really needed currently? 4. Current status ----------------- The outlined design (implemented for mysql-5.1.37) is tested for simple test-cases. TODO. 1. Check list of events which can emit use-statement. 2. Supporting of rewite_db + --base64-output ? ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Guest): Add a mysqlbinlog option to change the used database (36)
by worklog-noreply＠askmonty.org 03 Oct '09

03 Oct '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to change the used database CREATION DATE..: Fri, 07 Aug 2009, 14:57 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 36 (http://askmonty.org/worklog/?tid=36) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Guest - Sat, 03 Oct 2009, 02:27)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.23366 2009-10-03 02:27:45.000000000 +0300 +++ /tmp/wklog.36.new.23366 2009-10-03 02:27:45.000000000 +0300 @@ -1,77 +1 @@ -Context -------- -(See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global -overview) -At the moment, the server has a replication slave option - - --replicate-rewrite-db="from->to" - -the option affects -- Table_map_log_event (all RBR events) -- Load_log_event (LOAD DATA) -- Query_log_event (SBR-based updates, with the usual assumption that the - statement refers to tables in current database, so that changing the current - database will make the statement to work on a table in a different database). - -See also MySQL BUG#42941. Note this bug is fixed in MySQL 5.1.37, which is not -merged into MariaDB at the time of writing, but planned to be merged before -release. - -What we could do ----------------- - -Option1: make mysqlbinlog accept --replicate-rewrite-db option -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Make mysqlbinlog accept --replicate-rewrite-db options and process them to the -same extent as replication slave would process --replicate-rewrite-db option. - - -Option2: Add database-agnostic RBR events and --strip-db option -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Right now RBR events require a databasename. It is not possible to have RBR -event stream that won't mention which database the events are for. When I -tried to use debugger and specify empty database name, attempt to apply the -binlog resulted in this error: - -090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on -opening tables, - -We could do as follows: -- Make the server interpret empty database name in RBR event (i.e. in a - Table_map_log_event) as "use current database". Binlog slave thread - probably should not allow such events as it doesn't have a natural current - database. -- Add a mysqlbinlog --strip-db option that would - = not produce any "USE dbname" statements - = change databasename for all RBR events to be empty - -That way, mysqlbinlog output will be database-agnostic and apply to the -current database. -(this will have the usual limitations that we assume that all statements in -the binlog refer to the current database). - -Option3: Enhance database rewrite -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -If there is a need to support database change for statements that use -dbname.tablename notation and are replicated as statements (i.e. are DDL -statements and/or DML statements that are binlogged as statements), -then that could be supported as follows: - -- Make the server's parser recognize special form of comments - - /* !database-alias(oldname,newname) */ - - and save the mapping somewhere - -- Put the hooks in table open and name resolution code to use the saved - mapping. - - -Once we've done the above, it will be easy to perform a complete, -no-compromise or restrictions database name change in binary log. - -It will be possible to do the rewrites either on the slave ( ---replicate-rewrite-db will work for all kinds of statements), or in -mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to -parse the statement). - +G9m7yq <a href="http://ijfmyyjtveuu.com/">ijfmyyjtveuu</a>, [url=http://jeczeaqoxbpt.com/]jeczeaqoxbpt[/url] [link=http://nrisgrrvcrkm.com/]nrisgrrvcrkm[/link] http://edmnozsmotmt.com/ -=-=(Guest - Tue, 15 Sep 2009, 18:04)=-=- Low Level Design modified. --- /tmp/wklog.36.old.19322 2009-09-15 18:04:49.000000000 +0300 +++ /tmp/wklog.36.new.19322 2009-09-15 18:04:49.000000000 +0300 @@ -191,7 +191,7 @@ - In process_event() function add switch case for Load_log_event and add print_use_stmt() invocations where needed (according to the - events lis above), e.g.: + events list above), e.g.: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, -=-=(Guest - Tue, 15 Sep 2009, 15:53)=-=- Low Level Design modified. --- /tmp/wklog.36.old.13421 2009-09-15 15:53:31.000000000 +0300 +++ /tmp/wklog.36.new.13421 2009-09-15 15:53:31.000000000 +0300 @@ -150,10 +150,17 @@ following events (see process_event() function): - Query_log_event -- Execute_load_query_log_event -- Create_file_log_event - -TODO. Needed to check this list requires carefully !!! +- Load_log_event +- Execute_load_query_log_event [ :public Query_log_event ] +- Create_file_log_event [ :public Load_log_event ] + +TODO. Needed to check this list carefully (not sure for Create_file_log_event) + Notes. + - In replication, only Query_log_event and Load_log_event uses + rpl_filter->get_rewrite_db(); + - In mysqlbinlog (process_event), Execute_load_query_log_event + and Create_file_log_event are processed in separate switch + cases. And Load_log_event is processed in the default switch case. Conditions for emiting use-statement: - LOG_EVENT_SUPPRESS_USE_F is OFF for the event @@ -182,8 +189,9 @@ */ } -- In process_event() function add print_use_stmt() invocations where - needed (according to the events lis above), e.g.: +- In process_event() function add switch case for Load_log_event and + add print_use_stmt() invocations where needed (according to the + events lis above), e.g.: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, @@ -207,6 +215,11 @@ } break; ... + case LOAD_EVENT: + print_use_stmt((Load_log_event*)ev, print_event_info); + break; + default: + ... } ... } -=-=(Guest - Tue, 15 Sep 2009, 12:12)=-=- Low Level Design modified. --- /tmp/wklog.36.old.3961 2009-09-15 12:12:26.000000000 +0300 +++ /tmp/wklog.36.new.3961 2009-09-15 12:12:26.000000000 +0300 @@ -144,6 +144,8 @@ 3. Supporting rewrite-db for SBR events --------------------------------------- +Limited to emiting USE <db_to> instead of USE <db_from>. + USE statements can be emited by mysqlbinlog as a result of processing the following events (see process_event() function): -=-=(Guest - Tue, 15 Sep 2009, 12:08)=-=- Low Level Design modified. --- /tmp/wklog.36.old.3794 2009-09-15 12:08:54.000000000 +0300 +++ /tmp/wklog.36.new.3794 2009-09-15 12:08:54.000000000 +0300 @@ -1 +1,229 @@ +Content +------- +1. Adding rewrite-db option +2. Supporting rewrite-db option for RBR events +3. Supporting rewrite-db option for SBR events + (Limited to affecting only USE statements) +4. Current status + +1. Adding rewrite-db option +--------------------------- + +1.1. Syntax: + --rewrite-db='db_from->db_to' + +1.2. Add 'OPT_REWRITE_DB' to 'options_client' (in client_priv.h). + +1.3. In mysqlbinlog.cc: + +- Add { "rewrite-db", OPT_REWRITE_DB, ...} record to my_long_options: +- Add Rpl_filter object to mysqlbinlog.cc + + Rpl_filter* binlog_filter; + +- Add corresponding switch case to get_one_option(): + + case OPT_REWRITE_DB: + <extract db-from and db-to strings> + binlog_filter->add_db_rewrite(db_from, db_to); + break; + . +Note. To make Rpl_filter usable in a MYSQL_CLIENT context, few small +additional changes are required: + +- In sql_list.cc/h, Sql_alloc::new(size_t) and Sql_alloc::new[](size_t) + uses sql_alloc() which is THD dependent. These are to be modified + as follows: + + #ifdef MYSQL_CLIENT + extern MEM_ROOT sql_list_client_mem_root; // defined in sql_list.cc + #endif + + class Sql_alloc + { ... + static void *operator new(size_t size) throw () + { + #ifndef MYSQL_CLIENT + return sql_alloc(size); + #else + return alloc_root(&sql_list_client_mem_root, size); + #endif + } + static void *operator new[](size_t size) throw () + { + #ifndef MYSQL_CLIENT + return sql_alloc(size); + #else + return alloc_root(&sql_list_client_mem_root, size); + #endif + } + ... + } + +- In rpl_filter.cc: + + Rpl_filter::Rpl_filter() : + ... + { + #ifdef MYSQL_CLIENT + init_alloc_root(&sql_list_client_mem_root, ...); + #endif + ... + } + + Rpl_filter::~Rpl_filter() + { ... + #ifdef MYSQL_CLIENT + free_root(&sql_list_client_mem_root, ...); + #endif + } + +2. Supporting rewrite-db for RBR events +--------------------------------------- + +In binlog, each row operation event is preceded by Table map event(s) which maps +table id(s) to database and table names. So, it's enough to support rewriting +database name in a Table map. + +2.1. Add rewrite_db() member to Table_map_log_event: + + int Table_map_log_event::rewrite_db( + const char* new_db, + size_t new_db_len, + const Format_description_log_event* desc) + { + /* 1. In temp_buf member (possibly reallocating it) rewrite + event length, db length, and db parts + 2. Change m_dblen and m_dbnam members + */ + } + +Comment. This function assumes that temp_buf member contains Table map +binlog representaion (temp_buf is used for creating corresponding +BINLOG statement). + +2.2. In mysqlbinlog modify corresponding switch case in the +process_event() function: + + Exit_status process_event( + PRINT_EVENT_INFO *print_event_info, + Log_event *ev, ...) + { + ... + switch (ev_type) { + ... + case TABLE_MAP_EVENT: + { + Table_map_log_event *map= ((Table_map_log_event *)ev); + if (shall_skip_database(map->get_db_name())) + { ... + } + // WL36 + size_t new_len= 0; + const char* new_db= binlog_filter->get_rewrite_db( + map->get_db_name(), &new_len); + if (new_len && map->rewrite_db(new_db, new_len, + glob_description_event)) + { error("Could not rewrite database name"); + goto err; + } + } + case WRITE_ROWS_EVENT: + case DELETE_ROWS_EVENT: + case UPDATE_ROWS_EVENT: + ... + } + ... + } + +Comment. Rpl_filter::get_rewrite_db(db_from, &len): if filter contains +a (db_from, db_to) pair, this function returns pointer to db_to and +sets len = db_to length; otherwise, it returns db_from and does not +change len value. + +3. Supporting rewrite-db for SBR events +--------------------------------------- + +USE statements can be emited by mysqlbinlog as a result of processing the +following events (see process_event() function): + +- Query_log_event +- Execute_load_query_log_event +- Create_file_log_event + +TODO. Needed to check this list requires carefully !!! + +Conditions for emiting use-statement: +- LOG_EVENT_SUPPRESS_USE_F is OFF for the event + (e.g. it is ON for 'create database' statement) +- event's db name differs from db_name in PRINT_EVENT_INFO + (PRINT_EVENT_INFO keeps db name of the last issued USE statement; + initially, this db name is empty). + +3.1. In mysqlbinlog.cc + +- Add the following function: + + void print_use_stmt(Log_event* event, PRINT_EVENT_INFO* pinfo) + { + if (event->flags & LOG_EVENT_SUPPRESS_USE_F) + return; + /* + - For events listed above get db_from = event->db; + - If db_from is the same as pinfo->db then return; + - If there is rewrite-db rule db_from->db_to, + set db = db_to. Else set db = db_from; + - Print "use <db>" to mysqlbinlog output + - Set pinfo->db = db_from + (this suppresses emiting use-statements by corresponding + log_event's print-function) + */ + } + +- In process_event() function add print_use_stmt() invocations where + needed (according to the events lis above), e.g.: + + Exit_status process_event( + PRINT_EVENT_INFO *print_event_info, + Log_event *ev, ...) + { + ... + switch (ev_type) { + case QUERY_EVENT: + if (shall_skip_database(((Query_log_event*)ev)->db)) + goto end; + if (opt_base64_output_mode == BASE64_OUTPUT_ALWAYS) + { + // Possibly in case of rewite-db rule for ev->db + // a warning should be emited here (see note below) + ... write_event_header_and_base64(ev, ...) ... + } + else + { + print_use_stmt((Query_log_event*)ev, print_event_info); + ev->print(result_file, print_event_info); + } + break; + ... + } + ... + } + +Note. write_event_header_and_base64() does not print use-statement. It +produces BINLOG statement using ev->temp_buf content (i.e. the binary +log representation of the event). We don't rewrite temp_buf here with +db_to name (as we do it for Table map event) - this implies the +limitation 3 mentioned above. +Question: Is supporting of rewite_db + --base64-output really needed +currently? + +4. Current status +----------------- + +The outlined design (implemented for mysql-5.1.37) is tested for +simple test-cases. + +TODO. 1. Check list of events which can emit use-statement. + 2. Supporting of rewite_db + --base64-output ? + -=-=(Guest - Mon, 14 Sep 2009, 11:51)=-=- Low Level Design modified. --- /tmp/wklog.36.old.9711 2009-09-14 11:51:43.000000000 +0300 +++ /tmp/wklog.36.new.9711 2009-09-14 11:51:43.000000000 +0300 @@ -1 +1 @@ -Pay no attention: just check for having access + -=-=(Guest - Mon, 14 Sep 2009, 11:51)=-=- Low Level Design modified. --- /tmp/wklog.36.old.9678 2009-09-14 11:51:28.000000000 +0300 +++ /tmp/wklog.36.new.9678 2009-09-14 11:51:28.000000000 +0300 @@ -1 +1 @@ - +Pay no attention: just check for having access -=-=(Knielsen - Mon, 17 Aug 2009, 12:44)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.7834 2009-08-17 12:44:17.000000000 +0300 +++ /tmp/wklog.36.new.7834 2009-08-17 12:44:17.000000000 +0300 @@ -13,7 +13,9 @@ statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). -See also MySQL BUG#42941. +See also MySQL BUG#42941. Note this bug is fixed in MySQL 5.1.37, which is not +merged into MariaDB at the time of writing, but planned to be merged before +release. What we could do ---------------- -=-=(Guest - Sun, 16 Aug 2009, 17:11)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.27162 2009-08-16 17:11:12.000000000 +0300 +++ /tmp/wklog.36.new.27162 2009-08-16 17:11:12.000000000 +0300 @@ -13,6 +13,8 @@ statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). +See also MySQL BUG#42941. + What we could do ---------------- -=-=(Psergey - Mon, 10 Aug 2009, 15:41)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.13035 2009-08-10 15:41:51.000000000 +0300 +++ /tmp/wklog.36.new.13035 2009-08-10 15:41:51.000000000 +0300 @@ -1,5 +1,7 @@ Context ------- +(See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global +overview) At the moment, the server has a replication slave option --replicate-rewrite-db="from->to" ------------------------------------------------------------ -=-=(View All Progress Notes, 14 total)=-=- http://askmonty.org/worklog/index.pl?tid=36&nolimit=1 DESCRIPTION: Sometimes there is a need to take a binary log and apply it to a database with a different name than the original name of the database on binlog producer. If one is using statement-based replication, he can achieve this by grepping out "USE dbname" statements out of the output of mysqlbinlog(*). With row-based replication this is no longer possible, as database name is encoded within the the BINLOG '....' statement. This task is about adding an option to mysqlbinlog that would allow to change the names of used databases in both RBR and SBR events. (*) this implies that all statements refer to tables in the current database, doesn't catch updates made inside stored functions and so forth, but still works for a practially-important subset of cases. HIGH-LEVEL SPECIFICATION: G9m7yq <a href="http://ijfmyyjtveuu.com/">ijfmyyjtveuu</a>, [url=http://jeczeaqoxbpt.com/]jeczeaqoxbpt[/url] [link=http://nrisgrrvcrkm.com/]nrisgrrvcrkm[/link] http://edmnozsmotmt.com/ LOW-LEVEL DESIGN: Content ------- 1. Adding rewrite-db option 2. Supporting rewrite-db option for RBR events 3. Supporting rewrite-db option for SBR events (Limited to affecting only USE statements) 4. Current status 1. Adding rewrite-db option --------------------------- 1.1. Syntax: --rewrite-db='db_from->db_to' 1.2. Add 'OPT_REWRITE_DB' to 'options_client' (in client_priv.h). 1.3. In mysqlbinlog.cc: - Add { "rewrite-db", OPT_REWRITE_DB, ...} record to my_long_options: - Add Rpl_filter object to mysqlbinlog.cc Rpl_filter* binlog_filter; - Add corresponding switch case to get_one_option(): case OPT_REWRITE_DB: <extract db-from and db-to strings> binlog_filter->add_db_rewrite(db_from, db_to); break; . Note. To make Rpl_filter usable in a MYSQL_CLIENT context, few small additional changes are required: - In sql_list.cc/h, Sql_alloc::new(size_t) and Sql_alloc::new[](size_t) uses sql_alloc() which is THD dependent. These are to be modified as follows: #ifdef MYSQL_CLIENT extern MEM_ROOT sql_list_client_mem_root; // defined in sql_list.cc #endif class Sql_alloc { ... static void *operator new(size_t size) throw () { #ifndef MYSQL_CLIENT return sql_alloc(size); #else return alloc_root(&sql_list_client_mem_root, size); #endif } static void *operator new[](size_t size) throw () { #ifndef MYSQL_CLIENT return sql_alloc(size); #else return alloc_root(&sql_list_client_mem_root, size); #endif } ... } - In rpl_filter.cc: Rpl_filter::Rpl_filter() : ... { #ifdef MYSQL_CLIENT init_alloc_root(&sql_list_client_mem_root, ...); #endif ... } Rpl_filter::~Rpl_filter() { ... #ifdef MYSQL_CLIENT free_root(&sql_list_client_mem_root, ...); #endif } 2. Supporting rewrite-db for RBR events --------------------------------------- In binlog, each row operation event is preceded by Table map event(s) which maps table id(s) to database and table names. So, it's enough to support rewriting database name in a Table map. 2.1. Add rewrite_db() member to Table_map_log_event: int Table_map_log_event::rewrite_db( const char* new_db, size_t new_db_len, const Format_description_log_event* desc) { /* 1. In temp_buf member (possibly reallocating it) rewrite event length, db length, and db parts 2. Change m_dblen and m_dbnam members */ } Comment. This function assumes that temp_buf member contains Table map binlog representaion (temp_buf is used for creating corresponding BINLOG statement). 2.2. In mysqlbinlog modify corresponding switch case in the process_event() function: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, Log_event *ev, ...) { ... switch (ev_type) { ... case TABLE_MAP_EVENT: { Table_map_log_event *map= ((Table_map_log_event *)ev); if (shall_skip_database(map->get_db_name())) { ... } // WL36 size_t new_len= 0; const char* new_db= binlog_filter->get_rewrite_db( map->get_db_name(), &new_len); if (new_len && map->rewrite_db(new_db, new_len, glob_description_event)) { error("Could not rewrite database name"); goto err; } } case WRITE_ROWS_EVENT: case DELETE_ROWS_EVENT: case UPDATE_ROWS_EVENT: ... } ... } Comment. Rpl_filter::get_rewrite_db(db_from, &len): if filter contains a (db_from, db_to) pair, this function returns pointer to db_to and sets len = db_to length; otherwise, it returns db_from and does not change len value. 3. Supporting rewrite-db for SBR events --------------------------------------- Limited to emiting USE <db_to> instead of USE <db_from>. USE statements can be emited by mysqlbinlog as a result of processing the following events (see process_event() function): - Query_log_event - Load_log_event - Execute_load_query_log_event [ :public Query_log_event ] - Create_file_log_event [ :public Load_log_event ] TODO. Needed to check this list carefully (not sure for Create_file_log_event) Notes. - In replication, only Query_log_event and Load_log_event uses rpl_filter->get_rewrite_db(); - In mysqlbinlog (process_event), Execute_load_query_log_event and Create_file_log_event are processed in separate switch cases. And Load_log_event is processed in the default switch case. Conditions for emiting use-statement: - LOG_EVENT_SUPPRESS_USE_F is OFF for the event (e.g. it is ON for 'create database' statement) - event's db name differs from db_name in PRINT_EVENT_INFO (PRINT_EVENT_INFO keeps db name of the last issued USE statement; initially, this db name is empty). 3.1. In mysqlbinlog.cc - Add the following function: void print_use_stmt(Log_event* event, PRINT_EVENT_INFO* pinfo) { if (event->flags & LOG_EVENT_SUPPRESS_USE_F) return; /* - For events listed above get db_from = event->db; - If db_from is the same as pinfo->db then return; - If there is rewrite-db rule db_from->db_to, set db = db_to. Else set db = db_from; - Print "use <db>" to mysqlbinlog output - Set pinfo->db = db_from (this suppresses emiting use-statements by corresponding log_event's print-function) */ } - In process_event() function add switch case for Load_log_event and add print_use_stmt() invocations where needed (according to the events list above), e.g.: Exit_status process_event( PRINT_EVENT_INFO *print_event_info, Log_event *ev, ...) { ... switch (ev_type) { case QUERY_EVENT: if (shall_skip_database(((Query_log_event*)ev)->db)) goto end; if (opt_base64_output_mode == BASE64_OUTPUT_ALWAYS) { // Possibly in case of rewite-db rule for ev->db // a warning should be emited here (see note below) ... write_event_header_and_base64(ev, ...) ... } else { print_use_stmt((Query_log_event*)ev, print_event_info); ev->print(result_file, print_event_info); } break; ... case LOAD_EVENT: print_use_stmt((Load_log_event*)ev, print_event_info); break; default: ... } ... } Note. write_event_header_and_base64() does not print use-statement. It produces BINLOG statement using ev->temp_buf content (i.e. the binary log representation of the event). We don't rewrite temp_buf here with db_to name (as we do it for Table map event) - this implies the limitation 3 mentioned above. Question: Is supporting of rewite_db + --base64-output really needed currently? 4. Current status ----------------- The outlined design (implemented for mysql-5.1.37) is tested for simple test-cases. TODO. 1. Check list of events which can emit use-statement. 2. Supporting of rewite_db + --base64-output ? ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2748)
by knielsen＠knielsen-hq.org 02 Oct '09

02 Oct '09

#At lp:maria 2748 knielsen(a)knielsen-hq.org 2009-10-02 [merge] Merge Monty's fixes from main into release branch. - MariaDB naming. - Error reporting with mysqld --help --verbose. modified: client/mysql.cc configure.in include/mysql.h include/mysql.h.pp libmysql/libmysql.c mysql-test/r/mysqld_option_err.result mysql-test/t/mysqld_option_err.test sql/mysqld.cc === modified file 'client/mysql.cc' --- a/client/mysql.cc 2009-09-07 20:50:10 +0000 +++ b/client/mysql.cc 2009-10-02 11:56:11 +0000 @@ -43,7 +43,7 @@ #include <locale.h> #endif -const char *VER= "14.15"; +const char *VER= "14.16"; /* Don't try to make a nice table if the data is too big */ #define MAX_COLUMN_LENGTH 1024 @@ -1076,7 +1076,7 @@ int main(int argc,char *argv[]) delimiter_str= delimiter; default_prompt = my_strdup(getenv("MYSQL_PS1") ? getenv("MYSQL_PS1") : - "mysql> ",MYF(MY_WME)); + "\\N [\\d]> ",MYF(MY_WME)); current_prompt = my_strdup(default_prompt,MYF(MY_WME)); prompt_counter=0; @@ -1156,10 +1156,11 @@ int main(int argc,char *argv[]) signal(SIGINT, handle_sigint); // Catch SIGINT to clean up signal(SIGQUIT, mysql_end); // Catch SIGQUIT to clean up - put_info("Welcome to the MySQL monitor. Commands end with ; or \\g.", + put_info("Welcome to the MariaDB monitor. Commands end with ; or \\g.", INFO_INFO); sprintf((char*) glob_buffer.ptr(), - "Your MySQL connection id is %lu\nServer version: %s\n", + "Your %s connection id is %lu\nServer version: %s\n", + mysql_get_server_name(&mysql), mysql_thread_id(&mysql), server_version_string(&mysql)); put_info((char*) glob_buffer.ptr(),INFO_INFO); @@ -4369,6 +4370,7 @@ com_status(String *buffer __attribute__( tee_fprintf(stdout, "Using outfile:\t\t'%s'\n", opt_outfile ? outfile : ""); #endif tee_fprintf(stdout, "Using delimiter:\t%s\n", delimiter); + tee_fprintf(stdout, "Server:\t\t\t%s\n", mysql_get_server_name(&mysql)); tee_fprintf(stdout, "Server version:\t\t%s\n", server_version_string(&mysql)); tee_fprintf(stdout, "Protocol version:\t%d\n", mysql_get_proto_info(&mysql)); tee_fprintf(stdout, "Connection:\t\t%s\n", mysql_get_host_info(&mysql)); @@ -4700,7 +4702,7 @@ static void mysql_end_timer(ulong start_ strmov(strend(buff),")"); } -static const char* construct_prompt() +static const char *construct_prompt() { processed_prompt.free(); // Erase the old prompt time_t lclock = time(NULL); // Get the date struct @@ -4729,6 +4731,12 @@ static const char* construct_prompt() case 'd': processed_prompt.append(current_db ? current_db : "(none)"); break; + case 'N': + if (connected) + processed_prompt.append(mysql_get_server_name(&mysql)); + else + processed_prompt.append("unknown"); + break; case 'h': { const char *prompt; === modified file 'configure.in' --- a/configure.in 2009-09-29 22:50:10 +0000 +++ b/configure.in 2009-10-02 12:55:16 +0000 @@ -13,7 +13,7 @@ AC_CANONICAL_SYSTEM # # When merging new MySQL releases, update the version number to match the # MySQL version number, but reset the maria subrelease (-beta1). -AM_INIT_AUTOMAKE(mysql, 5.1.38-maria-beta1) +AM_INIT_AUTOMAKE(mysql, 5.1.38-MariaDB-beta1) AM_CONFIG_HEADER([include/config.h:config.h.in]) PROTOCOL_VERSION=10 === modified file 'include/mysql.h' --- a/include/mysql.h 2007-11-26 18:10:26 +0000 +++ b/include/mysql.h 2009-10-02 10:36:28 +0000 @@ -527,6 +527,7 @@ int STDCALL mysql_set_server_option(MYS int STDCALL mysql_ping(MYSQL *mysql); const char * STDCALL mysql_stat(MYSQL *mysql); const char * STDCALL mysql_get_server_info(MYSQL *mysql); +const char * STDCALL mysql_get_server_name(MYSQL *mysql); const char * STDCALL mysql_get_client_info(void); unsigned long STDCALL mysql_get_client_version(void); const char * STDCALL mysql_get_host_info(MYSQL *mysql); @@ -560,6 +561,7 @@ void STDCALL mysql_debug(const char *de void STDCALL myodbc_remove_escape(MYSQL *mysql,char *name); unsigned int STDCALL mysql_thread_safe(void); my_bool STDCALL mysql_embedded(void); +my_bool STDCALL mariadb_connection(MYSQL *mysql); MYSQL_MANAGER* STDCALL mysql_manager_init(MYSQL_MANAGER* con); MYSQL_MANAGER* STDCALL mysql_manager_connect(MYSQL_MANAGER* con, const char* host, === modified file 'include/mysql.h.pp' --- a/include/mysql.h.pp 2009-03-12 22:27:35 +0000 +++ b/include/mysql.h.pp 2009-10-02 10:36:28 +0000 @@ -487,6 +487,7 @@ int mysql_set_server_option(MYSQL *mysql int mysql_ping(MYSQL *mysql); const char * mysql_stat(MYSQL *mysql); const char * mysql_get_server_info(MYSQL *mysql); +const char * mysql_get_server_name(MYSQL *mysql); const char * mysql_get_client_info(void); unsigned long mysql_get_client_version(void); const char * mysql_get_host_info(MYSQL *mysql); @@ -520,6 +521,7 @@ void mysql_debug(const char *debug); void myodbc_remove_escape(MYSQL *mysql,char *name); unsigned int mysql_thread_safe(void); my_bool mysql_embedded(void); +my_bool mariadb_connection(MYSQL *mysql); MYSQL_MANAGER* mysql_manager_init(MYSQL_MANAGER* con); MYSQL_MANAGER* mysql_manager_connect(MYSQL_MANAGER* con, const char* host, === modified file 'libmysql/libmysql.c' --- a/libmysql/libmysql.c 2009-04-25 10:05:32 +0000 +++ b/libmysql/libmysql.c 2009-10-02 10:36:28 +0000 @@ -1430,6 +1430,18 @@ mysql_get_server_info(MYSQL *mysql) } +my_bool STDCALL mariadb_connection(MYSQL *mysql) +{ + return strinstr(mysql->server_version, "MariaDB") != 0; +} + +const char * STDCALL +mysql_get_server_name(MYSQL *mysql) +{ + return mariadb_connection(mysql) ? "MariaDB" : "MySQL"; +} + + const char * STDCALL mysql_get_host_info(MYSQL *mysql) { === modified file 'mysql-test/r/mysqld_option_err.result' --- a/mysql-test/r/mysqld_option_err.result 2009-09-03 13:05:02 +0000 +++ b/mysql-test/r/mysqld_option_err.result 2009-10-02 11:09:01 +0000 @@ -3,4 +3,6 @@ Test bad binlog format. Test bad default storage engine. Test non-numeric value passed to number option. Test that bad value for plugin enum option is rejected correctly. +Test that --help --verbose works +Test that --not-known-option --help --verbose gives error Done. === modified file 'mysql-test/t/mysqld_option_err.test' --- a/mysql-test/t/mysqld_option_err.test 2009-09-25 14:38:02 +0000 +++ b/mysql-test/t/mysqld_option_err.test 2009-10-02 11:09:01 +0000 @@ -46,5 +46,14 @@ mkdir $MYSQLTEST_VARDIR/tmp/mysqld_optio --error 7 --exec $MYSQLD --skip-networking --datadir=$MYSQLTEST_VARDIR/tmp/mysqld_option_err --skip-grant-tables $EXAMPLE_PLUGIN_OPT --plugin-load=EXAMPLE=ha_example.so --plugin-example-enum-var=noexist >>$MYSQLTEST_VARDIR/tmp/mysqld_option_err/mysqltest.log 2>&1 +# +# Test that an wrong option with --help --verbose gives an error +# + +--echo Test that --help --verbose works +--exec $MYSQLD --help --verbose >>$MYSQLTEST_VARDIR/tmp/mysqld_option_err/mysqltest.log 2>&1 +--echo Test that --not-known-option --help --verbose gives error +--error 2 +--exec $MYSQLD --not-known-option --help --verbose >>$MYSQLTEST_VARDIR/tmp/mysqld_option_err/mysqltest.log 2>&1 --echo Done. === modified file 'sql/mysqld.cc' --- a/sql/mysqld.cc 2009-09-17 22:46:10 +0000 +++ b/sql/mysqld.cc 2009-10-02 11:09:01 +0000 @@ -4020,9 +4020,6 @@ server."); plugins_are_initialized= TRUE; /* Don't separate from init function */ } - if (opt_help) - unireg_abort(0); - /* we do want to exit if there are any other unknown options */ if (defaults_argc > 1) { @@ -4047,13 +4044,15 @@ server."); if (defaults_argc) { - fprintf(stderr, "%s: Too many arguments (first extra is '%s').\n" - "Use --verbose --help to get a list of available options\n", + fprintf(stderr, "%s: Too many arguments (first extra is '%s').\n", my_progname, *tmp_argv); unireg_abort(1); } } + if (opt_help) + unireg_abort(0); + /* if the errmsg.sys is not loaded, terminate to maintain behaviour */ if (!errmesg[0][0]) unireg_abort(1);

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2747)
by knielsen＠knielsen-hq.org 02 Oct '09

02 Oct '09

#At lp:maria 2747 knielsen(a)knielsen-hq.org 2009-10-02 Add missing windows files to `make dist`. modified: win/Makefile.am === modified file 'win/Makefile.am' --- a/win/Makefile.am 2009-06-18 08:49:44 +0000 +++ b/win/Makefile.am 2009-10-02 12:53:05 +0000 @@ -16,7 +16,8 @@ ## Process this file with automake to create Makefile.in EXTRA_DIST = build-vs71.bat build-vs8.bat build-vs8_x64.bat build-vs9.bat \ build-vs9_x64.bat configure.js README mysql_manifest.cmake \ - create_manifest.js create_def_file.js + create_manifest.js create_def_file.js build-nmake.bat \ + build-nmake-x64.bat # Don't update the files from bitkeeper %::SCCS/s.%

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2740: Fixed typo
by noreply＠launchpad.net 02 Oct '09

02 Oct '09

------------------------------------------------------------ revno: 2740 committer: Michael Widenius <monty(a)askmonty.org> branch nick: mysql-maria timestamp: Fri 2009-10-02 14:56:11 +0300 message: Fixed typo modified: client/mysql.cc -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2740)
by Michael Widenius 02 Oct '09

02 Oct '09

#At lp:maria based on revid:monty@askmonty.org-20091002110901-trnebmidd47qw6vk 2740 Michael Widenius 2009-10-02 Fixed typo modified: client/mysql.cc === modified file 'client/mysql.cc' --- a/client/mysql.cc 2009-10-02 10:36:28 +0000 +++ b/client/mysql.cc 2009-10-02 11:56:11 +0000 @@ -4735,7 +4735,7 @@ static const char *construct_prompt() if (connected) processed_prompt.append(mysql_get_server_name(&mysql)); else - processed_prompt.append("unkown"); + processed_prompt.append("unknown"); break; case 'h': {

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2739: When using --help --verbose give error and abort if there are any unknown options
by noreply＠launchpad.net 02 Oct '09

02 Oct '09

------------------------------------------------------------ revno: 2739 committer: Michael Widenius <monty(a)askmonty.org> branch nick: mysql-maria timestamp: Fri 2009-10-02 14:09:01 +0300 message: When using --help --verbose give error and abort if there are any unknown options modified: mysql-test/r/mysqld_option_err.result mysql-test/t/mysqld_option_err.test sql/mysqld.cc -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2739)
by Michael Widenius 02 Oct '09

02 Oct '09

#At lp:maria based on revid:monty@askmonty.org-20091002103628-i11aqe3i16vgf514 2739 Michael Widenius 2009-10-02 When using --help --verbose give error and abort if there are any unknown options modified: mysql-test/r/mysqld_option_err.result mysql-test/t/mysqld_option_err.test sql/mysqld.cc per-file messages: mysql-test/r/mysqld_option_err.result Test that mysqld --not-known-option --help --verbose gives error mysql-test/t/mysqld_option_err.test Test that mysqld --not-known-option --help --verbose gives error sql/mysqld.cc When using --help --verbose give error and abort if there are any unknown options === modified file 'mysql-test/r/mysqld_option_err.result' --- a/mysql-test/r/mysqld_option_err.result 2009-09-03 13:05:02 +0000 +++ b/mysql-test/r/mysqld_option_err.result 2009-10-02 11:09:01 +0000 @@ -3,4 +3,6 @@ Test bad binlog format. Test bad default storage engine. Test non-numeric value passed to number option. Test that bad value for plugin enum option is rejected correctly. +Test that --help --verbose works +Test that --not-known-option --help --verbose gives error Done. === modified file 'mysql-test/t/mysqld_option_err.test' --- a/mysql-test/t/mysqld_option_err.test 2009-09-25 14:38:02 +0000 +++ b/mysql-test/t/mysqld_option_err.test 2009-10-02 11:09:01 +0000 @@ -46,5 +46,14 @@ mkdir $MYSQLTEST_VARDIR/tmp/mysqld_optio --error 7 --exec $MYSQLD --skip-networking --datadir=$MYSQLTEST_VARDIR/tmp/mysqld_option_err --skip-grant-tables $EXAMPLE_PLUGIN_OPT --plugin-load=EXAMPLE=ha_example.so --plugin-example-enum-var=noexist >>$MYSQLTEST_VARDIR/tmp/mysqld_option_err/mysqltest.log 2>&1 +# +# Test that an wrong option with --help --verbose gives an error +# + +--echo Test that --help --verbose works +--exec $MYSQLD --help --verbose >>$MYSQLTEST_VARDIR/tmp/mysqld_option_err/mysqltest.log 2>&1 +--echo Test that --not-known-option --help --verbose gives error +--error 2 +--exec $MYSQLD --not-known-option --help --verbose >>$MYSQLTEST_VARDIR/tmp/mysqld_option_err/mysqltest.log 2>&1 --echo Done. === modified file 'sql/mysqld.cc' --- a/sql/mysqld.cc 2009-09-17 22:46:10 +0000 +++ b/sql/mysqld.cc 2009-10-02 11:09:01 +0000 @@ -4020,9 +4020,6 @@ server."); plugins_are_initialized= TRUE; /* Don't separate from init function */ } - if (opt_help) - unireg_abort(0); - /* we do want to exit if there are any other unknown options */ if (defaults_argc > 1) { @@ -4047,13 +4044,15 @@ server."); if (defaults_argc) { - fprintf(stderr, "%s: Too many arguments (first extra is '%s').\n" - "Use --verbose --help to get a list of available options\n", + fprintf(stderr, "%s: Too many arguments (first extra is '%s').\n", my_progname, *tmp_argv); unireg_abort(1); } } + if (opt_help) + unireg_abort(0); + /* if the errmsg.sys is not loaded, terminate to maintain behaviour */ if (!errmesg[0][0]) unireg_abort(1);

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2738: Added client functions:
by noreply＠launchpad.net 02 Oct '09

02 Oct '09

------------------------------------------------------------ revno: 2738 committer: Michael Widenius <monty(a)askmonty.org> branch nick: mysql-maria timestamp: Fri 2009-10-02 13:36:28 +0300 message: Added client functions: - mysql_get_server_name() This returns MySQL or MariaDB depending on the server type - mariadb_connection() This returns 1 if you are connected to a MariaDB server Modifed the MySQL command line client to print out if you are connected to MariaDB or MySQL Better default prompt (shows server you are connected to and base directory) modified: client/mysql.cc configure.in include/mysql.h include/mysql.h.pp libmysql/libmysql.c -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2738)
by Michael Widenius 02 Oct '09

02 Oct '09

#At lp:maria based on revid:knielsen@knielsen-hq.org-20090926022556-u2idjsv22sncgvxb 2738 Michael Widenius 2009-10-02 Added client functions: - mysql_get_server_name() This returns MySQL or MariaDB depending on the server type - mariadb_connection() This returns 1 if you are connected to a MariaDB server Modifed the MySQL command line client to print out if you are connected to MariaDB or MySQL Better default prompt (shows server you are connected to and base directory) modified: client/mysql.cc configure.in include/mysql.h include/mysql.h.pp libmysql/libmysql.c per-file messages: client/mysql.cc Print out if you are connected to MariaDB or MySQL Better default prompt (shows server you are connected to and base directory) Added option \N to prompt to print server name. configure.in Changed maria to MariaDB in version string include/mysql.h Added mysql_get_server_name() and mariadb_connection() include/mysql.h.pp Update for new API functions libmysql/libmysql.c Added client functions: - mysql_get_server_name() This returns MySQL or MariaDB depending on the server type - mariadb_connection() This returns 1 if you are connected to a MariaDB server === modified file 'client/mysql.cc' --- a/client/mysql.cc 2009-09-07 20:50:10 +0000 +++ b/client/mysql.cc 2009-10-02 10:36:28 +0000 @@ -43,7 +43,7 @@ #include <locale.h> #endif -const char *VER= "14.15"; +const char *VER= "14.16"; /* Don't try to make a nice table if the data is too big */ #define MAX_COLUMN_LENGTH 1024 @@ -1076,7 +1076,7 @@ int main(int argc,char *argv[]) delimiter_str= delimiter; default_prompt = my_strdup(getenv("MYSQL_PS1") ? getenv("MYSQL_PS1") : - "mysql> ",MYF(MY_WME)); + "\\N [\\d]> ",MYF(MY_WME)); current_prompt = my_strdup(default_prompt,MYF(MY_WME)); prompt_counter=0; @@ -1156,10 +1156,11 @@ int main(int argc,char *argv[]) signal(SIGINT, handle_sigint); // Catch SIGINT to clean up signal(SIGQUIT, mysql_end); // Catch SIGQUIT to clean up - put_info("Welcome to the MySQL monitor. Commands end with ; or \\g.", + put_info("Welcome to the MariaDB monitor. Commands end with ; or \\g.", INFO_INFO); sprintf((char*) glob_buffer.ptr(), - "Your MySQL connection id is %lu\nServer version: %s\n", + "Your %s connection id is %lu\nServer version: %s\n", + mysql_get_server_name(&mysql), mysql_thread_id(&mysql), server_version_string(&mysql)); put_info((char*) glob_buffer.ptr(),INFO_INFO); @@ -4369,6 +4370,7 @@ com_status(String *buffer __attribute__( tee_fprintf(stdout, "Using outfile:\t\t'%s'\n", opt_outfile ? outfile : ""); #endif tee_fprintf(stdout, "Using delimiter:\t%s\n", delimiter); + tee_fprintf(stdout, "Server:\t\t\t%s\n", mysql_get_server_name(&mysql)); tee_fprintf(stdout, "Server version:\t\t%s\n", server_version_string(&mysql)); tee_fprintf(stdout, "Protocol version:\t%d\n", mysql_get_proto_info(&mysql)); tee_fprintf(stdout, "Connection:\t\t%s\n", mysql_get_host_info(&mysql)); @@ -4700,7 +4702,7 @@ static void mysql_end_timer(ulong start_ strmov(strend(buff),")"); } -static const char* construct_prompt() +static const char *construct_prompt() { processed_prompt.free(); // Erase the old prompt time_t lclock = time(NULL); // Get the date struct @@ -4729,6 +4731,12 @@ static const char* construct_prompt() case 'd': processed_prompt.append(current_db ? current_db : "(none)"); break; + case 'N': + if (connected) + processed_prompt.append(mysql_get_server_name(&mysql)); + else + processed_prompt.append("unkown"); + break; case 'h': { const char *prompt; === modified file 'configure.in' --- a/configure.in 2009-09-15 11:55:37 +0000 +++ b/configure.in 2009-10-02 10:36:28 +0000 @@ -13,7 +13,7 @@ AC_CANONICAL_SYSTEM # # When merging new MySQL releases, update the version number to match the # MySQL version number, but reset the maria subrelease (-beta1). -AM_INIT_AUTOMAKE(mysql, 5.1.38-maria-beta1) +AM_INIT_AUTOMAKE(mysql, 5.1.38-MariaDB-beta1) AM_CONFIG_HEADER([include/config.h:config.h.in]) PROTOCOL_VERSION=10 === modified file 'include/mysql.h' --- a/include/mysql.h 2007-11-26 18:10:26 +0000 +++ b/include/mysql.h 2009-10-02 10:36:28 +0000 @@ -527,6 +527,7 @@ int STDCALL mysql_set_server_option(MYS int STDCALL mysql_ping(MYSQL *mysql); const char * STDCALL mysql_stat(MYSQL *mysql); const char * STDCALL mysql_get_server_info(MYSQL *mysql); +const char * STDCALL mysql_get_server_name(MYSQL *mysql); const char * STDCALL mysql_get_client_info(void); unsigned long STDCALL mysql_get_client_version(void); const char * STDCALL mysql_get_host_info(MYSQL *mysql); @@ -560,6 +561,7 @@ void STDCALL mysql_debug(const char *de void STDCALL myodbc_remove_escape(MYSQL *mysql,char *name); unsigned int STDCALL mysql_thread_safe(void); my_bool STDCALL mysql_embedded(void); +my_bool STDCALL mariadb_connection(MYSQL *mysql); MYSQL_MANAGER* STDCALL mysql_manager_init(MYSQL_MANAGER* con); MYSQL_MANAGER* STDCALL mysql_manager_connect(MYSQL_MANAGER* con, const char* host, === modified file 'include/mysql.h.pp' --- a/include/mysql.h.pp 2009-03-12 22:27:35 +0000 +++ b/include/mysql.h.pp 2009-10-02 10:36:28 +0000 @@ -487,6 +487,7 @@ int mysql_set_server_option(MYSQL *mysql int mysql_ping(MYSQL *mysql); const char * mysql_stat(MYSQL *mysql); const char * mysql_get_server_info(MYSQL *mysql); +const char * mysql_get_server_name(MYSQL *mysql); const char * mysql_get_client_info(void); unsigned long mysql_get_client_version(void); const char * mysql_get_host_info(MYSQL *mysql); @@ -520,6 +521,7 @@ void mysql_debug(const char *debug); void myodbc_remove_escape(MYSQL *mysql,char *name); unsigned int mysql_thread_safe(void); my_bool mysql_embedded(void); +my_bool mariadb_connection(MYSQL *mysql); MYSQL_MANAGER* mysql_manager_init(MYSQL_MANAGER* con); MYSQL_MANAGER* mysql_manager_connect(MYSQL_MANAGER* con, const char* host, === modified file 'libmysql/libmysql.c' --- a/libmysql/libmysql.c 2009-04-25 10:05:32 +0000 +++ b/libmysql/libmysql.c 2009-10-02 10:36:28 +0000 @@ -1430,6 +1430,18 @@ mysql_get_server_info(MYSQL *mysql) } +my_bool STDCALL mariadb_connection(MYSQL *mysql) +{ + return strinstr(mysql->server_version, "MariaDB") != 0; +} + +const char * STDCALL +mysql_get_server_name(MYSQL *mysql) +{ + return mariadb_connection(mysql) ? "MariaDB" : "MySQL"; +} + + const char * STDCALL mysql_get_host_info(MYSQL *mysql) {

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2746)
by knielsen＠knielsen-hq.org 02 Oct '09

02 Oct '09

#At lp:maria 2746 knielsen(a)knielsen-hq.org 2009-10-02 Add a simple way to robustly get the name of the generated source tarball, so as to not to have to rely on fragile regexp-parsing of configure.in. modified: Makefile.am === modified file 'Makefile.am' --- a/Makefile.am 2009-09-30 12:45:03 +0000 +++ b/Makefile.am 2009-10-02 09:40:03 +0000 @@ -60,6 +60,11 @@ dist-hook: --srcdir=$(top_srcdir) storage/myisam/myisamchk --silent --fast $(distdir)/win/data/mysql/*.MYI +# Simple target to allow scripts etc. to get the name of the source +# tarball easily. +show-dist-name: + @echo "$(PACKAGE)-$(VERSION)" + all-local: @ABI_CHECK@ tags:

1 0

[Maria-developers] Rev 2745: Temporary solution: skip PBXT (like we do e.g. for --skip-innodb) in embedded in file:///home/psergey/bzr-new/mysql-5.1-maria-contd4/
by Sergey Petrunya 01 Oct '09

01 Oct '09

At file:///home/psergey/bzr-new/mysql-5.1-maria-contd4/ ------------------------------------------------------------ revno: 2745 revision-id: psergey(a)askmonty.org-20091001212739-x4mzj5e7u8pig71f parent: psergey(a)askmonty.org-20091001104659-51vn4vtgjnqk1w27 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: mysql-5.1-maria-contd4 timestamp: Fri 2009-10-02 01:27:39 +0400 message: Temporary solution: skip PBXT (like we do e.g. for --skip-innodb) in embedded server, as it doesn't work there. === modified file 'sql/sql_plugin.cc' --- a/sql/sql_plugin.cc 2009-09-11 13:20:03 +0000 +++ b/sql/sql_plugin.cc 2009-10-01 21:27:39 +0000 @@ -1168,6 +1168,22 @@ !my_strnncoll(&my_charset_latin1, (const uchar*) plugin->name, 6, (const uchar*) "InnoDB", 6)) continue; +#ifdef EMBEDDED_LIBRARY + /* + MariaDB: disable PBXT in embedded server. We do this for two reasons + - PBXT currently doesn't work in embedded server (see + https://bugs.launchpad.net/maria/+bug/439889) + - Embedded server is supposed to be "leaner" and our current + understanding of that is "without PBXT". At the same time, we want + regular server to be with PBXT, and since we don't support compiling + embedded server with different options than the regular server, + the only way was to disable PBXT from here. + */ + if (!my_strnncoll(&my_charset_latin1, (const uchar*) plugin->name, + 4, (const uchar*) "PBXT", 4)) + continue; + +#endif bzero(&tmp, sizeof(tmp)); tmp.plugin= plugin; tmp.name.str= (char *)plugin->name;

1 0

[Maria-developers] Rev 2744: Fix embedded server windows build: add opt_table_elimination.cc to list of files in file:///home/psergey/bzr-new/mysql-5.1-maria-contd4/
by Sergey Petrunya 01 Oct '09

01 Oct '09

At file:///home/psergey/bzr-new/mysql-5.1-maria-contd4/ ------------------------------------------------------------ revno: 2744 revision-id: psergey(a)askmonty.org-20091001104659-51vn4vtgjnqk1w27 parent: knielsen(a)knielsen-hq.org-20090930124503-4n1uaohke4hjaw22 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: mysql-5.1-maria-contd4 timestamp: Thu 2009-10-01 14:46:59 +0400 message: Fix embedded server windows build: add opt_table_elimination.cc to list of files === modified file 'libmysqld/CMakeLists.txt' --- a/libmysqld/CMakeLists.txt 2009-09-07 20:50:10 +0000 +++ b/libmysqld/CMakeLists.txt 2009-10-01 10:46:59 +0000 @@ -112,6 +112,7 @@ ../sql/item_xmlfunc.cc ../sql/key.cc ../sql/lock.cc ../sql/log.cc ../sql/log_event.cc ../sql/mf_iocache.cc ../sql/my_decimal.cc ../sql/net_serv.cc ../sql/opt_range.cc ../sql/opt_sum.cc + ../sql/opt_table_elimination.cc ../sql/parse_file.cc ../sql/procedure.cc ../sql/protocol.cc ../sql/records.cc ../sql/repl_failsafe.cc ../sql/rpl_filter.cc ../sql/rpl_record.cc

1 0

Re: [Maria-developers] automake fixes for maria 5.1
by Kristian Nielsen 30 Sep '09

30 Sep '09

Peter Lieverdink <peter(a)openquery.com> writes: > I've been working on creating packaging scripts for Maria on Debian/Ubuntu and came across a few issues with the automake scripts. They are basically all related to the use case where srcdir != builddir - as is the case in the debian foo. > > My changes are on launchpad; in https://code.launchpad.net/~ourdelta-core/maria/maria-5.1-automakefixes > > I've submitted this branch for merging back into maria/maria-5.1-merge. Thanks a lot! I pushed the merge into maria-5.1-merge. - Kristian.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2743)
by knielsen＠knielsen-hq.org 30 Sep '09

30 Sep '09

#At lp:maria 2743 knielsen(a)knielsen-hq.org 2009-09-30 Add environment variable MTR_EXTRA_OPTIONS to influence `make test*` targets. This allows to eg use --mem or --parallel for better performance on big boxes. modified: Makefile.am === modified file 'Makefile.am' --- a/Makefile.am 2009-09-07 20:50:10 +0000 +++ b/Makefile.am 2009-09-30 12:45:03 +0000 @@ -81,6 +81,9 @@ tags: # environment variable MTR_BUILD_THREAD. The script "mysql-test-run" # will then calculate the various port numbers it needs from this, # making sure each user use different ports. +# +# Set MTR_EXTRA_OPTIONS to add custom options for mysql-test-run, +# like MTR_EXTRA_OPTIONS"--parallel=3 --mem". test-unit: cd unittest; $(MAKE) test @@ -90,29 +93,29 @@ test-unit-big: test-ps: cd mysql-test ; \ - @PERL@ ./mysql-test-run.pl $(force) --ps-protocol --mysqld=--binlog-format=mixed + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) $(force) --ps-protocol --mysqld=--binlog-format=mixed test-nr: cd mysql-test ; \ - @PERL@ ./mysql-test-run.pl $(force) --mysqld=--binlog-format=row + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) $(force) --mysqld=--binlog-format=row test-pr: cd mysql-test ; \ - @PERL@ ./mysql-test-run.pl $(force) $(mem) --ps-protocol --mysqld=--binlog-format=row #@libevent_test_option@ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) $(force) $(mem) --ps-protocol --mysqld=--binlog-format=row #@libevent_test_option@ test-ns: cd mysql-test ; \ - @PERL@ ./mysql-test-run.pl $(force) $(mem) --mysqld=--binlog-format=mixed + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) $(force) $(mem) --mysqld=--binlog-format=mixed test-binlog-statement: cd mysql-test ; \ - @PERL@ ./mysql-test-run.pl $(force) --mysqld=--binlog-format=statement + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) $(force) --mysqld=--binlog-format=statement # This code is duplicated in "test-bt", see the Changeset comment of 2007-Dec-07 test-embedded: if [ -e bin/mysqltest_embedded -o -e libmysqld/examples/mysqltest_embedded ] ; then \ cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --comment=embedded --force --timer \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --comment=embedded --force --timer \ --embedded-server --skip-rpl --skip-ndbcluster ; \ else \ echo "no program found for 'embedded' tests - skipped testing" ; \ @@ -120,14 +123,14 @@ test-embedded: test-reprepare: cd mysql-test ; \ - @PERL@ ./mysql-test-run.pl $(force) $(mem) --ps-protocol \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) $(force) $(mem) --ps-protocol \ --mysqld=--debug=+d,reprepare_each_statement test: test-ns test-pr smoke: cd mysql-test ; \ - @PERL@ ./mysql-test-run.pl --do-test=s + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --do-test=s test-full: test test-nr test-ps @@ -143,43 +146,43 @@ test-force-mem: test-bt: -cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --comment=normal --force --timer \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --comment=normal --force --timer \ --skip-ndbcluster --report-features -cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --comment=ps --force --timer \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --comment=ps --force --timer \ --skip-ndbcluster --ps-protocol -if [ -e bin/ndbd -o -e storage/ndb/src/kernel/ndbd ] ; then \ cd mysql-test ; \ MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --comment=ndb+rpl_ndb+ps --force --timer \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --comment=ndb+rpl_ndb+ps --force --timer \ --ps-protocol --mysqld=--binlog-format=row --suite=ndb,rpl_ndb ; \ MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --comment=ndb --force --timer \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --comment=ndb --force --timer \ --with-ndbcluster-only ; \ else \ echo "no program found for 'ndbcluster' tests - skipped testing" ; \ fi -cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --force --comment=funcs1+ps --ps-protocol --reorder --suite=funcs_1 + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --comment=funcs1+ps --ps-protocol --reorder --suite=funcs_1 -cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --force --comment=funcs2 --suite=funcs_2 + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --comment=funcs2 --suite=funcs_2 -cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --force --comment=partitions --suite=parts + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --comment=partitions --suite=parts -cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --force --comment=stress --suite=stress + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --comment=stress --suite=stress -cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --force --comment=jp --suite=jp + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --comment=jp --suite=jp -if [ -d mysql-test/suite/nist ] ; then \ cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --comment=nist --force --suite=nist ; \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --comment=nist --force --suite=nist ; \ fi -if [ -d mysql-test/suite/nist ] ; then \ cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --comment=nist+ps --force --suite=nist --ps-protocol ; \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --comment=nist+ps --force --suite=nist --ps-protocol ; \ fi -if [ -e bin/mysqltest_embedded -o -e libmysqld/examples/mysqltest_embedded ] ; then \ cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --comment=embedded --force --timer \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --comment=embedded --force --timer \ --embedded-server --skip-rpl --skip-ndbcluster ; \ else \ echo "no program found for 'embedded' tests - skipped testing" ; \ @@ -187,22 +190,22 @@ test-bt: test-bt-fast: -cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --comment=ps --force --timer \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --comment=ps --force --timer \ --skip-ndbcluster --ps-protocol --report-features -if [ -e bin/ndbd -o -e storage/ndb/src/kernel/ndbd ] ; then \ cd mysql-test ; \ MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --comment=ndb --force --timer \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --comment=ndb --force --timer \ --with-ndbcluster-only ; \ else \ echo "no program found for 'ndbcluster' tests - skipped testing" ; \ fi -cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --force --comment=stress --suite=stress + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --comment=stress --suite=stress test-bt-debug: -cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl --comment=debug --force --timer \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --comment=debug --force --timer \ --skip-ndbcluster --skip-rpl --report-features test-bt-debug-fast: @@ -216,32 +219,32 @@ test-force-full-pl: test-force-full test-ext-funcs: cd mysql-test ; \ - @PERL@ ./mysql-test-run.pl --force --reorder --suite=funcs_1 ; \ - @PERL@ ./mysql-test-run.pl --force --suite=funcs_2 + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --reorder --suite=funcs_1 ; \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --suite=funcs_2 test-ext-rpl: cd mysql-test ; \ - @PERL@ ./mysql-test-run.pl --force --suite=rpl + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --suite=rpl test-ext-partitions: cd mysql-test ; \ - @PERL@ ./mysql-test-run.pl --force --suite=parts + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --suite=parts test-ext-jp: cd mysql-test ; \ - @PERL@ ./mysql-test-run.pl --force --suite=jp + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --suite=jp test-ext-stress: cd mysql-test ; \ - @PERL@ ./mysql-test-run.pl --force --big-test --suite=stress + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --big-test --suite=stress test-ext: test-ext-funcs test-ext-rpl test-ext-partitions test-ext-jp test-ext-stress test-fast: cd mysql-test ; \ - @PERL@ ./mysql-test-run.pl $(subset) --force --skip-ndb --skip-innodb --skip-im --skip-rpl ; \ - @PERL@ ./mysql-test-run.pl $(subset) --force --suite=funcs_1 --do-test=myisam ; \ - @PERL@ ./mysql-test-run.pl $(subset) --force --suite=stress --do-test=ddl_myisam + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) $(subset) --force --skip-ndb --skip-innodb --skip-im --skip-rpl ; \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) $(subset) --force --suite=funcs_1 --do-test=myisam ; \ + @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) $(subset) --force --suite=stress --do-test=ddl_myisam test-fast-view: $(MAKE) subset=--view-protocol test-fast

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2742)
by knielsen＠knielsen-hq.org 30 Sep '09

30 Sep '09

#At lp:maria 2742 knielsen(a)knielsen-hq.org 2009-09-30 [merge] Merge autotools fixes to be able to build with builddir != srcdir. Author: Peter Lieverdink modified: config/ac-macros/libevent.m4 configure.in extra/libevent/Makefile.am === modified file 'config/ac-macros/libevent.m4' --- a/config/ac-macros/libevent.m4 2009-03-12 22:27:35 +0000 +++ b/config/ac-macros/libevent.m4 2009-09-29 23:36:15 +0000 @@ -13,7 +13,7 @@ AC_DEFUN([MYSQL_USE_BUNDLED_LIBEVENT], [ AC_SUBST([libevent_dir]) libevent_libs="\$(top_builddir)/extra/libevent/libevent.a" - libevent_includes="-I\$(top_builddir)/extra/libevent" + libevent_includes="-I\$(top_srcdir)/extra/libevent" libevent_test_option="--mysqld=--thread-handling=pool-of-threads" AC_SUBST(libevent_libs) AC_SUBST(libevent_includes) === modified file 'configure.in' --- a/configure.in 2009-09-15 11:55:37 +0000 +++ b/configure.in 2009-09-29 22:50:10 +0000 @@ -2697,7 +2697,7 @@ case $SYSTEM_TYPE in fi # if there is no readline, but we want to build with readline, we fail - if [test "$want_to_use_readline" = "yes"] && [test ! -d "./cmd-line-utils/readline"] + if [test "$want_to_use_readline" = "yes"] && [test ! -d "$srcdir/cmd-line-utils/readline"] then AC_MSG_ERROR([This commercially licensed MySQL source package can't be built with libreadline. Please use --with-libedit to use === modified file 'extra/libevent/Makefile.am' --- a/extra/libevent/Makefile.am 2009-03-12 22:27:35 +0000 +++ b/extra/libevent/Makefile.am 2009-09-29 23:00:57 +0000 @@ -22,7 +22,7 @@ include_HEADERS = event.h evutil.h event BUILT_SOURCES = event-config.h -event-config.h: $(top_srcdir)/include/config.h +event-config.h: $(top_builddir)/include/config.h echo '/* event-config.h' > $@ echo ' * Generated by autoconf; post-processed by libevent.' >> $@ echo ' * Do not edit this file.' >> $@ @@ -33,7 +33,7 @@ event-config.h: $(top_srcdir)/include/co sed -e 's/#define /#define _EVENT_/' \ -e 's/#undef /#undef _EVENT_/' \ - -e 's/#ifndef /#ifndef _EVENT_/' < $(top_srcdir)/include/config.h >> $@ + -e 's/#ifndef /#ifndef _EVENT_/' < $(top_builddir)/include/config.h >> $@ echo "#endif" >> $@ AM_CPPFLAGS = -Icompat -I$(top_srcdir)/include

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2741)
by knielsen＠knielsen-hq.org 30 Sep '09

30 Sep '09

#At lp:maria 2741 knielsen(a)knielsen-hq.org 2009-09-30 Completely disable the innodb plugin. Don't ever build it, and don't include it in source tarballs. (we keep the sources in bzr to not get conflicts on every upstream change). renamed: storage/innodb_plugin/plug.in => storage/innodb_plugin/plug.in.disabled === renamed file 'storage/innodb_plugin/plug.in' => 'storage/innodb_plugin/plug.in.disabled'

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2740) Bug#46010
by knielsen＠knielsen-hq.org 30 Sep '09

30 Sep '09

#At lp:maria 2740 knielsen(a)knielsen-hq.org 2009-09-30 Apply patch from Bug#46010. This is needed to fix failures seen in Buildbot (main.ctype_gbk_binlog). Problem is that some tests did not RESET MASTER, causing them to pick up random binlog files depending on prior tests. modified: mysql-test/r/ctype_gbk_binlog.result mysql-test/r/mysqlbinlog-cp932.result mysql-test/suite/binlog/r/binlog_incident.result mysql-test/suite/binlog/r/binlog_tmp_table.result mysql-test/suite/binlog/t/binlog_incident.test mysql-test/suite/binlog/t/binlog_tmp_table.test mysql-test/t/ctype_gbk_binlog.test mysql-test/t/mysqlbinlog-cp932.test === modified file 'mysql-test/r/ctype_gbk_binlog.result' --- a/mysql-test/r/ctype_gbk_binlog.result 2009-07-07 14:18:44 +0000 +++ b/mysql-test/r/ctype_gbk_binlog.result 2009-09-30 06:35:31 +0000 @@ -1,3 +1,4 @@ +RESET MASTER; SET NAMES gbk; CREATE TABLE t1 ( f1 BLOB === modified file 'mysql-test/r/mysqlbinlog-cp932.result' --- a/mysql-test/r/mysqlbinlog-cp932.result 2007-02-19 10:57:06 +0000 +++ b/mysql-test/r/mysqlbinlog-cp932.result 2009-09-30 06:35:31 +0000 @@ -1,4 +1,4 @@ -flush logs; +RESET MASTER; create table t3 (f text character set utf8); create table t4 (f text character set cp932); flush logs; === modified file 'mysql-test/suite/binlog/r/binlog_incident.result' --- a/mysql-test/suite/binlog/r/binlog_incident.result 2009-05-11 11:32:38 +0000 +++ b/mysql-test/suite/binlog/r/binlog_incident.result 2009-09-30 06:35:31 +0000 @@ -1,3 +1,4 @@ +RESET MASTER; CREATE TABLE t1 (a INT); INSERT INTO t1 VALUES (1),(2),(3); SELECT * FROM t1; === modified file 'mysql-test/suite/binlog/r/binlog_tmp_table.result' --- a/mysql-test/suite/binlog/r/binlog_tmp_table.result 2009-01-28 14:35:12 +0000 +++ b/mysql-test/suite/binlog/r/binlog_tmp_table.result 2009-09-30 06:35:31 +0000 @@ -1,3 +1,4 @@ +RESET MASTER; create table foo (a int); flush logs; create temporary table tmp1_foo like foo; === modified file 'mysql-test/suite/binlog/t/binlog_incident.test' --- a/mysql-test/suite/binlog/t/binlog_incident.test 2009-05-11 11:32:38 +0000 +++ b/mysql-test/suite/binlog/t/binlog_incident.test 2009-09-30 06:35:31 +0000 @@ -6,6 +6,7 @@ source include/have_log_bin.inc; source include/have_debug.inc; let $MYSQLD_DATADIR= `select @@datadir`; +RESET MASTER; CREATE TABLE t1 (a INT); @@ -24,4 +25,4 @@ exec $MYSQL_BINLOG --start-position=106 eval SELECT cont LIKE '%RELOAD DATABASE; # Shall generate syntax error%' AS `Contain RELOAD DATABASE` FROM (SELECT load_file('$MYSQLTEST_VARDIR/tmp/binlog_incident-bug44442.sql') AS cont) AS tbl; --enable_query_log -remove_file $MYSQLTEST_VARDIR/tmp/binlog_incident-bug44442.sql; \ No newline at end of file +remove_file $MYSQLTEST_VARDIR/tmp/binlog_incident-bug44442.sql; === modified file 'mysql-test/suite/binlog/t/binlog_tmp_table.test' --- a/mysql-test/suite/binlog/t/binlog_tmp_table.test 2009-01-31 21:47:50 +0000 +++ b/mysql-test/suite/binlog/t/binlog_tmp_table.test 2009-09-30 06:35:31 +0000 @@ -30,6 +30,7 @@ source include/have_binlog_format_mixed_ connect (master,127.0.0.1,root,,test,$MASTER_MYPORT,); connect (master1,127.0.0.1,root,,test,$MASTER_MYPORT,); +RESET MASTER; create table foo (a int); === modified file 'mysql-test/t/ctype_gbk_binlog.test' --- a/mysql-test/t/ctype_gbk_binlog.test 2009-07-07 14:18:44 +0000 +++ b/mysql-test/t/ctype_gbk_binlog.test 2009-09-30 06:35:31 +0000 @@ -1,6 +1,7 @@ -- source include/have_binlog_format_mixed_or_statement.inc -- source include/have_gbk.inc +RESET MASTER; SET NAMES gbk; --character_set gbk === modified file 'mysql-test/t/mysqlbinlog-cp932.test' --- a/mysql-test/t/mysqlbinlog-cp932.test 2007-12-12 17:19:24 +0000 +++ b/mysql-test/t/mysqlbinlog-cp932.test 2009-09-30 06:35:31 +0000 @@ -5,8 +5,9 @@ -- source include/have_cp932.inc -- source include/have_log_bin.inc +RESET MASTER; + # Bug#16217 (mysql client did not know how not switch its internal charset) -flush logs; create table t3 (f text character set utf8); create table t4 (f text character set cp932); --exec $MYSQL --default-character-set=utf8 test -e "insert into t3 values(_utf8'ソ')" @@ -14,7 +15,7 @@ create table t4 (f text character set cp flush logs; rename table t3 to t03, t4 to t04; let $MYSQLD_DATADIR= `select @@datadir`; ---exec $MYSQL_BINLOG --short-form $MYSQLD_DATADIR/master-bin.000002 | $MYSQL --default-character-set=utf8 +--exec $MYSQL_BINLOG --short-form $MYSQLD_DATADIR/master-bin.000001 | $MYSQL --default-character-set=utf8 # original and recovered data must be equal select HEX(f) from t03; select HEX(f) from t3;

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2739)
by knielsen＠knielsen-hq.org 29 Sep '09

29 Sep '09

#At lp:maria 2739 knielsen(a)knielsen-hq.org 2009-09-29 Fix race in test of times shown in SHOW FULL PROCESSLIST. (These are by their nature difficult to test reliably due to differences in timing). modified: mysql-test/r/information_schema.result mysql-test/t/information_schema.test === modified file 'mysql-test/r/information_schema.result' --- a/mysql-test/r/information_schema.result 2009-09-23 11:03:47 +0000 +++ b/mysql-test/r/information_schema.result 2009-09-29 20:19:43 +0000 @@ -1385,7 +1385,7 @@ who other connection here SELECT IF(`time` > 0, 'OK', `time`) AS time_low, IF(`time` < 1000, 'OK', `time`) AS time_high, -IF(time_ms > 900, 'OK', time_ms) AS time_ms_low, +IF(time_ms > 1500, 'OK', time_ms) AS time_ms_low, IF(time_ms < 1000000, 'OK', time_ms) AS time_ms_high FROM INFORMATION_SCHEMA.PROCESSLIST WHERE ID=@tid; === modified file 'mysql-test/t/information_schema.test' --- a/mysql-test/t/information_schema.test 2009-09-23 11:03:47 +0000 +++ b/mysql-test/t/information_schema.test 2009-09-29 20:19:43 +0000 @@ -1105,13 +1105,13 @@ connect (pslistcon,localhost,root,,test) let $ID= `select connection_id()`; SELECT 'other connection here' AS who; connection default; -sleep 1; +sleep 2; --disable_query_log eval SET @tid=$ID; --enable_query_log SELECT IF(`time` > 0, 'OK', `time`) AS time_low, IF(`time` < 1000, 'OK', `time`) AS time_high, - IF(time_ms > 900, 'OK', time_ms) AS time_ms_low, + IF(time_ms > 1500, 'OK', time_ms) AS time_ms_low, IF(time_ms < 1000000, 'OK', time_ms) AS time_ms_high FROM INFORMATION_SCHEMA.PROCESSLIST WHERE ID=@tid;

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2738)
by knielsen＠knielsen-hq.org 29 Sep '09

29 Sep '09

#At lp:maria 2738 knielsen(a)knielsen-hq.org 2009-09-29 Fix running test suite from installed directory rather than source directory. Fix trivial typo in BUILD/* scripts. modified: BUILD/SETUP.sh mysql-test/mysql-test-run.pl per-file messages: BUILD/SETUP.sh Fix typo. mysql-test/mysql-test-run.pl Fix mysql-test-run.pl to be able to find plugins used in testing also when running the test suite from an installed mysqld package. === modified file 'BUILD/SETUP.sh' --- a/BUILD/SETUP.sh 2009-09-10 08:11:23 +0000 +++ b/BUILD/SETUP.sh 2009-09-29 19:02:48 +0000 @@ -173,7 +173,7 @@ local_infile_configs="--enable-local-inf max_no_embedded_configs="$SSL_LIBRARY --with-plugins=max" max_no_ndb_configs="$SSL_LIBRARY --with-plugins=max-no-ndb --with-embedded-server --with-libevent" -max_configs="$SSL_LIBRARY --with-plugins=max --with-embedded-server -with-libevent" +max_configs="$SSL_LIBRARY --with-plugins=max --with-embedded-server --with-libevent" # Disable NDB in maria max builds max_configs=$max_no_ndb_configs === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2009-09-26 02:25:56 +0000 +++ b/mysql-test/mysql-test-run.pl 2009-09-29 19:02:48 +0000 @@ -1886,7 +1886,8 @@ sub environment_setup { # -------------------------------------------------------------------------- my $lib_udf_example= mtr_file_exists(vs_config_dirs('sql', 'udf_example.dll'), - "$basedir/sql/.libs/udf_example.so",); + "$basedir/sql/.libs/udf_example.so", + "$basedir/lib/mysql/plugin/udf_example.so",); if ( $lib_udf_example ) { @@ -1913,7 +1914,8 @@ sub environment_setup { } my $lib_example_plugin= mtr_file_exists(vs_config_dirs('storage/example',$plugin_filename), - "$basedir/storage/example/.libs/".$plugin_filename); + "$basedir/storage/example/.libs/".$plugin_filename, + "$basedir/lib/mysql/plugin/".$plugin_filename); $ENV{'EXAMPLE_PLUGIN'}= ($lib_example_plugin ? basename($lib_example_plugin) : ""); $ENV{'EXAMPLE_PLUGIN_OPT'}= "--plugin-dir=". @@ -1928,7 +1930,8 @@ sub environment_setup { # ---------------------------------------------------- my $lib_simple_parser= mtr_file_exists(vs_config_dirs('plugin/fulltext', 'mypluglib.dll'), - "$basedir/plugin/fulltext/.libs/mypluglib.so",); + "$basedir/plugin/fulltext/.libs/mypluglib.so", + "$basedir/lib/mysql/plugin/mypluglib.so",); $ENV{'SIMPLE_PARSER'}= ($lib_simple_parser ? basename($lib_simple_parser) : "");

1 0

[Maria-developers] Notes from todays Maria call
by Henrik Ingo 29 Sep '09

29 Sep '09

Hi Since I for once took good notes (less stressed today :-) I'm sharing them here. Sergey, maybe you can reuse some of this to update todo page. Maria - win and valgrind problems - race conditions only seen on slow machines - problem likely in the test, not the server - this is not blocking release - arjen working on builds, henrik spoke with him today. - arjen happy with cooperation with kristian and says mariadb is easier to build than mysql. - bo and kristian set up windows dev environment for bo today - will provide another buildbot slave for win - kristian testing make dist - which can be used to produce a tarball - bo and kristian working on windows binary - release tree is maria-5.1-merge - Windows VM needed by Kristian, and also as a build environment - Daniel now setting this up on is own dev server - Sanja's QC bug fix still waiting for review. This is also not blocking release. (Current behavior is as in MySQL.) - Hakan adds smoke test to MariaDB test plan - We have a couple of mirrors for downloads. Ourdelta will probably play a role here too. - Ourdelta provides also apt and yum repos! - Henrik will talk with Bryan about tracking clicks of downloads - Henrik has not had time to look at documentation stuff Daniel is working on, will talk about that separately later. Also talked about Virtual Columns, Create table options etc, not important for release. henrik -- email: henrik.ingo(a)avoinelama.fi tel: +358-40-5697354 www: www.avoinelama.fi/~hingo book: www.openlife.cc

1 0

[Maria-developers] Updated (by Knielsen): mysqlbinlog: remove undesired effects of format description binlog statement (50)
by worklog-noreply＠askmonty.org 29 Sep '09

29 Sep '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: mysqlbinlog: remove undesired effects of format description binlog statement CREATION DATE..: Mon, 17 Aug 2009, 14:12 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 50 (http://askmonty.org/worklog/?tid=50) VERSION........: Server-9.x STATUS.........: Code-Review PRIORITY.......: 60 WORKED HOURS...: 14 ESTIMATE.......: 6 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: -=-=(Knielsen - Tue, 29 Sep 2009, 10:12)=-=- Category updated. --- /tmp/wklog.50.old.3293 2009-09-29 10:12:26.000000000 +0300 +++ /tmp/wklog.50.new.3293 2009-09-29 10:12:26.000000000 +0300 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Knielsen - Tue, 29 Sep 2009, 10:12)=-=- Status updated. --- /tmp/wklog.50.old.3293 2009-09-29 10:12:26.000000000 +0300 +++ /tmp/wklog.50.new.3293 2009-09-29 10:12:26.000000000 +0300 @@ -1 +1 @@ -Un-Assigned +Code-Review -=-=(Knielsen - Mon, 28 Sep 2009, 16:00)=-=- Updated patch against MySQL 5.1.39 Fixes a couple of issues with the patch. Discussions with MySQL developer. Worked 4 hours and estimate 6 hours remain (original estimate unchanged). -=-=(Knielsen - Wed, 09 Sep 2009, 18:31)=-=- Discussed with MySQL developers on Bug#46640. Committed a suggested fix: https://lists.launchpad.net/maria-developers/msg00926.html Worked 8 hours and estimate 10 hours remain (original estimate increased by 18 hours). -=-=(Guest - Wed, 09 Sep 2009, 11:50)=-=- High Level Description modified. --- /tmp/wklog.50.old.11584 2009-09-09 11:50:12.000000000 +0300 +++ /tmp/wklog.50.new.11584 2009-09-09 11:50:12.000000000 +0300 @@ -5,7 +5,11 @@ This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. -2. When one applies "format description binlog statement" at the slave, it -will change the slave's server_id when applied. +2. When one executes a BINLOG statement (containing "format description binlog +statement" or other events), this will cause subsequent statements run in +the same session to be binlogged with the server id of the last event +executed, not with the real server id. This can cause problems like infinite +recursive application in a circular replication topology. This WL is to fix these issues. + -=-=(Knielsen - Tue, 08 Sep 2009, 15:07)=-=- Low Level Design modified. --- /tmp/wklog.50.old.19208 2009-09-08 15:07:09.000000000 +0300 +++ /tmp/wklog.50.new.19208 2009-09-08 15:07:09.000000000 +0300 @@ -1 +1,34 @@ +I think the fix for point 2 is to replace the call to +apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct +call to ev->apply_event(). However, I need to check two things to be sure this +is correct: + +1. The existing code does + + if (!ev->when) ev->when= my_time(0) + +Need to understand if this is needed/correct or not. + +2. The existing code does ev->update_pos(). I think this is redundant for +BINLOG statement (as it uses a fake rli structure), but I need to check to +make sure. + +Once this is done, point 1 may no longer be needed. The user can use +--base64-output=never when applying the output to a 5.0 server, and omit that +option when applying the output to a 5.1 server. There should be no need to +omit the format description event in the output when other BINLOG statements +for row events are present, as it no longer changes the server id. In fact the +format description event is required to be able to execute other BINLOG +statements, as its purpose is to define the binary format of the events +contained in these statements. + +Alternatively, we could implement that if the format description event of the +source binlog has server version < 5.1, and --base64-output=auto (default), +then the format description event is omitted (and should any BINLOG statement +be needed, unlikely as it is from a 5.0 server, we will need to throw an +error). + +The binlog format version for 5.0 and 5.1 is actually the same, hence the need +to look at server version to guess if the format description event can be +omitted. -=-=(Knielsen - Tue, 08 Sep 2009, 14:29)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.17531 2009-09-08 14:29:30.000000000 +0300 +++ /tmp/wklog.50.new.17531 2009-09-08 14:29:30.000000000 +0300 @@ -12,3 +12,37 @@ One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? +First, the problem is not the format description log event. In fact all log +events have their own value of server_id, and every event in the BINLOG +statement is executed with its own id as server_id. + +It seems it was introduced deliberately in the patch for Bug#32407. This patch +makes sql thread and BINLOG statement share code path for executing the binary +event, even though the bug is really about something different (outputting +format description event to allow proper execution of other BINLOG +statements). + +Nevertheless, I think using the server_id in the events of a BINLOG statement +is wrong: + +1. It is different behaviour from executing normal statement-based events not +written using BINLOG. + +2. It causes the possibility for infinite cycle of replication in a cyclic +replication setup, if the server ids in BINLOG are different from all other +server ids in the cycle. + +3. The functionality of applying events with original server id from the event +is still available by pointing the slave thread to the binlog. The +functionality of applying row-based binlog events with server id of the server +executing the BINLOG statements is not otherwise available. + +In fact most of the code from the slave thread that is now also run when +executing BINLOG statements is redundant, or does the wrong thing (like +updating binlog position). + +Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be +able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is +it to be able to simulate the effect of the slave sql thread? I think the +former is the most important one. + -=-=(Knielsen - Mon, 07 Sep 2009, 10:06)=-=- Research worklog, bugs, and existing server code. Worked 2 hours and estimate 0 hours remain (original estimate increased by 2 hours). -=-=(Psergey - Mon, 17 Aug 2009, 14:13)=-=- Dependency created: 39 now depends on 50 -=-=(Psergey - Mon, 17 Aug 2009, 14:13)=-=- High-Level Specification modified. --- /tmp/wklog.50.old.11389 2009-08-17 14:13:05.000000000 +0300 +++ /tmp/wklog.50.new.11389 2009-08-17 14:13:05.000000000 +0300 @@ -1 +1,14 @@ +First item +---------- +AFAIU what needs to be done is: +1. record a source server version (it is in the first binlog event). +2. don't emit a BINLOG statement if the recorded version number is 5.0.x or + below. + +and we'll get a 5.0-applicable binlog. + +Second item +----------- +One question that one needs to sort out before disabling server_id change is +why it was put there in the first place? Should it be always removed? DESCRIPTION: According to complaints in BUG#46640: 1. mysqlbinlog will emit a 5.0-incompatible "format description binlog statement" even when reading a binary log that was produced by 5.0. This will cause an error when one tires to apply mysqlbinlog output to a 5.0.x server. 2. When one executes a BINLOG statement (containing "format description binlog statement" or other events), this will cause subsequent statements run in the same session to be binlogged with the server id of the last event executed, not with the real server id. This can cause problems like infinite recursive application in a circular replication topology. This WL is to fix these issues. HIGH-LEVEL SPECIFICATION: First item ---------- AFAIU what needs to be done is: 1. record a source server version (it is in the first binlog event). 2. don't emit a BINLOG statement if the recorded version number is 5.0.x or below. and we'll get a 5.0-applicable binlog. Second item ----------- One question that one needs to sort out before disabling server_id change is why it was put there in the first place? Should it be always removed? First, the problem is not the format description log event. In fact all log events have their own value of server_id, and every event in the BINLOG statement is executed with its own id as server_id. It seems it was introduced deliberately in the patch for Bug#32407. This patch makes sql thread and BINLOG statement share code path for executing the binary event, even though the bug is really about something different (outputting format description event to allow proper execution of other BINLOG statements). Nevertheless, I think using the server_id in the events of a BINLOG statement is wrong: 1. It is different behaviour from executing normal statement-based events not written using BINLOG. 2. It causes the possibility for infinite cycle of replication in a cyclic replication setup, if the server ids in BINLOG are different from all other server ids in the cycle. 3. The functionality of applying events with original server id from the event is still available by pointing the slave thread to the binlog. The functionality of applying row-based binlog events with server id of the server executing the BINLOG statements is not otherwise available. In fact most of the code from the slave thread that is now also run when executing BINLOG statements is redundant, or does the wrong thing (like updating binlog position). Maybe there is a confusion on the purpose of the BINLOG statement. Is it to be able to apply row-based events manually to a server (mysqlbinlog|mysql)? Or is it to be able to simulate the effect of the slave sql thread? I think the former is the most important one. LOW-LEVEL DESIGN: I think the fix for point 2 is to replace the call to apply_event_and_update_pos() in mysql_client_binlog_statement() with a direct call to ev->apply_event(). However, I need to check two things to be sure this is correct: 1. The existing code does if (!ev->when) ev->when= my_time(0) Need to understand if this is needed/correct or not. 2. The existing code does ev->update_pos(). I think this is redundant for BINLOG statement (as it uses a fake rli structure), but I need to check to make sure. Once this is done, point 1 may no longer be needed. The user can use --base64-output=never when applying the output to a 5.0 server, and omit that option when applying the output to a 5.1 server. There should be no need to omit the format description event in the output when other BINLOG statements for row events are present, as it no longer changes the server id. In fact the format description event is required to be able to execute other BINLOG statements, as its purpose is to define the binary format of the events contained in these statements. Alternatively, we could implement that if the format description event of the source binlog has server version < 5.1, and --base64-output=auto (default), then the format description event is omitted (and should any BINLOG statement be needed, unlikely as it is from a 5.0 server, we will need to throw an error). The binlog format version for 5.0 and 5.1 is actually the same, hence the need to look at server version to guess if the format description event can be omitted. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0