- developers - lists.mariadb.org

[Maria-developers] Updated (by Guest): Add a mysqlbinlog option to filter updates to certain tables (40)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to filter updates to certain tables CREATION DATE..: Mon, 10 Aug 2009, 13:25 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Psergey CATEGORY.......: Server-RawIdeaBin TASK ID........: 40 (http://askmonty.org/worklog/?tid=40) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Guest - Mon, 10 Aug 2009, 14:51)=-=- High-Level Specification modified. --- /tmp/wklog.40.old.16949 2009-08-10 14:51:33.000000000 +0300 +++ /tmp/wklog.40.new.16949 2009-08-10 14:51:33.000000000 +0300 @@ -1 +1,73 @@ +1. Context +---------- +At the moment, the server has these replication slave options: + + --replicate-do-table=db.tbl + --replicate-ignore-table=db.tbl + --replicate-wild-do-table=pattern.pattern + --replicate-wild-ignore-table=pattern.pattern + +They affect both RBR and SBR events. SBR events are checked after the +statement has been parsed, the server iterates over list of used tables and +checks them againist --replicate instructions. + +What is interesting is that this scheme still allows to update the ignored +table through a VIEW. + +2. Table filtering in mysqlbinlog +--------------------------------- + +Per-table filtering of RBR events is easy (as it is relatively easy to extract +the name of the table that the event applies to). + +Per-table filtering of SBR events is hard, as generally it is not apparent +which tables the statement refers to. + +This opens possible options: + +2.1 Put the parser into mysqlbinlog +----------------------------------- +Once we have a full parser in mysqlbinlog, we'll be able to check which tables +are used by a statement, and will allow to show behaviour identical to those +that one obtains when using --replicate-* slave options. + +(It is not clear how much effort is needed to put the parser into mysqlbinlog. +Any guesses?) + + +2.2 Use dumb regexp match +------------------------- +Use a really dumb approach. A query is considered to be modifying table X if +it matches an expression + +CREATE TABLE $tablename +DROP $tablename +UPDATE ...$tablename ... SET // here '...' can't contain the word 'SET' +DELETE ...$tablename ... WHERE // same as above +ALTER TABLE $tablename +.. etc (go get from the grammar) .. + +The advantage over doing the same in awk is that mysqlbinlog will also process +RBR statements, and together with that will provide a working solution for +those who are careful with their table names not mixing with string constants +and such. + +(TODO: string constants are of particular concern as they come from +[potentially hostile] users, unlike e.g. table aliases which come from +[not hostile] developers. Remove also all string constants before attempting +to do match?) + +2.3 Have the master put annotations +----------------------------------- +We could add a master option so that it injects into query a mark that tells +which tables the query will affect, e.g. for the query + + UPDATE t1 LEFT JOIN db3.t2 ON ... WHERE ... + + +the binlog will have + + /* !mysqlbinlog: updates t1,db3.t2 */ UPDATE t1 LEFT JOIN ... + +and further processing in mysqlbinlog will be trivial. DESCRIPTION: Replication slave can be set to filter updates to certain tables with --replicate-[wild-]{do,ignore}-table options. This task is about adding similar functionality to mysqlbinlog. HIGH-LEVEL SPECIFICATION: 1. Context ---------- At the moment, the server has these replication slave options: --replicate-do-table=db.tbl --replicate-ignore-table=db.tbl --replicate-wild-do-table=pattern.pattern --replicate-wild-ignore-table=pattern.pattern They affect both RBR and SBR events. SBR events are checked after the statement has been parsed, the server iterates over list of used tables and checks them againist --replicate instructions. What is interesting is that this scheme still allows to update the ignored table through a VIEW. 2. Table filtering in mysqlbinlog --------------------------------- Per-table filtering of RBR events is easy (as it is relatively easy to extract the name of the table that the event applies to). Per-table filtering of SBR events is hard, as generally it is not apparent which tables the statement refers to. This opens possible options: 2.1 Put the parser into mysqlbinlog ----------------------------------- Once we have a full parser in mysqlbinlog, we'll be able to check which tables are used by a statement, and will allow to show behaviour identical to those that one obtains when using --replicate-* slave options. (It is not clear how much effort is needed to put the parser into mysqlbinlog. Any guesses?) 2.2 Use dumb regexp match ------------------------- Use a really dumb approach. A query is considered to be modifying table X if it matches an expression CREATE TABLE $tablename DROP $tablename UPDATE ...$tablename ... SET // here '...' can't contain the word 'SET' DELETE ...$tablename ... WHERE // same as above ALTER TABLE $tablename .. etc (go get from the grammar) .. The advantage over doing the same in awk is that mysqlbinlog will also process RBR statements, and together with that will provide a working solution for those who are careful with their table names not mixing with string constants and such. (TODO: string constants are of particular concern as they come from [potentially hostile] users, unlike e.g. table aliases which come from [not hostile] developers. Remove also all string constants before attempting to do match?) 2.3 Have the master put annotations ----------------------------------- We could add a master option so that it injects into query a mark that tells which tables the query will affect, e.g. for the query UPDATE t1 LEFT JOIN db3.t2 ON ... WHERE ... the binlog will have /* !mysqlbinlog: updates t1,db3.t2 */ UPDATE t1 LEFT JOIN ... and further processing in mysqlbinlog will be trivial. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): Using the Valgrind API in mysqld (23)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Using the Valgrind API in mysqld CREATION DATE..: Fri, 22 May 2009, 11:43 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 23 (http://askmonty.org/worklog/?tid=23) VERSION........: Server-5.1 STATUS.........: Code-Review PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: -=-=(Knielsen - Mon, 10 Aug 2009, 14:27)=-=- Low Level Design modified. --- /tmp/wklog.23.old.16018 2009-08-10 14:27:09.000000000 +0300 +++ /tmp/wklog.23.new.16018 2009-08-10 14:27:09.000000000 +0300 @@ -5,3 +5,5 @@ - sql/item_strfunc.cc (Item_func_compress). +Another good place is in the TRASH_MEM macro. + -=-=(Knielsen - Wed, 24 Jun 2009, 15:55)=-=- Supervisor updated. --- /tmp/wklog.23.old.944 2009-06-24 15:55:57.000000000 +0300 +++ /tmp/wklog.23.new.944 2009-06-24 15:55:57.000000000 +0300 @@ -1 +1 @@ -Knielsen +Monty -=-=(Knielsen - Wed, 24 Jun 2009, 15:53)=-=- Version updated. --- /tmp/wklog.23.old.911 2009-06-24 15:53:32.000000000 +0300 +++ /tmp/wklog.23.new.911 2009-06-24 15:53:32.000000000 +0300 @@ -1 +1 @@ -Maria-1.0 +Server-5.1 -=-=(Knielsen - Wed, 24 Jun 2009, 15:52)=-=- Version updated. --- /tmp/wklog.23.old.897 2009-06-24 15:52:43.000000000 +0300 +++ /tmp/wklog.23.new.897 2009-06-24 15:52:43.000000000 +0300 @@ -1 +1 @@ -Connector/.NET-2.1 +Maria-1.0 -=-=(Knielsen - Wed, 24 Jun 2009, 15:52)=-=- Version updated. --- /tmp/wklog.23.old.895 2009-06-24 15:52:28.000000000 +0300 +++ /tmp/wklog.23.new.895 2009-06-24 15:52:28.000000000 +0300 @@ -1 +1 @@ -Maria-1.0 +Connector/.NET-2.1 -=-=(Knielsen - Wed, 24 Jun 2009, 15:35)=-=- Version updated. --- /tmp/wklog.23.old.32742 2009-06-24 15:35:48.000000000 +0300 +++ /tmp/wklog.23.new.32742 2009-06-24 15:35:48.000000000 +0300 @@ -1 +1 @@ -Server-5.1 +Maria-1.0 -=-=(Knielsen - Fri, 22 May 2009, 14:31)=-=- Low Level Design modified. --- /tmp/wklog.23.old.24587 2009-05-22 14:31:52.000000000 +0300 +++ /tmp/wklog.23.new.24587 2009-05-22 14:31:52.000000000 +0300 @@ -1 +1,7 @@ +Two places where we call into libz, and where checking for defined parameters +would be good: + + - mysys/my_compress.c + + - sql/item_strfunc.cc (Item_func_compress). -=-=(Guest - Fri, 22 May 2009, 12:04)=-=- High-Level Specification modified. --- /tmp/wklog.23.old.18061 2009-05-22 12:04:05.000000000 +0300 +++ /tmp/wklog.23.new.18061 2009-05-22 12:04:05.000000000 +0300 @@ -26,3 +26,5 @@ initialised, it is possible to detect problems earlier, speeding up debugging. Such code can be added in more places over time as development and debugging goes on. + +See also a patch here: http://bugs.mysql.com/bug.php?id=44582 -=-=(Knielsen - Fri, 22 May 2009, 11:52)=-=- High-Level Specification modified. --- /tmp/wklog.23.old.17628 2009-05-22 11:52:33.000000000 +0300 +++ /tmp/wklog.23.new.17628 2009-05-22 11:52:33.000000000 +0300 @@ -1 +1,28 @@ +With custom memory allocators, using the Valgrind APIs we can tell Valgrind when +a memory block is allocated (so that data read from memory is marked as undefined +instead of being defined or not at random depending on prior use); and when a +memory block is freed (so that use after freeing can be reported as an error). +In some cases cheking for leaks may also be appropriate. + +Another possibility is to add an explicit check for whether memory is defined. + +One place this would be useful is when calling libz. Due to the design of that +library, Valgrind produces lots of false alarms about using undefined values +(I think the issue is that it runs a few bytes off of initialized memory to +reduce boundary checks in each loop iteration, then after the loop has checks to +avoid using the undefined part of the result). This means we have lots of libz +Valgrind suppressions and continue to add more as new warnings surface. So we +might easily miss a real problem in this area. This could be improved by adding +explicit checks at the call to libz functions that the passed memory is properly +defined. + +Another use is to improve debugging. It is often the case when debugging a +warning about using un-initialised memory that the detection happens long after +the real problem, the un-initialized value being passed along through the code +for a long time before being detected. This makes debugging the problem slow. + +By adding in strategic places code that asserts that a specific value must be +initialised, it is possible to detect problems earlier, speeding up debugging. +Such code can be added in more places over time as development and debugging +goes on. DESCRIPTION: Valgrind (the memcheck tool) has some very useful APIs that can be used in mysqld when testing with Valgrind to improve testing and/or debugging: file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.clientreqs file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.mempools This worklog is about adding configure checks and headers to allow to use these in a way that continues to work on machines where the Valgrind headers or functionality is missing. It also includes adding some basic Valgrind enhancements: - Adding Valgrind annotations to custom memory allocators so that Valgrind can detect leaks, use-before-init, and use-after-free problems also for these allocators. - Adding checks for definedness in appropriate places (eg. when calling libz). HIGH-LEVEL SPECIFICATION: With custom memory allocators, using the Valgrind APIs we can tell Valgrind when a memory block is allocated (so that data read from memory is marked as undefined instead of being defined or not at random depending on prior use); and when a memory block is freed (so that use after freeing can be reported as an error). In some cases cheking for leaks may also be appropriate. Another possibility is to add an explicit check for whether memory is defined. One place this would be useful is when calling libz. Due to the design of that library, Valgrind produces lots of false alarms about using undefined values (I think the issue is that it runs a few bytes off of initialized memory to reduce boundary checks in each loop iteration, then after the loop has checks to avoid using the undefined part of the result). This means we have lots of libz Valgrind suppressions and continue to add more as new warnings surface. So we might easily miss a real problem in this area. This could be improved by adding explicit checks at the call to libz functions that the passed memory is properly defined. Another use is to improve debugging. It is often the case when debugging a warning about using un-initialised memory that the detection happens long after the real problem, the un-initialized value being passed along through the code for a long time before being detected. This makes debugging the problem slow. By adding in strategic places code that asserts that a specific value must be initialised, it is possible to detect problems earlier, speeding up debugging. Such code can be added in more places over time as development and debugging goes on. See also a patch here: http://bugs.mysql.com/bug.php?id=44582 LOW-LEVEL DESIGN: Two places where we call into libz, and where checking for defined parameters would be good: - mysys/my_compress.c - sql/item_strfunc.cc (Item_func_compress). Another good place is in the TRASH_MEM macro. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): Using the Valgrind API in mysqld (23)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Using the Valgrind API in mysqld CREATION DATE..: Fri, 22 May 2009, 11:43 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 23 (http://askmonty.org/worklog/?tid=23) VERSION........: Server-5.1 STATUS.........: Code-Review PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: -=-=(Knielsen - Mon, 10 Aug 2009, 14:27)=-=- Low Level Design modified. --- /tmp/wklog.23.old.16018 2009-08-10 14:27:09.000000000 +0300 +++ /tmp/wklog.23.new.16018 2009-08-10 14:27:09.000000000 +0300 @@ -5,3 +5,5 @@ - sql/item_strfunc.cc (Item_func_compress). +Another good place is in the TRASH_MEM macro. + -=-=(Knielsen - Wed, 24 Jun 2009, 15:55)=-=- Supervisor updated. --- /tmp/wklog.23.old.944 2009-06-24 15:55:57.000000000 +0300 +++ /tmp/wklog.23.new.944 2009-06-24 15:55:57.000000000 +0300 @@ -1 +1 @@ -Knielsen +Monty -=-=(Knielsen - Wed, 24 Jun 2009, 15:53)=-=- Version updated. --- /tmp/wklog.23.old.911 2009-06-24 15:53:32.000000000 +0300 +++ /tmp/wklog.23.new.911 2009-06-24 15:53:32.000000000 +0300 @@ -1 +1 @@ -Maria-1.0 +Server-5.1 -=-=(Knielsen - Wed, 24 Jun 2009, 15:52)=-=- Version updated. --- /tmp/wklog.23.old.897 2009-06-24 15:52:43.000000000 +0300 +++ /tmp/wklog.23.new.897 2009-06-24 15:52:43.000000000 +0300 @@ -1 +1 @@ -Connector/.NET-2.1 +Maria-1.0 -=-=(Knielsen - Wed, 24 Jun 2009, 15:52)=-=- Version updated. --- /tmp/wklog.23.old.895 2009-06-24 15:52:28.000000000 +0300 +++ /tmp/wklog.23.new.895 2009-06-24 15:52:28.000000000 +0300 @@ -1 +1 @@ -Maria-1.0 +Connector/.NET-2.1 -=-=(Knielsen - Wed, 24 Jun 2009, 15:35)=-=- Version updated. --- /tmp/wklog.23.old.32742 2009-06-24 15:35:48.000000000 +0300 +++ /tmp/wklog.23.new.32742 2009-06-24 15:35:48.000000000 +0300 @@ -1 +1 @@ -Server-5.1 +Maria-1.0 -=-=(Knielsen - Fri, 22 May 2009, 14:31)=-=- Low Level Design modified. --- /tmp/wklog.23.old.24587 2009-05-22 14:31:52.000000000 +0300 +++ /tmp/wklog.23.new.24587 2009-05-22 14:31:52.000000000 +0300 @@ -1 +1,7 @@ +Two places where we call into libz, and where checking for defined parameters +would be good: + + - mysys/my_compress.c + + - sql/item_strfunc.cc (Item_func_compress). -=-=(Guest - Fri, 22 May 2009, 12:04)=-=- High-Level Specification modified. --- /tmp/wklog.23.old.18061 2009-05-22 12:04:05.000000000 +0300 +++ /tmp/wklog.23.new.18061 2009-05-22 12:04:05.000000000 +0300 @@ -26,3 +26,5 @@ initialised, it is possible to detect problems earlier, speeding up debugging. Such code can be added in more places over time as development and debugging goes on. + +See also a patch here: http://bugs.mysql.com/bug.php?id=44582 -=-=(Knielsen - Fri, 22 May 2009, 11:52)=-=- High-Level Specification modified. --- /tmp/wklog.23.old.17628 2009-05-22 11:52:33.000000000 +0300 +++ /tmp/wklog.23.new.17628 2009-05-22 11:52:33.000000000 +0300 @@ -1 +1,28 @@ +With custom memory allocators, using the Valgrind APIs we can tell Valgrind when +a memory block is allocated (so that data read from memory is marked as undefined +instead of being defined or not at random depending on prior use); and when a +memory block is freed (so that use after freeing can be reported as an error). +In some cases cheking for leaks may also be appropriate. + +Another possibility is to add an explicit check for whether memory is defined. + +One place this would be useful is when calling libz. Due to the design of that +library, Valgrind produces lots of false alarms about using undefined values +(I think the issue is that it runs a few bytes off of initialized memory to +reduce boundary checks in each loop iteration, then after the loop has checks to +avoid using the undefined part of the result). This means we have lots of libz +Valgrind suppressions and continue to add more as new warnings surface. So we +might easily miss a real problem in this area. This could be improved by adding +explicit checks at the call to libz functions that the passed memory is properly +defined. + +Another use is to improve debugging. It is often the case when debugging a +warning about using un-initialised memory that the detection happens long after +the real problem, the un-initialized value being passed along through the code +for a long time before being detected. This makes debugging the problem slow. + +By adding in strategic places code that asserts that a specific value must be +initialised, it is possible to detect problems earlier, speeding up debugging. +Such code can be added in more places over time as development and debugging +goes on. DESCRIPTION: Valgrind (the memcheck tool) has some very useful APIs that can be used in mysqld when testing with Valgrind to improve testing and/or debugging: file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.clientreqs file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.mempools This worklog is about adding configure checks and headers to allow to use these in a way that continues to work on machines where the Valgrind headers or functionality is missing. It also includes adding some basic Valgrind enhancements: - Adding Valgrind annotations to custom memory allocators so that Valgrind can detect leaks, use-before-init, and use-after-free problems also for these allocators. - Adding checks for definedness in appropriate places (eg. when calling libz). HIGH-LEVEL SPECIFICATION: With custom memory allocators, using the Valgrind APIs we can tell Valgrind when a memory block is allocated (so that data read from memory is marked as undefined instead of being defined or not at random depending on prior use); and when a memory block is freed (so that use after freeing can be reported as an error). In some cases cheking for leaks may also be appropriate. Another possibility is to add an explicit check for whether memory is defined. One place this would be useful is when calling libz. Due to the design of that library, Valgrind produces lots of false alarms about using undefined values (I think the issue is that it runs a few bytes off of initialized memory to reduce boundary checks in each loop iteration, then after the loop has checks to avoid using the undefined part of the result). This means we have lots of libz Valgrind suppressions and continue to add more as new warnings surface. So we might easily miss a real problem in this area. This could be improved by adding explicit checks at the call to libz functions that the passed memory is properly defined. Another use is to improve debugging. It is often the case when debugging a warning about using un-initialised memory that the detection happens long after the real problem, the un-initialized value being passed along through the code for a long time before being detected. This makes debugging the problem slow. By adding in strategic places code that asserts that a specific value must be initialised, it is possible to detect problems earlier, speeding up debugging. Such code can be added in more places over time as development and debugging goes on. See also a patch here: http://bugs.mysql.com/bug.php?id=44582 LOW-LEVEL DESIGN: Two places where we call into libz, and where checking for defined parameters would be good: - mysys/my_compress.c - sql/item_strfunc.cc (Item_func_compress). Another good place is in the TRASH_MEM macro. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Add a mysqlbinlog option to filter updates to certain tables (40)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to filter updates to certain tables CREATION DATE..: Mon, 10 Aug 2009, 13:25 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Psergey CATEGORY.......: Server-RawIdeaBin TASK ID........: 40 (http://askmonty.org/worklog/?tid=40) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Replication slave can be set to filter updates to certain tables with --replicate-[wild-]{do,ignore}-table options. This task is about adding similar functionality to mysqlbinlog. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Add a mysqlbinlog option to filter updates to certain tables (40)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to filter updates to certain tables CREATION DATE..: Mon, 10 Aug 2009, 13:25 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Psergey CATEGORY.......: Server-RawIdeaBin TASK ID........: 40 (http://askmonty.org/worklog/?tid=40) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Replication slave can be set to filter updates to certain tables with --replicate-[wild-]{do,ignore}-table options. This task is about adding similar functionality to mysqlbinlog. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Guest): Add a mysqlbinlog option to change the used database (36)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to change the used database CREATION DATE..: Fri, 07 Aug 2009, 14:57 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 36 (http://askmonty.org/worklog/?tid=36) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Guest - Mon, 10 Aug 2009, 11:12)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.6580 2009-08-10 11:12:36.000000000 +0300 +++ /tmp/wklog.36.new.6580 2009-08-10 11:12:36.000000000 +0300 @@ -1,4 +1,3 @@ - Context ------- At the moment, the server has a replication slave option @@ -67,6 +66,6 @@ It will be possible to do the rewrites either on the slave ( --replicate-rewrite-db will work for all kinds of statements), or in -mysqlbinlog (adding a comment is easy and doesn't require use to parse the -statement). +mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to +parse the statement). -=-=(Psergey - Sun, 09 Aug 2009, 23:53)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.13425 2009-08-09 23:53:54.000000000 +0300 +++ /tmp/wklog.36.new.13425 2009-08-09 23:53:54.000000000 +0300 @@ -1 +1,72 @@ +Context +------- +At the moment, the server has a replication slave option + + --replicate-rewrite-db="from->to" + +the option affects +- Table_map_log_event (all RBR events) +- Load_log_event (LOAD DATA) +- Query_log_event (SBR-based updates, with the usual assumption that the + statement refers to tables in current database, so that changing the current + database will make the statement to work on a table in a different database). + +What we could do +---------------- + +Option1: make mysqlbinlog accept --replicate-rewrite-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make mysqlbinlog accept --replicate-rewrite-db options and process them to the +same extent as replication slave would process --replicate-rewrite-db option. + + +Option2: Add database-agnostic RBR events and --strip-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Right now RBR events require a databasename. It is not possible to have RBR +event stream that won't mention which database the events are for. When I +tried to use debugger and specify empty database name, attempt to apply the +binlog resulted in this error: + +090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on +opening tables, + +We could do as follows: +- Make the server interpret empty database name in RBR event (i.e. in a + Table_map_log_event) as "use current database". Binlog slave thread + probably should not allow such events as it doesn't have a natural current + database. +- Add a mysqlbinlog --strip-db option that would + = not produce any "USE dbname" statements + = change databasename for all RBR events to be empty + +That way, mysqlbinlog output will be database-agnostic and apply to the +current database. +(this will have the usual limitations that we assume that all statements in +the binlog refer to the current database). + +Option3: Enhance database rewrite +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +If there is a need to support database change for statements that use +dbname.tablename notation and are replicated as statements (i.e. are DDL +statements and/or DML statements that are binlogged as statements), +then that could be supported as follows: + +- Make the server's parser recognize special form of comments + + /* !database-alias(oldname,newname) */ + + and save the mapping somewhere + +- Put the hooks in table open and name resolution code to use the saved + mapping. + + +Once we've done the above, it will be easy to perform a complete, +no-compromise or restrictions database name change in binary log. + +It will be possible to do the rewrites either on the slave ( +--replicate-rewrite-db will work for all kinds of statements), or in +mysqlbinlog (adding a comment is easy and doesn't require use to parse the +statement). + -=-=(Psergey - Sun, 09 Aug 2009, 12:27)=-=- Dependency created: 39 now depends on 36 -=-=(Psergey - Fri, 07 Aug 2009, 14:57)=-=- Title modified. --- /tmp/wklog.36.old.14687 2009-08-07 14:57:49.000000000 +0300 +++ /tmp/wklog.36.new.14687 2009-08-07 14:57:49.000000000 +0300 @@ -1 +1 @@ -Add a mysqlbinlog option to change the database +Add a mysqlbinlog option to change the used database DESCRIPTION: Sometimes there is a need to take a binary log and apply it to a database with a different name than the original name of the database on binlog producer. If one is using statement-based replication, he can achieve this by grepping out "USE dbname" statements out of the output of mysqlbinlog(*). With row-based replication this is no longer possible, as database name is encoded within the the BINLOG '....' statement. This task is about adding an option to mysqlbinlog that would allow to change the names of used databases in both RBR and SBR events. (*) this implies that all statements refer to tables in the current database, doesn't catch updates made inside stored functions and so forth, but still works for a practially-important subset of cases. HIGH-LEVEL SPECIFICATION: Context ------- At the moment, the server has a replication slave option --replicate-rewrite-db="from->to" the option affects - Table_map_log_event (all RBR events) - Load_log_event (LOAD DATA) - Query_log_event (SBR-based updates, with the usual assumption that the statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). What we could do ---------------- Option1: make mysqlbinlog accept --replicate-rewrite-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Make mysqlbinlog accept --replicate-rewrite-db options and process them to the same extent as replication slave would process --replicate-rewrite-db option. Option2: Add database-agnostic RBR events and --strip-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Right now RBR events require a databasename. It is not possible to have RBR event stream that won't mention which database the events are for. When I tried to use debugger and specify empty database name, attempt to apply the binlog resulted in this error: 090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on opening tables, We could do as follows: - Make the server interpret empty database name in RBR event (i.e. in a Table_map_log_event) as "use current database". Binlog slave thread probably should not allow such events as it doesn't have a natural current database. - Add a mysqlbinlog --strip-db option that would = not produce any "USE dbname" statements = change databasename for all RBR events to be empty That way, mysqlbinlog output will be database-agnostic and apply to the current database. (this will have the usual limitations that we assume that all statements in the binlog refer to the current database). Option3: Enhance database rewrite ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If there is a need to support database change for statements that use dbname.tablename notation and are replicated as statements (i.e. are DDL statements and/or DML statements that are binlogged as statements), then that could be supported as follows: - Make the server's parser recognize special form of comments /* !database-alias(oldname,newname) */ and save the mapping somewhere - Put the hooks in table open and name resolution code to use the saved mapping. Once we've done the above, it will be easy to perform a complete, no-compromise or restrictions database name change in binary log. It will be possible to do the rewrites either on the slave ( --replicate-rewrite-db will work for all kinds of statements), or in mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to parse the statement). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Guest): Add a mysqlbinlog option to change the used database (36)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to change the used database CREATION DATE..: Fri, 07 Aug 2009, 14:57 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 36 (http://askmonty.org/worklog/?tid=36) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Guest - Mon, 10 Aug 2009, 11:12)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.6580 2009-08-10 11:12:36.000000000 +0300 +++ /tmp/wklog.36.new.6580 2009-08-10 11:12:36.000000000 +0300 @@ -1,4 +1,3 @@ - Context ------- At the moment, the server has a replication slave option @@ -67,6 +66,6 @@ It will be possible to do the rewrites either on the slave ( --replicate-rewrite-db will work for all kinds of statements), or in -mysqlbinlog (adding a comment is easy and doesn't require use to parse the -statement). +mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to +parse the statement). -=-=(Psergey - Sun, 09 Aug 2009, 23:53)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.13425 2009-08-09 23:53:54.000000000 +0300 +++ /tmp/wklog.36.new.13425 2009-08-09 23:53:54.000000000 +0300 @@ -1 +1,72 @@ +Context +------- +At the moment, the server has a replication slave option + + --replicate-rewrite-db="from->to" + +the option affects +- Table_map_log_event (all RBR events) +- Load_log_event (LOAD DATA) +- Query_log_event (SBR-based updates, with the usual assumption that the + statement refers to tables in current database, so that changing the current + database will make the statement to work on a table in a different database). + +What we could do +---------------- + +Option1: make mysqlbinlog accept --replicate-rewrite-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make mysqlbinlog accept --replicate-rewrite-db options and process them to the +same extent as replication slave would process --replicate-rewrite-db option. + + +Option2: Add database-agnostic RBR events and --strip-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Right now RBR events require a databasename. It is not possible to have RBR +event stream that won't mention which database the events are for. When I +tried to use debugger and specify empty database name, attempt to apply the +binlog resulted in this error: + +090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on +opening tables, + +We could do as follows: +- Make the server interpret empty database name in RBR event (i.e. in a + Table_map_log_event) as "use current database". Binlog slave thread + probably should not allow such events as it doesn't have a natural current + database. +- Add a mysqlbinlog --strip-db option that would + = not produce any "USE dbname" statements + = change databasename for all RBR events to be empty + +That way, mysqlbinlog output will be database-agnostic and apply to the +current database. +(this will have the usual limitations that we assume that all statements in +the binlog refer to the current database). + +Option3: Enhance database rewrite +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +If there is a need to support database change for statements that use +dbname.tablename notation and are replicated as statements (i.e. are DDL +statements and/or DML statements that are binlogged as statements), +then that could be supported as follows: + +- Make the server's parser recognize special form of comments + + /* !database-alias(oldname,newname) */ + + and save the mapping somewhere + +- Put the hooks in table open and name resolution code to use the saved + mapping. + + +Once we've done the above, it will be easy to perform a complete, +no-compromise or restrictions database name change in binary log. + +It will be possible to do the rewrites either on the slave ( +--replicate-rewrite-db will work for all kinds of statements), or in +mysqlbinlog (adding a comment is easy and doesn't require use to parse the +statement). + -=-=(Psergey - Sun, 09 Aug 2009, 12:27)=-=- Dependency created: 39 now depends on 36 -=-=(Psergey - Fri, 07 Aug 2009, 14:57)=-=- Title modified. --- /tmp/wklog.36.old.14687 2009-08-07 14:57:49.000000000 +0300 +++ /tmp/wklog.36.new.14687 2009-08-07 14:57:49.000000000 +0300 @@ -1 +1 @@ -Add a mysqlbinlog option to change the database +Add a mysqlbinlog option to change the used database DESCRIPTION: Sometimes there is a need to take a binary log and apply it to a database with a different name than the original name of the database on binlog producer. If one is using statement-based replication, he can achieve this by grepping out "USE dbname" statements out of the output of mysqlbinlog(*). With row-based replication this is no longer possible, as database name is encoded within the the BINLOG '....' statement. This task is about adding an option to mysqlbinlog that would allow to change the names of used databases in both RBR and SBR events. (*) this implies that all statements refer to tables in the current database, doesn't catch updates made inside stored functions and so forth, but still works for a practially-important subset of cases. HIGH-LEVEL SPECIFICATION: Context ------- At the moment, the server has a replication slave option --replicate-rewrite-db="from->to" the option affects - Table_map_log_event (all RBR events) - Load_log_event (LOAD DATA) - Query_log_event (SBR-based updates, with the usual assumption that the statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). What we could do ---------------- Option1: make mysqlbinlog accept --replicate-rewrite-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Make mysqlbinlog accept --replicate-rewrite-db options and process them to the same extent as replication slave would process --replicate-rewrite-db option. Option2: Add database-agnostic RBR events and --strip-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Right now RBR events require a databasename. It is not possible to have RBR event stream that won't mention which database the events are for. When I tried to use debugger and specify empty database name, attempt to apply the binlog resulted in this error: 090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on opening tables, We could do as follows: - Make the server interpret empty database name in RBR event (i.e. in a Table_map_log_event) as "use current database". Binlog slave thread probably should not allow such events as it doesn't have a natural current database. - Add a mysqlbinlog --strip-db option that would = not produce any "USE dbname" statements = change databasename for all RBR events to be empty That way, mysqlbinlog output will be database-agnostic and apply to the current database. (this will have the usual limitations that we assume that all statements in the binlog refer to the current database). Option3: Enhance database rewrite ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If there is a need to support database change for statements that use dbname.tablename notation and are replicated as statements (i.e. are DDL statements and/or DML statements that are binlogged as statements), then that could be supported as follows: - Make the server's parser recognize special form of comments /* !database-alias(oldname,newname) */ and save the mapping somewhere - Put the hooks in table open and name resolution code to use the saved mapping. Once we've done the above, it will be easy to perform a complete, no-compromise or restrictions database name change in binary log. It will be possible to do the rewrites either on the slave ( --replicate-rewrite-db will work for all kinds of statements), or in mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to parse the statement). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Psergey): Add a mysqlbinlog option to change the used database (36)
by worklog-noreply＠askmonty.org 09 Aug '09

09 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to change the used database CREATION DATE..: Fri, 07 Aug 2009, 14:57 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 36 (http://askmonty.org/worklog/?tid=36) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Sun, 09 Aug 2009, 23:53)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.13425 2009-08-09 23:53:54.000000000 +0300 +++ /tmp/wklog.36.new.13425 2009-08-09 23:53:54.000000000 +0300 @@ -1 +1,72 @@ +Context +------- +At the moment, the server has a replication slave option + + --replicate-rewrite-db="from->to" + +the option affects +- Table_map_log_event (all RBR events) +- Load_log_event (LOAD DATA) +- Query_log_event (SBR-based updates, with the usual assumption that the + statement refers to tables in current database, so that changing the current + database will make the statement to work on a table in a different database). + +What we could do +---------------- + +Option1: make mysqlbinlog accept --replicate-rewrite-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make mysqlbinlog accept --replicate-rewrite-db options and process them to the +same extent as replication slave would process --replicate-rewrite-db option. + + +Option2: Add database-agnostic RBR events and --strip-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Right now RBR events require a databasename. It is not possible to have RBR +event stream that won't mention which database the events are for. When I +tried to use debugger and specify empty database name, attempt to apply the +binlog resulted in this error: + +090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on +opening tables, + +We could do as follows: +- Make the server interpret empty database name in RBR event (i.e. in a + Table_map_log_event) as "use current database". Binlog slave thread + probably should not allow such events as it doesn't have a natural current + database. +- Add a mysqlbinlog --strip-db option that would + = not produce any "USE dbname" statements + = change databasename for all RBR events to be empty + +That way, mysqlbinlog output will be database-agnostic and apply to the +current database. +(this will have the usual limitations that we assume that all statements in +the binlog refer to the current database). + +Option3: Enhance database rewrite +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +If there is a need to support database change for statements that use +dbname.tablename notation and are replicated as statements (i.e. are DDL +statements and/or DML statements that are binlogged as statements), +then that could be supported as follows: + +- Make the server's parser recognize special form of comments + + /* !database-alias(oldname,newname) */ + + and save the mapping somewhere + +- Put the hooks in table open and name resolution code to use the saved + mapping. + + +Once we've done the above, it will be easy to perform a complete, +no-compromise or restrictions database name change in binary log. + +It will be possible to do the rewrites either on the slave ( +--replicate-rewrite-db will work for all kinds of statements), or in +mysqlbinlog (adding a comment is easy and doesn't require use to parse the +statement). + -=-=(Psergey - Sun, 09 Aug 2009, 12:27)=-=- Dependency created: 39 now depends on 36 -=-=(Psergey - Fri, 07 Aug 2009, 14:57)=-=- Title modified. --- /tmp/wklog.36.old.14687 2009-08-07 14:57:49.000000000 +0300 +++ /tmp/wklog.36.new.14687 2009-08-07 14:57:49.000000000 +0300 @@ -1 +1 @@ -Add a mysqlbinlog option to change the database +Add a mysqlbinlog option to change the used database DESCRIPTION: Sometimes there is a need to take a binary log and apply it to a database with a different name than the original name of the database on binlog producer. If one is using statement-based replication, he can achieve this by grepping out "USE dbname" statements out of the output of mysqlbinlog(*). With row-based replication this is no longer possible, as database name is encoded within the the BINLOG '....' statement. This task is about adding an option to mysqlbinlog that would allow to change the names of used databases in both RBR and SBR events. (*) this implies that all statements refer to tables in the current database, doesn't catch updates made inside stored functions and so forth, but still works for a practially-important subset of cases. HIGH-LEVEL SPECIFICATION: Context ------- At the moment, the server has a replication slave option --replicate-rewrite-db="from->to" the option affects - Table_map_log_event (all RBR events) - Load_log_event (LOAD DATA) - Query_log_event (SBR-based updates, with the usual assumption that the statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). What we could do ---------------- Option1: make mysqlbinlog accept --replicate-rewrite-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Make mysqlbinlog accept --replicate-rewrite-db options and process them to the same extent as replication slave would process --replicate-rewrite-db option. Option2: Add database-agnostic RBR events and --strip-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Right now RBR events require a databasename. It is not possible to have RBR event stream that won't mention which database the events are for. When I tried to use debugger and specify empty database name, attempt to apply the binlog resulted in this error: 090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on opening tables, We could do as follows: - Make the server interpret empty database name in RBR event (i.e. in a Table_map_log_event) as "use current database". Binlog slave thread probably should not allow such events as it doesn't have a natural current database. - Add a mysqlbinlog --strip-db option that would = not produce any "USE dbname" statements = change databasename for all RBR events to be empty That way, mysqlbinlog output will be database-agnostic and apply to the current database. (this will have the usual limitations that we assume that all statements in the binlog refer to the current database). Option3: Enhance database rewrite ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If there is a need to support database change for statements that use dbname.tablename notation and are replicated as statements (i.e. are DDL statements and/or DML statements that are binlogged as statements), then that could be supported as follows: - Make the server's parser recognize special form of comments /* !database-alias(oldname,newname) */ and save the mapping somewhere - Put the hooks in table open and name resolution code to use the saved mapping. Once we've done the above, it will be easy to perform a complete, no-compromise or restrictions database name change in binary log. It will be possible to do the rewrites either on the slave ( --replicate-rewrite-db will work for all kinds of statements), or in mysqlbinlog (adding a comment is easy and doesn't require use to parse the statement). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Psergey): Add a mysqlbinlog option to change the used database (36)
by worklog-noreply＠askmonty.org 09 Aug '09

09 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to change the used database CREATION DATE..: Fri, 07 Aug 2009, 14:57 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 36 (http://askmonty.org/worklog/?tid=36) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Sun, 09 Aug 2009, 23:53)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.13425 2009-08-09 23:53:54.000000000 +0300 +++ /tmp/wklog.36.new.13425 2009-08-09 23:53:54.000000000 +0300 @@ -1 +1,72 @@ +Context +------- +At the moment, the server has a replication slave option + + --replicate-rewrite-db="from->to" + +the option affects +- Table_map_log_event (all RBR events) +- Load_log_event (LOAD DATA) +- Query_log_event (SBR-based updates, with the usual assumption that the + statement refers to tables in current database, so that changing the current + database will make the statement to work on a table in a different database). + +What we could do +---------------- + +Option1: make mysqlbinlog accept --replicate-rewrite-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make mysqlbinlog accept --replicate-rewrite-db options and process them to the +same extent as replication slave would process --replicate-rewrite-db option. + + +Option2: Add database-agnostic RBR events and --strip-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Right now RBR events require a databasename. It is not possible to have RBR +event stream that won't mention which database the events are for. When I +tried to use debugger and specify empty database name, attempt to apply the +binlog resulted in this error: + +090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on +opening tables, + +We could do as follows: +- Make the server interpret empty database name in RBR event (i.e. in a + Table_map_log_event) as "use current database". Binlog slave thread + probably should not allow such events as it doesn't have a natural current + database. +- Add a mysqlbinlog --strip-db option that would + = not produce any "USE dbname" statements + = change databasename for all RBR events to be empty + +That way, mysqlbinlog output will be database-agnostic and apply to the +current database. +(this will have the usual limitations that we assume that all statements in +the binlog refer to the current database). + +Option3: Enhance database rewrite +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +If there is a need to support database change for statements that use +dbname.tablename notation and are replicated as statements (i.e. are DDL +statements and/or DML statements that are binlogged as statements), +then that could be supported as follows: + +- Make the server's parser recognize special form of comments + + /* !database-alias(oldname,newname) */ + + and save the mapping somewhere + +- Put the hooks in table open and name resolution code to use the saved + mapping. + + +Once we've done the above, it will be easy to perform a complete, +no-compromise or restrictions database name change in binary log. + +It will be possible to do the rewrites either on the slave ( +--replicate-rewrite-db will work for all kinds of statements), or in +mysqlbinlog (adding a comment is easy and doesn't require use to parse the +statement). + -=-=(Psergey - Sun, 09 Aug 2009, 12:27)=-=- Dependency created: 39 now depends on 36 -=-=(Psergey - Fri, 07 Aug 2009, 14:57)=-=- Title modified. --- /tmp/wklog.36.old.14687 2009-08-07 14:57:49.000000000 +0300 +++ /tmp/wklog.36.new.14687 2009-08-07 14:57:49.000000000 +0300 @@ -1 +1 @@ -Add a mysqlbinlog option to change the database +Add a mysqlbinlog option to change the used database DESCRIPTION: Sometimes there is a need to take a binary log and apply it to a database with a different name than the original name of the database on binlog producer. If one is using statement-based replication, he can achieve this by grepping out "USE dbname" statements out of the output of mysqlbinlog(*). With row-based replication this is no longer possible, as database name is encoded within the the BINLOG '....' statement. This task is about adding an option to mysqlbinlog that would allow to change the names of used databases in both RBR and SBR events. (*) this implies that all statements refer to tables in the current database, doesn't catch updates made inside stored functions and so forth, but still works for a practially-important subset of cases. HIGH-LEVEL SPECIFICATION: Context ------- At the moment, the server has a replication slave option --replicate-rewrite-db="from->to" the option affects - Table_map_log_event (all RBR events) - Load_log_event (LOAD DATA) - Query_log_event (SBR-based updates, with the usual assumption that the statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). What we could do ---------------- Option1: make mysqlbinlog accept --replicate-rewrite-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Make mysqlbinlog accept --replicate-rewrite-db options and process them to the same extent as replication slave would process --replicate-rewrite-db option. Option2: Add database-agnostic RBR events and --strip-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Right now RBR events require a databasename. It is not possible to have RBR event stream that won't mention which database the events are for. When I tried to use debugger and specify empty database name, attempt to apply the binlog resulted in this error: 090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on opening tables, We could do as follows: - Make the server interpret empty database name in RBR event (i.e. in a Table_map_log_event) as "use current database". Binlog slave thread probably should not allow such events as it doesn't have a natural current database. - Add a mysqlbinlog --strip-db option that would = not produce any "USE dbname" statements = change databasename for all RBR events to be empty That way, mysqlbinlog output will be database-agnostic and apply to the current database. (this will have the usual limitations that we assume that all statements in the binlog refer to the current database). Option3: Enhance database rewrite ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If there is a need to support database change for statements that use dbname.tablename notation and are replicated as statements (i.e. are DDL statements and/or DML statements that are binlogged as statements), then that could be supported as follows: - Make the server's parser recognize special form of comments /* !database-alias(oldname,newname) */ and save the mapping somewhere - Put the hooks in table open and name resolution code to use the saved mapping. Once we've done the above, it will be easy to perform a complete, no-compromise or restrictions database name change in binary log. It will be possible to do the rewrites either on the slave ( --replicate-rewrite-db will work for all kinds of statements), or in mysqlbinlog (adding a comment is easy and doesn't require use to parse the statement). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Psergey): Add an option to mysqlbinlog to produce SQL script with fewer roundtrips (37)
by worklog-noreply＠askmonty.org 09 Aug '09

09 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add an option to mysqlbinlog to produce SQL script with fewer roundtrips CREATION DATE..: Fri, 07 Aug 2009, 17:14 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 37 (http://askmonty.org/worklog/?tid=37) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Sun, 09 Aug 2009, 12:56)=-=- High-Level Specification modified. --- /tmp/wklog.37.old.22083 2009-08-09 12:56:36.000000000 +0300 +++ /tmp/wklog.37.new.22083 2009-08-09 12:56:36.000000000 +0300 @@ -11,3 +11,16 @@ if (my_b_tell(&cache) != 0) my_b_write(&cache,";;",2); +Note: mysqlbinlog already uses + + DELIMITER /*!*/; + +so that it can process "multi-statements" like + + CREATE PROCEDURE ... BEGIN stmt1; stmt2; ... END + +what remains to be done is to print the /*!*/; only when we're about to exceed +$args[combine-statements] bytes. In all other cases, delimit statements with +regular semicolon. + + -=-=(Psergey - Sun, 09 Aug 2009, 12:30)=-=- High Level Description modified. --- /tmp/wklog.37.old.21090 2009-08-09 12:30:26.000000000 +0300 +++ /tmp/wklog.37.new.21090 2009-08-09 12:30:26.000000000 +0300 @@ -1,6 +1,6 @@ SQL scripts generated by mysqlbinlog can be slow to load because they have many small queries, hence applying the script against a remote server requires a lot -of roundtrips, and they become a bottleneck. +of roundtrips, and the network roundtrips become the bottleneck. This bottleneck can be addressed by having mysqlbinlog combine multiple statements into one: @@ -14,7 +14,7 @@ loading such sql script will require fewer roundtrips. -The behavior can be controlled using a command line option +The behaviour can be controlled using a command line option mysqlbinlog --combine-statements=# -=-=(Psergey - Sun, 09 Aug 2009, 12:24)=-=- Dependency created: 39 now depends on 37 -=-=(Psergey - Fri, 07 Aug 2009, 17:16)=-=- High-Level Specification modified. --- /tmp/wklog.37.old.20454 2009-08-07 17:16:54.000000000 +0300 +++ /tmp/wklog.37.new.20454 2009-08-07 17:16:54.000000000 +0300 @@ -1 +1,13 @@ +Implementation overview: + +- At start, print "--delimiter=;;" +- Modify the start of each print functions as follows + + if (my_b_tell(&cache) - my_start_of_combine_statement) + + estimiated_size_of_log_event) > combine_statement_size) + my_b_write(&cache,";;",2); + +- And we should end mysqlbinlog with; + if (my_b_tell(&cache) != 0) + my_b_write(&cache,";;",2); DESCRIPTION: SQL scripts generated by mysqlbinlog can be slow to load because they have many small queries, hence applying the script against a remote server requires a lot of roundtrips, and the network roundtrips become the bottleneck. This bottleneck can be addressed by having mysqlbinlog combine multiple statements into one: +delimiter // binlog statement1; binlog statement2; binlog statement3; +// binlog statement4; loading such sql script will require fewer roundtrips. The behaviour can be controlled using a command line option mysqlbinlog --combine-statements=# Where the # is maximum allowed packet length. HIGH-LEVEL SPECIFICATION: Implementation overview: - At start, print "--delimiter=;;" - Modify the start of each print functions as follows if (my_b_tell(&cache) - my_start_of_combine_statement) + estimiated_size_of_log_event) > combine_statement_size) my_b_write(&cache,";;",2); - And we should end mysqlbinlog with; if (my_b_tell(&cache) != 0) my_b_write(&cache,";;",2); Note: mysqlbinlog already uses DELIMITER /*!*/; so that it can process "multi-statements" like CREATE PROCEDURE ... BEGIN stmt1; stmt2; ... END what remains to be done is to print the /*!*/; only when we're about to exceed $args[combine-statements] bytes. In all other cases, delimit statements with regular semicolon. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0