March 2010 - developers - lists.mariadb.org

[Maria-developers] Updated (by Knielsen): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 29 Mar '10

29 Mar '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen, Serg CATEGORY.......: Server-Sprint TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: Code-Review PRIORITY.......: 60 WORKED HOURS...: 40 ESTIMATE.......: 15 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Knielsen - Mon, 29 Mar 2010, 10:59)=-=- Status updated. --- /tmp/wklog.47.old.27790 2010-03-29 10:59:53.000000000 +0000 +++ /tmp/wklog.47.new.27790 2010-03-29 10:59:53.000000000 +0000 @@ -1 +1 @@ -In-Progress +Code-Review -=-=(Alexi - Thu, 18 Feb 2010, 19:29)=-=- Worked 20 hours (alexi) Worked 20 hours and estimate 15 hours remain (original estimate unchanged). -=-=(Serg - Fri, 05 Feb 2010, 14:04)=-=- Observers changed: Knielsen,Serg -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Category updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Status updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +In-Progress -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { ------------------------------------------------------------ -=-=(View All Progress Notes, 25 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 29 Mar '10

29 Mar '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen, Serg CATEGORY.......: Server-Sprint TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: Code-Review PRIORITY.......: 60 WORKED HOURS...: 40 ESTIMATE.......: 15 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Knielsen - Mon, 29 Mar 2010, 10:59)=-=- Status updated. --- /tmp/wklog.47.old.27790 2010-03-29 10:59:53.000000000 +0000 +++ /tmp/wklog.47.new.27790 2010-03-29 10:59:53.000000000 +0000 @@ -1 +1 @@ -In-Progress +Code-Review -=-=(Alexi - Thu, 18 Feb 2010, 19:29)=-=- Worked 20 hours (alexi) Worked 20 hours and estimate 15 hours remain (original estimate unchanged). -=-=(Serg - Fri, 05 Feb 2010, 14:04)=-=- Observers changed: Knielsen,Serg -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Category updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Status updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +In-Progress -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { ------------------------------------------------------------ -=-=(View All Progress Notes, 25 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Rev 29: Increased warmup time. in file:///Users/hakan/work/monty_program/mariadb-tools/
by Hakan Kuecuekyilmaz 29 Mar '10

29 Mar '10

At file:///Users/hakan/work/monty_program/mariadb-tools/ ------------------------------------------------------------ revno: 29 revision-id: hakan(a)askmonty.org-20100329105306-6hthpbelo9xop3kw parent: hakan(a)askmonty.org-20100325014205-kwsruwixwlymz1ti committer: Hakan Kuecuekyilmaz <hakan(a)askmonty.org> branch nick: mariadb-tools timestamp: Mon 2010-03-29 12:53:06 +0200 message: Increased warmup time. === modified file 'sysbench/run-sysbench-myisam.sh' --- a/sysbench/run-sysbench-myisam.sh 2010-03-25 01:42:05 +0000 +++ b/sysbench/run-sysbench-myisam.sh 2010-03-29 10:53:06 +0000 @@ -92,10 +92,10 @@ TABLE_SIZE=20000000 # The run time we use for sysbench. -RUN_TIME=900 +RUN_TIME=1800 # Warm up time we use for sysbench. -WARM_UP_TIME=180 +WARM_UP_TIME=300 # How many times we run each test. LOOP_COUNT=3 @@ -314,7 +314,7 @@ echo "[$(date "+%Y-%m-%d %H:%M:%S")] Running $SYSBENCH_TEST with $THREADS threads and $LOOP_COUNT iterations for $PRODUCT" | tee ${THIS_RESULT_DIR}/results.txt echo '' >> ${THIS_RESULT_DIR}/results.txt - SYSBENCH_OPTIONS_WARM_UP="${SYSBENCH_OPTIONS} --num-threads=1 --max-time=$WARM_UP_TIME" + SYSBENCH_OPTIONS_WARM_UP="${SYSBENCH_OPTIONS} --num-threads=3 --max-time=$WARM_UP_TIME" SYSBENCH_OPTIONS_RUN="${SYSBENCH_OPTIONS} --num-threads=$THREADS --max-time=$RUN_TIME" k=0 @@ -345,7 +345,7 @@ grep "write requests:" ${THIS_RESULT_DIR}/result${k}.txt | awk '{ print $4 }' | sed -e 's/(//' >> ${THIS_RESULT_DIR}/results.txt - echo 'SELECT * FROM INFORMATION_SCHEMA.KEY_CACHES' | $MYSQL -uroot > ${THIS_RESULT_DIR}/key_cache_stats{k}.txt + echo 'SELECT * FROM INFORMATION_SCHEMA.KEY_CACHES' | $MYSQL -uroot > ${THIS_RESULT_DIR}/key_cache_stats${k}.txt k=$(($k + 1)) done

1 0

[Maria-developers] Updated (by Psergey): Make EXPLAIN show where subquery predicates are in the WHERE clause (111)
by worklog-noreply＠askmonty.org 29 Mar '10

29 Mar '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Make EXPLAIN show where subquery predicates are in the WHERE clause CREATION DATE..: Mon, 29 Mar 2010, 07:07 SUPERVISOR.....: Psergey IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 111 (http://askmonty.org/worklog/?tid=111) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Mon, 29 Mar 2010, 08:23)=-=- High-Level Specification modified. --- /tmp/wklog.111.old.18396 2010-03-29 08:23:06.000000000 +0000 +++ /tmp/wklog.111.new.18396 2010-03-29 08:23:06.000000000 +0000 @@ -1 +1,9 @@ +* Implement Item::walk()-compatible method that calls a callback + function for each found subquery predicate. Enumeration should enter + condition/function parameters but do not descend into subquery + expressions. + +* Let select_describe() call the above method with a callback that appends + "subquery#n" for every encountered subquery. + -=-=(Psergey - Mon, 29 Mar 2010, 08:16)=-=- High Level Description modified. --- /tmp/wklog.111.old.18136 2010-03-29 08:16:36.000000000 +0000 +++ /tmp/wklog.111.new.18136 2010-03-29 08:16:36.000000000 +0000 @@ -30,8 +30,15 @@ to expressions being too long for current EXPLAIN output format. A simplified solution would be to show "Subquery#n" in Extra column if the -clause in "Using Where" has a subquery. +clause in "Using Where" has a subquery. That is, if table's condition has +subquery expressions, Extra column will look like this: + Using where; Subquery#n1; Subquery#n2,... + +When table's condition has only subquery expression, we could omit printing +"Using where" (note that this case is quite rare now. the subquery must be +correlated, non-semi-join subquery predicate that is an AND-part of the WHERE. +such cases are in minority) [1] http://forge.mysql.com/wiki/Image:NewSubqueryOptimizationsIn6_0_UC2008.pdf -=-=(Guest - Mon, 29 Mar 2010, 07:09)=-=- Supervisor updated. --- /tmp/wklog.111.old.13259 2010-03-29 07:09:55.000000000 +0000 +++ /tmp/wklog.111.new.13259 2010-03-29 07:09:55.000000000 +0000 @@ -1 +1 @@ -Bothorsen +Psergey -=-=(Guest - Mon, 29 Mar 2010, 07:09)=-=- Category updated. --- /tmp/wklog.111.old.13259 2010-03-29 07:09:55.000000000 +0000 +++ /tmp/wklog.111.new.13259 2010-03-29 07:09:55.000000000 +0000 @@ -1 +1 @@ -Client-BackLog +Server-RawIdeaBin -=-=(Guest - Mon, 29 Mar 2010, 07:09)=-=- Version updated. --- /tmp/wklog.111.old.13259 2010-03-29 07:09:55.000000000 +0000 +++ /tmp/wklog.111.new.13259 2010-03-29 07:09:55.000000000 +0000 @@ -1 +1 @@ -Benchmarks-3.0 +9.x -=-=(Guest - Mon, 29 Mar 2010, 07:09)=-=- Version updated. --- /tmp/wklog.111.old.13259 2010-03-29 07:09:55.000000000 +0000 +++ /tmp/wklog.111.new.13259 2010-03-29 07:09:55.000000000 +0000 @@ -1 +1 @@ -9.x +Server-5.3 DESCRIPTION: Current EXPLAIN does not show where the subquery predicate is attached to. For example, see [1], slide#5 "Straightforward subquery evaluation (contd)", or look here: MariaDB [test]> explain select * from ot1, ot2 where ot1.a=ot2.a and (ot2.a in (select it1.b from it1) or ot1.b<3); +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ | 1 | PRIMARY | ot1 | ALL | NULL | NULL | NULL | NULL | 10 | Using where | | 1 | PRIMARY | ot2 | ALL | NULL | NULL | NULL | NULL | 10 | Using where; Using join buffer | | 2 | SUBQUERY | it1 | ALL | NULL | NULL | NULL | NULL | 20 | | +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ 3 rows in set (0.01 sec) Here there are two "Using where", and it is not clear where the predicate is attached to. If one has sufficient knowledge, they could know that - "ot2.a in (select )" will be substituted for "ot1.a" (provided datatypes allow equality propagation) - For correlated subqueries, equality propagation will not affect outside references (so, if the subquery in the above example was correlated, it would have been attached to table ot2, not ot1). As one can see, the rules are quite complicated. The full solution would be to show expressions behind "Using Where", but that has additional complications due to expressions being too long for current EXPLAIN output format. A simplified solution would be to show "Subquery#n" in Extra column if the clause in "Using Where" has a subquery. That is, if table's condition has subquery expressions, Extra column will look like this: Using where; Subquery#n1; Subquery#n2,... When table's condition has only subquery expression, we could omit printing "Using where" (note that this case is quite rare now. the subquery must be correlated, non-semi-join subquery predicate that is an AND-part of the WHERE. such cases are in minority) [1] http://forge.mysql.com/wiki/Image:NewSubqueryOptimizationsIn6_0_UC2008.pdf HIGH-LEVEL SPECIFICATION: * Implement Item::walk()-compatible method that calls a callback function for each found subquery predicate. Enumeration should enter condition/function parameters but do not descend into subquery expressions. * Let select_describe() call the above method with a callback that appends "subquery#n" for every encountered subquery. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Psergey): Make EXPLAIN show where subquery predicates are in the WHERE clause (111)
by worklog-noreply＠askmonty.org 29 Mar '10

29 Mar '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Make EXPLAIN show where subquery predicates are in the WHERE clause CREATION DATE..: Mon, 29 Mar 2010, 07:07 SUPERVISOR.....: Psergey IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 111 (http://askmonty.org/worklog/?tid=111) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Mon, 29 Mar 2010, 08:16)=-=- High Level Description modified. --- /tmp/wklog.111.old.18136 2010-03-29 08:16:36.000000000 +0000 +++ /tmp/wklog.111.new.18136 2010-03-29 08:16:36.000000000 +0000 @@ -30,8 +30,15 @@ to expressions being too long for current EXPLAIN output format. A simplified solution would be to show "Subquery#n" in Extra column if the -clause in "Using Where" has a subquery. +clause in "Using Where" has a subquery. That is, if table's condition has +subquery expressions, Extra column will look like this: + Using where; Subquery#n1; Subquery#n2,... + +When table's condition has only subquery expression, we could omit printing +"Using where" (note that this case is quite rare now. the subquery must be +correlated, non-semi-join subquery predicate that is an AND-part of the WHERE. +such cases are in minority) [1] http://forge.mysql.com/wiki/Image:NewSubqueryOptimizationsIn6_0_UC2008.pdf -=-=(Guest - Mon, 29 Mar 2010, 07:09)=-=- Supervisor updated. --- /tmp/wklog.111.old.13259 2010-03-29 07:09:55.000000000 +0000 +++ /tmp/wklog.111.new.13259 2010-03-29 07:09:55.000000000 +0000 @@ -1 +1 @@ -Bothorsen +Psergey -=-=(Guest - Mon, 29 Mar 2010, 07:09)=-=- Category updated. --- /tmp/wklog.111.old.13259 2010-03-29 07:09:55.000000000 +0000 +++ /tmp/wklog.111.new.13259 2010-03-29 07:09:55.000000000 +0000 @@ -1 +1 @@ -Client-BackLog +Server-RawIdeaBin -=-=(Guest - Mon, 29 Mar 2010, 07:09)=-=- Version updated. --- /tmp/wklog.111.old.13259 2010-03-29 07:09:55.000000000 +0000 +++ /tmp/wklog.111.new.13259 2010-03-29 07:09:55.000000000 +0000 @@ -1 +1 @@ -Benchmarks-3.0 +9.x -=-=(Guest - Mon, 29 Mar 2010, 07:09)=-=- Version updated. --- /tmp/wklog.111.old.13259 2010-03-29 07:09:55.000000000 +0000 +++ /tmp/wklog.111.new.13259 2010-03-29 07:09:55.000000000 +0000 @@ -1 +1 @@ -9.x +Server-5.3 DESCRIPTION: Current EXPLAIN does not show where the subquery predicate is attached to. For example, see [1], slide#5 "Straightforward subquery evaluation (contd)", or look here: MariaDB [test]> explain select * from ot1, ot2 where ot1.a=ot2.a and (ot2.a in (select it1.b from it1) or ot1.b<3); +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ | 1 | PRIMARY | ot1 | ALL | NULL | NULL | NULL | NULL | 10 | Using where | | 1 | PRIMARY | ot2 | ALL | NULL | NULL | NULL | NULL | 10 | Using where; Using join buffer | | 2 | SUBQUERY | it1 | ALL | NULL | NULL | NULL | NULL | 20 | | +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ 3 rows in set (0.01 sec) Here there are two "Using where", and it is not clear where the predicate is attached to. If one has sufficient knowledge, they could know that - "ot2.a in (select )" will be substituted for "ot1.a" (provided datatypes allow equality propagation) - For correlated subqueries, equality propagation will not affect outside references (so, if the subquery in the above example was correlated, it would have been attached to table ot2, not ot1). As one can see, the rules are quite complicated. The full solution would be to show expressions behind "Using Where", but that has additional complications due to expressions being too long for current EXPLAIN output format. A simplified solution would be to show "Subquery#n" in Extra column if the clause in "Using Where" has a subquery. That is, if table's condition has subquery expressions, Extra column will look like this: Using where; Subquery#n1; Subquery#n2,... When table's condition has only subquery expression, we could omit printing "Using where" (note that this case is quite rare now. the subquery must be correlated, non-semi-join subquery predicate that is an AND-part of the WHERE. such cases are in minority) [1] http://forge.mysql.com/wiki/Image:NewSubqueryOptimizationsIn6_0_UC2008.pdf ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Progress (by Knielsen): New replication APIs (107)
by worklog-noreply＠askmonty.org 29 Mar '10

29 Mar '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: New replication APIs CREATION DATE..: Mon, 15 Mar 2010, 13:55 SUPERVISOR.....: Knielsen IMPLEMENTOR....: Knielsen COPIES TO......: Sergei CATEGORY.......: Server-Sprint TASK ID........: 107 (http://askmonty.org/worklog/?tid=107) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 50 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Knielsen - Mon, 29 Mar 2010, 07:33)=-=- Research and design discussions: Galera, 2pc/XA, group commit, multi-engine transactions. Worked 14 hours and estimate 0 hours remain (original estimate increased by 14 hours). -=-=(Knielsen - Wed, 24 Mar 2010, 10:39)=-=- Design discussions Worked 11 hours and estimate 0 hours remain (original estimate increased by 11 hours). -=-=(Knielsen - Mon, 15 Mar 2010, 14:28)=-=- Research into the problem, and discussions on phone/mailing list Worked 25 hours and estimate 0 hours remain (original estimate increased by 25 hours). -=-=(Guest - Mon, 15 Mar 2010, 14:18)=-=- High-Level Specification modified. --- /tmp/wklog.107.old.9086 2010-03-15 14:18:18.000000000 +0000 +++ /tmp/wklog.107.new.9086 2010-03-15 14:18:18.000000000 +0000 @@ -1 +1,43 @@ +Current ideas/status after discussions on the mailing list: + + - Implement a set of plugin APIs and use them to move all of the existing + MySQL replication into a (set of) plugins. + + - Design the APIs so that they can support full MySQL replication, but also + so that they do not hardcode assumptions about how this replication + implementation is done, and so that they will be suitable for other types of + replication (Tungsten, Galera, parallel replication, ...). + + - APIs need to include the concept of a global transaction ID. Need to + determine the extent to which the semantics of such ID will be defined + by the API, and to which extend it will be defined by the plugin + implementations. + + - APIs should properly support reliable crash-recovery with decent + performance (eg. not require multiple mandatory fsync()s per commit, and + not make group commit impossible). + + - Would be nice if the API provided facilities for implementing good + consistency checking support (mainly checking master tables against slave + tables is hard here I think, but also applying wrong binlog data and + individual event checksums). + + +Steps to make this more concrete: + + - Investigate the current MySQL replication, and list all of the places where + a plugin implementation will need to connect/hook into the MySQL server. + * handler::{write,update,delete}_row() + * Statement execution + * Transaction start/commit + * Table open + * Query safe/not/safe for statement based replication + * Statement-based logging details (user variables, random seed, etc.) + * ... + + - Use this list to make an initial sketch of the set of APIs we need. + + - Use the list to determine the feasibility of this project and the level of + detail in the API needed to support a full replication implementation as a + plugin. -=-=(Serg - Mon, 15 Mar 2010, 14:13)=-=- Observers changed: Sergei DESCRIPTION: This is a top-level task for the project of designing a new set of replication APIs for MariaDB. This task is for the initial discussion of what to do and where to focus. The project is started in this email thread: https://lists.launchpad.net/maria-developers/msg01998.html HIGH-LEVEL SPECIFICATION: Current ideas/status after discussions on the mailing list: - Implement a set of plugin APIs and use them to move all of the existing MySQL replication into a (set of) plugins. - Design the APIs so that they can support full MySQL replication, but also so that they do not hardcode assumptions about how this replication implementation is done, and so that they will be suitable for other types of replication (Tungsten, Galera, parallel replication, ...). - APIs need to include the concept of a global transaction ID. Need to determine the extent to which the semantics of such ID will be defined by the API, and to which extend it will be defined by the plugin implementations. - APIs should properly support reliable crash-recovery with decent performance (eg. not require multiple mandatory fsync()s per commit, and not make group commit impossible). - Would be nice if the API provided facilities for implementing good consistency checking support (mainly checking master tables against slave tables is hard here I think, but also applying wrong binlog data and individual event checksums). Steps to make this more concrete: - Investigate the current MySQL replication, and list all of the places where a plugin implementation will need to connect/hook into the MySQL server. * handler::{write,update,delete}_row() * Statement execution * Transaction start/commit * Table open * Query safe/not/safe for statement based replication * Statement-based logging details (user variables, random seed, etc.) * ... - Use this list to make an initial sketch of the set of APIs we need. - Use the list to determine the feasibility of this project and the level of detail in the API needed to support a full replication implementation as a plugin. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Progress (by Knielsen): New replication APIs (107)
by worklog-noreply＠askmonty.org 29 Mar '10

29 Mar '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: New replication APIs CREATION DATE..: Mon, 15 Mar 2010, 13:55 SUPERVISOR.....: Knielsen IMPLEMENTOR....: Knielsen COPIES TO......: Sergei CATEGORY.......: Server-Sprint TASK ID........: 107 (http://askmonty.org/worklog/?tid=107) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 50 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Knielsen - Mon, 29 Mar 2010, 07:33)=-=- Research and design discussions: Galera, 2pc/XA, group commit, multi-engine transactions. Worked 14 hours and estimate 0 hours remain (original estimate increased by 14 hours). -=-=(Knielsen - Wed, 24 Mar 2010, 10:39)=-=- Design discussions Worked 11 hours and estimate 0 hours remain (original estimate increased by 11 hours). -=-=(Knielsen - Mon, 15 Mar 2010, 14:28)=-=- Research into the problem, and discussions on phone/mailing list Worked 25 hours and estimate 0 hours remain (original estimate increased by 25 hours). -=-=(Guest - Mon, 15 Mar 2010, 14:18)=-=- High-Level Specification modified. --- /tmp/wklog.107.old.9086 2010-03-15 14:18:18.000000000 +0000 +++ /tmp/wklog.107.new.9086 2010-03-15 14:18:18.000000000 +0000 @@ -1 +1,43 @@ +Current ideas/status after discussions on the mailing list: + + - Implement a set of plugin APIs and use them to move all of the existing + MySQL replication into a (set of) plugins. + + - Design the APIs so that they can support full MySQL replication, but also + so that they do not hardcode assumptions about how this replication + implementation is done, and so that they will be suitable for other types of + replication (Tungsten, Galera, parallel replication, ...). + + - APIs need to include the concept of a global transaction ID. Need to + determine the extent to which the semantics of such ID will be defined + by the API, and to which extend it will be defined by the plugin + implementations. + + - APIs should properly support reliable crash-recovery with decent + performance (eg. not require multiple mandatory fsync()s per commit, and + not make group commit impossible). + + - Would be nice if the API provided facilities for implementing good + consistency checking support (mainly checking master tables against slave + tables is hard here I think, but also applying wrong binlog data and + individual event checksums). + + +Steps to make this more concrete: + + - Investigate the current MySQL replication, and list all of the places where + a plugin implementation will need to connect/hook into the MySQL server. + * handler::{write,update,delete}_row() + * Statement execution + * Transaction start/commit + * Table open + * Query safe/not/safe for statement based replication + * Statement-based logging details (user variables, random seed, etc.) + * ... + + - Use this list to make an initial sketch of the set of APIs we need. + + - Use the list to determine the feasibility of this project and the level of + detail in the API needed to support a full replication implementation as a + plugin. -=-=(Serg - Mon, 15 Mar 2010, 14:13)=-=- Observers changed: Sergei DESCRIPTION: This is a top-level task for the project of designing a new set of replication APIs for MariaDB. This task is for the initial discussion of what to do and where to focus. The project is started in this email thread: https://lists.launchpad.net/maria-developers/msg01998.html HIGH-LEVEL SPECIFICATION: Current ideas/status after discussions on the mailing list: - Implement a set of plugin APIs and use them to move all of the existing MySQL replication into a (set of) plugins. - Design the APIs so that they can support full MySQL replication, but also so that they do not hardcode assumptions about how this replication implementation is done, and so that they will be suitable for other types of replication (Tungsten, Galera, parallel replication, ...). - APIs need to include the concept of a global transaction ID. Need to determine the extent to which the semantics of such ID will be defined by the API, and to which extend it will be defined by the plugin implementations. - APIs should properly support reliable crash-recovery with decent performance (eg. not require multiple mandatory fsync()s per commit, and not make group commit impossible). - Would be nice if the API provided facilities for implementing good consistency checking support (mainly checking master tables against slave tables is hard here I think, but also applying wrong binlog data and individual event checksums). Steps to make this more concrete: - Investigate the current MySQL replication, and list all of the places where a plugin implementation will need to connect/hook into the MySQL server. * handler::{write,update,delete}_row() * Statement execution * Transaction start/commit * Table open * Query safe/not/safe for statement based replication * Statement-based logging details (user variables, random seed, etc.) * ... - Use this list to make an initial sketch of the set of APIs we need. - Use the list to determine the feasibility of this project and the level of detail in the API needed to support a full replication implementation as a plugin. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Guest): Make EXPLAIN show where subquery predicates are in the WHERE clause (111)
by worklog-noreply＠askmonty.org 29 Mar '10

29 Mar '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Make EXPLAIN show where subquery predicates are in the WHERE clause CREATION DATE..: Mon, 29 Mar 2010, 07:07 SUPERVISOR.....: Psergey IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 111 (http://askmonty.org/worklog/?tid=111) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Guest - Mon, 29 Mar 2010, 07:09)=-=- Supervisor updated. --- /tmp/wklog.111.old.13259 2010-03-29 07:09:55.000000000 +0000 +++ /tmp/wklog.111.new.13259 2010-03-29 07:09:55.000000000 +0000 @@ -1 +1 @@ -Bothorsen +Psergey -=-=(Guest - Mon, 29 Mar 2010, 07:09)=-=- Category updated. --- /tmp/wklog.111.old.13259 2010-03-29 07:09:55.000000000 +0000 +++ /tmp/wklog.111.new.13259 2010-03-29 07:09:55.000000000 +0000 @@ -1 +1 @@ -Client-BackLog +Server-RawIdeaBin -=-=(Guest - Mon, 29 Mar 2010, 07:09)=-=- Version updated. --- /tmp/wklog.111.old.13259 2010-03-29 07:09:55.000000000 +0000 +++ /tmp/wklog.111.new.13259 2010-03-29 07:09:55.000000000 +0000 @@ -1 +1 @@ -Benchmarks-3.0 +9.x -=-=(Guest - Mon, 29 Mar 2010, 07:09)=-=- Version updated. --- /tmp/wklog.111.old.13259 2010-03-29 07:09:55.000000000 +0000 +++ /tmp/wklog.111.new.13259 2010-03-29 07:09:55.000000000 +0000 @@ -1 +1 @@ -9.x +Server-5.3 DESCRIPTION: Current EXPLAIN does not show where the subquery predicate is attached to. For example, see [1], slide#5 "Straightforward subquery evaluation (contd)", or look here: MariaDB [test]> explain select * from ot1, ot2 where ot1.a=ot2.a and (ot2.a in (select it1.b from it1) or ot1.b<3); +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ | 1 | PRIMARY | ot1 | ALL | NULL | NULL | NULL | NULL | 10 | Using where | | 1 | PRIMARY | ot2 | ALL | NULL | NULL | NULL | NULL | 10 | Using where; Using join buffer | | 2 | SUBQUERY | it1 | ALL | NULL | NULL | NULL | NULL | 20 | | +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ 3 rows in set (0.01 sec) Here there are two "Using where", and it is not clear where the predicate is attached to. If one has sufficient knowledge, they could know that - "ot2.a in (select )" will be substituted for "ot1.a" (provided datatypes allow equality propagation) - For correlated subqueries, equality propagation will not affect outside references (so, if the subquery in the above example was correlated, it would have been attached to table ot2, not ot1). As one can see, the rules are quite complicated. The full solution would be to show expressions behind "Using Where", but that has additional complications due to expressions being too long for current EXPLAIN output format. A simplified solution would be to show "Subquery#n" in Extra column if the clause in "Using Where" has a subquery. [1] http://forge.mysql.com/wiki/Image:NewSubqueryOptimizationsIn6_0_UC2008.pdf ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Make EXPLAIN show where subquery predicates are in the WHERE clause (111)
by worklog-noreply＠askmonty.org 29 Mar '10

29 Mar '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Make EXPLAIN show where subquery predicates are in the WHERE clause CREATION DATE..: Mon, 29 Mar 2010, 07:07 SUPERVISOR.....: Bothorsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-BackLog TASK ID........: 111 (http://askmonty.org/worklog/?tid=111) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Current EXPLAIN does not show where the subquery predicate is attached to. For example, see [1], slide#5 "Straightforward subquery evaluation (contd)", or look here: MariaDB [test]> explain select * from ot1, ot2 where ot1.a=ot2.a and (ot2.a in (select it1.b from it1) or ot1.b<3); +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ | 1 | PRIMARY | ot1 | ALL | NULL | NULL | NULL | NULL | 10 | Using where | | 1 | PRIMARY | ot2 | ALL | NULL | NULL | NULL | NULL | 10 | Using where; Using join buffer | | 2 | SUBQUERY | it1 | ALL | NULL | NULL | NULL | NULL | 20 | | +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ 3 rows in set (0.01 sec) Here there are two "Using where", and it is not clear where the predicate is attached to. If one has sufficient knowledge, they could know that - "ot2.a in (select )" will be substituted for "ot1.a" (provided datatypes allow equality propagation) - For correlated subqueries, equality propagation will not affect outside references (so, if the subquery in the above example was correlated, it would have been attached to table ot2, not ot1). As one can see, the rules are quite complicated. The full solution would be to show expressions behind "Using Where", but that has additional complications due to expressions being too long for current EXPLAIN output format. A simplified solution would be to show "Subquery#n" in Extra column if the clause in "Using Where" has a subquery. [1] http://forge.mysql.com/wiki/Image:NewSubqueryOptimizationsIn6_0_UC2008.pdf ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Make EXPLAIN show where subquery predicates are in the WHERE clause (111)
by worklog-noreply＠askmonty.org 29 Mar '10

29 Mar '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Make EXPLAIN show where subquery predicates are in the WHERE clause CREATION DATE..: Mon, 29 Mar 2010, 07:07 SUPERVISOR.....: Bothorsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-BackLog TASK ID........: 111 (http://askmonty.org/worklog/?tid=111) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Current EXPLAIN does not show where the subquery predicate is attached to. For example, see [1], slide#5 "Straightforward subquery evaluation (contd)", or look here: MariaDB [test]> explain select * from ot1, ot2 where ot1.a=ot2.a and (ot2.a in (select it1.b from it1) or ot1.b<3); +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ | 1 | PRIMARY | ot1 | ALL | NULL | NULL | NULL | NULL | 10 | Using where | | 1 | PRIMARY | ot2 | ALL | NULL | NULL | NULL | NULL | 10 | Using where; Using join buffer | | 2 | SUBQUERY | it1 | ALL | NULL | NULL | NULL | NULL | 20 | | +----+-------------+-------+------+---------------+------+---------+------+------+--------------------------------+ 3 rows in set (0.01 sec) Here there are two "Using where", and it is not clear where the predicate is attached to. If one has sufficient knowledge, they could know that - "ot2.a in (select )" will be substituted for "ot1.a" (provided datatypes allow equality propagation) - For correlated subqueries, equality propagation will not affect outside references (so, if the subquery in the above example was correlated, it would have been attached to table ot2, not ot1). As one can see, the rules are quite complicated. The full solution would be to show expressions behind "Using Where", but that has additional complications due to expressions being too long for current EXPLAIN output format. A simplified solution would be to show "Subquery#n" in Extra column if the clause in "Using Where" has a subquery. [1] http://forge.mysql.com/wiki/Image:NewSubqueryOptimizationsIn6_0_UC2008.pdf ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0