- developers - lists.mariadb.org

[Maria-developers] Progress (by Monty): Backporting pool of threads to MariaDB (6)
by worklog-noreply＠askmonty.org 11 Aug '09

11 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Backporting pool of threads to MariaDB CREATION DATE..: Mon, 09 Mar 2009, 17:21 SUPERVISOR.....: Monty IMPLEMENTOR....: Monty COPIES TO......: Monty CATEGORY.......: Server-Sprint TASK ID........: 6 (http://askmonty.org/worklog/?tid=6) VERSION........: Server-5.1 STATUS.........: Complete PRIORITY.......: 60 WORKED HOURS...: 16 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 8 PROGRESS NOTES: -=-=(Monty - Tue, 11 Aug 2009, 16:58)=-=- Done, ages ago Worked 16 hours and estimate 0 hours remain (original estimate increased by 8 hours). -=-=(Monty - Thu, 26 Mar 2009, 00:32)=-=- Privacy level updated. --- /tmp/wklog.6.old.6586 2009-03-26 00:32:23.000000000 +0200 +++ /tmp/wklog.6.new.6586 2009-03-26 00:32:23.000000000 +0200 @@ -1 +1 @@ -y +n -=-=(Monty - Thu, 26 Mar 2009, 00:31)=-=- Supervisor updated. --- /tmp/wklog.6.old.6580 2009-03-26 00:31:30.000000000 +0200 +++ /tmp/wklog.6.new.6580 2009-03-26 00:31:30.000000000 +0200 @@ -1 +1 @@ -Knielsen +Monty -=-=(Monty - Fri, 13 Mar 2009, 02:43)=-=- Low Level Design modified. --- /tmp/wklog.6.old.26076 2009-03-13 02:43:17.000000000 +0200 +++ /tmp/wklog.6.new.26076 2009-03-13 02:43:17.000000000 +0200 @@ -1 +1,20 @@ +To be able to work with both one-thread-per-connection and pool-of-threads at +the same time, I added a new global scheduler variable 'extra_thread_scheduler' +that is always using the one-thread-per-connection method. + +To the THD structure was added a pointer to the 'scheduler' variable that should +be used for this connection. + +To do easy handing of two connect counter and two max_connection variables, I +added pointer to these pointer in the scheduler variable.: + +Other changes was: + +- If extra-port was <> 0, start listing to this port too +- At connect time, set THD->scheduler to point to the given scheduler (based on +the port that was used to connect) +- Change some calls that was done trough functions pointer in the scheduler to +instead use thd->scheduler-> +- Change max_connections to *thd->scheduler->max_connections +- Change connection_count to *thd->scheduler->connection_count -=-=(Monty - Fri, 13 Mar 2009, 02:29)=-=- Version updated. --- /tmp/wklog.6.old.25818 2009-03-13 02:29:16.000000000 +0200 +++ /tmp/wklog.6.new.25818 2009-03-13 02:29:16.000000000 +0200 @@ -1 +1 @@ -Server-9.x +Server-5.1 -=-=(Monty - Fri, 13 Mar 2009, 02:29)=-=- Status updated. --- /tmp/wklog.6.old.25818 2009-03-13 02:29:16.000000000 +0200 +++ /tmp/wklog.6.new.25818 2009-03-13 02:29:16.000000000 +0200 @@ -1 +1 @@ -Assigned +Complete -=-=(Monty - Fri, 13 Mar 2009, 02:28)=-=- High Level Description modified. --- /tmp/wklog.6.old.25790 2009-03-13 02:28:25.000000000 +0200 +++ /tmp/wklog.6.new.25790 2009-03-13 02:28:25.000000000 +0200 @@ -8,3 +8,6 @@ Add option --extra-port to allow connections with old one-thread-per-connection method. This is needed to allow root to login and kill threads if something goes wrong. +Add option --extra-max-connections to regulate how many connections can be made +to 'extra-port'. This should work in a similar way as 'max-connections', in the +way that one connection is reserved for a SUPER user. -=-=(Knielsen - Mon, 09 Mar 2009, 19:02)=-=- Version updated. --- /tmp/wklog.6.old.10740 2009-03-09 19:02:38.000000000 +0200 +++ /tmp/wklog.6.new.10740 2009-03-09 19:02:38.000000000 +0200 @@ -1 +1 @@ -WorkLog-3.4 +Server-9.x -=-=(Knielsen - Mon, 09 Mar 2009, 19:02)=-=- Title modified. --- /tmp/wklog.6.old.10740 2009-03-09 19:02:38.000000000 +0200 +++ /tmp/wklog.6.new.10740 2009-03-09 19:02:38.000000000 +0200 @@ -1 +1 @@ -Backporting pool of threads tro MariaDB +Backporting pool of threads to MariaDB DESCRIPTION: Back porting pool of threads to MariaDB We will use code for Maria 6.0, with the following extensions: Add option: --test-ignore-wrong-options to ignore errors in enum values for testing pool-of-threads. (Better than having --pool-of-threads command line option just for testing) Add option --extra-port to allow connections with old one-thread-per-connection method. This is needed to allow root to login and kill threads if something goes wrong. Add option --extra-max-connections to regulate how many connections can be made to 'extra-port'. This should work in a similar way as 'max-connections', in the way that one connection is reserved for a SUPER user. LOW-LEVEL DESIGN: To be able to work with both one-thread-per-connection and pool-of-threads at the same time, I added a new global scheduler variable 'extra_thread_scheduler' that is always using the one-thread-per-connection method. To the THD structure was added a pointer to the 'scheduler' variable that should be used for this connection. To do easy handing of two connect counter and two max_connection variables, I added pointer to these pointer in the scheduler variable.: Other changes was: - If extra-port was <> 0, start listing to this port too - At connect time, set THD->scheduler to point to the given scheduler (based on the port that was used to connect) - Change some calls that was done trough functions pointer in the scheduler to instead use thd->scheduler-> - Change max_connections to *thd->scheduler->max_connections - Change connection_count to *thd->scheduler->connection_count ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Monty): Add Sphinx storage engine to MariaDB (42)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add Sphinx storage engine to MariaDB CREATION DATE..: Mon, 10 Aug 2009, 23:57 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Maria-BackLog TASK ID........: 42 (http://askmonty.org/worklog/?tid=42) VERSION........: Connector/.NET-5.1 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 16 (hours remain) ORIG. ESTIMATE.: 16 PROGRESS NOTES: DESCRIPTION: Add the Sphinx storage engine to the MariaDB tree ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

Re: [Maria-developers] keycache flush problem
by Michael Widenius 10 Aug '09

10 Aug '09

Hi! For those that don't know what this is about: - This is a fix for the case where you do a DROP TABLE of a MyISAM table with key delayed MyISAM writes all the changed key pages for the file to disk before closing and then deleting the table. This patch is a first attempt to fix that we don't write the key pages in case of drop. >>>>> "Oleksandr" == Oleksandr Byelkin <Oleksandr> writes: Oleksandr> Hi! Oleksandr> I made different patch from you suggested maybe I am wrong but IMHO it Oleksandr> is better because: Oleksandr> 1) use already existing keycache calls Oleksandr> 2) do not require additional finding table MI_INFO by name and locking Oleksandr> around it Oleksandr> The main idea was that if we are going to drop table it can be passed Oleksandr> via existing table descriptors to the place where we call flush and it Oleksandr> does not matter if other threads trying to do something with the table Oleksandr> we will drop it in any case. Oleksandr> === modified file 'sql/handler.h' Oleksandr> --- sql/handler.h 2009-06-29 21:03:30 +0000 Oleksandr> +++ sql/handler.h 2009-08-09 19:52:01 +0000 Oleksandr> @@ -1342,6 +1342,7 @@ public: Oleksandr> virtual void column_bitmaps_signal(); Oleksandr> uint get_index(void) const { return active_index; } Oleksandr> virtual int close(void)=0; Oleksandr> + virtual void prepare_for_delete() {} Oleksandr> /** Oleksandr> @retval 0 Bulk update used by handler Why using prepare_for_delete(), instead of adding a more general call ? Oleksandr> === modified file 'sql/lock.cc' Oleksandr> --- sql/lock.cc 2009-04-25 10:05:32 +0000 Oleksandr> +++ sql/lock.cc 2009-08-09 22:08:41 +0000 Oleksandr> @@ -1049,10 +1049,12 @@ int lock_table_name(THD *thd, TABLE_LIST Oleksandr> DBUG_RETURN(-1); Oleksandr> table_list->table=table; Oleksandr> + table->s->deleting= table_list->deleting; Oleksandr> /* Return 1 if table is in use */ Oleksandr> DBUG_RETURN(test(remove_table_from_cache(thd, db, table_list->table_name, Oleksandr> - check_in_use ? RTFC_NO_FLAG : RTFC_WAIT_OTHER_THREAD_FLAG))); Oleksandr> + check_in_use ? RTFC_NO_FLAG : RTFC_WAIT_OTHER_THREAD_FLAG, Oleksandr> + table_list->deleting))); Oleksandr> } Oleksandr> === modified file 'sql/mysql_priv.h' Oleksandr> --- sql/mysql_priv.h 2009-04-25 10:05:32 +0000 Oleksandr> +++ sql/mysql_priv.h 2009-08-09 21:51:48 +0000 Oleksandr> @@ -1609,7 +1609,7 @@ uint prep_alter_part_table(THD *thd, TAB Oleksandr> #define RTFC_WAIT_OTHER_THREAD_FLAG 0x0002 Oleksandr> #define RTFC_CHECK_KILLED_FLAG 0x0004 Oleksandr> bool remove_table_from_cache(THD *thd, const char *db, const char *table, Oleksandr> - uint flags); Oleksandr> + uint flags, my_bool deleting); Oleksandr> #define NORMAL_PART_NAME 0 Oleksandr> #define TEMP_PART_NAME 1 Oleksandr> === modified file 'sql/sql_base.cc' Oleksandr> --- sql/sql_base.cc 2009-05-19 09:28:05 +0000 Oleksandr> +++ sql/sql_base.cc 2009-08-09 21:54:54 +0000 Oleksandr> @@ -927,7 +927,7 @@ bool close_cached_tables(THD *thd, TABLE Oleksandr> for (TABLE_LIST *table= tables; table; table= table->next_local) Oleksandr> { Oleksandr> if (remove_table_from_cache(thd, table->db, table->table_name, Oleksandr> - RTFC_OWNED_BY_THD_FLAG)) Oleksandr> + RTFC_OWNED_BY_THD_FLAG, table->deleting)) Oleksandr> found=1; Oleksandr> } Oleksandr> if (!found) Oleksandr> @@ -8395,7 +8395,7 @@ void flush_tables() Oleksandr> */ Oleksandr> bool remove_table_from_cache(THD *thd, const char *db, const char *table_name, Oleksandr> - uint flags) Oleksandr> + uint flags, my_bool deleting) Oleksandr> { Oleksandr> char key[MAX_DBKEY_LENGTH]; Oleksandr> uint key_length; Oleksandr> @@ -8482,7 +8482,10 @@ bool remove_table_from_cache(THD *thd, c Oleksandr> } Oleksandr> } Oleksandr> while (unused_tables && !unused_tables->s->version) Oleksandr> + { Oleksandr> + unused_tables->s->deleting= deleting; Oleksandr> VOID(hash_delete(&open_cache,(uchar*) unused_tables)); Oleksandr> + } Oleksandr> DBUG_PRINT("info", ("Removing table from table_def_cache")); Oleksandr> /* Remove table from table definition cache if it's not in use */ Oleksandr> @@ -8676,7 +8679,8 @@ int abort_and_upgrade_lock(ALTER_PARTITI Oleksandr> /* If MERGE child, forward lock handling to parent. */ Oleksandr> mysql_lock_abort(lpt->thd, lpt->table->parent ? lpt->table->parent : Oleksandr> lpt->table, TRUE); Oleksandr> - VOID(remove_table_from_cache(lpt->thd, lpt->db, lpt->table_name, flags)); Oleksandr> + VOID(remove_table_from_cache(lpt->thd, lpt->db, lpt->table_name, flags, Oleksandr> + FALSE)); Oleksandr> VOID(pthread_mutex_unlock(&LOCK_open)); Oleksandr> DBUG_RETURN(0); Oleksandr> } Oleksandr> @@ -8701,7 +8705,7 @@ void close_open_tables_and_downgrade(ALT Oleksandr> { Oleksandr> VOID(pthread_mutex_lock(&LOCK_open)); Oleksandr> remove_table_from_cache(lpt->thd, lpt->db, lpt->table_name, Oleksandr> - RTFC_WAIT_OTHER_THREAD_FLAG); Oleksandr> + RTFC_WAIT_OTHER_THREAD_FLAG, FALSE); Oleksandr> VOID(pthread_mutex_unlock(&LOCK_open)); Oleksandr> /* If MERGE child, forward lock handling to parent. */ Oleksandr> mysql_lock_downgrade_write(lpt->thd, lpt->table->parent ? lpt->table->parent : Oleksandr> === modified file 'sql/sql_table.cc' Oleksandr> --- sql/sql_table.cc 2009-06-18 12:39:21 +0000 Oleksandr> +++ sql/sql_table.cc 2009-08-09 21:48:04 +0000 Oleksandr> @@ -1599,6 +1599,8 @@ int mysql_rm_table_part2(THD *thd, TABLE Oleksandr> if ((share= get_cached_table_share(table->db, table->table_name))) Oleksandr> table->db_type= share->db_type(); Oleksandr> + table->deleting= TRUE; Oleksandr> + Oleksandr> /* Disable drop of enabled log tables */ Oleksandr> if (share && (share->table_category == TABLE_CATEGORY_PERFORMANCE) && Oleksandr> check_if_log_table(table->db_length, table->db, Oleksandr> @@ -1676,7 +1678,7 @@ int mysql_rm_table_part2(THD *thd, TABLE Oleksandr> abort_locked_tables(thd, db, table->table_name); Oleksandr> remove_table_from_cache(thd, db, table->table_name, Oleksandr> RTFC_WAIT_OTHER_THREAD_FLAG | Oleksandr> - RTFC_CHECK_KILLED_FLAG); Oleksandr> + RTFC_CHECK_KILLED_FLAG, TRUE); Oleksandr> /* Oleksandr> If the table was used in lock tables, remember it so that Oleksandr> unlock_table_names can free it Oleksandr> @@ -3862,7 +3864,7 @@ void wait_while_table_is_used(THD *thd,T Oleksandr> /* Wait until all there are no other threads that has this table open */ Oleksandr> remove_table_from_cache(thd, table->s->db.str, Oleksandr> table->s->table_name.str, Oleksandr> - RTFC_WAIT_OTHER_THREAD_FLAG); Oleksandr> + RTFC_WAIT_OTHER_THREAD_FLAG, FALSE); Oleksandr> /* extra() call must come only after all instances above are closed */ Oleksandr> VOID(table->file->extra(function)); Oleksandr> DBUG_VOID_RETURN; Oleksandr> @@ -4366,7 +4368,7 @@ static bool mysql_admin_table(THD* thd, Oleksandr> remove_table_from_cache(thd, table->table->s->db.str, Oleksandr> table->table->s->table_name.str, Oleksandr> RTFC_WAIT_OTHER_THREAD_FLAG | Oleksandr> - RTFC_CHECK_KILLED_FLAG); Oleksandr> + RTFC_CHECK_KILLED_FLAG, FALSE); Oleksandr> thd->exit_cond(old_message); Oleksandr> DBUG_EXECUTE_IF("wait_in_mysql_admin_table", wait_for_kill_signal(thd);); Oleksandr> if (thd->killed) Oleksandr> @@ -4624,7 +4626,8 @@ send_result_message: Oleksandr> { Oleksandr> pthread_mutex_lock(&LOCK_open); Oleksandr> remove_table_from_cache(thd, table->table->s->db.str, Oleksandr> - table->table->s->table_name.str, RTFC_NO_FLAG); Oleksandr> + table->table->s->table_name.str, Oleksandr> + RTFC_NO_FLAG, FALSE); Oleksandr> pthread_mutex_unlock(&LOCK_open); Oleksandr> } Oleksandr> /* May be something modified consequently we have to invalidate cache */ Oleksandr> === modified file 'sql/table.cc' Oleksandr> --- sql/table.cc 2009-06-29 21:03:30 +0000 Oleksandr> +++ sql/table.cc 2009-08-09 20:46:07 +0000 Oleksandr> @@ -1960,7 +1960,12 @@ int closefrm(register TABLE *table, bool Oleksandr> DBUG_PRINT("enter", ("table: 0x%lx", (long) table)); Oleksandr> if (table->db_stat) Oleksandr> - error=table->file->close(); Oleksandr> + { Oleksandr> + if (table->s->deleting) Oleksandr> + table->file->prepare_for_delete(); Oleksandr> + error= table->file->close(); Oleksandr> + } Oleksandr> + As we have a handler here, we not instead do ? table->file->extra(HA_EXTRA_PREPARE_FOR_DROP); There is no reason to add an extra prepare_for_delete() here. Oleksandr> --- storage/myisam/ha_myisam.cc 2009-06-29 21:03:30 +0000 Oleksandr> +++ storage/myisam/ha_myisam.cc 2009-08-09 20:42:15 +0000 Oleksandr> @@ -26,7 +26,9 @@ Oleksandr> #include <myisampack.h> Oleksandr> #include "ha_myisam.h" Oleksandr> #include <stdarg.h> Oleksandr> +C_MODE_START Oleksandr> #include "myisamdef.h" Oleksandr> +C_MODE_END Oleksandr> #include "rt_index.h" With my suggested change, no reason to do any changes in ha_myisam.cc or ha_myisam.h <cut> Oleksandr> +++ storage/myisam/mi_close.c 2009-08-09 22:01:32 +0000 Oleksandr> @@ -65,8 +65,9 @@ int mi_close(register MI_INFO *info) Oleksandr> { Oleksandr> if (share->kfile >= 0 && Oleksandr> flush_key_blocks(share->key_cache, share->kfile, Oleksandr> - share->temporary ? FLUSH_IGNORE_CHANGED : Oleksandr> - FLUSH_RELEASE)) Oleksandr> + (share->temporary || share->deleting) ? Oleksandr> + FLUSH_IGNORE_CHANGED : Oleksandr> + FLUSH_RELEASE)) Oleksandr> error=my_errno; Oleksandr> if (share->kfile >= 0) Oleksandr> { No reason for the above change. 1) In my suggestion, no reason to do this. 2) If we implement it your way, we could reuse 'share->temporary' for this cse. Oleksandr> === modified file 'storage/myisam/mi_locking.c' Oleksandr> --- storage/myisam/mi_locking.c 2009-04-01 09:34:52 +0000 Oleksandr> +++ storage/myisam/mi_locking.c 2009-08-09 20:42:00 +0000 Oleksandr> @@ -68,7 +68,10 @@ int mi_lock_database(MI_INFO *info, int Oleksandr> --share->tot_locks; Oleksandr> if (info->lock_type == F_WRLCK && !share->w_locks && Oleksandr> !share->delay_key_write && flush_key_blocks(share->key_cache, Oleksandr> - share->kfile,FLUSH_KEEP)) Oleksandr> + share->kfile, Oleksandr> + (share->deleting ? Oleksandr> + FLUSH_IGNORE_CHANGED : Oleksandr> + FLUSH_KEEP))) No reason to do the above. Reasons: - In case of delay_key_write, they above code will not be executed. - If delay_key_write is not set, things was flushed at previous statement. Did I miss some case? Oleksandr> --- storage/myisam/myisamdef.h 2009-04-25 09:04:38 +0000 Oleksandr> +++ storage/myisam/myisamdef.h 2009-08-09 20:41:25 +0000 Oleksandr> @@ -218,6 +218,7 @@ typedef struct st_mi_isam_share Oleksandr> my_bool changed, /* If changed since lock */ Oleksandr> global_changed, /* If changed since open */ Oleksandr> not_flushed, temporary, delay_key_write, concurrent_insert; Oleksandr> + my_bool deleting; /* we are going to delete this table */ Not needed. --------------- Other fixes: Please fix mi_extra.c as we dicussed (move the #ifdef so that things are flushed) do also the folloing fix to ma_extra.c: if (share->kfile.file >= 0) _ma_decrement_open_count(info); -> if (share->kfile.file >= 0 && do_flush) _ma_decrement_open_count(info); The idea is that we should not decrement the open_count in case of drop. This will ensure that if we die between flushing the key cache and close, the index will be rechecked. ------------- Regards, Monty

1 0

[Maria-developers] Progress (by Guest): Replication tasks (39)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Replication tasks CREATION DATE..: Sun, 09 Aug 2009, 12:24 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-RawIdeaBin TASK ID........: 39 (http://askmonty.org/worklog/?tid=39) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 17 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Guest - Mon, 10 Aug 2009, 16:32)=-=- Adding 1 hour for Monty's initial work on starting the architecture review. Worked 1 hour and estimate 0 hours remain (original estimate increased by 1 hour). -=-=(Psergey - Mon, 10 Aug 2009, 15:59)=-=- Re-searched and added subtasks. Worked 16 hours and estimate 0 hours remain (original estimate increased by 16 hours). -=-=(Psergey - Mon, 10 Aug 2009, 15:31)=-=- Dependency created: 39 now depends on 41 -=-=(Guest - Mon, 10 Aug 2009, 14:52)=-=- Dependency created: 39 now depends on 40 -=-=(Psergey - Sun, 09 Aug 2009, 12:27)=-=- Dependency created: 39 now depends on 36 -=-=(Psergey - Sun, 09 Aug 2009, 12:24)=-=- Dependency created: 39 now depends on 38 -=-=(Psergey - Sun, 09 Aug 2009, 12:24)=-=- Dependency created: 39 now depends on 37 DESCRIPTION: A combine task for all replication tasks. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Progress (by Guest): Replication tasks (39)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Replication tasks CREATION DATE..: Sun, 09 Aug 2009, 12:24 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-RawIdeaBin TASK ID........: 39 (http://askmonty.org/worklog/?tid=39) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 17 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Guest - Mon, 10 Aug 2009, 16:32)=-=- Adding 1 hour for Monty's initial work on starting the architecture review. Worked 1 hour and estimate 0 hours remain (original estimate increased by 1 hour). -=-=(Psergey - Mon, 10 Aug 2009, 15:59)=-=- Re-searched and added subtasks. Worked 16 hours and estimate 0 hours remain (original estimate increased by 16 hours). -=-=(Psergey - Mon, 10 Aug 2009, 15:31)=-=- Dependency created: 39 now depends on 41 -=-=(Guest - Mon, 10 Aug 2009, 14:52)=-=- Dependency created: 39 now depends on 40 -=-=(Psergey - Sun, 09 Aug 2009, 12:27)=-=- Dependency created: 39 now depends on 36 -=-=(Psergey - Sun, 09 Aug 2009, 12:24)=-=- Dependency created: 39 now depends on 38 -=-=(Psergey - Sun, 09 Aug 2009, 12:24)=-=- Dependency created: 39 now depends on 37 DESCRIPTION: A combine task for all replication tasks. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Progress (by Psergey): Replication tasks (39)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Replication tasks CREATION DATE..: Sun, 09 Aug 2009, 12:24 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-RawIdeaBin TASK ID........: 39 (http://askmonty.org/worklog/?tid=39) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 16 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Mon, 10 Aug 2009, 15:59)=-=- Re-searched and added subtasks. Worked 16 hours and estimate 0 hours remain (original estimate increased by 16 hours). -=-=(Psergey - Mon, 10 Aug 2009, 15:31)=-=- Dependency created: 39 now depends on 41 -=-=(Guest - Mon, 10 Aug 2009, 14:52)=-=- Dependency created: 39 now depends on 40 -=-=(Psergey - Sun, 09 Aug 2009, 12:27)=-=- Dependency created: 39 now depends on 36 -=-=(Psergey - Sun, 09 Aug 2009, 12:24)=-=- Dependency created: 39 now depends on 38 -=-=(Psergey - Sun, 09 Aug 2009, 12:24)=-=- Dependency created: 39 now depends on 37 DESCRIPTION: A combine task for all replication tasks. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Progress (by Psergey): Replication tasks (39)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Replication tasks CREATION DATE..: Sun, 09 Aug 2009, 12:24 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-RawIdeaBin TASK ID........: 39 (http://askmonty.org/worklog/?tid=39) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 16 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Mon, 10 Aug 2009, 15:59)=-=- Re-searched and added subtasks. Worked 16 hours and estimate 0 hours remain (original estimate increased by 16 hours). -=-=(Psergey - Mon, 10 Aug 2009, 15:31)=-=- Dependency created: 39 now depends on 41 -=-=(Guest - Mon, 10 Aug 2009, 14:52)=-=- Dependency created: 39 now depends on 40 -=-=(Psergey - Sun, 09 Aug 2009, 12:27)=-=- Dependency created: 39 now depends on 36 -=-=(Psergey - Sun, 09 Aug 2009, 12:24)=-=- Dependency created: 39 now depends on 38 -=-=(Psergey - Sun, 09 Aug 2009, 12:24)=-=- Dependency created: 39 now depends on 37 DESCRIPTION: A combine task for all replication tasks. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Psergey): Add a mysqlbinlog option to filter certain kinds of statements (41)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to filter certain kinds of statements CREATION DATE..: Mon, 10 Aug 2009, 15:30 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-BackLog TASK ID........: 41 (http://askmonty.org/worklog/?tid=41) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Mon, 10 Aug 2009, 15:47)=-=- High-Level Specification modified. --- /tmp/wklog.41.old.13282 2009-08-10 15:47:13.000000000 +0300 +++ /tmp/wklog.41.new.13282 2009-08-10 15:47:13.000000000 +0300 @@ -2,3 +2,10 @@ - If we decide to parse the statement, SQL-verb filtering will be trivial - If we decide not to parse the statement, we still can reliably distinguish the statement by matching the first characters against a set of patterns. + +If we chose the second, we'll have to perform certain normalization before +matching the patterns: + - Remove all comments from the command + - Remove all pre-space + - Compare the string case-insensitively + - etc -=-=(Psergey - Mon, 10 Aug 2009, 15:35)=-=- High-Level Specification modified. --- /tmp/wklog.41.old.12689 2009-08-10 15:35:04.000000000 +0300 +++ /tmp/wklog.41.new.12689 2009-08-10 15:35:04.000000000 +0300 @@ -1 +1,4 @@ - +The implementation will depend on design choices made in WL#40: +- If we decide to parse the statement, SQL-verb filtering will be trivial +- If we decide not to parse the statement, we still can reliably distinguish the +statement by matching the first characters against a set of patterns. -=-=(Psergey - Mon, 10 Aug 2009, 15:31)=-=- Dependency created: 39 now depends on 41 DESCRIPTION: Add a mysqlbinlog option to filter certain kinds of statements, i.e. (syntax subject to discussion): mysqlbinlog --exclude='alter table,drop table,alter database,...' HIGH-LEVEL SPECIFICATION: The implementation will depend on design choices made in WL#40: - If we decide to parse the statement, SQL-verb filtering will be trivial - If we decide not to parse the statement, we still can reliably distinguish the statement by matching the first characters against a set of patterns. If we chose the second, we'll have to perform certain normalization before matching the patterns: - Remove all comments from the command - Remove all pre-space - Compare the string case-insensitively - etc ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Psergey): Add a mysqlbinlog option to filter certain kinds of statements (41)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to filter certain kinds of statements CREATION DATE..: Mon, 10 Aug 2009, 15:30 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-BackLog TASK ID........: 41 (http://askmonty.org/worklog/?tid=41) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Mon, 10 Aug 2009, 15:47)=-=- High-Level Specification modified. --- /tmp/wklog.41.old.13282 2009-08-10 15:47:13.000000000 +0300 +++ /tmp/wklog.41.new.13282 2009-08-10 15:47:13.000000000 +0300 @@ -2,3 +2,10 @@ - If we decide to parse the statement, SQL-verb filtering will be trivial - If we decide not to parse the statement, we still can reliably distinguish the statement by matching the first characters against a set of patterns. + +If we chose the second, we'll have to perform certain normalization before +matching the patterns: + - Remove all comments from the command + - Remove all pre-space + - Compare the string case-insensitively + - etc -=-=(Psergey - Mon, 10 Aug 2009, 15:35)=-=- High-Level Specification modified. --- /tmp/wklog.41.old.12689 2009-08-10 15:35:04.000000000 +0300 +++ /tmp/wklog.41.new.12689 2009-08-10 15:35:04.000000000 +0300 @@ -1 +1,4 @@ - +The implementation will depend on design choices made in WL#40: +- If we decide to parse the statement, SQL-verb filtering will be trivial +- If we decide not to parse the statement, we still can reliably distinguish the +statement by matching the first characters against a set of patterns. -=-=(Psergey - Mon, 10 Aug 2009, 15:31)=-=- Dependency created: 39 now depends on 41 DESCRIPTION: Add a mysqlbinlog option to filter certain kinds of statements, i.e. (syntax subject to discussion): mysqlbinlog --exclude='alter table,drop table,alter database,...' HIGH-LEVEL SPECIFICATION: The implementation will depend on design choices made in WL#40: - If we decide to parse the statement, SQL-verb filtering will be trivial - If we decide not to parse the statement, we still can reliably distinguish the statement by matching the first characters against a set of patterns. If we chose the second, we'll have to perform certain normalization before matching the patterns: - Remove all comments from the command - Remove all pre-space - Compare the string case-insensitively - etc ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Psergey): Add a mysqlbinlog option to change the used database (36)
by worklog-noreply＠askmonty.org 10 Aug '09

10 Aug '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to change the used database CREATION DATE..: Fri, 07 Aug 2009, 14:57 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 36 (http://askmonty.org/worklog/?tid=36) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Mon, 10 Aug 2009, 15:41)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.13035 2009-08-10 15:41:51.000000000 +0300 +++ /tmp/wklog.36.new.13035 2009-08-10 15:41:51.000000000 +0300 @@ -1,5 +1,7 @@ Context ------- +(See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global +overview) At the moment, the server has a replication slave option --replicate-rewrite-db="from->to" -=-=(Guest - Mon, 10 Aug 2009, 11:12)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.6580 2009-08-10 11:12:36.000000000 +0300 +++ /tmp/wklog.36.new.6580 2009-08-10 11:12:36.000000000 +0300 @@ -1,4 +1,3 @@ - Context ------- At the moment, the server has a replication slave option @@ -67,6 +66,6 @@ It will be possible to do the rewrites either on the slave ( --replicate-rewrite-db will work for all kinds of statements), or in -mysqlbinlog (adding a comment is easy and doesn't require use to parse the -statement). +mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to +parse the statement). -=-=(Psergey - Sun, 09 Aug 2009, 23:53)=-=- High-Level Specification modified. --- /tmp/wklog.36.old.13425 2009-08-09 23:53:54.000000000 +0300 +++ /tmp/wklog.36.new.13425 2009-08-09 23:53:54.000000000 +0300 @@ -1 +1,72 @@ +Context +------- +At the moment, the server has a replication slave option + + --replicate-rewrite-db="from->to" + +the option affects +- Table_map_log_event (all RBR events) +- Load_log_event (LOAD DATA) +- Query_log_event (SBR-based updates, with the usual assumption that the + statement refers to tables in current database, so that changing the current + database will make the statement to work on a table in a different database). + +What we could do +---------------- + +Option1: make mysqlbinlog accept --replicate-rewrite-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make mysqlbinlog accept --replicate-rewrite-db options and process them to the +same extent as replication slave would process --replicate-rewrite-db option. + + +Option2: Add database-agnostic RBR events and --strip-db option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Right now RBR events require a databasename. It is not possible to have RBR +event stream that won't mention which database the events are for. When I +tried to use debugger and specify empty database name, attempt to apply the +binlog resulted in this error: + +090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on +opening tables, + +We could do as follows: +- Make the server interpret empty database name in RBR event (i.e. in a + Table_map_log_event) as "use current database". Binlog slave thread + probably should not allow such events as it doesn't have a natural current + database. +- Add a mysqlbinlog --strip-db option that would + = not produce any "USE dbname" statements + = change databasename for all RBR events to be empty + +That way, mysqlbinlog output will be database-agnostic and apply to the +current database. +(this will have the usual limitations that we assume that all statements in +the binlog refer to the current database). + +Option3: Enhance database rewrite +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +If there is a need to support database change for statements that use +dbname.tablename notation and are replicated as statements (i.e. are DDL +statements and/or DML statements that are binlogged as statements), +then that could be supported as follows: + +- Make the server's parser recognize special form of comments + + /* !database-alias(oldname,newname) */ + + and save the mapping somewhere + +- Put the hooks in table open and name resolution code to use the saved + mapping. + + +Once we've done the above, it will be easy to perform a complete, +no-compromise or restrictions database name change in binary log. + +It will be possible to do the rewrites either on the slave ( +--replicate-rewrite-db will work for all kinds of statements), or in +mysqlbinlog (adding a comment is easy and doesn't require use to parse the +statement). + -=-=(Psergey - Sun, 09 Aug 2009, 12:27)=-=- Dependency created: 39 now depends on 36 -=-=(Psergey - Fri, 07 Aug 2009, 14:57)=-=- Title modified. --- /tmp/wklog.36.old.14687 2009-08-07 14:57:49.000000000 +0300 +++ /tmp/wklog.36.new.14687 2009-08-07 14:57:49.000000000 +0300 @@ -1 +1 @@ -Add a mysqlbinlog option to change the database +Add a mysqlbinlog option to change the used database DESCRIPTION: Sometimes there is a need to take a binary log and apply it to a database with a different name than the original name of the database on binlog producer. If one is using statement-based replication, he can achieve this by grepping out "USE dbname" statements out of the output of mysqlbinlog(*). With row-based replication this is no longer possible, as database name is encoded within the the BINLOG '....' statement. This task is about adding an option to mysqlbinlog that would allow to change the names of used databases in both RBR and SBR events. (*) this implies that all statements refer to tables in the current database, doesn't catch updates made inside stored functions and so forth, but still works for a practially-important subset of cases. HIGH-LEVEL SPECIFICATION: Context ------- (See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global overview) At the moment, the server has a replication slave option --replicate-rewrite-db="from->to" the option affects - Table_map_log_event (all RBR events) - Load_log_event (LOAD DATA) - Query_log_event (SBR-based updates, with the usual assumption that the statement refers to tables in current database, so that changing the current database will make the statement to work on a table in a different database). What we could do ---------------- Option1: make mysqlbinlog accept --replicate-rewrite-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Make mysqlbinlog accept --replicate-rewrite-db options and process them to the same extent as replication slave would process --replicate-rewrite-db option. Option2: Add database-agnostic RBR events and --strip-db option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Right now RBR events require a databasename. It is not possible to have RBR event stream that won't mention which database the events are for. When I tried to use debugger and specify empty database name, attempt to apply the binlog resulted in this error: 090809 17:38:44 [ERROR] Slave SQL: Error 'Table '.tablename' doesn't exist' on opening tables, We could do as follows: - Make the server interpret empty database name in RBR event (i.e. in a Table_map_log_event) as "use current database". Binlog slave thread probably should not allow such events as it doesn't have a natural current database. - Add a mysqlbinlog --strip-db option that would = not produce any "USE dbname" statements = change databasename for all RBR events to be empty That way, mysqlbinlog output will be database-agnostic and apply to the current database. (this will have the usual limitations that we assume that all statements in the binlog refer to the current database). Option3: Enhance database rewrite ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If there is a need to support database change for statements that use dbname.tablename notation and are replicated as statements (i.e. are DDL statements and/or DML statements that are binlogged as statements), then that could be supported as follows: - Make the server's parser recognize special form of comments /* !database-alias(oldname,newname) */ and save the mapping somewhere - Put the hooks in table open and name resolution code to use the saved mapping. Once we've done the above, it will be easy to perform a complete, no-compromise or restrictions database name change in binary log. It will be possible to do the rewrites either on the slave ( --replicate-rewrite-db will work for all kinds of statements), or in mysqlbinlog (adding a comment is easy and doesn't require mysqlbinlog to parse the statement). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0