November 2012 - developers - lists.mariadb.org

[Maria-developers] MDEV-520: consider parallel replication patch from taobao patches
by Kristian Nielsen 14 Jan '13

14 Jan '13

Hi xiaobin, I read through your patch found here: http://mysql.taobao.org/index.php/RLPR_for_MariaDB I am still a bit confused about the status of this patch. It would be useful if you could explain a bit about the status of the patch: - How have you tested the patch. - How are you currently using the patch? Or is it still just development/test? - What are your current plans for the patch? (You mentioned that you have some changes you are implementing at the moment). Anyway, I think I have a fair understanding of the overall idea, which seems very interesting! I will try to summarise here in a bit more detail, as well as give a few general comments on the patch. Please correct me if I misunderstood anything. ---- The basic idea is to build a set (implemented as a hash table) of every unique key in every row-event in each transaction received from the master. If two transactions have no overlapping keys, they can be applied in parallel (foreign_key_checks is set to OFF to make this work). There is a single thread that processes each event received in sequence. It checks the key set of the new transaction against the key sets of all transactions queued for each of N worker threads: - If there is no conflict with any already queued transaction, the new transaction can be queued to any worker thread (the one with fewest queued transaction is chosen). - If there is a conflict with just one thread, then the transaction can be queued for that thread - it will then execute in correct sequence with the conflicting transactions. - If there is a conflict with multiple threads, wait until the conflicting transactions have been processed. No new transactions are queued while waiting. Then there are N (default 16) threads that apply the queued transactions. Queued transactions are processsed in parallel within one thread, but transaction in different queues are processed in parallel with each other. If a transaction is received which is not using row events (statement-based replication or DDL), then we wait until all queued transactions are processed. Only after that is the new transaction queued, so that such events are processed correctly (but not in parallel, of course). Likewise, if the worker threads get too far behind, we wait with queuing any more transactions. ---- I think this is a very interesting approach, and I am happy that someone have actually tried implementing it. The biggest potential issue seems to be the performance of the thread that checks the transactions for conflicts and queues them for the appropriate thread: - Main issue will be huge transactions, if the size of hash tables of key values become too big for available memory. Maybe this could be solved by having some limit above which transactions are not tried in parallel. - There will be a lot of hash lookups of key values, all done in a single thread. This could become a limit I suppose - testing will be needed to understand this fully. But I think there will be many workloads where this will be useful. Overall, it seems to me this technique is very promising. It seems to be better at getting maximum parallelism on the slave than any of the other approaches that have been considered. As to the patch itself, it seems a bit immature in the present form, but I understood you are still working on it. Let us discuss what the plans could be for finishing it and get it ready for inclusion in MariaDB. The main criteria is that it must be reasonably complete - it must not crash or silently corrupt data in any usage. However, reasonable restrictions for what is supported is fine. (For example, we could say that LOAD DATA is not supported and give an error, but not crash or silently ignore such statement). But we can discuss this in more details. ---- Finally, a couple general comments on the code: 1. get_pk_value() does not seem to handle collations, eg. case insensitivity? It needs to understand that eg. "abab" and "ABAB" can conflict. I think this should be solved by copying in values with non-binary collations using the appropriate strxfrm() variant. 2. remote_query() retries the query in case of deadlock - but that seems wrrong, doesn't it need to retry the whole _transaction_, not just the query? 3. What about other events like User_var_event ? Will they be included in a later patch? 4. There is a my_sleep(10000) in the code when a transaction conflicts with two or more workers. It would be better to replace this with a condition variable so we can wake up as soon as the conflict is resolved. Same for some other sleeps like in wait_for_all_dml_done(). 5. Shouldn't there be some locking in dispatch_transaction()? To protect the hash lookups when checking conflicts against worker threads doing pk_hash_minus() ? 6. get_pk_value(). There seems to be a buffer overflow - st_hash_item::key is fixed size, but there seems no check against overrunning it. One solution is to just truncate if data does not fit - this can lead to false positives where potential parallelism is not exploited, but that is still safe and should be ok. Thanks! - Kristian.

5 21

Re: [Maria-developers] MDEV-318 IF (NOT) EXIST clauses for ALTER TABLE
by Sergei Golubchik 13 Jan '13

13 Jan '13

Hi, Holyfoot! On Jul 08, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-318/ > > ------------------------------------------------------------ > revno: 3552 > revision-id: holyfoot(a)askmonty.org-20120708152354-d7iu7yoqo3cohidz > parent: sanja(a)montyprogram.com-20120626184334-ptpg39ptq3t79dg5 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-318 > timestamp: Sun 2012-07-08 20:23:54 +0500 > message: > MDEV-318 IF (NOT) EXIST clauses for ALTER TABLE (MWL #252). > ALTER TABLE syntax modified and related implementations added to allow: > ALTER TABLE ADD/DROP COLUMN > ALTER TABLE ADD/DROP INDEX > ALTER TABLE ADD/DROP FOREIGN KEY > ALTER TABLE ADD/DROP PARTITION > ALTER TABLE CHANGE COLUMN > ALTER TABLE MODIFY COLUMN Please, find my comments below: > === modified file 'mysql-test/r/alter_table.result' > --- mysql-test/r/alter_table.result 2011-11-21 17:13:14 +0000 > +++ mysql-test/r/alter_table.result 2012-07-11 14:24:32 +0000 > @@ -1282,3 +1282,30 @@ > # Clean-up. > drop table t1; > End of 5.1 tests > +CREATE TABLE t1 ( > +id INT(11) NOT NULL, > +x_param INT(11) DEFAULT NULL, > +PRIMARY KEY (id), > +KEY x_param (x_param) > +); > +ALTER TABLE t1 ADD COLUMN IF NOT EXISTS id INT, > +ADD COLUMN IF NOT EXISTS lol INT AFTER id; > +Warnings: > +Note 1060 Duplicate column name 'id' > +ALTER TABLE t1 ADD COLUMN IF NOT EXISTS lol INT AFTER id; > +Warnings: > +Note 1060 Duplicate column name 'lol' > +ALTER TABLE t1 DROP COLUMN IF EXISTS lol; > +ALTER TABLE t1 DROP COLUMN IF EXISTS lol; > +Warnings: > +Note 1091 Can't DROP 'lol'; check that column/key exists > +ALTER TABLE t1 ADD KEY IF NOT EXISTS x_param(x_param); > +Warnings: > +Note 1061 Duplicate key name 'x_param' > +ALTER TABLE t1 ADD KEY IF NOT EXISTS x_param(x_param); typo? you test ADD KEY IF NOT EXISTS twice. > +Warnings: > +Note 1061 Duplicate key name 'x_param' > +ALTER TABLE t1 MODIFY IF EXISTS lol INT; > +Warnings: > +Note 1054 Unknown column 'lol' in 't1' > +DROP TABLE t1; > === modified file 'sql/field.cc' > --- sql/field.cc 2012-03-13 14:38:43 +0000 > +++ sql/field.cc 2012-07-11 14:28:40 +0000 > @@ -9182,6 +9182,7 @@ void Create_field::init_for_tmp_table(en First: please, see the last section in http://kb.askmonty.org/en/how-to-get-more-out-of-bzr-when-working-on-mariad… and do as it suggests I had to apply your patch locally and re-run bzr diff to get the function names in the diff :( > (maybe_null ? FIELDFLAG_MAYBE_NULL : 0) | > (is_unsigned ? 0 : FIELDFLAG_DECIMAL)); > vcol_info= 0; > + create_if_not_exists= FALSE; > stored_in_db= TRUE; > } > > === modified file 'sql/sql_lex.h' > --- sql/sql_lex.h 2011-12-11 16:39:33 +0000 > +++ sql/sql_lex.h 2012-07-11 14:28:40 +0000 > @@ -1816,6 +1816,7 @@ typedef struct st_lex : public Query_tab > uint16 create_view_algorithm; > uint8 create_view_check; > bool drop_if_exists, drop_temporary, local_file, one_shot_set; > + bool check_exists; /* Enabled if the IF [NOT] EXISTS specified for ALTER */ I think, you can remove drop_if_exists and use check_exists instead > bool autocommit; > bool verbose, no_write_to_binlog; > > === modified file 'sql/sql_yacc.yy' > --- sql/sql_yacc.yy 2012-06-10 09:53:06 +0000 > +++ sql/sql_yacc.yy 2012-07-11 14:28:40 +0000 > @@ -4656,8 +4659,16 @@ table_option: > ; > > opt_if_not_exists: > - /* empty */ { $$= 0; } > - | IF not EXISTS { $$=HA_LEX_CREATE_IF_NOT_EXISTS; } > + /* empty */ > + { > + Lex->check_exists= FALSE; Let's initialize check_exists either before the grammar is parsed (like in the add_create_index_prepare and in the create rule) of in the grammar (here, in the opt_if_not_exists). But don't do it twice. > + $$= 0; > + } > + | IF not EXISTS > + { > + Lex->check_exists= TRUE; > + $$=HA_LEX_CREATE_IF_NOT_EXISTS; > + } > ; > > opt_create_table_options: > @@ -5879,6 +5891,18 @@ opt_ident: > | field_ident { $$=$1.str; } > ; > > +opt_if_not_exists_ident: > + opt_if_not_exists opt_ident > + { > + LEX *lex= Lex; > + if (lex->check_exists && lex->sql_command != SQLCOM_ALTER_TABLE) > + { > + my_parse_error(ER(ER_SYNTAX_ERROR)); > + MYSQL_YYABORT; > + } > + $$= $2; > + }; > + why did you create opt_if_not_exists_ident rule, instead of adding opt_if_not_exists before ident in the alter table grammar? > opt_component: > /* empty */ { $$= null_lex_str; } > | '.' ident { $$= $2; } > @@ -6455,6 +6482,11 @@ opt_column: > | COLUMN_SYM {} > ; > > +opt_column_if_exists: > + opt_column if_exists > + {} > + ; > + same as above - why a special rule? > opt_ignore: > /* empty */ { Lex->ignore= 0;} > | IGNORE_SYM { Lex->ignore= 1;} > @@ -10096,8 +10128,16 @@ table_alias_ref: > ; > > if_exists: please, rename this old rule to opt_if_exists > - /* empty */ { $$= 0; } > - | IF EXISTS { $$= 1; } > + /* empty */ > + { > + Lex->check_exists= FALSE; > + $$= 0; > + } > + | IF EXISTS > + { > + Lex->check_exists= TRUE; > + $$= 1; > + } > ; > > opt_temporary: you didn't change CREATE INDEX and DROP INDEX, did you? and ALTER TABLE DROP PRIMARY KEY isn't covered either > === modified file 'sql/sql_table.cc' > --- sql/sql_table.cc 2012-04-05 21:07:18 +0000 > +++ sql/sql_table.cc 2012-07-11 14:28:40 +0000 > @@ -5788,6 +5788,226 @@ is_index_maintenance_unique (TABLE *tabl > > > /* > + Preparation for table creation > + > + SYNOPSIS > + handle_if_exists_option() Why did you do that in a separate function? You have basically duplicated all checks. normal alter table code checks for duplicate columns/key names and for non-existant columns/keys for DROP. this code is still in place after your patch. so, basically, you'll check for duplicates and non-existant names twice. what's the point of that? I'd expect you to take check_exists flag into account in the existing checking code and convert errors into warnings as appropriate. > + thd Thread object. > + table The altered table. > + alter_info List of columns and indexes to create > + > + DESCRIPTION > + Looks for the IF [NOT] EXISTS options, checks the states and remove items > + from the list if existing found. > + > + RETURN VALUES > + NONE > +*/ > + > +static void > +handle_if_exists_options(THD *thd, TABLE *table, Alter_info *alter_info) > +{ > + Field **f_ptr; Regards, Sergei

2 9

Re: [Maria-developers] [Commits] Rev 3435: MDEV-532: Async InnoDB commit checkpoint. in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Kristian Nielsen 14 Dec '12

14 Dec '12

Hi Serg, As we discussed under review of MDEV-232, here is a separate patch that makes InnoDB/XtraDB commit checkpointing be more asynchroneous. See MDEV-532 for further description of this task. I hope you will review this, at your convenience. - Kristian. knielsen(a)knielsen-hq.org writes: > At http://bazaar.launchpad.net/~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 3435 > revision-id: knielsen(a)knielsen-hq.org-20120914124453-zsap6hjclq3vrb6n > parent: knielsen(a)knielsen-hq.org-20120913123129-kaujy4cw0jc9o08k > committer: knielsen(a)knielsen-hq.org > branch nick: work-10.0-mdev225-181-232 > timestamp: Fri 2012-09-14 14:44:53 +0200 > message: > MDEV-532: Async InnoDB commit checkpoint. > > Make the commit checkpoint inside InnoDB be asynchroneous. > Implement a background thread in binlog to do the writing and flushing of > binlog checkpoint events to disk. > === modified file 'mysql-test/suite/binlog/r/binlog_checkpoint.result' > --- a/mysql-test/suite/binlog/r/binlog_checkpoint.result 2012-09-13 12:31:29 +0000 > +++ b/mysql-test/suite/binlog/r/binlog_checkpoint.result 2012-09-14 12:44:53 +0000 > @@ -70,8 +70,14 @@ show binlog events in 'master-bin.000003 > Log_name Pos Event_type Server_id End_log_pos Info > master-bin.000003 # Format_desc # # SERVER_VERSION, BINLOG_VERSION > master-bin.000003 # Binlog_checkpoint # # master-bin.000001 > +SET DEBUG_SYNC= "RESET"; > +SET @old_dbug= @@global.DEBUG_DBUG; > +SET GLOBAL debug_dbug="+d,binlog_background_checkpoint_processed"; > SET DEBUG_SYNC= "now SIGNAL con2_continue"; > con1 is still pending, no new binlog checkpoint should have been logged. > +SET DEBUG_SYNC= "now WAIT_FOR binlog_background_checkpoint_processed"; > +SET GLOBAL debug_dbug= @old_dbug; > +SET DEBUG_SYNC= "RESET"; > show binlog events in 'master-bin.000003' from <binlog_start>; > Log_name Pos Event_type Server_id End_log_pos Info > master-bin.000003 # Format_desc # # SERVER_VERSION, BINLOG_VERSION > > === modified file 'mysql-test/suite/binlog/r/binlog_xa_recover.result' > --- a/mysql-test/suite/binlog/r/binlog_xa_recover.result 2012-09-13 12:31:29 +0000 > +++ b/mysql-test/suite/binlog/r/binlog_xa_recover.result 2012-09-14 12:44:53 +0000 > @@ -118,7 +118,11 @@ master-bin.00000<binlog_start> # Table_m > master-bin.00000<binlog_start> # Write_rows # # table_id: # flags: STMT_END_F > master-bin.00000<binlog_start> # Xid # # COMMIT /* XID */ > SET DEBUG_SYNC= "now SIGNAL con10_cont"; > +SET @old_dbug= @@global.DEBUG_DBUG; > +SET GLOBAL debug_dbug="+d,binlog_background_checkpoint_processed"; > SET DEBUG_SYNC= "now SIGNAL con12_cont"; > +SET DEBUG_SYNC= "now WAIT_FOR binlog_background_checkpoint_processed"; > +SET GLOBAL debug_dbug= @old_dbug; > SET DEBUG_SYNC= "now SIGNAL con11_cont"; > Checking that master-bin.000004 is the last binlog checkpoint > show binlog events in 'master-bin.00000<binlog_start>' from <binlog_start>; > > === modified file 'mysql-test/suite/binlog/t/binlog_checkpoint.test' > --- a/mysql-test/suite/binlog/t/binlog_checkpoint.test 2012-09-13 12:31:29 +0000 > +++ b/mysql-test/suite/binlog/t/binlog_checkpoint.test 2012-09-14 12:44:53 +0000 > @@ -71,6 +71,12 @@ SET DEBUG_SYNC= "now WAIT_FOR con2_ready > --let $binlog_file= master-bin.000003 > --source include/show_binlog_events.inc > > +# We need to sync the test case with the background processing of the > +# commit checkpoint, otherwise we get nondeterministic results. > +SET DEBUG_SYNC= "RESET"; > +SET @old_dbug= @@global.DEBUG_DBUG; > +SET GLOBAL debug_dbug="+d,binlog_background_checkpoint_processed"; > + > SET DEBUG_SYNC= "now SIGNAL con2_continue"; > > connection con2; > @@ -78,6 +84,12 @@ reap; > > connection default; > --echo con1 is still pending, no new binlog checkpoint should have been logged. > +# Make sure commit checkpoint is processed before we check that no checkpoint > +# event has been binlogged. > +SET DEBUG_SYNC= "now WAIT_FOR binlog_background_checkpoint_processed"; > +SET GLOBAL debug_dbug= @old_dbug; > +SET DEBUG_SYNC= "RESET"; > + > --let $binlog_file= master-bin.000003 > --source include/show_binlog_events.inc > > > === modified file 'mysql-test/suite/binlog/t/binlog_xa_recover.test' > --- a/mysql-test/suite/binlog/t/binlog_xa_recover.test 2012-09-13 12:31:29 +0000 > +++ b/mysql-test/suite/binlog/t/binlog_xa_recover.test 2012-09-14 12:44:53 +0000 > @@ -14,8 +14,24 @@ CREATE TABLE t1 (a INT PRIMARY KEY, b ME > # Insert some data to force a couple binlog rotations (3), so we get some > # normal binlog checkpoints before starting the test. > INSERT INTO t1 VALUES (100, REPEAT("x", 4100)); > +# Wait for the master-bin.000002 binlog checkpoint to appear. > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000002" > +--let $field= Info > +--let $condition= = "master-bin.000002" > +--source include/wait_show_condition.inc > INSERT INTO t1 VALUES (101, REPEAT("x", 4100)); > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000003" > +--let $field= Info > +--let $condition= = "master-bin.000003" > +--source include/wait_show_condition.inc > INSERT INTO t1 VALUES (102, REPEAT("x", 4100)); > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000004" > +--let $field= Info > +--let $condition= = "master-bin.000004" > +--source include/wait_show_condition.inc > > # Now start a bunch of transactions that span multiple binlog > # files. Leave then in the state prepared-but-not-committed in the engine > @@ -153,10 +169,19 @@ SET DEBUG_SYNC= "now SIGNAL con10_cont"; > connection con10; > reap; > connection default; > + > +# We need to sync the test case with the background processing of the > +# commit checkpoint, otherwise we get nondeterministic results. > +SET @old_dbug= @@global.DEBUG_DBUG; > +SET GLOBAL debug_dbug="+d,binlog_background_checkpoint_processed"; > + > SET DEBUG_SYNC= "now SIGNAL con12_cont"; > connection con12; > reap; > connection default; > +SET DEBUG_SYNC= "now WAIT_FOR binlog_background_checkpoint_processed"; > +SET GLOBAL debug_dbug= @old_dbug; > + > SET DEBUG_SYNC= "now SIGNAL con11_cont"; > connection con11; > reap; > @@ -210,7 +235,20 @@ RESET MASTER; > # crash recovery fails due to the error insert used for previous test. > INSERT INTO t1 VALUES (21, REPEAT("x", 4100)); > INSERT INTO t1 VALUES (22, REPEAT("x", 4100)); > +# Wait for the master-bin.000003 binlog checkpoint to appear. > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000003" > +--let $field= Info > +--let $condition= = "master-bin.000003" > +--source include/wait_show_condition.inc > INSERT INTO t1 VALUES (23, REPEAT("x", 4100)); > +# Wait for the last (master-bin.000004) binlog checkpoint to appear. > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000004" > +--let $field= Info > +--let $condition= = "master-bin.000004" > +--source include/wait_show_condition.inc > + > --write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > wait-binlog_xa_recover.test > EOF > > === modified file 'mysql-test/suite/perfschema/r/all_instances.result' > --- a/mysql-test/suite/perfschema/r/all_instances.result 2012-06-22 09:46:28 +0000 > +++ b/mysql-test/suite/perfschema/r/all_instances.result 2012-09-14 12:44:53 +0000 > @@ -76,6 +76,7 @@ wait/synch/mutex/sql/Master_info::run_lo > wait/synch/mutex/sql/Master_info::sleep_lock > wait/synch/mutex/sql/MDL_map::mutex > wait/synch/mutex/sql/MDL_wait::LOCK_wait_status > +wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_binlog_thread > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_index > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_xid_list > wait/synch/mutex/sql/MYSQL_RELAY_LOG::LOCK_index > @@ -129,6 +130,8 @@ wait/synch/cond/sql/Master_info::sleep_c > wait/synch/cond/sql/Master_info::start_cond > wait/synch/cond/sql/Master_info::stop_cond > wait/synch/cond/sql/MDL_context::COND_wait_status > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread_end > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_queue_busy > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_xid_list > wait/synch/cond/sql/MYSQL_BIN_LOG::update_cond > > === modified file 'mysql-test/suite/perfschema/r/relaylog.result' > --- a/mysql-test/suite/perfschema/r/relaylog.result 2012-06-22 09:46:28 +0000 > +++ b/mysql-test/suite/perfschema/r/relaylog.result 2012-09-14 12:44:53 +0000 > @@ -56,8 +56,11 @@ where event_name like "%MYSQL_BIN_LOG%" > and event_name not like "%MYSQL_BIN_LOG::update_cond" > order by event_name; > EVENT_NAME COUNT_STAR > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread NONE > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread_end NONE > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_queue_busy NONE > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_xid_list NONE > +wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_binlog_thread MANY > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_index MANY > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_xid_list MANY > "Expect no slave relay log" > @@ -131,8 +134,11 @@ where event_name like "%MYSQL_BIN_LOG%" > and event_name not like "%MYSQL_BIN_LOG::update_cond" > order by event_name; > EVENT_NAME COUNT_STAR > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread MANY > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread_end NONE > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_queue_busy NONE > -wait/synch/cond/sql/MYSQL_BIN_LOG::COND_xid_list NONE > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_xid_list MANY > +wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_binlog_thread MANY > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_index MANY > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_xid_list MANY > "Expect a slave relay log" > > === modified file 'sql/debug_sync.cc' > --- a/sql/debug_sync.cc 2012-03-28 17:26:00 +0000 > +++ b/sql/debug_sync.cc 2012-09-14 12:44:53 +0000 > @@ -984,6 +984,7 @@ static bool debug_sync_eval_action(THD * > DBUG_ENTER("debug_sync_eval_action"); > DBUG_ASSERT(thd); > DBUG_ASSERT(action_str); > + DBUG_PRINT("debug_sync", ("action_str='%s'", action_str)); > > /* > Get debug sync point name. Or a special command. > > === modified file 'sql/log.cc' > --- a/sql/log.cc 2012-09-13 12:31:29 +0000 > +++ b/sql/log.cc 2012-09-14 12:44:53 +0000 > @@ -53,6 +53,7 @@ > #include "rpl_handler.h" > #include "debug_sync.h" > #include "sql_show.h" > +#include "my_pthread.h" > > /* max size of the log message */ > #define MAX_LOG_BUFFER_SIZE 1024 > @@ -106,6 +107,14 @@ static SHOW_VAR binlog_status_vars_detai > {NullS, NullS, SHOW_LONG} > }; > > +/* Variables for the binlog background thread. */ > +static bool binlog_thread_started= false; > +static bool binlog_background_thread_stop= false; > +static MYSQL_BIN_LOG::xid_count_per_binlog * > + binlog_background_thread_queue= NULL; > + > +static bool start_binlog_background_thread(); > + > > /** > purge logs, master and slave sides both, related error code > @@ -2957,12 +2966,27 @@ void MYSQL_BIN_LOG::cleanup() > my_free(b); > } > > + /* Wait for the binlog thread to stop. */ > + if (!is_relay_log && binlog_thread_started) > + { > + mysql_mutex_lock(&LOCK_binlog_thread); > + binlog_background_thread_stop= true; > + mysql_cond_signal(&COND_binlog_thread); > + while (binlog_background_thread_stop) > + mysql_cond_wait(&COND_binlog_thread_end, &LOCK_binlog_thread); > + mysql_mutex_unlock(&LOCK_binlog_thread); > + binlog_thread_started= false; > + } > + > mysql_mutex_destroy(&LOCK_log); > mysql_mutex_destroy(&LOCK_index); > mysql_mutex_destroy(&LOCK_xid_list); > + mysql_mutex_destroy(&LOCK_binlog_thread); > mysql_cond_destroy(&update_cond); > mysql_cond_destroy(&COND_queue_busy); > mysql_cond_destroy(&COND_xid_list); > + mysql_cond_destroy(&COND_binlog_thread); > + mysql_cond_destroy(&COND_binlog_thread_end); > } > DBUG_VOID_RETURN; > } > @@ -2988,6 +3012,11 @@ void MYSQL_BIN_LOG::init_pthread_objects > mysql_cond_init(m_key_update_cond, &update_cond, 0); > mysql_cond_init(m_key_COND_queue_busy, &COND_queue_busy, 0); > mysql_cond_init(key_BINLOG_COND_xid_list, &COND_xid_list, 0); > + > + mysql_mutex_init(key_BINLOG_LOCK_binlog_thread, > + &LOCK_binlog_thread, MY_MUTEX_INIT_FAST); > + mysql_cond_init(key_BINLOG_COND_binlog_thread, &COND_binlog_thread, 0); > + mysql_cond_init(key_BINLOG_COND_binlog_thread_end, &COND_binlog_thread_end, 0); > } > > > @@ -3085,6 +3114,10 @@ bool MYSQL_BIN_LOG::open(const char *log > DBUG_ENTER("MYSQL_BIN_LOG::open"); > DBUG_PRINT("enter",("log_type: %d",(int) log_type_arg)); > > + if (!is_relay_log && !binlog_thread_started && > + start_binlog_background_thread()) > + DBUG_RETURN(1); > + > if (init_and_set_log_file_name(log_name, new_name, log_type_arg, > io_cache_type_arg)) > { > @@ -5540,11 +5573,7 @@ bool general_log_write(THD *thd, enum en > } > > > -/* > - I would like to make this function static, but this causes compiler warnings > - when it is declared as friend function in log.h. > -*/ > -void > +static void > binlog_checkpoint_callback(void *cookie) > { > MYSQL_BIN_LOG::xid_count_per_binlog *entry= > @@ -8116,9 +8145,128 @@ int TC_LOG_BINLOG::unlog(ulong cookie, m > void > TC_LOG_BINLOG::commit_checkpoint_notify(void *cookie) > { > - mark_xid_done(((xid_count_per_binlog *)cookie)->binlog_id, true); > + xid_count_per_binlog *entry= static_cast<xid_count_per_binlog *>(cookie); > + mysql_mutex_lock(&LOCK_binlog_thread); > + entry->next_in_queue= binlog_background_thread_queue; > + binlog_background_thread_queue= entry; > + mysql_cond_signal(&COND_binlog_thread); > + mysql_mutex_unlock(&LOCK_binlog_thread); > } > > +/* > + Binlog service thread. > + > + This thread is used to log binlog checkpoints in the background, rather than > + in the context of random storage engine threads that happen to call > + commit_checkpoint_notify_ha() and may not like the delays while syncing > + binlog to disk or may not be setup with all my_thread_init() and other > + necessary stuff. > + > + In the future, this thread could also be used to do log rotation in the > + background, which could elimiate all stalls around binlog rotations. > +*/ > +pthread_handler_t > +binlog_background_thread(void *arg __attribute__((unused))) > +{ > + bool stop; > + MYSQL_BIN_LOG::xid_count_per_binlog *queue, *next; > + THD *thd; > + > + my_thread_init(); > + thd= new THD; > + thd->system_thread= SYSTEM_THREAD_BINLOG_BACKGROUND; > + my_pthread_setspecific_ptr(THR_THD, thd); > + mysql_mutex_lock(&LOCK_thread_count); > + thd->thread_id= thread_id++; > + mysql_mutex_unlock(&LOCK_thread_count); > + > + for (;;) > + { > + /* > + Wait until there is something in the queue to process, or we are asked > + to shut down. > + */ > + thd_proc_info(thd, "Waiting for background binlog tasks"); > + mysql_mutex_lock(&mysql_bin_log.LOCK_binlog_thread); > + for (;;) > + { > + stop= binlog_background_thread_stop; > + queue= binlog_background_thread_queue; > + if (stop || queue) > + break; > + mysql_cond_wait(&mysql_bin_log.COND_binlog_thread, > + &mysql_bin_log.LOCK_binlog_thread); > + } > + /* Grab the queue, if any. */ > + binlog_background_thread_queue= NULL; > + mysql_mutex_unlock(&mysql_bin_log.LOCK_binlog_thread); > + > + /* Process any incoming commit_checkpoint_notify() calls. */ > + while (queue) > + { > + thd_proc_info(thd, "Processing binlog checkpoint notification"); > + /* Grab next pointer first, as mark_xid_done() may free the element. */ > + next= queue->next_in_queue; > + mysql_bin_log.mark_xid_done(queue->binlog_id, true); > + queue= next; > + > + DBUG_EXECUTE_IF("binlog_background_checkpoint_processed", > + DBUG_ASSERT(!debug_sync_set_action( > + thd, > + STRING_WITH_LEN("now SIGNAL binlog_background_checkpoint_processed"))); > + ); > + } > + > + if (stop) > + break; > + } > + > + thd_proc_info(thd, "Stopping binlog background thread"); > + > + mysql_mutex_lock(&LOCK_thread_count); > + delete thd; > + mysql_mutex_unlock(&LOCK_thread_count); > + > + my_thread_end(); > + > + /* Signal that we are (almost) stopped. */ > + mysql_mutex_lock(&mysql_bin_log.LOCK_binlog_thread); > + binlog_background_thread_stop= false; > + mysql_cond_signal(&mysql_bin_log.COND_binlog_thread_end); > + mysql_mutex_unlock(&mysql_bin_log.LOCK_binlog_thread); > + > + return 0; > +} > + > +#ifdef HAVE_PSI_INTERFACE > +static PSI_thread_key key_thread_binlog; > + > +static PSI_thread_info all_binlog_threads[]= > +{ > + { &key_thread_binlog, "binlog_background", PSI_FLAG_GLOBAL}, > +}; > +#endif /* HAVE_PSI_INTERFACE */ > + > +static bool > +start_binlog_background_thread() > +{ > + pthread_t th; > + > +#ifdef HAVE_PSI_INTERFACE > + if (PSI_server) > + PSI_server->register_thread("sql", all_binlog_threads, > + array_elements(all_binlog_threads)); > +#endif > + > + if (mysql_thread_create(key_thread_binlog, &th, NULL, > + binlog_background_thread, NULL)) > + return 1; > + > + binlog_thread_started= true; > + return 0; > +} > + > + > int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > IO_CACHE *first_log, > Format_description_log_event *fdle) > > === modified file 'sql/log.h' > --- a/sql/log.h 2012-09-13 12:31:29 +0000 > +++ b/sql/log.h 2012-09-14 12:44:53 +0000 > @@ -395,8 +395,6 @@ class MYSQL_QUERY_LOG: public MYSQL_LOG > #define BINLOG_COOKIE_IS_DUMMY(c) \ > ( ((ulong)(c)>>1) == BINLOG_COOKIE_DUMMY_ID ) > > -void binlog_checkpoint_callback(void *cookie); > - > class binlog_cache_mngr; > class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG > { > @@ -451,27 +449,6 @@ class MYSQL_BIN_LOG: public TC_LOG, priv > }; > > /* > - A list of struct xid_count_per_binlog is used to keep track of how many > - XIDs are in prepared, but not committed, state in each binlog. And how > - many commit_checkpoint_request()'s are pending. > - > - When count drops to zero in a binlog after rotation, it means that there > - are no more XIDs in prepared state, so that binlog is no longer needed > - for XA crash recovery, and we can log a new binlog checkpoint event. > - > - The list is protected against simultaneous access from multiple > - threads by LOCK_xid_list. > - */ > - struct xid_count_per_binlog : public ilink { > - char *binlog_name; > - uint binlog_name_len; > - ulong binlog_id; > - /* Total prepared XIDs and pending checkpoint requests in this binlog. */ > - long xid_count; > - xid_count_per_binlog(); /* Give link error if constructor used. */ > - }; > - I_List<xid_count_per_binlog> binlog_xid_count_list; > - /* > When this is set, a RESET MASTER is in progress. > > Then we should not write any binlog checkpoints into the binlog (that > @@ -480,7 +457,6 @@ class MYSQL_BIN_LOG: public TC_LOG, priv > checkpoint arrives - when all have arrived, RESET MASTER will complete. > */ > bool reset_master_pending; > - friend void binlog_checkpoint_callback(void *cookie); > > /* LOCK_log and LOCK_index are inited by init_pthread_objects() */ > mysql_mutex_t LOCK_index; > @@ -553,10 +529,35 @@ class MYSQL_BIN_LOG: public TC_LOG, priv > int write_transaction_or_stmt(group_commit_entry *entry); > bool write_transaction_to_binlog_events(group_commit_entry *entry); > void trx_group_commit_leader(group_commit_entry *leader); > - void mark_xid_done(ulong cookie, bool write_checkpoint); > - void mark_xids_active(ulong cookie, uint xid_count); > > public: > + /* > + A list of struct xid_count_per_binlog is used to keep track of how many > + XIDs are in prepared, but not committed, state in each binlog. And how > + many commit_checkpoint_request()'s are pending. > + > + When count drops to zero in a binlog after rotation, it means that there > + are no more XIDs in prepared state, so that binlog is no longer needed > + for XA crash recovery, and we can log a new binlog checkpoint event. > + > + The list is protected against simultaneous access from multiple > + threads by LOCK_xid_list. > + */ > + struct xid_count_per_binlog : public ilink { > + char *binlog_name; > + uint binlog_name_len; > + ulong binlog_id; > + /* Total prepared XIDs and pending checkpoint requests in this binlog. */ > + long xid_count; > + /* For linking in requests to the binlog background thread. */ > + xid_count_per_binlog *next_in_queue; > + xid_count_per_binlog(); /* Give link error if constructor used. */ > + }; > + I_List<xid_count_per_binlog> binlog_xid_count_list; > + mysql_mutex_t LOCK_binlog_thread; > + mysql_cond_t COND_binlog_thread; > + mysql_cond_t COND_binlog_thread_end; > + > using MYSQL_LOG::generate_name; > using MYSQL_LOG::is_open; > > @@ -712,6 +713,8 @@ class MYSQL_BIN_LOG: public TC_LOG, priv > bool appendv(const char* buf,uint len,...); > bool append(Log_event* ev); > > + void mark_xids_active(ulong cookie, uint xid_count); > + void mark_xid_done(ulong cookie, bool write_checkpoint); > void make_log_name(char* buf, const char* log_ident); > bool is_active(const char* log_file_name); > bool can_purge_log(const char *log_file_name); > > === modified file 'sql/mysqld.cc' > --- a/sql/mysqld.cc 2012-09-13 12:31:29 +0000 > +++ b/sql/mysqld.cc 2012-09-14 12:44:53 +0000 > @@ -724,6 +724,7 @@ PSI_mutex_key key_LOCK_des_key_file; > #endif /* HAVE_OPENSSL */ > > PSI_mutex_key key_BINLOG_LOCK_index, key_BINLOG_LOCK_xid_list, > + key_BINLOG_LOCK_binlog_thread, > key_delayed_insert_mutex, key_hash_filo_lock, key_LOCK_active_mi, > key_LOCK_connection_count, key_LOCK_crypt, key_LOCK_delayed_create, > key_LOCK_delayed_insert, key_LOCK_delayed_status, key_LOCK_error_log, > @@ -766,6 +767,7 @@ static PSI_mutex_info all_server_mutexes > > { &key_BINLOG_LOCK_index, "MYSQL_BIN_LOG::LOCK_index", 0}, > { &key_BINLOG_LOCK_xid_list, "MYSQL_BIN_LOG::LOCK_xid_list", 0}, > + { &key_BINLOG_LOCK_binlog_thread, "MYSQL_BIN_LOG::LOCK_binlog_thread", 0}, > { &key_RELAYLOG_LOCK_index, "MYSQL_RELAY_LOG::LOCK_index", 0}, > { &key_delayed_insert_mutex, "Delayed_insert::mutex", 0}, > { &key_hash_filo_lock, "hash_filo::lock", 0}, > @@ -834,6 +836,7 @@ PSI_cond_key key_PAGE_cond, key_COND_act > #endif /* HAVE_MMAP */ > > PSI_cond_key key_BINLOG_COND_xid_list, key_BINLOG_update_cond, > + key_BINLOG_COND_binlog_thread, key_BINLOG_COND_binlog_thread_end, > key_COND_cache_status_changed, key_COND_manager, > key_COND_rpl_status, key_COND_server_started, > key_delayed_insert_cond, key_delayed_insert_cond_client, > @@ -863,6 +866,8 @@ static PSI_cond_info all_server_conds[]= > #endif /* HAVE_MMAP */ > { &key_BINLOG_COND_xid_list, "MYSQL_BIN_LOG::COND_xid_list", 0}, > { &key_BINLOG_update_cond, "MYSQL_BIN_LOG::update_cond", 0}, > + { &key_BINLOG_COND_binlog_thread, "MYSQL_BIN_LOG::COND_binlog_thread", 0}, > + { &key_BINLOG_COND_binlog_thread_end, "MYSQL_BIN_LOG::COND_binlog_thread_end", 0}, > { &key_BINLOG_COND_queue_busy, "MYSQL_BIN_LOG::COND_queue_busy", 0}, > { &key_RELAYLOG_update_cond, "MYSQL_RELAY_LOG::update_cond", 0}, > { &key_RELAYLOG_COND_queue_busy, "MYSQL_RELAY_LOG::COND_queue_busy", 0}, > > === modified file 'sql/mysqld.h' > --- a/sql/mysqld.h 2012-09-13 12:31:29 +0000 > +++ b/sql/mysqld.h 2012-09-14 12:44:53 +0000 > @@ -226,6 +226,7 @@ extern PSI_mutex_key key_LOCK_des_key_fi > #endif > > extern PSI_mutex_key key_BINLOG_LOCK_index, key_BINLOG_LOCK_xid_list, > + key_BINLOG_LOCK_binlog_thread, > key_delayed_insert_mutex, key_hash_filo_lock, key_LOCK_active_mi, > key_LOCK_connection_count, key_LOCK_crypt, key_LOCK_delayed_create, > key_LOCK_delayed_insert, key_LOCK_delayed_status, key_LOCK_error_log, > @@ -257,6 +258,7 @@ extern PSI_cond_key key_PAGE_cond, key_C > #endif /* HAVE_MMAP */ > > extern PSI_cond_key key_BINLOG_COND_xid_list, key_BINLOG_update_cond, > + key_BINLOG_COND_binlog_thread, key_BINLOG_COND_binlog_thread_end, > key_COND_cache_status_changed, key_COND_manager, > key_COND_rpl_status, key_COND_server_started, > key_delayed_insert_cond, key_delayed_insert_cond_client, > > === modified file 'sql/rpl_rli.cc' > --- a/sql/rpl_rli.cc 2012-09-13 12:31:29 +0000 > +++ b/sql/rpl_rli.cc 2012-09-14 12:44:53 +0000 > @@ -58,6 +58,7 @@ Relay_log_info::Relay_log_info(bool is_s > { > DBUG_ENTER("Relay_log_info::Relay_log_info"); > > + relay_log.is_relay_log= TRUE; > #ifdef HAVE_PSI_INTERFACE > relay_log.set_psi_keys(key_RELAYLOG_LOCK_index, > key_RELAYLOG_update_cond, > @@ -206,8 +207,6 @@ a file name for --relay-log-index option > name_warning_sent= 1; > } > > - rli->relay_log.is_relay_log= TRUE; > - > /* > note, that if open() fails, we'll still have index file open > but a destructor will take care of that > > === modified file 'sql/sql_class.h' > --- a/sql/sql_class.h 2012-09-13 12:31:29 +0000 > +++ b/sql/sql_class.h 2012-09-14 12:44:53 +0000 > @@ -1244,7 +1244,8 @@ enum enum_thread_type > SYSTEM_THREAD_SLAVE_SQL= 4, > SYSTEM_THREAD_NDBCLUSTER_BINLOG= 8, > SYSTEM_THREAD_EVENT_SCHEDULER= 16, > - SYSTEM_THREAD_EVENT_WORKER= 32 > + SYSTEM_THREAD_EVENT_WORKER= 32, > + SYSTEM_THREAD_BINLOG_BACKGROUND= 64 > }; > > inline char const * > > === modified file 'storage/innobase/handler/ha_innodb.cc' > --- a/storage/innobase/handler/ha_innodb.cc 2012-09-13 12:31:29 +0000 > +++ b/storage/innobase/handler/ha_innodb.cc 2012-09-14 12:44:53 +0000 > @@ -106,6 +106,7 @@ static ulong commit_threads = 0; > static mysql_mutex_t commit_threads_m; > static mysql_cond_t commit_cond; > static mysql_mutex_t commit_cond_m; > +static mysql_mutex_t pending_checkpoint_mutex; > static bool innodb_inited = 0; > > #define INSIDE_HA_INNOBASE_CC > @@ -222,11 +223,13 @@ static mysql_pfs_key_t innobase_share_mu > static mysql_pfs_key_t commit_threads_m_key; > static mysql_pfs_key_t commit_cond_mutex_key; > static mysql_pfs_key_t commit_cond_key; > +static mysql_pfs_key_t pending_checkpoint_mutex_key; > > static PSI_mutex_info all_pthread_mutexes[] = { > {&commit_threads_m_key, "commit_threads_m", 0}, > {&commit_cond_mutex_key, "commit_cond_mutex", 0}, > - {&innobase_share_mutex_key, "innobase_share_mutex", 0} > + {&innobase_share_mutex_key, "innobase_share_mutex", 0}, > + {&pending_checkpoint_mutex_key, "pending_checkpoint_mutex", 0} > }; > > static PSI_cond_info all_innodb_conds[] = { > @@ -2592,6 +2595,9 @@ innobase_init( > mysql_mutex_init(commit_cond_mutex_key, > &commit_cond_m, MY_MUTEX_INIT_FAST); > mysql_cond_init(commit_cond_key, &commit_cond, NULL); > + mysql_mutex_init(pending_checkpoint_mutex_key, > + &pending_checkpoint_mutex, > + MY_MUTEX_INIT_FAST); > innodb_inited= 1; > #ifdef MYSQL_DYNAMIC_PLUGIN > if (innobase_hton != p) { > @@ -2639,6 +2645,7 @@ innobase_end( > mysql_mutex_destroy(&commit_threads_m); > mysql_mutex_destroy(&commit_cond_m); > mysql_cond_destroy(&commit_cond); > + mysql_mutex_destroy(&pending_checkpoint_mutex); > } > > DBUG_RETURN(err); > @@ -3008,6 +3015,16 @@ innobase_rollback_trx( > DBUG_RETURN(convert_error_code_to_mysql(error, 0, NULL)); > } > > + > +struct pending_checkpoint { > + struct pending_checkpoint *next; > + handlerton *hton; > + void *cookie; > + ib_uint64_t lsn; > +}; > +static struct pending_checkpoint *pending_checkpoint_list; > +static struct pending_checkpoint *pending_checkpoint_list_end; > + > /*****************************************************************//** > Handle a commit checkpoint request from server layer. > We simply flush the redo log immediately and do the notify call.*/ > @@ -3017,8 +3034,113 @@ innobase_checkpoint_request( > handlerton *hton, > void *cookie) > { > - log_buffer_flush_to_disk(); > - commit_checkpoint_notify_ha(hton, cookie); > + ib_uint64_t lsn; > + ib_uint64_t flush_lsn; > + struct pending_checkpoint * entry; > + > + /* Do the allocation outside of lock to reduce contention. The normal > + case is that not everything is flushed, so we will need to enqueue. */ > + entry = static_cast<struct pending_checkpoint *> > + (my_malloc(sizeof(*entry), MYF(MY_WME))); > + if (!entry) { > + sql_print_error("Failed to allocate %u bytes." > + " Commit checkpoint will be skipped.", > + static_cast<unsigned>(sizeof(*entry))); > + return; > + } > + > + entry->next = NULL; > + entry->hton = hton; > + entry->cookie = cookie; > + > + mysql_mutex_lock(&pending_checkpoint_mutex); > + lsn = log_get_lsn(); > + flush_lsn = log_get_flush_lsn(); > + if (lsn > flush_lsn) { > + /* Put the request in queue. > + When the log gets flushed past the lsn, we will remove the > + entry from the queue and notify the upper layer. */ > + entry->lsn = lsn; > + if (pending_checkpoint_list_end) { > + pending_checkpoint_list_end->next = entry; > + } else { > + pending_checkpoint_list = entry; > + } > + pending_checkpoint_list_end = entry; > + entry = NULL; > + } > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + > + if (entry) { > + /* We are already flushed. Notify the checkpoint immediately. */ > + commit_checkpoint_notify_ha(entry->hton, entry->cookie); > + my_free(entry); > + } > +} > + > +/*****************************************************************//** > +Log code calls this whenever log has been written and/or flushed up > +to a new position. We use this to notify upper layer of a new commit > +checkpoint when necessary.*/ > +extern "C" UNIV_INTERN > +void > +innobase_mysql_log_notify( > +/*===============*/ > + ib_uint64_t write_lsn, /*!< in: LSN written to log file */ > + ib_uint64_t flush_lsn) /*!< in: LSN flushed to disk */ > +{ > + struct pending_checkpoint * pending; > + struct pending_checkpoint * entry; > + struct pending_checkpoint * last_ready; > + > + /* It is safe to do a quick check for NULL first without lock. > + Even if we should race, we will at most skip one checkpoint and > + take the next one, which is harmless. */ > + if (!pending_checkpoint_list) > + return; > + > + mysql_mutex_lock(&pending_checkpoint_mutex); > + pending = pending_checkpoint_list; > + if (!pending) > + { > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + return; > + } > + > + last_ready = NULL; > + for (entry = pending; entry != NULL; entry = entry -> next) > + { > + if (entry->lsn > flush_lsn) > + break; > + last_ready = entry; > + } > + > + if (last_ready) > + { > + /* We found some pending checkpoints that are now flushed to > + disk. So remove them from the list. */ > + pending_checkpoint_list = entry; > + if (!entry) > + pending_checkpoint_list_end = NULL; > + } > + > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + > + if (!last_ready) > + return; > + > + /* Now that we have released the lock, notify upper layer about all > + commit checkpoints that have now completed. */ > + for (;;) { > + entry = pending; > + pending = pending->next; > + > + commit_checkpoint_notify_ha(entry->hton, entry->cookie); > + > + my_free(entry); > + if (entry == last_ready) > + break; > + } > } > > /*****************************************************************//** > > === modified file 'storage/innobase/include/ha_prototypes.h' > --- a/storage/innobase/include/ha_prototypes.h 2011-04-26 17:55:52 +0000 > +++ b/storage/innobase/include/ha_prototypes.h 2012-09-14 12:44:53 +0000 > @@ -136,6 +136,17 @@ innobase_mysql_print_thd( > uint max_query_len); /*!< in: max query length to print, or 0 to > use the default max length */ > > +/*****************************************************************//** > +Log code calls this whenever log has been written and/or flushed up > +to a new position. We use this to notify upper layer of a new commit > +checkpoint when necessary.*/ > +UNIV_INTERN > +void > +innobase_mysql_log_notify( > +/*===============*/ > + ib_uint64_t write_lsn, /*!< in: LSN written to log file */ > + ib_uint64_t flush_lsn); /*!< in: LSN flushed to disk */ > + > /**************************************************************//** > Converts a MySQL type to an InnoDB type. Note that this function returns > the 'mtype' of InnoDB. InnoDB differentiates between MySQL's old <= 4.1 > > === modified file 'storage/innobase/include/log0log.h' > --- a/storage/innobase/include/log0log.h 2012-06-07 13:44:26 +0000 > +++ b/storage/innobase/include/log0log.h 2012-09-14 12:44:53 +0000 > @@ -151,6 +151,13 @@ UNIV_INLINE > ib_uint64_t > log_get_lsn(void); > /*=============*/ > +/************************************************************//** > +Gets the last lsn that is fully flushed to disk. > +@return last flushed lsn */ > +UNIV_INLINE > +ib_uint64_t > +log_get_flush_lsn(void); > +/*=============*/ > /**************************************************************** > Gets the log group capacity. It is OK to read the value without > holding log_sys->mutex because it is constant. > > === modified file 'storage/innobase/include/log0log.ic' > --- a/storage/innobase/include/log0log.ic 2011-04-05 07:18:43 +0000 > +++ b/storage/innobase/include/log0log.ic 2012-09-14 12:44:53 +0000 > @@ -411,6 +411,25 @@ log_get_lsn(void) > return(lsn); > } > > +/************************************************************//** > +Gets the last lsn that is fully flushed to disk. > +@return last flushed lsn */ > +UNIV_INLINE > +ib_uint64_t > +log_get_flush_lsn(void) > +/*=============*/ > +{ > + ib_uint64_t lsn; > + > + mutex_enter(&(log_sys->mutex)); > + > + lsn = log_sys->flushed_to_disk_lsn; > + > + mutex_exit(&(log_sys->mutex)); > + > + return(lsn); > +} > + > /**************************************************************** > Gets the log group capacity. It is OK to read the value without > holding log_sys->mutex because it is constant. > > === modified file 'storage/innobase/log/log0log.c' > --- a/storage/innobase/log/log0log.c 2012-03-21 03:48:12 +0000 > +++ b/storage/innobase/log/log0log.c 2012-09-14 12:44:53 +0000 > @@ -1353,6 +1353,8 @@ log_write_up_to( > ulint loop_count = 0; > #endif /* UNIV_DEBUG */ > ulint unlock; > + ib_uint64_t write_lsn; > + ib_uint64_t flush_lsn; > > if (recv_no_ibuf_operations) { > /* Recovery is running and no operations on the log files are > @@ -1530,8 +1532,13 @@ log_write_up_to( > > log_flush_do_unlocks(unlock); > > + write_lsn = log_sys->write_lsn; > + flush_lsn = log_sys->flushed_to_disk_lsn; > + > mutex_exit(&(log_sys->mutex)); > > + innobase_mysql_log_notify(write_lsn, flush_lsn); > + > return; > > do_waits: > > === modified file 'storage/xtradb/handler/ha_innodb.cc' > --- a/storage/xtradb/handler/ha_innodb.cc 2012-09-13 12:31:29 +0000 > +++ b/storage/xtradb/handler/ha_innodb.cc 2012-09-14 12:44:53 +0000 > @@ -120,6 +120,7 @@ static ulong commit_threads = 0; > static mysql_mutex_t commit_threads_m; > static mysql_cond_t commit_cond; > static mysql_mutex_t commit_cond_m; > +static mysql_mutex_t pending_checkpoint_mutex; > static bool innodb_inited = 0; > > > @@ -253,11 +254,13 @@ static mysql_pfs_key_t innobase_share_mu > static mysql_pfs_key_t commit_threads_m_key; > static mysql_pfs_key_t commit_cond_mutex_key; > static mysql_pfs_key_t commit_cond_key; > +static mysql_pfs_key_t pending_checkpoint_mutex_key; > > static PSI_mutex_info all_pthread_mutexes[] = { > {&commit_threads_m_key, "commit_threads_m", 0}, > {&commit_cond_mutex_key, "commit_cond_mutex", 0}, > - {&innobase_share_mutex_key, "innobase_share_mutex", 0} > + {&innobase_share_mutex_key, "innobase_share_mutex", 0}, > + {&pending_checkpoint_mutex_key, "pending_checkpoint_mutex", 0} > }; > > static PSI_cond_info all_innodb_conds[] = { > @@ -3060,6 +3063,9 @@ innobase_init( > mysql_mutex_init(commit_cond_mutex_key, > &commit_cond_m, MY_MUTEX_INIT_FAST); > mysql_cond_init(commit_cond_key, &commit_cond, NULL); > + mysql_mutex_init(pending_checkpoint_mutex_key, > + &pending_checkpoint_mutex, > + MY_MUTEX_INIT_FAST); > innodb_inited= 1; > #ifdef MYSQL_DYNAMIC_PLUGIN > if (innobase_hton != p) { > @@ -3107,6 +3113,7 @@ innobase_end( > mysql_mutex_destroy(&commit_threads_m); > mysql_mutex_destroy(&commit_cond_m); > mysql_cond_destroy(&commit_cond); > + mysql_mutex_destroy(&pending_checkpoint_mutex); > } > > DBUG_RETURN(err); > @@ -3500,6 +3507,16 @@ innobase_rollback_trx( > DBUG_RETURN(convert_error_code_to_mysql(error, 0, NULL)); > } > > + > +struct pending_checkpoint { > + struct pending_checkpoint *next; > + handlerton *hton; > + void *cookie; > + ib_uint64_t lsn; > +}; > +static struct pending_checkpoint *pending_checkpoint_list; > +static struct pending_checkpoint *pending_checkpoint_list_end; > + > /*****************************************************************//** > Handle a commit checkpoint request from server layer. > We simply flush the redo log immediately and do the notify call.*/ > @@ -3509,8 +3526,113 @@ innobase_checkpoint_request( > handlerton *hton, > void *cookie) > { > - log_buffer_flush_to_disk(); > - commit_checkpoint_notify_ha(hton, cookie); > + ib_uint64_t lsn; > + ib_uint64_t flush_lsn; > + struct pending_checkpoint * entry; > + > + /* Do the allocation outside of lock to reduce contention. The normal > + case is that not everything is flushed, so we will need to enqueue. */ > + entry = static_cast<struct pending_checkpoint *> > + (my_malloc(sizeof(*entry), MYF(MY_WME))); > + if (!entry) { > + sql_print_error("Failed to allocate %u bytes." > + " Commit checkpoint will be skipped.", > + static_cast<unsigned>(sizeof(*entry))); > + return; > + } > + > + entry->next = NULL; > + entry->hton = hton; > + entry->cookie = cookie; > + > + mysql_mutex_lock(&pending_checkpoint_mutex); > + lsn = log_get_lsn(); > + flush_lsn = log_get_flush_lsn(); > + if (lsn > flush_lsn) { > + /* Put the request in queue. > + When the log gets flushed past the lsn, we will remove the > + entry from the queue and notify the upper layer. */ > + entry->lsn = lsn; > + if (pending_checkpoint_list_end) { > + pending_checkpoint_list_end->next = entry; > + } else { > + pending_checkpoint_list = entry; > + } > + pending_checkpoint_list_end = entry; > + entry = NULL; > + } > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + > + if (entry) { > + /* We are already flushed. Notify the checkpoint immediately. */ > + commit_checkpoint_notify_ha(entry->hton, entry->cookie); > + my_free(entry); > + } > +} > + > +/*****************************************************************//** > +Log code calls this whenever log has been written and/or flushed up > +to a new position. We use this to notify upper layer of a new commit > +checkpoint when necessary.*/ > +extern "C" UNIV_INTERN > +void > +innobase_mysql_log_notify( > +/*===============*/ > + ib_uint64_t write_lsn, /*!< in: LSN written to log file */ > + ib_uint64_t flush_lsn) /*!< in: LSN flushed to disk */ > +{ > + struct pending_checkpoint * pending; > + struct pending_checkpoint * entry; > + struct pending_checkpoint * last_ready; > + > + /* It is safe to do a quick check for NULL first without lock. > + Even if we should race, we will at most skip one checkpoint and > + take the next one, which is harmless. */ > + if (!pending_checkpoint_list) > + return; > + > + mysql_mutex_lock(&pending_checkpoint_mutex); > + pending = pending_checkpoint_list; > + if (!pending) > + { > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + return; > + } > + > + last_ready = NULL; > + for (entry = pending; entry != NULL; entry = entry -> next) > + { > + if (entry->lsn > flush_lsn) > + break; > + last_ready = entry; > + } > + > + if (last_ready) > + { > + /* We found some pending checkpoints that are now flushed to > + disk. So remove them from the list. */ > + pending_checkpoint_list = entry; > + if (!entry) > + pending_checkpoint_list_end = NULL; > + } > + > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + > + if (!last_ready) > + return; > + > + /* Now that we have released the lock, notify upper layer about all > + commit checkpoints that have now completed. */ > + for (;;) { > + entry = pending; > + pending = pending->next; > + > + commit_checkpoint_notify_ha(entry->hton, entry->cookie); > + > + my_free(entry); > + if (entry == last_ready) > + break; > + } > } > > /*****************************************************************//** > > === modified file 'storage/xtradb/include/ha_prototypes.h' > --- a/storage/xtradb/include/ha_prototypes.h 2012-02-21 19:51:56 +0000 > +++ b/storage/xtradb/include/ha_prototypes.h 2012-09-14 12:44:53 +0000 > @@ -136,6 +136,17 @@ innobase_mysql_print_thd( > uint max_query_len); /*!< in: max query length to print, or 0 to > use the default max length */ > > +/*****************************************************************//** > +Log code calls this whenever log has been written and/or flushed up > +to a new position. We use this to notify upper layer of a new commit > +checkpoint when necessary.*/ > +UNIV_INTERN > +void > +innobase_mysql_log_notify( > +/*===============*/ > + ib_uint64_t write_lsn, /*!< in: LSN written to log file */ > + ib_uint64_t flush_lsn); /*!< in: LSN flushed to disk */ > + > /**************************************************************//** > Converts a MySQL type to an InnoDB type. Note that this function returns > the 'mtype' of InnoDB. InnoDB differentiates between MySQL's old <= 4.1 > > === modified file 'storage/xtradb/include/log0log.h' > --- a/storage/xtradb/include/log0log.h 2012-08-27 16:13:17 +0000 > +++ b/storage/xtradb/include/log0log.h 2012-09-14 12:44:53 +0000 > @@ -151,6 +151,13 @@ UNIV_INLINE > ib_uint64_t > log_get_lsn(void); > /*=============*/ > +/************************************************************//** > +Gets the last lsn that is fully flushed to disk. > +@return last flushed lsn */ > +UNIV_INLINE > +ib_uint64_t > +log_get_flush_lsn(void); > +/*=============*/ > /**************************************************************** > Gets the log group capacity. It is OK to read the value without > holding log_sys->mutex because it is constant. > > === modified file 'storage/xtradb/include/log0log.ic' > --- a/storage/xtradb/include/log0log.ic 2011-07-14 19:22:41 +0000 > +++ b/storage/xtradb/include/log0log.ic 2012-09-14 12:44:53 +0000 > @@ -411,6 +411,25 @@ log_get_lsn(void) > return(lsn); > } > > +/************************************************************//** > +Gets the last lsn that is fully flushed to disk. > +@return last flushed lsn */ > +UNIV_INLINE > +ib_uint64_t > +log_get_flush_lsn(void) > +/*=============*/ > +{ > + ib_uint64_t lsn; > + > + mutex_enter(&(log_sys->mutex)); > + > + lsn = log_sys->flushed_to_disk_lsn; > + > + mutex_exit(&(log_sys->mutex)); > + > + return(lsn); > +} > + > /**************************************************************** > Gets the log group capacity. It is OK to read the value without > holding log_sys->mutex because it is constant. > > === modified file 'storage/xtradb/log/log0log.c' > --- a/storage/xtradb/log/log0log.c 2012-08-27 16:13:17 +0000 > +++ b/storage/xtradb/log/log0log.c 2012-09-14 12:44:53 +0000 > @@ -1390,6 +1390,8 @@ log_write_up_to( > ulint loop_count = 0; > #endif /* UNIV_DEBUG */ > ulint unlock; > + ib_uint64_t write_lsn; > + ib_uint64_t flush_lsn; > > if (recv_no_ibuf_operations) { > /* Recovery is running and no operations on the log files are > @@ -1568,8 +1570,13 @@ log_write_up_to( > > log_flush_do_unlocks(unlock); > > + write_lsn = log_sys->write_lsn; > + flush_lsn = log_sys->flushed_to_disk_lsn; > + > mutex_exit(&(log_sys->mutex)); > > + innobase_mysql_log_notify(write_lsn, flush_lsn); > + > return; > > do_waits: > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 2

[Maria-developers] API Documentaion for Replication Features
by Gordan Bobic 03 Dec '12

03 Dec '12

Hi, I have a need for replicating data from multiple masters (independent data going to different tables). I have looked into the following with no success: Tungsten: Slow, CPU hungry, outrageously bloated, and unacceptably unreliable (replication will stall silently at times, and the only way to resume it is to offline and online the replication service). mysql-replication-listener: Just about got it to work with local binlogs, but haven't managed to get it to connect via TCP to a MySQL server. So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: 1) Slave TCP connections to the master and binlog fetching 2) Binlog parsing 3) Config file parsing In the longer term I'd like to write a patch that adds multiple-master functionality into MySQL itself. For configuring it, I was thinking about the following WRT configuration in my.cnf with multiple section instances, e.g.: [replicator] server-id=12345 master-host=server1 master-port=3306 master-user=user1 master-password=password1 replicate-do-table=db1.table1 [replicator] server-id=12345 master-host=server2 master-port=3306 master-user=user2 master-password=password2 replicate-do-table=db2.table2 etc. Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand. TIA. Gordan

4 9

[Maria-developers] MariaDB 5.5.28a release prep
by Sergei Golubchik 28 Nov '12

28 Nov '12

Preliminary release notes and changelog pages have been created for 5.5.28a https://kb.askmonty.org/en/mariadb-5528a-release-notes/ https://kb.askmonty.org/en/mariadb-5528a-changelog/ Please let me know if there is anything in particular which should be mentioned in the release notes. Thanks, Sergei _______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers(a)lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp

1 0

[Maria-developers] Review of mdev-452
by Michael Widenius 28 Nov '12

28 Nov '12

Hi! Here is the review of https://mariadb.atlassian.net/browse/MDEV-452 (Add full support for auto-initialized/updated timestamp and datetime) === modified file 'mysql-test/r/log_tables.result' --- mysql-test/r/log_tables.result 2011-10-19 19:45:18 +0000 +++ mysql-test/r/log_tables.result 2012-10-17 12:43:56 +0000 @@ -53,7 +53,7 @@ ERROR HY000: You can't use locks with lo show create table mysql.general_log; Table Create Table general_log CREATE TABLE `general_log` ( - `event_time` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + `event_time` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), Why does all log tables now have CURRENT_TIMESTAMP(6) instead of CURRENT_TIMESTAMP ? This would cause a problem for someone that updates to MySQL 10.0 and then want to go back... Many of the other result file changes comes from the same change for general_log and slow_log. Is this also the case for MySQL 5.6 or only for MariaDB? Answer from Timour on IRC: Timour> This comes from MySQL 5.6 and we need to stay compatbile. === modified file 'sql/event_db_repository.cc' --- sql/event_db_repository.cc 2012-09-04 16:26:30 +0000 +++ sql/event_db_repository.cc 2012-10-17 12:43:56 +0000 @@ -829,9 +829,6 @@ Event_db_repository::update_event(THD *t (int) table->field[ET_FIELD_ON_COMPLETION]->val_int())) goto end; - /* Don't update create on row update. */ - table->timestamp_field_type= TIMESTAMP_NO_AUTO_SET; - Why don't we anymore need to tell the rest of the code that the automatic fields shouldn't be updated from this function? Timour> As I remember, the main change compared to the old design is Timour> that we no longer update timestamps inside the handler Timour> So the place where timestamp updates are triggered is such Timour> that such updates will no longer be triggered in these cases. Yes, mysql_event_fill_row() doesn't call fill_record_n_invoke_before_triggers(). It instead has own code that probably not calling setting of any default fields. This does however mean that one can never get automatic timestamps with events. === modified file 'sql/field.cc' --- sql/field.cc 2012-09-08 22:22:06 +0000 +++ sql/field.cc 2012-10-17 12:43:56 +0000 @@ -4419,10 +4419,12 @@ Field_timestamp::Field_timestamp(uchar * { /* For 4.0 MYD and 4.0 InnoDB compatibility */ flags|= UNSIGNED_FLAG | BINARY_FLAG; - if (unireg_check != NONE && !share->timestamp_field) + if (unireg_check != NONE) { - /* This timestamp has auto-update */ - share->timestamp_field= this; + /* + This TIMESTAMP column is hereby quietly assumed to have an insert or + update default function. + */ We need to here add a comment what the TIMESTAMP_FLAG now means, as it has lost it's orginal meaning. Add to the above comment: /* We mark the flag with TIMESTAMP_FLAG to indicate to the client that this field will be automaticly updated on insert */ flags|= TIMESTAMP_FLAG; if (unireg_check != TIMESTAMP_DN_FIELD) flags|= ON_UPDATE_NOW_FLAG; Please also fix the function comment that refers to TABLE::timestamp_field <cut> +void Field_timestamp::set_explicit_default(Item *value) +{ + if (value && + ((value->type() == Item::DEFAULT_VALUE_ITEM && + !((Item_default_value*)value)->arg) || + (!maybe_null() && value->is_null()))) + return; + flags|= HAS_EXPLICIT_DEFAULT; +} The above function confused me greatly (mostly because of names) You should not have to check for value != 0. An item should never be 0. I noticed that some code called set_explicit_default() with NULL, but it would be better to for this add a general item function: field->set_explicit_default(NULL) -> field->mark_that_field_has_been_given_a_value() Where: Field::mark_that_field_has_been_given_a_value() { flags|= HAS_EXPLICIT_VALUE; } (Note name change for HAS_EXPLICIT_DEFAULT) This can be inline and we will win some function calls. The set_explicit_default() function also needs a comment as it's very confusing. To clarify the code, lets first add a function: /* Returns 1 if the default returns the default value for a column */ Item_default_value::default_column() { return arg != 0; } And then original function would look like: /** Mark columns that has been given a value with HAS_EXPLICIT_VALUE. For timestamp columns, the only case where a column is not marked with been given a value are: - It's explicitely assigned with DEFAULT - We assign NULL to a timestamp field that is defined as NOT NULL. This is how MySQL has worked since it's start. */ void Field_timestamp::set_explicit_default(Item *value) { if (((value->type() == Item::DEFAULT_VALUE_ITEM && !(Item_default_value*) value)->default_column()) || (!maybe_null() && value->is_null())) return; mark_that_field_has_been_given_a_value(); } <cut> + +int Field_datetime::set_time() +{ + THD *thd= current_thd; current_thd -> table->in_use; + MYSQL_TIME now_time; + thd->variables.time_zone->gmt_sec_to_TIME(&now_time, thd->query_start()); + now_time.second_part= thd->query_start_sec_part(); + set_notnull(); + store_TIME(&now_time); + thd->time_zone_used= 1; + return 0; +} @@ -8857,16 +8866,37 @@ bool Create_field::init(THD *thd, char * { uint sign_len, allowed_type_modifier= 0; ulong max_field_charlength= MAX_FIELD_CHARLENGTH; + const bool on_update_is_function= + (fld_on_update_value != NULL && + fld_on_update_value->type() == Item::FUNC_ITEM); DBUG_ENTER("Create_field::init()"); field= 0; field_name= fld_name; - def= fld_default_value; flags= fld_type_modifier; option_list= create_opt; - unireg_check= (fld_type_modifier & AUTO_INCREMENT_FLAG ? - Field::NEXT_NUMBER : Field::NONE); + + if (fld_default_value != NULL && fld_default_value->type() == Item::FUNC_ITEM) + { + /* There is a function default for insertions. */ + def= NULL; + unireg_check= on_update_is_function ? + Field::TIMESTAMP_DNUN_FIELD : // for insertions and for updates. + Field::TIMESTAMP_DN_FIELD; // only for insertions. Add braces around the on_update_is... to get better indentation. + } + else + { + /* No function default for insertions. Either NULL or a constant. */ + def= fld_default_value; + if (on_update_is_function) + unireg_check= Field::TIMESTAMP_UN_FIELD; // function default for updates + else + unireg_check= (fld_type_modifier & AUTO_INCREMENT_FLAG) != 0 ? + Field::NEXT_NUMBER : // Automatic increment. + Field::NONE; Add () around expression. + } + <cut> + + +/** + Mark the field as having an explicit default value. + + @param value if available, the value that the field is being set to + + @note + Fields that have an explicit default value should not be updated + automatically via the DEFAULT or ON UPDATE functions. The functions + that deal with data change functionality (INSERT/UPDATE/LOAD), + determine if there is an explicit value for each field before performing + the data change, and call this method to mark the field. + + If the 'value' parameter is NULL, then the field is marked unconditionally + as having an explicit value. If 'value' is not NULL, then it can be further + analyzed to check if it really should count as a value. +*/ + +void Field::set_explicit_default(Item *value) +{ /* Return if value is DEFAULT */ + if (value && value->type() == Item::DEFAULT_VALUE_ITEM && + !((Item_default_value*)value)->arg) + return; + flags|= HAS_EXPLICIT_DEFAULT; +} Remove checking if value is null. flags|= HAS_EXPLICIT_DEFAULT -> flags|= HAS_EXPLICIT_VALUE === modified file 'sql/field.h' @@ -1239,12 +1270,26 @@ class Field_timestamp :public Field_str virtual int set_time(); <cut> + virtual void set_explicit_default(Item *value); + virtual int evaluate_insert_default_function() + { + int res= 0; + if (has_insert_default_function()) + res= set_time(); + return res; + } + virtual int evaluate_update_default_function() + { + int res= 0; + if (has_update_default_function()) + res= set_time(); + return res; + } /* Get TIMESTAMP field value as seconds since begging of Unix Epoch */ virtual my_time_t get_timestamp(ulong *sec_part) const; virtual void store_TIME(my_time_t timestamp, ulong sec_part) @@ -1252,7 +1297,6 @@ class Field_timestamp :public Field_str int4store(ptr,timestamp); } bool get_date(MYSQL_TIME *ltime, ulonglong fuzzydate); - timestamp_auto_set_type get_auto_set_type() const; uchar *pack(uchar *to, const uchar *from, uint max_length __attribute__((unused))) { @@ -1503,6 +1547,28 @@ class Field_datetime :public Field_tempo void sql_type(String &str) const; bool zero_pack() const { return 1; } bool get_date(MYSQL_TIME *ltime, ulonglong fuzzydate); + virtual int set_time(); + virtual void set_default() + { + if (has_insert_default_function()) + set_time(); + else + Field::set_default(); + } + virtual int evaluate_insert_default_function() + { + int res= 0; + if (has_insert_default_function()) + res= set_time(); + return res; + } + virtual int evaluate_update_default_function() + { + int res= 0; + if (has_update_default_function()) + res= set_time(); + return res; + } It's a bit of a shame that we need to have the exact same functions twice? One way to solve it would be to add these functions to Field_str, as both Field_timestamp() and Field_datetime() depends on this class. === modified file 'sql/sql_base.cc' <cut> static bool -fill_record(THD * thd, List<Item> &fields, List<Item> &values, +fill_record(THD * thd, TABLE *table_arg, List<Item> &fields, List<Item> &values, bool ignore_errors) { Why this change (was not needed for this functionality). timour> I think I had a different implementation before, then I kept this timrou> change, because it looked simpler. bool -fill_record_n_invoke_before_triggers(THD *thd, List<Item> &fields, +fill_record_n_invoke_before_triggers(THD *thd, TABLE *table, List<Item> &fields, List<Item> &values, bool ignore_errors, - Table_triggers_list *triggers, enum trg_event_type event) { bool result; - result= (fill_record(thd, fields, values, ignore_errors) || + Table_triggers_list *triggers= table->triggers; + result= (fill_record(thd, table, fields, values, ignore_errors) || (triggers && triggers->process_triggers(thd, event, TRG_ACTION_BEFORE, TRUE))); /* @@ -8925,7 +8924,6 @@ fill_record_n_invoke_before_triggers(THD */ if (!result && triggers) { - TABLE *table= 0; List_iterator_fast<Item> f(fields); Item *fld; Item_field *item_field; @@ -8933,9 +8931,7 @@ fill_record_n_invoke_before_triggers(THD { fld= (Item_field*)f++; item_field= fld->filed_for_view_update(); - if (item_field && item_field->field && - (table= item_field->field->table) && - table->vfield) + if (item_field && item_field->field && table && table->vfield) result= update_virtual_fields(thd, table, TRUE); Add: DBUG_ASSERT(table == item_field->field->table) Note that if we can only have one table here, than we could also replace: if (!result && triggers) with: if (!result && triggers && table) As the loop will not do anything if table == 0. <cut> bool -fill_record_n_invoke_before_triggers(THD *thd, Field **ptr, +fill_record_n_invoke_before_triggers(THD *thd, TABLE *table, Field **ptr, List<Item> &values, bool ignore_errors, - Table_triggers_list *triggers, enum trg_event_type event) { bool result; - result= (fill_record(thd, ptr, values, ignore_errors, FALSE) || + Table_triggers_list *triggers= table->triggers; + result= (fill_record(thd, table, ptr, values, ignore_errors, FALSE) || (triggers && triggers->process_triggers(thd, event, TRG_ACTION_BEFORE, TRUE))); /* @@ -9071,7 +9064,6 @@ fill_record_n_invoke_before_triggers(THD */ if (!result && triggers && *ptr) { - TABLE *table= (*ptr)->table; Add DBUG_ASSERT(table == (*ptr)->table); if (table->vfield) result= update_virtual_fields(thd, table, TRUE); } <cut> === modified file 'sql/sql_insert.cc' @@ -1696,13 +1660,32 @@ int write_record(THD *thd, TABLE *table, restore_record(table,record[1]); DBUG_ASSERT(info->update_fields->elements == info->update_values->elements); - if (fill_record_n_invoke_before_triggers(thd, *info->update_fields, + if (fill_record_n_invoke_before_triggers(thd, table, *info->update_fields, *info->update_values, info->ignore, - table->triggers, TRG_EVENT_UPDATE)) goto before_trg_err; + bool different_records= (!records_are_comparable(table) || + compare_record(table)); + /* + Default fields must be updated before checking view updateability. + This branch of INSERT is executed only when a UNIQUE key was violated + with the ON DUPLICATE KEY UPDATE option. In this case the INSERT + operation is transformed to an UPDATE, and the default fields must + be updated as if this is an UPDATE. + */ + if (different_records && table->default_field) + { + bool res; + enum_sql_command cmd= thd->lex->sql_command; + thd->lex->sql_command= SQLCOM_UPDATE; + res= table->update_default_fields(); + thd->lex->sql_command= cmd; + if (res) + goto err; + } Another way to do the above nicer would be to give thd->lex->sql_command as an argument to update_default_fields(). Another option would be to have: update_defaults_fields(table) { return update_defaults_fields(table, table->in_use->lex->cmd); } And use the later in this case. However this is not that critical to fix... <cut> + Field **field,**org_field, *found_next_number_field, **dfield_ptr; You need to do dfield_ptr= 0 to avoid a compiler warning. @@ -2390,6 +2370,12 @@ TABLE *Delayed_insert::get_local_table(T bitmap= (uchar*) (field + share->fields + 1); copy->record[0]= (bitmap + share->column_bitmap_size*3); memcpy((char*) copy->record[0], (char*) table->record[0], share->reclength); + if (share->default_fields) + { + copy->default_field= (Field**) client_thd->alloc((share->default_fields+1)* + sizeof(Field**)); You should check for error condition here. + dfield_ptr= copy->default_field; + } /* Make a copy of all fields. The copied fields need to point into the copied record. This is done @@ -2407,18 +2393,19 @@ TABLE *Delayed_insert::get_local_table(T (*field)->move_field_offset(adjust_ptrs); // Point at copy->record[0] if (*org_field == found_next_number_field) (*field)->table->found_next_number_field= *field; + if (share->default_fields && + ((*org_field)->has_insert_default_function() || + (*org_field)->has_update_default_function())) + { + /* Put the newly copied field into the set of default fields. */ + *dfield_ptr= *field; + (*dfield_ptr)->unireg_check= (*org_field)->unireg_check; It's strange that new_field() resets unireg_check. Don't know why this is done and we should at some point fix that it's not reset and see what happens. + dfield_ptr++; + } } *field=0; <cut> @@ -850,7 +837,7 @@ read_fixed_length(THD *thd, COPY_INFO &i ER(ER_WARN_TOO_FEW_RECORDS), thd->warning_info->current_row_for_warning()); if (!field->maybe_null() && field->type() == FIELD_TYPE_TIMESTAMP) - ((Field_timestamp*) field)->set_time(); + field->set_time(); Shouldn't the above be done now by testing for if the field has an insert_default_function? @@ -994,10 +982,11 @@ read_sep_field(THD *thd, COPY_INFO &info DBUG_RETURN(1); } field->set_null(); + field->set_explicit_default(NULL); if (!field->maybe_null()) { if (field->type() == MYSQL_TYPE_TIMESTAMP) - ((Field_timestamp*) field)->set_time(); + field->set_time(); shouldn't we use an insert_default_function here too ? else if (field != table->next_number_field) field->set_warning(MYSQL_ERROR::WARN_LEVEL_WARN, ER_WARN_NULL_TO_NOTNULL, 1); @@ -1066,7 +1056,7 @@ read_sep_field(THD *thd, COPY_INFO &info DBUG_RETURN(1); } if (!field->maybe_null() && field->type() == FIELD_TYPE_TIMESTAMP) - ((Field_timestamp*) field)->set_time(); + field->set_time(); shouldn't we use an insert_default_function here too ? /* TODO: We probably should not throw warning for each field. But how about intention to always have the same number @@ -1206,7 +1196,7 @@ read_xml_field(THD *thd, COPY_INFO &info if (!field->maybe_null()) { if (field->type() == FIELD_TYPE_TIMESTAMP) - ((Field_timestamp *) field)->set_time(); + field->set_time(); shouldn't we use an insert_default_function here too ? === modified file 'sql/sql_parse.cc' --- sql/sql_parse.cc 2012-10-06 07:30:52 +0000 +++ sql/sql_parse.cc 2012-10-18 12:57:12 +0000 @@ -269,12 +269,14 @@ void init_update_queries(void) CF_CAN_GENERATE_ROW_EVENTS; sql_command_flags[SQLCOM_CREATE_INDEX]= CF_CHANGES_DATA | CF_AUTO_COMMIT_TRANS; sql_command_flags[SQLCOM_ALTER_TABLE]= CF_CHANGES_DATA | CF_WRITE_LOGS_COMMAND | - CF_AUTO_COMMIT_TRANS | CF_REPORT_PROGRESS ; + CF_AUTO_COMMIT_TRANS | CF_REPORT_PROGRESS | + CF_INSERTS_DATA; Should we really have CF_INSERTS_DATA here? We should not update any timestamps for ALTER TABLE? === modified file 'sql/table.cc' @@ -2472,7 +2471,7 @@ int open_table_from_share(THD *thd, TABL } /* - Process virtual columns, if any. + Process virtual and default columns, if any. */ if (!share->vfields) outparam->vfield= NULL; @@ -2484,10 +2483,26 @@ int open_table_from_share(THD *thd, TABL goto err; outparam->vfield= vfield_ptr; + } + + if (!share->default_fields) + outparam->default_field= NULL; You don't need to set default_fields to NULL as outparam is zeroed in beginning of the function. (Same is true for outparam->vfield) <cut> Regards, Monty

1 0

[Maria-developers] EXISTS-to-IN can work anywhere?
by Sergei Petrunia 26 Nov '12

26 Nov '12

Hello, Currently, EXISTS-to-IN rewrite only works on WHERE's top level, or under NOT, where the rewrite doesn't change semantics wrt NULL values. It seems, it is possible to make it work in other contexts, too. We need to convert it into a "NULL-ignoring IN", a special kind of IN predicate. For normal IN: NULL IN (select that returns something) --> NULL NULL IN (select that returns nothing) --> FALSE $value IN (select that doesn't return $value) --> FALSE $value IN (select that doesn't return $value but returns NULL) --> NULL null-ignoring-IN should return FALSE in all of the above cases. In fact, the optimizer is perfectly capable of doing this. What we need is just to disable special handling for NULLs. Let's discuss this tomorrow (Tuesday). BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB 5.1.66 release prep
by Sergei Golubchik 26 Nov '12

26 Nov '12

Preliminary release notes and changelog pages have been created for 5.1.66, 5.2.13, 5.3.11. https://kb.askmonty.org/en/mariadb-5166-release-notes/ https://kb.askmonty.org/en/mariadb-5166-changelog/ https://kb.askmonty.org/en/mariadb-5213-release-notes/ https://kb.askmonty.org/en/mariadb-5213-changelog/ https://kb.askmonty.org/en/mariadb-5311-release-notes/ https://kb.askmonty.org/en/mariadb-5311-changelog/ Please let me know if there is anything in particular which should be mentioned in the release notes. Thanks, Sergei

1 0

[Maria-developers] EXISTS-to-IN is ok to push
by Sergei Petrunia 23 Nov '12

23 Nov '12

Hi Sanja, I've made some fixes to comments in lp:~maria-captains/maria/10.0-exists2in. With that, it is ok to push EXISTS-to-IN into 10.0. Sorry for the delay. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] crash in TC_LOG_MMAP::log_one_transaction in maria-10.0.0
by Rich Prohaska 19 Nov '12

19 Nov '12

Hello, I am having problems with the shinny new mariadb-10.0.0. I have a simple test case that replaces a row in innodb and tokudb and then commits. Eventually, mysqld hits this assert: mysqld: /home/prohaska/maria10-build/mariadb-10.0.0/sql/log.cc:7547: int TC_LOG_MMAP::log_one_transaction(my_xid): Assertion `p->ptr < p->end' failed. The test works great when the binlog is enabled. The test crashes when the binlog is OFF. So, maybe the problem is isolated to the TC_LOG_MMAP logic. I noticed that the TC_LOG_MMAP::unlog function is quite different than the mariadb-5.5 version. What is the best way to get this fixed? Would it help if i made the test case available? Thanks Rich Prohaska Here is the stack trace: Program received signal SIGABRT, Aborted. 0x0000003006a32885 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003006a32885 in raise () from /lib64/libc.so.6 #1 0x0000003006a34065 in abort () from /lib64/libc.so.6 #2 0x0000003006a2b9fe in __assert_fail_base () from /lib64/libc.so.6 #3 0x0000003006a2bac0 in __assert_fail () from /lib64/libc.so.6 #4 0x00000000008862bf in TC_LOG_MMAP::log_one_transaction (this=0x1541f00 <tc_log_mmap>, xid=142018) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/log.cc:7547 #5 0x0000000000885442 in TC_LOG_MMAP::log_and_order (this=0x1541f00 <tc_log_mmap>, thd=0x27f0000, xid=142018, all=true, need_prepare_ordered=false, need_commit_ordered=true) at /home/prohaska/maria10-\ build/mariadb-10.0.0/sql/log.cc:7180 #6 0x00000000007b819f in ha_commit_trans (thd=0x27f0000, all=true) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/handler.cc:1329 #7 0x0000000000710307 in trans_commit (thd=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/transaction.cc:213 #8 0x000000000060a2df in mysql_execute_command (thd=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_parse.cc:4084 #9 0x000000000060f1a7 in mysql_parse (thd=0x27f0000, rawbuf=0x7fff6c0050b8 "commit", length=6, parser_state=0x7fffe4112660) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_parse.cc:6056 #10 0x00000000006029a3 in dispatch_command (command=COM_QUERY, thd=0x27f0000, packet=0x27fa341 "commit", packet_length=6) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_parse.cc:1216 #11 0x0000000000601bdb in do_command (thd=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_parse.cc:945 #12 0x00000000006ff6d1 in do_handle_one_connection (thd_arg=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_connect.cc:1254 #13 0x00000000006ff179 in handle_one_connection (arg=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_connect.cc:1168 #14 0x0000003006e077f1 in start_thread () from /lib64/libpthread.so.0 #15 0x0000003006ae592d in clone () from /lib64/libc.so.6

3 9