- developers - lists.mariadb.org

Re: [Maria-developers] parallel replication performance
by Kristian Nielsen 25 Mar '14

25 Mar '14

Axel Schwenke <axel(a)askmonty.org> writes: > The poor performance of MySQL with 1 database is expected. However MySQL is > much better with 32 databases. Even better than MariaDB :( I haven't had time to go through all the numbers and details yet, but just a quick check: In this test, are you sure that you are running MariaDB in GTID mode? CHANGE MASTER TO master_use_gtid=slave_pos If not, you won't get any parallelism out of using 32 databases with different domain_ids. So two interesting, but different comparisons to make here: 1. MySQL multi-threaded slave compared to MariaDB GTID with different domain_id per database. If MySQL is significantly better here it could indicate some problem in MariaDB parallel replication that should be investigated; this is possible as this part has not been tested a lot. 2. MySQL multi-threaded slave with load split into multiple databases, compared to MariaDB without configuring different domains. Or just MariaDB compared without and with different domain ids. This is an "unfair" benchmark; by explicitly declaring that different databases/domains can run in parallel, more opportunity for parallellism is available. But it could still be interesting to see how much of that can be obtained from group commit parallelism, without changing applications. Both of these are interesting, but it was not clear to me which of them you were describing? - Kristian.

2 3

[Maria-developers] Patch for MDEV 5873
by Rohit Kalhans 25 Mar '14

25 Mar '14

Hello Kristian, All, As mentioned earlier i have a patch for MDEV 5873 and i am attaching here in the bundle of the patch. I have run tests on this patch and all of them have passed. Kindly review the patch. Also i am not sure which is the suitable mailing list for sending patches, so it will be great if you can point me to the same. Thanks and regards Rohit. -- Cheerio! *Rohit*

2 1

[Maria-developers] Two 10.0-related things.
by Alexey Botchkov 25 Mar '14

25 Mar '14

Hi, Sergei. I'd like to share my recently obtained knowledge :) 1. Mac OS X and sockets. As you know there on Labrador machine that mysql-test-run.pl failed if not run with --tmpdir=/tmp. The problem apparently is that the /Users directory has the access restrictions that /tmp doesn't. I still don't know what exactly was that, but it helps if we revoke all the ACLs with 'chmod -R -N /Users/mariadb'. 2. The 10.0 fails building on my machine, with linking errors in the extra/yassl/, about missing 'operator new[]' and similar. I fixed it locally with that patch: === modified file 'extra/yassl/CMakeLists.txt' --- extra/yassl/CMakeLists.txt 2014-02-26 14:28:07 +0000 +++ extra/yassl/CMakeLists.txt 2014-03-25 07:45:46 +0000 @@ -33,9 +33,10 @@ SET(YASSL_SOURCES src/buffer.cpp src/ce ADD_CONVENIENCE_LIBRARY(yassl ${YASSL_SOURCES}) RESTRICT_SYMBOL_EXPORTS(yassl) +TARGET_LINK_LIBRARIES(stdc++) + INSTALL_DEBUG_SYMBOLS(yassl) Best regards. HF

2 1

[Maria-developers] valgrind failures in ConnectSE
by Alexander Barkov 23 Mar '14

23 Mar '14

Hi Olivier, Valgrind complains about uninitialized memory read: http://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/5… It seems the problem was introduced by this change: http://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/5… Can you please fix this asap? We're going to make a release early next week. It would be nice to merge the latest ConnectSE. Thanks.

2 1

[Maria-developers] A fix for MDEV-5689 ExtractValue(xml, 'substring(/x, /y)') crashes
by Alexander Barkov 23 Mar '14

23 Mar '14

Hi Sergei, please review a fix for MDEV-5689. It also fixes MDEV-5709 ExtractValue() with XPath variable references returns wrong result. Description: 1. The main problem was that that nodeset_func->fix_fields() was called in Item_func_xml_extractvalue::val_str() and Item_func_xml_update::val_str(), which led in some cases to execution of the XPath engine *before* having a parsed XML value. Moved to Item_xml_str_func::fix_fields(). 2. Cleanup: added a new method Item_xml_str_func::fix_fields() and moved most of the code from Item_xml_str_func::fix_length_and_dec() to Item_xml_str_func::fix_fields(), to follow the usual Item layout. 3. Cleanup: a parsed XML value is useless without the raw XML value it was built from. Previously the parsed and the raw values where stored in separate String instances. It was hard to follow how they are synchronized. Added a helper class XML which contains both parsed and raw values. Makes things easier to read and modify. 4. MDEV-5709: const_item() could incorrectly return a "true" result when XPath expression contains users/SP variable references. Now nodeset_func->const_item() is also taken into account to catch such cases. 5. Minor code enhancements.

2 5

[Maria-developers] Transactions behind a failed transaction could commit in parallel replication
by nanyi607rao 21 Mar '14

21 Mar '14

hi kristian, In parallel replication if a transaction executed failed, replication should stop immediately(unconsider retry), and transactions behind the failed one should skip or rollback. you did try to do like this, there are codes like: if (unlikely(entry->stop_on_error_sub_id <= rgi->wait_commit_sub_id)) skip_event_group= true; this codes can tell latter transactions to skip but can't tell them rollback. because if a transaction started commiting before a former transaction failed (such as Lock timeout for unknown reason), the commiting transaction will not be affectd by stop_on_error_sub_id. Then the failed transaction should wakeup latter commiting transactions and tell them to rollback, unfortunately it won't. codes like if (!rgi->is_error && !skip_event_group) err= rpt_handle_event(events, rpt); else err= thd->wait_for_prior_commit(); ... ... finish_event_group(thd, err, event_gtid_sub_id, entry, rgi); if the failed transaction didn't fail at end event, err's value would come from wait_for_prior_commit, the err would be 0 if its former transaction has successed, then the failed transaction would tell latter transactions ok to commit in finish_event_group. 2014-03-19 nanyi607rao

2 2

Re: [Maria-developers] MDEV-5914
by Kristian Nielsen 21 Mar '14

21 Mar '14

Jan Lindström <jplindst(a)mariadb.org> writes: > Here is necessary changes. I have run whole mysql-test-run and I did not see > any failures, thus is is mostly safe. I say mostly, > because we really do not have much test cases that would use multimaster setup > and try to get conflicting transactions > generated from local clients against slave thread. Thanks a lot! I will try to come up with some test cases for this, with replication and external transactions that conflict and show the problem. In parallel, I tried running the parallel replication slaves with READ COMMITTED; that also seems to work and can get Elena able to continue testing. But this change is potentially more correct; I will let you know when I have some test results. - Kristian. > ================================= > > jan@jan-GE70-0NC-0ND ~/mysql/10.0-mdev5914 $ bzr diff > === modified file 'include/mysql/plugin.h' > --- include/mysql/plugin.h 2014-02-26 14:28:07 +0000 > +++ include/mysql/plugin.h 2014-03-21 09:05:58 +0000 > @@ -617,6 +617,7 @@ > int thd_in_lock_tables(const MYSQL_THD thd); > int thd_tablespace_op(const MYSQL_THD thd); > long long thd_test_options(const MYSQL_THD thd, long long test_options); > +unsigned long long thd_group_commit_id(const MYSQL_THD thd); > int thd_sql_command(const MYSQL_THD thd); > void **thd_ha_data(const MYSQL_THD thd, const struct handlerton *hton); > void thd_storage_lock_wait(MYSQL_THD thd, long long value); > > === modified file 'include/mysql/plugin_audit.h.pp' > --- include/mysql/plugin_audit.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_audit.h.pp 2014-03-21 09:06:17 +0000 > @@ -298,6 +298,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'include/mysql/plugin_auth.h.pp' > --- include/mysql/plugin_auth.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_auth.h.pp 2014-03-21 09:06:35 +0000 > @@ -298,6 +298,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'include/mysql/plugin_ftparser.h.pp' > --- include/mysql/plugin_ftparser.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_ftparser.h.pp 2014-03-21 09:06:59 +0000 > @@ -251,6 +251,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'sql/sql_class.cc' > --- sql/sql_class.cc 2014-03-20 14:36:45 +0000 > +++ sql/sql_class.cc 2014-03-21 09:07:12 +0000 > @@ -675,6 +675,12 @@ > } > > extern "C" > +unsigned long long thd_group_commit_id(const THD *thd) > +{ > + return thd->rpl_group_commit_id; > +} > + > +extern "C" > int thd_sql_command(const THD *thd) > { > return (int) thd->lex->sql_command; > > === modified file 'storage/innobase/handler/ha_innodb.cc' > --- storage/innobase/handler/ha_innodb.cc 2014-02-26 18:36:33 +0000 > +++ storage/innobase/handler/ha_innodb.cc 2014-03-20 15:42:27 +0000 > @@ -2048,6 +2048,8 @@ > trx->check_unique_secondary = !thd_test_options( > thd, OPTION_RELAXED_UNIQUE_CHECKS); > > + trx->group_commit_id = thd_group_commit_id(thd); > + > DBUG_VOID_RETURN; > } > > > === modified file 'storage/innobase/include/trx0trx.h' > --- storage/innobase/include/trx0trx.h 2014-02-26 18:36:33 +0000 > +++ storage/innobase/include/trx0trx.h 2014-03-20 15:28:19 +0000 > @@ -840,6 +840,7 @@ > or the state last time became > TRX_STATE_ACTIVE */ > trx_id_t id; /*!< transaction id */ > + ib_uint64_t group_commit_id;/*!< Group commit id or 0 */ > XID xid; /*!< X/Open XA transaction > identification to identify a > transaction branch */ > > === modified file 'storage/innobase/lock/lock0lock.cc' > --- storage/innobase/lock/lock0lock.cc 2014-02-26 18:22:48 +0000 > +++ storage/innobase/lock/lock0lock.cc 2014-03-21 09:32:54 +0000 > @@ -1020,6 +1020,23 @@ > return(FALSE); > } > > + if (trx->group_commit_id != 0 && > + lock2->trx->group_commit_id != 0 && > + trx->group_commit_id == lock2->trx->group_commit_id && > + (type_mode & LOCK_GAP || lock_rec_get_gap(lock2))) { > + /* No lock request needs to wait GAP locks if > + transactions belong to the same group commit group. > + This is because we have already verified these > + transactions on master that they do not conflict > + on record locks but as they could be executed on > + different order on slave we should not wait for > + gap locks here. Note that parallel replication > + makes sure that transaction commits are executed > + in the same order in master and slave. */ > + > + return (FALSE); > + } > + > return(TRUE); > } > > > === modified file 'storage/xtradb/handler/ha_innodb.cc' > --- storage/xtradb/handler/ha_innodb.cc 2014-02-26 18:21:23 +0000 > +++ storage/xtradb/handler/ha_innodb.cc 2014-03-20 15:42:41 +0000 > @@ -2308,6 +2308,8 @@ > > trx->fake_changes = THDVAR(thd, fake_changes); > > + trx->group_commit_id = thd_group_commit_id(thd); > + > #ifdef EXTENDED_SLOWLOG > if (thd_log_slow_verbosity(thd) & (1ULL << SLOG_V_INNODB)) { > trx->take_stats = TRUE; > > === modified file 'storage/xtradb/include/trx0trx.h' > --- storage/xtradb/include/trx0trx.h 2014-02-26 18:21:23 +0000 > +++ storage/xtradb/include/trx0trx.h 2014-03-20 15:28:20 +0000 > @@ -861,6 +861,7 @@ > or the state last time became > TRX_STATE_ACTIVE */ > trx_id_t id; /*!< transaction id */ > + ib_uint64_t group_commit_id;/*!< Group commit id or 0 */ > XID xid; /*!< X/Open XA transaction > identification to identify a > transaction branch */ > > === modified file 'storage/xtradb/lock/lock0lock.cc' > --- storage/xtradb/lock/lock0lock.cc 2013-12-22 16:06:50 +0000 > +++ storage/xtradb/lock/lock0lock.cc 2014-03-21 09:32:46 +0000 > @@ -1021,6 +1021,23 @@ > return(FALSE); > } > > + if (trx->group_commit_id != 0 && > + lock2->trx->group_commit_id != 0 && > + trx->group_commit_id == lock2->trx->group_commit_id && > + (type_mode & LOCK_GAP || lock_rec_get_gap(lock2))) { > + /* No lock request needs to wait GAP locks if > + transactions belong to the same group commit group. > + This is because we have already verified these > + transactions on master that they do not conflict > + on record locks but as they could be executed on > + different order on slave we should not wait for > + gap locks here. Note that parallel replication > + makes sure that transaction commits are executed > + in the same order in master and slave. */ > + > + return (FALSE); > + } > + > return(TRUE); > }

1 0

[Maria-developers] [GSoC 2014] CREATE OR REPLACE, CREATE IF NOT EXISTS and DROP IF EXISTS
by sriram patil 21 Mar '14

21 Mar '14

Hi Sergei, I just went through WL#3129 (thanks to James Briggs for the link). It gave me a detailed insight on what exactly is expected. I have missed 2 object types in my proposal which are, USER and SERVER. I can edit my proposal to incorporate the changes in the deliverables timeline if you want me to. Thanks, Sriram

1 0

Re: [Maria-developers] [GSOC 2014] "CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS" Project
by Sergei Golubchik 21 Mar '14

21 Mar '14

Hi, Sriram! First: please don't forget to cc: the list maria-developers(a)lists.launchpad.net Thank you. On Mar 18, sriram patil wrote: > Hi Sergei, > > I was going through the source code and was able to figure out the > flow of query execution and major functions through which a query goes > before execution. > > The query execution majorly dwells in the "sql" directory of the > source code. So, I think I will have to make changes in the following > files throughout the project, > > - sql_parse.cc (do_command, dispatch_command, mysql_execute_command) Yes. And also sql_yacc.yy - to extend the parser. And the implementation of these commands, for example, sql_acl.cc for CREATE/DROP USER, sql_trigger.cc for CREATE/DROP TRIGGER, etc. > - sql_prepare.cc (Prepared_statement::execute_loop, > Prepared_statement::execute) No, hopefully, this will work automatically. > What I am not sure about is will I have to create a handler for some of > these commands? You are not creating new commands > And will lower level storage engine code changes required? I don't think so. > > > I am interested to work on "CREATE OR REPLACE, CREATE IF NOT EXISTS, and > > > DROP IF EXISTS" project as part of GSOC 2014. I want to apply my subject > > > knowledge at a bigger level which will also give me an idea about the > > > practical issues while implementing the features. > > > Regards, Sergei

3 9

[Maria-developers] [GSoC] MDEV-371- index on BLOB
by smit hinsu 21 Mar '14

21 Mar '14

Hi Sergei and Roberto, My name is Smit Hinsu and I am final year undergrad student. I have good knowledge about database internals and good experience with C++. I find feature of creating index on BLOB quickly using MD-5 interesting. I have understood basic requirements for this feature. I have following questions in my mind currently. I think, HASH index is not supported in MySQL's version of MyISAM storage engine. Has MariaDB removed this restriction? We should support index for BLOB for BTREE index also in which we store only hash value of the value in BTREE. Is this possible? Can you suggest me some startup task related to index or storage engine which can make me familiar with the related parts? I have started to look at code. From Smit Hinsu

2 2