March 2014 - developers - lists.mariadb.org

Re: [Maria-developers] MDEV-5914
by Kristian Nielsen 21 Mar '14

21 Mar '14

Jan Lindström <jplindst(a)mariadb.org> writes: > Here is necessary changes. I have run whole mysql-test-run and I did not see > any failures, thus is is mostly safe. I say mostly, > because we really do not have much test cases that would use multimaster setup > and try to get conflicting transactions > generated from local clients against slave thread. Thanks a lot! I will try to come up with some test cases for this, with replication and external transactions that conflict and show the problem. In parallel, I tried running the parallel replication slaves with READ COMMITTED; that also seems to work and can get Elena able to continue testing. But this change is potentially more correct; I will let you know when I have some test results. - Kristian. > ================================= > > jan@jan-GE70-0NC-0ND ~/mysql/10.0-mdev5914 $ bzr diff > === modified file 'include/mysql/plugin.h' > --- include/mysql/plugin.h 2014-02-26 14:28:07 +0000 > +++ include/mysql/plugin.h 2014-03-21 09:05:58 +0000 > @@ -617,6 +617,7 @@ > int thd_in_lock_tables(const MYSQL_THD thd); > int thd_tablespace_op(const MYSQL_THD thd); > long long thd_test_options(const MYSQL_THD thd, long long test_options); > +unsigned long long thd_group_commit_id(const MYSQL_THD thd); > int thd_sql_command(const MYSQL_THD thd); > void **thd_ha_data(const MYSQL_THD thd, const struct handlerton *hton); > void thd_storage_lock_wait(MYSQL_THD thd, long long value); > > === modified file 'include/mysql/plugin_audit.h.pp' > --- include/mysql/plugin_audit.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_audit.h.pp 2014-03-21 09:06:17 +0000 > @@ -298,6 +298,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'include/mysql/plugin_auth.h.pp' > --- include/mysql/plugin_auth.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_auth.h.pp 2014-03-21 09:06:35 +0000 > @@ -298,6 +298,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'include/mysql/plugin_ftparser.h.pp' > --- include/mysql/plugin_ftparser.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_ftparser.h.pp 2014-03-21 09:06:59 +0000 > @@ -251,6 +251,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'sql/sql_class.cc' > --- sql/sql_class.cc 2014-03-20 14:36:45 +0000 > +++ sql/sql_class.cc 2014-03-21 09:07:12 +0000 > @@ -675,6 +675,12 @@ > } > > extern "C" > +unsigned long long thd_group_commit_id(const THD *thd) > +{ > + return thd->rpl_group_commit_id; > +} > + > +extern "C" > int thd_sql_command(const THD *thd) > { > return (int) thd->lex->sql_command; > > === modified file 'storage/innobase/handler/ha_innodb.cc' > --- storage/innobase/handler/ha_innodb.cc 2014-02-26 18:36:33 +0000 > +++ storage/innobase/handler/ha_innodb.cc 2014-03-20 15:42:27 +0000 > @@ -2048,6 +2048,8 @@ > trx->check_unique_secondary = !thd_test_options( > thd, OPTION_RELAXED_UNIQUE_CHECKS); > > + trx->group_commit_id = thd_group_commit_id(thd); > + > DBUG_VOID_RETURN; > } > > > === modified file 'storage/innobase/include/trx0trx.h' > --- storage/innobase/include/trx0trx.h 2014-02-26 18:36:33 +0000 > +++ storage/innobase/include/trx0trx.h 2014-03-20 15:28:19 +0000 > @@ -840,6 +840,7 @@ > or the state last time became > TRX_STATE_ACTIVE */ > trx_id_t id; /*!< transaction id */ > + ib_uint64_t group_commit_id;/*!< Group commit id or 0 */ > XID xid; /*!< X/Open XA transaction > identification to identify a > transaction branch */ > > === modified file 'storage/innobase/lock/lock0lock.cc' > --- storage/innobase/lock/lock0lock.cc 2014-02-26 18:22:48 +0000 > +++ storage/innobase/lock/lock0lock.cc 2014-03-21 09:32:54 +0000 > @@ -1020,6 +1020,23 @@ > return(FALSE); > } > > + if (trx->group_commit_id != 0 && > + lock2->trx->group_commit_id != 0 && > + trx->group_commit_id == lock2->trx->group_commit_id && > + (type_mode & LOCK_GAP || lock_rec_get_gap(lock2))) { > + /* No lock request needs to wait GAP locks if > + transactions belong to the same group commit group. > + This is because we have already verified these > + transactions on master that they do not conflict > + on record locks but as they could be executed on > + different order on slave we should not wait for > + gap locks here. Note that parallel replication > + makes sure that transaction commits are executed > + in the same order in master and slave. */ > + > + return (FALSE); > + } > + > return(TRUE); > } > > > === modified file 'storage/xtradb/handler/ha_innodb.cc' > --- storage/xtradb/handler/ha_innodb.cc 2014-02-26 18:21:23 +0000 > +++ storage/xtradb/handler/ha_innodb.cc 2014-03-20 15:42:41 +0000 > @@ -2308,6 +2308,8 @@ > > trx->fake_changes = THDVAR(thd, fake_changes); > > + trx->group_commit_id = thd_group_commit_id(thd); > + > #ifdef EXTENDED_SLOWLOG > if (thd_log_slow_verbosity(thd) & (1ULL << SLOG_V_INNODB)) { > trx->take_stats = TRUE; > > === modified file 'storage/xtradb/include/trx0trx.h' > --- storage/xtradb/include/trx0trx.h 2014-02-26 18:21:23 +0000 > +++ storage/xtradb/include/trx0trx.h 2014-03-20 15:28:20 +0000 > @@ -861,6 +861,7 @@ > or the state last time became > TRX_STATE_ACTIVE */ > trx_id_t id; /*!< transaction id */ > + ib_uint64_t group_commit_id;/*!< Group commit id or 0 */ > XID xid; /*!< X/Open XA transaction > identification to identify a > transaction branch */ > > === modified file 'storage/xtradb/lock/lock0lock.cc' > --- storage/xtradb/lock/lock0lock.cc 2013-12-22 16:06:50 +0000 > +++ storage/xtradb/lock/lock0lock.cc 2014-03-21 09:32:46 +0000 > @@ -1021,6 +1021,23 @@ > return(FALSE); > } > > + if (trx->group_commit_id != 0 && > + lock2->trx->group_commit_id != 0 && > + trx->group_commit_id == lock2->trx->group_commit_id && > + (type_mode & LOCK_GAP || lock_rec_get_gap(lock2))) { > + /* No lock request needs to wait GAP locks if > + transactions belong to the same group commit group. > + This is because we have already verified these > + transactions on master that they do not conflict > + on record locks but as they could be executed on > + different order on slave we should not wait for > + gap locks here. Note that parallel replication > + makes sure that transaction commits are executed > + in the same order in master and slave. */ > + > + return (FALSE); > + } > + > return(TRUE); > }

1 0

[Maria-developers] [GSoC 2014] CREATE OR REPLACE, CREATE IF NOT EXISTS and DROP IF EXISTS
by sriram patil 21 Mar '14

21 Mar '14

Hi Sergei, I just went through WL#3129 (thanks to James Briggs for the link). It gave me a detailed insight on what exactly is expected. I have missed 2 object types in my proposal which are, USER and SERVER. I can edit my proposal to incorporate the changes in the deliverables timeline if you want me to. Thanks, Sriram

1 0

Re: [Maria-developers] [GSOC 2014] "CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS" Project
by Sergei Golubchik 21 Mar '14

21 Mar '14

Hi, Sriram! First: please don't forget to cc: the list maria-developers(a)lists.launchpad.net Thank you. On Mar 18, sriram patil wrote: > Hi Sergei, > > I was going through the source code and was able to figure out the > flow of query execution and major functions through which a query goes > before execution. > > The query execution majorly dwells in the "sql" directory of the > source code. So, I think I will have to make changes in the following > files throughout the project, > > - sql_parse.cc (do_command, dispatch_command, mysql_execute_command) Yes. And also sql_yacc.yy - to extend the parser. And the implementation of these commands, for example, sql_acl.cc for CREATE/DROP USER, sql_trigger.cc for CREATE/DROP TRIGGER, etc. > - sql_prepare.cc (Prepared_statement::execute_loop, > Prepared_statement::execute) No, hopefully, this will work automatically. > What I am not sure about is will I have to create a handler for some of > these commands? You are not creating new commands > And will lower level storage engine code changes required? I don't think so. > > > I am interested to work on "CREATE OR REPLACE, CREATE IF NOT EXISTS, and > > > DROP IF EXISTS" project as part of GSOC 2014. I want to apply my subject > > > knowledge at a bigger level which will also give me an idea about the > > > practical issues while implementing the features. > > > Regards, Sergei

3 9

[Maria-developers] [GSoC] MDEV-371- index on BLOB
by smit hinsu 21 Mar '14

21 Mar '14

Hi Sergei and Roberto, My name is Smit Hinsu and I am final year undergrad student. I have good knowledge about database internals and good experience with C++. I find feature of creating index on BLOB quickly using MD-5 interesting. I have understood basic requirements for this feature. I have following questions in my mind currently. I think, HASH index is not supported in MySQL's version of MyISAM storage engine. Has MariaDB removed this restriction? We should support index for BLOB for BTREE index also in which we store only hash value of the value in BTREE. Is this possible? Can you suggest me some startup task related to index or storage engine which can make me familiar with the related parts? I have started to look at code. From Smit Hinsu

2 2

[Maria-developers] Engine independent index statistics - infrastructure questions
by Sergey Petrunia 20 Mar '14

20 Mar '14

Hello Igor and others, I've been playing around with engine independent statistics, and got a few questions. == mysqldump == mysqldump dumps table' data, but besides that it can dump other related objects, e.g. it will dump triggers that are attached to the table. Should it dump table statistics values, also? This question is of interest for DBAs, but also for optimizer developers. If a user has a complaint about the optimizer, we frequently ask them to upload a dataset. How do we request table/column statistics? We can't request output of SELECT * ..., because histogram data is binary and will be corrupted. We should provide users a convinent way to pass us relevant part of the EITS tables. == dropping table statistics == How does one drop table statistics? are we satisfied with needing to run three statements DELETE FROM mysql.column_stats WHERE schema_name=$db AND table_name=$table_name; DELETE FROM mysql.index_stats WHERE table_name=$db AND table_name=$table_name; DELETE FROM mysql.column_stats WHERE table_name=$db AND table_name=$table_name; or we want a simpler SQL, like DROP STATISTICS FOR t1; == Permissions on statistic tables == What should be the permissions on statistics tables? I guess we should follow pattern from other system tables - a user is typically able to read the table but not write to it (so that he can't mess with other users' SPs/triggers/etc). However, if the user is not allowed to modify the table directly, we should provide enough ways to modify the table indirectly. For example, we should provide a way to drop statistics for one's tables (which brings us back to the previous item). We might also want to allow to make other modifications to statistics of tables that one has control of? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] [GSoC 2014]
by Ivan Nikulin 20 Mar '14

20 Mar '14

Hello, I would like to take part into one of your projects on base of GSoC 2014. I am an experienced C/C++ programmer with wide knowledge of algorithms and data structures and huge interest into this sphere. I have some basic knowledge of SQL language, but unfortunatelly I don't have any database development experience. Nevertheless, gaining the latter is my essential purpose. Can I count on a chance of taking part in one of your projects? Sorry for writing to you that late. P.S. I tried writing to you a bit earlier, but the mail wouldn't appear on mailing list. -- Best regards, Ivan Nikulin

2 1

Re: [Maria-developers] [Commits] Rev 4053: MDEV-5864 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.free_tables
by Sergei Golubchik 19 Mar '14

19 Mar '14

Hi, Sergey! On Mar 15, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 4053 > revision-id: svoj(a)mariadb.org-20140315185301-ce4r6br023co80dm > parent: sanja(a)montyprogram.com-20140314073116-nklb0jkjd4t1w40n > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Sat 2014-03-15 22:53:01 +0400 > message: > MDEV-5864 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.free_tables > > Let TABLE_SHARE::tdc.free_tables, TABLE_SHARE::tdc.all_tables, > TABLE_SHARE::tdc.flushed and corresponding invariants be protected by > per-share TABLE_SHARE::tdc.LOCK_table_share instead of global LOCK_open. This looks pretty good to me! A couple of comments, see below. > === modified file 'sql/sql_plist.h' > --- a/sql/sql_plist.h 2013-11-20 11:05:39 +0000 > +++ b/sql/sql_plist.h 2014-03-15 18:53:01 +0000 > @@ -142,6 +142,14 @@ class I_P_List : public C, public I > > return result; > } > + inline T *back() > + { > + T *result= m_first; > + if (result) > + while (*B::next_ptr(result)) > + result= *B::next_ptr(result); > + return result; > + } I think this is wrong. The correct way would be to use I_P_List<> with the I_P_List_fast_push_back policy, then you could use have get_last() method to get the last element in the list. May be as inline T *back() { return *I::get_last(); } > void swap(I_P_List<T, B, C> &rhs) > { > swap_variables(T *, m_first, rhs.m_first); > > === modified file 'sql/table_cache.cc' > --- a/sql/table_cache.cc 2014-03-06 12:19:12 +0000 > +++ b/sql/table_cache.cc 2014-03-15 18:53:01 +0000 > @@ -70,13 +70,6 @@ static int32 tc_count; /**< Number of TA > > > /** > - Protects TABLE_SHARE::tdc.free_tables, TABLE_SHARE::tdc.all_tables. > -*/ > - > -mysql_mutex_t LOCK_open; So, you've now removed LOCK_open completely? Cool! > - > - > -/** > Protects unused shares list. > > TABLE_SHARE::tdc.prev > @@ -360,12 +378,15 @@ bool tc_release_table(TABLE *table) > table->in_use= 0; > /* Add table to the list of unused TABLE objects for this share. */ > table->s->tdc.free_tables.push_front(table); > - mysql_mutex_unlock(&LOCK_open); > + mysql_mutex_unlock(&table->s->tdc.LOCK_table_share); > return false; > > purge: > + /* Wait for MDL deadlock detector */ > + while (table->s->tdc.all_tables_refs) > + mysql_cond_wait(&table->s->tdc.COND_release, &table->s->tdc.LOCK_table_share); you're using it many times, I'd put it in a small static inline function, like static wait_for_mdl_deadlock_detector(TABLE *table) { mysql_mutex_assert_owner(&table->s->tdc.LOCK_table_share); while (table->s->tdc.all_tables_refs) mysql_cond_wait(&table->s->tdc.COND_release, &table->s->tdc.LOCK_table_share); } > tc_remove_table(table); > - mysql_mutex_unlock(&LOCK_open); > + mysql_mutex_unlock(&table->s->tdc.LOCK_table_share); > table->in_use= 0; > intern_close_table(table); > return true; Regards, Sergei

2 2

[Maria-developers] [GSOC 2014] GIS Enhancement
by bang honam 19 Mar '14

19 Mar '14

Hi everybody. I am Honam Bang, a researcher of Korean Institute of Science Technology and I will be a master student of UCSD, USA from this Sep. I have been working with GIS research for about one year. Therefore, I decided to work on the 'GIS Enhancement topic' because I have a lot of interest for dealing with geospatial data. My topic is 'implementing shp to sql command tool for MariaDB'. #MDEV-5813. Most shptosql tools are enable to convert 3D coordinates as I know. The shape file white paper( http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf) specifies a lot of 3D coordinates like POINT Z, MULTIPOINTZ which does not exist in MariaDB's Geometry type. Does my topic include implementing those type converting as well? Thank you Honam Bang

2 1

[Maria-developers] first results from replication benchmarks
by Axel Schwenke 18 Mar '14

18 Mar '14

Hi everybody, here are some first results from my benchmarks of replication / group commit performance. The benchmark is sysbench OLTP rw, data set fitting in memory. I tested a total of 27 configurations, 9 each for Maria-5.5.36, Maria-10.0.9 and MySQL-5.6.16. Variations were for binlog on/off, SBR vs. RBR and sync_binlog=0/1. For MariaDB I also tested XtraDB vs. InnoDB, for MySQL it was GTID on vs. GTID off. Not surprisingly turning on the binlog comes with a performance penalty. Also sync_binlog=1 adds another penalty. Quite consistently RBR was a little bit slower than SBR for all combinations (I also observed the binlogs to be bigger for RBR by a factor of nearly 2). The only real surprise was the cost of GTID in MySQL as can be seen in the attached plot. MariaDB-10.0 has GTID enabled by default (you cannot even turn it off) Right now the same benchmark is running again, this time using a single disk for binlogs (was a fast RAID before). Next step is applying binlogs to a slave with different settings for number of slave threads etc. XL

2 2

[Maria-developers] Gsoc proposal submitted for user defined event
by Savita Agrawal 18 Mar '14

18 Mar '14

Hi, I have submitted my proposal for user defined event support project. I am still working on the proposal. Please give feedback. Regards, Savita

1 0