- developers - lists.mariadb.org

[Maria-developers] [GSoC] Week Nine!
by sriram patil 21 Jul '14

21 Jul '14

Hi All, There will be a little delay in the ninth week blog post. I need to get the code reviewed first to make the final changes. You can check the commit here <https://github.com/srirampatil/server/commit/3611bfc644714b797acff8b9884e7d…> though. I have also fixed MDEV-6409 <https://mariadb.atlassian.net/browse/MDEV-6409> and it is under review too. Regards, Sriram

1 0

[Maria-developers] Java Connector Coding till 20th July
by Puneet Dewan 20 Jul '14

20 Jul '14

Hi Everyone, Sorry,I did not update in the last week. It is because of my convocation and was meeting my parents after one year in Singapore, so enjoyed a small vacation of 5 days with them. I had code review and discussion with Massimo in that week. But I thought it is better to update next week. Things done: 1.I had initially written code in MySQLPreparedStatement.java but the code was getting ugly so as per recommnedation from Vladislav,I also discussed with my mentor.So we thought of using MySQLServerSidePreparedStatement as this will make code more cleaner. Now prepareStatement method in MySQLConnection will become the factory method to provide either MySQLPreparedStatement or MySQLServerSidePreparedStatement instance. I have migrated my code to MySQLServerSidePreparedStatement. 2.Test case for determining the type of instance, MySQLPreparedStatement or MySQLServerSidePreparedStatement. Next plan: 1.Write executeQuery and executeUpdate methods 2.For executeQuery, need to decide how I can use BinaryProtocolResultset to fetch results from a select query. 3.Write methods for batchstatements and multiqueries. Regards Puneet.

1 0

[Maria-developers] [GSoC] Optimize mysql-test-runs - Results of new strategy
by Pablo Estrada 17 Jul '14

17 Jul '14

Hello everyone, I took the last couple of days working on a new strategy to calculate the relevance of a test. The results are not sufficient by themselves, but I believe they point to an interesting direction. This strategy uses that rate of co-occurrence of events to estimate the relevance of a test, and the events that it uses are the following: - File editions since last run - Test failure in last run The strategy has also two stages: 1. Training stage 2. Executing stage In the training stage, it goes through the available data, and does the following: - If test A failed: - It counts and stores all the files that were edited since the last test_run (the last test_run depends on BRANCH, PLATFORM, and other factors) - If test A failed also in the previous test run, it also counts that In the executing stage, the training algorithm is still applied, but the decision of whether a test runs is based on its relevance, the relevance is calculated as the sum of the following: - The percentage of times a test has failed in two subsequent test_runs, multiplied by whether the test failed in the previous run (if the test didn't fail in the previous run, this quantity is 0) - For each file that was edited since the last test_run, the percentage of times that the test has failed after this file was edited (The explanation is a bit clumsy, I can clear it up if you wish so) The results have not been too bad, nor too good. With a running set of 200 tests, a training phase of 3000 test runs, and an executing stage of 2000 test runs, I have achieved recall of 0.50. It's not too great, nor too bad. Nonetheless, while running tests, I found something interesting: - I removed the first factor of the relevance. I decided to not care about whether a test failed in the previous test run. I was only using the file-change factor. Naturally, the recall decreased, from 0.50 to 0.39 (the decrease was not too big)... and the distribution of failed tests in the priority queue had a good skew towards the front of the queue (so it seems that the files help somewhat, to indicate the likelihood of a failure). I attached this chart. An interesting problem that I encountered was that about 50% of the test_runs don't have any file changes nor test failures, and so the relevance of all tests is zero. Here is where the original strategy (a weighted average of failures) could be useful, so that even if we don't have any information to guess which tests to run, we just go ahead and run the ones that have failed the most, recently. I will work on mixing up both strategies a bit in the next few days, and see what comes of that. By the way, I pushed the code to github. The code is completely different, so may be better to wait until I have new results soon. Regards! Pablo

2 11

[Maria-developers] Java Connector Coding till 5th June
by Puneet Dewan 17 Jul '14

17 Jul '14

Hi Everyone, The work done till 5th June: 1.I started studying the connector immediately after my semester exams got over(5th May). 2. Understood the relationship between different classes. 3.Read the MySql documentation for text protocol and map the commands from documentation onto code.This thing gave a clear idea about how things are moving from client to server. 4.Setup the code on Ubuntu using eclipse and tested against database. Setup the Team branch in launchpad. 5.Debug the test classes to see how different classes are interlinked and understood that MySQLProtocol.java is the heart of our JDBC driver. 6.Then I started reading Client Server Protocol documentation i.e ServerSide prepared statement and try to see how and where I have to code it. 7.Then I saw that there is a class MySqlServerSidePreparedStatement.java, Initially I coded the prepareStatement() ,close(),and execute() in that class seperately. But after (Code review) discussion with Massimo, we donot need to use that class. Instead use MySqlPreparedStatement.java to do the same work.So made changes accordingly. 8.I had pushed some of the code which I have written to launchpad and discussed with Massimo last week. 9.Last week I was struck with Wireshark on windows and ubuntu both. I was unable to see the packets going to database and seeing the response. 10.On Monday the problem was resolved as I was using the wrong interface.I should have used loopback.io(127.0.0.1) instead on ubuntu(thanks to Georg). Unfortunately the thing does not work on windows.Tried a lot many times. 11.Now I am working on the execute method of client server protocol as I am getting exception while using that method.Wireshark is playing a great role in how the packets are flowing. >From now I will do weekly blogging and keep everyone updated. Thanks & Regards Puneet.

2 9

[Maria-developers] Help needed TokuDB build errors in Debian
by Otto Kekäläinen 16 Jul '14

16 Jul '14

Hello Rich! For some strange reason, re-building 5.5.37 in Debian unstable stopped working. Any ideas what might be going on here?: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=753222 Recently a build of 5.5.38 generated a similar error: http://paste.ubuntu.com/7747594/ Are these maybe related? I have no idea how to fix these apart from disabling TokuDB in Debian when uploading 5.5.38. -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

3 5

[Maria-developers] [GSoC] Week Eight!
by sriram patil 14 Jul '14

14 Jul '14

Hi all, Following is the blog link about my eighth week into GSoC. http://goo.gl/h2mnbv Suggestions and/or reviews are welcome! Thanks, Sriram

1 0

Re: [Maria-developers] [Commits] Rev 4876: Post-fixes after UNIQUE index support was enabled without uniquenessenforcement. in file:///home/psergey/chroot/saucy-x64/home/psergey/dev2/mysql-5.6-rocksdb-r2/
by Roberto Spadim 11 Jul '14

11 Jul '14

=] nice rocksdb, any devel code to test it? i have some key-value databases to test 2014-07-11 13:21 GMT-03:00 Sergey Petrunya <psergey(a)askmonty.org>: > At > file:///home/psergey/chroot/saucy-x64/home/psergey/dev2/mysql-5.6-rocksdb-r2/ > > ------------------------------------------------------------ > revno: 4876 > revision-id: psergey(a)askmonty.org-20140711162114-051drr4tocqpbx30 > parent: psergey(a)askmonty.org-20140711161708-b50tdmlr60djrror > committer: Sergey Petrunya <psergey(a)askmonty.org> > branch nick: mysql-5.6-rocksdb-r2 > timestamp: Fri 2014-07-11 20:21:14 +0400 > message: > Post-fixes after UNIQUE index support was enabled without > uniquenessenforcement. > - Fix compile warning > - Fix the rocksdb.test/result > === modified file 'mysql-test/r/rocksdb.result' > --- a/mysql-test/r/rocksdb.result 2014-07-03 20:58:25 +0000 > +++ b/mysql-test/r/rocksdb.result 2014-07-11 16:21:14 +0000 > @@ -686,8 +686,8 @@ DROP TABLE t33; > # > # MDEV-4081: RocksDB throws error 122 on an attempt to create a table > with unique index > # > +# Unique indexes can be created, but uniqueness won't be enforced > create table t33 (pk int primary key, u int, unique index(u)) > engine=RocksDB; > -ERROR HY000: Got error 187 'Unique indexes are not supported' from ROCKSDB > # > # MDEV-4077: RocksDB: Wrong result (duplicate row) on select with range > # > > === modified file 'mysql-test/t/rocksdb.test' > --- a/mysql-test/t/rocksdb.test 2014-07-03 20:58:25 +0000 > +++ b/mysql-test/t/rocksdb.test 2014-07-11 16:21:14 +0000 > @@ -610,7 +610,8 @@ DROP TABLE t33; > --echo # > --echo # MDEV-4081: RocksDB throws error 122 on an attempt to create a > table with unique index > --echo # > ---error ER_GET_ERRMSG > +#--error ER_GET_ERRMSG > +--echo # Unique indexes can be created, but uniqueness won't be enforced > create table t33 (pk int primary key, u int, unique index(u)) > engine=RocksDB; > > > > === modified file 'storage/rocksdb/ha_rocksdb.cc' > --- a/storage/rocksdb/ha_rocksdb.cc 2014-07-09 23:46:24 +0000 > +++ b/storage/rocksdb/ha_rocksdb.cc 2014-07-11 16:21:14 +0000 > @@ -1051,7 +1051,7 @@ int ha_rocksdb::create(const char *name, > /* > We only support auto_increment at start of the PRIMARY KEY. > */ > - Field *field; > + //Field *field; > //if ((field= table_arg->next_number_field)) > /* TODO mdcallag: disable this for now to let UNIQUE indexes kind of > work > if ((field= table_arg->found_next_number_field)) > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits > -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

Re: [Maria-developers] apt-get upgrade on buildbot
by Kristian Nielsen 11 Jul '14

11 Jul '14

Otto Kekäläinen <otto(a)seravo.fi> writes: > Hello Kristian! > > Could you do a small favour and run 'apt-get upgrade' and then trigger > manually a rebuild in the virtual machine Debian unstable the buildbot > uses at http://buildbot.askmonty.org/buildbot/builders/debpkg-sid > > Thanks! Hm, I tried this, but I got the below errors. Any ideas what I did wrong? - Kristian. buildbot@debian-sid-amd64:~$ sudo apt-get update Hit http://ftp.us.debian.org sid InRelease Hit http://ftp.us.debian.org sid/main Sources/DiffIndex Hit http://ftp.us.debian.org sid/main amd64 Packages/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-en/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-cs/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ru/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-nl/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-eo/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ca/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-pl/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-fi/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-hr/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-fr/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-eu/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-sr/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-hu/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-da/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ro/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-es/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-pt/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-sv/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-sk/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ko/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-vi/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-uk/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-km/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ja/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-de/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-it/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-nb/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-zh/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-id/DiffIndex Get:1 http://ftp.us.debian.org sid/main Translation-ru [423 kB] Get:2 http://ftp.us.debian.org sid/main Translation-pl [575 kB] 100% [2 Translation-pl bzip2 0 B] 66.3 kB/s 0sFalling back to normal index file acquire Ign http://ftp.us.debian.org sid/main 2014-07-09-0854.18.pdiff Get:3 http://ftp.us.debian.org sid/main Translation-fr [731 kB] 100% [3 Translation-fr bzip2 0 B] 90.5 kB/s 0sFalling back to normal index file acquire Ign http://ftp.us.debian.org sid/main 2014-07-09-0854.18.pdiff Get:4 http://ftp.us.debian.org sid/main Translation-da [2,414 kB] Get:5 http://ftp.us.debian.org sid/main Translation-es [323 kB] Get:6 http://ftp.us.debian.org sid/main Translation-pt [444 kB] Get:7 http://ftp.us.debian.org sid/main Translation-sk [421 kB] Get:8 http://ftp.us.debian.org sid/main Translation-uk [776 kB] Get:9 http://ftp.us.debian.org sid/main Translation-ja [786 kB] Get:10 http://ftp.us.debian.org sid/main Translation-de [1,772 kB] Get:11 http://ftp.us.debian.org sid/main Translation-it [3,259 kB] Get:12 http://ftp.us.debian.org sid/main Sources [7,333 kB] Get:13 http://ftp.us.debian.org sid/main amd64 Packages [6,820 kB] Get:14 http://ftp.us.debian.org sid/main [116 kB] Get:15 http://ftp.us.debian.org sid/main [35.0 kB] Fetched 26.2 MB in 1min 31s (288 kB/s) W: Failed to fetch xz:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_source_Sources Hash Sum mismatch W: Failed to fetch xz:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_binary-amd64_Packages Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-ru Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-pl Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-fr Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-da Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-es Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-pt Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-sk Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-uk Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-ja Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-de Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-it Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-fi Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-hu Hash Sum mismatch E: Some index files failed to download. They have been ignored, or old ones used instead. buildbot@debian-sid-amd64:~$ sudo apt-get dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

2 2

[Maria-developers] [GSoC] Suggestions on adding a new method
by sriram patil 11 Jul '14

11 Jul '14

Hi All, I was thinking of adding two new inline methods in struct LEX as follows inline bool is_create_or_replace() { return (create_info.options & HA_LEX_CREATE_REPLACE); } inline bool is_create_if_not_exists() { return (create_info.options & HA_LEX_CREATE_IF_NOT_EXISTS); } With these, statements like if(lex->create_info.options & HA_LEX_CREATE_REPLACE) will be changed to if(lex->is_create_or_replace()) which definitely is more readable. This kind of statements are used in sql_yacc.yy, sql_parse.cc and many object specific implementation .cc files. Should I make the changes or is it unnecessary? Thanks, Sriram

1 0

Re: [Maria-developers] [JIRA] (MDEV-4937) NEED REVIEW: sql_slave_skip_counter does not work with GTID
by Kristian Nielsen 09 Jul '14

09 Jul '14

"Michael Widenius (JIRA)" <jira(a)mariadb.atlassian.net> writes: > Michael Widenius commented on MDEV-4937: > ---------------------------------------- Thanks for review Monty! I have some comments inline, let me know if you still want me to change something. - Kristian. > revno: 4260 > revision-id: knielsen at knielsen-hq.org-20140620104939-xdvn985cgmwy3odd > committer: Kristian Nielsen <knielsen at knielsen-hq.org> > timestamp: Fri 2014-06-20 12:49:39 +0200 > message: > MDEV-4937: sql_slave_skip_counter does not work with GTID > === modified file 'sql/slave.cc' > --- a/sql/slave.cc 2014-06-19 12:50:43 +0000 > +++ b/sql/slave.cc 2014-06-20 10:49:39 +0000 > @@ -3521,9 +3521,6 @@ static int exec_relay_log_event(THD* thd > > if (opt_gtid_ignore_duplicates) > { > - serial_rgi->current_gtid.domain_id= gev->domain_id; > - serial_rgi->current_gtid.server_id= gev->server_id; > - serial_rgi->current_gtid.seq_no= gev->seq_no; > int res= rpl_global_gtid_slave_state.check_duplicate_gtid > (&serial_rgi->current_gtid, serial_rgi); > if (res < 0) > What was the reason for removing the above assignments? > Was it a bug in the old code ? Not really a bug, however, just before this code we have: if (event_group_new_gtid(serial_rgi, gev)) and event_group_new_gtid() does the exact same assignments, so the ones I removed are completely redundant. I just happened to notice this redundancy, so I removed them... > (I have to ask as I am not 100% of what the old code did). > @@ -4594,16 +4598,27 @@ log '%s' at position %s, relay log '%s' > > if (saved_skip && rli->slave_skip_counter == 0) > { > + String tmp; > + if (mi->using_gtid != Master_info::USE_GTID_NO) > + { > + tmp.append(STRING_WITH_LEN(", GTID '")); > + rpl_append_gtid_state(&tmp, false); > + tmp.append(STRING_WITH_LEN("'; ")); > + } > > {noformat} > > If you want to avoid checking if tmp.append() fails, you could > create tmp with a static buffer. > > char tmp_buff[MAX_GTID_LENGTH+10]; > String tmp(tmp_buff, sizeof(tmp_buff), &my_charset_bin); > tmp.length(0); Right, hm. It does not really feel right to enforce a limit on the length; if the user uses a longer GTID position (which would be rather unusual), it might be confusing to have it silently cut. If the append() should fail, well, then it _will_ be cut, but then it's just an informational message, and that kind of out-of-memory is probably unlikely (and the error log will likely be filled up with other complaints about out-of-memory). But let me know if you still think I should change it. (If so - can you think of a place where MAX_GTID_LENGTH can be defined, that would be accessible to both client/mysqldump.c and sql/slave.cc? I could spot only my_global.h, which seems a bit too generic to contain GTID-related stuff...?) > if (saved_skip && rli->slave_skip_counter == 0) > { > String tmp; > saved_skip= 0; > + saved_skip_gtid_pos.free(); > } > If you free saved_skip_gtid_pos, it good to also free tmp! Right ... but it should be freed automatically anyway, when tmp goes out of scope, right? > === modified file 'sql/sys_vars.cc' > --- a/sql/sys_vars.cc 2014-06-09 18:00:23 +0000 > +++ b/sql/sys_vars.cc 2014-06-20 10:49:39 +0000 > @@ -4287,11 +4287,6 @@ bool update_multi_source_variable(sys_va > > static bool update_slave_skip_counter(sys_var *self, THD *thd, Master_info *mi) > { > - if (mi->using_gtid != Master_info::USE_GTID_NO) > - { > - my_error(ER_SLAVE_SKIP_NOT_IN_GTID, MYF(0)); > - return true; > - } > > {noformat} > > Shouldn't you also remove the error ER_SLAVE_SKIP_NOT_IN_GTID from the > error file? I think we cannot remove error codes from a GA release once introduced, right? So I left it, to not shift other error numbers. But let me know if I'm wrong, and I'll remove it. - Kristian.

1 0

Re: [Maria-developers] [JIRA] (MDEV-6336) NEED REVIEW: mysqldump --master-data does not work with GTID setups
by Kristian Nielsen 09 Jul '14

09 Jul '14

"Michael Widenius (JIRA)" <jira(a)mariadb.atlassian.net> writes: > Here is the review: > > ------------------------------------------------------------ > revno: 4245 > revision-id: knielsen at knielsen-hq.org-20140616093803-gub60pda6t77ch3k > committer: knielsen at knielsen-hq.org > timestamp: Mon 2014-06-16 11:38:03 +0200 > message: > MDEV-6336: mysqldump --master-data does not work with GTID setups > MDEV-6344: mysqldump issues FLUSH TABLES, which gets written into binlog and replicated Thanks for review Monty! I fixed all of your suggestions. - Kristian.

1 0

Re: [Maria-developers] [Commits] Rev 4281: MDEV-5262, MDEV-5914, MDEV-5941, MDEV-6020: Deadlocks during parallel replication causing replication to fail. in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Kristian Nielsen 09 Jul '14

09 Jul '14

Jan Lindström <jan.lindstrom(a)skysql.com> writes: > I have mostly InnoDB style comments, but some questions below: Thanks for comments! Sorry about style violations, and thanks for pointing then out. I'll try to fix all of it. Answers to other questions below: > Why you need to define these, why #include "mysql/plugin.h" file is not > enough ? > > +extern "C" void thd_report_wait_for(const MYSQL_THD thd, MYSQL_THD > other_thd); > +extern "C" int thd_need_wait_for(const MYSQL_THD thd); > +extern "C" > +int thd_need_ordering_with(const MYSQL_THD thd, const MYSQL_THD other_thd); See the discussions here: https://lists.launchpad.net/maria-developers/msg07494.html The conclusion with Serg was that doing a proper public API in 10.0 was too big a change. We will do it in 10.1 (MDEV-6429). plugin.h should not be modified in a GA version and with an API that has not been thought through wrt. all engines. Also, the API should be a service, not in plugin.h. > Why you need to introduce a new global structure for this, can't this be > part > of some already existing system structure e.g. lock_sys and why you do not > use ut_list ? > > +/* Buffer to collect THDs to report waits for. */ > +struct thd_wait_reports { > + struct thd_wait_reports *next; > + ulint used; > + trx_t *waitees[64]; > +}; So the idea here is to collect a bunch of trx pointers during the graph traversal, and pass each of them to thd_report_wait_for(). I wanted to pass them after completing the graph traversal to avoid issues if thd_report_wait_for() needs to kill one of them; this goes to lock_cancel_waiting_and_release() eventually which I assume might modify the graph being traversed. This is time-critical code (it runs under global lock_sys->mutex), so it is important to avoid memory allocations. So I use an array allocated on the stack, this is really short-lived data, it only lives for the duration of lock_deadlock_check_and_resolve(). In all normal cases the single array should be sufficient, but the code needs to handle the overflow also, of course. So in case of overflow, I malloc further arrays and link them into the first. I suppose another solution would be to abort in case of overflow and roll back the transaction like in case of ctx->too_deep==true, however I don't like to introduce such arbitrary limits when it's not necessary. I looked into using ib_list, but this seems to always do mallocs, which I want to avoid. I also looked at ut0vec and dyn0dyn, but they also seem to always use malloc and thus do not seem appropriate. So I did not find anything existing I could use, unfortunately, but then I know little of the InnoDB code. Do you have a suggestion for an existing data structure to use that can take an inital buffer on the stack and malloc extra memory only if needed? Thanks for comments! - Kristian.

1 0

Re: [Maria-developers] [Commits] Rev 4275: MDEV-5867 ALTER TABLE t1 ENGINE=InnoDB keeps bad options when t1 ENGINE is CONNECT in lp:~maria-captains/maria/10.0
by Sergey Vojtovich 08 Jul '14

08 Jul '14

Hi Sergei, I believe options handling is way too confusing in general: - CREATE TABLE warns/errors about unknown options - ALTER TABLE ... ENGINE doesn't - ALTER TABLE ... UNKNOWN_OPTION=1 does - CREATE TABLE ... LIKE doesn't - different engines may assign different meaning to the same option - IGNORE_BAD_TABLE_OPTIONS doesn't actually _ignore_ _bad_ options but rather _accepts_ _unkown_ options But the question is if we should show unknown options by default. I have no opinion: both cases are equally wrong to me. On the one hand we get unusable CREATE TABLE statement, on the other hand hidden metadata which may affect further statements. I'm fine with hidding unknown options that were accepted due to IGNORE_BAD_TABLE_OPTIONS, but hidding things that were accepted by fully successful ALTER TABLE statement sounds confusing. Regards, Sergey On Sat, Jul 05, 2014 at 11:42:25PM +0200, Sergei Golubchik wrote: > At lp:~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4275 > revision-id: sergii(a)pisem.net-20140705214225-p0k45asdaq27xocr > parent: sergii(a)pisem.net-20140705214216-npqtc7tz854xqjxo > fixes bug: https://mariadb.atlassian.net/browse/MDEV-5867 > committer: Sergei Golubchik <sergii(a)pisem.net> > branch nick: 10.0 > timestamp: Sat 2014-07-05 23:42:25 +0200 > message: > MDEV-5867 ALTER TABLE t1 ENGINE=InnoDB keeps bad options when t1 ENGINE is CONNECT > > don't print unknown options in SHOW CREATE TABLE unless IGNORE_BAD_TABLE_OPTIONS is used > === added file 'mysql-test/r/table_options-5867.result' > --- a/mysql-test/r/table_options-5867.result 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/r/table_options-5867.result 2014-07-05 21:42:25 +0000 > @@ -0,0 +1,37 @@ > +install soname 'ha_example'; > +set sql_mode='ignore_bad_table_options'; > +create table t1 ( > +a int complex='c,f,f,f' invalid=3 > +) engine=example ull=10000 str='dskj' one_or_two='one' yesno=0 > +foobar=barfoo; > +Warnings: > +Warning 1911 Unknown option 'invalid' > +Warning 1911 Unknown option 'foobar' > +create table t2 (a int, key (a) some_option=2014); > +Warnings: > +Warning 1911 Unknown option 'some_option' > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL `complex`='c,f,f,f' `invalid`=3 > +) ENGINE=EXAMPLE DEFAULT CHARSET=latin1 `ull`=10000 `str`='dskj' `one_or_two`='one' `yesno`=0 `foobar`=barfoo `VAROPT`='5' > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `a` int(11) DEFAULT NULL, > + KEY `a` (`a`) `some_option`=2014 > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +set sql_mode=''; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL `complex`='c,f,f,f' > +) ENGINE=EXAMPLE DEFAULT CHARSET=latin1 `ull`=10000 `str`='dskj' `one_or_two`='one' `yesno`=0 `VAROPT`='5' > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `a` int(11) DEFAULT NULL, > + KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +drop table t1, t2; > +uninstall soname 'ha_example'; > > === modified file 'mysql-test/suite/rpl/r/rpl_table_options.result' > --- a/mysql-test/suite/rpl/r/rpl_table_options.result 2013-09-14 01:09:36 +0000 > +++ b/mysql-test/suite/rpl/r/rpl_table_options.result 2014-07-05 21:42:25 +0000 > @@ -12,6 +12,12 @@ show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > `a` int(11) NOT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +set sql_mode=ignore_bad_table_options; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) NOT NULL > ) ENGINE=MyISAM DEFAULT CHARSET=latin1 `ull`=12340 > drop table t1; > set storage_engine=default; > > === modified file 'mysql-test/suite/rpl/t/rpl_table_options.test' > --- a/mysql-test/suite/rpl/t/rpl_table_options.test 2011-10-19 19:45:18 +0000 > +++ b/mysql-test/suite/rpl/t/rpl_table_options.test 2014-07-05 21:42:25 +0000 > @@ -23,6 +23,8 @@ show create table t1; > sync_slave_with_master; > connection slave; > show create table t1; > +set sql_mode=ignore_bad_table_options; > +show create table t1; > > connection master; > drop table t1; > > === added file 'mysql-test/t/table_options-5867.test' > --- a/mysql-test/t/table_options-5867.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/table_options-5867.test 2014-07-05 21:42:25 +0000 > @@ -0,0 +1,30 @@ > +# > +# MDEV-5867 ALTER TABLE t1 ENGINE=InnoDB keeps bad options when t1 ENGINE is CONNECT > +# > +# verify that SHOW CREATE TABLE hides unknown options when IGNORE_BAD_TABLE_OPTIONS is not set > + > +--source include/have_example_plugin.inc > +--source include/not_embedded.inc > + > +install soname 'ha_example'; > + > +set sql_mode='ignore_bad_table_options'; > +create table t1 ( > + a int complex='c,f,f,f' invalid=3 > +) engine=example ull=10000 str='dskj' one_or_two='one' yesno=0 > + foobar=barfoo; > + > +create table t2 (a int, key (a) some_option=2014); > + > +show create table t1; > +show create table t2; > + > +set sql_mode=''; > + > +show create table t1; > +show create table t2; > + > +drop table t1, t2; > + > +uninstall soname 'ha_example'; > + > > === modified file 'sql/create_options.cc' > --- a/sql/create_options.cc 2014-03-26 08:33:03 +0000 > +++ b/sql/create_options.cc 2014-07-05 21:42:25 +0000 > @@ -775,3 +775,20 @@ engine_option_value *merge_engine_table_ > &first, &end); > DBUG_RETURN(first); > } > + > +bool is_engine_option_known(engine_option_value *opt, > + ha_create_table_option *rules) > +{ > + if (!rules) > + return false; > + > + for (; rules->name; rules++) > + { > + if (!my_strnncoll(system_charset_info, > + (uchar*)rules->name, rules->name_length, > + (uchar*)opt->name.str, opt->name.length)) > + return true; > + } > + return false; > +} > + > > === modified file 'sql/create_options.h' > --- a/sql/create_options.h 2013-11-20 11:05:39 +0000 > +++ b/sql/create_options.h 2014-07-05 21:42:25 +0000 > @@ -99,4 +99,6 @@ uchar *engine_table_options_frm_image(uc > > bool engine_options_differ(void *old_struct, void *new_struct, > ha_create_table_option *rules); > +bool is_engine_option_known(engine_option_value *opt, > + ha_create_table_option *rules); > #endif > > === modified file 'sql/sql_show.cc' > --- a/sql/sql_show.cc 2014-07-05 21:42:16 +0000 > +++ b/sql/sql_show.cc 2014-07-05 21:42:25 +0000 > @@ -1519,13 +1519,20 @@ static bool get_field_default_value(THD > @param thd thread handler > @param packet string to append > @param opt list of options > + @param check_options only print known options > + @param rules list of known options > */ > > static void append_create_options(THD *thd, String *packet, > - engine_option_value *opt) > + engine_option_value *opt, > + bool check_options, > + ha_create_table_option *rules) > { > for(; opt; opt= opt->next) > { > + if (check_options && !is_engine_option_known(opt, rules)) > + continue; > + > DBUG_ASSERT(opt->value.str); > packet->append(' '); > append_identifier(thd, packet, opt->name.str, opt->name.length); > @@ -1584,6 +1591,8 @@ int show_create_table(THD *thd, TABLE_LI > MODE_MAXDB | MODE_ANSI); > bool limited_mysql_mode= sql_mode & (MODE_NO_FIELD_OPTIONS | MODE_MYSQL323 | > MODE_MYSQL40); > + bool check_options= !(sql_mode & MODE_IGNORE_BAD_TABLE_OPTIONS) && > + !create_info_arg; > handlerton *hton; > my_bitmap_map *old_map; > int error= 0; > @@ -1733,7 +1742,8 @@ int show_create_table(THD *thd, TABLE_LI > packet->append(STRING_WITH_LEN(" COMMENT ")); > append_unescaped(packet, field->comment.str, field->comment.length); > } > - append_create_options(thd, packet, field->option_list); > + append_create_options(thd, packet, field->option_list, check_options, > + hton->field_options); > } > > key_info= table->key_info; > @@ -1800,7 +1810,8 @@ int show_create_table(THD *thd, TABLE_LI > append_identifier(thd, packet, parser_name->str, parser_name->length); > packet->append(STRING_WITH_LEN(" */ ")); > } > - append_create_options(thd, packet, key_info->option_list); > + append_create_options(thd, packet, key_info->option_list, check_options, > + hton->index_options); > } > > /* > @@ -1953,7 +1964,8 @@ int show_create_table(THD *thd, TABLE_LI > packet->append(STRING_WITH_LEN(" CONNECTION=")); > append_unescaped(packet, share->connect_string.str, share->connect_string.length); > } > - append_create_options(thd, packet, share->option_list); > + append_create_options(thd, packet, share->option_list, check_options, > + hton->table_options); > append_directory(thd, packet, "DATA", create_info.data_file_name); > append_directory(thd, packet, "INDEX", create_info.index_file_name); > } > @@ -5130,7 +5142,7 @@ static int get_schema_tables_record(THD > str.qs_append(STRING_WITH_LEN(" transactional=")); > str.qs_append(ha_choice_values[(uint) share->transactional]); > } > - append_create_options(thd, &str, share->option_list); > + append_create_options(thd, &str, share->option_list, false, 0); > > if (str.length()) > table->field[19]->store(str.ptr()+1, str.length()-1, cs); > > === modified file 'storage/connect/mysql-test/connect/r/alter.result' > --- a/storage/connect/mysql-test/connect/r/alter.result 2014-03-23 14:50:39 +0000 > +++ b/storage/connect/mysql-test/connect/r/alter.result 2014-07-05 21:42:25 +0000 > @@ -218,13 +218,22 @@ Three 3 > # Changing to another engine is Ok > # However, the data file is not deleted. > # > -ALTER TABLE t1 ENGINE=MARIA; > +ALTER TABLE t1 ENGINE=ARIA; > +SHOW CREATE TABLE t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` char(10) NOT NULL, > + `c` int(11) NOT NULL > +) ENGINE=Aria DEFAULT CHARSET=latin1 PAGE_CHECKSUM=1 > +set @old_sql_mode=@@sql_mode; > +set sql_mode=ignore_bad_table_options; > SHOW CREATE TABLE t1; > Table Create Table > t1 CREATE TABLE `t1` ( > `d` char(10) NOT NULL `FLAG`=11, > `c` int(11) NOT NULL `FLAG`=0 > ) ENGINE=Aria DEFAULT CHARSET=latin1 PAGE_CHECKSUM=1 `TABLE_TYPE`=fix `FILE_NAME`='tf1.txt' `ENDING`=1 > +set sql_mode=@old_sql_mode; > SELECT * from t1; > d c > One 1 > > === modified file 'storage/connect/mysql-test/connect/t/alter.test' > --- a/storage/connect/mysql-test/connect/t/alter.test 2014-03-23 14:50:39 +0000 > +++ b/storage/connect/mysql-test/connect/t/alter.test 2014-07-05 21:42:25 +0000 > @@ -105,8 +105,12 @@ SELECT * FROM t1; > --echo # Changing to another engine is Ok > --echo # However, the data file is not deleted. > --echo # > -ALTER TABLE t1 ENGINE=MARIA; > +ALTER TABLE t1 ENGINE=ARIA; > SHOW CREATE TABLE t1; > +set @old_sql_mode=@@sql_mode; > +set sql_mode=ignore_bad_table_options; > +SHOW CREATE TABLE t1; > +set sql_mode=@old_sql_mode; > SELECT * from t1; > SELECT * from t2; > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

4 7

[Maria-developers] Java Connector coding till 6th July
by Puneet Dewan 08 Jul '14

08 Jul '14

Hi Everyone, Sorry for the late mail. Last week I had code review with Massimo. I wrote the test cases for transactions. Then I clean the code written in MySQLProtocol.java for executePreparedStatement method. Then added new methods like writeBinaryTo(for BinaryProtocol) in IntParameter and will write similar methods in other classes. In the coming week: *a)*I will be making use of MYSQLServerSidePreparedStatement.java for performing the all the operations.Currently they are embedded in MYSQLPreparedStatement.java *b)*I have done the analysis of the above as to what classes will be changed. *c)*Also I will be modifying the execute method for Serverside prepared statements for sending the *new params bound flag* properly. Currently it is always going as 0x01 but will modify it to send 0x00 if same statement gets executed again I will have code review with Massimo on thursday.. Your suggestions are welcome. Regards Puneet.

2 2

Re: [Maria-developers] comments to the plugin.h
by Kristian Nielsen 08 Jul '14

08 Jul '14

Sergei Golubchik <vuvova(a)gmail.com> writes: > This is what I've written in the patch: I've tried to answer all points in detail, also to help myself better understand all the issues: >> + The storage engine can report such conflicting locks using this call. This >> + will allow parallel replication to detect such conflicts and resolve the >> + deadlock (by killing the second transaction to release the locks that the >> + first is waiting for, and then later re-try the second killed transaction). > > What will happen if the engine doesn't? Deadlocks will timeout. > Which is annoying, but it only affects the performance, not the > correctness. No, unfortunately that is not so. Let's say we have T1 and T2. T1 goes to wait on a row lock held by T2. Then T2 completes its prepare phase, it goes to the group commit queue and waits for T1 to complete and commit for both of them. T1 gets lock wait timeout, so T1 will roll back and try again. But T1 will just go to wait on the same row lock again that is held by T2. So T1 will retry over and over until slave_transaction_retries is exceeded, at which point replication will abort with an error. There is no lock wait timeout when T2 waits in the group commit queue. > But can it cause the first transaction to be killed, not > the second? Then it's worse - it'll violate the strict commit ordering. No, this can never happen. Commit order is enforced by the wait_for_commit facility. T2 will not be allowed to commit before T1. If T1 is killed and retried, T2 will continue waiting for T1 to commit the second time. And if T1 fails, it will call wakeup_subsequent_commits() with an error, so that T1 can abort its commit. > And, again, the engine needs to do it for every lock wait. Even while most of > them won't cause any deadlocks. Right. This is only needed for parallel replication worker threads. That is why I introduced thd_need_wait_for(). The storage engine can call this once for a transaction or even for a THD, and save the result. And then it won't need to call thd_report_wait_for() except for the parallel replication worker threads. And for those, lock waits should be rare. >> + The storage engine should not report false positives. That is, it should not >> + report any lock waits that do not actually require one transaction to wait >> + for the other. Nor should it report waits for locks that will be released >> + before the commit of the other transactions. > > What will happen if the engine reports false positives? Lock waist > that don't require one transaction to wait for another. What if it > report waits for locks that will be released before commit of the other > transaction? If it reports that T1 will wait for T2, then we will kill T2 and retry it. This will cause unnecessary work if there is no actual deadlock, so mainly a performance issue. A deeper problem is caused by the async kill processing. If T1 is reported as waiting for T2, we will kill T2 in the background thread. But if there is no such wait, or if the wait is resolved due to T2 releasing locks before commit, then T1 and T2 could both complete, and their worker threads move on to new transactions. Then the background thread will be trying to kill a T2 which no longer exists, which will access incorrect rpl_group_info objects. This is the part of the patch I am most unhappy about :-/ But we started to discuss on IRC to possibly avoid the async kill, instead making it possible to call THD::awake() directly inside thd_report_wait_for(). This would be good, as it would solve the deeper problem. In this case, a false positive would then cause only performance issues, I think, though I should think the problem through carefully to be sure. >> +void thd_report_wait_for(const MYSQL_THD thd, MYSQL_THD other_thd); > > You kill the second thread in a background. > What if you convey the ordering to the engine and let it know about a > deadlock? Like, if you return 1 from thd_report_wait_for() > it means a deadlock and the engine will kill the transaction > just normally as for any normal deadlock case. The problem is that T2 could be executing anywhere in the server, including server code or another storage engine, where InnoDB does not really have any possibility to kill it. This is different from a normal InnoDB deadlock, which only happens if both T1 and T2 are doing an actual row lock wait inside the storage engine itself. But in this case, only T1 is waiting, T2 is continuing whatever processing it is doing, eventually it will end up in the group commit queue in binlog or in wait_for_prior_commit(), and wait for T1 to commit first. >> +int thd_need_wait_for(const MYSQL_THD thd); > > I'd suggest to remove it. If you think it's useful, you can call it from > your thd_report_wait_for() and spare storage engine the knowledge of yet > another API function. See discussion above. The intention is to avoid lots of calls to thd_report_wait_for() from the storage engine for non-replication threads. So calling it from inside thd_report_wait_for() makes no sense. I don't understand the benefits of reducing the functionality available in the API when that functionality is completely optional? But on the other hand, I do not feel strongly about this function. I do not even use it in the patch, as InnoDB does a rather expensive graph traversal of all waiters anyway in its deadlock detector, so it can be just omitted. I included this function because without it, there seems no way for a storage engine to avoid introducing O(N**2) overhead for N transactions contending for a row lock. That seemed unacceptable to me. >> +int thd_need_ordering_with(const MYSQL_THD thd, const MYSQL_THD other_thd); > > Hmm, really? Couldn't you simply disable gap locks in some other already > existing way? Like decrease the transaction isolation level or > tell the engine that it's RBR? No, that is not safe. Suppose we have two transactions: T1: INSERT INTO t1 VALUES (1); T2: INSERT INTO t2 SELECT * FROM t1; T1 and T2 are not run in parallel, because they cannot group commit together on the master. However, on the slave, as soon as T1 _starts_ to commit, T2 is allowed to start. But T2 cannot see the changes done by T1 until T1 _completes_ its commit. Thus it is critical that gap locks are disabled _only_ among transactions within the same group commit, not between different group commits. In fact there is a bug in the patch you got regarding this, as it did not check the commit_id before relaxing gap locks. I currently have the following extra patch in my tree to fix this: === modified file 'sql/sql_class.cc' --- sql/sql_class.cc 2014-06-10 08:13:15 +0000 +++ sql/sql_class.cc 2014-06-27 11:47:53 +0000 @@ -4277,6 +4277,8 @@ thd_need_ordering_with(const MYSQL_THD t return 1; if (rgi->current_gtid.domain_id != other_rgi->current_gtid.domain_id) return 1; + if (rgi->commit_id != other_rgi->commit_id) + return 1; /* These two threads are doing parallel replication within the same replication domain. Their commit order is already fixed, so we do not need On the other hand, if we were to omit this part of the patch completely and just keep the gap locks always, then things would still work, because thd_report_wait_for() would detect a deadlock and do a retry to resolve it. But we would get extra rollbacks. Maybe such extra rollbacks would be rare, though. >> +int thd_deadlock_victim_preference(const MYSQL_THD thd1, const MYSQL_THD thd2); >> + > > Okay, I see why this one is needed. > > On the other hand... > It seems like it's the only function that you realy need. > Because > > thd_report_wait_for(const MYSQL_THD thd, MYSQL_THD other_thd); > > can be replaced with > > if (thd_deadlock_victim_preference(thd, other_thd) < 0) goto deadlock; > > And > > thd_need_wait_for(const MYSQL_THD thd); > > can be removed (replaced with noop). And > > thd_need_ordering_with(const MYSQL_THD thd, const MYSQL_THD other_thd); > > can be replaced with > > thd_deadlock_victim_preference(thd, other_thd) != 0 I don't understand the purpose of using the same function for these? They do completely different things, using a single function will only cause confusion, it seems to me: 1. thd_report_wait_for() is used to allow the _replication code_ to detect a deadlock related to ordering. 2. thd_deadlock_victim_preference() is only used when the _storage engine_ detects a deadlock, replication related or not. 3. thd_need_ordering_with() is used by the storage engine to decide on lock wait semantics. > Although in that case thd_deadlock_victim_preference() should better be > renamed, because it won't be a generic "victim preference", > but specifically victim preference, depending on the fixed transaction > commit ordering. A better name could be, for example, > > thd_get_commit_order(thd, other_thd) > -1 - thd first > 0 - unspecified > +1 - other_thd first How can the storage engine know the difference between T1 and T2 being in the same group commit, or in different group commits? How can the storage engine inquire about need for gap locks, without accidentally reporting a lock wait to the upper layer? > I think it'll be much simpler with only one function. Why? I think it's quite the opposite... > And also it becomes clear what happens if the engine doesn't use it. > This function is used to enquire about the commit order, if it is > not used - the commit order cannot be guaranteed. No, that is not the case. On the contrary - the storage engine is free not to use this, commit order will be guaranteed in any case by wait_for_commit(). Hm. So my idea was to make the new API simple and generic, the upper layer would handle most of the detail about commit ordering or whatever. The storage engine only needs to think about simple stuff like lock waits or deadlock victims, both of which it likely already needs to deal with anyway. But if I understand correctly, another idea could be to make the new API specific just to this particular problem of enforced commit order in parallel replication. So the storage engine would be able to know of such ordering, and should then handle any necessary actions on this by itself? > And you won't need background killing for slave threads The need for background killing comes because it does not work to call THD::awake() from the context of where the InnoDB deadlock detector calls thd_report_wait_for(). The problem is that innobase_kill_query() does a recursive mutex lock on lock_sys->mutex. We discussed on IRC to fix this so background kill is not necessary. One way would be to document that hton->kill_query must be safe to call from within thd_report_wait_for(). Maybe that is the simplest way. It should not be hard to fix the InnoDB code to make this work. For example, InnoDB could just set a flag in the transaction that it already has lock_sys->mutex held, thus avoiding the recursive mutex lock. You also had a different suggestion: > here's another idea. To have thd_get_commit_order() as I thought originally > *and* allow innodb to kill other threads, that is, add a function for > plugins to invoke THD::awake. This function should not call into the engine > that invoked it. That is, supposedly the engine knows how to kill threads > stuck in that engine, to kill other threads it should use this new helper. > In that case innodb can detect the deadlock and can kill the other thread > and it won't break on lock mutex > The underlying logic of this killer helper would be "an engine knows how to > kill a transaction waiting in this engine, so we wouldn't bother doing > that. but to kill a transaction waiting elsewhere the engine needs help" It seems better to me if all the details of enforcing commit order and killing worker threads in case of deadlocks could be kept inside the replication code? Otherwise every storage engine would potentially need to re-implement this? I think that is what I tried to do with my proposed storage engine API, to cleanly separate the storage engine's responsibilities, which is about row-level locks and internal deadlock victims, from the replication codes responsibilities. Wouldn't it be simpler to do it like this? 1. InnoDB deadlock detector reports lock waits to whatever function serves the thd_report_wait_for() purpose. 2. We document that thd_report_wait_for() may need to call hton->kill_query(). 3. We fix InnoDB to handle (2) correctly (assuming this is possible to do in a simple way). > And anyway, this absolutely must be in a service, not in the global plugin.h. Why? My understanding was that we do not support ABI stability for storage engines. (And I think we should absolutely not provide such support for storage engines. Storage engines are all about performance and flexibility - ABI stability will make this harder to provide without sufficient other benefits to justify this. ABI seems mainly suited for eg. authentication plugins). > See? In this patch you've removed a function from the global plugin API Right, the thd_rpl_is_parallel(). That should have never be allowed into the code in the first place, it was a temporary hack only. :-( > header, strictly speaking, you must increase the API version - the major > number of - making all older plugins incompatible with the new server. This > is a bit extreme, taking into account that no plugin actually uses > thd_rpl_is_parallel() > To avoid this, please only add new APIs as services. > See libservices/HOWTO I can do that, if you want. It just seems a needless complication to me... Are there any performance penalties for using a service over a normal function call (for built-in storage engines)? I seem to recall there is not? - Kristian.

2 6

Re: [Maria-developers] [Commits] Rev 4274: small cleanup of the SHOW CREATE TABLE code in lp:~maria-captains/maria/10.0
by Sergey Vojtovich 08 Jul '14

08 Jul '14

Hi Sergei, looks good. One question inline. On Sat, Jul 05, 2014 at 11:42:16PM +0200, Sergei Golubchik wrote: > At lp:~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4274 > revision-id: sergii(a)pisem.net-20140705214216-npqtc7tz854xqjxo > parent: sergii(a)pisem.net-20140704190257-1z1inye8jd7k7umc > committer: Sergei Golubchik <sergii(a)pisem.net> > branch nick: 10.0 > timestamp: Sat 2014-07-05 23:42:16 +0200 > message: > small cleanup of the SHOW CREATE TABLE code ...skip... > === modified file 'sql/sql_show.cc' > --- a/sql/sql_show.cc 2014-06-05 22:07:27 +0000 > +++ b/sql/sql_show.cc 2014-07-05 21:42:16 +0000 > @@ -1578,27 +1577,32 @@ int store_create_info(THD *thd, TABLE_LI > handler *file= table->file; > TABLE_SHARE *share= table->s; > HA_CREATE_INFO create_info; > -#ifdef WITH_PARTITION_STORAGE_ENGINE > bool show_table_options= FALSE; > -#endif /* WITH_PARTITION_STORAGE_ENGINE */ Why did you remove ifdef-s here? This variable seem to be unused without partitioning. ...skip... Regards, Sergey

2 1

[Maria-developers] doubt about commit and myisam/aria engine
by Roberto Spadim 07 Jul '14

07 Jul '14

hi guys, i'm with a doubt... when a aria or myisam file is fsync to disk? when i execute commit? when i execute a autocommit? -- Roberto Spadim SPAEmpresarial

1 0

[Maria-developers] [GSoC] Week Seven!
by sriram patil 07 Jul '14

07 Jul '14

Hi all, Following is the blog link about my seventh week into GSoC. http://goo.gl/jBeubY Suggestions and/or reviews are welcome! Thanks, Sriram

1 0

[Maria-developers] comments on parallel applying
by Jonas Oreland 04 Jul '14

04 Jul '14

hi there, disclaimer: i haven't followed all turns and details on this task so sorry if my comments are outdated (or generally uninteresting). --- i've been glancing on the review comments on the parallel applying, and have some comments to make. i think it will be well invested time to add infrastructure to be able to restrict when parallel applying will be performed. typically i think it would be nice to - only allow parallel transaction in same engine - decrease parallelism when performing DDL ...(more of these can probably be added) restrictions that will make it easier to reason about correctness and that will not affect performance for 99% of the cases (i think :-) if adding such restrictions was an option, the resolving review comments on weird corner cases would be much easier. --- from what I can tell, there are currently no such restrictions, and no infrastructure to add them either. i realize that it might not be trivial to introduce such restrictions, but personally i think it will be worth the effort to get a solid solution in place. <quick thoughts on implementation> for row-based replication this seems quite "easy". for statement-based replication i image that you would have to add hooks into the "real" code after parsing has been performed, but before the actual execution is started (and yes, i know that there is sometimes a blurry line here) </thoughts> --- /Jonas

3 4

[Maria-developers] Review request for MDEV-6020, MDEV-5914, MDEV-5941, and MDEV-5262
by Kristian Nielsen 04 Jul '14

04 Jul '14

Hi Serg, We discussed on IRC a review of the $subj bugs related to deadlocks in parallel replication and handling of automatic retry of transactions in case of deadlock. The patches are in my tree: lp:~maria-captains/maria/10.0-knielsen The complete diff can be obtained eg. with: bzr diff -rrevid:psergey@askmonty.org-20140429143317-tkmxrzu6y36hpi7t..revid:knielsen@knielsen-hq.org-20140610081315-o5nx3u32uvqaai7q Let me give some background for these bugs: When two transactions T1 and T2 group-commit on the master, we assume that they have no lock conflicts (otherwise one would have had to wait for the other to commit). So we run them in parallel on the slave. But before we let T2 commit, we wait until T1 has committed first, so that we get the same binlog order on the slave as we had on the master. But if T1 and T2 somehow _do_ get a lock conflict on the slave, and T1 ends up waiting for some lock held by T2, we get a deadlock. This causes the replication to hang (the wait of T2 for T1 to commit is not part of InnoDB deadlock detection), which is quite bad. Elena's testing of the parallel replication in fact uncovered a number of cases where such deadlock issues occur: 1. MDEV-5914. It turns out that InnoDB gap locks can work in a non-symmetric way. If T1 is an INSERT, and T2 is a DELETE, then T1 can run first and not block T2. But if T2 runs first, then it may set a gap lock that blocks T1. Thus, depending on exact execution order, we can get T1 and T2 to group commit on the master, but T2 block T1 on the slave and cause a deadlock. 2. MDEV-5941. If eg. an UPDATE uses different execution plans on the master and slave, it might use index scan on the master, but table scan on the slave. Thus it may take more locks on the slave than on the master, causing it to conflict with another transaction that is run in parallel and cause a deadlock. 3. MDEV-6020. Here, there is a multi-value insert into multiple partitions of a table with auto-increment key. It appears that in this case InnoDB takes an auto-increment table lock on each partition. Since these locks are taken in the order of the inserted values, they can be taken in different order between different INSERT statements, and can cause a deadlock. Since these locks are released during the statement execution (before the commit), such deadlock can occur on the slave even though it did not occur on the master. (Note that this deadlock _is_ detected by InnoDB, and is not related to the commit order on the parallel slave). I know a way to prevent (1) on the slave (and it is implemented in the patches). I do not know if there is a way to prevent (3), it would require understanding better how innodb autoincrement and partitioning works. I do not see any way to prevent (2) from occuring on the slave, in the general case. It also seems to me that there are likely to be more corner cases lurking, and even if I somehow managed to fix the three above problems, there would be some other cases where an extra lock conflict can occur on the slave and cause the slave to hang. So I decided that a general solution is needed to such deadlocks. So the currently proposed patch does two things: A. It adds code to InnoDB, so that whenever T1 starts to wait inside InnoDB for a lock held by T2, this wait is reported to the upper server layer. The replication code will then check if T1 and T2 are part of parallel replication such that T1 must commit before T2. If so, we have a deadlock, and the code will terminate T2 with a deadlock error, allowing T1 to proceed. B. It implements code to handle deadlocks during parallel replication. When a transaction gets a deadlock error (normal, or provoked by the new deadlock detection code from (B)), it is rolled back, and re-executed. The result should be that even if there are other non-discovered cases where locks can conflict in different ways on the master from on the parallel slave, such conflicts will not cause replication to fail. Instead they will be detected, and handled automatically by resolving the deadlock. I think this will be important to ensure the reliability of parallel replication for users. I went through quite a few different ideas and attempts before ending with the present solution. So I wanted to give this background about how the patch relates to the basic problem that needs to be solved. I found a _lot_ of tricky issues during testing, and so I am somewhat unhappy about the complexity of this, even though the final patch may look simple enough on the surface. Maybe you can have some ideas for what could be done simpler, or at least question anything that looks incorrect or unnecessary. Thanks, - Kristian.

2 3

[Maria-developers] Debian packaging - option to enable feedback plugin
by Colin Charles 04 Jul '14

04 Jul '14

Hi! This is probably a great idea: https://answers.launchpad.net/maria/+question/248639 Let's ask dpkg to optionally enable feedback plugin? This could be good for us Just to be clear: I'm referring to this for the packages at mariadb.org I don't know if Debian upstream would like that -- Otto, any thoughts? cheers, -colin -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

5 5

Re: [Maria-developers] Review request for MDEV-6020, MDEV-5914, MDEV-5941, and MDEV-5262
by Kristian Nielsen 04 Jul '14

04 Jul '14

Jan Lindström <jan.lindstrom(a)skysql.com> writes: > 1) In innodb deadlock detection it is not certain that there is actually a > deadlock, deadlock detection is done when a lock request is issued. At that > point yes, we can see from lock manager that there is a deadlock, however > if waiting transaction is rolled back e.g. by user request concurrently, > the deadlock detected is not valid anymore. Hm, interesting. It was my impression that the deadlock detection algorithm only considers transactions that are already waiting for a row lock. That is, if we detect a simple deadlock between two transactions T1 and T2, then it is because T2 is waiting for a row lock held by T1 while executing some user query, and T1 is executing another user query that requests another row lock held by T2. So since both T1 and T2 are already executing user queries, and will not complete until the deadlock is resolved, how could the user roll either of them back concurrently? > 2) It is fine to roll back all transactions participating to deadlock, this > is correct. Note that there are some preferences which transaction is > rolled back on normal InnoDB server and this could be different when we > have Galera > 3) How this new needs to wait work with Galera where we have so called > brute force transactions that may not wait anything ? Note that this is only for resolving deadlocks between replicated transactions from a single master. If T1 and T2 are both replicated from the same master, we might decide to kill T2 to let T1 commit first. If I understand correctly, "brute force" transactions are only for transactions that are replicated from other Galera nodes, and which _must_ be committed in a specific order to preserve correctness. Thus, a brute-force transaction would not be affected by a deadlock kill from parallel replication. Galera code might cause a parallel replication transaction to be rolled back and retried, but this should be fine. Note also that in general for MySQL/MariaDB style replication, it is assumed (and it is the user's responsibility to ensure it) that replicated transactions will not conflict with anything done on the local node. This might also help to provide that Galera and parallel replication will not conflict. If Galera had been written with a proper general high-level API that could work with different storage-engines, then we could probably have used the "brute force transaction" mechanism to solve this problem in parallel replication also. But when I checked it, it seemed that it was hardcoded inside InnoDB with lots of Galera-specific code and data structures. If work were to start to fix this lack of proper API in Galera, then perhaps it would be interesting to work together to design a general API for enforcing a specific commit order... - Kristian.

1 0

[Maria-developers] [GSoC] optimize mysql-test runs - Preparing next steps
by Pablo Estrada 04 Jul '14

04 Jul '14

Hello Elena, I understand that the recall that we are achieving so far should still be improved; nonetheless, since we are already in the second half of the summer, I was wondering if we should spend some time designing the interfaces for the implementation of the project; such that I can start building them (even though the research phase can't be marked off as completed yet). Regards Pablo

2 1

Re: [Maria-developers] MDEV-5756 CMake option to build without thread pool
by Sergei Golubchik 03 Jul '14

03 Jul '14

Hi, Holyfoot! On Jul 01, holyfoot(a)askmonty.org wrote: > revno: 4270 > revision-id: holyfoot(a)askmonty.org-20140701104158-4gj5ztnjqlr1alwa > parent: holyfoot(a)askmonty.org-20140630193024-56wpcvwwy24kuzt4 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5756 > timestamp: Tue 2014-07-01 15:41:58 +0500 > message: > MDEV-5756 CMake option to build without thread pool. > Check if the threadpool is available on the system and set HAVE_POOL_OF_THREADS respectively. > === modified file 'sql/CMakeLists.txt' > --- a/sql/CMakeLists.txt 2014-05-30 11:24:25 +0000 > +++ b/sql/CMakeLists.txt 2014-07-01 10:41:58 +0000 > @@ -30,7 +30,15 @@ ${CMAKE_CURRENT_BINARY_DIR}/lex_hash.h > > SET_SOURCE_FILES_PROPERTIES(${GEN_SOURCES} PROPERTIES GENERATED 1) > > -ADD_DEFINITIONS(-DMYSQL_SERVER -DHAVE_EVENT_SCHEDULER -DHAVE_POOL_OF_THREADS) > +ADD_DEFINITIONS(-DMYSQL_SERVER -DHAVE_EVENT_SCHEDULER) > + > +IF (CMAKE_SYSTEM_NAME MATCHES "Linux" OR > + CMAKE_SYSTEM_NAME MATCHES "Windows" OR > + CMAKE_SYSTEM_NAME MATCHES "SunOS" OR > + HAVE_KQUEUE) > + ADD_DEFINITIONS(-DHAVE_POOL_OF_THREADS) > +ENDIF() > + > IF(SSL_DEFINES) > ADD_DEFINITIONS(${SSL_DEFINES}) > ENDIF() Did you test it on any of the platforms that that didn't support thread pool? If you would've done it, you'd noticed that your patch doesn't work. Because scheduler.h contains: #undef HAVE_POOL_OF_THREADS #if !defined(EMBEDDED_LIBRARY) && !defined(_AIX) #define HAVE_POOL_OF_THREADS 1 So, HAVE_POOL_OF_THREADS is only disabled in embedded and on AIX. The AIX part was added recently by Bar, as a quick fix to make MariaDB to compile on AIX. Your patch should remove the AIX test from scheduler.h. Actually, you should remove all #define and #undef HAVE_POOL_OF_THREADS from all files, only CMakeLists.txt should do it. Otherwise it's ok. I don't like that it explicitly checks for OS (Linux, Windows, SunOS) instead of checking for specific functions, libraries, or headers. But it's the same check that threadpool_unix.cc does, so it's ok. Regards, Sergei

1 0

[Maria-developers] [GSoC] Optimize mysql-test-runs - Analyzing some data
by Pablo Estrada 02 Jul '14

02 Jul '14

Hello Elena and all, I have spent a couple of days writing up some scripts to analyze the data, and try to find good candidates for metrics to improve recall, or some vices in the way the relevance is calculated for tests, such that they are not close enough in the priority queue to be spotted and run. Here's a writedown of some of the findings: *Of the failures that we miss, how many of them come from tests that have never failed before?* I added a small function in the simulator.py code, to keep diagnostics data of the test failures that we missed. I made a few charts regarding missed failures of tests that have never failed before (and, in the current model, we can't calculate a relevancy for them, because we have no data about them). In the next chart, we observe the percentage of missed failures that come from tests that had never failed before increases with time (as expected): By the way, this info comes from a standard run with 3000 learning cycles and 2000 prediction cycles. I feel that this is an important area. If we can figure a way to predict failures of tests that have not failed before, then we can predict subsequent failures of these tests, and thus improve recall significantly, specially over the long term. On the next section I analyze the option that I considered. *Is there a relationship between a test file being modified and the test failing any soon?* I figured that a good idea to catch failures of tests that had never failed before could be using the test file changes. If a test file is changed, I figured that the test itself would change behavior and might fail soon, and thus we could gather some data about that failure. I coded a small script that computes the number of test runs between a test file being edited, and the same test failing sometime in the future. The relationship, unfortunately is not very strong. I have attached the chart with the cummulative histogram of number of test runs that occurred between a test file being edited, and the next failure of that test. Unfortunately, the results were not too exciting. It turns out that tests don't tend to fail too soon after they have been changed (I guess this also makes sense). The following figures present the data (and a zoom in of the part of the graph we care most about), showing that failures are not very subsequent to test file editions. Not even 10% of failures occur in the first 100 runs. As we can see from the charts, there is not a very significant relationship between test file edition and subsequent failure. It seems this indicator won't be too useful. I am* still looking for ideas* on how to catch tests failing for the first time. They are all welcome : ) This is all the data that I collected so far. I am also trying to answer the following question: Failures of tests with non-zero relevance. How come we are missing those that we are missing? I have collected and charted some data, but I still haven't found anything that feels conclusive or revealing. I expect to write an email with that info soon as well, but I decided to send this out first. In conclusion: - I am thinking about how to predict first failures of a test. Any ideas? - I tested the relation between edition of a test result file and subsequent test failure. It was not a good predictor. Regards Pablo

1 0

[Maria-developers] [GSoC] Week Six!
by sriram patil 30 Jun '14

30 Jun '14

Hi all, Following is the blog link about my sixth week into GSoC. http://goo.gl/1dzgHe Suggestions and/or reviews are welcome! Thanks, Sriram

3 2

Re: [Maria-developers] [Commits] Rev 3855: Bumping server version. (10.0.13-galera) in lp:~maria-captains/maria/maria-10.0-galera
by Roberto Spadim 30 Jun '14

30 Jun '14

=] nice nirbhay 2014-06-30 10:33 GMT-03:00 Nirbhay Choubey <nirbhay(a)skysql.com>: > Hi, > > On Mon, Jun 30, 2014 at 9:13 AM, Roberto Spadim <roberto(a)spadim.com.br> > wrote: >> >> this will be a production ready version? > > > 10.0.12-galera (released over weekend) is the first 10.0-galera stable > release. This 'patch version bump' is > for preparation towards the next release. > > Best, > Nirbhay > >> >> >> 2014-06-30 10:03 GMT-03:00 Nirbhay Choubey <nirbhay(a)skysql.com>: >> > At lp:~maria-captains/maria/maria-10.0-galera >> > >> > ------------------------------------------------------------ >> > revno: 3855 >> > revision-id: nirbhay(a)skysql.com-20140630130329-w8kiirtf3blli7l1 >> > parent: nirbhay(a)skysql.com-20140626161112-q1iqtoyq5owutvvb >> > committer: Nirbhay Choubey <nirbhay(a)skysql.com> >> > branch nick: maria-10.0-galera >> > timestamp: Mon 2014-06-30 09:03:29 -0400 >> > message: >> > Bumping server version. (10.0.13-galera) >> > >> > === modified file 'VERSION' >> > --- a/VERSION 2014-06-19 17:12:38 +0000 >> > +++ b/VERSION 2014-06-30 13:03:29 +0000 >> > @@ -4,5 +4,5 @@ >> > # >> > MYSQL_VERSION_MAJOR=10 >> > MYSQL_VERSION_MINOR=0 >> > -MYSQL_VERSION_PATCH=12 >> > +MYSQL_VERSION_PATCH=13 >> > MYSQL_VERSION_EXTRA= >> > >> > >> > _______________________________________________ >> > commits mailing list >> > commits(a)mariadb.org >> > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >> >> >> >> -- >> Roberto Spadim >> SPAEmpresarial >> Eng. Automação e Controle >> _______________________________________________ >> commits mailing list >> commits(a)mariadb.org >> https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits > > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

Re: [Maria-developers] [Commits] Rev 3855: Bumping server version. (10.0.13-galera) in lp:~maria-captains/maria/maria-10.0-galera
by Roberto Spadim 30 Jun '14

30 Jun '14

this will be a production ready version? 2014-06-30 10:03 GMT-03:00 Nirbhay Choubey <nirbhay(a)skysql.com>: > At lp:~maria-captains/maria/maria-10.0-galera > > ------------------------------------------------------------ > revno: 3855 > revision-id: nirbhay(a)skysql.com-20140630130329-w8kiirtf3blli7l1 > parent: nirbhay(a)skysql.com-20140626161112-q1iqtoyq5owutvvb > committer: Nirbhay Choubey <nirbhay(a)skysql.com> > branch nick: maria-10.0-galera > timestamp: Mon 2014-06-30 09:03:29 -0400 > message: > Bumping server version. (10.0.13-galera) > > === modified file 'VERSION' > --- a/VERSION 2014-06-19 17:12:38 +0000 > +++ b/VERSION 2014-06-30 13:03:29 +0000 > @@ -4,5 +4,5 @@ > # > MYSQL_VERSION_MAJOR=10 > MYSQL_VERSION_MINOR=0 > -MYSQL_VERSION_PATCH=12 > +MYSQL_VERSION_PATCH=13 > MYSQL_VERSION_EXTRA= > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

[Maria-developers] Merge request: Cross-compile fixes (MySQL bug #61340)
by James Le Cuirot 30 Jun '14

30 Jun '14

Hello all, I posted a merge request to Launchpad back in April. A Launchpad glitch meant that it didn't get any attention but I was told on IRC to post about it here. The merge request: https://code.launchpad.net/~chewi/maria/mysql-fix-bug61340/+merge/217351 The commit diff: http://bazaar.launchpad.net/~chewi/maria/mysql-fix-bug61340/revision/4165 Given the nature of the fix, I can appreciate that it is difficult for other developers to test but I can tell you that I have just tried out a cross-built MariaDB under QEMU/proot and it seems to work. I have also seen posts from others stating that they made similar temporary adjustments to the CMake files with successful results. I am providing this contribution to the MariaDB Foundation under the MCA and also wish for it to be applied to MySQL. Regards, James

2 2

[Maria-developers] Connector Coding till 29th June
by Puneet Dewan 29 Jun '14

29 Jun '14

Hi Everyone, This week I fixed the null bitmap and types that were being sent to the server. So now I am able to successfully insert multiple values in the DB. Currently only Integer and String. This week 1.I will extend all the TypeParameter classes and ParameterWriter class so as to send the value in binary format. 2.Also try insert on different data types. 3. Write test cases Currently I am extending the PacketOutputStream.java class. Tomorrow I will have code review with Massimo. Regards Puneet.

1 0

[Maria-developers] TIME/DATETIME/TIMESTAMP data types with fractional precision
by Alexander Barkov 26 Jun '14

26 Jun '14

Hello, I propose to switch to MySQL-5.6 TIME/DATETIME/TIMESTAMP formats by default and deprecate MariaDB native temporal data types, together with pre-5.6 non-fractional types. The problem is that MariaDB native data types (and MySQL-5.5 non-fractional data types) do not work with row based replication in case of: TIME(N) -> TIME(M) There is no metadata telling about the column precision in the binary log file. It can only work when N=M. MySQL-5.6 uses separate type codes for the temporal data types: MYSQL_TYPE_TIME2 MYSQL_TYPE_DATETIME2 MYSQL_TYPE_TIMESTAMP2 and does put precision into metadata in the binary log. So it works for N != M. There is a JIRA task for this: MDEV-5377 Row-based replication of MariaDB temporal data types with FSP>0 into a different column type I'd propose to do it in 10.1.x, as 10.0.x is too late. Greetings.

2 1

Re: [Maria-developers] [Commits] Rev 4245: MDEV-6336: mysqldump --master-data does not work with GTID setups in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Sergey Vojtovich 25 Jun '14

25 Jun '14

Hi Kristian, MDEV-6344 looks fine. But I don't feel confident reviewing MDEV-6336. Regards, Sergey On Mon, Jun 16, 2014 at 11:38:04AM +0200, knielsen(a)knielsen-hq.org wrote: > At http://bazaar.launchpad.net/~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4245 > revision-id: knielsen(a)knielsen-hq.org-20140616093803-gub60pda6t77ch3k > parent: igor(a)askmonty.org-20140610223256-nqs0kkp7i45qzyvx > committer: knielsen(a)knielsen-hq.org > branch nick: work-10.0 > timestamp: Mon 2014-06-16 11:38:03 +0200 > message: > MDEV-6336: mysqldump --master-data does not work with GTID setups > MDEV-6344: mysqldump issues FLUSH TABLES, which gets written into binlog and replicated > > Add a --gtid option (for compatibility, the original behaviour is preserved > when --gtid is not used). > > With --gtid, --master-data and --dump-slave output the GTID position (the > old-style file/offset position is still output, but commented out). Also, a > CHANGE MASTER TO master_use_gtid=slave_pos is output to ensure a provisioned > slave is configured in GTID, as requested. > > Without --gtid, the GTID position is still output, if available, but commented > out. > > Also fix MDEV-6344, to avoid FLUSH TABLES getting into the binlog. Otherwise a > mysqldump on a slave server will silently inject a GTID which does not exist > on the master, which is highly undesirable. > === modified file 'client/client_priv.h' > --- a/client/client_priv.h 2013-09-14 01:09:36 +0000 > +++ b/client/client_priv.h 2014-06-16 09:38:03 +0000 > @@ -92,6 +92,7 @@ enum options_client > OPT_REPORT_PROGRESS, > OPT_SKIP_ANNOTATE_ROWS_EVENTS, > OPT_SSL_CRL, OPT_SSL_CRLPATH, > + OPT_USE_GTID, > OPT_MAX_CLIENT_OPTION /* should be always the last */ > }; > > > === modified file 'client/mysqldump.c' > --- a/client/mysqldump.c 2014-05-09 10:35:11 +0000 > +++ b/client/mysqldump.c 2014-06-16 09:38:03 +0000 > @@ -134,6 +134,7 @@ static ulong opt_compatible_mode= 0; > #define MYSQL_OPT_SLAVE_DATA_COMMENTED_SQL 2 > static uint opt_mysql_port= 0, opt_master_data; > static uint opt_slave_data; > +static uint opt_use_gtid; > static uint my_end_arg; > static char * opt_mysql_unix_port=0; > static int first_error=0; > @@ -346,6 +347,13 @@ static struct my_option my_long_options[ > {"force", 'f', "Continue even if we get an SQL error.", > &ignore_errors, &ignore_errors, 0, GET_BOOL, NO_ARG, > 0, 0, 0, 0, 0, 0}, > + {"gtid", OPT_USE_GTID, "Used together with --master-data=1 or --dump-slave=1." > + "When enabled, the output from those options will set the GTID position " > + "instead of the binlog file and offset; the file/offset will appear only as " > + "a comment. When disabled, the GTID position will still appear in the " > + "output, but only commented.", > + &opt_use_gtid, &opt_use_gtid, 0, GET_BOOL, NO_ARG, > + 0, 0, 0, 0, 0, 0}, > {"help", '?', "Display this help message and exit.", 0, 0, 0, GET_NO_ARG, > NO_ARG, 0, 0, 0, 0, 0, 0}, > {"hex-blob", OPT_HEXBLOB, "Dump binary strings (BINARY, " > @@ -1177,7 +1185,7 @@ check_consistent_binlog_pos(char *binlog > > if (mysql_query_with_error_report(mysql, &res, > "SHOW STATUS LIKE 'binlog_snapshot_%'")) > - return 1; > + return 0; > > found= 0; > while ((row= mysql_fetch_row(res))) > @@ -1200,6 +1208,90 @@ check_consistent_binlog_pos(char *binlog > return (found == 2); > } > > + > +/* > + Get the GTID position corresponding to a given old-style binlog position. > + This uses BINLOG_GTID_POS(). The advantage is that the GTID position can > + be obtained completely non-blocking in this way (without the need for > + FLUSH TABLES WITH READ LOCK), as the old-style position can be obtained > + with START TRANSACTION WITH CONSISTENT SNAPSHOT. > + > + Returns 0 if ok, non-zero if error. > +*/ > +static int > +get_binlog_gtid_pos(char *binlog_pos_file, char *binlog_pos_offset, > + char *out_gtid_pos) > +{ > + DYNAMIC_STRING query; > + MYSQL_RES *res; > + MYSQL_ROW row; > + int err; > + char file_buf[FN_REFLEN*2+1], offset_buf[LONGLONG_LEN*2+1]; > + size_t len_pos_file= strlen(binlog_pos_file); > + size_t len_pos_offset= strlen(binlog_pos_offset); > + > + if (len_pos_file >= FN_REFLEN || len_pos_offset > LONGLONG_LEN) > + return 0; > + mysql_real_escape_string(mysql, file_buf, binlog_pos_file, len_pos_file); > + mysql_real_escape_string(mysql, offset_buf, binlog_pos_offset, len_pos_offset); > + init_dynamic_string_checked(&query, "SELECT BINLOG_GTID_POS('", 256, 1024); > + dynstr_append_checked(&query, file_buf); > + dynstr_append_checked(&query, "', '"); > + dynstr_append_checked(&query, offset_buf); > + dynstr_append_checked(&query, "')"); > + > + err= mysql_query_with_error_report(mysql, &res, query.str); > + dynstr_free(&query); > + if (err) > + return err; > + > + err= 1; > + if ((row= mysql_fetch_row(res))) > + { > + strmake(out_gtid_pos, row[0], FN_REFLEN-1); > + err= 0; > + } > + mysql_free_result(res); > + > + return err; > +} > + > + > +/* > + Get the GTID position on a master or slave. > + The parameter MASTER is non-zero to get the position on a master > + (@@gtid_binlog_pos) or zero for a slave (@@gtid_slave_pos). > + > + This uses the @@gtid_binlog_pos or @@gtid_slave_pos, so requires FLUSH TABLES > + WITH READ LOCK or similar to be consistent. > + > + Returns 0 if ok, non-zero for error. > +*/ > +static int > +get_gtid_pos(char *out_gtid_pos, int master) > +{ > + MYSQL_RES *res; > + MYSQL_ROW row; > + int found; > + > + if (mysql_query_with_error_report(mysql, &res, > + (master ? > + "SELECT @@GLOBAL.gtid_binlog_pos" : > + "SELECT @@GLOBAL.gtid_slave_pos"))) > + return 1; > + > + found= 0; > + if ((row= mysql_fetch_row(res))) > + { > + strmake(out_gtid_pos, row[0], FN_REFLEN-1); > + found++; > + } > + mysql_free_result(res); > + > + return (found != 1); > +} > + > + > static char *my_case_str(const char *str, > uint str_len, > const char *token, > @@ -4799,12 +4891,14 @@ static int dump_selected_tables(char *db > } /* dump_selected_tables */ > > > -static int do_show_master_status(MYSQL *mysql_con, int consistent_binlog_pos) > +static int do_show_master_status(MYSQL *mysql_con, int consistent_binlog_pos, > + int have_mariadb_gtid, int use_gtid) > { > MYSQL_ROW row; > MYSQL_RES *UNINIT_VAR(master); > char binlog_pos_file[FN_REFLEN]; > char binlog_pos_offset[LONGLONG_LEN+1]; > + char gtid_pos[FN_REFLEN]; > char *file, *offset; > const char *comment_prefix= > (opt_master_data == MYSQL_OPT_MASTER_DATA_COMMENTED_SQL) ? "-- " : ""; > @@ -4815,6 +4909,9 @@ static int do_show_master_status(MYSQL * > return 1; > file= binlog_pos_file; > offset= binlog_pos_offset; > + if (have_mariadb_gtid && > + get_binlog_gtid_pos(binlog_pos_file, binlog_pos_offset, gtid_pos)) > + return 1; > } > else > { > @@ -4844,6 +4941,9 @@ static int do_show_master_status(MYSQL * > return 0; > } > } > + > + if (have_mariadb_gtid && get_gtid_pos(gtid_pos, 1)) > + return 1; > } > > /* SHOW MASTER STATUS reports file and position */ > @@ -4852,7 +4952,19 @@ static int do_show_master_status(MYSQL * > "recovery from\n--\n\n"); > fprintf(md_result_file, > "%sCHANGE MASTER TO MASTER_LOG_FILE='%s', MASTER_LOG_POS=%s;\n", > - comment_prefix, file, offset); > + (use_gtid ? "-- " : comment_prefix), file, offset); > + if (have_mariadb_gtid) > + { > + print_comment(md_result_file, 0, > + "\n--\n-- GTID to start replication from\n--\n\n"); > + if (use_gtid) > + fprintf(md_result_file, > + "%sCHANGE MASTER TO MASTER_USE_GTID=slave_pos;\n", > + comment_prefix); > + fprintf(md_result_file, > + "%sSET GLOBAL gtid_slave_pos='%s';\n", > + (!use_gtid ? "-- " : comment_prefix), gtid_pos); > + } > check_io(md_result_file); > > if (!consistent_binlog_pos) > @@ -4922,12 +5034,16 @@ static int add_slave_statements(void) > return(0); > } > > -static int do_show_slave_status(MYSQL *mysql_con) > +static int do_show_slave_status(MYSQL *mysql_con, int use_gtid, > + int have_mariadb_gtid) > { > MYSQL_RES *UNINIT_VAR(slave); > MYSQL_ROW row; > const char *comment_prefix= > (opt_slave_data == MYSQL_OPT_SLAVE_DATA_COMMENTED_SQL) ? "-- " : ""; > + const char *gtid_comment_prefix= (use_gtid ? comment_prefix : "-- "); > + const char *nogtid_comment_prefix= (!use_gtid ? comment_prefix : "-- "); > + int set_gtid_done= 0; > > if (mysql_query_with_error_report(mysql_con, &slave, > multi_source ? > @@ -4945,8 +5061,30 @@ static int do_show_slave_status(MYSQL *m > > while ((row= mysql_fetch_row(slave))) > { > + if (multi_source && !set_gtid_done) > + { > + char gtid_pos[FN_REFLEN]; > + if (have_mariadb_gtid && get_gtid_pos(gtid_pos, 0)) > + return 1; > + if (opt_comments) > + fprintf(md_result_file, "\n--\n-- Gtid position to start replication " > + "from\n--\n\n"); > + fprintf(md_result_file, "%sSET GLOBAL gtid_slave_pos='%s';\n", > + gtid_comment_prefix, gtid_pos); > + set_gtid_done= 1; > + } > if (row[9 + multi_source] && row[21 + multi_source]) > { > + if (use_gtid) > + { > + if (multi_source) > + fprintf(md_result_file, "%sCHANGE MASTER '%.80s' TO " > + "MASTER_USE_GTID=slave_pos;\n", gtid_comment_prefix, row[0]); > + else > + fprintf(md_result_file, "%sCHANGE MASTER TO " > + "MASTER_USE_GTID=slave_pos;\n", gtid_comment_prefix); > + } > + > /* SHOW MASTER STATUS reports file and position */ > if (opt_comments) > fprintf(md_result_file, > @@ -4955,9 +5093,9 @@ static int do_show_slave_status(MYSQL *m > > if (multi_source) > fprintf(md_result_file, "%sCHANGE MASTER '%.80s' TO ", > - comment_prefix, row[0]); > + nogtid_comment_prefix, row[0]); > else > - fprintf(md_result_file, "%sCHANGE MASTER TO ", comment_prefix); > + fprintf(md_result_file, "%sCHANGE MASTER TO ", nogtid_comment_prefix); > > if (opt_include_master_host_port) > { > @@ -5030,12 +5168,13 @@ static int do_flush_tables_read_lock(MYS > FLUSH TABLES is to lower the probability of a stage where both mysqldump > and most client connections are stalled. Of course, if a second long > update starts between the two FLUSHes, we have that bad stall. > + > + We use the LOCAL option, as we do not want the FLUSH TABLES replicated to > + other servers. > */ > return > - ( mysql_query_with_error_report(mysql_con, 0, > - ((opt_master_data != 0) ? > - "FLUSH /*!40101 LOCAL */ TABLES" : > - "FLUSH TABLES")) || > + ( mysql_query_with_error_report(mysql_con, 0, > + "FLUSH /*!40101 LOCAL */ TABLES") || > mysql_query_with_error_report(mysql_con, 0, > "FLUSH TABLES WITH READ LOCK") ); > } > @@ -5656,6 +5795,7 @@ int main(int argc, char **argv) > char bin_log_name[FN_REFLEN]; > int exit_code; > int consistent_binlog_pos= 0; > + int have_mariadb_gtid= 0; > MY_INIT(argv[0]); > > sf_leaking_memory=1; /* don't report memory leaks on early exits */ > @@ -5696,7 +5836,10 @@ int main(int argc, char **argv) > > /* Check if the server support multi source */ > if (mysql_get_server_version(mysql) >= 100000) > + { > multi_source= 2; > + have_mariadb_gtid= 1; > + } > > if (opt_slave_data && do_stop_slave_sql(mysql)) > goto err; > @@ -5743,9 +5886,11 @@ int main(int argc, char **argv) > /* Add 'STOP SLAVE to beginning of dump */ > if (opt_slave_apply && add_stop_slave()) > goto err; > - if (opt_master_data && do_show_master_status(mysql, consistent_binlog_pos)) > + if (opt_master_data && do_show_master_status(mysql, consistent_binlog_pos, > + have_mariadb_gtid, opt_use_gtid)) > goto err; > - if (opt_slave_data && do_show_slave_status(mysql)) > + if (opt_slave_data && do_show_slave_status(mysql, opt_use_gtid, > + have_mariadb_gtid)) > goto err; > if (opt_single_transaction && do_unlock_tables(mysql)) /* unlock but no commit! */ > goto err; > > === modified file 'mysql-test/r/rpl_mysqldump_slave.result' > --- a/mysql-test/r/rpl_mysqldump_slave.result 2014-02-26 14:28:07 +0000 > +++ b/mysql-test/r/rpl_mysqldump_slave.result 2014-06-16 09:38:03 +0000 > @@ -4,18 +4,59 @@ include/master-slave.inc > # New --dump-slave, --apply-slave-statements functionality > # > use test; > +-- SET GLOBAL gtid_slave_pos=''; > CHANGE MASTER '' TO MASTER_LOG_FILE='master-bin.000001', MASTER_LOG_POS=BINLOG_START; > STOP ALL SLAVES; > +-- SET GLOBAL gtid_slave_pos=''; > CHANGE MASTER '' TO MASTER_LOG_FILE='master-bin.000001', MASTER_LOG_POS=BINLOG_START; > START ALL SLAVES; > STOP ALL SLAVES; > +-- SET GLOBAL gtid_slave_pos=''; > CHANGE MASTER '' TO MASTER_HOST='127.0.0.1', MASTER_PORT=MASTER_MYPORT, MASTER_LOG_FILE='master-bin.000001', MASTER_LOG_POS=BINLOG_START; > START ALL SLAVES; > start slave; > Warnings: > Note 1254 Slave is already running > +-- SET GLOBAL gtid_slave_pos=''; > CHANGE MASTER '' TO MASTER_LOG_FILE='master-bin.000001', MASTER_LOG_POS=BINLOG_START; > start slave; > Warnings: > Note 1254 Slave is already running > +*** Test mysqldump --dump-slave GTID functionality. > +SET gtid_seq_no = 1000; > +CREATE TABLE t1 (a INT PRIMARY KEY); > +DROP TABLE t1; > +CREATE TABLE t2 (a INT PRIMARY KEY); > +DROP TABLE t2; > + > +1. --dump-slave=1 > + > +SET GLOBAL gtid_slave_pos='0-1-1001'; > +CHANGE MASTER '' TO MASTER_USE_GTID=slave_pos; > +-- CHANGE MASTER '' TO MASTER_LOG_FILE='master-bin.000001', MASTER_LOG_POS=BINLOG_START; > + > +2. --dump-slave=2 > + > +-- SET GLOBAL gtid_slave_pos='0-1-1001'; > +-- CHANGE MASTER '' TO MASTER_USE_GTID=slave_pos; > +-- CHANGE MASTER '' TO MASTER_LOG_FILE='master-bin.000001', MASTER_LOG_POS=BINLOG_START; > +*** Test mysqldump --master-data GTID functionality. > + > +1. --master-data=1 > + > +-- CHANGE MASTER TO MASTER_LOG_FILE='slave-bin.000001', MASTER_LOG_POS=BINLOG_START; > +CHANGE MASTER TO MASTER_USE_GTID=slave_pos; > +SET GLOBAL gtid_slave_pos='0-2-1003'; > + > +2. --master-data=2 > + > +-- CHANGE MASTER TO MASTER_LOG_FILE='slave-bin.000001', MASTER_LOG_POS=BINLOG_START; > +-- CHANGE MASTER TO MASTER_USE_GTID=slave_pos; > +-- SET GLOBAL gtid_slave_pos='0-2-1003'; > + > +3. --master-data --single-transaction > + > +-- CHANGE MASTER TO MASTER_LOG_FILE='slave-bin.000001', MASTER_LOG_POS=BINLOG_START; > +CHANGE MASTER TO MASTER_USE_GTID=slave_pos; > +SET GLOBAL gtid_slave_pos='0-2-1003'; > include/rpl_end.inc > > === modified file 'mysql-test/t/rpl_mysqldump_slave.test' > --- a/mysql-test/t/rpl_mysqldump_slave.test 2014-02-26 14:28:07 +0000 > +++ b/mysql-test/t/rpl_mysqldump_slave.test 2014-06-16 09:38:03 +0000 > @@ -36,4 +36,53 @@ start slave; > --exec $MYSQL_DUMP_SLAVE --compact --dump-slave no_such_db > start slave; > > + > +--echo *** Test mysqldump --dump-slave GTID functionality. > + > +--connection master > +SET gtid_seq_no = 1000; > +CREATE TABLE t1 (a INT PRIMARY KEY); > +DROP TABLE t1; > +--sync_slave_with_master > + > +--connection slave > +# Inject a local transaction on the slave to check that this is not considered > +# for --dump-slave. > +CREATE TABLE t2 (a INT PRIMARY KEY); > +DROP TABLE t2; > + > +--echo > +--echo 1. --dump-slave=1 > +--echo > +--replace_regex /MASTER_LOG_POS=[0-9]+/MASTER_LOG_POS=BINLOG_START/ > +--exec $MYSQL_DUMP_SLAVE --compact --dump-slave=1 --gtid test > + > +--echo > +--echo 2. --dump-slave=2 > +--echo > +--replace_regex /MASTER_LOG_POS=[0-9]+/MASTER_LOG_POS=BINLOG_START/ > +--exec $MYSQL_DUMP_SLAVE --compact --dump-slave=2 --gtid test > + > + > +--echo *** Test mysqldump --master-data GTID functionality. > +--echo > +--echo 1. --master-data=1 > +--echo > +--replace_regex /MASTER_LOG_POS=[0-9]+/MASTER_LOG_POS=BINLOG_START/ > +--exec $MYSQL_DUMP_SLAVE --compact --master-data=1 --gtid test > + > +--echo > +--echo 2. --master-data=2 > +--echo > +--replace_regex /MASTER_LOG_POS=[0-9]+/MASTER_LOG_POS=BINLOG_START/ > +--exec $MYSQL_DUMP_SLAVE --compact --master-data=2 --gtid test > + > +--echo > +--echo 3. --master-data --single-transaction > +--echo > +--replace_regex /MASTER_LOG_POS=[0-9]+/MASTER_LOG_POS=BINLOG_START/ > +--exec $MYSQL_DUMP_SLAVE --compact --master-data --single-transaction --gtid test > + > + > + > --source include/rpl_end.inc > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 1

Re: [Maria-developers] [Commits] Rev 4259: MDEV-6120: When slave stops with error, error message should indicate the failing GTID in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Sergey Vojtovich 25 Jun '14

25 Jun '14

Hi Kristian, looks good. One minor question below. On Thu, Jun 19, 2014 at 02:50:44PM +0200, knielsen(a)knielsen-hq.org wrote: > At http://bazaar.launchpad.net/~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4259 > revision-id: knielsen(a)knielsen-hq.org-20140619125043-acw2gx7rw90y3vqp > parent: svoj(a)mariadb.org-20140618141504-kqq6ws9gm93r2b2d > committer: Kristian Nielsen <knielsen(a)knielsen-hq.org> > branch nick: work-10.0 > timestamp: Thu 2014-06-19 14:50:43 +0200 > message: > MDEV-6120: When slave stops with error, error message should indicate the failing GTID > > If replication breaks in GTID mode, it is not trivial to determine the GTID of > the failing event group. This is a problem, as such GTID is needed eg. to > explicitly set @@gtid_slave_pos to skip to after that event group, or to > compare errors on different servers, etc. > > Fix by ensuring that relevant slave errors logged to the error log include the > GTID of the event group containing the problem event. ...skip... > === added file 'mysql-test/suite/rpl/t/rpl_gtid_errorlog.test' > --- a/mysql-test/suite/rpl/t/rpl_gtid_errorlog.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/rpl/t/rpl_gtid_errorlog.test 2014-06-19 12:50:43 +0000 > @@ -0,0 +1,94 @@ > +--source include/have_debug.inc > +--source include/master-slave.inc > + > +--echo *** Test MDEV-6120, output of current GTID when a replication error is logged to the errorlog *** > +--connection master > +CREATE TABLE t1(a INT PRIMARY KEY); > +--sync_slave_with_master > + > +--connection slave > +--source include/stop_slave.inc > +CHANGE MASTER TO master_use_gtid=slave_pos; > + > +--connection master > +INSERT INTO t1 VALUES (1); > +SET gtid_seq_no=100; > +INSERT INTO t1 VALUES (2); > +INSERT INTO t1 VALUES (3); > +INSERT INTO t1 VALUES (4); > +--save_master_pos > + > +--connection slave > +SET sql_log_bin=0; > +INSERT INTO t1 VALUES (2); > +SET sql_log_bin=1; > + > +START SLAVE; > +--let $slave_sql_errno=1062 > +--source include/wait_for_slave_sql_error.inc > + > +--source include/stop_slave.inc > +# Skip the problem event from the master. > +SET GLOBAL gtid_slave_pos= "0-1-100"; > +--source include/start_slave.inc > +--sync_with_master > + > +SELECT * FROM t1 ORDER BY a; > + > +--connection master > + > +SET @dbug_save= @@debug_dbug; > +SET debug_dbug= '+d,incident_database_resync_on_replace'; > +REPLACE INTO t1 VALUES (5); > +SET debug_dbug= @dbug_save; > +--save_master_pos > + > +--connection slave > +--let $slave_sql_errno=1590 > +--source include/wait_for_slave_sql_error.inc > +# ToDo no need to switch off GTID once MDEV-4937 is fixed > +--source include/stop_slave.inc > +CHANGE MASTER TO master_use_gtid=no; > +SET sql_slave_skip_counter=1; > +--source include/start_slave.inc > +--sync_with_master > +--source include/stop_slave.inc > +CHANGE MASTER TO master_use_gtid=slave_pos; > +--source include/start_slave.inc > + > +SELECT * FROM t1 ORDER BY a; > + > + > +# Check error log for correct messages. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.2.err; > +} > +let LOG_ERROR=$log_error_; > + > +perl; > + use strict; > + use warnings; > + my $log_error= $ENV{'LOG_ERROR'} or die "LOG_ERROR not set"; > + open(FILE, "<", "$log_error") or die("Unable to open $log_error: $!\n"); > + my $found_dup= 0; > + my $found_incident= 0; > + while (<FILE>) { > + $found_dup= 1 > + if /Slave SQL: Error 'Duplicate entry .* on query\. .*Query: '.*', Gtid 0-1-100, Internal MariaDB error code:/ > + or /Slave SQL: Could not execute Write_rows.*table test.t1; Duplicate entry.*, Gtid 0-1-100, Internal MariaDB error/ > + ; > + $found_incident= 1 > + if /Slave SQL: The incident LOST_EVENTS occured on the master\. Message: <none>, Internal MariaDB error code: 1590/; > + } > + print "Duplicate key found: $found_dup; incident found: $found_incident\n"; > + close(FILE); > +EOF > + > +--connection master > +DROP TABLE t1; > + > +--source include/rpl_end.inc Why didn't you like include/search_pattern_in_file.inc? It should perfectly able to match your patterns. ...skip... Regards, Sergey

2 1

[Maria-developers] Review of MDEV-5730: enhance security using special compilation options
by Kristian Nielsen 24 Jun '14

24 Jun '14

> commit feab48657528b9bb40fb035d51bee28d93710c1e > Author: Sergei Golubchik <serg(a)mariadb.org> > Date: Mon Jun 16 21:39:09 2014 +0200 > > MDEV-5730 enhance security using special compilation options > > -Wl,-z,relro,-z,now > -pie > -fstack-protector --param=ssp-buffer-size=4 > -D_FORTIFY_SOURCE=2 The patch is ok to push with a clarified comment as requested below. I have checked the compiler/linker options added and commented some on them below. > diff --git a/CMakeLists.txt b/CMakeLists.txt > index a24f000..5b0d348 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -201,6 +201,20 @@ IF (WITH_ASAN) > ENDIF() > ENDIF() > > +OPTION(SECURITY_HARDENED "Use security-enhancing compiler features (stack protector, relro, etc)" ON) So these are on by default. This means that this patch will cause some performance impact. So having them on by default is a compromise between performance and security. You should probably include in the commit comment a discussion of why the security viewpoint was prefered over the performance viewpoint in this case (or a comment in the cmake file if you prefer). The proper way to handle a patch like this would be to measure the performance impact, eg. for single-threaded performance with some various simple sysbench loads. However, it is quite hard to do such measurements, as you need a setup that has low enough noise in results to detect sub-percentage differences of performance. I have had some success by locking processes to a single core and using `perf` measurements of cycles spent (rather than wall-clock time), but even so the results can be tricky to interpret. So it may be too much work for a small patch like this. (And even if the performance improvement can be measured, how can we measure the security impact to weight them against one another?) My personal opinion is that I do not buy into this "security" mindset much. I do not think these options improve security significantly in practise for most users. So while the performance impact is probably also not very significant, why accept it at all (by default)? But the opposite viewpoint exists and is valid as well. So I think this patch is ok if you add a discussion of why it was chosen to do it this way. (Even if the reason is "Most distros do it" - at least then it is documented what the reason is). Also, the option should mention the possible performance impact. Suggestion: "Use compiler options (stack protector, relro, etc) that may slightly enhance security of the resulting binary, at the possible cost of some small performance impact." > +IF(SECURITY_HARDENED) > + # security-enhancing flags > + MY_CHECK_AND_SET_COMPILER_FLAG("-pie -fPIC") -fPIC can have a significant impact on performance on some platforms. For example on 32-bit x86, it requires extra instructions at the head of every function, and it makes one less register available for code generation on a CPU already starved for registers. On 64-bit x86, the impact is much smaller, as the architecture was enhanced to better support -fPIC. However, I seem to recall that the server is anyway built with -fPIC, for libmysqld or something like that? In which case this has no extra impact. > + MY_CHECK_AND_SET_COMPILER_FLAG("-Wl,-z,relro,-z,now") This option seems to cause all relocations to happen at load time, rather than lazily on-demand. The impact should be higher memory usage and some extra CPU time for these relocations, many of which may never be needed. This option will have the biggest effect on the client programs probably, as server startup is already fairly heavy. Probably even for client programs the added cost will be not very significant, as even the client programs are usually called to do significant processing (eg. connect to a database and so on). I guess these options would mostly hurt small utility programs for shell scripts, like `mv` or so. > + MY_CHECK_AND_SET_COMPILER_FLAG("-fstack-protector --param=ssp-buffer-size=4") This also adds extra code and memory loads to each function, so it will have some impact to performance. It is probably the most significant impact of all the added options, though without detailed analysis this is only guesswork, of course. > + # sometimes _FORTIFY_SOURCE is predefined > + INCLUDE(CheckSymbolExists) > + CHECK_SYMBOL_EXISTS(_FORTIFY_SOURCE "" HAVE_FORTIFY_SOURCE) > + IF(NOT HAVE_FORTIFY_SOURCE) > + ADD_DEFINITIONS(-D_FORTIFY_SOURCE=2) > + ENDIF() > +ENDIF() FORTIFY_SOURCE also adds some runtime overhead, but it is not very much, it is an extra argument to memcpy() and friends about destination buffer size. Probably the way or code is written, there will be few places where this option makes any difference, so the performance impact does not seem likely to be very significant. Hope this helps, - Kristian.

3 2

[Maria-developers] [GSoC] Optimize mysql-test-runs - Setback
by Pablo Estrada 24 Jun '14

24 Jun '14

Hello Sergei, Elena and all, Today while working on the script, I found and fixed an issue: There is some faulty code code in my script that is in charge of collecting the statistics about whether a test failure was caught or not (here <https://github.com/pabloem/Kokiri/blob/master/basic_simulator.py#L393>). I looked into fixing it, and then I could see another *problem*: The *recall numbers* that I had collected previously were *too high*. The actual recall numbers, once we consider the test failures that are *not caught*, are disappointingly lower. I won't show you results yet, since I want to make sure that the code has been fixed, and I have accurate tests first. This is all for now. The strategy that I was using is a lot less effective than it seemed initially. I will send out a more detailed report with results, my opinion on the weak points of the strategy, and ideas, including a roadmap to try to improve results. Regards. All feedback is welcome. Pablo

3 18

[Maria-developers] Doubt in bin logging for CREATE VIEW
by sriram patil 24 Jun '14

24 Jun '14

Hi All, I was checking the behavior of bin logging in various methods. I found out CREATE VIEW differs a little bit than others in bin logging. The query is written into bin log if query is executed successfully. If there is any error the query is not logged. This behavior is observed across all the commands. In case of CREATE VIEW (method: mysql_create_view file: sql/sql_view.cc), while the bin logging is skipped for most of the errors by jumping to "err" label, it is not skipped in following code snippet (line number: 616) res= mysql_register_view(thd, view, mode); /* View TABLE_SHARE must be removed from the table definition cache in order to make ALTER VIEW work properly. Otherwise, we would not be able to detect meta-data changes after ALTER VIEW. */ if (!res) tdc_remove_table(thd, TDC_RT_REMOVE_ALL, view->db, view->table_name, false); if (mysql_bin_log.is_open()) { // this is where the log is written } If an error occurs in mysql_register_view, e.g. ER_OUT_OF_RESOURCES, ER_TABLE_EXISTS_ERROR, etc. the query is still logged. Is there a specific reason why it is logged even though an error occurs? Or is it a bug? Thanks & Regards, Sriram

4 9

Re: [Maria-developers] [Commits] 7d386bb: MDEV-4549 [PATCH] Clean up code working with ACL tables
by Sergey Vojtovich 23 Jun '14

23 Jun '14

Hi Sergei, ok to push. A few minor questions inline. On Mon, Jun 16, 2014 at 09:34:21AM +0200, serg(a)mariadb.org wrote: > revision-id: 7d386bbce216745ba28df728f45a0b796fcced98 > parent(s): f61f36b386e8d0c6448297bb84c92e8d9b82be6a > committer: Sergei Golubchik > branch nick: maria > timestamp: 2014-06-15 17:12:57 +0200 > message: > > MDEV-4549 [PATCH] Clean up code working with ACL tables > > enum values to index different ACL tables, instead of hard-coded numbers > (even different in diffent functions). Also factor out TABLE_LIST > initialization into a separate function. > > --- > mysql-test/r/not_embedded_server.result | 4 +- > mysql-test/r/sp-destruct.result | 3 - > .../suite/rpl/r/rpl_tmp_table_and_DDL.result | 22 +- > mysql-test/t/not_embedded_server.test | 2 +- > mysql-test/t/sp-destruct.test | 1 - > sql/sql_acl.cc | 533 +++++++++------------ > 6 files changed, 235 insertions(+), 330 deletions(-) > > diff --git a/mysql-test/r/not_embedded_server.result b/mysql-test/r/not_embedded_server.result > index 2295276..6dda855 100644 > --- a/mysql-test/r/not_embedded_server.result > +++ b/mysql-test/r/not_embedded_server.result > @@ -1,4 +1,4 @@ > -call mtr.add_suppression("Can't open and lock privilege tables: Table 'host' was not locked with LOCK TABLES"); > +call mtr.add_suppression("Can't open and lock privilege tables: Table 'roles_mapping' was not locked with LOCK TABLES"); > SHOW VARIABLES like 'slave_skip_errors'; > Variable_name Value > slave_skip_errors OFF Why did it change (here and in other tests)? I didn't notice any table order change in the code. ...skip... > diff --git a/mysql-test/r/sp-destruct.result b/mysql-test/r/sp-destruct.result > index 172e40c..fe68229 100644 > --- a/mysql-test/r/sp-destruct.result > +++ b/mysql-test/r/sp-destruct.result > @@ -128,11 +128,8 @@ CREATE FUNCTION f1() RETURNS INT RETURN 1; > RENAME TABLE mysql.procs_priv TO procs_priv_backup; > FLUSH TABLE mysql.procs_priv; > DROP FUNCTION f1; > -ERROR 42S02: Table 'mysql.procs_priv' doesn't exist > SHOW WARNINGS; > Level Code Message > -Error 1146 Table 'mysql.procs_priv' doesn't exist > -Warning 1405 Failed to revoke all privileges to dropped routine > # Restore the procs_priv table > RENAME TABLE procs_priv_backup TO mysql.procs_priv; > FLUSH TABLE mysql.procs_priv; Hmm... why? ...skip... > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 5e8e0e7..442d512 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -380,7 +380,7 @@ class ACL_PROXY_USER :public ACL_ACCESS > MYSQL_PROXIES_PRIV_PROXIED_USER, > MYSQL_PROXIES_PRIV_WITH_GRANT, > MYSQL_PROXIES_PRIV_GRANTOR, > - MYSQL_PROXIES_PRIV_TIMESTAMP } old_acl_proxy_users; > + MYSQL_PROXIES_PRIV_TIMESTAMP } proxy_table_fields; > public: > ACL_PROXY_USER () {}; > > @@ -760,6 +760,99 @@ static int traverse_role_graph_down(ACL_USER_BASE *, void *, > int (*) (ACL_USER_BASE *, ACL_ROLE *, void *)); > > /* > + Enumeration of ACL/GRANT tables in the mysql database > +*/ > +enum enum_acl_tables > +{ > + USER_TABLE, > + DB_TABLE, > + TABLES_PRIV_TABLE, > + COLUMNS_PRIV_TABLE, > +#define FIRST_OPTIONAL_TABLE HOST_TABLE > + HOST_TABLE, > + PROCS_PRIV_TABLE, > + PROXIES_PRIV_TABLE, > + ROLES_MAPPING_TABLE, > + TABLES_MAX // <== always the last > +}; > +// bits for init_priv_tables > +static const int Table_user= 1 << USER_TABLE; > +static const int Table_db= 1 << DB_TABLE; > +static const int Table_tables_priv= 1 << TABLES_PRIV_TABLE; > +static const int Table_columns_priv= 1 << COLUMNS_PRIV_TABLE; > +static const int Table_host= 1 << HOST_TABLE; > +static const int Table_procs_priv= 1 << PROCS_PRIV_TABLE; > +static const int Table_proxies_priv= 1 << PROXIES_PRIV_TABLE; > +static const int Table_roles_mapping= 1 << ROLES_MAPPING_TABLE; > + > +const LEX_STRING acl_table_names[]= // matches enum_acl_tables > +{ > + { C_STRING_WITH_LEN("user") }, > + { C_STRING_WITH_LEN("db") }, > + { C_STRING_WITH_LEN("tables_priv") }, > + { C_STRING_WITH_LEN("columns_priv") }, > + { C_STRING_WITH_LEN("host") }, > + { C_STRING_WITH_LEN("procs_priv") }, > + { C_STRING_WITH_LEN("proxies_priv") }, > + { C_STRING_WITH_LEN("roles_mapping") } > +}; > + > +/** > + Initialize a TABLE_LIST array to a set of acl/grant tables. > + All tables will be opened with the same lock type, either read or write. > + > + @retval !0 a pointer to the first initialized element in the TABLE_LIST > + @retval 0 replication filters matched. Abort operation, but return OK (!) > +*/ > +TABLE_LIST *init_priv_tables(THD *thd, TABLE_LIST *tables, > + enum thr_lock_type lock_type, int tables_to_open) Nice clean-up, I assume there was a reason not rewrite open_grant_tables() instead. > +{ > + int prev= -1; > + bzero(tables, sizeof(TABLE_LIST) * TABLES_MAX); > + for (int cur=0, mask=1; mask <= tables_to_open; cur++, mask <<= 1) > + { > + if ((tables_to_open & mask) == 0) > + continue; > + tables[cur].init_one_table(C_STRING_WITH_LEN("mysql"), > + acl_table_names[cur].str, > + acl_table_names[cur].length, > + acl_table_names[cur].str, lock_type); > + tables[cur].open_type= OT_BASE_ONLY; > + if (lock_type >= TL_WRITE_ALLOW_WRITE) > + tables[cur].updating= 1; > + if (cur >= FIRST_OPTIONAL_TABLE) > + tables[cur].open_strategy= TABLE_LIST::OPEN_IF_EXISTS; > + if (prev != -1) > + tables[cur].next_local= tables[cur].next_global= & tables[prev]; > + prev= cur; > + } > + > +#ifdef HAVE_REPLICATION > + if (lock_type >= TL_WRITE_ALLOW_WRITE && thd->slave_thread && !thd->spcont) > + { > + /* > + GRANT and REVOKE are applied the slave in/exclusion rules as they are > + some kind of updates to the mysql.% tables. > + */ > + Rpl_filter *rpl_filter= thd->system_thread_info.rpl_sql_info->rpl_filter; > + if (rpl_filter->is_on() && !rpl_filter->tables_ok(0, tables)) > + return NULL; > + } > +#endif Was it a bug that mysql_grant_role() didn't have this code? open_grant_tables() does updating= 0 after tables_ok(). Is that needed? ...skip... > @@ -5783,8 +5796,9 @@ bool mysql_routine_grant(THD *thd, TABLE_LIST *table_list, bool is_proc, > } > } > > - if (replace_routine_table(thd, grant_name, tables[1].table, *Str, > - db_name, table_name, is_proc, rights, > + if (no_such_table(tables + PROCS_PRIV_TABLE) || > + replace_routine_table(thd, grant_name, tables[PROCS_PRIV_TABLE].table, > + *Str, db_name, table_name, is_proc, rights, > revoke_grant) != 0) > { > result= TRUE; Why no_such_table() here? ...skip... > @@ -6250,7 +6232,7 @@ bool mysql_grant(THD *thd, const char *db, List <LEX_USER> &list, > ulong db_rights= rights & DB_ACLS; > if (db_rights == rights) > { > - if (replace_db_table(tables[1].table, db, *Str, db_rights, > + if (replace_db_table(tables[DB_TABLE].table, db, *Str, db_rights, > revoke_grant)) > result= -1; > } Tabs here (and in a few hunks below). > @@ -6262,9 +6244,11 @@ bool mysql_grant(THD *thd, const char *db, List <LEX_USER> &list, > } > else if (is_proxy) > { > - if (replace_proxies_priv_table (thd, tables[1].table, Str, proxied_user, > - rights & GRANT_ACL ? TRUE : FALSE, > - revoke_grant)) > + if (no_such_table(tables + PROXIES_PRIV_TABLE) || > + replace_proxies_priv_table (thd, tables[PROXIES_PRIV_TABLE].table, > + Str, proxied_user, > + rights & GRANT_ACL ? TRUE : FALSE, > + revoke_grant)) > result= -1; > } > if (Str->is_role()) Why no_such_table() here? ...skip... Regards, Sergey

2 2

[Maria-developers] [GSoC] Week Five!
by sriram patil 23 Jun '14

23 Jun '14

Hi All, Following is the link to my blog post of fifth week into GSoC. http://goo.gl/x7dtcn Suggestions and/or review are welcome! Thanks, Sriram

1 0

[Maria-developers] Java Connector Coding till 21st June
by Puneet Dewan 21 Jun '14

21 Jun '14

Hi Everyone. Test case class(ServersidePreparedStatementTest.java) was written for checking the value of statement id. This week code review was done by Massimo and Georg. There pointers were really valuable as data was not being send in a proper format to the server for execute(com_stmt_execute) method : 1.NullBitMap was set incorrectly. 2.Creation of a new class extending PacketOuptutStream e.g integer of 4 bytes can be sent.Currently if an integer is sent it is sent as a byte. 3.Also I was expecting when we do a *prepare* for a statement like insert into XYZ(id) values(?) where id is of type Integer. Then server shall be returning integer as a type. This was a false assumption.It does not return the type. So now on the basis of value set for *id as example*,decide on the type to be sent from execute. So this week will fix the above things. Looking forward for suggestions. Regards Puneet.

1 0

Re: [Maria-developers] [Commits] 22f532c: MDEV-4260 Don't create frm files for temporary tables
by Sergey Vojtovich 20 Jun '14

20 Jun '14

Hi Sergei, ok to push. One note inline. On Fri, Jun 20, 2014 at 11:29:36AM +0200, serg(a)mariadb.org wrote: > revision-id: 22f532cfb2f40132e2f7b079a3aafb109af92c26 > parent(s): fb8818c1aff9ce67b74f965c019cad93addd811e > committer: Sergei Golubchik > branch nick: maria > timestamp: 2014-06-20 11:29:12 +0200 > message: > > MDEV-4260 Don't create frm files for temporary tables > > * Don't write frm for tmp tables > * pass frm image down to open_table_uncached, when possible > * don't use truncate-by-recreate for temp tables - cannot recreate > without frm, and delete_all_rows is faster anyway > * remove HTON_FLUSH_AFTER_RENAME (BDB-ism, unused for years) > > --- > mysql-test/r/temp_table_frm.result | 20 +++++++++++ > mysql-test/t/temp_table_frm.test | 13 ++++++++ > sql/handler.h | 2 +- > sql/sql_base.cc | 14 +++++++- > sql/sql_base.h | 3 +- > sql/sql_table.cc | 27 +++------------ > sql/sql_truncate.cc | 68 +------------------------------------- > sql/unireg.cc | 3 +- > 8 files changed, 56 insertions(+), 94 deletions(-) > ...skip... > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 0558066..7d69ccb 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -5561,7 +5563,17 @@ TABLE *open_table_uncached(THD *thd, handlerton *hton, > strend(saved_cache_key)+1, tmp_path); > share->db_plugin= ha_lock_engine(thd, hton); > > - if (open_table_def(thd, share, GTS_TABLE | GTS_USE_DISCOVERY)) > + /* > + Use the frm image, if possible, open the file otherwise. > + > + The image might be unavailable in ALTER TABLE, when the discovering > + engine took over the ownership (see TABLE::read_frm_image). > + */ > + int res= frm->str > + ? res= share->init_from_binary_frm_image(thd, false, frm->str, frm->length) Excessive "res="? Regards, Sergey

2 1

[Maria-developers] [GSoC] Week Four!
by sriram patil 19 Jun '14

19 Jun '14

Hi All, Sorry for a little delay in posting the blog link this week. Following is the blog link about my fourth week into GSoC. http://goo.gl/W54cvE Suggestions and/or reviews are welcome! Thanks, Sriram

4 4

Re: [Maria-developers] [Commits] Rev 4258: MDEV-6039 - WebScaleSQL patches in lp:maria/10.0
by Sergei Golubchik 18 Jun '14

18 Jun '14

Hi, Sergey! On Jun 18, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 4258 > revision-id: svoj(a)mariadb.org-20140618141504-kqq6ws9gm93r2b2d > parent: svoj(a)mariadb.org-20140618081243-jxs6yo9tx9cahdv6 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Wed 2014-06-18 18:15:04 +0400 > message: > MDEV-6039 - WebScaleSQL patches > > Stop spawning dummy threads on client library initialization > > Let's revert the fix for Bug#24507. To quote Monty from 2006: > > "After 1/2 a year, when all glibc versions are updated, we can delete > this code." > > Note: The upstream glibc bug was fixed in 2006. Ok to push Regards, Sergei

1 0

Re: [Maria-developers] [GSoC] self-tuning optimizer
by Sergei Golubchik 18 Jun '14

18 Jun '14

Hi, Anshu! Please, don't forget to cc: maria-developers@ when discussing development topics. On Jun 18, Anshu Avinash wrote: > > What I meant to say was we need N "linearly independent" equations to > solve for N variables and that might not be the case. Some of the > queries might not use any factor at all or use one factor more often. Yes, you're right, of course. > > Later we'll see what to do - perhaps we'll need to decrease the limit, > > or set it exactly to N (which means, the solution won't be "best > > fitting" anymore). But this will be not in this GSoC, not your project. > > I'm sure that in this project we won't have enough factors for memory > > usage to become an issue :) > > Yes, I'll start with a basic least squares fitting model without caring > much about the memory usage for now. Sounds good. Regards, Sergei

1 0

Re: [Maria-developers] [GSoC] self-tuning optimizer
by Sergei Golubchik 18 Jun '14

18 Jun '14

Hi, Anshu! On Jun 18, Anshu Avinash wrote: > Hi serg, > > Sorry that I had been out of touch for 1-2 days. I got caught up in some > other work. > > I have read through the comments you made on MDEV and I have just started > the implementation for the first approach, the approximate method. I have > pushed a prototype at: > https://github.com/igniting/server/commit/3bd4f5b6fd20d11575207ac36e8a3cc2c… > > Here is what I'm thinking, we add to the array of measurement data, after > each query and try to solve the system of equations because we don't know > when the system of equations would become solvable. Eh. To solve a system of linear equations with N unknowns, you need N equations. So, you wait until you get N equations and solve it. Few thoughts: 1. it might be that out of all factors that you have in your data structures (currently it's 2 + 2 * MAX_HA), only few will be actually used in queries (for example, only TIME_FOR_COMPARE and read_time for InnoDB). Then you only need two queries to solve that, not 2+2*MAX_HA. But it's an optimization, try to do the basic implementation without it first. 2. You normally store these data (for N queries) in the THD, right? It might be that the thread will disconnect before it'll do the required number of queries. To avoid throwing the useful data away, you can store one extra set of data globally and when some THD disconnects and it doesn't have enough data to solve the equation, you append this data to the global set and solve that as a whole. Again, this is an extension, only do it when the basic implementation works. > However this might be time consuming, trying to solve equations after > each query. Can we store the entire data (from all the queries) and > apply a best fitting model, instead of solving them exactly? This we > will do at disconnect. I don't have a good answer for that. It's a trade-off. Either you collect exactly N equations and solve them. Then you collect the next N equations (or you get another one and delete the first one) - and solve that. And so on. Or you collect many equations, more than N. And find a best fitting solution. This way you do less solving, but you'll need more memory. I think, for now you can do that - use a best fitting solution. We only have very few factors at the moment. Still, even now you'll need some limit to how many equations you'll collect. Later we'll see what to do - perhaps we'll need to decrease the limit, or set it exactly to N (which means, the solution won't be "best fitting" anymore). But this will be not in this GSoC, not your project. I'm sure that in this project we won't have enough factors for memory usage to become an issue :) > Should it be in ~THD() or somewhere else? Yes, ~THD is fine, just make sure you only do it for real connection THD's, where SELECT queries were issues and optimizer was run. Not for various internal THD's, like the one you create in cost_factors::init. For example, as you've noticed, I've mentioned in the last comment about using read_time() result and Nrows_estimated (the same applies to scan_time(), of course). So, when ha_read_time() is called, you need to remember (in your data set) the return value of read_time() and Nrows_estimated (that's the last argument of read_time()). This way, you only solve your equations if you have this read_time() and Nrows_estimated (or scan_time()) stored. If not - the optimizer was never run and you don't do anything. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 4257: Empty commit to check email setting in commit mails in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Roberto Spadim 18 Jun '14

18 Jun '14

working :) 2014-06-18 7:53 GMT-03:00 <knielsen(a)knielsen-hq.org>: > At http://bazaar.launchpad.net/~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4257 > revision-id: knielsen(a)knielsen-hq.org-20140618105333-44ez01iye29sljur > parent: knielsen(a)knielsen-hq.org-20140618090308-frd0tuo8rycgyi32 > committer: Kristian Nielsen <knielsen(a)knielsen-hq.org> > branch nick: work-10.0 > timestamp: Wed 2014-06-18 12:53:33 +0200 > message: > Empty commit to check email setting in commit mails > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits > -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

[Maria-developers] [MDEV-4903] Regarding range support for multi-column IN predicates
by Sergey Petrunia 18 Jun '14

18 Jun '14

Hi, So, MySQL 5.7 has got support for range optimization for multi-column IN predicates: http://optimize-this.blogspot.ru/2013/12/range-access-now-in-in-predicate-n… We've also got a user issue about it: https://mariadb.atlassian.net/browse/MDEV-4903. I'm looking at their IN handling. I don't see anything apparently wrong with it. They don't process multi-column NOT IN, but one can say that they just have a limited scope. Relevant code in opt_range.cc for single-column IN: case Item_func::IN_FUNC: { Item_func_in *func=(Item_func_in*) cond_func; if (inv) { // complicated code to handle NOT IN } else { tree= get_mm_parts(param, cond_func, field, Item_func::EQ_FUNC, func->arguments()[1], cmp_type); if (tree) { Item **arg, **end; for (arg= func->arguments()+2, end= arg+func->argument_count()-2; arg < end ; arg++) { tree= tree_or(param, tree, get_mm_parts(param, cond_func, field, Item_func::EQ_FUNC, *arg, cmp_type)); } } } mysql-5.7 extends this approach to handle multi-column case (by invoking get_mm_parts() for each compared value, then doing tree_and(), and then tree_or()). Should we just go ahead and backport it? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] problem with update and myisam
by Roberto Spadim 18 Jun '14

18 Jun '14

hi guys, could anyone help me with https://mariadb.atlassian.net/browse/MDEV-6355 thanks -- Roberto Spadim

2 5

Re: [Maria-developers] [Commits] 0eb94ca: MDEV-6137 better help for SET/ENUM sysvars
by Sergey Vojtovich 17 Jun '14

17 Jun '14

Hi Sergei, ok to push. A few minor questions inline. On Mon, Jun 16, 2014 at 06:51:53PM +0200, serg(a)mariadb.org wrote: > revision-id: 0eb94ca810500977eb0336ee30b19e9c0b769bca > parent(s): f61f36b386e8d0c6448297bb84c92e8d9b82be6a > committer: Sergei Golubchik > branch nick: maria > timestamp: 2014-06-16 18:43:26 +0200 > message: > > MDEV-6137 better help for SET/ENUM sysvars > > Auto-generate the allowed list of values for enum/set/flagset options > in --help output. But don't do that when the help text already has them. > > Also, remove lists of values from help strings of various options, where > they were simply listed without any additional information. > > --- > mysys/my_getopt.c | 81 +++++++++++++++++++++++++++------ > plugin/server_audit/server_audit.c | 2 +- > sql/log.cc | 3 +- > sql/mysqld.cc | 8 ++-- > sql/sql_plugin.cc | 2 +- > sql/sys_vars.cc | 90 ++++++++----------------------------- > storage/maria/ha_maria.cc | 14 ++---- > storage/myisam/ha_myisam.cc | 7 ++- > storage/xtradb/handler/ha_innodb.cc | 35 +++++++-------- > 9 files changed, 116 insertions(+), 126 deletions(-) Hmmm... no test case affected? > > diff --git a/mysys/my_getopt.c b/mysys/my_getopt.c > index 2e97417..3662757 100644 > --- a/mysys/my_getopt.c > +++ b/mysys/my_getopt.c > @@ -1300,6 +1300,48 @@ static uint print_name(const struct my_option *optp) > return s - optp->name; > } > > +/** prints option comment with indentation and wrapping. > + > + The comment column starts at startpos, and has width of width > + Current cursor position is curpos, returns new cursor position > + > +Note: can print one character beyond width! The above line, is it properly indented? > +*/ > +static uint print_comment(const char *comment, > + int curpos, int startpos, int width) Probably make width local to this function, it doesn't seem to be used by the caller anyway? > +{ > + const char *end= strend(comment); > + int endpos= startpos + width; > + > + for (; curpos < startpos; curpos++) > + putchar(' '); > + > + if (*comment == '.' || *comment == ',') > + { > + putchar(*comment); > + comment++; > + curpos++; > + } > + > + while (end - comment > endpos - curpos) > + { > + const char *line_end; > + for (line_end= comment + endpos - curpos; > + line_end > comment && *line_end != ' '; > + line_end--); > + for (; comment < line_end; comment++) > + putchar(*comment); I would probably replace it with fwrite(), but fine with this form too. > + while (*comment == ' ') > + comment++; /* skip the space, as a newline will take it's place now */ > + putchar('\n'); > + for (curpos= 0; curpos < startpos; curpos++) > + putchar(' '); > + } > + printf("%s", comment); > + return curpos + (end - comment); > +} > + > + > /* > function: my_print_options > > @@ -1309,12 +1351,12 @@ static uint print_name(const struct my_option *optp) > void my_print_help(const struct my_option *options) > { > uint col, name_space= 22, comment_space= 57; > - const char *line_end; > const struct my_option *optp; > DBUG_ENTER("my_print_help"); > > for (optp= options; optp->name; optp++) > { > + const char *typelib_help= 0; > if (!optp->comment) > continue; > if (optp->id && optp->id < 256) > @@ -1359,25 +1401,36 @@ void my_print_help(const struct my_option *options) > col= 0; > } > } > - for (; col < name_space; col++) > - putchar(' '); > if (optp->comment && *optp->comment) Cleanup suggestion: optp->comment can't be NULL here. > { > - const char *comment= optp->comment, *end= strend(comment); > + col= print_comment(optp->comment, col, name_space, comment_space); > > - while ((uint) (end - comment) > comment_space) > + switch (optp->var_type & GET_TYPE_MASK) { > + case GET_ENUM: > + typelib_help= ". One of: "; > + break; > + case GET_SET: > + typelib_help= ". Any combination of: "; > + break; > + case GET_FLAGSET: > + typelib_help= ". Takes a comma-separated list of option=value pairs, " > + "where value is on or off, and options are: "; > + break; > + } > + if (typelib_help && > + strstr(optp->comment, optp->typelib->type_names[0]) == NULL) Should we handle case when optp->typelib == NULL || optp->typelib->type_names[0] == NULL? Probably not. I believe this condition is fragile: comment string may mention type_names[0], but not list possible values. OTOH since this condition is for very limited data set it should cause problems only in rare cases. > { > - for (line_end= comment + comment_space; *line_end != ' '; line_end--); > - for (; comment != line_end; comment++) > - putchar(*comment); > - comment++; /* skip the space, as a newline will take it's place now */ > - putchar('\n'); > - for (col= 0; col < name_space; col++) > - putchar(' '); > + int i; > + col= print_comment(typelib_help, col, name_space, comment_space); > + col= print_comment(optp->typelib->type_names[0], col, name_space, comment_space); > + for (i= 1; i < optp->typelib->count; i++) > + { > + col= print_comment(", ", col, name_space, comment_space); > + col= print_comment(optp->typelib->type_names[i], col, name_space, comment_space); > + } > } > - printf("%s", comment); > + putchar('\n'); > } > - putchar('\n'); Does this mean if comment is empty there won't be a new line? > if ((optp->var_type & GET_TYPE_MASK) == GET_BOOL) > { > if (optp->def_value != 0) ...skip... > @@ -2785,7 +2745,7 @@ static Sys_var_mybool Sys_master_verify_checksum( > > /* These names must match RPL_SKIP_XXX #defines in slave.h. */ > static const char *replicate_events_marked_for_skip_names[]= { > - "replicate", "filter_on_slave", "filter_on_master", 0 > + "REPLICATE", "FILTER_ON_SLAVE", "FILTER_ON_MASTER", 0 > }; Does this mean there is a convention that values must be in upper case? ...skip... Regards, Sergey

2 1

[Maria-developers] WebScaleSQL: Actually fix all the array bounds warnings
by Sergey Vojtovich 16 Jun '14

16 Jun '14

Hi Sergei and Alexey, during our recent discussion re WebScaleSQL patches we agreed to review $subj: https://github.com/webscalesql/webscalesql-5.6/commit/094187c4a20e8d0c02f17… All these warnings come from strings/decimal.c. I was able to reproduce them using gcc-4.8.1, cmake -DBUILD_CONFIG=mysql_release (basically needs -Wall -O3). I simplified this function to the following: gcc -Wall -O3 -c dec.c -o dec.o static const int powers10[2 + 1]= { 1, 10, 100 }; int remove_leading_zeroes(int decimals, int var) { decimals%= 2; while (var < powers10[decimals--]) /* no-op */; return decimals; } The only allowed value for decimals is 1 and var must be > 0. Similar assertion is enforced by original function. So far I see nothing wrong with this code. I believe we should skip this patch, but still we should probably do something about these warning. What's your take? Regards, Sergey

2 2

[Maria-developers] MariaDB 10.0.12 release prep
by Daniel Bartholomew 14 Jun '14

14 Jun '14

I've begun prep work for the MariaDB 10.0.12 release. Draft Release Notes and Changelog are here: - https://mariadb.com/kb/en/mariadb-10012-release-notes/ - https://mariadb.com/kb/en/mariadb-10012-changelog/ Please add/modify/etc... as needed. Current plan is to announce the release on Mon, 16 Jun 2014. Thanks! -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

1 0

Re: [Maria-developers] Rev 3986: MDEV-5231: Per query variables from Percona Server
by Sergei Golubchik 14 Jun '14

14 Jun '14

Hi, Sanja! On May 22, sanja(a)askmonty.org wrote: > === added file 'mysql-test/suite/percona/percona_rpl_stm_per_query_variables_settings.test' > --- a/mysql-test/suite/percona/percona_rpl_stm_per_query_variables_settings.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/percona/percona_rpl_stm_per_query_variables_settings.test 2014-05-22 12:50:01 +0000 could you rename tests to have shorter names? rpl_set_statement looks good enough to me > @@ -0,0 +1,57 @@ > +--source include/master-slave.inc > +--source include/have_binlog_format_statement.inc > + > +--disable_warnings > +DROP TABLE IF EXISTS t1; > +DROP TABLE IF EXISTS t2; > +--enable_warnings > + > +call mtr.add_suppression("Unsafe statement written to the binary log*"); > +CREATE TABLE t1 (a bigint unsigned not null); > +CREATE TABLE t2 (a char(255) not null); okay, although I wouldn't bother creating *two* tables, I'd create one and hard-coded it into percona_rpl_set_statement_variable_test.inc (removing rpl_ssvt_table parameter) > + > +--echo > +--echo There are the following types of variables: > +--echo 1) variables that are NOT replicated correctly when using STATEMENT mode; > +--echo > + > +--let $rpl_ssvt_var_name=max_join_size > +--let $rpl_ssvt_var_value=2 > +--let $rpl_ssvt_table=t1 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > + > +--echo > +--echo 2) variables thar ARE replicated correctly > +--echo They must be replicated correctly with "SET STATEMENT" too. > +--echo > +--let $rpl_ssvt_var_name=auto_increment_increment > +--let $rpl_ssvt_var_value=10 > +--let $rpl_ssvt_table=t1 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > + > +--echo > +--echo 3) sql_mode which is replicated correctly exept NO_DIR_IN_CREATE value; > +--echo > +--let $rpl_ssvt_var_name=sql_mode > +--let $rpl_ssvt_var_value='ERROR_FOR_DIVISION_BY_ZERO' > +--let $rpl_ssvt_table=t2 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > +--let $rpl_ssvt_var_name=sql_mode > +--let $rpl_ssvt_var_value='NO_DIR_IN_CREATE' > +--let $rpl_ssvt_table=t2 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > + > +--echo > +--echo 4) variables that are not replicated at all: > +--echo default_storage_engine, storage_engine, max_heap_table_size > +--echo > +--let $rpl_ssvt_var_name=max_heap_table_size > +--let $rpl_ssvt_var_value=16384 > +--let $rpl_ssvt_table=t1 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > + > +connection master; > +DROP TABLE t1; > +DROP TABLE t2; > +sync_slave_with_master; > +source include/stop_slave.inc; > > === added file 'mysql-test/suite/percona/percona_statement_set.result' > --- a/mysql-test/suite/percona/percona_statement_set.result 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/percona/percona_statement_set.result 2014-05-22 12:50:01 +0000 reverse polish notation in percona someone loves. starwars too much seen he apparently. Please, rename to "set_statement.result" > @@ -0,0 +1,875 @@ > +'# SET STATEMENT ..... FOR .... TEST' > +DROP TABLE IF EXISTS t1; > +DROP FUNCTION IF EXISTS myProc; > +DROP PROCEDURE IF EXISTS p1; > +DROP PROCEDURE IF EXISTS p2; > +DROP PROCEDURE IF EXISTS p3; > +DROP PROCEDURE IF EXISTS p4; > +DROP PROCEDURE IF EXISTS p5; > +DROP TABLE IF EXISTS STATEMENT; > +'# Setup database' > +CREATE TABLE t1 (v1 INT, v2 INT); > +INSERT INTO t1 VALUES (1,2); > +INSERT INTO t1 VALUES (3,4); > +'' > +'#------------------ STATEMENT Test 1 -----------------------#' > +'# Initialize variables to known setting' > +SET SESSION sort_buffer_size=100000; > +'' > +'# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +Variable_name Value > +sort_buffer_size 100000 > +SET STATEMENT sort_buffer_size=150000 FOR SELECT * FROM t1; > +v1 v2 > +1 2 > +3 4 > +'' > +'# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +Variable_name Value > +sort_buffer_size 100000 > +'' > +'#------------------ STATEMENT Test 2 -----------------------#' > +'# Initialize variables to known setting' > +SET SESSION binlog_format=mixed; > +SET SESSION sort_buffer_size=100000; > +'# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +Variable_name Value > +sort_buffer_size 100000 > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +Variable_name Value > +binlog_format MIXED > +SET STATEMENT sort_buffer_size=150000, binlog_format=row > +FOR SELECT * FROM t1; > +v1 v2 > +1 2 > +3 4 > +'# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +Variable_name Value > +sort_buffer_size 100000 > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +Variable_name Value > +binlog_format MIXED > +'' > +'#------------------ STATEMENT Test 3 -----------------------#' > +'# set initial variable value, make prepared statement > +SET SESSION binlog_format=row; > +PREPARE stmt1 FROM 'SET STATEMENT binlog_format=row FOR SELECT * FROM t1'; > +'' > +'# Change variable setting' > +SET SESSION binlog_format=mixed; > +'' > +'# Pre-STATEMENT variable value' > +'' > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +Variable_name Value > +binlog_format MIXED > +'' > +EXECUTE stmt1; > +v1 v2 > +1 2 > +3 4 > +'' > +'# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +Variable_name Value > +binlog_format MIXED > +'' > +DEALLOCATE PREPARE stmt1; > +'#------------------ STATEMENT Test 4 -----------------------#' > +'# set initial variable value, make prepared statement > +SET SESSION myisam_sort_buffer_size=500000, myisam_repair_threads=1; > +'' > +'# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'myisam_sort_buffer_size'; > +Variable_name Value > +myisam_sort_buffer_size 500000 > +SHOW SESSION VARIABLES LIKE 'myisam_repair_threads'; > +Variable_name Value > +myisam_repair_threads 1 > +'' > +SET STATEMENT myisam_sort_buffer_size=800000, > +myisam_repair_threads=2 FOR OPTIMIZE TABLE t1; how comes that is "make prepared statement" (see comment above) ? > +Table Op Msg_type Msg_text > +test.t1 optimize status OK > +'' > +'# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'myisam_sort_buffer_size'; > +Variable_name Value > +myisam_sort_buffer_size 500000 > +SHOW SESSION VARIABLES LIKE 'myisam_repair_threads'; > +Variable_name Value > +myisam_repair_threads 1 > +'' ... > === added file 'mysql-test/suite/percona/percona_statement_set.test' > --- a/mysql-test/suite/percona/percona_statement_set.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/percona/percona_statement_set.test 2014-05-22 12:50:01 +0000 > @@ -0,0 +1,847 @@ > +--echo '# SET STATEMENT ..... FOR .... TEST' many of the tests below are pretty useless. they don't test that the value is actually set. See below: > +############################ STATEMENT_SET ############################# > +# # > +# Testing working functionality of SET STATEMENT # > +# # > +# # > +# There is important documentation within # > +# # > +# # > +# Author: Joe Lukas # > +# Creation: # > +# 2009-08-02 Implement this test as part of # > +# WL#681 Per query variable settings # > +# # > +######################################################################## > + > +--disable_warnings > +DROP TABLE IF EXISTS t1; > +DROP FUNCTION IF EXISTS myProc; > +DROP PROCEDURE IF EXISTS p1; > +DROP PROCEDURE IF EXISTS p2; > +DROP PROCEDURE IF EXISTS p3; > +DROP PROCEDURE IF EXISTS p4; > +DROP PROCEDURE IF EXISTS p5; > +DROP TABLE IF EXISTS STATEMENT; > +--enable_warnings > +#################################################################### > +#Set up current database > +#################################################################### > +--echo '# Setup database' > +CREATE TABLE t1 (v1 INT, v2 INT); > +INSERT INTO t1 VALUES (1,2); > +INSERT INTO t1 VALUES (3,4); > +--echo '' > +--echo '#------------------ STATEMENT Test 1 -----------------------#' > +#################################################################### > +# Checks with variable value type ulong # > +#################################################################### > +--echo '# Initialize variables to known setting' > +SET SESSION sort_buffer_size=100000; > +--echo '' > +--echo '# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +SET STATEMENT sort_buffer_size=150000 FOR SELECT * FROM t1; How do you know that SET STATEMENT did anything at all? > +--echo '' > +--echo '# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +--echo '' > +--echo '#------------------ STATEMENT Test 2 -----------------------#' > +#################################################################### > +# Checks for multiple set values inside STATEMENT ... FOR # > +#################################################################### > +--echo '# Initialize variables to known setting' > +SET SESSION binlog_format=mixed; > +SET SESSION sort_buffer_size=100000; > +--echo '# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +SET STATEMENT sort_buffer_size=150000, binlog_format=row > + FOR SELECT * FROM t1; > +--echo '# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > + > +--echo '' > +--echo '#------------------ STATEMENT Test 3 -----------------------#' > +#################################################################### > +# Check current variable value is stored in using stored # > +# statements. # > +#################################################################### > +--echo '# set initial variable value, make prepared statement > +SET SESSION binlog_format=row; > +PREPARE stmt1 FROM 'SET STATEMENT binlog_format=row FOR SELECT * FROM t1'; How will you know that SET STATEMENT did anything at all? (the same applies to most tests below, I'll stop repeating it) > +--echo '' > +--echo '# Change variable setting' > +SET SESSION binlog_format=mixed; > +--echo '' > +--echo '# Pre-STATEMENT variable value' > +--echo '' > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +--echo '' > +EXECUTE stmt1; > +--echo '' > +--echo '# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > + > +--echo '' > +DEALLOCATE PREPARE stmt1; ... > === modified file 'sql/share/errmsg-utf8.txt' > --- a/sql/share/errmsg-utf8.txt 2014-02-11 13:21:48 +0000 > +++ b/sql/share/errmsg-utf8.txt 2014-05-22 12:50:01 +0000 > @@ -7067,3 +7067,5 @@ ER_IT_IS_A_VIEW 42S02 > eng "'%-.192s' is a view" > ER_SLAVE_SKIP_NOT_IN_GTID > eng "When using GTID, @@sql_slave_skip_counter can not be used. Instead, setting @@gtid_slave_pos explicitly can be used to skip to after a given GTID position." > +ER_SET_STATEMENT_TABLES_IS_NOT_SUPPORTED > + eng "SET STATEMENT does not support using tables in the assignment variables expressions" why not? > > === modified file 'sql/sql_parse.cc' > --- a/sql/sql_parse.cc 2014-03-29 10:33:25 +0000 > +++ b/sql/sql_parse.cc 2014-05-22 12:50:01 +0000 > @@ -2430,6 +2432,33 @@ mysql_execute_command(THD *thd) > thd->get_binlog_format(&orig_binlog_format, > &orig_current_stmt_binlog_format); > > + if (lex->set_statement && !lex->var_list.is_empty()) Hmm? No separate list of variables, but a common lex->var_list? Then the following won't work: SET STATEMENT A=1 SET B=2 I remember in the WL and in GSoC we agreed that for SET STATEMENT A=2 SET A=2 (same variable in SET STATEMENT and SET), it's acceptable to restore the original value of A, basically ignoring SET. But I doubt that the same logic applies when SET STATEMENT and SET modify *different* variables. > + { > + MEM_ROOT *save_mem_root; > + /* > + We should be sure that variables backup and assignment made on > + runtime memory > + */ > + save_mem_root= thd->mem_root; > + thd->mem_root= thd->get_execution_mem_root(); Why? > + per_query_variables_backup= copy_system_variables(&thd->variables); > + if (!per_query_variables_backup || > + (res= sql_set_variables(thd, &lex->var_list))) > + { > + thd->mem_root= save_mem_root; > + /* > + We encountered some sort of error, but no message was sent. > + Send something semi-generic here since we don't know which > + assignment in the list caused the error. > + */ > + if (!thd->is_error()) > + my_error(ER_WRONG_ARGUMENTS, MYF(0), "SET"); > + goto error; > + } > + thd->mem_root= save_mem_root; > + } > + > /* > Force statement logging for DDL commands to allow us to update > privilege, system or statistic tables directly without the updates > @@ -5144,6 +5179,30 @@ finish: > > lex->unit.cleanup(); > > + if (lex->set_statement && !lex->var_list.is_empty()) { > + List_iterator_fast<set_var_base> it(thd->lex->var_list); > + set_var *var; > + > + free_system_variables(&thd->variables); > + thd->variables= *per_query_variables_backup; > + /* per_query_variables_backup was in a mem_root */ > + per_query_variables_backup= NULL; > + /* > + When variables are restored after "SET STATEMENT ... FOR ..." statement > + execution an update callback must be invoked for the system variables > + to save special logic if it is. set_var_base class does not contain > + refference to variable as it is just an interface class. But only > + system variables are allowed to be used in "SET STATEMENT ... FOR ..." > + statement, so cast from set_var_base* to set_var* can be used here. > + */ > + while ((var=(set_var *)it++)) > + { > + var->var->stmt_update(thd); > + } I don't think it'll work. Compare set_var::update() and set_var::stmt_update(). The former calls ::session_update(), while the latter does not. Normally, session_update() simply updates the value, performs the assignment. And it is, indeed, not needed here. but in some custom classes session_update() is more complex, and it's not safe to omit it. Please 1) see all non-trivial implementations of sys_var::session_update() 2) write SET SESSION tests for these variables 3) if they'll all work - please explain why 4) if they won't - I'm afraid, this whole approach needs to be redone, then > + > + thd->lex->set_statement= false; > + } > + > if (! thd->in_sub_stmt) > { > if (thd->killed != NOT_KILLED) > > === modified file 'sql/sql_plugin.cc' > --- a/sql/sql_plugin.cc 2014-03-26 21:25:38 +0000 > +++ b/sql/sql_plugin.cc 2014-05-22 12:50:01 +0000 > @@ -4088,3 +4088,48 @@ static void restore_ptr_backup(uint n, s > (backup++)->restore(); > } > > + > +/** > + Create deep copy of system_variables instance. > +*/ > +struct system_variables * > +copy_system_variables(const struct system_variables *src) What is it doing in sql_plugin.cc? > +{ > + struct system_variables *dst; > + > + DBUG_ASSERT(src); > + > + dst= (struct system_variables *) > + sql_alloc(sizeof(struct system_variables)); > + if (!dst) > + return NULL; > + *dst= *src; > + > + if (dst->dynamic_variables_ptr) > + { > + if (!(dst->dynamic_variables_ptr= > + (char *)my_malloc(dst->dynamic_variables_size, MYF(MY_WME | MY_FAE)))) > + return NULL; > + memcpy(dst->dynamic_variables_ptr, > + src->dynamic_variables_ptr, > + src->dynamic_variables_size); > + } > + > + mysql_mutex_lock(&LOCK_plugin); > + dst->table_plugin= > + intern_plugin_lock(NULL, src->table_plugin); I'm surprised that plugin locking actually works in here, (I believe it does, otherwise you'd noticed it). What about SET STATEMENT default_engine=MEMORY ? What about SET STATEMENT default_engine=MyISAM (where MyISAM is already default)? please, try both in debug build. > + mysql_mutex_unlock(&LOCK_plugin); > + > + return dst; > +} > + > +void free_system_variables(struct system_variables *v) > +{ > + DBUG_ASSERT(v); > + > + mysql_mutex_lock(&LOCK_plugin); > + intern_plugin_unlock(NULL, v->table_plugin); > + mysql_mutex_unlock(&LOCK_plugin); > + > + my_free(v->dynamic_variables_ptr); > +} > > === modified file 'sql/sql_yacc.yy' > --- a/sql/sql_yacc.yy 2014-03-26 21:25:38 +0000 > +++ b/sql/sql_yacc.yy 2014-05-22 12:50:01 +0000 > @@ -813,8 +813,10 @@ static void sp_create_assignment_lex(THD > lex->one_shot_set= 0; > lex->autocommit= 0; > /* get_ptr() is only correct with no lookahead. */ > - DBUG_ASSERT(no_lookahead); > + if (no_lookahead) > lex->sphead->m_tmp_query= lip->get_ptr(); > + else > + lex->sphead->m_tmp_query= lip->get_tok_end(); Why? > /* Inherit from outer lex. */ > lex->option_type= old_lex->option_type; > } > @@ -955,10 +957,10 @@ bool my_yyoverflow(short **a, YYSTYPE ** > %parse-param { THD *thd } > %lex-param { THD *thd } > /* > - Currently there are 163 shift/reduce conflicts. > + Currently there are 164 shift/reduce conflicts. > We should not introduce new conflicts any more. > */ > -%expect 163 > +%expect 164 Why? > > /* > Comments for TOKENS. > @@ -14427,8 +14431,44 @@ set: > } > start_option_value_list > {} > + | SET STATEMENT_SYM > + { > + LEX *lex= Lex; > + mysql_init_select(lex); > + lex->sql_command= SQLCOM_SET_OPTION; > + /* Don't clear var_list in the case of recursive statement */ > + if (!lex->set_statement) > + lex->var_list.empty(); When var_list is not empty here (and set_statement not set)? In other words, when this statement will actually have any effect? > + lex->one_shot_set= 0; > + lex->autocommit= 0; > + lex->set_statement= true; > + sp_head *sp= lex->sphead; > + if (sp && !sp->is_invoked()) > + { > + sp->m_param_begin= ((yychar == YYEMPTY) ? YYLIP->get_ptr() : YYLIP->get_tok_start()); > + sp->m_param_end= ((yychar == YYEMPTY) ? YYLIP->get_ptr() : YYLIP->get_tok_end()); Why? > + } > + } > + set_stmt_option_value_following_option_type_list Eh? What does "set stmt option value following option type list" mean? Please, rename to set_stmt_option_value_list or even set_stmt_list > + { > + if (Lex->query_tables) > + { > + my_error(ER_SET_STATEMENT_TABLES_IS_NOT_SUPPORTED, MYF(0)); > + MYSQL_YYABORT; > + } > + } > + FOR_SYM statement > + {} > ; > > +set_stmt_option_value_following_option_type_list: > + /* > + Only system variables can be used here. If this condition is changed > + please check careful code under lex->option_type == OPT_STATEMENT > + condition on wrong type casts. > + */ > + option_value_following_option_type > + | set_stmt_option_value_following_option_type_list ',' option_value_following_option_type > > // Start of option value list > start_option_value_list: Regards, Sergei

1 0

Re: [Maria-developers] [GSoC] self-tuning optimizer
by Sergei Golubchik 13 Jun '14

13 Jun '14

Hi, Anshu! On Jun 13, Anshu Avinash wrote: > > So I have refactored Cost_factors::init() into Cost_factors::init() and > Cost_factors::re_init(). The diff is at: > https://github.com/igniting/server/commit/41e360309c067d072d788495e4e948054… Looks great. I've added one comment there. > I had started writing a test for checking whether if loading a new > storage engine works or not. I loaded 'Example' storage engine, but > was unable to test the current t1 table on it, as t1 contains a > primary key. Also I need to write test cases for 'time_for_compare'. > Any hints on that? Hm. Yes. Example engine, is pretty useless for you, as you cannot create tables in it. You can use InnoDB plugin. But create a separate test file for that. For example (I'll use the short name below, don't commit it): a.test: ============================================== if (!$HA_INNODB_SO) { skip Need InnoDB plugin; } install plugin innodb soname 'ha_innodb'; ... do your tests here uninstall plugin innodb; ============================================== you will also need a.opt file: ============================================== --ignore-builtin-innodb --loose-innodb ============================================== As for time_for_compare... Here's a universal approach, you can do it yourself. This time_for_compare is 5 by default, right? Change it to be, say, 500, and run existing tests (start from the main suite, and there from optimizer tests, e.g. from select.test). If some test will fail - here's your query, extract it, simplify, perhaps, and put in your test file. That's exactly what I just did :) And here's a simple test: create table t1 (a blob, index(a(20))); create table t2 (a blob, index(a(20))); insert into t1 values ('one'),('two'),('three'),('four'),('five'); insert into t2 values ('one'),('two'),('three'),('four'),('five'); insert into t2 values ('one'),('two'),('three'),('four'),('five'); drop table t1, t2; Perhaps you can get it down to one table - I didn't try. > Also I think I can start with the next part, i.e. collect statistics. We > can have a structure like: > > struct Cost_factor_stats > { > char *cost_factor_name; > double cost_factor_value; > double total_queries; total_queries should be ulonglong, not double. > double total_query_time; > double total_query_time_squared; > }; > > We will be updating these structures after every query per THD and write > the data into a persistent table on disconnect. Is this workflow correct? Almost. You need a Cost_factor_stats structure per cost factor (and per engine where appropriate). You don't need the name there. You have these structures per THD and globally. Basically, instead of your current double time_for_compare_val; you will have Cost_factor_stats time_for_compare_val; hmm, perhaps better to name the structure simply "Cost_factor": Cost_factor time_for_compare; And you put them all in one structure: struct Engine_cost_factors { Cost_factor read_time; Cost_factor scan_time; }; struct Global_cost_factors { Cost_factor time_for_compare; Cost_factor time_for_compare_rowid; }; class Cost_factors { Global_cost_factors global; Engine_cost_factors engine[MAX_HA]; double time_for_compare() const { return global.read_time.value; } double time_for_compare_rowid() const; double read_time(const handler*) const; double scan_time(const handler*) const; }; Now, you stick this structure in the THD, and also declare one instance globally. On connect you initialize THD::Cost_factor values from the global Cost_factor object. On disconnect you add THD::Cost_factor values back to the global Cost_factor object. There's no writing to disk so far, we'll think about it later (you don't want to write to disk on *every* disconnect). > Few more questions: > 1.What will we do if the query had used more than one cost factor? > 2.How would joins be handled? Yes, a query will certainly use more than one cost factor. In fact, this is something we might want to experiment with. I see two approaches - exact and approximate. Approximate one is described in MDEV. Say, you've counted 500 index reads, 20,000 sequential row reads (table scan), some 10,000,000 comparisons, etc. And you know how long the query took. For another query you've counted different numbers, and so on. Basically, you'll have a system of linear equations that one can solve to know how the time was split. The problem with this approach - the server is doing much more than purely index reads, table scans, and comparisons. So this unaccounted time will be split between cost factors. I don't know if it's a problem. The exact approach - we only measure times spent on the actual operation. Basically, you measure how long *each row read* took, how long *each index read* took, etc. This will show no unaccounted time. But it's more expensive, it might be too expensive to be practically useful - I don't know that. These two approaches need different data structures, the first one needs more memory, because you cannot start aggregating before you solve the system of equations. Basically, if you have four constant (time_for_compare, time_for_compare_rowid, read_time, scan_time, only one engine), you'll need four equations, that is, per THD you need to store four measurements: struct measurement { ulong time_for_compare; ulong time_for_compare_rowid; struct per_engine { ulong scan_time; ulong read_time; } per_engine[MAX_HA]; double total_time; }; and in THD you need struct measurement data[4]; Also, note that I'm writing per_engine[MAX_HA], but it's a simplification, good enough for a prototype. But in the final code we cannot allocate that much memory per THD, as most of the time we'll have only few engines, not MAX_HA. Regards, Sergei

1 0

[Maria-developers] WebScaleSQL: Adjust linking flags to not embed some libs
by Sergey Vojtovich 13 Jun '14

13 Jun '14

Hi Sergei, during our recent discussion re WebScaleSQL patches we agreed to accept $subj: https://github.com/webscalesql/webscalesql-5.6/commit/361e3b83145402945b737… I studied our binaries and it looks like we embed zlib and ssl libraries only if WITH_ZLIB=bundled/WITH_SSL=bundled. E.g. DEB/RPM packages use system libs and do not embed zlib/ssl, whereas tarball uses bundled zlib/yassl and embeds them. Not embedding yassl into the client library will make the latter almost useless. Same probably applies to zlib even though it is normally offered by OS-es. Said the above I believe our build scripts behave properly and we should skip this patch. What's your take? Regards, Sergey

2 1

[Maria-developers] Java Connector Coding till 12th June
by Puneet Dewan 13 Jun '14

13 Jun '14

Hi Everyone, This week, I was able to send the com_stmt_execute command from the *Driver * in the way it is accepted by the MariaDB server So, Now I am able to insert a row in the database using com_stmt_execute command. But from the console when I try to see the values I see *NULL* I used *Wireshark* to capture the packets,but I can see the value(Varchar /Int/..etc) is passed in the packet which wireshark captures. I am investigating the code but could not find anything till now. In Wireshark it shows* [Expert Info (warn/undecoded) :FIXME: Excute Dissector Incomplete]* Now I think I am not sending the values in a correct format.Will look and try to resolve it. Regards Puneet. On Thu, Jun 5, 2014 at 10:42 PM, Puneet Dewan <puneetd30(a)gmail.com> wrote: > Hi Everyone, > > The work done till 5th June: > 1.I started studying the connector immediately after my semester exams got > over(5th May). > 2. Understood the relationship between different classes. > 3.Read the MySql documentation for text protocol and map the commands from > documentation onto code.This thing gave a clear idea about how things are > moving from client to server. > 4.Setup the code on Ubuntu using eclipse and tested against database. > Setup the Team branch in launchpad. > 5.Debug the test classes to see how different classes are interlinked and > understood that MySQLProtocol.java is the heart of our JDBC driver. > 6.Then I started reading Client Server Protocol documentation i.e > ServerSide prepared statement and try to see how and where I have to code > it. > 7.Then I saw that there is a class MySqlServerSidePreparedStatement.java, > Initially I coded the prepareStatement() ,close(),and execute() in that > class seperately. > But after (Code review) discussion with Massimo, we donot need to use that > class. > Instead use MySqlPreparedStatement.java to do the same work.So made > changes accordingly. > 8.I had pushed some of the code which I have written to launchpad and > discussed with Massimo last week. > 9.Last week I was struck with Wireshark on windows and ubuntu both. > I was unable to see the packets going to database and seeing the response. > 10.On Monday the problem was resolved as I was using the wrong interface.I > should have used loopback.io(127.0.0.1) instead on ubuntu(thanks to > Georg). > Unfortunately the thing does not work on windows.Tried a lot many times. > 11.Now I am working on the execute method of client server protocol as I > am getting exception while using that method.Wireshark is playing a great > role in how the packets are flowing. > > From now I will do weekly blogging and keep everyone updated. > > Thanks & Regards > Puneet. >

1 0

[Maria-developers] Allow to turn off max_connect_errors
by Honza Horak 13 Jun '14

13 Jun '14

Hi guys, there are apparently some tools out there [1], that check if the server is up quite often, while not closing the connection properly. It eventually ends in 'many connection errors', because max_connect_errors is always limited now. I understand that this way of checking may be wrong, but there may be scenarios where we do not want to check for `max_connect_errors` at all. So, would it be acceptable for mariadb to change behaviour of max_connect_errors option, so that it accepts also 0 as a possible value, which would mean 'do not check connect errors at all'? I'm bringing the idea here first, but will submit a report and possibly patch if it does not seem to be undesired behaviour. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1104957 TIA and regards, Honza

4 4

Re: [Maria-developers] [Commits] Rev 4018: MDEV-4784 - merge test cases from 5.6 in lp:maria/10.0
by Sergei Golubchik 12 Jun '14

12 Jun '14

Hi, Sergey! > === added file 'mysql-test/include/have_daemon_example_plugin.inc' > --- a/mysql-test/include/have_daemon_example_plugin.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/have_daemon_example_plugin.inc 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,16 @@ > +disable_query_log; > +# > +# Check if server has support for loading plugins > +# > +if (`SELECT @@have_dynamic_loading != 'YES'`) { > + --skip daemon example plugin requires dynamic loading > +} > + > +# > +# Check if the variable DAEMONEXAMPLE is set > +# > +if (!$LIBDAEMON_EXAMPLE_SO) { > + --skip daemon_example plugin requires the environment variable \$DAEMONEXAMPLE to be set (normally done by mtr) Eh? It was in your first patch. I thought it's an additional patch, but it looks like it includes the first one? Okay, I will copy all my comments from the first patch here to have a complete review in one email. So, here I said ************************ it's always done by mtr, a better error message should be "No libdaemon_example.so was found" or simply "Need daemon_example plugin" and, btw, the first if() is redundant, if no dynamic loading was enabled no dynamic plugins will be built whatsoever. > +} > + > +enable_query_log; > > === added file 'mysql-test/include/ib_logfile_size_check.inc' > --- a/mysql-test/include/ib_logfile_size_check.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/ib_logfile_size_check.inc 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,14 @@ I'm not a great fan of obscure include files that are only used once in one single test. Perhaps it's better to inline it? > +perl; > + my $dir = $ENV{'MYSQLD_DATADIR'}; > + my $size; > + opendir(DIR, $dir) or die $!; > + while (my $file = readdir(DIR)) > + { > + > + next unless ($file =~ m/\ib_logfile.$/); > + $size = -s "$dir/$file"; > + print "The size of the ib_logfile(0/1): $size \n"; > + } > + close(DIR); > + exit(0) > +EOF > > === added file 'mysql-test/include/vardir_size_check.inc' > --- a/mysql-test/include/vardir_size_check.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/vardir_size_check.inc 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,26 @@ Same here. Although it's worse than with the previous file :) ib_logfile_size_check.inc is only used in innodb_log_file_size_functionality.test. This file - too. But in 5.6 this one is also used in query_cache_type_functionality, table_open_cache_functionality, and table_definition_cache_functionality, sort_buffer_size_functionality, query_cache_size_functionality. I suppose somebody just copied innodb_log_file_size_functionality over and edited it. I don't see why would anyone really want to check the vardir size in all these other tests. So, yes, better to inline this file too, to avoid silly mistakes like that. > +--echo Check the size of the vardir > +--echo The output size is in unit blocks > +perl; > +#!/usr/bin/perl > +use warnings; > +use strict; > +use File::Find; > +my $dir = $ENV{'MYSQLTEST_VARDIR'}; > +my $size = 0; > +find( sub { $size += -f $_ ? -s _ : 0 }, $dir ); > + > +if ( $size < 1717987000 ) > + { print "TRUE", "\n";} > +else > + { print "FALSE $size", "\n";} > + > + > +## Expected size for the VAR dir being changed for the PB2 runs > + > +##if ( $size =~ /^496[\d]{5}$/ ) > +## { > +## { print "TRUE", "\n";} > +## } > +##else > +## { print "FALSE $size", "\n";} > +EOF > > === modified file 'mysql-test/mysql-test-run.pl' > --- a/mysql-test/mysql-test-run.pl 2014-02-26 14:28:07 +0000 > +++ b/mysql-test/mysql-test-run.pl 2014-04-14 09:20:24 +0000 > @@ -5350,6 +5350,14 @@ sub mysqld_arguments ($$$) { > mtr_add_arg($args, "--gdb"); > } > > + # Enable the debug sync facility, set default wait timeout. > + # Facility stays disabled if timeout value is zero. > + mtr_add_arg($args, "--loose-debug-sync-timeout=%s", > + $opt_debug_sync_timeout) unless $opt_user_args; Why was it moved? > + > + # Options specified in .opt files should be added last so they can > + # override defaults above. > + > my $found_skip_core= 0; > foreach my $arg ( @$extra_opts ) > { > === added file 'mysql-test/suite/archive/archive_no_symlink-master.opt' > --- a/mysql-test/suite/archive/archive_no_symlink-master.opt 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/archive/archive_no_symlink-master.opt 2014-04-14 09:20:24 +0000 > @@ -0,0 +1 @@ > +--skip-symbolic-links just fyi: in MariaDB the file name can simply be archive_no_symlink.opt -master and -slave are only needed for replication tests to have different options for the master and the slave. > > === added file 'mysql-test/t/bug12969156.test' > --- a/mysql-test/t/bug12969156.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/bug12969156.test 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,18 @@ > +--source include/have_daemon_example_plugin.inc > +--source include/not_embedded.inc > +# TODO: the windows emulation of pthreads doesn't behave predictably > +--source include/not_windows.inc > + > +--echo # > +--echo # Bug #12969156 : SEGMENTATION FAULT ON UNINSTALLING > +--echo # DAEMON_EXAMPLE PLUGIN > +--echo # > + > +let $counter= 0; > +while ($counter < 10) > +{ > +--replace_result $LIBDAEMON_EXAMPLE_SO DAEMONEXAMPLE > + eval INSTALL PLUGIN daemon_example SONAME '$LIBDAEMON_EXAMPLE_SO'; in our own tests (not copied from 5.6) better use install soname 'libdaemon_example'; it's shorter, doesn't need --eval and doesn't need --replace_result > + UNINSTALL PLUGIN daemon_example; > + inc $counter; > +} > > === added file 'mysql-test/t/innodb_log_file_size_functionality.test' > --- a/mysql-test/t/innodb_log_file_size_functionality.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/innodb_log_file_size_functionality.test 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,124 @@ This should be renamed to sys_vars.innodb_log_file_size_func test > +############################################################################### > +# # > +# Variable Name: innodb_log_file_size # > +# Scope: Global # > +# Access Type: Static # > +# Data Type: numeric # > +# # > +# # > +# Creation Date: 2012-08-20 # > +# Author : Tanjot Singh Uppal # > +# # > +# # > +# Description:Test Cases of Static System Variable innodb_log_file_size # > +# that checks the behavior of this variable in the following ways # > +# * Value Check # > +# * Scope Check # > +# * Functionality Check # > +# * Accessability Check # > +# # > +# This test does not perform the crash recovery on this variable # > +# For crash recovery test on default change please run the ibtest # > +############################################################################### Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 4018: MDEV-4784 - merge test cases from 5.6 in lp:maria/10.0
by Sergei Golubchik 12 Jun '14

12 Jun '14

Hi, Sergey! > === added file 'mysql-test/include/have_daemon_example_plugin.inc' > --- a/mysql-test/include/have_daemon_example_plugin.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/have_daemon_example_plugin.inc 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,16 @@ > +disable_query_log; > +# > +# Check if server has support for loading plugins > +# > +if (`SELECT @@have_dynamic_loading != 'YES'`) { > + --skip daemon example plugin requires dynamic loading > +} > + > +# > +# Check if the variable DAEMONEXAMPLE is set > +# > +if (!$LIBDAEMON_EXAMPLE_SO) { > + --skip daemon_example plugin requires the environment variable \$DAEMONEXAMPLE to be set (normally done by mtr) it's always done by mtr (not just "normally"). So a better error message should be "No libdaemon_example.so was found" or simply "Need daemon_example plugin" and, btw, the first if() is redundant, if no dynamic loading was enabled no dynamic plugins will be built whatsoever. > +} > + > +enable_query_log; > > === added file 'mysql-test/include/ib_logfile_size_check.inc' > --- a/mysql-test/include/ib_logfile_size_check.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/ib_logfile_size_check.inc 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,14 @@ I'm not a great fan of obscure include files that are only used once in one single test. Perhaps it's better to inline it? > +perl; > + my $dir = $ENV{'MYSQLD_DATADIR'}; > + my $size; > + opendir(DIR, $dir) or die $!; > + while (my $file = readdir(DIR)) > + { > + > + next unless ($file =~ m/\ib_logfile.$/); > + $size = -s "$dir/$file"; > + print "The size of the ib_logfile(0/1): $size \n"; > + } > + close(DIR); > + exit(0) > +EOF > > === added file 'mysql-test/include/vardir_size_check.inc' > --- a/mysql-test/include/vardir_size_check.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/vardir_size_check.inc 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,26 @@ Same here. Although it's worse than with the previous file :) ib_logfile_size_check.inc is only used in innodb_log_file_size_functionality.test. This file - too. But in 5.6 this one is also used in query_cache_type_functionality, table_open_cache_functionality, and table_definition_cache_functionality, sort_buffer_size_functionality, query_cache_size_functionality. I suppose somebody just copied innodb_log_file_size_functionality over and edited it without much thinking. I don't see why would anyone really want to check the vardir size in all these other tests. So, yes, better to inline this file too, to avoid silly mistakes like that. > +--echo Check the size of the vardir > +--echo The output size is in unit blocks > +perl; > +#!/usr/bin/perl > +use warnings; > +use strict; > +use File::Find; > +my $dir = $ENV{'MYSQLTEST_VARDIR'}; > +my $size = 0; > +find( sub { $size += -f $_ ? -s _ : 0 }, $dir ); > + > +if ( $size < 1717987000 ) > + { print "TRUE", "\n";} > +else > + { print "FALSE $size", "\n";} > + > + > +## Expected size for the VAR dir being changed for the PB2 runs > + > +##if ( $size =~ /^496[\d]{5}$/ ) > +## { > +## { print "TRUE", "\n";} > +## } > +##else > +## { print "FALSE $size", "\n";} > +EOF > > === added file 'mysql-test/suite/archive/archive_no_symlink-master.opt' > --- a/mysql-test/suite/archive/archive_no_symlink-master.opt 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/archive/archive_no_symlink-master.opt 2014-03-11 09:28:16 +0000 > @@ -0,0 +1 @@ just fyi: in MariaDB the file name can simply be archive_no_symlink.opt -master and -slave are only needed for replication tests to have different options for the master and the slave. > +--skip-symbolic-links > > === added file 'mysql-test/t/bug12969156.test' > --- a/mysql-test/t/bug12969156.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/bug12969156.test 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,18 @@ > +--source include/have_daemon_example_plugin.inc > +--source include/not_embedded.inc > +# TODO: the windows emulation of pthreads doesn't behave predictably > +--source include/not_windows.inc > + > +--echo # > +--echo # Bug #12969156 : SEGMENTATION FAULT ON UNINSTALLING > +--echo # DAEMON_EXAMPLE PLUGIN > +--echo # > + > +let $counter= 0; > +while ($counter < 10) > +{ > +--replace_result $LIBDAEMON_EXAMPLE_SO DAEMONEXAMPLE > + eval INSTALL PLUGIN daemon_example SONAME '$LIBDAEMON_EXAMPLE_SO'; in our own tests (not copied from 5.6) better use install soname 'libdaemon_example'; it's shorter, doesn't need --eval and doesn't need --replace_result > + UNINSTALL PLUGIN daemon_example; > + inc $counter; > +} > > === added file 'mysql-test/t/innodb_log_file_size_functionality.test' > --- a/mysql-test/t/innodb_log_file_size_functionality.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/innodb_log_file_size_functionality.test 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,124 @@ This should be renamed to sys_vars.innodb_log_file_size_func test > +############################################################################### > +# # > +# Variable Name: innodb_log_file_size # > +# Scope: Global # > +# Access Type: Static # > +# Data Type: numeric # > +# # > +# # > +# Creation Date: 2012-08-20 # > +# Author : Tanjot Singh Uppal # > +# # > +# # > +# Description:Test Cases of Static System Variable innodb_log_file_size # > +# that checks the behavior of this variable in the following ways # > +# * Value Check # > +# * Scope Check # > +# * Functionality Check # > +# * Accessability Check # > +# # > +# This test does not perform the crash recovery on this variable # > +# For crash recovery test on default change please run the ibtest # > +############################################################################### > + > +-- source include/have_innodb.inc > +-- source include/not_embedded.inc > +-- source include/have_innodb_16k.inc > + > +echo '#________________________VAR_09_INNODB_LOG_FILE_SIZE__________________#' > +echo '##' > +--echo '#---------------------WL6372_VAR_9_01----------------------#' > +#################################################################### > +# Checking default value # > +#################################################################### > +SELECT COUNT(@@GLOBAL.innodb_log_file_size); > +--echo 1 Expected > + > +SELECT @@GLOBAL.innodb_log_file_size; > +--echo 5242880 Expected > + > + > + > +--echo '#---------------------WL6372_VAR_9_02----------------------#' > +#################################################################### > +# Checking Value can be set - Static # > +#################################################################### > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SET @@local.innodb_log_file_size=1; > +--echo Expected error 'Read only variable' > + > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SET @@session.innodb_log_file_size=1; > +--echo Expected error 'Read only variable' > + > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SET @@GLOBAL.innodb_log_file_size=1; > +--echo Expected error 'Read only variable' > + > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SET @@GLOBAL.innodb_log_file_size=DEFAULT; > +--echo Expected error 'Read only variable' > + > +SELECT COUNT(@@GLOBAL.innodb_log_file_size); > +--echo 1 Expected > + > + > + > +--echo '#---------------------WL6372_VAR_9_03----------------------#' > +################################################################# > +# Check if the value in GLOBAL Table matches value in variable # > +################################################################# > +SELECT @@GLOBAL.innodb_log_file_size = VARIABLE_VALUE > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='innodb_log_file_size'; > +--echo 1 Expected > + > +SELECT COUNT(@@GLOBAL.innodb_log_file_size); > +--echo 1 Expected > + > +SELECT COUNT(VARIABLE_VALUE) > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='innodb_log_file_size'; > +--echo 1 Expected > + > + > + > +--echo '#---------------------WL6372_VAR_9_04----------------------#' > +################################################################################ > +# Checking Variable Scope # > +################################################################################ > +SELECT @@innodb_log_file_size = @@GLOBAL.innodb_log_file_size; > +--echo 1 Expected > + > +--Error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SELECT COUNT(@@local.innodb_log_file_size); > +--echo Expected error 'Variable is a GLOBAL variable' > + > +--Error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SELECT COUNT(@@SESSION.innodb_log_file_size); > +--echo Expected error 'Variable is a GLOBAL variable' > + > +SELECT COUNT(@@GLOBAL.innodb_log_file_size); > +--echo 1 Expected > + > +--Error ER_BAD_FIELD_ERROR > +SELECT innodb_log_file_size = @@SESSION.innodb_log_file_size; > +--echo Expected error 'Unknown column innodb_log_file_size in field list' > + > + > + > +--echo '#---------------------WL6372_VAR_9_05----------------------#' > +############################################################################### > +# Checking the /Var directory size # > +############################################################################### > +-- source include/vardir_size_check.inc > +--echo TRUE Expected > + > +--echo '#---------------------WL6372_VAR_9_06----------------------#' > +################################################################################# > +# Checking the size of ib_logfile # > +################################################################################# > +let MYSQLD_DATADIR=`SELECT @@datadir`; > +--source include/ib_logfile_size_check.inc > +--echo 5242880 expected > + > Regards, Sergei

1 0

[Maria-developers] PERFORMANCE_SCHEMA to be disabled in 10.0 (10.0.12)
by Colin Charles 12 Jun '14

12 Jun '14

Hi all, Recently there was chat about how PERFORMANCE_SCHEMA was enabled by mistake and it should be disabled in the 10.0 series. I'm curious - how many of you are using PERFORMANCE_SCHEMA? Is it a problem to turn it on, if you use it? I'm referring to: https://mariadb.atlassian.net/browse/MDEV-6249 Looking from the webscalesql list, there are examples of performance degradation: https://groups.google.com/d/msg/webscalesql/B-4bPIb8mHI/eUNsl0lSfbMJ https://groups.google.com/d/msg/webscalesql/B-4bPIb8mHI/-YZMMCuPw3QJ quote: "Our perf testing agrees with your assessment (we see about a 5%-6% perf hit when it's included and on, and a 2%-3% hit when it's included but off)" Please discuss this, either here or on MDEV-6249 Thanks cheers, -colin -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

6 10

[Maria-developers] Extendet FederatedX storage engine to support updating Sphinx RT indexes
by Markus Lidel 12 Jun '14

12 Jun '14

Hello, the Sphinx Search engine has RT indexes, which could be updated with a subset of SQL commands, refered to as SphinxQL: http://sphinxsearch.com/docs/2.1.1/sphinxql-reference.html Because the FederatedX storage engine uses some commands, which Sphinx does not know, here is a patch, wich extends the FederatedX storage engine to support this subset of commands. Now its possible to update Sphinx RT indexes directly from MariaDB. To create a connection to the RT index in the Sphinx default RT-index: index rt { type = rt path = @CONFDIR@/data/rt rt_field = title rt_field = content rt_attr_string = title rt_attr_string = content rt_attr_uint = gid } you just have to call: CREATE TABLE `rt` ( `id` BIGINT UNSIGNED NOT NULL, `title` TEXT, `content` TEXT, `gid` INT UNSIGNED ) ENGINE=FEDERATED CONNECTION='sphinxql://root@localhost:9306/rt'; It would be great, if the attached patch could be integrated into MariaDB. If you have any suggestions, please feel free to contact me. Best regards, Markus Lidel

4 8

Re: [Maria-developers] [GSoC] Accepted student ready to work : )
by Pablo Estrada 10 Jun '14

10 Jun '14

Hello all, I'm still interested on a call, (I would make sure to write down everything as part of an email to keep it clearly organized here), but as Elena suggested, I'll try to look at Sergei's bzr repo first, and all the documentation. In any case, I already have a decent idea of how the data is arranged. I can see how it is difficult to trace test results back to code changes directly, but it might be possible. I still haven't figured out how to recognize what code changes pertain to the same 'history', or 'line' (I think that you were not considering this in your experiments... and it might turn out to not be relevant at all, but I still want to consider it, to try to make a more complete model). So I have just two questions that should be easy to answer by email: 1. In your graphs, exactly, what do you mean by Precision? What about Recall? And cutoff? 2. I looked at your launchpad profile but I can't seem to find the repo with your tests (if you even pushed it to lp at all). Is it available? That's it for today : ) Until next time. Pablo On Thu, Apr 24, 2014 at 12:28 AM, Sergei Golubchik <serg(a)skysql.com> wrote: > On April 23, 2014 10:07:23 AM CEST, Pablo Estrada wrote: > > >If it's not too inconvenient, do you think we could set up a Google > >Hangout > >or a Skype call on Monday to go over a few questions quickly? > > > Definitely. This week I'm on vacations and without a laptop, back home on > Sunday. So, Monday is fine. But please if you have specific questions - ask > them now in emails. I'll prepare answers for Monday, but it'll take time to > remember everything. That is, if you will wait with questions till the > call, I might not be able to remember the answer on the spot. > Regards, > Sergei >

4 17

Re: [Maria-developers] Rev 4234: Fixed bug mdev-6071
by Sergei Golubchik 10 Jun '14

10 Jun '14

Hi, Igor! On Jun 09, Igor Babaev wrote: > revno: 4234 > revision-id: igor(a)askmonty.org-20140610050716-0x8xaquf3itvofjc > parent: bar(a)mariadb.org-20140606062952-x2jwt4mgrgr6tpw6 > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-10.0-bugs > timestamp: Mon 2014-06-09 22:07:16 -0700 > message: > Fixed bug mdev-6071. > The method JOIN_CACHE::init may fail (return 1) if some conditions on the > used join buffer is not satisfied. For example it fails if join_buffer_size > is greater than join_buffer_space_limit. The conditions should be checked > when running the EXPLAIN command for the query. That's why the method > JOIN_CACHE::init has to be called for EXPLAIN commands as well. It's an improvement over the old behavior - as EXPLAIN shows correctly whether BKA is used. But I think there's still not enough diagnostics. In the bug report Sergey wrote that " I think it's a big gotcha that @join_buffer_size must be lower than @@join_buffer_space_limit when optimizer_space_limit optimization is off. " So, I'd believe users will be very confused that in some cases BKA is suddenly turned off, when they expect it to be used. Of course, without this bugfix they would've not even noticed that, which is even worse. So, this bugfix is good. But it would be nice to have some kind of an explanation of why BKA is not used. At the very least in the manual. Better - here as a warning, like if (for_explain) push_warning("BKA was not used because ..."; goto fail; > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2014-06-05 22:07:27 +0000 > +++ b/sql/sql_select.cc 2014-06-10 05:07:16 +0000 > @@ -10541,7 +10541,7 @@ > if (cache_level == 1) > prev_cache= 0; > if ((tab->cache= new JOIN_CACHE_BNL(join, tab, prev_cache)) && > - ((options & SELECT_DESCRIBE) || !tab->cache->init())) > + !tab->cache->init(MY_TEST(options & SELECT_DESCRIBE))) MY_TEST here (and in other similar places) is redundant, you could've simply written !tab->cache->init(options & SELECT_DESCRIBE)) > { > tab->icp_other_tables_ok= FALSE; > return (2 - MY_TEST(!prev_cache)); and while it wasn't part of your bug fix, I cannot help but noticing that MY_TEST() is redundant here too. !prev_cache is boolean, it's always 0 or 1. Regards, Sergei

1 0

Re: [Maria-developers] Google Code for Encryption of MariaDB
by Elmar Eperiesi-Beck 09 Jun '14

09 Jun '14

Hi, 1. what about an NDA between us, to give me the chance to have a pre-view. I will make sure, that your architecture and development fits into our work. Everything we can think of in an early stage does not make lot of work, but… 2. like. 3. We will post our concept next week into the MariaDB Developers community. 4. … or 1. Regards Elmar Am 09.06.2014 um 11:14 schrieb Jonas Oreland : > Hi there, > > 1) repo: as I said in my mail, the code is currently not yet published, > but the guy doing this said that he expects it to be with in a couple of weeks or so. > before this, i don't think you can see it. > > once it's published, I think you can find it at https://code.google.com, > but I'll check and let you know. > > 2) This is my personal email address, so I prefer if we keep the conversation > @maria-developers where I'm subscribed with work email address, if this is ok > by you ? > > 3) concept paper: sure > > 4) we don't really have a concept paper, but the code is complete and being rolled out. > I can answer questions about it, but I think it's easiest once it has been published. > > /Jonas > > > > On Mon, Jun 9, 2014 at 11:02 AM, Elmar Eperiesi-Beck wrote: > Jonas, > is there a repository, where I can find the actual source code of your encryption implementation? > Or do we have to sign an NDA to exchange these kind of information? > > We can have a look at your code and merge it with our existing implementation. > Additionally I would like to send you our concept paper, to give you an overview of our ideas. > > Regards > Elmar >

1 0

[Maria-developers] [GSoC] Week 3!
by sriram patil 09 Jun '14

09 Jun '14

Hi all, Here is my post for the 3rd week into GSoC project. http://goo.gl/bhl5ZW Suggestions and review are welcome! Thanks, Sriram

1 0

[Maria-developers] MariaDB 5.5.38 Release Prep
by Daniel Bartholomew 07 Jun '14

07 Jun '14

All, I've begun prep work on the MariaDB 5.5.38 release. Draft release notes and changelog are here: - https://mariadb.com/kb/en/mariadb-5538-release-notes/ - https://mariadb.com/kb/en/mariadb-5538-changelog/ Current plan is to activate and announce the release on Mon, 9 jun 2014. Thanks. -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

1 0

[Maria-developers] MDEV-5533 - Backport Threadpool improvements from Percona Server
by Sergey Vojtovich 04 Jun '14

04 Jun '14

Hi Sergei, Since MDEV-5533 is marked critical for 10.0.12 it is a good time to start this discussion. IIRC we agreed that our intention is to accept default value change for thread_pool_max_threads into 10.0 and accept priority queue throttling into 10.1. So default value for thread_pool_max_threads has bumped in percona server from 500 to 100 000. The reason is: deadlocks and uneven distribution of worker threads between thread groups in case of stalled connections. But the above is contradicting to suggestions in KB: https://mariadb.com/kb/en/threadpool-in-55/ ... so ideally, there should be a single active thread for each CPU on the machine. ... If your application constantly pegs at 500 threads, it might be a strong indicator for high contention in the application, and threadpool does not help much. ... One way to solve the problem would be to set thread_pool_max_threads to a very high value, but this might be less than ideal, and negatively affect performance. ... Said the above, do you still think it is a good idea to raise default value of thread_pool_max_threads? If so, what should KB article state wrt this variable? Thanks, Sergey

1 0

Re: [Maria-developers] MDEV-6213: Query cache is not invalidated on cascade delete when database name contains special symbols
by Sergei Golubchik 03 Jun '14

03 Jun '14

Hi, Sanja! Please add in your revision a test case for MySQL bug#13919851 http://bazaar.launchpad.net/~mysql/mysql-server/5.6/revision/4514 and make sure that your fix doesn't break it. On May 09, sanja(a)askmonty.org wrote: > revno: 4207 > revision-id: sanja(a)askmonty.org-20140509104240-ksrk10tef5dq157z > branch nick: work-maria-10.0-MDEV-6213 > timestamp: Fri 2014-05-09 13:42:40 +0300 > message: > MDEV-6213: Query cache is not invalidated on cascade delete when > database name contains special symbols. > > Innodb should de-escape db/table names when sending them to QC Regards, Sergei

1 0

[Maria-developers] Please review a patch fixing a few MDEV-6001 blockers
by Alexander Barkov 03 Jun '14

03 Jun '14

Hi Sergei, can you please review my patch fixing a few bugs that are blockers for MDEV-6001? - MDEV-4858 Wrong results for a huge unsigned value inserted into a TIME column - MDEV-6097 Inconsistent results for CAST(int,decimal,double AS DATETIME) - MDEV-6099 Bad results for DATE_ADD(.., INTERVAL 2000000000000000000.0 SECOND) - MDEV-6100 No warning on CAST(9000000 AS TIME) - MDEV-6101 Hybrid functions do not add CURRENT_DATE when converting TIME to DATETIME - MDEV-6102 Comparison between TIME and DATETIME does not use CURRENT_DATE Thanks.

2 10

[Maria-developers] Review request[major 5.3/5.5 bug]: MDEV-4511 Assertion `scale <= precision' fails on GROUP BY TIMEDIFF with incorrect types, , Agile Board, , Export
by Alexander Barkov 03 Jun '14

03 Jun '14

Hello Sergei, A few weeks ago we fixed this bug for 10.0 by changing temporal items not to have decimals=NOT_FIXED_DEC (more MySQL-5.6 compatibility), and agreed to have a separate patch for the pre-10.0 branches. This patch is for 5.3 and 5.5 only. It back-ports the mdev-4511 tests from 10.0 as is. But it does not touch the NOT_FIXED_DEC behavior of the temporal items and fixes the problem in a different way. Greetings.

1 0

[Maria-developers] POWER interest
by Colin Charles 03 Jun '14

03 Jun '14

Hi, It seems we have a POWER8 box and soon will start doing builds on this platform Stewart Smith (currently at IBM) seems to be doing some work on MySQL 5.6 and MySQL 5.7 for POWER8 systems - there doesn't seem to be much attention being paid to MariaDB 10. Ideally, our fixes that are being picked up from Stewart or focused on POWER should be targeted at 10.1 I'm thinking (unless there's customer interest for the 10.0 series -- which may very likely be the case) So far, I've noted GIS precise has an open failure, and maybe its a great idea for Holyfoot to use the POWER8 system to start fixing it: https://mariadb.atlassian.net/browse/MDEV-4153 From MySQL bugs system we see multiple examples from Stewart's blog posts - some are InnoDB specific so we will inherit them during a merge - some aren't like mysql#72715 (character set code endianness dependent on CPU type rather than endianness of CP) https://www.flamingspork.com/blog/2014/06/03/1-million-sql-queries-per-seco… https://www.flamingspork.com/blog/2014/06/03/mysql-5-7-on-power/ Relevant stuff that we may want to look at: http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=PowerPC cheers, -colin -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

1 0

[Maria-developers] [GSoC] First two weeks into coding!
by sriram patil 03 Jun '14

03 Jun '14

Hi all, I have created a blog to write the details of the work that I do as part of GSoC'14. Following is the link to my first blog post explaining the details of the work that I have done during the last two weeks. http://goo.gl/PYoVZb I will be writing these blog posts every weekend from now on and will share it on the developer mailing list every Monday. Suggestions/reviews are welcome. Thanks, Sriram

2 2

[Maria-developers] Possible race condition or NULL pointer triggered by OQGraph
by Andrew McDonnell 02 Jun '14

02 Jun '14

Hi all (Cross-posted to oqgraph-developers, maria-developers) I am trying to track down a segfault apparently triggered by concurrent execution of queries through OQGraph - for context, see: https://mariadb.atlassian.net/browse/MDEV-6282 I am however a bit confused as to what is going on, because at least some of the information leads me to suspect that there could be more going on here than just a simple race condition. --Immediate Cause-- On the surface, a segfault appears to be caused in a method in sql_class.h, Statement::check_limit_rows_examined() dereferencing a NULL pointer, `lex`. The method check_limit_rows_examined() was called from sql_class.h method handler::increment_statistics() . --Analysis-- A quick inspection reveals that a NULL value for `lex` is actually perfectly valid some of the time, although I am nowhere near familiar enough with the code to really understand when or why. Specifically, `lex` is a member of the base class Statement, which is initialised to either point to THD::main_lex via a constructor , or intentionally to NULL from construction of an object Prepared_statement() (sql_prepare.cc) To my mind, if there is going to be a possibility of a pointer being NULL, then all uses of that pointer should be checking for that. I am hoping that the bug reporter can prove or otherwise my hypothesis with his particular scenario, although I am suspicious that there may be additional issues to be dealt with; it is quite possible that lex being NULL is a red herring, being caused by a race condition memory overrun elsewhere. However, I think it is still a good idea to NULL check pointers that could be set to NULL intentionally. I can submit a pull request if the following patch is agreeable: diff --git a/sql/sql_class.h b/sql/sql_class.h index 5898d9e..957acf3 100644 --- a/sql/sql_class.h +++ b/sql/sql_class.h @@ -2475,8 +2475,10 @@ public: */ void check_limit_rows_examined() { - if (++accessed_rows_and_keys > lex->limit_rows_examined_cnt) + ++accessed_rows_and_keys; + if (lex && accessed_rows_and_keys > lex->limit_rows_examined_cnt) { killed= ABORT_QUERY; + } } --Aside-- Immediately prior to the check_limit_rows_examined() call is a call to macro status_var_increment(). Comments around the macro indicate it is often likely to be called in a mutli-threaded context, but the data is such that it is possible to disable protection with the only side effect being an occasionally 'inaccurate' result. Is it possible however, that check_limit_rows_examined() is not thread-safe and should have some protection? cheers, Andrew --A

3 3

[Maria-developers] MDEV-4051 INET6_ATON() and INET6_NTOA()
by Alexander Barkov 02 Jun '14

02 Jun '14

Hello Ian, I have merged these functions from MySQL-5.6 into 10.0: INET6_ATON() INET6_NTOA() IS_IPV4() IS_IPV4_COMPAT() IS_IPV4_MAPPED() IS_IPV6() Can you please document them? The corresponding article in the MySQL manual is here: http://dev.mysql.com/doc/refman/5.6/en/miscellaneous-functions.html Thanks.

2 2

[Maria-developers] KB page on switching from bazaar to git.
by Sergey Petrunia 27 May '14

27 May '14

Hi, I was doing some stuff in git already, and so I have created a git cheatsheet with equivalents of bzr commands that were useful in MariaDB development: https://mariadb.com/kb/en/using-git/ It is not finished, any additions are welcome. Elena has agreed to look at creating a commit trigger that sends commit emails (currently, github sends emails on push (not commit), and these do not include diffs) BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] CREATE OR REPLACE DATABASE algorithm
by sriram patil 27 May '14

27 May '14

Hi Alexander and all, Following are the steps that I am following for implementing "CREATE OR REPLACE" for DATABASE "..." signifies, there exists one or more lines of code in that block of the function. mysql_create_db(...) ... ... if(db exists) { if(create_or_replace) { if(DROP_ACL access not granted) { // Error 1 DBUG_RETURN(-1) } if(mysql_rm_db(...) unsuccessful) { // Error 2 DBUG_RETURN(-1) } Reset m_status from diagnostics cause it is changed by mysql_rm_db. Proceed with normal create db. } else if(if not exists) { ... } else ... } ... ... } Is the above algorithm correct? Also, I am not yet sure which errors should be displayed at both the places "Error 1" and "Error 2". I have written and tested the above code. It is working fine. Testing involves running all the existing test cases and executing "CREATE OR REPLACE DATABASE" command on the console manually. "IF NOT EXISTS" gives error if used with "CREATE OR REPLACE". Any other test cases I need to consider? Thanks, Sriram

2 2

[Maria-developers] FOR JSON
by Alexander Barkov 26 May '14

26 May '14

Hello Monty, We discussed extracting data in JSON format today, to make it possible to return data as a single JSON value (as a result set consisting of a single row and a single column): SELECT id,name,address FROM users FOR JSON; Note, in some cases the result of a such query can obviously be very huge. So it would be nice to be able to use "FOR JSON" in subselects, like this: SELECT id, (SELECT name,address FROM users AS b WHERE a.id= b.id FOR JSON) AS RowJSON FROM users AS a; This query will return a result set with multiple rows (one row per one record) and with two columns: - the ID, as a regular number - the JSON value with "name" and "address" embedded into it. Or, another example, if one does not need "id" as a separate column: SELECT (SELECT id,name,address FROM users AS b WHERE a.id= b.id FOR JSON) AS RowJSON FROM users AS a; This query will return a result set with multiple rows (one row per one record) with a single JSON value embedding all three columns: "id", "name" and "address".

2 4

[Maria-developers] MariaDB 10.1 is now on github!
by Sergei Golubchik 26 May '14

26 May '14

Hi! I'm happy to announce that MariaDB-10.1 tree has been completely migrated to github. Since now on we'll use github for the new development in MariaDB. It's https://github.com/MariaDB/server, go on, fork it, hack around, submit pull requests. Have fun! Older trees (10.0, 5.5, 5.3, 5.2, 5.1) are not on github - we do hope to migrate them too eventually, but at the moment they are still on launchpad. If you're a maria-captain on launchpad - for you to get write access to the mariadb repository on github you need to tell me your github account name. Regards, Sergei P.S.: Don't forget that 10.1 is still *pre-alpha*. You've been warned.

4 5

Re: [Maria-developers] MariaDB 10.1 is now on github!
by Sergei Golubchik 26 May '14

26 May '14

Hi, Binod! First: please send questions like this to maria-developers(a)lists.launchpad.net it's a public mailing list dedicated to MySQL and MariaDB internals, source code, and related things. I am subscribed, so I'll see you mail there, and you may be sure I will, because it won't be accidentally catched by my spam filter, or sorted out in some obscure folder. Furthermore other subscribers will see your question and could reply if I will be not available (e.g. I could be travelling). Thank you. On May 26, Binod Bista wrote: > Hi Sergei, > > Thanks for the info. Just to interrupt, is there a page you recommend for > anyone starting out in MariaDB development? Particularly any database or > MySQL internals material that help understand the codebase. That depends on what you're trying to do. There are two books: http://shop.oreilly.com/product/9780596009571.do http://www.packtpub.com/mysql-51-plugin-development/book There's MySQL internals manual: http://dev.mysql.com/doc/internals/en/index.html There's MariaDB Knowledge Base that includes pages about MariaDB internals: http://kb.askmonty.org/ There're numerous blog posts and presentations about specific topics. Regards, Sergei

1 0

[Maria-developers] Getting started with the coding.
by sriram patil 23 May '14

23 May '14

Hi Alexander, I have started working on "CREATE OR REPLACE DATABASE" implementation. I thought this one has the least complexity among all the other commands. I am getting my hand-on debugging the source code too. I have made the required changes in sql_yacc.yy. The command is parsed properly. Handled the error where CREATE OR REPLACE and IF NOT EXISTS cannot occur at the same time. This is how I am thinking of proceeding with the implementation, In method mysql_create_db after checking the flag HA_LEX_CREATE_REPLACE. I am removing the database by calling mysql_rm_db and then the current implementation will take care of creating the new db with the same name. This seems correct? Or is there anything I need to consider? Thanks, Sriram

2 1

[Maria-developers] delete + offset
by Roberto Spadim 22 May '14

22 May '14

hi guys! i was thinking at maria-discuss, and i have a new "feature request" to mariadb DELETE statment, that's the OFFSET clausule, like sqlite my idea is... i have a table with history about for example temperature, and i will only record last 5000 records, my idea is execute a 'cleanup' for example every day instead of: select date_time from history where channel=1234 order by date_time desc limit 1 offset 5000 delete from history where channel=1234 and data_time<value from previous query i was considering something like delete from history where channel=1234 order by date_time desc limit 99999999999 offset 5000 what you think about this? -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

[Maria-developers] Regarding debugging MariaDB
by sriram patil 17 May '14

17 May '14

Hi Alexander, I wanted to run mysqld in debug mode to check the trace file and get to know workflow of SQL commands. I tried following command mysqld --user=<username> --debug Then, after connecting I executed some commands. But, the trace file is not created in /tmp folder. Is there anything I need to do while building the source code in order for the debug mode to work? Thanks, Sriram

2 1

[Maria-developers] Merging FusionIO changes to 10.0.12
by Colin Charles 16 May '14

16 May '14

Hi all, MariaDB has worked actively with Fusion-io, and we did release some improvements in a preview release a little over a month ago: https://blog.mariadb.org/significant-performance-boost-with-new-mariadb-pag… This was originally planned to be included into 10.0.10, but when we released the GA, we thought that the feature was not tested thoroughly. Now the code has proven stable we are thinking of merging this into 10.0.12. Yes, we know its already a GA, but this feature will be turned off by default, and the code is safely encapsulated. If you are not using Fusion-io or do not turn it on, there is no effect to you. To check, the changes are in the XtraDB and InnoDB that we ship (storage/xtradb & storage/innobase). Naturally, for those using Fusion-io devices, the benefits are clear - you will see significant improvements in speed and you will increase the lifetime of your cards. Think of this like adding drivers to the Linux kernel from time to time ;) What does everyone think? Package maintainers? Thanks! -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

3 2

[Maria-developers] LIMIT ROWS MATCHED
by Colin Charles 16 May '14

16 May '14

Hi! http://seonguck.blogspot.kr/2014/05/limit-rows-matched.html We have a potential contribution here for a LIMIT ROWS MATCHED feature - might be interesting for us to consider this for 10.1 ? cheers, -colin -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

2 1

[Maria-developers] tokudb database - generated "should never happen" status when mysql log partition ENOSP
by Daniel Black 15 May '14

15 May '14

1 0

[Maria-developers] New to the list & question about CONNECT
by Martijn Tonies (Upscene Productions) 13 May '14

13 May '14

Hello everyone, My name is Martijn Tonies and I’m a Windows tools developer, we have a tool called “Database Workbench”, a developer tool for several database engines. For it’s next major version, we’re planning to officially support MariaDB in the MySQL module. I’ve investigated several of the features, implemented support for Roles, Virtual Columns etc, but I’m having trouble with the CONNECT engine. Here’s what I did, using MariaDB 10.0.10 1) registered a server with CREATE SERVER: CREATE SERVER mysql55_at_local FOREIGN DATA WRAPPER mysql OPTIONS(HOST 'MT-XP-VM-MYSQL', PORT 3355, DATABASE 'test', USER 'root', PASSWORD ‘secret’); 2) created a table in the MySQL database: CREATE TABLE sfdata ( mark1 Integer(11), mark2 Integer(11), mark3 Integer(11), mark4 Integer(11), `name` VarChar(50) CHARACTER SET latin1 COLLATE latin1_swedish_ci ) ENGINE=InnoDB DEFAULT CHARACTER SET latin1 COLLATE latin1_swedish_ci; 3) created a table in the MariaDB database: CREATE TABLE `sfdata_proxy3` ( `mark1` int(11) DEFAULT NULL, `mark2` int(11) DEFAULT NULL, `mark3` int(11) DEFAULT NULL, `mark4` int(11) DEFAULT NULL, `name` varchar(50) DEFAULT NULL ) ENGINE=CONNECT DEFAULT CHARSET=utf8 CONNECTION='mysql55_at_local/sfdata' I then expected that table “sfdata_proxy3” has the rows as “sfdata” has, but I got nothing returned. Any clue where to look? With regards, Martijn Tonies Upscene Productions http://www.upscene.com Download Database Workbench for Oracle, MS SQL Server, Sybase SQL Anywhere, MySQL, InterBase, NexusDB and Firebird!

1 1

Re: [Maria-developers] Question about GTIDs in MariaBD
by Kristian Nielsen 12 May '14

12 May '14

"AL13N" <alien(a)rmail.be> writes: >> Peter Laursen <peter_laursen(a)webyog.com> writes: >> So yes, there are others that consider Oracle's design of GTID 'a dirty >> hack', >> but MariaDB 10.0 GTID is not affected by that design. > What about mariadb being a slave of a Oracle MySQL GTID? The restrictions are on the master, not on the slaves. So what do you mean? Surely, connecting a MariaDB slave to an Oracle MySQL master will not remove those restrictions. Note that when a MariaDB server connects as a slave to an Oracle MySQL master server, it is not possible to use GTID for that connection (because the GTID implementations are not compatible). Old style non-GTID replication must be used. - Kristian.

2 1

Re: [Maria-developers] Question about GTIDs in MariaBD
by Peter Laursen 12 May '14

12 May '14

With autocommit=1 every statement runs in its own transaction, so obviousluy .. yes! MyISAM is/used to be completely 'transaction-agnosstic'. But GTIDs seem to confict with non-transactional storage engines (as far as I can understand). MySQL/Oracle have "solved" (!) this by not allowing updates to tables using transactional engines and non-transactional storage engines in same transaction. However to me it is nonsense that transactions apply (have any effect at all) for non-transactional storage engines. As a consequence I consider this solution (what must have been enforced in the server layer and not the storeage engine layer) by Oracle 'a dirty hack' (unless somewone can explain me why it it necessary). -- Peter On Sun, May 11, 2014 at 4:42 PM, Roberto Spadim <roberto(a)spadim.com.br>wrote: > With autoclmmit=1 works? > > Em domingo, 11 de maio de 2014, Peter Laursen <peter_laursen(a)webyog.com> > escreveu: > > Thanks for the reply, but I am afraid I don't fully understand. >> >> Here is a simple test case with mySQL 5.6.17: >> >> 1) >> You need all those in configuration [mysqld) section >> >> log-bin >> log-slave-updates >> gtid_mode = ON >> enforce_gtid_consistency = ON >> >> >> 2) >> Then this small script will reproduce: >> >> >> DROP DATABASE IF EXISTS `gtid_test`; >> CREATE DATABASE `gtid_test`; >> USE `gtid_test`; >> >> CREATE TABLE `tab1` ( >> `a` INT DEFAULT NULL >> ) ENGINE=MYISAM ; >> >> CREATE TABLE `tab2` ( >> `b` INT >> ) ENGINE=INNODB; >> >> --start a transaction >> SET autocommit=0; >> >> INSERT INTO `tab1` VALUES (1); >> INSERT INTO `tab2` VALUES (1); >> >> -- now the first UPDATE to the MyISAM table in the transaction triggers >> this error: >> >> UPDATE `tab1` SET `a` = 5 WHERE `a` = 1; >> >> -- and you'll get: >> -- >> -- Error Code: 1785 >> -- When @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1, updates to >> non-transactional tables can only be done in either autocommitted >> statements or single-statement transactions, and never in the same >> statement as updates to transactional tables. >> >> >> 3) >> ShouldI understand that MariaDB dos not prevent this (does not raise an >> error), but it is still unsafe as binlog - and thus replication - may be >> corrupted? >> >> >> Thanks! >> >> -- Peter >> >> On Sat, May 10, 2014 at 6:09 PM, Pavel Ivanov <pivanof(a)google.com> wrote: >> >>> There is no such limitation in MariaDB as well as no limitation on >>> CREATE TABLE ... SELECT. >>> >>> Although beware that mixing statements changing MyISAM and InnoDB >>> tables in statement binlog mode you may get different results from the >>> statements on slaves. >>> >>> Pavel >>> >>> On Sat, May 10, 2014 at 3:48 AM, Peter Laursen <peter_laursen(a)webyog.com> >>> wrote: >>> > GTID implementation in MySQL 5.6 has the limitation as described here: >>> > >>> https://dev.mysql.com/doc/refman/5.6/en/replication-gtids-restrictions.html >>> > >>> > >>> > "nontransactional storage engines such as MyISAM cannot be made in the >>> same >>> > statement or transaction as updates to tables using transactional >>> storage >>> > engines such as InnoDB." >>> > >>> > If you try you will get the error: >>> > Error Code: 1785 >>> > When @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1, updates to >>> non-transactional >>> > tables can only be done in either autocommitted statements or >>> > single-statement transactions, and never in the same statement as >>> updates to >>> > transactional tables. >>> > >>> > Is there same or similar/other limitations with GTID in MariaDB 10? >>> > >>> > -- Peter Laursen >>> > -- Webyog >>> > >>> > _______________________________________________ >>> > Mailing list: https://launchpad.net/~maria-developers >>> > Post to : maria-developers(a)lists.launchpad.net >>> > Unsubscribe : https://launchpad.net/~maria-developers >>> > More help : https://help.launchpad.net/ListHelp >>> > >>> >> >> > > -- > Roberto Spadim > SPAEmpresarial > Eng. Automação e Controle > >

3 3

[Maria-developers] Question about GTIDs in MariaBD
by Peter Laursen 11 May '14

11 May '14

GTID implementation in MySQL 5.6 has the limitation as described here: https://dev.mysql.com/doc/refman/5.6/en/replication-gtids-restrictions.html "nontransactional storage engines such as MyISAM<https://dev.mysql.com/doc/refman/5.6/en/myisam-storage-engine.html> cannot be made in the same statement or transaction as updates to tables using transactional storage engines such as InnoDB<https://dev.mysql.com/doc/refman/5.6/en/innodb-storage-engine.html> ." If you try you will get the error: Error Code: 1785 When @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1, updates to non-transactional tables can only be done in either autocommitted statements or single-statement transactions, and never in the same statement as updates to transactional tables. Is there same or similar/other limitations with GTID in MariaDB 10? -- Peter Laursen -- Webyog

2 3

[Maria-developers] MariaDB 10.0.11 release prep
by Daniel Bartholomew 10 May '14

10 May '14

I've started the release prep for MariaDB 10.0.11. Draft release notes and changelog pages are here: - https://mariadb.com/kb/en/mariadb-10011-release-notes/ - https://mariadb.com/kb/en/mariadb-10011-changelog/ Please make necessary additions/modifications/etc... The release is still building at the moment, but when finished I'll upload to the mirrors later today or tomorrow. Current plan, if everything goes well, is to activate and announce the release on Monday, 12 May. Thanks! -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

1 0

[Maria-developers] 10.0 -> 10.1 merge
by Sergey Petrunia 08 May '14

08 May '14

Hello Igor, As discussed, I've done the 10.0 -> 10.1 merge. I didn't get any warnings or new test failures in the 'main' test suite, so I've pushed the merge into 10.1 tree. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] connect.alter fails
by Alexander Barkov 07 May '14

07 May '14

Hello Olivier, connect.alter fails when built without XML support complaining about unsupported table type XML. I guess it should be skipped. connect.alter also fails with libxml2 with these symptoms: --- /home/bar/maria-bzr/maria-10.0-connect/storage/connect/mysql-test/connect/r/alter.result 2014-03-23 11:32:34.519624000 +0400 +++ /home/bar/maria-bzr/maria-10.0-connect/storage/connect/mysql-test/connect/r/alter.reject 2014-03-23 11:48:43.251680097 +0400 @@ -1,3 +1,5 @@ +Warnings: +Warning 1105 No file name. Table will use t1.xml Please fix. A remainder: please check every push in BuildBot: http://buildbot.askmonty.org/buildbot/grid?branch=10.0-connect

3 4

[Maria-developers] Barkov: report for the week starting Apr 29, 2014
by Alexander Barkov 05 May '14

05 May '14

Apr 28 - May 04 =============== Apr 28 Apr 29 Apr 30 May 01 Holiday May 02 Holiday May 03 Weekend May 04 Weekend Vacation days taken: 10/35 - MDEV-5702 Incorrect results are returned with NULLIF() Pushed into 5.5 - MDEV-5459 Illegal mix of collations for datetime Pushed into 5.5 - MDEV-6146 Can't mix (latin1_swedish_ci,NUMERIC) and (utf8_unicode_ci,IMPLICIT An after-fix pushed (the original patch broke "mtr --ps fulltext" - Merging gb18030 from MySQL-5.7 In progress - Discussion the manual changes with Ian

1 0

[Maria-developers] sources won't compile
by AugustQ 03 May '14

03 May '14

Hi, I encountered a problem by compiling the sources of MariaDB 10.04. Some days ago I upgraded my system from Ubuntu 13.10 to 14.04 and now I encounter these problems: gcc used is this: gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) a minor glitch: cmake complains: Syntax Warning in cmake code at /home/august/workspace/MariaDB/mariadb-10.0.4/contrib_AQ_01/cmake/ssl.cmake:210:55 Argument not separated from preceding token by whitespace. This is the reason for this: ELSE() MESSAGE(SEND_ERROR "Wrong option for WITH_SSL. Valid values are : "${WITH_SSL_DOC}) ENDIF() I added a space between "$(WITH and this works now. But in compiling the sources this error was reported: Scanning dependencies of target innobase [ 45%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/row/row0log.cc.o /home/august/workspace/MariaDB/mariadb-10.0.4/contrib_AQ_01/storage/innobase/row/row0log.cc: In function ‘dberr_t row_log_table_apply_ops(que_thr_t*, row_merge_dup_t*)’: /home/august/workspace/MariaDB/mariadb-10.0.4/contrib_AQ_01/storage/innobase/row/row0log.cc:2201:11: error: ‘srv_buf_size’ was not declared in this scope ofs, srv_buf_size); ^ By looking into the code of row0log.cc I found that FALLOC_FL_PUNCH_HOLE seems to be defined now. This is defined in /usr/include/linux/falloc.h. So what happened here during the upgrade of Linux? Regards AugustQ PS: by looking into Ubuntu 12.04 I found that this file also existed here.

2 2

[Maria-developers] Analysing degraded performance at high concurrency in sysbench OLTP
by Kristian Nielsen 02 May '14

02 May '14

At the Barcelona meeting in January, I promised to take a look at the high-concurrency sysbench OLTP benchmarks, and now I finally had the time do do this. There was a lot of work on LOCK_open by Svoj and Serg. If I have understood correctly, the basic problem was that at high concurrency (like, 512 threads), the TPS is only a small fraction of the peak throughput at lower concurrency. Basically, the server "falls over" and starts trashing instead of doing real work, due to some kind of inter-processor communication overhead. I started from Axel' OLTP sysbench runs and scripts, using 10.0 from bzr revno:4151 (revid:svoj@mariadb.org-20140415072957-yeir4jvokyilw5hp). I compiled without performance schema and with PGO, and ran sysbench 0.5 OLTP. (I just realised that my runs are with 32 tables, while I think the benchmarks in January focused on single-table runs. Maybe I need to re-do my analysis with the single-table benchmark, or perhaps it is too artificial to matter much?). In the read-only sysbench, the server mostly does not fall over. I guess this is due to the work by Svoj on eliminating LOCK_open? But in read-write, performance drops dramatically at high concurrency. TPS drops to 2600 at 512 threads compared to a peak of around 13000 (numbers here are approximate only, they vary somewhat between different runs). So I analysed the r/w benchmark with the linux `perf` tool. It turns out two-thirds of the time is spent in a single kernel function _raw_spin_lock(): - 66.26% mysqld [kernel.kallsyms] [k] _raw_spin_lock Digging further using --call-graph, this turns out to be mostly futex waits (and futex wakeups) from inside InnoDB locking primitives. Calls like sync_array_get_and_reserve_cell() and sync_array_wait_event() stand out in particular. So this is related to the non-scalable implementation in InnoDB of locking primitives, which is a known problem. I think Mark Callaghan has written about it a couple of times. Last time I looked at the code, every single mutex wait has to take a global mutex protecting some global arrays and stuff. I even remember seeing code that at mutex release would pthread_signal_broadcast() _every_ waiter, all of them waking up, only to all (except one) go do another wait. This is a kiler for scalability. While investigating, I discovered the variable innodb_sync_array_size, which I did not know about. It seems to split the mutex for some of the synchronisation operations. So I tried to re-run the benchmark with innodb_sync_array_size set to 8 and 64. In both cases, I got significant improvement, TPS increase to 5900, twice the value with innodb_sync_array_size set to the default of 1. So it is clear that the main limitation in this benchmark was the non-scalable InnoDB synchronisation implementation. After tuning innodb_sync_array_size, time spent in _raw_spin_lock() is down to half what it was before (33% of total time): + 33.77% mysqld [kernel.kallsyms] [k] _raw_spin_lock Now investigating call-graphs show that the sync_array operations are much less visible. Instead mutex_create_func(), called from dict_mem_table_create(), is the one that turns up prominently in the profile. I am not familiar with what this part of the InnoDB code is doing, but what I saw from a quick look is that it creates a mutex - and there is another global mutex needed for this, which again limits scalability. It is a bit surprising to see mutex creation being the most significant bottleneck in the benchmark. I would have assumed that most mutexes could be created up-front and re-used? It is possible that this is a warm-up thing, maybe the code is filling up the buffer pool or some table-cache like thing inside InnoDB? Because I see TPS being rather low for the first 150 seconds of the run (around 3000), and then increasing suddenly to around 8000-9000 for the rest. This might be worth investigating further. So in summary, my investigations found that the bottleneck in this benchmark, and the likely cause of the fall-over, is a scalability problem with InnoDB locking primitives. The sync_array part seems to be mitigated to some degree by innodb_sync_array_size, the mutex creation part still needs to be investigated. I wonder if the InnoDB team @ Oracle is doing something for this in 5.7? Does anyone know? I vaguely recall reading something about it, but I am not sure. It would seem a waste to duplicate their efforts. In any case, I hope this was useful. As part of this investigation, I installed a new 3.14 kernel on the lizard2 machine and a new `perf` installation, which seems to work well to do more detailed investigations of these kind of issues. So let me know if there are other benchmarks that I should look into. One thing that could be interesting is to look for false sharing; there are some performance counters that Intel manuals describe can be used for this. As an aside: In my tests, once concurrency becomes high enough that the server falls over, the actual TPS number becomes mostly meaningless. Eg. I saw putting dummy pause loops into the code increasing TPS. If TPS stabilises at N% of peak throughput as concurrency goes to infinity, then we can compare N. But if N goes to zero as concurrency goes to infinite, I think it is meaningless to compare actual TPS numbers - we should instead focus on removing the fall-over behaviour. (Maybe this is already obvious to you, I have not followed the previous benchmark efforts that closely). Hope this helps, - Kristian.

4 4

Re: [Maria-developers] MDEV-5792 - Deadlock between SELECTs from METADATA_LOCK_INFO and another
by Sergei Golubchik 01 May '14

01 May '14

Hi, Sergey! On Apr 23, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 4155 > revision-id: svoj(a)mariadb.org-20140423073406-o20n7jvwfjinvxxr > parent: psergey(a)askmonty.org-20140418100754-v5b5sdqbavbcznh7 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0-mdev5792 > timestamp: Wed 2014-04-23 11:34:06 +0400 > message: > MDEV-5792 - Deadlock between SELECTs from METADATA_LOCK_INFO and another > I_S table > > mdl_iterate() helper function (which is used by the plugin to iterate mdl > locks) acquired mutexes in reverse order. > > Fixed by iterating MDL locks in two stages: > 1. Iterate locks hash under the protection of hash mutex, store all > lock pointers in a thread local array and increment reference counter > for the lock. > 2. Iterate local array without protection of hash mutex, handle destroyed > locks. > > It somewhat echoes hack in MDL_map_partition::move_from_hash_to_lock_mutex. Looks ok to me. Regards, Sergei

1 0

[Maria-developers] Next steps in improving single-threaded performance
by Kristian Nielsen 01 May '14

01 May '14

I have been analysing CPU bottlenecks in single-threaded sysbench read-only load. I found that icache misses is the main bottleneck, and that profile-guided compiler optimisation (PGO) with GCC gives a large speedup, 25% or more. (More details in my blog posts: http://kristiannielsen.livejournal.com/17676.html http://kristiannielsen.livejournal.com/18168.html ) Now I would like to ask for some discussions/help in how to get this implemented in practice. It involves changing the build process for our binaries: First compile with gcc --coverage, then run some profile workload, then recompile with -fprofile-use. I implemented a simple program to generate some profile load: https://github.com/knielsen/gen_profile_load It runs a bunch of simple insert/select/update/delete, with different combinations of storage engine, binlog format, and client API. It is designed to run inside the build tree and handle starting and stopping the server being tested, so it is pretty close to a working setup. These commands work to generate a binary that is faster due to PGO: mkdir bld cd bld cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" .. make tests/gen_profile_load cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" make So all the pieces really are there, it should be possible to implement it. But we need to find a good way to integrate it into our build system. The best would be to integrate it into our cmake files. The gen_profile_load.c could go into tests/, ideally we would build both a static and dynamically linked version (so we get PGO for both libmysqlclient.a and libmysqlclient.so). Anyone can help me get cmake to do that? And it would be cool if we could get the above procedure to work completely within cmake, so that the user could just do: cmake -DWITH_PGO ... ; make and cmake would itself handle first building with --coverage, then running gen_profile_load.static and gen_profile_load.dynamic, then rebuilding with -fprofile-use. Anyone know if this is possible with cmake, and if so could help implement it? But alternatively, we could integrate a double build, like the commands above, into the buildbot scripts (.deb, .rpm, bintar). Any comments? Here are some more points: - I tested that gen_profile_load gives a good speedup of sysbench read-only (around 30%, so still very significant even though it generates a different and more varied load). - As another test, I removed all SELECT from gen_profile_load, and ran the resulting PGO binary with sysbench read-only. This still gave a fair speedup, despite the PGO load being completely different from the benchmark load. This gives me confidence that the PGO should not cause performance regressions in cases not covered well by gen_profile_load - More tests would be nice, of course. Axel, would you be able to build some binaries following above procedure, and test some different random benchmarks? Anything that is easy to run could be interesting, both to test for improvement, and to check against regressions. - We probably need a recent GCC version to get good results. I used GCC version 4.7.2. Maybe we should install this GCC version in all the VMs we use to build binaries? - Should we do this in 5.5? I think we might want to. The speedup is quite significant, and it seems very safe - no code modifications are involved, only different compiler options. Any thoughts? Volunteeres for helping with the cmake or buildbot parts? - Kristian.

4 14

[Maria-developers] valgrind failure in main.statistics
by Sergey Petrunia 29 Apr '14

29 Apr '14

Hello Igor, Valgrind build in buildbot currently fails in main.statistics test: http://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/5… I have tracked it down: the failure appeared after this push by you: revno: 4157 [merge] committer: Igor Babaev <igor(a)askmonty.org> branch nick: maria-10.0 timestamp: Tue 2014-04-22 14:43:13 -0700 message: Merge. ------------------------------------------------------------ revno: 4152.1.1 committer: Igor Babaev <igor(a)askmonty.org> branch nick: maria-10.0-bugs timestamp: Tue 2014-04-22 14:39:57 -0700 message: Fixed the problem of mdev-5947. Back-ported from the mysql 5.6 code line the patch with the following comment: Fix for Bug#11757108 CHANGE IN EXECUTION PLAN FOR COUNT_DISTINCT_GROUP_ON_KEY CAUSES PEFORMANCE REGRESSION I suppose it would take you much less time than anybody else to fix this? The failure is not repeatable on every machine: it doesn't repeat on my machines, but it can be repeated on work.askmonty.org. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 10.1 planning
by Kristian Nielsen 29 Apr '14

29 Apr '14

Sergei Golubchik <serg(a)mariadb.org> writes: > See below. > What do you think about it? > > On Apr 23, Elena Stepanova wrote: >> >> There is also https://mariadb.atlassian.net/browse/MDEV-4487 >> (Replication from 5.6 with enabled GTID) to be considered. It's >> currently Minor and has 10.0.x as a target version, both of which are >> questionable: recently Colin said on IRC that it should be much more >> than a minor FR, and yet I doubt that somebody will get to it in 10.0. >> Maybe it's more realistic to raise the priority but to re-target it for >> 10.1? I think Monty had some ideas for doing this, IIRC it was discussed at the meeting in Barcelona in January. If what I remember is true, it is actually just that a 5.6 master refuses to replicate if the slave does not say it is configured for MySQL 5.6 GTID. The slave does not need to actually _use_ GTID. So I think Monty's plan was for the MariaDB slave to just lie to the master and say that it's a 5.6 slave with appropriate config. And the MariaDB slave would need to recognise and ignore the new MySQL 5.6 events, such as GTID. - Kristian.

2 2

[Maria-developers] Old autotools cruft in source tree?
by Otto Kekäläinen 28 Apr '14

28 Apr '14

Hello The Debian Lintian QA scanner yields this warning: W outdated-autotools-helper-file (http://lintian.debian.org/tags/outdated-autotools-helper-file.html) extra/jemalloc/config.guess 2012-02-10 extra/jemalloc/config.sub 2012-02-10 storage/tokudb/ft-index/third_party/xz-4.999.9beta/build-aux/config.guess 2009-04-27 storage/tokudb/ft-index/third_party/xz-4.999.9beta/build-aux/config.sub 2009-04-17 Are these still needed in MariaDB or is this simply old cruft from old autotools time that could be removed? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 1

[Maria-developers] "SELECT ... LIMIT 0" is now deterministic
by Alexander Barkov 28 Apr '14

28 Apr '14

Hello Ian, A fix for MDEV-6170 fixed another problem as a side effect: "SELECT ... LIMIT 0" is now deterministic for statement based binary log. Queries with "LIMIT 0" do not return ER_BINLOG_UNSAFE_STATEMENT any more. Can you please mention this in the manual? Some discussion is here: https://mariadb.atlassian.net/browse/MDEV-6170 Related articles in the MySQL manual: https://dev.mysql.com/doc/refman/5.6/en/replication-rbr-safe-unsafe.html https://dev.mysql.com/doc/refman/5.6/en/replication-features-limit.html Thanks.

1 0

[Maria-developers] a Query_log_event charset bug in parallel replication
by nanyi607rao 25 Apr '14

25 Apr '14

hi, If character_set in different Query_log_events changed, worker threads may apply them with wrong character_set. the codes leading this problem is in Query_log_event::do_apply_event, that is: if (charset_inited) { if (rli->cached_charset_compare(charset)) { /* Verify that we support the charsets found in the event. */ if (!(thd->variables.character_set_client= get_charset(uint2korr(charset), MYF(MY_WME))) || !(thd->variables.collation_connection= get_charset(uint2korr(charset+2), MYF(MY_WME))) || !(thd->variables.collation_server= get_charset(uint2korr(charset+4), MYF(MY_WME)))) { There is a charset[6] in rli, which cached last Query_log_event's charset in serial replication. But in parallel replication it would lead mistake, because every worker thread can read and set rli->charset[6], so rli->charset[6] isn't any worker threads' last Query_log_event charset. but rli->charset[6] can affect every worker threads' thd->variables.character_set_* setting. 2014-04-17 nanyi607rao

2 7