July 2014 - developers - lists.mariadb.org

[Maria-developers] [GSoC] Optimize mysql-test-runs - Results of new strategy
by Pablo Estrada 17 Jul '14

17 Jul '14

Hello everyone, I took the last couple of days working on a new strategy to calculate the relevance of a test. The results are not sufficient by themselves, but I believe they point to an interesting direction. This strategy uses that rate of co-occurrence of events to estimate the relevance of a test, and the events that it uses are the following: - File editions since last run - Test failure in last run The strategy has also two stages: 1. Training stage 2. Executing stage In the training stage, it goes through the available data, and does the following: - If test A failed: - It counts and stores all the files that were edited since the last test_run (the last test_run depends on BRANCH, PLATFORM, and other factors) - If test A failed also in the previous test run, it also counts that In the executing stage, the training algorithm is still applied, but the decision of whether a test runs is based on its relevance, the relevance is calculated as the sum of the following: - The percentage of times a test has failed in two subsequent test_runs, multiplied by whether the test failed in the previous run (if the test didn't fail in the previous run, this quantity is 0) - For each file that was edited since the last test_run, the percentage of times that the test has failed after this file was edited (The explanation is a bit clumsy, I can clear it up if you wish so) The results have not been too bad, nor too good. With a running set of 200 tests, a training phase of 3000 test runs, and an executing stage of 2000 test runs, I have achieved recall of 0.50. It's not too great, nor too bad. Nonetheless, while running tests, I found something interesting: - I removed the first factor of the relevance. I decided to not care about whether a test failed in the previous test run. I was only using the file-change factor. Naturally, the recall decreased, from 0.50 to 0.39 (the decrease was not too big)... and the distribution of failed tests in the priority queue had a good skew towards the front of the queue (so it seems that the files help somewhat, to indicate the likelihood of a failure). I attached this chart. An interesting problem that I encountered was that about 50% of the test_runs don't have any file changes nor test failures, and so the relevance of all tests is zero. Here is where the original strategy (a weighted average of failures) could be useful, so that even if we don't have any information to guess which tests to run, we just go ahead and run the ones that have failed the most, recently. I will work on mixing up both strategies a bit in the next few days, and see what comes of that. By the way, I pushed the code to github. The code is completely different, so may be better to wait until I have new results soon. Regards! Pablo

2 11

[Maria-developers] Java Connector Coding till 5th June
by Puneet Dewan 17 Jul '14

17 Jul '14

Hi Everyone, The work done till 5th June: 1.I started studying the connector immediately after my semester exams got over(5th May). 2. Understood the relationship between different classes. 3.Read the MySql documentation for text protocol and map the commands from documentation onto code.This thing gave a clear idea about how things are moving from client to server. 4.Setup the code on Ubuntu using eclipse and tested against database. Setup the Team branch in launchpad. 5.Debug the test classes to see how different classes are interlinked and understood that MySQLProtocol.java is the heart of our JDBC driver. 6.Then I started reading Client Server Protocol documentation i.e ServerSide prepared statement and try to see how and where I have to code it. 7.Then I saw that there is a class MySqlServerSidePreparedStatement.java, Initially I coded the prepareStatement() ,close(),and execute() in that class seperately. But after (Code review) discussion with Massimo, we donot need to use that class. Instead use MySqlPreparedStatement.java to do the same work.So made changes accordingly. 8.I had pushed some of the code which I have written to launchpad and discussed with Massimo last week. 9.Last week I was struck with Wireshark on windows and ubuntu both. I was unable to see the packets going to database and seeing the response. 10.On Monday the problem was resolved as I was using the wrong interface.I should have used loopback.io(127.0.0.1) instead on ubuntu(thanks to Georg). Unfortunately the thing does not work on windows.Tried a lot many times. 11.Now I am working on the execute method of client server protocol as I am getting exception while using that method.Wireshark is playing a great role in how the packets are flowing. >From now I will do weekly blogging and keep everyone updated. Thanks & Regards Puneet.

2 9

[Maria-developers] Help needed TokuDB build errors in Debian
by Otto Kekäläinen 16 Jul '14

16 Jul '14

Hello Rich! For some strange reason, re-building 5.5.37 in Debian unstable stopped working. Any ideas what might be going on here?: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=753222 Recently a build of 5.5.38 generated a similar error: http://paste.ubuntu.com/7747594/ Are these maybe related? I have no idea how to fix these apart from disabling TokuDB in Debian when uploading 5.5.38. -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

3 5

[Maria-developers] [GSoC] Week Eight!
by sriram patil 14 Jul '14

14 Jul '14

Hi all, Following is the blog link about my eighth week into GSoC. http://goo.gl/h2mnbv Suggestions and/or reviews are welcome! Thanks, Sriram

1 0

Re: [Maria-developers] [Commits] Rev 4876: Post-fixes after UNIQUE index support was enabled without uniquenessenforcement. in file:///home/psergey/chroot/saucy-x64/home/psergey/dev2/mysql-5.6-rocksdb-r2/
by Roberto Spadim 11 Jul '14

11 Jul '14

=] nice rocksdb, any devel code to test it? i have some key-value databases to test 2014-07-11 13:21 GMT-03:00 Sergey Petrunya <psergey(a)askmonty.org>: > At > file:///home/psergey/chroot/saucy-x64/home/psergey/dev2/mysql-5.6-rocksdb-r2/ > > ------------------------------------------------------------ > revno: 4876 > revision-id: psergey(a)askmonty.org-20140711162114-051drr4tocqpbx30 > parent: psergey(a)askmonty.org-20140711161708-b50tdmlr60djrror > committer: Sergey Petrunya <psergey(a)askmonty.org> > branch nick: mysql-5.6-rocksdb-r2 > timestamp: Fri 2014-07-11 20:21:14 +0400 > message: > Post-fixes after UNIQUE index support was enabled without > uniquenessenforcement. > - Fix compile warning > - Fix the rocksdb.test/result > === modified file 'mysql-test/r/rocksdb.result' > --- a/mysql-test/r/rocksdb.result 2014-07-03 20:58:25 +0000 > +++ b/mysql-test/r/rocksdb.result 2014-07-11 16:21:14 +0000 > @@ -686,8 +686,8 @@ DROP TABLE t33; > # > # MDEV-4081: RocksDB throws error 122 on an attempt to create a table > with unique index > # > +# Unique indexes can be created, but uniqueness won't be enforced > create table t33 (pk int primary key, u int, unique index(u)) > engine=RocksDB; > -ERROR HY000: Got error 187 'Unique indexes are not supported' from ROCKSDB > # > # MDEV-4077: RocksDB: Wrong result (duplicate row) on select with range > # > > === modified file 'mysql-test/t/rocksdb.test' > --- a/mysql-test/t/rocksdb.test 2014-07-03 20:58:25 +0000 > +++ b/mysql-test/t/rocksdb.test 2014-07-11 16:21:14 +0000 > @@ -610,7 +610,8 @@ DROP TABLE t33; > --echo # > --echo # MDEV-4081: RocksDB throws error 122 on an attempt to create a > table with unique index > --echo # > ---error ER_GET_ERRMSG > +#--error ER_GET_ERRMSG > +--echo # Unique indexes can be created, but uniqueness won't be enforced > create table t33 (pk int primary key, u int, unique index(u)) > engine=RocksDB; > > > > === modified file 'storage/rocksdb/ha_rocksdb.cc' > --- a/storage/rocksdb/ha_rocksdb.cc 2014-07-09 23:46:24 +0000 > +++ b/storage/rocksdb/ha_rocksdb.cc 2014-07-11 16:21:14 +0000 > @@ -1051,7 +1051,7 @@ int ha_rocksdb::create(const char *name, > /* > We only support auto_increment at start of the PRIMARY KEY. > */ > - Field *field; > + //Field *field; > //if ((field= table_arg->next_number_field)) > /* TODO mdcallag: disable this for now to let UNIQUE indexes kind of > work > if ((field= table_arg->found_next_number_field)) > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits > -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

Re: [Maria-developers] apt-get upgrade on buildbot
by Kristian Nielsen 11 Jul '14

11 Jul '14

Otto Kekäläinen <otto(a)seravo.fi> writes: > Hello Kristian! > > Could you do a small favour and run 'apt-get upgrade' and then trigger > manually a rebuild in the virtual machine Debian unstable the buildbot > uses at http://buildbot.askmonty.org/buildbot/builders/debpkg-sid > > Thanks! Hm, I tried this, but I got the below errors. Any ideas what I did wrong? - Kristian. buildbot@debian-sid-amd64:~$ sudo apt-get update Hit http://ftp.us.debian.org sid InRelease Hit http://ftp.us.debian.org sid/main Sources/DiffIndex Hit http://ftp.us.debian.org sid/main amd64 Packages/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-en/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-cs/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ru/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-nl/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-eo/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ca/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-pl/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-fi/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-hr/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-fr/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-eu/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-sr/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-hu/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-da/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ro/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-es/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-pt/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-sv/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-sk/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ko/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-vi/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-uk/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-km/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ja/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-de/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-it/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-nb/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-zh/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-id/DiffIndex Get:1 http://ftp.us.debian.org sid/main Translation-ru [423 kB] Get:2 http://ftp.us.debian.org sid/main Translation-pl [575 kB] 100% [2 Translation-pl bzip2 0 B] 66.3 kB/s 0sFalling back to normal index file acquire Ign http://ftp.us.debian.org sid/main 2014-07-09-0854.18.pdiff Get:3 http://ftp.us.debian.org sid/main Translation-fr [731 kB] 100% [3 Translation-fr bzip2 0 B] 90.5 kB/s 0sFalling back to normal index file acquire Ign http://ftp.us.debian.org sid/main 2014-07-09-0854.18.pdiff Get:4 http://ftp.us.debian.org sid/main Translation-da [2,414 kB] Get:5 http://ftp.us.debian.org sid/main Translation-es [323 kB] Get:6 http://ftp.us.debian.org sid/main Translation-pt [444 kB] Get:7 http://ftp.us.debian.org sid/main Translation-sk [421 kB] Get:8 http://ftp.us.debian.org sid/main Translation-uk [776 kB] Get:9 http://ftp.us.debian.org sid/main Translation-ja [786 kB] Get:10 http://ftp.us.debian.org sid/main Translation-de [1,772 kB] Get:11 http://ftp.us.debian.org sid/main Translation-it [3,259 kB] Get:12 http://ftp.us.debian.org sid/main Sources [7,333 kB] Get:13 http://ftp.us.debian.org sid/main amd64 Packages [6,820 kB] Get:14 http://ftp.us.debian.org sid/main [116 kB] Get:15 http://ftp.us.debian.org sid/main [35.0 kB] Fetched 26.2 MB in 1min 31s (288 kB/s) W: Failed to fetch xz:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_source_Sources Hash Sum mismatch W: Failed to fetch xz:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_binary-amd64_Packages Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-ru Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-pl Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-fr Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-da Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-es Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-pt Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-sk Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-uk Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-ja Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-de Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-it Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-fi Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-hu Hash Sum mismatch E: Some index files failed to download. They have been ignored, or old ones used instead. buildbot@debian-sid-amd64:~$ sudo apt-get dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

2 2

[Maria-developers] [GSoC] Suggestions on adding a new method
by sriram patil 11 Jul '14

11 Jul '14

Hi All, I was thinking of adding two new inline methods in struct LEX as follows inline bool is_create_or_replace() { return (create_info.options & HA_LEX_CREATE_REPLACE); } inline bool is_create_if_not_exists() { return (create_info.options & HA_LEX_CREATE_IF_NOT_EXISTS); } With these, statements like if(lex->create_info.options & HA_LEX_CREATE_REPLACE) will be changed to if(lex->is_create_or_replace()) which definitely is more readable. This kind of statements are used in sql_yacc.yy, sql_parse.cc and many object specific implementation .cc files. Should I make the changes or is it unnecessary? Thanks, Sriram

1 0

Re: [Maria-developers] [JIRA] (MDEV-4937) NEED REVIEW: sql_slave_skip_counter does not work with GTID
by Kristian Nielsen 09 Jul '14

09 Jul '14

"Michael Widenius (JIRA)" <jira(a)mariadb.atlassian.net> writes: > Michael Widenius commented on MDEV-4937: > ---------------------------------------- Thanks for review Monty! I have some comments inline, let me know if you still want me to change something. - Kristian. > revno: 4260 > revision-id: knielsen at knielsen-hq.org-20140620104939-xdvn985cgmwy3odd > committer: Kristian Nielsen <knielsen at knielsen-hq.org> > timestamp: Fri 2014-06-20 12:49:39 +0200 > message: > MDEV-4937: sql_slave_skip_counter does not work with GTID > === modified file 'sql/slave.cc' > --- a/sql/slave.cc 2014-06-19 12:50:43 +0000 > +++ b/sql/slave.cc 2014-06-20 10:49:39 +0000 > @@ -3521,9 +3521,6 @@ static int exec_relay_log_event(THD* thd > > if (opt_gtid_ignore_duplicates) > { > - serial_rgi->current_gtid.domain_id= gev->domain_id; > - serial_rgi->current_gtid.server_id= gev->server_id; > - serial_rgi->current_gtid.seq_no= gev->seq_no; > int res= rpl_global_gtid_slave_state.check_duplicate_gtid > (&serial_rgi->current_gtid, serial_rgi); > if (res < 0) > What was the reason for removing the above assignments? > Was it a bug in the old code ? Not really a bug, however, just before this code we have: if (event_group_new_gtid(serial_rgi, gev)) and event_group_new_gtid() does the exact same assignments, so the ones I removed are completely redundant. I just happened to notice this redundancy, so I removed them... > (I have to ask as I am not 100% of what the old code did). > @@ -4594,16 +4598,27 @@ log '%s' at position %s, relay log '%s' > > if (saved_skip && rli->slave_skip_counter == 0) > { > + String tmp; > + if (mi->using_gtid != Master_info::USE_GTID_NO) > + { > + tmp.append(STRING_WITH_LEN(", GTID '")); > + rpl_append_gtid_state(&tmp, false); > + tmp.append(STRING_WITH_LEN("'; ")); > + } > > {noformat} > > If you want to avoid checking if tmp.append() fails, you could > create tmp with a static buffer. > > char tmp_buff[MAX_GTID_LENGTH+10]; > String tmp(tmp_buff, sizeof(tmp_buff), &my_charset_bin); > tmp.length(0); Right, hm. It does not really feel right to enforce a limit on the length; if the user uses a longer GTID position (which would be rather unusual), it might be confusing to have it silently cut. If the append() should fail, well, then it _will_ be cut, but then it's just an informational message, and that kind of out-of-memory is probably unlikely (and the error log will likely be filled up with other complaints about out-of-memory). But let me know if you still think I should change it. (If so - can you think of a place where MAX_GTID_LENGTH can be defined, that would be accessible to both client/mysqldump.c and sql/slave.cc? I could spot only my_global.h, which seems a bit too generic to contain GTID-related stuff...?) > if (saved_skip && rli->slave_skip_counter == 0) > { > String tmp; > saved_skip= 0; > + saved_skip_gtid_pos.free(); > } > If you free saved_skip_gtid_pos, it good to also free tmp! Right ... but it should be freed automatically anyway, when tmp goes out of scope, right? > === modified file 'sql/sys_vars.cc' > --- a/sql/sys_vars.cc 2014-06-09 18:00:23 +0000 > +++ b/sql/sys_vars.cc 2014-06-20 10:49:39 +0000 > @@ -4287,11 +4287,6 @@ bool update_multi_source_variable(sys_va > > static bool update_slave_skip_counter(sys_var *self, THD *thd, Master_info *mi) > { > - if (mi->using_gtid != Master_info::USE_GTID_NO) > - { > - my_error(ER_SLAVE_SKIP_NOT_IN_GTID, MYF(0)); > - return true; > - } > > {noformat} > > Shouldn't you also remove the error ER_SLAVE_SKIP_NOT_IN_GTID from the > error file? I think we cannot remove error codes from a GA release once introduced, right? So I left it, to not shift other error numbers. But let me know if I'm wrong, and I'll remove it. - Kristian.

1 0

Re: [Maria-developers] [JIRA] (MDEV-6336) NEED REVIEW: mysqldump --master-data does not work with GTID setups
by Kristian Nielsen 09 Jul '14

09 Jul '14

"Michael Widenius (JIRA)" <jira(a)mariadb.atlassian.net> writes: > Here is the review: > > ------------------------------------------------------------ > revno: 4245 > revision-id: knielsen at knielsen-hq.org-20140616093803-gub60pda6t77ch3k > committer: knielsen at knielsen-hq.org > timestamp: Mon 2014-06-16 11:38:03 +0200 > message: > MDEV-6336: mysqldump --master-data does not work with GTID setups > MDEV-6344: mysqldump issues FLUSH TABLES, which gets written into binlog and replicated Thanks for review Monty! I fixed all of your suggestions. - Kristian.

1 0

Re: [Maria-developers] [Commits] Rev 4281: MDEV-5262, MDEV-5914, MDEV-5941, MDEV-6020: Deadlocks during parallel replication causing replication to fail. in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Kristian Nielsen 09 Jul '14

09 Jul '14

Jan Lindström <jan.lindstrom(a)skysql.com> writes: > I have mostly InnoDB style comments, but some questions below: Thanks for comments! Sorry about style violations, and thanks for pointing then out. I'll try to fix all of it. Answers to other questions below: > Why you need to define these, why #include "mysql/plugin.h" file is not > enough ? > > +extern "C" void thd_report_wait_for(const MYSQL_THD thd, MYSQL_THD > other_thd); > +extern "C" int thd_need_wait_for(const MYSQL_THD thd); > +extern "C" > +int thd_need_ordering_with(const MYSQL_THD thd, const MYSQL_THD other_thd); See the discussions here: https://lists.launchpad.net/maria-developers/msg07494.html The conclusion with Serg was that doing a proper public API in 10.0 was too big a change. We will do it in 10.1 (MDEV-6429). plugin.h should not be modified in a GA version and with an API that has not been thought through wrt. all engines. Also, the API should be a service, not in plugin.h. > Why you need to introduce a new global structure for this, can't this be > part > of some already existing system structure e.g. lock_sys and why you do not > use ut_list ? > > +/* Buffer to collect THDs to report waits for. */ > +struct thd_wait_reports { > + struct thd_wait_reports *next; > + ulint used; > + trx_t *waitees[64]; > +}; So the idea here is to collect a bunch of trx pointers during the graph traversal, and pass each of them to thd_report_wait_for(). I wanted to pass them after completing the graph traversal to avoid issues if thd_report_wait_for() needs to kill one of them; this goes to lock_cancel_waiting_and_release() eventually which I assume might modify the graph being traversed. This is time-critical code (it runs under global lock_sys->mutex), so it is important to avoid memory allocations. So I use an array allocated on the stack, this is really short-lived data, it only lives for the duration of lock_deadlock_check_and_resolve(). In all normal cases the single array should be sufficient, but the code needs to handle the overflow also, of course. So in case of overflow, I malloc further arrays and link them into the first. I suppose another solution would be to abort in case of overflow and roll back the transaction like in case of ctx->too_deep==true, however I don't like to introduce such arbitrary limits when it's not necessary. I looked into using ib_list, but this seems to always do mallocs, which I want to avoid. I also looked at ut0vec and dyn0dyn, but they also seem to always use malloc and thus do not seem appropriate. So I did not find anything existing I could use, unfortunately, but then I know little of the InnoDB code. Do you have a suggestion for an existing data structure to use that can take an inital buffer on the stack and malloc extra memory only if needed? Thanks for comments! - Kristian.

1 0