- developers - lists.mariadb.org

[Maria-developers] Connector Coding till 28th July
by Puneet Dewan 28 Jul '14

28 Jul '14

Hi Everyone, Last week, 1.Migrated the code from MySQLPreparedStatement to MySQLServerSidePreparedStatement. 2.Removed duplicate code in classes like IntParameter, DoubleParameter, LongParameter and used methods of class ParameterWriter(provides utility methods) in the above classes. 3.Did the analysis of TextProtocol Resultset has been implemented in connector code and find a way on how to write for Binary Protocol Resultset. 4.Fixed the new_params_bound_flag issue. 5.Code review with Massimo. This week, 1.Code review with Massimo 2.Complete the coding for Binary Protocol Resultset. 3.Write code for allowMultiQueries, Batchstatements. Regards Puneet.

1 0

[Maria-developers] [GSoC] Week Nine and Ten!
by sriram patil 28 Jul '14

28 Jul '14

Hi All, Following is the blog link about my ninth and tenth week into GSoC. http://goo.gl/N5Le32 Suggestions and/or reviews are welcome! Thanks, Sriram

1 0

[Maria-developers] MDEV-4511 Assertion `scale <= precision' fails on GROUP BY TIMEDIFF with incorrect types
by Alexander Barkov 25 Jul '14

25 Jul '14

Hello Sergei, Please review a new version of the patch for MDEV-4511, it also fixes more crashing problems found, as well as non-crashing problems reported in MDEV-6302 I removed find_num_type() and fix_num_length_and_dec() which were hard to track, and for some Items these methods were erroneously called twice. fix_num_length_and_dec() now only exists as a non-virtual method in Item_sum and Item_udf_func. The patch is for 5.3, but it is still valid for 10.0, as we still have temporal functions with NOT_FIXED_DEC in 10.0. Thanks.

2 3

[Maria-developers] Any plans about starting to use /var/lib/mariadb for data instead of /var/lib/mysql?
by Otto Kekäläinen 24 Jul '14

24 Jul '14

Hello! Does the MariaDB project have any plans about switching the data dir to /var/lib/mariadb in some release? In Debian there has been discussion about separating the binary-incompatible MySQL forks into separate /var/lib/<fork> to ease maintenance and secure data so that purging some packages will not accidentally delete data that another package uses. The thread is named "Discussion on handling 5.6 related matters" and available at http://lists.alioth.debian.org/pipermail/pkg-mysql-maint/2014-July/thread.h… I'd appreciate any guidelines here on what the upstream plans are. The Debian MySQL/MariaDB packaging team willl have an IRC meeting tomorrow at 16:00 UTC and I need to express my opinion on the mattes and what kind of changes I'm willing to do to the MariaDB package. Obviously I'd like to follow upstream as much as possible to keep the delta between Debian and upstream small and thus the maintenance burden reasonable. -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

3 2

[Maria-developers] please review mdev-5745 analyze MySQL fix for bug#12368495
by Alexander Barkov 23 Jul '14

23 Jul '14

Hi Serg, please review a patch for mdev-5745. Thanks.

1 0

Re: [Maria-developers] [Commits] 00ed367: MDEV-4262 - P_S discovery
by Sergei Golubchik 22 Jul '14

22 Jul '14

Hi, Sergey! On Jul 07, svoj(a)mariadb.org wrote: > revision-id: 00ed36700addfbce91940223cce4ccd6ee5823a1 > parent(s): f37f19a6c790100cc2ae5f483ba4db34a517f0ae > committer: Sergey Vojtovich > branch nick: 10.1 > timestamp: 2014-07-07 14:32:13 +0400 > message: > > MDEV-4262 - P_S discovery > > Discover P_S tables automatically. > > Most of this patch is code clean-up: > - removed tests and code responsible for P_S tables correctness verification > - always return error from ha_perfschema::create() > - install/upgrade scripts won't create P_S tables anymore Great work, thanks! The patch is pretty much "ok to push", but, first I want to understand why you deleted table_file_summary.cc file. See other minor comments below. > diff --git a/mysql-test/suite/perfschema/include/upgrade_check.inc b/mysql-test/suite/perfschema/include/upgrade_check.inc > index 52d4cfd..1532de8 100644 > --- a/mysql-test/suite/perfschema/include/upgrade_check.inc > +++ b/mysql-test/suite/perfschema/include/upgrade_check.inc > @@ -3,7 +3,6 @@ > # > > --source include/count_sessions.inc > ---error 1 > --exec $MYSQL_UPGRADE --skip-verbose --force > $out_file 2> $err_file > --source include/wait_until_count_sessions.inc The rest of this upgrade_check.inc file does (according to the comment) "Verify that mysql_upgrade complained about the performance_schema" perhaps you should change the comment to say "does not complain" ? > > diff --git a/scripts/mysql_performance_tables.sql b/scripts/mysql_performance_tables.sql > index d495578..3061603 100644 > --- a/scripts/mysql_performance_tables.sql > +++ b/scripts/mysql_performance_tables.sql > @@ -53,1427 +53,3 @@ SET @str = IF(@broken_pfs = 0, @cmd, 'SET @dummy = 0'); > PREPARE stmt FROM @str; > EXECUTE stmt; > DROP PREPARE stmt; Why did you keep the above? I'd think that a simple DROP DATABASE IF EXISTS performance_schema should be enough to replace the whole file. And it could be moved back to mysql_system_tables.sql, so this file could be completely removed. > - > -# > -# From this point, only create the performance schema tables > -# if the server is built with performance schema > -# > - > -set @have_pfs= (select count(engine) from information_schema.engines where engine='PERFORMANCE_SCHEMA' and support != 'NO'); > - > -# > -# TABLE COND_INSTANCES > -# ... > diff --git a/storage/perfschema/pfs_engine_table.cc b/storage/perfschema/pfs_engine_table.cc > index 958a2bd..6f0a363 100644 > --- a/storage/perfschema/pfs_engine_table.cc > +++ b/storage/perfschema/pfs_engine_table.cc > @@ -1440,5 +1319,17 @@ end: > DBUG_RETURN(false); > } > > +int pfs_discover_table_names(handlerton *hton, LEX_STRING *db, > + MY_DIR *dir, hm, no "unused argument" compiler warnings? okay... > + handlerton::discovered_list *result) > +{ > + if (compare_table_names(db->str, PERFORMANCE_SCHEMA_str.str)) > + return 0; > + for (size_t i= 0; i < array_elements(all_shares) - 1; i++) > + result->add_table(all_shares[i]->m_name.str, > + all_shares[i]->m_name.length); > + return 0; > +} > + > /** @} */ > > diff --git a/storage/perfschema/table_file_summary.cc b/storage/perfschema/table_file_summary.cc > deleted file mode 100644 > index 104fa0f..0000000 > --- a/storage/perfschema/table_file_summary.cc > +++ /dev/null > @@ -1,372 +0,0 @@ Why did you remove this file? > -/* Copyright (c) 2008, 2010, Oracle and/or its affiliates. All rights reserved. > - > - This program is free software; you can redistribute it and/or modify > - it under the terms of the GNU General Public License as published by > - the Free Software Foundation; version 2 of the License. Regards, Sergei

2 4

Re: [Maria-developers] MDEV-4904 - Add query cache hit information to Percona Response time distribution plugin (MDEV-4568)
by Sergei Golubchik 22 Jul '14

22 Jul '14

Hi, svoj! On Jun 26, svoj(a)mariadb.org wrote: > revision-id: f37f19a6c790100cc2ae5f483ba4db34a517f0ae > parent(s): 4a3f0684195703996b23c51af79cf946b85b8053 > committer: Sergey Vojtovich > branch nick: 10.1 > timestamp: 2014-06-26 16:37:46 +0400 > message: > > MDEV-4904 - Add query cache hit information to Percona Response time distribution plugin (MDEV-4568) > > Added two new fields to INFORMATION_SCHEMA.QUERY_RESPONSE_TIME: > QC_HITS_COUNT - query cache hits count > QC_HITS_TOTAL - cumulative response time for query cache hits. This is somewhat questionable feature. Queries served from qc don't take much time, which means that @@query_response_time_range_base should be different for them. Otherwise all qc hits will be in the first few slots. But a different base means a different table (QUERY_CACHE_RESPONSE_TIME ?) and a different set of configuration variables. On the other hand, this current implementation is much simpler. So I think we can accept this patch. But I don't know how useful it'll be. Alternatively, you can do a separate I_S table for QC. As you like. > diff --git a/plugin/query_response_time/mysql-test/query_response_time/query_response_time-stored.result b/plugin/query_response_time/mysql-test/query_response_time/query_response_time-stored.result > index bec7007..e487fc5 100644 > --- a/plugin/query_response_time/mysql-test/query_response_time/query_response_time-stored.result > +++ b/plugin/query_response_time/mysql-test/query_response_time/query_response_time-stored.result > @@ -40,51 +40,51 @@ SHOW GLOBAL VARIABLES where Variable_name like 'QUERY_RESPONSE_TIME_RANGE_BASE'; > Variable_name Value > query_response_time_range_base 2 > SELECT * FROM INFORMATION_SCHEMA.QUERY_RESPONSE_TIME; I'd rather replaced * with TIME,COUNT,TOTAL - less changes and more future proof. > -TIME COUNT TOTAL > - 0.000001 45 0.000000 > +TIME COUNT TOTAL QC_HITS_COUNT QC_HITS_TOTAL > + 0.000001 45 0.000000 0 0.000000 ... Regards, Sergei

1 0

[Maria-developers] Why does name resolution stop in crash stack trace?
by Sergey Petrunia 21 Jul '14

21 Jul '14

Hi, Look at today's crash report in maria-discuss@. A question not related to the cause of that crash: why do stack traces that go inside the storage engine cannot be resolved? Is something wrong with our build process? On Fri, Jun 13, 2014 at 08:15:39PM -0700, grantksupport(a)operamail.com wrote: > ==> /var/log/mariadb/mariadb-err.log <== > 140613 20:07:24 [ERROR] mysqld got signal 11 ; ... > Server version: 10.0.11-MariaDB-log > key_buffer_size=268435456 > read_buffer_size=1048576 > max_used_connections=1 > max_threads=66 > thread_count=1 > It is possible that mysqld could use up to > key_buffer_size + (read_buffer_size + > sort_buffer_size)*max_threads = 601410 K bytes of > memory > Hope that's ok; if not, decrease some variables in the > equation. > > Thread pointer: 0x0x7fcf132fa008 > Attempting backtrace. You can use the following > information to find out > where mysqld died. If you see no messages after this, > something went > terribly wrong... > stack_bottom = 0x7fcf4a6e4e78 thread_stack 0x100000 > /usr/local/mariadb/bin/mysqld(my_print_stacktrace+0x2e)[0xacb60e] > /usr/local/mariadb/bin/mysqld(handle_fatal_signal+0x390)[0x6f5a10] > /lib64/libpthread.so.0(+0xf9f0)[0x7fcf4ff2a9f0] > /usr/local/mariadb/bin/mysqld[0x915cba] > /usr/local/mariadb/bin/mysqld[0x909e27] > /usr/local/mariadb/bin/mysqld[0x90a35b] > /usr/local/mariadb/bin/mysqld[0x90c759] > /usr/local/mariadb/bin/mysqld[0x9169d5] > /usr/local/mariadb/bin/mysqld[0x8aaf2b] > /usr/local/mariadb/bin/mysqld[0x8ab3cf] > /usr/local/mariadb/bin/mysqld[0x8ab95a] > /usr/local/mariadb/bin/mysqld[0x895e80] > /usr/local/mariadb/bin/mysqld[0x89643f] > /usr/local/mariadb/bin/mysqld[0x9657da] > /usr/local/mariadb/bin/mysqld[0x965a16] > /usr/local/mariadb/bin/mysqld[0x96a51d] > /usr/local/mariadb/bin/mysqld[0x96ac28] > /usr/local/mariadb/bin/mysqld[0x82de54] > /usr/local/mariadb/bin/mysqld(_ZN7handler9ha_createEPKcP5TABLEP14HA_CREATE_INFO+0x70)[0x6fcbf0] > /usr/local/mariadb/bin/mysqld(_Z15ha_create_tableP3THDPKcS2_S2_P14HA_CREATE_INFOP34st_mysql_const_unsigned_lex_string+0x220)[0x6fd560] > /usr/local/mariadb/bin/mysqld(_Z16rea_create_tableP3THDP34st_mysql_const_unsigned_lex_stringPKcS4_S4_P14HA_CREATE_INFOP7handlerb+0xbb)[0x659d1b] > /usr/local/mariadb/bin/mysqld[0x62e50d] > /usr/local/mariadb/bin/mysqld(_Z26mysql_create_table_no_lockP3THDPKcS2_P14HA_CREATE_INFOP10Alter_infoPbi+0xd3)[0x62ee13] > /usr/local/mariadb/bin/mysqld(_Z18mysql_create_tableP3THDP10TABLE_LISTP14HA_CREATE_INFOP10Alter_info+0xb6)[0x62ef76] > /usr/local/mariadb/bin/mysqld(_Z21mysql_execute_commandP3THD+0x672d)[0x5be04d] > /usr/local/mariadb/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x1d7)[0x5bf4f7] > /usr/local/mariadb/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x15d7)[0x5c1057] > /usr/local/mariadb/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x20c)[0x66f0ec] > /usr/local/mariadb/bin/mysqld(handle_one_connection+0x38)[0x66f138] > /usr/local/mariadb/bin/mysqld[0x9d5184] > /lib64/libpthread.so.0(+0x80db)[0x7fcf4ff230db] > /lib64/libc.so.6(clone+0x6d)[0x7fcf4ebd390d] > > Trying to get some variables. > Some pointers may be invalid and cause the dump to > abort. > Query (0x7fcf1143d020): CREATE TABLE menu_router ( BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] [GSoC] Week Nine!
by sriram patil 21 Jul '14

21 Jul '14

Hi All, There will be a little delay in the ninth week blog post. I need to get the code reviewed first to make the final changes. You can check the commit here <https://github.com/srirampatil/server/commit/3611bfc644714b797acff8b9884e7d…> though. I have also fixed MDEV-6409 <https://mariadb.atlassian.net/browse/MDEV-6409> and it is under review too. Regards, Sriram

1 0

[Maria-developers] Java Connector Coding till 20th July
by Puneet Dewan 20 Jul '14

20 Jul '14

Hi Everyone, Sorry,I did not update in the last week. It is because of my convocation and was meeting my parents after one year in Singapore, so enjoyed a small vacation of 5 days with them. I had code review and discussion with Massimo in that week. But I thought it is better to update next week. Things done: 1.I had initially written code in MySQLPreparedStatement.java but the code was getting ugly so as per recommnedation from Vladislav,I also discussed with my mentor.So we thought of using MySQLServerSidePreparedStatement as this will make code more cleaner. Now prepareStatement method in MySQLConnection will become the factory method to provide either MySQLPreparedStatement or MySQLServerSidePreparedStatement instance. I have migrated my code to MySQLServerSidePreparedStatement. 2.Test case for determining the type of instance, MySQLPreparedStatement or MySQLServerSidePreparedStatement. Next plan: 1.Write executeQuery and executeUpdate methods 2.For executeQuery, need to decide how I can use BinaryProtocolResultset to fetch results from a select query. 3.Write methods for batchstatements and multiqueries. Regards Puneet.

1 0

[Maria-developers] [GSoC] Optimize mysql-test-runs - Results of new strategy
by Pablo Estrada 17 Jul '14

17 Jul '14

Hello everyone, I took the last couple of days working on a new strategy to calculate the relevance of a test. The results are not sufficient by themselves, but I believe they point to an interesting direction. This strategy uses that rate of co-occurrence of events to estimate the relevance of a test, and the events that it uses are the following: - File editions since last run - Test failure in last run The strategy has also two stages: 1. Training stage 2. Executing stage In the training stage, it goes through the available data, and does the following: - If test A failed: - It counts and stores all the files that were edited since the last test_run (the last test_run depends on BRANCH, PLATFORM, and other factors) - If test A failed also in the previous test run, it also counts that In the executing stage, the training algorithm is still applied, but the decision of whether a test runs is based on its relevance, the relevance is calculated as the sum of the following: - The percentage of times a test has failed in two subsequent test_runs, multiplied by whether the test failed in the previous run (if the test didn't fail in the previous run, this quantity is 0) - For each file that was edited since the last test_run, the percentage of times that the test has failed after this file was edited (The explanation is a bit clumsy, I can clear it up if you wish so) The results have not been too bad, nor too good. With a running set of 200 tests, a training phase of 3000 test runs, and an executing stage of 2000 test runs, I have achieved recall of 0.50. It's not too great, nor too bad. Nonetheless, while running tests, I found something interesting: - I removed the first factor of the relevance. I decided to not care about whether a test failed in the previous test run. I was only using the file-change factor. Naturally, the recall decreased, from 0.50 to 0.39 (the decrease was not too big)... and the distribution of failed tests in the priority queue had a good skew towards the front of the queue (so it seems that the files help somewhat, to indicate the likelihood of a failure). I attached this chart. An interesting problem that I encountered was that about 50% of the test_runs don't have any file changes nor test failures, and so the relevance of all tests is zero. Here is where the original strategy (a weighted average of failures) could be useful, so that even if we don't have any information to guess which tests to run, we just go ahead and run the ones that have failed the most, recently. I will work on mixing up both strategies a bit in the next few days, and see what comes of that. By the way, I pushed the code to github. The code is completely different, so may be better to wait until I have new results soon. Regards! Pablo

2 11

[Maria-developers] Java Connector Coding till 5th June
by Puneet Dewan 17 Jul '14

17 Jul '14

Hi Everyone, The work done till 5th June: 1.I started studying the connector immediately after my semester exams got over(5th May). 2. Understood the relationship between different classes. 3.Read the MySql documentation for text protocol and map the commands from documentation onto code.This thing gave a clear idea about how things are moving from client to server. 4.Setup the code on Ubuntu using eclipse and tested against database. Setup the Team branch in launchpad. 5.Debug the test classes to see how different classes are interlinked and understood that MySQLProtocol.java is the heart of our JDBC driver. 6.Then I started reading Client Server Protocol documentation i.e ServerSide prepared statement and try to see how and where I have to code it. 7.Then I saw that there is a class MySqlServerSidePreparedStatement.java, Initially I coded the prepareStatement() ,close(),and execute() in that class seperately. But after (Code review) discussion with Massimo, we donot need to use that class. Instead use MySqlPreparedStatement.java to do the same work.So made changes accordingly. 8.I had pushed some of the code which I have written to launchpad and discussed with Massimo last week. 9.Last week I was struck with Wireshark on windows and ubuntu both. I was unable to see the packets going to database and seeing the response. 10.On Monday the problem was resolved as I was using the wrong interface.I should have used loopback.io(127.0.0.1) instead on ubuntu(thanks to Georg). Unfortunately the thing does not work on windows.Tried a lot many times. 11.Now I am working on the execute method of client server protocol as I am getting exception while using that method.Wireshark is playing a great role in how the packets are flowing. >From now I will do weekly blogging and keep everyone updated. Thanks & Regards Puneet.

2 9

[Maria-developers] Help needed TokuDB build errors in Debian
by Otto Kekäläinen 16 Jul '14

16 Jul '14

Hello Rich! For some strange reason, re-building 5.5.37 in Debian unstable stopped working. Any ideas what might be going on here?: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=753222 Recently a build of 5.5.38 generated a similar error: http://paste.ubuntu.com/7747594/ Are these maybe related? I have no idea how to fix these apart from disabling TokuDB in Debian when uploading 5.5.38. -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

3 5

[Maria-developers] [GSoC] Week Eight!
by sriram patil 14 Jul '14

14 Jul '14

Hi all, Following is the blog link about my eighth week into GSoC. http://goo.gl/h2mnbv Suggestions and/or reviews are welcome! Thanks, Sriram

1 0

Re: [Maria-developers] [Commits] Rev 4876: Post-fixes after UNIQUE index support was enabled without uniquenessenforcement. in file:///home/psergey/chroot/saucy-x64/home/psergey/dev2/mysql-5.6-rocksdb-r2/
by Roberto Spadim 11 Jul '14

11 Jul '14

=] nice rocksdb, any devel code to test it? i have some key-value databases to test 2014-07-11 13:21 GMT-03:00 Sergey Petrunya <psergey(a)askmonty.org>: > At > file:///home/psergey/chroot/saucy-x64/home/psergey/dev2/mysql-5.6-rocksdb-r2/ > > ------------------------------------------------------------ > revno: 4876 > revision-id: psergey(a)askmonty.org-20140711162114-051drr4tocqpbx30 > parent: psergey(a)askmonty.org-20140711161708-b50tdmlr60djrror > committer: Sergey Petrunya <psergey(a)askmonty.org> > branch nick: mysql-5.6-rocksdb-r2 > timestamp: Fri 2014-07-11 20:21:14 +0400 > message: > Post-fixes after UNIQUE index support was enabled without > uniquenessenforcement. > - Fix compile warning > - Fix the rocksdb.test/result > === modified file 'mysql-test/r/rocksdb.result' > --- a/mysql-test/r/rocksdb.result 2014-07-03 20:58:25 +0000 > +++ b/mysql-test/r/rocksdb.result 2014-07-11 16:21:14 +0000 > @@ -686,8 +686,8 @@ DROP TABLE t33; > # > # MDEV-4081: RocksDB throws error 122 on an attempt to create a table > with unique index > # > +# Unique indexes can be created, but uniqueness won't be enforced > create table t33 (pk int primary key, u int, unique index(u)) > engine=RocksDB; > -ERROR HY000: Got error 187 'Unique indexes are not supported' from ROCKSDB > # > # MDEV-4077: RocksDB: Wrong result (duplicate row) on select with range > # > > === modified file 'mysql-test/t/rocksdb.test' > --- a/mysql-test/t/rocksdb.test 2014-07-03 20:58:25 +0000 > +++ b/mysql-test/t/rocksdb.test 2014-07-11 16:21:14 +0000 > @@ -610,7 +610,8 @@ DROP TABLE t33; > --echo # > --echo # MDEV-4081: RocksDB throws error 122 on an attempt to create a > table with unique index > --echo # > ---error ER_GET_ERRMSG > +#--error ER_GET_ERRMSG > +--echo # Unique indexes can be created, but uniqueness won't be enforced > create table t33 (pk int primary key, u int, unique index(u)) > engine=RocksDB; > > > > === modified file 'storage/rocksdb/ha_rocksdb.cc' > --- a/storage/rocksdb/ha_rocksdb.cc 2014-07-09 23:46:24 +0000 > +++ b/storage/rocksdb/ha_rocksdb.cc 2014-07-11 16:21:14 +0000 > @@ -1051,7 +1051,7 @@ int ha_rocksdb::create(const char *name, > /* > We only support auto_increment at start of the PRIMARY KEY. > */ > - Field *field; > + //Field *field; > //if ((field= table_arg->next_number_field)) > /* TODO mdcallag: disable this for now to let UNIQUE indexes kind of > work > if ((field= table_arg->found_next_number_field)) > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits > -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

Re: [Maria-developers] apt-get upgrade on buildbot
by Kristian Nielsen 11 Jul '14

11 Jul '14

Otto Kekäläinen <otto(a)seravo.fi> writes: > Hello Kristian! > > Could you do a small favour and run 'apt-get upgrade' and then trigger > manually a rebuild in the virtual machine Debian unstable the buildbot > uses at http://buildbot.askmonty.org/buildbot/builders/debpkg-sid > > Thanks! Hm, I tried this, but I got the below errors. Any ideas what I did wrong? - Kristian. buildbot@debian-sid-amd64:~$ sudo apt-get update Hit http://ftp.us.debian.org sid InRelease Hit http://ftp.us.debian.org sid/main Sources/DiffIndex Hit http://ftp.us.debian.org sid/main amd64 Packages/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-en/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-cs/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ru/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-nl/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-eo/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ca/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-pl/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-fi/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-hr/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-fr/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-eu/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-sr/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-hu/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-da/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ro/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-es/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-pt/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-sv/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-sk/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ko/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-vi/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-uk/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-km/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-ja/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-de/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-it/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-nb/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-zh/DiffIndex Hit http://ftp.us.debian.org sid/main Translation-id/DiffIndex Get:1 http://ftp.us.debian.org sid/main Translation-ru [423 kB] Get:2 http://ftp.us.debian.org sid/main Translation-pl [575 kB] 100% [2 Translation-pl bzip2 0 B] 66.3 kB/s 0sFalling back to normal index file acquire Ign http://ftp.us.debian.org sid/main 2014-07-09-0854.18.pdiff Get:3 http://ftp.us.debian.org sid/main Translation-fr [731 kB] 100% [3 Translation-fr bzip2 0 B] 90.5 kB/s 0sFalling back to normal index file acquire Ign http://ftp.us.debian.org sid/main 2014-07-09-0854.18.pdiff Get:4 http://ftp.us.debian.org sid/main Translation-da [2,414 kB] Get:5 http://ftp.us.debian.org sid/main Translation-es [323 kB] Get:6 http://ftp.us.debian.org sid/main Translation-pt [444 kB] Get:7 http://ftp.us.debian.org sid/main Translation-sk [421 kB] Get:8 http://ftp.us.debian.org sid/main Translation-uk [776 kB] Get:9 http://ftp.us.debian.org sid/main Translation-ja [786 kB] Get:10 http://ftp.us.debian.org sid/main Translation-de [1,772 kB] Get:11 http://ftp.us.debian.org sid/main Translation-it [3,259 kB] Get:12 http://ftp.us.debian.org sid/main Sources [7,333 kB] Get:13 http://ftp.us.debian.org sid/main amd64 Packages [6,820 kB] Get:14 http://ftp.us.debian.org sid/main [116 kB] Get:15 http://ftp.us.debian.org sid/main [35.0 kB] Fetched 26.2 MB in 1min 31s (288 kB/s) W: Failed to fetch xz:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_source_Sources Hash Sum mismatch W: Failed to fetch xz:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_binary-amd64_Packages Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-ru Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-pl Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-fr Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-da Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-es Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-pt Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-sk Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-uk Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-ja Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-de Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-it Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-fi Hash Sum mismatch W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.us.debian.org_debian_dists_sid_main_i18n_Translation-hu Hash Sum mismatch E: Some index files failed to download. They have been ignored, or old ones used instead. buildbot@debian-sid-amd64:~$ sudo apt-get dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

2 2

[Maria-developers] [GSoC] Suggestions on adding a new method
by sriram patil 11 Jul '14

11 Jul '14

Hi All, I was thinking of adding two new inline methods in struct LEX as follows inline bool is_create_or_replace() { return (create_info.options & HA_LEX_CREATE_REPLACE); } inline bool is_create_if_not_exists() { return (create_info.options & HA_LEX_CREATE_IF_NOT_EXISTS); } With these, statements like if(lex->create_info.options & HA_LEX_CREATE_REPLACE) will be changed to if(lex->is_create_or_replace()) which definitely is more readable. This kind of statements are used in sql_yacc.yy, sql_parse.cc and many object specific implementation .cc files. Should I make the changes or is it unnecessary? Thanks, Sriram

1 0

Re: [Maria-developers] [JIRA] (MDEV-4937) NEED REVIEW: sql_slave_skip_counter does not work with GTID
by Kristian Nielsen 09 Jul '14

09 Jul '14

"Michael Widenius (JIRA)" <jira(a)mariadb.atlassian.net> writes: > Michael Widenius commented on MDEV-4937: > ---------------------------------------- Thanks for review Monty! I have some comments inline, let me know if you still want me to change something. - Kristian. > revno: 4260 > revision-id: knielsen at knielsen-hq.org-20140620104939-xdvn985cgmwy3odd > committer: Kristian Nielsen <knielsen at knielsen-hq.org> > timestamp: Fri 2014-06-20 12:49:39 +0200 > message: > MDEV-4937: sql_slave_skip_counter does not work with GTID > === modified file 'sql/slave.cc' > --- a/sql/slave.cc 2014-06-19 12:50:43 +0000 > +++ b/sql/slave.cc 2014-06-20 10:49:39 +0000 > @@ -3521,9 +3521,6 @@ static int exec_relay_log_event(THD* thd > > if (opt_gtid_ignore_duplicates) > { > - serial_rgi->current_gtid.domain_id= gev->domain_id; > - serial_rgi->current_gtid.server_id= gev->server_id; > - serial_rgi->current_gtid.seq_no= gev->seq_no; > int res= rpl_global_gtid_slave_state.check_duplicate_gtid > (&serial_rgi->current_gtid, serial_rgi); > if (res < 0) > What was the reason for removing the above assignments? > Was it a bug in the old code ? Not really a bug, however, just before this code we have: if (event_group_new_gtid(serial_rgi, gev)) and event_group_new_gtid() does the exact same assignments, so the ones I removed are completely redundant. I just happened to notice this redundancy, so I removed them... > (I have to ask as I am not 100% of what the old code did). > @@ -4594,16 +4598,27 @@ log '%s' at position %s, relay log '%s' > > if (saved_skip && rli->slave_skip_counter == 0) > { > + String tmp; > + if (mi->using_gtid != Master_info::USE_GTID_NO) > + { > + tmp.append(STRING_WITH_LEN(", GTID '")); > + rpl_append_gtid_state(&tmp, false); > + tmp.append(STRING_WITH_LEN("'; ")); > + } > > {noformat} > > If you want to avoid checking if tmp.append() fails, you could > create tmp with a static buffer. > > char tmp_buff[MAX_GTID_LENGTH+10]; > String tmp(tmp_buff, sizeof(tmp_buff), &my_charset_bin); > tmp.length(0); Right, hm. It does not really feel right to enforce a limit on the length; if the user uses a longer GTID position (which would be rather unusual), it might be confusing to have it silently cut. If the append() should fail, well, then it _will_ be cut, but then it's just an informational message, and that kind of out-of-memory is probably unlikely (and the error log will likely be filled up with other complaints about out-of-memory). But let me know if you still think I should change it. (If so - can you think of a place where MAX_GTID_LENGTH can be defined, that would be accessible to both client/mysqldump.c and sql/slave.cc? I could spot only my_global.h, which seems a bit too generic to contain GTID-related stuff...?) > if (saved_skip && rli->slave_skip_counter == 0) > { > String tmp; > saved_skip= 0; > + saved_skip_gtid_pos.free(); > } > If you free saved_skip_gtid_pos, it good to also free tmp! Right ... but it should be freed automatically anyway, when tmp goes out of scope, right? > === modified file 'sql/sys_vars.cc' > --- a/sql/sys_vars.cc 2014-06-09 18:00:23 +0000 > +++ b/sql/sys_vars.cc 2014-06-20 10:49:39 +0000 > @@ -4287,11 +4287,6 @@ bool update_multi_source_variable(sys_va > > static bool update_slave_skip_counter(sys_var *self, THD *thd, Master_info *mi) > { > - if (mi->using_gtid != Master_info::USE_GTID_NO) > - { > - my_error(ER_SLAVE_SKIP_NOT_IN_GTID, MYF(0)); > - return true; > - } > > {noformat} > > Shouldn't you also remove the error ER_SLAVE_SKIP_NOT_IN_GTID from the > error file? I think we cannot remove error codes from a GA release once introduced, right? So I left it, to not shift other error numbers. But let me know if I'm wrong, and I'll remove it. - Kristian.

1 0

Re: [Maria-developers] [JIRA] (MDEV-6336) NEED REVIEW: mysqldump --master-data does not work with GTID setups
by Kristian Nielsen 09 Jul '14

09 Jul '14

"Michael Widenius (JIRA)" <jira(a)mariadb.atlassian.net> writes: > Here is the review: > > ------------------------------------------------------------ > revno: 4245 > revision-id: knielsen at knielsen-hq.org-20140616093803-gub60pda6t77ch3k > committer: knielsen at knielsen-hq.org > timestamp: Mon 2014-06-16 11:38:03 +0200 > message: > MDEV-6336: mysqldump --master-data does not work with GTID setups > MDEV-6344: mysqldump issues FLUSH TABLES, which gets written into binlog and replicated Thanks for review Monty! I fixed all of your suggestions. - Kristian.

1 0

Re: [Maria-developers] [Commits] Rev 4281: MDEV-5262, MDEV-5914, MDEV-5941, MDEV-6020: Deadlocks during parallel replication causing replication to fail. in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Kristian Nielsen 09 Jul '14

09 Jul '14

Jan Lindström <jan.lindstrom(a)skysql.com> writes: > I have mostly InnoDB style comments, but some questions below: Thanks for comments! Sorry about style violations, and thanks for pointing then out. I'll try to fix all of it. Answers to other questions below: > Why you need to define these, why #include "mysql/plugin.h" file is not > enough ? > > +extern "C" void thd_report_wait_for(const MYSQL_THD thd, MYSQL_THD > other_thd); > +extern "C" int thd_need_wait_for(const MYSQL_THD thd); > +extern "C" > +int thd_need_ordering_with(const MYSQL_THD thd, const MYSQL_THD other_thd); See the discussions here: https://lists.launchpad.net/maria-developers/msg07494.html The conclusion with Serg was that doing a proper public API in 10.0 was too big a change. We will do it in 10.1 (MDEV-6429). plugin.h should not be modified in a GA version and with an API that has not been thought through wrt. all engines. Also, the API should be a service, not in plugin.h. > Why you need to introduce a new global structure for this, can't this be > part > of some already existing system structure e.g. lock_sys and why you do not > use ut_list ? > > +/* Buffer to collect THDs to report waits for. */ > +struct thd_wait_reports { > + struct thd_wait_reports *next; > + ulint used; > + trx_t *waitees[64]; > +}; So the idea here is to collect a bunch of trx pointers during the graph traversal, and pass each of them to thd_report_wait_for(). I wanted to pass them after completing the graph traversal to avoid issues if thd_report_wait_for() needs to kill one of them; this goes to lock_cancel_waiting_and_release() eventually which I assume might modify the graph being traversed. This is time-critical code (it runs under global lock_sys->mutex), so it is important to avoid memory allocations. So I use an array allocated on the stack, this is really short-lived data, it only lives for the duration of lock_deadlock_check_and_resolve(). In all normal cases the single array should be sufficient, but the code needs to handle the overflow also, of course. So in case of overflow, I malloc further arrays and link them into the first. I suppose another solution would be to abort in case of overflow and roll back the transaction like in case of ctx->too_deep==true, however I don't like to introduce such arbitrary limits when it's not necessary. I looked into using ib_list, but this seems to always do mallocs, which I want to avoid. I also looked at ut0vec and dyn0dyn, but they also seem to always use malloc and thus do not seem appropriate. So I did not find anything existing I could use, unfortunately, but then I know little of the InnoDB code. Do you have a suggestion for an existing data structure to use that can take an inital buffer on the stack and malloc extra memory only if needed? Thanks for comments! - Kristian.

1 0

Re: [Maria-developers] [Commits] Rev 4275: MDEV-5867 ALTER TABLE t1 ENGINE=InnoDB keeps bad options when t1 ENGINE is CONNECT in lp:~maria-captains/maria/10.0
by Sergey Vojtovich 08 Jul '14

08 Jul '14

Hi Sergei, I believe options handling is way too confusing in general: - CREATE TABLE warns/errors about unknown options - ALTER TABLE ... ENGINE doesn't - ALTER TABLE ... UNKNOWN_OPTION=1 does - CREATE TABLE ... LIKE doesn't - different engines may assign different meaning to the same option - IGNORE_BAD_TABLE_OPTIONS doesn't actually _ignore_ _bad_ options but rather _accepts_ _unkown_ options But the question is if we should show unknown options by default. I have no opinion: both cases are equally wrong to me. On the one hand we get unusable CREATE TABLE statement, on the other hand hidden metadata which may affect further statements. I'm fine with hidding unknown options that were accepted due to IGNORE_BAD_TABLE_OPTIONS, but hidding things that were accepted by fully successful ALTER TABLE statement sounds confusing. Regards, Sergey On Sat, Jul 05, 2014 at 11:42:25PM +0200, Sergei Golubchik wrote: > At lp:~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4275 > revision-id: sergii(a)pisem.net-20140705214225-p0k45asdaq27xocr > parent: sergii(a)pisem.net-20140705214216-npqtc7tz854xqjxo > fixes bug: https://mariadb.atlassian.net/browse/MDEV-5867 > committer: Sergei Golubchik <sergii(a)pisem.net> > branch nick: 10.0 > timestamp: Sat 2014-07-05 23:42:25 +0200 > message: > MDEV-5867 ALTER TABLE t1 ENGINE=InnoDB keeps bad options when t1 ENGINE is CONNECT > > don't print unknown options in SHOW CREATE TABLE unless IGNORE_BAD_TABLE_OPTIONS is used > === added file 'mysql-test/r/table_options-5867.result' > --- a/mysql-test/r/table_options-5867.result 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/r/table_options-5867.result 2014-07-05 21:42:25 +0000 > @@ -0,0 +1,37 @@ > +install soname 'ha_example'; > +set sql_mode='ignore_bad_table_options'; > +create table t1 ( > +a int complex='c,f,f,f' invalid=3 > +) engine=example ull=10000 str='dskj' one_or_two='one' yesno=0 > +foobar=barfoo; > +Warnings: > +Warning 1911 Unknown option 'invalid' > +Warning 1911 Unknown option 'foobar' > +create table t2 (a int, key (a) some_option=2014); > +Warnings: > +Warning 1911 Unknown option 'some_option' > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL `complex`='c,f,f,f' `invalid`=3 > +) ENGINE=EXAMPLE DEFAULT CHARSET=latin1 `ull`=10000 `str`='dskj' `one_or_two`='one' `yesno`=0 `foobar`=barfoo `VAROPT`='5' > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `a` int(11) DEFAULT NULL, > + KEY `a` (`a`) `some_option`=2014 > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +set sql_mode=''; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL `complex`='c,f,f,f' > +) ENGINE=EXAMPLE DEFAULT CHARSET=latin1 `ull`=10000 `str`='dskj' `one_or_two`='one' `yesno`=0 `VAROPT`='5' > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `a` int(11) DEFAULT NULL, > + KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +drop table t1, t2; > +uninstall soname 'ha_example'; > > === modified file 'mysql-test/suite/rpl/r/rpl_table_options.result' > --- a/mysql-test/suite/rpl/r/rpl_table_options.result 2013-09-14 01:09:36 +0000 > +++ b/mysql-test/suite/rpl/r/rpl_table_options.result 2014-07-05 21:42:25 +0000 > @@ -12,6 +12,12 @@ show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > `a` int(11) NOT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +set sql_mode=ignore_bad_table_options; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) NOT NULL > ) ENGINE=MyISAM DEFAULT CHARSET=latin1 `ull`=12340 > drop table t1; > set storage_engine=default; > > === modified file 'mysql-test/suite/rpl/t/rpl_table_options.test' > --- a/mysql-test/suite/rpl/t/rpl_table_options.test 2011-10-19 19:45:18 +0000 > +++ b/mysql-test/suite/rpl/t/rpl_table_options.test 2014-07-05 21:42:25 +0000 > @@ -23,6 +23,8 @@ show create table t1; > sync_slave_with_master; > connection slave; > show create table t1; > +set sql_mode=ignore_bad_table_options; > +show create table t1; > > connection master; > drop table t1; > > === added file 'mysql-test/t/table_options-5867.test' > --- a/mysql-test/t/table_options-5867.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/table_options-5867.test 2014-07-05 21:42:25 +0000 > @@ -0,0 +1,30 @@ > +# > +# MDEV-5867 ALTER TABLE t1 ENGINE=InnoDB keeps bad options when t1 ENGINE is CONNECT > +# > +# verify that SHOW CREATE TABLE hides unknown options when IGNORE_BAD_TABLE_OPTIONS is not set > + > +--source include/have_example_plugin.inc > +--source include/not_embedded.inc > + > +install soname 'ha_example'; > + > +set sql_mode='ignore_bad_table_options'; > +create table t1 ( > + a int complex='c,f,f,f' invalid=3 > +) engine=example ull=10000 str='dskj' one_or_two='one' yesno=0 > + foobar=barfoo; > + > +create table t2 (a int, key (a) some_option=2014); > + > +show create table t1; > +show create table t2; > + > +set sql_mode=''; > + > +show create table t1; > +show create table t2; > + > +drop table t1, t2; > + > +uninstall soname 'ha_example'; > + > > === modified file 'sql/create_options.cc' > --- a/sql/create_options.cc 2014-03-26 08:33:03 +0000 > +++ b/sql/create_options.cc 2014-07-05 21:42:25 +0000 > @@ -775,3 +775,20 @@ engine_option_value *merge_engine_table_ > &first, &end); > DBUG_RETURN(first); > } > + > +bool is_engine_option_known(engine_option_value *opt, > + ha_create_table_option *rules) > +{ > + if (!rules) > + return false; > + > + for (; rules->name; rules++) > + { > + if (!my_strnncoll(system_charset_info, > + (uchar*)rules->name, rules->name_length, > + (uchar*)opt->name.str, opt->name.length)) > + return true; > + } > + return false; > +} > + > > === modified file 'sql/create_options.h' > --- a/sql/create_options.h 2013-11-20 11:05:39 +0000 > +++ b/sql/create_options.h 2014-07-05 21:42:25 +0000 > @@ -99,4 +99,6 @@ uchar *engine_table_options_frm_image(uc > > bool engine_options_differ(void *old_struct, void *new_struct, > ha_create_table_option *rules); > +bool is_engine_option_known(engine_option_value *opt, > + ha_create_table_option *rules); > #endif > > === modified file 'sql/sql_show.cc' > --- a/sql/sql_show.cc 2014-07-05 21:42:16 +0000 > +++ b/sql/sql_show.cc 2014-07-05 21:42:25 +0000 > @@ -1519,13 +1519,20 @@ static bool get_field_default_value(THD > @param thd thread handler > @param packet string to append > @param opt list of options > + @param check_options only print known options > + @param rules list of known options > */ > > static void append_create_options(THD *thd, String *packet, > - engine_option_value *opt) > + engine_option_value *opt, > + bool check_options, > + ha_create_table_option *rules) > { > for(; opt; opt= opt->next) > { > + if (check_options && !is_engine_option_known(opt, rules)) > + continue; > + > DBUG_ASSERT(opt->value.str); > packet->append(' '); > append_identifier(thd, packet, opt->name.str, opt->name.length); > @@ -1584,6 +1591,8 @@ int show_create_table(THD *thd, TABLE_LI > MODE_MAXDB | MODE_ANSI); > bool limited_mysql_mode= sql_mode & (MODE_NO_FIELD_OPTIONS | MODE_MYSQL323 | > MODE_MYSQL40); > + bool check_options= !(sql_mode & MODE_IGNORE_BAD_TABLE_OPTIONS) && > + !create_info_arg; > handlerton *hton; > my_bitmap_map *old_map; > int error= 0; > @@ -1733,7 +1742,8 @@ int show_create_table(THD *thd, TABLE_LI > packet->append(STRING_WITH_LEN(" COMMENT ")); > append_unescaped(packet, field->comment.str, field->comment.length); > } > - append_create_options(thd, packet, field->option_list); > + append_create_options(thd, packet, field->option_list, check_options, > + hton->field_options); > } > > key_info= table->key_info; > @@ -1800,7 +1810,8 @@ int show_create_table(THD *thd, TABLE_LI > append_identifier(thd, packet, parser_name->str, parser_name->length); > packet->append(STRING_WITH_LEN(" */ ")); > } > - append_create_options(thd, packet, key_info->option_list); > + append_create_options(thd, packet, key_info->option_list, check_options, > + hton->index_options); > } > > /* > @@ -1953,7 +1964,8 @@ int show_create_table(THD *thd, TABLE_LI > packet->append(STRING_WITH_LEN(" CONNECTION=")); > append_unescaped(packet, share->connect_string.str, share->connect_string.length); > } > - append_create_options(thd, packet, share->option_list); > + append_create_options(thd, packet, share->option_list, check_options, > + hton->table_options); > append_directory(thd, packet, "DATA", create_info.data_file_name); > append_directory(thd, packet, "INDEX", create_info.index_file_name); > } > @@ -5130,7 +5142,7 @@ static int get_schema_tables_record(THD > str.qs_append(STRING_WITH_LEN(" transactional=")); > str.qs_append(ha_choice_values[(uint) share->transactional]); > } > - append_create_options(thd, &str, share->option_list); > + append_create_options(thd, &str, share->option_list, false, 0); > > if (str.length()) > table->field[19]->store(str.ptr()+1, str.length()-1, cs); > > === modified file 'storage/connect/mysql-test/connect/r/alter.result' > --- a/storage/connect/mysql-test/connect/r/alter.result 2014-03-23 14:50:39 +0000 > +++ b/storage/connect/mysql-test/connect/r/alter.result 2014-07-05 21:42:25 +0000 > @@ -218,13 +218,22 @@ Three 3 > # Changing to another engine is Ok > # However, the data file is not deleted. > # > -ALTER TABLE t1 ENGINE=MARIA; > +ALTER TABLE t1 ENGINE=ARIA; > +SHOW CREATE TABLE t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` char(10) NOT NULL, > + `c` int(11) NOT NULL > +) ENGINE=Aria DEFAULT CHARSET=latin1 PAGE_CHECKSUM=1 > +set @old_sql_mode=@@sql_mode; > +set sql_mode=ignore_bad_table_options; > SHOW CREATE TABLE t1; > Table Create Table > t1 CREATE TABLE `t1` ( > `d` char(10) NOT NULL `FLAG`=11, > `c` int(11) NOT NULL `FLAG`=0 > ) ENGINE=Aria DEFAULT CHARSET=latin1 PAGE_CHECKSUM=1 `TABLE_TYPE`=fix `FILE_NAME`='tf1.txt' `ENDING`=1 > +set sql_mode=@old_sql_mode; > SELECT * from t1; > d c > One 1 > > === modified file 'storage/connect/mysql-test/connect/t/alter.test' > --- a/storage/connect/mysql-test/connect/t/alter.test 2014-03-23 14:50:39 +0000 > +++ b/storage/connect/mysql-test/connect/t/alter.test 2014-07-05 21:42:25 +0000 > @@ -105,8 +105,12 @@ SELECT * FROM t1; > --echo # Changing to another engine is Ok > --echo # However, the data file is not deleted. > --echo # > -ALTER TABLE t1 ENGINE=MARIA; > +ALTER TABLE t1 ENGINE=ARIA; > SHOW CREATE TABLE t1; > +set @old_sql_mode=@@sql_mode; > +set sql_mode=ignore_bad_table_options; > +SHOW CREATE TABLE t1; > +set sql_mode=@old_sql_mode; > SELECT * from t1; > SELECT * from t2; > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

4 7

[Maria-developers] Java Connector coding till 6th July
by Puneet Dewan 08 Jul '14

08 Jul '14

Hi Everyone, Sorry for the late mail. Last week I had code review with Massimo. I wrote the test cases for transactions. Then I clean the code written in MySQLProtocol.java for executePreparedStatement method. Then added new methods like writeBinaryTo(for BinaryProtocol) in IntParameter and will write similar methods in other classes. In the coming week: *a)*I will be making use of MYSQLServerSidePreparedStatement.java for performing the all the operations.Currently they are embedded in MYSQLPreparedStatement.java *b)*I have done the analysis of the above as to what classes will be changed. *c)*Also I will be modifying the execute method for Serverside prepared statements for sending the *new params bound flag* properly. Currently it is always going as 0x01 but will modify it to send 0x00 if same statement gets executed again I will have code review with Massimo on thursday.. Your suggestions are welcome. Regards Puneet.

2 2

Re: [Maria-developers] comments to the plugin.h
by Kristian Nielsen 08 Jul '14

08 Jul '14

Sergei Golubchik <vuvova(a)gmail.com> writes: > This is what I've written in the patch: I've tried to answer all points in detail, also to help myself better understand all the issues: >> + The storage engine can report such conflicting locks using this call. This >> + will allow parallel replication to detect such conflicts and resolve the >> + deadlock (by killing the second transaction to release the locks that the >> + first is waiting for, and then later re-try the second killed transaction). > > What will happen if the engine doesn't? Deadlocks will timeout. > Which is annoying, but it only affects the performance, not the > correctness. No, unfortunately that is not so. Let's say we have T1 and T2. T1 goes to wait on a row lock held by T2. Then T2 completes its prepare phase, it goes to the group commit queue and waits for T1 to complete and commit for both of them. T1 gets lock wait timeout, so T1 will roll back and try again. But T1 will just go to wait on the same row lock again that is held by T2. So T1 will retry over and over until slave_transaction_retries is exceeded, at which point replication will abort with an error. There is no lock wait timeout when T2 waits in the group commit queue. > But can it cause the first transaction to be killed, not > the second? Then it's worse - it'll violate the strict commit ordering. No, this can never happen. Commit order is enforced by the wait_for_commit facility. T2 will not be allowed to commit before T1. If T1 is killed and retried, T2 will continue waiting for T1 to commit the second time. And if T1 fails, it will call wakeup_subsequent_commits() with an error, so that T1 can abort its commit. > And, again, the engine needs to do it for every lock wait. Even while most of > them won't cause any deadlocks. Right. This is only needed for parallel replication worker threads. That is why I introduced thd_need_wait_for(). The storage engine can call this once for a transaction or even for a THD, and save the result. And then it won't need to call thd_report_wait_for() except for the parallel replication worker threads. And for those, lock waits should be rare. >> + The storage engine should not report false positives. That is, it should not >> + report any lock waits that do not actually require one transaction to wait >> + for the other. Nor should it report waits for locks that will be released >> + before the commit of the other transactions. > > What will happen if the engine reports false positives? Lock waist > that don't require one transaction to wait for another. What if it > report waits for locks that will be released before commit of the other > transaction? If it reports that T1 will wait for T2, then we will kill T2 and retry it. This will cause unnecessary work if there is no actual deadlock, so mainly a performance issue. A deeper problem is caused by the async kill processing. If T1 is reported as waiting for T2, we will kill T2 in the background thread. But if there is no such wait, or if the wait is resolved due to T2 releasing locks before commit, then T1 and T2 could both complete, and their worker threads move on to new transactions. Then the background thread will be trying to kill a T2 which no longer exists, which will access incorrect rpl_group_info objects. This is the part of the patch I am most unhappy about :-/ But we started to discuss on IRC to possibly avoid the async kill, instead making it possible to call THD::awake() directly inside thd_report_wait_for(). This would be good, as it would solve the deeper problem. In this case, a false positive would then cause only performance issues, I think, though I should think the problem through carefully to be sure. >> +void thd_report_wait_for(const MYSQL_THD thd, MYSQL_THD other_thd); > > You kill the second thread in a background. > What if you convey the ordering to the engine and let it know about a > deadlock? Like, if you return 1 from thd_report_wait_for() > it means a deadlock and the engine will kill the transaction > just normally as for any normal deadlock case. The problem is that T2 could be executing anywhere in the server, including server code or another storage engine, where InnoDB does not really have any possibility to kill it. This is different from a normal InnoDB deadlock, which only happens if both T1 and T2 are doing an actual row lock wait inside the storage engine itself. But in this case, only T1 is waiting, T2 is continuing whatever processing it is doing, eventually it will end up in the group commit queue in binlog or in wait_for_prior_commit(), and wait for T1 to commit first. >> +int thd_need_wait_for(const MYSQL_THD thd); > > I'd suggest to remove it. If you think it's useful, you can call it from > your thd_report_wait_for() and spare storage engine the knowledge of yet > another API function. See discussion above. The intention is to avoid lots of calls to thd_report_wait_for() from the storage engine for non-replication threads. So calling it from inside thd_report_wait_for() makes no sense. I don't understand the benefits of reducing the functionality available in the API when that functionality is completely optional? But on the other hand, I do not feel strongly about this function. I do not even use it in the patch, as InnoDB does a rather expensive graph traversal of all waiters anyway in its deadlock detector, so it can be just omitted. I included this function because without it, there seems no way for a storage engine to avoid introducing O(N**2) overhead for N transactions contending for a row lock. That seemed unacceptable to me. >> +int thd_need_ordering_with(const MYSQL_THD thd, const MYSQL_THD other_thd); > > Hmm, really? Couldn't you simply disable gap locks in some other already > existing way? Like decrease the transaction isolation level or > tell the engine that it's RBR? No, that is not safe. Suppose we have two transactions: T1: INSERT INTO t1 VALUES (1); T2: INSERT INTO t2 SELECT * FROM t1; T1 and T2 are not run in parallel, because they cannot group commit together on the master. However, on the slave, as soon as T1 _starts_ to commit, T2 is allowed to start. But T2 cannot see the changes done by T1 until T1 _completes_ its commit. Thus it is critical that gap locks are disabled _only_ among transactions within the same group commit, not between different group commits. In fact there is a bug in the patch you got regarding this, as it did not check the commit_id before relaxing gap locks. I currently have the following extra patch in my tree to fix this: === modified file 'sql/sql_class.cc' --- sql/sql_class.cc 2014-06-10 08:13:15 +0000 +++ sql/sql_class.cc 2014-06-27 11:47:53 +0000 @@ -4277,6 +4277,8 @@ thd_need_ordering_with(const MYSQL_THD t return 1; if (rgi->current_gtid.domain_id != other_rgi->current_gtid.domain_id) return 1; + if (rgi->commit_id != other_rgi->commit_id) + return 1; /* These two threads are doing parallel replication within the same replication domain. Their commit order is already fixed, so we do not need On the other hand, if we were to omit this part of the patch completely and just keep the gap locks always, then things would still work, because thd_report_wait_for() would detect a deadlock and do a retry to resolve it. But we would get extra rollbacks. Maybe such extra rollbacks would be rare, though. >> +int thd_deadlock_victim_preference(const MYSQL_THD thd1, const MYSQL_THD thd2); >> + > > Okay, I see why this one is needed. > > On the other hand... > It seems like it's the only function that you realy need. > Because > > thd_report_wait_for(const MYSQL_THD thd, MYSQL_THD other_thd); > > can be replaced with > > if (thd_deadlock_victim_preference(thd, other_thd) < 0) goto deadlock; > > And > > thd_need_wait_for(const MYSQL_THD thd); > > can be removed (replaced with noop). And > > thd_need_ordering_with(const MYSQL_THD thd, const MYSQL_THD other_thd); > > can be replaced with > > thd_deadlock_victim_preference(thd, other_thd) != 0 I don't understand the purpose of using the same function for these? They do completely different things, using a single function will only cause confusion, it seems to me: 1. thd_report_wait_for() is used to allow the _replication code_ to detect a deadlock related to ordering. 2. thd_deadlock_victim_preference() is only used when the _storage engine_ detects a deadlock, replication related or not. 3. thd_need_ordering_with() is used by the storage engine to decide on lock wait semantics. > Although in that case thd_deadlock_victim_preference() should better be > renamed, because it won't be a generic "victim preference", > but specifically victim preference, depending on the fixed transaction > commit ordering. A better name could be, for example, > > thd_get_commit_order(thd, other_thd) > -1 - thd first > 0 - unspecified > +1 - other_thd first How can the storage engine know the difference between T1 and T2 being in the same group commit, or in different group commits? How can the storage engine inquire about need for gap locks, without accidentally reporting a lock wait to the upper layer? > I think it'll be much simpler with only one function. Why? I think it's quite the opposite... > And also it becomes clear what happens if the engine doesn't use it. > This function is used to enquire about the commit order, if it is > not used - the commit order cannot be guaranteed. No, that is not the case. On the contrary - the storage engine is free not to use this, commit order will be guaranteed in any case by wait_for_commit(). Hm. So my idea was to make the new API simple and generic, the upper layer would handle most of the detail about commit ordering or whatever. The storage engine only needs to think about simple stuff like lock waits or deadlock victims, both of which it likely already needs to deal with anyway. But if I understand correctly, another idea could be to make the new API specific just to this particular problem of enforced commit order in parallel replication. So the storage engine would be able to know of such ordering, and should then handle any necessary actions on this by itself? > And you won't need background killing for slave threads The need for background killing comes because it does not work to call THD::awake() from the context of where the InnoDB deadlock detector calls thd_report_wait_for(). The problem is that innobase_kill_query() does a recursive mutex lock on lock_sys->mutex. We discussed on IRC to fix this so background kill is not necessary. One way would be to document that hton->kill_query must be safe to call from within thd_report_wait_for(). Maybe that is the simplest way. It should not be hard to fix the InnoDB code to make this work. For example, InnoDB could just set a flag in the transaction that it already has lock_sys->mutex held, thus avoiding the recursive mutex lock. You also had a different suggestion: > here's another idea. To have thd_get_commit_order() as I thought originally > *and* allow innodb to kill other threads, that is, add a function for > plugins to invoke THD::awake. This function should not call into the engine > that invoked it. That is, supposedly the engine knows how to kill threads > stuck in that engine, to kill other threads it should use this new helper. > In that case innodb can detect the deadlock and can kill the other thread > and it won't break on lock mutex > The underlying logic of this killer helper would be "an engine knows how to > kill a transaction waiting in this engine, so we wouldn't bother doing > that. but to kill a transaction waiting elsewhere the engine needs help" It seems better to me if all the details of enforcing commit order and killing worker threads in case of deadlocks could be kept inside the replication code? Otherwise every storage engine would potentially need to re-implement this? I think that is what I tried to do with my proposed storage engine API, to cleanly separate the storage engine's responsibilities, which is about row-level locks and internal deadlock victims, from the replication codes responsibilities. Wouldn't it be simpler to do it like this? 1. InnoDB deadlock detector reports lock waits to whatever function serves the thd_report_wait_for() purpose. 2. We document that thd_report_wait_for() may need to call hton->kill_query(). 3. We fix InnoDB to handle (2) correctly (assuming this is possible to do in a simple way). > And anyway, this absolutely must be in a service, not in the global plugin.h. Why? My understanding was that we do not support ABI stability for storage engines. (And I think we should absolutely not provide such support for storage engines. Storage engines are all about performance and flexibility - ABI stability will make this harder to provide without sufficient other benefits to justify this. ABI seems mainly suited for eg. authentication plugins). > See? In this patch you've removed a function from the global plugin API Right, the thd_rpl_is_parallel(). That should have never be allowed into the code in the first place, it was a temporary hack only. :-( > header, strictly speaking, you must increase the API version - the major > number of - making all older plugins incompatible with the new server. This > is a bit extreme, taking into account that no plugin actually uses > thd_rpl_is_parallel() > To avoid this, please only add new APIs as services. > See libservices/HOWTO I can do that, if you want. It just seems a needless complication to me... Are there any performance penalties for using a service over a normal function call (for built-in storage engines)? I seem to recall there is not? - Kristian.

2 6

Re: [Maria-developers] [Commits] Rev 4274: small cleanup of the SHOW CREATE TABLE code in lp:~maria-captains/maria/10.0
by Sergey Vojtovich 08 Jul '14

08 Jul '14

Hi Sergei, looks good. One question inline. On Sat, Jul 05, 2014 at 11:42:16PM +0200, Sergei Golubchik wrote: > At lp:~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4274 > revision-id: sergii(a)pisem.net-20140705214216-npqtc7tz854xqjxo > parent: sergii(a)pisem.net-20140704190257-1z1inye8jd7k7umc > committer: Sergei Golubchik <sergii(a)pisem.net> > branch nick: 10.0 > timestamp: Sat 2014-07-05 23:42:16 +0200 > message: > small cleanup of the SHOW CREATE TABLE code ...skip... > === modified file 'sql/sql_show.cc' > --- a/sql/sql_show.cc 2014-06-05 22:07:27 +0000 > +++ b/sql/sql_show.cc 2014-07-05 21:42:16 +0000 > @@ -1578,27 +1577,32 @@ int store_create_info(THD *thd, TABLE_LI > handler *file= table->file; > TABLE_SHARE *share= table->s; > HA_CREATE_INFO create_info; > -#ifdef WITH_PARTITION_STORAGE_ENGINE > bool show_table_options= FALSE; > -#endif /* WITH_PARTITION_STORAGE_ENGINE */ Why did you remove ifdef-s here? This variable seem to be unused without partitioning. ...skip... Regards, Sergey

2 1

[Maria-developers] doubt about commit and myisam/aria engine
by Roberto Spadim 07 Jul '14

07 Jul '14

hi guys, i'm with a doubt... when a aria or myisam file is fsync to disk? when i execute commit? when i execute a autocommit? -- Roberto Spadim SPAEmpresarial

1 0

[Maria-developers] [GSoC] Week Seven!
by sriram patil 07 Jul '14

07 Jul '14

Hi all, Following is the blog link about my seventh week into GSoC. http://goo.gl/jBeubY Suggestions and/or reviews are welcome! Thanks, Sriram

1 0

[Maria-developers] comments on parallel applying
by Jonas Oreland 04 Jul '14

04 Jul '14

hi there, disclaimer: i haven't followed all turns and details on this task so sorry if my comments are outdated (or generally uninteresting). --- i've been glancing on the review comments on the parallel applying, and have some comments to make. i think it will be well invested time to add infrastructure to be able to restrict when parallel applying will be performed. typically i think it would be nice to - only allow parallel transaction in same engine - decrease parallelism when performing DDL ...(more of these can probably be added) restrictions that will make it easier to reason about correctness and that will not affect performance for 99% of the cases (i think :-) if adding such restrictions was an option, the resolving review comments on weird corner cases would be much easier. --- from what I can tell, there are currently no such restrictions, and no infrastructure to add them either. i realize that it might not be trivial to introduce such restrictions, but personally i think it will be worth the effort to get a solid solution in place. <quick thoughts on implementation> for row-based replication this seems quite "easy". for statement-based replication i image that you would have to add hooks into the "real" code after parsing has been performed, but before the actual execution is started (and yes, i know that there is sometimes a blurry line here) </thoughts> --- /Jonas

3 4

[Maria-developers] Review request for MDEV-6020, MDEV-5914, MDEV-5941, and MDEV-5262
by Kristian Nielsen 04 Jul '14

04 Jul '14

Hi Serg, We discussed on IRC a review of the $subj bugs related to deadlocks in parallel replication and handling of automatic retry of transactions in case of deadlock. The patches are in my tree: lp:~maria-captains/maria/10.0-knielsen The complete diff can be obtained eg. with: bzr diff -rrevid:psergey@askmonty.org-20140429143317-tkmxrzu6y36hpi7t..revid:knielsen@knielsen-hq.org-20140610081315-o5nx3u32uvqaai7q Let me give some background for these bugs: When two transactions T1 and T2 group-commit on the master, we assume that they have no lock conflicts (otherwise one would have had to wait for the other to commit). So we run them in parallel on the slave. But before we let T2 commit, we wait until T1 has committed first, so that we get the same binlog order on the slave as we had on the master. But if T1 and T2 somehow _do_ get a lock conflict on the slave, and T1 ends up waiting for some lock held by T2, we get a deadlock. This causes the replication to hang (the wait of T2 for T1 to commit is not part of InnoDB deadlock detection), which is quite bad. Elena's testing of the parallel replication in fact uncovered a number of cases where such deadlock issues occur: 1. MDEV-5914. It turns out that InnoDB gap locks can work in a non-symmetric way. If T1 is an INSERT, and T2 is a DELETE, then T1 can run first and not block T2. But if T2 runs first, then it may set a gap lock that blocks T1. Thus, depending on exact execution order, we can get T1 and T2 to group commit on the master, but T2 block T1 on the slave and cause a deadlock. 2. MDEV-5941. If eg. an UPDATE uses different execution plans on the master and slave, it might use index scan on the master, but table scan on the slave. Thus it may take more locks on the slave than on the master, causing it to conflict with another transaction that is run in parallel and cause a deadlock. 3. MDEV-6020. Here, there is a multi-value insert into multiple partitions of a table with auto-increment key. It appears that in this case InnoDB takes an auto-increment table lock on each partition. Since these locks are taken in the order of the inserted values, they can be taken in different order between different INSERT statements, and can cause a deadlock. Since these locks are released during the statement execution (before the commit), such deadlock can occur on the slave even though it did not occur on the master. (Note that this deadlock _is_ detected by InnoDB, and is not related to the commit order on the parallel slave). I know a way to prevent (1) on the slave (and it is implemented in the patches). I do not know if there is a way to prevent (3), it would require understanding better how innodb autoincrement and partitioning works. I do not see any way to prevent (2) from occuring on the slave, in the general case. It also seems to me that there are likely to be more corner cases lurking, and even if I somehow managed to fix the three above problems, there would be some other cases where an extra lock conflict can occur on the slave and cause the slave to hang. So I decided that a general solution is needed to such deadlocks. So the currently proposed patch does two things: A. It adds code to InnoDB, so that whenever T1 starts to wait inside InnoDB for a lock held by T2, this wait is reported to the upper server layer. The replication code will then check if T1 and T2 are part of parallel replication such that T1 must commit before T2. If so, we have a deadlock, and the code will terminate T2 with a deadlock error, allowing T1 to proceed. B. It implements code to handle deadlocks during parallel replication. When a transaction gets a deadlock error (normal, or provoked by the new deadlock detection code from (B)), it is rolled back, and re-executed. The result should be that even if there are other non-discovered cases where locks can conflict in different ways on the master from on the parallel slave, such conflicts will not cause replication to fail. Instead they will be detected, and handled automatically by resolving the deadlock. I think this will be important to ensure the reliability of parallel replication for users. I went through quite a few different ideas and attempts before ending with the present solution. So I wanted to give this background about how the patch relates to the basic problem that needs to be solved. I found a _lot_ of tricky issues during testing, and so I am somewhat unhappy about the complexity of this, even though the final patch may look simple enough on the surface. Maybe you can have some ideas for what could be done simpler, or at least question anything that looks incorrect or unnecessary. Thanks, - Kristian.

2 3

[Maria-developers] Debian packaging - option to enable feedback plugin
by Colin Charles 04 Jul '14

04 Jul '14

Hi! This is probably a great idea: https://answers.launchpad.net/maria/+question/248639 Let's ask dpkg to optionally enable feedback plugin? This could be good for us Just to be clear: I'm referring to this for the packages at mariadb.org I don't know if Debian upstream would like that -- Otto, any thoughts? cheers, -colin -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

5 5

Re: [Maria-developers] Review request for MDEV-6020, MDEV-5914, MDEV-5941, and MDEV-5262
by Kristian Nielsen 04 Jul '14

04 Jul '14

Jan Lindström <jan.lindstrom(a)skysql.com> writes: > 1) In innodb deadlock detection it is not certain that there is actually a > deadlock, deadlock detection is done when a lock request is issued. At that > point yes, we can see from lock manager that there is a deadlock, however > if waiting transaction is rolled back e.g. by user request concurrently, > the deadlock detected is not valid anymore. Hm, interesting. It was my impression that the deadlock detection algorithm only considers transactions that are already waiting for a row lock. That is, if we detect a simple deadlock between two transactions T1 and T2, then it is because T2 is waiting for a row lock held by T1 while executing some user query, and T1 is executing another user query that requests another row lock held by T2. So since both T1 and T2 are already executing user queries, and will not complete until the deadlock is resolved, how could the user roll either of them back concurrently? > 2) It is fine to roll back all transactions participating to deadlock, this > is correct. Note that there are some preferences which transaction is > rolled back on normal InnoDB server and this could be different when we > have Galera > 3) How this new needs to wait work with Galera where we have so called > brute force transactions that may not wait anything ? Note that this is only for resolving deadlocks between replicated transactions from a single master. If T1 and T2 are both replicated from the same master, we might decide to kill T2 to let T1 commit first. If I understand correctly, "brute force" transactions are only for transactions that are replicated from other Galera nodes, and which _must_ be committed in a specific order to preserve correctness. Thus, a brute-force transaction would not be affected by a deadlock kill from parallel replication. Galera code might cause a parallel replication transaction to be rolled back and retried, but this should be fine. Note also that in general for MySQL/MariaDB style replication, it is assumed (and it is the user's responsibility to ensure it) that replicated transactions will not conflict with anything done on the local node. This might also help to provide that Galera and parallel replication will not conflict. If Galera had been written with a proper general high-level API that could work with different storage-engines, then we could probably have used the "brute force transaction" mechanism to solve this problem in parallel replication also. But when I checked it, it seemed that it was hardcoded inside InnoDB with lots of Galera-specific code and data structures. If work were to start to fix this lack of proper API in Galera, then perhaps it would be interesting to work together to design a general API for enforcing a specific commit order... - Kristian.

1 0

[Maria-developers] [GSoC] optimize mysql-test runs - Preparing next steps
by Pablo Estrada 04 Jul '14

04 Jul '14

Hello Elena, I understand that the recall that we are achieving so far should still be improved; nonetheless, since we are already in the second half of the summer, I was wondering if we should spend some time designing the interfaces for the implementation of the project; such that I can start building them (even though the research phase can't be marked off as completed yet). Regards Pablo

2 1

Re: [Maria-developers] MDEV-5756 CMake option to build without thread pool
by Sergei Golubchik 03 Jul '14

03 Jul '14

Hi, Holyfoot! On Jul 01, holyfoot(a)askmonty.org wrote: > revno: 4270 > revision-id: holyfoot(a)askmonty.org-20140701104158-4gj5ztnjqlr1alwa > parent: holyfoot(a)askmonty.org-20140630193024-56wpcvwwy24kuzt4 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5756 > timestamp: Tue 2014-07-01 15:41:58 +0500 > message: > MDEV-5756 CMake option to build without thread pool. > Check if the threadpool is available on the system and set HAVE_POOL_OF_THREADS respectively. > === modified file 'sql/CMakeLists.txt' > --- a/sql/CMakeLists.txt 2014-05-30 11:24:25 +0000 > +++ b/sql/CMakeLists.txt 2014-07-01 10:41:58 +0000 > @@ -30,7 +30,15 @@ ${CMAKE_CURRENT_BINARY_DIR}/lex_hash.h > > SET_SOURCE_FILES_PROPERTIES(${GEN_SOURCES} PROPERTIES GENERATED 1) > > -ADD_DEFINITIONS(-DMYSQL_SERVER -DHAVE_EVENT_SCHEDULER -DHAVE_POOL_OF_THREADS) > +ADD_DEFINITIONS(-DMYSQL_SERVER -DHAVE_EVENT_SCHEDULER) > + > +IF (CMAKE_SYSTEM_NAME MATCHES "Linux" OR > + CMAKE_SYSTEM_NAME MATCHES "Windows" OR > + CMAKE_SYSTEM_NAME MATCHES "SunOS" OR > + HAVE_KQUEUE) > + ADD_DEFINITIONS(-DHAVE_POOL_OF_THREADS) > +ENDIF() > + > IF(SSL_DEFINES) > ADD_DEFINITIONS(${SSL_DEFINES}) > ENDIF() Did you test it on any of the platforms that that didn't support thread pool? If you would've done it, you'd noticed that your patch doesn't work. Because scheduler.h contains: #undef HAVE_POOL_OF_THREADS #if !defined(EMBEDDED_LIBRARY) && !defined(_AIX) #define HAVE_POOL_OF_THREADS 1 So, HAVE_POOL_OF_THREADS is only disabled in embedded and on AIX. The AIX part was added recently by Bar, as a quick fix to make MariaDB to compile on AIX. Your patch should remove the AIX test from scheduler.h. Actually, you should remove all #define and #undef HAVE_POOL_OF_THREADS from all files, only CMakeLists.txt should do it. Otherwise it's ok. I don't like that it explicitly checks for OS (Linux, Windows, SunOS) instead of checking for specific functions, libraries, or headers. But it's the same check that threadpool_unix.cc does, so it's ok. Regards, Sergei

1 0

[Maria-developers] [GSoC] Optimize mysql-test-runs - Analyzing some data
by Pablo Estrada 02 Jul '14

02 Jul '14

Hello Elena and all, I have spent a couple of days writing up some scripts to analyze the data, and try to find good candidates for metrics to improve recall, or some vices in the way the relevance is calculated for tests, such that they are not close enough in the priority queue to be spotted and run. Here's a writedown of some of the findings: *Of the failures that we miss, how many of them come from tests that have never failed before?* I added a small function in the simulator.py code, to keep diagnostics data of the test failures that we missed. I made a few charts regarding missed failures of tests that have never failed before (and, in the current model, we can't calculate a relevancy for them, because we have no data about them). In the next chart, we observe the percentage of missed failures that come from tests that had never failed before increases with time (as expected): By the way, this info comes from a standard run with 3000 learning cycles and 2000 prediction cycles. I feel that this is an important area. If we can figure a way to predict failures of tests that have not failed before, then we can predict subsequent failures of these tests, and thus improve recall significantly, specially over the long term. On the next section I analyze the option that I considered. *Is there a relationship between a test file being modified and the test failing any soon?* I figured that a good idea to catch failures of tests that had never failed before could be using the test file changes. If a test file is changed, I figured that the test itself would change behavior and might fail soon, and thus we could gather some data about that failure. I coded a small script that computes the number of test runs between a test file being edited, and the same test failing sometime in the future. The relationship, unfortunately is not very strong. I have attached the chart with the cummulative histogram of number of test runs that occurred between a test file being edited, and the next failure of that test. Unfortunately, the results were not too exciting. It turns out that tests don't tend to fail too soon after they have been changed (I guess this also makes sense). The following figures present the data (and a zoom in of the part of the graph we care most about), showing that failures are not very subsequent to test file editions. Not even 10% of failures occur in the first 100 runs. As we can see from the charts, there is not a very significant relationship between test file edition and subsequent failure. It seems this indicator won't be too useful. I am* still looking for ideas* on how to catch tests failing for the first time. They are all welcome : ) This is all the data that I collected so far. I am also trying to answer the following question: Failures of tests with non-zero relevance. How come we are missing those that we are missing? I have collected and charted some data, but I still haven't found anything that feels conclusive or revealing. I expect to write an email with that info soon as well, but I decided to send this out first. In conclusion: - I am thinking about how to predict first failures of a test. Any ideas? - I tested the relation between edition of a test result file and subsequent test failure. It was not a good predictor. Regards Pablo

1 0

[Maria-developers] [GSoC] Week Six!
by sriram patil 30 Jun '14

30 Jun '14

Hi all, Following is the blog link about my sixth week into GSoC. http://goo.gl/1dzgHe Suggestions and/or reviews are welcome! Thanks, Sriram

3 2

Re: [Maria-developers] [Commits] Rev 3855: Bumping server version. (10.0.13-galera) in lp:~maria-captains/maria/maria-10.0-galera
by Roberto Spadim 30 Jun '14

30 Jun '14

=] nice nirbhay 2014-06-30 10:33 GMT-03:00 Nirbhay Choubey <nirbhay(a)skysql.com>: > Hi, > > On Mon, Jun 30, 2014 at 9:13 AM, Roberto Spadim <roberto(a)spadim.com.br> > wrote: >> >> this will be a production ready version? > > > 10.0.12-galera (released over weekend) is the first 10.0-galera stable > release. This 'patch version bump' is > for preparation towards the next release. > > Best, > Nirbhay > >> >> >> 2014-06-30 10:03 GMT-03:00 Nirbhay Choubey <nirbhay(a)skysql.com>: >> > At lp:~maria-captains/maria/maria-10.0-galera >> > >> > ------------------------------------------------------------ >> > revno: 3855 >> > revision-id: nirbhay(a)skysql.com-20140630130329-w8kiirtf3blli7l1 >> > parent: nirbhay(a)skysql.com-20140626161112-q1iqtoyq5owutvvb >> > committer: Nirbhay Choubey <nirbhay(a)skysql.com> >> > branch nick: maria-10.0-galera >> > timestamp: Mon 2014-06-30 09:03:29 -0400 >> > message: >> > Bumping server version. (10.0.13-galera) >> > >> > === modified file 'VERSION' >> > --- a/VERSION 2014-06-19 17:12:38 +0000 >> > +++ b/VERSION 2014-06-30 13:03:29 +0000 >> > @@ -4,5 +4,5 @@ >> > # >> > MYSQL_VERSION_MAJOR=10 >> > MYSQL_VERSION_MINOR=0 >> > -MYSQL_VERSION_PATCH=12 >> > +MYSQL_VERSION_PATCH=13 >> > MYSQL_VERSION_EXTRA= >> > >> > >> > _______________________________________________ >> > commits mailing list >> > commits(a)mariadb.org >> > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >> >> >> >> -- >> Roberto Spadim >> SPAEmpresarial >> Eng. Automação e Controle >> _______________________________________________ >> commits mailing list >> commits(a)mariadb.org >> https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits > > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

Re: [Maria-developers] [Commits] Rev 3855: Bumping server version. (10.0.13-galera) in lp:~maria-captains/maria/maria-10.0-galera
by Roberto Spadim 30 Jun '14

30 Jun '14

this will be a production ready version? 2014-06-30 10:03 GMT-03:00 Nirbhay Choubey <nirbhay(a)skysql.com>: > At lp:~maria-captains/maria/maria-10.0-galera > > ------------------------------------------------------------ > revno: 3855 > revision-id: nirbhay(a)skysql.com-20140630130329-w8kiirtf3blli7l1 > parent: nirbhay(a)skysql.com-20140626161112-q1iqtoyq5owutvvb > committer: Nirbhay Choubey <nirbhay(a)skysql.com> > branch nick: maria-10.0-galera > timestamp: Mon 2014-06-30 09:03:29 -0400 > message: > Bumping server version. (10.0.13-galera) > > === modified file 'VERSION' > --- a/VERSION 2014-06-19 17:12:38 +0000 > +++ b/VERSION 2014-06-30 13:03:29 +0000 > @@ -4,5 +4,5 @@ > # > MYSQL_VERSION_MAJOR=10 > MYSQL_VERSION_MINOR=0 > -MYSQL_VERSION_PATCH=12 > +MYSQL_VERSION_PATCH=13 > MYSQL_VERSION_EXTRA= > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

[Maria-developers] Merge request: Cross-compile fixes (MySQL bug #61340)
by James Le Cuirot 30 Jun '14

30 Jun '14

Hello all, I posted a merge request to Launchpad back in April. A Launchpad glitch meant that it didn't get any attention but I was told on IRC to post about it here. The merge request: https://code.launchpad.net/~chewi/maria/mysql-fix-bug61340/+merge/217351 The commit diff: http://bazaar.launchpad.net/~chewi/maria/mysql-fix-bug61340/revision/4165 Given the nature of the fix, I can appreciate that it is difficult for other developers to test but I can tell you that I have just tried out a cross-built MariaDB under QEMU/proot and it seems to work. I have also seen posts from others stating that they made similar temporary adjustments to the CMake files with successful results. I am providing this contribution to the MariaDB Foundation under the MCA and also wish for it to be applied to MySQL. Regards, James

2 2

[Maria-developers] Connector Coding till 29th June
by Puneet Dewan 29 Jun '14

29 Jun '14

Hi Everyone, This week I fixed the null bitmap and types that were being sent to the server. So now I am able to successfully insert multiple values in the DB. Currently only Integer and String. This week 1.I will extend all the TypeParameter classes and ParameterWriter class so as to send the value in binary format. 2.Also try insert on different data types. 3. Write test cases Currently I am extending the PacketOutputStream.java class. Tomorrow I will have code review with Massimo. Regards Puneet.

1 0

[Maria-developers] TIME/DATETIME/TIMESTAMP data types with fractional precision
by Alexander Barkov 26 Jun '14

26 Jun '14

Hello, I propose to switch to MySQL-5.6 TIME/DATETIME/TIMESTAMP formats by default and deprecate MariaDB native temporal data types, together with pre-5.6 non-fractional types. The problem is that MariaDB native data types (and MySQL-5.5 non-fractional data types) do not work with row based replication in case of: TIME(N) -> TIME(M) There is no metadata telling about the column precision in the binary log file. It can only work when N=M. MySQL-5.6 uses separate type codes for the temporal data types: MYSQL_TYPE_TIME2 MYSQL_TYPE_DATETIME2 MYSQL_TYPE_TIMESTAMP2 and does put precision into metadata in the binary log. So it works for N != M. There is a JIRA task for this: MDEV-5377 Row-based replication of MariaDB temporal data types with FSP>0 into a different column type I'd propose to do it in 10.1.x, as 10.0.x is too late. Greetings.

2 1

Re: [Maria-developers] [Commits] Rev 4245: MDEV-6336: mysqldump --master-data does not work with GTID setups in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Sergey Vojtovich 25 Jun '14

25 Jun '14

Hi Kristian, MDEV-6344 looks fine. But I don't feel confident reviewing MDEV-6336. Regards, Sergey On Mon, Jun 16, 2014 at 11:38:04AM +0200, knielsen(a)knielsen-hq.org wrote: > At http://bazaar.launchpad.net/~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4245 > revision-id: knielsen(a)knielsen-hq.org-20140616093803-gub60pda6t77ch3k > parent: igor(a)askmonty.org-20140610223256-nqs0kkp7i45qzyvx > committer: knielsen(a)knielsen-hq.org > branch nick: work-10.0 > timestamp: Mon 2014-06-16 11:38:03 +0200 > message: > MDEV-6336: mysqldump --master-data does not work with GTID setups > MDEV-6344: mysqldump issues FLUSH TABLES, which gets written into binlog and replicated > > Add a --gtid option (for compatibility, the original behaviour is preserved > when --gtid is not used). > > With --gtid, --master-data and --dump-slave output the GTID position (the > old-style file/offset position is still output, but commented out). Also, a > CHANGE MASTER TO master_use_gtid=slave_pos is output to ensure a provisioned > slave is configured in GTID, as requested. > > Without --gtid, the GTID position is still output, if available, but commented > out. > > Also fix MDEV-6344, to avoid FLUSH TABLES getting into the binlog. Otherwise a > mysqldump on a slave server will silently inject a GTID which does not exist > on the master, which is highly undesirable. > === modified file 'client/client_priv.h' > --- a/client/client_priv.h 2013-09-14 01:09:36 +0000 > +++ b/client/client_priv.h 2014-06-16 09:38:03 +0000 > @@ -92,6 +92,7 @@ enum options_client > OPT_REPORT_PROGRESS, > OPT_SKIP_ANNOTATE_ROWS_EVENTS, > OPT_SSL_CRL, OPT_SSL_CRLPATH, > + OPT_USE_GTID, > OPT_MAX_CLIENT_OPTION /* should be always the last */ > }; > > > === modified file 'client/mysqldump.c' > --- a/client/mysqldump.c 2014-05-09 10:35:11 +0000 > +++ b/client/mysqldump.c 2014-06-16 09:38:03 +0000 > @@ -134,6 +134,7 @@ static ulong opt_compatible_mode= 0; > #define MYSQL_OPT_SLAVE_DATA_COMMENTED_SQL 2 > static uint opt_mysql_port= 0, opt_master_data; > static uint opt_slave_data; > +static uint opt_use_gtid; > static uint my_end_arg; > static char * opt_mysql_unix_port=0; > static int first_error=0; > @@ -346,6 +347,13 @@ static struct my_option my_long_options[ > {"force", 'f', "Continue even if we get an SQL error.", > &ignore_errors, &ignore_errors, 0, GET_BOOL, NO_ARG, > 0, 0, 0, 0, 0, 0}, > + {"gtid", OPT_USE_GTID, "Used together with --master-data=1 or --dump-slave=1." > + "When enabled, the output from those options will set the GTID position " > + "instead of the binlog file and offset; the file/offset will appear only as " > + "a comment. When disabled, the GTID position will still appear in the " > + "output, but only commented.", > + &opt_use_gtid, &opt_use_gtid, 0, GET_BOOL, NO_ARG, > + 0, 0, 0, 0, 0, 0}, > {"help", '?', "Display this help message and exit.", 0, 0, 0, GET_NO_ARG, > NO_ARG, 0, 0, 0, 0, 0, 0}, > {"hex-blob", OPT_HEXBLOB, "Dump binary strings (BINARY, " > @@ -1177,7 +1185,7 @@ check_consistent_binlog_pos(char *binlog > > if (mysql_query_with_error_report(mysql, &res, > "SHOW STATUS LIKE 'binlog_snapshot_%'")) > - return 1; > + return 0; > > found= 0; > while ((row= mysql_fetch_row(res))) > @@ -1200,6 +1208,90 @@ check_consistent_binlog_pos(char *binlog > return (found == 2); > } > > + > +/* > + Get the GTID position corresponding to a given old-style binlog position. > + This uses BINLOG_GTID_POS(). The advantage is that the GTID position can > + be obtained completely non-blocking in this way (without the need for > + FLUSH TABLES WITH READ LOCK), as the old-style position can be obtained > + with START TRANSACTION WITH CONSISTENT SNAPSHOT. > + > + Returns 0 if ok, non-zero if error. > +*/ > +static int > +get_binlog_gtid_pos(char *binlog_pos_file, char *binlog_pos_offset, > + char *out_gtid_pos) > +{ > + DYNAMIC_STRING query; > + MYSQL_RES *res; > + MYSQL_ROW row; > + int err; > + char file_buf[FN_REFLEN*2+1], offset_buf[LONGLONG_LEN*2+1]; > + size_t len_pos_file= strlen(binlog_pos_file); > + size_t len_pos_offset= strlen(binlog_pos_offset); > + > + if (len_pos_file >= FN_REFLEN || len_pos_offset > LONGLONG_LEN) > + return 0; > + mysql_real_escape_string(mysql, file_buf, binlog_pos_file, len_pos_file); > + mysql_real_escape_string(mysql, offset_buf, binlog_pos_offset, len_pos_offset); > + init_dynamic_string_checked(&query, "SELECT BINLOG_GTID_POS('", 256, 1024); > + dynstr_append_checked(&query, file_buf); > + dynstr_append_checked(&query, "', '"); > + dynstr_append_checked(&query, offset_buf); > + dynstr_append_checked(&query, "')"); > + > + err= mysql_query_with_error_report(mysql, &res, query.str); > + dynstr_free(&query); > + if (err) > + return err; > + > + err= 1; > + if ((row= mysql_fetch_row(res))) > + { > + strmake(out_gtid_pos, row[0], FN_REFLEN-1); > + err= 0; > + } > + mysql_free_result(res); > + > + return err; > +} > + > + > +/* > + Get the GTID position on a master or slave. > + The parameter MASTER is non-zero to get the position on a master > + (@@gtid_binlog_pos) or zero for a slave (@@gtid_slave_pos). > + > + This uses the @@gtid_binlog_pos or @@gtid_slave_pos, so requires FLUSH TABLES > + WITH READ LOCK or similar to be consistent. > + > + Returns 0 if ok, non-zero for error. > +*/ > +static int > +get_gtid_pos(char *out_gtid_pos, int master) > +{ > + MYSQL_RES *res; > + MYSQL_ROW row; > + int found; > + > + if (mysql_query_with_error_report(mysql, &res, > + (master ? > + "SELECT @@GLOBAL.gtid_binlog_pos" : > + "SELECT @@GLOBAL.gtid_slave_pos"))) > + return 1; > + > + found= 0; > + if ((row= mysql_fetch_row(res))) > + { > + strmake(out_gtid_pos, row[0], FN_REFLEN-1); > + found++; > + } > + mysql_free_result(res); > + > + return (found != 1); > +} > + > + > static char *my_case_str(const char *str, > uint str_len, > const char *token, > @@ -4799,12 +4891,14 @@ static int dump_selected_tables(char *db > } /* dump_selected_tables */ > > > -static int do_show_master_status(MYSQL *mysql_con, int consistent_binlog_pos) > +static int do_show_master_status(MYSQL *mysql_con, int consistent_binlog_pos, > + int have_mariadb_gtid, int use_gtid) > { > MYSQL_ROW row; > MYSQL_RES *UNINIT_VAR(master); > char binlog_pos_file[FN_REFLEN]; > char binlog_pos_offset[LONGLONG_LEN+1]; > + char gtid_pos[FN_REFLEN]; > char *file, *offset; > const char *comment_prefix= > (opt_master_data == MYSQL_OPT_MASTER_DATA_COMMENTED_SQL) ? "-- " : ""; > @@ -4815,6 +4909,9 @@ static int do_show_master_status(MYSQL * > return 1; > file= binlog_pos_file; > offset= binlog_pos_offset; > + if (have_mariadb_gtid && > + get_binlog_gtid_pos(binlog_pos_file, binlog_pos_offset, gtid_pos)) > + return 1; > } > else > { > @@ -4844,6 +4941,9 @@ static int do_show_master_status(MYSQL * > return 0; > } > } > + > + if (have_mariadb_gtid && get_gtid_pos(gtid_pos, 1)) > + return 1; > } > > /* SHOW MASTER STATUS reports file and position */ > @@ -4852,7 +4952,19 @@ static int do_show_master_status(MYSQL * > "recovery from\n--\n\n"); > fprintf(md_result_file, > "%sCHANGE MASTER TO MASTER_LOG_FILE='%s', MASTER_LOG_POS=%s;\n", > - comment_prefix, file, offset); > + (use_gtid ? "-- " : comment_prefix), file, offset); > + if (have_mariadb_gtid) > + { > + print_comment(md_result_file, 0, > + "\n--\n-- GTID to start replication from\n--\n\n"); > + if (use_gtid) > + fprintf(md_result_file, > + "%sCHANGE MASTER TO MASTER_USE_GTID=slave_pos;\n", > + comment_prefix); > + fprintf(md_result_file, > + "%sSET GLOBAL gtid_slave_pos='%s';\n", > + (!use_gtid ? "-- " : comment_prefix), gtid_pos); > + } > check_io(md_result_file); > > if (!consistent_binlog_pos) > @@ -4922,12 +5034,16 @@ static int add_slave_statements(void) > return(0); > } > > -static int do_show_slave_status(MYSQL *mysql_con) > +static int do_show_slave_status(MYSQL *mysql_con, int use_gtid, > + int have_mariadb_gtid) > { > MYSQL_RES *UNINIT_VAR(slave); > MYSQL_ROW row; > const char *comment_prefix= > (opt_slave_data == MYSQL_OPT_SLAVE_DATA_COMMENTED_SQL) ? "-- " : ""; > + const char *gtid_comment_prefix= (use_gtid ? comment_prefix : "-- "); > + const char *nogtid_comment_prefix= (!use_gtid ? comment_prefix : "-- "); > + int set_gtid_done= 0; > > if (mysql_query_with_error_report(mysql_con, &slave, > multi_source ? > @@ -4945,8 +5061,30 @@ static int do_show_slave_status(MYSQL *m > > while ((row= mysql_fetch_row(slave))) > { > + if (multi_source && !set_gtid_done) > + { > + char gtid_pos[FN_REFLEN]; > + if (have_mariadb_gtid && get_gtid_pos(gtid_pos, 0)) > + return 1; > + if (opt_comments) > + fprintf(md_result_file, "\n--\n-- Gtid position to start replication " > + "from\n--\n\n"); > + fprintf(md_result_file, "%sSET GLOBAL gtid_slave_pos='%s';\n", > + gtid_comment_prefix, gtid_pos); > + set_gtid_done= 1; > + } > if (row[9 + multi_source] && row[21 + multi_source]) > { > + if (use_gtid) > + { > + if (multi_source) > + fprintf(md_result_file, "%sCHANGE MASTER '%.80s' TO " > + "MASTER_USE_GTID=slave_pos;\n", gtid_comment_prefix, row[0]); > + else > + fprintf(md_result_file, "%sCHANGE MASTER TO " > + "MASTER_USE_GTID=slave_pos;\n", gtid_comment_prefix); > + } > + > /* SHOW MASTER STATUS reports file and position */ > if (opt_comments) > fprintf(md_result_file, > @@ -4955,9 +5093,9 @@ static int do_show_slave_status(MYSQL *m > > if (multi_source) > fprintf(md_result_file, "%sCHANGE MASTER '%.80s' TO ", > - comment_prefix, row[0]); > + nogtid_comment_prefix, row[0]); > else > - fprintf(md_result_file, "%sCHANGE MASTER TO ", comment_prefix); > + fprintf(md_result_file, "%sCHANGE MASTER TO ", nogtid_comment_prefix); > > if (opt_include_master_host_port) > { > @@ -5030,12 +5168,13 @@ static int do_flush_tables_read_lock(MYS > FLUSH TABLES is to lower the probability of a stage where both mysqldump > and most client connections are stalled. Of course, if a second long > update starts between the two FLUSHes, we have that bad stall. > + > + We use the LOCAL option, as we do not want the FLUSH TABLES replicated to > + other servers. > */ > return > - ( mysql_query_with_error_report(mysql_con, 0, > - ((opt_master_data != 0) ? > - "FLUSH /*!40101 LOCAL */ TABLES" : > - "FLUSH TABLES")) || > + ( mysql_query_with_error_report(mysql_con, 0, > + "FLUSH /*!40101 LOCAL */ TABLES") || > mysql_query_with_error_report(mysql_con, 0, > "FLUSH TABLES WITH READ LOCK") ); > } > @@ -5656,6 +5795,7 @@ int main(int argc, char **argv) > char bin_log_name[FN_REFLEN]; > int exit_code; > int consistent_binlog_pos= 0; > + int have_mariadb_gtid= 0; > MY_INIT(argv[0]); > > sf_leaking_memory=1; /* don't report memory leaks on early exits */ > @@ -5696,7 +5836,10 @@ int main(int argc, char **argv) > > /* Check if the server support multi source */ > if (mysql_get_server_version(mysql) >= 100000) > + { > multi_source= 2; > + have_mariadb_gtid= 1; > + } > > if (opt_slave_data && do_stop_slave_sql(mysql)) > goto err; > @@ -5743,9 +5886,11 @@ int main(int argc, char **argv) > /* Add 'STOP SLAVE to beginning of dump */ > if (opt_slave_apply && add_stop_slave()) > goto err; > - if (opt_master_data && do_show_master_status(mysql, consistent_binlog_pos)) > + if (opt_master_data && do_show_master_status(mysql, consistent_binlog_pos, > + have_mariadb_gtid, opt_use_gtid)) > goto err; > - if (opt_slave_data && do_show_slave_status(mysql)) > + if (opt_slave_data && do_show_slave_status(mysql, opt_use_gtid, > + have_mariadb_gtid)) > goto err; > if (opt_single_transaction && do_unlock_tables(mysql)) /* unlock but no commit! */ > goto err; > > === modified file 'mysql-test/r/rpl_mysqldump_slave.result' > --- a/mysql-test/r/rpl_mysqldump_slave.result 2014-02-26 14:28:07 +0000 > +++ b/mysql-test/r/rpl_mysqldump_slave.result 2014-06-16 09:38:03 +0000 > @@ -4,18 +4,59 @@ include/master-slave.inc > # New --dump-slave, --apply-slave-statements functionality > # > use test; > +-- SET GLOBAL gtid_slave_pos=''; > CHANGE MASTER '' TO MASTER_LOG_FILE='master-bin.000001', MASTER_LOG_POS=BINLOG_START; > STOP ALL SLAVES; > +-- SET GLOBAL gtid_slave_pos=''; > CHANGE MASTER '' TO MASTER_LOG_FILE='master-bin.000001', MASTER_LOG_POS=BINLOG_START; > START ALL SLAVES; > STOP ALL SLAVES; > +-- SET GLOBAL gtid_slave_pos=''; > CHANGE MASTER '' TO MASTER_HOST='127.0.0.1', MASTER_PORT=MASTER_MYPORT, MASTER_LOG_FILE='master-bin.000001', MASTER_LOG_POS=BINLOG_START; > START ALL SLAVES; > start slave; > Warnings: > Note 1254 Slave is already running > +-- SET GLOBAL gtid_slave_pos=''; > CHANGE MASTER '' TO MASTER_LOG_FILE='master-bin.000001', MASTER_LOG_POS=BINLOG_START; > start slave; > Warnings: > Note 1254 Slave is already running > +*** Test mysqldump --dump-slave GTID functionality. > +SET gtid_seq_no = 1000; > +CREATE TABLE t1 (a INT PRIMARY KEY); > +DROP TABLE t1; > +CREATE TABLE t2 (a INT PRIMARY KEY); > +DROP TABLE t2; > + > +1. --dump-slave=1 > + > +SET GLOBAL gtid_slave_pos='0-1-1001'; > +CHANGE MASTER '' TO MASTER_USE_GTID=slave_pos; > +-- CHANGE MASTER '' TO MASTER_LOG_FILE='master-bin.000001', MASTER_LOG_POS=BINLOG_START; > + > +2. --dump-slave=2 > + > +-- SET GLOBAL gtid_slave_pos='0-1-1001'; > +-- CHANGE MASTER '' TO MASTER_USE_GTID=slave_pos; > +-- CHANGE MASTER '' TO MASTER_LOG_FILE='master-bin.000001', MASTER_LOG_POS=BINLOG_START; > +*** Test mysqldump --master-data GTID functionality. > + > +1. --master-data=1 > + > +-- CHANGE MASTER TO MASTER_LOG_FILE='slave-bin.000001', MASTER_LOG_POS=BINLOG_START; > +CHANGE MASTER TO MASTER_USE_GTID=slave_pos; > +SET GLOBAL gtid_slave_pos='0-2-1003'; > + > +2. --master-data=2 > + > +-- CHANGE MASTER TO MASTER_LOG_FILE='slave-bin.000001', MASTER_LOG_POS=BINLOG_START; > +-- CHANGE MASTER TO MASTER_USE_GTID=slave_pos; > +-- SET GLOBAL gtid_slave_pos='0-2-1003'; > + > +3. --master-data --single-transaction > + > +-- CHANGE MASTER TO MASTER_LOG_FILE='slave-bin.000001', MASTER_LOG_POS=BINLOG_START; > +CHANGE MASTER TO MASTER_USE_GTID=slave_pos; > +SET GLOBAL gtid_slave_pos='0-2-1003'; > include/rpl_end.inc > > === modified file 'mysql-test/t/rpl_mysqldump_slave.test' > --- a/mysql-test/t/rpl_mysqldump_slave.test 2014-02-26 14:28:07 +0000 > +++ b/mysql-test/t/rpl_mysqldump_slave.test 2014-06-16 09:38:03 +0000 > @@ -36,4 +36,53 @@ start slave; > --exec $MYSQL_DUMP_SLAVE --compact --dump-slave no_such_db > start slave; > > + > +--echo *** Test mysqldump --dump-slave GTID functionality. > + > +--connection master > +SET gtid_seq_no = 1000; > +CREATE TABLE t1 (a INT PRIMARY KEY); > +DROP TABLE t1; > +--sync_slave_with_master > + > +--connection slave > +# Inject a local transaction on the slave to check that this is not considered > +# for --dump-slave. > +CREATE TABLE t2 (a INT PRIMARY KEY); > +DROP TABLE t2; > + > +--echo > +--echo 1. --dump-slave=1 > +--echo > +--replace_regex /MASTER_LOG_POS=[0-9]+/MASTER_LOG_POS=BINLOG_START/ > +--exec $MYSQL_DUMP_SLAVE --compact --dump-slave=1 --gtid test > + > +--echo > +--echo 2. --dump-slave=2 > +--echo > +--replace_regex /MASTER_LOG_POS=[0-9]+/MASTER_LOG_POS=BINLOG_START/ > +--exec $MYSQL_DUMP_SLAVE --compact --dump-slave=2 --gtid test > + > + > +--echo *** Test mysqldump --master-data GTID functionality. > +--echo > +--echo 1. --master-data=1 > +--echo > +--replace_regex /MASTER_LOG_POS=[0-9]+/MASTER_LOG_POS=BINLOG_START/ > +--exec $MYSQL_DUMP_SLAVE --compact --master-data=1 --gtid test > + > +--echo > +--echo 2. --master-data=2 > +--echo > +--replace_regex /MASTER_LOG_POS=[0-9]+/MASTER_LOG_POS=BINLOG_START/ > +--exec $MYSQL_DUMP_SLAVE --compact --master-data=2 --gtid test > + > +--echo > +--echo 3. --master-data --single-transaction > +--echo > +--replace_regex /MASTER_LOG_POS=[0-9]+/MASTER_LOG_POS=BINLOG_START/ > +--exec $MYSQL_DUMP_SLAVE --compact --master-data --single-transaction --gtid test > + > + > + > --source include/rpl_end.inc > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 1

Re: [Maria-developers] [Commits] Rev 4259: MDEV-6120: When slave stops with error, error message should indicate the failing GTID in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Sergey Vojtovich 25 Jun '14

25 Jun '14

Hi Kristian, looks good. One minor question below. On Thu, Jun 19, 2014 at 02:50:44PM +0200, knielsen(a)knielsen-hq.org wrote: > At http://bazaar.launchpad.net/~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4259 > revision-id: knielsen(a)knielsen-hq.org-20140619125043-acw2gx7rw90y3vqp > parent: svoj(a)mariadb.org-20140618141504-kqq6ws9gm93r2b2d > committer: Kristian Nielsen <knielsen(a)knielsen-hq.org> > branch nick: work-10.0 > timestamp: Thu 2014-06-19 14:50:43 +0200 > message: > MDEV-6120: When slave stops with error, error message should indicate the failing GTID > > If replication breaks in GTID mode, it is not trivial to determine the GTID of > the failing event group. This is a problem, as such GTID is needed eg. to > explicitly set @@gtid_slave_pos to skip to after that event group, or to > compare errors on different servers, etc. > > Fix by ensuring that relevant slave errors logged to the error log include the > GTID of the event group containing the problem event. ...skip... > === added file 'mysql-test/suite/rpl/t/rpl_gtid_errorlog.test' > --- a/mysql-test/suite/rpl/t/rpl_gtid_errorlog.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/rpl/t/rpl_gtid_errorlog.test 2014-06-19 12:50:43 +0000 > @@ -0,0 +1,94 @@ > +--source include/have_debug.inc > +--source include/master-slave.inc > + > +--echo *** Test MDEV-6120, output of current GTID when a replication error is logged to the errorlog *** > +--connection master > +CREATE TABLE t1(a INT PRIMARY KEY); > +--sync_slave_with_master > + > +--connection slave > +--source include/stop_slave.inc > +CHANGE MASTER TO master_use_gtid=slave_pos; > + > +--connection master > +INSERT INTO t1 VALUES (1); > +SET gtid_seq_no=100; > +INSERT INTO t1 VALUES (2); > +INSERT INTO t1 VALUES (3); > +INSERT INTO t1 VALUES (4); > +--save_master_pos > + > +--connection slave > +SET sql_log_bin=0; > +INSERT INTO t1 VALUES (2); > +SET sql_log_bin=1; > + > +START SLAVE; > +--let $slave_sql_errno=1062 > +--source include/wait_for_slave_sql_error.inc > + > +--source include/stop_slave.inc > +# Skip the problem event from the master. > +SET GLOBAL gtid_slave_pos= "0-1-100"; > +--source include/start_slave.inc > +--sync_with_master > + > +SELECT * FROM t1 ORDER BY a; > + > +--connection master > + > +SET @dbug_save= @@debug_dbug; > +SET debug_dbug= '+d,incident_database_resync_on_replace'; > +REPLACE INTO t1 VALUES (5); > +SET debug_dbug= @dbug_save; > +--save_master_pos > + > +--connection slave > +--let $slave_sql_errno=1590 > +--source include/wait_for_slave_sql_error.inc > +# ToDo no need to switch off GTID once MDEV-4937 is fixed > +--source include/stop_slave.inc > +CHANGE MASTER TO master_use_gtid=no; > +SET sql_slave_skip_counter=1; > +--source include/start_slave.inc > +--sync_with_master > +--source include/stop_slave.inc > +CHANGE MASTER TO master_use_gtid=slave_pos; > +--source include/start_slave.inc > + > +SELECT * FROM t1 ORDER BY a; > + > + > +# Check error log for correct messages. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.2.err; > +} > +let LOG_ERROR=$log_error_; > + > +perl; > + use strict; > + use warnings; > + my $log_error= $ENV{'LOG_ERROR'} or die "LOG_ERROR not set"; > + open(FILE, "<", "$log_error") or die("Unable to open $log_error: $!\n"); > + my $found_dup= 0; > + my $found_incident= 0; > + while (<FILE>) { > + $found_dup= 1 > + if /Slave SQL: Error 'Duplicate entry .* on query\. .*Query: '.*', Gtid 0-1-100, Internal MariaDB error code:/ > + or /Slave SQL: Could not execute Write_rows.*table test.t1; Duplicate entry.*, Gtid 0-1-100, Internal MariaDB error/ > + ; > + $found_incident= 1 > + if /Slave SQL: The incident LOST_EVENTS occured on the master\. Message: <none>, Internal MariaDB error code: 1590/; > + } > + print "Duplicate key found: $found_dup; incident found: $found_incident\n"; > + close(FILE); > +EOF > + > +--connection master > +DROP TABLE t1; > + > +--source include/rpl_end.inc Why didn't you like include/search_pattern_in_file.inc? It should perfectly able to match your patterns. ...skip... Regards, Sergey

2 1

[Maria-developers] Review of MDEV-5730: enhance security using special compilation options
by Kristian Nielsen 24 Jun '14

24 Jun '14

> commit feab48657528b9bb40fb035d51bee28d93710c1e > Author: Sergei Golubchik <serg(a)mariadb.org> > Date: Mon Jun 16 21:39:09 2014 +0200 > > MDEV-5730 enhance security using special compilation options > > -Wl,-z,relro,-z,now > -pie > -fstack-protector --param=ssp-buffer-size=4 > -D_FORTIFY_SOURCE=2 The patch is ok to push with a clarified comment as requested below. I have checked the compiler/linker options added and commented some on them below. > diff --git a/CMakeLists.txt b/CMakeLists.txt > index a24f000..5b0d348 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -201,6 +201,20 @@ IF (WITH_ASAN) > ENDIF() > ENDIF() > > +OPTION(SECURITY_HARDENED "Use security-enhancing compiler features (stack protector, relro, etc)" ON) So these are on by default. This means that this patch will cause some performance impact. So having them on by default is a compromise between performance and security. You should probably include in the commit comment a discussion of why the security viewpoint was prefered over the performance viewpoint in this case (or a comment in the cmake file if you prefer). The proper way to handle a patch like this would be to measure the performance impact, eg. for single-threaded performance with some various simple sysbench loads. However, it is quite hard to do such measurements, as you need a setup that has low enough noise in results to detect sub-percentage differences of performance. I have had some success by locking processes to a single core and using `perf` measurements of cycles spent (rather than wall-clock time), but even so the results can be tricky to interpret. So it may be too much work for a small patch like this. (And even if the performance improvement can be measured, how can we measure the security impact to weight them against one another?) My personal opinion is that I do not buy into this "security" mindset much. I do not think these options improve security significantly in practise for most users. So while the performance impact is probably also not very significant, why accept it at all (by default)? But the opposite viewpoint exists and is valid as well. So I think this patch is ok if you add a discussion of why it was chosen to do it this way. (Even if the reason is "Most distros do it" - at least then it is documented what the reason is). Also, the option should mention the possible performance impact. Suggestion: "Use compiler options (stack protector, relro, etc) that may slightly enhance security of the resulting binary, at the possible cost of some small performance impact." > +IF(SECURITY_HARDENED) > + # security-enhancing flags > + MY_CHECK_AND_SET_COMPILER_FLAG("-pie -fPIC") -fPIC can have a significant impact on performance on some platforms. For example on 32-bit x86, it requires extra instructions at the head of every function, and it makes one less register available for code generation on a CPU already starved for registers. On 64-bit x86, the impact is much smaller, as the architecture was enhanced to better support -fPIC. However, I seem to recall that the server is anyway built with -fPIC, for libmysqld or something like that? In which case this has no extra impact. > + MY_CHECK_AND_SET_COMPILER_FLAG("-Wl,-z,relro,-z,now") This option seems to cause all relocations to happen at load time, rather than lazily on-demand. The impact should be higher memory usage and some extra CPU time for these relocations, many of which may never be needed. This option will have the biggest effect on the client programs probably, as server startup is already fairly heavy. Probably even for client programs the added cost will be not very significant, as even the client programs are usually called to do significant processing (eg. connect to a database and so on). I guess these options would mostly hurt small utility programs for shell scripts, like `mv` or so. > + MY_CHECK_AND_SET_COMPILER_FLAG("-fstack-protector --param=ssp-buffer-size=4") This also adds extra code and memory loads to each function, so it will have some impact to performance. It is probably the most significant impact of all the added options, though without detailed analysis this is only guesswork, of course. > + # sometimes _FORTIFY_SOURCE is predefined > + INCLUDE(CheckSymbolExists) > + CHECK_SYMBOL_EXISTS(_FORTIFY_SOURCE "" HAVE_FORTIFY_SOURCE) > + IF(NOT HAVE_FORTIFY_SOURCE) > + ADD_DEFINITIONS(-D_FORTIFY_SOURCE=2) > + ENDIF() > +ENDIF() FORTIFY_SOURCE also adds some runtime overhead, but it is not very much, it is an extra argument to memcpy() and friends about destination buffer size. Probably the way or code is written, there will be few places where this option makes any difference, so the performance impact does not seem likely to be very significant. Hope this helps, - Kristian.

3 2

[Maria-developers] [GSoC] Optimize mysql-test-runs - Setback
by Pablo Estrada 24 Jun '14

24 Jun '14

Hello Sergei, Elena and all, Today while working on the script, I found and fixed an issue: There is some faulty code code in my script that is in charge of collecting the statistics about whether a test failure was caught or not (here <https://github.com/pabloem/Kokiri/blob/master/basic_simulator.py#L393>). I looked into fixing it, and then I could see another *problem*: The *recall numbers* that I had collected previously were *too high*. The actual recall numbers, once we consider the test failures that are *not caught*, are disappointingly lower. I won't show you results yet, since I want to make sure that the code has been fixed, and I have accurate tests first. This is all for now. The strategy that I was using is a lot less effective than it seemed initially. I will send out a more detailed report with results, my opinion on the weak points of the strategy, and ideas, including a roadmap to try to improve results. Regards. All feedback is welcome. Pablo

3 18

[Maria-developers] Doubt in bin logging for CREATE VIEW
by sriram patil 24 Jun '14

24 Jun '14

Hi All, I was checking the behavior of bin logging in various methods. I found out CREATE VIEW differs a little bit than others in bin logging. The query is written into bin log if query is executed successfully. If there is any error the query is not logged. This behavior is observed across all the commands. In case of CREATE VIEW (method: mysql_create_view file: sql/sql_view.cc), while the bin logging is skipped for most of the errors by jumping to "err" label, it is not skipped in following code snippet (line number: 616) res= mysql_register_view(thd, view, mode); /* View TABLE_SHARE must be removed from the table definition cache in order to make ALTER VIEW work properly. Otherwise, we would not be able to detect meta-data changes after ALTER VIEW. */ if (!res) tdc_remove_table(thd, TDC_RT_REMOVE_ALL, view->db, view->table_name, false); if (mysql_bin_log.is_open()) { // this is where the log is written } If an error occurs in mysql_register_view, e.g. ER_OUT_OF_RESOURCES, ER_TABLE_EXISTS_ERROR, etc. the query is still logged. Is there a specific reason why it is logged even though an error occurs? Or is it a bug? Thanks & Regards, Sriram

4 9

Re: [Maria-developers] [Commits] 7d386bb: MDEV-4549 [PATCH] Clean up code working with ACL tables
by Sergey Vojtovich 23 Jun '14

23 Jun '14

Hi Sergei, ok to push. A few minor questions inline. On Mon, Jun 16, 2014 at 09:34:21AM +0200, serg(a)mariadb.org wrote: > revision-id: 7d386bbce216745ba28df728f45a0b796fcced98 > parent(s): f61f36b386e8d0c6448297bb84c92e8d9b82be6a > committer: Sergei Golubchik > branch nick: maria > timestamp: 2014-06-15 17:12:57 +0200 > message: > > MDEV-4549 [PATCH] Clean up code working with ACL tables > > enum values to index different ACL tables, instead of hard-coded numbers > (even different in diffent functions). Also factor out TABLE_LIST > initialization into a separate function. > > --- > mysql-test/r/not_embedded_server.result | 4 +- > mysql-test/r/sp-destruct.result | 3 - > .../suite/rpl/r/rpl_tmp_table_and_DDL.result | 22 +- > mysql-test/t/not_embedded_server.test | 2 +- > mysql-test/t/sp-destruct.test | 1 - > sql/sql_acl.cc | 533 +++++++++------------ > 6 files changed, 235 insertions(+), 330 deletions(-) > > diff --git a/mysql-test/r/not_embedded_server.result b/mysql-test/r/not_embedded_server.result > index 2295276..6dda855 100644 > --- a/mysql-test/r/not_embedded_server.result > +++ b/mysql-test/r/not_embedded_server.result > @@ -1,4 +1,4 @@ > -call mtr.add_suppression("Can't open and lock privilege tables: Table 'host' was not locked with LOCK TABLES"); > +call mtr.add_suppression("Can't open and lock privilege tables: Table 'roles_mapping' was not locked with LOCK TABLES"); > SHOW VARIABLES like 'slave_skip_errors'; > Variable_name Value > slave_skip_errors OFF Why did it change (here and in other tests)? I didn't notice any table order change in the code. ...skip... > diff --git a/mysql-test/r/sp-destruct.result b/mysql-test/r/sp-destruct.result > index 172e40c..fe68229 100644 > --- a/mysql-test/r/sp-destruct.result > +++ b/mysql-test/r/sp-destruct.result > @@ -128,11 +128,8 @@ CREATE FUNCTION f1() RETURNS INT RETURN 1; > RENAME TABLE mysql.procs_priv TO procs_priv_backup; > FLUSH TABLE mysql.procs_priv; > DROP FUNCTION f1; > -ERROR 42S02: Table 'mysql.procs_priv' doesn't exist > SHOW WARNINGS; > Level Code Message > -Error 1146 Table 'mysql.procs_priv' doesn't exist > -Warning 1405 Failed to revoke all privileges to dropped routine > # Restore the procs_priv table > RENAME TABLE procs_priv_backup TO mysql.procs_priv; > FLUSH TABLE mysql.procs_priv; Hmm... why? ...skip... > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 5e8e0e7..442d512 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -380,7 +380,7 @@ class ACL_PROXY_USER :public ACL_ACCESS > MYSQL_PROXIES_PRIV_PROXIED_USER, > MYSQL_PROXIES_PRIV_WITH_GRANT, > MYSQL_PROXIES_PRIV_GRANTOR, > - MYSQL_PROXIES_PRIV_TIMESTAMP } old_acl_proxy_users; > + MYSQL_PROXIES_PRIV_TIMESTAMP } proxy_table_fields; > public: > ACL_PROXY_USER () {}; > > @@ -760,6 +760,99 @@ static int traverse_role_graph_down(ACL_USER_BASE *, void *, > int (*) (ACL_USER_BASE *, ACL_ROLE *, void *)); > > /* > + Enumeration of ACL/GRANT tables in the mysql database > +*/ > +enum enum_acl_tables > +{ > + USER_TABLE, > + DB_TABLE, > + TABLES_PRIV_TABLE, > + COLUMNS_PRIV_TABLE, > +#define FIRST_OPTIONAL_TABLE HOST_TABLE > + HOST_TABLE, > + PROCS_PRIV_TABLE, > + PROXIES_PRIV_TABLE, > + ROLES_MAPPING_TABLE, > + TABLES_MAX // <== always the last > +}; > +// bits for init_priv_tables > +static const int Table_user= 1 << USER_TABLE; > +static const int Table_db= 1 << DB_TABLE; > +static const int Table_tables_priv= 1 << TABLES_PRIV_TABLE; > +static const int Table_columns_priv= 1 << COLUMNS_PRIV_TABLE; > +static const int Table_host= 1 << HOST_TABLE; > +static const int Table_procs_priv= 1 << PROCS_PRIV_TABLE; > +static const int Table_proxies_priv= 1 << PROXIES_PRIV_TABLE; > +static const int Table_roles_mapping= 1 << ROLES_MAPPING_TABLE; > + > +const LEX_STRING acl_table_names[]= // matches enum_acl_tables > +{ > + { C_STRING_WITH_LEN("user") }, > + { C_STRING_WITH_LEN("db") }, > + { C_STRING_WITH_LEN("tables_priv") }, > + { C_STRING_WITH_LEN("columns_priv") }, > + { C_STRING_WITH_LEN("host") }, > + { C_STRING_WITH_LEN("procs_priv") }, > + { C_STRING_WITH_LEN("proxies_priv") }, > + { C_STRING_WITH_LEN("roles_mapping") } > +}; > + > +/** > + Initialize a TABLE_LIST array to a set of acl/grant tables. > + All tables will be opened with the same lock type, either read or write. > + > + @retval !0 a pointer to the first initialized element in the TABLE_LIST > + @retval 0 replication filters matched. Abort operation, but return OK (!) > +*/ > +TABLE_LIST *init_priv_tables(THD *thd, TABLE_LIST *tables, > + enum thr_lock_type lock_type, int tables_to_open) Nice clean-up, I assume there was a reason not rewrite open_grant_tables() instead. > +{ > + int prev= -1; > + bzero(tables, sizeof(TABLE_LIST) * TABLES_MAX); > + for (int cur=0, mask=1; mask <= tables_to_open; cur++, mask <<= 1) > + { > + if ((tables_to_open & mask) == 0) > + continue; > + tables[cur].init_one_table(C_STRING_WITH_LEN("mysql"), > + acl_table_names[cur].str, > + acl_table_names[cur].length, > + acl_table_names[cur].str, lock_type); > + tables[cur].open_type= OT_BASE_ONLY; > + if (lock_type >= TL_WRITE_ALLOW_WRITE) > + tables[cur].updating= 1; > + if (cur >= FIRST_OPTIONAL_TABLE) > + tables[cur].open_strategy= TABLE_LIST::OPEN_IF_EXISTS; > + if (prev != -1) > + tables[cur].next_local= tables[cur].next_global= & tables[prev]; > + prev= cur; > + } > + > +#ifdef HAVE_REPLICATION > + if (lock_type >= TL_WRITE_ALLOW_WRITE && thd->slave_thread && !thd->spcont) > + { > + /* > + GRANT and REVOKE are applied the slave in/exclusion rules as they are > + some kind of updates to the mysql.% tables. > + */ > + Rpl_filter *rpl_filter= thd->system_thread_info.rpl_sql_info->rpl_filter; > + if (rpl_filter->is_on() && !rpl_filter->tables_ok(0, tables)) > + return NULL; > + } > +#endif Was it a bug that mysql_grant_role() didn't have this code? open_grant_tables() does updating= 0 after tables_ok(). Is that needed? ...skip... > @@ -5783,8 +5796,9 @@ bool mysql_routine_grant(THD *thd, TABLE_LIST *table_list, bool is_proc, > } > } > > - if (replace_routine_table(thd, grant_name, tables[1].table, *Str, > - db_name, table_name, is_proc, rights, > + if (no_such_table(tables + PROCS_PRIV_TABLE) || > + replace_routine_table(thd, grant_name, tables[PROCS_PRIV_TABLE].table, > + *Str, db_name, table_name, is_proc, rights, > revoke_grant) != 0) > { > result= TRUE; Why no_such_table() here? ...skip... > @@ -6250,7 +6232,7 @@ bool mysql_grant(THD *thd, const char *db, List <LEX_USER> &list, > ulong db_rights= rights & DB_ACLS; > if (db_rights == rights) > { > - if (replace_db_table(tables[1].table, db, *Str, db_rights, > + if (replace_db_table(tables[DB_TABLE].table, db, *Str, db_rights, > revoke_grant)) > result= -1; > } Tabs here (and in a few hunks below). > @@ -6262,9 +6244,11 @@ bool mysql_grant(THD *thd, const char *db, List <LEX_USER> &list, > } > else if (is_proxy) > { > - if (replace_proxies_priv_table (thd, tables[1].table, Str, proxied_user, > - rights & GRANT_ACL ? TRUE : FALSE, > - revoke_grant)) > + if (no_such_table(tables + PROXIES_PRIV_TABLE) || > + replace_proxies_priv_table (thd, tables[PROXIES_PRIV_TABLE].table, > + Str, proxied_user, > + rights & GRANT_ACL ? TRUE : FALSE, > + revoke_grant)) > result= -1; > } > if (Str->is_role()) Why no_such_table() here? ...skip... Regards, Sergey

2 2

[Maria-developers] [GSoC] Week Five!
by sriram patil 23 Jun '14

23 Jun '14

Hi All, Following is the link to my blog post of fifth week into GSoC. http://goo.gl/x7dtcn Suggestions and/or review are welcome! Thanks, Sriram

1 0

[Maria-developers] Java Connector Coding till 21st June
by Puneet Dewan 21 Jun '14

21 Jun '14

Hi Everyone. Test case class(ServersidePreparedStatementTest.java) was written for checking the value of statement id. This week code review was done by Massimo and Georg. There pointers were really valuable as data was not being send in a proper format to the server for execute(com_stmt_execute) method : 1.NullBitMap was set incorrectly. 2.Creation of a new class extending PacketOuptutStream e.g integer of 4 bytes can be sent.Currently if an integer is sent it is sent as a byte. 3.Also I was expecting when we do a *prepare* for a statement like insert into XYZ(id) values(?) where id is of type Integer. Then server shall be returning integer as a type. This was a false assumption.It does not return the type. So now on the basis of value set for *id as example*,decide on the type to be sent from execute. So this week will fix the above things. Looking forward for suggestions. Regards Puneet.

1 0

Re: [Maria-developers] [Commits] 22f532c: MDEV-4260 Don't create frm files for temporary tables
by Sergey Vojtovich 20 Jun '14

20 Jun '14

Hi Sergei, ok to push. One note inline. On Fri, Jun 20, 2014 at 11:29:36AM +0200, serg(a)mariadb.org wrote: > revision-id: 22f532cfb2f40132e2f7b079a3aafb109af92c26 > parent(s): fb8818c1aff9ce67b74f965c019cad93addd811e > committer: Sergei Golubchik > branch nick: maria > timestamp: 2014-06-20 11:29:12 +0200 > message: > > MDEV-4260 Don't create frm files for temporary tables > > * Don't write frm for tmp tables > * pass frm image down to open_table_uncached, when possible > * don't use truncate-by-recreate for temp tables - cannot recreate > without frm, and delete_all_rows is faster anyway > * remove HTON_FLUSH_AFTER_RENAME (BDB-ism, unused for years) > > --- > mysql-test/r/temp_table_frm.result | 20 +++++++++++ > mysql-test/t/temp_table_frm.test | 13 ++++++++ > sql/handler.h | 2 +- > sql/sql_base.cc | 14 +++++++- > sql/sql_base.h | 3 +- > sql/sql_table.cc | 27 +++------------ > sql/sql_truncate.cc | 68 +------------------------------------- > sql/unireg.cc | 3 +- > 8 files changed, 56 insertions(+), 94 deletions(-) > ...skip... > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 0558066..7d69ccb 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -5561,7 +5563,17 @@ TABLE *open_table_uncached(THD *thd, handlerton *hton, > strend(saved_cache_key)+1, tmp_path); > share->db_plugin= ha_lock_engine(thd, hton); > > - if (open_table_def(thd, share, GTS_TABLE | GTS_USE_DISCOVERY)) > + /* > + Use the frm image, if possible, open the file otherwise. > + > + The image might be unavailable in ALTER TABLE, when the discovering > + engine took over the ownership (see TABLE::read_frm_image). > + */ > + int res= frm->str > + ? res= share->init_from_binary_frm_image(thd, false, frm->str, frm->length) Excessive "res="? Regards, Sergey

2 1

[Maria-developers] [GSoC] Week Four!
by sriram patil 19 Jun '14

19 Jun '14

Hi All, Sorry for a little delay in posting the blog link this week. Following is the blog link about my fourth week into GSoC. http://goo.gl/W54cvE Suggestions and/or reviews are welcome! Thanks, Sriram

4 4

Re: [Maria-developers] [Commits] Rev 4258: MDEV-6039 - WebScaleSQL patches in lp:maria/10.0
by Sergei Golubchik 18 Jun '14

18 Jun '14

Hi, Sergey! On Jun 18, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 4258 > revision-id: svoj(a)mariadb.org-20140618141504-kqq6ws9gm93r2b2d > parent: svoj(a)mariadb.org-20140618081243-jxs6yo9tx9cahdv6 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Wed 2014-06-18 18:15:04 +0400 > message: > MDEV-6039 - WebScaleSQL patches > > Stop spawning dummy threads on client library initialization > > Let's revert the fix for Bug#24507. To quote Monty from 2006: > > "After 1/2 a year, when all glibc versions are updated, we can delete > this code." > > Note: The upstream glibc bug was fixed in 2006. Ok to push Regards, Sergei

1 0

Re: [Maria-developers] [GSoC] self-tuning optimizer
by Sergei Golubchik 18 Jun '14

18 Jun '14

Hi, Anshu! Please, don't forget to cc: maria-developers@ when discussing development topics. On Jun 18, Anshu Avinash wrote: > > What I meant to say was we need N "linearly independent" equations to > solve for N variables and that might not be the case. Some of the > queries might not use any factor at all or use one factor more often. Yes, you're right, of course. > > Later we'll see what to do - perhaps we'll need to decrease the limit, > > or set it exactly to N (which means, the solution won't be "best > > fitting" anymore). But this will be not in this GSoC, not your project. > > I'm sure that in this project we won't have enough factors for memory > > usage to become an issue :) > > Yes, I'll start with a basic least squares fitting model without caring > much about the memory usage for now. Sounds good. Regards, Sergei

1 0

Re: [Maria-developers] [GSoC] self-tuning optimizer
by Sergei Golubchik 18 Jun '14

18 Jun '14

Hi, Anshu! On Jun 18, Anshu Avinash wrote: > Hi serg, > > Sorry that I had been out of touch for 1-2 days. I got caught up in some > other work. > > I have read through the comments you made on MDEV and I have just started > the implementation for the first approach, the approximate method. I have > pushed a prototype at: > https://github.com/igniting/server/commit/3bd4f5b6fd20d11575207ac36e8a3cc2c… > > Here is what I'm thinking, we add to the array of measurement data, after > each query and try to solve the system of equations because we don't know > when the system of equations would become solvable. Eh. To solve a system of linear equations with N unknowns, you need N equations. So, you wait until you get N equations and solve it. Few thoughts: 1. it might be that out of all factors that you have in your data structures (currently it's 2 + 2 * MAX_HA), only few will be actually used in queries (for example, only TIME_FOR_COMPARE and read_time for InnoDB). Then you only need two queries to solve that, not 2+2*MAX_HA. But it's an optimization, try to do the basic implementation without it first. 2. You normally store these data (for N queries) in the THD, right? It might be that the thread will disconnect before it'll do the required number of queries. To avoid throwing the useful data away, you can store one extra set of data globally and when some THD disconnects and it doesn't have enough data to solve the equation, you append this data to the global set and solve that as a whole. Again, this is an extension, only do it when the basic implementation works. > However this might be time consuming, trying to solve equations after > each query. Can we store the entire data (from all the queries) and > apply a best fitting model, instead of solving them exactly? This we > will do at disconnect. I don't have a good answer for that. It's a trade-off. Either you collect exactly N equations and solve them. Then you collect the next N equations (or you get another one and delete the first one) - and solve that. And so on. Or you collect many equations, more than N. And find a best fitting solution. This way you do less solving, but you'll need more memory. I think, for now you can do that - use a best fitting solution. We only have very few factors at the moment. Still, even now you'll need some limit to how many equations you'll collect. Later we'll see what to do - perhaps we'll need to decrease the limit, or set it exactly to N (which means, the solution won't be "best fitting" anymore). But this will be not in this GSoC, not your project. I'm sure that in this project we won't have enough factors for memory usage to become an issue :) > Should it be in ~THD() or somewhere else? Yes, ~THD is fine, just make sure you only do it for real connection THD's, where SELECT queries were issues and optimizer was run. Not for various internal THD's, like the one you create in cost_factors::init. For example, as you've noticed, I've mentioned in the last comment about using read_time() result and Nrows_estimated (the same applies to scan_time(), of course). So, when ha_read_time() is called, you need to remember (in your data set) the return value of read_time() and Nrows_estimated (that's the last argument of read_time()). This way, you only solve your equations if you have this read_time() and Nrows_estimated (or scan_time()) stored. If not - the optimizer was never run and you don't do anything. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 4257: Empty commit to check email setting in commit mails in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Roberto Spadim 18 Jun '14

18 Jun '14

working :) 2014-06-18 7:53 GMT-03:00 <knielsen(a)knielsen-hq.org>: > At http://bazaar.launchpad.net/~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4257 > revision-id: knielsen(a)knielsen-hq.org-20140618105333-44ez01iye29sljur > parent: knielsen(a)knielsen-hq.org-20140618090308-frd0tuo8rycgyi32 > committer: Kristian Nielsen <knielsen(a)knielsen-hq.org> > branch nick: work-10.0 > timestamp: Wed 2014-06-18 12:53:33 +0200 > message: > Empty commit to check email setting in commit mails > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits > -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

[Maria-developers] [MDEV-4903] Regarding range support for multi-column IN predicates
by Sergey Petrunia 18 Jun '14

18 Jun '14

Hi, So, MySQL 5.7 has got support for range optimization for multi-column IN predicates: http://optimize-this.blogspot.ru/2013/12/range-access-now-in-in-predicate-n… We've also got a user issue about it: https://mariadb.atlassian.net/browse/MDEV-4903. I'm looking at their IN handling. I don't see anything apparently wrong with it. They don't process multi-column NOT IN, but one can say that they just have a limited scope. Relevant code in opt_range.cc for single-column IN: case Item_func::IN_FUNC: { Item_func_in *func=(Item_func_in*) cond_func; if (inv) { // complicated code to handle NOT IN } else { tree= get_mm_parts(param, cond_func, field, Item_func::EQ_FUNC, func->arguments()[1], cmp_type); if (tree) { Item **arg, **end; for (arg= func->arguments()+2, end= arg+func->argument_count()-2; arg < end ; arg++) { tree= tree_or(param, tree, get_mm_parts(param, cond_func, field, Item_func::EQ_FUNC, *arg, cmp_type)); } } } mysql-5.7 extends this approach to handle multi-column case (by invoking get_mm_parts() for each compared value, then doing tree_and(), and then tree_or()). Should we just go ahead and backport it? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] problem with update and myisam
by Roberto Spadim 18 Jun '14

18 Jun '14

hi guys, could anyone help me with https://mariadb.atlassian.net/browse/MDEV-6355 thanks -- Roberto Spadim

2 5

Re: [Maria-developers] [Commits] 0eb94ca: MDEV-6137 better help for SET/ENUM sysvars
by Sergey Vojtovich 17 Jun '14

17 Jun '14

Hi Sergei, ok to push. A few minor questions inline. On Mon, Jun 16, 2014 at 06:51:53PM +0200, serg(a)mariadb.org wrote: > revision-id: 0eb94ca810500977eb0336ee30b19e9c0b769bca > parent(s): f61f36b386e8d0c6448297bb84c92e8d9b82be6a > committer: Sergei Golubchik > branch nick: maria > timestamp: 2014-06-16 18:43:26 +0200 > message: > > MDEV-6137 better help for SET/ENUM sysvars > > Auto-generate the allowed list of values for enum/set/flagset options > in --help output. But don't do that when the help text already has them. > > Also, remove lists of values from help strings of various options, where > they were simply listed without any additional information. > > --- > mysys/my_getopt.c | 81 +++++++++++++++++++++++++++------ > plugin/server_audit/server_audit.c | 2 +- > sql/log.cc | 3 +- > sql/mysqld.cc | 8 ++-- > sql/sql_plugin.cc | 2 +- > sql/sys_vars.cc | 90 ++++++++----------------------------- > storage/maria/ha_maria.cc | 14 ++---- > storage/myisam/ha_myisam.cc | 7 ++- > storage/xtradb/handler/ha_innodb.cc | 35 +++++++-------- > 9 files changed, 116 insertions(+), 126 deletions(-) Hmmm... no test case affected? > > diff --git a/mysys/my_getopt.c b/mysys/my_getopt.c > index 2e97417..3662757 100644 > --- a/mysys/my_getopt.c > +++ b/mysys/my_getopt.c > @@ -1300,6 +1300,48 @@ static uint print_name(const struct my_option *optp) > return s - optp->name; > } > > +/** prints option comment with indentation and wrapping. > + > + The comment column starts at startpos, and has width of width > + Current cursor position is curpos, returns new cursor position > + > +Note: can print one character beyond width! The above line, is it properly indented? > +*/ > +static uint print_comment(const char *comment, > + int curpos, int startpos, int width) Probably make width local to this function, it doesn't seem to be used by the caller anyway? > +{ > + const char *end= strend(comment); > + int endpos= startpos + width; > + > + for (; curpos < startpos; curpos++) > + putchar(' '); > + > + if (*comment == '.' || *comment == ',') > + { > + putchar(*comment); > + comment++; > + curpos++; > + } > + > + while (end - comment > endpos - curpos) > + { > + const char *line_end; > + for (line_end= comment + endpos - curpos; > + line_end > comment && *line_end != ' '; > + line_end--); > + for (; comment < line_end; comment++) > + putchar(*comment); I would probably replace it with fwrite(), but fine with this form too. > + while (*comment == ' ') > + comment++; /* skip the space, as a newline will take it's place now */ > + putchar('\n'); > + for (curpos= 0; curpos < startpos; curpos++) > + putchar(' '); > + } > + printf("%s", comment); > + return curpos + (end - comment); > +} > + > + > /* > function: my_print_options > > @@ -1309,12 +1351,12 @@ static uint print_name(const struct my_option *optp) > void my_print_help(const struct my_option *options) > { > uint col, name_space= 22, comment_space= 57; > - const char *line_end; > const struct my_option *optp; > DBUG_ENTER("my_print_help"); > > for (optp= options; optp->name; optp++) > { > + const char *typelib_help= 0; > if (!optp->comment) > continue; > if (optp->id && optp->id < 256) > @@ -1359,25 +1401,36 @@ void my_print_help(const struct my_option *options) > col= 0; > } > } > - for (; col < name_space; col++) > - putchar(' '); > if (optp->comment && *optp->comment) Cleanup suggestion: optp->comment can't be NULL here. > { > - const char *comment= optp->comment, *end= strend(comment); > + col= print_comment(optp->comment, col, name_space, comment_space); > > - while ((uint) (end - comment) > comment_space) > + switch (optp->var_type & GET_TYPE_MASK) { > + case GET_ENUM: > + typelib_help= ". One of: "; > + break; > + case GET_SET: > + typelib_help= ". Any combination of: "; > + break; > + case GET_FLAGSET: > + typelib_help= ". Takes a comma-separated list of option=value pairs, " > + "where value is on or off, and options are: "; > + break; > + } > + if (typelib_help && > + strstr(optp->comment, optp->typelib->type_names[0]) == NULL) Should we handle case when optp->typelib == NULL || optp->typelib->type_names[0] == NULL? Probably not. I believe this condition is fragile: comment string may mention type_names[0], but not list possible values. OTOH since this condition is for very limited data set it should cause problems only in rare cases. > { > - for (line_end= comment + comment_space; *line_end != ' '; line_end--); > - for (; comment != line_end; comment++) > - putchar(*comment); > - comment++; /* skip the space, as a newline will take it's place now */ > - putchar('\n'); > - for (col= 0; col < name_space; col++) > - putchar(' '); > + int i; > + col= print_comment(typelib_help, col, name_space, comment_space); > + col= print_comment(optp->typelib->type_names[0], col, name_space, comment_space); > + for (i= 1; i < optp->typelib->count; i++) > + { > + col= print_comment(", ", col, name_space, comment_space); > + col= print_comment(optp->typelib->type_names[i], col, name_space, comment_space); > + } > } > - printf("%s", comment); > + putchar('\n'); > } > - putchar('\n'); Does this mean if comment is empty there won't be a new line? > if ((optp->var_type & GET_TYPE_MASK) == GET_BOOL) > { > if (optp->def_value != 0) ...skip... > @@ -2785,7 +2745,7 @@ static Sys_var_mybool Sys_master_verify_checksum( > > /* These names must match RPL_SKIP_XXX #defines in slave.h. */ > static const char *replicate_events_marked_for_skip_names[]= { > - "replicate", "filter_on_slave", "filter_on_master", 0 > + "REPLICATE", "FILTER_ON_SLAVE", "FILTER_ON_MASTER", 0 > }; Does this mean there is a convention that values must be in upper case? ...skip... Regards, Sergey

2 1

[Maria-developers] WebScaleSQL: Actually fix all the array bounds warnings
by Sergey Vojtovich 16 Jun '14

16 Jun '14

Hi Sergei and Alexey, during our recent discussion re WebScaleSQL patches we agreed to review $subj: https://github.com/webscalesql/webscalesql-5.6/commit/094187c4a20e8d0c02f17… All these warnings come from strings/decimal.c. I was able to reproduce them using gcc-4.8.1, cmake -DBUILD_CONFIG=mysql_release (basically needs -Wall -O3). I simplified this function to the following: gcc -Wall -O3 -c dec.c -o dec.o static const int powers10[2 + 1]= { 1, 10, 100 }; int remove_leading_zeroes(int decimals, int var) { decimals%= 2; while (var < powers10[decimals--]) /* no-op */; return decimals; } The only allowed value for decimals is 1 and var must be > 0. Similar assertion is enforced by original function. So far I see nothing wrong with this code. I believe we should skip this patch, but still we should probably do something about these warning. What's your take? Regards, Sergey

2 2

[Maria-developers] MariaDB 10.0.12 release prep
by Daniel Bartholomew 14 Jun '14

14 Jun '14

I've begun prep work for the MariaDB 10.0.12 release. Draft Release Notes and Changelog are here: - https://mariadb.com/kb/en/mariadb-10012-release-notes/ - https://mariadb.com/kb/en/mariadb-10012-changelog/ Please add/modify/etc... as needed. Current plan is to announce the release on Mon, 16 Jun 2014. Thanks! -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

1 0

Re: [Maria-developers] Rev 3986: MDEV-5231: Per query variables from Percona Server
by Sergei Golubchik 14 Jun '14

14 Jun '14

Hi, Sanja! On May 22, sanja(a)askmonty.org wrote: > === added file 'mysql-test/suite/percona/percona_rpl_stm_per_query_variables_settings.test' > --- a/mysql-test/suite/percona/percona_rpl_stm_per_query_variables_settings.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/percona/percona_rpl_stm_per_query_variables_settings.test 2014-05-22 12:50:01 +0000 could you rename tests to have shorter names? rpl_set_statement looks good enough to me > @@ -0,0 +1,57 @@ > +--source include/master-slave.inc > +--source include/have_binlog_format_statement.inc > + > +--disable_warnings > +DROP TABLE IF EXISTS t1; > +DROP TABLE IF EXISTS t2; > +--enable_warnings > + > +call mtr.add_suppression("Unsafe statement written to the binary log*"); > +CREATE TABLE t1 (a bigint unsigned not null); > +CREATE TABLE t2 (a char(255) not null); okay, although I wouldn't bother creating *two* tables, I'd create one and hard-coded it into percona_rpl_set_statement_variable_test.inc (removing rpl_ssvt_table parameter) > + > +--echo > +--echo There are the following types of variables: > +--echo 1) variables that are NOT replicated correctly when using STATEMENT mode; > +--echo > + > +--let $rpl_ssvt_var_name=max_join_size > +--let $rpl_ssvt_var_value=2 > +--let $rpl_ssvt_table=t1 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > + > +--echo > +--echo 2) variables thar ARE replicated correctly > +--echo They must be replicated correctly with "SET STATEMENT" too. > +--echo > +--let $rpl_ssvt_var_name=auto_increment_increment > +--let $rpl_ssvt_var_value=10 > +--let $rpl_ssvt_table=t1 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > + > +--echo > +--echo 3) sql_mode which is replicated correctly exept NO_DIR_IN_CREATE value; > +--echo > +--let $rpl_ssvt_var_name=sql_mode > +--let $rpl_ssvt_var_value='ERROR_FOR_DIVISION_BY_ZERO' > +--let $rpl_ssvt_table=t2 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > +--let $rpl_ssvt_var_name=sql_mode > +--let $rpl_ssvt_var_value='NO_DIR_IN_CREATE' > +--let $rpl_ssvt_table=t2 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > + > +--echo > +--echo 4) variables that are not replicated at all: > +--echo default_storage_engine, storage_engine, max_heap_table_size > +--echo > +--let $rpl_ssvt_var_name=max_heap_table_size > +--let $rpl_ssvt_var_value=16384 > +--let $rpl_ssvt_table=t1 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > + > +connection master; > +DROP TABLE t1; > +DROP TABLE t2; > +sync_slave_with_master; > +source include/stop_slave.inc; > > === added file 'mysql-test/suite/percona/percona_statement_set.result' > --- a/mysql-test/suite/percona/percona_statement_set.result 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/percona/percona_statement_set.result 2014-05-22 12:50:01 +0000 reverse polish notation in percona someone loves. starwars too much seen he apparently. Please, rename to "set_statement.result" > @@ -0,0 +1,875 @@ > +'# SET STATEMENT ..... FOR .... TEST' > +DROP TABLE IF EXISTS t1; > +DROP FUNCTION IF EXISTS myProc; > +DROP PROCEDURE IF EXISTS p1; > +DROP PROCEDURE IF EXISTS p2; > +DROP PROCEDURE IF EXISTS p3; > +DROP PROCEDURE IF EXISTS p4; > +DROP PROCEDURE IF EXISTS p5; > +DROP TABLE IF EXISTS STATEMENT; > +'# Setup database' > +CREATE TABLE t1 (v1 INT, v2 INT); > +INSERT INTO t1 VALUES (1,2); > +INSERT INTO t1 VALUES (3,4); > +'' > +'#------------------ STATEMENT Test 1 -----------------------#' > +'# Initialize variables to known setting' > +SET SESSION sort_buffer_size=100000; > +'' > +'# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +Variable_name Value > +sort_buffer_size 100000 > +SET STATEMENT sort_buffer_size=150000 FOR SELECT * FROM t1; > +v1 v2 > +1 2 > +3 4 > +'' > +'# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +Variable_name Value > +sort_buffer_size 100000 > +'' > +'#------------------ STATEMENT Test 2 -----------------------#' > +'# Initialize variables to known setting' > +SET SESSION binlog_format=mixed; > +SET SESSION sort_buffer_size=100000; > +'# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +Variable_name Value > +sort_buffer_size 100000 > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +Variable_name Value > +binlog_format MIXED > +SET STATEMENT sort_buffer_size=150000, binlog_format=row > +FOR SELECT * FROM t1; > +v1 v2 > +1 2 > +3 4 > +'# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +Variable_name Value > +sort_buffer_size 100000 > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +Variable_name Value > +binlog_format MIXED > +'' > +'#------------------ STATEMENT Test 3 -----------------------#' > +'# set initial variable value, make prepared statement > +SET SESSION binlog_format=row; > +PREPARE stmt1 FROM 'SET STATEMENT binlog_format=row FOR SELECT * FROM t1'; > +'' > +'# Change variable setting' > +SET SESSION binlog_format=mixed; > +'' > +'# Pre-STATEMENT variable value' > +'' > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +Variable_name Value > +binlog_format MIXED > +'' > +EXECUTE stmt1; > +v1 v2 > +1 2 > +3 4 > +'' > +'# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +Variable_name Value > +binlog_format MIXED > +'' > +DEALLOCATE PREPARE stmt1; > +'#------------------ STATEMENT Test 4 -----------------------#' > +'# set initial variable value, make prepared statement > +SET SESSION myisam_sort_buffer_size=500000, myisam_repair_threads=1; > +'' > +'# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'myisam_sort_buffer_size'; > +Variable_name Value > +myisam_sort_buffer_size 500000 > +SHOW SESSION VARIABLES LIKE 'myisam_repair_threads'; > +Variable_name Value > +myisam_repair_threads 1 > +'' > +SET STATEMENT myisam_sort_buffer_size=800000, > +myisam_repair_threads=2 FOR OPTIMIZE TABLE t1; how comes that is "make prepared statement" (see comment above) ? > +Table Op Msg_type Msg_text > +test.t1 optimize status OK > +'' > +'# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'myisam_sort_buffer_size'; > +Variable_name Value > +myisam_sort_buffer_size 500000 > +SHOW SESSION VARIABLES LIKE 'myisam_repair_threads'; > +Variable_name Value > +myisam_repair_threads 1 > +'' ... > === added file 'mysql-test/suite/percona/percona_statement_set.test' > --- a/mysql-test/suite/percona/percona_statement_set.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/percona/percona_statement_set.test 2014-05-22 12:50:01 +0000 > @@ -0,0 +1,847 @@ > +--echo '# SET STATEMENT ..... FOR .... TEST' many of the tests below are pretty useless. they don't test that the value is actually set. See below: > +############################ STATEMENT_SET ############################# > +# # > +# Testing working functionality of SET STATEMENT # > +# # > +# # > +# There is important documentation within # > +# # > +# # > +# Author: Joe Lukas # > +# Creation: # > +# 2009-08-02 Implement this test as part of # > +# WL#681 Per query variable settings # > +# # > +######################################################################## > + > +--disable_warnings > +DROP TABLE IF EXISTS t1; > +DROP FUNCTION IF EXISTS myProc; > +DROP PROCEDURE IF EXISTS p1; > +DROP PROCEDURE IF EXISTS p2; > +DROP PROCEDURE IF EXISTS p3; > +DROP PROCEDURE IF EXISTS p4; > +DROP PROCEDURE IF EXISTS p5; > +DROP TABLE IF EXISTS STATEMENT; > +--enable_warnings > +#################################################################### > +#Set up current database > +#################################################################### > +--echo '# Setup database' > +CREATE TABLE t1 (v1 INT, v2 INT); > +INSERT INTO t1 VALUES (1,2); > +INSERT INTO t1 VALUES (3,4); > +--echo '' > +--echo '#------------------ STATEMENT Test 1 -----------------------#' > +#################################################################### > +# Checks with variable value type ulong # > +#################################################################### > +--echo '# Initialize variables to known setting' > +SET SESSION sort_buffer_size=100000; > +--echo '' > +--echo '# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +SET STATEMENT sort_buffer_size=150000 FOR SELECT * FROM t1; How do you know that SET STATEMENT did anything at all? > +--echo '' > +--echo '# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +--echo '' > +--echo '#------------------ STATEMENT Test 2 -----------------------#' > +#################################################################### > +# Checks for multiple set values inside STATEMENT ... FOR # > +#################################################################### > +--echo '# Initialize variables to known setting' > +SET SESSION binlog_format=mixed; > +SET SESSION sort_buffer_size=100000; > +--echo '# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +SET STATEMENT sort_buffer_size=150000, binlog_format=row > + FOR SELECT * FROM t1; > +--echo '# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > + > +--echo '' > +--echo '#------------------ STATEMENT Test 3 -----------------------#' > +#################################################################### > +# Check current variable value is stored in using stored # > +# statements. # > +#################################################################### > +--echo '# set initial variable value, make prepared statement > +SET SESSION binlog_format=row; > +PREPARE stmt1 FROM 'SET STATEMENT binlog_format=row FOR SELECT * FROM t1'; How will you know that SET STATEMENT did anything at all? (the same applies to most tests below, I'll stop repeating it) > +--echo '' > +--echo '# Change variable setting' > +SET SESSION binlog_format=mixed; > +--echo '' > +--echo '# Pre-STATEMENT variable value' > +--echo '' > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +--echo '' > +EXECUTE stmt1; > +--echo '' > +--echo '# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > + > +--echo '' > +DEALLOCATE PREPARE stmt1; ... > === modified file 'sql/share/errmsg-utf8.txt' > --- a/sql/share/errmsg-utf8.txt 2014-02-11 13:21:48 +0000 > +++ b/sql/share/errmsg-utf8.txt 2014-05-22 12:50:01 +0000 > @@ -7067,3 +7067,5 @@ ER_IT_IS_A_VIEW 42S02 > eng "'%-.192s' is a view" > ER_SLAVE_SKIP_NOT_IN_GTID > eng "When using GTID, @@sql_slave_skip_counter can not be used. Instead, setting @@gtid_slave_pos explicitly can be used to skip to after a given GTID position." > +ER_SET_STATEMENT_TABLES_IS_NOT_SUPPORTED > + eng "SET STATEMENT does not support using tables in the assignment variables expressions" why not? > > === modified file 'sql/sql_parse.cc' > --- a/sql/sql_parse.cc 2014-03-29 10:33:25 +0000 > +++ b/sql/sql_parse.cc 2014-05-22 12:50:01 +0000 > @@ -2430,6 +2432,33 @@ mysql_execute_command(THD *thd) > thd->get_binlog_format(&orig_binlog_format, > &orig_current_stmt_binlog_format); > > + if (lex->set_statement && !lex->var_list.is_empty()) Hmm? No separate list of variables, but a common lex->var_list? Then the following won't work: SET STATEMENT A=1 SET B=2 I remember in the WL and in GSoC we agreed that for SET STATEMENT A=2 SET A=2 (same variable in SET STATEMENT and SET), it's acceptable to restore the original value of A, basically ignoring SET. But I doubt that the same logic applies when SET STATEMENT and SET modify *different* variables. > + { > + MEM_ROOT *save_mem_root; > + /* > + We should be sure that variables backup and assignment made on > + runtime memory > + */ > + save_mem_root= thd->mem_root; > + thd->mem_root= thd->get_execution_mem_root(); Why? > + per_query_variables_backup= copy_system_variables(&thd->variables); > + if (!per_query_variables_backup || > + (res= sql_set_variables(thd, &lex->var_list))) > + { > + thd->mem_root= save_mem_root; > + /* > + We encountered some sort of error, but no message was sent. > + Send something semi-generic here since we don't know which > + assignment in the list caused the error. > + */ > + if (!thd->is_error()) > + my_error(ER_WRONG_ARGUMENTS, MYF(0), "SET"); > + goto error; > + } > + thd->mem_root= save_mem_root; > + } > + > /* > Force statement logging for DDL commands to allow us to update > privilege, system or statistic tables directly without the updates > @@ -5144,6 +5179,30 @@ finish: > > lex->unit.cleanup(); > > + if (lex->set_statement && !lex->var_list.is_empty()) { > + List_iterator_fast<set_var_base> it(thd->lex->var_list); > + set_var *var; > + > + free_system_variables(&thd->variables); > + thd->variables= *per_query_variables_backup; > + /* per_query_variables_backup was in a mem_root */ > + per_query_variables_backup= NULL; > + /* > + When variables are restored after "SET STATEMENT ... FOR ..." statement > + execution an update callback must be invoked for the system variables > + to save special logic if it is. set_var_base class does not contain > + refference to variable as it is just an interface class. But only > + system variables are allowed to be used in "SET STATEMENT ... FOR ..." > + statement, so cast from set_var_base* to set_var* can be used here. > + */ > + while ((var=(set_var *)it++)) > + { > + var->var->stmt_update(thd); > + } I don't think it'll work. Compare set_var::update() and set_var::stmt_update(). The former calls ::session_update(), while the latter does not. Normally, session_update() simply updates the value, performs the assignment. And it is, indeed, not needed here. but in some custom classes session_update() is more complex, and it's not safe to omit it. Please 1) see all non-trivial implementations of sys_var::session_update() 2) write SET SESSION tests for these variables 3) if they'll all work - please explain why 4) if they won't - I'm afraid, this whole approach needs to be redone, then > + > + thd->lex->set_statement= false; > + } > + > if (! thd->in_sub_stmt) > { > if (thd->killed != NOT_KILLED) > > === modified file 'sql/sql_plugin.cc' > --- a/sql/sql_plugin.cc 2014-03-26 21:25:38 +0000 > +++ b/sql/sql_plugin.cc 2014-05-22 12:50:01 +0000 > @@ -4088,3 +4088,48 @@ static void restore_ptr_backup(uint n, s > (backup++)->restore(); > } > > + > +/** > + Create deep copy of system_variables instance. > +*/ > +struct system_variables * > +copy_system_variables(const struct system_variables *src) What is it doing in sql_plugin.cc? > +{ > + struct system_variables *dst; > + > + DBUG_ASSERT(src); > + > + dst= (struct system_variables *) > + sql_alloc(sizeof(struct system_variables)); > + if (!dst) > + return NULL; > + *dst= *src; > + > + if (dst->dynamic_variables_ptr) > + { > + if (!(dst->dynamic_variables_ptr= > + (char *)my_malloc(dst->dynamic_variables_size, MYF(MY_WME | MY_FAE)))) > + return NULL; > + memcpy(dst->dynamic_variables_ptr, > + src->dynamic_variables_ptr, > + src->dynamic_variables_size); > + } > + > + mysql_mutex_lock(&LOCK_plugin); > + dst->table_plugin= > + intern_plugin_lock(NULL, src->table_plugin); I'm surprised that plugin locking actually works in here, (I believe it does, otherwise you'd noticed it). What about SET STATEMENT default_engine=MEMORY ? What about SET STATEMENT default_engine=MyISAM (where MyISAM is already default)? please, try both in debug build. > + mysql_mutex_unlock(&LOCK_plugin); > + > + return dst; > +} > + > +void free_system_variables(struct system_variables *v) > +{ > + DBUG_ASSERT(v); > + > + mysql_mutex_lock(&LOCK_plugin); > + intern_plugin_unlock(NULL, v->table_plugin); > + mysql_mutex_unlock(&LOCK_plugin); > + > + my_free(v->dynamic_variables_ptr); > +} > > === modified file 'sql/sql_yacc.yy' > --- a/sql/sql_yacc.yy 2014-03-26 21:25:38 +0000 > +++ b/sql/sql_yacc.yy 2014-05-22 12:50:01 +0000 > @@ -813,8 +813,10 @@ static void sp_create_assignment_lex(THD > lex->one_shot_set= 0; > lex->autocommit= 0; > /* get_ptr() is only correct with no lookahead. */ > - DBUG_ASSERT(no_lookahead); > + if (no_lookahead) > lex->sphead->m_tmp_query= lip->get_ptr(); > + else > + lex->sphead->m_tmp_query= lip->get_tok_end(); Why? > /* Inherit from outer lex. */ > lex->option_type= old_lex->option_type; > } > @@ -955,10 +957,10 @@ bool my_yyoverflow(short **a, YYSTYPE ** > %parse-param { THD *thd } > %lex-param { THD *thd } > /* > - Currently there are 163 shift/reduce conflicts. > + Currently there are 164 shift/reduce conflicts. > We should not introduce new conflicts any more. > */ > -%expect 163 > +%expect 164 Why? > > /* > Comments for TOKENS. > @@ -14427,8 +14431,44 @@ set: > } > start_option_value_list > {} > + | SET STATEMENT_SYM > + { > + LEX *lex= Lex; > + mysql_init_select(lex); > + lex->sql_command= SQLCOM_SET_OPTION; > + /* Don't clear var_list in the case of recursive statement */ > + if (!lex->set_statement) > + lex->var_list.empty(); When var_list is not empty here (and set_statement not set)? In other words, when this statement will actually have any effect? > + lex->one_shot_set= 0; > + lex->autocommit= 0; > + lex->set_statement= true; > + sp_head *sp= lex->sphead; > + if (sp && !sp->is_invoked()) > + { > + sp->m_param_begin= ((yychar == YYEMPTY) ? YYLIP->get_ptr() : YYLIP->get_tok_start()); > + sp->m_param_end= ((yychar == YYEMPTY) ? YYLIP->get_ptr() : YYLIP->get_tok_end()); Why? > + } > + } > + set_stmt_option_value_following_option_type_list Eh? What does "set stmt option value following option type list" mean? Please, rename to set_stmt_option_value_list or even set_stmt_list > + { > + if (Lex->query_tables) > + { > + my_error(ER_SET_STATEMENT_TABLES_IS_NOT_SUPPORTED, MYF(0)); > + MYSQL_YYABORT; > + } > + } > + FOR_SYM statement > + {} > ; > > +set_stmt_option_value_following_option_type_list: > + /* > + Only system variables can be used here. If this condition is changed > + please check careful code under lex->option_type == OPT_STATEMENT > + condition on wrong type casts. > + */ > + option_value_following_option_type > + | set_stmt_option_value_following_option_type_list ',' option_value_following_option_type > > // Start of option value list > start_option_value_list: Regards, Sergei

1 0

Re: [Maria-developers] [GSoC] self-tuning optimizer
by Sergei Golubchik 13 Jun '14

13 Jun '14

Hi, Anshu! On Jun 13, Anshu Avinash wrote: > > So I have refactored Cost_factors::init() into Cost_factors::init() and > Cost_factors::re_init(). The diff is at: > https://github.com/igniting/server/commit/41e360309c067d072d788495e4e948054… Looks great. I've added one comment there. > I had started writing a test for checking whether if loading a new > storage engine works or not. I loaded 'Example' storage engine, but > was unable to test the current t1 table on it, as t1 contains a > primary key. Also I need to write test cases for 'time_for_compare'. > Any hints on that? Hm. Yes. Example engine, is pretty useless for you, as you cannot create tables in it. You can use InnoDB plugin. But create a separate test file for that. For example (I'll use the short name below, don't commit it): a.test: ============================================== if (!$HA_INNODB_SO) { skip Need InnoDB plugin; } install plugin innodb soname 'ha_innodb'; ... do your tests here uninstall plugin innodb; ============================================== you will also need a.opt file: ============================================== --ignore-builtin-innodb --loose-innodb ============================================== As for time_for_compare... Here's a universal approach, you can do it yourself. This time_for_compare is 5 by default, right? Change it to be, say, 500, and run existing tests (start from the main suite, and there from optimizer tests, e.g. from select.test). If some test will fail - here's your query, extract it, simplify, perhaps, and put in your test file. That's exactly what I just did :) And here's a simple test: create table t1 (a blob, index(a(20))); create table t2 (a blob, index(a(20))); insert into t1 values ('one'),('two'),('three'),('four'),('five'); insert into t2 values ('one'),('two'),('three'),('four'),('five'); insert into t2 values ('one'),('two'),('three'),('four'),('five'); drop table t1, t2; Perhaps you can get it down to one table - I didn't try. > Also I think I can start with the next part, i.e. collect statistics. We > can have a structure like: > > struct Cost_factor_stats > { > char *cost_factor_name; > double cost_factor_value; > double total_queries; total_queries should be ulonglong, not double. > double total_query_time; > double total_query_time_squared; > }; > > We will be updating these structures after every query per THD and write > the data into a persistent table on disconnect. Is this workflow correct? Almost. You need a Cost_factor_stats structure per cost factor (and per engine where appropriate). You don't need the name there. You have these structures per THD and globally. Basically, instead of your current double time_for_compare_val; you will have Cost_factor_stats time_for_compare_val; hmm, perhaps better to name the structure simply "Cost_factor": Cost_factor time_for_compare; And you put them all in one structure: struct Engine_cost_factors { Cost_factor read_time; Cost_factor scan_time; }; struct Global_cost_factors { Cost_factor time_for_compare; Cost_factor time_for_compare_rowid; }; class Cost_factors { Global_cost_factors global; Engine_cost_factors engine[MAX_HA]; double time_for_compare() const { return global.read_time.value; } double time_for_compare_rowid() const; double read_time(const handler*) const; double scan_time(const handler*) const; }; Now, you stick this structure in the THD, and also declare one instance globally. On connect you initialize THD::Cost_factor values from the global Cost_factor object. On disconnect you add THD::Cost_factor values back to the global Cost_factor object. There's no writing to disk so far, we'll think about it later (you don't want to write to disk on *every* disconnect). > Few more questions: > 1.What will we do if the query had used more than one cost factor? > 2.How would joins be handled? Yes, a query will certainly use more than one cost factor. In fact, this is something we might want to experiment with. I see two approaches - exact and approximate. Approximate one is described in MDEV. Say, you've counted 500 index reads, 20,000 sequential row reads (table scan), some 10,000,000 comparisons, etc. And you know how long the query took. For another query you've counted different numbers, and so on. Basically, you'll have a system of linear equations that one can solve to know how the time was split. The problem with this approach - the server is doing much more than purely index reads, table scans, and comparisons. So this unaccounted time will be split between cost factors. I don't know if it's a problem. The exact approach - we only measure times spent on the actual operation. Basically, you measure how long *each row read* took, how long *each index read* took, etc. This will show no unaccounted time. But it's more expensive, it might be too expensive to be practically useful - I don't know that. These two approaches need different data structures, the first one needs more memory, because you cannot start aggregating before you solve the system of equations. Basically, if you have four constant (time_for_compare, time_for_compare_rowid, read_time, scan_time, only one engine), you'll need four equations, that is, per THD you need to store four measurements: struct measurement { ulong time_for_compare; ulong time_for_compare_rowid; struct per_engine { ulong scan_time; ulong read_time; } per_engine[MAX_HA]; double total_time; }; and in THD you need struct measurement data[4]; Also, note that I'm writing per_engine[MAX_HA], but it's a simplification, good enough for a prototype. But in the final code we cannot allocate that much memory per THD, as most of the time we'll have only few engines, not MAX_HA. Regards, Sergei

1 0

[Maria-developers] WebScaleSQL: Adjust linking flags to not embed some libs
by Sergey Vojtovich 13 Jun '14

13 Jun '14

Hi Sergei, during our recent discussion re WebScaleSQL patches we agreed to accept $subj: https://github.com/webscalesql/webscalesql-5.6/commit/361e3b83145402945b737… I studied our binaries and it looks like we embed zlib and ssl libraries only if WITH_ZLIB=bundled/WITH_SSL=bundled. E.g. DEB/RPM packages use system libs and do not embed zlib/ssl, whereas tarball uses bundled zlib/yassl and embeds them. Not embedding yassl into the client library will make the latter almost useless. Same probably applies to zlib even though it is normally offered by OS-es. Said the above I believe our build scripts behave properly and we should skip this patch. What's your take? Regards, Sergey

2 1

[Maria-developers] Java Connector Coding till 12th June
by Puneet Dewan 13 Jun '14

13 Jun '14

Hi Everyone, This week, I was able to send the com_stmt_execute command from the *Driver * in the way it is accepted by the MariaDB server So, Now I am able to insert a row in the database using com_stmt_execute command. But from the console when I try to see the values I see *NULL* I used *Wireshark* to capture the packets,but I can see the value(Varchar /Int/..etc) is passed in the packet which wireshark captures. I am investigating the code but could not find anything till now. In Wireshark it shows* [Expert Info (warn/undecoded) :FIXME: Excute Dissector Incomplete]* Now I think I am not sending the values in a correct format.Will look and try to resolve it. Regards Puneet. On Thu, Jun 5, 2014 at 10:42 PM, Puneet Dewan <puneetd30(a)gmail.com> wrote: > Hi Everyone, > > The work done till 5th June: > 1.I started studying the connector immediately after my semester exams got > over(5th May). > 2. Understood the relationship between different classes. > 3.Read the MySql documentation for text protocol and map the commands from > documentation onto code.This thing gave a clear idea about how things are > moving from client to server. > 4.Setup the code on Ubuntu using eclipse and tested against database. > Setup the Team branch in launchpad. > 5.Debug the test classes to see how different classes are interlinked and > understood that MySQLProtocol.java is the heart of our JDBC driver. > 6.Then I started reading Client Server Protocol documentation i.e > ServerSide prepared statement and try to see how and where I have to code > it. > 7.Then I saw that there is a class MySqlServerSidePreparedStatement.java, > Initially I coded the prepareStatement() ,close(),and execute() in that > class seperately. > But after (Code review) discussion with Massimo, we donot need to use that > class. > Instead use MySqlPreparedStatement.java to do the same work.So made > changes accordingly. > 8.I had pushed some of the code which I have written to launchpad and > discussed with Massimo last week. > 9.Last week I was struck with Wireshark on windows and ubuntu both. > I was unable to see the packets going to database and seeing the response. > 10.On Monday the problem was resolved as I was using the wrong interface.I > should have used loopback.io(127.0.0.1) instead on ubuntu(thanks to > Georg). > Unfortunately the thing does not work on windows.Tried a lot many times. > 11.Now I am working on the execute method of client server protocol as I > am getting exception while using that method.Wireshark is playing a great > role in how the packets are flowing. > > From now I will do weekly blogging and keep everyone updated. > > Thanks & Regards > Puneet. >

1 0

[Maria-developers] Allow to turn off max_connect_errors
by Honza Horak 13 Jun '14

13 Jun '14

Hi guys, there are apparently some tools out there [1], that check if the server is up quite often, while not closing the connection properly. It eventually ends in 'many connection errors', because max_connect_errors is always limited now. I understand that this way of checking may be wrong, but there may be scenarios where we do not want to check for `max_connect_errors` at all. So, would it be acceptable for mariadb to change behaviour of max_connect_errors option, so that it accepts also 0 as a possible value, which would mean 'do not check connect errors at all'? I'm bringing the idea here first, but will submit a report and possibly patch if it does not seem to be undesired behaviour. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1104957 TIA and regards, Honza

4 4

Re: [Maria-developers] [Commits] Rev 4018: MDEV-4784 - merge test cases from 5.6 in lp:maria/10.0
by Sergei Golubchik 12 Jun '14

12 Jun '14

Hi, Sergey! > === added file 'mysql-test/include/have_daemon_example_plugin.inc' > --- a/mysql-test/include/have_daemon_example_plugin.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/have_daemon_example_plugin.inc 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,16 @@ > +disable_query_log; > +# > +# Check if server has support for loading plugins > +# > +if (`SELECT @@have_dynamic_loading != 'YES'`) { > + --skip daemon example plugin requires dynamic loading > +} > + > +# > +# Check if the variable DAEMONEXAMPLE is set > +# > +if (!$LIBDAEMON_EXAMPLE_SO) { > + --skip daemon_example plugin requires the environment variable \$DAEMONEXAMPLE to be set (normally done by mtr) Eh? It was in your first patch. I thought it's an additional patch, but it looks like it includes the first one? Okay, I will copy all my comments from the first patch here to have a complete review in one email. So, here I said ************************ it's always done by mtr, a better error message should be "No libdaemon_example.so was found" or simply "Need daemon_example plugin" and, btw, the first if() is redundant, if no dynamic loading was enabled no dynamic plugins will be built whatsoever. > +} > + > +enable_query_log; > > === added file 'mysql-test/include/ib_logfile_size_check.inc' > --- a/mysql-test/include/ib_logfile_size_check.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/ib_logfile_size_check.inc 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,14 @@ I'm not a great fan of obscure include files that are only used once in one single test. Perhaps it's better to inline it? > +perl; > + my $dir = $ENV{'MYSQLD_DATADIR'}; > + my $size; > + opendir(DIR, $dir) or die $!; > + while (my $file = readdir(DIR)) > + { > + > + next unless ($file =~ m/\ib_logfile.$/); > + $size = -s "$dir/$file"; > + print "The size of the ib_logfile(0/1): $size \n"; > + } > + close(DIR); > + exit(0) > +EOF > > === added file 'mysql-test/include/vardir_size_check.inc' > --- a/mysql-test/include/vardir_size_check.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/vardir_size_check.inc 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,26 @@ Same here. Although it's worse than with the previous file :) ib_logfile_size_check.inc is only used in innodb_log_file_size_functionality.test. This file - too. But in 5.6 this one is also used in query_cache_type_functionality, table_open_cache_functionality, and table_definition_cache_functionality, sort_buffer_size_functionality, query_cache_size_functionality. I suppose somebody just copied innodb_log_file_size_functionality over and edited it. I don't see why would anyone really want to check the vardir size in all these other tests. So, yes, better to inline this file too, to avoid silly mistakes like that. > +--echo Check the size of the vardir > +--echo The output size is in unit blocks > +perl; > +#!/usr/bin/perl > +use warnings; > +use strict; > +use File::Find; > +my $dir = $ENV{'MYSQLTEST_VARDIR'}; > +my $size = 0; > +find( sub { $size += -f $_ ? -s _ : 0 }, $dir ); > + > +if ( $size < 1717987000 ) > + { print "TRUE", "\n";} > +else > + { print "FALSE $size", "\n";} > + > + > +## Expected size for the VAR dir being changed for the PB2 runs > + > +##if ( $size =~ /^496[\d]{5}$/ ) > +## { > +## { print "TRUE", "\n";} > +## } > +##else > +## { print "FALSE $size", "\n";} > +EOF > > === modified file 'mysql-test/mysql-test-run.pl' > --- a/mysql-test/mysql-test-run.pl 2014-02-26 14:28:07 +0000 > +++ b/mysql-test/mysql-test-run.pl 2014-04-14 09:20:24 +0000 > @@ -5350,6 +5350,14 @@ sub mysqld_arguments ($$$) { > mtr_add_arg($args, "--gdb"); > } > > + # Enable the debug sync facility, set default wait timeout. > + # Facility stays disabled if timeout value is zero. > + mtr_add_arg($args, "--loose-debug-sync-timeout=%s", > + $opt_debug_sync_timeout) unless $opt_user_args; Why was it moved? > + > + # Options specified in .opt files should be added last so they can > + # override defaults above. > + > my $found_skip_core= 0; > foreach my $arg ( @$extra_opts ) > { > === added file 'mysql-test/suite/archive/archive_no_symlink-master.opt' > --- a/mysql-test/suite/archive/archive_no_symlink-master.opt 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/archive/archive_no_symlink-master.opt 2014-04-14 09:20:24 +0000 > @@ -0,0 +1 @@ > +--skip-symbolic-links just fyi: in MariaDB the file name can simply be archive_no_symlink.opt -master and -slave are only needed for replication tests to have different options for the master and the slave. > > === added file 'mysql-test/t/bug12969156.test' > --- a/mysql-test/t/bug12969156.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/bug12969156.test 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,18 @@ > +--source include/have_daemon_example_plugin.inc > +--source include/not_embedded.inc > +# TODO: the windows emulation of pthreads doesn't behave predictably > +--source include/not_windows.inc > + > +--echo # > +--echo # Bug #12969156 : SEGMENTATION FAULT ON UNINSTALLING > +--echo # DAEMON_EXAMPLE PLUGIN > +--echo # > + > +let $counter= 0; > +while ($counter < 10) > +{ > +--replace_result $LIBDAEMON_EXAMPLE_SO DAEMONEXAMPLE > + eval INSTALL PLUGIN daemon_example SONAME '$LIBDAEMON_EXAMPLE_SO'; in our own tests (not copied from 5.6) better use install soname 'libdaemon_example'; it's shorter, doesn't need --eval and doesn't need --replace_result > + UNINSTALL PLUGIN daemon_example; > + inc $counter; > +} > > === added file 'mysql-test/t/innodb_log_file_size_functionality.test' > --- a/mysql-test/t/innodb_log_file_size_functionality.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/innodb_log_file_size_functionality.test 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,124 @@ This should be renamed to sys_vars.innodb_log_file_size_func test > +############################################################################### > +# # > +# Variable Name: innodb_log_file_size # > +# Scope: Global # > +# Access Type: Static # > +# Data Type: numeric # > +# # > +# # > +# Creation Date: 2012-08-20 # > +# Author : Tanjot Singh Uppal # > +# # > +# # > +# Description:Test Cases of Static System Variable innodb_log_file_size # > +# that checks the behavior of this variable in the following ways # > +# * Value Check # > +# * Scope Check # > +# * Functionality Check # > +# * Accessability Check # > +# # > +# This test does not perform the crash recovery on this variable # > +# For crash recovery test on default change please run the ibtest # > +############################################################################### Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 4018: MDEV-4784 - merge test cases from 5.6 in lp:maria/10.0
by Sergei Golubchik 12 Jun '14

12 Jun '14

Hi, Sergey! > === added file 'mysql-test/include/have_daemon_example_plugin.inc' > --- a/mysql-test/include/have_daemon_example_plugin.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/have_daemon_example_plugin.inc 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,16 @@ > +disable_query_log; > +# > +# Check if server has support for loading plugins > +# > +if (`SELECT @@have_dynamic_loading != 'YES'`) { > + --skip daemon example plugin requires dynamic loading > +} > + > +# > +# Check if the variable DAEMONEXAMPLE is set > +# > +if (!$LIBDAEMON_EXAMPLE_SO) { > + --skip daemon_example plugin requires the environment variable \$DAEMONEXAMPLE to be set (normally done by mtr) it's always done by mtr (not just "normally"). So a better error message should be "No libdaemon_example.so was found" or simply "Need daemon_example plugin" and, btw, the first if() is redundant, if no dynamic loading was enabled no dynamic plugins will be built whatsoever. > +} > + > +enable_query_log; > > === added file 'mysql-test/include/ib_logfile_size_check.inc' > --- a/mysql-test/include/ib_logfile_size_check.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/ib_logfile_size_check.inc 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,14 @@ I'm not a great fan of obscure include files that are only used once in one single test. Perhaps it's better to inline it? > +perl; > + my $dir = $ENV{'MYSQLD_DATADIR'}; > + my $size; > + opendir(DIR, $dir) or die $!; > + while (my $file = readdir(DIR)) > + { > + > + next unless ($file =~ m/\ib_logfile.$/); > + $size = -s "$dir/$file"; > + print "The size of the ib_logfile(0/1): $size \n"; > + } > + close(DIR); > + exit(0) > +EOF > > === added file 'mysql-test/include/vardir_size_check.inc' > --- a/mysql-test/include/vardir_size_check.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/vardir_size_check.inc 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,26 @@ Same here. Although it's worse than with the previous file :) ib_logfile_size_check.inc is only used in innodb_log_file_size_functionality.test. This file - too. But in 5.6 this one is also used in query_cache_type_functionality, table_open_cache_functionality, and table_definition_cache_functionality, sort_buffer_size_functionality, query_cache_size_functionality. I suppose somebody just copied innodb_log_file_size_functionality over and edited it without much thinking. I don't see why would anyone really want to check the vardir size in all these other tests. So, yes, better to inline this file too, to avoid silly mistakes like that. > +--echo Check the size of the vardir > +--echo The output size is in unit blocks > +perl; > +#!/usr/bin/perl > +use warnings; > +use strict; > +use File::Find; > +my $dir = $ENV{'MYSQLTEST_VARDIR'}; > +my $size = 0; > +find( sub { $size += -f $_ ? -s _ : 0 }, $dir ); > + > +if ( $size < 1717987000 ) > + { print "TRUE", "\n";} > +else > + { print "FALSE $size", "\n";} > + > + > +## Expected size for the VAR dir being changed for the PB2 runs > + > +##if ( $size =~ /^496[\d]{5}$/ ) > +## { > +## { print "TRUE", "\n";} > +## } > +##else > +## { print "FALSE $size", "\n";} > +EOF > > === added file 'mysql-test/suite/archive/archive_no_symlink-master.opt' > --- a/mysql-test/suite/archive/archive_no_symlink-master.opt 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/archive/archive_no_symlink-master.opt 2014-03-11 09:28:16 +0000 > @@ -0,0 +1 @@ just fyi: in MariaDB the file name can simply be archive_no_symlink.opt -master and -slave are only needed for replication tests to have different options for the master and the slave. > +--skip-symbolic-links > > === added file 'mysql-test/t/bug12969156.test' > --- a/mysql-test/t/bug12969156.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/bug12969156.test 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,18 @@ > +--source include/have_daemon_example_plugin.inc > +--source include/not_embedded.inc > +# TODO: the windows emulation of pthreads doesn't behave predictably > +--source include/not_windows.inc > + > +--echo # > +--echo # Bug #12969156 : SEGMENTATION FAULT ON UNINSTALLING > +--echo # DAEMON_EXAMPLE PLUGIN > +--echo # > + > +let $counter= 0; > +while ($counter < 10) > +{ > +--replace_result $LIBDAEMON_EXAMPLE_SO DAEMONEXAMPLE > + eval INSTALL PLUGIN daemon_example SONAME '$LIBDAEMON_EXAMPLE_SO'; in our own tests (not copied from 5.6) better use install soname 'libdaemon_example'; it's shorter, doesn't need --eval and doesn't need --replace_result > + UNINSTALL PLUGIN daemon_example; > + inc $counter; > +} > > === added file 'mysql-test/t/innodb_log_file_size_functionality.test' > --- a/mysql-test/t/innodb_log_file_size_functionality.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/innodb_log_file_size_functionality.test 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,124 @@ This should be renamed to sys_vars.innodb_log_file_size_func test > +############################################################################### > +# # > +# Variable Name: innodb_log_file_size # > +# Scope: Global # > +# Access Type: Static # > +# Data Type: numeric # > +# # > +# # > +# Creation Date: 2012-08-20 # > +# Author : Tanjot Singh Uppal # > +# # > +# # > +# Description:Test Cases of Static System Variable innodb_log_file_size # > +# that checks the behavior of this variable in the following ways # > +# * Value Check # > +# * Scope Check # > +# * Functionality Check # > +# * Accessability Check # > +# # > +# This test does not perform the crash recovery on this variable # > +# For crash recovery test on default change please run the ibtest # > +############################################################################### > + > +-- source include/have_innodb.inc > +-- source include/not_embedded.inc > +-- source include/have_innodb_16k.inc > + > +echo '#________________________VAR_09_INNODB_LOG_FILE_SIZE__________________#' > +echo '##' > +--echo '#---------------------WL6372_VAR_9_01----------------------#' > +#################################################################### > +# Checking default value # > +#################################################################### > +SELECT COUNT(@@GLOBAL.innodb_log_file_size); > +--echo 1 Expected > + > +SELECT @@GLOBAL.innodb_log_file_size; > +--echo 5242880 Expected > + > + > + > +--echo '#---------------------WL6372_VAR_9_02----------------------#' > +#################################################################### > +# Checking Value can be set - Static # > +#################################################################### > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SET @@local.innodb_log_file_size=1; > +--echo Expected error 'Read only variable' > + > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SET @@session.innodb_log_file_size=1; > +--echo Expected error 'Read only variable' > + > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SET @@GLOBAL.innodb_log_file_size=1; > +--echo Expected error 'Read only variable' > + > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SET @@GLOBAL.innodb_log_file_size=DEFAULT; > +--echo Expected error 'Read only variable' > + > +SELECT COUNT(@@GLOBAL.innodb_log_file_size); > +--echo 1 Expected > + > + > + > +--echo '#---------------------WL6372_VAR_9_03----------------------#' > +################################################################# > +# Check if the value in GLOBAL Table matches value in variable # > +################################################################# > +SELECT @@GLOBAL.innodb_log_file_size = VARIABLE_VALUE > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='innodb_log_file_size'; > +--echo 1 Expected > + > +SELECT COUNT(@@GLOBAL.innodb_log_file_size); > +--echo 1 Expected > + > +SELECT COUNT(VARIABLE_VALUE) > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='innodb_log_file_size'; > +--echo 1 Expected > + > + > + > +--echo '#---------------------WL6372_VAR_9_04----------------------#' > +################################################################################ > +# Checking Variable Scope # > +################################################################################ > +SELECT @@innodb_log_file_size = @@GLOBAL.innodb_log_file_size; > +--echo 1 Expected > + > +--Error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SELECT COUNT(@@local.innodb_log_file_size); > +--echo Expected error 'Variable is a GLOBAL variable' > + > +--Error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SELECT COUNT(@@SESSION.innodb_log_file_size); > +--echo Expected error 'Variable is a GLOBAL variable' > + > +SELECT COUNT(@@GLOBAL.innodb_log_file_size); > +--echo 1 Expected > + > +--Error ER_BAD_FIELD_ERROR > +SELECT innodb_log_file_size = @@SESSION.innodb_log_file_size; > +--echo Expected error 'Unknown column innodb_log_file_size in field list' > + > + > + > +--echo '#---------------------WL6372_VAR_9_05----------------------#' > +############################################################################### > +# Checking the /Var directory size # > +############################################################################### > +-- source include/vardir_size_check.inc > +--echo TRUE Expected > + > +--echo '#---------------------WL6372_VAR_9_06----------------------#' > +################################################################################# > +# Checking the size of ib_logfile # > +################################################################################# > +let MYSQLD_DATADIR=`SELECT @@datadir`; > +--source include/ib_logfile_size_check.inc > +--echo 5242880 expected > + > Regards, Sergei

1 0

[Maria-developers] PERFORMANCE_SCHEMA to be disabled in 10.0 (10.0.12)
by Colin Charles 12 Jun '14

12 Jun '14

Hi all, Recently there was chat about how PERFORMANCE_SCHEMA was enabled by mistake and it should be disabled in the 10.0 series. I'm curious - how many of you are using PERFORMANCE_SCHEMA? Is it a problem to turn it on, if you use it? I'm referring to: https://mariadb.atlassian.net/browse/MDEV-6249 Looking from the webscalesql list, there are examples of performance degradation: https://groups.google.com/d/msg/webscalesql/B-4bPIb8mHI/eUNsl0lSfbMJ https://groups.google.com/d/msg/webscalesql/B-4bPIb8mHI/-YZMMCuPw3QJ quote: "Our perf testing agrees with your assessment (we see about a 5%-6% perf hit when it's included and on, and a 2%-3% hit when it's included but off)" Please discuss this, either here or on MDEV-6249 Thanks cheers, -colin -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

6 10

[Maria-developers] Extendet FederatedX storage engine to support updating Sphinx RT indexes
by Markus Lidel 12 Jun '14

12 Jun '14

Hello, the Sphinx Search engine has RT indexes, which could be updated with a subset of SQL commands, refered to as SphinxQL: http://sphinxsearch.com/docs/2.1.1/sphinxql-reference.html Because the FederatedX storage engine uses some commands, which Sphinx does not know, here is a patch, wich extends the FederatedX storage engine to support this subset of commands. Now its possible to update Sphinx RT indexes directly from MariaDB. To create a connection to the RT index in the Sphinx default RT-index: index rt { type = rt path = @CONFDIR@/data/rt rt_field = title rt_field = content rt_attr_string = title rt_attr_string = content rt_attr_uint = gid } you just have to call: CREATE TABLE `rt` ( `id` BIGINT UNSIGNED NOT NULL, `title` TEXT, `content` TEXT, `gid` INT UNSIGNED ) ENGINE=FEDERATED CONNECTION='sphinxql://root@localhost:9306/rt'; It would be great, if the attached patch could be integrated into MariaDB. If you have any suggestions, please feel free to contact me. Best regards, Markus Lidel

4 8

Re: [Maria-developers] [GSoC] Accepted student ready to work : )
by Pablo Estrada 10 Jun '14

10 Jun '14

Hello all, I'm still interested on a call, (I would make sure to write down everything as part of an email to keep it clearly organized here), but as Elena suggested, I'll try to look at Sergei's bzr repo first, and all the documentation. In any case, I already have a decent idea of how the data is arranged. I can see how it is difficult to trace test results back to code changes directly, but it might be possible. I still haven't figured out how to recognize what code changes pertain to the same 'history', or 'line' (I think that you were not considering this in your experiments... and it might turn out to not be relevant at all, but I still want to consider it, to try to make a more complete model). So I have just two questions that should be easy to answer by email: 1. In your graphs, exactly, what do you mean by Precision? What about Recall? And cutoff? 2. I looked at your launchpad profile but I can't seem to find the repo with your tests (if you even pushed it to lp at all). Is it available? That's it for today : ) Until next time. Pablo On Thu, Apr 24, 2014 at 12:28 AM, Sergei Golubchik <serg(a)skysql.com> wrote: > On April 23, 2014 10:07:23 AM CEST, Pablo Estrada wrote: > > >If it's not too inconvenient, do you think we could set up a Google > >Hangout > >or a Skype call on Monday to go over a few questions quickly? > > > Definitely. This week I'm on vacations and without a laptop, back home on > Sunday. So, Monday is fine. But please if you have specific questions - ask > them now in emails. I'll prepare answers for Monday, but it'll take time to > remember everything. That is, if you will wait with questions till the > call, I might not be able to remember the answer on the spot. > Regards, > Sergei >

4 17

Re: [Maria-developers] Rev 4234: Fixed bug mdev-6071
by Sergei Golubchik 10 Jun '14

10 Jun '14

Hi, Igor! On Jun 09, Igor Babaev wrote: > revno: 4234 > revision-id: igor(a)askmonty.org-20140610050716-0x8xaquf3itvofjc > parent: bar(a)mariadb.org-20140606062952-x2jwt4mgrgr6tpw6 > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-10.0-bugs > timestamp: Mon 2014-06-09 22:07:16 -0700 > message: > Fixed bug mdev-6071. > The method JOIN_CACHE::init may fail (return 1) if some conditions on the > used join buffer is not satisfied. For example it fails if join_buffer_size > is greater than join_buffer_space_limit. The conditions should be checked > when running the EXPLAIN command for the query. That's why the method > JOIN_CACHE::init has to be called for EXPLAIN commands as well. It's an improvement over the old behavior - as EXPLAIN shows correctly whether BKA is used. But I think there's still not enough diagnostics. In the bug report Sergey wrote that " I think it's a big gotcha that @join_buffer_size must be lower than @@join_buffer_space_limit when optimizer_space_limit optimization is off. " So, I'd believe users will be very confused that in some cases BKA is suddenly turned off, when they expect it to be used. Of course, without this bugfix they would've not even noticed that, which is even worse. So, this bugfix is good. But it would be nice to have some kind of an explanation of why BKA is not used. At the very least in the manual. Better - here as a warning, like if (for_explain) push_warning("BKA was not used because ..."; goto fail; > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2014-06-05 22:07:27 +0000 > +++ b/sql/sql_select.cc 2014-06-10 05:07:16 +0000 > @@ -10541,7 +10541,7 @@ > if (cache_level == 1) > prev_cache= 0; > if ((tab->cache= new JOIN_CACHE_BNL(join, tab, prev_cache)) && > - ((options & SELECT_DESCRIBE) || !tab->cache->init())) > + !tab->cache->init(MY_TEST(options & SELECT_DESCRIBE))) MY_TEST here (and in other similar places) is redundant, you could've simply written !tab->cache->init(options & SELECT_DESCRIBE)) > { > tab->icp_other_tables_ok= FALSE; > return (2 - MY_TEST(!prev_cache)); and while it wasn't part of your bug fix, I cannot help but noticing that MY_TEST() is redundant here too. !prev_cache is boolean, it's always 0 or 1. Regards, Sergei

1 0

Re: [Maria-developers] Google Code for Encryption of MariaDB
by Elmar Eperiesi-Beck 09 Jun '14

09 Jun '14

Hi, 1. what about an NDA between us, to give me the chance to have a pre-view. I will make sure, that your architecture and development fits into our work. Everything we can think of in an early stage does not make lot of work, but… 2. like. 3. We will post our concept next week into the MariaDB Developers community. 4. … or 1. Regards Elmar Am 09.06.2014 um 11:14 schrieb Jonas Oreland : > Hi there, > > 1) repo: as I said in my mail, the code is currently not yet published, > but the guy doing this said that he expects it to be with in a couple of weeks or so. > before this, i don't think you can see it. > > once it's published, I think you can find it at https://code.google.com, > but I'll check and let you know. > > 2) This is my personal email address, so I prefer if we keep the conversation > @maria-developers where I'm subscribed with work email address, if this is ok > by you ? > > 3) concept paper: sure > > 4) we don't really have a concept paper, but the code is complete and being rolled out. > I can answer questions about it, but I think it's easiest once it has been published. > > /Jonas > > > > On Mon, Jun 9, 2014 at 11:02 AM, Elmar Eperiesi-Beck wrote: > Jonas, > is there a repository, where I can find the actual source code of your encryption implementation? > Or do we have to sign an NDA to exchange these kind of information? > > We can have a look at your code and merge it with our existing implementation. > Additionally I would like to send you our concept paper, to give you an overview of our ideas. > > Regards > Elmar >

1 0

[Maria-developers] [GSoC] Week 3!
by sriram patil 09 Jun '14

09 Jun '14

Hi all, Here is my post for the 3rd week into GSoC project. http://goo.gl/bhl5ZW Suggestions and review are welcome! Thanks, Sriram

1 0

[Maria-developers] MariaDB 5.5.38 Release Prep
by Daniel Bartholomew 07 Jun '14

07 Jun '14

All, I've begun prep work on the MariaDB 5.5.38 release. Draft release notes and changelog are here: - https://mariadb.com/kb/en/mariadb-5538-release-notes/ - https://mariadb.com/kb/en/mariadb-5538-changelog/ Current plan is to activate and announce the release on Mon, 9 jun 2014. Thanks. -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

1 0

[Maria-developers] MDEV-5533 - Backport Threadpool improvements from Percona Server
by Sergey Vojtovich 04 Jun '14

04 Jun '14

Hi Sergei, Since MDEV-5533 is marked critical for 10.0.12 it is a good time to start this discussion. IIRC we agreed that our intention is to accept default value change for thread_pool_max_threads into 10.0 and accept priority queue throttling into 10.1. So default value for thread_pool_max_threads has bumped in percona server from 500 to 100 000. The reason is: deadlocks and uneven distribution of worker threads between thread groups in case of stalled connections. But the above is contradicting to suggestions in KB: https://mariadb.com/kb/en/threadpool-in-55/ ... so ideally, there should be a single active thread for each CPU on the machine. ... If your application constantly pegs at 500 threads, it might be a strong indicator for high contention in the application, and threadpool does not help much. ... One way to solve the problem would be to set thread_pool_max_threads to a very high value, but this might be less than ideal, and negatively affect performance. ... Said the above, do you still think it is a good idea to raise default value of thread_pool_max_threads? If so, what should KB article state wrt this variable? Thanks, Sergey

1 0

Re: [Maria-developers] MDEV-6213: Query cache is not invalidated on cascade delete when database name contains special symbols
by Sergei Golubchik 03 Jun '14

03 Jun '14

Hi, Sanja! Please add in your revision a test case for MySQL bug#13919851 http://bazaar.launchpad.net/~mysql/mysql-server/5.6/revision/4514 and make sure that your fix doesn't break it. On May 09, sanja(a)askmonty.org wrote: > revno: 4207 > revision-id: sanja(a)askmonty.org-20140509104240-ksrk10tef5dq157z > branch nick: work-maria-10.0-MDEV-6213 > timestamp: Fri 2014-05-09 13:42:40 +0300 > message: > MDEV-6213: Query cache is not invalidated on cascade delete when > database name contains special symbols. > > Innodb should de-escape db/table names when sending them to QC Regards, Sergei

1 0

[Maria-developers] Please review a patch fixing a few MDEV-6001 blockers
by Alexander Barkov 03 Jun '14

03 Jun '14

Hi Sergei, can you please review my patch fixing a few bugs that are blockers for MDEV-6001? - MDEV-4858 Wrong results for a huge unsigned value inserted into a TIME column - MDEV-6097 Inconsistent results for CAST(int,decimal,double AS DATETIME) - MDEV-6099 Bad results for DATE_ADD(.., INTERVAL 2000000000000000000.0 SECOND) - MDEV-6100 No warning on CAST(9000000 AS TIME) - MDEV-6101 Hybrid functions do not add CURRENT_DATE when converting TIME to DATETIME - MDEV-6102 Comparison between TIME and DATETIME does not use CURRENT_DATE Thanks.

2 10

[Maria-developers] Review request[major 5.3/5.5 bug]: MDEV-4511 Assertion `scale <= precision' fails on GROUP BY TIMEDIFF with incorrect types, , Agile Board, , Export
by Alexander Barkov 03 Jun '14

03 Jun '14

Hello Sergei, A few weeks ago we fixed this bug for 10.0 by changing temporal items not to have decimals=NOT_FIXED_DEC (more MySQL-5.6 compatibility), and agreed to have a separate patch for the pre-10.0 branches. This patch is for 5.3 and 5.5 only. It back-ports the mdev-4511 tests from 10.0 as is. But it does not touch the NOT_FIXED_DEC behavior of the temporal items and fixes the problem in a different way. Greetings.

1 0

[Maria-developers] POWER interest
by Colin Charles 03 Jun '14

03 Jun '14

Hi, It seems we have a POWER8 box and soon will start doing builds on this platform Stewart Smith (currently at IBM) seems to be doing some work on MySQL 5.6 and MySQL 5.7 for POWER8 systems - there doesn't seem to be much attention being paid to MariaDB 10. Ideally, our fixes that are being picked up from Stewart or focused on POWER should be targeted at 10.1 I'm thinking (unless there's customer interest for the 10.0 series -- which may very likely be the case) So far, I've noted GIS precise has an open failure, and maybe its a great idea for Holyfoot to use the POWER8 system to start fixing it: https://mariadb.atlassian.net/browse/MDEV-4153 From MySQL bugs system we see multiple examples from Stewart's blog posts - some are InnoDB specific so we will inherit them during a merge - some aren't like mysql#72715 (character set code endianness dependent on CPU type rather than endianness of CP) https://www.flamingspork.com/blog/2014/06/03/1-million-sql-queries-per-seco… https://www.flamingspork.com/blog/2014/06/03/mysql-5-7-on-power/ Relevant stuff that we may want to look at: http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=PowerPC cheers, -colin -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

1 0

[Maria-developers] [GSoC] First two weeks into coding!
by sriram patil 03 Jun '14

03 Jun '14

Hi all, I have created a blog to write the details of the work that I do as part of GSoC'14. Following is the link to my first blog post explaining the details of the work that I have done during the last two weeks. http://goo.gl/PYoVZb I will be writing these blog posts every weekend from now on and will share it on the developer mailing list every Monday. Suggestions/reviews are welcome. Thanks, Sriram

2 2

[Maria-developers] Possible race condition or NULL pointer triggered by OQGraph
by Andrew McDonnell 02 Jun '14

02 Jun '14

Hi all (Cross-posted to oqgraph-developers, maria-developers) I am trying to track down a segfault apparently triggered by concurrent execution of queries through OQGraph - for context, see: https://mariadb.atlassian.net/browse/MDEV-6282 I am however a bit confused as to what is going on, because at least some of the information leads me to suspect that there could be more going on here than just a simple race condition. --Immediate Cause-- On the surface, a segfault appears to be caused in a method in sql_class.h, Statement::check_limit_rows_examined() dereferencing a NULL pointer, `lex`. The method check_limit_rows_examined() was called from sql_class.h method handler::increment_statistics() . --Analysis-- A quick inspection reveals that a NULL value for `lex` is actually perfectly valid some of the time, although I am nowhere near familiar enough with the code to really understand when or why. Specifically, `lex` is a member of the base class Statement, which is initialised to either point to THD::main_lex via a constructor , or intentionally to NULL from construction of an object Prepared_statement() (sql_prepare.cc) To my mind, if there is going to be a possibility of a pointer being NULL, then all uses of that pointer should be checking for that. I am hoping that the bug reporter can prove or otherwise my hypothesis with his particular scenario, although I am suspicious that there may be additional issues to be dealt with; it is quite possible that lex being NULL is a red herring, being caused by a race condition memory overrun elsewhere. However, I think it is still a good idea to NULL check pointers that could be set to NULL intentionally. I can submit a pull request if the following patch is agreeable: diff --git a/sql/sql_class.h b/sql/sql_class.h index 5898d9e..957acf3 100644 --- a/sql/sql_class.h +++ b/sql/sql_class.h @@ -2475,8 +2475,10 @@ public: */ void check_limit_rows_examined() { - if (++accessed_rows_and_keys > lex->limit_rows_examined_cnt) + ++accessed_rows_and_keys; + if (lex && accessed_rows_and_keys > lex->limit_rows_examined_cnt) { killed= ABORT_QUERY; + } } --Aside-- Immediately prior to the check_limit_rows_examined() call is a call to macro status_var_increment(). Comments around the macro indicate it is often likely to be called in a mutli-threaded context, but the data is such that it is possible to disable protection with the only side effect being an occasionally 'inaccurate' result. Is it possible however, that check_limit_rows_examined() is not thread-safe and should have some protection? cheers, Andrew --A

3 3

[Maria-developers] MDEV-4051 INET6_ATON() and INET6_NTOA()
by Alexander Barkov 02 Jun '14

02 Jun '14

Hello Ian, I have merged these functions from MySQL-5.6 into 10.0: INET6_ATON() INET6_NTOA() IS_IPV4() IS_IPV4_COMPAT() IS_IPV4_MAPPED() IS_IPV6() Can you please document them? The corresponding article in the MySQL manual is here: http://dev.mysql.com/doc/refman/5.6/en/miscellaneous-functions.html Thanks.

2 2

[Maria-developers] KB page on switching from bazaar to git.
by Sergey Petrunia 27 May '14

27 May '14

Hi, I was doing some stuff in git already, and so I have created a git cheatsheet with equivalents of bzr commands that were useful in MariaDB development: https://mariadb.com/kb/en/using-git/ It is not finished, any additions are welcome. Elena has agreed to look at creating a commit trigger that sends commit emails (currently, github sends emails on push (not commit), and these do not include diffs) BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] CREATE OR REPLACE DATABASE algorithm
by sriram patil 27 May '14

27 May '14

Hi Alexander and all, Following are the steps that I am following for implementing "CREATE OR REPLACE" for DATABASE "..." signifies, there exists one or more lines of code in that block of the function. mysql_create_db(...) ... ... if(db exists) { if(create_or_replace) { if(DROP_ACL access not granted) { // Error 1 DBUG_RETURN(-1) } if(mysql_rm_db(...) unsuccessful) { // Error 2 DBUG_RETURN(-1) } Reset m_status from diagnostics cause it is changed by mysql_rm_db. Proceed with normal create db. } else if(if not exists) { ... } else ... } ... ... } Is the above algorithm correct? Also, I am not yet sure which errors should be displayed at both the places "Error 1" and "Error 2". I have written and tested the above code. It is working fine. Testing involves running all the existing test cases and executing "CREATE OR REPLACE DATABASE" command on the console manually. "IF NOT EXISTS" gives error if used with "CREATE OR REPLACE". Any other test cases I need to consider? Thanks, Sriram

2 2

[Maria-developers] FOR JSON
by Alexander Barkov 26 May '14

26 May '14

Hello Monty, We discussed extracting data in JSON format today, to make it possible to return data as a single JSON value (as a result set consisting of a single row and a single column): SELECT id,name,address FROM users FOR JSON; Note, in some cases the result of a such query can obviously be very huge. So it would be nice to be able to use "FOR JSON" in subselects, like this: SELECT id, (SELECT name,address FROM users AS b WHERE a.id= b.id FOR JSON) AS RowJSON FROM users AS a; This query will return a result set with multiple rows (one row per one record) and with two columns: - the ID, as a regular number - the JSON value with "name" and "address" embedded into it. Or, another example, if one does not need "id" as a separate column: SELECT (SELECT id,name,address FROM users AS b WHERE a.id= b.id FOR JSON) AS RowJSON FROM users AS a; This query will return a result set with multiple rows (one row per one record) with a single JSON value embedding all three columns: "id", "name" and "address".

2 4

[Maria-developers] MariaDB 10.1 is now on github!
by Sergei Golubchik 26 May '14

26 May '14

Hi! I'm happy to announce that MariaDB-10.1 tree has been completely migrated to github. Since now on we'll use github for the new development in MariaDB. It's https://github.com/MariaDB/server, go on, fork it, hack around, submit pull requests. Have fun! Older trees (10.0, 5.5, 5.3, 5.2, 5.1) are not on github - we do hope to migrate them too eventually, but at the moment they are still on launchpad. If you're a maria-captain on launchpad - for you to get write access to the mariadb repository on github you need to tell me your github account name. Regards, Sergei P.S.: Don't forget that 10.1 is still *pre-alpha*. You've been warned.

4 5

Re: [Maria-developers] MariaDB 10.1 is now on github!
by Sergei Golubchik 26 May '14

26 May '14

Hi, Binod! First: please send questions like this to maria-developers(a)lists.launchpad.net it's a public mailing list dedicated to MySQL and MariaDB internals, source code, and related things. I am subscribed, so I'll see you mail there, and you may be sure I will, because it won't be accidentally catched by my spam filter, or sorted out in some obscure folder. Furthermore other subscribers will see your question and could reply if I will be not available (e.g. I could be travelling). Thank you. On May 26, Binod Bista wrote: > Hi Sergei, > > Thanks for the info. Just to interrupt, is there a page you recommend for > anyone starting out in MariaDB development? Particularly any database or > MySQL internals material that help understand the codebase. That depends on what you're trying to do. There are two books: http://shop.oreilly.com/product/9780596009571.do http://www.packtpub.com/mysql-51-plugin-development/book There's MySQL internals manual: http://dev.mysql.com/doc/internals/en/index.html There's MariaDB Knowledge Base that includes pages about MariaDB internals: http://kb.askmonty.org/ There're numerous blog posts and presentations about specific topics. Regards, Sergei

1 0

[Maria-developers] Getting started with the coding.
by sriram patil 23 May '14

23 May '14

Hi Alexander, I have started working on "CREATE OR REPLACE DATABASE" implementation. I thought this one has the least complexity among all the other commands. I am getting my hand-on debugging the source code too. I have made the required changes in sql_yacc.yy. The command is parsed properly. Handled the error where CREATE OR REPLACE and IF NOT EXISTS cannot occur at the same time. This is how I am thinking of proceeding with the implementation, In method mysql_create_db after checking the flag HA_LEX_CREATE_REPLACE. I am removing the database by calling mysql_rm_db and then the current implementation will take care of creating the new db with the same name. This seems correct? Or is there anything I need to consider? Thanks, Sriram

2 1

[Maria-developers] delete + offset
by Roberto Spadim 21 May '14

21 May '14

hi guys! i was thinking at maria-discuss, and i have a new "feature request" to mariadb DELETE statment, that's the OFFSET clausule, like sqlite my idea is... i have a table with history about for example temperature, and i will only record last 5000 records, my idea is execute a 'cleanup' for example every day instead of: select date_time from history where channel=1234 order by date_time desc limit 1 offset 5000 delete from history where channel=1234 and data_time<value from previous query i was considering something like delete from history where channel=1234 order by date_time desc limit 99999999999 offset 5000 what you think about this? -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

[Maria-developers] Regarding debugging MariaDB
by sriram patil 17 May '14

17 May '14

Hi Alexander, I wanted to run mysqld in debug mode to check the trace file and get to know workflow of SQL commands. I tried following command mysqld --user=<username> --debug Then, after connecting I executed some commands. But, the trace file is not created in /tmp folder. Is there anything I need to do while building the source code in order for the debug mode to work? Thanks, Sriram

2 1

[Maria-developers] Merging FusionIO changes to 10.0.12
by Colin Charles 16 May '14

16 May '14

Hi all, MariaDB has worked actively with Fusion-io, and we did release some improvements in a preview release a little over a month ago: https://blog.mariadb.org/significant-performance-boost-with-new-mariadb-pag… This was originally planned to be included into 10.0.10, but when we released the GA, we thought that the feature was not tested thoroughly. Now the code has proven stable we are thinking of merging this into 10.0.12. Yes, we know its already a GA, but this feature will be turned off by default, and the code is safely encapsulated. If you are not using Fusion-io or do not turn it on, there is no effect to you. To check, the changes are in the XtraDB and InnoDB that we ship (storage/xtradb & storage/innobase). Naturally, for those using Fusion-io devices, the benefits are clear - you will see significant improvements in speed and you will increase the lifetime of your cards. Think of this like adding drivers to the Linux kernel from time to time ;) What does everyone think? Package maintainers? Thanks! -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

3 2

[Maria-developers] LIMIT ROWS MATCHED
by Colin Charles 16 May '14

16 May '14

Hi! http://seonguck.blogspot.kr/2014/05/limit-rows-matched.html We have a potential contribution here for a LIMIT ROWS MATCHED feature - might be interesting for us to consider this for 10.1 ? cheers, -colin -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

2 1

[Maria-developers] tokudb database - generated "should never happen" status when mysql log partition ENOSP
by Daniel Black 15 May '14

15 May '14

1 0

[Maria-developers] New to the list & question about CONNECT
by Martijn Tonies (Upscene Productions) 13 May '14

13 May '14

Hello everyone, My name is Martijn Tonies and I’m a Windows tools developer, we have a tool called “Database Workbench”, a developer tool for several database engines. For it’s next major version, we’re planning to officially support MariaDB in the MySQL module. I’ve investigated several of the features, implemented support for Roles, Virtual Columns etc, but I’m having trouble with the CONNECT engine. Here’s what I did, using MariaDB 10.0.10 1) registered a server with CREATE SERVER: CREATE SERVER mysql55_at_local FOREIGN DATA WRAPPER mysql OPTIONS(HOST 'MT-XP-VM-MYSQL', PORT 3355, DATABASE 'test', USER 'root', PASSWORD ‘secret’); 2) created a table in the MySQL database: CREATE TABLE sfdata ( mark1 Integer(11), mark2 Integer(11), mark3 Integer(11), mark4 Integer(11), `name` VarChar(50) CHARACTER SET latin1 COLLATE latin1_swedish_ci ) ENGINE=InnoDB DEFAULT CHARACTER SET latin1 COLLATE latin1_swedish_ci; 3) created a table in the MariaDB database: CREATE TABLE `sfdata_proxy3` ( `mark1` int(11) DEFAULT NULL, `mark2` int(11) DEFAULT NULL, `mark3` int(11) DEFAULT NULL, `mark4` int(11) DEFAULT NULL, `name` varchar(50) DEFAULT NULL ) ENGINE=CONNECT DEFAULT CHARSET=utf8 CONNECTION='mysql55_at_local/sfdata' I then expected that table “sfdata_proxy3” has the rows as “sfdata” has, but I got nothing returned. Any clue where to look? With regards, Martijn Tonies Upscene Productions http://www.upscene.com Download Database Workbench for Oracle, MS SQL Server, Sybase SQL Anywhere, MySQL, InterBase, NexusDB and Firebird!

1 1

Re: [Maria-developers] Question about GTIDs in MariaBD
by Kristian Nielsen 12 May '14

12 May '14

"AL13N" <alien(a)rmail.be> writes: >> Peter Laursen <peter_laursen(a)webyog.com> writes: >> So yes, there are others that consider Oracle's design of GTID 'a dirty >> hack', >> but MariaDB 10.0 GTID is not affected by that design. > What about mariadb being a slave of a Oracle MySQL GTID? The restrictions are on the master, not on the slaves. So what do you mean? Surely, connecting a MariaDB slave to an Oracle MySQL master will not remove those restrictions. Note that when a MariaDB server connects as a slave to an Oracle MySQL master server, it is not possible to use GTID for that connection (because the GTID implementations are not compatible). Old style non-GTID replication must be used. - Kristian.

2 1

Re: [Maria-developers] Question about GTIDs in MariaBD
by Peter Laursen 12 May '14

12 May '14

With autocommit=1 every statement runs in its own transaction, so obviousluy .. yes! MyISAM is/used to be completely 'transaction-agnosstic'. But GTIDs seem to confict with non-transactional storage engines (as far as I can understand). MySQL/Oracle have "solved" (!) this by not allowing updates to tables using transactional engines and non-transactional storage engines in same transaction. However to me it is nonsense that transactions apply (have any effect at all) for non-transactional storage engines. As a consequence I consider this solution (what must have been enforced in the server layer and not the storeage engine layer) by Oracle 'a dirty hack' (unless somewone can explain me why it it necessary). -- Peter On Sun, May 11, 2014 at 4:42 PM, Roberto Spadim <roberto(a)spadim.com.br>wrote: > With autoclmmit=1 works? > > Em domingo, 11 de maio de 2014, Peter Laursen <peter_laursen(a)webyog.com> > escreveu: > > Thanks for the reply, but I am afraid I don't fully understand. >> >> Here is a simple test case with mySQL 5.6.17: >> >> 1) >> You need all those in configuration [mysqld) section >> >> log-bin >> log-slave-updates >> gtid_mode = ON >> enforce_gtid_consistency = ON >> >> >> 2) >> Then this small script will reproduce: >> >> >> DROP DATABASE IF EXISTS `gtid_test`; >> CREATE DATABASE `gtid_test`; >> USE `gtid_test`; >> >> CREATE TABLE `tab1` ( >> `a` INT DEFAULT NULL >> ) ENGINE=MYISAM ; >> >> CREATE TABLE `tab2` ( >> `b` INT >> ) ENGINE=INNODB; >> >> --start a transaction >> SET autocommit=0; >> >> INSERT INTO `tab1` VALUES (1); >> INSERT INTO `tab2` VALUES (1); >> >> -- now the first UPDATE to the MyISAM table in the transaction triggers >> this error: >> >> UPDATE `tab1` SET `a` = 5 WHERE `a` = 1; >> >> -- and you'll get: >> -- >> -- Error Code: 1785 >> -- When @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1, updates to >> non-transactional tables can only be done in either autocommitted >> statements or single-statement transactions, and never in the same >> statement as updates to transactional tables. >> >> >> 3) >> ShouldI understand that MariaDB dos not prevent this (does not raise an >> error), but it is still unsafe as binlog - and thus replication - may be >> corrupted? >> >> >> Thanks! >> >> -- Peter >> >> On Sat, May 10, 2014 at 6:09 PM, Pavel Ivanov <pivanof(a)google.com> wrote: >> >>> There is no such limitation in MariaDB as well as no limitation on >>> CREATE TABLE ... SELECT. >>> >>> Although beware that mixing statements changing MyISAM and InnoDB >>> tables in statement binlog mode you may get different results from the >>> statements on slaves. >>> >>> Pavel >>> >>> On Sat, May 10, 2014 at 3:48 AM, Peter Laursen <peter_laursen(a)webyog.com> >>> wrote: >>> > GTID implementation in MySQL 5.6 has the limitation as described here: >>> > >>> https://dev.mysql.com/doc/refman/5.6/en/replication-gtids-restrictions.html >>> > >>> > >>> > "nontransactional storage engines such as MyISAM cannot be made in the >>> same >>> > statement or transaction as updates to tables using transactional >>> storage >>> > engines such as InnoDB." >>> > >>> > If you try you will get the error: >>> > Error Code: 1785 >>> > When @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1, updates to >>> non-transactional >>> > tables can only be done in either autocommitted statements or >>> > single-statement transactions, and never in the same statement as >>> updates to >>> > transactional tables. >>> > >>> > Is there same or similar/other limitations with GTID in MariaDB 10? >>> > >>> > -- Peter Laursen >>> > -- Webyog >>> > >>> > _______________________________________________ >>> > Mailing list: https://launchpad.net/~maria-developers >>> > Post to : maria-developers(a)lists.launchpad.net >>> > Unsubscribe : https://launchpad.net/~maria-developers >>> > More help : https://help.launchpad.net/ListHelp >>> > >>> >> >> > > -- > Roberto Spadim > SPAEmpresarial > Eng. Automação e Controle > >

3 3

[Maria-developers] Question about GTIDs in MariaBD
by Peter Laursen 11 May '14

11 May '14

GTID implementation in MySQL 5.6 has the limitation as described here: https://dev.mysql.com/doc/refman/5.6/en/replication-gtids-restrictions.html "nontransactional storage engines such as MyISAM<https://dev.mysql.com/doc/refman/5.6/en/myisam-storage-engine.html> cannot be made in the same statement or transaction as updates to tables using transactional storage engines such as InnoDB<https://dev.mysql.com/doc/refman/5.6/en/innodb-storage-engine.html> ." If you try you will get the error: Error Code: 1785 When @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1, updates to non-transactional tables can only be done in either autocommitted statements or single-statement transactions, and never in the same statement as updates to transactional tables. Is there same or similar/other limitations with GTID in MariaDB 10? -- Peter Laursen -- Webyog

2 3

[Maria-developers] MariaDB 10.0.11 release prep
by Daniel Bartholomew 10 May '14

10 May '14

I've started the release prep for MariaDB 10.0.11. Draft release notes and changelog pages are here: - https://mariadb.com/kb/en/mariadb-10011-release-notes/ - https://mariadb.com/kb/en/mariadb-10011-changelog/ Please make necessary additions/modifications/etc... The release is still building at the moment, but when finished I'll upload to the mirrors later today or tomorrow. Current plan, if everything goes well, is to activate and announce the release on Monday, 12 May. Thanks! -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

1 0

[Maria-developers] 10.0 -> 10.1 merge
by Sergey Petrunia 08 May '14

08 May '14

Hello Igor, As discussed, I've done the 10.0 -> 10.1 merge. I didn't get any warnings or new test failures in the 'main' test suite, so I've pushed the merge into 10.1 tree. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] connect.alter fails
by Alexander Barkov 07 May '14

07 May '14

Hello Olivier, connect.alter fails when built without XML support complaining about unsupported table type XML. I guess it should be skipped. connect.alter also fails with libxml2 with these symptoms: --- /home/bar/maria-bzr/maria-10.0-connect/storage/connect/mysql-test/connect/r/alter.result 2014-03-23 11:32:34.519624000 +0400 +++ /home/bar/maria-bzr/maria-10.0-connect/storage/connect/mysql-test/connect/r/alter.reject 2014-03-23 11:48:43.251680097 +0400 @@ -1,3 +1,5 @@ +Warnings: +Warning 1105 No file name. Table will use t1.xml Please fix. A remainder: please check every push in BuildBot: http://buildbot.askmonty.org/buildbot/grid?branch=10.0-connect

3 4

[Maria-developers] Barkov: report for the week starting Apr 29, 2014
by Alexander Barkov 05 May '14

05 May '14

Apr 28 - May 04 =============== Apr 28 Apr 29 Apr 30 May 01 Holiday May 02 Holiday May 03 Weekend May 04 Weekend Vacation days taken: 10/35 - MDEV-5702 Incorrect results are returned with NULLIF() Pushed into 5.5 - MDEV-5459 Illegal mix of collations for datetime Pushed into 5.5 - MDEV-6146 Can't mix (latin1_swedish_ci,NUMERIC) and (utf8_unicode_ci,IMPLICIT An after-fix pushed (the original patch broke "mtr --ps fulltext" - Merging gb18030 from MySQL-5.7 In progress - Discussion the manual changes with Ian

1 0

[Maria-developers] sources won't compile
by AugustQ 03 May '14

03 May '14

Hi, I encountered a problem by compiling the sources of MariaDB 10.04. Some days ago I upgraded my system from Ubuntu 13.10 to 14.04 and now I encounter these problems: gcc used is this: gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) a minor glitch: cmake complains: Syntax Warning in cmake code at /home/august/workspace/MariaDB/mariadb-10.0.4/contrib_AQ_01/cmake/ssl.cmake:210:55 Argument not separated from preceding token by whitespace. This is the reason for this: ELSE() MESSAGE(SEND_ERROR "Wrong option for WITH_SSL. Valid values are : "${WITH_SSL_DOC}) ENDIF() I added a space between "$(WITH and this works now. But in compiling the sources this error was reported: Scanning dependencies of target innobase [ 45%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/row/row0log.cc.o /home/august/workspace/MariaDB/mariadb-10.0.4/contrib_AQ_01/storage/innobase/row/row0log.cc: In function ‘dberr_t row_log_table_apply_ops(que_thr_t*, row_merge_dup_t*)’: /home/august/workspace/MariaDB/mariadb-10.0.4/contrib_AQ_01/storage/innobase/row/row0log.cc:2201:11: error: ‘srv_buf_size’ was not declared in this scope ofs, srv_buf_size); ^ By looking into the code of row0log.cc I found that FALLOC_FL_PUNCH_HOLE seems to be defined now. This is defined in /usr/include/linux/falloc.h. So what happened here during the upgrade of Linux? Regards AugustQ PS: by looking into Ubuntu 12.04 I found that this file also existed here.

2 2

[Maria-developers] Analysing degraded performance at high concurrency in sysbench OLTP
by Kristian Nielsen 02 May '14

02 May '14

At the Barcelona meeting in January, I promised to take a look at the high-concurrency sysbench OLTP benchmarks, and now I finally had the time do do this. There was a lot of work on LOCK_open by Svoj and Serg. If I have understood correctly, the basic problem was that at high concurrency (like, 512 threads), the TPS is only a small fraction of the peak throughput at lower concurrency. Basically, the server "falls over" and starts trashing instead of doing real work, due to some kind of inter-processor communication overhead. I started from Axel' OLTP sysbench runs and scripts, using 10.0 from bzr revno:4151 (revid:svoj@mariadb.org-20140415072957-yeir4jvokyilw5hp). I compiled without performance schema and with PGO, and ran sysbench 0.5 OLTP. (I just realised that my runs are with 32 tables, while I think the benchmarks in January focused on single-table runs. Maybe I need to re-do my analysis with the single-table benchmark, or perhaps it is too artificial to matter much?). In the read-only sysbench, the server mostly does not fall over. I guess this is due to the work by Svoj on eliminating LOCK_open? But in read-write, performance drops dramatically at high concurrency. TPS drops to 2600 at 512 threads compared to a peak of around 13000 (numbers here are approximate only, they vary somewhat between different runs). So I analysed the r/w benchmark with the linux `perf` tool. It turns out two-thirds of the time is spent in a single kernel function _raw_spin_lock(): - 66.26% mysqld [kernel.kallsyms] [k] _raw_spin_lock Digging further using --call-graph, this turns out to be mostly futex waits (and futex wakeups) from inside InnoDB locking primitives. Calls like sync_array_get_and_reserve_cell() and sync_array_wait_event() stand out in particular. So this is related to the non-scalable implementation in InnoDB of locking primitives, which is a known problem. I think Mark Callaghan has written about it a couple of times. Last time I looked at the code, every single mutex wait has to take a global mutex protecting some global arrays and stuff. I even remember seeing code that at mutex release would pthread_signal_broadcast() _every_ waiter, all of them waking up, only to all (except one) go do another wait. This is a kiler for scalability. While investigating, I discovered the variable innodb_sync_array_size, which I did not know about. It seems to split the mutex for some of the synchronisation operations. So I tried to re-run the benchmark with innodb_sync_array_size set to 8 and 64. In both cases, I got significant improvement, TPS increase to 5900, twice the value with innodb_sync_array_size set to the default of 1. So it is clear that the main limitation in this benchmark was the non-scalable InnoDB synchronisation implementation. After tuning innodb_sync_array_size, time spent in _raw_spin_lock() is down to half what it was before (33% of total time): + 33.77% mysqld [kernel.kallsyms] [k] _raw_spin_lock Now investigating call-graphs show that the sync_array operations are much less visible. Instead mutex_create_func(), called from dict_mem_table_create(), is the one that turns up prominently in the profile. I am not familiar with what this part of the InnoDB code is doing, but what I saw from a quick look is that it creates a mutex - and there is another global mutex needed for this, which again limits scalability. It is a bit surprising to see mutex creation being the most significant bottleneck in the benchmark. I would have assumed that most mutexes could be created up-front and re-used? It is possible that this is a warm-up thing, maybe the code is filling up the buffer pool or some table-cache like thing inside InnoDB? Because I see TPS being rather low for the first 150 seconds of the run (around 3000), and then increasing suddenly to around 8000-9000 for the rest. This might be worth investigating further. So in summary, my investigations found that the bottleneck in this benchmark, and the likely cause of the fall-over, is a scalability problem with InnoDB locking primitives. The sync_array part seems to be mitigated to some degree by innodb_sync_array_size, the mutex creation part still needs to be investigated. I wonder if the InnoDB team @ Oracle is doing something for this in 5.7? Does anyone know? I vaguely recall reading something about it, but I am not sure. It would seem a waste to duplicate their efforts. In any case, I hope this was useful. As part of this investigation, I installed a new 3.14 kernel on the lizard2 machine and a new `perf` installation, which seems to work well to do more detailed investigations of these kind of issues. So let me know if there are other benchmarks that I should look into. One thing that could be interesting is to look for false sharing; there are some performance counters that Intel manuals describe can be used for this. As an aside: In my tests, once concurrency becomes high enough that the server falls over, the actual TPS number becomes mostly meaningless. Eg. I saw putting dummy pause loops into the code increasing TPS. If TPS stabilises at N% of peak throughput as concurrency goes to infinity, then we can compare N. But if N goes to zero as concurrency goes to infinite, I think it is meaningless to compare actual TPS numbers - we should instead focus on removing the fall-over behaviour. (Maybe this is already obvious to you, I have not followed the previous benchmark efforts that closely). Hope this helps, - Kristian.

4 4

Re: [Maria-developers] MDEV-5792 - Deadlock between SELECTs from METADATA_LOCK_INFO and another
by Sergei Golubchik 01 May '14

01 May '14

Hi, Sergey! On Apr 23, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 4155 > revision-id: svoj(a)mariadb.org-20140423073406-o20n7jvwfjinvxxr > parent: psergey(a)askmonty.org-20140418100754-v5b5sdqbavbcznh7 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0-mdev5792 > timestamp: Wed 2014-04-23 11:34:06 +0400 > message: > MDEV-5792 - Deadlock between SELECTs from METADATA_LOCK_INFO and another > I_S table > > mdl_iterate() helper function (which is used by the plugin to iterate mdl > locks) acquired mutexes in reverse order. > > Fixed by iterating MDL locks in two stages: > 1. Iterate locks hash under the protection of hash mutex, store all > lock pointers in a thread local array and increment reference counter > for the lock. > 2. Iterate local array without protection of hash mutex, handle destroyed > locks. > > It somewhat echoes hack in MDL_map_partition::move_from_hash_to_lock_mutex. Looks ok to me. Regards, Sergei

1 0

[Maria-developers] Next steps in improving single-threaded performance
by Kristian Nielsen 01 May '14

01 May '14

I have been analysing CPU bottlenecks in single-threaded sysbench read-only load. I found that icache misses is the main bottleneck, and that profile-guided compiler optimisation (PGO) with GCC gives a large speedup, 25% or more. (More details in my blog posts: http://kristiannielsen.livejournal.com/17676.html http://kristiannielsen.livejournal.com/18168.html ) Now I would like to ask for some discussions/help in how to get this implemented in practice. It involves changing the build process for our binaries: First compile with gcc --coverage, then run some profile workload, then recompile with -fprofile-use. I implemented a simple program to generate some profile load: https://github.com/knielsen/gen_profile_load It runs a bunch of simple insert/select/update/delete, with different combinations of storage engine, binlog format, and client API. It is designed to run inside the build tree and handle starting and stopping the server being tested, so it is pretty close to a working setup. These commands work to generate a binary that is faster due to PGO: mkdir bld cd bld cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" .. make tests/gen_profile_load cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" make So all the pieces really are there, it should be possible to implement it. But we need to find a good way to integrate it into our build system. The best would be to integrate it into our cmake files. The gen_profile_load.c could go into tests/, ideally we would build both a static and dynamically linked version (so we get PGO for both libmysqlclient.a and libmysqlclient.so). Anyone can help me get cmake to do that? And it would be cool if we could get the above procedure to work completely within cmake, so that the user could just do: cmake -DWITH_PGO ... ; make and cmake would itself handle first building with --coverage, then running gen_profile_load.static and gen_profile_load.dynamic, then rebuilding with -fprofile-use. Anyone know if this is possible with cmake, and if so could help implement it? But alternatively, we could integrate a double build, like the commands above, into the buildbot scripts (.deb, .rpm, bintar). Any comments? Here are some more points: - I tested that gen_profile_load gives a good speedup of sysbench read-only (around 30%, so still very significant even though it generates a different and more varied load). - As another test, I removed all SELECT from gen_profile_load, and ran the resulting PGO binary with sysbench read-only. This still gave a fair speedup, despite the PGO load being completely different from the benchmark load. This gives me confidence that the PGO should not cause performance regressions in cases not covered well by gen_profile_load - More tests would be nice, of course. Axel, would you be able to build some binaries following above procedure, and test some different random benchmarks? Anything that is easy to run could be interesting, both to test for improvement, and to check against regressions. - We probably need a recent GCC version to get good results. I used GCC version 4.7.2. Maybe we should install this GCC version in all the VMs we use to build binaries? - Should we do this in 5.5? I think we might want to. The speedup is quite significant, and it seems very safe - no code modifications are involved, only different compiler options. Any thoughts? Volunteeres for helping with the cmake or buildbot parts? - Kristian.

4 14

[Maria-developers] valgrind failure in main.statistics
by Sergey Petrunia 29 Apr '14

29 Apr '14

Hello Igor, Valgrind build in buildbot currently fails in main.statistics test: http://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/5… I have tracked it down: the failure appeared after this push by you: revno: 4157 [merge] committer: Igor Babaev <igor(a)askmonty.org> branch nick: maria-10.0 timestamp: Tue 2014-04-22 14:43:13 -0700 message: Merge. ------------------------------------------------------------ revno: 4152.1.1 committer: Igor Babaev <igor(a)askmonty.org> branch nick: maria-10.0-bugs timestamp: Tue 2014-04-22 14:39:57 -0700 message: Fixed the problem of mdev-5947. Back-ported from the mysql 5.6 code line the patch with the following comment: Fix for Bug#11757108 CHANGE IN EXECUTION PLAN FOR COUNT_DISTINCT_GROUP_ON_KEY CAUSES PEFORMANCE REGRESSION I suppose it would take you much less time than anybody else to fix this? The failure is not repeatable on every machine: it doesn't repeat on my machines, but it can be repeated on work.askmonty.org. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 10.1 planning
by Kristian Nielsen 29 Apr '14

29 Apr '14

Sergei Golubchik <serg(a)mariadb.org> writes: > See below. > What do you think about it? > > On Apr 23, Elena Stepanova wrote: >> >> There is also https://mariadb.atlassian.net/browse/MDEV-4487 >> (Replication from 5.6 with enabled GTID) to be considered. It's >> currently Minor and has 10.0.x as a target version, both of which are >> questionable: recently Colin said on IRC that it should be much more >> than a minor FR, and yet I doubt that somebody will get to it in 10.0. >> Maybe it's more realistic to raise the priority but to re-target it for >> 10.1? I think Monty had some ideas for doing this, IIRC it was discussed at the meeting in Barcelona in January. If what I remember is true, it is actually just that a 5.6 master refuses to replicate if the slave does not say it is configured for MySQL 5.6 GTID. The slave does not need to actually _use_ GTID. So I think Monty's plan was for the MariaDB slave to just lie to the master and say that it's a 5.6 slave with appropriate config. And the MariaDB slave would need to recognise and ignore the new MySQL 5.6 events, such as GTID. - Kristian.

2 2

[Maria-developers] Old autotools cruft in source tree?
by Otto Kekäläinen 28 Apr '14

28 Apr '14

Hello The Debian Lintian QA scanner yields this warning: W outdated-autotools-helper-file (http://lintian.debian.org/tags/outdated-autotools-helper-file.html) extra/jemalloc/config.guess 2012-02-10 extra/jemalloc/config.sub 2012-02-10 storage/tokudb/ft-index/third_party/xz-4.999.9beta/build-aux/config.guess 2009-04-27 storage/tokudb/ft-index/third_party/xz-4.999.9beta/build-aux/config.sub 2009-04-17 Are these still needed in MariaDB or is this simply old cruft from old autotools time that could be removed? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 1

[Maria-developers] "SELECT ... LIMIT 0" is now deterministic
by Alexander Barkov 28 Apr '14

28 Apr '14

Hello Ian, A fix for MDEV-6170 fixed another problem as a side effect: "SELECT ... LIMIT 0" is now deterministic for statement based binary log. Queries with "LIMIT 0" do not return ER_BINLOG_UNSAFE_STATEMENT any more. Can you please mention this in the manual? Some discussion is here: https://mariadb.atlassian.net/browse/MDEV-6170 Related articles in the MySQL manual: https://dev.mysql.com/doc/refman/5.6/en/replication-rbr-safe-unsafe.html https://dev.mysql.com/doc/refman/5.6/en/replication-features-limit.html Thanks.

1 0

[Maria-developers] a Query_log_event charset bug in parallel replication
by nanyi607rao 25 Apr '14

25 Apr '14

hi, If character_set in different Query_log_events changed, worker threads may apply them with wrong character_set. the codes leading this problem is in Query_log_event::do_apply_event, that is: if (charset_inited) { if (rli->cached_charset_compare(charset)) { /* Verify that we support the charsets found in the event. */ if (!(thd->variables.character_set_client= get_charset(uint2korr(charset), MYF(MY_WME))) || !(thd->variables.collation_connection= get_charset(uint2korr(charset+2), MYF(MY_WME))) || !(thd->variables.collation_server= get_charset(uint2korr(charset+4), MYF(MY_WME)))) { There is a charset[6] in rli, which cached last Query_log_event's charset in serial replication. But in parallel replication it would lead mistake, because every worker thread can read and set rli->charset[6], so rli->charset[6] isn't any worker threads' last Query_log_event charset. but rli->charset[6] can affect every worker threads' thd->variables.character_set_* setting. 2014-04-17 nanyi607rao

2 7

[Maria-developers] About the query performance issue reported on IRC (Percona 5.5 vs MariaDB 10.0)
by Elena Stepanova 24 Apr '14

24 Apr '14

Hi all, Over the last couple of days, some of you have seen or even participated in a "thread" on #maria IRC channel regarding the performance problem on query execution that james_woods observed on MariaDB 10.0 instance comparing to a Percona 5.5 instance. This email is for those who are curious about the outcome/current status. Summary ------- The original complaint sounded like this: "After switching to MariaDB 10 I see 5-10 very long running queries - which are not using indices as they do on Percona 5.5." The data in both instances is exactly the same. The HW is essentially different, but it's better on the machine with MariaDB 10. After some back-and-forth, the complaint got split into three main parts: - the query is considerably slower on MariaDB 10.0 instance than on Percona 5.5 instance; - the query plans show very different row counts, although the data is identical; - Percona uses indices when MariaDB doesn't. Sergei Petrunia investigated the alleged issues with query plans and found that there was no problem: - the row counts that Percona 5.5 reported for derived tables were smaller because EXPLAIN in Percona 5.5 had actually executed the query. - the difference in index usage was an effect of table elimination. A table was removed, hence no index use for it was shown in the explain. He also broke down the query and got a simple flat one, without subqueries, which still revealed ~4x difference in execution time. Upon his request, I ran some tests locally to see if I could reproduce such a difference between Percona 5.5 and MariaDB 10.0 and found that I couldn't (of course I don't have the HW that james_woods uses, the but the difference this big was likely to be seen anywhere). Meanwhile, james_woods installed Percona 5.5 on the machine where MariaDB 10.0 was running, and didn't get any performance gain -- that is, the difference was not between Percona 5.5 and MariaDB 10.0, but was caused by something else (HW, environment, ...). As of now, we don't know what it is. (end of summary). Sergei asked me to share the numbers I got while running the local tests. I will also put here the initial queries and explains, for the reference. It's mainly for the record, there is nothing particularly interesting in here; if you stop reading now, you won't lose much. Queries and plans ----------------- The initial offending query: SELECT SUM(visits) FROM ( SELECT IFNULL(ref.probename_unique_visits,'tr_trafficother2_uv') AS probename, SUM(x.value) AS visits, IFNULL(ref.category,'Other') AS category FROM ( SELECT aggregated_data._getSource(source, source2, source3, fb_source, request_ids) AS source, COUNT(DISTINCT u.fbuserid) AS value FROM g9_tracker.app_logins u WHERE u.created_date = '2014-04-22' GROUP BY 1 ) x LEFT JOIN aggregated_data.ref_parameter_mappings ref ON (ref.value_of_ref_parameter = x.source) GROUP BY 1,3 ) y; EXPLAINs (that's where row counts were off; note EXPLAIN execution time): MariaDB 10.0: +------+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 3394453 | | | 2 | DERIVED | <derived3> | ALL | NULL | NULL | NULL | NULL | 3394453 | Using temporary; Using filesort | | 2 | DERIVED | ref | ref | value_of_ref_parameter_UNIQUE | value_of_ref_parameter_UNIQUE | 603 | x.source | 1 | Using where | | 3 | DERIVED | u | ALL | NULL | NULL | NULL | NULL | 3394453 | Using where; Using filesort | +------+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ 4 rows in set (0,03 sec) Percona 5.5: +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 15 | | | 2 | DERIVED | <derived3> | ALL | NULL | NULL | NULL | NULL | 75 | Using temporary; Using filesort | | 2 | DERIVED | ref | ref | value_of_ref_parameter_UNIQUE | value_of_ref_parameter_UNIQUE | 603 | x.source | 1 | | | 3 | DERIVED | u | ALL | NULL | NULL | NULL | NULL | 3669609 | Using filesort | +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ 4 rows in set (1 min 40,20 sec) Example of the indices difference (caused by table elimination): Query: SELECT COUNT(*) FROM ( SELECT aggregated_data._getSource(source, source2, source3, fb_source, request_ids) AS source, COUNT(DISTINCT u.fbuserid) AS value FROM g9_tracker.app_logins u WHERE u.created_date = '2014-04-22' GROUP BY 1 ) x LEFT JOIN aggregated_data.ref_parameter_mappings ref ON (ref.value_of_ref_parameter = x.source); EXPLAINs: MariaDB 10.0: +------+-------------+------------+------+---------------+------+---------+------+---------+----------+-----------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +------+-------------+------------+------+---------------+------+---------+------+---------+----------+-----------------------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 3394453 | 100.00 | | | 2 | DERIVED | u | ALL | NULL | NULL | NULL | NULL | 3394453 | 100.00 | Using where; Using filesort | +------+-------------+------------+------+---------------+------+---------+------+---------+----------+-----------------------------+ 2 rows in set, 1 warning (0,02 sec) Percona 5.5: +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+----------+----------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+----------+----------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 75 | 100.00 | | | 1 | PRIMARY | ref | ref | value_of_ref_parameter_UNIQUE | value_of_ref_parameter_UNIQUE | 603 | x.source | 1 | 100.00 | Using index | | 2 | DERIVED | u | ALL | NULL | NULL | NULL | NULL | 3669609 | 100.00 | Using filesort | +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+----------+----------------+ 3 rows in set, 1 warning (1 min 43,62 sec) Numbers ------- The query that Sergei suggested to run in local tests: SELECT SUM(varchar_non_indexed_column_with_non_empty_different_values='foo') FROM table_with_3_mio_rows I ran tests on Percona 5.5/5.6, MariaDB 5.5/10.0, MySQL 5.5/5.6. Only the relevant results are here, I'll skip the rest. For every run the caches were cleaned, the server was started, and then the queries were executed, 3 times each. Initially servers were run with the default options (not server default, but packaging default, how they are installed on Wheezy). The difference between 5.5 and 5.6/10.0 was astonishing, but it was explained by Performance Schema. It went like this: 5.5 select sum(c2='foo') from t_innodb; 1 row in set (8.17 sec) 1 row in set (0.99 sec) 1 row in set (0.95 sec) select sum(c2='foo') from t_myisam; 1 row in set (3.04 sec) 1 row in set (0.56 sec) 1 row in set (0.55 sec) 5.6 / 10.0 select sum(c2='foo') from t_innodb; 1 row in set (24.24 sec) 1 row in set (21.65 sec) 1 row in set (23.43 sec) select sum(c2='foo') from t_myisam; 1 row in set (18.83 sec) 1 row in set (17.23 sec) 1 row in set (16.05 sec) Pretty much the same picture for MariaDB, Percona and MySQL. It would explain everything, but it turned out that james_woods already had Performance Schema turned off. (Note: we *really* should consider switching off Performance schema, at least in configs provided upon the new package installation). Then I ran another test, with the PS turned off, and otherwise default options (now really default, without any pre-installed config). There was still a difference between 5.5 and 5.6/10.0, although not that big, and for InnoDB only: Percona 5.5 select sum(c2='foo') from t_innodb; 1 row in set (5.97 sec) 1 row in set (0.79 sec) 1 row in set (0.79 sec) select sum(c2='foo') from t_myisam; 1 row in set (3.68 sec) 1 row in set (0.56 sec) 1 row in set (0.56 sec) Percona 5.6 select sum(c2='foo') from t_innodb; 1 row in set (7.33 sec) 1 row in set (2.32 sec) 1 row in set (1.48 sec) select sum(c2='foo') from t_myisam; 1 row in set (2.90 sec) 1 row in set (0.59 sec) 1 row in set (0.59 sec) MariaDB 5.5 select sum(c2='foo') from t_innodb; 1 row in set (8.17 sec) 1 row in set (0.99 sec) 1 row in set (0.95 sec) select sum(c2='foo') from t_myisam; 1 row in set (3.04 sec) 1 row in set (0.56 sec) 1 row in set (0.55 sec) MariaDB 10.0 select sum(c2='foo') from t_innodb; 1 row in set (7.34 sec) 1 row in set (2.07 sec) 1 row in set (2.07 sec) select sum(c2='foo') from t_myisam; 1 row in set (3.49 sec) 1 row in set (0.58 sec) 1 row in set (0.57 sec) MySQL 5.6 select sum(c2='foo') from t_innodb; 1 row in set (7.88 sec) 1 row in set (1.95 sec) 1 row in set (2.11 sec) select sum(c2='foo') from t_myisam; 1 row in set (4.11 sec) 1 row in set (0.55 sec) 1 row in set (0.57 sec) So, on 5.6/10.0 with default options the query from the "warm" InnoDB table was approximately 2x slower than on 5.5. Then I took non-default options from james_woods and applied them, as much as I could, considering the HW difference, to my modest VM. In fact, only innodb_buffer_pool_size mattered. After doubling it on 5.6/10.0, I got pretty much the same results as on 5.5. Percona 5.5 (default options) select sum(c2='foo') from t1; 1 row in set (5.97 sec) 1 row in set (0.79 sec) 1 row in set (0.79 sec) Percona 5.6 (default options + no PS + increased buffer pool) select sum(c2='foo') from t1; 1 row in set (8.33 sec) 1 row in set (0.83 sec) 1 row in set (0.82 sec) MariaDB 5.5 (default options) select sum(c2='foo') from t1; 1 row in set (8.17 sec) 1 row in set (0.99 sec) 1 row in set (0.95 sec) MariaDB 10.0 (default options + no PS + increased buffer pool) select sum(c2='foo') from t1; 1 row in set (7.85 sec) 1 row in set (0.86 sec) 1 row in set (0.86 sec) There are still some minor differences, but my VM is not suitable for fine-grained performance testing. If somebody wants to investigate it further, it should be done in a proper way; in any case I'm sure it can be tuned and made to perform better on 5.6/10.0, nobody who cares about differences this small actually runs the server with default options. Regards, /E

1 0

[Maria-developers] Difference between 0xHHHH and X'HHHH' hex literals
by Alexander Barkov 24 Apr '14

24 Apr '14

Hello Ian, I tried to explain in https://mariadb.atlassian.net/browse/MDEV-6092 to the complainer why it's not a bug. Can you please document the difference between the two hex notations and then close as "not a bug"? Thanks.

1 0

Re: [Maria-developers] bb01.mariadb.net
by Kristian Nielsen 23 Apr '14

23 Apr '14

Daniel Bartholomew <dbart(a)mariadb.com> writes: > Serg and Knielsen: I've got the box mostly configured I think, but > could be (very) wrong, so I wanted to give you a chance to make sure > everything looks good on it before I go ahead and add it to the It looks like it already mostly works. I noticed an error in a build though: https://buildbot.askmonty.org/buildbot/builders/kvm-tarbake-jaunty-x86/buil… The "kernel" log has this: ----------------------------------------------------------------------- /dev/sda1 contains a file system with errors, check forced. /dev/sda1: Entry 'pack-names' in /home/buildbot/.bzr/repository (3415) has an incorrect filetype (was 1, should be 2). /dev/sda1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. (i.e., without -a or -p options) ----------------------------------------------------------------------- One likely reason is a failed rsync step. This is the tarbake build; unlike most other virtual machines, this one is actually modified during a build. So it's quite possible that the rsync would go wrong if a tarbake build happened to run in the meantime. So just re-sync that particular VM, and that might solve the problem. Thanks for setting this up! - Kristian.

2 1

[Maria-developers] [GSoC] Accepted student ready to work : )
by Pablo Estrada 23 Apr '14

23 Apr '14

Hello Sergei, Elena and everyone, I'm Pablo, and I was selected as part of Google Summer of Code 2014 to work on the project "*Statistically optimize mysql-test runs by running less tests*". First of all, thank you very much for having selected me for the project. I am very excited to start reading, reviewing data and coding up. I see that Elena will be my mentor. Nice to meet you Elena : ) To begin with, I am focusing on the *Buildbot documentation*, to *(1)understand how the data is organized* inside it, and to *(2)*figure out *how to add the extra functionality* into the BuildBot. I want to get started with the data analysis, so it would be great if you could help me with the following things: 1. Sergei, you said you did a bit of research into this. Do you have your *previous results*, or *any papers* that you have read that you could suggest to me? 2. Can I get *access to the buildbot database* to look at the data? If not, can I get the E/R model, or any scheme of *how the data looks like?* - I understand that maybe the data is not just a single database. I am starting by *reading the buildbot documentation*, to understand how the stuff is organized in it. 3. *Any other information* that you consider useful for me to know? Any advice? And that's all I have for now. I'm looking forward to getting started! Regards Pablo

4 5

[Maria-developers] mdev6027 RLIKE: "." no longer matching new line (default_regex_flags)
by Alexander Barkov 23 Apr '14

23 Apr '14

Hello Serg, Please review a patch implementing a new system variable default_regex_flags, to address the remaining incompatibilities between PCRE and the old regex library. Greetings.

3 5

[Maria-developers] Proposal Acceptance - GSOC
by Puneet Dewan 22 Apr '14

22 Apr '14

Hi Everyone, I, Puneet Dewan have got accepted into Google Summer Of Code 2014. This is the first and the last time I applied for GSOC and getting accepted is *really motivating*. :) My Project is : *Support of UseServerPrepStmts to MariaDB Java Connector.* *Massimo* is my mentor. I will start working on the project after 25th April as my semester exams are going on. I am excited to be part of MariaDB and looking forward to contribute to the community. Regards Puneet.

1 0

[Maria-developers] [GSOC] Proposal accepted!
by sriram patil 22 Apr '14

22 Apr '14

Hi Sergei, Alexander and all, Thank you for accepting my proposal. This was the first time I applied for GSoC, so I was kind of nervous. I was selected to work on the project *"CREATE OR REPLACE, CREATE IF NOT EXISTS and DROP IF EXISTS" *as part of GSoC'14. My end semester exams are going on right now. I will be done with my exams and projects by 29th April. I am totally excited to get started. I have built the source code already. Have learnt a few things about the source code as you must have observed in my proposal. I will continue with the documentation and source code from 1st May. Is there anything else you want me to look into? Thanks, Sriram

1 0

Re: [Maria-developers] [Commits] Rev 4153: MDEV-5963: InnoDB: Assertion failure in file row0sel.cc line 2503 ... in file:///home/psergey/dev2/10.0-cp/
by Jan Lindström 18 Apr '14

18 Apr '14

Sergey, > === modified file 'sql/opt_range.cc' > --- a/sql/opt_range.cc 2014-04-01 16:59:51 +0000 > +++ b/sql/opt_range.cc 2014-04-18 10:07:54 +0000 > @@ -2181,7 +2181,7 @@ int QUICK_ROR_INTERSECT_SELECT::init_ror > quick->record= head->record[0]; > } > > - if (need_to_fetch_row && head->file->ha_rnd_init_with_error(1)) > + if (need_to_fetch_row && head->file->ha_rnd_init_with_error(false)) You had so nice and informative reason why this has to be like this, why not add big fat comment here ? > { > DBUG_PRINT("error", ("ROR index_merge rnd_init call failed")); > DBUG_RETURN(1); > @@ -2363,8 +2363,13 @@ int QUICK_ROR_UNION_SELECT::reset() > quick->save_last_pos(); > queue_insert(&queue, (uchar*)quick); > } > - > - if ((error= head->file->ha_rnd_init(1))) > + /* Prepare for ha_rnd_pos calls. */ > + if (head->file->inited && (error= head->file->ha_rnd_end())) > + { > + DBUG_PRINT("error", ("ROR index_merge rnd_end call failed")); > + DBUG_RETURN(error); > + } > + if ((error= head->file->ha_rnd_init(false))) > { > DBUG_PRINT("error", ("ROR index_merge rnd_init call failed")); > DBUG_RETURN(error); > > _______________________________________________ > R: -- -- Jan Lindström, Principal Engineer SkySQL - The MariaDB Company MariaDB | MaxScale | skype: jan_p_lindstrom www.skysql.com <http://www.skysql.com/> Twitter <http://twitter.com/skysql> Blog <http://www.skysql.com/blog/> Facebook <http://www.facebook.com/skysql> LinkedIn <http://www.linkedin.com/company/1214250> Google+ <https://plus.google.com/117544963211695643458/posts>

1 0

[Maria-developers] TokuDB disabled in Debian and Ubuntu - help me get it working?
by Otto Kekäläinen 18 Apr '14

18 Apr '14

Hello Rick, Last year I spent a lot of time packaging MariaDB 5.5 for Debian and finally this year it has landed in Ubuntu 14.04 and Debian testing. Unfortunately the Debian/Ubuntu version does not include TokuDB and I need your help to get it there. In 5.5.35 (I think) the TokuDB plugn was added to MariaDB but I had issues getting it build 100% correctly and I eventually dropped it (added build parameter -DWITHOUT_TOKUDB=true), as getting MariaDB in Debian at all was a bigger priority than getting it there with every possible plugin. The root cause seems to be that when Debian and Ubuntu packages are built in chroot environments (the build systems of Debian and Ubuntu use pbuilder/sbuilder systems, see https://en.wikipedia.org/wiki/Debian_build_toolchain#Isolated_build_environ…) the code that builds the plugin does not seem to correctly detect the CPU features. It seems to read the values from the build machine and not the inputted target values (in a cross-compile situation). There are two related issues that needs a solution: 1) Currenlty the code that checks what the architecture is (32-bit/64-bit) is the first lines of https://bazaar.launchpad.net/~maria-captains/maria/10.0/view/head:/storage/…. This works well for real and virtual machhines, but it does not seem to work in the pbuilder/sbuilder chroots, as CMAKE_SYSTEM_PROCESSOR always shows the chroot host CPU, not the cross-compile target CPU. Could you please investigate pbuilder/sbuilder and search for some solution that works for reliable target CPU checking? 2) When building TokuDB in Ubuntu (amd64) sbuilder environments something in crashes in the 'toku_os_get_processor_frequency' function. For this too, could you investigate the sbuilde chroot environment and figure out what goes on and how to fix it? Issue 2 has a bug report with the (a bit messy) debugging history documented: https://mariadb.atlassian.net/browse/MDEV-5618 Both of these issues requires learning a bit about sbuilder CPU things, so I assume it is most efficient if the same persons looks into both of these. Thanks! -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

3 5

[Maria-developers] Ubuntu 14.04 and MariaDB
by Otto Kekäläinen 17 Apr '14

17 Apr '14

Ubuntu 14.04 was released today. >From https://wiki.ubuntu.com/TrustyTahr/ReleaseNotes#Ubuntu_Server "MySQL Ubuntu 14.04 LTS continues to provide official support for MySQL 5.5. Three other community supported alternatives of MySQL are also included: * MariaDB 5.5 - a drop in replacement for MySQL 5.5. * Percona XtraDB Cluster 5.5 - a Galera based Active/Active MySQL solution. * MySQL 5.6 - the next release of Oracle's MySQL. Note that upgrading to MySQL 5.6 is an automatic one way process; it is possible to downgrade manually - see the upstream documentation on details of how to perform this process." I have been working on 5.5.37 today, I hope to soon get it uploaded into Debian and then via security update process into Ubuntu too. -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

1 0

Re: [Maria-developers] [GSoC] Student interested in "Optimize mysql-test runs" or adding SQL features
by Sergei Golubchik 17 Apr '14

17 Apr '14

Hi, Pablo! First: please always cc: maria-developers(a)lists.launchpad.net, don't send emails to me only. Now: On Mar 11, Pablo Estrada wrote: > Hello! > I am getting ready to submit my proposal. Is there any feedback for my > application document? Anything that would be good to add, remove, rephrase, > etc? Go on and submit it. There will be a comment form under your proposal, if I will want you to change something, I will leave my comments there. Please mention what tools/languages you think you'll be using. > Also, if I could see the data available (the history of test results), I > could write more in detail : ) Hm. Okay. Attached is the preprocessed table that I used two years ago. It lists test failures in buildbot in a given period of time. Columns are (as far as I remember): * time of the test run start * code branch that was tested * revision id that was tested - it's fake, our buildbot tables don't store revision id. I've constructed it from the set of changed files, revision number, etc. hopefully it's as unique as a real revision id * build number * platform where the tests were run * test type - basically, the set of tests to run and what protocol to use. one test may be run with different protocols * test name * test variant - or combination. certain tests may be run in many combinations Using tis file I tried to rearrange test execution order for every specific build number (using failure data from previous builds in the same table) to detect test failures as early as possible. I managed to have 90% of all test failures happening within first 10% of the tests. The goal of this project is to have something we can put back into buildbot, so that it could actually run only 10% of the tests, still having 90% probability of discovering a failure (or 20%, or 95%, whaveter). See also http://buildbot.askmonty.org/buildbot/ you can click on a link for any branch, for example, http://buildbot.askmonty.org/buildbot/grid?branch=10.0 then on any particular build, for example, http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297 and on a stdio log for a particular test run: http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297/ste… on these pages you'll see what a platform (builder) is, build number, test types, combinations, etc. Regards, Sergei

2 2

[Maria-developers] Problems with Launchpad?
by Andrew McDonnell 17 Apr '14

17 Apr '14

Hi, my mariab bzr branch on launchpad doesnt update the summary page properly, and hasn't done so for a while now, which means recent work I committed is not visible unless you drill right down into the branch change history (My related question on Launchpad Answers - https://answers.launchpad.net/launchpad/+question/246850 - hasn't been touched either) Is anyone else experiencing this? cheers, Andrew -- http://blog.oldcomputerjunk.net https://launchpad.net/~andymc73 https://github.com/andymc73 Twitter: @pastcompute GPG: http://www.andrewmcdonnell.net/gpg.html

2 1

[Maria-developers] MariaDB 5.5.37 release prep
by Daniel Bartholomew 17 Apr '14

17 Apr '14

All, I've begun release prep for MariaDB 5.5.37. Draft changelog and release notes are at: - https://mariadb.com/kb/en/mariadb-5537-release-notes/ - https://mariadb.com/kb/en/mariadb-5537-changelog/ Please let me know ASAP of any notable things that should be mentioned in the release notes. Thanks! -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

3 3

[Maria-developers] MariaDB scalability: mutexes
by Sergey Vojtovich 16 Apr '14

16 Apr '14

Hi Axel, I benchmarked various mutexes available to MariaDB. Results are here: http://svoj-db.blogspot.ru/2014/02/mariadb-mutexes-scalability.html Among other things I noticed that normal mutexes scale better than adaptive. And it looks like it gives over 5 000 TPS in my benchmarks: Original -------- 64 threads, time spent: 60s, queries executed: 11500723, qps: 191678, 1 thread qps: 2994, tps: 13691 64 threads, time spent: 60s, queries executed: 10983665, qps: 183061, 1 thread qps: 2860, tps: 13075 64 threads, time spent: 60s, queries executed: 11080325, qps: 184672, 1 thread qps: 2885, tps: 13190 slow LOCK_open -------------- 64 threads, time spent: 60s, queries executed: 12948365, qps: 215806, 1 thread qps: 3371, tps: 15414 64 threads, time spent: 60s, queries executed: 13191584, qps: 219859, 1 thread qps: 3435, tps: 15704 64 threads, time spent: 60s, queries executed: 13050647, qps: 217510, 1 thread qps: 3398, tps: 15536 slow LOCK_open + THR_LOCK::mutex -------------------------------- 64 threads, time spent: 60s, queries executed: 15863174, qps: 264386, 1 thread qps: 4131, tps: 18884 64 threads, time spent: 60s, queries executed: 15761944, qps: 262699, 1 thread qps: 4104, tps: 18764 64 threads, time spent: 60s, queries executed: 15773239, qps: 262887, 1 thread qps: 4107, tps: 18777 Could you do official benchmarks (patch attached)? It would be nice to also benchmark MySQL-5.6 with thr_lock part of the patch. Thanks, Sergey

3 8

[Maria-developers] Error on 'make' of mariadb source
by Kesha Shah 14 Apr '14

14 Apr '14

Hi all, I did some changes with in the sql_yacc.yy. During building the source , I am getting following error (attached screenshot) when I try to execute the "make" command after doing a cmake of the source. It says permission denied while trying to make jemalloc-configure. Although i am executing the make command with root user. [image: Inline image 1] [image: Inline image 2] Can somebody provide/help me with some pointers on how to proceed to resolve it ? -- Thanks and Regards, Kesha Shah. www.keshashah.com

2 3

[Maria-developers] MariaDB release and support policies
by Otto Kekäläinen 14 Apr '14

14 Apr '14

Hello! Does MariaDB promise a 8 year release cycle for 5.5 as MySQL does? Does all of the points listed at https://lists.ubuntu.com/archives/technical-board/2014-February/001812.html apply for MariaDB as well? Can we promise Ubuntu that all MariaDB minor releases fulfill the quality requirements listed above? If so, I'll then file a micro release update permission request for Ubuntu. AFAIK as I know it is the only way to get security releases to users anyway. PS. Do you people know about semver.org and try to follow the general versioning policy? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 2

[Maria-developers] Third bunch of unmerged items
by Sergey Vojtovich 14 Apr '14

14 Apr '14

Hi Sergei, here comes third bunch of items which I'm unsure about: WL#5706/Bug#58712/Bug#11746378 - Encrypt or remove passwords from slow, query, and binary logs WL#4677 - Unique Server Ids for Replication Topology (UUIDs) WL#6004 - Server extension for allowing user tables to be put into transactional SE Bug #14112682 : MYSQLTEST: PROVIDE ACCESS TO ERROR NAMES THROUGH NUMERIC CODES AND VICE VERSA Bug #11747313 31741: MYSQLTEST - DEPRECATE "SYSTEM" COMMAND WL#2739: Auditing Password Security (plugin + API) WL#6587: disconnect_on_expired_password WL#5825 Using C++ Standard Library with MySQL code WL#2111: #6 Set CLASS_ORIGIN and SUBCLASS_ORIGIN Regards, Sergey

2 1

Re: [Maria-developers] MariaDB 10.x GTID
by Kristian Nielsen 11 Apr '14

11 Apr '14

Jan Lindström <jplindst(a)mariadb.org> writes: > some background, lets assume we have two galera clusters that are > geographically distributed (lets say for example that cluster A is on Europe > and cluster B on USA). Inside a cluster all nodes are set up so that server_id > and domain_id are the same and we will use strict GTID mode (for a example on > cluster A server_id = 1 and domain_id =1 and in cluster B server_id = 2 and > domain_id = 2). In cluster we naturally use Galera replication. Additionally, > we have asynchronous replication (MySQL replication) between these clusters. > 1) We would like to have some special domain_id (i.e. GTID) for transactions > that are executed only on certain cluster e.g. on cluster A. This would mean > that there is some dynamic option to enable "local" transaction execution mode, > where transactions are executed on cluster A but not on cluster B. I know that > this kind of feature does not exists but is it even possible ? Basically, this > would mean for MySQL replication point of view a fact that if GTID is formed > like this special GTID, they are not replicated to slave, but they are written > to binlog. It sounds like what you want is some option on the slave that would cause it to ignore all GTIDs with some specific domain id, similar to how it is currently possible to filter on server id. That seems a reasonable request; such option does not exist currently though. Another option is to use the @@skip_replication variable. Set this for the sessions doing the "local" transactions. Then set replicate_events_marked_for_skip to either FILTER_ON_SLAVE or FILTER_ON_MASTER, as you like. This is specifically designed to allow to log certain transactions to the binlog but not replicate them, so it sounds like it is what you asked for. > 2) There will be some challenge in maintaining the sequence number for GTID > correctly: for global GTID purposes, we can just count commits, and skip every > local transaction commit, but, we must also initialize the sequence for every > node joining the cluster. In MySQL-5.6 this happens by copying binlog files > (only header part needed) to joining node. I wonder if same strategy works for > MGC-10, is MariaDB GTID is stored in binlog header ? Yes, the GTID is stored in the binlog header. It is also stored in a file master-bin.state. This file is read at server start to initialise the GTID sequence numbers. However, if the file is not found for any reason, the information is extracted from the last binlog file (the master-bin.state is just a cache). However, you need the whole binlog file (if master-bin.state is missing), not just the header. I am sure it must be the same for MySQL-5.6 - you need to scan to the end of the binlog file, as there may be extra GTIDs logged in the file that were not there when the header was written. Of course, just after server start, the most recent binlog file is empty, so just the header works. It sounds like you would want to just copy over the master-bin.state file. However, this whole approach sounds wrong to me. The current GTID state is a cluster-wide property, not a local property, I do not understand why you want to maintain it locally. As I understand it, Galera decides on commit order. The certification process decides for parallel transactions T1 and T2 which one will be committed before the other. The GTID sequence has to match the commit order. So the GTIDs need to be generated by Galera, at the point in the code where the commit order is decided. This Galera code then also needs to do the binlog writing, so that binlog order matches GTID order and commit order. With Galera handling GTID number allocations, I do not understand why you would need to worry about binlog headers. But maybe I have misunderstood, I do not know Galera in details. Maybe you could explain how Galera ensures matching commit order, binlog order, and GTID order? Or alternatively, explain how things work without such ensurance? > 3) Independent table (in data dictionary) or file for replication filters. I > can change the replication filters dynamically, but if the server crash and i > don?t report to my.cnf it?s lost. Therefore, there is a need to store > replication filters persistently and these should be loaded automatically when > server is restarted. Currently we do not have this, replication filters are configuration options, and those are stored in the config file, not in dynamic tables or files. The server_id filters are set by CHANGE MASTER though, IIRC, so they are stored in master.info / relay_log.info. > 4) Dynamic binlog. Currently options are read-only. This we do not have. There has been talk of allowing it under global read lock; this might work, but it would require some work to carefully check all of the code for possible issues. There are a lot of places in the code where expensive synchronisation/locking is avoided based on the knowledge that the binlog can not be enabled or disabled on-the-fly. Hope this helps, - Kristian.

1 0

[Maria-developers] debugging MDEV-5932, wrong result set for tokudb
by Rich Prohaska 10 Apr '14

10 Apr '14

Hello, TokuDB generates an empty result set for a select with sub-select, but generated a non-empty result set for the same query when creating a new table. The result set should be the same in both cases. What is going on here? InnoDB computes an empty result set for both cases. I traced the problem to the tokudb::index_read function. The tokudb::index_read does not handle the null byte key prefix. Unfortunately, I do not understand how the null byte (byte 0) in the key handed to the ::index_read function is supposed to work. Most of the time, the value of this byte is 0. Sometimes it is set to 1. How should the storage engine work WRT this? BTW, this problem exists with TokuDB in MySQL 5.5.36, MySQL 5.6.16. I have not been able to consistently see this on MariaDB 5.5.36 for some reason. In any case, I am interested in how the handler's index_read function is supposed to process the null byte. Thanks

2 1

[Maria-developers] Buildbot machines
by Rasmus Johansson 09 Apr '14

09 Apr '14

Hi, I discussed both with Dbart and Serg last week about the situation with buildbot. Kristian and Elena also had similar discussion on IRC today. To sum it up, there seems to be two aspects to it; 1) more machines, i.e. slaves could/should be added immediately and 2) some development work could/should be done on buildbot to support some other ways of building, like automatically spinning up and down platform specific virtual machines in a public cloud somewhere whenever they are needed. I looked into alternatives for the 1) immediate need of more machines and thought. I found a pretty interesting alternative from OVH.COM, who offers dedicated servers to a very reasonable price. Here is a good example of what they can offer: https://www.ovh.com/us/dedicated-servers/infra/2014-EG-32.xml . I would be prepared to give it a try. Do you think the specs are good enough to help with the load buildbot currently has? BR, Rasmus -- Rasmus Johansson, VP Engineering MariaDB | t: +358 50 499 9589 | Skype: ratzpo

3 5

Re: [Maria-developers] [Commits] Rev 3991: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 09 Apr '14

09 Apr '14

Hi, Alexey! On Apr 02, Alexey Botchkov wrote: > > Still, another question, why did you have to rename net_store_data() to > > net_store_data_cs() ? > > That's personal. > I really don't like functions with exactly same names - makes code harder to > understand and debug. > Do you belive it's better not to rename these here? That's up to you. Sometimes I create functions with the same name, sometimes I don't. For example, tdc_acquire_share() - there are few functions with the same name, because they do exactly the same. Actually, one of tdc_acquire_share() is a real function, others are inline convenience wrappers - they eventually call the "real" one. But tdc_acquire_share_shortlived() - while also only a convenience wrapper - has a different name to emphasize that it has different semantics. But anyway, that's your patch. If you'd like to rename - go ahead. Regards, Sergei

1 0

[Maria-developers] INT thread variables in mysql and mariadb (all versions as far as i know)
by Rich Prohaska 09 Apr '14

09 Apr '14

suppose that I have a THDVAR_INT variable with min = -1 and max = 1, and i set it to -1. how come select @@var returns 4294967295 instead of -1?

2 1

[Maria-developers] Error in "make install"
by sriram patil 09 Apr '14

09 Apr '14

Hello all, I have had built and installed the mariadb source code before, no hassles whatsoever. I was trying to install it again but getting the following error when trying to "make install" ....... -- Installing: /usr/local/mysql/mysql-test/././var/tmp -- Installing: /usr/local/mysql/mysql-test/././var/tmp/mysqld.2 -- Installing: /usr/local/mysql/mysql-test/././var/tmp/mysqld.1.sock CMake Error at mysql-test/cmake_install.cmake:36 (FILE): file INSTALL cannot copy file "/home/sriram/Downloads/mariadb/10.0/mysql-test/./var/tmp/mysqld.1.sock" to "/usr/local/mysql/mysql-test/././var/tmp/mysqld.1.sock". Call Stack (most recent call first): cmake_install.cmake:119 (INCLUDE) make: *** [install] Error 1 I have changed the owner and group of the folder "/usr/local/mysql", so root access is not an issue. I am using Ubuntu 12.04. Did anyone get such error while installing and was able to solve it? Thanks, Sriram

2 3

[Maria-developers] Vision for next 10.x minor releases
by Honza Horak 08 Apr '14

08 Apr '14

Hi guys, first, congratulations for the 10.0 release, well done! Even though it can look insane, I'm already wondering about next minor releases ;) Especially if you have already some idea about 10.1 and later. Specifically, I'm wondering if these releases will still be somehow tight to MySQL's release cycle (say 10.1 will be somehow following 5.7) or if you plan totally independent release schedule like slower/faster the release cadence? In other words, since MySQL changed only minor version and MariaDB bumped the major (we all know the reason), I'm wondering if something changed in thinking about major/minor versioning -- like if the difference between 10.0 and 10.1 will still be considered the same like between say 5.3 and 5.5. I don't expect you can give us any dates, but some expectations, if 10.1 can be expected in months or rather years? If that was already discussed somewhere, please, just point me to the correct place. Thanks, Honza

2 2

Re: [Maria-developers] Rev 4077: MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client
by Sergei Golubchik 07 Apr '14

07 Apr '14

Hi, Holyfoot! On Feb 22, holyfoot(a)askmonty.org wrote: > revno: 4077 > revision-id: holyfoot(a)askmonty.org-20140222173543-xkzlp5rpga41evur > parent: svoj(a)mariadb.org-20140218065405-vw3bfhvhrfjsc6hh > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-4856 > timestamp: Sat 2014-02-22 21:35:43 +0400 > message: > MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client. > The fill_schema_table() function used to call get_table_share() > for a table name in WHERE then clear the error list. That way > plugins receive the superfluous error notification if it happens > in it. Also the problem was that error handler didn't prevent > the suppressed error message from logging anyway as the logging > happens in THD::raise_condition before the handler call. > Fixed by adding the Warnings_only_error_handler before the call. > raise_condition() also fixed. The Internal_error_handler class > was exteded with virtual functions to handle storing the error > message. This looks too complex to me. Why would you extend the base Internal_error_handler class - no other error handler did that or needed that, fwiw. I'd simply push an error handler, store the first message there and access it in get_schema_tables_record() via THD::get_internal_handler(). Regards, Sergei

2 4

[Maria-developers] md5sum mariadb-10.0.10.tar.gz
by Jakub QB Dorňák 07 Apr '14

07 Apr '14

Hi guys, I am getting mariadb-10.0.10.tar.gz with md5 checksum of d41d8cd98f00b204e9800998ecf8427e, while it should be 14ce22b8197d4eae88d237776d47220f. Thank You for any help, QB

2 2

[Maria-developers] Getting started
by sriram patil 04 Apr '14

04 Apr '14

Hi Sergei, I have got myself familiarized with parsing and execution source code. I wanted to start working on the source code a little bit. But I could not find any open bugs on MariaDB launchpad. It would really help if you could give me a few open bugs or enhancements to start with. Thanks, Sriram

2 1

[Maria-developers] [GSOC 2014] CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS
by Kesha Shah 03 Apr '14

03 Apr '14

Dear Sergei Golubchik, Hope this mail finds you well. I am Kesha Shah, a third year undergraduate from DA-IICT, Gandhinagar, India. I was a Google Summer of Code student previous year as well as Google Code-In mentor with BRL-CAD open source solid modeling software. The activities related to my contributions at BRL-CAD at logged by me at - http://brlcad.org/wiki/User:KeshaSShah/GSoC13/Reports. My other projects can be found at - https://github.com/keshashah?tab=repositories. I have good programming hands on C and C++ and would like to take up and contribute at the Mariadb task of 'CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS' [MDEV-5359] as part of GSOC 2014. I have studied database systems and compiler construction during my academics. Your project would help to apply my knowledge by working at deeper level of coding database internals, thereby contributing to open source software. I had checked out the mariadb master branch from bazaar and started looking into the source code. I found the task more related to update of the existing mariadb SQL parser. The parser seems to be implemented as ascendant parser using bison parser generator (GNU version of YACC). From the current findings I could see the YACC grammar file sql_yacc.yy under maria/sql/ where I can see the rules defined for CREATE PROCEDURE_SYM, CREATE FUNCTION_SYM, CREATE TRIGGER_SYM etc. Alternatively I can see the *_SYM lexical tokens defined under the /maria/sql/lex.h file. Similarly for DROP TABLESPACE_SYM, DROP TRIGGER_SYM etc could be found. So when sql text input stream is encountered from SQL interface, it is read and categorized into lexical tokens by the lexer /maria/sql/sql_lex.cc in the lexical analysis step. I tried to figure out that we will have to define additional token 'IF' in the lex.h and 'NOT' and 'EXISTS' are already defined. I am also looking at the existing user defined actions that builds the AST nodes (sp_head being the root node for most types) when relevant grammar rule is satisfied in case of DDL statements. After semantic analysis (create of stored procs/functions), the next step would be to amend/define actions for the newly introduced grammar of handling check of existence of db artefact at the time of DDL execution. Also at this step the compiler will have to query the relevant storage engine on DB level or repository level to check if the DB object exists. This would require me to spend more time on understanding design pattern followed for the implementation of classes and behaviours of each of the underlying db artefacts. It would be interesting for me to know if the steps proposed by me above are in the right direction or not ? Also I am aware that I am late in discussing my proposal with you. But I can assure that I will be able to deliver the requirements in timely manner under your guidance. I have formulated initial draft of my proposal and would like to submit it at the earliest. Looking forward to you reply. Have a nice day! Thanks and Regards, Kesha Shah

2 2

[Maria-developers] new histogram code in 10.0.10
by Axel Schwenke 02 Apr '14

02 Apr '14

Hi Sergei, attached the results from a DBT3/SF30 run vs. 10.0.9 (as released) and 10.0-bzr rev 4128. Both runs use EITS + histograms. I see different EXPLAIN plans for queries 2, 3, 5, 7, 8, 18 and 21. Execution times only for few queries: 10.0.9 rev4128 Q2 44,44,46 44,45,45 Q3 298,298,299 236,243,243 Q5 264,302,303 293,308,308 Q7 199,200,201 204,205,206 Q8 375,375,377 264,274,276 Q18 217,218,221 178,180,182 Q21 251,259,261 258,261,262 Details in attached zip file. XL

1 0

[Maria-developers] PIC build options and MariaDB
by Otto Kekäläinen 02 Apr '14

02 Apr '14

Hello, I noticed that the MySQL packages in Debian have PIC stuff in their alternative build arguments, see http://anonscm.debian.org/gitweb/?p=pkg-mysql/mysql-5.6.git;a=blob;f=debian… http://anonscm.debian.org/gitweb/?p=pkg-mysql/mysql-5.5.git;a=blob;f=debian… Anybody here who understands deeply the build arguments who could say if having these PIC additions is desirable? Also the MySQL rules file has a section like this: # See http://blogs.innodb.com/wp/2010/04/innodb-performance-aio-linux/ 67 # Also #659565 68 ifneq (,$(findstring linux-gnu,$(DEB_HOST_GNU_TYPE))) 69 USE_LINUX_NATIVE_AIO:=-DLINUX_NATIVE_AIO=ON 70 endif Should MariaDB use this too, is it valid for XtraDB? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 1

[Maria-developers] Cassandra plugin build error: _S_empty_rep_storage' can not be used when making a shared object; recompile with -fPIC
by Otto Kekäläinen 02 Apr '14

02 Apr '14

Hello! I am testing out building MariaDB 10.0.10 on a Ubuntu 12.04 system with libthrift-dev installed (latest 0.9.0 from repo 'deb http://ppa.launchpad.net/wnoronha/thrift/ubuntu precise main'). Building fails with this error: make[4]: Siirrytään hakemistoon "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" Scanning dependencies of target cassandra make[4]: Poistutaan hakemistosta "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" make[4]: Siirrytään hakemistoon "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/ha_cassandra.cc.o [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/cassandra_se.cc.o [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/Cassandra.cpp.o [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/cassandra_types.cpp.o [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/cassandra_constants.cpp.o Linking CXX shared module ha_cassandra.so /usr/bin/ld.bfd.real: /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libthrift.a(TApplicationException.o): relocation R_X86_64_32S against `std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_empty_rep_storage' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libthrift.a: could not read symbols: Bad value collect2: ld:n paluuarvo oli 1 make[4]: *** [storage/cassandra/ha_cassandra.so] Virhe 1 Ideas? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

3 3

Re: [Maria-developers] Cassandra plugin build error: _S_empty_rep_storage' can not be used when making a shared object; recompile with -fPIC
by Otto Kekäläinen 01 Apr '14

01 Apr '14

Hello! Thanks for a quick reply! So I should add -fPIC to cmake arguments? Why isn't it in MariaDB sources by default, what are the downsides? GCC manual said "This option makes a difference on the m68k, PowerPC and SPARC." -> I am running amd64 For background, in my Debian packaging the build command looks like this: sh -c 'PATH=${MYSQL_BUILD_PATH:-"/bin:/usr/bin"} \ CC=${MYSQL_BUILD_CC:-gcc} \ CFLAGS=${MYSQL_BUILD_CFLAGS:-"-O2 -DBIG_JOINS=1 -fno-strict-aliasing "} \ CXX=${MYSQL_BUILD_CXX:-g++} \ CXXFLAGS=${MYSQL_BUILD_CXXFLAGS:-"-O3 -DBIG_JOINS=1 -felide-constructors -fno-exceptions -fno-rtti -fno-strict-aliasing "} \ cmake -DCMAKE_INSTALL_PREFIX=/usr \ \ -DCOMPILATION_COMMENT="(Ubuntu)" \ -DMYSQL_SERVER_SUFFIX="-1" \ -DSYSTEM_TYPE="debian-linux-gnu" \ -DWITHOUT_TOKUDB=true \ -DBUILD_CONFIG=mysql_release \ -DINSTALL_LIBDIR=lib/x86_64-linux-gnu \ -DINSTALL_PLUGINDIR=lib/mysql/plugin \ -DINSTALL_SUPPORTFILESDIR=share/mysql \ -DDEB=1 ..' In upstream MariaDB the build command is: sh -c 'PATH=$${MYSQL_BUILD_PATH:-"/bin:/usr/bin:/usr/local/bin"} \ CC=$${MYSQL_BUILD_CC:-gcc} \ CFLAGS=$${MYSQL_BUILD_CFLAGS:-"-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized $$(case `lsb_release -sc` in (lenny) echo -DWORKAROUND_GCC_4_3_2_BUG ;; esac)"} \ CXX=$${MYSQL_BUILD_CXX:-g++} \ CXXFLAGS=$${MYSQL_BUILD_CXXFLAGS:-"-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized"} \ cmake .. \ -DMYSQL_SERVER_SUFFIX="-$(DEBVERSION)" \ -DBUILD_CONFIG=mysql_release \ -DCOMPILATION_COMMENT="mariadb.org binary distribution" \ -DSYSTEM_TYPE="debian-linux-gnu" \ -DDEB=debian' 2014-04-01 16:21 GMT+03:00 Peter W Bowey <support(a)pbcomp.com.au>: > You require -fPIC to compile. See the info you reported = "> object; > recompile with -fPIC" > > Info is here (via GCC manual): > > -fPIC > If supported for the target machine, emit position-independent code, > suitable for dynamic linking and avoiding any limit on the size of the > global offset table. This option makes a difference on the m68k, PowerPC and > SPARC. > Position-independent code requires special support, and therefore works only > on certain machines. > > When this flag is set, the macros __pic__ and __PIC__ are defined to 2. > > In brief, the term position independent code (PIC) refers to the generated > machine code which is memory address agnostic, i.e. does not make any > assumptions about where it was loaded into RAM. Only position independent > code is supposed to be included into shared objects (SO) as they should have > an ability to dynamically change their location in RAM. > > > ----- Original Message ----- From: "Otto Kekäläinen" <otto(a)seravo.fi> > To: <maria-developers(a)lists.launchpad.net> > Cc: "Roger Meier" <roger(a)bufferoverflow.ch>; "Jake Farrell" > <jfarrell(a)apache.org> > Sent: Tuesday, April 01, 2014 11:16 PM > Subject: [Maria-developers] Cassandra plugin build error: > _S_empty_rep_storage' can not be used when making a shared object; recompile > with -fPIC > > >> Hello! >> >> I am testing out building MariaDB 10.0.10 on a Ubuntu 12.04 system >> with libthrift-dev installed (latest 0.9.0 from repo 'deb >> http://ppa.launchpad.net/wnoronha/thrift/ubuntu precise main'). >> >> Building fails with this error: >> >> make[4]: Siirrytään hakemistoon >> "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" >> Scanning dependencies of target cassandra >> make[4]: Poistutaan hakemistosta >> "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" >> make[4]: Siirrytään hakemistoon >> "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" >> [ 75%] Building CXX object >> storage/cassandra/CMakeFiles/cassandra.dir/ha_cassandra.cc.o >> [ 75%] Building CXX object >> storage/cassandra/CMakeFiles/cassandra.dir/cassandra_se.cc.o >> [ 75%] Building CXX object >> storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/Cassandra.cpp.o >> [ 75%] Building CXX object >> storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/cassandra_types.cpp.o >> [ 75%] Building CXX object >> >> storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/cassandra_constants.cpp.o >> Linking CXX shared module ha_cassandra.so >> /usr/bin/ld.bfd.real: >> >> /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libthrift.a(TApplicationException.o): >> relocation R_X86_64_32S against `std::basic_string<char, >> std::char_traits<char>, std::allocator<char> >>> >>> ::_Rep::_S_empty_rep_storage' can not be used when making a shared >> >> object; recompile with -fPIC >> /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libthrift.a: could >> not read symbols: Bad value >> collect2: ld:n paluuarvo oli 1 >> make[4]: *** [storage/cassandra/ha_cassandra.so] Virhe 1 >> >> >> Ideas? >> >> >> -- >> Check out our blog at http://seravo.fi/blog >> and follow @ottokekalainen >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~maria-developers >> Post to : maria-developers(a)lists.launchpad.net >> Unsubscribe : https://launchpad.net/~maria-developers >> More help : https://help.launchpad.net/ListHelp >> > > > --- > This email is free from viruses and malware because avast! Antivirus > protection is active. > http://www.avast.com > -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

1 0

[Maria-developers] DBT-3 benchmarks: MariaDB 10 vs MySQL 5.6
by Colin Charles 31 Mar '14

31 Mar '14

Hi XL, Nice work on posting new benchmarks in replication. Are you planning on running DBT-3 benchmarks with 10.0.10 and latest 5.6? Maybe even 5.7 DMR? Thanks -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

1 0

[Maria-developers] maria-developers@lists.launchpad.net
by Sergey Vojtovich 31 Mar '14

31 Mar '14

Hi! This is second list of questions I came up while merging test cases: - --ignore-builtin-innodb is deprecated in 5.6, do we want to deprecate it too? - LOW_PRIORITY WRITE is deprecate in 5.6, do we want to deprecate it too? The following items seem to be not merged to MariaDB. Do we want to merge them? WL#5331 Support Unicode for Windows command line client Bug#12688860 : SECURITY RECOMMENDATION: PASSWORDS ON CLI WL#5855 (EXPLAIN FORMAT=JSON) partition test WL#5980 - Allow remote tablespaces (need to check) WL5602 Providing password hashing with appropriate strength Bug#12316645: Wrong cost calculation with optimizer_join_cache_level settings WL#5602 - SHA256 password authentication Bugs#12635232: VALGRIND WARNINGS: IS_IPV6, IS_IPV4, INET6_ATON, INET6_NTOA + MULTIBYTE CHARSET Bug#14040277 UNINITIALIZED VALUE REFERENCED IN STR_TO_IPV6 And a lot of other IPV6 related stuff. Regards, Sergey

2 2

[Maria-developers] Intermediate status for test cases merge
by Sergey Vojtovich 31 Mar '14

31 Mar '14

Hi! So far I processed 431 test cases from the main suite. Most of them are up to date, a few merged without problems, a few tests fail (bugs reported), a few need extra analysis. There are a few items which I'd like to discuss: - INSERT DELAYED is deprecated in 5.6, do we want to deprecate it too? - SHOW PROFILE is deprecated in 5.6, do we want to deprecate it too? - YEAR(2) is deprecated in 5.6, do we want to deprecate it too? Monty suggests that we shouldn't deprecate it. I find it Ok too. Reasons for YEAR(2) deprecation are not obvious, relevant worklog is private. Relevant revision comment says: "YEAR(2) is a subject to deprecation since it has ill design." - EXPLAIN gives NULL in Extra column in 5.6 (empty string in 10.0), do we want behave similarly? - Do we want to merge fix for BUG#60269 - mysql should reject attempts to create system tables in incorrect engine http://bugs.mysql.com/bug.php?id=60269 - NO_ENGINE_SUBSTITUTION is default mode in 5.6 now, do we want it be default too? - 5.6 has many updates to default values, do we like to merge them too? Thanks, Sergey

2 6

Re: [Maria-developers] [Commits] Rev 3995: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 31 Mar '14

31 Mar '14

Hi, Holyfoot! On Dec 09, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3995 > revision-id: holyfoot(a)askmonty.org-20131209020845-9o3l30ao90vzf600 > parent: holyfoot(a)askmonty.org-20131208143921-3veatgg9wngyzb47 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Mon 2013-12-09 06:08:45 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > If a prepared statement calls an stored procedure, > the thd->server_status out of the SP goes up > to the PS and then to the client. So that the > client gets the SERVER_STATUS_CURSOR_EXISTS status > if the SP uses a cursor. Which makes the embedded > server fail. Why does embedded fail while a normal client-server protocol is ok with that? > Fixed by saving/restoring the upper-level server_status > in sp_head::execute(). Regards, Sergei

2 3

[Maria-developers] Engine independent statistics - naming issues?
by Sergey Petrunia 31 Mar '14

31 Mar '14

Hello, I was looking at whether column names used by EIS feature are consistent with themselves and with ther parts of the server. == Database / table / column names == There are a few naming conventions arready in use: 1. Old school: mysql.columns_priv, tables_priv Db Table_name Column_name 2. SQL STANDARD ALL CAPS HELLO FROM 1970S: information_schema.* (tables, columns, indexes, index_statistics, etc) TABLE_CATALOG (always 'def') TABLE_SCHEMA TABLE_NAME COLUMN_NAME INDEX_NAME 3. InnoDB has its own convention: mysql.innodb_table_stats, innodb_index_stats database_name table_name index_name 4. EITS feature will add: mysql.column_stats, index_stats, table_stats db_name table_name column_name index_name == Other == There are variables @@histogram_size, @@histogram_type. mysql.column_stats has columns named hist_size, hist_type. What's your opinion, should we make some renames to make things more consistent? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] [Commits] Rev 3991: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 31 Mar '14

31 Mar '14

Hi, Alexey! On Mar 26, Alexey Botchkov wrote: > Hi, Sergei. > > The Protocol::net_store_data is virtual in the embedded-server case > because we need it implemented differently for Protocol_binary. Ok, got it (finally), sorry. I'm almost always use ctags when looking for definitions, but in this case ctags didn't handle this correctly: #ifndef EMBEDDED_LIBRARY bool Protocol::net_store_data(const uchar *from, size_t length) #else bool Protocol_binary::net_store_data(const uchar *from, size_t length) #endif { ... so I never noticed that Protocol_binary actually has its own implementation of ::net_store_data() in the embedded case. Still, another question, why did you have to rename net_store_data() to net_store_data_cs() ? I've simply made it virtual and everything compiled just fine - it has a different signature so the compiler can distinguish them all right. Regards, Sergei

1 0

[Maria-developers] MariaDB 10.0.10 release prep
by Daniel Bartholomew 28 Mar '14

28 Mar '14

All, To get a jump on things, I've created the initial changelog page and a placeholder release notes page for the upcoming MariaDB 10.0.10 release. - https://mariadb.com/kb/en/mariadb-10010-changelog/ - https://mariadb.com/kb/en/mariadb-10010-release-notes/ Please let me know of things that need to be in the release notes for this release as soon as you can (or just add them yourself). The changelog page will also likely need to be updated as final pushes are made. As of right now it is up to revision 4135. FYI: I will be unresponsive on IRC or email until around mid-afternoon on Saturday due to some family events, but I will be around in the afternoon/evening on Saturday and Sunday to finish the various necessary tasks in preparation for the release. Thanks! -- Daniel Bartholomew MariaDB

1 0

[Maria-developers] A joint patch for MDEV-5969 and MDEV-5971
by Alexander Barkov 28 Mar '14

28 Mar '14

Hello, please review a joint patch fixing a crashing bug MDEV-5969, together with a non-crashing bug MDEV-5971. Thanks.

1 0

Re: [Maria-developers] parallel replication performance
by Kristian Nielsen 27 Mar '14

27 Mar '14

Axel Schwenke <axel(a)askmonty.org> writes: > The poor performance of MySQL with 1 database is expected. However MySQL is > much better with 32 databases. Even better than MariaDB :( Ok, so I looked closer at the numbers (of which there were quite a few, thanks for doing this work :-). I have some idea of what could be going on. The short story is that for this kind of configuration, it will be necessary to configure @@slave_domain_parallel_threads to be at most the value of @@slave_parallel_threads divided by the number of domain ids used. Eg. with 32 databases/domains, --slave-parallel-threads=64 --slave-domain-parallel-threads=2. Otherwise the domains will starve each other for threads, leading to poor parallelism. In fact, with such high domain parallelism @ 32 databases, I think just using --slave-domain-parallel-threads=1 could give much better performance. The longer story: MariaDB parallel replication uses a single pool of worker threads, of size @@slave_parallel_threads. GTIDs are read serially from the relay log in the SQL driver thread, and scheduled to worker threads. Each replication domain is scheduled independently from all other domains. Each GTID is allocated to a new worker thread if possible. If no worker thread is available (because all workers already have at least one GTID queued for them), then the SQL driver thread waits for one to become free; this stalls _all_ domains from getting more work queued. When your benchmark starts, the SQL driver thread will quickly read 64 events from the relay logs and queue them each to a separate worker thread. Then no more free worker threads are available, and the SQL thread will have to wait. With 64 worker threads and 32 domains, it is quite likely that some domains will not have any work to do. At the same time, the group commit size will most likely be much less than 64, so many workers will have to wait for the previous GTID in their domain without being able to do any work, especially at the "fast" configuration where commit (which can always run in parallel) does not take much time. The result will be a lot of threads waiting on each other and waking each other up, and little real work done. By setting @@slave_domain_parallel_threads, it is possible to limit how many worker threads a single domain is allowed to grab, thereby ensuring that worker threads will be available for all domains to be active doing work. Those limited number of threads will be used round-robin to queue work for; this also reduces thread sleep and wakeup, as with luck a worker thread that completes one commit will already have more work queued up for it at that point that it can process without first returning to the thread pool and going to sleep. In particular, with @@slave_domain_parallel_threads=1, we will just have 32 threads, one for each domain, each processing events independently in parallel at full speed, which is probably the best way to handle a workload as evenly distributed across domains as this one is. It would be very useful to see new benchmarks with those results. And if time permits, @@slave_domain_parallel_threads=2 could also be interesting; maybe there could be some extra performance due to more group commit in the "peer" configuration, or maybe the cost of repeatedly alternating between two worker threads within the domain will cause reduced performance. We can actually see some evidence of this theory in the numbers. Where number of worker threads is much larger than number of domains (so starvation is less of a problem), MariaDB does much better, even better than MySQL 5.6 in some cases. But performance drops as the number of domains increase, possibly because starvation kicks in. Still, even if this theory holds, we do see 5-10% slower performance of MariaDB at eg. slave@1 compared to MySQL. This suggests some higher overhead somewhere, which I need to look into at some point, though it will be a while yet before I have time for that, I am afraid. And in any case, this benchmark is highly useful as a test - as I said before, serious load testing of the domain_id parallelism has received very little testing, so it is quite possible that there is simply some bugs that cause things to work poorly. The extra testing with @@slave_domain_parallel_threads=1 (and possibly 2) will be very useful to help understand this better. > I think 32 databases is a bit too much, not only that MySQL then shines so > bright, it's also unrealistic in the real world. What do you think? I can > rerun the benchmark with 4 or 8 databases (will take 4-5 hours). Quite > certainly MySQL will have worse numbers then :) Well, I think 32 databases is not that unrealistic - shared hosting for example could have many more. The completely even spacing of the load among those databases however is probably unrealistic. But I think we should investigate and solve the issues for MariaDB @ 32 databases. More databases, and thus more parallelism, should be better for the parallel replication, if it is worse then that is a bug that should be fixed. Even if it is just the need to configure @@slave_domain_parallel_threads, maybe we could document that better. > I also notice that for MariaDB 10.0 the tuning of the commit-wait-* > variables is critical. Settings that give trx per commit group tend to slow > down the master a lot. The default settings give only few transactions per Right, this is something to be aware of. Note that for a typical web client application or similar, --commit-wait-* probably will not be much of an issue. An individual client will see slightly higher latency, which should not be much of an issue if it is in the order of a few milliseconds. Throughput is not likely to be affected, just the number of concurrent connections will increase (at least as long as system does not get choked). On the other hand, a load with fixed parallelism, like batch processing or your sysbench runs, can be hurt significantly on throughput by --commit-wait-*, as your tests show. Also, I have heard from several users that the ability to throttle a busy master in order to prevent it from running too far ahead of the slaves, is actually often seen as a benefit rather than a performance problem. Semi-sync replication also is sometimes used to achieve similar throttling. So yes, it is a tuning parameter that is unfortunately rather sensitive depending on the actual load and applications. > Please also check the "master" page of the comparison sheet. It shows the > cost of turning on the binlog. Both MySQL and MariaDB show significant > performance loss at high concurrency when the binlog is turned off. This is > completely counter-intuitive. Kristian, can this be related to MDEV-5802? It might be related. Another possibility is that this is related to InnoDB's known weaknesses at high concurrency. When the binlog is enabled, group commit will handle the commit of a lot of transactions serially, in a tight loop inside a single thread. When the binlog is disabled, all of those transactions will be committing each in their separate thread, all of those threads heavily contending each other for the hot mutexes inside InnoDB. It would need a lot of research to be sure that this is the cause, but it's a plausible guess, at least. Thanks for your efforts so far! Overall, I think the results look pretty good, actually. We see a good speedup from MariaDB parallel replication in many of the configurations. And your comparison shows that using group commit, a large percentage of the speedup can be obtained automatically from group commit, without any need to change applications and partition the data into separate databases. - Kristian.

2 2

Re: [Maria-developers] [Commits] Rev 3991: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 27 Mar '14

27 Mar '14

Hi, Holyfoot! On Nov 30, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3991 > revision-id: holyfoot(a)askmonty.org-20131130132432-etlmes8qhhqhr1iu > parent: elenst(a)wheezy-64.home-20131128160251-vn857wx5qlon6j1p > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Sat 2013-11-30 17:24:32 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > The function Protocol::net_store_data(a, b, CHARSET_A, CHARSET_B) should > be adapted to be working in the embedded server as it's done > with the Protocol::net_store_data(a, b). > That new function renamed as net_store_data_cs, so we can make it > virtual. Why does Protocol::net_store_data() have to be virtual? (if it doesn't, Protocol::net_store_data_cs doesn't have to be virtual either and doesn't have to be renamed) Regards, Sergei

2 1

[Maria-developers] Adding a storage engine - compilation
by Amith Nagarajappa 27 Mar '14

27 Mar '14

Hi, When i am creating a folder in storage/<customEngine>. I have placed the makefile from example folder and in the customEngine folder and have made required changes. However, makefile is not generated in the storage/<customEngine> folder. What change I have to make in configure or configure.in file? Right now, my custom engine code is similar to example code. I would like to build it first and then make the changes accordingly. But makefile is not generated. Appreciate your help. Thanks and Regards, Amith

3 2

Re: [Maria-developers] [Commits] Rev 3994: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 26 Mar '14

26 Mar '14

Hi, Alexey! On Mar 26, Alexey Botchkov wrote: > Hi, Sergei. > > > What do you mean by that? > > It can be seen in the sys_vars.completion_type_func test. oops. obscure feature. Okay, thanks. ok to push that. > > revno: 3994 > > message: > > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > > Thread can be disconnected internally for example after COMMIT statements. > > So we should check this for the statement execution. Regards, Sergei

1 0

[Maria-developers] Core MariaDB 10. Task suggestion.
by Сергей Дудоладов 26 Mar '14

26 Mar '14

2 1

Re: [Maria-developers] parallel replication performance
by Kristian Nielsen 25 Mar '14

25 Mar '14

Axel Schwenke <axel(a)askmonty.org> writes: > The poor performance of MySQL with 1 database is expected. However MySQL is > much better with 32 databases. Even better than MariaDB :( I haven't had time to go through all the numbers and details yet, but just a quick check: In this test, are you sure that you are running MariaDB in GTID mode? CHANGE MASTER TO master_use_gtid=slave_pos If not, you won't get any parallelism out of using 32 databases with different domain_ids. So two interesting, but different comparisons to make here: 1. MySQL multi-threaded slave compared to MariaDB GTID with different domain_id per database. If MySQL is significantly better here it could indicate some problem in MariaDB parallel replication that should be investigated; this is possible as this part has not been tested a lot. 2. MySQL multi-threaded slave with load split into multiple databases, compared to MariaDB without configuring different domains. Or just MariaDB compared without and with different domain ids. This is an "unfair" benchmark; by explicitly declaring that different databases/domains can run in parallel, more opportunity for parallellism is available. But it could still be interesting to see how much of that can be obtained from group commit parallelism, without changing applications. Both of these are interesting, but it was not clear to me which of them you were describing? - Kristian.

2 3

[Maria-developers] Patch for MDEV 5873
by Rohit Kalhans 25 Mar '14

25 Mar '14

Hello Kristian, All, As mentioned earlier i have a patch for MDEV 5873 and i am attaching here in the bundle of the patch. I have run tests on this patch and all of them have passed. Kindly review the patch. Also i am not sure which is the suitable mailing list for sending patches, so it will be great if you can point me to the same. Thanks and regards Rohit. -- Cheerio! *Rohit*

2 1

[Maria-developers] Two 10.0-related things.
by Alexey Botchkov 25 Mar '14

25 Mar '14

Hi, Sergei. I'd like to share my recently obtained knowledge :) 1. Mac OS X and sockets. As you know there on Labrador machine that mysql-test-run.pl failed if not run with --tmpdir=/tmp. The problem apparently is that the /Users directory has the access restrictions that /tmp doesn't. I still don't know what exactly was that, but it helps if we revoke all the ACLs with 'chmod -R -N /Users/mariadb'. 2. The 10.0 fails building on my machine, with linking errors in the extra/yassl/, about missing 'operator new[]' and similar. I fixed it locally with that patch: === modified file 'extra/yassl/CMakeLists.txt' --- extra/yassl/CMakeLists.txt 2014-02-26 14:28:07 +0000 +++ extra/yassl/CMakeLists.txt 2014-03-25 07:45:46 +0000 @@ -33,9 +33,10 @@ SET(YASSL_SOURCES src/buffer.cpp src/ce ADD_CONVENIENCE_LIBRARY(yassl ${YASSL_SOURCES}) RESTRICT_SYMBOL_EXPORTS(yassl) +TARGET_LINK_LIBRARIES(stdc++) + INSTALL_DEBUG_SYMBOLS(yassl) Best regards. HF

2 1

[Maria-developers] valgrind failures in ConnectSE
by Alexander Barkov 23 Mar '14

23 Mar '14

Hi Olivier, Valgrind complains about uninitialized memory read: http://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/5… It seems the problem was introduced by this change: http://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/5… Can you please fix this asap? We're going to make a release early next week. It would be nice to merge the latest ConnectSE. Thanks.

2 1

[Maria-developers] A fix for MDEV-5689 ExtractValue(xml, 'substring(/x, /y)') crashes
by Alexander Barkov 23 Mar '14

23 Mar '14

Hi Sergei, please review a fix for MDEV-5689. It also fixes MDEV-5709 ExtractValue() with XPath variable references returns wrong result. Description: 1. The main problem was that that nodeset_func->fix_fields() was called in Item_func_xml_extractvalue::val_str() and Item_func_xml_update::val_str(), which led in some cases to execution of the XPath engine *before* having a parsed XML value. Moved to Item_xml_str_func::fix_fields(). 2. Cleanup: added a new method Item_xml_str_func::fix_fields() and moved most of the code from Item_xml_str_func::fix_length_and_dec() to Item_xml_str_func::fix_fields(), to follow the usual Item layout. 3. Cleanup: a parsed XML value is useless without the raw XML value it was built from. Previously the parsed and the raw values where stored in separate String instances. It was hard to follow how they are synchronized. Added a helper class XML which contains both parsed and raw values. Makes things easier to read and modify. 4. MDEV-5709: const_item() could incorrectly return a "true" result when XPath expression contains users/SP variable references. Now nodeset_func->const_item() is also taken into account to catch such cases. 5. Minor code enhancements.

2 5

[Maria-developers] Transactions behind a failed transaction could commit in parallel replication
by nanyi607rao 21 Mar '14

21 Mar '14

hi kristian, In parallel replication if a transaction executed failed, replication should stop immediately(unconsider retry), and transactions behind the failed one should skip or rollback. you did try to do like this, there are codes like: if (unlikely(entry->stop_on_error_sub_id <= rgi->wait_commit_sub_id)) skip_event_group= true; this codes can tell latter transactions to skip but can't tell them rollback. because if a transaction started commiting before a former transaction failed (such as Lock timeout for unknown reason), the commiting transaction will not be affectd by stop_on_error_sub_id. Then the failed transaction should wakeup latter commiting transactions and tell them to rollback, unfortunately it won't. codes like if (!rgi->is_error && !skip_event_group) err= rpt_handle_event(events, rpt); else err= thd->wait_for_prior_commit(); ... ... finish_event_group(thd, err, event_gtid_sub_id, entry, rgi); if the failed transaction didn't fail at end event, err's value would come from wait_for_prior_commit, the err would be 0 if its former transaction has successed, then the failed transaction would tell latter transactions ok to commit in finish_event_group. 2014-03-19 nanyi607rao

2 2

Re: [Maria-developers] MDEV-5914
by Kristian Nielsen 21 Mar '14

21 Mar '14

Jan Lindström <jplindst(a)mariadb.org> writes: > Here is necessary changes. I have run whole mysql-test-run and I did not see > any failures, thus is is mostly safe. I say mostly, > because we really do not have much test cases that would use multimaster setup > and try to get conflicting transactions > generated from local clients against slave thread. Thanks a lot! I will try to come up with some test cases for this, with replication and external transactions that conflict and show the problem. In parallel, I tried running the parallel replication slaves with READ COMMITTED; that also seems to work and can get Elena able to continue testing. But this change is potentially more correct; I will let you know when I have some test results. - Kristian. > ================================= > > jan@jan-GE70-0NC-0ND ~/mysql/10.0-mdev5914 $ bzr diff > === modified file 'include/mysql/plugin.h' > --- include/mysql/plugin.h 2014-02-26 14:28:07 +0000 > +++ include/mysql/plugin.h 2014-03-21 09:05:58 +0000 > @@ -617,6 +617,7 @@ > int thd_in_lock_tables(const MYSQL_THD thd); > int thd_tablespace_op(const MYSQL_THD thd); > long long thd_test_options(const MYSQL_THD thd, long long test_options); > +unsigned long long thd_group_commit_id(const MYSQL_THD thd); > int thd_sql_command(const MYSQL_THD thd); > void **thd_ha_data(const MYSQL_THD thd, const struct handlerton *hton); > void thd_storage_lock_wait(MYSQL_THD thd, long long value); > > === modified file 'include/mysql/plugin_audit.h.pp' > --- include/mysql/plugin_audit.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_audit.h.pp 2014-03-21 09:06:17 +0000 > @@ -298,6 +298,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'include/mysql/plugin_auth.h.pp' > --- include/mysql/plugin_auth.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_auth.h.pp 2014-03-21 09:06:35 +0000 > @@ -298,6 +298,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'include/mysql/plugin_ftparser.h.pp' > --- include/mysql/plugin_ftparser.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_ftparser.h.pp 2014-03-21 09:06:59 +0000 > @@ -251,6 +251,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'sql/sql_class.cc' > --- sql/sql_class.cc 2014-03-20 14:36:45 +0000 > +++ sql/sql_class.cc 2014-03-21 09:07:12 +0000 > @@ -675,6 +675,12 @@ > } > > extern "C" > +unsigned long long thd_group_commit_id(const THD *thd) > +{ > + return thd->rpl_group_commit_id; > +} > + > +extern "C" > int thd_sql_command(const THD *thd) > { > return (int) thd->lex->sql_command; > > === modified file 'storage/innobase/handler/ha_innodb.cc' > --- storage/innobase/handler/ha_innodb.cc 2014-02-26 18:36:33 +0000 > +++ storage/innobase/handler/ha_innodb.cc 2014-03-20 15:42:27 +0000 > @@ -2048,6 +2048,8 @@ > trx->check_unique_secondary = !thd_test_options( > thd, OPTION_RELAXED_UNIQUE_CHECKS); > > + trx->group_commit_id = thd_group_commit_id(thd); > + > DBUG_VOID_RETURN; > } > > > === modified file 'storage/innobase/include/trx0trx.h' > --- storage/innobase/include/trx0trx.h 2014-02-26 18:36:33 +0000 > +++ storage/innobase/include/trx0trx.h 2014-03-20 15:28:19 +0000 > @@ -840,6 +840,7 @@ > or the state last time became > TRX_STATE_ACTIVE */ > trx_id_t id; /*!< transaction id */ > + ib_uint64_t group_commit_id;/*!< Group commit id or 0 */ > XID xid; /*!< X/Open XA transaction > identification to identify a > transaction branch */ > > === modified file 'storage/innobase/lock/lock0lock.cc' > --- storage/innobase/lock/lock0lock.cc 2014-02-26 18:22:48 +0000 > +++ storage/innobase/lock/lock0lock.cc 2014-03-21 09:32:54 +0000 > @@ -1020,6 +1020,23 @@ > return(FALSE); > } > > + if (trx->group_commit_id != 0 && > + lock2->trx->group_commit_id != 0 && > + trx->group_commit_id == lock2->trx->group_commit_id && > + (type_mode & LOCK_GAP || lock_rec_get_gap(lock2))) { > + /* No lock request needs to wait GAP locks if > + transactions belong to the same group commit group. > + This is because we have already verified these > + transactions on master that they do not conflict > + on record locks but as they could be executed on > + different order on slave we should not wait for > + gap locks here. Note that parallel replication > + makes sure that transaction commits are executed > + in the same order in master and slave. */ > + > + return (FALSE); > + } > + > return(TRUE); > } > > > === modified file 'storage/xtradb/handler/ha_innodb.cc' > --- storage/xtradb/handler/ha_innodb.cc 2014-02-26 18:21:23 +0000 > +++ storage/xtradb/handler/ha_innodb.cc 2014-03-20 15:42:41 +0000 > @@ -2308,6 +2308,8 @@ > > trx->fake_changes = THDVAR(thd, fake_changes); > > + trx->group_commit_id = thd_group_commit_id(thd); > + > #ifdef EXTENDED_SLOWLOG > if (thd_log_slow_verbosity(thd) & (1ULL << SLOG_V_INNODB)) { > trx->take_stats = TRUE; > > === modified file 'storage/xtradb/include/trx0trx.h' > --- storage/xtradb/include/trx0trx.h 2014-02-26 18:21:23 +0000 > +++ storage/xtradb/include/trx0trx.h 2014-03-20 15:28:20 +0000 > @@ -861,6 +861,7 @@ > or the state last time became > TRX_STATE_ACTIVE */ > trx_id_t id; /*!< transaction id */ > + ib_uint64_t group_commit_id;/*!< Group commit id or 0 */ > XID xid; /*!< X/Open XA transaction > identification to identify a > transaction branch */ > > === modified file 'storage/xtradb/lock/lock0lock.cc' > --- storage/xtradb/lock/lock0lock.cc 2013-12-22 16:06:50 +0000 > +++ storage/xtradb/lock/lock0lock.cc 2014-03-21 09:32:46 +0000 > @@ -1021,6 +1021,23 @@ > return(FALSE); > } > > + if (trx->group_commit_id != 0 && > + lock2->trx->group_commit_id != 0 && > + trx->group_commit_id == lock2->trx->group_commit_id && > + (type_mode & LOCK_GAP || lock_rec_get_gap(lock2))) { > + /* No lock request needs to wait GAP locks if > + transactions belong to the same group commit group. > + This is because we have already verified these > + transactions on master that they do not conflict > + on record locks but as they could be executed on > + different order on slave we should not wait for > + gap locks here. Note that parallel replication > + makes sure that transaction commits are executed > + in the same order in master and slave. */ > + > + return (FALSE); > + } > + > return(TRUE); > }

1 0

[Maria-developers] [GSoC 2014] CREATE OR REPLACE, CREATE IF NOT EXISTS and DROP IF EXISTS
by sriram patil 21 Mar '14

21 Mar '14

Hi Sergei, I just went through WL#3129 (thanks to James Briggs for the link). It gave me a detailed insight on what exactly is expected. I have missed 2 object types in my proposal which are, USER and SERVER. I can edit my proposal to incorporate the changes in the deliverables timeline if you want me to. Thanks, Sriram

1 0

Re: [Maria-developers] [GSOC 2014] "CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS" Project
by Sergei Golubchik 21 Mar '14

21 Mar '14

Hi, Sriram! First: please don't forget to cc: the list maria-developers(a)lists.launchpad.net Thank you. On Mar 18, sriram patil wrote: > Hi Sergei, > > I was going through the source code and was able to figure out the > flow of query execution and major functions through which a query goes > before execution. > > The query execution majorly dwells in the "sql" directory of the > source code. So, I think I will have to make changes in the following > files throughout the project, > > - sql_parse.cc (do_command, dispatch_command, mysql_execute_command) Yes. And also sql_yacc.yy - to extend the parser. And the implementation of these commands, for example, sql_acl.cc for CREATE/DROP USER, sql_trigger.cc for CREATE/DROP TRIGGER, etc. > - sql_prepare.cc (Prepared_statement::execute_loop, > Prepared_statement::execute) No, hopefully, this will work automatically. > What I am not sure about is will I have to create a handler for some of > these commands? You are not creating new commands > And will lower level storage engine code changes required? I don't think so. > > > I am interested to work on "CREATE OR REPLACE, CREATE IF NOT EXISTS, and > > > DROP IF EXISTS" project as part of GSOC 2014. I want to apply my subject > > > knowledge at a bigger level which will also give me an idea about the > > > practical issues while implementing the features. > > > Regards, Sergei

3 9

[Maria-developers] [GSoC] MDEV-371- index on BLOB
by smit hinsu 21 Mar '14

21 Mar '14

Hi Sergei and Roberto, My name is Smit Hinsu and I am final year undergrad student. I have good knowledge about database internals and good experience with C++. I find feature of creating index on BLOB quickly using MD-5 interesting. I have understood basic requirements for this feature. I have following questions in my mind currently. I think, HASH index is not supported in MySQL's version of MyISAM storage engine. Has MariaDB removed this restriction? We should support index for BLOB for BTREE index also in which we store only hash value of the value in BTREE. Is this possible? Can you suggest me some startup task related to index or storage engine which can make me familiar with the related parts? I have started to look at code. From Smit Hinsu

2 2

[Maria-developers] Engine independent index statistics - infrastructure questions
by Sergey Petrunia 20 Mar '14

20 Mar '14

Hello Igor and others, I've been playing around with engine independent statistics, and got a few questions. == mysqldump == mysqldump dumps table' data, but besides that it can dump other related objects, e.g. it will dump triggers that are attached to the table. Should it dump table statistics values, also? This question is of interest for DBAs, but also for optimizer developers. If a user has a complaint about the optimizer, we frequently ask them to upload a dataset. How do we request table/column statistics? We can't request output of SELECT * ..., because histogram data is binary and will be corrupted. We should provide users a convinent way to pass us relevant part of the EITS tables. == dropping table statistics == How does one drop table statistics? are we satisfied with needing to run three statements DELETE FROM mysql.column_stats WHERE schema_name=$db AND table_name=$table_name; DELETE FROM mysql.index_stats WHERE table_name=$db AND table_name=$table_name; DELETE FROM mysql.column_stats WHERE table_name=$db AND table_name=$table_name; or we want a simpler SQL, like DROP STATISTICS FOR t1; == Permissions on statistic tables == What should be the permissions on statistics tables? I guess we should follow pattern from other system tables - a user is typically able to read the table but not write to it (so that he can't mess with other users' SPs/triggers/etc). However, if the user is not allowed to modify the table directly, we should provide enough ways to modify the table indirectly. For example, we should provide a way to drop statistics for one's tables (which brings us back to the previous item). We might also want to allow to make other modifications to statistics of tables that one has control of? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] [GSoC 2014]
by Ivan Nikulin 20 Mar '14

20 Mar '14

Hello, I would like to take part into one of your projects on base of GSoC 2014. I am an experienced C/C++ programmer with wide knowledge of algorithms and data structures and huge interest into this sphere. I have some basic knowledge of SQL language, but unfortunatelly I don't have any database development experience. Nevertheless, gaining the latter is my essential purpose. Can I count on a chance of taking part in one of your projects? Sorry for writing to you that late. P.S. I tried writing to you a bit earlier, but the mail wouldn't appear on mailing list. -- Best regards, Ivan Nikulin

2 1

Re: [Maria-developers] [Commits] Rev 4053: MDEV-5864 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.free_tables
by Sergei Golubchik 19 Mar '14

19 Mar '14

Hi, Sergey! On Mar 15, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 4053 > revision-id: svoj(a)mariadb.org-20140315185301-ce4r6br023co80dm > parent: sanja(a)montyprogram.com-20140314073116-nklb0jkjd4t1w40n > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Sat 2014-03-15 22:53:01 +0400 > message: > MDEV-5864 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.free_tables > > Let TABLE_SHARE::tdc.free_tables, TABLE_SHARE::tdc.all_tables, > TABLE_SHARE::tdc.flushed and corresponding invariants be protected by > per-share TABLE_SHARE::tdc.LOCK_table_share instead of global LOCK_open. This looks pretty good to me! A couple of comments, see below. > === modified file 'sql/sql_plist.h' > --- a/sql/sql_plist.h 2013-11-20 11:05:39 +0000 > +++ b/sql/sql_plist.h 2014-03-15 18:53:01 +0000 > @@ -142,6 +142,14 @@ class I_P_List : public C, public I > > return result; > } > + inline T *back() > + { > + T *result= m_first; > + if (result) > + while (*B::next_ptr(result)) > + result= *B::next_ptr(result); > + return result; > + } I think this is wrong. The correct way would be to use I_P_List<> with the I_P_List_fast_push_back policy, then you could use have get_last() method to get the last element in the list. May be as inline T *back() { return *I::get_last(); } > void swap(I_P_List<T, B, C> &rhs) > { > swap_variables(T *, m_first, rhs.m_first); > > === modified file 'sql/table_cache.cc' > --- a/sql/table_cache.cc 2014-03-06 12:19:12 +0000 > +++ b/sql/table_cache.cc 2014-03-15 18:53:01 +0000 > @@ -70,13 +70,6 @@ static int32 tc_count; /**< Number of TA > > > /** > - Protects TABLE_SHARE::tdc.free_tables, TABLE_SHARE::tdc.all_tables. > -*/ > - > -mysql_mutex_t LOCK_open; So, you've now removed LOCK_open completely? Cool! > - > - > -/** > Protects unused shares list. > > TABLE_SHARE::tdc.prev > @@ -360,12 +378,15 @@ bool tc_release_table(TABLE *table) > table->in_use= 0; > /* Add table to the list of unused TABLE objects for this share. */ > table->s->tdc.free_tables.push_front(table); > - mysql_mutex_unlock(&LOCK_open); > + mysql_mutex_unlock(&table->s->tdc.LOCK_table_share); > return false; > > purge: > + /* Wait for MDL deadlock detector */ > + while (table->s->tdc.all_tables_refs) > + mysql_cond_wait(&table->s->tdc.COND_release, &table->s->tdc.LOCK_table_share); you're using it many times, I'd put it in a small static inline function, like static wait_for_mdl_deadlock_detector(TABLE *table) { mysql_mutex_assert_owner(&table->s->tdc.LOCK_table_share); while (table->s->tdc.all_tables_refs) mysql_cond_wait(&table->s->tdc.COND_release, &table->s->tdc.LOCK_table_share); } > tc_remove_table(table); > - mysql_mutex_unlock(&LOCK_open); > + mysql_mutex_unlock(&table->s->tdc.LOCK_table_share); > table->in_use= 0; > intern_close_table(table); > return true; Regards, Sergei

2 2

[Maria-developers] [GSOC 2014] GIS Enhancement
by bang honam 19 Mar '14

19 Mar '14

Hi everybody. I am Honam Bang, a researcher of Korean Institute of Science Technology and I will be a master student of UCSD, USA from this Sep. I have been working with GIS research for about one year. Therefore, I decided to work on the 'GIS Enhancement topic' because I have a lot of interest for dealing with geospatial data. My topic is 'implementing shp to sql command tool for MariaDB'. #MDEV-5813. Most shptosql tools are enable to convert 3D coordinates as I know. The shape file white paper( http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf) specifies a lot of 3D coordinates like POINT Z, MULTIPOINTZ which does not exist in MariaDB's Geometry type. Does my topic include implementing those type converting as well? Thank you Honam Bang

2 1

[Maria-developers] first results from replication benchmarks
by Axel Schwenke 18 Mar '14

18 Mar '14

Hi everybody, here are some first results from my benchmarks of replication / group commit performance. The benchmark is sysbench OLTP rw, data set fitting in memory. I tested a total of 27 configurations, 9 each for Maria-5.5.36, Maria-10.0.9 and MySQL-5.6.16. Variations were for binlog on/off, SBR vs. RBR and sync_binlog=0/1. For MariaDB I also tested XtraDB vs. InnoDB, for MySQL it was GTID on vs. GTID off. Not surprisingly turning on the binlog comes with a performance penalty. Also sync_binlog=1 adds another penalty. Quite consistently RBR was a little bit slower than SBR for all combinations (I also observed the binlogs to be bigger for RBR by a factor of nearly 2). The only real surprise was the cost of GTID in MySQL as can be seen in the attached plot. MariaDB-10.0 has GTID enabled by default (you cannot even turn it off) Right now the same benchmark is running again, this time using a single disk for binlogs (was a fast RAID before). Next step is applying binlogs to a slave with different settings for number of slave threads etc. XL

2 2

[Maria-developers] Gsoc proposal submitted for user defined event
by Savita Agrawal 18 Mar '14

18 Mar '14

Hi, I have submitted my proposal for user defined event support project. I am still working on the proposal. Please give feedback. Regards, Savita

1 0

[Maria-developers] Group commit id in mysqlbinlog output
by Kristian Nielsen 18 Mar '14

18 Mar '14

There was a question on the output of mysqlbinlog related to group commit. In 10.0, if two transactions group commit together on the master, their GTID event contains a "commit id" - a 64-bit number. This is used by parallel replication; if two transactions have the same commit id, they can be executed in parallel. In mysqlbinlog output, this looks like this: #140314 13:42:56 server id 1 end_log_pos 772 GTID 0-1-12 cid=180 ... #140314 13:42:56 server id 1 end_log_pos 1027 GTID 0-1-13 cid=180 But if a transaction commits alone on the master (no other transactions participate in the group commit), the mysqlbinlog outpus has no commit id: #140314 13:42:56 server id 1 end_log_pos 437 GTID 0-1-10 The question was why there is no cid=X in the second case, as it makes scripting/grepping the output harder. The reason is that there is no commit id in the event in the binlog in this case (to reduce the size of the binlog). So there is no valid number to put in there. One option could be to make the output something like this: #140314 13:42:56 server id 1 end_log_pos 437 GTID 0-1-10 cid=<none> This would allow to grep for "cid=" and catch everything. I do not have a strong opinion one way or the other, if there are people who find this useful, then let me know and I can change it. - Kristian.

4 6

Re: [Maria-developers] rbr triggers (all)
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, Oleksandr! Ok to push. See a couple of comments below. On Mar 18, Oleksandr Byelkin wrote: > === added file 'mysql-test/suite/rpl/t/rpl_row_triggers.test' > --- mysql-test/suite/rpl/t/rpl_row_triggers.test 1970-01-01 00:00:00 +0000 > +++ mysql-test/suite/rpl/t/rpl_row_triggers.test 2014-03-11 08:49:31 +0000 > @@ -0,0 +1,280 @@ > +-- source include/have_binlog_format_row.inc > +-- source include/have_rbr_triggers.inc > +-- source include/master-slave.inc > +-- source include/have_innodb.inc Note that master-slave.inc should always be the last in the series of includes. Everything that can fail should be before it, otherwise you can abort and skip the test after configuring replication, but without a corresponding replication cleanup. It will affect following tests. > === added file 'mysql-test/suite/rpl/t/rpl_row_triggers_sbr.test' > --- mysql-test/suite/rpl/t/rpl_row_triggers_sbr.test 1970-01-01 00:00:00 +0000 > +++ mysql-test/suite/rpl/t/rpl_row_triggers_sbr.test 2014-03-11 08:49:48 +0000 > @@ -0,0 +1,43 @@ > +--source include/master-slave.inc > +--source include/have_binlog_format_statement.inc > +--source include/have_rbr_triggers.inc Same here. > === modified file 'sql/log_event.cc' > --- sql/log_event.cc 2014-03-05 22:20:10 +0000 > +++ sql/log_event.cc 2014-03-11 18:39:51 +0000 > @@ -11121,6 +11233,13 @@ Rows_log_event::write_row(rpl_group_info > TABLE *table= m_table; // pointer to event's table > int error; > int UNINIT_VAR(keynum); > + > +#ifdef RBR_TRIGGERS > + bool invoke_triggers= > + slave_run_triggers_for_rbr && !master_had_triggers && table->triggers; > +#else > +#define invoke_triggers 0 > +#endif //RBR_TRIGGERS I think you don't need this ifdef, just declare invoke_triggers as const: const bool invoke_triggers= slave_run_triggers_for_rbr && !master_had_triggers && table->triggers; > auto_afree_ptr<char> key(NULL); > > prepare_record(table, m_width, Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3994: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, Holyfoot! On Dec 08, holyfoot(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 3994 > revision-id: holyfoot(a)askmonty.org-20131208143921-3veatgg9wngyzb47 > parent: holyfoot(a)askmonty.org-20131202104753-srqqm7hv1rtcurx4 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Sun 2013-12-08 18:39:21 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > Thread can be disconnected internally for example after COMMIT statements. What do you mean by that? > So we should check this for the statement execution. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3993: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, holyfoot! On Dec 02, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3993 > revision-id: holyfoot(a)askmonty.org-20131202104753-srqqm7hv1rtcurx4 > parent: holyfoot(a)askmonty.org-20131130142537-7j272euoaoy0svg1 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Mon 2013-12-02 14:47:53 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > mysqltest in the 'embedded-server' mode runs queries in a separate thread, > but it didn't do so for the statements - they were run in the main thread. > That leads to inconsistencies. > When a test sets SESSION 'dbug' variable like > SET SESSION debug_dbug="+d,warn_during_ha_commit_trans"; > it is run as a plain query in that separate thread, so the main thread remains unaffected. > After that the statement run in the main thread doesn't produce expected 'dbug' errors, > so the test fail. > To fix that I made statement to be run in that special thread along with the plain queries. > That makes the environment consistent. Ok to push. But sayng "statements" in the comment is a big misleading, it'd be cleared if you'd say "prepared statements". Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3992: MDEV-5138 Numerous test failures in "mtr --ps --embedded". in file:///home/hf/wmar/mdev-5138/
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, holyfoot! On Nov 30, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3992 > revision-id: holyfoot(a)askmonty.org-20131130142537-7j272euoaoy0svg1 > parent: holyfoot(a)askmonty.org-20131130132432-etlmes8qhhqhr1iu > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Sat 2013-11-30 18:25:37 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > As Davi added code like > sav_protocol= thd->protocol > thd->protocol= &thd->protocol_binary > ... > thd->protocol= sav_protocol > the fucntions like emb_store_querycache_result() cannot determine > the used protocol testing thd->protocol == &thd->protocol_binary. > Fixed by additional check thd->command == COM_STMT_EXECUTE. What about PREPARE? Regards, Sergei

1 0

Re: [Maria-developers] MaxScale as a binlog server
by Kristian Nielsen 18 Mar '14

18 Mar '14

Mark Riddoch <mark.riddoch(a)skysql.com> writes: > we have been approached by a user with a suggestion for a project that we > might do with them. They have a replication environment with a large fan out > (100 slaves per master) and are looking at doing something to cut one the > load on the master. the thought they had was to use MaxScale to effectively > cache the bin log files, so MaxScale would act as a single Slave to the > Master and would itself have a number of Slaves that read the bin log from > MaxScale. My initial response was why not just use an instance of MariaDB as > the intermediate node and connect slaves to one of those instances, in a > tree structure. They had a few reasons why they did not want to do this; > Looking in more detail I think it is feasible, we essentially write a router > module for MaxSale that acts as a slave, aching the bin log locally on the > MaxScale node. We then have another router, or maybe even the same one, that > relays that binlog to the real slaves. We are looking at doing something for > MariaDB 10, since we would want to utilise the GTID to get the best failover > semantics, I am assuming there must be some differences between the > replication stream that I can find documented for MySQL and what MariaDB 10 > does? Is this stuff anywhere on the Knowledge Base? I tried looking but Right, so I agree with what Serg said, that maybe going back to the original problem and seeing if there are better solutions possible would be better. However, I will try to answer the technical aspect here. I suppose caching and routing binlogs outside of the server should be possible, but it might be more complex than you would think at first. I am not aware of any comprehensive documentation for this. However, it should be relatively easy to see from the code. Basically, it is the function mysql_binlog_send() in sql/sql_repl.cc that handles sending binlog data to a slave. There are good comments in there describing many of the trickier points. You need to be aware that there is more logic in sending binlog data to a slave than just streaming raw binlog files. Maybe you can implement a subset and document any limitations. The old-style replication (not using GTID) is the simplest, but it still does a bit extra, like sending some extra events (FORMAT_DESCRIPTION_EVENT and fake ROTATE_EVENT). There is also the handling of old slaves. A slave sends a value in @mariadb_slave_capability, and the master will rewrite or remove any events that the slave does not understand, as appropriate. And there is the @@skip_replication flag; when the slave sets this, the master removes events that were logged with @@skip_replication set (to reduce bandwidth needs). The slave also does several SQL queries to obtain additional information. For example, the slave does a SELECT binlog_gtid_pos(file, offset) to obtain the GTID position associated with a given old-format replication position, so that it knows how to switch to GTID mode later. Maybe those queries could be handled by relaying them back to a real master server. The GTID mode is significantly more complex, due to a GTID position having multiple streams, one per replication domain. The slave sends @slave_connect_state, @slave_gtid_strict_mode, @slave_gtid_ignore_duplicates, and optionally @slave_until_gtid values. The router would need to parse these values and act accordingly. The binlog sender needs to keep track of each domain as the binlog files are scanned. Events in a domain must be skipped until the point specified in @slave_connect_state is reached. Then events are sent until the point in @slave_until_gtid is reached, at which point further events must again be skipped. The master also sends extra GTID_LIST events, which contain the state of the binlog at the point of the event, which is needed by the slave to correctly handle START SLAVE UNTIL master_gtid_pos=XXX and MASTER_GTID_WAIT(). A lot of effort was put into error handling, to support all reasonable, and even many unreasonable uses of replication, while still giving an error in cases where things are obviously wrong (to prevent silently doing the wrong thing or hanging). For example, if the slave tries to connect at a GTID that the master does not have, this is normally an error. This is important, because that GTID might be some transaction that was executed manually on the slave by mistake; if it only exists on the slave, then the master would endlessly skip events looking for the GTID that never shows up, and replication would silently hang. However, there are special cases where the slave _is_ allowed to connect at a GTID that does not exist on the master. For example, one might have a slave with --log-slave-updates=0. Then if this slave is promoted as a new master, the binlogs will not have the current position, but it is still allowed to connect to it at the point where it became the new master (but not to a point earlier than that). Another special case is if the requested GTID (but no GTIDs following that) was purged from the binary logs, as can easily happen if a replication domain is unused for long time. A _full_ implementation of a router for sending binlog to slaves would be some effort, but it might be reasonable to implement some subset. I suppose the first step would be for you to go through the code in mysql_binlog_send() and the functions it calls, and decide on which parts to implement. I will be happy to answer any questions that might pop up on the way, of course. > 2. They wanted to maintain the same group commit groups as the original > master, so as to benefit from the parallel replication. Note that this may no longer apply to MariaDB 10.0.9. The issue is that an intermediate slave that is itself a master may have less group commit than the original master, which reduces the possibility for a third-level slave to efficiently do parallel replication. But in 10.0.9, it should be possible to configure the intermediate slave with --binlog-commit-wait-* to get better group commit. > 3. They think that can get better failover semantics because they will have > n (maybe 5 or 10) MaxScales and they will all have the same bin log, so > failover between them will be easier. I am curious how it will be assured that they all have the same binlog in case of crash? But maybe you just mean that all will be a subset of the largest one. Hope this helps, - Kristian.

2 1

[Maria-developers] Java Connector Discussion
by Puneet Dewan 18 Mar '14

18 Mar '14

Hi Georg, As we missed our scheduled meeting yesterday, can you suggest a suitable time to discuss? Regards Puneet.

1 0

Re: [Maria-developers] [Maria-discuss] MariaDB vs. MySQL error codes (1900+)
by Kristian Nielsen 17 Mar '14

17 Mar '14

Felipe Gasper <felipe(a)felipegasper.com> writes: > Hi everyone, > > I notice that MariaDB is using the 1900+ range for its own > error codes; however, MySQL 5.7, at least, has a few errors in that > range as well. > > Are error codes just going to be an area of increasing > conflict between the two, or is there a plan to migrate MariaDB codes > to a different numbering scheme? Thanks for bringing this to attention. This is a pretty serious issue, and I've forwarded it to Tomas Ulin, who should be in a position to get it fixed. Tomas, please note this issue. MariaDB has used error codes from 1900 from the very start to avoid these kinds of conflicts. They can not be changed, as the codes are already in GA releases and applications may be relying on the numbers. You will need to ensure that MySQL 5.7 avoids conflicts with these before 5.7 goes GA, or there will be a conflict that will hurt all users and will be very hard to fix later. Felipe, you should probably also file a bug on bugs.mysql.com to try to get this fixed before MySQL 5.7 GA. Thanks, - Kristian.

2 1

[Maria-developers] redundant compilation of targets in mariadb 10.0
by Rohit Kalhans 17 Mar '14

17 Mar '14

hello all, I am trying to fix a few bugs but ever after small modifications in specific files, make command build all targets irrespective of whether or not they are dependent on the modified source. I am no expert in build commands but this seem off since it keeps taking exceptionally long time to build. Is there a trick/workaround that can prevent all targets from being built. Or am I missing something. -- Cheerio! *Rohit* *http://rohitkalhans.com <http://rohitkalhans.com>*

3 6

[Maria-developers] For Google Summer of Code 2014, Interested in the task of "statistically optimize mysql-test runs by running less tests"
by 胡仲义 17 Mar '14

17 Mar '14

Dear Sergei Golubchik, I am a post graduate student of Institute of Software, Chinese Academy of Sciences and my name is Zhongyi Hu. I major in computer science and my research field is data stream mining. Because I have got enough papers and works for graduation, I want to do something interesting, meaningful and valuable in the rest time as student. I have participated in two projects about database, one is main memory database and the other is database cluster. I got some experience of database system design and implementaion from them. Although I am just a beginner of this area, I really like it and expect to make it as my career. I often use Mysql in research and work, but MariaDB is not very familiar to me. I am tremendously optimistic about it's future because all of you. Well, let's come to the point. I am interested in the task of "statistically optimize mysql-test runs by running less tests". I chose this task because I have written a few tools for automatic test. I know the performance is very important if there are a large amount of data or cases to test. I read the MDEV-5776 and I think the major job is as follow. When the code is changed, the mysql-test is used to do the requisite tests. We need to integrate the information of the changes and the scenarios to predict the probability of failure for each test and get the relationships of the tests. Then decide what to test and what test cases should be used. The purpose is to optimize the efficiency of testing. All of these should be done by algorithm and program. In addition, I think that the job is in some ways like mining in data stream, such as many data need to be statistical analyzed and the hidden patterns changing over time. At last, I have two basic questions. 1) What exactly are the builder and the combination? I thought they refer to compiler and runtime environment. 2) What does the "individual tests within a big test file" mean? Maybe I am completely wrong, but I still look forward to your reply. I hope to have the opportunity to learn from you in work and discussion. Best regards Zhongyi Hu 20140314

3 5

[Maria-developers] [GSOC 2014] "CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS" Project
by sriram patil 16 Mar '14

16 Mar '14

Hi Sergei, I am Sriram Patil. I am a Masters' student at International Institute of Information Technology, Hyderabad, India. I have been working on a small DB Engine as part of my course work in C++. I have also worked on a SQLite DB abstraction library FMDB in Objective-C. Also, I am currently working on lock-free concurrent B-trees (Java), and going to evaluate it with an in-memory database to check the performance. I have also worked on libdaap and Rhythembox source code during my undergraduate project. You can check my GitHub<https://github.com/srirampatil?tab=repositories>and BitBucket <https://bitbucket.org/srirampatil> accounts which have some of my projects. I am interested to work on "CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS" project as part of GSOC 2014. I want to apply my subject knowledge at a bigger level which will also give me an idea about the practical issues while implementing the features. I have gone through the project description and understood that a generic interface needs to be implemented for "CREATE OR REPLACE" and "CREATE IF NOT EXISTS" for functions, procedures, tables, triggers, etc. But, I understand I am a pretty late to discuss these things with you. Is it okay if I submit my proposal document directly? Also, do you recommend me to go through some specific things with respect to this project before I start working on my proposal draft? Thanks, Sriram Patil IIIT Hyderabad, India

2 1

[Maria-developers] MDEV-5681- audit log not rotating
by Tan Tran 14 Mar '14

14 Mar '14

Hi maria-developers, This is my patch for MDEV-5681: " audit log will not rotate when the file size exceeds global variable setting": http://bazaar.launchpad.net/~tktran/maria/maria-fix-bug5681/revision/4102 To test, 1. set server_audit_events = 'connect' and keep default rotate size of 1000000. 2. connect with a bad password about 20 times: for i in {1..20}; do mysql -ufail -pfail; done 3. see that server_audit.log is X bytes 4. set server_audit_rotate_size = 1000 5. connect with bad password for another 20 times 6. see original server_audit.log is still X bytes (meaning log rotated immediately) and subsequent server_audit.log.* logs are ~1000 bytes or less Please review. Thanks, Tan Tran Prospective GSoC student

2 1

Re: [Maria-developers] [Commits] Rev 4066: MDEV-5089 - possible deadlocks between rwlocks and mutexes
by Sergei Golubchik 13 Mar '14

13 Mar '14

Hi, Sergey! On Feb 13, Sergey Vojtovich wrote: > revno: 4066 > revision-id: svoj(a)mariadb.org-20140213074049-70zbqa98y95sfpox > parent: sanja(a)askmonty.org-20140212150705-e1tfabmhjggqws7q > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5 > timestamp: Thu 2014-02-13 11:40:49 +0400 > message: > MDEV-5089 - possible deadlocks between rwlocks and mutexes > > Pre-MDL versions had direct relationship between LOCK_open and > LOCK_global_system_variables, e.g.: > intern_sys_var_ptr // locks LOCK_global_system_variable > mysql_sys_var_char > create_options_are_valid > ha_innobase::create > handler::ha_create > ha_create_table > rea_create_table > mysql_create_table_no_lock // locks LOCK_open > mysql_create_table > > With MDL this relationship was removed, but mutex order was still > recorded. In fact there is indirect relationship between LOCK_open > and LOCK_global_system_variables via rwlocks in reverse order. > > Removed LOCK_open and LOCK_global_system_variables order recording, > instead assert that LOCK_open is never held in intern_sys_var_ptr(). > > This solves only one of many problems detected with MDEV-5089. Looks good, thanks! ok to push! Regards, Sergei

1 0

[Maria-developers] TokuDB Performance Regression (MDEV-5777)
by Tim Callaghan 13 Mar '14

13 Mar '14

I'm very concerned about the regression I reported in https://mariadb.atlassian.net/browse/MDEV-5777, as users of the MariaDB.org supplied builds of both 5.5 and 10.0 are getting substantially less throughput than the MariaDB 5.5 build supplied by us at tokutek.com. I see that the fix version is now 5.5.37 and 10.0.10, is there a plan for how that will get done? I'll assist if possible, I just don't know how I can. -Tim Callaghan (Tokutek)

2 1

[Maria-developers] Fwd: Database/kernel community topic at Collaboration Summit 2014
by Hartmut Holzgraefe 13 Mar '14

13 Mar '14

Forwarding this as the original message doesn't seem to have made it to this list ... -------- Original Message -------- Subject: Database/kernel community topic at Collaboration Summit 2014 Date: Mon, 10 Mar 2014 10:15:37 +0000 From: Mel Gorman <mgorman(a)suse.de> To: pgsql-hackers(a)postgresql.org, mysql(a)lists.mysql.com, maria-developers(a)lists.launchpad.net, linux-kernel(a)vger.kernel.org Hi, Arrangements have been made to hold a meeting between database and kernel developers at Collaboration Summit 2014 http://sched.co/1hEBRuq on March 27th 2014. This was organised after discussions on pain points encountered by the PostgreSQL community. Originally the plan had been to just have a topic for LSF/MM there was much more interest in the topic than anticipated so the Collaboration Summit meeting will be much more open. If there are developers attending Collaboration Summit that work in the database or kernel communities, it would be great if you could come along. Previous discussions were on the PostgreSQL list and that should be expanded in case we accidentally build postgres-only features. The intent is to identify the problems encountered by databases and where relevant, test cases that can be used to demonstrate them if they exist. While the kernel community may be aware of some of the problems, they are not always widely known or understood. There is a belief that some interfaces are fine when in reality applications cannot use them properly. The ideal outcome of the meeting would be concrete proposals on kernel features that could be developed over the course of time to address any identified problem. For reference, this is a summary of the discussion that took place when the topic was proposed for LSF/MM. Thanks. ---8<--- On testing of modern kernels ---------------------------- Josh Berkus claims that most people are using Postgres with 2.6.19 and consequently there may be poor awareness of recent kernel developments. This is a disturbingly large window of opportunity for problems to have been introduced. Minimally, Postgres has concerns about IO-related stalls which may or may not exist in current kernels. There were indications that large writes starve reads. There have been variants of this style of bug in the past but it's unclear what the exact shape of this problem is and if IO-less dirty throttling affected it. It is possible that Postgres was burned in the past by data being written back from reclaim context in low memory situations. That would have looked like massive stalls with drops in IO throughput but it was fixed in relatively recent kernels. Any data on historical tests would be helpful. Alternatively, a pgbench-based reproduction test could potentially be used by people in the kernel community that track performance over time and have access to a suitable testing rig. It was mentioned that Postgres has an tool called pg_test_fsync which was mentioned in the context of testing different wal_sync_methods. Potentially it could also be used for evaluating some kernel patches. Gregory Smith highlighted the existence of a benchmark wrapper for pgbench called pgbench-tools: https://github.com/gregs1104/pgbench-tools . It can track statistics of interest to Postgres as well as report in interesting metrics such as transaction latency. He had a lot of information on testing requirements and some very interesting tuning information and it's worth reading the whole mail http://www.postgresql.org/message-id/52D99161.60305@gmail.com Postgres bug reports and LKML ----------------------------- It is claimed that LKML does not welcome bug reports but it's less clear what the basis of this claim is. Is it because the reports are ignored? A possible explanation is that they are simply getting lost in the LKML noise and there would be better luck if the bug report was cc'd to a specific subsystem list. A second possibility is the bug report is against an old kernel and unless it is reproduced on a recent kernel the bug report will be ignored. Finally it is possible that there is not enough data available to debug the problem. The worst explanation is that to date the problem has not been fixable but the details of this have been lost and are now unknown. Is is possible that some of these bug reports can be refreshed so at least there is a chance they get addressed? Apparently there were changes to the reclaim algorithms that crippled performance without any sysctls. The problem may be compounded by the introduction of adaptive replacement cache in the shape of the thrash detection patches currently being reviewed. Postgres investigated the use of ARC in the past and ultimately abandoned it. Details are in the archives (http://www.Postgres.org/search/?m=1&q=arc&l=1&d=-1&s=r) I have not read then, just noting they exist for future reference. Sysctls to control VM behaviour are not popular as such tuning parameters are often used as an excuse to not properly fix the problem. Would it be possible to describe a test case that shows 2.6.19 performing well and a modern kernel failing? That would give the VM people a concrete basis to work from to either fix the problem or identify exactly what sysctls are required to make this work. I am confident that any bug related to VM reclaim in this area has been lost. At least, I recall no instances of it being discussed on linux-mm and it has not featured on LSF/MM during the last years. IO Scheduling ------------- Kevin Grittner has stated that it is known that the DEADLINE and NOOP schedulers perform better than any alternatives for most database loads. It would be desirable to quantify this for some test case and see can the default scheduler cope in some way. The deadline scheduler makes sense to a large extent though. Postgres is sensitive to large latencies due to IO write spikes. It is at least plausible that deadline would give more deterministic behaviour for parallel reads in the presence of large writes assuming there were not ordering problems between the reads/writes and the underlying filesystem. For reference, these IO spikes can be massive. If the shared buffer is completely dirtied in a short space of time then it could be 20-25% of RAM being dirtied and writeback required in typical configurations. There have been cases where it was worked around by limiting the size of the shared buffer to a small enough size so that it can be written back quickly. There are other tuning options available such as altering when dirty background writing starts within the kernel but that will not help if the dirtying happens in a very short space of time. Dave Chinner described the considerations as follows There's no absolute rule here, but the threshold for background writeback needs to consider the amount of dirty data being generated, the rate at which it can be retired and the checkpoint period the application is configured with. i.e. it needs to be slow enough to not cause serious read IO perturbations, but still fast enough that it avoids peaks at synchronisation points. And most importantly, it needs to be fast enought that it can complete writeback of all the dirty data in a checkpoint before the next checkpoint is triggered. In general, I find that threshold to be somewhere around 2-5s worth of data writeback - enough to keep a good amount of write combining and the IO pipeline full as work is done, but no more. e.g. if your workload results in writeback rates of 500MB/s, then I'd be setting the dirty limit somewhere around 1-2GB as an initial guess. It's basically a simple trade off buffering space for writeback latency. Some applications perform well with increased buffering space (e.g. 10-20s of writeback) while others perform better with extremely low writeback latency (e.g. 0.5-1s). Some of this may have been addressed in recent changes with IO-less dirty throttling. When considering stalls related to excessive IO it will be important to check if the kernel was later than 3.2 and what the underlying filesystem was. Again, it really should be possible to demonstrate this with a test case, one driven by pgbench maybe? Workload would generate a bunch of test data, dirty a large percentage of it and try to sync. Metrics would be measuring average read-only query latency when reading in parallel to the write, average latencies from the underlying storage, IO queue lengths etc and comparing default IO scheduler with deadline or noop. NUMA Optimisations ------------------ The primary one that showed up was zone_reclaim_mode. Enabling that parameter is a disaster for many workloads and apparently Postgres is one. It might be time to revisit leaving that thing disabled by default and explicitly requiring that NUMA-aware workloads that are correctly partitioned enable it. Otherwise NUMA considerations are not that much of a concern right now. Bruce Momjian highlighted this block entry that covered zone_reclaim http://frosty-postgres.blogspot.com/2012/08/postgresql-numa-and-zone-reclai… Direct IO, buffered IO, double buffering and wishlists ------------------------------------------------------ The general position of Postgres is that the kernel knows more about storage geometries and IO scheduling that an application can or should know. It would be preferred to have interfaces that allow Postgres to give hints to the kernel about how and when data should be written back. The alternative is exposing details of the underlying storage to userspace so Postgres can implement a full IO scheduler using direct IO. It has been asserted on the kernel side that the optimal IO size and alignment is the most important detail should be all the details that are required in the majority of cases. While some database vendors have this option, the Postgres community do not have the resources to implement something of this magnitude. They also have tried direct IO in the past in the areas where it should have mattered and had mixed results. I can understand Postgres preference for using the kernel to handle these details for them. They are a cross-platform application and the kernel should not be washing its hands of the problem and hiding behind direct IO as a solution. Ted Ts'o summarises the issues as The high order bit is what's the right thing to do when database programmers come to kernel engineers saying, we want to do <FOO> and the performance sucks. Do we say, "Use O_DIRECT, dummy", not withstanding Linus's past comments on the issue? Or do we have some general design principles that we tell database engineers that they should do for better performance, and then all developers for all of the file systems can then try to optimize for a set of new API's, or recommended ways of using the existing API's? In an effort to avoid depending on direct IO there were some proposals and/or wishlist items. These are listed in order of likliehood to be implemented and usefulness to Postgres. 1. Hint to asynchronously queue writeback now in preparation for a fsync in the near future. Postgres dirties a large amount of data and asks the kernel to push it to disk over the next few minutes. Postgres still is required to fsync later but the fsync time should be minimised. vm.dirty_writeback_centisecs is unreliable for this. One possibility would be an fadvise call that queues the data for writeback by a flusher thread now and returns immediately 2. Hint that a page is a prime candidate for reclaim but only if there is reclaim pressure. This avoids a problem where fadvise(DONTNEED) discards a page only to have a read/write or WILLNEED hint immediately read it back in again. The requirements are similar to the volatile range hinting but they do not use mmap() currently and would need a file-descriptor based interface. Robert Hass had some concerns with the general concept and described them thusly This is an interesting idea but it stinks of impracticality. Essentially when the last buffer pin on a page is dropped we'd have to mark it as discardable, and then the next person wanting to pin it would have to check whether it's still there. But the system call overhead of calling vrange() every time the last pin on a page was dropped would probably hose us. Well, I guess it could be done lazily: make periodic sweeps through shared_buffers, looking for pages that haven't been touched in a while, and vrange() them. That's quite a bit of new mechanism, but in theory it could work out to a win. vrange() would have to scale well to millions of separate ranges, though. Will it? And a lot depends on whether the kernel makes the right decision about whether to chunk data from our vrange() vs. any other page it could have reclaimed. 3. Hint that a page should be dropped immediately when IO completes. There is already something like this buried in the kernel internals and sometimes called "immediate reclaim" which comes into play when pages are bgin invalidated. It should just be a case of investigating if that is visible to userspace, if not why not and do it in a semi-sensible fashion. 4. 8kB atomic write with OS support to avoid writing full page images in the WAL. This is a feature that is likely to be delivered anyway and one that Postgres is interested in. 5. Only writeback some pages if explicitly synced or dirty limits are violated. Jeff Janes states that he has problems with large temporary files that generate IO spikes when the data starts hitting the platter even though the data does not need to be preserved. Jim Nasby agreed and commented that he "also frequently see this, and it has an even larger impact if pgsql_tmp is on the same filesystem as WAL. Which *theoretically* shouldn't matter with a BBU controller, except that when the kernel suddenly +decides your *temporary* data needs to hit the media you're screwed." One proposal that may address this is Allow a process with an open fd to hint that pages managed by this inode will have dirty-sticky pages. Pages will be ignored by dirty background writing unless there is an fsync call or dirty page limits are hit. The hint is cleared when no process has the file open. 6. Only writeback pages if explicitly synced. Postgres has strict write ordering requirements. In the words of Tom Lane -- "As things currently stand, we dirty the page in our internal buffers, and we don't write it to the kernel until we've written and fsync'd the WAL data that needs to get to disk first". mmap() would avoid double buffering but it has no control about the write ordering which is a show-stopper. As Andres Freund described; Postgres' durability works by guaranteeing that our journal entries (called WAL := Write Ahead Log) are written & synced to disk before the corresponding entries of tables and indexes reach the disk. That also allows to group together many random-writes into a few contiguous writes fdatasync()ed at once. Only during a checkpointing phase the big bulk of the data is then (slowly, in the background) synced to disk. I don't see how that's doable with holding all pages in mmap()ed buffers. There are also concerns there would be an absurd number of mappings. The problem with this sort of dirty pinning interface is that it can deadlock the kernel if all dirty pages in the system cannot be written back by the kernel. James Bottomley stated No, I'm sorry, that's never going to be possible. No user space application has all the facts. If we give you an interface to force unconditional holding of dirty pages in core you'll livelock the system eventually because you made a wrong decision to hold too many dirty pages. However, it was very clearly stated that the writing ordering is critical. If the kernel breaks the requirement then the database can get trashed in the event of a power failure. This led to a discussion on write barriers which the kernel uses internally but there are scaling concerns both with the number of constraints that would exist and the requirement that Postgres use mapped buffers. There were few solid conclusions on this. It would need major reworking on all sides and it would handing control of system safety to userspace which is going to cause layering violations. This whole idea may be a bust but it is still worth recording. Greg Stark outlined the motivation best as follows; Ted T'so was concerned this would all be a massive layering violation and I have to admit that's a huge risk. It would take some clever API engineering to come with a clean set of primitives to express the kind of ordering guarantees we need without being too tied to Postgres's specific implementation. The reason I think it's more interesting though is that Postgres's journalling and checkpointing architecture is pretty bog-standard CS stuff and there are hundreds or thousands of pieces of software out there that do pretty much the same work and trying to do it efficiently with fsync or O_DIRECT is like working with both hands tied to your feet. 7. Allow userspace process to insert data into the kernel page cache without marking the page dirty. This would allow the application to request that the OS use the application copy of data as page cache if it does not have a copy already. The difficulty here is that the application has no way of knowing if something else has altered the underlying file in the meantime via something like direct IO. Granted, such activity has probably corrupted the database already but initial reactions are that this is not a safe interface and there are coherency concerns. Dave Chinner asked "why, exactly, do you even need the kernel page cache here?" when Postgres already knows how and when data should be written back to disk. The answer boiled down to "To let kernel do the job that it is good at, namely managing the write-back of dirty buffers to disk and to manage (possible) read-ahead pages". Postgres has some ordering requirements but it does not want to be responsible for all cache replacement and IO scheduling. Hannu Krosing summarised it best as Again, as said above the linux file system is doing fine. What we want is a few ways to interact with it to let it do even better when working with Postgres by telling it some stuff it otherwise would have to second guess and by sometimes giving it back some cache pages which were copied away for potential modifying but ended up clean in the end. And let the linux kernel decide if and how long to keep these pages in its cache using its superior knowledge of disk subsystem and about what else is going on in the system in general. 8. Allow copy-on-write of page-cache pages to anonymous. This would limit the double ram usage to some extent. It's not as simple as having a MAP_PRIVATE mapping of a file-backed page because presumably they want this data in a shared buffer shared between Postgres processes. The implementation details of something like this are hairy because it's mmap()-like but not mmap() as it does not have the same writeback semantics due to the write ordering requirements Postgres has for database integrity. Completely nuts and this was not mentioned on the list, but arguably you could try implementing something like this as a character device that allows MAP_SHARED with ioctls with ioctls controlling that file and offset backs pages within the mapping. A new mapping would be forced resident and read-only. A write would COW the page. It's a crazy way of doing something like this but avoids a lot of overhead. Even considering the stupid solution might make the general solution a bit more obvious. For reference, Tom Lane comprehensively described the problems with mmap at http://www.Postgres.org/message-id/17515.1389715715@sss.pgh.pa.us There were some variants of how something like this could be achieved but no finalised proposal at the time of writing. 9. Hint that a page in an anonymous buffer is a copy of a page cache page and invalidate the page cache page on COW. This limits the amount of double buffering. It's in as a low priority item as it's unclear if it's really necessary and also I suspect the implementation would be very heavy because of the amount of information we'd have to track in the kernel. It is important to note in general that Postgres has a problem with some files being written back too aggressively and other files not written back aggressively enough. Temp files for purposes such as sorting should have writeback deferred as long as possible. Data file writes that must complete before portions of the WAL can be discarded should begin writeback early so the final fsync does not stall for too long. As Dave Chinner says IOWs, there are two very different IO and caching requirements in play here and tuning the kernel for one actively degrades the performance of the other. Robert Hass categorised the IO patterns as follows - WAL files are written (and sometimes read) sequentially and fsync'd very frequently and it's always good to write the data out to disk as soon as possible - Temp files are written and read sequentially and never fsync'd. They should only be written to disk when memory pressure demands it (but are a good candidate when that situation comes up) - Data files are read and written randomly. They are fsync'd at checkpoint time; between checkpoints, it's best not to write them sooner than necessary, but when the checkpoint arrives, they all need to get out to the disk without bringing the system to a standstill No matter it was pointed out that fsync should never be able to screw the system. Robert Hass again summaried it as follows IMHO, the problem is simpler than that: no single process should be allowed to completely screw over every other process on the system. When the checkpointer process starts calling fsync(), the system begins writing out the data that needs to be fsync()'d so aggressively that service times for I/O requests from other process go through the roof. It's difficult for me to imagine that any application on any I/O scheduler is ever happy with that behavior. We shouldn't need to sprinkle of fsync() calls with special magic juju sauce that says "hey, when you do this, could you try to avoid causing the rest of the system to COMPLETELY GRIND TO A HALT?". That should be the *default* behavior, if not the *only* behavior. It is important to keep this in mind although sometimes the ordering requirements of the filesystem may make it impossible to achieve. At LSF/MM last year there was a discussion on whether userspace should hint that files are "hot" or "cold" so the underlying layers could decide to relocate some data to faster storage. I tuned out a bit during the discussion and did not track what happened with it since but I guess that any developments of that sort would be of interest to the Postgres community. Some of these wish lists still need polish but could potentially be discussed further at LSF/MM with a wider audience as well as on the lists. Then in a of unicorns and ponies it's a case of picking some of these hinting wishlists, seeing what it takes to implement it in kernel and testing it with a suitably patched version of postgres running a test case driven by something (pgbench presumably). -- Mel Gorman SUSE Labs -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/mysql

1 0

[Maria-developers] GSOC Proposal-Java Connector For MariaDB
by Puneet Dewan 12 Mar '14

12 Mar '14

Hi All, Please find attached my initial proposal for Java Connector for Maria DB. Awaiting for your comments.:) Regards Puneet.

1 0

[Maria-developers] InnoDB selected projects (corrected links)
by Jan Lindström 12 Mar '14

12 Mar '14

Hi all, After careful weighting and selection process, I have selected following two projects as a starting point to improve InnoDB (1) InnoDB file space defragmentation Description: External tool to physically delete delete marked rows from InnoDB file space and freeing unused file space. https://mariadb.atlassian.net/browse/MDEV-5834 (2) InnoDB fast bulk load Description: External tool to load CVS file directly to InnoDB file space (innodb_file_per_table = 1). https://mariadb.atlassian.net/browse/MDEV-5835 Now, I hope these projects mostly represent the most frequently requested features and are most usable to current customers and/or potential new customers. Next, I would hope some indication which one of the selected project you would like to see first. Votes can be added above links. R: -- -- Jan Lindström, Principal Engineer SkySQL - The MariaDB Company MariaDB | MaxScale | skype: jan_p_lindstrom www.skysql.com <http://www.skysql.com/> Twitter <http://twitter.com/skysql> Blog <http://www.skysql.com/blog/> Facebook <http://www.facebook.com/skysql> LinkedIn <http://www.linkedin.com/company/1214250> Google+ <https://plus.google.com/117544963211695643458/posts>

1 0

Re: [Maria-developers] [Commits] Rev 3768: MDEV-5717: Server crash with insert statement containing DEFAULT into view
by Sergei Golubchik 12 Mar '14

12 Mar '14

Hi, Sanja! On Mar 12, sanja(a)montyprogram.com wrote: > revno: 3768 > revision-id: sanja(a)montyprogram.com-20140312103416-3v6tan06crsqry28 > parent: sanja(a)montyprogram.com-20140307115707-poaet8iwv9d55azm > committer: sanja(a)montyprogram.com > branch nick: work-maria-5.3-MDEV-5717 > timestamp: Wed 2014-03-12 12:34:16 +0200 > message: > MDEV-5717: Server crash with insert statement containing DEFAULT into view > > Item_default_value::arg can be NULL so walk() should take it into consideration. > === modified file 'sql/item.h' > --- a/sql/item.h 2014-02-19 13:45:33 +0000 > +++ b/sql/item.h 2014-03-12 10:34:16 +0000 > @@ -3551,7 +3551,7 @@ public: > > bool walk(Item_processor processor, bool walk_subquery, uchar *args) > { > - return arg->walk(processor, walk_subquery, args) || > + return (arg && arg->walk(processor, walk_subquery, args)) || > (this->*processor)(args); > } Ok to push. But *please* install the diff_p bzr plugin as described here: https://mariadb.com/kb/en/how-to-get-more-out-of-bzr-when-working-on-mariad… Regards, Sergei

1 0

Re: [Maria-developers] Maria DB project aspirant for GSoC 2014 : CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS
by Sergei 11 Mar '14

11 Mar '14

Hi, Siddharth! First: please send questions like this to maria-developers(a)lists.launchpad.net it's a public mailing list dedicated to MySQL and MariaDB internals, source code, and related things. I am subscribed, so I'll see you mail there, and you may be sure I will, because it won't be accidentally catched by my spam filter, or sorted out in some obscure folder. Furthermore other subscribers will see your question and could reply if I will be not available (e.g. I could be travelling). Thank you. On Mar 11, Siddharth Dalmia wrote: > Hi Sergei, > > I am interested in working on the project. I am new to the open source > world so, can you tell how i should go forward? Do I need to work on > something and show? No, this is not required, but if you do it'll be a bonus point for you - you'll get familiar with MariaDB source code and would be able to write a better proposal. Everything you have to do and how you should move forward is written at the GSoC site. See http://www.google-melange.com/gsoc/homepage/google/gsoc2014 Regards, Sergei

1 0

[Maria-developers] GSOC 14: Discussion on "GIS enhancements to MariaDB"
by Abhishek Kumar 11 Mar '14

11 Mar '14

Hi, I am Abhishek and am interested in knowing about the gsoc project "GIS enhancements to MariaDB". I participate often in the programming contents hosted at these sites: Codeforces: abhi170893 (Division 1) TopCoder: abhi170893 ( Divsion 1) Codechef: abhi_iiit_17 ( Ranked 6 in India in DEC13 . 3 times in top 20 in country.) Spoj: abhi170893 (over 100 problems) The ideas page mentions C as the required skill and I am good at coding in C. Also on the ideas page not much is written about the project. Where can i get more details about it..? what things should i explore first before trying for this project..? the complexity level..? the competition involved..?Do i need to fix bugs first..? If yes, their links..? Thanks Abhishek Kumar CSE, IIIT Hyderabad (India)

2 1

Re: [Maria-developers] Rev 4033: MDEV-5766 - my_atomic_load does memory writes
by Sergei Golubchik 10 Mar '14

10 Mar '14

Hi, Sergey! On Mar 07, Sergey Vojtovich wrote: > revno: 4033 > revision-id: svoj(a)mariadb.org-20140307074306-i3l4fllu2rj5pouf > parent: svoj(a)mariadb.org-20140306121912-i1p8iilpsgc8m4bn > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Fri 2014-03-07 11:43:06 +0400 > message: > MDEV-5766 - my_atomic_load does memory writes > > my_atomic_load() is implemented as __sync_fetch_and_or(var, 0) which > writes or-ed value back to var. Memory writes as such have worse > performance and scalability than reads. > > gcc 4.7 and up offers better facility for atomic loads/stores. Use it > whenever it is available. Thanks. ok to push Regards, Sergei

1 0

[Maria-developers] Some questions on GSOC project User Defined Event
by Savita Agrawal 10 Mar '14

10 Mar '14

Hi, what is the difference between a trigger and user defined event? In postgres there is a clear distintion between trigger and event trigger. Triggers are activated when DML statements are executed whereas event triggers are activated when DDL statements are executed. But in soliddb (from which our post in JIRA page is inspired), it seems like user defined events can be activated for DML statements. What is it that can't be done with triggers and can be done with user defined event? Why is the restriction that events can only be posted and received by stored procedures only? Cheers, Savita

3 6

[Maria-developers] Project in GSOC 2014.
by Puneet Dewan 10 Mar '14

10 Mar '14

Hi All, I,Puneet Dewan, am participating in Google Summer of Code 2014. I am interested in contributing to Java Client Library for Maria DB. I have started investigating in Maria DB Java connector and have found one potential feature that can be added to MariaDB-Java connector. The feature is: * RowSetFactoryInterface and RowSetProviderClass as per JDBC Spec 4.1 which is not part of latest Java connector version JDBC 1.1.6* Benefit: *Any types of row sets that are supported by the JDBC Driver can be created.* *One Last Question:* I have seen the source code of Java Connector and I understood they are made in Java 6. Are there any plans to migrate the code from Java 6 to Java 7? Awaiting for the response. Regards Puneet

2 5

[Maria-developers] [Gsoc] LDAP authentication plug-in project
by Sudheera Palihakkara 10 Mar '14

10 Mar '14

Hi, I'm started to work above project. I want to know that if I have to use a local ldap server to test the feature like OpenLdapServer or are there any remote servers that I can connect to? thanks. -- *Sudheera Palihakkara.* Undergraduate Department of *Computer Science and Engineering,*Faculty of Engineering, *University of Moratuwa*, Sri Lanka.

3 5

Re: [Maria-developers] Maria DB project aspirant for GSoC 2014 : CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS
by Sergei 10 Mar '14

10 Mar '14

Hi, Siddharth! On Mar 08, Siddharth Dalmia wrote: > Hello Sergei and all, > > I am Siddharth Dalmia a second year undergraduate pursuing Computer > Science Engineering from Birla Institute Technology and Science (BITS) > Pilani, Hyderabad Campus. > > I have a keen interest in programming and learning algorithms. I have a > deep knowledge in C , C++ and sql. I have even done various projects in > C++ including graphics. I have an average knowledge of data structures, > java, Linux, html, css. I am also currently working on a project on > concurrency in databases. I have a thorough knowledge in relational > algebra and i have been using sql since 3 years (oracle and mysql). > > I have gone through various projects of your's for GSOC and the one > which interest me a most is CREATE OR REPLACE, CREATE IF NOT EXISTS, and > DROP IF EXISTS . I am very keen to be a part of this project as it is > something which i think its something which i am learning/learnt to put > to use. > > I know that i am a bit late, but it is not an end and i believe that i > can work hard with dedication to make a good and convincing proposal. It > will be very nice of you if you guide me on how to go about this project > and from where i could start and elaborate what i need to do. There's not much to add to the specifications. There is CREATE TABLE statement, CREATE TABLE IF NOT EXISTS statement, CREATE OR REPLACE TABLE statement. There is CREATE USER statetement, but no CREATE USER IF NOT EXISTS and no CREATE OR REPLACE USER. There is CREATE PROCEDURE, but neither CREATE PROCEDURE IF NOT EXISTS nor CREATE OR REPLACE PROCEDURE. And so on. We'd like to have a consistent syntax and support "IF NOT EXISTS" and "OR REPLACE" for all CREATE statements where possible. And the same for DROP IF EXISTS Regards, Sergei

1 0

[Maria-developers] Interest in the "LDAP Authentication plugin" project for GSoC14
by Marta Sequeira 09 Mar '14

09 Mar '14

Hi everyone! First of all I would like to introduce myself. My name is Marta Sequeira and I'm studying for my master's degree in Computer Science, in Instituto Superior Técnico, Portugal. I have skills in C from the academic projects I developed. I did a course in relational DBMS and I have managed a few MySQL databases. I also am currently maintaining some PHP code that uses a large LDAP directory (but not for authentication in this particular case), and I have worked with a Samba server that uses LDAP authentication, so I am somewhat familiar with LDAP. This is the reason why I'm very interested in the LDAP Authentication plugin project. I would like to apply for this project, so any guidelines on what I should include in my proposal are very welcome. I would also like to know if you consider this project idea to fullfil the workload goals of GSoC, or if I should apply for more projects from your ideas page. Since I am not yet familiar with the MariaDB's code, I have no notion of the amount of work it will require. Best regards, -- Marta Sequeira

1 0

[Maria-developers] Review to fix for mdev-5437
by Savita Agrawal 08 Mar '14

08 Mar '14

Hi, I am a student from India interested in participating gsoc. This is my first fix to mariadb. Please review it. Link is: http://bazaar.launchpad.net/~savitarit0/maria/maria-fix-mdev-5437/revision/…

2 1

[Maria-developers] [GSoC] proposal draft
by Tan Tran 08 Mar '14

08 Mar '14

Hi Colin Charles, maria-developers (cc'd), I'm looking to do my GSoC project on porting the InnoDB-memcached plugin to MariaDB, which you will be the mentor for. I asked maria-developers about it earlier, but haven't had any responses, so I've written a draft proposal (below) to elicit some responses. I would appreciate your technical corrections and suggestions. Thanks, Tan Tran tankimtran(a)gmail.com Proposal The MySQL+memcached combination is popular for high-speed, high-throughput websites that also need durable, long-term storage. By storing data mostly on disk, MySQL provides the ACID compliance and longevity of a traditional RDMS; by caching commonly accessed data solely on memory, memcached provides fast lookup for most transactions. In general, memcached can be used in two different ways: first, where the program tries to retrieve a cached copy from a separate memcached process before resorting to accessing the database; and second, where memcached is integrated into MySQL, invisibly to the client program. This is possible because MySQL versions 5.6 and up come with a daemon plugin that connects the InnoDB storage engine to memcached, bypassing the SQL layer and replicating data behind the scenes. This plugin is not in MariaDB, as it forked from MySQL before 5.6. But it would be very useful if it was. The plugin has a couple advantages over the separate process; for example, the plugin can automatically serialize binary data to store it as text in memcached, making it much less error-prone than manual efforts. As a GSoC student, I would port the InnoDB-memcached plugin from MySQL 5.6/5.7 to MariaDB 10's InnoDB and XtraDB engines. The original code is GPL-licensed, so my job would be to use the existing code but modify it to match the variety of codebase changes that have happened since the MariaDB fork. For example, upon compiling the MySQL plugin code with MariaDB instead, I got a compile error about a missing "binlog.h" in the handler API code for the InnoDB half of the plugin. Apparently, the binary logging code is different for MySQL and MariaDB, and my job would be to convert those references over. Additionally, I will write an XtraDB version of the plugin that, for example, scales well with many memcached nodes. This project would provide an alternative to MDEV-4674, where the writer requests a feature to connect MariaDB to memcached using TCP/UDP, and to MDEV-4675, where the writer requests a feature to detect and address low memory in the memcached server; instead, for example, the plugin could take parameters on a strategy to allocate memcached memory. Deliverables: MariaDB InnoDB+memcached plugin XtraDB+memcached plugin Documentation similar to MySQL's "Getting Started" guides on using the plugin

1 0

[Maria-developers] [GSoC] Student interested in "Optimize mysql-test runs" or adding SQL features
by Pablo Estrada 08 Mar '14

08 Mar '14

Hello! I'm Pablo, from Mexico. Before going into detail, I wanna give a bit of background: I am a master's student, pursuing a degree on Scientific Computing in Korea. My program is 3 years long, and I'm at the middle of the first one. The reason why I think I have skills that will be useful for MariaDB is that before being a master's student, I was working for Oracle during 2 years, committing into the Oracle executable, so I have experience testing, coding and working inside a database executable. Specifically, I was working on the DBMS_SCHEDULER package. Although I am interested in software development, I am more interested in data analysis. That's why I switched from computer science to scientific computing, and that's why the project that interests me the most is the one called "Statistically optimize mysql-test runs by running less tests". I am very excited by the idea of using data to try to test more smartly. If it was not possible to participate in the latter project, I would also be interested to work on adding support for "OR REPLACE, CREATE IF NOT EXISTS and DROP IF EXISTS". Depending on your feedback, I can get to work on a proposal for either one of the two projects. I'd be very happy to be able to participate in GSoC with MariaDB. Regards Pablo

2 2

Re: [Maria-developers] Rev 4004: RBR triggers compiled-out with ifdefs in 10.0
by Sergei Golubchik 07 Mar '14

07 Mar '14

Hi, Sanja! On Feb 12, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-10.0-MDEV-5095/ > > ------------------------------------------------------------ > revno: 4004 > revision-id: sanja(a)askmonty.org-20140212204600-n89hns6zvrut1nl7 > parent: sanja(a)askmonty.org-20140212190858-38n5izb9m3pwz7wc > committer: sanja(a)askmonty.org > branch nick: work-maria-10.0-MDEV-5095 > timestamp: Wed 2014-02-12 22:46:00 +0200 > message: > RBR triggers compiled-out with ifdefs in 10.0 Here're my comments, see below: Note, that I didn't review the implementation of RBR triggers, I looked at ifdefs only. > === modified file 'mysql-test/suite/rpl/disabled.def' > --- mysql-test/suite/rpl/disabled.def 2013-09-27 12:58:49 +0000 > +++ mysql-test/suite/rpl/disabled.def 2014-02-28 10:56:24 +0000 > @@ -14,3 +14,5 @@ rpl_row_create_table : Bug#11759274 > rpl_spec_variables : BUG#11755836 2009-10-27 jasonh rpl_spec_variables fails on PB2 hpux > rpl_get_master_version_and_clock : Bug#11766137 Jan 05 2011 joro Valgrind warnings rpl_get_master_version_and_clock > rpl_partition_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table > +rpl_row_triggers : IFDEF-ed in 10.0 (RBR_TRIGGERS) > +rpl_row_triggers_sbr : IFDEF-ed in 10.0 (RBR_TRIGGERS) better than disabled.def would be to have a file, say --source include/have_rbr_triggers.inc that would do, like if (`select count(*) = 0 from information_schema.session_variables where variable_name = 'slave_run_triggers_for_rbr'`) { skip RBR triggers are not available; } > === modified file 'sql/log_event.cc' > --- sql/log_event.cc 2013-11-13 22:03:48 +0000 > +++ sql/log_event.cc 2014-02-28 10:56:24 +0000 > @@ -9282,10 +9284,31 @@ int Rows_log_event::do_add_row_data(ucha > } > #endif > > -#if !defined(MYSQL_CLIENT) && defined(HAVE_REPLICATION) > +#if defined(MYSQL_SERVER) && defined(HAVE_REPLICATION) why is this change? > + > + > +#ifdef RBR_TRIGGERS > +/** > + Restores empty table list as it was before trigger processing. > + > + @note We have a lot of ASSERTS that check the lists when we close tables. > + There was the same problem with MERGE MYISAM tables and so here we try to > + go the same way. > +*/ > +static void restore_empty_query_table_list(LEX *lex) > +{ > + if (lex->first_not_own_table()) > + (*lex->first_not_own_table()->prev_global)= NULL; > + lex->query_tables= NULL; > + lex->query_tables_last= &lex->query_tables; > +} > +#endif //RBR_TRIGGERS > + > + > int Rows_log_event::do_apply_event(rpl_group_info *rgi) > { > Relay_log_info const *rli= rgi->rli; > + TABLE* table; > DBUG_ENTER("Rows_log_event::do_apply_event(Relay_log_info*)"); > int error= 0; > /* > @@ -9632,9 +9691,15 @@ int Rows_log_event::do_apply_event(rpl_g > */ > thd->reset_current_stmt_binlog_format_row(); > thd->is_slave_error= 1; > - DBUG_RETURN(error); > + /* remove trigger's tables */ > + goto err; hmm, here you've added rgi->slave_close_thread_tables(thd); > } > > +#ifdef RBR_TRIGGERS > + /* remove trigger's tables */ > + if (slave_run_triggers_for_rbr) > + restore_empty_query_table_list(thd->lex); > +#endif //RBR_TRIGGERS > if (get_flags(STMT_END_F) && (error= rows_event_stmt_cleanup(rgi, thd))) > slave_rows_error_report(ERROR_LEVEL, > thd->is_error() ? 0 : error, > @@ -10132,6 +10206,9 @@ Table_map_log_event::Table_map_log_event > DBUG_ASSERT(static_cast<size_t>(cbuf_end - cbuf) <= sizeof(cbuf)); > m_data_size+= (cbuf_end - cbuf) + m_colcnt; // COLCNT and column types > > + if (tbl->triggers) > + m_flags|= TM_BIT_HAS_TRIGGERS_F; why didn't you ifdef this? > + > /* If malloc fails, caught in is_valid() */ > if ((m_memory= (uchar*) my_malloc(m_colcnt, MYF(MY_WME)))) > { > @@ -10538,7 +10616,11 @@ int Table_map_log_event::do_apply_event( > table_list->table_id= DBUG_EVALUATE_IF("inject_tblmap_same_id_maps_diff_table", 0, m_table_id); > table_list->updating= 1; > table_list->required_type= FRMTYPE_TABLE; > - DBUG_PRINT("debug", ("table: %s is mapped to %u", table_list->table_name, table_list->table_id)); > + table_list->master_had_triggers= ((m_flags & TM_BIT_HAS_TRIGGERS_F) ? 1 : 0); why didn't you ifdef this? > + DBUG_PRINT("debug", ("table: %s is mapped to %u%s", > + table_list->table_name, table_list->table_id, > + (table_list->master_had_triggers ? > + " (master had triggers)" : ""))); > enum_tbl_map_status tblmap_status= check_table_map(rgi, table_list); > if (tblmap_status == OK_TO_PROCESS) > { > @@ -10801,6 +10887,10 @@ Write_rows_log_event::do_before_row_oper > from the start. > */ > } > +#ifdef RBR_TRIGGERS > + if (slave_run_triggers_for_rbr && !master_had_triggers && m_table->triggers) > + m_table->prepare_triggers_for_insert_stmt_or_event(); > +#endif //RBR_TRIGGERS But you don't need to ifdef this and many blocks in Rows_log_event::do_apply_event. If you only do #define slave_run_triggers_for_rbr 0 the compiler will remove the whole if() block anyway. > > /* Honor next number column if present */ > m_table->next_number_field= m_table->found_next_number_field; > @@ -10928,6 +11039,13 @@ Rows_log_event::write_row(rpl_group_info > TABLE *table= m_table; // pointer to event's table > int error; > int UNINIT_VAR(keynum); > + > +#ifdef RBR_TRIGGERS > + bool invoke_triggers= > + slave_run_triggers_for_rbr && !master_had_triggers && table->triggers; > +#else > +#define invoke_triggers 0 > +#endif if you have slave_run_triggers_for_rbr defined to 0 then you don't need this idef either. although I'd still declare invoke_triggers as const, to make it easier for the compiler. > auto_afree_ptr<char> key(NULL); > > prepare_record(table, m_width, > @@ -10953,6 +11075,14 @@ Rows_log_event::write_row(rpl_group_info > DBUG_PRINT_BITSET("debug", "read_set = %s", table->read_set); > #endif > > +#ifdef RBR_TRIGGERS > + if (invoke_triggers && > + process_triggers(TRG_EVENT_INSERT, TRG_ACTION_BEFORE, TRUE)) > + { > + DBUG_RETURN(HA_ERR_GENERIC); // in case if error is not set yet > + } > +#endif //RBR_TRIGGERS why ifdef if invoke_triggers is 0 anyway? > + > /* > Try to write record. If a corresponding record already exists in the table, > we try to change it using ha_update_row() if possible. Otherwise we delete > @@ -11079,38 +11209,71 @@ Rows_log_event::write_row(rpl_group_info > !table->file->referenced_by_foreign_key()) > { > DBUG_PRINT("info",("Updating row using ha_update_row()")); > - error=table->file->ha_update_row(table->record[1], > - table->record[0]); > - switch (error) { > - > - case HA_ERR_RECORD_IS_THE_SAME: > - DBUG_PRINT("info",("ignoring HA_ERR_RECORD_IS_THE_SAME error from" > - " ha_update_row()")); > - error= 0; > - > - case 0: > - break; > - > - default: > - DBUG_PRINT("info",("ha_update_row() returns error %d",error)); > - table->file->print_error(error, MYF(0)); > +#ifdef RBR_TRIGGERS why ifdef if invoke_triggers is 0 anyway? > + if (invoke_triggers && > + process_triggers(TRG_EVENT_UPDATE, TRG_ACTION_BEFORE, FALSE)) > + error= HA_ERR_GENERIC; // in case if error is not set yet > + else > +#endif > + { > + error= table->file->ha_update_row(table->record[1], > + table->record[0]); > + switch (error) { > + > + case HA_ERR_RECORD_IS_THE_SAME: > + DBUG_PRINT("info",("ignoring HA_ERR_RECORD_IS_THE_SAME error from" > + " ha_update_row()")); > + error= 0; > + > + case 0: > + break; > + > + default: > + DBUG_PRINT("info",("ha_update_row() returns error %d",error)); > + table->file->print_error(error, MYF(0)); > + } > +#ifdef RBR_TRIGGERS > + if (invoke_triggers && !error && > + (process_triggers(TRG_EVENT_UPDATE, TRG_ACTION_AFTER, TRUE) || > + process_triggers(TRG_EVENT_INSERT, TRG_ACTION_AFTER, TRUE))) > + error= HA_ERR_GENERIC; // in case if error is not set yet > +#endif //RBR_TRIGGERS > } > - > + > DBUG_RETURN(error); > } > else > { > DBUG_PRINT("info",("Deleting offending row and trying to write new one again")); > - if ((error= table->file->ha_delete_row(table->record[1]))) > +#ifdef RBR_TRIGGERS > + if (invoke_triggers && > + process_triggers(TRG_EVENT_DELETE, TRG_ACTION_BEFORE, TRUE)) > + error= HA_ERR_GENERIC; // in case if error is not set yet > + else > +#endif //RBR_TRIGGERS > { > - DBUG_PRINT("info",("ha_delete_row() returns error %d",error)); > - table->file->print_error(error, MYF(0)); > - DBUG_RETURN(error); > + if ((error= table->file->ha_delete_row(table->record[1]))) > + { > + DBUG_PRINT("info",("ha_delete_row() returns error %d",error)); > + table->file->print_error(error, MYF(0)); > + DBUG_RETURN(error); > + } > +#ifdef RBR_TRIGGERS > + if (invoke_triggers && > + process_triggers(TRG_EVENT_DELETE, TRG_ACTION_AFTER, TRUE)) > + DBUG_RETURN(HA_ERR_GENERIC); // in case if error is not set yet > +#endif //RBR_TRIGGERS > } > /* Will retry ha_write_row() with the offending row removed. */ > } > } > > +#ifdef RBR_TRIGGERS > + if (invoke_triggers && > + process_triggers(TRG_EVENT_INSERT, TRG_ACTION_AFTER, TRUE)) > + error= HA_ERR_GENERIC; // in case if error is not set yet > +#endif //RBR_TRIGGERS > + > DBUG_RETURN(error); > } > > === modified file 'sql/sys_vars.cc' > --- sql/sys_vars.cc 2013-11-25 14:49:40 +0000 > +++ sql/sys_vars.cc 2014-02-28 10:56:24 +0000 > @@ -2601,6 +2601,22 @@ static Sys_var_enum Slave_exec_mode( > GLOBAL_VAR(slave_exec_mode_options), CMD_LINE(REQUIRED_ARG), > slave_exec_mode_names, DEFAULT(SLAVE_EXEC_MODE_STRICT)); > > +#ifdef RBR_TRIGGERS > +static const char *slave_run_triggers_for_rbr_names[]= > + {"NO", "YES", "LOGGING", 0}; > +static Sys_var_enum Slave_run_triggers_for_rbr( > + "slave_run_triggers_for_rbr", > + "Modes for how triggers in row-base replication on slave side will be " > + "executed. Legal values are NO (default), YES and LOGGING. NO means " > + "that trigger for RBR will not be running on slave. YES and LOGGING " > + "means that triggers will be running on slave, if there was not " > + "triggers running on the master for the statement. LOGGING also means " > + "results of that the executed triggers work will be written to " > + "the binlog.", > + GLOBAL_VAR(slave_run_triggers_for_rbr), CMD_LINE(REQUIRED_ARG), > + slave_run_triggers_for_rbr_names, > + DEFAULT(SLAVE_RUN_TRIGGERS_FOR_RBR_NO)); > +#endif //RBR_TRIGGERS empty line between variables would be nice > static const char *slave_type_conversions_name[]= {"ALL_LOSSY", "ALL_NON_LOSSY", 0}; > static Sys_var_set Slave_type_conversions( > "slave_type_conversions", > > === modified file 'sql/table.cc' > --- sql/table.cc 2013-11-11 18:46:14 +0000 > +++ sql/table.cc 2014-02-27 19:10:57 +0000 > @@ -6660,6 +6660,81 @@ int TABLE::update_default_fields() > > > /* > + Prepare triggers for INSERT-like statement. > + > + SYNOPSIS > + prepare_triggers_for_insert_stmt_or_event() > + > + NOTE > + Prepare triggers for INSERT-like statement by marking fields > + used by triggers and inform handlers that batching of UPDATE/DELETE > + cannot be done if there are BEFORE UPDATE/DELETE triggers. > +*/ > + > +void TABLE::prepare_triggers_for_insert_stmt_or_event() > +{ > + if (triggers) > + { > + if (triggers->has_triggers(TRG_EVENT_DELETE, > + TRG_ACTION_AFTER)) > + { > + /* > + The table has AFTER DELETE triggers that might access to > + subject table and therefore might need delete to be done > + immediately. So we turn-off the batching. > + */ > + (void) file->extra(HA_EXTRA_DELETE_CANNOT_BATCH); > + } > + if (triggers->has_triggers(TRG_EVENT_UPDATE, > + TRG_ACTION_AFTER)) > + { > + /* > + The table has AFTER UPDATE triggers that might access to subject > + table and therefore might need update to be done immediately. > + So we turn-off the batching. > + */ > + (void) file->extra(HA_EXTRA_UPDATE_CANNOT_BATCH); > + } > + } > +} > + > + > +bool TABLE::prepare_triggers_for_delete_stmt_or_event() > +{ > + if (triggers && > + triggers->has_triggers(TRG_EVENT_DELETE, > + TRG_ACTION_AFTER)) > + { > + /* > + The table has AFTER DELETE triggers that might access to subject table > + and therefore might need delete to be done immediately. So we turn-off > + the batching. > + */ > + (void) file->extra(HA_EXTRA_DELETE_CANNOT_BATCH); > + return TRUE; > + } > + return FALSE; > +} why didn't you put this in the .h file? it's something that should rather be inlined > + > + > +bool TABLE::prepare_triggers_for_update_stmt_or_event() > +{ > + if (triggers && > + triggers->has_triggers(TRG_EVENT_UPDATE, > + TRG_ACTION_AFTER)) > + { > + /* > + The table has AFTER UPDATE triggers that might access to subject > + table and therefore might need update to be done immediately. > + So we turn-off the batching. > + */ > + (void) file->extra(HA_EXTRA_UPDATE_CANNOT_BATCH); > + return TRUE; > + } > + return FALSE; > +} > + > +/* > @brief Reset const_table flag > > @detail > Regards, Sergei

1 0

[Maria-developers] two places maybe need adjusted in rpl_parallel.cc
by nanyi607rao 07 Mar '14

07 Mar '14

hi Kistian, I afraid there are two palces need adjusted in rpl_parallel.cc First is in rpl_parallel_thread::get_rgi() adjusted like this ... else { if(!(rgi= new rpl_group_info(rli))) { my_error(ER_OUTOFMEMORY, MYF(0), (int)sizeof(*rgi)); return NULL; } rgi->is_parallel_exec = true; - if ((rgi->deferred_events_collecting= rli->mi->rpl_filter->is_on())) _ rgi->deferred_events= new Deferred_log_events(rli); } + if ((rgi->deferred_events_collecting= rli->mi->rpl_filter->is_on())) + rgi->deferred_events= new Deferred_log_events(rli); because rgi->deferred_events was freed in rpl_parallel_thread::free_rgi() if it wasn't NULL Second is in rpl_parallel_change_thread_count() I think it need add mysql_cond_broadcast(&pool->COND_rpl_thread_pool) after all new threads are ok, this is for the case that skip_slave_start=OFF,then SQL thread would start automaticlly, it could wait forever in rpl_parallel_thread_pool::get_thread(), because SQL thread may reach this function before another thread reach the rpl_parallel_change_thread_count(). 2014-03-04 nanyi607rao

2 2