February 2014 - developers - lists.mariadb.org

[Maria-developers] New MariaDB builds: regular trusty and special kvm-deb-sid-pbuilder-amd and (Re: MariaDB)
by Otto Kekäläinen 01 Sep '14

01 Sep '14

Hello! 2014-02-04 20:41 GMT+02:00 Daniel Bartholomew <dbart(a)mariadb.com>: > On Tue, Feb 4, 2014 at 5:39 AM, Kristian Nielsen > <knielsen(a)knielsen-hq.org> wrote: >>> And while you are at it, please also consider setting up an instance >>> 'kvm-deb-sid-pbuilder-amd64' that provides a Debian unstable >> >> Daniel, can you prepare new sets of VMs for Ubuntu trusty and Debian sid? >> Then I can try to setup Buildbot to do these builds. > > Ok, will do. I should have them up and running before the end of the week. Any updates on this?

3 22

[Maria-developers] Extendet FederatedX storage engine to support updating Sphinx RT indexes
by Markus Lidel 12 Jun '14

12 Jun '14

Hello, the Sphinx Search engine has RT indexes, which could be updated with a subset of SQL commands, refered to as SphinxQL: http://sphinxsearch.com/docs/2.1.1/sphinxql-reference.html Because the FederatedX storage engine uses some commands, which Sphinx does not know, here is a patch, wich extends the FederatedX storage engine to support this subset of commands. Now its possible to update Sphinx RT indexes directly from MariaDB. To create a connection to the RT index in the Sphinx default RT-index: index rt { type = rt path = @CONFDIR@/data/rt rt_field = title rt_field = content rt_attr_string = title rt_attr_string = content rt_attr_uint = gid } you just have to call: CREATE TABLE `rt` ( `id` BIGINT UNSIGNED NOT NULL, `title` TEXT, `content` TEXT, `gid` INT UNSIGNED ) ENGINE=FEDERATED CONNECTION='sphinxql://root@localhost:9306/rt'; It would be great, if the attached patch could be integrated into MariaDB. If you have any suggestions, please feel free to contact me. Best regards, Markus Lidel

4 8

[Maria-developers] Next steps in improving single-threaded performance
by Kristian Nielsen 01 May '14

01 May '14

I have been analysing CPU bottlenecks in single-threaded sysbench read-only load. I found that icache misses is the main bottleneck, and that profile-guided compiler optimisation (PGO) with GCC gives a large speedup, 25% or more. (More details in my blog posts: http://kristiannielsen.livejournal.com/17676.html http://kristiannielsen.livejournal.com/18168.html ) Now I would like to ask for some discussions/help in how to get this implemented in practice. It involves changing the build process for our binaries: First compile with gcc --coverage, then run some profile workload, then recompile with -fprofile-use. I implemented a simple program to generate some profile load: https://github.com/knielsen/gen_profile_load It runs a bunch of simple insert/select/update/delete, with different combinations of storage engine, binlog format, and client API. It is designed to run inside the build tree and handle starting and stopping the server being tested, so it is pretty close to a working setup. These commands work to generate a binary that is faster due to PGO: mkdir bld cd bld cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" .. make tests/gen_profile_load cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" make So all the pieces really are there, it should be possible to implement it. But we need to find a good way to integrate it into our build system. The best would be to integrate it into our cmake files. The gen_profile_load.c could go into tests/, ideally we would build both a static and dynamically linked version (so we get PGO for both libmysqlclient.a and libmysqlclient.so). Anyone can help me get cmake to do that? And it would be cool if we could get the above procedure to work completely within cmake, so that the user could just do: cmake -DWITH_PGO ... ; make and cmake would itself handle first building with --coverage, then running gen_profile_load.static and gen_profile_load.dynamic, then rebuilding with -fprofile-use. Anyone know if this is possible with cmake, and if so could help implement it? But alternatively, we could integrate a double build, like the commands above, into the buildbot scripts (.deb, .rpm, bintar). Any comments? Here are some more points: - I tested that gen_profile_load gives a good speedup of sysbench read-only (around 30%, so still very significant even though it generates a different and more varied load). - As another test, I removed all SELECT from gen_profile_load, and ran the resulting PGO binary with sysbench read-only. This still gave a fair speedup, despite the PGO load being completely different from the benchmark load. This gives me confidence that the PGO should not cause performance regressions in cases not covered well by gen_profile_load - More tests would be nice, of course. Axel, would you be able to build some binaries following above procedure, and test some different random benchmarks? Anything that is easy to run could be interesting, both to test for improvement, and to check against regressions. - We probably need a recent GCC version to get good results. I used GCC version 4.7.2. Maybe we should install this GCC version in all the VMs we use to build binaries? - Should we do this in 5.5? I think we might want to. The speedup is quite significant, and it seems very safe - no code modifications are involved, only different compiler options. Any thoughts? Volunteeres for helping with the cmake or buildbot parts? - Kristian.

4 14

[Maria-developers] MariaDB scalability: mutexes
by Sergey Vojtovich 16 Apr '14

16 Apr '14

Hi Axel, I benchmarked various mutexes available to MariaDB. Results are here: http://svoj-db.blogspot.ru/2014/02/mariadb-mutexes-scalability.html Among other things I noticed that normal mutexes scale better than adaptive. And it looks like it gives over 5 000 TPS in my benchmarks: Original -------- 64 threads, time spent: 60s, queries executed: 11500723, qps: 191678, 1 thread qps: 2994, tps: 13691 64 threads, time spent: 60s, queries executed: 10983665, qps: 183061, 1 thread qps: 2860, tps: 13075 64 threads, time spent: 60s, queries executed: 11080325, qps: 184672, 1 thread qps: 2885, tps: 13190 slow LOCK_open -------------- 64 threads, time spent: 60s, queries executed: 12948365, qps: 215806, 1 thread qps: 3371, tps: 15414 64 threads, time spent: 60s, queries executed: 13191584, qps: 219859, 1 thread qps: 3435, tps: 15704 64 threads, time spent: 60s, queries executed: 13050647, qps: 217510, 1 thread qps: 3398, tps: 15536 slow LOCK_open + THR_LOCK::mutex -------------------------------- 64 threads, time spent: 60s, queries executed: 15863174, qps: 264386, 1 thread qps: 4131, tps: 18884 64 threads, time spent: 60s, queries executed: 15761944, qps: 262699, 1 thread qps: 4104, tps: 18764 64 threads, time spent: 60s, queries executed: 15773239, qps: 262887, 1 thread qps: 4107, tps: 18777 Could you do official benchmarks (patch attached)? It would be nice to also benchmark MySQL-5.6 with thr_lock part of the patch. Thanks, Sergey

3 8

Re: [Maria-developers] Rev 4077: MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client
by Sergei Golubchik 07 Apr '14

07 Apr '14

Hi, Holyfoot! On Feb 22, holyfoot(a)askmonty.org wrote: > revno: 4077 > revision-id: holyfoot(a)askmonty.org-20140222173543-xkzlp5rpga41evur > parent: svoj(a)mariadb.org-20140218065405-vw3bfhvhrfjsc6hh > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-4856 > timestamp: Sat 2014-02-22 21:35:43 +0400 > message: > MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client. > The fill_schema_table() function used to call get_table_share() > for a table name in WHERE then clear the error list. That way > plugins receive the superfluous error notification if it happens > in it. Also the problem was that error handler didn't prevent > the suppressed error message from logging anyway as the logging > happens in THD::raise_condition before the handler call. > Fixed by adding the Warnings_only_error_handler before the call. > raise_condition() also fixed. The Internal_error_handler class > was exteded with virtual functions to handle storing the error > message. This looks too complex to me. Why would you extend the base Internal_error_handler class - no other error handler did that or needed that, fwiw. I'd simply push an error handler, store the first message there and access it in get_schema_tables_record() via THD::get_internal_handler(). Regards, Sergei

2 4

[Maria-developers] A fix for MDEV-5689 ExtractValue(xml, 'substring(/x, /y)') crashes
by Alexander Barkov 23 Mar '14

23 Mar '14

Hi Sergei, please review a fix for MDEV-5689. It also fixes MDEV-5709 ExtractValue() with XPath variable references returns wrong result. Description: 1. The main problem was that that nodeset_func->fix_fields() was called in Item_func_xml_extractvalue::val_str() and Item_func_xml_update::val_str(), which led in some cases to execution of the XPath engine *before* having a parsed XML value. Moved to Item_xml_str_func::fix_fields(). 2. Cleanup: added a new method Item_xml_str_func::fix_fields() and moved most of the code from Item_xml_str_func::fix_length_and_dec() to Item_xml_str_func::fix_fields(), to follow the usual Item layout. 3. Cleanup: a parsed XML value is useless without the raw XML value it was built from. Previously the parsed and the raw values where stored in separate String instances. It was hard to follow how they are synchronized. Added a helper class XML which contains both parsed and raw values. Makes things easier to read and modify. 4. MDEV-5709: const_item() could incorrectly return a "true" result when XPath expression contains users/SP variable references. Now nodeset_func->const_item() is also taken into account to catch such cases. 5. Minor code enhancements.

2 5

Re: [Maria-developers] Rev 4029: MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client
by Sergei Golubchik 03 Mar '14

03 Mar '14

Hi, Holyfoot! On Jan 31, holyfoot(a)askmonty.org wrote: > revno: 4029 > revision-id: holyfoot(a)askmonty.org-20140131103303-1rx7ielo83f8iahe > parent: holyfoot(a)askmonty.org-20140124020722-dd5twtwlcc8o1xiy > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-4856 > timestamp: Fri 2014-01-31 14:33:03 +0400 > message: > MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client. > The fill_schema_table() function used to call get_table_share() for > a table name in WHERE then clear the error list. That way plugins > receive the superfluous error notification if it happens in it. Also > the problem was that error handler didn't prevent the suppressed > error message from logging anyway as the logging happens in > THD::raise_condition before the handler call. > Fixed by adding the No_table_error_handler before the call. > raise_condition() also fixed. > === modified file 'sql/sql_class.cc' > --- a/sql/sql_class.cc 2014-01-23 18:21:02 +0000 > +++ b/sql/sql_class.cc 2014-01-31 10:33:03 +0000 > @@ -1145,7 +1145,6 @@ MYSQL_ERROR* THD::raise_condition(uint s > got_warning= 1; > break; > case MYSQL_ERROR::WARN_LEVEL_ERROR: > - mysql_audit_general(this, MYSQL_AUDIT_GENERAL_ERROR, sql_errno, msg); > break; > default: > DBUG_ASSERT(FALSE); > @@ -1156,6 +1155,8 @@ MYSQL_ERROR* THD::raise_condition(uint s > > if (level == MYSQL_ERROR::WARN_LEVEL_ERROR) > { > + mysql_audit_general(this, MYSQL_AUDIT_GENERAL_ERROR, sql_errno, msg); > + This is good. > === modified file 'sql/sql_show.cc' > --- a/sql/sql_show.cc 2013-11-19 12:16:25 +0000 > +++ b/sql/sql_show.cc 2014-01-31 10:33:03 +0000 > @@ -4199,8 +4199,13 @@ static int fill_schema_table_from_frm(TH > key_length= create_table_def_key(thd, key, &table_list, 0); > hash_value= my_calc_hash(&table_def_cache, (uchar*) key, key_length); > mysql_mutex_lock(&LOCK_open); > - share= get_table_share(thd, &table_list, key, > - key_length, OPEN_VIEW, &not_used, hash_value); > + { > + No_such_table_error_handler no_such_table_handler; > + thd->push_internal_handler(&no_such_table_handler); > + share= get_table_share(thd, &table_list, key, > + key_length, OPEN_VIEW, &not_used, hash_value); > + thd->pop_internal_handler(); > + } But I don't understand that. Do you mean, old code did *not* suppress errors here? How could selects from I_S tables work without it? Regards, Sergei

2 5

[Maria-developers] slave_ddl_exec_mode and incompatible change in MariaDB 10.0.8
by Pavel Ivanov 03 Mar '14

03 Mar '14

Hi, I've discovered a change in MariaDB 10.0.8 that I don't quite understand motivation for and which looks really surprising to me. So I would appreciate if you could tell me what's the motivation and why you think it's appropriate to do that. The change in question is http://bazaar.launchpad.net/~maria-captains/maria/10.0/revision/3961.1.7. I see several issues with it: 1. It introduces a pretty significant variable slave_ddl_exec_mode which wasn't announced in the Release Notes. 2. The default variable value was set to IDEMPOTENT which changes MariaDB slave behavior in an incompatible way. Not everyone will want to have this variable set to IDEMPOTENT (I'd actually argue that no one should set it to IDEMPOTENT), but without any mentioning of this variable in the Release Notes people basically have the only chance to get knowledge about this change (let me stress out: _incompatible_ change) from production incidents or from accidental code inspection. 3. When slave_ddl_exec_mode is set to IDEMPOTENT every "DROP TABLE" event in master's binlog is converted into "DROP TABLE IF EXISTS" in slave's binlog, which I believe is a major no-no for replication. Binlogs on master and slave should be identical, otherwise after failover even if one would set slave_ddl_exec_mode = STRICT it will still execute "DROP TABLE IF EXISTS" because new master has that in binlogs. 4. Why this change introduces OPTION_GTID_BEGIN in the code? Why OPTION_BEGIN wasn't enough? From reading the code the only thing I understand is that with broken binlogs and sequence of events "GTID BEGIN; CREATE TABLE; CREATE TABLE" MariaDB will try to execute both CREATE TABLE statements in the same transaction without auto-committing at the end. This sounds pretty crazy to me. And given the fact that in normal situation such sequence of events on the master is not possible it seems that existence of both OPTION_BEGIN and OPTION_GTID_BEGIN only introduces confusion (MariaDB have GTID always on, right? So it should always execute GTID_BEGIN and never BEGIN without GTID, right?). So what is your word on this? Can you at least set default value for slave_ddl_exec_mode to STRICT so that slave behavior didn't change with the version upgrade? Thank you, Pavel

2 7

[Maria-developers] MDEV-5081 - Simple performance improvement for MariaDB
by Sergey Vojtovich 28 Feb '14

28 Feb '14

Hi Sergei, do you know if we ship any binaries with fast mutex (-DMY_PTHREAD_FASTMUTEX) enabled or officially suggest it? What was it's intention and what loads it was supposed to make faster? I'm trying to understand if fastmutex is worth fixing. It looks like duplication of PTHREAD_MUTEX_ADAPTIVE_NP, so we have double spinning when they're enabled. According to my recent benchmarks it shown worst performance. Thanks, Sergey

3 5

[Maria-developers] Keystone authentication plugin GSoc 14
by Abhishek Solanki 28 Feb '14

28 Feb '14

Hello MariaDB members , I am interested to work on "Keystone authentication plugin" project , i have a good knowledge of C/C++ and OpenStack. So can anyone help me how to get started with this project, so that i cold dig something before applying for this project. -- Thanks, Abhishek

2 1