March 2014 - developers - lists.mariadb.org

[Maria-developers] New MariaDB builds: regular trusty and special kvm-deb-sid-pbuilder-amd and (Re: MariaDB)
by Otto Kekäläinen 01 Sep '14

01 Sep '14

Hello! 2014-02-04 20:41 GMT+02:00 Daniel Bartholomew <dbart(a)mariadb.com>: > On Tue, Feb 4, 2014 at 5:39 AM, Kristian Nielsen > <knielsen(a)knielsen-hq.org> wrote: >>> And while you are at it, please also consider setting up an instance >>> 'kvm-deb-sid-pbuilder-amd64' that provides a Debian unstable >> >> Daniel, can you prepare new sets of VMs for Ubuntu trusty and Debian sid? >> Then I can try to setup Buildbot to do these builds. > > Ok, will do. I should have them up and running before the end of the week. Any updates on this?

3 22

[Maria-developers] Extendet FederatedX storage engine to support updating Sphinx RT indexes
by Markus Lidel 11 Jun '14

11 Jun '14

Hello, the Sphinx Search engine has RT indexes, which could be updated with a subset of SQL commands, refered to as SphinxQL: http://sphinxsearch.com/docs/2.1.1/sphinxql-reference.html Because the FederatedX storage engine uses some commands, which Sphinx does not know, here is a patch, wich extends the FederatedX storage engine to support this subset of commands. Now its possible to update Sphinx RT indexes directly from MariaDB. To create a connection to the RT index in the Sphinx default RT-index: index rt { type = rt path = @CONFDIR@/data/rt rt_field = title rt_field = content rt_attr_string = title rt_attr_string = content rt_attr_uint = gid } you just have to call: CREATE TABLE `rt` ( `id` BIGINT UNSIGNED NOT NULL, `title` TEXT, `content` TEXT, `gid` INT UNSIGNED ) ENGINE=FEDERATED CONNECTION='sphinxql://root@localhost:9306/rt'; It would be great, if the attached patch could be integrated into MariaDB. If you have any suggestions, please feel free to contact me. Best regards, Markus Lidel

4 8

[Maria-developers] connect.alter fails
by Alexander Barkov 07 May '14

07 May '14

Hello Olivier, connect.alter fails when built without XML support complaining about unsupported table type XML. I guess it should be skipped. connect.alter also fails with libxml2 with these symptoms: --- /home/bar/maria-bzr/maria-10.0-connect/storage/connect/mysql-test/connect/r/alter.result 2014-03-23 11:32:34.519624000 +0400 +++ /home/bar/maria-bzr/maria-10.0-connect/storage/connect/mysql-test/connect/r/alter.reject 2014-03-23 11:48:43.251680097 +0400 @@ -1,3 +1,5 @@ +Warnings: +Warning 1105 No file name. Table will use t1.xml Please fix. A remainder: please check every push in BuildBot: http://buildbot.askmonty.org/buildbot/grid?branch=10.0-connect

3 4

[Maria-developers] Next steps in improving single-threaded performance
by Kristian Nielsen 01 May '14

01 May '14

I have been analysing CPU bottlenecks in single-threaded sysbench read-only load. I found that icache misses is the main bottleneck, and that profile-guided compiler optimisation (PGO) with GCC gives a large speedup, 25% or more. (More details in my blog posts: http://kristiannielsen.livejournal.com/17676.html http://kristiannielsen.livejournal.com/18168.html ) Now I would like to ask for some discussions/help in how to get this implemented in practice. It involves changing the build process for our binaries: First compile with gcc --coverage, then run some profile workload, then recompile with -fprofile-use. I implemented a simple program to generate some profile load: https://github.com/knielsen/gen_profile_load It runs a bunch of simple insert/select/update/delete, with different combinations of storage engine, binlog format, and client API. It is designed to run inside the build tree and handle starting and stopping the server being tested, so it is pretty close to a working setup. These commands work to generate a binary that is faster due to PGO: mkdir bld cd bld cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" .. make tests/gen_profile_load cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" make So all the pieces really are there, it should be possible to implement it. But we need to find a good way to integrate it into our build system. The best would be to integrate it into our cmake files. The gen_profile_load.c could go into tests/, ideally we would build both a static and dynamically linked version (so we get PGO for both libmysqlclient.a and libmysqlclient.so). Anyone can help me get cmake to do that? And it would be cool if we could get the above procedure to work completely within cmake, so that the user could just do: cmake -DWITH_PGO ... ; make and cmake would itself handle first building with --coverage, then running gen_profile_load.static and gen_profile_load.dynamic, then rebuilding with -fprofile-use. Anyone know if this is possible with cmake, and if so could help implement it? But alternatively, we could integrate a double build, like the commands above, into the buildbot scripts (.deb, .rpm, bintar). Any comments? Here are some more points: - I tested that gen_profile_load gives a good speedup of sysbench read-only (around 30%, so still very significant even though it generates a different and more varied load). - As another test, I removed all SELECT from gen_profile_load, and ran the resulting PGO binary with sysbench read-only. This still gave a fair speedup, despite the PGO load being completely different from the benchmark load. This gives me confidence that the PGO should not cause performance regressions in cases not covered well by gen_profile_load - More tests would be nice, of course. Axel, would you be able to build some binaries following above procedure, and test some different random benchmarks? Anything that is easy to run could be interesting, both to test for improvement, and to check against regressions. - We probably need a recent GCC version to get good results. I used GCC version 4.7.2. Maybe we should install this GCC version in all the VMs we use to build binaries? - Should we do this in 5.5? I think we might want to. The speedup is quite significant, and it seems very safe - no code modifications are involved, only different compiler options. Any thoughts? Volunteeres for helping with the cmake or buildbot parts? - Kristian.

4 14

Re: [Maria-developers] [GSoC] Student interested in "Optimize mysql-test runs" or adding SQL features
by Sergei Golubchik 17 Apr '14

17 Apr '14

Hi, Pablo! First: please always cc: maria-developers(a)lists.launchpad.net, don't send emails to me only. Now: On Mar 11, Pablo Estrada wrote: > Hello! > I am getting ready to submit my proposal. Is there any feedback for my > application document? Anything that would be good to add, remove, rephrase, > etc? Go on and submit it. There will be a comment form under your proposal, if I will want you to change something, I will leave my comments there. Please mention what tools/languages you think you'll be using. > Also, if I could see the data available (the history of test results), I > could write more in detail : ) Hm. Okay. Attached is the preprocessed table that I used two years ago. It lists test failures in buildbot in a given period of time. Columns are (as far as I remember): * time of the test run start * code branch that was tested * revision id that was tested - it's fake, our buildbot tables don't store revision id. I've constructed it from the set of changed files, revision number, etc. hopefully it's as unique as a real revision id * build number * platform where the tests were run * test type - basically, the set of tests to run and what protocol to use. one test may be run with different protocols * test name * test variant - or combination. certain tests may be run in many combinations Using tis file I tried to rearrange test execution order for every specific build number (using failure data from previous builds in the same table) to detect test failures as early as possible. I managed to have 90% of all test failures happening within first 10% of the tests. The goal of this project is to have something we can put back into buildbot, so that it could actually run only 10% of the tests, still having 90% probability of discovering a failure (or 20%, or 95%, whaveter). See also http://buildbot.askmonty.org/buildbot/ you can click on a link for any branch, for example, http://buildbot.askmonty.org/buildbot/grid?branch=10.0 then on any particular build, for example, http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297 and on a stdio log for a particular test run: http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297/ste… on these pages you'll see what a platform (builder) is, build number, test types, combinations, etc. Regards, Sergei

2 2

[Maria-developers] MariaDB scalability: mutexes
by Sergey Vojtovich 16 Apr '14

16 Apr '14

Hi Axel, I benchmarked various mutexes available to MariaDB. Results are here: http://svoj-db.blogspot.ru/2014/02/mariadb-mutexes-scalability.html Among other things I noticed that normal mutexes scale better than adaptive. And it looks like it gives over 5 000 TPS in my benchmarks: Original -------- 64 threads, time spent: 60s, queries executed: 11500723, qps: 191678, 1 thread qps: 2994, tps: 13691 64 threads, time spent: 60s, queries executed: 10983665, qps: 183061, 1 thread qps: 2860, tps: 13075 64 threads, time spent: 60s, queries executed: 11080325, qps: 184672, 1 thread qps: 2885, tps: 13190 slow LOCK_open -------------- 64 threads, time spent: 60s, queries executed: 12948365, qps: 215806, 1 thread qps: 3371, tps: 15414 64 threads, time spent: 60s, queries executed: 13191584, qps: 219859, 1 thread qps: 3435, tps: 15704 64 threads, time spent: 60s, queries executed: 13050647, qps: 217510, 1 thread qps: 3398, tps: 15536 slow LOCK_open + THR_LOCK::mutex -------------------------------- 64 threads, time spent: 60s, queries executed: 15863174, qps: 264386, 1 thread qps: 4131, tps: 18884 64 threads, time spent: 60s, queries executed: 15761944, qps: 262699, 1 thread qps: 4104, tps: 18764 64 threads, time spent: 60s, queries executed: 15773239, qps: 262887, 1 thread qps: 4107, tps: 18777 Could you do official benchmarks (patch attached)? It would be nice to also benchmark MySQL-5.6 with thr_lock part of the patch. Thanks, Sergey

3 8

[Maria-developers] MariaDB release and support policies
by Otto Kekäläinen 14 Apr '14

14 Apr '14

Hello! Does MariaDB promise a 8 year release cycle for 5.5 as MySQL does? Does all of the points listed at https://lists.ubuntu.com/archives/technical-board/2014-February/001812.html apply for MariaDB as well? Can we promise Ubuntu that all MariaDB minor releases fulfill the quality requirements listed above? If so, I'll then file a micro release update permission request for Ubuntu. AFAIK as I know it is the only way to get security releases to users anyway. PS. Do you people know about semver.org and try to follow the general versioning policy? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 2

[Maria-developers] Third bunch of unmerged items
by Sergey Vojtovich 14 Apr '14

14 Apr '14

Hi Sergei, here comes third bunch of items which I'm unsure about: WL#5706/Bug#58712/Bug#11746378 - Encrypt or remove passwords from slow, query, and binary logs WL#4677 - Unique Server Ids for Replication Topology (UUIDs) WL#6004 - Server extension for allowing user tables to be put into transactional SE Bug #14112682 : MYSQLTEST: PROVIDE ACCESS TO ERROR NAMES THROUGH NUMERIC CODES AND VICE VERSA Bug #11747313 31741: MYSQLTEST - DEPRECATE "SYSTEM" COMMAND WL#2739: Auditing Password Security (plugin + API) WL#6587: disconnect_on_expired_password WL#5825 Using C++ Standard Library with MySQL code WL#2111: #6 Set CLASS_ORIGIN and SUBCLASS_ORIGIN Regards, Sergey

2 1

[Maria-developers] debugging MDEV-5932, wrong result set for tokudb
by Rich Prohaska 10 Apr '14

10 Apr '14

Hello, TokuDB generates an empty result set for a select with sub-select, but generated a non-empty result set for the same query when creating a new table. The result set should be the same in both cases. What is going on here? InnoDB computes an empty result set for both cases. I traced the problem to the tokudb::index_read function. The tokudb::index_read does not handle the null byte key prefix. Unfortunately, I do not understand how the null byte (byte 0) in the key handed to the ::index_read function is supposed to work. Most of the time, the value of this byte is 0. Sometimes it is set to 1. How should the storage engine work WRT this? BTW, this problem exists with TokuDB in MySQL 5.5.36, MySQL 5.6.16. I have not been able to consistently see this on MariaDB 5.5.36 for some reason. In any case, I am interested in how the handler's index_read function is supposed to process the null byte. Thanks

2 1

Re: [Maria-developers] Rev 4077: MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client
by Sergei Golubchik 07 Apr '14

07 Apr '14

Hi, Holyfoot! On Feb 22, holyfoot(a)askmonty.org wrote: > revno: 4077 > revision-id: holyfoot(a)askmonty.org-20140222173543-xkzlp5rpga41evur > parent: svoj(a)mariadb.org-20140218065405-vw3bfhvhrfjsc6hh > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-4856 > timestamp: Sat 2014-02-22 21:35:43 +0400 > message: > MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client. > The fill_schema_table() function used to call get_table_share() > for a table name in WHERE then clear the error list. That way > plugins receive the superfluous error notification if it happens > in it. Also the problem was that error handler didn't prevent > the suppressed error message from logging anyway as the logging > happens in THD::raise_condition before the handler call. > Fixed by adding the Warnings_only_error_handler before the call. > raise_condition() also fixed. The Internal_error_handler class > was exteded with virtual functions to handle storing the error > message. This looks too complex to me. Why would you extend the base Internal_error_handler class - no other error handler did that or needed that, fwiw. I'd simply push an error handler, store the first message there and access it in get_schema_tables_record() via THD::get_internal_handler(). Regards, Sergei

2 4