January 2014 - developers - lists.mariadb.org

[Maria-developers] Extendet FederatedX storage engine to support updating Sphinx RT indexes
by Markus Lidel 12 Jun '14

12 Jun '14

Hello, the Sphinx Search engine has RT indexes, which could be updated with a subset of SQL commands, refered to as SphinxQL: http://sphinxsearch.com/docs/2.1.1/sphinxql-reference.html Because the FederatedX storage engine uses some commands, which Sphinx does not know, here is a patch, wich extends the FederatedX storage engine to support this subset of commands. Now its possible to update Sphinx RT indexes directly from MariaDB. To create a connection to the RT index in the Sphinx default RT-index: index rt { type = rt path = @CONFDIR@/data/rt rt_field = title rt_field = content rt_attr_string = title rt_attr_string = content rt_attr_uint = gid } you just have to call: CREATE TABLE `rt` ( `id` BIGINT UNSIGNED NOT NULL, `title` TEXT, `content` TEXT, `gid` INT UNSIGNED ) ENGINE=FEDERATED CONNECTION='sphinxql://root@localhost:9306/rt'; It would be great, if the attached patch could be integrated into MariaDB. If you have any suggestions, please feel free to contact me. Best regards, Markus Lidel

4 8

[Maria-developers] Next steps in improving single-threaded performance
by Kristian Nielsen 01 May '14

01 May '14

I have been analysing CPU bottlenecks in single-threaded sysbench read-only load. I found that icache misses is the main bottleneck, and that profile-guided compiler optimisation (PGO) with GCC gives a large speedup, 25% or more. (More details in my blog posts: http://kristiannielsen.livejournal.com/17676.html http://kristiannielsen.livejournal.com/18168.html ) Now I would like to ask for some discussions/help in how to get this implemented in practice. It involves changing the build process for our binaries: First compile with gcc --coverage, then run some profile workload, then recompile with -fprofile-use. I implemented a simple program to generate some profile load: https://github.com/knielsen/gen_profile_load It runs a bunch of simple insert/select/update/delete, with different combinations of storage engine, binlog format, and client API. It is designed to run inside the build tree and handle starting and stopping the server being tested, so it is pretty close to a working setup. These commands work to generate a binary that is faster due to PGO: mkdir bld cd bld cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" .. make tests/gen_profile_load cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" make So all the pieces really are there, it should be possible to implement it. But we need to find a good way to integrate it into our build system. The best would be to integrate it into our cmake files. The gen_profile_load.c could go into tests/, ideally we would build both a static and dynamically linked version (so we get PGO for both libmysqlclient.a and libmysqlclient.so). Anyone can help me get cmake to do that? And it would be cool if we could get the above procedure to work completely within cmake, so that the user could just do: cmake -DWITH_PGO ... ; make and cmake would itself handle first building with --coverage, then running gen_profile_load.static and gen_profile_load.dynamic, then rebuilding with -fprofile-use. Anyone know if this is possible with cmake, and if so could help implement it? But alternatively, we could integrate a double build, like the commands above, into the buildbot scripts (.deb, .rpm, bintar). Any comments? Here are some more points: - I tested that gen_profile_load gives a good speedup of sysbench read-only (around 30%, so still very significant even though it generates a different and more varied load). - As another test, I removed all SELECT from gen_profile_load, and ran the resulting PGO binary with sysbench read-only. This still gave a fair speedup, despite the PGO load being completely different from the benchmark load. This gives me confidence that the PGO should not cause performance regressions in cases not covered well by gen_profile_load - More tests would be nice, of course. Axel, would you be able to build some binaries following above procedure, and test some different random benchmarks? Anything that is easy to run could be interesting, both to test for improvement, and to check against regressions. - We probably need a recent GCC version to get good results. I used GCC version 4.7.2. Maybe we should install this GCC version in all the VMs we use to build binaries? - Should we do this in 5.5? I think we might want to. The speedup is quite significant, and it seems very safe - no code modifications are involved, only different compiler options. Any thoughts? Volunteeres for helping with the cmake or buildbot parts? - Kristian.

4 14

[Maria-developers] Plugin init()/deinit() call convention
by Sergey Vojtovich 17 Feb '14

17 Feb '14

Hi Sergei, IIRC during early days of plugins framework there was a strict convention when plugin deinit() function must be called. Specifically only after successful call to init() function. I'm now fixing a bug in related code (5.5) and noticed that: - deinit() may (or may not) be called without prior call to init() - deinit() may (or may not) be called with prior failed call to init() Do you think the above is correct? Thanks, Sergey

2 1

[Maria-developers] system variable value
by Rich Prohaska 17 Feb '14

17 Feb '14

I have a ULONGLONG system variable in my storage engine, and I set its value to '-1' on the mysqld command line. Its value gets set to all ones without any warnings or errors. The number conversion functions just cast between longlong and ulonglong types without regard to the sign of the number. Is this correct behavior?

2 1

Re: [Maria-developers] uint6korr optimization
by Michael Widenius 16 Feb '14

16 Feb '14

Hi! >>>>> "Alexey" == Alexey Botchkov <holyfoot(a)askmonty.org> writes: Alexey> Hi, Monty. Alexey> I've looked at that part of the code, and tried to make the improved Alexey> versions of uint6korr and mi_uint6korr. Alexey> The small program attached that benchmarks these. Alexey> Depending on enabled macros it loops uint6korr, hf_uint6korr, Alexey> mi_uint6korr and hf_mi_uint6korr respectively. It performs 2 loops on Alexey> each functions first runs it once, the second - twice, so we can Alexey> calculate how much time was spent on the operation itself. Alexey> The results i got so far are: Alexey> elapsed 103 seconds on korr6-1 Alexey> elapsed 190 seconds on korr6-2 Alexey> elapsed 50 seconds on hf_korr6-1 Alexey> elapsed 79 seconds on hf_korr6-2 Alexey> elapsed 106 seconds on mi6-1 Alexey> elapsed 195 seconds on mi6-2 Alexey> elapsed 56 seconds on hf_mi6B-1 Alexey> elapsed 88 seconds on hf_mi6-2 Alexey> So the Alexey> hf_uint6korr is 3 times faster than uint6korr. Alexey> hf_mi_uint6korr is 2.8 times faster than mi_uint6korr. Alexey> You're welcome to check the code out. Thanks. What is important is to get fast versions of mi_uint3korr (Used a lot in ma_dynrec.c and gis) mi_uint4korr mi_uint5korr mi_uint6korr mi_uint7korr mi_uint8korr (Used for some variables) uint5korr uint6korr (Used a lot in Aria) uint8korr (I am including the full test so that anyone can comment upon this) ---------------------------------------------------------------------- #include <stdio.h> #include <time.h> #include <malloc.h> #define TEST_KORR6 #define TEST_HF_KORR6 #define TEST_MI6 #define TEST_HF_MI6 #define uint6korr(A) ((ulonglong)(((uint32) ((uchar) (A)[0])) + \ (((uint32) ((uchar) (A)[1])) << 8) + \ (((uint32) ((uchar) (A)[2])) << 16) + \ (((uint32) ((uchar) (A)[3])) << 24)) + \ (((ulonglong) ((uchar) (A)[4])) << 32) + \ (((ulonglong) ((uchar) (A)[5])) << 40)) #define mi_uint6korr(A) ((ulonglong)(((uint32) (((const uchar*) (A))[5])) +\ (((uint32) (((const uchar*) (A))[4])) << 8) +\ (((uint32) (((const uchar*) (A))[3])) << 16) +\ (((uint32) (((const uchar*) (A))[2])) << 24)) +\ (((ulonglong) (((uint32) (((const uchar*) (A))[1])) +\ (((uint32) (((const uchar*) (A))[0]) << 8)))) <<\ 32)) #define hf_uint6korr(A) (((ulonglong) ((uint32 *) (A))[0]) + (((ulonglong) ((uint16 *) (A))[2]) << 32)) #define hf_mi_uint6korr(src, dest) \ __asm__ ( \ "bswapq %1;" \ "mov %1, %0;" \ :"=r"(dest) \ :"r"(hf_uint6korr(src)<<16) \ : \ ) typedef unsigned long long int ulonglong; typedef unsigned int uint32; typedef unsigned char uchar; typedef unsigned short uint16; time_t t0, t1; ulonglong i; #define GM 10000000000LL #define BAS 2000000 int main() { ulonglong *pb, *pb2; char *art, *art2; ulonglong ci; art= malloc(6*BAS); pb= malloc(sizeof(ulonglong)*BAS); for (i=0; i<6*BAS; i++) art[i]= (char)i; art2= malloc(6*BAS); pb2= malloc(sizeof(ulonglong)*BAS); for (i=0; i<6*BAS; i++) art2[i]= (char)i; #ifdef TEST_KORR6 ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; pb[ci]= uint6korr(art+ci*6); ci++; } t1= time(0); printf("elapsed %d seconds on korr6-1\n", t1 - t0); ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; pb[ci]= uint6korr(art+ci*6); pb2[ci]= uint6korr(art2+ci*6); ci++; } t1= time(0); printf("elapsed %d seconds on korr6-2\n", t1 - t0); #endif /*KORR6*/ #ifdef TEST_HF_KORR6 ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; pb[ci]= hf_uint6korr(art+ci*6); ci++; } t1= time(0); printf("elapsed %d seconds on hf_korr6-1\n", t1 - t0); ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; pb[ci]= hf_uint6korr(art+ci*6); pb2[ci]= hf_uint6korr(art2+ci*6); ci++; } t1= time(0); printf("elapsed %d seconds on hf_korr6-2\n", t1 - t0); #endif /*HF_KORR6*/ #ifdef TEST_MI6 ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; pb[ci]= mi_uint6korr(art+ci*6); ci++; } t1= time(0); printf("elapsed %d seconds on mi6-1\n", t1 - t0); ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; pb[ci]= mi_uint6korr(art+ci*6); pb2[ci]= mi_uint6korr(art2+ci*6); ci++; } t1= time(0); printf("elapsed %d seconds on mi6-2\n", t1 - t0); #endif /*MI6*/ #ifdef TEST_HF_MI6 ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; hf_mi_uint6korr(art+ci*6, pb[ci]); ci++; } t1= time(0); printf("elapsed %d seconds on hf_mi6B-1\n", t1 - t0); ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; hf_mi_uint6korr(art+ci*6, pb[ci]); hf_mi_uint6korr(art2+ci*6, pb2[ci]); ci++; } t1= time(0); printf("elapsed %d seconds on hf_mi6-2\n", t1 - t0); #endif /*HF_MI6*/ return 0; } Looks ok. Did you try with: #define hf_uint6korr(A) (*((uint64 *) A) & 0xffffffffffffLL) That may be faster, with the extra cost problem of reading 2 bytes extra, so this only safe to use if one knows that there is a few extra bytes in the buffer. Can you do a patch to my_global.h and myisampack and add the new functions there. What to do: - All improvements for now only for x64 architecture - Replace uint6korr(A) and any other function that can be improved without reading extra bytes with a faster version. - Improve with faster variants: mi_uint3korr mi_uint4korr mi_uint5korr mi_uint6korr mi_uint7korr mi_uint8korr uint5korr uint6korr uint8korr If the above suggested version of uint6korr() that reads extra bytes is notable faster, add a function: uint6korr_unsafe() Which of course should map to uint6korr for other architectures. ok? Regards, Monty

4 5

Re: [Maria-developers] Rev 3967: MDEV-5597 - Reduce usage of LOCK_open: LOCK_flush
by Sergei Golubchik 12 Feb '14

12 Feb '14

Hi, Sergey! On Jan 31, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 3967 > revision-id: svoj(a)mariadb.org-20140131141943-fjyyih3h92lek62n > parent: svoj(a)mariadb.org-20140130124905-vy28zbs9swlpg8uq > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Fri 2014-01-31 18:19:43 +0400 > message: > MDEV-5597 - Reduce usage of LOCK_open: LOCK_flush > > Replaced LOCK_flush with per-share conditional variable. > === modified file 'sql/table_cache.cc' > --- a/sql/table_cache.cc 2014-01-30 12:49:05 +0000 > +++ b/sql/table_cache.cc 2014-01-31 14:19:43 +0000 > @@ -945,6 +949,7 @@ bool tdc_remove_table(THD *thd, enum_tdc > if ((share= tdc_delete_share(db, table_name))) > { > I_P_List <TABLE, TABLE_share> purge_tables; > + uint my_refs= 1; > > mysql_mutex_lock(&LOCK_open); > /* > @@ -969,28 +974,48 @@ bool tdc_remove_table(THD *thd, enum_tdc > if (kill_delayed_threads) > kill_delayed_threads_for_table(share); > > -#ifndef DBUG_OFF > - if (remove_type == TDC_RT_REMOVE_NOT_OWN) > + if (remove_type == TDC_RT_REMOVE_NOT_OWN || > + remove_type == TDC_RT_REMOVE_NOT_OWN_KEEP_SHARE) > { > TABLE_SHARE::All_share_tables_list::Iterator it(share->tdc.all_tables); > while ((table= it++)) > + { > + my_refs++; > DBUG_ASSERT(table->in_use == thd); > + } > } > -#endif > - > - mysql_rwlock_rdlock(&LOCK_flush); > + DBUG_ASSERT(share->tdc.all_tables.is_empty() || remove_type != TDC_RT_REMOVE_ALL); > mysql_mutex_unlock(&LOCK_open); > > while ((table= purge_tables.pop_front())) > intern_close_table(table); > - mysql_rwlock_unlock(&LOCK_flush); > > - DBUG_ASSERT(share->tdc.all_tables.is_empty() || remove_type != TDC_RT_REMOVE_ALL); > - tdc_release_share(share); > + if (remove_type != TDC_RT_REMOVE_UNUSED) > + { > + /* > + Even though current thread holds exclusive metadata lock on this share > + (asserted above), concurrent FLUSH TABLES threads may be in process of > + closing unused table instances belonging to this share. E.g.: > + thr1 (FLUSH TABLES): table= share->tdc.free_tables.pop_front(); > + thr1 (FLUSH TABLES): share->tdc.all_tables.remove(table); > + thr2 (ALTER TABLE): tdc_remove_table(); > + thr1 (FLUSH TABLES): intern_close_table(table); > + > + Current remove type assumes that all table instances (except for those > + that are owned by current thread) must be closed before > + thd_remove_table() returns. Wait for such tables here. > + > + Unfortunately TABLE_SHARE::wait_for_old_version() cannot be used here > + because it waits for all table instances, whereas we have to wait only > + for those that are not owned by current thread. > + */ > + mysql_mutex_lock(&share->tdc.LOCK_table_share); > + while (share->tdc.ref_count > my_refs) > + mysql_cond_wait(&share->tdc.COND_release, &share->tdc.LOCK_table_share); > + mysql_mutex_unlock(&share->tdc.LOCK_table_share); I'm sorry, I don't understand that at all. Why my_refs is only incremented in debug builds? In non-debug builds you wait until all tables but one are closed, in debug builds you may keep more than one table opened. You wait here on a condition, how do you know there's FLUSH TABLES (or anything at all) running that will signal this condition? > + } > > - /* Wait for concurrent threads to free unused objects. */ > - mysql_rwlock_wrlock(&LOCK_flush); > - mysql_rwlock_unlock(&LOCK_flush); > + tdc_release_share(share); > > found= true; > } Regards, Sergei

2 4

[Maria-developers] Buildbot trees for spider and mroonga
by Alexander Barkov 10 Feb '14

10 Feb '14

Hi Kentoku, Currently you use your own trees for Mroonga and Spider development, e.g. bzr+ssh://bazaar.launchpad.net/~mroonga/maria/mroonga/ Would you mind to move development into trees under ~maria-captains location, for example: bzr+ssh://bazaar.launchpad.net/~maria-captains/maria/10.0-mroonga/ bzr+ssh://bazaar.launchpad.net/~maria-captains/maria/10.0-spider/ This will give an great advantage to have every revision compiled and tested on a wide variety of operating systems (this is done automatically by MariaDB Build Bot), and also will help us to merge your updates into the main 10.0 tree easier. Note, development in these trees should be done "incrementally". Things like "bzr push --overwrite" are not desirable, to avoid problems with BiildBot and with merges. Hope this is not a big problem. Thanks.

3 4

Re: [Maria-developers] MariaDB
by Kristian Nielsen 04 Feb '14

04 Feb '14

Otto Kekäläinen <otto(a)seravo.fi> writes: > 2013-10-01 Otto Kekäläinen <otto(a)fsfe.org>: >> 2013/10/1 Ondřej Surý <ondrej(a)sury.org>: >>> - it would be better to repack the tarball and kill the upstream debian/ >>> directory to not mess with our packaging >> >> I hope to merge my work upstream and that is easier if my changes are >> a made on top of current upstream. I think most of the changes should >> be useful for upstream too. Maybe you can participate in the process >> to create patches and test that they work for upstream OK? > there much easier. Although I will be needing help from somebody to > make autobuild.sh and standard Debian builders work nicely on the same > code base. Wouldn't it be better to discard the existing debian/ stuff in the upstream sources, and work from the native Debian packaging as the base? The existing stuff has _so_ much legacy and crap, and little of value. It has origins in some ancient copy of native Debian packages, then OurDelta hacked it for their build infrastructure (the autobuild.sh stuff), then MariaDB for our build infrastructure. It has desperately needed a cleanup for so long. Since this cleanup has now been done in native Debian packaging, which also makes unnecessary some of the hacks to work with native Debian packages, it seems better to just use Otto's work as the base going forwards. And in any case, the old reason for having debian packaging in-tree no longer applies. Now, we need to make sure before each release of MariaDB that it will play nicely with the upstream Debian packaging, and for this it seems most useful to pull in the packaging from the Debian packaging git repo? Well, in the end it is up to you. I suppose in any case there will be a wish to provide 3rd-party MariaDB .debs, eg. to provide backports to older distros of new versions. It just seems more natural to focus primarily on having good native packages and then accommodating the 3rd-party stuff, rather than the other way around? - Kristian.

3 4

[Maria-developers] MDEV-5273
by Alexey Botchkov 04 Feb '14

04 Feb '14

Hi, Serg, all. I'm not sure how about fixing this one: https://mariadb.atlassian.net/browse/MDEV-5273 So would like your opinion. The thing is that the SHOW commands with old-fasioned implementations (those that aren't mapped to the SELECT FROM information_schema.something) return stmt->field_count == 0 after the mysql_prepare_statement. The fix i see involves some boring coding. Is it worth to get into it right now? Best regards. HF

2 1

[Maria-developers] Extra copying in debian/rules causes troubles
by Elena Stepanova 04 Feb '14

04 Feb '14

Hi all. There is a fragment in MariaDB debian/dist/*/rules files: ... ifeq ($(findstring nocheck,$(DEB_BUILD_OPTIONS)),) # Don't know why the following is necessary... cp unittest/unit.pl $(builddir)/unittest/ cp -r mysql-test/* $(builddir)/mysql-test/ cp -r sql/share/* $(builddir)/sql/share/ cp -r scripts/*sql $(builddir)/scripts/ if [ ! -f testsuite-stamp ] ; then \ ... Recently it started causing a problem. MTR assumes that the tests are run either in a proper source build (be it in-source or out-of-source), or in a binary build. With the extra-copying above, particularly due to copying of mysql-test folder, the builddir by the moment of running tests is in the "neither fish nor fowl" state. So if one runs dpkg-buildpackage with the tests enabled, it fails when it comes to the MTR stage. We had this problem reported at least 3 times over the last 24 hours, so it's not hypothetical. It's currently filed as https://mariadb.atlassian.net/browse/MDEV-5591. I tried to remove these 4 'cp' lines and the build seemed to work all right; but there might also be consequences that I fail to predict, so I would like to have the 2nd (3rd, 4th, ... Nth) opinion. Does anyone have any idea what might go wrong without these 4 lines? Also, *Kristian*, These lines are coming from the initial draft of the rules. I understand from the comment that there is no reason to ask *why* the lines were necessary, but I'm wondering if you remember why you felt they *were* necessary (even if didn't know why). Were you getting some errors without them? Or were they copied from some previous version of the file? I realize it was long time ago so you probably won't remember, but I thought it was worth checking anyway. Regards, Elena

3 4