- developers - lists.mariadb.org

[Maria-developers] Thread pool (request for review)
by Vladislav Vaintroub 27 Dec '11

27 Dec '11

Hello, Worklog#246 http://askmonty.org/worklog/Server-BackLog/?tid=246 is ready to review. The worklog is implemented in lp:~maria-captains/maria/5.5-threadpool, and if someone prefers to have a one big diff, it is in the attachment(diff.diff) It would be great if Monty, Serg (optionally Kristian if there is interest) could take a some time to review it in the near future. The worklog has extensive description of the implementation, might be helpful to consult it before reviewing. The branch currently has thread-handling=pool-if-threads as default, I found it easier for testing. Based on results of benchmark (see below), it seems to be generally performing well. So maybe we should use it as default, beta version of 5.5 for better testing coverage . Maybe we can keep it as new default after testing default, if there are no complains. Benchmarking: I also run some sysbench on 3 machines (2 Linuxes - fb-maria1 with 16 cores and pitbull with 24 cores, and windows one with 8 cores but good IO), results in https://docs.google.com/spreadsheet/ccc?key=0AqvHLyfRWjmbdDY0U1MzZGlXY3VGdUx xcC1pUlBZeHc I used sysbench 0.4 to run some of "unit" oltp tests (point-select and update-nokey), as well as mere "classic" OLTP-readonly and OLTP-readwrite. All tests are run with number of concurrent clients ranging from 16 to 4096, with warm cache, with sysbench table having 1Mio records. The results seem to be quite different on all of the machines tested (the machines are very different, in terms of cores, IO etc) , yet threadpool has a positive effect on all 3 machines, and the positive effect seems to grow with the number of cores. Some words about how I ran the benchmarks 1. Benchmark client and server used different CPUs - (I used 'taskset' (Linux), or 'start /affinity' (Windows) to run benchmark client on #CPUs/4, the rest of CPUs are used by the server). On Linuxes, I used --thread_pool_size=<N> where N is number of cores I dedicate to server. 2.I used innodb_flush_log_at_trx_commit=0 and innodb_flush_method= ALL_O_DIRECT to avoid extensive fsyncing, which I think is ok for the purposes of the testing for this WL. 3. Every "write" benchmark (oltp_rw and update_nokey) started with new server (i.e I kill mysqld, remove innodb files, and restart mysqld for each test),whereas every "read" benchmark reused the same running server instance. Starting afresh with a new server on write benchmarks is done mailnly to eliminate the effects of the purge lag. I attach config file sysbench.ini is the one I used on fb-maria1 In case I missed to say it before, Happy Xmas and Happy New Year! Wlad PS. Elena, we need to have some plan on testing it. PPS. Kristian, tried to integrate the 5.5-threadpool branch into Buildbot, and stuck. Can you help?

2 2

Re: [Maria-developers] request for a feature -- LIMIT_ROWS_EXAMINED
by MARK CALLAGHAN 23 Dec '11

23 Dec '11

A warning would be excellent. An error would be a problem. Returning OK is probably not the best result when the examined limit is reached. On Wed, Dec 21, 2011 at 7:17 AM, Timour Katchaounov <timour(a)montyprogram.com> wrote: >> >> Hi! >> >>>>>>> "MARK" == MARK CALLAGHAN<mdcallag(a)gmail.com> writes: >> >> >> MARK> I want a variant of the LIMIT clause that limits the number of rows >> MARK> examined during SELECT processing. This should return OK when the >> MARK> limit is reached -- something like NESTED_LOOP_QUERY_LIMIT. While >> MARK> LIMIT can be LIMIT x and LIMIT x,y. This only supports one argument >> -- >> MARK> LIMIT_ROWS_EXAMINED x. Are you interested in implementing this >> feature >> MARK> for me? We will sponsor the work. The purpose of the feature is to >> MARK> prevent queries from taking too long when there isn't a great index >> MARK> and many rows might be filtered. >> >> I am a bit afraid of returning 'ok'. Would it not be better to return >> an error? >> >> Another option would be to generate a warning so that the application >> can know that they didn't get all rows. > > > This is what I would prefer to do: > - process the new LIMIT_ROWS_EXAMINED clause in the same way as LIMIT, > just with a different condition to end query execution, > - in addition produce a warning that not all rows were returned. > > Mark, does the above suit you? If you don't want to check for > warnings, you can skip them, however users will have the option > to detect that not all rows were returned. > > Timour > >> >> We can do this and it should be relatively easy to do. >> >> Regards, >> Monty >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~maria-developers >> Post to : maria-developers(a)lists.launchpad.net >> Unsubscribe : https://launchpad.net/~maria-developers >> More help : https://help.launchpad.net/ListHelp -- Mark Callaghan mdcallag(a)gmail.com

3 3

[Maria-developers] Fixes for PAM plugin in packages
by Kristian Nielsen 21 Dec '11

21 Dec '11

Hi Serg, As promised, here a summary of what I did to fix PAM in linux packages. I managed to fix most issues I think (except for RHEL5 as noted below), but it needs testing, which takes time due to waiting for Buildbot. - I installed libpam0g-dev in all the VMs for .deb and bintar packages. I checked on one of them that this really does cause pam modules to build and install. - I installed pam-devel on the centos5 VMs. I also tried on the rhel5, but failed. These VMs seem to require some sort of license key or something to install packages with yum, which I don't have. Maybe Daniel can sort it out (all that is needed is `sudo yum install pam-devel` in vm-rhel5-amd64-build.qcow2 and vm-rhel5-x86-build.qcow2). - I hacked .deb packaging scripts (ourdelta scripts) so that dialog.so is in the libmariadbclient16 package (it used to be in the mariadb-server-5.[23] package). I did this for 5.2 and 5.3 (not 5.1). - The dialog.so change could cause an upgrade problem I think - installing new libmariadbclient16 package on top of old mariadb-server-5.[23] package will conflict because they both contain /usr/lib/mysql/plugin/dialog.so. I tried to fix this by declaring a conflicts: dependency which should allow Apt to sort it out. I hope it works - .deb dependencies can be tricky, hopefully buildbot Upgrade tests will catch any problems. - I did not change RPM packages wrt. dialog.so. I don't know if it is needed, nor how to do it... What remains is basically testing. First checking Buildbot when it has chewed through all the builds. It might be a good idea also to test manually a couple packages that the PAM plugin is in there and actually works according to instructions on the knowledgebase. - Kristian.

2 1

[Maria-developers] request for a feature -- LIMIT_ROWS_EXAMINED
by MARK CALLAGHAN 21 Dec '11

21 Dec '11

I want a variant of the LIMIT clause that limits the number of rows examined during SELECT processing. This should return OK when the limit is reached -- something like NESTED_LOOP_QUERY_LIMIT. While LIMIT can be LIMIT x and LIMIT x,y. This only supports one argument -- LIMIT_ROWS_EXAMINED x. Are you interested in implementing this feature for me? We will sponsor the work. The purpose of the feature is to prevent queries from taking too long when there isn't a great index and many rows might be filtered. -- Mark Callaghan mdcallag(a)gmail.com

2 1

Re: [Maria-developers] request for a feature -- LIMIT_ROWS_EXAMINED
by MARK CALLAGHAN 18 Dec '11

18 Dec '11

On Sat, Dec 17, 2011 at 3:18 PM, Timour Katchaounov <timour(a)montyprogram.com> wrote: > Mark, > > >> I want a variant of the LIMIT clause that limits the number of rows >> examined during SELECT processing. This should return OK when the > > > I had a look at the code, and it seems you need a limit on the value > of Rows_examined that is currently available in the slow query log. > Did I understand correctly that this is what you need? yes > > >> limit is reached -- something like NESTED_LOOP_QUERY_LIMIT. While >> LIMIT can be LIMIT x and LIMIT x,y. This only supports one argument -- >> LIMIT_ROWS_EXAMINED x. Are you interested in implementing this feature >> for me? We will sponsor the work. The purpose of the feature is to >> prevent queries from taking too long when there isn't a great index >> and many rows might be filtered. >> > > Have you considered a system variable instead of extending the query > syntax? If you rejected this idea, why? > > It seems quite logical to me that one may want to set the same limit > for more than one query. In this case a system variable would make > life easier, because no queries need to be changed. Personally I > would prefer not to extend the query language with constructs whose > purpose is to compensate for the imperfection of the optimizer. This isn't about limits on the optimizer. We don't have a great index for some queries and never will. We prefer to stop queries early rather than have them examine 1,000,000 rows. Real users might be waiting on the query response so we really need to finish queries fast and in some cases our apps work as expected were queries allowed to terminate early. I almost always prefer to attach the semantics to the statement and that includes this case. Anyone looking at the statement via a log (slow query, general), SHOW PROCESSLIST, tcpdump should be able to understand what the statement does. Making the LIMIT part of a session variable prevents that. I also don't want to risk an additional round trip to the server and for drivers that don't support multi-statement or for limitations on the code that generates the SQL, I don't want to do: set session limit_rows_examined=1000; SELECT /* with rows examined limit */ ... ; set session limit_rows_examined=0; -- Mark Callaghan mdcallag(a)gmail.com

1 0

[Maria-developers] Patch for xtradb compilation on visual studio
by Kristian Nielsen 15 Dec '11

15 Dec '11

Here is a patch for XtraDB (from Percona Server 5.5.17-rel22.1). We needed this in MariaDB 5.5 to compile on Windows. The problem is that this compiler apparently does not like having #ifdef inside macro invocations. I thought Percona might want to include it upstream, in case there are other compilers that also do not like this. - Kristian. === modified file 'storage/innobase/handler/ha_innodb.cc' --- storage/innobase/handler/ha_innodb.cc 2011-12-15 09:34:14 +0000 +++ storage/innobase/handler/ha_innodb.cc 2011-12-15 12:35:48 +0000 @@ -12369,14 +12369,16 @@ static MYSQL_SYSVAR_ULONG(concurrency_ti "Number of times a thread is allowed to enter InnoDB within the same SQL query after it has once got the ticket", NULL, NULL, 500L, 1L, ~0L, 0); -static MYSQL_SYSVAR_LONG(kill_idle_transaction, srv_kill_idle_transaction, - PLUGIN_VAR_RQCMDARG, #ifdef EXTENDED_FOR_KILLIDLE - "If non-zero value, the idle session with transaction which is idle over the value in seconds is killed by InnoDB.", +#define TMP_STR "If non-zero value, the idle session with transaction which is idle over the value in seconds is killed by InnoDB." #else - "No effect for this build.", +#define TMP_STR "No effect for this build." #endif +static MYSQL_SYSVAR_LONG(kill_idle_transaction, srv_kill_idle_transaction, + PLUGIN_VAR_RQCMDARG, + TMP_STR, NULL, NULL, 0, 0, LONG_MAX, 0); +#undef TMP_STR static MYSQL_SYSVAR_LONG(file_io_threads, innobase_file_io_threads, PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY | PLUGIN_VAR_NOSYSVAR,

2 1

Re: [Maria-developers] Please fix: innodb_plugin.innodb_gis fails on 'gcov' builder
by Sergei Petrunia 14 Dec '11

14 Dec '11

Hi Alexey, We have a repeatable failure like this: https://internal.askmonty.org/buildbot/builders/gcov/builds/830/steps/test_… innodb_plugin.innodb_gis w3 [ fail ] Test ended at 2011-12-14 06:52:31 CURRENT_TEST: innodb_plugin.innodb_gis mysqltest: At line 1: query 'SET storage_engine=innodb' failed: 1286: Unknown table engine 'innodb' The result from queries just before the failure was: SET storage_engine=innodb; - saving '/home/buildbot/buildbot_basedir/perpush-gcov/build/mysql-test/var/3/log/innodb_plugin.innodb_gis/' to '/home/buildbot/buildbot_basedir/perpush-gcov/build/mysql-test/var/log/innodb_plugin.innodb_gis/' It's in GIS, so I assume it is on your turf. Could you please fix this? BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

3 2

Re: [Maria-developers] Please do a test of 5.3 with these new @@optimizer_switch settings
by Sergei Petrunia 14 Dec '11

14 Dec '11

Hi Elena, On Thu, Dec 08, 2011 at 06:29:01AM +0400, Elena Stepanova wrote: > I tried to set them run overnight, but it isn't going too well. Most of > tests are crashing, see https://bugs.launchpad.net/maria/+bug/901506 . > This bug is finally fixed and pushed. Could you please run more tests on the fix asap? BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] Compilation problem on an old 32-bit machine
by Kristian Nielsen 12 Dec '11

12 Dec '11

Hi Paul, It seems a patch is needed for PBXT to build on 32-bit x86 in some cases: Elena Stepanova <elenst(a)montyprogram.com> writes: > lock_xt.h: Assembler messages: > lock_xt.h:70: Error: bad register name `%sil' > After some re-installation and long digging, I found the ancient bug > report http://gcc.gnu.org/bugzilla/show_bug.cgi?id=10153 which gets > re-filed regularly, the last time was a few months ago. They say it's > a code problem, and suggest replacing the '=r' constraint with '=q'. I > did so in all 4 lines that gcc complained about, and finally got it > work. It seems correct to me that the "q" constraint must be used rather than "r" for the 8-bit case. Below patch should do it (we will apply this in MariaDB). - Kristian. === modified file 'storage/pbxt/src/lock_xt.h' --- storage/pbxt/src/lock_xt.h 2010-05-06 12:57:15 +0000 +++ storage/pbxt/src/lock_xt.h 2011-12-12 11:34:54 +0000 @@ -67,9 +67,9 @@ inline void xt_atomic_inc1(volatile xtWo #elif defined(XT_ATOMIC_GNUC_X86) xtWord1 val; - asm volatile ("movb %1,%0" : "=r" (val) : "m" (*mptr) : "memory"); + asm volatile ("movb %1,%0" : "=q" (val) : "m" (*mptr) : "memory"); val++; - asm volatile ("xchgb %1,%0" : "=r" (val) : "m" (*mptr), "0" (val) : "memory"); + asm volatile ("xchgb %1,%0" : "=q" (val) : "m" (*mptr), "0" (val) : "memory"); #elif defined(XT_ATOMIC_SOLARIS_LIB) atomic_inc_8(mptr); #else @@ -91,9 +91,9 @@ inline xtWord1 xt_atomic_dec1(volatile x #elif defined(XT_ATOMIC_GNUC_X86) xtWord1 val2; - asm volatile ("movb %1, %0" : "=r" (val) : "m" (*mptr) : "memory"); + asm volatile ("movb %1, %0" : "=q" (val) : "m" (*mptr) : "memory"); val--; - asm volatile ("xchgb %1,%0" : "=r" (val2) : "m" (*mptr), "0" (val) : "memory"); + asm volatile ("xchgb %1,%0" : "=q" (val2) : "m" (*mptr), "0" (val) : "memory"); /* Should work, but compiler makes a mistake? * asm volatile ("xchgb %1, %0" : : "r" (val), "m" (*mptr) : "memory"); */

2 1

Re: [Maria-developers] Please fix: gis-precise crashes on solaris
by Alexey Botchkov 12 Dec '11

12 Dec '11

Hi, Adam. Yes i need the access to the machine. Wasn't able to figure out what's wrong by looking at the traces. The crashing query is probably the most trivial of them all. Regards HF 10.12.2011 15:29, Adam M Dutko wrote: > > Let me know if you need access to the machine and I will coordinate > off-list. > > -Adam > >

1 0