- developers - lists.mariadb.org

Re: [Maria-developers] [Commits] Rev 3077: Fixed LP bug #904345. in file:///home/igor/maria/maria-5.2-bug904345/
by Oleksandr Byelkin 27 Dec '11

27 Dec '11

Hi! 27.12.2011 20:46, Igor Babaev пишет: > At file:///home/igor/maria/maria-5.2-bug904345/ > > ------------------------------------------------------------ > revno: 3077 > revision-id: igor(a)askmonty.org-20111227184559-v7hlo1h98dz8ui32 > parent: wlad(a)montyprogram.com-20111223140257-k34gmwhs5msbc57k > committer: Igor Babaev<igor(a)askmonty.org> > branch nick: maria-5.2-bug904345 > timestamp: Tue 2011-12-27 10:45:59 -0800 > message: > Fixed LP bug #904345. > > The MIN/MAX optimizer code from the function opt_sum_query erroneously > did not take into account conjunctive conditions that did not depend on > any table, yet were not identified as constant items. These could be > items containing rand() or PS/SP parameters. These items are supposed > to be evaluated at the execution phase. That's why if such conditions > can be extracted from the WHERE condition the MIN/MAX optimization is > not applied as currently it is always done at the optimization phase. > > (In 5.3 expensive subqueries are also evaluated only at the execution > phase. So, if a constant condition with such subquery can be extracted > from the WHERE clause the MIN/MAX optimization should not be applied > in 5.3.) > > IF an IN/ALL/SOME predicate with a constant left part is transformed > into an EXISTS subquery the resulting subquery should not be considered > uncacheable if the right part of the predicate is not uncacheable. > > Backported the function dbug_print_item() from 5.3. The function is used > only for debugging. [skip] > === modified file 'sql/item.cc' > --- a/sql/item.cc 2011-11-24 16:48:58 +0000 > +++ b/sql/item.cc 2011-12-27 18:45:59 +0000 > @@ -7915,6 +7915,22 @@ > ((TABLE_LIST *)data)->hide_view_error(thd); > } > > +/* Debugger help function */ > +static char dbug_item_print_buf[256]; > + > +const char *dbug_print_item(Item *item) > +{ > + char *buf= dbug_item_print_buf; > + String str(buf, sizeof(dbug_item_print_buf),&my_charset_bin); > + str.length(0); > + if (!item) > + return "(Item*)NULL"; > + item->print(&str ,QT_ORDINARY); > + if (str.c_ptr() == buf) > + return buf; > + else > + return "Couldn't fit into buffer"; > +} This should be put under #ifndef DBUG_OFF Everything else looks OK to be pushed. [skip]

1 0

Re: [Maria-developers] [Commits] Rev 3170: After-merge fixes for 5.5 merge. in http://bazaar.launchpad.net/~maria-captains/maria/5.5
by Kristian Nielsen 27 Dec '11

27 Dec '11

Hi Monty, While fixing test failures in the MariaDB 5.5 tree, I encountered problems in sql_string, related to changes you did there in 5.2. I managed to come up with some solution, but please see below patch and see if you can suggest something better. Two issues: 1. You added realloc_with_extra() / realloc_with_extra_if_needed(), and used them to replace calls to realloc(). But realloc() as part of its operation puts a terminating '\0' at the end of the allocation; with your change this is now at a different (and not very useful) position in the buffer. My change tries to put it in the original place, but it is not terribly elegant, and perhaps your original change needs to be rethought. 2. I found a terrible mess (IMHO): We have two distict, subtly differing, sql_string.{h,cc}, one in /sql/ and one in /client/. We even seem to have a requirement for ABI compatibility between the two, since mysql_embedded uses client/sql_string.h with sql/sql_string.cc :-( And in fact your change seems to break such ABI compatibility, so I suspect mysql_embedded might be subtly broken in 5.2. But why do we need to have two sql_string in the first place? Especially given that they are apparently sufficiently close to be ABI compatible ... can't we eliminate on of them to get at least a little sanity into the sql_string story? - Kristian. knielsen(a)knielsen-hq.org writes: > At http://bazaar.launchpad.net/~maria-captains/maria/5.5 > > ------------------------------------------------------------ > revno: 3170 > revision-id: knielsen(a)knielsen-hq.org-20111209144027-yf5as4v9lb6jt3s7 > parent: knielsen(a)knielsen-hq.org-20111208170848-pijq4i8ceqwcl19q > committer: knielsen(a)knielsen-hq.org > branch nick: mariadb-5.5 > timestamp: Fri 2011-12-09 15:40:27 +0100 > message: > After-merge fixes for 5.5 merge. > > Fix typo causing too low timeout value for wait_for_slave_param.inc. > Fix wrong/missing '\0' termination in String:realloc() after Monty's > optimisations to reduce reallocation calls. > === modified file 'mysql-test/include/wait_for_slave_param.inc' > --- a/mysql-test/include/wait_for_slave_param.inc 2010-12-20 14:15:01 +0000 > +++ b/mysql-test/include/wait_for_slave_param.inc 2011-12-09 14:40:27 +0000 > @@ -79,7 +79,7 @@ if ($_slave_check_configured == 'No such > > # mysqltest doesn't provide any better way to multiply by 10 > --let $_wait_for_slave_param_zero= 0 > ---let $_slave_timeout_counter= $_slave_timeout$zero > +--let $_slave_timeout_counter= $_slave_timeout$_wait_for_slave_param_zero > --let $_slave_continue= 1 > while ($_slave_continue) > { > > === modified file 'sql/rpl_handler.cc' > --- a/sql/rpl_handler.cc 2011-11-03 18:17:05 +0000 > +++ b/sql/rpl_handler.cc 2011-12-09 14:40:27 +0000 > @@ -397,7 +397,7 @@ int Binlog_transmit_delegate::before_sen > > int ret= 0; > FOREACH_OBSERVER(ret, before_send_event, thd, > - (&param, (uchar *)packet->c_ptr(), > + (&param, (uchar *)packet->c_ptr_safe(), > packet->length(), > log_file+dirname_length(log_file), log_pos)); > return ret; > @@ -411,7 +411,7 @@ int Binlog_transmit_delegate::after_send > > int ret= 0; > FOREACH_OBSERVER(ret, after_send_event, thd, > - (&param, packet->c_ptr(), packet->length())); > + (&param, packet->c_ptr_safe(), packet->length())); > return ret; > } > > > === modified file 'sql/sql_string.cc' > --- a/sql/sql_string.cc 2011-11-03 18:17:05 +0000 > +++ b/sql/sql_string.cc 2011-12-09 14:40:27 +0000 > @@ -80,8 +80,12 @@ bool String::real_alloc(uint32 length) > no allocation occured. > > @retval true An error occured when attempting to allocate memory. > + > + The _internal() version is just to share code between realloc() and > + realloc_with_extra(), so that we can put the NULL termination at the right > + position. > */ > -bool String::realloc(uint32 alloc_length) > +bool String::realloc_internal(uint32 alloc_length) > { > if (Alloced_length <= alloc_length) > { > @@ -109,6 +113,13 @@ bool String::realloc(uint32 alloc_length > Ptr= new_ptr; > Alloced_length= len; > } > + return FALSE; > +} > + > +bool String::realloc(uint32 alloc_length) > +{ > + if (realloc_internal(alloc_length)) > + return TRUE; > Ptr[alloc_length]=0; // This make other funcs shorter > return FALSE; > } > > === modified file 'sql/sql_string.h' > --- a/sql/sql_string.h 2011-11-03 18:17:05 +0000 > +++ b/sql/sql_string.h 2011-12-09 14:40:27 +0000 > @@ -255,12 +255,18 @@ public: > return real_alloc(arg_length); > } > bool real_alloc(uint32 arg_length); // Empties old string > +private: > + bool realloc_internal(uint32 arg_length); > +public: > bool realloc(uint32 arg_length); > bool realloc_with_extra(uint32 arg_length) > { > if (extra_alloc < 4096) > extra_alloc= extra_alloc*2+128; > - return realloc(arg_length + extra_alloc); > + if (realloc_internal(arg_length + extra_alloc)) > + return TRUE; > + Ptr[arg_length]= 0; > + return FALSE; > } > bool realloc_with_extra_if_needed(uint32 arg_length) > { > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

3 2

[Maria-developers] Thread pool (request for review)
by Vladislav Vaintroub 27 Dec '11

27 Dec '11

Hello, Worklog#246 http://askmonty.org/worklog/Server-BackLog/?tid=246 is ready to review. The worklog is implemented in lp:~maria-captains/maria/5.5-threadpool, and if someone prefers to have a one big diff, it is in the attachment(diff.diff) It would be great if Monty, Serg (optionally Kristian if there is interest) could take a some time to review it in the near future. The worklog has extensive description of the implementation, might be helpful to consult it before reviewing. The branch currently has thread-handling=pool-if-threads as default, I found it easier for testing. Based on results of benchmark (see below), it seems to be generally performing well. So maybe we should use it as default, beta version of 5.5 for better testing coverage . Maybe we can keep it as new default after testing default, if there are no complains. Benchmarking: I also run some sysbench on 3 machines (2 Linuxes - fb-maria1 with 16 cores and pitbull with 24 cores, and windows one with 8 cores but good IO), results in https://docs.google.com/spreadsheet/ccc?key=0AqvHLyfRWjmbdDY0U1MzZGlXY3VGdUx xcC1pUlBZeHc I used sysbench 0.4 to run some of "unit" oltp tests (point-select and update-nokey), as well as mere "classic" OLTP-readonly and OLTP-readwrite. All tests are run with number of concurrent clients ranging from 16 to 4096, with warm cache, with sysbench table having 1Mio records. The results seem to be quite different on all of the machines tested (the machines are very different, in terms of cores, IO etc) , yet threadpool has a positive effect on all 3 machines, and the positive effect seems to grow with the number of cores. Some words about how I ran the benchmarks 1. Benchmark client and server used different CPUs - (I used 'taskset' (Linux), or 'start /affinity' (Windows) to run benchmark client on #CPUs/4, the rest of CPUs are used by the server). On Linuxes, I used --thread_pool_size=<N> where N is number of cores I dedicate to server. 2.I used innodb_flush_log_at_trx_commit=0 and innodb_flush_method= ALL_O_DIRECT to avoid extensive fsyncing, which I think is ok for the purposes of the testing for this WL. 3. Every "write" benchmark (oltp_rw and update_nokey) started with new server (i.e I kill mysqld, remove innodb files, and restart mysqld for each test),whereas every "read" benchmark reused the same running server instance. Starting afresh with a new server on write benchmarks is done mailnly to eliminate the effects of the purge lag. I attach config file sysbench.ini is the one I used on fb-maria1 In case I missed to say it before, Happy Xmas and Happy New Year! Wlad PS. Elena, we need to have some plan on testing it. PPS. Kristian, tried to integrate the 5.5-threadpool branch into Buildbot, and stuck. Can you help?

2 2

Re: [Maria-developers] request for a feature -- LIMIT_ROWS_EXAMINED
by MARK CALLAGHAN 23 Dec '11

23 Dec '11

A warning would be excellent. An error would be a problem. Returning OK is probably not the best result when the examined limit is reached. On Wed, Dec 21, 2011 at 7:17 AM, Timour Katchaounov <timour(a)montyprogram.com> wrote: >> >> Hi! >> >>>>>>> "MARK" == MARK CALLAGHAN<mdcallag(a)gmail.com> writes: >> >> >> MARK> I want a variant of the LIMIT clause that limits the number of rows >> MARK> examined during SELECT processing. This should return OK when the >> MARK> limit is reached -- something like NESTED_LOOP_QUERY_LIMIT. While >> MARK> LIMIT can be LIMIT x and LIMIT x,y. This only supports one argument >> -- >> MARK> LIMIT_ROWS_EXAMINED x. Are you interested in implementing this >> feature >> MARK> for me? We will sponsor the work. The purpose of the feature is to >> MARK> prevent queries from taking too long when there isn't a great index >> MARK> and many rows might be filtered. >> >> I am a bit afraid of returning 'ok'. Would it not be better to return >> an error? >> >> Another option would be to generate a warning so that the application >> can know that they didn't get all rows. > > > This is what I would prefer to do: > - process the new LIMIT_ROWS_EXAMINED clause in the same way as LIMIT, > just with a different condition to end query execution, > - in addition produce a warning that not all rows were returned. > > Mark, does the above suit you? If you don't want to check for > warnings, you can skip them, however users will have the option > to detect that not all rows were returned. > > Timour > >> >> We can do this and it should be relatively easy to do. >> >> Regards, >> Monty >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~maria-developers >> Post to : maria-developers(a)lists.launchpad.net >> Unsubscribe : https://launchpad.net/~maria-developers >> More help : https://help.launchpad.net/ListHelp -- Mark Callaghan mdcallag(a)gmail.com

3 3

[Maria-developers] Fixes for PAM plugin in packages
by Kristian Nielsen 21 Dec '11

21 Dec '11

Hi Serg, As promised, here a summary of what I did to fix PAM in linux packages. I managed to fix most issues I think (except for RHEL5 as noted below), but it needs testing, which takes time due to waiting for Buildbot. - I installed libpam0g-dev in all the VMs for .deb and bintar packages. I checked on one of them that this really does cause pam modules to build and install. - I installed pam-devel on the centos5 VMs. I also tried on the rhel5, but failed. These VMs seem to require some sort of license key or something to install packages with yum, which I don't have. Maybe Daniel can sort it out (all that is needed is `sudo yum install pam-devel` in vm-rhel5-amd64-build.qcow2 and vm-rhel5-x86-build.qcow2). - I hacked .deb packaging scripts (ourdelta scripts) so that dialog.so is in the libmariadbclient16 package (it used to be in the mariadb-server-5.[23] package). I did this for 5.2 and 5.3 (not 5.1). - The dialog.so change could cause an upgrade problem I think - installing new libmariadbclient16 package on top of old mariadb-server-5.[23] package will conflict because they both contain /usr/lib/mysql/plugin/dialog.so. I tried to fix this by declaring a conflicts: dependency which should allow Apt to sort it out. I hope it works - .deb dependencies can be tricky, hopefully buildbot Upgrade tests will catch any problems. - I did not change RPM packages wrt. dialog.so. I don't know if it is needed, nor how to do it... What remains is basically testing. First checking Buildbot when it has chewed through all the builds. It might be a good idea also to test manually a couple packages that the PAM plugin is in there and actually works according to instructions on the knowledgebase. - Kristian.

2 1

[Maria-developers] request for a feature -- LIMIT_ROWS_EXAMINED
by MARK CALLAGHAN 21 Dec '11

21 Dec '11

I want a variant of the LIMIT clause that limits the number of rows examined during SELECT processing. This should return OK when the limit is reached -- something like NESTED_LOOP_QUERY_LIMIT. While LIMIT can be LIMIT x and LIMIT x,y. This only supports one argument -- LIMIT_ROWS_EXAMINED x. Are you interested in implementing this feature for me? We will sponsor the work. The purpose of the feature is to prevent queries from taking too long when there isn't a great index and many rows might be filtered. -- Mark Callaghan mdcallag(a)gmail.com

2 1

Re: [Maria-developers] request for a feature -- LIMIT_ROWS_EXAMINED
by MARK CALLAGHAN 18 Dec '11

18 Dec '11

On Sat, Dec 17, 2011 at 3:18 PM, Timour Katchaounov <timour(a)montyprogram.com> wrote: > Mark, > > >> I want a variant of the LIMIT clause that limits the number of rows >> examined during SELECT processing. This should return OK when the > > > I had a look at the code, and it seems you need a limit on the value > of Rows_examined that is currently available in the slow query log. > Did I understand correctly that this is what you need? yes > > >> limit is reached -- something like NESTED_LOOP_QUERY_LIMIT. While >> LIMIT can be LIMIT x and LIMIT x,y. This only supports one argument -- >> LIMIT_ROWS_EXAMINED x. Are you interested in implementing this feature >> for me? We will sponsor the work. The purpose of the feature is to >> prevent queries from taking too long when there isn't a great index >> and many rows might be filtered. >> > > Have you considered a system variable instead of extending the query > syntax? If you rejected this idea, why? > > It seems quite logical to me that one may want to set the same limit > for more than one query. In this case a system variable would make > life easier, because no queries need to be changed. Personally I > would prefer not to extend the query language with constructs whose > purpose is to compensate for the imperfection of the optimizer. This isn't about limits on the optimizer. We don't have a great index for some queries and never will. We prefer to stop queries early rather than have them examine 1,000,000 rows. Real users might be waiting on the query response so we really need to finish queries fast and in some cases our apps work as expected were queries allowed to terminate early. I almost always prefer to attach the semantics to the statement and that includes this case. Anyone looking at the statement via a log (slow query, general), SHOW PROCESSLIST, tcpdump should be able to understand what the statement does. Making the LIMIT part of a session variable prevents that. I also don't want to risk an additional round trip to the server and for drivers that don't support multi-statement or for limitations on the code that generates the SQL, I don't want to do: set session limit_rows_examined=1000; SELECT /* with rows examined limit */ ... ; set session limit_rows_examined=0; -- Mark Callaghan mdcallag(a)gmail.com

1 0

[Maria-developers] Patch for xtradb compilation on visual studio
by Kristian Nielsen 15 Dec '11

15 Dec '11

Here is a patch for XtraDB (from Percona Server 5.5.17-rel22.1). We needed this in MariaDB 5.5 to compile on Windows. The problem is that this compiler apparently does not like having #ifdef inside macro invocations. I thought Percona might want to include it upstream, in case there are other compilers that also do not like this. - Kristian. === modified file 'storage/innobase/handler/ha_innodb.cc' --- storage/innobase/handler/ha_innodb.cc 2011-12-15 09:34:14 +0000 +++ storage/innobase/handler/ha_innodb.cc 2011-12-15 12:35:48 +0000 @@ -12369,14 +12369,16 @@ static MYSQL_SYSVAR_ULONG(concurrency_ti "Number of times a thread is allowed to enter InnoDB within the same SQL query after it has once got the ticket", NULL, NULL, 500L, 1L, ~0L, 0); -static MYSQL_SYSVAR_LONG(kill_idle_transaction, srv_kill_idle_transaction, - PLUGIN_VAR_RQCMDARG, #ifdef EXTENDED_FOR_KILLIDLE - "If non-zero value, the idle session with transaction which is idle over the value in seconds is killed by InnoDB.", +#define TMP_STR "If non-zero value, the idle session with transaction which is idle over the value in seconds is killed by InnoDB." #else - "No effect for this build.", +#define TMP_STR "No effect for this build." #endif +static MYSQL_SYSVAR_LONG(kill_idle_transaction, srv_kill_idle_transaction, + PLUGIN_VAR_RQCMDARG, + TMP_STR, NULL, NULL, 0, 0, LONG_MAX, 0); +#undef TMP_STR static MYSQL_SYSVAR_LONG(file_io_threads, innobase_file_io_threads, PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY | PLUGIN_VAR_NOSYSVAR,

2 1

Re: [Maria-developers] Please fix: innodb_plugin.innodb_gis fails on 'gcov' builder
by Sergei Petrunia 14 Dec '11

14 Dec '11

Hi Alexey, We have a repeatable failure like this: https://internal.askmonty.org/buildbot/builders/gcov/builds/830/steps/test_… innodb_plugin.innodb_gis w3 [ fail ] Test ended at 2011-12-14 06:52:31 CURRENT_TEST: innodb_plugin.innodb_gis mysqltest: At line 1: query 'SET storage_engine=innodb' failed: 1286: Unknown table engine 'innodb' The result from queries just before the failure was: SET storage_engine=innodb; - saving '/home/buildbot/buildbot_basedir/perpush-gcov/build/mysql-test/var/3/log/innodb_plugin.innodb_gis/' to '/home/buildbot/buildbot_basedir/perpush-gcov/build/mysql-test/var/log/innodb_plugin.innodb_gis/' It's in GIS, so I assume it is on your turf. Could you please fix this? BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

3 2

Re: [Maria-developers] Please do a test of 5.3 with these new @@optimizer_switch settings
by Sergei Petrunia 14 Dec '11

14 Dec '11

Hi Elena, On Thu, Dec 08, 2011 at 06:29:01AM +0400, Elena Stepanova wrote: > I tried to set them run overnight, but it isn't going too well. Most of > tests are crashing, see https://bugs.launchpad.net/maria/+bug/901506 . > This bug is finally fixed and pushed. Could you please run more tests on the fix asap? BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0