- developers - lists.mariadb.org

[Maria-developers] Problems with packages in 5.3 buildbot
by Kristian Nielsen 30 Dec '11

30 Dec '11

I learned that there were many test failures with packages in 5.3. I think it may be due to the version in configure.in not being increased after last release. I pushed a commit that changes it to 5.3.4-rc, we will see if it helps. - Kristian.

1 0

Re: [Maria-developers] License issues
by Henrik Ingo 30 Dec '11

30 Dec '11

I'm assuming you left out the list in accident and won't mind that I quote you a little: On Thu, Dec 29, 2011 at 10:22 PM, Sergei Golubchik <serg(a)askmonty.org> wrote: > On Dec 29, Henrik Ingo wrote: >> (I'm pretty sure readline developers didn't quite realize what they >> were doing here. Their original goal with their choice of license was >> to create an advantage for GPL licensed apps, now they are doing >> exactly the opposite.) > > I think it was intentional. > > readline is a gnu project, hosted on gnu.org. > It looks like an FSF policy to use GPLv3. Other gnu software than I've > checked is also on GPLv3 (in particular, libbfd - that is used by > binutils, gdb, etc - so it's kind of fundamental for development chain). Yes, I'm aware of this. However, most FSF *libraries* are licensed LGPL. Only readline and a few others (apparently then libbfd, if you say so) are GPL. The purpose of this is to make those libraries exclusively available to GPL software, or also compatible FOSS software. (eg. a BSD licensed application could use a GPL licensed library as long as source code of the app is in fact made available.) > I might be wrong, but I think using GPLv3 for all (or most) GNU > projects, including essencial libraries, is quite intentional. Well, it is intentional for FSF in the sense that it is policy. For the libraries that use LGPL it is also quite fine to force everyone to use them under LGPLv3, as it won't have similar bad effects on the apps that use those libraries. However, in the case of readline and other libraries that are GPL, this has the effect of hurting projects that are GPLv2 only, while it in fact improves the situation for non-copyleft projects such as those under Apache license. (Apache license is considered by FSF to be incompatible with GPLv2 but compatible with GPLv3.) I very much doubt that this particular consequence was intentional. To support GPL licensed software, the FSF should in this case have licensed readline as both GPLv2 and GPLv3. henrik -- henrik.ingo(a)avoinelama.fi +358-40-8211286 skype: henrik.ingo irc: hingo www.openlife.cc My LinkedIn profile: http://www.linkedin.com/profile/view?id=9522559

2 1

[Maria-developers] License issues
by Maarten Vanraes 30 Dec '11

30 Dec '11

Hi, I'm a packager for mageia, and i'm working to get branch 5.5 submitted on our system, replacing mysql. However, we have readline v6 on mageia, which is GPLv3, and serg added the NOT_FOR_DISTRIBUTION variable, which makes it build a bundled readline v5. well, it builds for us too, but later i get a conflict situation on our system between mariadb-common and mariadb-client (i didn't look deep into it). but normally we try to use as much system shared libraries as possible, also, what about security fixes + bug fixes in this bundled readline v5, etc... i donno how long v5 will be maintained by gnu, this is not an ideal situation. i've checked and noted that mariadb server is GPLv2only and mariadb client has a FLOSS exception... which doesn't mention GPLv3 so i'm a bit at a loss here, ideally we want to use system shared readline v6 on our distribution... it's a bit of a PITA for me, tbh, i mean, it's all open source, i don't really get why this is all so difficult... :-( I guess the easiest way for us is if mariadb was GPLv2+ we could go ahead and use readline v6. I guess i'm asking for a way to proceed for us. Regards, AL13N

4 6

Re: [Maria-developers] [Commits] Rev 3077: Fixed LP bug #904345. in file:///home/igor/maria/maria-5.2-bug904345/
by Oleksandr Byelkin 27 Dec '11

27 Dec '11

Hi! 27.12.2011 20:46, Igor Babaev пишет: > At file:///home/igor/maria/maria-5.2-bug904345/ > > ------------------------------------------------------------ > revno: 3077 > revision-id: igor(a)askmonty.org-20111227184559-v7hlo1h98dz8ui32 > parent: wlad(a)montyprogram.com-20111223140257-k34gmwhs5msbc57k > committer: Igor Babaev<igor(a)askmonty.org> > branch nick: maria-5.2-bug904345 > timestamp: Tue 2011-12-27 10:45:59 -0800 > message: > Fixed LP bug #904345. > > The MIN/MAX optimizer code from the function opt_sum_query erroneously > did not take into account conjunctive conditions that did not depend on > any table, yet were not identified as constant items. These could be > items containing rand() or PS/SP parameters. These items are supposed > to be evaluated at the execution phase. That's why if such conditions > can be extracted from the WHERE condition the MIN/MAX optimization is > not applied as currently it is always done at the optimization phase. > > (In 5.3 expensive subqueries are also evaluated only at the execution > phase. So, if a constant condition with such subquery can be extracted > from the WHERE clause the MIN/MAX optimization should not be applied > in 5.3.) > > IF an IN/ALL/SOME predicate with a constant left part is transformed > into an EXISTS subquery the resulting subquery should not be considered > uncacheable if the right part of the predicate is not uncacheable. > > Backported the function dbug_print_item() from 5.3. The function is used > only for debugging. [skip] > === modified file 'sql/item.cc' > --- a/sql/item.cc 2011-11-24 16:48:58 +0000 > +++ b/sql/item.cc 2011-12-27 18:45:59 +0000 > @@ -7915,6 +7915,22 @@ > ((TABLE_LIST *)data)->hide_view_error(thd); > } > > +/* Debugger help function */ > +static char dbug_item_print_buf[256]; > + > +const char *dbug_print_item(Item *item) > +{ > + char *buf= dbug_item_print_buf; > + String str(buf, sizeof(dbug_item_print_buf),&my_charset_bin); > + str.length(0); > + if (!item) > + return "(Item*)NULL"; > + item->print(&str ,QT_ORDINARY); > + if (str.c_ptr() == buf) > + return buf; > + else > + return "Couldn't fit into buffer"; > +} This should be put under #ifndef DBUG_OFF Everything else looks OK to be pushed. [skip]

1 0

Re: [Maria-developers] [Commits] Rev 3170: After-merge fixes for 5.5 merge. in http://bazaar.launchpad.net/~maria-captains/maria/5.5
by Kristian Nielsen 27 Dec '11

27 Dec '11

Hi Monty, While fixing test failures in the MariaDB 5.5 tree, I encountered problems in sql_string, related to changes you did there in 5.2. I managed to come up with some solution, but please see below patch and see if you can suggest something better. Two issues: 1. You added realloc_with_extra() / realloc_with_extra_if_needed(), and used them to replace calls to realloc(). But realloc() as part of its operation puts a terminating '\0' at the end of the allocation; with your change this is now at a different (and not very useful) position in the buffer. My change tries to put it in the original place, but it is not terribly elegant, and perhaps your original change needs to be rethought. 2. I found a terrible mess (IMHO): We have two distict, subtly differing, sql_string.{h,cc}, one in /sql/ and one in /client/. We even seem to have a requirement for ABI compatibility between the two, since mysql_embedded uses client/sql_string.h with sql/sql_string.cc :-( And in fact your change seems to break such ABI compatibility, so I suspect mysql_embedded might be subtly broken in 5.2. But why do we need to have two sql_string in the first place? Especially given that they are apparently sufficiently close to be ABI compatible ... can't we eliminate on of them to get at least a little sanity into the sql_string story? - Kristian. knielsen(a)knielsen-hq.org writes: > At http://bazaar.launchpad.net/~maria-captains/maria/5.5 > > ------------------------------------------------------------ > revno: 3170 > revision-id: knielsen(a)knielsen-hq.org-20111209144027-yf5as4v9lb6jt3s7 > parent: knielsen(a)knielsen-hq.org-20111208170848-pijq4i8ceqwcl19q > committer: knielsen(a)knielsen-hq.org > branch nick: mariadb-5.5 > timestamp: Fri 2011-12-09 15:40:27 +0100 > message: > After-merge fixes for 5.5 merge. > > Fix typo causing too low timeout value for wait_for_slave_param.inc. > Fix wrong/missing '\0' termination in String:realloc() after Monty's > optimisations to reduce reallocation calls. > === modified file 'mysql-test/include/wait_for_slave_param.inc' > --- a/mysql-test/include/wait_for_slave_param.inc 2010-12-20 14:15:01 +0000 > +++ b/mysql-test/include/wait_for_slave_param.inc 2011-12-09 14:40:27 +0000 > @@ -79,7 +79,7 @@ if ($_slave_check_configured == 'No such > > # mysqltest doesn't provide any better way to multiply by 10 > --let $_wait_for_slave_param_zero= 0 > ---let $_slave_timeout_counter= $_slave_timeout$zero > +--let $_slave_timeout_counter= $_slave_timeout$_wait_for_slave_param_zero > --let $_slave_continue= 1 > while ($_slave_continue) > { > > === modified file 'sql/rpl_handler.cc' > --- a/sql/rpl_handler.cc 2011-11-03 18:17:05 +0000 > +++ b/sql/rpl_handler.cc 2011-12-09 14:40:27 +0000 > @@ -397,7 +397,7 @@ int Binlog_transmit_delegate::before_sen > > int ret= 0; > FOREACH_OBSERVER(ret, before_send_event, thd, > - (&param, (uchar *)packet->c_ptr(), > + (&param, (uchar *)packet->c_ptr_safe(), > packet->length(), > log_file+dirname_length(log_file), log_pos)); > return ret; > @@ -411,7 +411,7 @@ int Binlog_transmit_delegate::after_send > > int ret= 0; > FOREACH_OBSERVER(ret, after_send_event, thd, > - (&param, packet->c_ptr(), packet->length())); > + (&param, packet->c_ptr_safe(), packet->length())); > return ret; > } > > > === modified file 'sql/sql_string.cc' > --- a/sql/sql_string.cc 2011-11-03 18:17:05 +0000 > +++ b/sql/sql_string.cc 2011-12-09 14:40:27 +0000 > @@ -80,8 +80,12 @@ bool String::real_alloc(uint32 length) > no allocation occured. > > @retval true An error occured when attempting to allocate memory. > + > + The _internal() version is just to share code between realloc() and > + realloc_with_extra(), so that we can put the NULL termination at the right > + position. > */ > -bool String::realloc(uint32 alloc_length) > +bool String::realloc_internal(uint32 alloc_length) > { > if (Alloced_length <= alloc_length) > { > @@ -109,6 +113,13 @@ bool String::realloc(uint32 alloc_length > Ptr= new_ptr; > Alloced_length= len; > } > + return FALSE; > +} > + > +bool String::realloc(uint32 alloc_length) > +{ > + if (realloc_internal(alloc_length)) > + return TRUE; > Ptr[alloc_length]=0; // This make other funcs shorter > return FALSE; > } > > === modified file 'sql/sql_string.h' > --- a/sql/sql_string.h 2011-11-03 18:17:05 +0000 > +++ b/sql/sql_string.h 2011-12-09 14:40:27 +0000 > @@ -255,12 +255,18 @@ public: > return real_alloc(arg_length); > } > bool real_alloc(uint32 arg_length); // Empties old string > +private: > + bool realloc_internal(uint32 arg_length); > +public: > bool realloc(uint32 arg_length); > bool realloc_with_extra(uint32 arg_length) > { > if (extra_alloc < 4096) > extra_alloc= extra_alloc*2+128; > - return realloc(arg_length + extra_alloc); > + if (realloc_internal(arg_length + extra_alloc)) > + return TRUE; > + Ptr[arg_length]= 0; > + return FALSE; > } > bool realloc_with_extra_if_needed(uint32 arg_length) > { > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

3 2

[Maria-developers] Thread pool (request for review)
by Vladislav Vaintroub 27 Dec '11

27 Dec '11

Hello, Worklog#246 http://askmonty.org/worklog/Server-BackLog/?tid=246 is ready to review. The worklog is implemented in lp:~maria-captains/maria/5.5-threadpool, and if someone prefers to have a one big diff, it is in the attachment(diff.diff) It would be great if Monty, Serg (optionally Kristian if there is interest) could take a some time to review it in the near future. The worklog has extensive description of the implementation, might be helpful to consult it before reviewing. The branch currently has thread-handling=pool-if-threads as default, I found it easier for testing. Based on results of benchmark (see below), it seems to be generally performing well. So maybe we should use it as default, beta version of 5.5 for better testing coverage . Maybe we can keep it as new default after testing default, if there are no complains. Benchmarking: I also run some sysbench on 3 machines (2 Linuxes - fb-maria1 with 16 cores and pitbull with 24 cores, and windows one with 8 cores but good IO), results in https://docs.google.com/spreadsheet/ccc?key=0AqvHLyfRWjmbdDY0U1MzZGlXY3VGdUx xcC1pUlBZeHc I used sysbench 0.4 to run some of "unit" oltp tests (point-select and update-nokey), as well as mere "classic" OLTP-readonly and OLTP-readwrite. All tests are run with number of concurrent clients ranging from 16 to 4096, with warm cache, with sysbench table having 1Mio records. The results seem to be quite different on all of the machines tested (the machines are very different, in terms of cores, IO etc) , yet threadpool has a positive effect on all 3 machines, and the positive effect seems to grow with the number of cores. Some words about how I ran the benchmarks 1. Benchmark client and server used different CPUs - (I used 'taskset' (Linux), or 'start /affinity' (Windows) to run benchmark client on #CPUs/4, the rest of CPUs are used by the server). On Linuxes, I used --thread_pool_size=<N> where N is number of cores I dedicate to server. 2.I used innodb_flush_log_at_trx_commit=0 and innodb_flush_method= ALL_O_DIRECT to avoid extensive fsyncing, which I think is ok for the purposes of the testing for this WL. 3. Every "write" benchmark (oltp_rw and update_nokey) started with new server (i.e I kill mysqld, remove innodb files, and restart mysqld for each test),whereas every "read" benchmark reused the same running server instance. Starting afresh with a new server on write benchmarks is done mailnly to eliminate the effects of the purge lag. I attach config file sysbench.ini is the one I used on fb-maria1 In case I missed to say it before, Happy Xmas and Happy New Year! Wlad PS. Elena, we need to have some plan on testing it. PPS. Kristian, tried to integrate the 5.5-threadpool branch into Buildbot, and stuck. Can you help?

2 2

Re: [Maria-developers] request for a feature -- LIMIT_ROWS_EXAMINED
by MARK CALLAGHAN 23 Dec '11

23 Dec '11

A warning would be excellent. An error would be a problem. Returning OK is probably not the best result when the examined limit is reached. On Wed, Dec 21, 2011 at 7:17 AM, Timour Katchaounov <timour(a)montyprogram.com> wrote: >> >> Hi! >> >>>>>>> "MARK" == MARK CALLAGHAN<mdcallag(a)gmail.com> writes: >> >> >> MARK> I want a variant of the LIMIT clause that limits the number of rows >> MARK> examined during SELECT processing. This should return OK when the >> MARK> limit is reached -- something like NESTED_LOOP_QUERY_LIMIT. While >> MARK> LIMIT can be LIMIT x and LIMIT x,y. This only supports one argument >> -- >> MARK> LIMIT_ROWS_EXAMINED x. Are you interested in implementing this >> feature >> MARK> for me? We will sponsor the work. The purpose of the feature is to >> MARK> prevent queries from taking too long when there isn't a great index >> MARK> and many rows might be filtered. >> >> I am a bit afraid of returning 'ok'. Would it not be better to return >> an error? >> >> Another option would be to generate a warning so that the application >> can know that they didn't get all rows. > > > This is what I would prefer to do: > - process the new LIMIT_ROWS_EXAMINED clause in the same way as LIMIT, > just with a different condition to end query execution, > - in addition produce a warning that not all rows were returned. > > Mark, does the above suit you? If you don't want to check for > warnings, you can skip them, however users will have the option > to detect that not all rows were returned. > > Timour > >> >> We can do this and it should be relatively easy to do. >> >> Regards, >> Monty >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~maria-developers >> Post to : maria-developers(a)lists.launchpad.net >> Unsubscribe : https://launchpad.net/~maria-developers >> More help : https://help.launchpad.net/ListHelp -- Mark Callaghan mdcallag(a)gmail.com

3 3

[Maria-developers] Fixes for PAM plugin in packages
by Kristian Nielsen 22 Dec '11

22 Dec '11

Hi Serg, As promised, here a summary of what I did to fix PAM in linux packages. I managed to fix most issues I think (except for RHEL5 as noted below), but it needs testing, which takes time due to waiting for Buildbot. - I installed libpam0g-dev in all the VMs for .deb and bintar packages. I checked on one of them that this really does cause pam modules to build and install. - I installed pam-devel on the centos5 VMs. I also tried on the rhel5, but failed. These VMs seem to require some sort of license key or something to install packages with yum, which I don't have. Maybe Daniel can sort it out (all that is needed is `sudo yum install pam-devel` in vm-rhel5-amd64-build.qcow2 and vm-rhel5-x86-build.qcow2). - I hacked .deb packaging scripts (ourdelta scripts) so that dialog.so is in the libmariadbclient16 package (it used to be in the mariadb-server-5.[23] package). I did this for 5.2 and 5.3 (not 5.1). - The dialog.so change could cause an upgrade problem I think - installing new libmariadbclient16 package on top of old mariadb-server-5.[23] package will conflict because they both contain /usr/lib/mysql/plugin/dialog.so. I tried to fix this by declaring a conflicts: dependency which should allow Apt to sort it out. I hope it works - .deb dependencies can be tricky, hopefully buildbot Upgrade tests will catch any problems. - I did not change RPM packages wrt. dialog.so. I don't know if it is needed, nor how to do it... What remains is basically testing. First checking Buildbot when it has chewed through all the builds. It might be a good idea also to test manually a couple packages that the PAM plugin is in there and actually works according to instructions on the knowledgebase. - Kristian.

2 1

[Maria-developers] request for a feature -- LIMIT_ROWS_EXAMINED
by MARK CALLAGHAN 21 Dec '11

21 Dec '11

I want a variant of the LIMIT clause that limits the number of rows examined during SELECT processing. This should return OK when the limit is reached -- something like NESTED_LOOP_QUERY_LIMIT. While LIMIT can be LIMIT x and LIMIT x,y. This only supports one argument -- LIMIT_ROWS_EXAMINED x. Are you interested in implementing this feature for me? We will sponsor the work. The purpose of the feature is to prevent queries from taking too long when there isn't a great index and many rows might be filtered. -- Mark Callaghan mdcallag(a)gmail.com

2 1

Re: [Maria-developers] request for a feature -- LIMIT_ROWS_EXAMINED
by MARK CALLAGHAN 18 Dec '11

18 Dec '11

On Sat, Dec 17, 2011 at 3:18 PM, Timour Katchaounov <timour(a)montyprogram.com> wrote: > Mark, > > >> I want a variant of the LIMIT clause that limits the number of rows >> examined during SELECT processing. This should return OK when the > > > I had a look at the code, and it seems you need a limit on the value > of Rows_examined that is currently available in the slow query log. > Did I understand correctly that this is what you need? yes > > >> limit is reached -- something like NESTED_LOOP_QUERY_LIMIT. While >> LIMIT can be LIMIT x and LIMIT x,y. This only supports one argument -- >> LIMIT_ROWS_EXAMINED x. Are you interested in implementing this feature >> for me? We will sponsor the work. The purpose of the feature is to >> prevent queries from taking too long when there isn't a great index >> and many rows might be filtered. >> > > Have you considered a system variable instead of extending the query > syntax? If you rejected this idea, why? > > It seems quite logical to me that one may want to set the same limit > for more than one query. In this case a system variable would make > life easier, because no queries need to be changed. Personally I > would prefer not to extend the query language with constructs whose > purpose is to compensate for the imperfection of the optimizer. This isn't about limits on the optimizer. We don't have a great index for some queries and never will. We prefer to stop queries early rather than have them examine 1,000,000 rows. Real users might be waiting on the query response so we really need to finish queries fast and in some cases our apps work as expected were queries allowed to terminate early. I almost always prefer to attach the semantics to the statement and that includes this case. Anyone looking at the statement via a log (slow query, general), SHOW PROCESSLIST, tcpdump should be able to understand what the statement does. Making the LIMIT part of a session variable prevents that. I also don't want to risk an additional round trip to the server and for drivers that don't support multi-statement or for limitations on the code that generates the SQL, I don't want to do: set session limit_rows_examined=1000; SELECT /* with rows examined limit */ ... ; set session limit_rows_examined=0; -- Mark Callaghan mdcallag(a)gmail.com

1 0