- developers - lists.mariadb.org

[Maria-developers] mysql_upgrade issue for libmysqld and amarok
by Kristian Nielsen 18 Mar '11

18 Mar '11

This is a follow up on a discussion I had with eean on Freenode IRC #maria. The issue is how to upgrade libmysqd from MySQL version 5.1 to version 5.5 (or more generally any major version upgrade), in particular as relates to Amarok's use of libmysqld. The procedure for upgrading a normal MySQL server is to run the mysql_upgrade command. However, this requires a running mysqld server, so it is not appropriate for libmysqld, or at least very inconvenient. We discussed this briefly at our MariaDB meeting in Lisbon, and here is what we came up with: One idea is to also build mysql_upgrade_embedded; this would be a variant of mysql_upgrade linked with libmysqld, so it does not depend on the mysqld server. Similarly, we would need mysqlcheck_embedded (mysqlcheck is called from mysql_upgrade). These two binaries could be included in the libmysqld package (they are small). The upgrade procedure for amarok would then be to stop amarok linked with 5.1, then run mysql_upgrade_embedded (from 5.5) against the amarok libmysqld data directory, then start amarok linked with 5.5. Does that sound useful? The other idea that came up was to add this as extra calls in the library. So that the amarok application could just call some mysql_upgrade() function instead of having to run an external upgrade binary. Would this be more useful for amarok? Either way, we then need to think on how to implement this change. We could offer to implement this (either solution I think) in MariaDB (a version of MariaDB that includes MySQL 5.5 is in the works). This would allow to use the above to upgrade Amarok to using MariaDB 5.5. Would this work? (I think it would even work to keep Amarok using MySQL 5.5 libmysqld, but using MariaDB 5.5 mysql_upgrade_embedded, though that is kind of a strange solution. Or we could provide the patch for MariaDB and you could patch libmysqld similarly to provide the same functionality in relevant distros.) Does this sound like a feasible way forward? One question about how Amarok uses libmysqld, which will help understand what is needed to solve the upgrade issue: Does Amarok use libmysqld built with the --with-embedded-privilege-control option, or use stored procedures, or other things that require the `mysql' database schema to be available in MySQL? If this does not make sense, the answer is probably "no" :-) The background: When a normal standalone mysql database server is installed, it is necessary to create the `mysql' database schema using mysql_create_db. This database schema contains such things as user accounts and passwords, stored procedure code, etc. Normally, libmysqld does not do any access control on accounts, and it is possible to use it without running mysql_install_db. In fact it's a bit hard to run mysql_install_db, as it has the same issue as mysql_upgrade. So I assume that Amarok does not do this and just does not use account privileges or stored procedures/functions, but just wanted to check? This affects how much mysql_upgrade_embedded has to do, as part of the upgrade is to add any missing columns in tables in the `mysql' database schema. - Kristian.

1 0

Re: [Maria-developers] [All] How are we going to handle incompatible changes in MariaDB?
by Kristian Nielsen 18 Mar '11

18 Mar '11

"Philip Stoev" <pstoev(a)askmonty.org> writes: > We should discuss how do we handle new developments in MariaDB that cause > our behavior to diverge from the one in MySQL. Since we claim to be a Yes, agree. Thank you for bringing it up! Note that this is a general MariaDB discussion, not a Monty Program specific one. So I move the thread to maria-developers@. So far, we have generally been ok at handing incompatible changes. The basic approach is to shape new features to be backwards compatible as much as possible, and to add any extra needed code to handle interoperability. For example, if an option is changed due to a feature, we should keep the old option around and emulate the bahaviour, eventually deprecating it in a future version. Another example, with replication, we check the version of the master or slave at the other end, and take care to only send what the other end understands, and likewise interpret what it sends correctly based on its version. However, this has been mostly up to the individual developer (or often reviewer), and if we can get some more structure/process to this to avoid it being forgotten, then that would be very good. > The particular situation we are facing now is the microsecond precision > patch from Serg. As part of introducing precision date and time datatypes, > he made some refactoring around the Item_* tree. > > As a result, various corner cases, e.g. invalid dates, partially valid > dates, invalid conversions between date and time, etc. are now handled > differently, with either different warnings being printed (or no warnings at > all), or with the result of the expression being different, or with a NULL > or 0000-00-00 returned. > > For example, mysql reports that DATE ( SEC_TO_TIME( 8 ) ) is equal to > "2000-00-08" whereas serg's patch returns "0000-00-00" . In other words, > MySQL chose to place the "8" literal into the days portion of the date > value, whereas Serg's patch filtered the literal somewhere along the way. > There are numerous other such examples. > > While in most cases Serg's new behavior is more intuitive and a program that > is written to conform to postgresql-level type safety will not be affected > either way, it entirely possible that a program written under mysql's > relaxed type safety rules, and with MySQL's behavior in mind, would start > behaving unexpectedly. It sounds like in this case, the changes in behaviour could be considered improvements, or even bug fixes, when seen in isolation. And the behaviour they change is not something that an application would deliberately be written to rely on. But since it is a change in behaviour, it has the potential to break applications that by accident happen to depend on such behaviour. My opinion on this is that we should do the change in a major version (eg. 5.2->5.3), but not in a minor version (eg. 5.2.5->5.2.6). And document it in release notes as an incompatible change. Basically the same way we did in MySQL (and still do, I suppose). It is ok to change behaviour in a major version, if it is necessary, and documented. I believe users anticipate such changes, and consequently are more careful about going eg. 5.1->5.5 than 5.1.x->5.1.y. We should not needlessly change things. Every behaviour change should be carefully considered. If it has the potential to seriously break major applications, we should not change it (or use deprecation, compatibility mode, etc). If the change is not necessary or useful, it should probably be left. But if the change makes sense and the new behaviour is much more reasonable, then we should be ready to make the change. We cannot afford to be stuck forever on old poor decisions. Another example is optimiser changes. In my opinion, we cannot totally avoid that improvements to the optimiser will make some corner case queries slower in some cases. The optimiser has only limited knowledge of the data and application. If, based on this knowledge, execution plan A is better than execution plan B, then we should be prepared to change (in a major version) the code to select A over B. This will invaribly cause slowdown in some cases, where the knowledge is incomplete. But it is important to be able to make improvements. But again, we need to consider each case carefully. We should not do things that are obviously stupid (eg. do not select table scan over range scan unless _really_ sure it will be better), and we should avoid breaking major applications or use patterns, even if it theoretically makes sense. So summary of my opinion: We should be very careful about incompatible changes. We should generally avoid them unless there is a very good reason for them. We should create backward compatibility where possible and where it makes sense. And if we do change behaviour, we should only do it in a major version and document it clearly in an "incompatible changes" list. And being able to continually improve the server long-term is important. > So, how we should handle this particular case, and more importantly, how do > we handle future cases like that in a stable and predictable manner? I > understand that in the early days of MySQL it would have been a no-brainer > to change everything as needed, we are by now dealing with a huge installed > base and powerful competitors, so we can not simply apply the happy hacking > approach that was used in the 1990s. Agree. No "happy hacking" approach to incompatible changes! - Kristian.

1 0

[Maria-developers] request for feedback on live schema changes patch
by Zardosht Kasheff 18 Mar '11

18 Mar '11

Hello all, In another thread, there was a discussion about what it would take to allow the storage engine to support live schema changes in mysql and mariadb. I said that I was working porting this existing feature from MySQL Cluster to MySQL 5.1.46. I would like to show the work I have done so far and solicit feedback. Attached is the diff based off of 5.1.46. Before looking at the patch, please be aware of the following things: - I am not an expert in this code. As a result, I had to use an approach of that minimizes the risk of introducing regressions. \ - My main interest is to get something safe that works and can be integrated in 5.1. As a result, kept the current mysql_alter_table functionality in place (as old_mysql_alter_table), and added MySQL Cluster's as new_mysql_alter_table. For an engine to get new_mysql_alter_table to be used, the engine must expose an alter table flag HA_GENERAL_ONLINE. - Another reason why I kept old_mysql_alter_table was that I did not want to deal with engines that implement handler::add_index and make sure that they work properly under the new API. Also, I did not want to port alter table changes related to partitioned tables. - I have run the current 5,1 mysql test suite (that is in the mysql-test directory of source tarballs), and verified that using old_mysql_alter_table and new_mysql_alter_table works fine with them. That being said, I would love to hear feedback on this patch, and would also like to understand what (if anything) can be done to get this into MariaDB 5.1. I realize that may not be possible, but if it is, I would like to try. Thanks -Zardosht

2 5

[Maria-developers] WL#182 New (by Mdcallag): Explain running statements
by worklog-noreply＠askmonty.org 13 Mar '11

13 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Explain running statements CREATION DATE..: Sun, 13 Mar 2011, 09:44 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-BackLog TASK ID........: 182 (http://askmonty.org/worklog/?tid=182) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: I want to know the query plan for running statements -- something like "SHOW PLAN FOR <connection_id>". This requires running statements to generate a string that describes their plan in case another thread needs to examine it. For me it would be good enough to know the join order and indexes. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] URLs for worklog tasks include bin name
by MARK CALLAGHAN 13 Mar '11

13 Mar '11

When I click on http://askmonty.org/worklog/?tid=180 I end up at http://askmonty.org/worklog/WorkLog-RawIdeaBin/?tid=180. Is that last URL only valid as long as tid=180 remains in the RawIdea bin? If so, then when I share it in email or include that URL elsewhere how are people to use it when tid=180 is moved from the RawIdeaBin to another bin? -- Mark Callaghan mdcallag(a)gmail.com

1 0

[Maria-developers] User Comment -> Add Comment is bugging in MariaDB worklogs
by MARK CALLAGHAN 13 Mar '11

13 Mar '11

Go to a worklog, Scroll down, Click on "Add a new comment" This takes me back to the top of the page. This is confusing and now I must scroll down again to reach the comment text box. Add text, click on OK This takes me to another page that has a text box. The text I previously entered is lost. This is a bug. If I enter text on this page and click OK it is not lost. -- Mark Callaghan mdcallag(a)gmail.com

1 1

Re: [Maria-developers] [Commits] Rev 2935: Fixed LP bugs BUG#729067/730466. in file:///home/igor/maria/maria-5.3-bug729067/
by Oleksandr Byelkin 12 Mar '11

12 Mar '11

Hi! Ok to push. 12 марта 2011, в 10:49, Igor Babaev написал(а): > At file:///home/igor/maria/maria-5.3-bug729067/ > > ------------------------------------------------------------ > revno: 2935 > revision-id: igor(a)askmonty.org-20110312084903-b9mmi2wdgbdar3tc > parent: timour(a)askmonty.org-20110308212344-7r4ue17b3rxschlj > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-bug729067 > timestamp: Sat 2011-03-12 00:49:03 -0800 > message: > Fixed LP bugs BUG#729067/730466. > Do not reset the value of the item_equal field in the Item_field object > once it has been set. > === modified file 'mysql-test/r/join.result' > --- a/mysql-test/r/join.result 2011-02-28 21:29:59 +0000 > +++ b/mysql-test/r/join.result 2011-03-12 08:49:03 +0000 > @@ -1285,3 +1285,38 @@ > 1 1 1 1 1 > 1 1 1 1 1 > drop table t1,t2,t3; > +# > +# BUG#729067/730466: unexpected 'Range checked for each record' > +# for queries with OR in WHERE clause > +# > +CREATE TABLE t1 (f1 int, f2 int) ; > +INSERT INTO t1 VALUES (4,0),(5,1); > +CREATE TABLE t2 (f1 int, f2 int, KEY (f2)) ; > +INSERT INTO t2 VALUES (5,7), (8,9); > +EXPLAIN > +SELECT * FROM t1 STRAIGHT_JOIN t2 ON t2.f1 = t1.f1 > +WHERE t1.f1<>0 OR t1.f2<>0 AND t1.f1 = t2.f2; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 Using where > +1 SIMPLE t2 ALL f2 NULL NULL NULL 2 Using where; Using join buffer (flat, BNL join) > +SELECT * FROM t1 STRAIGHT_JOIN t2 ON t2.f1 = t1.f1 > +WHERE t1.f1<>0 OR t1.f2<>0 AND t1.f1 = t2.f2; > +f1 f2 f1 f2 > +5 1 5 7 > +DROP TABLE t1,t2; > +CREATE TABLE t1(f1 int PRIMARY KEY, f2 int) ; > +INSERT INTO t1 VALUES (9,4), (10,9); > +CREATE TABLE t2(f1 int PRIMARY KEY, f2 int) ; > +INSERT INTO t2 VALUES (9,4), (10,9); > +EXPLAIN > +SELECT STRAIGHT_JOIN * FROM t1 JOIN t2 ON t2.f2 = t1.f1 > +WHERE t1.f1 IN (SELECT f1 FROM t1) AND t1.f1 = t2.f1 OR t1.f1 = 9; > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY t1 ALL PRIMARY NULL NULL NULL 2 Using where > +1 PRIMARY t2 ALL PRIMARY NULL NULL NULL 2 Using where; Using join buffer (flat, BNL join) > +2 SUBQUERY t1 index NULL PRIMARY 4 NULL 2 Using index > +SELECT STRAIGHT_JOIN * FROM t1 JOIN t2 ON t2.f2 = t1.f1 > +WHERE t1.f1 IN (SELECT f1 FROM t1) AND t1.f1 = t2.f1 OR t1.f1 = 9; > +f1 f2 f1 f2 > +9 4 10 9 > +DROP TABLE t1,t2; > > === modified file 'mysql-test/t/join.test' > --- a/mysql-test/t/join.test 2011-02-28 21:29:59 +0000 > +++ b/mysql-test/t/join.test 2011-03-12 08:49:03 +0000 > @@ -964,3 +964,36 @@ > > drop table t1,t2,t3; > > +--echo # > +--echo # BUG#729067/730466: unexpected 'Range checked for each record' > +--echo # for queries with OR in WHERE clause > +--echo # > + > +CREATE TABLE t1 (f1 int, f2 int) ; > +INSERT INTO t1 VALUES (4,0),(5,1); > + > +CREATE TABLE t2 (f1 int, f2 int, KEY (f2)) ; > +INSERT INTO t2 VALUES (5,7), (8,9); > + > +EXPLAIN > +SELECT * FROM t1 STRAIGHT_JOIN t2 ON t2.f1 = t1.f1 > + WHERE t1.f1<>0 OR t1.f2<>0 AND t1.f1 = t2.f2; > +SELECT * FROM t1 STRAIGHT_JOIN t2 ON t2.f1 = t1.f1 > + WHERE t1.f1<>0 OR t1.f2<>0 AND t1.f1 = t2.f2; > + > +DROP TABLE t1,t2; > + > +CREATE TABLE t1(f1 int PRIMARY KEY, f2 int) ; > +INSERT INTO t1 VALUES (9,4), (10,9); > + > +CREATE TABLE t2(f1 int PRIMARY KEY, f2 int) ; > +INSERT INTO t2 VALUES (9,4), (10,9); > + > +EXPLAIN > +SELECT STRAIGHT_JOIN * FROM t1 JOIN t2 ON t2.f2 = t1.f1 > + WHERE t1.f1 IN (SELECT f1 FROM t1) AND t1.f1 = t2.f1 OR t1.f1 = 9; > +SELECT STRAIGHT_JOIN * FROM t1 JOIN t2 ON t2.f2 = t1.f1 > + WHERE t1.f1 IN (SELECT f1 FROM t1) AND t1.f1 = t2.f1 OR t1.f1 = 9; > + > +DROP TABLE t1,t2; > + > > === modified file 'sql/item_cmpfunc.cc' > --- a/sql/item_cmpfunc.cc 2011-02-06 04:57:03 +0000 > +++ b/sql/item_cmpfunc.cc 2011-03-12 08:49:03 +0000 > @@ -5723,7 +5723,8 @@ > not_null_tables_cache|= tmp_table_map; > if (item->maybe_null) > maybe_null=1; > - item->item_equal= this; > + if (!item->item_equal) > + item->item_equal= this; > } > fix_length_and_dec(); > fixed= 1; > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] [Commits] Rev 2934: Fix LP BUG#719198 in file:///home/tsk/mprog/src/5.3/
by Sergey Petrunya 08 Mar '11

08 Mar '11

Hi Timour, On Tue, Mar 08, 2011 at 04:35:24PM +0200, timour(a)askmonty.org wrote: > At file:///home/tsk/mprog/src/5.3/ > > ------------------------------------------------------------ > revno: 2934 > revision-id: timour(a)askmonty.org-20110308143512-0m7uotvxp9rmegak > parent: psergey(a)askmonty.org-20110304155430-l0ad4iag9gn5zeka > committer: timour(a)askmonty.org > branch nick: 5.3 > timestamp: Tue 2011-03-08 16:35:12 +0200 > message: > Fix LP BUG#719198 > > Analysis: > The assert failed because the execution code for > partial matching is designed with the assumption that > NULLs on the left side are detected as early as possible, > and a NULL result is returned before any lookups are > performed at all. > > However, in the case of an Item_cache object on the left > side, null was not detected properly, because detection > was done via Item::is_null(), which is not implemented at > all for Item_cache, and resolved to the default Item::is_null() > which always returns FALSE. > > Solution: > Use the property Item::null_value instead of is_null(), which > is properly updated for Item_cache objects as well. > > Alternatively one could implement Item_cache:is_null(), but > its not quite clear how that will interact with the cache > update methods, so the current fix seems simpler and cleaner. Ok to push. Since it is guaranteed that cache_value() have been called for the object, it's ok to use null_value instead of is_null(). Btw, out of interest I've implemented Item_cache::is_null() and t/subselect*.test ran successfully, so it seems to be ok to have. I don't insist that it is added as part of this fix, though. > === modified file 'mysql-test/r/subselect_mat.result' > --- a/mysql-test/r/subselect_mat.result 2011-01-14 10:51:30 +0000 > +++ b/mysql-test/r/subselect_mat.result 2011-03-08 14:35:12 +0000 > @@ -1371,3 +1371,41 @@ SELECT pk FROM t1 WHERE (b,c,d) IN (SELE > pk > 2 > DROP TABLE t1, t2; > +# > +# LPBUG#719198 Ordered_key::cmp_key_with_search_key(rownum_t): Assertion `!compare_pred[i]->null_value' > +# failed with subquery on both sides of NOT IN and materialization > +# > +CREATE TABLE t1 (f1a int, f1b int) ; > +INSERT IGNORE INTO t1 VALUES (1,1),(2,2); > +CREATE TABLE t2 ( f2 int); > +INSERT IGNORE INTO t2 VALUES (3),(4); > +CREATE TABLE t3 (f3a int, f3b int); > +set session optimizer_switch='materialization=on,partial_match_rowid_merge=on,partial_match_table_scan=off'; > +EXPLAIN > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3) NOT IN (SELECT f1a FROM t1); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY t2 ALL NULL NULL NULL NULL 2 Using where > +3 SUBQUERY t1 ALL NULL NULL NULL NULL 2 > +2 SUBQUERY t3 system NULL NULL NULL NULL 0 const row not found > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3) NOT IN (SELECT f1a FROM t1); > +f2 > +EXPLAIN > +SELECT * FROM t2 WHERE (SELECT f3a, f3b FROM t3) NOT IN (SELECT f1a, f1b FROM t1); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY t2 ALL NULL NULL NULL NULL 2 Using where > +3 SUBQUERY t1 ALL NULL NULL NULL NULL 2 > +2 SUBQUERY t3 system NULL NULL NULL NULL 0 const row not found > +SELECT * FROM t2 WHERE (SELECT f3a, f3b FROM t3) NOT IN (SELECT f1a, f1b FROM t1); > +f2 > +insert into t3 values (1,1),(2,2); > +EXPLAIN > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3 where f3a > 3) NOT IN (SELECT f1a FROM t1); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY t2 ALL NULL NULL NULL NULL 2 Using where > +3 SUBQUERY t1 ALL NULL NULL NULL NULL 2 > +2 SUBQUERY t3 ALL NULL NULL NULL NULL 2 Using where > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3 where f3a > 3) NOT IN (SELECT f1a FROM t1); > +f2 > +3 > +4 > +drop table t1, t2, t3; > > === modified file 'mysql-test/t/subselect_mat.test' > --- a/mysql-test/t/subselect_mat.test 2011-01-14 10:51:30 +0000 > +++ b/mysql-test/t/subselect_mat.test 2011-03-08 14:35:12 +0000 > @@ -1011,3 +1011,31 @@ SELECT pk FROM t1 WHERE (a) IN (SELECT a > SELECT pk FROM t1 WHERE (b,c,d) IN (SELECT b,c,d FROM t2 WHERE pk > 0); > DROP TABLE t1, t2; > > +--echo # > +--echo # LPBUG#719198 Ordered_key::cmp_key_with_search_key(rownum_t): Assertion `!compare_pred[i]->null_value' > +--echo # failed with subquery on both sides of NOT IN and materialization > +--echo # > + > +CREATE TABLE t1 (f1a int, f1b int) ; > +INSERT IGNORE INTO t1 VALUES (1,1),(2,2); > +CREATE TABLE t2 ( f2 int); > +INSERT IGNORE INTO t2 VALUES (3),(4); > +CREATE TABLE t3 (f3a int, f3b int); > + > +set session optimizer_switch='materialization=on,partial_match_rowid_merge=on,partial_match_table_scan=off'; > + > +EXPLAIN > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3) NOT IN (SELECT f1a FROM t1); > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3) NOT IN (SELECT f1a FROM t1); > + > +EXPLAIN > +SELECT * FROM t2 WHERE (SELECT f3a, f3b FROM t3) NOT IN (SELECT f1a, f1b FROM t1); > +SELECT * FROM t2 WHERE (SELECT f3a, f3b FROM t3) NOT IN (SELECT f1a, f1b FROM t1); > + > +insert into t3 values (1,1),(2,2); > + > +EXPLAIN > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3 where f3a > 3) NOT IN (SELECT f1a FROM t1); > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3 where f3a > 3) NOT IN (SELECT f1a FROM t1); > + > +drop table t1, t2, t3; > > === modified file 'sql/item_subselect.cc' > --- a/sql/item_subselect.cc 2011-03-03 21:48:31 +0000 > +++ b/sql/item_subselect.cc 2011-03-08 14:35:12 +0000 > @@ -5069,7 +5069,7 @@ bool subselect_rowid_merge_engine::parti > for (uint i= test(non_null_key); i < keys_count; i++) > { > DBUG_ASSERT(merge_keys[i]->get_column_count() == 1); > - if (merge_keys[i]->get_search_key(0)->is_null()) > + if (merge_keys[i]->get_search_key(0)->null_value) > { > ++count_nulls_in_search_key; > bitmap_set_bit(&matching_outer_cols, merge_keys[i]->get_keyid()); > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MWL#90 combined patch for review
by Sergey Petrunya 08 Mar '11

08 Mar '11

Hi Monty, Please find attached the complete patch of MWL#90 for review. The tree with the code is on launchpad/buildbot: https://code.launchpad.net/~maria-captains/maria/5.3-subqueries-mwl90 http://buildbot.askmonty.org/buildbot/grid?branch=5.3-subqueries-mwl90 and it has no failures other than those also found in mainline 5.3. The merge from 5.3-main was about a week ago. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 5

[Maria-developers] Report from OpenSourceDays2011
by Kristian Nielsen 08 Mar '11

08 Mar '11

Hi, On Saturday, March 5, I gave a talk at the OpenSourceDays 2011 conference in Copenhagen, Denmark: http://opensourcedays.org/ My talk was titled "Latest news from the MariaDB (and MySQL) community". I put the slides from the talk up on my web server here: http://knielsen-hq.org/maria/osd2011.pdf This year the OpenSourceDays conference was reduced somewhat from what it usually is, only one day and I heard something like 400 tickes available (which were sold out rather quickly). Still, I had a good bunch of people at the talk, maybe 30 or so. There were many questions, people seemed interested, so I think it went well. I also talked to some people outside of the talk, again there seems to be a lot of interest in what is happening around MySQL generally and in MariaDB specifically. I think generally people are pleasantly surprised to learn the depth and dedication of the MariaDB effort, that we really are a major effort and a viable alternative. There was one good question at the talk about if there were reference customer statements or so to show that companies are deploying MariaDB with success at large scale; I guess this will be useful for people that want to sell the idea of using MariaDB at their job or institution or whatever. I did not have a very good answer to this, except that I knew there were some. I guess we already have this on one of the web sites, or if not it would be worth it to collect the different press releases and such as we get them, I think. The night before the conference there was a speaker's dinner, and I had a great talk with Magnus from Postgresql. I got the impression that things are also really moving in the Postgres space, and it seems to be picking up a lot of interest and development effort. I think that is great, I am starting to be hopeful that we are now seeing the open source databases really take off as a community effort, as opposed to just a project of whatever "acquirer of the year". That's it :-) - Kristian.

2 1

[Maria-developers] Pbxt loses rows
by Michael Widenius 08 Mar '11

08 Mar '11

Hi! This is a bug that has been around a long time but not yet fixed (at least not in 1.0.11 which is in the MariaDB 5.1 tree). Once in a while when running the pbxt.insert test, pbxt fails to find all rows in the table: pbxt.insert [ fail ] Test ended at 2011-02-25 16:44:03 CURRENT_TEST: pbxt.insert --- /home/my/maria-5.1-merge/mysql-test/suite/pbxt/r/insert.result 2011-02-10 22:57:21.000000000 +0200 +++ /home/my/maria-5.1-merge/mysql-test/suite/pbxt/r/insert.reject 2011-02-25 16:44:03.030856755 +0200 @@ -298,7 +298,7 @@ insert into t2 select t1.* from t1, t2 t, t3 where t1.id1 = t.id2 and t.id2 = t3.id3; select count(*) from t2; count(*) -25500 +25495 drop table t1,t2,t3; create table t1 (a int, b int); insert into t1 (a,b) values (a,b); Rerunning the test usually works It would be really good to get this fixes ASAP as it's not good if a storage engines looses rows randomly... The above happend in my latest 5.1 tree on Linux. Regards, Monty

2 2

[Maria-developers] adutko-centos-amd64
by Adam M. Dutko 07 Mar '11

07 Mar '11

The machine running the buildbot adutko-centos-amd64 experienced a hardware failure last night around 1AM EST (UTC 0600). I believe it is the motherboard and due to funding constraints I am not certain when it will be brought back online. I will make an attempt to migrate the buildbot when I have time over the next few days. Sorry for the interruption. If someone has a spare board that supports AMD Phenom X3/X4 processors please write me off list. Thanks.

1 0

Re: [Maria-developers] RedHat Buildslaves
by Kristian Nielsen 07 Mar '11

07 Mar '11

Daniel Bartholomew <dbart(a)montyprogram.com> writes: > Daniel> I'm creating the build-slaves based on the KVM setup > Daniel> instructions you have here: > Daniel> > Daniel> http://askmonty.org/wiki/BuildBot:vm-setup#Centos_5_i386 > I now have a rhel6-amd64 vm setup on terrier with base, serial, build, > and install vms. The steps I went through to set it up are here: > > http://kb.askmonty.org/v/buildbot-setup-for-red-hat > I'm not sure what the next step should be. Probably copying over the > correct OurDelta file. Can you copy one out After that, do you want to > take a look at the VMs and see if they're ok for building and install > testing? I did this, and set up new builders in Buildbot. They fail, but the failure looks related to the .spec file and build scripts, rather than the VMs. So I would say the VMs are now ok, and the next step is to find someone to modify the ourdelta build scripts and/or spec files to work with RHEL. > Once this VM is certifed as good to use as a buildbot buildslave I'll > move on to rhel4 and rhel5. As I said, the VMs look reasonable. So if you want, you can start doing the rhel4/5 ones the same way, or if you prefer, you can wait until the .spec files are done for rhel6, in case that work turns up something else that is needed. > One thing I just realized is that > the /kvm/MySQL-OurDelta-shared-5.0.87.d10-65.el5.i386.rpm file that I > placed at /srv/shared/yum/CentOS/5/i386/RPMS/ is not the right > architecture for an amd64 system, and it's probably not at the right > path, either (what with CentOS being in the path). Yeah. I grabbed the amd64 one and copied it in there with my best guess for the path (and also put it on terrier:/kvm/). The thing is, it probably needs more than this, someone who understands (or takes the time to learn) .rpm will have to develop the right solution. The full story is like this: The Redhat rpms (original, not ours) put the `mysql` binary in the same package as libmysqlclient. Then it has other packages depend on this package to get libmysqlclient.so.15 (old version of client library). So when we replace the package with a new version of the mysql binary (from MariaDB), we also have to provide a copy of the _old_ client library (as well as a copy of the new one of course, libmysqlclient.so.16). (In contrast, Debian does this correctly, placing the client library in package libmysqlclient15, which can co-exist with new MySQL/MariaDB versions). The question is how to get this old library? The MySQL build team told me they grab it from an old binary package from MySQL 4.1 (or whatever), so the OurDelta stuff does something similar. Admittedly not a nice solution. Maybe it is not needed for RHEL 6, if it does not have dependencies on libmysqlclient15. Or maybe we need to find an old RHEL MySQL .rpm to use instead. I do not know rpm well enough to have a meaningful opinion. > I built this directly on terrier, since my local desktop can only run > with "-cpu qemu32". I tried to build an x86 version on terrier, but I > kept getting stuck in a VM reboot loop. So I'll have to upload the Yeah, there is something wierd/different with the qemu-kvm I built on terrier to fix another problem. But it works with `-cpu qemu64` (also for a 32-bit VM). Doing it on your local machine should be fine as well. Thanks, - Kristian.

1 0

[Maria-developers] Igor please review: Rev 2933: BUG#727667 Wrong result with OR + NOT NULL in maria-5.3 in file:///home/psergey/dev2/5.3/]
by Sergey Petrunya 05 Mar '11

05 Mar '11

Hello Igor, Could you please review the below? The bug page https://bugs.launchpad.net/maria/+bug/727667 has verbose explanations of how we ended up having this bug, as well as justifications for fixing it in this particular way. ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- From: Sergey Petrunya <psergey(a)askmonty.org> To: commits(a)mariadb.org X-Mailer: mail (GNU Mailutils 1.2) Date: Sat, 5 Mar 2011 12:56:26 +0300 (MSK) Subject: [Commits] Rev 2933: BUG#727667 Wrong result with OR + NOT NULL in maria-5.3 in file:///home/psergey/dev2/5.3/ At file:///home/psergey/dev2/5.3/ ------------------------------------------------------------ revno: 2933 revision-id: psergey(a)askmonty.org-20110305095622-wzlnwr9srfbsu3pv parent: psergey(a)askmonty.org-20110304091446-qmsezak3c0cfm9b1 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: 5.3 timestamp: Sat 2011-03-05 12:56:22 +0300 message: BUG#727667 Wrong result with OR + NOT NULL in maria-5.3 - put the code that sets HA_NULL_PART bit back - Fix test_if_ref/part_of_refkey() so that = NULL-ability of lookup columns does not prevent the equality from being removed (we now have early/late NULLs filtering which will filter out NULL values) = equality is not removed if it is ref_or_null access, and the value of the lookup column can alternate between the lookup value and NULL. === modified file 'mysql-test/r/null.result' --- a/mysql-test/r/null.result 2009-12-15 07:16:46 +0000 +++ b/mysql-test/r/null.result 2011-03-05 09:56:22 +0000 @@ -170,7 +170,7 @@ insert into t1 values(null); explain select * from t1 where i=2 or i is null; id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null i i 5 const 9 Using index +1 SIMPLE t1 ref_or_null i i 5 const 9 Using where; Using index select count(*) from t1 where i=2 or i is null; count(*) 10 === modified file 'mysql-test/r/null_key.result' --- a/mysql-test/r/null_key.result 2010-10-06 20:27:12 +0000 +++ b/mysql-test/r/null_key.result 2011-03-05 09:56:22 +0000 @@ -21,10 +21,10 @@ 1 SIMPLE t1 range a,b a 9 NULL 3 Using where; Using index explain select * from t1 where (a is null or a = 7) and b=7; id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null a,b a 9 const,const 2 Using index +1 SIMPLE t1 ref_or_null a,b a 9 const,const 2 Using where; Using index explain select * from t1 where (a is null or a = 7) and b=7 order by a; id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null a,b a 9 const,const 2 Using index; Using filesort +1 SIMPLE t1 ref_or_null a,b a 9 const,const 2 Using where; Using index; Using filesort explain select * from t1 where (a is null and b>a) or a is null and b=7 limit 2; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t1 ref a,b a 5 const 3 Using where; Using index @@ -151,7 +151,7 @@ insert into t1 values (7,null), (8,null), (8,7); explain select * from t1 where a = 7 and (b=7 or b is null); id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null a,b a 10 const,const 2 Using index +1 SIMPLE t1 ref_or_null a,b a 10 const,const 2 Using where; Using index select * from t1 where a = 7 and (b=7 or b is null); a b 7 7 @@ -166,7 +166,7 @@ NULL 7 explain select * from t1 where (a = 7 or a is null) and (a = 7 or a is null); id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null a a 5 const 5 Using index +1 SIMPLE t1 ref_or_null a a 5 const 5 Using where; Using index select * from t1 where (a = 7 or a is null) and (a = 7 or a is null); a b 7 NULL @@ -192,7 +192,7 @@ explain select * from t2,t1 where t1.a=t2.a and (b= 7 or b is null); id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t2 ALL NULL NULL NULL NULL 2 Using where -1 SIMPLE t1 ref_or_null a a 10 test.t2.a,const 4 Using index +1 SIMPLE t1 ref_or_null a a 10 test.t2.a,const 4 Using where; Using index select * from t2,t1 where t1.a=t2.a and (b= 7 or b is null); a a b 7 7 7 @@ -202,7 +202,7 @@ explain select * from t2,t1 where (t1.a=t2.a or t1.a is null) and b= 7; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t2 ALL NULL NULL NULL NULL 2 -1 SIMPLE t1 ref_or_null a a 10 test.t2.a,const 4 Using index +1 SIMPLE t1 ref_or_null a a 10 test.t2.a,const 4 Using where; Using index select * from t2,t1 where (t1.a=t2.a or t1.a is null) and b= 7; a a b 7 7 7 @@ -226,7 +226,7 @@ explain select * from t2,t1 where t1.a=t2.a or t1.a is null; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t2 ALL NULL NULL NULL NULL 4 -1 SIMPLE t1 ref_or_null a a 5 test.t2.a 4 Using index +1 SIMPLE t1 ref_or_null a a 5 test.t2.a 4 Using where; Using index explain select * from t2,t1 where t1.a<=>t2.a or (t1.a is null and t1.b <> 9); id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t2 ALL NULL NULL NULL NULL 4 @@ -447,3 +447,15 @@ 3 12 0 12 drop table t1, t2; End of 5.0 tests +# +# BUG#727667 Wrong result with OR + NOT NULL in maria-5.3 +# +CREATE TABLE t1 ( +f3 int(11), +f10 varchar(1), +KEY (f3) +); +INSERT INTO t1 VALUES ('9','k'),(NULL,'r'); +SELECT * FROM t1 WHERE (f3 = 83) OR (f10 = 'z' AND f3 IS NULL); +f3 f10 +DROP TABLE t1; === modified file 'mysql-test/r/order_by.result' --- a/mysql-test/r/order_by.result 2011-01-27 04:45:23 +0000 +++ b/mysql-test/r/order_by.result 2011-03-05 09:56:22 +0000 @@ -643,7 +643,7 @@ insert into t1 values (2, 1), (1, 1), (4, NULL), (3, NULL), (6, 2), (5, 2); explain select * from t1 where b=1 or b is null order by a; id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null b b 5 const 3 Using filesort +1 SIMPLE t1 ref_or_null b b 5 const 3 Using index condition; Using filesort select * from t1 where b=1 or b is null order by a; a b 1 1 @@ -652,7 +652,7 @@ 4 NULL explain select * from t1 where b=2 or b is null order by a; id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null b b 5 const 4 Using filesort +1 SIMPLE t1 ref_or_null b b 5 const 4 Using index condition; Using filesort select * from t1 where b=2 or b is null order by a; a b 3 NULL === modified file 'mysql-test/r/subselect.result' --- a/mysql-test/r/subselect.result 2011-02-22 09:15:47 +0000 +++ b/mysql-test/r/subselect.result 2011-03-05 09:56:22 +0000 @@ -906,7 +906,7 @@ explain extended SELECT t1.a, t1.a in (select t2.a from t2,t3 where t3.a=t2.a) FROM t1; id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 index NULL PRIMARY 4 NULL 4 100.00 Using index -2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using index +2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using where; Using index 2 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 3 100.00 Using where; Using join buffer (flat, BNL join) Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,<expr_cache><`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`a`,<exists>(select 1 from `test`.`t2` join `test`.`t3` where ((`test`.`t3`.`a` = `test`.`t2`.`a`) and ((<cache>(`test`.`t1`.`a`) = `test`.`t2`.`a`) or isnull(`test`.`t2`.`a`))) having <is_not_null_test>(`test`.`t2`.`a`)))) AS `t1.a in (select t2.a from t2,t3 where t3.a=t2.a)` from `test`.`t1` === modified file 'mysql-test/r/subselect_no_mat.result' --- a/mysql-test/r/subselect_no_mat.result 2011-02-22 09:15:47 +0000 +++ b/mysql-test/r/subselect_no_mat.result 2011-03-05 09:56:22 +0000 @@ -910,7 +910,7 @@ explain extended SELECT t1.a, t1.a in (select t2.a from t2,t3 where t3.a=t2.a) FROM t1; id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 index NULL PRIMARY 4 NULL 4 100.00 Using index -2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using index +2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using where; Using index 2 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 3 100.00 Using where; Using join buffer (flat, BNL join) Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,<expr_cache><`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`a`,<exists>(select 1 from `test`.`t2` join `test`.`t3` where ((`test`.`t3`.`a` = `test`.`t2`.`a`) and ((<cache>(`test`.`t1`.`a`) = `test`.`t2`.`a`) or isnull(`test`.`t2`.`a`))) having <is_not_null_test>(`test`.`t2`.`a`)))) AS `t1.a in (select t2.a from t2,t3 where t3.a=t2.a)` from `test`.`t1` === modified file 'mysql-test/r/subselect_no_opts.result' --- a/mysql-test/r/subselect_no_opts.result 2011-02-22 09:15:47 +0000 +++ b/mysql-test/r/subselect_no_opts.result 2011-03-05 09:56:22 +0000 @@ -907,7 +907,7 @@ explain extended SELECT t1.a, t1.a in (select t2.a from t2,t3 where t3.a=t2.a) FROM t1; id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 index NULL PRIMARY 4 NULL 4 100.00 Using index -2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using index +2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using where; Using index 2 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 3 100.00 Using where; Using join buffer (flat, BNL join) Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,<expr_cache><`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`a`,<exists>(select 1 from `test`.`t2` join `test`.`t3` where ((`test`.`t3`.`a` = `test`.`t2`.`a`) and ((<cache>(`test`.`t1`.`a`) = `test`.`t2`.`a`) or isnull(`test`.`t2`.`a`))) having <is_not_null_test>(`test`.`t2`.`a`)))) AS `t1.a in (select t2.a from t2,t3 where t3.a=t2.a)` from `test`.`t1` === modified file 'mysql-test/r/subselect_no_semijoin.result' --- a/mysql-test/r/subselect_no_semijoin.result 2011-02-22 09:15:47 +0000 +++ b/mysql-test/r/subselect_no_semijoin.result 2011-03-05 09:56:22 +0000 @@ -907,7 +907,7 @@ explain extended SELECT t1.a, t1.a in (select t2.a from t2,t3 where t3.a=t2.a) FROM t1; id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 index NULL PRIMARY 4 NULL 4 100.00 Using index -2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using index +2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using where; Using index 2 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 3 100.00 Using where; Using join buffer (flat, BNL join) Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,<expr_cache><`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`a`,<exists>(select 1 from `test`.`t2` join `test`.`t3` where ((`test`.`t3`.`a` = `test`.`t2`.`a`) and ((<cache>(`test`.`t1`.`a`) = `test`.`t2`.`a`) or isnull(`test`.`t2`.`a`))) having <is_not_null_test>(`test`.`t2`.`a`)))) AS `t1.a in (select t2.a from t2,t3 where t3.a=t2.a)` from `test`.`t1` === modified file 'mysql-test/t/null_key.test' --- a/mysql-test/t/null_key.test 2008-03-03 17:35:44 +0000 +++ b/mysql-test/t/null_key.test 2011-03-05 09:56:22 +0000 @@ -263,3 +263,16 @@ drop table t1, t2; -- echo End of 5.0 tests +--echo # +--echo # BUG#727667 Wrong result with OR + NOT NULL in maria-5.3 +--echo # + +CREATE TABLE t1 ( + f3 int(11), + f10 varchar(1), + KEY (f3) +); +INSERT INTO t1 VALUES ('9','k'),(NULL,'r'); +SELECT * FROM t1 WHERE (f3 = 83) OR (f10 = 'z' AND f3 IS NULL); +DROP TABLE t1; + === modified file 'sql/sql_select.cc' --- a/sql/sql_select.cc 2011-02-28 21:29:59 +0000 +++ b/sql/sql_select.cc 2011-03-05 09:56:22 +0000 @@ -6564,10 +6564,12 @@ j->ref.has_record= FALSE; j->ref.null_rejecting= 0; j->ref.disable_cache= FALSE; + j->ref.null_ref_part= (uint)-1; keyuse=org_keyuse; store_key **ref_key= j->ref.key_copy; uchar *key_buff=j->ref.key_buff, *null_ref_key= 0; + uint null_ref_part= (uint)-1; bool keyuse_uses_no_tables= TRUE; if (ftkey) { @@ -6621,7 +6623,10 @@ instead of JT_REF_OR_NULL in case if field can't be null */ if ((keyuse->optimize & KEY_OPTIMIZE_REF_OR_NULL) && maybe_null) + { null_ref_key= key_buff; + null_ref_part= i; + } key_buff+= keyinfo->key_part[i].store_length; } } /* not ftkey */ @@ -6636,6 +6641,7 @@ /* Must read with repeat */ j->type= null_ref_key ? JT_REF_OR_NULL : JT_REF; j->ref.null_ref_key= null_ref_key; + j->ref.null_ref_part= null_ref_part; } else if (keyuse_uses_no_tables) { @@ -15287,8 +15293,12 @@ *****************************************************************************/ /** + Check if "left_item=right_item" equality is guaranteed to be true by use of + [eq]ref access on left_item->field->table. + @return - 1 if right_item is used removable reference key on left_item + TRUE if right_item is used removable reference key on left_item + FALSE otherwise */ bool test_if_ref(Item *root_cond, Item_field *left_item,Item *right_item) @@ -15301,7 +15311,11 @@ (!join_tab->first_inner || *join_tab->first_inner->on_expr_ref == root_cond)) { - // Cond guards + /* + If ref access uses "Full scan on NULL key" (i.e. it actually alternates + between ref access and full table scan), then no equality can be + guaranteed to be true. + */ for (uint i = 0; i < join_tab->ref.key_parts; i++) { if (join_tab->ref.cond_guards[i]) @@ -15309,7 +15323,7 @@ return FALSE; } } - // + Item *ref_item=part_of_refkey(field->table,field); if (ref_item && ref_item->eq(right_item,1)) { @@ -15611,6 +15625,29 @@ } +/* + @brief + + Check if + - @table uses "ref"-like access + - it is based on "@field=certain_item" equality + - the equality will be true for any record returned by the access method + and return the certain_item if yes. + + @detail + + Equality won't necessarily hold if: + - the used index covers only part of the @field. + Suppose, we have a CHAR(5) field and INDEX(field(3)). if you make a lookup + for 'abc', you will get both record with 'abc' and with 'abcde'. + - The type of access is actually ref_or_null, and so @field can be either + a value or NULL. + + @return + Item that the field will be equal to + NULL if no such item +*/ + static Item * part_of_refkey(TABLE *table,Field *field) { @@ -15619,17 +15656,30 @@ return (Item*) 0; // field from outer non-select (UPDATE,...) uint ref_parts= join_tab->ref.key_parts; - if (ref_parts) + if (ref_parts) /* if it's ref/eq_ref/ref_or_null */ { - uint key= join_tab->ref.key; KEY *key_info= join_tab->get_keyinfo_by_key_no(key); KEY_PART_INFO *key_part= key_info->key_part; for (uint part=0 ; part < ref_parts ; part++,key_part++) - if (field->eq(key_part->field) && - !(key_part->key_part_flag & (HA_PART_KEY_SEG | HA_NULL_PART))) - return join_tab->ref.items[part]; + { + if (field->eq(key_part->field)) + { + /* + Found the field in the key. Check that + 1. ref_or_null doesn't alternate this component between a value and + a NULL + 2. index fully covers the key + */ + if (part != join_tab->ref.null_ref_part && // (1) + !(key_part->key_part_flag & HA_PART_KEY_SEG)) // (2) + { + return join_tab->ref.items[part]; + } + break; + } + } } return (Item*) 0; } === modified file 'sql/sql_select.h' --- a/sql/sql_select.h 2011-02-24 06:23:12 +0000 +++ b/sql/sql_select.h 2011-03-05 09:56:22 +0000 @@ -109,8 +109,15 @@ */ key_part_map null_rejecting; table_map depend_map; ///< Table depends on these tables. + /* null byte position in the key_buf. Used for REF_OR_NULL optimization */ uchar *null_ref_key; + /* + ref_or_null optimization: number of key part can be NULL (there's only one) + uint(-1) if no part can be NULL. + */ + uint null_ref_part; + /* The number of times the record associated with this key was used in the join. === modified file 'sql/table.cc' --- a/sql/table.cc 2011-02-22 09:15:47 +0000 +++ b/sql/table.cc 2011-03-05 09:56:22 +0000 @@ -1581,6 +1581,8 @@ #endif key_part->key_part_flag|= HA_PART_KEY_SEG; } + if (field->real_maybe_null()) + key_part->key_part_flag|= HA_NULL_PART; /* Sometimes we can compare key parts for equality with memcmp. But not always. _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB 5.2.5
by Daniel Bartholomew 03 Mar '11

03 Mar '11

Here are the download, release notes, and changelog pages for 5.2.5: * http://downloads.askmonty.org/mariadb/5.2.5 * http://kb.askmonty.org/v/mariadb-525-release-notes * http://kb.askmonty.org/v/mariadb-525-changelog If any of you have a moment, please check over the above pages to see if there are any mistakes or omissions and let me know (or jump in and fix if you have editing rights). Community team: I tested a sample of the mirrors and they all had the files so please announce the release on the announce list, blogs, and so on. Thanks. -- Daniel Bartholomew MariaDB - http://mariadb.org Monty Program - http://montyprogram.com AskMonty Knowledgebase - http://kb.askmonty.org

2 2

Re: [Maria-developers] [Commits] Rev 2929: Fix LP BUG#718763 in file:///home/tsk/mprog/src/5.3/
by Oleksandr Byelkin 03 Mar '11

03 Mar '11

Hi! Everything looks OK, except one small style error. Also it would be nice to put somewhere 'TODO:' to bring in order 'uncachable' flags. On 03.03.2011 14:07, timour(a)askmonty.org wrote: > At file:///home/tsk/mprog/src/5.3/ > > ------------------------------------------------------------ > revno: 2929 > revision-id: timour(a)askmonty.org-20110303120726-la5o40rldim2t93y > parent: psergey(a)askmonty.org-20110301072222-1rmapkelx9l1kw8u > committer: timour(a)askmonty.org > branch nick: 5.3 > timestamp: Thu 2011-03-03 14:07:26 +0200 > message: > Fix LP BUG#718763 > > Analysis: > The reason for the crash was that the inner subquery was executed > via a scan on a final temporary table applied after all other > operations. This final operation is implemented by changing the > contents of the JOIN object of the subquery to represent a table > scan over the temp table. At the same time query optimization of > the outer subquery required evaluation of the inner subquery, which > happened before the actual EXPLAIN. The evaluation left the JOIN > object of the inner subquery in the changed state, where it represented > a table scan over a temp table, and EXPLAIN crashed because the temp > table is not associated with any table reference (TABLE_LIST object). > The reason the JOIN was not restored was because its saving/restoration > was controlled by the join->select_lex->uncacheable flag, which was > not set in the case of materialization. > > Solution: > In the methods Item_in_subselect::[single | row]_value_transformer() set: > select_lex->uncacheable|= UNCACHEABLE_EXPLAIN; > In addition, for symmetry, change: > master_unit->uncacheable|= UNCACHEABLE_EXPLAIN; > instead of UNCACHEABLE_DEPENDENT because if a subquery was not > dependent initially, the changed methods do not change this > fact. The subquery may later become correlated if it is transformed > to an EXISTS query, but it may stay uncorrelated if executed via > materialization. [skip] > > > @@ -1742,7 +1752,15 @@ Item_in_subselect::row_value_transformer > optimizer->keep_top_level_cache(); > > thd->lex->current_select= current; > - master_unit->uncacheable|= UNCACHEABLE_DEPENDENT; > + /* > + The uncacheable property controls a number of actions, e.g. whether to > + save/restore (via init_save_join_tab/restore_tmp) the original JOIN for > + plans with a temp table where the original JOIN was overriden by > + make_simple_join. The UNCACHEABLE_EXPLAIN is ignored by EXPLAIN, thus > + non-correlated subqueries will not appear as such to EXPLAIN. > + */ > + master_unit->uncacheable|=UNCACHEABLE_EXPLAIN; > + select_lex->uncacheable|= UNCACHEABLE_EXPLAIN; > > if (!abort_on_null&& left_expr->maybe_null&& !pushed_cond_guards) > { put space (' ') after '=' [skip] > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] MariaDB 5.2.5 packages ready for release
by Kristian Nielsen 02 Mar '11

02 Mar '11

From: Kristian Nielsen <knielsen(a)knielsen-hq.org> Subject: Release of MariaDB 5.1.55 To: Daniel Bartholomew <dbart(a)askmonty.org> Cc: Internal Dev List <dev(a)askmonty.org> Date: Mon, 28 Feb 2011 12:40:13 +0100 Hi Daniel, The packages for release of MariaDB 5.2.5 are ready. The main changes I know of are the same changes as in the MariaDB 5.1.55 release: merges of MySQL 5.1.55 and of XtraDB from Percona Server 5.1.54-12.5. This version was released off the main "5.2" branch: lp:maria/5.2 I added the mariadb-5.2.5 tag. The Buildbot build number, for locating packages in /archive/pack, is 829. As for the 5.1.55 release, I prepared apt repositories and signed .rpm files. The repositories are here: hasky:/home/knielsen/repo The signed .rpms are here: hasky:/home/knielsen/build-829 - Kristian.

1 0

Re: [Maria-developers] Fwd: [Commits] Rev 2901: BNLH algorithm always used a full table scan over the joined table in file:///home/igor/maria/maria-5.3-mwl128-hashrange/
by Sergey Petrunya 02 Mar '11

02 Mar '11

Hello Igor, Here's the promised proof of deadcode: On Fri, Feb 11, 2011 at 08:49:37PM -0800, Igor Babaev wrote: > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2011-02-06 04:57:03 +0000 > +++ b/sql/sql_select.cc 2011-02-12 04:41:22 +0000 > @@ -8419,6 +8438,11 @@ > sort_by_tab->type= JT_ALL; > sort_by_tab->read_first_record= join_init_read_record; > } > + else if (sort_by_tab->type == JT_HASH_NEXT) > + { > + sort_by_tab->type= JT_HASH; > + sort_by_tab->read_first_record= join_init_read_record; > + } > } > break; > } Statement: the above else-if branch can never be taken, i.e. equality sort_by_tab->type == JT_HASH_NEXT (1) can never be true. Proof: First, let's see what JOIN_TABs can be in sort_by_tab. sort_by_tab is assigned in this statement: JOIN_TAB *sort_by_tab= join->group && join->simple_group && join->group_list ? join->join_tab+join->const_tables : join->get_sort_by_join_tab(); i.e. possible values are 1. join->join_tab+join->const_tables 2. join->get_sort_by_join_tab() get_sort_by_join_tab is defined in sql_select.h as: JOIN_TAB *get_sort_by_join_tab() { return (need_tmp || !sort_by_table || skip_sort_order || ((group || tmp_table_param.sum_func_count) && !group_list)) ? NULL : join_tab+const_tables; } i.e. it can return: 1. NULL (which is not interesting) 2. join->join_tab + const_tables. this draws us to conclusion that sort_by_tab can be - NULL - join->join_tab + const_tables. so equality (1) can be rewritten as join->join_tab[const_tables].type == JT_HASH_NEXT (2) Now, let's explore the question, what join_tab elements can have type==JT_HASH_NEXT. The value JT_HASH_NEXT is assigned in only one place: sql_select.cc|8399| tab->type= tab->type == JT_ALL ? JT_NEXT : JT_HASH_NEXT; this line is inside the case block of code: break; case JT_ALL: case JT_HASH: .... tab->type= tab->type == JT_ALL ? JT_NEXT : JT_HASH_NEXT; which means that tab->type can become JT_HASH_NEXT only if it was JT_HASH before. Let's see when a join_tab can get type==JT_HASH. There's only one assignment that can assign type=JT_HASH. It is at sql_select.cc:8279 : if (tab->cache && tab->cache->get_join_alg() == JOIN_CACHE::BNLH_JOIN_ALG) tab->type= JT_HASH; i.e. tab->type can become JT_HASH only if tab->cache!=NULL. tab->cache can be set to non-NULL value only inside check_join_cache_usage(). However, the first lines in check_join_cache_usage() have this code: uint i= tab - join->join_tab; join->return_tab= 0; if (cache_level == 0 || i == join->const_tables || !prev_tab) return 0; from which we can conclude that join->join_tab[join->const_tables].cache == NULL will always hold. Going back on our chain of reasoning, we get that join->join_tab[join->const_tables].cache == NULL at all times=> join->join_tab[join->const_tables].type != JT_HASH at all times => join->join_tab[join->const_tables].type != JT_HASH_NEXT at all times => equality (2) is false always => original equality (1) is false always, q.e.d. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Igor please review: BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) in file:///home/psergey/dev2/5.3/]
by Sergey Petrunya 02 Mar '11

02 Mar '11

On Wed, Mar 02, 2011 at 08:04:39AM -0800, Igor Babaev wrote: > > +SELECT alias2.f1 AS field1 , alias1.f2 AS field2 > > +FROM t1 AS alias1 > > +LEFT OUTER JOIN t1 AS alias2 ON alias1.f2 = alias2.f5 > > +WHERE alias2.f3 > 'k' > > +AND alias2.f3 <= 'z' > > +OR alias2.f4 IN ('o', 'y') > > +AND alias2.f4 >= 'g' AND alias2.f4 < 'z'; > > +field1 field2 > > +Warnings: > > +Warning 1292 Truncated incorrect DOUBLE value: 'k' > > Why do we need comparison of a string with an integer that generates the > above warning? Can we do without it? Why not to use char(4) instead > if integer? Got rid of string vs integer comparison, as well as simplified the testcase a bit (further simplification is difficult). Please find the result below: ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- From: Sergey Petrunya <psergey(a)askmonty.org> To: commits(a)mariadb.org X-Mailer: mail (GNU Mailutils 1.2) Date: Wed, 2 Mar 2011 17:06:15 +0300 (MSK) Subject: [Commits] Rev 2929: BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) in file:///home/psergey/dev2/5.3/ At file:///home/psergey/dev2/5.3/ ------------------------------------------------------------ revno: 2929 revision-id: psergey(a)askmonty.org-20110302140613-4qt8xvdvxs4t4m98 parent: psergey(a)askmonty.org-20110301072222-1rmapkelx9l1kw8u committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: 5.3 timestamp: Wed 2011-03-02 17:06:13 +0300 message: BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) - Add debugging aid which maps MRR's range_ids (which are pointers so are different on every execution and hard to track) to range numbers (which allow much easier debugging). The mapper is not compiled unless the developer puts a #define into code to request it. - Fix the bug: the problem was that Mrr_ordered_index_reader's interrupt_read() and resume_read() would save and restore 1) index tuple 2) the rowid (as bytes returned by handler->position()). Clustered primary key columns were not saved/restored. They are not explicitly present in the index tuple (i.e. table->key_info[secondary_key].key_parts doesn't list them), but they are actually there, in particular table->field[clustered_primary_key_member].part_of_key(secondary_key) == 1. Index condition pushdown code [correctly] uses the latter as inidication that pushed index condition can refer to clustered PK members. The fix was to make interrupt_read()/resume_read() to save/restore clustered primary key members as well, so that we get correct values for them when evaluating pushed index condition. [Second attempt, make the code compile without the debugging aid] === modified file 'mysql-test/r/innodb_mrr.result' --- a/mysql-test/r/innodb_mrr.result 2011-01-17 21:26:04 +0000 +++ b/mysql-test/r/innodb_mrr.result 2011-03-02 14:06:13 +0000 @@ -736,3 +736,31 @@ c1 c2 c3 08:29:45 NULL 2009-02-01 drop table `t1`; +# +# BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) +# +set @_save_join_cache_level= @@join_cache_level; +set join_cache_level = 6; +CREATE TABLE t1 ( +f1 int(11), f2 int(11), f3 varchar(1), f4 varchar(1), +PRIMARY KEY (f1), +KEY (f3), +KEY (f2) +) ENGINE=InnoDB; +INSERT INTO t1 VALUES ('11','8','f','f'),('12','5','v','v'),('13','8','s','s'), +('14','8','a','a'),('15','6','p','p'),('16','7','z','z'),('17','2','a','a'), +('18','5','h','h'),('19','7','h','h'),('20','2','v','v'),('21','9','v','v'), +('22','142','b','b'),('23','3','y','y'),('24','0','v','v'),('25','3','m','m'), +('26','5','z','z'),('27','9','n','n'),('28','1','d','d'),('29','107','a','a'); +select count(*) from ( +SELECT alias1.f2 +FROM +t1 AS alias1 JOIN ( +t1 AS alias2 FORCE KEY (f3) JOIN +t1 AS alias3 FORCE KEY (f2) ON alias3.f2 = alias2.f2 AND alias3.f4 = alias2.f3 +) ON alias3.f1 <= alias2.f1 +) X; +count(*) +361 +set join_cache_level=@_save_join_cache_level; +drop table t1; === modified file 'mysql-test/t/innodb_mrr.test' --- a/mysql-test/t/innodb_mrr.test 2011-01-17 21:26:04 +0000 +++ b/mysql-test/t/innodb_mrr.test 2011-03-02 14:06:13 +0000 @@ -426,3 +426,31 @@ SELECT * FROM t1 WHERE c2 <=> NULL ORDER BY c2 LIMIT 2; drop table `t1`; +--echo # +--echo # BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) +--echo # +set @_save_join_cache_level= @@join_cache_level; +set join_cache_level = 6; +CREATE TABLE t1 ( + f1 int(11), f2 int(11), f3 varchar(1), f4 varchar(1), + PRIMARY KEY (f1), + KEY (f3), + KEY (f2) +) ENGINE=InnoDB; +INSERT INTO t1 VALUES ('11','8','f','f'),('12','5','v','v'),('13','8','s','s'), +('14','8','a','a'),('15','6','p','p'),('16','7','z','z'),('17','2','a','a'), +('18','5','h','h'),('19','7','h','h'),('20','2','v','v'),('21','9','v','v'), +('22','142','b','b'),('23','3','y','y'),('24','0','v','v'),('25','3','m','m'), +('26','5','z','z'),('27','9','n','n'),('28','1','d','d'),('29','107','a','a'); + +select count(*) from ( + SELECT alias1.f2 + FROM + t1 AS alias1 JOIN ( + t1 AS alias2 FORCE KEY (f3) JOIN + t1 AS alias3 FORCE KEY (f2) ON alias3.f2 = alias2.f2 AND alias3.f4 = alias2.f3 + ) ON alias3.f1 <= alias2.f1 +) X; + +set join_cache_level=@_save_join_cache_level; +drop table t1; === modified file 'sql/multi_range_read.cc' --- a/sql/multi_range_read.cc 2011-02-18 22:31:01 +0000 +++ b/sql/multi_range_read.cc 2011-03-02 14:06:13 +0000 @@ -387,15 +387,25 @@ bool Mrr_ordered_index_reader::set_interruption_temp_buffer(uint rowid_length, uint key_len, + uint saved_pk_len, uchar **space_start, uchar *space_end) { - if (space_end - *space_start <= (ptrdiff_t)(rowid_length + key_len)) + if (space_end - *space_start <= (ptrdiff_t)(rowid_length + key_len + saved_pk_len)) return TRUE; support_scan_interruptions= TRUE; saved_rowid= *space_start; *space_start += rowid_length; + + if (saved_pk_len) + { + saved_primary_key= *space_start; + *space_start += saved_pk_len; + } + else + saved_primary_key= NULL; + saved_key_tuple= *space_start; *space_start += key_len; @@ -406,17 +416,25 @@ void Mrr_ordered_index_reader::set_no_interruption_temp_buffer() { support_scan_interruptions= FALSE; - saved_key_tuple= saved_rowid= NULL; /* safety */ + saved_key_tuple= saved_rowid= saved_primary_key= NULL; /* safety */ have_saved_rowid= FALSE; } void Mrr_ordered_index_reader::interrupt_read() { DBUG_ASSERT(support_scan_interruptions); + TABLE *table= file->get_table(); /* Save the current key value */ - key_copy(saved_key_tuple, file->get_table()->record[0], - &file->get_table()->key_info[file->active_index], + key_copy(saved_key_tuple, table->record[0], + &table->key_info[file->active_index], keypar.key_tuple_length); + + if (saved_primary_key) + { + key_copy(saved_primary_key, table->record[0], + &table->key_info[table->s->primary_key], + table->key_info[table->s->primary_key].key_length); + } /* Save the last rowid */ memcpy(saved_rowid, file->ref, file->ref_length); @@ -433,9 +451,16 @@ void Mrr_ordered_index_reader::resume_read() { - key_restore(file->get_table()->record[0], saved_key_tuple, - &file->get_table()->key_info[file->active_index], + TABLE *table= file->get_table(); + key_restore(table->record[0], saved_key_tuple, + &table->key_info[file->active_index], keypar.key_tuple_length); + if (saved_primary_key) + { + key_restore(table->record[0], saved_primary_key, + &table->key_info[table->s->primary_key], + table->key_info[table->s->primary_key].key_length); + } } @@ -738,8 +763,8 @@ @retval other Error */ -int DsMrr_impl::dsmrr_init(handler *h_arg, RANGE_SEQ_IF *seq_funcs, - void *seq_init_param, uint n_ranges, uint mode, +int DsMrr_impl::dsmrr_init(handler *h_arg, RANGE_SEQ_IF *seq_funcs_arg, + void *seq_init_param_arg, uint n_ranges, uint mode, HANDLER_BUFFER *buf) { THD *thd= current_thd; @@ -748,6 +773,8 @@ uint key_buff_elem_size; handler *h_idx; Mrr_ordered_rndpos_reader *disk_strategy= NULL; + RANGE_SEQ_IF seq_funcs= *seq_funcs_arg; + void *seq_init_param= seq_init_param_arg; bool do_sort_keys= FALSE; DBUG_ENTER("DsMrr_impl::dsmrr_init"); LINT_INIT(key_buff_elem_size); /* set/used when do_sort_keys==TRUE */ @@ -810,8 +837,8 @@ { /* Pre-calculate some parameters of key sorting */ keypar.use_key_pointers= test(mode & HA_MRR_MATERIALIZED_KEYS); - seq_funcs->get_key_info(seq_init_param, &keypar.key_tuple_length, - &keypar.key_tuple_map); + seq_funcs.get_key_info(seq_init_param, &keypar.key_tuple_length, + &keypar.key_tuple_map); keypar.key_size_in_keybuf= keypar.use_key_pointers? sizeof(char*) : keypar.key_tuple_length; key_buff_elem_size= keypar.key_size_in_keybuf + (int)is_mrr_assoc * sizeof(void*); @@ -819,9 +846,18 @@ /* Ordered index reader needs some space to store an index tuple */ if (strategy != index_strategy) { + /* Normally we don't need to save/restore PK columns*/ + uint saved_pk_length=0; + if (h_idx->primary_key_is_clustered()) + { + uint pk= h_idx->get_table()->s->primary_key; + saved_pk_length= h_idx->get_table()->key_info[pk].key_length; + } + if (reader_factory.ordered_index_reader. set_interruption_temp_buffer(primary_file->ref_length, keypar.key_tuple_length, + saved_pk_length, &full_buf, full_buf_end)) goto use_default_impl; } @@ -845,7 +881,9 @@ if (do_sort_keys && !key_buffer->have_space_for(key_buff_elem_size)) goto use_default_impl; - if ((res= index_strategy->init(primary_file, seq_funcs, seq_init_param, n_ranges, + MRR_DEBUG(setup_range_id_mapping(&seq_funcs, &seq_init_param, n_ranges)); + + if ((res= index_strategy->init(primary_file, &seq_funcs, seq_init_param, n_ranges, mode, &keypar, key_buffer, &buf_manager))) goto error; } @@ -873,7 +911,9 @@ if ((res= setup_two_handlers())) goto error; - if ((res= index_strategy->init(secondary_file, seq_funcs, seq_init_param, + MRR_DEBUG(setup_range_id_mapping(&seq_funcs, &seq_init_param, n_ranges)); + + if ((res= index_strategy->init(secondary_file, &seq_funcs, seq_init_param, n_ranges, mode, &keypar, key_buffer, &buf_manager)) || (res= disk_strategy->init(primary_file, index_strategy, mode, @@ -883,6 +923,7 @@ } } + res= strategy->refill_buffer(TRUE); if (res) { @@ -910,7 +951,7 @@ DBUG_ASSERT(primary_file->inited == handler::INDEX); /* Call correct init function and assign to top level object */ Mrr_simple_index_reader *s= &reader_factory.simple_index_reader; - res= s->init(primary_file, seq_funcs, seq_init_param, n_ranges, mode, NULL, + res= s->init(primary_file, &seq_funcs, seq_init_param, n_ranges, mode, NULL, NULL, NULL); strategy= s; DBUG_RETURN(res); @@ -1033,6 +1074,7 @@ DBUG_ENTER("DsMrr_impl::dsmrr_close"); close_second_handler(); strategy= NULL; + MRR_DEBUG(clear_range_id_mapping()); DBUG_VOID_RETURN; } @@ -1690,7 +1732,110 @@ DBUG_VOID_RETURN; } - +#ifdef DBUG_MRR +range_seq_t DsMrr_impl::hook_init(void *init_params, uint n_ranges, uint flags) +{ + DsMrr_impl *mrr= (DsMrr_impl*)init_params; + mrr->backend_seq= mrr->backend_init(mrr->backend_init_params, n_ranges, flags); + return (range_seq_t)mrr; +} + +bool DsMrr_impl::hook_next(range_seq_t seq, KEY_MULTI_RANGE *range) +{ + DsMrr_impl *mrr= (DsMrr_impl*)seq; + int res; + if (!(res= mrr->backend_next(mrr->backend_seq, range))) + { + DBUG_ASSERT(mrr->cur_range_no < mrr->n_range_ids); + mrr->range_ids[mrr->cur_range_no++]= range->ptr; + } + return res; +} + +void DsMrr_impl::hook_get_key_info(void *init_params, uint *length, key_part_map *map) +{ + DsMrr_impl *mrr= (DsMrr_impl*)init_params; + mrr->backend_get_key_info(mrr->backend_init_params, length, map); +} + +bool DsMrr_impl::hook_skip_index_tuple(range_seq_t seq, char *range_info) +{ + DsMrr_impl *mrr= (DsMrr_impl*)seq; + int rangeno= mrr->range_id_to_number(range_info); + bool res= mrr->backend_skip_index_tuple(mrr->backend_seq, range_info); + fprintf(stderr, "checking index tuple %d -> %d\n", + rangeno, + (int)res); + return res; +} + +bool DsMrr_impl::hook_skip_record(range_seq_t seq, char *range_info, uchar *rowid) +{ + DsMrr_impl *mrr= (DsMrr_impl*)seq; + bool res= mrr->backend_skip_record(mrr->backend_seq, range_info, rowid); + return res; +} + +void DsMrr_impl::setup_range_id_mapping(RANGE_SEQ_IF *rseq, void **init_params, uint n_ranges) +{ + /* + Hook ourselves into RANGE_SEQ_IF so that we can watch ranges as they are + enumerated + */ + backend_init= rseq->init; + rseq->init= hook_init; + + backend_next= rseq->next; + rseq->next= hook_next; + + backend_init_params= *init_params; + *init_params= (void*)this; + + if (rseq->get_key_info) + { + backend_get_key_info= rseq->get_key_info; + rseq->get_key_info= hook_get_key_info; + } + + if (rseq->skip_index_tuple) + { + backend_skip_index_tuple= rseq->skip_index_tuple; + rseq->skip_index_tuple= hook_skip_index_tuple; + } + + if (rseq->skip_record) + { + backend_skip_record= rseq->skip_record; + rseq->skip_record= hook_skip_record; + } + + /* Alloc space for the mapping */ + n_range_ids= n_ranges; + if (n_ranges) + range_ids= (char**)malloc(sizeof(char*) * n_ranges); + else + range_ids= NULL; + cur_range_no= 0; +} + +void DsMrr_impl::clear_range_id_mapping() +{ + if (range_ids) + free(range_ids); + range_ids= NULL; + n_range_ids= 0; +} + +int DsMrr_impl::range_id_to_number(char *range_id) +{ + for (int i=0; i < n_range_ids; i++) + { + if (range_ids[i] == range_id) + return i; + } + return -1; +} +#endif /* ************************************************************************** * DS-MRR implementation ends ***************************************************************************/ === modified file 'sql/multi_range_read.h' --- a/sql/multi_range_read.h 2010-12-20 11:40:12 +0000 +++ b/sql/multi_range_read.h 2011-03-02 14:06:13 +0000 @@ -271,7 +271,8 @@ mrr_funcs.skip_index_tuple(mrr_iter, range_info)); } - bool set_interruption_temp_buffer(uint rowid_length, uint key_len, + bool set_interruption_temp_buffer(uint rowid_length, uint key_len, + uint saved_pk_len, uchar **space_start, uchar *space_end); void set_no_interruption_temp_buffer(); @@ -314,6 +315,7 @@ bool have_saved_rowid; uchar *saved_key_tuple; + uchar *saved_primary_key; static int compare_keys(void* arg, uchar* key1, uchar* key2); static int compare_keys_reverse(void* arg, uchar* key1, uchar* key2); @@ -520,7 +522,12 @@ typedef void (handler::*range_check_toggle_func_t)(bool on); DsMrr_impl() - : secondary_file(NULL) {}; + : secondary_file(NULL) + { +#ifdef DBUG_MRR + range_id_mapping_ctor(); +#endif + } void init(handler *h_arg, TABLE *table_arg) { @@ -615,6 +622,52 @@ int setup_two_handlers(); void close_second_handler(); + +#ifdef DBUG_MRR +#define MRR_DEBUG(x) x + /* + Debugging aid: support for converting range_ids (which are pointers) to + range numbers. Range's number specifies when the range was returned by + RANGE_SEQ_IF::next(). First returned range gets number 0, second gets + number 1, and so on. + */ + char **range_ids; /* range_ids[N] == range_id of range with number N */ + int n_range_ids; /* Total number of range ids */ + + /* "current" range number (makes sense when walking the ranges) */ + int cur_range_no; + + + /* RANGE_SEQ_IF override: "backend" functions */ + range_seq_t (*backend_init)(void *init_params, uint n_ranges, uint flags); + bool (*backend_next)(range_seq_t seq, KEY_MULTI_RANGE *range); + void (*backend_get_key_info)(void *init_params, uint *length, key_part_map *map); + bool (*backend_skip_index_tuple)(range_seq_t seq, char *range_info); + bool (*backend_skip_record)(range_seq_t seq, char *range_info, uchar *rowid); + void *backend_init_params; + range_seq_t backend_seq; + + /* RANGE_SEQ_IF override: our hook-functions */ + static range_seq_t hook_init(void *init_params, uint n_ranges, uint flags); + static bool hook_next(range_seq_t seq, KEY_MULTI_RANGE *range); + static void hook_get_key_info(void *init_params, uint *length, key_part_map *map); + static bool hook_skip_index_tuple(range_seq_t seq, char *range_info); + static bool hook_skip_record(range_seq_t seq, char *range_info, uchar *rowid); + + /* Mapper [de]initialization functions*/ + void range_id_mapping_ctor() + { + range_ids= NULL; + n_range_ids= 0; + } + void setup_range_id_mapping(RANGE_SEQ_IF *rseq, void **init_params, uint n_ranges); + void clear_range_id_mapping(); + + /* The reason of all of the above: function to map range ids to range numbers */ + int range_id_to_number(char *range_id); +#else +#define MRR_DEBUG(x) +#endif }; /** _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] Rev 2905: Merge Percona patch MWL#47 into mariadb 5.2-percona. in http://bazaar.launchpad.net/~maria-captains/maria/5.2
by Kristian Nielsen 02 Mar '11

02 Mar '11

knielsen(a)knielsen-hq.org writes: > message: > Merge Percona patch MWL#47 into mariadb 5.2-percona. > > This patch adds options to annotate the binlog (and the mysqlbinlog > output) with the original SQL query for queries that are logged > using row-based replication. Serg raised the very relevant issue about what to do about the similar feature in MySQL 5.6: http://forge.mysql.com/worklog/task.php?id=4033 http://forge.mysql.com/worklog/task.php?id=5404 The Oracle team has worked more on this since last I looked, and the two features are now very similar. Here are the main differences, as I see it: - The MySQL 5.6 feature is enabled by --binlog_rows_query_log_events, while the MWL#47 option is called --binlog-annotate-rows-events. - MWL#47 has --replicate-annotate-rows-events which makes the slave receive and binlog the annotations. The MySQL feature always receives annotation events if present in the binlog (so replication 5.6->pre-5.6 will break if annotations are enabled). The MySQL feature binlogs the annotations based on the same --binlog-annotate-rows-events option as the master. - MWL#47 has a mysqlbinlog option --skip-annotate-rows-events, MySQL 5.6 does not. - MySQL 5.6 mysqlbinlog can log the annotations with BINLOG statements so that they are preserved with `mysqlbinlog | mysql`; MWL#47 does not have this. - MWL#47 uses binlog event type 160, MySQL 5.6 uses an ignorable event of type 29. As far as I can tell, the actual event format is identical, except for the LOG_EVENT_IGNORABLE_F flag (which says it's ok for a slave to ignore this event if it doesn't know about it, however there is no server version that knows the LOG_EVENT_IGNORABLE_F but not the MySQL annotation event). - I think there are a couple of issues with the MySQL patch (newlines in queries breaking mysqlbinlog output, duplicated annotations for INSERT DELAYED). There might be issues with the MWL#47 patch also, of course. So the question is how to prepare for when we merge MySQL 5.6. An easy option could be to just ditch MWL#47 (and possibly backport the MySQL 5.6 patch). However, as I understand it, the original sponsor of MWL#47 is already using it, and so we may have to remain compatible with the existing MWL#47 implementation. Can we get some definite information on that? Another option might be to simply change the MWL#47 to use the same event type as MySQL (the event format is the same, and we can add the LOG_EVENT_IGNORABLE_F). And replace the MWL#47 options --binlog-annotate-rows-events and --replicate-annotate-rows-events with the MySQL --binlog_rows_query_log_events option (though I think the MySQL option name is not good). But again, I am not sure if this is a valid option given possible existing usage of MWL#47. The third option I can think of is to leave MWL#47 as it is. Then when we get to merge MySQL 5.6, we will basically drop the MySQL part, doing the following to remain backwards compatible with MySQL: - Make the --binlog_rows_query_log_events option set --binlog-annotate-rows-events and --replicate-annotate-rows-events. - Make a slave read a MySQL rows_query_log_event as a MWL#47 annotation event. - Merge over the bit of the MySQL patch that makes mysqlbinlog output annotation events using a BINLOG '...' statement. This should make upgrade from MySQL 5.6 -> MariaDB fully work. Using MySQL5.6+ with MariaDB will also work, only MySQL slaves will not receive MariaDB master annotation events. The third option seems to preserve the existing MWL#47 most faithfully and does not seem to need much work in terms of the eventual merge, so this is what I suggest as the best way forward. But I am open to other (justified) opinions. - Kristian.

2 1

Re: [Maria-developers] [Commits] Rev 3014: New table creation/renaming block added if old encoded table present. in file:///home/bell/maria/bzr/work-maria-5.1-CREATE/
by Sergei Golubchik 02 Mar '11

02 Mar '11

Hi, sanja! On Jan 13, sanja(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 3014 > revision-id: sanja(a)askmonty.org-20110113180607-e5hc8zuugxxjtn6k > parent: psergey(a)askmonty.org-20110112083018-90rw058f00uslvoz > committer: sanja(a)askmonty.org > branch nick: work-maria-5.1-CREATE > timestamp: Thu 2011-01-13 20:06:07 +0200 > message: > New table creation/renaming block added if old encoded table present. you've extended the fix, but not the test case. please add a test case for the new code and ok to push Regards, Sergei

1 0

[Maria-developers] Igor please review: Rev 2929: BUG#693747: Assertion multi_range_read.cc:908: int DsMrr_impl::dsmrr_init in file:///home/psergey/dev2/5.3-2/]
by Sergey Petrunya 02 Mar '11

02 Mar '11

Hello Igor, Could you please review the below: ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- From: Sergey Petrunya <psergey(a)askmonty.org> To: commits(a)mariadb.org X-Mailer: mail (GNU Mailutils 1.2) Date: Wed, 2 Mar 2011 00:17:53 +0300 (MSK) Subject: [Commits] Rev 2929: BUG#693747: Assertion multi_range_read.cc:908: int DsMrr_impl::dsmrr_init in file:///home/psergey/dev2/5.3-2/ At file:///home/psergey/dev2/5.3-2/ ------------------------------------------------------------ revno: 2929 revision-id: psergey(a)askmonty.org-20110301211747-8wvngodwxzr9i2e9 parent: psergey(a)askmonty.org-20110301072222-1rmapkelx9l1kw8u committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: 5.3-2 timestamp: Wed 2011-03-02 00:17:47 +0300 message: BUG#693747: Assertion multi_range_read.cc:908: int DsMrr_impl::dsmrr_init - Make DsMrr_impl::dsmrr_init() handle the case of 1. 1st MRR scan using DS-MRR strategy (i.e. doing key sorting and rowid sorting) 2. 2nd MRR scan getting a buffer that's too small to fit one key element and one rowid element, and so falling back to default MRR implementation In this case, dsmrr_init() is invoked with {primary_handler, secondary_handler} initialized for DS-MRR scan and have to reset them to be initialized for the default MRR scan. === modified file 'mysql-test/r/maria_mrr.result' --- a/mysql-test/r/maria_mrr.result 2010-12-02 11:10:52 +0000 +++ b/mysql-test/r/maria_mrr.result 2011-03-01 21:17:47 +0000 @@ -403,3 +403,30 @@ count(*) 0 drop table t1, t2; +# +# BUG#693747: Assertion multi_range_read.cc:908: int DsMrr_impl::dsmrr_init( +# +set @_save_join_cache_level= @@join_cache_level; +set @_save_join_buffer_size= @@join_buffer_size; +set join_cache_level=8; +set join_buffer_size=10000; +CREATE TABLE t1 ( +f1 int(11), f2 varchar(32) COLLATE latin1_swedish_ci, +f3 int(11), f4 varchar(1024) COLLATE utf8_bin, +f5 varchar(1024) COLLATE latin1_bin, +KEY (f5) +) ENGINE=Aria TRANSACTIONAL=0 ; +# Fill the table with some data +SELECT alias2.f1 AS field1 , alias1.f2 AS field2 +FROM t1 AS alias1 +LEFT OUTER JOIN t1 AS alias2 ON alias1.f2 = alias2.f5 +WHERE alias2.f3 > 'k' +AND alias2.f3 <= 'z' +OR alias2.f4 IN ('o', 'y') +AND alias2.f4 >= 'g' AND alias2.f4 < 'z'; +field1 field2 +Warnings: +Warning 1292 Truncated incorrect DOUBLE value: 'k' +set join_cache_level=@_save_join_cache_level; +set join_buffer_size=@_save_join_buffer_size; +drop table t1; === modified file 'mysql-test/t/maria_mrr.test' --- a/mysql-test/t/maria_mrr.test 2010-11-09 16:02:08 +0000 +++ b/mysql-test/t/maria_mrr.test 2011-03-01 21:17:47 +0000 @@ -125,3 +125,81 @@ table1.col_varchar_1024_latin1_key = table2.col_varchar_10_latin1 AND table1.pk<>0 ; drop table t1, t2; + +--echo # +--echo # BUG#693747: Assertion multi_range_read.cc:908: int DsMrr_impl::dsmrr_init( +--echo # +set @_save_join_cache_level= @@join_cache_level; +set @_save_join_buffer_size= @@join_buffer_size; + +set join_cache_level=8; +set join_buffer_size=10000; + +CREATE TABLE t1 ( + f1 int(11), f2 varchar(32) COLLATE latin1_swedish_ci, + f3 int(11), f4 varchar(1024) COLLATE utf8_bin, + f5 varchar(1024) COLLATE latin1_bin, + KEY (f5) +) ENGINE=Aria TRANSACTIONAL=0 ; + +--echo # Fill the table with some data +--disable_query_log +INSERT IGNORE INTO t1 VALUES +('0','cueikuirqr','0','okay','hcueikuirqrzflno'),('2009','her','0','ULWAH','ehcueikuirqrzfln'), +('0','YKAOE','0','of','qieehcueikuirqrz'),('2009','you\'re','0','k','nkqieehcueikuirq'), +('2003','b','0','bgsnkqieehcueiku','the'),('0','MGUDG','0','p','m'), +('0','UXAGU','0','elfhjawbwbgsnkqi','HZXVA'),('2004','bwbgsnkqie','0','x','something'), +('2007','s','0','okay','slelfhjawbwbgsnk'),('2007','the','0','or','if'), +('2008','TDLKE','0','mean','MGWNJ'),('2006','do','0','KGFVV','see'), +('2001','why','0','a','mean'),(NULL,'THKCG','0','no','YFLDY'), +(NULL,'x','0','mean','e'),('2007','yncitaeysb','0','WDOSC','tgyncitaeysbgucs'), +(NULL,'ZEOXX','0','awbwbgsnkqieehcu','jawbwbgsnkqieehc'),('2006','hjawbwbgsn','0','just','fhjawbwbgsnkqiee'), +('0','all','0','k','sbgucsgqslelfhja'),('2009','the','0','f','would'), +('2005','mtgyncitae','0','w','ISNQQ'),('2003','KNCUI','0','r','want'), +('2006','is','0','j','i'),('2000','out','0','o','jvcmjlmtgyncitae'), +('2001','it','0','txyjvcmjlmtgynci','you'),('2008','LHDIH','0','ZKFJK','txmtxyjvcmjlmtgy'), +('0','z','0','the','ntxmtxyjvcmjlmtg'),(NULL,'vyhnmvgmcn','0','PWEYL','AIGQK'), +('0','ytvyhnmvgm','0','gbytvyhnmvgmcntx','z'),('2002','t','0','just','on'), +('2002','xqegbytvyh','0','to','ixqegbytvyhnmvgm'),('2004','WGVRU','0','GHBZB','h'), +('0','b','0','yhnmvgmcntxmtxyj','z'),('2003','who','0','ddixqegbytvyhnmv','gddixqegbytvy'), +('0','PMLFL','0','n','vgmcntxmtxyjvcmj'),(NULL,'back','0','FIXPA','n'), +('2005','i','0','I\'ll','PZGUB'),('0','f','0','t','the'), +(NULL,'PNXVP','0','t','v'),('2004','MAKKL','0','u','CGCWF'), +('0','RMDAV','0','noypgddixqegbytv','v'),('0','l','0','ESAVW','n'), +('2008','rhnoypgddi','0','s','VIZNE'),('2007','t','0','XZECS','a'), +(NULL,'like','0','k','JSHPZ'),('2009','pskeywslmk','0','epskeywslmkdrhno','q'), +(NULL,'QZZJJ','0','LOTFB','c'),('0','atlxepskey','0','b','YJRMA'), +(NULL,'YUVOU','0','wslmkdrhnoypgddi','eywslmkdrhnoypgd'),('2006','some','0','can','r'), +('2002','c','0','k','her'),(NULL,'o','0','ALBVZ','EMURT'), +('0','if','0','okay','had'),('2006','when','0','e','CLVWT'), +(NULL,'blfufrcdjm','0','ZAKUQ','IZCZN'),('2007','vutblfufrc','0','MPKMI','how'), +(NULL,'why','0','UBUAG','I'),('2004','IXLYQ','0','ILPBE','weuwuvutblfufrcd'), +('0','here','0','your','m'),(NULL,'ZOCTJ','0','GTPOO','IDSFD'), +('0','kqsweuwuvu','0','CXQLW','oh'),('0','ykqsweuwuv','0','CLJWP','zykqsweuwuvutblf'), +('0','zezykqsweu','0','had','t'),(NULL,'q','0','jisuzezykqsweuwu','o'), +('2007','IBKAU','0','t','oh'),('0','ivjisuzezy','0','y','XHXKE'), +('0','xsivjisuze','0','right','plxsivjisuzezykq'),('2009','have','0','so','uvplxsivjisuzezy'), +('0','on','0','b','me'),('0','ijkfuvplxs','0','RZXDY','OGEHV'), +('0','u','0','all','okay'),('0','i','0','s','pajzbbojshnijkfu'), +('2001','of','0','xpajzbbojshnijkf','g'),('0','for','0','ITBAQ','I\'m'), +('2006','or','0','now','ZOJHX'),('0','n','0','k','you'), +(NULL,'that','0','suzezykqsweuwuvu','just'),('0','bbojshnijk','0','QGYOA','JYGSJ'), +('2009','k','0','ajzbbojshnijkfuv','y'),('2007','k','0','affpegvavdgxpajz','y'), +('2007','be','0','RCTIT','m'),('2000','fnbmxwicrk','0','lcfnbmxwicrkdyma','t'), +('0','yaffpegvav','0','v','have'),('0','crkdymahya','0','icrkdymahyaffpeg','QQWQI'), +('0','t','0','nijkfuvplxsivjis','hnijkfuvplxsivji'),('0','dgxpajzbbo','0','vdgxpajzbbojshni','vavdgxpajzbbojsh'), +('2006','g','0','gvavdgxpajzbbojs','pegvavdgxpajzbbo'),('2007','I\'m','0','g','ffpegvavdgxpajzb'); +--enable_query_log + + +SELECT alias2.f1 AS field1 , alias1.f2 AS field2 +FROM t1 AS alias1 +LEFT OUTER JOIN t1 AS alias2 ON alias1.f2 = alias2.f5 +WHERE alias2.f3 > 'k' +AND alias2.f3 <= 'z' +OR alias2.f4 IN ('o', 'y') +AND alias2.f4 >= 'g' AND alias2.f4 < 'z'; + +set join_cache_level=@_save_join_cache_level; +set join_buffer_size=@_save_join_buffer_size; +drop table t1; === modified file 'sql/multi_range_read.cc' --- a/sql/multi_range_read.cc 2011-02-18 22:31:01 +0000 +++ b/sql/multi_range_read.cc 2011-03-01 21:17:47 +0000 @@ -907,7 +907,26 @@ DBUG_RETURN(res); use_default_impl: - DBUG_ASSERT(primary_file->inited == handler::INDEX); + if (primary_file->inited != handler::INDEX) + { + /* We can get here when + - we've previously successfully done a DS-MRR scan (and so have + secondary_file!= NULL, secondary_file->inited= INDEX, + primary_file->inited=RND) + - for this invocation, we haven't got enough buffer space, and so we + have to use the default MRR implementation. + + note: primary_file->ha_index_end() will call dsmrr_close() which will + close/destroy the secondary_file, this is intentional. + (Yes this is slow, but one can't expect performance with join buffer + so small that it can accomodate one rowid and one index tuple) + */ + if ((res= primary_file->ha_rnd_end()) || + (res= primary_file->ha_index_init(keyno, test(mode & HA_MRR_SORTED)))) + { + DBUG_RETURN(res); + } + } /* Call correct init function and assign to top level object */ Mrr_simple_index_reader *s= &reader_factory.simple_index_reader; res= s->init(primary_file, seq_funcs, seq_init_param, n_ranges, mode, NULL, _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Igor please review: BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) in file:///home/psergey/dev2/5.3/]
by Sergey Petrunya 02 Mar '11

02 Mar '11

Hello Igor, Could you please review the below: ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- From: Sergey Petrunya <psergey(a)askmonty.org> To: commits(a)mariadb.org X-Mailer: mail (GNU Mailutils 1.2) Date: Wed, 2 Mar 2011 16:27:43 +0300 (MSK) Subject: [Commits] Rev 2929: BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) in file:///home/psergey/dev2/5.3/ At file:///home/psergey/dev2/5.3/ ------------------------------------------------------------ revno: 2929 revision-id: psergey(a)askmonty.org-20110302132739-s6vasca1nm6eh2le parent: psergey(a)askmonty.org-20110301072222-1rmapkelx9l1kw8u committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: 5.3 timestamp: Wed 2011-03-02 16:27:39 +0300 message: BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) - Add debugging aid which maps MRR's range_ids (which are pointers so are different on every execution and hard to track) to range numbers (which allow much easier debugging). The mapper is not compiled unless the developer puts a #define into code to request it. - Fix the bug: the problem was that Mrr_ordered_index_reader's interrupt_read() and resume_read() would save and restore 1) index tuple 2) the rowid (as bytes returned by handler->position()). Clustered primary key columns were not saved/restored. They are not explicitly present in the index tuple (i.e. table->key_info[secondary_key].key_parts doesn't list them), but they are actually there, in particular table->field[clustered_primary_key_member].part_of_key(secondary_key) == 1. Index condition pushdown code [correctly] uses the latter as inidication that pushed index condition can refer to clustered PK members. The fix was to make interrupt_read()/resume_read() to save/restore clustered primary key members as well, so that we get correct values for them when evaluating pushed index condition. === modified file 'mysql-test/r/innodb_mrr.result' --- a/mysql-test/r/innodb_mrr.result 2011-01-17 21:26:04 +0000 +++ b/mysql-test/r/innodb_mrr.result 2011-03-02 13:27:39 +0000 @@ -736,3 +736,31 @@ c1 c2 c3 08:29:45 NULL 2009-02-01 drop table `t1`; +# +# BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) +# +set @_save_join_cache_level= @@join_cache_level; +set join_cache_level = 6; +CREATE TABLE t1 ( +f1 int(11), f2 int(11), f3 varchar(1), f4 varchar(1), +PRIMARY KEY (f1), +KEY (f3), +KEY (f2) +) ENGINE=InnoDB; +INSERT INTO t1 VALUES ('11','8','f','f'),('12','5','v','v'),('13','8','s','s'), +('14','8','a','a'),('15','6','p','p'),('16','7','z','z'),('17','2','a','a'), +('18','5','h','h'),('19','7','h','h'),('20','2','v','v'),('21','9','v','v'), +('22','142','b','b'),('23','3','y','y'),('24','0','v','v'),('25','3','m','m'), +('26','5','z','z'),('27','9','n','n'),('28','1','d','d'),('29','107','a','a'); +select count(*) from ( +SELECT alias1.f2 +FROM +t1 AS alias1 JOIN ( +t1 AS alias2 FORCE KEY (f3) JOIN +t1 AS alias3 FORCE KEY (f2) ON alias3.f2 = alias2.f2 AND alias3.f4 = alias2.f3 +) ON alias3.f1 <= alias2.f1 +) X; +count(*) +361 +set join_cache_level=@_save_join_cache_level; +drop table t1; === modified file 'mysql-test/t/innodb_mrr.test' --- a/mysql-test/t/innodb_mrr.test 2011-01-17 21:26:04 +0000 +++ b/mysql-test/t/innodb_mrr.test 2011-03-02 13:27:39 +0000 @@ -426,3 +426,31 @@ SELECT * FROM t1 WHERE c2 <=> NULL ORDER BY c2 LIMIT 2; drop table `t1`; +--echo # +--echo # BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) +--echo # +set @_save_join_cache_level= @@join_cache_level; +set join_cache_level = 6; +CREATE TABLE t1 ( + f1 int(11), f2 int(11), f3 varchar(1), f4 varchar(1), + PRIMARY KEY (f1), + KEY (f3), + KEY (f2) +) ENGINE=InnoDB; +INSERT INTO t1 VALUES ('11','8','f','f'),('12','5','v','v'),('13','8','s','s'), +('14','8','a','a'),('15','6','p','p'),('16','7','z','z'),('17','2','a','a'), +('18','5','h','h'),('19','7','h','h'),('20','2','v','v'),('21','9','v','v'), +('22','142','b','b'),('23','3','y','y'),('24','0','v','v'),('25','3','m','m'), +('26','5','z','z'),('27','9','n','n'),('28','1','d','d'),('29','107','a','a'); + +select count(*) from ( + SELECT alias1.f2 + FROM + t1 AS alias1 JOIN ( + t1 AS alias2 FORCE KEY (f3) JOIN + t1 AS alias3 FORCE KEY (f2) ON alias3.f2 = alias2.f2 AND alias3.f4 = alias2.f3 + ) ON alias3.f1 <= alias2.f1 +) X; + +set join_cache_level=@_save_join_cache_level; +drop table t1; === modified file 'sql/multi_range_read.cc' --- a/sql/multi_range_read.cc 2011-02-18 22:31:01 +0000 +++ b/sql/multi_range_read.cc 2011-03-02 13:27:39 +0000 @@ -387,15 +387,25 @@ bool Mrr_ordered_index_reader::set_interruption_temp_buffer(uint rowid_length, uint key_len, + uint saved_pk_len, uchar **space_start, uchar *space_end) { - if (space_end - *space_start <= (ptrdiff_t)(rowid_length + key_len)) + if (space_end - *space_start <= (ptrdiff_t)(rowid_length + key_len + saved_pk_len)) return TRUE; support_scan_interruptions= TRUE; saved_rowid= *space_start; *space_start += rowid_length; + + if (saved_pk_len) + { + saved_primary_key= *space_start; + *space_start += saved_pk_len; + } + else + saved_primary_key= NULL; + saved_key_tuple= *space_start; *space_start += key_len; @@ -406,17 +416,25 @@ void Mrr_ordered_index_reader::set_no_interruption_temp_buffer() { support_scan_interruptions= FALSE; - saved_key_tuple= saved_rowid= NULL; /* safety */ + saved_key_tuple= saved_rowid= saved_primary_key= NULL; /* safety */ have_saved_rowid= FALSE; } void Mrr_ordered_index_reader::interrupt_read() { DBUG_ASSERT(support_scan_interruptions); + TABLE *table= file->get_table(); /* Save the current key value */ - key_copy(saved_key_tuple, file->get_table()->record[0], - &file->get_table()->key_info[file->active_index], + key_copy(saved_key_tuple, table->record[0], + &table->key_info[file->active_index], keypar.key_tuple_length); + + if (saved_primary_key) + { + key_copy(saved_primary_key, table->record[0], + &table->key_info[table->s->primary_key], + table->key_info[table->s->primary_key].key_length); + } /* Save the last rowid */ memcpy(saved_rowid, file->ref, file->ref_length); @@ -433,9 +451,16 @@ void Mrr_ordered_index_reader::resume_read() { - key_restore(file->get_table()->record[0], saved_key_tuple, - &file->get_table()->key_info[file->active_index], + TABLE *table= file->get_table(); + key_restore(table->record[0], saved_key_tuple, + &table->key_info[file->active_index], keypar.key_tuple_length); + if (saved_primary_key) + { + key_restore(table->record[0], saved_primary_key, + &table->key_info[table->s->primary_key], + table->key_info[table->s->primary_key].key_length); + } } @@ -738,8 +763,8 @@ @retval other Error */ -int DsMrr_impl::dsmrr_init(handler *h_arg, RANGE_SEQ_IF *seq_funcs, - void *seq_init_param, uint n_ranges, uint mode, +int DsMrr_impl::dsmrr_init(handler *h_arg, RANGE_SEQ_IF *seq_funcs_arg, + void *seq_init_param_arg, uint n_ranges, uint mode, HANDLER_BUFFER *buf) { THD *thd= current_thd; @@ -748,6 +773,8 @@ uint key_buff_elem_size; handler *h_idx; Mrr_ordered_rndpos_reader *disk_strategy= NULL; + RANGE_SEQ_IF seq_funcs= *seq_funcs_arg; + void *seq_init_param= seq_init_param_arg; bool do_sort_keys= FALSE; DBUG_ENTER("DsMrr_impl::dsmrr_init"); LINT_INIT(key_buff_elem_size); /* set/used when do_sort_keys==TRUE */ @@ -810,8 +837,8 @@ { /* Pre-calculate some parameters of key sorting */ keypar.use_key_pointers= test(mode & HA_MRR_MATERIALIZED_KEYS); - seq_funcs->get_key_info(seq_init_param, &keypar.key_tuple_length, - &keypar.key_tuple_map); + seq_funcs.get_key_info(seq_init_param, &keypar.key_tuple_length, + &keypar.key_tuple_map); keypar.key_size_in_keybuf= keypar.use_key_pointers? sizeof(char*) : keypar.key_tuple_length; key_buff_elem_size= keypar.key_size_in_keybuf + (int)is_mrr_assoc * sizeof(void*); @@ -819,9 +846,18 @@ /* Ordered index reader needs some space to store an index tuple */ if (strategy != index_strategy) { + /* Normally we don't need to save/restore PK columns*/ + uint saved_pk_length=0; + if (h_idx->primary_key_is_clustered()) + { + uint pk= h_idx->get_table()->s->primary_key; + saved_pk_length= h_idx->get_table()->key_info[pk].key_length; + } + if (reader_factory.ordered_index_reader. set_interruption_temp_buffer(primary_file->ref_length, keypar.key_tuple_length, + saved_pk_length, &full_buf, full_buf_end)) goto use_default_impl; } @@ -845,7 +881,9 @@ if (do_sort_keys && !key_buffer->have_space_for(key_buff_elem_size)) goto use_default_impl; - if ((res= index_strategy->init(primary_file, seq_funcs, seq_init_param, n_ranges, + MRR_DEBUG(setup_range_id_mapping(&seq_funcs, &seq_init_param, n_ranges)); + + if ((res= index_strategy->init(primary_file, &seq_funcs, seq_init_param, n_ranges, mode, &keypar, key_buffer, &buf_manager))) goto error; } @@ -873,7 +911,9 @@ if ((res= setup_two_handlers())) goto error; - if ((res= index_strategy->init(secondary_file, seq_funcs, seq_init_param, + MRR_DEBUG(setup_range_id_mapping(&seq_funcs, &seq_init_param, n_ranges)); + + if ((res= index_strategy->init(secondary_file, &seq_funcs, seq_init_param, n_ranges, mode, &keypar, key_buffer, &buf_manager)) || (res= disk_strategy->init(primary_file, index_strategy, mode, @@ -883,6 +923,7 @@ } } + res= strategy->refill_buffer(TRUE); if (res) { @@ -910,7 +951,7 @@ DBUG_ASSERT(primary_file->inited == handler::INDEX); /* Call correct init function and assign to top level object */ Mrr_simple_index_reader *s= &reader_factory.simple_index_reader; - res= s->init(primary_file, seq_funcs, seq_init_param, n_ranges, mode, NULL, + res= s->init(primary_file, &seq_funcs, seq_init_param, n_ranges, mode, NULL, NULL, NULL); strategy= s; DBUG_RETURN(res); @@ -1033,6 +1074,7 @@ DBUG_ENTER("DsMrr_impl::dsmrr_close"); close_second_handler(); strategy= NULL; + MRR_DEBUG(clear_range_id_mapping()); DBUG_VOID_RETURN; } @@ -1690,7 +1732,110 @@ DBUG_VOID_RETURN; } - +#ifdef DBUG_MRR +range_seq_t DsMrr_impl::hook_init(void *init_params, uint n_ranges, uint flags) +{ + DsMrr_impl *mrr= (DsMrr_impl*)init_params; + mrr->backend_seq= mrr->backend_init(mrr->backend_init_params, n_ranges, flags); + return (range_seq_t)mrr; +} + +bool DsMrr_impl::hook_next(range_seq_t seq, KEY_MULTI_RANGE *range) +{ + DsMrr_impl *mrr= (DsMrr_impl*)seq; + int res; + if (!(res= mrr->backend_next(mrr->backend_seq, range))) + { + DBUG_ASSERT(mrr->cur_range_no < mrr->n_range_ids); + mrr->range_ids[mrr->cur_range_no++]= range->ptr; + } + return res; +} + +void DsMrr_impl::hook_get_key_info(void *init_params, uint *length, key_part_map *map) +{ + DsMrr_impl *mrr= (DsMrr_impl*)init_params; + mrr->backend_get_key_info(mrr->backend_init_params, length, map); +} + +bool DsMrr_impl::hook_skip_index_tuple(range_seq_t seq, char *range_info) +{ + DsMrr_impl *mrr= (DsMrr_impl*)seq; + int rangeno= mrr->range_id_to_number(range_info); + bool res= mrr->backend_skip_index_tuple(mrr->backend_seq, range_info); + fprintf(stderr, "checking index tuple %d -> %d\n", + rangeno, + (int)res); + return res; +} + +bool DsMrr_impl::hook_skip_record(range_seq_t seq, char *range_info, uchar *rowid) +{ + DsMrr_impl *mrr= (DsMrr_impl*)seq; + bool res= mrr->backend_skip_record(mrr->backend_seq, range_info, rowid); + return res; +} + +void DsMrr_impl::setup_range_id_mapping(RANGE_SEQ_IF *rseq, void **init_params, uint n_ranges) +{ + /* + Hook ourselves into RANGE_SEQ_IF so that we can watch ranges as they are + enumerated + */ + backend_init= rseq->init; + rseq->init= hook_init; + + backend_next= rseq->next; + rseq->next= hook_next; + + backend_init_params= *init_params; + *init_params= (void*)this; + + if (rseq->get_key_info) + { + backend_get_key_info= rseq->get_key_info; + rseq->get_key_info= hook_get_key_info; + } + + if (rseq->skip_index_tuple) + { + backend_skip_index_tuple= rseq->skip_index_tuple; + rseq->skip_index_tuple= hook_skip_index_tuple; + } + + if (rseq->skip_record) + { + backend_skip_record= rseq->skip_record; + rseq->skip_record= hook_skip_record; + } + + /* Alloc space for the mapping */ + n_range_ids= n_ranges; + if (n_ranges) + range_ids= (char**)malloc(sizeof(char*) * n_ranges); + else + range_ids= NULL; + cur_range_no= 0; +} + +void DsMrr_impl::clear_range_id_mapping() +{ + if (range_ids) + free(range_ids); + range_ids= NULL; + n_range_ids= 0; +} + +int DsMrr_impl::range_id_to_number(char *range_id) +{ + for (int i=0; i < n_range_ids; i++) + { + if (range_ids[i] == range_id) + return i; + } + return -1; +} +#endif /* ************************************************************************** * DS-MRR implementation ends ***************************************************************************/ === modified file 'sql/multi_range_read.h' --- a/sql/multi_range_read.h 2010-12-20 11:40:12 +0000 +++ b/sql/multi_range_read.h 2011-03-02 13:27:39 +0000 @@ -271,7 +271,8 @@ mrr_funcs.skip_index_tuple(mrr_iter, range_info)); } - bool set_interruption_temp_buffer(uint rowid_length, uint key_len, + bool set_interruption_temp_buffer(uint rowid_length, uint key_len, + uint saved_pk_len, uchar **space_start, uchar *space_end); void set_no_interruption_temp_buffer(); @@ -314,6 +315,7 @@ bool have_saved_rowid; uchar *saved_key_tuple; + uchar *saved_primary_key; static int compare_keys(void* arg, uchar* key1, uchar* key2); static int compare_keys_reverse(void* arg, uchar* key1, uchar* key2); @@ -520,7 +522,12 @@ typedef void (handler::*range_check_toggle_func_t)(bool on); DsMrr_impl() - : secondary_file(NULL) {}; + : secondary_file(NULL) + { +#ifdef DBUG_MRR + range_id_mapping_ctor(); +#endif + } void init(handler *h_arg, TABLE *table_arg) { @@ -615,6 +622,50 @@ int setup_two_handlers(); void close_second_handler(); + +#ifdef DBUG_MRR +#define MRR_DEBUG(x) x + /* + Debugging aid: support for converting range_ids (which are pointers) to + range numbers. Range's number specifies when the range was returned by + RANGE_SEQ_IF::next(). First returned range gets number 0, second gets + number 1, and so on. + */ + char **range_ids; /* range_ids[N] == range_id of range with number N */ + int n_range_ids; /* Total number of range ids */ + + /* "current" range number (makes sense when walking the ranges) */ + int cur_range_no; + + + /* RANGE_SEQ_IF override: "backend" functions */ + range_seq_t (*backend_init)(void *init_params, uint n_ranges, uint flags); + bool (*backend_next)(range_seq_t seq, KEY_MULTI_RANGE *range); + void (*backend_get_key_info)(void *init_params, uint *length, key_part_map *map); + bool (*backend_skip_index_tuple)(range_seq_t seq, char *range_info); + bool (*backend_skip_record)(range_seq_t seq, char *range_info, uchar *rowid); + void *backend_init_params; + range_seq_t backend_seq; + + /* RANGE_SEQ_IF override: our hook-functions */ + static range_seq_t hook_init(void *init_params, uint n_ranges, uint flags); + static bool hook_next(range_seq_t seq, KEY_MULTI_RANGE *range); + static void hook_get_key_info(void *init_params, uint *length, key_part_map *map); + static bool hook_skip_index_tuple(range_seq_t seq, char *range_info); + static bool hook_skip_record(range_seq_t seq, char *range_info, uchar *rowid); + + /* Mapper [de]initialization functions*/ + void range_id_mapping_ctor() + { + range_ids= NULL; + n_range_ids= 0; + } + void setup_range_id_mapping(RANGE_SEQ_IF *rseq, void **init_params, uint n_ranges); + void clear_range_id_mapping(); + + /* The reason of all of the above: function to map range ids to range numbers */ + int range_id_to_number(char *range_id); +#endif }; /** _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] dgcov.pl
by Kristian Nielsen 02 Mar '11

02 Mar '11

Hi Serg, I pushed to lp:mariadb-tools a "fix" for the negative numbers output by gcov which causes dgcov.pl to break. (Sergey Petrunya needed it). It just strips the leading '-' from negative numbers, so it does not help to understand what the problem is, I would still like to figure that out... but meanwhile, I hope you are ok with the workaround, or at least I wanted to make you aware of it. - Kristian.

1 0

Re: [Maria-developers] Release of MariaDB 5.1.55
by Daniel Bartholomew 02 Mar '11

02 Mar '11

On Mon, 28 Feb 2011 12:40:13 +0100 Kristian Nielsen <knielsen(a)knielsen-hq.org> wrote: Kristian> Monty gave the go for releasing MariaDB 5.1.55. Thanks Kristian for all of your work in getting the packages ready, especially the new .deb repository. And thank you to all of the developers who worked on this release! Here are the download, release notes, and changelog pages for 5.1.55: * http://downloads.askmonty.org/mariadb/5.1.55 * http://kb.askmonty.org/v/mariadb-5155-release-notes * http://kb.askmonty.org/v/mariadb-5155-changelog All: If you have a moment, please check over the above pages to see if there are any mistakes or omissions and let me know (or jump in and fix if you have editing rights). One brief note about the new downloads system: I'll also be adding some redirects to the old wiki-based downloads site to direct people to the new downloads site (in case they have the old site bookmarked). Community team: I tested a sample of the mirrors and they all had the files so please announce the release on the announce list, blogs, and so on. After sending out this email I will add an entry to the releases RSS feed. Thanks. -- Daniel Bartholomew MariaDB - http://mariadb.org Monty Program - http://montyprogram.com AskMonty Knowledgebase - http://kb.askmonty.org

2 1

[Maria-developers] Buildbot Update
by Adam M. Dutko 01 Mar '11

01 Mar '11

All of the buildbots I run should be back online.

2 2

[Maria-developers] Igor please review: [Commits] Rev 2926: BUG#724275: Crash in JOIN::optimize in maria-5.3 in file:///home/psergey/dev2/5.3-2/]
by Sergey Petrunya 28 Feb '11

28 Feb '11

Hello Igor, Could you please review the below: ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- From: Sergey Petrunya <psergey(a)askmonty.org> To: commits(a)mariadb.org X-Mailer: mail (GNU Mailutils 1.2) Date: Tue, 1 Mar 2011 00:30:02 +0300 (MSK) Subject: [Commits] Rev 2926: BUG#724275: Crash in JOIN::optimize in maria-5.3 in file:///home/psergey/dev2/5.3-2/ At file:///home/psergey/dev2/5.3-2/ ------------------------------------------------------------ revno: 2926 revision-id: psergey(a)askmonty.org-20110228212959-hl3selurcdou7d22 parent: igor(a)askmonty.org-20110227181411-k81vmi35o0no21ab committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: 5.3-2 timestamp: Tue 2011-03-01 00:29:59 +0300 message: BUG#724275: Crash in JOIN::optimize in maria-5.3 - Make equality-substitution-for-ref-access code in JOIN::optimize() treat join_tab->ref.key_copy correctly (in the way create_ref_for_key() has filled it). === modified file 'mysql-test/r/join.result' --- a/mysql-test/r/join.result 2010-10-30 13:07:45 +0000 +++ b/mysql-test/r/join.result 2011-02-28 21:29:59 +0000 @@ -1251,3 +1251,37 @@ Handler_read_rnd_next 1 DROP TABLE t1, t2; End of 5.1 tests +# +# BUG#724275: Crash in JOIN::optimize in maria-5.3 +# +create table t1 (a int); +insert into t1 values (1),(2); +insert into t1 select * from t1; +create table t2 (a int, b int, key(a,b)); +insert into t2 values (1,1),(1,2),(1,3),(1,4),(2,5),(2,6),(2,7),(2,8),(2,9); +insert into t2 select * from t2; +insert into t2 select * from t2; +insert into t2 select * from t2; +create table t3 (a int, b int, key(a)); +insert into t3 values (1,1),(2,2); +select * from +t3 straight_join t1 straight_join t2 force index(a) +where t2.a=1 and t2.b=t1.a and t1.a=t3.b and t3.a=1; +a b a a b +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +drop table t1,t2,t3; === modified file 'mysql-test/t/join.test' --- a/mysql-test/t/join.test 2010-10-30 13:07:45 +0000 +++ b/mysql-test/t/join.test 2011-02-28 21:29:59 +0000 @@ -941,3 +941,26 @@ DROP TABLE t1, t2; --echo End of 5.1 tests + +--echo # +--echo # BUG#724275: Crash in JOIN::optimize in maria-5.3 +--echo # + +create table t1 (a int); +insert into t1 values (1),(2); +insert into t1 select * from t1; + +create table t2 (a int, b int, key(a,b)); +insert into t2 values (1,1),(1,2),(1,3),(1,4),(2,5),(2,6),(2,7),(2,8),(2,9); +insert into t2 select * from t2; +insert into t2 select * from t2; +insert into t2 select * from t2; + +create table t3 (a int, b int, key(a)); +insert into t3 values (1,1),(2,2); +select * from + t3 straight_join t1 straight_join t2 force index(a) +where t2.a=1 and t2.b=t1.a and t1.a=t3.b and t3.a=1; + +drop table t1,t2,t3; + === modified file 'sql/sql_select.cc' --- a/sql/sql_select.cc 2011-02-27 18:14:11 +0000 +++ b/sql/sql_select.cc 2011-02-28 21:29:59 +0000 @@ -1056,6 +1056,7 @@ */ for (JOIN_TAB *tab= join_tab + const_tables; tab < join_tab + tables; tab++) { + uint key_copy_index=0; for (uint i=0; i < tab->ref.key_parts; i++) { @@ -1071,13 +1072,14 @@ { *ref_item_ptr= ref_item; Item *item= ref_item->real_item(); - store_key *key_copy= tab->ref.key_copy[i]; + store_key *key_copy= tab->ref.key_copy[key_copy_index]; if (key_copy->type() == store_key::FIELD_STORE_KEY) { store_key_field *field_copy= ((store_key_field *)key_copy); field_copy->change_source_field((Item_field *) item); } } + key_copy_index++; } } _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Buildbot Outage
by Adam M. Dutko 28 Feb '11

28 Feb '11

It is possible the buildbots I sponsor are offline today: mariadb-brs, adutko-centos-amd64 and adutko-ultrasparc3. The transformer that feeds our house was hit by lightning last night. It was still operational this morning but I believe the power company is replacing it today.

1 0

[Maria-developers] Fix for PBXT for problem with 'repair-pending'
by Michael Widenius 25 Feb '11

25 Feb '11

Hi! We have got some failures of the following type in pushbuild: pbxt.mysqlslap w2 [ fail ] Test ended at 2011-02-25 14:37:57 CURRENT_TEST: pbxt.mysqlslap mysqltest: At line 43: query 'drop database pbxt' failed: 1010: Error dropping database (can't rmdir './pbxt', errno: 39) The reason is that at the end of the test there was left in the pbxt directory a file "repair-pending" with a reference to some old already deleted table. There was nothing else in the directory. My assumption (and the code seams to agree with it) is that when you delete a table, you don't remove the table from the repair-pending file even if it had been recorded there. Here is a patch that fixes this: === modified file 'storage/pbxt/src/table_xt.cc' --- storage/pbxt/src/table_xt.cc 2010-11-26 22:37:34 +0000 +++ storage/pbxt/src/table_xt.cc 2011-02-25 14:40:53 +0000 @@ -1755,6 +1755,8 @@ xtPublic void xt_drop_table(XTThreadPtr tab_close_mapped_files(self, tab); tab_delete_table_files(self, tab_name, tab_id); + /* Remove table from "repair-pending" */ + xt_tab_table_repaired(tab); ASSERT(xt_get_self() == self); if ((te_ptr = (XTTableEntryPtr) xt_sl_find(self, db->db_ Regards, Monty

2 1

[Maria-developers] Please consider LP BUG#725050 - related to join_cache_hashed
by Timour Katchaounov 25 Feb '11

25 Feb '11

Igor, the following bug is present in different ways both in 5.3 and in 5.3-mwl89. https://bugs.launchpad.net/maria/+bug/725050 The crash in 5.3-mwl89 appeared after merging with the latest 5.3. The crash in 5.3 requires a rather artificial example, but it shows that there is some inconsistency in JOIN::keyuse in 5.3. Please look at the last bug comment - it contains a very small test case. I will ping you tonight after putting kids to sleep around 21:30. Timour

1 0

[Maria-developers] [psergey@askmonty.org: [Commits] Rev 2921: BUG#723822: Crash in get_constant_key_infix with EXISTS ( SELECT .. DISTINCT ) in file:///home/psergey/dev2/5.3/]
by Sergey Petrunya 25 Feb '11

25 Feb '11

Hi Timour, Could you please review the below? (Asking you because the fix is in the loose index scan optimizer) ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- From: Sergey Petrunya <psergey(a)askmonty.org> To: commits(a)mariadb.org X-Mailer: mail (GNU Mailutils 1.2) Date: Fri, 25 Feb 2011 12:59:00 +0300 (MSK) Subject: [Commits] Rev 2921: BUG#723822: Crash in get_constant_key_infix with EXISTS ( SELECT .. DISTINCT ) in file:///home/psergey/dev2/5.3/ At file:///home/psergey/dev2/5.3/ ------------------------------------------------------------ revno: 2921 revision-id: psergey(a)askmonty.org-20110225095856-qgykwziqv5h1suqr parent: psergey(a)askmonty.org-20110224203200-4uf4e4kngwdmtmgw committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: 5.3 timestamp: Fri 2011-02-25 12:58:56 +0300 message: BUG#723822: Crash in get_constant_key_infix with EXISTS ( SELECT .. DISTINCT ) - Make get_constant_key_infix() take into account that there may be SEL_TREEs with type=SEL_ARG::MAYBE_KEY, which it cannot process. check_quick_select() already has a check for such condition. === modified file 'mysql-test/r/subselect4.result' --- a/mysql-test/r/subselect4.result 2011-02-10 08:36:43 +0000 +++ b/mysql-test/r/subselect4.result 2011-02-25 09:58:56 +0000 @@ -689,3 +689,21 @@ 2 DEPENDENT SUBQUERY X ALL NULL NULL NULL NULL 10 2 DEPENDENT SUBQUERY B ALL a,b NULL NULL NULL 1000 Range checked for each record (index map: 0x3) drop table t1, t2; +# +# BUG#723822: Crash in get_constant_key_infix with EXISTS ( SELECT .. DISTINCT ) +# +CREATE TABLE t1 ( f1 int(11), f3 varchar(1)) ; +INSERT INTO t1 VALUES ('8','c'),('5','f'); +ALTER TABLE t1 ADD KEY (f3,f1); +CREATE TABLE t2 ( f4 varchar(1)) ; +INSERT INTO t2 VALUES ('f'),('d'); +SELECT * FROM t2 +WHERE EXISTS ( +SELECT DISTINCT f3 +FROM t1 +WHERE f3 <= t2.f4 +); +f4 +f +d +drop table t1,t2; === modified file 'mysql-test/t/subselect4.test' --- a/mysql-test/t/subselect4.test 2011-02-10 08:36:43 +0000 +++ b/mysql-test/t/subselect4.test 2011-02-25 09:58:56 +0000 @@ -619,3 +619,24 @@ from t1 A; drop table t1, t2; + +--echo # +--echo # BUG#723822: Crash in get_constant_key_infix with EXISTS ( SELECT .. DISTINCT ) +--echo # +CREATE TABLE t1 ( f1 int(11), f3 varchar(1)) ; +INSERT INTO t1 VALUES ('8','c'),('5','f'); + +ALTER TABLE t1 ADD KEY (f3,f1); + +CREATE TABLE t2 ( f4 varchar(1)) ; +INSERT INTO t2 VALUES ('f'),('d'); + +SELECT * FROM t2 +WHERE EXISTS ( + SELECT DISTINCT f3 + FROM t1 + WHERE f3 <= t2.f4 +); + +drop table t1,t2; + === modified file 'sql/opt_range.cc' --- a/sql/opt_range.cc 2011-02-22 09:15:47 +0000 +++ b/sql/opt_range.cc 2011-02-25 09:58:56 +0000 @@ -11857,7 +11857,8 @@ Find the range tree for the current keypart. We assume that index_range_tree points to the leftmost keypart in the index. */ - for (cur_range= index_range_tree; cur_range; + for (cur_range= index_range_tree; + cur_range && cur_range->type == SEL_ARG::KEY_RANGE; cur_range= cur_range->next_key_part) { if (cur_range->field->eq(cur_part->field)) _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Using tilde in the version for debs?
by Henrik Ingo 24 Feb '11

24 Feb '11

Hi Kristian I've been a bit invisible for a few months now, as I had to spend time on job interviews, family and luckily also some coding! By March life is turning back to normal with a new job and everything. As you know by now, I have in the past months fulfilled my desire to spend some hands on time with the MariaDB build system. I intend to blog about it in the coming week, after which I'll email some MariaDB specific details about what improvements you may want to consider incorporating, and what I think are great improvements but probably not worth focusing on for you in the short term. But just as a quickie, I noticed you are changing the versioning of debs to include the distribution codename (lucid, maverick, etc..): http://bazaar.launchpad.net/~maria-captains/ourdelta/ourdelta-knielsen/revi… What caught my eye was the usage of tilde character for punctuation. In dpkg tilde is a special character with superpowers, it is sorted as greater than anything else: http://man.he.net/man5/deb-version You probably know that, of course, and have a good reason to use it (related to getting upgrades right, perhaps?). I just wanted to comment on it "just in case". "In case" there is no particular reason to use tilde, then of course using a dot would have been more natural. Percona uses a dot http://www.percona.com/downloads/Percona-Server-5.1/Percona-Server-5.1.54-1… ...and I use that with the MepSQL packages too. As a final observation - apart from the different puntuation, the version string seems to be similar in all three of percona, mepsql and ourdelta-knielsen. (So it is kind of a pity if you are forced to use the tilde for some reason - otoh MariaDB versioning is a bit different anyway as it doesn't follow the <mysql-base-ver>-<my-release-ver> approach.) cheers and I'll get back later with more interesting comments. henrik -- henrik.ingo(a)avoinelama.fi +358-40-5697354 skype: henrik.ingo irc: hingo www.openlife.cc My LinkedIn profile: http://www.linkedin.com/profile/view?id=9522559

2 2

Re: [Maria-developers] [Commits] Rev 2875: The patch adds the code that allows to use equi-join conditions in file:///home/igor/maria/maria-5.3-mwl128-hjkey/
by Timour Katchaounov 24 Feb '11

24 Feb '11

Igor, While merging 5.3-mwl89 with 5.3 I noticed that when you extracted a block of code from add_key_part() into the new function add_keyuse(), you skipped the initialization: keyuse.ref_table_rows= 0; was it intentional, and if so, why? Timour > At file:///home/igor/maria/maria-5.3-mwl128-hjkey/ > > ------------------------------------------------------------ > revno: 2875 > revision-id: igor(a)askmonty.org-20110105055941-xht2mpqe66a64fbm > parent: igor(a)askmonty.org-20101230021403-ly4ku1yq675l5d6b > committer: Igor Babaev<igor(a)askmonty.org> > branch nick: maria-5.3-mwl128-hjkey > timestamp: Tue 2011-01-04 21:59:41 -0800 > message: > The patch adds the code that allows to use equi-join conditions > for hash join in the cases when there are no suitable indexes > for these conditions. > > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] Fwd: [Commits] Rev 2921: Added build script which works with ICC v12.0 in file:///home/hakan/work/monty_program/maria-5.2/
by Kristian Nielsen 24 Feb '11

24 Feb '11

Hakan Kuecuekyilmaz <hakan(a)askmonty.org> writes: > Kristian, can you please review the new build script, so that I can > push it into our 5.2 tree. After we cleared the license details with Ok to push. - Kristian. > Begin forwarded message: > >> From: "Hakan Kuecuekyilmaz" <hakan.kuecuekyilmaz(a)t-online.de> >> Date: 10. Februar 2011 16:41:30 MEZ >> To: <commits(a)mariadb.org> >> Subject: [Commits] Rev 2921: Added build script which works with ICC v12.0 in file:///home/hakan/work/monty_program/maria-5.2/ >> Reply-To: maria-developers(a)lists.launchpad.net >> >> At file:///home/hakan/work/monty_program/maria-5.2/ >> >> ------------------------------------------------------------ >> revno: 2921 >> revision-id: hakan(a)askmonty.org-20110210154128-ncaubaj5i8rhf2pe >> parent: monty(a)askmonty.org-20110203144529-ah7majfmqscp4yn1 >> committer: Hakan Kuecuekyilmaz <hakan(a)askmonty.org> >> branch nick: maria-5.2 >> timestamp: Thu 2011-02-10 16:41:28 +0100 >> message: >> Added build script which works with ICC v12.0 >> === added file 'BUILD/compile-pentium64-icc-max' >> --- a/BUILD/compile-pentium64-icc-max 1970-01-01 00:00:00 +0000 >> +++ b/BUILD/compile-pentium64-icc-max 2011-02-10 15:41:28 +0000 >> @@ -0,0 +1,25 @@ >> +#! /bin/sh >> + >> +path=`dirname $0` >> +. "$path/SETUP.sh" >> + >> +# Note that we can't use ccache with icc as the generated .deps file will >> +# then contain wrong information. >> +CC=icc >> +CXX=icpc >> +LD=xild >> +AR=xiar >> +export CC CXX LD AR >> + >> +c_warnings="" >> +cxx_warnings="" >> +extra_flags="-O2 -unroll2 -fno-exceptions -fno-rtti -fno-omit-frame-pointer -static-intel -static-libgcc -fPIE" >> + >> +# Use -no-ipo if you get this error >> +# IPO link: can not find "-lstdc++_shared" >> +# icpc: error: problem during multi-file optimization compilation (code 1) >> +base_cxxflags="$extra_flags" >> +extra_configs="$pentium_configs" >> + >> +. "$path/FINISH.sh" >> +

1 0

[Maria-developers] WL#177 New (by Mdcallag): add server-side error logging
by worklog-noreply＠askmonty.org 22 Feb '11

22 Feb '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: add server-side error logging CREATION DATE..: Tue, 22 Feb 2011, 19:42 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 177 (http://askmonty.org/worklog/?tid=177) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: I want to log all or a sample of errors on the server side into a new log file (not general, error or slow query log) and into a file that can be rotated. In the facebook patch I added code to sample query errors into the slow query log using - -log-sample-error-rate: * when 0 do not sample * when 1 then log all errors * when > 1 then sample 1 of --log-sample-error-rate queries This should have: * query text * username * timestamp * database * errno * error text * client IP The log should support rotation and possibly auto-rotation when it reaches a certain size. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

Re: [Maria-developers] Fwd: [Commits] Rev 2901: BNLH algorithm always used a full table scan over the joined table in file:///home/igor/maria/maria-5.3-mwl128-hashrange/
by Sergey Petrunya 22 Feb '11

22 Feb '11

On Fri, Feb 11, 2011 at 08:49:37PM -0800, Igor Babaev wrote: > Hi Sergey, > > Please review this patch that fixes a defect of the current > implementation of the mwl #128 that never couples hash join > with range/index_merge scans. > The patch also adds prefix #hash# to the names of the used hash indexes. > (we agreed upon this at the last optimizer meeting). > > Regards, > Igor. > > -------- Original Message -------- > Subject: [Commits] Rev 2901: BNLH algorithm always used a full table > scan over the joined table in > file:///home/igor/maria/maria-5.3-mwl128-hashrange/ > Date: Fri, 11 Feb 2011 20:41:23 -0800 (PST) > From: Igor Babaev <igor(a)askmonty.org> > Reply-To: maria-developers(a)lists.launchpad.net > To: <commits(a)mariadb.org> > > At file:///home/igor/maria/maria-5.3-mwl128-hashrange/ > > ------------------------------------------------------------ > revno: 2901 > revision-id: igor(a)askmonty.org-20110212044122-w9n3jdk3d2ps0w3o > parent: igor(a)askmonty.org-20110207231903-3rqbfs50d33lk3r9 > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-mwl128-hashrange > timestamp: Fri 2011-02-11 20:41:22 -0800 > message: > BNLH algorithm always used a full table scan over the joined table > even in the cases when there existed range/index-merge scans that > were cheaper than the full table scan. > This was a defect/bug of the implementation of mwl #128. > Now hash join can work not only with full table scan of the joined > table, but also with full index scan, range and index-merge scans. > Accordingly, in the cases when hash join is used the column 'type' > in the EXPLAINs can contain now 'hash_ALL', 'hash_index', 'hash_range' > and 'hash_index_merge'. If hash join is coupled with a range/index_merge > scan then the columns 'key' and 'key_len' contain info not only on > the used hash index, but also on the indexes used for the scan. ... > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2011-02-06 04:57:03 +0000 > +++ b/sql/sql_select.cc 2011-02-12 04:41:22 +0000 ... > @@ -8419,6 +8438,11 @@ > sort_by_tab->type= JT_ALL; > sort_by_tab->read_first_record= join_init_read_record; > } > + else if (sort_by_tab->type == JT_HASH_NEXT) > + { > + sort_by_tab->type= JT_HASH; > + sort_by_tab->read_first_record= join_init_read_record; > + } > } > break; > } I have a question only to this part of the patch: I don't understand how the above code could work. The if statement above the one you've added checks if sort_by_tab was using [ordered] index scan to produce records in orderer, and if yes, switches it to a full scan (because use of join buffering will break ordering anyway). I suppose the part you've added does something similar, but I totally fail to understand what exactly it does. Now, some comments on the new EXPLAIN output: ISSUE1. Consider a query: MariaDB [j1]> explain select * from t3, t4 where t3.col1=t4.col2; +----+-------------+-------+----------+---------------+-----------+---------+------------+------+--------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+----------+---------------+-----------+---------+------------+------+--------------------------------------------------+ | 1 | SIMPLE | t3 | ALL | NULL | NULL | NULL | NULL | 1000 | Using where | | 1 | SIMPLE | t4 | hash_ALL | NULL | #hash#$hj | 5 | j1.t3.col1 | 1000 | Using where; Using join buffer (flat, BNLH join) | +----+-------------+-------+----------+---------------+-----------+---------+------------+------+--------------------------------------------------+ 2 rows in set (0.00 sec) AFAIU #hash#$hj means the hash key on the hashtable that's used for hash join. I think the name is too convoluted. The hashtable has only one key, if we need to tell it from t4's indexes it would be sufficient to use "#hashkey#", without the cryptic "$hj". As for "#" characters: EXPLAIN already shows "internal" tables/indexes: MariaDB [j1]> explain select * from (select * from t1) X; +----+-------------+------------+------+---------------+------+---------+------+------+-------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+------+---------------+------+---------+------+------+-------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 1000 | | | 2 | DERIVED | t1 | ALL | NULL | NULL | NULL | NULL | 1000 | | +----+-------------+------------+------+---------------+------+---------+------+------+-------+ # The following is from MWL#90 tree: MariaDB [test]> explain select * from t10 where a in (select max(b) from t10 group by a); +----+-------------+-------------+--------+---------------+--------------+---------+------------+------+---------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------------+--------+---------------+--------------+---------+------------+------+---------------------------------+ | 1 | PRIMARY | t10 | ALL | NULL | NULL | NULL | NULL | 1000 | Using where | | 1 | PRIMARY | <subquery2> | eq_ref | distinct_key | distinct_key | 5 | test.t10.a | 1 | | | 2 | SUBQUERY | t10 | ALL | NULL | NULL | NULL | NULL | 1000 | Using temporary; Using filesort | +----+-------------+-------------+--------+---------------+--------------+---------+------------+------+---------------------------------+ 3 rows in set (0.01 sec) # The following is from current 5.3: MariaDB [j1]> explain select distinct * from t1 where a in (select b from t2); +----+-------------+------------+--------+---------------+------------+---------+------+------+-----------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+--------+---------------+------------+---------+------+------+-----------------+ | 1 | PRIMARY | t1 | ALL | NULL | NULL | NULL | NULL | 1000 | Using temporary | | 1 | PRIMARY | subselect2 | eq_ref | unique_key | unique_key | 5 | func | 1 | | | 2 | SUBQUERY | t2 | ALL | NULL | NULL | NULL | NULL | 1000 | Distinct | +----+-------------+------------+--------+---------------+------------+---------+------+------+-----------------+ ... and it either uses no quoting at all (see "unique_key"), or angle brackets quoting (see <subquery2>). I think we should not multiply the ways of quoting, so please consider switching to <quoting> or to not using any quoting at all. ISSUE2 MariaDB [j1]> EXPLAIN -> SELECT City.Name, Country.Name, CountryLanguage.Language -> FROM City,Country,CountryLanguage -> WHERE City.Country=Country.Code AND -> CountryLanguage.Country=Country.Code AND -> City.Name LIKE 'L%' AND Country.Population > 3000000 AND -> CountryLanguage.Percentage > 50 AND -> LENGTH(Language) < LENGTH(City.Name) - 2; +----+-------------+-----------------+----------+--------------------+---------------+---------+----------------------------+------+--------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-----------------+----------+--------------------+---------------+---------+----------------------------+------+--------------------------------------------------+ | 1 | SIMPLE | CountryLanguage | ALL | PRIMARY,Percentage | NULL | NULL | NULL | 984 | Using where | | 1 | SIMPLE | Country | hash_ALL | PRIMARY | #hash#PRIMARY | 3 | j1.CountryLanguage.Country | 239 | Using where; Using join buffer (flat, BNLH join) | | 1 | SIMPLE | City | hash_ALL | Country | #hash#Country | 3 | j1.CountryLanguage.Country | 4079 | Using where; Using join buffer (flat, BNLH join) | +----+-------------+-----------------+----------+--------------------+---------------+---------+----------------------------+------+--------------------------------------------------+ 3 rows in set (0.01 sec) "key" column shows "#hash#PRIMARY". I suppose, this is because hash join opimizer is internally hooked into ref optimizer, and it used the PRIMARY index. This might be useful for developer, but is this information meaningful for the user? If one changes #hash#PRIMARY to #hash#IDX2 (suppose there is an index called IDX2 which has the same definition as primary key), or to #hash#$hj, will it be a different query plan? I think it should be "<hashkey>" always. ISSUE3 For hash_range, "key" column shows {hashkey}:{range_key}, which allows to determine that which key was used for range access. For hash_index type, "key" column shows only the hash's pseudo-index: MariaDB [j3]> EXPLAIN SELECT t2.i FROM t1,t2 WHERE t1.cu = t2.cl; +----+-------------+-------+------------+---------------+-----------+---------+------+------+--------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------------+---------------+-----------+---------+------+------+--------------------------------------------------+ | 1 | SIMPLE | t2 | ALL | NULL | NULL | NULL | NULL | 6 | | | 1 | SIMPLE | t1 | hash_index | cu1,cu2 | #hash#cu1 | 33 | func | 10 | Using where; Using join buffer (flat, BNLH join) | +----+-------------+-------+------------+---------------+-----------+---------+------+------+--------------------------------------------------+ What is the index that's used for scanning the table t1? "#hash#cu1" shows that hash join optimizer used access on index "cu1" in its analysis, but generally it doesn't mean that full index scan on table t1 was done on index cu1, doesn't it? I would very much prefer to see #hash#cu1:USED_INDEX there (or taking into account previous suggestions, hashkey:USED_INDEX). It will also be consistent with how hash_range and hash_index_merge use that column. ISSUE4 If key and key_len use hash_part:real_part notation, i.e. use semicolon, why use underscore in hash_ALL, hash_index, etc? Won't hash:ALL, hash:index look more consistent with other columns? BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 2

Re: [Maria-developers] [Commits] Rev 2833: Fixed LP bug #664508. in file:///home/igor/maria/maria-5.3-mwl128-bug664508/
by Sergey Petrunya 21 Feb '11

21 Feb '11

Hello Igor, On Sun, Oct 24, 2010 at 02:22:02PM -0700, Igor Babaev wrote: > At file:///home/igor/maria/maria-5.3-mwl128-bug664508/ > > ------------------------------------------------------------ > revno: 2833 > revision-id: igor(a)askmonty.org-20101024212201-t8iius4jikcvm3zb > parent: igor(a)askmonty.org-20101022223047-phunrh5g3xzrx2d9 > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-mwl128-bug664508 > timestamp: Sun 2010-10-24 14:22:01 -0700 > message: > Fixed LP bug #664508. > When join buffers are employed no index scan for the first > table with grouping columns can be used. ... > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2010-10-18 20:56:48 +0000 > +++ b/sql/sql_select.cc 2010-10-24 21:22:01 +0000 > @@ -7980,8 +7980,11 @@ > JOIN_TAB *tab=join->join_tab+i; > if (tab->use_join_cache) > { > - JOIN_TAB *sort_by_tab= join->get_sort_by_join_tab(); > - if (sort_by_tab && !join->need_tmp) > + JOIN_TAB *sort_by_tab= join->group && join->simple_group && > + join->group_list ? > + join->join_tab+join->const_tables : > + join->get_sort_by_join_tab(); > + if (sort_by_tab) > { > join->need_tmp= 1; > join->simple_order= join->simple_group= 0; Here sort_by_tab is assigned inside the " for (i=join->const_tables ; ..." loop. I think the code will look much more cleaner if it was calculated before the loop (and actually we don't need to loop at all if sort_by_tab==NULL). > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB Binary Log File Corruption
by Daniel Bartholomew 20 Feb '11

20 Feb '11

Hello everyone, I came across this question in the Knowledgebase: http://kb.askmonty.org/v/binary-log-file-corruption-happens-each-day The poster was able to solve the problem by increasing the slave max_allowed_packet size from 32M to 256M. Any thoughts from the list about this? My guess is that this not a MariaDB-specific thing; just a difference between 5.0 and 5.1 (and higher). But I'm asking anyway in case I'm mistaken and/or if a note should be added to the documentation for max_allowed_packet. Thanks. -- Daniel Bartholomew MariaDB - http://mariadb.org Monty Program - http://montyprogram.com AskMonty Knowledgebase - http://kb.askmonty.org

2 1

[Maria-developers] question about when locks are released
by MARK CALLAGHAN 18 Feb '11

18 Feb '11

Vamsi, In the innodb-release-locks-early patch, do you remember why you release the locks only _after_ the sync in prepare(), and not before? -- Mark Callaghan mdcallag(a)gmail.com

1 1

[Maria-developers] WL#176 New (by Dreas): protocl typo in replication error
by worklog-noreply＠askmonty.org 17 Feb '11

17 Feb '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: protocl typo in replication error CREATION DATE..: Thu, 17 Feb 2011, 16:58 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Maria-RawIdeaBin TASK ID........: 176 (http://askmonty.org/worklog/?tid=176) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 10 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Using 5.1.50-mariadb84: ===== Last_IO_Error: error connecting to master 'repl2@mysql1.*****.com:3306' - retry- time: 60 retries: 86400 message: SSL error: protocl version mismatch ==== "protocl" seems to be a typo. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] work in progress for the Facebook patch
by MARK CALLAGHAN 16 Feb '11

16 Feb '11

I am writing this to describe work in progress for the Facebook patch. Comments are welcome. In the future we hope to get parts of the Facebook patch into public forks. We have two big projects in progress: * admission control * InnoDB compression for OLTP --- Admission control enforces max concurrent queries per account. MySQL already limits max concurrent connections per account, but we want to set the connection limit to a much larger value than the query limit. The limit will be enforced at statement start and will allow queries to timeout in the queue while waiting to run. By "descheduled" I mean that a connection decrements the count of concurrent queries for a particular account. Connections will be descheduled when: * a statement ends * a connection is about to send results back to a client -- only when the socket write is to be done, not when data is buffered * srv_suspend_mysql_thread is called on a row lock wait * possibly when a disk read is done within innodb * when enter_cond is called By "scheduled" I mean that a connection increments the count of concurrent queries for an account. If done with "noforce" then the caller will wait until this doesn't exceed concurrency limits for the account. If done with "force" then this doesn't wait and can exceed the concurrency limits. Schedule with noforce is done: * at statement start * after a socket write Schedule with force is done: * when exit_cond is called * when a thread wakes after calling srv_suspend_mysql_thread This change will allow us to run with higher values for max concurrent connections and we assume that will significantly increase mutex contention on kernel_mutex so we have begun to evaluate: * using the pthread mutex implementation that does not spin in more cases * limit the max number of threads that can do concurrent busy-wait loops for InnoDB's mutex and rw-lock * removing use of the sync array within InnoDB * changing Innodb to wake one rather than all waiters when a mutex is unlocked * somehow making read_view_open_now faster --- InnoDB compression for OLTP tries to make compression faster for OLTP workloads. We have work in progress to change InnoDB to not log page images for many but not all insert/update/delete operations. When InnoDB compression is used, InnoDB can log page images. That was not done before and for my production workloads this doubles the rate at which IO is done to the transaction log and increases the dirty page write rate by at least 20% because checkpoint IO must be done sooner. We previously added an option that allows the LZ compression level to be set globally. I think the default is level 6. We will evaluate: * using QuickLZ which provides faster compression and decompression than zlib * adding compress_ops and compress_ops_ok per table to information_schema.table_statistics * adding an option to limit compression to the clustered index and not compress secondary indexes -- Mark Callaghan mdcallag(a)gmail.com

1 0

[Maria-developers] MariaDB and MySQL 5.5
by MARK CALLAGHAN 11 Feb '11

11 Feb '11

When will there be a MariaDB branch based on MySQL 5.5? I want to use that as the basis for the 5.5 version of the Facebook patch. -- Mark Callaghan mdcallag(a)gmail.com

2 1

[Maria-developers] WL#175 New (by Mdcallag): Unique ID in binlog event metadata
by worklog-noreply＠askmonty.org 11 Feb '11

11 Feb '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Unique ID in binlog event metadata CREATION DATE..: Fri, 11 Feb 2011, 05:03 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 175 (http://askmonty.org/worklog/?tid=175) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: While I would love to have global transaction IDs for MySQL replication (http://askmonty.org/worklog/Server-RawIdeaBin/?tid=31) that is a huge project. I think we can automate slave failover if binlog events have a metadata attribute that is a unique ID and that metadata is sticky like server ID when a slave is run with --log-slave-updates. Assuming this metadata is added, when it is time to do slave failover, first determine the serverID + uniqueID for the last binlog event group executed on the slave, then find that filename/offset for that event on the new master, then CHANGE MASTER on the slave to start replication on the new master just past that offset. I mention that serverID can be used in addition to the uniqueID, so maybe uniqueID only needs to be unique within the binlog events generated for one serverID. The search on the new master is more efficient when the uniqueID are generated in increasing order. But I am not sure whether that is required. We might need a new server command that maps (serverID, uniqueID) to (filename, offset), but I am not sure that is required. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

Re: [Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 08 Feb '11

08 Feb '11

> -----Original Message----- > From: kahzoo(a)gmail.com [mailto:kahzoo@gmail.com] On Behalf Of Kazuhisa > Ichikawa > Sent: Dienstag, 8. Februar 2011 04:46 > To: Vladislav Vaintroub > Cc: Philip Stoev; maria-developers(a)lists.launchpad.net > Subject: Re: [Maria-developers] Windows installer MWL#55 finished. > > Hello, Vladislav. > All previous 5.2.5 installer created its data directory even when > upgrading existing instance. Therefore, if datadir was not set > explicity in the inherited my.ini, it was using the new data directory > (in other words, it was not reusing the previous database files, I'm > afraid). Yes. Somewhat unusual case my.ini without datadir was not handled correctly though (case without my.ini at all was handled). Fixed that, MSIs are in the usual location. Thanks for input! > - Kazuhisa

1 0

Re: [Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 05 Feb '11

05 Feb '11

> -----Original Message----- > From: kahzoo(a)gmail.com [mailto:kahzoo@gmail.com] On Behalf Of Kazuhisa > Ichikawa > Sent: Samstag, 5. Februar 2011 04:34 > To: Vladislav Vaintroub > Cc: maria-developers(a)lists.launchpad.net > Subject: Re: [Maria-developers] Windows installer MWL#55 finished. Hello Kazuhisa, > I BROWSED through it but did not read it carefully line by line before > reporting bugs. Sorry about that. Yet, all of your reports were valid . > > By the way, after finished installing 5.2.5 and upgrading the previous > (5.1.53) service instance, I have 2 service definitions for 5.2.5, one > that was newly created and the other that was successfully upgraded > from 5.1.53. > > I think this is a human (that is my) error rather than a bug, because > I checked "Install as service" in the "Defaults settings" dialog > before the installer starts installing 5.2.5 files and then chose to > upgrade the existing service in the Upgrade Wizard that was started > after installation. If you think installing instance was a mistake on you part (you only wanted to upgrade what you have), you can correct that easily. In maintenance mode install (Add/Remove Programs=>MariaDB5.2=>Change/Uninstall), select "Change" button. Deselect "Database Instance". Choose "Remove data" afterwards. Now the mistakenly installed instance is gone. > My expectation was that the installer and upgrade > wizard will install a new service instance based on the information > from the existing service definition. Hmm, have you see that in any of MySQL installers ;)? > If I could (mutually) exclusively select between "installing a new > service" and "upgrading from existing service" in the same dialog, I > could have avoided this mis-configuration. I understand, and agree from usability point of view this would be ideal. Currently, "Database Instance" feature can only create new databases, while "Upgrade Wizard" (runs outside of the installer) upgrades existing instances. Their functionality is non-overlapping. This allows for painless (for user) and simple to implement installer rollback in case of errors. So it will stay this way for the version 1 of the new installer. One thing that is possible to do to handle your case, is to deselect "database instance" by default, if upgradable services are detected. I'm not sure this is the best way to handle it , and whether it would confuse the user thus I'll consult our QA for his opinion. Philip, what do you think? Vladislav

1 0

Re: [Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 04 Feb '11

04 Feb '11

> -----Original Message----- > From: kahzoo(a)gmail.com [mailto:kahzoo@gmail.com] On Behalf Of Kazuhisa Ichikawa > Sent: Freitag, 4. Februar 2011 03:17 > To: Vladislav Vaintroub > Cc: maria-developers(a)lists.launchpad.net > Subject: Re: [Maria-developers] Windows installer MWL#55 finished. > > Hello Vladislav, Hello Kazuhisa, > First of all, thanks for all the information on "MSI v.s. NSIS" issues. > > > On a bright side, this is a valid bug that was simple to fix forcing per-machine install, so installer does not need to guess. I > > did that and refreshed the MSI on the download page. > > I confirmed the fix with the same machine/user with which I had the > original problem. Thanks. > > However, I'm afraid I hit a new one. If I have MariaDB 5.1.53 > installed (by the 5.1 installer) on the same machine, the new > installer doesn't allow me to install 5.2.5 saying: > "Previous version of MariaDB was found, that used incompatible > installer. Please remove "MariaDB 5.1.53" before you proceed with this > installation." Ok, valid bug. Thanks for reporting. There will *still* be this message if you're trying to install 5.2.5 msi with existing 5.2.(<5) NSIS . For me personally ,main reason NOT to offer 5.2.4=>5.2.5 autoupgrade is the fact that MariaDBs NSIS was not designed to run in silent mode, its popups "Directory not deleted" during upgrade would only alienate users . > I think this is not an intended behavior, because this kills the big > benefit (of your new installer) of automatically upgrading older > MariaDB instance. The upgrade is not always automatic, sometimes it is semiautomatic (supported by the UI tool launched at the end of installation). Did you look in the WL on how upgrades are handled? Here is link just in case http://askmonty.org/worklog/Server-Sprint/?tid=55 . Feedback on it is welcome. I have fixed 5.2.5 over 5.1 NSIS installation ,the update is on usual place. http://cid-ff0c950417b4f8a4.office.live.com/browse.aspx/share Thanks for your efforts, Vladislav

2 3

Re: [Maria-developers] [Commits] Rev 2892: BUG#711565 Index Condition Pushdown can make a thread hold MyISAM locks as well as be unKILLable for long time in file:///home/psergey/dev2/5.3/
by Michael Widenius 03 Feb '11

03 Feb '11

Hi! >>>>> "Sergey" == Sergey Petrunya <psergey(a)askmonty.org> writes: Sergey> At file:///home/psergey/dev2/5.3/ Sergey> ------------------------------------------------------------ Sergey> revno: 2892 Sergey> revision-id: psergey(a)askmonty.org-20110201233314-6j8yc7jrx9425f3b Sergey> parent: sergii(a)pisem.net-20110114205155-i8poq0jti8o8mob1 Sergey> committer: Sergey Petrunya <psergey(a)askmonty.org> Sergey> branch nick: 5.3 Sergey> timestamp: Wed 2011-02-02 02:33:14 +0300 Sergey> message: Sergey> BUG#711565 Index Condition Pushdown can make a thread hold MyISAM locks as well as be unKILLable for long time <cut> Sergey> === modified file 'mysql-test/t/myisam_icp.test' Sergey> --- a/mysql-test/t/myisam_icp.test 2011-01-14 13:38:41 +0000 Sergey> +++ b/mysql-test/t/myisam_icp.test 2011-02-01 23:33:14 +0000 Sergey> @@ -4,3 +4,183 @@ Sergey> --source include/icp_tests.inc <cut> Instead of using # for each row, please use --disable_parsing --enable_parsing This makes it easier to enable the test temporarly when testing. <cut> Sergey> === modified file 'sql/multi_range_read.cc' Sergey> --- a/sql/multi_range_read.cc 2010-12-21 23:26:35 +0000 Sergey> +++ b/sql/multi_range_read.cc 2011-02-01 23:33:14 +0000 Sergey> @@ -902,7 +902,7 @@ Sergey> close_second_handler(); Sergey> /* Safety, not really needed but: */ Sergey> strategy= NULL; Sergey> - DBUG_RETURN(1); Sergey> + DBUG_RETURN(res); Sergey> use_default_impl: Sergey> DBUG_ASSERT(primary_file->inited == handler::INDEX); Sergey> === modified file 'sql/sql_base.cc' Sergey> --- a/sql/sql_base.cc 2010-12-06 08:25:44 +0000 Sergey> +++ b/sql/sql_base.cc 2011-02-01 23:33:14 +0000 Sergey> @@ -8283,11 +8283,9 @@ Sergey> List_iterator_fast<Item> f(fields),v(values); Sergey> Item *value, *fld; Sergey> Item_field *field; Sergey> - TABLE *table= 0; Sergey> - List<TABLE> tbl_list; Sergey> + TABLE *table= 0, *vcol_table= 0; Sergey> bool abort_on_warning_saved= thd->abort_on_warning; Sergey> DBUG_ENTER("fill_record"); Sergey> - tbl_list.empty(); Sergey> /* Sergey> Reset the table->auto_increment_field_not_null as it is valid for Sergey> @@ -8310,7 +8308,7 @@ Sergey> f.rewind(); Sergey> } Sergey> else if (thd->lex->unit.insert_table_with_stored_vcol) Sergey> - tbl_list.push_back(thd->lex->unit.insert_table_with_stored_vcol); Sergey> + vcol_table= thd->lex->unit.insert_table_with_stored_vcol; Sergey> while ((fld= f++)) Sergey> { Sergey> if (!(field= fld->filed_for_view_update())) Sergey> @@ -8340,31 +8338,17 @@ Sergey> my_message(ER_UNKNOWN_ERROR, ER(ER_UNKNOWN_ERROR), MYF(0)); Sergey> goto err; Sergey> } Sergey> - tbl_list.push_back(table); Sergey> + DBUG_ASSERT(vcol_table == 0 || vcol_table == table); Sergey> + vcol_table= table; Sergey> } Sergey> /* Update virtual fields*/ Sergey> thd->abort_on_warning= FALSE; Sergey> - if (tbl_list.head()) Sergey> + if (vcol_table) Sergey> { Sergey> - List_iterator_fast<TABLE> it(tbl_list); Sergey> - TABLE *prev_table= 0; Sergey> - while ((table= it++)) Sergey> + if (vcol_table->vfield) Sergey> { Sergey> - /* Sergey> - Do simple optimization to prevent unnecessary re-generating Sergey> - values for virtual fields Sergey> - */ Sergey> - if (table != prev_table) Sergey> - { Sergey> - prev_table= table; Sergey> - if (table->vfield) Sergey> - { Sergey> - if (update_virtual_fields(thd, table, TRUE)) Sergey> - { Sergey> - goto err; Sergey> - } Sergey> - } Sergey> - } Sergey> + if (update_virtual_fields(thd, vcol_table, TRUE)) Sergey> + goto err; Sergey> } Sergey> } Sergey> thd->abort_on_warning= abort_on_warning_saved; What has the above to do with the IPC patch? In any case, how will the above work when there is more than one table involved in fill_tables ? (By the way, agree that the old code as stupid; There is no reason to add all involved tables to tbl_list (trivial to avoid). Sergey> === modified file 'sql/sql_select.cc' Sergey> --- a/sql/sql_select.cc 2011-01-14 11:03:41 +0000 Sergey> +++ b/sql/sql_select.cc 2011-02-01 23:33:14 +0000 Sergey> @@ -13618,13 +13618,17 @@ Sergey> " cond: %p error: %d", join, join_tab, select_cond, error)); Sergey> if (error > 0 || (join->thd->is_error())) // Fatal error Sergey> DBUG_RETURN(NESTED_LOOP_ERROR); Sergey> + /* Sergey> + Check if we're killed before checking for EOF. This is important because Sergey> + if we're killed while reading a record, the handler will return EOF. Sergey> + */ Please update the comment; We should not send EOF from the handler if the query is killed; There is too many things that can go wrong if we send EOF on killed from the handler. Sergey> + if (join->thd->killed) // Aborted by user Sergey> + { Sergey> + join->thd->send_kill_message(); Sergey> + DBUG_RETURN(NESTED_LOOP_KILLED); /* purecov: inspected */ Sergey> + } <cut> Sergey> === modified file 'storage/maria/ma_rkey.c' Sergey> --- a/storage/maria/ma_rkey.c 2011-01-14 11:05:46 +0000 Sergey> +++ b/storage/maria/ma_rkey.c 2011-02-01 23:33:14 +0000 Sergey> @@ -34,7 +34,7 @@ Sergey> HA_KEYSEG *last_used_keyseg; Sergey> uint32 nextflag; Sergey> MARIA_KEY key; Sergey> - int icp_res= 1; Sergey> + ICP_RESULT icp_res= ICP_MATCH; Sergey> DBUG_ENTER("maria_rkey"); Sergey> DBUG_PRINT("enter", ("base: 0x%lx buf: 0x%lx inx: %d search_flag: %d", Sergey> (long) info, (long) buf, inx, search_flag)); Sergey> @@ -115,7 +115,7 @@ Sergey> not satisfied with an out-of-range condition. Sergey> */ Sergey> if ((*share->row_is_visible)(info) && Sergey> - ((icp_res= ma_check_index_cond(info, inx, buf)) != 0)) Sergey> + ((icp_res= ma_check_index_cond(info, inx, buf)) != ICP_NO_MATCH)) Sergey> break; Sergey> /* The key references a concurrently inserted record. */ Sergey> @@ -145,6 +145,19 @@ Sergey> if (_ma_search_next(info, &lastkey, maria_readnext_vec[search_flag], Sergey> info->s->state.key_root[inx])) Sergey> break; /* purecov: inspected */ Sergey> + Sergey> + /* Sergey> + If we are at the last key on the key page, allow writers to Sergey> + access the index. Sergey> + */ Sergey> + if (info->int_keypos >= info->int_maxpos && Sergey> + ma_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + buf= 0; /* Fast abort */ Sergey> + icp_res= ICP_ERROR; We should not return an IPC_ERROR here as this test has nothing to do with IPC. It can also happens when we are scanning for a visible row. Better to just threat this as a normal handler error. See later comments... Sergey> + break; Sergey> + } Sergey> + Sergey> /* Sergey> Check that the found key does still match the search. Sergey> _ma_search_next() delivers the next key regardless of its Sergey> @@ -168,9 +181,9 @@ Sergey> if (share->lock_key_trees) Sergey> rw_unlock(&keyinfo->root_lock); Sergey> - if (info->cur_row.lastpos == HA_OFFSET_ERROR || (icp_res != 1)) Sergey> + if (info->cur_row.lastpos == HA_OFFSET_ERROR || (icp_res != ICP_MATCH)) Sergey> { Sergey> - if (icp_res == 2) Sergey> + if (icp_res == ICP_OUT_OF_RANGE || icp_res == ICP_ERROR) Sergey> { Sergey> info->cur_row.lastpos= HA_OFFSET_ERROR; Sergey> my_errno= HA_ERR_KEY_NOT_FOUND; We should not return HA_ERR_KEY_NOT_FOUND in the case of a real error. (Too big risk we loose the reason that something went wrong). Sergey> +/* Sergey> + Yield to possible other writers during a index scan. Sergey> + Check also if we got killed by the user and if yes, return Sergey> + HA_ERR_LOCK_WAIT_TIMEOUT Sergey> + Sergey> + return 0 ok Sergey> + return 1 Query has been requested to be killed Sergey> +*/ Sergey> + Sergey> +my_bool ma_yield_and_check_if_killed(MARIA_HA *info, int inx) Sergey> +{ Sergey> + MARIA_SHARE *share; Sergey> + if (ma_killed(info)) Sergey> + { Sergey> + //info->lastpos= HA_OFFSET_ERROR; Sergey> + my_errno= HA_ERR_KEY_NOT_FOUND; Sergey> + return 1; The above is not safe. Please use: info->cur_row.lastpos= HA_OFFSET_ERROR; my_errno= HA_ERR_ABORTED_BY_USER; In this case we can not return HA_ERR_KEY_NOT_FOUND as it's too big risk that the calling code may think that there was really no key and send a result to the client instead of aborting. By sending a fatal error message to the user, we can be sure that the calling code will not ignore it. That is why we probably should introduce HA_ERR_ABORTED_BY_USER. If we do this, then we have to fix that handler::print_error() ignores this error if thd->killed is set. dispatch_command will generate the necessary error message. Sergey> + } Sergey> + Sergey> + if ((share= info->s)->lock_key_trees) Sergey> + { Sergey> + /* Give writers a chance to access index */ Sergey> + rw_unlock(&share->keyinfo[inx].root_lock); Sergey> + rw_rdlock(&share->keyinfo[inx].root_lock); Sergey> + } Sergey> + return 0; Sergey> +} Sergey> + Sergey> === modified file 'storage/maria/ma_rnext.c' Sergey> --- a/storage/maria/ma_rnext.c 2009-12-15 07:16:46 +0000 Sergey> +++ b/storage/maria/ma_rnext.c 2011-02-01 23:33:14 +0000 <cut> Sergey> + if (info->int_keypos >= info->int_maxpos && Sergey> + ma_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + error= 0; Sergey> + icp_res= ICP_ERROR; Sergey> + break; Sergey> + } Same thing as above, don't use IPC_ERROR, treat this as a real error! Sergey> + Sergey> /* Skip rows inserted by other threads since we got a lock */ Sergey> if ((error= _ma_search_next(info, &info->last_key, Sergey> SEARCH_BIGGER, Sergey> @@ -108,10 +120,10 @@ Sergey> info->update&= (HA_STATE_CHANGED | HA_STATE_ROW_CHANGED); Sergey> info->update|= HA_STATE_NEXT_FOUND; Sergey> - if (icp_res == 2) Sergey> + if (icp_res == ICP_OUT_OF_RANGE) Sergey> my_errno=HA_ERR_END_OF_FILE; /* got beyond the end of scanned range */ Sergey> - if (error || icp_res != 1) Sergey> + if (error || icp_res != ICP_MATCH) Sergey> { Sergey> if (my_errno == HA_ERR_KEY_NOT_FOUND) Sergey> my_errno=HA_ERR_END_OF_FILE; Don't return HA_ERR_END_OF_FILE or HA_ERR_KEY_NOT_FOUND for IPC_ERROR Sergey> === modified file 'storage/maria/ma_rnext_same.c' Sergey> --- a/storage/maria/ma_rnext_same.c 2010-10-30 01:59:39 +0000 Sergey> +++ b/storage/maria/ma_rnext_same.c 2011-02-01 23:33:14 +0000 Sergey> @@ -30,7 +30,7 @@ Sergey> int error; Sergey> uint inx,not_used[2]; Sergey> MARIA_KEYDEF *keyinfo; Sergey> - int icp_res= 1; Sergey> + ICP_RESULT icp_res= ICP_MATCH; Sergey> DBUG_ENTER("maria_rnext_same"); Sergey> if ((int) (inx= info->lastinx) < 0 || Sergey> @@ -80,9 +80,20 @@ Sergey> info->cur_row.lastpos= HA_OFFSET_ERROR; Sergey> break; Sergey> } Sergey> + /* Sergey> + If we are at the last key on the key page, allow writers to Sergey> + access the index. Sergey> + */ Sergey> + if (info->int_keypos >= info->int_maxpos && Sergey> + ma_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + error= 0; Sergey> + icp_res= ICP_ERROR; Sergey> + break; Sergey> + } Same thing as above, don't use IPC_ERROR, treat this as a real error! <cut> Sergey> - if (error || icp_res != 1) Sergey> + if (error || icp_res != ICP_MATCH) Sergey> { Sergey> if (my_errno == HA_ERR_KEY_NOT_FOUND) Sergey> my_errno=HA_ERR_END_OF_FILE; Don't return HA_ERR_END_OF_FILE or HA_ERR_KEY_NOT_FOUND for IPC_ERROR Sergey> === modified file 'storage/maria/ma_rprev.c' <cut> Sergey> + if (ma_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + error= 0; Sergey> + icp_res= ICP_ERROR; Sergey> + break; Sergey> + } Return a real error! Sergey> + } Sergey> + Sergey> /* Skip rows that are inserted by other threads since we got a lock */ Sergey> if ((error= _ma_search_next(info, &info->last_key, Sergey> SEARCH_SMALLER, Sergey> @@ -66,6 +84,10 @@ Sergey> } Sergey> if (share->lock_key_trees) Sergey> rw_unlock(&keyinfo->root_lock); Sergey> + if (icp_res == ICP_ERROR || icp_res == ICP_OUT_OF_RANGE) Sergey> + { Sergey> + DBUG_RETURN(my_errno= HA_ERR_END_OF_FILE); Don't return HA_ERR_END_OF_FILE in case of IPC_ERROR Sergey> === modified file 'storage/maria/maria_def.h' Sergey> --- a/storage/maria/maria_def.h 2011-01-14 11:03:41 +0000 Sergey> +++ b/storage/maria/maria_def.h 2011-02-01 23:33:14 +0000 Sergey> @@ -505,6 +505,7 @@ Sergey> DYNAMIC_ARRAY pinned_pages; Sergey> /* accumulate indexfile changes between write's */ Sergey> TREE *bulk_insert; Sergey> + void *__external_ref; /* For MariaDB TABLE */ /* psergey-todo: need this?*/ No, we don't need this as we have external_ptr already in Aria that we can use. Sergey> --- a/storage/myisam/mi_rkey.c 2009-12-22 12:33:21 +0000 Sergey> @@ -168,26 +176,37 @@ Sergey> if (share->concurrent_insert) Sergey> rw_unlock(&share->key_root_lock[inx]); Sergey> - /* Calculate length of the found key; Used by mi_rnext_same */ Sergey> - if ((keyinfo->flag & HA_VAR_LENGTH_KEY) && last_used_keyseg && Sergey> - info->lastpos != HA_OFFSET_ERROR) Sergey> - info->last_rkey_length= _mi_keylength_part(keyinfo, info->lastkey, Sergey> - last_used_keyseg); Sergey> + info->last_rkey_length= pack_key_length; Sergey> + Sergey> + if (info->lastpos == HA_OFFSET_ERROR) /* No such record */ Sergey> + { Sergey> + /* Sergey> + psergey: what doe the next line do:vv initially there were no writeinfo Sergey> + calls! Sergey> + */ Sergey> + fast_mi_writeinfo(info); The original code had a bug. At start of mi_rkey.c we do fast_mi_readinfo(), which must be followed by a fast_mi_writeinfo() before return, which didn't happen if buf was 0. In the normal case 'read_record()' will do the fast_mi_writeinfo() call for the other cases. Sergey> + if (!buf) Sergey> + DBUG_RETURN(my_errno); Sergey> + } Sergey> else Sergey> + /* Calculate length of the found key; Used by mi_rnext_same */ Sergey> + if ((keyinfo->flag & HA_VAR_LENGTH_KEY) && last_used_keyseg) Sergey> + info->last_rkey_length= _mi_keylength_part(keyinfo, info->lastkey, Sergey> + last_used_keyseg); Sergey> + Sergey> + /* Check if we don't want to have record back, only error message */ Sergey> + if (!buf) Sergey> + DBUG_RETURN(0); We have to add fast_mi_writeinfo(info) before the DBUG_RETURN above. Sergey> + if (!(*info->read_record)(info,info->lastpos,buf)) Sergey> + { Sergey> + info->update|= HA_STATE_AKTIV; /* Record is read */ Sergey> + DBUG_RETURN(0); Sergey> + } <cut> Sergey> +my_bool mi_yield_and_check_if_killed(MI_INFO *info, int inx) Sergey> +{ Sergey> + MYISAM_SHARE *share; Sergey> + if (mi_killed(info)) Sergey> + { Sergey> + info->lastpos= HA_OFFSET_ERROR; Sergey> + my_errno= HA_ERR_KEY_NOT_FOUND; HA_ERR_KEY_NOT_FOUND -> HA_ERR_ABORTED_BY_USER Sergey> + return 1; Sergey> + } Sergey> + Sergey> + if ((share= info->s)->concurrent_insert) Sergey> + { Sergey> + /* Give writers a chance to access index */ Sergey> + rw_unlock(&share->key_root_lock[inx]); Sergey> + rw_rdlock(&share->key_root_lock[inx]); Sergey> + } Sergey> + return 0; Sergey> +} Sergey> === modified file 'storage/myisam/mi_rnext.c' Sergey> --- a/storage/myisam/mi_rnext.c 2010-06-26 10:05:41 +0000 Sergey> +++ b/storage/myisam/mi_rnext.c 2011-02-01 23:33:14 +0000 Sergey> @@ -104,6 +104,18 @@ Sergey> (info->index_cond_func && Sergey> (res= mi_check_index_cond(info, inx, buf)) == ICP_NO_MATCH)) Sergey> { Sergey> + /* Sergey> + If we are at the last key on the key page, allow writers to Sergey> + access the index. Sergey> + */ Sergey> + if (info->int_keypos >= info->int_maxpos && Sergey> + mi_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + error= 0; Sergey> + res= ICP_ERROR; Return a real error! Sergey> + break; Sergey> + } Sergey> + Sergey> /* Sergey> Skip rows that are either inserted by other threads since Sergey> we got a lock or do not match pushed index conditions Sergey> @@ -115,7 +127,8 @@ Sergey> info->s->state.key_root[inx]))) Sergey> break; Sergey> } Sergey> - if (!error && res == ICP_OUT_OF_RANGE) Sergey> + Sergey> + if (!error && (res == ICP_OUT_OF_RANGE || res == ICP_ERROR)) Sergey> { Sergey> if (info->s->concurrent_insert) Sergey> rw_unlock(&info->s->key_root_lock[inx]); Return error code in case of ICP_ERROR Sergey> === modified file 'storage/myisam/mi_rnext_same.c' Sergey> --- a/storage/myisam/mi_rnext_same.c 2009-12-22 12:33:21 +0000 Sergey> +++ b/storage/myisam/mi_rnext_same.c 2011-02-01 23:33:14 +0000 Sergey> @@ -63,6 +63,18 @@ Sergey> } Sergey> for (;;) Sergey> { Sergey> + /* Sergey> + If we are at the last key on the key page, allow writers to Sergey> + access the index. Sergey> + */ Sergey> + if (info->int_keypos >= info->int_maxpos && Sergey> + mi_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + error=1; Sergey> + buf= 0; /* Fast abort */ Sergey> + break; Return real error Sergey> + } Sergey> + Sergey> if ((error=_mi_search_next(info,keyinfo,info->lastkey, Sergey> info->lastkey_length,SEARCH_BIGGER, Sergey> info->s->state.key_root[inx]))) Sergey> @@ -87,6 +99,10 @@ Sergey> } Sergey> if (info->s->concurrent_insert) Sergey> rw_unlock(&info->s->key_root_lock[inx]); Sergey> + Sergey> + if (!buf) Sergey> + DBUG_RETURN(my_errno); Sergey> + Remove above; We have to update info->update even if buf is 0 Sergey> /* Don't clear if database-changed */ Sergey> info->update&= (HA_STATE_CHANGED | HA_STATE_ROW_CHANGED); Sergey> info->update|= HA_STATE_NEXT_FOUND | HA_STATE_RNEXT_SAME; Sergey> === modified file 'storage/myisam/mi_rprev.c' Sergey> --- a/storage/myisam/mi_rprev.c 2009-12-15 07:16:46 +0000 Sergey> +++ b/storage/myisam/mi_rprev.c 2011-02-01 23:33:14 +0000 Sergey> @@ -53,12 +53,28 @@ Sergey> if (!error) Sergey> { Sergey> - int res= 0; Sergey> + ICP_RESULT res= 0; Sergey> + my_off_t cur_keypage= info->last_keypage; Sergey> while ((share->concurrent_insert && Sergey> info->lastpos >= info->state->data_file_length) || Sergey> (info->index_cond_func && Sergey> !(res= mi_check_index_cond(info, inx, buf)))) Sergey> { Sergey> + /* Sergey> + If we are at the last (i.e. first?) key on the key page, Sergey> + allow writers to access the index. Sergey> + */ Sergey> + if (info->last_keypage != cur_keypage) Sergey> + { Sergey> + cur_keypage= info->last_keypage; Sergey> + if (mi_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + error= 0; Sergey> + res= ICP_ERROR; Sergey> + break; Return error Sergey> + } Sergey> + } Sergey> + Sergey> /* Sergey> Skip rows that are either inserted by other threads since Sergey> we got a lock or do not match pushed index conditions Sergey> @@ -69,7 +85,7 @@ Sergey> share->state.key_root[inx]))) Sergey> break; Sergey> } Sergey> - if (!error && res == 2) Sergey> + if (!error && (res == ICP_OUT_OF_RANGE || res == ICP_ERROR)) Sergey> { Sergey> if (share->concurrent_insert) Sergey> rw_unlock(&share->key_root_lock[inx]); Return error code in case of ICP_ERROR <cut> Sergey> --- a/storage/xtradb/handler/ha_innodb.cc 2011-01-12 12:00:10 +0000 Sergey> +++ b/storage/xtradb/handler/ha_innodb.cc 2011-02-01 23:33:14 +0000 <cut> Sergey> @@ -12094,15 +12102,18 @@ Sergey> See note on ICP_RESULT for return values description. Sergey> */ Sergey> -static int index_cond_func_innodb(void *arg) Sergey> +static xtradb_icp_result_t index_cond_func_innodb(void *arg) Sergey> { Sergey> ha_innobase *h= (ha_innobase*)arg; Sergey> + if (h->is_thd_killed()) Sergey> + return XTRADB_ICP_ERROR; Sergey> + rename to XTRADB_ABORTED_BY_USER; Sergey> if (h->end_range) Sergey> { Sergey> if (h->compare_key2(h->end_range) > 0) Sergey> - return ICP_OUT_OF_RANGE; /* caller should return HA_ERR_END_OF_FILE already */ Sergey> + return XTRADB_ICP_OUT_OF_RANGE; /* caller should return HA_ERR_END_OF_FILE already */ Sergey> } Sergey> - return h->pushed_idx_cond->val_int()? ICP_MATCH : ICP_NO_MATCH; Sergey> + return h->pushed_idx_cond->val_int()? XTRADB_ICP_MATCH : XTRADB_ICP_NO_MATCH; Sergey> } Sergey> C_MODE_END <cut> Sergey> --- a/storage/xtradb/include/row0mysql.h 2010-12-06 08:25:44 +0000 Sergey> +++ b/storage/xtradb/include/row0mysql.h 2011-02-01 23:33:14 +0000 Sergey> @@ -577,7 +577,15 @@ Sergey> #define ROW_PREBUILT_ALLOCATED 78540783 Sergey> #define ROW_PREBUILT_FREED 26423527 Sergey> -typedef int (*index_cond_func_t)(void *param); Sergey> + Sergey> +typedef enum xtradb_icp_result { Sergey> + XTRADB_ICP_ERROR=-1, rename to XTRADB_ABORTED_BY_USER Sergey> + XTRADB_ICP_NO_MATCH=0, Sergey> + XTRADB_ICP_MATCH=1, Sergey> + XTRADB_ICP_OUT_OF_RANGE=2 Sergey> +} xtradb_icp_result_t; Sergey> + Sergey> +typedef xtradb_icp_result_t (*index_cond_func_t)(void *param); Sergey> /** A struct for (sometimes lazily) prebuilt structures in an Innobase table Sergey> handle used within MySQL; these are used to save CPU time. */ Sergey> === modified file 'storage/xtradb/row/row0sel.c' Sergey> --- a/storage/xtradb/row/row0sel.c 2010-12-06 17:44:17 +0000 Sergey> +++ b/storage/xtradb/row/row0sel.c 2011-02-01 23:33:14 +0000 Sergey> @@ -4396,12 +4396,13 @@ Sergey> */ Sergey> ut_ad(ib_res); Sergey> res= prebuilt->idx_cond_func(prebuilt->idx_cond_func_arg); Sergey> - if (res == 0) Sergey> + if (res == XTRADB_ICP_NO_MATCH) Sergey> goto next_rec; Sergey> - if (res == 2) { Sergey> + if (res == XTRADB_ICP_OUT_OF_RANGE || res == XTRADB_ICP_ERROR) { Sergey> err = DB_RECORD_NOT_FOUND; Sergey> goto idx_cond_failed; Don't return DB_RECORD_NOT_FOUND in case of XTRADB_ICP_ERROR. Return an actual error like DB_ABORTED_BY_USER which can be translated to HA_ERR_ABORTED_BY_USER. Regards, Monty

1 0

Re: [Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 03 Feb '11

03 Feb '11

> -----Original Message----- > From: kahzoo(a)gmail.com [mailto:kahzoo@gmail.com] On Behalf Of Kazuhisa Ichikawa > Sent: Donnerstag, 3. Februar 2011 02:00 > To: Vladislav Vaintroub > Cc: maria-developers(a)lists.launchpad.net > Subject: Re: [Maria-developers] Windows installer MWL#55 finished. > Hello Kazuhisa, > > But to reiterate, this package is not really thought to be run by non-admin users. It installs into program files, and it writes to > > the HKLM registry hive, and it creates a service. All this stuff is not possible without being an administrator. I'm quite surprised > > Well, if that is how it is designed, that's fine with me. I just > thought that there might be many users who are new to MariaDB, not > sure about its quality and want to install/run it in a sandbox > environment. If switching to HKCU registry hive and disabling the > option to run it as a service when the installing user is non-admin > can satisfy the needs, implementing them might not be a bad idea. But > again, I do not have a strong opinion on that. (The user can choose to > install from the zipped package as long as zip version is also > distributed). I this the per-user install is a feature in NSIS which you discovered, but which is was not planned from the start on. From the start on, it was planned as an installation where you get a running service at the end, and that was not implemented in NSIS. if it was, it would not work in this sandboxed environment. With MSI I checked, and the rules for per-user install appear to be relatively complicated. Officially dual package authoring (single package that can be installed either per-user or per-machine) is supported with MSI 5.0 and Win7, so there is no hope to get that working in the same for unprivileged XP user. I might do some experimentation with it on Win7, however the benefit it provide compared to ZIP is low: is a menu shortcut with PATH including bin directory of the installation and Add/Remove Program entry. On a bright side, this is a valid bug that was simple to fix forcing per-machine install, so installer does not need to guess. I did that and refreshed the MSI on the download page. > One more thing that I noticed is, when installing 5.1 all the messages > in the dialogs are in Japanese (with the exception of GPL license > statement), while they are all in English for 5.2. Maybe this is a > bonus that came from NSIS. Maybe. NSIS guys deserve a compliment for that effort. With MSI dealing with multiple languages inside single MSI is not that simple. But, localization is definitely possible in Wix or generally MSI, I so far ignored that topic altogether;) But back to NSIS package .I believe, just like with per-user installation, you discovered hidden and untested feature, an Easter egg. It has more localization bugs that just GPL text. The feature tree is not localized, neither uninstall popup box. If that NSIS package had more functionality than plain unpack, and had own dialogs, there would be more localization bugs.

1 0

Re: [Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 02 Feb '11

02 Feb '11

> -----Original Message----- > From: kahzoo(a)gmail.com [mailto:kahzoo@gmail.com] On Behalf Of Kazuhisa Ichikawa > Sent: Mittwoch, 2. Februar 2011 23:01 > To: Vladislav Vaintroub > Cc: maria-developers(a)lists.launchpad.net > Subject: Re: [Maria-developers] Windows installer MWL#55 finished. > > Vladislav, > > I tried the new 5.2.5 installer in Japanese Windows XP (32bit) > environment using a user account which does NOT have administrator > privilege. I wonder why it has started at all. It definitely needs administrator and installer on later Windows brings all the necessary UAC popups. > During the installation I encountered a lot of warning > dialogs (see below for an example) about not being able to write into > the registry, although I did not have any noticeable problem startnig > mysqld.exe after the installation is finished (No, the installation > was not aborted and it said "Completed the MariaDB 5.2 Setup WIzard.") > > (Example warning message) > "Could not write value InstallLocation to key > \Software\Microsoft\Windows\CurrenVersion\Uninstall\MariaDB 5.2. > Verify that you have sufficient access to that key, or contact your > support personnel." If you do not have that key, you do not have an entry in "Add/remove programs" > I think what this error message means is that the user account used > for the installation needs to have an administrator privilege, but the > current (older) installer for MariaDB 5.1.53 did not have this > problem. This installer should be run as administrator, there is no per-user option. > The other (related) issue is that I cannot uninstall mariadb 5.2 now > because it is not listed in the installed program list in the control > panel (most likely because the necessary information is not written > into the registry). You will need to run msiexec /x mariadb-5.2.5-win32.msi from the command line. This uninstalls installed package (I'm not really sure it was installed, but it does not hurt to try, it nothing is installed it will tell) > Again, the installer for 5.1 did not have this problem. Well, it is not installer for 5.1, the old one had more or less the same functionality as downloading and unpacking a zip file. It might even be that NSIS guys automatically switched to HKCU registry keys or so for a weak user. > Also, unlike the current installer for 5.1, the new installer > does not seem to install a standalone uninstaller (Uninstall.exe) on > the disk. So, I'm stuck and wondering how to uninstall 5.2 (Well, not > really, because if nothing is written into the registry, deleting all > the files and directories should 'cleanly' uninstall everything, > right?.). I would stick to documented stuff. msiexec /x mariadb-5.2.5-win32.msi But to reiterate, this package is not really thought to be run by non-admin users. It installs into program files, and it writes to the HKLM registry hive, and it creates a service. All this stuff is not possible without being an administrator. I'm quite surprised that MSI let you go that far actually, and that it would not rollback and cleanup after the error. On the other hand, thanks for an interesting error report, makes sense for me to have a test for anomalies like that. Can you tell service pack of your XP and MSI version (launch winver and msiexec /?) Thanks, Vladislav > - Kazuhisa >

1 0

[Maria-developers] Benchmarking mrr_sort_keys=on|off.
by Sergey Petrunya 02 Feb '11

02 Feb '11

Hi! Following the discussion of benchmarks of mrr_sort_keys=on|off functionality, I've done a re-run. Please find the results below. ## Dataset: DBT-3, scale=10, xtradb, ## innodb_file_per_table=1 ## innodb_file_format=barracuda ## innodb_log_file_size=256M ## innodb_buffer_pool_size=4000M ## ## Platform: work.askmonty.org host ## Cold-cache is achieved by: ## - stopping the server ## - echo 1 > /proc/sys/vm/drop_caches ## - starting the server ## - running EXPLAIN {query} ## ## MariaDB: 5.3-current, release build (compile-pentium-max) ## Result summary: * mrr_sort_keys=on provides speedup (2.25 min vs 13.5 min) on cold cache. * on hot cache, there is no big difference (~1.3 sec in both cases). * I was able to repeat Igor's August results with the current 5.3. It is not clear why Igor can't repeat them anymore. (An evasive bug? or just mistake in experiments?) set join_buffer_size=1024*1024*32; set join_buffer_space_limit=1024*1024*32; set join_cache_level=6; ## ## Cold cache, mrr_sort_keys=on: ## [22:12] root@dbt3sf10>explain select avg(l_extendedprice) from orders, lineitem where l_orderkey=o_orderkey and o_orderdate between date '1992-07-01' and date '1992-07-31'; +----+-------------+----------+-------+--------------------------------------------+---------------+---------+----------------------------+--------+------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+-------+--------------------------------------------+---------------+---------+----------------------------+--------+------------------------------------+ | 1 | SIMPLE | orders | range | PRIMARY,i_o_orderdate | i_o_orderdate | 4 | NULL | 142680 | Using where; Using index | | 1 | SIMPLE | lineitem | ref | PRIMARY,i_l_orderkey,i_l_orderkey_quantity | PRIMARY | 4 | dbt3sf10.orders.o_orderkey | 1 | Using join buffer (flat, BKA join) | +----+-------------+----------+-------+--------------------------------------------+---------------+---------+----------------------------+--------+------------------------------------+ 2 rows in set (3.04 sec) [22:06] root@dbt3sf10>select avg(l_extendedprice) from orders, lineitem where l_orderkey=o_orderkey and o_orderdate between date '1992-07-01' and date '1992-07-31'; +----------------------+ | avg(l_extendedprice) | +----------------------+ | 38229.0683265894 | +----------------------+ 1 row in set (2 min 13.40 sec) [22:09] root@dbt3sf10> show status like "%handler%"; +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_commit | 2 | | Handler_read_key | 193396 | | Handler_read_next | 966535 | +----------------------------+--------+ 15 rows in set (0.29 sec) # # mrr_sort_keys=off, cold cache: # [22:20] root@dbt3sf10>explain select avg(l_extendedprice) from orders, lineitem where l_orderkey=o_orderkey and o_orderdate between date '1992-07-01' and date '1992-07-31'; +----+-------------+----------+-------+--------------------------------------------+---------------+---------+----------------------------+--------+--------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+-------+--------------------------------------------+---------------+---------+----------------------------+--------+--------------------------+ | 1 | SIMPLE | orders | range | PRIMARY,i_o_orderdate | i_o_orderdate | 4 | NULL | 142680 | Using where; Using index | | 1 | SIMPLE | lineitem | ref | PRIMARY,i_l_orderkey,i_l_orderkey_quantity | PRIMARY | 4 | dbt3sf10.orders.o_orderkey | 2 | | +----+-------------+----------+-------+--------------------------------------------+---------------+---------+----------------------------+--------+--------------------------+ [22:20] root@dbt3sf10>select avg(l_extendedprice) from orders, lineitem where l_orderkey=o_orderkey and o_orderdate between date '1992-07-01' and date '1992-07-31'; +----------------------+ | avg(l_extendedprice) | +----------------------+ | 38229.0683265898 | +----------------------+ 1 row in set (13 min 23.68 sec) [22:09] root@dbt3sf10> show status like "%handler%"; +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_commit | 2 | | Handler_read_key | 193396 | | Handler_read_next | 966535 | +----------------------------+--------+ ## ## Hot caches (re-run the query until time stabilizes): ## mrr_sort_keys=off: 1.34 sec mrr_sort_keys=on: 1.38 sec BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 02 Feb '11

02 Feb '11

From: Peter Laursen [mailto:peter_laursen@webyog.com] Sent: Mittwoch, 2. Februar 2011 17:14 To: Vladislav Vaintroub Cc: maria-developers(a)lists.launchpad.net Subject: Re: [Maria-developers] Windows installer MWL#55 finished. Hi Peter, >Your default install path is \program files [(x86)]. You copy the /datadir as a subfolder of /basedirr. >This will cause problems on Vista+ with UAC enabled if the server is ever started from commandline as a non-service (also if >the user doing so is an administrator user). For the user who installed MySQL, it won't cause problems. After you stop the service you can start mysqld --defaults-file=path\to\datadir\my.ini --console from the non-elevated command line, with/without UAC. No file system virtualization will happen. That bug http://bugs.mysql.com/bug.php?id=38373 (user can not from mysqld from command line after installation) was filed by myself 2.5 years ago , I could not leave that unfixed now. I know, writing to ProgramFiles might not be kosher, but it works if ACLs on our tree are set. (And Microsoft should not tell people that writing to program files is bad, they are doing it with SQLServer, and also document it http://msdn.microsoft.com/en-us/library/ms143547.aspx , so we are again as good/bad as SQLServer here) >2) There is no my.ini generated with a fresh install as far as I can see - so the server probably runs (it does!) with very low default buffer settings etc. Also latin1 charset and MyISAM will be default. I think a my.ini with settings fit for a 1GB-2GB RAM machine should be generated - and with a few configuration options. And a my.ini will be required to solve 1) anyway. There is a my.ini, it is in database directory. It is spartanic, it has port and datadir, nothing else. You can edit it using non-elevated notepad (with or without UAC) and there is a shortcut for it created in start menu, MariaDB 5.2 folder (rsp MariaDB 5.2 (x64) if for 64 bit package). There are zero performance related options, which means in fact it won't run on full speed. I have no strong opinion on it, nor I feel qualified for configuring "optimal performance out of the box" . If there is a strong demand on specific parameters, they should probably go there. I would like to hear more opinions on this, one thing that will definitely not be in the first incarnation of this installer is fully-featured ConfigWizard like stuff. Where I see differences between this my.ini and MySQL installer's default is * sql_mode (MySQL's default is "strict transactional tables") * default storage engine (MySQL default is innodb) * innodb buffer pool size (IIRC, MySQL's default is something like 12% of RAM) Those are trivial to change modifying my.ini Stuff like innodb log file size (calculated from innodb buffer pool size) not trivial to change with simple my.ini modification, there must be a clean shutdown and exiting files need to be removed. > But still it is a good installer (and there is a 64 bit one as well), so this is an important step in the right direction. So thanks for your effort! Thanks! -- Peter On Wed, Feb 2, 2011 at 14:37, Peter Laursen <peter_laursen(a)webyog.com> wrote: I am a little bit busy now, but will confirm tonight. -- Peter On Wed, Feb 2, 2011 at 14:12, Vladislav Vaintroub <wlad(a)montyprogram.com> wrote: From: Peter Laursen [mailto:peter_laursen@webyog.com] Sent: Dienstag, 1. Februar 2011 20:31 To: Vladislav Vaintroub Cc: maria-developers(a)lists.launchpad.net Subject: Re: [Maria-developers] Windows instller MWL#55 finished. >It is a Danish Windows (Home Premium 64 bit). Hi Peter, As it turns out, installer had the same "localization" problem registering service on German Windows. Fixed now, I uploaded the msis to the same place http://cid-ff0c950417b4f8a4.office.live.com/browse.aspx/share Thanks, Wlad

2 1

[Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 02 Feb '11

02 Feb '11

Hi all, I updated the MWL#55 (Windows installer) with the upgrade information. Now, people will be able to upgrade between prior MariaDB (or prior MySQL) versions using upgrade wizard, which is by default launched at the end of installation, if upgradable instances are found . Users will also be able to use Upgrade Wizard after the installation, at any the time convenient for them. Upgrade wizard will also be included into ZIP distribution. Besides GUI Wizard, there is a command line utility mysql_upgrade_service.exe to perform all operations necessary to upgrade an existing database instance registered as service (in fact GUI wizard uses this mysql_upgrade_service.exe as "backend"). Wizard is an MFC application. I'm sorry for that, but my Windows C/C++ GUI programming experience is not only rusty (prior to this WL last time I used it was more than a decade ago), it is also restricted to MFC only. Being MFC application, it will require at least VS Professional to build. There is no MFC in free Visual Studio Express edition, nor in free Windows SDK. Build and packaging process will handle missing MFC gracefully (build won't try to compile upgrade wizard, package won't include it , and installer won't start it at the end of installation). The repository that includes the final windows installer is lp:~maria-captains/maria/5.2-windows-packaging-upgrade Building installer is described in http://askmonty.org/worklog/Server-Sprint/?tid=55 (Section "Building installer", at the very start of the WL). Monty or Sergei (or volunteers if any): Can you please have a look the above repo and do a review . I tried to split the big patch into parts, but I did not find the result to be too satisfactory Philip: would you be able to test the upgrade wizard? You need a service running under some older mysqld.exe binary (e.g. install MySQL 5.1), you'll should get upgrade wizard launched at the end of install. You should also get Wizard at the end of installation any time you have an older version of Maria running as service. Thanks and best regards, Wlad

6 27

[Maria-developers] IMPORTANT: LP bug #611648 does not allow running benchmarks for DS-MRR normally
by Igor Babaev 02 Feb '11

02 Feb '11

Hi, I've just reported bug #611648 that probably explains the strange results I had when trying to run my benchmark queries for DS-MRR about which I talked at today's optimizer meeting. Philip, please assign this bug to SergeyP. Regards, Igor.

2 1

[Maria-developers] Buildbot Outage
by Adam M. Dutko 31 Jan '11

31 Jan '11

Sorry about adutko-centos5-amd64 being down. I've not finished changes to my monitoring script to tell me when a builder is offline. I'll try to get that done soon and send it to the list when I've updated it to work with the new buildbot status page format.

1 0

Re: [Maria-developers] replication improvements going on here and there
by Kristian Nielsen 28 Jan '11

28 Jan '11

"Rasmus Johansson" <rasmus(a)montyprogram.com> writes: > I would need your help to understand the different replication topics that > are very active now: > > - The replication work that you have been doing As far as implementation goes, it is these four worklogs, which are in code review and available in the feature preview tree lp:~maria-captains/maria/mariadb-5.2-rpl: http://askmonty.org/worklog/Server-Sprint/?tid=116 http://askmonty.org/worklog/Server-RawIdeaBin/?tid=136 http://askmonty.org/worklog/Server-Sprint/?tid=163 http://askmonty.org/worklog/Server-Sprint/?tid=132 This work improves replication in a number of ways: - Implement working group commit for the replication binlog. This greatly improves performance when running with sync_binlog=1, which is necessary to ensure the ability to reliably recover replication state after a master crash. - Ensure same commit order between innodb and binlog, without the need for the expensive prepare_commit_mutex. This is needed by XtraBackup and InnoDB hot backup to do a non-blocking online backup that can be used to provision a slave. The other existing patches I have seen solve this by having the user need to turn on/off the prepare_commit_mutex depending on whether backup is going on or not. - Allow to obtain an InnoDB consistent read snapshot and the corresponding binlog position in a fully non-blocking way. This allows for example to provision a slave from a fully non-blocking `mysqldump --master-data --single-transaction`. Previously, this required FLUSH TABLES WITH READ LOCK, which blocks queries in the server for some time. - Port the Facebook patch to release InnoDB row locks early, already during the prepare phase. This can improve performance in the presense of hot-spot rows in busy OLTP applications. (This feature must be enabled explicitly, it is off by default). - Make the transaction coordinator inside MariaDB plugable, allowing to plug in different implementations. This is mostly preparatory work for pluggable replication, as the replication binlog functions as a transaction coordinator. So with this, a new binlog implementation can be plugged in and do the same transaction coordinator role that the legacy binlog normally handles. In addition to this, there was a long discussion on the MariaDB mailing list about various replication topics, and I wrote up the following specifications (but did not start on the implementation yet): http://askmonty.org/worklog/?tid=107 http://askmonty.org/worklog/?tid=120 http://askmonty.org/worklog/?tid=133 The idea with these is to gradually make the existing legacy MySQL replication and binlog pluggable, opening the way for writing completely new replication implementations. It builds on top of the implementation work above. I also worked on the specifications of the parallel replication worklogs MWL#169 and MWL#170 mentioned below. > - The parallel replication that Oracle is talking about for MySQL 5.6 Do you mean this? http://forge.mysql.com/wiki/ReplicationFeatures/ParallelSlave http://forge.mysql.com/worklog/task.php?id=4648 The idea is to take one transaction at a time, and run the operations inside it in parallel. That is a bit unusual, most other projects work by running multiple transaction in parallel. I think this is motivated by NDB (though it is just me reading between the lines, I do not know for sure). Since in NDB, multiple parallel transactions on the master are replicated as a single, bigger transaction called an epoch. So they need something on the slave that can take apart the epoch again and make it parallel. Frankly, when I first saw this, I dismissed it, thinking it was the wrong approach to the problem (outside of NDB replication at least). I am still quite sceptical. Though it may be possible to use this for normal OLTP workloads (with small transactions) by artificially combining multiple transactions, at the cost of higher delay between master and slave. > - The replication functionality that <customer> was interested in together > with <partner> and us This is a request for implementing another algorithm for parallel replication. I wrote up some designs for this: http://askmonty.org/worklog/Server-RawIdeaBin/?tid=169 http://askmonty.org/worklog/Server-RawIdeaBin/?tid=170 This overlaps with the above-mentioned parallel replication project in Oracle, however the chosen approach/algorithm is quite different between the two. In this work, it would be the responsibility of the application to ensure that parallel transactions modifying tables in two different databases are independent. Then such transactions can be replicated in parallel on the slave. > Could you just give a short description on each of those and then describe > how they connect to each other or are there something that duplicates? Hope this helps, - Kristian.

1 0

Re: [Maria-developers] [Commits] Rev 2896: Fixed LP bug #707827. in file:///home/igor/maria/maria-5.3-bug707827/
by Oleksandr Byelkin 28 Jan '11

28 Jan '11

Hi! On 28.01.2011 07:23, Igor Babaev wrote: > At file:///home/igor/maria/maria-5.3-bug707827/ > > ------------------------------------------------------------ > revno: 2896 > revision-id: igor(a)askmonty.org-20110128052302-09617460yxj1kck7 > parent: igor(a)askmonty.org-20110127044523-9kvx1t3x3166hjyr > committer: Igor Babaev<igor(a)askmonty.org> > branch nick: maria-5.3-bug707827 > timestamp: Thu 2011-01-27 21:23:02 -0800 > message: > Fixed LP bug #707827. > This bug could manifest itself when hash join over a varchar column > with NULL values in some rows was used. It happened because the > function key_buf_cmp erroneously returned FALSE when one of the joined > key fields was null while the second was not. > Also fixed two other bugs in the functions key_hashnr and key_buf_cmp > that could possibly lead to wrong results for some queries that > used hash join over several columns with nulls. > Also reverted the latest addition of the test case for bug #45092. It > had been already backported earlier. It looks good to be pushed. Thank you for fixing it!

1 0

[Maria-developers] Benchmarking index_merge sort_intersect
by Sergey Petrunya 26 Jan '11

26 Jan '11

Hi! Please find below results of my attempts to figure out if/when one should "index_merge_sort_intersection=on" setting. General setup information: * Experiment was done on my laptop * I picked "on time flight statistics" dataset, because - it has lots of columns, so one can't be expected to have all possible composite indexes - it allows for meaningful intersection queries - the data is real and I don't expect it to be extremely correlated (although some aiports/airlines are probably worse than others, so it's not totally uncrellated, either). * Storage engine is XtraDB * I loaded data for Q1 2009, which is 1.5M rows, and ontime.ibd file is 704M: MariaDB [ontime]> select count(*), min(FlightDate), max(FlightDate) from ontime; +----------+-----------------+-----------------+ | count(*) | min(FlightDate) | max(FlightDate) | +----------+-----------------+-----------------+ | 1578171 | 2009-01-01 | 2009-03-31 | +----------+-----------------+-----------------+ 1 row in set (15.92 sec) MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=on'; MariaDB [ontime]> explain select count(*) from ontime where depdelay > 30 and OriginState IN ('WA', 'CA'); +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay,OriginState | 5,3 | NULL | 41832 | Using sort_intersect(DepDelay,OriginState); Using where | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ 1 row in set (0.01 sec) ## (Run query several times until query time stabilizes) MariaDB [ontime]> select count(*) from ontime where depdelay > 30 and OriginState IN ('WA', 'CA'); +----------+ | count(*) | +----------+ | 18824 | +----------+ 1 row in set (2.37 sec) MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=off'; MariaDB [ontime]> explain select count(*) from ontime where depdelay > 30 and OriginState IN ('WA', 'CA'); +----+-------------+--------+-------+----------------------+----------+---------+------+--------+-----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------+---------+------+--------+-----------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay | 5 | NULL | 201894 | Using index condition; Using where; Using MRR | +----+-------------+--------+-------+----------------------+----------+---------+------+--------+-----------------------------------------------+ 1 row in set (0.01 sec) ## (Run query several times until query time stabilizes) MariaDB [ontime]> select count(*) from ontime where depdelay > 30 and OriginState IN ('WA', 'CA'); +----------+ | count(*) | +----------+ | 18824 | +----------+ 1 row in set (18.27 sec) ## For comparison, without indexes whatsoever: MariaDB [ontime]> select count(*) from ontime ignore index (depdelay, originstate) where depdelay > 30 and OriginState IN ('WA', 'CA'); +----------+ | count(*) | +----------+ | 18824 | +----------+ 1 row in set (15.50 sec) "iostat -x" showed disk utilization as 0% during all of the above listed queries, so all accessed data was either in cache or in buffer pool. For an idea about predicate selectivities and correlation: ROWS FRACTION whole table 1578171 100.0% depdelay>30 151806 9.6% OriginState IN ('WA', 'CA') 206576 13.1% both 18824 1.1% Conclusions: - sort-intersect gives about 7x improvement - Igor's statement that sort-intersect can't be useful when the load is not IO-bound is not supported by the experiment. My next step was to try with narrower ranges. I changed the predicates to - OriginState IN ('IL', 'VW') (100K rows instead of 200K we had with WA+CA) - depdelay > 60 (75 K rows instead of 150K we had with depdelay>30) MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=on'; MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState IN ('IL', 'VT'); +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay,OriginState | 5,3 | NULL | 10105 | Using sort_intersect(DepDelay,OriginState); Using where | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ 1 row in set (0.02 sec) # Run a few times to stabilize MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState IN ('IL', 'VT'); +----------+ | count(*) | +----------+ | 6283 | +----------+ 1 row in set (0.99 sec) MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=off'; Query OK, 0 rows affected (0.00 sec) MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState IN ('IL', 'VT'); +----+-------------+--------+-------+----------------------+----------+---------+------+-------+-----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------+---------+------+-------+-----------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay | 5 | NULL | 99636 | Using index condition; Using where; Using MRR | +----+-------------+--------+-------+----------------------+----------+---------+------+-------+-----------------------------------------------+ 1 row in set (0.02 sec) # Run a few times to stabilize MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState IN ('IL', 'VT'); +----------+ | count(*) | +----------+ | 6283 | +----------+ 1 row in set (8.63 sec) MariaDB [ontime]> select count(*) from ontime ignore index (OriginState,DepDelay) where depdelay > 60 and OriginState IN ('IL', 'VT'); +----------+ | count(*) | +----------+ | 6283 | +----------+ 1 row in set (14.00 sec) The picture is similar: sort-intersect is still about 8 times better than single-index range access plan. I'm not sure about the conclusions. I suspect that there is some threshold (#records to be read through the most selective index) after which sort-intersect becomes worse than using one of the indexes. I wasn't able to hit it, though. Then I've tried changing the OriginState predicate to being simple equality, to see if we could compare againist ref access: MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=on'; MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState='IL'; +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay,OriginState | 5,3 | NULL | 10043 | Using sort_intersect(DepDelay,OriginState); Using where | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState='IL'; +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (1.00 sec) MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=off'; MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState='IL'; +----+-------------+--------+------+----------------------+-------------+---------+-------+--------+------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+------+----------------------+-------------+---------+-------+--------+------------------------------------+ | 1 | SIMPLE | ontime | ref | OriginState,DepDelay | OriginState | 3 | const | 201600 | Using index condition; Using where | +----+-------------+--------+------+----------------------+-------------+---------+-------+--------+------------------------------------+ 1 row in set (0.01 sec) MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState='IL'; +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (0.95 sec) And the benefit was gone. sort_intersect happens to be just as fast as ref access. I've changed OriginState='IL' to OriginState IN ('IL','ZZ'), where 'ZZ' is the non existant state code: MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState IN ('IL','ZZ'); +----+-------------+--------+-------+----------------------+----------+---------+------+-------+-----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------+---------+------+-------+-----------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay | 5 | NULL | 99636 | Using index condition; Using where; Using MRR | +----+-------------+--------+-------+----------------------+----------+---------+------+-------+-----------------------------------------------+ The used index has changed, MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState IN ('IL','ZZ'); +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (9.10 sec) And the query time went up greatly. Using FORCE INDEX to use the index that ref access was using improved the situation slightly: MariaDB [ontime]> explain select count(*) from ontime force index(originstate) where depdelay > 60 and OriginState IN ('IL','ZZ'); +----+-------------+--------+-------+---------------+-------------+---------+------+--------+-----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+---------------+-------------+---------+------+--------+-----------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState | OriginState | 3 | NULL | 201601 | Using index condition; Using where; Using MRR | +----+-------------+--------+-------+---------------+-------------+---------+------+--------+-----------------------------------------------+ MariaDB [ontime]> select count(*) from ontime force index(originstate) where depdelay > 60 and OriginState IN ('IL','ZZ'); +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (2.69 sec) Disabling MRR further improved the situation: MariaDB [ontime]> set optimizer_use_mrr='disable'; Query OK, 0 rows affected (0.00 sec) MariaDB [ontime]> explain select count(*) from ontime force index(originstate) where depdelay > 60 and OriginState IN ('IL','ZZ'); +----+-------------+--------+-------+---------------+-------------+---------+------+--------+------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+---------------+-------------+---------+------+--------+------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState | OriginState | 3 | NULL | 201601 | Using index condition; Using where | +----+-------------+--------+-------+---------------+-------------+---------+------+--------+------------------------------------+ 1 row in set (0.01 sec) MariaDB [ontime]> select count(*) from ontime force index(originstate) where depdelay > 60 and OriginState IN ('IL','ZZ'); +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (1.23 sec) But it seems we can't put all the blame on MRR. The supposedly best query plan without MRR is just as bad as it was with MRR: MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState IN ('IL','ZZ'); +----+-------------+--------+-------+----------------------+----------+---------+------+-------+------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------+---------+------+-------+------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay | 5 | NULL | 99636 | Using index condition; Using where | +----+-------------+--------+-------+----------------------+----------+---------+------+-------+------------------------------------+ 1 row in set (0.02 sec) MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState IN ('IL','ZZ'); +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (8.96 sec) In any case, sort_intersect doesn't care about whether one of the predicates is an equality or IN: MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=on'; Query OK, 0 rows affected (0.00 sec) MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState IN ('IL','ZZ'); +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay,OriginState | 5,3 | NULL | 10043 | Using sort_intersect(DepDelay,OriginState); Using where | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ 1 row in set (0.02 sec) MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState IN ('IL','ZZ'); +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (0.97 sec) Conclusions: I'm not sure if the "failure" to demonstrate speedup against ref access is a genuine symptom (I think - not). Perhaps, enabling all of our improvements at once (BKA, DS-MRR, sort-intersect) will produce more coherent results, because all of these strategies use rowid-ordered disk-sweep reads, which allows to make fairer comparison between them. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 2

[Maria-developers] Buildbot will be down start of next week due to upgrade
by Kristian Nielsen 24 Jan '11

24 Jan '11

Next week, on Monday, I plan to take down Buildbot and start the upgrade process to version 0.8.x. This is a somewhat major upgrade, as Buildbot 0.8 switches to storing all its state and history in a MySQL (well MariaDB probably ;-) database rather than gazilions of Python-encoded files. Hopefully this will speed up our Buildbot, which has frankly gotten painfully slow. I plan to stop Buildbot, take a full backup, then attempt the upgrade and data conversion. Buildbot will be down this time, which I estimate could be 1-2 days, but it is hard to tell for sure without knowing what issues could arise. If the upgrade fails in some way, I will revert from the backup (but I hope of course to get it running on the new version). >From reading the docs, it looks like we will loose the history of past builds. I will see if there is some way to avoid that, but if not I suppose we can still live with it (note that the cross-reference search of old test failures is not affected by the upgrade, and will not loose any history). - Kristian.

2 3

Re: [Maria-developers] Optimization about count(*) of mysql 5.1.38
by Lichao Xie(谢立超) 24 Jan '11

24 Jan '11

> Hi, Lichao! > > On Jan 10, Lichao Xie wrote: >> hi all: >> I have a question about the optimization of count(*) in a range. >> Sql for mysql like:select count(*) from t where id > 10 and id < 1000000; >> In table t, there is an B-tree index on id field, this sql will read >> hundreds of thousands of records from the storage engine, that is a >> time-consuming operation. >> I am developing a storage engine, I want to know, is there a good >> method to caculate the records in a range, rather than read all >> records or keys in the range? > > Yes, see how MyISAM does it - it's pretty straightforward, and only > requires two index lookups. MyISAM seemingly read all keys in the range. And I test MyISAM table use mysqlslap, the test case is: ./mysqlslap --query="select count(xn_id) as num from roman1.r_paihang where dengji > 4 and dengji < 11" --number-of-queries=10000000 --concurrency=30 -uroot --host=10.134.11.17 --port=10001 the result is : 10.134.11.17:/data/mqq_data$ mysqladmin extended-status -i 1 -r -uroot -S /tmp/mysql-tdmdb-14.sock | grep Com_select | Com_select | 41115084 | | Com_select | 9 | | Com_select | 14 | | Com_select | 23 | | Com_select | 25 | | Com_select | 36 | | Com_select | 8 | | Com_select | 26 | | Com_select | 34 | | Com_select | 24 | | Com_select | 16 | | Com_select | 10 | | Com_select | 8 | the table scheme is: CREATE TABLE `r_paihang` ( `xn_id` int(11) NOT NULL DEFAULT '0', `company` int(8) DEFAULT NULL, `dengji` int(6) DEFAULT NULL, `position` tinyint(3) DEFAULT '1', `tili` int(11) DEFAULT '0', `zhili` int(11) DEFAULT '0', `meili` int(11) DEFAULT '0', `worth` int(11) DEFAULT '0', `sex` int(1) DEFAULT '0', `online_time` int(11) DEFAULT NULL, `sign_time` datetime DEFAULT NULL, `last_time` datetime DEFAULT NULL, `yq_id` int(12) NOT NULL DEFAULT '0', PRIMARY KEY (`xn_id`), KEY `company` (`company`), KEY `worth` (`worth`), KEY `zhili` (`zhili`), KEY `position` (`position`), KEY `meili` (`meili`), KEY `tili` (`tili`), KEY `sex` (`sex`), KEY `dengji` (`dengji`), KEY `sign_time` (`sign_time`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8; the case runs too slow.... If there is a storage engine api like records_in_range() to get the exact records in the range from the storage engine, the result will be 10^4 qps for this case... > > Regards, > Sergei > 2011/1/20 Sergei Golubchik <serg(a)askmonty.org>: > Hi, Lichao! > > Please send questions like this to maria-developers(a)lists.launchpad.net. > In your replies make sure send replies to the list, e.g. use "Reply to All". > It's a public mailing list dedicated to MySQL and MariaDB internals, source > code, and related things. I am subscribed, so I'll see you mail there, and you > may be sure I will, because it won't be accidentally catched by my spam filter, > or sorted out in some obscure folder. Furthermore other subscribers will see > your question and could reply if I will be not available (e.g. I could be > travelling). > Thank you. Sarry, my fault! I am resending the email. > > On Jan 20, Lichao Xie wrote: >> Hi, Sergei >> >> 2011/1/20 Sergei Golubchik <serg(a)askmonty.org>: >> > >> > On Jan 10, Lichao Xie wrote: >> >> hi all: >> >> I have a question about the optimization of count(*) in a range. >> >> Sql for mysql like:select count(*) from t where id > 10 and id < 1000000; >> >> In table t, there is an B-tree index on id field, this sql will read >> >> hundreds of thousands of records from the storage engine, that is a >> >> time-consuming operation. >> >> I am developing a storage engine, I want to know, is there a good >> >> method to caculate the records in a range, rather than read all >> >> records or keys in the range? >> > >> > Yes, see how MyISAM does it - it's pretty straightforward, and only >> > requires two index lookups. >> > Regards, > Sergei >

2 6

Re: [Maria-developers] [Commits] Rev 3014: New table creation/renaming block added if old encoded table present. in file:///home/bell/maria/bzr/work-maria-5.1-CREATE/
by Sergei Golubchik 20 Jan '11

20 Jan '11

Hi, sanja! On Jan 13, sanja(a)askmonty.org wrote: > revno: 3014 > revision-id: sanja(a)askmonty.org-20110113180607-e5hc8zuugxxjtn6k > parent: psergey(a)askmonty.org-20110112083018-90rw058f00uslvoz > committer: sanja(a)askmonty.org > branch nick: work-maria-5.1-CREATE > timestamp: Thu 2011-01-13 20:06:07 +0200 > message: > New table creation/renaming block added if old encoded table present. First: there is no test case for the new (as compared with the previous patch) change. Second: I think "old_path" does not belong to the check_table_file_presence() function. The function simply checks if the file exists, there is no place for the concept of "old path" here. You need to take care of the renaming for upgrade and old path outside of check_table_file_presence(). Regards, Sergei

2 1

[Maria-developers] index_merge, @@sort_buffer_size, and Sort_XXX counters
by Sergey Petrunya 20 Jan '11

20 Jan '11

Hello, index_merge uses @@sort_buffer_size as a guide of how much space it can use for sorting. I think there is an issue with the way it is done. Before index_merge was introduced, @@sort_buffer_size was used only by filesort (also for couple of other cases like GROUP_CONCAT with sorting, but I consider them to be too rare to be of practical importance). This allowed the following: 1. One can set @@sort_buffer_size to control how much space is to be allocated 2. SHOW STATUS will produce these numbers: Sort_merge_passes Sort_range Sort_scan Sort_range+Sort_scan gives the total number of filesort invocations. Sort_merge_passes is the number of merges. One can get average amount of merges per sort operation by calculating: Sort_merge_passes / (Sort_range + Sort_scan) (1) If the number is high, then it means that it could be useful to increase @@sort_buffer_size. Now, if we take index_merge into the picture, the counters are much less useful. index_merge causes Sort_merge_passes to be incremented, but we dont have a counter for index_merge operations, so formula (1) becomes invalid. Possible solutions are: Option#1: introduce a counter Sort_index_merge_count (name subject to discussion), which will mean "number of index_merge sort union/intersection scans". Option#2: make it so that index_merge does not cause Sort_merge_passes to be incremented. I think that in this case we'll need a separate variable to control index_merge's buffer size. it could have @@sort_buffer_size value as default. Any comments? BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Optimization about count(*) of mysql 5.1.38
by Lichao Xie(谢立超) 19 Jan '11

19 Jan '11

hi all: I have a question about the optimization of count(*) in a range. Sql for mysql like:select count(*) from t where id > 10 and id < 1000000; In table t, there is an B-tree index on id field, this sql will read hundreds of thousands of records from the storage engine, that is a time-consuming operation. I am developing a storage engine, I want to know, is there a good method to caculate the records in a range, rather than read all records or keys in the range? For example, does mysql call an API like records_in_range() to get the records in a range? After all, the range could be caculated by reading twice index.

2 1

[Maria-developers] [psergey@askmonty.org: No "Using index" support in sort-intersect - intentional?]
by Sergey Petrunya 19 Jan '11

19 Jan '11

Another question: why is type='range' for sort_intersect? We have type=index_merge for intersect and [sort]_union, so it seems rather confusing that we have type=range for sort_intersect? ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- Date: Wed, 19 Jan 2011 11:11:14 +0300 From: Sergey Petrunya <psergey(a)askmonty.org> To: igor(a)askmonty.org Cc: Maria Developers <maria-developers(a)lists.launchpad.net> Subject: No "Using index" support in sort-intersect - intentional? Hello Igor, I was looking at sort-intersect and noticed the following: Consider this table: CREATE TABLE `t1` ( `k1` int(11) DEFAULT NULL, `k2` int(11) DEFAULT NULL, `filler` char(100) DEFAULT NULL, KEY `k1` (`k1`), KEY `k2` (`k2`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1; Let's try a sort-intersect query: MariaDB [timerge1]> EXPLAIN select 1 from t1 where k1<30 and k2 < 30\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: range possible_keys: k1,k2 key: k1,k2 key_len: 5,5 ref: NULL rows: 1 Extra: Using sort_intersect(k1,k2); Using where 1 row in set (0.01 sec) The query doesn't use any columns not covered by the used indexes, however, we don't see "using index". Analogous ROR-intersect query will use "Using index": MariaDB [timerge1]> EXPLAIN select 1 from t1 where k1=30 and k2 = 30\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: index_merge possible_keys: k1,k2 key: k1,k2 key_len: 5,5 ref: NULL rows: 1 Extra: Using intersect(k1,k2); Using where; Using index 1 row in set (0.00 sec) BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] No "Using index" support in sort-intersect - intentional?
by Sergey Petrunya 19 Jan '11

19 Jan '11

Hello Igor, I was looking at sort-intersect and noticed the following: Consider this table: CREATE TABLE `t1` ( `k1` int(11) DEFAULT NULL, `k2` int(11) DEFAULT NULL, `filler` char(100) DEFAULT NULL, KEY `k1` (`k1`), KEY `k2` (`k2`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1; Let's try a sort-intersect query: MariaDB [timerge1]> EXPLAIN select 1 from t1 where k1<30 and k2 < 30\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: range possible_keys: k1,k2 key: k1,k2 key_len: 5,5 ref: NULL rows: 1 Extra: Using sort_intersect(k1,k2); Using where 1 row in set (0.01 sec) The query doesn't use any columns not covered by the used indexes, however, we don't see "using index". Analogous ROR-intersect query will use "Using index": MariaDB [timerge1]> EXPLAIN select 1 from t1 where k1=30 and k2 = 30\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: index_merge possible_keys: k1,k2 key: k1,k2 key_len: 5,5 ref: NULL rows: 1 Extra: Using intersect(k1,k2); Using where; Using index 1 row in set (0.00 sec) BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] review: [Commits] Rev 2875: Fix problem of making on-disk cache table. in file:///home/bell/maria/bzr/work-maria-5.3-subquerycache/
by Michael Widenius 17 Jan '11

17 Jan '11

Hi! Here is the review: >>>>> "sanja" == sanja <sanja(a)askmonty.org> writes: <cut> sanja> Fix problem of making on-disk cache table. sanja> Removed 'uniques=1' which lead to creating unneeded unique index creation. <cut> ---------------------------------------------------------------------- === modified file 'sql/sql_expression_cache.cc' --- a/sql/sql_expression_cache.cc 2010-11-02 09:47:36 +0000 +++ b/sql/sql_expression_cache.cc 2011-01-05 19:37:56 +0000 <cut> @@ -123,7 +126,6 @@ void Expression_cache_tmptable::init() goto error; } cache_table->s->keys= 1; - cache_table->s->uniques= 1; ref.null_rejecting= 1; ref.disable_cache= FALSE; ref.has_record= 0; The above should now already be in 5.3 tree as part of Sergei's push of my patch). (Just for your information) @@ -195,10 +197,12 @@ Expression_cache::result Expression_cach if (res) { subquery_cache_miss++; + miss++; DBUG_RETURN(MISS); } subquery_cache_hit++; + hit++; *value= cached_result; DBUG_RETURN(Expression_cache::HIT); As subquery_cache_miss and hit are global statistics variables, please use statistic_increment(subquery_cache_miss, &LOCK_status) with these. } @@ -225,7 +229,7 @@ my_bool Expression_cache_tmptable::put_v DBUG_ENTER("Expression_cache_tmptable::put_value"); DBUG_ASSERT(inited); - if (!cache_table) + if (!cache_table || frozen) { Is it not eonugh to test for 'frozen'? (You should set frozen to 0 if cache_table is not set) @@ -239,12 +243,22 @@ my_bool Expression_cache_tmptable::put_v if ((error= cache_table->file->ha_write_row(cache_table->record[0]))) { /* create_myisam_from_heap will generate error if needed */ + /* + TODO: add estimation if ondisk table is efficient if (cache_table->file->is_fatal_error(error, HA_CHECK_DUP) && create_internal_tmp_table_from_heap(table_thd, cache_table, cache_table_param.start_recinfo, &cache_table_param.recinfo, error, 1)) goto err; + */ Please change comment to #ifdef OVERFLOW_CACHE_TO_DISK_BASED_TABLE + if (cache_table->file->is_fatal_error(error, HA_CHECK_DUP)) + goto err; + /* should we leave the table */ + if ((((ulonglong)hit) * 100) / miss < EXPRASSION_TMPTABLE_ALLOWED_HITRATE) What happens if miss is 0 Shouldn't the above be? ((ulonglong) hit*100) / (hit + miss))... + goto err; // remove the temporary table + else Remove else + frozen= TRUE; // leave the temporary table } Try to move the above to before #ifdef and make it in such that one can just define the ifdef if one wants overflow to disk. === modified file 'sql/sql_expression_cache.h' <cut> @@ -50,6 +50,8 @@ class Item_field; class Expression_cache_tmptable :public Expression_cache { + ulong hit, miss; + bool frozen; ulong -> ulonglong (in a join you can have a LOT of row combinations) public: Expression_cache_tmptable(THD *thd, List<Item*> &dependants, Item *value); virtual ~Expression_cache_tmptable(); Ok to push after above changes. Regards, Monty

2 1

Re: [Maria-developers] test this list
by Vladislav Vaintroub 17 Jan '11

17 Jan '11

> -----Original Message----- > From: Vladislav Vaintroub [mailto:wlad@montyprogram.com] > Sent: Montag, 17. Januar 2011 13:12 > To: maria-developers(a)lists.launchpad.net > Subject: test this list > > Please ignore

1 0

[Maria-developers] lp: url support in firefox
by Sergei Golubchik 15 Jan '11

15 Jan '11

Hi. I've just written a small firefox add-on that adds support for lp: URI scheme to firefox. Nothing too magical, the add-on simply rewrites the url, replacing lp: with http://code.launchpad.net/ but at least now one can paste bzr urls (e.g. from bzr info, or email) directly into firefox and go to the corresponding branch page. https://addons.mozilla.org/en-US/firefox/addon/lp-uri-handler/ Regards, Sergei

1 0

[Maria-developers] test please ignore
by Vladislav Vaintroub 14 Jan '11

14 Jan '11

Checking mailing list access

1 0

Re: [Maria-developers] [Commits] Rev 2876: Fixed LP bug #698882. in file:///home/igor/maria/maria-5.3-bugs/
by Oleksandr Byelkin 14 Jan '11

14 Jan '11

Hi! On 13.01.2011 01:11, Igor Babaev wrote: > At file:///home/igor/maria/maria-5.3-bugs/ > > ------------------------------------------------------------ > revno: 2876 > revision-id: igor(a)askmonty.org-20110112231059-prhv2vncj6y2cizn > parent: igor(a)askmonty.org-20110105230330-2nzlm8641wcrwzio > committer: Igor Babaev<igor(a)askmonty.org> > branch nick: maria-5.3-bugs > timestamp: Wed 2011-01-12 15:10:59 -0800 > message: > Fixed LP bug #698882. > Made sure that the optimal fields are used by TABLE_REF objects > when building index access keys to joined tables. > Fixed a bug in the template function that sorts the elements of > a list using the bubble sort algorithm. The bug caused poor > performance of the function. Also added an optimization that > skips comparison with the most heavy elements that has been > already properly placed in the list. > Made the comparison of the fields belonging to the same Item_equal > more granular: fields belonging to the same table are also ordered > according to some rules. The patch as OK. But I'd prefer to have comment that the lists are mostly short and because of this used bubble sort algorithm (or you could use more detailed explanation from your letter to me). Also in big 'if (!cmp)' used tabulation (it is visible if you check diff where layout looks not correct, but it is correct in the result file), if it is not difficult it would be better to remove them. [skip]

1 0

[Maria-developers] WL#174 New (by Monty): Safety improvements for HANDLER commands
by worklog-noreply＠askmonty.org 14 Jan '11

14 Jan '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Safety improvements for HANDLER commands CREATION DATE..: Fri, 14 Jan 2011, 12:43 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 174 (http://askmonty.org/worklog/?tid=174) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 20 (hours remain) ORIG. ESTIMATE.: 20 PROGRESS NOTES: DESCRIPTION: There is some known problems with HANDLER implementation in some storage engines, that could potentially cause crashes that should be fixed ASAP. We should also do some trivial fixes to ensure that one gets Error 1020 (ER_CHECKREAD) Record has changed since last read in table '...' in so few cases as possible. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Documentation for Percona patch for row-based replication of pk-less tables
by Kristian Nielsen 13 Jan '11

13 Jan '11

Hi Daniel, I merged the Percona patch row_based_replication_without_primary_key.patch into 5.2-percona. This will later be merged into MariaDB 5.3. Here is some documentation, I was not sure where to put it, so hopefully you can help. The patch adds a new feature (performance improvement), and also fixes a bug. Incidentally, do we have a list of all bugs fixed in MariaDB over MySQL? I think that would be quite useful, even if it might get a bit out of date as MySQL also fixes some bugs. I suppose it could mostly be compiled from your release notes. This would also provide motivation for developers to actually make a proper bug report on issues fixed and proper commit messages etc :-) Anyway, just a suggestion. - Kristian. ----------------------------------------------------------------------- Row-based replication of tables with no primary key. MariaDB improves on row-based replication of tables which have no primary key but do have some other index. This is based in part on the original Percona patch "row_based_replication_without_primary_key.patch", with some additional fixes and enhancements. When row-based replication is used with UPDATE or DELETE, the slave needs to locate each replicated row based on the value in columns. If the table contains at least one index, an index lookup will be used (otherwise a table scan is needed for each row, which is extremely inefficient for all but the smallest table and generally to be avoided). >From MariaDB 5.3, the slave will try to choose a good index among any available: - The primary key is used, if there is one. - Else, the first unique index without NULL-able columns is used, if there is one. - Else, a choice is made among any normal indexes on the table (eg. a FULLTEXT index is not considered). The choice of which of several non-unique indexes to use is based on the cardinality of indexes; the one that is most selective (has the smallest average number of rows per distinct tuple of column values) is prefered. Note that for this choice to be effective, it is for most storage engines (like MyISAM, InnoDB) necessary to make sure ANALYZE TABLE has been run on the slave, otherwise statistics about index cardinality will not be available. In the absense of index cardinality, the first unique index will be chosen, if any, else the first non-unique index. Prior to MariaDB 5.3, the slave would always choose the first index without considering cardinality. The slave could even choose an unusable index (like FULLTEXT) if no other index was available (Bug #58997), causing row-based replication to break in this case; this is also fixed in MariaDB 5.3.

2 1

[Maria-developers] MWL#47 documentation
by Kristian Nielsen 12 Jan '11

12 Jan '11

Hi Daniel, I merged the MWL#47 patch into 5.2-percona, which will later be included in 5.3. I have attached the documentation for this worklog that Alexi wrote (he also implemented the feature itself). I hope you can adapt it as needed and find a good place for it in the server documentation. - Kristian. WL#47: Store in binlog text of statements that caused RBR events ================================================================ Content ~~~~~~~ 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] 2. Example 3. Master option: --binlog-annotate-rows-events 4. Slave option: --replicate-annotate-rows-events 5. mysqlbinlog option: --skip-annotate-rows-events 6. Example of mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ New binlog event type - Annotate_rows_log_event - is added. The events of this type accompany rows events and describe the query which caused these rows events. In binary log, each Annotate_rows event precedes the corresponding Table map event or the first of Table map events, if there are more than one (e.g. in case of multi-delete or insert delayed). 2. Example ~~~~~~~~~~ master> DROP DATABASE IF EXISTS test; master> CREATE DATABASE test; master> USE test; master> CREATE TABLE t1(a int); master> INSERT INTO t1 VALUES (1), (2), (3); master> CREATE TABLE t2(a int); master> INSERT INTO t2 VALUES (1), (2), (3); master> CREATE TABLE t3(a int); master> INSERT DELAYED INTO t3 VALUES (1), (2), (3); master> DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 -> WHERE t1.a=t2.a AND t2.a=t3.a; master> SHOW BINLOG EVENTS IN 'master-bin.000001'; +-------------------+------+---------------+-----------+-------------+---------------------------------------------------+ | Log_name | Pos | Event_type | Server_id | End_log_pos | Info | +-------------------+------+---------------+-----------+-------------+---------------------------------------------------+ | master-bin.000001 | 4 | Format_desc | 100 | 240 | Server ver: 5.1.44-debug-log, Binlog ver: 4 | | master-bin.000001 | 240 | Query | 100 | 331 | DROP DATABASE IF EXISTS test | | master-bin.000001 | 331 | Query | 100 | 414 | CREATE DATABASE test | | master-bin.000001 | 414 | Query | 100 | 499 | use `test`; CREATE TABLE t1(a int) | | master-bin.000001 | 499 | Query | 100 | 567 | BEGIN | | master-bin.000001 | 567 | Annotate_rows | 100 | 621 | INSERT INTO t1 VALUES (1), (2), (3) | | master-bin.000001 | 621 | Table_map | 100 | 662 | table_id: 16 (test.t1) | | master-bin.000001 | 662 | Write_rows | 100 | 706 | table_id: 16 flags: STMT_END_F | | master-bin.000001 | 706 | Query | 100 | 775 | COMMIT | | master-bin.000001 | 775 | Query | 100 | 860 | use `test`; CREATE TABLE t2(a int) | | master-bin.000001 | 860 | Query | 100 | 928 | BEGIN | | master-bin.000001 | 928 | Annotate_rows | 100 | 982 | INSERT INTO t2 VALUES (1), (2), (3) | | master-bin.000001 | 982 | Table_map | 100 | 1023 | table_id: 17 (test.t2) | | master-bin.000001 | 1023 | Write_rows | 100 | 1067 | table_id: 17 flags: STMT_END_F | | master-bin.000001 | 1067 | Query | 100 | 1136 | COMMIT | | master-bin.000001 | 1136 | Query | 100 | 1221 | use `test`; CREATE TABLE t3(a int) | | master-bin.000001 | 1221 | Query | 100 | 1289 | BEGIN | | master-bin.000001 | 1289 | Annotate_rows | 100 | 1351 | INSERT DELAYED INTO t3 VALUES (1), (2), (3) | | master-bin.000001 | 1351 | Table_map | 100 | 1392 | table_id: 18 (test.t3) | | master-bin.000001 | 1392 | Write_rows | 100 | 1426 | table_id: 18 flags: STMT_END_F | | master-bin.000001 | 1426 | Table_map | 100 | 1467 | table_id: 18 (test.t3) | | master-bin.000001 | 1467 | Write_rows | 100 | 1506 | table_id: 18 flags: STMT_END_F | | master-bin.000001 | 1506 | Query | 100 | 1575 | COMMIT | | master-bin.000001 | 1575 | Query | 100 | 1643 | BEGIN | | master-bin.000001 | 1643 | Annotate_rows | 100 | 1748 | DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a | | master-bin.000001 | 1748 | Table_map | 100 | 1789 | table_id: 16 (test.t1) | | master-bin.000001 | 1789 | Table_map | 100 | 1830 | table_id: 17 (test.t2) | | master-bin.000001 | 1830 | Delete_rows | 100 | 1874 | table_id: 16 | | master-bin.000001 | 1874 | Delete_rows | 100 | 1918 | table_id: 17 flags: STMT_END_F | | master-bin.000001 | 1918 | Query | 100 | 1987 | COMMIT | +-------------------+------+---------------+-----------+-------------+---------------------------------------------------+ 3. Master option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 4. Slave option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only (only in config file) * Data Type: bool * Default Value: OFF 5. mysqlbinlog option: --skip-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells mysqlbinlog to skip all Annotate_rows events in its output (by default, mysqlbinlog prints Annotate_rows events, if the binary log does contain them). 6. Example of mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ...> mysqlbinlog.exe -vv -R --user=root --port=3306 --host=localhost master-bin.000001 /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; # at 4 #100516 15:36:00 server id 100 end_log_pos 240 Start: binlog v 4, server v 5.1.44-debug-log created 100516 15:36:00 at startup ROLLBACK/*!*/; BINLOG ' oNjvSw9kAAAA7AAAAPAAAAAAAAQANS4xLjQ0LWRlYnVnLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAACg2O9LEzgNAAgAEgAEBAQEEgAA2QAEGggAAAAICAgCAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAA= '/*!*/; # at 240 #100516 15:36:18 server id 100 end_log_pos 331 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; SET @@session.pseudo_thread_id=1/*!*/; SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=1, @@session.unique_checks=1, @@session.autocommit=1 /*!*/; SET @@session.sql_mode=0/*!*/; SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/; /*!\C latin1 *//*!*/; SET @@session.character_set_client=8,@@session.collation_connection=8,@@session.collation_server=8/*!*/; SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; DROP DATABASE IF EXISTS test /*!*/; # at 331 #100516 15:36:18 server id 100 end_log_pos 414 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; CREATE DATABASE test /*!*/; # at 414 #100516 15:36:18 server id 100 end_log_pos 499 Query thread_id=1 exec_time=0 error_code=0 use test/*!*/; SET TIMESTAMP=1274009778/*!*/; CREATE TABLE t1(a int) /*!*/; # at 499 #100516 15:36:18 server id 100 end_log_pos 567 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; BEGIN /*!*/; # at 567 # at 621 # at 662 #100516 15:36:18 server id 100 end_log_pos 621 Annotate_rows: #Q> INSERT INTO t1 VALUES (1), (2), (3) #100516 15:36:18 server id 100 end_log_pos 662 Table_map: `test`.`t1` mapped to number 16 #100516 15:36:18 server id 100 end_log_pos 706 Write_rows: table id 16 flags: STMT_END_F BINLOG ' stjvSxNkAAAAKQAAAJYCAAAAABAAAAAAAAAABHRlc3QAAnQxAAEDAAE= stjvSxdkAAAALAAAAMICAAAQABAAAAAAAAEAAf/+AQAAAP4CAAAA/gMAAAA= '/*!*/; ### INSERT INTO test.t1 ### SET ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ### INSERT INTO test.t1 ### SET ### @1=2 /* INT meta=0 nullable=1 is_null=0 */ ### INSERT INTO test.t1 ### SET ### @1=3 /* INT meta=0 nullable=1 is_null=0 */ # at 706 #100516 15:36:18 server id 100 end_log_pos 775 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; COMMIT /*!*/; # at 775 #100516 15:36:18 server id 100 end_log_pos 860 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; CREATE TABLE t2(a int) /*!*/; # at 860 #100516 15:36:18 server id 100 end_log_pos 928 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; BEGIN /*!*/; # at 928 # at 982 # at 1023 #100516 15:36:18 server id 100 end_log_pos 982 Annotate_rows: #Q> INSERT INTO t2 VALUES (1), (2), (3) #100516 15:36:18 server id 100 end_log_pos 1023 Table_map: `test`.`t2` mapped to number 17 #100516 15:36:18 server id 100 end_log_pos 1067 Write_rows: table id 17 flags: STMT_END_F BINLOG ' stjvSxNkAAAAKQAAAP8DAAAAABEAAAAAAAAABHRlc3QAAnQyAAEDAAE= stjvSxdkAAAALAAAACsEAAAQABEAAAAAAAEAAf/+AQAAAP4CAAAA/gMAAAA= '/*!*/; ### INSERT INTO test.t2 ### SET ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ### INSERT INTO test.t2 ### SET ### @1=2 /* INT meta=0 nullable=1 is_null=0 */ ### INSERT INTO test.t2 ### SET ### @1=3 /* INT meta=0 nullable=1 is_null=0 */ # at 1067 #100516 15:36:18 server id 100 end_log_pos 1136 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; COMMIT /*!*/; # at 1136 #100516 15:36:18 server id 100 end_log_pos 1221 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; CREATE TABLE t3(a int) /*!*/; # at 1221 #100516 15:36:18 server id 100 end_log_pos 1289 Query thread_id=2 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; BEGIN /*!*/; # at 1289 # at 1351 # at 1392 #100516 15:36:18 server id 100 end_log_pos 1351 Annotate_rows: #Q> INSERT DELAYED INTO t3 VALUES (1), (2), (3) #100516 15:36:18 server id 100 end_log_pos 1392 Table_map: `test`.`t3` mapped to number 18 #100516 15:36:18 server id 100 end_log_pos 1426 Write_rows: table id 18 flags: STMT_END_F BINLOG ' stjvSxNkAAAAKQAAAHAFAAAAABIAAAAAAAAABHRlc3QAAnQzAAEDAAE= stjvSxdkAAAAIgAAAJIFAAAQABIAAAAAAAEAAf/+AQAAAA== '/*!*/; ### INSERT INTO test.t3 ### SET ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ # at 1426 # at 1467 #100516 15:36:18 server id 100 end_log_pos 1467 Table_map: `test`.`t3` mapped to number 18 #100516 15:36:18 server id 100 end_log_pos 1506 Write_rows: table id 18 flags: STMT_END_F BINLOG ' stjvSxNkAAAAKQAAALsFAAAAABIAAAAAAAAABHRlc3QAAnQzAAEDAAE= stjvSxdkAAAAJwAAAOIFAAAQABIAAAAAAAEAAf/+AgAAAP4DAAAA '/*!*/; ### INSERT INTO test.t3 ### SET ### @1=2 /* INT meta=0 nullable=1 is_null=0 */ ### INSERT INTO test.t3 ### SET ### @1=3 /* INT meta=0 nullable=1 is_null=0 */ # at 1506 #100516 15:36:18 server id 100 end_log_pos 1575 Query thread_id=2 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; COMMIT /*!*/; # at 1575 #100516 15:36:18 server id 100 end_log_pos 1643 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; BEGIN /*!*/; # at 1643 # at 1748 # at 1789 # at 1830 # at 1874 #100516 15:36:18 server id 100 end_log_pos 1748 Annotate_rows: #Q> DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 #Q> WHERE t1.a=t2.a AND t2.a=t3. #100516 15:36:18 server id 100 end_log_pos 1789 Table_map: `test`.`t1` mapped to number 16 #100516 15:36:18 server id 100 end_log_pos 1830 Table_map: `test`.`t2` mapped to number 17 #100516 15:36:18 server id 100 end_log_pos 1874 Delete_rows: table id 16 #100516 15:36:18 server id 100 end_log_pos 1918 Delete_rows: table id 17 flags: STMT_END_F BINLOG ' stjvSxNkAAAAKQAAAP0GAAAAABAAAAAAAAAABHRlc3QAAnQxAAEDAAE= stjvSxNkAAAAKQAAACYHAAAAABEAAAAAAAAABHRlc3QAAnQyAAEDAAE= stjvSxlkAAAALAAAAFIHAAAAABAAAAAAAAAAAf/+AQAAAP4CAAAA/gMAAAA= ### DELETE FROM test.t1 ### WHERE ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ### DELETE FROM test.t1 ### WHERE ### @1=2 /* INT meta=0 nullable=1 is_null=0 */ ### DELETE FROM test.t1 ### WHERE ### @1=3 /* INT meta=0 nullable=1 is_null=0 */ stjvSxlkAAAALAAAAH4HAAAQABEAAAAAAAEAAf/+AQAAAP4CAAAA/gMAAAA= '/*!*/; ### DELETE FROM test.t2 ### WHERE ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ### DELETE FROM test.t2 ### WHERE ### @1=2 /* INT meta=0 nullable=1 is_null=0 */ ### DELETE FROM test.t2 ### WHERE ### @1=3 /* INT meta=0 nullable=1 is_null=0 */ # at 1918 #100516 15:36:18 server id 100 end_log_pos 1987 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; COMMIT /*!*/; DELIMITER ; # End of log file ROLLBACK /* added by mysqlbinlog */; /*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;

1 0

[Maria-developers] MWL#36 documentation
by Kristian Nielsen 12 Jan '11

12 Jan '11

Hi Daniel, While merging Percona patches, I noticed a couple of issues with our documentation relating to MWL#36, I was hoping you could handle them: - The MWL#36 is mentioned in the changelog/release notes: http://kb.askmonty.org/v/mariadb-520-release-notes However it seems to be missing from "mariadb-vs-mysql": http://kb.askmonty.org/v/mariadb-versus-mysql-features - There is some documentation of the MWL#36 in the form of a table entry for --rewrite-db here: http://kb.askmonty.org/v/mysqlbinlog However, that page mentions the feature introduced in MariaDB 5.2.1, while the release notes say 5.2.0. I wonder which is correct? - Kristian.

1 0

[Maria-developers] Multiple default values for timestamps
by Dolf Schimmel 09 Jan '11

09 Jan '11

Hey folks, On IRC I read somewhere that multiple timestamps cant have multiple default values using NOW() (error ER_TOO_MUCH_AUTO_TIMESTAMP_COLS). I tried to find out why this limitation is in place, but was not able to do so. The only thing I (or rather someone in #mysql and #maria) was able to find was this, in sql/field.cc (line 4729): > Actually SQL-99 says that we should allow niladic functions (like NOW()) > as defaults for any field. Current limitations (only NOW() and only > for one TIMESTAMP field) are because of restricted binary .frm format > and should go away in the future. I removed the check that spits out this error and updated the tests. Now I'll admit I'm not sure how to properly test this, but so far the entire testsuite passes. It may be that I'm missing any reasons for this limitation, or that tests may be missing/inaccurate. If you have any opinions on this, please let me know. You can find it in: lp:~freeaqingme/maria/improv-allowMultipleCurTimestampInDefaultOnTable Regards, Dolf Schimmel -- Freeaqingme p.s. Sorry for the long branchname, couldn't think of something shorter that was still descriptive :D

2 3

[Maria-developers] WL#173 New (by Sergei): temporal types with sub-second resolution
by worklog-noreply＠askmonty.org 09 Jan '11

09 Jan '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: temporal types with sub-second resolution CREATION DATE..: Sun, 09 Jan 2011, 20:08 SUPERVISOR.....: Sergei IMPLEMENTOR....: Sergei COPIES TO......: Monty CATEGORY.......: Server-Sprint TASK ID........: 173 (https://askmonty.org/worklog/?tid=173) VERSION........: Server-5.3 STATUS.........: Assigned PRIORITY.......: 80 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: We will add a support for timestamps (and other temporal types) with a sub-second resolution. Syntax ^^^^^^ The syntax for declaring a column is the conventional TIMESTAMP(X) DATETIME(X) TIME(X) where X is the scale - that is, number of digits after the decimal dot. TIMESTAMP(0) is the same as the current TIMESTAMP. The existing syntax (without specifying the scale) will, of course, work as before. X=6 means microseconds, X=9 means nanoseconds, X=12 means picoseconds. The max supported value of X is [undecided], but will be no less than 6. Possibly, we will do 6 in this WL - because it keeps syntax and protocol changes to the minimum, microseconds are already supported there. If needed, we will increase the timestamp precision later - the implementation (see below) needs to keep it in mind. The time and datetime literals can be written with the fractional part, for example '2011-01-08 20:29:26.1234567890'. This is alrteady supported. Client-server protocol ^^^^^^^^^^^^^^^^^^^^^^ We will not change the binary client-server protocol. It only supports microseconds in the MYSQL_TIME::second_part - we will truncate longer timestamps to this precision. Show commands and I_S ^^^^^^^^^^^^^^^^^^^^^ SHOW COLUMNS and SHOW CREATE TABLE will show TIMESTAMP(X) as appropriate (similar to INT(N) and DECIMAL(M,N) types). I_S.COLUMNS will *not* show '(X)' in the DATA_TYPE column, but it *will* show it in the COLUMN_TYPE column. It will show X in the new DATETIME_PRECISION column. Functions ^^^^^^^^^ The following functions need to be fixed to support sub-seconds: CAST() - as DATETIME(X), etc CONVERT_TZ() SEC_TO_TIME() TIME_TO_SEC() MAKETIME() - perhaps? FROM_UNIXTIME() - probably not UNIX_TIMESTAMP() - probably not UTC_TIMESTAMP(), UTC_TIME(), NOW(), CURTIME() and synonyms - probably not, high-precision NOW() is a separate task P.S. See also http://forge.mysql.com/worklog/task.php?id=946 ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] The query cache patch fixed by me and then Monty (most changes by him)
by Oleksandr Byelkin 07 Jan '11

07 Jan '11

Hi! Here is changes which was done: - Fixed switching off of the query cache -- memory freed now -- after completely switching of it could be switched on again - Cleaned up saving the query without comments and space - Added all the missing test cases (and checked results) - Added more symbols to LEX to make parsing faster (not part of this thing, but I thought it was better to get this done now as I was looking at the code) - Fixed all the issues that I had in my review of query parsing - Cleaned up mysql-tests -- Made test results easier to read -- removed 'percona' prefix from tests names -- Made test result files notable smaller - Added 'thd' to query_cache::invalidate calls, to remove some current_thd usages - Added variable thd->query_cache_is_valid, that is set to 1 if query_cache could possible read query from the query_cache -- If not, there is no reason to put the query in the cache The above change allowed me to remove safe_to_cache_queries from send_result_to_client() and make this part of the code not depending in lex_start() - simplified a couple of functions to use pointers instead of index - Fixed bug where SQL_NO_CACHE was not detected if followed by tab, newline or return i> that's about it 2 Vadim & Oleg: If you OK with changes above and the patch then it will be included in our tree.

2 3

[Maria-developers] [patch] A few typos
by Dolf Schimmel 07 Jan '11

07 Jan '11

Morning folks, I was looking at the source for a certain function in MariaDB (see the patch and you'll know which one :D ). While grepping I found a couple of typos and since I don't have commit rights, and couldn't find where to send the patch, I decided to send it here :) The diff is made against trunk, but I guess it should be fairly easy to patch against other branches as well. Kind regards, Dolf Schimmel -- Freeaqingme

2 1

Re: [Maria-developers] Restarting cooperation of Percona and Monty Programming AB
by Kristian Nielsen 06 Jan '11

06 Jan '11

Michael Widenius <monty(a)askmonty.org> writes: >>>>>> "Kristian" == Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > > <cut> > > Kristian> Anyway, I fixed the patch, added test cases, and merged into MariaDB. It is > Kristian> pushed here: > > Kristian> lp:~knielsen/maria/5.2-percona-row_based_replication_without_primary_key > > Kristian> Here is the commit message: > > Kristian> Merge Percona patch row_based_replication_without_primary_key.patch into MariaDB. > > Kristian> This patch improves the selection of index to use to apply row-based > Kristian> DELETE and UPDATE events on tables with no primary key (original code > Kristian> picks the first index unconditionally). > Kristian> If ANALYZE TABLE is done, the index cardinalities will be compared and > Kristian> the best index will be used. > > Kristian> Fixes some problems in the original patch: > > Kristian> - Without ANALYZE TABLE, rec_per_key statistics is not available; in this > Kristian> case the original patch could choose a really bad index, even ignoring > Kristian> a primary key. > > Kristian> - The original patch did not consider multi-column keys correctly, and > Kristian> could thus pick a less desirable single-column key over a good > Kristian> multi-column index. > > Kristian> Also fixes Bug#58997, and adds test cases. > > Kristian> Monty, how do I include it in our percona tree?, can I just push here: > > Kristian> lp:~maria-captains/maria/5.2-percona > > Yes, feel free to push if no one from Percona has any problems with > your patch. So, any objections or comments from Percona on this patch? I've included the patch below for convenience. - Kristian.

2 2

[Maria-developers] WL#172 New (by Monty): Add support for prepared statements to HANDLER READ
by worklog-noreply＠askmonty.org 03 Jan '11

03 Jan '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add support for prepared statements to HANDLER READ CREATION DATE..: Mon, 03 Jan 2011, 19:19 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 172 (http://askmonty.org/worklog/?tid=172) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: DESCRIPTION: Add support for prepared statements to HANDLER READ. As part of this work, move some of the word done during handler_read to handler_open to speed up HANDLER READ calls. Example: handler t1 open; prepare stmt from 'handler t1 read a=(?) limit ?,?'; set @a=16,@b=1,@c=100; execute stmt using @a,@b,@c; deallocate prepare stmt; ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

Re: [Maria-developers] [Commits] Rev 2999: Speed up `mtr --parallel=<lots>` by scheduling some slow tests earlier. in http://bazaar.launchpad.net/~maria-captains/maria/5.1
by Sergei Golubchik 03 Jan '11

03 Jan '11

Hi, knielsen! On Jan 03, knielsen(a)knielsen-hq.org wrote: > At http://bazaar.launchpad.net/~maria-captains/maria/5.1 Hm. Why do you prefer to add a dummy include file to every "long" test, instead of using a separate file, like disabled.def ? I think it'd be easier to maintain in a separate file, and easier to extend later on. Regards, Sergei

2 2

[Maria-developers] Logoff trigger for MariaDB
by Oli Sennhauser 31 Dec '10

31 Dec '10

Hi all, As suggested by Sergei we should discuss this here. Copying the login trigger stuff to a place in the code where I thought it makes sense I have created a logout trigger. This gives the capability to track a session from its beginning to its end and collects all the data from each session. With this capability we can start thinking about auditing of MySQL users like it is needed/wanted from big companies. I am NOT a developer and I was just lucky that it worked out how I did it. So I am not of much help for an other approach as Sergei hinted in his comment. Now: free to discuss... http://www.fromdual.com/mysql-logon-and-logoff-trigger-for-auditing Regards, Oli -- FromDual - Vendor independent and neutral MySQL consulting. Website http://www.fromdual.com/ Skype fromdual Jabber fromdual(a)swissjabber.ch Yahoo IM fromdual Xing https://www.xing.com/profile/Oliver_Sennhauser LinkedIn http://ch.linkedin.com/in/shinguz Twitter http://twitter.com/shinguz

2 2

Re: [Maria-developers] [Fwd: [Commits] Rev 2998: Fixed LP bug #639935 (bug #58727). in file:///home/igor/maria/maria-5.1-bug693935/]
by Oleksandr Byelkin 24 Dec '10

24 Dec '10

Hi! 24 Dec 2010, в 06:45, Igor Babaev написал(а): > Sanja, > > Please review this fix ASAP. > We need it to move on with 5.3 The patch is OK, but I would prefer to have comment before Item::quick_fix_field() that this method is only for optimizer needs, it can be used only for certain Items controlled by optimizer and maybe add that only Item_func_* need recursion for this. But you could write comments much better then I so it is up to you how write the comment. > > Regards. > Igor. > > -------- Original Message -------- > Subject: [Commits] Rev 2998: Fixed LP bug #639935 (bug #58727). in > file:///home/igor/maria/maria-5.1-bug693935/ > Date: Fri, 24 Dec 2010 06:42:05 +0200 (EET) > From: <igor(a)askmonty.org> > Reply-To: maria-developers(a)lists.launchpad.net > To: <commits(a)mariadb.org> > > At file:///home/igor/maria/maria-5.1-bug693935/ > > ------------------------------------------------------------ > revno: 2998 > revision-id: igor(a)askmonty.org-20101224044150-p9yja1e87riwjh7g > parent: monty(a)askmonty.org-20101221172350-wqtf0t6evpc0un4b > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.1-bug693935 > timestamp: Thu 2010-12-23 20:41:50 -0800 > message: > Fixed LP bug #639935 (bug #58727). > When the optimizer creates items out of other items it does > not have to call the fix_fields method. Usually in these > cases it calls quick_fix_field() that just marks the > created item as fixed. If the created item is an Item_func > object then calling quick_fix_field() works fine if the > arguments of the created functional item are already fixed. > Otherwise some unfixed nodes remain in the item tree and > it triggers an assertion failure whenever the item is > evaluated. > > Fixed the problem by making the method quick_fix_field > virtual and providing an implementation for the class > Item_func objects that recursively calls the method > for unfixed arguments of any functional item. [skip]

1 0

Re: [Maria-developers] pre-allocating binlog to speed up sync_binlog=1
by Sergei Golubchik 23 Dec '10

23 Dec '10

Hi, Arjen! [moving to the public list] On Dec 14, Arjen Lentz wrote: > Hi all > > Can we adopt/implement http://forge.mysql.com/worklog/task.php?id=4925 in > MariaDB? > The benchmark info is in the item, and looks quite interesting. > The author tested it using a separate tool to preallocate the binlog, but > it should be straightforward implementing this inside the server, that'd > also be much easier to manage without extra fuss for a DBA. You mean - before it gets into 5.6 and we merge it the normal way? Or you mean - with a different implementation (w/o mysqlbinlogalloc) ? Regards, Sergei P.S. These are the changesets: final version: http://lists.mysql.com/commits/113309 http://lists.mysql.com/commits/113306 http://lists.mysql.com/commits/113307 review comments: http://lists.mysql.com/commits/116965 http://lists.mysql.com/commits/121478

4 6

Re: [Maria-developers] [Commits] Rev 2959: New table creation/renaming block added if old encoded table present. in file:///home/bell/maria/bzr/work-maria-5.1-CREATE/
by Sergei Golubchik 20 Dec '10

20 Dec '10

Hi, sanja! On Oct 22, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-5.1-CREATE/ > > ------------------------------------------------------------ > revno: 2959 > revision-id: sanja(a)askmonty.org-20101021220232-eqso4ey46985wfzf > parent: sergii(a)pisem.net-20101020105843-7ybuqyc9n5328tie > committer: sanja(a)askmonty.org > branch nick: work-maria-5.1-CREATE > timestamp: Fri 2010-10-22 01:02:32 +0300 > message: > New table creation/renaming block added if old encoded table present. ok to push Regards, Sergei

1 0

[Maria-developers] Proposals for improving BKA/MRR visibility in EXPLAIN and diagnostics
by Sergey Petrunya 17 Dec '10

17 Dec '10

Hello, Below are some ideas on how to make DS-MRR/BKA easier to work with for the users (= those who don't run mysqld under debugger). Questions are marked with 'Q:' but any comments are welcome. Better EXPLAIN -------------- Philip has complained numerous times that it is not shown in EXPLAIN whether DS-MRR will use key sorting. Currently, MRR alone is show like this: MariaDB [test]> explain select * from t1 where key1<30; +----+-------------+-------+-------+---------------+------+---------+------+------+----------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+-------+---------------+------+---------+------+------+----------------------------------+ | 1 | SIMPLE | t1 | range | key1 | key1 | 5 | NULL | 1 | Using index condition; Using MRR | +----+-------------+-------+-------+---------------+------+---------+------+------+----------------------------------+ MRR with BKA are shown like this: MariaDB [test]> explain select * from t2,t1 where t2.a=t1.key1; +----+-------------+-------+------+---------------+------+---------+-----------+------+------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------+---------------+------+---------+-----------+------+------------------------------------+ | 1 | SIMPLE | t2 | ALL | NULL | NULL | NULL | NULL | 10 | Using where | | 1 | SIMPLE | t1 | ref | key1 | key1 | 5 | test.t2.a | 1 | Using join buffer (flat, BKA join) | +----+-------------+-------+------+---------------+------+---------+-----------+------+------------------------------------+ The suggestion is to - get rid of "Using MRR" word, - instead, show "Sort rowids" and/or "Sort keys". - Show the above two whenever rowid-ordered and/or key-ordered retrieval is performed. The above examples will look as follows: MariaDB [test]> explain select * from t1 where key1<30; +----+-------------+-------+-------+---------------+------+---------+------+------+-----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+-------+---------------+------+---------+------+------+-----------------------------------------------+ | 1 | SIMPLE | t1 | range | key1 | key1 | 5 | NULL | 1 | Using index condition; Sort keys; Sort rowids | +----+-------------+-------+-------+---------------+------+---------+------+------+-----------------------------------------------+ MRR with BKA are shown like this: MariaDB [test]> explain select * from t2,t1 where t2.a=t1.key1; +----+-------------+-------+------+---------------+------+---------+-----------+------+------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------+---------------+------+---------+-----------+------+------------------------------------------------------------+ | 1 | SIMPLE | t2 | ALL | NULL | NULL | NULL | NULL | 10 | Using where | | 1 | SIMPLE | t1 | ref | key1 | key1 | 5 | test.t2.a | 1 | Using join buffer (flat, BKA join); Sort keys; Sort rowids | +----+-------------+-------+------+---------------+------+---------+-----------+------+------------------------------------------------------------+ Q: any comments? Counters -------- Like it is done with other kinds of table accesses, we want to have counter-based way of analyzing of what has been happening around MRR. The most obvious are: 1. Handler_multi_range_read_init_count status variable This will tell how many multi_range_read_init() calls have been made that used non-default MRR implementation, i.e. one will be able to see how many times real MRR scans were performed. Q: this counter doesn't show how many times key sorting/rowid sorting/both strategies were used. Is that ok? One could argue that information about strategy choice is not in high demand as MRR strategy choice is based mostly on system settings and DDLs. 2. Handler_multi_range_read_next_count status variable This will tell how many records were returned by MRR to the upper layer. Q: MRR does index and rnd_pos scans under the hood, and these scans do increase counters. This means that, for a single row returned by MRR, multiple counters will be incremented. Less obvious suggestions: 3. It would be useful to have an idea about whether DS-MRR had sufficient buffer space to operate. One can get a rough picture by adding counters of key/rowid sort operations: - Handler_mrr_rowid_sort_count - Handler_mrr_key_sort_count If Handler_mrr_key_sort_count == Handler_multi_range_read_init_count then we've had enough buffer space. If key_sort_count is two times greater than the number of init operations, then one could conclude that on average two buffer refills were needed and increasing buffer size up to 2x would be beneficial. 4. Other possible things - Index condition pushdown check count, true/false ratio - Some statistics to globally check if the optimizer's guesses about required sizes for key/rowid parts of the buffer were any good. Q: do we see the need for any of the above? BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

3 3

[Maria-developers] WL#171 New (by Monty): new free MariaDB client library
by worklog-noreply＠askmonty.org 14 Dec '10

14 Dec '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: new free MariaDB client library CREATION DATE..: Tue, 14 Dec 2010, 19:51 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-Sprint TASK ID........: 171 (http://askmonty.org/worklog/?tid=171) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Create a new MariaDB/MySQL/Drizzle client library that is Apache, BSD or LGPL. This task is to correct the limitations of the old lgpl client library, MWL#134, as outlined in http://kb.askmonty.org/v/lgpl-mysql-client-library-32358. Minimum requirements are: * Should be binary link level compatible with the current MySQL 5.1 client library. To use it, one would just have to relink the application with the new library. No re-compilation of the source should be necessary. ** It should be able to connect to all current versions of MariaDB, MySQL and Drizzle. * It should support all the new functionality in MariaDB's client library: ** New functions ** Pluggable authentication ** The license should be one of BSD/LGPL/Apache The different development options are: * Use the old LGPL MySQL library from 3.23 as a base. * Use Drizzles client library as base. * Use one of the free MySQL client libraries used by some of the languages as a base. * Writing a new one from scratch. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

Re: [Maria-developers] [Commits] Rev 2867: Fix LP BUG#685411 in file:///home/tsk/mprog/src/5.3-lpb-685411/
by Oleksandr Byelkin 14 Dec '10

14 Dec '10

Hi! On 14.12.2010 14:08, timour(a)askmonty.org wrote: > At file:///home/tsk/mprog/src/5.3-lpb-685411/ > > ------------------------------------------------------------ > revno: 2867 > revision-id: timour(a)askmonty.org-20101214120805-vafrxb6i59u9gyef > parent: igor(a)askmonty.org-20101211205039-b1alcmyethjpddzu > committer: timour(a)askmonty.org > branch nick: 5.3-lpb-685411 > timestamp: Tue 2010-12-14 14:08:05 +0200 > message: > Fix LP BUG#685411 Ok, to push. [skip]

1 0

[Maria-developers] [PATCH] Make innodb_bug57255.test 10-20 times faster
by Stewart Smith 14 Dec '10

14 Dec '10

Real simple patch, makes the test run much faster if using a real storage device without libeatmydata. patch should be appliable (just change the path) ------------------------------------------------------------ revno: 2477 committer: Stewart Smith <stewart(a)flamingspork.com> branch nick: update-innobase timestamp: Tue 2010-12-07 13:40:36 +1100 message: make innodb_bug57255 test run 10-20 times faster (use a txn when loading data) modified: plugin/innobase/tests/t/innodb_bug57255.test diff: === modified file 'plugin/innobase/tests/t/innodb_bug57255.test' --- plugin/innobase/tests/t/innodb_bug57255.test 2010-12-07 02:40:01 +0000 +++ plugin/innobase/tests/t/innodb_bug57255.test 2010-12-07 02:40:36 +0000 @@ -12,6 +12,7 @@ insert into A values(1), (2); --disable_query_log +begin; let $i=257; while ($i) { @@ -24,6 +25,7 @@ insert into C(f1) values(2); dec $i; } +commit; --enable_query_log # Following Deletes should not report error -- Stewart Smith

3 2

[Maria-developers] DBT-3/TPC-H RQG tests are now available
by Philip Stoev 14 Dec '10

14 Dec '10

Hello, In response to popular demand, the DBT-3 dataset will be used for testing along with a new set of grammars that generate queries against that dataset. While I personally doubt that the realism of the DBT-3, it being 99% random, here is what we have so far: 1. DBT-3 datasets for scales 0.1 0.01 and 0.001 2. RQG grammars that implement the following: - a grammar on the range optimizer via single-table queries against the lineitem database. The WHERE clause consists of nested AND and OR expressions of varying depth, where each individual expression involves an indexed column and is generated to be realistic. E.g. for a date column, we generate expressions that filter records for a specific month or a specific year. - a grammar for general join tests - realistic multi-table joins are generated by observing the star structure of the dataset. The WHERE and HAVING conditions are generated to be realistic with respect to the columns being queried or filtered out. GROUP BY is used so that it matches the ONLY_FULL_GROUP_BY mode. - (forthcoming) - subquery tests where subqueries that return a currency value are used in various locations and expressions within the query where a currency value would generally be expected; In addition, I have reviewed the actual queries from the benchmark and I think the RQG grammars more or less cover the scenarios described in the specification. >From the test runs that have been performed so far, no new bugs are being discovered as compared to the existing RQG grammars that use purely random queries against purely random data. Philip Stoev

1 0

[Maria-developers] WL#170 New (by Knielsen): Promotion of new master in " full independence" parallel replication.
by worklog-noreply＠askmonty.org 13 Dec '10

13 Dec '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Promotion of new master in "full independence" parallel replication. CREATION DATE..: Mon, 13 Dec 2010, 13:55 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 170 (http://askmonty.org/worklog/?tid=170) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: MWL#169: "Parallel replication of transactions in distinct databases" allows replication events and transactions to be applied on the slave in parallel, and potentially out-of-order (in "full independence"). This makes it more complex to promote a new master in case the old master dies or needs to be taken offline. The issue is that the states of any two slaves may not be commensurable, meaning that either slave may have a transaction applied that is missing on the other. This is more complicated that with old single-threaded replication, where there is a single ordered replication stream; then between any two slaves one will be ahead (or equal) to the other, and the one ahead can serve immediately as a master for the other. So with single-threaded replication, the problem is merely to find a slave that is maximally progressed. This worklog proposes an algorithm that allows to keep track of the states on slaves and a way to use this information to promote a new master reliably and potentially automated. This assumes that some kind of *global transaction ID* is implemented, along the lines of the Google patch for this [1]. A global transaction ID uniquely identifies the event(s) of a transaction (or event group in the case of non-transactional statements) across the entire replication topology. Global transaction IDs are monotonic, so E1 occurs before E2 if and only if the ID of E1 is less that the ID of E2. Note that we do not require all IDs to be comparable; in the case of two independent masters, events from different masters will be incomparable. (A common implementation is to use (server_id, incrementing_64bit_int) as the ID). Also note that we do not require ID sequences to be without holes, just that they are increasing. So it is permissible that IDs (server1,10) and (server1,12) exist, yet that there never existed any event with ID (server1,11). [1] http://code.google.com/p/google-mysql-tools/wiki/GlobalTransactionIds ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#169 New (by Knielsen): Parallel replication of transactions in distinct databases
by worklog-noreply＠askmonty.org 13 Dec '10

13 Dec '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Parallel replication of transactions in distinct databases CREATION DATE..: Mon, 13 Dec 2010, 13:54 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 169 (http://askmonty.org/worklog/?tid=169) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: The main challenge in parallel replication is to determine which events are safe and valid to run in parallel on the slave. This worklog is about parallel replication of transactions that each access only tables in a single database. Two such transactions can be safely applied in parallel on a slave if the two databases they each access are distinct. The term "database" is used here in the usual MySQL sense which is similar to "schema" in other RDBMS. There will be a server option "--fail-cross-database" to disallow multi-database statements and cross-database transactions. Ie. cause such queries to fail on the master so that the assumption remains valid on the slave that transactions do not span databases. This option defaults to FALSE. There will be a fixed number of apply threads on the slave, configured by the user with the option "--slave-apply-threads". This will default to 1, which will give the old non-parallel slave apply behaviour. Setting this to >1 will enable parallel slave apply. For this first version of parallel slave apply, it should probably be an error to set --slave-apply-threads >1 without also setting --fail-across-database to TRUE. If other options for discovering opportunities for parallel apply are implemented later, this restriction can be lifted. For this worklog, it is permissible, but not required, to use a simple scheduling of parallel events to apply threads: take a numeric hash of the database name and queue it for execution by thread (HASH mod N). This might be simpler to implement than a more clever scheduling, though it might reduce the opportunity for parallelism in case of hash collisions between distinct database names. When two transactions are applied in parallel on the slave, there is the question whether they are allowed to commit in different order than they did on the master. If commit in different order is allowed, this is called "full independence". If not, this is called "binlog order". With "binlog order", it is necessary to implement some mechanism to delay the commit of one thread until other threads have applied and committed all prior transactions. This waiting also has the potential to reduce the opportunities for parallelism. For "full independence", as a consequence of committing in different order, there will be database states seen on the slave that never existed on the master; the application will have to be tolerant of this. Also, the binlogs on the slaves will have events in different order than on the master; this complicates the promotion of a new master, as different servers will in general have incommensurable states, neither of which is a subset of the other. It still needs to be determined which of "binlog order" and "full independence" should be implemented. Probably a good idea is to start with just "full independence" due to its simplicity. Then "binlog order" and/or automated master promotion can be added as follow-up projects later. Related worklogs: MWL#XXX: Promotion of new master with full independence parallel replication. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Implementing new "group commit" API in PBXT?
by Kristian Nielsen 13 Dec '10

13 Dec '10

Hi Paul! I want to ask your opinion about implementing in PBXT an extension to the storage engine API that I am working on. There are lots of details in http://askmonty.org/worklog/Server-BackLog/?tid=116 (and even more details in other places), but I thought you would appreciate the short version :-) The idea is to get a well-defined ordering of commits in the server in an efficient way (eg. not break group commit like InnoDB does currently, Bug#13669). For this, two new (optional) storage engine methods are introduced: void (*prepare_ordered)(handlerton *hton, THD *thd, bool all); void (*commit_ordered)(handlerton *hton, THD *thd, bool all); The prepare_ordered() method is called after prepare(), as soon as commit order is decided. The commit_ordered() method is called before commit(), just after the transaction coordinator has made the final decision that the transaction will be durably committed (and not rolled back). The calls into commit_ordered() among different transactions will happen in the order that these transactions are committed, consistently across all engines and the binary log. Same for prepare_ordered(). The idea is that the storage engine should do the minimal amount of work in commit_ordered() necessary to make the commit visible to other threads. And to make sure commits appear to be done in the order of calls to these methods. Do you think either (or both) of these methods could be implemented in PBXT with reasonable effort (and if so, how)? ---- In InnoDB, this was trivial to do, as the InnoDB commit() method already had a "fast" part (which fixed the transaction log order (= "commit order") and made the transaction visible) and a "slow" part (which did the fsync() to make the transaction durable, and handled group commit). (It is necessary that commit_ordered() is fast, as it runs under a global lock. Ideally, it will just allocate an LSN in the transaction log to fix commit order, and perhaps whatever else already needs to happen serialised during engine commit). I hope my explanation was sufficiently clear for you to make a qualified answer. Maybe you can point me to where in the PBXT code a commit becomes visible and the commit order is fixed? In case you were wondering, here are some of the motivations for this feature: 1. For taking a hot backup, it is useful to have consistent commit order between binlog and storage engines. Without it, it can happen that the backed up state of the server has transaction A (but not B) committed in the storage engine, and transaction B (but not A) written to the binlog. Using such backup to provision a new master or slave would leave replication in an inconsistent state. 2. This feature implements working group commit for the binlog while still preserving consistent order as per (1). 3. This will allow to implement START TRANSACTION WITH CONSISTENT SNAPSHOT for multi-engine transactions that is truly consistent (currently is is possible for a transaction to be visible in one engine but not another in such "consistent" snapshot). 4. Galera relies on a consistent commit order, and I believe this feature will allow it to get this in a more engine-independent way. 5. We are planning to use consistent commit order to allow MySQL to recover after a crash transactions that were synced to disk in the binlog but not in the engine. This will allow to reduce the number of fsyncs() during prepare() / commit() from 3 to 1; it only needs to be done in the binlog (with group commit); the engine does not need to fsync(), as any lost transactions will be recovered from the binlog after crash. 6. The prepare_ordered() method is inspired by the Facebook patch to release InnoDB row-level read locks early (before syncing the binlog to disk) to improve performance in the presence of hot spots (probably does not apply to PBXT). - Kristian.

2 9

Re: [Maria-developers] Philip please test maria-5.3-mwl128-dsmrr-cpk
by Sergey Petrunya 13 Dec '10

13 Dec '10

On Thu, Dec 09, 2010 at 11:34:05AM +0200, Philip Stoev wrote: > I have a few questions with respect to the feature so that I can test it > better (we may have discussed those previously but I can not seem to find > the email): > > - apart from mrr_sort_keys ON and OFF and engine_condition_pushdown = ON > | OFF and all join_cache_levels , is there any other optimizer switch > that I should fiddle with? Is there any switch that I can set to ON or > OFF in order to drive more queries to use your code? @@engine_condition_pushdown variable does not have any effect with the current code. It controls table condition pushdown, which is only supported by NDB. XtraDB/MyISAM/Maria have *index* condition pushdown, which can be turned on and off by using SET @@optimizer_switch='index_condition_pushdown=on|off'; Overall, DS-MRR behavior can be modified by: - @@join_cache_level setting (need to be such that BKA is enabled in order for new code to wor) - @@join_buffer_size (sets size of the buffer that will be shared between BKA module and DS-MRR module) - mrr_sort_keys=on|off @@optimizer_switch flag - index_condition_pushdown=on|off @@optimizer_switch flag. > - apart from mrr_buffer_size , is there any other mysqld option I should > be conserned with. What values of mrr_buffer_size should be tested? > - what table sizes should I use and also how many tables should I join? > Is up to 1000 rows per table and 2-3-table joins sufficient? Or should I > go for the DBT-2 data set? I think that we should first test with up to 1K rows per table and 2-4 way joins. I was quite impressed by the issue you've found in BUG#671340 (it was a specific interaction of key sorting and rowid sorting, which could occur only when we're scanning a particular kind of data and exhaust the buffer at a certain point). The testcase for that bug was rather small, I take the original testcase found by RQG wasn't big either? > - how about multipart and prefix indexes? Yes. I don't expect bugs to be there, but the code does work with index tuples, so it would be nice to check how scanning on multipart indexes (and their prefixes) work. Prefix indexes can be checked, too. > - should I test enum/set/bit and other marginal data types? I can't think of realistic examples with those types, so we better focus on something else. > - are there any engine specific code or considerations? XtraDB is different in the regard that it has clustered primary keys, and MWL#121-125 adds support for MRR/BKA to work for clustered primary keys. Maria/MyISAM should be checked too, they both have very similar DS-MRR handling. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 2

[Maria-developers] DS-MRR improvements patch R5 ready for review.
by Sergey Petrunya 10 Dec '10

10 Dec '10

Hi Monty, I've finished addressing the review feedback and I'm submitting the patches for another review. The attached are: - cumulative all-included diff againist 5.3-main (should just apply) - diff with post-review changes ( I needed to tweak that file a bit becasue I've done a merge from -mwl128 tree, so I needed to remove those changes. The tree is at lp:~maria-captains/maria/maria-5.3-mwl128-dsmrr-cpk. I'm not aware of any buildbot failures except the solaris32 crashes. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Volunteer my machine for buildbot and database with test results
by Haidong Ji 10 Dec '10

10 Dec '10

To whom it may concern, Per information shown on URL below, with relevant paragraph pasted, I'd love to volunteer to run a build slave and other tests. Let me know how to participate, such as: Will an old laptop work? If so, what do you need? What do I need to do on my side? Does it mean that I need to leave my machine on all the time? What if a reboot happens while it is being used? I am sure more questions as we proceed. Thanks, Haidong "Alex" Ji http://kb.askmonty.org/v/about-buildbot Volunteering to run a build slave Many of our build hosts are run by community members, and we are always looking for additional volunteers to help us cover additional platforms or build options in BuildBot. If you are able to provide a spare machine for this purpose, your help is greatly appreciated! This is a good way to get involved without having to spend a lot of time on it. Get started by writing an email to maria-developers(a)lists.launchpad.net with an offer to run a BuildBot slave. Database with test results Buildbot saves the results of test runs in a database, to be used for enhanced reporting on web pages without need to change the Buildbot code, and for data mining when investigating test failures. The database schema is documented under Buildbot Database Schema. The schema is likely to evolve as we gradually add more kinds of information. For now, the data is not externally available. But the plan is to set up a slave database to replicate the data, and provide access (eg. remote database accounts) to members of the community with interesting ideas about how to present or mine this data, or who are just curious to play with it. If anyone has an interest in this, or wants to volunteer a slave host for this purpose, please send a mail to maria-developers(a)lists.launchpad.net. The more people show interest in this, the faster it is likely to happen!

2 5

[Maria-developers] problem with partitioning and our storage engine in 5.2
by Zardosht Kasheff 10 Dec '10

10 Dec '10

Hello all, We have been working on testing our storage engine, TokuDB, against MariaDB 5.2.3, and we have encountered a problem with partitioning that does not exist on MySQL 5.1, MySQL 5.5, and MariaDB 5.1.50. This problem also does not exist with any other storage engine that we have tried. It ONLY exists with TokuDB and MariaDB 5.2. On MariaDB 5.2, the attached test produces the attached result file, that has incorrect results. Here is what the test is doing: 1. Create a table with some partitions 2. update the table 3. run select max(f_int1) 4. run select * 5. run select max(f_int1) again. The problem is that the query results for 3 are incorrect, even though the query results for 4 and 5 are correct. MySQL 5.1, MySQL 5.5, and MariaDB 5.1 produce the correct results for 3. What makes this even stranger is that the query results for 4 and 5 are always correct. Any ideas on how we can go about figuring out what is wrong? Thanks -Zardosht

6 11

[Maria-developers] MWL#90: need a list of "obvious things to improve"
by Sergey Petrunya 09 Dec '10

09 Dec '10

Hi Timour, In Helsinki, we've noted the following: > MWL#90: Subqueries: Inside-out execution for non-semijoin materialized subqueries in WHERE (4K diff); Critical task > = Get the code reviewed/pushed > TODO: > - Timour will tell the "obvious things to improve"; Timour 1 day I'm going to need those "obvious things" fairly soon. Do you need to have anything as a prerequisite to produce the list? BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Philip please test maria-5.3-mwl128-dsmrr-cpk
by Sergey Petrunya 09 Dec '10

09 Dec '10

Hi Philip, The new version of MWL#121-125, DS-MRR impovements, is ready for your testing. I believe all previously reported crashes and wrong query result bugs are no longer there (they've been either targeted and fixed, or were gone after code re-working I've made when addressing review feedback). The slowdown bugs (https://bugs.launchpad.net/maria/+bug/637160, https://bugs.launchpad.net/maria/+bug/631504) are most likely still there (I didn't check if they are, but I haven't made any changes that could impact them). The code to be tested is at lp:~maria-captains/maria/maria-5.3-mwl128-dsmrr-cpk. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] sql-bench results for MariaDB string optimization
by Hakan Kuecuekyilmaz 09 Dec '10

09 Dec '10

Hi Monty, here are the results comparing your string optimization patch with MariaDB 5.2 and sql-bench. I added the configuration used, a short summary, and a full summary of the results. I ran the test on our new machine "pitbull", a two socket hexa-core Intel Xeon X5660 @ 2.80GHz with 23GB RAM with a single S-ATA disk. Todo: the tests show a couple of results with 0 - 1 wallclock seconds, which is due the quite faster machine. I will adjust the tests to run for at least 5 seconds (again). I attached the result file for the that the results are not good readable in this email. Best, Hakan hakan@pitbull:~/work/monty_program/mariadb-tools/sql-bench$ more conf/sql-bench-only_myisam.cnf # Base configuration for running sql-bench tests for MariaDB > 5.2.2. # # This is the minimal base test version # # Hakan Kuecuekyilmaz, <hakan at askmonty dot org>, 2010-11-18. # Configure environment variables. # Note: ccache is checked in the host specific config file. $ENV{'CC'} = "gcc"; $ENV{'CFLAGS'} = "-Wimplicit -Wreturn-type -Wswitch -Wtrigraphs -Wcomment -W -Wchar-subscripts -Wformat -Wparenthese s -Wsign-compare -Wwrite-strings -Wunused-function -Wunused-label -Wunused-value -Wunused-variable -Wunused-paramete r -O3 -fno-omit-frame-pointer"; $ENV{'CXX'} = "gcc"; $ENV{'CXXFLAGS'} = "-Wimplicit -Wreturn-type -Wswitch -Wtrigraphs -Wcomment -W -Wchar-subscripts -Wformat -Wparenthe ses -Wsign-compare -Wwrite-strings -Wunused-function -Wunused-label -Wunused-value -Wunused-variable -Wreorder -Wcto r-dtor-privacy -Wnon-virtual-dtor -felide-constructors -fno-exceptions -fno-rtti -O3 -fno-omit-frame-pointer"; $ENV{'CXXLDFLAGS'} = ""; $config = { # Configure invocation line. # Do not use --prefix here. configure_line => '--enable-assembler --with-extra-charsets=complex --enable-thread-safe-client --with-big-tables --with-plugin-aria --with-aria-tmp-tables --without-plugin-innodb_plugin --with-mysqld-ldflags=-static --with-client -ldflags=-static --with-readline --with-ssl --with-plugins=max-no-ndb --with-embedded-server --with-libevent --enabl e-local-infile', }; # # Different test to be run with the above configuration. # The sql-bench-test will automaticly get --suffix=_test_name as an argument # The different options that can be used are: # # mysqld_start_options Options given to the mysqld that is started. # This is prefixed with --skip-grant. # mysqld_init_command SQL commands given to mysqld before tests starts. # sql_bench_options Options given to sql-bench. # $sql_bench_test->{'base'} = { mysqld_start_options => '', mysqld_init_command => '', sql_bench_options => '--comment="base test (with MyISAM)"', }; # This has to be the last line. See "perldoc -f require" for details. 1; RESULTS: Comparing 5.2 plain against patched 5.2 (three runs opt1/opt2/opt3) Overall the string optimization patch shows better results. Highlights are * Biggest gain we see in "connect": ~ 4 - 15% * Biggest loss we see in "insert": ~ 1 - 3% Biggest loss in detail: connect (50000) | 41.00| 1.024| 1.024| 1.000| connect+select_1_row (50000) | 48.00| 1.042| 1.042| 1.021| connect+select_simple (50000) | 45.00| 1.022| 1.022| 1.022| count_distinct_big (720) | 19.00| 1.105| 1.000| 1.053| count_distinct_group_on_key (7000) | 29.00| 1.034| 1.000| 1.034| count_group_on_key_parts (7000) | 27.00| 1.074| 1.000| 1.037| create_MANY_tables (400) | 17.00| 1.059| 1.059| 1.059| delete_all_many_keys (1) | 5.00| 1.200| 1.200| 1.000| delete_big_many_keys (128) | 5.00| 1.200| 1.200| 1.000| insert (350768) | 39.00| 1.103| 1.077| 1.051| insert_duplicates (100000) | 9.00| 1.000| 1.444| 1.333| insert_duplicates (100000) | 9.00| 1.000| 1.444| 1.333| min_max (60) | 1.00| 2.000| 1.000| 2.000| once_prepared_select (100000) | 13.00| 1.000| 1.308| 1.231| order_by_big (10) | 5.00| 1.000| 1.200| 1.200| order_by_big_key_desc (10) | 5.00| 1.200| 1.000| 1.200| outer_join_not_found (500) | 4.00| 1.250| 1.000| 1.250| select_big (80) | 5.00| 1.200| 1.000| 1.200| select_big_str (50000) | 14.00| 1.071| 1.071| 1.071| update_of_key (50000) | 6.00| 1.167| 1.333| 1.333| update_with_key (300000) | 30.00| 1.133| 1.333| 1.333| update_with_key_prefix (100000) | 10.00| 1.100| 1.400| 1.400| Full results: hakan@pitbull:~/sql-bench-results/pitbull$ /tmp/tmp.jAANb0DlLs/install/sql-bench/compare-results 2010-12-08-5.2-base/base/RUN-mysql-Linux_2.6.35_23_server_x86_64 2010-12-08-5.2-string-opt1/base/RUN-mysql-Linux_2.6.35_23_server_x86_64 2010-12-08-5.2-string-opt2/base/RUN-mysql-Linux_2.6.35_23_server_x86_64 2010-12-09-5.2-string-opt3/base/RUN-mysql-Linux_2.6.35_23_server_x86_64 --relative This is the result file of the different benchmark tests. The number in () after each tests shows how many SQL commands the particular test did. As one test may have many different parameters this gives only a rough picture of what was done. Check the source for more information :) Keep in mind that one can't compare benchmarks run with different --cmp options. The --cmp options sets the all limits according to the worst limit for all server in the benchmark. Numbers marked with '+' are estimated according to previous runs because the query took longer than a given time-limit to finish. The estimation shouldn't be far from the real result thought. Numbers marked with '?' contains results that gave wrong result. This can only be used as an indication of how long it took for the server to produce a wrong result :) Numbers marked with '*' are tests that was run a different number times than the test in the first column. The reason for this is normally that the marked test was run with different options that affects the number of tests or that the test was from another version of the MySQL benchmarks. Hope this will give you some idea how each db is performing at what thing .... Hope you like it .... Luuk & Monty (1997) Column 1 is in seconds. All other columns are presented relative to this. 1.00 is the same, bigger numbers indicates slower The result logs which where found and the options: 1 mysql-Linux_2.6.35_23_server_x86_64 : MySQL 5.2.4 MariaDB --socket=/tmp/tmp.bn05ikyeM5/mysql.sock base test (with MyISAM) 2 mysql-Linux_2.6.35_23_server_x86_64 : MySQL 5.2.4 MariaDB --socket=/tmp/tmp.eNBvB5BqYb/mysql.sock base test (with MyISAM) 3 mysql-Linux_2.6.35_23_server_x86_64 : MySQL 5.2.4 MariaDB --socket=/tmp/tmp.AG0msBbvC2/mysql.sock base test (with MyISAM) 4 mysql-Linux_2.6.35_23_server_x86_64 : MySQL 5.2.4 MariaDB --socket=/tmp/tmp.jAANb0DlLs/mysql.sock base test (with MyISAM) ================================================================================= Operation | 1| 2| 3| 4| |mysql-Linu|mysql-Linu|mysql-Linu|mysql-Linu| --------------------------------------------------------------------------------- Relative results per test (First column is in seconds): | --------------------------------------------------------------------------------- ATIS | 407.00| 0.980| 0.983| 0.978| alter-table | 112.00| 0.991| 0.955| 0.991| big-tables | 425.00| 0.965| 0.953| 0.958| connect | 403.00| 0.893| 0.963| 0.846| create | 918.00| 0.997| 1.001| 0.971| insert | 536.00| 1.007| 1.034| 1.011| select | 417.00| 0.959| 0.868| 0.986| wisconsin | 408.00| 1.007| 0.983| 0.985| --------------------------------------------------------------------------------- The results per operation: | --------------------------------------------------------------------------------- alter_table_add (992) | 73.00| 0.973| 0.945| 0.973| alter_table_drop (496) | 38.00| 1.026| 0.947| 1.000| connect (50000) | 41.00| 1.024| 1.024| 1.000| connect+select_1_row (50000) | 48.00| 1.042| 1.042| 1.021| connect+select_simple (50000) | 45.00| 1.022| 1.022| 1.022| count (100) | 3.00| 0.667| 0.667| 0.667| count_distinct (7000) | 22.00| 1.000| 1.000| 1.000| count_distinct_2 (7000) | 38.00| 1.000| 1.000| 1.000| count_distinct_big (720) | 19.00| 1.105| 1.000| 1.053| count_distinct_group (7000) | 36.00| 0.667| 0.667| 0.972| count_distinct_group_on_key (7000) | 29.00| 1.034| 1.000| 1.034| count_distinct_group_on_key_parts (7| 33.00| 1.000| 0.697| 0.818| count_distinct_key_prefix (7000) | 15.00| 1.000| 0.933| 1.000| count_group (7000) | 33.00| 1.000| 0.606| 1.030| count_group_on_key_parts (7000) | 27.00| 1.074| 1.000| 1.037| count_group_with_order (7000) | 34.00| 1.000| 0.618| 1.029| count_on_key (50100) | 66.00| 0.848| 0.955| 0.955| create+drop (10000) | 450.00| 1.002| 1.007| 0.973| create_MANY_tables (400) | 17.00| 1.059| 1.059| 1.059| create_index (8) | 0.00| 1.000| 0.000| 1.000| create_key+drop (10000) | 450.00| 0.991| 0.996| 0.967| create_table (31) | 2.00| 0.500| 0.500| 0.500| delete_all_many_keys (1) | 5.00| 1.200| 1.200| 1.000| delete_big (1) | 0.00| 0.000| 0.000| 0.000| delete_big_many_keys (128) | 5.00| 1.200| 1.200| 1.000| delete_key (10000) | 2.00| 0.500| 0.500| 0.500| delete_range (12) | 1.00| 2.000| 1.000| 1.000| drop_index (8) | 1.00| 0.000| 1.000| 1.000| drop_table (28) | 0.00| 0.000| 0.000| 0.000| drop_table_when_MANY_tables (400) | 1.00| 0.000| 0.000| 0.000| insert (350768) | 39.00| 1.103| 1.077| 1.051| insert_duplicates (100000) | 9.00| 1.000| 1.444| 1.333| insert_key (100000) | 25.00| 0.960| 1.040| 0.960| insert_many_fields (140000) | 112.00| 0.982| 0.982| 0.982| insert_select_1_key (1) | 1.00| 1.000| 1.000| 1.000| insert_select_2_keys (1) | 0.00| 0.000| 1.000| 1.000| min_max (60) | 1.00| 2.000| 1.000| 2.000| min_max_on_key (85000) | 16.00| 1.000| 1.000| 1.000| multiple_value_insert (100000) | 1.00| 0.000| 1.000| 1.000| once_prepared_select (100000) | 13.00| 1.000| 1.308| 1.231| order_by_big (10) | 5.00| 1.000| 1.200| 1.200| order_by_big_key (10) | 6.00| 0.833| 1.000| 0.833| order_by_big_key2 (10) | 5.00| 1.000| 1.000| 1.200| order_by_big_key_desc (10) | 5.00| 1.200| 1.000| 1.200| order_by_big_key_diff (10) | 6.00| 1.000| 0.833| 0.833| order_by_big_key_prefix (10) | 5.00| 1.000| 1.200| 1.000| order_by_key2_diff (500) | 1.00| 1.000| 1.000| 1.000| order_by_key_prefix (500) | 1.00| 1.000| 1.000| 1.000| order_by_range (500) | 1.00| 1.000| 0.000| 0.000| outer_join (10) | 5.00| 1.000| 1.200| 1.000| outer_join_found (10) | 5.00| 1.000| 1.000| 1.000| outer_join_not_found (500) | 4.00| 1.250| 1.000| 1.250| outer_join_on_key (10) | 5.00| 1.000| 0.800| 1.000| prepared_select (100000) | 22.00| 0.955| 0.955| 0.955| select_1_row (500000) | 52.00| 0.750| 0.788| 0.673| select_1_row_cache (500000) | 50.00| 0.660| 0.700| 0.660| select_2_rows (500000) | 57.00| 0.772| 0.947| 0.684| select_big (80) | 5.00| 1.200| 1.000| 1.200| select_big_str (50000) | 14.00| 1.071| 1.071| 1.071| select_cache (10000) | 18.00| 1.000| 0.944| 0.944| select_cache2 (10000) | 18.00| 0.944| 0.944| 0.944| select_column+column (500000) | 53.00| 0.925| 0.830| 0.792| select_diff_key (500) | 0.00| 0.000| 0.000| 0.000| select_distinct (40000) | 113.00| 1.018| 1.009| 1.009| select_group (140111) | 121.00| 1.008| 1.041| 1.000| select_group_when_MANY_tables (400) | 0.00| 0.000| 0.000| 0.000| select_join (5000) | 22.00| 1.000| 1.000| 1.000| select_key (200000) | 43.00| 1.000| 0.977| 0.953| select_key2 (200000) | 46.00| 0.978| 0.957| 0.957| select_key2_return_key (200000) | 44.00| 1.000| 0.977| 0.955| select_key2_return_prim (200000) | 46.00| 0.978| 0.935| 0.935| select_key_prefix (200000) | 46.00| 1.022| 0.957| 0.957| select_key_prefix_join (5000) | 131.00| 0.908| 0.893| 0.916| select_key_return_key (200000) | 43.00| 0.953| 0.953| 0.953| select_many_fields (140000) | 313.00| 0.958| 0.942| 0.949| select_range (410) | 22.00| 0.955| 0.955| 1.000| select_range_key2 (25010) | 5.00| 1.000| 1.200| 1.000| select_range_prefix (25010) | 6.00| 1.000| 1.000| 1.000| select_simple (500000) | 22.00| 1.000| 1.318| 0.955| select_simple_cache (500000) | 21.00| 0.952| 1.524| 0.952| select_simple_join (25000) | 22.00| 1.045| 1.045| 1.000| update_big (10) | 4.00| 1.000| 1.000| 1.000| update_of_key (50000) | 6.00| 1.167| 1.333| 1.333| update_of_key_big (501) | 3.00| 1.000| 1.000| 0.667| update_of_primary_key_many_keys (256| 3.00| 0.667| 1.000| 0.667| update_with_key (300000) | 30.00| 1.133| 1.333| 1.333| update_with_key_prefix (100000) | 10.00| 1.100| 1.400| 1.400| wisc_benchmark (50014) | 403.00| 1.010| 0.988| 0.988| --------------------------------------------------------------------------------- TOTALS | 3619.00| 0.978| 0.975| 0.966| ================================================================================= -- Hakan Küçükyılmaz, QA/Benchmark Engineer, Stuttgart/Germany Monty Program Ab, http://askmonty.org/ Skype: hakank_ Phone: +49 171 1919839

1 0

[Maria-developers] bringing mysql-5.5 cmake goodies into MariaDB?
by Vladislav Vaintroub 08 Dec '10

08 Dec '10

Hi, I spend about some time (took about a day) trying to improve the current state of windows build and packaging. Partly, I it was to improve my own experience working on Windows (spoiled by MySQL5.5 :) with out-of-source build, partly because I think the fact that the packaging (even zip) requires specially prepared source on Unix or Cygwin (known as bootstrapping in the old MySQL, gone in MySQL-5.5) , is not the way to go forward. It is hard to explain it to our Windows users that Cygwin might is required or recommended, die-hard windows people try to avoid it, and it is known to be broken in recent Windows (Vista +) too... So, the proposed patch would eliminate every Cygwin dependency. Those cmake goodies are not the full 5.5 cmake stuff (nothing Unix related, no improvements for plugins, no improvements to avoid useless recompilation in embedded or in libmysql etc.), but mainly Windows-related packaging plus out-of-source build and test. I have created a repo with my changes on Launchpad https://code.launchpad.net/~maria-captains/maria/5.1-windows-packaging to configure the build (for releases) one would execute % cmake . -DWITH_EMBEDDED_SERVER=1 (all engines that should be compiled in are set to compile in by default, I do not use embedded by default since the build is taking too long with it) To build the ZIP package % devenv mysql.sln /build relwithdebinfo /project package This creates ./mariadb-<major>.<minor>.<patch>-win32.zip To build MSI (assuming Wix 3.0 is installed from http://wix.sourceforge.net/downloadv3.html ) % devenv mysql.sln /build relwithdebinfo % devenv mysql.sln /build relwithdebinfo /project MSI the resulting msi is in win\cmake\ mariadb-<major>.<minor>.<patch>-win32.msi This procedure resembles very closely the official MySQL5.5 release-type builds on Windows ( Jonathan blogged about it some time ago http://www.perkin.org.uk/blog/2010/11/how-to-build-mysql-releases/ ) NSIS is probably broken. Frankly speaking I do not know it at all, neither I believe it would be a good idea to learn, as it will disappear in 5.5 at least. I appreciate comments on the idea. Especially from Philip as QA, and from Kristian as build automator ( should we decide to use this patch, some things will have to change on Buildbot). On the patch itself: - Preventively (to avoid 5.5 merge errors), I created new files in different places from where they live in MySQL 5.5 (e.g. not in cmake/* but in win/cmake/* , not packaging/Wix/* but win/packaging/*). I have done it as bazaar cannot handle situations, where files are "retrospectively" added in old releases. - I also had to fix Maria's mtr for out-of-source build. - When merging with 5.5 most of this stuff (except MTR changes) should disappear, it is already there I can create worklog about this ,and do other procedurally required stuff. I'm not sure in which MariaDB version it should be applied , if we decide to apply it at all. Thanks, Vladislav

1 0