- developers - lists.mariadb.org

Re: [Maria-developers] [Commits] Rev 2947: Fix LP BUG#613029 in file:///home/tsk/mprog/src/5.3/
by Sergey Petrunya 29 Mar '11

29 Mar '11

Hi Timour, Ok to push. On Mon, Mar 28, 2011 at 12:55:43PM +0300, timour(a)askmonty.org wrote: > At file:///home/tsk/mprog/src/5.3/ > > ------------------------------------------------------------ > revno: 2947 > revision-id: timour(a)askmonty.org-20110328095536-wbmu1hiwsnhw6bs8 > parent: timour(a)askmonty.org-20110324143406-04q5peh1r7nthcyb > committer: timour(a)askmonty.org > branch nick: 5.3 > timestamp: Mon 2011-03-28 12:55:36 +0300 > message: > Fix LP BUG#613029 > > Analysis: > There are two code paths through which JOIN::exec may produce > an all-NULL row for an empty result set. One goes via the > function return_zero_rows(), when query processing detectes > early that the where clause is false, the other one is via > do_select() in the case of join execution. > > In the case of do_select(), the problem was that the executioner > didn't set TABLE::null_row to 1. As result when sending the only > result row, the evaluation of each field didn't detect that all > non-aggregated fields are NULL, because Field::is_null returned > true, after checking that field->table->null_row was false. > > Given that the each non-aggregated field was not considered NULL, > select_result::send_data sent whatever was in the buffer of each > field. However, since there was no actual data in the field buffer, > send_data() accessed and sent whatever junk was in the field's > data buffer. > > Solution: > Similar to the analogous case in return_zero_rows() mark all > tables that their current row is NULL before sending the > artificailly created NULL row. > === modified file 'mysql-test/r/subselect4.result' > --- a/mysql-test/r/subselect4.result 2011-03-24 14:34:06 +0000 > +++ b/mysql-test/r/subselect4.result 2011-03-28 09:55:36 +0000 > @@ -1133,3 +1133,55 @@ SELECT * FROM t1 WHERE (2, 0) NOT IN (SE > f1 f2 > set @@optimizer_switch=@save_optimizer_switch; > drop table t1,t2; > +# > +# LP BUG#613029 Wrong result with materialization and semijoin, and > +# valgrind warnings in Protocol::net_store_data with materialization > +# for implicit grouping > +# > +CREATE TABLE t1 ( > +pk int(11) NOT NULL AUTO_INCREMENT, > +f2 int(11) NOT NULL, > +f3 varchar(1) NOT NULL, > +PRIMARY KEY (pk), > +KEY f2 (f2)); > +INSERT INTO t1 VALUES (1,9,'x'); > +INSERT INTO t1 VALUES (2,5,'g'); > +CREATE TABLE t2 ( > +pk int(11) NOT NULL AUTO_INCREMENT, > +f2 int(11) NOT NULL, > +f3 varchar(1) NOT NULL, > +PRIMARY KEY (pk), > +KEY f2 (f2)); > +INSERT INTO t2 VALUES (1,7,'p'); > +set @save_optimizer_switch=@@optimizer_switch; > +set @@optimizer_switch='materialization=off,semijoin=off'; > +EXPLAIN > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables > +2 DEPENDENT SUBQUERY t1 index_subquery f2 f2 4 func 2 Using index > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > +f3 MAX(t1.f2) > +NULL NULL > +set @@optimizer_switch='materialization=on,semijoin=off'; > +EXPLAIN > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY t2 system PRIMARY NULL NULL NULL 1 > +1 PRIMARY t1 const PRIMARY PRIMARY 4 const 1 > +2 SUBQUERY t1 index NULL f2 4 NULL 2 Using index > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > +f3 MAX(t1.f2) > +NULL NULL > +TODO: add a test case for semijoin when the wrong result is fixed > +set @@optimizer_switch='materialization=off,semijoin=on'; > +set @@optimizer_switch=@save_optimizer_switch; > +drop table t1, t2; > > === modified file 'mysql-test/t/subselect4.test' > --- a/mysql-test/t/subselect4.test 2011-03-24 14:34:06 +0000 > +++ b/mysql-test/t/subselect4.test 2011-03-28 09:55:36 +0000 > @@ -906,3 +906,60 @@ SELECT * FROM t1 WHERE (2, 0) NOT IN (SE > set @@optimizer_switch=@save_optimizer_switch; > > drop table t1,t2; > + > +--echo # > +--echo # LP BUG#613029 Wrong result with materialization and semijoin, and > +--echo # valgrind warnings in Protocol::net_store_data with materialization > +--echo # for implicit grouping > +--echo # > + > +CREATE TABLE t1 ( > + pk int(11) NOT NULL AUTO_INCREMENT, > + f2 int(11) NOT NULL, > + f3 varchar(1) NOT NULL, > + PRIMARY KEY (pk), > + KEY f2 (f2)); > + > +INSERT INTO t1 VALUES (1,9,'x'); > +INSERT INTO t1 VALUES (2,5,'g'); > + > +CREATE TABLE t2 ( > + pk int(11) NOT NULL AUTO_INCREMENT, > + f2 int(11) NOT NULL, > + f3 varchar(1) NOT NULL, > + PRIMARY KEY (pk), > + KEY f2 (f2)); > + > +INSERT INTO t2 VALUES (1,7,'p'); > + > +set @save_optimizer_switch=@@optimizer_switch; > + > +set @@optimizer_switch='materialization=off,semijoin=off'; > + > +EXPLAIN > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > + > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > + > +set @@optimizer_switch='materialization=on,semijoin=off'; > + > +EXPLAIN > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > + > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > + > +-- echo TODO: add a test case for semijoin when the wrong result is fixed > +-- echo set @@optimizer_switch='materialization=off,semijoin=on'; > + > + > +set @@optimizer_switch=@save_optimizer_switch; > + > +drop table t1, t2; > > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2011-03-24 14:34:06 +0000 > +++ b/sql/sql_select.cc 2011-03-28 09:55:36 +0000 > @@ -13452,6 +13452,13 @@ do_select(JOIN *join,List<Item> *fields, > { > List<Item> *columns_list= (procedure ? &join->procedure_fields_list : > fields); > + /* > + With implicit grouping all fields of special row produced for an > + empty result are NULL. See return_zero_rows() for the same behavior. > + */ > + for (TABLE_LIST *table= join->select_lex->leaf_tables; > + table; table= table->next_leaf) > + mark_as_null_row(table->table); > rc= join->result->send_data(*columns_list); > } > } > > === modified file 'sql/sql_select.h' > --- a/sql/sql_select.h 2011-03-13 10:50:14 +0000 > +++ b/sql/sql_select.h 2011-03-28 09:55:36 +0000 > @@ -954,8 +954,8 @@ public: > bool init_save_join_tab(); > bool send_row_on_empty_set() > { > - return (do_send_rows && tmp_table_param.sum_func_count != 0 && > - !group_list && having_value != Item::COND_FALSE); > + return (do_send_rows && implicit_grouping && > + having_value != Item::COND_FALSE); > } > bool change_result(select_result *result); > bool is_top_level_join() const > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] WL#191 New (by Knielsen): status variables for last InnoDB binlog position
by worklog-noreply＠askmonty.org 29 Mar '11

29 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: status variables for last InnoDB binlog position CREATION DATE..: Tue, 29 Mar 2011, 09:06 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 191 (http://askmonty.org/worklog/?tid=191) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: In every commit, InnoDB/XtraDB stores the binlog position in the system tablespace (in a transactional-safe way). At server start, this position is printed to the error log. This is useful after taking a backup to provision a slave, as it provides the values to use for CHANGE MASTER TO. It would be useful if these values printed to the error log were also directly available to SQL as read-only status variables: innodb_startup_binlog_master_file innodb_startup_binlog_master_pos This would be generally useful. When this is implemented, the test case main.group_commit_binlog_pos can be fixed to not require `grep` and `tail`, and so work on Windows with no posix tools installed. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Improved table elimination
by Lars Rönnbäck 29 Mar '11

29 Mar '11

When doing some comparisons of query optimization in Microsoft SQL Server and MariaDB I noticed that there is a second type of table elimination which MariaDB has not yet implemented. It depends on foreign keys being declared, and can under some circumstances substantially improve performance. Rather than to write a lengthy discussion, here is a test script for MariaDB 5.2.5 in which the queries with the condition "where ... is not null" should result in this type of table elimination: http://pastebin.com/s68RwgG2 This screencast shows the table elimination in effect in MSSQL: http://www.screenr.com/tPG We rely on table elimination to get better performance in 3NF-views and functions used in our Anchor Modeling (6NF-table) technique. However, the need for table elimination has previously come from denormalized modeling, where it is used to speed up queries against star schemas (e g eliminating dimension tables). It should therefore not only be appreciated by our crowd, if implemented in full. We were very happy to see the 'outer join table elimination' implemented and are wondering if someone could put in the effort to add the 'foreign key table elimination'? Since everything we do with Anchor Modeling is Open Source, it would be nice to have an Open Source database that fully supports the technique. Support in other databases is described here, if anyone is interested: http://www.anchormodeling.com/?page_id=22 Thanks, Lars

1 0

[Maria-developers] WL#190 New (by Yakatz): Optional parameters support in stored procedures
by worklog-noreply＠askmonty.org 28 Mar '11

28 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Optional parameters support in stored procedures CREATION DATE..: Mon, 28 Mar 2011, 20:26 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 190 (http://askmonty.org/worklog/?tid=190) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: This bug has been open on MySQL since 2005. In a nutshell, support PROCEDUREs like this: CREATE PROCEDURE ListUsers ( @userID int, @authorized bit = NULL ) AS.... ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Review of replication/group commit work
by Kristian Nielsen 28 Mar '11

28 Mar '11

Hi Monty, Thanks for offering to review my replication worklogs. Here are pointers to the various worklogs and patches, and suggestions for order to things in etc. Some background information is in this series on my blog: http://kristiannielsen.livejournal.com/12254.html http://kristiannielsen.livejournal.com/12408.html http://kristiannielsen.livejournal.com/12553.html http://kristiannielsen.livejournal.com/12810.html There is a tree on launchpad based on 5.2 with all the patches merged, if you want to look at the actual code in one piece: lp:~maria-captains/maria/mariadb-5.2-rpl ---- The main worklog, and the place to start, is MWL#116: http://askmonty.org/worklog/Server-Sprint/?tid=116 I think you'll have to read through the worklog/design first (it already had architecture review by Serg) in order to properly review the code. The patch for MWL#116 is here: https://lists.launchpad.net/maria-developers/msg03693.html It also has links to a lauchpad tree with just this patch. This worklog extends the storage engine API with a facility to efficiently coordinate commit order among transaction participants, and also implements group commit for binlog. ---- The next worklog is MWL#136: http://askmonty.org/worklog/Server-RawIdeaBin/?tid=136 Here is the commit mail: https://lists.askmonty.org/pipermail/commits/2010-November/000735.html (also this one, but it just renames some variables to bettter names): https://lists.askmonty.org/pipermail/commits/2010-November/000741.html This worklog makes START TRANSACTION WITH CONSISTENT SNAPSHOT, well, consistent. It also uses this for the binlog to provide non-blocking mysqldump --single-transaction --master-data. ---- The last worklog is MWL#163: http://askmonty.org/worklog/Server-Sprint/?tid=163 Here is the commit mail: https://lists.askmonty.org/pipermail/commits/2010-November/000711.html This is a re-write of a Facebook patch. Mark has mentioned this on several occasions as being important for them to fix a hot-spot. It implements an option to release InnoDB row locks earlier, before committing to binlog and storage engine (which is not safe in general, but can be safe in certain applications such as Facebook presumably). ---- That's it for now! - Kristian.

2 1

[Maria-developers] review: [Commits] Rev 2863: MWL#163: release of row locks in InnoDB during prepare in http://bazaar.launchpad.net/~maria-captains/maria/5.1
by Michael Widenius 28 Mar '11

28 Mar '11

Hi! >>>>> "knielsen" == knielsen <knielsen(a)knielsen-hq.org> writes: knielsen> At http://bazaar.launchpad.net/~maria-captains/maria/5.1 knielsen> ------------------------------------------------------------ knielsen> revno: 2863 knielsen> revision-id: knielsen(a)knielsen-hq.org-20101103160708-88w8p8ibe9iaoggj knielsen> parent: knielsen(a)knielsen-hq.org-20101102074027-jlnx0rnbdpovl122 knielsen> committer: knielsen(a)knielsen-hq.org knielsen> branch nick: work-5.1-mwl163 knielsen> timestamp: Wed 2010-11-03 17:07:08 +0100 knielsen> message: knielsen> MWL#163: release of row locks in InnoDB during prepare knielsen> Port the Facebook patch for releasing InnoDB row locks early to the knielsen> MWL#116 framework. knielsen> === modified file 'storage/xtradb/handler/ha_innodb.cc' knielsen> --- a/storage/xtradb/handler/ha_innodb.cc 2010-11-01 14:41:09 +0000 <cut> Spent 30 minutes checking the innodb code and things looked ok. Regards, Monty

1 0

[Maria-developers] Writeup of how to run my group commit benchmark
by Kristian Nielsen 28 Mar '11

28 Mar '11

Hi, For anyone interested in trying to redo my group commit benchmarks on more interesting hardware than what I have available, here is a writeup of the steps needed to run it. The benchmarks are described here: http://kristiannielsen.livejournal.com/15739.html http://kristiannielsen.livejournal.com/15515.html 1. Compile gypsy. This is used to generate the load. I use a specific version as the latest in trunk seemed to not build on non-windows last I tried: bzr branch '-rrevid:shane@mysql.com-20100224065101-jk3s5sro2p914nvp' lp:gypsy cd gypsy make This requires `mysql_config` in the path to get includes/libraries for libmysqlclient. It also needs openssl headers/libraries (eg. install libssl-dev package). 2. Install MariaDB with group commit enabled. Packages/sources are available for this here: http://kb.askmonty.org/v/mariadb-52-replication-feature-preview These packages are however missing the --binlog-optimize-thread-scheduling option that I used to compare the two ways to schedule threads in the above blog posts. To get that option, grab this bzr tree and build: bzr branch lp:~knielsen/maria/mariadb-5.1-mwl116-threadscheduling cd mariadb-5.1-mwl116-threadscheduling BUILD/compile-pentium64-max 3. Configure and start mysqld This involves `make install` and `mysql_install_db` when building from source, check README/INSTALL for details. For configuration, set whatever parameters are interesting to test. Eg. set InnoDB buffer pool size and log file/buffer size to something sensible, set innodb_flush_log_at_trx_commit=1 and sync_binlog=1. Also create a mysql account and a database with suitable access permissions. 4. Create the table to use: mysql --host=127.0.0.1 -utest -ptestpass test MariaDB [test]> create table t (a int primary key, b int) engine=innodb; 5. Create the script for Gypsy to generate the load. Just create a file simpleload.gypsy containing this single line: ----------------------------------------------------------------------- p|1|REPLACE INTO t (a,b) VALUES (? MOD 1000000, ? MOD 1000000)|int,int ----------------------------------------------------------------------- 6. Run the bemchmark. I run it with `script` to capture the output (yeah, I know it's crude). The `gypsy` binary is found in the directory where it was compiled in step (1). script for i in 1 4 8 12 16 24 32 48 64 96 128 ; do gypsy --host=127.0.0.1 --user=test --password=testpass --database=test --queryfile=simpleload.gypsy --duration=180 --threads=$i ; done Adjust host/user/password as appropriate for the installation of mariadb. 7. That should be it! This is obviously a rather crude benchmark, sorry about that, I did not have experience with something like sysbench or tcp-c, would be interesting to see results from such benchmarks also. - Kristian.

1 0

Re: [Maria-developers] Bundle more relevant cnf files in MariaDB package
by Arjen Lentz 28 Mar '11

28 Mar '11

Hi Haidong On 12/03/2011, at 7:56 AM, Haidong Ji wrote: > I alluded to this in IRC maybe a couple of weeks back. That is, it is > really to have intelligent default for MariaDB servers. there already is a baseline my.cnf which is installed by the DEB (Debian/Ubuntu) packages. You'll find it in the debian build subdirs. It can a) be updated with more parameters b) also be put into RPM, tarball and other builds. I would recommend not redoing it from scratch, as a number of practical issues have to be considered for this kind of default config, and reinventing the wheel there is just not necessary. Simply extend what's already there! > In particular, the bundled sample my.cnf files, under > /usr/local/mysql/support-files, such as my-small.cnf, my-medium.cnf, > my-large.cnf, and my-huge.cnf, are laughable in today's world. yep they are crap and should be removed. However, since they've already been replaced in the real world this is a separate issue. The my-*.cnf are not proof of "we need to create a good sample config" but rather a "this is old junk that should be removed from the bzr tree." The hindrance there is that some builds currently use them, so simply deleting the files is not sufficient. I'll see if I can find those dependencies, and create a modified tree for merging. Cheers, Arjen. -- Arjen Lentz, Exec.Director @ Open Query (http://openquery.com) Remote expertise & maintenance for MySQL/MariaDB server environments. Follow us at http://openquery.com/blog/ & http://twitter.com/openquery _______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss(a)lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

1 0

[Maria-developers] Pushing/merging sample my.cnf
by Haidong Ji 25 Mar '11

25 Mar '11

All, I've created my trunk on launchpad for MariaDB, and am ready to make some updates to the sample my.cnf file(s) packaged with MariaDB. I have a few questions regarding how to proceed: 1. I've seen work log creation and assignment in MariaDB developer mailing list. Is work log creation necessary for changes like this, where no C/C++ source code is touched? If yes, any guidelines will be appreciated; 2. As this is more packaging related, what are the testing requirement? 3. What about peer review? I've blogged this and also just sent another email for more input. 4. After I make the change on my machine, build it, and push the changes back to Lauchpad, I will do a merge request. 5. Anything else? Comments/suggestions? Thanks, Haidong "Alex" Ji

2 4

[Maria-developers] review: [Commits] Rev 2946: Fix LP BUG#715738 in file:///home/tsk/mprog/src/5.3/
by Michael Widenius 23 Mar '11

23 Mar '11

Hi! >>>>> "timour" == timour <timour(a)askmonty.org> writes: timour> At file:///home/tsk/mprog/src/5.3/ timour> ------------------------------------------------------------ timour> revno: 2946 timour> revision-id: timour(a)askmonty.org-20110322132154-moeuyey4saatnm5x timour> parent: igor(a)askmonty.org-20110315193048-akp4yq0y079dkkx2 timour> committer: timour(a)askmonty.org timour> branch nick: 5.3 timour> timestamp: Tue 2011-03-22 15:21:54 +0200 timour> message: timour> Fix LP BUG#715738 timour> Analysis: timour> A query with implicit grouping is one with aggregate functions and timour> no GROUP BY clause. MariaDB inherits from MySQL an SQL extenstion timour> that allows mixing aggregate functions with non-aggregate fields. timour> If a query with such mixed select clause produces an empty result timour> set, the meaning of aggregate functions is well defined - either timour> NULL (MIN, MAX, etc.), or 0 (count(*)). However the non-aggregated timour> fields must also have some value, and the only reasonable value in timour> the case of empty result is NULL. <cut> === modified file 'sql/sql_lex.cc' --- a/sql/sql_lex.cc 2011-03-09 13:47:59 +0000 +++ b/sql/sql_lex.cc 2011-03-22 13:21:54 +0000 @@ -1667,6 +1667,11 @@ void st_select_lex::init_select() cond_value= having_value= Item::COND_UNDEF; inner_refs_list.empty(); full_group_by_flag= 0; + n_sum_items= 0; + n_child_sum_items= 0; + select_n_sum_item= 0; + select_n_nonsum_item= 0; + ok, but I am not totally happy with that we get even more things that needs to be initialized for every select (we already initialize a lot of things...) Please change the names of 'n_' to something more understandable. 'number_of_' or 'use the _count' prefix that we use in other places. === modified file 'sql/sql_yacc.yy' --- a/sql/sql_yacc.yy 2011-03-09 13:47:59 +0000 +++ b/sql/sql_yacc.yy 2011-03-22 13:21:54 +0000 @@ -1338,7 +1338,7 @@ bool my_yyoverflow(short **a, YYSTYPE ** ev_alter_on_schedule_completion opt_ev_rename_to opt_ev_sql_stmt %type <ulong_num> - ulong_num real_ulong_num merge_insert_types + ulong_num real_ulong_num merge_insert_types remember_n_sum_func %type <ulonglong_number> ulonglong_num real_ulonglong_num size_number @@ -7019,31 +7019,43 @@ select_item_list: ; select_item: - remember_name select_item2 remember_end select_alias + remember_n_sum_func remember_name select_item2 remember_end select_alias { THD *thd= YYTHD; - DBUG_ASSERT($1 < $3); + DBUG_ASSERT($2 < $4); - if (add_item_to_list(thd, $2)) + if (thd->lex->current_select->n_sum_items > $1) + ++thd->lex->current_select->select_n_sum_item; + else + ++thd->lex->current_select->select_n_nonsum_item; Not sure if the counting of select_n_nonsum_item is correct. For example, we may have an item: SELECT MAX(field)+const_field from t1 ... In this case 'MAX(field)+const_field' will be counted as select_n_sum_item even if it should be counted also as select_n_nonsum_item. Not sure we can do this test properly at the parsing stage... Regards, Monty

1 0

[Maria-developers] WL#189 New (by Psergey): Pending bulk renames in server codebase
by worklog-noreply＠askmonty.org 22 Mar '11

22 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Pending bulk renames in server codebase CREATION DATE..: Tue, 22 Mar 2011, 16:31 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-BackLog TASK ID........: 189 (http://askmonty.org/worklog/?tid=189) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: This task is to file things that we intend to rename in the server codebase but can't do it just yet because of the merge problems it will create. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#188 New (by Knielsen): Using --log-slave-updates to ensure crash safe/transactional slave state
by worklog-noreply＠askmonty.org 21 Mar '11

21 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Using --log-slave-updates to ensure crash safe/transactional slave state CREATION DATE..: Mon, 21 Mar 2011, 12:56 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 188 (http://askmonty.org/worklog/?tid=188) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Overview -------- A replication slave needs to preserve certain state between slave server restarts to be able to correctly resume replication from where it left off. In current MySQL replication, this state is kept in multiple simple files (master.info, relay-log.info). This is a big problem if the slave server crashes, as there is no guarantee that these files will be in a consistent state with what is in the table data (and binlog if using --log-slow-updates), or even with each other. The Google patch rpl_transaction_enabled has a partial solution for this, by duplicating the state inside InnoDB in a transactional way. At slave server startup, InnoDB can then decide to overwrite the files keeping the slave state with its own, hopefully more correct, information. There are some remaining problems with this approach, some of which may be fixable, some not. However, the basic problem here is the need to maintain state in a transactional/crash-safe way across multiple subsystems of the server (eg. replication and storage engine(s)). And we already have such a mechanism, in the form of the two-phase commit between engines and binlog. This worklog describes how we could use this existing mechanism to keep the replication slave state across server restarts in a crash-safe way, rather than introduce new complex mechanisms for every new piece of state to be kept. Idea ---- The main state we need to make crash-safe is the binlog position (filename,offset) of the next event from the master to execute on the slave. There is more state stored current, but there is less need for that to be transactional: - relay-log.info also stores the position in the relay log files on the slave which event execution has reached. In the case of normal shutdown this can be used file as-is. In case of recovery after a crash, it is possible that the relay logs are not consistent with the master and/or slave SQL thread, so it is probably better to just discard any existing relay log and re-fetch all necessary events from the master. - master.info mainly stores the connection information from CHANGE MASTER TO, which does not change often. The basic idea is that on slave server start, we get the required information from the binlog on the slave, rather than from these files (this requires that --log-slave-updates is enabled). This also has the advantage that we avoid the need to constantly update the state files after every transaction, saving some execution cost. When we shut down the server normally, we close the binlog in a way that allows at startup to detect if we are recovering from a crash or not. As part of this close, we can write whatever state we need to recover at the end of the binlog. Recovering the state then depends on whether we crashed or not. If we did not crash, then all we need is to able to find the position of whatever event was written at the end of the binlog with the state. One way is to just have a fixed offset from the end where this event starts; however this is not too robust against finding wrong data there, especially with respect to binlogs from different versions of the server possibly with different events (or event sizes) at the end of the log. Another way is that when we close the binlog, we seek back to the start of the log and write the position of the last event there (either in the format_description event, which already has version information, or a new event written just after format_description event). We already do this seek anyway to overwrite the flag in format_description event which signals that the binlog has been closed properly (not crashed). We just need to be sure to fsync() the binlog _before_ overwriting the crashed-or-not flag, so we are sure all data will be there in the not-crashed case. If we did crash, then we can not rely on any information at the end of the binlog. In this case, we can instead build on top of the already existing crash recovery mechanism. This mechanism scans the last binlog to build a list of all committed transactions, then uses this list to tell storage engines which previously prepared transactions to commit and which to roll back. As part of this scan, we can determine the last event executed, and from this we get the necessary state to continue replication in terms of the binlog position (filename,offset) on the master. Note that if something like group ID (MWL#175) or global transaction ID is implemented, then the state to preserve is the last executed ID rather than binlog position; however, the basic mechanism remains the same. Discussion ---------- The main disadvantage I see with this approach is that in order for this to be really crash safe, we need to run with innodb_flush_log_at_trx_commit=1 and sync_binlog=1. This requires 3 fsync() calls per commit, and group commit is not possible since the slave is single-threaded. This is likely to be too expensive for many installations to be able to use it. However, it may be possible to reduce this overhead sufficiently by using MWL#185 (grouping multiple commits together on the slave to reduce fsync() overhead), or by implementing parallel replication to utilise group commit (MWL#169, MWL#184, MWL#186). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#187 New (by Knielsen): Consistent (but non-durable) crash recovery from binlog 2-phase commit
by worklog-noreply＠askmonty.org 21 Mar '11

21 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Consistent (but non-durable) crash recovery from binlog 2-phase commit CREATION DATE..: Mon, 21 Mar 2011, 11:54 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 187 (http://askmonty.org/worklog/?tid=187) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Overview -------- Current MySQL needs no less than three fsync() calls for every commit in order to ensure the ability to recover into a consistent state (between master binlog and storage engine(s)). Without this, after a master crash, the binlog may be inconsistent with the data in tables on the master, and replication slaves will diverge from the master, possibly causing replication to eventually break. The MWL#116 implements group commit, so that each of the three fsync() calls can be shared among multiple transaction commits. The MWL#164 describes a way to reduce this to a single fsync() call shared among multiple commits, by recovering any transactions lost in storage engines from the binlog, thereby avoiding the need for fsync() calls in the engines. Both MWL#116 and MWL#164 improve performance without reducing functionality or ability to do crash recovery. This worklog describes a method to reduce the overhead to *zero* fsync() calls per commit. Though one will still want to have say one fsync() per second or whatever of the binlog. After a master crash, crash recovery is still guaranteed to recover the system into a consistent state, where every transaction committed in the storage engine is also committed in the binlog, and vice versa. However, the cost of this is that durability is no longer guaranteed: after a crash, a number of transactions committed before the crash may be lost. Thus this worklog is similar to the InnoDB feature innodb_flush_log_at_trx_commit=2 (or 0). That feature makes InnoDB not call fsync() after every commit, possibly loosing some transactions in case of crash, but still guaranteeing that the table data can be recovered into a consistent state. This worklog would do the same, but for the binlog and InnoDB together, whereas innodb_flush_log_at_trx_commit=2/0 only guarantees consistency for InnoDB in isolation. Idea ---- This worklog builds on top of the group commit framework implemented in MWL#116. The basic idea is similar to MWL#163, "release of row locks in InnoDB during prepare() phase". As soon as we have successfully run the prepare step in InnoDB, we make the transaction committed in memory so that it is visible to other transactions from that point on and all row locks are released. However, at this point we go further than MWL#163, and return successful commit to the client connection. The rest of the group commit work is delegated to a separate backgroud thread: writing and fsync()'ing to the binlog, running commit_ordered() and commit() inside the storage engine. The client is thus free to continue with new transactions without waiting for any fsync() delay. The background thread will do an fsync() call, but since no client is waiting for it, the background thread is free to do a long wait (eg. 1 second) between fsync() calls, potentially collecting lots of transtions for a single huge group commit. Note that any replication slave _will_ be waiting for the background thread; we must wait for fsync() of the binary log before sending events to the slaves, or we could in case of crash end up with more transactions on the slaves than on the master, which would break replication. So there will be a cost of extra replication latency. Discussion ---------- A main complication with this worklog is that the main part of the commit will run in a background thread, which is different from the thread that ran the main transaction. This requires that storage engines are checked, and modified as needed, so that they do not depend on thread local storage in that part of the code, and also do not do things like release a mutex that was aquired from another thread, etc. We also need to handle the issue that we need the THD object of the original transaction/thread to be able to finish the commit in the background thread. I think we will need to remove the THD object from the original thread before returning to the client and keep a reference to it in the queue of transactions waiting to group commit in the background thread. The background thread must then use this THD for the last part of commit and afterwards return the THD object to a pool of spare THDs; when/if the client starts a new transaction, it will then need to obtain a new THD from this pool of spare THDs. ---- Currently, the MWL#116 API has prepare_ordered() being called _after_ prepare(). I am not sure if there is any particularly good reason for this. The only reason I know is that this is how the Facebook innodb_release_locks_early patch did things, however when I asked I did not get any answer from Facebook why they did it in this order. If we switched this, we should be able to return to the client even earlier, already before prepare(), so we could keep the fsync() in prepare(). Alternatively, we should use MWL#164 to not have to call fsync() in prepare(); then we can return to the client only after prepare(), and still avoid any fsync() latency visible to the client. Note that we have to make the transaction committed to memory and visible to other transactions before returning to the client, just as with innodb_release_locks_early in MWL#163. The client would be quite confused if COMMIT returns ok, yet the committed transaction is not visible to following SELECT statements! ---- The first transaction to enter the group commit queue becomes the group commit leader, and would just signal the background thread to continue the commit sequence and then return to the client. Subsequent transactions that enqueue after would not signal the background thread; instead they would set a flag in their entry in the queue, so that the background thread knows that they are asynchroneous and should not be woken up after commit is done (the original thread may no longer exist at that point, or may be doing something else). ---- The main benefit of this worklog would be that we could get crash safe replication state on the master without any fsync() overhead visible to the client. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#186 New (by Knielsen): Parallel replication using interleaved binlog events to detect potential parallelism
by worklog-noreply＠askmonty.org 21 Mar '11

21 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Parallel replication using interleaved binlog events to detect potential parallelism CREATION DATE..: Mon, 21 Mar 2011, 10:45 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 186 (http://askmonty.org/worklog/?tid=186) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Similar to MWL#169 and MWL#184, this worklog describes a way to discover binlog events that can be executed in parallel on the slave, improving scalability with increasing CPU and I/O resources. This worklog only deals with parallel replication of row-based events. Statement-based events should still be executed in serial order. We also require that commits on the slave still happen in the same order as on the master (this guarantees that applications can never see a state on a slave that did not exist on the master, among other nice properties). Basic idea ---------- The idea is to work with row-based events. Suppose that on the master, we have two parallel transactions running, T_1 and T_2. Suppose that we have the following sequence of events: 1. T_2 modifies a row R_1, resulting in row event E_1 2. T_1 commits 3. T_2 commits. In this case, it is safe on a slave to apply the event E_1 of T_2 in parallel with any events of transaction T_1. Since we know T_1 committed before T_2, we know that T_1 did not have to wait for T_2 to release any row lock of R_1 in commit, so T_1 could not have modified R_1 after T_2 did. Similarly, since we know that T_2 executed E_1 before T_1 committed, we know E_1 did not have to wait for T_1 to release any row locks on R_1, so T_1 could not have modified R_1 before T_2 did. Thus, we know that T_1 and T_2 are not in conflict on this particular row R_1 and row event E_1. Thus, on a slave, with this knowledge we can start transaction T_2 and execute E_1 (in a separate thread from T_1 events) while T_1 is still running. Refinement: optimistic apply ---------------------------- Suppose now that instead we have the following sequence of events: 1. T_3 commits 2. T_4 modifies a row R_2, resulting in row event E_2 3. T_4 commits In this case, we can not be sure that event E_2 is independent of transaction T_3. It could well be that T_3 also modified R_2, and E_2 was waiting for T_3 to commit and release row locks. However, on a slave, we can nevertheless try to apply E_2 (in a separate thread) while T_3 is still running. However, then we have to deal with the situation that T_3 gets stuck waiting for T_4 to commit and release the lock on R_2. In this situation, we have a deadlock; T3 is waiting for T_4, but T_4 also needs to wait for T_3 (as we want to commit in the right order, and even more so want the end result to have the changes of T_4 in row R_2, not those of T_3). So we need to abort and restart one of the transactions. As long as we choose always to abort the transaction with the later commit time (T_4 in this case), things will work out: We will abort T_4, commit T_3, then re-start and commit T_4. Using optimistic apply has the potential to increase parallelism on the slave. However, this must be carefully weighted against the extra cost of having to roll back and retry some transactions. So it needs suitable status variables to be able to monitor number of transactions and number of events that were rolled back and re-started (as well as number of transactions and events successfully applied). And user options to control if and to what extend to try optimistic apply. But note that since we always have at least one transaction running that will not be aborted due to optimistic apply (the one with earliest commit), and since all other transactions run in parallel, the cost of restart may not be too bad, as long as we do not start to starve the CPU(s). Any extra I/O reads done in the aborted transaction to bring in pages to the buffer pool will in any case benefit us when we immediately re-start the transaction. Requirements ------------ Implementing this worklog will require several non-trivial extensions of the server and storage engine code (these should eventually be filed as separate worklog items): 1. Ability for server layer to detect and control what to do in case of conflicts between two transactions. Currently, the storage engine (eg. InnoDB/XtraDB) will just let the latter access block waiting for the first transaction to commit, and in the case of a deadlock the storage engine will choose an arbitrary transaction to roll back. For this worklog, the server layer (replication slave SQL thread(s)) must be able to (a) cause abort of the former transaction rather than wait of the second, and (b) choose which of several possible transactions to abort in case of deadlock. Note that such a mechanism is also needed for Galera replication, and the Galera wsrep patch implements something like this for InnoDB already. 2. Interleaving of events from different transactions in the binlog. Currently, the master buffers events during a transaction and writes them all out sequentially during COMMIT; this destroys any knowledge about which events and transactions ran in parallel on the master. We need to change this so that events are tagged with a (local) transaction ID and written out to the binlog as they occur, interleaved with events from other transactions. We will probably also want to add "begin transaction" events and "prepare" event, to increase the knowledge available on the slave about in which order events occured. We will still want to do _some_ buffering of events to avoid heavy contention on writing to the binlog. We can buffer a small number of normal events; The cost of such buffering is we may write a row event E_r _after_ a commit event E_c where it could have been written before, which loses some opportunity for (non-optimistic) parallel apply on the slave. We may however _not_ buffer commit events; such buffering could wrongly put E_r ahead of E_c where it must be after, causing an unexpected conflict in parallel apply on the slave. 3. A mechanism on the slave for distributing binlog events among multiple threads and coordinating execution and commit among the threads. Such mechanism should ideally be the same as used in MWL#184 and MWL#169. Discussion ---------- One nice property of this worklog is that the parallelism exposed to the slave is derived from whatever parallelism was there already on the master. We know E_1 in T_2 ran in parallel with T_1 on the master, and use this to similarly run E_1 in parallel with T_1 on the slave. This increases the likelyhood that if parallelism is available on the master, it will also be available on the slave (note that we will need parallelism on both master and slave to be able to scale an entire replication topology). ---- For short OLTP-like transactions, it seems likely that this method will be particularly suited to be combined with the method from MWL#184, which allows to run in parallel on the slave all transactions that participate in a single group commit on the master. Eg. for single-row transactions T_1 (with row event E_1) and T_2 (with row event E_2) that run in parallel on the master, we can have the following two situations (among others): 1. E_1 and E_2 execute start 1. E_1 and E_2 execute start 2. E_1 execute finish 2. E_1 execute finish 3. E_2 execute finish 3. T_1 commit 4. T_1 commit 4. E_2 execute finish 5. T_2 commit 5. T_2 commit In the scenario to the left, on a busy master the two commits are likely to be done as a single group commit, and MWL#184 will allow parallelism on the slave. In the scenario on the right, optimistic apply as per this worklog will allow to run T_1 and T_2 in parallel on the slave. ---- Note that by interleaving events from different transactions in the binlog, we will be writing events before they are committed, and so will need to issue ROLLBACK statements in case of rollback. Similarly, on the slave, we can start applying the events from a transaction even before we have received the commit event from the master (in fact even before such commit event has even occured on the master). This can potentially reduce master->slave delay. However, there is a need to be careful, as on the slave we need to keep the correct commit order. And such order is not known until the commit events are received. For example, if we had a limited number N of slave execution threads and they all have started running T_1, ..., T_N, and we then receive a commit event for T_(N+1), we would be deadlocked. It might be a good idea to only start executing new transactions when we have at least one pending commit in the relay log, or at least keep one spare thread available for execution of one more transaction once we receive such commit event. Similarly, if we get a conflict in opportunistic apply of transactions T_1 and T_2, neither of which we have received the commit event from, then we will need to wait for either commit event to arrive before we can decide which of T_1 and T_2 to abort. Note that in any case what we need to do is no worse than what we do in currentl MySQL replication. There, we never do any execution until we have not only seen but also executed any prior commit. Similarly, whatever buffering of events we need to do on the parallel slave will be no worse than what we currently have to do on the master (where we buffer _all_ events until commit). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#185 New (by Knielsen): Group commits on slave to reduce fsync() load
by worklog-noreply＠askmonty.org 18 Mar '11

18 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Group commits on slave to reduce fsync() load CREATION DATE..: Fri, 18 Mar 2011, 14:25 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 185 (http://askmonty.org/worklog/?tid=185) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: MWL#116 introduces group commit for the binary log; this can reduce the number of fsync() calls needed when running with sync_binlog=1. But this does not currently help on the slave. The reason is that the slave SQL thread is single-threaded, so there will never be any opportunity for group commit. However, an alternative way to reduce fsync() on the slave when sync_binlog=1 is to combine multiple transactions from the binlog into one transaction on the slave. If we have N transactions T_1, ..., T_N in the binlog, we can combine them simply into one transaction on the slave like this: - Change COMMIT in T_1, ..., T_(N-1) into SAVEPOINT T_i - Change any ROLLBACK into ROLLBACK to T_i This will reduce the number of fsync() calls, since it results in fewer COMMITs. The downside is that the latency between a transaction being visible on the master and on the slave will increase. There should probably be some option to limit how much transactions may be delayed on the slave. It would also be useful to have some kind of timestamps in the binlog to be able to estimate the duration of a transaction on the slave before starting to apply it; this would allow to avoid combining a later long transaction with a previous short transaction, which could cause undue delay for the short transaction to become visible on the slave. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

Re: [Maria-developers] mysql_upgrade issue for libmysqld and amarok
by Kristian Nielsen 18 Mar '11

18 Mar '11

Ian Monroe <ian(a)monroe.nu> writes: > On Fri, Mar 18, 2011 at 05:44, Kristian Nielsen > <knielsen(a)knielsen-hq.org> wrote: >> The issue is how to upgrade libmysqd from MySQL version 5.1 to version 5.5 (or >> more generally any major version upgrade), in particular as relates to >> Amarok's use of libmysqld. >> One idea is to also build mysql_upgrade_embedded; this would be a variant of >> mysql_upgrade linked with libmysqld, so it does not depend on the mysqld >> server. Similarly, we would need mysqlcheck_embedded (mysqlcheck is called >> from mysql_upgrade). These two binaries could be included in the libmysqld >> package (they are small). > > wouldn't they be huge? libmysqld is a static library, so anything > linking to them is going to be at least 50mb. In MariaDB, we have both static and dynamic libmysqld. So we should link with the dynamic libmysqld.so, and binaries should be small. >> The upgrade procedure for amarok would then be to stop amarok linked with 5.1, >> then run mysql_upgrade_embedded (from 5.5) against the amarok libmysqld data >> directory, then start amarok linked with 5.5. > > There can't be any procedure that the user has to do. Agree. > But if there is some way for us to determine if mysql_upgrade_embedded > needs to be run, I don't see why this would a problem. Amarok could > run it before initializing the database. I would suggest to write a small file in the user's data directory with the version last upgraded to. If this is < 5.5 at startup (eg. "5.1"), then Amarok can run the upgrade and afterwards update the file to say "5.5". This will also allow to detect and warn about downgrading 5.5->5.1. Debian does something similar for the main MySQL server packages. >> The other idea that came up was to add this as extra calls in the library. So >> that the amarok application could just call some mysql_upgrade() function >> instead of having to run an external upgrade binary. Would this be more useful >> for amarok? > > This sounds much more straightforward. Ok, I will see if we can do this. Probably we could do both (then mysql_upgrade_embedded will just be a stub calling into mysql_upgrade() in the library, might be useful in other applications). >> We could offer to implement this (either solution I think) in MariaDB (a >> version of MariaDB that includes MySQL 5.5 is in the works). This would allow >> to use the above to upgrade Amarok to using MariaDB 5.5. Would this work? > > Sounds good. > We really could care less about maria vs. mysql, we just want > something that works. Ok, great :-) > Since I talked to you last time, I've had reports of people going from > mysql 5.1 to 5.5 without needing mysql_upgrade. mysql_upgrade seemed > to fix things for me, but maybe it was just a coincidence. Yes. The main issue I can think of now is that when character sets/collations change, the indexes of affected tables may need to be rebuilt. Otherwise they may contain incorrect ordering or indexing of affected characters. Since such changes are rare, and tables with affected characters probably even rarer, a missing upgrade is likely to go ok or at least unnoticed for some time. > Either way though, relying on newer versions of mysql simply not > needing to do any upgrade procedures seems like poor planning. Agree, even if it works for 5.5 it will be sure to break sooner or later. Monty said he would look into implementing this. He is quite busy, so it may be a few weeks, but I will try to follow up with him to see if we can move forward with it reasonably quickly. - Kristian.

1 0

[Maria-developers] WL#184 New (by Knielsen): Parallel replication of group-committed transactions
by worklog-noreply＠askmonty.org 18 Mar '11

18 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Parallel replication of group-committed transactions CREATION DATE..: Fri, 18 Mar 2011, 14:06 SUPERVISOR.....: Sergei IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 184 (http://askmonty.org/worklog/?tid=184) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Overview -------- Parallel replication is the idea of executing multiple transactions (or more precisely event groups from the binlog) in parallel in multiple threads on the slave, in order to make the slave applier more scalable to multiple CPU cores and disk spindles. The main problem with parallel replication is identifying transactions that are *independent*, meaning that they can be safely executed in parallel on the slave without dead-locking each other or producing different results than on the master. This worklog describes one way of identifying such independence. MWL#169 describes another. Apart from discovering independent transactions, it is also necessary to implement a mechanism for actually distributing events to different execution threads, coordinating commit order, handling errors and crash recovery, etc. It is expected that different ways of determining independent transactions, such as this worklog and MWL#169, will share the same mechanism for actual parallel execution. It should even be possible to combine different ways to discover independence at the same time. Idea ---- MWL#116 introduces group commit, where multiple transactions are committed together and also written to the binlog together. This works by collecting parallel transactions that are waiting inside COMMIT (or autocommit) for access to the binary log. The idea in this worklog is that all transactions participating in a single group commit must necessarily be independent, and can be safely applied in parallel on a slave. Suppose we are about to write N transactions T_1, ..., T_N into the binlog in group commit. Then all statements for each transaction have completed, but none of them are committed yet. This means that (1) there can be no conflicting row locks between any two T_i and T_j, and (2) no transaction T_i was able to see any changes from T_j (for i<>j and assuming isolation level greater than read uncommitted). Another way to see this is to note that on the master, the order of T_1, ..., T_N is essentially random and depends only on how the kernel decided to schedule the threads during the COMMIT. Any ordering would have been possible just by having different thread scheduling delays when the transactions are put in the ordered list used for group commit; there is no inter-thread dependencies at that point in the code. So any order of the transactions in the binlog would have been possible, and thus any order of slave execution. Therefore, as long as COMMIT cannot change what modifications the transaction already did, any execution order of the transactions on the slave is permissible. Implementation -------------- I suggest that when this feature is enabled, we introduce two new binlog events, group_commit_begin and group_commit_end. We write a group_commit_begin event just before all the events for the transactions in a group commit, and a group_commit_end event just after. We can use the "ignorable events" from MySQL 5.5 for these events; this will allow a slave that does not implement these events to safely ignore them. If needed, we can further increase compatibility by - Having a server option to enable/disable the writing of these two events. - Add a flag that the slave sets to request the sending of these events, and only send them if requested (similar to MWL#47). It still needs to be decided how much of this to do. The slave can then collect all the event groups between the group_commit_begin and group_commit_end pair, and schedule them in parallel. Limitations ----------- The main limitation of this approach is that on the slave, it can only run transactions in parallel if they happened to commit at roughly the same time on the master. This means that there will be little opportunity for parallelism on the slave if 1. The master has low write load 2. The master is mainly running few but long transactions 3. The master has fast fsync() performance for the commit In the case (1), there is in any case less need for scaling the event execution. In the case (2), another method (for example MWL#169) will be needed to achieve efficient parallel replication. For (3), we can improve parallelism by deliberately delaying commits on the master. By inserting a short sleep just before group commit, we allow more independent transactions time to participate, giving more opportunity for parallelism on the slave. At the same time, we reduce fsync() load on the master, potentially freeing more I/O capacity for other work. For example, we might add an option to rate-limit COMMIT fsync() to 25 per seconds or whatever. Care will be needed when using this option, as if the master load has insufficient parallelism, such rate-limitation will reduce throughput on the master. The MWL#163 innodb_release_locks_early option causes transactions to effectively be committed early in COMMIT during the prepare phase. This allows another transaction to see and/or modify rows also modified by the first transaction, and still participate in the same group commit. This means that when innodb_release_locks_early is set, the transactions cannot be safely used for parallel replication (eg. in this case we must disable the writing of group_commit_begin / group_commit_end). Events that modify non-transactional tables or do DDL causes more complications, as they are written directly to the binlog and make changes visible immediately to other transactions (basically they are not transactional). To simplify things, we will not declare any such events independent with respect to any other transactions according to this worklog. There are many useful applications of parallel replications where such events are rare. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] mysql_upgrade issue for libmysqld and amarok
by Kristian Nielsen 18 Mar '11

18 Mar '11

This is a follow up on a discussion I had with eean on Freenode IRC #maria. The issue is how to upgrade libmysqd from MySQL version 5.1 to version 5.5 (or more generally any major version upgrade), in particular as relates to Amarok's use of libmysqld. The procedure for upgrading a normal MySQL server is to run the mysql_upgrade command. However, this requires a running mysqld server, so it is not appropriate for libmysqld, or at least very inconvenient. We discussed this briefly at our MariaDB meeting in Lisbon, and here is what we came up with: One idea is to also build mysql_upgrade_embedded; this would be a variant of mysql_upgrade linked with libmysqld, so it does not depend on the mysqld server. Similarly, we would need mysqlcheck_embedded (mysqlcheck is called from mysql_upgrade). These two binaries could be included in the libmysqld package (they are small). The upgrade procedure for amarok would then be to stop amarok linked with 5.1, then run mysql_upgrade_embedded (from 5.5) against the amarok libmysqld data directory, then start amarok linked with 5.5. Does that sound useful? The other idea that came up was to add this as extra calls in the library. So that the amarok application could just call some mysql_upgrade() function instead of having to run an external upgrade binary. Would this be more useful for amarok? Either way, we then need to think on how to implement this change. We could offer to implement this (either solution I think) in MariaDB (a version of MariaDB that includes MySQL 5.5 is in the works). This would allow to use the above to upgrade Amarok to using MariaDB 5.5. Would this work? (I think it would even work to keep Amarok using MySQL 5.5 libmysqld, but using MariaDB 5.5 mysql_upgrade_embedded, though that is kind of a strange solution. Or we could provide the patch for MariaDB and you could patch libmysqld similarly to provide the same functionality in relevant distros.) Does this sound like a feasible way forward? One question about how Amarok uses libmysqld, which will help understand what is needed to solve the upgrade issue: Does Amarok use libmysqld built with the --with-embedded-privilege-control option, or use stored procedures, or other things that require the `mysql' database schema to be available in MySQL? If this does not make sense, the answer is probably "no" :-) The background: When a normal standalone mysql database server is installed, it is necessary to create the `mysql' database schema using mysql_create_db. This database schema contains such things as user accounts and passwords, stored procedure code, etc. Normally, libmysqld does not do any access control on accounts, and it is possible to use it without running mysql_install_db. In fact it's a bit hard to run mysql_install_db, as it has the same issue as mysql_upgrade. So I assume that Amarok does not do this and just does not use account privileges or stored procedures/functions, but just wanted to check? This affects how much mysql_upgrade_embedded has to do, as part of the upgrade is to add any missing columns in tables in the `mysql' database schema. - Kristian.

1 0

Re: [Maria-developers] [All] How are we going to handle incompatible changes in MariaDB?
by Kristian Nielsen 18 Mar '11

18 Mar '11

"Philip Stoev" <pstoev(a)askmonty.org> writes: > We should discuss how do we handle new developments in MariaDB that cause > our behavior to diverge from the one in MySQL. Since we claim to be a Yes, agree. Thank you for bringing it up! Note that this is a general MariaDB discussion, not a Monty Program specific one. So I move the thread to maria-developers@. So far, we have generally been ok at handing incompatible changes. The basic approach is to shape new features to be backwards compatible as much as possible, and to add any extra needed code to handle interoperability. For example, if an option is changed due to a feature, we should keep the old option around and emulate the bahaviour, eventually deprecating it in a future version. Another example, with replication, we check the version of the master or slave at the other end, and take care to only send what the other end understands, and likewise interpret what it sends correctly based on its version. However, this has been mostly up to the individual developer (or often reviewer), and if we can get some more structure/process to this to avoid it being forgotten, then that would be very good. > The particular situation we are facing now is the microsecond precision > patch from Serg. As part of introducing precision date and time datatypes, > he made some refactoring around the Item_* tree. > > As a result, various corner cases, e.g. invalid dates, partially valid > dates, invalid conversions between date and time, etc. are now handled > differently, with either different warnings being printed (or no warnings at > all), or with the result of the expression being different, or with a NULL > or 0000-00-00 returned. > > For example, mysql reports that DATE ( SEC_TO_TIME( 8 ) ) is equal to > "2000-00-08" whereas serg's patch returns "0000-00-00" . In other words, > MySQL chose to place the "8" literal into the days portion of the date > value, whereas Serg's patch filtered the literal somewhere along the way. > There are numerous other such examples. > > While in most cases Serg's new behavior is more intuitive and a program that > is written to conform to postgresql-level type safety will not be affected > either way, it entirely possible that a program written under mysql's > relaxed type safety rules, and with MySQL's behavior in mind, would start > behaving unexpectedly. It sounds like in this case, the changes in behaviour could be considered improvements, or even bug fixes, when seen in isolation. And the behaviour they change is not something that an application would deliberately be written to rely on. But since it is a change in behaviour, it has the potential to break applications that by accident happen to depend on such behaviour. My opinion on this is that we should do the change in a major version (eg. 5.2->5.3), but not in a minor version (eg. 5.2.5->5.2.6). And document it in release notes as an incompatible change. Basically the same way we did in MySQL (and still do, I suppose). It is ok to change behaviour in a major version, if it is necessary, and documented. I believe users anticipate such changes, and consequently are more careful about going eg. 5.1->5.5 than 5.1.x->5.1.y. We should not needlessly change things. Every behaviour change should be carefully considered. If it has the potential to seriously break major applications, we should not change it (or use deprecation, compatibility mode, etc). If the change is not necessary or useful, it should probably be left. But if the change makes sense and the new behaviour is much more reasonable, then we should be ready to make the change. We cannot afford to be stuck forever on old poor decisions. Another example is optimiser changes. In my opinion, we cannot totally avoid that improvements to the optimiser will make some corner case queries slower in some cases. The optimiser has only limited knowledge of the data and application. If, based on this knowledge, execution plan A is better than execution plan B, then we should be prepared to change (in a major version) the code to select A over B. This will invaribly cause slowdown in some cases, where the knowledge is incomplete. But it is important to be able to make improvements. But again, we need to consider each case carefully. We should not do things that are obviously stupid (eg. do not select table scan over range scan unless _really_ sure it will be better), and we should avoid breaking major applications or use patterns, even if it theoretically makes sense. So summary of my opinion: We should be very careful about incompatible changes. We should generally avoid them unless there is a very good reason for them. We should create backward compatibility where possible and where it makes sense. And if we do change behaviour, we should only do it in a major version and document it clearly in an "incompatible changes" list. And being able to continually improve the server long-term is important. > So, how we should handle this particular case, and more importantly, how do > we handle future cases like that in a stable and predictable manner? I > understand that in the early days of MySQL it would have been a no-brainer > to change everything as needed, we are by now dealing with a huge installed > base and powerful competitors, so we can not simply apply the happy hacking > approach that was used in the 1990s. Agree. No "happy hacking" approach to incompatible changes! - Kristian.

1 0

[Maria-developers] request for feedback on live schema changes patch
by Zardosht Kasheff 18 Mar '11

18 Mar '11

Hello all, In another thread, there was a discussion about what it would take to allow the storage engine to support live schema changes in mysql and mariadb. I said that I was working porting this existing feature from MySQL Cluster to MySQL 5.1.46. I would like to show the work I have done so far and solicit feedback. Attached is the diff based off of 5.1.46. Before looking at the patch, please be aware of the following things: - I am not an expert in this code. As a result, I had to use an approach of that minimizes the risk of introducing regressions. \ - My main interest is to get something safe that works and can be integrated in 5.1. As a result, kept the current mysql_alter_table functionality in place (as old_mysql_alter_table), and added MySQL Cluster's as new_mysql_alter_table. For an engine to get new_mysql_alter_table to be used, the engine must expose an alter table flag HA_GENERAL_ONLINE. - Another reason why I kept old_mysql_alter_table was that I did not want to deal with engines that implement handler::add_index and make sure that they work properly under the new API. Also, I did not want to port alter table changes related to partitioned tables. - I have run the current 5,1 mysql test suite (that is in the mysql-test directory of source tarballs), and verified that using old_mysql_alter_table and new_mysql_alter_table works fine with them. That being said, I would love to hear feedback on this patch, and would also like to understand what (if anything) can be done to get this into MariaDB 5.1. I realize that may not be possible, but if it is, I would like to try. Thanks -Zardosht

2 5

[Maria-developers] WL#182 New (by Mdcallag): Explain running statements
by worklog-noreply＠askmonty.org 13 Mar '11

13 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Explain running statements CREATION DATE..: Sun, 13 Mar 2011, 09:44 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-BackLog TASK ID........: 182 (http://askmonty.org/worklog/?tid=182) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: I want to know the query plan for running statements -- something like "SHOW PLAN FOR <connection_id>". This requires running statements to generate a string that describes their plan in case another thread needs to examine it. For me it would be good enough to know the join order and indexes. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] URLs for worklog tasks include bin name
by MARK CALLAGHAN 13 Mar '11

13 Mar '11

When I click on http://askmonty.org/worklog/?tid=180 I end up at http://askmonty.org/worklog/WorkLog-RawIdeaBin/?tid=180. Is that last URL only valid as long as tid=180 remains in the RawIdea bin? If so, then when I share it in email or include that URL elsewhere how are people to use it when tid=180 is moved from the RawIdeaBin to another bin? -- Mark Callaghan mdcallag(a)gmail.com

1 0

[Maria-developers] User Comment -> Add Comment is bugging in MariaDB worklogs
by MARK CALLAGHAN 13 Mar '11

13 Mar '11

Go to a worklog, Scroll down, Click on "Add a new comment" This takes me back to the top of the page. This is confusing and now I must scroll down again to reach the comment text box. Add text, click on OK This takes me to another page that has a text box. The text I previously entered is lost. This is a bug. If I enter text on this page and click OK it is not lost. -- Mark Callaghan mdcallag(a)gmail.com

1 1

Re: [Maria-developers] [Commits] Rev 2935: Fixed LP bugs BUG#729067/730466. in file:///home/igor/maria/maria-5.3-bug729067/
by Oleksandr Byelkin 12 Mar '11

12 Mar '11

Hi! Ok to push. 12 марта 2011, в 10:49, Igor Babaev написал(а): > At file:///home/igor/maria/maria-5.3-bug729067/ > > ------------------------------------------------------------ > revno: 2935 > revision-id: igor(a)askmonty.org-20110312084903-b9mmi2wdgbdar3tc > parent: timour(a)askmonty.org-20110308212344-7r4ue17b3rxschlj > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-bug729067 > timestamp: Sat 2011-03-12 00:49:03 -0800 > message: > Fixed LP bugs BUG#729067/730466. > Do not reset the value of the item_equal field in the Item_field object > once it has been set. > === modified file 'mysql-test/r/join.result' > --- a/mysql-test/r/join.result 2011-02-28 21:29:59 +0000 > +++ b/mysql-test/r/join.result 2011-03-12 08:49:03 +0000 > @@ -1285,3 +1285,38 @@ > 1 1 1 1 1 > 1 1 1 1 1 > drop table t1,t2,t3; > +# > +# BUG#729067/730466: unexpected 'Range checked for each record' > +# for queries with OR in WHERE clause > +# > +CREATE TABLE t1 (f1 int, f2 int) ; > +INSERT INTO t1 VALUES (4,0),(5,1); > +CREATE TABLE t2 (f1 int, f2 int, KEY (f2)) ; > +INSERT INTO t2 VALUES (5,7), (8,9); > +EXPLAIN > +SELECT * FROM t1 STRAIGHT_JOIN t2 ON t2.f1 = t1.f1 > +WHERE t1.f1<>0 OR t1.f2<>0 AND t1.f1 = t2.f2; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 Using where > +1 SIMPLE t2 ALL f2 NULL NULL NULL 2 Using where; Using join buffer (flat, BNL join) > +SELECT * FROM t1 STRAIGHT_JOIN t2 ON t2.f1 = t1.f1 > +WHERE t1.f1<>0 OR t1.f2<>0 AND t1.f1 = t2.f2; > +f1 f2 f1 f2 > +5 1 5 7 > +DROP TABLE t1,t2; > +CREATE TABLE t1(f1 int PRIMARY KEY, f2 int) ; > +INSERT INTO t1 VALUES (9,4), (10,9); > +CREATE TABLE t2(f1 int PRIMARY KEY, f2 int) ; > +INSERT INTO t2 VALUES (9,4), (10,9); > +EXPLAIN > +SELECT STRAIGHT_JOIN * FROM t1 JOIN t2 ON t2.f2 = t1.f1 > +WHERE t1.f1 IN (SELECT f1 FROM t1) AND t1.f1 = t2.f1 OR t1.f1 = 9; > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY t1 ALL PRIMARY NULL NULL NULL 2 Using where > +1 PRIMARY t2 ALL PRIMARY NULL NULL NULL 2 Using where; Using join buffer (flat, BNL join) > +2 SUBQUERY t1 index NULL PRIMARY 4 NULL 2 Using index > +SELECT STRAIGHT_JOIN * FROM t1 JOIN t2 ON t2.f2 = t1.f1 > +WHERE t1.f1 IN (SELECT f1 FROM t1) AND t1.f1 = t2.f1 OR t1.f1 = 9; > +f1 f2 f1 f2 > +9 4 10 9 > +DROP TABLE t1,t2; > > === modified file 'mysql-test/t/join.test' > --- a/mysql-test/t/join.test 2011-02-28 21:29:59 +0000 > +++ b/mysql-test/t/join.test 2011-03-12 08:49:03 +0000 > @@ -964,3 +964,36 @@ > > drop table t1,t2,t3; > > +--echo # > +--echo # BUG#729067/730466: unexpected 'Range checked for each record' > +--echo # for queries with OR in WHERE clause > +--echo # > + > +CREATE TABLE t1 (f1 int, f2 int) ; > +INSERT INTO t1 VALUES (4,0),(5,1); > + > +CREATE TABLE t2 (f1 int, f2 int, KEY (f2)) ; > +INSERT INTO t2 VALUES (5,7), (8,9); > + > +EXPLAIN > +SELECT * FROM t1 STRAIGHT_JOIN t2 ON t2.f1 = t1.f1 > + WHERE t1.f1<>0 OR t1.f2<>0 AND t1.f1 = t2.f2; > +SELECT * FROM t1 STRAIGHT_JOIN t2 ON t2.f1 = t1.f1 > + WHERE t1.f1<>0 OR t1.f2<>0 AND t1.f1 = t2.f2; > + > +DROP TABLE t1,t2; > + > +CREATE TABLE t1(f1 int PRIMARY KEY, f2 int) ; > +INSERT INTO t1 VALUES (9,4), (10,9); > + > +CREATE TABLE t2(f1 int PRIMARY KEY, f2 int) ; > +INSERT INTO t2 VALUES (9,4), (10,9); > + > +EXPLAIN > +SELECT STRAIGHT_JOIN * FROM t1 JOIN t2 ON t2.f2 = t1.f1 > + WHERE t1.f1 IN (SELECT f1 FROM t1) AND t1.f1 = t2.f1 OR t1.f1 = 9; > +SELECT STRAIGHT_JOIN * FROM t1 JOIN t2 ON t2.f2 = t1.f1 > + WHERE t1.f1 IN (SELECT f1 FROM t1) AND t1.f1 = t2.f1 OR t1.f1 = 9; > + > +DROP TABLE t1,t2; > + > > === modified file 'sql/item_cmpfunc.cc' > --- a/sql/item_cmpfunc.cc 2011-02-06 04:57:03 +0000 > +++ b/sql/item_cmpfunc.cc 2011-03-12 08:49:03 +0000 > @@ -5723,7 +5723,8 @@ > not_null_tables_cache|= tmp_table_map; > if (item->maybe_null) > maybe_null=1; > - item->item_equal= this; > + if (!item->item_equal) > + item->item_equal= this; > } > fix_length_and_dec(); > fixed= 1; > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] [Commits] Rev 2934: Fix LP BUG#719198 in file:///home/tsk/mprog/src/5.3/
by Sergey Petrunya 08 Mar '11

08 Mar '11

Hi Timour, On Tue, Mar 08, 2011 at 04:35:24PM +0200, timour(a)askmonty.org wrote: > At file:///home/tsk/mprog/src/5.3/ > > ------------------------------------------------------------ > revno: 2934 > revision-id: timour(a)askmonty.org-20110308143512-0m7uotvxp9rmegak > parent: psergey(a)askmonty.org-20110304155430-l0ad4iag9gn5zeka > committer: timour(a)askmonty.org > branch nick: 5.3 > timestamp: Tue 2011-03-08 16:35:12 +0200 > message: > Fix LP BUG#719198 > > Analysis: > The assert failed because the execution code for > partial matching is designed with the assumption that > NULLs on the left side are detected as early as possible, > and a NULL result is returned before any lookups are > performed at all. > > However, in the case of an Item_cache object on the left > side, null was not detected properly, because detection > was done via Item::is_null(), which is not implemented at > all for Item_cache, and resolved to the default Item::is_null() > which always returns FALSE. > > Solution: > Use the property Item::null_value instead of is_null(), which > is properly updated for Item_cache objects as well. > > Alternatively one could implement Item_cache:is_null(), but > its not quite clear how that will interact with the cache > update methods, so the current fix seems simpler and cleaner. Ok to push. Since it is guaranteed that cache_value() have been called for the object, it's ok to use null_value instead of is_null(). Btw, out of interest I've implemented Item_cache::is_null() and t/subselect*.test ran successfully, so it seems to be ok to have. I don't insist that it is added as part of this fix, though. > === modified file 'mysql-test/r/subselect_mat.result' > --- a/mysql-test/r/subselect_mat.result 2011-01-14 10:51:30 +0000 > +++ b/mysql-test/r/subselect_mat.result 2011-03-08 14:35:12 +0000 > @@ -1371,3 +1371,41 @@ SELECT pk FROM t1 WHERE (b,c,d) IN (SELE > pk > 2 > DROP TABLE t1, t2; > +# > +# LPBUG#719198 Ordered_key::cmp_key_with_search_key(rownum_t): Assertion `!compare_pred[i]->null_value' > +# failed with subquery on both sides of NOT IN and materialization > +# > +CREATE TABLE t1 (f1a int, f1b int) ; > +INSERT IGNORE INTO t1 VALUES (1,1),(2,2); > +CREATE TABLE t2 ( f2 int); > +INSERT IGNORE INTO t2 VALUES (3),(4); > +CREATE TABLE t3 (f3a int, f3b int); > +set session optimizer_switch='materialization=on,partial_match_rowid_merge=on,partial_match_table_scan=off'; > +EXPLAIN > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3) NOT IN (SELECT f1a FROM t1); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY t2 ALL NULL NULL NULL NULL 2 Using where > +3 SUBQUERY t1 ALL NULL NULL NULL NULL 2 > +2 SUBQUERY t3 system NULL NULL NULL NULL 0 const row not found > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3) NOT IN (SELECT f1a FROM t1); > +f2 > +EXPLAIN > +SELECT * FROM t2 WHERE (SELECT f3a, f3b FROM t3) NOT IN (SELECT f1a, f1b FROM t1); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY t2 ALL NULL NULL NULL NULL 2 Using where > +3 SUBQUERY t1 ALL NULL NULL NULL NULL 2 > +2 SUBQUERY t3 system NULL NULL NULL NULL 0 const row not found > +SELECT * FROM t2 WHERE (SELECT f3a, f3b FROM t3) NOT IN (SELECT f1a, f1b FROM t1); > +f2 > +insert into t3 values (1,1),(2,2); > +EXPLAIN > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3 where f3a > 3) NOT IN (SELECT f1a FROM t1); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY t2 ALL NULL NULL NULL NULL 2 Using where > +3 SUBQUERY t1 ALL NULL NULL NULL NULL 2 > +2 SUBQUERY t3 ALL NULL NULL NULL NULL 2 Using where > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3 where f3a > 3) NOT IN (SELECT f1a FROM t1); > +f2 > +3 > +4 > +drop table t1, t2, t3; > > === modified file 'mysql-test/t/subselect_mat.test' > --- a/mysql-test/t/subselect_mat.test 2011-01-14 10:51:30 +0000 > +++ b/mysql-test/t/subselect_mat.test 2011-03-08 14:35:12 +0000 > @@ -1011,3 +1011,31 @@ SELECT pk FROM t1 WHERE (a) IN (SELECT a > SELECT pk FROM t1 WHERE (b,c,d) IN (SELECT b,c,d FROM t2 WHERE pk > 0); > DROP TABLE t1, t2; > > +--echo # > +--echo # LPBUG#719198 Ordered_key::cmp_key_with_search_key(rownum_t): Assertion `!compare_pred[i]->null_value' > +--echo # failed with subquery on both sides of NOT IN and materialization > +--echo # > + > +CREATE TABLE t1 (f1a int, f1b int) ; > +INSERT IGNORE INTO t1 VALUES (1,1),(2,2); > +CREATE TABLE t2 ( f2 int); > +INSERT IGNORE INTO t2 VALUES (3),(4); > +CREATE TABLE t3 (f3a int, f3b int); > + > +set session optimizer_switch='materialization=on,partial_match_rowid_merge=on,partial_match_table_scan=off'; > + > +EXPLAIN > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3) NOT IN (SELECT f1a FROM t1); > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3) NOT IN (SELECT f1a FROM t1); > + > +EXPLAIN > +SELECT * FROM t2 WHERE (SELECT f3a, f3b FROM t3) NOT IN (SELECT f1a, f1b FROM t1); > +SELECT * FROM t2 WHERE (SELECT f3a, f3b FROM t3) NOT IN (SELECT f1a, f1b FROM t1); > + > +insert into t3 values (1,1),(2,2); > + > +EXPLAIN > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3 where f3a > 3) NOT IN (SELECT f1a FROM t1); > +SELECT * FROM t2 WHERE (SELECT f3a FROM t3 where f3a > 3) NOT IN (SELECT f1a FROM t1); > + > +drop table t1, t2, t3; > > === modified file 'sql/item_subselect.cc' > --- a/sql/item_subselect.cc 2011-03-03 21:48:31 +0000 > +++ b/sql/item_subselect.cc 2011-03-08 14:35:12 +0000 > @@ -5069,7 +5069,7 @@ bool subselect_rowid_merge_engine::parti > for (uint i= test(non_null_key); i < keys_count; i++) > { > DBUG_ASSERT(merge_keys[i]->get_column_count() == 1); > - if (merge_keys[i]->get_search_key(0)->is_null()) > + if (merge_keys[i]->get_search_key(0)->null_value) > { > ++count_nulls_in_search_key; > bitmap_set_bit(&matching_outer_cols, merge_keys[i]->get_keyid()); > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MWL#90 combined patch for review
by Sergey Petrunya 08 Mar '11

08 Mar '11

Hi Monty, Please find attached the complete patch of MWL#90 for review. The tree with the code is on launchpad/buildbot: https://code.launchpad.net/~maria-captains/maria/5.3-subqueries-mwl90 http://buildbot.askmonty.org/buildbot/grid?branch=5.3-subqueries-mwl90 and it has no failures other than those also found in mainline 5.3. The merge from 5.3-main was about a week ago. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 5

[Maria-developers] Report from OpenSourceDays2011
by Kristian Nielsen 08 Mar '11

08 Mar '11

Hi, On Saturday, March 5, I gave a talk at the OpenSourceDays 2011 conference in Copenhagen, Denmark: http://opensourcedays.org/ My talk was titled "Latest news from the MariaDB (and MySQL) community". I put the slides from the talk up on my web server here: http://knielsen-hq.org/maria/osd2011.pdf This year the OpenSourceDays conference was reduced somewhat from what it usually is, only one day and I heard something like 400 tickes available (which were sold out rather quickly). Still, I had a good bunch of people at the talk, maybe 30 or so. There were many questions, people seemed interested, so I think it went well. I also talked to some people outside of the talk, again there seems to be a lot of interest in what is happening around MySQL generally and in MariaDB specifically. I think generally people are pleasantly surprised to learn the depth and dedication of the MariaDB effort, that we really are a major effort and a viable alternative. There was one good question at the talk about if there were reference customer statements or so to show that companies are deploying MariaDB with success at large scale; I guess this will be useful for people that want to sell the idea of using MariaDB at their job or institution or whatever. I did not have a very good answer to this, except that I knew there were some. I guess we already have this on one of the web sites, or if not it would be worth it to collect the different press releases and such as we get them, I think. The night before the conference there was a speaker's dinner, and I had a great talk with Magnus from Postgresql. I got the impression that things are also really moving in the Postgres space, and it seems to be picking up a lot of interest and development effort. I think that is great, I am starting to be hopeful that we are now seeing the open source databases really take off as a community effort, as opposed to just a project of whatever "acquirer of the year". That's it :-) - Kristian.

2 1

[Maria-developers] Pbxt loses rows
by Michael Widenius 08 Mar '11

08 Mar '11

Hi! This is a bug that has been around a long time but not yet fixed (at least not in 1.0.11 which is in the MariaDB 5.1 tree). Once in a while when running the pbxt.insert test, pbxt fails to find all rows in the table: pbxt.insert [ fail ] Test ended at 2011-02-25 16:44:03 CURRENT_TEST: pbxt.insert --- /home/my/maria-5.1-merge/mysql-test/suite/pbxt/r/insert.result 2011-02-10 22:57:21.000000000 +0200 +++ /home/my/maria-5.1-merge/mysql-test/suite/pbxt/r/insert.reject 2011-02-25 16:44:03.030856755 +0200 @@ -298,7 +298,7 @@ insert into t2 select t1.* from t1, t2 t, t3 where t1.id1 = t.id2 and t.id2 = t3.id3; select count(*) from t2; count(*) -25500 +25495 drop table t1,t2,t3; create table t1 (a int, b int); insert into t1 (a,b) values (a,b); Rerunning the test usually works It would be really good to get this fixes ASAP as it's not good if a storage engines looses rows randomly... The above happend in my latest 5.1 tree on Linux. Regards, Monty

2 2

[Maria-developers] adutko-centos-amd64
by Adam M. Dutko 07 Mar '11

07 Mar '11

The machine running the buildbot adutko-centos-amd64 experienced a hardware failure last night around 1AM EST (UTC 0600). I believe it is the motherboard and due to funding constraints I am not certain when it will be brought back online. I will make an attempt to migrate the buildbot when I have time over the next few days. Sorry for the interruption. If someone has a spare board that supports AMD Phenom X3/X4 processors please write me off list. Thanks.

1 0

Re: [Maria-developers] RedHat Buildslaves
by Kristian Nielsen 07 Mar '11

07 Mar '11

Daniel Bartholomew <dbart(a)montyprogram.com> writes: > Daniel> I'm creating the build-slaves based on the KVM setup > Daniel> instructions you have here: > Daniel> > Daniel> http://askmonty.org/wiki/BuildBot:vm-setup#Centos_5_i386 > I now have a rhel6-amd64 vm setup on terrier with base, serial, build, > and install vms. The steps I went through to set it up are here: > > http://kb.askmonty.org/v/buildbot-setup-for-red-hat > I'm not sure what the next step should be. Probably copying over the > correct OurDelta file. Can you copy one out After that, do you want to > take a look at the VMs and see if they're ok for building and install > testing? I did this, and set up new builders in Buildbot. They fail, but the failure looks related to the .spec file and build scripts, rather than the VMs. So I would say the VMs are now ok, and the next step is to find someone to modify the ourdelta build scripts and/or spec files to work with RHEL. > Once this VM is certifed as good to use as a buildbot buildslave I'll > move on to rhel4 and rhel5. As I said, the VMs look reasonable. So if you want, you can start doing the rhel4/5 ones the same way, or if you prefer, you can wait until the .spec files are done for rhel6, in case that work turns up something else that is needed. > One thing I just realized is that > the /kvm/MySQL-OurDelta-shared-5.0.87.d10-65.el5.i386.rpm file that I > placed at /srv/shared/yum/CentOS/5/i386/RPMS/ is not the right > architecture for an amd64 system, and it's probably not at the right > path, either (what with CentOS being in the path). Yeah. I grabbed the amd64 one and copied it in there with my best guess for the path (and also put it on terrier:/kvm/). The thing is, it probably needs more than this, someone who understands (or takes the time to learn) .rpm will have to develop the right solution. The full story is like this: The Redhat rpms (original, not ours) put the `mysql` binary in the same package as libmysqlclient. Then it has other packages depend on this package to get libmysqlclient.so.15 (old version of client library). So when we replace the package with a new version of the mysql binary (from MariaDB), we also have to provide a copy of the _old_ client library (as well as a copy of the new one of course, libmysqlclient.so.16). (In contrast, Debian does this correctly, placing the client library in package libmysqlclient15, which can co-exist with new MySQL/MariaDB versions). The question is how to get this old library? The MySQL build team told me they grab it from an old binary package from MySQL 4.1 (or whatever), so the OurDelta stuff does something similar. Admittedly not a nice solution. Maybe it is not needed for RHEL 6, if it does not have dependencies on libmysqlclient15. Or maybe we need to find an old RHEL MySQL .rpm to use instead. I do not know rpm well enough to have a meaningful opinion. > I built this directly on terrier, since my local desktop can only run > with "-cpu qemu32". I tried to build an x86 version on terrier, but I > kept getting stuck in a VM reboot loop. So I'll have to upload the Yeah, there is something wierd/different with the qemu-kvm I built on terrier to fix another problem. But it works with `-cpu qemu64` (also for a 32-bit VM). Doing it on your local machine should be fine as well. Thanks, - Kristian.

1 0

[Maria-developers] Igor please review: Rev 2933: BUG#727667 Wrong result with OR + NOT NULL in maria-5.3 in file:///home/psergey/dev2/5.3/]
by Sergey Petrunya 05 Mar '11

05 Mar '11

Hello Igor, Could you please review the below? The bug page https://bugs.launchpad.net/maria/+bug/727667 has verbose explanations of how we ended up having this bug, as well as justifications for fixing it in this particular way. ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- From: Sergey Petrunya <psergey(a)askmonty.org> To: commits(a)mariadb.org X-Mailer: mail (GNU Mailutils 1.2) Date: Sat, 5 Mar 2011 12:56:26 +0300 (MSK) Subject: [Commits] Rev 2933: BUG#727667 Wrong result with OR + NOT NULL in maria-5.3 in file:///home/psergey/dev2/5.3/ At file:///home/psergey/dev2/5.3/ ------------------------------------------------------------ revno: 2933 revision-id: psergey(a)askmonty.org-20110305095622-wzlnwr9srfbsu3pv parent: psergey(a)askmonty.org-20110304091446-qmsezak3c0cfm9b1 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: 5.3 timestamp: Sat 2011-03-05 12:56:22 +0300 message: BUG#727667 Wrong result with OR + NOT NULL in maria-5.3 - put the code that sets HA_NULL_PART bit back - Fix test_if_ref/part_of_refkey() so that = NULL-ability of lookup columns does not prevent the equality from being removed (we now have early/late NULLs filtering which will filter out NULL values) = equality is not removed if it is ref_or_null access, and the value of the lookup column can alternate between the lookup value and NULL. === modified file 'mysql-test/r/null.result' --- a/mysql-test/r/null.result 2009-12-15 07:16:46 +0000 +++ b/mysql-test/r/null.result 2011-03-05 09:56:22 +0000 @@ -170,7 +170,7 @@ insert into t1 values(null); explain select * from t1 where i=2 or i is null; id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null i i 5 const 9 Using index +1 SIMPLE t1 ref_or_null i i 5 const 9 Using where; Using index select count(*) from t1 where i=2 or i is null; count(*) 10 === modified file 'mysql-test/r/null_key.result' --- a/mysql-test/r/null_key.result 2010-10-06 20:27:12 +0000 +++ b/mysql-test/r/null_key.result 2011-03-05 09:56:22 +0000 @@ -21,10 +21,10 @@ 1 SIMPLE t1 range a,b a 9 NULL 3 Using where; Using index explain select * from t1 where (a is null or a = 7) and b=7; id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null a,b a 9 const,const 2 Using index +1 SIMPLE t1 ref_or_null a,b a 9 const,const 2 Using where; Using index explain select * from t1 where (a is null or a = 7) and b=7 order by a; id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null a,b a 9 const,const 2 Using index; Using filesort +1 SIMPLE t1 ref_or_null a,b a 9 const,const 2 Using where; Using index; Using filesort explain select * from t1 where (a is null and b>a) or a is null and b=7 limit 2; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t1 ref a,b a 5 const 3 Using where; Using index @@ -151,7 +151,7 @@ insert into t1 values (7,null), (8,null), (8,7); explain select * from t1 where a = 7 and (b=7 or b is null); id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null a,b a 10 const,const 2 Using index +1 SIMPLE t1 ref_or_null a,b a 10 const,const 2 Using where; Using index select * from t1 where a = 7 and (b=7 or b is null); a b 7 7 @@ -166,7 +166,7 @@ NULL 7 explain select * from t1 where (a = 7 or a is null) and (a = 7 or a is null); id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null a a 5 const 5 Using index +1 SIMPLE t1 ref_or_null a a 5 const 5 Using where; Using index select * from t1 where (a = 7 or a is null) and (a = 7 or a is null); a b 7 NULL @@ -192,7 +192,7 @@ explain select * from t2,t1 where t1.a=t2.a and (b= 7 or b is null); id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t2 ALL NULL NULL NULL NULL 2 Using where -1 SIMPLE t1 ref_or_null a a 10 test.t2.a,const 4 Using index +1 SIMPLE t1 ref_or_null a a 10 test.t2.a,const 4 Using where; Using index select * from t2,t1 where t1.a=t2.a and (b= 7 or b is null); a a b 7 7 7 @@ -202,7 +202,7 @@ explain select * from t2,t1 where (t1.a=t2.a or t1.a is null) and b= 7; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t2 ALL NULL NULL NULL NULL 2 -1 SIMPLE t1 ref_or_null a a 10 test.t2.a,const 4 Using index +1 SIMPLE t1 ref_or_null a a 10 test.t2.a,const 4 Using where; Using index select * from t2,t1 where (t1.a=t2.a or t1.a is null) and b= 7; a a b 7 7 7 @@ -226,7 +226,7 @@ explain select * from t2,t1 where t1.a=t2.a or t1.a is null; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t2 ALL NULL NULL NULL NULL 4 -1 SIMPLE t1 ref_or_null a a 5 test.t2.a 4 Using index +1 SIMPLE t1 ref_or_null a a 5 test.t2.a 4 Using where; Using index explain select * from t2,t1 where t1.a<=>t2.a or (t1.a is null and t1.b <> 9); id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t2 ALL NULL NULL NULL NULL 4 @@ -447,3 +447,15 @@ 3 12 0 12 drop table t1, t2; End of 5.0 tests +# +# BUG#727667 Wrong result with OR + NOT NULL in maria-5.3 +# +CREATE TABLE t1 ( +f3 int(11), +f10 varchar(1), +KEY (f3) +); +INSERT INTO t1 VALUES ('9','k'),(NULL,'r'); +SELECT * FROM t1 WHERE (f3 = 83) OR (f10 = 'z' AND f3 IS NULL); +f3 f10 +DROP TABLE t1; === modified file 'mysql-test/r/order_by.result' --- a/mysql-test/r/order_by.result 2011-01-27 04:45:23 +0000 +++ b/mysql-test/r/order_by.result 2011-03-05 09:56:22 +0000 @@ -643,7 +643,7 @@ insert into t1 values (2, 1), (1, 1), (4, NULL), (3, NULL), (6, 2), (5, 2); explain select * from t1 where b=1 or b is null order by a; id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null b b 5 const 3 Using filesort +1 SIMPLE t1 ref_or_null b b 5 const 3 Using index condition; Using filesort select * from t1 where b=1 or b is null order by a; a b 1 1 @@ -652,7 +652,7 @@ 4 NULL explain select * from t1 where b=2 or b is null order by a; id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE t1 ref_or_null b b 5 const 4 Using filesort +1 SIMPLE t1 ref_or_null b b 5 const 4 Using index condition; Using filesort select * from t1 where b=2 or b is null order by a; a b 3 NULL === modified file 'mysql-test/r/subselect.result' --- a/mysql-test/r/subselect.result 2011-02-22 09:15:47 +0000 +++ b/mysql-test/r/subselect.result 2011-03-05 09:56:22 +0000 @@ -906,7 +906,7 @@ explain extended SELECT t1.a, t1.a in (select t2.a from t2,t3 where t3.a=t2.a) FROM t1; id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 index NULL PRIMARY 4 NULL 4 100.00 Using index -2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using index +2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using where; Using index 2 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 3 100.00 Using where; Using join buffer (flat, BNL join) Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,<expr_cache><`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`a`,<exists>(select 1 from `test`.`t2` join `test`.`t3` where ((`test`.`t3`.`a` = `test`.`t2`.`a`) and ((<cache>(`test`.`t1`.`a`) = `test`.`t2`.`a`) or isnull(`test`.`t2`.`a`))) having <is_not_null_test>(`test`.`t2`.`a`)))) AS `t1.a in (select t2.a from t2,t3 where t3.a=t2.a)` from `test`.`t1` === modified file 'mysql-test/r/subselect_no_mat.result' --- a/mysql-test/r/subselect_no_mat.result 2011-02-22 09:15:47 +0000 +++ b/mysql-test/r/subselect_no_mat.result 2011-03-05 09:56:22 +0000 @@ -910,7 +910,7 @@ explain extended SELECT t1.a, t1.a in (select t2.a from t2,t3 where t3.a=t2.a) FROM t1; id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 index NULL PRIMARY 4 NULL 4 100.00 Using index -2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using index +2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using where; Using index 2 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 3 100.00 Using where; Using join buffer (flat, BNL join) Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,<expr_cache><`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`a`,<exists>(select 1 from `test`.`t2` join `test`.`t3` where ((`test`.`t3`.`a` = `test`.`t2`.`a`) and ((<cache>(`test`.`t1`.`a`) = `test`.`t2`.`a`) or isnull(`test`.`t2`.`a`))) having <is_not_null_test>(`test`.`t2`.`a`)))) AS `t1.a in (select t2.a from t2,t3 where t3.a=t2.a)` from `test`.`t1` === modified file 'mysql-test/r/subselect_no_opts.result' --- a/mysql-test/r/subselect_no_opts.result 2011-02-22 09:15:47 +0000 +++ b/mysql-test/r/subselect_no_opts.result 2011-03-05 09:56:22 +0000 @@ -907,7 +907,7 @@ explain extended SELECT t1.a, t1.a in (select t2.a from t2,t3 where t3.a=t2.a) FROM t1; id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 index NULL PRIMARY 4 NULL 4 100.00 Using index -2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using index +2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using where; Using index 2 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 3 100.00 Using where; Using join buffer (flat, BNL join) Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,<expr_cache><`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`a`,<exists>(select 1 from `test`.`t2` join `test`.`t3` where ((`test`.`t3`.`a` = `test`.`t2`.`a`) and ((<cache>(`test`.`t1`.`a`) = `test`.`t2`.`a`) or isnull(`test`.`t2`.`a`))) having <is_not_null_test>(`test`.`t2`.`a`)))) AS `t1.a in (select t2.a from t2,t3 where t3.a=t2.a)` from `test`.`t1` === modified file 'mysql-test/r/subselect_no_semijoin.result' --- a/mysql-test/r/subselect_no_semijoin.result 2011-02-22 09:15:47 +0000 +++ b/mysql-test/r/subselect_no_semijoin.result 2011-03-05 09:56:22 +0000 @@ -907,7 +907,7 @@ explain extended SELECT t1.a, t1.a in (select t2.a from t2,t3 where t3.a=t2.a) FROM t1; id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 index NULL PRIMARY 4 NULL 4 100.00 Using index -2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using index +2 DEPENDENT SUBQUERY t2 ref_or_null a a 5 func 2 100.00 Using where; Using index 2 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 3 100.00 Using where; Using join buffer (flat, BNL join) Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,<expr_cache><`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`a`,<exists>(select 1 from `test`.`t2` join `test`.`t3` where ((`test`.`t3`.`a` = `test`.`t2`.`a`) and ((<cache>(`test`.`t1`.`a`) = `test`.`t2`.`a`) or isnull(`test`.`t2`.`a`))) having <is_not_null_test>(`test`.`t2`.`a`)))) AS `t1.a in (select t2.a from t2,t3 where t3.a=t2.a)` from `test`.`t1` === modified file 'mysql-test/t/null_key.test' --- a/mysql-test/t/null_key.test 2008-03-03 17:35:44 +0000 +++ b/mysql-test/t/null_key.test 2011-03-05 09:56:22 +0000 @@ -263,3 +263,16 @@ drop table t1, t2; -- echo End of 5.0 tests +--echo # +--echo # BUG#727667 Wrong result with OR + NOT NULL in maria-5.3 +--echo # + +CREATE TABLE t1 ( + f3 int(11), + f10 varchar(1), + KEY (f3) +); +INSERT INTO t1 VALUES ('9','k'),(NULL,'r'); +SELECT * FROM t1 WHERE (f3 = 83) OR (f10 = 'z' AND f3 IS NULL); +DROP TABLE t1; + === modified file 'sql/sql_select.cc' --- a/sql/sql_select.cc 2011-02-28 21:29:59 +0000 +++ b/sql/sql_select.cc 2011-03-05 09:56:22 +0000 @@ -6564,10 +6564,12 @@ j->ref.has_record= FALSE; j->ref.null_rejecting= 0; j->ref.disable_cache= FALSE; + j->ref.null_ref_part= (uint)-1; keyuse=org_keyuse; store_key **ref_key= j->ref.key_copy; uchar *key_buff=j->ref.key_buff, *null_ref_key= 0; + uint null_ref_part= (uint)-1; bool keyuse_uses_no_tables= TRUE; if (ftkey) { @@ -6621,7 +6623,10 @@ instead of JT_REF_OR_NULL in case if field can't be null */ if ((keyuse->optimize & KEY_OPTIMIZE_REF_OR_NULL) && maybe_null) + { null_ref_key= key_buff; + null_ref_part= i; + } key_buff+= keyinfo->key_part[i].store_length; } } /* not ftkey */ @@ -6636,6 +6641,7 @@ /* Must read with repeat */ j->type= null_ref_key ? JT_REF_OR_NULL : JT_REF; j->ref.null_ref_key= null_ref_key; + j->ref.null_ref_part= null_ref_part; } else if (keyuse_uses_no_tables) { @@ -15287,8 +15293,12 @@ *****************************************************************************/ /** + Check if "left_item=right_item" equality is guaranteed to be true by use of + [eq]ref access on left_item->field->table. + @return - 1 if right_item is used removable reference key on left_item + TRUE if right_item is used removable reference key on left_item + FALSE otherwise */ bool test_if_ref(Item *root_cond, Item_field *left_item,Item *right_item) @@ -15301,7 +15311,11 @@ (!join_tab->first_inner || *join_tab->first_inner->on_expr_ref == root_cond)) { - // Cond guards + /* + If ref access uses "Full scan on NULL key" (i.e. it actually alternates + between ref access and full table scan), then no equality can be + guaranteed to be true. + */ for (uint i = 0; i < join_tab->ref.key_parts; i++) { if (join_tab->ref.cond_guards[i]) @@ -15309,7 +15323,7 @@ return FALSE; } } - // + Item *ref_item=part_of_refkey(field->table,field); if (ref_item && ref_item->eq(right_item,1)) { @@ -15611,6 +15625,29 @@ } +/* + @brief + + Check if + - @table uses "ref"-like access + - it is based on "@field=certain_item" equality + - the equality will be true for any record returned by the access method + and return the certain_item if yes. + + @detail + + Equality won't necessarily hold if: + - the used index covers only part of the @field. + Suppose, we have a CHAR(5) field and INDEX(field(3)). if you make a lookup + for 'abc', you will get both record with 'abc' and with 'abcde'. + - The type of access is actually ref_or_null, and so @field can be either + a value or NULL. + + @return + Item that the field will be equal to + NULL if no such item +*/ + static Item * part_of_refkey(TABLE *table,Field *field) { @@ -15619,17 +15656,30 @@ return (Item*) 0; // field from outer non-select (UPDATE,...) uint ref_parts= join_tab->ref.key_parts; - if (ref_parts) + if (ref_parts) /* if it's ref/eq_ref/ref_or_null */ { - uint key= join_tab->ref.key; KEY *key_info= join_tab->get_keyinfo_by_key_no(key); KEY_PART_INFO *key_part= key_info->key_part; for (uint part=0 ; part < ref_parts ; part++,key_part++) - if (field->eq(key_part->field) && - !(key_part->key_part_flag & (HA_PART_KEY_SEG | HA_NULL_PART))) - return join_tab->ref.items[part]; + { + if (field->eq(key_part->field)) + { + /* + Found the field in the key. Check that + 1. ref_or_null doesn't alternate this component between a value and + a NULL + 2. index fully covers the key + */ + if (part != join_tab->ref.null_ref_part && // (1) + !(key_part->key_part_flag & HA_PART_KEY_SEG)) // (2) + { + return join_tab->ref.items[part]; + } + break; + } + } } return (Item*) 0; } === modified file 'sql/sql_select.h' --- a/sql/sql_select.h 2011-02-24 06:23:12 +0000 +++ b/sql/sql_select.h 2011-03-05 09:56:22 +0000 @@ -109,8 +109,15 @@ */ key_part_map null_rejecting; table_map depend_map; ///< Table depends on these tables. + /* null byte position in the key_buf. Used for REF_OR_NULL optimization */ uchar *null_ref_key; + /* + ref_or_null optimization: number of key part can be NULL (there's only one) + uint(-1) if no part can be NULL. + */ + uint null_ref_part; + /* The number of times the record associated with this key was used in the join. === modified file 'sql/table.cc' --- a/sql/table.cc 2011-02-22 09:15:47 +0000 +++ b/sql/table.cc 2011-03-05 09:56:22 +0000 @@ -1581,6 +1581,8 @@ #endif key_part->key_part_flag|= HA_PART_KEY_SEG; } + if (field->real_maybe_null()) + key_part->key_part_flag|= HA_NULL_PART; /* Sometimes we can compare key parts for equality with memcmp. But not always. _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB 5.2.5
by Daniel Bartholomew 03 Mar '11

03 Mar '11

Here are the download, release notes, and changelog pages for 5.2.5: * http://downloads.askmonty.org/mariadb/5.2.5 * http://kb.askmonty.org/v/mariadb-525-release-notes * http://kb.askmonty.org/v/mariadb-525-changelog If any of you have a moment, please check over the above pages to see if there are any mistakes or omissions and let me know (or jump in and fix if you have editing rights). Community team: I tested a sample of the mirrors and they all had the files so please announce the release on the announce list, blogs, and so on. Thanks. -- Daniel Bartholomew MariaDB - http://mariadb.org Monty Program - http://montyprogram.com AskMonty Knowledgebase - http://kb.askmonty.org

2 2

Re: [Maria-developers] [Commits] Rev 2929: Fix LP BUG#718763 in file:///home/tsk/mprog/src/5.3/
by Oleksandr Byelkin 03 Mar '11

03 Mar '11

Hi! Everything looks OK, except one small style error. Also it would be nice to put somewhere 'TODO:' to bring in order 'uncachable' flags. On 03.03.2011 14:07, timour(a)askmonty.org wrote: > At file:///home/tsk/mprog/src/5.3/ > > ------------------------------------------------------------ > revno: 2929 > revision-id: timour(a)askmonty.org-20110303120726-la5o40rldim2t93y > parent: psergey(a)askmonty.org-20110301072222-1rmapkelx9l1kw8u > committer: timour(a)askmonty.org > branch nick: 5.3 > timestamp: Thu 2011-03-03 14:07:26 +0200 > message: > Fix LP BUG#718763 > > Analysis: > The reason for the crash was that the inner subquery was executed > via a scan on a final temporary table applied after all other > operations. This final operation is implemented by changing the > contents of the JOIN object of the subquery to represent a table > scan over the temp table. At the same time query optimization of > the outer subquery required evaluation of the inner subquery, which > happened before the actual EXPLAIN. The evaluation left the JOIN > object of the inner subquery in the changed state, where it represented > a table scan over a temp table, and EXPLAIN crashed because the temp > table is not associated with any table reference (TABLE_LIST object). > The reason the JOIN was not restored was because its saving/restoration > was controlled by the join->select_lex->uncacheable flag, which was > not set in the case of materialization. > > Solution: > In the methods Item_in_subselect::[single | row]_value_transformer() set: > select_lex->uncacheable|= UNCACHEABLE_EXPLAIN; > In addition, for symmetry, change: > master_unit->uncacheable|= UNCACHEABLE_EXPLAIN; > instead of UNCACHEABLE_DEPENDENT because if a subquery was not > dependent initially, the changed methods do not change this > fact. The subquery may later become correlated if it is transformed > to an EXISTS query, but it may stay uncorrelated if executed via > materialization. [skip] > > > @@ -1742,7 +1752,15 @@ Item_in_subselect::row_value_transformer > optimizer->keep_top_level_cache(); > > thd->lex->current_select= current; > - master_unit->uncacheable|= UNCACHEABLE_DEPENDENT; > + /* > + The uncacheable property controls a number of actions, e.g. whether to > + save/restore (via init_save_join_tab/restore_tmp) the original JOIN for > + plans with a temp table where the original JOIN was overriden by > + make_simple_join. The UNCACHEABLE_EXPLAIN is ignored by EXPLAIN, thus > + non-correlated subqueries will not appear as such to EXPLAIN. > + */ > + master_unit->uncacheable|=UNCACHEABLE_EXPLAIN; > + select_lex->uncacheable|= UNCACHEABLE_EXPLAIN; > > if (!abort_on_null&& left_expr->maybe_null&& !pushed_cond_guards) > { put space (' ') after '=' [skip] > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] MariaDB 5.2.5 packages ready for release
by Kristian Nielsen 02 Mar '11

02 Mar '11

From: Kristian Nielsen <knielsen(a)knielsen-hq.org> Subject: Release of MariaDB 5.1.55 To: Daniel Bartholomew <dbart(a)askmonty.org> Cc: Internal Dev List <dev(a)askmonty.org> Date: Mon, 28 Feb 2011 12:40:13 +0100 Hi Daniel, The packages for release of MariaDB 5.2.5 are ready. The main changes I know of are the same changes as in the MariaDB 5.1.55 release: merges of MySQL 5.1.55 and of XtraDB from Percona Server 5.1.54-12.5. This version was released off the main "5.2" branch: lp:maria/5.2 I added the mariadb-5.2.5 tag. The Buildbot build number, for locating packages in /archive/pack, is 829. As for the 5.1.55 release, I prepared apt repositories and signed .rpm files. The repositories are here: hasky:/home/knielsen/repo The signed .rpms are here: hasky:/home/knielsen/build-829 - Kristian.

1 0

Re: [Maria-developers] Fwd: [Commits] Rev 2901: BNLH algorithm always used a full table scan over the joined table in file:///home/igor/maria/maria-5.3-mwl128-hashrange/
by Sergey Petrunya 02 Mar '11

02 Mar '11

Hello Igor, Here's the promised proof of deadcode: On Fri, Feb 11, 2011 at 08:49:37PM -0800, Igor Babaev wrote: > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2011-02-06 04:57:03 +0000 > +++ b/sql/sql_select.cc 2011-02-12 04:41:22 +0000 > @@ -8419,6 +8438,11 @@ > sort_by_tab->type= JT_ALL; > sort_by_tab->read_first_record= join_init_read_record; > } > + else if (sort_by_tab->type == JT_HASH_NEXT) > + { > + sort_by_tab->type= JT_HASH; > + sort_by_tab->read_first_record= join_init_read_record; > + } > } > break; > } Statement: the above else-if branch can never be taken, i.e. equality sort_by_tab->type == JT_HASH_NEXT (1) can never be true. Proof: First, let's see what JOIN_TABs can be in sort_by_tab. sort_by_tab is assigned in this statement: JOIN_TAB *sort_by_tab= join->group && join->simple_group && join->group_list ? join->join_tab+join->const_tables : join->get_sort_by_join_tab(); i.e. possible values are 1. join->join_tab+join->const_tables 2. join->get_sort_by_join_tab() get_sort_by_join_tab is defined in sql_select.h as: JOIN_TAB *get_sort_by_join_tab() { return (need_tmp || !sort_by_table || skip_sort_order || ((group || tmp_table_param.sum_func_count) && !group_list)) ? NULL : join_tab+const_tables; } i.e. it can return: 1. NULL (which is not interesting) 2. join->join_tab + const_tables. this draws us to conclusion that sort_by_tab can be - NULL - join->join_tab + const_tables. so equality (1) can be rewritten as join->join_tab[const_tables].type == JT_HASH_NEXT (2) Now, let's explore the question, what join_tab elements can have type==JT_HASH_NEXT. The value JT_HASH_NEXT is assigned in only one place: sql_select.cc|8399| tab->type= tab->type == JT_ALL ? JT_NEXT : JT_HASH_NEXT; this line is inside the case block of code: break; case JT_ALL: case JT_HASH: .... tab->type= tab->type == JT_ALL ? JT_NEXT : JT_HASH_NEXT; which means that tab->type can become JT_HASH_NEXT only if it was JT_HASH before. Let's see when a join_tab can get type==JT_HASH. There's only one assignment that can assign type=JT_HASH. It is at sql_select.cc:8279 : if (tab->cache && tab->cache->get_join_alg() == JOIN_CACHE::BNLH_JOIN_ALG) tab->type= JT_HASH; i.e. tab->type can become JT_HASH only if tab->cache!=NULL. tab->cache can be set to non-NULL value only inside check_join_cache_usage(). However, the first lines in check_join_cache_usage() have this code: uint i= tab - join->join_tab; join->return_tab= 0; if (cache_level == 0 || i == join->const_tables || !prev_tab) return 0; from which we can conclude that join->join_tab[join->const_tables].cache == NULL will always hold. Going back on our chain of reasoning, we get that join->join_tab[join->const_tables].cache == NULL at all times=> join->join_tab[join->const_tables].type != JT_HASH at all times => join->join_tab[join->const_tables].type != JT_HASH_NEXT at all times => equality (2) is false always => original equality (1) is false always, q.e.d. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Igor please review: BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) in file:///home/psergey/dev2/5.3/]
by Sergey Petrunya 02 Mar '11

02 Mar '11

On Wed, Mar 02, 2011 at 08:04:39AM -0800, Igor Babaev wrote: > > +SELECT alias2.f1 AS field1 , alias1.f2 AS field2 > > +FROM t1 AS alias1 > > +LEFT OUTER JOIN t1 AS alias2 ON alias1.f2 = alias2.f5 > > +WHERE alias2.f3 > 'k' > > +AND alias2.f3 <= 'z' > > +OR alias2.f4 IN ('o', 'y') > > +AND alias2.f4 >= 'g' AND alias2.f4 < 'z'; > > +field1 field2 > > +Warnings: > > +Warning 1292 Truncated incorrect DOUBLE value: 'k' > > Why do we need comparison of a string with an integer that generates the > above warning? Can we do without it? Why not to use char(4) instead > if integer? Got rid of string vs integer comparison, as well as simplified the testcase a bit (further simplification is difficult). Please find the result below: ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- From: Sergey Petrunya <psergey(a)askmonty.org> To: commits(a)mariadb.org X-Mailer: mail (GNU Mailutils 1.2) Date: Wed, 2 Mar 2011 17:06:15 +0300 (MSK) Subject: [Commits] Rev 2929: BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) in file:///home/psergey/dev2/5.3/ At file:///home/psergey/dev2/5.3/ ------------------------------------------------------------ revno: 2929 revision-id: psergey(a)askmonty.org-20110302140613-4qt8xvdvxs4t4m98 parent: psergey(a)askmonty.org-20110301072222-1rmapkelx9l1kw8u committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: 5.3 timestamp: Wed 2011-03-02 17:06:13 +0300 message: BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) - Add debugging aid which maps MRR's range_ids (which are pointers so are different on every execution and hard to track) to range numbers (which allow much easier debugging). The mapper is not compiled unless the developer puts a #define into code to request it. - Fix the bug: the problem was that Mrr_ordered_index_reader's interrupt_read() and resume_read() would save and restore 1) index tuple 2) the rowid (as bytes returned by handler->position()). Clustered primary key columns were not saved/restored. They are not explicitly present in the index tuple (i.e. table->key_info[secondary_key].key_parts doesn't list them), but they are actually there, in particular table->field[clustered_primary_key_member].part_of_key(secondary_key) == 1. Index condition pushdown code [correctly] uses the latter as inidication that pushed index condition can refer to clustered PK members. The fix was to make interrupt_read()/resume_read() to save/restore clustered primary key members as well, so that we get correct values for them when evaluating pushed index condition. [Second attempt, make the code compile without the debugging aid] === modified file 'mysql-test/r/innodb_mrr.result' --- a/mysql-test/r/innodb_mrr.result 2011-01-17 21:26:04 +0000 +++ b/mysql-test/r/innodb_mrr.result 2011-03-02 14:06:13 +0000 @@ -736,3 +736,31 @@ c1 c2 c3 08:29:45 NULL 2009-02-01 drop table `t1`; +# +# BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) +# +set @_save_join_cache_level= @@join_cache_level; +set join_cache_level = 6; +CREATE TABLE t1 ( +f1 int(11), f2 int(11), f3 varchar(1), f4 varchar(1), +PRIMARY KEY (f1), +KEY (f3), +KEY (f2) +) ENGINE=InnoDB; +INSERT INTO t1 VALUES ('11','8','f','f'),('12','5','v','v'),('13','8','s','s'), +('14','8','a','a'),('15','6','p','p'),('16','7','z','z'),('17','2','a','a'), +('18','5','h','h'),('19','7','h','h'),('20','2','v','v'),('21','9','v','v'), +('22','142','b','b'),('23','3','y','y'),('24','0','v','v'),('25','3','m','m'), +('26','5','z','z'),('27','9','n','n'),('28','1','d','d'),('29','107','a','a'); +select count(*) from ( +SELECT alias1.f2 +FROM +t1 AS alias1 JOIN ( +t1 AS alias2 FORCE KEY (f3) JOIN +t1 AS alias3 FORCE KEY (f2) ON alias3.f2 = alias2.f2 AND alias3.f4 = alias2.f3 +) ON alias3.f1 <= alias2.f1 +) X; +count(*) +361 +set join_cache_level=@_save_join_cache_level; +drop table t1; === modified file 'mysql-test/t/innodb_mrr.test' --- a/mysql-test/t/innodb_mrr.test 2011-01-17 21:26:04 +0000 +++ b/mysql-test/t/innodb_mrr.test 2011-03-02 14:06:13 +0000 @@ -426,3 +426,31 @@ SELECT * FROM t1 WHERE c2 <=> NULL ORDER BY c2 LIMIT 2; drop table `t1`; +--echo # +--echo # BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) +--echo # +set @_save_join_cache_level= @@join_cache_level; +set join_cache_level = 6; +CREATE TABLE t1 ( + f1 int(11), f2 int(11), f3 varchar(1), f4 varchar(1), + PRIMARY KEY (f1), + KEY (f3), + KEY (f2) +) ENGINE=InnoDB; +INSERT INTO t1 VALUES ('11','8','f','f'),('12','5','v','v'),('13','8','s','s'), +('14','8','a','a'),('15','6','p','p'),('16','7','z','z'),('17','2','a','a'), +('18','5','h','h'),('19','7','h','h'),('20','2','v','v'),('21','9','v','v'), +('22','142','b','b'),('23','3','y','y'),('24','0','v','v'),('25','3','m','m'), +('26','5','z','z'),('27','9','n','n'),('28','1','d','d'),('29','107','a','a'); + +select count(*) from ( + SELECT alias1.f2 + FROM + t1 AS alias1 JOIN ( + t1 AS alias2 FORCE KEY (f3) JOIN + t1 AS alias3 FORCE KEY (f2) ON alias3.f2 = alias2.f2 AND alias3.f4 = alias2.f3 + ) ON alias3.f1 <= alias2.f1 +) X; + +set join_cache_level=@_save_join_cache_level; +drop table t1; === modified file 'sql/multi_range_read.cc' --- a/sql/multi_range_read.cc 2011-02-18 22:31:01 +0000 +++ b/sql/multi_range_read.cc 2011-03-02 14:06:13 +0000 @@ -387,15 +387,25 @@ bool Mrr_ordered_index_reader::set_interruption_temp_buffer(uint rowid_length, uint key_len, + uint saved_pk_len, uchar **space_start, uchar *space_end) { - if (space_end - *space_start <= (ptrdiff_t)(rowid_length + key_len)) + if (space_end - *space_start <= (ptrdiff_t)(rowid_length + key_len + saved_pk_len)) return TRUE; support_scan_interruptions= TRUE; saved_rowid= *space_start; *space_start += rowid_length; + + if (saved_pk_len) + { + saved_primary_key= *space_start; + *space_start += saved_pk_len; + } + else + saved_primary_key= NULL; + saved_key_tuple= *space_start; *space_start += key_len; @@ -406,17 +416,25 @@ void Mrr_ordered_index_reader::set_no_interruption_temp_buffer() { support_scan_interruptions= FALSE; - saved_key_tuple= saved_rowid= NULL; /* safety */ + saved_key_tuple= saved_rowid= saved_primary_key= NULL; /* safety */ have_saved_rowid= FALSE; } void Mrr_ordered_index_reader::interrupt_read() { DBUG_ASSERT(support_scan_interruptions); + TABLE *table= file->get_table(); /* Save the current key value */ - key_copy(saved_key_tuple, file->get_table()->record[0], - &file->get_table()->key_info[file->active_index], + key_copy(saved_key_tuple, table->record[0], + &table->key_info[file->active_index], keypar.key_tuple_length); + + if (saved_primary_key) + { + key_copy(saved_primary_key, table->record[0], + &table->key_info[table->s->primary_key], + table->key_info[table->s->primary_key].key_length); + } /* Save the last rowid */ memcpy(saved_rowid, file->ref, file->ref_length); @@ -433,9 +451,16 @@ void Mrr_ordered_index_reader::resume_read() { - key_restore(file->get_table()->record[0], saved_key_tuple, - &file->get_table()->key_info[file->active_index], + TABLE *table= file->get_table(); + key_restore(table->record[0], saved_key_tuple, + &table->key_info[file->active_index], keypar.key_tuple_length); + if (saved_primary_key) + { + key_restore(table->record[0], saved_primary_key, + &table->key_info[table->s->primary_key], + table->key_info[table->s->primary_key].key_length); + } } @@ -738,8 +763,8 @@ @retval other Error */ -int DsMrr_impl::dsmrr_init(handler *h_arg, RANGE_SEQ_IF *seq_funcs, - void *seq_init_param, uint n_ranges, uint mode, +int DsMrr_impl::dsmrr_init(handler *h_arg, RANGE_SEQ_IF *seq_funcs_arg, + void *seq_init_param_arg, uint n_ranges, uint mode, HANDLER_BUFFER *buf) { THD *thd= current_thd; @@ -748,6 +773,8 @@ uint key_buff_elem_size; handler *h_idx; Mrr_ordered_rndpos_reader *disk_strategy= NULL; + RANGE_SEQ_IF seq_funcs= *seq_funcs_arg; + void *seq_init_param= seq_init_param_arg; bool do_sort_keys= FALSE; DBUG_ENTER("DsMrr_impl::dsmrr_init"); LINT_INIT(key_buff_elem_size); /* set/used when do_sort_keys==TRUE */ @@ -810,8 +837,8 @@ { /* Pre-calculate some parameters of key sorting */ keypar.use_key_pointers= test(mode & HA_MRR_MATERIALIZED_KEYS); - seq_funcs->get_key_info(seq_init_param, &keypar.key_tuple_length, - &keypar.key_tuple_map); + seq_funcs.get_key_info(seq_init_param, &keypar.key_tuple_length, + &keypar.key_tuple_map); keypar.key_size_in_keybuf= keypar.use_key_pointers? sizeof(char*) : keypar.key_tuple_length; key_buff_elem_size= keypar.key_size_in_keybuf + (int)is_mrr_assoc * sizeof(void*); @@ -819,9 +846,18 @@ /* Ordered index reader needs some space to store an index tuple */ if (strategy != index_strategy) { + /* Normally we don't need to save/restore PK columns*/ + uint saved_pk_length=0; + if (h_idx->primary_key_is_clustered()) + { + uint pk= h_idx->get_table()->s->primary_key; + saved_pk_length= h_idx->get_table()->key_info[pk].key_length; + } + if (reader_factory.ordered_index_reader. set_interruption_temp_buffer(primary_file->ref_length, keypar.key_tuple_length, + saved_pk_length, &full_buf, full_buf_end)) goto use_default_impl; } @@ -845,7 +881,9 @@ if (do_sort_keys && !key_buffer->have_space_for(key_buff_elem_size)) goto use_default_impl; - if ((res= index_strategy->init(primary_file, seq_funcs, seq_init_param, n_ranges, + MRR_DEBUG(setup_range_id_mapping(&seq_funcs, &seq_init_param, n_ranges)); + + if ((res= index_strategy->init(primary_file, &seq_funcs, seq_init_param, n_ranges, mode, &keypar, key_buffer, &buf_manager))) goto error; } @@ -873,7 +911,9 @@ if ((res= setup_two_handlers())) goto error; - if ((res= index_strategy->init(secondary_file, seq_funcs, seq_init_param, + MRR_DEBUG(setup_range_id_mapping(&seq_funcs, &seq_init_param, n_ranges)); + + if ((res= index_strategy->init(secondary_file, &seq_funcs, seq_init_param, n_ranges, mode, &keypar, key_buffer, &buf_manager)) || (res= disk_strategy->init(primary_file, index_strategy, mode, @@ -883,6 +923,7 @@ } } + res= strategy->refill_buffer(TRUE); if (res) { @@ -910,7 +951,7 @@ DBUG_ASSERT(primary_file->inited == handler::INDEX); /* Call correct init function and assign to top level object */ Mrr_simple_index_reader *s= &reader_factory.simple_index_reader; - res= s->init(primary_file, seq_funcs, seq_init_param, n_ranges, mode, NULL, + res= s->init(primary_file, &seq_funcs, seq_init_param, n_ranges, mode, NULL, NULL, NULL); strategy= s; DBUG_RETURN(res); @@ -1033,6 +1074,7 @@ DBUG_ENTER("DsMrr_impl::dsmrr_close"); close_second_handler(); strategy= NULL; + MRR_DEBUG(clear_range_id_mapping()); DBUG_VOID_RETURN; } @@ -1690,7 +1732,110 @@ DBUG_VOID_RETURN; } - +#ifdef DBUG_MRR +range_seq_t DsMrr_impl::hook_init(void *init_params, uint n_ranges, uint flags) +{ + DsMrr_impl *mrr= (DsMrr_impl*)init_params; + mrr->backend_seq= mrr->backend_init(mrr->backend_init_params, n_ranges, flags); + return (range_seq_t)mrr; +} + +bool DsMrr_impl::hook_next(range_seq_t seq, KEY_MULTI_RANGE *range) +{ + DsMrr_impl *mrr= (DsMrr_impl*)seq; + int res; + if (!(res= mrr->backend_next(mrr->backend_seq, range))) + { + DBUG_ASSERT(mrr->cur_range_no < mrr->n_range_ids); + mrr->range_ids[mrr->cur_range_no++]= range->ptr; + } + return res; +} + +void DsMrr_impl::hook_get_key_info(void *init_params, uint *length, key_part_map *map) +{ + DsMrr_impl *mrr= (DsMrr_impl*)init_params; + mrr->backend_get_key_info(mrr->backend_init_params, length, map); +} + +bool DsMrr_impl::hook_skip_index_tuple(range_seq_t seq, char *range_info) +{ + DsMrr_impl *mrr= (DsMrr_impl*)seq; + int rangeno= mrr->range_id_to_number(range_info); + bool res= mrr->backend_skip_index_tuple(mrr->backend_seq, range_info); + fprintf(stderr, "checking index tuple %d -> %d\n", + rangeno, + (int)res); + return res; +} + +bool DsMrr_impl::hook_skip_record(range_seq_t seq, char *range_info, uchar *rowid) +{ + DsMrr_impl *mrr= (DsMrr_impl*)seq; + bool res= mrr->backend_skip_record(mrr->backend_seq, range_info, rowid); + return res; +} + +void DsMrr_impl::setup_range_id_mapping(RANGE_SEQ_IF *rseq, void **init_params, uint n_ranges) +{ + /* + Hook ourselves into RANGE_SEQ_IF so that we can watch ranges as they are + enumerated + */ + backend_init= rseq->init; + rseq->init= hook_init; + + backend_next= rseq->next; + rseq->next= hook_next; + + backend_init_params= *init_params; + *init_params= (void*)this; + + if (rseq->get_key_info) + { + backend_get_key_info= rseq->get_key_info; + rseq->get_key_info= hook_get_key_info; + } + + if (rseq->skip_index_tuple) + { + backend_skip_index_tuple= rseq->skip_index_tuple; + rseq->skip_index_tuple= hook_skip_index_tuple; + } + + if (rseq->skip_record) + { + backend_skip_record= rseq->skip_record; + rseq->skip_record= hook_skip_record; + } + + /* Alloc space for the mapping */ + n_range_ids= n_ranges; + if (n_ranges) + range_ids= (char**)malloc(sizeof(char*) * n_ranges); + else + range_ids= NULL; + cur_range_no= 0; +} + +void DsMrr_impl::clear_range_id_mapping() +{ + if (range_ids) + free(range_ids); + range_ids= NULL; + n_range_ids= 0; +} + +int DsMrr_impl::range_id_to_number(char *range_id) +{ + for (int i=0; i < n_range_ids; i++) + { + if (range_ids[i] == range_id) + return i; + } + return -1; +} +#endif /* ************************************************************************** * DS-MRR implementation ends ***************************************************************************/ === modified file 'sql/multi_range_read.h' --- a/sql/multi_range_read.h 2010-12-20 11:40:12 +0000 +++ b/sql/multi_range_read.h 2011-03-02 14:06:13 +0000 @@ -271,7 +271,8 @@ mrr_funcs.skip_index_tuple(mrr_iter, range_info)); } - bool set_interruption_temp_buffer(uint rowid_length, uint key_len, + bool set_interruption_temp_buffer(uint rowid_length, uint key_len, + uint saved_pk_len, uchar **space_start, uchar *space_end); void set_no_interruption_temp_buffer(); @@ -314,6 +315,7 @@ bool have_saved_rowid; uchar *saved_key_tuple; + uchar *saved_primary_key; static int compare_keys(void* arg, uchar* key1, uchar* key2); static int compare_keys_reverse(void* arg, uchar* key1, uchar* key2); @@ -520,7 +522,12 @@ typedef void (handler::*range_check_toggle_func_t)(bool on); DsMrr_impl() - : secondary_file(NULL) {}; + : secondary_file(NULL) + { +#ifdef DBUG_MRR + range_id_mapping_ctor(); +#endif + } void init(handler *h_arg, TABLE *table_arg) { @@ -615,6 +622,52 @@ int setup_two_handlers(); void close_second_handler(); + +#ifdef DBUG_MRR +#define MRR_DEBUG(x) x + /* + Debugging aid: support for converting range_ids (which are pointers) to + range numbers. Range's number specifies when the range was returned by + RANGE_SEQ_IF::next(). First returned range gets number 0, second gets + number 1, and so on. + */ + char **range_ids; /* range_ids[N] == range_id of range with number N */ + int n_range_ids; /* Total number of range ids */ + + /* "current" range number (makes sense when walking the ranges) */ + int cur_range_no; + + + /* RANGE_SEQ_IF override: "backend" functions */ + range_seq_t (*backend_init)(void *init_params, uint n_ranges, uint flags); + bool (*backend_next)(range_seq_t seq, KEY_MULTI_RANGE *range); + void (*backend_get_key_info)(void *init_params, uint *length, key_part_map *map); + bool (*backend_skip_index_tuple)(range_seq_t seq, char *range_info); + bool (*backend_skip_record)(range_seq_t seq, char *range_info, uchar *rowid); + void *backend_init_params; + range_seq_t backend_seq; + + /* RANGE_SEQ_IF override: our hook-functions */ + static range_seq_t hook_init(void *init_params, uint n_ranges, uint flags); + static bool hook_next(range_seq_t seq, KEY_MULTI_RANGE *range); + static void hook_get_key_info(void *init_params, uint *length, key_part_map *map); + static bool hook_skip_index_tuple(range_seq_t seq, char *range_info); + static bool hook_skip_record(range_seq_t seq, char *range_info, uchar *rowid); + + /* Mapper [de]initialization functions*/ + void range_id_mapping_ctor() + { + range_ids= NULL; + n_range_ids= 0; + } + void setup_range_id_mapping(RANGE_SEQ_IF *rseq, void **init_params, uint n_ranges); + void clear_range_id_mapping(); + + /* The reason of all of the above: function to map range ids to range numbers */ + int range_id_to_number(char *range_id); +#else +#define MRR_DEBUG(x) +#endif }; /** _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] Rev 2905: Merge Percona patch MWL#47 into mariadb 5.2-percona. in http://bazaar.launchpad.net/~maria-captains/maria/5.2
by Kristian Nielsen 02 Mar '11

02 Mar '11

knielsen(a)knielsen-hq.org writes: > message: > Merge Percona patch MWL#47 into mariadb 5.2-percona. > > This patch adds options to annotate the binlog (and the mysqlbinlog > output) with the original SQL query for queries that are logged > using row-based replication. Serg raised the very relevant issue about what to do about the similar feature in MySQL 5.6: http://forge.mysql.com/worklog/task.php?id=4033 http://forge.mysql.com/worklog/task.php?id=5404 The Oracle team has worked more on this since last I looked, and the two features are now very similar. Here are the main differences, as I see it: - The MySQL 5.6 feature is enabled by --binlog_rows_query_log_events, while the MWL#47 option is called --binlog-annotate-rows-events. - MWL#47 has --replicate-annotate-rows-events which makes the slave receive and binlog the annotations. The MySQL feature always receives annotation events if present in the binlog (so replication 5.6->pre-5.6 will break if annotations are enabled). The MySQL feature binlogs the annotations based on the same --binlog-annotate-rows-events option as the master. - MWL#47 has a mysqlbinlog option --skip-annotate-rows-events, MySQL 5.6 does not. - MySQL 5.6 mysqlbinlog can log the annotations with BINLOG statements so that they are preserved with `mysqlbinlog | mysql`; MWL#47 does not have this. - MWL#47 uses binlog event type 160, MySQL 5.6 uses an ignorable event of type 29. As far as I can tell, the actual event format is identical, except for the LOG_EVENT_IGNORABLE_F flag (which says it's ok for a slave to ignore this event if it doesn't know about it, however there is no server version that knows the LOG_EVENT_IGNORABLE_F but not the MySQL annotation event). - I think there are a couple of issues with the MySQL patch (newlines in queries breaking mysqlbinlog output, duplicated annotations for INSERT DELAYED). There might be issues with the MWL#47 patch also, of course. So the question is how to prepare for when we merge MySQL 5.6. An easy option could be to just ditch MWL#47 (and possibly backport the MySQL 5.6 patch). However, as I understand it, the original sponsor of MWL#47 is already using it, and so we may have to remain compatible with the existing MWL#47 implementation. Can we get some definite information on that? Another option might be to simply change the MWL#47 to use the same event type as MySQL (the event format is the same, and we can add the LOG_EVENT_IGNORABLE_F). And replace the MWL#47 options --binlog-annotate-rows-events and --replicate-annotate-rows-events with the MySQL --binlog_rows_query_log_events option (though I think the MySQL option name is not good). But again, I am not sure if this is a valid option given possible existing usage of MWL#47. The third option I can think of is to leave MWL#47 as it is. Then when we get to merge MySQL 5.6, we will basically drop the MySQL part, doing the following to remain backwards compatible with MySQL: - Make the --binlog_rows_query_log_events option set --binlog-annotate-rows-events and --replicate-annotate-rows-events. - Make a slave read a MySQL rows_query_log_event as a MWL#47 annotation event. - Merge over the bit of the MySQL patch that makes mysqlbinlog output annotation events using a BINLOG '...' statement. This should make upgrade from MySQL 5.6 -> MariaDB fully work. Using MySQL5.6+ with MariaDB will also work, only MySQL slaves will not receive MariaDB master annotation events. The third option seems to preserve the existing MWL#47 most faithfully and does not seem to need much work in terms of the eventual merge, so this is what I suggest as the best way forward. But I am open to other (justified) opinions. - Kristian.

2 1

Re: [Maria-developers] [Commits] Rev 3014: New table creation/renaming block added if old encoded table present. in file:///home/bell/maria/bzr/work-maria-5.1-CREATE/
by Sergei Golubchik 02 Mar '11

02 Mar '11

Hi, sanja! On Jan 13, sanja(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 3014 > revision-id: sanja(a)askmonty.org-20110113180607-e5hc8zuugxxjtn6k > parent: psergey(a)askmonty.org-20110112083018-90rw058f00uslvoz > committer: sanja(a)askmonty.org > branch nick: work-maria-5.1-CREATE > timestamp: Thu 2011-01-13 20:06:07 +0200 > message: > New table creation/renaming block added if old encoded table present. you've extended the fix, but not the test case. please add a test case for the new code and ok to push Regards, Sergei

1 0

[Maria-developers] Igor please review: Rev 2929: BUG#693747: Assertion multi_range_read.cc:908: int DsMrr_impl::dsmrr_init in file:///home/psergey/dev2/5.3-2/]
by Sergey Petrunya 02 Mar '11

02 Mar '11

Hello Igor, Could you please review the below: ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- From: Sergey Petrunya <psergey(a)askmonty.org> To: commits(a)mariadb.org X-Mailer: mail (GNU Mailutils 1.2) Date: Wed, 2 Mar 2011 00:17:53 +0300 (MSK) Subject: [Commits] Rev 2929: BUG#693747: Assertion multi_range_read.cc:908: int DsMrr_impl::dsmrr_init in file:///home/psergey/dev2/5.3-2/ At file:///home/psergey/dev2/5.3-2/ ------------------------------------------------------------ revno: 2929 revision-id: psergey(a)askmonty.org-20110301211747-8wvngodwxzr9i2e9 parent: psergey(a)askmonty.org-20110301072222-1rmapkelx9l1kw8u committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: 5.3-2 timestamp: Wed 2011-03-02 00:17:47 +0300 message: BUG#693747: Assertion multi_range_read.cc:908: int DsMrr_impl::dsmrr_init - Make DsMrr_impl::dsmrr_init() handle the case of 1. 1st MRR scan using DS-MRR strategy (i.e. doing key sorting and rowid sorting) 2. 2nd MRR scan getting a buffer that's too small to fit one key element and one rowid element, and so falling back to default MRR implementation In this case, dsmrr_init() is invoked with {primary_handler, secondary_handler} initialized for DS-MRR scan and have to reset them to be initialized for the default MRR scan. === modified file 'mysql-test/r/maria_mrr.result' --- a/mysql-test/r/maria_mrr.result 2010-12-02 11:10:52 +0000 +++ b/mysql-test/r/maria_mrr.result 2011-03-01 21:17:47 +0000 @@ -403,3 +403,30 @@ count(*) 0 drop table t1, t2; +# +# BUG#693747: Assertion multi_range_read.cc:908: int DsMrr_impl::dsmrr_init( +# +set @_save_join_cache_level= @@join_cache_level; +set @_save_join_buffer_size= @@join_buffer_size; +set join_cache_level=8; +set join_buffer_size=10000; +CREATE TABLE t1 ( +f1 int(11), f2 varchar(32) COLLATE latin1_swedish_ci, +f3 int(11), f4 varchar(1024) COLLATE utf8_bin, +f5 varchar(1024) COLLATE latin1_bin, +KEY (f5) +) ENGINE=Aria TRANSACTIONAL=0 ; +# Fill the table with some data +SELECT alias2.f1 AS field1 , alias1.f2 AS field2 +FROM t1 AS alias1 +LEFT OUTER JOIN t1 AS alias2 ON alias1.f2 = alias2.f5 +WHERE alias2.f3 > 'k' +AND alias2.f3 <= 'z' +OR alias2.f4 IN ('o', 'y') +AND alias2.f4 >= 'g' AND alias2.f4 < 'z'; +field1 field2 +Warnings: +Warning 1292 Truncated incorrect DOUBLE value: 'k' +set join_cache_level=@_save_join_cache_level; +set join_buffer_size=@_save_join_buffer_size; +drop table t1; === modified file 'mysql-test/t/maria_mrr.test' --- a/mysql-test/t/maria_mrr.test 2010-11-09 16:02:08 +0000 +++ b/mysql-test/t/maria_mrr.test 2011-03-01 21:17:47 +0000 @@ -125,3 +125,81 @@ table1.col_varchar_1024_latin1_key = table2.col_varchar_10_latin1 AND table1.pk<>0 ; drop table t1, t2; + +--echo # +--echo # BUG#693747: Assertion multi_range_read.cc:908: int DsMrr_impl::dsmrr_init( +--echo # +set @_save_join_cache_level= @@join_cache_level; +set @_save_join_buffer_size= @@join_buffer_size; + +set join_cache_level=8; +set join_buffer_size=10000; + +CREATE TABLE t1 ( + f1 int(11), f2 varchar(32) COLLATE latin1_swedish_ci, + f3 int(11), f4 varchar(1024) COLLATE utf8_bin, + f5 varchar(1024) COLLATE latin1_bin, + KEY (f5) +) ENGINE=Aria TRANSACTIONAL=0 ; + +--echo # Fill the table with some data +--disable_query_log +INSERT IGNORE INTO t1 VALUES +('0','cueikuirqr','0','okay','hcueikuirqrzflno'),('2009','her','0','ULWAH','ehcueikuirqrzfln'), +('0','YKAOE','0','of','qieehcueikuirqrz'),('2009','you\'re','0','k','nkqieehcueikuirq'), +('2003','b','0','bgsnkqieehcueiku','the'),('0','MGUDG','0','p','m'), +('0','UXAGU','0','elfhjawbwbgsnkqi','HZXVA'),('2004','bwbgsnkqie','0','x','something'), +('2007','s','0','okay','slelfhjawbwbgsnk'),('2007','the','0','or','if'), +('2008','TDLKE','0','mean','MGWNJ'),('2006','do','0','KGFVV','see'), +('2001','why','0','a','mean'),(NULL,'THKCG','0','no','YFLDY'), +(NULL,'x','0','mean','e'),('2007','yncitaeysb','0','WDOSC','tgyncitaeysbgucs'), +(NULL,'ZEOXX','0','awbwbgsnkqieehcu','jawbwbgsnkqieehc'),('2006','hjawbwbgsn','0','just','fhjawbwbgsnkqiee'), +('0','all','0','k','sbgucsgqslelfhja'),('2009','the','0','f','would'), +('2005','mtgyncitae','0','w','ISNQQ'),('2003','KNCUI','0','r','want'), +('2006','is','0','j','i'),('2000','out','0','o','jvcmjlmtgyncitae'), +('2001','it','0','txyjvcmjlmtgynci','you'),('2008','LHDIH','0','ZKFJK','txmtxyjvcmjlmtgy'), +('0','z','0','the','ntxmtxyjvcmjlmtg'),(NULL,'vyhnmvgmcn','0','PWEYL','AIGQK'), +('0','ytvyhnmvgm','0','gbytvyhnmvgmcntx','z'),('2002','t','0','just','on'), +('2002','xqegbytvyh','0','to','ixqegbytvyhnmvgm'),('2004','WGVRU','0','GHBZB','h'), +('0','b','0','yhnmvgmcntxmtxyj','z'),('2003','who','0','ddixqegbytvyhnmv','gddixqegbytvy'), +('0','PMLFL','0','n','vgmcntxmtxyjvcmj'),(NULL,'back','0','FIXPA','n'), +('2005','i','0','I\'ll','PZGUB'),('0','f','0','t','the'), +(NULL,'PNXVP','0','t','v'),('2004','MAKKL','0','u','CGCWF'), +('0','RMDAV','0','noypgddixqegbytv','v'),('0','l','0','ESAVW','n'), +('2008','rhnoypgddi','0','s','VIZNE'),('2007','t','0','XZECS','a'), +(NULL,'like','0','k','JSHPZ'),('2009','pskeywslmk','0','epskeywslmkdrhno','q'), +(NULL,'QZZJJ','0','LOTFB','c'),('0','atlxepskey','0','b','YJRMA'), +(NULL,'YUVOU','0','wslmkdrhnoypgddi','eywslmkdrhnoypgd'),('2006','some','0','can','r'), +('2002','c','0','k','her'),(NULL,'o','0','ALBVZ','EMURT'), +('0','if','0','okay','had'),('2006','when','0','e','CLVWT'), +(NULL,'blfufrcdjm','0','ZAKUQ','IZCZN'),('2007','vutblfufrc','0','MPKMI','how'), +(NULL,'why','0','UBUAG','I'),('2004','IXLYQ','0','ILPBE','weuwuvutblfufrcd'), +('0','here','0','your','m'),(NULL,'ZOCTJ','0','GTPOO','IDSFD'), +('0','kqsweuwuvu','0','CXQLW','oh'),('0','ykqsweuwuv','0','CLJWP','zykqsweuwuvutblf'), +('0','zezykqsweu','0','had','t'),(NULL,'q','0','jisuzezykqsweuwu','o'), +('2007','IBKAU','0','t','oh'),('0','ivjisuzezy','0','y','XHXKE'), +('0','xsivjisuze','0','right','plxsivjisuzezykq'),('2009','have','0','so','uvplxsivjisuzezy'), +('0','on','0','b','me'),('0','ijkfuvplxs','0','RZXDY','OGEHV'), +('0','u','0','all','okay'),('0','i','0','s','pajzbbojshnijkfu'), +('2001','of','0','xpajzbbojshnijkf','g'),('0','for','0','ITBAQ','I\'m'), +('2006','or','0','now','ZOJHX'),('0','n','0','k','you'), +(NULL,'that','0','suzezykqsweuwuvu','just'),('0','bbojshnijk','0','QGYOA','JYGSJ'), +('2009','k','0','ajzbbojshnijkfuv','y'),('2007','k','0','affpegvavdgxpajz','y'), +('2007','be','0','RCTIT','m'),('2000','fnbmxwicrk','0','lcfnbmxwicrkdyma','t'), +('0','yaffpegvav','0','v','have'),('0','crkdymahya','0','icrkdymahyaffpeg','QQWQI'), +('0','t','0','nijkfuvplxsivjis','hnijkfuvplxsivji'),('0','dgxpajzbbo','0','vdgxpajzbbojshni','vavdgxpajzbbojsh'), +('2006','g','0','gvavdgxpajzbbojs','pegvavdgxpajzbbo'),('2007','I\'m','0','g','ffpegvavdgxpajzb'); +--enable_query_log + + +SELECT alias2.f1 AS field1 , alias1.f2 AS field2 +FROM t1 AS alias1 +LEFT OUTER JOIN t1 AS alias2 ON alias1.f2 = alias2.f5 +WHERE alias2.f3 > 'k' +AND alias2.f3 <= 'z' +OR alias2.f4 IN ('o', 'y') +AND alias2.f4 >= 'g' AND alias2.f4 < 'z'; + +set join_cache_level=@_save_join_cache_level; +set join_buffer_size=@_save_join_buffer_size; +drop table t1; === modified file 'sql/multi_range_read.cc' --- a/sql/multi_range_read.cc 2011-02-18 22:31:01 +0000 +++ b/sql/multi_range_read.cc 2011-03-01 21:17:47 +0000 @@ -907,7 +907,26 @@ DBUG_RETURN(res); use_default_impl: - DBUG_ASSERT(primary_file->inited == handler::INDEX); + if (primary_file->inited != handler::INDEX) + { + /* We can get here when + - we've previously successfully done a DS-MRR scan (and so have + secondary_file!= NULL, secondary_file->inited= INDEX, + primary_file->inited=RND) + - for this invocation, we haven't got enough buffer space, and so we + have to use the default MRR implementation. + + note: primary_file->ha_index_end() will call dsmrr_close() which will + close/destroy the secondary_file, this is intentional. + (Yes this is slow, but one can't expect performance with join buffer + so small that it can accomodate one rowid and one index tuple) + */ + if ((res= primary_file->ha_rnd_end()) || + (res= primary_file->ha_index_init(keyno, test(mode & HA_MRR_SORTED)))) + { + DBUG_RETURN(res); + } + } /* Call correct init function and assign to top level object */ Mrr_simple_index_reader *s= &reader_factory.simple_index_reader; res= s->init(primary_file, seq_funcs, seq_init_param, n_ranges, mode, NULL, _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Igor please review: BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) in file:///home/psergey/dev2/5.3/]
by Sergey Petrunya 02 Mar '11

02 Mar '11

Hello Igor, Could you please review the below: ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- From: Sergey Petrunya <psergey(a)askmonty.org> To: commits(a)mariadb.org X-Mailer: mail (GNU Mailutils 1.2) Date: Wed, 2 Mar 2011 16:27:43 +0300 (MSK) Subject: [Commits] Rev 2929: BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) in file:///home/psergey/dev2/5.3/ At file:///home/psergey/dev2/5.3/ ------------------------------------------------------------ revno: 2929 revision-id: psergey(a)askmonty.org-20110302132739-s6vasca1nm6eh2le parent: psergey(a)askmonty.org-20110301072222-1rmapkelx9l1kw8u committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: 5.3 timestamp: Wed 2011-03-02 16:27:39 +0300 message: BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) - Add debugging aid which maps MRR's range_ids (which are pointers so are different on every execution and hard to track) to range numbers (which allow much easier debugging). The mapper is not compiled unless the developer puts a #define into code to request it. - Fix the bug: the problem was that Mrr_ordered_index_reader's interrupt_read() and resume_read() would save and restore 1) index tuple 2) the rowid (as bytes returned by handler->position()). Clustered primary key columns were not saved/restored. They are not explicitly present in the index tuple (i.e. table->key_info[secondary_key].key_parts doesn't list them), but they are actually there, in particular table->field[clustered_primary_key_member].part_of_key(secondary_key) == 1. Index condition pushdown code [correctly] uses the latter as inidication that pushed index condition can refer to clustered PK members. The fix was to make interrupt_read()/resume_read() to save/restore clustered primary key members as well, so that we get correct values for them when evaluating pushed index condition. === modified file 'mysql-test/r/innodb_mrr.result' --- a/mysql-test/r/innodb_mrr.result 2011-01-17 21:26:04 +0000 +++ b/mysql-test/r/innodb_mrr.result 2011-03-02 13:27:39 +0000 @@ -736,3 +736,31 @@ c1 c2 c3 08:29:45 NULL 2009-02-01 drop table `t1`; +# +# BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) +# +set @_save_join_cache_level= @@join_cache_level; +set join_cache_level = 6; +CREATE TABLE t1 ( +f1 int(11), f2 int(11), f3 varchar(1), f4 varchar(1), +PRIMARY KEY (f1), +KEY (f3), +KEY (f2) +) ENGINE=InnoDB; +INSERT INTO t1 VALUES ('11','8','f','f'),('12','5','v','v'),('13','8','s','s'), +('14','8','a','a'),('15','6','p','p'),('16','7','z','z'),('17','2','a','a'), +('18','5','h','h'),('19','7','h','h'),('20','2','v','v'),('21','9','v','v'), +('22','142','b','b'),('23','3','y','y'),('24','0','v','v'),('25','3','m','m'), +('26','5','z','z'),('27','9','n','n'),('28','1','d','d'),('29','107','a','a'); +select count(*) from ( +SELECT alias1.f2 +FROM +t1 AS alias1 JOIN ( +t1 AS alias2 FORCE KEY (f3) JOIN +t1 AS alias3 FORCE KEY (f2) ON alias3.f2 = alias2.f2 AND alias3.f4 = alias2.f3 +) ON alias3.f1 <= alias2.f1 +) X; +count(*) +361 +set join_cache_level=@_save_join_cache_level; +drop table t1; === modified file 'mysql-test/t/innodb_mrr.test' --- a/mysql-test/t/innodb_mrr.test 2011-01-17 21:26:04 +0000 +++ b/mysql-test/t/innodb_mrr.test 2011-03-02 13:27:39 +0000 @@ -426,3 +426,31 @@ SELECT * FROM t1 WHERE c2 <=> NULL ORDER BY c2 LIMIT 2; drop table `t1`; +--echo # +--echo # BUG#707925: Wrong result with join_cache_level=6 optimizer_use_mrr = force (incremental, BKA join) +--echo # +set @_save_join_cache_level= @@join_cache_level; +set join_cache_level = 6; +CREATE TABLE t1 ( + f1 int(11), f2 int(11), f3 varchar(1), f4 varchar(1), + PRIMARY KEY (f1), + KEY (f3), + KEY (f2) +) ENGINE=InnoDB; +INSERT INTO t1 VALUES ('11','8','f','f'),('12','5','v','v'),('13','8','s','s'), +('14','8','a','a'),('15','6','p','p'),('16','7','z','z'),('17','2','a','a'), +('18','5','h','h'),('19','7','h','h'),('20','2','v','v'),('21','9','v','v'), +('22','142','b','b'),('23','3','y','y'),('24','0','v','v'),('25','3','m','m'), +('26','5','z','z'),('27','9','n','n'),('28','1','d','d'),('29','107','a','a'); + +select count(*) from ( + SELECT alias1.f2 + FROM + t1 AS alias1 JOIN ( + t1 AS alias2 FORCE KEY (f3) JOIN + t1 AS alias3 FORCE KEY (f2) ON alias3.f2 = alias2.f2 AND alias3.f4 = alias2.f3 + ) ON alias3.f1 <= alias2.f1 +) X; + +set join_cache_level=@_save_join_cache_level; +drop table t1; === modified file 'sql/multi_range_read.cc' --- a/sql/multi_range_read.cc 2011-02-18 22:31:01 +0000 +++ b/sql/multi_range_read.cc 2011-03-02 13:27:39 +0000 @@ -387,15 +387,25 @@ bool Mrr_ordered_index_reader::set_interruption_temp_buffer(uint rowid_length, uint key_len, + uint saved_pk_len, uchar **space_start, uchar *space_end) { - if (space_end - *space_start <= (ptrdiff_t)(rowid_length + key_len)) + if (space_end - *space_start <= (ptrdiff_t)(rowid_length + key_len + saved_pk_len)) return TRUE; support_scan_interruptions= TRUE; saved_rowid= *space_start; *space_start += rowid_length; + + if (saved_pk_len) + { + saved_primary_key= *space_start; + *space_start += saved_pk_len; + } + else + saved_primary_key= NULL; + saved_key_tuple= *space_start; *space_start += key_len; @@ -406,17 +416,25 @@ void Mrr_ordered_index_reader::set_no_interruption_temp_buffer() { support_scan_interruptions= FALSE; - saved_key_tuple= saved_rowid= NULL; /* safety */ + saved_key_tuple= saved_rowid= saved_primary_key= NULL; /* safety */ have_saved_rowid= FALSE; } void Mrr_ordered_index_reader::interrupt_read() { DBUG_ASSERT(support_scan_interruptions); + TABLE *table= file->get_table(); /* Save the current key value */ - key_copy(saved_key_tuple, file->get_table()->record[0], - &file->get_table()->key_info[file->active_index], + key_copy(saved_key_tuple, table->record[0], + &table->key_info[file->active_index], keypar.key_tuple_length); + + if (saved_primary_key) + { + key_copy(saved_primary_key, table->record[0], + &table->key_info[table->s->primary_key], + table->key_info[table->s->primary_key].key_length); + } /* Save the last rowid */ memcpy(saved_rowid, file->ref, file->ref_length); @@ -433,9 +451,16 @@ void Mrr_ordered_index_reader::resume_read() { - key_restore(file->get_table()->record[0], saved_key_tuple, - &file->get_table()->key_info[file->active_index], + TABLE *table= file->get_table(); + key_restore(table->record[0], saved_key_tuple, + &table->key_info[file->active_index], keypar.key_tuple_length); + if (saved_primary_key) + { + key_restore(table->record[0], saved_primary_key, + &table->key_info[table->s->primary_key], + table->key_info[table->s->primary_key].key_length); + } } @@ -738,8 +763,8 @@ @retval other Error */ -int DsMrr_impl::dsmrr_init(handler *h_arg, RANGE_SEQ_IF *seq_funcs, - void *seq_init_param, uint n_ranges, uint mode, +int DsMrr_impl::dsmrr_init(handler *h_arg, RANGE_SEQ_IF *seq_funcs_arg, + void *seq_init_param_arg, uint n_ranges, uint mode, HANDLER_BUFFER *buf) { THD *thd= current_thd; @@ -748,6 +773,8 @@ uint key_buff_elem_size; handler *h_idx; Mrr_ordered_rndpos_reader *disk_strategy= NULL; + RANGE_SEQ_IF seq_funcs= *seq_funcs_arg; + void *seq_init_param= seq_init_param_arg; bool do_sort_keys= FALSE; DBUG_ENTER("DsMrr_impl::dsmrr_init"); LINT_INIT(key_buff_elem_size); /* set/used when do_sort_keys==TRUE */ @@ -810,8 +837,8 @@ { /* Pre-calculate some parameters of key sorting */ keypar.use_key_pointers= test(mode & HA_MRR_MATERIALIZED_KEYS); - seq_funcs->get_key_info(seq_init_param, &keypar.key_tuple_length, - &keypar.key_tuple_map); + seq_funcs.get_key_info(seq_init_param, &keypar.key_tuple_length, + &keypar.key_tuple_map); keypar.key_size_in_keybuf= keypar.use_key_pointers? sizeof(char*) : keypar.key_tuple_length; key_buff_elem_size= keypar.key_size_in_keybuf + (int)is_mrr_assoc * sizeof(void*); @@ -819,9 +846,18 @@ /* Ordered index reader needs some space to store an index tuple */ if (strategy != index_strategy) { + /* Normally we don't need to save/restore PK columns*/ + uint saved_pk_length=0; + if (h_idx->primary_key_is_clustered()) + { + uint pk= h_idx->get_table()->s->primary_key; + saved_pk_length= h_idx->get_table()->key_info[pk].key_length; + } + if (reader_factory.ordered_index_reader. set_interruption_temp_buffer(primary_file->ref_length, keypar.key_tuple_length, + saved_pk_length, &full_buf, full_buf_end)) goto use_default_impl; } @@ -845,7 +881,9 @@ if (do_sort_keys && !key_buffer->have_space_for(key_buff_elem_size)) goto use_default_impl; - if ((res= index_strategy->init(primary_file, seq_funcs, seq_init_param, n_ranges, + MRR_DEBUG(setup_range_id_mapping(&seq_funcs, &seq_init_param, n_ranges)); + + if ((res= index_strategy->init(primary_file, &seq_funcs, seq_init_param, n_ranges, mode, &keypar, key_buffer, &buf_manager))) goto error; } @@ -873,7 +911,9 @@ if ((res= setup_two_handlers())) goto error; - if ((res= index_strategy->init(secondary_file, seq_funcs, seq_init_param, + MRR_DEBUG(setup_range_id_mapping(&seq_funcs, &seq_init_param, n_ranges)); + + if ((res= index_strategy->init(secondary_file, &seq_funcs, seq_init_param, n_ranges, mode, &keypar, key_buffer, &buf_manager)) || (res= disk_strategy->init(primary_file, index_strategy, mode, @@ -883,6 +923,7 @@ } } + res= strategy->refill_buffer(TRUE); if (res) { @@ -910,7 +951,7 @@ DBUG_ASSERT(primary_file->inited == handler::INDEX); /* Call correct init function and assign to top level object */ Mrr_simple_index_reader *s= &reader_factory.simple_index_reader; - res= s->init(primary_file, seq_funcs, seq_init_param, n_ranges, mode, NULL, + res= s->init(primary_file, &seq_funcs, seq_init_param, n_ranges, mode, NULL, NULL, NULL); strategy= s; DBUG_RETURN(res); @@ -1033,6 +1074,7 @@ DBUG_ENTER("DsMrr_impl::dsmrr_close"); close_second_handler(); strategy= NULL; + MRR_DEBUG(clear_range_id_mapping()); DBUG_VOID_RETURN; } @@ -1690,7 +1732,110 @@ DBUG_VOID_RETURN; } - +#ifdef DBUG_MRR +range_seq_t DsMrr_impl::hook_init(void *init_params, uint n_ranges, uint flags) +{ + DsMrr_impl *mrr= (DsMrr_impl*)init_params; + mrr->backend_seq= mrr->backend_init(mrr->backend_init_params, n_ranges, flags); + return (range_seq_t)mrr; +} + +bool DsMrr_impl::hook_next(range_seq_t seq, KEY_MULTI_RANGE *range) +{ + DsMrr_impl *mrr= (DsMrr_impl*)seq; + int res; + if (!(res= mrr->backend_next(mrr->backend_seq, range))) + { + DBUG_ASSERT(mrr->cur_range_no < mrr->n_range_ids); + mrr->range_ids[mrr->cur_range_no++]= range->ptr; + } + return res; +} + +void DsMrr_impl::hook_get_key_info(void *init_params, uint *length, key_part_map *map) +{ + DsMrr_impl *mrr= (DsMrr_impl*)init_params; + mrr->backend_get_key_info(mrr->backend_init_params, length, map); +} + +bool DsMrr_impl::hook_skip_index_tuple(range_seq_t seq, char *range_info) +{ + DsMrr_impl *mrr= (DsMrr_impl*)seq; + int rangeno= mrr->range_id_to_number(range_info); + bool res= mrr->backend_skip_index_tuple(mrr->backend_seq, range_info); + fprintf(stderr, "checking index tuple %d -> %d\n", + rangeno, + (int)res); + return res; +} + +bool DsMrr_impl::hook_skip_record(range_seq_t seq, char *range_info, uchar *rowid) +{ + DsMrr_impl *mrr= (DsMrr_impl*)seq; + bool res= mrr->backend_skip_record(mrr->backend_seq, range_info, rowid); + return res; +} + +void DsMrr_impl::setup_range_id_mapping(RANGE_SEQ_IF *rseq, void **init_params, uint n_ranges) +{ + /* + Hook ourselves into RANGE_SEQ_IF so that we can watch ranges as they are + enumerated + */ + backend_init= rseq->init; + rseq->init= hook_init; + + backend_next= rseq->next; + rseq->next= hook_next; + + backend_init_params= *init_params; + *init_params= (void*)this; + + if (rseq->get_key_info) + { + backend_get_key_info= rseq->get_key_info; + rseq->get_key_info= hook_get_key_info; + } + + if (rseq->skip_index_tuple) + { + backend_skip_index_tuple= rseq->skip_index_tuple; + rseq->skip_index_tuple= hook_skip_index_tuple; + } + + if (rseq->skip_record) + { + backend_skip_record= rseq->skip_record; + rseq->skip_record= hook_skip_record; + } + + /* Alloc space for the mapping */ + n_range_ids= n_ranges; + if (n_ranges) + range_ids= (char**)malloc(sizeof(char*) * n_ranges); + else + range_ids= NULL; + cur_range_no= 0; +} + +void DsMrr_impl::clear_range_id_mapping() +{ + if (range_ids) + free(range_ids); + range_ids= NULL; + n_range_ids= 0; +} + +int DsMrr_impl::range_id_to_number(char *range_id) +{ + for (int i=0; i < n_range_ids; i++) + { + if (range_ids[i] == range_id) + return i; + } + return -1; +} +#endif /* ************************************************************************** * DS-MRR implementation ends ***************************************************************************/ === modified file 'sql/multi_range_read.h' --- a/sql/multi_range_read.h 2010-12-20 11:40:12 +0000 +++ b/sql/multi_range_read.h 2011-03-02 13:27:39 +0000 @@ -271,7 +271,8 @@ mrr_funcs.skip_index_tuple(mrr_iter, range_info)); } - bool set_interruption_temp_buffer(uint rowid_length, uint key_len, + bool set_interruption_temp_buffer(uint rowid_length, uint key_len, + uint saved_pk_len, uchar **space_start, uchar *space_end); void set_no_interruption_temp_buffer(); @@ -314,6 +315,7 @@ bool have_saved_rowid; uchar *saved_key_tuple; + uchar *saved_primary_key; static int compare_keys(void* arg, uchar* key1, uchar* key2); static int compare_keys_reverse(void* arg, uchar* key1, uchar* key2); @@ -520,7 +522,12 @@ typedef void (handler::*range_check_toggle_func_t)(bool on); DsMrr_impl() - : secondary_file(NULL) {}; + : secondary_file(NULL) + { +#ifdef DBUG_MRR + range_id_mapping_ctor(); +#endif + } void init(handler *h_arg, TABLE *table_arg) { @@ -615,6 +622,50 @@ int setup_two_handlers(); void close_second_handler(); + +#ifdef DBUG_MRR +#define MRR_DEBUG(x) x + /* + Debugging aid: support for converting range_ids (which are pointers) to + range numbers. Range's number specifies when the range was returned by + RANGE_SEQ_IF::next(). First returned range gets number 0, second gets + number 1, and so on. + */ + char **range_ids; /* range_ids[N] == range_id of range with number N */ + int n_range_ids; /* Total number of range ids */ + + /* "current" range number (makes sense when walking the ranges) */ + int cur_range_no; + + + /* RANGE_SEQ_IF override: "backend" functions */ + range_seq_t (*backend_init)(void *init_params, uint n_ranges, uint flags); + bool (*backend_next)(range_seq_t seq, KEY_MULTI_RANGE *range); + void (*backend_get_key_info)(void *init_params, uint *length, key_part_map *map); + bool (*backend_skip_index_tuple)(range_seq_t seq, char *range_info); + bool (*backend_skip_record)(range_seq_t seq, char *range_info, uchar *rowid); + void *backend_init_params; + range_seq_t backend_seq; + + /* RANGE_SEQ_IF override: our hook-functions */ + static range_seq_t hook_init(void *init_params, uint n_ranges, uint flags); + static bool hook_next(range_seq_t seq, KEY_MULTI_RANGE *range); + static void hook_get_key_info(void *init_params, uint *length, key_part_map *map); + static bool hook_skip_index_tuple(range_seq_t seq, char *range_info); + static bool hook_skip_record(range_seq_t seq, char *range_info, uchar *rowid); + + /* Mapper [de]initialization functions*/ + void range_id_mapping_ctor() + { + range_ids= NULL; + n_range_ids= 0; + } + void setup_range_id_mapping(RANGE_SEQ_IF *rseq, void **init_params, uint n_ranges); + void clear_range_id_mapping(); + + /* The reason of all of the above: function to map range ids to range numbers */ + int range_id_to_number(char *range_id); +#endif }; /** _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] dgcov.pl
by Kristian Nielsen 02 Mar '11

02 Mar '11

Hi Serg, I pushed to lp:mariadb-tools a "fix" for the negative numbers output by gcov which causes dgcov.pl to break. (Sergey Petrunya needed it). It just strips the leading '-' from negative numbers, so it does not help to understand what the problem is, I would still like to figure that out... but meanwhile, I hope you are ok with the workaround, or at least I wanted to make you aware of it. - Kristian.

1 0

Re: [Maria-developers] Release of MariaDB 5.1.55
by Daniel Bartholomew 02 Mar '11

02 Mar '11

On Mon, 28 Feb 2011 12:40:13 +0100 Kristian Nielsen <knielsen(a)knielsen-hq.org> wrote: Kristian> Monty gave the go for releasing MariaDB 5.1.55. Thanks Kristian for all of your work in getting the packages ready, especially the new .deb repository. And thank you to all of the developers who worked on this release! Here are the download, release notes, and changelog pages for 5.1.55: * http://downloads.askmonty.org/mariadb/5.1.55 * http://kb.askmonty.org/v/mariadb-5155-release-notes * http://kb.askmonty.org/v/mariadb-5155-changelog All: If you have a moment, please check over the above pages to see if there are any mistakes or omissions and let me know (or jump in and fix if you have editing rights). One brief note about the new downloads system: I'll also be adding some redirects to the old wiki-based downloads site to direct people to the new downloads site (in case they have the old site bookmarked). Community team: I tested a sample of the mirrors and they all had the files so please announce the release on the announce list, blogs, and so on. After sending out this email I will add an entry to the releases RSS feed. Thanks. -- Daniel Bartholomew MariaDB - http://mariadb.org Monty Program - http://montyprogram.com AskMonty Knowledgebase - http://kb.askmonty.org

2 1

[Maria-developers] Buildbot Update
by Adam M. Dutko 01 Mar '11

01 Mar '11

All of the buildbots I run should be back online.

2 2

[Maria-developers] Igor please review: [Commits] Rev 2926: BUG#724275: Crash in JOIN::optimize in maria-5.3 in file:///home/psergey/dev2/5.3-2/]
by Sergey Petrunya 28 Feb '11

28 Feb '11

Hello Igor, Could you please review the below: ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- From: Sergey Petrunya <psergey(a)askmonty.org> To: commits(a)mariadb.org X-Mailer: mail (GNU Mailutils 1.2) Date: Tue, 1 Mar 2011 00:30:02 +0300 (MSK) Subject: [Commits] Rev 2926: BUG#724275: Crash in JOIN::optimize in maria-5.3 in file:///home/psergey/dev2/5.3-2/ At file:///home/psergey/dev2/5.3-2/ ------------------------------------------------------------ revno: 2926 revision-id: psergey(a)askmonty.org-20110228212959-hl3selurcdou7d22 parent: igor(a)askmonty.org-20110227181411-k81vmi35o0no21ab committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: 5.3-2 timestamp: Tue 2011-03-01 00:29:59 +0300 message: BUG#724275: Crash in JOIN::optimize in maria-5.3 - Make equality-substitution-for-ref-access code in JOIN::optimize() treat join_tab->ref.key_copy correctly (in the way create_ref_for_key() has filled it). === modified file 'mysql-test/r/join.result' --- a/mysql-test/r/join.result 2010-10-30 13:07:45 +0000 +++ b/mysql-test/r/join.result 2011-02-28 21:29:59 +0000 @@ -1251,3 +1251,37 @@ Handler_read_rnd_next 1 DROP TABLE t1, t2; End of 5.1 tests +# +# BUG#724275: Crash in JOIN::optimize in maria-5.3 +# +create table t1 (a int); +insert into t1 values (1),(2); +insert into t1 select * from t1; +create table t2 (a int, b int, key(a,b)); +insert into t2 values (1,1),(1,2),(1,3),(1,4),(2,5),(2,6),(2,7),(2,8),(2,9); +insert into t2 select * from t2; +insert into t2 select * from t2; +insert into t2 select * from t2; +create table t3 (a int, b int, key(a)); +insert into t3 values (1,1),(2,2); +select * from +t3 straight_join t1 straight_join t2 force index(a) +where t2.a=1 and t2.b=t1.a and t1.a=t3.b and t3.a=1; +a b a a b +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +1 1 1 1 1 +drop table t1,t2,t3; === modified file 'mysql-test/t/join.test' --- a/mysql-test/t/join.test 2010-10-30 13:07:45 +0000 +++ b/mysql-test/t/join.test 2011-02-28 21:29:59 +0000 @@ -941,3 +941,26 @@ DROP TABLE t1, t2; --echo End of 5.1 tests + +--echo # +--echo # BUG#724275: Crash in JOIN::optimize in maria-5.3 +--echo # + +create table t1 (a int); +insert into t1 values (1),(2); +insert into t1 select * from t1; + +create table t2 (a int, b int, key(a,b)); +insert into t2 values (1,1),(1,2),(1,3),(1,4),(2,5),(2,6),(2,7),(2,8),(2,9); +insert into t2 select * from t2; +insert into t2 select * from t2; +insert into t2 select * from t2; + +create table t3 (a int, b int, key(a)); +insert into t3 values (1,1),(2,2); +select * from + t3 straight_join t1 straight_join t2 force index(a) +where t2.a=1 and t2.b=t1.a and t1.a=t3.b and t3.a=1; + +drop table t1,t2,t3; + === modified file 'sql/sql_select.cc' --- a/sql/sql_select.cc 2011-02-27 18:14:11 +0000 +++ b/sql/sql_select.cc 2011-02-28 21:29:59 +0000 @@ -1056,6 +1056,7 @@ */ for (JOIN_TAB *tab= join_tab + const_tables; tab < join_tab + tables; tab++) { + uint key_copy_index=0; for (uint i=0; i < tab->ref.key_parts; i++) { @@ -1071,13 +1072,14 @@ { *ref_item_ptr= ref_item; Item *item= ref_item->real_item(); - store_key *key_copy= tab->ref.key_copy[i]; + store_key *key_copy= tab->ref.key_copy[key_copy_index]; if (key_copy->type() == store_key::FIELD_STORE_KEY) { store_key_field *field_copy= ((store_key_field *)key_copy); field_copy->change_source_field((Item_field *) item); } } + key_copy_index++; } } _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Buildbot Outage
by Adam M. Dutko 28 Feb '11

28 Feb '11

It is possible the buildbots I sponsor are offline today: mariadb-brs, adutko-centos-amd64 and adutko-ultrasparc3. The transformer that feeds our house was hit by lightning last night. It was still operational this morning but I believe the power company is replacing it today.

1 0

[Maria-developers] Fix for PBXT for problem with 'repair-pending'
by Michael Widenius 25 Feb '11

25 Feb '11

Hi! We have got some failures of the following type in pushbuild: pbxt.mysqlslap w2 [ fail ] Test ended at 2011-02-25 14:37:57 CURRENT_TEST: pbxt.mysqlslap mysqltest: At line 43: query 'drop database pbxt' failed: 1010: Error dropping database (can't rmdir './pbxt', errno: 39) The reason is that at the end of the test there was left in the pbxt directory a file "repair-pending" with a reference to some old already deleted table. There was nothing else in the directory. My assumption (and the code seams to agree with it) is that when you delete a table, you don't remove the table from the repair-pending file even if it had been recorded there. Here is a patch that fixes this: === modified file 'storage/pbxt/src/table_xt.cc' --- storage/pbxt/src/table_xt.cc 2010-11-26 22:37:34 +0000 +++ storage/pbxt/src/table_xt.cc 2011-02-25 14:40:53 +0000 @@ -1755,6 +1755,8 @@ xtPublic void xt_drop_table(XTThreadPtr tab_close_mapped_files(self, tab); tab_delete_table_files(self, tab_name, tab_id); + /* Remove table from "repair-pending" */ + xt_tab_table_repaired(tab); ASSERT(xt_get_self() == self); if ((te_ptr = (XTTableEntryPtr) xt_sl_find(self, db->db_ Regards, Monty

2 1

[Maria-developers] Please consider LP BUG#725050 - related to join_cache_hashed
by Timour Katchaounov 25 Feb '11

25 Feb '11

Igor, the following bug is present in different ways both in 5.3 and in 5.3-mwl89. https://bugs.launchpad.net/maria/+bug/725050 The crash in 5.3-mwl89 appeared after merging with the latest 5.3. The crash in 5.3 requires a rather artificial example, but it shows that there is some inconsistency in JOIN::keyuse in 5.3. Please look at the last bug comment - it contains a very small test case. I will ping you tonight after putting kids to sleep around 21:30. Timour

1 0

[Maria-developers] [psergey@askmonty.org: [Commits] Rev 2921: BUG#723822: Crash in get_constant_key_infix with EXISTS ( SELECT .. DISTINCT ) in file:///home/psergey/dev2/5.3/]
by Sergey Petrunya 25 Feb '11

25 Feb '11

Hi Timour, Could you please review the below? (Asking you because the fix is in the loose index scan optimizer) ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- From: Sergey Petrunya <psergey(a)askmonty.org> To: commits(a)mariadb.org X-Mailer: mail (GNU Mailutils 1.2) Date: Fri, 25 Feb 2011 12:59:00 +0300 (MSK) Subject: [Commits] Rev 2921: BUG#723822: Crash in get_constant_key_infix with EXISTS ( SELECT .. DISTINCT ) in file:///home/psergey/dev2/5.3/ At file:///home/psergey/dev2/5.3/ ------------------------------------------------------------ revno: 2921 revision-id: psergey(a)askmonty.org-20110225095856-qgykwziqv5h1suqr parent: psergey(a)askmonty.org-20110224203200-4uf4e4kngwdmtmgw committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: 5.3 timestamp: Fri 2011-02-25 12:58:56 +0300 message: BUG#723822: Crash in get_constant_key_infix with EXISTS ( SELECT .. DISTINCT ) - Make get_constant_key_infix() take into account that there may be SEL_TREEs with type=SEL_ARG::MAYBE_KEY, which it cannot process. check_quick_select() already has a check for such condition. === modified file 'mysql-test/r/subselect4.result' --- a/mysql-test/r/subselect4.result 2011-02-10 08:36:43 +0000 +++ b/mysql-test/r/subselect4.result 2011-02-25 09:58:56 +0000 @@ -689,3 +689,21 @@ 2 DEPENDENT SUBQUERY X ALL NULL NULL NULL NULL 10 2 DEPENDENT SUBQUERY B ALL a,b NULL NULL NULL 1000 Range checked for each record (index map: 0x3) drop table t1, t2; +# +# BUG#723822: Crash in get_constant_key_infix with EXISTS ( SELECT .. DISTINCT ) +# +CREATE TABLE t1 ( f1 int(11), f3 varchar(1)) ; +INSERT INTO t1 VALUES ('8','c'),('5','f'); +ALTER TABLE t1 ADD KEY (f3,f1); +CREATE TABLE t2 ( f4 varchar(1)) ; +INSERT INTO t2 VALUES ('f'),('d'); +SELECT * FROM t2 +WHERE EXISTS ( +SELECT DISTINCT f3 +FROM t1 +WHERE f3 <= t2.f4 +); +f4 +f +d +drop table t1,t2; === modified file 'mysql-test/t/subselect4.test' --- a/mysql-test/t/subselect4.test 2011-02-10 08:36:43 +0000 +++ b/mysql-test/t/subselect4.test 2011-02-25 09:58:56 +0000 @@ -619,3 +619,24 @@ from t1 A; drop table t1, t2; + +--echo # +--echo # BUG#723822: Crash in get_constant_key_infix with EXISTS ( SELECT .. DISTINCT ) +--echo # +CREATE TABLE t1 ( f1 int(11), f3 varchar(1)) ; +INSERT INTO t1 VALUES ('8','c'),('5','f'); + +ALTER TABLE t1 ADD KEY (f3,f1); + +CREATE TABLE t2 ( f4 varchar(1)) ; +INSERT INTO t2 VALUES ('f'),('d'); + +SELECT * FROM t2 +WHERE EXISTS ( + SELECT DISTINCT f3 + FROM t1 + WHERE f3 <= t2.f4 +); + +drop table t1,t2; + === modified file 'sql/opt_range.cc' --- a/sql/opt_range.cc 2011-02-22 09:15:47 +0000 +++ b/sql/opt_range.cc 2011-02-25 09:58:56 +0000 @@ -11857,7 +11857,8 @@ Find the range tree for the current keypart. We assume that index_range_tree points to the leftmost keypart in the index. */ - for (cur_range= index_range_tree; cur_range; + for (cur_range= index_range_tree; + cur_range && cur_range->type == SEL_ARG::KEY_RANGE; cur_range= cur_range->next_key_part) { if (cur_range->field->eq(cur_part->field)) _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Using tilde in the version for debs?
by Henrik Ingo 24 Feb '11

24 Feb '11

Hi Kristian I've been a bit invisible for a few months now, as I had to spend time on job interviews, family and luckily also some coding! By March life is turning back to normal with a new job and everything. As you know by now, I have in the past months fulfilled my desire to spend some hands on time with the MariaDB build system. I intend to blog about it in the coming week, after which I'll email some MariaDB specific details about what improvements you may want to consider incorporating, and what I think are great improvements but probably not worth focusing on for you in the short term. But just as a quickie, I noticed you are changing the versioning of debs to include the distribution codename (lucid, maverick, etc..): http://bazaar.launchpad.net/~maria-captains/ourdelta/ourdelta-knielsen/revi… What caught my eye was the usage of tilde character for punctuation. In dpkg tilde is a special character with superpowers, it is sorted as greater than anything else: http://man.he.net/man5/deb-version You probably know that, of course, and have a good reason to use it (related to getting upgrades right, perhaps?). I just wanted to comment on it "just in case". "In case" there is no particular reason to use tilde, then of course using a dot would have been more natural. Percona uses a dot http://www.percona.com/downloads/Percona-Server-5.1/Percona-Server-5.1.54-1… ...and I use that with the MepSQL packages too. As a final observation - apart from the different puntuation, the version string seems to be similar in all three of percona, mepsql and ourdelta-knielsen. (So it is kind of a pity if you are forced to use the tilde for some reason - otoh MariaDB versioning is a bit different anyway as it doesn't follow the <mysql-base-ver>-<my-release-ver> approach.) cheers and I'll get back later with more interesting comments. henrik -- henrik.ingo(a)avoinelama.fi +358-40-5697354 skype: henrik.ingo irc: hingo www.openlife.cc My LinkedIn profile: http://www.linkedin.com/profile/view?id=9522559

2 2

Re: [Maria-developers] [Commits] Rev 2875: The patch adds the code that allows to use equi-join conditions in file:///home/igor/maria/maria-5.3-mwl128-hjkey/
by Timour Katchaounov 24 Feb '11

24 Feb '11

Igor, While merging 5.3-mwl89 with 5.3 I noticed that when you extracted a block of code from add_key_part() into the new function add_keyuse(), you skipped the initialization: keyuse.ref_table_rows= 0; was it intentional, and if so, why? Timour > At file:///home/igor/maria/maria-5.3-mwl128-hjkey/ > > ------------------------------------------------------------ > revno: 2875 > revision-id: igor(a)askmonty.org-20110105055941-xht2mpqe66a64fbm > parent: igor(a)askmonty.org-20101230021403-ly4ku1yq675l5d6b > committer: Igor Babaev<igor(a)askmonty.org> > branch nick: maria-5.3-mwl128-hjkey > timestamp: Tue 2011-01-04 21:59:41 -0800 > message: > The patch adds the code that allows to use equi-join conditions > for hash join in the cases when there are no suitable indexes > for these conditions. > > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] Fwd: [Commits] Rev 2921: Added build script which works with ICC v12.0 in file:///home/hakan/work/monty_program/maria-5.2/
by Kristian Nielsen 24 Feb '11

24 Feb '11

Hakan Kuecuekyilmaz <hakan(a)askmonty.org> writes: > Kristian, can you please review the new build script, so that I can > push it into our 5.2 tree. After we cleared the license details with Ok to push. - Kristian. > Begin forwarded message: > >> From: "Hakan Kuecuekyilmaz" <hakan.kuecuekyilmaz(a)t-online.de> >> Date: 10. Februar 2011 16:41:30 MEZ >> To: <commits(a)mariadb.org> >> Subject: [Commits] Rev 2921: Added build script which works with ICC v12.0 in file:///home/hakan/work/monty_program/maria-5.2/ >> Reply-To: maria-developers(a)lists.launchpad.net >> >> At file:///home/hakan/work/monty_program/maria-5.2/ >> >> ------------------------------------------------------------ >> revno: 2921 >> revision-id: hakan(a)askmonty.org-20110210154128-ncaubaj5i8rhf2pe >> parent: monty(a)askmonty.org-20110203144529-ah7majfmqscp4yn1 >> committer: Hakan Kuecuekyilmaz <hakan(a)askmonty.org> >> branch nick: maria-5.2 >> timestamp: Thu 2011-02-10 16:41:28 +0100 >> message: >> Added build script which works with ICC v12.0 >> === added file 'BUILD/compile-pentium64-icc-max' >> --- a/BUILD/compile-pentium64-icc-max 1970-01-01 00:00:00 +0000 >> +++ b/BUILD/compile-pentium64-icc-max 2011-02-10 15:41:28 +0000 >> @@ -0,0 +1,25 @@ >> +#! /bin/sh >> + >> +path=`dirname $0` >> +. "$path/SETUP.sh" >> + >> +# Note that we can't use ccache with icc as the generated .deps file will >> +# then contain wrong information. >> +CC=icc >> +CXX=icpc >> +LD=xild >> +AR=xiar >> +export CC CXX LD AR >> + >> +c_warnings="" >> +cxx_warnings="" >> +extra_flags="-O2 -unroll2 -fno-exceptions -fno-rtti -fno-omit-frame-pointer -static-intel -static-libgcc -fPIE" >> + >> +# Use -no-ipo if you get this error >> +# IPO link: can not find "-lstdc++_shared" >> +# icpc: error: problem during multi-file optimization compilation (code 1) >> +base_cxxflags="$extra_flags" >> +extra_configs="$pentium_configs" >> + >> +. "$path/FINISH.sh" >> +

1 0

[Maria-developers] WL#177 New (by Mdcallag): add server-side error logging
by worklog-noreply＠askmonty.org 22 Feb '11

22 Feb '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: add server-side error logging CREATION DATE..: Tue, 22 Feb 2011, 19:42 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 177 (http://askmonty.org/worklog/?tid=177) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: I want to log all or a sample of errors on the server side into a new log file (not general, error or slow query log) and into a file that can be rotated. In the facebook patch I added code to sample query errors into the slow query log using - -log-sample-error-rate: * when 0 do not sample * when 1 then log all errors * when > 1 then sample 1 of --log-sample-error-rate queries This should have: * query text * username * timestamp * database * errno * error text * client IP The log should support rotation and possibly auto-rotation when it reaches a certain size. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

Re: [Maria-developers] Fwd: [Commits] Rev 2901: BNLH algorithm always used a full table scan over the joined table in file:///home/igor/maria/maria-5.3-mwl128-hashrange/
by Sergey Petrunya 22 Feb '11

22 Feb '11

On Fri, Feb 11, 2011 at 08:49:37PM -0800, Igor Babaev wrote: > Hi Sergey, > > Please review this patch that fixes a defect of the current > implementation of the mwl #128 that never couples hash join > with range/index_merge scans. > The patch also adds prefix #hash# to the names of the used hash indexes. > (we agreed upon this at the last optimizer meeting). > > Regards, > Igor. > > -------- Original Message -------- > Subject: [Commits] Rev 2901: BNLH algorithm always used a full table > scan over the joined table in > file:///home/igor/maria/maria-5.3-mwl128-hashrange/ > Date: Fri, 11 Feb 2011 20:41:23 -0800 (PST) > From: Igor Babaev <igor(a)askmonty.org> > Reply-To: maria-developers(a)lists.launchpad.net > To: <commits(a)mariadb.org> > > At file:///home/igor/maria/maria-5.3-mwl128-hashrange/ > > ------------------------------------------------------------ > revno: 2901 > revision-id: igor(a)askmonty.org-20110212044122-w9n3jdk3d2ps0w3o > parent: igor(a)askmonty.org-20110207231903-3rqbfs50d33lk3r9 > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-mwl128-hashrange > timestamp: Fri 2011-02-11 20:41:22 -0800 > message: > BNLH algorithm always used a full table scan over the joined table > even in the cases when there existed range/index-merge scans that > were cheaper than the full table scan. > This was a defect/bug of the implementation of mwl #128. > Now hash join can work not only with full table scan of the joined > table, but also with full index scan, range and index-merge scans. > Accordingly, in the cases when hash join is used the column 'type' > in the EXPLAINs can contain now 'hash_ALL', 'hash_index', 'hash_range' > and 'hash_index_merge'. If hash join is coupled with a range/index_merge > scan then the columns 'key' and 'key_len' contain info not only on > the used hash index, but also on the indexes used for the scan. ... > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2011-02-06 04:57:03 +0000 > +++ b/sql/sql_select.cc 2011-02-12 04:41:22 +0000 ... > @@ -8419,6 +8438,11 @@ > sort_by_tab->type= JT_ALL; > sort_by_tab->read_first_record= join_init_read_record; > } > + else if (sort_by_tab->type == JT_HASH_NEXT) > + { > + sort_by_tab->type= JT_HASH; > + sort_by_tab->read_first_record= join_init_read_record; > + } > } > break; > } I have a question only to this part of the patch: I don't understand how the above code could work. The if statement above the one you've added checks if sort_by_tab was using [ordered] index scan to produce records in orderer, and if yes, switches it to a full scan (because use of join buffering will break ordering anyway). I suppose the part you've added does something similar, but I totally fail to understand what exactly it does. Now, some comments on the new EXPLAIN output: ISSUE1. Consider a query: MariaDB [j1]> explain select * from t3, t4 where t3.col1=t4.col2; +----+-------------+-------+----------+---------------+-----------+---------+------------+------+--------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+----------+---------------+-----------+---------+------------+------+--------------------------------------------------+ | 1 | SIMPLE | t3 | ALL | NULL | NULL | NULL | NULL | 1000 | Using where | | 1 | SIMPLE | t4 | hash_ALL | NULL | #hash#$hj | 5 | j1.t3.col1 | 1000 | Using where; Using join buffer (flat, BNLH join) | +----+-------------+-------+----------+---------------+-----------+---------+------------+------+--------------------------------------------------+ 2 rows in set (0.00 sec) AFAIU #hash#$hj means the hash key on the hashtable that's used for hash join. I think the name is too convoluted. The hashtable has only one key, if we need to tell it from t4's indexes it would be sufficient to use "#hashkey#", without the cryptic "$hj". As for "#" characters: EXPLAIN already shows "internal" tables/indexes: MariaDB [j1]> explain select * from (select * from t1) X; +----+-------------+------------+------+---------------+------+---------+------+------+-------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+------+---------------+------+---------+------+------+-------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 1000 | | | 2 | DERIVED | t1 | ALL | NULL | NULL | NULL | NULL | 1000 | | +----+-------------+------------+------+---------------+------+---------+------+------+-------+ # The following is from MWL#90 tree: MariaDB [test]> explain select * from t10 where a in (select max(b) from t10 group by a); +----+-------------+-------------+--------+---------------+--------------+---------+------------+------+---------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------------+--------+---------------+--------------+---------+------------+------+---------------------------------+ | 1 | PRIMARY | t10 | ALL | NULL | NULL | NULL | NULL | 1000 | Using where | | 1 | PRIMARY | <subquery2> | eq_ref | distinct_key | distinct_key | 5 | test.t10.a | 1 | | | 2 | SUBQUERY | t10 | ALL | NULL | NULL | NULL | NULL | 1000 | Using temporary; Using filesort | +----+-------------+-------------+--------+---------------+--------------+---------+------------+------+---------------------------------+ 3 rows in set (0.01 sec) # The following is from current 5.3: MariaDB [j1]> explain select distinct * from t1 where a in (select b from t2); +----+-------------+------------+--------+---------------+------------+---------+------+------+-----------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+--------+---------------+------------+---------+------+------+-----------------+ | 1 | PRIMARY | t1 | ALL | NULL | NULL | NULL | NULL | 1000 | Using temporary | | 1 | PRIMARY | subselect2 | eq_ref | unique_key | unique_key | 5 | func | 1 | | | 2 | SUBQUERY | t2 | ALL | NULL | NULL | NULL | NULL | 1000 | Distinct | +----+-------------+------------+--------+---------------+------------+---------+------+------+-----------------+ ... and it either uses no quoting at all (see "unique_key"), or angle brackets quoting (see <subquery2>). I think we should not multiply the ways of quoting, so please consider switching to <quoting> or to not using any quoting at all. ISSUE2 MariaDB [j1]> EXPLAIN -> SELECT City.Name, Country.Name, CountryLanguage.Language -> FROM City,Country,CountryLanguage -> WHERE City.Country=Country.Code AND -> CountryLanguage.Country=Country.Code AND -> City.Name LIKE 'L%' AND Country.Population > 3000000 AND -> CountryLanguage.Percentage > 50 AND -> LENGTH(Language) < LENGTH(City.Name) - 2; +----+-------------+-----------------+----------+--------------------+---------------+---------+----------------------------+------+--------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-----------------+----------+--------------------+---------------+---------+----------------------------+------+--------------------------------------------------+ | 1 | SIMPLE | CountryLanguage | ALL | PRIMARY,Percentage | NULL | NULL | NULL | 984 | Using where | | 1 | SIMPLE | Country | hash_ALL | PRIMARY | #hash#PRIMARY | 3 | j1.CountryLanguage.Country | 239 | Using where; Using join buffer (flat, BNLH join) | | 1 | SIMPLE | City | hash_ALL | Country | #hash#Country | 3 | j1.CountryLanguage.Country | 4079 | Using where; Using join buffer (flat, BNLH join) | +----+-------------+-----------------+----------+--------------------+---------------+---------+----------------------------+------+--------------------------------------------------+ 3 rows in set (0.01 sec) "key" column shows "#hash#PRIMARY". I suppose, this is because hash join opimizer is internally hooked into ref optimizer, and it used the PRIMARY index. This might be useful for developer, but is this information meaningful for the user? If one changes #hash#PRIMARY to #hash#IDX2 (suppose there is an index called IDX2 which has the same definition as primary key), or to #hash#$hj, will it be a different query plan? I think it should be "<hashkey>" always. ISSUE3 For hash_range, "key" column shows {hashkey}:{range_key}, which allows to determine that which key was used for range access. For hash_index type, "key" column shows only the hash's pseudo-index: MariaDB [j3]> EXPLAIN SELECT t2.i FROM t1,t2 WHERE t1.cu = t2.cl; +----+-------------+-------+------------+---------------+-----------+---------+------+------+--------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------------+---------------+-----------+---------+------+------+--------------------------------------------------+ | 1 | SIMPLE | t2 | ALL | NULL | NULL | NULL | NULL | 6 | | | 1 | SIMPLE | t1 | hash_index | cu1,cu2 | #hash#cu1 | 33 | func | 10 | Using where; Using join buffer (flat, BNLH join) | +----+-------------+-------+------------+---------------+-----------+---------+------+------+--------------------------------------------------+ What is the index that's used for scanning the table t1? "#hash#cu1" shows that hash join optimizer used access on index "cu1" in its analysis, but generally it doesn't mean that full index scan on table t1 was done on index cu1, doesn't it? I would very much prefer to see #hash#cu1:USED_INDEX there (or taking into account previous suggestions, hashkey:USED_INDEX). It will also be consistent with how hash_range and hash_index_merge use that column. ISSUE4 If key and key_len use hash_part:real_part notation, i.e. use semicolon, why use underscore in hash_ALL, hash_index, etc? Won't hash:ALL, hash:index look more consistent with other columns? BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 2

Re: [Maria-developers] [Commits] Rev 2833: Fixed LP bug #664508. in file:///home/igor/maria/maria-5.3-mwl128-bug664508/
by Sergey Petrunya 21 Feb '11

21 Feb '11

Hello Igor, On Sun, Oct 24, 2010 at 02:22:02PM -0700, Igor Babaev wrote: > At file:///home/igor/maria/maria-5.3-mwl128-bug664508/ > > ------------------------------------------------------------ > revno: 2833 > revision-id: igor(a)askmonty.org-20101024212201-t8iius4jikcvm3zb > parent: igor(a)askmonty.org-20101022223047-phunrh5g3xzrx2d9 > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-mwl128-bug664508 > timestamp: Sun 2010-10-24 14:22:01 -0700 > message: > Fixed LP bug #664508. > When join buffers are employed no index scan for the first > table with grouping columns can be used. ... > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2010-10-18 20:56:48 +0000 > +++ b/sql/sql_select.cc 2010-10-24 21:22:01 +0000 > @@ -7980,8 +7980,11 @@ > JOIN_TAB *tab=join->join_tab+i; > if (tab->use_join_cache) > { > - JOIN_TAB *sort_by_tab= join->get_sort_by_join_tab(); > - if (sort_by_tab && !join->need_tmp) > + JOIN_TAB *sort_by_tab= join->group && join->simple_group && > + join->group_list ? > + join->join_tab+join->const_tables : > + join->get_sort_by_join_tab(); > + if (sort_by_tab) > { > join->need_tmp= 1; > join->simple_order= join->simple_group= 0; Here sort_by_tab is assigned inside the " for (i=join->const_tables ; ..." loop. I think the code will look much more cleaner if it was calculated before the loop (and actually we don't need to loop at all if sort_by_tab==NULL). > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB Binary Log File Corruption
by Daniel Bartholomew 20 Feb '11

20 Feb '11

Hello everyone, I came across this question in the Knowledgebase: http://kb.askmonty.org/v/binary-log-file-corruption-happens-each-day The poster was able to solve the problem by increasing the slave max_allowed_packet size from 32M to 256M. Any thoughts from the list about this? My guess is that this not a MariaDB-specific thing; just a difference between 5.0 and 5.1 (and higher). But I'm asking anyway in case I'm mistaken and/or if a note should be added to the documentation for max_allowed_packet. Thanks. -- Daniel Bartholomew MariaDB - http://mariadb.org Monty Program - http://montyprogram.com AskMonty Knowledgebase - http://kb.askmonty.org

2 1

[Maria-developers] question about when locks are released
by MARK CALLAGHAN 18 Feb '11

18 Feb '11

Vamsi, In the innodb-release-locks-early patch, do you remember why you release the locks only _after_ the sync in prepare(), and not before? -- Mark Callaghan mdcallag(a)gmail.com

1 1

[Maria-developers] WL#176 New (by Dreas): protocl typo in replication error
by worklog-noreply＠askmonty.org 17 Feb '11

17 Feb '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: protocl typo in replication error CREATION DATE..: Thu, 17 Feb 2011, 16:58 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Maria-RawIdeaBin TASK ID........: 176 (http://askmonty.org/worklog/?tid=176) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 10 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Using 5.1.50-mariadb84: ===== Last_IO_Error: error connecting to master 'repl2@mysql1.*****.com:3306' - retry- time: 60 retries: 86400 message: SSL error: protocl version mismatch ==== "protocl" seems to be a typo. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] work in progress for the Facebook patch
by MARK CALLAGHAN 16 Feb '11

16 Feb '11

I am writing this to describe work in progress for the Facebook patch. Comments are welcome. In the future we hope to get parts of the Facebook patch into public forks. We have two big projects in progress: * admission control * InnoDB compression for OLTP --- Admission control enforces max concurrent queries per account. MySQL already limits max concurrent connections per account, but we want to set the connection limit to a much larger value than the query limit. The limit will be enforced at statement start and will allow queries to timeout in the queue while waiting to run. By "descheduled" I mean that a connection decrements the count of concurrent queries for a particular account. Connections will be descheduled when: * a statement ends * a connection is about to send results back to a client -- only when the socket write is to be done, not when data is buffered * srv_suspend_mysql_thread is called on a row lock wait * possibly when a disk read is done within innodb * when enter_cond is called By "scheduled" I mean that a connection increments the count of concurrent queries for an account. If done with "noforce" then the caller will wait until this doesn't exceed concurrency limits for the account. If done with "force" then this doesn't wait and can exceed the concurrency limits. Schedule with noforce is done: * at statement start * after a socket write Schedule with force is done: * when exit_cond is called * when a thread wakes after calling srv_suspend_mysql_thread This change will allow us to run with higher values for max concurrent connections and we assume that will significantly increase mutex contention on kernel_mutex so we have begun to evaluate: * using the pthread mutex implementation that does not spin in more cases * limit the max number of threads that can do concurrent busy-wait loops for InnoDB's mutex and rw-lock * removing use of the sync array within InnoDB * changing Innodb to wake one rather than all waiters when a mutex is unlocked * somehow making read_view_open_now faster --- InnoDB compression for OLTP tries to make compression faster for OLTP workloads. We have work in progress to change InnoDB to not log page images for many but not all insert/update/delete operations. When InnoDB compression is used, InnoDB can log page images. That was not done before and for my production workloads this doubles the rate at which IO is done to the transaction log and increases the dirty page write rate by at least 20% because checkpoint IO must be done sooner. We previously added an option that allows the LZ compression level to be set globally. I think the default is level 6. We will evaluate: * using QuickLZ which provides faster compression and decompression than zlib * adding compress_ops and compress_ops_ok per table to information_schema.table_statistics * adding an option to limit compression to the clustered index and not compress secondary indexes -- Mark Callaghan mdcallag(a)gmail.com

1 0

[Maria-developers] MariaDB and MySQL 5.5
by MARK CALLAGHAN 11 Feb '11

11 Feb '11

When will there be a MariaDB branch based on MySQL 5.5? I want to use that as the basis for the 5.5 version of the Facebook patch. -- Mark Callaghan mdcallag(a)gmail.com

2 1

[Maria-developers] WL#175 New (by Mdcallag): Unique ID in binlog event metadata
by worklog-noreply＠askmonty.org 11 Feb '11

11 Feb '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Unique ID in binlog event metadata CREATION DATE..: Fri, 11 Feb 2011, 05:03 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 175 (http://askmonty.org/worklog/?tid=175) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: While I would love to have global transaction IDs for MySQL replication (http://askmonty.org/worklog/Server-RawIdeaBin/?tid=31) that is a huge project. I think we can automate slave failover if binlog events have a metadata attribute that is a unique ID and that metadata is sticky like server ID when a slave is run with --log-slave-updates. Assuming this metadata is added, when it is time to do slave failover, first determine the serverID + uniqueID for the last binlog event group executed on the slave, then find that filename/offset for that event on the new master, then CHANGE MASTER on the slave to start replication on the new master just past that offset. I mention that serverID can be used in addition to the uniqueID, so maybe uniqueID only needs to be unique within the binlog events generated for one serverID. The search on the new master is more efficient when the uniqueID are generated in increasing order. But I am not sure whether that is required. We might need a new server command that maps (serverID, uniqueID) to (filename, offset), but I am not sure that is required. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

Re: [Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 08 Feb '11

08 Feb '11

> -----Original Message----- > From: kahzoo(a)gmail.com [mailto:kahzoo@gmail.com] On Behalf Of Kazuhisa > Ichikawa > Sent: Dienstag, 8. Februar 2011 04:46 > To: Vladislav Vaintroub > Cc: Philip Stoev; maria-developers(a)lists.launchpad.net > Subject: Re: [Maria-developers] Windows installer MWL#55 finished. > > Hello, Vladislav. > All previous 5.2.5 installer created its data directory even when > upgrading existing instance. Therefore, if datadir was not set > explicity in the inherited my.ini, it was using the new data directory > (in other words, it was not reusing the previous database files, I'm > afraid). Yes. Somewhat unusual case my.ini without datadir was not handled correctly though (case without my.ini at all was handled). Fixed that, MSIs are in the usual location. Thanks for input! > - Kazuhisa

1 0

Re: [Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 05 Feb '11

05 Feb '11

> -----Original Message----- > From: kahzoo(a)gmail.com [mailto:kahzoo@gmail.com] On Behalf Of Kazuhisa > Ichikawa > Sent: Samstag, 5. Februar 2011 04:34 > To: Vladislav Vaintroub > Cc: maria-developers(a)lists.launchpad.net > Subject: Re: [Maria-developers] Windows installer MWL#55 finished. Hello Kazuhisa, > I BROWSED through it but did not read it carefully line by line before > reporting bugs. Sorry about that. Yet, all of your reports were valid . > > By the way, after finished installing 5.2.5 and upgrading the previous > (5.1.53) service instance, I have 2 service definitions for 5.2.5, one > that was newly created and the other that was successfully upgraded > from 5.1.53. > > I think this is a human (that is my) error rather than a bug, because > I checked "Install as service" in the "Defaults settings" dialog > before the installer starts installing 5.2.5 files and then chose to > upgrade the existing service in the Upgrade Wizard that was started > after installation. If you think installing instance was a mistake on you part (you only wanted to upgrade what you have), you can correct that easily. In maintenance mode install (Add/Remove Programs=>MariaDB5.2=>Change/Uninstall), select "Change" button. Deselect "Database Instance". Choose "Remove data" afterwards. Now the mistakenly installed instance is gone. > My expectation was that the installer and upgrade > wizard will install a new service instance based on the information > from the existing service definition. Hmm, have you see that in any of MySQL installers ;)? > If I could (mutually) exclusively select between "installing a new > service" and "upgrading from existing service" in the same dialog, I > could have avoided this mis-configuration. I understand, and agree from usability point of view this would be ideal. Currently, "Database Instance" feature can only create new databases, while "Upgrade Wizard" (runs outside of the installer) upgrades existing instances. Their functionality is non-overlapping. This allows for painless (for user) and simple to implement installer rollback in case of errors. So it will stay this way for the version 1 of the new installer. One thing that is possible to do to handle your case, is to deselect "database instance" by default, if upgradable services are detected. I'm not sure this is the best way to handle it , and whether it would confuse the user thus I'll consult our QA for his opinion. Philip, what do you think? Vladislav

1 0

Re: [Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 04 Feb '11

04 Feb '11

> -----Original Message----- > From: kahzoo(a)gmail.com [mailto:kahzoo@gmail.com] On Behalf Of Kazuhisa Ichikawa > Sent: Freitag, 4. Februar 2011 03:17 > To: Vladislav Vaintroub > Cc: maria-developers(a)lists.launchpad.net > Subject: Re: [Maria-developers] Windows installer MWL#55 finished. > > Hello Vladislav, Hello Kazuhisa, > First of all, thanks for all the information on "MSI v.s. NSIS" issues. > > > On a bright side, this is a valid bug that was simple to fix forcing per-machine install, so installer does not need to guess. I > > did that and refreshed the MSI on the download page. > > I confirmed the fix with the same machine/user with which I had the > original problem. Thanks. > > However, I'm afraid I hit a new one. If I have MariaDB 5.1.53 > installed (by the 5.1 installer) on the same machine, the new > installer doesn't allow me to install 5.2.5 saying: > "Previous version of MariaDB was found, that used incompatible > installer. Please remove "MariaDB 5.1.53" before you proceed with this > installation." Ok, valid bug. Thanks for reporting. There will *still* be this message if you're trying to install 5.2.5 msi with existing 5.2.(<5) NSIS . For me personally ,main reason NOT to offer 5.2.4=>5.2.5 autoupgrade is the fact that MariaDBs NSIS was not designed to run in silent mode, its popups "Directory not deleted" during upgrade would only alienate users . > I think this is not an intended behavior, because this kills the big > benefit (of your new installer) of automatically upgrading older > MariaDB instance. The upgrade is not always automatic, sometimes it is semiautomatic (supported by the UI tool launched at the end of installation). Did you look in the WL on how upgrades are handled? Here is link just in case http://askmonty.org/worklog/Server-Sprint/?tid=55 . Feedback on it is welcome. I have fixed 5.2.5 over 5.1 NSIS installation ,the update is on usual place. http://cid-ff0c950417b4f8a4.office.live.com/browse.aspx/share Thanks for your efforts, Vladislav

2 3

Re: [Maria-developers] [Commits] Rev 2892: BUG#711565 Index Condition Pushdown can make a thread hold MyISAM locks as well as be unKILLable for long time in file:///home/psergey/dev2/5.3/
by Michael Widenius 03 Feb '11

03 Feb '11

Hi! >>>>> "Sergey" == Sergey Petrunya <psergey(a)askmonty.org> writes: Sergey> At file:///home/psergey/dev2/5.3/ Sergey> ------------------------------------------------------------ Sergey> revno: 2892 Sergey> revision-id: psergey(a)askmonty.org-20110201233314-6j8yc7jrx9425f3b Sergey> parent: sergii(a)pisem.net-20110114205155-i8poq0jti8o8mob1 Sergey> committer: Sergey Petrunya <psergey(a)askmonty.org> Sergey> branch nick: 5.3 Sergey> timestamp: Wed 2011-02-02 02:33:14 +0300 Sergey> message: Sergey> BUG#711565 Index Condition Pushdown can make a thread hold MyISAM locks as well as be unKILLable for long time <cut> Sergey> === modified file 'mysql-test/t/myisam_icp.test' Sergey> --- a/mysql-test/t/myisam_icp.test 2011-01-14 13:38:41 +0000 Sergey> +++ b/mysql-test/t/myisam_icp.test 2011-02-01 23:33:14 +0000 Sergey> @@ -4,3 +4,183 @@ Sergey> --source include/icp_tests.inc <cut> Instead of using # for each row, please use --disable_parsing --enable_parsing This makes it easier to enable the test temporarly when testing. <cut> Sergey> === modified file 'sql/multi_range_read.cc' Sergey> --- a/sql/multi_range_read.cc 2010-12-21 23:26:35 +0000 Sergey> +++ b/sql/multi_range_read.cc 2011-02-01 23:33:14 +0000 Sergey> @@ -902,7 +902,7 @@ Sergey> close_second_handler(); Sergey> /* Safety, not really needed but: */ Sergey> strategy= NULL; Sergey> - DBUG_RETURN(1); Sergey> + DBUG_RETURN(res); Sergey> use_default_impl: Sergey> DBUG_ASSERT(primary_file->inited == handler::INDEX); Sergey> === modified file 'sql/sql_base.cc' Sergey> --- a/sql/sql_base.cc 2010-12-06 08:25:44 +0000 Sergey> +++ b/sql/sql_base.cc 2011-02-01 23:33:14 +0000 Sergey> @@ -8283,11 +8283,9 @@ Sergey> List_iterator_fast<Item> f(fields),v(values); Sergey> Item *value, *fld; Sergey> Item_field *field; Sergey> - TABLE *table= 0; Sergey> - List<TABLE> tbl_list; Sergey> + TABLE *table= 0, *vcol_table= 0; Sergey> bool abort_on_warning_saved= thd->abort_on_warning; Sergey> DBUG_ENTER("fill_record"); Sergey> - tbl_list.empty(); Sergey> /* Sergey> Reset the table->auto_increment_field_not_null as it is valid for Sergey> @@ -8310,7 +8308,7 @@ Sergey> f.rewind(); Sergey> } Sergey> else if (thd->lex->unit.insert_table_with_stored_vcol) Sergey> - tbl_list.push_back(thd->lex->unit.insert_table_with_stored_vcol); Sergey> + vcol_table= thd->lex->unit.insert_table_with_stored_vcol; Sergey> while ((fld= f++)) Sergey> { Sergey> if (!(field= fld->filed_for_view_update())) Sergey> @@ -8340,31 +8338,17 @@ Sergey> my_message(ER_UNKNOWN_ERROR, ER(ER_UNKNOWN_ERROR), MYF(0)); Sergey> goto err; Sergey> } Sergey> - tbl_list.push_back(table); Sergey> + DBUG_ASSERT(vcol_table == 0 || vcol_table == table); Sergey> + vcol_table= table; Sergey> } Sergey> /* Update virtual fields*/ Sergey> thd->abort_on_warning= FALSE; Sergey> - if (tbl_list.head()) Sergey> + if (vcol_table) Sergey> { Sergey> - List_iterator_fast<TABLE> it(tbl_list); Sergey> - TABLE *prev_table= 0; Sergey> - while ((table= it++)) Sergey> + if (vcol_table->vfield) Sergey> { Sergey> - /* Sergey> - Do simple optimization to prevent unnecessary re-generating Sergey> - values for virtual fields Sergey> - */ Sergey> - if (table != prev_table) Sergey> - { Sergey> - prev_table= table; Sergey> - if (table->vfield) Sergey> - { Sergey> - if (update_virtual_fields(thd, table, TRUE)) Sergey> - { Sergey> - goto err; Sergey> - } Sergey> - } Sergey> - } Sergey> + if (update_virtual_fields(thd, vcol_table, TRUE)) Sergey> + goto err; Sergey> } Sergey> } Sergey> thd->abort_on_warning= abort_on_warning_saved; What has the above to do with the IPC patch? In any case, how will the above work when there is more than one table involved in fill_tables ? (By the way, agree that the old code as stupid; There is no reason to add all involved tables to tbl_list (trivial to avoid). Sergey> === modified file 'sql/sql_select.cc' Sergey> --- a/sql/sql_select.cc 2011-01-14 11:03:41 +0000 Sergey> +++ b/sql/sql_select.cc 2011-02-01 23:33:14 +0000 Sergey> @@ -13618,13 +13618,17 @@ Sergey> " cond: %p error: %d", join, join_tab, select_cond, error)); Sergey> if (error > 0 || (join->thd->is_error())) // Fatal error Sergey> DBUG_RETURN(NESTED_LOOP_ERROR); Sergey> + /* Sergey> + Check if we're killed before checking for EOF. This is important because Sergey> + if we're killed while reading a record, the handler will return EOF. Sergey> + */ Please update the comment; We should not send EOF from the handler if the query is killed; There is too many things that can go wrong if we send EOF on killed from the handler. Sergey> + if (join->thd->killed) // Aborted by user Sergey> + { Sergey> + join->thd->send_kill_message(); Sergey> + DBUG_RETURN(NESTED_LOOP_KILLED); /* purecov: inspected */ Sergey> + } <cut> Sergey> === modified file 'storage/maria/ma_rkey.c' Sergey> --- a/storage/maria/ma_rkey.c 2011-01-14 11:05:46 +0000 Sergey> +++ b/storage/maria/ma_rkey.c 2011-02-01 23:33:14 +0000 Sergey> @@ -34,7 +34,7 @@ Sergey> HA_KEYSEG *last_used_keyseg; Sergey> uint32 nextflag; Sergey> MARIA_KEY key; Sergey> - int icp_res= 1; Sergey> + ICP_RESULT icp_res= ICP_MATCH; Sergey> DBUG_ENTER("maria_rkey"); Sergey> DBUG_PRINT("enter", ("base: 0x%lx buf: 0x%lx inx: %d search_flag: %d", Sergey> (long) info, (long) buf, inx, search_flag)); Sergey> @@ -115,7 +115,7 @@ Sergey> not satisfied with an out-of-range condition. Sergey> */ Sergey> if ((*share->row_is_visible)(info) && Sergey> - ((icp_res= ma_check_index_cond(info, inx, buf)) != 0)) Sergey> + ((icp_res= ma_check_index_cond(info, inx, buf)) != ICP_NO_MATCH)) Sergey> break; Sergey> /* The key references a concurrently inserted record. */ Sergey> @@ -145,6 +145,19 @@ Sergey> if (_ma_search_next(info, &lastkey, maria_readnext_vec[search_flag], Sergey> info->s->state.key_root[inx])) Sergey> break; /* purecov: inspected */ Sergey> + Sergey> + /* Sergey> + If we are at the last key on the key page, allow writers to Sergey> + access the index. Sergey> + */ Sergey> + if (info->int_keypos >= info->int_maxpos && Sergey> + ma_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + buf= 0; /* Fast abort */ Sergey> + icp_res= ICP_ERROR; We should not return an IPC_ERROR here as this test has nothing to do with IPC. It can also happens when we are scanning for a visible row. Better to just threat this as a normal handler error. See later comments... Sergey> + break; Sergey> + } Sergey> + Sergey> /* Sergey> Check that the found key does still match the search. Sergey> _ma_search_next() delivers the next key regardless of its Sergey> @@ -168,9 +181,9 @@ Sergey> if (share->lock_key_trees) Sergey> rw_unlock(&keyinfo->root_lock); Sergey> - if (info->cur_row.lastpos == HA_OFFSET_ERROR || (icp_res != 1)) Sergey> + if (info->cur_row.lastpos == HA_OFFSET_ERROR || (icp_res != ICP_MATCH)) Sergey> { Sergey> - if (icp_res == 2) Sergey> + if (icp_res == ICP_OUT_OF_RANGE || icp_res == ICP_ERROR) Sergey> { Sergey> info->cur_row.lastpos= HA_OFFSET_ERROR; Sergey> my_errno= HA_ERR_KEY_NOT_FOUND; We should not return HA_ERR_KEY_NOT_FOUND in the case of a real error. (Too big risk we loose the reason that something went wrong). Sergey> +/* Sergey> + Yield to possible other writers during a index scan. Sergey> + Check also if we got killed by the user and if yes, return Sergey> + HA_ERR_LOCK_WAIT_TIMEOUT Sergey> + Sergey> + return 0 ok Sergey> + return 1 Query has been requested to be killed Sergey> +*/ Sergey> + Sergey> +my_bool ma_yield_and_check_if_killed(MARIA_HA *info, int inx) Sergey> +{ Sergey> + MARIA_SHARE *share; Sergey> + if (ma_killed(info)) Sergey> + { Sergey> + //info->lastpos= HA_OFFSET_ERROR; Sergey> + my_errno= HA_ERR_KEY_NOT_FOUND; Sergey> + return 1; The above is not safe. Please use: info->cur_row.lastpos= HA_OFFSET_ERROR; my_errno= HA_ERR_ABORTED_BY_USER; In this case we can not return HA_ERR_KEY_NOT_FOUND as it's too big risk that the calling code may think that there was really no key and send a result to the client instead of aborting. By sending a fatal error message to the user, we can be sure that the calling code will not ignore it. That is why we probably should introduce HA_ERR_ABORTED_BY_USER. If we do this, then we have to fix that handler::print_error() ignores this error if thd->killed is set. dispatch_command will generate the necessary error message. Sergey> + } Sergey> + Sergey> + if ((share= info->s)->lock_key_trees) Sergey> + { Sergey> + /* Give writers a chance to access index */ Sergey> + rw_unlock(&share->keyinfo[inx].root_lock); Sergey> + rw_rdlock(&share->keyinfo[inx].root_lock); Sergey> + } Sergey> + return 0; Sergey> +} Sergey> + Sergey> === modified file 'storage/maria/ma_rnext.c' Sergey> --- a/storage/maria/ma_rnext.c 2009-12-15 07:16:46 +0000 Sergey> +++ b/storage/maria/ma_rnext.c 2011-02-01 23:33:14 +0000 <cut> Sergey> + if (info->int_keypos >= info->int_maxpos && Sergey> + ma_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + error= 0; Sergey> + icp_res= ICP_ERROR; Sergey> + break; Sergey> + } Same thing as above, don't use IPC_ERROR, treat this as a real error! Sergey> + Sergey> /* Skip rows inserted by other threads since we got a lock */ Sergey> if ((error= _ma_search_next(info, &info->last_key, Sergey> SEARCH_BIGGER, Sergey> @@ -108,10 +120,10 @@ Sergey> info->update&= (HA_STATE_CHANGED | HA_STATE_ROW_CHANGED); Sergey> info->update|= HA_STATE_NEXT_FOUND; Sergey> - if (icp_res == 2) Sergey> + if (icp_res == ICP_OUT_OF_RANGE) Sergey> my_errno=HA_ERR_END_OF_FILE; /* got beyond the end of scanned range */ Sergey> - if (error || icp_res != 1) Sergey> + if (error || icp_res != ICP_MATCH) Sergey> { Sergey> if (my_errno == HA_ERR_KEY_NOT_FOUND) Sergey> my_errno=HA_ERR_END_OF_FILE; Don't return HA_ERR_END_OF_FILE or HA_ERR_KEY_NOT_FOUND for IPC_ERROR Sergey> === modified file 'storage/maria/ma_rnext_same.c' Sergey> --- a/storage/maria/ma_rnext_same.c 2010-10-30 01:59:39 +0000 Sergey> +++ b/storage/maria/ma_rnext_same.c 2011-02-01 23:33:14 +0000 Sergey> @@ -30,7 +30,7 @@ Sergey> int error; Sergey> uint inx,not_used[2]; Sergey> MARIA_KEYDEF *keyinfo; Sergey> - int icp_res= 1; Sergey> + ICP_RESULT icp_res= ICP_MATCH; Sergey> DBUG_ENTER("maria_rnext_same"); Sergey> if ((int) (inx= info->lastinx) < 0 || Sergey> @@ -80,9 +80,20 @@ Sergey> info->cur_row.lastpos= HA_OFFSET_ERROR; Sergey> break; Sergey> } Sergey> + /* Sergey> + If we are at the last key on the key page, allow writers to Sergey> + access the index. Sergey> + */ Sergey> + if (info->int_keypos >= info->int_maxpos && Sergey> + ma_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + error= 0; Sergey> + icp_res= ICP_ERROR; Sergey> + break; Sergey> + } Same thing as above, don't use IPC_ERROR, treat this as a real error! <cut> Sergey> - if (error || icp_res != 1) Sergey> + if (error || icp_res != ICP_MATCH) Sergey> { Sergey> if (my_errno == HA_ERR_KEY_NOT_FOUND) Sergey> my_errno=HA_ERR_END_OF_FILE; Don't return HA_ERR_END_OF_FILE or HA_ERR_KEY_NOT_FOUND for IPC_ERROR Sergey> === modified file 'storage/maria/ma_rprev.c' <cut> Sergey> + if (ma_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + error= 0; Sergey> + icp_res= ICP_ERROR; Sergey> + break; Sergey> + } Return a real error! Sergey> + } Sergey> + Sergey> /* Skip rows that are inserted by other threads since we got a lock */ Sergey> if ((error= _ma_search_next(info, &info->last_key, Sergey> SEARCH_SMALLER, Sergey> @@ -66,6 +84,10 @@ Sergey> } Sergey> if (share->lock_key_trees) Sergey> rw_unlock(&keyinfo->root_lock); Sergey> + if (icp_res == ICP_ERROR || icp_res == ICP_OUT_OF_RANGE) Sergey> + { Sergey> + DBUG_RETURN(my_errno= HA_ERR_END_OF_FILE); Don't return HA_ERR_END_OF_FILE in case of IPC_ERROR Sergey> === modified file 'storage/maria/maria_def.h' Sergey> --- a/storage/maria/maria_def.h 2011-01-14 11:03:41 +0000 Sergey> +++ b/storage/maria/maria_def.h 2011-02-01 23:33:14 +0000 Sergey> @@ -505,6 +505,7 @@ Sergey> DYNAMIC_ARRAY pinned_pages; Sergey> /* accumulate indexfile changes between write's */ Sergey> TREE *bulk_insert; Sergey> + void *__external_ref; /* For MariaDB TABLE */ /* psergey-todo: need this?*/ No, we don't need this as we have external_ptr already in Aria that we can use. Sergey> --- a/storage/myisam/mi_rkey.c 2009-12-22 12:33:21 +0000 Sergey> @@ -168,26 +176,37 @@ Sergey> if (share->concurrent_insert) Sergey> rw_unlock(&share->key_root_lock[inx]); Sergey> - /* Calculate length of the found key; Used by mi_rnext_same */ Sergey> - if ((keyinfo->flag & HA_VAR_LENGTH_KEY) && last_used_keyseg && Sergey> - info->lastpos != HA_OFFSET_ERROR) Sergey> - info->last_rkey_length= _mi_keylength_part(keyinfo, info->lastkey, Sergey> - last_used_keyseg); Sergey> + info->last_rkey_length= pack_key_length; Sergey> + Sergey> + if (info->lastpos == HA_OFFSET_ERROR) /* No such record */ Sergey> + { Sergey> + /* Sergey> + psergey: what doe the next line do:vv initially there were no writeinfo Sergey> + calls! Sergey> + */ Sergey> + fast_mi_writeinfo(info); The original code had a bug. At start of mi_rkey.c we do fast_mi_readinfo(), which must be followed by a fast_mi_writeinfo() before return, which didn't happen if buf was 0. In the normal case 'read_record()' will do the fast_mi_writeinfo() call for the other cases. Sergey> + if (!buf) Sergey> + DBUG_RETURN(my_errno); Sergey> + } Sergey> else Sergey> + /* Calculate length of the found key; Used by mi_rnext_same */ Sergey> + if ((keyinfo->flag & HA_VAR_LENGTH_KEY) && last_used_keyseg) Sergey> + info->last_rkey_length= _mi_keylength_part(keyinfo, info->lastkey, Sergey> + last_used_keyseg); Sergey> + Sergey> + /* Check if we don't want to have record back, only error message */ Sergey> + if (!buf) Sergey> + DBUG_RETURN(0); We have to add fast_mi_writeinfo(info) before the DBUG_RETURN above. Sergey> + if (!(*info->read_record)(info,info->lastpos,buf)) Sergey> + { Sergey> + info->update|= HA_STATE_AKTIV; /* Record is read */ Sergey> + DBUG_RETURN(0); Sergey> + } <cut> Sergey> +my_bool mi_yield_and_check_if_killed(MI_INFO *info, int inx) Sergey> +{ Sergey> + MYISAM_SHARE *share; Sergey> + if (mi_killed(info)) Sergey> + { Sergey> + info->lastpos= HA_OFFSET_ERROR; Sergey> + my_errno= HA_ERR_KEY_NOT_FOUND; HA_ERR_KEY_NOT_FOUND -> HA_ERR_ABORTED_BY_USER Sergey> + return 1; Sergey> + } Sergey> + Sergey> + if ((share= info->s)->concurrent_insert) Sergey> + { Sergey> + /* Give writers a chance to access index */ Sergey> + rw_unlock(&share->key_root_lock[inx]); Sergey> + rw_rdlock(&share->key_root_lock[inx]); Sergey> + } Sergey> + return 0; Sergey> +} Sergey> === modified file 'storage/myisam/mi_rnext.c' Sergey> --- a/storage/myisam/mi_rnext.c 2010-06-26 10:05:41 +0000 Sergey> +++ b/storage/myisam/mi_rnext.c 2011-02-01 23:33:14 +0000 Sergey> @@ -104,6 +104,18 @@ Sergey> (info->index_cond_func && Sergey> (res= mi_check_index_cond(info, inx, buf)) == ICP_NO_MATCH)) Sergey> { Sergey> + /* Sergey> + If we are at the last key on the key page, allow writers to Sergey> + access the index. Sergey> + */ Sergey> + if (info->int_keypos >= info->int_maxpos && Sergey> + mi_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + error= 0; Sergey> + res= ICP_ERROR; Return a real error! Sergey> + break; Sergey> + } Sergey> + Sergey> /* Sergey> Skip rows that are either inserted by other threads since Sergey> we got a lock or do not match pushed index conditions Sergey> @@ -115,7 +127,8 @@ Sergey> info->s->state.key_root[inx]))) Sergey> break; Sergey> } Sergey> - if (!error && res == ICP_OUT_OF_RANGE) Sergey> + Sergey> + if (!error && (res == ICP_OUT_OF_RANGE || res == ICP_ERROR)) Sergey> { Sergey> if (info->s->concurrent_insert) Sergey> rw_unlock(&info->s->key_root_lock[inx]); Return error code in case of ICP_ERROR Sergey> === modified file 'storage/myisam/mi_rnext_same.c' Sergey> --- a/storage/myisam/mi_rnext_same.c 2009-12-22 12:33:21 +0000 Sergey> +++ b/storage/myisam/mi_rnext_same.c 2011-02-01 23:33:14 +0000 Sergey> @@ -63,6 +63,18 @@ Sergey> } Sergey> for (;;) Sergey> { Sergey> + /* Sergey> + If we are at the last key on the key page, allow writers to Sergey> + access the index. Sergey> + */ Sergey> + if (info->int_keypos >= info->int_maxpos && Sergey> + mi_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + error=1; Sergey> + buf= 0; /* Fast abort */ Sergey> + break; Return real error Sergey> + } Sergey> + Sergey> if ((error=_mi_search_next(info,keyinfo,info->lastkey, Sergey> info->lastkey_length,SEARCH_BIGGER, Sergey> info->s->state.key_root[inx]))) Sergey> @@ -87,6 +99,10 @@ Sergey> } Sergey> if (info->s->concurrent_insert) Sergey> rw_unlock(&info->s->key_root_lock[inx]); Sergey> + Sergey> + if (!buf) Sergey> + DBUG_RETURN(my_errno); Sergey> + Remove above; We have to update info->update even if buf is 0 Sergey> /* Don't clear if database-changed */ Sergey> info->update&= (HA_STATE_CHANGED | HA_STATE_ROW_CHANGED); Sergey> info->update|= HA_STATE_NEXT_FOUND | HA_STATE_RNEXT_SAME; Sergey> === modified file 'storage/myisam/mi_rprev.c' Sergey> --- a/storage/myisam/mi_rprev.c 2009-12-15 07:16:46 +0000 Sergey> +++ b/storage/myisam/mi_rprev.c 2011-02-01 23:33:14 +0000 Sergey> @@ -53,12 +53,28 @@ Sergey> if (!error) Sergey> { Sergey> - int res= 0; Sergey> + ICP_RESULT res= 0; Sergey> + my_off_t cur_keypage= info->last_keypage; Sergey> while ((share->concurrent_insert && Sergey> info->lastpos >= info->state->data_file_length) || Sergey> (info->index_cond_func && Sergey> !(res= mi_check_index_cond(info, inx, buf)))) Sergey> { Sergey> + /* Sergey> + If we are at the last (i.e. first?) key on the key page, Sergey> + allow writers to access the index. Sergey> + */ Sergey> + if (info->last_keypage != cur_keypage) Sergey> + { Sergey> + cur_keypage= info->last_keypage; Sergey> + if (mi_yield_and_check_if_killed(info, inx)) Sergey> + { Sergey> + error= 0; Sergey> + res= ICP_ERROR; Sergey> + break; Return error Sergey> + } Sergey> + } Sergey> + Sergey> /* Sergey> Skip rows that are either inserted by other threads since Sergey> we got a lock or do not match pushed index conditions Sergey> @@ -69,7 +85,7 @@ Sergey> share->state.key_root[inx]))) Sergey> break; Sergey> } Sergey> - if (!error && res == 2) Sergey> + if (!error && (res == ICP_OUT_OF_RANGE || res == ICP_ERROR)) Sergey> { Sergey> if (share->concurrent_insert) Sergey> rw_unlock(&share->key_root_lock[inx]); Return error code in case of ICP_ERROR <cut> Sergey> --- a/storage/xtradb/handler/ha_innodb.cc 2011-01-12 12:00:10 +0000 Sergey> +++ b/storage/xtradb/handler/ha_innodb.cc 2011-02-01 23:33:14 +0000 <cut> Sergey> @@ -12094,15 +12102,18 @@ Sergey> See note on ICP_RESULT for return values description. Sergey> */ Sergey> -static int index_cond_func_innodb(void *arg) Sergey> +static xtradb_icp_result_t index_cond_func_innodb(void *arg) Sergey> { Sergey> ha_innobase *h= (ha_innobase*)arg; Sergey> + if (h->is_thd_killed()) Sergey> + return XTRADB_ICP_ERROR; Sergey> + rename to XTRADB_ABORTED_BY_USER; Sergey> if (h->end_range) Sergey> { Sergey> if (h->compare_key2(h->end_range) > 0) Sergey> - return ICP_OUT_OF_RANGE; /* caller should return HA_ERR_END_OF_FILE already */ Sergey> + return XTRADB_ICP_OUT_OF_RANGE; /* caller should return HA_ERR_END_OF_FILE already */ Sergey> } Sergey> - return h->pushed_idx_cond->val_int()? ICP_MATCH : ICP_NO_MATCH; Sergey> + return h->pushed_idx_cond->val_int()? XTRADB_ICP_MATCH : XTRADB_ICP_NO_MATCH; Sergey> } Sergey> C_MODE_END <cut> Sergey> --- a/storage/xtradb/include/row0mysql.h 2010-12-06 08:25:44 +0000 Sergey> +++ b/storage/xtradb/include/row0mysql.h 2011-02-01 23:33:14 +0000 Sergey> @@ -577,7 +577,15 @@ Sergey> #define ROW_PREBUILT_ALLOCATED 78540783 Sergey> #define ROW_PREBUILT_FREED 26423527 Sergey> -typedef int (*index_cond_func_t)(void *param); Sergey> + Sergey> +typedef enum xtradb_icp_result { Sergey> + XTRADB_ICP_ERROR=-1, rename to XTRADB_ABORTED_BY_USER Sergey> + XTRADB_ICP_NO_MATCH=0, Sergey> + XTRADB_ICP_MATCH=1, Sergey> + XTRADB_ICP_OUT_OF_RANGE=2 Sergey> +} xtradb_icp_result_t; Sergey> + Sergey> +typedef xtradb_icp_result_t (*index_cond_func_t)(void *param); Sergey> /** A struct for (sometimes lazily) prebuilt structures in an Innobase table Sergey> handle used within MySQL; these are used to save CPU time. */ Sergey> === modified file 'storage/xtradb/row/row0sel.c' Sergey> --- a/storage/xtradb/row/row0sel.c 2010-12-06 17:44:17 +0000 Sergey> +++ b/storage/xtradb/row/row0sel.c 2011-02-01 23:33:14 +0000 Sergey> @@ -4396,12 +4396,13 @@ Sergey> */ Sergey> ut_ad(ib_res); Sergey> res= prebuilt->idx_cond_func(prebuilt->idx_cond_func_arg); Sergey> - if (res == 0) Sergey> + if (res == XTRADB_ICP_NO_MATCH) Sergey> goto next_rec; Sergey> - if (res == 2) { Sergey> + if (res == XTRADB_ICP_OUT_OF_RANGE || res == XTRADB_ICP_ERROR) { Sergey> err = DB_RECORD_NOT_FOUND; Sergey> goto idx_cond_failed; Don't return DB_RECORD_NOT_FOUND in case of XTRADB_ICP_ERROR. Return an actual error like DB_ABORTED_BY_USER which can be translated to HA_ERR_ABORTED_BY_USER. Regards, Monty

1 0

Re: [Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 03 Feb '11

03 Feb '11

> -----Original Message----- > From: kahzoo(a)gmail.com [mailto:kahzoo@gmail.com] On Behalf Of Kazuhisa Ichikawa > Sent: Donnerstag, 3. Februar 2011 02:00 > To: Vladislav Vaintroub > Cc: maria-developers(a)lists.launchpad.net > Subject: Re: [Maria-developers] Windows installer MWL#55 finished. > Hello Kazuhisa, > > But to reiterate, this package is not really thought to be run by non-admin users. It installs into program files, and it writes to > > the HKLM registry hive, and it creates a service. All this stuff is not possible without being an administrator. I'm quite surprised > > Well, if that is how it is designed, that's fine with me. I just > thought that there might be many users who are new to MariaDB, not > sure about its quality and want to install/run it in a sandbox > environment. If switching to HKCU registry hive and disabling the > option to run it as a service when the installing user is non-admin > can satisfy the needs, implementing them might not be a bad idea. But > again, I do not have a strong opinion on that. (The user can choose to > install from the zipped package as long as zip version is also > distributed). I this the per-user install is a feature in NSIS which you discovered, but which is was not planned from the start on. From the start on, it was planned as an installation where you get a running service at the end, and that was not implemented in NSIS. if it was, it would not work in this sandboxed environment. With MSI I checked, and the rules for per-user install appear to be relatively complicated. Officially dual package authoring (single package that can be installed either per-user or per-machine) is supported with MSI 5.0 and Win7, so there is no hope to get that working in the same for unprivileged XP user. I might do some experimentation with it on Win7, however the benefit it provide compared to ZIP is low: is a menu shortcut with PATH including bin directory of the installation and Add/Remove Program entry. On a bright side, this is a valid bug that was simple to fix forcing per-machine install, so installer does not need to guess. I did that and refreshed the MSI on the download page. > One more thing that I noticed is, when installing 5.1 all the messages > in the dialogs are in Japanese (with the exception of GPL license > statement), while they are all in English for 5.2. Maybe this is a > bonus that came from NSIS. Maybe. NSIS guys deserve a compliment for that effort. With MSI dealing with multiple languages inside single MSI is not that simple. But, localization is definitely possible in Wix or generally MSI, I so far ignored that topic altogether;) But back to NSIS package .I believe, just like with per-user installation, you discovered hidden and untested feature, an Easter egg. It has more localization bugs that just GPL text. The feature tree is not localized, neither uninstall popup box. If that NSIS package had more functionality than plain unpack, and had own dialogs, there would be more localization bugs.

1 0

Re: [Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 02 Feb '11

02 Feb '11

> -----Original Message----- > From: kahzoo(a)gmail.com [mailto:kahzoo@gmail.com] On Behalf Of Kazuhisa Ichikawa > Sent: Mittwoch, 2. Februar 2011 23:01 > To: Vladislav Vaintroub > Cc: maria-developers(a)lists.launchpad.net > Subject: Re: [Maria-developers] Windows installer MWL#55 finished. > > Vladislav, > > I tried the new 5.2.5 installer in Japanese Windows XP (32bit) > environment using a user account which does NOT have administrator > privilege. I wonder why it has started at all. It definitely needs administrator and installer on later Windows brings all the necessary UAC popups. > During the installation I encountered a lot of warning > dialogs (see below for an example) about not being able to write into > the registry, although I did not have any noticeable problem startnig > mysqld.exe after the installation is finished (No, the installation > was not aborted and it said "Completed the MariaDB 5.2 Setup WIzard.") > > (Example warning message) > "Could not write value InstallLocation to key > \Software\Microsoft\Windows\CurrenVersion\Uninstall\MariaDB 5.2. > Verify that you have sufficient access to that key, or contact your > support personnel." If you do not have that key, you do not have an entry in "Add/remove programs" > I think what this error message means is that the user account used > for the installation needs to have an administrator privilege, but the > current (older) installer for MariaDB 5.1.53 did not have this > problem. This installer should be run as administrator, there is no per-user option. > The other (related) issue is that I cannot uninstall mariadb 5.2 now > because it is not listed in the installed program list in the control > panel (most likely because the necessary information is not written > into the registry). You will need to run msiexec /x mariadb-5.2.5-win32.msi from the command line. This uninstalls installed package (I'm not really sure it was installed, but it does not hurt to try, it nothing is installed it will tell) > Again, the installer for 5.1 did not have this problem. Well, it is not installer for 5.1, the old one had more or less the same functionality as downloading and unpacking a zip file. It might even be that NSIS guys automatically switched to HKCU registry keys or so for a weak user. > Also, unlike the current installer for 5.1, the new installer > does not seem to install a standalone uninstaller (Uninstall.exe) on > the disk. So, I'm stuck and wondering how to uninstall 5.2 (Well, not > really, because if nothing is written into the registry, deleting all > the files and directories should 'cleanly' uninstall everything, > right?.). I would stick to documented stuff. msiexec /x mariadb-5.2.5-win32.msi But to reiterate, this package is not really thought to be run by non-admin users. It installs into program files, and it writes to the HKLM registry hive, and it creates a service. All this stuff is not possible without being an administrator. I'm quite surprised that MSI let you go that far actually, and that it would not rollback and cleanup after the error. On the other hand, thanks for an interesting error report, makes sense for me to have a test for anomalies like that. Can you tell service pack of your XP and MSI version (launch winver and msiexec /?) Thanks, Vladislav > - Kazuhisa >

1 0

[Maria-developers] Benchmarking mrr_sort_keys=on|off.
by Sergey Petrunya 02 Feb '11

02 Feb '11

Hi! Following the discussion of benchmarks of mrr_sort_keys=on|off functionality, I've done a re-run. Please find the results below. ## Dataset: DBT-3, scale=10, xtradb, ## innodb_file_per_table=1 ## innodb_file_format=barracuda ## innodb_log_file_size=256M ## innodb_buffer_pool_size=4000M ## ## Platform: work.askmonty.org host ## Cold-cache is achieved by: ## - stopping the server ## - echo 1 > /proc/sys/vm/drop_caches ## - starting the server ## - running EXPLAIN {query} ## ## MariaDB: 5.3-current, release build (compile-pentium-max) ## Result summary: * mrr_sort_keys=on provides speedup (2.25 min vs 13.5 min) on cold cache. * on hot cache, there is no big difference (~1.3 sec in both cases). * I was able to repeat Igor's August results with the current 5.3. It is not clear why Igor can't repeat them anymore. (An evasive bug? or just mistake in experiments?) set join_buffer_size=1024*1024*32; set join_buffer_space_limit=1024*1024*32; set join_cache_level=6; ## ## Cold cache, mrr_sort_keys=on: ## [22:12] root@dbt3sf10>explain select avg(l_extendedprice) from orders, lineitem where l_orderkey=o_orderkey and o_orderdate between date '1992-07-01' and date '1992-07-31'; +----+-------------+----------+-------+--------------------------------------------+---------------+---------+----------------------------+--------+------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+-------+--------------------------------------------+---------------+---------+----------------------------+--------+------------------------------------+ | 1 | SIMPLE | orders | range | PRIMARY,i_o_orderdate | i_o_orderdate | 4 | NULL | 142680 | Using where; Using index | | 1 | SIMPLE | lineitem | ref | PRIMARY,i_l_orderkey,i_l_orderkey_quantity | PRIMARY | 4 | dbt3sf10.orders.o_orderkey | 1 | Using join buffer (flat, BKA join) | +----+-------------+----------+-------+--------------------------------------------+---------------+---------+----------------------------+--------+------------------------------------+ 2 rows in set (3.04 sec) [22:06] root@dbt3sf10>select avg(l_extendedprice) from orders, lineitem where l_orderkey=o_orderkey and o_orderdate between date '1992-07-01' and date '1992-07-31'; +----------------------+ | avg(l_extendedprice) | +----------------------+ | 38229.0683265894 | +----------------------+ 1 row in set (2 min 13.40 sec) [22:09] root@dbt3sf10> show status like "%handler%"; +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_commit | 2 | | Handler_read_key | 193396 | | Handler_read_next | 966535 | +----------------------------+--------+ 15 rows in set (0.29 sec) # # mrr_sort_keys=off, cold cache: # [22:20] root@dbt3sf10>explain select avg(l_extendedprice) from orders, lineitem where l_orderkey=o_orderkey and o_orderdate between date '1992-07-01' and date '1992-07-31'; +----+-------------+----------+-------+--------------------------------------------+---------------+---------+----------------------------+--------+--------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+-------+--------------------------------------------+---------------+---------+----------------------------+--------+--------------------------+ | 1 | SIMPLE | orders | range | PRIMARY,i_o_orderdate | i_o_orderdate | 4 | NULL | 142680 | Using where; Using index | | 1 | SIMPLE | lineitem | ref | PRIMARY,i_l_orderkey,i_l_orderkey_quantity | PRIMARY | 4 | dbt3sf10.orders.o_orderkey | 2 | | +----+-------------+----------+-------+--------------------------------------------+---------------+---------+----------------------------+--------+--------------------------+ [22:20] root@dbt3sf10>select avg(l_extendedprice) from orders, lineitem where l_orderkey=o_orderkey and o_orderdate between date '1992-07-01' and date '1992-07-31'; +----------------------+ | avg(l_extendedprice) | +----------------------+ | 38229.0683265898 | +----------------------+ 1 row in set (13 min 23.68 sec) [22:09] root@dbt3sf10> show status like "%handler%"; +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_commit | 2 | | Handler_read_key | 193396 | | Handler_read_next | 966535 | +----------------------------+--------+ ## ## Hot caches (re-run the query until time stabilizes): ## mrr_sort_keys=off: 1.34 sec mrr_sort_keys=on: 1.38 sec BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 02 Feb '11

02 Feb '11

From: Peter Laursen [mailto:peter_laursen@webyog.com] Sent: Mittwoch, 2. Februar 2011 17:14 To: Vladislav Vaintroub Cc: maria-developers(a)lists.launchpad.net Subject: Re: [Maria-developers] Windows installer MWL#55 finished. Hi Peter, >Your default install path is \program files [(x86)]. You copy the /datadir as a subfolder of /basedirr. >This will cause problems on Vista+ with UAC enabled if the server is ever started from commandline as a non-service (also if >the user doing so is an administrator user). For the user who installed MySQL, it won't cause problems. After you stop the service you can start mysqld --defaults-file=path\to\datadir\my.ini --console from the non-elevated command line, with/without UAC. No file system virtualization will happen. That bug http://bugs.mysql.com/bug.php?id=38373 (user can not from mysqld from command line after installation) was filed by myself 2.5 years ago , I could not leave that unfixed now. I know, writing to ProgramFiles might not be kosher, but it works if ACLs on our tree are set. (And Microsoft should not tell people that writing to program files is bad, they are doing it with SQLServer, and also document it http://msdn.microsoft.com/en-us/library/ms143547.aspx , so we are again as good/bad as SQLServer here) >2) There is no my.ini generated with a fresh install as far as I can see - so the server probably runs (it does!) with very low default buffer settings etc. Also latin1 charset and MyISAM will be default. I think a my.ini with settings fit for a 1GB-2GB RAM machine should be generated - and with a few configuration options. And a my.ini will be required to solve 1) anyway. There is a my.ini, it is in database directory. It is spartanic, it has port and datadir, nothing else. You can edit it using non-elevated notepad (with or without UAC) and there is a shortcut for it created in start menu, MariaDB 5.2 folder (rsp MariaDB 5.2 (x64) if for 64 bit package). There are zero performance related options, which means in fact it won't run on full speed. I have no strong opinion on it, nor I feel qualified for configuring "optimal performance out of the box" . If there is a strong demand on specific parameters, they should probably go there. I would like to hear more opinions on this, one thing that will definitely not be in the first incarnation of this installer is fully-featured ConfigWizard like stuff. Where I see differences between this my.ini and MySQL installer's default is * sql_mode (MySQL's default is "strict transactional tables") * default storage engine (MySQL default is innodb) * innodb buffer pool size (IIRC, MySQL's default is something like 12% of RAM) Those are trivial to change modifying my.ini Stuff like innodb log file size (calculated from innodb buffer pool size) not trivial to change with simple my.ini modification, there must be a clean shutdown and exiting files need to be removed. > But still it is a good installer (and there is a 64 bit one as well), so this is an important step in the right direction. So thanks for your effort! Thanks! -- Peter On Wed, Feb 2, 2011 at 14:37, Peter Laursen <peter_laursen(a)webyog.com> wrote: I am a little bit busy now, but will confirm tonight. -- Peter On Wed, Feb 2, 2011 at 14:12, Vladislav Vaintroub <wlad(a)montyprogram.com> wrote: From: Peter Laursen [mailto:peter_laursen@webyog.com] Sent: Dienstag, 1. Februar 2011 20:31 To: Vladislav Vaintroub Cc: maria-developers(a)lists.launchpad.net Subject: Re: [Maria-developers] Windows instller MWL#55 finished. >It is a Danish Windows (Home Premium 64 bit). Hi Peter, As it turns out, installer had the same "localization" problem registering service on German Windows. Fixed now, I uploaded the msis to the same place http://cid-ff0c950417b4f8a4.office.live.com/browse.aspx/share Thanks, Wlad

2 1

[Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 02 Feb '11

02 Feb '11

Hi all, I updated the MWL#55 (Windows installer) with the upgrade information. Now, people will be able to upgrade between prior MariaDB (or prior MySQL) versions using upgrade wizard, which is by default launched at the end of installation, if upgradable instances are found . Users will also be able to use Upgrade Wizard after the installation, at any the time convenient for them. Upgrade wizard will also be included into ZIP distribution. Besides GUI Wizard, there is a command line utility mysql_upgrade_service.exe to perform all operations necessary to upgrade an existing database instance registered as service (in fact GUI wizard uses this mysql_upgrade_service.exe as "backend"). Wizard is an MFC application. I'm sorry for that, but my Windows C/C++ GUI programming experience is not only rusty (prior to this WL last time I used it was more than a decade ago), it is also restricted to MFC only. Being MFC application, it will require at least VS Professional to build. There is no MFC in free Visual Studio Express edition, nor in free Windows SDK. Build and packaging process will handle missing MFC gracefully (build won't try to compile upgrade wizard, package won't include it , and installer won't start it at the end of installation). The repository that includes the final windows installer is lp:~maria-captains/maria/5.2-windows-packaging-upgrade Building installer is described in http://askmonty.org/worklog/Server-Sprint/?tid=55 (Section "Building installer", at the very start of the WL). Monty or Sergei (or volunteers if any): Can you please have a look the above repo and do a review . I tried to split the big patch into parts, but I did not find the result to be too satisfactory Philip: would you be able to test the upgrade wizard? You need a service running under some older mysqld.exe binary (e.g. install MySQL 5.1), you'll should get upgrade wizard launched at the end of install. You should also get Wizard at the end of installation any time you have an older version of Maria running as service. Thanks and best regards, Wlad

6 27

[Maria-developers] IMPORTANT: LP bug #611648 does not allow running benchmarks for DS-MRR normally
by Igor Babaev 02 Feb '11

02 Feb '11

Hi, I've just reported bug #611648 that probably explains the strange results I had when trying to run my benchmark queries for DS-MRR about which I talked at today's optimizer meeting. Philip, please assign this bug to SergeyP. Regards, Igor.

2 1

[Maria-developers] Buildbot Outage
by Adam M. Dutko 31 Jan '11

31 Jan '11

Sorry about adutko-centos5-amd64 being down. I've not finished changes to my monitoring script to tell me when a builder is offline. I'll try to get that done soon and send it to the list when I've updated it to work with the new buildbot status page format.

1 0

Re: [Maria-developers] replication improvements going on here and there
by Kristian Nielsen 28 Jan '11

28 Jan '11

"Rasmus Johansson" <rasmus(a)montyprogram.com> writes: > I would need your help to understand the different replication topics that > are very active now: > > - The replication work that you have been doing As far as implementation goes, it is these four worklogs, which are in code review and available in the feature preview tree lp:~maria-captains/maria/mariadb-5.2-rpl: http://askmonty.org/worklog/Server-Sprint/?tid=116 http://askmonty.org/worklog/Server-RawIdeaBin/?tid=136 http://askmonty.org/worklog/Server-Sprint/?tid=163 http://askmonty.org/worklog/Server-Sprint/?tid=132 This work improves replication in a number of ways: - Implement working group commit for the replication binlog. This greatly improves performance when running with sync_binlog=1, which is necessary to ensure the ability to reliably recover replication state after a master crash. - Ensure same commit order between innodb and binlog, without the need for the expensive prepare_commit_mutex. This is needed by XtraBackup and InnoDB hot backup to do a non-blocking online backup that can be used to provision a slave. The other existing patches I have seen solve this by having the user need to turn on/off the prepare_commit_mutex depending on whether backup is going on or not. - Allow to obtain an InnoDB consistent read snapshot and the corresponding binlog position in a fully non-blocking way. This allows for example to provision a slave from a fully non-blocking `mysqldump --master-data --single-transaction`. Previously, this required FLUSH TABLES WITH READ LOCK, which blocks queries in the server for some time. - Port the Facebook patch to release InnoDB row locks early, already during the prepare phase. This can improve performance in the presense of hot-spot rows in busy OLTP applications. (This feature must be enabled explicitly, it is off by default). - Make the transaction coordinator inside MariaDB plugable, allowing to plug in different implementations. This is mostly preparatory work for pluggable replication, as the replication binlog functions as a transaction coordinator. So with this, a new binlog implementation can be plugged in and do the same transaction coordinator role that the legacy binlog normally handles. In addition to this, there was a long discussion on the MariaDB mailing list about various replication topics, and I wrote up the following specifications (but did not start on the implementation yet): http://askmonty.org/worklog/?tid=107 http://askmonty.org/worklog/?tid=120 http://askmonty.org/worklog/?tid=133 The idea with these is to gradually make the existing legacy MySQL replication and binlog pluggable, opening the way for writing completely new replication implementations. It builds on top of the implementation work above. I also worked on the specifications of the parallel replication worklogs MWL#169 and MWL#170 mentioned below. > - The parallel replication that Oracle is talking about for MySQL 5.6 Do you mean this? http://forge.mysql.com/wiki/ReplicationFeatures/ParallelSlave http://forge.mysql.com/worklog/task.php?id=4648 The idea is to take one transaction at a time, and run the operations inside it in parallel. That is a bit unusual, most other projects work by running multiple transaction in parallel. I think this is motivated by NDB (though it is just me reading between the lines, I do not know for sure). Since in NDB, multiple parallel transactions on the master are replicated as a single, bigger transaction called an epoch. So they need something on the slave that can take apart the epoch again and make it parallel. Frankly, when I first saw this, I dismissed it, thinking it was the wrong approach to the problem (outside of NDB replication at least). I am still quite sceptical. Though it may be possible to use this for normal OLTP workloads (with small transactions) by artificially combining multiple transactions, at the cost of higher delay between master and slave. > - The replication functionality that <customer> was interested in together > with <partner> and us This is a request for implementing another algorithm for parallel replication. I wrote up some designs for this: http://askmonty.org/worklog/Server-RawIdeaBin/?tid=169 http://askmonty.org/worklog/Server-RawIdeaBin/?tid=170 This overlaps with the above-mentioned parallel replication project in Oracle, however the chosen approach/algorithm is quite different between the two. In this work, it would be the responsibility of the application to ensure that parallel transactions modifying tables in two different databases are independent. Then such transactions can be replicated in parallel on the slave. > Could you just give a short description on each of those and then describe > how they connect to each other or are there something that duplicates? Hope this helps, - Kristian.

1 0

Re: [Maria-developers] [Commits] Rev 2896: Fixed LP bug #707827. in file:///home/igor/maria/maria-5.3-bug707827/
by Oleksandr Byelkin 28 Jan '11

28 Jan '11

Hi! On 28.01.2011 07:23, Igor Babaev wrote: > At file:///home/igor/maria/maria-5.3-bug707827/ > > ------------------------------------------------------------ > revno: 2896 > revision-id: igor(a)askmonty.org-20110128052302-09617460yxj1kck7 > parent: igor(a)askmonty.org-20110127044523-9kvx1t3x3166hjyr > committer: Igor Babaev<igor(a)askmonty.org> > branch nick: maria-5.3-bug707827 > timestamp: Thu 2011-01-27 21:23:02 -0800 > message: > Fixed LP bug #707827. > This bug could manifest itself when hash join over a varchar column > with NULL values in some rows was used. It happened because the > function key_buf_cmp erroneously returned FALSE when one of the joined > key fields was null while the second was not. > Also fixed two other bugs in the functions key_hashnr and key_buf_cmp > that could possibly lead to wrong results for some queries that > used hash join over several columns with nulls. > Also reverted the latest addition of the test case for bug #45092. It > had been already backported earlier. It looks good to be pushed. Thank you for fixing it!

1 0

[Maria-developers] Benchmarking index_merge sort_intersect
by Sergey Petrunya 26 Jan '11

26 Jan '11

Hi! Please find below results of my attempts to figure out if/when one should "index_merge_sort_intersection=on" setting. General setup information: * Experiment was done on my laptop * I picked "on time flight statistics" dataset, because - it has lots of columns, so one can't be expected to have all possible composite indexes - it allows for meaningful intersection queries - the data is real and I don't expect it to be extremely correlated (although some aiports/airlines are probably worse than others, so it's not totally uncrellated, either). * Storage engine is XtraDB * I loaded data for Q1 2009, which is 1.5M rows, and ontime.ibd file is 704M: MariaDB [ontime]> select count(*), min(FlightDate), max(FlightDate) from ontime; +----------+-----------------+-----------------+ | count(*) | min(FlightDate) | max(FlightDate) | +----------+-----------------+-----------------+ | 1578171 | 2009-01-01 | 2009-03-31 | +----------+-----------------+-----------------+ 1 row in set (15.92 sec) MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=on'; MariaDB [ontime]> explain select count(*) from ontime where depdelay > 30 and OriginState IN ('WA', 'CA'); +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay,OriginState | 5,3 | NULL | 41832 | Using sort_intersect(DepDelay,OriginState); Using where | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ 1 row in set (0.01 sec) ## (Run query several times until query time stabilizes) MariaDB [ontime]> select count(*) from ontime where depdelay > 30 and OriginState IN ('WA', 'CA'); +----------+ | count(*) | +----------+ | 18824 | +----------+ 1 row in set (2.37 sec) MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=off'; MariaDB [ontime]> explain select count(*) from ontime where depdelay > 30 and OriginState IN ('WA', 'CA'); +----+-------------+--------+-------+----------------------+----------+---------+------+--------+-----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------+---------+------+--------+-----------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay | 5 | NULL | 201894 | Using index condition; Using where; Using MRR | +----+-------------+--------+-------+----------------------+----------+---------+------+--------+-----------------------------------------------+ 1 row in set (0.01 sec) ## (Run query several times until query time stabilizes) MariaDB [ontime]> select count(*) from ontime where depdelay > 30 and OriginState IN ('WA', 'CA'); +----------+ | count(*) | +----------+ | 18824 | +----------+ 1 row in set (18.27 sec) ## For comparison, without indexes whatsoever: MariaDB [ontime]> select count(*) from ontime ignore index (depdelay, originstate) where depdelay > 30 and OriginState IN ('WA', 'CA'); +----------+ | count(*) | +----------+ | 18824 | +----------+ 1 row in set (15.50 sec) "iostat -x" showed disk utilization as 0% during all of the above listed queries, so all accessed data was either in cache or in buffer pool. For an idea about predicate selectivities and correlation: ROWS FRACTION whole table 1578171 100.0% depdelay>30 151806 9.6% OriginState IN ('WA', 'CA') 206576 13.1% both 18824 1.1% Conclusions: - sort-intersect gives about 7x improvement - Igor's statement that sort-intersect can't be useful when the load is not IO-bound is not supported by the experiment. My next step was to try with narrower ranges. I changed the predicates to - OriginState IN ('IL', 'VW') (100K rows instead of 200K we had with WA+CA) - depdelay > 60 (75 K rows instead of 150K we had with depdelay>30) MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=on'; MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState IN ('IL', 'VT'); +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay,OriginState | 5,3 | NULL | 10105 | Using sort_intersect(DepDelay,OriginState); Using where | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ 1 row in set (0.02 sec) # Run a few times to stabilize MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState IN ('IL', 'VT'); +----------+ | count(*) | +----------+ | 6283 | +----------+ 1 row in set (0.99 sec) MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=off'; Query OK, 0 rows affected (0.00 sec) MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState IN ('IL', 'VT'); +----+-------------+--------+-------+----------------------+----------+---------+------+-------+-----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------+---------+------+-------+-----------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay | 5 | NULL | 99636 | Using index condition; Using where; Using MRR | +----+-------------+--------+-------+----------------------+----------+---------+------+-------+-----------------------------------------------+ 1 row in set (0.02 sec) # Run a few times to stabilize MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState IN ('IL', 'VT'); +----------+ | count(*) | +----------+ | 6283 | +----------+ 1 row in set (8.63 sec) MariaDB [ontime]> select count(*) from ontime ignore index (OriginState,DepDelay) where depdelay > 60 and OriginState IN ('IL', 'VT'); +----------+ | count(*) | +----------+ | 6283 | +----------+ 1 row in set (14.00 sec) The picture is similar: sort-intersect is still about 8 times better than single-index range access plan. I'm not sure about the conclusions. I suspect that there is some threshold (#records to be read through the most selective index) after which sort-intersect becomes worse than using one of the indexes. I wasn't able to hit it, though. Then I've tried changing the OriginState predicate to being simple equality, to see if we could compare againist ref access: MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=on'; MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState='IL'; +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay,OriginState | 5,3 | NULL | 10043 | Using sort_intersect(DepDelay,OriginState); Using where | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState='IL'; +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (1.00 sec) MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=off'; MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState='IL'; +----+-------------+--------+------+----------------------+-------------+---------+-------+--------+------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+------+----------------------+-------------+---------+-------+--------+------------------------------------+ | 1 | SIMPLE | ontime | ref | OriginState,DepDelay | OriginState | 3 | const | 201600 | Using index condition; Using where | +----+-------------+--------+------+----------------------+-------------+---------+-------+--------+------------------------------------+ 1 row in set (0.01 sec) MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState='IL'; +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (0.95 sec) And the benefit was gone. sort_intersect happens to be just as fast as ref access. I've changed OriginState='IL' to OriginState IN ('IL','ZZ'), where 'ZZ' is the non existant state code: MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState IN ('IL','ZZ'); +----+-------------+--------+-------+----------------------+----------+---------+------+-------+-----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------+---------+------+-------+-----------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay | 5 | NULL | 99636 | Using index condition; Using where; Using MRR | +----+-------------+--------+-------+----------------------+----------+---------+------+-------+-----------------------------------------------+ The used index has changed, MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState IN ('IL','ZZ'); +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (9.10 sec) And the query time went up greatly. Using FORCE INDEX to use the index that ref access was using improved the situation slightly: MariaDB [ontime]> explain select count(*) from ontime force index(originstate) where depdelay > 60 and OriginState IN ('IL','ZZ'); +----+-------------+--------+-------+---------------+-------------+---------+------+--------+-----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+---------------+-------------+---------+------+--------+-----------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState | OriginState | 3 | NULL | 201601 | Using index condition; Using where; Using MRR | +----+-------------+--------+-------+---------------+-------------+---------+------+--------+-----------------------------------------------+ MariaDB [ontime]> select count(*) from ontime force index(originstate) where depdelay > 60 and OriginState IN ('IL','ZZ'); +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (2.69 sec) Disabling MRR further improved the situation: MariaDB [ontime]> set optimizer_use_mrr='disable'; Query OK, 0 rows affected (0.00 sec) MariaDB [ontime]> explain select count(*) from ontime force index(originstate) where depdelay > 60 and OriginState IN ('IL','ZZ'); +----+-------------+--------+-------+---------------+-------------+---------+------+--------+------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+---------------+-------------+---------+------+--------+------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState | OriginState | 3 | NULL | 201601 | Using index condition; Using where | +----+-------------+--------+-------+---------------+-------------+---------+------+--------+------------------------------------+ 1 row in set (0.01 sec) MariaDB [ontime]> select count(*) from ontime force index(originstate) where depdelay > 60 and OriginState IN ('IL','ZZ'); +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (1.23 sec) But it seems we can't put all the blame on MRR. The supposedly best query plan without MRR is just as bad as it was with MRR: MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState IN ('IL','ZZ'); +----+-------------+--------+-------+----------------------+----------+---------+------+-------+------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------+---------+------+-------+------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay | 5 | NULL | 99636 | Using index condition; Using where | +----+-------------+--------+-------+----------------------+----------+---------+------+-------+------------------------------------+ 1 row in set (0.02 sec) MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState IN ('IL','ZZ'); +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (8.96 sec) In any case, sort_intersect doesn't care about whether one of the predicates is an equality or IN: MariaDB [ontime]> set optimizer_switch='index_merge_sort_intersection=on'; Query OK, 0 rows affected (0.00 sec) MariaDB [ontime]> explain select count(*) from ontime where depdelay > 60 and OriginState IN ('IL','ZZ'); +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ | 1 | SIMPLE | ontime | range | OriginState,DepDelay | DepDelay,OriginState | 5,3 | NULL | 10043 | Using sort_intersect(DepDelay,OriginState); Using where | +----+-------------+--------+-------+----------------------+----------------------+---------+------+-------+---------------------------------------------------------+ 1 row in set (0.02 sec) MariaDB [ontime]> select count(*) from ontime where depdelay > 60 and OriginState IN ('IL','ZZ'); +----------+ | count(*) | +----------+ | 6215 | +----------+ 1 row in set (0.97 sec) Conclusions: I'm not sure if the "failure" to demonstrate speedup against ref access is a genuine symptom (I think - not). Perhaps, enabling all of our improvements at once (BKA, DS-MRR, sort-intersect) will produce more coherent results, because all of these strategies use rowid-ordered disk-sweep reads, which allows to make fairer comparison between them. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 2

[Maria-developers] Buildbot will be down start of next week due to upgrade
by Kristian Nielsen 24 Jan '11

24 Jan '11

Next week, on Monday, I plan to take down Buildbot and start the upgrade process to version 0.8.x. This is a somewhat major upgrade, as Buildbot 0.8 switches to storing all its state and history in a MySQL (well MariaDB probably ;-) database rather than gazilions of Python-encoded files. Hopefully this will speed up our Buildbot, which has frankly gotten painfully slow. I plan to stop Buildbot, take a full backup, then attempt the upgrade and data conversion. Buildbot will be down this time, which I estimate could be 1-2 days, but it is hard to tell for sure without knowing what issues could arise. If the upgrade fails in some way, I will revert from the backup (but I hope of course to get it running on the new version). >From reading the docs, it looks like we will loose the history of past builds. I will see if there is some way to avoid that, but if not I suppose we can still live with it (note that the cross-reference search of old test failures is not affected by the upgrade, and will not loose any history). - Kristian.

2 3

Re: [Maria-developers] Optimization about count(*) of mysql 5.1.38
by Lichao Xie(谢立超) 24 Jan '11

24 Jan '11

> Hi, Lichao! > > On Jan 10, Lichao Xie wrote: >> hi all: >> I have a question about the optimization of count(*) in a range. >> Sql for mysql like:select count(*) from t where id > 10 and id < 1000000; >> In table t, there is an B-tree index on id field, this sql will read >> hundreds of thousands of records from the storage engine, that is a >> time-consuming operation. >> I am developing a storage engine, I want to know, is there a good >> method to caculate the records in a range, rather than read all >> records or keys in the range? > > Yes, see how MyISAM does it - it's pretty straightforward, and only > requires two index lookups. MyISAM seemingly read all keys in the range. And I test MyISAM table use mysqlslap, the test case is: ./mysqlslap --query="select count(xn_id) as num from roman1.r_paihang where dengji > 4 and dengji < 11" --number-of-queries=10000000 --concurrency=30 -uroot --host=10.134.11.17 --port=10001 the result is : 10.134.11.17:/data/mqq_data$ mysqladmin extended-status -i 1 -r -uroot -S /tmp/mysql-tdmdb-14.sock | grep Com_select | Com_select | 41115084 | | Com_select | 9 | | Com_select | 14 | | Com_select | 23 | | Com_select | 25 | | Com_select | 36 | | Com_select | 8 | | Com_select | 26 | | Com_select | 34 | | Com_select | 24 | | Com_select | 16 | | Com_select | 10 | | Com_select | 8 | the table scheme is: CREATE TABLE `r_paihang` ( `xn_id` int(11) NOT NULL DEFAULT '0', `company` int(8) DEFAULT NULL, `dengji` int(6) DEFAULT NULL, `position` tinyint(3) DEFAULT '1', `tili` int(11) DEFAULT '0', `zhili` int(11) DEFAULT '0', `meili` int(11) DEFAULT '0', `worth` int(11) DEFAULT '0', `sex` int(1) DEFAULT '0', `online_time` int(11) DEFAULT NULL, `sign_time` datetime DEFAULT NULL, `last_time` datetime DEFAULT NULL, `yq_id` int(12) NOT NULL DEFAULT '0', PRIMARY KEY (`xn_id`), KEY `company` (`company`), KEY `worth` (`worth`), KEY `zhili` (`zhili`), KEY `position` (`position`), KEY `meili` (`meili`), KEY `tili` (`tili`), KEY `sex` (`sex`), KEY `dengji` (`dengji`), KEY `sign_time` (`sign_time`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8; the case runs too slow.... If there is a storage engine api like records_in_range() to get the exact records in the range from the storage engine, the result will be 10^4 qps for this case... > > Regards, > Sergei > 2011/1/20 Sergei Golubchik <serg(a)askmonty.org>: > Hi, Lichao! > > Please send questions like this to maria-developers(a)lists.launchpad.net. > In your replies make sure send replies to the list, e.g. use "Reply to All". > It's a public mailing list dedicated to MySQL and MariaDB internals, source > code, and related things. I am subscribed, so I'll see you mail there, and you > may be sure I will, because it won't be accidentally catched by my spam filter, > or sorted out in some obscure folder. Furthermore other subscribers will see > your question and could reply if I will be not available (e.g. I could be > travelling). > Thank you. Sarry, my fault! I am resending the email. > > On Jan 20, Lichao Xie wrote: >> Hi, Sergei >> >> 2011/1/20 Sergei Golubchik <serg(a)askmonty.org>: >> > >> > On Jan 10, Lichao Xie wrote: >> >> hi all: >> >> I have a question about the optimization of count(*) in a range. >> >> Sql for mysql like:select count(*) from t where id > 10 and id < 1000000; >> >> In table t, there is an B-tree index on id field, this sql will read >> >> hundreds of thousands of records from the storage engine, that is a >> >> time-consuming operation. >> >> I am developing a storage engine, I want to know, is there a good >> >> method to caculate the records in a range, rather than read all >> >> records or keys in the range? >> > >> > Yes, see how MyISAM does it - it's pretty straightforward, and only >> > requires two index lookups. >> > Regards, > Sergei >

2 6

Re: [Maria-developers] [Commits] Rev 3014: New table creation/renaming block added if old encoded table present. in file:///home/bell/maria/bzr/work-maria-5.1-CREATE/
by Sergei Golubchik 20 Jan '11

20 Jan '11

Hi, sanja! On Jan 13, sanja(a)askmonty.org wrote: > revno: 3014 > revision-id: sanja(a)askmonty.org-20110113180607-e5hc8zuugxxjtn6k > parent: psergey(a)askmonty.org-20110112083018-90rw058f00uslvoz > committer: sanja(a)askmonty.org > branch nick: work-maria-5.1-CREATE > timestamp: Thu 2011-01-13 20:06:07 +0200 > message: > New table creation/renaming block added if old encoded table present. First: there is no test case for the new (as compared with the previous patch) change. Second: I think "old_path" does not belong to the check_table_file_presence() function. The function simply checks if the file exists, there is no place for the concept of "old path" here. You need to take care of the renaming for upgrade and old path outside of check_table_file_presence(). Regards, Sergei

2 1

[Maria-developers] index_merge, @@sort_buffer_size, and Sort_XXX counters
by Sergey Petrunya 20 Jan '11

20 Jan '11

Hello, index_merge uses @@sort_buffer_size as a guide of how much space it can use for sorting. I think there is an issue with the way it is done. Before index_merge was introduced, @@sort_buffer_size was used only by filesort (also for couple of other cases like GROUP_CONCAT with sorting, but I consider them to be too rare to be of practical importance). This allowed the following: 1. One can set @@sort_buffer_size to control how much space is to be allocated 2. SHOW STATUS will produce these numbers: Sort_merge_passes Sort_range Sort_scan Sort_range+Sort_scan gives the total number of filesort invocations. Sort_merge_passes is the number of merges. One can get average amount of merges per sort operation by calculating: Sort_merge_passes / (Sort_range + Sort_scan) (1) If the number is high, then it means that it could be useful to increase @@sort_buffer_size. Now, if we take index_merge into the picture, the counters are much less useful. index_merge causes Sort_merge_passes to be incremented, but we dont have a counter for index_merge operations, so formula (1) becomes invalid. Possible solutions are: Option#1: introduce a counter Sort_index_merge_count (name subject to discussion), which will mean "number of index_merge sort union/intersection scans". Option#2: make it so that index_merge does not cause Sort_merge_passes to be incremented. I think that in this case we'll need a separate variable to control index_merge's buffer size. it could have @@sort_buffer_size value as default. Any comments? BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Optimization about count(*) of mysql 5.1.38
by Lichao Xie(谢立超) 19 Jan '11

19 Jan '11

hi all: I have a question about the optimization of count(*) in a range. Sql for mysql like:select count(*) from t where id > 10 and id < 1000000; In table t, there is an B-tree index on id field, this sql will read hundreds of thousands of records from the storage engine, that is a time-consuming operation. I am developing a storage engine, I want to know, is there a good method to caculate the records in a range, rather than read all records or keys in the range? For example, does mysql call an API like records_in_range() to get the records in a range? After all, the range could be caculated by reading twice index.

2 1

[Maria-developers] [psergey@askmonty.org: No "Using index" support in sort-intersect - intentional?]
by Sergey Petrunya 19 Jan '11

19 Jan '11

Another question: why is type='range' for sort_intersect? We have type=index_merge for intersect and [sort]_union, so it seems rather confusing that we have type=range for sort_intersect? ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- Date: Wed, 19 Jan 2011 11:11:14 +0300 From: Sergey Petrunya <psergey(a)askmonty.org> To: igor(a)askmonty.org Cc: Maria Developers <maria-developers(a)lists.launchpad.net> Subject: No "Using index" support in sort-intersect - intentional? Hello Igor, I was looking at sort-intersect and noticed the following: Consider this table: CREATE TABLE `t1` ( `k1` int(11) DEFAULT NULL, `k2` int(11) DEFAULT NULL, `filler` char(100) DEFAULT NULL, KEY `k1` (`k1`), KEY `k2` (`k2`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1; Let's try a sort-intersect query: MariaDB [timerge1]> EXPLAIN select 1 from t1 where k1<30 and k2 < 30\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: range possible_keys: k1,k2 key: k1,k2 key_len: 5,5 ref: NULL rows: 1 Extra: Using sort_intersect(k1,k2); Using where 1 row in set (0.01 sec) The query doesn't use any columns not covered by the used indexes, however, we don't see "using index". Analogous ROR-intersect query will use "Using index": MariaDB [timerge1]> EXPLAIN select 1 from t1 where k1=30 and k2 = 30\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: index_merge possible_keys: k1,k2 key: k1,k2 key_len: 5,5 ref: NULL rows: 1 Extra: Using intersect(k1,k2); Using where; Using index 1 row in set (0.00 sec) BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] No "Using index" support in sort-intersect - intentional?
by Sergey Petrunya 19 Jan '11

19 Jan '11

Hello Igor, I was looking at sort-intersect and noticed the following: Consider this table: CREATE TABLE `t1` ( `k1` int(11) DEFAULT NULL, `k2` int(11) DEFAULT NULL, `filler` char(100) DEFAULT NULL, KEY `k1` (`k1`), KEY `k2` (`k2`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1; Let's try a sort-intersect query: MariaDB [timerge1]> EXPLAIN select 1 from t1 where k1<30 and k2 < 30\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: range possible_keys: k1,k2 key: k1,k2 key_len: 5,5 ref: NULL rows: 1 Extra: Using sort_intersect(k1,k2); Using where 1 row in set (0.01 sec) The query doesn't use any columns not covered by the used indexes, however, we don't see "using index". Analogous ROR-intersect query will use "Using index": MariaDB [timerge1]> EXPLAIN select 1 from t1 where k1=30 and k2 = 30\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: index_merge possible_keys: k1,k2 key: k1,k2 key_len: 5,5 ref: NULL rows: 1 Extra: Using intersect(k1,k2); Using where; Using index 1 row in set (0.00 sec) BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] review: [Commits] Rev 2875: Fix problem of making on-disk cache table. in file:///home/bell/maria/bzr/work-maria-5.3-subquerycache/
by Michael Widenius 17 Jan '11

17 Jan '11

Hi! Here is the review: >>>>> "sanja" == sanja <sanja(a)askmonty.org> writes: <cut> sanja> Fix problem of making on-disk cache table. sanja> Removed 'uniques=1' which lead to creating unneeded unique index creation. <cut> ---------------------------------------------------------------------- === modified file 'sql/sql_expression_cache.cc' --- a/sql/sql_expression_cache.cc 2010-11-02 09:47:36 +0000 +++ b/sql/sql_expression_cache.cc 2011-01-05 19:37:56 +0000 <cut> @@ -123,7 +126,6 @@ void Expression_cache_tmptable::init() goto error; } cache_table->s->keys= 1; - cache_table->s->uniques= 1; ref.null_rejecting= 1; ref.disable_cache= FALSE; ref.has_record= 0; The above should now already be in 5.3 tree as part of Sergei's push of my patch). (Just for your information) @@ -195,10 +197,12 @@ Expression_cache::result Expression_cach if (res) { subquery_cache_miss++; + miss++; DBUG_RETURN(MISS); } subquery_cache_hit++; + hit++; *value= cached_result; DBUG_RETURN(Expression_cache::HIT); As subquery_cache_miss and hit are global statistics variables, please use statistic_increment(subquery_cache_miss, &LOCK_status) with these. } @@ -225,7 +229,7 @@ my_bool Expression_cache_tmptable::put_v DBUG_ENTER("Expression_cache_tmptable::put_value"); DBUG_ASSERT(inited); - if (!cache_table) + if (!cache_table || frozen) { Is it not eonugh to test for 'frozen'? (You should set frozen to 0 if cache_table is not set) @@ -239,12 +243,22 @@ my_bool Expression_cache_tmptable::put_v if ((error= cache_table->file->ha_write_row(cache_table->record[0]))) { /* create_myisam_from_heap will generate error if needed */ + /* + TODO: add estimation if ondisk table is efficient if (cache_table->file->is_fatal_error(error, HA_CHECK_DUP) && create_internal_tmp_table_from_heap(table_thd, cache_table, cache_table_param.start_recinfo, &cache_table_param.recinfo, error, 1)) goto err; + */ Please change comment to #ifdef OVERFLOW_CACHE_TO_DISK_BASED_TABLE + if (cache_table->file->is_fatal_error(error, HA_CHECK_DUP)) + goto err; + /* should we leave the table */ + if ((((ulonglong)hit) * 100) / miss < EXPRASSION_TMPTABLE_ALLOWED_HITRATE) What happens if miss is 0 Shouldn't the above be? ((ulonglong) hit*100) / (hit + miss))... + goto err; // remove the temporary table + else Remove else + frozen= TRUE; // leave the temporary table } Try to move the above to before #ifdef and make it in such that one can just define the ifdef if one wants overflow to disk. === modified file 'sql/sql_expression_cache.h' <cut> @@ -50,6 +50,8 @@ class Item_field; class Expression_cache_tmptable :public Expression_cache { + ulong hit, miss; + bool frozen; ulong -> ulonglong (in a join you can have a LOT of row combinations) public: Expression_cache_tmptable(THD *thd, List<Item*> &dependants, Item *value); virtual ~Expression_cache_tmptable(); Ok to push after above changes. Regards, Monty

2 1

Re: [Maria-developers] test this list
by Vladislav Vaintroub 17 Jan '11

17 Jan '11

> -----Original Message----- > From: Vladislav Vaintroub [mailto:wlad@montyprogram.com] > Sent: Montag, 17. Januar 2011 13:12 > To: maria-developers(a)lists.launchpad.net > Subject: test this list > > Please ignore

1 0

[Maria-developers] lp: url support in firefox
by Sergei Golubchik 15 Jan '11

15 Jan '11

Hi. I've just written a small firefox add-on that adds support for lp: URI scheme to firefox. Nothing too magical, the add-on simply rewrites the url, replacing lp: with http://code.launchpad.net/ but at least now one can paste bzr urls (e.g. from bzr info, or email) directly into firefox and go to the corresponding branch page. https://addons.mozilla.org/en-US/firefox/addon/lp-uri-handler/ Regards, Sergei

1 0

[Maria-developers] test please ignore
by Vladislav Vaintroub 14 Jan '11

14 Jan '11

Checking mailing list access

1 0

Re: [Maria-developers] [Commits] Rev 2876: Fixed LP bug #698882. in file:///home/igor/maria/maria-5.3-bugs/
by Oleksandr Byelkin 14 Jan '11

14 Jan '11

Hi! On 13.01.2011 01:11, Igor Babaev wrote: > At file:///home/igor/maria/maria-5.3-bugs/ > > ------------------------------------------------------------ > revno: 2876 > revision-id: igor(a)askmonty.org-20110112231059-prhv2vncj6y2cizn > parent: igor(a)askmonty.org-20110105230330-2nzlm8641wcrwzio > committer: Igor Babaev<igor(a)askmonty.org> > branch nick: maria-5.3-bugs > timestamp: Wed 2011-01-12 15:10:59 -0800 > message: > Fixed LP bug #698882. > Made sure that the optimal fields are used by TABLE_REF objects > when building index access keys to joined tables. > Fixed a bug in the template function that sorts the elements of > a list using the bubble sort algorithm. The bug caused poor > performance of the function. Also added an optimization that > skips comparison with the most heavy elements that has been > already properly placed in the list. > Made the comparison of the fields belonging to the same Item_equal > more granular: fields belonging to the same table are also ordered > according to some rules. The patch as OK. But I'd prefer to have comment that the lists are mostly short and because of this used bubble sort algorithm (or you could use more detailed explanation from your letter to me). Also in big 'if (!cmp)' used tabulation (it is visible if you check diff where layout looks not correct, but it is correct in the result file), if it is not difficult it would be better to remove them. [skip]

1 0

[Maria-developers] WL#174 New (by Monty): Safety improvements for HANDLER commands
by worklog-noreply＠askmonty.org 14 Jan '11

14 Jan '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Safety improvements for HANDLER commands CREATION DATE..: Fri, 14 Jan 2011, 12:43 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 174 (http://askmonty.org/worklog/?tid=174) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 20 (hours remain) ORIG. ESTIMATE.: 20 PROGRESS NOTES: DESCRIPTION: There is some known problems with HANDLER implementation in some storage engines, that could potentially cause crashes that should be fixed ASAP. We should also do some trivial fixes to ensure that one gets Error 1020 (ER_CHECKREAD) Record has changed since last read in table '...' in so few cases as possible. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Documentation for Percona patch for row-based replication of pk-less tables
by Kristian Nielsen 13 Jan '11

13 Jan '11

Hi Daniel, I merged the Percona patch row_based_replication_without_primary_key.patch into 5.2-percona. This will later be merged into MariaDB 5.3. Here is some documentation, I was not sure where to put it, so hopefully you can help. The patch adds a new feature (performance improvement), and also fixes a bug. Incidentally, do we have a list of all bugs fixed in MariaDB over MySQL? I think that would be quite useful, even if it might get a bit out of date as MySQL also fixes some bugs. I suppose it could mostly be compiled from your release notes. This would also provide motivation for developers to actually make a proper bug report on issues fixed and proper commit messages etc :-) Anyway, just a suggestion. - Kristian. ----------------------------------------------------------------------- Row-based replication of tables with no primary key. MariaDB improves on row-based replication of tables which have no primary key but do have some other index. This is based in part on the original Percona patch "row_based_replication_without_primary_key.patch", with some additional fixes and enhancements. When row-based replication is used with UPDATE or DELETE, the slave needs to locate each replicated row based on the value in columns. If the table contains at least one index, an index lookup will be used (otherwise a table scan is needed for each row, which is extremely inefficient for all but the smallest table and generally to be avoided). >From MariaDB 5.3, the slave will try to choose a good index among any available: - The primary key is used, if there is one. - Else, the first unique index without NULL-able columns is used, if there is one. - Else, a choice is made among any normal indexes on the table (eg. a FULLTEXT index is not considered). The choice of which of several non-unique indexes to use is based on the cardinality of indexes; the one that is most selective (has the smallest average number of rows per distinct tuple of column values) is prefered. Note that for this choice to be effective, it is for most storage engines (like MyISAM, InnoDB) necessary to make sure ANALYZE TABLE has been run on the slave, otherwise statistics about index cardinality will not be available. In the absense of index cardinality, the first unique index will be chosen, if any, else the first non-unique index. Prior to MariaDB 5.3, the slave would always choose the first index without considering cardinality. The slave could even choose an unusable index (like FULLTEXT) if no other index was available (Bug #58997), causing row-based replication to break in this case; this is also fixed in MariaDB 5.3.

2 1

[Maria-developers] MWL#47 documentation
by Kristian Nielsen 12 Jan '11

12 Jan '11

Hi Daniel, I merged the MWL#47 patch into 5.2-percona, which will later be included in 5.3. I have attached the documentation for this worklog that Alexi wrote (he also implemented the feature itself). I hope you can adapt it as needed and find a good place for it in the server documentation. - Kristian. WL#47: Store in binlog text of statements that caused RBR events ================================================================ Content ~~~~~~~ 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] 2. Example 3. Master option: --binlog-annotate-rows-events 4. Slave option: --replicate-annotate-rows-events 5. mysqlbinlog option: --skip-annotate-rows-events 6. Example of mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ New binlog event type - Annotate_rows_log_event - is added. The events of this type accompany rows events and describe the query which caused these rows events. In binary log, each Annotate_rows event precedes the corresponding Table map event or the first of Table map events, if there are more than one (e.g. in case of multi-delete or insert delayed). 2. Example ~~~~~~~~~~ master> DROP DATABASE IF EXISTS test; master> CREATE DATABASE test; master> USE test; master> CREATE TABLE t1(a int); master> INSERT INTO t1 VALUES (1), (2), (3); master> CREATE TABLE t2(a int); master> INSERT INTO t2 VALUES (1), (2), (3); master> CREATE TABLE t3(a int); master> INSERT DELAYED INTO t3 VALUES (1), (2), (3); master> DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 -> WHERE t1.a=t2.a AND t2.a=t3.a; master> SHOW BINLOG EVENTS IN 'master-bin.000001'; +-------------------+------+---------------+-----------+-------------+---------------------------------------------------+ | Log_name | Pos | Event_type | Server_id | End_log_pos | Info | +-------------------+------+---------------+-----------+-------------+---------------------------------------------------+ | master-bin.000001 | 4 | Format_desc | 100 | 240 | Server ver: 5.1.44-debug-log, Binlog ver: 4 | | master-bin.000001 | 240 | Query | 100 | 331 | DROP DATABASE IF EXISTS test | | master-bin.000001 | 331 | Query | 100 | 414 | CREATE DATABASE test | | master-bin.000001 | 414 | Query | 100 | 499 | use `test`; CREATE TABLE t1(a int) | | master-bin.000001 | 499 | Query | 100 | 567 | BEGIN | | master-bin.000001 | 567 | Annotate_rows | 100 | 621 | INSERT INTO t1 VALUES (1), (2), (3) | | master-bin.000001 | 621 | Table_map | 100 | 662 | table_id: 16 (test.t1) | | master-bin.000001 | 662 | Write_rows | 100 | 706 | table_id: 16 flags: STMT_END_F | | master-bin.000001 | 706 | Query | 100 | 775 | COMMIT | | master-bin.000001 | 775 | Query | 100 | 860 | use `test`; CREATE TABLE t2(a int) | | master-bin.000001 | 860 | Query | 100 | 928 | BEGIN | | master-bin.000001 | 928 | Annotate_rows | 100 | 982 | INSERT INTO t2 VALUES (1), (2), (3) | | master-bin.000001 | 982 | Table_map | 100 | 1023 | table_id: 17 (test.t2) | | master-bin.000001 | 1023 | Write_rows | 100 | 1067 | table_id: 17 flags: STMT_END_F | | master-bin.000001 | 1067 | Query | 100 | 1136 | COMMIT | | master-bin.000001 | 1136 | Query | 100 | 1221 | use `test`; CREATE TABLE t3(a int) | | master-bin.000001 | 1221 | Query | 100 | 1289 | BEGIN | | master-bin.000001 | 1289 | Annotate_rows | 100 | 1351 | INSERT DELAYED INTO t3 VALUES (1), (2), (3) | | master-bin.000001 | 1351 | Table_map | 100 | 1392 | table_id: 18 (test.t3) | | master-bin.000001 | 1392 | Write_rows | 100 | 1426 | table_id: 18 flags: STMT_END_F | | master-bin.000001 | 1426 | Table_map | 100 | 1467 | table_id: 18 (test.t3) | | master-bin.000001 | 1467 | Write_rows | 100 | 1506 | table_id: 18 flags: STMT_END_F | | master-bin.000001 | 1506 | Query | 100 | 1575 | COMMIT | | master-bin.000001 | 1575 | Query | 100 | 1643 | BEGIN | | master-bin.000001 | 1643 | Annotate_rows | 100 | 1748 | DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a | | master-bin.000001 | 1748 | Table_map | 100 | 1789 | table_id: 16 (test.t1) | | master-bin.000001 | 1789 | Table_map | 100 | 1830 | table_id: 17 (test.t2) | | master-bin.000001 | 1830 | Delete_rows | 100 | 1874 | table_id: 16 | | master-bin.000001 | 1874 | Delete_rows | 100 | 1918 | table_id: 17 flags: STMT_END_F | | master-bin.000001 | 1918 | Query | 100 | 1987 | COMMIT | +-------------------+------+---------------+-----------+-------------+---------------------------------------------------+ 3. Master option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 4. Slave option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only (only in config file) * Data Type: bool * Default Value: OFF 5. mysqlbinlog option: --skip-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells mysqlbinlog to skip all Annotate_rows events in its output (by default, mysqlbinlog prints Annotate_rows events, if the binary log does contain them). 6. Example of mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ...> mysqlbinlog.exe -vv -R --user=root --port=3306 --host=localhost master-bin.000001 /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; # at 4 #100516 15:36:00 server id 100 end_log_pos 240 Start: binlog v 4, server v 5.1.44-debug-log created 100516 15:36:00 at startup ROLLBACK/*!*/; BINLOG ' oNjvSw9kAAAA7AAAAPAAAAAAAAQANS4xLjQ0LWRlYnVnLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAACg2O9LEzgNAAgAEgAEBAQEEgAA2QAEGggAAAAICAgCAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAA= '/*!*/; # at 240 #100516 15:36:18 server id 100 end_log_pos 331 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; SET @@session.pseudo_thread_id=1/*!*/; SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=1, @@session.unique_checks=1, @@session.autocommit=1 /*!*/; SET @@session.sql_mode=0/*!*/; SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/; /*!\C latin1 *//*!*/; SET @@session.character_set_client=8,@@session.collation_connection=8,@@session.collation_server=8/*!*/; SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; DROP DATABASE IF EXISTS test /*!*/; # at 331 #100516 15:36:18 server id 100 end_log_pos 414 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; CREATE DATABASE test /*!*/; # at 414 #100516 15:36:18 server id 100 end_log_pos 499 Query thread_id=1 exec_time=0 error_code=0 use test/*!*/; SET TIMESTAMP=1274009778/*!*/; CREATE TABLE t1(a int) /*!*/; # at 499 #100516 15:36:18 server id 100 end_log_pos 567 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; BEGIN /*!*/; # at 567 # at 621 # at 662 #100516 15:36:18 server id 100 end_log_pos 621 Annotate_rows: #Q> INSERT INTO t1 VALUES (1), (2), (3) #100516 15:36:18 server id 100 end_log_pos 662 Table_map: `test`.`t1` mapped to number 16 #100516 15:36:18 server id 100 end_log_pos 706 Write_rows: table id 16 flags: STMT_END_F BINLOG ' stjvSxNkAAAAKQAAAJYCAAAAABAAAAAAAAAABHRlc3QAAnQxAAEDAAE= stjvSxdkAAAALAAAAMICAAAQABAAAAAAAAEAAf/+AQAAAP4CAAAA/gMAAAA= '/*!*/; ### INSERT INTO test.t1 ### SET ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ### INSERT INTO test.t1 ### SET ### @1=2 /* INT meta=0 nullable=1 is_null=0 */ ### INSERT INTO test.t1 ### SET ### @1=3 /* INT meta=0 nullable=1 is_null=0 */ # at 706 #100516 15:36:18 server id 100 end_log_pos 775 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; COMMIT /*!*/; # at 775 #100516 15:36:18 server id 100 end_log_pos 860 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; CREATE TABLE t2(a int) /*!*/; # at 860 #100516 15:36:18 server id 100 end_log_pos 928 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; BEGIN /*!*/; # at 928 # at 982 # at 1023 #100516 15:36:18 server id 100 end_log_pos 982 Annotate_rows: #Q> INSERT INTO t2 VALUES (1), (2), (3) #100516 15:36:18 server id 100 end_log_pos 1023 Table_map: `test`.`t2` mapped to number 17 #100516 15:36:18 server id 100 end_log_pos 1067 Write_rows: table id 17 flags: STMT_END_F BINLOG ' stjvSxNkAAAAKQAAAP8DAAAAABEAAAAAAAAABHRlc3QAAnQyAAEDAAE= stjvSxdkAAAALAAAACsEAAAQABEAAAAAAAEAAf/+AQAAAP4CAAAA/gMAAAA= '/*!*/; ### INSERT INTO test.t2 ### SET ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ### INSERT INTO test.t2 ### SET ### @1=2 /* INT meta=0 nullable=1 is_null=0 */ ### INSERT INTO test.t2 ### SET ### @1=3 /* INT meta=0 nullable=1 is_null=0 */ # at 1067 #100516 15:36:18 server id 100 end_log_pos 1136 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; COMMIT /*!*/; # at 1136 #100516 15:36:18 server id 100 end_log_pos 1221 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; CREATE TABLE t3(a int) /*!*/; # at 1221 #100516 15:36:18 server id 100 end_log_pos 1289 Query thread_id=2 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; BEGIN /*!*/; # at 1289 # at 1351 # at 1392 #100516 15:36:18 server id 100 end_log_pos 1351 Annotate_rows: #Q> INSERT DELAYED INTO t3 VALUES (1), (2), (3) #100516 15:36:18 server id 100 end_log_pos 1392 Table_map: `test`.`t3` mapped to number 18 #100516 15:36:18 server id 100 end_log_pos 1426 Write_rows: table id 18 flags: STMT_END_F BINLOG ' stjvSxNkAAAAKQAAAHAFAAAAABIAAAAAAAAABHRlc3QAAnQzAAEDAAE= stjvSxdkAAAAIgAAAJIFAAAQABIAAAAAAAEAAf/+AQAAAA== '/*!*/; ### INSERT INTO test.t3 ### SET ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ # at 1426 # at 1467 #100516 15:36:18 server id 100 end_log_pos 1467 Table_map: `test`.`t3` mapped to number 18 #100516 15:36:18 server id 100 end_log_pos 1506 Write_rows: table id 18 flags: STMT_END_F BINLOG ' stjvSxNkAAAAKQAAALsFAAAAABIAAAAAAAAABHRlc3QAAnQzAAEDAAE= stjvSxdkAAAAJwAAAOIFAAAQABIAAAAAAAEAAf/+AgAAAP4DAAAA '/*!*/; ### INSERT INTO test.t3 ### SET ### @1=2 /* INT meta=0 nullable=1 is_null=0 */ ### INSERT INTO test.t3 ### SET ### @1=3 /* INT meta=0 nullable=1 is_null=0 */ # at 1506 #100516 15:36:18 server id 100 end_log_pos 1575 Query thread_id=2 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; COMMIT /*!*/; # at 1575 #100516 15:36:18 server id 100 end_log_pos 1643 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; BEGIN /*!*/; # at 1643 # at 1748 # at 1789 # at 1830 # at 1874 #100516 15:36:18 server id 100 end_log_pos 1748 Annotate_rows: #Q> DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 #Q> WHERE t1.a=t2.a AND t2.a=t3. #100516 15:36:18 server id 100 end_log_pos 1789 Table_map: `test`.`t1` mapped to number 16 #100516 15:36:18 server id 100 end_log_pos 1830 Table_map: `test`.`t2` mapped to number 17 #100516 15:36:18 server id 100 end_log_pos 1874 Delete_rows: table id 16 #100516 15:36:18 server id 100 end_log_pos 1918 Delete_rows: table id 17 flags: STMT_END_F BINLOG ' stjvSxNkAAAAKQAAAP0GAAAAABAAAAAAAAAABHRlc3QAAnQxAAEDAAE= stjvSxNkAAAAKQAAACYHAAAAABEAAAAAAAAABHRlc3QAAnQyAAEDAAE= stjvSxlkAAAALAAAAFIHAAAAABAAAAAAAAAAAf/+AQAAAP4CAAAA/gMAAAA= ### DELETE FROM test.t1 ### WHERE ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ### DELETE FROM test.t1 ### WHERE ### @1=2 /* INT meta=0 nullable=1 is_null=0 */ ### DELETE FROM test.t1 ### WHERE ### @1=3 /* INT meta=0 nullable=1 is_null=0 */ stjvSxlkAAAALAAAAH4HAAAQABEAAAAAAAEAAf/+AQAAAP4CAAAA/gMAAAA= '/*!*/; ### DELETE FROM test.t2 ### WHERE ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ### DELETE FROM test.t2 ### WHERE ### @1=2 /* INT meta=0 nullable=1 is_null=0 */ ### DELETE FROM test.t2 ### WHERE ### @1=3 /* INT meta=0 nullable=1 is_null=0 */ # at 1918 #100516 15:36:18 server id 100 end_log_pos 1987 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1274009778/*!*/; COMMIT /*!*/; DELIMITER ; # End of log file ROLLBACK /* added by mysqlbinlog */; /*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;

1 0

[Maria-developers] MWL#36 documentation
by Kristian Nielsen 12 Jan '11

12 Jan '11

Hi Daniel, While merging Percona patches, I noticed a couple of issues with our documentation relating to MWL#36, I was hoping you could handle them: - The MWL#36 is mentioned in the changelog/release notes: http://kb.askmonty.org/v/mariadb-520-release-notes However it seems to be missing from "mariadb-vs-mysql": http://kb.askmonty.org/v/mariadb-versus-mysql-features - There is some documentation of the MWL#36 in the form of a table entry for --rewrite-db here: http://kb.askmonty.org/v/mysqlbinlog However, that page mentions the feature introduced in MariaDB 5.2.1, while the release notes say 5.2.0. I wonder which is correct? - Kristian.

1 0

[Maria-developers] Multiple default values for timestamps
by Dolf Schimmel 09 Jan '11

09 Jan '11

Hey folks, On IRC I read somewhere that multiple timestamps cant have multiple default values using NOW() (error ER_TOO_MUCH_AUTO_TIMESTAMP_COLS). I tried to find out why this limitation is in place, but was not able to do so. The only thing I (or rather someone in #mysql and #maria) was able to find was this, in sql/field.cc (line 4729): > Actually SQL-99 says that we should allow niladic functions (like NOW()) > as defaults for any field. Current limitations (only NOW() and only > for one TIMESTAMP field) are because of restricted binary .frm format > and should go away in the future. I removed the check that spits out this error and updated the tests. Now I'll admit I'm not sure how to properly test this, but so far the entire testsuite passes. It may be that I'm missing any reasons for this limitation, or that tests may be missing/inaccurate. If you have any opinions on this, please let me know. You can find it in: lp:~freeaqingme/maria/improv-allowMultipleCurTimestampInDefaultOnTable Regards, Dolf Schimmel -- Freeaqingme p.s. Sorry for the long branchname, couldn't think of something shorter that was still descriptive :D

2 3

[Maria-developers] WL#173 New (by Sergei): temporal types with sub-second resolution
by worklog-noreply＠askmonty.org 09 Jan '11

09 Jan '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: temporal types with sub-second resolution CREATION DATE..: Sun, 09 Jan 2011, 20:08 SUPERVISOR.....: Sergei IMPLEMENTOR....: Sergei COPIES TO......: Monty CATEGORY.......: Server-Sprint TASK ID........: 173 (https://askmonty.org/worklog/?tid=173) VERSION........: Server-5.3 STATUS.........: Assigned PRIORITY.......: 80 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: We will add a support for timestamps (and other temporal types) with a sub-second resolution. Syntax ^^^^^^ The syntax for declaring a column is the conventional TIMESTAMP(X) DATETIME(X) TIME(X) where X is the scale - that is, number of digits after the decimal dot. TIMESTAMP(0) is the same as the current TIMESTAMP. The existing syntax (without specifying the scale) will, of course, work as before. X=6 means microseconds, X=9 means nanoseconds, X=12 means picoseconds. The max supported value of X is [undecided], but will be no less than 6. Possibly, we will do 6 in this WL - because it keeps syntax and protocol changes to the minimum, microseconds are already supported there. If needed, we will increase the timestamp precision later - the implementation (see below) needs to keep it in mind. The time and datetime literals can be written with the fractional part, for example '2011-01-08 20:29:26.1234567890'. This is alrteady supported. Client-server protocol ^^^^^^^^^^^^^^^^^^^^^^ We will not change the binary client-server protocol. It only supports microseconds in the MYSQL_TIME::second_part - we will truncate longer timestamps to this precision. Show commands and I_S ^^^^^^^^^^^^^^^^^^^^^ SHOW COLUMNS and SHOW CREATE TABLE will show TIMESTAMP(X) as appropriate (similar to INT(N) and DECIMAL(M,N) types). I_S.COLUMNS will *not* show '(X)' in the DATA_TYPE column, but it *will* show it in the COLUMN_TYPE column. It will show X in the new DATETIME_PRECISION column. Functions ^^^^^^^^^ The following functions need to be fixed to support sub-seconds: CAST() - as DATETIME(X), etc CONVERT_TZ() SEC_TO_TIME() TIME_TO_SEC() MAKETIME() - perhaps? FROM_UNIXTIME() - probably not UNIX_TIMESTAMP() - probably not UTC_TIMESTAMP(), UTC_TIME(), NOW(), CURTIME() and synonyms - probably not, high-precision NOW() is a separate task P.S. See also http://forge.mysql.com/worklog/task.php?id=946 ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] The query cache patch fixed by me and then Monty (most changes by him)
by Oleksandr Byelkin 07 Jan '11

07 Jan '11

Hi! Here is changes which was done: - Fixed switching off of the query cache -- memory freed now -- after completely switching of it could be switched on again - Cleaned up saving the query without comments and space - Added all the missing test cases (and checked results) - Added more symbols to LEX to make parsing faster (not part of this thing, but I thought it was better to get this done now as I was looking at the code) - Fixed all the issues that I had in my review of query parsing - Cleaned up mysql-tests -- Made test results easier to read -- removed 'percona' prefix from tests names -- Made test result files notable smaller - Added 'thd' to query_cache::invalidate calls, to remove some current_thd usages - Added variable thd->query_cache_is_valid, that is set to 1 if query_cache could possible read query from the query_cache -- If not, there is no reason to put the query in the cache The above change allowed me to remove safe_to_cache_queries from send_result_to_client() and make this part of the code not depending in lex_start() - simplified a couple of functions to use pointers instead of index - Fixed bug where SQL_NO_CACHE was not detected if followed by tab, newline or return i> that's about it 2 Vadim & Oleg: If you OK with changes above and the patch then it will be included in our tree.

2 3

[Maria-developers] [patch] A few typos
by Dolf Schimmel 07 Jan '11

07 Jan '11

Morning folks, I was looking at the source for a certain function in MariaDB (see the patch and you'll know which one :D ). While grepping I found a couple of typos and since I don't have commit rights, and couldn't find where to send the patch, I decided to send it here :) The diff is made against trunk, but I guess it should be fairly easy to patch against other branches as well. Kind regards, Dolf Schimmel -- Freeaqingme

2 1

Re: [Maria-developers] Restarting cooperation of Percona and Monty Programming AB
by Kristian Nielsen 06 Jan '11

06 Jan '11

Michael Widenius <monty(a)askmonty.org> writes: >>>>>> "Kristian" == Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > > <cut> > > Kristian> Anyway, I fixed the patch, added test cases, and merged into MariaDB. It is > Kristian> pushed here: > > Kristian> lp:~knielsen/maria/5.2-percona-row_based_replication_without_primary_key > > Kristian> Here is the commit message: > > Kristian> Merge Percona patch row_based_replication_without_primary_key.patch into MariaDB. > > Kristian> This patch improves the selection of index to use to apply row-based > Kristian> DELETE and UPDATE events on tables with no primary key (original code > Kristian> picks the first index unconditionally). > Kristian> If ANALYZE TABLE is done, the index cardinalities will be compared and > Kristian> the best index will be used. > > Kristian> Fixes some problems in the original patch: > > Kristian> - Without ANALYZE TABLE, rec_per_key statistics is not available; in this > Kristian> case the original patch could choose a really bad index, even ignoring > Kristian> a primary key. > > Kristian> - The original patch did not consider multi-column keys correctly, and > Kristian> could thus pick a less desirable single-column key over a good > Kristian> multi-column index. > > Kristian> Also fixes Bug#58997, and adds test cases. > > Kristian> Monty, how do I include it in our percona tree?, can I just push here: > > Kristian> lp:~maria-captains/maria/5.2-percona > > Yes, feel free to push if no one from Percona has any problems with > your patch. So, any objections or comments from Percona on this patch? I've included the patch below for convenience. - Kristian.

2 2

[Maria-developers] WL#172 New (by Monty): Add support for prepared statements to HANDLER READ
by worklog-noreply＠askmonty.org 03 Jan '11

03 Jan '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add support for prepared statements to HANDLER READ CREATION DATE..: Mon, 03 Jan 2011, 19:19 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 172 (http://askmonty.org/worklog/?tid=172) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: DESCRIPTION: Add support for prepared statements to HANDLER READ. As part of this work, move some of the word done during handler_read to handler_open to speed up HANDLER READ calls. Example: handler t1 open; prepare stmt from 'handler t1 read a=(?) limit ?,?'; set @a=16,@b=1,@c=100; execute stmt using @a,@b,@c; deallocate prepare stmt; ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

Re: [Maria-developers] [Commits] Rev 2999: Speed up `mtr --parallel=<lots>` by scheduling some slow tests earlier. in http://bazaar.launchpad.net/~maria-captains/maria/5.1
by Sergei Golubchik 03 Jan '11

03 Jan '11

Hi, knielsen! On Jan 03, knielsen(a)knielsen-hq.org wrote: > At http://bazaar.launchpad.net/~maria-captains/maria/5.1 Hm. Why do you prefer to add a dummy include file to every "long" test, instead of using a separate file, like disabled.def ? I think it'd be easier to maintain in a separate file, and easier to extend later on. Regards, Sergei

2 2

[Maria-developers] Logoff trigger for MariaDB
by Oli Sennhauser 31 Dec '10

31 Dec '10

Hi all, As suggested by Sergei we should discuss this here. Copying the login trigger stuff to a place in the code where I thought it makes sense I have created a logout trigger. This gives the capability to track a session from its beginning to its end and collects all the data from each session. With this capability we can start thinking about auditing of MySQL users like it is needed/wanted from big companies. I am NOT a developer and I was just lucky that it worked out how I did it. So I am not of much help for an other approach as Sergei hinted in his comment. Now: free to discuss... http://www.fromdual.com/mysql-logon-and-logoff-trigger-for-auditing Regards, Oli -- FromDual - Vendor independent and neutral MySQL consulting. Website http://www.fromdual.com/ Skype fromdual Jabber fromdual(a)swissjabber.ch Yahoo IM fromdual Xing https://www.xing.com/profile/Oliver_Sennhauser LinkedIn http://ch.linkedin.com/in/shinguz Twitter http://twitter.com/shinguz

2 2

Re: [Maria-developers] [Fwd: [Commits] Rev 2998: Fixed LP bug #639935 (bug #58727). in file:///home/igor/maria/maria-5.1-bug693935/]
by Oleksandr Byelkin 24 Dec '10

24 Dec '10

Hi! 24 Dec 2010, в 06:45, Igor Babaev написал(а): > Sanja, > > Please review this fix ASAP. > We need it to move on with 5.3 The patch is OK, but I would prefer to have comment before Item::quick_fix_field() that this method is only for optimizer needs, it can be used only for certain Items controlled by optimizer and maybe add that only Item_func_* need recursion for this. But you could write comments much better then I so it is up to you how write the comment. > > Regards. > Igor. > > -------- Original Message -------- > Subject: [Commits] Rev 2998: Fixed LP bug #639935 (bug #58727). in > file:///home/igor/maria/maria-5.1-bug693935/ > Date: Fri, 24 Dec 2010 06:42:05 +0200 (EET) > From: <igor(a)askmonty.org> > Reply-To: maria-developers(a)lists.launchpad.net > To: <commits(a)mariadb.org> > > At file:///home/igor/maria/maria-5.1-bug693935/ > > ------------------------------------------------------------ > revno: 2998 > revision-id: igor(a)askmonty.org-20101224044150-p9yja1e87riwjh7g > parent: monty(a)askmonty.org-20101221172350-wqtf0t6evpc0un4b > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.1-bug693935 > timestamp: Thu 2010-12-23 20:41:50 -0800 > message: > Fixed LP bug #639935 (bug #58727). > When the optimizer creates items out of other items it does > not have to call the fix_fields method. Usually in these > cases it calls quick_fix_field() that just marks the > created item as fixed. If the created item is an Item_func > object then calling quick_fix_field() works fine if the > arguments of the created functional item are already fixed. > Otherwise some unfixed nodes remain in the item tree and > it triggers an assertion failure whenever the item is > evaluated. > > Fixed the problem by making the method quick_fix_field > virtual and providing an implementation for the class > Item_func objects that recursively calls the method > for unfixed arguments of any functional item. [skip]

1 0

Re: [Maria-developers] pre-allocating binlog to speed up sync_binlog=1
by Sergei Golubchik 23 Dec '10

23 Dec '10

Hi, Arjen! [moving to the public list] On Dec 14, Arjen Lentz wrote: > Hi all > > Can we adopt/implement http://forge.mysql.com/worklog/task.php?id=4925 in > MariaDB? > The benchmark info is in the item, and looks quite interesting. > The author tested it using a separate tool to preallocate the binlog, but > it should be straightforward implementing this inside the server, that'd > also be much easier to manage without extra fuss for a DBA. You mean - before it gets into 5.6 and we merge it the normal way? Or you mean - with a different implementation (w/o mysqlbinlogalloc) ? Regards, Sergei P.S. These are the changesets: final version: http://lists.mysql.com/commits/113309 http://lists.mysql.com/commits/113306 http://lists.mysql.com/commits/113307 review comments: http://lists.mysql.com/commits/116965 http://lists.mysql.com/commits/121478

4 6