- developers - lists.mariadb.org

[Maria-developers] MWL#89 patch for review
by Timour Katchaounov 30 Mar '11

30 Mar '11

Monty, Please find attached an updated patch for MWL#89. The patch is merged with the most current 5.3 as of today, March 30. The tree itself is here: https://code.launchpad.net/~maria-captains/maria/5.3-mwl89 Timour

1 0

Re: [Maria-developers] [Commits] Rev 2947: Fix LP BUG#613029 in file:///home/tsk/mprog/src/5.3/
by Sergey Petrunya 29 Mar '11

29 Mar '11

Hi Timour, Ok to push. On Mon, Mar 28, 2011 at 12:55:43PM +0300, timour(a)askmonty.org wrote: > At file:///home/tsk/mprog/src/5.3/ > > ------------------------------------------------------------ > revno: 2947 > revision-id: timour(a)askmonty.org-20110328095536-wbmu1hiwsnhw6bs8 > parent: timour(a)askmonty.org-20110324143406-04q5peh1r7nthcyb > committer: timour(a)askmonty.org > branch nick: 5.3 > timestamp: Mon 2011-03-28 12:55:36 +0300 > message: > Fix LP BUG#613029 > > Analysis: > There are two code paths through which JOIN::exec may produce > an all-NULL row for an empty result set. One goes via the > function return_zero_rows(), when query processing detectes > early that the where clause is false, the other one is via > do_select() in the case of join execution. > > In the case of do_select(), the problem was that the executioner > didn't set TABLE::null_row to 1. As result when sending the only > result row, the evaluation of each field didn't detect that all > non-aggregated fields are NULL, because Field::is_null returned > true, after checking that field->table->null_row was false. > > Given that the each non-aggregated field was not considered NULL, > select_result::send_data sent whatever was in the buffer of each > field. However, since there was no actual data in the field buffer, > send_data() accessed and sent whatever junk was in the field's > data buffer. > > Solution: > Similar to the analogous case in return_zero_rows() mark all > tables that their current row is NULL before sending the > artificailly created NULL row. > === modified file 'mysql-test/r/subselect4.result' > --- a/mysql-test/r/subselect4.result 2011-03-24 14:34:06 +0000 > +++ b/mysql-test/r/subselect4.result 2011-03-28 09:55:36 +0000 > @@ -1133,3 +1133,55 @@ SELECT * FROM t1 WHERE (2, 0) NOT IN (SE > f1 f2 > set @@optimizer_switch=@save_optimizer_switch; > drop table t1,t2; > +# > +# LP BUG#613029 Wrong result with materialization and semijoin, and > +# valgrind warnings in Protocol::net_store_data with materialization > +# for implicit grouping > +# > +CREATE TABLE t1 ( > +pk int(11) NOT NULL AUTO_INCREMENT, > +f2 int(11) NOT NULL, > +f3 varchar(1) NOT NULL, > +PRIMARY KEY (pk), > +KEY f2 (f2)); > +INSERT INTO t1 VALUES (1,9,'x'); > +INSERT INTO t1 VALUES (2,5,'g'); > +CREATE TABLE t2 ( > +pk int(11) NOT NULL AUTO_INCREMENT, > +f2 int(11) NOT NULL, > +f3 varchar(1) NOT NULL, > +PRIMARY KEY (pk), > +KEY f2 (f2)); > +INSERT INTO t2 VALUES (1,7,'p'); > +set @save_optimizer_switch=@@optimizer_switch; > +set @@optimizer_switch='materialization=off,semijoin=off'; > +EXPLAIN > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables > +2 DEPENDENT SUBQUERY t1 index_subquery f2 f2 4 func 2 Using index > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > +f3 MAX(t1.f2) > +NULL NULL > +set @@optimizer_switch='materialization=on,semijoin=off'; > +EXPLAIN > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY t2 system PRIMARY NULL NULL NULL 1 > +1 PRIMARY t1 const PRIMARY PRIMARY 4 const 1 > +2 SUBQUERY t1 index NULL f2 4 NULL 2 Using index > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > +f3 MAX(t1.f2) > +NULL NULL > +TODO: add a test case for semijoin when the wrong result is fixed > +set @@optimizer_switch='materialization=off,semijoin=on'; > +set @@optimizer_switch=@save_optimizer_switch; > +drop table t1, t2; > > === modified file 'mysql-test/t/subselect4.test' > --- a/mysql-test/t/subselect4.test 2011-03-24 14:34:06 +0000 > +++ b/mysql-test/t/subselect4.test 2011-03-28 09:55:36 +0000 > @@ -906,3 +906,60 @@ SELECT * FROM t1 WHERE (2, 0) NOT IN (SE > set @@optimizer_switch=@save_optimizer_switch; > > drop table t1,t2; > + > +--echo # > +--echo # LP BUG#613029 Wrong result with materialization and semijoin, and > +--echo # valgrind warnings in Protocol::net_store_data with materialization > +--echo # for implicit grouping > +--echo # > + > +CREATE TABLE t1 ( > + pk int(11) NOT NULL AUTO_INCREMENT, > + f2 int(11) NOT NULL, > + f3 varchar(1) NOT NULL, > + PRIMARY KEY (pk), > + KEY f2 (f2)); > + > +INSERT INTO t1 VALUES (1,9,'x'); > +INSERT INTO t1 VALUES (2,5,'g'); > + > +CREATE TABLE t2 ( > + pk int(11) NOT NULL AUTO_INCREMENT, > + f2 int(11) NOT NULL, > + f3 varchar(1) NOT NULL, > + PRIMARY KEY (pk), > + KEY f2 (f2)); > + > +INSERT INTO t2 VALUES (1,7,'p'); > + > +set @save_optimizer_switch=@@optimizer_switch; > + > +set @@optimizer_switch='materialization=off,semijoin=off'; > + > +EXPLAIN > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > + > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > + > +set @@optimizer_switch='materialization=on,semijoin=off'; > + > +EXPLAIN > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > + > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > + > +-- echo TODO: add a test case for semijoin when the wrong result is fixed > +-- echo set @@optimizer_switch='materialization=off,semijoin=on'; > + > + > +set @@optimizer_switch=@save_optimizer_switch; > + > +drop table t1, t2; > > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2011-03-24 14:34:06 +0000 > +++ b/sql/sql_select.cc 2011-03-28 09:55:36 +0000 > @@ -13452,6 +13452,13 @@ do_select(JOIN *join,List<Item> *fields, > { > List<Item> *columns_list= (procedure ? &join->procedure_fields_list : > fields); > + /* > + With implicit grouping all fields of special row produced for an > + empty result are NULL. See return_zero_rows() for the same behavior. > + */ > + for (TABLE_LIST *table= join->select_lex->leaf_tables; > + table; table= table->next_leaf) > + mark_as_null_row(table->table); > rc= join->result->send_data(*columns_list); > } > } > > === modified file 'sql/sql_select.h' > --- a/sql/sql_select.h 2011-03-13 10:50:14 +0000 > +++ b/sql/sql_select.h 2011-03-28 09:55:36 +0000 > @@ -954,8 +954,8 @@ public: > bool init_save_join_tab(); > bool send_row_on_empty_set() > { > - return (do_send_rows && tmp_table_param.sum_func_count != 0 && > - !group_list && having_value != Item::COND_FALSE); > + return (do_send_rows && implicit_grouping && > + having_value != Item::COND_FALSE); > } > bool change_result(select_result *result); > bool is_top_level_join() const > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] WL#191 New (by Knielsen): status variables for last InnoDB binlog position
by worklog-noreply＠askmonty.org 29 Mar '11

29 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: status variables for last InnoDB binlog position CREATION DATE..: Tue, 29 Mar 2011, 09:06 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 191 (http://askmonty.org/worklog/?tid=191) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: In every commit, InnoDB/XtraDB stores the binlog position in the system tablespace (in a transactional-safe way). At server start, this position is printed to the error log. This is useful after taking a backup to provision a slave, as it provides the values to use for CHANGE MASTER TO. It would be useful if these values printed to the error log were also directly available to SQL as read-only status variables: innodb_startup_binlog_master_file innodb_startup_binlog_master_pos This would be generally useful. When this is implemented, the test case main.group_commit_binlog_pos can be fixed to not require `grep` and `tail`, and so work on Windows with no posix tools installed. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Improved table elimination
by Lars Rönnbäck 29 Mar '11

29 Mar '11

When doing some comparisons of query optimization in Microsoft SQL Server and MariaDB I noticed that there is a second type of table elimination which MariaDB has not yet implemented. It depends on foreign keys being declared, and can under some circumstances substantially improve performance. Rather than to write a lengthy discussion, here is a test script for MariaDB 5.2.5 in which the queries with the condition "where ... is not null" should result in this type of table elimination: http://pastebin.com/s68RwgG2 This screencast shows the table elimination in effect in MSSQL: http://www.screenr.com/tPG We rely on table elimination to get better performance in 3NF-views and functions used in our Anchor Modeling (6NF-table) technique. However, the need for table elimination has previously come from denormalized modeling, where it is used to speed up queries against star schemas (e g eliminating dimension tables). It should therefore not only be appreciated by our crowd, if implemented in full. We were very happy to see the 'outer join table elimination' implemented and are wondering if someone could put in the effort to add the 'foreign key table elimination'? Since everything we do with Anchor Modeling is Open Source, it would be nice to have an Open Source database that fully supports the technique. Support in other databases is described here, if anyone is interested: http://www.anchormodeling.com/?page_id=22 Thanks, Lars

1 0

[Maria-developers] WL#190 New (by Yakatz): Optional parameters support in stored procedures
by worklog-noreply＠askmonty.org 28 Mar '11

28 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Optional parameters support in stored procedures CREATION DATE..: Mon, 28 Mar 2011, 20:26 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 190 (http://askmonty.org/worklog/?tid=190) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: This bug has been open on MySQL since 2005. In a nutshell, support PROCEDUREs like this: CREATE PROCEDURE ListUsers ( @userID int, @authorized bit = NULL ) AS.... ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Review of replication/group commit work
by Kristian Nielsen 28 Mar '11

28 Mar '11

Hi Monty, Thanks for offering to review my replication worklogs. Here are pointers to the various worklogs and patches, and suggestions for order to things in etc. Some background information is in this series on my blog: http://kristiannielsen.livejournal.com/12254.html http://kristiannielsen.livejournal.com/12408.html http://kristiannielsen.livejournal.com/12553.html http://kristiannielsen.livejournal.com/12810.html There is a tree on launchpad based on 5.2 with all the patches merged, if you want to look at the actual code in one piece: lp:~maria-captains/maria/mariadb-5.2-rpl ---- The main worklog, and the place to start, is MWL#116: http://askmonty.org/worklog/Server-Sprint/?tid=116 I think you'll have to read through the worklog/design first (it already had architecture review by Serg) in order to properly review the code. The patch for MWL#116 is here: https://lists.launchpad.net/maria-developers/msg03693.html It also has links to a lauchpad tree with just this patch. This worklog extends the storage engine API with a facility to efficiently coordinate commit order among transaction participants, and also implements group commit for binlog. ---- The next worklog is MWL#136: http://askmonty.org/worklog/Server-RawIdeaBin/?tid=136 Here is the commit mail: https://lists.askmonty.org/pipermail/commits/2010-November/000735.html (also this one, but it just renames some variables to bettter names): https://lists.askmonty.org/pipermail/commits/2010-November/000741.html This worklog makes START TRANSACTION WITH CONSISTENT SNAPSHOT, well, consistent. It also uses this for the binlog to provide non-blocking mysqldump --single-transaction --master-data. ---- The last worklog is MWL#163: http://askmonty.org/worklog/Server-Sprint/?tid=163 Here is the commit mail: https://lists.askmonty.org/pipermail/commits/2010-November/000711.html This is a re-write of a Facebook patch. Mark has mentioned this on several occasions as being important for them to fix a hot-spot. It implements an option to release InnoDB row locks earlier, before committing to binlog and storage engine (which is not safe in general, but can be safe in certain applications such as Facebook presumably). ---- That's it for now! - Kristian.

2 1

[Maria-developers] review: [Commits] Rev 2863: MWL#163: release of row locks in InnoDB during prepare in http://bazaar.launchpad.net/~maria-captains/maria/5.1
by Michael Widenius 28 Mar '11

28 Mar '11

Hi! >>>>> "knielsen" == knielsen <knielsen(a)knielsen-hq.org> writes: knielsen> At http://bazaar.launchpad.net/~maria-captains/maria/5.1 knielsen> ------------------------------------------------------------ knielsen> revno: 2863 knielsen> revision-id: knielsen(a)knielsen-hq.org-20101103160708-88w8p8ibe9iaoggj knielsen> parent: knielsen(a)knielsen-hq.org-20101102074027-jlnx0rnbdpovl122 knielsen> committer: knielsen(a)knielsen-hq.org knielsen> branch nick: work-5.1-mwl163 knielsen> timestamp: Wed 2010-11-03 17:07:08 +0100 knielsen> message: knielsen> MWL#163: release of row locks in InnoDB during prepare knielsen> Port the Facebook patch for releasing InnoDB row locks early to the knielsen> MWL#116 framework. knielsen> === modified file 'storage/xtradb/handler/ha_innodb.cc' knielsen> --- a/storage/xtradb/handler/ha_innodb.cc 2010-11-01 14:41:09 +0000 <cut> Spent 30 minutes checking the innodb code and things looked ok. Regards, Monty

1 0

[Maria-developers] Writeup of how to run my group commit benchmark
by Kristian Nielsen 28 Mar '11

28 Mar '11

Hi, For anyone interested in trying to redo my group commit benchmarks on more interesting hardware than what I have available, here is a writeup of the steps needed to run it. The benchmarks are described here: http://kristiannielsen.livejournal.com/15739.html http://kristiannielsen.livejournal.com/15515.html 1. Compile gypsy. This is used to generate the load. I use a specific version as the latest in trunk seemed to not build on non-windows last I tried: bzr branch '-rrevid:shane@mysql.com-20100224065101-jk3s5sro2p914nvp' lp:gypsy cd gypsy make This requires `mysql_config` in the path to get includes/libraries for libmysqlclient. It also needs openssl headers/libraries (eg. install libssl-dev package). 2. Install MariaDB with group commit enabled. Packages/sources are available for this here: http://kb.askmonty.org/v/mariadb-52-replication-feature-preview These packages are however missing the --binlog-optimize-thread-scheduling option that I used to compare the two ways to schedule threads in the above blog posts. To get that option, grab this bzr tree and build: bzr branch lp:~knielsen/maria/mariadb-5.1-mwl116-threadscheduling cd mariadb-5.1-mwl116-threadscheduling BUILD/compile-pentium64-max 3. Configure and start mysqld This involves `make install` and `mysql_install_db` when building from source, check README/INSTALL for details. For configuration, set whatever parameters are interesting to test. Eg. set InnoDB buffer pool size and log file/buffer size to something sensible, set innodb_flush_log_at_trx_commit=1 and sync_binlog=1. Also create a mysql account and a database with suitable access permissions. 4. Create the table to use: mysql --host=127.0.0.1 -utest -ptestpass test MariaDB [test]> create table t (a int primary key, b int) engine=innodb; 5. Create the script for Gypsy to generate the load. Just create a file simpleload.gypsy containing this single line: ----------------------------------------------------------------------- p|1|REPLACE INTO t (a,b) VALUES (? MOD 1000000, ? MOD 1000000)|int,int ----------------------------------------------------------------------- 6. Run the bemchmark. I run it with `script` to capture the output (yeah, I know it's crude). The `gypsy` binary is found in the directory where it was compiled in step (1). script for i in 1 4 8 12 16 24 32 48 64 96 128 ; do gypsy --host=127.0.0.1 --user=test --password=testpass --database=test --queryfile=simpleload.gypsy --duration=180 --threads=$i ; done Adjust host/user/password as appropriate for the installation of mariadb. 7. That should be it! This is obviously a rather crude benchmark, sorry about that, I did not have experience with something like sysbench or tcp-c, would be interesting to see results from such benchmarks also. - Kristian.

1 0

Re: [Maria-developers] Bundle more relevant cnf files in MariaDB package
by Arjen Lentz 28 Mar '11

28 Mar '11

Hi Haidong On 12/03/2011, at 7:56 AM, Haidong Ji wrote: > I alluded to this in IRC maybe a couple of weeks back. That is, it is > really to have intelligent default for MariaDB servers. there already is a baseline my.cnf which is installed by the DEB (Debian/Ubuntu) packages. You'll find it in the debian build subdirs. It can a) be updated with more parameters b) also be put into RPM, tarball and other builds. I would recommend not redoing it from scratch, as a number of practical issues have to be considered for this kind of default config, and reinventing the wheel there is just not necessary. Simply extend what's already there! > In particular, the bundled sample my.cnf files, under > /usr/local/mysql/support-files, such as my-small.cnf, my-medium.cnf, > my-large.cnf, and my-huge.cnf, are laughable in today's world. yep they are crap and should be removed. However, since they've already been replaced in the real world this is a separate issue. The my-*.cnf are not proof of "we need to create a good sample config" but rather a "this is old junk that should be removed from the bzr tree." The hindrance there is that some builds currently use them, so simply deleting the files is not sufficient. I'll see if I can find those dependencies, and create a modified tree for merging. Cheers, Arjen. -- Arjen Lentz, Exec.Director @ Open Query (http://openquery.com) Remote expertise & maintenance for MySQL/MariaDB server environments. Follow us at http://openquery.com/blog/ & http://twitter.com/openquery _______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss(a)lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

1 0

[Maria-developers] Pushing/merging sample my.cnf
by Haidong Ji 25 Mar '11

25 Mar '11

All, I've created my trunk on launchpad for MariaDB, and am ready to make some updates to the sample my.cnf file(s) packaged with MariaDB. I have a few questions regarding how to proceed: 1. I've seen work log creation and assignment in MariaDB developer mailing list. Is work log creation necessary for changes like this, where no C/C++ source code is touched? If yes, any guidelines will be appreciated; 2. As this is more packaging related, what are the testing requirement? 3. What about peer review? I've blogged this and also just sent another email for more input. 4. After I make the change on my machine, build it, and push the changes back to Lauchpad, I will do a merge request. 5. Anything else? Comments/suggestions? Thanks, Haidong "Alex" Ji

2 4