- developers - lists.mariadb.org

[Maria-developers] Idea for parallel replication with statement-based binlogging
by Kristian Nielsen 17 Feb '13

17 Feb '13

Hi Xiaobin, I hope you are enjoying the Spring Festival! All the best wishes for you and your family for the coming year. I had an idea for your parallel replication work, inspired by my work on in-order commit. I wanted to hear your opinion on it, if you are interested. The idea is to make your patch work with statement-based replication. For row-based InnoDB events, you hash all the unique keys to check for conflicts, and if no conflict you can run transactions in parallel even against the same table. For MyISAM, I get deadlock issues in my in-order-commit patch, if I try to replicate in parallel against the same table. So I hash just the table names, so that two transactions on the same table always conflict. Then I can still replicate in parallel events against different tables. Now suppose that for statement-based events, we obtain the names of all tables used in the queries. We could add a new event to the binlog with the table names, or perhaps parse the statement on the slave, if that is not too expensive. Then we could hash just the names to check for conflicts, just like I did for MyISAM with row-based events. It seems to me that we could then safely replicate statement-based events in parallel on the slave if they are to different tables. This would be great for people that do not want to use row-based replication, it would surely be better than the MySQL 5.6 multi-threaded slave which can only do queries against different databases in parallel. What do you think? - Kristian.

2 1

[Maria-developers] implementing index condition pushdown in MariaDB 5.5
by Zardosht Kasheff 15 Feb '13

15 Feb '13

Hello, Is there any documentation for what a storage engine needs to do to implement index condition pushdown in MariaDB 5.5? I see some related things, such as handler_index_cond_check, HA_DO_INDEX_COND_PUSHDOWN, and idx_cond_push, but I don't understand how these all interact with each other. Thanks -Zardosht

2 6

[Maria-developers] Help with running a transaction inside the server
by Kristian Nielsen 14 Feb '13

14 Feb '13

Hi Serg, Can you give me a tip for doing the following (for MDEV-26, global transaction ID): I need to run a transaction inside the server. This is in the slave I/O thread, when it connects to the master it receives the current replication position as a GTID, and I want to store it in the mysql.rpl_slave_state so that next reconnect of the slave can use the GTID automatically. So I basically need to run the equivalent of something like this: BEGIN; INSERT INTO mysql.rpl_slave_state VALUES (...); DELETE FROM mysql.rpl_slave_state WHERE id=...; COMMIT; I think I already have everything except the "BEGIN" part (if you want, you can check the code in rpl_slave_state::record_gtid() in my branch lp:~maria-captains/maria/10.0-mdev26/ on Launchpad). So how do I do the equivalent of BEGIN inside the server code? Can you think of any existing code I could use as example? - Kristian.

3 7

[Maria-developers] Fwd: debugging how query plans are made
by Zardosht Kasheff 12 Feb '13

12 Feb '13

Is there something similar in MariaDB? ---------- Forwarded message ---------- From: Øystein Grøvlen <oystein.grovlen(a)oracle.com> Date: Tue, Feb 12, 2013 at 9:11 AM Subject: Re: debugging how query plans are made To: Zardosht Kasheff <zardosht(a)gmail.com> Cc: "internals(a)lists.mysql.com" <internals(a)lists.mysql.com> On 02/12/13 02:54 PM, Zardosht Kasheff wrote: > > Hello all, > > In MySQL and/or MariaDB, are there any hooks or methods one can use to > learn why a query plan is chosen over alternatives? We sometimes > notice that a different query plan is chosen for us than InnoDB (which > may be fine, given we are a different storage engine), and we would > like to better understand why. Is there any debug parameter that gets > MySQL to output decisions made in the optimizer? MySQL 5.6 has optimizer tracing which outputs the decisions made by the optimizer. See http://dev.mysql.com/doc/internals/en/optimizer-tracing.html Hope this helps, -- Øystein

2 1

Re: [Maria-developers] Files under /etc/my.cnf.d/ read by default
by Sergei Golubchik 12 Feb '13

12 Feb '13

Hi, Honza! On Feb 05, Honza Horak wrote: > Hi guys, > > while we were packaging MariaDB into Fedora (finally) we used the > upstream's cnf files structure, which means /etc/my.cnf, that includes > all files from directory /etc/my.cnf.d/* using statement !includedir. > > That works fine when user installs (not updates) the MariaDB packages > without having them installed before. The problem is if user has changed > /etc/my.cnf before updating -- then /etc/my.cnf doesn't get updated > (!includedir won't be added) and we end with files under /etc/my.cnf.d/* > but nothing what includes them. That is quite confusing for users that > may change something in /etc/my.cnf.d/client.cnf but nothing happens. Agree. Especially as we've started to use /etc/my.cnf.d/ now - e.g. CassandraSE comes in a separate rpm, that (besides the plugin itself) drops a file into /etc/my.cnf.d/ to enable the engine in the server. I expect that more separately-packed plugins will start doing the same. > Since files under /etc/my.cnf.d/* are empty by default it shouldn't do > any harm if all files corresponding with /etc/my.cnf.d/*.cnf template > get loaded even without !includedir in /etc/my.cnf. Actually I believe > that is what users expect to happen when they see files under > /etc/my.cnf.d.. > > So, that's basically what I'm proposing -- read files under > /etc/my.cnf.d/* by default, without a need to use !includedir. What do > you think? What if we add !includedir to the /etc/my.cnf from the post-inst script? Something like grep -q '!includedir /etc/my.cnf.d' /etc/my.cnf || \ (echo; echo '!includedir /etc/my.cnf.d') >> /etc/my.cnf Would that be enough? Regards, Sergei

2 1

[Maria-developers] No Doxygen?
by Christian Convey 11 Feb '13

11 Feb '13

I was kind of surprised to not find anything in MariaDB's build system for using Doxygen to create HTML docs. Have people decided this isn't a worthwhile thing to do? Or has just no one gotten around to it yet? - Christian

3 4

Re: [Maria-developers] Question
by Michael Widenius 10 Feb '13

10 Feb '13

Hi! First, the MariaDB developers list is the right place to ask questions like this. >>>>> "Aamir" == Aamir Abbas <Abbas> writes: Aamir> Hi Monty, Aamir> I have researched this question about MySQL without luck and if you cannot answer it then nobody can. On some MySQL servers the version number returns a "-log" suffixed to the version number. For example, 5.0.45-log. What does the "-log" mean or signify? This is not a standard version number as far as I can tell. Should it not simply return 5.0.45? I have posted the question to a couple of forums but nobody's able to answer. I have just updated the documentation at https://kb.askmonty.org/en/version/ to answer this questions in detail. Here is a copy of the new documentation: The VERSION() string may have one or more of the following suffixes: Suffix Description -embedded The server is an embedded server (libmysqld). -log General logging, slow logging or binary (replication) logging is enabled. -debug The server is compiled for debugging. -valgrind The server is compiled to be instrumented with valgrind. Regards, Monty

1 0

[Maria-developers] Coverity scan results
by Christian Convey 07 Feb '13

07 Feb '13

Hey guys, For those of you who missed it, I volunteered to try gettin MariaDB scanned as part of the Coverity Scan service. I just got the trunk scanned for the first time, and a lot of potential-problem reports came up. I did a spot check on one of them, and the report looks like a true-positive. Coverity Scan reports: 178 high-impact problems 1020 medium-impact problems 47 low-impact problems. Any suggestions for how to get these looked at by the appropriate developers? It seems crazy for me to enter 1000+ bug reports into the bug tracker. I believe I have the power to create new user accounts in Coverity Scan's MariaDB project, for any MariaDB developer who wants to see the results. Thanks, Christian

3 12

Re: [Maria-developers] Fwd: [Commits] Rev 3622: Fix for bug MDEV-765 (LP:825075) in file:///home/tsk/mprog/src/5.3-md765/
by Sergei Petrunia 07 Feb '13

07 Feb '13

Hi Timour, On Mon, Feb 04, 2013 at 05:39:02PM +0200, Timour Katchaounov wrote: > > Sergey, > > Could you please review the following patch. The fix implements > your suggestion from your previous review: > https://lists.launchpad.net/maria-developers/msg04597.html > Ok to push. > Timour > > ------------------------------------------------------------ > revno: 3622 > revision-id: timour(a)askmonty.org-20130204153548-njv08hcdskv6ttjk > parent: sergii(a)pisem.net-20130128081223-mp9rsd3t9soz8lly > fixes bug: https://mariadb.atlassian.net/browse/MDEV-765 > committer: timour(a)askmonty.org > branch nick: 5.3-md765 > timestamp: Mon 2013-02-04 17:35:48 +0200 > message: > Fix for bug MDEV-765 (LP:825075) > > Analys: > The cause for the wrong result was that the optimizer > incorrectly chose min/max loose scan when it is not > applicable. The applicability test missed the case when > a condition on the MIN/MAX argument was OR-ed with a > condition on some other field. In this case, the MIN/MAX > condition cannot be used for loose scan. > > Solution: > Extend the test check_group_min_max_predicates() to check > that the WHERE clause is of the form: "cond1 AND cond2" > where > cond1 - does not use min_max_column at all. > cond2 - is an AND/OR tree with leaves in form "min_max_column $CMP$ const" > or $CMP$ is one of the functions between, is [not] null > > > === modified file 'mysql-test/r/group_min_max.result' > --- a/mysql-test/r/group_min_max.result 2013-01-28 08:12:23 +0000 > +++ b/mysql-test/r/group_min_max.result 2013-02-04 15:35:48 +0000 > @@ -3186,3 +3186,61 @@ a b > 109 19 > drop table t1; > End of 5.1 tests > +# > +# MDEV-765: LP:825075 - Wrong result with GROUP BY + multipart key + MIN/MAX loose scan > +# > +CREATE TABLE t1 (a varchar(1), b varchar(1), KEY (b,a)); > +INSERT INTO t1 VALUES > +('0',NULL),('9',NULL),('8','c'),('4','d'),('7','d'),(NULL,'f'), > +('7','f'),('8','g'),(NULL,'j'); > +explain > +SELECT max(a) , b FROM t1 WHERE a IS NULL OR b = 'z' GROUP BY b; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index b b 8 NULL 9 Using where; Using index > +SELECT max(a) , b FROM t1 WHERE a IS NULL OR b = 'z' GROUP BY b; > +max(a) b > +NULL f > +NULL j > +explain > +SELECT b, min(a) FROM t1 WHERE a = '7' OR b = 'z' GROUP BY b; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index b b 8 NULL 9 Using where; Using index > +SELECT b, min(a) FROM t1 WHERE a = '7' OR b = 'z' GROUP BY b; > +b min(a) > +d 7 > +f 7 > +explain > +SELECT b, min(a) FROM t1 WHERE (a = b OR b = 'd' OR b is NULL) GROUP BY b; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index b b 8 NULL 9 Using where; Using index > +SELECT b, min(a) FROM t1 WHERE (a = b OR b = 'd' OR b is NULL) GROUP BY b; > +b min(a) > +NULL 0 > +d 4 > +explain > +SELECT b, min(a) FROM t1 WHERE a > ('0' = b) AND b = 'z' GROUP BY b; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ref b b 4 const 1 Using where; Using index > +SELECT b, min(a) FROM t1 WHERE a > ('0' = b) AND b = 'z' GROUP BY b; > +b min(a) > +explain > +SELECT b, min(a) FROM t1 WHERE a > '0' AND (b < (a = '7')) GROUP BY b; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index NULL b 8 NULL 9 Using where; Using index > +SELECT b, min(a) FROM t1 WHERE a > '0' AND (b < (a = '7')) GROUP BY b; > +b min(a) > +d 7 > +f 7 > +explain > +SELECT b, min(a) FROM t1 WHERE (a > '0' AND (a > '1' OR b = 'd')) GROUP BY b; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index b b 8 NULL 9 Using where; Using index > +SELECT b, min(a) FROM t1 WHERE (a > '0' AND (a > '1' OR b = 'd')) GROUP BY b; > +b min(a) > +NULL 9 > +c 8 > +d 4 > +f 7 > +g 8 > +drop table t1; > +End of 5.3 tests > > === modified file 'mysql-test/t/group_min_max.test' > --- a/mysql-test/t/group_min_max.test 2013-01-28 08:12:23 +0000 > +++ b/mysql-test/t/group_min_max.test 2013-02-04 15:35:48 +0000 > @@ -1203,3 +1203,40 @@ WHERE EXISTS ( SELECT DISTINCT b FROM t1 > drop table t1; > > --echo End of 5.1 tests > + > +--echo # > +--echo # MDEV-765: LP:825075 - Wrong result with GROUP BY + multipart key + MIN/MAX loose scan > +--echo # > + > +CREATE TABLE t1 (a varchar(1), b varchar(1), KEY (b,a)); > +INSERT INTO t1 VALUES > +('0',NULL),('9',NULL),('8','c'),('4','d'),('7','d'),(NULL,'f'), > +('7','f'),('8','g'),(NULL,'j'); > + > +explain > +SELECT max(a) , b FROM t1 WHERE a IS NULL OR b = 'z' GROUP BY b; > +SELECT max(a) , b FROM t1 WHERE a IS NULL OR b = 'z' GROUP BY b; > + > +explain > +SELECT b, min(a) FROM t1 WHERE a = '7' OR b = 'z' GROUP BY b; > +SELECT b, min(a) FROM t1 WHERE a = '7' OR b = 'z' GROUP BY b; > + > +explain > +SELECT b, min(a) FROM t1 WHERE (a = b OR b = 'd' OR b is NULL) GROUP BY b; > +SELECT b, min(a) FROM t1 WHERE (a = b OR b = 'd' OR b is NULL) GROUP BY b; > + > +explain > +SELECT b, min(a) FROM t1 WHERE a > ('0' = b) AND b = 'z' GROUP BY b; > +SELECT b, min(a) FROM t1 WHERE a > ('0' = b) AND b = 'z' GROUP BY b; > + > +explain > +SELECT b, min(a) FROM t1 WHERE a > '0' AND (b < (a = '7')) GROUP BY b; > +SELECT b, min(a) FROM t1 WHERE a > '0' AND (b < (a = '7')) GROUP BY b; > + > +explain > +SELECT b, min(a) FROM t1 WHERE (a > '0' AND (a > '1' OR b = 'd')) GROUP BY b; > +SELECT b, min(a) FROM t1 WHERE (a > '0' AND (a > '1' OR b = 'd')) GROUP BY b; > + > +drop table t1; > + > +--echo End of 5.3 tests > > === modified file 'sql/opt_range.cc' > --- a/sql/opt_range.cc 2013-01-28 08:12:23 +0000 > +++ b/sql/opt_range.cc 2013-02-04 15:35:48 +0000 > @@ -11485,7 +11485,8 @@ static bool get_constant_key_infix(KEY * > KEY_PART_INFO **first_non_infix_part); > static bool > check_group_min_max_predicates(Item *cond, Item_field *min_max_arg_item, > - Field::imagetype image_type); > + Field::imagetype image_type, > + bool *has_min_max_fld, bool *has_other_fld); > > static void > cost_group_min_max(TABLE* table, KEY *index_info, uint used_key_parts, > @@ -12005,10 +12006,12 @@ get_best_group_min_max(PARAM *param, SEL > DBUG_RETURN(NULL); > > /* Check (SA3) for the where clause. */ > + bool has_min_max_fld= false, has_other_fld= false; > if (join->conds && min_max_arg_item && > !check_group_min_max_predicates(join->conds, min_max_arg_item, > (index_info->flags & HA_SPATIAL) ? > - Field::itMBR : Field::itRAW)) > + Field::itMBR : Field::itRAW, > + &has_min_max_fld, &has_other_fld)) > DBUG_RETURN(NULL); > > /* The query passes all tests, so construct a new TRP object. */ > @@ -12043,16 +12046,24 @@ get_best_group_min_max(PARAM *param, SEL > > SYNOPSIS > check_group_min_max_predicates() > - cond tree (or subtree) describing all or part of the WHERE > - clause being analyzed > - min_max_arg_item the field referenced by the MIN/MAX function(s) > - min_max_arg_part the keypart of the MIN/MAX argument if any > + cond [in] the expression tree being analyzed > + min_max_arg [in] the field referenced by the MIN/MAX function(s) > + image_type [in] > + has_min_max_arg [out] true if the subtree being analyzed references min_max_arg > + has_other_arg [out] true if the subtree being analyzed references a column > + other min_max_arg > > DESCRIPTION > The function walks recursively over the cond tree representing a WHERE > clause, and checks condition (SA3) - if a field is referenced by a MIN/MAX > aggregate function, it is referenced only by one of the following > - predicates: {=, !=, <, <=, >, >=, between, is null, is not null}. > + predicates $FUNC$: > + {=, !=, <, <=, >, >=, between, is [not] null, multiple equal}. > + In addition the function checks that the WHERE condition is equivalent to > + "cond1 AND cond2" where : > + cond1 - does not use min_max_column at all. > + cond2 - is an AND/OR tree with leaves in form > + "$FUNC$(min_max_column[, const])". > > RETURN > TRUE if cond passes the test > @@ -12061,7 +12072,8 @@ get_best_group_min_max(PARAM *param, SEL > > static bool > check_group_min_max_predicates(Item *cond, Item_field *min_max_arg_item, > - Field::imagetype image_type) > + Field::imagetype image_type, > + bool *has_min_max_arg, bool *has_other_arg) > { > DBUG_ENTER("check_group_min_max_predicates"); > DBUG_ASSERT(cond && min_max_arg_item); > @@ -12073,12 +12085,24 @@ check_group_min_max_predicates(Item *con > DBUG_PRINT("info", ("Analyzing: %s", ((Item_func*) cond)->func_name())); > List_iterator_fast<Item> li(*((Item_cond*) cond)->argument_list()); > Item *and_or_arg; > + Item_func::Functype func_type= ((Item_cond*) cond)->functype(); > + bool has_min_max= false, has_other= false; > while ((and_or_arg= li++)) > { > - if (!check_group_min_max_predicates(and_or_arg, min_max_arg_item, > - image_type)) > + /* > + The WHERE clause doesn't pass the condition if: > + (1) any subtree doesn't pass the condition or > + (2) the subtree passes the test, but it is an OR and it references both > + the min/max argument and other columns. > + */ > + if (!check_group_min_max_predicates(and_or_arg, min_max_arg_item, //1 > + image_type, > + &has_min_max, &has_other) || > + (func_type == Item_func::COND_OR_FUNC && has_min_max && has_other))//2 > DBUG_RETURN(FALSE); > } > + *has_min_max_arg= has_min_max || *has_min_max_arg; > + *has_other_arg= has_other || *has_other_arg; > DBUG_RETURN(TRUE); > } > > @@ -12112,6 +12136,10 @@ check_group_min_max_predicates(Item *con > if (cond_type == Item::FIELD_ITEM) > { > DBUG_PRINT("info", ("Analyzing: %s", cond->full_name())); > + if (min_max_arg_item->eq((Item_field*)cond, 1)) > + *has_min_max_arg= true; > + else > + *has_other_arg= true; > DBUG_RETURN(TRUE); > } > > @@ -12120,9 +12148,33 @@ check_group_min_max_predicates(Item *con > > /* Test if cond references only group-by or non-group fields. */ > Item_func *pred= (Item_func*) cond; > + Item_func::Functype pred_type= pred->functype(); > + DBUG_PRINT("info", ("Analyzing: %s", pred->func_name())); > + if (pred_type == Item_func::MULT_EQUAL_FUNC) > + { > + /* > + Check that each field in a multiple equality is either a constant or > + it is a reference to the min/max argument, or it doesn't contain the > + min/max argument at all. > + */ > + Item_equal_fields_iterator eq_it(*((Item_equal*)pred)); > + Item *eq_item; > + bool has_min_max= false, has_other= false; > + while ((eq_item= eq_it++)) > + { > + if (min_max_arg_item->eq(eq_item->real_item(), 1)) > + has_min_max= true; > + else > + has_other= true; > + } > + *has_min_max_arg= has_min_max || *has_min_max_arg; > + *has_other_arg= has_other || *has_other_arg; > + DBUG_RETURN(!(has_min_max && has_other)); > + } > + > Item **arguments= pred->arguments(); > Item *cur_arg; > - DBUG_PRINT("info", ("Analyzing: %s", pred->func_name())); > + bool has_min_max= false, has_other= false; > for (uint arg_idx= 0; arg_idx < pred->argument_count (); arg_idx++) > { > cur_arg= arguments[arg_idx]->real_item(); > @@ -12131,11 +12183,11 @@ check_group_min_max_predicates(Item *con > { > if (min_max_arg_item->eq(cur_arg, 1)) > { > - /* > - If pred references the MIN/MAX argument, check whether pred is a range > - condition that compares the MIN/MAX argument with a constant. > - */ > - Item_func::Functype pred_type= pred->functype(); > + has_min_max= true; > + /* > + If pred references the MIN/MAX argument, check whether pred is a range > + condition that compares the MIN/MAX argument with a constant. > + */ > if (pred_type != Item_func::EQUAL_FUNC && > pred_type != Item_func::LT_FUNC && > pred_type != Item_func::LE_FUNC && > @@ -12175,11 +12227,13 @@ check_group_min_max_predicates(Item *con > min_max_arg_item->field->cmp_type() != args[1]->result_type()))) > DBUG_RETURN(FALSE); > } > + else > + has_other= true; > } > else if (cur_arg->type() == Item::FUNC_ITEM) > { > - if (!check_group_min_max_predicates(cur_arg, min_max_arg_item, > - image_type)) > + if (!check_group_min_max_predicates(cur_arg, min_max_arg_item, image_type, > + &has_min_max, &has_other)) > DBUG_RETURN(FALSE); > } > else if (cur_arg->const_item()) > @@ -12192,7 +12246,11 @@ check_group_min_max_predicates(Item *con > } > else > DBUG_RETURN(FALSE); > + if(has_min_max && has_other) > + DBUG_RETURN(FALSE); > } > + *has_min_max_arg= has_min_max || *has_min_max_arg; > + *has_other_arg= has_other || *has_other_arg; > > DBUG_RETURN(TRUE); > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Igor please review: EXPLAIN for DELETE
by Sergei Petrunia 06 Feb '13

06 Feb '13

Hello Igor, Hello Igor, Could you please review the following two patches: http://lists.askmonty.org/pipermail/commits/2012-December/003944.html http://lists.askmonty.org/pipermail/commits/2012-December/003945.html I think they implement the ideas about MDEV-3798 we've discussed earlier. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1