- developers - lists.mariadb.org

[Maria-developers] Updated (by Timour): Subquery optimization: Efficient NOT IN execution with NULLs (68)
by worklog-noreply＠askmonty.org 27 Nov '09

27 Nov '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subquery optimization: Efficient NOT IN execution with NULLs CREATION DATE..: Fri, 27 Nov 2009, 13:22 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 68 (http://askmonty.org/worklog/?tid=68) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Timour - Fri, 27 Nov 2009, 13:23)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.17140 2009-11-27 11:23:17.000000000 +0000 +++ /tmp/wklog.68.new.17140 2009-11-27 11:23:17.000000000 +0000 @@ -1 +1,72 @@ +This a copy of the initial algorithm proposed by Igor: +====================================================== +For each left side tuple (v_1,...,v_n) we have to find the following set +of rowids for the temp table containing N rows as the result of +materialization of the subquery: + + R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs +trough all indexes from [1..n] such that v_i is not null. + +Bear in mind the following specifics of this intersection: + (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint + (2) For each i: rowid{a_i is null} is the same for each tuple + +Due to (2) it makes sense to build rowid{a_i is null} only once. +A good representation for such sets would be bitmaps: +- it requires minimum memory: not more than N*n bits in total +- search of an element in a set is extremely cheap + +Taken all above into account I could suggest the following algorithm to +build R: + + Using indexes (read about them below) for each column participating in the + intersection, + merge ordered sets rowid{a_i=v_i} in the following manner. + If a rowid r has been encountered maximum in k sets +rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), + then it has to be checked against all rowid{a_i=v_i} such that i is +not in {i1,...,ik}. + As soon as we fail to find r in one of these sets we discard it. + If r has been found in all of them then r belongs to the set R. + +Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i +is null} is either +belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can +infer that for any r from R +indexes a_i can be uniquely divided into two groups: one contains +indexes a_i where r belongs to +the sets rowid{a_i=v_i}, the other contains indexes a_j such that r +belongs to rowid{a_j is null}. + +Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order +needed for the merge procedure. We could use BTREE indexes for temp +table. But they are rather expensive and +take a lot of memory as the are implemented with RB trees. +I would suggest creating for each column from the temporary table just +an array of rowids sorted by the value from column a. +Index lookup in such an array is cheap. It's also rather cheap to check +that the next rowid refers to a row with a different value in column a. +The array can be created on demand. + +Other consideration that may be taken into account: +1. If columns a_j1,...,a_jm do not contain null values in the temporary +table at all, create for them only one index array (and of course do not +create any bitmaps for them). +2. Consider the ratio d(a_i)=N'/ V(a_i), where N' is the number of rows +where a_i is not null and V(a_i) is the number of distinct values for +a_i excluding nulls. + If d(a_i) is close to 1 then do not create any index array: check +whether there is a match running through the records that have been +filtered in. Anyway if d(a_i) is close to 1 then a intersection with +rowid{a_i=v_i} would not reduce the number of remaining rowids +significantly. + If additionally N-N' is small do not create a bitmap for this column +either. +3. If for a column a_i d(a_i) is not close to 1, but N-N' is small a +sorted array of rowids from the set rowid{a_i is null} can be used +instead of a bitmap. +4. We always have a match if R0= INTERSECT rowid{a_i is null} is not +empty. Here i runs through all indexes from [1..n] such that v_i is not +null. For a given subset of columns this fact has to be checked only +once. It can be easily done with bitmap intersection. DESCRIPTION: The goal of this task is to implement efficient execution of NOT IN subquery predicates of the form: <oe_1,...,oe_n> NOT IN <non_correlated subquery> when either some oe_i, or some subqury result column contains NULLs. The problem with such predicates is that it is possible to use index lookups only when neither argument of the predicate contains NULLs. If some argument contains a NULL, then due to NULL semantics, it plays the role of a wildcard. If we were to use regular index lookups, then we would get 'no match' for some outer tuple (thus the predicate evaluates to FALSE), while the SQL semantics means 'partial match', and the predicate should evaluate to NULL. This task implements an efficient algorithm to compute such 'parial matches', where a NULL matches any value. HIGH-LEVEL SPECIFICATION: This a copy of the initial algorithm proposed by Igor: ====================================================== For each left side tuple (v_1,...,v_n) we have to find the following set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs trough all indexes from [1..n] such that v_i is not null. Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint (2) For each i: rowid{a_i is null} is the same for each tuple Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap Taken all above into account I could suggest the following algorithm to build R: Using indexes (read about them below) for each column participating in the intersection, merge ordered sets rowid{a_i=v_i} in the following manner. If a rowid r has been encountered maximum in k sets rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can infer that for any r from R indexes a_i can be uniquely divided into two groups: one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, the other contains indexes a_j such that r belongs to rowid{a_j is null}. Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order needed for the merge procedure. We could use BTREE indexes for temp table. But they are rather expensive and take a lot of memory as the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all, create for them only one index array (and of course do not create any bitmaps for them). 2. Consider the ratio d(a_i)=N'/ V(a_i), where N' is the number of rows where a_i is not null and V(a_i) is the number of distinct values for a_i excluding nulls. If d(a_i) is close to 1 then do not create any index array: check whether there is a match running through the records that have been filtered in. Anyway if d(a_i) is close to 1 then a intersection with rowid{a_i=v_i} would not reduce the number of remaining rowids significantly. If additionally N-N' is small do not create a bitmap for this column either. 3. If for a column a_i d(a_i) is not close to 1, but N-N' is small a sorted array of rowids from the set rowid{a_i is null} can be used instead of a bitmap. 4. We always have a match if R0= INTERSECT rowid{a_i is null} is not empty. Here i runs through all indexes from [1..n] such that v_i is not null. For a given subset of columns this fact has to be checked only once. It can be easily done with bitmap intersection. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Timour): Subquery optimization: Efficient NOT IN execution with NULLs (68)
by worklog-noreply＠askmonty.org 27 Nov '09

27 Nov '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subquery optimization: Efficient NOT IN execution with NULLs CREATION DATE..: Fri, 27 Nov 2009, 13:22 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 68 (http://askmonty.org/worklog/?tid=68) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Timour - Fri, 27 Nov 2009, 13:23)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.17140 2009-11-27 11:23:17.000000000 +0000 +++ /tmp/wklog.68.new.17140 2009-11-27 11:23:17.000000000 +0000 @@ -1 +1,72 @@ +This a copy of the initial algorithm proposed by Igor: +====================================================== +For each left side tuple (v_1,...,v_n) we have to find the following set +of rowids for the temp table containing N rows as the result of +materialization of the subquery: + + R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs +trough all indexes from [1..n] such that v_i is not null. + +Bear in mind the following specifics of this intersection: + (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint + (2) For each i: rowid{a_i is null} is the same for each tuple + +Due to (2) it makes sense to build rowid{a_i is null} only once. +A good representation for such sets would be bitmaps: +- it requires minimum memory: not more than N*n bits in total +- search of an element in a set is extremely cheap + +Taken all above into account I could suggest the following algorithm to +build R: + + Using indexes (read about them below) for each column participating in the + intersection, + merge ordered sets rowid{a_i=v_i} in the following manner. + If a rowid r has been encountered maximum in k sets +rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), + then it has to be checked against all rowid{a_i=v_i} such that i is +not in {i1,...,ik}. + As soon as we fail to find r in one of these sets we discard it. + If r has been found in all of them then r belongs to the set R. + +Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i +is null} is either +belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can +infer that for any r from R +indexes a_i can be uniquely divided into two groups: one contains +indexes a_i where r belongs to +the sets rowid{a_i=v_i}, the other contains indexes a_j such that r +belongs to rowid{a_j is null}. + +Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order +needed for the merge procedure. We could use BTREE indexes for temp +table. But they are rather expensive and +take a lot of memory as the are implemented with RB trees. +I would suggest creating for each column from the temporary table just +an array of rowids sorted by the value from column a. +Index lookup in such an array is cheap. It's also rather cheap to check +that the next rowid refers to a row with a different value in column a. +The array can be created on demand. + +Other consideration that may be taken into account: +1. If columns a_j1,...,a_jm do not contain null values in the temporary +table at all, create for them only one index array (and of course do not +create any bitmaps for them). +2. Consider the ratio d(a_i)=N'/ V(a_i), where N' is the number of rows +where a_i is not null and V(a_i) is the number of distinct values for +a_i excluding nulls. + If d(a_i) is close to 1 then do not create any index array: check +whether there is a match running through the records that have been +filtered in. Anyway if d(a_i) is close to 1 then a intersection with +rowid{a_i=v_i} would not reduce the number of remaining rowids +significantly. + If additionally N-N' is small do not create a bitmap for this column +either. +3. If for a column a_i d(a_i) is not close to 1, but N-N' is small a +sorted array of rowids from the set rowid{a_i is null} can be used +instead of a bitmap. +4. We always have a match if R0= INTERSECT rowid{a_i is null} is not +empty. Here i runs through all indexes from [1..n] such that v_i is not +null. For a given subset of columns this fact has to be checked only +once. It can be easily done with bitmap intersection. DESCRIPTION: The goal of this task is to implement efficient execution of NOT IN subquery predicates of the form: <oe_1,...,oe_n> NOT IN <non_correlated subquery> when either some oe_i, or some subqury result column contains NULLs. The problem with such predicates is that it is possible to use index lookups only when neither argument of the predicate contains NULLs. If some argument contains a NULL, then due to NULL semantics, it plays the role of a wildcard. If we were to use regular index lookups, then we would get 'no match' for some outer tuple (thus the predicate evaluates to FALSE), while the SQL semantics means 'partial match', and the predicate should evaluate to NULL. This task implements an efficient algorithm to compute such 'parial matches', where a NULL matches any value. HIGH-LEVEL SPECIFICATION: This a copy of the initial algorithm proposed by Igor: ====================================================== For each left side tuple (v_1,...,v_n) we have to find the following set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs trough all indexes from [1..n] such that v_i is not null. Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint (2) For each i: rowid{a_i is null} is the same for each tuple Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap Taken all above into account I could suggest the following algorithm to build R: Using indexes (read about them below) for each column participating in the intersection, merge ordered sets rowid{a_i=v_i} in the following manner. If a rowid r has been encountered maximum in k sets rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can infer that for any r from R indexes a_i can be uniquely divided into two groups: one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, the other contains indexes a_j such that r belongs to rowid{a_j is null}. Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order needed for the merge procedure. We could use BTREE indexes for temp table. But they are rather expensive and take a lot of memory as the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all, create for them only one index array (and of course do not create any bitmaps for them). 2. Consider the ratio d(a_i)=N'/ V(a_i), where N' is the number of rows where a_i is not null and V(a_i) is the number of distinct values for a_i excluding nulls. If d(a_i) is close to 1 then do not create any index array: check whether there is a match running through the records that have been filtered in. Anyway if d(a_i) is close to 1 then a intersection with rowid{a_i=v_i} would not reduce the number of remaining rowids significantly. If additionally N-N' is small do not create a bitmap for this column either. 3. If for a column a_i d(a_i) is not close to 1, but N-N' is small a sorted array of rowids from the set rowid{a_i is null} can be used instead of a bitmap. 4. We always have a match if R0= INTERSECT rowid{a_i is null} is not empty. Here i runs through all indexes from [1..n] such that v_i is not null. For a given subset of columns this fact has to be checked only once. It can be easily done with bitmap intersection. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Timour): Subquery optimization: Efficient NOT IN execution with NULLs (68)
by worklog-noreply＠askmonty.org 27 Nov '09

27 Nov '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subquery optimization: Efficient NOT IN execution with NULLs CREATION DATE..: Fri, 27 Nov 2009, 13:22 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 68 (http://askmonty.org/worklog/?tid=68) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: The goal of this task is to implement efficient execution of NOT IN subquery predicates of the form: <oe_1,...,oe_n> NOT IN <non_correlated subquery> when either some oe_i, or some subqury result column contains NULLs. The problem with such predicates is that it is possible to use index lookups only when neither argument of the predicate contains NULLs. If some argument contains a NULL, then due to NULL semantics, it plays the role of a wildcard. If we were to use regular index lookups, then we would get 'no match' for some outer tuple (thus the predicate evaluates to FALSE), while the SQL semantics means 'partial match', and the predicate should evaluate to NULL. This task implements an efficient algorithm to compute such 'parial matches', where a NULL matches any value. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Timour): Subquery optimization: Efficient NOT IN execution with NULLs (68)
by worklog-noreply＠askmonty.org 27 Nov '09

27 Nov '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subquery optimization: Efficient NOT IN execution with NULLs CREATION DATE..: Fri, 27 Nov 2009, 13:22 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 68 (http://askmonty.org/worklog/?tid=68) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: The goal of this task is to implement efficient execution of NOT IN subquery predicates of the form: <oe_1,...,oe_n> NOT IN <non_correlated subquery> when either some oe_i, or some subqury result column contains NULLs. The problem with such predicates is that it is possible to use index lookups only when neither argument of the predicate contains NULLs. If some argument contains a NULL, then due to NULL semantics, it plays the role of a wildcard. If we were to use regular index lookups, then we would get 'no match' for some outer tuple (thus the predicate evaluates to FALSE), while the SQL semantics means 'partial match', and the predicate should evaluate to NULL. This task implements an efficient algorithm to compute such 'parial matches', where a NULL matches any value. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Monty): Provide key cache statistics (58)
by worklog-noreply＠askmonty.org 27 Nov '09

27 Nov '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Provide key cache statistics CREATION DATE..: Thu, 22 Oct 2009, 12:28 SUPERVISOR.....: Bothorsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 58 (http://askmonty.org/worklog/?tid=58) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Monty - Fri, 27 Nov 2009, 12:36)=-=- High Level Description modified. --- /tmp/wklog.58.old.15108 2009-11-27 10:36:13.000000000 +0000 +++ /tmp/wklog.58.new.15108 2009-11-27 10:36:13.000000000 +0000 @@ -1,8 +1,7 @@ -Provide the key cache -statistics available for the default cache (key blocks used, -unflushed, etc.) for all named caches. +Provide the same key cache statistics for all MyISAM key caches as we provide +for the default key cache. -Monty answers in an email: +Background: Monty answers in an email to customer: This is a much simpler task and something that is important to get done. We would have to introduce a 'show keycache statistics' command @@ -11,3 +10,44 @@ The raw coding is probably 1-2 days, but on top if this we need testing, a test environment and building of biniaries for you. Henrik/Bo and Igor will come back to you with a more exact estimate. + +----------------- +Currently we have: + +show status like "key%"; ++------------------------+-------+ +| Variable_name | Value | ++------------------------+-------+ +| Key_blocks_not_flushed | 0 | +| Key_blocks_unused | 13389 | +| Key_blocks_used | 7 | +| Key_read_requests | 22 | +| Key_reads | 7 | +| Key_write_requests | 0 | +| Key_writes | 0 | ++------------------------+-------+ + +Sergei suggested we introduce for each new key cache a set of variables prefixed +with the keycache name: + +show status like "keycache1.%"; +Which should show: + ++----------------------------------+-------+ +| Variable_name | Value | ++----------------------------------+-------+ +| keycache1.Key_blocks_not_flushed | 0 | +| keycache1.Key_blocks_unused | 13389 | +| keycache1.Key_blocks_used | 7 | +| keycache1.Key_read_requests | 22 | +| keycache1.Key_reads | 7 | +| keycache1.Key_write_requests | 0 | +| keycache1.Key_writes | 0 | ++----------------------------------+-------+ + +The task would thus to be to automatically introduce new variables when we +create a new key cache and automatically remove these variables when the cache +is deleted. + +The other option would be to add an information schema where we populate the +information schema with data from all existing key caches. DESCRIPTION: Provide the same key cache statistics for all MyISAM key caches as we provide for the default key cache. Background: Monty answers in an email to customer: This is a much simpler task and something that is important to get done. We would have to introduce a 'show keycache statistics' command that would iterate over all keycaches and provide the statistics. The raw coding is probably 1-2 days, but on top if this we need testing, a test environment and building of biniaries for you. Henrik/Bo and Igor will come back to you with a more exact estimate. ----------------- Currently we have: show status like "key%"; +------------------------+-------+ | Variable_name | Value | +------------------------+-------+ | Key_blocks_not_flushed | 0 | | Key_blocks_unused | 13389 | | Key_blocks_used | 7 | | Key_read_requests | 22 | | Key_reads | 7 | | Key_write_requests | 0 | | Key_writes | 0 | +------------------------+-------+ Sergei suggested we introduce for each new key cache a set of variables prefixed with the keycache name: show status like "keycache1.%"; Which should show: +----------------------------------+-------+ | Variable_name | Value | +----------------------------------+-------+ | keycache1.Key_blocks_not_flushed | 0 | | keycache1.Key_blocks_unused | 13389 | | keycache1.Key_blocks_used | 7 | | keycache1.Key_read_requests | 22 | | keycache1.Key_reads | 7 | | keycache1.Key_write_requests | 0 | | keycache1.Key_writes | 0 | +----------------------------------+-------+ The task would thus to be to automatically introduce new variables when we create a new key cache and automatically remove these variables when the cache is deleted. The other option would be to add an information schema where we populate the information schema with data from all existing key caches. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Test failures in PBXT after merging with MySQL-5.1.41
by Kristian Nielsen 27 Nov '09

27 Nov '09

Hi Paul, Vladimir, We are working on merging latest MySQL (5.1.41) into MariaDB. After the merge, I see some test failures in the PBXT test suite. See here: https://askmonty.org/buildbot/builders/hardy-amd64-fulltest/builds/118/step… Or if you prefer, see here for a search for these failures in the build history database: http://askmonty.org/buildbot/reports/cross_reference#branch=5.1-merge&revis… >From a quick look, it seems to me most/all of these are just the need to adjust result files for changes/bugfixes in MySQL. But before spending time on fixing, I though it was better to coordinate with you, to not duplicate effort. So perhaps you already did the similar changes for the PBXT testsuite against MySQL 5.1.41? Is there maybe a newer version of PBXT that we should merge? Or alternatively, do you want me to fix the testsuite, and then send you a patch for merging into PBXT to update to 5.1.41? - Kristian.

3 11

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2769)
by Michael Widenius 26 Nov '09

26 Nov '09

#At lp:maria based on revid:monty@askmonty.org-20091126155705-5g7fmkbcpdu6ad6j 2769 Michael Widenius 2009-11-26 Added protection around usage of thd->mysys_var (May be changed to 0 by scheduler) modified: sql/mysqld.cc sql/scheduler.cc sql/sql_class.cc sql/sql_class.h sql/sql_insert.cc sql/sql_show.cc === modified file 'sql/mysqld.cc' --- a/sql/mysqld.cc 2009-10-31 19:22:50 +0000 +++ b/sql/mysqld.cc 2009-11-26 20:19:33 +0000 @@ -957,6 +957,7 @@ static void close_connections(void) tmp->killed= THD::KILL_CONNECTION; thread_scheduler.post_kill_notification(tmp); + pthread_mutex_lock(&tmp->LOCK_thd_data); if (tmp->mysys_var) { tmp->mysys_var->abort=1; @@ -979,6 +980,7 @@ static void close_connections(void) } pthread_mutex_unlock(&tmp->mysys_var->mutex); } + pthread_mutex_unlock(&tmp->LOCK_thd_data); } (void) pthread_mutex_unlock(&LOCK_thread_count); // For unlink from list === modified file 'sql/scheduler.cc' --- a/sql/scheduler.cc 2009-09-07 20:50:10 +0000 +++ b/sql/scheduler.cc 2009-11-26 20:19:33 +0000 @@ -235,9 +235,7 @@ void thd_scheduler::thread_detach() if (thread_attached) { THD* thd = (THD*)list.data; - pthread_mutex_lock(&thd->LOCK_thd_data); - thd->mysys_var= NULL; - pthread_mutex_unlock(&thd->LOCK_thd_data); + thd->reset_globals(); thread_attached= FALSE; #ifndef DBUG_OFF /* === modified file 'sql/sql_class.cc' --- a/sql/sql_class.cc 2009-09-15 10:46:35 +0000 +++ b/sql/sql_class.cc 2009-11-26 20:19:33 +0000 @@ -1166,6 +1166,19 @@ bool THD::store_globals() } +/** + Untie THD from current thread + + Used when using --thread-handling=pool-of-threads +*/ + +void THD::reset_globals() +{ + pthread_mutex_lock(&LOCK_thd_data); + mysys_var= 0; + pthread_mutex_unlock(&LOCK_thd_data); +} + /* Cleanup after query. === modified file 'sql/sql_class.h' --- a/sql/sql_class.h 2009-09-15 10:46:35 +0000 +++ b/sql/sql_class.h 2009-11-26 20:19:33 +0000 @@ -1906,6 +1906,7 @@ public: void cleanup(void); void cleanup_after_query(); bool store_globals(); + void reset_globals(); #ifdef SIGNAL_WITH_VIO_CLOSE inline void set_active_vio(Vio* vio) { === modified file 'sql/sql_insert.cc' --- a/sql/sql_insert.cc 2009-11-07 10:34:19 +0000 +++ b/sql/sql_insert.cc 2009-11-26 20:19:33 +0000 @@ -2279,6 +2279,7 @@ void kill_delayed_threads(void) while ((di= it++)) { di->thd.killed= THD::KILL_CONNECTION; + pthread_mutex_lock(&di->thd.LOCK_thd_data); if (di->thd.mysys_var) { pthread_mutex_lock(&di->thd.mysys_var->mutex); @@ -2297,6 +2298,7 @@ void kill_delayed_threads(void) } pthread_mutex_unlock(&di->thd.mysys_var->mutex); } + pthread_mutex_unlock(&di->thd.LOCK_thd_data); } VOID(pthread_mutex_unlock(&LOCK_delayed_insert)); // For unlink from list } === modified file 'sql/sql_show.cc' --- a/sql/sql_show.cc 2009-11-02 09:30:21 +0000 +++ b/sql/sql_show.cc 2009-11-26 20:19:33 +0000 @@ -1747,6 +1747,7 @@ void mysqld_list_processes(THD *thd,cons if ((thd_info->db=tmp->db)) // Safe test thd_info->db=thd->strdup(thd_info->db); thd_info->command=(int) tmp->command; + pthread_mutex_lock(&tmp->LOCK_thd_data); if ((mysys_var= tmp->mysys_var)) pthread_mutex_lock(&mysys_var->mutex); thd_info->proc_info= (char*) (tmp->killed == THD::KILL_CONNECTION? "Killed" : 0); @@ -1766,6 +1767,7 @@ void mysqld_list_processes(THD *thd,cons #endif if (mysys_var) pthread_mutex_unlock(&mysys_var->mutex); + pthread_mutex_unlock(&tmp->LOCK_thd_data); thd_info->start_time= tmp->start_time; thd_info->query=0;

1 0

[Maria-developers] Updated (by Psergey): MRR backport (67)
by worklog-noreply＠askmonty.org 26 Nov '09

26 Nov '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: MRR backport CREATION DATE..: Thu, 26 Nov 2009, 15:19 SUPERVISOR.....: Monty IMPLEMENTOR....: Psergey COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 67 (http://askmonty.org/worklog/?tid=67) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Thu, 26 Nov 2009, 20:21)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.9329 2009-11-26 20:21:28.000000000 +0200 +++ /tmp/wklog.67.new.9329 2009-11-26 20:21:28.000000000 +0200 @@ -65,17 +65,19 @@ 2.5 Make MRR code more of a module ---------------------------------- -Some code in handler.cc can be moved to separate file. -But changes in opt_range.cc can't. -TODO: Sort out how much we really can do here. Initial guess is not much as the -code consists of: +It is not possible to make MRR to be a totally separate module, as its code +consists of : - Default MRR implementation in handler.cc - Changes in opt_range.cc to use MRR instead of multiple records_in_range() - calls. These rely on opt_range.cc's internal structures like SEL_ARG trees and + calls. These rely on opt_range.cc's internal stuctures like SEL_ARG trees and so there is not much point in moving them out. -- DS-MRR implementations which are spread over storage engines. -and the only modularization we see is to move #1 into a separate file which -won't achieve much. +- DS-MRR impelementations which are spread over storage engines. + +We'll try to modularize what we can: +- Move out default MRR implementation from handler.cc +- Move possible parts out of opt_range.cc into a separate file. + + 2.6 Improve the cost model -------------------------- -=-=(Psergey - Thu, 26 Nov 2009, 19:06)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.6449 2009-11-26 19:06:04.000000000 +0200 +++ /tmp/wklog.67.new.6449 2009-11-26 19:06:04.000000000 +0200 @@ -1,4 +1,3 @@ - <contents> 1. Requirements 2. Required actions @@ -44,6 +43,7 @@ http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=index_condi… http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=mrr +http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=icp 2.2 Backport DS-MRR code to MariaDB 5.2 --------------------------------------- -=-=(Psergey - Thu, 26 Nov 2009, 18:15)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.4161 2009-11-26 18:15:36.000000000 +0200 +++ /tmp/wklog.67.new.4161 2009-11-26 18:15:36.000000000 +0200 @@ -1,3 +1,17 @@ + +<contents> +1. Requirements +2. Required actions +2.1 Fix DS-MRR/InnoDB bugs +2.2 Backport DS-MRR code to MariaDB 5.2 +2.3 Introduce control variables +2.4 Other backport issues +2.5 Make MRR code more of a module +2.6 Improve the cost model +2.7 Let DS-MRR support clustered primary keys +</contents> + + 1. Requirements =============== @@ -63,4 +77,28 @@ and the only modularization we see is to move #1 into a separate file which won't achieve much. +2.6 Improve the cost model +-------------------------- +At the moment DS-MRR cost formula re-uses non-MRR scan costs, which uses +records_in_range() calls, followed by index_only_read_time() or read_time() +calls to produce the estimate for read cost. + + We should change this (TODO sort out how exactly) + +Note: this means that the query plans will change from MariaDB 5.2. + +2.7 Let DS-MRR support clustered primary keys +--------------------------------------------- +At the moment DS-MRR is not supported for clustered primary keys. It is not +needed when MRR is used for range access, because range access is done over +an ordered list of ranges, but it is useful for BKA. + +TODO: + it's useful for BKA because BKA makes MRR scans over un-orderered + non-disjoint lists of ranges. Then we can sort these and do ordered scans. + There is still no use for DS-MRR over clustered primary key for range + access, where the ranges are disjoint and ordered. + How about postponing this item until BKA is backported? + + -=-=(Guest - Thu, 26 Nov 2009, 16:52)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.694 2009-11-26 14:52:53.000000000 +0000 +++ /tmp/wklog.67.new.694 2009-11-26 14:52:53.000000000 +0000 @@ -1 +1,66 @@ +1. Requirements +=============== + +We need the following: + +1. Latest MRR interface support, including extensions to support ICP when + using BKA. +2. Let DS-MRR support clustered primary keys (needed when using BKA). +3. Remove conditions used for key access from the condition pushed to index + (ATM this manifests itself as "Using index condition" appearing where there + was no "Using where". TODO: example of this?) +4. Introduce a separate @@optimizer_switch flag for turning on/out ICP (atm it + is switched on/off by @@engine_condition_pushdown) +5. Introduce a separate @@mrr_buffer_size variable to control MRR buffer size + for range+MRR scans. ATM it is controlled by @@read_rnd_size flag and that + makes it unobvious for a number of users. +6. Rename multi_range_read_info_const() to look like it is not a part of MRR + interface. +8. Try to make MRR to be more of a module +7. Improve MRR's cost model. + +2. Required actions +=================== + +Roughly in the order in which it will be done: + +2.1 Fix DS-MRR/InnoDB bugs +-------------------------- +We need to fix the bugs listed here: + +http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=index_condition_pushdown +http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=mrr + +2.2 Backport DS-MRR code to MariaDB 5.2 +--------------------------------------- +The easiest way seems to be to to manually move the needed code from mysql-6.0 +(or whatever it's called now) to MariaDB. + +2.3 Introduce control variables +------------------------------- +Act on items #4 and #5 from the requirements. Should be easy as +@@optimizer_switch is supported in 5.1 codebase. + +2.4 Other backport issues +------------------------- +* Figure out what to do with NDB/MRR. 5.1 codebase has "old" NDB/MRR + implementation. mysql-6.0 (and NDB's branch) have the updated NDB/MRR + but merging it into 5.1 can be very labor-intensive. + Will it be ok to disable NDB/MRR altogether? + + +2.5 Make MRR code more of a module +---------------------------------- +Some code in handler.cc can be moved to separate file. +But changes in opt_range.cc can't. +TODO: Sort out how much we really can do here. Initial guess is not much as the +code consists of: +- Default MRR implementation in handler.cc +- Changes in opt_range.cc to use MRR instead of multiple records_in_range() + calls. These rely on opt_range.cc's internal structures like SEL_ARG trees and + so there is not much point in moving them out. +- DS-MRR implementations which are spread over storage engines. +and the only modularization we see is to move #1 into a separate file which +won't achieve much. + DESCRIPTION: Backport DS-MRR into MariaDB-5.2 codebase, also adding certain extra features to make it more usable. HIGH-LEVEL SPECIFICATION: <contents> 1. Requirements 2. Required actions 2.1 Fix DS-MRR/InnoDB bugs 2.2 Backport DS-MRR code to MariaDB 5.2 2.3 Introduce control variables 2.4 Other backport issues 2.5 Make MRR code more of a module 2.6 Improve the cost model 2.7 Let DS-MRR support clustered primary keys </contents> 1. Requirements =============== We need the following: 1. Latest MRR interface support, including extensions to support ICP when using BKA. 2. Let DS-MRR support clustered primary keys (needed when using BKA). 3. Remove conditions used for key access from the condition pushed to index (ATM this manifests itself as "Using index condition" appearing where there was no "Using where". TODO: example of this?) 4. Introduce a separate @@optimizer_switch flag for turning on/out ICP (atm it is switched on/off by @@engine_condition_pushdown) 5. Introduce a separate @@mrr_buffer_size variable to control MRR buffer size for range+MRR scans. ATM it is controlled by @@read_rnd_size flag and that makes it unobvious for a number of users. 6. Rename multi_range_read_info_const() to look like it is not a part of MRR interface. 8. Try to make MRR to be more of a module 7. Improve MRR's cost model. 2. Required actions =================== Roughly in the order in which it will be done: 2.1 Fix DS-MRR/InnoDB bugs -------------------------- We need to fix the bugs listed here: http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=index_condi… http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=mrr http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=icp 2.2 Backport DS-MRR code to MariaDB 5.2 --------------------------------------- The easiest way seems to be to to manually move the needed code from mysql-6.0 (or whatever it's called now) to MariaDB. 2.3 Introduce control variables ------------------------------- Act on items #4 and #5 from the requirements. Should be easy as @@optimizer_switch is supported in 5.1 codebase. 2.4 Other backport issues ------------------------- * Figure out what to do with NDB/MRR. 5.1 codebase has "old" NDB/MRR implementation. mysql-6.0 (and NDB's branch) have the updated NDB/MRR but merging it into 5.1 can be very labor-intensive. Will it be ok to disable NDB/MRR altogether? 2.5 Make MRR code more of a module ---------------------------------- It is not possible to make MRR to be a totally separate module, as its code consists of : - Default MRR implementation in handler.cc - Changes in opt_range.cc to use MRR instead of multiple records_in_range() calls. These rely on opt_range.cc's internal stuctures like SEL_ARG trees and so there is not much point in moving them out. - DS-MRR impelementations which are spread over storage engines. We'll try to modularize what we can: - Move out default MRR implementation from handler.cc - Move possible parts out of opt_range.cc into a separate file. 2.6 Improve the cost model -------------------------- At the moment DS-MRR cost formula re-uses non-MRR scan costs, which uses records_in_range() calls, followed by index_only_read_time() or read_time() calls to produce the estimate for read cost. We should change this (TODO sort out how exactly) Note: this means that the query plans will change from MariaDB 5.2. 2.7 Let DS-MRR support clustered primary keys --------------------------------------------- At the moment DS-MRR is not supported for clustered primary keys. It is not needed when MRR is used for range access, because range access is done over an ordered list of ranges, but it is useful for BKA. TODO: it's useful for BKA because BKA makes MRR scans over un-orderered non-disjoint lists of ranges. Then we can sort these and do ordered scans. There is still no use for DS-MRR over clustered primary key for range access, where the ranges are disjoint and ordered. How about postponing this item until BKA is backported? ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Psergey): MRR backport (67)
by worklog-noreply＠askmonty.org 26 Nov '09

26 Nov '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: MRR backport CREATION DATE..: Thu, 26 Nov 2009, 15:19 SUPERVISOR.....: Monty IMPLEMENTOR....: Psergey COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 67 (http://askmonty.org/worklog/?tid=67) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Thu, 26 Nov 2009, 20:21)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.9329 2009-11-26 20:21:28.000000000 +0200 +++ /tmp/wklog.67.new.9329 2009-11-26 20:21:28.000000000 +0200 @@ -65,17 +65,19 @@ 2.5 Make MRR code more of a module ---------------------------------- -Some code in handler.cc can be moved to separate file. -But changes in opt_range.cc can't. -TODO: Sort out how much we really can do here. Initial guess is not much as the -code consists of: +It is not possible to make MRR to be a totally separate module, as its code +consists of : - Default MRR implementation in handler.cc - Changes in opt_range.cc to use MRR instead of multiple records_in_range() - calls. These rely on opt_range.cc's internal structures like SEL_ARG trees and + calls. These rely on opt_range.cc's internal stuctures like SEL_ARG trees and so there is not much point in moving them out. -- DS-MRR implementations which are spread over storage engines. -and the only modularization we see is to move #1 into a separate file which -won't achieve much. +- DS-MRR impelementations which are spread over storage engines. + +We'll try to modularize what we can: +- Move out default MRR implementation from handler.cc +- Move possible parts out of opt_range.cc into a separate file. + + 2.6 Improve the cost model -------------------------- -=-=(Psergey - Thu, 26 Nov 2009, 19:06)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.6449 2009-11-26 19:06:04.000000000 +0200 +++ /tmp/wklog.67.new.6449 2009-11-26 19:06:04.000000000 +0200 @@ -1,4 +1,3 @@ - <contents> 1. Requirements 2. Required actions @@ -44,6 +43,7 @@ http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=index_condi… http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=mrr +http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=icp 2.2 Backport DS-MRR code to MariaDB 5.2 --------------------------------------- -=-=(Psergey - Thu, 26 Nov 2009, 18:15)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.4161 2009-11-26 18:15:36.000000000 +0200 +++ /tmp/wklog.67.new.4161 2009-11-26 18:15:36.000000000 +0200 @@ -1,3 +1,17 @@ + +<contents> +1. Requirements +2. Required actions +2.1 Fix DS-MRR/InnoDB bugs +2.2 Backport DS-MRR code to MariaDB 5.2 +2.3 Introduce control variables +2.4 Other backport issues +2.5 Make MRR code more of a module +2.6 Improve the cost model +2.7 Let DS-MRR support clustered primary keys +</contents> + + 1. Requirements =============== @@ -63,4 +77,28 @@ and the only modularization we see is to move #1 into a separate file which won't achieve much. +2.6 Improve the cost model +-------------------------- +At the moment DS-MRR cost formula re-uses non-MRR scan costs, which uses +records_in_range() calls, followed by index_only_read_time() or read_time() +calls to produce the estimate for read cost. + + We should change this (TODO sort out how exactly) + +Note: this means that the query plans will change from MariaDB 5.2. + +2.7 Let DS-MRR support clustered primary keys +--------------------------------------------- +At the moment DS-MRR is not supported for clustered primary keys. It is not +needed when MRR is used for range access, because range access is done over +an ordered list of ranges, but it is useful for BKA. + +TODO: + it's useful for BKA because BKA makes MRR scans over un-orderered + non-disjoint lists of ranges. Then we can sort these and do ordered scans. + There is still no use for DS-MRR over clustered primary key for range + access, where the ranges are disjoint and ordered. + How about postponing this item until BKA is backported? + + -=-=(Guest - Thu, 26 Nov 2009, 16:52)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.694 2009-11-26 14:52:53.000000000 +0000 +++ /tmp/wklog.67.new.694 2009-11-26 14:52:53.000000000 +0000 @@ -1 +1,66 @@ +1. Requirements +=============== + +We need the following: + +1. Latest MRR interface support, including extensions to support ICP when + using BKA. +2. Let DS-MRR support clustered primary keys (needed when using BKA). +3. Remove conditions used for key access from the condition pushed to index + (ATM this manifests itself as "Using index condition" appearing where there + was no "Using where". TODO: example of this?) +4. Introduce a separate @@optimizer_switch flag for turning on/out ICP (atm it + is switched on/off by @@engine_condition_pushdown) +5. Introduce a separate @@mrr_buffer_size variable to control MRR buffer size + for range+MRR scans. ATM it is controlled by @@read_rnd_size flag and that + makes it unobvious for a number of users. +6. Rename multi_range_read_info_const() to look like it is not a part of MRR + interface. +8. Try to make MRR to be more of a module +7. Improve MRR's cost model. + +2. Required actions +=================== + +Roughly in the order in which it will be done: + +2.1 Fix DS-MRR/InnoDB bugs +-------------------------- +We need to fix the bugs listed here: + +http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=index_condition_pushdown +http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=mrr + +2.2 Backport DS-MRR code to MariaDB 5.2 +--------------------------------------- +The easiest way seems to be to to manually move the needed code from mysql-6.0 +(or whatever it's called now) to MariaDB. + +2.3 Introduce control variables +------------------------------- +Act on items #4 and #5 from the requirements. Should be easy as +@@optimizer_switch is supported in 5.1 codebase. + +2.4 Other backport issues +------------------------- +* Figure out what to do with NDB/MRR. 5.1 codebase has "old" NDB/MRR + implementation. mysql-6.0 (and NDB's branch) have the updated NDB/MRR + but merging it into 5.1 can be very labor-intensive. + Will it be ok to disable NDB/MRR altogether? + + +2.5 Make MRR code more of a module +---------------------------------- +Some code in handler.cc can be moved to separate file. +But changes in opt_range.cc can't. +TODO: Sort out how much we really can do here. Initial guess is not much as the +code consists of: +- Default MRR implementation in handler.cc +- Changes in opt_range.cc to use MRR instead of multiple records_in_range() + calls. These rely on opt_range.cc's internal structures like SEL_ARG trees and + so there is not much point in moving them out. +- DS-MRR implementations which are spread over storage engines. +and the only modularization we see is to move #1 into a separate file which +won't achieve much. + DESCRIPTION: Backport DS-MRR into MariaDB-5.2 codebase, also adding certain extra features to make it more usable. HIGH-LEVEL SPECIFICATION: <contents> 1. Requirements 2. Required actions 2.1 Fix DS-MRR/InnoDB bugs 2.2 Backport DS-MRR code to MariaDB 5.2 2.3 Introduce control variables 2.4 Other backport issues 2.5 Make MRR code more of a module 2.6 Improve the cost model 2.7 Let DS-MRR support clustered primary keys </contents> 1. Requirements =============== We need the following: 1. Latest MRR interface support, including extensions to support ICP when using BKA. 2. Let DS-MRR support clustered primary keys (needed when using BKA). 3. Remove conditions used for key access from the condition pushed to index (ATM this manifests itself as "Using index condition" appearing where there was no "Using where". TODO: example of this?) 4. Introduce a separate @@optimizer_switch flag for turning on/out ICP (atm it is switched on/off by @@engine_condition_pushdown) 5. Introduce a separate @@mrr_buffer_size variable to control MRR buffer size for range+MRR scans. ATM it is controlled by @@read_rnd_size flag and that makes it unobvious for a number of users. 6. Rename multi_range_read_info_const() to look like it is not a part of MRR interface. 8. Try to make MRR to be more of a module 7. Improve MRR's cost model. 2. Required actions =================== Roughly in the order in which it will be done: 2.1 Fix DS-MRR/InnoDB bugs -------------------------- We need to fix the bugs listed here: http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=index_condi… http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=mrr http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=icp 2.2 Backport DS-MRR code to MariaDB 5.2 --------------------------------------- The easiest way seems to be to to manually move the needed code from mysql-6.0 (or whatever it's called now) to MariaDB. 2.3 Introduce control variables ------------------------------- Act on items #4 and #5 from the requirements. Should be easy as @@optimizer_switch is supported in 5.1 codebase. 2.4 Other backport issues ------------------------- * Figure out what to do with NDB/MRR. 5.1 codebase has "old" NDB/MRR implementation. mysql-6.0 (and NDB's branch) have the updated NDB/MRR but merging it into 5.1 can be very labor-intensive. Will it be ok to disable NDB/MRR altogether? 2.5 Make MRR code more of a module ---------------------------------- It is not possible to make MRR to be a totally separate module, as its code consists of : - Default MRR implementation in handler.cc - Changes in opt_range.cc to use MRR instead of multiple records_in_range() calls. These rely on opt_range.cc's internal stuctures like SEL_ARG trees and so there is not much point in moving them out. - DS-MRR impelementations which are spread over storage engines. We'll try to modularize what we can: - Move out default MRR implementation from handler.cc - Move possible parts out of opt_range.cc into a separate file. 2.6 Improve the cost model -------------------------- At the moment DS-MRR cost formula re-uses non-MRR scan costs, which uses records_in_range() calls, followed by index_only_read_time() or read_time() calls to produce the estimate for read cost. We should change this (TODO sort out how exactly) Note: this means that the query plans will change from MariaDB 5.2. 2.7 Let DS-MRR support clustered primary keys --------------------------------------------- At the moment DS-MRR is not supported for clustered primary keys. It is not needed when MRR is used for range access, because range access is done over an ordered list of ranges, but it is useful for BKA. TODO: it's useful for BKA because BKA makes MRR scans over un-orderered non-disjoint lists of ranges. Then we can sort these and do ordered scans. There is still no use for DS-MRR over clustered primary key for range access, where the ranges are disjoint and ordered. How about postponing this item until BKA is backported? ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Psergey): MRR backport (67)
by worklog-noreply＠askmonty.org 26 Nov '09

26 Nov '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: MRR backport CREATION DATE..: Thu, 26 Nov 2009, 15:19 SUPERVISOR.....: Monty IMPLEMENTOR....: Psergey COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 67 (http://askmonty.org/worklog/?tid=67) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Thu, 26 Nov 2009, 19:06)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.6449 2009-11-26 19:06:04.000000000 +0200 +++ /tmp/wklog.67.new.6449 2009-11-26 19:06:04.000000000 +0200 @@ -1,4 +1,3 @@ - <contents> 1. Requirements 2. Required actions @@ -44,6 +43,7 @@ http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=index_condi… http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=mrr +http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=icp 2.2 Backport DS-MRR code to MariaDB 5.2 --------------------------------------- -=-=(Psergey - Thu, 26 Nov 2009, 18:15)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.4161 2009-11-26 18:15:36.000000000 +0200 +++ /tmp/wklog.67.new.4161 2009-11-26 18:15:36.000000000 +0200 @@ -1,3 +1,17 @@ + +<contents> +1. Requirements +2. Required actions +2.1 Fix DS-MRR/InnoDB bugs +2.2 Backport DS-MRR code to MariaDB 5.2 +2.3 Introduce control variables +2.4 Other backport issues +2.5 Make MRR code more of a module +2.6 Improve the cost model +2.7 Let DS-MRR support clustered primary keys +</contents> + + 1. Requirements =============== @@ -63,4 +77,28 @@ and the only modularization we see is to move #1 into a separate file which won't achieve much. +2.6 Improve the cost model +-------------------------- +At the moment DS-MRR cost formula re-uses non-MRR scan costs, which uses +records_in_range() calls, followed by index_only_read_time() or read_time() +calls to produce the estimate for read cost. + + We should change this (TODO sort out how exactly) + +Note: this means that the query plans will change from MariaDB 5.2. + +2.7 Let DS-MRR support clustered primary keys +--------------------------------------------- +At the moment DS-MRR is not supported for clustered primary keys. It is not +needed when MRR is used for range access, because range access is done over +an ordered list of ranges, but it is useful for BKA. + +TODO: + it's useful for BKA because BKA makes MRR scans over un-orderered + non-disjoint lists of ranges. Then we can sort these and do ordered scans. + There is still no use for DS-MRR over clustered primary key for range + access, where the ranges are disjoint and ordered. + How about postponing this item until BKA is backported? + + -=-=(Guest - Thu, 26 Nov 2009, 16:52)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.694 2009-11-26 14:52:53.000000000 +0000 +++ /tmp/wklog.67.new.694 2009-11-26 14:52:53.000000000 +0000 @@ -1 +1,66 @@ +1. Requirements +=============== + +We need the following: + +1. Latest MRR interface support, including extensions to support ICP when + using BKA. +2. Let DS-MRR support clustered primary keys (needed when using BKA). +3. Remove conditions used for key access from the condition pushed to index + (ATM this manifests itself as "Using index condition" appearing where there + was no "Using where". TODO: example of this?) +4. Introduce a separate @@optimizer_switch flag for turning on/out ICP (atm it + is switched on/off by @@engine_condition_pushdown) +5. Introduce a separate @@mrr_buffer_size variable to control MRR buffer size + for range+MRR scans. ATM it is controlled by @@read_rnd_size flag and that + makes it unobvious for a number of users. +6. Rename multi_range_read_info_const() to look like it is not a part of MRR + interface. +8. Try to make MRR to be more of a module +7. Improve MRR's cost model. + +2. Required actions +=================== + +Roughly in the order in which it will be done: + +2.1 Fix DS-MRR/InnoDB bugs +-------------------------- +We need to fix the bugs listed here: + +http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=index_condition_pushdown +http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=mrr + +2.2 Backport DS-MRR code to MariaDB 5.2 +--------------------------------------- +The easiest way seems to be to to manually move the needed code from mysql-6.0 +(or whatever it's called now) to MariaDB. + +2.3 Introduce control variables +------------------------------- +Act on items #4 and #5 from the requirements. Should be easy as +@@optimizer_switch is supported in 5.1 codebase. + +2.4 Other backport issues +------------------------- +* Figure out what to do with NDB/MRR. 5.1 codebase has "old" NDB/MRR + implementation. mysql-6.0 (and NDB's branch) have the updated NDB/MRR + but merging it into 5.1 can be very labor-intensive. + Will it be ok to disable NDB/MRR altogether? + + +2.5 Make MRR code more of a module +---------------------------------- +Some code in handler.cc can be moved to separate file. +But changes in opt_range.cc can't. +TODO: Sort out how much we really can do here. Initial guess is not much as the +code consists of: +- Default MRR implementation in handler.cc +- Changes in opt_range.cc to use MRR instead of multiple records_in_range() + calls. These rely on opt_range.cc's internal structures like SEL_ARG trees and + so there is not much point in moving them out. +- DS-MRR implementations which are spread over storage engines. +and the only modularization we see is to move #1 into a separate file which +won't achieve much. + DESCRIPTION: Backport DS-MRR into MariaDB-5.2 codebase, also adding certain extra features to make it more usable. HIGH-LEVEL SPECIFICATION: <contents> 1. Requirements 2. Required actions 2.1 Fix DS-MRR/InnoDB bugs 2.2 Backport DS-MRR code to MariaDB 5.2 2.3 Introduce control variables 2.4 Other backport issues 2.5 Make MRR code more of a module 2.6 Improve the cost model 2.7 Let DS-MRR support clustered primary keys </contents> 1. Requirements =============== We need the following: 1. Latest MRR interface support, including extensions to support ICP when using BKA. 2. Let DS-MRR support clustered primary keys (needed when using BKA). 3. Remove conditions used for key access from the condition pushed to index (ATM this manifests itself as "Using index condition" appearing where there was no "Using where". TODO: example of this?) 4. Introduce a separate @@optimizer_switch flag for turning on/out ICP (atm it is switched on/off by @@engine_condition_pushdown) 5. Introduce a separate @@mrr_buffer_size variable to control MRR buffer size for range+MRR scans. ATM it is controlled by @@read_rnd_size flag and that makes it unobvious for a number of users. 6. Rename multi_range_read_info_const() to look like it is not a part of MRR interface. 8. Try to make MRR to be more of a module 7. Improve MRR's cost model. 2. Required actions =================== Roughly in the order in which it will be done: 2.1 Fix DS-MRR/InnoDB bugs -------------------------- We need to fix the bugs listed here: http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=index_condi… http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=mrr http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=icp 2.2 Backport DS-MRR code to MariaDB 5.2 --------------------------------------- The easiest way seems to be to to manually move the needed code from mysql-6.0 (or whatever it's called now) to MariaDB. 2.3 Introduce control variables ------------------------------- Act on items #4 and #5 from the requirements. Should be easy as @@optimizer_switch is supported in 5.1 codebase. 2.4 Other backport issues ------------------------- * Figure out what to do with NDB/MRR. 5.1 codebase has "old" NDB/MRR implementation. mysql-6.0 (and NDB's branch) have the updated NDB/MRR but merging it into 5.1 can be very labor-intensive. Will it be ok to disable NDB/MRR altogether? 2.5 Make MRR code more of a module ---------------------------------- Some code in handler.cc can be moved to separate file. But changes in opt_range.cc can't. TODO: Sort out how much we really can do here. Initial guess is not much as the code consists of: - Default MRR implementation in handler.cc - Changes in opt_range.cc to use MRR instead of multiple records_in_range() calls. These rely on opt_range.cc's internal structures like SEL_ARG trees and so there is not much point in moving them out. - DS-MRR implementations which are spread over storage engines. and the only modularization we see is to move #1 into a separate file which won't achieve much. 2.6 Improve the cost model -------------------------- At the moment DS-MRR cost formula re-uses non-MRR scan costs, which uses records_in_range() calls, followed by index_only_read_time() or read_time() calls to produce the estimate for read cost. We should change this (TODO sort out how exactly) Note: this means that the query plans will change from MariaDB 5.2. 2.7 Let DS-MRR support clustered primary keys --------------------------------------------- At the moment DS-MRR is not supported for clustered primary keys. It is not needed when MRR is used for range access, because range access is done over an ordered list of ranges, but it is useful for BKA. TODO: it's useful for BKA because BKA makes MRR scans over un-orderered non-disjoint lists of ranges. Then we can sort these and do ordered scans. There is still no use for DS-MRR over clustered primary key for range access, where the ranges are disjoint and ordered. How about postponing this item until BKA is backported? ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0