----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: ICP/MRR backport CREATION DATE..: Thu, 26 Nov 2009, 15:19 SUPERVISOR.....: Monty IMPLEMENTOR....: Psergey COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 67 (http://askmonty.org/worklog/?tid=67) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Igor - Wed, 10 Mar 2010, 19:14)=-=- High Level Description modified. --- /tmp/wklog.67.old.25641 2010-03-10 19:14:45.000000000 +0000 +++ /tmp/wklog.67.new.25641 2010-03-10 19:14:45.000000000 +0000 @@ -1,2 +1,2 @@ -Backport DS-MRR into MariaDB-5.2 codebase, also adding certain extra features to -make it more usable. +Backport ICP and DS-MRR into MariaDB-5.2 codebase, also adding certain extra +features to make it more usable. -=-=(Guest - Wed, 10 Mar 2010, 19:12)=-=- Title modified. --- /tmp/wklog.67.old.25456 2010-03-10 19:12:57.000000000 +0000 +++ /tmp/wklog.67.new.25456 2010-03-10 19:12:57.000000000 +0000 @@ -1 +1 @@ -MRR backport +ICP/MRR backport -=-=(Psergey - Sun, 28 Feb 2010, 14:56)=-=- Dependency created: 91 now depends on 67 -=-=(Psergey - Sun, 28 Feb 2010, 14:54)=-=- Dependency deleted: 94 no longer depends on 67 -=-=(Psergey - Sun, 28 Feb 2010, 14:09)=-=- Dependency created: 94 now depends on 67 -=-=(Psergey - Thu, 26 Nov 2009, 20:21)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.9329 2009-11-26 20:21:28.000000000 +0200 +++ /tmp/wklog.67.new.9329 2009-11-26 20:21:28.000000000 +0200 @@ -65,17 +65,19 @@ 2.5 Make MRR code more of a module ---------------------------------- -Some code in handler.cc can be moved to separate file. -But changes in opt_range.cc can't. -TODO: Sort out how much we really can do here. Initial guess is not much as the -code consists of: +It is not possible to make MRR to be a totally separate module, as its code +consists of : - Default MRR implementation in handler.cc - Changes in opt_range.cc to use MRR instead of multiple records_in_range() - calls. These rely on opt_range.cc's internal structures like SEL_ARG trees and + calls. These rely on opt_range.cc's internal stuctures like SEL_ARG trees and so there is not much point in moving them out. -- DS-MRR implementations which are spread over storage engines. -and the only modularization we see is to move #1 into a separate file which -won't achieve much. +- DS-MRR impelementations which are spread over storage engines. + +We'll try to modularize what we can: +- Move out default MRR implementation from handler.cc +- Move possible parts out of opt_range.cc into a separate file. + + 2.6 Improve the cost model -------------------------- -=-=(Psergey - Thu, 26 Nov 2009, 19:06)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.6449 2009-11-26 19:06:04.000000000 +0200 +++ /tmp/wklog.67.new.6449 2009-11-26 19:06:04.000000000 +0200 @@ -1,4 +1,3 @@ - <contents> 1. Requirements 2. Required actions @@ -44,6 +43,7 @@ http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=index_condition_pushdown http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=mrr +http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=icp 2.2 Backport DS-MRR code to MariaDB 5.2 --------------------------------------- -=-=(Psergey - Thu, 26 Nov 2009, 18:15)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.4161 2009-11-26 18:15:36.000000000 +0200 +++ /tmp/wklog.67.new.4161 2009-11-26 18:15:36.000000000 +0200 @@ -1,3 +1,17 @@ + +<contents> +1. Requirements +2. Required actions +2.1 Fix DS-MRR/InnoDB bugs +2.2 Backport DS-MRR code to MariaDB 5.2 +2.3 Introduce control variables +2.4 Other backport issues +2.5 Make MRR code more of a module +2.6 Improve the cost model +2.7 Let DS-MRR support clustered primary keys +</contents> + + 1. Requirements =============== @@ -63,4 +77,28 @@ and the only modularization we see is to move #1 into a separate file which won't achieve much. +2.6 Improve the cost model +-------------------------- +At the moment DS-MRR cost formula re-uses non-MRR scan costs, which uses +records_in_range() calls, followed by index_only_read_time() or read_time() +calls to produce the estimate for read cost. + + We should change this (TODO sort out how exactly) + +Note: this means that the query plans will change from MariaDB 5.2. + +2.7 Let DS-MRR support clustered primary keys +--------------------------------------------- +At the moment DS-MRR is not supported for clustered primary keys. It is not +needed when MRR is used for range access, because range access is done over +an ordered list of ranges, but it is useful for BKA. + +TODO: + it's useful for BKA because BKA makes MRR scans over un-orderered + non-disjoint lists of ranges. Then we can sort these and do ordered scans. + There is still no use for DS-MRR over clustered primary key for range + access, where the ranges are disjoint and ordered. + How about postponing this item until BKA is backported? + + -=-=(Guest - Thu, 26 Nov 2009, 16:52)=-=- High-Level Specification modified. --- /tmp/wklog.67.old.694 2009-11-26 14:52:53.000000000 +0000 +++ /tmp/wklog.67.new.694 2009-11-26 14:52:53.000000000 +0000 @@ -1 +1,66 @@ +1. Requirements +=============== + +We need the following: + +1. Latest MRR interface support, including extensions to support ICP when + using BKA. +2. Let DS-MRR support clustered primary keys (needed when using BKA). +3. Remove conditions used for key access from the condition pushed to index + (ATM this manifests itself as "Using index condition" appearing where there + was no "Using where". TODO: example of this?) +4. Introduce a separate @@optimizer_switch flag for turning on/out ICP (atm it + is switched on/off by @@engine_condition_pushdown) +5. Introduce a separate @@mrr_buffer_size variable to control MRR buffer size + for range+MRR scans. ATM it is controlled by @@read_rnd_size flag and that + makes it unobvious for a number of users. +6. Rename multi_range_read_info_const() to look like it is not a part of MRR + interface. +8. Try to make MRR to be more of a module +7. Improve MRR's cost model. + +2. Required actions +=================== + +Roughly in the order in which it will be done: + +2.1 Fix DS-MRR/InnoDB bugs +-------------------------- +We need to fix the bugs listed here: + +http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=index_condition_pushdown +http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=mrr + +2.2 Backport DS-MRR code to MariaDB 5.2 +--------------------------------------- +The easiest way seems to be to to manually move the needed code from mysql-6.0 +(or whatever it's called now) to MariaDB. + +2.3 Introduce control variables +------------------------------- +Act on items #4 and #5 from the requirements. Should be easy as +@@optimizer_switch is supported in 5.1 codebase. + +2.4 Other backport issues +------------------------- +* Figure out what to do with NDB/MRR. 5.1 codebase has "old" NDB/MRR + implementation. mysql-6.0 (and NDB's branch) have the updated NDB/MRR + but merging it into 5.1 can be very labor-intensive. + Will it be ok to disable NDB/MRR altogether? + + +2.5 Make MRR code more of a module +---------------------------------- +Some code in handler.cc can be moved to separate file. +But changes in opt_range.cc can't. +TODO: Sort out how much we really can do here. Initial guess is not much as the +code consists of: +- Default MRR implementation in handler.cc +- Changes in opt_range.cc to use MRR instead of multiple records_in_range() + calls. These rely on opt_range.cc's internal structures like SEL_ARG trees and + so there is not much point in moving them out. +- DS-MRR implementations which are spread over storage engines. +and the only modularization we see is to move #1 into a separate file which +won't achieve much. + DESCRIPTION: Backport ICP and DS-MRR into MariaDB-5.2 codebase, also adding certain extra features to make it more usable. HIGH-LEVEL SPECIFICATION: <contents> 1. Requirements 2. Required actions 2.1 Fix DS-MRR/InnoDB bugs 2.2 Backport DS-MRR code to MariaDB 5.2 2.3 Introduce control variables 2.4 Other backport issues 2.5 Make MRR code more of a module 2.6 Improve the cost model 2.7 Let DS-MRR support clustered primary keys </contents> 1. Requirements =============== We need the following: 1. Latest MRR interface support, including extensions to support ICP when using BKA. 2. Let DS-MRR support clustered primary keys (needed when using BKA). 3. Remove conditions used for key access from the condition pushed to index (ATM this manifests itself as "Using index condition" appearing where there was no "Using where". TODO: example of this?) 4. Introduce a separate @@optimizer_switch flag for turning on/out ICP (atm it is switched on/off by @@engine_condition_pushdown) 5. Introduce a separate @@mrr_buffer_size variable to control MRR buffer size for range+MRR scans. ATM it is controlled by @@read_rnd_size flag and that makes it unobvious for a number of users. 6. Rename multi_range_read_info_const() to look like it is not a part of MRR interface. 8. Try to make MRR to be more of a module 7. Improve MRR's cost model. 2. Required actions =================== Roughly in the order in which it will be done: 2.1 Fix DS-MRR/InnoDB bugs -------------------------- We need to fix the bugs listed here: http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=index_condition_pushdown http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=mrr http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=icp 2.2 Backport DS-MRR code to MariaDB 5.2 --------------------------------------- The easiest way seems to be to to manually move the needed code from mysql-6.0 (or whatever it's called now) to MariaDB. 2.3 Introduce control variables ------------------------------- Act on items #4 and #5 from the requirements. Should be easy as @@optimizer_switch is supported in 5.1 codebase. 2.4 Other backport issues ------------------------- * Figure out what to do with NDB/MRR. 5.1 codebase has "old" NDB/MRR implementation. mysql-6.0 (and NDB's branch) have the updated NDB/MRR but merging it into 5.1 can be very labor-intensive. Will it be ok to disable NDB/MRR altogether? 2.5 Make MRR code more of a module ---------------------------------- It is not possible to make MRR to be a totally separate module, as its code consists of : - Default MRR implementation in handler.cc - Changes in opt_range.cc to use MRR instead of multiple records_in_range() calls. These rely on opt_range.cc's internal stuctures like SEL_ARG trees and so there is not much point in moving them out. - DS-MRR impelementations which are spread over storage engines. We'll try to modularize what we can: - Move out default MRR implementation from handler.cc - Move possible parts out of opt_range.cc into a separate file. 2.6 Improve the cost model -------------------------- At the moment DS-MRR cost formula re-uses non-MRR scan costs, which uses records_in_range() calls, followed by index_only_read_time() or read_time() calls to produce the estimate for read cost. We should change this (TODO sort out how exactly) Note: this means that the query plans will change from MariaDB 5.2. 2.7 Let DS-MRR support clustered primary keys --------------------------------------------- At the moment DS-MRR is not supported for clustered primary keys. It is not needed when MRR is used for range access, because range access is done over an ordered list of ranges, but it is useful for BKA. TODO: it's useful for BKA because BKA makes MRR scans over un-orderered non-disjoint lists of ranges. Then we can sort these and do ordered scans. There is still no use for DS-MRR over clustered primary key for range access, where the ranges are disjoint and ordered. How about postponing this item until BKA is backported? ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)