- developers - lists.mariadb.org

[Maria-developers] WL#220 New (by Monty): Dissallow table scan in SELECT
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Dissallow table scan in SELECT CREATION DATE..: Thu, 30 Jun 2011, 17:28 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 220 (http://askmonty.org/worklog/?tid=220) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 16 (hours remain) ORIG. ESTIMATE.: 16 PROGRESS NOTES: DESCRIPTION: Add a variable SQL_FORCE_INDEX_USAGE= 0 | 1 | 2 0 is normal. 1 would give an error if you are not specifying a usable index for all tables. It would still allow MariaDB to do a table scan if that's the best method. 2 Give an error for any SELECT's that would use a table scan on a normal (not internal temporary) table. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#219 New (by Monty): Query statistics
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Query statistics CREATION DATE..: Thu, 30 Jun 2011, 17:08 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 219 (http://askmonty.org/worklog/?tid=219) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 120 (hours remain) ORIG. ESTIMATE.: 120 PROGRESS NOTES: DESCRIPTION: Log skeletons for queries in server (live from parser) but without constants with statistics. Things to consider: * Should use a fixed memory size. Queries should be removed as FIFO * Small CPU overhead * IN(a,b,c) should be equal recorded as IN(#) (One constant) * multi row inserts needs to be handled as one insert. We also need a method to do regular expression replacement for table names. This is needed to be able to recognize similar tables like log tables with a _YYMMDD prefix. Statistics to collect per query: - Number of times executed - Total time executed (in microseconds) - Rows read, inserted, updated and deleted. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#218 New (by Monty): Better EXPLAIN
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Better EXPLAIN CREATION DATE..: Thu, 30 Jun 2011, 16:58 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 218 (http://askmonty.org/worklog/?tid=218) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: This is a master task for all tasks that makes EXPLAIN better. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#217 New (by Monty): Get list of active GET_LOCK
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Get list of active GET_LOCK CREATION DATE..: Thu, 30 Jun 2011, 16:51 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 217 (http://askmonty.org/worklog/?tid=217) VERSION........: Server-5.6 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 16 (hours remain) ORIG. ESTIMATE.: 16 PROGRESS NOTES: DESCRIPTION: Add a command SHOW GET_LOCKS to show all active GET_LOCK commands. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#216 New (by Monty): Make aria_pack an internal command in mysqld
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Make aria_pack an internal command in mysqld CREATION DATE..: Thu, 30 Jun 2011, 16:45 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 216 (http://askmonty.org/worklog/?tid=216) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: DESCRIPTION: Make aria_pack an internal command in mysqld Add the aria_pack source inside mysqld and add a new handler commands 'compress table' and 'uncompress table' ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#215 New (by Monty): Add functional indexes
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add functional indexes CREATION DATE..: Thu, 30 Jun 2011, 16:17 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 215 (http://askmonty.org/worklog/?tid=215) VERSION........: Server-5.6 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 200 (hours remain) ORIG. ESTIMATE.: 200 PROGRESS NOTES: DESCRIPTION: Add support for functional indexes. This is needed for MWL#34 Dynamic columns. The idea would be to extend the handler interface with a callback function that provides the handler with a key in normal key-read format of the index entry. In addition we need to enhance the optimizer to recognize the function used for the functional index parts so that it can use the index. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#214 New (by Monty): Make Indexes first level objects
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Make Indexes first level objects CREATION DATE..: Thu, 30 Jun 2011, 16:11 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Maria-RawIdeaBin TASK ID........: 214 (http://askmonty.org/worklog/?tid=214) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: An index should be able to exist as an object that is not tied to a physical table. That would allow for indexes that are global to a > partition, and for functional indexes or indexes on virtual columns. The above is extremely hard to do as Indexes are very closely tied to storage engines. What is easier to do is to do functional indexes, where the storage engine will ask MySQL for the index data for a row. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#213 New (by Monty): Run SP on binary log rotation
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Run SP on binary log rotation CREATION DATE..: Thu, 30 Jun 2011, 16:03 SUPERVISOR.....: Knielsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 213 (http://askmonty.org/worklog/?tid=213) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: On binlog rotation, run a SP asynchronously (for example to run a shell command to copy/compress the binary logs) Provide sufficient environmental data to the SP (binlog names old and new, binlog position for old and new log, etc). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#212 New (by Monty): Compressed binary log
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Compressed binary log CREATION DATE..: Thu, 30 Jun 2011, 15:34 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 212 (http://askmonty.org/worklog/?tid=212) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: Compress the binary log to make it smaller. The following options exists: a) Compress old binary log when they are not in use. On access from delayed slave, decompress it from start to memory and feed the slave the data from the uncompressed stream. b) Compress each binary log event separately with a format: <length><data>. Ship the binary log in the compressed format. I assume that b) is much easier to do. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#211 New (by Monty): conflict resolution for RBR replication
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: conflict resolution for RBR replication CREATION DATE..: Thu, 30 Jun 2011, 15:28 SUPERVISOR.....: Knielsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 211 (http://askmonty.org/worklog/?tid=211) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: In RBR, when there is any kind of RBR problem, call a SP. The SP gets all required data made available somehow, can define a resolution policy. 4 Outcomes: Choose old, choose new, fail (error) and synthesize (merge) new row. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#210 New (by Monty): Make all replication variables changeable without server restart
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Make all replication variables changeable without server restart CREATION DATE..: Thu, 30 Jun 2011, 15:17 SUPERVISOR.....: IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 210 (http://askmonty.org/worklog/?tid=210) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Make all replication variables changeable without server restart. This requires some work as there is many variables and some variables require more work than others. Here we will list the variables in priority order that should be made changeable. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#209 New (by Monty): Setting up a new slave easily
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Setting up a new slave easily CREATION DATE..: Thu, 30 Jun 2011, 15:12 SUPERVISOR.....: Knielsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 209 (http://askmonty.org/worklog/?tid=209) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: The current procedure to setup a slave is: * mylvmbackup (or similar snapshot) on a slave * restore it to a new box * recover InnoDB * connect it to the master. We'd like to have a command that can be issues to an empty database, will download the data from a master (or a co-slave), and then attach the new slave to the master (using the configuration provided on the command line or by the co-slave). Requirements: * Should be done at SQL level. * With progress report. * The data should move at disk speed; No index creation, ship binary index data from the co-slave or master. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#208 New (by Monty): Replicate ALTER TABLE in parallel
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Replicate ALTER TABLE in parallel CREATION DATE..: Thu, 30 Jun 2011, 14:55 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 208 (http://askmonty.org/worklog/?tid=208) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: Replicate ALTER TABLE in parallel For ALTER TABLES on the slave we could start another thread to do the ALTER TABLE while the replication thread continues. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#202 New (by Monty): Extend progress reporting
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Extend progress reporting CREATION DATE..: Thu, 30 Jun 2011, 11:04 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 202 (http://askmonty.org/worklog/?tid=202) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: MariaDB 5.3 supports progress reporting for ALTER TABLE and some other comands. See http://kb.askmonty.org/en/progress-reporting The task is to expand this to other commands and storage engines. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

3 7

[Maria-developers] WL#207 New (by Monty): Replicate LOAD DATA INFILE 'out of band'
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Replicate LOAD DATA INFILE 'out of band' CREATION DATE..: Thu, 30 Jun 2011, 14:47 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 207 (http://askmonty.org/worklog/?tid=207) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: Adding support for replication some LOAD DATA INFILE commands 'out of band' The problem to solve is to allow upload of large new tables to slaves, without having the data downloading stop the normal replication stream. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#206 New (by Monty): Parallel replication
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Parallel replication CREATION DATE..: Thu, 30 Jun 2011, 14:29 SUPERVISOR.....: Knielsen IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 206 (http://askmonty.org/worklog/?tid=206) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: This is a master task for all tasks that makes replication more parallel ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#205 New (by Monty): DELETE with result set
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: DELETE with result set CREATION DATE..: Thu, 30 Jun 2011, 14:13 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 205 (http://askmonty.org/worklog/?tid=205) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: DESCRIPTION: Add a DELETE operations that returns a result set of the deleted rows to the client. Suggested syntax: The DB2 syntax: DELETE from (SELECT column_list from single_table WHERE ... ORDER BY) another option would be DELETE FROM table_name WHERE ... [LIMIT #]] SELECT expression_list but I think that the first syntax is more clear. It's also easier to expand it to handle multiple tables. Oracle supports the following syntax: DELETE <table or expression> <alias> WHERE <where_clause> RETURNING <exprs> INTO <data_items> but this is not useful as it can only be used to record single row deletes. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#204 New (by Monty): Add audit information to the bnary log
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add audit information to the bnary log CREATION DATE..: Thu, 30 Jun 2011, 13:06 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 204 (http://askmonty.org/worklog/?tid=204) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Add audit information to the binary log. As the audit information takes extra place, this should be done conditionally on the master. The audit information should be printed by mysqlbinlog -v -v The slave should just skip the audit information. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#203 New (by Monty): Add efficient support for BLOB and VARCHAR to HEAP/MEMORY
by worklog-noreply＠askmonty.org 30 Jun '11

30 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add efficient support for BLOB and VARCHAR to HEAP/MEMORY CREATION DATE..: Thu, 30 Jun 2011, 11:41 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 203 (http://askmonty.org/worklog/?tid=203) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 60 (hours remain) ORIG. ESTIMATE.: 60 PROGRESS NOTES: DESCRIPTION: Add efficient support for BLOB and VARCHAR to the HEAP/MEMORY engine. Requirements: * Should work for all MEMORY tables (no requirement of field order) * Should work for internal memory tables. * Low overhead for not changed rows. * Fixed storage size (as now) for total size of MEMORY TABLE. Should be optimized for insert and select (most common operations for internal temporary tables). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Philip please test: Duplicate Elimination for outer joins + semi joins
by Sergey Petrunya 30 Jun '11

30 Jun '11

Hello Philip, I'd like to get the tree at lp:~maria-captains/maria/5.3-subqueries-mwl90 tested for Duplicate Elimination with outer+semi-join processing. I'm interested in crashes, as well as wrong query result bugs. The code path that needs to be tested is selected with this @@optimizer_switch setting: semijoin=on,materialization=off,firstmatch=off,loosescan=off this set of switches disables all semi-join subquery strategies except DuplicateElimination, so you should have no problem with hitting the If you also set "semijoin=off" then you get the execution path that we assume will always produce correct results. Query pattern SELECT .... FROM complex_join WHERE top_where AND (SELECT ... FROM complex_join WHERE ...) where - subqueries must be "mergeable semi-joins": = no aggregates, unions, or ORDER BYs - both correlated and ucorrelated subqueries are OK. - complex_join is arbitrary join structure (comma join, "t1 JOIN t2", various kinds of outer joins - everything should work) - multiple sibling subqueries, as well as nested subqueries are covered by what you're testing. EXPLAIN - Should have id=='1' && select_type='PRIMARY' on all lines. - Should use Start Temporary/End temporary. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 2

Re: [Maria-developers] [Commits] Rev 2983: WL#77 - INSTALL PLUGIN * in http://bazaar.launchpad.net/~maria-captains/maria/5.2/
by Alexey Botchkov 29 Jun '11

29 Jun '11

I'm fine with this last patch. One thing i can think of is if we should add a new error instead of adding static english texts to the ER_CANT_OPEN_LIBRARY like 'API version for ... plugin ... not supported'. Second question if we should have INSTALL PLUGIN ... IF NOT INSTALLED; to be even more user-friendly. Regards HF 17.06.2011 21:49, serg(a)askmonty.org wrote: > At http://bazaar.launchpad.net/~maria-captains/maria/5.2/ >

1 0

[Maria-developers] MariaDB 5.3 docs
by Daniel Bartholomew 29 Jun '11

29 Jun '11

All, We're getting closer to the first beta release of MariaDB 5.3 and several of the new features have been documented in the Knowledgebase and linked to from this page: http://kb.askmonty.org/en/what-is-mariadb-53 What I don't know is if there are any features which have either: 1. not been documented in the Knowledgebase at all, -or- 2. been incompletely documented in the Knowledgebase but not added to the above page, -or- 3. been added to the page but incompletely documented. If you have worked on 5.3, please review the above page to make sure the items you worked on are present and fully documented. If they aren't please document them in the Knowledgebase and add them to the above page (or send me enough information about them so that I can document them). If they are present, but inadequately documented, please add what is missing or let me know what is missing. Don't worry about your documentation being perfect, I will clean it up. If you need any help, please contact me via email or on IRC. Thanks. -- Daniel Bartholomew MariaDB - http://mariadb.org Monty Program - http://montyprogram.com AskMonty Knowledgebase - http://kb.askmonty.org

2 2

[Maria-developers] Proposal to split @@optimizer_switch
by Sergey Petrunya 28 Jun '11

28 Jun '11

Hello, Please find below descriptions of problems with current @@optimizer_switch variable, as well as a proposal on how to fix it. <contents> Problems with @@optimizer_switch Proposed solution Details about hooking this into parser Alternate approach1: grouping </contents> Problems with @@optimizer_switch -------------------------------- @@optimizer_switch variable use has outgrown its syntax. There are too many settings to fit on the single line: MariaDB [(none)]> select @@optimizer_switch; +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | @@optimizer_switch | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=off,mrr_sort_keys=on,outer_join_with_cache=off,semijoin_with_cache=off,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) The line is now 510 characters long. It's hard to find the flag you're looking for, and one needs to write regular expressions if they want to check the value of a particular flag in some program. There is no easy way to get settings for a subset of flags (which means we've banned "select @@optimizer_switch" from almost all .test files, because one had to change .result file all the time) The extra-long line also gets into "SHOW VARIABLES" output, and makes it hard-to-read, too. The situation is going to get worse, as we've already committed to adding even to add even more switches (Igor's patch adds some, then we've decided that two semi-join optimizations that do not have their switches, should have them) The benefits of having all settings grouped together are: - It was easy to implement - One can easily see all optimizer switches without having to fish them out of the list of all server variables. - One can reset all optimizer settings with a single statement: SET @@optimizer_switch=default Proposed solution ----------------- Break optimizer_switch into multiple variables, with names having the dots in them. That is, current @@optimizer_switch flags will change into this set of variables: optimizer.index_merge=on optimizer.index_merge.union=on optimizer.index_merge.sort_union=on optimizer.index_merge.intersection=on optimizer.index_merge.sort_intersection=off optimizer.index_condition_pushdown=on optimizer.join_cache.bka=on optimizer.join_cache.hashed=on optimizer.join_cache.incremental=on optimizer.join_cache.optimize_buffer_size=on optimizer.join_cache.outer_join=off optimizer.join_cache.semijoin=off optimizer.mrr=on optimizer.mrr.cost_based=off optimizer.mrr.sort_keys=on optimizer.semijoin=on optimizer.semijoin.firstmatch=on optimizer.semijoin.loosescan=on optimizer.subquery.cache=on optimizer.subquery.in_to_exists=on optimizer.subquery.materialization=off optimizer.subquery.partial_match_rowid_merge=on optimizer.subquery.partial_match_table_scan=on optimizer.table_elimination=on The dots are there so that - the structuring is more apparent - it is possible to reset groups of variables to their defaults. Current idea is to use syntax like this: SET optimizer.semijoin.*= default; - in the future we may also allow to select and set groups of settings. Details about hooking this into parser ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Currently, dotted syntax is allowed to some extent: we allow - GLOBAL/SESSION/DEFAULT prefix for any variable - key_cache_name prefix for multiple named key caches. There is one key cache named default, it is not possible to make a key cache named "session". To make variable lookup rules easy, we will diallow creation/use of key caches named 'optimizer'. Overall, system variables can be encountered - as part of @@variable syntax when their value is selected - as part of SET statement. in both cases, bison can easily see it's parsing a system variable name, so I was able to add support for dotted variable syntax into the parser without introducing any extra conflicts. Alternate approach1: grouping ----------------------------- (Suggested by Igor (or was it Timour?)) Replace one @@optimizer_switch variable with multiple: @@optimizer_switch_semijoin @@optimizer_switch_mrr .. and so forth. The advantage of this approach is that it's easy and we don't need to change the parser. The disadavantage is that it may turn out that groups of flags may be still too big, and so either look ugly or require further splits. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

4 9

Re: [Maria-developers] [Commits] Rev 3062: Fixed bug lp:800679 in file:///home/tsk/mprog/src/5.3-mwl89/
by Sergey Petrunya 28 Jun '11

28 Jun '11

Hi Timour, Ok to push. On Mon, Jun 27, 2011 at 05:40:25PM +0300, timour(a)askmonty.org wrote: > At file:///home/tsk/mprog/src/5.3-mwl89/ > > ------------------------------------------------------------ > revno: 3062 > revision-id: timour(a)askmonty.org-20110627144014-r14ahro6uzf77423 > parent: igor(a)askmonty.org-20110625210227-kdneawayskx5jl4q > fixes bug(s): https://launchpad.net/bugs/800679 > committer: timour(a)askmonty.org > branch nick: 5.3-mwl89 > timestamp: Mon 2011-06-27 17:40:14 +0300 > message: > Fixed bug lp:800679 > > Analysis: > The failed assert ensured that the choice of subquery strategy > is performed only for queries with at least one table. If there > is a LIMIT 0 clause all tables are removed, and the subquery is > neither optimized, nor executed during actual optimization. However, > if the query is EXPLAIN-ed, the EXPLAIN execution path doesn't remove > the query tables if there is a LIMIT 0 clause. As a result, the > subquery optimization code is called, which violates the ASSERT > condition. > > Solution: > Transform the assert into a condition, and if the outer query > has no tables assume that there will be at most one subquery > execution. > > There is potentially a better solution by reengineering the > EXPLAIN/optimize code, so that subquery optimization is not > done if not needed. Such a solution would be a lot bigger and > more complex than a bug fix. > === modified file 'mysql-test/r/subselect_mat_cost_bugs.result' > --- a/mysql-test/r/subselect_mat_cost_bugs.result 2011-06-05 02:56:06 +0000 > +++ b/mysql-test/r/subselect_mat_cost_bugs.result 2011-06-27 14:40:14 +0000 > @@ -295,3 +295,24 @@ id select_type table type possible_keys > select c1 from t1 where c1 in (select kp1 from t2 where kp2 = 10 and c2 = 4) or c1 > 7; > c1 > drop table t1, t2; > +# > +# LP BUG#800679: Assertion `outer_join->table_count > 0' failed in > +# JOIN::choose_subquery_plan() with materialization=on,semijoin=off > +# > +CREATE TABLE t1 ( f1 int); > +insert into t1 values (1),(2); > +CREATE TABLE t2 ( f1 int); > +insert into t2 values (1),(2); > +SET @@optimizer_switch='materialization=on,semijoin=off'; > +EXPLAIN > +SELECT * FROM t1 > +WHERE (f1) IN (SELECT f1 FROM t2) > +LIMIT 0; > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE > +2 DEPENDENT SUBQUERY t2 ALL NULL NULL NULL NULL 2 Using where > +SELECT * FROM t1 > +WHERE (f1) IN (SELECT f1 FROM t2) > +LIMIT 0; > +f1 > +drop table t1, t2; > > === modified file 'mysql-test/t/subselect_mat_cost_bugs.test' > --- a/mysql-test/t/subselect_mat_cost_bugs.test 2011-05-23 07:56:05 +0000 > +++ b/mysql-test/t/subselect_mat_cost_bugs.test 2011-06-27 14:40:14 +0000 > @@ -326,3 +326,26 @@ select c1 from t1 where c1 in (select kp > select c1 from t1 where c1 in (select kp1 from t2 where kp2 = 10 and c2 = 4) or c1 > 7; > > drop table t1, t2; > + > +--echo # > +--echo # LP BUG#800679: Assertion `outer_join->table_count > 0' failed in > +--echo # JOIN::choose_subquery_plan() with materialization=on,semijoin=off > +--echo # > + > +CREATE TABLE t1 ( f1 int); > +insert into t1 values (1),(2); > +CREATE TABLE t2 ( f1 int); > +insert into t2 values (1),(2); > + > +SET @@optimizer_switch='materialization=on,semijoin=off'; > + > +EXPLAIN > +SELECT * FROM t1 > +WHERE (f1) IN (SELECT f1 FROM t2) > +LIMIT 0; > + > +SELECT * FROM t1 > +WHERE (f1) IN (SELECT f1 FROM t2) > +LIMIT 0; > + > +drop table t1, t2; > > === modified file 'sql/opt_subselect.cc' > --- a/sql/opt_subselect.cc 2011-06-21 13:00:41 +0000 > +++ b/sql/opt_subselect.cc 2011-06-27 14:40:14 +0000 > @@ -4358,7 +4358,7 @@ bool JOIN::choose_subquery_plan(table_ma > by the IN predicate. > */ > outer_join= unit->outer_select() ? unit->outer_select()->join : NULL; > - if (outer_join) > + if (outer_join && outer_join->table_count > 0) > { > /* > The index of the last JOIN_TAB in the outer JOIN where in_subs is > @@ -4371,7 +4371,6 @@ bool JOIN::choose_subquery_plan(table_ma > JOIN_TAB, and their join_tab_idx remains MAX_TABLES. Such predicates > are evaluated for each complete row of the outer join. > */ > - DBUG_ASSERT(outer_join->table_count > 0); > max_outer_join_tab_idx= (in_subs->get_join_tab_idx() == MAX_TABLES) ? > outer_join->table_count - 1: > in_subs->get_join_tab_idx(); > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] can "set read_only" be non-blocking?
by MARK CALLAGHAN 27 Jun '11

27 Jun '11

It can be dangerous for us to run "set read_only" on a production server because it can block in close_cached_tables. More details about the pain this caused at a previous job are at: http://mysqlha.blogspot.com/2008/07/what-exactly-does-flush-tables-with.html Per the code in set_var.cc: /* Perform a 'FLUSH TABLES WITH READ LOCK'. This is a 3 step process: - [1] lock_global_read_lock() - [2] close_cached_tables() - [3] make_global_read_lock_block_commit() [1] prevents new connections from obtaining tables locked for write. [2] waits until all existing connections close their tables. [3] prevents transactions from being committed. */ Can there be a variant that doesn't do #2? My workload doesn't use MyISAM and I don't know if #2 is done because of MyISAM. Calling close_cached_tables seems like a heavy way to force LOCK TABLEs to be unlocked. Any long running queries will cause #2 to block. -- Mark Callaghan mdcallag(a)gmail.com

2 4

Re: [Maria-developers] [Commits] Rev 3062: Fixed LP bug #801536. in file:///home/igor/maria/maria-5.3-merge-tmp/
by Timour Katchaounov 27 Jun '11

27 Jun '11

Igor, Ok to push in the sense that the new patch cannot any more result in overflow. Both the old and new code are not equivalent. Given there is no comment, I cannot fully understand whether the change produces at least as good estimate of the record length as the old patch. Could you please add some general comment to the whole function how do we compute the field length. Timour On 27.06.2011 07:55, Igor Babaev wrote: > At file:///home/igor/maria/maria-5.3-merge-tmp/ > > ------------------------------------------------------------ > revno: 3062 > revision-id: igor(a)askmonty.org-20110627045532-663o9eih0flql0n6 > parent: igor(a)askmonty.org-20110625210227-kdneawayskx5jl4q > committer: Igor Babaev<igor(a)askmonty.org> > branch nick: maria-5.3-merge-tmp > timestamp: Sun 2011-06-26 21:55:32 -0700 > message: > Fixed LP bug #801536. > Ensured valid calculations of the estimates stored in JOIN_TAB::used_fieldlength. >

1 0

[Maria-developers] WL#201 New (by Knielsen): Multi-source replication
by worklog-noreply＠askmonty.org 23 Jun '11

23 Jun '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Multi-source replication CREATION DATE..: Thu, 23 Jun 2011, 12:50 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 201 (http://askmonty.org/worklog/?tid=201) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: This worklog is about implementing a way for a MariaDB replication slave to replicate from more than one master simultaneously. Each master is handled by a specific replicator instance in the slave server. Each replicator instance consists of separate I/O thread, SQL thread, and associated state and configuration. In efffect, several replication slaves are running at the same time, each replicating from a separate master, but all replicating into a common data store (typically, but not necessarily to separate databases/tables). A replicator instance is identified with a user-chosen name used in replication SQL statements such as CHANGE MASTER TO ... This name is also included in file names to distinguish the files that keep the replication state (relay logs, master.info, relay-log.info). This way, each separate instance can be configured separately, but otherwise the same way as existing single-source replication. In order to remain backwards-compatible with existing third-party scripts etc., the replicator instance name is made optional in all existing replication statements. If it is omitted, the name "default" is used, and for this particular name, master.info and the other files retain their old names to allow seamless upgrades of slaves. In this worklog, there is no extra conflict resolution proposed. The effect of updates from one master on replication from another master will be the same as the effect of direct user SQL queries on the slave server, ie. it is the responsibility of the user/DBA to ensure no conflicts occur. If a conflict causes replication of an event to fail (with duplicate key violation eg.), the corresponding slave SQL thread will stop, requiring manual intervention to fix and start. An easy and typical way to avoid conflicts would be to eg. to use separate databases for each master->slave replication channel. RBR idempotent slave application can also be used to help resolve conflicts, for example. See also MySQL WL#1697: http://forge.mysql.com/worklog/task.php?id=1697 ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] 5.5 merge: sql_select.cc
by Sergei Golubchik 23 Jun '11

23 Jun '11

Hi. Here I'll list changesets that cause conflicts and ask what to do. 1. http://tinyurl.com/63tey39 vs. http://tinyurl.com/5v2pluf it's a fix for bug#49322. both changes are similar, return is moved down. But Igor put it after the while() loop, while Ole John - in the loop, as soon a non-match is found. What difference does it make, I mean the earlier return, what consequences? 2. http://tinyurl.com/69lwrp9 - a big chunk is moved around in test_if_skip_sort_order() that's all, the rest of the file I've merged already. Regards, Sergei

3 2

[Maria-developers] forwarded message from Kristian Köhntopp
by Michael Widenius 21 Jun '11

21 Jun '11

Hi! Here is booking.com's wish list! Regards, Monty

3 2

[Maria-developers] MariaDB limits
by Marian Marinov 21 Jun '11

21 Jun '11

Hello guys, I have a lot of clients requesting some kind of per user limiting based on the information from the STATISTICS tables. I'm thinking of creating a patch that will add this functionality by creating two more tables in the INFORMATION_SCHEMA: USER_LIMITS CLIENT_LIMITS They will have some parts of the structure of USER_STATISTICS and CLIENT_STATISTICS and will keep only values for limits. By default a 0 would mean no limit. Do you have any comments about my aproach? Would you be willing to accept a patch that adds such functionality? I want to know before I start work on this project if the community is interested in such enhancement of MariaDB. Regards, Marian

3 4

Re: [Maria-developers] Notes from the two IT Day Sessions
by Michael Widenius 20 Jun '11

20 Jun '11

Hi! Follow up to last email. I will hope to have the worklog items for all the wishes done by Thursday. Regards, Monty

1 0

[Maria-developers] MariaDB in EPEL
by Marian Marinov 20 Jun '11

20 Jun '11

Hello, I wish to package MariaDB for the EPEL repository. I have searched a bit to see if there were any efforts in that direction but I saw none. In 2009 there was some talk about MariaDB and Fedora, but that was it, only some talk. I have the time to prepare and test the MariaDB RPMs and to continue support them later. My idea is to have MariaDB RPMs as replacements of the default mysql installation. In order to do so, I first want the approval of the MariaDB developers. Best regards, Marian Marinov

2 3

Re: [Maria-developers] Rev 2994: BUG#790513: MariaDB crashes on startup
by Sergei Golubchik 17 Jun '11

17 Jun '11

Hi, Sergey! On Jun 17, Sergey Petrunya wrote: > Hi Sergei, > > Could you please review the below: > > ------------------------------------------------------------ > revno: 2994 > committer: Sergey Petrunya <psergey(a)askmonty.org> > timestamp: Fri 2011-06-17 01:05:20 +0400 > message: > BUG#790513: MariaDB crashes on startup please use lp:790513 abbreviation for bugs. see 'bzr help bugs' for details. > - Let ha_resolve_by_name() print error in case it could locate > the needed plugin but the plugin hasn't yet been initialized. > One can hit this problem by having mysql.plugin of type=innodb. Okay, I see the problem. But why couldn't we just initialize *all* built-in plugins earlier, before accessing mysql.plugin table ? > === modified file 'sql/handler.cc' > --- a/sql/handler.cc 2011-05-03 16:10:10 +0000 > +++ b/sql/handler.cc 2011-06-16 21:05:20 +0000 > @@ -135,6 +135,12 @@ > if ((plugin= my_plugin_lock_by_name(thd, name, MYSQL_STORAGE_ENGINE_PLUGIN))) > { > handlerton *hton= plugin_data(plugin, handlerton *); > + if (!hton) > + { > + sql_print_error("Attempt to use a built-in plugin %*s before it " > + "has been initialized", name->length, name->str); > + return NULL; > + } > if (!(hton->flags & HTON_NOT_USER_SELECTABLE)) > return plugin; > Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3049: BUG#778406: Crash in hp_movelink with Aria engine and subqueries in file:///home/psergey/dev2/5.3-push3/
by Timour Katchaounov 17 Jun '11

17 Jun '11

Sergey, Ok to push. I'd rather add a comment to the block of code starting with: " // 1. Store the length " At least to me it was not obvious why it is the way it is. Timour > At file:///home/psergey/dev2/5.3-push3/ > > ------------------------------------------------------------ > revno: 3049 > revision-id: psergey(a)askmonty.org-20110615143701-cgzzcr5l4ot2cxjy > parent: psergey(a)askmonty.org-20110615134018-5g5jbhqhce43onz4 > committer: Sergey Petrunya<psergey(a)askmonty.org> > branch nick: 5.3-push3 > timestamp: Wed 2011-06-15 18:37:01 +0400 > message: > BUG#778406: Crash in hp_movelink with Aria engine and subqueries > -In do_sj_dups_weedout(), set nulls_ptr to point to NULL bytes (and not to length bytes) of the DuplicateWeedout column. > === modified file 'mysql-test/r/subselect_sj.result' > --- a/mysql-test/r/subselect_sj.result 2011-06-15 09:43:04 +0000 > +++ b/mysql-test/r/subselect_sj.result 2011-06-15 14:37:01 +0000 > @@ -1305,4 +1305,25 @@ > GROUP BY field2 ; > field2 > drop table t1, t2, t3; > +# > +# BUG#778406 Crash in hp_movelink with Aria engine and subqueries > +# > +CREATE TABLE t4 (f10 varchar(32) , KEY (f10)) ENGINE=Aria; > +INSERT INTO t4 VALUES ('x'),('m'),('c'); > +CREATE TABLE t1 (f11 int) ENGINE=Aria; > +INSERT INTO t1 VALUES (0),(0),(0); > +CREATE TABLE t2 ( f10 int) ENGINE=Aria; > +INSERT INTO t2 VALUES (0),(0),(0); > +CREATE TABLE t3 ( f10 int, f11 int) ENGINE=Aria; > +SELECT * > +FROM t4 > +WHERE f10 IN > +( SELECT t1.f11 > +FROM t1 > +LEFT JOIN t2 JOIN t3 ON t3.f10 = t2.f10 ON t3.f11 != 0 ); > +f10 > +x > +m > +c > +drop table t1,t2,t3,t4; > set @@optimizer_switch=@save_optimizer_switch; > > === modified file 'mysql-test/r/subselect_sj_jcl6.result' > --- a/mysql-test/r/subselect_sj_jcl6.result 2011-06-15 09:43:04 +0000 > +++ b/mysql-test/r/subselect_sj_jcl6.result 2011-06-15 14:37:01 +0000 > @@ -1313,6 +1313,27 @@ > GROUP BY field2 ; > field2 > drop table t1, t2, t3; > +# > +# BUG#778406 Crash in hp_movelink with Aria engine and subqueries > +# > +CREATE TABLE t4 (f10 varchar(32) , KEY (f10)) ENGINE=Aria; > +INSERT INTO t4 VALUES ('x'),('m'),('c'); > +CREATE TABLE t1 (f11 int) ENGINE=Aria; > +INSERT INTO t1 VALUES (0),(0),(0); > +CREATE TABLE t2 ( f10 int) ENGINE=Aria; > +INSERT INTO t2 VALUES (0),(0),(0); > +CREATE TABLE t3 ( f10 int, f11 int) ENGINE=Aria; > +SELECT * > +FROM t4 > +WHERE f10 IN > +( SELECT t1.f11 > +FROM t1 > +LEFT JOIN t2 JOIN t3 ON t3.f10 = t2.f10 ON t3.f11 != 0 ); > +f10 > +x > +m > +c > +drop table t1,t2,t3,t4; > set @@optimizer_switch=@save_optimizer_switch; > # > # BUG#49129: Wrong result with IN-subquery with join_cache_level=6 and firstmatch=off > > === modified file 'mysql-test/t/subselect_sj.test' > --- a/mysql-test/t/subselect_sj.test 2011-06-15 09:43:04 +0000 > +++ b/mysql-test/t/subselect_sj.test 2011-06-15 14:37:01 +0000 > @@ -1189,5 +1189,28 @@ > > drop table t1, t2, t3; > > +--echo # > +--echo # BUG#778406 Crash in hp_movelink with Aria engine and subqueries > +--echo # > +CREATE TABLE t4 (f10 varchar(32) , KEY (f10)) ENGINE=Aria; > +INSERT INTO t4 VALUES ('x'),('m'),('c'); > + > +CREATE TABLE t1 (f11 int) ENGINE=Aria; > +INSERT INTO t1 VALUES (0),(0),(0); > + > +CREATE TABLE t2 ( f10 int) ENGINE=Aria; > +INSERT INTO t2 VALUES (0),(0),(0); > + > +CREATE TABLE t3 ( f10 int, f11 int) ENGINE=Aria; > + > +SELECT * > +FROM t4 > +WHERE f10 IN > +( SELECT t1.f11 > +FROM t1 > +LEFT JOIN t2 JOIN t3 ON t3.f10 = t2.f10 ON t3.f11 != 0 ); > + > +drop table t1,t2,t3,t4; > + > # The following command must be the last one the file > set @@optimizer_switch=@save_optimizer_switch; > > === modified file 'sql/opt_subselect.cc' > --- a/sql/opt_subselect.cc 2011-06-06 22:50:46 +0000 > +++ b/sql/opt_subselect.cc 2011-06-15 14:37:01 +0000 > @@ -3515,7 +3515,6 @@ > } > > ptr= sjtbl->tmp_table->record[0] + 1; > - nulls_ptr= ptr; > > /* Put the the rowids tuple into table->record[0]: */ > > @@ -3531,6 +3530,7 @@ > ptr += 2; > } > > + nulls_ptr= ptr; > // 2. Zero the null bytes > if (sjtbl->null_bytes) > { > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] MariaDB 5.2.7
by Daniel Bartholomew 15 Jun '11

15 Jun '11

Hello all, Just writing to let you know that MariaDB 5.2.7 is now almost completely mirrored worldwide, so I've activated the downloads page and the release notes and changelog. Here are the links: http://kb.askmonty.org/en/mariadb-527-release-notes http://kb.askmonty.org/en/mariadb-527-changelog http://downloads.askmonty.org/mariadb/5.2.7/ If any of you have a moment, please review the release notes and make (or suggest) any necessary changes. I tried to incorporate everything that I was told about this release, but I may have missed something. Now turning things over to the community team to announce and promote. Thanks. -- Daniel Bartholomew MariaDB - http://mariadb.org Monty Program - http://montyprogram.com AskMonty Knowledgebase - http://kb.askmonty.org

1 0

[Maria-developers] Preparing to release MariaDB 5.2.7
by Daniel Bartholomew 14 Jun '11

14 Jun '11

Monty, I saw your note about the new MariaDB 5.2.7 release being ready, but it looks like the "mariadb-5.2.7" tag has not been added to the 5.2 tree on launchpad. Can you or one of the other developers add it? I need the tag as part of my process for creating the changelog. Apart from that, I just need to determine which buildbot build is the correct one. Kristian, You sent me some instructions a while back for how to do this but I can't remember them. I guess you've spoiled me by usually just telling me the build number. :) Can you send the instructions to me again? Thanks. -- Daniel Bartholomew MariaDB - http://mariadb.org Monty Program - http://montyprogram.com AskMonty Knowledgebase - http://kb.askmonty.org

3 4

[Maria-developers] tmp-table-tests MariaDB 5.2.6 vs. 5.5.12 vs. 5.1.57
by erkan yanar 09 Jun '11

09 Jun '11

Moin, I did some tmp-table tests. As it is told MariaDB is faster on tmp-tables because of the Aria-Engine. Therefore Ive got 3 Severs running: onyx:/data/dbod # for i in 6051 6053 6060; do mysql -u erkan -h 10.255.128.4 -P $i -e 'select version()'; done +------------+ | version() | +------------+ | 5.5.12-log | +------------+ +-------------------+ | version() | +-------------------+ | 5.2.6-MariaDB-log | +-------------------+ +------------+ | version() | +------------+ | 5.1.57-log | key_buffer_size is 16MB. For MariaDB there is also: mysql> show global variables like 'aria_pagecache_buffer_size'; +----------------------------+----------+ | Variable_name | Value | +----------------------------+----------+ | aria_pagecache_buffer_size | 16777216 | +----------------------------+----------+ To provoke tmp-tables I did: onyx:/data/dbod # for i in 6051 6053 6060; do mysql -u erkan -h 10.255.128.4 -P $i -e 'show global variables like "tmp_table_size"'; done +----------------+-------+ | Variable_name | Value | +----------------+-------+ | tmp_table_size | 1024 | +----------------+-------+ +----------------+-------+ | Variable_name | Value | +----------------+-------+ | tmp_table_size | 1024 | +----------------+-------+ +----------------+-------+ | Variable_name | Value | +----------------+-------+ | tmp_table_size | 1024 | +----------------+-------+ We got two tables: mysql> desc sort_id; +-------+---------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+---------+------+-----+---------+-------+ | id | int(11) | YES | | NULL | | +-------+---------+------+-----+---------+-------+ 1 row in set (0.00 sec) mysql> desc sort_1; +-------+---------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+---------+------+-----+---------+-------+ | id | int(11) | YES | | NULL | | | tea | text | YES | | NULL | | +-------+---------+------+-----+---------+-------+ 2 rows in set (0.00 sec) With 10000 Rows each. Tests: onyx:/var/tmp # for i in 6051 6053 6060; do mysqlslap -u erkan -P $i -h 10.255.128.4 --create-schema=sbtest -q "SELECT * from sbtest.sort_1 group by id;" --number-of-queries=20 -i 3; done Benchmark [[5.5.12]] Average number of seconds to run all queries: 1.803 seconds Minimum number of seconds to run all queries: 1.791 seconds Maximum number of seconds to run all queries: 1.826 seconds Number of clients running queries: 1 Average number of queries per client: 20 Benchmark [[MariaDB]] Average number of seconds to run all queries: 5.547 seconds Minimum number of seconds to run all queries: 5.542 seconds Maximum number of seconds to run all queries: 5.556 seconds Number of clients running queries: 1 Average number of queries per client: 20 Benchmark [[5.1.57]] Average number of seconds to run all queries: 1.605 seconds Minimum number of seconds to run all queries: 1.603 seconds Maximum number of seconds to run all queries: 1.606 seconds Number of clients running queries: 1 Average number of queries per client: 20 onyx:/var/tmp # for i in 6051 6053 6060; do mysqlslap -u erkan -P $i -h 10.255.128.4 --create-schema=sbtest -q "SELECT * from sbtest.sort_id group by id;" --number-of-queries=20 -i 3; done Benchmark [[5.5.12]] Average number of seconds to run all queries: 1.146 seconds Minimum number of seconds to run all queries: 1.135 seconds Maximum number of seconds to run all queries: 1.170 seconds Number of clients running queries: 1 Average number of queries per client: 20 Benchmark [[MariaDB]] Average number of seconds to run all queries: 4.359 seconds Minimum number of seconds to run all queries: 4.352 seconds Maximum number of seconds to run all queries: 4.366 seconds Number of clients running queries: 1 Average number of queries per client: 20 Benchmark [[5.1.57]] Average number of seconds to run all queries: 1.049 seconds Minimum number of seconds to run all queries: 1.032 seconds Maximum number of seconds to run all queries: 1.068 seconds Number of clients running queries: 1 Average number of queries per client: 20 So MariaDb doesn't perform in this test. So I wonder I did something wrong ... Regards Erkan -- über den grenzen muß die freiheit wohl wolkenlos sein

3 2

[Maria-developers] Reqeust for SHOW FULL FIELDS FROM and virtual tables in MariaDB
by Peter Laursen 06 Jun '11

06 Jun '11

Refer http://kb.askmonty.org/v/virtual-columns - the example CREATE TABLE table1 ( a INT NOT NULL, b VARCHAR(32), c INT AS (a MOD 10) virtual, d VARCHAR(5) AS (LEFT(b,5)) persistent); now SHOW FULL FIELDS FROM table1; Field Type Collation Null Key Default Extra Privileges Comment ------ ----------- ----------------- ------ ------ ------- ------- ------------------------------- ------- a int(11) (NULL) NO (NULL) select,insert,update,references b varchar(32) latin1_swedish_ci YES (NULL) select,insert,update,references c int(11) (NULL) YES (NULL) VIRTUAL select,insert,update,references d varchar(5) latin1_swedish_ci YES (NULL) VIRTUAL select,insert,update,references I request that 'extra' column should return either "PERSISTENT" or " VIRTUAL, PERSISTENT" for column `d`. The reason is that the output from SHOW FULL FIELDS returns an *array* (that will not need to be parsed) as opposite to SHOW CREATE TABLE. Accordingly SHOW FULL FIELDS is much easier to use for an 'unintelligent client' as opposed to an 'intelligent (or ignorant for that sake :-) ) human user to decide on special properties for a column. Peter Webyog

3 4

[Maria-developers] Buildbot failures: Microsecond-related changes break innodb_plugin but not xtradb
by Sergey Petrunya 06 Jun '11

06 Jun '11

Hi! All of the recent 5.3 builds in buildbot have two failures: innodb_plugin.innodb_bug54044 'innodb_plugin' innodb_plugin.innodb_information_schema 'innodb_plugin' The failure happens only with innodb_plugin. XtraDB is fine. I've investigated the first problem. Its cause is in storage/xtradb/handler/ha_innodb.cc:get_innobase_type_from_mysql_type(): - On 2010-08-04, Kristian did a merge from XtraDB in Percona-Server-5.1.47-11 and among everything else pulled in the code that will produce error when the function's argument is Field_null. This seems to be the bugfix for BUG#54044 - On 2011-03-29, Sergei was making fix for lp:743017 in 5.1-micro, switched the function from analyzing field->type() to analyzing field->key_type() and in the process removed the part of code that Kristian has added. For some reason, he did that storage/innodb_plugin and storage/innobase but not to storage/xtradb. - On May, 28th Monty has merged in Sergei's change into 5.3-main and we started to get the failure. I see two problems here: P1: (the apparent one) Sergei's new code doesn't produce error when passed a Field_null object. P2: (less apparent one) Why do we have Microsecond changes in innodb_plugin but not in xtradb? Could there be more fixes missing in a similar way? Sergei, could you please take a look? BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

4 4

Re: [Maria-developers] [Commits] Rev 2983: WL#77 - INSTALL PLUGIN * in http://bazaar.launchpad.net/~maria-captains/maria/5.2/
by Vladislav Vaintroub 03 Jun '11

03 Jun '11

> -----Original Message----- > From: commits-bounces(a)mariadb.org [mailto:commits- > bounces(a)mariadb.org] On Behalf Of serg(a)askmonty.org > Sent: Mittwoch, 1. Juni 2011 22:27 > To: commits(a)mariadb.org > Subject: [Commits] Rev 2983: WL#77 - INSTALL PLUGIN * in > http://bazaar.launchpad.net/~maria-captains/maria/5.2/ > > At http://bazaar.launchpad.net/~maria-captains/maria/5.2/ > > ------------------------------------------------------------ > revno: 2983 > revision-id: sergii(a)pisem.net-20110601202627-mcgxt0c3fgxoxxrq > parent: wlad(a)montyprogram.com-20110527170940-vl12r4mni30valsh > committer: Sergei Golubchik <sergii(a)pisem.net> > branch nick: 5.2 > timestamp: Wed 2011-06-01 22:26:27 +0200 > message: > WL#77 - INSTALL PLUGIN * > > new statements INSTALL PLUGIN * SONAME 'xxx' > and UNINSTALL PLUGIN * SONAME 'xxx' Serg, while simplifying "install plugin", do you think it could be even better to get rid of PLUGIN * SONAME altogether and just have INSTALL 'xxx' ? What do you think? Perhaps even more, make shared module/dynamic load library extension optional. It will simplify documentation greatly (so one would not have to mentally translate .so in examples to .dll) . It would also simplify test cases, we can get rid of obligatory --replace_regex /\.dll/.so/ and of $HA_EXAMPLE_SO. Wlad

2 1

[Maria-developers] Buildbot failure in filesort.cc:make_sortkey
by Sergey Petrunya 03 Jun '11

03 Jun '11

Hi Sergei, Current 5.3 fails as follows: http://buildbot.askmonty.org/buildbot/builders/mac-mini-x86-dbg/builds/771/… g++ -DMYSQL_SERVER -DDEFAULT_MYSQL_HOME='"/usr/local/mysql"' -DMYSQL_DATADIR='"/usr/local/mysql/var"' -DSHAREDIR='"/usr/local/mysql/share/mysql"' -DPLUGINDIR='"/usr/local/mysql/lib/mysql/plugin"' -DHAVE_EVENT_SCHEDULER -DHAVE_CONFIG_H -I. -I../include -I../include -I../include -I../regex -I. -I../extra/libevent -Wall -Wextra -Wunused -Wwrite-strings -Wno-strict-aliasing -Werror -DFORCE_INIT_OF_VARS -Wno-unused-parameter -g -Wall -Wextra -Wunused -Wwrite-strings -Wno-uninitialized -Wno-unused-parameter -Wnon-virtual-dtor -felide-constructors -fno-exceptions -fno-rtti -mtune=native -DUNIV_MUST_NOT_INLINE -DEXTRA_DEBUG -DSAFEMALLOC -DPEDANTIC_SAFEMALLOC -O0 -g3 -gdwarf-2 -O2 -Wuninitialized -DFORCE_INIT_OF_VARS -fno-implicit-templates -fno-exceptions -fno-rtti -D_P1003_1B_VISIBLE -DSIGNAL_WITH_VIO_CLOSE -DSIGNALS_DONT_BREAK_READ -DIGNORE_SIGHUP_SIGQUIT -DDONT_DECLARE_CXA_PURE_VIRTUAL -MT debug_sync.o -MD -MP -MF .deps/debug_sync.Tpo -c -o debug_sync.o debug_sync.cc cc1plus: warnings being treated as errors filesort.cc: In function 'void make_sortkey(SORTPARAM*, uchar*, uchar*)': filesort.cc:878: warning: 'value' may be used uninitialized in this function make[3]: *** [filesort.o] Error 1 make[3]: *** Waiting for unfinished jobs.... It is not apparent for me whether this is a real error or not, bazaar history shows that the failure is in your code of MWL#173. Could you please take a look? BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] FYI: MWL#90 pushed into 5.3 main
by Sergey Petrunya 31 May '11

31 May '11

Hello, This is to inform you that today I've pushed MWL#90 into 5.3-main. There was a buildbot run on 5.3-subqueries-mwl90 tree before the push, and it did not show any failures that weren't also present in 5.3-main. The result of my merge has an issue that conceptually it is not a full merge, because MWL#90 code is not able yet to take advantage of the possibily of dynamic choice between IN->EXISTS (which was introduced in MWL#89 which Timour pushed two weeks ago). Adding support for MWL#89 and MWL#90 working together seems easy. Actually, I initially intended to do it while merging, but then decided to do one thing at a time. My further plans are: - look at the open bugs I have - look at making MWL#89 and MWL#90 work together. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] how do I count errors per database user?
by MARK CALLAGHAN 26 May '11

26 May '11

I want to count errors by database user for IS.user_statistics. I have code that does it in my_message_sql but hfisk thinks the counts might be inflated. I don't understand the error reporting framework in MySQL so it is possible that the counts are wrong or count errors and warnings rather than only counting errors. The code below is from my_message_sql and shows where I try to maintain the count. The assignment (thd->is_slave_error= 1) makes me think that an error has occurred by the time I count it. But code that follows has comments about clearing an error or converting an error to a warning. if ((thd= current_thd)) { /* TODO: There are two exceptions mechanism (THD and sp_rcontext), this could be improved by having a common stack of handlers. */ if (thd->handle_error(error, str, MYSQL_ERROR::WARN_LEVEL_ERROR)) DBUG_RETURN(0); { USER_STATS *us= thd_get_user_stats(thd); --------------------------------> added by me my_atomic_add_bigint(&(us->errors_total), 1); --------------------------------> added by me } thd->is_slave_error= 1; // needed to catch query errors during replication /* thd->lex->current_select == 0 if lex structure is not inited (not query command (COM_QUERY)) */ if (thd->lex->current_select && thd->lex->current_select->no_error && !thd->is_fatal_error) { DBUG_PRINT("error", ("Error converted to warning: current_select: no_error %d " "fatal_error: %d", (thd->lex->current_select ? thd->lex->current_select->no_error : 0), (int) thd->is_fatal_error)); } else { if (! thd->main_da.is_error()) // Return only first message { thd->main_da.set_error_status(thd, error, str); } query_cache_abort(&thd->net); } /* If a continue handler is found, the error message will be cleared by the stored procedures code. */ if (thd->spcont && ! (MyFlags & ME_NO_SP_HANDLER) && thd->spcont->handle_error(error, MYSQL_ERROR::WARN_LEVEL_ERROR, thd)) { /* Do not push any warnings, a handled error must be completely silenced. */ DBUG_RETURN(0); } /* When simulating OOM, skip writing to error log to avoid mtr errors */ DBUG_EXECUTE_IF("simulate_out_of_memory", DBUG_RETURN(0);); if (!thd->no_warnings_for_error && !(MyFlags & ME_NO_WARNING_FOR_ERROR)) { /* Suppress infinite recursion if there a memory allocation error inside push_warning. */ thd->no_warnings_for_error= TRUE; push_warning(thd, MYSQL_ERROR::WARN_LEVEL_ERROR, error, str); thd->no_warnings_for_error= FALSE; } } /* When simulating OOM, skip writing to error log to avoid mtr errors */ DBUG_EXECUTE_IF("simulate_out_of_memory", DBUG_RETURN(0);); if (!thd || MyFlags & ME_NOREFRESH) sql_print_error("%s: %s",my_progname,str); /* purecov: inspected */ DBUG_RETURN(0); } -- Mark Callaghan mdcallag(a)gmail.com

2 1

[Maria-developers] More MWL#89 questions
by Sergey Petrunya 25 May '11

25 May '11

Hi Timour, While merging, I also got the following questions: == exclude_expensive_cond == make_cond_for_table()'s exclude_expensive_cond parameter is not used anymore. Was it intentional that you kept it in the code? (minimizing MWL#89's diff size could not be a reason because the diff changes make_cond_for_table signature anyway) == exec_const_cond == The patch introduces JOIN::exec_const_cond. On the other hand, the code already had JOIN::outer_ref_cond which seems to be nearly the same. == join_tab_idx == make_join_select() attaches parts of WHERE/ON clauses to JOIN_TABs. It does so with a help of two functions: 1. make_cond_for_table() 2. make_cond_after_sjm() The first one does make set_join_tab_idx() calls, the second one doesn't. Why? == join_tab_idx #2 === > @@ -7150,24 +7174,27 @@ > there inside the triggers. > */ > { // Check const tables > + join->exec_const_cond= > + make_cond_for_table(thd, cond, > join->const_table_map, > - (table_map) 0, TRUE, FALSE); > + (table_map) 0, MAX_TABLES, FALSE, FALSE); As far as I understand the code, the MAX_TABLES part informs the constant subquery predicate that it will be evaluated O(join_output_records) times, which is not true. This is not the only such case, I see plenty of make_cond_for_table() calls which use MAX_TABLES for join_tab_idx argument, where we can actually tell for certain that the condition will be evaluated much fewer than O(join_output_records) times. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] [Commits] Rev 2983: MWL#89 - Automatic merge with 5.3 in file:///home/tsk/mprog/src/5.3-mwl89-merge/
by Sergey Petrunya 23 May '11

23 May '11

Hi Timour, Congratulations on having pushed MWL#89 into 5.3-main! This is quite an achievement which really moves us forward. I've been studying the new code and has got some feedback which I thought I need to share. Please find the notes below, marked with 'psergey'. > diff -urN --exclude='.*' 5.3-timours-wl-clean/sql/filesort.cc 5.3-timours-wl/sql/filesort.cc > --- 5.3-timours-wl-clean/sql/filesort.cc 2011-05-11 20:12:50.000000000 +0100 > +++ 5.3-timours-wl/sql/filesort.cc 2011-05-11 20:08:50.000000000 +0100 > @@ -612,10 +612,34 @@ > } > DBUG_RETURN(HA_POS_ERROR); /* purecov: inspected */ > } > + > + bool write_record= false; > if (error == 0) > + { > param->examined_rows++; > - > - if (error == 0 && (!select || select->skip_record(thd) > 0)) > + if (select && select->cond) > + { > + /* > + If the condition 'select->cond' contains a subquery, restore the > + original read/write sets of the table 'sort_form' because when > + SQL_SELECT::skip_record evaluates this condition. it may include a psergey: - The last sentence doesn't parse, please fix. - Why is the check done *for each record we enumerate*? Please move it outside of the loop. - I suspect that now, with Sanja's subquery code, each subquery has a list of references it makes to outside, so this shortcut is not necessary (let's discuss that on the next optimizer call) > + correlated subquery predicate, such that some field in the subquery > + refers to 'sort_form'. > + */ > + if (select->cond->with_subselect) > + sort_form->column_bitmaps_set(save_read_set, save_write_set, > + save_vcol_set); > + write_record= (select->skip_record(thd) > 0); > + if (select->cond->with_subselect) > + sort_form->column_bitmaps_set(&sort_form->tmp_set, > + &sort_form->tmp_set, > + &sort_form->tmp_set); > + } > + else > + write_record= true; > + } > + > + if (write_record) > { > if (idx == param->keys) > { > diff -urN --exclude='.*' 5.3-timours-wl-clean/sql/item_cmpfunc.cc 5.3-timours-wl/sql/item_cmpfunc.cc > --- 5.3-timours-wl-clean/sql/item_cmpfunc.cc 2011-05-11 20:12:50.000000000 +0100 > +++ 5.3-timours-wl/sql/item_cmpfunc.cc 2011-05-11 20:08:50.000000000 +0100 > @@ -2072,6 +2072,18 @@ > } > > > +bool Item_in_optimizer::is_expensive_processor(uchar *arg) > +{ > + return args[1]->is_expensive_processor(arg); > +} > + > + > +bool Item_in_optimizer::is_expensive() > +{ > + return args[1]->is_expensive(); > +} > + psergey: what about the cases when args[0] is expensive? It should either be checked, or a comment here is needed with explanation why we don't need to check it. > longlong Item_func_eq::val_int() > { > DBUG_ASSERT(fixed == 1); > diff -urN --exclude='.*' 5.3-timours-wl-clean/sql/item.h 5.3-timours-wl/sql/item.h > --- 5.3-timours-wl-clean/sql/item.h 2011-05-11 20:12:50.000000000 +0100 > +++ 5.3-timours-wl/sql/item.h 2011-05-11 20:08:50.000000000 +0100 > @@ -510,7 +521,7 @@ > SUBSELECT_ITEM, ROW_ITEM, CACHE_ITEM, TYPE_HOLDER, > PARAM_ITEM, TRIGGER_FIELD_ITEM, DECIMAL_ITEM, > XPATH_NODESET, XPATH_NODESET_CMP, > - VIEW_FIXER_ITEM, EXPR_CACHE_ITEM}; > + VIEW_FIXER_ITEM, EXPR_CACHE_ITEM, UNKNOWN_ITEM}; psergey: When I grep for UNKNOWN_ITEM, I find only this occurence. Why add it? > > enum cond_result { COND_UNDEF,COND_OK,COND_TRUE,COND_FALSE }; > ... > diff -urN --exclude='.*' 5.3-timours-wl-clean/sql/item_subselect.cc 5.3-timours-wl/sql/item_subselect.cc > --- 5.3-timours-wl-clean/sql/item_subselect.cc 2011-05-11 20:12:50.000000000 +0100 > +++ 5.3-timours-wl/sql/item_subselect.cc 2011-05-11 20:08:50.000000000 +0100 ... > @@ -1659,49 +1696,40 @@ > { > if (!(new_having= new Item_func_trig_cond(new_having, > get_cond_guard(0)))) > - DBUG_RETURN(RES_ERROR); > + DBUG_RETURN(true); > } > - new_having->name= (char*)in_having_cond; > - select_lex->having= join->having= new_having; > - select_lex->having_fix_field= 1; > - > - /* > - we do not check join->having->fixed, because comparison function > - (from func->create) can't be fixed after creation > - */ > - tmp= join->having->fix_fields(thd, 0); > - select_lex->having_fix_field= 0; > - if (tmp) > - DBUG_RETURN(RES_ERROR); > + > + new_having->name= (char*) in_having_cond; > + if (fix_having(new_having, select_lex)) > + DBUG_RETURN(true); > + *having_item= new_having; > } > else > - { > - // it is single select without tables => possible optimization > - // remove the dependence mark since the item is moved to upper > - // select and is not outer anymore. > - item->walk(&Item::remove_dependence_processor, 0, > - (uchar *) select_lex->outer_select()); > - item= func->create(left_expr, item); > - // fix_field of item will be done in time of substituting > - substitution= item; > - have_to_be_excluded= 1; > - if (thd->lex->describe) > - { > - char warn_buff[MYSQL_ERRMSG_SIZE]; > - sprintf(warn_buff, ER(ER_SELECT_REDUCED), select_lex->select_number); > - push_warning(thd, MYSQL_ERROR::WARN_LEVEL_NOTE, > - ER_SELECT_REDUCED, warn_buff); > - } > - DBUG_RETURN(RES_REDUCE); > - } > + DBUG_ASSERT(FALSE); psergey: (one) > } > } > > - DBUG_RETURN(RES_OK); > + DBUG_RETURN(false); psergey: (two). I think using both 'false' and 'FALSE' in the same patch is too much. Please change to TRUE/FALSE since we have it everywhere else. > } > > > -Item_subselect::trans_res > +/** > + Wrap a multi-column IN/ALL/ANY subselect into an Item_in_optimizer. > + > + @param join Join object of the subquery (i.e. 'child' join). > + > + @details > + The subquery predicate is wrapped into an Item_in_optimizer. Later the query > + optimization phase chooses whether the subquery under the Item_in_optimizer > + will be further transformed into an equivalent correlated EXISTS by injecting > + additional predicates, or will be executed via subquery materialization in its > + unmodified form. > + > + @retval false The subquery was transformed > + @retval true Error > +*/ > + > +bool > Item_in_subselect::row_value_transformer(JOIN *join) > { > SELECT_LEX *select_lex= join->select_lex; ... > diff -urN --exclude='.*' 5.3-timours-wl-clean/sql/opt_subselect.cc 5.3-timours-wl/sql/opt_subselect.cc > --- 5.3-timours-wl-clean/sql/opt_subselect.cc 2011-05-11 20:12:50.000000000 +0100 > +++ 5.3-timours-wl/sql/opt_subselect.cc 2011-05-11 20:08:50.000000000 +0100 > @@ -175,63 +189,80 @@ > else > { > DBUG_PRINT("info", ("Subquery can't be converted to semi-join")); > - /* > - Check if the subquery predicate can be executed via materialization. > - The required conditions are: > - 1. Subquery predicate is an IN/=ANY subq predicate > - 2. Subquery is a single SELECT (not a UNION) > - 3. Subquery is not a table-less query. In this case there is no > - point in materializing. > - 3A The upper query is not a table-less SELECT ... FROM DUAL. We > + /* Test if the user has set a legal combination of optimizer switches. */ > + if (!optimizer_flag(thd, OPTIMIZER_SWITCH_IN_TO_EXISTS) && > + !optimizer_flag(thd, OPTIMIZER_SWITCH_MATERIALIZATION)) > + my_error(ER_ILLEGAL_SUBQUERY_OPTIMIZER_SWITCHES, MYF(0)); > + > + if (in_subs) > + { > + /* Subquery predicate is an IN/=ANY predicate. */ > + if (optimizer_flag(thd, OPTIMIZER_SWITCH_IN_TO_EXISTS)) > + in_subs->in_strategy|= SUBS_IN_TO_EXISTS; > + if (optimizer_flag(thd, OPTIMIZER_SWITCH_MATERIALIZATION)) > + in_subs->in_strategy|= SUBS_MATERIALIZATION; > + > + /* > + Check if the subquery predicate can be executed via materialization. > + The required conditions are: > + 1. Subquery is a single SELECT (not a UNION) > + 2. Subquery is not a table-less query. In this case there is no > + point in materializing. > + 2A The upper query is not a table-less SELECT ... FROM DUAL. We > can't do materialization for SELECT .. FROM DUAL because it > does not call setup_subquery_materialization(). We could make > SELECT ... FROM DUAL call that function but that doesn't seem > to be the case that is worth handling. > - 4. Either the subquery predicate is a top-level predicate, or at > - least one partial match strategy is enabled. If no partial match > - strategy is enabled, then materialization cannot be used for > - non-top-level queries because it cannot handle NULLs correctly. > - 5. Subquery is non-correlated > - TODO: > - This is an overly restrictive condition. It can be extended to: > - (Subquery is non-correlated || > - Subquery is correlated to any query outer to IN predicate || > - (Subquery is correlated to the immediate outer query && > - Subquery !contains {GROUP BY, ORDER BY [LIMIT], > - aggregate functions}) && subquery predicate is not under "NOT IN")) > - 6. No execution method was already chosen (by a prepared statement). > - > - (*) The subquery must be part of a SELECT statement. The current > - condition also excludes multi-table update statements. > - > - Determine whether we will perform subquery materialization before > - calling the IN=>EXISTS transformation, so that we know whether to > - perform the whole transformation or only that part of it which wraps > - Item_in_subselect in an Item_in_optimizer. > - */ > - if (optimizer_flag(thd, OPTIMIZER_SWITCH_MATERIALIZATION) && > - in_subs && // 1 > - !select_lex->is_part_of_union() && // 2 > - select_lex->master_unit()->first_select()->leaf_tables && // 3 > - thd->lex->sql_command == SQLCOM_SELECT && // * > - select_lex->outer_select()->leaf_tables && // 3A > - subquery_types_allow_materialization(in_subs) && > - // psergey-todo: duplicated_subselect_card_check: where it's done? > - (in_subs->is_top_level_item() || > - optimizer_flag(thd, OPTIMIZER_SWITCH_PARTIAL_MATCH_ROWID_MERGE) || > - optimizer_flag(thd, OPTIMIZER_SWITCH_PARTIAL_MATCH_TABLE_SCAN)) &&//4 > - !in_subs->is_correlated && // 5 > - in_subs->exec_method == Item_in_subselect::NOT_TRANSFORMED) // 6 > - { > - in_subs->exec_method= Item_in_subselect::MATERIALIZATION; > - } > - > - Item_subselect::trans_res trans_res; > - if ((trans_res= subselect->select_transformer(join)) != > - Item_subselect::RES_OK) > - { > - DBUG_RETURN((trans_res == Item_subselect::RES_ERROR)); > + 3. Either the subquery predicate is a top-level predicate, or at > + least one partial match strategy is enabled. If no partial match > + strategy is enabled, then materialization cannot be used for > + non-top-level queries because it cannot handle NULLs correctly. > + 4. Subquery is non-correlated > + TODO: > + This is an overly restrictive condition. It can be extended to: > + (Subquery is non-correlated || > + Subquery is correlated to any query outer to IN predicate || > + (Subquery is correlated to the immediate outer query && > + Subquery !contains {GROUP BY, ORDER BY [LIMIT], > + aggregate functions}) && subquery predicate is not under "NOT IN")) > + > + (*) The subquery must be part of a SELECT statement. The current > + condition also excludes multi-table update statements. > + */ > + if (!(in_subs->in_strategy & SUBS_MATERIALIZATION && > + !select_lex->is_part_of_union() && // 1 > + parent_unit->first_select()->leaf_tables && // 2 > + thd->lex->sql_command == SQLCOM_SELECT && // * > + select_lex->outer_select()->leaf_tables && // 2A > + subquery_types_allow_materialization(in_subs) && > + // psergey-todo: duplicated_subselect_card_check: where it's done? > + (in_subs->is_top_level_item() || //3 > + optimizer_flag(thd, > + OPTIMIZER_SWITCH_PARTIAL_MATCH_ROWID_MERGE) || //3 > + optimizer_flag(thd, > + OPTIMIZER_SWITCH_PARTIAL_MATCH_TABLE_SCAN)) && //3 > + !in_subs->is_correlated)) //4 > + { > + /* Materialization is not possible based on syntactic properties. */ > + in_subs->in_strategy&= ~SUBS_MATERIALIZATION; > + } > + > + if (!in_subs->in_strategy) > + { > + /* > + If neither materialization is possible, nor the user chose > + IN-TO-EXISTS, choose IN-TO-EXISTS as the only universal strategy. > + */ > + in_subs->in_strategy|= SUBS_IN_TO_EXISTS; > + } > } psergey: The above seems unneccessarily convoluted. This is not apprent from the patch, but the resulting code looks like this: if (in_subs) { /* Subquery predicate is an IN/=ANY predicate. */ if (optimizer_flag(thd, OPTIMIZER_SWITCH_IN_TO_EXISTS)) in_subs->in_strategy|= SUBS_IN_TO_EXISTS; if (optimizer_flag(thd, OPTIMIZER_SWITCH_MATERIALIZATION)) in_subs->in_strategy|= SUBS_MATERIALIZATION; /* a long comment */ if (!(long condition)) { /* Materialization is not possible based on syntactic properties. */ in_subs->in_strategy&= ~SUBS_MATERIALIZATION; } ... Is there any particular reason to first set SUBS_MATERIALIZATION flag, and then optionally clear it? It seems it would be more straightforward to only set it when materialization is really applicable? > + > + /* > + Transform each subquery predicate according to its overloaded > + transformer. > + */ > + if (subselect->select_transformer(join)) > + DBUG_RETURN(-11); > } > } > DBUG_RETURN(0); ... > @@ -1830,15 +1919,15 @@ > - sj_inner_fanout*sj_outer_fanout lookups. > > */ > - double one_lookup_cost; > - if (sj_outer_fanout*temptable_rec_size > > - join->thd->variables.max_heap_table_size) > - one_lookup_cost= DISK_TEMPTABLE_LOOKUP_COST; > - else > - one_lookup_cost= HEAP_TEMPTABLE_LOOKUP_COST; > + double one_lookup_cost= get_tmp_table_lookup_cost(join->thd, > + sj_outer_fanout, > + temptable_rec_size); > + double one_write_cost= get_tmp_table_write_cost(join->thd, > + sj_outer_fanout, > + temptable_rec_size); > > double write_cost= join->positions[first_tab].prefix_record_count* > - sj_outer_fanout * one_lookup_cost; > + sj_outer_fanout * one_write_cost; > double full_lookup_cost= join->positions[first_tab].prefix_record_count* > sj_outer_fanout* sj_inner_fanout * > one_lookup_cost; > @@ -3360,9 +3449,23 @@ > JOIN_TAB* join_tab=join->join_tab; > SELECT_LEX_UNIT *unit= join->unit; > DBUG_ENTER("rewrite_to_index_subquery_engine"); > + > /* > is this simple IN subquery? > */ > + /* TODO: In order to use these more efficient subquery engines in more cases, > + the following problems need to be solved: > + - the code that removes GROUP BY (group_list), also adds an ORDER BY > + (order), thus GROUP BY queries (almost?) never pass through this branch. > + Solution: remove the test below '!join->order', because we remove the > + ORDER clase for subqueries anyway. > + - in order to set a more efficient engine, the optimizer needs to both > + decide to remove GROUP BY, *and* select one of the JT_[EQ_]REF[_OR_NULL] > + access methods, *and* loose scan should be more expensive or > + inapliccable. When is that possible? > + - Consider expanding the applicability of this rewrite for loose scan > + for group by queries. > + */ > if (!join->group_list && !join->order && > join->unit->item && > join->unit->item->substype() == Item_subselect::IN_SUBS && > @@ -3503,3 +3606,349 @@ > } > > > +/** > + Optimize all subqueries of a query that have were flattened into a semijoin. psergey: a typo on the above line? > + > + @details > + Optimize all immediate children subqueries of a query. > + > + This phase must be called after substitute_for_best_equal_field() because > + that function may replace items with other items from a multiple equality, > + and we need to reference the correct items in the index access method of the > + IN predicate. > + > + @return Operation status > + @retval FALSE success. > + @retval TRUE error occurred. > +*/ > + > +bool JOIN::optimize_unflattened_subqueries() > +{ > + return select_lex->optimize_unflattened_subqueries(); > +} > + > + > +/** > + Choose an optimal strategy to execute an IN/ALL/ANY subquery predicate > + based on cost. > + > + @param join_tables the set of tables joined in the subquery > + > + @notes > + The method chooses between the materialization and IN=>EXISTS rewrite > + strategies for the execution of a non-flattened subquery IN predicate. > + The cost-based decision is made as follows: > + > + 1. compute materialize_strategy_cost based on the unmodified subquery > + 2. reoptimize the subquery taking into account the IN-EXISTS predicates > + 3. compute in_exists_strategy_cost based on the reoptimized plan > + 4. compare and set the cheaper strategy > + if (materialize_strategy_cost >= in_exists_strategy_cost) > + in_strategy = MATERIALIZATION > + else > + in_strategy = IN_TO_EXISTS > + 5. if in_strategy = MATERIALIZATION and it is not possible to initialize it > + revert to IN_TO_EXISTS > + 6. if (in_strategy == MATERIALIZATION) > + revert the subquery plan to the original one before reoptimizing > + else > + inject the IN=>EXISTS predicates into the new EXISTS subquery plan > + > + The implementation itself is a bit more complicated because it takes into > + account two more factors: > + - whether the user allowed both strategies through an optimizer_switch, and > + - if materialization was the cheaper strategy, whether it can be executed > + or not. > + > + @retval FALSE success. > + @retval TRUE error occurred. > +*/ > + > +bool JOIN::choose_subquery_plan(table_map join_tables) > +{ > + Query_plan_state save_qep; /* The original QEP of the subquery. */ > + enum_reopt_result reopt_result= REOPT_NONE; > + Item_in_subselect *in_subs; > + > + if (select_lex->master_unit()->item && > + select_lex->master_unit()->item->is_in_predicate()) > + { > + in_subs= (Item_in_subselect*) select_lex->master_unit()->item; > + if (in_subs->create_in_to_exists_cond(this)) > + return true; > + } > + else > + return false; > + > + DBUG_ASSERT(in_subs->in_strategy); /* A strategy must be chosen earlier. */ > + DBUG_ASSERT(in_to_exists_where || in_to_exists_having); > + DBUG_ASSERT(!in_to_exists_where || in_to_exists_where->fixed); > + DBUG_ASSERT(!in_to_exists_having || in_to_exists_having->fixed); > + > + /* > + Compute and compare the costs of materialization and in-exists if both > + strategies are possible and allowed by the user (checked during the prepare > + phase. > + */ > + if (in_subs->in_strategy & SUBS_MATERIALIZATION && > + in_subs->in_strategy & SUBS_IN_TO_EXISTS) psergey: when I write conditions like the one above, Monty tells me to change it to in_subs->in_strategy & (SUBS_MATERIALIZATION |SUBS_IN_TO_EXISTS) Did he not tell the same to you? > + { > + JOIN *outer_join; > + JOIN *inner_join= this; > + /* Number of (partial) rows of the outer JOIN filtered by the IN predicate. */ > + double outer_record_count; > + /* Number of unique value combinations filtered by the IN predicate. */ > + double outer_lookup_keys; > + /* Cost and row count of the unmodified subquery. */ > + double inner_read_time_1, inner_record_count_1; > + /* Cost of the subquery with injected IN-EXISTS predicates. */ > + double inner_read_time_2; > + /* The cost to compute IN via materialization. */ > + double materialize_strategy_cost; > + /* The cost of the IN->EXISTS strategy. */ > + double in_exists_strategy_cost; > + double dummy; > + > + /* > + A. Estimate the number of rows of the outer table that will be filtered > + by the IN predicate. > + */ > + outer_join= unit->outer_select() ? unit->outer_select()->join : NULL; > + if (outer_join) > + { > + uint outer_partial_plan_len; > + /* > + Make_cond_for_table is called for predicates only in the WHERE/ON > + clauses. In all other cases, predicates are not pushed to any > + JOIN_TAB, and their joi_tab_idx remains MAX_TABLES. Such predicates > + are evaluated for each complete row of the outer join. > + */ > + outer_partial_plan_len= (in_subs->get_join_tab_idx() == MAX_TABLES) ? > + outer_join->tables : > + in_subs->get_join_tab_idx() + 1; > + outer_join->get_partial_join_cost(outer_partial_plan_len, &dummy, > + &outer_record_count); > + if (outer_join->tables > outer_join->const_tables) > + outer_lookup_keys= prev_record_reads(outer_join->best_positions, > + outer_partial_plan_len, > + in_subs->used_tables()); > + else > + { > + /* If all tables are constant, positions is undefined. */ > + outer_lookup_keys= 1; > + } > + } > + else > + { > + /* > + TODO: outer_join can be NULL for DELETE statements. > + How to compute its cost? > + */ > + outer_record_count= 1; > + outer_lookup_keys=1; > + } > + /* > + There cannot be more lookup keys than the total number of records. > + TODO: this a temporary solution until we find a better way to compute > + get_partial_join_cost() and prev_record_reads() in a consitent manner, > + where it is guaranteed that (outer_lookup_keys <= outer_record_count). > + */ > + if (outer_lookup_keys > outer_record_count) > + outer_lookup_keys= outer_record_count; > + > + /* > + B. Estimate the cost and number of records of the subquery both > + unmodified, and with injected IN->EXISTS predicates. > + */ > + inner_read_time_1= inner_join->best_read; > + inner_record_count_1= inner_join->record_count; > + > + if (in_to_exists_where && const_tables != tables) > + { > + /* > + Re-optimize and cost the subquery taking into account the IN-EXISTS > + conditions. > + */ > + reopt_result= reoptimize(in_to_exists_where, join_tables, &save_qep); > + if (reopt_result == REOPT_ERROR) > + return TRUE; > + > + /* Get the cost of the modified IN-EXISTS plan. */ > + inner_read_time_2= inner_join->best_read; > + > + } > + else > + { > + /* Reoptimization would not produce any better plan. */ > + inner_read_time_2= inner_read_time_1; > + } > + > + /* > + C. Compute execution costs. > + */ > + /* C.1 Compute the cost of the materialization strategy. */ > + uint rowlen= get_tmp_table_rec_length(unit->first_select()->item_list); > + /* The cost of writing one row into the temporary table. */ > + double write_cost= get_tmp_table_write_cost(thd, inner_record_count_1, > + rowlen); > + /* The cost of a lookup into the unique index of the materialized table. */ > + double lookup_cost= get_tmp_table_lookup_cost(thd, inner_record_count_1, > + rowlen); > + /* > + The cost of executing the subquery and storing its result in an indexed > + temporary table. > + */ > + double materialization_cost= inner_read_time_1 + > + write_cost * inner_record_count_1; > + > + materialize_strategy_cost= materialization_cost + > + outer_record_count * lookup_cost; > + > + /* C.2 Compute the cost of the IN=>EXISTS strategy. */ > + in_exists_strategy_cost= outer_lookup_keys * inner_read_time_2; > + > + /* C.3 Compare the costs and choose the cheaper strategy. */ > + if (materialize_strategy_cost >= in_exists_strategy_cost) > + in_subs->in_strategy&= ~SUBS_MATERIALIZATION; > + else > + in_subs->in_strategy&= ~SUBS_IN_TO_EXISTS; > + } > + > + /* > + If (1) materialization is a possible strategy based on semantic analysis > + during the prepare phase, then if > + (2) it is more expensive than the IN->EXISTS transformation, and > + (3) it is not possible to create usable indexes for the materialization > + strategy, > + fall back to IN->EXISTS. > + otherwise > + use materialization. > + */ > + if (in_subs->in_strategy & SUBS_MATERIALIZATION && > + in_subs->setup_mat_engine()) > + { > + /* > + If materialization was the cheaper or the only user-selected strategy, > + but it is not possible to execute it due to limitations in the > + implementation, fall back to IN-TO-EXISTS. > + */ > + in_subs->in_strategy&= ~SUBS_MATERIALIZATION; > + in_subs->in_strategy|= SUBS_IN_TO_EXISTS; > + } > + > + if (in_subs->in_strategy & SUBS_MATERIALIZATION) > + { > + /* Restore the original query plan used for materialization. */ > + if (reopt_result == REOPT_NEW_PLAN) > + restore_query_plan(&save_qep); > + > + /* TODO: should we set/unset this flag for both select_lex and its unit? */ > + in_subs->unit->uncacheable&= ~UNCACHEABLE_DEPENDENT; > + select_lex->uncacheable&= ~UNCACHEABLE_DEPENDENT; > + > + /* > + Reset the "LIMIT 1" set in Item_exists_subselect::fix_length_and_dec. > + TODO: > + Currently we set the subquery LIMIT to infinity, and this is correct > + because we forbid at parse time LIMIT inside IN subqueries (see > + Item_in_subselect::test_limit). However, once we allow this, here > + we should set the correct limit if given in the query. > + */ > + in_subs->unit->global_parameters->select_limit= NULL; > + in_subs->unit->set_limit(unit->global_parameters); > + /* > + Set the limit of this JOIN object as well, because normally its being > + set in the beginning of JOIN::optimize, which was already done. > + */ > + select_limit= in_subs->unit->select_limit_cnt; > + } > + else if (in_subs->in_strategy & SUBS_IN_TO_EXISTS) > + { > + if (reopt_result == REOPT_NONE && in_to_exists_where && > + const_tables != tables) > + { > + /* > + The subquery was not reoptimized either because the user allowed only the > + IN-EXISTS strategy, or because materialization was not possible based on > + semantic analysis. Clenup the original plan and reoptimize. > + */ > + for (uint i= 0; i < tables; i++) > + { > + join_tab[i].keyuse= NULL; > + join_tab[i].checked_keys.clear_all(); > + } > + if ((reopt_result= reoptimize(in_to_exists_where, join_tables, NULL)) == > + REOPT_ERROR) > + return TRUE; > + } > + > + if (in_subs->inject_in_to_exists_cond(this)) > + return TRUE; > + } > + else > + DBUG_ASSERT(FALSE); > + > + return FALSE; > +} > + > + > +/** > + Choose a query plan for a table-less subquery. > + > + @notes > + > + @retval FALSE success. > + @retval TRUE error occurred. > +*/ > + > +bool JOIN::choose_tableless_subquery_plan() > +{ > + DBUG_ASSERT(!tables_list || !tables); > + if (select_lex->master_unit()->item) > + { > + DBUG_ASSERT(select_lex->master_unit()->item->type() == > + Item::SUBSELECT_ITEM); > + Item_subselect *subs_predicate= select_lex->master_unit()->item; > + > + /* > + If the optimizer determined that his query has an empty result, > + in most cases the subquery predicate is a known constant value - > + either FALSE or NULL. The implementation of Item_subselect::reset() > + determines which one. > + */ > + if (zero_result_cause) > + { > + if (!implicit_grouping) > + { > + /* > + Both group by queries and non-group by queries without aggregate > + functions produce empty subquery result. > + */ > + subs_predicate->reset(); > + subs_predicate->make_const(); > + return FALSE; > + } > + > + /* TODO: > + A further optimization is possible when a non-group query with > + MIN/MAX/COUNT is optimized by opt_sum_query. Then, if there are > + only MIN/MAX functions over an empty result set, the subquery > + result is a NULL value/row, thus the value of subs_predicate is > + NULL. > + */ > + } > + > + if (subs_predicate->is_in_predicate()) > + { > + Item_in_subselect *in_subs; > + in_subs= (Item_in_subselect*) subs_predicate; > + in_subs->in_strategy= SUBS_IN_TO_EXISTS; > + if (in_subs->create_in_to_exists_cond(this) || > + in_subs->inject_in_to_exists_cond(this)) > + return TRUE; > + tmp_having= having; > + } > + } > + return FALSE; > +} > + > diff -urN --exclude='.*' 5.3-timours-wl-clean/sql/sql_lex.cc 5.3-timours-wl/sql/sql_lex.cc > --- 5.3-timours-wl-clean/sql/sql_lex.cc 2011-05-11 20:12:50.000000000 +0100 > +++ 5.3-timours-wl/sql/sql_lex.cc 2011-05-11 20:08:50.000000000 +0100 > @@ -1684,6 +1684,31 @@ > slave= 0; > } > > + > +void st_select_lex_node::add_slave(st_select_lex_node *slave_arg) > +{ > + for (; slave; slave= slave->next) > + if (slave == slave_arg) > + return; > + > + if (slave) > + { > + st_select_lex_node *slave_arg_slave= slave_arg->slave; > + /* Insert in the front of list of slaves if any. */ > + slave_arg->include_neighbour(slave); > + /* include_neighbour() sets slave_arg->slave=0, restore it. */ > + slave_arg->slave= slave_arg_slave; > + /* Count on include_neighbour() setting the master. */ > + DBUG_ASSERT(slave_arg->master == this); > + } > + else > + { > + slave= slave_arg; > + slave_arg->master= this; > + } > +} psergey: 1. Suppose slave_arg is already in the list of slaves (based on the function's code, this seems to be possible) Why does this function adjust this->slave in this case? 2. I've made the following change: === modified file 'sql/sql_lex.cc' --- sql/sql_lex.cc 2011-05-05 12:24:28 +0000 +++ sql/sql_lex.cc 2011-05-13 21:34:49 +0000 @@ -1693,6 +1693,7 @@ void st_select_lex_node::add_slave(st_se if (slave) { + DBUG_ASSERT(0); st_select_lex_node *slave_arg_slave= slave_arg->slave; /* Insert in the front of list of slaves if any. */ slave_arg->include_neighbour(slave); and all t/subselect*.test still passed. I think the code inside if () { ... } is deadcode. If yes, please remove it, if not, please add coverage to the testsuite. > + > + > /* > include on level down (but do not link) > ... > diff -urN --exclude='.*' 5.3-timours-wl-clean/sql/sql_list.h 5.3-timours-wl/sql/sql_list.h > --- 5.3-timours-wl-clean/sql/sql_list.h 2011-05-11 20:12:50.000000000 +0100 > +++ 5.3-timours-wl/sql/sql_list.h 2011-05-11 20:08:50.000000000 +0100 > @@ -260,7 +260,7 @@ > list_node *node= first; > list_node *list_first= list->first; > elements=0; > - while (node && node != list_first) > + while (node->info && node != list_first) psergey: Why the above change? Does this mean that List<T> now can contain (T*)NULL ? > { > prev= &node->next; > node= node->next; > diff -urN --exclude='.*' 5.3-timours-wl-clean/sql/sql_select.cc 5.3-timours-wl/sql/sql_select.cc > --- 5.3-timours-wl-clean/sql/sql_select.cc 2011-05-11 20:12:50.000000000 +0100 > +++ 5.3-timours-wl/sql/sql_select.cc 2011-05-11 20:08:50.000000000 +0100 ... > @@ -882,6 +887,7 @@ > "Impossible HAVING" : "Impossible WHERE"; > tables= 0; > error= 0; > + choose_tableless_subquery_plan(); > goto setup_subq_exit; > } psergey: there are three instances of this piece of code: if (.. we've figured out this is a degenerate join which produces no rows ...) { ... choose_tableless_subquery_plan(); goto setup_subq_exit; } I'm afraid if somebody invents a fourth way to figure out that the join is degenerate, they will forget to make call to choose_tableless_subquery_plan(). Could you prevent this by introducing empty_join_output: choose_tableless_subquery_plan(); ... whatever else ... setup_subq_exit: and using appropriate goto's? > } > @@ -926,12 +932,13 @@ > */ > if ((res=opt_sum_query(select_lex->leaf_tables, all_fields, conds))) > { > - if (res == HA_ERR_KEY_NOT_FOUND) > + if (res == HA_ERR_KEY_NOT_FOUND || res < 0) psergey: What is this for? I am looking at opt_sum_query code and I don't see any cases where it will return a negative value? > { > DBUG_PRINT("info",("No matching min/max row")); > zero_result_cause= "No matching min/max row"; > tables= 0; > error=0; > + choose_tableless_subquery_plan(); > goto setup_subq_exit; > } > if (res > 1) ... > @@ -9036,11 +9070,16 @@ > *simple_order=0; // Must do a temp table to sort > else if (!(order_tables & not_const_tables)) > { > - if (order->item[0]->with_subselect && > - !(join->select_lex->options & SELECT_DESCRIBE)) > - order->item[0]->val_str(&order->item[0]->str_value); > + if (order->item[0]->with_subselect) > + { > + /* > + Delay the evaluation of constant ORDER and/or GROUP expressions that > + contain subqueries until the execution phase. > + */ > + join->exec_const_order_group_cond.push_back(order->item[0]); psergey: I'm wondering why would one want to evaluate them at all in that case? If we have SELECT ... GROUP BY const_subquery_expression, ... we don't need to evaluate const_subquery_expression at all, do we? > + } > DBUG_PRINT("info",("removing: %s", order->item[0]->full_name())); > - continue; // skip const item > + continue; > } > else > { ... > @@ -20316,5 +20363,154 @@ > > > /** > + Save a query execution plan so that the caller can revert to it if needed, > + and reset the current query plan so that it can be reoptimized. > + > + @param save_to The object into which the current query plan state is saved > +*/ > + > +void JOIN::save_query_plan(Query_plan_state *save_to) > +{ > + if (keyuse.elements) > + { > + DYNAMIC_ARRAY tmp_keyuse; > + /* Swap the current and the backup keyuse internal arrays. */ > + tmp_keyuse= keyuse; > + keyuse= save_to->keyuse; /* keyuse is reset to an empty array. */ > + save_to->keyuse= tmp_keyuse; > + > + for (uint i= 0; i < tables; i++) > + { > + save_to->join_tab_keyuse[i]= join_tab[i].keyuse; > + join_tab[i].keyuse= NULL; > + save_to->join_tab_checked_keys[i]= join_tab[i].checked_keys; > + join_tab[i].checked_keys.clear_all(); > + } > + } > + memcpy((uchar*) save_to->best_positions, (uchar*) best_positions, > + sizeof(POSITION) * (tables + 1)); > + memset(best_positions, 0, sizeof(POSITION) * (tables + 1)); > +} > + > + > +/** > + Restore a query execution plan previously saved by the caller. > + > + @param The object from which the current query plan state is restored. > +*/ > + > +void JOIN::restore_query_plan(Query_plan_state *restore_from) > +{ > + if (restore_from->keyuse.elements) > + { > + DYNAMIC_ARRAY tmp_keyuse; > + tmp_keyuse= keyuse; > + keyuse= restore_from->keyuse; > + restore_from->keyuse= tmp_keyuse; > + > + for (uint i= 0; i < tables; i++) > + { > + join_tab[i].keyuse= restore_from->join_tab_keyuse[i]; > + join_tab[i].checked_keys= restore_from->join_tab_checked_keys[i]; > + } > + > + } > + memcpy((uchar*) best_positions, (uchar*) restore_from->best_positions, > + sizeof(POSITION) * (tables + 1)); > +} > + > + > +/** > + Reoptimize a query plan taking into account an additional conjunct to the > + WHERE clause. > + > + @param added_where An extra conjunct to the WHERE clause to reoptimize with > + @param join_tables The set of tables to reoptimize > + @param save_to If != NULL, save here the state of the current query plan > + > + @notes > + Given a query plan that was already optimized taking into account some WHERE > + clause 'C', reoptimize this plan with a new WHERE clause 'C AND added_where'. > + The reoptimization works as follows: > + > + 1. Call update_ref_and_keys *only* for the new conditions 'added_where' > + that are about to be injected into the query. > + 2. Expand if necessary the original KEYUSE array JOIN::keyuse to > + accommodate the new REF accesses computed for the 'added_where' condition. > + 3. Add the new KEYUSEs into JOIN::keyuse. > + 4. Re-sort and re-filter the JOIN::keyuse array with the newly added > + KEYUSE elements. psergey: I think the above scheme has some flaws. What if the initial filtering has discarded elements that would be useful with the new injected condition? Consider this example: create table ten (a int); insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); create table one_k (a int); insert into one_k select A.a + B.a*10 + C.a*100 from ten A, ten B, ten C; create table t11 (kp1 int, kp2 int, a int, filler char(100), key(kp1, kp2)); insert into t11 select a/100, a, a, 'filler' from one_k; If I debug this query: explain select a from one_k where a in (select kp1 from t11 where kp2=10 and a=4) or a+1 < 100000; I see that the optimizer considers access to t11 via lookup on t11.kp1=... but not on (t11.kp1=... AND t11.kp2=..). Please note that from user point of view this will look like a regression. > + > + @retval REOPT_NEW_PLAN there is a new plan. > + @retval REOPT_OLD_PLAN no new improved plan was produced, use the old one. > + @retval REOPT_ERROR an irrecovarable error occured during reoptimization. > +*/ > + > +JOIN::enum_reopt_result > +JOIN::reoptimize(Item *added_where, table_map join_tables, > + Query_plan_state *save_to) > +{ psergey: This function looks very odd. We touch "keyuse" only when restore_from->keyuse.elements!=NULL What about the case when this->keyuse.elements!=NULL && restore_from->keyuse.elements==NULL ? This looks like a possible scenario to me. Suppose that - original (i.e. Materialization) plan produces no KEYUSE-s - IN-to-EXISTS plan does produce KEYUSE and we're restoring from IN-to-EXISTS to Materialization plan? > + DYNAMIC_ARRAY added_keyuse; > + SARGABLE_PARAM *sargables= 0; /* Used only as a dummy parameter. */ > + uint org_keyuse_elements; > + > + /* Re-run the REF optimizer to take into account the new conditions. */ > + if (update_ref_and_keys(thd, &added_keyuse, join_tab, tables, added_where, > + ~outer_join, select_lex, &sargables)) > + { > + delete_dynamic(&added_keyuse); > + return REOPT_ERROR; > + } > + > + if (!added_keyuse.elements) > + { > + delete_dynamic(&added_keyuse); > + return REOPT_OLD_PLAN; > + } > + > + if (save_to) > + save_query_plan(save_to); > + > + if (!keyuse.buffer && > + my_init_dynamic_array(&keyuse, sizeof(KEYUSE), 20, 64)) > + { > + delete_dynamic(&added_keyuse); > + return REOPT_ERROR; > + } > + > + org_keyuse_elements= save_to ? save_to->keyuse.elements : keyuse.elements; > + allocate_dynamic(&keyuse, org_keyuse_elements + added_keyuse.elements); > + > + /* If needed, add the access methods from the original query plan. */ > + if (save_to) > + { > + DBUG_ASSERT(!keyuse.elements); > + memcpy(keyuse.buffer, > + save_to->keyuse.buffer, > + (size_t) save_to->keyuse.elements * keyuse.size_of_element); > + keyuse.elements= save_to->keyuse.elements; > + } > + > + /* Add the new access methods to the keyuse array. */ > + memcpy(keyuse.buffer + keyuse.elements * keyuse.size_of_element, > + added_keyuse.buffer, > + (size_t) added_keyuse.elements * added_keyuse.size_of_element); > + keyuse.elements+= added_keyuse.elements; > + /* added_keyuse contents is copied, and it is no longer needed. */ > + delete_dynamic(&added_keyuse); > + > + if (sort_and_filter_keyuse(&keyuse)) > + return REOPT_ERROR; > + optimize_keyuse(this, &keyuse); > + > + /* Re-run the join optimizer to compute a new query plan. */ > + if (choose_plan(this, join_tables)) > + return REOPT_ERROR; > + > + return REOPT_NEW_PLAN; > +} > + > + > +/** > @} (end of group Query_Optimizer) > */ > diff -urN --exclude='.*' 5.3-timours-wl-clean/sql/sql_select.h 5.3-timours-wl/sql/sql_select.h > --- 5.3-timours-wl-clean/sql/sql_select.h 2011-05-11 20:12:50.000000000 +0100 > +++ 5.3-timours-wl/sql/sql_select.h 2011-05-11 20:08:50.000000000 +0100 > @@ -603,8 +603,53 @@ > > class JOIN :public Sql_alloc > { > +private: > JOIN(const JOIN &rhs); /**< not implemented */ > JOIN& operator=(const JOIN &rhs); /**< not implemented */ > + > +protected: > + > + /** > + The subset of the state of a JOIN that represents an optimized query > + execution plan. Allows saving/restoring different plans for the same query. > + */ > + class Query_plan_state { psergey: I have a concern about naming. This is really a join plan state, not a query plan state. We may get into mess if at some point we'll need to save plan state for the whole query. Is it still possible to rename this to something like Join_plan_state? > + public: > + DYNAMIC_ARRAY keyuse; /* Copy of the JOIN::keyuse array. */ > + POSITION best_positions[MAX_TABLES+1]; /* Copy of JOIN::best_positions */ > + /* Copies of the JOIN_TAB::keyuse pointers for each JOIN_TAB. */ > + KEYUSE *join_tab_keyuse[MAX_TABLES]; > + /* Copies of JOIN_TAB::checked_keys for each JOIN_TAB. */ > + key_map join_tab_checked_keys[MAX_TABLES]; > + public: > + Query_plan_state() > + { > + keyuse.elements= 0; > + keyuse.buffer= NULL; > + } > + Query_plan_state(JOIN *join); > + ~Query_plan_state() > + { > + delete_dynamic(&keyuse); > + } > + }; > + > + /* Results of reoptimizing a JOIN via JOIN::reoptimize(). */ > + enum enum_reopt_result { > + REOPT_NEW_PLAN, /* there is a new reoptimized plan */ > + REOPT_OLD_PLAN, /* no new improved plan can be found, use the old one */ > + REOPT_ERROR, /* an irrecovarable error occured during reoptimization */ > + REOPT_NONE /* not yet reoptimized */ > + }; > + > + /* Support for plan reoptimization with rewritten conditions. */ > + enum_reopt_result reoptimize(Item *added_where, table_map join_tables, > + Query_plan_state *save_to); > + void save_query_plan(Query_plan_state *save_to); > + void restore_query_plan(Query_plan_state *restore_from); > + /* Choose a subquery plan for a table-less subquery. */ > + bool choose_tableless_subquery_plan(); > + > public: > JOIN_TAB *join_tab,**best_ref; > JOIN_TAB **map2table; ///< mapping between table indexes and JOIN_TABs > @@ -702,6 +747,13 @@ > account the changes made by test_if_skip_sort_order()). > */ > double best_read; > + /* > + Estimated result rows (fanout) of the whole query. If this is a subquery psergey: I think it would be better to change "whole query" to "join operation" or "the select" here. > + that is reexecuted multiple times, this value includes the estiamted # of > + reexecutions. This value is equal to the multiplication of all > + join->positions[i].records_read of a JOIN. > + */ > + double record_count; > List<Item> *fields; > List<Cached_item> group_fields, group_fields_cache; > TABLE *tmp_table; .. > @@ -1233,6 +1313,9 @@ > if (!inited) > { > inited=1; psergey: I don't understand why this wasn't needed before MWL#89, but is needed after MWL#89. Do you still remember why you've added it? > + TABLE *table= to_field->table; > + my_bitmap_map *old_map= dbug_tmp_use_all_columns(table, > + table->write_set); > if ((res= item->save_in_field(to_field, 1))) > { > if (!err) BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 2

Re: [Maria-developers] [Commits] Rev 3005: Changed MariaDB error numbers to start from 1900 to not conflict with MySQL error numbers in lp:maria/5.3
by Sergei Golubchik 23 May '11

23 May '11

Hi, Michael! There are more MariaDB-only error messages, I think you need to move all of them. On May 21, Michael Widenius wrote: > === modified file 'sql/share/errmsg.txt' > --- a/sql/share/errmsg.txt 2011-05-18 13:27:19 +0000 > +++ b/sql/share/errmsg.txt 2011-05-20 21:46:18 +0000 > @@ -6214,6 +6214,19 @@ ER_DEBUG_SYNC_HIT_LIMIT > eng "debug sync point hit limit reached" > ger "Debug Sync Point Hit Limit erreicht" > > +# > +# MariaDB error messages section starts here > +# > + > +# The following is here to allow us to detect if there was missing > +# error messages in the errmsg.sys file > + > +ER_LAST_MYSQL_ERROR_MESSAGE > + eng "" > + > +# MariaDB error numbers starts from 1900 > +start-error-number 1900 > + > ER_VCOL_BASED_ON_VCOL > eng "A computed column cannot be based on a computed column" Regards, Sergei

2 1

[Maria-developers] mysql improvement idea - field type "file"
by Tõnu Samuel 20 May '11

20 May '11

( 1. Sent this mail few days ago to Monty but no answer yet. I know, he is busy man. 2. I use name "MySQL" because this is real name for me. Oracle still sucks:P ) Hi! We all have seen cases where people try to use SQL for something what does not fit there. Reason is simple - file system itself is very good database for some type of objects and queries. One common MySQL usage scenario is web portal. Often portal has some sort of news and photos accompanying it. My idea is to make some sort of clever way to store images within MySQL making their read/write as fast as possible and meanwhile giving extra value to user who shouldn't care about data integrity keeping data in and out of SQL coherent. Also would be cool to get all required data with single query. I think best way to implement this is to introduce field type "file" which is stored outside of usual data storage. Only his physical location on disk is stored in MyISAM or similar. This is exactly same as people keep file name in CHAR currently but we provide method to fetch file contents with same query. Also we can implement cool replication to keep same file on all servers consistent, way to backup things via mysqldump etc.... Actually idea is bit wider but what you think generally? Tõnu

3 2

[Maria-developers] Review of microsecond patch
by Michael Widenius 19 May '11

19 May '11

Hi! Here is finally a review of the microsecond patch. (Took me 5 days to complete! Lots of code!!!) I got this by doing bzr diff -r tag:mysql-5.1.54..-1 in the 5.1-micro tree. > === modified file 'client/mysqltest.cc' > --- client/mysqltest.cc 2010-11-22 09:21:10 +0000 > +++ client/mysqltest.cc 2011-02-23 17:26:02 +0000 > @@ -7342,7 +7342,8 @@ int util_query(MYSQL* org_mysql, const c > cur_con->util_mysql= mysql; > } > > - return mysql_query(mysql, query); > + int ret= mysql_query(mysql, query); > + DBUG_RETURN(ret); > } Not necessary change, but not important enough to revert. > === modified file 'include/my_sys.h' <cut> > +#define my_micro_time() (my_getsystime()/10) I am not happy with the above change. The old my_micro_time() was defined to be significantly faster than getsystime() for some systems (especially windows and solaris) and we should consider keeping the old implementation.... <long phone conversation> Conclusion: my_getsystime() is not depending on the clock and only to be used when calculating intervals. > +#define hrtime_to_time(X) ((X).val/1000000) > +#define hrtime_from_time(X) ((ulonglong)((X)*1000000ULL)) > +#define hrtime_to_double(X) ((X).val/1e6) I think you need to add a cast to double avoid compiler warnings on all platforms. (check buildbot!) <cut> > > === modified file 'include/my_time.h' > --- include/my_time.h 2008-04-03 17:14:57 +0000 > +++ include/my_time.h 2011-03-26 10:59:34 +0000 > @@ -55,6 +55,7 @@ typedef long my_time_t; > /* Flags to str_to_datetime */ > #define TIME_FUZZY_DATE 1 > #define TIME_DATETIME_ONLY 2 > +#define TIME_TIME_ONLY 4 > /* Must be same as MODE_NO_ZERO_IN_DATE */ > #define TIME_NO_ZERO_IN_DATE (65536L*2*2*2*2*2*2*2) > /* Must be same as MODE_NO_ZERO_DATE */ > @@ -68,8 +69,9 @@ typedef long my_time_t; > #define TIME_MAX_HOUR 838 > #define TIME_MAX_MINUTE 59 > #define TIME_MAX_SECOND 59 > +#define TIME_MAX_SECOND_PART 999999 > #define TIME_MAX_VALUE (TIME_MAX_HOUR*10000 + TIME_MAX_MINUTE*100 + \ > - TIME_MAX_SECOND) > + TIME_MAX_SECOND + TIME_MAX_SECOND_PART/1e6) Make 1e6 a constant and use this in my_sys.h and here. > #define TIME_MAX_VALUE_SECONDS (TIME_MAX_HOUR * 3600L + \ > TIME_MAX_MINUTE * 60L + TIME_MAX_SECOND) > > @@ -80,16 +82,17 @@ str_to_datetime(const char *str, uint le > uint flags, int *was_cut); > longlong number_to_datetime(longlong nr, MYSQL_TIME *time_res, > uint flags, int *was_cut); > +int number_to_time(double nr, MYSQL_TIME *ltime, int *was_cut); > ulonglong TIME_to_ulonglong_datetime(const MYSQL_TIME *); > ulonglong TIME_to_ulonglong_date(const MYSQL_TIME *); > ulonglong TIME_to_ulonglong_time(const MYSQL_TIME *); > ulonglong TIME_to_ulonglong(const MYSQL_TIME *); > +double TIME_to_double(const MYSQL_TIME *my_time); > > +longlong pack_time(MYSQL_TIME *my_time); > +MYSQL_TIME *unpack_time(longlong packed, MYSQL_TIME *my_time); > > -my_bool str_to_time(const char *str,uint length, MYSQL_TIME *l_time, > - int *warning); > - > -int check_time_range(struct st_mysql_time *, int *warning); > +int check_time_range(struct st_mysql_time *my_time, uint dec, int *warning); > > long calc_daynr(uint year,uint month,uint day); > uint calc_days_in_year(uint year); > @@ -136,11 +139,30 @@ void set_zero_time(MYSQL_TIME *tm, enum > sent using binary protocol fit in this buffer. > */ > #define MAX_DATE_STRING_REP_LENGTH 30 > +#define MAX_SEC_PART_VALUE 999999 > +#define MAX_SEC_PART_DIGITS 6 > +#define AUTO_SEC_PART_DIGITS 31 /* same as NOT_FIXED_DEC */ Do we really need to have MAX_SEC_PART_VALUE and TIME_MAX_SECOND_PART ? Shouldn't these always be the same for mysys functions? Also MAX_SEC_PART_DIGITS should be changed to TIME_MAX_SECOND_PART_DIGITS <cut> > === modified file 'mysql-test/include/ps_conv.inc' > @@ -1153,19 +1146,19 @@ select '-- select .. where date/time col > set @arg00= '1991-01-01 01:01:01' ; > select 'true' as found from t9 > where c1= 20 and c13= CAST('1991-01-01 01:01:01' AS DATE) and c14= '1991-01-01 01:01:01' and > - c15= '1991-01-01 01:01:01' and c16= '1991-01-01 01:01:01' and > + c15= '1991-01-01 01:01:01' and You need to add at least one test that shows the failure of doing a comparison between datetime and time. > === added file 'mysql-test/include/type_hrtime.inc' > --- mysql-test/include/type_hrtime.inc 1970-01-01 00:00:00 +0000 > +++ mysql-test/include/type_hrtime.inc 2011-03-01 12:24:36 +0000 > @@ -0,0 +1,94 @@ > + > +--source include/have_innodb.inc > + > +--disable_warnings > +drop table if exists t1, t2, t3; > +--enable_warnings > + > +--error ER_TOO_BIG_PRECISION > +eval create table t1 (a $type(7)); > + > +eval create table t1 (a $type(3)); > +insert t1 values ('2010-12-11 01:02:03.4567'); > +insert t1 values (20101211010203.45678); > +insert t1 values (20101211030405.789e0); > +insert t1 values (99991231235959e1); > +select * from t1; > +select truncate(a, 6) from t1; # Field::val_real() > +select a DIV 1 from t1; # Field::val_int() > +select group_concat(distinct a) from t1; # Field::cmp() > +alter table t1 engine=innodb; > +select * from t1; > +drop table t1; > +eval create table t1 (a $type(4)) engine=innodb; > +insert t1 values ('2010-12-11 01:02:03.456789'); > +select * from t1; > +select extract(microsecond from a + interval 100 microsecond) from t1 where a>'2010-11-12 01:02:03.456'; > +select a from t1 where a>'2010-11-12 01:02:03.456' group by a; Shouldn't we do the above for all engines ? If you can, please add at test of this in suite/engines/time.test It would be good to add a combinations file: [xtradb] --default-engine=innodb [aira] --default-engine=aria [myisam] --default-engine=myisam [heap] --default-engine=heap The problem I see with this is: - How to ignore running tests if an engine doesn't exist? - How to handle different result files ? - Adding the combination name to the result file may be a good temporary solution until we can do something based on 'diff' <cut> > +# > +# Views > +# > + > +create view v1 as select * from t1 group by a,b; > +select * from v1; > +show columns from v1; > +create table t2 select * from v1; > +show create table t2; Please also select some data from t2. > + > +drop view v1; > +drop table t1, t2; Good set of tests. The things that was not tested, as far as I can see: - INSERT SELECT between tables that has different column types. It would be good to test all common combinations. date -> datetime date -> string time -> datetime time -> string datetime -> time datetime -> date datetime -> timestamp datetime -> string timestamp -> datetime timestamp -> date timestamp -> time timestamp -> string string -> date string -> datetime string -> time string -> timestamp Doing this with different microsecond precission would be also good And the same with alter table. > === modified file 'mysql-test/r/cast.result' > --- mysql-test/r/cast.result 2009-05-21 08:06:43 +0000 > +++ mysql-test/r/cast.result 2011-03-18 13:52:20 +0000 > @@ -254,7 +254,7 @@ cast("2001-1-1" as datetime) = "2001-01- > 1 > select cast("1:2:3" as TIME) = "1:02:03"; > cast("1:2:3" as TIME) = "1:02:03" > -0 > +1 > select cast(NULL as DATE); > cast(NULL as DATE) > NULL > @@ -452,3 +452,9 @@ SELECT CONVERT(t2.a USING UTF8) FROM t1, > 1 > DROP TABLE t1; > End of 5.1 tests > +create table t1 (f1 time, f2 date, f3 datetime); > +insert into t1 values ('11:22:33','2011-12-13','2011-12-13 11:22:33'); > +select cast(f1 as unsigned), cast(f2 as unsigned), cast(f3 as unsigned) from t1; > +cast(f1 as unsigned) cast(f2 as unsigned) cast(f3 as unsigned) > +112233 20111213 20111213112233 > +drop table t1; Please also add a field datetime with microseconds to test cast to unsigned and double. > === modified file 'mysql-test/r/date_formats.result' > --- mysql-test/r/date_formats.result 2010-11-12 10:12:15 +0000 > +++ mysql-test/r/date_formats.result 2011-03-08 18:41:58 +0000 > @@ -196,16 +196,16 @@ date format datetime > 0003-01-02 8:11:2.123456 %Y-%m-%d %H:%i:%S.%# 0003-01-02 08:11:02 > 03-01-02 8:11:2.123456 %Y-%m-%d %H:%i:%S.%# 2003-01-02 08:11:02 > 2003-01-02 10:11:12 PM %Y-%m-%d %h:%i:%S %p 2003-01-02 22:11:12 > -2003-01-02 01:11:12.12345AM %Y-%m-%d %h:%i:%S.%f%p 2003-01-02 01:11:12.123450 > -2003-01-02 02:11:12.12345AM %Y-%m-%d %h:%i:%S.%f %p 2003-01-02 02:11:12.123450 > -2003-01-02 12:11:12.12345 am %Y-%m-%d %h:%i:%S.%f%p 2003-01-02 00:11:12.123450 > +2003-01-02 01:11:12.12345AM %Y-%m-%d %h:%i:%S.%f%p 2003-01-02 01:11:12 > +2003-01-02 02:11:12.12345AM %Y-%m-%d %h:%i:%S.%f %p 2003-01-02 02:11:12 > +2003-01-02 12:11:12.12345 am %Y-%m-%d %h:%i:%S.%f%p 2003-01-02 00:11:12 Why has microseconds been removed from the output ? It looks like 'cast(str_to_date(date, format) as datetime)' removes all microseconds, which is an incompatible change. In the past cast(..., datetime) as same as cast(..., datetime(6)) You should at least add a new test that uses 'cast(str_to_date(date, format) as datetime(6))' Please also compile a list for the KB about all incompatible changes. <cut> I think that it's more important that cast() produces the same value as MySQL did than that create ... select cast() produces the same type as before. With the first issue, you will get lost data (without any warnings) for old applictions if they would do things like: select * from t1 where cast(char_column as time) = "12:23:24.123456"; which works with MySQL but not with this patch. > === modified file 'mysql-test/r/func_in.result' > --- mysql-test/r/func_in.result 2010-06-22 18:53:08 +0000 > +++ mysql-test/r/func_in.result 2011-03-01 12:24:36 +0000 > @@ -544,15 +544,9 @@ id select_type table type possible_keys > select f2 from t2 where f2 in ('a','b'); > f2 > 0 > -Warnings: > -Warning 1292 Truncated incorrect DOUBLE value: 'a' > -Warning 1292 Truncated incorrect DOUBLE value: 'b' > explain select f2 from t2 where f2 in ('a','b'); > id select_type table type possible_keys key key_len ref rows Extra > 1 SIMPLE t2 index t2f2 t2f2 5 NULL 3 Using where; Using index > -Warnings: > -Warning 1292 Truncated incorrect DOUBLE value: 'a' > -Warning 1292 Truncated incorrect DOUBLE value: 'b' Good that we got rid of confusing internal error messages! > === modified file 'mysql-test/r/func_sapdb.result' > --- mysql-test/r/func_sapdb.result 2010-08-13 13:05:46 +0000 > +++ mysql-test/r/func_sapdb.result 2011-03-08 18:41:58 +0000 > @@ -113,10 +113,10 @@ subtime("01:00:00.999999", "02:00:00.999 > -00:59:59.999999 > select subtime("02:01:01.999999", "01:01:01.999999"); > subtime("02:01:01.999999", "01:01:01.999999") > -01:00:00.000000 > +01:00:00 Add also a test for: select subtime("02:01:01.999999", "01:01:01.999998"); To ensure we don't drop the sub seconds. > === modified file 'mysql-test/r/func_time.result' > --- mysql-test/r/func_time.result 2010-10-31 16:04:38 +0000 > +++ mysql-test/r/func_time.result 2011-03-19 14:49:36 +0000 > @@ -8,20 +8,20 @@ period_add("9602",-12) period_diff(19950 <cut> > +1994-03-02 10:11:12 1994-03-02 10:11:12 19940302101112 > select sec_to_time(9001),sec_to_time(9001)+0,time_to_sec("15:12:22"), > sec_to_time(time_to_sec("0:30:47")/6.21); > sec_to_time(9001) sec_to_time(9001)+0 time_to_sec("15:12:22") sec_to_time(time_to_sec("0:30:47")/6.21) > -02:30:01 23001.000000 54742 00:04:57 > +02:30:01 23001 54742 00:04:57.423510 Please add test for 'sec_to_time(9001.1)' and 'time_to_sec("15:12:22.123456')' and 'time_to_sec(15.55)'. I noticed that: select time_to_sec(15.55023124); -> 15.55023100 select time_to_sec('15.55023124'); -> 15.550231 Wouldn't this be better to limit the first one also to 6 digits? (Serg agreed to this on IRC) Add test for above cases too > @@ -798,9 +798,7 @@ TIMESTAMPDIFF(year,'2006-01-10 14:30:28' > 2 > select date_add(time,INTERVAL 1 SECOND) from t1; > date_add(time,INTERVAL 1 SECOND) > -NULL > -Warnings: > -Warning 1264 Out of range value for column 'time' at row 1 > +06:07:09 Add a note to compatible matrix that 'date_add()' now also works for time types. > @@ -941,10 +939,10 @@ sec_to_time(1) + 0, from_unixtime(1) + 0 > show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > - `now() - now()` double(23,6) NOT NULL DEFAULT '0.000000', > - `curtime() - curtime()` double(23,6) NOT NULL DEFAULT '0.000000', > - `sec_to_time(1) + 0` double(23,6) DEFAULT NULL, > - `from_unixtime(1) + 0` double(23,6) DEFAULT NULL > + `now() - now()` double(17,0) NOT NULL DEFAULT '0', > + `curtime() - curtime()` double(17,0) NOT NULL DEFAULT '0', > + `sec_to_time(1) + 0` double(17,0) DEFAULT NULL, > + `from_unixtime(1) + 0` double(17,0) DEFAULT NULL > ) ENGINE=MyISAM DEFAULT CHARSET=latin1 To make example complete, add to the original create time some decimals for sec_to_time and also some decimals to one of the now() arguments. The CREATE was: create table t1 select now() - now(), curtime() - curtime(), sec_to_time(1) + 0, from_unixtime(1) + 0; As part of this, I noticed that from_unixtime() and unix_timestamp() is not enhanced to support microseconds. This needs to work: create table t2 (a int, b timestamp(6)) insert into t2 (a) values(1); select * from t2; I got '1 | 2011-03-29 15:38:53.395516' select unix_timestamp(b) from t2; -> 1301402333 (no decimals) This needs to returns sub seconds! I tested that: set timestamp=1301402333.55; select now(6); works (good)! <cut> > +create table t1 (f1 integer, f2 date); > +insert into t1 values (1,'2011-05-05'),(2,'2011-05-05'),(3,'2011-05-05'),(4,'2011-05-05'),(5,'2011-05-05'); > +select * from t1 where 1 and concat(f2)=MAKEDATE(2011, 125); > +f1 f2 > +1 2011-05-05 > +2 2011-05-05 > +3 2011-05-05 > +4 2011-05-05 > +5 2011-05-05 Add also at least one value for which there is no match. <cut> > --- mysql-test/r/func_time_hires.result 1970-01-01 00:00:00 +0000 > +++ mysql-test/r/func_time_hires.result 2011-03-17 13:13:03 +0000 > @@ -0,0 +1,177 @@ > +set timestamp=unix_timestamp('2011-01-01 01:01:01') + 0.123456, time_zone='+03:00'; > +select sec_to_time(12345), sec_to_time(12345.6789), sec_to_time(1234567e-2); > +sec_to_time(12345) 03:25:45 > +sec_to_time(12345.6789) 03:25:45.6789 > +sec_to_time(1234567e-2) 03:25:45.670000 > +select now(), curtime(0), utc_timestamp(1), utc_time(2), current_time(3), > +current_timestamp(4), localtime(5), localtimestamp(6), time_to_sec('12:34:56'), > +time_to_sec('12:34:56.789'); > +now() 2011-01-01 01:01:01 > +curtime(0) 01:01:01 > +utc_timestamp(1) 2010-12-31 22:01:01.1 > +utc_time(2) 22:01:01.12 > +current_time(3) 01:01:01.123 > +current_timestamp(4) 2011-01-01 01:01:01.1234 > +localtime(5) 2011-01-01 01:01:01.12345 > +localtimestamp(6) 2011-01-01 01:01:01.123456 > +time_to_sec('12:34:56') 45296 > +time_to_sec('12:34:56.789') 45296.789 > +select sec_to_time(time_to_sec('1:2:3')), sec_to_time(time_to_sec('2:3:4.567890')); > +sec_to_time(time_to_sec('1:2:3')) 01:02:03 > +sec_to_time(time_to_sec('2:3:4.567890')) 02:03:04.567890 > +select time_to_sec(sec_to_time(11111)), time_to_sec(sec_to_time(11111.22222)); > +time_to_sec(sec_to_time(11111)) 11111 > +time_to_sec(sec_to_time(11111.22222)) 11111.22222 > +select current_timestamp(7); > +ERROR HY000: Incorrect arguments to now <cut> > +select CAST(@a AS DATETIME(7)); > +ERROR 42000: Too big precision 7 specified for column '(@a)'. Maximum is 6. The above would be a better error message than 'Incorrect arguments to now'. Can you add precision and maximum to this error message? > +select time(f1) from t1 union all select time(f1) from t1; Add a time operation on the second time(f1) to make the rows a bit difference (and test more code) > +time(f1) > +21:00:00.000000 > +21:00:00.000000 > +drop table t1; > === modified file 'mysql-test/r/information_schema.result' > --- mysql-test/r/information_schema.result 2010-06-23 16:25:31 +0000 > +++ mysql-test/r/information_schema.result 2011-03-01 12:24:36 +0000 > @@ -189,8 +189,8 @@ Field Type Collation Null Key Default Ex > c varchar(64) utf8_general_ci NO select,insert,update,references > select * from information_schema.COLUMNS where table_name="t1" > and column_name= "a"; > -TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT > -NULL mysqltest t1 a 1 NULL YES int NULL NULL 10 0 NULL NULL int(11) select,insert,update,references > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE DATETIME_PRECISION CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT > +NULL mysqltest t1 a 1 NULL YES int NULL NULL 10 0 NULL NULL NULL int(11) select,insert,update,references This is also an incompatible change that should be listed. > === modified file 'mysql-test/r/parser.result' > --- mysql-test/r/parser.result 2009-04-17 20:00:53 +0000 > +++ mysql-test/r/parser.result 2011-03-01 12:24:36 +0000 > @@ -556,9 +556,13 @@ DROP TABLE IF EXISTS t1; > SELECT STR_TO_DATE('10:00 PM', '%h:%i %p') + INTERVAL 10 MINUTE; > STR_TO_DATE('10:00 PM', '%h:%i %p') + INTERVAL 10 MINUTE > NULL > +Warnings: > +Error 1411 Incorrect datetime value: '10:00 PM' for function str_to_date Strange that 'str_to_date()' doesn't work with TIME arguments, but 'date_add()' does. > === modified file 'mysql-test/r/ps.result' > --- mysql-test/r/ps.result 2010-10-04 08:51:26 +0000 > +++ mysql-test/r/ps.result 2011-03-01 12:24:36 +0000 > @@ -3040,3 +3040,18 @@ id select_type table type possible_keys > DEALLOCATE PREPARE stmt; > DROP TABLE t1; > End of 5.1 tests. > +prepare stmt from "select date('2010-10-10') between '2010-09-09' and ?"; > +set @a='2010-11-11'; > +execute stmt using @a; > +date('2010-10-10') between '2010-09-09' and ? > +1 > +execute stmt using @a; > +date('2010-10-10') between '2010-09-09' and ? > +1 > +set @a='2010-08-08'; > +execute stmt using @a; > +date('2010-10-10') between '2010-09-09' and ? > +0 > +execute stmt using @a; > +date('2010-10-10') between '2010-09-09' and ? > +0 Please add a test that uses sub seconds (for completeness) set @a='2010-08-08 02:03:04.567890'; execute stmt using @a; > === modified file 'mysql-test/r/select.result' <cut> > -select str_to_date('2007-10-09','%Y-%m-%d') <= '2007/10/2000:00:00 GMT-6'; > -str_to_date('2007-10-09','%Y-%m-%d') <= '2007/10/2000:00:00 GMT-6' > +select str_to_date('2007-10-09','%Y-%m-%d') <= '2007/10/20 00:00:00 GMT-6'; > +str_to_date('2007-10-09','%Y-%m-%d') <= '2007/10/20 00:00:00 GMT-6' Please add a new test that shows that this fails: select str_to_date('2007-10-09','%Y-%m-%d') <= '2007/10/2000:00:00 GMT-6' As this is an incompatible change (but of course not critical) > Warnings: > -Warning 1292 Truncated incorrect date value: '2007/10/2000:00:00 GMT-6' > +Warning 1292 Truncated incorrect date value: '2007/10/20 00:00:00 GMT-6' > select str_to_date('2007-10-01','%Y-%m-%d') = '2007-10-1 00:00:00 GMT-6'; > str_to_date('2007-10-01','%Y-%m-%d') = '2007-10-1 00:00:00 GMT-6' > 1 > @@ -4179,22 +4179,17 @@ Warning 1292 Truncated incorrect datetim > select str_to_date('2007-10-00 12:34','%Y-%m-%d %H:%i') = '2007-10-01 12:34'; > str_to_date('2007-10-00 12:34','%Y-%m-%d %H:%i') = '2007-10-01 12:34' > 0 > -Warnings: > -Warning 1292 Truncated incorrect datetime value: '2007-10-00 12:34:00' Add a test here (which is run with SQL_MODE=traditional): select str_to_date('2007-10-00', "%Y-%m-%d %H:%i"); (Makes the test easier to understand) > @@ -4221,18 +4216,18 @@ str_to_date('1','%Y-%m-%d') = '1' > Warning 1292 Truncated incorrect date value: '1' > select str_to_date('','%Y-%m-%d') = ''; > str_to_date('','%Y-%m-%d') = '' This is an incompatible change (but for the better) In comparison string to date, we don't give errors for zero part dates independent of SQL_MODE. And as agreed on IRC, we can't have str_to_date('','%Y-%m-%d') return NULL as otherwise we couldn't use = to find rows with zero date in a table which would be worse. <cut> > Warnings: > Warning 1292 Truncated incorrect date value: '' > select str_to_date('1000-01-01','%Y-%m-%d') between '0000-00-00' and NULL; > str_to_date('1000-01-01','%Y-%m-%d') between '0000-00-00' and NULL > -0 > +NULL > select str_to_date('1000-01-01','%Y-%m-%d') between NULL and '2000-00-00'; > str_to_date('1000-01-01','%Y-%m-%d') between NULL and '2000-00-00' > -0 > +NULL > select str_to_date('1000-01-01','%Y-%m-%d') between NULL and NULL; > str_to_date('1000-01-01','%Y-%m-%d') between NULL and NULL > -0 > +NULL For completeness sake add: select str_to_date('2000-01-02','%Y-%m-%d') between NULL and '2000-01-01'; -> 0 select str_to_date('2000-01-02','%Y-%m-%d') between '2000-01-03' and NULL; -> 0 MariaDB 5.1 returns wrong result (NULL) for the above while 5.1-micro returns the correct answer. > === modified file 'mysql-test/r/subselect.result' <cut> > -delete from t1 where b = (select b from t1); > +delete from t1 where b in (select b from t1); Why the above change ? Please also add the original row, so that we can ensure it works the same way in the future. > === modified file 'mysql-test/r/type_datetime.result' > @@ -352,7 +352,7 @@ least(cast('01-01-01' as date), '01-01-0 > 2001-01-01 > select greatest(cast('01-01-01' as date), '01-01-02'); > greatest(cast('01-01-01' as date), '01-01-02') > -01-01-02 > +2001-01-02 Incompatible change (but ok) > === modified file 'mysql-test/r/type_time.result' > --- mysql-test/r/type_time.result 2010-05-31 09:25:11 +0000 > +++ mysql-test/r/type_time.result 2011-03-08 21:01:40 +0000 > @@ -1,6 +1,8 @@ > drop table if exists t1; > create table t1 (t time); > insert into t1 values("10:22:33"),("12:34:56.78"),(10),(1234),(123456.78),(1234559.99),("1"),("1:23"),("1:23:45"), ("10.22"), ("-10 1:22:33.45"),("20 10:22:33"),("1999-02-03 20:33:34"); > +Warnings: > +Note 1265 Data truncated for column 't' at row 13 Why the above note ? (I checked that the data is ok) Please document this in the compatible matrix. > === modified file 'mysql-test/suite/funcs_1/r/is_columns.result' > --- mysql-test/suite/funcs_1/r/is_columns.result 2008-11-13 09:50:20 +0000 > +++ mysql-test/suite/funcs_1/r/is_columns.result 2011-03-24 14:55:52 +0000 <cut> > +++ mysql-test/suite/funcs_1/r/is_columns_innodb.result 2011-03-24 14:55:52 +0000 > @@ -382,333 +382,333 @@ LOAD DATA INFILE '<MYSQLTEST_VARDIR>/std > SELECT * FROM information_schema.columns > WHERE table_schema LIKE 'test%' > ORDER BY table_schema, table_name, column_name; > -TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT > -NULL test t1 f1 1 NULL YES char 20 20 NULL NULL latin1 latin1_swedish_ci char(20) select,insert,update,references .... lots of changes How about changing the SELECT * FROM information_schema.columns to just list the old columns in suite/funcs_1/datadict/columns.inc This gives a smaller change list and will not cause a diff if we add another column... Especially this change caused really huge diff and will likely cause merge conflicts sooner or later... <cut> > === modified file 'mysys/my_getsystime.c' <cut> > +/* > + get time since epoc in 100 nanosec units > + > + NOTE: > + Thus to get the current time we should use the system function > + with the highest possible resolution > + > + The value is not subject to resetting or drifting by way of adjtime() or > + settimeofday(), and thus it is *NOT* appropriate for getting the current > + timestamp. It can be used for calculating time intervals, though. > + And it's good enough for UUID. > +*/ > + > ulonglong my_getsystime() > { > #ifdef HAVE_CLOCK_GETTIME > struct timespec tp; > clock_gettime(CLOCK_REALTIME, &tp); > return (ulonglong)tp.tv_sec*10000000+(ulonglong)tp.tv_nsec/100; > +#elif defined(HAVE_GETHRTIME) > + return gethrtime()/100-gethrtime_offset; > #elif defined(__WIN__) > LARGE_INTEGER t_cnt; > if (query_performance_frequency) > @@ -57,169 +73,70 @@ ulonglong my_getsystime() > #endif > } As discussed on IRC, I don't think it's make sense to try to have this aligned to 'epoch'. Better to just remove gethrtime_offset; Beeing 'free' from epoch will allow us to use other even faster clock's in the future like clock_gettime(CLOCK_MONOTONIC) > +my_hrtime_t my_hrtime() > { > + my_hrtime_t hrtime; > #if defined(__WIN__) > ulonglong newtime; > GetSystemTimeAsFileTime((FILETIME*)&newtime); > + hrtime.val= newtime/10; > #elif defined(HAVE_GETHRTIME) Shouldn't the above be 'HAVE_GETTIMEOFDAY' ? > struct timeval t; > /* > The following loop is here because gettimeofday may fail on some systems > */ > while (gettimeofday(&t, NULL) != 0) > {} > + hrtime.val= t.tv_sec*1000000 + t.tv_usec; > +#else > + hrtime.val= my_getsystime()/10; Don't think this option is correct as my_getsystime() is defined to be not reliable for timing. I would assume that all systems have either GetSystemTimeAsFileTime, gettimeofday or clock_gettime(). So I would instead add here: #ifdef HAVE_CLOCK_GETTIME struct timespec tp; clock_gettime(CLOCK_REALTIME, &tp); return (ulonglong)tp.tv_sec*10000000+(ulonglong)tp.tv_nsec/100; #else #error Need one of GetSystemTimeAsFileTime(), gettimeofday() or clock_gettime() #endif > + return hrtime; <cut> > +void my_time_init() > { <cut> > + gethrtime_offset= gethrtime(); I suggest we remove this > +#endif > } > === modified file 'sql-common/my_time.c' > --- sql-common/my_time.c 2010-07-09 12:00:17 +0000 > +++ sql-common/my_time.c 2011-03-26 10:59:34 +0000 > @@ -19,6 +19,9 @@ > /* Windows version of localtime_r() is declared in my_ptrhead.h */ > #include <my_pthread.h> > > +static enum enum_mysql_timestamp_type > +str_to_time(const char *str, uint length, MYSQL_TIME *l_time, int *warning) Make this again global; There are places where we can call this directly and there is no reason to do an extra jump to get here. <cut> > @@ -681,24 +692,31 @@ my_bool str_to_time(const char *str, uin > 1 time value is invalid > */ > > +int check_time_range(struct st_mysql_time *my_time, uint dec, int *warning) > { > longlong hour; > + static ulong max_sec_part[MAX_SEC_PART_DIGITS+1]= {000000, 900000, 990000, > + 999000, 999900, 999990, 999999}; > > if (my_time->minute >= 60 || my_time->second >= 60) > return 1; > > hour= my_time->hour + (24*my_time->day); > + > + if (dec == AUTO_SEC_PART_DIGITS) > + dec= MAX_SEC_PART_DIGITS; > + > if (hour <= TIME_MAX_HOUR && > (hour != TIME_MAX_HOUR || my_time->minute != TIME_MAX_MINUTE || > + my_time->second != TIME_MAX_SECOND || > + my_time->second_part <= max_sec_part[dec])) > return 0; > > my_time->day= 0; > my_time->hour= TIME_MAX_HOUR; > my_time->minute= TIME_MAX_MINUTE; > my_time->second= TIME_MAX_SECOND; > - my_time->second_part= 0; > + my_time->second_part= TIME_MAX_SECOND_PART; Shouldn't this be max_sec_part[dec] ? (Otherwise you may return a bigger value than what the function was called with) > *warning|= MYSQL_TIME_WARN_OUT_OF_RANGE; > return 0; > } <cut> > +int number_to_time(double nr, MYSQL_TIME *ltime, int *was_cut) > +{ > + ulong tmp; > + *was_cut= 0; > + ltime->year= ltime->month= ltime->day= 0; > + ltime->time_type= MYSQL_TIMESTAMP_TIME; > + > + if ((ltime->neg= nr < 0)) > + nr= -nr; > + > + if (nr > TIME_MAX_VALUE) > + { > + nr= TIME_MAX_VALUE; > + *was_cut= MYSQL_TIME_WARN_OUT_OF_RANGE; > + } > + tmp=(ulong)floor(nr); > + ltime->hour = tmp/100/100; > + ltime->minute= tmp/100%100; > + ltime->second= tmp%100; > + ltime->second_part= (ulong)((nr-tmp)*1e6); Change 1e6 to a define like 'TIME_SECOND_PART_TO_INT_MULTIPLIER' here and in all other places. > + if (ltime->minute < 60 && ltime->second < 60) > + return 0; > + > + *was_cut= MYSQL_TIME_WARN_TRUNCATED; > + return 1; > +} <cut> > === modified file 'sql/event_db_repository.cc' > --- sql/event_db_repository.cc 2010-01-22 09:38:21 +0000 > +++ sql/event_db_repository.cc 2011-03-01 12:24:36 +0000 > @@ -231,6 +231,9 @@ mysql_event_fill_row(THD *thd, > rs|= fields[ET_FIELD_STATUS]->store((longlong)et->status, TRUE); > rs|= fields[ET_FIELD_ORIGINATOR]->store((longlong)et->originator, TRUE); > > + if (!is_update) > + rs|= fields[ET_FIELD_CREATED]->set_time(); > + Don't understand why this was moved from Event_db_repository::create_event(). Probably safe but it would be nice to know the reason. (The bzr file comment didn't say anything about this) > === modified file 'sql/event_queue.cc' > --- sql/event_queue.cc 2008-05-09 07:43:02 +0000 > +++ sql/event_queue.cc 2011-03-01 12:24:36 +0000 > @@ -513,9 +513,10 @@ Event_queue::empty_queue() > */ > > void > +Event_queue::dbug_dump_queue(my_time_t when) > { > #ifndef DBUG_OFF > + my_time_t now= when; > Event_queue_element *et; > uint i; > DBUG_ENTER("Event_queue::dbug_dump_queue"); > @@ -592,14 +593,13 @@ Event_queue::get_top_for_execution_if_ti > thd->set_current_time(); /* Get current time */ > > next_activation_at= top->execute_at; > + if (next_activation_at >thd->query_start()) > { > /* > Not yet time for top event, wait on condition with > time or until signaled. Release LOCK_queue while waiting. > */ > - struct timespec top_time; > - set_timespec(top_time, next_activation_at - thd->query_start()); > + struct timespec top_time= { next_activation_at, 0 }; Not sure if the above change is portable. That is why we have the set_timespec() defines. Any particular reason why you did the change ? I would prefer to have the original code as we know it's works and compiles on a lot of platforms. However, after your definition change of my_getsystime() you have to redefine all the macros on my_pthread.h that calls my_getsystime() to call my_hrtime() instead as my_getsystime() can't anymore be used as a start point for timespec. If you want to keep the above change, you should also remove all the set_timespec() defines from my_pthread.h and fix all other code that uses this! > cond_wait(thd, &top_time, queue_wait_msg, SCHED_FUNC, __LINE__); > > continue; > === modified file 'sql/field.cc' > @@ -3151,13 +3141,9 @@ int Field_short::store(const char *from, > > error= get_int(cs, from, len, &rnd, UINT_MAX16, INT_MIN16, INT_MAX16); > store_tmp= unsigned_flag ? (int) (ulonglong) rnd : (int) rnd; > -#ifdef WORDS_BIGENDIAN > - if (table->s->db_low_byte_first) > - { > + if (ARCH_BIGENDIAN && table->s->db_low_byte_first) > int2store(ptr, store_tmp); > - } The above code can produce warnings on some compilers with the error 'if is always true'. I also find it much harder to read and understand! Please add back the original code and don't try to play smart games with the compiler! This is not part of microsecond patch and should not be done here. Same goes for all other changes of WORDS_BIGENDIAN. That said, we can probably remove the db_low_byte_first code in the future as no engine uses this anymore. This should however be a different patch! > @@ -4822,136 +4682,136 @@ timestamp_auto_set_type Field_timestamp: > } > } > > +long Field_timestamp::get_timestamp(ulong *sec_part) const > +{ > + ASSERT_COLUMN_MARKED_FOR_READ; > + *sec_part= 0; > + if (ARCH_BIGENDIAN && table && table->s->db_low_byte_first) > + return sint4korr(ptr); > + long tmp; > + longget(tmp,ptr); > + return tmp; > +} > Shouldn't this be of type 'my_time_t' for completeness? > -int Field_timestamp::store(const char *from,uint len,CHARSET_INFO *cs) > +int Field_timestamp::store_TIME_with_warning(THD *thd, MYSQL_TIME *l_time, > + const Lazy_string *str, > + bool was_cut, bool have_smth_to_conv) > { > ASSERT_COLUMN_MARKED_FOR_WRITE; > + uint error = 0; > + my_time_t timestamp; > my_bool in_dst_time_gap; > + if (was_cut || !have_smth_to_conv) > { > error= 1; > set_datetime_warning(MYSQL_ERROR::WARN_LEVEL_WARN, WARN_DATA_TRUNCATED, > + str, MYSQL_TIMESTAMP_DATETIME, 1); > } Add a comment here. The following test is not clear > + if (have_smth_to_conv && l_time->month) > { > + if (!(timestamp= TIME_to_timestamp(thd, l_time, &in_dst_time_gap))) > { > set_datetime_warning(MYSQL_ERROR::WARN_LEVEL_WARN, > ER_WARN_DATA_OUT_OF_RANGE, > + str, MYSQL_TIMESTAMP_DATETIME, !error); > error= 1; > } > else if (in_dst_time_gap) > { > set_datetime_warning(MYSQL_ERROR::WARN_LEVEL_WARN, > ER_WARN_INVALID_TIMESTAMP, > + str, MYSQL_TIMESTAMP_DATETIME, !error); > error= 1; > } > } As we are fixing things, it would be nice to change the two above 'if' in the common case to one if. We could do it by changing the last parameter in TIME_to_timestamp() to an enum and do: timestamp= TIME_to_timestamp(thd, l_time, &timstamp_error) if (timestamp_error) { set_datetime_warning(MYSQL_ERROR::WARN_LEVEL_WARN, (timestamp_error == MYSQL_TIMESTAMP_IN_DST_TIME_GAP ? ER_WARN_INVALID_TIMESTAMP : ER_WARN_DATA_OUT_OF_RANGE), str, MYSQL_TIMESTAMP_DATETIME, !error); error= 1; } Less code and only one if... > + else > + { > + timestamp= 0; > + l_time->second_part= 0; > + } > + store_TIME(timestamp, l_time->second_part); > return error; > } > +int Field_timestamp::store_time(MYSQL_TIME *ltime,timestamp_type time_type) > +{ > + THD *thd= table ? table->in_use : current_thd; You can assume that table->in_use is always set. (Other place in the code already assumes that). Feel free to fix this everywhere in field.cc > + int unused; > + MYSQL_TIME l_time= *ltime; > + Lazy_string_time str(ltime); > + bool valid= !check_date(&l_time, pack_time(&l_time) != 0, > + (thd->variables.sql_mode & MODE_NO_ZERO_DATE) | > + MODE_NO_ZERO_IN_DATE, &unused); > + > + return store_TIME_with_warning(thd, &l_time, &str, false, valid); > +} > + Add an empty line here > +int Field_timestamp::store(const char *from,uint len,CHARSET_INFO *cs) > +{ <cut> > int Field_timestamp::store(double nr) > + MYSQL_TIME l_time; > + int error; > + Lazy_string_dbl str(nr); > + THD *thd= table ? table->in_use : current_thd; > + > + /* We don't want to store invalid or fuzzy datetime values in TIMESTAMP */ > + longlong tmp= number_to_datetime((longlong) floor(nr), > + &l_time, (thd->variables.sql_mode & > + MODE_NO_ZERO_DATE) | > + MODE_NO_ZERO_IN_DATE, &error); > + l_time.second_part= (ulong)((nr-floor(nr))*1e6); > + return store_TIME_with_warning(thd, &l_time, &str, error, tmp != LL(-1)); > } You should add back the test: if (nr < 0 || nr > 99991231235959.9999999) the reason is that on some machines you get a trap or wrong result if you try to convert a too big double to a longlong! > longlong Field_timestamp::val_int(void) > { > MYSQL_TIME time_tmp; > THD *thd= table ? table->in_use : current_thd; > > thd->time_zone_used= 1; > ulong sec_part; > uint32 temp= get_timestamp(&sec_part); Add a comment here: /* Field_timestamp() and Field_timestamp_hres() shares this code. This is why are also testing sec_part below. */ > if (temp == 0 && sec_part == 0) > return(0); <cut> > longlong Field_timestamp::val_int(void) > { > MYSQL_TIME time_tmp; > THD *thd= table ? table->in_use : current_thd; Please add: ASSERT_COLUMN_MARKED_FOR_READ; > +static const char *zero_timestamp="0000-00-00 00:00:00.000000"; Move to start of file. > String *Field_timestamp::val_str(String *val_buffer, String *val_ptr) > { > - ASSERT_COLUMN_MARKED_FOR_READ; Add back ASSERT_COLUMN_MARKED_FOR_READ; <cut> > -void Field_timestamp::set_time() > +int Field_timestamp::set_time() > { > THD *thd= table ? table->in_use : current_thd; > - long tmp= (long) thd->query_start(); > set_notnull(); > - store_timestamp(tmp); > + store_TIME(thd->query_start(), 0); > + return 0; > } The set_time() function could be a void as it can never fail. (Yes, I agree that it means that we have to test type of field in event code, but that something we should do at event startup anyway...) <cut> > +static void store_native(ulonglong num, uchar *to, uint bytes) > { > + switch(bytes) { > + case 1: *to= (uchar)num; break; > + case 2: shortstore(to, (ushort)num); break; > + case 3: int3store(to, num); /* Sic!*/ break; > + case 4: longstore(to, (ulong)num); break; > + case 8: longlongstore(to, num); break; > + default: DBUG_ASSERT(0); > + } > +} What would be even better if each field would have a pointers to c functions with the right read and write functions; Would avoid a switch for every call... <cut> > +static uint sec_part_bytes[MAX_DATETIME_PRECISION+1]= { 0, 1, 1, 2, 2, 3, 3 }; Move to start of file and document this. <cut> > + if (!tmp) > + return fuzzydate & TIME_NO_ZERO_DATE; should be: > + return (fuzzydate & TIME_NO_ZERO_DATE) != 0; as the function is a bool function > +bool Field_datetime_hires::get_date(MYSQL_TIME *ltime, uint fuzzydate) > +{ > + ulonglong packed= read_bigendian(ptr, Field_datetime_hires::pack_length()); > + unpack_time(sec_part_unshift(packed, dec), ltime); > + if (!packed) > + return fuzzydate & TIME_NO_ZERO_DATE; Should be: return (fuzzydate & TIME_NO_ZERO_DATE) != 0; > + if (!ltime->month || !ltime->day) > + return !(fuzzydate & TIME_FUZZY_DATE); > + return 0; > +} > + > +uint32 Field_datetime_hires::pack_length() const > +{ > + return 5 + sec_part_bytes[dec]; > +} move to include file. <cut> > @@ -8217,7 +8020,7 @@ Field_blob::unpack_key(uchar *to, const > length+= *from++ << 8; > > /* put the length into the record buffer */ > - put_length(to, length); > + store_length(to, packlength, length, table->s->db_low_byte_first); Note that the old format was not depending on table->s->db_low_byte_first ! it should alway stored in low byte first order! Change above to store_lowendian() > === modified file 'sql/field.h' > --- sql/field.h 2010-03-17 18:15:41 +0000 > +++ sql/field.h 2011-03-24 11:30:03 +0000 > @@ -22,8 +22,13 @@ > #pragma interface /* gcc class implementation */ > #endif > > +#ifdef WORDS_ARCH_BIGENDIAN > +#define ARCH_BIGENDIAN 1 > +#else > +#define ARCH_BIGENDIAN 0 > +#endif > + <cut> > class Field_datetime :public Field_temporal { ... > enum Item_result cmp_type () const { return TIME_RESULT; } remove cmp_type() as it's already defined identically by Field_temporal > === modified file 'sql/item.cc' > --- sql/item.cc 2010-11-18 13:11:18 +0000 > +++ sql/item.cc 2011-03-24 14:55:52 +0000 > @@ -461,6 +461,34 @@ void Item::print_item_w_name(String *str > } > > > +void Item::print_value(String *str) > +{ > + char buff[MAX_FIELD_WIDTH]; > + String *ptr, tmp(buff,sizeof(buff),str->charset()); > + ptr= val_str(&tmp); > + if (!ptr) > + str->append("NULL"); > + else > + { > + switch (result_type()) > + { > + default: > + DBUG_ASSERT(0); > + case STRING_RESULT: > + str->append('\''); > + str->append(*ptr); > + str->append('\''); Shouldn't we also escape \' ? (As we never know what the print is used for) Doing 'append_unescaped(str, ptr->ptr(), ptr->length())' instead of above would fix that. (Yes, I know that the original code didn't do that, but that was also a bug). > + break; > + case DECIMAL_RESULT: > + case REAL_RESULT: > + case INT_RESULT: > + str->append(*ptr); > + break; > + } Shouldn't you have TIME_RESULT: here too ? If this never happens, you should at least have this as part of default to make it clear it wasn't forgotten. (Actually removing the default would be a good idea as then we will get a compiler warning if someone adds a new result type but don't take care if it here). > + } > +} > + > + > void Item::cleanup() > { > DBUG_ENTER("Item::cleanup"); > @@ -503,6 +531,45 @@ void Item::rename(char *new_name) > name= new_name; > } > > +Item_result Item::cmp_type() const > +{ > + switch(field_type()) { Add space after switch > bool Item::get_time(MYSQL_TIME *ltime) > { > return get_date(ltime, TIME_TIME_ONLY); > } You should probably add TIME_FUZZY_DATE to the above. This should fix the error that I have described at the end of the review. <cut> > /** > Traverse item tree possibly transforming it (replacing items). > @@ -915,13 +982,15 @@ bool Item_string::eq(const Item *item, b > > bool Item::get_date(MYSQL_TIME *ltime,uint fuzzydate) > { > - if (result_type() == STRING_RESULT) > + if (field_type() == MYSQL_TYPE_TIME) > + fuzzydate|= TIME_TIME_ONLY; > + if (result_type() == STRING_RESULT || fuzzydate & TIME_TIME_ONLY) > { Please add a comment under what conditions or why field_type() can be MYSQL_TYPE_TIME and result_type() is STRING_RESULT. (Looks like this only happens for Item_field(Field_time)) I am actually a bit confused by the fact that only Item_field can have TIME_RESULT. Why doesn't all time functions say they have TIME_RESULT? > @@ -2698,22 +2750,23 @@ void Item_param::set_time(MYSQL_TIME *tm > value.time= *tm; > value.time.time_type= time_type; > > + decimals= value.time.second_part > 0 ? MAX_SEC_PART_DIGITS : 0; > + > if (value.time.year > 9999 || value.time.month > 12 || > value.time.day > 31 || > (time_type != MYSQL_TIMESTAMP_TIME && value.time.hour > 23) || > - value.time.minute > 59 || value.time.second > 59) > + value.time.minute > 59 || value.time.second > 59 || > + value.time.second_part > MAX_SEC_PART_VALUE) > { > + Lazy_string_time str(&value.time); > make_truncated_value_warning(current_thd, MYSQL_ERROR::WARN_LEVEL_WARN, > + &str, time_type, 0); > set_zero_time(&value.time, MYSQL_TIMESTAMP_ERROR); > } > > state= TIME_VALUE; > maybe_null= 0; > max_length= max_length_arg; > - decimals= 0; > DBUG_VOID_RETURN; Please move setting of decimal to end of function (where the old setting was). all Item_param::set... functions first checks the arguments and then sets all values at the end (in the same place). (Just makes the function easier to read) > } > > @@ -2790,10 +2843,12 @@ bool Item_param::set_from_user_var(THD * > case REAL_RESULT: > set_double(*(double*)entry->value); > item_type= Item::REAL_ITEM; > + param_type= MYSQL_TYPE_DOUBLE; > break; Here you set the param_type, but I wonder if there could be a problem that the param_type is different over two execution of a prepared statement. I tried some things to see if there is a problem, but couldn't see any behavior changes because of this, so this change may be safe. What did this change fix? (At at least a code comment why param_type is reset here) > @@ -4651,12 +4693,7 @@ Item *Item_field::equal_fields_propagato > item= this; > else if (field && (field->flags & ZEROFILL_FLAG) && IS_NUM(field->type())) > { > - if (item && field->type() != FIELD_TYPE_TIMESTAMP && cmp_context != INT_RESULT) > + if (item && (cmp_context == STRING_RESULT || cmp_context == (Item_result)-1)) When adding to 5.3, don't forget to change to use IMPOSSIBLE_RESULT > @@ -6852,11 +6879,19 @@ void resolve_const_item(THD *thd, Item * <cut> > + case TIME_RESULT: > + { > + bool is_null; > + Item **ref_copy= ref; Please add a short comment why we don't create an Item_datetime value here (as this is what we do in all the other cases) > + get_datetime_value(thd, &ref_copy, &new_item, comp_item, &is_null); > + if (is_null) > + new_item= new Item_null(name); > + break; > + } > @@ -7009,6 +7047,21 @@ int stored_field_cmp_to_item(THD *thd, F > field_val= field->val_decimal(&field_buf); > return my_decimal_cmp(item_val, field_val); > } Add comment: /* We have to check field->cmp_type() here as res_type is TIME_RESULT if either field or item is of type TIME_RESULT */ > + if (field->cmp_type() == TIME_RESULT) > + { > + MYSQL_TIME field_time, item_time; Fix indentation (should be 2 space, not 4) > + if (field->type() == MYSQL_TYPE_TIME) > + { > + field->get_time(&field_time); > + item->get_time(&item_time); > + } > + else > + { > + field->get_date(&field_time, TIME_FUZZY_DATE | TIME_INVALID_DATES); > + item->get_date(&item_time, TIME_FUZZY_DATE | TIME_INVALID_DATES); > + } > + return my_time_compare(&field_time, &item_time); > + } Looking at the code, it would probably be simpler if we replaced all the 'if (res_type)' with a swtich. > @@ -7048,6 +7101,8 @@ Item_cache* Item_cache::get_cache(const > return new Item_cache_str(item); > case ROW_RESULT: > return new Item_cache_row(); > + case TIME_RESULT: > + return new Item_cache_int(MYSQL_TYPE_DATETIME); Shouldn't this be a double or decimal as the time can contain microseconds? Looks like a potential bug! > === modified file 'sql/item.h' > --- sql/item.h 2010-07-30 13:35:06 +0000 > +++ sql/item.h 2011-03-24 11:30:03 +0000 > @@ -569,7 +569,8 @@ class Item { > virtual bool send(Protocol *protocol, String *str); > virtual bool eq(const Item *, bool binary_cmp) const; > virtual Item_result result_type() const { return REAL_RESULT; } > - virtual Item_result cast_to_int_type() const { return result_type(); } > + virtual Item_result cmp_type() const; Please add documentation what's the difference between cmp_type() and result_type() and when one should choose which. By the way, you can remove Field->cast_to_int_type() as this is always same as field->result_type(). I also noticed that you have a Field->cmp_type() but there is no Item_field->cmp_type() mapped to Field->cmp_type(). I couldn't As I assume that for all fields the following should hold: Item_field->cmp_type() == Item_field->field->cmp_type() If not that needs to be changed or properly documented as otherwise things are very confusing! > @@ -3031,7 +3022,7 @@ class Item_cache_int: public Item_cache > protected: > longlong value; > public: > - Item_cache_int(): Item_cache(), > + Item_cache_int(): Item_cache(MYSQL_TYPE_LONGLONG), > value(0) {} > Item_cache_int(enum_field_types field_type_arg): > Item_cache(field_type_arg), value(0) {} > @@ -3043,8 +3034,13 @@ class Item_cache_int: public Item_cache > String* val_str(String *str); > my_decimal *val_decimal(my_decimal *); > enum Item_result result_type() const { return INT_RESULT; } > - bool result_as_longlong() { return TRUE; } > bool cache_value(); > + Item *clone_item() > + { > + Item_cache_int *item= new Item_cache_int(cached_field_type); > + item->store(this, value); > + return item; > + } > }; Don't understand why Item_cache_int() needs a clone_item when Item_cache_real() or Item_cache_decimal() dosn't have one. If this was needed, please check if the other Item_cache_xxx() functions needs it. > === modified file 'sql/item_cmpfunc.cc' <cut> Please add documentation for this function: > @@ -891,65 +713,15 @@ int Arg_comparator::set_cmp_func(Item_re > Item **a1, Item **a2, > Item_result type) <cut> > @@ -1026,9 +761,14 @@ Item** Arg_comparator::cache_converted_c > Item **cache_item, > Item_result type) > { > - /* Don't need cache if doing context analysis only. */ > + /* > + Don't need cache if doing context analysis only. > + Also, get_datetime_value creates Item_cache internally. > + Unless fixed, we should not do it here. > + */ > if (!thd_arg->is_context_analysis_only() && > - (*value)->const_item() && type != (*value)->result_type()) > + (*value)->const_item() && type != (*value)->result_type() && > + type != TIME_RESULT) > { > Item_cache *cache= Item_cache::get_cache(*value, type); > cache->setup(*value); Why don't we cache it there instead of doing it in get_datetime_value()? This would save us one if and 3 test for every call to get_datetime_value(). You could here call get_datetime_value() and use this value in the cache. If this doesn't easily work (as indicated on IRC), at least expand the comment to explain the issue so that we can later try to fix it in a better way. > @@ -1071,81 +806,120 @@ void Arg_comparator::set_datetime_cmp_fu > DESCRIPTION > Retrieves the correct DATETIME value from given item for comparison by the > compare_datetime() function. > + > + If the value should be compared as time (TIME_RESULT), it's retrieved as > + MYSQL_TIME. Otherwise it's read as a number/string and converted to time. > + Constant items are cached, so the convertion is only done once for them. > + > + Note the f_type behavior: if the item can be compared as time, then > + f_type is this item's field_type(). Otherwise it's field_type() of > + warn_item (which is the other operand of the comparison operator). > + This logic provides correct string/number to date/time conversion > + depending on the other operand (when comparing a string with a date, it's > + parsed as a date, when comparing a string with a time it's parsed as a time) > > RETURN > + MYSQL_TIME value, packed in a longlong, suitable for comparison. > */ > Please add the following to the comment: For example, if we are executing WHERE datetime_6_column = 20010101000001.000020 for the first call the 'number-item' will have item->cmp_type == REAL_RESULT. It will create an item_cache with cmp_type() == TIME_RESULT and item->result_type() == INT_RESULT which will replace the item. No other item than an item_cache item will have cmp_type() == TIME_RESULT and result_type() == INT_RESULT, which makes the TIME_RESULT case safe. > longlong > get_datetime_value(THD *thd, Item ***item_arg, Item **cache_arg, > Item *warn_item, bool *is_null) > { > + longlong UNINIT_VAR(value); > Item *item= **item_arg; > + enum_field_types f_type= warn_item->field_type(); > + timestamp_type t_type= > + f_type == MYSQL_TYPE_DATE ? MYSQL_TIMESTAMP_DATE : > + f_type == MYSQL_TYPE_TIME ? MYSQL_TIMESTAMP_TIME : > + MYSQL_TIMESTAMP_DATETIME; > + > + switch (item->cmp_type()) { > + case TIME_RESULT: > + /* if it's our Item_cache_int, as created below, we simply use the value */ > + if (item->result_type() == INT_RESULT) > + value= item->val_int(); Please also set cache_item= 0 to make the end function test simpler. <cut> > + default: DBUG_ASSERT(0); > } > - if (*is_null) > + if ((*is_null= item->null_value)) > return ~(ulonglong) 0; > + if (cache_arg && item->const_item() && item->type() != Item::CACHE_ITEM) > { With my suggested change above, you can remove test of CACHE_ITEM. Another option would be to add to function start: if (item->type() == Item::CACHE_ITEM) { value= item->val_int(); *is_null= item->null_value; return; } Which would save us the calcuation of f_type and t_type for all cached objects. I would also suggest delaying calcuation of t_type until it's needed as in most cases we will not need it (constant items or fields) <cut> > +int Arg_comparator::compare_e_datetime() > +{ > + bool a_is_null, b_is_null; > + longlong a_value, b_value; > + > + if (set_null) > + owner->null_value= 0; Remove the above set; No other compare_e... function sets null_value > int Arg_comparator::compare_string() > { > @@ -2247,9 +1963,7 @@ void Item_func_between::fix_length_and_d > { > max_length= 1; > int i; > - bool datetime_found= FALSE; > - int time_items_found= 0; > - compare_as_dates= TRUE; > + compare_as_dates= 0; > THD *thd= current_thd; Please move declarations first and assignment of variables later. (Even better, move 'int i' to the block where it's used) > /* > @@ -2265,43 +1979,33 @@ void Item_func_between::fix_length_and_d > return; > > /* > + When comparing as date/time, we need to convert non-temporal values > + (e.g. strings) to MYSQL_TIME. get_datetime_value() doees it doees -> does > + automatically when one of the operands is a date/time. But here we > + may need to compare two strings as dates (str1 BETWEEN str2 AND date). > + For this to work, we need to know what date/time type we compare > + strings as. > */ > + if (cmp_type == TIME_RESULT) > { > for (i= 0; i < 3; i++) > { > + if (args[i]->cmp_type() == TIME_RESULT) > { > - datetime_found= TRUE; > + if (args[i]->field_type() != MYSQL_TYPE_TIME || > + (args[i]->field_type() == MYSQL_TYPE_TIME && compare_as_dates==0)) > + compare_as_dates= args[i]; > continue; > } > } > } I find it strange that you set compare_as_dates to the last not MYSQL_TYPE_TIME column as this is used to give warnings as the warning may in theory come from some other field. I tried to make a test case to show the problem but didn't succeed :( > @@ -2319,27 +2023,46 @@ void Item_func_between::fix_length_and_d > longlong Item_func_between::val_int() > { // ANSI BETWEEN > DBUG_ASSERT(fixed == 1); > - if (compare_as_dates) > + > + switch (cmp_type) { > + case TIME_RESULT: > { > - int ge_res, le_res; > + THD *thd= current_thd; > + longlong value, a, b; > + Item *cache, **ptr; > + bool value_is_null, a_is_null, b_is_null; > + Add a comment here that it's ok to have cache and ptr on stack as if we created a newcache item, it will be taken care of by the call to change_item_tree(). <cut> > + if (!a_is_null && !b_is_null) > + return (longlong) ((value >= a && value <= b) != negated); > + if (a_is_null && b_is_null) > + null_value=1; > + else if (a_is_null) > + null_value= value <= b; // not null if false range. > else > + null_value= value >= a; > + break; A slightly simpler if: if (a_is_null) null_value= b_is_null || value <= b; // not null if false range. else null_value= value >= a; <cut> > @@ -3193,6 +2940,13 @@ void Item_func_coalesce::fix_length_and_ > { > cached_field_type= agg_field_type(args, arg_count); > agg_result_type(&hybrid_type, args, arg_count); > + Item_result cmp_type; > + agg_cmp_type(&cmp_type, args, arg_count); > + if (cmp_type == TIME_RESULT) > + { > + count_real_length(); > + return; > + } Add a comment that this code is not needed when we have TIME_RESULT also as a result type. > === modified file 'sql/item_create.cc' > --- sql/item_create.cc 2010-07-02 18:30:47 +0000 > +++ sql/item_create.cc 2011-03-08 18:41:58 +0000 > @@ -5044,6 +5044,14 @@ find_qualified_function_builder(THD *thd > return & Create_sp_func::s_singleton; > } > > +static inline const char* item_name(Item *a, String *str) > +{ > + if (a->name) > + return a->name; > + str->length(0); > + a->print(str, QT_ORDINARY); > + return str->c_ptr_safe(); > +} Add a comment that 'item_name()' always returns a \0 ended string. > @@ -5051,6 +5059,8 @@ create_func_cast(THD *thd, Item *a, Cast > CHARSET_INFO *cs) <cut> > case ITEM_CAST_DATETIME: > - res= new (thd->mem_root) Item_datetime_typecast(a); > + { > + uint len; > + if (c_len) > + { > + errno= 0; > + len= strtoul(c_len, NULL, 10); Note that strtoul() on error doesn't set errno but my_errno (if we are using using our version of strtoul()). Anyway you don't have to check errno as it will return UTYPE_MAX on error, which is bigger than MAX_DATETIME_PRECISION. In general, it's better to use my_strtoll10() than strtoul() because this is faster and it returns the error in a variable that is easier to check than errno. Please also fix all other calls to strtoul() in this function > + if (errno != 0 || len > MAX_DATETIME_PRECISION) > + { > + my_error(ER_TOO_BIG_PRECISION, MYF(0), len, > + item_name(a, &buf), MAX_DATETIME_PRECISION); > + return NULL; > + } > + } > + else > + len= 0; Set len= 0 before the 'if'. This is faster and less code lines. > case ITEM_CAST_DECIMAL: > { > ulong len= 0; > @@ -5083,8 +5111,8 @@ create_func_cast(THD *thd, Item *a, Cast > decoded_size= strtoul(c_len, NULL, 10); > if (errno != 0) > { > - my_error(ER_TOO_BIG_PRECISION, MYF(0), c_len, a->name, > - DECIMAL_MAX_PRECISION); > + my_error(ER_TOO_BIG_PRECISION, MYF(0), decoded_size, > + item_name(a, &buf), DECIMAL_MAX_PRECISION); Good bug fix, but still not perfect as the value could be a longlong and then you get a strange number in the result. Best would be to print the result as ulonglong, but that would require us to add another error message which is not optimal either. Example: select cast('2001-01-01' as decimal(99999999999999)); -> ERROR 1426 (42000): Too big precision 276447231 specified for column '2001-01-01'. Maximum is 65 <cut> > @@ -5135,7 +5163,7 @@ create_func_cast(THD *thd, Item *a, Cast > decoded_size= strtoul(c_len, NULL, 10); > if ((errno != 0) || (decoded_size > MAX_FIELD_BLOBLENGTH)) > { > - my_error(ER_TOO_BIG_DISPLAYWIDTH, MYF(0), "cast as char", MAX_FIELD_BLOBLENGTH); > + my_error(ER_TOO_BIG_DISPLAYWIDTH, MYF(0), "cast as char", MAX_FIELD_BLOBLENGTH); > return NULL; > } > len= (int) decoded_size; > > === modified file 'sql/item_func.cc' > --- sql/item_func.cc 2010-11-23 13:08:10 +0000 > +++ sql/item_func.cc 2011-03-19 13:54:46 +0000 > @@ -940,8 +940,7 @@ longlong Item_func_signed::val_int() > longlong value; > int error; > > - if (args[0]->cast_to_int_type() != STRING_RESULT || > - args[0]->result_as_longlong()) > + if (args[0]->cast_to_int_type() != STRING_RESULT) > { > value= args[0]->val_int(); > null_value= args[0]->null_value; > @@ -982,8 +981,7 @@ longlong Item_func_unsigned::val_int() > value= 0; > return value; > } > - else if (args[0]->cast_to_int_type() != STRING_RESULT || > - args[0]->result_as_longlong()) > + else if (args[0]->cast_to_int_type() != STRING_RESULT) > { > value= args[0]->val_int(); > null_value= args[0]->null_value; > @@ -2220,10 +2218,10 @@ double Item_func_units::val_real() > void Item_func_min_max::fix_length_and_dec() > { > int max_int_part=0; > decimals=0; > max_length=0; > maybe_null=0; > + thd= current_thd; We should consider in the future adding thd as an argument to fix_length_and_dec(). The alternative is that in the functions one wants to access thd, one implements fix_fields() that just sets thd and call parent fix_fields(). As almost all Item_date_func's uses thd, it may be a good idea to add thd to this item and set it in fix_fields(). This would remove a LOT of calls to current_thd. <cut> > +bool Item_func_min_max::get_date(MYSQL_TIME *ltime, uint fuzzy_date) > { > longlong UNINIT_VAR(min_max); > + DBUG_ASSERT(fixed == 1); > + if (!compare_as_dates) > + return Item_func::get_date(ltime, fuzzy_date); Add a comment in which case the above may not be true. > > for (uint i=0; i < arg_count ; i++) > { > Item **arg= args + i; > bool is_null; > + longlong res= get_datetime_value(thd, &arg, 0, compare_as_dates, &is_null); > > /* Check if we need to stop (because of error or KILL) and stop the loop */ Using 'compare_as_dates' can give a strange warning: select greatest(cast("2001-01-5" as date), "2002-02-7aaa"); -> Incorrect date value: '2002-02-7aaa' for column 'cast("2001-01-5" as date)' at row 1 In cases like this when we don't store the value in a field, it would may be better to just give an error: Incorrect date value: '2002-02-7aaa' for function 'greatest' at row 1 We could get the name by just sending 'this' to the get_datetime_value instead of a found argument and we could choose error message based if the 'warn_item->real_item()->type()' is a FIELD_ITEM or not. > if (thd->is_error()) > { > null_value= 1; > + return 1; > } > > if ((null_value= args[i]->null_value)) > + return 1; You could combine the above two tests to: if (thd->is_error() || args[i]->null_value) { null_value= 1; return 1; } > if (i == 0 || (res < min_max ? cmp_sign : -cmp_sign) > 0) > min_max= res; > } > + unpack_time(min_max, ltime); > + if (compare_as_dates->field_type() == MYSQL_TYPE_DATE) > + ltime->time_type= MYSQL_TIMESTAMP_DATE; I would rather cache the correct value on upper level and store it in ltime->time_type directly. The above test also looks a bit suspicion if you would compare date with time. greatest(cast("2001-01-5" as date), cast("10:20:05" as time)) -> 2001-01-05 another strange effect is: greatest(cast("0-0-0" as date), cast("10:20:05" as time)) -> 0000-00-00 but: greatest(cast("0-0-0" as date), cast("10:20:05" as time)) = "0000-00-00" -> 0 (Which is wrong) compare to: concat(greatest(cast("0-0-0" as date), cast("10:20:05" as time))) = "0000-00-00" -> true This shows what is really happening: cast(greatest(cast("0-0-0" as date), cast("10:20:05" as time)) as datetime(6)) -> 0000-00-00 10:20:05.000000 Suggested fix: a) Do as before, when comparing datetime or date with time, convert to time, not datetime. b) Give an error when comparing date or datetime with time. c) When doing compare of date and time, upgrade type to compare to DATETIME. > @@ -2319,19 +2308,14 @@ String *Item_func_min_max::val_str(Strin > DBUG_ASSERT(fixed == 1); > if (compare_as_dates) > { > + MYSQL_TIME ltime; > + if (get_date(&ltime, TIME_FUZZY_DATE)) > return 0; > + char buf[MAX_DATE_STRING_REP_LENGTH]; > + int len= my_TIME_to_str(&ltime, buf, decimals); > + str->copy(buf, len, collation.collation); > + return str; A better way would be to to do: str->alloc(MAX_DATE_STRING_REP_LENGTH); str->set_charset(collation.collation). str->length(my_TIME_to_str(&ltime, str->ptr(), decimals)); This avoids the extra strcpy() (One should also check the return value for alloc() and copy() here). You could also make item_timefunc.cc:make_datetime() public and just call this. <cut> > @@ -3979,7 +3968,7 @@ double user_var_entry::val_real(my_bool > } > case STRING_RESULT: > return my_atof(value); // This is null terminated > - case ROW_RESULT: > + default: Add instead ROW_RESULT and TIME_RESULT here. This is safer as it's less chance we miss something when we add more result types later. > DBUG_ASSERT(1); // Impossible > break; > } > @@ -4010,7 +3999,7 @@ longlong user_var_entry::val_int(my_bool > int error; > return my_strtoll10(value, (char**) 0, &error);// String is null terminated > } > - case ROW_RESULT: > + default: Same as above. > DBUG_ASSERT(1); // Impossible > break; > } > @@ -4042,7 +4031,7 @@ String *user_var_entry::val_str(my_bool > case STRING_RESULT: > if (str->copy(value, length, collation.collation)) > str= 0; // EOM error > - case ROW_RESULT: > + default: > DBUG_ASSERT(1); // Impossible > break; Same as above. > } > @@ -4069,7 +4058,7 @@ my_decimal *user_var_entry::val_decimal( > case STRING_RESULT: > str2my_decimal(E_DEC_FATAL_ERROR, value, length, collation.collation, val); > break; > - case ROW_RESULT: > + default: Same as above. > DBUG_ASSERT(1); // Impossible > break; > } > > === modified file 'sql/item_timefunc.cc' <cut> > -static bool sec_to_time(longlong seconds, bool unsigned_flag, MYSQL_TIME *ltime) > +static bool sec_to_time(double seconds, MYSQL_TIME *ltime) > { > uint sec; > > bzero((char *)ltime, sizeof(*ltime)); > + > + ltime->time_type= MYSQL_TIMESTAMP_TIME; > > if (seconds < 0) > { > - if (unsigned_flag) > - goto overflow; > ltime->neg= 1; > if (seconds < -3020399) > goto overflow; Shouldn't this be: if (seconds < -3020399.999999) Same for the other test. (Instead of a constant number this should of course be TIME_MAX_VALUE_SECONDS + TIME_MAX_SECOND_PART/1e6) Add also tests for this in the test suite! > @@ -211,9 +95,8 @@ static bool sec_to_time(longlong seconds > ltime->minute= TIME_MAX_MINUTE; > ltime->second= TIME_MAX_SECOND; > > - char buf[22]; > - int len= (int)(longlong10_to_str(seconds, buf, unsigned_flag ? 10 : -10) > - - buf); > + char buf[100]; > + uint len= snprintf(buf, sizeof(buf), "%.80g", ltime->neg ? -seconds: seconds); I don't think we want to have 80 digit precission here; Looks strange.; 30 should be more than enough. <cut> > @@ -578,6 +464,10 @@ static bool extract_date_time(DATE_TIME_ > l_time->minute > 59 || l_time->second > 59) > goto err; > > + if ((fuzzy_date & TIME_NO_ZERO_DATE) && > + (l_time->year == 0 || l_time->month == 0 || l_time->day == 0)) > + goto err; > + This is something we didn't check before. If this is a compatibily issue, you should list it in your matrix. I tried to get the above to execute, but could not. I tried with: select str_to_date("1997-00-04 22:23:00","%Y-%m-%D"); but this returned "1997-00-04" independent of the sql mode. Please add a comment when the above case will be used. > @@ -1569,8 +1459,9 @@ void Item_func_curdate_local::store_now_ > */ > void Item_func_curdate_utc::store_now_in_TIME(MYSQL_TIME *now_time) > { > - my_tz_UTC->gmt_sec_to_TIME(now_time, > - (my_time_t)(current_thd->query_start())); > + THD *thd= current_thd; > + my_tz_UTC->gmt_sec_to_TIME(now_time, thd->query_start()); > + thd->time_zone_used= 1; > /* > We are not flagging this query as using time zone, since it uses fixed > UTC-SYSTEM time-zone. Why do you flag the time_zone_used when the function comment says that you don't need to do this? You also don't set time_zone_used in Item_func_now_utc::store_now_in_TIME() <cut> <cut> > bool Item_func_sysdate_local::get_date(MYSQL_TIME *res, > uint fuzzy_date __attribute__((unused))) > { > + MYSQL_TIME ltime; > store_now_in_TIME(&ltime); > *res= ltime; > return 0; > } Why not do directly: store_now_in_TIME(res); Actaully, I don't undestand why Item_func_sysdate_local() have to have a :store_now_in_TIME() function at all. As this function can only be called by a Item_func_sysdate_local object and we only call it from this place it would be more logical to have the code from store_now_in_time() here. It actually doesn't make sense that this function inherits from Item_func_now() as we get an extra store_now_in_TIME() call from Item_func_now::fix_length_and_dec() that we don't need or want. Not critical to do it now, but should be done eventually. <cut> > @@ -2050,42 +1845,14 @@ bool Item_func_from_unixtime::get_date(M > void Item_func_convert_tz::fix_length_and_dec() > { > collation.set(&my_charset_bin); > - decimals= 0; > - max_length= MAX_DATETIME_WIDTH*MY_CHARSET_BIN_MB_MAXLEN; > + max_length= MAX_DATETIME_WIDTH; Should probably be: max_length= MAX_DATETIME_WIDTH*MY_CHARSET_BIN_MB_MAXLEN; This is how all the other functions are defined. > + decimals= args[0]->decimals; > + if (decimals && decimals != NOT_FIXED_DEC) > + max_length+= min(decimals, MAX_SEC_PART_DIGITS) + 1; > maybe_null= 1; > } I was actually wondering why we don't have an Item_temporal_func::fix_fields() function that was defined to set the above as this should be the default for all DATETIME results. This would allow us to do something like: bool Item_func_convert_tz::fix_fields(thd, items) { decimals= args[0]->decimals; decimals= min(decimals, MAX_SEC_PART_DIGITS); return Item_temporal_func::fix_fields(thd, items); } And of course Item_temporal_func::fix_fields() could then check if decimals > MAX_SEC_PART_DIGITS and give an error. (Just an idea) <cut> > @@ -2137,8 +1908,7 @@ void Item_date_add_interval::fix_length_ > > collation.set(&my_charset_bin); > maybe_null=1; > - max_length=MAX_DATETIME_FULL_WIDTH*MY_CHARSET_BIN_MB_MAXLEN; > + max_length=MAX_DATETIME_WIDTH*MY_CHARSET_BIN_MB_MAXLEN; Please move setting of the above to where you do: if (decimals) max_length+= min(decimals, MAX_SEC_PART_DIGITS) + 1; So that the result is: max_length= MAX_DATETIME_WIDTH*MY_CHARSET_BIN_MB_MAXLEN; if (decimals) max_length+= min(decimals, MAX_SEC_PART_DIGITS) + 1; This makes it much easier to ensure that max_length is correct > @@ -2146,7 +1916,11 @@ void Item_date_add_interval::fix_length_ > > - If first arg is a MYSQL_TYPE_DATETIME result is MYSQL_TYPE_DATETIME > - If first arg is a MYSQL_TYPE_DATE and the interval type uses hours, > + minutes or seconds then type is MYSQL_TYPE_DATETIME > + otherwise it's MYSQL_TYPE_DATE > + - if first arg is a MYSQL_TYPE_TIME and the interval type isn't using > + anything larger than days, then the result is MYSQL_TYPE_TIME, > + otherwise - MYSQL_TYPE_DATETIME. > - Otherwise the result is MYSQL_TYPE_STRING > (This is because you can't know if the string contains a DATE, MYSQL_TIME or > DATETIME argument) > @@ -2163,6 +1937,20 @@ void Item_date_add_interval::fix_length_ > else > cached_field_type= MYSQL_TYPE_DATETIME; > } > + else if (arg0_field_type == MYSQL_TYPE_TIME) > + { > + if (int_type >= INTERVAL_DAY && int_type != INTERVAL_YEAR_MONTH) > + cached_field_type= arg0_field_type; > + else > + cached_field_type= MYSQL_TYPE_DATETIME; > + } > + if (int_type == INTERVAL_MICROSECOND || int_type >= INTERVAL_DAY_MICROSECOND) > + decimals= 6; > + else > + decimals= args[0]->decimals; Shouldn't we above use: decimals= min(args[0]->decimals, 6); I suspect that some of the other code may find it confusing if decimals have a too high value. I tried: select 20010101000203.000000004 + interval 1 day; and this did die in an assert, which seams to indicate the the above change is needed. > + if (decimals) > + max_length+= min(decimals, MAX_SEC_PART_DIGITS) + 1; > + value.alloc(max_length); > } > > > @@ -2172,7 +1960,7 @@ bool Item_date_add_interval::get_date(MY > { > INTERVAL interval; > > - if (args[0]->get_date(ltime, TIME_NO_ZERO_DATE) || > + if (args[0]->get_date(ltime, TIME_NO_ZERO_DATE|TIME_FUZZY_DATE) || Please add space around | to make the above readable > void Item_func_add_time::fix_length_and_dec() > { > enum_field_types arg0_field_type; > - decimals=0; > - max_length=MAX_DATETIME_FULL_WIDTH*MY_CHARSET_BIN_MB_MAXLEN; > + max_length= MAX_DATETIME_WIDTH; Move setting of max_length down before 'if (decimals)' (see earlier comment). > + decimals= max(args[0]->decimals, args[1]->decimals); Add: decimals= min(decimals, MAX_DATETIME_PRECISION); <cut> > + MYSQL_TIME copy= *ltime; > + Lazy_string_time str(&copy); > + > + check_time_range(ltime, decimals, &was_cut); > + if (was_cut) > + make_truncated_value_warning(current_thd, MYSQL_ERROR::WARN_LEVEL_WARN, > + &str, MYSQL_TIMESTAMP_TIME, NullS); Move setting of Lazy_string_time str(&copy) inside 'if (was_cut)...' > +bool Item_func_timediff::get_date(MYSQL_TIME *ltime, uint fuzzy_date) > { > DBUG_ASSERT(fixed == 1); > longlong seconds; > long microseconds; > - int l_sign= 1; > - MYSQL_TIME l_time1 ,l_time2, l_time3; > + int l_sign= 1, was_cut= 0; > + MYSQL_TIME l_time1,l_time2,l_time3; > + Lazy_string_time str(&l_time3); Move setting of 'str' to where it's used. > @@ -2899,6 +2535,9 @@ String *Item_func_timediff::val_str(Stri > l_time1.time_type != l_time2.time_type) > goto null_date; > > + if (fuzzy_date & TIME_NO_ZERO_IN_DATE) > + goto null_date; It took some time to find a test to trigger this code. Please add a comment: /* The following test maybe true when we are called by a function that require a datetime object. For example when doing: select timediff('...') + interval 1 month */ In any case, this should be moved to start of the function as there is no way this function can ever return anything else than NULL if this flag is set and there is no reason to calculate any else in this case. > @@ -2915,16 +2554,22 @@ String *Item_func_timediff::val_str(Stri > if (l_time1.neg && (seconds || microseconds)) > l_time3.neg= 1-l_time3.neg; // Swap sign of result > Add comment here: /* Adjust second so that we don't loose information from (long) cast. This is safe to to as calc_time_from_sec() will overflow also for INT_MAX32 and we will get the warning we expect */ > + set_if_smaller(seconds, INT_MAX32); > calc_time_from_sec(&l_time3, (long) seconds, microseconds); > > + if ((fuzzy_date & TIME_NO_ZERO_DATE) && (seconds == 0) && (microseconds == 0)) > + goto null_date; TIME_NO_ZERO_DATE means that neither day or month should not be 0 As this is always the case for time_diff() we could merge this test with the above 'if (fuzzy_date & TIME_NO_ZERO_IN_DATE)' For some examples where this now goes strange: dayofyear(timediff('0000-00-01 00:00:05', '0000-00-01 00:00:05')) -> NULL dayofyear(timediff('0000-00-01 00:00:05', '0000-00-01 00:00:04')) -> 0 dayofyear(cast('0000-00-01 00:00:05' as datetime)) -> NULL I think it should return NULL in all cases (which is what happens if you move the TIME_NO_ZERO_DATE test as suggested above). <cut>> > > @@ -3292,36 +2930,45 @@ get_date_time_result_type(const char *fo > void Item_func_str_to_date::fix_length_and_dec() > { > maybe_null= 1; > - decimals=0; > cached_field_type= MYSQL_TYPE_DATETIME; > - max_length= MAX_DATETIME_FULL_WIDTH*MY_CHARSET_BIN_MB_MAXLEN; > - cached_timestamp_type= MYSQL_TIMESTAMP_NONE; > + max_length= MAX_DATETIME_WIDTH+MAX_SEC_PART_DIGITS; Add '*MY_CHARSET_BIN_MB_MAXLEN' for completeness here and in the other places we set max_length.. <cut> > +++ sql/item_timefunc.h 2011-03-23 12:31:06 +0000 > @@ -323,147 +323,100 @@ class Item_func_unix_timestamp :public I > }; > > > -class Item_func_time_to_sec :public Item_int_func > +class Item_func_time_to_sec :public Item_real_func > { > public: > - Item_func_time_to_sec(Item *item) :Item_int_func(item) {} > - longlong val_int(); > + Item_func_time_to_sec(Item *item) :Item_real_func(item) {} > const char *func_name() const { return "time_to_sec"; } > + double val_real(); > void fix_length_and_dec() > { > maybe_null= TRUE; > - decimals=0; > - max_length=10*MY_CHARSET_BIN_MB_MAXLEN; > + decimals=args[0]->decimals; > + max_length=17; max_length= 17*MY_CHARSET_BIN_MB_MAXLEN; <cut> > +class Item_temporal_func: public Item_func > { > public: > + Item_temporal_func() :Item_func() {} > + Item_temporal_func(Item *a) :Item_func(a) {} > + Item_temporal_func(Item *a, Item *b) :Item_func(a,b) {} > + Item_temporal_func(Item *a, Item *b, Item *c) :Item_func(a,b,c) {} > enum Item_result result_type () const { return STRING_RESULT; } > + enum_field_types field_type() const { return MYSQL_TYPE_DATETIME; } > String *val_str(String *str); > longlong val_int(); > + double val_real(); > + bool get_date(MYSQL_TIME *res, uint fuzzy_date) { DBUG_ASSERT(0); return 1; } > my_decimal *val_decimal(my_decimal *decimal_value) > + { return val_decimal_from_date(decimal_value); } > + Field *tmp_table_field(TABLE *table) > + { return tmp_table_field_from_field_type(table, 0); } > int save_in_field(Field *field, bool no_conversions) > + { return save_date_in_field(field); } > }; > See earlier suggestion to add a fix_fields() function and add a 'THD' variable. The only problem with this is to decide upon the value of 'decimal' for functions that depends on it's arguments for the value of decimal. One way to do this would bo first call item->fix_length_and_dec() and then use the value of decimals and field_type() to calculate max_length. Having a Item_temporal_func::fix_fields() would also make it trivial to ensure we have always a legal value for decimals! We also would not have to do the test: max_length+= (decimals ? min(decimals, MAX_SEC_PART_DIGITS)+1 : 0); In a lot of places. <cut> > @@ -563,16 +502,10 @@ class Item_func_now_utc :public Item_fun > class Item_func_sysdate_local :public Item_func_now Inheriting from Item_temporal_func() may be better. (See earlier comment) <cut> > +class Item_func_sec_to_time :public Item_timefunc > { > public: > + Item_func_sec_to_time(Item *item) :Item_timefunc(item) {} > + bool get_date(MYSQL_TIME *res, uint fuzzy_date); > void fix_length_and_dec() > { > collation.set(&my_charset_bin); > maybe_null=1; > + decimals= args[0]->decimals; > + if (decimals != NOT_FIXED_DEC && decimals > MAX_SEC_PART_DIGITS) > + decimals= MAX_SEC_PART_DIGITS; I am not sure that NOT_FIXED_DEC works for time; It may be better to always limit this to MAX_SEC_PART_DIGITS. <cut> > +class Item_date_typecast :public Item_temporal_typecast > { > public: > - Item_date_typecast(Item *a) :Item_typecast_maybe_null(a) {} > + Item_date_typecast(Item *a) :Item_temporal_typecast(a) {} > const char *func_name() const { return "cast_as_date"; } > - String *val_str(String *str); > bool get_date(MYSQL_TIME *ltime, uint fuzzy_date); > - bool get_time(MYSQL_TIME *ltime); > const char *cast_type() const { return "date"; } > enum_field_types field_type() const { return MYSQL_TYPE_DATE; } > - Field *tmp_table_field(TABLE *table) > - { > - return tmp_table_field_from_field_type(table, 0); > - } > void fix_length_and_dec() > { > collation.set(&my_charset_bin); > + decimals= 0; You don't need to set decimals to 0; This is already done in the item constructor. Of course, if we have Item_temporal::fix_fields() this function would not be needed at all... > + max_length= MAX_DATE_WIDTH; > maybe_null= 1; <cut> > +class Item_time_typecast :public Item_temporal_typecast > { > public: > - Item_time_typecast(Item *a) :Item_typecast_maybe_null(a) {} > + Item_time_typecast(Item *a, uint dec_arg) > + :Item_temporal_typecast(a) { decimals= dec_arg; } > const char *func_name() const { return "cast_as_time"; } > + bool get_date(MYSQL_TIME *ltime, uint fuzzy_date); > const char *cast_type() const { return "time"; } > enum_field_types field_type() const { return MYSQL_TYPE_TIME; } > + void fix_length_and_dec() > { > + collation.set(&my_charset_bin); > + maybe_null= 1; > + max_length= MIN_TIME_WIDTH; > + if (decimals == NOT_FIXED_DEC) > + decimals= args[0]->decimals; You proabaly should test for AUTO_SEC_PART_DIGITS as this is the define used in sql_yacc.yy > + if (decimals && decimals != NOT_FIXED_DEC) > + max_length+= min(decimals, MAX_SEC_PART_DIGITS) + 1; You need to limit decimals to be <= MAX_SEC_PART_DIGITS or you will get an assert in some code (see earlier test). However, I agree that this is nice: create table t1 (a double); insert into t1 values(1.0001),(1.000000000001) select time(a) from t1; -> | 00:00:01.000100 | | 00:00:01 | If I change the create to: create table t1 (a double(10,10)); the above will cause an assert in: my_time.c:1047: my_time_to_str: Assertion `digits >= 0 && digits <= 6' failed. <cut> > +class Item_func_last_day :public Item_datefunc > { > public: > - Item_func_last_day(Item *a) :Item_date(a) {} > + Item_func_last_day(Item *a) :Item_datefunc(a) {} > const char *func_name() const { return "last_day"; } > bool get_date(MYSQL_TIME *res, uint fuzzy_date); > + void fix_length_and_dec() > + { > + maybe_null=1; Better if Item_datefunc::fix_length_and_dec() or Item_temporal_func::fix_fields() sets maybe_null= 1 by default. > === modified file 'sql/log_event.cc' In the file you have: #ifdef MYSQL_CLIENT ... #ifdef MYSQL_CLIENT void free_table_map_log_event(...) #endif #endif Please remove the inner #ifdef MYSQL_CLIENT > @@ -3598,7 +3614,7 @@ bool Start_log_event_v3::write(IO_CACHE* > int2store(buff + ST_BINLOG_VER_OFFSET,binlog_version); > memcpy(buff + ST_SERVER_VER_OFFSET,server_version,ST_SERVER_VER_LEN); > if (!dont_set_created) > - created= when= get_time(); > + created= get_time(); Add a comment before get_time(): /* get_time sets 'when' and 'when_sec_part' as a side effect */ <cut> > @@ -1000,16 +1004,27 @@ class Log_event > { return 0; } > virtual bool write_data_body(IO_CACHE* file __attribute__((unused))) > { return 0; } > - inline time_t get_time() > + inline my_time_t get_time() > { > THD *tmp_thd; > if (when) > return when; > if (thd) > + { > + when= thd->start_time; > + when_sec_part= thd->start_time_sec_part; > + return when; > + } thd should alway be true. Can you at some point in time check when this is not the case and if needed set thd for the class ! > if ((tmp_thd= current_thd)) current_thd() should never fail. If it can, there should be a comment in the code about this! > + { > + when= tmp_thd->start_time; > + when_sec_part= tmp_thd->start_time_sec_part; > + return when; > + } > + my_hrtime_t hrtime= my_hrtime(); > + when= hrtime_to_my_time(hrtime); > + when_sec_part= hrtime_sec_part(hrtime); > + return when; > } > #endif > virtual Log_event_type get_type_code() = 0; > > === modified file 'sql/mysql_priv.h' <cut> > +/* > + to unify the code that differs only in the argument passed to the > + error message (string vs. number) > + > + We pass this container around, and only convert the number > + to a string when necessary. > +*/ Good idea, but please consider using a better name for the string. This is more a value wrapper than an string. > +class Lazy_string > +{ > +public: > + Lazy_string() {} > + virtual ~Lazy_string() {} > + virtual void copy(String *str) const = 0; > +}; > + <cut> > +class Lazy_string_dbl: public Lazy_string dbl -> double (No reason to save a couple of characters here) <cut> > === modified file 'sql/set_var.cc' > --- sql/set_var.cc 2010-10-20 18:21:40 +0000 > +++ sql/set_var.cc 2011-03-01 12:24:36 +0000 > @@ -2715,38 +2715,38 @@ int set_var_collation_client::update(THD > > bool sys_var_timestamp::check(THD *thd, set_var *var) > { > + double val= var->value->val_real(); > + if (val < 0 || val > MY_TIME_T_MAX) > { > my_message(ER_UNKNOWN_ERROR, > "This version of MySQL doesn't support dates later than 2038", > MYF(0)); > return TRUE; > } > + var->save_result.ulonglong_value= hrtime_from_time(var->value->val_real()); > return FALSE; > } > > > bool sys_var_timestamp::update(THD *thd, set_var *var) > { > - thd->set_time((time_t) var->save_result.ulonglong_value); > + my_hrtime_t hrtime = { var->save_result.ulonglong_value }; > + thd->set_time(hrtime); > return FALSE; > } > > > void sys_var_timestamp::set_default(THD *thd, enum_var_type type) > { > - thd->user_time=0; > + thd->user_time.val= 0; > } > > > uchar *sys_var_timestamp::value_ptr(THD *thd, enum_var_type type, > LEX_STRING *base) > { > - thd->sys_var_tmp.long_value= (long) thd->start_time; > - return (uchar*) &thd->sys_var_tmp.long_value; > + thd->sys_var_tmp.double_value= thd->start_time + thd->start_time_sec_part/1e6; > + return (uchar*) &thd->sys_var_tmp.double_value; > } > > > @@ -3045,10 +3045,10 @@ void sys_var_microseconds::set_default(T > uchar *sys_var_microseconds::value_ptr(THD *thd, enum_var_type type, > LEX_STRING *base) > { > - thd->tmp_double_value= (double) ((type == OPT_GLOBAL) ? > + thd->sys_var_tmp.double_value= (double) ((type == OPT_GLOBAL) ? > global_system_variables.*offset : > thd->variables.*offset) / 1000000.0; Change 100000.0 to same define constant as 1e6 > > === modified file 'sql/sql_class.cc' > @@ -2338,7 +2338,7 @@ bool select_max_min_finder_subselect::se > case DECIMAL_RESULT: > op= &select_max_min_finder_subselect::cmp_decimal; > break; > - case ROW_RESULT: > + default: Add back ROW_RESULT and add TIME_RESULT > +++ sql/sql_class.h 2011-03-26 10:59:34 +0000 <cut> > inline void set_time() > { > + if (user_time.val) > { > + start_time= hrtime_to_my_time(user_time); > + start_time_sec_part= hrtime_sec_part(user_time); > start_utime= utime_after_lock= my_micro_time(); > } > else > + { > + my_hrtime_t hrtime; > + my_timediff_t timediff; > + my_micro_and_hrtime(&timediff, &hrtime); > + start_time= hrtime_to_my_time(hrtime); > + start_time_sec_part= hrtime_sec_part(hrtime); > + utime_after_lock= start_utime= timediff.val; > + } For the future: As we are now always calling hrtime(), we should consider using hrtime also for start_utime. At least on systems with a fast hrtime() better to use this for both my_micro_time() and hrtime(). > +++ sql/sql_prepare.cc 2011-03-01 12:24:36 +0000 > @@ -487,8 +487,7 @@ static void set_param_time(Item_param *p > } > else > set_zero_time(&tm, MYSQL_TIMESTAMP_TIME); > - param->set_time(&tm, MYSQL_TIMESTAMP_TIME, > - MAX_TIME_WIDTH * MY_CHARSET_BIN_MB_MAXLEN); > + param->set_time(&tm, MYSQL_TIMESTAMP_TIME, MAX_TIME_FULL_WIDTH); Should be: MAX_TIME_FULL_WIDTH * MY_CHARSET_BIN_MB_MAXLEN > === modified file 'sql/sql_select.cc' > @@ -9455,7 +9445,7 @@ test_if_equality_guarantees_uniqueness(I > { > return r->const_item() && > /* elements must be compared as dates */ > - (Arg_comparator::can_compare_as_dates(l, r, 0) || > + (l->cmp_type() == TIME_RESULT || > /* or of the same result type */ > (r->result_type() == l->result_type() && > /* and must have the same collation if compared as strings */ Add a comment: /* It's enough to test 'l' for TIME_RESULT as only 'l' can be a field in this context. */ > === modified file 'sql/sql_show.cc' > --- sql/sql_show.cc 2010-11-02 13:20:02 +0000 <cut> > @@ -6288,6 +6305,8 @@ ST_FIELD_INFO columns_fields_info[]= > 0, (MY_I_S_MAYBE_NULL | MY_I_S_UNSIGNED), 0, OPEN_FRM_ONLY}, > {"NUMERIC_SCALE", MY_INT64_NUM_DECIMAL_DIGITS , MYSQL_TYPE_LONGLONG, > 0, (MY_I_S_MAYBE_NULL | MY_I_S_UNSIGNED), 0, OPEN_FRM_ONLY}, > + {"DATETIME_PRECISION", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, > + 0, (MY_I_S_MAYBE_NULL | MY_I_S_UNSIGNED), 0, OPEN_FRM_ONLY}, You have added a new field; Please add to compability documentation! > {"CHARACTER_SET_NAME", MY_CS_NAME_SIZE, MYSQL_TYPE_STRING, 0, 1, 0, > OPEN_FRM_ONLY}, > {"COLLATION_NAME", MY_CS_NAME_SIZE, MYSQL_TYPE_STRING, 0, 1, "Collation", > > +++ sql/sql_table.cc 2011-03-01 12:24:36 +0000 > @@ -34,26 +34,16 @@ const char *primary_key_name="PRIMARY"; > > static bool check_if_keyname_exists(const char *name,KEY *start, KEY *end); > static char *make_unique_key_name(const char *field_name,KEY *start,KEY *end); > -static int copy_data_between_tables(TABLE *from,TABLE *to, > - List<Create_field> &create, bool ignore, > - uint order_num, ORDER *order, > - ha_rows *copied,ha_rows *deleted, > - enum enum_enable_or_disable keys_onoff, > - bool error_if_not_empty); > +static int copy_data_between_tables(TABLE *,TABLE *, List<Create_field> &, bool, > + uint, ORDER *, ha_rows *,ha_rows *, > + enum enum_enable_or_disable, bool); I don't understand why you did this change here and in other places. I usually prefer to have the field names also in the prototype as it makes the prototype easier to read! <cut> > void make_time(const DATE_TIME_FORMAT *format __attribute__((unused)), > const MYSQL_TIME *l_time, String *str) > { > + uint length= (uint) my_time_to_str(l_time, (char*) str->ptr(), 0); > str->length(length); > str->set_charset(&my_charset_bin); It would be best to do str->alloc(MAX_DATETIME_FULL_WIDTH) at start of function. > } > @@ -713,15 +693,15 @@ void make_date(const DATE_TIME_FORMAT *f > void make_datetime(const DATE_TIME_FORMAT *format __attribute__((unused)), > const MYSQL_TIME *l_time, String *str) > { > + uint length= (uint) my_datetime_to_str(l_time, (char*) str->ptr(), 0); It would be best to do str->alloc(MAX_DATETIME_FULL_WIDTH) at start of function. > str->length(length); > str->set_charset(&my_charset_bin); > } <cut> > bool date_add_interval(MYSQL_TIME *ltime, interval_type int_type, INTERVAL interval) > { > long period, sign; > > + sign= (interval.neg == ltime->neg ? 1 : -1); Add a comment that shows why the above is needed > > switch (int_type) { > case INTERVAL_SECOND: > @@ -789,35 +770,29 @@ bool date_add_interval(MYSQL_TIME *ltime > case INTERVAL_DAY_SECOND: > case INTERVAL_DAY_MINUTE: > case INTERVAL_DAY_HOUR: > + case INTERVAL_DAY: > { > + longlong usec, daynr; > + my_bool neg= ltime->neg; > + enum enum_mysql_timestamp_type time_type= ltime->time_type; > + > + if (ltime->time_type != MYSQL_TIMESTAMP_TIME) Change to: if (time_type != MYSQL_TIMESTAMP_TIME) > + ltime->day+= calc_daynr(ltime->year, ltime->month, 1) - 1; > + > + usec= COMBINE(ltime) + sign*COMBINE(&interval); > + > + unpack_time(usec, ltime); > + ltime->time_type= time_type; > + ltime->neg^= neg; > + > + if (time_type == MYSQL_TIMESTAMP_TIME) > + break; > + > + if (int_type != INTERVAL_DAY) > + ltime->time_type= MYSQL_TIMESTAMP_DATETIME; // Return full date > + > + daynr= ltime->day + 32*(ltime->month + 13*ltime->year); Add a comment for the above: /* unpack_time() calculates month and day based on the assumption that the date was packed with 'pack_time()'. The following code restores the number of days. */ another option would be to do: daynr= usec / 1000000/24/60/60; This would at least be easier to understand... <cut> Here is some bugs I found: CREATE TABLE t3 (a time(6)); insert into t3 values(-100); select * from t3; ->8299:28:36.710656 If you use just (a time) you get: "-00:01:00" which is correct. cast("2101-00-01 02:03:04" as datetime) -> "2101-00-01 02:03:04" but cast(cast("2101-00-01 02:03:04" as datetime) as time) -> NULL Other things: Item_func_unix_timestamp() should return microseconds. Otherwise one can't acccess the full precision of Field_timestamp. One should also be able to do: select unix_timestamp(now(6)); For fast access to the current clock in maximum precision. --------- general_log.timestamp should be of type timestamp(6) slow_log.query_time and slow_log.lock_time() should be of type TIME(6) ----------- Regards, Monty

2 3

Re: [Maria-developers] [Commits] Rev 2992: Removed some alias warnings in lp:maria/5.3
by Kristian Nielsen 19 May '11

19 May '11

Michael Widenius <monty(a)askmonty.org> writes: > ------------------------------------------------------------ > revno: 2992 > revision-id: monty(a)askmonty.org-20110517214756-6f7fv5uk68i4hb3b > parent: sanja(a)askmonty.org-20110516132109-h2g9qzrfq3g9t6em > committer: Michael Widenius <monty(a)askmonty.org> > branch nick: maria-5.3 > timestamp: Wed 2011-05-18 00:47:56 +0300 > message: > Removed some alias warnings > Fixed alias bug when compiling with gcc 4.2.4 that caused subselect.test to fail "Fix strict aliasing bug that caused subselect.test to fail" ("alias bug" can mean different things. And the problem is not GCC specific, see below) > === modified file 'sql/item_subselect.cc' > --- a/sql/item_subselect.cc 2011-05-16 12:07:04 +0000 > +++ b/sql/item_subselect.cc 2011-05-17 21:47:56 +0000 > @@ -3563,7 +3562,15 @@ subselect_single_select_engine::change_r > } > else > result= res; > - return select_lex->join->change_result(result); > + > + /* > + We can't use 'result' below as gcc 4.2.4's alias optimization > + assumes that result was not changed by thd->change_item_tree(). > + I tried to find a solution to make gcc happy, but could not find anything > + that would not require a lot of extra code that would be harder to manage > + than the current code. > + */ > + return select_lex->join->change_result(res); > } I do not agree with this patch. Instead do something like this: @@ -3559,7 +3559,9 @@ subselect_single_select_engine::change_r nothing special about Item* pointer so it is safe conversion. We do not change the interface to be compatible with MySQL. */ - thd->change_item_tree((Item**) &result, (Item*)res); + Item *temp_item; + thd->change_item_tree(&temp_item, (Item*)res); + result= (select_result_interceptor *)temp_item; } else result= res; This makes the code correct (at least with respect to standard C++ strict aliasing), not just work-around the problem for one specific combination of compiler flags for one specific compiler version. The problem has nothing to do with any particular GCC version (except that this is where we happened to detect it), the problem is passing a select_result_interceptor** when an Item** is needed. This is in violation of the strict aliasing rule in standard C++. ---- However, I think we urgently need to start using -fno-strict-aliasing (and probably -fno-strict-overflow as well). Davi told me MySQL@Oracle is doing this as well. In effect there are two dialects of C/C++, even though they are rarely identified as such. - "System C/C++", which is a high-level assembler, where everything is bytes and pointers are just 8 (or 4) byte integers. - "Application C/C++", which is a type-safe language, and pointers are abstract objects that cannot be arbitrarily manipulated. Clearly, many (most?) developers are programming in "System C/C++", including you. That is fine, Postgress, the Linux Kernel, they use the same dialect. However, GCC by default uses "Application C/C++" (-fstrict-aliasing). We need to tell GCC which dialict of C/C++ we are using (-fno-strict-aliasing for "System C/C++"). If we could just agree on which dialect we use, and let GCC know our choice, things will be much better. - Kristian.

2 1

Re: [Maria-developers] [Commits] Rev 2994: Original idea from Zardosht Kasheff to add HA_CLUSTERED_INDEX in lp:maria/5.3
by Sergei Golubchik 19 May '11

19 May '11

Hi, Michael! On May 18, Michael Widenius wrote: > === modified file 'sql/handler.h' > --- a/sql/handler.h 2011-05-16 11:05:45 +0000 > +++ b/sql/handler.h 2011-05-18 16:26:30 +0000 > @@ -161,8 +161,11 @@ > */ > #define HA_KEY_SCAN_NOT_ROR 128 > #define HA_DO_INDEX_COND_PUSHDOWN 256 /* Supports Index Condition Pushdown */ > - > - > +/* > + Data is clustered on this key. This means that when you read the key > + you also get the row data in the same block. perhaps, better to say that "you get the row data without an extra seek (or even without an extra disk read)". A question: what about memory-based (and SSD) engines? Could/Should one consider all indexes clustered? > +*/ > +#define HA_CLUSTERED_INDEX 512 > > /* > bits in alter_table_flags: > @@ -2311,9 +2314,22 @@ class handler :public Sql_alloc > > > /* > - @retval TRUE Primary key (if there is one) is clustered > - key covering all fields > - @retval FALSE otherwise > + Check if the primary key (if there is one) is a clustered key covering > + all fields. This means: I wouldn't say that it "checks if the primary key is clustered", but something like "historical method that returns TRUE if everything below holds:" > + > + - Data is stored together with the primary key (no secondary lookup > + needed to find the row data). The optimizer uses this to find out > + the cost of fetching data. > + - The primary key is part of each secondary key and is used > + to find the row data in the primary index when reading trough > + secondary indexes. > + - When doing a HA_KEYREAD_ONLY we get also all the primary key parts > + into the row. This is critical property used by index_merge. and here "all the above is usually true for engines that store the row data in the primary key index (e.g. in a b-tree), and use the primary key value as a position()" > + > + For a clustered primary key, index_flags() returns also HA_CLUSTERED_INDEX > + > + @retval TRUE yes > + @retval FALSE No. > */ > virtual bool primary_key_is_clustered() { return FALSE; } > virtual int cmp_ref(const uchar *ref1, const uchar *ref2) > > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2011-05-16 12:07:04 +0000 > +++ b/sql/sql_select.cc 2011-05-18 16:26:30 +0000 > @@ -16525,9 +16527,9 @@ test_if_skip_sort_order(JOIN_TAB *tab,OR > */ > DBUG_ASSERT (ref_key != (int) nr); > > - bool is_covering= table->covering_keys.is_set(nr) || > - (nr == table->s->primary_key && > - table->file->primary_key_is_clustered()); > + bool is_covering= (table->covering_keys.is_set(nr) || > + (table->file->index_flags(nr, 0, 1) & > + HA_CLUSTERED_INDEX)); See, that's what I said. It make sense to set covering_keys bitmap for all clustered indexes once, when the table is opened, and then you not only get this condition simpler, but probably will also cover other cases where table->covering_keys is checked (or may be checked in the future). > /* > Don't use an index scan with ORDER BY without limit. > === modified file 'sql/sql_table.cc' > --- a/sql/sql_table.cc 2011-05-16 11:05:45 +0000 > +++ b/sql/sql_table.cc 2011-05-18 16:26:30 +0000 > @@ -7983,7 +7983,8 @@ copy_data_between_tables(TABLE *from,TAB > > if (order) > { > - if (to->s->primary_key != MAX_KEY && to->file->primary_key_is_clustered()) > + if (to->s->primary_key != MAX_KEY && > + to->file->ha_table_flags() & HA_TABLE_SCAN_ON_INDEX) Hmm, good. It took me a while to come up with a realistic use case where to->file->primary_key_is_clustered() is false, but to->file->ha_table_flags() & HA_TABLE_SCAN_ON_INDEX is true. Example: new InnoDB optimization that makes it to use internal generated primary key when user creates a PK of, say, VARCHAR(255). That is, in this optimization InnoDB treat this key as UNIQUE but not PRIMARY. Not completely unrealistic idea. In this case to->s->primary_key != MAX_KEY is true, there is a primary key. to->file->primary_key_is_clustered() is false, but to->file->ha_table_flags() & HA_TABLE_SCAN_ON_INDEX is true. Heh, this is a good example, perhaps we need to apply it as a test to other code that uses primary_key_is_clustered() and see if the condition is used correctly. > { > char warn_buff[MYSQL_ERRMSG_SIZE]; > my_snprintf(warn_buff, sizeof(warn_buff), > Regards, Sergei

2 2

[Maria-developers] feedback on proposed fix to MySQL bug 57430
by Zardosht Kasheff 19 May '11

19 May '11

Hello all, Continuing with patch contributions (as encouraged at the storage engine summit last Friday), here is a very simple patch we have to fix MySQL bug 57430 (http://bugs.mysql.com/bug.php?id=57430) The problem is described in the bug report we sent to MySQL. We think this simple fix addresses the problem that we ran into, but we do not understand the code well enough to know if this fix is sufficient. Feedback is welcome. Thanks -Zardosht

3 4

[Maria-developers] proposed patches for MWL#113: many clustered keys (not only primary)
by Zardosht Kasheff 19 May '11

19 May '11

Hello all, First off, thanks for hosting the storage engine summit last Friday. At the beginning of the day, I was encouraged to send the patches that I had for MWL#113 to the MariaDB developers list, in hopes possibly getting them into MariaDB 5.3. Here they are. They are patched off of MariaDB 5.2.3. They are all relatively simple. Here is what the patches do. - 1.diff - add an index flag HA_CLUSTERED_INDEX. If a storage engine exposes this flag for an index, then that index is clustered. This was proposed in http://bugs.mysql.com/bug.php?id=51687 - 2.diff - simple fix to get select ... order by DESC working - 3.diff - fix for index merge and clustering keys. This was inspired by Sergey Petrunya originally proposing http://lists.mysql.com/internals/37165. It has been altered a bit now that HA_CLUSTERED_INDEX is added. - 4.diff - extend fix of MySQL bug 50843 for clustering keys - http://lists.askmonty.org/pipermail/commits/2010-October/000626.html. I cannot find this checked in anywhere, but it seems to be a good patch for clustering keys. This was inspired from http://www.mail-archive.com/maria-developers@lists.launchpad.net/msg03491.h…. We currently have a hacky workaround for this issue, but this seems like the right fix. I also have one more fix for clustering keys and joins (in make_join_readinfo), but I want to look over it and before sending it. Also, if for whatever reason all of the patches are not good enough for checking in, please consider some subset of the patches. This is by no means "all or none". That is why I broke it up into separate diff files. I would love to hear feedback on these patches. Thanks -Zardosht

3 4

[Maria-developers] proposed patch to add new handler error code
by Zardosht Kasheff 18 May '11

18 May '11

Hello all, In continuing with the suggestion to provide patches (given at the storage engine summit last friday), here is another one (last one for tonight). Our storage engine wanted to add the ability to return an error to the user that states that the disk is full. To do so, we added, we added the handler error code HA_ERR_DISK_FULL. When returned, the error ER_DISK_FULL (which already exists) is returned to the user. This does not have a bug number or worklog number. Is there something I should do besides sending this patch to the list? Thanks -Zardosht

2 2

Re: [Maria-developers] NEW Percona QC patch for review
by Michael Widenius 17 May '11

17 May '11

Hi! >>>>> "Oleksandr" == Oleksandr Byelkin <sanja(a)askmonty.org> writes: Oleksandr> Hi! Oleksandr> Here is new patch. I hope I understood you. I assume that since your last patch, you have only changed the things I commented on in my last review. <cut> > === modified file 'sql/sql_cache.cc' > --- sql/sql_cache.cc 2010-11-30 21:11:03 +0000 > +++ sql/sql_cache.cc 2011-05-11 05:44:27 +0000 <cut> > -bool Query_cache::try_lock(bool use_timeout) > +bool Query_cache::try_lock(THD *thd, Cache_try_lock_mode mode) > { > bool interrupt= FALSE; > + const char* old_proc_info; > DBUG_ENTER("Query_cache::try_lock"); > + if (!thd) > + thd = current_thd; Why testing and setting thd ? Please fix this in the calling functions (as caller should always know if thd is set or not). If it can't be easily fixed, add at least a comment in which context thd could not be set. > + old_proc_info= thd->proc_info; > + thd_proc_info(thd,"Waiting on query cache mutex"); > > pthread_mutex_lock(&structure_guard_mutex); > + DBUG_EXECUTE_IF("status_wait_query_cache_mutex_sleep", { > + sleep(5); > + }); > + if (m_cache_status == DISABLED) > + { > + pthread_mutex_unlock(&structure_guard_mutex); You need to add: thd->proc_info = old_proc_info; > + return TRUE; > + } > @@ -488,11 +641,16 @@ > } > else > { > - pthread_cond_wait(&COND_cache_status_changed, &structure_guard_mutex); > + DBUG_ASSERT(mode == TRY); > + interrupt= TRUE; > + break; The following above the assert: /** If we are here, then mode is == TRY and there was someone else using the query cache. (m_cache_lock_status != Query_cache::UNLOCKED). Signal that we didn't get a lock. */ DBUG_ASSERT(m_requests_in_progress > 1); ... In this function, it will also be less code if you set interrupt to TRUE at start of function. Another options is they you use LINT_INIT(interrupt) and set the variable also when m_cache_lock_status == Query_cache::UNLOCKED. <cut> > @@ -866,14 +1034,18 @@ > { > DBUG_ENTER("query_cache_insert"); > > - /* See the comment on double-check locking usage above. */ > if (net->query_cache_query == 0) > DBUG_VOID_RETURN; > > + DBUG_ASSERT(current_thd); > + > DBUG_EXECUTE_IF("wait_in_query_cache_insert", > debug_wait_for_kill("wait_in_query_cache_insert"); ); > Add a comment here: /* First we check if query cache is disable without doing a mutex lock */ > - if (query_cache.try_lock()) > + if(query_cache.is_disabled()) > + DBUG_VOID_RETURN; Add a comment here: /* Lock the cache with try_lock(). try_lock() will fail if cache was disabled between the above test and lock. */ > + > + if (query_cache.try_lock(NULL, Query_cache::WAIT)) > DBUG_VOID_RETURN; > > Query_cache_block *query_block= (Query_cache_block*)net->query_cache_query; > @@ -931,7 +1103,10 @@ > if (net->query_cache_query == 0) > DBUG_VOID_RETURN; > > - if (query_cache.try_lock()) > + if(query_cache.is_disabled()) > + DBUG_VOID_RETURN; You can remove the above test, as we know the query cache was enabled when the query started. So there is two cases: 1) Either it's still enabled (which is probably the case) and we don't need the test. 2) It's not enabled and the below try_lock() will catch it. The extra test for is_disabled() is only relevant for query_cache.insert() to ensure that in normal operations we don't need to take any mutex when the cache is disabled. > @@ -1096,6 +1272,12 @@ > query_cache_size_arg)); > DBUG_ASSERT(initialized); > > + if (global_system_variables.query_cache_type == 0) > + { > + my_error(ER_QUERY_CACHE_IS_DISABLED, MYF(0)); > + DBUG_RETURN(0); > + } > + > lock_and_suspend(); > Shouldn't we do lock_and_suspend() BEFORE testing for (query_cache_type == 0)? For example the above may fail if we get two concurrent resize or disable commands ? <cut> > @@ -1985,11 +2268,14 @@ > { > DBUG_ENTER("Query_cache::pack"); > > + if (is_disabled()) > + DBUG_VOID_RETURN; > + We don't need the above. Same argument as for abort() > /* > If the entire qc is being invalidated we can bail out early > instead of waiting for the lock. > */ > - if (try_lock()) > + if (try_lock(NULL, Query_cache::WAIT)) > DBUG_VOID_RETURN; > > if (query_cache_size == 0) <cut> Regards, Monty

1 0

[Maria-developers] 5.2.6 .msi isntaller questions issues
by Peter Laursen 14 May '11

14 May '11

I think this is for @wlad! :-) I just installed 5.2.6 using the new .msi installer. This was an upgrade 'on top' of the previous (5.2.3 or 5.2.4) .msi build distributed by wlad in mailing list. The installer worked fine, The previos 5.2 instance was upgraded (ie. executables and similar files replaced, old my.ini and datadir is still used etc.). So far so good. The problem is with the configuration wizard. What is it supposed to do? Run 'mysql_upgrade'? More than this or something else? Let me explain my setup. Except for this MariaDB instance (running as service 'maria52') I have both MySQL/Oracle 5.0, 5.1, 5.5 and 5.6 (running as services 'MySQL50, 'MySQL51' , 'MySQL55' , 'MySQL56') . The config wizard prompts me which instance should be upgraded an gives me a choice of 'MySQL50 and 'MySQL51' but not 'MySQL55', 'MySQL56' and 'maria52'. 1) Now if the wizard starts 'mysql_upgrade' it makes perfect sense not to offer to upgrade 'MySQL55' and 'MySQL56' (as MySQL51/MariaDb52 are 'lower' than those - it would be a 'downgrade' - not an 'upgrade' then - what 'mysql_upgrade' does not handle). But ideally I think they should be displayed but 'greyed out' or otherwise indicated that upgrading MySQL 5.5/5.6 --> MariaDB 5.2 is not possible. 2) More important: the instance that I *really* wanted to upgrade ('maria52') was not displayed. I assume that the services' list is filtered by the prefix 'mysql'. Am I right? In that case I suggest the filtering is done on both (OR'ed) prefixes 'mysql' and 'maria'. Anyway after just exiting the wizard before executing it everything seems fine. But the wizard GUI can still be 'polished' a little I think. -- Peter Laursen -- Webyog

2 1

Re: [Maria-developers] [Commits] Rev 2988: Added directories with .dylib files to DYLD_LYBRARY_PATH to allow plugin to use them. in file:///Users/bell/maria/bzr/work-maria-5.3-machandlersocket/
by Sergei Golubchik 13 May '11

13 May '11

Hi, Oleksandr! On May 13, Oleksandr Byelkin wrote: > revno: 2988 > revision-id: sanja(a)askmonty.org-20110513150858-ba2sowmbv1l39xxm > parent: sanja(a)askmonty.org-20110512150136-5yzagaev1ssohykp > committer: Oleksandr Byelkin <sanja(a)askmonty.org> > branch nick: work-maria-5.3-machandlersocket > timestamp: Fri 2011-05-13 18:08:58 +0300 > message: > Added directories with .dylib files to DYLD_LYBRARY_PATH to allow plugin to use them. > === modified file 'mysql-test/mysql-test-run.pl' > --- a/mysql-test/mysql-test-run.pl 2011-04-19 12:34:40 +0000 > +++ b/mysql-test/mysql-test-run.pl 2011-05-13 15:08:58 +0000 > @@ -2051,6 +2051,13 @@ > push(@ld_library_paths, "$basedir/libmysql/.libs/", > "$basedir/libmysql_r/.libs/", > "$basedir/zlib/.libs/"); > + if ($^O eq "darwin") > + { > + # it is MAC OS and we have to add dynamic libraries paths > + push @ld_library_paths, grep {<$_/*.dylib>} > + (<$bindir/storage/*/.libs/>,<$bindir/plugin/*/.libs/>, > + <$bindir/plugin/*/*/.libs/>); > + } I think you forgot to add storage/*/*/.libs. add it and ok to push Regards, Sergei

1 0

[Maria-developers] MariaDB 5.2.6 release preparations
by Kristian Nielsen 13 May '11

13 May '11

Hi Daniel, We are ready to release MariaDB 5.2.6! I tagged the release with the tag mariadb-5.2.6. The tree is lp:maria/5.2. This release introduces packages for Ubuntu 11.04 "Natty". I updated the scripts in lp:mariadb-tools, but you need to be aware if something on the download pages needs changing. Apart from this we merged XtraDB from Percona Server 5.1.56-12.7. Monty also merged latest MySQL 5.1 tree, but this does not correspond to any particular released version, so I afraid you have nothing but the bzr changelog to tell which bugfixes are included. The built packages should be in the usual place. The build number from Buildbot is 907. This release also introduces the new windows MSI/.zip packages that Wlad and Philip set up. Philip sent mail about how to obtain those (I think on a private mailing list), or you can ask one of them for help if needed. Unfortunately, despite Philip spending a lot of effort (and half his sanity I'm afraid) to get the windows hosts to run stably, there are still intermittent problems, so the 32-bit packages are not ready yet. However, I requested a rebuild from Buildbot, and they should hopefully appear eventually. I think that is it, let me know if there is anything else you need! - Kristian.

4 11

Re: [Maria-developers] [Commits] Rev 2988: Added directories with .dylib files to DYLD_LYBRARY_PATH to allow plugin to use them. in file:///Users/bell/maria/bzr/work-maria-5.3-machandlersocket/
by Kristian Nielsen 13 May '11

13 May '11

Oleksandr Byelkin <sanja(a)askmonty.org> writes: > At file:///Users/bell/maria/bzr/work-maria-5.3-machandlersocket/ > > ------------------------------------------------------------ > revno: 2988 > revision-id: sanja(a)askmonty.org-20110513135817-kspibqh2830aj5tk > parent: sanja(a)askmonty.org-20110512150136-5yzagaev1ssohykp > committer: Oleksandr Byelkin <sanja(a)askmonty.org> > branch nick: work-maria-5.3-machandlersocket > timestamp: Fri 2011-05-13 16:58:17 +0300 > message: > Added directories with .dylib files to DYLD_LYBRARY_PATH to allow plugin to use them. > === modified file 'mysql-test/mysql-test-run.pl' > --- a/mysql-test/mysql-test-run.pl 2011-04-19 12:34:40 +0000 > +++ b/mysql-test/mysql-test-run.pl 2011-05-13 13:58:17 +0000 > @@ -2051,6 +2051,16 @@ > push(@ld_library_paths, "$basedir/libmysql/.libs/", > "$basedir/libmysql_r/.libs/", > "$basedir/zlib/.libs/"); > + if ($^O eq "darwin") > + { > + my %dirs= (); > + # it is MAC OS and we have to add dynamic libraries paths > + for (<$bindir/storage/*/.libs/*.dylib>,<$bindir/plugin/*/.libs/*.dylib>,<$bindir/plugin/*/*/.libs/*.dylib>,<$bindir/sql/.libs/*.dylib>) > + { > + $dirs{dirname($_)}= 1; > + } > + push @ld_library_paths, keys(%dirs); > + } > } > else > { > > === modified file 'plugin/handler_socket/libhsclient/Makefile.am' > --- a/plugin/handler_socket/libhsclient/Makefile.am 2011-05-12 15:01:36 +0000 > +++ b/plugin/handler_socket/libhsclient/Makefile.am 2011-05-13 13:58:17 +0000 > @@ -8,6 +8,5 @@ > lib_LTLIBRARIES = libhsclient.la > libhsclient_la_SOURCES = config.cpp escape.cpp fatal.cpp hstcpcli.cpp \ > socket.cpp string_util.cpp > -libhsclient_la_LDFLAGS = -static > libhsclient_la_CFLAGS = $(AM_CFLAGS) > libhsclient_la_CXXFLAGS = $(AM_CXXFLAGS) Ok to push from me. - Kristian.

1 0

[Maria-developers] mwl148 for review
by Oleksandr Byelkin 11 May '11

11 May '11

=== modified file 'mysql-test/r/explain.result' --- mysql-test/r/explain.result 2011-05-10 15:28:05 +0000 +++ mysql-test/r/explain.result 2011-05-10 20:17:04 +0000 @@ -176,11 +176,15 @@ SET SESSION sql_mode='ONLY_FULL_GROUP_BY'; EXPLAIN EXTENDED SELECT 1 FROM t1 WHERE f1 > ALL( SELECT t.f1 FROM t1,t1 AS t ); -ERROR 42000: Mixing of GROUP columns (MIN(),MAX(),COUNT(),...) with no GROUP columns is illegal if there is no GROUP BY clause +id select_type table type possible_keys key key_len ref rows filtered Extra +1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables +2 SUBQUERY t1 system NULL NULL NULL NULL 0 0.00 const row not found +2 SUBQUERY t system NULL NULL NULL NULL 0 0.00 const row not found +Warnings: +Note 1003 select 1 AS `1` from `test`.`t1` where 0 SHOW WARNINGS; Level Code Message -Error 1140 Mixing of GROUP columns (MIN(),MAX(),COUNT(),...) with no GROUP columns is illegal if there is no GROUP BY clause -Note 1003 select 1 AS `1` from `test`.`t1` where <not>(<exists>(...)) +Note 1003 select 1 AS `1` from `test`.`t1` where 0 SET SESSION sql_mode=@old_sql_mode; DROP TABLE t1; End of 5.0 tests. @@ -272,12 +276,12 @@ EXECUTE stmt; id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t1 system NULL NULL NULL NULL 1 -2 SUBQUERY a system NULL NULL NULL NULL 1 Using filesort +2 SUBQUERY a system NULL NULL NULL NULL 1 2 SUBQUERY t1 fulltext f1 f1 0 1 Using where EXECUTE stmt; id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t1 system NULL NULL NULL NULL 1 -2 SUBQUERY a system NULL NULL NULL NULL 1 Using filesort +2 SUBQUERY a system NULL NULL NULL NULL 1 2 SUBQUERY t1 fulltext f1 f1 0 1 Using where DEALLOCATE PREPARE stmt; PREPARE stmt FROM @@ -288,12 +292,12 @@ EXECUTE stmt; id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t1 system NULL NULL NULL NULL 1 -2 SUBQUERY a system NULL NULL NULL NULL 1 Using filesort +2 SUBQUERY a system NULL NULL NULL NULL 1 2 SUBQUERY t1 fulltext f1 f1 0 1 Using where EXECUTE stmt; id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t1 system NULL NULL NULL NULL 1 -2 SUBQUERY a system NULL NULL NULL NULL 1 Using filesort +2 SUBQUERY a system NULL NULL NULL NULL 1 2 SUBQUERY t1 fulltext f1 f1 0 1 Using where DEALLOCATE PREPARE stmt; DROP TABLE t1; === modified file 'mysql-test/r/subselect.result' --- mysql-test/r/subselect.result 2011-05-04 22:35:03 +0000 +++ mysql-test/r/subselect.result 2011-05-10 20:17:04 +0000 @@ -273,7 +273,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 3 100.00 Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= (select min(`test`.`t2`.`b`) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= (select min(`test`.`t2`.`b`) from `test`.`t2`)))) select * from t3 where a >= all (select b from t2); a 7 @@ -1503,7 +1503,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` < (select max(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` < (select max(NULL) from `test`.`t2`)))) select * from t3 where a >= some (select b from t2); a explain extended select * from t3 where a >= some (select b from t2); @@ -1511,7 +1511,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= (select min(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= (select min(NULL) from `test`.`t2`)))) select * from t3 where a >= all (select b from t2 group by 1); a 6 @@ -1522,7 +1522,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` < <max>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` < <max>(select NULL from `test`.`t2` group by 1)))) select * from t3 where a >= some (select b from t2 group by 1); a explain extended select * from t3 where a >= some (select b from t2 group by 1); @@ -1530,7 +1530,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= <min>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= <min>(select NULL from `test`.`t2` group by 1)))) select * from t3 where NULL >= any (select b from t2); a explain extended select * from t3 where NULL >= any (select b from t2); @@ -1538,7 +1538,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= (select min(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= (select min(NULL) from `test`.`t2`)))) select * from t3 where NULL >= any (select b from t2 group by 1); a explain extended select * from t3 where NULL >= any (select b from t2 group by 1); @@ -1546,7 +1546,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= <min>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= <min>(select NULL from `test`.`t2` group by 1)))) select * from t3 where NULL >= some (select b from t2); a explain extended select * from t3 where NULL >= some (select b from t2); @@ -1554,7 +1554,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= (select min(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= (select min(NULL) from `test`.`t2`)))) select * from t3 where NULL >= some (select b from t2 group by 1); a explain extended select * from t3 where NULL >= some (select b from t2 group by 1); @@ -1562,7 +1562,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= <min>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= <min>(select NULL from `test`.`t2` group by 1)))) insert into t2 values (2,2), (2,1), (3,3), (3,1); select * from t3 where a > all (select max(b) from t2 group by a); a @@ -1573,7 +1573,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 4 100.00 Using temporary Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` <= <max>(select max(`test`.`t2`.`b`) from `test`.`t2` group by `test`.`t2`.`a`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` <= <max>(select max(`test`.`t2`.`b`) from `test`.`t2` group by `test`.`t2`.`a`)))) drop table t2, t3; CREATE TABLE `t1` ( `id` mediumint(9) NOT NULL auto_increment, `taskid` bigint(20) NOT NULL default '0', `dbid` int(11) NOT NULL default '0', `create_date` datetime NOT NULL default '0000-00-00 00:00:00', `last_update` datetime NOT NULL default '0000-00-00 00:00:00', PRIMARY KEY (`id`)) ENGINE=MyISAM CHARSET=latin1 AUTO_INCREMENT=3 ; INSERT INTO `t1` (`id`, `taskid`, `dbid`, `create_date`,`last_update`) VALUES (1, 1, 15, '2003-09-29 10:31:36', '2003-09-29 10:31:36'), (2, 1, 21, now(), now()); @@ -1624,7 +1624,7 @@ 3 UNION t1 system NULL NULL NULL NULL 1 100.00 NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL Warnings: -Note 1003 select 'e' AS `s1` from `test`.`t1` where <nop>(('f' > <min>(select 'e' from `test`.`t1` union select 'e' from `test`.`t1`))) +Note 1003 select 'e' AS `s1` from `test`.`t1` where <nop>(<in_optimizer>('f',('f' > <min>(select 'e' from `test`.`t1` union select 'e' from `test`.`t1`)))) drop table t1; CREATE TABLE t1 (number char(11) NOT NULL default '') ENGINE=MyISAM CHARSET=latin1; INSERT INTO t1 VALUES ('69294728265'),('18621828126'),('89356874041'),('95895001874'); === modified file 'mysql-test/r/subselect_no_mat.result' --- mysql-test/r/subselect_no_mat.result 2011-05-04 22:35:03 +0000 +++ mysql-test/r/subselect_no_mat.result 2011-05-10 20:17:04 +0000 @@ -277,7 +277,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 3 100.00 Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= (select min(`test`.`t2`.`b`) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= (select min(`test`.`t2`.`b`) from `test`.`t2`)))) select * from t3 where a >= all (select b from t2); a 7 @@ -1507,7 +1507,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` < (select max(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` < (select max(NULL) from `test`.`t2`)))) select * from t3 where a >= some (select b from t2); a explain extended select * from t3 where a >= some (select b from t2); @@ -1515,7 +1515,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= (select min(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= (select min(NULL) from `test`.`t2`)))) select * from t3 where a >= all (select b from t2 group by 1); a 6 @@ -1526,7 +1526,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` < <max>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` < <max>(select NULL from `test`.`t2` group by 1)))) select * from t3 where a >= some (select b from t2 group by 1); a explain extended select * from t3 where a >= some (select b from t2 group by 1); @@ -1534,7 +1534,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= <min>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= <min>(select NULL from `test`.`t2` group by 1)))) select * from t3 where NULL >= any (select b from t2); a explain extended select * from t3 where NULL >= any (select b from t2); @@ -1542,7 +1542,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= (select min(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= (select min(NULL) from `test`.`t2`)))) select * from t3 where NULL >= any (select b from t2 group by 1); a explain extended select * from t3 where NULL >= any (select b from t2 group by 1); @@ -1550,7 +1550,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= <min>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= <min>(select NULL from `test`.`t2` group by 1)))) select * from t3 where NULL >= some (select b from t2); a explain extended select * from t3 where NULL >= some (select b from t2); @@ -1558,7 +1558,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= (select min(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= (select min(NULL) from `test`.`t2`)))) select * from t3 where NULL >= some (select b from t2 group by 1); a explain extended select * from t3 where NULL >= some (select b from t2 group by 1); @@ -1566,7 +1566,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= <min>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= <min>(select NULL from `test`.`t2` group by 1)))) insert into t2 values (2,2), (2,1), (3,3), (3,1); select * from t3 where a > all (select max(b) from t2 group by a); a @@ -1577,7 +1577,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 4 100.00 Using temporary Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` <= <max>(select max(`test`.`t2`.`b`) from `test`.`t2` group by `test`.`t2`.`a`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` <= <max>(select max(`test`.`t2`.`b`) from `test`.`t2` group by `test`.`t2`.`a`)))) drop table t2, t3; CREATE TABLE `t1` ( `id` mediumint(9) NOT NULL auto_increment, `taskid` bigint(20) NOT NULL default '0', `dbid` int(11) NOT NULL default '0', `create_date` datetime NOT NULL default '0000-00-00 00:00:00', `last_update` datetime NOT NULL default '0000-00-00 00:00:00', PRIMARY KEY (`id`)) ENGINE=MyISAM CHARSET=latin1 AUTO_INCREMENT=3 ; INSERT INTO `t1` (`id`, `taskid`, `dbid`, `create_date`,`last_update`) VALUES (1, 1, 15, '2003-09-29 10:31:36', '2003-09-29 10:31:36'), (2, 1, 21, now(), now()); @@ -1628,7 +1628,7 @@ 3 UNION t1 system NULL NULL NULL NULL 1 100.00 NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL Warnings: -Note 1003 select 'e' AS `s1` from `test`.`t1` where <nop>(('f' > <min>(select 'e' from `test`.`t1` union select 'e' from `test`.`t1`))) +Note 1003 select 'e' AS `s1` from `test`.`t1` where <nop>(<in_optimizer>('f',('f' > <min>(select 'e' from `test`.`t1` union select 'e' from `test`.`t1`)))) drop table t1; CREATE TABLE t1 (number char(11) NOT NULL default '') ENGINE=MyISAM CHARSET=latin1; INSERT INTO t1 VALUES ('69294728265'),('18621828126'),('89356874041'),('95895001874'); === modified file 'mysql-test/r/subselect_no_opts.result' --- mysql-test/r/subselect_no_opts.result 2011-05-04 22:35:03 +0000 +++ mysql-test/r/subselect_no_opts.result 2011-05-10 20:17:04 +0000 @@ -274,7 +274,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 3 100.00 Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= (select min(`test`.`t2`.`b`) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= (select min(`test`.`t2`.`b`) from `test`.`t2`)))) select * from t3 where a >= all (select b from t2); a 7 @@ -1504,7 +1504,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` < (select max(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` < (select max(NULL) from `test`.`t2`)))) select * from t3 where a >= some (select b from t2); a explain extended select * from t3 where a >= some (select b from t2); @@ -1512,7 +1512,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= (select min(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= (select min(NULL) from `test`.`t2`)))) select * from t3 where a >= all (select b from t2 group by 1); a 6 @@ -1523,7 +1523,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` < <max>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` < <max>(select NULL from `test`.`t2` group by 1)))) select * from t3 where a >= some (select b from t2 group by 1); a explain extended select * from t3 where a >= some (select b from t2 group by 1); @@ -1531,7 +1531,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= <min>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= <min>(select NULL from `test`.`t2` group by 1)))) select * from t3 where NULL >= any (select b from t2); a explain extended select * from t3 where NULL >= any (select b from t2); @@ -1539,7 +1539,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= (select min(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= (select min(NULL) from `test`.`t2`)))) select * from t3 where NULL >= any (select b from t2 group by 1); a explain extended select * from t3 where NULL >= any (select b from t2 group by 1); @@ -1547,7 +1547,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= <min>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= <min>(select NULL from `test`.`t2` group by 1)))) select * from t3 where NULL >= some (select b from t2); a explain extended select * from t3 where NULL >= some (select b from t2); @@ -1555,7 +1555,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= (select min(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= (select min(NULL) from `test`.`t2`)))) select * from t3 where NULL >= some (select b from t2 group by 1); a explain extended select * from t3 where NULL >= some (select b from t2 group by 1); @@ -1563,7 +1563,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= <min>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= <min>(select NULL from `test`.`t2` group by 1)))) insert into t2 values (2,2), (2,1), (3,3), (3,1); select * from t3 where a > all (select max(b) from t2 group by a); a @@ -1574,7 +1574,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 4 100.00 Using temporary Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` <= <max>(select max(`test`.`t2`.`b`) from `test`.`t2` group by `test`.`t2`.`a`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` <= <max>(select max(`test`.`t2`.`b`) from `test`.`t2` group by `test`.`t2`.`a`)))) drop table t2, t3; CREATE TABLE `t1` ( `id` mediumint(9) NOT NULL auto_increment, `taskid` bigint(20) NOT NULL default '0', `dbid` int(11) NOT NULL default '0', `create_date` datetime NOT NULL default '0000-00-00 00:00:00', `last_update` datetime NOT NULL default '0000-00-00 00:00:00', PRIMARY KEY (`id`)) ENGINE=MyISAM CHARSET=latin1 AUTO_INCREMENT=3 ; INSERT INTO `t1` (`id`, `taskid`, `dbid`, `create_date`,`last_update`) VALUES (1, 1, 15, '2003-09-29 10:31:36', '2003-09-29 10:31:36'), (2, 1, 21, now(), now()); @@ -1625,7 +1625,7 @@ 3 UNION t1 system NULL NULL NULL NULL 1 100.00 NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL Warnings: -Note 1003 select 'e' AS `s1` from `test`.`t1` where <nop>(('f' > <min>(select 'e' from `test`.`t1` union select 'e' from `test`.`t1`))) +Note 1003 select 'e' AS `s1` from `test`.`t1` where <nop>(<in_optimizer>('f',('f' > <min>(select 'e' from `test`.`t1` union select 'e' from `test`.`t1`)))) drop table t1; CREATE TABLE t1 (number char(11) NOT NULL default '') ENGINE=MyISAM CHARSET=latin1; INSERT INTO t1 VALUES ('69294728265'),('18621828126'),('89356874041'),('95895001874'); === modified file 'mysql-test/r/subselect_no_semijoin.result' --- mysql-test/r/subselect_no_semijoin.result 2011-05-04 22:35:03 +0000 +++ mysql-test/r/subselect_no_semijoin.result 2011-05-10 20:17:04 +0000 @@ -274,7 +274,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 3 100.00 Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= (select min(`test`.`t2`.`b`) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= (select min(`test`.`t2`.`b`) from `test`.`t2`)))) select * from t3 where a >= all (select b from t2); a 7 @@ -1504,7 +1504,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` < (select max(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` < (select max(NULL) from `test`.`t2`)))) select * from t3 where a >= some (select b from t2); a explain extended select * from t3 where a >= some (select b from t2); @@ -1512,7 +1512,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= (select min(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= (select min(NULL) from `test`.`t2`)))) select * from t3 where a >= all (select b from t2 group by 1); a 6 @@ -1523,7 +1523,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` < <max>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` < <max>(select NULL from `test`.`t2` group by 1)))) select * from t3 where a >= some (select b from t2 group by 1); a explain extended select * from t3 where a >= some (select b from t2 group by 1); @@ -1531,7 +1531,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= <min>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= <min>(select NULL from `test`.`t2` group by 1)))) select * from t3 where NULL >= any (select b from t2); a explain extended select * from t3 where NULL >= any (select b from t2); @@ -1539,7 +1539,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= (select min(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= (select min(NULL) from `test`.`t2`)))) select * from t3 where NULL >= any (select b from t2 group by 1); a explain extended select * from t3 where NULL >= any (select b from t2 group by 1); @@ -1547,7 +1547,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= <min>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= <min>(select NULL from `test`.`t2` group by 1)))) select * from t3 where NULL >= some (select b from t2); a explain extended select * from t3 where NULL >= some (select b from t2); @@ -1555,7 +1555,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= (select min(NULL) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= (select min(NULL) from `test`.`t2`)))) select * from t3 where NULL >= some (select b from t2 group by 1); a explain extended select * from t3 where NULL >= some (select b from t2 group by 1); @@ -1563,7 +1563,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 system NULL NULL NULL NULL 0 0.00 const row not found Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= <min>(select NULL from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= <min>(select NULL from `test`.`t2` group by 1)))) insert into t2 values (2,2), (2,1), (3,3), (3,1); select * from t3 where a > all (select max(b) from t2 group by a); a @@ -1574,7 +1574,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 4 100.00 Using temporary Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` <= <max>(select max(`test`.`t2`.`b`) from `test`.`t2` group by `test`.`t2`.`a`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` <= <max>(select max(`test`.`t2`.`b`) from `test`.`t2` group by `test`.`t2`.`a`)))) drop table t2, t3; CREATE TABLE `t1` ( `id` mediumint(9) NOT NULL auto_increment, `taskid` bigint(20) NOT NULL default '0', `dbid` int(11) NOT NULL default '0', `create_date` datetime NOT NULL default '0000-00-00 00:00:00', `last_update` datetime NOT NULL default '0000-00-00 00:00:00', PRIMARY KEY (`id`)) ENGINE=MyISAM CHARSET=latin1 AUTO_INCREMENT=3 ; INSERT INTO `t1` (`id`, `taskid`, `dbid`, `create_date`,`last_update`) VALUES (1, 1, 15, '2003-09-29 10:31:36', '2003-09-29 10:31:36'), (2, 1, 21, now(), now()); @@ -1625,7 +1625,7 @@ 3 UNION t1 system NULL NULL NULL NULL 1 100.00 NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL Warnings: -Note 1003 select 'e' AS `s1` from `test`.`t1` where <nop>(('f' > <min>(select 'e' from `test`.`t1` union select 'e' from `test`.`t1`))) +Note 1003 select 'e' AS `s1` from `test`.`t1` where <nop>(<in_optimizer>('f',('f' > <min>(select 'e' from `test`.`t1` union select 'e' from `test`.`t1`)))) drop table t1; CREATE TABLE t1 (number char(11) NOT NULL default '') ENGINE=MyISAM CHARSET=latin1; INSERT INTO t1 VALUES ('69294728265'),('18621828126'),('89356874041'),('95895001874'); === modified file 'mysql-test/suite/pbxt/r/subselect.result' --- mysql-test/suite/pbxt/r/subselect.result 2011-04-20 19:55:29 +0000 +++ mysql-test/suite/pbxt/r/subselect.result 2011-05-10 20:17:04 +0000 @@ -270,7 +270,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 3 100.00 Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= (select min(`test`.`t2`.`b`) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= (select min(`test`.`t2`.`b`) from `test`.`t2`)))) select * from t3 where a >= all (select b from t2); a 7 @@ -1498,7 +1498,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 0 0.00 Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` < (select max(`test`.`t2`.`b`) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` < (select max(`test`.`t2`.`b`) from `test`.`t2`)))) select * from t3 where a >= some (select b from t2); a explain extended select * from t3 where a >= some (select b from t2); @@ -1506,7 +1506,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 0 0.00 Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= (select min(`test`.`t2`.`b`) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= (select min(`test`.`t2`.`b`) from `test`.`t2`)))) select * from t3 where a >= all (select b from t2 group by 1); a 6 @@ -1517,7 +1517,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 0 0.00 Using temporary Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` < <max>(select `test`.`t2`.`b` from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` < <max>(select `test`.`t2`.`b` from `test`.`t2` group by 1)))) select * from t3 where a >= some (select b from t2 group by 1); a explain extended select * from t3 where a >= some (select b from t2 group by 1); @@ -1525,7 +1525,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 0 0.00 Using temporary Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((`test`.`t3`.`a` >= <min>(select `test`.`t2`.`b` from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` >= <min>(select `test`.`t2`.`b` from `test`.`t2` group by 1)))) select * from t3 where NULL >= any (select b from t2); a explain extended select * from t3 where NULL >= any (select b from t2); @@ -1533,7 +1533,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 ALL NULL NULL NULL NULL 0 0.00 Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= (select min(`test`.`t2`.`b`) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= (select min(`test`.`t2`.`b`) from `test`.`t2`)))) select * from t3 where NULL >= any (select b from t2 group by 1); a explain extended select * from t3 where NULL >= any (select b from t2 group by 1); @@ -1541,7 +1541,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 ALL NULL NULL NULL NULL 0 0.00 Using temporary Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= <min>(select `test`.`t2`.`b` from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= <min>(select `test`.`t2`.`b` from `test`.`t2` group by 1)))) select * from t3 where NULL >= some (select b from t2); a explain extended select * from t3 where NULL >= some (select b from t2); @@ -1549,7 +1549,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 ALL NULL NULL NULL NULL 0 0.00 Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= (select min(`test`.`t2`.`b`) from `test`.`t2`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= (select min(`test`.`t2`.`b`) from `test`.`t2`)))) select * from t3 where NULL >= some (select b from t2 group by 1); a explain extended select * from t3 where NULL >= some (select b from t2 group by 1); @@ -1557,7 +1557,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 2 SUBQUERY t2 ALL NULL NULL NULL NULL 0 0.00 Using temporary Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>((NULL >= <min>(select `test`.`t2`.`b` from `test`.`t2` group by 1))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <nop>(<in_optimizer>(NULL,(NULL >= <min>(select `test`.`t2`.`b` from `test`.`t2` group by 1)))) insert into t2 values (2,2), (2,1), (3,3), (3,1); select * from t3 where a > all (select max(b) from t2 group by a); a @@ -1568,7 +1568,7 @@ 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 Using where 2 SUBQUERY t2 ALL NULL NULL NULL NULL 4 100.00 Using temporary Warnings: -Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>((`test`.`t3`.`a` <= <max>(select max(`test`.`t2`.`b`) from `test`.`t2` group by `test`.`t2`.`a`))) +Note 1003 select `test`.`t3`.`a` AS `a` from `test`.`t3` where <not>(<in_optimizer>(`test`.`t3`.`a`,(`test`.`t3`.`a` <= <max>(select max(`test`.`t2`.`b`) from `test`.`t2` group by `test`.`t2`.`a`)))) drop table t2, t3; CREATE TABLE `t1` ( `id` mediumint(9) NOT NULL auto_increment, `taskid` bigint(20) NOT NULL default '0', `dbid` int(11) NOT NULL default '0', `create_date` datetime NOT NULL default '0000-00-00 00:00:00', `last_update` datetime NOT NULL default '0000-00-00 00:00:00', PRIMARY KEY (`id`)) ENGINE=MyISAM CHARSET=latin1 AUTO_INCREMENT=3 ; INSERT INTO `t1` (`id`, `taskid`, `dbid`, `create_date`,`last_update`) VALUES (1, 1, 15, '2003-09-29 10:31:36', '2003-09-29 10:31:36'), (2, 1, 21, now(), now()); @@ -1619,7 +1619,7 @@ 3 UNION t1 ALL NULL NULL NULL NULL 1 100.00 NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL Warnings: -Note 1003 select `test`.`t1`.`s1` AS `s1` from `test`.`t1` where <nop>(('f' > <min>(select `test`.`t1`.`s1` from `test`.`t1` union select `test`.`t1`.`s1` from `test`.`t1`))) +Note 1003 select `test`.`t1`.`s1` AS `s1` from `test`.`t1` where <nop>(<in_optimizer>('f',('f' > <min>(select `test`.`t1`.`s1` from `test`.`t1` union select `test`.`t1`.`s1` from `test`.`t1`)))) drop table t1; CREATE TABLE t1 (number char(11) NOT NULL default '') ENGINE=MyISAM CHARSET=latin1; INSERT INTO t1 VALUES ('69294728265'),('18621828126'),('89356874041'),('95895001874'); === modified file 'mysql-test/t/explain.test' --- mysql-test/t/explain.test 2011-05-05 01:08:44 +0000 +++ mysql-test/t/explain.test 2011-05-10 20:17:04 +0000 @@ -158,7 +158,10 @@ SET SESSION sql_mode='ONLY_FULL_GROUP_BY'; # EXPLAIN EXTENDED (with subselect). used to crash. should give NOTICE. ---error ER_MIX_OF_GROUP_FUNC_AND_FIELDS +# Before moving max/min optimization to optimize phase this statement +# generated error, but as far as original query do not contain aggregate +# function user should not see error +# --error ER_MIX_OF_GROUP_FUNC_AND_FIELDS EXPLAIN EXTENDED SELECT 1 FROM t1 WHERE f1 > ALL( SELECT t.f1 FROM t1,t1 AS t ); SHOW WARNINGS; === modified file 'sql/item_cmpfunc.cc' --- sql/item_cmpfunc.cc 2011-05-10 15:28:05 +0000 +++ sql/item_cmpfunc.cc 2011-05-10 20:17:04 +0000 @@ -1798,6 +1798,8 @@ { THD *thd= (THD*) thd_arg; DBUG_ENTER("Item_in_optimizer::expr_cache_insert_transformer"); + if (args[1]->type() != Item::SUBSELECT_ITEM) + DBUG_RETURN(this); // MAX/MIN transformed => do nothing List<Item*> &depends_on= ((Item_subselect *)args[1])->depends_on; if (expr_cache) @@ -1901,7 +1903,15 @@ DBUG_ASSERT(fixed == 1); cache->store(args[0]); cache->cache_value(); - + + if (args[1]->type() != Item::SUBSELECT_ITEM) + { + /* MAX/MIN transformed => pass through */ + longlong res= args[1]->val_int(); + null_value= args[1]->null_value; + return (res); + } + if (cache->null_value) { /* @@ -2050,24 +2060,35 @@ if ((*args) != new_item) current_thd->change_item_tree(args, new_item); - /* - Transform the right IN operand which should be an Item_in_subselect or a - subclass of it. The left operand of the IN must be the same as the left - operand of this Item_in_optimizer, so in this case there is no further - transformation, we only make both operands the same. - TODO: is it the way it should be? - */ - DBUG_ASSERT((args[1])->type() == Item::SUBSELECT_ITEM && - (((Item_subselect*)(args[1]))->substype() == - Item_subselect::IN_SUBS || - ((Item_subselect*)(args[1]))->substype() == - Item_subselect::ALL_SUBS || - ((Item_subselect*)(args[1]))->substype() == - Item_subselect::ANY_SUBS)); - - Item_in_subselect *in_arg= (Item_in_subselect*)args[1]; - in_arg->left_expr= args[0]; - + if (args[1]->type() != Item::SUBSELECT_ITEM) + { + /* MAX/MIN transformed => pass through */ + new_item= args[1]->transform(transformer, argument); + if (!new_item) + return 0; + if (args[1] != new_item) + current_thd->change_item_tree(args, new_item); + } + else + { + /* + Transform the right IN operand which should be an Item_in_subselect or a + subclass of it. The left operand of the IN must be the same as the left + operand of this Item_in_optimizer, so in this case there is no further + transformation, we only make both operands the same. + TODO: is it the way it should be? + */ + DBUG_ASSERT((args[1])->type() == Item::SUBSELECT_ITEM && + (((Item_subselect*)(args[1]))->substype() == + Item_subselect::IN_SUBS || + ((Item_subselect*)(args[1]))->substype() == + Item_subselect::ALL_SUBS || + ((Item_subselect*)(args[1]))->substype() == + Item_subselect::ANY_SUBS)); + + Item_in_subselect *in_arg= (Item_in_subselect*)args[1]; + in_arg->left_expr= args[0]; + } return (this->*transformer)(argument); } === modified file 'sql/item_subselect.cc' --- sql/item_subselect.cc 2011-04-28 14:15:05 +0000 +++ sql/item_subselect.cc 2011-05-10 20:17:04 +0000 @@ -33,12 +33,14 @@ Item_subselect::Item_subselect(): - Item_result_field(), value_assigned(0), thd(0), substitution(0), + Item_result_field(), value_assigned(0), thd(0), substitution(0), expr_cache(0), engine(0), old_engine(0), used_tables_cache(0), have_to_be_excluded(0), const_item_cache(1), inside_first_fix_fields(0), done_first_fix_fields(FALSE), forced_const(FALSE), eliminated(FALSE), engine_changed(0), changed(0), is_correlated(FALSE) { + DBUG_ENTER("Item_subselect::Item_subselect"); + DBUG_PRINT("enter", ("this: 0x%lx", (ulong) this)); with_subselect= 1; reset(); /* @@ -46,6 +48,7 @@ (i.e. some rows will be found returned) */ null_value= TRUE; + DBUG_VOID_RETURN; } @@ -58,7 +61,8 @@ */ DBUG_ENTER("Item_subselect::init"); - DBUG_PRINT("enter", ("select_lex: 0x%lx", (long) select_lex)); + DBUG_PRINT("enter", ("select_lex: 0x%lx this: 0x%lx", + (ulong) select_lex, (ulong) this)); unit= select_lex->master_unit(); thd= unit->thd; @@ -69,10 +73,12 @@ => we do not copy old_engine here */ engine= unit->item->engine; + borrowed_engine= TRUE; parsing_place= unit->item->parsing_place; - unit->item->engine= 0; - unit->item= this; - engine->change_result(this, result); + //unit->item->engine= 0; + thd->change_item_tree((Item**)&unit->item, this); + //unit->item= this; + engine->change_result(this, result, TRUE); } else { @@ -97,6 +103,7 @@ /* The subquery is an expression cache candidate */ upper->expr_cache_may_be_used[upper->parsing_place]= TRUE; } + DBUG_PRINT("info", ("engine: 0x%lx", (ulong)engine)); DBUG_VOID_RETURN; } @@ -156,8 +163,14 @@ Item_subselect::~Item_subselect() { - delete engine; + DBUG_ENTER("Item_subselect::~Item_subselect"); + DBUG_PRINT("enter", ("this: 0x%lx", (ulong) this)); + if (!borrowed_engine) + delete engine; + else + engine->cleanup(); engine= NULL; + DBUG_VOID_RETURN; } bool @@ -1119,11 +1132,23 @@ void Item_exists_subselect::fix_length_and_dec() { - decimals= 0; - max_length= 1; - max_columns= engine->cols(); + DBUG_ENTER("Item_exists_subselect::fix_length_and_dec"); + decimals= 0; + max_length= 1; + max_columns= engine->cols(); /* We need only 1 row to determine existence */ unit->global_parameters->select_limit= new Item_int((int32) 1); + DBUG_PRINT("info", ("Set limit to 1")); + DBUG_VOID_RETURN; +} + +void Item_in_subselect::fix_length_and_dec() +{ + DBUG_ENTER("Item_in_subselect::fix_length_and_dec"); + decimals= 0; + max_length= 1; + max_columns= engine->cols(); + DBUG_VOID_RETURN; } @@ -1388,88 +1413,6 @@ DBUG_RETURN(true); } - /* - If this is an ALL/ANY single-value subselect, try to rewrite it with - a MIN/MAX subselect. We can do that if a possible NULL result of the - subselect can be ignored. - E.g. SELECT * FROM t1 WHERE b > ANY (SELECT a FROM t2) can be rewritten - with SELECT * FROM t1 WHERE b > (SELECT MAX(a) FROM t2). - We can't check that this optimization is safe if it's not a top-level - item of the WHERE clause (e.g. because the WHERE clause can contain IS - NULL/IS NOT NULL functions). If so, we rewrite ALL/ANY with NOT EXISTS - later in this method. - */ - if ((abort_on_null || (upper_item && upper_item->top_level())) && - !select_lex->master_unit()->uncacheable && !func->eqne_op()) - { - if (substitution) - { - /* It is second (third, ...) SELECT of UNION => All is done */ - DBUG_RETURN(false); - } - - Item *subs; - if (!select_lex->group_list.elements && - !select_lex->having && - !select_lex->with_sum_func && - !(select_lex->next_select()) && - select_lex->table_list.elements) - { - Item_sum_hybrid *item; - nesting_map save_allow_sum_func; - if (func->l_op()) - { - /* - (ALL && (> || =>)) || (ANY && (< || =<)) - for ALL condition is inverted - */ - item= new Item_sum_max(*select_lex->ref_pointer_array); - } - else - { - /* - (ALL && (< || =<)) || (ANY && (> || =>)) - for ALL condition is inverted - */ - item= new Item_sum_min(*select_lex->ref_pointer_array); - } - if (upper_item) - upper_item->set_sum_test(item); - *select_lex->ref_pointer_array= item; - { - List_iterator<Item> it(select_lex->item_list); - it++; - it.replace(item); - } - - save_allow_sum_func= thd->lex->allow_sum_func; - thd->lex->allow_sum_func|= 1 << thd->lex->current_select->nest_level; - /* - Item_sum_(max|min) can't substitute other item => we can use 0 as - reference, also Item_sum_(max|min) can't be fixed after creation, so - we do not check item->fixed - */ - if (item->fix_fields(thd, 0)) - DBUG_RETURN(true); - thd->lex->allow_sum_func= save_allow_sum_func; - /* we added aggregate function => we have to change statistic */ - count_field_types(select_lex, &join->tmp_table_param, join->all_fields, - 0); - - subs= new Item_singlerow_subselect(select_lex); - } - else - { - Item_maxmin_subselect *item; - subs= item= new Item_maxmin_subselect(thd, this, select_lex, func->l_op()); - if (upper_item) - upper_item->set_sub_test(item); - } - /* fix fields is already called for left expression */ - substitution= func->create(left_expr, subs); - DBUG_RETURN(false); - } - Item* join_having= join->having ? join->having : join->tmp_having; if (!(join_having || select_lex->with_sum_func || select_lex->group_list.elements) && @@ -1503,7 +1446,6 @@ if (!substitution) { /* We're invoked for the 1st (or the only) SELECT in the subquery UNION */ - SELECT_LEX_UNIT *master_unit= select_lex->master_unit(); substitution= optimizer; SELECT_LEX *current= thd->lex->current_select; @@ -1529,16 +1471,102 @@ (char *)"<no matter>", (char *)in_left_expr_name); + } + + DBUG_RETURN(false); +} + +bool Item_allany_subselect::transform_allany(JOIN *join) +{ + DBUG_ENTER("Item_allany_subselect::transform_allany"); + if (!(in_strategy & SUBS_MAXMIN)) + DBUG_RETURN(0); + Item **place= optimizer->arguments() + 1; + THD *thd= join->thd; + SELECT_LEX *select_lex= join->select_lex; + Item *subs; + + /* + If this is an ALL/ANY single-value subselect, try to rewrite it with + a MIN/MAX subselect. We can do that if a possible NULL result of the + subselect can be ignored. + E.g. SELECT * FROM t1 WHERE b > ANY (SELECT a FROM t2) can be rewritten + with SELECT * FROM t1 WHERE b > (SELECT MAX(a) FROM t2). + We can't check that this optimization is safe if it's not a top-level + item of the WHERE clause (e.g. because the WHERE clause can contain IS + NULL/IS NOT NULL functions). If so, we rewrite ALL/ANY with NOT EXISTS + later in this method. + */ + DBUG_ASSERT(!substitution); + + if (!select_lex->group_list.elements && + !select_lex->having && + !select_lex->with_sum_func && + !(select_lex->next_select()) && + select_lex->table_list.elements) + { + Item_sum_hybrid *item; + nesting_map save_allow_sum_func; + if (func->l_op()) + { + /* + (ALL && (> || =>)) || (ANY && (< || =<)) + for ALL condition is inverted + */ + item= new Item_sum_max(*select_lex->ref_pointer_array); + } + else + { + /* + (ALL && (< || =<)) || (ANY && (> || =>)) + for ALL condition is inverted + */ + item= new Item_sum_min(*select_lex->ref_pointer_array); + } + if (upper_item) + upper_item->set_sum_test(item); + thd->change_item_tree(select_lex->ref_pointer_array, item); + { + List_iterator<Item> it(select_lex->item_list); + it++; + thd->change_item_tree(it.ref(), item); + } + + save_allow_sum_func= thd->lex->allow_sum_func; + thd->lex->allow_sum_func|= 1 << thd->lex->current_select->nest_level; /* - The uncacheable property controls a number of actions, e.g. whether to - save/restore (via init_save_join_tab/restore_tmp) the original JOIN for - plans with a temp table where the original JOIN was overriden by - make_simple_join. The UNCACHEABLE_EXPLAIN is ignored by EXPLAIN, thus - non-correlated subqueries will not appear as such to EXPLAIN. + Item_sum_(max|min) can't substitute other item => we can use 0 as + reference, also Item_sum_(max|min) can't be fixed after creation, so + we do not check item->fixed */ - master_unit->uncacheable|= UNCACHEABLE_EXPLAIN; - select_lex->uncacheable|= UNCACHEABLE_EXPLAIN; - } + if (item->fix_fields(thd, 0)) + DBUG_RETURN(true); + thd->lex->allow_sum_func= save_allow_sum_func; + /* we added aggregate function => we have to change statistic */ + count_field_types(select_lex, &join->tmp_table_param, join->all_fields, + 0); + if (join->prepare_stage2()) + DBUG_RETURN(true); + subs= new Item_singlerow_subselect(select_lex); + } + else + { + Item_maxmin_subselect *item; + subs= item= new Item_maxmin_subselect(thd, this, select_lex, func->l_op()); + if (upper_item) + upper_item->set_sub_test(item); + } + /* fix fields is already called for left expression */ + subs= func->create(left_expr, subs); + thd->change_item_tree(place, subs); + if (subs->fix_fields(thd, &subs)) + DBUG_RETURN(1); + DBUG_ASSERT(subs == (*place)); // There was no substitutions + + select_lex->master_unit()->uncacheable&= ~UNCACHEABLE_DEPENDENT_INJECTED; + select_lex->uncacheable&= ~UNCACHEABLE_DEPENDENT_INJECTED; + /* remove other strategies if there was (just to be safe) */ + in_strategy= SUBS_MAXMIN; DBUG_RETURN(false); } @@ -1556,6 +1584,16 @@ return fix_res; } +bool Item_allany_subselect::is_maxmin_applicable(JOIN *join) +{ + /* + Check if max/min optimization applicable: It is top item of + WHERE condition. + */ + return (abort_on_null || (upper_item && upper_item->top_level())) && + !join->select_lex->master_unit()->uncacheable && !func->eqne_op(); +} + /** Create the predicates needed to transform a single-column IN/ALL/ANY @@ -2028,7 +2066,7 @@ /* The IN=>EXISTS transformation makes non-correlated subqueries correlated. */ - join_arg->select_lex->uncacheable|= UNCACHEABLE_DEPENDENT; + join_arg->select_lex->uncacheable|= UNCACHEABLE_DEPENDENT_INJECTED; if (left_expr->cols() == 1) res= create_single_in_to_exists_cond(join_arg, &(join_arg->in_to_exists_where), @@ -2037,6 +2075,16 @@ res= create_row_in_to_exists_cond(join_arg, &(join_arg->in_to_exists_where), &(join_arg->in_to_exists_having)); + + /* + The uncacheable property controls a number of actions, e.g. whether to + save/restore (via init_save_join_tab/restore_tmp) the original JOIN for + plans with a temp table where the original JOIN was overriden by + make_simple_join. The UNCACHEABLE_EXPLAIN is ignored by EXPLAIN, thus + non-correlated subqueries will not appear as such to EXPLAIN. + */ + join_arg->select_lex->master_unit()->uncacheable|= UNCACHEABLE_EXPLAIN; + join_arg->select_lex->uncacheable|= UNCACHEABLE_EXPLAIN; return (res); } @@ -2107,6 +2155,9 @@ select_lex->having->top_level_item(); join_arg->having= select_lex->having; } + join_arg->thd->change_item_tree(&unit->global_parameters->select_limit, + new Item_int((int32) 1)); + unit->select_limit_cnt= 1; DBUG_RETURN(false); } @@ -2407,7 +2458,8 @@ Item_allany_subselect::select_transformer(JOIN *join) { DBUG_ENTER("Item_allany_subselect::select_transformer"); - in_strategy= SUBS_IN_TO_EXISTS; + DBUG_ASSERT((in_strategy & ~(SUBS_MAXMIN | SUBS_IN_TO_EXISTS)) == 0); + in_strategy|= SUBS_IN_TO_EXISTS; if (upper_item) upper_item->show= 1; DBUG_RETURN(select_in_like_transformer(join)); @@ -2458,6 +2510,7 @@ prepared= executed= 0; join= 0; result->cleanup(); + select_lex->uncacheable&= ~UNCACHEABLE_DEPENDENT_INJECTED; DBUG_VOID_RETURN; } @@ -2467,6 +2520,9 @@ DBUG_ENTER("subselect_union_engine::cleanup"); unit->reinit_exec_mechanism(); result->cleanup(); + unit->uncacheable&= ~UNCACHEABLE_DEPENDENT_INJECTED; + for (SELECT_LEX *sl= unit->first_select(); sl; sl= sl->next_select()) + sl->uncacheable&= ~UNCACHEABLE_DEPENDENT_INJECTED; DBUG_VOID_RETURN; } @@ -3466,11 +3522,16 @@ TRUE error */ -bool subselect_single_select_engine::change_result(Item_subselect *si, - select_result_interceptor *res) +bool +subselect_single_select_engine::change_result(Item_subselect *si, + select_result_interceptor *res, + bool temp) { item= si; - result= res; + if (temp) + thd->change_item_tree((Item**) &result, (Item*)res); + else + result= res; return select_lex->join->change_result(result); } @@ -3488,11 +3549,15 @@ */ bool subselect_union_engine::change_result(Item_subselect *si, - select_result_interceptor *res) + select_result_interceptor *res, + bool temp) { item= si; int rc= unit->change_result(res, result); - result= res; + if (temp) + thd->change_item_tree((Item**) &result, (Item*)res); + else + result= res; return rc; } @@ -3509,8 +3574,11 @@ TRUE error */ -bool subselect_uniquesubquery_engine::change_result(Item_subselect *si, - select_result_interceptor *res) +bool +subselect_uniquesubquery_engine::change_result(Item_subselect *si, + select_result_interceptor *res, + bool temp + __attribute__((unused))) { DBUG_ASSERT(0); return TRUE; @@ -4301,7 +4369,8 @@ } bool subselect_hash_sj_engine::change_result(Item_subselect *si, - select_result_interceptor *res) + select_result_interceptor *res, + bool temp __attribute__((unused))) { DBUG_ASSERT(FALSE); return TRUE; === modified file 'sql/item_subselect.h' --- sql/item_subselect.h 2011-03-30 07:10:59 +0000 +++ sql/item_subselect.h 2011-05-10 20:17:04 +0000 @@ -32,7 +32,8 @@ class Item_subselect :public Item_result_field { - bool value_assigned; /* value already assigned to subselect */ + bool value_assigned; /* value already assigned to subselect */ + bool borrowed_engine; /* the engine was taken from other Item_subselect */ protected: /* thread handler, will be assigned in fix_fields only */ THD *thd; @@ -356,6 +357,9 @@ /* Partial matching substrategies of MATERIALIZATION. */ #define SUBS_PARTIAL_MATCH_ROWID_MERGE 8 #define SUBS_PARTIAL_MATCH_TABLE_SCAN 16 +/* ALL/ANY will be transformed with max/min optimization */ +#define SUBS_MAXMIN 32 + /** Representation of IN subquery predicates of the form @@ -486,6 +490,7 @@ bool test_limit(st_select_lex_unit *unit); virtual void print(String *str, enum_query_type query_type); bool fix_fields(THD *thd, Item **ref); + void fix_length_and_dec(); void fix_after_pullout(st_select_lex *new_parent, Item **ref); void update_used_tables(); bool setup_mat_engine(); @@ -523,6 +528,8 @@ bool select_transformer(JOIN *join); void create_comp_func(bool invert) { func= func_creator(invert); } virtual void print(String *str, enum_query_type query_type); + bool is_maxmin_applicable(JOIN *join); + bool transform_allany(JOIN *join); }; @@ -594,7 +601,8 @@ static table_map calc_const_tables(TABLE_LIST *); virtual void print(String *str, enum_query_type query_type)= 0; virtual bool change_result(Item_subselect *si, - select_result_interceptor *result)= 0; + select_result_interceptor *result, + bool temp= FALSE)= 0; virtual bool no_tables()= 0; virtual bool is_executed() const { return FALSE; } /* Check if subquery produced any rows during last query execution */ @@ -626,7 +634,9 @@ void exclude(); table_map upper_select_const_tables(); virtual void print (String *str, enum_query_type query_type); - bool change_result(Item_subselect *si, select_result_interceptor *result); + bool change_result(Item_subselect *si, + select_result_interceptor *result, + bool temp); bool no_tables(); bool may_be_null(); bool is_executed() const { return executed; } @@ -655,7 +665,9 @@ void exclude(); table_map upper_select_const_tables(); virtual void print (String *str, enum_query_type query_type); - bool change_result(Item_subselect *si, select_result_interceptor *result); + bool change_result(Item_subselect *si, + select_result_interceptor *result, + bool temp= FALSE); bool no_tables(); bool is_executed() const; bool no_rows(); @@ -707,11 +719,13 @@ void fix_length_and_dec(Item_cache** row); int exec(); uint cols() { return 1; } - uint8 uncacheable() { return UNCACHEABLE_DEPENDENT; } + uint8 uncacheable() { return UNCACHEABLE_DEPENDENT_INJECTED; } void exclude(); table_map upper_select_const_tables() { return 0; } virtual void print (String *str, enum_query_type query_type); - bool change_result(Item_subselect *si, select_result_interceptor *result); + bool change_result(Item_subselect *si, + select_result_interceptor *result, + bool temp= FALSE); bool no_tables(); int index_lookup(); /* TIMOUR: this method needs refactoring. */ int scan_table(); @@ -879,7 +893,9 @@ void fix_length_and_dec(Item_cache** row);//=>base class void exclude(); //=>base class //=>base class - bool change_result(Item_subselect *si, select_result_interceptor *result); + bool change_result(Item_subselect *si, + select_result_interceptor *result, + bool temp= FALSE); bool no_tables();//=>base class }; @@ -1106,7 +1122,9 @@ uint8 uncacheable() { return UNCACHEABLE_DEPENDENT; } void exclude() {} table_map upper_select_const_tables() { return 0; } - bool change_result(Item_subselect*, select_result_interceptor*) + bool change_result(Item_subselect*, + select_result_interceptor*, + bool temp= FALSE) { DBUG_ASSERT(FALSE); return false; } bool no_tables() { return false; } bool no_rows() === modified file 'sql/mysql_priv.h' --- sql/mysql_priv.h 2011-05-10 15:28:05 +0000 +++ sql/mysql_priv.h 2011-05-10 20:17:04 +0000 @@ -679,14 +679,18 @@ #define CONTEXT_ANALYSIS_ONLY_DERIVED 4 // uncachable cause -#define UNCACHEABLE_DEPENDENT 1 -#define UNCACHEABLE_RAND 2 -#define UNCACHEABLE_SIDEEFFECT 4 +#define UNCACHEABLE_DEPENDENT_GENERATED 1 +#define UNCACHEABLE_RAND 2 +#define UNCACHEABLE_SIDEEFFECT 4 /// forcing to save JOIN for explain -#define UNCACHEABLE_EXPLAIN 8 +#define UNCACHEABLE_EXPLAIN 8 /* For uncorrelated SELECT in an UNION with some correlated SELECTs */ -#define UNCACHEABLE_UNITED 16 -#define UNCACHEABLE_CHECKOPTION 32 +#define UNCACHEABLE_UNITED 16 +#define UNCACHEABLE_CHECKOPTION 32 +#define UNCACHEABLE_DEPENDENT_INJECTED 64 + +#define UNCACHEABLE_DEPENDENT (UNCACHEABLE_DEPENDENT_GENERATED | \ + UNCACHEABLE_DEPENDENT_INJECTED) /* Used to check GROUP BY list in the MODE_ONLY_FULL_GROUP_BY mode */ #define UNDEF_POS (-1) === modified file 'sql/opt_subselect.cc' --- sql/opt_subselect.cc 2011-05-02 18:59:16 +0000 +++ sql/opt_subselect.cc 2011-05-10 20:17:04 +0000 @@ -93,8 +93,19 @@ (subselect= parent_unit->item)) // (2) { Item_in_subselect *in_subs= NULL; - if (subselect->substype() == Item_subselect::IN_SUBS) - in_subs= (Item_in_subselect*)subselect; + Item_allany_subselect *allany_subs= NULL; + switch (subselect->substype()) { + case Item_subselect::IN_SUBS: + in_subs= (Item_in_subselect *)subselect; + break; + case Item_subselect::ALL_SUBS: + case Item_subselect::ANY_SUBS: + allany_subs= (Item_allany_subselect *)subselect; + break; + default: + break; + } + /* Resolve expressions and perform semantic analysis for IN query */ if (in_subs != NULL) @@ -257,12 +268,18 @@ } } + /* Check if max/min optimization applicable */ + if (allany_subs) + allany_subs->in_strategy|= (allany_subs->is_maxmin_applicable(join) ? + SUBS_MAXMIN : + SUBS_IN_TO_EXISTS); + /* Transform each subquery predicate according to its overloaded transformer. */ if (subselect->select_transformer(join)) - DBUG_RETURN(-11); + DBUG_RETURN(-1); } } DBUG_RETURN(0); @@ -369,6 +386,21 @@ } +/** + Apply max min optimization of all/any subselect +*/ + +bool convert_max_min_subquery(JOIN *join) +{ + DBUG_ENTER("convert_max_min_subquery"); + Item_subselect *subselect= join->unit->item; + if (!subselect || (subselect->substype() != Item_subselect::ALL_SUBS && + subselect->substype() != Item_subselect::ANY_SUBS)) + DBUG_RETURN(0); + DBUG_RETURN(((Item_allany_subselect *) subselect)->transform_allany(join)); +} + + /* Convert semi-join subquery predicates into semi-join join nests @@ -3843,8 +3875,8 @@ restore_query_plan(&save_qep); /* TODO: should we set/unset this flag for both select_lex and its unit? */ - in_subs->unit->uncacheable&= ~UNCACHEABLE_DEPENDENT; - select_lex->uncacheable&= ~UNCACHEABLE_DEPENDENT; + in_subs->unit->uncacheable&= ~UNCACHEABLE_DEPENDENT_INJECTED; + select_lex->uncacheable&= ~UNCACHEABLE_DEPENDENT_INJECTED; /* Reset the "LIMIT 1" set in Item_exists_subselect::fix_length_and_dec. @@ -3884,6 +3916,9 @@ if (in_subs->inject_in_to_exists_cond(this)) return TRUE; + in_subs->unit->uncacheable|= UNCACHEABLE_DEPENDENT_INJECTED; + select_lex->uncacheable|= UNCACHEABLE_DEPENDENT_INJECTED; + select_limit= 1; } else DBUG_ASSERT(FALSE); === modified file 'sql/opt_subselect.h' --- sql/opt_subselect.h 2010-12-11 07:23:34 +0000 +++ sql/opt_subselect.h 2011-05-10 20:17:04 +0000 @@ -6,6 +6,7 @@ int check_and_do_in_subquery_rewrites(JOIN *join); bool convert_join_subqueries_to_semijoins(JOIN *join); +bool convert_max_min_subquery(JOIN *join); int pull_out_semijoin_tables(JOIN *join); bool optimize_semijoin_nests(JOIN *join, table_map all_table_map); === modified file 'sql/sql_class.h' --- sql/sql_class.h 2011-05-10 15:28:05 +0000 +++ sql/sql_class.h 2011-05-10 20:17:04 +0000 @@ -2727,7 +2727,12 @@ class select_result_interceptor: public select_result { public: - select_result_interceptor() {} /* Remove gcc warning */ + select_result_interceptor() + { + DBUG_ENTER("select_result_interceptor::select_result_interceptor"); + DBUG_PRINT("enter", ("this 0x%lx", (ulong) this)); + DBUG_VOID_RETURN; + } /* Remove gcc warning */ uint field_count(List<Item> &fields) const { return 0; } bool send_fields(List<Item> &fields, uint flag) { return FALSE; } }; === modified file 'sql/sql_lex.cc' --- sql/sql_lex.cc 2011-05-05 12:24:28 +0000 +++ sql/sql_lex.cc 2011-05-10 20:17:04 +0000 @@ -1942,18 +1942,19 @@ SELECT_LEX *s= this; do { - if (!(s->uncacheable & UNCACHEABLE_DEPENDENT)) + if (!(s->uncacheable & UNCACHEABLE_DEPENDENT_GENERATED)) { // Select is dependent of outer select s->uncacheable= (s->uncacheable & ~UNCACHEABLE_UNITED) | - UNCACHEABLE_DEPENDENT; + UNCACHEABLE_DEPENDENT_GENERATED; SELECT_LEX_UNIT *munit= s->master_unit(); munit->uncacheable= (munit->uncacheable & ~UNCACHEABLE_UNITED) | - UNCACHEABLE_DEPENDENT; + UNCACHEABLE_DEPENDENT_GENERATED; for (SELECT_LEX *sl= munit->first_select(); sl ; sl= sl->next_select()) { if (sl != s && - !(sl->uncacheable & (UNCACHEABLE_DEPENDENT | UNCACHEABLE_UNITED))) + !(sl->uncacheable & (UNCACHEABLE_DEPENDENT_GENERATED | + UNCACHEABLE_UNITED))) sl->uncacheable|= UNCACHEABLE_UNITED; } } @@ -2178,17 +2179,6 @@ subs_type == Item_subselect::IN_SUBS || subs_type == Item_subselect::ALL_SUBS) { - DBUG_ASSERT(!item->fixed || - /* - If not using materialization both: - select_limit == 1, and there should be no offset_limit. - */ - (((subs_type == Item_subselect::IN_SUBS) && - ((Item_in_subselect*)item)->in_strategy & - SUBS_MATERIALIZATION) ? - TRUE : - (select_limit->val_int() == 1LL) && - offset_limit == 0)); return; } } === modified file 'sql/sql_lex.h' --- sql/sql_lex.h 2011-05-02 18:59:16 +0000 +++ sql/sql_lex.h 2011-05-10 20:17:04 +0000 @@ -417,7 +417,8 @@ /* result of this query can't be cached, bit field, can be : - UNCACHEABLE_DEPENDENT + UNCACHEABLE_DEPENDENT_GENERATED + UNCACHEABLE_DEPENDENT_INJECTED UNCACHEABLE_RAND UNCACHEABLE_SIDEEFFECT UNCACHEABLE_EXPLAIN === modified file 'sql/sql_select.cc' --- sql/sql_select.cc 2011-05-05 12:24:28 +0000 +++ sql/sql_select.cc 2011-05-10 20:17:04 +0000 @@ -736,11 +736,28 @@ if (!procedure && result && result->prepare(fields_list, unit_arg)) goto err; /* purecov: inspected */ + unit= unit_arg; + if (prepare_stage2()) + goto err; + + DBUG_RETURN(0); // All OK + +err: + delete procedure; /* purecov: inspected */ + procedure= 0; + DBUG_RETURN(-1); /* purecov: inspected */ +} + + +bool JOIN::prepare_stage2() +{ + bool res= TRUE; + DBUG_ENTER("JOIN::prepare_stage2"); + /* Init join struct */ count_field_types(select_lex, &tmp_table_param, all_fields, 0); ref_pointer_array_size= all_fields.elements*sizeof(Item*); this->group= group_list != 0; - unit= unit_arg; if (tmp_table_param.sum_func_count && !group_list) implicit_grouping= TRUE; @@ -757,12 +774,9 @@ if (alloc_func_list()) goto err; - DBUG_RETURN(0); // All OK - + res= FALSE; err: - delete procedure; /* purecov: inspected */ - procedure= 0; - DBUG_RETURN(-1); /* purecov: inspected */ + DBUG_RETURN(res); /* purecov: inspected */ } @@ -795,7 +809,8 @@ set_allowed_join_cache_types(); /* dump_TABLE_LIST_graph(select_lex, select_lex->leaf_tables); */ - if (convert_join_subqueries_to_semijoins(this)) + if (convert_max_min_subquery(this) || + convert_join_subqueries_to_semijoins(this)) DBUG_RETURN(1); /* purecov: inspected */ /* dump_TABLE_LIST_graph(select_lex, select_lex->leaf_tables); */ @@ -8599,6 +8614,10 @@ void JOIN_TAB::cleanup() { + DBUG_ENTER("JOIN_TAB::cleanup"); + DBUG_PRINT("enter", ("table %s.%s", + (table ? table->s->db.str : "?"), + (table ? table->s->table_name.str : "?"))); delete select; select= 0; delete quick; @@ -8620,6 +8639,7 @@ table->reginfo.join_tab= 0; } end_read_record(&read_record); + DBUG_VOID_RETURN; } @@ -8740,7 +8760,8 @@ Optimization: if not EXPLAIN and we are done with the JOIN, free all tables. */ - bool full= (!select_lex->uncacheable && !thd->lex->describe); + bool full= (!(select_lex->uncacheable) && + !thd->lex->describe); bool can_unlock= full; DBUG_ENTER("JOIN::join_free"); @@ -8804,6 +8825,7 @@ void JOIN::cleanup(bool full) { DBUG_ENTER("JOIN::cleanup"); + DBUG_PRINT("enter", ("full %u", (uint) full)); if (table) { @@ -8829,7 +8851,11 @@ for (tab= join_tab, end= tab+tables; tab != end; tab++) { if (tab->table) + { + DBUG_PRINT("info", ("close index: %s.%s", tab->table->s->db.str, + tab->table->s->table_name.str)); tab->table->file->ha_index_or_rnd_end(); + } } } } @@ -20307,6 +20333,7 @@ change select_result object of JOIN. @param res new select_result object + @param temp temporary assignment @retval FALSE OK === modified file 'sql/sql_select.h' --- sql/sql_select.h 2011-05-02 18:59:16 +0000 +++ sql/sql_select.h 2011-05-10 20:17:04 +0000 @@ -999,6 +999,7 @@ COND *conds, uint og_num, ORDER *order, ORDER *group, Item *having, ORDER *proc_param, SELECT_LEX *select, SELECT_LEX_UNIT *unit); + bool prepare_stage2(); int optimize(); int reinit(); int init_execution();

2 1

[Maria-developers] feature documentation
by Kurt von Finck 11 May '11

11 May '11

IMO, the Knowledgbase entry for the virtual columns http://kb.askmonty.org/v/virtual-columns is among the best and most useful entries we have. It describes the feature, suggests use cases and demonstrates functionality in a straightforward manner. I'd like to see equal care taken to provide the same information for all user-facing features implemented in MariaDB. If developers can devote a little time to ensuring such documentation is started, the community (myself included) will be happy to expand the articles to usable states. Just ask on -discuss or IRC. I think KB articles can be started in the feature planning phase, and updated as the feature is implemented. I imagine the KB acting as an ongoing feature implementation and use log is more likely to maintain quality developer input than is asking an entire document be written from scratch after the feature is merged. Thoughts? -- ./k Kurt von Finck Chief Community And Communications Officer Monty Program http://montyprogram.com

2 1

[Maria-developers] Summary of my buildbot and bakery work
by Henrik Ingo 11 May '11

11 May '11

Kristian, Serg, Colin Thanks for showing an interest in my work with the MariaDB buildsystem and bakery scripts. During Open DB camp I finally managed to finish the last pending task and published a blog post about it. So in this mail I'll summarize once more what was done and highlight how it may be relevant to MariaDB, so you can pick enhancements that you find useful. This blog post has links to the series published in February: http://openlife.cc/blogs/2011/march/overview-and-archeological-exploration-… It links to posts 1-7. And this is the new one: http://openlife.cc/blogs/2011/may/easy-way-manage-virtualcloud-images-outsi… let's call it number 8. So the high level of what I've done is this: 5) http://openlife.cc/blogs/2011/february/mariadb-and-mepsql-buildbot-setups Instead of the standard BuildSlave class, use my own subclass of EC2LatentBuildSlave. This allows migrating the buildbot configuration to EC2 cloud. Relevance to MariaDB: If you don't want to use a cloud service (faster turnaround times due to more servers in parallel, but costs more than what you do now) not much. However, there is a small value in the refactoring effort itself. Now you have the runvm stuff hardwired into the test cases. The approach used with EC2LatentBuildSlave abstracts the handling of virtual instances into python code, and the test cases are identical whether you have the classic physical slaves or virtual/cloud slaves. They are cleaner to read and can be moved between classic slave and virtual slave without modifications. Unfortunately, to do this, you'd need to both a) write a subclass of AbstractLatentBuildSlave that does what runvm does for you today, and b) clean up the runvm parts from your builds/tests, since I only did so for a small part and we have probably diverged since then anyway. As part of (5), I also produced an apt repository, not just standalone deb packages. But you have implemented that separately too. I advised you to use something else than ~ for the separator in filename. 6) http://openlife.cc/blogs/2011/february/dealing-cambrian-explosion-12-how-pa… 7) http://openlife.cc/blogs/2011/february/dealing-cambrian-explosion-22-parame… Parameterize the package name (mysql, mariadb, percona-server, mepsql...) in the scripts that build packages (with sed and other shell magic). See http://bazaar.launchpad.net/~mepsql-committers/mepsql-bakery/trunk/view/hea… Relevance to MariaDB: Nothing as such, however, if you adopted these changes, it would enable you to share the packaging scripts and even buildbot setup with Percona. Possibly also with Linux distributions that are packaging multiple MySQL forks now, although their problem is not quite the same. (you produce a fork for many distros, they package many forks for one distro) Personally, I like this enhancement both because it centralizes all the variables that end up in package names and version strings, and because it would enable all forks (including, theoretically, Oracle MySQL) to share the same packaging script. This is not at all possible with how things are packaged traditionally, where you assume the package name to be constant and it is used as a hardwired value around all scripts and filenames. I didn't yet do this for RPMs and don't know when, if ever, I will. 8) http://openlife.cc/blogs/2011/may/easy-way-manage-virtualcloud-images-outsi… Instead of configuring a vm image and saving it, only use generic images and run external script on boot. Relevance to MariaDB: high. I know you complained about this in your otherwise perfectly automated build system. Even in my small project I experienced huge gains during development thanks to this approach. You can use this technique even with your current runvm setup. (You just need to edit your images one last time to enable fetching of the remote configuration script.) That's all. I hope some of this is seen useful. henrik -- henrik.ingo(a)avoinelama.fi +358-40-8211286 skype: henrik.ingo irc: hingo www.openlife.cc My LinkedIn profile: http://www.linkedin.com/profile/view?id=9522559

1 0

[Maria-developers] WL#199 New (by Monty): Change Item:name to LEX_STRING
by worklog-noreply＠askmonty.org 02 May '11

02 May '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Change Item:name to LEX_STRING CREATION DATE..: Mon, 02 May 2011, 15:04 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-BackLog TASK ID........: 199 (http://askmonty.org/worklog/?tid=199) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 6 (hours remain) ORIG. ESTIMATE.: 6 PROGRESS NOTES: DESCRIPTION: Change Item::name, Item::org_name to LEX_STRING objects (from char*) This allows us to remove Item::max_length and some calls to strlen(Item->name). Reasons: - Simpler and faster code - Allow us to remove calls to strlen() - Fixes some known bugs where Item->max_length and Item::name is not kept in sync. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Dynamic column for review
by Oleksandr Byelkin 29 Apr '11

29 Apr '11

Hi! Here is 2 patches: one cut version - to review, other one full version.

1 0

Re: [Maria-developers] [Commits] Rev 2908: Fixed LP bugs #717577, #724942. in file:///home/igor/maria/maria-5.3-bug717577/
by Sergey Petrunya 25 Apr '11

25 Apr '11

Hello Igor, First, an overall comment: there are lots of typos/coding style violations in the patch. To reduce amount of effort spent on such things, I was just fixing them as I saw them and I'm attaching the patch with all the fixes (i.e. this patch should be applied on top of the patch that I was reviewing). More important comments are bellow, marked with 'psergey:' On Fri, Mar 25, 2011 at 10:31:19PM -0700, Igor Babaev wrote: > At file:///home/igor/maria/maria-5.3-bug717577/ > > ------------------------------------------------------------ > revno: 2908 > revision-id: igor(a)askmonty.org-20110326053117-x50cah9krd4f1345 > parent: wlad(a)montyprogram.com-20110212174322-f7ptnc0u32vasdwy > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-bug717577 > timestamp: Fri 2011-03-25 22:31:17 -0700 > message: > Fixed LP bugs #717577, #724942. > > Both these two bugs happened due to the following problem. > When a view column is referenced in the query an Item_direct_view_ref > object is created that is refers to the Item_field for the column. > All references to the same view column refer to the same Item_field. > Different references can belong to different AND/OR levels and, > as a result, can be included in different Item_equal object. > These Item_equal objects may include different constant objects. > If these constant objects are substituted for the Item_field created > for a view column we have a conflict situation when the second > substitution annuls the first substitution. This leads to > wrong result sets returned by the query. Bug #724942 demonstrates > such an erroneous behaviour. > Test case of the bug #717577 produces wrong result sets because best > equal fields of the multiple equalities built for different OR levels > of the WHERE condition differs. The subsitution for the best equal field > in the second OR branch overwrites the the substitution made for the > first branch. > > To avoid such conflicts we have to substitute for the references > to the view columns rather than for the underlying field items. > To make such substitutions possible we have to include into > multiple equalities references to view columns rather than > field items created for such columns. > > This patch modifies the Item_equal class to include references > to view columns into multiple equality objects. It also performs > a clean up of the class methods and adds more comments. The methods > of the Item_direct_view_ref class that assist substitutions for > references to view columns has been also added by this patch. > === modified file 'sql/item.cc' > --- a/sql/item.cc 2011-02-01 03:33:32 +0000 > +++ b/sql/item.cc 2011-03-26 05:31:17 +0000 > @@ -7226,6 +7233,129 @@ > return FALSE; > } > > + > +Item_equal *Item_direct_view_ref::find_item_equal(COND_EQUAL *cond_equal) > +{ > + Item* field_item= real_item(); > + if (field_item->type() != FIELD_ITEM) > + return NULL; > + return ((Item_field *) field_item)->find_item_equal(cond_equal); > +} > + > + > +/** > + Check whether a reference to field item can be substituted b an equal item > + > + @details > + The function checks whether a substitution of a reference to field item for > + an equal item is valid. > + > + @param arg *arg != NULL && **arg <-> the reference is in the context > + where substitution for an equal item is valid > + > + @note > + See also the note for Item_field::subst_argument_checker > + > + @retval > + TRUE substitution is valid > + @retval > + FALSE otherwise > +*/ > +bool Item_direct_view_ref::subst_argument_checker(uchar **arg) > +{ > + bool res= (!(*arg) && (result_type() != STRING_RESULT)) || > + ((*arg) && (**arg)); > + if (*arg) > + **arg= (uchar) 0; psergey: What is the above for? Do I understand it correctly that this is needed so that Item_field that this item is wrapping is not substituted? Please add a comment. > + return res; > +} > + > + > +/** > + Set a pointer to the multiple equality the view field reference belongs to > + (if any). > + > + @details > + The function looks for a multiple equality containing this item of the type > + Item_direct_view_ref among those referenced by arg. > + In the case such equality exists the function does the following. > + If the found multiple equality contains a constant, then the item > + is substituted for this constant, otherwise the function sets a pointer > + to the multiple equality in the item. > + > + @param arg reference to list of multiple equalities where > + the item (this object) is to be looked for > + > + @note > + This function is supposed to be called as a callback parameter in calls > + of the compile method. > + > + @note > + The function calls Item_field::equal_fields_propagator for the field item > + this->real_item() to do the job. Then it takes the pointer to equal_item > + from this field item and assigns it to this->item_equal. > + > + @return > + - pointer to the replacing constant item, if the field item was substituted > + - pointer to the field item, otherwise. > +*/ > + > +Item *Item_direct_view_ref::equal_fields_propagator(uchar *arg) > +{ > + Item *field_item= real_item(); > + if (field_item->type() != FIELD_ITEM) > + return this; > + Item *item= field_item->equal_fields_propagator(arg); > + set_item_equal(field_item->get_item_equal()); > + field_item->set_item_equal(0); psergey: Please use NULL here, we use that for pointers. > + if (item != field_item) > + return item; > + return this; > +} > + ... > === modified file 'sql/item_cmpfunc.cc' > --- a/sql/item_cmpfunc.cc 2011-02-06 04:57:03 +0000 > +++ b/sql/item_cmpfunc.cc 2011-03-26 05:31:17 +0000 > @@ -5516,43 +5516,93 @@ > return 0; > } > > -Item_equal::Item_equal(Item_field *f1, Item_field *f2) > - : Item_bool_func(), const_item(0), eval_item(0), cond_false(0), > - compare_as_dates(FALSE) > -{ > - const_item_cache= 0; > - fields.push_back(f1); > - fields.push_back(f2); > -} > - > -Item_equal::Item_equal(Item *c, Item_field *f) > + > +/** > + Construct a minimal multiple equality item > + > + @param f1 the first equal item > + @param f2 the second equal item > + @param with_const_item TRUE if the first item is constant > + > + @details > + The constructor builds a new item equal object for the equality f1=f2. > + One if the equal items can be constant. If this is the case it is passed > + always as the first parameter and the parameter with_const_item serves > + as an indicator of this case. > + Currently any non-constant parameter items must refer to an item of the > + of the type FIELD_ITEM. psergey: The above is still true for the case where passed item is an Item_direct_view_ref which refers to an Item_field. The wording may be confusing for the new readers, though. Please change to explicitly indicate that f1 and f2 may point to Item_field or Item_direct_view_ref(Item_field) > +*/ > + > +Item_equal::Item_equal(Item *f1, Item *f2, bool with_const_item) > : Item_bool_func(), eval_item(0), cond_false(0) > { > const_item_cache= 0; > - fields.push_back(f); > - const_item= c; > - compare_as_dates= f->is_datetime(); > + with_const= with_const_item; > + compare_as_dates= with_const_item && f2->is_datetime(); > + equal_items.push_back(f1); > + equal_items.push_back(f2); > } > > > +/** > + Copy constructor for a multiple equality > + > + @param item_equal source item for the constructor > + > + @details > + The function creates a copy of an Item_equal object. > + This constructor is used when an item belongs to a multiple equality > + of an upper level (an upper AND/OR level or an upper level of a nested > + outer join). > +*/ > + > Item_equal::Item_equal(Item_equal *item_equal) > : Item_bool_func(), eval_item(0), cond_false(0) > { > const_item_cache= 0; > - List_iterator_fast<Item_field> li(item_equal->fields); > - Item_field *item; > + List_iterator_fast<Item> li(item_equal->equal_items); > + Item *item; > while ((item= li++)) > { > - fields.push_back(item); > + equal_items.push_back(item); > } > - const_item= item_equal->const_item; > + with_const= item_equal->with_const; > compare_as_dates= item_equal->compare_as_dates; > cond_false= item_equal->cond_false; > } > > > -void Item_equal::compare_const(Item *c) > +/* > + @brief > + Add a constant item to the Item_equal object > + > + @param[in] c the constant to add > + @param[in] f item from the list equal_items the item c is equal to > + (this parameter is optional) > + > + @details > + The method adds the constant item c to the list equal_items. If the list > + hasn't not contained any constant item yet the item c is just added > + to the very beginning of the list. Otherwise the value of c is compared > + with the value of the constant item from equal_items. If they are not > + equal cond_false is set to TRUE. This serves as an indicator that this > + Item_equal is always FALSE. psergey: Does function comment really need to duplicate the function body? In my opinion, the above should be split into multiple comments inside the function body. > + The optional parameter f is used to adjust the flag compare_as_dates. > +*/ > + > +void Item_equal::add_const(Item *c, Item *f) > { > + if (cond_false) > + return; > + if (!with_const) > + { > + with_const= TRUE; > + if (f) > + compare_as_dates= f->is_datetime(); > + equal_items.push_front(c); > + return; > + } > + Item *const_item= get_const(); > if (compare_as_dates) > { > cmp.set_datetime_cmp_func(this, &c, &const_item); ... > @@ -5811,19 +5911,15 @@ > return Item_func::transform(transformer, arg); > } > > + > void Item_equal::print(String *str, enum_query_type query_type) > { > str->append(func_name()); > str->append('('); > - List_iterator_fast<Item_field> it(fields); > + List_iterator_fast<Item> it(equal_items); > Item *item; > - if (const_item) > - const_item->print(str, query_type); > - else > - { > - item= it++; > - item->print(str, query_type); > - } > + item= it++; > + item->print(str, query_type); > while ((item= it++)) > { > str->append(','); > @@ -5834,6 +5930,14 @@ > } > > > +CHARSET_INFO *Item_equal::compare_collation() psergey: One expects compare_xxx() functions to compare something, in particular this function looks like it's going to compare some collation. Could you please rename it to e.g. comparison_collation()? > +{ > + Item_equal_fields_iterator it(*this); > + Item *item= it++; > + return item->collation.collation; > +} > + > + > /* > @brief Get the first equal field of multiple equality. > @param[in] field the field to get equal field to ... > === modified file 'sql/item_cmpfunc.h' > --- a/sql/item_cmpfunc.h 2010-10-10 14:18:11 +0000 > +++ b/sql/item_cmpfunc.h 2011-03-26 05:31:17 +0000 > @@ -1672,23 +1707,47 @@ > }; > > psergey: Please add a one-line comment describing what this class does. > -class Item_equal_iterator : public List_iterator_fast<Item_field> > +class Item_equal_fields_iterator : public List_iterator_fast<Item> > { > + Item_equal *item_equal; > + Item *curr_item; > public: > - inline Item_equal_iterator(Item_equal &item_equal) > - :List_iterator_fast<Item_field> (item_equal.fields) > - {} > - inline Item_field* operator++(int) > - { > - Item_field *item= (*(List_iterator_fast<Item_field> *) this)++; > - return item; > - } > - inline void rewind(void) > - { > - List_iterator_fast<Item_field>::rewind(); > - } > + Item_equal_fields_iterator(Item_equal &item_eq) > + :List_iterator_fast<Item> (item_eq.equal_items) > + { > + curr_item= NULL; > + item_equal= &item_eq; > + if (item_eq.with_const) > + { > + List_iterator_fast<Item> *list_it= (List_iterator_fast<Item> *) this; psergey: Why do redundant explicit typecasting? > + curr_item= (*list_it)++; > + } > + } > + Item* operator++(int) > + { > + List_iterator_fast<Item> *list_it= (List_iterator_fast<Item> *) this; psergey: Why do redundant explicit typecasting? > + curr_item= (*list_it)++; > + return curr_item; > + } > + Item ** ref() > + { > + return List_iterator_fast<Item>::ref(); > + } > + void rewind(void) > + { > + List_iterator_fast<Item> *list_it= (List_iterator_fast<Item> *) this; > + list_it->rewind(); > + if (item_equal->with_const) > + curr_item= (*list_it)++; > + } > + Field *get_curr_field() > + { > + Item_field *item= (Item_field *) (curr_item->real_item()); > + return item->field; > + } > }; > > + > class Item_cond_and :public Item_cond > { > public: > > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2011-02-11 10:27:35 +0000 > +++ b/sql/sql_select.cc 2011-03-26 05:31:17 +0000 > @@ -9623,7 +9630,8 @@ > args->concat((List<Item> *)&cond_equal.current_level); > } > } > - else if (cond->type() == Item::FUNC_ITEM) > + else if (cond->type() == Item::FUNC_ITEM || > + cond->real_item()->type() == Item::FIELD_ITEM) psergey: As far as I understand the last part of the condition is there so that direct references like ... AND view.column AND ... are processed with subst_argument_checker/equal_fields_propagator. Is it intentional that direct references like ... AND base_table.column AND ... are now processed with subst_argument_checker/equal_fields_propagator as well? > { > List<Item> eq_list; > /* -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 3

Re: [Maria-developers] A serious merge bug in 5.3.
by Sergei Golubchik 24 Apr '11

24 Apr '11

Hi, Igor! On Apr 24, Igor Babaev wrote: > > When debugging I discovered the following bad code in sql_select.cc of > the current 5.3 tree: > tmp= table->file->read_time(key, 1, > min(tmp,s->worst_seeks)-1); <= !!!!! ? > tmp*= record_count; > > gannotate says that it appeared in the code after your merge with 5.2. > > The fact is that 5.2 contains the correct expression here > min(tmp,s->worst_seeks) > not > min(tmp,s->worst_seeks)-1 > > As a result if the number of records is 1 then min(tmp,s->worst_seeks)-1 > is equal to 0 and we pass 0 as the last parameter to > table->file->read_time. Naturally it returns 0. > > Will you fix this problem or you'd rather want me to do it? In 5.2 it is tmp= record_count*min(tmp,s->worst_seeks); In 5.3, record_count* is moved to a separate line, so it should just be tmp= min(tmp,s->worst_seeks); But it uses table->file->read_time(). Which is defined as virtual double read_time(uint index, uint ranges, ha_rows rows) { return rows2double(ranges+rows); } so, when ranges=1 and we pass rows=min(tmp,s->worst_seeks)-1, we get the same value of min(tmp,s->worst_seeks) as the result. This was the change of introducing handler::keyread_time(), and at the same time using handler::read_time() where appropriate, to have all values comparable. Regards, Sergei

2 2

[Maria-developers] MariaDB/MySQL error code issue
by Kristian Nielsen 19 Apr '11

19 Apr '11

Hi, For merging MySQL into MariaDB, we need to work out what to do about error code. The problem occurs as (different) new error messages are added in both MySQL and MariaDB. When we then merge from MySQL, we get conflicts between the error numbers. If we do not handle this somehow, it means that error numbers for a given error message will be different, either between MySQL and MariaDB, or alternatively between one version of MariaDB and another (depending on how conflicts are resolved). The problem with error code changing is that this makes it much harder for applications to check error codes and handle errors as it want in a portable way between MySQL and MariaDB. I think we have these options: 1. Cooperate with MySQL@Oracle, so that whenever we add an error code in MariaDB, the corresponding code is also reserved in MySQL (even if not used), and vice versa. 2. Create different "namespaces" for error codes added to MariaDB and MySQL, eg. new MariaDB codes are assigned with some offset to MySQL codes, so we avoid conflicts. 3. Keep MariaDB error code numbers stable, re-assign MySQL error code numbers when merging from MySQL (I rather dislike this as well because it introduces incompatibility with MySQL). 4. Keep MySQL error code numbers when merging from MySQL, so that MariaDB-only errors will fluctuate between releases (I really dislike this option). It seems option (4) is the best option. Is someone from MySQL/Oracle willing to work on getting such a procedure running? Failing that, it seems (3) is the best option. In this case, we need (at least) to assign a suitable error code range to new MariaDB errors and change the source code to be able to handle this (eg. maybe we need a sql/share/errmsg-mariadb.txt or something). We will also need to handle the error codes already assigned to virtual columns, which I think already conflict with new MySQL error codes. I would suggest doing a one-time incompatible change to map them into the MariaDB range. So, any hope for getting solution (4) working with MySQL@Oracle? And failing that, any opinions on how to proceed? Thanks, - Kristian.

4 6

[Maria-developers] WL#198 New (by Sergei): Roles
by worklog-noreply＠askmonty.org 19 Apr '11

19 Apr '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Roles CREATION DATE..: Tue, 19 Apr 2011, 10:01 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 198 (http://askmonty.org/worklog/?tid=198) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 250 (hours remain) ORIG. ESTIMATE.: 250 PROGRESS NOTES: DESCRIPTION: Roles, as defined by the SQL standard. See also http://forge.mysql.com/worklog/task.php?id=988 ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Testing group commit with XtraBackup
by Kristian Nielsen 19 Apr '11

19 Apr '11

Hi Philip, I thought of another test of group commit that ought to be done (not sure why I didn't think to mention this earlier, sorry about that). It's not urgent, just something for the list. The test is basically the same you already did for non-blocking `mysqldump --master-data --single-transaction` used to provision a slave, but using XtraBackup instead of mysqldump. >From what I could learn, the original reason that InnoDB added the mutex that kills group commit when binlog is enabled is the following: To ensure that commit order in the InnoDB transaction log is the same as in the binlog. This is needed when XtraBackup (innobackup really, but they work the same and XtraBackup is Free Software) takes a non-blocking backup of the InnoDB data (only), and this is used to provision a new slave. If commit order was different, there might not be any binlog position on the master corresponding to the backup snapshot used to setup the new slave. Now, my group commit work also guarantees consistent commit order, but using a completely different method that does not kill group commit. So we ought to test this. So the test would be similar to your existing mysqldump test. But while the master is running, we would use non-blocking XtraBackup to take a snapshot. This snapshot we would then restore onto a new slave. The slave would then be configured with CHANGE MASTER TO, I believe one gets the correct binlog position to start from in the log file of XtraDB and/or slave mysqld (I need to check up on details). Then the test proceeds as the mysqldump case, letting the slave catch up and checking that replication is ok etc. What do you think? Now, to do this we would need to be able to use XtraBackup. I am wondering if it wouldn't make sense to at the same time include XtraBackup into MariaDB? XtraBackup seems to me quite mature already, and a very good product besides. It is arguably long overdue for us to include it. So if we include XtraBackup in mariadb-5.2-rpl (/5.3), I think this should be possible to setup without too much trouble (I can provide specific details/scripts for how to actually run XtraBackup and setup slave start position, I just need to research that a bit). Or we could also do the test first with XtraBackup as external, shouldn't be much harder, and would give us some experience with it. What do you think? Maybe we can look at it in more detail in Lisbon. - Kristian.

2 3

[Maria-developers] WL#197 New (by Mdcallag): table lock wait timeouts
by worklog-noreply＠askmonty.org 13 Apr '11

13 Apr '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: table lock wait timeouts CREATION DATE..: Wed, 13 Apr 2011, 17:09 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 197 (http://askmonty.org/worklog/?tid=197) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: MySQL has table_lock_wait_timeout but that doesn't do anything. For some apps it would help to timeout sessions blocked on a table lock. For example, when ALTER TABLE runs it implicitly gets an X lock on the table cache entry. Queries that follow are blocked until it finishes. The client-side read timeout might expire and disconnect the client but the server-side thread will continue blocking and the server can fill with blocked sessions. Right now the stack when the client blocks is open_tables -> open_table -> wait_for_condition. I implemented a demo for this by: 1) use pthread_cond_timedwait in wait_for_condition 2) check for timeout expiration in open_tables and error out Without the change in 2) the session will loop within open_tables on calls to open_table. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

Re: [Maria-developers] Windows installer MWL#55 finished.
by Vladislav Vaintroub 13 Apr '11

13 Apr '11

> -----Original Message----- > From: kahzoo(a)gmail.com [mailto:kahzoo@gmail.com] On Behalf Of Kazuhisa > Ichikawa > Sent: Sonntag, 6. Februar 2011 09:32 > To: Vladislav Vaintroub > Cc: Philip Stoev; maria-developers(a)lists.launchpad.net > Subject: Re: [Maria-developers] Windows installer MWL#55 finished. Hello, Kazuhisa > No :-). My experience with MySQL installers is limited to fresh > install. This is the first time I tried 'upgrade' and my expectation > was purely imaginary. Right, there is no upgrade functionality in MySQLs installers up-to-now (nor I heard or any planning for it recently). But I noticed that both you and Philip had quite similar requests for the changed UI in case of "upgrade possible". Both of you recommended changes at the "Database Properties" dialog, I think it is wrong place, by the time you enter this dialog it is too late, you should not have entered this dialog, as the intention was not "create new", but "upgrade existing" anyway.. So since both of my users have pointed to usability issues, I added a new dialog that comes after license agreement that and tells user that upgradable instances were found and please make a choice between installing a new instance and not installing a new instance(optional upgrade one or more existing ones). This sounds almost like your first suggestion, but in own dialog. This dialog comes right before feature selection, and only thing it might do is to unselect "Database Instance" feature in the following dialog . Technically, it duplicates functionality of the feature selection tree. So what... I hope it adds some clarity for the user - at least tells the user that his existing instances *can* be upgraded. Or maybe it won't help that much, since user never looks at the UI anyway :) (nice article about that is here http://www.codinghorror.com/blog/2009/10/treating-user-myopia.html ) Anyway, I uploaded the newest msis to the usual place, if somebody wants to play with it. Thanks, Vladislav.

2 4

[Maria-developers] MariaDB 5.6
by Peter Laursen 12 Apr '11

12 Apr '11

I noticed this on planet.mysql: http://kb.askmonty.org/v/plans-for-56 I *again* strongly want to discourage a major version number identical with a MySQL/Oracle release. MySQL plans a 5.6 too and I believe that there is already a source-tree available on launchpad. I think I understand that MariaDB 5.6 is planned to use MySQL 5.5 codebase. Am I correct? The problem arises with *generic clients* (ie clients that are supposed to work with not only one specific server version but a broad range of server versions) - including all GUI clients. Such clients will execute a "SELECT VERSION()" after connection to decide what features in the client should be enabled or not (necessary as "SHOW VARIABLES LIKE 'have_***"" and "SELECT ... FROM I_S" cannot always be used due to incomplete instrumentation with those) I have now posted this complaint 2 or 3 times (including the times I complained about the use of 5.2 for a mariaDB based on MySQL 5.1 code as there is alose a (now abandoned) MySQL 5.2 tree. I never had a reply from a MariaDB 'decisionmaker'. Could I at least request a reply this time? Please! :-) I repeat: It is really causing problems for *generic clients* if there are too different 5.6's with different feature sets. What solution would you suggest for a client that shall be able to connect to both MySQL5.6 and MariaDB 5.6 without confusing the features of one server with the other? Peter Webyog

3 5

[Maria-developers] PBXT test failures after your patch for lp:732124
by Kristian Nielsen 12 Apr '11

12 Apr '11

Hi Monty, After you pushed the patch for lp:732124, test failures appeared in Buildbot for test case pbxt.union. Most were just that the pbxt union.result file needed same update as for main.union. But there was one more issue: create table t1 (a int); insert into t1 values (1),(2),(3); create table t2 (a int); insert into t2 values (3),(4),(5); # ... SELECT SQL_CALC_FOUND_ROWS * FROM t1 UNION SELECT * FROM t2 LIMIT 2; select found_rows(); In pbxt.union, this now (after your patch) return 5. However, the result file for main.union has a return value of 6, and the test succeeds with MyISAM. The thing is ... the value that 5 PBXT returns is correct, and what MyISAM does and what is in the result file is wrong. At least according to the docs. The full query would return 5 rows (as it is union distinct), not 6... What I will do is that I will fix the PBXT union.result to have the correct value "5" to get the test suite to pass in Buildbot. However, there seems to be some deeper problem here, given that MyISAM does something different (and wrong?). There are also other cases that are wrong: SELECT SQL_CALC_FOUND_ROWS * FROM t1 LIMIT 1 UNION SELECT * FROM t2; a 1 3 4 5 select found_rows(); found_rows() 6 SELECT SQL_CALC_FOUND_ROWS * FROM t1 LIMIT 1 UNION SELECT * FROM t2 LIMIT 2; a 1 3 select found_rows(); found_rows() 6 (the result should have been 5) So maybe there are still bugs in the code? Or if this is a limitation of SQL_CALC_FOUND_ROWS with UNION (there is a comment in the test case "# The following examples will not be exact" that might hint at this), then we should document this at least ... - Kristian.

1 0

[Maria-developers] WL#196 New (by Mdcallag): add is_connection_alive for server-side checks to find disconnected clients
by worklog-noreply＠askmonty.org 08 Apr '11

08 Apr '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: add is_connection_alive for server-side checks to find disconnected clients CREATION DATE..: Fri, 08 Apr 2011, 14:06 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 196 (http://askmonty.org/worklog/?tid=196) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: There are a few places in the server where the thread for a user connection will wait. With upcoming features in the FB patch there will be more places. I don't want a thread to continue waiting after the client has disconnected. Is it possible to check whether the client has disconnected without reading/writing more than 0 bytes from the socket? Assuming the check is is_connection_alive(), it can return TRUE when the client has disconnected (but not all of the time). But it must not return FALSE when the client has not disconnected. The check does not have to be cheap. It can require a system call as it won't be made frequently. For example, assume that a row lock wait is implemented as a sequent of timed pthread condition variable waits (1 second each). At the end of each 1 second timeout there can be a check for whether the connection was killed and a check for whether the client has disconnected. Disconnected clients on the server-side can be a serious problem for high throughput servers. The worst case is when the client starts a long-running and blocking query (external sort) that won't return a row for a long time and thus won't detect the disconnected client for a long time. I have used tool in the past that join output from netstat with SHOW PROCESSLIST to find sockets for disconnected clients. This works but is very expensive. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#195 New (by Mdcallag): SHOW QUERY STATUS
by worklog-noreply＠askmonty.org 07 Apr '11

07 Apr '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: SHOW QUERY STATUS CREATION DATE..: Thu, 07 Apr 2011, 13:54 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 195 (http://askmonty.org/worklog/?tid=195) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: I want some performance counters to be maintained per-query in addition to per- session. This could then be used to implement SHOW QUERY STATUS. It could also be used to augment slow query log output with additional performance data per statement. The Facebook patch and probably Percona already put more data per statement in the slow query log. With the Facebook patch the option --log-slow- extra enables this. Unfortunately, that uses the session counters and can be confusing. Some details on the use of this in the facebook patch are at: https://www.facebook.com/note.php?note_id=390420710932 ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Summary of the storage engine API changes for group commit
by Kristian Nielsen 07 Apr '11

07 Apr '11

Hi Serg, As promised, here is a summary of the changes to the storage engine API that I made as part of group commit. Two new handlerton methods are added: void (*prepare_ordered)(handlerton *hton, THD *thd, bool all); void (*commit_ordered)(handlerton *hton, THD *thd, bool all); commit_ordered() will be called just before commit(). It should commit the transaction in memory; calls to commit_ordered() will be serialised, so order of commit is determined by order of calls to commit_ordered(). MVCC engines should make the transaction visible. No time-consuming operations like flushing logs to disk should be done; this should happen in commit(). prepare_ordered() will be called just after prepare(). Calls will again be serialised and happen in commit order, so an engine can do here any part of the prepare step that depends on commit order. Again no time-consuming operations like I/O should be done. Order of calls to the new methods is guaranteed to be consistent between engines and with commit order in the binlog. Locking is done so that START TRANSACTION WITH CONSISTENT SNAPSHOT becomes consistent between storage engines. The methods are optional; a storage engine is free to not set them. Group commit will still work, but there will be no guarantee that commits happen in the same order in engine and in binlog, nor that START TRANSACTION WITH CONSISTENT SNAPSHOT will be consistent between engines/binlog. (The prepare_commit_mutex in built-in InnoDB will still ensure consistent commit order, and will still destroy group commit). There are no other changes to the storage engine API by me. Hope this helps, - Kristian.

3 4

[Maria-developers] Default parameter settings / Windows MSI installer
by Vladislav Vaintroub 07 Apr '11

07 Apr '11

Hello, After recent discussion about appropriate or not appropriate default setting on in my.cnf , I have created a prototype minimal settings for Windows installer. Here is the picture of this attempt http://cid-ff0c950417b4f8a4.office.live.com/self.aspx/share/dbsettings.png For those who have familiar the 5.2 MSI installer already, what I have added is a checkbox "Standard configuration", explanation text, and input field for innodb bufferpool size (preset with RAM/8, in my case it is 8191/8=1023). "Standard configuration" is on by default, if it is "on", then 4 parameters are added to the my.ini file 1) default_storage_engine=innodb 2) innodb_buffer_pool_size=[VALUE_IN_EDIT_BOX]M 3) innodb_log_file_size= min(innodb_buffer_pool_size/4, 50) I made this formula almost out of thin air and using some blog reading and comments in the templates. It is likely that 50M is more than the average user needs, however it only takes some disk space, and If user has to increase from default 5MB , the procedure is awkward. I checked, MySQL Config Wizard set this parameter to 54M on my box, so it is in the same range. 4) sql_mode=STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION If "Standard configuration" is off, hen input field for bufferpool size is disabled, and none of those 4 parameters is added to my.ini Do you think it is useful in this form? So far I tried to avoid "calibration" topics altogether since it is a controversial topic where people never have same opinions. Also I tried to avoid it because I cannot write GUIs . However, this one change is small in and it does not introduce additional dialogs. It makes some important defaults the same as "standard configuration" in existing MySQL installers. I think it was PeterL who already asked for non-default buffer pool size already, also Monty also spoke about it in Lisbon. Comments welcome, also those on the picture above - I'm not sure if wording and use of brackets and quotes is correct. Wlad

5 8

Re: [Maria-developers] dist-hook in top-level Makefile.am
by Kristian Nielsen 06 Apr '11

06 Apr '11

"Vladislav Vaintroub" <wlad(a)montyprogram.com> writes: > Kristian, now that Windows builds in 5.2+ are self-sufficient and can > produce different packages from bzr, do you see any reason to keep this hook > in the top-level Makefile.am? > > If not, I can remove, or you can remove it if you prefer. Will make our > source distribution smaller/cleaner , if nothing else. > > I'm talking about this place in Makefile.am > > # Remove BK's "SCCS" subdirectories from source distribution > # Create initial database files for Windows installations and check them. > dist-hook: > mkdir -p $(distdir)/win > scripts/mysql_install_db --no-defaults --cross-bootstrap \ > --builddir=$(top_builddir) \ > --datadir=$(distdir)/win/data \ > --srcdir=$(top_srcdir) > storage/myisam/myisamchk --silent --fast > $(distdir)/win/data/mysql/*.MYI I do not have time to look deeply into this at the moment, but I did not want you to have to wait any longer for an answer, sorry for taking so long already. It is very good news that Windows builds are now self-sufficient! This need to compile the whole server just to produce a source tarball has been nagging me forever, and I suppose this has also been an annoyance on the Windows side when installing from source. You need to be careful when you remove it. Update any documentation that refers to the old way of installing from source on Windows, making sure existing stuff still works, etc. Well, I guess you are already aware of that. It would be nice to also fix the autotools builds so that `make dist` to produce a source tarball no longer needs to compile the whole server. But then, as we are phasing out autotools, it is probably not worth it. So I suppose just removing the dist-hook is a reasonable step if you make sure it does not break anything. I guess the CMake system already has this done properly, so that a source tarball is produced without needing to build the server (and without generating these obsolete database files for Windows)? - Kristian.

2 1

Re: [Maria-developers] [Commits] Rev 2864: MWL#136: Cross-engine consistency for START TRANSACTION WITH CONSISTENT SNAPSHOT in http://bazaar.launchpad.net/~maria-captains/maria/5.1
by Michael Widenius 05 Apr '11

05 Apr '11

Hi! >>>>> "knielsen" == knielsen <knielsen(a)knielsen-hq.org> writes: knielsen> At http://bazaar.launchpad.net/~maria-captains/maria/5.1 knielsen> ------------------------------------------------------------ knielsen> revno: 2864 knielsen> revision-id: knielsen(a)knielsen-hq.org-20101107213743-3luszsivft2vt7t6 knielsen> parent: knielsen(a)knielsen-hq.org-20101103155438-vdou0fngj6hpgpsd knielsen> committer: knielsen(a)knielsen-hq.org knielsen> branch nick: work-5.1-mwl136 knielsen> timestamp: Sun 2010-11-07 22:37:43 +0100 knielsen> message: knielsen> MWL#136: Cross-engine consistency for START TRANSACTION WITH CONSISTENT SNAPSHOT <cut> knielsen> === modified file 'client/mysqldump.c' <cut> knielsen> -static int do_show_master_status(MYSQL *mysql_con) knielsen> +static int do_show_master_status(MYSQL *mysql_con, int consistent_binlog_pos) knielsen> { knielsen> MYSQL_ROW row; knielsen> MYSQL_RES *master; knielsen> + char binlog_pos_file[FN_REFLEN]; knielsen> + char binlog_pos_offset[LONGLONG_LEN+1]; knielsen> + char *file, *offset; knielsen> const char *comment_prefix= knielsen> (opt_master_data == MYSQL_OPT_MASTER_DATA_COMMENTED_SQL) ? "-- " : ""; knielsen> - if (mysql_query_with_error_report(mysql_con, &master, "SHOW MASTER STATUS")) knielsen> + knielsen> + if (consistent_binlog_pos) knielsen> { knielsen> - return 1; knielsen> + if(!check_consistent_binlog_pos(binlog_pos_file, binlog_pos_offset)) knielsen> + return 1; knielsen> + file= binlog_pos_file; knielsen> + offset= binlog_pos_offset; knielsen> } knielsen> else knielsen> { knielsen> + if (mysql_query_with_error_report(mysql_con, &master, "SHOW MASTER STATUS")) knielsen> + return 1; knielsen> + knielsen> row= mysql_fetch_row(master); knielsen> if (row && row[0] && row[1]) knielsen> { knielsen> + file= row[0]; knielsen> + offset= row[1]; knielsen> } knielsen> + else knielsen> { You need a mysql_free_result(master) here. knielsen> + if (!ignore_errors) knielsen> + { knielsen> + /* SHOW MASTER STATUS reports nothing and --force is not enabled */ knielsen> + my_printf_error(0, "Error: Binlogging on server not active", knielsen> + MYF(0)); knielsen> + maybe_exit(EX_MYSQLERR); knielsen> + return 1; knielsen> + } knielsen> + else knielsen> + { knielsen> + return 0; knielsen> + } knielsen> } knielsen> } <cut> knielsen> === modified file 'sql/log.cc' knielsen> --- a/sql/log.cc 2010-11-02 07:40:27 +0000 knielsen> +++ b/sql/log.cc 2010-11-07 21:37:43 +0000 knielsen> @@ -62,6 +62,7 @@ static int binlog_savepoint_rollback(han knielsen> static int binlog_commit(handlerton *hton, THD *thd, bool all); knielsen> static int binlog_rollback(handlerton *hton, THD *thd, bool all); knielsen> static int binlog_prepare(handlerton *hton, THD *thd, bool all); knielsen> +static int binlog_start_consistent_snapshot(handlerton *hton, THD *thd); knielsen> /** knielsen> Silence all errors and warnings reported when performing a write knielsen> @@ -155,9 +156,10 @@ class binlog_trx_data { knielsen> public: knielsen> binlog_trx_data() knielsen> : at_least_one_stmt_committed(0), incident(FALSE), m_pending(0), knielsen> - before_stmt_pos(MY_OFF_T_UNDEF), commit_bin_log_file_pos(0), using_xa(0) knielsen> + before_stmt_pos(MY_OFF_T_UNDEF), last_commit_pos_offset(0), using_xa(0) knielsen> { knielsen> trans_log.end_of_file= max_binlog_cache_size; knielsen> + strcpy(last_commit_pos_file, ""); strcpy() -> strmov() or even better last_commit_pos_file[0]= 0; (I want to get rid of all strcpy() and replace with strmov() as strcpy() is often used wrongly; I would like to have this thing changed to not get 'false positives' when searching for strcpy() to remove. knielsen> } knielsen> ~binlog_trx_data() knielsen> @@ -215,7 +217,8 @@ public: knielsen> incident= FALSE; knielsen> trans_log.end_of_file= max_binlog_cache_size; knielsen> using_xa= FALSE; knielsen> - commit_bin_log_file_pos= 0; knielsen> + strcpy(last_commit_pos_file, ""); strcpy -> strmov or last_commit_pos_file[0]= 0; knielsen> + last_commit_pos_offset= 0; knielsen> DBUG_ASSERT(empty()); knielsen> } knielsen> /* knielsen> Write a table map to the binary log. knielsen> @@ -4337,6 +4369,9 @@ MYSQL_BIN_LOG::flush_and_set_pending_row knielsen> rotate_and_purge(RP_LOCK_LOG_IS_ALREADY_LOCKED); knielsen> } Add a comment that this mutex is a protection for partial reads of last_commit_pos_offset on 32 bit systems. (at least I can't come up with any better reason for this mutex). Maybe we should introduce pthread_mutex_lock_for_32_bit() that would be void on 64 bit systems in cases like this. knielsen> + pthread_mutex_lock(&LOCK_commit_ordered); knielsen> + last_commit_pos_offset= my_b_tell(&log_file); knielsen> + pthread_mutex_unlock(&LOCK_commit_ordered); knielsen> pthread_mutex_unlock(&LOCK_log); knielsen> } knielsen> @@ -4526,7 +4561,12 @@ bool MYSQL_BIN_LOG::write(Log_event *eve knielsen> err_unlock: knielsen> if (file == &log_file) knielsen> + { knielsen> + pthread_mutex_lock(&LOCK_commit_ordered); knielsen> + last_commit_pos_offset= my_b_tell(&log_file); knielsen> + pthread_mutex_unlock(&LOCK_commit_ordered); knielsen> pthread_mutex_unlock(&LOCK_log); knielsen> + } Same comment needed for this. knielsen> err: knielsen> if (error) knielsen> @@ -4827,6 +4867,9 @@ bool MYSQL_BIN_LOG::write_incident(THD * knielsen> signal_update(); knielsen> rotate_and_purge(RP_LOCK_LOG_IS_ALREADY_LOCKED); knielsen> } knielsen> + pthread_mutex_lock(&LOCK_commit_ordered); knielsen> + last_commit_pos_offset= my_b_tell(&log_file); knielsen> + pthread_mutex_unlock(&LOCK_commit_ordered); knielsen> pthread_mutex_unlock(&LOCK_log); and here... knielsen> static ulonglong binlog_status_var_num_commits; knielsen> static ulonglong binlog_status_var_num_group_commits; knielsen> +static char binlog_trx_file[FN_REFLEN]; knielsen> +static ulonglong binlog_trx_position; knielsen> static SHOW_VAR binlog_status_vars_detail[]= knielsen> { knielsen> @@ -6562,12 +6609,16 @@ static SHOW_VAR binlog_status_vars_detai knielsen> (char *)&binlog_status_var_num_commits, SHOW_LONGLONG}, knielsen> {"group_commits", knielsen> (char *)&binlog_status_var_num_group_commits, SHOW_LONGLONG}, knielsen> + {"trx_file", knielsen> + (char *)&binlog_trx_file, SHOW_CHAR}, knielsen> + {"trx_position", knielsen> + (char *)&binlog_trx_position, SHOW_LONGLONG}, knielsen> {NullS, NullS, SHOW_LONG} knielsen> }; knielsen> static int show_binlog_vars(THD *thd, SHOW_VAR *var, char *buff) knielsen> { knielsen> - mysql_bin_log.set_status_variables(); knielsen> + mysql_bin_log.set_status_variables(thd); knielsen> var->type= SHOW_ARRAY; knielsen> var->value= (char *)&binlog_status_vars_detail; knielsen> return 0; knielsen> @@ -6606,17 +6657,31 @@ static struct st_mysql_sys_var *binlog_s knielsen> This is called only under LOCK_status, so we can fill in a static array. knielsen> */ knielsen> void knielsen> -TC_LOG_BINLOG::set_status_variables() knielsen> +TC_LOG_BINLOG::set_status_variables(THD *thd) knielsen> { knielsen> - ulonglong num_commits, num_group_commits; knielsen> + binlog_trx_data *trx_data; knielsen> + knielsen> + if (thd) knielsen> + trx_data= (binlog_trx_data*) thd_get_ha_data(thd, binlog_hton); knielsen> + else knielsen> + trx_data= NULL; knielsen> pthread_mutex_lock(&LOCK_commit_ordered); knielsen> - num_commits= this->num_commits; knielsen> - num_group_commits= this->num_group_commits; knielsen> + binlog_status_var_num_commits= this->num_commits; knielsen> + binlog_status_var_num_group_commits= this->num_group_commits; knielsen> + if (!trx_data || 0 == strcmp(trx_data->last_commit_pos_file, "")) -> if (!trx_data || trx_data->last_commit_pos_file[0] == 0) knielsen> + { knielsen> + strmake(binlog_trx_file, last_commit_pos_file, sizeof(binlog_trx_file)-1); knielsen> + binlog_trx_position= last_commit_pos_offset; knielsen> + } knielsen> pthread_mutex_unlock(&LOCK_commit_ordered); knielsen> + if (trx_data && 0 != strcmp(trx_data->last_commit_pos_file, "")) -> if (trx_data && trx_data->last_commit_pos_file[0] != 0) It was a bit tricky to see that this was a negative test for the earlier one. I would sugges to change first to: -> if ((binlog_pos_set= (!trx_data || trx_data->last_commit_pos_file[0] == 0)) and the later to if (!binlog_pos_set) knielsen> + { knielsen> + strmake(binlog_trx_file, trx_data->last_commit_pos_file, knielsen> + sizeof(binlog_trx_file)-1); knielsen> + binlog_trx_position= trx_data->last_commit_pos_offset; knielsen> + } knielsen> } <cut> Regards, Monty

4 4

[Maria-developers] WL#194 New (by Sergei): remove the difference between standalone and bundled plugins
by worklog-noreply＠askmonty.org 04 Apr '11

04 Apr '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: remove the difference between standalone and bundled plugins CREATION DATE..: Mon, 04 Apr 2011, 13:29 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: Sergei CATEGORY.......: Server-RawIdeaBin TASK ID........: 194 (http://askmonty.org/worklog/?tid=194) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 20 (hours remain) ORIG. ESTIMATE.: 20 PROGRESS NOTES: DESCRIPTION: Now, standalone plugins needs to be distributed witha different set of files, as compared to bundled plugins. For example, standalone plugins that use autotools need NEWS, AUTHORS, README, configure.ac, Makefile.am, ChangeLog. Bundled plugins need Makefile.am and plug.in - or CMakeLists.txt for 5.5. It means that standalone plugin cannot be built from within the tree, and bundled plugin - outside of it. It would be beneficial for plugin authors if we remove this difference. Speaking of 5.5 only, it means providing cmake include file (?) that would allow to use MYSQL_ADD_PLUGIN() macro from outside the source tree. Such a cmake file would be installed with the mysql headers and libraries. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#193 New (by Sergei): Logging service
by worklog-noreply＠askmonty.org 04 Apr '11

04 Apr '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Logging service CREATION DATE..: Mon, 04 Apr 2011, 12:47 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: Sergei CATEGORY.......: Server-RawIdeaBin TASK ID........: 193 (https://askmonty.org/worklog/?tid=193) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: logging service for plugins. destinations: * syslog * file. can be automatically rotated. * stderr, stdout (or FILE? or handle?) * table * event log on windows rewrite general, error, slow logs to use that. not all destinations and not all features need to be implemented in the first version. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] MWL#116: Efficient group commit, ready for code review
by Kristian Nielsen 04 Apr '11

04 Apr '11

Hi Serg, Please find attached the patch for code review of MWL#116: Efficient group commit. I went through all your comments in the architecture review and updated the worklog description and the code accordingly. I hope I got everything, otherwise I hope you have the patience to point out any omissions once again! The patch is pushed here if you want to see the complete code: lp:~maria-captains/maria/mariadb-5.1-mwl116 Once the code passes code review, I think the plan is to include this feature in MariaDB 5.3. Please let me know if you need anything else from me for the review. - Kristian.

2 3

[Maria-developers] WL#192 New (by Mdcallag): optional nonblocking client API
by worklog-noreply＠askmonty.org 30 Mar '11

30 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: optional nonblocking client API CREATION DATE..: Wed, 30 Mar 2011, 22:13 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-RawIdeaBin TASK ID........: 192 (http://askmonty.org/worklog/?tid=192) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: We want some of the calls in libmysqlclient to optionally be non-blocking so it can be used with libevent as described by http://jan.kneschke.de/2008/9/9/async- mysql-queries-with-c-api. We prefer to extend libmysqlclient rather than replace it. Is this possible? ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] MWL#89 patch for review
by Timour Katchaounov 30 Mar '11

30 Mar '11

Monty, Please find attached an updated patch for MWL#89. The patch is merged with the most current 5.3 as of today, March 30. The tree itself is here: https://code.launchpad.net/~maria-captains/maria/5.3-mwl89 Timour

1 0

Re: [Maria-developers] [Commits] Rev 2947: Fix LP BUG#613029 in file:///home/tsk/mprog/src/5.3/
by Sergey Petrunya 29 Mar '11

29 Mar '11

Hi Timour, Ok to push. On Mon, Mar 28, 2011 at 12:55:43PM +0300, timour(a)askmonty.org wrote: > At file:///home/tsk/mprog/src/5.3/ > > ------------------------------------------------------------ > revno: 2947 > revision-id: timour(a)askmonty.org-20110328095536-wbmu1hiwsnhw6bs8 > parent: timour(a)askmonty.org-20110324143406-04q5peh1r7nthcyb > committer: timour(a)askmonty.org > branch nick: 5.3 > timestamp: Mon 2011-03-28 12:55:36 +0300 > message: > Fix LP BUG#613029 > > Analysis: > There are two code paths through which JOIN::exec may produce > an all-NULL row for an empty result set. One goes via the > function return_zero_rows(), when query processing detectes > early that the where clause is false, the other one is via > do_select() in the case of join execution. > > In the case of do_select(), the problem was that the executioner > didn't set TABLE::null_row to 1. As result when sending the only > result row, the evaluation of each field didn't detect that all > non-aggregated fields are NULL, because Field::is_null returned > true, after checking that field->table->null_row was false. > > Given that the each non-aggregated field was not considered NULL, > select_result::send_data sent whatever was in the buffer of each > field. However, since there was no actual data in the field buffer, > send_data() accessed and sent whatever junk was in the field's > data buffer. > > Solution: > Similar to the analogous case in return_zero_rows() mark all > tables that their current row is NULL before sending the > artificailly created NULL row. > === modified file 'mysql-test/r/subselect4.result' > --- a/mysql-test/r/subselect4.result 2011-03-24 14:34:06 +0000 > +++ b/mysql-test/r/subselect4.result 2011-03-28 09:55:36 +0000 > @@ -1133,3 +1133,55 @@ SELECT * FROM t1 WHERE (2, 0) NOT IN (SE > f1 f2 > set @@optimizer_switch=@save_optimizer_switch; > drop table t1,t2; > +# > +# LP BUG#613029 Wrong result with materialization and semijoin, and > +# valgrind warnings in Protocol::net_store_data with materialization > +# for implicit grouping > +# > +CREATE TABLE t1 ( > +pk int(11) NOT NULL AUTO_INCREMENT, > +f2 int(11) NOT NULL, > +f3 varchar(1) NOT NULL, > +PRIMARY KEY (pk), > +KEY f2 (f2)); > +INSERT INTO t1 VALUES (1,9,'x'); > +INSERT INTO t1 VALUES (2,5,'g'); > +CREATE TABLE t2 ( > +pk int(11) NOT NULL AUTO_INCREMENT, > +f2 int(11) NOT NULL, > +f3 varchar(1) NOT NULL, > +PRIMARY KEY (pk), > +KEY f2 (f2)); > +INSERT INTO t2 VALUES (1,7,'p'); > +set @save_optimizer_switch=@@optimizer_switch; > +set @@optimizer_switch='materialization=off,semijoin=off'; > +EXPLAIN > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables > +2 DEPENDENT SUBQUERY t1 index_subquery f2 f2 4 func 2 Using index > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > +f3 MAX(t1.f2) > +NULL NULL > +set @@optimizer_switch='materialization=on,semijoin=off'; > +EXPLAIN > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY t2 system PRIMARY NULL NULL NULL 1 > +1 PRIMARY t1 const PRIMARY PRIMARY 4 const 1 > +2 SUBQUERY t1 index NULL f2 4 NULL 2 Using index > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > +f3 MAX(t1.f2) > +NULL NULL > +TODO: add a test case for semijoin when the wrong result is fixed > +set @@optimizer_switch='materialization=off,semijoin=on'; > +set @@optimizer_switch=@save_optimizer_switch; > +drop table t1, t2; > > === modified file 'mysql-test/t/subselect4.test' > --- a/mysql-test/t/subselect4.test 2011-03-24 14:34:06 +0000 > +++ b/mysql-test/t/subselect4.test 2011-03-28 09:55:36 +0000 > @@ -906,3 +906,60 @@ SELECT * FROM t1 WHERE (2, 0) NOT IN (SE > set @@optimizer_switch=@save_optimizer_switch; > > drop table t1,t2; > + > +--echo # > +--echo # LP BUG#613029 Wrong result with materialization and semijoin, and > +--echo # valgrind warnings in Protocol::net_store_data with materialization > +--echo # for implicit grouping > +--echo # > + > +CREATE TABLE t1 ( > + pk int(11) NOT NULL AUTO_INCREMENT, > + f2 int(11) NOT NULL, > + f3 varchar(1) NOT NULL, > + PRIMARY KEY (pk), > + KEY f2 (f2)); > + > +INSERT INTO t1 VALUES (1,9,'x'); > +INSERT INTO t1 VALUES (2,5,'g'); > + > +CREATE TABLE t2 ( > + pk int(11) NOT NULL AUTO_INCREMENT, > + f2 int(11) NOT NULL, > + f3 varchar(1) NOT NULL, > + PRIMARY KEY (pk), > + KEY f2 (f2)); > + > +INSERT INTO t2 VALUES (1,7,'p'); > + > +set @save_optimizer_switch=@@optimizer_switch; > + > +set @@optimizer_switch='materialization=off,semijoin=off'; > + > +EXPLAIN > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > + > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > + > +set @@optimizer_switch='materialization=on,semijoin=off'; > + > +EXPLAIN > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > + > +SELECT t1.f3, MAX(t1.f2) > +FROM t1, t2 > +WHERE (t2.pk = t1.pk) AND t2.pk IN (SELECT f2 FROM t1); > + > +-- echo TODO: add a test case for semijoin when the wrong result is fixed > +-- echo set @@optimizer_switch='materialization=off,semijoin=on'; > + > + > +set @@optimizer_switch=@save_optimizer_switch; > + > +drop table t1, t2; > > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2011-03-24 14:34:06 +0000 > +++ b/sql/sql_select.cc 2011-03-28 09:55:36 +0000 > @@ -13452,6 +13452,13 @@ do_select(JOIN *join,List<Item> *fields, > { > List<Item> *columns_list= (procedure ? &join->procedure_fields_list : > fields); > + /* > + With implicit grouping all fields of special row produced for an > + empty result are NULL. See return_zero_rows() for the same behavior. > + */ > + for (TABLE_LIST *table= join->select_lex->leaf_tables; > + table; table= table->next_leaf) > + mark_as_null_row(table->table); > rc= join->result->send_data(*columns_list); > } > } > > === modified file 'sql/sql_select.h' > --- a/sql/sql_select.h 2011-03-13 10:50:14 +0000 > +++ b/sql/sql_select.h 2011-03-28 09:55:36 +0000 > @@ -954,8 +954,8 @@ public: > bool init_save_join_tab(); > bool send_row_on_empty_set() > { > - return (do_send_rows && tmp_table_param.sum_func_count != 0 && > - !group_list && having_value != Item::COND_FALSE); > + return (do_send_rows && implicit_grouping && > + having_value != Item::COND_FALSE); > } > bool change_result(select_result *result); > bool is_top_level_join() const > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] WL#191 New (by Knielsen): status variables for last InnoDB binlog position
by worklog-noreply＠askmonty.org 29 Mar '11

29 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: status variables for last InnoDB binlog position CREATION DATE..: Tue, 29 Mar 2011, 09:06 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 191 (http://askmonty.org/worklog/?tid=191) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: In every commit, InnoDB/XtraDB stores the binlog position in the system tablespace (in a transactional-safe way). At server start, this position is printed to the error log. This is useful after taking a backup to provision a slave, as it provides the values to use for CHANGE MASTER TO. It would be useful if these values printed to the error log were also directly available to SQL as read-only status variables: innodb_startup_binlog_master_file innodb_startup_binlog_master_pos This would be generally useful. When this is implemented, the test case main.group_commit_binlog_pos can be fixed to not require `grep` and `tail`, and so work on Windows with no posix tools installed. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Improved table elimination
by Lars Rönnbäck 29 Mar '11

29 Mar '11

When doing some comparisons of query optimization in Microsoft SQL Server and MariaDB I noticed that there is a second type of table elimination which MariaDB has not yet implemented. It depends on foreign keys being declared, and can under some circumstances substantially improve performance. Rather than to write a lengthy discussion, here is a test script for MariaDB 5.2.5 in which the queries with the condition "where ... is not null" should result in this type of table elimination: http://pastebin.com/s68RwgG2 This screencast shows the table elimination in effect in MSSQL: http://www.screenr.com/tPG We rely on table elimination to get better performance in 3NF-views and functions used in our Anchor Modeling (6NF-table) technique. However, the need for table elimination has previously come from denormalized modeling, where it is used to speed up queries against star schemas (e g eliminating dimension tables). It should therefore not only be appreciated by our crowd, if implemented in full. We were very happy to see the 'outer join table elimination' implemented and are wondering if someone could put in the effort to add the 'foreign key table elimination'? Since everything we do with Anchor Modeling is Open Source, it would be nice to have an Open Source database that fully supports the technique. Support in other databases is described here, if anyone is interested: http://www.anchormodeling.com/?page_id=22 Thanks, Lars

1 0

[Maria-developers] WL#190 New (by Yakatz): Optional parameters support in stored procedures
by worklog-noreply＠askmonty.org 28 Mar '11

28 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Optional parameters support in stored procedures CREATION DATE..: Mon, 28 Mar 2011, 20:26 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 190 (http://askmonty.org/worklog/?tid=190) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: This bug has been open on MySQL since 2005. In a nutshell, support PROCEDUREs like this: CREATE PROCEDURE ListUsers ( @userID int, @authorized bit = NULL ) AS.... ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Review of replication/group commit work
by Kristian Nielsen 28 Mar '11

28 Mar '11

Hi Monty, Thanks for offering to review my replication worklogs. Here are pointers to the various worklogs and patches, and suggestions for order to things in etc. Some background information is in this series on my blog: http://kristiannielsen.livejournal.com/12254.html http://kristiannielsen.livejournal.com/12408.html http://kristiannielsen.livejournal.com/12553.html http://kristiannielsen.livejournal.com/12810.html There is a tree on launchpad based on 5.2 with all the patches merged, if you want to look at the actual code in one piece: lp:~maria-captains/maria/mariadb-5.2-rpl ---- The main worklog, and the place to start, is MWL#116: http://askmonty.org/worklog/Server-Sprint/?tid=116 I think you'll have to read through the worklog/design first (it already had architecture review by Serg) in order to properly review the code. The patch for MWL#116 is here: https://lists.launchpad.net/maria-developers/msg03693.html It also has links to a lauchpad tree with just this patch. This worklog extends the storage engine API with a facility to efficiently coordinate commit order among transaction participants, and also implements group commit for binlog. ---- The next worklog is MWL#136: http://askmonty.org/worklog/Server-RawIdeaBin/?tid=136 Here is the commit mail: https://lists.askmonty.org/pipermail/commits/2010-November/000735.html (also this one, but it just renames some variables to bettter names): https://lists.askmonty.org/pipermail/commits/2010-November/000741.html This worklog makes START TRANSACTION WITH CONSISTENT SNAPSHOT, well, consistent. It also uses this for the binlog to provide non-blocking mysqldump --single-transaction --master-data. ---- The last worklog is MWL#163: http://askmonty.org/worklog/Server-Sprint/?tid=163 Here is the commit mail: https://lists.askmonty.org/pipermail/commits/2010-November/000711.html This is a re-write of a Facebook patch. Mark has mentioned this on several occasions as being important for them to fix a hot-spot. It implements an option to release InnoDB row locks earlier, before committing to binlog and storage engine (which is not safe in general, but can be safe in certain applications such as Facebook presumably). ---- That's it for now! - Kristian.

2 1

[Maria-developers] review: [Commits] Rev 2863: MWL#163: release of row locks in InnoDB during prepare in http://bazaar.launchpad.net/~maria-captains/maria/5.1
by Michael Widenius 28 Mar '11

28 Mar '11

Hi! >>>>> "knielsen" == knielsen <knielsen(a)knielsen-hq.org> writes: knielsen> At http://bazaar.launchpad.net/~maria-captains/maria/5.1 knielsen> ------------------------------------------------------------ knielsen> revno: 2863 knielsen> revision-id: knielsen(a)knielsen-hq.org-20101103160708-88w8p8ibe9iaoggj knielsen> parent: knielsen(a)knielsen-hq.org-20101102074027-jlnx0rnbdpovl122 knielsen> committer: knielsen(a)knielsen-hq.org knielsen> branch nick: work-5.1-mwl163 knielsen> timestamp: Wed 2010-11-03 17:07:08 +0100 knielsen> message: knielsen> MWL#163: release of row locks in InnoDB during prepare knielsen> Port the Facebook patch for releasing InnoDB row locks early to the knielsen> MWL#116 framework. knielsen> === modified file 'storage/xtradb/handler/ha_innodb.cc' knielsen> --- a/storage/xtradb/handler/ha_innodb.cc 2010-11-01 14:41:09 +0000 <cut> Spent 30 minutes checking the innodb code and things looked ok. Regards, Monty

1 0

[Maria-developers] Writeup of how to run my group commit benchmark
by Kristian Nielsen 28 Mar '11

28 Mar '11

Hi, For anyone interested in trying to redo my group commit benchmarks on more interesting hardware than what I have available, here is a writeup of the steps needed to run it. The benchmarks are described here: http://kristiannielsen.livejournal.com/15739.html http://kristiannielsen.livejournal.com/15515.html 1. Compile gypsy. This is used to generate the load. I use a specific version as the latest in trunk seemed to not build on non-windows last I tried: bzr branch '-rrevid:shane@mysql.com-20100224065101-jk3s5sro2p914nvp' lp:gypsy cd gypsy make This requires `mysql_config` in the path to get includes/libraries for libmysqlclient. It also needs openssl headers/libraries (eg. install libssl-dev package). 2. Install MariaDB with group commit enabled. Packages/sources are available for this here: http://kb.askmonty.org/v/mariadb-52-replication-feature-preview These packages are however missing the --binlog-optimize-thread-scheduling option that I used to compare the two ways to schedule threads in the above blog posts. To get that option, grab this bzr tree and build: bzr branch lp:~knielsen/maria/mariadb-5.1-mwl116-threadscheduling cd mariadb-5.1-mwl116-threadscheduling BUILD/compile-pentium64-max 3. Configure and start mysqld This involves `make install` and `mysql_install_db` when building from source, check README/INSTALL for details. For configuration, set whatever parameters are interesting to test. Eg. set InnoDB buffer pool size and log file/buffer size to something sensible, set innodb_flush_log_at_trx_commit=1 and sync_binlog=1. Also create a mysql account and a database with suitable access permissions. 4. Create the table to use: mysql --host=127.0.0.1 -utest -ptestpass test MariaDB [test]> create table t (a int primary key, b int) engine=innodb; 5. Create the script for Gypsy to generate the load. Just create a file simpleload.gypsy containing this single line: ----------------------------------------------------------------------- p|1|REPLACE INTO t (a,b) VALUES (? MOD 1000000, ? MOD 1000000)|int,int ----------------------------------------------------------------------- 6. Run the bemchmark. I run it with `script` to capture the output (yeah, I know it's crude). The `gypsy` binary is found in the directory where it was compiled in step (1). script for i in 1 4 8 12 16 24 32 48 64 96 128 ; do gypsy --host=127.0.0.1 --user=test --password=testpass --database=test --queryfile=simpleload.gypsy --duration=180 --threads=$i ; done Adjust host/user/password as appropriate for the installation of mariadb. 7. That should be it! This is obviously a rather crude benchmark, sorry about that, I did not have experience with something like sysbench or tcp-c, would be interesting to see results from such benchmarks also. - Kristian.

1 0

Re: [Maria-developers] Bundle more relevant cnf files in MariaDB package
by Arjen Lentz 28 Mar '11

28 Mar '11

Hi Haidong On 12/03/2011, at 7:56 AM, Haidong Ji wrote: > I alluded to this in IRC maybe a couple of weeks back. That is, it is > really to have intelligent default for MariaDB servers. there already is a baseline my.cnf which is installed by the DEB (Debian/Ubuntu) packages. You'll find it in the debian build subdirs. It can a) be updated with more parameters b) also be put into RPM, tarball and other builds. I would recommend not redoing it from scratch, as a number of practical issues have to be considered for this kind of default config, and reinventing the wheel there is just not necessary. Simply extend what's already there! > In particular, the bundled sample my.cnf files, under > /usr/local/mysql/support-files, such as my-small.cnf, my-medium.cnf, > my-large.cnf, and my-huge.cnf, are laughable in today's world. yep they are crap and should be removed. However, since they've already been replaced in the real world this is a separate issue. The my-*.cnf are not proof of "we need to create a good sample config" but rather a "this is old junk that should be removed from the bzr tree." The hindrance there is that some builds currently use them, so simply deleting the files is not sufficient. I'll see if I can find those dependencies, and create a modified tree for merging. Cheers, Arjen. -- Arjen Lentz, Exec.Director @ Open Query (http://openquery.com) Remote expertise & maintenance for MySQL/MariaDB server environments. Follow us at http://openquery.com/blog/ & http://twitter.com/openquery _______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss(a)lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

1 0

[Maria-developers] Pushing/merging sample my.cnf
by Haidong Ji 25 Mar '11

25 Mar '11

All, I've created my trunk on launchpad for MariaDB, and am ready to make some updates to the sample my.cnf file(s) packaged with MariaDB. I have a few questions regarding how to proceed: 1. I've seen work log creation and assignment in MariaDB developer mailing list. Is work log creation necessary for changes like this, where no C/C++ source code is touched? If yes, any guidelines will be appreciated; 2. As this is more packaging related, what are the testing requirement? 3. What about peer review? I've blogged this and also just sent another email for more input. 4. After I make the change on my machine, build it, and push the changes back to Lauchpad, I will do a merge request. 5. Anything else? Comments/suggestions? Thanks, Haidong "Alex" Ji

2 4

[Maria-developers] review: [Commits] Rev 2946: Fix LP BUG#715738 in file:///home/tsk/mprog/src/5.3/
by Michael Widenius 23 Mar '11

23 Mar '11

Hi! >>>>> "timour" == timour <timour(a)askmonty.org> writes: timour> At file:///home/tsk/mprog/src/5.3/ timour> ------------------------------------------------------------ timour> revno: 2946 timour> revision-id: timour(a)askmonty.org-20110322132154-moeuyey4saatnm5x timour> parent: igor(a)askmonty.org-20110315193048-akp4yq0y079dkkx2 timour> committer: timour(a)askmonty.org timour> branch nick: 5.3 timour> timestamp: Tue 2011-03-22 15:21:54 +0200 timour> message: timour> Fix LP BUG#715738 timour> Analysis: timour> A query with implicit grouping is one with aggregate functions and timour> no GROUP BY clause. MariaDB inherits from MySQL an SQL extenstion timour> that allows mixing aggregate functions with non-aggregate fields. timour> If a query with such mixed select clause produces an empty result timour> set, the meaning of aggregate functions is well defined - either timour> NULL (MIN, MAX, etc.), or 0 (count(*)). However the non-aggregated timour> fields must also have some value, and the only reasonable value in timour> the case of empty result is NULL. <cut> === modified file 'sql/sql_lex.cc' --- a/sql/sql_lex.cc 2011-03-09 13:47:59 +0000 +++ b/sql/sql_lex.cc 2011-03-22 13:21:54 +0000 @@ -1667,6 +1667,11 @@ void st_select_lex::init_select() cond_value= having_value= Item::COND_UNDEF; inner_refs_list.empty(); full_group_by_flag= 0; + n_sum_items= 0; + n_child_sum_items= 0; + select_n_sum_item= 0; + select_n_nonsum_item= 0; + ok, but I am not totally happy with that we get even more things that needs to be initialized for every select (we already initialize a lot of things...) Please change the names of 'n_' to something more understandable. 'number_of_' or 'use the _count' prefix that we use in other places. === modified file 'sql/sql_yacc.yy' --- a/sql/sql_yacc.yy 2011-03-09 13:47:59 +0000 +++ b/sql/sql_yacc.yy 2011-03-22 13:21:54 +0000 @@ -1338,7 +1338,7 @@ bool my_yyoverflow(short **a, YYSTYPE ** ev_alter_on_schedule_completion opt_ev_rename_to opt_ev_sql_stmt %type <ulong_num> - ulong_num real_ulong_num merge_insert_types + ulong_num real_ulong_num merge_insert_types remember_n_sum_func %type <ulonglong_number> ulonglong_num real_ulonglong_num size_number @@ -7019,31 +7019,43 @@ select_item_list: ; select_item: - remember_name select_item2 remember_end select_alias + remember_n_sum_func remember_name select_item2 remember_end select_alias { THD *thd= YYTHD; - DBUG_ASSERT($1 < $3); + DBUG_ASSERT($2 < $4); - if (add_item_to_list(thd, $2)) + if (thd->lex->current_select->n_sum_items > $1) + ++thd->lex->current_select->select_n_sum_item; + else + ++thd->lex->current_select->select_n_nonsum_item; Not sure if the counting of select_n_nonsum_item is correct. For example, we may have an item: SELECT MAX(field)+const_field from t1 ... In this case 'MAX(field)+const_field' will be counted as select_n_sum_item even if it should be counted also as select_n_nonsum_item. Not sure we can do this test properly at the parsing stage... Regards, Monty

1 0

[Maria-developers] WL#189 New (by Psergey): Pending bulk renames in server codebase
by worklog-noreply＠askmonty.org 22 Mar '11

22 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Pending bulk renames in server codebase CREATION DATE..: Tue, 22 Mar 2011, 16:31 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-BackLog TASK ID........: 189 (http://askmonty.org/worklog/?tid=189) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: This task is to file things that we intend to rename in the server codebase but can't do it just yet because of the merge problems it will create. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#188 New (by Knielsen): Using --log-slave-updates to ensure crash safe/transactional slave state
by worklog-noreply＠askmonty.org 21 Mar '11

21 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Using --log-slave-updates to ensure crash safe/transactional slave state CREATION DATE..: Mon, 21 Mar 2011, 12:56 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 188 (http://askmonty.org/worklog/?tid=188) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Overview -------- A replication slave needs to preserve certain state between slave server restarts to be able to correctly resume replication from where it left off. In current MySQL replication, this state is kept in multiple simple files (master.info, relay-log.info). This is a big problem if the slave server crashes, as there is no guarantee that these files will be in a consistent state with what is in the table data (and binlog if using --log-slow-updates), or even with each other. The Google patch rpl_transaction_enabled has a partial solution for this, by duplicating the state inside InnoDB in a transactional way. At slave server startup, InnoDB can then decide to overwrite the files keeping the slave state with its own, hopefully more correct, information. There are some remaining problems with this approach, some of which may be fixable, some not. However, the basic problem here is the need to maintain state in a transactional/crash-safe way across multiple subsystems of the server (eg. replication and storage engine(s)). And we already have such a mechanism, in the form of the two-phase commit between engines and binlog. This worklog describes how we could use this existing mechanism to keep the replication slave state across server restarts in a crash-safe way, rather than introduce new complex mechanisms for every new piece of state to be kept. Idea ---- The main state we need to make crash-safe is the binlog position (filename,offset) of the next event from the master to execute on the slave. There is more state stored current, but there is less need for that to be transactional: - relay-log.info also stores the position in the relay log files on the slave which event execution has reached. In the case of normal shutdown this can be used file as-is. In case of recovery after a crash, it is possible that the relay logs are not consistent with the master and/or slave SQL thread, so it is probably better to just discard any existing relay log and re-fetch all necessary events from the master. - master.info mainly stores the connection information from CHANGE MASTER TO, which does not change often. The basic idea is that on slave server start, we get the required information from the binlog on the slave, rather than from these files (this requires that --log-slave-updates is enabled). This also has the advantage that we avoid the need to constantly update the state files after every transaction, saving some execution cost. When we shut down the server normally, we close the binlog in a way that allows at startup to detect if we are recovering from a crash or not. As part of this close, we can write whatever state we need to recover at the end of the binlog. Recovering the state then depends on whether we crashed or not. If we did not crash, then all we need is to able to find the position of whatever event was written at the end of the binlog with the state. One way is to just have a fixed offset from the end where this event starts; however this is not too robust against finding wrong data there, especially with respect to binlogs from different versions of the server possibly with different events (or event sizes) at the end of the log. Another way is that when we close the binlog, we seek back to the start of the log and write the position of the last event there (either in the format_description event, which already has version information, or a new event written just after format_description event). We already do this seek anyway to overwrite the flag in format_description event which signals that the binlog has been closed properly (not crashed). We just need to be sure to fsync() the binlog _before_ overwriting the crashed-or-not flag, so we are sure all data will be there in the not-crashed case. If we did crash, then we can not rely on any information at the end of the binlog. In this case, we can instead build on top of the already existing crash recovery mechanism. This mechanism scans the last binlog to build a list of all committed transactions, then uses this list to tell storage engines which previously prepared transactions to commit and which to roll back. As part of this scan, we can determine the last event executed, and from this we get the necessary state to continue replication in terms of the binlog position (filename,offset) on the master. Note that if something like group ID (MWL#175) or global transaction ID is implemented, then the state to preserve is the last executed ID rather than binlog position; however, the basic mechanism remains the same. Discussion ---------- The main disadvantage I see with this approach is that in order for this to be really crash safe, we need to run with innodb_flush_log_at_trx_commit=1 and sync_binlog=1. This requires 3 fsync() calls per commit, and group commit is not possible since the slave is single-threaded. This is likely to be too expensive for many installations to be able to use it. However, it may be possible to reduce this overhead sufficiently by using MWL#185 (grouping multiple commits together on the slave to reduce fsync() overhead), or by implementing parallel replication to utilise group commit (MWL#169, MWL#184, MWL#186). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#187 New (by Knielsen): Consistent (but non-durable) crash recovery from binlog 2-phase commit
by worklog-noreply＠askmonty.org 21 Mar '11

21 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Consistent (but non-durable) crash recovery from binlog 2-phase commit CREATION DATE..: Mon, 21 Mar 2011, 11:54 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 187 (http://askmonty.org/worklog/?tid=187) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Overview -------- Current MySQL needs no less than three fsync() calls for every commit in order to ensure the ability to recover into a consistent state (between master binlog and storage engine(s)). Without this, after a master crash, the binlog may be inconsistent with the data in tables on the master, and replication slaves will diverge from the master, possibly causing replication to eventually break. The MWL#116 implements group commit, so that each of the three fsync() calls can be shared among multiple transaction commits. The MWL#164 describes a way to reduce this to a single fsync() call shared among multiple commits, by recovering any transactions lost in storage engines from the binlog, thereby avoiding the need for fsync() calls in the engines. Both MWL#116 and MWL#164 improve performance without reducing functionality or ability to do crash recovery. This worklog describes a method to reduce the overhead to *zero* fsync() calls per commit. Though one will still want to have say one fsync() per second or whatever of the binlog. After a master crash, crash recovery is still guaranteed to recover the system into a consistent state, where every transaction committed in the storage engine is also committed in the binlog, and vice versa. However, the cost of this is that durability is no longer guaranteed: after a crash, a number of transactions committed before the crash may be lost. Thus this worklog is similar to the InnoDB feature innodb_flush_log_at_trx_commit=2 (or 0). That feature makes InnoDB not call fsync() after every commit, possibly loosing some transactions in case of crash, but still guaranteeing that the table data can be recovered into a consistent state. This worklog would do the same, but for the binlog and InnoDB together, whereas innodb_flush_log_at_trx_commit=2/0 only guarantees consistency for InnoDB in isolation. Idea ---- This worklog builds on top of the group commit framework implemented in MWL#116. The basic idea is similar to MWL#163, "release of row locks in InnoDB during prepare() phase". As soon as we have successfully run the prepare step in InnoDB, we make the transaction committed in memory so that it is visible to other transactions from that point on and all row locks are released. However, at this point we go further than MWL#163, and return successful commit to the client connection. The rest of the group commit work is delegated to a separate backgroud thread: writing and fsync()'ing to the binlog, running commit_ordered() and commit() inside the storage engine. The client is thus free to continue with new transactions without waiting for any fsync() delay. The background thread will do an fsync() call, but since no client is waiting for it, the background thread is free to do a long wait (eg. 1 second) between fsync() calls, potentially collecting lots of transtions for a single huge group commit. Note that any replication slave _will_ be waiting for the background thread; we must wait for fsync() of the binary log before sending events to the slaves, or we could in case of crash end up with more transactions on the slaves than on the master, which would break replication. So there will be a cost of extra replication latency. Discussion ---------- A main complication with this worklog is that the main part of the commit will run in a background thread, which is different from the thread that ran the main transaction. This requires that storage engines are checked, and modified as needed, so that they do not depend on thread local storage in that part of the code, and also do not do things like release a mutex that was aquired from another thread, etc. We also need to handle the issue that we need the THD object of the original transaction/thread to be able to finish the commit in the background thread. I think we will need to remove the THD object from the original thread before returning to the client and keep a reference to it in the queue of transactions waiting to group commit in the background thread. The background thread must then use this THD for the last part of commit and afterwards return the THD object to a pool of spare THDs; when/if the client starts a new transaction, it will then need to obtain a new THD from this pool of spare THDs. ---- Currently, the MWL#116 API has prepare_ordered() being called _after_ prepare(). I am not sure if there is any particularly good reason for this. The only reason I know is that this is how the Facebook innodb_release_locks_early patch did things, however when I asked I did not get any answer from Facebook why they did it in this order. If we switched this, we should be able to return to the client even earlier, already before prepare(), so we could keep the fsync() in prepare(). Alternatively, we should use MWL#164 to not have to call fsync() in prepare(); then we can return to the client only after prepare(), and still avoid any fsync() latency visible to the client. Note that we have to make the transaction committed to memory and visible to other transactions before returning to the client, just as with innodb_release_locks_early in MWL#163. The client would be quite confused if COMMIT returns ok, yet the committed transaction is not visible to following SELECT statements! ---- The first transaction to enter the group commit queue becomes the group commit leader, and would just signal the background thread to continue the commit sequence and then return to the client. Subsequent transactions that enqueue after would not signal the background thread; instead they would set a flag in their entry in the queue, so that the background thread knows that they are asynchroneous and should not be woken up after commit is done (the original thread may no longer exist at that point, or may be doing something else). ---- The main benefit of this worklog would be that we could get crash safe replication state on the master without any fsync() overhead visible to the client. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#186 New (by Knielsen): Parallel replication using interleaved binlog events to detect potential parallelism
by worklog-noreply＠askmonty.org 21 Mar '11

21 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Parallel replication using interleaved binlog events to detect potential parallelism CREATION DATE..: Mon, 21 Mar 2011, 10:45 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 186 (http://askmonty.org/worklog/?tid=186) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Similar to MWL#169 and MWL#184, this worklog describes a way to discover binlog events that can be executed in parallel on the slave, improving scalability with increasing CPU and I/O resources. This worklog only deals with parallel replication of row-based events. Statement-based events should still be executed in serial order. We also require that commits on the slave still happen in the same order as on the master (this guarantees that applications can never see a state on a slave that did not exist on the master, among other nice properties). Basic idea ---------- The idea is to work with row-based events. Suppose that on the master, we have two parallel transactions running, T_1 and T_2. Suppose that we have the following sequence of events: 1. T_2 modifies a row R_1, resulting in row event E_1 2. T_1 commits 3. T_2 commits. In this case, it is safe on a slave to apply the event E_1 of T_2 in parallel with any events of transaction T_1. Since we know T_1 committed before T_2, we know that T_1 did not have to wait for T_2 to release any row lock of R_1 in commit, so T_1 could not have modified R_1 after T_2 did. Similarly, since we know that T_2 executed E_1 before T_1 committed, we know E_1 did not have to wait for T_1 to release any row locks on R_1, so T_1 could not have modified R_1 before T_2 did. Thus, we know that T_1 and T_2 are not in conflict on this particular row R_1 and row event E_1. Thus, on a slave, with this knowledge we can start transaction T_2 and execute E_1 (in a separate thread from T_1 events) while T_1 is still running. Refinement: optimistic apply ---------------------------- Suppose now that instead we have the following sequence of events: 1. T_3 commits 2. T_4 modifies a row R_2, resulting in row event E_2 3. T_4 commits In this case, we can not be sure that event E_2 is independent of transaction T_3. It could well be that T_3 also modified R_2, and E_2 was waiting for T_3 to commit and release row locks. However, on a slave, we can nevertheless try to apply E_2 (in a separate thread) while T_3 is still running. However, then we have to deal with the situation that T_3 gets stuck waiting for T_4 to commit and release the lock on R_2. In this situation, we have a deadlock; T3 is waiting for T_4, but T_4 also needs to wait for T_3 (as we want to commit in the right order, and even more so want the end result to have the changes of T_4 in row R_2, not those of T_3). So we need to abort and restart one of the transactions. As long as we choose always to abort the transaction with the later commit time (T_4 in this case), things will work out: We will abort T_4, commit T_3, then re-start and commit T_4. Using optimistic apply has the potential to increase parallelism on the slave. However, this must be carefully weighted against the extra cost of having to roll back and retry some transactions. So it needs suitable status variables to be able to monitor number of transactions and number of events that were rolled back and re-started (as well as number of transactions and events successfully applied). And user options to control if and to what extend to try optimistic apply. But note that since we always have at least one transaction running that will not be aborted due to optimistic apply (the one with earliest commit), and since all other transactions run in parallel, the cost of restart may not be too bad, as long as we do not start to starve the CPU(s). Any extra I/O reads done in the aborted transaction to bring in pages to the buffer pool will in any case benefit us when we immediately re-start the transaction. Requirements ------------ Implementing this worklog will require several non-trivial extensions of the server and storage engine code (these should eventually be filed as separate worklog items): 1. Ability for server layer to detect and control what to do in case of conflicts between two transactions. Currently, the storage engine (eg. InnoDB/XtraDB) will just let the latter access block waiting for the first transaction to commit, and in the case of a deadlock the storage engine will choose an arbitrary transaction to roll back. For this worklog, the server layer (replication slave SQL thread(s)) must be able to (a) cause abort of the former transaction rather than wait of the second, and (b) choose which of several possible transactions to abort in case of deadlock. Note that such a mechanism is also needed for Galera replication, and the Galera wsrep patch implements something like this for InnoDB already. 2. Interleaving of events from different transactions in the binlog. Currently, the master buffers events during a transaction and writes them all out sequentially during COMMIT; this destroys any knowledge about which events and transactions ran in parallel on the master. We need to change this so that events are tagged with a (local) transaction ID and written out to the binlog as they occur, interleaved with events from other transactions. We will probably also want to add "begin transaction" events and "prepare" event, to increase the knowledge available on the slave about in which order events occured. We will still want to do _some_ buffering of events to avoid heavy contention on writing to the binlog. We can buffer a small number of normal events; The cost of such buffering is we may write a row event E_r _after_ a commit event E_c where it could have been written before, which loses some opportunity for (non-optimistic) parallel apply on the slave. We may however _not_ buffer commit events; such buffering could wrongly put E_r ahead of E_c where it must be after, causing an unexpected conflict in parallel apply on the slave. 3. A mechanism on the slave for distributing binlog events among multiple threads and coordinating execution and commit among the threads. Such mechanism should ideally be the same as used in MWL#184 and MWL#169. Discussion ---------- One nice property of this worklog is that the parallelism exposed to the slave is derived from whatever parallelism was there already on the master. We know E_1 in T_2 ran in parallel with T_1 on the master, and use this to similarly run E_1 in parallel with T_1 on the slave. This increases the likelyhood that if parallelism is available on the master, it will also be available on the slave (note that we will need parallelism on both master and slave to be able to scale an entire replication topology). ---- For short OLTP-like transactions, it seems likely that this method will be particularly suited to be combined with the method from MWL#184, which allows to run in parallel on the slave all transactions that participate in a single group commit on the master. Eg. for single-row transactions T_1 (with row event E_1) and T_2 (with row event E_2) that run in parallel on the master, we can have the following two situations (among others): 1. E_1 and E_2 execute start 1. E_1 and E_2 execute start 2. E_1 execute finish 2. E_1 execute finish 3. E_2 execute finish 3. T_1 commit 4. T_1 commit 4. E_2 execute finish 5. T_2 commit 5. T_2 commit In the scenario to the left, on a busy master the two commits are likely to be done as a single group commit, and MWL#184 will allow parallelism on the slave. In the scenario on the right, optimistic apply as per this worklog will allow to run T_1 and T_2 in parallel on the slave. ---- Note that by interleaving events from different transactions in the binlog, we will be writing events before they are committed, and so will need to issue ROLLBACK statements in case of rollback. Similarly, on the slave, we can start applying the events from a transaction even before we have received the commit event from the master (in fact even before such commit event has even occured on the master). This can potentially reduce master->slave delay. However, there is a need to be careful, as on the slave we need to keep the correct commit order. And such order is not known until the commit events are received. For example, if we had a limited number N of slave execution threads and they all have started running T_1, ..., T_N, and we then receive a commit event for T_(N+1), we would be deadlocked. It might be a good idea to only start executing new transactions when we have at least one pending commit in the relay log, or at least keep one spare thread available for execution of one more transaction once we receive such commit event. Similarly, if we get a conflict in opportunistic apply of transactions T_1 and T_2, neither of which we have received the commit event from, then we will need to wait for either commit event to arrive before we can decide which of T_1 and T_2 to abort. Note that in any case what we need to do is no worse than what we do in currentl MySQL replication. There, we never do any execution until we have not only seen but also executed any prior commit. Similarly, whatever buffering of events we need to do on the parallel slave will be no worse than what we currently have to do on the master (where we buffer _all_ events until commit). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#185 New (by Knielsen): Group commits on slave to reduce fsync() load
by worklog-noreply＠askmonty.org 18 Mar '11

18 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Group commits on slave to reduce fsync() load CREATION DATE..: Fri, 18 Mar 2011, 14:25 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 185 (http://askmonty.org/worklog/?tid=185) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: MWL#116 introduces group commit for the binary log; this can reduce the number of fsync() calls needed when running with sync_binlog=1. But this does not currently help on the slave. The reason is that the slave SQL thread is single-threaded, so there will never be any opportunity for group commit. However, an alternative way to reduce fsync() on the slave when sync_binlog=1 is to combine multiple transactions from the binlog into one transaction on the slave. If we have N transactions T_1, ..., T_N in the binlog, we can combine them simply into one transaction on the slave like this: - Change COMMIT in T_1, ..., T_(N-1) into SAVEPOINT T_i - Change any ROLLBACK into ROLLBACK to T_i This will reduce the number of fsync() calls, since it results in fewer COMMITs. The downside is that the latency between a transaction being visible on the master and on the slave will increase. There should probably be some option to limit how much transactions may be delayed on the slave. It would also be useful to have some kind of timestamps in the binlog to be able to estimate the duration of a transaction on the slave before starting to apply it; this would allow to avoid combining a later long transaction with a previous short transaction, which could cause undue delay for the short transaction to become visible on the slave. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

Re: [Maria-developers] mysql_upgrade issue for libmysqld and amarok
by Kristian Nielsen 18 Mar '11

18 Mar '11

Ian Monroe <ian(a)monroe.nu> writes: > On Fri, Mar 18, 2011 at 05:44, Kristian Nielsen > <knielsen(a)knielsen-hq.org> wrote: >> The issue is how to upgrade libmysqd from MySQL version 5.1 to version 5.5 (or >> more generally any major version upgrade), in particular as relates to >> Amarok's use of libmysqld. >> One idea is to also build mysql_upgrade_embedded; this would be a variant of >> mysql_upgrade linked with libmysqld, so it does not depend on the mysqld >> server. Similarly, we would need mysqlcheck_embedded (mysqlcheck is called >> from mysql_upgrade). These two binaries could be included in the libmysqld >> package (they are small). > > wouldn't they be huge? libmysqld is a static library, so anything > linking to them is going to be at least 50mb. In MariaDB, we have both static and dynamic libmysqld. So we should link with the dynamic libmysqld.so, and binaries should be small. >> The upgrade procedure for amarok would then be to stop amarok linked with 5.1, >> then run mysql_upgrade_embedded (from 5.5) against the amarok libmysqld data >> directory, then start amarok linked with 5.5. > > There can't be any procedure that the user has to do. Agree. > But if there is some way for us to determine if mysql_upgrade_embedded > needs to be run, I don't see why this would a problem. Amarok could > run it before initializing the database. I would suggest to write a small file in the user's data directory with the version last upgraded to. If this is < 5.5 at startup (eg. "5.1"), then Amarok can run the upgrade and afterwards update the file to say "5.5". This will also allow to detect and warn about downgrading 5.5->5.1. Debian does something similar for the main MySQL server packages. >> The other idea that came up was to add this as extra calls in the library. So >> that the amarok application could just call some mysql_upgrade() function >> instead of having to run an external upgrade binary. Would this be more useful >> for amarok? > > This sounds much more straightforward. Ok, I will see if we can do this. Probably we could do both (then mysql_upgrade_embedded will just be a stub calling into mysql_upgrade() in the library, might be useful in other applications). >> We could offer to implement this (either solution I think) in MariaDB (a >> version of MariaDB that includes MySQL 5.5 is in the works). This would allow >> to use the above to upgrade Amarok to using MariaDB 5.5. Would this work? > > Sounds good. > We really could care less about maria vs. mysql, we just want > something that works. Ok, great :-) > Since I talked to you last time, I've had reports of people going from > mysql 5.1 to 5.5 without needing mysql_upgrade. mysql_upgrade seemed > to fix things for me, but maybe it was just a coincidence. Yes. The main issue I can think of now is that when character sets/collations change, the indexes of affected tables may need to be rebuilt. Otherwise they may contain incorrect ordering or indexing of affected characters. Since such changes are rare, and tables with affected characters probably even rarer, a missing upgrade is likely to go ok or at least unnoticed for some time. > Either way though, relying on newer versions of mysql simply not > needing to do any upgrade procedures seems like poor planning. Agree, even if it works for 5.5 it will be sure to break sooner or later. Monty said he would look into implementing this. He is quite busy, so it may be a few weeks, but I will try to follow up with him to see if we can move forward with it reasonably quickly. - Kristian.

1 0

[Maria-developers] WL#184 New (by Knielsen): Parallel replication of group-committed transactions
by worklog-noreply＠askmonty.org 18 Mar '11

18 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Parallel replication of group-committed transactions CREATION DATE..: Fri, 18 Mar 2011, 14:06 SUPERVISOR.....: Sergei IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 184 (http://askmonty.org/worklog/?tid=184) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Overview -------- Parallel replication is the idea of executing multiple transactions (or more precisely event groups from the binlog) in parallel in multiple threads on the slave, in order to make the slave applier more scalable to multiple CPU cores and disk spindles. The main problem with parallel replication is identifying transactions that are *independent*, meaning that they can be safely executed in parallel on the slave without dead-locking each other or producing different results than on the master. This worklog describes one way of identifying such independence. MWL#169 describes another. Apart from discovering independent transactions, it is also necessary to implement a mechanism for actually distributing events to different execution threads, coordinating commit order, handling errors and crash recovery, etc. It is expected that different ways of determining independent transactions, such as this worklog and MWL#169, will share the same mechanism for actual parallel execution. It should even be possible to combine different ways to discover independence at the same time. Idea ---- MWL#116 introduces group commit, where multiple transactions are committed together and also written to the binlog together. This works by collecting parallel transactions that are waiting inside COMMIT (or autocommit) for access to the binary log. The idea in this worklog is that all transactions participating in a single group commit must necessarily be independent, and can be safely applied in parallel on a slave. Suppose we are about to write N transactions T_1, ..., T_N into the binlog in group commit. Then all statements for each transaction have completed, but none of them are committed yet. This means that (1) there can be no conflicting row locks between any two T_i and T_j, and (2) no transaction T_i was able to see any changes from T_j (for i<>j and assuming isolation level greater than read uncommitted). Another way to see this is to note that on the master, the order of T_1, ..., T_N is essentially random and depends only on how the kernel decided to schedule the threads during the COMMIT. Any ordering would have been possible just by having different thread scheduling delays when the transactions are put in the ordered list used for group commit; there is no inter-thread dependencies at that point in the code. So any order of the transactions in the binlog would have been possible, and thus any order of slave execution. Therefore, as long as COMMIT cannot change what modifications the transaction already did, any execution order of the transactions on the slave is permissible. Implementation -------------- I suggest that when this feature is enabled, we introduce two new binlog events, group_commit_begin and group_commit_end. We write a group_commit_begin event just before all the events for the transactions in a group commit, and a group_commit_end event just after. We can use the "ignorable events" from MySQL 5.5 for these events; this will allow a slave that does not implement these events to safely ignore them. If needed, we can further increase compatibility by - Having a server option to enable/disable the writing of these two events. - Add a flag that the slave sets to request the sending of these events, and only send them if requested (similar to MWL#47). It still needs to be decided how much of this to do. The slave can then collect all the event groups between the group_commit_begin and group_commit_end pair, and schedule them in parallel. Limitations ----------- The main limitation of this approach is that on the slave, it can only run transactions in parallel if they happened to commit at roughly the same time on the master. This means that there will be little opportunity for parallelism on the slave if 1. The master has low write load 2. The master is mainly running few but long transactions 3. The master has fast fsync() performance for the commit In the case (1), there is in any case less need for scaling the event execution. In the case (2), another method (for example MWL#169) will be needed to achieve efficient parallel replication. For (3), we can improve parallelism by deliberately delaying commits on the master. By inserting a short sleep just before group commit, we allow more independent transactions time to participate, giving more opportunity for parallelism on the slave. At the same time, we reduce fsync() load on the master, potentially freeing more I/O capacity for other work. For example, we might add an option to rate-limit COMMIT fsync() to 25 per seconds or whatever. Care will be needed when using this option, as if the master load has insufficient parallelism, such rate-limitation will reduce throughput on the master. The MWL#163 innodb_release_locks_early option causes transactions to effectively be committed early in COMMIT during the prepare phase. This allows another transaction to see and/or modify rows also modified by the first transaction, and still participate in the same group commit. This means that when innodb_release_locks_early is set, the transactions cannot be safely used for parallel replication (eg. in this case we must disable the writing of group_commit_begin / group_commit_end). Events that modify non-transactional tables or do DDL causes more complications, as they are written directly to the binlog and make changes visible immediately to other transactions (basically they are not transactional). To simplify things, we will not declare any such events independent with respect to any other transactions according to this worklog. There are many useful applications of parallel replications where such events are rare. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0