- developers - lists.mariadb.org

[Maria-developers] Updated (by Psergey): Subqueries: Inside-out execution for non-semijoin materialized subqueries that are AND-parts of the WHERE (92)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: Inside-out execution for non-semijoin materialized subqueries that are AND-parts of the WHERE CREATION DATE..: Sun, 28 Feb 2010, 14:02 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Psergey, Timour CATEGORY.......: Server-RawIdeaBin TASK ID........: 92 (http://askmonty.org/worklog/?tid=92) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Sun, 28 Feb 2010, 14:03)=-=- Observers changed: Psergey,Igor,Timour DESCRIPTION: Consider the following case: SELECT * FROM big_table WHERE oe IN (SELECT ie FROM table_with_few_groups WHERE ... GROUP BY group_col) AND ... Here the best way to execute the query is: Materialize the subquery; # now run the join: for each record R1 in materialized table for each record R2 in big_table such that oe=R1 pass R2 to output Semi-join materialization supports such strategy with SJM-Scan strategy. This WL entry is about adding support for such strategies for non-semijoin subqueries. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Subqueries: Inside-out execution for non-semijoin materialized subqueries that are AND-parts of the WHERE (92)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: Inside-out execution for non-semijoin materialized subqueries that are AND-parts of the WHERE CREATION DATE..: Sun, 28 Feb 2010, 14:02 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 92 (http://askmonty.org/worklog/?tid=92) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Consider the following case: SELECT * FROM big_table WHERE oe IN (SELECT ie FROM table_with_few_groups WHERE ... GROUP BY group_col) AND ... Here the best way to execute the query is: Materialize the subquery; # now run the join: for each record R1 in materialized table for each record R2 in big_table such that oe=R1 pass R2 to output Semi-join materialization supports such strategy with SJM-Scan strategy. This WL entry is about adding support for such strategies for non-semijoin subqueries. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Subqueries: Inside-out execution for non-semijoin materialized subqueries that are AND-parts of the WHERE (92)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: Inside-out execution for non-semijoin materialized subqueries that are AND-parts of the WHERE CREATION DATE..: Sun, 28 Feb 2010, 14:02 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 92 (http://askmonty.org/worklog/?tid=92) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Consider the following case: SELECT * FROM big_table WHERE oe IN (SELECT ie FROM table_with_few_groups WHERE ... GROUP BY group_col) AND ... Here the best way to execute the query is: Materialize the subquery; # now run the join: for each record R1 in materialized table for each record R2 in big_table such that oe=R1 pass R2 to output Semi-join materialization supports such strategy with SJM-Scan strategy. This WL entry is about adding support for such strategies for non-semijoin subqueries. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Subqueries: cost-based choice between Materialization and IN->EXISTS transformation (90)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: cost-based choice between Materialization and IN->EXISTS transformation CREATION DATE..: Sun, 28 Feb 2010, 13:45 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Psergey, Timour CATEGORY.......: Server-RawIdeaBin TASK ID........: 90 (http://askmonty.org/worklog/?tid=90) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: -1 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: For uncorrelated IN subqueries that can't be converted to semi-joins it is necessary to make a cost-based choice between IN->EXISTS and Materialization strategies. Both strategies handle two cases: 1. A simple case w/o NULLs handling 2. Handling NULLs. This WL is about making cost-based decision for #1. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Subqueries: cost-based choice between Materialization and IN->EXISTS transformation (90)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: cost-based choice between Materialization and IN->EXISTS transformation CREATION DATE..: Sun, 28 Feb 2010, 13:45 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Psergey, Timour CATEGORY.......: Server-RawIdeaBin TASK ID........: 90 (http://askmonty.org/worklog/?tid=90) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: -1 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: For uncorrelated IN subqueries that can't be converted to semi-joins it is necessary to make a cost-based choice between IN->EXISTS and Materialization strategies. Both strategies handle two cases: 1. A simple case w/o NULLs handling 2. Handling NULLs. This WL is about making cost-based decision for #1. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Subqueries: cost-based choice between Materialization and IN->EXISTS transformation (90)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: cost-based choice between Materialization and IN->EXISTS transformation CREATION DATE..: Sun, 28 Feb 2010, 13:45 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Psergey, Timour CATEGORY.......: Server-RawIdeaBin TASK ID........: 90 (http://askmonty.org/worklog/?tid=90) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: -1 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: For uncorrelated IN subqueries that can't be converted to semi-joins it is necessary to make a cost-based choice between IN->EXISTS and Materialization strategies. Both strategies handle two cases: 1. A simple case w/o NULLs handling 2. Handling NULLs. This WL is about making cost-based decision for #1. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Subqueries: cost-based choice between Materialization and IN->EXISTS transformation (90)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: cost-based choice between Materialization and IN->EXISTS transformation CREATION DATE..: Sun, 28 Feb 2010, 13:45 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Psergey, Timour CATEGORY.......: Server-RawIdeaBin TASK ID........: 90 (http://askmonty.org/worklog/?tid=90) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: -1 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: For uncorrelated IN subqueries that can't be converted to semi-joins it is necessary to make a cost-based choice between IN->EXISTS and Materialization strategies. Both strategies handle two cases: 1. A simple case w/o NULLs handling 2. Handling NULLs. This WL is about making cost-based decision for #1. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Knielsen): Update packaging scripts for MariaDB 5.2 (88)
by worklog-noreply＠askmonty.org 27 Feb '10

27 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Update packaging scripts for MariaDB 5.2 CREATION DATE..: Sat, 27 Feb 2010, 16:39 SUPERVISOR.....: Knielsen IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 88 (http://askmonty.org/worklog/?tid=88) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 30 (hours remain) ORIG. ESTIMATE.: 30 PROGRESS NOTES: DESCRIPTION: The packaging scripts need to be updated to work for MariaDB 5.2 Currently, 5.2 package builds fail in Buildbot. The .debs are missing a debian-5.2 subdirectory. The .rpm also need to be checked. Buildbot needs to be updated to do the new upgrade tests (mariadb-5.1 -> mariadb 5.2) ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): Enable the use of libstdc++ in MariaDB (63)
by worklog-noreply＠askmonty.org 27 Feb '10

27 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Enable the use of libstdc++ in MariaDB CREATION DATE..: Wed, 11 Nov 2009, 13:19 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-BackLog TASK ID........: 63 (http://askmonty.org/worklog/?tid=63) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 4 ESTIMATE.......: 10 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: -=-=(Knielsen - Sat, 27 Feb 2010, 16:02)=-=- Status updated. --- /tmp/wklog.63.old.30948 2010-02-27 16:02:20.000000000 +0000 +++ /tmp/wklog.63.new.30948 2010-02-27 16:02:20.000000000 +0000 @@ -1 +1 @@ -Un-Assigned +Assigned -=-=(Knielsen - Thu, 14 Jan 2010, 13:46)=-=- Research and updated description Worked 4 hours and estimate 10 hours remain (original estimate increased by 14 hours). -=-=(Knielsen - Thu, 14 Jan 2010, 13:45)=-=- High-Level Specification modified. --- /tmp/wklog.63.old.13967 2010-01-14 11:45:49.000000000 +0000 +++ /tmp/wklog.63.new.13967 2010-01-14 11:45:49.000000000 +0000 @@ -1 +1,37 @@ +I did some investigation into this. + +The simple way to do this is to simply use g++ to link C++ objects. So this +issue is really restricted to GCC compilation where we by default prefer to +link with gcc even for C++ code. So this means building with + + CXX=g++ + +The consequences of doing this for the binaries is the addition of two +additional run-time .so dependencies: libstdc++.so and libgcc_s.so. + +It still needs to be investigated if these additional dependencies are a +problem for binary tarball packages, or if the ABI for those libraries are now +as stable as libc.so. + +The libgcc_s.so is needed as a dependency to support exceptions between +different object files, as they need to use the same code for stack unwinding. +libstdc++.so is of course needed for access to C++ runtime. + +I researched into the possibility to instead link only specific plugins with +g++, and continue to link the rest of the server with gcc. Unfortunately, this +seems really hard to do in a proper way due to the way autotools works. At +configure time, a script ./libtool is created with hardcoded compiler commands +derived from $CC and $CXX. This script is then used to do the actual linking +in Makefiles generated by Automake. I thus did not find a way to change the +linker command on a per-makefile basis, as libtool is global to the project. + +One option would be to use separate configure.in for plugins, but this is +quite an intrusive change. + +My conclusion is that the best way is to start using g++ for linking the +entire server. This is no problem for binaries made for a specific +distribution (Ubuntu, Debian, Centos), where the dependencies are handled by +the package manager. If it is a big problem for binary tarball releases, at +worst we can build multiple binary tarball releases for the different library +versions we need to support. -=-=(Guest - Tue, 12 Jan 2010, 16:26)=-=- Version updated. --- /tmp/wklog.63.old.19522 2010-01-12 16:26:23.000000000 +0200 +++ /tmp/wklog.63.new.19522 2010-01-12 16:26:23.000000000 +0200 @@ -1 +1 @@ -Connector/.NET-5.2 +Server-5.2 -=-=(Guest - Tue, 12 Jan 2010, 16:26)=-=- Category updated. --- /tmp/wklog.63.old.19506 2010-01-12 16:26:15.000000000 +0200 +++ /tmp/wklog.63.new.19506 2010-01-12 16:26:15.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-BackLog DESCRIPTION: Enable the use of libstdc++ in MariaDB. As time goes on, more and more plugins and external code/library will need linking to libstdc++ for stuff it uses. I have already seen this happen several times, with extra work needed to integrate things properly. It would be nice to have a general solution for this so that it is not necessary to spend time on individual solutions in each case. It also needs to be considered what the impact of this will be for the server in terms of binary compatibility, performance etc. I think it should be mostly ok, except that it might introduce a problem for bintar packages with an external dependency on libstdc++. HIGH-LEVEL SPECIFICATION: I did some investigation into this. The simple way to do this is to simply use g++ to link C++ objects. So this issue is really restricted to GCC compilation where we by default prefer to link with gcc even for C++ code. So this means building with CXX=g++ The consequences of doing this for the binaries is the addition of two additional run-time .so dependencies: libstdc++.so and libgcc_s.so. It still needs to be investigated if these additional dependencies are a problem for binary tarball packages, or if the ABI for those libraries are now as stable as libc.so. The libgcc_s.so is needed as a dependency to support exceptions between different object files, as they need to use the same code for stack unwinding. libstdc++.so is of course needed for access to C++ runtime. I researched into the possibility to instead link only specific plugins with g++, and continue to link the rest of the server with gcc. Unfortunately, this seems really hard to do in a proper way due to the way autotools works. At configure time, a script ./libtool is created with hardcoded compiler commands derived from $CC and $CXX. This script is then used to do the actual linking in Makefiles generated by Automake. I thus did not find a way to change the linker command on a per-makefile basis, as libtool is global to the project. One option would be to use separate configure.in for plugins, but this is quite an intrusive change. My conclusion is that the best way is to start using g++ for linking the entire server. This is no problem for binaries made for a specific distribution (Ubuntu, Debian, Centos), where the dependencies are handled by the package manager. If it is a big problem for binary tarball releases, at worst we can build multiple binary tarball releases for the different library versions we need to support. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): Enable the use of libstdc++ in MariaDB (63)
by worklog-noreply＠askmonty.org 27 Feb '10

27 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Enable the use of libstdc++ in MariaDB CREATION DATE..: Wed, 11 Nov 2009, 13:19 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-BackLog TASK ID........: 63 (http://askmonty.org/worklog/?tid=63) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 4 ESTIMATE.......: 10 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: -=-=(Knielsen - Sat, 27 Feb 2010, 16:02)=-=- Status updated. --- /tmp/wklog.63.old.30948 2010-02-27 16:02:20.000000000 +0000 +++ /tmp/wklog.63.new.30948 2010-02-27 16:02:20.000000000 +0000 @@ -1 +1 @@ -Un-Assigned +Assigned -=-=(Knielsen - Thu, 14 Jan 2010, 13:46)=-=- Research and updated description Worked 4 hours and estimate 10 hours remain (original estimate increased by 14 hours). -=-=(Knielsen - Thu, 14 Jan 2010, 13:45)=-=- High-Level Specification modified. --- /tmp/wklog.63.old.13967 2010-01-14 11:45:49.000000000 +0000 +++ /tmp/wklog.63.new.13967 2010-01-14 11:45:49.000000000 +0000 @@ -1 +1,37 @@ +I did some investigation into this. + +The simple way to do this is to simply use g++ to link C++ objects. So this +issue is really restricted to GCC compilation where we by default prefer to +link with gcc even for C++ code. So this means building with + + CXX=g++ + +The consequences of doing this for the binaries is the addition of two +additional run-time .so dependencies: libstdc++.so and libgcc_s.so. + +It still needs to be investigated if these additional dependencies are a +problem for binary tarball packages, or if the ABI for those libraries are now +as stable as libc.so. + +The libgcc_s.so is needed as a dependency to support exceptions between +different object files, as they need to use the same code for stack unwinding. +libstdc++.so is of course needed for access to C++ runtime. + +I researched into the possibility to instead link only specific plugins with +g++, and continue to link the rest of the server with gcc. Unfortunately, this +seems really hard to do in a proper way due to the way autotools works. At +configure time, a script ./libtool is created with hardcoded compiler commands +derived from $CC and $CXX. This script is then used to do the actual linking +in Makefiles generated by Automake. I thus did not find a way to change the +linker command on a per-makefile basis, as libtool is global to the project. + +One option would be to use separate configure.in for plugins, but this is +quite an intrusive change. + +My conclusion is that the best way is to start using g++ for linking the +entire server. This is no problem for binaries made for a specific +distribution (Ubuntu, Debian, Centos), where the dependencies are handled by +the package manager. If it is a big problem for binary tarball releases, at +worst we can build multiple binary tarball releases for the different library +versions we need to support. -=-=(Guest - Tue, 12 Jan 2010, 16:26)=-=- Version updated. --- /tmp/wklog.63.old.19522 2010-01-12 16:26:23.000000000 +0200 +++ /tmp/wklog.63.new.19522 2010-01-12 16:26:23.000000000 +0200 @@ -1 +1 @@ -Connector/.NET-5.2 +Server-5.2 -=-=(Guest - Tue, 12 Jan 2010, 16:26)=-=- Category updated. --- /tmp/wklog.63.old.19506 2010-01-12 16:26:15.000000000 +0200 +++ /tmp/wklog.63.new.19506 2010-01-12 16:26:15.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-BackLog DESCRIPTION: Enable the use of libstdc++ in MariaDB. As time goes on, more and more plugins and external code/library will need linking to libstdc++ for stuff it uses. I have already seen this happen several times, with extra work needed to integrate things properly. It would be nice to have a general solution for this so that it is not necessary to spend time on individual solutions in each case. It also needs to be considered what the impact of this will be for the server in terms of binary compatibility, performance etc. I think it should be mostly ok, except that it might introduce a problem for bintar packages with an external dependency on libstdc++. HIGH-LEVEL SPECIFICATION: I did some investigation into this. The simple way to do this is to simply use g++ to link C++ objects. So this issue is really restricted to GCC compilation where we by default prefer to link with gcc even for C++ code. So this means building with CXX=g++ The consequences of doing this for the binaries is the addition of two additional run-time .so dependencies: libstdc++.so and libgcc_s.so. It still needs to be investigated if these additional dependencies are a problem for binary tarball packages, or if the ABI for those libraries are now as stable as libc.so. The libgcc_s.so is needed as a dependency to support exceptions between different object files, as they need to use the same code for stack unwinding. libstdc++.so is of course needed for access to C++ runtime. I researched into the possibility to instead link only specific plugins with g++, and continue to link the rest of the server with gcc. Unfortunately, this seems really hard to do in a proper way due to the way autotools works. At configure time, a script ./libtool is created with hardcoded compiler commands derived from $CC and $CXX. This script is then used to do the actual linking in Makefiles generated by Automake. I thus did not find a way to change the linker command on a per-makefile basis, as libtool is global to the project. One option would be to use separate configure.in for plugins, but this is quite an intrusive change. My conclusion is that the best way is to start using g++ for linking the entire server. This is no problem for binaries made for a specific distribution (Ubuntu, Debian, Centos), where the dependencies are handled by the package manager. If it is a big problem for binary tarball releases, at worst we can build multiple binary tarball releases for the different library versions we need to support. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Rev 2765: Change Field_enumerator to enumerate Item_field-s not Field-s. in file:///home/psergey/dev/maria-5.3-subqueries-r7/
by Sergey Petrunya 27 Feb '10

27 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r7/ ------------------------------------------------------------ revno: 2765 revision-id: psergey(a)askmonty.org-20100221063223-h0f7u2low7rtjixc parent: psergey(a)askmonty.org-20100221033618-83dgm2h9ingzmhcc committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r7 timestamp: Sun 2010-02-21 08:32:23 +0200 message: Change Field_enumerator to enumerate Item_field-s not Field-s. In Item_ref::fix_fields() do invoke mark_as_dependent() for outside references in all cases (see email for more details) === modified file 'sql/item.cc' --- a/sql/item.cc 2010-02-11 23:59:58 +0000 +++ b/sql/item.cc 2010-02-21 06:32:23 +0000 @@ -1959,7 +1959,7 @@ bool Item_field::enumerate_field_refs_processor(uchar *arg) { Field_enumerator *fe= (Field_enumerator*)arg; - fe->visit_field(field); + fe->visit_field(this); return FALSE; } @@ -5779,6 +5779,35 @@ set_properties(); } +/* + A Field_enumerator-compatible class that invokes mark_as_dependent() for + each field that is a reference to some ancestor of current_select. +*/ +class Dependency_marker: public Field_enumerator +{ +public: + THD *thd; + st_select_lex *current_select; + virtual void visit_field(Item_field *item) + { + // Find which select the field is in. This is achieved by walking up + // the select tree and looking for the table of interest. + st_select_lex *sel; + for (sel= current_select; sel; sel= sel->outer_select()) + { + TABLE_LIST *tbl; + for (tbl= sel->leaf_tables; tbl; tbl= tbl->next_leaf) + { + if (tbl->table == item->field->table) + { + if (sel != current_select) + mark_as_dependent(thd, sel, current_select, item, item); + return; + } + } + } + } +}; /** Resolve the name of a reference to a column reference. @@ -6038,6 +6067,20 @@ last_checked_context->select_lex->nest_level); } } + else + { + ; + /* + It could be that we're referring to something that's in ancestor selects. + We must make an appropriate mark_as_dependent() call for each such + outside reference. + */ + Dependency_marker dep_marker; + dep_marker.current_select= current_sel; + dep_marker.thd= thd; + (*ref)->walk(&Item::enumerate_field_refs_processor, FALSE, + (uchar*)&dep_marker); + } DBUG_ASSERT(*ref); /* === modified file 'sql/item.h' --- a/sql/item.h 2010-02-21 03:36:18 +0000 +++ b/sql/item.h 2010-02-21 06:32:23 +0000 @@ -1134,7 +1134,7 @@ class Field_enumerator { public: - virtual void visit_field(Field *field)= 0; + virtual void visit_field(Item_field *field)= 0; virtual ~Field_enumerator() {}; /* purecov: inspected */ }; === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-02-21 03:36:18 +0000 +++ b/sql/item_subselect.cc 2010-02-21 06:32:23 +0000 @@ -319,13 +319,13 @@ public: table_map used_tables; /* Collect used_tables here */ st_select_lex *new_parent; /* Select we're in */ - virtual void visit_field(Field *field) + virtual void visit_field(Item_field *item) { //for (TABLE_LIST *tbl= new_parent->leaf_tables; tbl; tbl= tbl->next_local) //{ // if (tbl->table == field->table) // { - used_tables|= field->table->map; + used_tables|= item->field->table->map; // return; // } //} === modified file 'sql/opt_table_elimination.cc' --- a/sql/opt_table_elimination.cc 2010-01-17 14:51:10 +0000 +++ b/sql/opt_table_elimination.cc 2010-02-21 06:32:23 +0000 @@ -922,8 +922,9 @@ Field_dependency_recorder(Dep_analysis_context *ctx_arg): ctx(ctx_arg) {} - void visit_field(Field *field) + void visit_field(Item_field *item) { + Field *field= item->field; Dep_value_table *tbl_dep; if ((tbl_dep= ctx->table_deps[field->table->tablenr])) {

1 2

[Maria-developers] Updated (by Guest): Subquery optimization: Efficient NOT IN execution with NULLs (68)
by worklog-noreply＠askmonty.org 27 Feb '10

27 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subquery optimization: Efficient NOT IN execution with NULLs CREATION DATE..: Fri, 27 Nov 2009, 13:22 SUPERVISOR.....: Monty IMPLEMENTOR....: Timour COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 68 (http://askmonty.org/worklog/?tid=68) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Guest - Sat, 27 Feb 2010, 10:11)=-=- Status updated. No change. -=-=(Guest - Sat, 27 Feb 2010, 10:11)=-=- Status updated. --- /tmp/wklog.68.old.24229 2010-02-27 10:11:57.000000000 +0000 +++ /tmp/wklog.68.new.24229 2010-02-27 10:11:57.000000000 +0000 @@ -1 +1 @@ -Assigned +In-Progress -=-=(Timour - Mon, 22 Feb 2010, 17:39)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.17116 2010-02-22 17:39:48.000000000 +0200 +++ /tmp/wklog.68.new.17116 2010-02-22 17:39:48.000000000 +0200 @@ -233,6 +233,7 @@ 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all and v_j1,...,v_jm cannot be null, create for these columns only one index array (and of course do not create any bitmaps for them). +[done] 2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number of rows, where a_i is not null and V(a_i) is the number of distinct @@ -264,6 +265,10 @@ 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. +[This is wrong, because if we don't fill the whole temp table, there may + be some tuple(s) that would match some outer tuple. In such cases, if we + stop filling the temp table, we would miss a TRUE result. Having a partial + match doesn't preclude us from having a complete match]. 8. [timour] Consider that due to materialization, we already have a unique index -=-=(Timour - Tue, 19 Jan 2010, 18:44)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.22569 2010-01-19 18:44:01.000000000 +0200 +++ /tmp/wklog.68.new.22569 2010-01-19 18:44:01.000000000 +0200 @@ -132,11 +132,10 @@ if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE - if (nonull_key) - pq.insert(nonull_key) for (i = 1; i <= n; i++) { + if (vkey[i] != nonull_key) vkey[i].lookup(outer_ref) if (! vkey[i].is_eof()) pq.insert(i) @@ -167,7 +166,7 @@ /* There cannot be a complete match, as we already checked for one. */ assert(matching_keys.elements < n) } - else if (cur_min_key == nonull_key) + else if (vkey[cur_min_key] == nonull_key) { /* The non-NULL key has no corresponding NULL index, so we know for @@ -183,8 +182,10 @@ /* Check if all null_keys contain a NULL at row 'min_row'. The procedure internally checks all keys in a special precomputed order. A prior - procedure determines an optimal order and a mapping - idx_no -> idx_order (encoded as an array). + procedure determines an optimal order and a mapping idx_no -> idx_order + (encoded as an array). + + This procedure makes sure not to match the non-NULL column. */ if (test_null_row(null_keys, min_row)) return TRUE @@ -198,6 +199,14 @@ vkey[cur_min_key].next() if (! vkey[cur_min_key].is_eof()) pq.insert(cur_min_key) + else if (vkey[cur_min_key] == nonull_key) + { + /* + If there can't be more matches for the nonull_key, we know for sure + there is no match, since there is no possible NULL match. + */ + return FALSE + } if (pq.is_empty()) { @@ -216,7 +225,6 @@ } - 3. Directions for improvement ======================================================================== -=-=(Timour - Tue, 19 Jan 2010, 18:29)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.21045 2010-01-19 18:29:12.000000000 +0200 +++ /tmp/wklog.68.new.21045 2010-01-19 18:29:12.000000000 +0200 @@ -132,6 +132,8 @@ if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE + if (nonull_key) + pq.insert(nonull_key) for (i = 1; i <= n; i++) { -=-=(Guest - Tue, 19 Jan 2010, 18:15)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.19825 2010-01-19 18:15:30.000000000 +0200 +++ /tmp/wklog.68.new.19825 2010-01-19 18:15:30.000000000 +0200 @@ -1,8 +1,16 @@ -This a copy of the initial algorithm proposed by Igor: -====================================================== +Contents +======================================================================== -For each left side tuple (v_1,...,v_n) we have to find the following set -of rowids for the temp table containing N rows as the result of +1. Initial idea as proposed by Igor +2. Algorithm for IN execution with partial matching +3. Directions for improvement + + +1. Initial idea as proposed by Igor +======================================================================== + +For each left side tuple (v_1,...,v_n) we have to find the following +set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs @@ -18,38 +26,198 @@ - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap -Taken all above into account I could suggest the following algorithm to -build R: +Taken all above into account I could suggest the following algorithm +to build R: - Using indexes (read about them below) for each column participating in the - intersection, - merge ordered sets rowid{a_i=v_i} in the following manner. + Using indexes (read about them below) for each column participating + in the intersection, merge ordered sets rowid{a_i=v_i} in the + following manner. If a rowid r has been encountered maximum in k sets -rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), + rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is -not in {i1,...,ik}. + not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. -Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i -is null} is either +Here we use the property (1): +any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can -infer that for any r from R -indexes a_i can be uniquely divided into two groups: one contains -indexes a_i where r belongs to -the sets rowid{a_i=v_i}, the other contains indexes a_j such that r -belongs to rowid{a_j is null}. - -Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order -needed for the merge procedure. We could use BTREE indexes for temp -table. But they are rather expensive and -take a lot of memory as the are implemented with RB trees. +infer that for any r from R indexes a_i can be uniquely divided into +two groups: +- one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, +- the other contains indexes a_j such that r belongs to + rowid{a_j is null}. + +Now let's talk how to get elements from rowid{a_i=v_i} in a sorted +order needed for the merge procedure. We could use BTREE indexes for +temp table. But they are rather expensive and take a lot of memory as +the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. +2. Algorithm for IN execution with partial matching +======================================================================== + +2.1 Below is shown the top-level algorithm to execute an IN predicate +with partial matching. This algorithm is essentially the implementation +of Item_subselect:exec(). + +int lookup_with_null_semantics(outer_ref[], mat_subquery) +{ + if (index_lookup(outer_ref, mat_subquery) + return TRUE + else + { + /* + Check if there is a partial match (UNKNOWN) or no match (NULL). + */ + if (this is the first partial match) + { + vkey[] = build array of value keys for each NULL-able column + of mat_subquery. + nkey[] = build a bitmap NULL index for each column of mat_subquery + that contains NULLs + nonull_key = build a key over all non-NULL columns of mat_subquery + } + if (partial_match(outer_ref, vkey[], nkey[], nonull_key) + return UNKNOWN + else + return FALSE + } +} + +2.2 The implementation of partial matching is as follows + +/* + Assumptions: + - It has already been checked if there is a complete match by a + regular index lookup, and the test failed. + - It has already been checked if there is a complete NULL row, + and if there was we wouldn't call this function. Thus we assume + that there is no complete NULL row. + - Not all vidx_i are empty, but some can be empty. If all were empty, + then the only possibility for a match is a complete NULL row, which + we already checked. + + @param outer_ref - the uter (left) IN argument. + @param vidx[] - array of value keys + Ordered sequences of rowids of the corresponding columns a_i, such + that all rowids in idx_i are the ones where column a_i contains some + value or NULL. Each idx_i is derived dynamically, for each different + left argument of an IN predicate. + @param nidx[] - array of NULL keys + Bitmpas, one per each column, where a bit is set if the corresponding + row has a NULL value for the corresponding column. + @nonull_key - the only key over all columns of the materialized subquery + that do not contain NULLs + + @returns + @retval FALSE if there is no match + @retval TRUE if there is a partial match +*/ + +Boolean partial_match(outer_ref, vkey[], nkey[], nonull_key) +{ + /* Set of the keys (columns) that form a partial match. */ + Set matching_keys = {} + /* A subset of all keys that need to be checked for NULL matches. */ + Set null_keys = {} + Int min_key /* Key that contains the current minimum position. */ + Int min_row /* Current row number of min_key. */ + Int cur_min_key, cur_min_row + PriorityQueue pq + + if (nonull_key && ! nonull_key->lookup(outer_ref)) + return FALSE + + for (i = 1; i <= n; i++) + { + vkey[i].lookup(outer_ref) + if (! vkey[i].is_eof()) + pq.insert(i) + } + /* + Not all value keys are empty, thus we don't have only NULL + keys. If we had, the only possible match is a NULL row, and + we cheked there is no such row, therefore the result is known + to be FALSE. + In fact this algorithm makes sense for at least two non-NULL + columns. + */ + assert(pq.elements > 1) + + (min_key, min_row) = pq.pop() + matching_keys.add(min_key) + vkey[min_key].next() + if (! vkey[min_key].is_eof()) + pq.insert(min_key) + + while (TRUE) + { + (cur_min_key, cur_min_row) = pq.pop() + + if (cur_min_row == min_row) + { + matching_keys.add(cur_min_key) + /* There cannot be a complete match, as we already checked for one. */ + assert(matching_keys.elements < n) + } + else if (cur_min_key == nonull_key) + { + /* + The non-NULL key has no corresponding NULL index, so we know for + sure that the row 'min_row' is not a match. + */ + (min_key, min_row) = (cur_min_key, cur_min_row) + matching_keys = {min_key} + } + else + { + assert(cur_min_row > min_row) /* Follows from the use of PQ. */ + null_keys = set_difference(all keys vkey[], matching_keys) + /* + Check if all null_keys contain a NULL at row 'min_row'. The procedure + internally checks all keys in a special precomputed order. A prior + procedure determines an optimal order and a mapping + idx_no -> idx_order (encoded as an array). + */ + if (test_null_row(null_keys, min_row)) + return TRUE + else + { + (min_key, min_row) = (cur_min_key, cur_min_row) + matching_keys = {min_key} + } + } + + vkey[cur_min_key].next() + if (! vkey[cur_min_key].is_eof()) + pq.insert(cur_min_key) + + if (pq.is_empty()) + { + /* Check the last row of the last column in PQ for NULL matches. */ + null_keys = set_difference(all keys vkey[], matching_keys) + if (test_null_row(null_keys, min_row)) + return TRUE + else + return FALSE + } + } + + /* We should never get here. */ + assert(FALSE) + return FALSE +} + + + +3. Directions for improvement +======================================================================== + Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary -=-=(Timour - Sun, 06 Dec 2009, 14:36)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.12919 2009-12-06 14:36:18.000000000 +0200 +++ /tmp/wklog.68.new.12919 2009-12-06 14:36:18.000000000 +0200 @@ -87,3 +87,8 @@ 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. +8. [timour] + Consider that due to materialization, we already have a unique index +on all columns <a_1,..., a_n>. We can use the first key part of this index +over column a_1, instead of the index rowid{a_i=v_i}. Thus we can avoid +creating the index rowid{a_i=v_i}. -=-=(Timour - Fri, 04 Dec 2009, 14:04)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.16724 2009-12-04 14:04:28.000000000 +0200 +++ /tmp/wklog.68.new.16724 2009-12-04 14:04:28.000000000 +0200 @@ -10,7 +10,8 @@ Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint - (2) For each i: rowid{a_i is null} is the same for each tuple + (2) For each i: rowid{a_i is null} is the same for each tuple, + that is, this set is independent of the left-side tuples. Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Version updated. --- /tmp/wklog.68.old.5257 2009-12-04 11:27:11.000000000 +0200 +++ /tmp/wklog.68.new.5257 2009-12-04 11:27:11.000000000 +0200 @@ -1 +1 @@ -Benchmarks-3.0 +Server-9.x -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Category updated. --- /tmp/wklog.68.old.5242 2009-12-04 11:27:02.000000000 +0200 +++ /tmp/wklog.68.new.5242 2009-12-04 11:27:02.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint ------------------------------------------------------------ -=-=(View All Progress Notes, 13 total)=-=- http://askmonty.org/worklog/index.pl?tid=68&nolimit=1 DESCRIPTION: The goal of this task is to implement efficient execution of NOT IN subquery predicates of the form: <oe_1,...,oe_n> NOT IN <non_correlated subquery> when either some oe_i, or some subqury result column contains NULLs. The problem with such predicates is that it is possible to use index lookups only when neither argument of the predicate contains NULLs. If some argument contains a NULL, then due to NULL semantics, it plays the role of a wildcard. If we were to use regular index lookups, then we would get 'no match' for some outer tuple (thus the predicate evaluates to FALSE), while the SQL semantics means 'partial match', and the predicate should evaluate to NULL. This task implements an efficient algorithm to compute such 'parial matches', where a NULL matches any value. HIGH-LEVEL SPECIFICATION: Contents ======================================================================== 1. Initial idea as proposed by Igor 2. Algorithm for IN execution with partial matching 3. Directions for improvement 1. Initial idea as proposed by Igor ======================================================================== For each left side tuple (v_1,...,v_n) we have to find the following set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs trough all indexes from [1..n] such that v_i is not null. Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint (2) For each i: rowid{a_i is null} is the same for each tuple, that is, this set is independent of the left-side tuples. Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap Taken all above into account I could suggest the following algorithm to build R: Using indexes (read about them below) for each column participating in the intersection, merge ordered sets rowid{a_i=v_i} in the following manner. If a rowid r has been encountered maximum in k sets rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can infer that for any r from R indexes a_i can be uniquely divided into two groups: - one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, - the other contains indexes a_j such that r belongs to rowid{a_j is null}. Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order needed for the merge procedure. We could use BTREE indexes for temp table. But they are rather expensive and take a lot of memory as the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. 2. Algorithm for IN execution with partial matching ======================================================================== 2.1 Below is shown the top-level algorithm to execute an IN predicate with partial matching. This algorithm is essentially the implementation of Item_subselect:exec(). int lookup_with_null_semantics(outer_ref[], mat_subquery) { if (index_lookup(outer_ref, mat_subquery) return TRUE else { /* Check if there is a partial match (UNKNOWN) or no match (NULL). */ if (this is the first partial match) { vkey[] = build array of value keys for each NULL-able column of mat_subquery. nkey[] = build a bitmap NULL index for each column of mat_subquery that contains NULLs nonull_key = build a key over all non-NULL columns of mat_subquery } if (partial_match(outer_ref, vkey[], nkey[], nonull_key) return UNKNOWN else return FALSE } } 2.2 The implementation of partial matching is as follows /* Assumptions: - It has already been checked if there is a complete match by a regular index lookup, and the test failed. - It has already been checked if there is a complete NULL row, and if there was we wouldn't call this function. Thus we assume that there is no complete NULL row. - Not all vidx_i are empty, but some can be empty. If all were empty, then the only possibility for a match is a complete NULL row, which we already checked. @param outer_ref - the uter (left) IN argument. @param vidx[] - array of value keys Ordered sequences of rowids of the corresponding columns a_i, such that all rowids in idx_i are the ones where column a_i contains some value or NULL. Each idx_i is derived dynamically, for each different left argument of an IN predicate. @param nidx[] - array of NULL keys Bitmpas, one per each column, where a bit is set if the corresponding row has a NULL value for the corresponding column. @nonull_key - the only key over all columns of the materialized subquery that do not contain NULLs @returns @retval FALSE if there is no match @retval TRUE if there is a partial match */ Boolean partial_match(outer_ref, vkey[], nkey[], nonull_key) { /* Set of the keys (columns) that form a partial match. */ Set matching_keys = {} /* A subset of all keys that need to be checked for NULL matches. */ Set null_keys = {} Int min_key /* Key that contains the current minimum position. */ Int min_row /* Current row number of min_key. */ Int cur_min_key, cur_min_row PriorityQueue pq if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE for (i = 1; i <= n; i++) { if (vkey[i] != nonull_key) vkey[i].lookup(outer_ref) if (! vkey[i].is_eof()) pq.insert(i) } /* Not all value keys are empty, thus we don't have only NULL keys. If we had, the only possible match is a NULL row, and we cheked there is no such row, therefore the result is known to be FALSE. In fact this algorithm makes sense for at least two non-NULL columns. */ assert(pq.elements > 1) (min_key, min_row) = pq.pop() matching_keys.add(min_key) vkey[min_key].next() if (! vkey[min_key].is_eof()) pq.insert(min_key) while (TRUE) { (cur_min_key, cur_min_row) = pq.pop() if (cur_min_row == min_row) { matching_keys.add(cur_min_key) /* There cannot be a complete match, as we already checked for one. */ assert(matching_keys.elements < n) } else if (vkey[cur_min_key] == nonull_key) { /* The non-NULL key has no corresponding NULL index, so we know for sure that the row 'min_row' is not a match. */ (min_key, min_row) = (cur_min_key, cur_min_row) matching_keys = {min_key} } else { assert(cur_min_row > min_row) /* Follows from the use of PQ. */ null_keys = set_difference(all keys vkey[], matching_keys) /* Check if all null_keys contain a NULL at row 'min_row'. The procedure internally checks all keys in a special precomputed order. A prior procedure determines an optimal order and a mapping idx_no -> idx_order (encoded as an array). This procedure makes sure not to match the non-NULL column. */ if (test_null_row(null_keys, min_row)) return TRUE else { (min_key, min_row) = (cur_min_key, cur_min_row) matching_keys = {min_key} } } vkey[cur_min_key].next() if (! vkey[cur_min_key].is_eof()) pq.insert(cur_min_key) else if (vkey[cur_min_key] == nonull_key) { /* If there can't be more matches for the nonull_key, we know for sure there is no match, since there is no possible NULL match. */ return FALSE } if (pq.is_empty()) { /* Check the last row of the last column in PQ for NULL matches. */ null_keys = set_difference(all keys vkey[], matching_keys) if (test_null_row(null_keys, min_row)) return TRUE else return FALSE } } /* We should never get here. */ assert(FALSE) return FALSE } 3. Directions for improvement ======================================================================== Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all and v_j1,...,v_jm cannot be null, create for these columns only one index array (and of course do not create any bitmaps for them). [done] 2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number of rows, where a_i is not null and V(a_i) is the number of distinct values for a_i excluding nulls. If d(a_i) is close to N'(a_i) then do not create any index array: check whether there is a match running through the records that have been filtered in. Anyway if d(a_i) is close to N'(a_i) then the intersection with rowid{a_i=v_i} will not reduce the number of remaining rowids significantly. In other words is V(a_i) exceeds some threshold there is no sense to create an index for a_i. If additionally N-N'(a_i) is small do not create a bitmap for this column either. 3. If for a column a_i d(a_i) is not close to N'(a_i), but N-N'(a_i) is small a sorted array of rowids from the set rowid{a_i is null} can be used instead of a bitmap. 4. We always have a match if R0= INTERSECT rowid{a_i is null} is not empty. Here i runs through all indexes from [1..n] such that v_i is not null. For a given subset of columns this fact has to be checked only once. It can be easily done with bitmap intersection. 5. If v1,...,vn never can be a null, then indexes (sorted arrays) can be created only for rows with nulls. 6. If v1,...,vn never can be a null and number of rows with nulls is small do not create indexes and do not create bitmaps. 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. [This is wrong, because if we don't fill the whole temp table, there may be some tuple(s) that would match some outer tuple. In such cases, if we stop filling the temp table, we would miss a TRUE result. Having a partial match doesn't preclude us from having a complete match]. 8. [timour] Consider that due to materialization, we already have a unique index on all columns <a_1,..., a_n>. We can use the first key part of this index over column a_1, instead of the index rowid{a_i=v_i}. Thus we can avoid creating the index rowid{a_i=v_i}. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Guest): Subquery optimization: Efficient NOT IN execution with NULLs (68)
by worklog-noreply＠askmonty.org 27 Feb '10

27 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subquery optimization: Efficient NOT IN execution with NULLs CREATION DATE..: Fri, 27 Nov 2009, 13:22 SUPERVISOR.....: Monty IMPLEMENTOR....: Timour COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 68 (http://askmonty.org/worklog/?tid=68) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Guest - Sat, 27 Feb 2010, 10:11)=-=- Status updated. No change. -=-=(Guest - Sat, 27 Feb 2010, 10:11)=-=- Status updated. --- /tmp/wklog.68.old.24229 2010-02-27 10:11:57.000000000 +0000 +++ /tmp/wklog.68.new.24229 2010-02-27 10:11:57.000000000 +0000 @@ -1 +1 @@ -Assigned +In-Progress -=-=(Timour - Mon, 22 Feb 2010, 17:39)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.17116 2010-02-22 17:39:48.000000000 +0200 +++ /tmp/wklog.68.new.17116 2010-02-22 17:39:48.000000000 +0200 @@ -233,6 +233,7 @@ 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all and v_j1,...,v_jm cannot be null, create for these columns only one index array (and of course do not create any bitmaps for them). +[done] 2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number of rows, where a_i is not null and V(a_i) is the number of distinct @@ -264,6 +265,10 @@ 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. +[This is wrong, because if we don't fill the whole temp table, there may + be some tuple(s) that would match some outer tuple. In such cases, if we + stop filling the temp table, we would miss a TRUE result. Having a partial + match doesn't preclude us from having a complete match]. 8. [timour] Consider that due to materialization, we already have a unique index -=-=(Timour - Tue, 19 Jan 2010, 18:44)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.22569 2010-01-19 18:44:01.000000000 +0200 +++ /tmp/wklog.68.new.22569 2010-01-19 18:44:01.000000000 +0200 @@ -132,11 +132,10 @@ if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE - if (nonull_key) - pq.insert(nonull_key) for (i = 1; i <= n; i++) { + if (vkey[i] != nonull_key) vkey[i].lookup(outer_ref) if (! vkey[i].is_eof()) pq.insert(i) @@ -167,7 +166,7 @@ /* There cannot be a complete match, as we already checked for one. */ assert(matching_keys.elements < n) } - else if (cur_min_key == nonull_key) + else if (vkey[cur_min_key] == nonull_key) { /* The non-NULL key has no corresponding NULL index, so we know for @@ -183,8 +182,10 @@ /* Check if all null_keys contain a NULL at row 'min_row'. The procedure internally checks all keys in a special precomputed order. A prior - procedure determines an optimal order and a mapping - idx_no -> idx_order (encoded as an array). + procedure determines an optimal order and a mapping idx_no -> idx_order + (encoded as an array). + + This procedure makes sure not to match the non-NULL column. */ if (test_null_row(null_keys, min_row)) return TRUE @@ -198,6 +199,14 @@ vkey[cur_min_key].next() if (! vkey[cur_min_key].is_eof()) pq.insert(cur_min_key) + else if (vkey[cur_min_key] == nonull_key) + { + /* + If there can't be more matches for the nonull_key, we know for sure + there is no match, since there is no possible NULL match. + */ + return FALSE + } if (pq.is_empty()) { @@ -216,7 +225,6 @@ } - 3. Directions for improvement ======================================================================== -=-=(Timour - Tue, 19 Jan 2010, 18:29)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.21045 2010-01-19 18:29:12.000000000 +0200 +++ /tmp/wklog.68.new.21045 2010-01-19 18:29:12.000000000 +0200 @@ -132,6 +132,8 @@ if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE + if (nonull_key) + pq.insert(nonull_key) for (i = 1; i <= n; i++) { -=-=(Guest - Tue, 19 Jan 2010, 18:15)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.19825 2010-01-19 18:15:30.000000000 +0200 +++ /tmp/wklog.68.new.19825 2010-01-19 18:15:30.000000000 +0200 @@ -1,8 +1,16 @@ -This a copy of the initial algorithm proposed by Igor: -====================================================== +Contents +======================================================================== -For each left side tuple (v_1,...,v_n) we have to find the following set -of rowids for the temp table containing N rows as the result of +1. Initial idea as proposed by Igor +2. Algorithm for IN execution with partial matching +3. Directions for improvement + + +1. Initial idea as proposed by Igor +======================================================================== + +For each left side tuple (v_1,...,v_n) we have to find the following +set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs @@ -18,38 +26,198 @@ - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap -Taken all above into account I could suggest the following algorithm to -build R: +Taken all above into account I could suggest the following algorithm +to build R: - Using indexes (read about them below) for each column participating in the - intersection, - merge ordered sets rowid{a_i=v_i} in the following manner. + Using indexes (read about them below) for each column participating + in the intersection, merge ordered sets rowid{a_i=v_i} in the + following manner. If a rowid r has been encountered maximum in k sets -rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), + rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is -not in {i1,...,ik}. + not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. -Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i -is null} is either +Here we use the property (1): +any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can -infer that for any r from R -indexes a_i can be uniquely divided into two groups: one contains -indexes a_i where r belongs to -the sets rowid{a_i=v_i}, the other contains indexes a_j such that r -belongs to rowid{a_j is null}. - -Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order -needed for the merge procedure. We could use BTREE indexes for temp -table. But they are rather expensive and -take a lot of memory as the are implemented with RB trees. +infer that for any r from R indexes a_i can be uniquely divided into +two groups: +- one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, +- the other contains indexes a_j such that r belongs to + rowid{a_j is null}. + +Now let's talk how to get elements from rowid{a_i=v_i} in a sorted +order needed for the merge procedure. We could use BTREE indexes for +temp table. But they are rather expensive and take a lot of memory as +the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. +2. Algorithm for IN execution with partial matching +======================================================================== + +2.1 Below is shown the top-level algorithm to execute an IN predicate +with partial matching. This algorithm is essentially the implementation +of Item_subselect:exec(). + +int lookup_with_null_semantics(outer_ref[], mat_subquery) +{ + if (index_lookup(outer_ref, mat_subquery) + return TRUE + else + { + /* + Check if there is a partial match (UNKNOWN) or no match (NULL). + */ + if (this is the first partial match) + { + vkey[] = build array of value keys for each NULL-able column + of mat_subquery. + nkey[] = build a bitmap NULL index for each column of mat_subquery + that contains NULLs + nonull_key = build a key over all non-NULL columns of mat_subquery + } + if (partial_match(outer_ref, vkey[], nkey[], nonull_key) + return UNKNOWN + else + return FALSE + } +} + +2.2 The implementation of partial matching is as follows + +/* + Assumptions: + - It has already been checked if there is a complete match by a + regular index lookup, and the test failed. + - It has already been checked if there is a complete NULL row, + and if there was we wouldn't call this function. Thus we assume + that there is no complete NULL row. + - Not all vidx_i are empty, but some can be empty. If all were empty, + then the only possibility for a match is a complete NULL row, which + we already checked. + + @param outer_ref - the uter (left) IN argument. + @param vidx[] - array of value keys + Ordered sequences of rowids of the corresponding columns a_i, such + that all rowids in idx_i are the ones where column a_i contains some + value or NULL. Each idx_i is derived dynamically, for each different + left argument of an IN predicate. + @param nidx[] - array of NULL keys + Bitmpas, one per each column, where a bit is set if the corresponding + row has a NULL value for the corresponding column. + @nonull_key - the only key over all columns of the materialized subquery + that do not contain NULLs + + @returns + @retval FALSE if there is no match + @retval TRUE if there is a partial match +*/ + +Boolean partial_match(outer_ref, vkey[], nkey[], nonull_key) +{ + /* Set of the keys (columns) that form a partial match. */ + Set matching_keys = {} + /* A subset of all keys that need to be checked for NULL matches. */ + Set null_keys = {} + Int min_key /* Key that contains the current minimum position. */ + Int min_row /* Current row number of min_key. */ + Int cur_min_key, cur_min_row + PriorityQueue pq + + if (nonull_key && ! nonull_key->lookup(outer_ref)) + return FALSE + + for (i = 1; i <= n; i++) + { + vkey[i].lookup(outer_ref) + if (! vkey[i].is_eof()) + pq.insert(i) + } + /* + Not all value keys are empty, thus we don't have only NULL + keys. If we had, the only possible match is a NULL row, and + we cheked there is no such row, therefore the result is known + to be FALSE. + In fact this algorithm makes sense for at least two non-NULL + columns. + */ + assert(pq.elements > 1) + + (min_key, min_row) = pq.pop() + matching_keys.add(min_key) + vkey[min_key].next() + if (! vkey[min_key].is_eof()) + pq.insert(min_key) + + while (TRUE) + { + (cur_min_key, cur_min_row) = pq.pop() + + if (cur_min_row == min_row) + { + matching_keys.add(cur_min_key) + /* There cannot be a complete match, as we already checked for one. */ + assert(matching_keys.elements < n) + } + else if (cur_min_key == nonull_key) + { + /* + The non-NULL key has no corresponding NULL index, so we know for + sure that the row 'min_row' is not a match. + */ + (min_key, min_row) = (cur_min_key, cur_min_row) + matching_keys = {min_key} + } + else + { + assert(cur_min_row > min_row) /* Follows from the use of PQ. */ + null_keys = set_difference(all keys vkey[], matching_keys) + /* + Check if all null_keys contain a NULL at row 'min_row'. The procedure + internally checks all keys in a special precomputed order. A prior + procedure determines an optimal order and a mapping + idx_no -> idx_order (encoded as an array). + */ + if (test_null_row(null_keys, min_row)) + return TRUE + else + { + (min_key, min_row) = (cur_min_key, cur_min_row) + matching_keys = {min_key} + } + } + + vkey[cur_min_key].next() + if (! vkey[cur_min_key].is_eof()) + pq.insert(cur_min_key) + + if (pq.is_empty()) + { + /* Check the last row of the last column in PQ for NULL matches. */ + null_keys = set_difference(all keys vkey[], matching_keys) + if (test_null_row(null_keys, min_row)) + return TRUE + else + return FALSE + } + } + + /* We should never get here. */ + assert(FALSE) + return FALSE +} + + + +3. Directions for improvement +======================================================================== + Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary -=-=(Timour - Sun, 06 Dec 2009, 14:36)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.12919 2009-12-06 14:36:18.000000000 +0200 +++ /tmp/wklog.68.new.12919 2009-12-06 14:36:18.000000000 +0200 @@ -87,3 +87,8 @@ 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. +8. [timour] + Consider that due to materialization, we already have a unique index +on all columns <a_1,..., a_n>. We can use the first key part of this index +over column a_1, instead of the index rowid{a_i=v_i}. Thus we can avoid +creating the index rowid{a_i=v_i}. -=-=(Timour - Fri, 04 Dec 2009, 14:04)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.16724 2009-12-04 14:04:28.000000000 +0200 +++ /tmp/wklog.68.new.16724 2009-12-04 14:04:28.000000000 +0200 @@ -10,7 +10,8 @@ Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint - (2) For each i: rowid{a_i is null} is the same for each tuple + (2) For each i: rowid{a_i is null} is the same for each tuple, + that is, this set is independent of the left-side tuples. Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Version updated. --- /tmp/wklog.68.old.5257 2009-12-04 11:27:11.000000000 +0200 +++ /tmp/wklog.68.new.5257 2009-12-04 11:27:11.000000000 +0200 @@ -1 +1 @@ -Benchmarks-3.0 +Server-9.x -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Category updated. --- /tmp/wklog.68.old.5242 2009-12-04 11:27:02.000000000 +0200 +++ /tmp/wklog.68.new.5242 2009-12-04 11:27:02.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint ------------------------------------------------------------ -=-=(View All Progress Notes, 13 total)=-=- http://askmonty.org/worklog/index.pl?tid=68&nolimit=1 DESCRIPTION: The goal of this task is to implement efficient execution of NOT IN subquery predicates of the form: <oe_1,...,oe_n> NOT IN <non_correlated subquery> when either some oe_i, or some subqury result column contains NULLs. The problem with such predicates is that it is possible to use index lookups only when neither argument of the predicate contains NULLs. If some argument contains a NULL, then due to NULL semantics, it plays the role of a wildcard. If we were to use regular index lookups, then we would get 'no match' for some outer tuple (thus the predicate evaluates to FALSE), while the SQL semantics means 'partial match', and the predicate should evaluate to NULL. This task implements an efficient algorithm to compute such 'parial matches', where a NULL matches any value. HIGH-LEVEL SPECIFICATION: Contents ======================================================================== 1. Initial idea as proposed by Igor 2. Algorithm for IN execution with partial matching 3. Directions for improvement 1. Initial idea as proposed by Igor ======================================================================== For each left side tuple (v_1,...,v_n) we have to find the following set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs trough all indexes from [1..n] such that v_i is not null. Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint (2) For each i: rowid{a_i is null} is the same for each tuple, that is, this set is independent of the left-side tuples. Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap Taken all above into account I could suggest the following algorithm to build R: Using indexes (read about them below) for each column participating in the intersection, merge ordered sets rowid{a_i=v_i} in the following manner. If a rowid r has been encountered maximum in k sets rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can infer that for any r from R indexes a_i can be uniquely divided into two groups: - one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, - the other contains indexes a_j such that r belongs to rowid{a_j is null}. Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order needed for the merge procedure. We could use BTREE indexes for temp table. But they are rather expensive and take a lot of memory as the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. 2. Algorithm for IN execution with partial matching ======================================================================== 2.1 Below is shown the top-level algorithm to execute an IN predicate with partial matching. This algorithm is essentially the implementation of Item_subselect:exec(). int lookup_with_null_semantics(outer_ref[], mat_subquery) { if (index_lookup(outer_ref, mat_subquery) return TRUE else { /* Check if there is a partial match (UNKNOWN) or no match (NULL). */ if (this is the first partial match) { vkey[] = build array of value keys for each NULL-able column of mat_subquery. nkey[] = build a bitmap NULL index for each column of mat_subquery that contains NULLs nonull_key = build a key over all non-NULL columns of mat_subquery } if (partial_match(outer_ref, vkey[], nkey[], nonull_key) return UNKNOWN else return FALSE } } 2.2 The implementation of partial matching is as follows /* Assumptions: - It has already been checked if there is a complete match by a regular index lookup, and the test failed. - It has already been checked if there is a complete NULL row, and if there was we wouldn't call this function. Thus we assume that there is no complete NULL row. - Not all vidx_i are empty, but some can be empty. If all were empty, then the only possibility for a match is a complete NULL row, which we already checked. @param outer_ref - the uter (left) IN argument. @param vidx[] - array of value keys Ordered sequences of rowids of the corresponding columns a_i, such that all rowids in idx_i are the ones where column a_i contains some value or NULL. Each idx_i is derived dynamically, for each different left argument of an IN predicate. @param nidx[] - array of NULL keys Bitmpas, one per each column, where a bit is set if the corresponding row has a NULL value for the corresponding column. @nonull_key - the only key over all columns of the materialized subquery that do not contain NULLs @returns @retval FALSE if there is no match @retval TRUE if there is a partial match */ Boolean partial_match(outer_ref, vkey[], nkey[], nonull_key) { /* Set of the keys (columns) that form a partial match. */ Set matching_keys = {} /* A subset of all keys that need to be checked for NULL matches. */ Set null_keys = {} Int min_key /* Key that contains the current minimum position. */ Int min_row /* Current row number of min_key. */ Int cur_min_key, cur_min_row PriorityQueue pq if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE for (i = 1; i <= n; i++) { if (vkey[i] != nonull_key) vkey[i].lookup(outer_ref) if (! vkey[i].is_eof()) pq.insert(i) } /* Not all value keys are empty, thus we don't have only NULL keys. If we had, the only possible match is a NULL row, and we cheked there is no such row, therefore the result is known to be FALSE. In fact this algorithm makes sense for at least two non-NULL columns. */ assert(pq.elements > 1) (min_key, min_row) = pq.pop() matching_keys.add(min_key) vkey[min_key].next() if (! vkey[min_key].is_eof()) pq.insert(min_key) while (TRUE) { (cur_min_key, cur_min_row) = pq.pop() if (cur_min_row == min_row) { matching_keys.add(cur_min_key) /* There cannot be a complete match, as we already checked for one. */ assert(matching_keys.elements < n) } else if (vkey[cur_min_key] == nonull_key) { /* The non-NULL key has no corresponding NULL index, so we know for sure that the row 'min_row' is not a match. */ (min_key, min_row) = (cur_min_key, cur_min_row) matching_keys = {min_key} } else { assert(cur_min_row > min_row) /* Follows from the use of PQ. */ null_keys = set_difference(all keys vkey[], matching_keys) /* Check if all null_keys contain a NULL at row 'min_row'. The procedure internally checks all keys in a special precomputed order. A prior procedure determines an optimal order and a mapping idx_no -> idx_order (encoded as an array). This procedure makes sure not to match the non-NULL column. */ if (test_null_row(null_keys, min_row)) return TRUE else { (min_key, min_row) = (cur_min_key, cur_min_row) matching_keys = {min_key} } } vkey[cur_min_key].next() if (! vkey[cur_min_key].is_eof()) pq.insert(cur_min_key) else if (vkey[cur_min_key] == nonull_key) { /* If there can't be more matches for the nonull_key, we know for sure there is no match, since there is no possible NULL match. */ return FALSE } if (pq.is_empty()) { /* Check the last row of the last column in PQ for NULL matches. */ null_keys = set_difference(all keys vkey[], matching_keys) if (test_null_row(null_keys, min_row)) return TRUE else return FALSE } } /* We should never get here. */ assert(FALSE) return FALSE } 3. Directions for improvement ======================================================================== Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all and v_j1,...,v_jm cannot be null, create for these columns only one index array (and of course do not create any bitmaps for them). [done] 2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number of rows, where a_i is not null and V(a_i) is the number of distinct values for a_i excluding nulls. If d(a_i) is close to N'(a_i) then do not create any index array: check whether there is a match running through the records that have been filtered in. Anyway if d(a_i) is close to N'(a_i) then the intersection with rowid{a_i=v_i} will not reduce the number of remaining rowids significantly. In other words is V(a_i) exceeds some threshold there is no sense to create an index for a_i. If additionally N-N'(a_i) is small do not create a bitmap for this column either. 3. If for a column a_i d(a_i) is not close to N'(a_i), but N-N'(a_i) is small a sorted array of rowids from the set rowid{a_i is null} can be used instead of a bitmap. 4. We always have a match if R0= INTERSECT rowid{a_i is null} is not empty. Here i runs through all indexes from [1..n] such that v_i is not null. For a given subset of columns this fact has to be checked only once. It can be easily done with bitmap intersection. 5. If v1,...,vn never can be a null, then indexes (sorted arrays) can be created only for rows with nulls. 6. If v1,...,vn never can be a null and number of rows with nulls is small do not create indexes and do not create bitmaps. 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. [This is wrong, because if we don't fill the whole temp table, there may be some tuple(s) that would match some outer tuple. In such cases, if we stop filling the temp table, we would miss a TRUE result. Having a partial match doesn't preclude us from having a complete match]. 8. [timour] Consider that due to materialization, we already have a unique index on all columns <a_1,..., a_n>. We can use the first key part of this index over column a_1, instead of the index rowid{a_i=v_i}. Thus we can avoid creating the index rowid{a_i=v_i}. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Guest): Subquery optimization: Efficient NOT IN execution with NULLs (68)
by worklog-noreply＠askmonty.org 27 Feb '10

27 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subquery optimization: Efficient NOT IN execution with NULLs CREATION DATE..: Fri, 27 Nov 2009, 13:22 SUPERVISOR.....: Monty IMPLEMENTOR....: Timour COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 68 (http://askmonty.org/worklog/?tid=68) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Guest - Sat, 27 Feb 2010, 10:11)=-=- Status updated. --- /tmp/wklog.68.old.24229 2010-02-27 10:11:57.000000000 +0000 +++ /tmp/wklog.68.new.24229 2010-02-27 10:11:57.000000000 +0000 @@ -1 +1 @@ -Assigned +In-Progress -=-=(Timour - Mon, 22 Feb 2010, 17:39)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.17116 2010-02-22 17:39:48.000000000 +0200 +++ /tmp/wklog.68.new.17116 2010-02-22 17:39:48.000000000 +0200 @@ -233,6 +233,7 @@ 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all and v_j1,...,v_jm cannot be null, create for these columns only one index array (and of course do not create any bitmaps for them). +[done] 2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number of rows, where a_i is not null and V(a_i) is the number of distinct @@ -264,6 +265,10 @@ 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. +[This is wrong, because if we don't fill the whole temp table, there may + be some tuple(s) that would match some outer tuple. In such cases, if we + stop filling the temp table, we would miss a TRUE result. Having a partial + match doesn't preclude us from having a complete match]. 8. [timour] Consider that due to materialization, we already have a unique index -=-=(Timour - Tue, 19 Jan 2010, 18:44)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.22569 2010-01-19 18:44:01.000000000 +0200 +++ /tmp/wklog.68.new.22569 2010-01-19 18:44:01.000000000 +0200 @@ -132,11 +132,10 @@ if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE - if (nonull_key) - pq.insert(nonull_key) for (i = 1; i <= n; i++) { + if (vkey[i] != nonull_key) vkey[i].lookup(outer_ref) if (! vkey[i].is_eof()) pq.insert(i) @@ -167,7 +166,7 @@ /* There cannot be a complete match, as we already checked for one. */ assert(matching_keys.elements < n) } - else if (cur_min_key == nonull_key) + else if (vkey[cur_min_key] == nonull_key) { /* The non-NULL key has no corresponding NULL index, so we know for @@ -183,8 +182,10 @@ /* Check if all null_keys contain a NULL at row 'min_row'. The procedure internally checks all keys in a special precomputed order. A prior - procedure determines an optimal order and a mapping - idx_no -> idx_order (encoded as an array). + procedure determines an optimal order and a mapping idx_no -> idx_order + (encoded as an array). + + This procedure makes sure not to match the non-NULL column. */ if (test_null_row(null_keys, min_row)) return TRUE @@ -198,6 +199,14 @@ vkey[cur_min_key].next() if (! vkey[cur_min_key].is_eof()) pq.insert(cur_min_key) + else if (vkey[cur_min_key] == nonull_key) + { + /* + If there can't be more matches for the nonull_key, we know for sure + there is no match, since there is no possible NULL match. + */ + return FALSE + } if (pq.is_empty()) { @@ -216,7 +225,6 @@ } - 3. Directions for improvement ======================================================================== -=-=(Timour - Tue, 19 Jan 2010, 18:29)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.21045 2010-01-19 18:29:12.000000000 +0200 +++ /tmp/wklog.68.new.21045 2010-01-19 18:29:12.000000000 +0200 @@ -132,6 +132,8 @@ if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE + if (nonull_key) + pq.insert(nonull_key) for (i = 1; i <= n; i++) { -=-=(Guest - Tue, 19 Jan 2010, 18:15)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.19825 2010-01-19 18:15:30.000000000 +0200 +++ /tmp/wklog.68.new.19825 2010-01-19 18:15:30.000000000 +0200 @@ -1,8 +1,16 @@ -This a copy of the initial algorithm proposed by Igor: -====================================================== +Contents +======================================================================== -For each left side tuple (v_1,...,v_n) we have to find the following set -of rowids for the temp table containing N rows as the result of +1. Initial idea as proposed by Igor +2. Algorithm for IN execution with partial matching +3. Directions for improvement + + +1. Initial idea as proposed by Igor +======================================================================== + +For each left side tuple (v_1,...,v_n) we have to find the following +set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs @@ -18,38 +26,198 @@ - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap -Taken all above into account I could suggest the following algorithm to -build R: +Taken all above into account I could suggest the following algorithm +to build R: - Using indexes (read about them below) for each column participating in the - intersection, - merge ordered sets rowid{a_i=v_i} in the following manner. + Using indexes (read about them below) for each column participating + in the intersection, merge ordered sets rowid{a_i=v_i} in the + following manner. If a rowid r has been encountered maximum in k sets -rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), + rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is -not in {i1,...,ik}. + not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. -Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i -is null} is either +Here we use the property (1): +any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can -infer that for any r from R -indexes a_i can be uniquely divided into two groups: one contains -indexes a_i where r belongs to -the sets rowid{a_i=v_i}, the other contains indexes a_j such that r -belongs to rowid{a_j is null}. - -Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order -needed for the merge procedure. We could use BTREE indexes for temp -table. But they are rather expensive and -take a lot of memory as the are implemented with RB trees. +infer that for any r from R indexes a_i can be uniquely divided into +two groups: +- one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, +- the other contains indexes a_j such that r belongs to + rowid{a_j is null}. + +Now let's talk how to get elements from rowid{a_i=v_i} in a sorted +order needed for the merge procedure. We could use BTREE indexes for +temp table. But they are rather expensive and take a lot of memory as +the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. +2. Algorithm for IN execution with partial matching +======================================================================== + +2.1 Below is shown the top-level algorithm to execute an IN predicate +with partial matching. This algorithm is essentially the implementation +of Item_subselect:exec(). + +int lookup_with_null_semantics(outer_ref[], mat_subquery) +{ + if (index_lookup(outer_ref, mat_subquery) + return TRUE + else + { + /* + Check if there is a partial match (UNKNOWN) or no match (NULL). + */ + if (this is the first partial match) + { + vkey[] = build array of value keys for each NULL-able column + of mat_subquery. + nkey[] = build a bitmap NULL index for each column of mat_subquery + that contains NULLs + nonull_key = build a key over all non-NULL columns of mat_subquery + } + if (partial_match(outer_ref, vkey[], nkey[], nonull_key) + return UNKNOWN + else + return FALSE + } +} + +2.2 The implementation of partial matching is as follows + +/* + Assumptions: + - It has already been checked if there is a complete match by a + regular index lookup, and the test failed. + - It has already been checked if there is a complete NULL row, + and if there was we wouldn't call this function. Thus we assume + that there is no complete NULL row. + - Not all vidx_i are empty, but some can be empty. If all were empty, + then the only possibility for a match is a complete NULL row, which + we already checked. + + @param outer_ref - the uter (left) IN argument. + @param vidx[] - array of value keys + Ordered sequences of rowids of the corresponding columns a_i, such + that all rowids in idx_i are the ones where column a_i contains some + value or NULL. Each idx_i is derived dynamically, for each different + left argument of an IN predicate. + @param nidx[] - array of NULL keys + Bitmpas, one per each column, where a bit is set if the corresponding + row has a NULL value for the corresponding column. + @nonull_key - the only key over all columns of the materialized subquery + that do not contain NULLs + + @returns + @retval FALSE if there is no match + @retval TRUE if there is a partial match +*/ + +Boolean partial_match(outer_ref, vkey[], nkey[], nonull_key) +{ + /* Set of the keys (columns) that form a partial match. */ + Set matching_keys = {} + /* A subset of all keys that need to be checked for NULL matches. */ + Set null_keys = {} + Int min_key /* Key that contains the current minimum position. */ + Int min_row /* Current row number of min_key. */ + Int cur_min_key, cur_min_row + PriorityQueue pq + + if (nonull_key && ! nonull_key->lookup(outer_ref)) + return FALSE + + for (i = 1; i <= n; i++) + { + vkey[i].lookup(outer_ref) + if (! vkey[i].is_eof()) + pq.insert(i) + } + /* + Not all value keys are empty, thus we don't have only NULL + keys. If we had, the only possible match is a NULL row, and + we cheked there is no such row, therefore the result is known + to be FALSE. + In fact this algorithm makes sense for at least two non-NULL + columns. + */ + assert(pq.elements > 1) + + (min_key, min_row) = pq.pop() + matching_keys.add(min_key) + vkey[min_key].next() + if (! vkey[min_key].is_eof()) + pq.insert(min_key) + + while (TRUE) + { + (cur_min_key, cur_min_row) = pq.pop() + + if (cur_min_row == min_row) + { + matching_keys.add(cur_min_key) + /* There cannot be a complete match, as we already checked for one. */ + assert(matching_keys.elements < n) + } + else if (cur_min_key == nonull_key) + { + /* + The non-NULL key has no corresponding NULL index, so we know for + sure that the row 'min_row' is not a match. + */ + (min_key, min_row) = (cur_min_key, cur_min_row) + matching_keys = {min_key} + } + else + { + assert(cur_min_row > min_row) /* Follows from the use of PQ. */ + null_keys = set_difference(all keys vkey[], matching_keys) + /* + Check if all null_keys contain a NULL at row 'min_row'. The procedure + internally checks all keys in a special precomputed order. A prior + procedure determines an optimal order and a mapping + idx_no -> idx_order (encoded as an array). + */ + if (test_null_row(null_keys, min_row)) + return TRUE + else + { + (min_key, min_row) = (cur_min_key, cur_min_row) + matching_keys = {min_key} + } + } + + vkey[cur_min_key].next() + if (! vkey[cur_min_key].is_eof()) + pq.insert(cur_min_key) + + if (pq.is_empty()) + { + /* Check the last row of the last column in PQ for NULL matches. */ + null_keys = set_difference(all keys vkey[], matching_keys) + if (test_null_row(null_keys, min_row)) + return TRUE + else + return FALSE + } + } + + /* We should never get here. */ + assert(FALSE) + return FALSE +} + + + +3. Directions for improvement +======================================================================== + Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary -=-=(Timour - Sun, 06 Dec 2009, 14:36)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.12919 2009-12-06 14:36:18.000000000 +0200 +++ /tmp/wklog.68.new.12919 2009-12-06 14:36:18.000000000 +0200 @@ -87,3 +87,8 @@ 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. +8. [timour] + Consider that due to materialization, we already have a unique index +on all columns <a_1,..., a_n>. We can use the first key part of this index +over column a_1, instead of the index rowid{a_i=v_i}. Thus we can avoid +creating the index rowid{a_i=v_i}. -=-=(Timour - Fri, 04 Dec 2009, 14:04)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.16724 2009-12-04 14:04:28.000000000 +0200 +++ /tmp/wklog.68.new.16724 2009-12-04 14:04:28.000000000 +0200 @@ -10,7 +10,8 @@ Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint - (2) For each i: rowid{a_i is null} is the same for each tuple + (2) For each i: rowid{a_i is null} is the same for each tuple, + that is, this set is independent of the left-side tuples. Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Version updated. --- /tmp/wklog.68.old.5257 2009-12-04 11:27:11.000000000 +0200 +++ /tmp/wklog.68.new.5257 2009-12-04 11:27:11.000000000 +0200 @@ -1 +1 @@ -Benchmarks-3.0 +Server-9.x -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Category updated. --- /tmp/wklog.68.old.5242 2009-12-04 11:27:02.000000000 +0200 +++ /tmp/wklog.68.new.5242 2009-12-04 11:27:02.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Status updated. --- /tmp/wklog.68.old.5242 2009-12-04 11:27:02.000000000 +0200 +++ /tmp/wklog.68.new.5242 2009-12-04 11:27:02.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +Assigned ------------------------------------------------------------ -=-=(View All Progress Notes, 12 total)=-=- http://askmonty.org/worklog/index.pl?tid=68&nolimit=1 DESCRIPTION: The goal of this task is to implement efficient execution of NOT IN subquery predicates of the form: <oe_1,...,oe_n> NOT IN <non_correlated subquery> when either some oe_i, or some subqury result column contains NULLs. The problem with such predicates is that it is possible to use index lookups only when neither argument of the predicate contains NULLs. If some argument contains a NULL, then due to NULL semantics, it plays the role of a wildcard. If we were to use regular index lookups, then we would get 'no match' for some outer tuple (thus the predicate evaluates to FALSE), while the SQL semantics means 'partial match', and the predicate should evaluate to NULL. This task implements an efficient algorithm to compute such 'parial matches', where a NULL matches any value. HIGH-LEVEL SPECIFICATION: Contents ======================================================================== 1. Initial idea as proposed by Igor 2. Algorithm for IN execution with partial matching 3. Directions for improvement 1. Initial idea as proposed by Igor ======================================================================== For each left side tuple (v_1,...,v_n) we have to find the following set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs trough all indexes from [1..n] such that v_i is not null. Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint (2) For each i: rowid{a_i is null} is the same for each tuple, that is, this set is independent of the left-side tuples. Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap Taken all above into account I could suggest the following algorithm to build R: Using indexes (read about them below) for each column participating in the intersection, merge ordered sets rowid{a_i=v_i} in the following manner. If a rowid r has been encountered maximum in k sets rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can infer that for any r from R indexes a_i can be uniquely divided into two groups: - one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, - the other contains indexes a_j such that r belongs to rowid{a_j is null}. Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order needed for the merge procedure. We could use BTREE indexes for temp table. But they are rather expensive and take a lot of memory as the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. 2. Algorithm for IN execution with partial matching ======================================================================== 2.1 Below is shown the top-level algorithm to execute an IN predicate with partial matching. This algorithm is essentially the implementation of Item_subselect:exec(). int lookup_with_null_semantics(outer_ref[], mat_subquery) { if (index_lookup(outer_ref, mat_subquery) return TRUE else { /* Check if there is a partial match (UNKNOWN) or no match (NULL). */ if (this is the first partial match) { vkey[] = build array of value keys for each NULL-able column of mat_subquery. nkey[] = build a bitmap NULL index for each column of mat_subquery that contains NULLs nonull_key = build a key over all non-NULL columns of mat_subquery } if (partial_match(outer_ref, vkey[], nkey[], nonull_key) return UNKNOWN else return FALSE } } 2.2 The implementation of partial matching is as follows /* Assumptions: - It has already been checked if there is a complete match by a regular index lookup, and the test failed. - It has already been checked if there is a complete NULL row, and if there was we wouldn't call this function. Thus we assume that there is no complete NULL row. - Not all vidx_i are empty, but some can be empty. If all were empty, then the only possibility for a match is a complete NULL row, which we already checked. @param outer_ref - the uter (left) IN argument. @param vidx[] - array of value keys Ordered sequences of rowids of the corresponding columns a_i, such that all rowids in idx_i are the ones where column a_i contains some value or NULL. Each idx_i is derived dynamically, for each different left argument of an IN predicate. @param nidx[] - array of NULL keys Bitmpas, one per each column, where a bit is set if the corresponding row has a NULL value for the corresponding column. @nonull_key - the only key over all columns of the materialized subquery that do not contain NULLs @returns @retval FALSE if there is no match @retval TRUE if there is a partial match */ Boolean partial_match(outer_ref, vkey[], nkey[], nonull_key) { /* Set of the keys (columns) that form a partial match. */ Set matching_keys = {} /* A subset of all keys that need to be checked for NULL matches. */ Set null_keys = {} Int min_key /* Key that contains the current minimum position. */ Int min_row /* Current row number of min_key. */ Int cur_min_key, cur_min_row PriorityQueue pq if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE for (i = 1; i <= n; i++) { if (vkey[i] != nonull_key) vkey[i].lookup(outer_ref) if (! vkey[i].is_eof()) pq.insert(i) } /* Not all value keys are empty, thus we don't have only NULL keys. If we had, the only possible match is a NULL row, and we cheked there is no such row, therefore the result is known to be FALSE. In fact this algorithm makes sense for at least two non-NULL columns. */ assert(pq.elements > 1) (min_key, min_row) = pq.pop() matching_keys.add(min_key) vkey[min_key].next() if (! vkey[min_key].is_eof()) pq.insert(min_key) while (TRUE) { (cur_min_key, cur_min_row) = pq.pop() if (cur_min_row == min_row) { matching_keys.add(cur_min_key) /* There cannot be a complete match, as we already checked for one. */ assert(matching_keys.elements < n) } else if (vkey[cur_min_key] == nonull_key) { /* The non-NULL key has no corresponding NULL index, so we know for sure that the row 'min_row' is not a match. */ (min_key, min_row) = (cur_min_key, cur_min_row) matching_keys = {min_key} } else { assert(cur_min_row > min_row) /* Follows from the use of PQ. */ null_keys = set_difference(all keys vkey[], matching_keys) /* Check if all null_keys contain a NULL at row 'min_row'. The procedure internally checks all keys in a special precomputed order. A prior procedure determines an optimal order and a mapping idx_no -> idx_order (encoded as an array). This procedure makes sure not to match the non-NULL column. */ if (test_null_row(null_keys, min_row)) return TRUE else { (min_key, min_row) = (cur_min_key, cur_min_row) matching_keys = {min_key} } } vkey[cur_min_key].next() if (! vkey[cur_min_key].is_eof()) pq.insert(cur_min_key) else if (vkey[cur_min_key] == nonull_key) { /* If there can't be more matches for the nonull_key, we know for sure there is no match, since there is no possible NULL match. */ return FALSE } if (pq.is_empty()) { /* Check the last row of the last column in PQ for NULL matches. */ null_keys = set_difference(all keys vkey[], matching_keys) if (test_null_row(null_keys, min_row)) return TRUE else return FALSE } } /* We should never get here. */ assert(FALSE) return FALSE } 3. Directions for improvement ======================================================================== Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all and v_j1,...,v_jm cannot be null, create for these columns only one index array (and of course do not create any bitmaps for them). [done] 2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number of rows, where a_i is not null and V(a_i) is the number of distinct values for a_i excluding nulls. If d(a_i) is close to N'(a_i) then do not create any index array: check whether there is a match running through the records that have been filtered in. Anyway if d(a_i) is close to N'(a_i) then the intersection with rowid{a_i=v_i} will not reduce the number of remaining rowids significantly. In other words is V(a_i) exceeds some threshold there is no sense to create an index for a_i. If additionally N-N'(a_i) is small do not create a bitmap for this column either. 3. If for a column a_i d(a_i) is not close to N'(a_i), but N-N'(a_i) is small a sorted array of rowids from the set rowid{a_i is null} can be used instead of a bitmap. 4. We always have a match if R0= INTERSECT rowid{a_i is null} is not empty. Here i runs through all indexes from [1..n] such that v_i is not null. For a given subset of columns this fact has to be checked only once. It can be easily done with bitmap intersection. 5. If v1,...,vn never can be a null, then indexes (sorted arrays) can be created only for rows with nulls. 6. If v1,...,vn never can be a null and number of rows with nulls is small do not create indexes and do not create bitmaps. 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. [This is wrong, because if we don't fill the whole temp table, there may be some tuple(s) that would match some outer tuple. In such cases, if we stop filling the temp table, we would miss a TRUE result. Having a partial match doesn't preclude us from having a complete match]. 8. [timour] Consider that due to materialization, we already have a unique index on all columns <a_1,..., a_n>. We can use the first key part of this index over column a_1, instead of the index rowid{a_i=v_i}. Thus we can avoid creating the index rowid{a_i=v_i}. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Guest): Subquery optimization: Efficient NOT IN execution with NULLs (68)
by worklog-noreply＠askmonty.org 27 Feb '10

27 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subquery optimization: Efficient NOT IN execution with NULLs CREATION DATE..: Fri, 27 Nov 2009, 13:22 SUPERVISOR.....: Monty IMPLEMENTOR....: Timour COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 68 (http://askmonty.org/worklog/?tid=68) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Guest - Sat, 27 Feb 2010, 10:11)=-=- Status updated. --- /tmp/wklog.68.old.24229 2010-02-27 10:11:57.000000000 +0000 +++ /tmp/wklog.68.new.24229 2010-02-27 10:11:57.000000000 +0000 @@ -1 +1 @@ -Assigned +In-Progress -=-=(Timour - Mon, 22 Feb 2010, 17:39)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.17116 2010-02-22 17:39:48.000000000 +0200 +++ /tmp/wklog.68.new.17116 2010-02-22 17:39:48.000000000 +0200 @@ -233,6 +233,7 @@ 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all and v_j1,...,v_jm cannot be null, create for these columns only one index array (and of course do not create any bitmaps for them). +[done] 2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number of rows, where a_i is not null and V(a_i) is the number of distinct @@ -264,6 +265,10 @@ 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. +[This is wrong, because if we don't fill the whole temp table, there may + be some tuple(s) that would match some outer tuple. In such cases, if we + stop filling the temp table, we would miss a TRUE result. Having a partial + match doesn't preclude us from having a complete match]. 8. [timour] Consider that due to materialization, we already have a unique index -=-=(Timour - Tue, 19 Jan 2010, 18:44)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.22569 2010-01-19 18:44:01.000000000 +0200 +++ /tmp/wklog.68.new.22569 2010-01-19 18:44:01.000000000 +0200 @@ -132,11 +132,10 @@ if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE - if (nonull_key) - pq.insert(nonull_key) for (i = 1; i <= n; i++) { + if (vkey[i] != nonull_key) vkey[i].lookup(outer_ref) if (! vkey[i].is_eof()) pq.insert(i) @@ -167,7 +166,7 @@ /* There cannot be a complete match, as we already checked for one. */ assert(matching_keys.elements < n) } - else if (cur_min_key == nonull_key) + else if (vkey[cur_min_key] == nonull_key) { /* The non-NULL key has no corresponding NULL index, so we know for @@ -183,8 +182,10 @@ /* Check if all null_keys contain a NULL at row 'min_row'. The procedure internally checks all keys in a special precomputed order. A prior - procedure determines an optimal order and a mapping - idx_no -> idx_order (encoded as an array). + procedure determines an optimal order and a mapping idx_no -> idx_order + (encoded as an array). + + This procedure makes sure not to match the non-NULL column. */ if (test_null_row(null_keys, min_row)) return TRUE @@ -198,6 +199,14 @@ vkey[cur_min_key].next() if (! vkey[cur_min_key].is_eof()) pq.insert(cur_min_key) + else if (vkey[cur_min_key] == nonull_key) + { + /* + If there can't be more matches for the nonull_key, we know for sure + there is no match, since there is no possible NULL match. + */ + return FALSE + } if (pq.is_empty()) { @@ -216,7 +225,6 @@ } - 3. Directions for improvement ======================================================================== -=-=(Timour - Tue, 19 Jan 2010, 18:29)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.21045 2010-01-19 18:29:12.000000000 +0200 +++ /tmp/wklog.68.new.21045 2010-01-19 18:29:12.000000000 +0200 @@ -132,6 +132,8 @@ if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE + if (nonull_key) + pq.insert(nonull_key) for (i = 1; i <= n; i++) { -=-=(Guest - Tue, 19 Jan 2010, 18:15)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.19825 2010-01-19 18:15:30.000000000 +0200 +++ /tmp/wklog.68.new.19825 2010-01-19 18:15:30.000000000 +0200 @@ -1,8 +1,16 @@ -This a copy of the initial algorithm proposed by Igor: -====================================================== +Contents +======================================================================== -For each left side tuple (v_1,...,v_n) we have to find the following set -of rowids for the temp table containing N rows as the result of +1. Initial idea as proposed by Igor +2. Algorithm for IN execution with partial matching +3. Directions for improvement + + +1. Initial idea as proposed by Igor +======================================================================== + +For each left side tuple (v_1,...,v_n) we have to find the following +set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs @@ -18,38 +26,198 @@ - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap -Taken all above into account I could suggest the following algorithm to -build R: +Taken all above into account I could suggest the following algorithm +to build R: - Using indexes (read about them below) for each column participating in the - intersection, - merge ordered sets rowid{a_i=v_i} in the following manner. + Using indexes (read about them below) for each column participating + in the intersection, merge ordered sets rowid{a_i=v_i} in the + following manner. If a rowid r has been encountered maximum in k sets -rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), + rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is -not in {i1,...,ik}. + not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. -Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i -is null} is either +Here we use the property (1): +any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can -infer that for any r from R -indexes a_i can be uniquely divided into two groups: one contains -indexes a_i where r belongs to -the sets rowid{a_i=v_i}, the other contains indexes a_j such that r -belongs to rowid{a_j is null}. - -Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order -needed for the merge procedure. We could use BTREE indexes for temp -table. But they are rather expensive and -take a lot of memory as the are implemented with RB trees. +infer that for any r from R indexes a_i can be uniquely divided into +two groups: +- one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, +- the other contains indexes a_j such that r belongs to + rowid{a_j is null}. + +Now let's talk how to get elements from rowid{a_i=v_i} in a sorted +order needed for the merge procedure. We could use BTREE indexes for +temp table. But they are rather expensive and take a lot of memory as +the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. +2. Algorithm for IN execution with partial matching +======================================================================== + +2.1 Below is shown the top-level algorithm to execute an IN predicate +with partial matching. This algorithm is essentially the implementation +of Item_subselect:exec(). + +int lookup_with_null_semantics(outer_ref[], mat_subquery) +{ + if (index_lookup(outer_ref, mat_subquery) + return TRUE + else + { + /* + Check if there is a partial match (UNKNOWN) or no match (NULL). + */ + if (this is the first partial match) + { + vkey[] = build array of value keys for each NULL-able column + of mat_subquery. + nkey[] = build a bitmap NULL index for each column of mat_subquery + that contains NULLs + nonull_key = build a key over all non-NULL columns of mat_subquery + } + if (partial_match(outer_ref, vkey[], nkey[], nonull_key) + return UNKNOWN + else + return FALSE + } +} + +2.2 The implementation of partial matching is as follows + +/* + Assumptions: + - It has already been checked if there is a complete match by a + regular index lookup, and the test failed. + - It has already been checked if there is a complete NULL row, + and if there was we wouldn't call this function. Thus we assume + that there is no complete NULL row. + - Not all vidx_i are empty, but some can be empty. If all were empty, + then the only possibility for a match is a complete NULL row, which + we already checked. + + @param outer_ref - the uter (left) IN argument. + @param vidx[] - array of value keys + Ordered sequences of rowids of the corresponding columns a_i, such + that all rowids in idx_i are the ones where column a_i contains some + value or NULL. Each idx_i is derived dynamically, for each different + left argument of an IN predicate. + @param nidx[] - array of NULL keys + Bitmpas, one per each column, where a bit is set if the corresponding + row has a NULL value for the corresponding column. + @nonull_key - the only key over all columns of the materialized subquery + that do not contain NULLs + + @returns + @retval FALSE if there is no match + @retval TRUE if there is a partial match +*/ + +Boolean partial_match(outer_ref, vkey[], nkey[], nonull_key) +{ + /* Set of the keys (columns) that form a partial match. */ + Set matching_keys = {} + /* A subset of all keys that need to be checked for NULL matches. */ + Set null_keys = {} + Int min_key /* Key that contains the current minimum position. */ + Int min_row /* Current row number of min_key. */ + Int cur_min_key, cur_min_row + PriorityQueue pq + + if (nonull_key && ! nonull_key->lookup(outer_ref)) + return FALSE + + for (i = 1; i <= n; i++) + { + vkey[i].lookup(outer_ref) + if (! vkey[i].is_eof()) + pq.insert(i) + } + /* + Not all value keys are empty, thus we don't have only NULL + keys. If we had, the only possible match is a NULL row, and + we cheked there is no such row, therefore the result is known + to be FALSE. + In fact this algorithm makes sense for at least two non-NULL + columns. + */ + assert(pq.elements > 1) + + (min_key, min_row) = pq.pop() + matching_keys.add(min_key) + vkey[min_key].next() + if (! vkey[min_key].is_eof()) + pq.insert(min_key) + + while (TRUE) + { + (cur_min_key, cur_min_row) = pq.pop() + + if (cur_min_row == min_row) + { + matching_keys.add(cur_min_key) + /* There cannot be a complete match, as we already checked for one. */ + assert(matching_keys.elements < n) + } + else if (cur_min_key == nonull_key) + { + /* + The non-NULL key has no corresponding NULL index, so we know for + sure that the row 'min_row' is not a match. + */ + (min_key, min_row) = (cur_min_key, cur_min_row) + matching_keys = {min_key} + } + else + { + assert(cur_min_row > min_row) /* Follows from the use of PQ. */ + null_keys = set_difference(all keys vkey[], matching_keys) + /* + Check if all null_keys contain a NULL at row 'min_row'. The procedure + internally checks all keys in a special precomputed order. A prior + procedure determines an optimal order and a mapping + idx_no -> idx_order (encoded as an array). + */ + if (test_null_row(null_keys, min_row)) + return TRUE + else + { + (min_key, min_row) = (cur_min_key, cur_min_row) + matching_keys = {min_key} + } + } + + vkey[cur_min_key].next() + if (! vkey[cur_min_key].is_eof()) + pq.insert(cur_min_key) + + if (pq.is_empty()) + { + /* Check the last row of the last column in PQ for NULL matches. */ + null_keys = set_difference(all keys vkey[], matching_keys) + if (test_null_row(null_keys, min_row)) + return TRUE + else + return FALSE + } + } + + /* We should never get here. */ + assert(FALSE) + return FALSE +} + + + +3. Directions for improvement +======================================================================== + Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary -=-=(Timour - Sun, 06 Dec 2009, 14:36)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.12919 2009-12-06 14:36:18.000000000 +0200 +++ /tmp/wklog.68.new.12919 2009-12-06 14:36:18.000000000 +0200 @@ -87,3 +87,8 @@ 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. +8. [timour] + Consider that due to materialization, we already have a unique index +on all columns <a_1,..., a_n>. We can use the first key part of this index +over column a_1, instead of the index rowid{a_i=v_i}. Thus we can avoid +creating the index rowid{a_i=v_i}. -=-=(Timour - Fri, 04 Dec 2009, 14:04)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.16724 2009-12-04 14:04:28.000000000 +0200 +++ /tmp/wklog.68.new.16724 2009-12-04 14:04:28.000000000 +0200 @@ -10,7 +10,8 @@ Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint - (2) For each i: rowid{a_i is null} is the same for each tuple + (2) For each i: rowid{a_i is null} is the same for each tuple, + that is, this set is independent of the left-side tuples. Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Version updated. --- /tmp/wklog.68.old.5257 2009-12-04 11:27:11.000000000 +0200 +++ /tmp/wklog.68.new.5257 2009-12-04 11:27:11.000000000 +0200 @@ -1 +1 @@ -Benchmarks-3.0 +Server-9.x -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Category updated. --- /tmp/wklog.68.old.5242 2009-12-04 11:27:02.000000000 +0200 +++ /tmp/wklog.68.new.5242 2009-12-04 11:27:02.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Status updated. --- /tmp/wklog.68.old.5242 2009-12-04 11:27:02.000000000 +0200 +++ /tmp/wklog.68.new.5242 2009-12-04 11:27:02.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +Assigned ------------------------------------------------------------ -=-=(View All Progress Notes, 12 total)=-=- http://askmonty.org/worklog/index.pl?tid=68&nolimit=1 DESCRIPTION: The goal of this task is to implement efficient execution of NOT IN subquery predicates of the form: <oe_1,...,oe_n> NOT IN <non_correlated subquery> when either some oe_i, or some subqury result column contains NULLs. The problem with such predicates is that it is possible to use index lookups only when neither argument of the predicate contains NULLs. If some argument contains a NULL, then due to NULL semantics, it plays the role of a wildcard. If we were to use regular index lookups, then we would get 'no match' for some outer tuple (thus the predicate evaluates to FALSE), while the SQL semantics means 'partial match', and the predicate should evaluate to NULL. This task implements an efficient algorithm to compute such 'parial matches', where a NULL matches any value. HIGH-LEVEL SPECIFICATION: Contents ======================================================================== 1. Initial idea as proposed by Igor 2. Algorithm for IN execution with partial matching 3. Directions for improvement 1. Initial idea as proposed by Igor ======================================================================== For each left side tuple (v_1,...,v_n) we have to find the following set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs trough all indexes from [1..n] such that v_i is not null. Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint (2) For each i: rowid{a_i is null} is the same for each tuple, that is, this set is independent of the left-side tuples. Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap Taken all above into account I could suggest the following algorithm to build R: Using indexes (read about them below) for each column participating in the intersection, merge ordered sets rowid{a_i=v_i} in the following manner. If a rowid r has been encountered maximum in k sets rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can infer that for any r from R indexes a_i can be uniquely divided into two groups: - one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, - the other contains indexes a_j such that r belongs to rowid{a_j is null}. Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order needed for the merge procedure. We could use BTREE indexes for temp table. But they are rather expensive and take a lot of memory as the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. 2. Algorithm for IN execution with partial matching ======================================================================== 2.1 Below is shown the top-level algorithm to execute an IN predicate with partial matching. This algorithm is essentially the implementation of Item_subselect:exec(). int lookup_with_null_semantics(outer_ref[], mat_subquery) { if (index_lookup(outer_ref, mat_subquery) return TRUE else { /* Check if there is a partial match (UNKNOWN) or no match (NULL). */ if (this is the first partial match) { vkey[] = build array of value keys for each NULL-able column of mat_subquery. nkey[] = build a bitmap NULL index for each column of mat_subquery that contains NULLs nonull_key = build a key over all non-NULL columns of mat_subquery } if (partial_match(outer_ref, vkey[], nkey[], nonull_key) return UNKNOWN else return FALSE } } 2.2 The implementation of partial matching is as follows /* Assumptions: - It has already been checked if there is a complete match by a regular index lookup, and the test failed. - It has already been checked if there is a complete NULL row, and if there was we wouldn't call this function. Thus we assume that there is no complete NULL row. - Not all vidx_i are empty, but some can be empty. If all were empty, then the only possibility for a match is a complete NULL row, which we already checked. @param outer_ref - the uter (left) IN argument. @param vidx[] - array of value keys Ordered sequences of rowids of the corresponding columns a_i, such that all rowids in idx_i are the ones where column a_i contains some value or NULL. Each idx_i is derived dynamically, for each different left argument of an IN predicate. @param nidx[] - array of NULL keys Bitmpas, one per each column, where a bit is set if the corresponding row has a NULL value for the corresponding column. @nonull_key - the only key over all columns of the materialized subquery that do not contain NULLs @returns @retval FALSE if there is no match @retval TRUE if there is a partial match */ Boolean partial_match(outer_ref, vkey[], nkey[], nonull_key) { /* Set of the keys (columns) that form a partial match. */ Set matching_keys = {} /* A subset of all keys that need to be checked for NULL matches. */ Set null_keys = {} Int min_key /* Key that contains the current minimum position. */ Int min_row /* Current row number of min_key. */ Int cur_min_key, cur_min_row PriorityQueue pq if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE for (i = 1; i <= n; i++) { if (vkey[i] != nonull_key) vkey[i].lookup(outer_ref) if (! vkey[i].is_eof()) pq.insert(i) } /* Not all value keys are empty, thus we don't have only NULL keys. If we had, the only possible match is a NULL row, and we cheked there is no such row, therefore the result is known to be FALSE. In fact this algorithm makes sense for at least two non-NULL columns. */ assert(pq.elements > 1) (min_key, min_row) = pq.pop() matching_keys.add(min_key) vkey[min_key].next() if (! vkey[min_key].is_eof()) pq.insert(min_key) while (TRUE) { (cur_min_key, cur_min_row) = pq.pop() if (cur_min_row == min_row) { matching_keys.add(cur_min_key) /* There cannot be a complete match, as we already checked for one. */ assert(matching_keys.elements < n) } else if (vkey[cur_min_key] == nonull_key) { /* The non-NULL key has no corresponding NULL index, so we know for sure that the row 'min_row' is not a match. */ (min_key, min_row) = (cur_min_key, cur_min_row) matching_keys = {min_key} } else { assert(cur_min_row > min_row) /* Follows from the use of PQ. */ null_keys = set_difference(all keys vkey[], matching_keys) /* Check if all null_keys contain a NULL at row 'min_row'. The procedure internally checks all keys in a special precomputed order. A prior procedure determines an optimal order and a mapping idx_no -> idx_order (encoded as an array). This procedure makes sure not to match the non-NULL column. */ if (test_null_row(null_keys, min_row)) return TRUE else { (min_key, min_row) = (cur_min_key, cur_min_row) matching_keys = {min_key} } } vkey[cur_min_key].next() if (! vkey[cur_min_key].is_eof()) pq.insert(cur_min_key) else if (vkey[cur_min_key] == nonull_key) { /* If there can't be more matches for the nonull_key, we know for sure there is no match, since there is no possible NULL match. */ return FALSE } if (pq.is_empty()) { /* Check the last row of the last column in PQ for NULL matches. */ null_keys = set_difference(all keys vkey[], matching_keys) if (test_null_row(null_keys, min_row)) return TRUE else return FALSE } } /* We should never get here. */ assert(FALSE) return FALSE } 3. Directions for improvement ======================================================================== Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all and v_j1,...,v_jm cannot be null, create for these columns only one index array (and of course do not create any bitmaps for them). [done] 2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number of rows, where a_i is not null and V(a_i) is the number of distinct values for a_i excluding nulls. If d(a_i) is close to N'(a_i) then do not create any index array: check whether there is a match running through the records that have been filtered in. Anyway if d(a_i) is close to N'(a_i) then the intersection with rowid{a_i=v_i} will not reduce the number of remaining rowids significantly. In other words is V(a_i) exceeds some threshold there is no sense to create an index for a_i. If additionally N-N'(a_i) is small do not create a bitmap for this column either. 3. If for a column a_i d(a_i) is not close to N'(a_i), but N-N'(a_i) is small a sorted array of rowids from the set rowid{a_i is null} can be used instead of a bitmap. 4. We always have a match if R0= INTERSECT rowid{a_i is null} is not empty. Here i runs through all indexes from [1..n] such that v_i is not null. For a given subset of columns this fact has to be checked only once. It can be easily done with bitmap intersection. 5. If v1,...,vn never can be a null, then indexes (sorted arrays) can be created only for rows with nulls. 6. If v1,...,vn never can be a null and number of rows with nulls is small do not create indexes and do not create bitmaps. 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. [This is wrong, because if we don't fill the whole temp table, there may be some tuple(s) that would match some outer tuple. In such cases, if we stop filling the temp table, we would miss a TRUE result. Having a partial match doesn't preclude us from having a complete match]. 8. [timour] Consider that due to materialization, we already have a unique index on all columns <a_1,..., a_n>. We can use the first key part of this index over column a_1, instead of the index rowid{a_i=v_i}. Thus we can avoid creating the index rowid{a_i=v_i}. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

Re: [Maria-developers] request for advice on managing a mysql patch in bzr
by MARK CALLAGHAN 27 Feb '10

27 Feb '10

On Fri, Feb 26, 2010 at 4:19 PM, Sergei Golubchik <serg(a)askmonty.org> wrote: > Eh, you omitted lots of crucial details here, but luckily you mentioned > them in your email to internals, so I was able to decrypt the above :) > > Anyway, what happens - you are not merging one mysql branch into > another. You merge two completely unrelated repositories. And bzr see > that both the source and the target repositories have the files with the > name "configure.in" for example - and this is obviously a conflict, as > two different files have the same name, bzr cannot merge that, it's not > the same file with the different content, they are two different files. > Two different file keys, at least. > > I seriously doubt that git would handle that situation the way you want. > By all means, bitkeeper would not - it would not even allow you to pull > from an unrelated repository, I'm a bit surprised that bzr let you do > that. > > I'm afraid your only option is to replay your changes on a mysql bzr > branch. I ended up doing that. Thanks for the advice. https://code.launchpad.net/~mysqlatfacebook/mysqlatfacebook/5.1 -- Mark Callaghan mdcallag(a)gmail.com

1 0

Re: [Maria-developers] request for advice on managing a mysql patch in bzr
by MARK CALLAGHAN 26 Feb '10

26 Feb '10

On Fri, Feb 26, 2010 at 6:36 AM, Sergei Golubchik <serg(a)askmonty.org> wrote: > When done, you'll still have your patches on top of the tree, and you > can trivially extract them (e.g. with bzr log -p) and publish them. > If you merge instead of rebasing, you'll have old patches in the tree > and merge changes spread over many merge changesets. There apparently isn't a trivial tool to import commits between two branches. Doing something like this does way more than apply the diff for that one change and copy the commit mesage. bzr merge -c revno:2 /s/bzr/fb/5.1 -- Mark Callaghan mdcallag(a)gmail.com

1 0

Re: [Maria-developers] request for advice on managing a mysql patch in bzr
by MARK CALLAGHAN 26 Feb '10

26 Feb '10

On Fri, Feb 26, 2010 at 6:36 AM, Sergei Golubchik <serg(a)askmonty.org> wrote: > Hi, MARK! > > On Feb 25, MARK CALLAGHAN wrote: >> >> I know this question is more about official MySQL than MariaDB, but it >> is also about sharing code we write at Facebook and all of the patches >> we publish can be reused by others as long as others accept BSD >> contributions. >> >> I am getting ready to publish facebook patches for mysql 5.1. We use >> git internally and are based on MySQL 5.1.44. We intend to stay >> current with 5.1 releases. What is the best way to manage the code in >> launchpad across new releases of official MySQL. My current plan is: >> >> 1) import MySQL 5.1.44, commit 2) apply patches from my git repo to my >> bzr repo, commit after each one 3) publish >> >> But there will soon be a release of 5.1.45 or 5.1.46 and I don't want >> to repeat the steps above each time that happens. Do I just publish a >> large patch for 5.1.44 -> 5.1.4X and get on with my work? Is there a >> better way? > > The least labor-intensive path, I think, could be something like > Branch mysql-5.1 tree (say, up to tag:mysql-5.1.44), apply your patches > and commit. One commit per logical patch, as usual. > > Then when 5.1.45 comes out you rebase (not merge) your tree to > tag:mysql-5.1.45. Some merges you'll need to resolve manually, others > bzr can handle automatically. > > When done, you'll still have your patches on top of the tree, and you > can trivially extract them (e.g. with bzr log -p) and publish them. > If you merge instead of rebasing, you'll have old patches in the tree > and merge changes spread over many merge changesets. That is the advice I need. Thanks. -- Mark Callaghan mdcallag(a)gmail.com

1 0

[Maria-developers] Rev 11: Merge. in file:///Users/hakan/work/monty_program/mariadb-tools/
by Hakan Kuecuekyilmaz 26 Feb '10

26 Feb '10

At file:///Users/hakan/work/monty_program/mariadb-tools/ ------------------------------------------------------------ revno: 11 [merge] revision-id: hakan(a)askmonty.org-20100226170755-uucbsqe4n60b50c1 parent: hakan(a)askmonty.org-20100226162413-118168qx5c9c02ml parent: knielsen@hasky-20100223074854-mg13sm3jnan2s51a committer: Hakan Kuecuekyilmaz <hakan(a)askmonty.org> branch nick: mariadb-tools timestamp: Fri 2010-02-26 17:07:55 +0000 message: Merge. modified: buildbot/maria-master.cfg mariamaster.cfg-20091218103450-cvifjz3i70oerkej-1 === modified file 'buildbot/maria-master.cfg' --- a/buildbot/maria-master.cfg 2010-02-02 15:10:12 +0000 +++ b/buildbot/maria-master.cfg 2010-02-23 07:48:54 +0000 @@ -113,7 +113,8 @@ "lp:~maria-captains/maria/maria-5.2-merge-5.1" : "maria-5.2-merge-5.1", "lp:~maria-captains/maria/5.2-dsmrr" : "5.2-dsmrr", "lp:~maria-captains/maria/5.3" : "5.3", - "lp:~maria-captains/maria/5.3-sj-subqueries" : "5.3-sj-subqueries" + "lp:~maria-captains/maria/5.3-sj-subqueries" : "5.3-sj-subqueries", + "lp:~maria-captains/maria/5.3-subqueries" : "5.3-subqueries" } mailSource = mail.BzrLaunchpadEmailMaildirSource("/var/lib/buildbot/Maildir", branchMap=myBranchMap) @@ -154,7 +155,7 @@ "mariadb-5.1-monty", "mariadb-5.2-monty", "mariadb-5.1-knielsen","5.1-release", "5.1-merge", "maria-5.1-wl24", "maria-5.1-vcol","maria-5.1-wl36","mysql-5.1-mwl36", - "5.1.39-oqgraph", "5.2-dsmrr", "5.3", "5.3-sj-subqueries" + "5.1.39-oqgraph", "5.2-dsmrr", "5.3", "5.3-sj-subqueries", "5.3-subqueries" ], treeStableTimer=60, # 1 sec for bzr builderNames=["centos5-debug", "hardy-x86-rtai", "hardy-amd64-makedist", @@ -531,7 +532,7 @@ "slavename": "adutko-centos5-amd64", "builddir": "centos5-amd64-minimal", "factory": f_minimal, - "category": "main", + "category": "experimental", } f_win32_rel_nmake = factory.BuildFactory()

1 0

[Maria-developers] Rev 10: Do one thing, and do that one thing proper. - The Unix Way. in file:///Users/hakan/work/monty_program/mariadb-tools/
by Hakan Kuecuekyilmaz 26 Feb '10

26 Feb '10

At file:///Users/hakan/work/monty_program/mariadb-tools/ ------------------------------------------------------------ revno: 10 revision-id: hakan(a)askmonty.org-20100226162413-118168qx5c9c02ml parent: hakan(a)askmonty.org-20100219052704-8nnlzsbu0nslnxba committer: Hakan Kuecuekyilmaz <hakan(a)askmonty.org> branch nick: mariadb-tools timestamp: Fri 2010-02-26 16:24:13 +0000 message: Do one thing, and do that one thing proper. - The Unix Way. We are running the benchmark for one source repository now. Also we run each test three times, so that peaks and glitches are balanced out. === modified file 'sysbench/run-sysbench.sh' --- a/sysbench/run-sysbench.sh 2010-02-19 05:27:04 +0000 +++ b/sysbench/run-sysbench.sh 2010-02-26 16:24:13 +0000 @@ -2,8 +2,10 @@ # # Run sysbench tests with MariaDB and MySQL # -# Note: Do not run this script with root privileges. -# We use killall -9, which can cause severe side effects! +# Notes: +# * Do not run this script with root privileges. We use +# killall -9, which can cause severe side effects! +# * By bzr pull we mean bzr merge --pull # # Hakan Kuecuekyilmaz <hakan at askmonty dot org> 2010-02-19. # @@ -19,6 +21,29 @@ exit 1 fi +if [ $# != 3 ]; then + echo '[ERROR]: Please provide exactly three options.' + echo " Example: $0 [pull | no-pull] [/path/to/bzr/repo] [name]" + echo " $0 pull ${HOME}/work/monty_program/maria-local-master MariaDB" + + exit 1 +else + PULL="$1" + LOCAL_MASTER="$2" + PRODUCT="$3" +fi + +# +# Binaries. +# +MYSQLADMIN='client/mysqladmin' + +# +# Adjust the following paths according to your installation. +# +SYSBENCH='/usr/local/bin/sysbench' +BZR='/usr/local/bin/bzr' + # # Variables. # @@ -27,32 +52,42 @@ MY_SOCKET="${TEMP_DIR}/mysql.sock" MYSQLADMIN_OPTIONS="--no-defaults -uroot --socket=$MY_SOCKET" MYSQL_OPTIONS="--no-defaults \ + --datadir=$DATA_DIR \ + --language=./sql/share/english \ + --max_connections=256 \ + --query_cache_size=0 \ + --query_cache_type=0 \ --skip-grant-tables \ - --language=./sql/share/english \ - --datadir=$DATA_DIR \ - --tmpdir=$TEMP_DIR \ --socket=$MY_SOCKET \ --table_open_cache=512 \ --thread_cache=512 \ - --query_cache_size=0 \ - --query_cache_type=0 \ + --tmpdir=$TEMP_DIR \ + --innodb_additional_mem_pool_size=32M \ + --innodb_buffer_pool_size=1024M \ + --innodb_data_file_path=ibdata1:32M:autoextend \ --innodb_data_home_dir=$DATA_DIR \ - --innodb_data_file_path=ibdata1:128M:autoextend \ - --innodb_log_group_home_dir=$DATA_DIR \ - --innodb_buffer_pool_size=1024M \ - --innodb_additional_mem_pool_size=32M \ - --innodb_log_file_size=256M \ - --innodb_log_buffer_size=16M \ + --innodb_doublewrite=0 \ --innodb_flush_log_at_trx_commit=1 \ + --innodb_flush_method=O_DIRECT \ --innodb_lock_wait_timeout=50 \ - --innodb_doublewrite=0 \ - --innodb_flush_method=O_DIRECT \ - --innodb_thread_concurrency=0 \ - --innodb_max_dirty_pages_pct=80" + --innodb_log_buffer_size=16M \ + --innodb_log_file_size=256M \ + --innodb_log_group_home_dir=$DATA_DIR \ + --innodb_max_dirty_pages_pct=80 \ + --innodb_thread_concurrency=0" +# Number of threads we run sysbench with. NUM_THREADS="1 4 8 16 32 64 128" + +# The table size we use for sysbench. TABLE_SIZE=2000000 + +# The run time we use for sysbench. RUN_TIME=300 + +# How many times we run each test. +LOOP_COUNT=3 + SYSBENCH_TESTS="delete.lua \ insert.lua \ oltp_complex_ro.lua \ @@ -61,6 +96,7 @@ select.lua \ update_index.lua \ update_non_index.lua" + SYSBENCH_OPTIONS="--oltp-table-size=$TABLE_SIZE \ --max-time=$RUN_TIME \ --max-requests=0 \ @@ -68,123 +104,86 @@ --mysql-user=root \ --mysql-engine-trx=yes" -PRODUCTS='MariaDB MySQL' - # Timeout in seconds for waiting for mysqld to start. TIMEOUT=100 # # Files # -MARIADB_BUILD_LOG='/tmp/mariadb_build.log' -MYSQL_BUILD_LOG='/tmp/mysql_build.log' +BUILD_LOG="/tmp/${PRODUCT}_build.log" # # Directories. # BASE="${HOME}/work" -MARIADB_LOCAL_MASTER="${BASE}/monty_program/maria-local-master" -MARIADB_WORK="${BASE}/monty_program/maria" -MYSQL_LOCAL_MASTER="${BASE}/mysql/mysql-server-local-master" -MYSQL_WORK="${BASE}/mysql/mysql-server" TEST_DIR="${BASE}/monty_program/sysbench/sysbench/tests/db" RESULT_DIR="${BASE}/sysbench-results" - -# -# Binaries. -# -MYSQLADMIN='./client/mysqladmin' -SYSBENCH='/usr/local/bin/sysbench' -BZR='/usr/local/bin/bzr' - -# -# Refresh repositories. -# -echo "[$(date "+%Y-%m-%d %H:%M:%S")] Refreshing source repositories." -rm -rf $MARIADB_WORK -if [ ! -d $MARIADB_LOCAL_MASTER ]; then - echo "[ERROR]: Local master of MariaDB does not exist." - echo " Please make a initial branch from lp:maria" - echo " Exiting." - exit 1 -else - cd $MARIADB_LOCAL_MASTER +WORK_DIR='/tmp' + +if [ ! -d $LOCAL_MASTER ]; then + echo "[ERROR]: Supplied local master $LOCAL_MASTER does not exists." + echo " Please provide a valid bzr repository." + echo " Exiting." + exit 1 +fi + +# +# Refresh repositories, if requested. +# +if [ x"$PULL" = x"pull" ]; then + echo "[$(date "+%Y-%m-%d %H:%M:%S")] Refreshing source repositories." + + cd $LOCAL_MASTER echo "Pulling latest MariaDB sources." - $BZR pull - if [ $? != 0 ]; then - echo "[ERROR]: $BZR pull for $MARIADB_LOCAL_MASTER failed" - echo " Please check your bzr setup" - echo " Exiting." - exit 1 - fi - - echo "Branching MariaDB working directory." - $BZR branch $MARIADB_LOCAL_MASTER $MARIADB_WORK - if [ $? != 0 ]; then - echo "[ERROR]: $BZR branch of $MARIADB_LOCAL_MASTER failed" - echo " Please check your bzr setup" - echo " Exiting." - exit 1 - fi -fi - -rm -rf $MYSQL_WORK -if [ ! -d $MYSQL_LOCAL_MASTER ]; then - echo "[ERROR]: Local master of MySQL does not exist." - echo " Please make a initial branch from lp:mysql-server" - echo " Exiting." - exit 1 -else - cd $MYSQL_LOCAL_MASTER - echo "Pulling latest MySQL sources." - $BZR pull - if [ $? != 0 ]; then - echo "[ERROR]: $BZR pull for $MYSQL_LOCAL_MASTER failed" - echo " Please check your bzr setup" - echo " Exiting." - exit 1 - fi - - echo "Branching MySQL working directory." - $BZR branch $MYSQL_LOCAL_MASTER $MYSQL_WORK - if [ $? != 0 ]; then - echo "[ERROR]: $BZR branch of $MYSQL_LOCAL_MASTER failed" - echo " Please check your bzr setup" - echo " Exiting." - exit 1 - fi -fi - -echo "[$(date "+%Y-%m-%d %H:%M:%S")] Done refreshing source repositories." - - -# -# TODO: Add platform detection and choose proper build script. -# -echo "[$(date "+%Y-%m-%d %H:%M:%S")] Starting to compile." - -echo "[$(date "+%Y-%m-%d %H:%M:%S")] Compiling MariaDB." -cd $MARIADB_WORK -BUILD/compile-amd64-max > $MARIADB_BUILD_LOG 2>&1 -if [ $? != 0 ]; then - echo "[ERROR]: Build of $MARIADB_WORK failed" - echo " Please check the log at $MARIDB_BUILD_LOG" - echo " Exiting." - exit 1 -fi -echo "[$(date "+%Y-%m-%d %H:%M:%S")] Finnished compiling MariaDB." - -echo "[$(date "+%Y-%m-%d %H:%M:%S")] Compiling MySQL." -cd $MYSQL_WORK -BUILD/compile-amd64-max > $MYSQL_BUILD_LOG 2>&1 -if [ $? != 0 ]; then - echo "[ERROR]: Build of $MYSQL_WORK failed" - echo " Please check the log at $MYSQL_BUILD_LOG" - echo " Exiting." - exit 1 -fi -echo "[$(date "+%Y-%m-%d %H:%M:%S")] Finnished compiling MySQL." -echo "[$(date "+%Y-%m-%d %H:%M:%S")] Finnished compiling." + $BZR merge --pull + if [ $? != 0 ]; then + echo "[ERROR]: $BZR pull for $LOCAL_MASTER failed" + echo " Please check your bzr setup and/or repository" + echo " Exiting." + exit 1 + fi + + echo "[$(date "+%Y-%m-%d %H:%M:%S")] Done refreshing source repositories." +fi + +cd $WORK_DIR +TEMP_DIR=$(mktemp -d) +if [ $? != 0 ]; then + echo "[ERROR]: mktemp in $WORK_DIR failed." + echo 'Exiting.' + + exit 1 +fi + +# +# bzr export refuses to export to an existing directory, +# therefore we use an extra build/ directory. +# +echo "Exporting from $LOCAL_MASTER to ${TEMP_DIR}/build" +$BZR export --format=dir ${TEMP_DIR}/build $LOCAL_MASTER +if [ $? != 0 ]; then + echo '[ERROR]: bzr export failed.' + echo 'Exiting.' + + exit 1 +fi + +# +# Compile sources. +# TODO: Add platform detection and choose proper build script accordingly. +# +echo "[$(date "+%Y-%m-%d %H:%M:%S")] Starting to compile $PRODUCT." + +cd ${TEMP_DIR}/build +BUILD/compile-amd64-max > $BUILD_LOG 2>&1 +if [ $? != 0 ]; then + echo "[ERROR]: Build of $PRODUCT failed" + echo " Please check your log at $BUILD_LOG" + echo " Exiting." + exit 1 +fi + +echo "[$(date "+%Y-%m-%d %H:%M:%S")] Finnished compiling $PRODUCT." # # Go to work. @@ -194,78 +193,79 @@ # # Prepare results directory. # -if [ ! -d $RESULT_DIRS ]; then - echo "[NOTE]: $RESULT_DIRS did not exist." +if [ ! -d $RESULT_DIR ]; then + echo "[NOTE]: $RESULT_DIR did not exist." echo " We are creating it for you!" - mkdir $RESULT_DIRS + mkdir $RESULT_DIR fi TODAY=$(date +%Y-%m-%d) mkdir ${RESULT_DIR}/${TODAY} - -for PRODUCT in $PRODUCTS; do - mkdir ${RESULT_DIR}/${TODAY}/${PRODUCT} - - killall -9 mysqld - rm -rf $DATA_DIR - rm -f $MY_SOCKET - mkdir $DATA_DIR - - if [ x"$PRODUCT" = x"MariaDB" ];then - cd $MARIADB_WORK - else - cd $MYSQL_WORK - fi - - sql/mysqld $MYSQL_OPTIONS & - - j=0 - STARTED=-1 - while [ $j -le $TIMEOUT ] - do - $MYSQLADMIN $MYSQLADMIN_OPTIONS ping > /dev/null 2>&1 - if [ $? = 0 ]; then - STARTED=0 - - break - fi - - sleep 1 - j=$(($j + 1)) - done - - if [ $STARTED != 0 ]; then - echo '[ERROR]: Start of mysqld failed.' - echo ' Please check your error log.' - echo ' Exiting.' - - exit 1 - fi - - for SYSBENCH_TEST in $SYSBENCH_TESTS; do - mkdir ${RESULT_DIR}/${TODAY}/${PRODUCT}/${SYSBENCH_TEST} - - for THREADS in $NUM_THREADS; do - THIS_RESULT_DIR="${RESULT_DIR}/${TODAY}/${PRODUCT}/${SYSBENCH_TEST}/${THREADS}" - mkdir $THIS_RESULT_DIR - echo "[$(date "+%Y-%m-%d %H:%M:%S")] Running $SYSBENCH_TEST with $THREADS threads for $PRODUCT" - - $MYSQLADMIN $MYSQLADMIN_OPTIONS -f drop sbtest - $MYSQLADMIN $MYSQLADMIN_OPTIONS create sbtest - if [ $? != 0 ]; then - echo "[ERROR]: Create of sbtest database failed" - echo " Please check your setup." - echo " Exiting" - exit 1 - fi - - SYSBENCH_OPTIONS="$SYSBENCH_OPTIONS --num-threads=$THREADS --test=${TEST_DIR}/${SYSBENCH_TEST}" - $SYSBENCH $SYSBENCH_OPTIONS prepare - $SYSBENCH $SYSBENCH_OPTIONS run > ${THIS_RESULT_DIR}/result.txt 2>&1 - - done - done +mkdir ${RESULT_DIR}/${TODAY}/${PRODUCT} + +killall -9 mysqld +rm -rf $DATA_DIR +rm -f $MY_SOCKET +mkdir $DATA_DIR + +sql/mysqld $MYSQL_OPTIONS & + +j=0 +STARTED=-1 +while [ $j -le $TIMEOUT ] + do + $MYSQLADMIN $MYSQLADMIN_OPTIONS ping > /dev/null 2>&1 + if [ $? = 0 ]; then + STARTED=0 + + break + fi + + sleep 1 + j=$(($j + 1)) +done + +if [ $STARTED != 0 ]; then + echo '[ERROR]: Start of mysqld failed.' + echo ' Please check your error log.' + echo ' Exiting.' + + exit 1 +fi + +for SYSBENCH_TEST in $SYSBENCH_TESTS + do + mkdir ${RESULT_DIR}/${TODAY}/${PRODUCT}/${SYSBENCH_TEST} + + for THREADS in $NUM_THREADS + do + THIS_RESULT_DIR="${RESULT_DIR}/${TODAY}/${PRODUCT}/${SYSBENCH_TEST}/${THREADS}" + mkdir $THIS_RESULT_DIR + echo "[$(date "+%Y-%m-%d %H:%M:%S")] Running $SYSBENCH_TEST with $THREADS threads and $LOOP_COUNT iterations for $PRODUCT" | tee ${THIS_RESULT_DIR}/results.txt + echo '' >> ${THIS_RESULT_DIR}/results.txt + + k=0 + while [ $k -lt $LOOP_COUNT ] + do + $MYSQLADMIN $MYSQLADMIN_OPTIONS -f drop sbtest + $MYSQLADMIN $MYSQLADMIN_OPTIONS create sbtest + if [ $? != 0 ]; then + echo "[ERROR]: Create of sbtest database failed" + echo " Please check your setup." + echo " Exiting" + exit 1 + fi + + SYSBENCH_OPTIONS="$SYSBENCH_OPTIONS --num-threads=$THREADS --test=${TEST_DIR}/${SYSBENCH_TEST}" + $SYSBENCH $SYSBENCH_OPTIONS prepare + $SYSBENCH $SYSBENCH_OPTIONS run > ${THIS_RESULT_DIR}/result${k}.txt 2>&1 + + grep "write requests:" ${THIS_RESULT_DIR}/result${k}.txt | awk '{ print $4 }' | sed -e 's/(//' >> ${THIS_RESULT_DIR}/results.txt + + k=$(($k + 1)) + done + done done #

1 0

Re: [Maria-developers] request for advice on managing a mysql patch in bzr
by MARK CALLAGHAN 26 Feb '10

26 Feb '10

On Fri, Feb 26, 2010 at 1:30 AM, Sergei Golubchik <serg(a)askmonty.org> wrote: > Hi, MARK! > > On Feb 25, MARK CALLAGHAN wrote: >> >> One other comment. I don't think the official releases are in >> launchpad. If they were I imagine this would be easier to do. Of >> course, anyone could republish the official releases there. > > Official MySQL releases ? > > What do you mean - mysql-5.1 tree is on launchpad, all releases are > tagged, you can branch up to any tag to get a release. > What am I missing ? > I missed that there were tags. I know that now -- Harrison explained it to me. -- Mark Callaghan mdcallag(a)gmail.com

1 0

[Maria-developers] askmonty.org moving to a new server this Thursday (25 Feb)
by Daniel Bartholomew 25 Feb '10

25 Feb '10

All, I'm sending this to both Maria Developers and Maria Discuss to make sure it gets to everyone, sorry for any duplicates. Just a heads up for everyone: I will be moving the askmonty.org website from its current home to a new server this Thursday, 25 Feb. If all goes according to plan, the website will remain up during the move but will be read-only during the actual switch. askmonty.org is the only domain/website that will be impacted by this move, websites on any other domain (such as buildbot.askmonty.org) are not part of this move and will continue to function normally. I will send a notice out on Thursday when I start the process, and a final one once the move is complete and I've tested that things are working properly. Thanks! -- Daniel Bartholomew Monty Program - http://askmonty.org

3 4

Re: [Maria-developers] protocol testing tool
by Eric Day 25 Feb '10

25 Feb '10

Hi Sergei! Like I mentioned, this will be a tool written in Python that tests all the edge cases for the MySQL and Drizzle protocols. Currently I'm through the handshake code, and will be looking at command and result processing next. This is to make sure we guarantee compatibility as we re-write the protocol code in Drizzle. This will be part of the libdrizzle project, so free to use and contribute to it! I've not pushed anything up yet, but probably will do so by the end of this week. I'll send you a link once I push it up. -Eric On Tue, Feb 09, 2010 at 01:18:24PM +0100, Sergei Golubchik wrote: > Hi, Eric > > > >>>>> "Eric" == Eric Day <eday(a)oddments.org> writes: > > > > Hi! > > I'm working on a MySQL and Drizzle protocol testing tool, and while > > testing various edge cases, > ...... > > This is interesting. I've just extended MySQL protocol [with lost of new > corner cases :)] and would love to see that tested too. > > Can you tell more about this work of yours ? > > Regards, > Sergei

3 4

[Maria-developers] Rev 2770: Prepare for OJ+SJ handling: Make replace_where_subcondition() not to assume in file:///home/psergey/dev/maria-5.3-subqueries-r7/
by Sergey Petrunya 25 Feb '10

25 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r7/ ------------------------------------------------------------ revno: 2770 revision-id: psergey(a)askmonty.org-20100225080910-38wapbgw5vpm6nax parent: psergey(a)askmonty.org-20100224113342-k558ogyr3cik1767 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r7 timestamp: Thu 2010-02-25 08:09:10 +0000 message: Prepare for OJ+SJ handling: Make replace_where_subcondition() not to assume it's working on the WHERE clause. === modified file 'sql/opt_subselect.cc' --- a/sql/opt_subselect.cc 2010-02-17 21:59:41 +0000 +++ b/sql/opt_subselect.cc 2010-02-25 08:09:10 +0000 @@ -536,7 +536,6 @@ *expr= new_cond; if (do_fix_fields) new_cond->fix_fields(join->thd, expr); - join->select_lex->where= *expr; return FALSE; }

1 0

[Maria-developers] Rev 2769: Bug#49198 Wrong result for second call of of procedure with view in subselect. in file:///home/psergey/dev/maria-5.3-subqueries-r7/
by Sergey Petrunya 24 Feb '10

24 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r7/ ------------------------------------------------------------ revno: 2769 revision-id: psergey(a)askmonty.org-20100224113342-k558ogyr3cik1767 parent: psergey(a)askmonty.org-20100223092202-sc3huivw9yymnj5d committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r7 timestamp: Wed 2010-02-24 12:33:42 +0100 message: Bug#49198 Wrong result for second call of of procedure with view in subselect. Re-worked fix of Tor Didriksen: The problem was that fix_after_pullout() after semijoin conversion wasn't propagated from the view to the underlying table. On subesequent executions of the prepared statement, we would mark the underlying table as 'dependent' and the predicate anlysis would lead to a different (and illegal) execution plan. === modified file 'mysql-test/r/subselect_sj.result' --- a/mysql-test/r/subselect_sj.result 2010-02-21 07:53:12 +0000 +++ b/mysql-test/r/subselect_sj.result 2010-02-24 11:33:42 +0000 @@ -824,3 +824,50 @@ 3 2 drop table t1, t2, t3; +# +# Bug#49198 Wrong result for second call of procedure +# with view in subselect. +# +CREATE TABLE t1 (t1field integer, primary key (t1field)); +CREATE TABLE t2 (t2field integer, primary key (t2field)); +CREATE TABLE t3 (t3field integer, primary key (t3field)); +CREATE VIEW v2 AS SELECT * FROM t2; +CREATE VIEW v3 AS SELECT * FROM t3; +INSERT INTO t1 VALUES(1),(2); +INSERT INTO t2 VALUES(1),(2); +INSERT INTO t3 VALUES(1),(2); +PREPARE stmt FROM +" +SELECT t1field +FROM t1 +WHERE t1field IN (SELECT * FROM v2); +"; +EXECUTE stmt; +t1field +1 +2 +EXECUTE stmt; +t1field +1 +2 +PREPARE stmt FROM +" +EXPLAIN +SELECT t1field +FROM t1 +WHERE t1field IN (SELECT * FROM v2) + AND t1field IN (SELECT * FROM v3) +"; +EXECUTE stmt; +id select_type table type possible_keys key key_len ref rows Extra +1 PRIMARY t1 index PRIMARY PRIMARY 4 NULL 2 Using index +1 PRIMARY t2 eq_ref PRIMARY PRIMARY 4 test.t1.t1field 1 Using index +1 PRIMARY t3 eq_ref PRIMARY PRIMARY 4 test.t1.t1field 1 Using index +EXECUTE stmt; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 index PRIMARY PRIMARY 4 NULL 2 Using index +1 SIMPLE t2 eq_ref PRIMARY PRIMARY 4 test.t1.t1field 1 Using index +1 SIMPLE t3 eq_ref PRIMARY PRIMARY 4 test.t1.t1field 1 Using index +DROP TABLE t1, t2, t3; +DROP VIEW v2, v3; +# End of Bug#49198 === modified file 'mysql-test/r/subselect_sj_jcl6.result' --- a/mysql-test/r/subselect_sj_jcl6.result 2010-02-21 07:53:12 +0000 +++ b/mysql-test/r/subselect_sj_jcl6.result 2010-02-24 11:33:42 +0000 @@ -828,6 +828,53 @@ 3 2 drop table t1, t2, t3; +# +# Bug#49198 Wrong result for second call of procedure +# with view in subselect. +# +CREATE TABLE t1 (t1field integer, primary key (t1field)); +CREATE TABLE t2 (t2field integer, primary key (t2field)); +CREATE TABLE t3 (t3field integer, primary key (t3field)); +CREATE VIEW v2 AS SELECT * FROM t2; +CREATE VIEW v3 AS SELECT * FROM t3; +INSERT INTO t1 VALUES(1),(2); +INSERT INTO t2 VALUES(1),(2); +INSERT INTO t3 VALUES(1),(2); +PREPARE stmt FROM +" +SELECT t1field +FROM t1 +WHERE t1field IN (SELECT * FROM v2); +"; +EXECUTE stmt; +t1field +1 +2 +EXECUTE stmt; +t1field +1 +2 +PREPARE stmt FROM +" +EXPLAIN +SELECT t1field +FROM t1 +WHERE t1field IN (SELECT * FROM v2) + AND t1field IN (SELECT * FROM v3) +"; +EXECUTE stmt; +id select_type table type possible_keys key key_len ref rows Extra +1 PRIMARY t1 index PRIMARY PRIMARY 4 NULL 2 Using index +1 PRIMARY t2 eq_ref PRIMARY PRIMARY 4 test.t1.t1field 1 Using index +1 PRIMARY t3 eq_ref PRIMARY PRIMARY 4 test.t1.t1field 1 Using index +EXECUTE stmt; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 index PRIMARY PRIMARY 4 NULL 2 Using index +1 SIMPLE t2 eq_ref PRIMARY PRIMARY 4 test.t1.t1field 1 Using index +1 SIMPLE t3 eq_ref PRIMARY PRIMARY 4 test.t1.t1field 1 Using index +DROP TABLE t1, t2, t3; +DROP VIEW v2, v3; +# End of Bug#49198 set join_cache_level=default; show variables like 'join_cache_level'; Variable_name Value === modified file 'mysql-test/t/subselect_sj.test' --- a/mysql-test/t/subselect_sj.test 2010-02-21 07:53:12 +0000 +++ b/mysql-test/t/subselect_sj.test 2010-02-24 11:33:42 +0000 @@ -728,3 +728,45 @@ drop table t1, t2, t3; +--echo # +--echo # Bug#49198 Wrong result for second call of procedure +--echo # with view in subselect. +--echo # + +CREATE TABLE t1 (t1field integer, primary key (t1field)); +CREATE TABLE t2 (t2field integer, primary key (t2field)); +CREATE TABLE t3 (t3field integer, primary key (t3field)); + +CREATE VIEW v2 AS SELECT * FROM t2; +CREATE VIEW v3 AS SELECT * FROM t3; + +INSERT INTO t1 VALUES(1),(2); +INSERT INTO t2 VALUES(1),(2); +INSERT INTO t3 VALUES(1),(2); + +PREPARE stmt FROM +" +SELECT t1field +FROM t1 +WHERE t1field IN (SELECT * FROM v2); +"; + +EXECUTE stmt; +EXECUTE stmt; + +PREPARE stmt FROM +" +EXPLAIN +SELECT t1field +FROM t1 +WHERE t1field IN (SELECT * FROM v2) + AND t1field IN (SELECT * FROM v3) +"; + +EXECUTE stmt; +EXECUTE stmt; + +DROP TABLE t1, t2, t3; +DROP VIEW v2, v3; + +--echo # End of Bug#49198 === modified file 'sql/item.cc' --- a/sql/item.cc 2010-02-21 06:32:23 +0000 +++ b/sql/item.cc 2010-02-24 11:33:42 +0000 @@ -6491,11 +6491,9 @@ void Item_ref::fix_after_pullout(st_select_lex *new_parent, Item **refptr) { + (*ref)->fix_after_pullout(new_parent, ref); if (depended_from == new_parent) - { - (*ref)->fix_after_pullout(new_parent, ref); depended_from= NULL; - } }

1 0

Re: [Maria-developers] [Merge] lp:~mydb08/maria/maria-5.1 into lp:maria
by Venu Kalyan 24 Feb '10

24 Feb '10

Looks like the diff is in-complete.. but you should get the updated one from here.. https://code.launchpad.net/~mydb08/maria/maria-5.1/+merge/20003 On Tue, Feb 23, 2010 at 1:17 PM, Venu Anuganti <mydb08(a)gmail.com> wrote: > Venu Anuganti has proposed merging lp:~mydb08/maria/maria-5.1 into lp:maria. > > Requested reviews: > Maria-captains (maria-captains) > > > Initial SHOW [GLOBAL] TEMPORARY TABLES [FROM/IN db] implementation along with associated Information Schema tables. > > Its been a while (probably 6 yrs+) I pushed any code to MySQL, so review is a must even though am working actively on MySQL code for last few years for developing storage engines. > > Give me the feedback in case if I missed anything. > > Todo: > > - ACL check on global temp tables, need an advice on what priv to use > - InnoDB has some issues on stat(tmp_file) as its not generating the full path, will work on a fix. > > Questions: > > Do we need to expose internal temp tables to this implementation ? I don't see any benefit doing so as it should be really part of slow query implementation. > > -- > https://code.launchpad.net/~mydb08/maria/maria-5.1/+merge/20003 > You are the owner of lp:~mydb08/maria/maria-5.1. > > === modified file 'sql/mysqld.cc' > --- sql/mysqld.cc 2010-02-11 19:15:24 +0000 > +++ sql/mysqld.cc 2010-02-23 21:17:16 +0000 > @@ -3272,6 +3272,7 @@ > {"show_storage_engines", (char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SHOW_STORAGE_ENGINES]), SHOW_LONG_STATUS}, > {"show_table_status", (char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SHOW_TABLE_STATUS]), SHOW_LONG_STATUS}, > {"show_tables", (char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SHOW_TABLES]), SHOW_LONG_STATUS}, > + {"show_temporary_tables",(char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SHOW_TEMPORARY_TABLES]), SHOW_LONG_STATUS}, > {"show_triggers", (char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SHOW_TRIGGERS]), SHOW_LONG_STATUS}, > {"show_variables", (char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SHOW_VARIABLES]), SHOW_LONG_STATUS}, > {"show_warnings", (char*) offsetof(STATUS_VAR, com_stat[(uint) SQLCOM_SHOW_WARNS]), SHOW_LONG_STATUS}, > > === modified file 'sql/sql_lex.h' > --- sql/sql_lex.h 2009-12-03 11:19:05 +0000 > +++ sql/sql_lex.h 2010-02-23 21:17:16 +0000 > @@ -67,7 +67,7 @@ > SQLCOM_UPDATE, SQLCOM_INSERT, SQLCOM_INSERT_SELECT, > SQLCOM_DELETE, SQLCOM_TRUNCATE, SQLCOM_DROP_TABLE, SQLCOM_DROP_INDEX, > > - SQLCOM_SHOW_DATABASES, SQLCOM_SHOW_TABLES, SQLCOM_SHOW_FIELDS, > + SQLCOM_SHOW_DATABASES, SQLCOM_SHOW_TABLES, SQLCOM_SHOW_TEMPORARY_TABLES, SQLCOM_SHOW_FIELDS, > SQLCOM_SHOW_KEYS, SQLCOM_SHOW_VARIABLES, SQLCOM_SHOW_STATUS, > SQLCOM_SHOW_ENGINE_LOGS, SQLCOM_SHOW_ENGINE_STATUS, SQLCOM_SHOW_ENGINE_MUTEX, > SQLCOM_SHOW_PROCESSLIST, SQLCOM_SHOW_MASTER_STAT, SQLCOM_SHOW_SLAVE_STAT, > > === modified file 'sql/sql_parse.cc' > --- sql/sql_parse.cc 2010-01-29 10:42:31 +0000 > +++ sql/sql_parse.cc 2010-02-23 21:17:16 +0000 > @@ -342,6 +342,9 @@ > sql_command_flags[SQLCOM_SHOW_TABLES]= (CF_STATUS_COMMAND | > CF_SHOW_TABLE_COMMAND | > CF_REEXECUTION_FRAGILE); > + sql_command_flags[SQLCOM_SHOW_TEMPORARY_TABLES]= (CF_STATUS_COMMAND | > + CF_SHOW_TABLE_COMMAND | > + CF_REEXECUTION_FRAGILE); > sql_command_flags[SQLCOM_SHOW_TABLE_STATUS]= (CF_STATUS_COMMAND | > CF_SHOW_TABLE_COMMAND | > CF_REEXECUTION_FRAGILE); > @@ -1774,6 +1777,8 @@ > > case SCH_TABLE_NAMES: > case SCH_TABLES: > + case SCH_TEMPORARY_TABLES: > + case SCH_GLOBAL_TEMPORARY_TABLES: > case SCH_VIEWS: > case SCH_TRIGGERS: > case SCH_EVENTS: > @@ -2262,6 +2267,7 @@ > } > case SQLCOM_SHOW_DATABASES: > case SQLCOM_SHOW_TABLES: > + case SQLCOM_SHOW_TEMPORARY_TABLES: > case SQLCOM_SHOW_TRIGGERS: > case SQLCOM_SHOW_TABLE_STATUS: > case SQLCOM_SHOW_OPEN_TABLES: > @@ -5354,6 +5360,8 @@ > > case SCH_TABLE_NAMES: > case SCH_TABLES: > + case SCH_TEMPORARY_TABLES: > + case SCH_GLOBAL_TEMPORARY_TABLES: > case SCH_VIEWS: > case SCH_TRIGGERS: > case SCH_EVENTS: > > === modified file 'sql/sql_show.cc' > --- sql/sql_show.cc 2010-01-15 15:27:55 +0000 > +++ sql/sql_show.cc 2010-02-23 21:17:16 +0000 > @@ -2716,6 +2716,7 @@ > return 0; > case SQLCOM_SHOW_TABLES: > case SQLCOM_SHOW_TABLE_STATUS: > + case SQLCOM_SHOW_TEMPORARY_TABLES: > case SQLCOM_SHOW_TRIGGERS: > case SQLCOM_SHOW_EVENTS: > lookup_field_values->db_value.str= lex->select_lex.db; > @@ -3171,6 +3172,212 @@ > return (uint) OPEN_FULL_TABLE; > } > > +/** > + @brief Change I_S table item list for SHOW [GLOBAL] TEMPORARY TABLES [FROM/IN db] > + > + @param[in] thd thread handler > + @param[in] schema_table I_S table > + > + @return Operation status > + @retval 0 success > + @retval 1 error > +*/ > +int make_temporary_tables_old_format(THD *thd, ST_SCHEMA_TABLE *schema_table) > +{ > + char tmp[128]; > + String buffer(tmp,sizeof(tmp), thd->charset()); > + LEX *lex= thd->lex; > + Name_resolution_context *context= &lex->select_lex.context; > + > + if (thd->lex->option_type == OPT_GLOBAL) { > + ST_FIELD_INFO *field_info= &schema_table->fields_info[0]; > + Item_field *field= new Item_field(context, NullS, NullS, field_info->field_name); > + if (add_item_to_list(thd, field)) > + return 1; > + field->set_name(field_info->old_name, strlen(field_info->old_name), system_charset_info); > + } > + > + ST_FIELD_INFO *field_info= &schema_table->fields_info[2]; > + buffer.length(0); > + buffer.append(field_info->old_name); > + buffer.append(lex->select_lex.db); > + > + if (lex->wild && lex->wild->ptr()) > + { > + buffer.append(STRING_WITH_LEN(" (")); > + buffer.append(lex->wild->ptr()); > + buffer.append(')'); > + } > + > + Item_field *field= new Item_field(context, NullS, NullS, field_info->field_name); > + if (add_item_to_list(thd, field)) > + return 1; > + > + field->set_name(buffer.ptr(), buffer.length(), system_charset_info); > + return 0; > +} > + > +/** > + @brief Fill records for temporary tables by reading info from table object > + > + @param[in] thd thread handler > + @param[in] table I_S table > + @param[in] tmp_table temporary table > + @param[in] db database name > + > + @return Operation status > + @retval 0 success > + @retval 1 error > +*/ > + > +static int store_temporary_table_record(THD *thd, TABLE *table, TABLE *tmp_table, const char *db, bool table_name_only) > +{ > + CHARSET_INFO *cs= system_charset_info; > + DBUG_ENTER("store_temporary_table_record"); > + > + if (db && my_strcasecmp(cs, db, tmp_table->s->db.str)) > + DBUG_RETURN(0); > + > + restore_record(table, s->default_values); > + > + //session_id > + table->field[0]->store((longlong) thd->thread_id, TRUE); > + > + //database > + table->field[1]->store(tmp_table->s->db.str, tmp_table->s->db.length, cs); > + > + //table > + table->field[2]->store(tmp_table->s->table_name.str, tmp_table->s->table_name.length, cs); > + > + if (table_name_only) > + DBUG_RETURN(schema_table_store_record(thd, table)); > + > + //engine > + handler *handle= tmp_table->file; > + char *engineType = (char *)(handle ? handle->table_type() : "UNKNOWN"); > + table->field[3]->store(engineType, strlen(engineType), cs); > + > + //name > + if (tmp_table->s->path.str) { > + char *p=strstr(tmp_table->s->path.str, "#sql"); > + int len=tmp_table->s->path.length-(p-tmp_table->s->path.str); > + table->field[4]->store(p, min(FN_REFLEN, len), cs); > + } > + > + // file stats > + handler *file= tmp_table->file; > + > + if (file) { > + > + MYSQL_TIME time; > + > + /** > + TODO: InnoDB stat(file) checks file on short names within data dictionary > + rather than using full path, because of that, temp files created in > + TMPDIR will not have access/create time as it will not find the file > + > + The fix is to patch InnoDB to use full path > + */ > + file->info(HA_STATUS_VARIABLE | HA_STATUS_TIME | HA_STATUS_NO_LOCK); > + > + table->field[5]->store((longlong) file->stats.records, TRUE); > + table->field[5]->set_notnull(); > + > + table->field[6]->store((longlong) file->stats.mean_rec_length, TRUE); > + table->field[7]->store((longlong) file->stats.data_file_length, TRUE); > + table->field[8]->store((longlong) file->stats.index_file_length, TRUE); > + if (file->stats.create_time) > + { > + thd->variables.time_zone->gmt_sec_to_TIME(&time, > + (my_time_t) file->stats.create_time); > + table->field[9]->store_time(&time, MYSQL_TIMESTAMP_DATETIME); > + table->field[9]->set_notnull(); > + } > + if (file->stats.update_time) > + { > + thd->variables.time_zone->gmt_sec_to_TIME(&time, > + (my_time_t) file->stats.update_time); > + table->field[10]->store_time(&time, MYSQL_TIMESTAMP_DATETIME); > + table->field[10]->set_notnull(); > + } > + } > + > + DBUG_RETURN(schema_table_store_record(thd, table)); > +} > + > +/** > + @brief Fill I_S tables with global temporary tables > + > + @param[in] thd thread handler > + @param[in] tables I_S table > + @param[in] cond 'WHERE' condition > + > + @return Operation status > + @retval 0 success > + @retval 1 error > +*/ > + > +static int fill_global_temporary_tables(THD *thd, TABLE_LIST *tables, COND *cond) > +{ > + DBUG_ENTER("fill_global_temporary_tables"); > + > + pthread_mutex_lock(&LOCK_thread_count); > + > + bool table_names_only= (thd->lex->sql_command == SQLCOM_SHOW_TEMPORARY_TABLES) ? 1 : 0; > + I_List_iterator<THD> it(threads); > + THD *thd_item; > + TABLE *tmp; > + > + //TODO: ACL check when I_S used directly on global temp tables > + while ((thd_item=it++)) { > + for (tmp=thd_item->temporary_tables; tmp; tmp=tmp->next) { > + > + THD *t= tmp->in_use; > + tmp->in_use= thd; > + > + if (store_temporary_table_record(thd_item, tables->table, tmp, thd->lex->select_lex.db, table_names_only)) { > + tmp->in_use= t; > + pthread_mutex_unlock(&LOCK_thread_count); > + DBUG_RETURN(1); > + } > + > + tmp->in_use= t; > + } > + } > + > + pthread_mutex_unlock(&LOCK_thread_count); > + DBUG_RETURN(0); > +} > + > +/** > + @brief Fill I_S tables with session temporary tables > + > + @param[in] thd thread handler > + @param[in] tables I_S table > + @param[in] cond 'WHERE' condition > + > + @return Operation status > + @retval 0 success > + @retval 1 error > +*/ > + > +int fill_temporary_tables(THD *thd, TABLE_LIST *tables, COND *cond) > +{ > + DBUG_ENTER("fill_temporary_tables"); > + > + if (thd->lex->option_type == OPT_GLOBAL) > + DBUG_RETURN(fill_global_temporary_tables(thd, tables, cond)); > + > + bool table_names_only= (thd->lex->sql_command == SQLCOM_SHOW_TEMPORARY_TABLES) ? 1 : 0; > + TABLE *tmp; > + > + for (tmp=thd->temporary_tables; tmp; tmp=tmp->next) { > + if (store_temporary_table_record(thd, tables->table, tmp, thd->lex->select_lex.db, table_names_only)) { > + DBUG_RETURN(1); > + } > + } > + DBUG_RETURN(0); > +} > > /** > @brief Fill I_S table with data from FRM file only > @@ -6259,6 +6466,25 @@ > {0, 0, MYSQL_TYPE_STRING, 0, 0, 0, SKIP_OPEN_TABLE} > }; > > +ST_FIELD_INFO temporary_table_fields_info[]= > +{ > + {"SESSION_ID", 4, MYSQL_TYPE_LONGLONG, 0, 0, "Session", SKIP_OPEN_TABLE}, > + {"TABLE_SCHEMA", NAME_CHAR_LEN, MYSQL_TYPE_STRING, 0, 0, "Db", SKIP_OPEN_TABLE}, > + {"TABLE_NAME", NAME_CHAR_LEN, MYSQL_TYPE_STRING, 0, 0, "Temp_tables_in_", SKIP_OPEN_TABLE}, > + {"ENGINE", NAME_CHAR_LEN, MYSQL_TYPE_STRING, 0, 0, "Engine", OPEN_FRM_ONLY}, > + {"NAME", FN_REFLEN, MYSQL_TYPE_STRING, 0, 0, "Name", SKIP_OPEN_TABLE}, > + {"TABLE_ROWS", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, 0, > + MY_I_S_UNSIGNED, "Rows", OPEN_FULL_TABLE}, > + {"AVG_ROW_LENGTH", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, 0, > + MY_I_S_UNSIGNED, "Avg Row", OPEN_FULL_TABLE}, > + {"DATA_LENGTH", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, 0, > + MY_I_S_UNSIGNED, "Data Length", OPEN_FULL_TABLE}, > + {"INDEX_LENGTH", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, 0, > + MY_I_S_UNSIGNED, "Index Size", OPEN_FULL_TABLE}, > + {"CREATE_TIME", 0, MYSQL_TYPE_DATETIME, 0, 1, "Create Time", OPEN_FULL_TABLE}, > + {"UPDATE_TIME", 0, MYSQL_TYPE_DATETIME, 0, 1, "Update Time", OPEN_FULL_TABLE}, > + {0, 0, MYSQL_TYPE_STRING, 0, 0, 0, SKIP_OPEN_TABLE} > +}; > > ST_FIELD_INFO columns_fields_info[]= > { > @@ -6817,6 +7043,9 @@ > fill_schema_files, 0, 0, -1, -1, 0, 0}, > {"GLOBAL_STATUS", variables_fields_info, create_schema_table, > fill_status, make_old_format, 0, 0, -1, 0, 0}, > + {"GLOBAL_TEMPORARY_TABLES", temporary_table_fields_info, create_schema_table, > + fill_global_temporary_tables, make_temporary_tables_old_format, 0, 2, 3, 0, > + OPEN_TABLE_ONLY|OPTIMIZE_I_S_TABLE}, > {"GLOBAL_VARIABLES", variables_fields_info, create_schema_table, > fill_variables, make_old_format, 0, 0, -1, 0, 0}, > {"KEY_COLUMN_USAGE", key_column_usage_fields_info, create_schema_table, > @@ -6860,6 +7089,9 @@ > get_all_tables, make_table_names_old_format, 0, 1, 2, 1, 0}, > {"TABLE_PRIVILEGES", table_privileges_fields_info, create_schema_table, > fill_schema_table_privileges, 0, 0, -1, -1, 0, 0}, > + {"TEMPORARY_TABLES", temporary_table_fields_info, create_schema_table, > + fill_temporary_tables, make_temporary_tables_old_format, 0, 2, 3, 0, > + OPEN_TABLE_ONLY|OPTIMIZE_I_S_TABLE}, > {"TRIGGERS", triggers_fields_info, create_schema_table, > get_all_tables, make_old_format, get_schema_triggers_record, 5, 6, 0, > OPEN_TABLE_ONLY}, > > === modified file 'sql/sql_yacc.yy' > --- sql/sql_yacc.yy 2010-01-17 17:22:46 +0000 > +++ sql/sql_yacc.yy 2010-02-23 21:17:16 +0000 > @@ -10035,6 +10035,15 @@ > if (prepare_schema_table(YYTHD, lex, 0, SCH_TABLE_NAMES)) > MYSQL_YYABORT; > } > + | opt_var_type TEMPORARY TABLES opt_db > + { > + LEX *lex= Lex; > + lex->sql_command= SQLCOM_SHOW_TEMPORARY_TABLES; > + lex->option_type= $1; > + lex->select_lex.db= $4; > + if (prepare_schema_table(YYTHD, lex, 0, SCH_TEMPORARY_TABLES)) > + MYSQL_YYABORT; > + } > | opt_full TRIGGERS_SYM opt_db wild_and_where > { > LEX *lex= Lex; > > === modified file 'sql/table.h' > --- sql/table.h 2010-02-10 19:06:24 +0000 > +++ sql/table.h 2010-02-23 21:17:16 +0000 > @@ -942,6 +942,7 @@ > SCH_FILES, > SCH_GLOBAL_STATUS, > SCH_GLOBAL_VARIABLES, > + SCH_GLOBAL_TEMPORARY_TABLES, > SCH_KEY_COLUMN_USAGE, > SCH_OPEN_TABLES, > SCH_PARTITIONS, > @@ -960,6 +961,7 @@ > SCH_TABLE_CONSTRAINTS, > SCH_TABLE_NAMES, > SCH_TABLE_PRIVILEGES, > + SCH_TEMPORARY_TABLES, > SCH_TRIGGERS, > SCH_USER_PRIVILEGES, > SCH_VARIABLES, > > >

1 0

[Maria-developers] Rev 2752: Subquery optimization: Duplicate Elimination: process in file:///home/psergey/dev/maria-5.3-subqueries-r3/
by Sergey Petrunya 23 Feb '10

23 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r3/ ------------------------------------------------------------ revno: 2752 revision-id: psergey(a)askmonty.org-20100208125530-l0o01n6lxk95cn78 parent: psergey(a)askmonty.org-20100208095616-ar502xdhoqqhe3nk committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r3 timestamp: Mon 2010-02-08 14:55:30 +0200 message: Subquery optimization: Duplicate Elimination: process temporary table overflow correctly. === modified file 'sql/sql_select.cc' --- a/sql/sql_select.cc 2010-01-28 13:48:33 +0000 +++ b/sql/sql_select.cc 2010-02-08 12:55:30 +0000 @@ -16250,12 +16250,12 @@ if (error) { /* create_internal_tmp_table_from_heap will generate error if needed */ - if (sjtbl->tmp_table->file->is_fatal_error(error, HA_CHECK_DUP) && - create_internal_tmp_table_from_heap(thd, sjtbl->tmp_table, + if (!sjtbl->tmp_table->file->is_fatal_error(error, HA_CHECK_DUP)) + DBUG_RETURN(1); /* Duplicate */ + if (create_internal_tmp_table_from_heap(thd, sjtbl->tmp_table, sjtbl->start_recinfo, &sjtbl->recinfo, error, 1)) DBUG_RETURN(-1); - DBUG_RETURN(1); } DBUG_RETURN(0); }

2 1

[Maria-developers] Rev 2751: Fix for previous: pass correct parameters to mark_as_dependent. in file:///home/psergey/dev/maria-5.3-subqueries-r3/
by Sergey Petrunya 23 Feb '10

23 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r3/ ------------------------------------------------------------ revno: 2751 revision-id: psergey(a)askmonty.org-20100208095616-ar502xdhoqqhe3nk parent: psergey(a)askmonty.org-20100202200045-13q0nb5dwzm739j6 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r3 timestamp: Mon 2010-02-08 11:56:16 +0200 message: Fix for previous: pass correct parameters to mark_as_dependent. === modified file 'sql/item.cc' --- a/sql/item.cc 2010-02-02 20:00:45 +0000 +++ b/sql/item.cc 2010-02-08 09:56:16 +0000 @@ -4220,7 +4220,7 @@ return -1; mark_as_dependent(thd, last_checked_context->select_lex, - context->select_lex, this, + context->select_lex, rf /*this*/ , //psergey-fix: s/this/rf/ rf); return 0; } @@ -6000,7 +6000,8 @@ goto error; thd->change_item_tree(reference, fld); mark_as_dependent(thd, last_checked_context->select_lex, - thd->lex->current_select, this, fld); + thd->lex->current_select, fld, fld); //psergey-fix-todo: use reference instead? + // was: this, fld) /* A reference is resolved to a nest level that's outer or the same as the nest level of the enclosing set function : adjust the value of

2 1

Re: [Maria-developers] Rev 2740: Group commit for maria storage engine. in file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/
by Michael Widenius 23 Feb '10

23 Feb '10

Hi! >>>>> "Sergei" == Sergei Golubchik <serg(a)askmonty.org> writes: Sergei> Hi, Michael! Sergei> On Feb 11, Michael Widenius wrote: >> >> Yes, but if you are only looking at one value, without regard of the >> other, then you don't need to use my_atomic_load32() Sergei> if 32-bit reads are always atomic - yes. Sergei> I suppose we can assume that they are. >> Another problem is that if the variables are related in any way, >> atomic_load32 is not good enough as you can get 'a' and 'b' from >> different instances in time. >> >> for example: >> >> a=b= 0; >> >> T1: my_atomic_store32(&a, 1); >> T1: my_atomic_store32(&b, 2); >> >> T2: my_atomic_store32(&a, 3); >> T2: my_atomic_store32(&b, 4); >> >> T3: a1= my_atomic_load32(&a); >> T3: b1= my_atomic_load32(&b); >> >> this means that you can get the following values for a1 and b1: >> >> a1= 0 b1= 0 >> a1= 0 b1= 2 >> a1= 1 b1= 2 >> a1= 1 b1= 4 >> a1= 3 b1= 2 >> a1= 3 b1= 4 Sergei> and also Sergei> a1=0 b1=4 Sergei> a1=1 b1=0 ok, I missed the above one. Sergei> a1=3 b1=0 Ok, that is right. However, my intention was actually to have T2 to be T1 (ie same process), in which case this option would be impossible. Sergei> all the 9 variants are possible. Sanja, when considering the above, are you sure your usage of my_atomic_load/my_atmoic_store is correct? For example: + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + max= my_atomic_load32(&soft_sync_max); + sync_request= my_atomic_load32(&soft_need_sync); + my_atomic_store32(&soft_sync_min, max); + my_atomic_store32(&soft_need_sync, 0); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); In the above code, there is no gurantee that you read or write related soft_sync_min and soft_sync_max values. (See above table). Also, as we concluded with Sergei, any code like: + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); There is no reason to have an my_atomic_rwlock instruction. Regards, Monty

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2821: fix for a possible DoS in the my_net_skip_rest()
by noreply＠launchpad.net 23 Feb '10

23 Feb '10

------------------------------------------------------------ revno: 2821 committer: Sergei Golubchik <sergii(a)pisem.net> branch nick: maria-5.1 timestamp: Tue 2010-02-23 13:04:58 +0100 message: fix for a possible DoS in the my_net_skip_rest() modified: include/mysql.h.pp include/mysql_com.h sql/net_serv.cc sql/sql_connect.cc -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2820: Fixed LP#524679: make test ORDER BY date_ord ASC
by noreply＠launchpad.net 23 Feb '10

23 Feb '10

------------------------------------------------------------ revno: 2820 committer: Michael Widenius <monty(a)askmonty.org> branch nick: maria-5.1 timestamp: Tue 2010-02-23 14:06:41 +0200 message: Fixed LP#524679: make test ORDER BY date_ord ASC (Problem was missing time_zone setting) modified: mysql-test/t/func_if.test -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2820)
by Michael Widenius 23 Feb '10

23 Feb '10

#At lp:maria based on revid:psergey@askmonty.org-20100219141349-bvecf7xaj7fzodw0 2820 Michael Widenius 2010-02-23 Fixed LP#524679: make test ORDER BY date_ord ASC (Problem was missing time_zone setting) modified: mysql-test/t/func_if.test === modified file 'mysql-test/t/func_if.test' --- a/mysql-test/t/func_if.test 2008-12-12 13:16:25 +0000 +++ b/mysql-test/t/func_if.test 2010-02-23 12:06:41 +0000 @@ -5,6 +5,11 @@ drop table if exists t1; --enable_warnings +--disable_query_log +# Set timezone to something specific, to make it possible to use unix_timestamp +set time_zone="+03:00"; +--enable_query_log + # # Simple IF tests # @@ -163,5 +168,9 @@ SELECT * FROM (SELECT MAX(IFNULL(CAST(c DROP TABLE t1; - --echo End of 5.0 tests + +--disable_query_log +# Restore timezone to default +set time_zone= @@global.time_zone; +--enable_query_log

1 0

[Maria-developers] Rev 2768: Subquery backport: update pbxt suite test results (checked). in file:///home/psergey/dev/maria-5.3-subqueries-r7/
by Sergey Petrunya 23 Feb '10

23 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r7/ ------------------------------------------------------------ revno: 2768 revision-id: psergey(a)askmonty.org-20100223092202-sc3huivw9yymnj5d parent: psergey(a)askmonty.org-20100221075312-fc08qgn72dnbudd5 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r7 timestamp: Tue 2010-02-23 11:22:02 +0200 message: Subquery backport: update pbxt suite test results (checked). === modified file 'mysql-test/suite/pbxt/r/group_min_max.result' --- a/mysql-test/suite/pbxt/r/group_min_max.result 2009-08-17 15:57:58 +0000 +++ b/mysql-test/suite/pbxt/r/group_min_max.result 2010-02-23 09:22:02 +0000 @@ -2257,7 +2257,7 @@ a IN (SELECT max(b) FROM t1 GROUP BY a HAVING a < 2); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t1_outer index NULL a 10 NULL 15 Using where; Using index -2 DEPENDENT SUBQUERY t1 index NULL a 10 NULL 1 Using index +2 SUBQUERY t1 index NULL a 10 NULL 15 Using index EXPLAIN SELECT 1 FROM t1 AS t1_outer GROUP BY a HAVING a > (SELECT max(b) FROM t1 GROUP BY a HAVING a < 2); id select_type table type possible_keys key key_len ref rows Extra === modified file 'mysql-test/suite/pbxt/r/subselect.result' --- a/mysql-test/suite/pbxt/r/subselect.result 2009-12-16 09:28:51 +0000 +++ b/mysql-test/suite/pbxt/r/subselect.result 2010-02-23 09:22:02 +0000 @@ -1293,31 +1293,31 @@ 4 explain extended select * from t2 where t2.a in (select a from t1); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL PRIMARY 4 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 unique_subquery PRIMARY PRIMARY 4 func 1 100.00 Using index +1 PRIMARY t2 index PRIMARY PRIMARY 4 NULL 4 100.00 Using index +1 PRIMARY t1 index PRIMARY PRIMARY 4 NULL 4 75.00 Using where; Using index; Using join buffer Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(<primary_index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on PRIMARY))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t1` join `test`.`t2` where (`test`.`t1`.`a` = `test`.`t2`.`a`) select * from t2 where t2.a in (select a from t1 where t1.b <> 30); a 2 4 explain extended select * from t2 where t2.a in (select a from t1 where t1.b <> 30); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL PRIMARY 4 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 unique_subquery PRIMARY PRIMARY 4 func 1 100.00 Using where +1 PRIMARY t2 index PRIMARY PRIMARY 4 NULL 4 100.00 Using index +1 PRIMARY t1 ALL PRIMARY NULL NULL NULL 4 75.00 Using where; Using join buffer Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(<primary_index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on PRIMARY where ((`test`.`t1`.`b` <> 30) and (<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`))))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t1` join `test`.`t2` where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t1`.`b` <> 30)) select * from t2 where t2.a in (select t1.a from t1,t3 where t1.b=t3.a); a 2 3 explain extended select * from t2 where t2.a in (select t1.a from t1,t3 where t1.b=t3.a); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL PRIMARY 4 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 eq_ref PRIMARY PRIMARY 4 func 1 100.00 -2 DEPENDENT SUBQUERY t3 eq_ref PRIMARY PRIMARY 4 test.t1.b 1 100.00 Using index +1 PRIMARY t2 index PRIMARY PRIMARY 4 NULL 4 100.00 Using index +1 PRIMARY t1 ALL PRIMARY NULL NULL NULL 4 75.00 Using where; Using join buffer +1 PRIMARY t3 eq_ref PRIMARY PRIMARY 4 test.t1.b 1 100.00 Using index Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(select 1 AS `Not_used` from `test`.`t1` join `test`.`t3` where ((`test`.`t3`.`a` = `test`.`t1`.`b`) and (<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`)))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t1` join `test`.`t3` join `test`.`t2` where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t3`.`a` = `test`.`t1`.`b`)) drop table t1, t2, t3; create table t1 (a int, b int, index a (a,b)); create table t2 (a int, index a (a)); @@ -1332,31 +1332,31 @@ 4 explain extended select * from t2 where t2.a in (select a from t1); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL a 5 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 index_subquery a a 5 func 1 100.00 Using index +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index +1 PRIMARY t1 ref a a 5 test.t2.a 1 100.00 Using index; FirstMatch(t2) Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on a))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1`) where (`test`.`t1`.`a` = `test`.`t2`.`a`) select * from t2 where t2.a in (select a from t1 where t1.b <> 30); a 2 4 explain extended select * from t2 where t2.a in (select a from t1 where t1.b <> 30); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL a 5 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 index_subquery a a 5 func 1 100.00 Using index; Using where +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index +1 PRIMARY t1 ref a a 5 test.t2.a 1 100.00 Using where; Using index; FirstMatch(t2) Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on a where ((`test`.`t1`.`b` <> 30) and (<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`))))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1`) where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t1`.`b` <> 30)) select * from t2 where t2.a in (select t1.a from t1,t3 where t1.b=t3.a); a 2 3 explain extended select * from t2 where t2.a in (select t1.a from t1,t3 where t1.b=t3.a); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL a 5 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 ref a a 5 func 1 100.00 Using index -2 DEPENDENT SUBQUERY t3 ref a a 5 test.t1.b 1 100.00 Using index +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index +1 PRIMARY t1 ref a a 5 test.t2.a 1 100.00 Using index +1 PRIMARY t3 ref a a 5 test.t1.b 1 100.00 Using index; FirstMatch(t2) Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(select 1 AS `Not_used` from `test`.`t1` join `test`.`t3` where ((`test`.`t3`.`a` = `test`.`t1`.`b`) and (<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`)))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1` join `test`.`t3`) where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t3`.`a` = `test`.`t1`.`b`)) insert into t1 values (3,31); select * from t2 where t2.a in (select a from t1 where t1.b <> 30); a @@ -1369,10 +1369,10 @@ 4 explain extended select * from t2 where t2.a in (select a from t1 where t1.b <> 30); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL a 5 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 index_subquery a a 5 func 1 100.00 Using index; Using where +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index +1 PRIMARY t1 ref a a 5 test.t2.a 1 100.00 Using where; Using index; FirstMatch(t2) Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on a where ((`test`.`t1`.`b` <> 30) and (<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`))))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1`) where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t1`.`b` <> 30)) drop table t1, t2, t3; create table t1 (a int, b int); create table t2 (a int, b int); @@ -2823,10 +2823,10 @@ Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one` AS `one`,`test`.`t2`.`two` AS `two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`))) and trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`)))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`))))) AS `test` from `test`.`t1` explain extended SELECT one,two from t1 where ROW(one,two) IN (SELECT one,two FROM t2 WHERE flag = 'N'); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 Using where -2 DEPENDENT SUBQUERY t2 ALL NULL NULL NULL NULL 9 100.00 Using where +1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 +1 PRIMARY t2 ALL NULL NULL NULL NULL 9 100.00 Using where; FirstMatch(t1) Warnings: -Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two` from `test`.`t1` where <in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one` AS `one`,`test`.`t2`.`two` AS `two` from `test`.`t2` where ((`test`.`t2`.`flag` = 'N') and (<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) and (<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`)))) +Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t2`.`two` = `test`.`t1`.`two`) and (`test`.`t2`.`one` = `test`.`t1`.`one`) and (`test`.`t2`.`flag` = 'N')) explain extended SELECT one,two,ROW(one,two) IN (SELECT one,two FROM t2 WHERE flag = '0' group by one,two) as 'test' from t1; id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 @@ -3412,7 +3412,7 @@ SELECT * FROM t1 WHERE (a,b) = ANY (SELECT a, max(b) FROM t1 GROUP BY a); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t1 ALL NULL NULL NULL NULL 9 Using where -2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 9 Using temporary; Using filesort +2 SUBQUERY t1 ALL NULL NULL NULL NULL 9 Using temporary; Using filesort ALTER TABLE t1 ADD INDEX(a); SELECT * FROM t1 WHERE (a,b) = ANY (SELECT a, max(b) FROM t1 GROUP BY a); a b @@ -3423,7 +3423,7 @@ SELECT * FROM t1 WHERE (a,b) = ANY (SELECT a, max(b) FROM t1 GROUP BY a); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t1 ALL NULL NULL NULL NULL 9 Using where -2 DEPENDENT SUBQUERY t1 index NULL a 8 NULL 1 Using filesort +2 SUBQUERY t1 ALL NULL NULL NULL NULL 9 Using temporary; Using filesort DROP TABLE t1; create table t1( f1 int,f2 int); insert into t1 values (1,1),(2,2); @@ -4213,8 +4213,8 @@ CREATE INDEX I2 ON t1 (b); EXPLAIN SELECT a,b FROM t1 WHERE b IN (SELECT a FROM t1); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 2 Using where -2 DEPENDENT SUBQUERY t1 index_subquery I1 I1 2 func 1 Using index; Using where +1 PRIMARY t1 index I1 I1 2 NULL 2 Using index; LooseScan +1 PRIMARY t1 ref I2 I2 13 test.t1.a 1 Using where SELECT a,b FROM t1 WHERE b IN (SELECT a FROM t1); a b CREATE TABLE t2 (a VARCHAR(1), b VARCHAR(10)); @@ -4223,15 +4223,15 @@ CREATE INDEX I2 ON t2 (b); EXPLAIN SELECT a,b FROM t2 WHERE b IN (SELECT a FROM t2); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t2 ALL NULL NULL NULL NULL 2 Using where -2 DEPENDENT SUBQUERY t2 index_subquery I1 I1 4 func 1 Using index; Using where +1 PRIMARY t2 index I1 I1 4 NULL 2 Using index; LooseScan +1 PRIMARY t2 ref I2 I2 13 test.t2.a 1 Using where SELECT a,b FROM t2 WHERE b IN (SELECT a FROM t2); a b EXPLAIN SELECT a,b FROM t1 WHERE b IN (SELECT a FROM t1 WHERE LENGTH(a)<500); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 2 Using where -2 DEPENDENT SUBQUERY t1 index_subquery I1 I1 2 func 1 Using index; Using where +1 PRIMARY t1 index I1 I1 2 NULL 2 Using where; Using index; LooseScan +1 PRIMARY t1 ref I2 I2 13 test.t1.a 1 Using where SELECT a,b FROM t1 WHERE b IN (SELECT a FROM t1 WHERE LENGTH(a)<500); a b DROP TABLE t1,t2;

1 0

[Maria-developers] New (by Monty): Add exit_connect, start_server, stop_server hooks (87)
by worklog-noreply＠askmonty.org 22 Feb '10

22 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add exit_connect, start_server, stop_server hooks CREATION DATE..: Mon, 22 Feb 2010, 19:51 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 87 (http://askmonty.org/worklog/?tid=87) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 20 (hours remain) ORIG. ESTIMATE.: 20 PROGRESS NOTES: DESCRIPTION: Add exit_connect, start_server, stop_server hooks. These would be similar to the init_connect command that is executed when a connection is created. The exit_connect command would be executed as the connection user. The start_server and stop_server should be run as user 'root' with SUPER_ACL privileges) (Note that this is troublesome as the server may not have a 'root' user, so we may want to allow the user to specify that name of the super user on the command line) ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Rev 2734: Maria WL#61 in file:///Users/bell/maria/bzr/work-maria-5.2-engine/
by sanja＠askmonty.org 22 Feb '10

22 Feb '10

At file:///Users/bell/maria/bzr/work-maria-5.2-engine/ ------------------------------------------------------------ revno: 2734 revision-id: sanja(a)askmonty.org-20091204114937-cfkvax0g36d3nq3j parent: psergey(a)askmonty.org-20091202142609-18bp41q8mejxl47t committer: sanja(a)askmonty.org branch nick: work-maria-5.2-engine timestamp: Fri 2009-12-04 13:49:37 +0200 message: Maria WL#61 Interface for maria extensions. Additional information about plugins (maturity and string version) interface for maria extensions. === modified file 'CMakeLists.txt' --- a/CMakeLists.txt 2009-10-03 19:24:13 +0000 +++ b/CMakeLists.txt 2009-12-04 11:49:37 +0000 @@ -251,6 +251,7 @@ IF (ENGINE_BUILD_TYPE STREQUAL "STATIC") SET (mysql_plugin_defs "${mysql_plugin_defs},builtin_${PLUGIN_NAME}_plugin") + SET (mariaext_plugin_defs "${mariaext_plugin_defs},bltnmext_${PLUGIN_NAME}_plugin") SET (MYSQLD_STATIC_ENGINE_LIBS ${MYSQLD_STATIC_ENGINE_LIBS} ${PLUGIN_NAME}) SET (STORAGE_ENGINE_DEFS "${STORAGE_ENGINE_DEFS} -DWITH_${ENGINE}_STORAGE_ENGINE") SET (WITH_${ENGINE}_STORAGE_ENGINE TRUE) @@ -269,6 +270,7 @@ IF(NOT WITHOUT_PARTITION_STORAGE_ENGINE) SET (STORAGE_ENGINE_DEFS "${STORAGE_ENGINE_DEFS} -DWITH_PARTITION_STORAGE_ENGINE") SET (mysql_plugin_defs "${mysql_plugin_defs},builtin_partition_plugin") + SET (mariaext_plugin_defs "${mariaext_plugin_defs},bltnmext_partition_plugin") ENDIF(NOT WITHOUT_PARTITION_STORAGE_ENGINE) # Special handling for tmp tables with the maria engine === modified file 'config/ac-macros/plugins.m4' --- a/config/ac-macros/plugins.m4 2009-04-25 10:05:32 +0000 +++ b/config/ac-macros/plugins.m4 2009-12-04 11:49:37 +0000 @@ -461,6 +461,7 @@ ]) ]) mysql_plugin_defs="$mysql_plugin_defs, [builtin_]$2[_plugin]" + mariaext_plugin_defs="$mariaext_plugin_defs, [bltnmext_]$2[_plugin]" [with_plugin_]$2=yes AC_MSG_RESULT([yes]) m4_ifdef([$11],[ === modified file 'configure.in' --- a/configure.in 2009-11-12 04:31:28 +0000 +++ b/configure.in 2009-12-04 11:49:37 +0000 @@ -2842,6 +2842,7 @@ AC_SUBST(mysql_plugin_dirs) AC_SUBST(mysql_plugin_libs) AC_SUBST(mysql_plugin_defs) +AC_SUBST(mariaext_plugin_defs) # Now that sql_client_dirs and sql_server_dirs are stable, determine the union. === modified file 'include/mysql/plugin.h' --- a/include/mysql/plugin.h 2009-09-07 20:50:10 +0000 +++ b/include/mysql/plugin.h 2009-12-04 11:49:37 +0000 @@ -65,7 +65,10 @@ Plugin API. Common for all plugin types. */ +/* MySQL plugin interface version */ #define MYSQL_PLUGIN_INTERFACE_VERSION 0x0100 +/* MariaDB extentsion interface version */ +#define MARIAEXT_PLUGIN_INTERFACE_VERSION 0x0100 /* The allowable types of plugins @@ -86,6 +89,21 @@ #define PLUGIN_LICENSE_GPL_STRING "GPL" #define PLUGIN_LICENSE_BSD_STRING "BSD" +/* definitions of code maturity for plugins */ +#define PLUGIN_MATURITY_UNKNOWN 0 +#define PLUGIN_MATURITY_TEST 1 +#define PLUGIN_MATURITY_ALPHA 2 +#define PLUGIN_MATURITY_BETA 3 +#define PLUGIN_MATURITY_GAMMA 4 +#define PLUGIN_MATURITY_RELEASE 5 + +#define PLUGIN_MATURITY_UNKNOWN_STR "Unknown" +#define PLUGIN_MATURITY_TEST_STR "Test" +#define PLUGIN_MATURITY_ALPHA_STR "Alpha" +#define PLUGIN_MATURITY_BETA_STR "Beta" +#define PLUGIN_MATURITY_GAMMA_STR "Gamma" +#define PLUGIN_MATURITY_RELEASE_STR "Release" + /* Macros for beginning and ending plugin declarations. Between mysql_declare_plugin and mysql_declare_plugin_end there should @@ -94,15 +112,29 @@ #ifndef MYSQL_DYNAMIC_PLUGIN + #define __MYSQL_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ int VERSION= MYSQL_PLUGIN_INTERFACE_VERSION; \ int PSIZE= sizeof(struct st_mysql_plugin); \ struct st_mysql_plugin DECLS[]= { + +#define __MARIAEXT_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ +int VERSION= MARIAEXT_PLUGIN_INTERFACE_VERSION; \ +int PSIZE= sizeof(struct st_mariaext_plugin); \ +struct st_mariaext_plugin DECLS[]= { + #else + #define __MYSQL_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ MYSQL_PLUGIN_EXPORT int _mysql_plugin_interface_version_= MYSQL_PLUGIN_INTERFACE_VERSION; \ MYSQL_PLUGIN_EXPORT int _mysql_sizeof_struct_st_plugin_= sizeof(struct st_mysql_plugin); \ MYSQL_PLUGIN_EXPORT struct st_mysql_plugin _mysql_plugin_declarations_[]= { + +#define __MARIAEXT_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ +MYSQL_PLUGIN_EXPORT int _mariaext_plugin_interface_version_= MARIAEXT_PLUGIN_INTERFACE_VERSION; \ +MYSQL_PLUGIN_EXPORT int _mariaext_sizeof_struct_st_plugin_= sizeof(struct st_mariaext_plugin); \ +MYSQL_PLUGIN_EXPORT struct st_mariaext_plugin _mariaext_plugin_declarations_[]= { + #endif #define mysql_declare_plugin(NAME) \ @@ -111,7 +143,14 @@ builtin_ ## NAME ## _sizeof_struct_st_plugin, \ builtin_ ## NAME ## _plugin) +#define mariaext_declare_plugin(NAME) \ +__MARIAEXT_DECLARE_PLUGIN(NAME, \ + bltnmext_ ## NAME ## _plugin_interface_version, \ + bltnmext_ ## NAME ## _sizeof_struct_st_plugin, \ + bltnmext_ ## NAME ## _plugin) + #define mysql_declare_plugin_end ,{0,0,0,0,0,0,0,0,0,0,0,0}} +#define mariaext_declare_plugin_end ,{0,0}} /* declarations for SHOW STATUS support in plugins @@ -407,6 +446,16 @@ void * __reserved1; /* reserved for dependency checking */ }; +/* + MariaDB extension for plugins declaration structure. +*/ + +struct st_mariaext_plugin +{ + const char *sversion; /* plugin version string */ + int maturity; /* HA_PLUGIN_MATURITY_XXX */ +}; + /************************************************************************* API for Full-text parser plugin. (MYSQL_FTPARSER_PLUGIN) */ === modified file 'include/mysql/plugin.h.pp' --- a/include/mysql/plugin.h.pp 2008-10-10 15:28:41 +0000 +++ b/include/mysql/plugin.h.pp 2009-12-04 11:49:37 +0000 @@ -46,6 +46,11 @@ struct st_mysql_sys_var **system_vars; void * __reserved1; }; +struct st_mariaext_plugin +{ + const char *sversion; + int maturity; +}; enum enum_ftparser_mode { MYSQL_FTPARSER_SIMPLE_MODE= 0, === modified file 'plugin/daemon_example/daemon_example.cc' --- a/plugin/daemon_example/daemon_example.cc 2007-06-27 14:49:12 +0000 +++ b/plugin/daemon_example/daemon_example.cc 2009-12-04 11:49:37 +0000 @@ -200,3 +200,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(daemon_example) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_TEST /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'plugin/fulltext/plugin_example.c' --- a/plugin/fulltext/plugin_example.c 2007-04-26 19:26:04 +0000 +++ b/plugin/fulltext/plugin_example.c 2009-12-04 11:49:37 +0000 @@ -270,4 +270,10 @@ NULL } mysql_declare_plugin_end; +mariaext_declare_plugin(ftexample) +{ + "0.01", /* string version */ + PLUGIN_MATURITY_TEST /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'sql/ha_ndbcluster.cc' --- a/sql/ha_ndbcluster.cc 2009-09-07 20:50:10 +0000 +++ b/sql/ha_ndbcluster.cc 2009-12-04 11:49:37 +0000 @@ -10561,5 +10561,11 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(ndbcluster) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_BETA /* maturity */ +} +mariaext_declare_plugin_end; #endif === modified file 'sql/ha_partition.cc' --- a/sql/ha_partition.cc 2009-11-12 04:31:28 +0000 +++ b/sql/ha_partition.cc 2009-12-04 11:49:37 +0000 @@ -6510,5 +6510,11 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(partition) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; #endif === modified file 'sql/log.cc' --- a/sql/log.cc 2009-11-12 04:31:28 +0000 +++ b/sql/log.cc 2009-12-04 11:49:37 +0000 @@ -5795,3 +5795,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(binlog) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'sql/sql_builtin.cc.in' --- a/sql/sql_builtin.cc.in 2006-12-31 01:29:11 +0000 +++ b/sql/sql_builtin.cc.in 2009-12-04 11:49:37 +0000 @@ -16,6 +16,7 @@ #include <mysql/plugin.h> typedef struct st_mysql_plugin builtin_plugin[]; +typedef struct st_mariaext_plugin bltnmext_plugin[]; extern builtin_plugin builtin_binlog_plugin@mysql_plugin_defs@; @@ -25,3 +26,10 @@ builtin_binlog_plugin@mysql_plugin_defs@,(struct st_mysql_plugin *)0 }; +extern bltnmext_plugin + bltnmext_binlog_plugin@mariaext_plugin_defs@; + +struct st_mariaext_plugin *mysqld_bltnmexts[]= +{ + bltnmext_binlog_plugin@mariaext_plugin_defs@,(struct st_mariaext_plugin *)0 +}; === modified file 'sql/sql_plugin.cc' --- a/sql/sql_plugin.cc 2009-11-12 04:31:28 +0000 +++ b/sql/sql_plugin.cc 2009-12-04 11:49:37 +0000 @@ -28,6 +28,9 @@ #endif extern struct st_mysql_plugin *mysqld_builtins[]; +extern struct st_mariaext_plugin *mysqld_bltnmexts[]; +static st_mariaext_plugin no_mariaext[2]= {{"Unknown", 0}, {0, 0}}; +static st_mariaext_plugin *empty_mariaext= no_mariaext; /** @note The order of the enumeration is critical. @@ -82,6 +85,14 @@ "_mysql_sizeof_struct_st_plugin_"; static const char *plugin_declarations_sym= "_mysql_plugin_declarations_"; static int min_plugin_interface_version= MYSQL_PLUGIN_INTERFACE_VERSION & ~0xFF; +static const char *mariaext_plugin_interface_version_sym= + "_mariaext_plugin_interface_version_"; +static const char *mariaext_sizeof_st_plugin_sym= + "_mariaext_sizeof_struct_st_plugin_"; +static const char *mariaext_plugin_declarations_sym= + "_mariaext_plugin_declarations_"; +static int min_mariaext_plugin_interface_version= + MARIAEXT_PLUGIN_INTERFACE_VERSION & ~0xFF; #endif /* Note that 'int version' must be the first field of every plugin @@ -352,6 +363,7 @@ char dlpath[FN_REFLEN]; uint plugin_dir_len, dummy_errors, dlpathlen; struct st_plugin_dl *tmp, plugin_dl; + struct st_mariaext_plugin *mariaext= no_mariaext; void *sym; DBUG_ENTER("plugin_dl_add"); plugin_dir_len= strlen(opt_plugin_dir); @@ -507,6 +519,15 @@ files_charset_info, dl->str, dl->length, system_charset_info, &dummy_errors); plugin_dl.dl.str[plugin_dl.dl.length]= 0; + + if ((sym= dlsym(plugin_dl.handle, mariaext_plugin_interface_version_sym)) && + (*(int *)sym == MARIAEXT_PLUGIN_INTERFACE_VERSION) && + (sym= dlsym(plugin_dl.handle, mariaext_plugin_declarations_sym))) + { + mariaext= (struct st_mariaext_plugin *) sym; + } + plugin_dl.mariaext= mariaext; + /* Add this dll to array */ if (! (tmp= plugin_dl_insert_or_reuse(&plugin_dl))) { @@ -719,6 +740,7 @@ { struct st_plugin_int tmp; struct st_mysql_plugin *plugin; + struct st_mariaext_plugin *ext, *mariaext; DBUG_ENTER("plugin_add"); if (plugin_find_internal(name, MYSQL_ANY_PLUGIN)) { @@ -732,9 +754,18 @@ bzero((char*) &tmp, sizeof(tmp)); if (! (tmp.plugin_dl= plugin_dl_add(dl, report))) DBUG_RETURN(TRUE); + /* Find plugin by name */ - for (plugin= tmp.plugin_dl->plugins; plugin->info; plugin++) + for (plugin= tmp.plugin_dl->plugins, ext= tmp.plugin_dl->mariaext; + plugin->info; + plugin++, ext++) { + mariaext= ext; + if (!ext->sversion) + { + mariaext= empty_mariaext; + ext--; + } uint name_len= strlen(plugin->name); if (plugin->type >= 0 && plugin->type < MYSQL_MAX_PLUGIN_TYPE_NUM && ! my_strnncoll(system_charset_info, @@ -759,6 +790,7 @@ goto err; } tmp.plugin= plugin; + tmp.mariaext= mariaext; tmp.name.str= (char *)plugin->name; tmp.name.length= name_len; tmp.ref_count= 0; @@ -1121,7 +1153,9 @@ uint i; bool is_myisam; struct st_mysql_plugin **builtins; + struct st_mariaext_plugin **bltnmexts; struct st_mysql_plugin *plugin; + struct st_mariaext_plugin *ext, *mariaext; struct st_plugin_int tmp, *plugin_ptr, **reap; MEM_ROOT tmp_root; bool reaped_mandatory_plugin= FALSE; @@ -1160,10 +1194,25 @@ /* First we register builtin plugins */ - for (builtins= mysqld_builtins; *builtins; builtins++) + for (builtins= mysqld_builtins, bltnmexts= mysqld_bltnmexts; + *builtins; + builtins++, bltnmexts++) { - for (plugin= *builtins; plugin->info; plugin++) + /* extensions should be the same numbers as static plugins) */ + DBUG_ASSERT(*bltnmexts); + for (plugin= *builtins, ext= *bltnmexts; + plugin->info; + plugin++, ext++) { + + /* in case if plugin describe less extensions then plugins */ + mariaext= ext; + if (!ext->sversion) + { + mariaext= empty_mariaext; + ext--; + } + if (opt_ignore_builtin_innodb && !my_strnncoll(&my_charset_latin1, (const uchar*) plugin->name, 6, (const uchar*) "InnoDB", 6)) @@ -1186,6 +1235,7 @@ #endif bzero(&tmp, sizeof(tmp)); tmp.plugin= plugin; + tmp.mariaext= mariaext; tmp.name.str= (char *)plugin->name; tmp.name.length= strlen(plugin->name); tmp.state= 0; === modified file 'sql/sql_plugin.h' --- a/sql/sql_plugin.h 2009-05-14 12:03:33 +0000 +++ b/sql/sql_plugin.h 2009-12-04 11:49:37 +0000 @@ -63,6 +63,7 @@ LEX_STRING dl; void *handle; struct st_mysql_plugin *plugins; + struct st_mariaext_plugin *mariaext; int version; uint ref_count; /* number of plugins loaded from the library */ }; @@ -74,6 +75,7 @@ LEX_STRING name; struct st_mysql_plugin *plugin; struct st_plugin_dl *plugin_dl; + struct st_mariaext_plugin *mariaext; uint state; uint ref_count; /* number of threads using the plugin */ void *data; /* plugin type specific, e.g. handlerton */ @@ -95,6 +97,7 @@ #define plugin_name(pi) (&((pi)->name)) #define plugin_state(pi) ((pi)->state) #define plugin_equals(p1,p2) ((p1) == (p2)) +#define plugin_ext(pi) ((pi)->mariaext) #else typedef struct st_plugin_int **plugin_ref; #define plugin_decl(pi) ((pi)[0]->plugin) @@ -103,6 +106,8 @@ #define plugin_name(pi) (&((pi)[0]->name)) #define plugin_state(pi) ((pi)[0]->state) #define plugin_equals(p1,p2) ((p1) && (p2) && (p1)[0] == (p2)[0]) +#define plugin_ext(pi) ((pi)[0]->mariaext) + #endif typedef int (*plugin_type_init)(struct st_plugin_int *); === modified file 'sql/sql_show.cc' --- a/sql/sql_show.cc 2009-11-12 04:31:28 +0000 +++ b/sql/sql_show.cc 2009-12-04 11:49:37 +0000 @@ -100,6 +100,7 @@ TABLE *table= (TABLE*) arg; struct st_mysql_plugin *plug= plugin_decl(plugin); struct st_plugin_dl *plugin_dl= plugin_dlib(plugin); + struct st_mariaext_plugin *mariaext= plugin_ext(plugin); CHARSET_INFO *cs= system_charset_info; char version_buf[20]; @@ -186,6 +187,48 @@ } table->field[9]->set_notnull(); + + switch (mariaext->maturity) { + case PLUGIN_MATURITY_UNKNOWN: + table->field[10]->store(PLUGIN_MATURITY_UNKNOWN_STR, + sizeof(PLUGIN_MATURITY_UNKNOWN_STR) - 1, cs); + break; + case PLUGIN_MATURITY_TEST: + table->field[10]->store(PLUGIN_MATURITY_TEST_STR, + sizeof(PLUGIN_MATURITY_TEST_STR) - 1, cs); + break; + case PLUGIN_MATURITY_ALPHA: + table->field[10]->store(PLUGIN_MATURITY_ALPHA_STR, + sizeof(PLUGIN_MATURITY_ALPHA_STR) - 1, cs); + break; + case PLUGIN_MATURITY_BETA: + table->field[10]->store(PLUGIN_MATURITY_BETA_STR, + sizeof(PLUGIN_MATURITY_BETA_STR) - 1, cs); + break; + case PLUGIN_MATURITY_GAMMA: + table->field[10]->store(PLUGIN_MATURITY_GAMMA_STR, + sizeof(PLUGIN_MATURITY_GAMMA_STR) - 1, cs); + break; + case PLUGIN_MATURITY_RELEASE: + table->field[10]->store(PLUGIN_MATURITY_RELEASE_STR, + sizeof(PLUGIN_MATURITY_RELEASE_STR) - 1, cs); + break; + default: + DBUG_ASSERT(0); + table->field[10]->store("Unknown", 7, cs); + break; + } + table->field[10]->set_notnull(); + + if (mariaext->sversion) + { + table->field[11]->store(mariaext->sversion, + strlen(mariaext->sversion), cs); + table->field[11]->set_notnull(); + } + else + table->field[11]->set_null(); + return schema_table_store_record(thd, table); } @@ -6990,6 +7033,8 @@ {"PLUGIN_AUTHOR", NAME_CHAR_LEN, MYSQL_TYPE_STRING, 0, 1, 0, SKIP_OPEN_TABLE}, {"PLUGIN_DESCRIPTION", 65535, MYSQL_TYPE_STRING, 0, 1, 0, SKIP_OPEN_TABLE}, {"PLUGIN_LICENSE", 80, MYSQL_TYPE_STRING, 0, 1, "License", SKIP_OPEN_TABLE}, + {"PLUGIN_MATURITY", 7, MYSQL_TYPE_STRING, 0, 1, 0, SKIP_OPEN_TABLE}, + {"PLUGIN_AUTH_VERSION", 80, MYSQL_TYPE_STRING, 0, 1, 0, SKIP_OPEN_TABLE}, {0, 0, MYSQL_TYPE_STRING, 0, 0, 0, SKIP_OPEN_TABLE} }; === modified file 'storage/archive/ha_archive.cc' --- a/storage/archive/ha_archive.cc 2009-09-07 20:50:10 +0000 +++ b/storage/archive/ha_archive.cc 2009-12-04 11:49:37 +0000 @@ -1642,4 +1642,10 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(archive) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/blackhole/ha_blackhole.cc' --- a/storage/blackhole/ha_blackhole.cc 2008-11-10 20:21:49 +0000 +++ b/storage/blackhole/ha_blackhole.cc 2009-12-04 11:49:37 +0000 @@ -369,3 +369,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(blackhole) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/csv/ha_tina.cc' --- a/storage/csv/ha_tina.cc 2009-04-25 10:05:32 +0000 +++ b/storage/csv/ha_tina.cc 2009-12-04 11:49:37 +0000 @@ -1636,4 +1636,9 @@ NULL /* config options */ } mysql_declare_plugin_end; - +mariaext_declare_plugin(csv) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/example/ha_example.cc' --- a/storage/example/ha_example.cc 2008-02-24 13:12:17 +0000 +++ b/storage/example/ha_example.cc 2009-12-04 11:49:37 +0000 @@ -906,3 +906,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(example) +{ + "0.1", /* string version */ + PLUGIN_MATURITY_TEST /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/federated/ha_federated.cc' --- a/storage/federated/ha_federated.cc 2009-09-07 20:50:10 +0000 +++ b/storage/federated/ha_federated.cc 2009-12-04 11:49:37 +0000 @@ -3379,3 +3379,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(federated) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_BETA /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/federatedx/ha_federatedx.cc' --- a/storage/federatedx/ha_federatedx.cc 2009-11-03 11:08:09 +0000 +++ b/storage/federatedx/ha_federatedx.cc 2009-12-04 11:49:37 +0000 @@ -3491,3 +3491,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(federated) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_UNKNOWN /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/heap/ha_heap.cc' --- a/storage/heap/ha_heap.cc 2009-09-07 20:50:10 +0000 +++ b/storage/heap/ha_heap.cc 2009-12-04 11:49:37 +0000 @@ -767,3 +767,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(heap) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/ibmdb2i/ha_ibmdb2i.cc' --- a/storage/ibmdb2i/ha_ibmdb2i.cc 2009-07-08 09:10:01 +0000 +++ b/storage/ibmdb2i/ha_ibmdb2i.cc 2009-12-04 11:49:37 +0000 @@ -3357,3 +3357,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(ibmdb2i) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_UNKNOWN /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/innobase/handler/ha_innodb.cc' --- a/storage/innobase/handler/ha_innodb.cc 2009-10-16 22:57:48 +0000 +++ b/storage/innobase/handler/ha_innodb.cc 2009-12-04 11:49:37 +0000 @@ -8684,6 +8684,12 @@ NULL /* reserved */ } mysql_declare_plugin_end; +mariaext_declare_plugin(innobase) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; /** @brief Initialize the default value of innodb_commit_concurrency. === modified file 'storage/innodb_plugin/handler/ha_innodb.cc' --- a/storage/innodb_plugin/handler/ha_innodb.cc 2009-08-04 08:02:48 +0000 +++ b/storage/innodb_plugin/handler/ha_innodb.cc 2009-12-04 11:49:37 +0000 @@ -10032,6 +10032,12 @@ i_s_innodb_cmpmem, i_s_innodb_cmpmem_reset mysql_declare_plugin_end; +mariaext_declare_plugin(innodb_plugin) +{ + INNODB_VERSION_STR, /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; /** @brief Initialize the default value of innodb_commit_concurrency. === modified file 'storage/maria/ha_maria.cc' --- a/storage/maria/ha_maria.cc 2009-10-26 11:35:42 +0000 +++ b/storage/maria/ha_maria.cc 2009-12-04 11:49:37 +0000 @@ -3352,3 +3352,9 @@ NULL } mysql_declare_plugin_end; +mariaext_declare_plugin(maria) +{ + "1.5", /* string version */ + PLUGIN_MATURITY_GAMMA /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/myisam/ha_myisam.cc' --- a/storage/myisam/ha_myisam.cc 2009-10-17 19:12:28 +0000 +++ b/storage/myisam/ha_myisam.cc 2009-12-04 11:49:37 +0000 @@ -2183,6 +2183,12 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(myisam) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; #ifdef HAVE_QUERY_CACHE === modified file 'storage/myisammrg/ha_myisammrg.cc' --- a/storage/myisammrg/ha_myisammrg.cc 2009-10-15 21:38:29 +0000 +++ b/storage/myisammrg/ha_myisammrg.cc 2009-12-04 11:49:37 +0000 @@ -1289,3 +1289,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(myisammrg) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/pbxt/src/ha_pbxt.cc' --- a/storage/pbxt/src/ha_pbxt.cc 2009-09-03 06:15:03 +0000 +++ b/storage/pbxt/src/ha_pbxt.cc 2009-12-04 11:49:37 +0000 @@ -5507,6 +5507,18 @@ drizzle_declare_plugin_end; #else mysql_declare_plugin_end; +#ifdef MARIADB_BASE_VERSION +mariaext_declare_plugin(pbxt) +{ /* PBXT */ + "1.0", /* string version */ + PLUGIN_MATURITY_UNKNOWN /* maturity */ +}, +{ /* PBXT_STATISTICS */ + "1.0", /* string version */ + PLUGIN_MATURITY_UNKNOWN /* maturity */ +} +mariaext_declare_plugin_end; +#endif #endif #if defined(XT_WIN) && defined(XT_COREDUMP) === modified file 'storage/xtradb/handler/ha_innodb.cc' --- a/storage/xtradb/handler/ha_innodb.cc 2009-10-16 22:57:48 +0000 +++ b/storage/xtradb/handler/ha_innodb.cc 2009-12-04 11:49:37 +0000 @@ -10540,6 +10540,69 @@ i_s_innodb_index_stats, i_s_innodb_patches mysql_declare_plugin_end; +mariaext_declare_plugin(innobase) +{ /* InnoDB */ + INNODB_VERSION_STR, /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +}, +{ /* INNODB_RSEG */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_BUFFER_POOL_PAGES */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_BUFFER_POOL_PAGES_INDEX */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_BUFFER_POOL_PAGES_BLOB */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_TRX */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_LOCKS */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_LOCK_WAITS */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_CMP */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_CMP_RESET */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_CMPMEM */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_CMPMEM_RESET */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_TABLE_STATS */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_INDEX_STATS */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* XTRADB_ENHANCEMENTS */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +} +mariaext_declare_plugin_end; + /** @brief Initialize the default value of innodb_commit_concurrency.

3 7

[Maria-developers] We Are Allowed To Add No More Than Two New Log Events :)
by Alexi1952 22 Feb '10

22 Feb '10

Hi Monty, I found out why MySQL/Sun generously allowed to add new log events starting only from number 196. How many log events with numbers >= 196 can be added? Only TWO !!! Indeed, each event must have post header length <= 254 because this length must be saved in the uchar post_header_len[] array (the value 255 is reserved as the "undefined" length value). Now, the post header of the Format description event has the length = number_of_events + 57. Hence max number of possible events in MySQL is 254 - 57 = 197. Funny :) Cheers, Alex

2 1

[Maria-developers] Updated (by Timour): Subquery optimization: Efficient NOT IN execution with NULLs (68)
by worklog-noreply＠askmonty.org 22 Feb '10

22 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subquery optimization: Efficient NOT IN execution with NULLs CREATION DATE..: Fri, 27 Nov 2009, 13:22 SUPERVISOR.....: Monty IMPLEMENTOR....: Timour COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 68 (http://askmonty.org/worklog/?tid=68) VERSION........: Server-9.x STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Timour - Mon, 22 Feb 2010, 17:39)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.17116 2010-02-22 17:39:48.000000000 +0200 +++ /tmp/wklog.68.new.17116 2010-02-22 17:39:48.000000000 +0200 @@ -233,6 +233,7 @@ 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all and v_j1,...,v_jm cannot be null, create for these columns only one index array (and of course do not create any bitmaps for them). +[done] 2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number of rows, where a_i is not null and V(a_i) is the number of distinct @@ -264,6 +265,10 @@ 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. +[This is wrong, because if we don't fill the whole temp table, there may + be some tuple(s) that would match some outer tuple. In such cases, if we + stop filling the temp table, we would miss a TRUE result. Having a partial + match doesn't preclude us from having a complete match]. 8. [timour] Consider that due to materialization, we already have a unique index -=-=(Timour - Tue, 19 Jan 2010, 18:44)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.22569 2010-01-19 18:44:01.000000000 +0200 +++ /tmp/wklog.68.new.22569 2010-01-19 18:44:01.000000000 +0200 @@ -132,11 +132,10 @@ if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE - if (nonull_key) - pq.insert(nonull_key) for (i = 1; i <= n; i++) { + if (vkey[i] != nonull_key) vkey[i].lookup(outer_ref) if (! vkey[i].is_eof()) pq.insert(i) @@ -167,7 +166,7 @@ /* There cannot be a complete match, as we already checked for one. */ assert(matching_keys.elements < n) } - else if (cur_min_key == nonull_key) + else if (vkey[cur_min_key] == nonull_key) { /* The non-NULL key has no corresponding NULL index, so we know for @@ -183,8 +182,10 @@ /* Check if all null_keys contain a NULL at row 'min_row'. The procedure internally checks all keys in a special precomputed order. A prior - procedure determines an optimal order and a mapping - idx_no -> idx_order (encoded as an array). + procedure determines an optimal order and a mapping idx_no -> idx_order + (encoded as an array). + + This procedure makes sure not to match the non-NULL column. */ if (test_null_row(null_keys, min_row)) return TRUE @@ -198,6 +199,14 @@ vkey[cur_min_key].next() if (! vkey[cur_min_key].is_eof()) pq.insert(cur_min_key) + else if (vkey[cur_min_key] == nonull_key) + { + /* + If there can't be more matches for the nonull_key, we know for sure + there is no match, since there is no possible NULL match. + */ + return FALSE + } if (pq.is_empty()) { @@ -216,7 +225,6 @@ } - 3. Directions for improvement ======================================================================== -=-=(Timour - Tue, 19 Jan 2010, 18:29)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.21045 2010-01-19 18:29:12.000000000 +0200 +++ /tmp/wklog.68.new.21045 2010-01-19 18:29:12.000000000 +0200 @@ -132,6 +132,8 @@ if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE + if (nonull_key) + pq.insert(nonull_key) for (i = 1; i <= n; i++) { -=-=(Guest - Tue, 19 Jan 2010, 18:15)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.19825 2010-01-19 18:15:30.000000000 +0200 +++ /tmp/wklog.68.new.19825 2010-01-19 18:15:30.000000000 +0200 @@ -1,8 +1,16 @@ -This a copy of the initial algorithm proposed by Igor: -====================================================== +Contents +======================================================================== -For each left side tuple (v_1,...,v_n) we have to find the following set -of rowids for the temp table containing N rows as the result of +1. Initial idea as proposed by Igor +2. Algorithm for IN execution with partial matching +3. Directions for improvement + + +1. Initial idea as proposed by Igor +======================================================================== + +For each left side tuple (v_1,...,v_n) we have to find the following +set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs @@ -18,38 +26,198 @@ - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap -Taken all above into account I could suggest the following algorithm to -build R: +Taken all above into account I could suggest the following algorithm +to build R: - Using indexes (read about them below) for each column participating in the - intersection, - merge ordered sets rowid{a_i=v_i} in the following manner. + Using indexes (read about them below) for each column participating + in the intersection, merge ordered sets rowid{a_i=v_i} in the + following manner. If a rowid r has been encountered maximum in k sets -rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), + rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is -not in {i1,...,ik}. + not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. -Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i -is null} is either +Here we use the property (1): +any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can -infer that for any r from R -indexes a_i can be uniquely divided into two groups: one contains -indexes a_i where r belongs to -the sets rowid{a_i=v_i}, the other contains indexes a_j such that r -belongs to rowid{a_j is null}. - -Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order -needed for the merge procedure. We could use BTREE indexes for temp -table. But they are rather expensive and -take a lot of memory as the are implemented with RB trees. +infer that for any r from R indexes a_i can be uniquely divided into +two groups: +- one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, +- the other contains indexes a_j such that r belongs to + rowid{a_j is null}. + +Now let's talk how to get elements from rowid{a_i=v_i} in a sorted +order needed for the merge procedure. We could use BTREE indexes for +temp table. But they are rather expensive and take a lot of memory as +the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. +2. Algorithm for IN execution with partial matching +======================================================================== + +2.1 Below is shown the top-level algorithm to execute an IN predicate +with partial matching. This algorithm is essentially the implementation +of Item_subselect:exec(). + +int lookup_with_null_semantics(outer_ref[], mat_subquery) +{ + if (index_lookup(outer_ref, mat_subquery) + return TRUE + else + { + /* + Check if there is a partial match (UNKNOWN) or no match (NULL). + */ + if (this is the first partial match) + { + vkey[] = build array of value keys for each NULL-able column + of mat_subquery. + nkey[] = build a bitmap NULL index for each column of mat_subquery + that contains NULLs + nonull_key = build a key over all non-NULL columns of mat_subquery + } + if (partial_match(outer_ref, vkey[], nkey[], nonull_key) + return UNKNOWN + else + return FALSE + } +} + +2.2 The implementation of partial matching is as follows + +/* + Assumptions: + - It has already been checked if there is a complete match by a + regular index lookup, and the test failed. + - It has already been checked if there is a complete NULL row, + and if there was we wouldn't call this function. Thus we assume + that there is no complete NULL row. + - Not all vidx_i are empty, but some can be empty. If all were empty, + then the only possibility for a match is a complete NULL row, which + we already checked. + + @param outer_ref - the uter (left) IN argument. + @param vidx[] - array of value keys + Ordered sequences of rowids of the corresponding columns a_i, such + that all rowids in idx_i are the ones where column a_i contains some + value or NULL. Each idx_i is derived dynamically, for each different + left argument of an IN predicate. + @param nidx[] - array of NULL keys + Bitmpas, one per each column, where a bit is set if the corresponding + row has a NULL value for the corresponding column. + @nonull_key - the only key over all columns of the materialized subquery + that do not contain NULLs + + @returns + @retval FALSE if there is no match + @retval TRUE if there is a partial match +*/ + +Boolean partial_match(outer_ref, vkey[], nkey[], nonull_key) +{ + /* Set of the keys (columns) that form a partial match. */ + Set matching_keys = {} + /* A subset of all keys that need to be checked for NULL matches. */ + Set null_keys = {} + Int min_key /* Key that contains the current minimum position. */ + Int min_row /* Current row number of min_key. */ + Int cur_min_key, cur_min_row + PriorityQueue pq + + if (nonull_key && ! nonull_key->lookup(outer_ref)) + return FALSE + + for (i = 1; i <= n; i++) + { + vkey[i].lookup(outer_ref) + if (! vkey[i].is_eof()) + pq.insert(i) + } + /* + Not all value keys are empty, thus we don't have only NULL + keys. If we had, the only possible match is a NULL row, and + we cheked there is no such row, therefore the result is known + to be FALSE. + In fact this algorithm makes sense for at least two non-NULL + columns. + */ + assert(pq.elements > 1) + + (min_key, min_row) = pq.pop() + matching_keys.add(min_key) + vkey[min_key].next() + if (! vkey[min_key].is_eof()) + pq.insert(min_key) + + while (TRUE) + { + (cur_min_key, cur_min_row) = pq.pop() + + if (cur_min_row == min_row) + { + matching_keys.add(cur_min_key) + /* There cannot be a complete match, as we already checked for one. */ + assert(matching_keys.elements < n) + } + else if (cur_min_key == nonull_key) + { + /* + The non-NULL key has no corresponding NULL index, so we know for + sure that the row 'min_row' is not a match. + */ + (min_key, min_row) = (cur_min_key, cur_min_row) + matching_keys = {min_key} + } + else + { + assert(cur_min_row > min_row) /* Follows from the use of PQ. */ + null_keys = set_difference(all keys vkey[], matching_keys) + /* + Check if all null_keys contain a NULL at row 'min_row'. The procedure + internally checks all keys in a special precomputed order. A prior + procedure determines an optimal order and a mapping + idx_no -> idx_order (encoded as an array). + */ + if (test_null_row(null_keys, min_row)) + return TRUE + else + { + (min_key, min_row) = (cur_min_key, cur_min_row) + matching_keys = {min_key} + } + } + + vkey[cur_min_key].next() + if (! vkey[cur_min_key].is_eof()) + pq.insert(cur_min_key) + + if (pq.is_empty()) + { + /* Check the last row of the last column in PQ for NULL matches. */ + null_keys = set_difference(all keys vkey[], matching_keys) + if (test_null_row(null_keys, min_row)) + return TRUE + else + return FALSE + } + } + + /* We should never get here. */ + assert(FALSE) + return FALSE +} + + + +3. Directions for improvement +======================================================================== + Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary -=-=(Timour - Sun, 06 Dec 2009, 14:36)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.12919 2009-12-06 14:36:18.000000000 +0200 +++ /tmp/wklog.68.new.12919 2009-12-06 14:36:18.000000000 +0200 @@ -87,3 +87,8 @@ 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. +8. [timour] + Consider that due to materialization, we already have a unique index +on all columns <a_1,..., a_n>. We can use the first key part of this index +over column a_1, instead of the index rowid{a_i=v_i}. Thus we can avoid +creating the index rowid{a_i=v_i}. -=-=(Timour - Fri, 04 Dec 2009, 14:04)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.16724 2009-12-04 14:04:28.000000000 +0200 +++ /tmp/wklog.68.new.16724 2009-12-04 14:04:28.000000000 +0200 @@ -10,7 +10,8 @@ Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint - (2) For each i: rowid{a_i is null} is the same for each tuple + (2) For each i: rowid{a_i is null} is the same for each tuple, + that is, this set is independent of the left-side tuples. Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Version updated. --- /tmp/wklog.68.old.5257 2009-12-04 11:27:11.000000000 +0200 +++ /tmp/wklog.68.new.5257 2009-12-04 11:27:11.000000000 +0200 @@ -1 +1 @@ -Benchmarks-3.0 +Server-9.x -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Category updated. --- /tmp/wklog.68.old.5242 2009-12-04 11:27:02.000000000 +0200 +++ /tmp/wklog.68.new.5242 2009-12-04 11:27:02.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Status updated. --- /tmp/wklog.68.old.5242 2009-12-04 11:27:02.000000000 +0200 +++ /tmp/wklog.68.new.5242 2009-12-04 11:27:02.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +Assigned -=-=(Timour - Fri, 04 Dec 2009, 11:26)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.5182 2009-12-04 11:26:25.000000000 +0200 +++ /tmp/wklog.68.new.5182 2009-12-04 11:26:25.000000000 +0200 @@ -50,23 +50,39 @@ The array can be created on demand. Other consideration that may be taken into account: + 1. If columns a_j1,...,a_jm do not contain null values in the temporary -table at all, create for them only one index array (and of course do not -create any bitmaps for them). -2. Consider the ratio d(a_i)=N'/ V(a_i), where N' is the number of rows -where a_i is not null and V(a_i) is the number of distinct values for -a_i excluding nulls. - If d(a_i) is close to 1 then do not create any index array: check +table at all and v_j1,...,v_jm cannot be null, create for these columns +only one index array (and of course do not create any bitmaps for them). + +2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number +of rows, where a_i is not null and V(a_i) is the number of distinct +values for a_i excluding nulls. +If d(a_i) is close to N'(a_i) then do not create any index array: check whether there is a match running through the records that have been -filtered in. Anyway if d(a_i) is close to 1 then a intersection with -rowid{a_i=v_i} would not reduce the number of remaining rowids +filtered in. Anyway if d(a_i) is close to N'(a_i) then the intersection + with rowid{a_i=v_i} will not reduce the number of remaining rowids significantly. - If additionally N-N' is small do not create a bitmap for this column -either. -3. If for a column a_i d(a_i) is not close to 1, but N-N' is small a -sorted array of rowids from the set rowid{a_i is null} can be used -instead of a bitmap. +In other words is V(a_i) exceeds some threshold there is no sense to +create an index for a_i. +If additionally N-N'(a_i) is small do not create a bitmap for this +column either. + +3. If for a column a_i d(a_i) is not close to N'(a_i), but N-N'(a_i) is +small a sorted array of rowids from the set rowid{a_i is null} can be +used instead of a bitmap. + 4. We always have a match if R0= INTERSECT rowid{a_i is null} is not empty. Here i runs through all indexes from [1..n] such that v_i is not null. For a given subset of columns this fact has to be checked only once. It can be easily done with bitmap intersection. + +5. If v1,...,vn never can be a null, then indexes (sorted arrays) can be +created only for rows with nulls. + +6. If v1,...,vn never can be a null and number of rows with nulls is +small do not create indexes and do not create bitmaps. + +7. If you get a row with nulls in all columns stop filling the temporary +table and return UNKNOWN for any tuple <v1,...,vn>. + ------------------------------------------------------------ -=-=(View All Progress Notes, 11 total)=-=- http://askmonty.org/worklog/index.pl?tid=68&nolimit=1 DESCRIPTION: The goal of this task is to implement efficient execution of NOT IN subquery predicates of the form: <oe_1,...,oe_n> NOT IN <non_correlated subquery> when either some oe_i, or some subqury result column contains NULLs. The problem with such predicates is that it is possible to use index lookups only when neither argument of the predicate contains NULLs. If some argument contains a NULL, then due to NULL semantics, it plays the role of a wildcard. If we were to use regular index lookups, then we would get 'no match' for some outer tuple (thus the predicate evaluates to FALSE), while the SQL semantics means 'partial match', and the predicate should evaluate to NULL. This task implements an efficient algorithm to compute such 'parial matches', where a NULL matches any value. HIGH-LEVEL SPECIFICATION: Contents ======================================================================== 1. Initial idea as proposed by Igor 2. Algorithm for IN execution with partial matching 3. Directions for improvement 1. Initial idea as proposed by Igor ======================================================================== For each left side tuple (v_1,...,v_n) we have to find the following set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs trough all indexes from [1..n] such that v_i is not null. Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint (2) For each i: rowid{a_i is null} is the same for each tuple, that is, this set is independent of the left-side tuples. Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap Taken all above into account I could suggest the following algorithm to build R: Using indexes (read about them below) for each column participating in the intersection, merge ordered sets rowid{a_i=v_i} in the following manner. If a rowid r has been encountered maximum in k sets rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can infer that for any r from R indexes a_i can be uniquely divided into two groups: - one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, - the other contains indexes a_j such that r belongs to rowid{a_j is null}. Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order needed for the merge procedure. We could use BTREE indexes for temp table. But they are rather expensive and take a lot of memory as the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. 2. Algorithm for IN execution with partial matching ======================================================================== 2.1 Below is shown the top-level algorithm to execute an IN predicate with partial matching. This algorithm is essentially the implementation of Item_subselect:exec(). int lookup_with_null_semantics(outer_ref[], mat_subquery) { if (index_lookup(outer_ref, mat_subquery) return TRUE else { /* Check if there is a partial match (UNKNOWN) or no match (NULL). */ if (this is the first partial match) { vkey[] = build array of value keys for each NULL-able column of mat_subquery. nkey[] = build a bitmap NULL index for each column of mat_subquery that contains NULLs nonull_key = build a key over all non-NULL columns of mat_subquery } if (partial_match(outer_ref, vkey[], nkey[], nonull_key) return UNKNOWN else return FALSE } } 2.2 The implementation of partial matching is as follows /* Assumptions: - It has already been checked if there is a complete match by a regular index lookup, and the test failed. - It has already been checked if there is a complete NULL row, and if there was we wouldn't call this function. Thus we assume that there is no complete NULL row. - Not all vidx_i are empty, but some can be empty. If all were empty, then the only possibility for a match is a complete NULL row, which we already checked. @param outer_ref - the uter (left) IN argument. @param vidx[] - array of value keys Ordered sequences of rowids of the corresponding columns a_i, such that all rowids in idx_i are the ones where column a_i contains some value or NULL. Each idx_i is derived dynamically, for each different left argument of an IN predicate. @param nidx[] - array of NULL keys Bitmpas, one per each column, where a bit is set if the corresponding row has a NULL value for the corresponding column. @nonull_key - the only key over all columns of the materialized subquery that do not contain NULLs @returns @retval FALSE if there is no match @retval TRUE if there is a partial match */ Boolean partial_match(outer_ref, vkey[], nkey[], nonull_key) { /* Set of the keys (columns) that form a partial match. */ Set matching_keys = {} /* A subset of all keys that need to be checked for NULL matches. */ Set null_keys = {} Int min_key /* Key that contains the current minimum position. */ Int min_row /* Current row number of min_key. */ Int cur_min_key, cur_min_row PriorityQueue pq if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE for (i = 1; i <= n; i++) { if (vkey[i] != nonull_key) vkey[i].lookup(outer_ref) if (! vkey[i].is_eof()) pq.insert(i) } /* Not all value keys are empty, thus we don't have only NULL keys. If we had, the only possible match is a NULL row, and we cheked there is no such row, therefore the result is known to be FALSE. In fact this algorithm makes sense for at least two non-NULL columns. */ assert(pq.elements > 1) (min_key, min_row) = pq.pop() matching_keys.add(min_key) vkey[min_key].next() if (! vkey[min_key].is_eof()) pq.insert(min_key) while (TRUE) { (cur_min_key, cur_min_row) = pq.pop() if (cur_min_row == min_row) { matching_keys.add(cur_min_key) /* There cannot be a complete match, as we already checked for one. */ assert(matching_keys.elements < n) } else if (vkey[cur_min_key] == nonull_key) { /* The non-NULL key has no corresponding NULL index, so we know for sure that the row 'min_row' is not a match. */ (min_key, min_row) = (cur_min_key, cur_min_row) matching_keys = {min_key} } else { assert(cur_min_row > min_row) /* Follows from the use of PQ. */ null_keys = set_difference(all keys vkey[], matching_keys) /* Check if all null_keys contain a NULL at row 'min_row'. The procedure internally checks all keys in a special precomputed order. A prior procedure determines an optimal order and a mapping idx_no -> idx_order (encoded as an array). This procedure makes sure not to match the non-NULL column. */ if (test_null_row(null_keys, min_row)) return TRUE else { (min_key, min_row) = (cur_min_key, cur_min_row) matching_keys = {min_key} } } vkey[cur_min_key].next() if (! vkey[cur_min_key].is_eof()) pq.insert(cur_min_key) else if (vkey[cur_min_key] == nonull_key) { /* If there can't be more matches for the nonull_key, we know for sure there is no match, since there is no possible NULL match. */ return FALSE } if (pq.is_empty()) { /* Check the last row of the last column in PQ for NULL matches. */ null_keys = set_difference(all keys vkey[], matching_keys) if (test_null_row(null_keys, min_row)) return TRUE else return FALSE } } /* We should never get here. */ assert(FALSE) return FALSE } 3. Directions for improvement ======================================================================== Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all and v_j1,...,v_jm cannot be null, create for these columns only one index array (and of course do not create any bitmaps for them). [done] 2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number of rows, where a_i is not null and V(a_i) is the number of distinct values for a_i excluding nulls. If d(a_i) is close to N'(a_i) then do not create any index array: check whether there is a match running through the records that have been filtered in. Anyway if d(a_i) is close to N'(a_i) then the intersection with rowid{a_i=v_i} will not reduce the number of remaining rowids significantly. In other words is V(a_i) exceeds some threshold there is no sense to create an index for a_i. If additionally N-N'(a_i) is small do not create a bitmap for this column either. 3. If for a column a_i d(a_i) is not close to N'(a_i), but N-N'(a_i) is small a sorted array of rowids from the set rowid{a_i is null} can be used instead of a bitmap. 4. We always have a match if R0= INTERSECT rowid{a_i is null} is not empty. Here i runs through all indexes from [1..n] such that v_i is not null. For a given subset of columns this fact has to be checked only once. It can be easily done with bitmap intersection. 5. If v1,...,vn never can be a null, then indexes (sorted arrays) can be created only for rows with nulls. 6. If v1,...,vn never can be a null and number of rows with nulls is small do not create indexes and do not create bitmaps. 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. [This is wrong, because if we don't fill the whole temp table, there may be some tuple(s) that would match some outer tuple. In such cases, if we stop filling the temp table, we would miss a TRUE result. Having a partial match doesn't preclude us from having a complete match]. 8. [timour] Consider that due to materialization, we already have a unique index on all columns <a_1,..., a_n>. We can use the first key part of this index over column a_1, instead of the index rowid{a_i=v_i}. Thus we can avoid creating the index rowid{a_i=v_i}. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Timour): Subquery optimization: Efficient NOT IN execution with NULLs (68)
by worklog-noreply＠askmonty.org 22 Feb '10

22 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subquery optimization: Efficient NOT IN execution with NULLs CREATION DATE..: Fri, 27 Nov 2009, 13:22 SUPERVISOR.....: Monty IMPLEMENTOR....: Timour COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 68 (http://askmonty.org/worklog/?tid=68) VERSION........: Server-9.x STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Timour - Mon, 22 Feb 2010, 17:39)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.17116 2010-02-22 17:39:48.000000000 +0200 +++ /tmp/wklog.68.new.17116 2010-02-22 17:39:48.000000000 +0200 @@ -233,6 +233,7 @@ 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all and v_j1,...,v_jm cannot be null, create for these columns only one index array (and of course do not create any bitmaps for them). +[done] 2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number of rows, where a_i is not null and V(a_i) is the number of distinct @@ -264,6 +265,10 @@ 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. +[This is wrong, because if we don't fill the whole temp table, there may + be some tuple(s) that would match some outer tuple. In such cases, if we + stop filling the temp table, we would miss a TRUE result. Having a partial + match doesn't preclude us from having a complete match]. 8. [timour] Consider that due to materialization, we already have a unique index -=-=(Timour - Tue, 19 Jan 2010, 18:44)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.22569 2010-01-19 18:44:01.000000000 +0200 +++ /tmp/wklog.68.new.22569 2010-01-19 18:44:01.000000000 +0200 @@ -132,11 +132,10 @@ if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE - if (nonull_key) - pq.insert(nonull_key) for (i = 1; i <= n; i++) { + if (vkey[i] != nonull_key) vkey[i].lookup(outer_ref) if (! vkey[i].is_eof()) pq.insert(i) @@ -167,7 +166,7 @@ /* There cannot be a complete match, as we already checked for one. */ assert(matching_keys.elements < n) } - else if (cur_min_key == nonull_key) + else if (vkey[cur_min_key] == nonull_key) { /* The non-NULL key has no corresponding NULL index, so we know for @@ -183,8 +182,10 @@ /* Check if all null_keys contain a NULL at row 'min_row'. The procedure internally checks all keys in a special precomputed order. A prior - procedure determines an optimal order and a mapping - idx_no -> idx_order (encoded as an array). + procedure determines an optimal order and a mapping idx_no -> idx_order + (encoded as an array). + + This procedure makes sure not to match the non-NULL column. */ if (test_null_row(null_keys, min_row)) return TRUE @@ -198,6 +199,14 @@ vkey[cur_min_key].next() if (! vkey[cur_min_key].is_eof()) pq.insert(cur_min_key) + else if (vkey[cur_min_key] == nonull_key) + { + /* + If there can't be more matches for the nonull_key, we know for sure + there is no match, since there is no possible NULL match. + */ + return FALSE + } if (pq.is_empty()) { @@ -216,7 +225,6 @@ } - 3. Directions for improvement ======================================================================== -=-=(Timour - Tue, 19 Jan 2010, 18:29)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.21045 2010-01-19 18:29:12.000000000 +0200 +++ /tmp/wklog.68.new.21045 2010-01-19 18:29:12.000000000 +0200 @@ -132,6 +132,8 @@ if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE + if (nonull_key) + pq.insert(nonull_key) for (i = 1; i <= n; i++) { -=-=(Guest - Tue, 19 Jan 2010, 18:15)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.19825 2010-01-19 18:15:30.000000000 +0200 +++ /tmp/wklog.68.new.19825 2010-01-19 18:15:30.000000000 +0200 @@ -1,8 +1,16 @@ -This a copy of the initial algorithm proposed by Igor: -====================================================== +Contents +======================================================================== -For each left side tuple (v_1,...,v_n) we have to find the following set -of rowids for the temp table containing N rows as the result of +1. Initial idea as proposed by Igor +2. Algorithm for IN execution with partial matching +3. Directions for improvement + + +1. Initial idea as proposed by Igor +======================================================================== + +For each left side tuple (v_1,...,v_n) we have to find the following +set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs @@ -18,38 +26,198 @@ - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap -Taken all above into account I could suggest the following algorithm to -build R: +Taken all above into account I could suggest the following algorithm +to build R: - Using indexes (read about them below) for each column participating in the - intersection, - merge ordered sets rowid{a_i=v_i} in the following manner. + Using indexes (read about them below) for each column participating + in the intersection, merge ordered sets rowid{a_i=v_i} in the + following manner. If a rowid r has been encountered maximum in k sets -rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), + rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is -not in {i1,...,ik}. + not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. -Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i -is null} is either +Here we use the property (1): +any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can -infer that for any r from R -indexes a_i can be uniquely divided into two groups: one contains -indexes a_i where r belongs to -the sets rowid{a_i=v_i}, the other contains indexes a_j such that r -belongs to rowid{a_j is null}. - -Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order -needed for the merge procedure. We could use BTREE indexes for temp -table. But they are rather expensive and -take a lot of memory as the are implemented with RB trees. +infer that for any r from R indexes a_i can be uniquely divided into +two groups: +- one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, +- the other contains indexes a_j such that r belongs to + rowid{a_j is null}. + +Now let's talk how to get elements from rowid{a_i=v_i} in a sorted +order needed for the merge procedure. We could use BTREE indexes for +temp table. But they are rather expensive and take a lot of memory as +the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. +2. Algorithm for IN execution with partial matching +======================================================================== + +2.1 Below is shown the top-level algorithm to execute an IN predicate +with partial matching. This algorithm is essentially the implementation +of Item_subselect:exec(). + +int lookup_with_null_semantics(outer_ref[], mat_subquery) +{ + if (index_lookup(outer_ref, mat_subquery) + return TRUE + else + { + /* + Check if there is a partial match (UNKNOWN) or no match (NULL). + */ + if (this is the first partial match) + { + vkey[] = build array of value keys for each NULL-able column + of mat_subquery. + nkey[] = build a bitmap NULL index for each column of mat_subquery + that contains NULLs + nonull_key = build a key over all non-NULL columns of mat_subquery + } + if (partial_match(outer_ref, vkey[], nkey[], nonull_key) + return UNKNOWN + else + return FALSE + } +} + +2.2 The implementation of partial matching is as follows + +/* + Assumptions: + - It has already been checked if there is a complete match by a + regular index lookup, and the test failed. + - It has already been checked if there is a complete NULL row, + and if there was we wouldn't call this function. Thus we assume + that there is no complete NULL row. + - Not all vidx_i are empty, but some can be empty. If all were empty, + then the only possibility for a match is a complete NULL row, which + we already checked. + + @param outer_ref - the uter (left) IN argument. + @param vidx[] - array of value keys + Ordered sequences of rowids of the corresponding columns a_i, such + that all rowids in idx_i are the ones where column a_i contains some + value or NULL. Each idx_i is derived dynamically, for each different + left argument of an IN predicate. + @param nidx[] - array of NULL keys + Bitmpas, one per each column, where a bit is set if the corresponding + row has a NULL value for the corresponding column. + @nonull_key - the only key over all columns of the materialized subquery + that do not contain NULLs + + @returns + @retval FALSE if there is no match + @retval TRUE if there is a partial match +*/ + +Boolean partial_match(outer_ref, vkey[], nkey[], nonull_key) +{ + /* Set of the keys (columns) that form a partial match. */ + Set matching_keys = {} + /* A subset of all keys that need to be checked for NULL matches. */ + Set null_keys = {} + Int min_key /* Key that contains the current minimum position. */ + Int min_row /* Current row number of min_key. */ + Int cur_min_key, cur_min_row + PriorityQueue pq + + if (nonull_key && ! nonull_key->lookup(outer_ref)) + return FALSE + + for (i = 1; i <= n; i++) + { + vkey[i].lookup(outer_ref) + if (! vkey[i].is_eof()) + pq.insert(i) + } + /* + Not all value keys are empty, thus we don't have only NULL + keys. If we had, the only possible match is a NULL row, and + we cheked there is no such row, therefore the result is known + to be FALSE. + In fact this algorithm makes sense for at least two non-NULL + columns. + */ + assert(pq.elements > 1) + + (min_key, min_row) = pq.pop() + matching_keys.add(min_key) + vkey[min_key].next() + if (! vkey[min_key].is_eof()) + pq.insert(min_key) + + while (TRUE) + { + (cur_min_key, cur_min_row) = pq.pop() + + if (cur_min_row == min_row) + { + matching_keys.add(cur_min_key) + /* There cannot be a complete match, as we already checked for one. */ + assert(matching_keys.elements < n) + } + else if (cur_min_key == nonull_key) + { + /* + The non-NULL key has no corresponding NULL index, so we know for + sure that the row 'min_row' is not a match. + */ + (min_key, min_row) = (cur_min_key, cur_min_row) + matching_keys = {min_key} + } + else + { + assert(cur_min_row > min_row) /* Follows from the use of PQ. */ + null_keys = set_difference(all keys vkey[], matching_keys) + /* + Check if all null_keys contain a NULL at row 'min_row'. The procedure + internally checks all keys in a special precomputed order. A prior + procedure determines an optimal order and a mapping + idx_no -> idx_order (encoded as an array). + */ + if (test_null_row(null_keys, min_row)) + return TRUE + else + { + (min_key, min_row) = (cur_min_key, cur_min_row) + matching_keys = {min_key} + } + } + + vkey[cur_min_key].next() + if (! vkey[cur_min_key].is_eof()) + pq.insert(cur_min_key) + + if (pq.is_empty()) + { + /* Check the last row of the last column in PQ for NULL matches. */ + null_keys = set_difference(all keys vkey[], matching_keys) + if (test_null_row(null_keys, min_row)) + return TRUE + else + return FALSE + } + } + + /* We should never get here. */ + assert(FALSE) + return FALSE +} + + + +3. Directions for improvement +======================================================================== + Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary -=-=(Timour - Sun, 06 Dec 2009, 14:36)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.12919 2009-12-06 14:36:18.000000000 +0200 +++ /tmp/wklog.68.new.12919 2009-12-06 14:36:18.000000000 +0200 @@ -87,3 +87,8 @@ 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. +8. [timour] + Consider that due to materialization, we already have a unique index +on all columns <a_1,..., a_n>. We can use the first key part of this index +over column a_1, instead of the index rowid{a_i=v_i}. Thus we can avoid +creating the index rowid{a_i=v_i}. -=-=(Timour - Fri, 04 Dec 2009, 14:04)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.16724 2009-12-04 14:04:28.000000000 +0200 +++ /tmp/wklog.68.new.16724 2009-12-04 14:04:28.000000000 +0200 @@ -10,7 +10,8 @@ Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint - (2) For each i: rowid{a_i is null} is the same for each tuple + (2) For each i: rowid{a_i is null} is the same for each tuple, + that is, this set is independent of the left-side tuples. Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Version updated. --- /tmp/wklog.68.old.5257 2009-12-04 11:27:11.000000000 +0200 +++ /tmp/wklog.68.new.5257 2009-12-04 11:27:11.000000000 +0200 @@ -1 +1 @@ -Benchmarks-3.0 +Server-9.x -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Category updated. --- /tmp/wklog.68.old.5242 2009-12-04 11:27:02.000000000 +0200 +++ /tmp/wklog.68.new.5242 2009-12-04 11:27:02.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Timour - Fri, 04 Dec 2009, 11:27)=-=- Status updated. --- /tmp/wklog.68.old.5242 2009-12-04 11:27:02.000000000 +0200 +++ /tmp/wklog.68.new.5242 2009-12-04 11:27:02.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +Assigned -=-=(Timour - Fri, 04 Dec 2009, 11:26)=-=- High-Level Specification modified. --- /tmp/wklog.68.old.5182 2009-12-04 11:26:25.000000000 +0200 +++ /tmp/wklog.68.new.5182 2009-12-04 11:26:25.000000000 +0200 @@ -50,23 +50,39 @@ The array can be created on demand. Other consideration that may be taken into account: + 1. If columns a_j1,...,a_jm do not contain null values in the temporary -table at all, create for them only one index array (and of course do not -create any bitmaps for them). -2. Consider the ratio d(a_i)=N'/ V(a_i), where N' is the number of rows -where a_i is not null and V(a_i) is the number of distinct values for -a_i excluding nulls. - If d(a_i) is close to 1 then do not create any index array: check +table at all and v_j1,...,v_jm cannot be null, create for these columns +only one index array (and of course do not create any bitmaps for them). + +2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number +of rows, where a_i is not null and V(a_i) is the number of distinct +values for a_i excluding nulls. +If d(a_i) is close to N'(a_i) then do not create any index array: check whether there is a match running through the records that have been -filtered in. Anyway if d(a_i) is close to 1 then a intersection with -rowid{a_i=v_i} would not reduce the number of remaining rowids +filtered in. Anyway if d(a_i) is close to N'(a_i) then the intersection + with rowid{a_i=v_i} will not reduce the number of remaining rowids significantly. - If additionally N-N' is small do not create a bitmap for this column -either. -3. If for a column a_i d(a_i) is not close to 1, but N-N' is small a -sorted array of rowids from the set rowid{a_i is null} can be used -instead of a bitmap. +In other words is V(a_i) exceeds some threshold there is no sense to +create an index for a_i. +If additionally N-N'(a_i) is small do not create a bitmap for this +column either. + +3. If for a column a_i d(a_i) is not close to N'(a_i), but N-N'(a_i) is +small a sorted array of rowids from the set rowid{a_i is null} can be +used instead of a bitmap. + 4. We always have a match if R0= INTERSECT rowid{a_i is null} is not empty. Here i runs through all indexes from [1..n] such that v_i is not null. For a given subset of columns this fact has to be checked only once. It can be easily done with bitmap intersection. + +5. If v1,...,vn never can be a null, then indexes (sorted arrays) can be +created only for rows with nulls. + +6. If v1,...,vn never can be a null and number of rows with nulls is +small do not create indexes and do not create bitmaps. + +7. If you get a row with nulls in all columns stop filling the temporary +table and return UNKNOWN for any tuple <v1,...,vn>. + ------------------------------------------------------------ -=-=(View All Progress Notes, 11 total)=-=- http://askmonty.org/worklog/index.pl?tid=68&nolimit=1 DESCRIPTION: The goal of this task is to implement efficient execution of NOT IN subquery predicates of the form: <oe_1,...,oe_n> NOT IN <non_correlated subquery> when either some oe_i, or some subqury result column contains NULLs. The problem with such predicates is that it is possible to use index lookups only when neither argument of the predicate contains NULLs. If some argument contains a NULL, then due to NULL semantics, it plays the role of a wildcard. If we were to use regular index lookups, then we would get 'no match' for some outer tuple (thus the predicate evaluates to FALSE), while the SQL semantics means 'partial match', and the predicate should evaluate to NULL. This task implements an efficient algorithm to compute such 'parial matches', where a NULL matches any value. HIGH-LEVEL SPECIFICATION: Contents ======================================================================== 1. Initial idea as proposed by Igor 2. Algorithm for IN execution with partial matching 3. Directions for improvement 1. Initial idea as proposed by Igor ======================================================================== For each left side tuple (v_1,...,v_n) we have to find the following set of rowids for the temp table containing N rows as the result of materialization of the subquery: R= INTERSECT (rowid{a_i=v_i} UNION rowid{a_i is null} where i runs trough all indexes from [1..n] such that v_i is not null. Bear in mind the following specifics of this intersection: (1) For each i: rowid{a_i=v_i} and rowid{a_i is null} are disjoint (2) For each i: rowid{a_i is null} is the same for each tuple, that is, this set is independent of the left-side tuples. Due to (2) it makes sense to build rowid{a_i is null} only once. A good representation for such sets would be bitmaps: - it requires minimum memory: not more than N*n bits in total - search of an element in a set is extremely cheap Taken all above into account I could suggest the following algorithm to build R: Using indexes (read about them below) for each column participating in the intersection, merge ordered sets rowid{a_i=v_i} in the following manner. If a rowid r has been encountered maximum in k sets rowid{a_i1=v_i1},...,rowid(a_ik=v_ik), then it has to be checked against all rowid{a_i=v_i} such that i is not in {i1,...,ik}. As soon as we fail to find r in one of these sets we discard it. If r has been found in all of them then r belongs to the set R. Here we use the property (1): any r from rowid{a_i=v_i} UNION rowid{a_i is null} is either belongs to rowid{a_i=v_i} or to rowid{a_i is null}. From this we can infer that for any r from R indexes a_i can be uniquely divided into two groups: - one contains indexes a_i where r belongs to the sets rowid{a_i=v_i}, - the other contains indexes a_j such that r belongs to rowid{a_j is null}. Now let's talk how to get elements from rowid{a_i=v_i} in a sorted order needed for the merge procedure. We could use BTREE indexes for temp table. But they are rather expensive and take a lot of memory as the are implemented with RB trees. I would suggest creating for each column from the temporary table just an array of rowids sorted by the value from column a. Index lookup in such an array is cheap. It's also rather cheap to check that the next rowid refers to a row with a different value in column a. The array can be created on demand. 2. Algorithm for IN execution with partial matching ======================================================================== 2.1 Below is shown the top-level algorithm to execute an IN predicate with partial matching. This algorithm is essentially the implementation of Item_subselect:exec(). int lookup_with_null_semantics(outer_ref[], mat_subquery) { if (index_lookup(outer_ref, mat_subquery) return TRUE else { /* Check if there is a partial match (UNKNOWN) or no match (NULL). */ if (this is the first partial match) { vkey[] = build array of value keys for each NULL-able column of mat_subquery. nkey[] = build a bitmap NULL index for each column of mat_subquery that contains NULLs nonull_key = build a key over all non-NULL columns of mat_subquery } if (partial_match(outer_ref, vkey[], nkey[], nonull_key) return UNKNOWN else return FALSE } } 2.2 The implementation of partial matching is as follows /* Assumptions: - It has already been checked if there is a complete match by a regular index lookup, and the test failed. - It has already been checked if there is a complete NULL row, and if there was we wouldn't call this function. Thus we assume that there is no complete NULL row. - Not all vidx_i are empty, but some can be empty. If all were empty, then the only possibility for a match is a complete NULL row, which we already checked. @param outer_ref - the uter (left) IN argument. @param vidx[] - array of value keys Ordered sequences of rowids of the corresponding columns a_i, such that all rowids in idx_i are the ones where column a_i contains some value or NULL. Each idx_i is derived dynamically, for each different left argument of an IN predicate. @param nidx[] - array of NULL keys Bitmpas, one per each column, where a bit is set if the corresponding row has a NULL value for the corresponding column. @nonull_key - the only key over all columns of the materialized subquery that do not contain NULLs @returns @retval FALSE if there is no match @retval TRUE if there is a partial match */ Boolean partial_match(outer_ref, vkey[], nkey[], nonull_key) { /* Set of the keys (columns) that form a partial match. */ Set matching_keys = {} /* A subset of all keys that need to be checked for NULL matches. */ Set null_keys = {} Int min_key /* Key that contains the current minimum position. */ Int min_row /* Current row number of min_key. */ Int cur_min_key, cur_min_row PriorityQueue pq if (nonull_key && ! nonull_key->lookup(outer_ref)) return FALSE for (i = 1; i <= n; i++) { if (vkey[i] != nonull_key) vkey[i].lookup(outer_ref) if (! vkey[i].is_eof()) pq.insert(i) } /* Not all value keys are empty, thus we don't have only NULL keys. If we had, the only possible match is a NULL row, and we cheked there is no such row, therefore the result is known to be FALSE. In fact this algorithm makes sense for at least two non-NULL columns. */ assert(pq.elements > 1) (min_key, min_row) = pq.pop() matching_keys.add(min_key) vkey[min_key].next() if (! vkey[min_key].is_eof()) pq.insert(min_key) while (TRUE) { (cur_min_key, cur_min_row) = pq.pop() if (cur_min_row == min_row) { matching_keys.add(cur_min_key) /* There cannot be a complete match, as we already checked for one. */ assert(matching_keys.elements < n) } else if (vkey[cur_min_key] == nonull_key) { /* The non-NULL key has no corresponding NULL index, so we know for sure that the row 'min_row' is not a match. */ (min_key, min_row) = (cur_min_key, cur_min_row) matching_keys = {min_key} } else { assert(cur_min_row > min_row) /* Follows from the use of PQ. */ null_keys = set_difference(all keys vkey[], matching_keys) /* Check if all null_keys contain a NULL at row 'min_row'. The procedure internally checks all keys in a special precomputed order. A prior procedure determines an optimal order and a mapping idx_no -> idx_order (encoded as an array). This procedure makes sure not to match the non-NULL column. */ if (test_null_row(null_keys, min_row)) return TRUE else { (min_key, min_row) = (cur_min_key, cur_min_row) matching_keys = {min_key} } } vkey[cur_min_key].next() if (! vkey[cur_min_key].is_eof()) pq.insert(cur_min_key) else if (vkey[cur_min_key] == nonull_key) { /* If there can't be more matches for the nonull_key, we know for sure there is no match, since there is no possible NULL match. */ return FALSE } if (pq.is_empty()) { /* Check the last row of the last column in PQ for NULL matches. */ null_keys = set_difference(all keys vkey[], matching_keys) if (test_null_row(null_keys, min_row)) return TRUE else return FALSE } } /* We should never get here. */ assert(FALSE) return FALSE } 3. Directions for improvement ======================================================================== Other consideration that may be taken into account: 1. If columns a_j1,...,a_jm do not contain null values in the temporary table at all and v_j1,...,v_jm cannot be null, create for these columns only one index array (and of course do not create any bitmaps for them). [done] 2. Consider the ratio d(a_i)=N'(a_i)/V(a_i), where N'(a_i) is the number of rows, where a_i is not null and V(a_i) is the number of distinct values for a_i excluding nulls. If d(a_i) is close to N'(a_i) then do not create any index array: check whether there is a match running through the records that have been filtered in. Anyway if d(a_i) is close to N'(a_i) then the intersection with rowid{a_i=v_i} will not reduce the number of remaining rowids significantly. In other words is V(a_i) exceeds some threshold there is no sense to create an index for a_i. If additionally N-N'(a_i) is small do not create a bitmap for this column either. 3. If for a column a_i d(a_i) is not close to N'(a_i), but N-N'(a_i) is small a sorted array of rowids from the set rowid{a_i is null} can be used instead of a bitmap. 4. We always have a match if R0= INTERSECT rowid{a_i is null} is not empty. Here i runs through all indexes from [1..n] such that v_i is not null. For a given subset of columns this fact has to be checked only once. It can be easily done with bitmap intersection. 5. If v1,...,vn never can be a null, then indexes (sorted arrays) can be created only for rows with nulls. 6. If v1,...,vn never can be a null and number of rows with nulls is small do not create indexes and do not create bitmaps. 7. If you get a row with nulls in all columns stop filling the temporary table and return UNKNOWN for any tuple <v1,...,vn>. [This is wrong, because if we don't fill the whole temp table, there may be some tuple(s) that would match some outer tuple. In such cases, if we stop filling the temp table, we would miss a TRUE result. Having a partial match doesn't preclude us from having a complete match]. 8. [timour] Consider that due to materialization, we already have a unique index on all columns <a_1,..., a_n>. We can use the first key part of this index over column a_1, instead of the index rowid{a_i=v_i}. Thus we can avoid creating the index rowid{a_i=v_i}. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Rev 2764: MWL#68 Subquery optimization: Efficient NOT IN execution with NULLs in file:///home/tsk/mprog/src/5.3-mwl68-unmerged/
by timour＠askmonty.org 22 Feb '10

22 Feb '10

At file:///home/tsk/mprog/src/5.3-mwl68-unmerged/ ------------------------------------------------------------ revno: 2764 revision-id: timour(a)askmonty.org-20100222151655-ltjv0rlv6z2sdiiu parent: timour(a)askmonty.org-20100222135709-3568ya6z76hkwfzs committer: timour(a)askmonty.org branch nick: 5.3-mwl68-unmerged timestamp: Mon 2010-02-22 17:16:55 +0200 message: MWL#68 Subquery optimization: Efficient NOT IN execution with NULLs This patch mainly adds sorting of all indexes for partial matching according to their NULL selectivity. The patch also fixes a related bug in subselect_rowid_merge_engine::test_null_row() where the wrong matched indexes were skipped. In addition the patch: - adds few ::print() methods, - renames few variables that had similar names but different purpose. === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-02-22 13:57:09 +0000 +++ b/sql/item_subselect.cc 2010-02-22 15:16:55 +0000 @@ -3424,7 +3424,10 @@ DBUG_ENTER("subselect_hash_sj_engine::set_strategy_using_schema"); if (item_in->is_top_level_item()) + { strategy= COMPLETE_MATCH; + DBUG_VOID_RETURN; + } else { List_iterator<Item> inner_col_it(*item_in->unit->get_unit_column_types()); @@ -3846,6 +3849,7 @@ thd->lex->current_select= materialize_engine->select_lex; if ((res= materialize_join->optimize())) goto err; /* purecov: inspected */ + DBUG_ASSERT(!is_materialized); /* We should materialize only once. */ materialize_join->exec(); if ((res= test(materialize_join->error || thd->is_fatal_error))) goto err; @@ -3952,7 +3956,7 @@ lookup_engine->print(str, query_type); else str->append(STRING_WITH_LEN( - "<the access method for lookups is not yet created>" + "<engine selected at execution time>" )); } @@ -3980,11 +3984,10 @@ } -Ordered_key::Ordered_key(uint key_idx_arg, TABLE *tbl_arg, - Item *search_key_arg, ha_rows null_count_arg, - ha_rows min_null_row_arg, ha_rows max_null_row_arg, - uchar *row_num_to_rowid_arg) - : key_idx(key_idx_arg), tbl(tbl_arg), search_key(search_key_arg), +Ordered_key::Ordered_key(uint keyid_arg, TABLE *tbl_arg, Item *search_key_arg, + ha_rows null_count_arg, ha_rows min_null_row_arg, + ha_rows max_null_row_arg, uchar *row_num_to_rowid_arg) + : keyid(keyid_arg), tbl(tbl_arg), search_key(search_key_arg), row_num_to_rowid(row_num_to_rowid_arg), null_count(null_count_arg) { DBUG_ASSERT(tbl->file->stats.records > null_count); @@ -4190,6 +4193,21 @@ /* + The probability that a certain row does not contain a NULL in some row in + a NULL-indexed column. + @retval 1 if there are no NULLs + @retval 0 if only NULLs +*/ + +double Ordered_key::null_selectivity() +{ + /* We should not be processing empty tables. */ + DBUG_ASSERT(tbl->file->stats.records); + return (1 - (double) null_count / (double) tbl->file->stats.records); +} + + +/* Compare the value(s) of the current key in 'search_key' with the data of the current table record. @@ -4307,6 +4325,34 @@ } +void Ordered_key::print(String *str) +{ + uint i; + str->append("{idx="); + str->qs_append(keyid); + str->append(", ("); + for (i= 0; i < key_column_count - 1; i++) + { + str->append(key_columns[i]->field->field_name); + str->append(", "); + } + str->append(key_columns[i]->field->field_name); + str->append("), "); + + str->append("null_bitmap: (bits="); + str->qs_append(null_key.n_bits); + str->append(", nulls= "); + str->qs_append((double)null_count); + str->append(", min_null= "); + str->qs_append((double)min_null_row); + str->append(", max_null= "); + str->qs_append((double)max_null_row); + str->append("), "); + + str->append('}'); +} + + /* @param non_null_key_parts @param partial_match_key_parts A union of all single-column NULL key parts. @@ -4323,7 +4369,7 @@ rownum_t cur_rownum= 0; select_materialize_with_stats *result_sink= (select_materialize_with_stats *) result; - uint cur_key= 0; + uint cur_keyid= 0; Item_in_subselect *item_in= (Item_in_subselect*) item; int error; @@ -4346,16 +4392,16 @@ /* Create the only non-NULL key if there is any. */ if (non_null_key_parts) { - non_null_key= new Ordered_key(cur_key, tmp_table, item_in->left_expr, + non_null_key= new Ordered_key(cur_keyid, tmp_table, item_in->left_expr, 0, 0, 0, row_num_to_rowid); if (non_null_key->init(non_null_key_parts)) { // TIMOUR: revert to partial matching via scanning return TRUE; } - merge_keys[cur_key]= non_null_key; - merge_keys[cur_key]->first(); - ++cur_key; + merge_keys[cur_keyid]= non_null_key; + merge_keys[cur_keyid]->first(); + ++cur_keyid; } /* @@ -4379,23 +4425,24 @@ continue; if (result_sink->get_null_count_of_col(i) == row_count) - bitmap_set_bit(&null_only_columns, cur_key); + bitmap_set_bit(&null_only_columns, cur_keyid); else { - merge_keys[cur_key]= new Ordered_key(cur_key, tmp_table, - item_in->left_expr->element_index(i), - result_sink->get_null_count_of_col(i), - result_sink->get_min_null_of_col(i), - result_sink->get_max_null_of_col(i), - row_num_to_rowid); - if (merge_keys[cur_key]->init(i)) + merge_keys[cur_keyid]= new Ordered_key( + cur_keyid, tmp_table, + item_in->left_expr->element_index(i), + result_sink->get_null_count_of_col(i), + result_sink->get_min_null_of_col(i), + result_sink->get_max_null_of_col(i), + row_num_to_rowid); + if (merge_keys[cur_keyid]->init(i)) { // TIMOUR: revert to partial matching via scanning return TRUE; } - merge_keys[cur_key]->first(); + merge_keys[cur_keyid]->first(); } - ++cur_key; + ++cur_keyid; } } @@ -4453,12 +4500,14 @@ tmp_table->file->ha_rnd_end(); + /* Sort all the keys by their NULL selectivity. */ + my_qsort(merge_keys, keys_count, sizeof(Ordered_key*), + (qsort_cmp) cmp_keys_by_null_selectivity); + /* Sort the keys in each of the indexes. */ for (uint i= 0; i < keys_count; i++) merge_keys[i]->sort_keys(); - // TIMOUR: sort all the keys by NULL selectivity - if (init_queue(&pq, keys_count, 0, FALSE, subselect_rowid_merge_engine::cmp_keys_by_cur_rownum, NULL)) { @@ -4486,20 +4535,38 @@ } +void subselect_rowid_merge_engine::print(String *str, enum_query_type query_type) +{ + str->append(STRING_WITH_LEN("<rowid_merge>(")); + for (uint i= 0; i < keys_count; i++) + merge_keys[i]->print(str); + str->append(')'); +} + + /* + Quick sort comparison function to compare keys in order of decreasing bitmap + selectivity, so that the most selective keys come first. + + @param k1 first key to compare + @param k2 second key to compare + + @retval 1 if k1 is less selective than k2 + @retval 0 if k1 is equally selective as k2 + @retval -1 if k1 is more selective than k2 */ int -subselect_rowid_merge_engine::cmp_keys_by_null_selectivity(Ordered_key *a, - Ordered_key *b) +subselect_rowid_merge_engine::cmp_keys_by_null_selectivity(Ordered_key **k1, + Ordered_key **k2) { - double a_sel= a->null_selectivity(); - double b_sel= b->null_selectivity(); - if (a_sel == b_sel) - return 0; - if (a_sel > b_sel) + double k1_sel= (*k1)->null_selectivity(); + double k2_sel= (*k2)->null_selectivity(); + if (k1_sel < k2_sel) return 1; - return -1; + if (k1_sel > k2_sel) + return -1; + return 0; } @@ -4527,17 +4594,21 @@ bool subselect_rowid_merge_engine::test_null_row(rownum_t row_num) { + Ordered_key *cur_key; + uint cur_id; for (uint i = 0; i < keys_count; i++) { - if (bitmap_is_set(&matching_keys, i)) + cur_key= merge_keys[i]; + cur_id= cur_key->get_keyid(); + if (bitmap_is_set(&matching_keys, cur_id)) { /* - The key 'i' already matches a value in row 'row_num', thus we - skip it as it can't possibly match a NULL. + The key 'i' (with id 'cur_keyid') already matches a value in row 'row_num', + thus we skip it as it can't possibly match a NULL. */ continue; } - if (!merge_keys[i]->is_null(row_num)) + if (!cur_key->is_null(row_num)) return FALSE; } return TRUE; @@ -4583,7 +4654,7 @@ if (merge_keys[i]->get_search_key(0)->is_null()) { ++count_nulls_in_search_key; - bitmap_set_bit(&matching_outer_cols, merge_keys[i]->get_key_idx()); + bitmap_set_bit(&matching_outer_cols, merge_keys[i]->get_keyid()); } else if (merge_keys[i]->lookup()) queue_insert(&pq, (uchar *) merge_keys[i]); @@ -4610,7 +4681,7 @@ min_key= (Ordered_key*) queue_remove(&pq, 0); min_row_num= min_key->current(); bitmap_copy(&matching_keys, &null_only_columns); - bitmap_set_bit(&matching_keys, min_key->get_key_idx()); + bitmap_set_bit(&matching_keys, min_key->get_keyid()); bitmap_union(&matching_keys, &matching_outer_cols); if (min_key->next_same()) queue_insert(&pq, (uchar *) min_key); @@ -4633,7 +4704,7 @@ cur_row_num= cur_key->current(); if (cur_row_num == min_row_num) - bitmap_set_bit(&matching_keys, cur_key->get_key_idx()); + bitmap_set_bit(&matching_keys, cur_key->get_keyid()); else { /* Follows from the correct use of priority queue. */ @@ -4645,7 +4716,7 @@ min_key= cur_key; min_row_num= cur_row_num; bitmap_copy(&matching_keys, &null_only_columns); - bitmap_set_bit(&matching_keys, min_key->get_key_idx()); + bitmap_set_bit(&matching_keys, min_key->get_keyid()); bitmap_union(&matching_keys, &matching_outer_cols); } } === modified file 'sql/item_subselect.h' --- a/sql/item_subselect.h 2010-02-19 21:55:57 +0000 +++ b/sql/item_subselect.h 2010-02-22 15:16:55 +0000 @@ -752,7 +752,7 @@ Index of the key in an array of keys. This index allows to construct (sub)sets of keys represented by bitmaps. */ - uint key_idx; + uint keyid; /* The table being indexed. */ TABLE *tbl; /* The columns being indexed. */ @@ -810,7 +810,7 @@ public: static void *operator new(size_t size) throw () { return sql_alloc(size); } - Ordered_key(uint key_idx_arg, TABLE *tbl_arg, + Ordered_key(uint keyid_arg, TABLE *tbl_arg, Item *search_key_arg, ha_rows null_count_arg, ha_rows min_null_row_arg, ha_rows max_null_row_arg, uchar *row_num_to_rowid_arg); @@ -822,7 +822,7 @@ bool init(int col_idx); uint get_column_count() { return key_column_count; } - uint get_key_idx() { return key_idx; } + uint get_keyid() { return keyid; } uint get_field_idx(uint i) { DBUG_ASSERT(i < key_column_count); @@ -841,8 +841,7 @@ } void sort_keys(); - - double null_selectivity() { return (1 - null_count / null_key.n_bits); } + double null_selectivity(); /* Position the current element at the first row that matches the key. @@ -896,6 +895,7 @@ return FALSE; return bitmap_is_set(&null_key, row_num); } + void print(String *str); }; @@ -957,10 +957,10 @@ bool has_covering_null_row; protected: /* - Comparison function to compare keys in order of increasing bitmap + Comparison function to compare keys in order of decreasing bitmap selectivity. */ - static int cmp_keys_by_null_selectivity(Ordered_key *a, Ordered_key *b); + static int cmp_keys_by_null_selectivity(Ordered_key **k1, Ordered_key **k2); /* Comparison function used by the priority queue pq, the 'smaller' key is the one with the smaller current row number. @@ -992,7 +992,7 @@ uint8 uncacheable() { return UNCACHEABLE_DEPENDENT; } void exclude() {} table_map upper_select_const_tables() { return 0; } - void print(String*, enum_query_type) {} + void print(String*, enum_query_type); bool change_result(Item_subselect*, select_result_interceptor*) { DBUG_ASSERT(FALSE); return false; } bool no_tables() { return false; }

1 0

[Maria-developers] bzr commit into file:///home/tsk/mprog/src/5.3-mwl68-unmerged/ branch (timour:2764)
by timour＠askmonty.org 22 Feb '10

22 Feb '10

#At file:///home/tsk/mprog/src/5.3-mwl68-unmerged/ based on revid:timour@askmonty.org-20100222135709-3568ya6z76hkwfzs 2764 timour(a)askmonty.org 2010-02-22 MWL#68 Subquery optimization: Efficient NOT IN execution with NULLs This patch mainly adds sorting of all indexes for partial matching according to their NULL selectivity. The patch also fixes a related bug in subselect_rowid_merge_engine::test_null_row() where the wrong matched indexes were skipped. In addition the patch: - adds few ::print() methods, - renames few variables that had similar names but different purpose. modified: sql/item_subselect.cc sql/item_subselect.h === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-02-22 13:57:09 +0000 +++ b/sql/item_subselect.cc 2010-02-22 15:16:55 +0000 @@ -3424,7 +3424,10 @@ void subselect_hash_sj_engine::set_strat DBUG_ENTER("subselect_hash_sj_engine::set_strategy_using_schema"); if (item_in->is_top_level_item()) + { strategy= COMPLETE_MATCH; + DBUG_VOID_RETURN; + } else { List_iterator<Item> inner_col_it(*item_in->unit->get_unit_column_types()); @@ -3846,6 +3849,7 @@ int subselect_hash_sj_engine::exec() thd->lex->current_select= materialize_engine->select_lex; if ((res= materialize_join->optimize())) goto err; /* purecov: inspected */ + DBUG_ASSERT(!is_materialized); /* We should materialize only once. */ materialize_join->exec(); if ((res= test(materialize_join->error || thd->is_fatal_error))) goto err; @@ -3952,7 +3956,7 @@ void subselect_hash_sj_engine::print(Str lookup_engine->print(str, query_type); else str->append(STRING_WITH_LEN( - "<the access method for lookups is not yet created>" + "<engine selected at execution time>" )); } @@ -3980,11 +3984,10 @@ bool subselect_hash_sj_engine::change_re } -Ordered_key::Ordered_key(uint key_idx_arg, TABLE *tbl_arg, - Item *search_key_arg, ha_rows null_count_arg, - ha_rows min_null_row_arg, ha_rows max_null_row_arg, - uchar *row_num_to_rowid_arg) - : key_idx(key_idx_arg), tbl(tbl_arg), search_key(search_key_arg), +Ordered_key::Ordered_key(uint keyid_arg, TABLE *tbl_arg, Item *search_key_arg, + ha_rows null_count_arg, ha_rows min_null_row_arg, + ha_rows max_null_row_arg, uchar *row_num_to_rowid_arg) + : keyid(keyid_arg), tbl(tbl_arg), search_key(search_key_arg), row_num_to_rowid(row_num_to_rowid_arg), null_count(null_count_arg) { DBUG_ASSERT(tbl->file->stats.records > null_count); @@ -4190,6 +4193,21 @@ void Ordered_key::sort_keys() /* + The probability that a certain row does not contain a NULL in some row in + a NULL-indexed column. + @retval 1 if there are no NULLs + @retval 0 if only NULLs +*/ + +double Ordered_key::null_selectivity() +{ + /* We should not be processing empty tables. */ + DBUG_ASSERT(tbl->file->stats.records); + return (1 - (double) null_count / (double) tbl->file->stats.records); +} + + +/* Compare the value(s) of the current key in 'search_key' with the data of the current table record. @@ -4307,6 +4325,34 @@ bool Ordered_key::next_same() } +void Ordered_key::print(String *str) +{ + uint i; + str->append("{idx="); + str->qs_append(keyid); + str->append(", ("); + for (i= 0; i < key_column_count - 1; i++) + { + str->append(key_columns[i]->field->field_name); + str->append(", "); + } + str->append(key_columns[i]->field->field_name); + str->append("), "); + + str->append("null_bitmap: (bits="); + str->qs_append(null_key.n_bits); + str->append(", nulls= "); + str->qs_append((double)null_count); + str->append(", min_null= "); + str->qs_append((double)min_null_row); + str->append(", max_null= "); + str->qs_append((double)max_null_row); + str->append("), "); + + str->append('}'); +} + + /* @param non_null_key_parts @param partial_match_key_parts A union of all single-column NULL key parts. @@ -4323,7 +4369,7 @@ subselect_rowid_merge_engine::init(MY_BI rownum_t cur_rownum= 0; select_materialize_with_stats *result_sink= (select_materialize_with_stats *) result; - uint cur_key= 0; + uint cur_keyid= 0; Item_in_subselect *item_in= (Item_in_subselect*) item; int error; @@ -4346,16 +4392,16 @@ subselect_rowid_merge_engine::init(MY_BI /* Create the only non-NULL key if there is any. */ if (non_null_key_parts) { - non_null_key= new Ordered_key(cur_key, tmp_table, item_in->left_expr, + non_null_key= new Ordered_key(cur_keyid, tmp_table, item_in->left_expr, 0, 0, 0, row_num_to_rowid); if (non_null_key->init(non_null_key_parts)) { // TIMOUR: revert to partial matching via scanning return TRUE; } - merge_keys[cur_key]= non_null_key; - merge_keys[cur_key]->first(); - ++cur_key; + merge_keys[cur_keyid]= non_null_key; + merge_keys[cur_keyid]->first(); + ++cur_keyid; } /* @@ -4379,23 +4425,24 @@ subselect_rowid_merge_engine::init(MY_BI continue; if (result_sink->get_null_count_of_col(i) == row_count) - bitmap_set_bit(&null_only_columns, cur_key); + bitmap_set_bit(&null_only_columns, cur_keyid); else { - merge_keys[cur_key]= new Ordered_key(cur_key, tmp_table, - item_in->left_expr->element_index(i), - result_sink->get_null_count_of_col(i), - result_sink->get_min_null_of_col(i), - result_sink->get_max_null_of_col(i), - row_num_to_rowid); - if (merge_keys[cur_key]->init(i)) + merge_keys[cur_keyid]= new Ordered_key( + cur_keyid, tmp_table, + item_in->left_expr->element_index(i), + result_sink->get_null_count_of_col(i), + result_sink->get_min_null_of_col(i), + result_sink->get_max_null_of_col(i), + row_num_to_rowid); + if (merge_keys[cur_keyid]->init(i)) { // TIMOUR: revert to partial matching via scanning return TRUE; } - merge_keys[cur_key]->first(); + merge_keys[cur_keyid]->first(); } - ++cur_key; + ++cur_keyid; } } @@ -4453,12 +4500,14 @@ subselect_rowid_merge_engine::init(MY_BI tmp_table->file->ha_rnd_end(); + /* Sort all the keys by their NULL selectivity. */ + my_qsort(merge_keys, keys_count, sizeof(Ordered_key*), + (qsort_cmp) cmp_keys_by_null_selectivity); + /* Sort the keys in each of the indexes. */ for (uint i= 0; i < keys_count; i++) merge_keys[i]->sort_keys(); - // TIMOUR: sort all the keys by NULL selectivity - if (init_queue(&pq, keys_count, 0, FALSE, subselect_rowid_merge_engine::cmp_keys_by_cur_rownum, NULL)) { @@ -4486,20 +4535,38 @@ void subselect_rowid_merge_engine::clean } +void subselect_rowid_merge_engine::print(String *str, enum_query_type query_type) +{ + str->append(STRING_WITH_LEN("<rowid_merge>(")); + for (uint i= 0; i < keys_count; i++) + merge_keys[i]->print(str); + str->append(')'); +} + + /* + Quick sort comparison function to compare keys in order of decreasing bitmap + selectivity, so that the most selective keys come first. + + @param k1 first key to compare + @param k2 second key to compare + + @retval 1 if k1 is less selective than k2 + @retval 0 if k1 is equally selective as k2 + @retval -1 if k1 is more selective than k2 */ int -subselect_rowid_merge_engine::cmp_keys_by_null_selectivity(Ordered_key *a, - Ordered_key *b) +subselect_rowid_merge_engine::cmp_keys_by_null_selectivity(Ordered_key **k1, + Ordered_key **k2) { - double a_sel= a->null_selectivity(); - double b_sel= b->null_selectivity(); - if (a_sel == b_sel) - return 0; - if (a_sel > b_sel) + double k1_sel= (*k1)->null_selectivity(); + double k2_sel= (*k2)->null_selectivity(); + if (k1_sel < k2_sel) return 1; - return -1; + if (k1_sel > k2_sel) + return -1; + return 0; } @@ -4527,17 +4594,21 @@ subselect_rowid_merge_engine::cmp_keys_b bool subselect_rowid_merge_engine::test_null_row(rownum_t row_num) { + Ordered_key *cur_key; + uint cur_id; for (uint i = 0; i < keys_count; i++) { - if (bitmap_is_set(&matching_keys, i)) + cur_key= merge_keys[i]; + cur_id= cur_key->get_keyid(); + if (bitmap_is_set(&matching_keys, cur_id)) { /* - The key 'i' already matches a value in row 'row_num', thus we - skip it as it can't possibly match a NULL. + The key 'i' (with id 'cur_keyid') already matches a value in row 'row_num', + thus we skip it as it can't possibly match a NULL. */ continue; } - if (!merge_keys[i]->is_null(row_num)) + if (!cur_key->is_null(row_num)) return FALSE; } return TRUE; @@ -4583,7 +4654,7 @@ bool subselect_rowid_merge_engine::parti if (merge_keys[i]->get_search_key(0)->is_null()) { ++count_nulls_in_search_key; - bitmap_set_bit(&matching_outer_cols, merge_keys[i]->get_key_idx()); + bitmap_set_bit(&matching_outer_cols, merge_keys[i]->get_keyid()); } else if (merge_keys[i]->lookup()) queue_insert(&pq, (uchar *) merge_keys[i]); @@ -4610,7 +4681,7 @@ bool subselect_rowid_merge_engine::parti min_key= (Ordered_key*) queue_remove(&pq, 0); min_row_num= min_key->current(); bitmap_copy(&matching_keys, &null_only_columns); - bitmap_set_bit(&matching_keys, min_key->get_key_idx()); + bitmap_set_bit(&matching_keys, min_key->get_keyid()); bitmap_union(&matching_keys, &matching_outer_cols); if (min_key->next_same()) queue_insert(&pq, (uchar *) min_key); @@ -4633,7 +4704,7 @@ bool subselect_rowid_merge_engine::parti cur_row_num= cur_key->current(); if (cur_row_num == min_row_num) - bitmap_set_bit(&matching_keys, cur_key->get_key_idx()); + bitmap_set_bit(&matching_keys, cur_key->get_keyid()); else { /* Follows from the correct use of priority queue. */ @@ -4645,7 +4716,7 @@ bool subselect_rowid_merge_engine::parti min_key= cur_key; min_row_num= cur_row_num; bitmap_copy(&matching_keys, &null_only_columns); - bitmap_set_bit(&matching_keys, min_key->get_key_idx()); + bitmap_set_bit(&matching_keys, min_key->get_keyid()); bitmap_union(&matching_keys, &matching_outer_cols); } } === modified file 'sql/item_subselect.h' --- a/sql/item_subselect.h 2010-02-19 21:55:57 +0000 +++ b/sql/item_subselect.h 2010-02-22 15:16:55 +0000 @@ -752,7 +752,7 @@ protected: Index of the key in an array of keys. This index allows to construct (sub)sets of keys represented by bitmaps. */ - uint key_idx; + uint keyid; /* The table being indexed. */ TABLE *tbl; /* The columns being indexed. */ @@ -810,7 +810,7 @@ protected: public: static void *operator new(size_t size) throw () { return sql_alloc(size); } - Ordered_key(uint key_idx_arg, TABLE *tbl_arg, + Ordered_key(uint keyid_arg, TABLE *tbl_arg, Item *search_key_arg, ha_rows null_count_arg, ha_rows min_null_row_arg, ha_rows max_null_row_arg, uchar *row_num_to_rowid_arg); @@ -822,7 +822,7 @@ public: bool init(int col_idx); uint get_column_count() { return key_column_count; } - uint get_key_idx() { return key_idx; } + uint get_keyid() { return keyid; } uint get_field_idx(uint i) { DBUG_ASSERT(i < key_column_count); @@ -841,8 +841,7 @@ public: } void sort_keys(); - - double null_selectivity() { return (1 - null_count / null_key.n_bits); } + double null_selectivity(); /* Position the current element at the first row that matches the key. @@ -896,6 +895,7 @@ public: return FALSE; return bitmap_is_set(&null_key, row_num); } + void print(String *str); }; @@ -957,10 +957,10 @@ protected: bool has_covering_null_row; protected: /* - Comparison function to compare keys in order of increasing bitmap + Comparison function to compare keys in order of decreasing bitmap selectivity. */ - static int cmp_keys_by_null_selectivity(Ordered_key *a, Ordered_key *b); + static int cmp_keys_by_null_selectivity(Ordered_key **k1, Ordered_key **k2); /* Comparison function used by the priority queue pq, the 'smaller' key is the one with the smaller current row number. @@ -992,7 +992,7 @@ public: uint8 uncacheable() { return UNCACHEABLE_DEPENDENT; } void exclude() {} table_map upper_select_const_tables() { return 0; } - void print(String*, enum_query_type) {} + void print(String*, enum_query_type); bool change_result(Item_subselect*, select_result_interceptor*) { DBUG_ASSERT(FALSE); return false; } bool no_tables() { return false; }

1 0

[Maria-developers] Rev 2763: Automerge with 5.3-subqueries in file:///home/tsk/mprog/src/5.3-mwl68-unmerged/
by timour＠askmonty.org 22 Feb '10

22 Feb '10

At file:///home/tsk/mprog/src/5.3-mwl68-unmerged/ ------------------------------------------------------------ revno: 2763 [merge] revision-id: timour(a)askmonty.org-20100222135709-3568ya6z76hkwfzs parent: timour(a)askmonty.org-20100219215557-g1uohrur8bopfv7n parent: psergey(a)askmonty.org-20100221075312-fc08qgn72dnbudd5 committer: timour(a)askmonty.org branch nick: 5.3-mwl68-unmerged timestamp: Mon 2010-02-22 15:57:09 +0200 message: Automerge with 5.3-subqueries modified: mysql-test/r/subselect_no_mat.result subselect_no_mat.res-20100117143924-hut18sl9k2c7qdj8-1 mysql-test/r/subselect_no_opts.result subselect_no_opts.re-20100117143925-pabg7o8iyokjlu93-1 mysql-test/r/subselect_no_semijoin.result subselect_no_semijoi-20100117143925-9yfygtcm7fwsuq2p-1 mysql-test/r/subselect_sj.result subselect_sj.result-20100117143926-nrop4ku355g3kv8b-1 mysql-test/r/subselect_sj_jcl6.result subselect_sj_jcl6.re-20100117143928-7vzk51yaf29cdavp-1 mysql-test/t/subselect_no_mat.test subselect_no_mat.tes-20100117143930-gdl0f5gsvv77pvtm-1 mysql-test/t/subselect_no_opts.test subselect_no_opts.te-20100117143930-hqzf7r8bjr0gu6td-1 mysql-test/t/subselect_no_semijoin.test subselect_no_semijoi-20100117143930-9yvcpokp2ayxhkju-1 mysql-test/t/subselect_sj.test subselect_sj.test-20100117143931-qp396ufpe3k0scre-1 sql/item.cc sp1f-item.cc-19700101030959-u7hxqopwpfly4kf5ctlyk2dvrq4l3dhn sql/item.h sp1f-item.h-19700101030959-rrkb43htudd62batmoteashkebcwykpa sql/item_subselect.cc sp1f-item_subselect.cc-20020512204640-qep43aqhsfrwkqmrobni6czc3fqj36oo sql/opt_table_elimination.cc opt_table_eliminatio-20090625095316-7ka9w3zr7n5114iv-1 === modified file 'mysql-test/r/subselect_no_mat.result' --- a/mysql-test/r/subselect_no_mat.result 2010-02-17 21:59:41 +0000 +++ b/mysql-test/r/subselect_no_mat.result 2010-02-21 07:33:54 +0000 @@ -1,6 +1,6 @@ show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='materialization=off'; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t11,t12; select (select 2); @@ -1360,9 +1360,9 @@ 3 explain extended select * from t2 where t2.a in (select t1.a from t1,t3 where t1.b=t3.a); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index a a 5 NULL 4 100.00 Using where; Using index -1 PRIMARY t3 index a a 5 NULL 3 100.00 Using where; Using index -1 PRIMARY t1 ref a a 10 test.t2.a,test.t3.a 116 100.00 Using index; FirstMatch(t2) +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index +1 PRIMARY t3 index a a 5 NULL 3 100.00 Using index +1 PRIMARY t1 ref a a 10 test.t2.a,test.t3.a 116 100.61 Using index; FirstMatch(t2) Warnings: Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1` join `test`.`t3`) where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t1`.`b` = `test`.`t3`.`a`)) insert into t1 values (3,31); @@ -4811,4 +4811,4 @@ set optimizer_switch=default; show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on === modified file 'mysql-test/r/subselect_no_opts.result' --- a/mysql-test/r/subselect_no_opts.result 2010-02-17 21:59:41 +0000 +++ b/mysql-test/r/subselect_no_opts.result 2010-02-21 07:33:54 +0000 @@ -1,6 +1,6 @@ show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='materialization=off,semijoin=off'; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t11,t12; select (select 2); @@ -4811,4 +4811,4 @@ set optimizer_switch=default; show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on === modified file 'mysql-test/r/subselect_no_semijoin.result' --- a/mysql-test/r/subselect_no_semijoin.result 2010-02-17 21:59:41 +0000 +++ b/mysql-test/r/subselect_no_semijoin.result 2010-02-21 07:33:54 +0000 @@ -1,6 +1,6 @@ show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='semijoin=off'; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t11,t12; select (select 2); @@ -4811,4 +4811,4 @@ set optimizer_switch=default; show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on === modified file 'mysql-test/r/subselect_sj.result' --- a/mysql-test/r/subselect_sj.result 2010-02-17 10:47:55 +0000 +++ b/mysql-test/r/subselect_sj.result 2010-02-21 07:53:12 +0000 @@ -202,39 +202,39 @@ select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='default,materialization=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=on set optimizer_switch='default,semijoin=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off set optimizer_switch='default,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=on set optimizer_switch='default,semijoin=off,materialization=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off set optimizer_switch='default,materialization=off,semijoin=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off set optimizer_switch='default,semijoin=off,materialization=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=off set optimizer_switch='default,semijoin=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=off set optimizer_switch='default,materialization=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=on set optimizer_switch=default; drop table t0, t1, t2; drop table t10, t11, t12; === modified file 'mysql-test/r/subselect_sj_jcl6.result' --- a/mysql-test/r/subselect_sj_jcl6.result 2010-02-17 10:47:55 +0000 +++ b/mysql-test/r/subselect_sj_jcl6.result 2010-02-21 07:53:12 +0000 @@ -206,39 +206,39 @@ select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='default,materialization=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=on set optimizer_switch='default,semijoin=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off set optimizer_switch='default,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=on set optimizer_switch='default,semijoin=off,materialization=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off set optimizer_switch='default,materialization=off,semijoin=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off set optimizer_switch='default,semijoin=off,materialization=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=off set optimizer_switch='default,semijoin=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=off set optimizer_switch='default,materialization=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=on set optimizer_switch=default; drop table t0, t1, t2; drop table t10, t11, t12; === modified file 'mysql-test/t/subselect_no_mat.test' --- a/mysql-test/t/subselect_no_mat.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_no_mat.test 2010-02-21 07:33:54 +0000 @@ -1,11 +1,13 @@ # # Run subselect.test without semi-join optimization (test materialize) # +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; set optimizer_switch='materialization=off'; --source t/subselect.test set optimizer_switch=default; +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; === modified file 'mysql-test/t/subselect_no_opts.test' --- a/mysql-test/t/subselect_no_opts.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_no_opts.test 2010-02-21 07:33:54 +0000 @@ -1,11 +1,13 @@ # # Run subselect.test without semi-join optimization (test materialize) # +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; set optimizer_switch='materialization=off,semijoin=off'; --source t/subselect.test set optimizer_switch=default; +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; === modified file 'mysql-test/t/subselect_no_semijoin.test' --- a/mysql-test/t/subselect_no_semijoin.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_no_semijoin.test 2010-02-21 07:33:54 +0000 @@ -1,11 +1,13 @@ # # Run subselect.test without semi-join optimization (test materialize) # +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; set optimizer_switch='semijoin=off'; --source t/subselect.test set optimizer_switch=default; +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; === modified file 'mysql-test/t/subselect_sj.test' --- a/mysql-test/t/subselect_sj.test 2010-02-11 23:59:58 +0000 +++ b/mysql-test/t/subselect_sj.test 2010-02-21 07:53:12 +0000 @@ -96,30 +96,39 @@ --echo BUG#37120 optimizer_switch allowable values not according to specification --echo +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,materialization=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,semijoin=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,loosescan=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,semijoin=off,materialization=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,materialization=off,semijoin=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,semijoin=off,materialization=off,loosescan=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,semijoin=off,loosescan=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,materialization=off,loosescan=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch=default; === modified file 'sql/item.cc' --- a/sql/item.cc 2010-02-11 23:59:58 +0000 +++ b/sql/item.cc 2010-02-21 06:32:23 +0000 @@ -1959,7 +1959,7 @@ bool Item_field::enumerate_field_refs_processor(uchar *arg) { Field_enumerator *fe= (Field_enumerator*)arg; - fe->visit_field(field); + fe->visit_field(this); return FALSE; } @@ -5779,6 +5779,35 @@ set_properties(); } +/* + A Field_enumerator-compatible class that invokes mark_as_dependent() for + each field that is a reference to some ancestor of current_select. +*/ +class Dependency_marker: public Field_enumerator +{ +public: + THD *thd; + st_select_lex *current_select; + virtual void visit_field(Item_field *item) + { + // Find which select the field is in. This is achieved by walking up + // the select tree and looking for the table of interest. + st_select_lex *sel; + for (sel= current_select; sel; sel= sel->outer_select()) + { + TABLE_LIST *tbl; + for (tbl= sel->leaf_tables; tbl; tbl= tbl->next_leaf) + { + if (tbl->table == item->field->table) + { + if (sel != current_select) + mark_as_dependent(thd, sel, current_select, item, item); + return; + } + } + } + } +}; /** Resolve the name of a reference to a column reference. @@ -6038,6 +6067,20 @@ last_checked_context->select_lex->nest_level); } } + else + { + ; + /* + It could be that we're referring to something that's in ancestor selects. + We must make an appropriate mark_as_dependent() call for each such + outside reference. + */ + Dependency_marker dep_marker; + dep_marker.current_select= current_sel; + dep_marker.thd= thd; + (*ref)->walk(&Item::enumerate_field_refs_processor, FALSE, + (uchar*)&dep_marker); + } DBUG_ASSERT(*ref); /* === modified file 'sql/item.h' --- a/sql/item.h 2010-02-17 10:05:27 +0000 +++ b/sql/item.h 2010-02-21 06:32:23 +0000 @@ -1134,7 +1134,7 @@ class Field_enumerator { public: - virtual void visit_field(Field *field)= 0; + virtual void visit_field(Item_field *field)= 0; virtual ~Field_enumerator() {}; /* purecov: inspected */ }; @@ -2378,7 +2378,12 @@ return ref ? (*ref)->real_item() : this; } bool walk(Item_processor processor, bool walk_subquery, uchar *arg) - { return (*ref)->walk(processor, walk_subquery, arg); } + { + if (ref && *ref) + return (*ref)->walk(processor, walk_subquery, arg); + else + return FALSE; + } bool enumerate_field_refs_processor(uchar *arg) { return (*ref)->enumerate_field_refs_processor(arg); } virtual void print(String *str, enum_query_type query_type); === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-02-19 21:55:57 +0000 +++ b/sql/item_subselect.cc 2010-02-22 13:57:09 +0000 @@ -168,6 +168,11 @@ { done_first_fix_fields= TRUE; inside_first_fix_fields= TRUE; + upper_refs.empty(); + /* + psergey-todo: remove _first_fix_fields calls, we need changes on every + execution + */ } eliminated= FALSE; @@ -182,6 +187,7 @@ changed= 1; inside_first_fix_fields= FALSE; + if (!res) { // all transformation is done (used by prepared statements) @@ -212,12 +218,14 @@ if (!(*ref)->fixed) ret= (*ref)->fix_fields(thd, ref); thd->where= save_where; + done_first_fix_fields= FALSE; return ret; } // Is it one field subselect? if (engine->cols() > max_columns) { my_error(ER_OPERAND_COLUMNS, MYF(0), 1); + done_first_fix_fields= FALSE; return TRUE; } fix_length_and_dec(); @@ -234,6 +242,7 @@ fixed= 1; err: + done_first_fix_fields= FALSE; thd->where= save_where; return res; } @@ -276,6 +285,7 @@ return FALSE; } + /* Adjust attributes after our parent select has been merged into grandparent @@ -304,18 +314,19 @@ parent_select= new_parent; } + class Field_fixer: public Field_enumerator { public: table_map used_tables; /* Collect used_tables here */ st_select_lex *new_parent; /* Select we're in */ - virtual void visit_field(Field *field) + virtual void visit_field(Item_field *item) { //for (TABLE_LIST *tbl= new_parent->leaf_tables; tbl; tbl= tbl->next_local) //{ // if (tbl->table == field->table) // { - used_tables|= field->table->map; + used_tables|= item->field->table->map; // return; // } //} === modified file 'sql/opt_table_elimination.cc' --- a/sql/opt_table_elimination.cc 2010-01-17 14:51:10 +0000 +++ b/sql/opt_table_elimination.cc 2010-02-21 06:32:23 +0000 @@ -922,8 +922,9 @@ Field_dependency_recorder(Dep_analysis_context *ctx_arg): ctx(ctx_arg) {} - void visit_field(Field *field) + void visit_field(Item_field *item) { + Field *field= item->field; Dep_value_table *tbl_dep; if ((tbl_dep= ctx->table_deps[field->table->tablenr])) {

1 0

[Maria-developers] bzr commit into file:///home/tsk/mprog/src/5.3-mwl68-unmerged/ branch (timour:2763)
by timour＠askmonty.org 22 Feb '10

22 Feb '10

#At file:///home/tsk/mprog/src/5.3-mwl68-unmerged/ based on revid:timour@askmonty.org-20100219215557-g1uohrur8bopfv7n 2763 timour(a)askmonty.org 2010-02-22 [merge] Automerge with 5.3-subqueries modified: mysql-test/r/subselect_no_mat.result mysql-test/r/subselect_no_opts.result mysql-test/r/subselect_no_semijoin.result mysql-test/r/subselect_sj.result mysql-test/r/subselect_sj_jcl6.result mysql-test/t/subselect_no_mat.test mysql-test/t/subselect_no_opts.test mysql-test/t/subselect_no_semijoin.test mysql-test/t/subselect_sj.test sql/item.cc sql/item.h sql/item_subselect.cc sql/opt_table_elimination.cc === modified file 'mysql-test/r/subselect_no_mat.result' --- a/mysql-test/r/subselect_no_mat.result 2010-02-17 21:59:41 +0000 +++ b/mysql-test/r/subselect_no_mat.result 2010-02-21 07:33:54 +0000 @@ -1,6 +1,6 @@ show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='materialization=off'; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t11,t12; select (select 2); @@ -1360,9 +1360,9 @@ a 3 explain extended select * from t2 where t2.a in (select t1.a from t1,t3 where t1.b=t3.a); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index a a 5 NULL 4 100.00 Using where; Using index -1 PRIMARY t3 index a a 5 NULL 3 100.00 Using where; Using index -1 PRIMARY t1 ref a a 10 test.t2.a,test.t3.a 116 100.00 Using index; FirstMatch(t2) +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index +1 PRIMARY t3 index a a 5 NULL 3 100.00 Using index +1 PRIMARY t1 ref a a 10 test.t2.a,test.t3.a 116 100.61 Using index; FirstMatch(t2) Warnings: Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1` join `test`.`t3`) where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t1`.`b` = `test`.`t3`.`a`)) insert into t1 values (3,31); @@ -4811,4 +4811,4 @@ End of 5.1 tests. set optimizer_switch=default; show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on === modified file 'mysql-test/r/subselect_no_opts.result' --- a/mysql-test/r/subselect_no_opts.result 2010-02-17 21:59:41 +0000 +++ b/mysql-test/r/subselect_no_opts.result 2010-02-21 07:33:54 +0000 @@ -1,6 +1,6 @@ show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='materialization=off,semijoin=off'; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t11,t12; select (select 2); @@ -4811,4 +4811,4 @@ End of 5.1 tests. set optimizer_switch=default; show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on === modified file 'mysql-test/r/subselect_no_semijoin.result' --- a/mysql-test/r/subselect_no_semijoin.result 2010-02-17 21:59:41 +0000 +++ b/mysql-test/r/subselect_no_semijoin.result 2010-02-21 07:33:54 +0000 @@ -1,6 +1,6 @@ show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='semijoin=off'; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t11,t12; select (select 2); @@ -4811,4 +4811,4 @@ End of 5.1 tests. set optimizer_switch=default; show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on === modified file 'mysql-test/r/subselect_sj.result' --- a/mysql-test/r/subselect_sj.result 2010-02-17 10:47:55 +0000 +++ b/mysql-test/r/subselect_sj.result 2010-02-21 07:53:12 +0000 @@ -202,39 +202,39 @@ BUG#37120 optimizer_switch allowable val select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='default,materialization=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=on set optimizer_switch='default,semijoin=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off set optimizer_switch='default,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=on set optimizer_switch='default,semijoin=off,materialization=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off set optimizer_switch='default,materialization=off,semijoin=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off set optimizer_switch='default,semijoin=off,materialization=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=off set optimizer_switch='default,semijoin=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=off set optimizer_switch='default,materialization=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=on set optimizer_switch=default; drop table t0, t1, t2; drop table t10, t11, t12; === modified file 'mysql-test/r/subselect_sj_jcl6.result' --- a/mysql-test/r/subselect_sj_jcl6.result 2010-02-17 10:47:55 +0000 +++ b/mysql-test/r/subselect_sj_jcl6.result 2010-02-21 07:53:12 +0000 @@ -206,39 +206,39 @@ BUG#37120 optimizer_switch allowable val select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='default,materialization=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=on set optimizer_switch='default,semijoin=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off set optimizer_switch='default,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=on set optimizer_switch='default,semijoin=off,materialization=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off set optimizer_switch='default,materialization=off,semijoin=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off set optimizer_switch='default,semijoin=off,materialization=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=off set optimizer_switch='default,semijoin=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=off set optimizer_switch='default,materialization=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=on set optimizer_switch=default; drop table t0, t1, t2; drop table t10, t11, t12; === modified file 'mysql-test/t/subselect_no_mat.test' --- a/mysql-test/t/subselect_no_mat.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_no_mat.test 2010-02-21 07:33:54 +0000 @@ -1,11 +1,13 @@ # # Run subselect.test without semi-join optimization (test materialize) # +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; set optimizer_switch='materialization=off'; --source t/subselect.test set optimizer_switch=default; +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; === modified file 'mysql-test/t/subselect_no_opts.test' --- a/mysql-test/t/subselect_no_opts.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_no_opts.test 2010-02-21 07:33:54 +0000 @@ -1,11 +1,13 @@ # # Run subselect.test without semi-join optimization (test materialize) # +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; set optimizer_switch='materialization=off,semijoin=off'; --source t/subselect.test set optimizer_switch=default; +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; === modified file 'mysql-test/t/subselect_no_semijoin.test' --- a/mysql-test/t/subselect_no_semijoin.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_no_semijoin.test 2010-02-21 07:33:54 +0000 @@ -1,11 +1,13 @@ # # Run subselect.test without semi-join optimization (test materialize) # +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; set optimizer_switch='semijoin=off'; --source t/subselect.test set optimizer_switch=default; +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; === modified file 'mysql-test/t/subselect_sj.test' --- a/mysql-test/t/subselect_sj.test 2010-02-11 23:59:58 +0000 +++ b/mysql-test/t/subselect_sj.test 2010-02-21 07:53:12 +0000 @@ -96,30 +96,39 @@ explain extended select * from t1 where --echo BUG#37120 optimizer_switch allowable values not according to specification --echo +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,materialization=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,semijoin=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,loosescan=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,semijoin=off,materialization=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,materialization=off,semijoin=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,semijoin=off,materialization=off,loosescan=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,semijoin=off,loosescan=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,materialization=off,loosescan=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch=default; === modified file 'sql/item.cc' --- a/sql/item.cc 2010-02-11 23:59:58 +0000 +++ b/sql/item.cc 2010-02-21 06:32:23 +0000 @@ -1959,7 +1959,7 @@ void Item_field::reset_field(Field *f) bool Item_field::enumerate_field_refs_processor(uchar *arg) { Field_enumerator *fe= (Field_enumerator*)arg; - fe->visit_field(field); + fe->visit_field(this); return FALSE; } @@ -5779,6 +5779,35 @@ Item_ref::Item_ref(Name_resolution_conte set_properties(); } +/* + A Field_enumerator-compatible class that invokes mark_as_dependent() for + each field that is a reference to some ancestor of current_select. +*/ +class Dependency_marker: public Field_enumerator +{ +public: + THD *thd; + st_select_lex *current_select; + virtual void visit_field(Item_field *item) + { + // Find which select the field is in. This is achieved by walking up + // the select tree and looking for the table of interest. + st_select_lex *sel; + for (sel= current_select; sel; sel= sel->outer_select()) + { + TABLE_LIST *tbl; + for (tbl= sel->leaf_tables; tbl; tbl= tbl->next_leaf) + { + if (tbl->table == item->field->table) + { + if (sel != current_select) + mark_as_dependent(thd, sel, current_select, item, item); + return; + } + } + } + } +}; /** Resolve the name of a reference to a column reference. @@ -6038,6 +6067,20 @@ bool Item_ref::fix_fields(THD *thd, Item last_checked_context->select_lex->nest_level); } } + else + { + ; + /* + It could be that we're referring to something that's in ancestor selects. + We must make an appropriate mark_as_dependent() call for each such + outside reference. + */ + Dependency_marker dep_marker; + dep_marker.current_select= current_sel; + dep_marker.thd= thd; + (*ref)->walk(&Item::enumerate_field_refs_processor, FALSE, + (uchar*)&dep_marker); + } DBUG_ASSERT(*ref); /* === modified file 'sql/item.h' --- a/sql/item.h 2010-02-17 10:05:27 +0000 +++ b/sql/item.h 2010-02-21 06:32:23 +0000 @@ -1134,7 +1134,7 @@ public: class Field_enumerator { public: - virtual void visit_field(Field *field)= 0; + virtual void visit_field(Item_field *field)= 0; virtual ~Field_enumerator() {}; /* purecov: inspected */ }; @@ -2378,7 +2378,12 @@ public: return ref ? (*ref)->real_item() : this; } bool walk(Item_processor processor, bool walk_subquery, uchar *arg) - { return (*ref)->walk(processor, walk_subquery, arg); } + { + if (ref && *ref) + return (*ref)->walk(processor, walk_subquery, arg); + else + return FALSE; + } bool enumerate_field_refs_processor(uchar *arg) { return (*ref)->enumerate_field_refs_processor(arg); } virtual void print(String *str, enum_query_type query_type); === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-02-19 21:55:57 +0000 +++ b/sql/item_subselect.cc 2010-02-22 13:57:09 +0000 @@ -168,6 +168,11 @@ bool Item_subselect::fix_fields(THD *thd { done_first_fix_fields= TRUE; inside_first_fix_fields= TRUE; + upper_refs.empty(); + /* + psergey-todo: remove _first_fix_fields calls, we need changes on every + execution + */ } eliminated= FALSE; @@ -182,6 +187,7 @@ bool Item_subselect::fix_fields(THD *thd changed= 1; inside_first_fix_fields= FALSE; + if (!res) { // all transformation is done (used by prepared statements) @@ -212,12 +218,14 @@ bool Item_subselect::fix_fields(THD *thd if (!(*ref)->fixed) ret= (*ref)->fix_fields(thd, ref); thd->where= save_where; + done_first_fix_fields= FALSE; return ret; } // Is it one field subselect? if (engine->cols() > max_columns) { my_error(ER_OPERAND_COLUMNS, MYF(0), 1); + done_first_fix_fields= FALSE; return TRUE; } fix_length_and_dec(); @@ -234,6 +242,7 @@ bool Item_subselect::fix_fields(THD *thd fixed= 1; err: + done_first_fix_fields= FALSE; thd->where= save_where; return res; } @@ -276,6 +285,7 @@ bool Item_subselect::mark_as_dependent(T return FALSE; } + /* Adjust attributes after our parent select has been merged into grandparent @@ -304,18 +314,19 @@ void Item_subselect::fix_after_pullout(s parent_select= new_parent; } + class Field_fixer: public Field_enumerator { public: table_map used_tables; /* Collect used_tables here */ st_select_lex *new_parent; /* Select we're in */ - virtual void visit_field(Field *field) + virtual void visit_field(Item_field *item) { //for (TABLE_LIST *tbl= new_parent->leaf_tables; tbl; tbl= tbl->next_local) //{ // if (tbl->table == field->table) // { - used_tables|= field->table->map; + used_tables|= item->field->table->map; // return; // } //} === modified file 'sql/opt_table_elimination.cc' --- a/sql/opt_table_elimination.cc 2010-01-17 14:51:10 +0000 +++ b/sql/opt_table_elimination.cc 2010-02-21 06:32:23 +0000 @@ -922,8 +922,9 @@ public: Field_dependency_recorder(Dep_analysis_context *ctx_arg): ctx(ctx_arg) {} - void visit_field(Field *field) + void visit_field(Item_field *item) { + Field *field= item->field; Dep_value_table *tbl_dep; if ((tbl_dep= ctx->table_deps[field->table->tablenr])) {

1 0

[Maria-developers] Rev 2767: Fix buildbot failure: take into account that there is no optimizer_switch flag in file:///home/psergey/dev/maria-5.3-subqueries-r7/
by Sergey Petrunya 21 Feb '10

21 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r7/ ------------------------------------------------------------ revno: 2767 revision-id: psergey(a)askmonty.org-20100221075312-fc08qgn72dnbudd5 parent: psergey(a)askmonty.org-20100221073354-d05otp2nklvja9pc committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r7 timestamp: Sun 2010-02-21 09:53:12 +0200 message: Fix buildbot failure: take into account that there is no optimizer_switch flag for table elimination in debug builds. (part 2) === modified file 'mysql-test/r/subselect_sj.result' --- a/mysql-test/r/subselect_sj.result 2010-02-21 07:33:54 +0000 +++ b/mysql-test/r/subselect_sj.result 2010-02-21 07:53:12 +0000 @@ -202,7 +202,7 @@ select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='default,materialization=off'; select @@optimizer_switch; @@optimizer_switch === modified file 'mysql-test/r/subselect_sj_jcl6.result' --- a/mysql-test/r/subselect_sj_jcl6.result 2010-02-21 07:33:54 +0000 +++ b/mysql-test/r/subselect_sj_jcl6.result 2010-02-21 07:53:12 +0000 @@ -206,7 +206,7 @@ select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='default,materialization=off'; select @@optimizer_switch; @@optimizer_switch === modified file 'mysql-test/t/subselect_sj.test' --- a/mysql-test/t/subselect_sj.test 2010-02-21 07:33:54 +0000 +++ b/mysql-test/t/subselect_sj.test 2010-02-21 07:53:12 +0000 @@ -96,6 +96,7 @@ --echo BUG#37120 optimizer_switch allowable values not according to specification --echo +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,materialization=off';

1 0

[Maria-developers] Rev 2766: Fix buildbot failure: take into account that there is no optimizer_switch flag in file:///home/psergey/dev/maria-5.3-subqueries-r7/
by Sergey Petrunya 21 Feb '10

21 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r7/ ------------------------------------------------------------ revno: 2766 revision-id: psergey(a)askmonty.org-20100221073354-d05otp2nklvja9pc parent: psergey(a)askmonty.org-20100221063223-h0f7u2low7rtjixc committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r7 timestamp: Sun 2010-02-21 09:33:54 +0200 message: Fix buildbot failure: take into account that there is no optimizer_switch flag for table elimination in debug builds. === modified file 'mysql-test/r/subselect_no_mat.result' --- a/mysql-test/r/subselect_no_mat.result 2010-02-17 22:54:59 +0000 +++ b/mysql-test/r/subselect_no_mat.result 2010-02-21 07:33:54 +0000 @@ -1,6 +1,6 @@ show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='materialization=off'; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t11,t12; select (select 2); @@ -4811,4 +4811,4 @@ set optimizer_switch=default; show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on === modified file 'mysql-test/r/subselect_no_opts.result' --- a/mysql-test/r/subselect_no_opts.result 2010-02-17 21:59:41 +0000 +++ b/mysql-test/r/subselect_no_opts.result 2010-02-21 07:33:54 +0000 @@ -1,6 +1,6 @@ show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='materialization=off,semijoin=off'; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t11,t12; select (select 2); @@ -4811,4 +4811,4 @@ set optimizer_switch=default; show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on === modified file 'mysql-test/r/subselect_no_semijoin.result' --- a/mysql-test/r/subselect_no_semijoin.result 2010-02-17 21:59:41 +0000 +++ b/mysql-test/r/subselect_no_semijoin.result 2010-02-21 07:33:54 +0000 @@ -1,6 +1,6 @@ show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on set optimizer_switch='semijoin=off'; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t11,t12; select (select 2); @@ -4811,4 +4811,4 @@ set optimizer_switch=default; show variables like 'optimizer_switch'; Variable_name Value -optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on,table_elimination=on +optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=on === modified file 'mysql-test/r/subselect_sj.result' --- a/mysql-test/r/subselect_sj.result 2010-02-17 10:47:55 +0000 +++ b/mysql-test/r/subselect_sj.result 2010-02-21 07:33:54 +0000 @@ -206,35 +206,35 @@ set optimizer_switch='default,materialization=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=on set optimizer_switch='default,semijoin=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off set optimizer_switch='default,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=on set optimizer_switch='default,semijoin=off,materialization=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off set optimizer_switch='default,materialization=off,semijoin=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off set optimizer_switch='default,semijoin=off,materialization=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=off set optimizer_switch='default,semijoin=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=off set optimizer_switch='default,materialization=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=on set optimizer_switch=default; drop table t0, t1, t2; drop table t10, t11, t12; === modified file 'mysql-test/r/subselect_sj_jcl6.result' --- a/mysql-test/r/subselect_sj_jcl6.result 2010-02-17 10:47:55 +0000 +++ b/mysql-test/r/subselect_sj_jcl6.result 2010-02-21 07:33:54 +0000 @@ -210,35 +210,35 @@ set optimizer_switch='default,materialization=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=on set optimizer_switch='default,semijoin=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off set optimizer_switch='default,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=on set optimizer_switch='default,semijoin=off,materialization=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off set optimizer_switch='default,materialization=off,semijoin=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=off,semijoin=off set optimizer_switch='default,semijoin=off,materialization=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=off set optimizer_switch='default,semijoin=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=off,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=on,semijoin=off set optimizer_switch='default,materialization=off,loosescan=off'; select @@optimizer_switch; @@optimizer_switch -index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=on,table_elimination=on +index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_condition_pushdown=on,firstmatch=on,loosescan=off,materialization=off,semijoin=on set optimizer_switch=default; drop table t0, t1, t2; drop table t10, t11, t12; === modified file 'mysql-test/t/subselect_no_mat.test' --- a/mysql-test/t/subselect_no_mat.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_no_mat.test 2010-02-21 07:33:54 +0000 @@ -1,11 +1,13 @@ # # Run subselect.test without semi-join optimization (test materialize) # +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; set optimizer_switch='materialization=off'; --source t/subselect.test set optimizer_switch=default; +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; === modified file 'mysql-test/t/subselect_no_opts.test' --- a/mysql-test/t/subselect_no_opts.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_no_opts.test 2010-02-21 07:33:54 +0000 @@ -1,11 +1,13 @@ # # Run subselect.test without semi-join optimization (test materialize) # +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; set optimizer_switch='materialization=off,semijoin=off'; --source t/subselect.test set optimizer_switch=default; +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; === modified file 'mysql-test/t/subselect_no_semijoin.test' --- a/mysql-test/t/subselect_no_semijoin.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_no_semijoin.test 2010-02-21 07:33:54 +0000 @@ -1,11 +1,13 @@ # # Run subselect.test without semi-join optimization (test materialize) # +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; set optimizer_switch='semijoin=off'; --source t/subselect.test set optimizer_switch=default; +--replace_regex /,table_elimination=on// show variables like 'optimizer_switch'; === modified file 'mysql-test/t/subselect_sj.test' --- a/mysql-test/t/subselect_sj.test 2010-02-11 23:59:58 +0000 +++ b/mysql-test/t/subselect_sj.test 2010-02-21 07:33:54 +0000 @@ -99,27 +99,35 @@ select @@optimizer_switch; set optimizer_switch='default,materialization=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,semijoin=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,loosescan=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,semijoin=off,materialization=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,materialization=off,semijoin=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,semijoin=off,materialization=off,loosescan=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,semijoin=off,loosescan=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch='default,materialization=off,loosescan=off'; +--replace_regex /,table_elimination=on// select @@optimizer_switch; set optimizer_switch=default;

1 0

[Maria-developers] Rev 2764: * Better self-recursion protection in Item_subselect::fix_fields. in file:///home/psergey/dev/maria-5.3-subqueries-r7/
by Sergey Petrunya 21 Feb '10

21 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r7/ ------------------------------------------------------------ revno: 2764 revision-id: psergey(a)askmonty.org-20100221033618-83dgm2h9ingzmhcc parent: psergey(a)askmonty.org-20100220082329-9esvom4n6mpgeqvk committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r7 timestamp: Sun 2010-02-21 05:36:18 +0200 message: * Better self-recursion protection in Item_subselect::fix_fields. Don't go into branch that calls upper_refs.empty() more than once per PREPARE or EXECUTE * Avoid crashing when processing references to outside from subquery's HAVING (will explain in more details in email) === modified file 'sql/item.h' --- a/sql/item.h 2010-02-17 10:05:27 +0000 +++ b/sql/item.h 2010-02-21 03:36:18 +0000 @@ -2378,7 +2378,12 @@ return ref ? (*ref)->real_item() : this; } bool walk(Item_processor processor, bool walk_subquery, uchar *arg) - { return (*ref)->walk(processor, walk_subquery, arg); } + { + if (ref && *ref) + return (*ref)->walk(processor, walk_subquery, arg); + else + return FALSE; + } bool enumerate_field_refs_processor(uchar *arg) { return (*ref)->enumerate_field_refs_processor(arg); } virtual void print(String *str, enum_query_type query_type); === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-02-20 08:23:29 +0000 +++ b/sql/item_subselect.cc 2010-02-21 03:36:18 +0000 @@ -186,7 +186,6 @@ changed= 1; inside_first_fix_fields= FALSE; - done_first_fix_fields= FALSE; if (!res) { @@ -218,12 +217,14 @@ if (!(*ref)->fixed) ret= (*ref)->fix_fields(thd, ref); thd->where= save_where; + done_first_fix_fields= FALSE; return ret; } // Is it one field subselect? if (engine->cols() > max_columns) { my_error(ER_OPERAND_COLUMNS, MYF(0), 1); + done_first_fix_fields= FALSE; return TRUE; } fix_length_and_dec(); @@ -240,6 +241,7 @@ fixed= 1; err: + done_first_fix_fields= FALSE; thd->where= save_where; return res; } @@ -282,6 +284,7 @@ return FALSE; } + /* Adjust attributes after our parent select has been merged into grandparent @@ -310,6 +313,7 @@ parent_select= new_parent; } + class Field_fixer: public Field_enumerator { public:

1 1

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2795)
by knielsen＠knielsen-hq.org 20 Feb '10

20 Feb '10

#At lp:maria 2795 knielsen(a)knielsen-hq.org 2010-02-20 [merge] automerge latest trunk for buildbot test added: mysql-test/suite/parts/t/partition_repair_myisam-master.opt modified: .bzrignore BUILD/util.sh client/mysqlslap.c client/mysqltest.cc config/ac-macros/libevent.m4 configure.in extra/libevent/devpoll.c extra/yassl/src/yassl_error.cpp extra/yassl/taocrypt/include/asn.hpp extra/yassl/taocrypt/src/asn.cpp include/ft_global.h include/maria.h include/my_base.h include/violite.h libmysqld/CMakeLists.txt mysql-test/mysql-test-run.pl mysql-test/r/grant.result mysql-test/r/openssl_1.result mysql-test/r/show_check.result mysql-test/r/sp-destruct.result mysql-test/r/table_elim.result mysql-test/r/udf.result mysql-test/r/variables.result mysql-test/r/view.result mysql-test/std_data/cacert.pem mysql-test/std_data/client-cert.pem mysql-test/std_data/client-key.pem mysql-test/std_data/server-cert.pem mysql-test/std_data/server-key.pem mysql-test/suite/maria/r/maria3.result mysql-test/suite/maria/t/maria-recovery2-master.opt mysql-test/suite/maria/t/maria3.test mysql-test/suite/rpl/t/rpl_rotate_logs.test mysql-test/t/grant.test mysql-test/t/openssl_1.test mysql-test/t/partition_innodb_semi_consistent.test mysql-test/t/query_cache_debug.test mysql-test/t/show_check-master.opt mysql-test/t/sp-destruct.test mysql-test/t/table_elim.test mysql-test/t/udf.test mysql-test/t/view.test mysys/my_file.c mysys/my_thr_init.c sql-bench/bench-init.pl.sh sql-bench/server-cfg.sh sql-bench/test-ATIS.sh sql-bench/test-alter-table.sh sql-bench/test-big-tables.sh sql-bench/test-connect.sh sql-bench/test-create.sh sql-bench/test-select.sh sql-bench/test-transactions.sh sql-bench/test-wisconsin.sh sql-common/client.c sql/CMakeLists.txt sql/item.cc sql/lock.cc sql/mysql_priv.h sql/mysqld.cc sql/net_serv.cc sql/opt_table_elimination.cc sql/rpl_record.cc sql/slave.cc sql/sql_base.cc sql/sql_cache.cc sql/sql_connect.cc sql/sql_delete.cc sql/sql_parse.cc sql/sql_profile.cc sql/sql_profile.h sql/sql_table.cc sql/table.cc sql/table.h sql/udf_example.def storage/maria/ft_maria.c storage/maria/ha_maria.cc storage/maria/ma_blockrec.c storage/maria/ma_close.c storage/maria/ma_extra.c storage/maria/ma_locking.c storage/maria/ma_recovery.c storage/maria/maria_def.h storage/myisam/ft_boolean_search.c storage/myisam/ft_myisam.c storage/myisam/ft_nlq_search.c storage/myisam/ft_parser.c storage/myisam/ft_stopwords.c storage/myisam/ftdefs.h storage/myisam/mi_close.c storage/myisam/mi_extra.c storage/myisam/mi_open.c storage/myisam/myisamdef.h storage/xtradb/buf/buf0buf.c storage/xtradb/handler/i_s.cc support-files/compiler_warnings.supp unittest/mysys/waiting_threads-t.c vio/viossl.c === modified file '.bzrignore' --- a/.bzrignore 2009-12-22 13:50:20 +0000 +++ b/.bzrignore 2010-01-29 18:42:22 +0000 @@ -40,15 +40,15 @@ *.dsp *.Po *.Plo -*/*.dir/* +*.dir/ */*_pure_*warnings */.deps */.libs/* */.pure -*/debug/* -*/minsizerel/* -*/release/* -*/relwithdebinfo/* +debug/ +MinSizeRel/ +Release/ +RelWithDebInfo/ *~ .*.swp ./CMakeCache.txt @@ -96,7 +96,7 @@ BitKeeper/tmp/gone BitKeeper/tmp BitKeeper/log BitKeeper/etc/SCCS -CMakeFiles/* +CMakeFiles/ COPYING COPYING.LIB Docs/#manual.texi# === modified file 'BUILD/util.sh' --- a/BUILD/util.sh 2010-01-07 12:02:18 +0000 +++ b/BUILD/util.sh 2010-01-28 22:34:22 +0000 @@ -16,6 +16,8 @@ # Setting cpu options. get_cpuopt () { + case "$(uname -o)" in + *Linux*) case "$(gcc -dumpmachine)" in x86_64-*) # gcc barfs on -march=... on x64 @@ -26,7 +28,13 @@ get_cpuopt () { CPUOPT="-m32 -march=i586 -mtune=generic" ;; esac - return 0 + ;; + *Solaris*) + # ToDo: handle 32-bit build? For now default to 64-bit. + CPUOPT="-D__sun -m64 -mtune=athlon64" + ;; + esac + return 0 } # Default to a parallel build, but only if AM_MAKEFLAGS is not set. === modified file 'client/mysqlslap.c' --- a/client/mysqlslap.c 2009-12-03 11:34:11 +0000 +++ b/client/mysqlslap.c 2010-01-29 18:42:22 +0000 @@ -292,6 +292,25 @@ static int gettimeofday(struct timeval * } #endif +void set_mysql_connect_options(MYSQL *mysql) +{ + if (opt_compress) + mysql_options(mysql,MYSQL_OPT_COMPRESS,NullS); +#ifdef HAVE_OPENSSL + if (opt_use_ssl) + mysql_ssl_set(mysql, opt_ssl_key, opt_ssl_cert, opt_ssl_ca, + opt_ssl_capath, opt_ssl_cipher); +#endif + if (opt_protocol) + mysql_options(mysql,MYSQL_OPT_PROTOCOL,(char*)&opt_protocol); +#ifdef HAVE_SMEM + if (shared_memory_base_name) + mysql_options(mysql,MYSQL_SHARED_MEMORY_BASE_NAME,shared_memory_base_name); +#endif + mysql_options(mysql, MYSQL_SET_CHARSET_NAME, default_charset); +} + + int main(int argc, char **argv) { MYSQL mysql; @@ -323,20 +342,7 @@ int main(int argc, char **argv) exit(1); } mysql_init(&mysql); - if (opt_compress) - mysql_options(&mysql,MYSQL_OPT_COMPRESS,NullS); -#ifdef HAVE_OPENSSL - if (opt_use_ssl) - mysql_ssl_set(&mysql, opt_ssl_key, opt_ssl_cert, opt_ssl_ca, - opt_ssl_capath, opt_ssl_cipher); -#endif - if (opt_protocol) - mysql_options(&mysql,MYSQL_OPT_PROTOCOL,(char*)&opt_protocol); -#ifdef HAVE_SMEM - if (shared_memory_base_name) - mysql_options(&mysql,MYSQL_SHARED_MEMORY_BASE_NAME,shared_memory_base_name); -#endif - mysql_options(&mysql, MYSQL_SET_CHARSET_NAME, default_charset); + set_mysql_connect_options(&mysql); if (!opt_only_print) { @@ -1815,6 +1821,7 @@ pthread_handler_t run_task(void *p) my_progname, mysql_error(mysql)); exit(0); } + set_mysql_connect_options(mysql); if (mysql_thread_init()) { @@ -1855,7 +1862,6 @@ limit_not_met: my_progname, mysql_error(mysql)); exit(0); } - if (slap_connect(mysql)) goto end; } @@ -2223,6 +2229,7 @@ slap_connect(MYSQL *mysql) int x, connect_error= 1; for (x= 0; x < 10; x++) { + set_mysql_connect_options(mysql); if (mysql_real_connect(mysql, host, user, opt_password, create_schema_string, opt_mysql_port, === modified file 'client/mysqltest.cc' --- a/client/mysqltest.cc 2010-01-15 15:27:55 +0000 +++ b/client/mysqltest.cc 2010-01-28 14:49:14 +0000 @@ -657,7 +657,9 @@ public: } while ((bytes= fread(buf, 1, sizeof(buf), m_file)) > 0) - fwrite(buf, 1, bytes, stderr); + if (fwrite(buf, 1, bytes, stderr)) + die("Failed to write to '%s', errno: %d", + m_file_name, errno); if (!lines) { === modified file 'config/ac-macros/libevent.m4' --- a/config/ac-macros/libevent.m4 2009-09-29 23:36:15 +0000 +++ b/config/ac-macros/libevent.m4 2010-01-30 12:39:39 +0000 @@ -14,13 +14,6 @@ AC_DEFUN([MYSQL_USE_BUNDLED_LIBEVENT], [ libevent_libs="\$(top_builddir)/extra/libevent/libevent.a" libevent_includes="-I\$(top_srcdir)/extra/libevent" - libevent_test_option="--mysqld=--thread-handling=pool-of-threads" - AC_SUBST(libevent_libs) - AC_SUBST(libevent_includes) - AC_SUBST(libevent_test_option) - - AC_DEFINE([HAVE_LIBEVENT], [1], [If we want to use libevent and have connection pooling]) - AC_MSG_RESULT([using bundled libevent]) dnl Get the upstream file with the original libevent configure macros. dnl Use builtin include for this, to work around path problems in old versions of aclocal. @@ -39,17 +32,64 @@ AC_DEFUN([MYSQL_CHECK_LIBEVENT], [ AC_CONFIG_FILES(extra/libevent/Makefile) - AC_MSG_CHECKING(for libevent) + AC_MSG_NOTICE([checking what libevent library to use]) + AC_ARG_WITH([libevent], - [ --with-libevent use libevent and have connection pooling], - [with_libevent=$withval], - [with_libevent=no] - ) - - if test "$with_libevent" != "no"; then - MYSQL_USE_BUNDLED_LIBEVENT - else - AC_MSG_RESULT([disabled]) + AC_HELP_STRING([--with-libevent=yes|no|bundled|DIR], + [Use libevent and have connection pooling. + A location of libevent library can be specified. + Given DIR, libevent library is + assumed to be in $DIR/lib and header files + in $DIR/include.]), + [with_libevent=${withval}], + [with_libevent=no]) + + case "$with_libevent" in + "no") + with_libevent=disabled + ;; + "bundled") + MYSQL_USE_BUNDLED_LIBEVENT + ;; + "" | "yes") + libevent_includes="" + libevent_libs="-levent" + AC_CHECK_LIB(event, evutil_socketpair,[with_libevent=system], + [with_libevent=bundled]) + AC_CHECK_HEADER(evutil.h,,[with_libevent=bundled]) + if test "$with_libevent" = "bundled"; then + MYSQL_USE_BUNDLED_LIBEVENT + fi + ;; + *) + # Test for libevent using all known library file endings + if test $ -f "$with_libevent/lib/libevent.a" -o \ + -f "$with_libevent/lib/libevent.so" -o \ + -f "$with_libevent/lib/libevent.sl" -o \ + -f "$with_libevent/lib/libevent.dylib" $ \ + -a -f "$with_libevent/include/evutil.h"; then + libevent_includes="-I$with_libevent/include" + libevent_libs="-L$with_libevent/lib -levent" + AC_CHECK_LIB(event, evutil_socketpair,[with_libevent=$with_libevent], + [with_libevent=no], [$libevent_libs]) + else + with_libevent=no + fi + if test "$with_libevent" = "no"; then + AC_MSG_ERROR([libevent headers or binaries were not found]) + fi + ;; + esac + AC_MSG_CHECKING(for libevent) + AC_MSG_RESULT([$with_libevent]) + + if test "$with_libevent" != "disabled"; then + libevent_test_option="--mysqld=--thread-handling=pool-of-threads" + AC_SUBST(libevent_libs) + AC_SUBST(libevent_includes) + AC_SUBST(libevent_test_option) + AC_DEFINE([HAVE_LIBEVENT], [1], [If we want to use libevent and have connection pooling]) fi - AM_CONDITIONAL([HAVE_LIBEVENT], [ test "$with_libevent" != "no" ]) + AM_CONDITIONAL([HAVE_LIBEVENT], [ test "$with_libevent" != "disabled" ]) ]) + === modified file 'configure.in' --- a/configure.in 2010-01-17 17:22:46 +0000 +++ b/configure.in 2010-01-29 20:37:22 +0000 @@ -10,7 +10,7 @@ AC_CANONICAL_SYSTEM # # When changing major version number please also check switch statement # in mysqlbinlog::check_master_version(). -AM_INIT_AUTOMAKE(mysql, 5.1.42-MariaDB-rc) +AM_INIT_AUTOMAKE(mysql, 5.1.42-MariaDB) AM_CONFIG_HEADER([include/config.h:config.h.in]) # Request support for automake silent-rules if available. @@ -604,7 +604,7 @@ AC_CHECK_TOOL([NM], [nm]) if test "$TARGET_LINUX" = "true" -a "$static_nss" = "" then - tmp=`$NM ${other_libc_lib:-/usr/lib*}/libc.a | grep _nss_files_getaliasent_r1` + tmp=`$NM ${other_libc_lib:-/usr/lib*}/libc.a 2>&1 | grep _nss_files_getaliasent_r1` if test -n "$tmp" then STATIC_NSS_FLAGS="-lc -lnss_files -lnss_dns -lresolv" === modified file 'extra/libevent/devpoll.c' --- a/extra/libevent/devpoll.c 2010-01-07 13:00:06 +0000 +++ b/extra/libevent/devpoll.c 2010-01-28 11:35:10 +0000 @@ -140,7 +140,7 @@ devpoll_init(struct event_base *base) return (NULL); if (getrlimit(RLIMIT_NOFILE, &rl) == 0 && - rl.rlim_cur != RLIM_INFINITY) + (unsigned long long) rl.rlim_cur != (unsigned long long) RLIM_INFINITY) nfiles = rl.rlim_cur - 1; /* Initialize the kernel queue */ === modified file 'extra/yassl/src/yassl_error.cpp' --- a/extra/yassl/src/yassl_error.cpp 2008-11-18 16:45:44 +0000 +++ b/extra/yassl/src/yassl_error.cpp 2010-01-29 10:42:31 +0000 @@ -60,7 +60,7 @@ void SetErrorString(YasslError error, ch using namespace TaoCrypt; const int max = MAX_ERROR_SZ; // shorthand - switch (error) { + switch ((int) error) { // yaSSL proper errors case range_error : === modified file 'extra/yassl/taocrypt/include/asn.hpp' --- a/extra/yassl/taocrypt/include/asn.hpp 2007-01-29 15:54:40 +0000 +++ b/extra/yassl/taocrypt/include/asn.hpp 2010-01-27 10:38:29 +0000 @@ -305,6 +305,7 @@ private: bool ValidateSignature(SignerList*); bool ConfirmSignature(Source&); void GetKey(); + char* AddTag(char*, const char*, const char*, word32, word32); void GetName(NameType); void GetValidity(); void GetDate(DateType); === modified file 'extra/yassl/taocrypt/src/asn.cpp' --- a/extra/yassl/taocrypt/src/asn.cpp 2009-09-15 11:22:39 +0000 +++ b/extra/yassl/taocrypt/src/asn.cpp 2010-01-27 10:38:29 +0000 @@ -652,6 +652,25 @@ word32 CertDecoder::GetDigest() } +// memory length checked add tag to buffer +char* CertDecoder::AddTag(char* ptr, const char* buf_end, const char* tag_name, + word32 tag_name_length, word32 tag_value_length) +{ + if (ptr + tag_name_length + tag_value_length > buf_end) { + source_.SetError(CONTENT_E); + return 0; + } + + memcpy(ptr, tag_name, tag_name_length); + ptr += tag_name_length; + + memcpy(ptr, source_.get_current(), tag_value_length); + ptr += tag_value_length; + + return ptr; +} + + // process NAME, either issuer or subject void CertDecoder::GetName(NameType nt) { @@ -659,11 +678,22 @@ void CertDecoder::GetName(NameType nt) SHA sha; word32 length = GetSequence(); // length of all distinguished names - assert (length < ASN_NAME_MAX); + + if (length >= ASN_NAME_MAX) + return; length += source_.get_index(); - char* ptr = (nt == ISSUER) ? issuer_ : subject_; - word32 idx = 0; + char* ptr; + char* buf_end; + + if (nt == ISSUER) { + ptr = issuer_; + buf_end = ptr + sizeof(issuer_) - 1; // 1 byte for trailing 0 + } + else { + ptr = subject_; + buf_end = ptr + sizeof(subject_) - 1; // 1 byte for trailing 0 + } while (source_.get_index() < length) { GetSet(); @@ -685,47 +715,36 @@ void CertDecoder::GetName(NameType nt) byte id = source_.next(); b = source_.next(); // strType word32 strLen = GetLength(source_); - bool copy = false; - - if (id == COMMON_NAME) { - memcpy(&ptr[idx], "/CN=", 4); - idx += 4; - copy = true; - } - else if (id == SUR_NAME) { - memcpy(&ptr[idx], "/SN=", 4); - idx += 4; - copy = true; - } - else if (id == COUNTRY_NAME) { - memcpy(&ptr[idx], "/C=", 3); - idx += 3; - copy = true; - } - else if (id == LOCALITY_NAME) { - memcpy(&ptr[idx], "/L=", 3); - idx += 3; - copy = true; - } - else if (id == STATE_NAME) { - memcpy(&ptr[idx], "/ST=", 4); - idx += 4; - copy = true; - } - else if (id == ORG_NAME) { - memcpy(&ptr[idx], "/O=", 3); - idx += 3; - copy = true; - } - else if (id == ORGUNIT_NAME) { - memcpy(&ptr[idx], "/OU=", 4); - idx += 4; - copy = true; - } - if (copy) { - memcpy(&ptr[idx], source_.get_current(), strLen); - idx += strLen; + switch (id) { + case COMMON_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/CN=", 4, strLen))) + return; + break; + case SUR_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/SN=", 4, strLen))) + return; + break; + case COUNTRY_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/C=", 3, strLen))) + return; + break; + case LOCALITY_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/L=", 3, strLen))) + return; + break; + case STATE_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/ST=", 4, strLen))) + return; + break; + case ORG_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/O=", 3, strLen))) + return; + break; + case ORGUNIT_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/OU=", 4, strLen))) + return; + break; } sha.Update(source_.get_current(), strLen); @@ -740,17 +759,15 @@ void CertDecoder::GetName(NameType nt) word32 length = GetLength(source_); if (email) { - memcpy(&ptr[idx], "/emailAddress=", 14); - idx += 14; - - memcpy(&ptr[idx], source_.get_current(), length); - idx += length; + if (!(ptr = AddTag(ptr, buf_end, "/emailAddress=", 14, length))) + return; } source_.advance(length); } } - ptr[idx++] = 0; + + *ptr = 0; if (nt == ISSUER) sha.Final(issuerHash_); === modified file 'include/ft_global.h' --- a/include/ft_global.h 2009-11-30 13:36:06 +0000 +++ b/include/ft_global.h 2010-01-27 21:53:08 +0000 @@ -62,7 +62,8 @@ void ft_free_stopwords(void); #define FT_SORTED 2 #define FT_EXPAND 4 /* query expansion */ -FT_INFO *ft_init_search(uint,void *, uint, uchar *, uint,CHARSET_INFO *, uchar *); +FT_INFO *ft_init_search(uint,void *, uint, uchar *, size_t, + CHARSET_INFO *, uchar *); my_bool ft_boolean_check_syntax_string(const uchar *); /* Internal symbols for fulltext between maria and MyISAM */ === modified file 'include/maria.h' --- a/include/maria.h 2009-02-19 09:01:25 +0000 +++ b/include/maria.h 2010-01-27 21:53:08 +0000 @@ -462,7 +462,7 @@ void maria_versioning(MARIA_HA *info, my void maria_ignore_trids(MARIA_HA *info); /* fulltext functions */ -FT_INFO *maria_ft_init_search(uint,void *, uint, uchar *, uint, +FT_INFO *maria_ft_init_search(uint,void *, uint, uchar *, size_t, CHARSET_INFO *, uchar *); /* 'Almost-internal' Maria functions */ === modified file 'include/my_base.h' --- a/include/my_base.h 2009-09-07 20:50:10 +0000 +++ b/include/my_base.h 2010-02-10 19:06:24 +0000 @@ -111,7 +111,7 @@ enum ha_storage_media { enum ha_extra_function { HA_EXTRA_NORMAL=0, /* Optimize for space (def) */ HA_EXTRA_QUICK=1, /* Optimize for speed */ - HA_EXTRA_NOT_USED=2, + HA_EXTRA_NOT_USED=2, /* Should be ignored by handler */ HA_EXTRA_CACHE=3, /* Cache record in HA_rrnd() */ HA_EXTRA_NO_CACHE=4, /* End caching of records (def) */ HA_EXTRA_NO_READCHECK=5, /* No readcheck on update */ === modified file 'include/violite.h' --- a/include/violite.h 2010-01-15 15:27:55 +0000 +++ b/include/violite.h 2010-01-29 10:42:31 +0000 @@ -123,8 +123,8 @@ struct st_VioSSLFd SSL_CTX *ssl_context; }; -int sslaccept(struct st_VioSSLFd*, Vio *, long timeout); -int sslconnect(struct st_VioSSLFd*, Vio *, long timeout); +int sslaccept(struct st_VioSSLFd*, Vio *, long timeout, char *error_string); +int sslconnect(struct st_VioSSLFd*, Vio *, long timeout, char *error_string); struct st_VioSSLFd *new_VioSSLConnectorFd(const char *key_file, const char *cert_file, === modified file 'libmysqld/CMakeLists.txt' --- a/libmysqld/CMakeLists.txt 2009-12-03 11:19:05 +0000 +++ b/libmysqld/CMakeLists.txt 2010-01-31 09:13:21 +0000 @@ -153,7 +153,7 @@ FILE(WRITE cmake_dummy.c " ") # custom targets to "sql/CMakeLists.txt" and reference them here. ADD_LIBRARY(mysqlserver STATIC ${LIBMYSQLD_SOURCES}) ADD_DEPENDENCIES(mysqlserver GenServerSource GenError) -TARGET_LINK_LIBRARIES(mysqlserver) +TARGET_LINK_LIBRARIES(mysqlserver psapi.lib) # Add any additional libraries requested by engine(s) FOREACH (ENGINE_LIB ${MYSQLD_STATIC_ENGINE_LIBS}) === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2010-01-27 17:41:05 +0000 +++ b/mysql-test/mysql-test-run.pl 2010-02-10 19:06:24 +0000 @@ -413,7 +413,6 @@ sub main { # Not all tests completed, failure mtr_report(); mtr_report("Only ", int(@$completed), " of $num_tests completed."); - mtr_error("Not all tests completed"); } mtr_print_line(); @@ -425,6 +424,10 @@ sub main { mtr_report_stats($fail, $completed, $extra_warnings); + if ( @$completed != $num_tests) + { + mtr_error("Not all tests completed"); + } exit(0); } @@ -5539,6 +5542,8 @@ sub usage ($) { if ( $message ) { print STDERR "$message\n"; + print STDERR "For full list of options, use $0 --help\n"; + exit; } print <<HERE; === modified file 'mysql-test/r/grant.result' --- a/mysql-test/r/grant.result 2009-10-27 10:09:36 +0000 +++ b/mysql-test/r/grant.result 2010-01-29 10:42:31 +0000 @@ -29,15 +29,15 @@ show grants for mysqltest_1@localhost; Grants for mysqltest_1@localhost GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' GRANT SELECT ON `mysqltest`.* TO 'mysqltest_1'@'localhost' -grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "MySQL AB"; +grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "Monty Program Ab"; show grants for mysqltest_1@localhost; Grants for mysqltest_1@localhost -GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'MySQL AB' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' +GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'Monty Program Ab' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' GRANT SELECT ON `mysqltest`.* TO 'mysqltest_1'@'localhost' revoke all privileges on mysqltest.* from mysqltest_1@localhost; show grants for mysqltest_1@localhost; Grants for mysqltest_1@localhost -GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'MySQL AB' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' +GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'Monty Program Ab' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' delete from mysql.user where user='mysqltest_1'; flush privileges; delete from mysql.user where user='mysqltest_1'; === modified file 'mysql-test/r/openssl_1.result' --- a/mysql-test/r/openssl_1.result 2009-05-25 13:00:18 +0000 +++ b/mysql-test/r/openssl_1.result 2010-01-29 10:42:31 +0000 @@ -3,8 +3,10 @@ create table t1(f1 int); insert into t1 values (5); grant select on test.* to ssl_user1@localhost require SSL; grant select on test.* to ssl_user2@localhost require cipher "DHE-RSA-AES256-SHA"; -grant select on test.* to ssl_user3@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com"; -grant select on test.* to ssl_user4@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com" ISSUER "/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB"; +grant select on test.* to ssl_user3@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; +grant select on test.* to ssl_user4@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org" ISSUER "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; grant select on test.* to ssl_user5@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "xxx"; flush privileges; connect(localhost,ssl_user5,,test,MASTER_PORT,MASTER_SOCKET); @@ -44,9 +46,9 @@ ERROR 42000: DELETE command denied to us drop user ssl_user1@localhost, ssl_user2@localhost, ssl_user3@localhost, ssl_user4@localhost, ssl_user5@localhost; drop table t1; -mysqltest: Could not open connection 'default': 2026 SSL connection error -mysqltest: Could not open connection 'default': 2026 SSL connection error -mysqltest: Could not open connection 'default': 2026 SSL connection error +mysqltest: Could not open connection 'default': 2026 SSL error: ASN: bad other signature confirmation +mysqltest: Could not open connection 'default': 2026 SSL error: ASN: bad other signature confirmation +mysqltest: Could not open connection 'default': 2026 SSL error: ASN: bad other signature confirmation SSL error: Unable to get private key from '' mysqltest: Could not open connection 'default': 2026 SSL connection error SSL error: Unable to get certificate from '' === modified file 'mysql-test/r/show_check.result' --- a/mysql-test/r/show_check.result 2009-12-15 09:03:24 +0000 +++ b/mysql-test/r/show_check.result 2010-01-27 21:53:08 +0000 @@ -252,13 +252,11 @@ drop table t1; flush tables; show open tables; Database Table In_use Name_locked -mysql general_log 0 0 create table t1(n int); insert into t1 values (1); show open tables; Database Table In_use Name_locked test t1 0 0 -mysql general_log 0 0 drop table t1; create table t1 (a int not null, b VARCHAR(10), INDEX (b) ) AVG_ROW_LENGTH=10 CHECKSUM=1 COMMENT="test" ENGINE=MYISAM MIN_ROWS=10 MAX_ROWS=100 PACK_KEYS=1 DELAY_KEY_WRITE=1 ROW_FORMAT=fixed; show create table t1; === modified file 'mysql-test/r/sp-destruct.result' --- a/mysql-test/r/sp-destruct.result 2009-11-21 11:18:21 +0000 +++ b/mysql-test/r/sp-destruct.result 2010-02-10 19:06:24 +0000 @@ -1,4 +1,5 @@ call mtr.add_suppression("Column count of mysql.proc is wrong. Expected 20, found 19. The table is probably corrupted"); +flush table mysql.proc; use test; drop procedure if exists bug14233; drop function if exists bug14233; === modified file 'mysql-test/r/table_elim.result' --- a/mysql-test/r/table_elim.result 2009-09-16 19:05:03 +0000 +++ b/mysql-test/r/table_elim.result 2010-02-18 12:03:52 +0000 @@ -418,3 +418,49 @@ id select_type table type possible_keys 1 SIMPLE t1 ALL NULL NULL NULL NULL 4 1 SIMPLE t2 range PRIMARY PRIMARY 4 NULL 2 Using where drop table t1, t2; +# +# LPBUG#523593: Running RQG optimizer_no_subquery crashes MariaDB +# +CREATE TABLE t1 ( +`pk` int(11) NOT NULL AUTO_INCREMENT, +`col_int_nokey` int(11) DEFAULT NULL, +`col_int_key` int(11) DEFAULT NULL, +`col_date_key` date DEFAULT NULL, +`col_date_nokey` date DEFAULT NULL, +`col_time_key` time DEFAULT NULL, +`col_time_nokey` time DEFAULT NULL, +`col_datetime_key` datetime DEFAULT NULL, +`col_datetime_nokey` datetime DEFAULT NULL, +`col_varchar_key` varchar(1) DEFAULT NULL, +`col_varchar_nokey` varchar(1) DEFAULT NULL, +PRIMARY KEY (`pk`), +KEY `col_int_key` (`col_int_key`), +KEY `col_date_key` (`col_date_key`), +KEY `col_time_key` (`col_time_key`), +KEY `col_datetime_key` (`col_datetime_key`), +KEY `col_varchar_key` (`col_varchar_key`,`col_int_key`) +); +CREATE TABLE t2 LIKE t1; +INSERT INTO t1 VALUES +(10,7,8,NULL,NULL,'01:27:35','01:27:35','2002-02-26 06:14:37','2002-02-26 06:14:37','v','v'), +(11,1,9,'2006-06-14','2006-06-14','19:48:31','19:48:31','1900-01-01 00:00:00','1900-01-01 00:00:00','r','r'); +INSERT INTO t2 SELECT * FROM t1; +SELECT table2.col_int_key AS field1 +FROM ( +t2 AS table1 +RIGHT OUTER JOIN +( +( t1 AS table2 STRAIGHT_JOIN +t1 AS table3 ON ( +(table3.col_varchar_nokey = table2.col_varchar_key ) AND +(table3.pk = table2.col_int_key)) +) +) ON +( +(table3.col_varchar_key = table2.col_varchar_key) OR +(table3.col_int_key = table2.pk) +) +) +HAVING field1 < 216; +field1 +DROP TABLE t1, t2; === modified file 'mysql-test/r/udf.result' --- a/mysql-test/r/udf.result 2010-01-11 13:15:28 +0000 +++ b/mysql-test/r/udf.result 2010-01-22 21:19:21 +0000 @@ -38,8 +38,6 @@ ERROR HY000: Can't initialize function ' select reverse_lookup("127.0.0.1"); select reverse_lookup(127,0,0,1); select reverse_lookup("localhost"); -reverse_lookup("localhost") -NULL select avgcost(); ERROR HY000: Can't initialize function 'avgcost'; wrong number of arguments: AVGCOST() requires two arguments select avgcost(100,23.76); === modified file 'mysql-test/r/variables.result' --- a/mysql-test/r/variables.result 2010-01-11 13:15:28 +0000 +++ b/mysql-test/r/variables.result 2010-02-10 19:06:24 +0000 @@ -1261,12 +1261,12 @@ ERROR HY000: Variable 'lower_case_table_ # SHOW VARIABLES like 'myisam_recover_options'; Variable_name Value -myisam_recover_options OFF +myisam_recover_options DEFAULT SELECT @@session.myisam_recover_options; ERROR HY000: Variable 'myisam_recover_options' is a GLOBAL variable SELECT @@global.myisam_recover_options; @@global.myisam_recover_options -OFF +DEFAULT SET @@session.myisam_recover_options= 'x'; ERROR HY000: Variable 'myisam_recover_options' is a read only variable SET @@global.myisam_recover_options= 'x'; === modified file 'mysql-test/r/view.result' --- a/mysql-test/r/view.result 2009-10-15 21:38:29 +0000 +++ b/mysql-test/r/view.result 2010-02-10 19:06:24 +0000 @@ -155,13 +155,13 @@ v5 VIEW v6 VIEW show table status; Name Engine Version Row_format Rows Avg_row_length Data_length Max_data_length Index_length Data_free Auto_increment Create_time Update_time Check_time Collation Checksum Create_options Comment -t1 MyISAM 10 Fixed 5 9 45 # 1024 0 NULL # # NULL latin1_swedish_ci NULL -v1 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # NULL NULL NULL NULL VIEW -v2 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # NULL NULL NULL NULL VIEW -v3 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # NULL NULL NULL NULL VIEW -v4 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # NULL NULL NULL NULL VIEW -v5 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # NULL NULL NULL NULL VIEW -v6 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # NULL NULL NULL NULL VIEW +t1 MyISAM 10 Fixed 5 9 45 # 1024 0 NULL # # # latin1_swedish_ci NULL +v1 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # # NULL NULL NULL VIEW +v2 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # # NULL NULL NULL VIEW +v3 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # # NULL NULL NULL VIEW +v4 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # # NULL NULL NULL VIEW +v5 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # # NULL NULL NULL VIEW +v6 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # # NULL NULL NULL VIEW drop view v1,v2,v3,v4,v5,v6; create view v1 (c,d,e,f) as select a,b, a in (select a+2 from t1), a = all (select a from t1) from t1; === modified file 'mysql-test/std_data/cacert.pem' --- a/mysql-test/std_data/cacert.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/cacert.pem 2010-01-29 10:42:31 +0000 @@ -1,17 +1,19 @@ -----BEGIN CERTIFICATE----- -MIICrTCCAhagAwIBAgIJAJXpePU0UOTVMA0GCSqGSIb3DQEBBQUAMEQxCzAJBgNV -BAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxhMRAwDgYDVQQHEwdVcHBzYWxhMREwDwYD -VQQKEwhNeVNRTCBBQjAeFw0wOTAxMjgxMDQ5NDZaFw0xNDAxMjcxMDQ5NDZaMEQx -CzAJBgNVBAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxhMRAwDgYDVQQHEwdVcHBzYWxh -MREwDwYDVQQKEwhNeVNRTCBBQjCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA -4XQHAe5R1+TXC8noZtWf+d5E0v1C59FWpn9SWEUCBjE5UiIwuJvi4Y+7xWGOXLAI -/JzJx5gNXLBiTsE/zh0uX9fKlajLhxB0GN+QU0ZlpQ1BeYipEcNXeI/7cT499f6v -XWabnTflivdCgHSWUOQ20/Lzs6kP6/e6OoZd/DPSjPECAwEAAaOBpjCBozAdBgNV -HQ4EFgQU8uLqVWWkmuKsnZf1RWz294wRrd8wdAYDVR0jBG0wa4AU8uLqVWWkmuKs -nZf1RWz294wRrd+hSKRGMEQxCzAJBgNVBAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxh -MRAwDgYDVQQHEwdVcHBzYWxhMREwDwYDVQQKEwhNeVNRTCBBQoIJAJXpePU0UOTV -MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADgYEAMMTE5sDN+Z0ZlV7KvH3g -6+aKvql8dTpRT3hYukeQlWua0nq74WPGVw0c4e/M/vbiMwmJcCYpB9pd4+dHqzSw -aPyoenjY6UF8n7B4quWy3SIUk2LSHeJLW+kzJn2afN9gvipFhdVh/uU2TIyLGOur -Z/vmJX2W7hF1uqPnbfa8Lrw= +MIIDIjCCAougAwIBAgIJAJhuvLP+2mGwMA0GCSqGSIb3DQEBBQUAMGoxCzAJBgNV +BAYTAkZJMRAwDgYDVQQIEwdUdXVzdWxhMRkwFwYDVQQKExBNb250eSBQcm9ncmFt +IEFiMS4wLAYJKoZIhvcNAQkBFh9hYnN0cmFjdC5kZXZlbG9wZXJAYXNrbW9udHku +b3JnMB4XDTEwMDEyODIxNTcyNVoXDTEwMDIyNzIxNTcyNVowajELMAkGA1UEBhMC +RkkxEDAOBgNVBAgTB1R1dXN1bGExGTAXBgNVBAoTEE1vbnR5IFByb2dyYW0gQWIx +LjAsBgkqhkiG9w0BCQEWH2Fic3RyYWN0LmRldmVsb3BlckBhc2ttb250eS5vcmcw +gZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMIaY4pwbst50S32xJH3bSXhPGep +6gx1AWwZKsHTXL3VeMO6PHmC8zu5HM0zbOcrIJcXL3YVnpmE4b9OQxIiMSx1Yd+U +u8/sTkxgpsEKhCbIzECIwPhppyT/JP5aSXCadEvg+PSjikv8dOVkD68wVG4CcFIX +MFttsPebBVzEokZZAgMBAAGjgc8wgcwwHQYDVR0OBBYEFOCKaNHFFPrju8AwzWxS +f96IKfRwMIGcBgNVHSMEgZQwgZGAFOCKaNHFFPrju8AwzWxSf96IKfRwoW6kbDBq +MQswCQYDVQQGEwJGSTEQMA4GA1UECBMHVHV1c3VsYTEZMBcGA1UEChMQTW9udHkg +UHJvZ3JhbSBBYjEuMCwGCSqGSIb3DQEJARYfYWJzdHJhY3QuZGV2ZWxvcGVyQGFz +a21vbnR5Lm9yZ4IJAJhuvLP+2mGwMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEF +BQADgYEAsmCX2/k9AInq2qhXtnkLip6cB0iOerLTNAzEijZc/aVf4wUjkL3cqhmC +kSTCwAHIOxp+ICwh6ky3xghXjoI9QnPFDVkRkzPT2tV0IoBaeQuI4e0CU2EY7L3P +XoDqp3oq1XtVcr9ZZdP68fBYUG/qcrWcXWk45ZFaBmBv3TotsGk= -----END CERTIFICATE----- === modified file 'mysql-test/std_data/client-cert.pem' --- a/mysql-test/std_data/client-cert.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/client-cert.pem 2010-01-29 10:42:31 +0000 @@ -1,55 +1,60 @@ Certificate: Data: Version: 3 (0x2) - Serial Number: 3 (0x3) + Serial Number: 2 (0x2) Signature Algorithm: sha1WithRSAEncryption - Issuer: C=SE, ST=Uppsala, L=Uppsala, O=MySQL AB + Issuer: C=FI, ST=Tuusula, O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org Validity - Not Before: Jan 28 11:04:39 2009 GMT - Not After : Jan 28 11:04:39 2010 GMT - Subject: C=SE, ST=Uppsala, O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com + Not Before: Jan 28 22:01:38 2010 GMT + Not After : Dec 7 22:01:38 2019 GMT + Subject: C=FI, ST=Tuusula, O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org Subject Public Key Info: Public Key Algorithm: rsaEncryption - RSA Public Key: (512 bit) - Modulus (512 bit): - 00:e1:52:30:2c:d9:be:64:28:91:5d:7a:fd:d9:e9: - 14:35:7a:d2:94:4e:91:46:e0:db:9f:6b:79:f4:4c: - ac:6e:07:61:34:86:74:62:a7:a8:44:af:fa:87:87: - a8:7d:42:61:ff:ab:50:d4:7b:bf:75:fa:d5:d5:b3: - 74:fb:56:1e:37 + RSA Public Key: (1024 bit) + Modulus (1024 bit): + 00:bd:dc:3d:f8:3c:0b:d4:d2:c0:a3:9c:34:2d:e7: + 11:ff:4d:43:35:17:f6:0d:91:01:92:9e:4f:4d:c0: + 38:d5:62:03:55:33:db:66:a2:91:d3:f2:b6:23:34: + 95:53:50:3d:4f:e3:0c:d7:76:fd:f6:54:64:e6:f6: + dc:70:74:7c:6b:74:41:59:b0:19:5d:62:90:3c:a7: + c8:5e:21:8f:2b:22:6b:c7:43:9b:be:79:84:60:da: + 16:c9:ce:ee:fd:66:cb:54:81:e2:b5:1c:cf:f9:74: + de:38:2b:28:d4:31:33:55:d2:30:1c:a3:e4:c2:c7: + 31:46:43:d5:33:3d:8a:0b:47 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE + Netscape Comment: + OpenSSL Generated Certificate X509v3 Subject Key Identifier: - 58:30:B5:9B:2C:05:94:06:BA:3D:3C:F0:B2:CD:1D:67:65:E3:7F:85 + BE:E6:DB:19:8D:DB:72:9A:85:EE:B2:B8:5D:E7:FF:61:DF:09:08:AF X509v3 Authority Key Identifier: - keyid:F2:E2:EA:55:65:A4:9A:E2:AC:9D:97:F5:45:6C:F6:F7:8C:11:AD:DF - DirName:/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB - serial:95:E9:78:F5:34:50:E4:D5 + keyid:E0:8A:68:D1:C5:14:FA:E3:BB:C0:30:CD:6C:52:7F:DE:88:29:F4:70 Signature Algorithm: sha1WithRSAEncryption - 05:19:e3:13:14:fc:c5:28:bf:69:f8:00:b3:25:cb:bd:ca:9f: - 2f:4c:b3:a8:04:11:f0:74:27:bd:82:2c:b4:49:9b:a7:59:f0: - f7:87:d1:e0:ba:99:a2:fe:4b:1d:10:6f:e4:a2:b3:cd:7f:8b: - 68:31:46:ee:cd:9e:e2:47:e1:4c:fa:74:d1:e2:8b:cc:a0:4b: - a8:24:d1:a4:c3:6b:2a:c6:28:cd:41:e0:06:48:e6:cf:f2:3c: - ca:37:95:d7:29:64:6b:91:91:83:e7:ac:c8:0b:87:bc:da:a6: - aa:f1:44:43:c8:74:7b:15:26:91:2e:03:c4:71:50:6c:f8:68: - dc:8c + 41:95:6d:0a:a4:ee:af:68:cd:94:26:59:9a:18:b7:75:3c:c5: + 0f:22:d3:5c:31:9b:85:a0:93:b3:f0:50:29:ba:1e:d3:5a:43: + 0b:77:2d:98:87:a7:a7:39:0f:40:8d:03:d3:b3:67:43:77:bc: + 3c:51:c2:f9:9e:7a:2d:39:c4:5c:16:d7:70:d6:74:d1:6c:e1: + 6a:4d:fd:1f:10:af:64:3b:f4:64:e9:b2:b3:fb:c8:cd:c5:41: + cd:99:e0:ac:83:1d:81:2c:6b:99:ba:80:02:12:72:f7:3b:bb: + 93:72:00:da:ff:d3:87:75:d2:3a:a4:ca:4d:c1:8b:c1:21:50: + cb:57 -----BEGIN CERTIFICATE----- -MIICfzCCAeigAwIBAgIBAzANBgkqhkiG9w0BAQUFADBEMQswCQYDVQQGEwJTRTEQ -MA4GA1UECBMHVXBwc2FsYTEQMA4GA1UEBxMHVXBwc2FsYTERMA8GA1UEChMITXlT -UUwgQUIwHhcNMDkwMTI4MTEwNDM5WhcNMTAwMTI4MTEwNDM5WjBlMQswCQYDVQQG -EwJTRTEQMA4GA1UECBMHVXBwc2FsYTERMA8GA1UEChMITXlTUUwgQUIxMTAvBgkq -hkiG9w0BCQEWImFic3RyYWN0Lm15c3FsLmRldmVsb3BlckBteXNxbC5jb20wXDAN -BgkqhkiG9w0BAQEFAANLADBIAkEA4VIwLNm+ZCiRXXr92ekUNXrSlE6RRuDbn2t5 -9EysbgdhNIZ0YqeoRK/6h4eofUJh/6tQ1Hu/dfrV1bN0+1YeNwIDAQABo4GjMIGg -MAkGA1UdEwQCMAAwHQYDVR0OBBYEFFgwtZssBZQGuj088LLNHWdl43+FMHQGA1Ud -IwRtMGuAFPLi6lVlpJrirJ2X9UVs9veMEa3foUikRjBEMQswCQYDVQQGEwJTRTEQ -MA4GA1UECBMHVXBwc2FsYTEQMA4GA1UEBxMHVXBwc2FsYTERMA8GA1UEChMITXlT -UUwgQUKCCQCV6Xj1NFDk1TANBgkqhkiG9w0BAQUFAAOBgQAFGeMTFPzFKL9p+ACz -Jcu9yp8vTLOoBBHwdCe9giy0SZunWfD3h9Hgupmi/ksdEG/korPNf4toMUbuzZ7i -R+FM+nTR4ovMoEuoJNGkw2sqxijNQeAGSObP8jzKN5XXKWRrkZGD56zIC4e82qaq -8URDyHR7FSaRLgPEcVBs+GjcjA== +MIICxTCCAi6gAwIBAgIBAjANBgkqhkiG9w0BAQUFADBqMQswCQYDVQQGEwJGSTEQ +MA4GA1UECBMHVHV1c3VsYTEZMBcGA1UEChMQTW9udHkgUHJvZ3JhbSBBYjEuMCwG +CSqGSIb3DQEJARYfYWJzdHJhY3QuZGV2ZWxvcGVyQGFza21vbnR5Lm9yZzAeFw0x +MDAxMjgyMjAxMzhaFw0xOTEyMDcyMjAxMzhaMGoxCzAJBgNVBAYTAkZJMRAwDgYD +VQQIEwdUdXVzdWxhMRkwFwYDVQQKExBNb250eSBQcm9ncmFtIEFiMS4wLAYJKoZI +hvcNAQkBFh9hYnN0cmFjdC5kZXZlbG9wZXJAYXNrbW9udHkub3JnMIGfMA0GCSqG +SIb3DQEBAQUAA4GNADCBiQKBgQC93D34PAvU0sCjnDQt5xH/TUM1F/YNkQGSnk9N +wDjVYgNVM9tmopHT8rYjNJVTUD1P4wzXdv32VGTm9txwdHxrdEFZsBldYpA8p8he +IY8rImvHQ5u+eYRg2hbJzu79ZstUgeK1HM/5dN44KyjUMTNV0jAco+TCxzFGQ9Uz +PYoLRwIDAQABo3sweTAJBgNVHRMEAjAAMCwGCWCGSAGG+EIBDQQfFh1PcGVuU1NM +IEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUvubbGY3bcpqF7rK4Xef/ +Yd8JCK8wHwYDVR0jBBgwFoAU4Ipo0cUU+uO7wDDNbFJ/3ogp9HAwDQYJKoZIhvcN +AQEFBQADgYEAQZVtCqTur2jNlCZZmhi3dTzFDyLTXDGbhaCTs/BQKboe01pDC3ct +mIenpzkPQI0D07NnQ3e8PFHC+Z56LTnEXBbXcNZ00Wzhak39HxCvZDv0ZOmys/vI +zcVBzZngrIMdgSxrmbqAAhJy9zu7k3IA2v/Th3XSOqTKTcGLwSFQy1c= -----END CERTIFICATE----- === modified file 'mysql-test/std_data/client-key.pem' --- a/mysql-test/std_data/client-key.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/client-key.pem 2010-01-29 10:42:31 +0000 @@ -1,9 +1,15 @@ -----BEGIN RSA PRIVATE KEY----- -MIIBOQIBAAJBAOFSMCzZvmQokV16/dnpFDV60pROkUbg259refRMrG4HYTSGdGKn -qESv+oeHqH1CYf+rUNR7v3X61dWzdPtWHjcCAwEAAQJAXYooM8ZlcuEgj+VKU1ee -qyEFIMqJJxqcMk+E/nWCM96WxCP3zHNSrqNfSpI3ld7QzMwhdRz+gFLxT2gGNpIw -MQIhAPxzM/lDihe67X3ADYtDl9ZjA8Pm430x9sXlcxI17tCZAiEA5H1SyFl4mUee -9VnfSC2XGW7lwz72ZygfVX+b7tLWF08CIEh40gzW5MfXM+KLxdea+fXjyursV5ZT -R6KcMiKiNQLRAiAcmHqlzFzFgisotai2Fc6VRkXHG7gmzOSvBJt1VjmpDQIge6jf -2N7whTdvC4ferB+zUlgWQdyvx1c3T4gnt6PYdaY= +MIICXgIBAAKBgQC93D34PAvU0sCjnDQt5xH/TUM1F/YNkQGSnk9NwDjVYgNVM9tm +opHT8rYjNJVTUD1P4wzXdv32VGTm9txwdHxrdEFZsBldYpA8p8heIY8rImvHQ5u+ +eYRg2hbJzu79ZstUgeK1HM/5dN44KyjUMTNV0jAco+TCxzFGQ9UzPYoLRwIDAQAB +AoGBAJa2lprPT7UJ99Ho1aL6ota/RnKHKtNqII17DgjyZis9OtgP6kJ3GrvdF6iq +vT79my4nVrJTyxYXuGF/5U1/qqNjuPPBE1Xbu1ubQlFv8CT0kKYynQ7Z3ls8fAHC +B3VJXnUVlG+GHtUEFFG4FQVX1fn/Sga67ioJ6ivAiBlHKaPBAkEA5f2ToWlj4u9O +KgfRkN54wdIp4yu2c40pbhMfKGjGGsBAHk92+qSBpzEmxLcI6Ay+4/QysSR4jYmK +jCJuxiTu1QJBANNU3Hx8Il2SF/2BqGLcIh2SHxzKQIT5wAyD2jb+P2cHvbk6pKGR +VTmw5bibxXmYMS6J/L2zUF2xtFe+Svwz96sCQEnKYSqBqOWvyBFeLtPfPTlal8vm +Q4SxfuBtTCrn6t+8XRYcgt0KGPsunvSwkS/6nuh+eiExxWgMACLUDVyPjv0CQQC4 +sJJc7LOv6Oy0bWr2swHRrBEqvQsz63zOszCzHPHWHirNxZV5aiT8XT/2XZRwlvRs +gsVyGFLk/1fn0vN/g/8vAkEAxUdzUKvC1ZwjzGhgcz2bQU0tEZN4C9jBCiwOI2ud +BpAsPG0xAGGL2+hz0B0n88XiTHobiTZ1bg4Z41i4pXx2ZA== -----END RSA PRIVATE KEY----- === modified file 'mysql-test/std_data/server-cert.pem' --- a/mysql-test/std_data/server-cert.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/server-cert.pem 2010-01-29 10:42:31 +0000 @@ -3,53 +3,59 @@ Certificate: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha1WithRSAEncryption - Issuer: C=SE, ST=Uppsala, L=Uppsala, O=MySQL AB + Issuer: C=FI, ST=Tuusula, O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org Validity - Not Before: Jan 28 10:55:13 2009 GMT - Not After : Jan 28 10:55:13 2010 GMT - Subject: C=SE, ST=Uppsala, O=MySQL AB, CN=localhost/emailAddress=abstract.mysql.developer(a)mysql.com + Not Before: Jan 28 21:59:14 2010 GMT + Not After : Dec 7 21:59:14 2019 GMT + Subject: C=FI, ST=Tuusula, O=Monty Program Ab, CN=localhost/emailAddress=abstract.developer(a)askmonty.org Subject Public Key Info: Public Key Algorithm: rsaEncryption - RSA Public Key: (512 bit) - Modulus (512 bit): - 00:b6:8f:e5:b7:b4:86:83:13:8a:f9:bf:63:cb:64: - 2d:b9:51:d1:de:ab:7b:45:1f:aa:b5:66:73:13:f9: - a6:07:d5:ba:7c:fa:92:bd:37:e2:ad:87:db:3e:b6: - 6a:12:64:f8:ee:17:e3:15:06:2f:a8:82:68:bf:57: - 8d:c3:04:98:27 + RSA Public Key: (1024 bit) + Modulus (1024 bit): + 00:cc:79:74:2a:14:7e:77:06:b0:ec:1b:b6:da:70: + 4c:4f:0e:94:04:8a:e7:69:f4:4c:9c:67:22:41:56: + 3a:96:53:9e:95:9d:17:55:80:41:c0:13:d6:94:0f: + cd:2c:51:fe:a4:6e:f2:74:d8:5d:3b:3a:80:e3:85: + 5b:a5:bc:7d:5c:25:55:e5:40:77:fe:f3:cb:5b:cd: + da:a5:f4:36:64:30:a2:a0:7f:93:b3:c4:56:75:2a: + c0:f7:65:2a:d7:e6:ff:88:25:03:e0:b4:40:2e:74: + 4c:cb:46:95:73:cb:25:5c:87:0e:ec:0f:5a:19:c2: + b6:dc:9e:e8:f0:30:b1:9c:99 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE + Netscape Comment: + OpenSSL Generated Certificate X509v3 Subject Key Identifier: - D9:9A:B8:5F:22:EA:04:10:C8:25:7D:82:57:E6:2E:FD:19:29:E7:DA + 6D:13:3B:40:52:3C:AF:18:EA:33:D1:B7:56:21:1B:05:FE:0B:9E:38 X509v3 Authority Key Identifier: - keyid:F2:E2:EA:55:65:A4:9A:E2:AC:9D:97:F5:45:6C:F6:F7:8C:11:AD:DF - DirName:/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB - serial:95:E9:78:F5:34:50:E4:D5 + keyid:E0:8A:68:D1:C5:14:FA:E3:BB:C0:30:CD:6C:52:7F:DE:88:29:F4:70 Signature Algorithm: sha1WithRSAEncryption - 54:07:2d:21:0b:a5:af:3b:58:23:32:5e:56:7f:ab:58:63:48: - 91:aa:38:90:89:16:f9:cc:bf:a4:0e:78:2b:9f:c5:1b:58:a6: - e6:08:8f:2e:ae:97:03:21:9b:f1:cd:c0:26:8f:1d:d7:28:27: - a0:8e:81:09:1b:1c:0f:c9:a5:41:3a:2d:44:3f:9c:fa:87:ff: - c8:4c:2b:44:f7:1b:c1:3e:4f:01:7f:e9:26:cc:9f:1c:06:b5: - 0b:27:d1:10:90:be:93:0c:9c:e7:b0:d1:ea:27:99:4e:06:14: - 0c:7a:e9:c1:52:c5:33:68:bc:61:0d:db:81:3b:57:48:57:bf: - 42:9a + 97:db:65:23:7f:f1:15:3c:1e:83:ac:0e:0a:50:a0:0c:22:b8: + 45:d4:ca:21:05:47:3b:3d:03:b5:6c:4b:8d:bb:5f:57:c3:c7: + 4e:71:23:cf:33:a3:7f:a0:3d:bd:58:75:b8:37:22:16:2f:e9: + ed:ae:9b:94:29:81:6e:34:79:cf:41:bd:3d:8d:17:d7:22:1c: + 1b:58:c7:0f:79:13:56:1d:e8:d8:4e:e5:07:3f:79:1b:dd:c4: + 06:9b:c5:b6:02:34:43:c5:bf:e5:87:ad:f1:c1:8a:f2:be:c2: + 00:1d:d4:27:1f:87:c8:80:31:ec:6e:97:95:b4:84:40:d1:73: + 42:71 -----BEGIN CERTIFICATE----- -MIICkzCCAfygAwIBAgIBATANBgkqhkiG9w0BAQUFADBEMQswCQYDVQQGEwJTRTEQ -MA4GA1UECBMHVXBwc2FsYTEQMA4GA1UEBxMHVXBwc2FsYTERMA8GA1UEChMITXlT -UUwgQUIwHhcNMDkwMTI4MTA1NTEzWhcNMTAwMTI4MTA1NTEzWjB5MQswCQYDVQQG -EwJTRTEQMA4GA1UECBMHVXBwc2FsYTERMA8GA1UEChMITXlTUUwgQUIxEjAQBgNV -BAMTCWxvY2FsaG9zdDExMC8GCSqGSIb3DQEJARYiYWJzdHJhY3QubXlzcWwuZGV2 -ZWxvcGVyQG15c3FsLmNvbTBcMA0GCSqGSIb3DQEBAQUAA0sAMEgCQQC2j+W3tIaD -E4r5v2PLZC25UdHeq3tFH6q1ZnMT+aYH1bp8+pK9N+Kth9s+tmoSZPjuF+MVBi+o -gmi/V43DBJgnAgMBAAGjgaMwgaAwCQYDVR0TBAIwADAdBgNVHQ4EFgQU2Zq4XyLq -BBDIJX2CV+Yu/Rkp59owdAYDVR0jBG0wa4AU8uLqVWWkmuKsnZf1RWz294wRrd+h -SKRGMEQxCzAJBgNVBAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxhMRAwDgYDVQQHEwdV -cHBzYWxhMREwDwYDVQQKEwhNeVNRTCBBQoIJAJXpePU0UOTVMA0GCSqGSIb3DQEB -BQUAA4GBAFQHLSELpa87WCMyXlZ/q1hjSJGqOJCJFvnMv6QOeCufxRtYpuYIjy6u -lwMhm/HNwCaPHdcoJ6COgQkbHA/JpUE6LUQ/nPqH/8hMK0T3G8E+TwF/6SbMnxwG -tQsn0RCQvpMMnOew0eonmU4GFAx66cFSxTNovGEN24E7V0hXv0Ka +MIIC2TCCAkKgAwIBAgIBATANBgkqhkiG9w0BAQUFADBqMQswCQYDVQQGEwJGSTEQ +MA4GA1UECBMHVHV1c3VsYTEZMBcGA1UEChMQTW9udHkgUHJvZ3JhbSBBYjEuMCwG +CSqGSIb3DQEJARYfYWJzdHJhY3QuZGV2ZWxvcGVyQGFza21vbnR5Lm9yZzAeFw0x +MDAxMjgyMTU5MTRaFw0xOTEyMDcyMTU5MTRaMH4xCzAJBgNVBAYTAkZJMRAwDgYD +VQQIEwdUdXVzdWxhMRkwFwYDVQQKExBNb250eSBQcm9ncmFtIEFiMRIwEAYDVQQD +Ewlsb2NhbGhvc3QxLjAsBgkqhkiG9w0BCQEWH2Fic3RyYWN0LmRldmVsb3BlckBh +c2ttb250eS5vcmcwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMx5dCoUfncG +sOwbttpwTE8OlASK52n0TJxnIkFWOpZTnpWdF1WAQcAT1pQPzSxR/qRu8nTYXTs6 +gOOFW6W8fVwlVeVAd/7zy1vN2qX0NmQwoqB/k7PEVnUqwPdlKtfm/4glA+C0QC50 +TMtGlXPLJVyHDuwPWhnCttye6PAwsZyZAgMBAAGjezB5MAkGA1UdEwQCMAAwLAYJ +YIZIAYb4QgENBB8WHU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1Ud +DgQWBBRtEztAUjyvGOoz0bdWIRsF/gueODAfBgNVHSMEGDAWgBTgimjRxRT647vA +MM1sUn/eiCn0cDANBgkqhkiG9w0BAQUFAAOBgQCX22Ujf/EVPB6DrA4KUKAMIrhF +1MohBUc7PQO1bEuNu19Xw8dOcSPPM6N/oD29WHW4NyIWL+ntrpuUKYFuNHnPQb09 +jRfXIhwbWMcPeRNWHejYTuUHP3kb3cQGm8W2AjRDxb/lh63xwYryvsIAHdQnH4fI +gDHsbpeVtIRA0XNCcQ== -----END CERTIFICATE----- === modified file 'mysql-test/std_data/server-key.pem' --- a/mysql-test/std_data/server-key.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/server-key.pem 2010-01-29 10:42:31 +0000 @@ -1,9 +1,15 @@ -----BEGIN RSA PRIVATE KEY----- -MIIBOQIBAAJBALaP5be0hoMTivm/Y8tkLblR0d6re0UfqrVmcxP5pgfVunz6kr03 -4q2H2z62ahJk+O4X4xUGL6iCaL9XjcMEmCcCAwEAAQJASA5VwgNb0CKHiPm0ntOk -hG+54SRX3DmafEy6gRjZIl/bZ/asSLhXUZ+CeohyrQh7BZgYWvykd8pRISL9eKsU -GQIhAOXkUrOtP/EtjyqNluEqZdG+RZi/7p61JS3Ce13Myu+LAiEAy0uMlV34AJpM -b40FPKqlHxw8DD/Dt1iKhNVAg8+LDVUCIFjv7fbJDbW2VG63/Cj8CAwOukoP5rbL -iaicVrHBKrllAiB9+MiaXeopZXNrxDS0jQFYr8Q9yt1aJVFgUkxx4Q9HKQIgZPs0 -KlF3NNNWw78INaAEkyf0IEssnLMsuoCWw0DIOak= +MIICXgIBAAKBgQDMeXQqFH53BrDsG7bacExPDpQEiudp9EycZyJBVjqWU56VnRdV +gEHAE9aUD80sUf6kbvJ02F07OoDjhVulvH1cJVXlQHf+88tbzdql9DZkMKKgf5Oz +xFZ1KsD3ZSrX5v+IJQPgtEAudEzLRpVzyyVchw7sD1oZwrbcnujwMLGcmQIDAQAB +AoGBAMdMYkNZsmJFbVDVOobzCg3Mgc1jrmeBrOKNS8AvUe+QFXRyp3m5B102eOHb +/PmD+hU/5qao9UZzoYkiRM/oRq45jrqJEYwWrX007bKK0F9hnErtC1ImM1nBFVhx +6+6cr+ShUkvtj8+wJ2d5bIccUzGCUfFR5tb5BnePTXK8IVoBAkEA7WGNxHAVKgjS +AzlpHr5fvpivA07hNVJizTwZdWGGYeETilZhkkuMRwREceeohF6ILMf0FTZdFSa/ +8EeLa3icIQJBANyDKFjynKwWy5pyRSz75mVwrEi+4eTQPsCPNWLkbpbEPwqPLYWJ +2VSFkISXF7b7Od48JkQWgiB8/kXqMDEdsXkCQQCzZvj3ryWvoP7nhOoXXBWMPGR4 +gZLe86bMKVGsTsp7CtnzwRj4sbQQr/7yfvvzHmaYQX4M0gtDQwfolomd7YdBAkEA +y24ETuqjNu9grf81aiaJipPDnOjcJOcovSRgr/blPxmUvv0Pld5yLNN7W5a4PgrO +fAMpmi7ZpXcqbP17sBQgoQJAWTDFKAmfHPVdDGZuCw4yceP5d+Tv7ABglZUvpPKx +kAvGN1WBASUuCQJDOIgzl6gvYX07S5p147i9mv7UBWOpvw== -----END RSA PRIVATE KEY----- === modified file 'mysql-test/suite/maria/r/maria3.result' --- a/mysql-test/suite/maria/r/maria3.result 2009-09-18 01:04:43 +0000 +++ b/mysql-test/suite/maria/r/maria3.result 2010-02-12 14:21:13 +0000 @@ -301,7 +301,7 @@ check table t1 extended; Table Op Msg_type Msg_text test.t1 check status OK drop table t1; -show variables like 'maria%'; +select lower(variable_name) as Variable_name, Variable_value as Value from information_schema.session_variables where variable_name like "maria%" and variable_name not like "maria_used_for_temp_tables" order by 1; Variable_name Value maria_block_size 8192 maria_checkpoint_interval 30 @@ -309,16 +309,15 @@ maria_force_start_after_recovery_failure maria_log_file_size 4294959104 maria_log_purge_type immediate maria_max_sort_file_size 9223372036853727232 -maria_page_checksum OFF maria_pagecache_age_threshold 300 maria_pagecache_buffer_size 8384512 maria_pagecache_division_limit 100 +maria_page_checksum OFF maria_recover OFF maria_repair_threads 1 maria_sort_buffer_size 8388608 maria_stats_method nulls_unequal maria_sync_log_dir NEWFILE -maria_used_for_temp_tables ON show status like 'maria%'; Variable_name Value Maria_pagecache_blocks_not_flushed # === modified file 'mysql-test/suite/maria/t/maria-recovery2-master.opt' --- a/mysql-test/suite/maria/t/maria-recovery2-master.opt 2009-01-15 14:29:14 +0000 +++ b/mysql-test/suite/maria/t/maria-recovery2-master.opt 2010-02-10 19:06:24 +0000 @@ -1 +1 @@ ---skip-stack-trace --skip-core-file --loose-maria-log-dir-path=$MYSQLTEST_VARDIR/tmp +--skip-stack-trace --skip-core-file --loose-maria-log-dir-path=$MYSQLTEST_VARDIR/tmp --myisam-recover= === modified file 'mysql-test/suite/maria/t/maria3.test' --- a/mysql-test/suite/maria/t/maria3.test 2009-06-02 09:58:27 +0000 +++ b/mysql-test/suite/maria/t/maria3.test 2010-02-12 14:21:13 +0000 @@ -259,7 +259,7 @@ drop table t1; # Fix if we are using safemalloc --replace_result 8388572 8388600 -show variables like 'maria%'; +select lower(variable_name) as Variable_name, Variable_value as Value from information_schema.session_variables where variable_name like "maria%" and variable_name not like "maria_used_for_temp_tables" order by 1; --replace_column 2 # show status like 'maria%'; === added file 'mysql-test/suite/parts/t/partition_repair_myisam-master.opt' --- a/mysql-test/suite/parts/t/partition_repair_myisam-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/parts/t/partition_repair_myisam-master.opt 2010-02-10 21:26:06 +0000 @@ -0,0 +1 @@ +--myisam-recover=off === modified file 'mysql-test/suite/rpl/t/rpl_rotate_logs.test' --- a/mysql-test/suite/rpl/t/rpl_rotate_logs.test 2009-03-12 09:48:41 +0000 +++ b/mysql-test/suite/rpl/t/rpl_rotate_logs.test 2010-01-27 21:53:08 +0000 @@ -153,7 +153,7 @@ disable_query_log; while ($1) { #eval means expand $ expressions - eval insert into t3 values($1 + 4); + eval insert HIGH_PRIORITY into t3 values($1 + 4); dec $1; } enable_query_log; === modified file 'mysql-test/t/grant.test' --- a/mysql-test/t/grant.test 2009-06-15 15:53:45 +0000 +++ b/mysql-test/t/grant.test 2010-01-29 10:42:31 +0000 @@ -35,7 +35,7 @@ revoke delete on mysqltest.* from mysqlt show grants for mysqltest_1@localhost; grant select on mysqltest.* to mysqltest_1@localhost require NONE; show grants for mysqltest_1@localhost; -grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "MySQL AB"; +grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "Monty Program Ab"; show grants for mysqltest_1@localhost; revoke all privileges on mysqltest.* from mysqltest_1@localhost; show grants for mysqltest_1@localhost; === modified file 'mysql-test/t/openssl_1.test' --- a/mysql-test/t/openssl_1.test 2009-05-25 13:00:18 +0000 +++ b/mysql-test/t/openssl_1.test 2010-01-29 10:42:31 +0000 @@ -15,8 +15,10 @@ insert into t1 values (5); grant select on test.* to ssl_user1@localhost require SSL; grant select on test.* to ssl_user2@localhost require cipher "DHE-RSA-AES256-SHA"; -grant select on test.* to ssl_user3@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com"; -grant select on test.* to ssl_user4@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com" ISSUER "/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB"; +grant select on test.* to ssl_user3@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; +grant select on test.* to ssl_user4@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org" ISSUER "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; grant select on test.* to ssl_user5@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "xxx"; flush privileges; === modified file 'mysql-test/t/partition_innodb_semi_consistent.test' --- a/mysql-test/t/partition_innodb_semi_consistent.test 2009-01-13 22:12:16 +0000 +++ b/mysql-test/t/partition_innodb_semi_consistent.test 2010-01-28 14:49:14 +0000 @@ -187,6 +187,7 @@ SELECT * FROM t1; --echo # Switch to connection con2 connection con2; +--reap SELECT * FROM t1; connection default; === modified file 'mysql-test/t/query_cache_debug.test' --- a/mysql-test/t/query_cache_debug.test 2009-09-07 20:50:10 +0000 +++ b/mysql-test/t/query_cache_debug.test 2010-01-29 10:42:31 +0000 @@ -1,5 +1,4 @@ --source include/not_embedded.inc ---source include/big_test.inc --source include/have_query_cache.inc --source include/have_debug.inc === modified file 'mysql-test/t/show_check-master.opt' --- a/mysql-test/t/show_check-master.opt 2009-12-03 11:19:05 +0000 +++ b/mysql-test/t/show_check-master.opt 2010-01-27 21:53:08 +0000 @@ -1 +1 @@ ---log-output=table --slow-query-log --log-long-format --log-queries-not-using-indexes --myisam-recover="" --general-log --general-log-file="" --slow-query-log-file="" +--log-output=file --slow-query-log --log-long-format --log-queries-not-using-indexes --myisam-recover="" --general-log --general-log-file="foo" --slow-query-log-file="" === modified file 'mysql-test/t/sp-destruct.test' --- a/mysql-test/t/sp-destruct.test 2009-11-21 11:18:21 +0000 +++ b/mysql-test/t/sp-destruct.test 2010-02-10 19:06:24 +0000 @@ -17,6 +17,7 @@ call mtr.add_suppression("Column count o # Backup proc table let $MYSQLD_DATADIR= `select @@datadir`; +flush table mysql.proc; --copy_file $MYSQLD_DATADIR/mysql/proc.frm $MYSQLTEST_VARDIR/tmp/proc.frm --copy_file $MYSQLD_DATADIR/mysql/proc.MYD $MYSQLTEST_VARDIR/tmp/proc.MYD --copy_file $MYSQLD_DATADIR/mysql/proc.MYI $MYSQLTEST_VARDIR/tmp/proc.MYI === modified file 'mysql-test/t/table_elim.test' --- a/mysql-test/t/table_elim.test 2009-09-16 19:05:03 +0000 +++ b/mysql-test/t/table_elim.test 2010-02-18 12:03:52 +0000 @@ -336,3 +336,54 @@ select t1.* from t1 left join t2 on (t2.pk=3 and t2.b=3) or (t2.pk= 4 and t2.b=3); drop table t1, t2; + +--echo # +--echo # LPBUG#523593: Running RQG optimizer_no_subquery crashes MariaDB +--echo # + +CREATE TABLE t1 ( + `pk` int(11) NOT NULL AUTO_INCREMENT, + `col_int_nokey` int(11) DEFAULT NULL, + `col_int_key` int(11) DEFAULT NULL, + `col_date_key` date DEFAULT NULL, + `col_date_nokey` date DEFAULT NULL, + `col_time_key` time DEFAULT NULL, + `col_time_nokey` time DEFAULT NULL, + `col_datetime_key` datetime DEFAULT NULL, + `col_datetime_nokey` datetime DEFAULT NULL, + `col_varchar_key` varchar(1) DEFAULT NULL, + `col_varchar_nokey` varchar(1) DEFAULT NULL, + PRIMARY KEY (`pk`), + KEY `col_int_key` (`col_int_key`), + KEY `col_date_key` (`col_date_key`), + KEY `col_time_key` (`col_time_key`), + KEY `col_datetime_key` (`col_datetime_key`), + KEY `col_varchar_key` (`col_varchar_key`,`col_int_key`) +); +CREATE TABLE t2 LIKE t1; + +INSERT INTO t1 VALUES + (10,7,8,NULL,NULL,'01:27:35','01:27:35','2002-02-26 06:14:37','2002-02-26 06:14:37','v','v'), + (11,1,9,'2006-06-14','2006-06-14','19:48:31','19:48:31','1900-01-01 00:00:00','1900-01-01 00:00:00','r','r'); +INSERT INTO t2 SELECT * FROM t1; + +SELECT table2.col_int_key AS field1 +FROM ( + t2 AS table1 + RIGHT OUTER JOIN + ( + ( t1 AS table2 STRAIGHT_JOIN + t1 AS table3 ON ( + (table3.col_varchar_nokey = table2.col_varchar_key ) AND + (table3.pk = table2.col_int_key)) + ) + ) ON + ( + (table3.col_varchar_key = table2.col_varchar_key) OR + (table3.col_int_key = table2.pk) + ) +) +HAVING field1 < 216; + +DROP TABLE t1, t2; + === modified file 'mysql-test/t/udf.test' --- a/mysql-test/t/udf.test 2010-01-11 13:15:28 +0000 +++ b/mysql-test/t/udf.test 2010-01-22 21:19:21 +0000 @@ -56,9 +56,9 @@ select reverse_lookup(); --disable_result_log select reverse_lookup("127.0.0.1"); select reverse_lookup(127,0,0,1); +select reverse_lookup("localhost"); --enable_result_log -select reverse_lookup("localhost"); --error ER_CANT_INITIALIZE_UDF select avgcost(); --error ER_CANT_INITIALIZE_UDF === modified file 'mysql-test/t/view.test' --- a/mysql-test/t/view.test 2009-10-15 21:38:29 +0000 +++ b/mysql-test/t/view.test 2010-02-10 19:06:24 +0000 @@ -87,7 +87,7 @@ explain extended select c from v6; # show table/table status test show tables; show full tables; ---replace_column 8 # 12 # 13 # +--replace_column 8 # 12 # 13 # 14 # show table status; drop view v1,v2,v3,v4,v5,v6; === modified file 'mysys/my_file.c' --- a/mysys/my_file.c 2006-12-23 19:20:40 +0000 +++ b/mysys/my_file.c 2010-01-28 11:35:10 +0000 @@ -51,7 +51,7 @@ static uint set_max_open_files(uint max_ DBUG_PRINT("info", ("rlim_cur: %u rlim_max: %u", (uint) rlimit.rlim_cur, (uint) rlimit.rlim_max)); - if (rlimit.rlim_cur == RLIM_INFINITY) + if ((ulonglong) rlimit.rlim_cur == (ulonglong) RLIM_INFINITY) rlimit.rlim_cur = max_file_limit; if (rlimit.rlim_cur >= max_file_limit) DBUG_RETURN(rlimit.rlim_cur); /* purecov: inspected */ === modified file 'mysys/my_thr_init.c' --- a/mysys/my_thr_init.c 2009-12-03 11:19:05 +0000 +++ b/mysys/my_thr_init.c 2010-01-29 18:42:22 +0000 @@ -317,7 +317,7 @@ my_bool my_thread_init(void) /* Skip initialization if the thread specific variable is already initialized */ - if (THR_KEY_mysys.id) + if (THR_KEY_mysys.init) goto end; tmp= &THR_KEY_mysys; #endif === modified file 'sql-bench/bench-init.pl.sh' --- a/sql-bench/bench-init.pl.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/bench-init.pl.sh 2010-02-10 21:26:06 +0000 @@ -39,8 +39,8 @@ require "$pwd/server-cfg" || die "Can't $|=1; # Output data immediately -$opt_skip_test=$opt_skip_create=$opt_skip_delete=$opt_verbose=$opt_fast_insert=$opt_lock_tables=$opt_debug=$opt_skip_delete=$opt_fast=$opt_force=$opt_log=$opt_use_old_results=$opt_help=$opt_odbc=$opt_small_test=$opt_small_tables=$opt_samll_key_tables=$opt_stage=$opt_old_headers=$opt_die_on_errors=$opt_tcpip=$opt_random=$opt_only_missing_tests=0; -$opt_cmp=$opt_user=$opt_password=$opt_connect_options=""; +$opt_skip_test=$opt_skip_create=$opt_skip_delete=$opt_verbose=$opt_fast_insert=$opt_lock_tables=$opt_debug=$opt_skip_delete=$opt_fast=$opt_force=$opt_log=$opt_use_old_results=$opt_help=$opt_odbc=$opt_small_test=$opt_small_tables=$opt_samll_key_tables=$opt_stage=$opt_old_headers=$opt_die_on_errors=$opt_tcpip=$opt_random=$opt_only_missing_tests=$opt_temporary_tables=0; +$opt_cmp=$opt_user=$opt_password=$opt_connect_options=$opt_connect_command= ""; $opt_server="mysql"; $opt_dir="output"; $opt_host="localhost";$opt_database="test"; $opt_machine=""; $opt_suffix=""; @@ -59,7 +59,7 @@ $log_prog_args=join(" ", skip_arguments( "use-old-results","skip-test", "optimization","hw", "machine", "dir", "suffix", "log")); -GetOptions("skip-test=s","comments=s","cmp=s","server=s","user=s","host=s","database=s","password=s","loop-count=i","row-count=i","skip-create","skip-delete","verbose","fast-insert","lock-tables","debug","fast","force","field-count=i","regions=i","groups=i","time-limit=i","log","use-old-results","machine=s","dir=s","suffix=s","help","odbc","small-test","small-tables","small-key-tables","stage=i","threads=i","random","old-headers","die-on-errors","create-options=s","hires","tcpip","silent","optimization=s","hw=s","socket=s","connect-options=s","only-missing-tests") || usage(); +GetOptions("skip-test=s","comments=s","cmp=s","server=s","user=s","host=s","database=s","password=s","loop-count=i","row-count=i","skip-create","skip-delete","verbose","fast-insert","lock-tables","debug","fast","force","field-count=i","regions=i","groups=i","time-limit=i","log","use-old-results","machine=s","dir=s","suffix=s","help","odbc","small-test","small-tables","small-key-tables","stage=i","threads=i","random","old-headers","die-on-errors","create-options=s","hires","tcpip","silent","optimization=s","hw=s","socket=s","connect-options=s","connect-command=s","only-missing-tests","temporary-tables") || usage(); usage() if ($opt_help); $server=get_server($opt_server,$opt_host,$opt_database,$opt_odbc, @@ -454,6 +454,9 @@ All benchmarks takes the following optio create all MySQL tables as InnoDB tables use: --create-options=ENGINE=InnoDB +--temporary-tables + Use temporary tables for all tests. + --database (Default $opt_database) In which database the test tables are created. @@ -595,6 +598,10 @@ All benchmarks takes the following optio Add options, which uses at DBI connect. For example --connect-options=mysql_read_default_file=/etc/my.cnf. +--connect-command='SQL command' + Initialization command to execute when logged in. Useful for setting + up the environment. + EOF exit(0); } === modified file 'sql-bench/server-cfg.sh' --- a/sql-bench/server-cfg.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/server-cfg.sh 2010-02-10 21:26:06 +0000 @@ -159,6 +159,7 @@ sub new $limits{'max_index'} = 16; # Max number of keys $limits{'max_index_parts'} = 16; # Max segments/key $limits{'max_tables'} = (($machine || '') =~ "^win") ? 5000 : 65000; + $limits{'max_temporary_tables'}= 400; $limits{'max_text_size'} = 1000000; # Good enough for tests $limits{'multi_drop'} = 1; # Drop table can take many tables $limits{'order_by_position'} = 1; # Can use 'ORDER BY 1' @@ -189,6 +190,7 @@ sub new $self->{'transactions'} = 1; # Transactions enabled $limits{'max_columns'} = 90; # Max number of columns in table $limits{'max_tables'} = 32; # No comments + $limits{'max_temporary_tables'}= $limits{"max_tables"}; } if (defined($main::opt_create_options) && $main::opt_create_options =~ /engine=bdb/i) @@ -200,6 +202,7 @@ sub new { $limits{'working_blobs'} = 0; # Blobs not implemented yet $limits{'max_tables'} = 500; + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $self->{'transactions'} = 1; # Transactions enabled } @@ -249,6 +252,11 @@ sub connect die "Got error: '$DBI::errstr' when connecting to " . $self->{'data_source'} ." with user: '$main::opt_user' password: '$main::opt_password'\n"; $dbh->do("SET OPTION LOG_OFF=1,UPDATE_LOG=0"); + if ($main::opt_connect_command ne "") + { + $dbh->do($main::opt_connect_command) or + die "Can't execute connect_command: $main::opt_connect_command error: $DBI::errstr\n"; + } return $dbh; } @@ -265,7 +273,14 @@ sub create my($self,$table_name,$fields,$index,$options) = @_; my($query,@queries); - $query="create table $table_name ("; + if ($main::opt_temporary_tables) + { + $query="create temporary table $table_name ("; + } + else + { + $query="create table $table_name ("; + } foreach $field (@$fields) { # $field =~ s/ decimal/ double(10,2)/i; @@ -388,6 +403,7 @@ sub new $limits{'max_conditions'} = 74; $limits{'max_columns'} = 75; $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $limits{'max_text_size'} = 32000; $limits{'query_size'} = 65535; $limits{'max_index'} = 5; @@ -617,7 +633,9 @@ sub new $limits{'max_conditions'} = 9999; # This makes Pg real slow $limits{'max_index'} = 64; # Big enough $limits{'max_index_parts'} = 16; - $limits{'max_tables'} = 5000; # 10000 crashes pg 7.0.2 + $limits{'max_tables'} = 65000; + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 65000; # Good enough for test $limits{'multi_drop'} = 1; $limits{'order_by_position'} = 1; @@ -868,6 +886,8 @@ sub new $limits{'max_conditions'} = 9999; # Probably big enough $limits{'max_columns'} = 2000; # From crash-me $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 65492; # According to tests $limits{'query_size'} = 65535; # Probably a limit $limits{'max_index'} = 64; # Probably big enough @@ -1099,6 +1119,7 @@ sub new # above this value .... but can handle 2419 columns # maybe something for crash-me ... but how to check ??? $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $limits{'max_text_size'} = 4095; # max returned .... $limits{'query_size'} = 65535; # Not a limit, big enough $limits{'max_index'} = 64; # Big enough @@ -1369,6 +1390,8 @@ sub new $limits{'max_conditions'} = 9999; # (Actually not a limit) $limits{'max_columns'} = 254; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 2000; # Limit for blob test-connect $limits{'query_size'} = 65525; # Max size with default buffers. $limits{'max_index'} = 16; # Max number of keys @@ -1642,6 +1665,8 @@ sub new $limits{'max_column_name'} = 18; # max table and column name $limits{'max_columns'} = 994; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_index'} = 64; # Max number of keys $limits{'max_index_parts'} = 15; # Max segments/key $limits{'max_text_size'} = 65535; # Max size with default buffers. ?? @@ -1830,6 +1855,8 @@ sub new $limits{'max_conditions'} = 97; # We get 'Query is too complex' $limits{'max_columns'} = 255; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 255; # Max size with default buffers. $limits{'query_size'} = 65535; # Not a limit, big enough $limits{'max_index'} = 32; # Max number of keys @@ -2015,6 +2042,8 @@ sub new $limits{'max_conditions'} = 1030; # We get 'Query is too complex' $limits{'max_columns'} = 250; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 9830; # Max size with default buffers. $limits{'query_size'} = 9830; # Max size with default buffers. $limits{'max_index'} = 64; # Max number of keys @@ -2211,6 +2240,8 @@ sub new $limits{'max_conditions'} = 1030; # We get 'Query is too complex' $limits{'max_columns'} = 250; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 9830; # Max size with default buffers. $limits{'query_size'} = 9830; # Max size with default buffers. $limits{'max_index'} = 64; # Max number of keys @@ -2443,6 +2474,8 @@ sub new $limits{'max_conditions'} = 50; # (Actually not a limit) $limits{'max_columns'} = 254; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 2000; # Limit for blob test-connect $limits{'query_size'} = 65525; # Max size with default buffers. $limits{'max_index'} = 16; # Max number of keys @@ -2647,6 +2680,8 @@ sub new $limits{'max_conditions'} = 418; # We get 'Query is too complex' $limits{'max_columns'} = 500; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 254; # Max size with default buffers. $limits{'query_size'} = 254; # Max size with default buffers. $limits{'max_index'} = 48; # Max number of keys @@ -2825,6 +2860,7 @@ sub new $limits{'max_conditions'} = 9999; # (Actually not a limit) $limits{'max_columns'} = 252; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $limits{'max_text_size'} = 15000; # Max size with default buffers. $limits{'query_size'} = 1000000; # Max size with default buffers. $limits{'max_index'} = 32; # Max number of keys @@ -3027,6 +3063,7 @@ sub new $limits{'max_conditions'} = 9999; # (Actually not a limit) $limits{'max_columns'} = 252; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $limits{'max_text_size'} = 15000; # Max size with default buffers. $limits{'query_size'} = 1000000; # Max size with default buffers. $limits{'max_index'} = 65000; # Max number of keys @@ -3223,6 +3260,7 @@ sub new # The following should be 8192, but is smaller because Frontbase crashes.. $limits{'max_columns'} = 150; # Max number of columns in table $limits{'max_tables'} = 5000; # 10000 crashed FrontBase + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $limits{'max_text_size'} = 65000; # Max size with default buffers. $limits{'query_size'} = 8000000; # Max size with default buffers. $limits{'max_index'} = 38; # Max number of keys @@ -3435,6 +3473,7 @@ sub new $limits{'max_conditions'} = 9999; # (Actually not a limit) * $limits{'max_columns'} = 1023; # Max number of columns in table * $limits{'max_tables'} = 65000; # Should be big enough * unlimited actually + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $limits{'max_text_size'} = 15000; # Max size with default buffers. $limits{'query_size'} = 64*1024; # Max size with default buffers. *64 kb by default. May be set by system variable $limits{'max_index'} = 510; # Max number of keys * === modified file 'sql-bench/test-ATIS.sh' --- a/sql-bench/test-ATIS.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/test-ATIS.sh 2010-02-17 20:10:02 +0000 @@ -28,7 +28,7 @@ use Cwd; use DBI; use Benchmark; -$opt_loop_count=100; # Run selects this many times +$opt_loop_count=5000; # Run selects this many times $pwd = cwd(); $pwd = "." if ($pwd eq ''); require "$pwd/bench-init.pl" || die "Can't read Configuration file: $!\n"; === modified file 'sql-bench/test-alter-table.sh' --- a/sql-bench/test-alter-table.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/test-alter-table.sh 2010-02-17 20:10:02 +0000 @@ -25,7 +25,7 @@ use DBI; use Benchmark; $opt_start_field_count=8; # start with this many fields -$opt_loop_count=100; # How many tests to do +$opt_loop_count=10000; # How many tests to do $opt_row_count=1000; # Rows in the table $opt_field_count=1000; # Add until this many fields. $opt_time_limit=10*60; # Don't wait more than 10 min for some tests === modified file 'sql-bench/test-big-tables.sh' --- a/sql-bench/test-big-tables.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/test-big-tables.sh 2010-02-17 20:10:02 +0000 @@ -25,7 +25,7 @@ use Cwd; use DBI; use Benchmark; -$opt_loop_count=1000; # Change this to make test harder/easier +$opt_loop_count=70000; # Change this to make test harder/easier $opt_field_count=1000; $pwd = cwd(); $pwd = "." if ($pwd eq ''); === modified file 'sql-bench/test-connect.sh' --- a/sql-bench/test-connect.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/test-connect.sh 2010-02-17 20:10:02 +0000 @@ -28,7 +28,7 @@ use Cwd; use DBI; use Benchmark; -$opt_loop_count=100000; # Change this to make test harder/easier +$opt_loop_count=500000; # Change this to make test harder/easier $str_length=65000; # This is the length of blob strings in PART:5 $max_test=20; # How many times to test if the server is busy @@ -161,41 +161,48 @@ if ($opt_fast && defined($server->{vacuu { $server->vacuum(0,\$dbh); } -$dbh->disconnect; +if (!$main::opt_temporary_tables) +{ + $dbh->disconnect; +} # # First test connect/select/disconnect # -print "Testing connect/select 1 row from table/disconnect\n"; +if (!$main::opt_temporary_tables) +{ + print "Testing connect/select 1 row from table/disconnect\n"; -$loop_time=new Benchmark; -$errors=0; + $loop_time=new Benchmark; + $errors=0; -for ($i=0 ; $i < $small_loop_count ; $i++) -{ - for ($j=0; $j < $max_test ; $j++) + for ($i=0 ; $i < $small_loop_count ; $i++) { - last if ($dbh = DBI->connect($server->{'data_source'}, $opt_user, $opt_password)); - $errors++; - } - die $DBI::errstr if ($j == $max_test); + for ($j=0; $j < $max_test ; $j++) + { + last if ($dbh = DBI->connect($server->{'data_source'}, $opt_user, $opt_password)); + $errors++; + } + die $DBI::errstr if ($j == $max_test); - $sth = $dbh->do("select a,i,s,$i from bench1") # Select * from table with 1 record + $sth = $dbh->do("select a,i,s,$i from bench1") # Select * from table with 1 record or die $DBI::errstr; - $dbh->disconnect; -} + $dbh->disconnect; + } -$end_time=new Benchmark; -print "Warning: $errors connections didn't work without a time delay\n" if ($errors); -print "Time to connect+select_1_row ($small_loop_count): " . + $end_time=new Benchmark; + print "Warning: $errors connections didn't work without a time delay\n" if ($errors); + print "Time to connect+select_1_row ($small_loop_count): " . timestr(timediff($end_time, $loop_time),"all") . "\n\n"; + $dbh = $server->connect(); +} + # # The same test, but without connect/disconnect # print "Testing select 1 row from table\n"; -$dbh = $server->connect(); $loop_time=new Benchmark; for ($i=0 ; $i < $opt_loop_count ; $i++) === modified file 'sql-bench/test-create.sh' --- a/sql-bench/test-create.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/test-create.sh 2010-02-10 21:26:06 +0000 @@ -47,7 +47,15 @@ if ($opt_small_test) $create_loop_count/=1000; } -$max_tables=min($limits->{'max_tables'},$opt_loop_count); +if ($opt_temporary_tables) +{ + $max_tables=min($limits->{'max_tables'},$opt_loop_count); +} +else +{ + $max_tables=min($limits->{'max_tables'},$opt_loop_count); + $max_tables=400; +} if ($opt_small_test) { @@ -71,7 +79,7 @@ $dbh = $server->connect(); if ($opt_force) # If tables used in this test exist, drop 'em { print "Okay..Let's make sure that our tables don't exist yet.\n\n"; - for ($i=1 ; $i <= $max_tables ; $i++) + for ($i=1 ; $i <= max($max_tables, $create_loop_count) ; $i++) { $dbh->do("drop table bench_$i" . $server->{'drop_attr'}); } @@ -245,7 +253,7 @@ for ($i=2 ; $i <= $keys ; $i++) } $loop_time=new Benchmark; -for ($i=1 ; $i <= $opt_loop_count ; $i++) +for ($i=1 ; $i <= $create_loop_count ; $i++) { do_many($dbh,$server->create("bench_$i", \@fields, \@keys)); $dbh->do("drop table bench_$i" . $server->{'drop_attr'}) or die $DBI::errstr; === modified file 'sql-bench/test-select.sh' --- a/sql-bench/test-select.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/test-select.sh 2010-02-17 20:10:02 +0000 @@ -26,7 +26,7 @@ use Getopt::Long; use Benchmark; $opt_loop_count=10000; -$opt_medium_loop_count=1000; +$opt_medium_loop_count=7000; $opt_small_loop_count=10; $opt_regions=6; $opt_groups=100; === modified file 'sql-bench/test-transactions.sh' --- a/sql-bench/test-transactions.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/test-transactions.sh 2010-02-17 20:10:02 +0000 @@ -28,8 +28,8 @@ use Benchmark; $opt_groups=27; # Characters are 'A' -> Z -$opt_loop_count=10000; # Change this to make test harder/easier -$opt_medium_loop_count=100; # Change this to make test harder/easier +$opt_loop_count=500000; # Change this to make test harder/easier +$opt_medium_loop_count=10000; # Change this to make test harder/easier $pwd = cwd(); $pwd = "." if ($pwd eq ''); require "$pwd/bench-init.pl" || die "Can't read Configuration file: $!\n"; === modified file 'sql-bench/test-wisconsin.sh' --- a/sql-bench/test-wisconsin.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/test-wisconsin.sh 2010-02-17 20:10:02 +0000 @@ -21,7 +21,7 @@ use Cwd; use DBI; use Benchmark; -$opt_loop_count=10; +$opt_loop_count=5000; $pwd = cwd(); $pwd = "." if ($pwd eq ''); require "$pwd/bench-init.pl" || die "Can't read Configuration file: $!\n"; === modified file 'sql-common/client.c' --- a/sql-common/client.c 2009-12-03 15:26:54 +0000 +++ b/sql-common/client.c 2010-01-29 18:42:22 +0000 @@ -1863,6 +1863,7 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons uint port, const char *unix_socket,ulong client_flag) { char buff[NAME_LEN+USERNAME_LENGTH+100]; + char error_string[1024]; char *end,*host_info= NULL; my_socket sock; in_addr_t ip_addr; @@ -1939,7 +1940,8 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons #if defined(HAVE_SMEM) if ((!mysql->options.protocol || mysql->options.protocol == MYSQL_PROTOCOL_MEMORY) && - (!host || !strcmp(host,LOCAL_HOST))) + (!host || !strcmp(host,LOCAL_HOST)) && + mysql->options.shared_memory_base_name) { if ((create_shared_memory(mysql,net, mysql->options.connect_timeout)) == INVALID_HANDLE_VALUE) @@ -1948,7 +1950,7 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons ("host: '%s' socket: '%s' shared memory: %s have_tcpip: %d", host ? host : "<null>", unix_socket ? unix_socket : "<null>", - (int) mysql->options.shared_memory_base_name, + mysql->options.shared_memory_base_name, (int) have_tcpip)); if (mysql->options.protocol == MYSQL_PROTOCOL_MEMORY) goto error; @@ -2331,9 +2333,14 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons /* Connect to the server */ DBUG_PRINT("info", ("IO layer change in progress...")); if (sslconnect(ssl_fd, mysql->net.vio, - (long) (mysql->options.connect_timeout))) + (long) (mysql->options.connect_timeout), + error_string)) { - set_mysql_error(mysql, CR_SSL_CONNECTION_ERROR, unknown_sqlstate); + set_mysql_extended_error(mysql, CR_SSL_CONNECTION_ERROR, + unknown_sqlstate, + "SSL error: %s", + error_string[0] ? error_string : + ER(CR_SSL_CONNECTION_ERROR)); goto error; } DBUG_PRINT("info", ("IO layer change done!")); @@ -2746,6 +2753,13 @@ void mysql_detach_stmt_list(LIST **stmt_ } +/* + Close a MySQL connection and free all resources attached to it. + + This function is coded in such that it can be called multiple times + (As some clients call this after mysql_real_connect() fails) +*/ + void STDCALL mysql_close(MYSQL *mysql) { DBUG_ENTER("mysql_close"); @@ -2779,10 +2793,16 @@ void STDCALL mysql_close(MYSQL *mysql) } #endif if (mysql != mysql->master) + { mysql_close(mysql->master); + mysql->master= 0; + } #ifndef MYSQL_SERVER if (mysql->thd) + { (*mysql->methods->free_embedded_thd)(mysql); + mysql->thd= 0; + } #endif if (mysql->free_me) my_free((uchar*) mysql,MYF(0)); === modified file 'sql/CMakeLists.txt' --- a/sql/CMakeLists.txt 2009-12-03 11:19:05 +0000 +++ b/sql/CMakeLists.txt 2010-01-29 18:42:22 +0000 @@ -97,7 +97,7 @@ SET_TARGET_PROPERTIES(mysqld PROPERTIES SET (MYSQLD_CORE_LIBS mysys zlib dbug strings yassl taocrypt vio regex sql libevent) TARGET_LINK_LIBRARIES(mysqld ${MYSQLD_CORE_LIBS} ${MYSQLD_STATIC_ENGINE_LIBS}) -TARGET_LINK_LIBRARIES(mysqld ws2_32.lib) +TARGET_LINK_LIBRARIES(mysqld ws2_32.lib psapi.lib) IF(MSVC AND NOT WITHOUT_DYNAMIC_PLUGINS) === modified file 'sql/item.cc' --- a/sql/item.cc 2010-01-15 15:27:55 +0000 +++ b/sql/item.cc 2010-01-27 21:53:08 +0000 @@ -7119,7 +7119,7 @@ double Item_cache_decimal::val_real() DBUG_ASSERT(fixed); double res; if (!value_cached && !cache_value()) - return NULL; + return 0.0; my_decimal2double(E_DEC_FATAL_ERROR, &decimal_value, &res); return res; } === modified file 'sql/lock.cc' --- a/sql/lock.cc 2009-10-15 21:38:29 +0000 +++ b/sql/lock.cc 2010-02-10 19:06:24 +0000 @@ -1049,10 +1049,14 @@ int lock_table_name(THD *thd, TABLE_LIST DBUG_RETURN(-1); table_list->table=table; + table->s->deleting= table_list->deleting; /* Return 1 if table is in use */ DBUG_RETURN(test(remove_table_from_cache(thd, db, table_list->table_name, - check_in_use ? RTFC_NO_FLAG : RTFC_WAIT_OTHER_THREAD_FLAG))); + (check_in_use ? + RTFC_NO_FLAG : + RTFC_WAIT_OTHER_THREAD_FLAG), + table_list->deleting))); } === modified file 'sql/mysql_priv.h' --- a/sql/mysql_priv.h 2009-12-03 11:19:05 +0000 +++ b/sql/mysql_priv.h 2010-02-10 19:06:24 +0000 @@ -1636,7 +1636,7 @@ uint prep_alter_part_table(THD *thd, TAB #define RTFC_WAIT_OTHER_THREAD_FLAG 0x0002 #define RTFC_CHECK_KILLED_FLAG 0x0004 bool remove_table_from_cache(THD *thd, const char *db, const char *table, - uint flags); + uint flags, my_bool deleting); #define NORMAL_PART_NAME 0 #define TEMP_PART_NAME 1 === modified file 'sql/mysqld.cc' --- a/sql/mysqld.cc 2010-01-15 15:27:55 +0000 +++ b/sql/mysqld.cc 2010-02-11 19:15:24 +0000 @@ -598,6 +598,7 @@ char *mysqld_unix_port, *opt_mysql_tmpdi const char **errmesg; /**< Error messages */ const char *myisam_recover_options_str="OFF"; const char *myisam_stats_method_str="nulls_unequal"; +const char *opt_thread_handling= thread_handling_typelib.type_names[0]; /** name of reference on left espression in rewritten IN subquery */ const char *in_left_expr_name= "<left expr>"; @@ -1173,11 +1174,14 @@ static void __cdecl kill_server(int sig_ /* Send event to smem_event_connect_request for aborting */ - if (!SetEvent(smem_event_connect_request)) + if (opt_enable_shared_memory) { - DBUG_PRINT("error", - ("Got error: %ld from SetEvent of smem_event_connect_request", - GetLastError())); + if (!SetEvent(smem_event_connect_request)) + { + DBUG_PRINT("error", + ("Got error: %ld from SetEvent of smem_event_connect_request", + GetLastError())); + } } #endif @@ -2704,7 +2708,7 @@ static void init_signals(void) { /* Change limits so that we will get a core file */ STRUCT_RLIMIT rl; - rl.rlim_cur = rl.rlim_max = RLIM_INFINITY; + rl.rlim_cur = rl.rlim_max = (rlim_t) RLIM_INFINITY; if (setrlimit(RLIMIT_CORE, &rl) && global_system_variables.log_warnings) sql_print_warning("setrlimit could not change the size of core files to 'infinity'; We may not be able to generate a core file on signals"); } @@ -7287,7 +7291,8 @@ The minimum value for this variable is 4 1024, 0}, {"thread_handling", OPT_THREAD_HANDLING, "Define threads usage for handling queries: " - "one-thread-per-connection or no-threads", 0, 0, + "one-thread-per-connection or no-threads", + (uchar**) &opt_thread_handling, (uchar**) &opt_thread_handling, 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, {"updatable_views_with_limit", OPT_UPDATABLE_VIEWS_WITH_LIMIT, "1 = YES = Don't issue an error message (warning only) if a VIEW without presence of a key of the underlying table is used in queries with a LIMIT clause for updating. 0 = NO = Prohibit update of a VIEW, which does not contain a key of the underlying table and the query uses a LIMIT clause (usually get from GUI tools).", @@ -7959,7 +7964,13 @@ static int mysql_init_variables(void) refresh_version= 1L; /* Increments on each reload */ global_query_id= thread_id= 1L; strmov(server_version, MYSQL_SERVER_VERSION); - myisam_recover_options_str= sql_mode_str= "OFF"; + sql_mode_str= ""; + + /* By default, auto-repair MyISAM tables after crash */ + myisam_recover_options_str= "DEFAULT"; + myisam_recover_options= HA_RECOVER_DEFAULT; + ha_open_options|= HA_OPEN_ABORT_IF_CRASHED; + myisam_stats_method_str= "nulls_unequal"; my_bind_addr = htonl(INADDR_ANY); threads.empty(); @@ -8613,26 +8624,31 @@ mysqld_get_one_option(int optid, #endif case OPT_MYISAM_RECOVER: { - if (!argument) - { - myisam_recover_options= HA_RECOVER_DEFAULT; - myisam_recover_options_str= myisam_recover_typelib.type_names[0]; - } - else if (!argument[0]) + if (argument && (!argument[0] || + my_strcasecmp(system_charset_info, argument, "OFF") == 0)) { myisam_recover_options= HA_RECOVER_NONE; myisam_recover_options_str= "OFF"; + ha_open_options&= ~HA_OPEN_ABORT_IF_CRASHED; } else { - myisam_recover_options_str=argument; - myisam_recover_options= - find_bit_type_or_exit(argument, &myisam_recover_typelib, opt->name, - &error); - if (error) - return 1; + if (!argument) + { + myisam_recover_options= HA_RECOVER_DEFAULT; + myisam_recover_options_str= myisam_recover_typelib.type_names[0]; + } + else + { + myisam_recover_options_str=argument; + myisam_recover_options= + find_bit_type_or_exit(argument, &myisam_recover_typelib, opt->name, + &error); + if (error) + return 1; + } + ha_open_options|=HA_OPEN_ABORT_IF_CRASHED; } - ha_open_options|=HA_OPEN_ABORT_IF_CRASHED; break; } case OPT_CONCURRENT_INSERT: @@ -8707,14 +8723,15 @@ mysqld_get_one_option(int optid, break; } case OPT_ONE_THREAD: - global_system_variables.thread_handling= - SCHEDULER_ONE_THREAD_PER_CONNECTION; + global_system_variables.thread_handling= SCHEDULER_NO_THREADS; + opt_thread_handling= thread_handling_typelib.type_names[global_system_variables.thread_handling]; break; case OPT_THREAD_HANDLING: { int id; if (!find_opt_type(argument, &thread_handling_typelib, opt->name, &id)) global_system_variables.thread_handling= id - 1; + opt_thread_handling= thread_handling_typelib.type_names[global_system_variables.thread_handling]; break; } case OPT_FT_BOOLEAN_SYNTAX: === modified file 'sql/net_serv.cc' --- a/sql/net_serv.cc 2009-09-07 20:50:10 +0000 +++ b/sql/net_serv.cc 2010-01-29 10:42:31 +0000 @@ -219,7 +219,7 @@ my_bool net_realloc(NET *net, size_t len -1 Don't know if data is ready or not */ -#if !defined(EMBEDDED_LIBRARY) +#if !defined(EMBEDDED_LIBRARY) && defined(DBUG_OFF) static int net_data_is_ready(my_socket sd) { @@ -280,13 +280,18 @@ static int net_data_is_ready(my_socket s void net_clear(NET *net, my_bool clear_buffer) { -#if !defined(EMBEDDED_LIBRARY) +#if !defined(EMBEDDED_LIBRARY) && defined(DBUG_OFF) size_t count; int ready; #endif DBUG_ENTER("net_clear"); -#if !defined(EMBEDDED_LIBRARY) +/* + We don't do a clear in case of DBUG_OFF to catch bugs + in the protocol handling +*/ + +#if !defined(EMBEDDED_LIBRARY) && defined(DBUG_OFF) if (clear_buffer) { while ((ready= net_data_is_ready(net->vio->sd)) > 0) === modified file 'sql/opt_table_elimination.cc' --- a/sql/opt_table_elimination.cc 2009-09-16 19:05:03 +0000 +++ b/sql/opt_table_elimination.cc 2010-02-18 12:03:52 +0000 @@ -1220,8 +1220,16 @@ void build_eq_mods_for_cond(Dep_analysis bound_item= item; } } - exchange_sort<Dep_value_field>(fvl, compare_field_values, NULL); - add_module_expr(ctx, eq_mod, *and_level, NULL, bound_item, fvl); + /* + Multiple equality is only useful if it includes at least one field from + the table that we could potentially eliminate: + */ + if (fvl->elements) + { + + exchange_sort<Dep_value_field>(fvl, compare_field_values, NULL); + add_module_expr(ctx, eq_mod, *and_level, NULL, bound_item, fvl); + } break; } default: === modified file 'sql/rpl_record.cc' --- a/sql/rpl_record.cc 2009-10-22 00:15:45 +0000 +++ b/sql/rpl_record.cc 2010-01-28 11:35:10 +0000 @@ -366,7 +366,6 @@ int prepare_record(TABLE *const table, */ for (Field **field_ptr= table->field+skip; *field_ptr; ++field_ptr) { - uint32 const mask= NOT_NULL_FLAG | NO_DEFAULT_VALUE_FLAG; Field *const f= *field_ptr; if ((f->flags & NO_DEFAULT_VALUE_FLAG) && (f->real_type() != MYSQL_TYPE_ENUM)) === modified file 'sql/slave.cc' --- a/sql/slave.cc 2009-12-06 17:51:48 +0000 +++ b/sql/slave.cc 2010-01-29 10:42:31 +0000 @@ -3803,10 +3803,11 @@ static int connect_to_master(THD* thd, M suppress_warnings= 0; mi->report(ERROR_LEVEL, last_errno, "error %s to master '%s@%s:%d'" - " - retry-time: %d retries: %lu", + " - retry-time: %d retries: %lu message: %s", (reconnect ? "reconnecting" : "connecting"), mi->user, mi->host, mi->port, - mi->connect_retry, master_retry_count); + mi->connect_retry, master_retry_count, + mysql_error(mysql)); } /* By default we try forever. The reason is that failure will trigger === modified file 'sql/sql_base.cc' --- a/sql/sql_base.cc 2010-01-15 15:27:55 +0000 +++ b/sql/sql_base.cc 2010-02-10 19:06:24 +0000 @@ -930,7 +930,7 @@ bool close_cached_tables(THD *thd, TABLE for (TABLE_LIST *table= tables; table; table= table->next_local) { if (remove_table_from_cache(thd, table->db, table->table_name, - RTFC_OWNED_BY_THD_FLAG)) + RTFC_OWNED_BY_THD_FLAG, table->deleting)) found=1; } if (!found) @@ -8404,6 +8404,11 @@ void remove_db_from_cache(const char *db if (!strcmp(table->s->db.str, db)) { table->s->version= 0L; /* Free when thread is ready */ + /* + This functions only called from DROP DATABASE code, so we are going + to drop all tables so we mark them as deleting + */ + table->s->deleting= TRUE; if (!table->in_use) relink_unused(table); } @@ -8446,7 +8451,7 @@ void flush_tables() */ bool remove_table_from_cache(THD *thd, const char *db, const char *table_name, - uint flags) + uint flags, my_bool deleting) { char key[MAX_DBKEY_LENGTH]; uint key_length; @@ -8540,7 +8545,10 @@ bool remove_table_from_cache(THD *thd, c } } while (unused_tables && !unused_tables->s->version) + { + unused_tables->s->deleting= deleting; VOID(hash_delete(&open_cache,(uchar*) unused_tables)); + } DBUG_PRINT("info", ("Removing table from table_def_cache")); /* Remove table from table definition cache if it's not in use */ @@ -8734,7 +8742,8 @@ int abort_and_upgrade_lock(ALTER_PARTITI /* If MERGE child, forward lock handling to parent. */ mysql_lock_abort(lpt->thd, lpt->table->parent ? lpt->table->parent : lpt->table, TRUE); - VOID(remove_table_from_cache(lpt->thd, lpt->db, lpt->table_name, flags)); + VOID(remove_table_from_cache(lpt->thd, lpt->db, lpt->table_name, flags, + FALSE)); VOID(pthread_mutex_unlock(&LOCK_open)); DBUG_RETURN(0); } @@ -8759,7 +8768,7 @@ void close_open_tables_and_downgrade(ALT { VOID(pthread_mutex_lock(&LOCK_open)); remove_table_from_cache(lpt->thd, lpt->db, lpt->table_name, - RTFC_WAIT_OTHER_THREAD_FLAG); + RTFC_WAIT_OTHER_THREAD_FLAG, FALSE); VOID(pthread_mutex_unlock(&LOCK_open)); /* If MERGE child, forward lock handling to parent. */ mysql_lock_downgrade_write(lpt->thd, lpt->table->parent ? lpt->table->parent : === modified file 'sql/sql_cache.cc' --- a/sql/sql_cache.cc 2010-01-15 15:27:55 +0000 +++ b/sql/sql_cache.cc 2010-01-29 10:42:31 +0000 @@ -377,9 +377,11 @@ TODO list: static void debug_wait_for_kill(const char *info) { - DBUG_ENTER("debug_wait_for_kill"); const char *prev_info; THD *thd; + char buff[1024]; + DBUG_ENTER("debug_wait_for_kill"); + thd= current_thd; prev_info= thd->proc_info; thd->proc_info= info; @@ -387,8 +389,16 @@ static void debug_wait_for_kill(const ch while(!thd->killed) my_sleep(1000); thd->killed= THD::NOT_KILLED; + /* + Remove the set debug variable, to ensure we don't get stuck on it again + This is needed as for MyISAM, invalidate_table() may be called twice + (Once from mysql_delete() and once from mi_update_status()) + */ + sprintf(buff, "-d,%s", info); + DBUG_SET(buff); sql_print_information("Exit debug_wait_for_kill"); thd->proc_info= prev_info; + DBUG_VOID_RETURN; } @@ -914,15 +924,18 @@ void query_cache_insert(NET *net, const void query_cache_abort(NET *net) { + THD *thd; DBUG_ENTER("query_cache_abort"); - THD *thd= current_thd; /* See the comment on double-check locking usage above. */ if (net->query_cache_query == 0) DBUG_VOID_RETURN; if (query_cache.try_lock()) + { + net->query_cache_query = 0; DBUG_VOID_RETURN; + } /* While we were waiting another thread might have changed the status @@ -932,6 +945,7 @@ void query_cache_abort(NET *net) net->query_cache_query); if (query_block) { + thd= current_thd; thd_proc_info(thd, "storing result in query cache"); DUMP(&query_cache); BLOCK_LOCK_WR(query_block); @@ -941,6 +955,7 @@ void query_cache_abort(NET *net) DBUG_EXECUTE("check_querycache",query_cache.check_integrity(1);); } + DBUG_ASSERT(!net->query_cache_query); query_cache.unlock(); DBUG_VOID_RETURN; } @@ -970,8 +985,12 @@ void query_cache_end_of_result(THD *thd) #endif if (query_cache.try_lock()) + { + thd->net.query_cache_query= 0; DBUG_VOID_RETURN; + } + /* thd->net.query_cache_query may have changed during resize */ query_block= ((Query_cache_block*) thd->net.query_cache_query); if (query_block) { @@ -997,8 +1016,8 @@ void query_cache_end_of_result(THD *thd) to this function. In the release version that query should be ignored and removed from QC. */ - DBUG_ASSERT(0); query_cache.free_query(query_block); + thd->net.query_cache_query= 0; query_cache.unlock(); DBUG_VOID_RETURN; } === modified file 'sql/sql_connect.cc' --- a/sql/sql_connect.cc 2009-12-03 11:19:05 +0000 +++ b/sql/sql_connect.cc 2010-01-29 10:42:31 +0000 @@ -798,6 +798,7 @@ static int check_connection(THD *thd) DBUG_PRINT("info", ("client capabilities: %lu", thd->client_capabilities)); if (thd->client_capabilities & CLIENT_SSL) { + char error_string[1024]; /* Do the SSL layering. */ if (!ssl_acceptor_fd) { @@ -806,7 +807,7 @@ static int check_connection(THD *thd) return 1; } DBUG_PRINT("info", ("IO layer change in progress...")); - if (sslaccept(ssl_acceptor_fd, net->vio, net->read_timeout)) + if (sslaccept(ssl_acceptor_fd, net->vio, net->read_timeout, error_string)) { DBUG_PRINT("error", ("Failed to accept new SSL connection")); inc_host_errors(&thd->remote.sin_addr); === modified file 'sql/sql_delete.cc' --- a/sql/sql_delete.cc 2010-01-15 15:27:55 +0000 +++ b/sql/sql_delete.cc 2010-02-10 19:06:24 +0000 @@ -1088,6 +1088,7 @@ bool mysql_truncate(THD *thd, TABLE_LIST HA_CREATE_INFO create_info; char path[FN_REFLEN + 1]; TABLE *table; + TABLE_LIST *tbl; bool error; uint path_length; bool is_temporary_table= false; @@ -1108,6 +1109,9 @@ bool mysql_truncate(THD *thd, TABLE_LIST if (!ha_check_storage_engine_flag(table_type, HTON_CAN_RECREATE)) goto trunc_by_del; + for (tbl= table_list; tbl; tbl= tbl->next_local) + tbl->deleting= TRUE; /* to trigger HA_PREPARE_FOR_DROP */ + table->file->info(HA_STATUS_AUTO | HA_STATUS_NO_LOCK); create_info.options|= HA_LEX_CREATE_TMP_TABLE; === modified file 'sql/sql_parse.cc' --- a/sql/sql_parse.cc 2010-01-15 15:27:55 +0000 +++ b/sql/sql_parse.cc 2010-01-29 10:42:31 +0000 @@ -1252,12 +1252,14 @@ bool dispatch_command(enum enum_server_c ha_maria::implicit_commit(thd, FALSE); #endif - net_end_statement(thd); - query_cache_end_of_result(thd); /* Multiple queries exits, execute them individually */ close_thread_tables(thd); + + net_end_statement(thd); + query_cache_end_of_result(thd); + ulong length= (ulong)(packet_end - beginning_of_next_stmt); log_slow_statement(thd); @@ -1645,13 +1647,22 @@ bool dispatch_command(enum enum_server_c ha_maria::implicit_commit(thd, FALSE); #endif - net_end_statement(thd); - query_cache_end_of_result(thd); - + if (!(sql_command_flags[thd->lex->sql_command] & CF_CHANGES_DATA)) + { + /* No changes in data; We can send ok at once to the client */ + net_end_statement(thd); + query_cache_end_of_result(thd); + } thd->proc_info= "closing tables"; /* Free tables */ close_thread_tables(thd); + if (sql_command_flags[thd->lex->sql_command] & CF_CHANGES_DATA) + { + net_end_statement(thd); + query_cache_end_of_result(thd); + } + log_slow_statement(thd); thd_proc_info(thd, "cleaning up"); === modified file 'sql/sql_profile.cc' --- a/sql/sql_profile.cc 2009-10-15 21:38:29 +0000 +++ b/sql/sql_profile.cc 2010-01-29 18:42:22 +0000 @@ -131,6 +131,23 @@ int make_profile_table_for_show(THD *thd #define RUSAGE_USEC(tv) ((tv).tv_sec*1000*1000 + (tv).tv_usec) #define RUSAGE_DIFF_USEC(tv1, tv2) (RUSAGE_USEC((tv1))-RUSAGE_USEC((tv2))) +#ifdef __WIN__ +inline ULONGLONG FileTimeToQuadWord(FILETIME *ft) +{ + ULONGLONG nrv = 0; + nrv |= ft->dwHighDateTime; + nrv <<= 32; + nrv |= ft->dwLowDateTime; + return nrv; +} + + +// Get time difference between to FILETIME objects in seconds. +inline double GetTimeDiffInSeconds(FILETIME *a, FILETIME *b) +{ + return ((FileTimeToQuadWord(a) - FileTimeToQuadWord(b)) / 1e7); +} +#endif /* __WIN__ */ PROF_MEASUREMENT::PROF_MEASUREMENT(QUERY_PROFILE *profile_arg, const char *status_arg) @@ -221,6 +238,11 @@ void PROF_MEASUREMENT::collect() time_usecs= (double) my_getsystime() / 10.0; /* 1 sec was 1e7, now is 1e6 */ #ifdef HAVE_GETRUSAGE getrusage(RUSAGE_SELF, &rusage); +#elif defined(__WIN__) + FILETIME ftDummy; + GetProcessTimes(GetCurrentProcess(), &ftDummy, &ftDummy, &ftKernel, &ftUser); + GetProcessIoCounters(GetCurrentProcess(), &io_count); + GetProcessMemoryInfo(GetCurrentProcess(), &mem_count, sizeof(mem_count)); #endif } @@ -590,6 +612,23 @@ int PROFILING::fill_statistics_info(THD table->field[5]->store_decimal(&cpu_stime_decimal); table->field[4]->set_notnull(); table->field[5]->set_notnull(); +#elif defined(__WIN__) + my_decimal cpu_utime_decimal, cpu_stime_decimal; + + double2my_decimal(E_DEC_FATAL_ERROR, + GetTimeDiffInSeconds(&entry->ftUser, + &previous->ftUser), + &cpu_utime_decimal); + double2my_decimal(E_DEC_FATAL_ERROR, + GetTimeDiffInSeconds(&entry->ftKernel, + &previous->ftKernel), + &cpu_stime_decimal); + + // Store the result. + table->field[4]->store_decimal(&cpu_utime_decimal); + table->field[5]->store_decimal(&cpu_stime_decimal); + table->field[4]->set_notnull(); + table->field[5]->set_notnull(); #else /* TODO: Add CPU-usage info for non-BSD systems */ #endif @@ -612,6 +651,17 @@ int PROFILING::fill_statistics_info(THD table->field[9]->store((uint32)(entry->rusage.ru_oublock - previous->rusage.ru_oublock)); table->field[9]->set_notnull(); +#elif defined(__WIN__) + ULONGLONG reads_delta = entry->io_count.ReadOperationCount - + previous->io_count.ReadOperationCount; + ULONGLONG writes_delta = entry->io_count.WriteOperationCount - + previous->io_count.WriteOperationCount; + + table->field[8]->store((uint32)reads_delta); + table->field[8]->set_notnull(); + + table->field[9]->store((uint32)writes_delta); + table->field[9]->set_notnull(); #else /* TODO: Add block IO info for non-BSD systems */ #endif @@ -634,6 +684,13 @@ int PROFILING::fill_statistics_info(THD table->field[13]->store((uint32)(entry->rusage.ru_minflt - previous->rusage.ru_minflt), true); table->field[13]->set_notnull(); +#elif defined(__WIN__) + /* Windows APIs don't easily distinguish between hard and soft page + faults, so we just fill the 'major' column and leave the second NULL. + */ + table->field[12]->store((uint32)(entry->mem_count.PageFaultCount - + previous->mem_count.PageFaultCount), true); + table->field[12]->set_notnull(); #else /* TODO: Add page fault info for non-BSD systems */ #endif === modified file 'sql/sql_profile.h' --- a/sql/sql_profile.h 2009-09-07 20:50:10 +0000 +++ b/sql/sql_profile.h 2010-01-29 18:42:22 +0000 @@ -36,6 +36,10 @@ int make_profile_table_for_show(THD *thd #if defined(ENABLED_PROFILING) && defined(COMMUNITY_SERVER) #include "mysql_priv.h" +#ifdef __WIN__ +#include <psapi.h> +#endif + #ifdef HAVE_SYS_RESOURCE_H #include <sys/resource.h> #endif @@ -165,6 +169,10 @@ private: char *status; #ifdef HAVE_GETRUSAGE struct rusage rusage; +#elif defined(__WIN__) + FILETIME ftKernel, ftUser; + IO_COUNTERS io_count; + PROCESS_MEMORY_COUNTERS mem_count; #endif char *function; === modified file 'sql/sql_table.cc' --- a/sql/sql_table.cc 2010-01-15 15:27:55 +0000 +++ b/sql/sql_table.cc 2010-02-10 19:06:24 +0000 @@ -1880,6 +1880,7 @@ int mysql_rm_table_part2(THD *thd, TABLE { TABLE_SHARE *share; table->db_type= NULL; + if ((share= get_cached_table_share(table->db, table->table_name))) table->db_type= share->db_type(); @@ -1974,9 +1975,10 @@ int mysql_rm_table_part2(THD *thd, TABLE { TABLE *locked_table; abort_locked_tables(thd, db, table->table_name); + table->deleting= TRUE; remove_table_from_cache(thd, db, table->table_name, RTFC_WAIT_OTHER_THREAD_FLAG | - RTFC_CHECK_KILLED_FLAG); + RTFC_CHECK_KILLED_FLAG, FALSE); /* If the table was used in lock tables, remember it so that unlock_table_names can free it @@ -4213,9 +4215,10 @@ void wait_while_table_is_used(THD *thd,T /* Wait until all there are no other threads that has this table open */ remove_table_from_cache(thd, table->s->db.str, table->s->table_name.str, - RTFC_WAIT_OTHER_THREAD_FLAG); + RTFC_WAIT_OTHER_THREAD_FLAG, FALSE); /* extra() call must come only after all instances above are closed */ - VOID(table->file->extra(function)); + if (function != HA_EXTRA_NOT_USED) + VOID(table->file->extra(function)); DBUG_VOID_RETURN; } @@ -4717,7 +4720,7 @@ static bool mysql_admin_table(THD* thd, remove_table_from_cache(thd, table->table->s->db.str, table->table->s->table_name.str, RTFC_WAIT_OTHER_THREAD_FLAG | - RTFC_CHECK_KILLED_FLAG); + RTFC_CHECK_KILLED_FLAG, FALSE); thd->exit_cond(old_message); DBUG_EXECUTE_IF("wait_in_mysql_admin_table", wait_for_kill_signal(thd);); if (thd->killed) @@ -4975,7 +4978,8 @@ send_result_message: { pthread_mutex_lock(&LOCK_open); remove_table_from_cache(thd, table->table->s->db.str, - table->table->s->table_name.str, RTFC_NO_FLAG); + table->table->s->table_name.str, + RTFC_NO_FLAG, FALSE); pthread_mutex_unlock(&LOCK_open); } /* May be something modified consequently we have to invalidate cache */ @@ -6738,7 +6742,9 @@ view_err: from concurrent DDL statements. */ VOID(pthread_mutex_lock(&LOCK_open)); - wait_while_table_is_used(thd, table, HA_EXTRA_FORCE_REOPEN); + wait_while_table_is_used(thd, table, + thd->locked_tables ? HA_EXTRA_NOT_USED : + HA_EXTRA_FORCE_REOPEN); VOID(pthread_mutex_unlock(&LOCK_open)); DBUG_EXECUTE_IF("sleep_alter_enable_indexes", my_sleep(6000000);); error= table->file->ha_enable_indexes(HA_KEY_SWITCH_NONUNIQ_SAVE); @@ -6746,7 +6752,9 @@ view_err: break; case DISABLE: VOID(pthread_mutex_lock(&LOCK_open)); - wait_while_table_is_used(thd, table, HA_EXTRA_FORCE_REOPEN); + wait_while_table_is_used(thd, table, + thd->locked_tables ? HA_EXTRA_NOT_USED : + HA_EXTRA_FORCE_REOPEN); VOID(pthread_mutex_unlock(&LOCK_open)); error=table->file->ha_disable_indexes(HA_KEY_SWITCH_NONUNIQ_SAVE); /* COND_refresh will be signaled in close_thread_tables() */ @@ -7192,7 +7200,9 @@ view_err: else { VOID(pthread_mutex_lock(&LOCK_open)); - wait_while_table_is_used(thd, table, HA_EXTRA_FORCE_REOPEN); + wait_while_table_is_used(thd, table, + thd->locked_tables ? HA_EXTRA_NOT_USED : + HA_EXTRA_FORCE_REOPEN); VOID(pthread_mutex_unlock(&LOCK_open)); thd_proc_info(thd, "manage keys"); alter_table_manage_keys(table, table->file->indexes_are_disabled(), === modified file 'sql/table.cc' --- a/sql/table.cc 2010-01-15 15:27:55 +0000 +++ b/sql/table.cc 2010-02-10 19:06:24 +0000 @@ -1977,7 +1977,11 @@ int closefrm(register TABLE *table, bool DBUG_PRINT("enter", ("table: 0x%lx", (long) table)); if (table->db_stat) + { + if (table->s->deleting) + table->file->extra(HA_EXTRA_PREPARE_FOR_DROP); error=table->file->close(); + } my_free((char*) table->alias, MYF(MY_ALLOW_ZERO_PTR)); table->alias= 0; if (table->field) === modified file 'sql/table.h' --- a/sql/table.h 2010-01-15 15:27:55 +0000 +++ b/sql/table.h 2010-02-10 19:06:24 +0000 @@ -431,6 +431,7 @@ typedef struct st_table_share bool is_view; bool name_lock, replace_with_name_lock; bool waiting_on_cond; /* Protection against free */ + bool deleting; /* going to delete this table */ ulong table_map_id; /* for row-based replication */ ulonglong table_map_version; @@ -1379,7 +1380,7 @@ struct TABLE_LIST */ bool create; bool internal_tmp_table; - + bool deleting; /* going to delete this table */ /* View creation context. */ === modified file 'sql/udf_example.def' --- a/sql/udf_example.def 2007-10-29 14:01:40 +0000 +++ b/sql/udf_example.def 2010-01-29 18:42:22 +0000 @@ -3,8 +3,10 @@ VERSION 1.0 EXPORTS lookup lookup_init + lookup_deinit reverse_lookup reverse_lookup_init + reverse_lookup_deinit metaphon_init metaphon_deinit metaphon === modified file 'storage/maria/ft_maria.c' --- a/storage/maria/ft_maria.c 2007-07-02 17:45:15 +0000 +++ b/storage/maria/ft_maria.c 2010-01-27 21:53:08 +0000 @@ -22,8 +22,8 @@ #include "ma_ftdefs.h" FT_INFO *maria_ft_init_search(uint flags, void *info, uint keynr, - uchar *query, uint query_len, CHARSET_INFO *cs, - uchar *record) + uchar *query, size_t query_len, + CHARSET_INFO *cs, uchar *record) { FT_INFO *res; if (flags & FT_BOOL) === modified file 'storage/maria/ha_maria.cc' --- a/storage/maria/ha_maria.cc 2009-12-03 11:34:11 +0000 +++ b/storage/maria/ha_maria.cc 2010-02-12 14:21:13 +0000 @@ -2255,9 +2255,12 @@ int ha_maria::extra(enum ha_extra_functi extern_lock(F_UNLOCK) (which resets file->trn) followed by maria_close() without calling commit/rollback in between. If file->trn is not set we can't remove file->share from the transaction list in the extra() call. + + table->in_use is not set in the case this is a done as part of closefrm() + as part of drop table. */ - if (!file->trn && + if (file->s->now_transactional && !file->trn && table->in_use && (operation == HA_EXTRA_PREPARE_FOR_DROP || operation == HA_EXTRA_PREPARE_FOR_RENAME)) { @@ -3275,11 +3278,11 @@ static struct st_mysql_sys_var* system_v MYSQL_SYSVAR(block_size), MYSQL_SYSVAR(checkpoint_interval), MYSQL_SYSVAR(force_start_after_recovery_failures), - MYSQL_SYSVAR(page_checksum), MYSQL_SYSVAR(log_dir_path), MYSQL_SYSVAR(log_file_size), MYSQL_SYSVAR(log_purge_type), MYSQL_SYSVAR(max_sort_file_size), + MYSQL_SYSVAR(page_checksum), MYSQL_SYSVAR(pagecache_age_threshold), MYSQL_SYSVAR(pagecache_buffer_size), MYSQL_SYSVAR(pagecache_division_limit), === modified file 'storage/maria/ma_blockrec.c' --- a/storage/maria/ma_blockrec.c 2010-01-06 21:27:53 +0000 +++ b/storage/maria/ma_blockrec.c 2010-02-10 19:06:24 +0000 @@ -430,8 +430,9 @@ my_bool _ma_once_end_block_record(MARIA_ if (share->bitmap.file.file >= 0) { if (flush_pagecache_blocks(share->pagecache, &share->bitmap.file, - share->temporary ? FLUSH_IGNORE_CHANGED : - FLUSH_RELEASE)) + ((share->temporary || share->deleting) ? + FLUSH_IGNORE_CHANGED : + FLUSH_RELEASE))) res= 1; /* File must be synced as it is going out of the maria_open_list and so @@ -1688,7 +1689,8 @@ static my_bool get_head_or_tail_page(MAR if (!page_link.changed) goto crashed; - DBUG_ASSERT((res->buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) == page_type); + DBUG_ASSERT((uint) (res->buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) == + page_type); if (!(dir= find_free_position(page_type == HEAD_PAGE ? info : 0, res->buff, block_size, &res->rownr, &res->length, &res->empty_space))) === modified file 'storage/maria/ma_close.c' --- a/storage/maria/ma_close.c 2009-02-05 22:38:30 +0000 +++ b/storage/maria/ma_close.c 2010-02-10 19:06:24 +0000 @@ -79,7 +79,7 @@ int maria_close(register MARIA_HA *info) if ((*share->once_end)(share)) error= my_errno; if (flush_pagecache_blocks(share->pagecache, &share->kfile, - (share->temporary ? + ((share->temporary || share->deleting) ? FLUSH_IGNORE_CHANGED : FLUSH_RELEASE))) error= my_errno; @@ -177,6 +177,7 @@ int maria_close(register MARIA_HA *info) { (void) pthread_mutex_destroy(&share->intern_lock); (void) pthread_mutex_destroy(&share->close_lock); + (void) pthread_cond_destroy(&share->key_del_cond); my_free((uchar *)share, MYF(0)); /* If share cannot be freed, it's because checkpoint has previously === modified file 'storage/maria/ma_extra.c' --- a/storage/maria/ma_extra.c 2009-10-06 06:13:56 +0000 +++ b/storage/maria/ma_extra.c 2010-02-10 19:06:24 +0000 @@ -305,6 +305,12 @@ int maria_extra(MARIA_HA *info, enum ha_ pthread_mutex_unlock(&THR_LOCK_maria); break; case HA_EXTRA_PREPARE_FOR_DROP: + /* Signals about intent to delete this table */ + share->deleting= TRUE; + share->global_changed= FALSE; /* force writing changed flag */ + /* To force repair if reopened */ + _ma_mark_file_changed(info); + /* Fall trough */ case HA_EXTRA_PREPARE_FOR_RENAME: { my_bool do_flush= test(function != HA_EXTRA_PREPARE_FOR_DROP); === modified file 'storage/maria/ma_locking.c' --- a/storage/maria/ma_locking.c 2009-10-06 06:13:56 +0000 +++ b/storage/maria/ma_locking.c 2010-02-10 19:06:24 +0000 @@ -387,6 +387,9 @@ int _ma_test_if_changed(register MARIA_H open_count is not maintained on disk for temporary tables. */ +#define _MA_ALREADY_MARKED_FILE_CHANGED \ + ((share->state.changed & STATE_CHANGED) && share->global_changed) + int _ma_mark_file_changed(MARIA_HA *info) { uchar buff[3]; @@ -394,8 +397,6 @@ int _ma_mark_file_changed(MARIA_HA *info int error= 1; DBUG_ENTER("_ma_mark_file_changed"); -#define _MA_ALREADY_MARKED_FILE_CHANGED \ - ((share->state.changed & STATE_CHANGED) && share->global_changed) if (_MA_ALREADY_MARKED_FILE_CHANGED) DBUG_RETURN(0); pthread_mutex_lock(&share->intern_lock); /* recheck under mutex */ === modified file 'storage/maria/ma_recovery.c' --- a/storage/maria/ma_recovery.c 2009-10-26 11:35:42 +0000 +++ b/storage/maria/ma_recovery.c 2010-02-10 19:06:24 +0000 @@ -312,11 +312,14 @@ int maria_apply_log(LSN from_lsn, enum m now= my_getsystime(); in_redo_phase= TRUE; + trnman_init(max_trid_in_control_file); if (run_redo_phase(from_lsn, apply)) { ma_message_no_user(0, "Redo phase failed"); + trnman_destroy(); goto err; } + trnman_destroy(); if ((uncommitted_trans= end_of_redo_phase(should_run_undo_phase)) == (uint)-1) === modified file 'storage/maria/maria_def.h' --- a/storage/maria/maria_def.h 2009-11-29 23:08:56 +0000 +++ b/storage/maria/maria_def.h 2010-02-10 19:06:24 +0000 @@ -390,6 +390,7 @@ typedef struct st_maria_share my_bool now_transactional; my_bool have_versioning; my_bool key_del_used; /* != 0 if key_del is locked */ + my_bool deleting; /* we are going to delete this table */ #ifdef THREAD THR_LOCK lock; void (*lock_restore_status)(void *); === modified file 'storage/myisam/ft_boolean_search.c' --- a/storage/myisam/ft_boolean_search.c 2010-01-15 15:27:55 +0000 +++ b/storage/myisam/ft_boolean_search.c 2010-01-28 14:49:14 +0000 @@ -299,7 +299,7 @@ static int ftb_parse_query_internal(MYSQ } -static int _ftb_parse_query(FTB *ftb, uchar *query, uint len, +static int _ftb_parse_query(FTB *ftb, uchar *query, mysql_ft_size_t len, struct st_mysql_ftparser *parser) { MYSQL_FTPARSER_PARAM *param; @@ -321,7 +321,7 @@ static int _ftb_parse_query(FTB *ftb, uc param->mysql_add_word= ftb_query_add_word; param->mysql_ftparam= (void *)&ftb_param; param->cs= ftb->charset; - param->doc= (char*) query; + param->doc= query; param->length= len; param->flags= 0; param->mode= MYSQL_FTPARSER_FULL_BOOLEAN_INFO; @@ -540,7 +540,7 @@ static void _ftb_init_index_search(FT_IN FT_INFO * ft_init_boolean_search(MI_INFO *info, uint keynr, uchar *query, - uint query_len, CHARSET_INFO *cs) + mysql_ft_size_t query_len, CHARSET_INFO *cs) { FTB *ftb; FTB_EXPR *ftbe; @@ -679,8 +679,9 @@ static int ftb_check_phrase_internal(MYS -1 is returned if error occurs. */ -static int _ftb_check_phrase(FTB *ftb, const uchar *document, uint len, - FTB_EXPR *ftbe, struct st_mysql_ftparser *parser) +static int _ftb_check_phrase(FTB *ftb, const uchar *document, + mysql_ft_size_t len, + FTB_EXPR *ftbe, struct st_mysql_ftparser *parser) { MY_FTB_PHRASE_PARAM ftb_param; MYSQL_FTPARSER_PARAM *param; @@ -701,7 +702,7 @@ static int _ftb_check_phrase(FTB *ftb, c param->mysql_add_word= ftb_phrase_add_word; param->mysql_ftparam= (void *)&ftb_param; param->cs= ftb->charset; - param->doc= (char *) document; + param->doc= document; param->length= len; param->flags= 0; param->mode= MYSQL_FTPARSER_WITH_STOPWORDS; @@ -999,7 +1000,7 @@ float ft_boolean_find_relevance(FT_INFO { if (!ftsi.pos) continue; - param->doc= (char *)ftsi.pos; + param->doc= ftsi.pos; param->length= ftsi.len; if (unlikely(parser->parse(param))) return 0; === modified file 'storage/myisam/ft_myisam.c' --- a/storage/myisam/ft_myisam.c 2007-07-02 17:45:15 +0000 +++ b/storage/myisam/ft_myisam.c 2010-01-27 21:53:08 +0000 @@ -23,8 +23,8 @@ #include "ftdefs.h" FT_INFO *ft_init_search(uint flags, void *info, uint keynr, - uchar *query, uint query_len, CHARSET_INFO *cs, - uchar *record) + uchar *query, size_t query_len, + CHARSET_INFO *cs, uchar *record) { FT_INFO *res; if (flags & FT_BOOL) === modified file 'storage/myisam/ft_nlq_search.c' --- a/storage/myisam/ft_nlq_search.c 2009-11-30 13:36:06 +0000 +++ b/storage/myisam/ft_nlq_search.c 2010-01-27 21:53:08 +0000 @@ -204,7 +204,8 @@ static int FT_DOC_cmp(void *unused __att FT_INFO *ft_init_nlq_search(MI_INFO *info, uint keynr, uchar *query, - uint query_len, uint flags, uchar *record) + mysql_ft_size_t query_len, uint flags, + uchar *record) { TREE wtree; ALL_IN_ONE aio; === modified file 'storage/myisam/ft_parser.c' --- a/storage/myisam/ft_parser.c 2009-11-30 13:36:06 +0000 +++ b/storage/myisam/ft_parser.c 2010-01-28 14:49:14 +0000 @@ -175,7 +175,7 @@ uchar ft_get_word(CHARSET_INFO *cs, cons if ((param->trunc=(doc<end && *doc == FTB_TRUNC))) doc++; - if (((length >= ft_min_word_len && !is_stopword((char*) word->pos, + if (((length >= ft_min_word_len && !is_stopword(word->pos, word->len)) || param->trunc) && length < ft_max_word_len) { @@ -298,13 +298,13 @@ static int ft_parse_internal(MYSQL_FTPAR DBUG_ENTER("ft_parse_internal"); while (ft_simple_get_word(wtree->custom_arg, &doc, end, &w, TRUE)) - if (param->mysql_add_word(param, (char*) w.pos, w.len, 0)) + if (param->mysql_add_word(param, w.pos, w.len, 0)) DBUG_RETURN(1); DBUG_RETURN(0); } -int ft_parse(TREE *wtree, const uchar *doc, int doclen, +int ft_parse(TREE *wtree, const uchar *doc, mysql_ft_size_t doclen, struct st_mysql_ftparser *parser, MYSQL_FTPARSER_PARAM *param, MEM_ROOT *mem_root) { @@ -319,7 +319,7 @@ int ft_parse(TREE *wtree, const uchar *d param->mysql_add_word= ft_add_word; param->mysql_ftparam= &my_param; param->cs= wtree->custom_arg; - param->doc= (char*) doc; + param->doc= doc; param->length= doclen; param->mode= MYSQL_FTPARSER_SIMPLE_MODE; DBUG_RETURN(parser->parse(param)); === modified file 'storage/myisam/ft_stopwords.c' --- a/storage/myisam/ft_stopwords.c 2009-11-29 23:08:56 +0000 +++ b/storage/myisam/ft_stopwords.c 2010-01-28 14:49:14 +0000 @@ -20,8 +20,8 @@ typedef struct st_ft_stopwords { - const char * pos; - uint len; + const uchar* pos; + size_t len; } FT_STOPWORD; static TREE *stopwords3=NULL; @@ -30,15 +30,15 @@ static int FT_STOPWORD_cmp(void* cmp_arg FT_STOPWORD *w1, FT_STOPWORD *w2) { return ha_compare_text(default_charset_info, - (uchar *)w1->pos,w1->len, - (uchar *)w2->pos,w2->len,0,0); + w1->pos, w1->len, + w2->pos, w2->len, 0, 0); } static void FT_STOPWORD_free(FT_STOPWORD *w, TREE_FREE action, void *arg __attribute__((unused))) { if (action == free_free) - my_free((uchar*) w->pos, MYF(0)); + my_free((void*) w->pos, MYF(0)); } static int ft_add_stopword(const char *w) === modified file 'storage/myisam/ftdefs.h' --- a/storage/myisam/ftdefs.h 2009-11-29 23:08:56 +0000 +++ b/storage/myisam/ftdefs.h 2010-01-27 21:53:08 +0000 @@ -97,8 +97,8 @@ typedef struct st_ft_word { const uchar *pos; - uint len; double weight; + size_t len; } FT_WORD; int is_stopword(const uchar *word, size_t len); @@ -111,7 +111,8 @@ uchar ft_simple_get_word(CHARSET_INFO *, FT_WORD *, my_bool); typedef struct _st_ft_seg_iterator { - uint num, len; + uint num; + mysql_ft_size_t len; HA_KEYSEG *seg; const uchar *rec, *pos; } FT_SEG_ITERATOR; @@ -128,8 +129,9 @@ FT_WORD * _mi_ft_parserecord(MI_INFO *, uint _mi_ft_parse(TREE *, MI_INFO *, uint, const uchar *, MYSQL_FTPARSER_PARAM *, MEM_ROOT *); -FT_INFO *ft_init_nlq_search(MI_INFO *, uint, uchar *, uint, uint, uchar *); -FT_INFO *ft_init_boolean_search(MI_INFO *, uint, uchar *, uint, CHARSET_INFO *); +FT_INFO *ft_init_nlq_search(MI_INFO *, uint, uchar *, mysql_ft_size_t, uint, + uchar *); +FT_INFO *ft_init_boolean_search(MI_INFO *, uint, uchar *, mysql_ft_size_t, CHARSET_INFO *); extern const struct _ft_vft _ft_vft_nlq; int ft_nlq_read_next(FT_INFO *, char *); === modified file 'storage/myisam/mi_close.c' --- a/storage/myisam/mi_close.c 2009-09-07 20:50:10 +0000 +++ b/storage/myisam/mi_close.c 2010-02-10 19:06:24 +0000 @@ -64,8 +64,9 @@ int mi_close(register MI_INFO *info) if (share->kfile >= 0) abort();); if (share->kfile >= 0 && flush_key_blocks(share->key_cache, share->kfile, - share->temporary ? FLUSH_IGNORE_CHANGED : - FLUSH_RELEASE)) + ((share->temporary || share->deleting) ? + FLUSH_IGNORE_CHANGED : + FLUSH_RELEASE))) error=my_errno; if (share->kfile >= 0) { === modified file 'storage/myisam/mi_extra.c' --- a/storage/myisam/mi_extra.c 2009-10-06 06:13:56 +0000 +++ b/storage/myisam/mi_extra.c 2010-02-10 19:06:24 +0000 @@ -256,8 +256,13 @@ int mi_extra(MI_INFO *info, enum ha_extr share->last_version= 0L; /* Impossible version */ pthread_mutex_unlock(&THR_LOCK_myisam); break; - case HA_EXTRA_PREPARE_FOR_RENAME: case HA_EXTRA_PREPARE_FOR_DROP: + /* Signals about intent to delete this table */ + share->deleting= TRUE; + share->global_changed= FALSE; /* force writing changed flag */ + _mi_mark_file_changed(info); + /* Fall trough */ + case HA_EXTRA_PREPARE_FOR_RENAME: pthread_mutex_lock(&THR_LOCK_myisam); share->last_version= 0L; /* Impossible version */ pthread_mutex_lock(&share->intern_lock); === modified file 'storage/myisam/mi_open.c' --- a/storage/myisam/mi_open.c 2009-12-03 11:19:05 +0000 +++ b/storage/myisam/mi_open.c 2010-02-10 19:06:24 +0000 @@ -58,6 +58,8 @@ MI_INFO *test_if_reopen(char *filename) { MI_INFO *info=(MI_INFO*) pos->data; MYISAM_SHARE *share=info->s; + DBUG_ASSERT(strcmp(share->unique_file_name,filename) || + share->last_version); if (!strcmp(share->unique_file_name,filename) && share->last_version) return info; } === modified file 'storage/myisam/myisamdef.h' --- a/storage/myisam/myisamdef.h 2009-12-03 11:34:11 +0000 +++ b/storage/myisam/myisamdef.h 2010-02-10 19:06:24 +0000 @@ -221,6 +221,7 @@ typedef struct st_mi_isam_share my_bool changed, /* If changed since lock */ global_changed, /* If changed since open */ not_flushed, temporary, delay_key_write, concurrent_insert; + my_bool deleting; /* we are going to delete this table */ #ifdef THREAD THR_LOCK lock; pthread_mutex_t intern_lock; /* Locking for use with _locking */ === modified file 'storage/xtradb/buf/buf0buf.c' --- a/storage/xtradb/buf/buf0buf.c 2010-01-06 12:00:14 +0000 +++ b/storage/xtradb/buf/buf0buf.c 2010-01-28 11:35:10 +0000 @@ -76,10 +76,10 @@ inline void _increment_page_get_statisti block->page.offset, DPAH_SIZE << 3); block_hash_byte = block_hash >> 3; block_hash_offset = (byte) block_hash & 0x07; - if (block_hash_byte < 0 || block_hash_byte >= DPAH_SIZE) - fprintf(stderr, "!!! block_hash_byte = %lu block_hash_offset = %lu !!!\n", block_hash_byte, block_hash_offset); - if (block_hash_offset < 0 || block_hash_offset > 7) - fprintf(stderr, "!!! block_hash_byte = %lu block_hash_offset = %lu !!!\n", block_hash_byte, block_hash_offset); + if (block_hash_byte >= DPAH_SIZE) + fprintf(stderr, "!!! block_hash_byte = %lu block_hash_offset = %lu !!!\n", (unsigned long) block_hash_byte, (unsigned long) block_hash_offset); + if (block_hash_offset > 7) + fprintf(stderr, "!!! block_hash_byte = %lu block_hash_offset = %lu !!!\n", (unsigned long) block_hash_byte, (unsigned long) block_hash_offset); if ((trx->distinct_page_access_hash[block_hash_byte] & ((byte) 0x01 << block_hash_offset)) == 0) trx->distinct_page_access++; trx->distinct_page_access_hash[block_hash_byte] |= (byte) 0x01 << block_hash_offset; === modified file 'storage/xtradb/handler/i_s.cc' --- a/storage/xtradb/handler/i_s.cc 2010-01-15 15:58:25 +0000 +++ b/storage/xtradb/handler/i_s.cc 2010-01-28 11:35:10 +0000 @@ -2992,7 +2992,7 @@ i_s_innodb_admin_command_fill( char** query_str; char* ptr; char quote = '\0'; - char* command_head = "XTRA_"; + const char* command_head = "XTRA_"; DBUG_ENTER("i_s_innodb_admin_command_fill"); === modified file 'support-files/compiler_warnings.supp' --- a/support-files/compiler_warnings.supp 2010-01-06 21:27:53 +0000 +++ b/support-files/compiler_warnings.supp 2010-01-28 14:49:14 +0000 @@ -4,6 +4,7 @@ # integer.cpp: .*control reaches end of non-void function.*: 1288-1427 +integer.cpp: .*no return statement in function returning non-void.*: 1288-1427 DictTabInfo.cpp : .*invalid access to non-static.* DictTabInfo.cpp : .*macro was used incorrectly.* DbdihMain.cpp : .*unused variable.* : 6666-7013 @@ -27,7 +28,17 @@ _flex_tmp.c: .*not enough actual paramet pars0lex.l: .*conversion from 'ulint' to 'int', possible loss of data.* btr/btr0cur\.c: .*value computed is not used.*: 3175-3375 include/buf0buf\.ic: unused parameter ‘mtr’ +fil/fil0fil\.c: pointer targets in passing argument.*differ in signedness fil/fil0fil\.c: comparison between signed and unsigned : 3100-3199 +fil/fil0fil\.c: unused parameter +log/log0recv\.c: unused variable +os/os0file\.c: unused parameter +os/os0file\.c: pointer targets in assignment differ in signedness +handler/i_s\.cc: unused variable +sync/sync0rw\.c: unused parameter +sync/sync0sync\.c: unused parameter +sync/sync0sync\.c: unused variable +ut/ut0ut\.c: ignoring return value of # # bdb is not critical to keep up to date @@ -50,6 +61,11 @@ db_vrfy.c : .*comparison is always false .*/extra/libevent/select\.c : .*comparison between signed and unsigned.* : 270-280 # +# Ignore warnings from system libraries +# +/usr/share/aclocal/audiofile.m4 : .* + +# # Ignore all conversion warnings on windows 64 # (Is safe as we are not yet supporting strings >= 2G) # @@ -87,10 +103,15 @@ storage/maria/ma_pagecache.c: .*'info_ch # I think these are due to mix of C and C++. # storage/pbxt/ : typedef.*was ignored in this declaration +ha_pbxt\.cc : variable.*might be clobbered by.*longjmp # # Yassl include/runtime.hpp: .*pure_error.* +.*/extra/yassl/taocrypt/.*: comparison with string literal +.*/extra/yassl/taocrypt/src/blowfish\.cpp: array subscript is above array bounds +.*/extra/yassl/taocrypt/src/file\.cpp: ignoring return value +.*/extra/yassl/taocrypt/src/integer\.cpp: control reaches end of non-void function # # Groff warnings on OpenSUSE. @@ -102,6 +123,8 @@ include/runtime.hpp: .*pure_error.* # listener.cc : .*conversion from 'SOCKET' to 'int'.* net_serv.cc : .*conversion from 'SOCKET' to 'int'.* +set_var.cc: right-hand operand of comma has no effect : 1000-1400 + # allow a little moving space for the warning below mi_packrec.c : .*result of 32-bit shift implicitly converted to 64 bits.* : 560-600 === modified file 'unittest/mysys/waiting_threads-t.c' --- a/unittest/mysys/waiting_threads-t.c 2009-09-15 10:46:35 +0000 +++ b/unittest/mysys/waiting_threads-t.c 2010-01-28 14:49:14 +0000 @@ -258,7 +258,7 @@ void do_tests() #define test_kill_strategy(X) \ diag("kill strategy: " #X); \ DBUG_EXECUTE("reset_file", \ - { rewind(DBUG_FILE); ftruncate(fileno(DBUG_FILE), 0); }); \ + { rewind(DBUG_FILE); (void) ftruncate(fileno(DBUG_FILE), 0); }); \ DBUG_PRINT("info", ("kill strategy: " #X)); \ kill_strategy=X; \ do_one_test(); === modified file 'vio/viossl.c' --- a/vio/viossl.c 2008-02-07 08:55:49 +0000 +++ b/vio/viossl.c 2010-01-29 10:42:31 +0000 @@ -74,8 +74,11 @@ report_errors(SSL* ssl) } if (ssl) - DBUG_PRINT("error", ("error: %s", - ERR_error_string(SSL_get_error(ssl, l), buf))); + { + int error= SSL_get_error(ssl, l); + DBUG_PRINT("error", ("error: %s (%d)", + ERR_error_string(error, buf), error)); + } DBUG_PRINT("info", ("socket_errno: %d", socket_errno)); DBUG_VOID_RETURN; @@ -174,16 +177,17 @@ void vio_ssl_delete(Vio *vio) static int ssl_do(struct st_VioSSLFd *ptr, Vio *vio, long timeout, - int (*connect_accept_func)(SSL*)) + int (*connect_accept_func)(SSL*), char *error_string) { SSL *ssl; my_bool unused; my_bool was_blocking; - DBUG_ENTER("ssl_do"); DBUG_PRINT("enter", ("ptr: 0x%lx, sd: %d ctx: 0x%lx", (long) ptr, vio->sd, (long) ptr->ssl_context)); + error_string[0]= 0; + /* Set socket to blocking if not already set */ vio_blocking(vio, 1, &was_blocking); @@ -191,6 +195,7 @@ static int ssl_do(struct st_VioSSLFd *pt { DBUG_PRINT("error", ("SSL_new failure")); report_errors(ssl); + strmov(error_string, "SSL_new failed"); vio_blocking(vio, was_blocking, &unused); DBUG_RETURN(1); } @@ -203,6 +208,7 @@ static int ssl_do(struct st_VioSSLFd *pt { DBUG_PRINT("error", ("SSL_connect/accept failure")); report_errors(ssl); + ERR_error_string(SSL_get_error(ssl, 0), error_string); SSL_free(ssl); vio_blocking(vio, was_blocking, &unused); DBUG_RETURN(1); @@ -250,17 +256,19 @@ static int ssl_do(struct st_VioSSLFd *pt } -int sslaccept(struct st_VioSSLFd *ptr, Vio *vio, long timeout) +int sslaccept(struct st_VioSSLFd *ptr, Vio *vio, long timeout, + char *error_string) { DBUG_ENTER("sslaccept"); - DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_accept)); + DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_accept, error_string)); } -int sslconnect(struct st_VioSSLFd *ptr, Vio *vio, long timeout) +int sslconnect(struct st_VioSSLFd *ptr, Vio *vio, long timeout, + char *error_string) { DBUG_ENTER("sslconnect"); - DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_connect)); + DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_connect, error_string)); }

1 0

[Maria-developers] Rev 2763: Fix Item_subselect::update_used_tables() and fix_after_pullout() to work with in file:///home/psergey/dev/maria-5.3-subqueries-r7/
by Sergey Petrunya 20 Feb '10

20 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r7/ ------------------------------------------------------------ revno: 2763 revision-id: psergey(a)askmonty.org-20100220082329-9esvom4n6mpgeqvk parent: psergey(a)askmonty.org-20100217225459-4mum80lxuu3cwmjz committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r7 timestamp: Sat 2010-02-20 11:23:29 +0300 message: Fix Item_subselect::update_used_tables() and fix_after_pullout() to work with prepared statements: re-collect list of upper refs on every PS re-execution. === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-02-17 21:59:41 +0000 +++ b/sql/item_subselect.cc 2010-02-20 08:23:29 +0000 @@ -167,6 +167,11 @@ { done_first_fix_fields= TRUE; inside_first_fix_fields= TRUE; + upper_refs.empty(); + /* + psergey-todo: remove _first_fix_fields calls, we need changes on every + execution + */ } eliminated= FALSE; @@ -181,6 +186,8 @@ changed= 1; inside_first_fix_fields= FALSE; + done_first_fix_fields= FALSE; + if (!res) { // all transformation is done (used by prepared statements)

1 0

[Maria-developers] Rev 2762: MWL#68 Subquery optimization: Efficient NOT IN execution with NULLs in file:///home/tsk/mprog/src/5.3-mwl68/
by timour＠askmonty.org 19 Feb '10

19 Feb '10

At file:///home/tsk/mprog/src/5.3-mwl68/ ------------------------------------------------------------ revno: 2762 revision-id: timour(a)askmonty.org-20100219215557-g1uohrur8bopfv7n parent: psergey(a)askmonty.org-20100217215941-5uxponk64p55xg4a committer: timour(a)askmonty.org branch nick: 5.3-mwl68 timestamp: Fri 2010-02-19 23:55:57 +0200 message: MWL#68 Subquery optimization: Efficient NOT IN execution with NULLs This patch implements correct NULL semantics for materialized subquery execution. The implementation has the following properties and main limitations: - It passes all query result tests, but fails a number of EXPLAIN tests because of changed plans. - The EXPLAIN output for partial matching is not decided yet. - It works only when all necessary indexes fit into main memory. Notice that these are not the general B-tree/Hash indexes, but instead much more compact ones, therefore this limitation may not be a problem in many practical cases. - It doesn't contain specialized tests. - In several places the implementation uses methods that are modified copies of other similar methods. These cases need to be refactored to avoid code duplication. - Add a test if the predicate is top-level just before deciding on partial matching. If it is top-level, use a more efficient exec method (index lookup). - Add sorting of indexes according to their selectivity. The code is almost there. - Needs more comments, and to sync existing ones with the implementation. Diff too large for email (2568 lines, the limit is 1000).

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2819: LPBUG#523593: Merge
by noreply＠launchpad.net 19 Feb '10

19 Feb '10

Merge authors: Sergey Petrunia (sergefp) ------------------------------------------------------------ revno: 2819 [merge] committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.1-push timestamp: Fri 2010-02-19 17:13:49 +0300 message: LPBUG#523593: Merge modified: mysql-test/r/table_elim.result mysql-test/t/table_elim.test sql/opt_table_elimination.cc -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] Rev 9: Added wrapper scripts for running all sysbench >= v0.5 test in file:///Users/hakan/work/monty_program/mariadb-tools/
by Hakan Kuecuekyilmaz 19 Feb '10

19 Feb '10

At file:///Users/hakan/work/monty_program/mariadb-tools/ ------------------------------------------------------------ revno: 9 revision-id: hakan(a)askmonty.org-20100219052704-8nnlzsbu0nslnxba parent: hakan(a)askmonty.org-20100204111840-0d9u3nyio2h8tavi committer: Hakan Kuecuekyilmaz <hakan(a)askmonty.org> branch nick: mariadb-tools timestamp: Fri 2010-02-19 06:27:04 +0100 message: Added wrapper scripts for running all sysbench >= v0.5 test with MariaDB and MySQL. === added directory 'sysbench' === added file 'sysbench/README' --- a/sysbench/README 1970-01-01 00:00:00 +0000 +++ b/sysbench/README 2010-02-19 05:27:04 +0000 @@ -0,0 +1,1 @@ +Automation scripts for sysbench >= v.05 === added file 'sysbench/run-sysbench.sh' --- a/sysbench/run-sysbench.sh 1970-01-01 00:00:00 +0000 +++ b/sysbench/run-sysbench.sh 2010-02-19 05:27:04 +0000 @@ -0,0 +1,275 @@ +#!/bin/bash +# +# Run sysbench tests with MariaDB and MySQL +# +# Note: Do not run this script with root privileges. +# We use killall -9, which can cause severe side effects! +# +# Hakan Kuecuekyilmaz <hakan at askmonty dot org> 2010-02-19. +# + +# +# Do not run this script as root! +# +RUN_BY=$(whoami) +if [ x"root" = x"$RUN_BY" ];then + echo '[ERROR]: Do not run this script as root!' + echo ' Exiting.' + + exit 1 +fi + +# +# Variables. +# +TEMP_DIR='/tmp' +DATA_DIR="${TEMP_DIR}/data" +MY_SOCKET="${TEMP_DIR}/mysql.sock" +MYSQLADMIN_OPTIONS="--no-defaults -uroot --socket=$MY_SOCKET" +MYSQL_OPTIONS="--no-defaults \ + --skip-grant-tables \ + --language=./sql/share/english \ + --datadir=$DATA_DIR \ + --tmpdir=$TEMP_DIR \ + --socket=$MY_SOCKET \ + --table_open_cache=512 \ + --thread_cache=512 \ + --query_cache_size=0 \ + --query_cache_type=0 \ + --innodb_data_home_dir=$DATA_DIR \ + --innodb_data_file_path=ibdata1:128M:autoextend \ + --innodb_log_group_home_dir=$DATA_DIR \ + --innodb_buffer_pool_size=1024M \ + --innodb_additional_mem_pool_size=32M \ + --innodb_log_file_size=256M \ + --innodb_log_buffer_size=16M \ + --innodb_flush_log_at_trx_commit=1 \ + --innodb_lock_wait_timeout=50 \ + --innodb_doublewrite=0 \ + --innodb_flush_method=O_DIRECT \ + --innodb_thread_concurrency=0 \ + --innodb_max_dirty_pages_pct=80" + +NUM_THREADS="1 4 8 16 32 64 128" +TABLE_SIZE=2000000 +RUN_TIME=300 +SYSBENCH_TESTS="delete.lua \ + insert.lua \ + oltp_complex_ro.lua \ + oltp_complex_rw.lua \ + oltp_simple.lua \ + select.lua \ + update_index.lua \ + update_non_index.lua" +SYSBENCH_OPTIONS="--oltp-table-size=$TABLE_SIZE \ + --max-time=$RUN_TIME \ + --max-requests=0 \ + --mysql-table-engine=InnoDB \ + --mysql-user=root \ + --mysql-engine-trx=yes" + +PRODUCTS='MariaDB MySQL' + +# Timeout in seconds for waiting for mysqld to start. +TIMEOUT=100 + +# +# Files +# +MARIADB_BUILD_LOG='/tmp/mariadb_build.log' +MYSQL_BUILD_LOG='/tmp/mysql_build.log' + +# +# Directories. +# +BASE="${HOME}/work" +MARIADB_LOCAL_MASTER="${BASE}/monty_program/maria-local-master" +MARIADB_WORK="${BASE}/monty_program/maria" +MYSQL_LOCAL_MASTER="${BASE}/mysql/mysql-server-local-master" +MYSQL_WORK="${BASE}/mysql/mysql-server" +TEST_DIR="${BASE}/monty_program/sysbench/sysbench/tests/db" +RESULT_DIR="${BASE}/sysbench-results" + +# +# Binaries. +# +MYSQLADMIN='./client/mysqladmin' +SYSBENCH='/usr/local/bin/sysbench' +BZR='/usr/local/bin/bzr' + +# +# Refresh repositories. +# +echo "[$(date "+%Y-%m-%d %H:%M:%S")] Refreshing source repositories." +rm -rf $MARIADB_WORK +if [ ! -d $MARIADB_LOCAL_MASTER ]; then + echo "[ERROR]: Local master of MariaDB does not exist." + echo " Please make a initial branch from lp:maria" + echo " Exiting." + exit 1 +else + cd $MARIADB_LOCAL_MASTER + echo "Pulling latest MariaDB sources." + $BZR pull + if [ $? != 0 ]; then + echo "[ERROR]: $BZR pull for $MARIADB_LOCAL_MASTER failed" + echo " Please check your bzr setup" + echo " Exiting." + exit 1 + fi + + echo "Branching MariaDB working directory." + $BZR branch $MARIADB_LOCAL_MASTER $MARIADB_WORK + if [ $? != 0 ]; then + echo "[ERROR]: $BZR branch of $MARIADB_LOCAL_MASTER failed" + echo " Please check your bzr setup" + echo " Exiting." + exit 1 + fi +fi + +rm -rf $MYSQL_WORK +if [ ! -d $MYSQL_LOCAL_MASTER ]; then + echo "[ERROR]: Local master of MySQL does not exist." + echo " Please make a initial branch from lp:mysql-server" + echo " Exiting." + exit 1 +else + cd $MYSQL_LOCAL_MASTER + echo "Pulling latest MySQL sources." + $BZR pull + if [ $? != 0 ]; then + echo "[ERROR]: $BZR pull for $MYSQL_LOCAL_MASTER failed" + echo " Please check your bzr setup" + echo " Exiting." + exit 1 + fi + + echo "Branching MySQL working directory." + $BZR branch $MYSQL_LOCAL_MASTER $MYSQL_WORK + if [ $? != 0 ]; then + echo "[ERROR]: $BZR branch of $MYSQL_LOCAL_MASTER failed" + echo " Please check your bzr setup" + echo " Exiting." + exit 1 + fi +fi + +echo "[$(date "+%Y-%m-%d %H:%M:%S")] Done refreshing source repositories." + + +# +# TODO: Add platform detection and choose proper build script. +# +echo "[$(date "+%Y-%m-%d %H:%M:%S")] Starting to compile." + +echo "[$(date "+%Y-%m-%d %H:%M:%S")] Compiling MariaDB." +cd $MARIADB_WORK +BUILD/compile-amd64-max > $MARIADB_BUILD_LOG 2>&1 +if [ $? != 0 ]; then + echo "[ERROR]: Build of $MARIADB_WORK failed" + echo " Please check the log at $MARIDB_BUILD_LOG" + echo " Exiting." + exit 1 +fi +echo "[$(date "+%Y-%m-%d %H:%M:%S")] Finnished compiling MariaDB." + +echo "[$(date "+%Y-%m-%d %H:%M:%S")] Compiling MySQL." +cd $MYSQL_WORK +BUILD/compile-amd64-max > $MYSQL_BUILD_LOG 2>&1 +if [ $? != 0 ]; then + echo "[ERROR]: Build of $MYSQL_WORK failed" + echo " Please check the log at $MYSQL_BUILD_LOG" + echo " Exiting." + exit 1 +fi +echo "[$(date "+%Y-%m-%d %H:%M:%S")] Finnished compiling MySQL." +echo "[$(date "+%Y-%m-%d %H:%M:%S")] Finnished compiling." + +# +# Go to work. +# +echo "[$(date "+%Y-%m-%d %H:%M:%S")] Starting sysbench runs." + +# +# Prepare results directory. +# +if [ ! -d $RESULT_DIRS ]; then + echo "[NOTE]: $RESULT_DIRS did not exist." + echo " We are creating it for you!" + + mkdir $RESULT_DIRS +fi + +TODAY=$(date +%Y-%m-%d) +mkdir ${RESULT_DIR}/${TODAY} + +for PRODUCT in $PRODUCTS; do + mkdir ${RESULT_DIR}/${TODAY}/${PRODUCT} + + killall -9 mysqld + rm -rf $DATA_DIR + rm -f $MY_SOCKET + mkdir $DATA_DIR + + if [ x"$PRODUCT" = x"MariaDB" ];then + cd $MARIADB_WORK + else + cd $MYSQL_WORK + fi + + sql/mysqld $MYSQL_OPTIONS & + + j=0 + STARTED=-1 + while [ $j -le $TIMEOUT ] + do + $MYSQLADMIN $MYSQLADMIN_OPTIONS ping > /dev/null 2>&1 + if [ $? = 0 ]; then + STARTED=0 + + break + fi + + sleep 1 + j=$(($j + 1)) + done + + if [ $STARTED != 0 ]; then + echo '[ERROR]: Start of mysqld failed.' + echo ' Please check your error log.' + echo ' Exiting.' + + exit 1 + fi + + for SYSBENCH_TEST in $SYSBENCH_TESTS; do + mkdir ${RESULT_DIR}/${TODAY}/${PRODUCT}/${SYSBENCH_TEST} + + for THREADS in $NUM_THREADS; do + THIS_RESULT_DIR="${RESULT_DIR}/${TODAY}/${PRODUCT}/${SYSBENCH_TEST}/${THREADS}" + mkdir $THIS_RESULT_DIR + echo "[$(date "+%Y-%m-%d %H:%M:%S")] Running $SYSBENCH_TEST with $THREADS threads for $PRODUCT" + + $MYSQLADMIN $MYSQLADMIN_OPTIONS -f drop sbtest + $MYSQLADMIN $MYSQLADMIN_OPTIONS create sbtest + if [ $? != 0 ]; then + echo "[ERROR]: Create of sbtest database failed" + echo " Please check your setup." + echo " Exiting" + exit 1 + fi + + SYSBENCH_OPTIONS="$SYSBENCH_OPTIONS --num-threads=$THREADS --test=${TEST_DIR}/${SYSBENCH_TEST}" + $SYSBENCH $SYSBENCH_OPTIONS prepare + $SYSBENCH $SYSBENCH_OPTIONS run > ${THIS_RESULT_DIR}/result.txt 2>&1 + + done + done +done + +# +# We are done! +# +echo "[$(date "+%Y-%m-%d %H:%M:%S")] Finished sysbench runs." +echo " You can check your results."

1 0

[Maria-developers] Progress (by Alexi): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 18 Feb '10

18 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen, Serg CATEGORY.......: Server-Sprint TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 40 ESTIMATE.......: 15 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Alexi - Thu, 18 Feb 2010, 19:29)=-=- Worked 20 hours (alexi) Worked 20 hours and estimate 15 hours remain (original estimate unchanged). -=-=(Serg - Fri, 05 Feb 2010, 14:04)=-=- Observers changed: Knielsen,Serg -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Category updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Status updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +In-Progress -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 ------------------------------------------------------------ -=-=(View All Progress Notes, 24 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Progress (by Alexi): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 18 Feb '10

18 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen, Serg CATEGORY.......: Server-Sprint TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 40 ESTIMATE.......: 15 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Alexi - Thu, 18 Feb 2010, 19:29)=-=- Worked 20 hours (alexi) Worked 20 hours and estimate 15 hours remain (original estimate unchanged). -=-=(Serg - Fri, 05 Feb 2010, 14:04)=-=- Observers changed: Knielsen,Serg -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Category updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Status updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +In-Progress -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 ------------------------------------------------------------ -=-=(View All Progress Notes, 24 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Progress (by Alexi): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 18 Feb '10

18 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen, Serg CATEGORY.......: Server-Sprint TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 40 ESTIMATE.......: 15 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Alexi - Thu, 18 Feb 2010, 19:29)=-=- Worked 20 hours (alexi) Worked 20 hours and estimate 15 hours remain (original estimate unchanged). -=-=(Serg - Fri, 05 Feb 2010, 14:04)=-=- Observers changed: Knielsen,Serg -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Category updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Status updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +In-Progress -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 ------------------------------------------------------------ -=-=(View All Progress Notes, 24 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Progress (by Alexi): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 18 Feb '10

18 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen, Serg CATEGORY.......: Server-Sprint TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 40 ESTIMATE.......: 15 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Alexi - Thu, 18 Feb 2010, 19:29)=-=- Worked 20 hours (alexi) Worked 20 hours and estimate 15 hours remain (original estimate unchanged). -=-=(Serg - Fri, 05 Feb 2010, 14:04)=-=- Observers changed: Knielsen,Serg -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Category updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Status updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +In-Progress -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 ------------------------------------------------------------ -=-=(View All Progress Notes, 24 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Rev 2818: LPBUG#523593: Running RQG optimizer_no_subquery crashes MariaDB in file:///home/psergey/dev/maria-5.1/
by Sergey Petrunya 18 Feb '10

18 Feb '10

At file:///home/psergey/dev/maria-5.1/ ------------------------------------------------------------ revno: 2818 revision-id: psergey(a)askmonty.org-20100218120352-fn9i4z9611nrkps5 parent: monty(a)askmonty.org-20100212142113-wdv50xx19quursaf committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.1 timestamp: Thu 2010-02-18 15:03:52 +0300 message: LPBUG#523593: Running RQG optimizer_no_subquery crashes MariaDB - When analying multiple equalities, take into account that they may not have a single table field that belongs to one of the tables that we're trying to eliminate (and they are not useful for table elimination in that case) === modified file 'mysql-test/r/table_elim.result' --- a/mysql-test/r/table_elim.result 2009-09-16 19:05:03 +0000 +++ b/mysql-test/r/table_elim.result 2010-02-18 12:03:52 +0000 @@ -418,3 +418,49 @@ 1 SIMPLE t1 ALL NULL NULL NULL NULL 4 1 SIMPLE t2 range PRIMARY PRIMARY 4 NULL 2 Using where drop table t1, t2; +# +# LPBUG#523593: Running RQG optimizer_no_subquery crashes MariaDB +# +CREATE TABLE t1 ( +`pk` int(11) NOT NULL AUTO_INCREMENT, +`col_int_nokey` int(11) DEFAULT NULL, +`col_int_key` int(11) DEFAULT NULL, +`col_date_key` date DEFAULT NULL, +`col_date_nokey` date DEFAULT NULL, +`col_time_key` time DEFAULT NULL, +`col_time_nokey` time DEFAULT NULL, +`col_datetime_key` datetime DEFAULT NULL, +`col_datetime_nokey` datetime DEFAULT NULL, +`col_varchar_key` varchar(1) DEFAULT NULL, +`col_varchar_nokey` varchar(1) DEFAULT NULL, +PRIMARY KEY (`pk`), +KEY `col_int_key` (`col_int_key`), +KEY `col_date_key` (`col_date_key`), +KEY `col_time_key` (`col_time_key`), +KEY `col_datetime_key` (`col_datetime_key`), +KEY `col_varchar_key` (`col_varchar_key`,`col_int_key`) +); +CREATE TABLE t2 LIKE t1; +INSERT INTO t1 VALUES +(10,7,8,NULL,NULL,'01:27:35','01:27:35','2002-02-26 06:14:37','2002-02-26 06:14:37','v','v'), +(11,1,9,'2006-06-14','2006-06-14','19:48:31','19:48:31','1900-01-01 00:00:00','1900-01-01 00:00:00','r','r'); +INSERT INTO t2 SELECT * FROM t1; +SELECT table2.col_int_key AS field1 +FROM ( +t2 AS table1 +RIGHT OUTER JOIN +( +( t1 AS table2 STRAIGHT_JOIN +t1 AS table3 ON ( +(table3.col_varchar_nokey = table2.col_varchar_key ) AND +(table3.pk = table2.col_int_key)) +) +) ON +( +(table3.col_varchar_key = table2.col_varchar_key) OR +(table3.col_int_key = table2.pk) +) +) +HAVING field1 < 216; +field1 +DROP TABLE t1, t2; === modified file 'mysql-test/t/table_elim.test' --- a/mysql-test/t/table_elim.test 2009-09-16 19:05:03 +0000 +++ b/mysql-test/t/table_elim.test 2010-02-18 12:03:52 +0000 @@ -336,3 +336,54 @@ from t1 left join t2 on (t2.pk=3 and t2.b=3) or (t2.pk= 4 and t2.b=3); drop table t1, t2; + +--echo # +--echo # LPBUG#523593: Running RQG optimizer_no_subquery crashes MariaDB +--echo # + +CREATE TABLE t1 ( + `pk` int(11) NOT NULL AUTO_INCREMENT, + `col_int_nokey` int(11) DEFAULT NULL, + `col_int_key` int(11) DEFAULT NULL, + `col_date_key` date DEFAULT NULL, + `col_date_nokey` date DEFAULT NULL, + `col_time_key` time DEFAULT NULL, + `col_time_nokey` time DEFAULT NULL, + `col_datetime_key` datetime DEFAULT NULL, + `col_datetime_nokey` datetime DEFAULT NULL, + `col_varchar_key` varchar(1) DEFAULT NULL, + `col_varchar_nokey` varchar(1) DEFAULT NULL, + PRIMARY KEY (`pk`), + KEY `col_int_key` (`col_int_key`), + KEY `col_date_key` (`col_date_key`), + KEY `col_time_key` (`col_time_key`), + KEY `col_datetime_key` (`col_datetime_key`), + KEY `col_varchar_key` (`col_varchar_key`,`col_int_key`) +); +CREATE TABLE t2 LIKE t1; + +INSERT INTO t1 VALUES + (10,7,8,NULL,NULL,'01:27:35','01:27:35','2002-02-26 06:14:37','2002-02-26 06:14:37','v','v'), + (11,1,9,'2006-06-14','2006-06-14','19:48:31','19:48:31','1900-01-01 00:00:00','1900-01-01 00:00:00','r','r'); +INSERT INTO t2 SELECT * FROM t1; + +SELECT table2.col_int_key AS field1 +FROM ( + t2 AS table1 + RIGHT OUTER JOIN + ( + ( t1 AS table2 STRAIGHT_JOIN + t1 AS table3 ON ( + (table3.col_varchar_nokey = table2.col_varchar_key ) AND + (table3.pk = table2.col_int_key)) + ) + ) ON + ( + (table3.col_varchar_key = table2.col_varchar_key) OR + (table3.col_int_key = table2.pk) + ) +) +HAVING field1 < 216; + +DROP TABLE t1, t2; + === modified file 'sql/opt_table_elimination.cc' --- a/sql/opt_table_elimination.cc 2009-09-16 19:05:03 +0000 +++ b/sql/opt_table_elimination.cc 2010-02-18 12:03:52 +0000 @@ -1220,8 +1220,16 @@ bound_item= item; } } - exchange_sort<Dep_value_field>(fvl, compare_field_values, NULL); - add_module_expr(ctx, eq_mod, *and_level, NULL, bound_item, fvl); + /* + Multiple equality is only useful if it includes at least one field from + the table that we could potentially eliminate: + */ + if (fvl->elements) + { + + exchange_sort<Dep_value_field>(fvl, compare_field_values, NULL); + add_module_expr(ctx, eq_mod, *and_level, NULL, bound_item, fvl); + } break; } default:

1 0

[Maria-developers] Rev 2762: Subquery backport: update test results (checked). in file:///home/psergey/dev/maria-5.3-subqueries-r7/
by Sergey Petrunya 17 Feb '10

17 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r7/ ------------------------------------------------------------ revno: 2762 revision-id: psergey(a)askmonty.org-20100217225459-4mum80lxuu3cwmjz parent: psergey(a)askmonty.org-20100217215941-5uxponk64p55xg4a committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r7 timestamp: Thu 2010-02-18 01:54:59 +0300 message: Subquery backport: update test results (checked). === modified file 'mysql-test/r/subselect_no_mat.result' --- a/mysql-test/r/subselect_no_mat.result 2010-02-17 21:59:41 +0000 +++ b/mysql-test/r/subselect_no_mat.result 2010-02-17 22:54:59 +0000 @@ -1360,9 +1360,9 @@ 3 explain extended select * from t2 where t2.a in (select t1.a from t1,t3 where t1.b=t3.a); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index a a 5 NULL 4 100.00 Using where; Using index -1 PRIMARY t3 index a a 5 NULL 3 100.00 Using where; Using index -1 PRIMARY t1 ref a a 10 test.t2.a,test.t3.a 116 100.00 Using index; FirstMatch(t2) +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index +1 PRIMARY t3 index a a 5 NULL 3 100.00 Using index +1 PRIMARY t1 ref a a 10 test.t2.a,test.t3.a 116 100.61 Using index; FirstMatch(t2) Warnings: Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1` join `test`.`t3`) where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t1`.`b` = `test`.`t3`.`a`)) insert into t1 values (3,31);

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2818: Increased loop counts of sql-bench tests to get run times around
by noreply＠launchpad.net 17 Feb '10

17 Feb '10

------------------------------------------------------------ revno: 2818 committer: Hakan Kuecuekyilmaz <hakan(a)askmonty.org> branch nick: maria timestamp: Wed 2010-02-17 21:10:02 +0100 message: Increased loop counts of sql-bench tests to get run times around 5 minutes on current machines. Tested on a Xeon machine and a new dual core laptop. modified: sql-bench/test-ATIS.sh sql-bench/test-alter-table.sh sql-bench/test-big-tables.sh sql-bench/test-connect.sh sql-bench/test-select.sh sql-bench/test-transactions.sh sql-bench/test-wisconsin.sh -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] Rev 2761: Subquery optimizations backport: fix test failures, update test results. in file:///home/psergey/dev/maria-5.3-subqueries-r7/
by Sergey Petrunya 17 Feb '10

17 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r7/ ------------------------------------------------------------ revno: 2761 revision-id: psergey(a)askmonty.org-20100217215941-5uxponk64p55xg4a parent: psergey(a)askmonty.org-20100217104755-3psvc5fmo3pqsnpy committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r7 timestamp: Thu 2010-02-18 00:59:41 +0300 message: Subquery optimizations backport: fix test failures, update test results. === modified file 'mysql-test/r/subselect.result' --- a/mysql-test/r/subselect.result 2010-02-17 10:05:27 +0000 +++ b/mysql-test/r/subselect.result 2010-02-17 21:59:41 +0000 @@ -1293,31 +1293,31 @@ 4 explain extended select * from t2 where t2.a in (select a from t1); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL PRIMARY 4 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 unique_subquery PRIMARY PRIMARY 4 func 1 100.00 Using index +1 PRIMARY t2 index PRIMARY PRIMARY 4 NULL 4 100.00 Using index +1 PRIMARY t1 index PRIMARY PRIMARY 4 NULL 4 75.00 Using where; Using index; Using join buffer Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(<primary_index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on PRIMARY))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t1` join `test`.`t2` where (`test`.`t1`.`a` = `test`.`t2`.`a`) select * from t2 where t2.a in (select a from t1 where t1.b <> 30); a 2 4 explain extended select * from t2 where t2.a in (select a from t1 where t1.b <> 30); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL PRIMARY 4 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 unique_subquery PRIMARY PRIMARY 4 func 1 100.00 Using where +1 PRIMARY t2 index PRIMARY PRIMARY 4 NULL 4 100.00 Using index +1 PRIMARY t1 ALL PRIMARY NULL NULL NULL 4 75.00 Using where; Using join buffer Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(<primary_index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on PRIMARY where ((`test`.`t1`.`b` <> 30) and (<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`))))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t1` join `test`.`t2` where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t1`.`b` <> 30)) select * from t2 where t2.a in (select t1.a from t1,t3 where t1.b=t3.a); a 2 3 explain extended select * from t2 where t2.a in (select t1.a from t1,t3 where t1.b=t3.a); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL PRIMARY 4 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 eq_ref PRIMARY PRIMARY 4 func 1 100.00 -2 DEPENDENT SUBQUERY t3 eq_ref PRIMARY PRIMARY 4 test.t1.b 1 100.00 Using index +1 PRIMARY t2 index PRIMARY PRIMARY 4 NULL 4 100.00 Using index +1 PRIMARY t1 ALL PRIMARY NULL NULL NULL 4 75.00 Using where; Using join buffer +1 PRIMARY t3 eq_ref PRIMARY PRIMARY 4 test.t1.b 1 100.00 Using index Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(select 1 AS `Not_used` from `test`.`t1` join `test`.`t3` where ((`test`.`t3`.`a` = `test`.`t1`.`b`) and (<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`)))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t1` join `test`.`t3` join `test`.`t2` where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t3`.`a` = `test`.`t1`.`b`)) drop table t1, t2, t3; create table t1 (a int, b int, index a (a,b)); create table t2 (a int, index a (a)); @@ -1336,31 +1336,31 @@ 4 explain extended select * from t2 where t2.a in (select a from t1); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL a 5 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 index_subquery a a 5 func 1001 100.00 Using index +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index +1 PRIMARY t1 ref a a 5 test.t2.a 101 100.00 Using index; FirstMatch(t2) Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on a))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1`) where (`test`.`t1`.`a` = `test`.`t2`.`a`) select * from t2 where t2.a in (select a from t1 where t1.b <> 30); a 2 4 explain extended select * from t2 where t2.a in (select a from t1 where t1.b <> 30); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL a 5 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 index_subquery a a 5 func 1001 100.00 Using index; Using where +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index +1 PRIMARY t1 ref a a 5 test.t2.a 101 100.00 Using where; Using index; FirstMatch(t2) Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on a where ((`test`.`t1`.`b` <> 30) and (<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`))))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1`) where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t1`.`b` <> 30)) select * from t2 where t2.a in (select t1.a from t1,t3 where t1.b=t3.a); a 2 3 explain extended select * from t2 where t2.a in (select t1.a from t1,t3 where t1.b=t3.a); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL a 5 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 ref a a 5 func 1001 100.00 Using index -2 DEPENDENT SUBQUERY t3 index a a 5 NULL 3 100.00 Using where; Using index; Using join buffer +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index +1 PRIMARY t3 index a a 5 NULL 3 100.00 Using index +1 PRIMARY t1 ref a a 10 test.t2.a,test.t3.a 116 100.61 Using index; FirstMatch(t2) Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(select 1 AS `Not_used` from `test`.`t1` join `test`.`t3` where ((`test`.`t3`.`a` = `test`.`t1`.`b`) and (<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`)))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1` join `test`.`t3`) where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t1`.`b` = `test`.`t3`.`a`)) insert into t1 values (3,31); select * from t2 where t2.a in (select a from t1 where t1.b <> 30); a @@ -1373,10 +1373,10 @@ 4 explain extended select * from t2 where t2.a in (select a from t1 where t1.b <> 30); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index NULL a 5 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 index_subquery a a 5 func 1001 100.00 Using index; Using where +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index +1 PRIMARY t1 ref a a 5 test.t2.a 101 100.00 Using where; Using index; FirstMatch(t2) Warnings: -Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on a where ((`test`.`t1`.`b` <> 30) and (<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`))))) +Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1`) where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t1`.`b` <> 30)) drop table t0, t1, t2, t3; create table t1 (a int, b int); create table t2 (a int, b int); @@ -2827,10 +2827,10 @@ Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one` AS `one`,`test`.`t2`.`two` AS `two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`))) and trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`)))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`))))) AS `test` from `test`.`t1` explain extended SELECT one,two from t1 where ROW(one,two) IN (SELECT one,two FROM t2 WHERE flag = 'N'); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 Using where -2 DEPENDENT SUBQUERY t2 ALL NULL NULL NULL NULL 9 100.00 Using where +1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 +1 PRIMARY t2 ALL NULL NULL NULL NULL 9 100.00 Using where; FirstMatch(t1) Warnings: -Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two` from `test`.`t1` where <in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one` AS `one`,`test`.`t2`.`two` AS `two` from `test`.`t2` where ((`test`.`t2`.`flag` = 'N') and (<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) and (<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`)))) +Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t2`.`two` = `test`.`t1`.`two`) and (`test`.`t2`.`one` = `test`.`t1`.`one`) and (`test`.`t2`.`flag` = 'N')) explain extended SELECT one,two,ROW(one,two) IN (SELECT one,two FROM t2 WHERE flag = '0' group by one,two) as 'test' from t1; id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 @@ -3416,7 +3416,7 @@ SELECT * FROM t1 WHERE (a,b) = ANY (SELECT a, max(b) FROM t1 GROUP BY a); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t1 ALL NULL NULL NULL NULL 9 Using where -2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 9 Using temporary; Using filesort +2 SUBQUERY t1 ALL NULL NULL NULL NULL 9 Using temporary; Using filesort ALTER TABLE t1 ADD INDEX(a); SELECT * FROM t1 WHERE (a,b) = ANY (SELECT a, max(b) FROM t1 GROUP BY a); a b @@ -3427,7 +3427,7 @@ SELECT * FROM t1 WHERE (a,b) = ANY (SELECT a, max(b) FROM t1 GROUP BY a); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t1 ALL NULL NULL NULL NULL 9 Using where -2 DEPENDENT SUBQUERY t1 index NULL a 8 NULL 1 Using filesort +2 SUBQUERY t1 ALL NULL NULL NULL NULL 9 Using temporary; Using filesort DROP TABLE t1; create table t1( f1 int,f2 int); insert into t1 values (1,1),(2,2); @@ -4200,8 +4200,8 @@ CREATE INDEX I2 ON t1 (b); EXPLAIN SELECT a,b FROM t1 WHERE b IN (SELECT a FROM t1); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 2 Using where -2 DEPENDENT SUBQUERY t1 index_subquery I1 I1 2 func 2 Using index; Using where +1 PRIMARY t1 index I1 I1 2 NULL 2 Using index; LooseScan +1 PRIMARY t1 ref I2 I2 13 test.t1.a 2 Using index condition SELECT a,b FROM t1 WHERE b IN (SELECT a FROM t1); a b CREATE TABLE t2 (a VARCHAR(1), b VARCHAR(10)); @@ -4210,15 +4210,15 @@ CREATE INDEX I2 ON t2 (b); EXPLAIN SELECT a,b FROM t2 WHERE b IN (SELECT a FROM t2); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t2 ALL NULL NULL NULL NULL 2 Using where -2 DEPENDENT SUBQUERY t2 index_subquery I1 I1 4 func 2 Using index; Using where +1 PRIMARY t2 index I1 I1 4 NULL 2 Using index; LooseScan +1 PRIMARY t2 ref I2 I2 13 test.t2.a 2 Using index condition SELECT a,b FROM t2 WHERE b IN (SELECT a FROM t2); a b EXPLAIN SELECT a,b FROM t1 WHERE b IN (SELECT a FROM t1 WHERE LENGTH(a)<500); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 2 Using where -2 DEPENDENT SUBQUERY t1 index_subquery I1 I1 2 func 2 Using index; Using where +1 PRIMARY t1 index I1 I1 2 NULL 2 Using where; Using index; LooseScan +1 PRIMARY t1 ref I2 I2 13 test.t1.a 2 Using index condition SELECT a,b FROM t1 WHERE b IN (SELECT a FROM t1 WHERE LENGTH(a)<500); a b DROP TABLE t1,t2; @@ -4320,16 +4320,16 @@ INSERT INTO t1 VALUES (1),(2); EXPLAIN EXTENDED SELECT 1 FROM t1 WHERE 1 IN (SELECT 1 FROM t1 GROUP BY a); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 2 100.00 -2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 2 100.00 Using temporary; Using filesort +1 PRIMARY t1 ALL NULL NULL NULL NULL 2 100.00 Using where +2 SUBQUERY t1 ALL NULL NULL NULL NULL 2 100.00 Using temporary; Using filesort Warnings: -Note 1003 select 1 AS `1` from `test`.`t1` where <in_optimizer>(1,<exists>(select 1 AS `1` from `test`.`t1` group by `test`.`t1`.`a` having 1)) +Note 1003 select 1 AS `1` from `test`.`t1` where <in_optimizer>(1,1 in ( <materialize> (select 1 AS `1` from `test`.`t1` group by `test`.`t1`.`a` ), <primary_index_lookup>(1 in <temporary table> on distinct_key where ((1 = `materialized subselect`.`1`))))) EXPLAIN EXTENDED SELECT 1 FROM t1 WHERE 1 IN (SELECT 1 FROM t1 WHERE a > 3 GROUP BY a); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables -2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 2 100.00 Using where; Using temporary; Using filesort +1 PRIMARY t1 ALL NULL NULL NULL NULL 2 100.00 Using where +2 SUBQUERY t1 ALL NULL NULL NULL NULL 2 100.00 Using where; Using temporary; Using filesort Warnings: -Note 1003 select 1 AS `1` from `test`.`t1` where <in_optimizer>(1,<exists>(select 1 AS `1` from `test`.`t1` where (`test`.`t1`.`a` > 3) group by `test`.`t1`.`a` having 1)) +Note 1003 select 1 AS `1` from `test`.`t1` where <in_optimizer>(1,1 in ( <materialize> (select 1 AS `1` from `test`.`t1` where (`test`.`t1`.`a` > 3) group by `test`.`t1`.`a` ), <primary_index_lookup>(1 in <temporary table> on distinct_key where ((1 = `materialized subselect`.`1`))))) DROP TABLE t1; # # Bug#45061: Incorrectly market field caused wrong result. === modified file 'mysql-test/r/subselect2.result' --- a/mysql-test/r/subselect2.result 2007-01-12 20:22:41 +0000 +++ b/mysql-test/r/subselect2.result 2010-02-17 21:59:41 +0000 @@ -126,11 +126,11 @@ 1 PRIMARY t2 ALL DDOCTYPEID_IDX NULL NULL NULL 9 Using where 1 PRIMARY t1 eq_ref PRIMARY PRIMARY 34 test.t2.DOCID 1 1 PRIMARY t4 eq_ref PRIMARY PRIMARY 34 test.t2.DOCTYPEID 1 -2 DEPENDENT SUBQUERY t3 unique_subquery PRIMARY,FFOLDERID_IDX PRIMARY 34 func 1 Using where -3 DEPENDENT SUBQUERY t3 unique_subquery PRIMARY,FFOLDERID_IDX PRIMARY 34 func 1 Using where -4 DEPENDENT SUBQUERY t3 unique_subquery PRIMARY,FFOLDERID_IDX PRIMARY 34 func 1 Using where -5 DEPENDENT SUBQUERY t3 unique_subquery PRIMARY,FFOLDERID_IDX PRIMARY 34 func 1 Using where -6 DEPENDENT SUBQUERY t3 unique_subquery PRIMARY,FFOLDERID_IDX,CMFLDRPARNT_IDX PRIMARY 34 func 1 Using where +2 DEPENDENT SUBQUERY t3 eq_ref PRIMARY,FFOLDERID_IDX,CMFLDRPARNT_IDX PRIMARY 34 func 1 Using where +2 DEPENDENT SUBQUERY t3 eq_ref PRIMARY,FFOLDERID_IDX,CMFLDRPARNT_IDX PRIMARY 34 test.t3.PARENTID 1 Using where +2 DEPENDENT SUBQUERY t3 eq_ref PRIMARY,FFOLDERID_IDX,CMFLDRPARNT_IDX PRIMARY 34 test.t3.PARENTID 1 Using where +2 DEPENDENT SUBQUERY t3 eq_ref PRIMARY,FFOLDERID_IDX,CMFLDRPARNT_IDX PRIMARY 34 test.t3.PARENTID 1 Using where +2 DEPENDENT SUBQUERY t3 eq_ref PRIMARY,FFOLDERID_IDX,CMFLDRPARNT_IDX PRIMARY 34 test.t3.PARENTID 1 Using where drop table t1, t2, t3, t4; CREATE TABLE t1 (a int(10) , PRIMARY KEY (a)) Engine=InnoDB; INSERT INTO t1 VALUES (1),(2); === modified file 'mysql-test/r/subselect_no_mat.result' --- a/mysql-test/r/subselect_no_mat.result 2010-01-17 20:52:20 +0000 +++ b/mysql-test/r/subselect_no_mat.result 2010-02-17 21:59:41 +0000 @@ -54,7 +54,7 @@ Warnings: Note 1276 Field or reference 'b.a' of SELECT #3 was resolved in SELECT #1 Note 1276 Field or reference 'b.a' of SELECT #3 was resolved in SELECT #1 -Note 1003 select 1 AS `1` from dual having ((select '1' AS `a`) = 1) +Note 1003 select 1 AS `1` from (select 1 AS `a`) `b` having ((select '1' AS `a`) = 1) SELECT 1 FROM (SELECT 1 as a) as b HAVING (SELECT a)=1; 1 1 @@ -207,7 +207,7 @@ 3 DERIVED t2 ALL NULL NULL NULL NULL 2 100.00 Using where 2 SUBQUERY t3 ALL NULL NULL NULL NULL 3 100.00 Using where; Using filesort Warnings: -Note 1003 select (select `test`.`t3`.`a` AS `a` from `test`.`t3` where (`test`.`t3`.`a` < 8) order by 1 desc limit 1) AS `(select t3.a from t3 where a<8 order by 1 desc limit 1)`,'2' AS `a` from dual +Note 1003 select (select `test`.`t3`.`a` AS `a` from `test`.`t3` where (`test`.`t3`.`a` < 8) order by 1 desc limit 1) AS `(select t3.a from t3 where a<8 order by 1 desc limit 1)`,'2' AS `a` from (select `test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b` from `test`.`t2` where (`test`.`t2`.`a` > 1)) `tt` select * from t1 where t1.a=(select t2.a from t2 where t2.b=(select max(a) from t3) order by 1 desc limit 1); a 2 @@ -318,7 +318,7 @@ Warnings: Note 1276 Field or reference 'test.t2.a' of SELECT #2 was resolved in SELECT #1 Note 1276 Field or reference 'test.t2.a' of SELECT #3 was resolved in SELECT #1 -Note 1003 select (select '2' AS `a` from dual where ('2' = `test`.`t2`.`a`) union select `test`.`t5`.`a` AS `a` from `test`.`t5` where (`test`.`t5`.`a` = `test`.`t2`.`a`)) AS `(select a from t1 where t1.a=t2.a union select a from t5 where t5.a=t2.a)`,`test`.`t2`.`a` AS `a` from `test`.`t2` +Note 1003 select (select '2' AS `a` from `test`.`t1` where ('2' = `test`.`t2`.`a`) union select `test`.`t5`.`a` AS `a` from `test`.`t5` where (`test`.`t5`.`a` = `test`.`t2`.`a`)) AS `(select a from t1 where t1.a=t2.a union select a from t5 where t5.a=t2.a)`,`test`.`t2`.`a` AS `a` from `test`.`t2` select (select a from t1 where t1.a=t2.a union all select a from t5 where t5.a=t2.a), a from t2; ERROR 21000: Subquery returns more than 1 row create table t6 (patient_uq int, clinic_uq int, index i1 (clinic_uq)); @@ -739,7 +739,7 @@ Warnings: Note 1249 Select 3 was reduced during optimization Note 1249 Select 2 was reduced during optimization -Note 1003 select `test`.`t2`.`id` AS `id` from `test`.`t2` where (`test`.`t2`.`id` = <cache>((1 + 1))) +Note 1003 select `test`.`t2`.`id` AS `id` from `test`.`t2` where (`test`.`t2`.`id` = (1 + 1)) EXPLAIN EXTENDED SELECT * FROM t2 WHERE id IN (SELECT 1 UNION SELECT 3); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t2 index NULL id 5 NULL 2 100.00 Using where; Using index @@ -1340,7 +1340,7 @@ 4 explain extended select * from t2 where t2.a in (select a from t1); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index a a 5 NULL 4 100.00 Using where; Using index +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index 1 PRIMARY t1 ref a a 5 test.t2.a 101 100.00 Using index; FirstMatch(t2) Warnings: Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1`) where (`test`.`t1`.`a` = `test`.`t2`.`a`) @@ -1350,7 +1350,7 @@ 4 explain extended select * from t2 where t2.a in (select a from t1 where t1.b <> 30); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index a a 5 NULL 4 100.00 Using where; Using index +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index 1 PRIMARY t1 ref a a 5 test.t2.a 101 100.00 Using where; Using index; FirstMatch(t2) Warnings: Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1`) where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t1`.`b` <> 30)) @@ -1377,7 +1377,7 @@ 4 explain extended select * from t2 where t2.a in (select a from t1 where t1.b <> 30); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t2 index a a 5 NULL 4 100.00 Using where; Using index +1 PRIMARY t2 index a a 5 NULL 4 100.00 Using index 1 PRIMARY t1 ref a a 5 test.t2.a 101 100.00 Using where; Using index; FirstMatch(t2) Warnings: Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` semi join (`test`.`t1`) where ((`test`.`t1`.`a` = `test`.`t2`.`a`) and (`test`.`t1`.`b` <> 30)) @@ -1437,7 +1437,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 1 SIMPLE t1 system NULL NULL NULL NULL 1 100.00 Warnings: -Note 1003 (select 'tttt' AS `s1` from dual) +Note 1003 (select 'tttt' AS `s1` from `test`.`t1`) (select * from t1); s1 tttt @@ -1625,7 +1625,7 @@ 3 UNION t1 system NULL NULL NULL NULL 1 100.00 NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL Warnings: -Note 1003 select 'e' AS `s1` from dual where 1 +Note 1003 select 'e' AS `s1` from `test`.`t1` where 1 drop table t1; CREATE TABLE t1 (number char(11) NOT NULL default '') ENGINE=MyISAM CHARSET=latin1; INSERT INTO t1 VALUES ('69294728265'),('18621828126'),('89356874041'),('95895001874'); @@ -4204,7 +4204,7 @@ CREATE INDEX I2 ON t1 (b); EXPLAIN SELECT a,b FROM t1 WHERE b IN (SELECT a FROM t1); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 index I1 I1 2 NULL 2 Using where; Using index; LooseScan +1 PRIMARY t1 index I1 I1 2 NULL 2 Using index; LooseScan 1 PRIMARY t1 ref I2 I2 13 test.t1.a 2 Using index condition SELECT a,b FROM t1 WHERE b IN (SELECT a FROM t1); a b @@ -4214,7 +4214,7 @@ CREATE INDEX I2 ON t2 (b); EXPLAIN SELECT a,b FROM t2 WHERE b IN (SELECT a FROM t2); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t2 index I1 I1 4 NULL 2 Using where; Using index; LooseScan +1 PRIMARY t2 index I1 I1 4 NULL 2 Using index; LooseScan 1 PRIMARY t2 ref I2 I2 13 test.t2.a 2 Using index condition SELECT a,b FROM t2 WHERE b IN (SELECT a FROM t2); a b @@ -4327,13 +4327,13 @@ 1 PRIMARY t1 ALL NULL NULL NULL NULL 2 100.00 2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 2 100.00 Using temporary; Using filesort Warnings: -Note 1003 select 1 AS `1` from `test`.`t1` where <in_optimizer>(1,<exists>(select 1 AS `1` from `test`.`t1` group by `test`.`t1`.`a` having (<cache>(1) = <ref_null_helper>(1)))) +Note 1003 select 1 AS `1` from `test`.`t1` where <in_optimizer>(1,<exists>(select 1 AS `1` from `test`.`t1` group by `test`.`t1`.`a` having 1)) EXPLAIN EXTENDED SELECT 1 FROM t1 WHERE 1 IN (SELECT 1 FROM t1 WHERE a > 3 GROUP BY a); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables 2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 2 100.00 Using where; Using temporary; Using filesort Warnings: -Note 1003 select 1 AS `1` from `test`.`t1` where <in_optimizer>(1,<exists>(select 1 AS `1` from `test`.`t1` where (`test`.`t1`.`a` > 3) group by `test`.`t1`.`a` having (<cache>(1) = <ref_null_helper>(1)))) +Note 1003 select 1 AS `1` from `test`.`t1` where <in_optimizer>(1,<exists>(select 1 AS `1` from `test`.`t1` where (`test`.`t1`.`a` > 3) group by `test`.`t1`.`a` having 1)) DROP TABLE t1; # # Bug#45061: Incorrectly market field caused wrong result. @@ -4377,7 +4377,8 @@ WHERE a = 230; id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables -2 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables +2 DEPENDENT SUBQUERY st1 index NULL a 5 NULL 2 Using index +2 DEPENDENT SUBQUERY st2 index b b 5 NULL 2 Using where; Using index; Using join buffer SELECT MAX(b), (SELECT COUNT(*) FROM st1,st2 WHERE st2.b <= t1.b) FROM t1 WHERE a = 230; @@ -4686,7 +4687,7 @@ explain SELECT t1.a, (SELECT 1 FROM t2 WHERE t2.b=t3.c AND t2.c=t1.a ORDER BY t2.d LIMIT 1) AS incorrect FROM t1, t3 WHERE t3.b=t1.a; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t3 index b,b_2 b 10 NULL 2 Using where; Using index +1 PRIMARY t3 index b,b_2 b 10 NULL 2 Using index 1 PRIMARY t1 eq_ref PRIMARY PRIMARY 4 test.t3.b 1 Using index 2 DEPENDENT SUBQUERY t2 index b,b_2,c d 5 NULL 1 Using where SELECT t1.a, (SELECT 1 FROM t2 WHERE t2.b=t3.c AND t2.c=t1.a ORDER BY t2.d LIMIT 1) AS incorrect FROM t1, t3 WHERE t3.b=t1.a; === modified file 'mysql-test/r/subselect_no_opts.result' --- a/mysql-test/r/subselect_no_opts.result 2010-01-17 20:52:20 +0000 +++ b/mysql-test/r/subselect_no_opts.result 2010-02-17 21:59:41 +0000 @@ -54,7 +54,7 @@ Warnings: Note 1276 Field or reference 'b.a' of SELECT #3 was resolved in SELECT #1 Note 1276 Field or reference 'b.a' of SELECT #3 was resolved in SELECT #1 -Note 1003 select 1 AS `1` from dual having ((select '1' AS `a`) = 1) +Note 1003 select 1 AS `1` from (select 1 AS `a`) `b` having ((select '1' AS `a`) = 1) SELECT 1 FROM (SELECT 1 as a) as b HAVING (SELECT a)=1; 1 1 @@ -207,7 +207,7 @@ 3 DERIVED t2 ALL NULL NULL NULL NULL 2 100.00 Using where 2 SUBQUERY t3 ALL NULL NULL NULL NULL 3 100.00 Using where; Using filesort Warnings: -Note 1003 select (select `test`.`t3`.`a` AS `a` from `test`.`t3` where (`test`.`t3`.`a` < 8) order by 1 desc limit 1) AS `(select t3.a from t3 where a<8 order by 1 desc limit 1)`,'2' AS `a` from dual +Note 1003 select (select `test`.`t3`.`a` AS `a` from `test`.`t3` where (`test`.`t3`.`a` < 8) order by 1 desc limit 1) AS `(select t3.a from t3 where a<8 order by 1 desc limit 1)`,'2' AS `a` from (select `test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b` from `test`.`t2` where (`test`.`t2`.`a` > 1)) `tt` select * from t1 where t1.a=(select t2.a from t2 where t2.b=(select max(a) from t3) order by 1 desc limit 1); a 2 @@ -318,7 +318,7 @@ Warnings: Note 1276 Field or reference 'test.t2.a' of SELECT #2 was resolved in SELECT #1 Note 1276 Field or reference 'test.t2.a' of SELECT #3 was resolved in SELECT #1 -Note 1003 select (select '2' AS `a` from dual where ('2' = `test`.`t2`.`a`) union select `test`.`t5`.`a` AS `a` from `test`.`t5` where (`test`.`t5`.`a` = `test`.`t2`.`a`)) AS `(select a from t1 where t1.a=t2.a union select a from t5 where t5.a=t2.a)`,`test`.`t2`.`a` AS `a` from `test`.`t2` +Note 1003 select (select '2' AS `a` from `test`.`t1` where ('2' = `test`.`t2`.`a`) union select `test`.`t5`.`a` AS `a` from `test`.`t5` where (`test`.`t5`.`a` = `test`.`t2`.`a`)) AS `(select a from t1 where t1.a=t2.a union select a from t5 where t5.a=t2.a)`,`test`.`t2`.`a` AS `a` from `test`.`t2` select (select a from t1 where t1.a=t2.a union all select a from t5 where t5.a=t2.a), a from t2; ERROR 21000: Subquery returns more than 1 row create table t6 (patient_uq int, clinic_uq int, index i1 (clinic_uq)); @@ -739,7 +739,7 @@ Warnings: Note 1249 Select 3 was reduced during optimization Note 1249 Select 2 was reduced during optimization -Note 1003 select `test`.`t2`.`id` AS `id` from `test`.`t2` where (`test`.`t2`.`id` = <cache>((1 + 1))) +Note 1003 select `test`.`t2`.`id` AS `id` from `test`.`t2` where (`test`.`t2`.`id` = (1 + 1)) EXPLAIN EXTENDED SELECT * FROM t2 WHERE id IN (SELECT 1 UNION SELECT 3); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t2 index NULL id 5 NULL 2 100.00 Using where; Using index @@ -1318,7 +1318,7 @@ explain extended select * from t2 where t2.a in (select t1.a from t1,t3 where t1.b=t3.a); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t2 index NULL PRIMARY 4 NULL 4 100.00 Using where; Using index -2 DEPENDENT SUBQUERY t1 eq_ref PRIMARY PRIMARY 4 func 1 100.00 Using where +2 DEPENDENT SUBQUERY t1 eq_ref PRIMARY PRIMARY 4 func 1 100.00 2 DEPENDENT SUBQUERY t3 eq_ref PRIMARY PRIMARY 4 test.t1.b 1 100.00 Using index Warnings: Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(select 1 AS `Not_used` from `test`.`t1` join `test`.`t3` where ((`test`.`t3`.`a` = `test`.`t1`.`b`) and (<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`)))) @@ -1437,7 +1437,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 1 SIMPLE t1 system NULL NULL NULL NULL 1 100.00 Warnings: -Note 1003 (select 'tttt' AS `s1` from dual) +Note 1003 (select 'tttt' AS `s1` from `test`.`t1`) (select * from t1); s1 tttt @@ -1625,7 +1625,7 @@ 3 UNION t1 system NULL NULL NULL NULL 1 100.00 NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL Warnings: -Note 1003 select 'e' AS `s1` from dual where 1 +Note 1003 select 'e' AS `s1` from `test`.`t1` where 1 drop table t1; CREATE TABLE t1 (number char(11) NOT NULL default '') ENGINE=MyISAM CHARSET=latin1; INSERT INTO t1 VALUES ('69294728265'),('18621828126'),('89356874041'),('95895001874'); @@ -4327,13 +4327,13 @@ 1 PRIMARY t1 ALL NULL NULL NULL NULL 2 100.00 2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 2 100.00 Using temporary; Using filesort Warnings: -Note 1003 select 1 AS `1` from `test`.`t1` where <in_optimizer>(1,<exists>(select 1 AS `1` from `test`.`t1` group by `test`.`t1`.`a` having (<cache>(1) = <ref_null_helper>(1)))) +Note 1003 select 1 AS `1` from `test`.`t1` where <in_optimizer>(1,<exists>(select 1 AS `1` from `test`.`t1` group by `test`.`t1`.`a` having 1)) EXPLAIN EXTENDED SELECT 1 FROM t1 WHERE 1 IN (SELECT 1 FROM t1 WHERE a > 3 GROUP BY a); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables 2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 2 100.00 Using where; Using temporary; Using filesort Warnings: -Note 1003 select 1 AS `1` from `test`.`t1` where <in_optimizer>(1,<exists>(select 1 AS `1` from `test`.`t1` where (`test`.`t1`.`a` > 3) group by `test`.`t1`.`a` having (<cache>(1) = <ref_null_helper>(1)))) +Note 1003 select 1 AS `1` from `test`.`t1` where <in_optimizer>(1,<exists>(select 1 AS `1` from `test`.`t1` where (`test`.`t1`.`a` > 3) group by `test`.`t1`.`a` having 1)) DROP TABLE t1; # # Bug#45061: Incorrectly market field caused wrong result. @@ -4377,7 +4377,8 @@ WHERE a = 230; id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables -2 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables +2 DEPENDENT SUBQUERY st1 index NULL a 5 NULL 2 Using index +2 DEPENDENT SUBQUERY st2 index b b 5 NULL 2 Using where; Using index; Using join buffer SELECT MAX(b), (SELECT COUNT(*) FROM st1,st2 WHERE st2.b <= t1.b) FROM t1 WHERE a = 230; @@ -4686,7 +4687,7 @@ explain SELECT t1.a, (SELECT 1 FROM t2 WHERE t2.b=t3.c AND t2.c=t1.a ORDER BY t2.d LIMIT 1) AS incorrect FROM t1, t3 WHERE t3.b=t1.a; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t3 index b,b_2 b 10 NULL 2 Using where; Using index +1 PRIMARY t3 index b,b_2 b 10 NULL 2 Using index 1 PRIMARY t1 eq_ref PRIMARY PRIMARY 4 test.t3.b 1 Using index 2 DEPENDENT SUBQUERY t2 index b,b_2,c d 5 NULL 1 Using where SELECT t1.a, (SELECT 1 FROM t2 WHERE t2.b=t3.c AND t2.c=t1.a ORDER BY t2.d LIMIT 1) AS incorrect FROM t1, t3 WHERE t3.b=t1.a; === modified file 'mysql-test/r/subselect_no_semijoin.result' --- a/mysql-test/r/subselect_no_semijoin.result 2010-02-17 10:47:55 +0000 +++ b/mysql-test/r/subselect_no_semijoin.result 2010-02-17 21:59:41 +0000 @@ -4377,7 +4377,8 @@ WHERE a = 230; id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables -2 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables +2 DEPENDENT SUBQUERY st1 index NULL a 5 NULL 2 Using index +2 DEPENDENT SUBQUERY st2 index b b 5 NULL 2 Using where; Using index; Using join buffer SELECT MAX(b), (SELECT COUNT(*) FROM st1,st2 WHERE st2.b <= t1.b) FROM t1 WHERE a = 230; === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-02-17 10:05:27 +0000 +++ b/sql/item_subselect.cc 2010-02-17 21:59:41 +0000 @@ -3267,7 +3267,7 @@ if (tmp_result_sink->create_result_table( thd, tmp_columns, TRUE, thd->options | TMP_TABLE_ALL_COLUMNS, - "materialized subselect")) + "materialized subselect", TRUE)) DBUG_RETURN(TRUE); tmp_table= tmp_result_sink->table; === modified file 'sql/opt_subselect.cc' --- a/sql/opt_subselect.cc 2010-02-17 10:05:27 +0000 +++ b/sql/opt_subselect.cc 2010-02-17 21:59:41 +0000 @@ -2199,7 +2199,7 @@ */ sjm->sjm_table_param.init(); sjm->sjm_table_param.field_count= item_list.elements; - // psergey-merge: the following is not in 5.x: sjm->sjm_table_param.bit_fields_as_long= TRUE; + sjm->sjm_table_param.bit_fields_as_long= TRUE; List_iterator<Item> it(item_list); Item *right_expr; while((right_expr= it++)) === modified file 'sql/sql_class.cc' --- a/sql/sql_class.cc 2009-11-12 04:31:28 +0000 +++ b/sql/sql_class.cc 2010-02-17 21:59:41 +0000 @@ -2890,6 +2890,7 @@ quick_group= 1; table_charset= 0; precomputed_group_by= 0; + bit_fields_as_long= 0; DBUG_VOID_RETURN; } === modified file 'sql/sql_class.h' --- a/sql/sql_class.h 2010-01-17 14:55:08 +0000 +++ b/sql/sql_class.h 2010-02-17 21:59:41 +0000 @@ -2710,11 +2710,18 @@ */ bool precomputed_group_by; bool force_copy_fields; + /* + If TRUE, create_tmp_field called from create_tmp_table will convert + all BIT fields to 64-bit longs. This is a workaround the limitation + that MEMORY tables cannot index BIT columns. + */ + bool bit_fields_as_long; TMP_TABLE_PARAM() :copy_field(0), group_parts(0), group_length(0), group_null_parts(0), convert_blob_length(0), - schema_table(0), precomputed_group_by(0), force_copy_fields(0) + schema_table(0), precomputed_group_by(0), force_copy_fields(0), + bit_fields_as_long(0) {} ~TMP_TABLE_PARAM() { @@ -2745,7 +2752,7 @@ bool create_result_table(THD *thd, List<Item> *column_types, bool is_distinct, ulonglong options, - const char *alias); + const char *alias, bool bit_fields_as_long); }; /* Base subselect interface class */ === modified file 'sql/sql_cursor.cc' --- a/sql/sql_cursor.cc 2009-10-19 17:14:48 +0000 +++ b/sql/sql_cursor.cc 2010-02-17 21:59:41 +0000 @@ -714,7 +714,8 @@ { DBUG_ASSERT(table == 0); if (create_result_table(unit->thd, unit->get_unit_column_types(), - FALSE, thd->options | TMP_TABLE_ALL_COLUMNS, "")) + FALSE, thd->options | TMP_TABLE_ALL_COLUMNS, "", + FALSE)) return TRUE; materialized_cursor= new (&table->mem_root) === modified file 'sql/sql_derived.cc' --- a/sql/sql_derived.cc 2009-05-27 15:19:44 +0000 +++ b/sql/sql_derived.cc 2010-02-17 21:59:41 +0000 @@ -168,7 +168,8 @@ */ if ((res= derived_result->create_result_table(thd, &unit->types, FALSE, create_options, - orig_table_list->alias))) + orig_table_list->alias, + FALSE))) goto exit; table= derived_result->table; === modified file 'sql/sql_select.cc' --- a/sql/sql_select.cc 2010-02-15 21:53:06 +0000 +++ b/sql/sql_select.cc 2010-02-17 21:59:41 +0000 @@ -10963,7 +10963,15 @@ group != 0, !force_copy_fields && (not_all_columns || group !=0), - item->marker == 4, force_copy_fields, + /* + If item->marker == 4 then we force create_tmp_field + to create a 64-bit longs for BIT fields because HEAP + tables can't index BIT fields directly. We do the same + for distinct, as we want the distinct index to be + usable in this case too. + */ + item->marker == 4 || param->bit_fields_as_long, // psergey-feb17 + force_copy_fields, param->convert_blob_length); if (!new_field) @@ -17838,6 +17846,17 @@ /* Add "filtered" field to item_list. */ if (join->thd->lex->describe & DESCRIBE_EXTENDED) { + /* + psergey-todo: + in the code above, we cast to integer when asssigning to + examined_rows. + In the code below, we may divide original value but result of + conversion of the same value to integer, which may produce a + value that's greater than 100%, which looks very odd. + I'm not fixing this right away because that might trigger a wave + of small EXPLAIN EXTENDED output changes, which I don't have time + to deal with right now. + */ float f= 0.0; if (examined_rows) f= (float) (100.0 * join->best_positions[i].records_read / === modified file 'sql/sql_union.cc' --- a/sql/sql_union.cc 2010-01-28 13:48:33 +0000 +++ b/sql/sql_union.cc 2010-02-17 21:59:41 +0000 @@ -103,6 +103,8 @@ is_union_distinct if set, the temporary table will eliminate duplicates on insert options create options + table_alias name of the temporary table + bit_fields_as_long convert bit fields to ulonglong DESCRIPTION Create a temporary table that is used to store the result of a UNION, @@ -116,11 +118,13 @@ bool select_union::create_result_table(THD *thd_arg, List<Item> *column_types, bool is_union_distinct, ulonglong options, - const char *alias) + const char *alias, + bool bit_fields_as_long) { DBUG_ASSERT(table == 0); tmp_table_param.init(); tmp_table_param.field_count= column_types->elements; + tmp_table_param.bit_fields_as_long= bit_fields_as_long; if (! (table= create_tmp_table(thd_arg, &tmp_table_param, *column_types, (ORDER*) 0, is_union_distinct, 1, @@ -350,7 +354,7 @@ create_options= create_options | TMP_TABLE_FORCE_MYISAM; if (union_result->create_result_table(thd, &types, test(union_distinct), - create_options, "")) + create_options, "", FALSE)) goto err; bzero((char*) &result_table_list, sizeof(result_table_list)); result_table_list.db= (char*) "";

1 0

[Maria-developers] Rev 2760: Subquery backport: in file:///home/psergey/dev/maria-5.3-subqueries-r7/
by Sergey Petrunya 17 Feb '10

17 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r7/ ------------------------------------------------------------ revno: 2760 revision-id: psergey(a)askmonty.org-20100217104755-3psvc5fmo3pqsnpy parent: psergey(a)askmonty.org-20100217100527-k81b7torhmj99moy committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r7 timestamp: Wed 2010-02-17 13:47:55 +0300 message: Subquery backport: - More test results updates (checked) === modified file 'mysql-test/r/subselect3_jcl6.result' --- a/mysql-test/r/subselect3_jcl6.result 2010-02-11 21:56:02 +0000 +++ b/mysql-test/r/subselect3_jcl6.result 2010-02-17 10:47:55 +0000 @@ -877,7 +877,7 @@ Note 1276 Field or reference 'test.t1.a' of SELECT #3 was resolved in SELECT #2 Note 1276 Field or reference 'test.t1.c' of SELECT #3 was resolved in SELECT #2 Error 1054 Unknown column 'c' in 'field list' -Note 1003 select `c` AS `c` from (select (select count(`test`.`t1`.`a`) AS `COUNT(a)` from dual group by `c`) AS `(SELECT COUNT(a) FROM +Note 1003 select `c` AS `c` from (select (select count(`test`.`t1`.`a`) AS `COUNT(a)` from (select count(`test`.`t1`.`b`) AS `COUNT(b)` from `test`.`t1`) `x` group by `t1`.`c`) AS `(SELECT COUNT(a) FROM (SELECT COUNT(b) FROM t1) AS x GROUP BY c )` from `test`.`t1` group by `test`.`t1`.`b`) `y` DROP TABLE t1; @@ -1122,7 +1122,7 @@ explain select * from t3 where a in (select t1.kp1 from t1,t4 where kp1<20 and t4.pk=t1.c); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 range kp1 kp1 5 NULL 48 Using index condition; Using where; Using MRR; LooseScan +1 PRIMARY t1 range kp1 kp1 5 NULL 48 Using index condition; Using MRR; LooseScan 1 PRIMARY t4 eq_ref PRIMARY PRIMARY 4 test.t1.c 1 Using index; FirstMatch(t1) 1 PRIMARY t3 ALL NULL NULL NULL NULL 100 Using where; Using join buffer drop table t1, t3, t4; === modified file 'mysql-test/r/subselect_no_semijoin.result' --- a/mysql-test/r/subselect_no_semijoin.result 2010-01-17 20:52:20 +0000 +++ b/mysql-test/r/subselect_no_semijoin.result 2010-02-17 10:47:55 +0000 @@ -54,7 +54,7 @@ Warnings: Note 1276 Field or reference 'b.a' of SELECT #3 was resolved in SELECT #1 Note 1276 Field or reference 'b.a' of SELECT #3 was resolved in SELECT #1 -Note 1003 select 1 AS `1` from dual having ((select '1' AS `a`) = 1) +Note 1003 select 1 AS `1` from (select 1 AS `a`) `b` having ((select '1' AS `a`) = 1) SELECT 1 FROM (SELECT 1 as a) as b HAVING (SELECT a)=1; 1 1 @@ -207,7 +207,7 @@ 3 DERIVED t2 ALL NULL NULL NULL NULL 2 100.00 Using where 2 SUBQUERY t3 ALL NULL NULL NULL NULL 3 100.00 Using where; Using filesort Warnings: -Note 1003 select (select `test`.`t3`.`a` AS `a` from `test`.`t3` where (`test`.`t3`.`a` < 8) order by 1 desc limit 1) AS `(select t3.a from t3 where a<8 order by 1 desc limit 1)`,'2' AS `a` from dual +Note 1003 select (select `test`.`t3`.`a` AS `a` from `test`.`t3` where (`test`.`t3`.`a` < 8) order by 1 desc limit 1) AS `(select t3.a from t3 where a<8 order by 1 desc limit 1)`,'2' AS `a` from (select `test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b` from `test`.`t2` where (`test`.`t2`.`a` > 1)) `tt` select * from t1 where t1.a=(select t2.a from t2 where t2.b=(select max(a) from t3) order by 1 desc limit 1); a 2 @@ -318,7 +318,7 @@ Warnings: Note 1276 Field or reference 'test.t2.a' of SELECT #2 was resolved in SELECT #1 Note 1276 Field or reference 'test.t2.a' of SELECT #3 was resolved in SELECT #1 -Note 1003 select (select '2' AS `a` from dual where ('2' = `test`.`t2`.`a`) union select `test`.`t5`.`a` AS `a` from `test`.`t5` where (`test`.`t5`.`a` = `test`.`t2`.`a`)) AS `(select a from t1 where t1.a=t2.a union select a from t5 where t5.a=t2.a)`,`test`.`t2`.`a` AS `a` from `test`.`t2` +Note 1003 select (select '2' AS `a` from `test`.`t1` where ('2' = `test`.`t2`.`a`) union select `test`.`t5`.`a` AS `a` from `test`.`t5` where (`test`.`t5`.`a` = `test`.`t2`.`a`)) AS `(select a from t1 where t1.a=t2.a union select a from t5 where t5.a=t2.a)`,`test`.`t2`.`a` AS `a` from `test`.`t2` select (select a from t1 where t1.a=t2.a union all select a from t5 where t5.a=t2.a), a from t2; ERROR 21000: Subquery returns more than 1 row create table t6 (patient_uq int, clinic_uq int, index i1 (clinic_uq)); @@ -739,7 +739,7 @@ Warnings: Note 1249 Select 3 was reduced during optimization Note 1249 Select 2 was reduced during optimization -Note 1003 select `test`.`t2`.`id` AS `id` from `test`.`t2` where (`test`.`t2`.`id` = <cache>((1 + 1))) +Note 1003 select `test`.`t2`.`id` AS `id` from `test`.`t2` where (`test`.`t2`.`id` = (1 + 1)) EXPLAIN EXTENDED SELECT * FROM t2 WHERE id IN (SELECT 1 UNION SELECT 3); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t2 index NULL id 5 NULL 2 100.00 Using where; Using index @@ -1437,7 +1437,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 1 SIMPLE t1 system NULL NULL NULL NULL 1 100.00 Warnings: -Note 1003 (select 'tttt' AS `s1` from dual) +Note 1003 (select 'tttt' AS `s1` from `test`.`t1`) (select * from t1); s1 tttt @@ -1625,7 +1625,7 @@ 3 UNION t1 system NULL NULL NULL NULL 1 100.00 NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL Warnings: -Note 1003 select 'e' AS `s1` from dual where 1 +Note 1003 select 'e' AS `s1` from `test`.`t1` where 1 drop table t1; CREATE TABLE t1 (number char(11) NOT NULL default '') ENGINE=MyISAM CHARSET=latin1; INSERT INTO t1 VALUES ('69294728265'),('18621828126'),('89356874041'),('95895001874'); @@ -4686,7 +4686,7 @@ explain SELECT t1.a, (SELECT 1 FROM t2 WHERE t2.b=t3.c AND t2.c=t1.a ORDER BY t2.d LIMIT 1) AS incorrect FROM t1, t3 WHERE t3.b=t1.a; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t3 index b,b_2 b 10 NULL 2 Using where; Using index +1 PRIMARY t3 index b,b_2 b 10 NULL 2 Using index 1 PRIMARY t1 eq_ref PRIMARY PRIMARY 4 test.t3.b 1 Using index 2 DEPENDENT SUBQUERY t2 index b,b_2,c d 5 NULL 1 Using where SELECT t1.a, (SELECT 1 FROM t2 WHERE t2.b=t3.c AND t2.c=t1.a ORDER BY t2.d LIMIT 1) AS incorrect FROM t1, t3 WHERE t3.b=t1.a; === modified file 'mysql-test/r/subselect_sj.result' --- a/mysql-test/r/subselect_sj.result 2010-02-11 23:59:58 +0000 +++ b/mysql-test/r/subselect_sj.result 2010-02-17 10:47:55 +0000 @@ -12,7 +12,7 @@ Flattened because of dependency, t10=func(t1) explain select * from t1 where a in (select pk from t10); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 3 Using where +1 PRIMARY t1 ALL NULL NULL NULL NULL 3 1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.a 1 Using index select * from t1 where a in (select pk from t10); a b @@ -39,7 +39,7 @@ a b explain select * from t1 where a in (select pk from t10) and b in (select pk from t10); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 3 Using where +1 PRIMARY t1 ALL NULL NULL NULL NULL 3 1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.a 1 Using index 1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.b 1 Using index select * from t1 where a in (select pk from t10) and b in (select pk from t10); @@ -50,8 +50,8 @@ flattening a nested subquery explain select * from t1 where a in (select pk from t10 where t10.a in (select pk from t12)); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 3 Using where -1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.a 1 Using where +1 PRIMARY t1 ALL NULL NULL NULL NULL 3 +1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.a 1 1 PRIMARY t12 eq_ref PRIMARY PRIMARY 4 test.t10.a 1 Using index select * from t1 where a in (select pk from t10 where t10.a in (select pk from t12)); a b @@ -61,8 +61,8 @@ flattening subquery w/ several tables explain extended select * from t1 where a in (select t10.pk from t10, t12 where t12.pk=t10.a); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 3 100.00 Using where -1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.a 1 100.00 Using where +1 PRIMARY t1 ALL NULL NULL NULL NULL 3 100.00 +1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.a 1 100.00 1 PRIMARY t12 eq_ref PRIMARY PRIMARY 4 test.t10.a 1 100.00 Using index Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,`test`.`t1`.`b` AS `b` from `test`.`t10` join `test`.`t12` join `test`.`t1` where ((`test`.`t10`.`pk` = `test`.`t1`.`a`) and (`test`.`t12`.`pk` = `test`.`t10`.`a`)) @@ -545,7 +545,7 @@ (SELECT t1.pk FROM t0 t1 JOIN t0 t2 ON t2.vkey = t1.vnokey); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t0 ALL PRIMARY NULL NULL NULL 5 100.00 -1 PRIMARY t1 eq_ref PRIMARY PRIMARY 4 test.t0.pk 1 100.00 Using where +1 PRIMARY t1 eq_ref PRIMARY PRIMARY 4 test.t0.pk 1 100.00 1 PRIMARY t2 ref vkey vkey 4 test.t1.vnokey 2 100.00 Using index; FirstMatch(t1) Warnings: Note 1003 select `test`.`t0`.`vkey` AS `vkey` from `test`.`t0` `t1` semi join (`test`.`t0` `t2`) join `test`.`t0` where ((`test`.`t2`.`vkey` = `test`.`t1`.`vnokey`) and (`test`.`t1`.`pk` = `test`.`t0`.`pk`)) === modified file 'mysql-test/r/subselect_sj2.result' --- a/mysql-test/r/subselect_sj2.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect_sj2.result 2010-02-17 10:47:55 +0000 @@ -32,7 +32,7 @@ 9 5 explain select * from t2 where b in (select a from t1); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 3 Using where; Materialize; Scan +1 PRIMARY t1 ALL NULL NULL NULL NULL 3 Materialize; Scan 1 PRIMARY t2 ref b b 5 test.t1.a 2 select * from t2 where b in (select a from t1); a b @@ -73,7 +73,7 @@ from t0 A, t0 B where B.a <5; explain select * from t3 where b in (select a from t0); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t0 ALL NULL NULL NULL NULL 10 Using where; Materialize; Scan +1 PRIMARY t0 ALL NULL NULL NULL NULL 10 Materialize; Scan 1 PRIMARY t3 ref b b 5 test.t0.a 1 set @save_ecp= @@engine_condition_pushdown; set engine_condition_pushdown=0; @@ -417,7 +417,7 @@ where t0.a in ( select t1.a from t1,t2 where t2.a=t0.a and t1.b=t2.b); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t0 ALL NULL NULL NULL NULL 5 100.00 Using where +1 PRIMARY t0 ALL NULL NULL NULL NULL 5 100.00 1 PRIMARY t1 ref a a 5 test.t0.a 1 100.00 Start temporary 1 PRIMARY t2 eq_ref PRIMARY PRIMARY 4 test.t0.a 1 100.00 Using where; End temporary Warnings: === modified file 'mysql-test/r/subselect_sj2_jcl6.result' --- a/mysql-test/r/subselect_sj2_jcl6.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect_sj2_jcl6.result 2010-02-17 10:47:55 +0000 @@ -36,7 +36,7 @@ 9 5 explain select * from t2 where b in (select a from t1); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 3 Using where; Materialize; Scan +1 PRIMARY t1 ALL NULL NULL NULL NULL 3 Materialize; Scan 1 PRIMARY t2 ref b b 5 test.t1.a 2 Using join buffer select * from t2 where b in (select a from t1); a b @@ -77,8 +77,8 @@ from t0 A, t0 B where B.a <5; explain select * from t3 where b in (select a from t0); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t0 ALL NULL NULL NULL NULL 10 Using where; Materialize; Scan -1 PRIMARY t3 ref b b 5 test.t0.a 1 +1 PRIMARY t0 ALL NULL NULL NULL NULL 10 Materialize; Scan +1 PRIMARY t3 ref b b 5 test.t0.a 1 Using join buffer set @save_ecp= @@engine_condition_pushdown; set engine_condition_pushdown=0; select * from t3 where b in (select A.a+B.a from t0 A, t0 B where B.a<5); @@ -421,7 +421,7 @@ where t0.a in ( select t1.a from t1,t2 where t2.a=t0.a and t1.b=t2.b); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t0 ALL NULL NULL NULL NULL 5 100.00 Using where +1 PRIMARY t0 ALL NULL NULL NULL NULL 5 100.00 1 PRIMARY t1 ref a a 5 test.t0.a 1 100.00 Start temporary; Using join buffer 1 PRIMARY t2 eq_ref PRIMARY PRIMARY 4 test.t0.a 1 100.00 Using where; End temporary; Using join buffer Warnings: === modified file 'mysql-test/r/subselect_sj_jcl6.result' --- a/mysql-test/r/subselect_sj_jcl6.result 2010-02-11 23:59:58 +0000 +++ b/mysql-test/r/subselect_sj_jcl6.result 2010-02-17 10:47:55 +0000 @@ -16,7 +16,7 @@ Flattened because of dependency, t10=func(t1) explain select * from t1 where a in (select pk from t10); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 3 Using where +1 PRIMARY t1 ALL NULL NULL NULL NULL 3 1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.a 1 Using index select * from t1 where a in (select pk from t10); a b @@ -43,7 +43,7 @@ a b explain select * from t1 where a in (select pk from t10) and b in (select pk from t10); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 3 Using where +1 PRIMARY t1 ALL NULL NULL NULL NULL 3 1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.a 1 Using index 1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.b 1 Using index select * from t1 where a in (select pk from t10) and b in (select pk from t10); @@ -54,8 +54,8 @@ flattening a nested subquery explain select * from t1 where a in (select pk from t10 where t10.a in (select pk from t12)); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 3 Using where -1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.a 1 Using where; Using join buffer +1 PRIMARY t1 ALL NULL NULL NULL NULL 3 +1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.a 1 Using join buffer 1 PRIMARY t12 eq_ref PRIMARY PRIMARY 4 test.t10.a 1 Using index select * from t1 where a in (select pk from t10 where t10.a in (select pk from t12)); a b @@ -65,8 +65,8 @@ flattening subquery w/ several tables explain extended select * from t1 where a in (select t10.pk from t10, t12 where t12.pk=t10.a); id select_type table type possible_keys key key_len ref rows filtered Extra -1 PRIMARY t1 ALL NULL NULL NULL NULL 3 100.00 Using where -1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.a 1 100.00 Using where; Using join buffer +1 PRIMARY t1 ALL NULL NULL NULL NULL 3 100.00 +1 PRIMARY t10 eq_ref PRIMARY PRIMARY 4 test.t1.a 1 100.00 Using join buffer 1 PRIMARY t12 eq_ref PRIMARY PRIMARY 4 test.t10.a 1 100.00 Using index Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,`test`.`t1`.`b` AS `b` from `test`.`t10` join `test`.`t12` join `test`.`t1` where ((`test`.`t10`.`pk` = `test`.`t1`.`a`) and (`test`.`t12`.`pk` = `test`.`t10`.`a`)) @@ -549,7 +549,7 @@ (SELECT t1.pk FROM t0 t1 JOIN t0 t2 ON t2.vkey = t1.vnokey); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t0 ALL PRIMARY NULL NULL NULL 5 100.00 -1 PRIMARY t1 eq_ref PRIMARY PRIMARY 4 test.t0.pk 1 100.00 Using where; Using join buffer +1 PRIMARY t1 eq_ref PRIMARY PRIMARY 4 test.t0.pk 1 100.00 Using join buffer 1 PRIMARY t2 ref vkey vkey 4 test.t1.vnokey 2 100.00 Using index; FirstMatch(t1) Warnings: Note 1003 select `test`.`t0`.`vkey` AS `vkey` from `test`.`t0` `t1` semi join (`test`.`t0` `t2`) join `test`.`t0` where ((`test`.`t2`.`vkey` = `test`.`t1`.`vnokey`) and (`test`.`t1`.`pk` = `test`.`t0`.`pk`)) === modified file 'mysql-test/r/view.result' --- a/mysql-test/r/view.result 2009-12-15 07:16:46 +0000 +++ b/mysql-test/r/view.result 2010-02-17 10:47:55 +0000 @@ -2342,11 +2342,11 @@ EXPLAIN SELECT t1.* FROM t1 JOIN t2 WHERE t1.a=t2.a AND t1.b=t2.b AND t1.a=1; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t1 ref a a 5 const 1 Using index -1 SIMPLE t2 ref a a 10 const,test.t1.b 2 Using index +1 SIMPLE t2 ref a a 10 const,test.t1.b 1 Using index EXPLAIN SELECT * FROM v1 WHERE a=1; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t1 ref a a 5 const 1 Using index -1 SIMPLE t2 ref a a 10 const,test.t1.b 2 Using index +1 SIMPLE t2 ref a a 10 const,test.t1.b 1 Using index EXPLAIN SELECT * FROM v2 WHERE a=1; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t1 ref a a 5 const 1 Using index

1 0

[Maria-developers] Rev 2759: Subquery optimizations backport: in file:///home/psergey/dev/maria-5.3-subqueries-r7/
by Sergey Petrunya 17 Feb '10

17 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r7/ ------------------------------------------------------------ revno: 2759 revision-id: psergey(a)askmonty.org-20100217100527-k81b7torhmj99moy parent: psergey(a)askmonty.org-20100215215306-hc0levm9ag1lv1b1 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r7 timestamp: Wed 2010-02-17 13:05:27 +0300 message: Subquery optimizations backport: - Update test results - More comments - Add Item_in_optimizer::transform() which was lost in backport === modified file 'mysql-test/r/subselect.result' --- a/mysql-test/r/subselect.result 2010-01-17 20:52:20 +0000 +++ b/mysql-test/r/subselect.result 2010-02-17 10:05:27 +0000 @@ -1377,7 +1377,7 @@ 2 DEPENDENT SUBQUERY t1 index_subquery a a 5 func 1001 100.00 Using index; Using where Warnings: Note 1003 select `test`.`t2`.`a` AS `a` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`a`,<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on a where ((`test`.`t1`.`b` <> 30) and (<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`))))) -drop table t1, t2, t3; +drop table t0, t1, t2, t3; create table t1 (a int, b int); create table t2 (a int, b int); create table t3 (a int, b int); === modified file 'mysql-test/r/subselect3.result' --- a/mysql-test/r/subselect3.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect3.result 2010-02-17 10:05:27 +0000 @@ -873,7 +873,7 @@ Note 1276 Field or reference 'test.t1.a' of SELECT #3 was resolved in SELECT #2 Note 1276 Field or reference 'test.t1.c' of SELECT #3 was resolved in SELECT #2 Error 1054 Unknown column 'c' in 'field list' -Note 1003 select `c` AS `c` from (select (select count(`test`.`t1`.`a`) AS `COUNT(a)` from dual group by `c`) AS `(SELECT COUNT(a) FROM +Note 1003 select `c` AS `c` from (select (select count(`test`.`t1`.`a`) AS `COUNT(a)` from (select count(`test`.`t1`.`b`) AS `COUNT(b)` from `test`.`t1`) `x` group by `t1`.`c`) AS `(SELECT COUNT(a) FROM (SELECT COUNT(b) FROM t1) AS x GROUP BY c )` from `test`.`t1` group by `test`.`t1`.`b`) `y` DROP TABLE t1; @@ -1117,7 +1117,7 @@ explain select * from t3 where a in (select t1.kp1 from t1,t4 where kp1<20 and t4.pk=t1.c); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 range kp1 kp1 5 NULL 48 Using index condition; Using where; Using MRR; LooseScan +1 PRIMARY t1 range kp1 kp1 5 NULL 48 Using index condition; Using MRR; LooseScan 1 PRIMARY t4 eq_ref PRIMARY PRIMARY 4 test.t1.c 1 Using index; FirstMatch(t1) 1 PRIMARY t3 ALL NULL NULL NULL NULL 100 Using where; Using join buffer drop table t1, t3, t4; === modified file 'mysql-test/r/subselect4.result' --- a/mysql-test/r/subselect4.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect4.result 2010-02-17 10:05:27 +0000 @@ -13,9 +13,9 @@ WHERE NOT EXISTS (SELECT 1 FROM t2 WHERE 1 = (SELECT MIN(t2.b) FROM t3)) ORDER BY count(*); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 index NULL a 5 NULL 2 Using index; Using temporary +1 PRIMARY t1 index NULL a 5 NULL 2 Using where; Using index; Using temporary 2 DEPENDENT SUBQUERY t2 ALL NULL NULL NULL NULL 2 Using where -3 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL no matching row in const table +3 DEPENDENT SUBQUERY t3 system NULL NULL NULL NULL 0 const row not found # should not crash the next statement SELECT 1 FROM t1 WHERE NOT EXISTS (SELECT 1 FROM t2 WHERE 1 = (SELECT MIN(t2.b) FROM t3)) @@ -77,10 +77,10 @@ EXPLAIN EXTENDED SELECT (SELECT 1 FROM t2 WHERE d = c) AS RESULT FROM t1 ; id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 system NULL NULL NULL NULL 1 100.00 -2 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables +2 DEPENDENT SUBQUERY t2 ref d d 5 const 1 100.00 Using index Warnings: Note 1276 Field or reference 'test.t1.c' of SELECT #2 was resolved in SELECT #1 -Note 1003 select (select 1 AS `1` from `test`.`t2` where (`test`.`t2`.`d` = '0')) AS `RESULT` from dual +Note 1003 select (select 1 AS `1` from `test`.`t2` where (`test`.`t2`.`d` = '0')) AS `RESULT` from `test`.`t1` first equivalent variant SELECT (SELECT 1 FROM t2 WHERE d = IFNULL(c,NULL)) AS RESULT FROM t1 GROUP BY c ; RESULT @@ -88,10 +88,10 @@ EXPLAIN EXTENDED SELECT (SELECT 1 FROM t2 WHERE d = IFNULL(c,NULL)) AS RESULT FROM t1 GROUP BY c; id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 system NULL NULL NULL NULL 1 100.00 -2 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables +2 DEPENDENT SUBQUERY t2 ref d d 5 const 1 100.00 Using where; Using index Warnings: Note 1276 Field or reference 'test.t1.c' of SELECT #2 was resolved in SELECT #1 -Note 1003 select (select 1 AS `1` from `test`.`t2` where (`test`.`t2`.`d` = ifnull('0',NULL))) AS `RESULT` from dual group by '0' +Note 1003 select (select 1 AS `1` from `test`.`t2` where (`test`.`t2`.`d` = ifnull('0',NULL))) AS `RESULT` from `test`.`t1` group by '0' second equivalent variant SELECT (SELECT 1 FROM t2 WHERE d = c) AS RESULT FROM t1 GROUP BY c ; RESULT @@ -99,10 +99,10 @@ EXPLAIN EXTENDED SELECT (SELECT 1 FROM t2 WHERE d = c) AS RESULT FROM t1 GROUP BY c ; id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 system NULL NULL NULL NULL 1 100.00 -2 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables +2 DEPENDENT SUBQUERY t2 ref d d 5 const 1 100.00 Using index Warnings: Note 1276 Field or reference 'test.t1.c' of SELECT #2 was resolved in SELECT #1 -Note 1003 select (select 1 AS `1` from `test`.`t2` where (`test`.`t2`.`d` = '0')) AS `RESULT` from dual group by '0' +Note 1003 select (select 1 AS `1` from `test`.`t2` where (`test`.`t2`.`d` = '0')) AS `RESULT` from `test`.`t1` group by '0' DROP TABLE t1,t2; # # BUG#45928 "Differing query results depending on MRR and === modified file 'sql/item.h' --- a/sql/item.h 2010-02-11 23:59:58 +0000 +++ b/sql/item.h 2010-02-17 10:05:27 +0000 @@ -2817,6 +2817,17 @@ }; +/* + Cached_item_XXX objects are not exactly caches. They do the following: + + Each Cached_item_XXX object has + - its source item + - saved value of the source item + - cmp() method that compares the saved value with the current value of the + source item, and if they were not equal saves item's value into the saved + value. +*/ + class Cached_item :public Sql_alloc { public: === modified file 'sql/item_cmpfunc.cc' --- a/sql/item_cmpfunc.cc 2010-01-17 14:55:08 +0000 +++ b/sql/item_cmpfunc.cc 2010-02-17 10:05:27 +0000 @@ -1649,6 +1649,70 @@ } +/** + Transform an Item_in_optimizer and its arguments with a callback function. + + @param transformer the transformer callback function to be applied to the + nodes of the tree of the object + @param parameter to be passed to the transformer + + @detail + Recursively transform the left and the right operand of this Item. The + Right operand is an Item_in_subselect or its subclass. To avoid the + creation of new Items, we use the fact the the left operand of the + Item_in_subselect is the same as the one of 'this', so instead of + transforming its operand, we just assign the left operand of the + Item_in_subselect to be equal to the left operand of 'this'. + The transformation is not applied further to the subquery operand + if the IN predicate. + + @returns + @retval pointer to the transformed item + @retval NULL if an error occurred +*/ + +Item *Item_in_optimizer::transform(Item_transformer transformer, uchar *argument) +{ + Item *new_item; + + DBUG_ASSERT(!current_thd->is_stmt_prepare()); + DBUG_ASSERT(arg_count == 2); + + /* Transform the left IN operand. */ + new_item= (*args)->transform(transformer, argument); + if (!new_item) + return 0; + /* + THD::change_item_tree() should be called only if the tree was + really transformed, i.e. when a new item has been created. + Otherwise we'll be allocating a lot of unnecessary memory for + change records at each execution. + */ + if ((*args) != new_item) + current_thd->change_item_tree(args, new_item); + + /* + Transform the right IN operand which should be an Item_in_subselect or a + subclass of it. The left operand of the IN must be the same as the left + operand of this Item_in_optimizer, so in this case there is no further + transformation, we only make both operands the same. + TODO: is it the way it should be? + */ + DBUG_ASSERT((args[1])->type() == Item::SUBSELECT_ITEM && + (((Item_subselect*)(args[1]))->substype() == + Item_subselect::IN_SUBS || + ((Item_subselect*)(args[1]))->substype() == + Item_subselect::ALL_SUBS || + ((Item_subselect*)(args[1]))->substype() == + Item_subselect::ANY_SUBS)); + + Item_in_subselect *in_arg= (Item_in_subselect*)args[1]; + in_arg->left_expr= args[0]; + + return (this->*transformer)(argument); +} + + longlong Item_func_eq::val_int() { DBUG_ASSERT(fixed == 1); === modified file 'sql/item_cmpfunc.h' --- a/sql/item_cmpfunc.h 2010-02-11 21:58:23 +0000 +++ b/sql/item_cmpfunc.h 2010-02-17 10:05:27 +0000 @@ -241,6 +241,7 @@ const char *func_name() const { return "<in_optimizer>"; } Item_cache **get_cache() { return &cache; } void keep_top_level_cache(); + Item *transform(Item_transformer transformer, uchar *arg); }; class Comp_creator === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-02-15 21:53:06 +0000 +++ b/sql/item_subselect.cc 2010-02-17 10:05:27 +0000 @@ -1312,7 +1312,7 @@ (char *)in_left_expr_name); master_unit->uncacheable|= UNCACHEABLE_DEPENDENT; - select_lex->uncacheable|= UNCACHEABLE_DEPENDENT; + //psergey: placed then removed: select_lex->uncacheable|= UNCACHEABLE_DEPENDENT; } if (!abort_on_null && left_expr->maybe_null && !pushed_cond_guards) === modified file 'sql/opt_subselect.cc' --- a/sql/opt_subselect.cc 2010-02-15 21:53:06 +0000 +++ b/sql/opt_subselect.cc 2010-02-17 10:05:27 +0000 @@ -120,7 +120,7 @@ SELECT_LEX *current= thd->lex->current_select; thd->lex->current_select= current->return_after_parsing(); char const *save_where= thd->where; - thd->where= " IN/ALL/ANY subquery"; + thd->where= "IN/ALL/ANY subquery"; bool failure= !in_subs->left_expr->fixed && in_subs->left_expr->fix_fields(thd, &in_subs->left_expr);

1 0

[Maria-developers] "Diff too large for email" problem
by Oleksandr Byelkin 17 Feb '10

17 Feb '10

Hi! Them who use bazaar email plugn. Most 'feature' diffs are too large so IMHO it is good to put following to ~/.bazaar/bazaar.conf ([DEFAULT] section): post_commit_difflimit = 10000 (or what limit you want)

2 1

[Maria-developers] bzr commit into Mariadb 5.2, with Maria 2.0:maria/5.2 branch (igor:2742) WL#86
by Igor Babaev 16 Feb '10

16 Feb '10

#At lp:maria/5.2 based on revid:sanja@askmonty.org-20100212131228-bgxli0wfybhjkvg9 2742 Igor Babaev 2010-02-16 WL#86: Partitioned key cache for MyISAM. This is the base patch for the task. added: mysql-test/r/select_pkeycache.result mysql-test/t/select_pkeycache-master.opt mysql-test/t/select_pkeycache.test modified: include/keycache.h mysql-test/r/information_schema.result mysql-test/r/information_schema_all_engines.result mysql-test/r/key_cache.result mysql-test/t/key_cache.test mysys/mf_keycache.c sql/handler.cc sql/handler.h sql/mysqld.cc sql/set_var.cc sql/set_var.h sql/sql_show.cc sql/sql_test.cc sql/table.h storage/myisam/mi_check.c storage/myisam/mi_close.c storage/myisam/mi_delete_all.c storage/myisam/mi_extra.c storage/myisam/mi_keycache.c storage/myisam/mi_locking.c storage/myisam/mi_page.c storage/myisam/mi_panic.c storage/myisam/mi_preload.c storage/myisam/mi_test1.c storage/myisam/mi_test2.c storage/myisam/mi_test3.c storage/myisam/myisam_ftdump.c storage/myisam/myisamchk.c storage/myisam/myisamdef.h storage/myisam/myisamlog.c === modified file 'include/keycache.h' --- a/include/keycache.h 2009-12-03 11:19:05 +0000 +++ b/include/keycache.h 2010-02-16 16:41:11 +0000 @@ -19,96 +19,121 @@ #define _keycache_h C_MODE_START -/* declare structures that is used by st_key_cache */ -struct st_block_link; -typedef struct st_block_link BLOCK_LINK; -struct st_keycache_page; -typedef struct st_keycache_page KEYCACHE_PAGE; -struct st_hash_link; -typedef struct st_hash_link HASH_LINK; -/* info about requests in a waiting queue */ -typedef struct st_keycache_wqueue +/* + Currently the default key cache is created as non-partitioned at + the start of the server unless the server is started with the parameter + --key-cache-partitions that is greater than 0 +*/ + +#define DEFAULT_KEY_CACHE_PARTITIONS 0 + +/* + MAX_KEY_CACHE_PARTITIONS cannot be greater than + sizeof(MYISAM_SHARE::dirty_part_map) + Currently sizeof(MYISAM_SHARE::dirty_part_map)=sizeof(ulonglong) +*/ + +#define MAX_KEY_CACHE_PARTITIONS 64 + + +/* The structure to get statistical data about a key cache */ + +typedef struct st_key_cache_statistics +{ + ulonglong mem_size; /* memory for cache buffers/auxiliary structures */ + ulonglong block_size; /* size of the each buffers in the key cache */ + ulonglong blocks_used; /* maximum number of used blocks/buffers */ + ulonglong blocks_unused; /* number of currently unused blocks */ + ulonglong blocks_changed; /* number of currently dirty blocks */ + ulonglong read_requests; /* number of read requests (read hits) */ + ulonglong reads; /* number of actual reads from files into buffers */ + ulonglong write_requests; /* number of write requests (write hits) */ + ulonglong writes; /* number of actual writes from buffers into files */ +} KEY_CACHE_STATISTICS; + +/* The type of a key cache object */ +typedef enum key_cache_type { - struct st_my_thread_var *last_thread; /* circular list of waiting threads */ -} KEYCACHE_WQUEUE; + SIMPLE_KEY_CACHE, + PARTITIONED_KEY_CACHE +} KEY_CACHE_TYPE; -#define CHANGED_BLOCKS_HASH 128 /* must be power of 2 */ /* - The key cache structure - It also contains read-only statistics parameters. + An object of the type KEY_CACHE_FUNCS contains pointers to all functions + from the key cache interface. + Currently a key cache can be of two types: simple and partitioned. + For each of them its own static structure of the type KEY_CACHE_FUNCS is + defined . The structures contain the pointers to the implementations of + the interface functions used by simple key caches and partitioned key + caches respectively. Pointers to these structures are assigned to key cache + objects at the time of their creation. */ +typedef struct st_key_cache_funcs +{ + int (*init) (void *, uint key_cache_block_size, + size_t use_mem, uint division_limit, uint age_threshold); + int (*resize) (void *, uint key_cache_block_size, + size_t use_mem, uint division_limit, uint age_threshold); + void (*change_param) (void *keycache_cb, + uint division_limit, uint age_threshold); + uchar* (*read) (void *keycache_cb, + File file, my_off_t filepos, int level, + uchar *buff, uint length, + uint block_length, int return_buffer); + int (*insert) (void *keycache_cb, + File file, my_off_t filepos, int level, + uchar *buff, uint length); + int (*write) (void *keycache_cb, + File file, void *file_extra, + my_off_t filepos, int level, + uchar *buff, uint length, + uint block_length, int force_write); + int (*flush) (void *keycache_cb, + int file, void *file_extra, + enum flush_type type); + int (*reset_counters) (const char *name, void *keycache_cb); + void (*end) (void *keycache_cb, my_bool cleanup); + void (*get_stats) (void *keycache_cb, uint partition_no, + KEY_CACHE_STATISTICS *key_cache_stats); + ulonglong (*get_stat_val) (void *keycache_cb, uint var_no); +} KEY_CACHE_FUNCS; + + typedef struct st_key_cache { - my_bool key_cache_inited; - my_bool in_resize; /* true during resize operation */ - my_bool resize_in_flush; /* true during flush of resize operation */ + KEY_CACHE_TYPE key_cache_type; /* type of the key cache used for debugging */ + void *keycache_cb; /* control block of the used key cache */ + KEY_CACHE_FUNCS *interface_funcs; /* interface functions of the key cache */ + ulonglong param_buff_size; /* size the memory allocated for the cache */ + ulong param_block_size; /* size of the blocks in the key cache */ + ulong param_division_limit; /* min. percentage of warm blocks */ + ulong param_age_threshold; /* determines when hot block is downgraded */ + ulong param_partitions; /* number of the key cache partitions */ + my_bool key_cache_inited; /* <=> key cache has been created */ my_bool can_be_used; /* usage of cache for read/write is allowed */ - size_t key_cache_mem_size; /* specified size of the cache memory */ - uint key_cache_block_size; /* size of the page buffer of a cache block */ - ulong min_warm_blocks; /* min number of warm blocks; */ - ulong age_threshold; /* age threshold for hot blocks */ - ulonglong keycache_time; /* total number of block link operations */ - uint hash_entries; /* max number of entries in the hash table */ - int hash_links; /* max number of hash links */ - int hash_links_used; /* number of hash links currently used */ - int disk_blocks; /* max number of blocks in the cache */ - ulong blocks_used; /* maximum number of concurrently used blocks */ - ulong blocks_unused; /* number of currently unused blocks */ - ulong blocks_changed; /* number of currently dirty blocks */ - ulong warm_blocks; /* number of blocks in warm sub-chain */ - ulong cnt_for_resize_op; /* counter to block resize operation */ - long blocks_available; /* number of blocks available in the LRU chain */ - HASH_LINK **hash_root; /* arr. of entries into hash table buckets */ - HASH_LINK *hash_link_root; /* memory for hash table links */ - HASH_LINK *free_hash_list; /* list of free hash links */ - BLOCK_LINK *free_block_list; /* list of free blocks */ - BLOCK_LINK *block_root; /* memory for block links */ - uchar HUGE_PTR *block_mem; /* memory for block buffers */ - BLOCK_LINK *used_last; /* ptr to the last block of the LRU chain */ - BLOCK_LINK *used_ins; /* ptr to the insertion block in LRU chain */ - pthread_mutex_t cache_lock; /* to lock access to the cache structure */ - KEYCACHE_WQUEUE resize_queue; /* threads waiting during resize operation */ - /* - Waiting for a zero resize count. Using a queue for symmetry though - only one thread can wait here. - */ - KEYCACHE_WQUEUE waiting_for_resize_cnt; - KEYCACHE_WQUEUE waiting_for_hash_link; /* waiting for a free hash link */ - KEYCACHE_WQUEUE waiting_for_block; /* requests waiting for a free block */ - BLOCK_LINK *changed_blocks[CHANGED_BLOCKS_HASH]; /* hash for dirty file bl.*/ - BLOCK_LINK *file_blocks[CHANGED_BLOCKS_HASH]; /* hash for other file bl.*/ - - /* - The following variables are and variables used to hold parameters for - initializing the key cache. - */ - - ulonglong param_buff_size; /* size the memory allocated for the cache */ - ulong param_block_size; /* size of the blocks in the key cache */ - ulong param_division_limit; /* min. percentage of warm blocks */ - ulong param_age_threshold; /* determines when hot block is downgraded */ - - /* Statistics variables. These are reset in reset_key_cache_counters(). */ - ulong global_blocks_changed; /* number of currently dirty blocks */ + my_bool in_init; /* Set to 1 in MySQL during init/resize */ + uint partitions; /* actual number of partitions */ + size_t key_cache_mem_size; /* specified size of the cache memory */ + ulong blocks_used; /* maximum number of concurrently used blocks */ + ulong blocks_unused; /* number of currently unused blocks */ + ulong global_blocks_changed; /* number of currently dirty blocks */ ulonglong global_cache_w_requests;/* number of write requests (write hits) */ ulonglong global_cache_write; /* number of writes from cache to files */ ulonglong global_cache_r_requests;/* number of read requests (read hits) */ ulonglong global_cache_read; /* number of reads from files to cache */ - - int blocks; /* max number of blocks in the cache */ - my_bool in_init; /* Set to 1 in MySQL during init/resize */ } KEY_CACHE; + /* The default key cache */ extern KEY_CACHE dflt_key_cache_var, *dflt_key_cache; extern int init_key_cache(KEY_CACHE *keycache, uint key_cache_block_size, size_t use_mem, uint division_limit, - uint age_threshold); + uint age_threshold, uint partitions); extern int resize_key_cache(KEY_CACHE *keycache, uint key_cache_block_size, size_t use_mem, uint division_limit, uint age_threshold); @@ -122,12 +147,18 @@ extern int key_cache_insert(KEY_CACHE *k File file, my_off_t filepos, int level, uchar *buff, uint length); extern int key_cache_write(KEY_CACHE *keycache, - File file, my_off_t filepos, int level, + File file, void *file_extra, + my_off_t filepos, int level, uchar *buff, uint length, - uint block_length,int force_write); + uint block_length, int force_write); extern int flush_key_blocks(KEY_CACHE *keycache, - int file, enum flush_type type); + int file, void *file_extra, + enum flush_type type); extern void end_key_cache(KEY_CACHE *keycache, my_bool cleanup); +extern void get_key_cache_statistics(KEY_CACHE *keycache, + uint partition_no, + KEY_CACHE_STATISTICS *key_cache_stats); +extern ulonglong get_key_cache_stat_value(KEY_CACHE *keycache, uint var_no); /* Functions to handle multiple key caches */ extern my_bool multi_keycache_init(void); @@ -140,5 +171,11 @@ extern void multi_key_cache_change(KEY_C KEY_CACHE *new_data); extern int reset_key_cache_counters(const char *name, KEY_CACHE *key_cache); +extern int repartition_key_cache(KEY_CACHE *keycache, + uint key_cache_block_size, + size_t use_mem, + uint division_limit, + uint age_threshold, + uint partitions); C_MODE_END #endif /* _keycache_h */ === modified file 'mysql-test/r/information_schema.result' --- a/mysql-test/r/information_schema.result 2010-02-01 06:14:12 +0000 +++ b/mysql-test/r/information_schema.result 2010-02-16 16:41:11 +0000 @@ -69,6 +69,7 @@ INNODB_LOCK_WAITS INNODB_RSEG INNODB_TABLE_STATS INNODB_TRX +KEY_CACHES KEY_COLUMN_USAGE PARTITIONS PLUGINS === modified file 'mysql-test/r/information_schema_all_engines.result' --- a/mysql-test/r/information_schema_all_engines.result 2010-02-01 06:14:12 +0000 +++ b/mysql-test/r/information_schema_all_engines.result 2010-02-16 16:41:11 +0000 @@ -13,6 +13,7 @@ FILES GLOBAL_STATUS GLOBAL_VARIABLES INDEX_STATISTICS +KEY_CACHES KEY_COLUMN_USAGE PARTITIONS PLUGINS @@ -76,6 +77,7 @@ FILES TABLE_SCHEMA GLOBAL_STATUS VARIABLE_NAME GLOBAL_VARIABLES VARIABLE_NAME INDEX_STATISTICS TABLE_SCHEMA +KEY_CACHES KEY_CACHE_NAME KEY_COLUMN_USAGE CONSTRAINT_SCHEMA PARTITIONS TABLE_SCHEMA PLUGINS PLUGIN_NAME @@ -139,6 +141,7 @@ FILES TABLE_SCHEMA GLOBAL_STATUS VARIABLE_NAME GLOBAL_VARIABLES VARIABLE_NAME INDEX_STATISTICS TABLE_SCHEMA +KEY_CACHES KEY_CACHE_NAME KEY_COLUMN_USAGE CONSTRAINT_SCHEMA PARTITIONS TABLE_SCHEMA PLUGINS PLUGIN_NAME @@ -221,6 +224,7 @@ INNODB_LOCK_WAITS information_schema.INN INNODB_RSEG information_schema.INNODB_RSEG 1 INNODB_TABLE_STATS information_schema.INNODB_TABLE_STATS 1 INNODB_TRX information_schema.INNODB_TRX 1 +KEY_CACHES information_schema.KEY_CACHES 1 KEY_COLUMN_USAGE information_schema.KEY_COLUMN_USAGE 1 PARTITIONS information_schema.PARTITIONS 1 PBXT_STATISTICS information_schema.PBXT_STATISTICS 1 @@ -259,6 +263,7 @@ Database: information_schema | GLOBAL_STATUS | | GLOBAL_VARIABLES | | INDEX_STATISTICS | +| KEY_CACHES | | KEY_COLUMN_USAGE | | PARTITIONS | | PLUGINS | @@ -312,6 +317,7 @@ Database: INFORMATION_SCHEMA | GLOBAL_STATUS | | GLOBAL_VARIABLES | | INDEX_STATISTICS | +| KEY_CACHES | | KEY_COLUMN_USAGE | | PARTITIONS | | PLUGINS | @@ -357,5 +363,5 @@ Wildcard: inf_rmation_schema +--------------------+ SELECT table_schema, count(*) FROM information_schema.TABLES WHERE table_schema IN ('mysql', 'INFORMATION_SCHEMA', 'test', 'mysqltest') AND table_name<>'ndb_binlog_index' AND table_name<>'ndb_apply_status' GROUP BY TABLE_SCHEMA; table_schema count(*) -information_schema 48 +information_schema 49 mysql 22 === modified file 'mysql-test/r/key_cache.result' --- a/mysql-test/r/key_cache.result 2009-03-16 19:54:50 +0000 +++ b/mysql-test/r/key_cache.result 2010-02-16 16:41:11 +0000 @@ -1,5 +1,7 @@ drop table if exists t1, t2, t3; -SET @save_key_buffer=@@key_buffer_size; +SET @save_key_buffer_size=@@key_buffer_size; +SET @save_key_cache_block_size=@@key_cache_block_size; +SET @save_key_cache_partitions=@@key_cache_partitions; SELECT @@key_buffer_size, @@small.key_buffer_size; @@key_buffer_size @@small.key_buffer_size 2097152 131072 @@ -37,7 +39,7 @@ SELECT @@small.key_buffer_size; SELECT @@medium.key_buffer_size; @@medium.key_buffer_size 0 -SET @@global.key_buffer_size=@save_key_buffer; +SET @@global.key_buffer_size=@save_key_buffer_size; SELECT @@default.key_buffer_size; ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'default.key_buffer_size' at line 1 SELECT @@skr.storage_engine="test"; @@ -366,3 +368,537 @@ Variable_name Value key_cache_block_size 1536 SET GLOBAL key_cache_block_size= @bug28478_key_cache_block_size; DROP TABLE t1; +set global key_buffer_size=@save_key_buffer_size; +set global key_cache_block_size=@save_key_cache_block_size; +select @@key_buffer_size; +@@key_buffer_size +2097152 +select @@key_cache_block_size; +@@key_cache_block_size +1024 +select @@key_cache_partitions; +@@key_cache_partitions +0 +create table t1 ( +p int not null auto_increment primary key, +a char(10)); +create table t2 ( +p int not null auto_increment primary key, +i int, a char(10), key k1(i), key k2(a)); +select @@key_cache_partitions; +@@key_cache_partitions +0 +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default NULL NULL 2097152 1024 0 # 0 0 0 0 0 +small NULL NULL 1048576 1024 1 # 0 1 0 2 1 +insert into t1 values (1, 'qqqq'), (2, 'yyyy'); +insert into t2 values (1, 1, 'qqqq'), (2, 1, 'pppp'), +(3, 1, 'yyyy'), (4, 3, 'zzzz'); +select * from t1; +p a +1 qqqq +2 yyyy +select * from t2; +p i a +1 1 qqqq +2 1 pppp +3 1 yyyy +4 3 zzzz +update t1 set p=3 where p=1; +update t2 set i=2 where i=1; +show status like 'key_%'; +Variable_name Value +Key_blocks_not_flushed 0 +Key_blocks_unused KEY_BLOCKS_UNUSED +Key_blocks_used 4 +Key_read_requests 22 +Key_reads 0 +Key_write_requests 26 +Key_writes 6 +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default NULL NULL 2097152 1024 4 # 0 22 0 26 6 +small NULL NULL 1048576 1024 1 # 0 1 0 2 1 +delete from t2 where a='zzzz'; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default NULL NULL 2097152 1024 4 # 0 29 0 32 9 +small NULL NULL 1048576 1024 1 # 0 1 0 2 1 +delete from t1; +delete from t2; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default NULL NULL 2097152 1024 4 # 0 29 0 32 9 +small NULL NULL 1048576 1024 1 # 0 1 0 2 1 +set global key_cache_partitions=2; +select @@key_cache_partitions; +@@key_cache_partitions +2 +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 1048576 1024 0 # 0 0 0 0 0 +default 2 2 1048576 1024 0 # 0 0 0 0 0 +default 2 NULL 2097152 1024 0 # 0 0 0 0 0 +small NULL NULL 1048576 1024 1 # 0 1 0 2 1 +insert into t1 values (1, 'qqqq'), (2, 'yyyy'); +insert into t2 values (1, 1, 'qqqq'), (2, 1, 'pppp'), +(3, 1, 'yyyy'), (4, 3, 'zzzz'); +select * from t1; +p a +1 qqqq +2 yyyy +select * from t2; +p i a +1 1 qqqq +2 1 pppp +3 1 yyyy +4 3 zzzz +update t1 set p=3 where p=1; +update t2 set i=2 where i=1; +show status like 'key_%'; +Variable_name Value +Key_blocks_not_flushed 0 +Key_blocks_unused KEY_BLOCKS_UNUSED +Key_blocks_used 4 +Key_read_requests 22 +Key_reads 0 +Key_write_requests 26 +Key_writes 6 +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 1048576 1024 3 # 0 10 0 13 4 +default 2 2 1048576 1024 1 # 0 12 0 13 2 +default 2 NULL 2097152 1024 4 # 0 22 0 26 6 +small NULL NULL 1048576 1024 1 # 0 1 0 2 1 +delete from t1; +delete from t2; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 1048576 1024 3 # 0 10 0 13 4 +default 2 2 1048576 1024 1 # 0 12 0 13 2 +default 2 NULL 2097152 1024 4 # 0 22 0 26 6 +small NULL NULL 1048576 1024 1 # 0 1 0 2 1 +set global key_cache_partitions=1; +select @@key_cache_partitions; +@@key_cache_partitions +1 +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 1 1 2097152 1024 0 # 0 0 0 0 0 +default 1 NULL 2097152 1024 0 # 0 0 0 0 0 +small NULL NULL 1048576 1024 1 # 0 1 0 2 1 +insert into t1 values (1, 'qqqq'), (2, 'yyyy'); +insert into t2 values (1, 1, 'qqqq'), (2, 1, 'pppp'), +(3, 1, 'yyyy'), (4, 3, 'zzzz'); +select * from t1; +p a +1 qqqq +2 yyyy +select * from t2; +p i a +1 1 qqqq +2 1 pppp +3 1 yyyy +4 3 zzzz +update t1 set p=3 where p=1; +update t2 set i=2 where i=1; +show status like 'key_%'; +Variable_name Value +Key_blocks_not_flushed 0 +Key_blocks_unused KEY_BLOCKS_UNUSED +Key_blocks_used 4 +Key_read_requests 22 +Key_reads 0 +Key_write_requests 26 +Key_writes 6 +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 1 1 2097152 1024 4 # 0 22 0 26 6 +default 1 NULL 2097152 1024 4 # 0 22 0 26 6 +small NULL NULL 1048576 1024 1 # 0 1 0 2 1 +delete from t1; +delete from t2; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 1 1 2097152 1024 4 # 0 22 0 26 6 +default 1 NULL 2097152 1024 4 # 0 22 0 26 6 +small NULL NULL 1048576 1024 1 # 0 1 0 2 1 +flush tables; +flush status; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 1 1 2097152 1024 4 # 0 0 0 0 0 +default 1 NULL 2097152 1024 4 # 0 0 0 0 0 +small NULL NULL 1048576 1024 1 # 0 0 0 0 0 +set global key_buffer_size=32*1024; +select @@key_buffer_size; +@@key_buffer_size +32768 +set global key_cache_partitions=2; +select @@key_cache_partitions; +@@key_cache_partitions +2 +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 0 # 0 0 0 0 0 +default 2 2 16384 1024 0 # 0 0 0 0 0 +default 2 NULL 32768 1024 0 # 0 0 0 0 0 +small NULL NULL 1048576 1024 1 # 0 0 0 0 0 +insert into t1 values (1, 'qqqq'), (2, 'yyyy'); +insert into t2 values (1, 1, 'qqqq'), (2, 1, 'pppp'), +(3, 1, 'yyyy'), (4, 3, 'zzzz'); +select * from t1; +p a +1 qqqq +2 yyyy +select * from t2; +p i a +1 1 qqqq +2 1 pppp +3 1 yyyy +4 3 zzzz +update t1 set p=3 where p=1; +update t2 set i=2 where i=1; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 1 # 0 12 0 13 2 +default 2 2 16384 1024 3 # 0 10 0 13 4 +default 2 NULL 32768 1024 4 # 0 22 0 26 6 +small NULL NULL 1048576 1024 1 # 0 0 0 0 0 +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 1951 # 1976 43 +default 2 2 16384 1024 # # 0 4782 # 1708 60 +default 2 NULL 32768 1024 # # 0 6733 # 3684 103 +small NULL NULL 1048576 1024 # # 0 0 # 0 0 +select * from t1 where p between 1010 and 1020 ; +p a +select * from t2 where p between 1010 and 1020 ; +p i a +1010 2 pppp +1011 2 yyyy +1012 3 zzzz +1013 2 qqqq +1014 2 pppp +1015 2 yyyy +1016 3 zzzz +1017 2 qqqq +1018 2 pppp +1019 2 yyyy +1020 3 zzzz +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 1954 # 1976 43 +default 2 2 16384 1024 # # 0 4796 # 1708 60 +default 2 NULL 32768 1024 # # 0 6750 # 3684 103 +small NULL NULL 1048576 1024 # # 0 0 # 0 0 +flush tables; +flush status; +update t1 set a='zzzz' where a='qqqq'; +update t2 set i=1 where i=2; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 940 10 939 10 +default 2 2 16384 1024 # # 0 2136 8 613 8 +default 2 NULL 32768 1024 # # 0 3076 18 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +set global keycache1.key_buffer_size=256*1024; +select @@keycache1.key_buffer_size; +@@keycache1.key_buffer_size +262144 +set global keycache1.key_cache_partitions=7; +select @@keycache1.key_cache_partitions; +@@keycache1.key_cache_partitions +7 +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 940 10 939 10 +default 2 2 16384 1024 # # 0 2136 8 613 8 +default 2 NULL 32768 1024 # # 0 3076 18 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +keycache1 7 1 37449 2048 # # 0 0 0 0 0 +keycache1 7 2 37449 2048 # # 0 0 0 0 0 +keycache1 7 3 37449 2048 # # 0 0 0 0 0 +keycache1 7 4 37449 2048 # # 0 0 0 0 0 +keycache1 7 5 37449 2048 # # 0 0 0 0 0 +keycache1 7 6 37449 2048 # # 0 0 0 0 0 +keycache1 7 7 37449 2048 # # 0 0 0 0 0 +keycache1 7 NULL 262143 2048 # # 0 0 0 0 0 +select * from information_schema.key_caches where key_cache_name like "key%"; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +keycache1 7 1 37449 2048 0 # 0 0 0 0 0 +keycache1 7 2 37449 2048 0 # 0 0 0 0 0 +keycache1 7 3 37449 2048 0 # 0 0 0 0 0 +keycache1 7 4 37449 2048 0 # 0 0 0 0 0 +keycache1 7 5 37449 2048 0 # 0 0 0 0 0 +keycache1 7 6 37449 2048 0 # 0 0 0 0 0 +keycache1 7 7 37449 2048 0 # 0 0 0 0 0 +keycache1 7 NULL 262143 2048 0 # 0 0 0 0 0 +cache index t1 key (`primary`) in keycache1; +Table Op Msg_type Msg_text +test.t1 assign_to_keycache status OK +explain select p from t1 where p between 1010 and 1020; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 range PRIMARY PRIMARY 4 NULL 1 Using where; Using index +select p from t1 where p between 1010 and 1020; +p +explain select i from t2 where p between 1010 and 1020; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 range PRIMARY PRIMARY 4 NULL 28 Using where +select i from t2 where p between 1010 and 1020; +i +1 +1 +3 +1 +1 +1 +3 +1 +1 +1 +3 +explain select count(*) from t1, t2 where t1.p = t2.i; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 index k1 k1 5 NULL 1024 Using index +1 SIMPLE t1 eq_ref PRIMARY PRIMARY 4 test.t2.i 1 Using index +select count(*) from t1, t2 where t1.p = t2.i; +count(*) +256 +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 966 12 939 10 +default 2 2 16384 1024 # # 0 2206 12 613 8 +default 2 NULL 32768 1024 # # 0 3172 24 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +keycache1 7 1 37449 2048 # # 0 2 1 0 0 +keycache1 7 2 37449 2048 # # 0 7 1 0 0 +keycache1 7 3 37449 2048 # # 0 0 0 0 0 +keycache1 7 4 37449 2048 # # 0 5 1 0 0 +keycache1 7 5 37449 2048 # # 0 0 0 0 0 +keycache1 7 6 37449 2048 # # 0 0 0 0 0 +keycache1 7 7 37449 2048 # # 0 0 0 0 0 +keycache1 7 NULL 262143 2048 # # 0 14 3 0 0 +select * from information_schema.key_caches where key_cache_name like "key%"; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +keycache1 7 1 37449 2048 1 # 0 2 1 0 0 +keycache1 7 2 37449 2048 1 # 0 7 1 0 0 +keycache1 7 3 37449 2048 0 # 0 0 0 0 0 +keycache1 7 4 37449 2048 1 # 0 5 1 0 0 +keycache1 7 5 37449 2048 0 # 0 0 0 0 0 +keycache1 7 6 37449 2048 0 # 0 0 0 0 0 +keycache1 7 7 37449 2048 0 # 0 0 0 0 0 +keycache1 7 NULL 262143 2048 3 # 0 14 3 0 0 +cache index t2 in keycache1; +Table Op Msg_type Msg_text +test.t2 assign_to_keycache status OK +update t2 set p=p+3000, i=2 where a='qqqq'; +select * from information_schema.key_caches where key_cache_name like "key%"; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +keycache1 7 1 37449 2048 3 # 0 44 3 43 2 +keycache1 7 2 37449 2048 4 # 0 61 4 51 1 +keycache1 7 3 37449 2048 4 # 0 177 4 176 3 +keycache1 7 4 37449 2048 4 # 0 122 4 119 3 +keycache1 7 5 37449 2048 4 # 0 840 4 335 4 +keycache1 7 6 37449 2048 3 # 0 627 3 133 3 +keycache1 7 7 37449 2048 3 # 0 211 3 214 3 +keycache1 7 NULL 262143 2048 25 # 0 2082 25 1071 19 +set global keycache2.key_buffer_size=1024*1024; +cache index t2 in keycache2; +Table Op Msg_type Msg_text +test.t2 assign_to_keycache status OK +insert into t2 values (2000, 3, 'yyyy'); +select * from information_schema.key_caches where key_cache_name like "keycache2"; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +keycache2 NULL NULL 1048576 1024 0 # 0 0 0 0 0 +select * from information_schema.key_caches where key_cache_name like "key%"; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +keycache1 7 1 37449 2048 3 # 0 44 3 43 2 +keycache1 7 2 37449 2048 4 # 0 61 4 51 1 +keycache1 7 3 37449 2048 4 # 0 177 4 176 3 +keycache1 7 4 37449 2048 4 # 0 122 4 119 3 +keycache1 7 5 37449 2048 4 # 0 840 4 335 4 +keycache1 7 6 37449 2048 3 # 0 627 3 133 3 +keycache1 7 7 37449 2048 3 # 0 211 3 214 3 +keycache1 7 NULL 262143 2048 25 # 0 2082 25 1071 19 +keycache2 NULL NULL 1048576 1024 0 # 0 0 0 0 0 +cache index t2 in keycache1; +Table Op Msg_type Msg_text +test.t2 assign_to_keycache status OK +update t2 set p=p+5000 where a='zzzz'; +select * from t2 where p between 1010 and 1020; +p i a +1010 1 pppp +1011 1 yyyy +1014 1 pppp +1015 1 yyyy +1018 1 pppp +1019 1 yyyy +explain select p from t2 where p between 1010 and 1020; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 range PRIMARY PRIMARY 4 NULL 7 Using where; Using index +select p from t2 where p between 1010 and 1020; +p +1010 +1011 +1014 +1015 +1018 +1019 +explain select i from t2 where a='yyyy' and i=3; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ref k1,k2 k1 5 const 188 Using where +select i from t2 where a='yyyy' and i=3; +i +3 +explain select a from t2 where a='yyyy' and i=3; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ref k1,k2 k1 5 const 188 Using where +select a from t2 where a='yyyy' and i=3 ; +a +yyyy +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 966 12 939 10 +default 2 2 16384 1024 # # 0 2206 12 613 8 +default 2 NULL 32768 1024 # # 0 3172 24 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +keycache1 7 1 37449 2048 # # 0 85 6 68 3 +keycache1 7 2 37449 2048 # # 0 122 6 102 2 +keycache1 7 3 37449 2048 # # 0 271 8 254 6 +keycache1 7 4 37449 2048 # # 0 179 6 170 4 +keycache1 7 5 37449 2048 # # 0 1445 7 416 6 +keycache1 7 6 37449 2048 # # 0 863 6 345 5 +keycache1 7 7 37449 2048 # # 0 236 4 239 4 +keycache1 7 NULL 262143 2048 # # 0 3201 43 1594 30 +keycache2 NULL NULL 1048576 1024 # # 0 0 0 0 0 +set global keycache1.key_cache_block_size=2*1024; +insert into t2 values (7000, 3, 'yyyy'); +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 966 12 939 10 +default 2 2 16384 1024 # # 0 2206 12 613 8 +default 2 NULL 32768 1024 # # 0 3172 24 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +keycache1 7 1 37449 2048 # # 0 1 1 1 1 +keycache1 7 2 37449 2048 # # 0 1 1 0 0 +keycache1 7 3 37449 2048 # # 0 0 0 0 0 +keycache1 7 4 37449 2048 # # 0 1 1 1 1 +keycache1 7 5 37449 2048 # # 0 1 1 0 0 +keycache1 7 6 37449 2048 # # 0 2 2 1 1 +keycache1 7 7 37449 2048 # # 0 0 0 0 0 +keycache1 7 NULL 262143 2048 # # 0 6 6 3 3 +keycache2 NULL NULL 1048576 1024 # # 0 0 0 0 0 +set global keycache1.key_cache_block_size=8*1024; +insert into t2 values (8000, 3, 'yyyy'); +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 966 12 939 10 +default 2 2 16384 1024 # # 0 2206 12 613 8 +default 2 NULL 32768 1024 # # 0 3172 24 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +keycache1 3 1 87381 8192 # # 0 1 1 1 1 +keycache1 3 2 87381 8192 # # 0 3 2 1 1 +keycache1 3 3 87381 8192 # # 0 2 2 1 1 +keycache1 3 NULL 262143 8192 # # 0 6 5 3 3 +keycache2 NULL NULL 1048576 1024 # # 0 0 0 0 0 +set global keycache1.key_buffer_size=64*1024; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 966 12 939 10 +default 2 2 16384 1024 # # 0 2206 12 613 8 +default 2 NULL 32768 1024 # # 0 3172 24 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +keycache2 NULL NULL 1048576 1024 # # 0 0 0 0 0 +set global keycache1.key_cache_block_size=2*1024; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 966 12 939 10 +default 2 2 16384 1024 # # 0 2206 12 613 8 +default 2 NULL 32768 1024 # # 0 3172 24 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +keycache1 3 1 21845 2048 # # 0 0 0 0 0 +keycache1 3 2 21845 2048 # # 0 0 0 0 0 +keycache1 3 3 21845 2048 # # 0 0 0 0 0 +keycache1 3 NULL 65535 2048 # # 0 0 0 0 0 +keycache2 NULL NULL 1048576 1024 # # 0 0 0 0 0 +set global keycache1.key_cache_block_size=8*1024; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 966 12 939 10 +default 2 2 16384 1024 # # 0 2206 12 613 8 +default 2 NULL 32768 1024 # # 0 3172 24 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +keycache2 NULL NULL 1048576 1024 # # 0 0 0 0 0 +set global keycache1.key_buffer_size=0; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 966 12 939 10 +default 2 2 16384 1024 # # 0 2206 12 613 8 +default 2 NULL 32768 1024 # # 0 3172 24 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +keycache2 NULL NULL 1048576 1024 # # 0 0 0 0 0 +set global keycache1.key_cache_block_size=8*1024; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 966 12 939 10 +default 2 2 16384 1024 # # 0 2206 12 613 8 +default 2 NULL 32768 1024 # # 0 3172 24 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +keycache2 NULL NULL 1048576 1024 # # 0 0 0 0 0 +set global keycache1.key_buffer_size=0; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 966 12 939 10 +default 2 2 16384 1024 # # 0 2206 12 613 8 +default 2 NULL 32768 1024 # # 0 3172 24 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +keycache2 NULL NULL 1048576 1024 # # 0 0 0 0 0 +set global keycache1.key_buffer_size=128*1024; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 966 12 939 10 +default 2 2 16384 1024 # # 0 2206 12 613 8 +default 2 NULL 32768 1024 # # 0 3172 24 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +keycache1 1 1 131072 8192 # # 0 0 0 0 0 +keycache1 1 NULL 131072 8192 # # 0 0 0 0 0 +keycache2 NULL NULL 1048576 1024 # # 0 0 0 0 0 +set global keycache1.key_cache_block_size=1024; +select * from information_schema.key_caches; +KEY_CACHE_NAME PARTITIONS PARTITION_NUMBER FULL_SIZE BLOCK_SIZE USED_BLOCKS UNUSED_BLOCKS DIRTY_BLOCKS READ_REQUESTS READS WRITE_REQUESTS WRITES +default 2 1 16384 1024 # # 0 966 12 939 10 +default 2 2 16384 1024 # # 0 2206 12 613 8 +default 2 NULL 32768 1024 # # 0 3172 24 1552 18 +small NULL NULL 1048576 1024 # # 0 0 0 0 0 +keycache1 7 1 18724 1024 # # 0 0 0 0 0 +keycache1 7 2 18724 1024 # # 0 0 0 0 0 +keycache1 7 3 18724 1024 # # 0 0 0 0 0 +keycache1 7 4 18724 1024 # # 0 0 0 0 0 +keycache1 7 5 18724 1024 # # 0 0 0 0 0 +keycache1 7 6 18724 1024 # # 0 0 0 0 0 +keycache1 7 7 18724 1024 # # 0 0 0 0 0 +keycache1 7 NULL 131068 1024 # # 0 0 0 0 0 +keycache2 NULL NULL 1048576 1024 # # 0 0 0 0 0 +drop table t1,t2; +set global keycache1.key_buffer_size=0; +set global keycache2.key_buffer_size=0; +set global key_buffer_size=@save_key_buffer_size; +set global key_cache_partitions=@save_key_cache_partitions; === added file 'mysql-test/r/select_pkeycache.result' --- a/mysql-test/r/select_pkeycache.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/r/select_pkeycache.result 2010-02-16 16:41:11 +0000 @@ -0,0 +1,4636 @@ +drop table if exists t1,t2,t3,t4,t11; +drop table if exists t1_1,t1_2,t9_1,t9_2,t1aa,t2aa; +drop view if exists v1; +CREATE TABLE t1 ( +Period smallint(4) unsigned zerofill DEFAULT '0000' NOT NULL, +Varor_period smallint(4) unsigned DEFAULT '0' NOT NULL +); +INSERT INTO t1 VALUES (9410,9412); +select period from t1; +period +9410 +select * from t1; +Period Varor_period +9410 9412 +select t1.* from t1; +Period Varor_period +9410 9412 +CREATE TABLE t2 ( +auto int not null auto_increment, +fld1 int(6) unsigned zerofill DEFAULT '000000' NOT NULL, +companynr tinyint(2) unsigned zerofill DEFAULT '00' NOT NULL, +fld3 char(30) DEFAULT '' NOT NULL, +fld4 char(35) DEFAULT '' NOT NULL, +fld5 char(35) DEFAULT '' NOT NULL, +fld6 char(4) DEFAULT '' NOT NULL, +UNIQUE fld1 (fld1), +KEY fld3 (fld3), +PRIMARY KEY (auto) +); +select t2.fld3 from t2 where companynr = 58 and fld3 like "%imaginable%"; +fld3 +imaginable +select fld3 from t2 where fld3 like "%cultivation" ; +fld3 +cultivation +select t2.fld3,companynr from t2 where companynr = 57+1 order by fld3; +fld3 companynr +concoct 58 +druggists 58 +engrossing 58 +Eurydice 58 +exclaimers 58 +ferociousness 58 +hopelessness 58 +Huey 58 +imaginable 58 +judges 58 +merging 58 +ostrich 58 +peering 58 +Phelps 58 +presumes 58 +Ruth 58 +sentences 58 +Shylock 58 +straggled 58 +synergy 58 +thanking 58 +tying 58 +unlocks 58 +select fld3,companynr from t2 where companynr = 58 order by fld3; +fld3 companynr +concoct 58 +druggists 58 +engrossing 58 +Eurydice 58 +exclaimers 58 +ferociousness 58 +hopelessness 58 +Huey 58 +imaginable 58 +judges 58 +merging 58 +ostrich 58 +peering 58 +Phelps 58 +presumes 58 +Ruth 58 +sentences 58 +Shylock 58 +straggled 58 +synergy 58 +thanking 58 +tying 58 +unlocks 58 +select fld3 from t2 order by fld3 desc limit 10; +fld3 +youthfulness +yelped +Wotan +workers +Witt +witchcraft +Winsett +Willy +willed +wildcats +select fld3 from t2 order by fld3 desc limit 5; +fld3 +youthfulness +yelped +Wotan +workers +Witt +select fld3 from t2 order by fld3 desc limit 5,5; +fld3 +witchcraft +Winsett +Willy +willed +wildcats +select t2.fld3 from t2 where fld3 = 'honeysuckle'; +fld3 +honeysuckle +select t2.fld3 from t2 where fld3 LIKE 'honeysuckl_'; +fld3 +honeysuckle +select t2.fld3 from t2 where fld3 LIKE 'hon_ysuckl_'; +fld3 +honeysuckle +select t2.fld3 from t2 where fld3 LIKE 'honeysuckle%'; +fld3 +honeysuckle +select t2.fld3 from t2 where fld3 LIKE 'h%le'; +fld3 +honeysuckle +select t2.fld3 from t2 where fld3 LIKE 'honeysuckle_'; +fld3 +select t2.fld3 from t2 where fld3 LIKE 'don_t_find_me_please%'; +fld3 +explain select t2.fld3 from t2 where fld3 = 'honeysuckle'; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ref fld3 fld3 30 const 1 Using where; Using index +explain select fld3 from t2 ignore index (fld3) where fld3 = 'honeysuckle'; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 Using where +explain select fld3 from t2 use index (fld1) where fld3 = 'honeysuckle'; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 Using where +explain select fld3 from t2 use index (fld3) where fld3 = 'honeysuckle'; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ref fld3 fld3 30 const 1 Using where; Using index +explain select fld3 from t2 use index (fld1,fld3) where fld3 = 'honeysuckle'; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ref fld3 fld3 30 const 1 Using where; Using index +explain select fld3 from t2 ignore index (fld3,not_used); +ERROR 42000: Key 'not_used' doesn't exist in table 't2' +explain select fld3 from t2 use index (not_used); +ERROR 42000: Key 'not_used' doesn't exist in table 't2' +select t2.fld3 from t2 where fld3 >= 'honeysuckle' and fld3 <= 'honoring' order by fld3; +fld3 +honeysuckle +honoring +explain select t2.fld3 from t2 where fld3 >= 'honeysuckle' and fld3 <= 'honoring' order by fld3; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 range fld3 fld3 30 NULL 2 Using where; Using index +select fld1,fld3 from t2 where fld3="Colombo" or fld3 = "nondecreasing" order by fld3; +fld1 fld3 +148504 Colombo +068305 Colombo +000000 nondecreasing +select fld1,fld3 from t2 where companynr = 37 and fld3 = 'appendixes'; +fld1 fld3 +232605 appendixes +1232605 appendixes +1232606 appendixes +1232607 appendixes +1232608 appendixes +1232609 appendixes +select fld1 from t2 where fld1=250501 or fld1="250502"; +fld1 +250501 +250502 +explain select fld1 from t2 where fld1=250501 or fld1="250502"; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 range fld1 fld1 4 NULL 2 Using where; Using index +select fld1 from t2 where fld1=250501 or fld1=250502 or fld1 >= 250505 and fld1 <= 250601 or fld1 between 250501 and 250502; +fld1 +250501 +250502 +250505 +250601 +explain select fld1 from t2 where fld1=250501 or fld1=250502 or fld1 >= 250505 and fld1 <= 250601 or fld1 between 250501 and 250502; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 range fld1 fld1 4 NULL 4 Using where; Using index +select fld1,fld3 from t2 where companynr = 37 and fld3 like 'f%'; +fld1 fld3 +218401 faithful +018007 fanatic +228311 fated +018017 featherweight +218022 feed +088303 feminine +058004 Fenton +038017 fetched +018054 fetters +208101 fiftieth +238007 filial +013606 fingerings +218008 finishers +038205 firearm +188505 fitting +202301 Fitzpatrick +238008 fixedly +012001 flanking +018103 flint +018104 flopping +188007 flurried +013602 foldout +226205 foothill +232102 forgivably +228306 forthcoming +186002 freakish +208113 freest +231315 freezes +036002 funereal +226209 furnishings +198006 furthermore +select fld3 from t2 where fld3 like "L%" and fld3 = "ok"; +fld3 +select fld3 from t2 where (fld3 like "C%" and fld3 = "Chantilly"); +fld3 +Chantilly +select fld1,fld3 from t2 where fld1 like "25050%"; +fld1 fld3 +250501 poisoning +250502 Iraqis +250503 heaving +250504 population +250505 bomb +select fld1,fld3 from t2 where fld1 like "25050_"; +fld1 fld3 +250501 poisoning +250502 Iraqis +250503 heaving +250504 population +250505 bomb +select distinct companynr from t2; +companynr +00 +37 +36 +50 +58 +29 +40 +53 +65 +41 +34 +68 +select distinct companynr from t2 order by companynr; +companynr +00 +29 +34 +36 +37 +40 +41 +50 +53 +58 +65 +68 +select distinct companynr from t2 order by companynr desc; +companynr +68 +65 +58 +53 +50 +41 +40 +37 +36 +34 +29 +00 +select distinct t2.fld3,period from t2,t1 where companynr=37 and fld3 like "O%"; +fld3 period +obliterates 9410 +offload 9410 +opaquely 9410 +organizer 9410 +overestimating 9410 +overlay 9410 +select distinct fld3 from t2 where companynr = 34 order by fld3; +fld3 +absentee +accessed +ahead +alphabetic +Asiaticizations +attitude +aye +bankruptcies +belays +Blythe +bomb +boulevard +bulldozes +cannot +caressing +charcoal +checksumming +chess +clubroom +colorful +cosy +creator +crying +Darius +diffusing +duality +Eiffel +Epiphany +Ernestine +explorers +exterminated +famine +forked +Gershwins +heaving +Hodges +Iraqis +Italianization +Lagos +landslide +libretto +Majorca +mastering +narrowed +occurred +offerers +Palestine +Peruvianizes +pharmaceutic +poisoning +population +Pygmalion +rats +realest +recording +regimented +retransmitting +reviver +rouses +scars +sicker +sleepwalk +stopped +sugars +translatable +uncles +unexpected +uprisings +versatility +vest +select distinct fld3 from t2 limit 10; +fld3 +abates +abiding +Abraham +abrogating +absentee +abut +accessed +accruing +accumulating +accuracies +select distinct fld3 from t2 having fld3 like "A%" limit 10; +fld3 +abates +abiding +Abraham +abrogating +absentee +abut +accessed +accruing +accumulating +accuracies +select distinct substring(fld3,1,3) from t2 where fld3 like "A%"; +substring(fld3,1,3) +aba +abi +Abr +abs +abu +acc +acq +acu +Ade +adj +Adl +adm +Ado +ads +adv +aer +aff +afi +afl +afo +agi +ahe +aim +air +Ald +alg +ali +all +alp +alr +ama +ame +amm +ana +and +ane +Ang +ani +Ann +Ant +api +app +aqu +Ara +arc +Arm +arr +Art +Asi +ask +asp +ass +ast +att +aud +Aug +aut +ave +avo +awe +aye +Azt +select distinct substring(fld3,1,3) as a from t2 having a like "A%" order by a limit 10; +a +aba +abi +Abr +abs +abu +acc +acq +acu +Ade +adj +select distinct substring(fld3,1,3) from t2 where fld3 like "A%" limit 10; +substring(fld3,1,3) +aba +abi +Abr +abs +abu +acc +acq +acu +Ade +adj +select distinct substring(fld3,1,3) as a from t2 having a like "A%" limit 10; +a +aba +abi +Abr +abs +abu +acc +acq +acu +Ade +adj +create table t3 ( +period int not null, +name char(32) not null, +companynr int not null, +price double(11,0), +price2 double(11,0), +key (period), +key (name) +); +create temporary table tmp engine = myisam select * from t3; +insert into t3 select * from tmp; +insert into tmp select * from t3; +insert into t3 select * from tmp; +insert into tmp select * from t3; +insert into t3 select * from tmp; +insert into tmp select * from t3; +insert into t3 select * from tmp; +insert into tmp select * from t3; +insert into t3 select * from tmp; +insert into tmp select * from t3; +insert into t3 select * from tmp; +insert into tmp select * from t3; +insert into t3 select * from tmp; +insert into tmp select * from t3; +insert into t3 select * from tmp; +insert into tmp select * from t3; +insert into t3 select * from tmp; +alter table t3 add t2nr int not null auto_increment primary key first; +drop table tmp; +SET SQL_BIG_TABLES=1; +select distinct concat(fld3," ",fld3) as namn from t2,t3 where t2.fld1=t3.t2nr order by namn limit 10; +namn +Abraham Abraham +abrogating abrogating +admonishing admonishing +Adolph Adolph +afield afield +aging aging +ammonium ammonium +analyzable analyzable +animals animals +animized animized +SET SQL_BIG_TABLES=0; +select distinct concat(fld3," ",fld3) from t2,t3 where t2.fld1=t3.t2nr order by fld3 limit 10; +concat(fld3," ",fld3) +Abraham Abraham +abrogating abrogating +admonishing admonishing +Adolph Adolph +afield afield +aging aging +ammonium ammonium +analyzable analyzable +animals animals +animized animized +select distinct fld5 from t2 limit 10; +fld5 +neat +Steinberg +jarring +tinily +balled +persist +attainments +fanatic +measures +rightfulness +select distinct fld3,count(*) from t2 group by companynr,fld3 limit 10; +fld3 count(*) +affixed 1 +and 1 +annoyers 1 +Anthony 1 +assayed 1 +assurers 1 +attendants 1 +bedlam 1 +bedpost 1 +boasted 1 +SET SQL_BIG_TABLES=1; +select distinct fld3,count(*) from t2 group by companynr,fld3 limit 10; +fld3 count(*) +affixed 1 +and 1 +annoyers 1 +Anthony 1 +assayed 1 +assurers 1 +attendants 1 +bedlam 1 +bedpost 1 +boasted 1 +SET SQL_BIG_TABLES=0; +select distinct fld3,repeat("a",length(fld3)),count(*) from t2 group by companynr,fld3 limit 100,10; +fld3 repeat("a",length(fld3)) count(*) +circus aaaaaa 1 +cited aaaaa 1 +Colombo aaaaaaa 1 +congresswoman aaaaaaaaaaaaa 1 +contrition aaaaaaaaaa 1 +corny aaaaa 1 +cultivation aaaaaaaaaaa 1 +definiteness aaaaaaaaaaaa 1 +demultiplex aaaaaaaaaaa 1 +disappointing aaaaaaaaaaaaa 1 +select distinct companynr,rtrim(space(512+companynr)) from t3 order by 1,2; +companynr rtrim(space(512+companynr)) +37 +78 +101 +154 +311 +447 +512 +select distinct fld3 from t2,t3 where t2.companynr = 34 and t2.fld1=t3.t2nr order by fld3; +fld3 +explain select t3.t2nr,fld3 from t2,t3 where t2.companynr = 34 and t2.fld1=t3.t2nr order by t3.t2nr,fld3; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ALL fld1 NULL NULL NULL 1199 Using where; Using temporary; Using filesort +1 SIMPLE t3 eq_ref PRIMARY PRIMARY 4 test.t2.fld1 1 Using where; Using index +explain select * from t3 as t1,t3 where t1.period=t3.period order by t3.period; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ALL period NULL NULL NULL 41810 Using temporary; Using filesort +1 SIMPLE t3 ref period period 4 test.t1.period 4181 +explain select * from t3 as t1,t3 where t1.period=t3.period order by t3.period limit 10; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t3 index period period 4 NULL 1 +1 SIMPLE t1 ref period period 4 test.t3.period 4181 +explain select * from t3 as t1,t3 where t1.period=t3.period order by t1.period limit 10; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 index period period 4 NULL 1 +1 SIMPLE t3 ref period period 4 test.t1.period 4181 +select period from t1; +period +9410 +select period from t1 where period=1900; +period +select fld3,period from t1,t2 where fld1 = 011401 order by period; +fld3 period +breaking 9410 +select fld3,period from t2,t3 where t2.fld1 = 011401 and t2.fld1=t3.t2nr and t3.period=1001; +fld3 period +breaking 1001 +explain select fld3,period from t2,t3 where t2.fld1 = 011401 and t3.t2nr=t2.fld1 and 1001 = t3.period; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 const fld1 fld1 4 const 1 +1 SIMPLE t3 const PRIMARY,period PRIMARY 4 const 1 +select fld3,period from t2,t1 where companynr*10 = 37*10; +fld3 period +breaking 9410 +Romans 9410 +intercepted 9410 +bewilderingly 9410 +astound 9410 +admonishing 9410 +sumac 9410 +flanking 9410 +combed 9410 +subjective 9410 +scatterbrain 9410 +Eulerian 9410 +Kane 9410 +overlay 9410 +perturb 9410 +goblins 9410 +annihilates 9410 +Wotan 9410 +snatching 9410 +concludes 9410 +laterally 9410 +yelped 9410 +grazing 9410 +Baird 9410 +celery 9410 +misunderstander 9410 +handgun 9410 +foldout 9410 +mystic 9410 +succumbed 9410 +Nabisco 9410 +fingerings 9410 +aging 9410 +afield 9410 +ammonium 9410 +boat 9410 +intelligibility 9410 +Augustine 9410 +teethe 9410 +dreaded 9410 +scholastics 9410 +audiology 9410 +wallet 9410 +parters 9410 +eschew 9410 +quitter 9410 +neat 9410 +Steinberg 9410 +jarring 9410 +tinily 9410 +balled 9410 +persist 9410 +attainments 9410 +fanatic 9410 +measures 9410 +rightfulness 9410 +capably 9410 +impulsive 9410 +starlet 9410 +terminators 9410 +untying 9410 +announces 9410 +featherweight 9410 +pessimist 9410 +daughter 9410 +decliner 9410 +lawgiver 9410 +stated 9410 +readable 9410 +attrition 9410 +cascade 9410 +motors 9410 +interrogate 9410 +pests 9410 +stairway 9410 +dopers 9410 +testicle 9410 +Parsifal 9410 +leavings 9410 +postulation 9410 +squeaking 9410 +contrasted 9410 +leftover 9410 +whiteners 9410 +erases 9410 +Punjab 9410 +Merritt 9410 +Quixotism 9410 +sweetish 9410 +dogging 9410 +scornfully 9410 +bellow 9410 +bills 9410 +cupboard 9410 +sureties 9410 +puddings 9410 +fetters 9410 +bivalves 9410 +incurring 9410 +Adolph 9410 +pithed 9410 +Miles 9410 +trimmings 9410 +tragedies 9410 +skulking 9410 +flint 9410 +flopping 9410 +relaxing 9410 +offload 9410 +suites 9410 +lists 9410 +animized 9410 +multilayer 9410 +standardizes 9410 +Judas 9410 +vacuuming 9410 +dentally 9410 +humanness 9410 +inch 9410 +Weissmuller 9410 +irresponsibly 9410 +luckily 9410 +culled 9410 +medical 9410 +bloodbath 9410 +subschema 9410 +animals 9410 +Micronesia 9410 +repetitions 9410 +Antares 9410 +ventilate 9410 +pityingly 9410 +interdependent 9410 +Graves 9410 +neonatal 9410 +chafe 9410 +honoring 9410 +realtor 9410 +elite 9410 +funereal 9410 +abrogating 9410 +sorters 9410 +Conley 9410 +lectured 9410 +Abraham 9410 +Hawaii 9410 +cage 9410 +hushes 9410 +Simla 9410 +reporters 9410 +Dutchman 9410 +descendants 9410 +groupings 9410 +dissociate 9410 +coexist 9410 +Beebe 9410 +Taoism 9410 +Connally 9410 +fetched 9410 +checkpoints 9410 +rusting 9410 +galling 9410 +obliterates 9410 +traitor 9410 +resumes 9410 +analyzable 9410 +terminator 9410 +gritty 9410 +firearm 9410 +minima 9410 +Selfridge 9410 +disable 9410 +witchcraft 9410 +betroth 9410 +Manhattanize 9410 +imprint 9410 +peeked 9410 +swelling 9410 +interrelationships 9410 +riser 9410 +Gandhian 9410 +peacock 9410 +bee 9410 +kanji 9410 +dental 9410 +scarf 9410 +chasm 9410 +insolence 9410 +syndicate 9410 +alike 9410 +imperial 9410 +convulsion 9410 +railway 9410 +validate 9410 +normalizes 9410 +comprehensive 9410 +chewing 9410 +denizen 9410 +schemer 9410 +chronicle 9410 +Kline 9410 +Anatole 9410 +partridges 9410 +brunch 9410 +recruited 9410 +dimensions 9410 +Chicana 9410 +announced 9410 +praised 9410 +employing 9410 +linear 9410 +quagmire 9410 +western 9410 +relishing 9410 +serving 9410 +scheduling 9410 +lore 9410 +eventful 9410 +arteriole 9410 +disentangle 9410 +cured 9410 +Fenton 9410 +avoidable 9410 +drains 9410 +detectably 9410 +husky 9410 +impelling 9410 +undoes 9410 +evened 9410 +squeezes 9410 +destroyer 9410 +rudeness 9410 +beaner 9410 +boorish 9410 +Everhart 9410 +encompass 9410 +mushrooms 9410 +Alison 9410 +externally 9410 +pellagra 9410 +cult 9410 +creek 9410 +Huffman 9410 +Majorca 9410 +governing 9410 +gadfly 9410 +reassigned 9410 +intentness 9410 +craziness 9410 +psychic 9410 +squabbled 9410 +burlesque 9410 +capped 9410 +extracted 9410 +DiMaggio 9410 +exclamation 9410 +subdirectory 9410 +Gothicism 9410 +feminine 9410 +metaphysically 9410 +sanding 9410 +Miltonism 9410 +freakish 9410 +index 9410 +straight 9410 +flurried 9410 +denotative 9410 +coming 9410 +commencements 9410 +gentleman 9410 +gifted 9410 +Shanghais 9410 +sportswriting 9410 +sloping 9410 +navies 9410 +leaflet 9410 +shooter 9410 +Joplin 9410 +babies 9410 +assails 9410 +admiring 9410 +swaying 9410 +Goldstine 9410 +fitting 9410 +Norwalk 9410 +analogy 9410 +deludes 9410 +cokes 9410 +Clayton 9410 +exhausts 9410 +causality 9410 +sating 9410 +icon 9410 +throttles 9410 +communicants 9410 +dehydrate 9410 +priceless 9410 +publicly 9410 +incidentals 9410 +commonplace 9410 +mumbles 9410 +furthermore 9410 +cautioned 9410 +parametrized 9410 +registration 9410 +sadly 9410 +positioning 9410 +babysitting 9410 +eternal 9410 +hoarder 9410 +congregates 9410 +rains 9410 +workers 9410 +sags 9410 +unplug 9410 +garage 9410 +boulder 9410 +specifics 9410 +Teresa 9410 +Winsett 9410 +convenient 9410 +buckboards 9410 +amenities 9410 +resplendent 9410 +sews 9410 +participated 9410 +Simon 9410 +certificates 9410 +Fitzpatrick 9410 +Evanston 9410 +misted 9410 +textures 9410 +save 9410 +count 9410 +rightful 9410 +chaperone 9410 +Lizzy 9410 +clenched 9410 +effortlessly 9410 +accessed 9410 +beaters 9410 +Hornblower 9410 +vests 9410 +indulgences 9410 +infallibly 9410 +unwilling 9410 +excrete 9410 +spools 9410 +crunches 9410 +overestimating 9410 +ineffective 9410 +humiliation 9410 +sophomore 9410 +star 9410 +rifles 9410 +dialysis 9410 +arriving 9410 +indulge 9410 +clockers 9410 +languages 9410 +Antarctica 9410 +percentage 9410 +ceiling 9410 +specification 9410 +regimented 9410 +ciphers 9410 +pictures 9410 +serpents 9410 +allot 9410 +realized 9410 +mayoral 9410 +opaquely 9410 +hostess 9410 +fiftieth 9410 +incorrectly 9410 +decomposition 9410 +stranglings 9410 +mixture 9410 +electroencephalography 9410 +similarities 9410 +charges 9410 +freest 9410 +Greenberg 9410 +tinting 9410 +expelled 9410 +warm 9410 +smoothed 9410 +deductions 9410 +Romano 9410 +bitterroot 9410 +corset 9410 +securing 9410 +environing 9410 +cute 9410 +Crays 9410 +heiress 9410 +inform 9410 +avenge 9410 +universals 9410 +Kinsey 9410 +ravines 9410 +bestseller 9410 +equilibrium 9410 +extents 9410 +relatively 9410 +pressure 9410 +critiques 9410 +befouled 9410 +rightfully 9410 +mechanizing 9410 +Latinizes 9410 +timesharing 9410 +Aden 9410 +embassies 9410 +males 9410 +shapelessly 9410 +mastering 9410 +Newtonian 9410 +finishers 9410 +abates 9410 +teem 9410 +kiting 9410 +stodgy 9410 +feed 9410 +guitars 9410 +airships 9410 +store 9410 +denounces 9410 +Pyle 9410 +Saxony 9410 +serializations 9410 +Peruvian 9410 +taxonomically 9410 +kingdom 9410 +stint 9410 +Sault 9410 +faithful 9410 +Ganymede 9410 +tidiness 9410 +gainful 9410 +contrary 9410 +Tipperary 9410 +tropics 9410 +theorizers 9410 +renew 9410 +already 9410 +terminal 9410 +Hegelian 9410 +hypothesizer 9410 +warningly 9410 +journalizing 9410 +nested 9410 +Lars 9410 +saplings 9410 +foothill 9410 +labeled 9410 +imperiously 9410 +reporters 9410 +furnishings 9410 +precipitable 9410 +discounts 9410 +excises 9410 +Stalin 9410 +despot 9410 +ripeness 9410 +Arabia 9410 +unruly 9410 +mournfulness 9410 +boom 9410 +slaughter 9410 +Sabine 9410 +handy 9410 +rural 9410 +organizer 9410 +shipyard 9410 +civics 9410 +inaccuracy 9410 +rules 9410 +juveniles 9410 +comprised 9410 +investigations 9410 +stabilizes 9410 +seminaries 9410 +Hunter 9410 +sporty 9410 +test 9410 +weasels 9410 +CERN 9410 +tempering 9410 +afore 9410 +Galatean 9410 +techniques 9410 +error 9410 +veranda 9410 +severely 9410 +Cassites 9410 +forthcoming 9410 +guides 9410 +vanish 9410 +lied 9410 +sawtooth 9410 +fated 9410 +gradually 9410 +widens 9410 +preclude 9410 +evenhandedly 9410 +percentage 9410 +disobedience 9410 +humility 9410 +gleaning 9410 +petted 9410 +bloater 9410 +minion 9410 +marginal 9410 +apiary 9410 +measures 9410 +precaution 9410 +repelled 9410 +primary 9410 +coverings 9410 +Artemia 9410 +navigate 9410 +spatial 9410 +Gurkha 9410 +meanwhile 9410 +Melinda 9410 +Butterfield 9410 +Aldrich 9410 +previewing 9410 +glut 9410 +unaffected 9410 +inmate 9410 +mineral 9410 +impending 9410 +meditation 9410 +ideas 9410 +miniaturizes 9410 +lewdly 9410 +title 9410 +youthfulness 9410 +creak 9410 +Chippewa 9410 +clamored 9410 +freezes 9410 +forgivably 9410 +reduce 9410 +McGovern 9410 +Nazis 9410 +epistle 9410 +socializes 9410 +conceptions 9410 +Kevin 9410 +uncovering 9410 +chews 9410 +appendixes 9410 +appendixes 9410 +appendixes 9410 +appendixes 9410 +appendixes 9410 +appendixes 9410 +raining 9410 +infest 9410 +compartment 9410 +minting 9410 +ducks 9410 +roped 9410 +waltz 9410 +Lillian 9410 +repressions 9410 +chillingly 9410 +noncritical 9410 +lithograph 9410 +spongers 9410 +parenthood 9410 +posed 9410 +instruments 9410 +filial 9410 +fixedly 9410 +relives 9410 +Pandora 9410 +watering 9410 +ungrateful 9410 +secures 9410 +poison 9410 +dusted 9410 +encompasses 9410 +presentation 9410 +Kantian 9410 +select fld3,period,price,price2 from t2,t3 where t2.fld1=t3.t2nr and period >= 1001 and period <= 1002 and t2.companynr = 37 order by fld3,period, price; +fld3 period price price2 +admonishing 1002 28357832 8723648 +analyzable 1002 28357832 8723648 +annihilates 1001 5987435 234724 +Antares 1002 28357832 8723648 +astound 1001 5987435 234724 +audiology 1001 5987435 234724 +Augustine 1002 28357832 8723648 +Baird 1002 28357832 8723648 +bewilderingly 1001 5987435 234724 +breaking 1001 5987435 234724 +Conley 1001 5987435 234724 +dentally 1002 28357832 8723648 +dissociate 1002 28357832 8723648 +elite 1001 5987435 234724 +eschew 1001 5987435 234724 +Eulerian 1001 5987435 234724 +flanking 1001 5987435 234724 +foldout 1002 28357832 8723648 +funereal 1002 28357832 8723648 +galling 1002 28357832 8723648 +Graves 1001 5987435 234724 +grazing 1001 5987435 234724 +groupings 1001 5987435 234724 +handgun 1001 5987435 234724 +humility 1002 28357832 8723648 +impulsive 1002 28357832 8723648 +inch 1001 5987435 234724 +intelligibility 1001 5987435 234724 +jarring 1001 5987435 234724 +lawgiver 1001 5987435 234724 +lectured 1002 28357832 8723648 +Merritt 1002 28357832 8723648 +neonatal 1001 5987435 234724 +offload 1002 28357832 8723648 +parters 1002 28357832 8723648 +pityingly 1002 28357832 8723648 +puddings 1002 28357832 8723648 +Punjab 1001 5987435 234724 +quitter 1002 28357832 8723648 +realtor 1001 5987435 234724 +relaxing 1001 5987435 234724 +repetitions 1001 5987435 234724 +resumes 1001 5987435 234724 +Romans 1002 28357832 8723648 +rusting 1001 5987435 234724 +scholastics 1001 5987435 234724 +skulking 1002 28357832 8723648 +stated 1002 28357832 8723648 +suites 1002 28357832 8723648 +sureties 1001 5987435 234724 +testicle 1002 28357832 8723648 +tinily 1002 28357832 8723648 +tragedies 1001 5987435 234724 +trimmings 1001 5987435 234724 +vacuuming 1001 5987435 234724 +ventilate 1001 5987435 234724 +wallet 1001 5987435 234724 +Weissmuller 1002 28357832 8723648 +Wotan 1002 28357832 8723648 +select t2.fld1,fld3,period,price,price2 from t2,t3 where t2.fld1>= 18201 and t2.fld1 <= 18811 and t2.fld1=t3.t2nr and period = 1001 and t2.companynr = 37; +fld1 fld3 period price price2 +018201 relaxing 1001 5987435 234724 +018601 vacuuming 1001 5987435 234724 +018801 inch 1001 5987435 234724 +018811 repetitions 1001 5987435 234724 +create table t4 ( +companynr tinyint(2) unsigned zerofill NOT NULL default '00', +companyname char(30) NOT NULL default '', +PRIMARY KEY (companynr), +UNIQUE KEY companyname(companyname) +) ENGINE=MyISAM MAX_ROWS=50 PACK_KEYS=1 COMMENT='companynames'; +select STRAIGHT_JOIN t2.companynr,companyname from t4,t2 where t2.companynr=t4.companynr group by t2.companynr; +companynr companyname +00 Unknown +29 company 1 +34 company 2 +36 company 3 +37 company 4 +40 company 5 +41 company 6 +50 company 11 +53 company 7 +58 company 8 +65 company 9 +68 company 10 +select SQL_SMALL_RESULT t2.companynr,companyname from t4,t2 where t2.companynr=t4.companynr group by t2.companynr; +companynr companyname +00 Unknown +29 company 1 +34 company 2 +36 company 3 +37 company 4 +40 company 5 +41 company 6 +50 company 11 +53 company 7 +58 company 8 +65 company 9 +68 company 10 +select * from t1,t1 t12; +Period Varor_period Period Varor_period +9410 9412 9410 9412 +select t2.fld1,t22.fld1 from t2,t2 t22 where t2.fld1 >= 250501 and t2.fld1 <= 250505 and t22.fld1 >= 250501 and t22.fld1 <= 250505; +fld1 fld1 +250501 250501 +250502 250501 +250503 250501 +250504 250501 +250505 250501 +250501 250502 +250502 250502 +250503 250502 +250504 250502 +250505 250502 +250501 250503 +250502 250503 +250503 250503 +250504 250503 +250505 250503 +250501 250504 +250502 250504 +250503 250504 +250504 250504 +250505 250504 +250501 250505 +250502 250505 +250503 250505 +250504 250505 +250505 250505 +insert into t2 (fld1, companynr) values (999999,99); +select t2.companynr,companyname from t2 left join t4 using (companynr) where t4.companynr is null; +companynr companyname +99 NULL +select count(*) from t2 left join t4 using (companynr) where t4.companynr is not null; +count(*) +1199 +explain select t2.companynr,companyname from t2 left join t4 using (companynr) where t4.companynr is null; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ALL NULL NULL NULL NULL 1200 +1 SIMPLE t4 eq_ref PRIMARY PRIMARY 1 test.t2.companynr 1 Using where; Not exists +explain select t2.companynr,companyname from t4 left join t2 using (companynr) where t2.companynr is null; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t4 ALL NULL NULL NULL NULL 12 +1 SIMPLE t2 ALL NULL NULL NULL NULL 1200 Using where; Not exists +select companynr,companyname from t2 left join t4 using (companynr) where companynr is null; +companynr companyname +select count(*) from t2 left join t4 using (companynr) where companynr is not null; +count(*) +1200 +explain select companynr,companyname from t2 left join t4 using (companynr) where companynr is null; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Impossible WHERE +explain select companynr,companyname from t4 left join t2 using (companynr) where companynr is null; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Impossible WHERE +delete from t2 where fld1=999999; +explain select t2.companynr,companyname from t4 left join t2 using (companynr) where t2.companynr > 0; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 Using where +1 SIMPLE t4 eq_ref PRIMARY PRIMARY 1 test.t2.companynr 1 +explain select t2.companynr,companyname from t4 left join t2 using (companynr) where t2.companynr > 0 or t2.companynr < 0; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 Using where +1 SIMPLE t4 eq_ref PRIMARY PRIMARY 1 test.t2.companynr 1 +explain select t2.companynr,companyname from t4 left join t2 using (companynr) where t2.companynr > 0 and t4.companynr > 0; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 Using where +1 SIMPLE t4 eq_ref PRIMARY PRIMARY 1 test.t2.companynr 1 +explain select companynr,companyname from t4 left join t2 using (companynr) where companynr > 0; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t4 ALL PRIMARY NULL NULL NULL 12 Using where +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 +explain select companynr,companyname from t4 left join t2 using (companynr) where companynr > 0 or companynr < 0; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t4 ALL PRIMARY NULL NULL NULL 12 Using where +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 +explain select companynr,companyname from t4 left join t2 using (companynr) where companynr > 0 and companynr > 0; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t4 ALL PRIMARY NULL NULL NULL 12 Using where +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 +explain select t2.companynr,companyname from t4 left join t2 using (companynr) where t2.companynr > 0 or t2.companynr is null; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t4 ALL NULL NULL NULL NULL 12 +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 Using where +explain select t2.companynr,companyname from t4 left join t2 using (companynr) where t2.companynr > 0 or t2.companynr < 0 or t4.companynr > 0; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t4 ALL PRIMARY NULL NULL NULL 12 +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 Using where +explain select t2.companynr,companyname from t4 left join t2 using (companynr) where ifnull(t2.companynr,1)>0; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t4 ALL NULL NULL NULL NULL 12 +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 Using where +explain select companynr,companyname from t4 left join t2 using (companynr) where companynr > 0 or companynr is null; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t4 ALL PRIMARY NULL NULL NULL 12 Using where +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 +explain select companynr,companyname from t4 left join t2 using (companynr) where companynr > 0 or companynr < 0 or companynr > 0; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t4 ALL PRIMARY NULL NULL NULL 12 Using where +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 +explain select companynr,companyname from t4 left join t2 using (companynr) where ifnull(companynr,1)>0; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t4 ALL NULL NULL NULL NULL 12 Using where +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 +select distinct t2.companynr,t4.companynr from t2,t4 where t2.companynr=t4.companynr+1; +companynr companynr +37 36 +41 40 +explain select distinct t2.companynr,t4.companynr from t2,t4 where t2.companynr=t4.companynr+1; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t4 index NULL PRIMARY 1 NULL 12 Using index; Using temporary +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 Using where; Using join buffer +select t2.fld1,t2.companynr,fld3,period from t3,t2 where t2.fld1 = 38208 and t2.fld1=t3.t2nr and period = 1008 or t2.fld1 = 38008 and t2.fld1 =t3.t2nr and period = 1008; +fld1 companynr fld3 period +038008 37 reporters 1008 +038208 37 Selfridge 1008 +select t2.fld1,t2.companynr,fld3,period from t3,t2 where (t2.fld1 = 38208 or t2.fld1 = 38008) and t2.fld1=t3.t2nr and period>=1008 and period<=1009; +fld1 companynr fld3 period +038008 37 reporters 1008 +038208 37 Selfridge 1008 +select t2.fld1,t2.companynr,fld3,period from t3,t2 where (t3.t2nr = 38208 or t3.t2nr = 38008) and t2.fld1=t3.t2nr and period>=1008 and period<=1009; +fld1 companynr fld3 period +038008 37 reporters 1008 +038208 37 Selfridge 1008 +select period from t1 where (((period > 0) or period < 10000 or (period = 1900)) and (period=1900 and period <= 1901) or (period=1903 and (period=1903)) and period>=1902) or ((period=1904 or period=1905) or (period=1906 or period>1907)) or (period=1908 and period = 1909); +period +9410 +select period from t1 where ((period > 0 and period < 1) or (((period > 0 and period < 100) and (period > 10)) or (period > 10)) or (period > 0 and (period > 5 or period > 6))); +period +9410 +select a.fld1 from t2 as a,t2 b where ((a.fld1 = 250501 and a.fld1=b.fld1) or a.fld1=250502 or a.fld1=250503 or (a.fld1=250505 and a.fld1<=b.fld1 and b.fld1>=a.fld1)) and a.fld1=b.fld1; +fld1 +250501 +250502 +250503 +250505 +select fld1 from t2 where fld1 in (250502,98005,98006,250503,250605,250606) and fld1 >=250502 and fld1 not in (250605,250606); +fld1 +250502 +250503 +select fld1 from t2 where fld1 between 250502 and 250504; +fld1 +250502 +250503 +250504 +select fld3 from t2 where (((fld3 like "_%L%" ) or (fld3 like "%ok%")) and ( fld3 like "L%" or fld3 like "G%")) and fld3 like "L%" ; +fld3 +label +labeled +labeled +landslide +laterally +leaflet +lewdly +Lillian +luckily +select count(*) from t1; +count(*) +1 +select companynr,count(*),sum(fld1) from t2 group by companynr; +companynr count(*) sum(fld1) +00 82 10355753 +29 95 14473298 +34 70 17788966 +36 215 22786296 +37 588 83602098 +40 37 6618386 +41 52 12816335 +50 11 1595438 +53 4 793210 +58 23 2254293 +65 10 2284055 +68 12 3097288 +select companynr,count(*) from t2 group by companynr order by companynr desc limit 5; +companynr count(*) +68 12 +65 10 +58 23 +53 4 +50 11 +select count(*),min(fld4),max(fld4),sum(fld1),avg(fld1),std(fld1),variance(fld1) from t2 where companynr = 34 and fld4<>""; +count(*) min(fld4) max(fld4) sum(fld1) avg(fld1) std(fld1) variance(fld1) +70 absentee vest 17788966 254128.0857 3272.5940 10709871.3069 +explain extended select count(*),min(fld4),max(fld4),sum(fld1),avg(fld1),std(fld1),variance(fld1) from t2 where companynr = 34 and fld4<>""; +id select_type table type possible_keys key key_len ref rows filtered Extra +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 100.00 Using where +Warnings: +Note 1003 select count(0) AS `count(*)`,min(`test`.`t2`.`fld4`) AS `min(fld4)`,max(`test`.`t2`.`fld4`) AS `max(fld4)`,sum(`test`.`t2`.`fld1`) AS `sum(fld1)`,avg(`test`.`t2`.`fld1`) AS `avg(fld1)`,std(`test`.`t2`.`fld1`) AS `std(fld1)`,variance(`test`.`t2`.`fld1`) AS `variance(fld1)` from `test`.`t2` where ((`test`.`t2`.`companynr` = 34) and (`test`.`t2`.`fld4` <> '')) +select companynr,count(*),min(fld4),max(fld4),sum(fld1),avg(fld1),std(fld1),variance(fld1) from t2 group by companynr limit 3; +companynr count(*) min(fld4) max(fld4) sum(fld1) avg(fld1) std(fld1) variance(fld1) +00 82 Anthony windmills 10355753 126289.6707 115550.9757 13352027981.7087 +29 95 abut wetness 14473298 152350.5053 8368.5480 70032594.9026 +34 70 absentee vest 17788966 254128.0857 3272.5940 10709871.3069 +select companynr,t2nr,count(price),sum(price),min(price),max(price),avg(price) from t3 where companynr = 37 group by companynr,t2nr limit 10; +companynr t2nr count(price) sum(price) min(price) max(price) avg(price) +37 1 1 5987435 5987435 5987435 5987435.0000 +37 2 1 28357832 28357832 28357832 28357832.0000 +37 3 1 39654943 39654943 39654943 39654943.0000 +37 11 1 5987435 5987435 5987435 5987435.0000 +37 12 1 28357832 28357832 28357832 28357832.0000 +37 13 1 39654943 39654943 39654943 39654943.0000 +37 21 1 5987435 5987435 5987435 5987435.0000 +37 22 1 28357832 28357832 28357832 28357832.0000 +37 23 1 39654943 39654943 39654943 39654943.0000 +37 31 1 5987435 5987435 5987435 5987435.0000 +select /*! SQL_SMALL_RESULT */ companynr,t2nr,count(price),sum(price),min(price),max(price),avg(price) from t3 where companynr = 37 group by companynr,t2nr limit 10; +companynr t2nr count(price) sum(price) min(price) max(price) avg(price) +37 1 1 5987435 5987435 5987435 5987435.0000 +37 2 1 28357832 28357832 28357832 28357832.0000 +37 3 1 39654943 39654943 39654943 39654943.0000 +37 11 1 5987435 5987435 5987435 5987435.0000 +37 12 1 28357832 28357832 28357832 28357832.0000 +37 13 1 39654943 39654943 39654943 39654943.0000 +37 21 1 5987435 5987435 5987435 5987435.0000 +37 22 1 28357832 28357832 28357832 28357832.0000 +37 23 1 39654943 39654943 39654943 39654943.0000 +37 31 1 5987435 5987435 5987435 5987435.0000 +select companynr,count(price),sum(price),min(price),max(price),avg(price) from t3 group by companynr ; +companynr count(price) sum(price) min(price) max(price) avg(price) +37 12543 309394878010 5987435 39654943 24666736.6667 +78 8362 414611089292 726498 98439034 49582766.0000 +101 4181 3489454238 834598 834598 834598.0000 +154 4181 4112197254950 983543950 983543950 983543950.0000 +311 4181 979599938 234298 234298 234298.0000 +447 4181 9929180954 2374834 2374834 2374834.0000 +512 4181 3288532102 786542 786542 786542.0000 +select distinct mod(companynr,10) from t4 group by companynr; +mod(companynr,10) +0 +9 +4 +6 +7 +1 +3 +8 +5 +select distinct 1 from t4 group by companynr; +1 +1 +select count(distinct fld1) from t2; +count(distinct fld1) +1199 +select companynr,count(distinct fld1) from t2 group by companynr; +companynr count(distinct fld1) +00 82 +29 95 +34 70 +36 215 +37 588 +40 37 +41 52 +50 11 +53 4 +58 23 +65 10 +68 12 +select companynr,count(*) from t2 group by companynr; +companynr count(*) +00 82 +29 95 +34 70 +36 215 +37 588 +40 37 +41 52 +50 11 +53 4 +58 23 +65 10 +68 12 +select companynr,count(distinct concat(fld1,repeat(65,1000))) from t2 group by companynr; +companynr count(distinct concat(fld1,repeat(65,1000))) +00 82 +29 95 +34 70 +36 215 +37 588 +40 37 +41 52 +50 11 +53 4 +58 23 +65 10 +68 12 +select companynr,count(distinct concat(fld1,repeat(65,200))) from t2 group by companynr; +companynr count(distinct concat(fld1,repeat(65,200))) +00 82 +29 95 +34 70 +36 215 +37 588 +40 37 +41 52 +50 11 +53 4 +58 23 +65 10 +68 12 +select companynr,count(distinct floor(fld1/100)) from t2 group by companynr; +companynr count(distinct floor(fld1/100)) +00 47 +29 35 +34 14 +36 69 +37 108 +40 16 +41 11 +50 9 +53 1 +58 1 +65 1 +68 1 +select companynr,count(distinct concat(repeat(65,1000),floor(fld1/100))) from t2 group by companynr; +companynr count(distinct concat(repeat(65,1000),floor(fld1/100))) +00 47 +29 35 +34 14 +36 69 +37 108 +40 16 +41 11 +50 9 +53 1 +58 1 +65 1 +68 1 +select sum(fld1),fld3 from t2 where fld3="Romans" group by fld1 limit 10; +sum(fld1) fld3 +11402 Romans +select name,count(*) from t3 where name='cloakroom' group by name; +name count(*) +cloakroom 4181 +select name,count(*) from t3 where name='cloakroom' and price>10 group by name; +name count(*) +cloakroom 4181 +select count(*) from t3 where name='cloakroom' and price2=823742; +count(*) +4181 +select name,count(*) from t3 where name='cloakroom' and price2=823742 group by name; +name count(*) +cloakroom 4181 +select name,count(*) from t3 where name >= "extramarital" and price <= 39654943 group by name; +name count(*) +extramarital 4181 +gazer 4181 +gems 4181 +Iranizes 4181 +spates 4181 +tucked 4181 +violinist 4181 +select t2.fld3,count(*) from t2,t3 where t2.fld1=158402 and t3.name=t2.fld3 group by t3.name; +fld3 count(*) +spates 4181 +select companynr|0,companyname from t4 group by 1; +companynr|0 companyname +0 Unknown +29 company 1 +34 company 2 +36 company 3 +37 company 4 +40 company 5 +41 company 6 +50 company 11 +53 company 7 +58 company 8 +65 company 9 +68 company 10 +select t2.companynr,companyname,count(*) from t2,t4 where t2.companynr=t4.companynr group by t2.companynr order by companyname; +companynr companyname count(*) +29 company 1 95 +68 company 10 12 +50 company 11 11 +34 company 2 70 +36 company 3 215 +37 company 4 588 +40 company 5 37 +41 company 6 52 +53 company 7 4 +58 company 8 23 +65 company 9 10 +00 Unknown 82 +select t2.fld1,count(*) from t2,t3 where t2.fld1=158402 and t3.name=t2.fld3 group by t3.name; +fld1 count(*) +158402 4181 +select sum(Period)/count(*) from t1; +sum(Period)/count(*) +9410.0000 +select companynr,count(price) as "count",sum(price) as "sum" ,abs(sum(price)/count(price)-avg(price)) as "diff",(0+count(price))*companynr as func from t3 group by companynr; +companynr count sum diff func +37 12543 309394878010 0.0000 464091 +78 8362 414611089292 0.0000 652236 +101 4181 3489454238 0.0000 422281 +154 4181 4112197254950 0.0000 643874 +311 4181 979599938 0.0000 1300291 +447 4181 9929180954 0.0000 1868907 +512 4181 3288532102 0.0000 2140672 +select companynr,sum(price)/count(price) as avg from t3 group by companynr having avg > 70000000 order by avg; +companynr avg +154 983543950.0000 +select companynr,count(*) from t2 group by companynr order by 2 desc; +companynr count(*) +37 588 +36 215 +29 95 +00 82 +34 70 +41 52 +40 37 +58 23 +68 12 +50 11 +65 10 +53 4 +select companynr,count(*) from t2 where companynr > 40 group by companynr order by 2 desc; +companynr count(*) +41 52 +58 23 +68 12 +50 11 +65 10 +53 4 +select t2.fld4,t2.fld1,count(price),sum(price),min(price),max(price),avg(price) from t3,t2 where t3.companynr = 37 and t2.fld1 = t3.t2nr group by fld1,t2.fld4; +fld4 fld1 count(price) sum(price) min(price) max(price) avg(price) +teethe 000001 1 5987435 5987435 5987435 5987435.0000 +dreaded 011401 1 5987435 5987435 5987435 5987435.0000 +scholastics 011402 1 28357832 28357832 28357832 28357832.0000 +audiology 011403 1 39654943 39654943 39654943 39654943.0000 +wallet 011501 1 5987435 5987435 5987435 5987435.0000 +parters 011701 1 5987435 5987435 5987435 5987435.0000 +eschew 011702 1 28357832 28357832 28357832 28357832.0000 +quitter 011703 1 39654943 39654943 39654943 39654943.0000 +neat 012001 1 5987435 5987435 5987435 5987435.0000 +Steinberg 012003 1 39654943 39654943 39654943 39654943.0000 +balled 012301 1 5987435 5987435 5987435 5987435.0000 +persist 012302 1 28357832 28357832 28357832 28357832.0000 +attainments 012303 1 39654943 39654943 39654943 39654943.0000 +capably 012501 1 5987435 5987435 5987435 5987435.0000 +impulsive 012602 1 28357832 28357832 28357832 28357832.0000 +starlet 012603 1 39654943 39654943 39654943 39654943.0000 +featherweight 012701 1 5987435 5987435 5987435 5987435.0000 +pessimist 012702 1 28357832 28357832 28357832 28357832.0000 +daughter 012703 1 39654943 39654943 39654943 39654943.0000 +lawgiver 013601 1 5987435 5987435 5987435 5987435.0000 +stated 013602 1 28357832 28357832 28357832 28357832.0000 +readable 013603 1 39654943 39654943 39654943 39654943.0000 +testicle 013801 1 5987435 5987435 5987435 5987435.0000 +Parsifal 013802 1 28357832 28357832 28357832 28357832.0000 +leavings 013803 1 39654943 39654943 39654943 39654943.0000 +squeaking 013901 1 5987435 5987435 5987435 5987435.0000 +contrasted 016001 1 5987435 5987435 5987435 5987435.0000 +leftover 016201 1 5987435 5987435 5987435 5987435.0000 +whiteners 016202 1 28357832 28357832 28357832 28357832.0000 +erases 016301 1 5987435 5987435 5987435 5987435.0000 +Punjab 016302 1 28357832 28357832 28357832 28357832.0000 +Merritt 016303 1 39654943 39654943 39654943 39654943.0000 +sweetish 018001 1 5987435 5987435 5987435 5987435.0000 +dogging 018002 1 28357832 28357832 28357832 28357832.0000 +scornfully 018003 1 39654943 39654943 39654943 39654943.0000 +fetters 018012 1 28357832 28357832 28357832 28357832.0000 +bivalves 018013 1 39654943 39654943 39654943 39654943.0000 +skulking 018021 1 5987435 5987435 5987435 5987435.0000 +flint 018022 1 28357832 28357832 28357832 28357832.0000 +flopping 018023 1 39654943 39654943 39654943 39654943.0000 +Judas 018032 1 28357832 28357832 28357832 28357832.0000 +vacuuming 018033 1 39654943 39654943 39654943 39654943.0000 +medical 018041 1 5987435 5987435 5987435 5987435.0000 +bloodbath 018042 1 28357832 28357832 28357832 28357832.0000 +subschema 018043 1 39654943 39654943 39654943 39654943.0000 +interdependent 018051 1 5987435 5987435 5987435 5987435.0000 +Graves 018052 1 28357832 28357832 28357832 28357832.0000 +neonatal 018053 1 39654943 39654943 39654943 39654943.0000 +sorters 018061 1 5987435 5987435 5987435 5987435.0000 +epistle 018062 1 28357832 28357832 28357832 28357832.0000 +Conley 018101 1 5987435 5987435 5987435 5987435.0000 +lectured 018102 1 28357832 28357832 28357832 28357832.0000 +Abraham 018103 1 39654943 39654943 39654943 39654943.0000 +cage 018201 1 5987435 5987435 5987435 5987435.0000 +hushes 018202 1 28357832 28357832 28357832 28357832.0000 +Simla 018402 1 28357832 28357832 28357832 28357832.0000 +reporters 018403 1 39654943 39654943 39654943 39654943.0000 +coexist 018601 1 5987435 5987435 5987435 5987435.0000 +Beebe 018602 1 28357832 28357832 28357832 28357832.0000 +Taoism 018603 1 39654943 39654943 39654943 39654943.0000 +Connally 018801 1 5987435 5987435 5987435 5987435.0000 +fetched 018802 1 28357832 28357832 28357832 28357832.0000 +checkpoints 018803 1 39654943 39654943 39654943 39654943.0000 +gritty 018811 1 5987435 5987435 5987435 5987435.0000 +firearm 018812 1 28357832 28357832 28357832 28357832.0000 +minima 019101 1 5987435 5987435 5987435 5987435.0000 +Selfridge 019102 1 28357832 28357832 28357832 28357832.0000 +disable 019103 1 39654943 39654943 39654943 39654943.0000 +witchcraft 019201 1 5987435 5987435 5987435 5987435.0000 +betroth 030501 1 5987435 5987435 5987435 5987435.0000 +Manhattanize 030502 1 28357832 28357832 28357832 28357832.0000 +imprint 030503 1 39654943 39654943 39654943 39654943.0000 +swelling 031901 1 5987435 5987435 5987435 5987435.0000 +interrelationships 036001 1 5987435 5987435 5987435 5987435.0000 +riser 036002 1 28357832 28357832 28357832 28357832.0000 +bee 038001 1 5987435 5987435 5987435 5987435.0000 +kanji 038002 1 28357832 28357832 28357832 28357832.0000 +dental 038003 1 39654943 39654943 39654943 39654943.0000 +railway 038011 1 5987435 5987435 5987435 5987435.0000 +validate 038012 1 28357832 28357832 28357832 28357832.0000 +normalizes 038013 1 39654943 39654943 39654943 39654943.0000 +Kline 038101 1 5987435 5987435 5987435 5987435.0000 +Anatole 038102 1 28357832 28357832 28357832 28357832.0000 +partridges 038103 1 39654943 39654943 39654943 39654943.0000 +recruited 038201 1 5987435 5987435 5987435 5987435.0000 +dimensions 038202 1 28357832 28357832 28357832 28357832.0000 +Chicana 038203 1 39654943 39654943 39654943 39654943.0000 +select t3.companynr,fld3,sum(price) from t3,t2 where t2.fld1 = t3.t2nr and t3.companynr = 512 group by companynr,fld3; +companynr fld3 sum(price) +512 boat 786542 +512 capably 786542 +512 cupboard 786542 +512 decliner 786542 +512 descendants 786542 +512 dopers 786542 +512 erases 786542 +512 Micronesia 786542 +512 Miles 786542 +512 skies 786542 +select t2.companynr,count(*),min(fld3),max(fld3),sum(price),avg(price) from t2,t3 where t3.companynr >= 30 and t3.companynr <= 58 and t3.t2nr = t2.fld1 and 1+1=2 group by t2.companynr; +companynr count(*) min(fld3) max(fld3) sum(price) avg(price) +00 1 Omaha Omaha 5987435 5987435.0000 +36 1 dubbed dubbed 28357832 28357832.0000 +37 83 Abraham Wotan 1908978016 22999735.1325 +50 2 scribbled tapestry 68012775 34006387.5000 +select t3.companynr+0,t3.t2nr,fld3,sum(price) from t3,t2 where t2.fld1 = t3.t2nr and t3.companynr = 37 group by 1,t3.t2nr,fld3,fld3,fld3,fld3,fld3 order by fld1; +t3.companynr+0 t2nr fld3 sum(price) +37 1 Omaha 5987435 +37 11401 breaking 5987435 +37 11402 Romans 28357832 +37 11403 intercepted 39654943 +37 11501 bewilderingly 5987435 +37 11701 astound 5987435 +37 11702 admonishing 28357832 +37 11703 sumac 39654943 +37 12001 flanking 5987435 +37 12003 combed 39654943 +37 12301 Eulerian 5987435 +37 12302 dubbed 28357832 +37 12303 Kane 39654943 +37 12501 annihilates 5987435 +37 12602 Wotan 28357832 +37 12603 snatching 39654943 +37 12701 grazing 5987435 +37 12702 Baird 28357832 +37 12703 celery 39654943 +37 13601 handgun 5987435 +37 13602 foldout 28357832 +37 13603 mystic 39654943 +37 13801 intelligibility 5987435 +37 13802 Augustine 28357832 +37 13803 teethe 39654943 +37 13901 scholastics 5987435 +37 16001 audiology 5987435 +37 16201 wallet 5987435 +37 16202 parters 28357832 +37 16301 eschew 5987435 +37 16302 quitter 28357832 +37 16303 neat 39654943 +37 18001 jarring 5987435 +37 18002 tinily 28357832 +37 18003 balled 39654943 +37 18012 impulsive 28357832 +37 18013 starlet 39654943 +37 18021 lawgiver 5987435 +37 18022 stated 28357832 +37 18023 readable 39654943 +37 18032 testicle 28357832 +37 18033 Parsifal 39654943 +37 18041 Punjab 5987435 +37 18042 Merritt 28357832 +37 18043 Quixotism 39654943 +37 18051 sureties 5987435 +37 18052 puddings 28357832 +37 18053 tapestry 39654943 +37 18061 trimmings 5987435 +37 18062 humility 28357832 +37 18101 tragedies 5987435 +37 18102 skulking 28357832 +37 18103 flint 39654943 +37 18201 relaxing 5987435 +37 18202 offload 28357832 +37 18402 suites 28357832 +37 18403 lists 39654943 +37 18601 vacuuming 5987435 +37 18602 dentally 28357832 +37 18603 humanness 39654943 +37 18801 inch 5987435 +37 18802 Weissmuller 28357832 +37 18803 irresponsibly 39654943 +37 18811 repetitions 5987435 +37 18812 Antares 28357832 +37 19101 ventilate 5987435 +37 19102 pityingly 28357832 +37 19103 interdependent 39654943 +37 19201 Graves 5987435 +37 30501 neonatal 5987435 +37 30502 scribbled 28357832 +37 30503 chafe 39654943 +37 31901 realtor 5987435 +37 36001 elite 5987435 +37 36002 funereal 28357832 +37 38001 Conley 5987435 +37 38002 lectured 28357832 +37 38003 Abraham 39654943 +37 38011 groupings 5987435 +37 38012 dissociate 28357832 +37 38013 coexist 39654943 +37 38101 rusting 5987435 +37 38102 galling 28357832 +37 38103 obliterates 39654943 +37 38201 resumes 5987435 +37 38202 analyzable 28357832 +37 38203 terminator 39654943 +select sum(price) from t3,t2 where t2.fld1 = t3.t2nr and t3.companynr = 512 and t3.t2nr = 38008 and t2.fld1 = 38008 or t2.fld1= t3.t2nr and t3.t2nr = 38008 and t2.fld1 = 38008; +sum(price) +234298 +select t2.fld1,sum(price) from t3,t2 where t2.fld1 = t3.t2nr and t3.companynr = 512 and t3.t2nr = 38008 and t2.fld1 = 38008 or t2.fld1 = t3.t2nr and t3.t2nr = 38008 and t2.fld1 = 38008 or t3.t2nr = t2.fld1 and t2.fld1 = 38008 group by t2.fld1; +fld1 sum(price) +038008 234298 +explain select fld3 from t2 where 1>2 or 2>3; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Impossible WHERE +explain select fld3 from t2 where fld1=fld1; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ALL NULL NULL NULL NULL 1199 +select companynr,fld1 from t2 HAVING fld1=250501 or fld1=250502; +companynr fld1 +34 250501 +34 250502 +select companynr,fld1 from t2 WHERE fld1>=250501 HAVING fld1<=250502; +companynr fld1 +34 250501 +34 250502 +select companynr,count(*) as count,sum(fld1) as sum from t2 group by companynr having count > 40 and sum/count >= 120000; +companynr count sum +00 82 10355753 +29 95 14473298 +34 70 17788966 +37 588 83602098 +41 52 12816335 +select companynr from t2 group by companynr having count(*) > 40 and sum(fld1)/count(*) >= 120000 ; +companynr +00 +29 +34 +37 +41 +select t2.companynr,companyname,count(*) from t2,t4 where t2.companynr=t4.companynr group by companyname having t2.companynr >= 40; +companynr companyname count(*) +68 company 10 12 +50 company 11 11 +40 company 5 37 +41 company 6 52 +53 company 7 4 +58 company 8 23 +65 company 9 10 +select count(*) from t2; +count(*) +1199 +select count(*) from t2 where fld1 < 098024; +count(*) +387 +select min(fld1) from t2 where fld1>= 098024; +min(fld1) +98024 +select max(fld1) from t2 where fld1>= 098024; +max(fld1) +1232609 +select count(*) from t3 where price2=76234234; +count(*) +4181 +select count(*) from t3 where companynr=512 and price2=76234234; +count(*) +4181 +explain select min(fld1),max(fld1),count(*) from t2; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Select tables optimized away +select min(fld1),max(fld1),count(*) from t2; +min(fld1) max(fld1) count(*) +0 1232609 1199 +select min(t2nr),max(t2nr) from t3 where t2nr=2115 and price2=823742; +min(t2nr) max(t2nr) +2115 2115 +select count(*),min(t2nr),max(t2nr) from t3 where name='spates' and companynr=78; +count(*) min(t2nr) max(t2nr) +4181 4 41804 +select t2nr,count(*) from t3 where name='gems' group by t2nr limit 20; +t2nr count(*) +9 1 +19 1 +29 1 +39 1 +49 1 +59 1 +69 1 +79 1 +89 1 +99 1 +109 1 +119 1 +129 1 +139 1 +149 1 +159 1 +169 1 +179 1 +189 1 +199 1 +select max(t2nr) from t3 where price=983543950; +max(t2nr) +41807 +select t1.period from t3 = t1 limit 1; +period +1001 +select t1.period from t1 as t1 limit 1; +period +9410 +select t1.period as "Nuvarande period" from t1 as t1 limit 1; +Nuvarande period +9410 +select period as ok_period from t1 limit 1; +ok_period +9410 +select period as ok_period from t1 group by ok_period limit 1; +ok_period +9410 +select 1+1 as summa from t1 group by summa limit 1; +summa +2 +select period as "Nuvarande period" from t1 group by "Nuvarande period" limit 1; +Nuvarande period +9410 +show tables; +Tables_in_test +t1 +t2 +t3 +t4 +show tables from test like "s%"; +Tables_in_test (s%) +show tables from test like "t?"; +Tables_in_test (t?) +show full columns from t2; +Field Type Collation Null Key Default Extra Privileges Comment +auto int(11) NULL NO PRI NULL auto_increment # +fld1 int(6) unsigned zerofill NULL NO UNI 000000 # +companynr tinyint(2) unsigned zerofill NULL NO 00 # +fld3 char(30) latin1_swedish_ci NO MUL # +fld4 char(35) latin1_swedish_ci NO # +fld5 char(35) latin1_swedish_ci NO # +fld6 char(4) latin1_swedish_ci NO # +show full columns from t2 from test like 'f%'; +Field Type Collation Null Key Default Extra Privileges Comment +fld1 int(6) unsigned zerofill NULL NO UNI 000000 # +fld3 char(30) latin1_swedish_ci NO MUL # +fld4 char(35) latin1_swedish_ci NO # +fld5 char(35) latin1_swedish_ci NO # +fld6 char(4) latin1_swedish_ci NO # +show full columns from t2 from test like 's%'; +Field Type Collation Null Key Default Extra Privileges Comment +show keys from t2; +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment +t2 0 PRIMARY 1 auto A 1199 NULL NULL BTREE +t2 0 fld1 1 fld1 A 1199 NULL NULL BTREE +t2 1 fld3 1 fld3 A NULL NULL NULL BTREE +drop table t4, t3, t2, t1; +DO 1; +DO benchmark(100,1+1),1,1; +do default; +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 1 +do foobar; +ERROR 42S22: Unknown column 'foobar' in 'field list' +CREATE TABLE t1 ( +id mediumint(8) unsigned NOT NULL auto_increment, +pseudo varchar(35) NOT NULL default '', +PRIMARY KEY (id), +UNIQUE KEY pseudo (pseudo) +); +INSERT INTO t1 (pseudo) VALUES ('test'); +INSERT INTO t1 (pseudo) VALUES ('test1'); +SELECT 1 as rnd1 from t1 where rand() > 2; +rnd1 +DROP TABLE t1; +CREATE TABLE t1 (gvid int(10) unsigned default NULL, hmid int(10) unsigned default NULL, volid int(10) unsigned default NULL, mmid int(10) unsigned default NULL, hdid int(10) unsigned default NULL, fsid int(10) unsigned default NULL, ctid int(10) unsigned default NULL, dtid int(10) unsigned default NULL, cost int(10) unsigned default NULL, performance int(10) unsigned default NULL, serialnumber bigint(20) unsigned default NULL, monitored tinyint(3) unsigned default '1', removed tinyint(3) unsigned default '0', target tinyint(3) unsigned default '0', dt_modified timestamp NOT NULL, name varchar(255) binary default NULL, description varchar(255) default NULL, UNIQUE KEY hmid (hmid,volid)) ENGINE=MyISAM; +INSERT INTO t1 VALUES (200001,2,1,1,100,1,1,1,0,0,0,1,0,1,20020425060057,'\\\\ARKIVIO-TESTPDC\\E$',''),(200002,2,2,1,101,1,1,1,0,0,0,1,0,1,20020425060057,'\\\\ARKIVIO-TESTPDC\\C$',''),(200003,1,3,2,NULL,NULL,NULL,NULL,NULL,NULL,NULL,1,0,1,20020425060427,'c:',NULL); +CREATE TABLE t2 ( hmid int(10) unsigned default NULL, volid int(10) unsigned default NULL, sampletid smallint(5) unsigned default NULL, sampletime datetime default NULL, samplevalue bigint(20) unsigned default NULL, KEY idx1 (hmid,volid,sampletid,sampletime)) ENGINE=MyISAM; +INSERT INTO t2 VALUES (1,3,10,'2002-06-01 08:00:00',35),(1,3,1010,'2002-06-01 12:00:01',35); +SELECT a.gvid, (SUM(CASE b.sampletid WHEN 140 THEN b.samplevalue ELSE 0 END)) as the_success,(SUM(CASE b.sampletid WHEN 141 THEN b.samplevalue ELSE 0 END)) as the_fail,(SUM(CASE b.sampletid WHEN 142 THEN b.samplevalue ELSE 0 END)) as the_size,(SUM(CASE b.sampletid WHEN 143 THEN b.samplevalue ELSE 0 END)) as the_time FROM t1 a, t2 b WHERE a.hmid = b.hmid AND a.volid = b.volid AND b.sampletime >= 'wrong-date-value' AND b.sampletime < 'wrong-date-value' AND b.sampletid IN (140, 141, 142, 143) GROUP BY a.gvid; +gvid the_success the_fail the_size the_time +Warnings: +Warning 1292 Incorrect datetime value: 'wrong-date-value' for column 'sampletime' at row 1 +Warning 1292 Incorrect datetime value: 'wrong-date-value' for column 'sampletime' at row 1 +SELECT a.gvid, (SUM(CASE b.sampletid WHEN 140 THEN b.samplevalue ELSE 0 END)) as the_success,(SUM(CASE b.sampletid WHEN 141 THEN b.samplevalue ELSE 0 END)) as the_fail,(SUM(CASE b.sampletid WHEN 142 THEN b.samplevalue ELSE 0 END)) as the_size,(SUM(CASE b.sampletid WHEN 143 THEN b.samplevalue ELSE 0 END)) as the_time FROM t1 a, t2 b WHERE a.hmid = b.hmid AND a.volid = b.volid AND b.sampletime >= NULL AND b.sampletime < NULL AND b.sampletid IN (140, 141, 142, 143) GROUP BY a.gvid; +gvid the_success the_fail the_size the_time +DROP TABLE t1,t2; +create table t1 ( A_Id bigint(20) NOT NULL default '0', A_UpdateBy char(10) NOT NULL default '', A_UpdateDate bigint(20) NOT NULL default '0', A_UpdateSerial int(11) NOT NULL default '0', other_types bigint(20) NOT NULL default '0', wss_type bigint(20) NOT NULL default '0'); +INSERT INTO t1 VALUES (102935998719055004,'brade',1029359987,2,102935229116544068,102935229216544093); +select wss_type from t1 where wss_type ='102935229216544106'; +wss_type +select wss_type from t1 where wss_type ='102935229216544105'; +wss_type +select wss_type from t1 where wss_type ='102935229216544104'; +wss_type +select wss_type from t1 where wss_type ='102935229216544093'; +wss_type +102935229216544093 +select wss_type from t1 where wss_type =102935229216544093; +wss_type +102935229216544093 +drop table t1; +select 1+2,"aaaa",3.13*2.0 into @a,@b,@c; +select @a; +@a +3 +select @b; +@b +aaaa +select @c; +@c +6.260 +create table t1 (a int not null auto_increment primary key); +insert into t1 values (); +insert into t1 values (); +insert into t1 values (); +select * from (t1 as t2 left join t1 as t3 using (a)), t1; +a a +1 1 +2 1 +3 1 +1 2 +2 2 +3 2 +1 3 +2 3 +3 3 +select * from t1, (t1 as t2 left join t1 as t3 using (a)); +a a +1 1 +2 1 +3 1 +1 2 +2 2 +3 2 +1 3 +2 3 +3 3 +select * from (t1 as t2 left join t1 as t3 using (a)) straight_join t1; +a a +1 1 +2 1 +3 1 +1 2 +2 2 +3 2 +1 3 +2 3 +3 3 +select * from t1 straight_join (t1 as t2 left join t1 as t3 using (a)); +a a +1 1 +2 1 +3 1 +1 2 +2 2 +3 2 +1 3 +2 3 +3 3 +select * from (t1 as t2 left join t1 as t3 using (a)) inner join t1 on t1.a>1; +a a +1 2 +2 2 +3 2 +1 3 +2 3 +3 3 +select * from t1 inner join (t1 as t2 left join t1 as t3 using (a)) on t1.a>1; +a a +2 1 +3 1 +2 2 +3 2 +2 3 +3 3 +select * from (t1 as t2 left join t1 as t3 using (a)) inner join t1 using ( a ); +a +1 +2 +3 +select * from t1 inner join (t1 as t2 left join t1 as t3 using (a)) using ( a ); +a +1 +2 +3 +select * from (t1 as t2 left join t1 as t3 using (a)) left outer join t1 on t1.a>1; +a a +1 2 +1 3 +2 2 +2 3 +3 2 +3 3 +select * from t1 left outer join (t1 as t2 left join t1 as t3 using (a)) on t1.a>1; +a a +1 NULL +2 1 +2 2 +2 3 +3 1 +3 2 +3 3 +select * from (t1 as t2 left join t1 as t3 using (a)) left join t1 using ( a ); +a +1 +2 +3 +select * from t1 left join (t1 as t2 left join t1 as t3 using (a)) using ( a ); +a +1 +2 +3 +select * from (t1 as t2 left join t1 as t3 using (a)) natural left join t1; +a +1 +2 +3 +select * from t1 natural left join (t1 as t2 left join t1 as t3 using (a)); +a +1 +2 +3 +select * from (t1 as t2 left join t1 as t3 using (a)) right join t1 on t1.a>1; +a a +NULL 1 +1 2 +2 2 +3 2 +1 3 +2 3 +3 3 +select * from t1 right join (t1 as t2 left join t1 as t3 using (a)) on t1.a>1; +a a +2 1 +3 1 +2 2 +3 2 +2 3 +3 3 +select * from (t1 as t2 left join t1 as t3 using (a)) right outer join t1 using ( a ); +a +1 +2 +3 +select * from t1 right outer join (t1 as t2 left join t1 as t3 using (a)) using ( a ); +a +1 +2 +3 +select * from (t1 as t2 left join t1 as t3 using (a)) natural right join t1; +a +1 +2 +3 +select * from t1 natural right join (t1 as t2 left join t1 as t3 using (a)); +a +1 +2 +3 +select * from t1 natural join (t1 as t2 left join t1 as t3 using (a)); +a +1 +2 +3 +select * from (t1 as t2 left join t1 as t3 using (a)) natural join t1; +a +1 +2 +3 +drop table t1; +CREATE TABLE t1 ( aa char(2), id int(11) NOT NULL auto_increment, t2_id int(11) NOT NULL default '0', PRIMARY KEY (id), KEY replace_id (t2_id)) ENGINE=MyISAM; +INSERT INTO t1 VALUES ("1",8264,2506),("2",8299,2517),("3",8301,2518),("4",8302,2519),("5",8303,2520),("6",8304,2521),("7",8305,2522); +CREATE TABLE t2 ( id int(11) NOT NULL auto_increment, PRIMARY KEY (id)) ENGINE=MyISAM; +INSERT INTO t2 VALUES (2517), (2518), (2519), (2520), (2521), (2522); +select * from t1, t2 WHERE t1.t2_id = t2.id and t1.t2_id > 0 order by t1.id LIMIT 0, 5; +aa id t2_id id +2 8299 2517 2517 +3 8301 2518 2518 +4 8302 2519 2519 +5 8303 2520 2520 +6 8304 2521 2521 +drop table t1,t2; +create table t1 (id1 int NOT NULL); +create table t2 (id2 int NOT NULL); +create table t3 (id3 int NOT NULL); +create table t4 (id4 int NOT NULL, id44 int NOT NULL, KEY (id4)); +insert into t1 values (1); +insert into t1 values (2); +insert into t2 values (1); +insert into t4 values (1,1); +explain select * from t1 left join t2 on id1 = id2 left join t3 on id1 = id3 +left join t4 on id3 = id4 where id2 = 1 or id4 = 1; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t3 system NULL NULL NULL NULL 0 const row not found +1 SIMPLE t4 const id4 NULL NULL NULL 1 +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 +1 SIMPLE t2 ALL NULL NULL NULL NULL 1 Using where +select * from t1 left join t2 on id1 = id2 left join t3 on id1 = id3 +left join t4 on id3 = id4 where id2 = 1 or id4 = 1; +id1 id2 id3 id4 id44 +1 1 NULL NULL NULL +drop table t1,t2,t3,t4; +create table t1(s varchar(10) not null); +create table t2(s varchar(10) not null primary key); +create table t3(s varchar(10) not null primary key); +insert into t1 values ('one\t'), ('two\t'); +insert into t2 values ('one\r'), ('two\t'); +insert into t3 values ('one '), ('two\t'); +select * from t1 where s = 'one'; +s +select * from t2 where s = 'one'; +s +select * from t3 where s = 'one'; +s +one +select * from t1,t2 where t1.s = t2.s; +s s +two two +select * from t2,t3 where t2.s = t3.s; +s s +two two +drop table t1, t2, t3; +create table t1 (a integer, b integer, index(a), index(b)); +create table t2 (c integer, d integer, index(c), index(d)); +insert into t1 values (1,2), (2,2), (3,2), (4,2); +insert into t2 values (1,3), (2,3), (3,4), (4,4); +explain select * from t1 left join t2 on a=c where d in (4); +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ref c,d d 5 const 2 Using where +1 SIMPLE t1 ALL a NULL NULL NULL 4 Using where; Using join buffer +select * from t1 left join t2 on a=c where d in (4); +a b c d +3 2 3 4 +4 2 4 4 +explain select * from t1 left join t2 on a=c where d = 4; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ref c,d d 5 const 2 Using where +1 SIMPLE t1 ALL a NULL NULL NULL 4 Using where; Using join buffer +select * from t1 left join t2 on a=c where d = 4; +a b c d +3 2 3 4 +4 2 4 4 +drop table t1, t2; +CREATE TABLE t1 ( +i int(11) NOT NULL default '0', +c char(10) NOT NULL default '', +PRIMARY KEY (i), +UNIQUE KEY c (c) +) ENGINE=MyISAM; +INSERT INTO t1 VALUES (1,'a'); +INSERT INTO t1 VALUES (2,'b'); +INSERT INTO t1 VALUES (3,'c'); +EXPLAIN SELECT i FROM t1 WHERE i=1; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 const PRIMARY PRIMARY 4 const 1 Using index +DROP TABLE t1; +CREATE TABLE t1 ( a BLOB, INDEX (a(20)) ); +CREATE TABLE t2 ( a BLOB, INDEX (a(20)) ); +INSERT INTO t1 VALUES ('one'),('two'),('three'),('four'),('five'); +INSERT INTO t2 VALUES ('one'),('two'),('three'),('four'),('five'); +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 USE INDEX (a) ON t1.a=t2.a; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ALL NULL NULL NULL NULL 5 +1 SIMPLE t2 ref a a 23 test.t1.a 2 +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 FORCE INDEX (a) ON t1.a=t2.a; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ALL NULL NULL NULL NULL 5 +1 SIMPLE t2 ref a a 23 test.t1.a 2 +DROP TABLE t1, t2; +CREATE TABLE t1 ( city char(30) ); +INSERT INTO t1 VALUES ('London'); +INSERT INTO t1 VALUES ('Paris'); +SELECT * FROM t1 WHERE city='London'; +city +London +SELECT * FROM t1 WHERE city='london'; +city +London +EXPLAIN SELECT * FROM t1 WHERE city='London' AND city='london'; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 Using where +SELECT * FROM t1 WHERE city='London' AND city='london'; +city +London +EXPLAIN SELECT * FROM t1 WHERE city LIKE '%london%' AND city='London'; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 Using where +SELECT * FROM t1 WHERE city LIKE '%london%' AND city='London'; +city +London +DROP TABLE t1; +create table t1 (a int(11) unsigned, b int(11) unsigned); +insert into t1 values (1,0), (1,1), (1,2); +select a-b from t1 order by 1; +a-b +0 +1 +18446744073709551615 +select a-b , (a-b < 0) from t1 order by 1; +a-b (a-b < 0) +0 0 +1 0 +18446744073709551615 0 +select a-b as d, (a-b >= 0), b from t1 group by b having d >= 0; +d (a-b >= 0) b +1 1 0 +0 1 1 +18446744073709551615 1 2 +select cast((a - b) as unsigned) from t1 order by 1; +cast((a - b) as unsigned) +0 +1 +18446744073709551615 +drop table t1; +create table t1 (a int(11)); +select all all * from t1; +a +select distinct distinct * from t1; +a +select all distinct * from t1; +ERROR HY000: Incorrect usage of ALL and DISTINCT +select distinct all * from t1; +ERROR HY000: Incorrect usage of ALL and DISTINCT +drop table t1; +CREATE TABLE t1 ( +kunde_intern_id int(10) unsigned NOT NULL default '0', +kunde_id int(10) unsigned NOT NULL default '0', +FK_firma_id int(10) unsigned NOT NULL default '0', +aktuell enum('Ja','Nein') NOT NULL default 'Ja', +vorname varchar(128) NOT NULL default '', +nachname varchar(128) NOT NULL default '', +geloescht enum('Ja','Nein') NOT NULL default 'Nein', +firma varchar(128) NOT NULL default '' +); +INSERT INTO t1 VALUES +(3964,3051,1,'Ja','Vorname1','1Nachname','Nein','Print Schau XXXX'), +(3965,3051111,1,'Ja','Vorname1111','1111Nachname','Nein','Print Schau XXXX'); +SELECT kunde_id ,FK_firma_id ,aktuell, vorname, nachname, geloescht FROM t1 +WHERE +( +( +( '' != '' AND firma LIKE CONCAT('%', '', '%')) +OR +(vorname LIKE CONCAT('%', 'Vorname1', '%') AND +nachname LIKE CONCAT('%', '1Nachname', '%') AND +'Vorname1' != '' AND 'xxxx' != '') +) +AND +( +aktuell = 'Ja' AND geloescht = 'Nein' AND FK_firma_id = 2 +) +) +; +kunde_id FK_firma_id aktuell vorname nachname geloescht +SELECT kunde_id ,FK_firma_id ,aktuell, vorname, nachname, +geloescht FROM t1 +WHERE +( +( +aktuell = 'Ja' AND geloescht = 'Nein' AND FK_firma_id = 2 +) +AND +( +( '' != '' AND firma LIKE CONCAT('%', '', '%') ) +OR +( vorname LIKE CONCAT('%', 'Vorname1', '%') AND +nachname LIKE CONCAT('%', '1Nachname', '%') AND 'Vorname1' != '' AND +'xxxx' != '') +) +) +; +kunde_id FK_firma_id aktuell vorname nachname geloescht +SELECT COUNT(*) FROM t1 WHERE +( 0 OR (vorname LIKE '%Vorname1%' AND nachname LIKE '%1Nachname%' AND 1)) +AND FK_firma_id = 2; +COUNT(*) +0 +drop table t1; +CREATE TABLE t1 (b BIGINT(20) UNSIGNED NOT NULL, PRIMARY KEY (b)); +INSERT INTO t1 VALUES (0x8000000000000000); +SELECT b FROM t1 WHERE b=0x8000000000000000; +b +9223372036854775808 +DROP TABLE t1; +CREATE TABLE `t1` ( `gid` int(11) default NULL, `uid` int(11) default NULL); +CREATE TABLE `t2` ( `ident` int(11) default NULL, `level` char(16) default NULL); +INSERT INTO `t2` VALUES (0,'READ'); +CREATE TABLE `t3` ( `id` int(11) default NULL, `name` char(16) default NULL); +INSERT INTO `t3` VALUES (1,'fs'); +select * from t3 left join t1 on t3.id = t1.uid, t2 where t2.ident in (0, t1.gid, t3.id, 0); +id name gid uid ident level +1 fs NULL NULL 0 READ +drop table t1,t2,t3; +CREATE TABLE t1 ( +acct_id int(11) NOT NULL default '0', +profile_id smallint(6) default NULL, +UNIQUE KEY t1$acct_id (acct_id), +KEY t1$profile_id (profile_id) +); +INSERT INTO t1 VALUES (132,17),(133,18); +CREATE TABLE t2 ( +profile_id smallint(6) default NULL, +queue_id int(11) default NULL, +seq int(11) default NULL, +KEY t2$queue_id (queue_id) +); +INSERT INTO t2 VALUES (17,31,4),(17,30,3),(17,36,2),(17,37,1); +CREATE TABLE t3 ( +id int(11) NOT NULL default '0', +qtype int(11) default NULL, +seq int(11) default NULL, +warn_lvl int(11) default NULL, +crit_lvl int(11) default NULL, +rr1 tinyint(4) NOT NULL default '0', +rr2 int(11) default NULL, +default_queue tinyint(4) NOT NULL default '0', +KEY t3$qtype (qtype), +KEY t3$id (id) +); +INSERT INTO t3 VALUES (30,1,29,NULL,NULL,0,NULL,0),(31,1,28,NULL,NULL,0,NULL,0), +(36,1,34,NULL,NULL,0,NULL,0),(37,1,35,NULL,NULL,0,121,0); +SELECT COUNT(*) FROM t1 a STRAIGHT_JOIN t2 pq STRAIGHT_JOIN t3 q +WHERE +(pq.profile_id = a.profile_id) AND (a.acct_id = 132) AND +(pq.queue_id = q.id) AND (q.rr1 <> 1); +COUNT(*) +4 +drop table t1,t2,t3; +create table t1 (f1 int); +insert into t1 values (1),(NULL); +create table t2 (f2 int, f3 int, f4 int); +create index idx1 on t2 (f4); +insert into t2 values (1,2,3),(2,4,6); +select A.f2 from t1 left join t2 A on A.f2 = f1 where A.f3=(select min(f3) +from t2 C where A.f4 = C.f4) or A.f3 IS NULL; +f2 +1 +NULL +drop table t1,t2; +create table t2 (a tinyint unsigned); +create index t2i on t2(a); +insert into t2 values (0), (254), (255); +explain select * from t2 where a > -1; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 index t2i t2i 2 NULL 3 Using where; Using index +select * from t2 where a > -1; +a +0 +254 +255 +drop table t2; +CREATE TABLE t1 (a int, b int, c int); +INSERT INTO t1 +SELECT 50, 3, 3 FROM DUAL +WHERE NOT EXISTS +(SELECT * FROM t1 WHERE a = 50 AND b = 3); +SELECT * FROM t1; +a b c +50 3 3 +INSERT INTO t1 +SELECT 50, 3, 3 FROM DUAL +WHERE NOT EXISTS +(SELECT * FROM t1 WHERE a = 50 AND b = 3); +select found_rows(); +found_rows() +0 +SELECT * FROM t1; +a b c +50 3 3 +select count(*) from t1; +count(*) +1 +select found_rows(); +found_rows() +1 +select count(*) from t1 limit 2,3; +count(*) +select found_rows(); +found_rows() +0 +select SQL_CALC_FOUND_ROWS count(*) from t1 limit 2,3; +count(*) +select found_rows(); +found_rows() +1 +DROP TABLE t1; +CREATE TABLE t1 (a INT, b INT); +(SELECT a, b AS c FROM t1) ORDER BY c+1; +a c +(SELECT a, b AS c FROM t1) ORDER BY b+1; +a c +SELECT a, b AS c FROM t1 ORDER BY c+1; +a c +SELECT a, b AS c FROM t1 ORDER BY b+1; +a c +drop table t1; +create table t1(f1 int, f2 int); +create table t2(f3 int); +select f1 from t1,t2 where f1=f2 and (f1,f2) = ((1,1)); +f1 +select f1 from t1,t2 where f1=f2 and (f1,NULL) = ((1,1)); +f1 +select f1 from t1,t2 where f1=f2 and (f1,f2) = ((1,NULL)); +f1 +insert into t1 values(1,1),(2,null); +insert into t2 values(2); +select * from t1,t2 where f1=f3 and (f1,f2) = (2,null); +f1 f2 f3 +select * from t1,t2 where f1=f3 and (f1,f2) <=> (2,null); +f1 f2 f3 +2 NULL 2 +drop table t1,t2; +create table t1 (f1 int not null auto_increment primary key, f2 varchar(10)); +create table t11 like t1; +insert into t1 values(1,""),(2,""); +show table status like 't1%'; +Name Engine Version Row_format Rows Avg_row_length Data_length Max_data_length Index_length Data_free Auto_increment Create_time Update_time Check_time Collation Checksum Create_options Comment +t1 MyISAM 10 Dynamic 2 20 X X X X X X X X latin1_swedish_ci NULL +t11 MyISAM 10 Dynamic 0 0 X X X X X X X X latin1_swedish_ci NULL +select 123 as a from t1 where f1 is null; +a +drop table t1,t11; +CREATE TABLE t1 ( a INT NOT NULL, b INT NOT NULL, UNIQUE idx (a,b) ); +INSERT INTO t1 VALUES (1,1),(1,2),(1,3),(1,4); +CREATE TABLE t2 ( a INT NOT NULL, b INT NOT NULL, e INT ); +INSERT INTO t2 VALUES ( 1,10,1), (1,10,2), (1,11,1), (1,11,2), (1,2,1), (1,2,2),(1,2,3); +SELECT t2.a, t2.b, IF(t1.b IS NULL,'',e) AS c, COUNT(*) AS d FROM t2 LEFT JOIN +t1 ON t2.a = t1.a AND t2.b = t1.b GROUP BY a, b, c; +a b c d +1 2 1 1 +1 2 2 1 +1 2 3 1 +1 10 2 +1 11 2 +SELECT t2.a, t2.b, IF(t1.b IS NULL,'',e) AS c, COUNT(*) AS d FROM t2 LEFT JOIN +t1 ON t2.a = t1.a AND t2.b = t1.b GROUP BY t1.a, t1.b, c; +a b c d +1 10 4 +1 2 1 1 +1 2 2 1 +1 2 3 1 +SELECT t2.a, t2.b, IF(t1.b IS NULL,'',e) AS c, COUNT(*) AS d FROM t2 LEFT JOIN +t1 ON t2.a = t1.a AND t2.b = t1.b GROUP BY t2.a, t2.b, c; +a b c d +1 2 1 1 +1 2 2 1 +1 2 3 1 +1 10 2 +1 11 2 +SELECT t2.a, t2.b, IF(t1.b IS NULL,'',e) AS c, COUNT(*) AS d FROM t2,t1 +WHERE t2.a = t1.a AND t2.b = t1.b GROUP BY a, b, c; +a b c d +1 2 1 1 +1 2 2 1 +1 2 3 1 +DROP TABLE IF EXISTS t1, t2; +create table t1 (f1 int primary key, f2 int); +create table t2 (f3 int, f4 int, primary key(f3,f4)); +insert into t1 values (1,1); +insert into t2 values (1,1),(1,2); +select distinct count(f2) >0 from t1 left join t2 on f1=f3 group by f1; +count(f2) >0 +1 +drop table t1,t2; +create table t1 (f1 int,f2 int); +insert into t1 values(1,1); +create table t2 (f3 int, f4 int, primary key(f3,f4)); +insert into t2 values(1,1); +select * from t1 where f1 in (select f3 from t2 where (f3,f4)= (select f3,f4 from t2)); +f1 f2 +1 1 +drop table t1,t2; +CREATE TABLE t1(a int, b int, c int, KEY b(b), KEY c(c)); +insert into t1 values (1,0,0),(2,0,0); +CREATE TABLE t2 (a int, b varchar(2), c varchar(2), PRIMARY KEY(a)); +insert into t2 values (1,'',''), (2,'',''); +CREATE TABLE t3 (a int, b int, PRIMARY KEY (a,b), KEY a (a), KEY b (b)); +insert into t3 values (1,1),(1,2); +explain select straight_join DISTINCT t2.a,t2.b, t1.c from t1, t3, t2 +where (t1.c=t2.a or (t1.c=t3.a and t2.a=t3.b)) and t1.b=556476786 and +t2.b like '%%' order by t2.b limit 0,1; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ref b,c b 5 const 1 Using where; Using temporary; Using filesort +1 SIMPLE t3 index PRIMARY,a,b PRIMARY 8 NULL 2 Using index; Using join buffer +1 SIMPLE t2 ALL PRIMARY NULL NULL NULL 2 Range checked for each record (index map: 0x1) +DROP TABLE t1,t2,t3; +CREATE TABLE t1 (a int, INDEX idx(a)); +INSERT INTO t1 VALUES (2), (3), (1); +EXPLAIN SELECT * FROM t1 IGNORE INDEX (idx); +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ALL NULL NULL NULL NULL 3 +EXPLAIN SELECT * FROM t1 IGNORE INDEX (a); +ERROR 42000: Key 'a' doesn't exist in table 't1' +EXPLAIN SELECT * FROM t1 FORCE INDEX (a); +ERROR 42000: Key 'a' doesn't exist in table 't1' +DROP TABLE t1; +CREATE TABLE t1 (a int, b int); +INSERT INTO t1 VALUES (1,1), (2,1), (4,10); +CREATE TABLE t2 (a int PRIMARY KEY, b int, KEY b (b)); +INSERT INTO t2 VALUES (1,NULL), (2,10); +ALTER TABLE t1 ENABLE KEYS; +EXPLAIN SELECT STRAIGHT_JOIN SQL_NO_CACHE COUNT(*) FROM t2, t1 WHERE t1.b = t2.b OR t2.b IS NULL; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 index b b 5 NULL 2 Using index +1 SIMPLE t1 ALL NULL NULL NULL NULL 3 Using where; Using join buffer +SELECT STRAIGHT_JOIN SQL_NO_CACHE * FROM t2, t1 WHERE t1.b = t2.b OR t2.b IS NULL; +a b a b +1 NULL 1 1 +1 NULL 2 1 +1 NULL 4 10 +2 10 4 10 +EXPLAIN SELECT STRAIGHT_JOIN SQL_NO_CACHE COUNT(*) FROM t2, t1 WHERE t1.b = t2.b OR t2.b IS NULL; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 index b b 5 NULL 2 Using index +1 SIMPLE t1 ALL NULL NULL NULL NULL 3 Using where; Using join buffer +SELECT STRAIGHT_JOIN SQL_NO_CACHE * FROM t2, t1 WHERE t1.b = t2.b OR t2.b IS NULL; +a b a b +1 NULL 1 1 +1 NULL 2 1 +1 NULL 4 10 +2 10 4 10 +DROP TABLE IF EXISTS t1,t2; +CREATE TABLE t1 (key1 float default NULL, UNIQUE KEY key1 (key1)); +CREATE TABLE t2 (key2 float default NULL, UNIQUE KEY key2 (key2)); +INSERT INTO t1 VALUES (0.3762),(0.3845),(0.6158),(0.7941); +INSERT INTO t2 VALUES (1.3762),(1.3845),(1.6158),(1.7941); +explain select max(key1) from t1 where key1 <= 0.6158; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Select tables optimized away +explain select max(key2) from t2 where key2 <= 1.6158; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Select tables optimized away +explain select min(key1) from t1 where key1 >= 0.3762; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Select tables optimized away +explain select min(key2) from t2 where key2 >= 1.3762; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Select tables optimized away +explain select max(key1), min(key2) from t1, t2 +where key1 <= 0.6158 and key2 >= 1.3762; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Select tables optimized away +explain select max(key1) from t1 where key1 <= 0.6158 and rand() + 0.5 >= 0.5; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Select tables optimized away +explain select min(key1) from t1 where key1 >= 0.3762 and rand() + 0.5 >= 0.5; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Select tables optimized away +select max(key1) from t1 where key1 <= 0.6158; +max(key1) +0.615800023078918 +select max(key2) from t2 where key2 <= 1.6158; +max(key2) +1.61580002307892 +select min(key1) from t1 where key1 >= 0.3762; +min(key1) +0.376199990510941 +select min(key2) from t2 where key2 >= 1.3762; +min(key2) +1.37619996070862 +select max(key1), min(key2) from t1, t2 +where key1 <= 0.6158 and key2 >= 1.3762; +max(key1) min(key2) +0.615800023078918 1.37619996070862 +select max(key1) from t1 where key1 <= 0.6158 and rand() + 0.5 >= 0.5; +max(key1) +0.615800023078918 +select min(key1) from t1 where key1 >= 0.3762 and rand() + 0.5 >= 0.5; +min(key1) +0.376199990510941 +DROP TABLE t1,t2; +CREATE TABLE t1 (i BIGINT UNSIGNED NOT NULL); +INSERT INTO t1 VALUES (10); +SELECT i='1e+01',i=1e+01, i in (1e+01,1e+01), i in ('1e+01','1e+01') FROM t1; +i='1e+01' i=1e+01 i in (1e+01,1e+01) i in ('1e+01','1e+01') +1 1 1 1 +DROP TABLE t1; +create table t1(a bigint unsigned, b bigint); +insert into t1 values (0xfffffffffffffffff, 0xfffffffffffffffff), +(0x10000000000000000, 0x10000000000000000), +(0x8fffffffffffffff, 0x8fffffffffffffff); +Warnings: +Warning 1264 Out of range value for column 'a' at row 1 +Warning 1264 Out of range value for column 'b' at row 1 +Warning 1264 Out of range value for column 'a' at row 2 +Warning 1264 Out of range value for column 'b' at row 2 +Warning 1264 Out of range value for column 'b' at row 3 +select hex(a), hex(b) from t1; +hex(a) hex(b) +FFFFFFFFFFFFFFFF 7FFFFFFFFFFFFFFF +FFFFFFFFFFFFFFFF 7FFFFFFFFFFFFFFF +8FFFFFFFFFFFFFFF 7FFFFFFFFFFFFFFF +drop table t1; +CREATE TABLE t1 (c0 int); +CREATE TABLE t2 (c0 int); +INSERT INTO t1 VALUES(@@connect_timeout); +INSERT INTO t2 VALUES(@@connect_timeout); +SELECT * FROM t1 JOIN t2 ON t1.c0 = t2.c0 WHERE (t1.c0 <=> @@connect_timeout); +c0 c0 +X X +DROP TABLE t1, t2; +End of 4.1 tests +CREATE TABLE t1 ( +K2C4 varchar(4) character set latin1 collate latin1_bin NOT NULL default '', +K4N4 varchar(4) character set latin1 collate latin1_bin NOT NULL default '0000', +F2I4 int(11) NOT NULL default '0' +) ENGINE=MyISAM DEFAULT CHARSET=latin1; +INSERT INTO t1 VALUES +('W%RT', '0100', 1), +('W-RT', '0100', 1), +('WART', '0100', 1), +('WART', '0200', 1), +('WERT', '0100', 2), +('WORT','0200', 2), +('WT', '0100', 2), +('W_RT', '0100', 2), +('WaRT', '0100', 3), +('WART', '0300', 3), +('WRT' , '0400', 3), +('WURM', '0500', 3), +('W%T', '0600', 4), +('WA%T', '0700', 4), +('WA_T', '0800', 4); +SELECT K2C4, K4N4, F2I4 FROM t1 +WHERE K2C4 = 'WART' AND +(F2I4 = 2 AND K2C4 = 'WART' OR (F2I4 = 2 OR K4N4 = '0200')); +K2C4 K4N4 F2I4 +WART 0200 1 +SELECT K2C4, K4N4, F2I4 FROM t1 +WHERE K2C4 = 'WART' AND (K2C4 = 'WART' OR K4N4 = '0200'); +K2C4 K4N4 F2I4 +WART 0100 1 +WART 0200 1 +WART 0300 3 +DROP TABLE t1; +create table t1 (a int, b int); +create table t2 like t1; +select t1.a from (t1 inner join t2 on t1.a=t2.a) where t2.a=1; +a +select t1.a from ((t1 inner join t2 on t1.a=t2.a)) where t2.a=1; +a +select x.a, y.a, z.a from ( (t1 x inner join t2 y on x.a=y.a) inner join t2 z on y.a=z.a) WHERE x.a=1; +a a a +drop table t1,t2; +create table t1 (s1 varchar(5)); +insert into t1 values ('Wall'); +select min(s1) from t1 group by s1 with rollup; +min(s1) +Wall +Wall +drop table t1; +create table t1 (s1 int) engine=myisam; +insert into t1 values (0); +select avg(distinct s1) from t1 group by s1 with rollup; +avg(distinct s1) +0.0000 +0.0000 +drop table t1; +create table t1 (s1 int); +insert into t1 values (null),(1); +select distinct avg(s1) as x from t1 group by s1 with rollup; +x +NULL +1.0000 +drop table t1; +CREATE TABLE t1 (a int); +CREATE TABLE t2 (a int); +INSERT INTO t1 VALUES (1), (2), (3), (4), (5); +INSERT INTO t2 VALUES (2), (4), (6); +SELECT t1.a FROM t1 STRAIGHT_JOIN t2 ON t1.a=t2.a; +a +2 +4 +EXPLAIN SELECT t1.a FROM t1 STRAIGHT_JOIN t2 ON t1.a=t2.a; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ALL NULL NULL NULL NULL 5 +1 SIMPLE t2 ALL NULL NULL NULL NULL 3 Using where; Using join buffer +EXPLAIN SELECT t1.a FROM t1 INNER JOIN t2 ON t1.a=t2.a; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ALL NULL NULL NULL NULL 3 +1 SIMPLE t1 ALL NULL NULL NULL NULL 5 Using where; Using join buffer +DROP TABLE t1,t2; +select x'10' + 0, X'10' + 0, b'10' + 0, B'10' + 0; +x'10' + 0 X'10' + 0 b'10' + 0 B'10' + 0 +16 16 2 2 +create table t1 (f1 varchar(6) default NULL, f2 int(6) primary key not null); +create table t2 (f3 varchar(5) not null, f4 varchar(5) not null, UNIQUE KEY UKEY (f3,f4)); +insert into t1 values (" 2", 2); +insert into t2 values (" 2", " one "),(" 2", " two "); +select * from t1 left join t2 on f1 = f3; +f1 f2 f3 f4 + 2 2 2 one + 2 2 2 two +drop table t1,t2; +create table t1 (empnum smallint, grp int); +create table t2 (empnum int, name char(5)); +insert into t1 values(1,1); +insert into t2 values(1,'bob'); +create view v1 as select * from t2 inner join t1 using (empnum); +select * from v1; +empnum name grp +1 bob 1 +drop table t1,t2; +drop view v1; +create table t1 (pk int primary key, b int); +create table t2 (pk int primary key, c int); +select pk from t1 inner join t2 using (pk); +pk +drop table t1,t2; +create table t1 (s1 int, s2 char(5), s3 decimal(10)); +create view v1 as select s1, s2, 'x' as s3 from t1; +select * from t1 natural join v1; +s1 s2 s3 +insert into t1 values (1,'x',5); +select * from t1 natural join v1; +s1 s2 s3 +Warnings: +Warning 1292 Truncated incorrect DOUBLE value: 'x' +drop table t1; +drop view v1; +create table t1(a1 int); +create table t2(a2 int); +insert into t1 values(1),(2); +insert into t2 values(1),(2); +create view v2 (c) as select a1 from t1; +select * from t1 natural left join t2; +a1 a2 +1 1 +1 2 +2 1 +2 2 +select * from t1 natural right join t2; +a2 a1 +1 1 +1 2 +2 1 +2 2 +select * from v2 natural left join t2; +c a2 +1 1 +1 2 +2 1 +2 2 +select * from v2 natural right join t2; +a2 c +1 1 +1 2 +2 1 +2 2 +drop table t1, t2; +drop view v2; +create table t1 (a int(10), t1_val int(10)); +create table t2 (b int(10), t2_val int(10)); +create table t3 (a int(10), b int(10)); +insert into t1 values (1,1),(2,2); +insert into t2 values (1,1),(2,2),(3,3); +insert into t3 values (1,1),(2,1),(3,1),(4,1); +select * from t1 natural join t2 natural join t3; +a b t1_val t2_val +1 1 1 1 +2 1 2 1 +select * from t1 natural join t3 natural join t2; +b a t1_val t2_val +1 1 1 1 +1 2 2 1 +drop table t1, t2, t3; +DO IFNULL(NULL, NULL); +SELECT CAST(IFNULL(NULL, NULL) AS DECIMAL); +CAST(IFNULL(NULL, NULL) AS DECIMAL) +NULL +SELECT ABS(IFNULL(NULL, NULL)); +ABS(IFNULL(NULL, NULL)) +NULL +SELECT IFNULL(NULL, NULL); +IFNULL(NULL, NULL) +NULL +SET @OLD_SQL_MODE12595=@@SQL_MODE, @@SQL_MODE=''; +SHOW LOCAL VARIABLES LIKE 'SQL_MODE'; +Variable_name Value +sql_mode +CREATE TABLE BUG_12595(a varchar(100)); +INSERT INTO BUG_12595 VALUES ('hakan%'), ('hakank'), ("ha%an"); +SELECT * FROM BUG_12595 WHERE a LIKE 'hakan\%'; +a +hakan% +SELECT * FROM BUG_12595 WHERE a LIKE 'hakan*%' ESCAPE '*'; +a +hakan% +SELECT * FROM BUG_12595 WHERE a LIKE 'hakan**%' ESCAPE '**'; +ERROR HY000: Incorrect arguments to ESCAPE +SELECT * FROM BUG_12595 WHERE a LIKE 'hakan%' ESCAPE ''; +a +hakan% +hakank +SELECT * FROM BUG_12595 WHERE a LIKE 'hakan\%' ESCAPE ''; +a +SELECT * FROM BUG_12595 WHERE a LIKE 'ha\%an' ESCAPE 0x5c; +a +ha%an +SELECT * FROM BUG_12595 WHERE a LIKE 'ha%%an' ESCAPE '%'; +a +ha%an +SELECT * FROM BUG_12595 WHERE a LIKE 'ha\%an' ESCAPE '\\'; +a +ha%an +SELECT * FROM BUG_12595 WHERE a LIKE 'ha|%an' ESCAPE '|'; +a +ha%an +SET @@SQL_MODE='NO_BACKSLASH_ESCAPES'; +SHOW LOCAL VARIABLES LIKE 'SQL_MODE'; +Variable_name Value +sql_mode NO_BACKSLASH_ESCAPES +SELECT * FROM BUG_12595 WHERE a LIKE 'hakan\%'; +a +SELECT * FROM BUG_12595 WHERE a LIKE 'hakan*%' ESCAPE '*'; +a +hakan% +SELECT * FROM BUG_12595 WHERE a LIKE 'hakan**%' ESCAPE '**'; +ERROR HY000: Incorrect arguments to ESCAPE +SELECT * FROM BUG_12595 WHERE a LIKE 'hakan\%' ESCAPE '\\'; +ERROR HY000: Incorrect arguments to ESCAPE +SELECT * FROM BUG_12595 WHERE a LIKE 'hakan%' ESCAPE ''; +ERROR HY000: Incorrect arguments to ESCAPE +SELECT * FROM BUG_12595 WHERE a LIKE 'ha\%an' ESCAPE 0x5c; +a +ha%an +SELECT * FROM BUG_12595 WHERE a LIKE 'ha|%an' ESCAPE '|'; +a +ha%an +SELECT * FROM BUG_12595 WHERE a LIKE 'hakan\n%' ESCAPE '\n'; +ERROR HY000: Incorrect arguments to ESCAPE +SET @@SQL_MODE=@OLD_SQL_MODE12595; +DROP TABLE BUG_12595; +create table t1 (a char(1)); +create table t2 (a char(1)); +insert into t1 values ('a'),('b'),('c'); +insert into t2 values ('b'),('c'),('d'); +select a from t1 natural join t2; +a +b +c +select * from t1 natural join t2 where a = 'b'; +a +b +drop table t1, t2; +CREATE TABLE t1 (`id` TINYINT); +CREATE TABLE t2 (`id` TINYINT); +CREATE TABLE t3 (`id` TINYINT); +INSERT INTO t1 VALUES (1),(2),(3); +INSERT INTO t2 VALUES (2); +INSERT INTO t3 VALUES (3); +SELECT t1.id,t3.id FROM t1 JOIN t2 ON (t2.id=t1.id) LEFT JOIN t3 USING (id); +ERROR 23000: Column 'id' in from clause is ambiguous +SELECT t1.id,t3.id FROM t1 JOIN t2 ON (t2.notacolumn=t1.id) LEFT JOIN t3 USING (id); +ERROR 23000: Column 'id' in from clause is ambiguous +SELECT id,t3.id FROM t1 JOIN t2 ON (t2.id=t1.id) LEFT JOIN t3 USING (id); +ERROR 23000: Column 'id' in from clause is ambiguous +SELECT id,t3.id FROM (t1 JOIN t2 ON (t2.id=t1.id)) LEFT JOIN t3 USING (id); +ERROR 23000: Column 'id' in from clause is ambiguous +drop table t1, t2, t3; +create table t1 (a int(10),b int(10)); +create table t2 (a int(10),b int(10)); +insert into t1 values (1,10),(2,20),(3,30); +insert into t2 values (1,10); +select * from t1 inner join t2 using (A); +a b b +1 10 10 +select * from t1 inner join t2 using (a); +a b b +1 10 10 +drop table t1, t2; +create table t1 (a int, c int); +create table t2 (b int); +create table t3 (b int, a int); +create table t4 (c int); +insert into t1 values (1,1); +insert into t2 values (1); +insert into t3 values (1,1); +insert into t4 values (1); +select * from t1 join t2 join t3 on (t2.b = t3.b and t1.a = t3.a); +a c b b a +1 1 1 1 1 +select * from t1, t2 join t3 on (t2.b = t3.b and t1.a = t3.a); +ERROR 42S22: Unknown column 't1.a' in 'on clause' +select * from t1 join t2 join t3 join t4 on (t1.a = t4.c and t2.b = t4.c); +a c b b a c +1 1 1 1 1 1 +select * from t1 join t2 join t4 using (c); +c a b +1 1 1 +drop table t1, t2, t3, t4; +create table t1(x int, y int); +create table t2(x int, y int); +create table t3(x int, primary key(x)); +insert into t1 values (1, 1), (2, 1), (3, 1), (4, 3), (5, 6), (6, 6); +insert into t2 values (1, 1), (2, 1), (3, 3), (4, 6), (5, 6); +insert into t3 values (1), (2), (3), (4), (5); +select t1.x, t3.x from t1, t2, t3 where t1.x = t2.x and t3.x >= t1.y and t3.x <= t2.y; +x x +1 1 +2 1 +3 1 +3 2 +3 3 +4 3 +4 4 +4 5 +drop table t1,t2,t3; +create table t1 (id char(16) not null default '', primary key (id)); +insert into t1 values ('100'),('101'),('102'); +create table t2 (id char(16) default null); +insert into t2 values (1); +create view v1 as select t1.id from t1; +create view v2 as select t2.id from t2; +create view v3 as select (t1.id+2) as id from t1 natural left join t2; +select t1.id from t1 left join v2 using (id); +id +100 +101 +102 +select t1.id from v2 right join t1 using (id); +id +100 +101 +102 +select t1.id from t1 left join v3 using (id); +id +100 +101 +102 +select * from t1 left join v2 using (id); +id +100 +101 +102 +select * from v2 right join t1 using (id); +id +100 +101 +102 +select * from t1 left join v3 using (id); +id +100 +101 +102 +select v1.id from v1 left join v2 using (id); +id +100 +101 +102 +select v1.id from v2 right join v1 using (id); +id +100 +101 +102 +select v1.id from v1 left join v3 using (id); +id +100 +101 +102 +select * from v1 left join v2 using (id); +id +100 +101 +102 +select * from v2 right join v1 using (id); +id +100 +101 +102 +select * from v1 left join v3 using (id); +id +100 +101 +102 +drop table t1, t2; +drop view v1, v2, v3; +create table t1 (id int(11) not null default '0'); +insert into t1 values (123),(191),(192); +create table t2 (id char(16) character set utf8 not null); +insert into t2 values ('58013'),('58014'),('58015'),('58016'); +create table t3 (a_id int(11) not null, b_id char(16) character set utf8); +insert into t3 values (123,null),(123,null),(123,null),(123,null),(123,null),(123,'58013'); +select count(*) +from t1 inner join (t3 left join t2 on t2.id = t3.b_id) on t1.id = t3.a_id; +count(*) +6 +select count(*) +from t1 inner join (t2 right join t3 on t2.id = t3.b_id) on t1.id = t3.a_id; +count(*) +6 +drop table t1,t2,t3; +create table t1 (a int); +create table t2 (b int); +create table t3 (c int); +select * from t1 join t2 join t3 on (t1.a=t3.c); +a b c +select * from t1 join t2 left join t3 on (t1.a=t3.c); +a b c +select * from t1 join t2 right join t3 on (t1.a=t3.c); +a b c +select * from t1 join t2 straight_join t3 on (t1.a=t3.c); +a b c +drop table t1, t2 ,t3; +create table t1(f1 int, f2 date); +insert into t1 values(1,'2005-01-01'),(2,'2005-09-01'),(3,'2005-09-30'), +(4,'2005-10-01'),(5,'2005-12-30'); +select * from t1 where f2 >= 0 order by f2; +f1 f2 +1 2005-01-01 +2 2005-09-01 +3 2005-09-30 +4 2005-10-01 +5 2005-12-30 +select * from t1 where f2 >= '0000-00-00' order by f2; +f1 f2 +1 2005-01-01 +2 2005-09-01 +3 2005-09-30 +4 2005-10-01 +5 2005-12-30 +select * from t1 where f2 >= '2005-09-31' order by f2; +f1 f2 +4 2005-10-01 +5 2005-12-30 +select * from t1 where f2 >= '2005-09-3a' order by f2; +f1 f2 +3 2005-09-30 +4 2005-10-01 +5 2005-12-30 +Warnings: +Warning 1292 Incorrect date value: '2005-09-3a' for column 'f2' at row 1 +select * from t1 where f2 <= '2005-09-31' order by f2; +f1 f2 +1 2005-01-01 +2 2005-09-01 +3 2005-09-30 +select * from t1 where f2 <= '2005-09-3a' order by f2; +f1 f2 +1 2005-01-01 +2 2005-09-01 +Warnings: +Warning 1292 Incorrect date value: '2005-09-3a' for column 'f2' at row 1 +drop table t1; +create table t1 (f1 int, f2 int); +insert into t1 values (1, 30), (2, 20), (3, 10); +create algorithm=merge view v1 as select f1, f2 from t1; +create algorithm=merge view v2 (f2, f1) as select f1, f2 from t1; +create algorithm=merge view v3 as select t1.f1 as f2, t1.f2 as f1 from t1; +select t1.f1 as x1, f1 from t1 order by t1.f1; +x1 f1 +1 1 +2 2 +3 3 +select v1.f1 as x1, f1 from v1 order by v1.f1; +x1 f1 +1 1 +2 2 +3 3 +select v2.f1 as x1, f1 from v2 order by v2.f1; +x1 f1 +10 10 +20 20 +30 30 +select v3.f1 as x1, f1 from v3 order by v3.f1; +x1 f1 +10 10 +20 20 +30 30 +select f1, f2, v1.f1 as x1 from v1 order by v1.f1; +f1 f2 x1 +1 30 1 +2 20 2 +3 10 3 +select f1, f2, v2.f1 as x1 from v2 order by v2.f1; +f1 f2 x1 +10 3 10 +20 2 20 +30 1 30 +select f1, f2, v3.f1 as x1 from v3 order by v3.f1; +f1 f2 x1 +10 3 10 +20 2 20 +30 1 30 +drop table t1; +drop view v1, v2, v3; +CREATE TABLE t1(key_a int4 NOT NULL, optimus varchar(32), PRIMARY KEY(key_a)); +CREATE TABLE t2(key_a int4 NOT NULL, prime varchar(32), PRIMARY KEY(key_a)); +CREATE table t3(key_a int4 NOT NULL, key_b int4 NOT NULL, foo varchar(32), +PRIMARY KEY(key_a,key_b)); +INSERT INTO t1 VALUES (0,''); +INSERT INTO t1 VALUES (1,'i'); +INSERT INTO t1 VALUES (2,'j'); +INSERT INTO t1 VALUES (3,'k'); +INSERT INTO t2 VALUES (1,'r'); +INSERT INTO t2 VALUES (2,'s'); +INSERT INTO t2 VALUES (3,'t'); +INSERT INTO t3 VALUES (1,5,'x'); +INSERT INTO t3 VALUES (1,6,'y'); +INSERT INTO t3 VALUES (2,5,'xx'); +INSERT INTO t3 VALUES (2,6,'yy'); +INSERT INTO t3 VALUES (2,7,'zz'); +INSERT INTO t3 VALUES (3,5,'xxx'); +SELECT t2.key_a,foo +FROM t1 INNER JOIN t2 ON t1.key_a = t2.key_a +INNER JOIN t3 ON t1.key_a = t3.key_a +WHERE t2.key_a=2 and key_b=5; +key_a foo +2 xx +EXPLAIN SELECT t2.key_a,foo +FROM t1 INNER JOIN t2 ON t1.key_a = t2.key_a +INNER JOIN t3 ON t1.key_a = t3.key_a +WHERE t2.key_a=2 and key_b=5; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 const PRIMARY PRIMARY 4 const 1 Using index +1 SIMPLE t2 const PRIMARY PRIMARY 4 const 1 Using index +1 SIMPLE t3 const PRIMARY PRIMARY 8 const,const 1 +SELECT t2.key_a,foo +FROM t1 INNER JOIN t2 ON t2.key_a = t1.key_a +INNER JOIN t3 ON t1.key_a = t3.key_a +WHERE t2.key_a=2 and key_b=5; +key_a foo +2 xx +EXPLAIN SELECT t2.key_a,foo +FROM t1 INNER JOIN t2 ON t2.key_a = t1.key_a +INNER JOIN t3 ON t1.key_a = t3.key_a +WHERE t2.key_a=2 and key_b=5; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 const PRIMARY PRIMARY 4 const 1 Using index +1 SIMPLE t2 const PRIMARY PRIMARY 4 const 1 Using index +1 SIMPLE t3 const PRIMARY PRIMARY 8 const,const 1 +DROP TABLE t1,t2,t3; +create table t1 (f1 int); +insert into t1 values(1),(2); +create table t2 (f2 int, f3 int, key(f2)); +insert into t2 values(1,1),(2,2); +create table t3 (f4 int not null); +insert into t3 values (2),(2),(2); +select f1,(select count(*) from t2,t3 where f2=f1 and f3=f4) as count from t1; +f1 count +1 0 +2 3 +drop table t1,t2,t3; +create table t1 (f1 int unique); +create table t2 (f2 int unique); +create table t3 (f3 int unique); +insert into t1 values(1),(2); +insert into t2 values(1),(2); +insert into t3 values(1),(NULL); +select * from t3 where f3 is null; +f3 +NULL +select t2.f2 from t1 left join t2 on f1=f2 join t3 on f1=f3 where f1=1; +f2 +1 +drop table t1,t2,t3; +create table t1(f1 char, f2 char not null); +insert into t1 values(null,'a'); +create table t2 (f2 char not null); +insert into t2 values('b'); +select * from t1 left join t2 on f1=t2.f2 where t1.f2='a'; +f1 f2 f2 +NULL a NULL +drop table t1,t2; +select * from (select * left join t on f1=f2) tt; +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'on f1=f2) tt' at line 1 +CREATE TABLE t1 (sku int PRIMARY KEY, pr int); +CREATE TABLE t2 (sku int PRIMARY KEY, sppr int, name varchar(255)); +INSERT INTO t1 VALUES +(10, 10), (20, 10), (30, 20), (40, 30), (50, 10), (60, 10); +INSERT INTO t2 VALUES +(10, 10, 'aaa'), (20, 10, 'bbb'), (30, 10, 'ccc'), (40, 20, 'ddd'), +(50, 10, 'eee'), (60, 20, 'fff'), (70, 20, 'ggg'), (80, 30, 'hhh'); +SELECT t2.sku, t2.sppr, t2.name, t1.sku, t1.pr +FROM t2, t1 WHERE t2.sku=20 AND (t2.sku=t1.sku OR t2.sppr=t1.sku); +sku sppr name sku pr +20 10 bbb 10 10 +20 10 bbb 20 10 +EXPLAIN +SELECT t2.sku, t2.sppr, t2.name, t1.sku, t1.pr +FROM t2, t1 WHERE t2.sku=20 AND (t2.sku=t1.sku OR t2.sppr=t1.sku); +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 const PRIMARY PRIMARY 4 const 1 +1 SIMPLE t1 range PRIMARY PRIMARY 4 NULL 2 Using where +DROP TABLE t1,t2; +CREATE TABLE t1 (i TINYINT UNSIGNED NOT NULL); +INSERT t1 SET i = 0; +UPDATE t1 SET i = -1; +Warnings: +Warning 1264 Out of range value for column 'i' at row 1 +SELECT * FROM t1; +i +0 +UPDATE t1 SET i = CAST(i - 1 AS SIGNED); +Warnings: +Warning 1264 Out of range value for column 'i' at row 1 +SELECT * FROM t1; +i +0 +UPDATE t1 SET i = i - 1; +Warnings: +Warning 1264 Out of range value for column 'i' at row 1 +SELECT * FROM t1; +i +255 +DROP TABLE t1; +create table t1 (a int); +insert into t1 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); +create table t2 (a int, b int, c int, e int, primary key(a,b,c)); +insert into t2 select A.a, B.a, C.a, C.a from t1 A, t1 B, t1 C; +analyze table t2; +Table Op Msg_type Msg_text +test.t2 analyze status OK +select 'In next EXPLAIN, B.rows must be exactly 10:' Z; +Z +In next EXPLAIN, B.rows must be exactly 10: +explain select * from t2 A, t2 B where A.a=5 and A.b=5 and A.C<5 +and B.a=5 and B.b=A.e and (B.b =1 or B.b = 3 or B.b=5); +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE A range PRIMARY PRIMARY 12 NULL 4 Using where +1 SIMPLE B ref PRIMARY PRIMARY 8 const,test.A.e 10 +drop table t1, t2; +CREATE TABLE t1 (a int PRIMARY KEY, b int, INDEX(b)); +INSERT INTO t1 VALUES (1, 3), (9,4), (7,5), (4,5), (6,2), +(3,1), (5,1), (8,9), (2,2), (0,9); +CREATE TABLE t2 (c int, d int, f int, INDEX(c,f)); +INSERT INTO t2 VALUES +(1,0,0), (1,0,1), (2,0,0), (2,0,1), (3,0,0), (4,0,1), +(5,0,0), (5,0,1), (6,0,0), (0,0,1), (7,0,0), (7,0,1), +(0,0,0), (0,0,1), (8,0,0), (8,0,1), (9,0,0), (9,0,1); +EXPLAIN +SELECT a, c, d, f FROM t1,t2 WHERE a=c AND b BETWEEN 4 AND 6; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 range PRIMARY,b b 5 NULL 3 Using where +1 SIMPLE t2 ref c c 5 test.t1.a 2 Using where +EXPLAIN +SELECT a, c, d, f FROM t1,t2 WHERE a=c AND b BETWEEN 4 AND 6 AND a > 0; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 range PRIMARY,b b 5 NULL 3 Using where +1 SIMPLE t2 ref c c 5 test.t1.a 2 Using where +DROP TABLE t1, t2; +create table t1 ( +a int unsigned not null auto_increment primary key, +b bit not null, +c bit not null +); +create table t2 ( +a int unsigned not null auto_increment primary key, +b bit not null, +c int unsigned not null, +d varchar(50) +); +insert into t1 (b,c) values (0,1), (0,1); +insert into t2 (b,c) values (0,1); +select t1.a, t1.b + 0, t1.c + 0, t2.a, t2.b + 0, t2.c, t2.d +from t1 left outer join t2 on t1.a = t2.c and t2.b <> 1 +where t1.b <> 1 order by t1.a; +a t1.b + 0 t1.c + 0 a t2.b + 0 c d +1 0 1 1 0 1 NULL +2 0 1 NULL NULL NULL NULL +drop table t1,t2; +SELECT 0.9888889889 * 1.011111411911; +0.9888889889 * 1.011111411911 +0.9998769417899202067879 +prepare stmt from 'select 1 as " a "'; +Warnings: +Warning 1466 Leading spaces are removed from name ' a ' +execute stmt; +a +1 +CREATE TABLE t1 (a int NOT NULL PRIMARY KEY, b int NOT NULL); +INSERT INTO t1 VALUES (1,1), (2,2), (3,3), (4,4); +CREATE TABLE t2 (c int NOT NULL, INDEX idx(c)); +INSERT INTO t2 VALUES +(1), (1), (1), (1), (1), (1), (1), (1), +(2), (2), (2), (2), +(3), (3), +(4); +EXPLAIN SELECT b FROM t1, t2 WHERE b=c AND a=1; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 const PRIMARY PRIMARY 4 const 1 +1 SIMPLE t2 ref idx idx 4 const 7 Using index +EXPLAIN SELECT b FROM t1, t2 WHERE b=c AND a=4; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 const PRIMARY PRIMARY 4 const 1 +1 SIMPLE t2 ref idx idx 4 const 1 Using index +DROP TABLE t1, t2; +CREATE TABLE t1 (id int NOT NULL PRIMARY KEY, a int); +INSERT INTO t1 VALUES (1,2), (2,NULL), (3,2); +CREATE TABLE t2 (b int, c INT, INDEX idx1(b)); +INSERT INTO t2 VALUES (2,1), (3,2); +CREATE TABLE t3 (d int, e int, INDEX idx1(d)); +INSERT INTO t3 VALUES (2,10), (2,20), (1,30), (2,40), (2,50); +EXPLAIN +SELECT * FROM t1 LEFT JOIN t2 ON t2.b=t1.a INNER JOIN t3 ON t3.d=t1.id +WHERE t1.id=2; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 const PRIMARY PRIMARY 4 const 1 +1 SIMPLE t2 const idx1 NULL NULL NULL 1 +1 SIMPLE t3 ref idx1 idx1 5 const 3 Using where +SELECT * FROM t1 LEFT JOIN t2 ON t2.b=t1.a INNER JOIN t3 ON t3.d=t1.id +WHERE t1.id=2; +id a b c d e +2 NULL NULL NULL 2 10 +2 NULL NULL NULL 2 20 +2 NULL NULL NULL 2 40 +2 NULL NULL NULL 2 50 +DROP TABLE t1,t2,t3; +create table t1 (c1 varchar(1), c2 int, c3 int, c4 int, c5 int, c6 int, +c7 int, c8 int, c9 int, fulltext key (`c1`)); +select distinct match (`c1`) against ('z') , c2, c3, c4,c5, c6,c7, c8 +from t1 where c9=1 order by c2, c2; +match (`c1`) against ('z') c2 c3 c4 c5 c6 c7 c8 +drop table t1; +CREATE TABLE t1 (pk varchar(10) PRIMARY KEY, fk varchar(16)); +CREATE TABLE t2 (pk varchar(16) PRIMARY KEY, fk varchar(10)); +INSERT INTO t1 VALUES +('d','dddd'), ('i','iii'), ('a','aa'), ('b','bb'), ('g','gg'), +('e','eee'), ('c','cccc'), ('h','hhh'), ('j','jjj'), ('f','fff'); +INSERT INTO t2 VALUES +('jjj', 'j'), ('cc','c'), ('ccc','c'), ('aaa', 'a'), ('jjjj','j'), +('hhh','h'), ('gg','g'), ('fff','f'), ('ee','e'), ('ffff','f'), +('bbb','b'), ('ff','f'), ('cccc','c'), ('dddd','d'), ('jj','j'), +('aaaa','a'), ('bb','b'), ('eeee','e'), ('aa','a'), ('hh','h'); +EXPLAIN SELECT t2.* +FROM t1 JOIN t2 ON t2.fk=t1.pk +WHERE t2.fk < 'c' AND t2.pk=t1.fk; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 range PRIMARY PRIMARY 12 NULL 3 Using where +1 SIMPLE t2 eq_ref PRIMARY PRIMARY 18 test.t1.fk 1 Using where +EXPLAIN SELECT t2.* +FROM t1 JOIN t2 ON t2.fk=t1.pk +WHERE t2.fk BETWEEN 'a' AND 'b' AND t2.pk=t1.fk; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 range PRIMARY PRIMARY 12 NULL 2 Using where +1 SIMPLE t2 eq_ref PRIMARY PRIMARY 18 test.t1.fk 1 Using where +EXPLAIN SELECT t2.* +FROM t1 JOIN t2 ON t2.fk=t1.pk +WHERE t2.fk IN ('a','b') AND t2.pk=t1.fk; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 range PRIMARY PRIMARY 12 NULL 2 Using where +1 SIMPLE t2 eq_ref PRIMARY PRIMARY 18 test.t1.fk 1 Using where +DROP TABLE t1,t2; +CREATE TABLE t1 (a int, b varchar(20) NOT NULL, PRIMARY KEY(a)); +CREATE TABLE t2 (a int, b varchar(20) NOT NULL, +PRIMARY KEY (a), UNIQUE KEY (b)); +INSERT INTO t1 VALUES (1,'a'),(2,'b'),(3,'c'); +INSERT INTO t2 VALUES (1,'a'),(2,'b'),(3,'c'); +EXPLAIN SELECT t1.a FROM t1 LEFT JOIN t2 ON t2.b=t1.b WHERE t1.a=3; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 const PRIMARY PRIMARY 4 const 1 +DROP TABLE t1,t2; +CREATE TABLE t1(id int PRIMARY KEY, b int, e int); +CREATE TABLE t2(i int, a int, INDEX si(i), INDEX ai(a)); +CREATE TABLE t3(a int PRIMARY KEY, c char(4), INDEX ci(c)); +INSERT INTO t1 VALUES +(1,10,19), (2,20,22), (4,41,42), (9,93,95), (7, 77,79), +(6,63,67), (5,55,58), (3,38,39), (8,81,89); +INSERT INTO t2 VALUES +(21,210), (41,410), (82,820), (83,830), (84,840), +(65,650), (51,510), (37,370), (94,940), (76,760), +(22,220), (33,330), (40,400), (95,950), (38,380), +(67,670), (88,880), (57,570), (96,960), (97,970); +INSERT INTO t3 VALUES +(210,'bb'), (950,'ii'), (400,'ab'), (500,'ee'), (220,'gg'), +(440,'gg'), (310,'eg'), (380,'ee'), (840,'bb'), (830,'ff'), +(230,'aa'), (960,'ii'), (410,'aa'), (510,'ee'), (290,'bb'), +(450,'gg'), (320,'dd'), (390,'hh'), (850,'jj'), (860,'ff'); +EXPLAIN +SELECT t3.a FROM t1,t2 FORCE INDEX (si),t3 +WHERE t1.id = 8 AND t2.i BETWEEN t1.b AND t1.e AND +t3.a=t2.a AND t3.c IN ('bb','ee'); +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 const PRIMARY PRIMARY 4 const 1 +1 SIMPLE t2 range si si 5 NULL 4 Using where +1 SIMPLE t3 eq_ref PRIMARY,ci PRIMARY 4 test.t2.a 1 Using where +EXPLAIN +SELECT t3.a FROM t1,t2,t3 +WHERE t1.id = 8 AND t2.i BETWEEN t1.b AND t1.e AND +t3.a=t2.a AND t3.c IN ('bb','ee') ; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 const PRIMARY PRIMARY 4 const 1 +1 SIMPLE t2 range si,ai si 5 NULL 4 Using where +1 SIMPLE t3 eq_ref PRIMARY,ci PRIMARY 4 test.t2.a 1 Using where +EXPLAIN +SELECT t3.a FROM t1,t2 FORCE INDEX (si),t3 +WHERE t1.id = 8 AND (t2.i=t1.b OR t2.i=t1.e) AND t3.a=t2.a AND +t3.c IN ('bb','ee'); +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 const PRIMARY PRIMARY 4 const 1 +1 SIMPLE t2 range si si 5 NULL 2 Using where +1 SIMPLE t3 eq_ref PRIMARY,ci PRIMARY 4 test.t2.a 1 Using where +EXPLAIN +SELECT t3.a FROM t1,t2,t3 +WHERE t1.id = 8 AND (t2.i=t1.b OR t2.i=t1.e) AND t3.a=t2.a AND +t3.c IN ('bb','ee'); +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 const PRIMARY PRIMARY 4 const 1 +1 SIMPLE t2 range si,ai si 5 NULL 2 Using where +1 SIMPLE t3 eq_ref PRIMARY,ci PRIMARY 4 test.t2.a 1 Using where +DROP TABLE t1,t2,t3; +CREATE TABLE t1 ( f1 int primary key, f2 int, f3 int, f4 int, f5 int, f6 int, checked_out int); +CREATE TABLE t2 ( f11 int PRIMARY KEY ); +INSERT INTO t1 VALUES (1,1,1,0,0,0,0),(2,1,1,3,8,1,0),(3,1,1,4,12,1,0); +INSERT INTO t2 VALUES (62); +SELECT * FROM t1 LEFT JOIN t2 ON f11 = t1.checked_out GROUP BY f1 ORDER BY f2, f3, f4, f5 LIMIT 0, 1; +f1 f2 f3 f4 f5 f6 checked_out f11 +1 1 1 0 0 0 0 NULL +DROP TABLE t1, t2; +DROP TABLE IF EXISTS t1; +CREATE TABLE t1(a int); +INSERT into t1 values (1), (2), (3); +SELECT * FROM t1 LIMIT 2, -1; +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '-1' at line 1 +DROP TABLE t1; +CREATE TABLE t1 ( +ID_with_null int NULL, +ID_better int NOT NULL, +INDEX idx1 (ID_with_null), +INDEX idx2 (ID_better) +); +INSERT INTO t1 VALUES (1,1), (2,1), (null,3), (null,3), (null,3), (null,3); +INSERT INTO t1 SELECT * FROM t1 WHERE ID_with_null IS NULL; +INSERT INTO t1 SELECT * FROM t1 WHERE ID_with_null IS NULL; +INSERT INTO t1 SELECT * FROM t1 WHERE ID_with_null IS NULL; +INSERT INTO t1 SELECT * FROM t1 WHERE ID_with_null IS NULL; +INSERT INTO t1 SELECT * FROM t1 WHERE ID_with_null IS NULL; +SELECT COUNT(*) FROM t1 WHERE ID_with_null IS NULL; +COUNT(*) +128 +SELECT COUNT(*) FROM t1 WHERE ID_better=1; +COUNT(*) +2 +EXPLAIN SELECT * FROM t1 WHERE ID_better=1 AND ID_with_null IS NULL; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ref idx1,idx2 idx2 4 const 1 Using where +DROP INDEX idx1 ON t1; +CREATE UNIQUE INDEX idx1 ON t1(ID_with_null); +EXPLAIN SELECT * FROM t1 WHERE ID_better=1 AND ID_with_null IS NULL; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ref idx1,idx2 idx2 4 const 1 Using where +DROP TABLE t1; +CREATE TABLE t1 ( +ID1_with_null int NULL, +ID2_with_null int NULL, +ID_better int NOT NULL, +INDEX idx1 (ID1_with_null, ID2_with_null), +INDEX idx2 (ID_better) +); +INSERT INTO t1 VALUES (1,1,1), (2,2,1), (3,null,3), (null,3,3), (null,null,3), +(3,null,3), (null,3,3), (null,null,3), (3,null,3), (null,3,3), (null,null,3); +INSERT INTO t1 SELECT * FROM t1 WHERE ID1_with_null IS NULL; +INSERT INTO t1 SELECT * FROM t1 WHERE ID2_with_null IS NULL; +INSERT INTO t1 SELECT * FROM t1 WHERE ID1_with_null IS NULL; +INSERT INTO t1 SELECT * FROM t1 WHERE ID2_with_null IS NULL; +INSERT INTO t1 SELECT * FROM t1 WHERE ID1_with_null IS NULL; +INSERT INTO t1 SELECT * FROM t1 WHERE ID2_with_null IS NULL; +SELECT COUNT(*) FROM t1 WHERE ID1_with_null IS NULL AND ID2_with_null=3; +COUNT(*) +24 +SELECT COUNT(*) FROM t1 WHERE ID1_with_null=3 AND ID2_with_null IS NULL; +COUNT(*) +24 +SELECT COUNT(*) FROM t1 WHERE ID1_with_null IS NULL AND ID2_with_null IS NULL; +COUNT(*) +192 +SELECT COUNT(*) FROM t1 WHERE ID_better=1; +COUNT(*) +2 +EXPLAIN SELECT * FROM t1 +WHERE ID_better=1 AND ID1_with_null IS NULL AND ID2_with_null=3 ; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ref idx1,idx2 idx2 4 const 1 Using where +EXPLAIN SELECT * FROM t1 +WHERE ID_better=1 AND ID1_with_null=3 AND ID2_with_null=3 IS NULL ; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ref idx1,idx2 idx2 4 const 1 Using where +EXPLAIN SELECT * FROM t1 +WHERE ID_better=1 AND ID1_with_null IS NULL AND ID2_with_null IS NULL; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ref idx1,idx2 idx2 4 const 1 Using where +DROP INDEX idx1 ON t1; +CREATE UNIQUE INDEX idx1 ON t1(ID1_with_null,ID2_with_null); +EXPLAIN SELECT * FROM t1 +WHERE ID_better=1 AND ID1_with_null IS NULL AND ID2_with_null=3 ; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ref idx1,idx2 idx2 4 const 1 Using where +EXPLAIN SELECT * FROM t1 +WHERE ID_better=1 AND ID1_with_null=3 AND ID2_with_null IS NULL ; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ref idx1,idx2 idx2 4 const 1 Using where +EXPLAIN SELECT * FROM t1 +WHERE ID_better=1 AND ID1_with_null IS NULL AND ID2_with_null IS NULL; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ref idx1,idx2 idx2 4 const 1 Using where +EXPLAIN SELECT * FROM t1 +WHERE ID_better=1 AND ID1_with_null IS NULL AND +(ID2_with_null=1 OR ID2_with_null=2); +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ref idx1,idx2 idx2 4 const 1 Using where +DROP TABLE t1; +CREATE TABLE t1 (a INT, ts TIMESTAMP, KEY ts(ts)); +INSERT INTO t1 VALUES (30,"2006-01-03 23:00:00"), (31,"2006-01-03 23:00:00"); +ANALYZE TABLE t1; +Table Op Msg_type Msg_text +test.t1 analyze status OK +CREATE TABLE t2 (a INT, dt1 DATETIME, dt2 DATETIME, PRIMARY KEY (a)); +INSERT INTO t2 VALUES (30, "2006-01-01 00:00:00", "2999-12-31 00:00:00"); +INSERT INTO t2 SELECT a+1,dt1,dt2 FROM t2; +ANALYZE TABLE t2; +Table Op Msg_type Msg_text +test.t2 analyze status OK +EXPLAIN +SELECT * FROM t1 LEFT JOIN t2 ON (t1.a=t2.a) WHERE t1.a=30 +AND t1.ts BETWEEN t2.dt1 AND t2.dt2 +AND t1.ts BETWEEN "2006-01-01" AND "2006-12-31"; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 const PRIMARY PRIMARY 4 const 1 +1 SIMPLE t1 range ts ts 4 NULL 1 Using where +Warnings: +Warning 1292 Incorrect datetime value: '2999-12-31 00:00:00' for column 'ts' at row 1 +SELECT * FROM t1 LEFT JOIN t2 ON (t1.a=t2.a) WHERE t1.a=30 +AND t1.ts BETWEEN t2.dt1 AND t2.dt2 +AND t1.ts BETWEEN "2006-01-01" AND "2006-12-31"; +a ts a dt1 dt2 +30 2006-01-03 23:00:00 30 2006-01-01 00:00:00 2999-12-31 00:00:00 +Warnings: +Warning 1292 Incorrect datetime value: '2999-12-31 00:00:00' for column 'ts' at row 1 +DROP TABLE t1,t2; +create table t1 (a bigint unsigned); +insert into t1 values +(if(1, 9223372036854775808, 1)), +(case when 1 then 9223372036854775808 else 1 end), +(coalesce(9223372036854775808, 1)); +select * from t1; +a +9223372036854775808 +9223372036854775808 +9223372036854775808 +drop table t1; +create table t1 select +if(1, 9223372036854775808, 1) i, +case when 1 then 9223372036854775808 else 1 end c, +coalesce(9223372036854775808, 1) co; +show create table t1; +Table Create Table +t1 CREATE TABLE `t1` ( + `i` decimal(19,0) NOT NULL DEFAULT '0', + `c` decimal(19,0) NOT NULL DEFAULT '0', + `co` decimal(19,0) NOT NULL DEFAULT '0' +) ENGINE=MyISAM DEFAULT CHARSET=latin1 +drop table t1; +select +if(1, cast(1111111111111111111 as unsigned), 1) i, +case when 1 then cast(1111111111111111111 as unsigned) else 1 end c, +coalesce(cast(1111111111111111111 as unsigned), 1) co; +i c co +1111111111111111111 1111111111111111111 1111111111111111111 +CREATE TABLE t1 (name varchar(255)); +CREATE TABLE t2 (name varchar(255), n int, KEY (name(3))); +INSERT INTO t1 VALUES ('ccc'), ('bb'), ('cc '), ('aa '), ('aa'); +INSERT INTO t2 VALUES ('bb',1), ('aa',2), ('cc ',3); +INSERT INTO t2 VALUES (concat('cc ', 0x06), 4); +INSERT INTO t2 VALUES ('cc',5), ('bb ',6), ('cc ',7); +SELECT * FROM t2; +name n +bb 1 +aa 2 +cc 3 +cc 4 +cc 5 +bb 6 +cc 7 +SELECT * FROM t2 ORDER BY name; +name n +aa 2 +bb 1 +bb 6 +cc 4 +cc 3 +cc 5 +cc 7 +SELECT name, LENGTH(name), n FROM t2 ORDER BY name; +name LENGTH(name) n +aa 2 2 +bb 2 1 +bb 3 6 +cc 4 4 +cc 5 3 +cc 2 5 +cc 3 7 +EXPLAIN SELECT name, LENGTH(name), n FROM t2 WHERE name='cc '; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ref name name 6 const 3 Using where +SELECT name, LENGTH(name), n FROM t2 WHERE name='cc '; +name LENGTH(name) n +cc 5 3 +cc 2 5 +cc 3 7 +EXPLAIN SELECT name , LENGTH(name), n FROM t2 WHERE name LIKE 'cc%'; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 range name name 6 NULL 3 Using where +SELECT name , LENGTH(name), n FROM t2 WHERE name LIKE 'cc%'; +name LENGTH(name) n +cc 5 3 +cc 4 4 +cc 2 5 +cc 3 7 +EXPLAIN SELECT name , LENGTH(name), n FROM t2 WHERE name LIKE 'cc%' ORDER BY name; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 range name name 6 NULL 3 Using where; Using filesort +SELECT name , LENGTH(name), n FROM t2 WHERE name LIKE 'cc%' ORDER BY name; +name LENGTH(name) n +cc 4 4 +cc 5 3 +cc 2 5 +cc 3 7 +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON t1.name=t2.name; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ALL NULL NULL NULL NULL 5 +1 SIMPLE t2 ref name name 6 test.t1.name 2 +SELECT * FROM t1 LEFT JOIN t2 ON t1.name=t2.name; +name name n +ccc NULL NULL +bb bb 1 +bb bb 6 +cc cc 3 +cc cc 5 +cc cc 7 +aa aa 2 +aa aa 2 +DROP TABLE t1,t2; +CREATE TABLE t1 (name text); +CREATE TABLE t2 (name text, n int, KEY (name(3))); +INSERT INTO t1 VALUES ('ccc'), ('bb'), ('cc '), ('aa '), ('aa'); +INSERT INTO t2 VALUES ('bb',1), ('aa',2), ('cc ',3); +INSERT INTO t2 VALUES (concat('cc ', 0x06), 4); +INSERT INTO t2 VALUES ('cc',5), ('bb ',6), ('cc ',7); +SELECT * FROM t2; +name n +bb 1 +aa 2 +cc 3 +cc 4 +cc 5 +bb 6 +cc 7 +SELECT * FROM t2 ORDER BY name; +name n +aa 2 +bb 1 +bb 6 +cc 4 +cc 3 +cc 5 +cc 7 +SELECT name, LENGTH(name), n FROM t2 ORDER BY name; +name LENGTH(name) n +aa 2 2 +bb 2 1 +bb 3 6 +cc 4 4 +cc 5 3 +cc 2 5 +cc 3 7 +EXPLAIN SELECT name, LENGTH(name), n FROM t2 WHERE name='cc '; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 ref name name 6 const 3 Using where +SELECT name, LENGTH(name), n FROM t2 WHERE name='cc '; +name LENGTH(name) n +cc 5 3 +cc 2 5 +cc 3 7 +EXPLAIN SELECT name , LENGTH(name), n FROM t2 WHERE name LIKE 'cc%'; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 range name name 6 NULL 3 Using where +SELECT name , LENGTH(name), n FROM t2 WHERE name LIKE 'cc%'; +name LENGTH(name) n +cc 5 3 +cc 4 4 +cc 2 5 +cc 3 7 +EXPLAIN SELECT name , LENGTH(name), n FROM t2 WHERE name LIKE 'cc%' ORDER BY name; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t2 range name name 6 NULL 3 Using where; Using filesort +SELECT name , LENGTH(name), n FROM t2 WHERE name LIKE 'cc%' ORDER BY name; +name LENGTH(name) n +cc 4 4 +cc 5 3 +cc 2 5 +cc 3 7 +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON t1.name=t2.name; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ALL NULL NULL NULL NULL 5 +1 SIMPLE t2 ref name name 6 test.t1.name 2 +SELECT * FROM t1 LEFT JOIN t2 ON t1.name=t2.name; +name name n +ccc NULL NULL +bb bb 1 +bb bb 6 +cc cc 3 +cc cc 5 +cc cc 7 +aa aa 2 +aa aa 2 +DROP TABLE t1,t2; +CREATE TABLE t1 ( +access_id int NOT NULL default '0', +name varchar(20) default NULL, +rank int NOT NULL default '0', +KEY idx (access_id) +); +CREATE TABLE t2 ( +faq_group_id int NOT NULL default '0', +faq_id int NOT NULL default '0', +access_id int default NULL, +UNIQUE KEY idx1 (faq_id), +KEY idx2 (faq_group_id,faq_id) +); +INSERT INTO t1 VALUES +(1,'Everyone',2),(2,'Help',3),(3,'Technical Support',1),(4,'Chat User',4); +INSERT INTO t2 VALUES +(261,265,1),(490,494,1); +SELECT t2.faq_id +FROM t1 INNER JOIN t2 IGNORE INDEX (idx1) +ON (t1.access_id = t2.access_id) +LEFT JOIN t2 t +ON (t.faq_group_id = t2.faq_group_id AND +find_in_set(t.access_id, '1,4') < find_in_set(t2.access_id, '1,4')) +WHERE +t2.access_id IN (1,4) AND t.access_id IS NULL AND t2.faq_id in (265); +faq_id +265 +SELECT t2.faq_id +FROM t1 INNER JOIN t2 +ON (t1.access_id = t2.access_id) +LEFT JOIN t2 t +ON (t.faq_group_id = t2.faq_group_id AND +find_in_set(t.access_id, '1,4') < find_in_set(t2.access_id, '1,4')) +WHERE +t2.access_id IN (1,4) AND t.access_id IS NULL AND t2.faq_id in (265); +faq_id +265 +DROP TABLE t1,t2; +CREATE TABLE t1 (a INT, b INT, KEY inx (b,a)); +INSERT INTO t1 VALUES (1,1), (1,2), (1,3), (1,4), (1,5), (1, 6), (1,7); +EXPLAIN SELECT COUNT(*) FROM t1 f1 INNER JOIN t1 f2 +ON ( f1.b=f2.b AND f1.a<f2.a ) +WHERE 1 AND f1.b NOT IN (100,2232,3343,51111); +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE f1 index inx inx 10 NULL 7 Using where; Using index +1 SIMPLE f2 ref inx inx 5 test.f1.b 1 Using where; Using index +DROP TABLE t1; +CREATE TABLE t1 (c1 INT, c2 INT); +INSERT INTO t1 VALUES (1,11), (2,22), (2,22); +EXPLAIN SELECT c1 FROM t1 WHERE (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT COUNT(c2)))))))))))))))))))))))))))))))) > 0; +id select_type table type possible_keys key key_len ref rows Extra +1 PRIMARY t1 ALL NULL NULL NULL NULL 3 Using where +31 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL No tables used +32 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL No tables used +EXPLAIN SELECT c1 FROM t1 WHERE (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT (SELECT COUNT(c2))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))) > 0; +ERROR HY000: Too high level of nesting for select +DROP TABLE t1; +CREATE TABLE t1 ( +c1 int(11) NOT NULL AUTO_INCREMENT, +c2 varchar(1000) DEFAULT NULL, +c3 bigint(20) DEFAULT NULL, +c4 bigint(20) DEFAULT NULL, +PRIMARY KEY (c1) +); +EXPLAIN EXTENDED +SELECT join_2.c1 +FROM +t1 AS join_0, +t1 AS join_1, +t1 AS join_2, +t1 AS join_3, +t1 AS join_4, +t1 AS join_5, +t1 AS join_6, +t1 AS join_7 +WHERE +join_0.c1=join_1.c1 AND +join_1.c1=join_2.c1 AND +join_2.c1=join_3.c1 AND +join_3.c1=join_4.c1 AND +join_4.c1=join_5.c1 AND +join_5.c1=join_6.c1 AND +join_6.c1=join_7.c1 +OR +join_0.c2 < '?' AND +join_1.c2 < '?' AND +join_2.c2 > '?' AND +join_2.c2 < '!' AND +join_3.c2 > '?' AND +join_4.c2 = '?' AND +join_5.c2 <> '?' AND +join_6.c2 <> '?' AND +join_7.c2 >= '?' AND +join_0.c1=join_1.c1 AND +join_1.c1=join_2.c1 AND +join_2.c1=join_3.c1 AND +join_3.c1=join_4.c1 AND +join_4.c1=join_5.c1 AND +join_5.c1=join_6.c1 AND +join_6.c1=join_7.c1 +GROUP BY +join_3.c1, +join_2.c1, +join_7.c1, +join_1.c1, +join_0.c1; +id select_type table type possible_keys key key_len ref rows filtered Extra +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables +Warnings: +Note 1003 select '0' AS `c1` from `test`.`t1` `join_0` join `test`.`t1` `join_1` join `test`.`t1` `join_2` join `test`.`t1` `join_3` join `test`.`t1` `join_4` join `test`.`t1` `join_5` join `test`.`t1` `join_6` join `test`.`t1` `join_7` where 0 group by '0','0','0','0','0' +SHOW WARNINGS; +Level Code Message +Note 1003 select '0' AS `c1` from `test`.`t1` `join_0` join `test`.`t1` `join_1` join `test`.`t1` `join_2` join `test`.`t1` `join_3` join `test`.`t1` `join_4` join `test`.`t1` `join_5` join `test`.`t1` `join_6` join `test`.`t1` `join_7` where 0 group by '0','0','0','0','0' +DROP TABLE t1; +SELECT 1 AS ` `; + +1 +Warnings: +Warning 1474 Name ' ' has become '' +SELECT 1 AS ` `; + +1 +Warnings: +Warning 1474 Name ' ' has become '' +SELECT 1 AS ` x`; +x +1 +Warnings: +Warning 1466 Leading spaces are removed from name ' x' +CREATE VIEW v1 AS SELECT 1 AS ``; +ERROR 42000: Incorrect column name '' +CREATE VIEW v1 AS SELECT 1 AS ` `; +ERROR 42000: Incorrect column name ' ' +CREATE VIEW v1 AS SELECT 1 AS ` `; +ERROR 42000: Incorrect column name ' ' +CREATE VIEW v1 AS SELECT (SELECT 1 AS ` `); +ERROR 42000: Incorrect column name ' ' +CREATE VIEW v1 AS SELECT 1 AS ` x`; +Warnings: +Warning 1466 Leading spaces are removed from name ' x' +SELECT `x` FROM v1; +x +1 +ALTER VIEW v1 AS SELECT 1 AS ` `; +ERROR 42000: Incorrect column name ' ' +DROP VIEW v1; +select str_to_date('2007-10-09','%Y-%m-%d') between '2007/10/01 00:00:00 GMT' + and '2007/10/20 00:00:00 GMT'; +str_to_date('2007-10-09','%Y-%m-%d') between '2007/10/01 00:00:00 GMT' + and '2007/10/20 00:00:00 GMT' +1 +Warnings: +Warning 1292 Truncated incorrect datetime value: '2007/10/01 00:00:00 GMT' +Warning 1292 Truncated incorrect datetime value: '2007/10/20 00:00:00 GMT' +select str_to_date('2007-10-09','%Y-%m-%d') > '2007/10/01 00:00:00 GMT-6'; +str_to_date('2007-10-09','%Y-%m-%d') > '2007/10/01 00:00:00 GMT-6' +1 +Warnings: +Warning 1292 Truncated incorrect date value: '2007/10/01 00:00:00 GMT-6' +select str_to_date('2007-10-09','%Y-%m-%d') <= '2007/10/2000:00:00 GMT-6'; +str_to_date('2007-10-09','%Y-%m-%d') <= '2007/10/2000:00:00 GMT-6' +1 +Warnings: +Warning 1292 Truncated incorrect date value: '2007/10/2000:00:00 GMT-6' +select str_to_date('2007-10-01','%Y-%m-%d') = '2007-10-1 00:00:00 GMT-6'; +str_to_date('2007-10-01','%Y-%m-%d') = '2007-10-1 00:00:00 GMT-6' +1 +Warnings: +Warning 1292 Truncated incorrect date value: '2007-10-1 00:00:00 GMT-6' +select str_to_date('2007-10-01','%Y-%m-%d') = '2007-10-01 x00:00:00 GMT-6'; +str_to_date('2007-10-01','%Y-%m-%d') = '2007-10-01 x00:00:00 GMT-6' +1 +Warnings: +Warning 1292 Truncated incorrect date value: '2007-10-01 x00:00:00 GMT-6' +select str_to_date('2007-10-01','%Y-%m-%d %H:%i:%s') = '2007-10-01 00:00:00 GMT-6'; +str_to_date('2007-10-01','%Y-%m-%d %H:%i:%s') = '2007-10-01 00:00:00 GMT-6' +1 +Warnings: +Warning 1292 Truncated incorrect datetime value: '2007-10-01 00:00:00 GMT-6' +select str_to_date('2007-10-01','%Y-%m-%d %H:%i:%s') = '2007-10-01 00:x00:00 GMT-6'; +str_to_date('2007-10-01','%Y-%m-%d %H:%i:%s') = '2007-10-01 00:x00:00 GMT-6' +1 +Warnings: +Warning 1292 Truncated incorrect datetime value: '2007-10-01 00:x00:00 GMT-6' +select str_to_date('2007-10-01','%Y-%m-%d %H:%i:%s') = '2007-10-01 x12:34:56 GMT-6'; +str_to_date('2007-10-01','%Y-%m-%d %H:%i:%s') = '2007-10-01 x12:34:56 GMT-6' +1 +Warnings: +Warning 1292 Truncated incorrect datetime value: '2007-10-01 x12:34:56 GMT-6' +select str_to_date('2007-10-01 12:34:00','%Y-%m-%d %H:%i:%s') = '2007-10-01 12:34x:56 GMT-6'; +str_to_date('2007-10-01 12:34:00','%Y-%m-%d %H:%i:%s') = '2007-10-01 12:34x:56 GMT-6' +1 +Warnings: +Warning 1292 Truncated incorrect datetime value: '2007-10-01 12:34x:56 GMT-6' +select str_to_date('2007-10-01 12:34:56','%Y-%m-%d %H:%i:%s') = '2007-10-01 12:34x:56 GMT-6'; +str_to_date('2007-10-01 12:34:56','%Y-%m-%d %H:%i:%s') = '2007-10-01 12:34x:56 GMT-6' +0 +Warnings: +Warning 1292 Truncated incorrect datetime value: '2007-10-01 12:34x:56 GMT-6' +select str_to_date('2007-10-01 12:34:56','%Y-%m-%d %H:%i:%s') = '2007-10-01 12:34:56'; +str_to_date('2007-10-01 12:34:56','%Y-%m-%d %H:%i:%s') = '2007-10-01 12:34:56' +1 +select str_to_date('2007-10-01','%Y-%m-%d') = '2007-10-01 12:00:00'; +str_to_date('2007-10-01','%Y-%m-%d') = '2007-10-01 12:00:00' +0 +select str_to_date('2007-10-01 12','%Y-%m-%d %H') = '2007-10-01 12:00:00'; +str_to_date('2007-10-01 12','%Y-%m-%d %H') = '2007-10-01 12:00:00' +1 +select str_to_date('2007-10-01 12:34','%Y-%m-%d %H') = '2007-10-01 12:00:00'; +str_to_date('2007-10-01 12:34','%Y-%m-%d %H') = '2007-10-01 12:00:00' +1 +Warnings: +Warning 1292 Truncated incorrect datetime value: '2007-10-01 12:34' +select str_to_date('2007-02-30 12:34','%Y-%m-%d %H:%i') = '2007-02-30 12:34'; +str_to_date('2007-02-30 12:34','%Y-%m-%d %H:%i') = '2007-02-30 12:34' +1 +select str_to_date('2007-10-00 12:34','%Y-%m-%d %H:%i') = '2007-10-00 12:34'; +str_to_date('2007-10-00 12:34','%Y-%m-%d %H:%i') = '2007-10-00 12:34' +1 +select str_to_date('2007-10-00','%Y-%m-%d') between '2007/09/01 00:00:00' + and '2007/10/20 00:00:00'; +str_to_date('2007-10-00','%Y-%m-%d') between '2007/09/01 00:00:00' + and '2007/10/20 00:00:00' +1 +set SQL_MODE=TRADITIONAL; +select str_to_date('2007-10-00 12:34','%Y-%m-%d %H:%i') = '2007-10-00 12:34'; +str_to_date('2007-10-00 12:34','%Y-%m-%d %H:%i') = '2007-10-00 12:34' +0 +Warnings: +Warning 1292 Truncated incorrect datetime value: '2007-10-00 12:34' +select str_to_date('2007-10-01 12:34','%Y-%m-%d %H:%i') = '2007-10-00 12:34'; +str_to_date('2007-10-01 12:34','%Y-%m-%d %H:%i') = '2007-10-00 12:34' +0 +Warnings: +Warning 1292 Truncated incorrect datetime value: '2007-10-00 12:34' +select str_to_date('2007-10-00 12:34','%Y-%m-%d %H:%i') = '2007-10-01 12:34'; +str_to_date('2007-10-00 12:34','%Y-%m-%d %H:%i') = '2007-10-01 12:34' +0 +Warnings: +Warning 1292 Truncated incorrect datetime value: '2007-10-00 12:34:00' +select str_to_date('2007-10-00','%Y-%m-%d') between '2007/09/01' + and '2007/10/20'; +str_to_date('2007-10-00','%Y-%m-%d') between '2007/09/01' + and '2007/10/20' +0 +Warnings: +Warning 1292 Incorrect datetime value: '2007-10-00' for column '2007/09/01' at row 1 +Warning 1292 Incorrect datetime value: '2007-10-00' for column '2007/10/20' at row 1 +set SQL_MODE=DEFAULT; +select str_to_date('2007-10-00','%Y-%m-%d') between '' and '2007/10/20'; +str_to_date('2007-10-00','%Y-%m-%d') between '' and '2007/10/20' +1 +Warnings: +Warning 1292 Truncated incorrect datetime value: '' +select str_to_date('','%Y-%m-%d') between '2007/10/01' and '2007/10/20'; +str_to_date('','%Y-%m-%d') between '2007/10/01' and '2007/10/20' +0 +select str_to_date('','%Y-%m-%d %H:%i') = '2007-10-01 12:34'; +str_to_date('','%Y-%m-%d %H:%i') = '2007-10-01 12:34' +0 +select str_to_date(NULL,'%Y-%m-%d %H:%i') = '2007-10-01 12:34'; +str_to_date(NULL,'%Y-%m-%d %H:%i') = '2007-10-01 12:34' +NULL +select str_to_date('2007-10-00 12:34','%Y-%m-%d %H:%i') = ''; +str_to_date('2007-10-00 12:34','%Y-%m-%d %H:%i') = '' +0 +Warnings: +Warning 1292 Truncated incorrect datetime value: '' +select str_to_date('1','%Y-%m-%d') = '1'; +str_to_date('1','%Y-%m-%d') = '1' +0 +Warnings: +Warning 1292 Truncated incorrect date value: '1' +select str_to_date('1','%Y-%m-%d') = '1'; +str_to_date('1','%Y-%m-%d') = '1' +0 +Warnings: +Warning 1292 Truncated incorrect date value: '1' +select str_to_date('','%Y-%m-%d') = ''; +str_to_date('','%Y-%m-%d') = '' +0 +Warnings: +Warning 1292 Truncated incorrect date value: '' +select str_to_date('1000-01-01','%Y-%m-%d') between '0000-00-00' and NULL; +str_to_date('1000-01-01','%Y-%m-%d') between '0000-00-00' and NULL +0 +select str_to_date('1000-01-01','%Y-%m-%d') between NULL and '2000-00-00'; +str_to_date('1000-01-01','%Y-%m-%d') between NULL and '2000-00-00' +0 +select str_to_date('1000-01-01','%Y-%m-%d') between NULL and NULL; +str_to_date('1000-01-01','%Y-%m-%d') between NULL and NULL +0 +CREATE TABLE t1 (c11 INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY); +CREATE TABLE t2 (c21 INT UNSIGNED NOT NULL, +c22 INT DEFAULT NULL, +KEY(c21, c22)); +CREATE TABLE t3 (c31 INT UNSIGNED NOT NULL DEFAULT 0, +c32 INT DEFAULT NULL, +c33 INT NOT NULL, +c34 INT UNSIGNED DEFAULT 0, +KEY (c33, c34, c32)); +INSERT INTO t1 values (),(),(),(),(); +INSERT INTO t2 SELECT a.c11, b.c11 FROM t1 a, t1 b; +INSERT INTO t3 VALUES (1, 1, 1, 0), +(2, 2, 0, 0), +(3, 3, 1, 0), +(4, 4, 0, 0), +(5, 5, 1, 0); +SELECT c32 FROM t1, t2, t3 WHERE t1.c11 IN (1, 3, 5) AND +t3.c31 = t1.c11 AND t2.c21 = t1.c11 AND +t3.c33 = 1 AND t2.c22 in (1, 3) +ORDER BY c32; +c32 +1 +1 +3 +3 +5 +5 +SELECT c32 FROM t1, t2, t3 WHERE t1.c11 IN (1, 3, 5) AND +t3.c31 = t1.c11 AND t2.c21 = t1.c11 AND +t3.c33 = 1 AND t2.c22 in (1, 3) +ORDER BY c32 DESC; +c32 +5 +5 +3 +3 +1 +1 +DROP TABLE t1, t2, t3; + +# +# Bug#30736: Row Size Too Large Error Creating a Table and +# Inserting Data. +# +DROP TABLE IF EXISTS t1; +DROP TABLE IF EXISTS t2; + +CREATE TABLE t1( +c1 DECIMAL(10, 2), +c2 FLOAT); + +INSERT INTO t1 VALUES (0, 1), (2, 3), (4, 5); + +CREATE TABLE t2( +c3 DECIMAL(10, 2)) +SELECT +c1 * c2 AS c3 +FROM t1; + +SELECT * FROM t1; +c1 c2 +0.00 1 +2.00 3 +4.00 5 + +SELECT * FROM t2; +c3 +0.00 +6.00 +20.00 + +DROP TABLE t1; +DROP TABLE t2; + +CREATE TABLE t1 (c1 BIGINT NOT NULL); +INSERT INTO t1 (c1) VALUES (1); +SELECT * FROM t1 WHERE c1 > NULL + 1; +c1 +DROP TABLE t1; + +CREATE TABLE t1 (a VARCHAR(10) NOT NULL PRIMARY KEY); +INSERT INTO t1 (a) VALUES ('foo0'), ('bar0'), ('baz0'); +SELECT * FROM t1 WHERE a IN (CONCAT('foo', 0), 'bar'); +a +foo0 +DROP TABLE t1; +CREATE TABLE t1 (a INT, b INT); +CREATE TABLE t2 (a INT, c INT, KEY(a)); +INSERT INTO t1 VALUES (1, 1), (2, 2); +INSERT INTO t2 VALUES (1, 1), (1, 2), (1, 3), (1, 4), (1, 5), +(2, 1), (2, 2), (2, 3), (2, 4), (2, 5), +(3, 1), (3, 2), (3, 3), (3, 4), (3, 5), +(4, 1), (4, 2), (4, 3), (4, 4), (4, 5); +FLUSH STATUS; +SELECT DISTINCT b FROM t1 LEFT JOIN t2 USING(a) WHERE c <= 3; +b +1 +2 +SHOW STATUS LIKE 'Handler_read%'; +Variable_name Value +Handler_read_first 0 +Handler_read_key 2 +Handler_read_next 0 +Handler_read_prev 0 +Handler_read_rnd 0 +Handler_read_rnd_next 6 +DROP TABLE t1, t2; +CREATE TABLE t1 (f1 bigint(20) NOT NULL default '0', +f2 int(11) NOT NULL default '0', +f3 bigint(20) NOT NULL default '0', +f4 varchar(255) NOT NULL default '', +PRIMARY KEY (f1), +KEY key1 (f4), +KEY key2 (f2)); +CREATE TABLE t2 (f1 int(11) NOT NULL default '0', +f2 enum('A1','A2','A3') NOT NULL default 'A1', +f3 int(11) NOT NULL default '0', +PRIMARY KEY (f1), +KEY key1 (f3)); +CREATE TABLE t3 (f1 bigint(20) NOT NULL default '0', +f2 datetime NOT NULL default '1980-01-01 00:00:00', +PRIMARY KEY (f1)); +insert into t1 values (1, 1, 1, 'abc'); +insert into t1 values (2, 1, 2, 'def'); +insert into t1 values (3, 1, 2, 'def'); +insert into t2 values (1, 'A1', 1); +insert into t3 values (1, '1980-01-01'); +SELECT a.f3, cr.f4, count(*) count +FROM t2 a +STRAIGHT_JOIN t1 cr ON cr.f2 = a.f1 +LEFT JOIN +(t1 cr2 +JOIN t3 ae2 ON cr2.f3 = ae2.f1 +) ON a.f1 = cr2.f2 AND ae2.f2 < now() - INTERVAL 7 DAY AND +cr.f4 = cr2.f4 +GROUP BY a.f3, cr.f4; +f3 f4 count +1 abc 1 +1 def 2 +drop table t1, t2, t3; +CREATE TABLE t1 (a INT KEY, b INT); +INSERT INTO t1 VALUES (1,1), (2,2), (3,3), (4,4); +EXPLAIN EXTENDED SELECT a, b FROM t1 WHERE a > 1 AND a = b LIMIT 2; +id select_type table type possible_keys key key_len ref rows filtered Extra +1 SIMPLE t1 range PRIMARY PRIMARY 4 NULL 3 100.00 Using where +Warnings: +Note 1003 select `test`.`t1`.`a` AS `a`,`test`.`t1`.`b` AS `b` from `test`.`t1` where ((`test`.`t1`.`b` = `test`.`t1`.`a`) and (`test`.`t1`.`a` > 1)) limit 2 +EXPLAIN EXTENDED SELECT a, b FROM t1 WHERE a > 1 AND b = a LIMIT 2; +id select_type table type possible_keys key key_len ref rows filtered Extra +1 SIMPLE t1 range PRIMARY PRIMARY 4 NULL 3 100.00 Using where +Warnings: +Note 1003 select `test`.`t1`.`a` AS `a`,`test`.`t1`.`b` AS `b` from `test`.`t1` where ((`test`.`t1`.`a` = `test`.`t1`.`b`) and (`test`.`t1`.`a` > 1)) limit 2 +DROP TABLE t1; +# +# Bug#47019: Assertion failed: 0, file .\rt_mbr.c, line 138 when +# forcing a spatial index +# +CREATE TABLE t1(a LINESTRING NOT NULL, SPATIAL KEY(a)); +INSERT INTO t1 VALUES +(GEOMFROMTEXT('LINESTRING(-1 -1, 1 -1, -1 -1, -1 1, 1 1)')), +(GEOMFROMTEXT('LINESTRING(-1 -1, 1 -1, -1 -1, -1 1, 1 1)')); +EXPLAIN SELECT 1 FROM t1 NATURAL LEFT JOIN t1 AS t2; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 +1 SIMPLE t2 ALL a NULL NULL NULL 2 +SELECT 1 FROM t1 NATURAL LEFT JOIN t1 AS t2; +1 +1 +1 +1 +1 +EXPLAIN SELECT 1 FROM t1 NATURAL LEFT JOIN t1 AS t2 FORCE INDEX(a); +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 +1 SIMPLE t2 ALL a NULL NULL NULL 2 +SELECT 1 FROM t1 NATURAL LEFT JOIN t1 AS t2 FORCE INDEX(a); +1 +1 +1 +1 +1 +DROP TABLE t1; +# +# Bug #48291 : crash with row() operator,select into @var, and +# subquery returning multiple rows +# +CREATE TABLE t1(a INT); +INSERT INTO t1 VALUES (2),(3); +# Should not crash +SELECT 1 FROM t1 WHERE a <> 1 AND NOT +ROW(a,a) <=> ROW((SELECT 1 FROM t1 WHERE 1=2),(SELECT 1 FROM t1)) +INTO @var0; +ERROR 21000: Subquery returns more than 1 row +DROP TABLE t1; +# +# Bug #48458: simple query tries to allocate enormous amount of +# memory +# +CREATE TABLE t1(a INT NOT NULL, b YEAR); +INSERT INTO t1 VALUES (); +Warnings: +Warning 1364 Field 'a' doesn't have a default value +CREATE TABLE t2(c INT); +# Should not err out because of out-of-memory +SELECT 1 FROM t2 JOIN t1 ON 1=1 +WHERE a != '1' AND NOT a >= b OR NOT ROW(b,a )<> ROW(a,a); +1 +DROP TABLE t1,t2; +End of 5.0 tests +create table t1(a INT, KEY (a)); +INSERT INTO t1 VALUES (1),(2),(3),(4),(5); +SELECT a FROM t1 ORDER BY a LIMIT 2; +a +1 +2 +SELECT a FROM t1 ORDER BY a LIMIT 2,4294967296; +a +3 +4 +5 +SELECT a FROM t1 ORDER BY a LIMIT 2,4294967297; +a +3 +4 +5 +DROP TABLE t1; +CREATE TABLE A (date_key date); +CREATE TABLE C ( +pk int, +int_nokey int, +int_key int, +date_key date NOT NULL, +date_nokey date, +varchar_key varchar(1) +); +INSERT INTO C VALUES +(1,1,1,'0000-00-00',NULL,NULL), +(1,1,1,'0000-00-00',NULL,NULL); +SELECT 1 FROM C WHERE pk > ANY (SELECT 1 FROM C); +1 +SELECT COUNT(DISTINCT 1) FROM C +WHERE date_key = (SELECT 1 FROM A WHERE C.date_key IS NULL) GROUP BY pk; +COUNT(DISTINCT 1) +SELECT date_nokey FROM C +WHERE int_key IN (SELECT 1 FROM A) +HAVING date_nokey = '10:41:7' +ORDER BY date_key; +date_nokey +Warnings: +Warning 1292 Incorrect date value: '10:41:7' for column 'date_nokey' at row 1 +DROP TABLE A,C; +CREATE TABLE t1 (a INT NOT NULL, b INT); +INSERT INTO t1 VALUES (1, 1); +EXPLAIN EXTENDED SELECT * FROM t1 WHERE (a=a AND a=a) OR b > 2; +id select_type table type possible_keys key key_len ref rows filtered Extra +1 SIMPLE t1 system NULL NULL NULL NULL 1 100.00 +Warnings: +Note 1003 select '1' AS `a`,'1' AS `b` from `test`.`t1` where 1 +SELECT * FROM t1 WHERE (a=a AND a=a) OR b > 2; +a b +1 1 +DROP TABLE t1; +CREATE TABLE t1 (a INT NOT NULL, b INT NOT NULL, c INT NOT NULL); +EXPLAIN EXTENDED SELECT * FROM t1 WHERE (a=a AND b=b AND c=c) OR b > 20; +id select_type table type possible_keys key key_len ref rows filtered Extra +1 SIMPLE t1 system NULL NULL NULL NULL 0 0.00 const row not found +Warnings: +Note 1003 select '0' AS `a`,'0' AS `b`,'0' AS `c` from `test`.`t1` where 1 +EXPLAIN EXTENDED SELECT * FROM t1 WHERE (a=a AND a=a AND b=b) OR b > 20; +id select_type table type possible_keys key key_len ref rows filtered Extra +1 SIMPLE t1 system NULL NULL NULL NULL 0 0.00 const row not found +Warnings: +Note 1003 select '0' AS `a`,'0' AS `b`,'0' AS `c` from `test`.`t1` where 1 +EXPLAIN EXTENDED SELECT * FROM t1 WHERE (a=a AND b=b AND a=a) OR b > 20; +id select_type table type possible_keys key key_len ref rows filtered Extra +1 SIMPLE t1 system NULL NULL NULL NULL 0 0.00 const row not found +Warnings: +Note 1003 select '0' AS `a`,'0' AS `b`,'0' AS `c` from `test`.`t1` where 1 +DROP TABLE t1; +# +# Bug#45266: Uninitialized variable lead to an empty result. +# +drop table if exists A,AA,B,BB; +CREATE TABLE `A` ( +`pk` int(11) NOT NULL AUTO_INCREMENT, +`date_key` date NOT NULL, +`date_nokey` date NOT NULL, +`datetime_key` datetime NOT NULL, +`int_nokey` int(11) NOT NULL, +`time_key` time NOT NULL, +`time_nokey` time NOT NULL, +PRIMARY KEY (`pk`), +KEY `date_key` (`date_key`), +KEY `time_key` (`time_key`), +KEY `datetime_key` (`datetime_key`) +); +CREATE TABLE `AA` ( +`pk` int(11) NOT NULL AUTO_INCREMENT, +`int_nokey` int(11) NOT NULL, +`time_key` time NOT NULL, +KEY `time_key` (`time_key`), +PRIMARY KEY (`pk`) +); +CREATE TABLE `B` ( +`date_nokey` date NOT NULL, +`date_key` date NOT NULL, +`time_key` time NOT NULL, +`datetime_nokey` datetime NOT NULL, +`varchar_key` varchar(1) NOT NULL, +KEY `date_key` (`date_key`), +KEY `time_key` (`time_key`), +KEY `varchar_key` (`varchar_key`) +); +INSERT INTO `B` VALUES ('2003-07-28','2003-07-28','15:13:38','0000-00-00 00:00:00','f'),('0000-00-00','0000-00-00','00:05:48','2004-07-02 14:34:13','x'); +CREATE TABLE `BB` ( +`pk` int(11) NOT NULL AUTO_INCREMENT, +`int_nokey` int(11) NOT NULL, +`date_key` date NOT NULL, +`varchar_nokey` varchar(1) NOT NULL, +`date_nokey` date NOT NULL, +PRIMARY KEY (`pk`), +KEY `date_key` (`date_key`) +); +INSERT INTO `BB` VALUES (10,8,'0000-00-00','i','0000-00-00'),(11,0,'2005-08-18','','2005-08-18'); +SELECT table1 . `pk` AS field1 +FROM +(BB AS table1 INNER JOIN +(AA AS table2 STRAIGHT_JOIN A AS table3 +ON ( table3 . `date_key` = table2 . `pk` )) +ON ( table3 . `datetime_key` = table2 . `int_nokey` )) +WHERE ( table3 . `date_key` <= 4 AND table2 . `pk` = table1 . `varchar_nokey`) +GROUP BY field1 ; +field1 +SELECT table3 .`date_key` field1 +FROM +B table1 LEFT JOIN B table3 JOIN +(BB table6 JOIN A table7 ON table6 .`varchar_nokey`) +ON table6 .`int_nokey` ON table6 .`date_key` + WHERE NOT ( table1 .`varchar_key` AND table7 .`pk`) GROUP BY field1; +field1 +NULL +SELECT table4 . `time_nokey` AS field1 FROM +(AA AS table1 CROSS JOIN +(AA AS table2 STRAIGHT_JOIN +(B AS table3 STRAIGHT_JOIN A AS table4 +ON ( table4 . `date_key` = table3 . `time_key` )) +ON ( table4 . `pk` = table3 . `date_nokey` )) +ON ( table4 . `time_key` = table3 . `datetime_nokey` )) +WHERE ( table4 . `time_key` < table1 . `time_key` AND +table1 . `int_nokey` != 'f') +GROUP BY field1 ORDER BY field1 , field1; +field1 +SELECT table1 .`time_key` field2 FROM B table1 LEFT JOIN BB JOIN A table5 ON table5 .`date_nokey` ON table5 .`int_nokey` GROUP BY field2; +field2 +00:05:48 +15:13:38 +drop table A,AA,B,BB; +#end of test for bug#45266 +# +# BUG#48052: Valgrind warning - uninitialized value in init_read_record() +# +CREATE TABLE t1 ( +pk int(11) NOT NULL, +i int(11) DEFAULT NULL, +v varchar(1) DEFAULT NULL, +PRIMARY KEY (pk) +); +INSERT INTO t1 VALUES (2,7,'m'); +INSERT INTO t1 VALUES (3,9,'m'); +SELECT v +FROM t1 +WHERE NOT pk > 0 +HAVING v <= 't' +ORDER BY pk; +v +DROP TABLE t1; +# +# Bug#49489 Uninitialized cache led to a wrong result. +# +CREATE TABLE t1(c1 DOUBLE(5,4)); +INSERT INTO t1 VALUES (9.1234); +SELECT * FROM t1 WHERE c1 < 9.12345; +c1 +9.1234 +DROP TABLE t1; +# End of test for bug#49489. +# +# Bug #49517: Inconsistent behavior while using +# NULLable BIGINT and INT columns in comparison +# +CREATE TABLE t1(a BIGINT UNSIGNED NOT NULL, b BIGINT NULL, c INT NULL); +INSERT INTO t1 VALUES(105, NULL, NULL); +SELECT * FROM t1 WHERE b < 102; +a b c +SELECT * FROM t1 WHERE c < 102; +a b c +SELECT * FROM t1 WHERE 102 < b; +a b c +SELECT * FROM t1 WHERE 102 < c; +a b c +DROP TABLE t1; +End of 5.1 tests === modified file 'mysql-test/t/key_cache.test' --- a/mysql-test/t/key_cache.test 2008-03-27 16:43:17 +0000 +++ b/mysql-test/t/key_cache.test 2010-02-16 16:41:11 +0000 @@ -1,11 +1,13 @@ # -# Test of multiple key caches +# Test of multiple key caches, simple an partitioned # --disable_warnings drop table if exists t1, t2, t3; --enable_warnings -SET @save_key_buffer=@@key_buffer_size; +SET @save_key_buffer_size=@@key_buffer_size; +SET @save_key_cache_block_size=@@key_cache_block_size; +SET @save_key_cache_partitions=@@key_cache_partitions; SELECT @@key_buffer_size, @@small.key_buffer_size; @@ -33,7 +35,7 @@ SELECT @@`default`.key_buffer_size; SELECT @@small.key_buffer_size; SELECT @@medium.key_buffer_size; -SET @@global.key_buffer_size=@save_key_buffer; +SET @@global.key_buffer_size=@save_key_buffer_size; # # Errors @@ -247,3 +249,263 @@ SET GLOBAL key_cache_block_size= @bug284 DROP TABLE t1; # End of 4.1 tests + +# +# Test cases for partitioned key caches +# + +# Test usage of the KEY_CACHE table from information schema +# for a simple key cache + +set global key_buffer_size=@save_key_buffer_size; +set global key_cache_block_size=@save_key_cache_block_size; +select @@key_buffer_size; +select @@key_cache_block_size; +select @@key_cache_partitions; + +create table t1 ( + p int not null auto_increment primary key, + a char(10)); +create table t2 ( + p int not null auto_increment primary key, + i int, a char(10), key k1(i), key k2(a)); + +select @@key_cache_partitions; +--replace_column 7 # +select * from information_schema.key_caches; + +insert into t1 values (1, 'qqqq'), (2, 'yyyy'); +insert into t2 values (1, 1, 'qqqq'), (2, 1, 'pppp'), + (3, 1, 'yyyy'), (4, 3, 'zzzz'); +select * from t1; +select * from t2; +update t1 set p=3 where p=1; +update t2 set i=2 where i=1; + +--replace_result 1808 KEY_BLOCKS_UNUSED 1670 KEY_BLOCKS_UNUSED +show status like 'key_%'; +--replace_column 7 # +select * from information_schema.key_caches; + +delete from t2 where a='zzzz'; +--replace_column 7 # +select * from information_schema.key_caches; + +delete from t1; +delete from t2; +--replace_column 7 # +select * from information_schema.key_caches; + +# For the key cache with 2 partitions execute the same sequence of +# statements as for the simple cache above. +# The statistical information on the number of i/o requests and +# the number of is expected to be the same. + +set global key_cache_partitions=2; +select @@key_cache_partitions; +--replace_column 7 # +select * from information_schema.key_caches; + +insert into t1 values (1, 'qqqq'), (2, 'yyyy'); +insert into t2 values (1, 1, 'qqqq'), (2, 1, 'pppp'), + (3, 1, 'yyyy'), (4, 3, 'zzzz'); +select * from t1; +select * from t2; +update t1 set p=3 where p=1; +update t2 set i=2 where i=1; + +--replace_result 1808 KEY_BLOCKS_UNUSED 1670 KEY_BLOCKS_UNUSED +show status like 'key_%'; +--replace_column 7 # +select * from information_schema.key_caches; + +delete from t1; +delete from t2; +--replace_column 7 # +select * from information_schema.key_caches; + +# Check that we can work with one partition with the same results + +set global key_cache_partitions=1; +select @@key_cache_partitions; +--replace_column 7 # +select * from information_schema.key_caches; + +insert into t1 values (1, 'qqqq'), (2, 'yyyy'); +insert into t2 values (1, 1, 'qqqq'), (2, 1, 'pppp'), + (3, 1, 'yyyy'), (4, 3, 'zzzz'); +select * from t1; +select * from t2; +update t1 set p=3 where p=1; +update t2 set i=2 where i=1; + +--replace_result 1808 KEY_BLOCKS_UNUSED 1670 KEY_BLOCKS_UNUSED +show status like 'key_%'; +--replace_column 7 # +select * from information_schema.key_caches; + +delete from t1; +delete from t2; +--replace_column 7 # +select * from information_schema.key_caches; + +flush tables; flush status; +--replace_column 7 # +select * from information_schema.key_caches; + +# Switch back to 2 partitions + +set global key_buffer_size=32*1024; +select @@key_buffer_size; +set global key_cache_partitions=2; +select @@key_cache_partitions; +--replace_column 7 # +select * from information_schema.key_caches; + +insert into t1 values (1, 'qqqq'), (2, 'yyyy'); +insert into t2 values (1, 1, 'qqqq'), (2, 1, 'pppp'), + (3, 1, 'yyyy'), (4, 3, 'zzzz'); +select * from t1; +select * from t2; +update t1 set p=3 where p=1; +update t2 set i=2 where i=1; + +--replace_column 7 # +select * from information_schema.key_caches; + +# Add more rows to tables t1 and t2 + +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; +insert into t1(a) select a from t1; + +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; +insert into t2(i,a) select i,a from t2; + +--replace_column 6 # 7 # 10 # +select * from information_schema.key_caches; + +select * from t1 where p between 1010 and 1020 ; +select * from t2 where p between 1010 and 1020 ; +--replace_column 6 # 7 # 10 # +select * from information_schema.key_caches; + +flush tables; flush status; +update t1 set a='zzzz' where a='qqqq'; +update t2 set i=1 where i=2; +--replace_column 6 # 7 # +select * from information_schema.key_caches; + +# Now test how we can work with 7 partitions + +set global keycache1.key_buffer_size=256*1024; +select @@keycache1.key_buffer_size; +set global keycache1.key_cache_partitions=7; +select @@keycache1.key_cache_partitions; + +--replace_column 6 # 7 # +select * from information_schema.key_caches; +--replace_column 7 # +select * from information_schema.key_caches where key_cache_name like "key%"; + +cache index t1 key (`primary`) in keycache1; + +explain select p from t1 where p between 1010 and 1020; +select p from t1 where p between 1010 and 1020; +explain select i from t2 where p between 1010 and 1020; +select i from t2 where p between 1010 and 1020; +explain select count(*) from t1, t2 where t1.p = t2.i; +select count(*) from t1, t2 where t1.p = t2.i; + +--replace_column 6 # 7 # +select * from information_schema.key_caches; +--replace_column 7 # +select * from information_schema.key_caches where key_cache_name like "key%"; + +cache index t2 in keycache1; +update t2 set p=p+3000, i=2 where a='qqqq'; +--replace_column 7 # +select * from information_schema.key_caches where key_cache_name like "key%"; + +set global keycache2.key_buffer_size=1024*1024; +cache index t2 in keycache2; +insert into t2 values (2000, 3, 'yyyy'); +--replace_column 7 # +select * from information_schema.key_caches where key_cache_name like "keycache2"; +--replace_column 7 # +select * from information_schema.key_caches where key_cache_name like "key%"; + +cache index t2 in keycache1; +update t2 set p=p+5000 where a='zzzz'; +select * from t2 where p between 1010 and 1020; +explain select p from t2 where p between 1010 and 1020; +select p from t2 where p between 1010 and 1020; +explain select i from t2 where a='yyyy' and i=3; +select i from t2 where a='yyyy' and i=3; +explain select a from t2 where a='yyyy' and i=3; +select a from t2 where a='yyyy' and i=3 ; +--replace_column 6 # 7 # +select * from information_schema.key_caches; + +set global keycache1.key_cache_block_size=2*1024; +insert into t2 values (7000, 3, 'yyyy'); +--replace_column 6 # 7 # +select * from information_schema.key_caches; + +set global keycache1.key_cache_block_size=8*1024; +insert into t2 values (8000, 3, 'yyyy'); +--replace_column 6 # 7 # +select * from information_schema.key_caches; + +set global keycache1.key_buffer_size=64*1024; +--replace_column 6 # 7 # +select * from information_schema.key_caches; + +set global keycache1.key_cache_block_size=2*1024; +--replace_column 6 # 7 # +select * from information_schema.key_caches; + +set global keycache1.key_cache_block_size=8*1024; +--replace_column 6 # 7 # +select * from information_schema.key_caches; + +set global keycache1.key_buffer_size=0; +--replace_column 6 # 7 # +select * from information_schema.key_caches; + +set global keycache1.key_cache_block_size=8*1024; +--replace_column 6 # 7 # +select * from information_schema.key_caches; + +set global keycache1.key_buffer_size=0; +--replace_column 6 # 7 # +select * from information_schema.key_caches; + +set global keycache1.key_buffer_size=128*1024; +--replace_column 6 # 7 # +select * from information_schema.key_caches; + +set global keycache1.key_cache_block_size=1024; +--replace_column 6 # 7 # +select * from information_schema.key_caches; + +drop table t1,t2; + +set global keycache1.key_buffer_size=0; +set global keycache2.key_buffer_size=0; + +set global key_buffer_size=@save_key_buffer_size; +set global key_cache_partitions=@save_key_cache_partitions; + +#End of 5.1 tests === added file 'mysql-test/t/select_pkeycache-master.opt' --- a/mysql-test/t/select_pkeycache-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/select_pkeycache-master.opt 2010-02-16 16:41:11 +0000 @@ -0,0 +1 @@ +--key_cache_partitions=7 === added file 'mysql-test/t/select_pkeycache.test' --- a/mysql-test/t/select_pkeycache.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/select_pkeycache.test 2010-02-16 16:41:11 +0000 @@ -0,0 +1,8 @@ + +# +# Run select.test partitioned default key cache (with 7 partitions) +# (see setting the number of partitions in select_pkecache-master.opt) +# The result is expected the same as for select.test +# + +--source t/select.test === modified file 'mysys/mf_keycache.c' --- a/mysys/mf_keycache.c 2009-12-03 11:19:05 +0000 +++ b/mysys/mf_keycache.c 2010-02-16 16:41:11 +0000 @@ -13,8 +13,38 @@ along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ + + /** - @file + @file + The file contains the following modules: + + Simple Key Cache Module + + Partitioned Key Cache Module + + Key Cache Interface Module + +*/ + +#include "mysys_priv.h" +#include "mysys_err.h" +#include <keycache.h> +#include "my_static.h" +#include <m_string.h> +#include <my_bit.h> +#include <errno.h> +#include <stdarg.h> + +/****************************************************************************** + Simple Key Cache Module + + The module contains implementations of all key cache interface functions + employed by partitioned key caches. + +******************************************************************************/ + +/* These functions handle keyblock cacheing for ISAM and MyISAM tables. One cache can handle many files. @@ -37,9 +67,7 @@ blocks_unused is the sum of never used blocks in the pool and of currently free blocks. blocks_used is the number of blocks fetched from the pool and as such gives the maximum number of in-use blocks at any time. -*/ -/* Key Cache Locking ================= @@ -104,14 +132,77 @@ I/O finished. */ -#include "mysys_priv.h" -#include "mysys_err.h" -#include <keycache.h> -#include "my_static.h" -#include <m_string.h> -#include <my_bit.h> -#include <errno.h> -#include <stdarg.h> +/* declare structures that is used by st_key_cache */ + +struct st_block_link; +typedef struct st_block_link BLOCK_LINK; +struct st_keycache_page; +typedef struct st_keycache_page KEYCACHE_PAGE; +struct st_hash_link; +typedef struct st_hash_link HASH_LINK; + +/* info about requests in a waiting queue */ +typedef struct st_keycache_wqueue +{ + struct st_my_thread_var *last_thread; /* circular list of waiting threads */ +} KEYCACHE_WQUEUE; + +#define CHANGED_BLOCKS_HASH 128 /* must be power of 2 */ + +/* Control block for a simple (non-partitioned) key cache */ + +typedef struct st_s_key_cache_cb +{ + my_bool key_cache_inited; /* <=> control block is allocated */ + my_bool in_resize; /* true during resize operation */ + my_bool resize_in_flush; /* true during flush of resize operation */ + my_bool can_be_used; /* usage of cache for read/write is allowed */ + size_t key_cache_mem_size; /* specified size of the cache memory */ + uint key_cache_block_size; /* size of the page buffer of a cache block */ + ulong min_warm_blocks; /* min number of warm blocks; */ + ulong age_threshold; /* age threshold for hot blocks */ + ulonglong keycache_time; /* total number of block link operations */ + uint hash_entries; /* max number of entries in the hash table */ + int hash_links; /* max number of hash links */ + int hash_links_used; /* number of hash links currently used */ + int disk_blocks; /* max number of blocks in the cache */ + ulong blocks_used; /* maximum number of concurrently used blocks */ + ulong blocks_unused; /* number of currently unused blocks */ + ulong blocks_changed; /* number of currently dirty blocks */ + ulong warm_blocks; /* number of blocks in warm sub-chain */ + ulong cnt_for_resize_op; /* counter to block resize operation */ + long blocks_available; /* number of blocks available in the LRU chain */ + HASH_LINK **hash_root; /* arr. of entries into hash table buckets */ + HASH_LINK *hash_link_root; /* memory for hash table links */ + HASH_LINK *free_hash_list; /* list of free hash links */ + BLOCK_LINK *free_block_list; /* list of free blocks */ + BLOCK_LINK *block_root; /* memory for block links */ + uchar HUGE_PTR *block_mem; /* memory for block buffers */ + BLOCK_LINK *used_last; /* ptr to the last block of the LRU chain */ + BLOCK_LINK *used_ins; /* ptr to the insertion block in LRU chain */ + pthread_mutex_t cache_lock; /* to lock access to the cache structure */ + KEYCACHE_WQUEUE resize_queue; /* threads waiting during resize operation */ + /* + Waiting for a zero resize count. Using a queue for symmetry though + only one thread can wait here. + */ + KEYCACHE_WQUEUE waiting_for_resize_cnt; + KEYCACHE_WQUEUE waiting_for_hash_link; /* waiting for a free hash link */ + KEYCACHE_WQUEUE waiting_for_block; /* requests waiting for a free block */ + BLOCK_LINK *changed_blocks[CHANGED_BLOCKS_HASH]; /* hash for dirty file bl.*/ + BLOCK_LINK *file_blocks[CHANGED_BLOCKS_HASH]; /* hash for other file bl.*/ + + /* Statistics variables. These are reset in reset_key_cache_counters(). */ + ulong global_blocks_changed; /* number of currently dirty blocks */ + ulonglong global_cache_w_requests;/* number of write requests (write hits) */ + ulonglong global_cache_write; /* number of writes from cache to files */ + ulonglong global_cache_r_requests;/* number of read requests (read hits) */ + ulonglong global_cache_read; /* number of reads from files to cache */ + + int blocks; /* max number of blocks in the cache */ + uint hash_factor; /* factor used to calculate hash function */ + my_bool in_init; /* Set to 1 in MySQL during init/resize */ +} S_KEY_CACHE_CB; /* Some compilation flags have been added specifically for this module @@ -223,7 +314,12 @@ KEY_CACHE *dflt_key_cache= &dflt_key_cac #define FLUSH_CACHE 2000 /* sort this many blocks at once */ -static int flush_all_key_blocks(KEY_CACHE *keycache); +static int flush_all_key_blocks(S_KEY_CACHE_CB *keycache); +/* +static void s_change_key_cache_param(void *keycache_cb, uint division_limit, + uint age_threshold); +*/ +static void s_end_key_cache(void *keycache_cb, my_bool cleanup); #ifdef THREAD static void wait_on_queue(KEYCACHE_WQUEUE *wqueue, pthread_mutex_t *mutex); @@ -232,15 +328,16 @@ static void release_whole_queue(KEYCACHE #define wait_on_queue(wqueue, mutex) do {} while (0) #define release_whole_queue(wqueue) do {} while (0) #endif -static void free_block(KEY_CACHE *keycache, BLOCK_LINK *block); +static void free_block(S_KEY_CACHE_CB *keycache, BLOCK_LINK *block); #if !defined(DBUG_OFF) -static void test_key_cache(KEY_CACHE *keycache, +static void test_key_cache(S_KEY_CACHE_CB *keycache, const char *where, my_bool lock); #endif - +#define KEYCACHE_BASE_EXPR(f, pos) \ + ((ulong) ((pos) / keycache->key_cache_block_size) + (ulong) (f)) #define KEYCACHE_HASH(f, pos) \ -(((ulong) ((pos) / keycache->key_cache_block_size) + \ - (ulong) (f)) & (keycache->hash_entries-1)) + ((KEYCACHE_BASE_EXPR(f, pos) / keycache->hash_factor) & \ + (keycache->hash_entries-1)) #define FILE_HASH(f) ((uint) (f) & (CHANGED_BLOCKS_HASH-1)) #define DEFAULT_KEYCACHE_DEBUG_LOG "keycache_debug.log" @@ -336,9 +433,10 @@ static int keycache_pthread_cond_signal( #define inline /* disabled inline for easier debugging */ static int fail_block(BLOCK_LINK *block); static int fail_hlink(HASH_LINK *hlink); -static int cache_empty(KEY_CACHE *keycache); +static int cache_empty(S_KEY_CACHE_CB *keycache); #endif + static inline uint next_power(uint value) { return (uint) my_round_up_to_next_power((uint32) value) << 1; @@ -346,19 +444,32 @@ static inline uint next_power(uint value /* - Initialize a key cache + Initialize a simple key cache SYNOPSIS - init_key_cache() - keycache pointer to a key cache data structure - key_cache_block_size size of blocks to keep cached data - use_mem total memory to use for the key cache - division_limit division limit (may be zero) - age_threshold age threshold (may be zero) + s_init_key_cache() + keycache_cb pointer to the control block of a simple key cache + key_cache_block_size size of blocks to keep cached data + use_mem memory to use for the key cache buferrs/structures + division_limit division limit (may be zero) + age_threshold age threshold (may be zero) + + DESCRIPTION + This function is the implementation of the init_key_cache interface + function that is employed by simple (non-partitioned) key caches. + The function builds a simple key cache and initializes the control block + structure of the type S_KEY_CACHE_CB that is used for this key cache. + The parameter keycache_cb is supposed to point to this structure. + The parameter key_cache_block_size specifies the size of the blocks in + the key cache to be built. The parameters division_limit and age_threshhold + determine the initial values of those characteristics of the key cache + that are used for midpoint insertion strategy. The parameter use_mem + specifies the total amount of memory to be allocated for key cache blocks + and auxiliary structures. RETURN VALUE number of blocks in the key cache, if successful, - 0 - otherwise. + <= 0 - otherwise. NOTES. if keycache->key_cache_inited != 0 we assume that the key cache @@ -370,10 +481,12 @@ static inline uint next_power(uint value */ -int init_key_cache(KEY_CACHE *keycache, uint key_cache_block_size, - size_t use_mem, uint division_limit, - uint age_threshold) +static +int s_init_key_cache(void *keycache_cb, uint key_cache_block_size, + size_t use_mem, uint division_limit, + uint age_threshold) { + S_KEY_CACHE_CB *keycache= (S_KEY_CACHE_CB *) keycache_cb; ulong blocks, hash_links; size_t length; int error; @@ -387,12 +500,15 @@ int init_key_cache(KEY_CACHE *keycache, DBUG_RETURN(0); } + keycache->blocks_used= keycache->blocks_unused= 0; + keycache->global_blocks_changed= 0; keycache->global_cache_w_requests= keycache->global_cache_r_requests= 0; keycache->global_cache_read= keycache->global_cache_write= 0; keycache->disk_blocks= -1; if (! keycache->key_cache_inited) { keycache->key_cache_inited= 1; + keycache->hash_factor= 1; /* Initialize these variables once only. Their value must survive re-initialization during resizing. @@ -534,51 +650,43 @@ err: /* - Resize a key cache + Prepare for resizing a simple key cache SYNOPSIS - resize_key_cache() - keycache pointer to a key cache data structure - key_cache_block_size size of blocks to keep cached data - use_mem total memory to use for the new key cache - division_limit new division limit (if not zero) - age_threshold new age threshold (if not zero) + s_prepare_resize_key_cache() + keycache_cb pointer to the control block of a simple key cache + with_resize_queue <=> resize queue is used + release_lock <=> release the key cache lock before return - RETURN VALUE - number of blocks in the key cache, if successful, - 0 - otherwise. + DESCRIPTION + This function flushes all dirty pages from a simple key cache and after + this it destroys the key cache calling s_end_key_cache. The function + considers the parameter keycache_cb as a pointer to the control block + structure of the type S_KEY_CACHE_CB for this key cache. + The parameter with_resize_queue determines weather the resize queue is + involved (MySQL server never uses this queue). The parameter release_lock + says weather the key cache lock must be released before return from + the function. - NOTES. - The function first compares the memory size and the block size parameters - with the key cache values. + RETURN VALUE + 0 - on success, + 1 - otherwise. - If they differ the function free the the memory allocated for the - old key cache blocks by calling the end_key_cache function and - then rebuilds the key cache with new blocks by calling - init_key_cache. + NOTES + This function is the called by s_resize_key_cache and p_resize_key_cache + that resize simple and partitioned key caches respectively. - The function starts the operation only when all other threads - performing operations with the key cache let her to proceed - (when cnt_for_resize=0). */ -int resize_key_cache(KEY_CACHE *keycache, uint key_cache_block_size, - size_t use_mem, uint division_limit, - uint age_threshold) +static +int s_prepare_resize_key_cache(void *keycache_cb, + my_bool with_resize_queue, + my_bool release_lock) { - int blocks; - DBUG_ENTER("resize_key_cache"); - - if (!keycache->key_cache_inited) - DBUG_RETURN(keycache->disk_blocks); - - if(key_cache_block_size == keycache->key_cache_block_size && - use_mem == keycache->key_cache_mem_size) - { - change_key_cache_param(keycache, division_limit, age_threshold); - DBUG_RETURN(keycache->disk_blocks); - } - + int res= 0; + S_KEY_CACHE_CB *keycache= (S_KEY_CACHE_CB *) keycache_cb; + DBUG_ENTER("s_prepare_resize_key_cache"); + keycache_pthread_mutex_lock(&keycache->cache_lock); #ifdef THREAD @@ -588,7 +696,7 @@ int resize_key_cache(KEY_CACHE *keycache one resizer only. In set_var.cc keycache->in_init is used to block multiple attempts. */ - while (keycache->in_resize) + while (with_resize_queue && keycache->in_resize) { /* purecov: begin inspected */ wait_on_queue(&keycache->resize_queue, &keycache->cache_lock); @@ -613,8 +721,8 @@ int resize_key_cache(KEY_CACHE *keycache { /* TODO: if this happens, we should write a warning in the log file ! */ keycache->resize_in_flush= 0; - blocks= 0; keycache->can_be_used= 0; + res= 1; goto finish; } DBUG_ASSERT(cache_empty(keycache)); @@ -640,29 +748,145 @@ int resize_key_cache(KEY_CACHE *keycache #else KEYCACHE_DBUG_ASSERT(keycache->cnt_for_resize_op == 0); #endif - - /* - Free old cache structures, allocate new structures, and initialize - them. Note that the cache_lock mutex and the resize_queue are left - untouched. We do not lose the cache_lock and will release it only at - the end of this function. - */ - end_key_cache(keycache, 0); /* Don't free mutex */ - /* The following will work even if use_mem is 0 */ - blocks= init_key_cache(keycache, key_cache_block_size, use_mem, - division_limit, age_threshold); + + s_end_key_cache(keycache_cb, 0); finish: + if (release_lock) + keycache_pthread_mutex_unlock(&keycache->cache_lock); + DBUG_RETURN(res); +} + + +/* + Finalize resizing a simple key cache + + SYNOPSIS + s_finish_resize_key_cache() + keycache_cb pointer to the control block of a simple key cache + with_resize_queue <=> resize queue is used + acquire_lock <=> acquire the key cache lock at start + + DESCRIPTION + This function performs finalizing actions for the operation of + resizing a simple key cache. The function considers the parameter + keycache_cb as a pointer to the control block structure of the type + S_KEY_CACHE_CB for this key cache. The function sets the flag + in_resize in this structure to FALSE. + The parameter with_resize_queue determines weather the resize queue + is involved (MySQL server never uses this queue). + The parameter acquire_lock says weather the key cache lock must be + acquired at the start of the function. + + RETURN VALUE + none + + NOTES + This function is the called by s_resize_key_cache and p_resize_key_cache + that resize simple and partitioned key caches respectively. + +*/ + +static +void s_finish_resize_key_cache(void *keycache_cb, + my_bool with_resize_queue, + my_bool acquire_lock) +{ + S_KEY_CACHE_CB *keycache= (S_KEY_CACHE_CB *) keycache_cb; + DBUG_ENTER("s_finish_resize_key_cache"); + + if (acquire_lock) + keycache_pthread_mutex_lock(&keycache->cache_lock); + /* Mark the resize finished. This allows other threads to start a resize or to request new cache blocks. */ keycache->in_resize= 0; - - /* Signal waiting threads. */ - release_whole_queue(&keycache->resize_queue); + + if (with_resize_queue) + { + /* Signal waiting threads. */ + release_whole_queue(&keycache->resize_queue); + } keycache_pthread_mutex_unlock(&keycache->cache_lock); + + DBUG_VOID_RETURN; +} + + +/* + Resize a simple key cache + + SYNOPSIS + s_resize_key_cache() + keycache_cb pointer to the control block of a simple key cache + key_cache_block_size size of blocks to keep cached data + use_mem memory to use for the key cache buffers/structures + division_limit new division limit (if not zero) + age_threshold new age threshold (if not zero) + + DESCRIPTION + This function is the implementation of the resize_key_cache interface + function that is employed by simple (non-partitioned) key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type S_KEY_CACHE_CB for the simple key + cache to be resized. + The parameter key_cache_block_size specifies the new size of the blocks in + the key cache. The parameters division_limit and age_threshold + determine the new initial values of those characteristics of the key cache + that are used for midpoint insertion strategy. The parameter use_mem + specifies the total amount of memory to be allocated for key cache blocks + and auxiliary structures in the new key cache. + + RETURN VALUE + number of blocks in the key cache, if successful, + 0 - otherwise. + + NOTES. + The function first calls the function s_prepare_resize_key_cache + to flush all dirty blocks from key cache, to free memory used + for key cache blocks and auxiliary structures. After this the + function builds a new key cache with new parameters. + + This implementation doesn't block the calls and executions of other + functions from the key cache interface. However it assumes that the + calls of s_resize_key_cache itself are serialized. + + The function starts the operation only when all other threads + performing operations with the key cache let her to proceed + (when cnt_for_resize=0). + +*/ + +static +int s_resize_key_cache(void *keycache_cb, uint key_cache_block_size, + size_t use_mem, uint division_limit, + uint age_threshold) +{ + S_KEY_CACHE_CB *keycache= (S_KEY_CACHE_CB *) keycache_cb; + int blocks= 0; + DBUG_ENTER("s_resize_key_cache"); + + if (!keycache->key_cache_inited) + DBUG_RETURN(keycache->disk_blocks); + + /* + Note that the cache_lock mutex and the resize_queue are left untouched. + We do not lose the cache_lock and will release it only at the end of + this function. + */ + if (s_prepare_resize_key_cache(keycache_cb, 1, 0)) + goto finish; + + /* The following will work even if use_mem is 0 */ + blocks= s_init_key_cache(keycache, key_cache_block_size, use_mem, + division_limit, age_threshold); + +finish: + s_finish_resize_key_cache(keycache_cb, 1, 0); + DBUG_RETURN(blocks); } @@ -670,7 +894,7 @@ finish: /* Increment counter blocking resize key cache operation */ -static inline void inc_counter_for_resize_op(KEY_CACHE *keycache) +static inline void inc_counter_for_resize_op(S_KEY_CACHE_CB *keycache) { keycache->cnt_for_resize_op++; } @@ -680,35 +904,49 @@ static inline void inc_counter_for_resiz Decrement counter blocking resize key cache operation; Signal the operation to proceed when counter becomes equal zero */ -static inline void dec_counter_for_resize_op(KEY_CACHE *keycache) +static inline void dec_counter_for_resize_op(S_KEY_CACHE_CB *keycache) { if (!--keycache->cnt_for_resize_op) release_whole_queue(&keycache->waiting_for_resize_cnt); } + /* - Change the key cache parameters + Change key cache parameters of a simple key cache SYNOPSIS - change_key_cache_param() - keycache pointer to a key cache data structure - division_limit new division limit (if not zero) - age_threshold new age threshold (if not zero) + s_change_key_cache_param() + keycache_cb pointer to the control block of a simple key cache + division_limit new division limit (if not zero) + age_threshold new age threshold (if not zero) + + DESCRIPTION + This function is the implementation of the change_key_cache_param interface + function that is employed by simple (non-partitioned) key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type S_KEY_CACHE_CB for the simple key + cache where new values of the division limit and the age threshold used + for midpoint insertion strategy are to be set. The parameters + division_limit and age_threshold provide these new values. RETURN VALUE none NOTES. - Presently the function resets the key cache parameters - concerning midpoint insertion strategy - division_limit and - age_threshold. + Presently the function resets the key cache parameters concerning + midpoint insertion strategy - division_limit and age_threshold. + This function changes some parameters of a given key cache without + reformatting it. The function does not touch the contents the key + cache blocks. + */ -void change_key_cache_param(KEY_CACHE *keycache, uint division_limit, - uint age_threshold) +static +void s_change_key_cache_param(void *keycache_cb, uint division_limit, + uint age_threshold) { - DBUG_ENTER("change_key_cache_param"); - + S_KEY_CACHE_CB *keycache= (S_KEY_CACHE_CB *) keycache_cb; + DBUG_ENTER("s_change_key_cache_param"); keycache_pthread_mutex_lock(&keycache->cache_lock); if (division_limit) keycache->min_warm_blocks= (keycache->disk_blocks * @@ -722,20 +960,32 @@ void change_key_cache_param(KEY_CACHE *k /* - Remove key_cache from memory + Destroy a simple key cache SYNOPSIS - end_key_cache() - keycache key cache handle - cleanup Complete free (Free also mutex for key cache) + s_end_key_cache() + keycache_cb pointer to the control block of a simple key cache + cleanup <=> complete free (free also mutex for key cache) + + DESCRIPTION + This function is the implementation of the end_key_cache interface + function that is employed by simple (non-partitioned) key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type S_KEY_CACHE_CB for the simple key + cache to be destroyed. + The function frees the memory allocated for the key cache blocks and + auxiliary structures. If the value of the parameter cleanup is TRUE + then even the key cache mutex is freed. RETURN VALUE none */ -void end_key_cache(KEY_CACHE *keycache, my_bool cleanup) +static +void s_end_key_cache(void *keycache_cb, my_bool cleanup) { - DBUG_ENTER("end_key_cache"); + S_KEY_CACHE_CB *keycache= (S_KEY_CACHE_CB *) keycache_cb; + DBUG_ENTER("s_end_key_cache"); DBUG_PRINT("enter", ("key_cache: 0x%lx", (long) keycache)); if (!keycache->key_cache_inited) @@ -1026,7 +1276,7 @@ static inline void link_changed(BLOCK_LI void */ -static void link_to_file_list(KEY_CACHE *keycache, +static void link_to_file_list(S_KEY_CACHE_CB *keycache, BLOCK_LINK *block, int file, my_bool unlink_block) { @@ -1067,7 +1317,7 @@ static void link_to_file_list(KEY_CACHE void */ -static void link_to_changed_list(KEY_CACHE *keycache, +static void link_to_changed_list(S_KEY_CACHE_CB *keycache, BLOCK_LINK *block) { DBUG_ASSERT(block->status & BLOCK_IN_USE); @@ -1122,7 +1372,7 @@ static void link_to_changed_list(KEY_CAC not linked in the LRU ring. */ -static void link_block(KEY_CACHE *keycache, BLOCK_LINK *block, my_bool hot, +static void link_block(S_KEY_CACHE_CB *keycache, BLOCK_LINK *block, my_bool hot, my_bool at_end) { BLOCK_LINK *ins; @@ -1243,7 +1493,7 @@ static void link_block(KEY_CACHE *keycac See NOTES for link_block */ -static void unlink_block(KEY_CACHE *keycache, BLOCK_LINK *block) +static void unlink_block(S_KEY_CACHE_CB *keycache, BLOCK_LINK *block) { DBUG_ASSERT((block->status & ~BLOCK_CHANGED) == (BLOCK_READ | BLOCK_IN_USE)); DBUG_ASSERT(block->hash_link); /*backptr to block NULL from free_block()*/ @@ -1301,7 +1551,7 @@ static void unlink_block(KEY_CACHE *keyc RETURN void */ -static void reg_requests(KEY_CACHE *keycache, BLOCK_LINK *block, int count) +static void reg_requests(S_KEY_CACHE_CB *keycache, BLOCK_LINK *block, int count) { DBUG_ASSERT(block->status & BLOCK_IN_USE); DBUG_ASSERT(block->hash_link); @@ -1344,7 +1594,7 @@ static void reg_requests(KEY_CACHE *keyc not linked in the LRU ring. */ -static void unreg_request(KEY_CACHE *keycache, +static void unreg_request(S_KEY_CACHE_CB *keycache, BLOCK_LINK *block, int at_end) { DBUG_ASSERT(block->status & (BLOCK_READ | BLOCK_IN_USE)); @@ -1433,7 +1683,7 @@ static void remove_reader(BLOCK_LINK *bl signals on its termination */ -static void wait_for_readers(KEY_CACHE *keycache, +static void wait_for_readers(S_KEY_CACHE_CB *keycache, BLOCK_LINK *block) { #ifdef THREAD @@ -1482,7 +1732,7 @@ static inline void link_hash(HASH_LINK * Remove a hash link from the hash table */ -static void unlink_hash(KEY_CACHE *keycache, HASH_LINK *hash_link) +static void unlink_hash(S_KEY_CACHE_CB *keycache, HASH_LINK *hash_link) { KEYCACHE_DBUG_PRINT("unlink_hash", ("fd: %u pos_ %lu #requests=%u", (uint) hash_link->file,(ulong) hash_link->diskpos, hash_link->requests)); @@ -1538,7 +1788,7 @@ static void unlink_hash(KEY_CACHE *keyca Get the hash link for a page */ -static HASH_LINK *get_hash_link(KEY_CACHE *keycache, +static HASH_LINK *get_hash_link(S_KEY_CACHE_CB *keycache, int file, my_off_t filepos) { reg1 HASH_LINK *hash_link, **start; @@ -1659,7 +1909,7 @@ restart: waits until first of this operations links any block back. */ -static BLOCK_LINK *find_key_block(KEY_CACHE *keycache, +static BLOCK_LINK *find_key_block(S_KEY_CACHE_CB *keycache, File file, my_off_t filepos, int init_hits_left, int wrmode, int *page_st) @@ -2419,7 +2669,7 @@ restart: portion is less than read_length, but not less than min_length. */ -static void read_block(KEY_CACHE *keycache, +static void read_block(S_KEY_CACHE_CB *keycache, BLOCK_LINK *block, uint read_length, uint min_length, my_bool primary) { @@ -2507,43 +2757,62 @@ static void read_block(KEY_CACHE *keycac /* - Read a block of data from a cached file into a buffer; + Read a block of data from a simple key cache into a buffer SYNOPSIS - key_cache_read() - keycache pointer to a key cache data structure - file handler for the file for the block of data to be read - filepos position of the block of data in the file - level determines the weight of the data - buff buffer to where the data must be placed - length length of the buffer - block_length length of the block in the key cache buffer - return_buffer return pointer to the key cache buffer with the data + s_key_cache_read() + keycache_cb pointer to the control block of a simple key cache + file handler for the file for the block of data to be read + filepos position of the block of data in the file + level determines the weight of the data + buff buffer to where the data must be placed + length length of the buffer + block_length length of the read data from a key cache block + return_buffer return pointer to the key cache buffer with the data + DESCRIPTION + This function is the implementation of the key_cache_read interface + function that is employed by simple (non-partitioned) key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type S_KEY_CACHE_CB for a simple key + cache. + In a general case the function reads a block of data from the key cache + into the buffer buff of the size specified by the parameter length. The + beginning of the block of data to be read is specified by the parameters + file and filepos. The length of the read data is the same as the length + of the buffer. The data is read into the buffer in key_cache_block_size + increments. If the next portion of the data is not found in any key cache + block, first it is read from file into the key cache. + If the parameter return_buffer is not ignored and its value is TRUE, and + the data to be read of the specified size block_length can be read from one + key cache buffer, then the function returns a pointer to the data in the + key cache buffer. + The function takse into account parameters block_length and return buffer + only in a single-threaded environment. + The parameter 'level' is used only by the midpoint insertion strategy + when the data or its portion cannot be found in the key cache. + RETURN VALUE - Returns address from where the data is placed if sucessful, 0 - otherwise. + Returns address from where the data is placed if successful, 0 - otherwise. - NOTES. - The function ensures that a block of data of size length from file - positioned at filepos is in the buffers for some key cache blocks. - Then the function either copies the data into the buffer buff, or, - if return_buffer is TRUE, it just returns the pointer to the key cache - buffer with the data. + NOTES Filepos must be a multiple of 'block_length', but it doesn't have to be a multiple of key_cache_block_size; + */ -uchar *key_cache_read(KEY_CACHE *keycache, - File file, my_off_t filepos, int level, - uchar *buff, uint length, - uint block_length __attribute__((unused)), - int return_buffer __attribute__((unused))) +uchar *s_key_cache_read(void *keycache_cb, + File file, my_off_t filepos, int level, + uchar *buff, uint length, + uint block_length __attribute__((unused)), + int return_buffer __attribute__((unused))) { + S_KEY_CACHE_CB *keycache= (S_KEY_CACHE_CB *) keycache_cb; my_bool locked_and_incremented= FALSE; int error=0; uchar *start= buff; - DBUG_ENTER("key_cache_read"); + DBUG_ENTER("s_key_cache_read"); DBUG_PRINT("enter", ("fd: %u pos: %lu length: %u", (uint) file, (ulong) filepos, length)); @@ -2738,29 +3007,49 @@ end: /* - Insert a block of file data from a buffer into key cache + Insert a block of file data from a buffer into a simple key cache SYNOPSIS - key_cache_insert() - keycache pointer to a key cache data structure + s_key_cache_insert() + keycache_cb pointer to the control block of a simple key cache file handler for the file to insert data from filepos position of the block of data in the file to insert level determines the weight of the data buff buffer to read data from length length of the data in the buffer - NOTES - This is used by MyISAM to move all blocks from a index file to the key - cache - + DESCRIPTION + This function is the implementation of the key_cache_insert interface + function that is employed by simple (non-partitioned) key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type S_KEY_CACHE_CB for a simple key + cache. + The function writes a block of file data from a buffer into the key cache. + The buffer is specified with the parameters buff and length - the pointer + to the beginning of the buffer and its size respectively. It's assumed + the buffer contains the data from 'file' allocated from the position + filepos. The data is copied from the buffer in key_cache_block_size + increments. + The parameter level is used to set one characteristic for the key buffers + loaded with the data from buff. The characteristic is used only by the + midpoint insertion strategy. + RETURN VALUE 0 if a success, 1 - otherwise. + + NOTES + The function is used by MyISAM to move all blocks from a index file to + the key cache. It can be performed in parallel with reading the file data + from the key buffers by other threads. + */ -int key_cache_insert(KEY_CACHE *keycache, - File file, my_off_t filepos, int level, - uchar *buff, uint length) +static +int s_key_cache_insert(void *keycache_cb, + File file, my_off_t filepos, int level, + uchar *buff, uint length) { + S_KEY_CACHE_CB *keycache= (S_KEY_CACHE_CB *) keycache_cb; int error= 0; DBUG_ENTER("key_cache_insert"); DBUG_PRINT("enter", ("fd: %u pos: %lu length: %u", @@ -2979,43 +3268,65 @@ int key_cache_insert(KEY_CACHE *keycache /* - Write a buffer into a cached file. + Write a buffer into a simple key cache SYNOPSIS - key_cache_write() - keycache pointer to a key cache data structure - file handler for the file to write data to - filepos position in the file to write data to - level determines the weight of the data - buff buffer with the data - length length of the buffer - dont_write if is 0 then all dirty pages involved in writing - should have been flushed from key cache + s_key_cache_write() + keycache_cb pointer to the control block of a simple key cache + file handler for the file to write data to + file_extra maps of key cache partitions containing + dirty pages from file + filepos position in the file to write data to + level determines the weight of the data + buff buffer with the data + length length of the buffer + dont_write if is 0 then all dirty pages involved in writing + should have been flushed from key cache + DESCRIPTION + This function is the implementation of the key_cache_write interface + function that is employed by simple (non-partitioned) key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type S_KEY_CACHE_CB for a simple key + cache. + In a general case the function copies data from a buffer into the key + cache. The buffer is specified with the parameters buff and length - + the pointer to the beginning of the buffer and its size respectively. + It's assumed the buffer contains the data to be written into 'file' + starting from the position filepos. The data is copied from the buffer + in key_cache_block_size increments. + If the value of the parameter dont_write is FALSE then the function + also writes the data into file. + The parameter level is used to set one characteristic for the key buffers + filled with the data from buff. The characteristic is employed only by + the midpoint insertion strategy. + The parameter file_extra currently makes sense only for simple key caches + that are elements of a partitioned key cache. It provides a pointer to the + shared bitmap of the partitions that may contains dirty pages for the file. + This bitmap is used to optimize the function p_flush_key_blocks. + RETURN VALUE 0 if a success, 1 - otherwise. - NOTES. - The function copies the data of size length from buff into buffers - for key cache blocks that are assigned to contain the portion of - the file starting with position filepos. - It ensures that this data is flushed to the file if dont_write is FALSE. - Filepos must be a multiple of 'block_length', but it doesn't - have to be a multiple of key_cache_block_size; + NOTES + This implementation exploits the fact that the function is called only + when a thread has got an exclusive lock for the key file. - dont_write is always TRUE in the server (info->lock_type is never F_UNLCK). */ -int key_cache_write(KEY_CACHE *keycache, - File file, my_off_t filepos, int level, - uchar *buff, uint length, - uint block_length __attribute__((unused)), - int dont_write) +static +int s_key_cache_write(void *keycache_cb, + File file, void *file_extra __attribute__((unused)), + my_off_t filepos, int level, + uchar *buff, uint length, + uint block_length __attribute__((unused)), + int dont_write) { + S_KEY_CACHE_CB *keycache= (S_KEY_CACHE_CB *) keycache_cb; my_bool locked_and_incremented= FALSE; int error=0; - DBUG_ENTER("key_cache_write"); + DBUG_ENTER("s_key_cache_write"); DBUG_PRINT("enter", ("fd: %u pos: %lu length: %u block_length: %u" " key_block_length: %u", @@ -3330,7 +3641,7 @@ end: Block must have a request registered on it. */ -static void free_block(KEY_CACHE *keycache, BLOCK_LINK *block) +static void free_block(S_KEY_CACHE_CB *keycache, BLOCK_LINK *block) { KEYCACHE_THREAD_TRACE("free block"); KEYCACHE_DBUG_PRINT("free_block", @@ -3470,7 +3781,7 @@ static int cmp_sec_link(BLOCK_LINK **a, free used blocks if requested */ -static int flush_cached_blocks(KEY_CACHE *keycache, +static int flush_cached_blocks(S_KEY_CACHE_CB *keycache, File file, BLOCK_LINK **cache, BLOCK_LINK **end, enum flush_type type) @@ -3514,9 +3825,9 @@ static int flush_cached_blocks(KEY_CACHE (BLOCK_READ | BLOCK_IN_FLUSH | BLOCK_CHANGED | BLOCK_IN_USE)); block->status|= BLOCK_IN_FLUSHWRITE; keycache_pthread_mutex_unlock(&keycache->cache_lock); - error= my_pwrite(file, block->buffer+block->offset, + error= my_pwrite(file, block->buffer + block->offset, block->length - block->offset, - block->hash_link->diskpos+ block->offset, + block->hash_link->diskpos + block->offset, MYF(MY_NABP | MY_WAIT_IF_FULL)); keycache_pthread_mutex_lock(&keycache->cache_lock); keycache->global_cache_write++; @@ -3576,7 +3887,7 @@ static int flush_cached_blocks(KEY_CACHE /* - Flush all key blocks for a file to disk, but don't do any mutex locks. + Flush all key blocks for a file to disk, but don't do any mutex locks SYNOPSIS flush_key_blocks_int() @@ -3598,7 +3909,7 @@ static int flush_cached_blocks(KEY_CACHE 1 error */ -static int flush_key_blocks_int(KEY_CACHE *keycache, +static int flush_key_blocks_int(S_KEY_CACHE_CB *keycache, File file, enum flush_type type) { BLOCK_LINK *cache_buff[FLUSH_CACHE],**cache; @@ -4034,23 +4345,49 @@ err: /* - Flush all blocks for a file to disk + Flush all blocks for a file from key buffers of a simple key cache SYNOPSIS - flush_key_blocks() - keycache pointer to a key cache data structure - file handler for the file to flush to - flush_type type of the flush + s_flush_key_blocks() + keycache_cb pointer to the control block of a simple key cache + file handler for the file to flush to + file_extra maps of key cache partitions containing + dirty pages from file (not used) + flush_type type of the flush operation + DESCRIPTION + This function is the implementation of the flush_key_blocks interface + function that is employed by simple (non-partitioned) key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type S_KEY_CACHE_CB for a simple key + cache. + In a general case the function flushes the data from all dirty key + buffers related to the file 'file' into this file. The function does + exactly this if the value of the parameter type is FLUSH_KEEP. If the + value of this parameter is FLUSH_RELEASE, the function additionally + releases the key buffers containing data from 'file' for new usage. + If the value of the parameter type is FLUSH_IGNORE_CHANGED the function + just releases the key buffers containing data from 'file'. + The parameter file_extra currently is not used by this function. + RETURN 0 ok 1 error + + NOTES + This implementation exploits the fact that the function is called only + when a thread has got an exclusive lock for the key file. + */ -int flush_key_blocks(KEY_CACHE *keycache, - File file, enum flush_type type) +static +int s_flush_key_blocks(void *keycache_cb, + File file, + void *file_extra __attribute__((unused)), + enum flush_type type) { + S_KEY_CACHE_CB *keycache= (S_KEY_CACHE_CB *) keycache_cb; int res= 0; DBUG_ENTER("flush_key_blocks"); DBUG_PRINT("enter", ("keycache: 0x%lx", (long) keycache)); @@ -4103,7 +4440,7 @@ int flush_key_blocks(KEY_CACHE *keycache != 0 Error */ -static int flush_all_key_blocks(KEY_CACHE *keycache) +static int flush_all_key_blocks(S_KEY_CACHE_CB *keycache) { BLOCK_LINK *block; uint total_found; @@ -4206,37 +4543,45 @@ static int flush_all_key_blocks(KEY_CACH /* - Reset the counters of a key cache. + Reset the counters of a simple key cache SYNOPSIS - reset_key_cache_counters() - name the name of a key cache - key_cache pointer to the key kache to be reset + s_reset_key_cache_counters() + name the name of a key cache + keycache_cb pointer to the control block of a simple key cache DESCRIPTION - This procedure is used by process_key_caches() to reset the counters of all - currently used key caches, both the default one and the named ones. + This function is the implementation of the reset_key_cache_counters + interface function that is employed by simple (non-partitioned) key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type S_KEY_CACHE_CB for a simple key cache. + This function resets the values of all statistical counters for the key + cache to 0. + The parameter name is currently not used. RETURN 0 on success (always because it can't fail) + */ -int reset_key_cache_counters(const char *name __attribute__((unused)), - KEY_CACHE *key_cache) +static +int s_reset_key_cache_counters(const char *name __attribute__((unused)), + void *keycache_cb) { - DBUG_ENTER("reset_key_cache_counters"); - if (!key_cache->key_cache_inited) + S_KEY_CACHE_CB *keycache= (S_KEY_CACHE_CB *) keycache_cb; + DBUG_ENTER("s_reset_key_cache_counters"); + if (!keycache->key_cache_inited) { DBUG_PRINT("info", ("Key cache %s not initialized.", name)); DBUG_RETURN(0); } DBUG_PRINT("info", ("Resetting counters for key cache %s.", name)); - key_cache->global_blocks_changed= 0; /* Key_blocks_not_flushed */ - key_cache->global_cache_r_requests= 0; /* Key_read_requests */ - key_cache->global_cache_read= 0; /* Key_reads */ - key_cache->global_cache_w_requests= 0; /* Key_write_requests */ - key_cache->global_cache_write= 0; /* Key_writes */ + keycache->global_blocks_changed= 0; /* Key_blocks_not_flushed */ + keycache->global_cache_r_requests= 0; /* Key_read_requests */ + keycache->global_cache_read= 0; /* Key_reads */ + keycache->global_cache_w_requests= 0; /* Key_write_requests */ + keycache->global_cache_write= 0; /* Key_writes */ DBUG_RETURN(0); } @@ -4245,7 +4590,7 @@ int reset_key_cache_counters(const char /* Test if disk-cache is ok */ -static void test_key_cache(KEY_CACHE *keycache __attribute__((unused)), +static void test_key_cache(S_KEY_CACHE_CB *keycache __attribute__((unused)), const char *where __attribute__((unused)), my_bool lock __attribute__((unused))) { @@ -4259,7 +4604,7 @@ static void test_key_cache(KEY_CACHE *ke #define MAX_QUEUE_LEN 100 -static void keycache_dump(KEY_CACHE *keycache) +static void keycache_dump(S_KEY_CACHE_CB *keycache) { FILE *keycache_dump_file=fopen(KEYCACHE_DUMP_FILE, "w"); struct st_my_thread_var *last; @@ -4499,7 +4844,7 @@ static int fail_hlink(HASH_LINK *hlink) return 0; /* Let the assert fail. */ } -static int cache_empty(KEY_CACHE *keycache) +static int cache_empty(S_KEY_CACHE_CB *keycache) { int errcnt= 0; int idx; @@ -4537,3 +4882,1675 @@ static int cache_empty(KEY_CACHE *keycac } #endif + +/* + Get statistics for a simple key cache + + SYNOPSIS + get_key_cache_statistics() + keycache_cb pointer to the control block of a simple key cache + partition_no partition number (not used) + key_cache_stats OUT pointer to the structure for the returned statistics + + DESCRIPTION + This function is the implementation of the get_key_cache_statistics + interface function that is employed by simple (non-partitioned) key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type S_KEY_CACHE_CB for a simple key cache. + This function returns the statistical data for the key cache. + The parameter partition_no is not used by this function. + + RETURN + none + +*/ + +static +void s_get_key_cache_statistics(void *keycache_cb, + uint partition_no __attribute__((unused)), + KEY_CACHE_STATISTICS *key_cache_stats) +{ + S_KEY_CACHE_CB *keycache= (S_KEY_CACHE_CB *) keycache_cb; + DBUG_ENTER("s_get_key_cache_statistics"); + + key_cache_stats->mem_size= (longlong) keycache->key_cache_mem_size; + key_cache_stats->block_size= (longlong) keycache->key_cache_block_size; + key_cache_stats->blocks_used= keycache->blocks_used; + key_cache_stats->blocks_unused= keycache->blocks_unused; + key_cache_stats->blocks_changed= keycache->global_blocks_changed; + key_cache_stats->read_requests= keycache->global_cache_r_requests; + key_cache_stats->reads= keycache->global_cache_read; + key_cache_stats->write_requests= keycache->global_cache_w_requests; + key_cache_stats->writes= keycache->global_cache_write; + DBUG_VOID_RETURN; +} + + +static size_t s_key_cache_stat_var_offsets[]= +{ + offsetof(S_KEY_CACHE_CB, blocks_used), + offsetof(S_KEY_CACHE_CB, blocks_unused), + offsetof(S_KEY_CACHE_CB, global_blocks_changed), + offsetof(S_KEY_CACHE_CB, global_cache_w_requests), + offsetof(S_KEY_CACHE_CB, global_cache_write), + offsetof(S_KEY_CACHE_CB, global_cache_r_requests), + offsetof(S_KEY_CACHE_CB, global_cache_read) +}; + + +/* + Get the value of a statistical variable for a simple key cache + + SYNOPSIS + s_get_key_cache_stat_value() + keycache_cb pointer to the control block of a simple key cache + var_no the ordered number of a statistical variable + + DESCRIPTION + This function is the implementation of the s_get_key_cache_stat_value + interface function that is employed by simple (non-partitioned) key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type S_KEY_CACHE_CB for a simple key cache. + This function returns the value of the statistical variable var_no + for this key cache. The variables are numbered starting from 0 to 6. + + RETURN + The value of the specified statistical variable + +*/ + +static +ulonglong s_get_key_cache_stat_value(void *keycache_cb, uint var_no) +{ + S_KEY_CACHE_CB *keycache= (S_KEY_CACHE_CB *) keycache_cb; + size_t var_ofs= s_key_cache_stat_var_offsets[var_no]; + ulonglong res= 0; + DBUG_ENTER("s_get_key_cache_stat_value"); + + if (var_no < 3) + res= (ulonglong) (*(long *) ((char *) keycache + var_ofs)); + else + res= *(ulonglong *) ((char *) keycache + var_ofs); + + DBUG_RETURN(res); +} + + +/* + The array of pointer to the key cache interface functions used for simple + key caches. Any simple key cache objects including those incorporated into + partitioned keys caches exploit this array. + + The current implementation of these functions allows to call them from + the MySQL server code directly. We don't do it though. +*/ + +static KEY_CACHE_FUNCS s_key_cache_funcs = +{ + s_init_key_cache, + s_resize_key_cache, + s_change_key_cache_param, + s_key_cache_read, + s_key_cache_insert, + s_key_cache_write, + s_flush_key_blocks, + s_reset_key_cache_counters, + s_end_key_cache, + s_get_key_cache_statistics, + s_get_key_cache_stat_value +}; + + +/****************************************************************************** + Partitioned Key Cache Module + + The module contains implementations of all key cache interface functions + employed by partitioned key caches. + + A partitioned key cache is a collection of structures for simple key caches + called key cache partitions. Any page from a file can be placed into a buffer + of only one partition. The number of the partition is calculated from + the file number and the position of the page in the file, and it's always the + same for the page. The function that maps pages into partitions takes care + of even distribution of pages among partitions. + + Partition key cache mitigate one of the major problem of simple key cache: + thread contention for key cache lock (mutex). Every call of a key cache + interface function must acquire this lock. So threads compete for this lock + even in the case when they have acquired shared locks for the file and + pages they want read from are in the key cache buffers. + When working with a partitioned key cache any key cache interface function + that needs only one page has to acquire the key cache lock only for the + partition the page is ascribed to. This makes the chances for threads not + compete for the same key cache lock better. Unfortunately if we use a + partitioned key cache with N partitions for B-tree indexes we can't say + that the chances becomes N times less. The fact is that any index lookup + operation requires reading from the root page that, for any index, is always + ascribed to the same partition. To resolve this problem we should have + employed more sophisticated mechanisms of working with root pages. + + Currently the number of partitions in a partitioned key cache is limited + by 64. We could increase this limit. Simultaneously we would have to increase + accordingly the size of the bitmap dirty_part_map from the MYISAM_SHARE + structure. + +******************************************************************************/ + +/* Control block for a partitioned key cache */ + +typedef struct st_p_key_cache_cb +{ + my_bool key_cache_inited; /*<=> control block is allocated */ + S_KEY_CACHE_CB **partition_array; /* array of the key cache partitions */ + uint partitions; /* number of partitions in the key cache */ + size_t key_cache_mem_size; /* specified size of the cache memory */ + uint key_cache_block_size; /* size of the page buffer of a cache block */ +} P_KEY_CACHE_CB; + +static +void p_end_key_cache(void *keycache_cb, my_bool cleanup); + +/* + Determine the partition to which the index block to read is ascribed + + SYNOPSIS + get_key_cache_partition() + keycache pointer to the control block of a partitioned key cache + file handler for the file for the block of data to be read + filepos position of the block of data in the file + + DESCRIPTION + The function determines the number of the partition in whose buffer the + block from 'file' at the position filepos has to be placed for reading. + The function returns the control block of the simple key cache for this + partition to the caller. + + RETURN VALUE + The pointer to the control block of the partition to which the specified + file block is ascribed. +*/ + +static +S_KEY_CACHE_CB *get_key_cache_partition(P_KEY_CACHE_CB *keycache, + File file, my_off_t filepos) +{ + uint i= KEYCACHE_BASE_EXPR( file, filepos) % keycache->partitions; + return keycache->partition_array[i]; +} + + +/* + Determine the partition to which the index block to write is ascribed + + SYNOPSIS + get_key_cache_partition() + keycache pointer to the control block of a partitioned key cache + file handler for the file for the block of data to be read + filepos position of the block of data in the file + dirty_part_map pointer to the bitmap of dirty partitions for the file + + DESCRIPTION + The function determines the number of the partition in whose buffer the + block from 'file' at the position filepos has to be placed for writing and + marks the partition as dirty in the dirty_part_map bitmap. + The function returns the control block of the simple key cache for this + partition to the caller. + + RETURN VALUE + The pointer to the control block of the partition to which the specified + file block is ascribed. +*/ + +static +S_KEY_CACHE_CB *get_key_cache_partition_for_write(P_KEY_CACHE_CB *keycache, + File file, my_off_t filepos, + ulonglong* dirty_part_map) +{ + uint i= KEYCACHE_BASE_EXPR( file, filepos) % keycache->partitions; + *dirty_part_map|= 1<<i; + return keycache->partition_array[i]; +} + + +/* + Initialize a partitioned key cache + + SYNOPSIS + p_init_key_cache() + keycache_cb pointer to the control block of a partitioned key cache + key_cache_block_size size of blocks to keep cached data + use_mem total memory to use for all key cache partitions + division_limit division limit (may be zero) + age_threshold age threshold (may be zero) + + DESCRIPTION + This function is the implementation of the init_key_cache interface function + that is employed by partitioned key caches. + The function builds and initializes an array of simple key caches, and then + initializes the control block structure of the type P_KEY_CACHE_CB that is + used for a partitioned key cache. The parameter keycache_cb is supposed to + point to this structure. The number of partitions in the partitioned key + cache to be built must be passed through the field 'partitions' of this + structure. The parameter key_cache_block_size specifies the size of the + blocks in the the simple key caches to be built. The parameters + division_limit and age_threshold determine the initial values of those + characteristics of the simple key caches that are used for midpoint + insertion strategy. The parameter use_mem specifies the total amount of + memory to be allocated for the key cache blocks in all simple key caches + and for all auxiliary structures. + + RETURN VALUE + total number of blocks in key cache partitions, if successful, + <= 0 - otherwise. + + NOTES + If keycache->key_cache_inited != 0 then we assume that the memory for + the array of partitions has been already allocated. + + It's assumed that no two threads call this function simultaneously + referring to the same key cache handle. +*/ + +static +int p_init_key_cache(void *keycache_cb, uint key_cache_block_size, + size_t use_mem, uint division_limit, + uint age_threshold) +{ + int i; + size_t mem_per_cache; + int cnt; + S_KEY_CACHE_CB *partition; + S_KEY_CACHE_CB **partition_ptr; + P_KEY_CACHE_CB *keycache= (P_KEY_CACHE_CB *) keycache_cb; + uint partitions= keycache->partitions; + int blocks= -1; + DBUG_ENTER("p_init_key_cache"); + + keycache->key_cache_block_size = key_cache_block_size; + + if (keycache->key_cache_inited) + partition_ptr= keycache->partition_array; + else + { + if(!(partition_ptr= + (S_KEY_CACHE_CB **) my_malloc(sizeof(S_KEY_CACHE_CB *) * partitions, + MYF(0)))) + DBUG_RETURN(blocks); + keycache->partition_array= partition_ptr; + } + + mem_per_cache = use_mem / partitions; + + for (i= 0; i < (int) partitions; i++) + { + my_bool key_cache_inited= keycache->key_cache_inited; + if (key_cache_inited) + partition= *partition_ptr; + else + { + if (!(partition= (S_KEY_CACHE_CB *) my_malloc(sizeof(S_KEY_CACHE_CB), + MYF(0)))) + continue; + partition->key_cache_inited= 0; + } + + if ((cnt= s_init_key_cache(partition, + key_cache_block_size, mem_per_cache, + division_limit, age_threshold)) <= 0) + { + s_end_key_cache(partition, 1); + my_free((uchar *) partition, MYF(0)); + partition= 0; + if (key_cache_inited) + { + memmove(partition_ptr, partition_ptr+1, + sizeof(partition_ptr)*(partitions-i-1)); + } + if (i == 0) + { + i--; + partitions--; + if (partitions) + mem_per_cache = use_mem / partitions; + } + continue; + } + + if (blocks < 0) + blocks= 0; + blocks+= cnt; + *partition_ptr++= partition; + } + + keycache->partitions= partitions= partition_ptr-keycache->partition_array; + keycache->key_cache_mem_size= mem_per_cache * partitions; + for (i= 0; i < (int) partitions; i++) + keycache->partition_array[i]->hash_factor= partitions; + + keycache->key_cache_inited= 1; + + DBUG_RETURN(blocks); +} + + +/* + Resize a partitioned key cache + + SYNOPSIS + p_resize_key_cache() + keycache_cb pointer to the control block of a partitioned key cache + key_cache_block_size size of blocks to keep cached data + use_mem total memory to use for the new key cache + division_limit new division limit (if not zero) + age_threshold new age threshold (if not zero) + + DESCRIPTION + This function is the implementation of the resize_key_cache interface + function that is employed by partitioned key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type P_KEY_CACHE_CB for the partitioned + key cache to be resized. + The parameter key_cache_block_size specifies the new size of the blocks in + the simple key caches that comprise the partitioned key cache. + The parameters division_limit and age_threshold determine the new initial + values of those characteristics of the simple key cache that are used for + midpoint insertion strategy. The parameter use-mem specifies the total + amount of memory to be allocated for the key cache blocks in all new + simple key caches and for all auxiliary structures. + + RETURN VALUE + number of blocks in the key cache, if successful, + 0 - otherwise. + + NOTES. + The function first calls s_prepare_resize_key_cache for each simple + key cache effectively flushing all dirty pages from it and destroying + the key cache. Then p_init_key cache is called. This call builds all + the new array of simple key caches containing the same number of + elements as the old one. After this the function calls the function + s_finish_resize_key_cache for each simple key cache from this array. + + This implementation doesn't block the calls and executions of other + functions from the key cache interface. However it assumes that the + calls of s_resize_key_cache itself are serialized. + +*/ + +static +int p_resize_key_cache(void *keycache_cb, uint key_cache_block_size, + size_t use_mem, uint division_limit, + uint age_threshold) +{ + uint i; + P_KEY_CACHE_CB *keycache= (P_KEY_CACHE_CB *) keycache_cb; + uint partitions= keycache->partitions; + my_bool cleanup= use_mem == 0; + int blocks= -1; + int err= 0; + DBUG_ENTER("p_resize_key_cache"); + if (use_mem == 0) + { + p_end_key_cache(keycache_cb, 0); + DBUG_RETURN(blocks); + } + for (i= 0; i < partitions; i++) + { + err|= s_prepare_resize_key_cache(keycache->partition_array[i], 0, 1); + } + if (!err && use_mem) + blocks= p_init_key_cache(keycache_cb, key_cache_block_size, use_mem, + division_limit, age_threshold); + if (blocks > 0 && !cleanup) + { + for (i= 0; i < partitions; i++) + { + s_finish_resize_key_cache(keycache->partition_array[i], 0, 1); + } + } + DBUG_RETURN(blocks); +} + + +/* + Change key cache parameters of a partitioned key cache + + SYNOPSIS + p_change_key_cache_param() + keycache_cb pointer to the control block of a partitioned key cache + division_limit new division limit (if not zero) + age_threshold new age threshold (if not zero) + + DESCRIPTION + This function is the implementation of the change_key_cache_param interface + function that is employed by partitioned key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type P_KEY_CACHE_CB for the simple key + cache where new values of the division limit and the age threshold used + for midpoint insertion strategy are to be set. The parameters + division_limit and age_threshold provide these new values. + + RETURN VALUE + none + + NOTES + The function just calls s_change_key_cache_param for each element from the + array of simple caches that comprise the partitioned key cache. + +*/ + +static +void p_change_key_cache_param(void *keycache_cb, uint division_limit, + uint age_threshold) +{ + uint i; + P_KEY_CACHE_CB *keycache= (P_KEY_CACHE_CB *) keycache_cb; + uint partitions= keycache->partitions; + DBUG_ENTER("p_change_key_cache_param"); + for (i= 0; i < partitions; i++) + { + s_change_key_cache_param(keycache->partition_array[i], division_limit, + age_threshold); + } + DBUG_VOID_RETURN; +} + + +/* + Destroy a partitioned key cache + + SYNOPSIS + p_end_key_cache() + keycache_cb pointer to the control block of a partitioned key cache + cleanup <=> complete free (free also control block structures + for all simple key caches) + + DESCRIPTION + This function is the implementation of the end_key_cache interface + function that is employed by partitioned key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type P_KEY_CACHE_CB for the partitioned + key cache to be destroyed. + The function frees the memory allocated for the cache blocks and + auxiliary structures used by simple key caches that comprise the + partitioned key cache. If the value of the parameter cleanup is TRUE + then even the memory used for control blocks of the simple key caches + and the array of pointers to them are freed. + + RETURN VALUE + none + +*/ + +static +void p_end_key_cache(void *keycache_cb, my_bool cleanup) +{ + uint i; + P_KEY_CACHE_CB *keycache= (P_KEY_CACHE_CB *) keycache_cb; + uint partitions= keycache->partitions; + DBUG_ENTER("p_end_key_cache"); + DBUG_PRINT("enter", ("key_cache: 0x%lx", (long) keycache)); + + for (i= 0; i < partitions; i++) + { + s_end_key_cache(keycache->partition_array[i], cleanup); + } + if (cleanup) { + for (i= 0; i < partitions; i++) + my_free((uchar*) keycache->partition_array[i], MYF(0)); + my_free((uchar*) keycache->partition_array, MYF(0)); + keycache->key_cache_inited= 0; + } + DBUG_VOID_RETURN; +} + + +/* + Read a block of data from a partitioned key cache into a buffer + + SYNOPSIS + + p_key_cache_read() + keycache_cb pointer to the control block of a partitioned key cache + file handler for the file for the block of data to be read + filepos position of the block of data in the file + level determines the weight of the data + buff buffer to where the data must be placed + length length of the buffer + block_length length of the read data from a key cache block + return_buffer return pointer to the key cache buffer with the data + + DESCRIPTION + This function is the implementation of the key_cache_read interface + function that is employed by partitioned key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type P_KEY_CACHE_CB for a partitioned + key cache. + In a general case the function reads a block of data from the key cache + into the buffer buff of the size specified by the parameter length. The + beginning of the block of data to be read is specified by the parameters + file and filepos. The length of the read data is the same as the length + of the buffer. The data is read into the buffer in key_cache_block_size + increments. To read each portion the function first finds out in what + partition of the key cache this portion(page) is to be saved, and calls + s_key_cache_read with the pointer to the corresponding simple key as + its first parameter. + If the parameter return_buffer is not ignored and its value is TRUE, and + the data to be read of the specified size block_length can be read from one + key cache buffer, then the function returns a pointer to the data in the + key cache buffer. + The function takes into account parameters block_length and return buffer + only in a single-threaded environment. + The parameter 'level' is used only by the midpoint insertion strategy + when the data or its portion cannot be found in the key cache. + + RETURN VALUE + Returns address from where the data is placed if successful, 0 - otherwise. + +*/ + +static +uchar *p_key_cache_read(void *keycache_cb, + File file, my_off_t filepos, int level, + uchar *buff, uint length, + uint block_length __attribute__((unused)), + int return_buffer __attribute__((unused))) +{ + uint r_length; + P_KEY_CACHE_CB *keycache= (P_KEY_CACHE_CB *) keycache_cb; + uint offset= (uint) (filepos % keycache->key_cache_block_size); + uchar *start= buff; + DBUG_ENTER("p_key_cache_read"); + DBUG_PRINT("enter", ("fd: %u pos: %lu length: %u", + (uint) file, (ulong) filepos, length)); + +#ifndef THREAD + if (block_length > keycache->key_cache_block_size || offset) + return_buffer=0; +#endif + + /* Read data in key_cache_block_size increments */ + do + { + S_KEY_CACHE_CB *partition= get_key_cache_partition(keycache, + file, filepos); + uchar *ret_buff= 0; + r_length= length; + set_if_smaller(r_length, keycache->key_cache_block_size - offset); + ret_buff= s_key_cache_read((void *) partition, + file, filepos, level, + buff, r_length, + block_length, return_buffer); + if (ret_buff == 0) + DBUG_RETURN(0); +#ifndef THREAD + /* This is only true if we were able to read everything in one block */ + if (return_buffer) + DBUG_RETURN(ret_buff); +#endif + filepos+= r_length; + buff+= r_length; + offset= 0; + } while ((length-= r_length)); + + DBUG_RETURN(start); +} + + +/* + Insert a block of file data from a buffer into a partitioned key cache + + SYNOPSIS + p_key_cache_insert() + keycache_cb pointer to the control block of a partitioned key cache + file handler for the file to insert data from + filepos position of the block of data in the file to insert + level determines the weight of the data + buff buffer to read data from + length length of the data in the buffer + + DESCRIPTION + This function is the implementation of the key_cache_insert interface + function that is employed by partitioned key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type P_KEY_CACHE_CB for a partitioned key + cache. + The function writes a block of file data from a buffer into the key cache. + The buffer is specified with the parameters buff and length - the pointer + to the beginning of the buffer and its size respectively. It's assumed + that the buffer contains the data from 'file' allocated from the position + filepos. The data is copied from the buffer in key_cache_block_size + increments. For every portion of data the function finds out in what simple + key cache from the array of partitions the data must be stored, and after + this calls s_key_cache_insert to copy the data into a key buffer of this + simple key cache. + The parameter level is used to set one characteristic for the key buffers + loaded with the data from buff. The characteristic is used only by the + midpoint insertion strategy. + + RETURN VALUE + 0 if a success, 1 - otherwise. + + NOTES + The function is used by MyISAM to move all blocks from a index file to + the key cache. It can be performed in parallel with reading the file data + from the key buffers by other threads. + +*/ + +static +int p_key_cache_insert(void *keycache_cb, + File file, my_off_t filepos, int level, + uchar *buff, uint length) +{ + uint w_length; + P_KEY_CACHE_CB *keycache= (P_KEY_CACHE_CB *) keycache_cb; + uint offset= (uint) (filepos % keycache->key_cache_block_size); + DBUG_ENTER("p_key_cache_insert"); + DBUG_PRINT("enter", ("fd: %u pos: %lu length: %u", + (uint) file,(ulong) filepos, length)); + + + /* Write data in key_cache_block_size increments */ + do + { + S_KEY_CACHE_CB *partition= get_key_cache_partition(keycache, + file, filepos); + w_length= length; + set_if_smaller(w_length, keycache->key_cache_block_size); + if (s_key_cache_insert((void *) partition, + file, filepos, level, + buff, w_length)) + DBUG_RETURN(1); + + filepos+= w_length; + buff+= w_length; + offset = 0; + } while ((length-= w_length)); + + DBUG_RETURN(0); +} + + +/* + Write data from a buffer into a partitioned key cache + + SYNOPSIS + + p_key_cache_write() + keycache_cb pointer to the control block of a partitioned key cache + file handler for the file to write data to + filepos position in the file to write data to + level determines the weight of the data + buff buffer with the data + length length of the buffer + dont_write if is 0 then all dirty pages involved in writing + should have been flushed from key cache + file_extra maps of key cache partitions containing + dirty pages from file + + DESCRIPTION + This function is the implementation of the key_cache_write interface + function that is employed by partitioned key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type P_KEY_CACHE_CB for a partitioned + key cache. + In a general case the function copies data from a buffer into the key + cache. The buffer is specified with the parameters buff and length - + the pointer to the beginning of the buffer and its size respectively. + It's assumed the buffer contains the data to be written into 'file' + starting from the position filepos. The data is copied from the buffer + in key_cache_block_size increments. For every portion of data the + function finds out in what simple key cache from the array of partitions + the data must be stored, and after this calls s_key_cache_write to copy + the data into a key buffer of this simple key cache. + If the value of the parameter dont_write is FALSE then the function + also writes the data into file. + The parameter level is used to set one characteristic for the key buffers + filled with the data from buff. The characteristic is employed only by + the midpoint insertion strategy. + The parameter file_expra provides a pointer to the shared bitmap of + the partitions that may contains dirty pages for the file. This bitmap + is used to optimize the function p_flush_key_blocks. + + RETURN VALUE + 0 if a success, 1 - otherwise. + + NOTES + This implementation exploits the fact that the function is called only + when a thread has got an exclusive lock for the key file. + +*/ + +static +int p_key_cache_write(void *keycache_cb, + File file, void *file_extra, + my_off_t filepos, int level, + uchar *buff, uint length, + uint block_length __attribute__((unused)), + int dont_write) +{ + uint w_length; + ulonglong *part_map= (ulonglong *) file_extra; + P_KEY_CACHE_CB *keycache= (P_KEY_CACHE_CB *) keycache_cb; + uint offset= (uint) (filepos % keycache->key_cache_block_size); + DBUG_ENTER("p_key_cache_write"); + DBUG_PRINT("enter", + ("fd: %u pos: %lu length: %u block_length: %u" + " key_block_length: %u", + (uint) file, (ulong) filepos, length, block_length, + keycache ? keycache->key_cache_block_size : 0)); + + + /* Write data in key_cache_block_size increments */ + do + { + S_KEY_CACHE_CB *partition= get_key_cache_partition_for_write(keycache, + file, filepos, + part_map); + w_length = length; + set_if_smaller(w_length, keycache->key_cache_block_size ); + if (s_key_cache_write(partition, + file, 0, filepos, level, + buff, w_length, block_length, + dont_write)) + DBUG_RETURN(1); + + filepos+= w_length; + buff+= w_length; + offset= 0; + } while ((length-= w_length)); + + DBUG_RETURN(0); +} + + +/* + Flush all blocks for a file from key buffers of a partitioned key cache + + SYNOPSIS + + p_flush_key_blocks() + keycache_cb pointer to the control block of a partitioned key cache + file handler for the file to flush to + file_extra maps of key cache partitions containing + dirty pages from file (not used) + flush_type type of the flush operation + + DESCRIPTION + This function is the implementation of the flush_key_blocks interface + function that is employed by partitioned key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type P_KEY_CACHE_CB for a partitioned + key cache. + In a general case the function flushes the data from all dirty key + buffers related to the file 'file' into this file. The function does + exactly this if the value of the parameter type is FLUSH_KEEP. If the + value of this parameter is FLUSH_RELEASE, the function additionally + releases the key buffers containing data from 'file' for new usage. + If the value of the parameter type is FLUSH_IGNORE_CHANGED the function + just releases the key buffers containing data from 'file'. + The function performs the operation by calling s_flush_key_blocks + for the elements of the array of the simple key caches that comprise + the partitioned key_cache. If the value of the parameter type is + FLUSH_KEEP s_flush_key_blocks is called only for the partitions with + possibly dirty pages marked in the bitmap pointed to by the parameter + file_extra. + + RETURN + 0 ok + 1 error + + NOTES + This implementation exploits the fact that the function is called only + when a thread has got an exclusive lock for the key file. + +*/ + +static +int p_flush_key_blocks(void *keycache_cb, + File file, void *file_extra, + enum flush_type type) +{ + uint i; + P_KEY_CACHE_CB *keycache= (P_KEY_CACHE_CB *) keycache_cb; + uint partitions= keycache->partitions; + int err= 0; + ulonglong *dirty_part_map= (ulonglong *) file_extra; + DBUG_ENTER("p_flush_key_blocks"); + DBUG_PRINT("enter", ("keycache: 0x%lx", (long) keycache)); + + for (i= 0; i < partitions; i++) + { + S_KEY_CACHE_CB *partition= keycache->partition_array[i]; + if ((type == FLUSH_KEEP || type == FLUSH_FORCE_WRITE) && + !((*dirty_part_map) & (1<<i))) + continue; + err+= test(s_flush_key_blocks(partition, file, 0, type)); + } + *dirty_part_map= 0; + + if (err > 0) + err= 1; + + DBUG_RETURN(err); +} + + +/* + Reset the counters of a partitioned key cache + + SYNOPSIS + p_reset_key_cache_counters() + name the name of a key cache + keycache_cb pointer to the control block of a partitioned key cache + + DESCRIPTION + This function is the implementation of the reset_key_cache_counters + interface function that is employed by partitioned key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type P_KEY_CACHE_CB for a partitioned + key cache. + This function resets the values of the statistical counters of the simple + key caches comprising partitioned key cache to 0. It does it by calling + s_reset_key_cache_counters for each key cache partition. + The parameter name is currently not used. + + RETURN + 0 on success (always because it can't fail) + +*/ + +static +int p_reset_key_cache_counters(const char *name __attribute__((unused)), + void *keycache_cb) +{ + uint i; + P_KEY_CACHE_CB *keycache= (P_KEY_CACHE_CB *) keycache_cb; + uint partitions= keycache->partitions; + DBUG_ENTER("p_reset_key_cache_counters"); + + for (i = 0; i < partitions; i++) + { + s_reset_key_cache_counters(name, keycache->partition_array[i]); + } + DBUG_RETURN(0); +} + + +/* + Get statistics for a partition key cache + + SYNOPSIS + p_get_key_cache_statistics() + keycache_cb pointer to the control block of a partitioned key cache + partition_no partition number to get statistics for + key_cache_stats OUT pointer to the structure for the returned statistics + + DESCRIPTION + This function is the implementation of the get_key_cache_statistics + interface function that is employed by partitioned key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type P_KEY_CACHE_CB for a partitioned + key cache. + If the value of the parameter partition_no is equal to 0 then aggregated + statistics for all partitions is returned in the fields of the + structure key_cache_stat of the type KEY_CACHE_STATISTICS . Otherwise + the function returns data for the partition number partition_no of the + key cache in the structure key_cache_stat. (Here partitions are numbered + starting from 1.) + + RETURN + none + +*/ + +static +void p_get_key_cache_statistics(void *keycache_cb, uint partition_no, + KEY_CACHE_STATISTICS *key_cache_stats) +{ + uint i; + S_KEY_CACHE_CB *partition; + P_KEY_CACHE_CB *keycache= (P_KEY_CACHE_CB *) keycache_cb; + uint partitions= keycache->partitions; + DBUG_ENTER("p_get_key_cache_statistics_"); + + if (partition_no != 0) + { + partition= keycache->partition_array[partition_no-1]; + s_get_key_cache_statistics((void *) partition, 0, key_cache_stats); + DBUG_VOID_RETURN; + } + key_cache_stats->mem_size= (longlong) keycache->key_cache_mem_size; + key_cache_stats->block_size= (longlong) keycache->key_cache_block_size; + for (i = 0; i < partitions; i++) + { + partition= keycache->partition_array[i]; + key_cache_stats->blocks_used+= partition->blocks_used; + key_cache_stats->blocks_unused+= partition->blocks_unused; + key_cache_stats->blocks_changed+= partition->global_blocks_changed; + key_cache_stats->read_requests+= partition->global_cache_r_requests; + key_cache_stats->reads+= partition->global_cache_read; + key_cache_stats->write_requests+= partition->global_cache_w_requests; + key_cache_stats->writes+= partition->global_cache_write; + } + DBUG_VOID_RETURN; +} + +/* + Get the value of a statistical variable for a partitioned key cache + + SYNOPSIS + p_get_key_cache_stat_value() + keycache_cb pointer to the control block of a partitioned key cache + var_no the ordered number of a statistical variable + + DESCRIPTION + This function is the implementation of the get_key_cache_stat_value + interface function that is employed by partitioned key caches. + The function considers the parameter keycache_cb as a pointer to the + control block structure of the type P_KEY_CACHE_CB for a partitioned + key cache. + This function returns the value of the statistical variable var_no + for this key cache. The variables are numbered starting from 0 to 6. + The returned value is calculated as the sum of the values of the + statistical variable with number var_no for all simple key caches that + comprise the partitioned key cache. + + RETURN + The value of the specified statistical variable + +*/ + +static +ulonglong p_get_key_cache_stat_value(void *keycache_cb, uint var_no) +{ + uint i; + P_KEY_CACHE_CB *keycache= (P_KEY_CACHE_CB *) keycache_cb; + uint partitions= keycache->partitions; + size_t var_ofs= s_key_cache_stat_var_offsets[var_no]; + ulonglong res= 0; + DBUG_ENTER("p_get_key_cache_stat_value"); + + if (var_no < 3) + { + for (i = 0; i < partitions; i++) + { + S_KEY_CACHE_CB *partition= keycache->partition_array[i]; + res+= (ulonglong) (*(long *) ((char *) partition + var_ofs)); + } + } + else + { + for (i = 0; i < partitions; i++) + { + S_KEY_CACHE_CB *partition= keycache->partition_array[i]; + res+= *(ulonglong *) ((char *) partition + var_ofs); + } + } + DBUG_RETURN(res); +} + + +/* + The array of pointers to the key cache interface functions used by + partitioned key caches. Any partitioned key cache object caches exploits + this array. + + The current implementation of these functions does not allow to call + them from the MySQL server code directly. The key cache interface + wrappers must be used for this purpose. +*/ + +static KEY_CACHE_FUNCS p_key_cache_funcs = +{ + p_init_key_cache, + p_resize_key_cache, + p_change_key_cache_param, + p_key_cache_read, + p_key_cache_insert, + p_key_cache_write, + p_flush_key_blocks, + p_reset_key_cache_counters, + p_end_key_cache, + p_get_key_cache_statistics, + p_get_key_cache_stat_value +}; + + +/****************************************************************************** + Key Cache Interface Module + + The module contains wrappers for all key cache interface functions. + + Currently there are key caches of two types: simple key caches and + partitioned key caches. Each type (class) has its own implementation of the + basic key cache operations used the MyISAM storage engine. The pointers + to the implementation functions are stored in two static structures of the + type KEY_CACHE_FUNC: s_key_cache_funcs - for simple key caches, and + p_key_cache_funcs - for partitioned key caches. When a key cache object is + created the constructor procedure init_key_cache places a pointer to the + corresponding table into one of its fields. The procedure also initializes + a control block for the key cache oject and saves the pointer to this + block in another field of the key cache object. + When a key cache wrapper function is invoked for a key cache object to + perform a basic key cache operation it looks into the interface table + associated with the key cache oject and calls the corresponding + implementation of the operation. It passes the saved key cache control + block to this implementation. If, for some reasons, the control block + has not been fully initialized yet, the wrapper function either does not + do anything or, in the case when it perform a read/write operation, the + function do it directly through the system i/o functions. + + As we can see the model with which the key cache interface is supported + as quite conventional for interfaces in general. + +******************************************************************************/ + + +/* + Initialize a key cache + + SYNOPSIS + init_key_cache() + keycache pointer to the key cache to be initialized + key_cache_block_size size of blocks to keep cached data + use_mem total memory to use for cache buffers/structures + division_limit division limit (may be zero) + age_threshold age threshold (may be zero) + partitions number of partitions in the key cache + + DESCRIPTION + The function creates a control block structure for a key cache and + places the pointer to this block in the structure keycache. + If the value of the parameter 'partitions' is 0 then a simple key cache + is created. Otherwise a partitioned key cache with the specified number + of partitions is created. + The parameter key_cache_block_size specifies the size of the blocks in + the key cache to be created. The parameters division_limit and + age_threshold determine the initial values of those characteristics of + the key cache that are used for midpoint insertion strategy. The parameter + use_mem specifies the total amount of memory to be allocated for the + key cache buffers and for all auxiliary structures. + + RETURN VALUE + total number of blocks in key cache partitions, if successful, + <= 0 - otherwise. + + NOTES + if keycache->key_cache_inited != 0 we assume that the memory + for the control block of the key cache has been already allocated. + + It's assumed that no two threads call this function simultaneously + referring to the same key cache handle. + +*/ + +int init_key_cache(KEY_CACHE *keycache, uint key_cache_block_size, + size_t use_mem, uint division_limit, + uint age_threshold, uint partitions) +{ + void *keycache_cb; + int blocks; + if (keycache->key_cache_inited) + keycache_cb= keycache->keycache_cb; + else + { + if (partitions == 0) + { + if (!(keycache_cb= (void *) my_malloc(sizeof(S_KEY_CACHE_CB), MYF(0)))) + return 0; + ((S_KEY_CACHE_CB *) keycache_cb)->key_cache_inited= 0; + keycache->key_cache_type= SIMPLE_KEY_CACHE; + keycache->interface_funcs= &s_key_cache_funcs; + } + else + { + if (!(keycache_cb= (void *) my_malloc(sizeof(P_KEY_CACHE_CB), MYF(0)))) + return 0; + ((P_KEY_CACHE_CB *) keycache_cb)->key_cache_inited= 0; + keycache->key_cache_type= PARTITIONED_KEY_CACHE; + keycache->interface_funcs= &p_key_cache_funcs; + } + keycache->keycache_cb= keycache_cb; + keycache->key_cache_inited= 1; + } + + if (partitions != 0) + { + ((P_KEY_CACHE_CB *) keycache_cb)->partitions= partitions; + } + keycache->can_be_used= 0; + blocks= keycache->interface_funcs->init(keycache_cb, key_cache_block_size, + use_mem, division_limit, + age_threshold); + keycache->partitions= partitions ? + ((P_KEY_CACHE_CB *) keycache_cb)->partitions : 0; + DBUG_ASSERT(partitions <= MAX_KEY_CACHE_PARTITIONS); + if (blocks > 0) + keycache->can_be_used= 1; + return blocks; +} + + +/* + Resize a key cache + + SYNOPSIS + resize_key_cache() + keycache pointer to the key cache to be resized + key_cache_block_size size of blocks to keep cached data + use_mem total memory to use for the new key cache + division_limit new division limit (if not zero) + age_threshold new age threshold (if not zero) + + DESCRIPTION + The function operates over the key cache key cache. + The parameter key_cache_block_size specifies the new size of the block + buffers in the key cache. The parameters division_limit and age_threshold + determine the new initial values of those characteristics of the key cache + that are used for midpoint insertion strategy. The parameter use_mem + specifies the total amount of memory to be allocated for the key cache + buffers and for all auxiliary structures. + + RETURN VALUE + number of blocks in the key cache, if successful, + 0 - otherwise. + + NOTES + The function does not block the calls and executions of other functions + from the key cache interface. However it assumes that the calls of + resize_key_cache itself are serialized. + + Currently the function is called when the values of the variables + key_buffer_size and/or key_cache_block_size are being reset for + the key cache keycache. + +*/ + +int resize_key_cache(KEY_CACHE *keycache, uint key_cache_block_size, + size_t use_mem, uint division_limit, uint age_threshold) +{ + int blocks= -1; + if (keycache->key_cache_inited) + { + if ((uint) keycache->param_partitions != keycache->partitions && use_mem) + blocks= repartition_key_cache (keycache, + key_cache_block_size, use_mem, + division_limit, age_threshold, + (uint) keycache->param_partitions); + else + { + blocks= keycache->interface_funcs->resize(keycache->keycache_cb, + key_cache_block_size, + use_mem, division_limit, + age_threshold); + + if (keycache->partitions) + keycache->partitions= + ((P_KEY_CACHE_CB *)(keycache->keycache_cb))->partitions; + } + if (blocks <= 0) + keycache->can_be_used= 0; + } + return blocks; +} + + +/* + Change key cache parameters of a key cache + + SYNOPSIS + change_key_cache_param() + keycache pointer to the key cache to change parameters for + division_limit new division limit (if not zero) + age_threshold new age threshold (if not zero) + + DESCRIPTION + The function sets new values of the division limit and the age threshold + used when the key cache keycach employs midpoint insertion strategy. + The parameters division_limit and age_threshold provide these new values. + + RETURN VALUE + none + + NOTES + Currently the function is called when the values of the variables + key_cache_division_limit and/or key_cache_age_threshold are being reset + for the key cache keycache. + +*/ + +void change_key_cache_param(KEY_CACHE *keycache, uint division_limit, + uint age_threshold) +{ + if (keycache->key_cache_inited) + { + + keycache->interface_funcs->change_param(keycache->keycache_cb, + division_limit, + age_threshold); + } +} + + +/* + Destroy a key cache + + SYNOPSIS + end_key_cache() + keycache pointer to the key cache to be destroyed + cleanup <=> complete free + + DESCRIPTION + The function frees the memory allocated for the cache blocks and + auxiliary structures used by the key cache keycache. If the value + of the parameter cleanup is TRUE then all resources used by the key + cache are to be freed. + + RETURN VALUE + none +*/ + +void end_key_cache(KEY_CACHE *keycache, my_bool cleanup) +{ + if (keycache->key_cache_inited) + { + keycache->interface_funcs->end(keycache->keycache_cb, cleanup); + if (cleanup) + { + if (keycache->keycache_cb) + { + my_free((uchar *) keycache->keycache_cb, MYF(0)); + keycache->keycache_cb= 0; + } + keycache->key_cache_inited= 0; + } + keycache->can_be_used= 0; + } +} + + +/* + Read a block of data from a key cache into a buffer + + SYNOPSIS + + key_cache_read() + keycache pointer to the key cache to read data from + file handler for the file for the block of data to be read + filepos position of the block of data in the file + level determines the weight of the data + buff buffer to where the data must be placed + length length of the buffer + block_length length of the data read from a key cache block + return_buffer return pointer to the key cache buffer with the data + + DESCRIPTION + The function operates over buffers of the key cache keycache. + In a general case the function reads a block of data from the key cache + into the buffer buff of the size specified by the parameter length. The + beginning of the block of data to be read is specified by the parameters + file and filepos. The length of the read data is the same as the length + of the buffer. + If the parameter return_buffer is not ignored and its value is TRUE, and + the data to be read of the specified size block_length can be read from one + key cache buffer, then the function returns a pointer to the data in the + key cache buffer. + The parameter 'level' is used only by the midpoint insertion strategy + when the data or its portion cannot be found in the key cache. + The function reads data into the buffer directly from file if the control + block of the key cache has not been initialized yet. + + RETURN VALUE + Returns address from where the data is placed if successful, 0 - otherwise. + + NOTES. + Filepos must be a multiple of 'block_length', but it doesn't + have to be a multiple of key_cache_block_size; +*/ + +uchar *key_cache_read(KEY_CACHE *keycache, + File file, my_off_t filepos, int level, + uchar *buff, uint length, + uint block_length, int return_buffer) +{ + if (keycache->key_cache_inited && keycache->can_be_used) + return keycache->interface_funcs->read(keycache->keycache_cb, + file, filepos, level, + buff, length, + block_length, return_buffer); + + /* We can't use mutex here as the key cache may not be initialized */ + keycache->global_cache_r_requests++; + keycache->global_cache_read++; + + if (my_pread(file, (uchar*) buff, length, filepos, MYF(MY_NABP))) + return (uchar *) 0; + + return buff; +} + + +/* + Insert a block of file data from a buffer into a key cache + + SYNOPSIS + key_cache_insert() + keycache pointer to the key cache to insert data into + file handler for the file to insert data from + filepos position of the block of data in the file to insert + level determines the weight of the data + buff buffer to read data from + length length of the data in the buffer + + DESCRIPTION + The function operates over buffers of the key cache keycache. + The function writes a block of file data from a buffer into the key cache. + The buffer is specified with the parameters buff and length - the pointer + to the beginning of the buffer and its size respectively. It's assumed + that the buffer contains the data from 'file' allocated from the position + filepos. + The parameter level is used to set one characteristic for the key buffers + loaded with the data from buff. The characteristic is used only by the + midpoint insertion strategy. + + RETURN VALUE + 0 if a success, 1 - otherwise. + + NOTES + The function is used by MyISAM to move all blocks from a index file to + the key cache. + It is assumed that it may be performed in parallel with reading the file + data from the key buffers by other threads. + +*/ + +int key_cache_insert(KEY_CACHE *keycache, + File file, my_off_t filepos, int level, + uchar *buff, uint length) +{ + if (keycache->key_cache_inited && keycache->can_be_used) + return keycache->interface_funcs->insert(keycache->keycache_cb, + file, filepos, level, + buff, length); + return 0; +} + + +/* + Write data from a buffer into a key cache + + SYNOPSIS + + key_cache_write() + keycache pointer to the key cache to write data to + file handler for the file to write data to + filepos position in the file to write data to + level determines the weight of the data + buff buffer with the data + length length of the buffer + dont_write if is 0 then all dirty pages involved in writing + should have been flushed from key cache + file_extra pointer to optional file attributes + + DESCRIPTION + The function operates over buffers of the key cache keycache. + In a general case the function writes data from a buffer into the key + cache. The buffer is specified with the parameters buff and length - + the pointer to the beginning of the buffer and its size respectively. + It's assumed the buffer contains the data to be written into 'file' + starting from the position filepos. + If the value of the parameter dont_write is FALSE then the function + also writes the data into file. + The parameter level is used to set one characteristic for the key buffers + filled with the data from buff. The characteristic is employed only by + the midpoint insertion strategy. + The parameter file_expra may point to additional file attributes used + for optimization or other purposes. + The function writes data from the buffer directly into file if the control + block of the key cache has not been initialized yet. + + RETURN VALUE + 0 if a success, 1 - otherwise. + + NOTES + This implementation may exploit the fact that the function is called only + when a thread has got an exclusive lock for the key file. + +*/ + +int key_cache_write(KEY_CACHE *keycache, + File file, void *file_extra, + my_off_t filepos, int level, + uchar *buff, uint length, + uint block_length, int force_write) +{ + if (keycache->key_cache_inited && keycache->can_be_used) + return keycache->interface_funcs->write(keycache->keycache_cb, + file, file_extra, + filepos, level, + buff, length, + block_length, force_write); + + /* We can't use mutex here as the key cache may not be initialized */ + keycache->global_cache_w_requests++; + keycache->global_cache_write++; + if (my_pwrite(file, buff, length, filepos, MYF(MY_NABP | MY_WAIT_IF_FULL))) + return 1; + + return 0; +} + + +/* + Flush all blocks for a file from key buffers of a key cache + + SYNOPSIS + + flush_key_blocks() + keycache pointer to the key cache whose blocks are to be flushed + file handler for the file to flush to + file_extra maps of key cache (used for partitioned key caches) + flush_type type of the flush operation + + DESCRIPTION + The function operates over buffers of the key cache keycache. + In a general case the function flushes the data from all dirty key + buffers related to the file 'file' into this file. The function does + exactly this if the value of the parameter type is FLUSH_KEEP. If the + value of this parameter is FLUSH_RELEASE, the function additionally + releases the key buffers containing data from 'file' for new usage. + If the value of the parameter type is FLUSH_IGNORE_CHANGED the function + just releases the key buffers containing data from 'file'. + If the value of the parameter type is FLUSH_KEEP the function may use + the value of the parameter file_extra pointing to possibly dirty + partitions to optimize the operation for partitioned key caches. + + RETURN + 0 ok + 1 error + + NOTES + Any implementation of the function may exploit the fact that the function + is called only when a thread has got an exclusive lock for the key file. + +*/ + +int flush_key_blocks(KEY_CACHE *keycache, + int file, void *file_extra, + enum flush_type type) +{ + if (keycache->key_cache_inited) + return keycache->interface_funcs->flush(keycache->keycache_cb, + file, file_extra, type); + return 0; +} + + +/* + Reset the counters of a key cache + + SYNOPSIS + reset_key_cache_counters() + name the name of a key cache (unused) + keycache pointer to the key cache for which to reset counters + + DESCRIPTION + This function resets the values of the statistical counters for the key + cache keycache. + The parameter name is currently not used. + + RETURN + 0 on success (always because it can't fail) + + NOTES + This procedure is used by process_key_caches() to reset the counters of all + currently used key caches, both the default one and the named ones. + +*/ + +int reset_key_cache_counters(const char *name __attribute__((unused)), + KEY_CACHE *keycache) +{ + if (keycache->key_cache_inited) + { + + return keycache->interface_funcs->reset_counters(name, + keycache->keycache_cb); + } + return 0; +} + + +/* + Get statistics for a key cache + + SYNOPSIS + get_key_cache_statistics() + keycache pointer to the key cache to get statistics for + partition_no partition number to get statistics for + key_cache_stats OUT pointer to the structure for the returned statistics + + DESCRIPTION + If the value of the parameter partition_no is equal to 0 then statistics + for the whole key cache keycache (aggregated statistics) is returned in the + fields of the structure key_cache_stat of the type KEY_CACHE_STATISTICS. + Otherwise the value of the parameter partition_no makes sense only for + a partitioned key cache. In this case the function returns statistics + for the partition with the specified number partition_no. + + RETURN + none + +*/ + +void get_key_cache_statistics(KEY_CACHE *keycache, uint partition_no, + KEY_CACHE_STATISTICS *key_cache_stats) +{ + bzero(key_cache_stats, sizeof(KEY_CACHE_STATISTICS)); + if (keycache->key_cache_inited) + { + keycache->interface_funcs->get_stats(keycache->keycache_cb, + partition_no, key_cache_stats); + } +} + + +/* + Get the value of a statistical variable for a key cache + + SYNOPSIS + get_key_cache_stat_value() + keycache pointer to the key cache to get statistics for + var_no the ordered number of a statistical variable + + DESCRIPTION + This function returns the value of the statistical variable var_no for + the key cache keycache. The variables are numbered starting from 0 to 6. + + RETURN + The value of the specified statistical variable. + + NOTES + Currently for any key cache the function can return values for the + following 7 statistical variables: + + Name Number + + blocks_used 0 + blocks_unused 1 + blocks_changed 2 + read_requests 3 + reads 4 + write_requests 5 + writes 6 + +*/ + +ulonglong get_key_cache_stat_value(KEY_CACHE *keycache, uint var_no) +{ + if (keycache->key_cache_inited) + { + return keycache->interface_funcs->get_stat_val(keycache->keycache_cb, + var_no); + } + else + return 0; +} + + +/* + Repartition a key cache + + SYNOPSIS + repartition_key_cache() + keycache pointer to the key cache to be repartitioned + key_cache_block_size size of blocks to keep cached data + use_mem total memory to use for the new key cache + division_limit new division limit (if not zero) + age_threshold new age threshold (if not zero) + partitions new number of partitions in the key cache + + DESCRIPTION + The function operates over the key cache keycache. + The parameter partitions specifies the number of partitions in the key + cache after repartitioning. If the value of this parameter is 0 then + a simple key cache must be created instead of the old one. + The parameter key_cache_block_size specifies the new size of the block + buffers in the key cache. The parameters division_limit and age_threshold + determine the new initial values of those characteristics of the key cache + that are used for midpoint insertion strategy. The parameter use_mem + specifies the total amount of memory to be allocated for the new key + cache buffers and for all auxiliary structures. + + RETURN VALUE + number of blocks in the key cache, if successful, + 0 - otherwise. + + NOTES + The function does not block the calls and executions of other functions + from the key cache interface. However it assumes that the calls of + resize_key_cache itself are serialized. + + Currently the function is called when the value of the variable + key_cache_partitions is being reset for the key cache keycache. + +*/ + +int repartition_key_cache(KEY_CACHE *keycache, uint key_cache_block_size, + size_t use_mem, uint division_limit, + uint age_threshold, uint partitions) +{ + uint blocks= -1; + if (keycache->key_cache_inited) + { + keycache->interface_funcs->resize(keycache->keycache_cb, + key_cache_block_size, 0, + division_limit, age_threshold); + end_key_cache(keycache, 1); + blocks= init_key_cache(keycache, key_cache_block_size, use_mem, + division_limit, age_threshold, partitions); + } + return blocks; +} + === modified file 'sql/handler.cc' --- a/sql/handler.cc 2010-02-01 06:14:12 +0000 +++ b/sql/handler.cc 2010-02-16 16:41:11 +0000 @@ -3828,11 +3828,13 @@ int ha_init_key_cache(const char *name, uint tmp_block_size= (uint) key_cache->param_block_size; uint division_limit= key_cache->param_division_limit; uint age_threshold= key_cache->param_age_threshold; + uint partitions= key_cache->param_partitions; pthread_mutex_unlock(&LOCK_global_system_variables); DBUG_RETURN(!init_key_cache(key_cache, tmp_block_size, tmp_buff_size, - division_limit, age_threshold)); + division_limit, age_threshold, + partitions)); } DBUG_RETURN(0); } @@ -3862,10 +3864,12 @@ int ha_resize_key_cache(KEY_CACHE *key_c /** - Change parameters for key cache (like size) + Change parameters for key cache (like division_limit) */ int ha_change_key_cache_param(KEY_CACHE *key_cache) { + DBUG_ENTER("ha_change_key_cache_param"); + if (key_cache->key_cache_inited) { pthread_mutex_lock(&LOCK_global_system_variables); @@ -3874,9 +3878,35 @@ int ha_change_key_cache_param(KEY_CACHE pthread_mutex_unlock(&LOCK_global_system_variables); change_key_cache_param(key_cache, division_limit, age_threshold); } - return 0; + DBUG_RETURN(0); } + +/** + Repartition key cache +*/ +int ha_repartition_key_cache(KEY_CACHE *key_cache) +{ + DBUG_ENTER("ha_repartition_key_cache"); + + if (key_cache->key_cache_inited) + { + pthread_mutex_lock(&LOCK_global_system_variables); + size_t tmp_buff_size= (size_t) key_cache->param_buff_size; + long tmp_block_size= (long) key_cache->param_block_size; + uint division_limit= key_cache->param_division_limit; + uint age_threshold= key_cache->param_age_threshold; + uint partitions= key_cache->param_partitions; + pthread_mutex_unlock(&LOCK_global_system_variables); + DBUG_RETURN(!repartition_key_cache(key_cache, tmp_block_size, + tmp_buff_size, + division_limit, age_threshold, + partitions)); + } + DBUG_RETURN(0); +} + + /** Free memory allocated by a key cache. */ === modified file 'sql/handler.h' --- a/sql/handler.h 2010-02-01 06:14:12 +0000 +++ b/sql/handler.h 2010-02-16 16:41:11 +0000 @@ -2188,6 +2188,7 @@ int ha_table_exists_in_engine(THD* thd, extern "C" int ha_init_key_cache(const char *name, KEY_CACHE *key_cache); int ha_resize_key_cache(KEY_CACHE *key_cache); int ha_change_key_cache_param(KEY_CACHE *key_cache); +int ha_repartition_key_cache(KEY_CACHE *key_cache); int ha_change_key_cache(KEY_CACHE *old_key_cache, KEY_CACHE *new_key_cache); int ha_end_key_cache(KEY_CACHE *key_cache); === modified file 'sql/mysqld.cc' --- a/sql/mysqld.cc 2010-02-12 08:47:31 +0000 +++ b/sql/mysqld.cc 2010-02-16 16:41:11 +0000 @@ -5825,6 +5825,7 @@ enum options_mysqld OPT_INTERACTIVE_TIMEOUT, OPT_JOIN_BUFF_SIZE, OPT_KEY_BUFFER_SIZE, OPT_KEY_CACHE_BLOCK_SIZE, OPT_KEY_CACHE_DIVISION_LIMIT, OPT_KEY_CACHE_AGE_THRESHOLD, + OPT_KEY_CACHE_PARTITIONS, OPT_LONG_QUERY_TIME, OPT_LOWER_CASE_TABLE_NAMES, OPT_MAX_ALLOWED_PACKET, OPT_MAX_BINLOG_CACHE_SIZE, OPT_MAX_BINLOG_SIZE, @@ -6915,6 +6916,12 @@ log and this option does nothing anymore (uchar**) 0, 0, (GET_ULONG | GET_ASK_ADDR) , REQUIRED_ARG, 100, 1, 100, 0, 1, 0}, + {"key_cache_partitions", OPT_KEY_CACHE_PARTITIONS, + "The number of partitions in key cache", + (uchar**) &dflt_key_cache_var.param_partitions, + (uchar**) 0, + 0, (GET_ULONG | GET_ASK_ADDR), REQUIRED_ARG, DEFAULT_KEY_CACHE_PARTITIONS, + 0, MAX_KEY_CACHE_PARTITIONS, 0, 1, 0}, {"log-slow-filter", OPT_LOG_SLOW_FILTER, "Log only the queries that followed certain execution plan. Multiple flags allowed in a comma-separated string. [admin, filesort, filesort_on_disk, full_join, full_scan, query_cache, query_cache_miss, tmp_table, tmp_table_on_disk]. Sets log-slow-admin-command to ON", 0, 0, 0, GET_STR, REQUIRED_ARG, 0, 0, 0, QPLAN_ALWAYS_SET, 0, 0}, @@ -8837,6 +8844,7 @@ mysql_getopt_value(const char *keyname, case OPT_KEY_CACHE_BLOCK_SIZE: case OPT_KEY_CACHE_DIVISION_LIMIT: case OPT_KEY_CACHE_AGE_THRESHOLD: + case OPT_KEY_CACHE_PARTITIONS: { KEY_CACHE *key_cache; if (!(key_cache= get_or_create_key_cache(keyname, key_length))) @@ -8854,6 +8862,8 @@ mysql_getopt_value(const char *keyname, return (uchar**) &key_cache->param_division_limit; case OPT_KEY_CACHE_AGE_THRESHOLD: return (uchar**) &key_cache->param_age_threshold; + case OPT_KEY_CACHE_PARTITIONS: + return (uchar**) &key_cache->param_partitions; } } } === modified file 'sql/set_var.cc' --- a/sql/set_var.cc 2010-02-01 06:14:12 +0000 +++ b/sql/set_var.cc 2010-02-16 16:41:11 +0000 @@ -317,15 +317,18 @@ static sys_var_thd_ulong sys_interactive static sys_var_thd_ulong sys_join_buffer_size(&vars, "join_buffer_size", &SV::join_buff_size); static sys_var_key_buffer_size sys_key_buffer_size(&vars, "key_buffer_size"); -static sys_var_key_cache_long sys_key_cache_block_size(&vars, "key_cache_block_size", - offsetof(KEY_CACHE, - param_block_size)); -static sys_var_key_cache_long sys_key_cache_division_limit(&vars, "key_cache_division_limit", - offsetof(KEY_CACHE, - param_division_limit)); -static sys_var_key_cache_long sys_key_cache_age_threshold(&vars, "key_cache_age_threshold", - offsetof(KEY_CACHE, - param_age_threshold)); +static sys_var_key_cache_long sys_key_cache_block_size(&vars, + "key_cache_block_size", + offsetof(KEY_CACHE,param_block_size)); +static sys_var_key_cache_long sys_key_cache_division_limit(&vars, + "key_cache_division_limit", + offsetof(KEY_CACHE, param_division_limit)); +static sys_var_key_cache_long sys_key_cache_age_threshold(&vars, + "key_cache_age_threshold", + offsetof(KEY_CACHE, param_age_threshold)); +static sys_var_key_cache_long sys_key_cache_partitions(&vars, + "key_cache_partitions", + offsetof(KEY_CACHE, param_partitions)); static sys_var_const sys_language(&vars, "language", OPT_GLOBAL, SHOW_CHAR, (uchar*) language); @@ -2540,7 +2543,21 @@ bool sys_var_key_cache_long::update(THD pthread_mutex_unlock(&LOCK_global_system_variables); - error= (bool) (ha_resize_key_cache(key_cache)); + switch (offset) { + + case offsetof(KEY_CACHE, param_block_size): + error= (bool) (ha_resize_key_cache(key_cache)); + break; + + case offsetof(KEY_CACHE, param_division_limit): + case offsetof(KEY_CACHE, param_age_threshold): + error= (bool) (ha_change_key_cache_param(key_cache)); + break; + + case offsetof(KEY_CACHE, param_partitions): + error= (bool) (ha_repartition_key_cache(key_cache)); + break; + } pthread_mutex_lock(&LOCK_global_system_variables); key_cache->in_init= 0; @@ -4142,6 +4159,7 @@ static KEY_CACHE *create_key_cache(const key_cache->param_block_size= dflt_key_cache_var.param_block_size; key_cache->param_division_limit= dflt_key_cache_var.param_division_limit; key_cache->param_age_threshold= dflt_key_cache_var.param_age_threshold; + key_cache->param_partitions= dflt_key_cache_var.param_partitions; } } DBUG_RETURN(key_cache); === modified file 'sql/set_var.h' --- a/sql/set_var.h 2009-12-03 11:19:05 +0000 +++ b/sql/set_var.h 2010-02-16 16:41:11 +0000 @@ -1427,6 +1427,7 @@ public: my_free((uchar*) name, MYF(0)); } friend bool process_key_caches(process_key_cache_t func); + friend int fill_key_cache_tables(THD *thd, TABLE_LIST *tables, COND *cond); friend void delete_elements(I_List<NAMED_LIST> *list, void (*free_element)(const char*, uchar*)); }; === modified file 'sql/sql_show.cc' --- a/sql/sql_show.cc 2010-02-01 06:14:12 +0000 +++ b/sql/sql_show.cc 2010-02-16 16:41:11 +0000 @@ -2220,6 +2220,31 @@ void remove_status_vars(SHOW_VAR *list) } + +static void update_key_cache_stat_var(KEY_CACHE *key_cache, size_t ofs) +{ + uint var_no; + switch (ofs) { + case offsetof(KEY_CACHE, blocks_used): + case offsetof(KEY_CACHE, blocks_unused): + case offsetof(KEY_CACHE, global_blocks_changed): + var_no= (ofs-offsetof(KEY_CACHE, blocks_used))/sizeof(ulong); + *(ulong *)((char *) key_cache + ofs)= + (ulong) get_key_cache_stat_value(key_cache, var_no); + break; + case offsetof(KEY_CACHE, global_cache_r_requests): + case offsetof(KEY_CACHE, global_cache_read): + case offsetof(KEY_CACHE, global_cache_w_requests): + case offsetof(KEY_CACHE, global_cache_write): + var_no= 3+(ofs-offsetof(KEY_CACHE, global_cache_w_requests))/ + sizeof(ulonglong); + *(ulonglong *)((char *) key_cache + ofs)= + get_key_cache_stat_value(key_cache, var_no); + break; + } +} + + static bool show_status_array(THD *thd, const char *wild, SHOW_VAR *variables, enum enum_var_type value_type, @@ -2352,10 +2377,12 @@ static bool show_status_array(THD *thd, break; } case SHOW_KEY_CACHE_LONG: + update_key_cache_stat_var(dflt_key_cache, (size_t) value); value= (char*) dflt_key_cache + (ulong)value; end= int10_to_str(*(long*) value, buff, 10); break; case SHOW_KEY_CACHE_LONGLONG: + update_key_cache_stat_var(dflt_key_cache, (size_t) value); value= (char*) dflt_key_cache + (ulong)value; end= longlong10_to_str(*(longlong*) value, buff, 10); break; @@ -6611,6 +6638,90 @@ int fill_schema_files(THD *thd, TABLE_LI } +static +int store_key_cache_table_record(THD *thd, TABLE *table, + const char *name, uint name_length, + KEY_CACHE *key_cache, + uint partitions, uint partition_no) +{ + KEY_CACHE_STATISTICS key_cache_stats; + uint err; + DBUG_ENTER("store_key_cache_table_record"); + + get_key_cache_statistics(key_cache, partition_no, &key_cache_stats); + + if (key_cache_stats.mem_size == 0) + DBUG_RETURN(0); + + restore_record(table, s->default_values); + table->field[0]->store(name, name_length, system_charset_info); + if (partitions == 0) + table->field[1]->set_null(); + else + { + table->field[1]->set_notnull(); + table->field[1]->store((long) partitions, TRUE); + } + + if (partition_no == 0) + table->field[2]->set_null(); + else + { + table->field[2]->set_notnull(); + table->field[2]->store((long) partition_no, TRUE); + } + table->field[3]->store(key_cache_stats.mem_size, TRUE); + table->field[4]->store(key_cache_stats.block_size, TRUE); + table->field[5]->store(key_cache_stats.blocks_used, TRUE); + table->field[6]->store(key_cache_stats.blocks_unused, TRUE); + table->field[7]->store(key_cache_stats.blocks_changed, TRUE); + table->field[8]->store(key_cache_stats.read_requests, TRUE); + table->field[9]->store(key_cache_stats.reads, TRUE); + table->field[10]->store(key_cache_stats.write_requests, TRUE); + table->field[11]->store(key_cache_stats.writes, TRUE); + + err= schema_table_store_record(thd, table); + DBUG_RETURN(err); +} + + +int fill_key_cache_tables(THD *thd, TABLE_LIST *tables, COND *cond) +{ + TABLE *table= tables->table; + I_List_iterator<NAMED_LIST> it(key_caches); + NAMED_LIST *element; + DBUG_ENTER("fill_key_cache_tables"); + + while ((element= it++)) + { + KEY_CACHE *key_cache= (KEY_CACHE *) element->data; + + if (!key_cache->key_cache_inited) + continue; + + uint partitions= key_cache->partitions; + DBUG_ASSERT(partitions <= MAX_KEY_CACHE_PARTITIONS); + + if (partitions) + { + for (uint i= 0; i < partitions; i++) + { + if (store_key_cache_table_record(thd, table, + element->name, element->name_length, + key_cache, partitions, i+1)) + DBUG_RETURN(1); + } + } + + if (store_key_cache_table_record(thd, table, + element->name, element->name_length, + key_cache, partitions, 0)) + DBUG_RETURN(1); + } + DBUG_RETURN(0); +} + + ST_FIELD_INFO schema_fields_info[]= { {"CATALOG_NAME", FN_REFLEN, MYSQL_TYPE_STRING, 0, 1, 0, SKIP_OPEN_TABLE}, @@ -7188,6 +7299,35 @@ ST_FIELD_INFO referential_constraints_fi }; +ST_FIELD_INFO keycache_fields_info[]= +{ + {"KEY_CACHE_NAME", NAME_LEN, MYSQL_TYPE_STRING, 0, 0, 0, SKIP_OPEN_TABLE}, + {"PARTITIONS", 3, MYSQL_TYPE_LONG, 0, + (MY_I_S_MAYBE_NULL | MY_I_S_UNSIGNED) , 0, SKIP_OPEN_TABLE}, + {"PARTITION_NUMBER", 3, MYSQL_TYPE_LONG, 0, + (MY_I_S_MAYBE_NULL | MY_I_S_UNSIGNED), 0, SKIP_OPEN_TABLE}, + {"FULL_SIZE", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, 0, + (MY_I_S_UNSIGNED), 0, SKIP_OPEN_TABLE}, + {"BLOCK_SIZE", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, 0, + (MY_I_S_UNSIGNED), 0, SKIP_OPEN_TABLE }, + {"USED_BLOCKS", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, 0, + (MY_I_S_UNSIGNED), "Key_blocks_used", SKIP_OPEN_TABLE}, + {"UNUSED_BLOCKS", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, 0, + (MY_I_S_UNSIGNED), "Key_blocks_unused", SKIP_OPEN_TABLE}, + {"DIRTY_BLOCKS", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, 0, + (MY_I_S_UNSIGNED), "Key_blocks_not_flushed", SKIP_OPEN_TABLE}, + {"READ_REQUESTS", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, 0, + (MY_I_S_UNSIGNED), "Key_read_requests", SKIP_OPEN_TABLE}, + {"READS", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, 0, + (MY_I_S_UNSIGNED), "Key_reads", SKIP_OPEN_TABLE}, + {"WRITE_REQUESTS", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, 0, + (MY_I_S_UNSIGNED), "Key_write_requests", SKIP_OPEN_TABLE}, + {"WRITES", MY_INT64_NUM_DECIMAL_DIGITS, MYSQL_TYPE_LONGLONG, 0, + (MY_I_S_UNSIGNED), "Key_writes", SKIP_OPEN_TABLE}, + {0, 0, MYSQL_TYPE_STRING, 0, 0, 0, SKIP_OPEN_TABLE} +}; + + /* Description of ST_FIELD_INFO in table.h @@ -7227,6 +7367,8 @@ ST_SCHEMA_TABLE schema_tables[]= fill_variables, make_old_format, 0, 0, -1, 0, 0}, {"INDEX_STATISTICS", index_stats_fields_info, create_schema_table, fill_schema_index_stats, make_old_format, 0, -1, -1, 0, 0}, + {"KEY_CACHES", keycache_fields_info, create_schema_table, + fill_key_cache_tables, make_old_format, 0, -1,-1, 0, 0}, {"KEY_COLUMN_USAGE", key_column_usage_fields_info, create_schema_table, get_all_tables, 0, get_schema_key_column_usage_record, 4, 5, 0, OPEN_TABLE_ONLY}, === modified file 'sql/sql_test.cc' --- a/sql/sql_test.cc 2009-09-07 20:50:10 +0000 +++ b/sql/sql_test.cc 2010-02-16 16:41:11 +0000 @@ -435,7 +435,8 @@ static int print_key_cache_status(const Buffer_size: %10lu\n\ Block_size: %10lu\n\ Division_limit: %10lu\n\ -Age_limit: %10lu\n\ +Age_threshold: %10lu\n\ +Partitions: %10lu\n\ blocks used: %10lu\n\ not flushed: %10lu\n\ w_requests: %10s\n\ @@ -445,6 +446,7 @@ reads: %10s\n\n", name, (ulong) key_cache->param_buff_size, key_cache->param_block_size, key_cache->param_division_limit, key_cache->param_age_threshold, + key_cache->param_partitions, key_cache->blocks_used,key_cache->global_blocks_changed, llstr(key_cache->global_cache_w_requests,llbuff1), llstr(key_cache->global_cache_write,llbuff2), === modified file 'sql/table.h' --- a/sql/table.h 2010-02-12 08:47:31 +0000 +++ b/sql/table.h 2010-02-16 16:41:11 +0000 @@ -953,6 +953,7 @@ enum enum_schema_tables SCH_GLOBAL_STATUS, SCH_GLOBAL_VARIABLES, SCH_INDEX_STATS, + SCH_KEY_CACHES, SCH_KEY_COLUMN_USAGE, SCH_OPEN_TABLES, SCH_PARTITIONS, === modified file 'storage/myisam/mi_check.c' --- a/storage/myisam/mi_check.c 2010-01-14 16:51:00 +0000 +++ b/storage/myisam/mi_check.c 2010-02-16 16:41:11 +0000 @@ -332,7 +332,8 @@ int chk_size(HA_CHECK *param, register M /* The following is needed if called externally (not from myisamchk) */ flush_key_blocks(info->s->key_cache, - info->s->kfile, FLUSH_FORCE_WRITE); + info->s->kfile, &info->s->dirty_part_map, + FLUSH_FORCE_WRITE); size= my_seek(info->s->kfile, 0L, MY_SEEK_END, MYF(MY_THREADSAFE)); if ((skr=(my_off_t) info->state->key_file_length) != size) @@ -1474,6 +1475,7 @@ static int mi_drop_all_indexes(HA_CHECK */ DBUG_PRINT("repair", ("all disabled are empty: create missing")); error= flush_key_blocks(share->key_cache, share->kfile, + &share->dirty_part_map, FLUSH_FORCE_WRITE); goto end; } @@ -1488,6 +1490,7 @@ static int mi_drop_all_indexes(HA_CHECK /* Remove all key blocks of this index file from key cache. */ if ((error= flush_key_blocks(share->key_cache, share->kfile, + &share->dirty_part_map, FLUSH_IGNORE_CHANGED))) goto end; /* purecov: inspected */ @@ -1549,7 +1552,7 @@ int mi_repair(HA_CHECK *param, register if (!param->using_global_keycache) VOID(init_key_cache(dflt_key_cache, param->key_cache_block_size, - (size_t) param->use_buffers, 0, 0)); + (size_t) param->use_buffers, 0, 0, 0)); if (init_io_cache(&param->read_cache,info->dfile, (uint) param->read_buffer_length, @@ -1762,7 +1765,8 @@ err: VOID(end_io_cache(&param->read_cache)); info->opt_flag&= ~(READ_CACHE_USED | WRITE_CACHE_USED); VOID(end_io_cache(&info->rec_cache)); - got_error|=flush_blocks(param, share->key_cache, share->kfile); + got_error|=flush_blocks(param, share->key_cache, share->kfile, + &share->dirty_part_map); if (!got_error && param->testflag & T_UNPACK) { share->state.header.options[0]&= (uchar) ~HA_OPTION_COMPRESS_RECORD; @@ -1908,9 +1912,10 @@ void lock_memory(HA_CHECK *param __attri /* Flush all changed blocks to disk */ -int flush_blocks(HA_CHECK *param, KEY_CACHE *key_cache, File file) +int flush_blocks(HA_CHECK *param, KEY_CACHE *key_cache, File file, + ulonglong *dirty_part_map) { - if (flush_key_blocks(key_cache, file, FLUSH_RELEASE)) + if (flush_key_blocks(key_cache, file, dirty_part_map, FLUSH_RELEASE)) { mi_check_print_error(param,"%d when trying to write bufferts",my_errno); return(1); @@ -1977,7 +1982,8 @@ int mi_sort_index(HA_CHECK *param, regis } /* Flush key cache for this file if we are calling this outside myisamchk */ - flush_key_blocks(share->key_cache,share->kfile, FLUSH_IGNORE_CHANGED); + flush_key_blocks(share->key_cache, share->kfile, &share->dirty_part_map, + FLUSH_IGNORE_CHANGED); share->state.version=(ulong) time((time_t*) 0); old_state= share->state; /* save state if not stored */ @@ -2535,7 +2541,8 @@ int mi_repair_by_sort(HA_CHECK *param, r memcpy( &share->state.state, info->state, sizeof(*info->state)); err: - got_error|= flush_blocks(param, share->key_cache, share->kfile); + got_error|= flush_blocks(param, share->key_cache, share->kfile, + &share->dirty_part_map); VOID(end_io_cache(&info->rec_cache)); if (!got_error) { @@ -3059,7 +3066,8 @@ int mi_repair_parallel(HA_CHECK *param, memcpy(&share->state.state, info->state, sizeof(*info->state)); err: - got_error|= flush_blocks(param, share->key_cache, share->kfile); + got_error|= flush_blocks(param, share->key_cache, share->kfile, + &share->dirty_part_map); /* Destroy the write cache. The master thread did already detach from the share by remove_io_thread() or it was not yet started (if the === modified file 'storage/myisam/mi_close.c' --- a/storage/myisam/mi_close.c 2010-02-10 19:06:24 +0000 +++ b/storage/myisam/mi_close.c 2010-02-16 16:41:11 +0000 @@ -64,6 +64,7 @@ int mi_close(register MI_INFO *info) if (share->kfile >= 0) abort();); if (share->kfile >= 0 && flush_key_blocks(share->key_cache, share->kfile, + &share->dirty_part_map, ((share->temporary || share->deleting) ? FLUSH_IGNORE_CHANGED : FLUSH_RELEASE))) === modified file 'storage/myisam/mi_delete_all.c' --- a/storage/myisam/mi_delete_all.c 2008-04-28 16:24:05 +0000 +++ b/storage/myisam/mi_delete_all.c 2010-02-16 16:41:11 +0000 @@ -52,7 +52,8 @@ int mi_delete_all_rows(MI_INFO *info) If we are using delayed keys or if the user has done changes to the tables since it was locked then there may be key blocks in the key cache */ - flush_key_blocks(share->key_cache, share->kfile, FLUSH_IGNORE_CHANGED); + flush_key_blocks(share->key_cache, share->kfile, &share->dirty_part_map, + FLUSH_IGNORE_CHANGED); #ifdef HAVE_MMAP if (share->file_map) _mi_unmap_file(info); === modified file 'storage/myisam/mi_extra.c' --- a/storage/myisam/mi_extra.c 2010-02-10 19:06:24 +0000 +++ b/storage/myisam/mi_extra.c 2010-02-16 16:41:11 +0000 @@ -268,6 +268,7 @@ int mi_extra(MI_INFO *info, enum ha_extr pthread_mutex_lock(&share->intern_lock); /* Flush pages that we don't need anymore */ if (flush_key_blocks(share->key_cache, share->kfile, + &share->dirty_part_map, (function == HA_EXTRA_PREPARE_FOR_DROP ? FLUSH_IGNORE_CHANGED : FLUSH_RELEASE))) { @@ -326,7 +327,8 @@ int mi_extra(MI_INFO *info, enum ha_extr break; case HA_EXTRA_FLUSH: if (!share->temporary) - flush_key_blocks(share->key_cache, share->kfile, FLUSH_KEEP); + flush_key_blocks(share->key_cache, share->kfile, &share->dirty_part_map, + FLUSH_KEEP); #ifdef HAVE_PWRITE _mi_decrement_open_count(info); #endif === modified file 'storage/myisam/mi_keycache.c' --- a/storage/myisam/mi_keycache.c 2008-03-29 15:56:33 +0000 +++ b/storage/myisam/mi_keycache.c 2010-02-16 16:41:11 +0000 @@ -75,7 +75,8 @@ int mi_assign_to_key_cache(MI_INFO *info in the old key cache. */ - if (flush_key_blocks(share->key_cache, share->kfile, FLUSH_RELEASE)) + if (flush_key_blocks(share->key_cache, share->kfile, &share->dirty_part_map, + FLUSH_RELEASE)) { error= my_errno; mi_print_error(info->s, HA_ERR_CRASHED); @@ -90,7 +91,8 @@ int mi_assign_to_key_cache(MI_INFO *info (This can never fail as there is never any not written data in the new key cache) */ - (void) flush_key_blocks(key_cache, share->kfile, FLUSH_RELEASE); + (void) flush_key_blocks(key_cache, share->kfile, &share->dirty_part_map, + FLUSH_RELEASE); /* ensure that setting the key cache and changing the multi_key_cache @@ -102,6 +104,7 @@ int mi_assign_to_key_cache(MI_INFO *info This should be seen at the lastes for the next call to an myisam function. */ share->key_cache= key_cache; + share->dirty_part_map= 0; /* store the key cache in the global hash structure for future opens */ if (multi_key_cache_set((uchar*) share->unique_file_name, === modified file 'storage/myisam/mi_locking.c' --- a/storage/myisam/mi_locking.c 2009-10-06 06:57:22 +0000 +++ b/storage/myisam/mi_locking.c 2010-02-16 16:41:11 +0000 @@ -68,7 +68,9 @@ int mi_lock_database(MI_INFO *info, int --share->tot_locks; if (info->lock_type == F_WRLCK && !share->w_locks && !share->delay_key_write && flush_key_blocks(share->key_cache, - share->kfile,FLUSH_KEEP)) + share->kfile, + &share->dirty_part_map, + FLUSH_KEEP)) { error=my_errno; mi_print_error(info->s, HA_ERR_CRASHED); @@ -513,7 +515,8 @@ int _mi_test_if_changed(register MI_INFO { /* Keyfile has changed */ DBUG_PRINT("info",("index file changed")); if (share->state.process != share->this_process) - VOID(flush_key_blocks(share->key_cache, share->kfile, FLUSH_RELEASE)); + VOID(flush_key_blocks(share->key_cache, share->kfile, + &share->dirty_part_map, FLUSH_RELEASE)); share->last_process=share->state.process; info->last_unique= share->state.unique; info->last_loop= share->state.update_count; === modified file 'storage/myisam/mi_page.c' --- a/storage/myisam/mi_page.c 2009-05-06 12:03:24 +0000 +++ b/storage/myisam/mi_page.c 2010-02-16 16:41:11 +0000 @@ -94,10 +94,11 @@ int _mi_write_keypage(register MI_INFO * } #endif DBUG_RETURN((key_cache_write(info->s->key_cache, - info->s->kfile,page, level, (uchar*) buff,length, - (uint) keyinfo->block_length, - (int) ((info->lock_type != F_UNLCK) || - info->s->delay_key_write)))); + info->s->kfile, &info->s->dirty_part_map, + page, level, (uchar*) buff, length, + (uint) keyinfo->block_length, + (int) ((info->lock_type != F_UNLCK) || + info->s->delay_key_write)))); } /* mi_write_keypage */ @@ -116,7 +117,8 @@ int _mi_dispose(register MI_INFO *info, mi_sizestore(buff,old_link); info->s->state.changed|= STATE_NOT_SORTED_PAGES; DBUG_RETURN(key_cache_write(info->s->key_cache, - info->s->kfile, pos , level, buff, + info->s->kfile, &info->s->dirty_part_map, + pos , level, buff, sizeof(buff), (uint) keyinfo->block_length, (int) (info->lock_type != F_UNLCK))); === modified file 'storage/myisam/mi_panic.c' --- a/storage/myisam/mi_panic.c 2006-12-31 00:32:21 +0000 +++ b/storage/myisam/mi_panic.c 2010-02-16 16:41:11 +0000 @@ -47,7 +47,8 @@ int mi_panic(enum ha_panic_function flag if (info->s->options & HA_OPTION_READ_ONLY_DATA) break; #endif - if (flush_key_blocks(info->s->key_cache, info->s->kfile, FLUSH_RELEASE)) + if (flush_key_blocks(info->s->key_cache, info->s->kfile, + &info->s->dirty_part_map, FLUSH_RELEASE)) error=my_errno; if (info->opt_flag & WRITE_CACHE_USED) if (flush_io_cache(&info->rec_cache)) === modified file 'storage/myisam/mi_preload.c' --- a/storage/myisam/mi_preload.c 2009-09-09 15:13:13 +0000 +++ b/storage/myisam/mi_preload.c 2010-02-16 16:41:11 +0000 @@ -65,7 +65,7 @@ int mi_preload(MI_INFO *info, ulonglong } } else - block_length= share->key_cache->key_cache_block_size; + block_length= share->key_cache->param_block_size; length= info->preload_buff_size/block_length * block_length; set_if_bigger(length, block_length); @@ -73,7 +73,8 @@ int mi_preload(MI_INFO *info, ulonglong if (!(buff= (uchar *) my_malloc(length, MYF(MY_WME)))) DBUG_RETURN(my_errno= HA_ERR_OUT_OF_MEM); - if (flush_key_blocks(share->key_cache,share->kfile, FLUSH_RELEASE)) + if (flush_key_blocks(share->key_cache, share->kfile, &share->dirty_part_map, + FLUSH_RELEASE)) goto err; do === modified file 'storage/myisam/mi_test1.c' --- a/storage/myisam/mi_test1.c 2008-04-28 16:24:05 +0000 +++ b/storage/myisam/mi_test1.c 2010-02-16 16:41:11 +0000 @@ -49,7 +49,8 @@ int main(int argc,char *argv[]) MY_INIT(argv[0]); my_init(); if (key_cacheing) - init_key_cache(dflt_key_cache,KEY_CACHE_BLOCK_SIZE,IO_SIZE*16,0,0); + init_key_cache(dflt_key_cache,KEY_CACHE_BLOCK_SIZE,IO_SIZE*16,0,0, + DEFAULT_KEY_CACHE_PARTITIONS); get_options(argc,argv); exit(run_test("test1")); === modified file 'storage/myisam/mi_test2.c' --- a/storage/myisam/mi_test2.c 2008-04-28 16:24:05 +0000 +++ b/storage/myisam/mi_test2.c 2010-02-16 16:41:11 +0000 @@ -215,7 +215,8 @@ int main(int argc, char *argv[]) if (!silent) printf("- Writing key:s\n"); if (key_cacheing) - init_key_cache(dflt_key_cache,key_cache_block_size,key_cache_size,0,0); + init_key_cache(dflt_key_cache,key_cache_block_size,key_cache_size,0,0, + DEFAULT_KEY_CACHE_PARTITIONS); if (do_locking) mi_lock_database(file,F_WRLCK); if (write_cacheing) === modified file 'storage/myisam/mi_test3.c' --- a/storage/myisam/mi_test3.c 2010-01-06 21:27:53 +0000 +++ b/storage/myisam/mi_test3.c 2010-02-16 16:41:11 +0000 @@ -177,7 +177,8 @@ void start_test(int id) exit(1); } if (key_cacheing && rnd(2) == 0) - init_key_cache(dflt_key_cache, KEY_CACHE_BLOCK_SIZE, 65536L, 0, 0); + init_key_cache(dflt_key_cache, KEY_CACHE_BLOCK_SIZE, 65536L, 0, 0, + DEFAULT_KEY_CACHE_PARTITIONS); printf("Process %d, pid: %ld\n", id, (long) getpid()); fflush(stdout); === modified file 'storage/myisam/myisam_ftdump.c' --- a/storage/myisam/myisam_ftdump.c 2009-11-30 13:36:06 +0000 +++ b/storage/myisam/myisam_ftdump.c 2010-02-16 16:41:11 +0000 @@ -83,7 +83,7 @@ int main(int argc,char *argv[]) usage(); } - init_key_cache(dflt_key_cache,MI_KEY_BLOCK_LENGTH,USE_BUFFER_INIT, 0, 0); + init_key_cache(dflt_key_cache,MI_KEY_BLOCK_LENGTH,USE_BUFFER_INIT, 0, 0, 0); if (!(info=mi_open(argv[0], O_RDONLY, HA_OPEN_ABORT_IF_LOCKED|HA_OPEN_FROM_SQL_LAYER))) === modified file 'storage/myisam/myisamchk.c' --- a/storage/myisam/myisamchk.c 2009-12-03 11:34:11 +0000 +++ b/storage/myisam/myisamchk.c 2010-02-16 16:41:11 +0000 @@ -1102,7 +1102,7 @@ static int myisamchk(HA_CHECK *param, ch { if (param->testflag & (T_EXTEND | T_MEDIUM)) VOID(init_key_cache(dflt_key_cache,opt_key_cache_block_size, - (size_t) param->use_buffers, 0, 0)); + (size_t) param->use_buffers, 0, 0, 0)); VOID(init_io_cache(&param->read_cache,datafile, (uint) param->read_buffer_length, READ_CACHE, @@ -1116,7 +1116,8 @@ static int myisamchk(HA_CHECK *param, ch HA_OPTION_COMPRESS_RECORD)) || (param->testflag & (T_EXTEND | T_MEDIUM))) error|=chk_data_link(param, info, test(param->testflag & T_EXTEND)); - error|=flush_blocks(param, share->key_cache, share->kfile); + error|=flush_blocks(param, share->key_cache, share->kfile, + &share->dirty_part_map); VOID(end_io_cache(&param->read_cache)); } if (!error) @@ -1526,7 +1527,7 @@ static int mi_sort_records(HA_CHECK *par DBUG_RETURN(0); /* Nothing to do */ init_key_cache(dflt_key_cache, opt_key_cache_block_size, - (size_t) param->use_buffers, 0, 0); + (size_t) param->use_buffers, 0, 0, 0); if (init_io_cache(&info->rec_cache,-1,(uint) param->write_buffer_length, WRITE_CACHE,share->pack.header_length,1, MYF(MY_WME | MY_WAIT_IF_FULL))) @@ -1641,8 +1642,8 @@ err: my_free(sort_info.buff,MYF(MY_ALLOW_ZERO_PTR)); sort_info.buff=0; share->state.sortkey=sort_key; - DBUG_RETURN(flush_blocks(param, share->key_cache, share->kfile) | - got_error); + DBUG_RETURN(flush_blocks(param, share->key_cache, share->kfile, + &share->dirty_part_map) | got_error); } /* sort_records */ === modified file 'storage/myisam/myisamdef.h' --- a/storage/myisam/myisamdef.h 2010-02-10 19:06:24 +0000 +++ b/storage/myisam/myisamdef.h 2010-02-16 16:41:11 +0000 @@ -174,6 +174,8 @@ typedef struct st_mi_isam_share *index_file_name; uchar *file_map; /* mem-map of file if possible */ KEY_CACHE *key_cache; /* ref to the current key cache */ + /* To mark the key cache partitions containing dirty pages for this file */ + ulonglong dirty_part_map; MI_DECODE_TREE *decode_trees; uint16 *decode_tables; /* Function to use for a row checksum. */ @@ -733,7 +735,8 @@ void mi_check_print_info _VARARGS((HA_CH #ifdef THREAD pthread_handler_t thr_find_all_keys(void *arg); #endif -int flush_blocks(HA_CHECK *param, KEY_CACHE *key_cache, File file); +int flush_blocks(HA_CHECK *param, KEY_CACHE *key_cache, File file, + ulonglong *dirty_part_map); #ifdef __cplusplus } #endif === modified file 'storage/myisam/myisamlog.c' --- a/storage/myisam/myisamlog.c 2009-11-29 23:08:56 +0000 +++ b/storage/myisam/myisamlog.c 2010-02-16 16:41:11 +0000 @@ -333,7 +333,7 @@ static int examine_log(char * file_name, init_tree(&tree,0,0,sizeof(file_info),(qsort_cmp2) file_info_compare,1, (tree_element_free) file_info_free, NULL); VOID(init_key_cache(dflt_key_cache,KEY_CACHE_BLOCK_SIZE,KEY_CACHE_SIZE, - 0, 0)); + 0, 0, 0)); files_open=0; access_time=0; while (access_time++ != number_of_commands &&

1 0

[Maria-developers] Updated (by Guest): Add a mysqlbinlog option to filter updates to certain tables (40)
by worklog-noreply＠askmonty.org 16 Feb '10

16 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to filter updates to certain tables CREATION DATE..: Mon, 10 Aug 2009, 13:25 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Psergey CATEGORY.......: Server-Sprint TASK ID........: 40 (http://askmonty.org/worklog/?tid=40) VERSION........: Server-9.x STATUS.........: Cancelled PRIORITY.......: 60 WORKED HOURS...: 32 ESTIMATE.......: 32 (hours remain) ORIG. ESTIMATE.: 48 PROGRESS NOTES: -=-=(Guest - Tue, 16 Feb 2010, 10:23)=-=- Status updated. --- /tmp/wklog.40.old.18300 2010-02-16 10:23:20.000000000 +0200 +++ /tmp/wklog.40.new.18300 2010-02-16 10:23:20.000000000 +0200 @@ -1 +1 @@ -Assigned +Cancelled -=-=(Guest - Wed, 25 Nov 2009, 11:41)=-=- Status updated. --- /tmp/wklog.40.old.5760 2009-11-25 11:41:09.000000000 +0200 +++ /tmp/wklog.40.new.5760 2009-11-25 11:41:09.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +Assigned -=-=(Guest - Wed, 25 Nov 2009, 11:41)=-=- Category updated. --- /tmp/wklog.40.old.5737 2009-11-25 11:41:03.000000000 +0200 +++ /tmp/wklog.40.new.5737 2009-11-25 11:41:03.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Bothorsen - Tue, 17 Nov 2009, 17:20)=-=- Alex is closer to a working patch now. Worked 14 hours and estimate 32 hours remain (original estimate unchanged). -=-=(Bothorsen - Thu, 12 Nov 2009, 13:13)=-=- Work hours by Alexi and Bo + estimated time for the task. Worked 16 hours and estimate 46 hours remain (original estimate increased by 14 hours). -=-=(Alexi - Sun, 08 Nov 2009, 15:18)=-=- Low Level Design modified. --- /tmp/wklog.40.old.15787 2009-11-08 15:18:11.000000000 +0200 +++ /tmp/wklog.40.new.15787 2009-11-08 15:18:11.000000000 +0200 @@ -62,7 +62,7 @@ it considers the query to extent to the end of the event. 2. For 'db' (current db) the trailing zero is redundant since the length is already known. -3. db_len = 0 means that this is the current db. +3. In tables_info, db_len = 0 means that this is the current db. When reading Query events from binary log, we can recognize its format by its post-header length: in extended case the post-header includes 4 @@ -75,6 +75,77 @@ + #define Q_QUERY_LEN_OFFSET Q_STATUS_VARS_LEN_OFFSET + 2 + #define Q_QUERY_TABLES_INFO_LEN_OFFSET Q_QUERY_LEN_OFFSET + 2 + +*********************************************************************** +HELP NEEDED +*********************************************************************** +The QUERY_HEADER_LEN is used in the definition of MAX_LOG_EVENT_HEADER: + +log_event.h +~~~~~~~~~~~ +#define MAX_LOG_EVENT_HEADER ( /* in order of Query_log_event::write */ \ + LOG_EVENT_HEADER_LEN + /* write_header */ \ + QUERY_HEADER_LEN + /* write_data */ \ + EXECUTE_LOAD_QUERY_EXTRA_HEADER_LEN + /*write_post_header_for_derived */ \ + MAX_SIZE_LOG_EVENT_STATUS + /* status */ \ + NAME_LEN + 1) + +which is used only for setting + + thd->variables.max_allowed_packet + mysql->net.max_packet_size + +Looks like (but I am not quite sure) that QUERY_HEADER_LEN can simply +(without making any other changes) be substituted in this definition by +QUERY_HEADER_LEN_EXT. + +Below I list all places where MAX_LOG_EVENT_HEADER is used: + +slave.cc +~~~~~~~~ +static int init_slave_thread(...) +{ ... + /* + Adding MAX_LOG_EVENT_HEADER_LEN to the max_allowed_packet on all + slave threads, since a replication event can become this much larger + than the corresponding packet (query) sent from client to master. + */ + thd->variables.max_allowed_packet= global_system_variables.max_allowed_packet + + MAX_LOG_EVENT_HEADER; /* note, incr over the global not session var */ + ... +} +pthread_handler_t handle_slave_io(...) +{ ... + /* + Adding MAX_LOG_EVENT_HEADER_LEN to the max_packet_size on the I/O + thread, since a replication event can become this much larger than + the corresponding packet (query) sent from client to master. + */ + mysql->net.max_packet_size= thd->net.max_packet_size+= MAX_LOG_EVENT_HEADER; + ... +} + +sql_repl.cc +~~~~~~~~~~~ +void mysql_binlog_send(...) +{ ... + /* + Adding MAX_LOG_EVENT_HEADER_LEN, since a binlog event can become + this larger than the corresponding packet (query) sent + from client to master. + */ + thd->variables.max_allowed_packet+= MAX_LOG_EVENT_HEADER; + ... +} +bool mysql_show_binlog_events(...) +{ ... + /* + to account binlog event header size + */ + thd->variables.max_allowed_packet+= MAX_LOG_EVENT_HEADER; + ... +} + 3. Changes in log events ************************ @@ -84,7 +155,7 @@ This setting is done in Format description event constructor which creates the event for writing to binary log: - if (binlog_with_tables_info) + if (opt_binlog_with_tables_info) post_header_len[QUERY_EVENT - 1] = QUERY_HEADER_LEN_EXT; else post_header_len[QUERY_EVENT - 1] = QUERY_HEADER_LEN; @@ -99,12 +170,12 @@ following manner: switch (binlog_ver) { - case 4: /* MySQL 5.0 and higher */ + #ifndef MYSQL_CLIENT + case 4: /* MySQL 5.0 and higher */ ... -+ #else -+ <error> + break; + #endif + case 1: case 3: ... @@ -132,7 +203,7 @@ -------------------------------- [Creates the event for binlogging] -In case of binlog_with_tables_info = TRUE, set additionally query_len, +In case of opt_binlog_with_tables_info = TRUE, set additionally query_len, tables_info_len, and tables_info members (the constructor is to have an additional 'tables_info' argument). @@ -140,7 +211,7 @@ ---------------- [Writes the event to binlog] -In case of binlog_with_tables_info = TRUE, write additional members +In case of opt_binlog_with_tables_info = TRUE, write additional members (query_len, tables_info_len, and tables_info) to binary log. Also write corresponding whole event length to the common-header. -=-=(Alexi - Sun, 08 Nov 2009, 10:40)=-=- Low Level Design modified. --- /tmp/wklog.40.old.5055 2009-11-08 08:40:02.000000000 +0000 +++ /tmp/wklog.40.new.5055 2009-11-08 08:40:02.000000000 +0000 @@ -3,6 +3,7 @@ 1. Adding --binlog-with-tables-info option ****************************************** +GLOBAL, read-only option. When set, Query events are to be written in the extended binary format which contains tables_info. When not set, Query events -=-=(Alexi - Thu, 05 Nov 2009, 12:37)=-=- Low Level Design modified. --- /tmp/wklog.40.old.11441 2009-11-05 12:37:16.000000000 +0200 +++ /tmp/wklog.40.new.11441 2009-11-05 12:37:16.000000000 +0200 @@ -1,9 +1,18 @@ OPTION: 2.5 Extend Query Events With Tables Info ================================================ -1. Query_log_event Binary Format -******************************** -Changes to be done: +1. Adding --binlog-with-tables-info option +****************************************** + +When set, Query events are to be written in the extended binary +format which contains tables_info. When not set, Query events +are to be written in usual format (without any changes). + +2. Query event extended binary format +************************************* + +When --binlog-with-tables-info is set, Query events are writen +to binary log in the following (extended) format. Query_log_event binary format --------------------------------- @@ -24,12 +33,12 @@ error_code 2 status_vars_len 2 + query_len 2 (see Note 1) -+ tables_info_len 2 (see Note 2) ++ tables_info_len 2 --------------------------------- BODY: status_vars status_vars_len - db db_len + 1 -+ db db_len (see Note 3) ++ db db_len (see Note 2) query query_len + tables_info @@ -37,7 +46,7 @@ --------------------------------- Name Size (bytes) --------------------------------- - db_len 1 (see Note 4) + db_len 1 (see Note 3) db db_len table_name_len 1 table_name table_name_len @@ -48,19 +57,99 @@ table_name table_name_len NOTES -1. Currently Query_log_event format doesn't include 'query_len' because +1. In usual format, Query_log_event doesn't include 'query_len' because it considers the query to extent to the end of the event. -2. If tables_info is not included in the event (--binlog-with-tables-info - option), tables_info_len = 0. -3. The trailing zero is redundant since the length is already known. -4. In case of db = current db, db_len = 0 and db = empty, because - current db is already included in the current event format. +2. For 'db' (current db) the trailing zero is redundant since the length + is already known. +3. db_len = 0 means that this is the current db. + +When reading Query events from binary log, we can recognize its format +by its post-header length: in extended case the post-header includes 4 +additional bytes. + + #define QUERY_HEADER_LEN (QUERY_HEADER_MINIMAL_LEN + 4) ++ #define QUERY_HEADER_LEN_EXT (QUERY_HEADER_LEN + 4) + ... + #define Q_STATUS_VARS_LEN_OFFSET 11 ++ #define Q_QUERY_LEN_OFFSET Q_STATUS_VARS_LEN_OFFSET + 2 ++ #define Q_QUERY_TABLES_INFO_LEN_OFFSET Q_QUERY_LEN_OFFSET + 2 + +3. Changes in log events +************************ + +3.1. Format description event +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Changes needed here concern setting post-header length for Query events. +This setting is done in Format description event constructor which creates +the event for writing to binary log: + + if (binlog_with_tables_info) + post_header_len[QUERY_EVENT - 1] = QUERY_HEADER_LEN_EXT; + else + post_header_len[QUERY_EVENT - 1] = QUERY_HEADER_LEN; + +This change is to be done only for case binlog_ver = 4. + +NOTE. The refered above constructor is allowed to be invoked in a client +context for creating "artificial" Format description events in case of +MySQL < 5.0 (e.g. see mysqlbinlog code). To avoid compilation problems +(because of 'binlog_with_tables_info') and taking into account the +"MySQL < 5.0" restriction, we have to #ifdef out the above code in +following manner: + + switch (binlog_ver) { + case 4: /* MySQL 5.0 and higher */ ++ #ifndef MYSQL_CLIENT + ... ++ #else ++ <error> ++ #endif + case 1: + case 3: + ... + } + +3.2. Query event +~~~~~~~~~~~~~~~~ +Changes needed here include adding tables_info and tables_info_len +members (member for query length already exists) and modifying the +following function-members: + +Query_log_event(buf) constructor +-------------------------------- +[Parses binary format written to the 'buf'] + +Getting post-header length from the Format description event (passed +to the constructor as an argument), define whether buf contains an +extended or usual Query event and parse the buf contents accordingly. + +NOTE. Defining Query event format here should be done with taking into +account that this constructor can be called within a Query-derived +event with the event_type argument != QUERY_EVENT. + +Query_log_event(thd) constructor +-------------------------------- +[Creates the event for binlogging] + +In case of binlog_with_tables_info = TRUE, set additionally query_len, +tables_info_len, and tables_info members (the constructor is to have +an additional 'tables_info' argument). + +write() function +---------------- +[Writes the event to binlog] + +In case of binlog_with_tables_info = TRUE, write additional members +(query_len, tables_info_len, and tables_info) to binary log. Also +write corresponding whole event length to the common-header. + +<To be continued> -2. Where to get tables info from? +4. Where to get tables info from? ********************************* -2.1. Case study: CREATE TABLE -****************************** +4.1. Case study: CREATE TABLE +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *** CREATE TABLE table [SELECT ...] @@ -129,4 +218,4 @@ } } -To be continued +<To be continued> -=-=(Alexi - Wed, 04 Nov 2009, 10:21)=-=- Low Level Design modified. --- /tmp/wklog.40.old.6734 2009-11-04 10:21:20.000000000 +0200 +++ /tmp/wklog.40.new.6734 2009-11-04 10:21:20.000000000 +0200 @@ -21,9 +21,9 @@ slave_proxy_id 4 exec_time 4 db_len 1 -+ query_len 2 (see Note 1) error_code 2 status_vars_len 2 ++ query_len 2 (see Note 1) + tables_info_len 2 (see Note 2) --------------------------------- BODY: -=-=(Alexi - Tue, 03 Nov 2009, 11:19)=-=- Low Level Design modified. --- /tmp/wklog.40.old.7187 2009-11-03 11:19:22.000000000 +0200 +++ /tmp/wklog.40.new.7187 2009-11-03 11:19:22.000000000 +0200 @@ -1 +1,132 @@ +OPTION: 2.5 Extend Query Events With Tables Info +================================================ +1. Query_log_event Binary Format +******************************** +Changes to be done: + + Query_log_event binary format + --------------------------------- + Name Size (bytes) + --------------------------------- + COMMON HEADER: + timestamp 4 + type 1 + server_id 4 + total_size 4 + master_position 4 + flags 2 + --------------------------------- + POST HEADER: + slave_proxy_id 4 + exec_time 4 + db_len 1 ++ query_len 2 (see Note 1) + error_code 2 + status_vars_len 2 ++ tables_info_len 2 (see Note 2) + --------------------------------- + BODY: + status_vars status_vars_len +- db db_len + 1 ++ db db_len (see Note 3) + query query_len ++ tables_info + + tables_info binary format + --------------------------------- + Name Size (bytes) + --------------------------------- + db_len 1 (see Note 4) + db db_len + table_name_len 1 + table_name table_name_len + ... + db_len 1 + db db_len + table_name_len 1 + table_name table_name_len + +NOTES +1. Currently Query_log_event format doesn't include 'query_len' because + it considers the query to extent to the end of the event. +2. If tables_info is not included in the event (--binlog-with-tables-info + option), tables_info_len = 0. +3. The trailing zero is redundant since the length is already known. +4. In case of db = current db, db_len = 0 and db = empty, because + current db is already included in the current event format. + +2. Where to get tables info from? +********************************* + +2.1. Case study: CREATE TABLE +****************************** + +*** CREATE TABLE table [SELECT ...] + + bool mysql_create_table_no_lock( + THD *thd, + const char *db, + const char *table_name, ...) + { + ... + // ------------------------------------- + // WL40: To be included in tables_info: + // * db, table_name + // * thd->lex->query_tables (tables refered to in + // the select-part; empty if no select-part) + // ------------------------------------- + write_bin_log(thd, TRUE, thd->query, thd->query_length); + } + +*** CREATE TABLE table LIKE src-table + + bool mysql_create_like_table( + ... + TABLE_LIST *table, + TABLE_LIST *src_table, + ...) + { + ... + if (thd->current_stmt_binlog_row_based) + { // RBR: In this case we don't replicate temp tables + if (!(create_info->options & HA_LEX_CREATE_TMP_TABLE)) + { + if (src_table->table->s->tmp_table) + { // CREATE normal-table LIKE temp-table: + + // Generate new query without LIKE-part + store_create_info(thd, table, &query, create_info, FALSE); + + // ------------------------------------- + // WL40: To include to tables_info: + // * table (src_table is not included) + // ------------------------------------- + write_bin_log(thd, TRUE, query.ptr(), query.length()); + } + else + { // CREATE normal-table LIKE normal-table + + // ------------------------------------- + // WL40: To include to log_tables_info: + // * table + // * src_table + // ------------------------------------- + write_bin_log(thd, TRUE, thd->query, thd->query_length); + } + } + // CREATE temp-table LIKE ... + // This case is not replicated + } + else + { // SBR: + // ------------------------------------- + // WL40: To include to tables_info: + // * table + // * src_table + // ------------------------------------- + write_bin_log(thd, TRUE, thd->query, thd->query_length); + } + } + +To be continued ------------------------------------------------------------ -=-=(View All Progress Notes, 17 total)=-=- http://askmonty.org/worklog/index.pl?tid=40&nolimit=1 DESCRIPTION: Replication slave can be set to filter updates to certain tables with --replicate-[wild-]{do,ignore}-table options. This task is about adding similar functionality to mysqlbinlog. HIGH-LEVEL SPECIFICATION: 1. Context ---------- (See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global overview) At the moment, the server has these replication slave options: --replicate-do-table=db.tbl --replicate-ignore-table=db.tbl --replicate-wild-do-table=pattern.pattern --replicate-wild-ignore-table=pattern.pattern They affect both RBR and SBR events. SBR events are checked after the statement has been parsed, the server iterates over list of used tables and checks them againist --replicate instructions. What is interesting is that this scheme still allows to update the ignored table through a VIEW. 2. Table filtering in mysqlbinlog --------------------------------- Per-table filtering of RBR events is easy (as it is relatively easy to extract the name of the table that the event applies to). Per-table filtering of SBR events is hard, as generally it is not apparent which tables the statement refers to. This opens possible options: 2.1 Put the parser into mysqlbinlog ----------------------------------- Once we have a full parser in mysqlbinlog, we'll be able to check which tables are used by a statement, and will allow to show behaviour identical to those that one obtains when using --replicate-* slave options. (It is not clear how much effort is needed to put the parser into mysqlbinlog. Any guesses?) 2.2 Use dumb regexp match ------------------------- Use a really dumb approach. A query is considered to be modifying table X if it matches an expression CREATE TABLE $tablename DROP $tablename UPDATE ...$tablename ... SET // here '...' can't contain the word 'SET' DELETE ...$tablename ... WHERE // same as above ALTER TABLE $tablename .. etc (go get from the grammar) .. The advantage over doing the same in awk is that mysqlbinlog will also process RBR statements, and together with that will provide a working solution for those who are careful with their table names not mixing with string constants and such. (TODO: string constants are of particular concern as they come from [potentially hostile] users, unlike e.g. table aliases which come from [not hostile] developers. Remove also all string constants before attempting to do match?) 2.3 Have the master put annotations ----------------------------------- We could add a master option so that it injects into query a mark that tells which tables the query will affect, e.g. for the query UPDATE t1 LEFT JOIN db3.t2 ON ... WHERE ... the binlog will have /* !mysqlbinlog: updates t1,db3.t2 */ UPDATE t1 LEFT JOIN ... and further processing in mysqlbinlog will be trivial. 2.4 Implement server functionality to ignore certain tables ----------------------------------------------------------- We could add a general facility in the server to ignore certain tables: SET SESSION ignored_tables = "db1.t1,db2.t2"; This would work similar to --replicate-ignore-table, but in a general way not restricted to the slave SQL thread. It would then be trivial for mysqlbinlog to add such statements at the start of the output, or probably the user could just do it manually with no need for additional options for mysqlbinlog. It might be useful to integrate this with the code that already handles --replicate-ignore-db and similar slave options. 2.5 Extend Query Events With Tables Info ---------------------------------------- We could extend query events structure with a tables info - a list of tables which the query refers to: <current query event structure> tables_info_len dbase_len dbase table_len table ... dbase_len dbase table_len table Note. In case of <dbase> = current data base, we can set dbase_len = 0 and dbase = empty because current query event structure already includes current data base name. Note. Possibly it is reasonable also to add a --binlog-with-tables-info option which defines whether tables info must be included to the query events. LOW-LEVEL DESIGN: OPTION: 2.5 Extend Query Events With Tables Info ================================================ 1. Adding --binlog-with-tables-info option ****************************************** GLOBAL, read-only option. When set, Query events are to be written in the extended binary format which contains tables_info. When not set, Query events are to be written in usual format (without any changes). 2. Query event extended binary format ************************************* When --binlog-with-tables-info is set, Query events are writen to binary log in the following (extended) format. Query_log_event binary format --------------------------------- Name Size (bytes) --------------------------------- COMMON HEADER: timestamp 4 type 1 server_id 4 total_size 4 master_position 4 flags 2 --------------------------------- POST HEADER: slave_proxy_id 4 exec_time 4 db_len 1 error_code 2 status_vars_len 2 + query_len 2 (see Note 1) + tables_info_len 2 --------------------------------- BODY: status_vars status_vars_len - db db_len + 1 + db db_len (see Note 2) query query_len + tables_info tables_info binary format --------------------------------- Name Size (bytes) --------------------------------- db_len 1 (see Note 3) db db_len table_name_len 1 table_name table_name_len ... db_len 1 db db_len table_name_len 1 table_name table_name_len NOTES 1. In usual format, Query_log_event doesn't include 'query_len' because it considers the query to extent to the end of the event. 2. For 'db' (current db) the trailing zero is redundant since the length is already known. 3. In tables_info, db_len = 0 means that this is the current db. When reading Query events from binary log, we can recognize its format by its post-header length: in extended case the post-header includes 4 additional bytes. #define QUERY_HEADER_LEN (QUERY_HEADER_MINIMAL_LEN + 4) + #define QUERY_HEADER_LEN_EXT (QUERY_HEADER_LEN + 4) ... #define Q_STATUS_VARS_LEN_OFFSET 11 + #define Q_QUERY_LEN_OFFSET Q_STATUS_VARS_LEN_OFFSET + 2 + #define Q_QUERY_TABLES_INFO_LEN_OFFSET Q_QUERY_LEN_OFFSET + 2 *********************************************************************** HELP NEEDED *********************************************************************** The QUERY_HEADER_LEN is used in the definition of MAX_LOG_EVENT_HEADER: log_event.h ~~~~~~~~~~~ #define MAX_LOG_EVENT_HEADER ( /* in order of Query_log_event::write */ \ LOG_EVENT_HEADER_LEN + /* write_header */ \ QUERY_HEADER_LEN + /* write_data */ \ EXECUTE_LOAD_QUERY_EXTRA_HEADER_LEN + /*write_post_header_for_derived */ \ MAX_SIZE_LOG_EVENT_STATUS + /* status */ \ NAME_LEN + 1) which is used only for setting thd->variables.max_allowed_packet mysql->net.max_packet_size Looks like (but I am not quite sure) that QUERY_HEADER_LEN can simply (without making any other changes) be substituted in this definition by QUERY_HEADER_LEN_EXT. Below I list all places where MAX_LOG_EVENT_HEADER is used: slave.cc ~~~~~~~~ static int init_slave_thread(...) { ... /* Adding MAX_LOG_EVENT_HEADER_LEN to the max_allowed_packet on all slave threads, since a replication event can become this much larger than the corresponding packet (query) sent from client to master. */ thd->variables.max_allowed_packet= global_system_variables.max_allowed_packet + MAX_LOG_EVENT_HEADER; /* note, incr over the global not session var */ ... } pthread_handler_t handle_slave_io(...) { ... /* Adding MAX_LOG_EVENT_HEADER_LEN to the max_packet_size on the I/O thread, since a replication event can become this much larger than the corresponding packet (query) sent from client to master. */ mysql->net.max_packet_size= thd->net.max_packet_size+= MAX_LOG_EVENT_HEADER; ... } sql_repl.cc ~~~~~~~~~~~ void mysql_binlog_send(...) { ... /* Adding MAX_LOG_EVENT_HEADER_LEN, since a binlog event can become this larger than the corresponding packet (query) sent from client to master. */ thd->variables.max_allowed_packet+= MAX_LOG_EVENT_HEADER; ... } bool mysql_show_binlog_events(...) { ... /* to account binlog event header size */ thd->variables.max_allowed_packet+= MAX_LOG_EVENT_HEADER; ... } 3. Changes in log events ************************ 3.1. Format description event ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Changes needed here concern setting post-header length for Query events. This setting is done in Format description event constructor which creates the event for writing to binary log: if (opt_binlog_with_tables_info) post_header_len[QUERY_EVENT - 1] = QUERY_HEADER_LEN_EXT; else post_header_len[QUERY_EVENT - 1] = QUERY_HEADER_LEN; This change is to be done only for case binlog_ver = 4. NOTE. The refered above constructor is allowed to be invoked in a client context for creating "artificial" Format description events in case of MySQL < 5.0 (e.g. see mysqlbinlog code). To avoid compilation problems (because of 'binlog_with_tables_info') and taking into account the "MySQL < 5.0" restriction, we have to #ifdef out the above code in following manner: switch (binlog_ver) { + #ifndef MYSQL_CLIENT case 4: /* MySQL 5.0 and higher */ ... break; + #endif case 1: case 3: ... } 3.2. Query event ~~~~~~~~~~~~~~~~ Changes needed here include adding tables_info and tables_info_len members (member for query length already exists) and modifying the following function-members: Query_log_event(buf) constructor -------------------------------- [Parses binary format written to the 'buf'] Getting post-header length from the Format description event (passed to the constructor as an argument), define whether buf contains an extended or usual Query event and parse the buf contents accordingly. NOTE. Defining Query event format here should be done with taking into account that this constructor can be called within a Query-derived event with the event_type argument != QUERY_EVENT. Query_log_event(thd) constructor -------------------------------- [Creates the event for binlogging] In case of opt_binlog_with_tables_info = TRUE, set additionally query_len, tables_info_len, and tables_info members (the constructor is to have an additional 'tables_info' argument). write() function ---------------- [Writes the event to binlog] In case of opt_binlog_with_tables_info = TRUE, write additional members (query_len, tables_info_len, and tables_info) to binary log. Also write corresponding whole event length to the common-header. <To be continued> 4. Where to get tables info from? ********************************* 4.1. Case study: CREATE TABLE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *** CREATE TABLE table [SELECT ...] bool mysql_create_table_no_lock( THD *thd, const char *db, const char *table_name, ...) { ... // ------------------------------------- // WL40: To be included in tables_info: // * db, table_name // * thd->lex->query_tables (tables refered to in // the select-part; empty if no select-part) // ------------------------------------- write_bin_log(thd, TRUE, thd->query, thd->query_length); } *** CREATE TABLE table LIKE src-table bool mysql_create_like_table( ... TABLE_LIST *table, TABLE_LIST *src_table, ...) { ... if (thd->current_stmt_binlog_row_based) { // RBR: In this case we don't replicate temp tables if (!(create_info->options & HA_LEX_CREATE_TMP_TABLE)) { if (src_table->table->s->tmp_table) { // CREATE normal-table LIKE temp-table: // Generate new query without LIKE-part store_create_info(thd, table, &query, create_info, FALSE); // ------------------------------------- // WL40: To include to tables_info: // * table (src_table is not included) // ------------------------------------- write_bin_log(thd, TRUE, query.ptr(), query.length()); } else { // CREATE normal-table LIKE normal-table // ------------------------------------- // WL40: To include to log_tables_info: // * table // * src_table // ------------------------------------- write_bin_log(thd, TRUE, thd->query, thd->query_length); } } // CREATE temp-table LIKE ... // This case is not replicated } else { // SBR: // ------------------------------------- // WL40: To include to tables_info: // * table // * src_table // ------------------------------------- write_bin_log(thd, TRUE, thd->query, thd->query_length); } } <To be continued> ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Guest): Add a mysqlbinlog option to filter updates to certain tables (40)
by worklog-noreply＠askmonty.org 16 Feb '10

16 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add a mysqlbinlog option to filter updates to certain tables CREATION DATE..: Mon, 10 Aug 2009, 13:25 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Psergey CATEGORY.......: Server-Sprint TASK ID........: 40 (http://askmonty.org/worklog/?tid=40) VERSION........: Server-9.x STATUS.........: Cancelled PRIORITY.......: 60 WORKED HOURS...: 32 ESTIMATE.......: 32 (hours remain) ORIG. ESTIMATE.: 48 PROGRESS NOTES: -=-=(Guest - Tue, 16 Feb 2010, 10:23)=-=- Status updated. --- /tmp/wklog.40.old.18300 2010-02-16 10:23:20.000000000 +0200 +++ /tmp/wklog.40.new.18300 2010-02-16 10:23:20.000000000 +0200 @@ -1 +1 @@ -Assigned +Cancelled -=-=(Guest - Wed, 25 Nov 2009, 11:41)=-=- Status updated. --- /tmp/wklog.40.old.5760 2009-11-25 11:41:09.000000000 +0200 +++ /tmp/wklog.40.new.5760 2009-11-25 11:41:09.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +Assigned -=-=(Guest - Wed, 25 Nov 2009, 11:41)=-=- Category updated. --- /tmp/wklog.40.old.5737 2009-11-25 11:41:03.000000000 +0200 +++ /tmp/wklog.40.new.5737 2009-11-25 11:41:03.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Bothorsen - Tue, 17 Nov 2009, 17:20)=-=- Alex is closer to a working patch now. Worked 14 hours and estimate 32 hours remain (original estimate unchanged). -=-=(Bothorsen - Thu, 12 Nov 2009, 13:13)=-=- Work hours by Alexi and Bo + estimated time for the task. Worked 16 hours and estimate 46 hours remain (original estimate increased by 14 hours). -=-=(Alexi - Sun, 08 Nov 2009, 15:18)=-=- Low Level Design modified. --- /tmp/wklog.40.old.15787 2009-11-08 15:18:11.000000000 +0200 +++ /tmp/wklog.40.new.15787 2009-11-08 15:18:11.000000000 +0200 @@ -62,7 +62,7 @@ it considers the query to extent to the end of the event. 2. For 'db' (current db) the trailing zero is redundant since the length is already known. -3. db_len = 0 means that this is the current db. +3. In tables_info, db_len = 0 means that this is the current db. When reading Query events from binary log, we can recognize its format by its post-header length: in extended case the post-header includes 4 @@ -75,6 +75,77 @@ + #define Q_QUERY_LEN_OFFSET Q_STATUS_VARS_LEN_OFFSET + 2 + #define Q_QUERY_TABLES_INFO_LEN_OFFSET Q_QUERY_LEN_OFFSET + 2 + +*********************************************************************** +HELP NEEDED +*********************************************************************** +The QUERY_HEADER_LEN is used in the definition of MAX_LOG_EVENT_HEADER: + +log_event.h +~~~~~~~~~~~ +#define MAX_LOG_EVENT_HEADER ( /* in order of Query_log_event::write */ \ + LOG_EVENT_HEADER_LEN + /* write_header */ \ + QUERY_HEADER_LEN + /* write_data */ \ + EXECUTE_LOAD_QUERY_EXTRA_HEADER_LEN + /*write_post_header_for_derived */ \ + MAX_SIZE_LOG_EVENT_STATUS + /* status */ \ + NAME_LEN + 1) + +which is used only for setting + + thd->variables.max_allowed_packet + mysql->net.max_packet_size + +Looks like (but I am not quite sure) that QUERY_HEADER_LEN can simply +(without making any other changes) be substituted in this definition by +QUERY_HEADER_LEN_EXT. + +Below I list all places where MAX_LOG_EVENT_HEADER is used: + +slave.cc +~~~~~~~~ +static int init_slave_thread(...) +{ ... + /* + Adding MAX_LOG_EVENT_HEADER_LEN to the max_allowed_packet on all + slave threads, since a replication event can become this much larger + than the corresponding packet (query) sent from client to master. + */ + thd->variables.max_allowed_packet= global_system_variables.max_allowed_packet + + MAX_LOG_EVENT_HEADER; /* note, incr over the global not session var */ + ... +} +pthread_handler_t handle_slave_io(...) +{ ... + /* + Adding MAX_LOG_EVENT_HEADER_LEN to the max_packet_size on the I/O + thread, since a replication event can become this much larger than + the corresponding packet (query) sent from client to master. + */ + mysql->net.max_packet_size= thd->net.max_packet_size+= MAX_LOG_EVENT_HEADER; + ... +} + +sql_repl.cc +~~~~~~~~~~~ +void mysql_binlog_send(...) +{ ... + /* + Adding MAX_LOG_EVENT_HEADER_LEN, since a binlog event can become + this larger than the corresponding packet (query) sent + from client to master. + */ + thd->variables.max_allowed_packet+= MAX_LOG_EVENT_HEADER; + ... +} +bool mysql_show_binlog_events(...) +{ ... + /* + to account binlog event header size + */ + thd->variables.max_allowed_packet+= MAX_LOG_EVENT_HEADER; + ... +} + 3. Changes in log events ************************ @@ -84,7 +155,7 @@ This setting is done in Format description event constructor which creates the event for writing to binary log: - if (binlog_with_tables_info) + if (opt_binlog_with_tables_info) post_header_len[QUERY_EVENT - 1] = QUERY_HEADER_LEN_EXT; else post_header_len[QUERY_EVENT - 1] = QUERY_HEADER_LEN; @@ -99,12 +170,12 @@ following manner: switch (binlog_ver) { - case 4: /* MySQL 5.0 and higher */ + #ifndef MYSQL_CLIENT + case 4: /* MySQL 5.0 and higher */ ... -+ #else -+ <error> + break; + #endif + case 1: case 3: ... @@ -132,7 +203,7 @@ -------------------------------- [Creates the event for binlogging] -In case of binlog_with_tables_info = TRUE, set additionally query_len, +In case of opt_binlog_with_tables_info = TRUE, set additionally query_len, tables_info_len, and tables_info members (the constructor is to have an additional 'tables_info' argument). @@ -140,7 +211,7 @@ ---------------- [Writes the event to binlog] -In case of binlog_with_tables_info = TRUE, write additional members +In case of opt_binlog_with_tables_info = TRUE, write additional members (query_len, tables_info_len, and tables_info) to binary log. Also write corresponding whole event length to the common-header. -=-=(Alexi - Sun, 08 Nov 2009, 10:40)=-=- Low Level Design modified. --- /tmp/wklog.40.old.5055 2009-11-08 08:40:02.000000000 +0000 +++ /tmp/wklog.40.new.5055 2009-11-08 08:40:02.000000000 +0000 @@ -3,6 +3,7 @@ 1. Adding --binlog-with-tables-info option ****************************************** +GLOBAL, read-only option. When set, Query events are to be written in the extended binary format which contains tables_info. When not set, Query events -=-=(Alexi - Thu, 05 Nov 2009, 12:37)=-=- Low Level Design modified. --- /tmp/wklog.40.old.11441 2009-11-05 12:37:16.000000000 +0200 +++ /tmp/wklog.40.new.11441 2009-11-05 12:37:16.000000000 +0200 @@ -1,9 +1,18 @@ OPTION: 2.5 Extend Query Events With Tables Info ================================================ -1. Query_log_event Binary Format -******************************** -Changes to be done: +1. Adding --binlog-with-tables-info option +****************************************** + +When set, Query events are to be written in the extended binary +format which contains tables_info. When not set, Query events +are to be written in usual format (without any changes). + +2. Query event extended binary format +************************************* + +When --binlog-with-tables-info is set, Query events are writen +to binary log in the following (extended) format. Query_log_event binary format --------------------------------- @@ -24,12 +33,12 @@ error_code 2 status_vars_len 2 + query_len 2 (see Note 1) -+ tables_info_len 2 (see Note 2) ++ tables_info_len 2 --------------------------------- BODY: status_vars status_vars_len - db db_len + 1 -+ db db_len (see Note 3) ++ db db_len (see Note 2) query query_len + tables_info @@ -37,7 +46,7 @@ --------------------------------- Name Size (bytes) --------------------------------- - db_len 1 (see Note 4) + db_len 1 (see Note 3) db db_len table_name_len 1 table_name table_name_len @@ -48,19 +57,99 @@ table_name table_name_len NOTES -1. Currently Query_log_event format doesn't include 'query_len' because +1. In usual format, Query_log_event doesn't include 'query_len' because it considers the query to extent to the end of the event. -2. If tables_info is not included in the event (--binlog-with-tables-info - option), tables_info_len = 0. -3. The trailing zero is redundant since the length is already known. -4. In case of db = current db, db_len = 0 and db = empty, because - current db is already included in the current event format. +2. For 'db' (current db) the trailing zero is redundant since the length + is already known. +3. db_len = 0 means that this is the current db. + +When reading Query events from binary log, we can recognize its format +by its post-header length: in extended case the post-header includes 4 +additional bytes. + + #define QUERY_HEADER_LEN (QUERY_HEADER_MINIMAL_LEN + 4) ++ #define QUERY_HEADER_LEN_EXT (QUERY_HEADER_LEN + 4) + ... + #define Q_STATUS_VARS_LEN_OFFSET 11 ++ #define Q_QUERY_LEN_OFFSET Q_STATUS_VARS_LEN_OFFSET + 2 ++ #define Q_QUERY_TABLES_INFO_LEN_OFFSET Q_QUERY_LEN_OFFSET + 2 + +3. Changes in log events +************************ + +3.1. Format description event +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Changes needed here concern setting post-header length for Query events. +This setting is done in Format description event constructor which creates +the event for writing to binary log: + + if (binlog_with_tables_info) + post_header_len[QUERY_EVENT - 1] = QUERY_HEADER_LEN_EXT; + else + post_header_len[QUERY_EVENT - 1] = QUERY_HEADER_LEN; + +This change is to be done only for case binlog_ver = 4. + +NOTE. The refered above constructor is allowed to be invoked in a client +context for creating "artificial" Format description events in case of +MySQL < 5.0 (e.g. see mysqlbinlog code). To avoid compilation problems +(because of 'binlog_with_tables_info') and taking into account the +"MySQL < 5.0" restriction, we have to #ifdef out the above code in +following manner: + + switch (binlog_ver) { + case 4: /* MySQL 5.0 and higher */ ++ #ifndef MYSQL_CLIENT + ... ++ #else ++ <error> ++ #endif + case 1: + case 3: + ... + } + +3.2. Query event +~~~~~~~~~~~~~~~~ +Changes needed here include adding tables_info and tables_info_len +members (member for query length already exists) and modifying the +following function-members: + +Query_log_event(buf) constructor +-------------------------------- +[Parses binary format written to the 'buf'] + +Getting post-header length from the Format description event (passed +to the constructor as an argument), define whether buf contains an +extended or usual Query event and parse the buf contents accordingly. + +NOTE. Defining Query event format here should be done with taking into +account that this constructor can be called within a Query-derived +event with the event_type argument != QUERY_EVENT. + +Query_log_event(thd) constructor +-------------------------------- +[Creates the event for binlogging] + +In case of binlog_with_tables_info = TRUE, set additionally query_len, +tables_info_len, and tables_info members (the constructor is to have +an additional 'tables_info' argument). + +write() function +---------------- +[Writes the event to binlog] + +In case of binlog_with_tables_info = TRUE, write additional members +(query_len, tables_info_len, and tables_info) to binary log. Also +write corresponding whole event length to the common-header. + +<To be continued> -2. Where to get tables info from? +4. Where to get tables info from? ********************************* -2.1. Case study: CREATE TABLE -****************************** +4.1. Case study: CREATE TABLE +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *** CREATE TABLE table [SELECT ...] @@ -129,4 +218,4 @@ } } -To be continued +<To be continued> -=-=(Alexi - Wed, 04 Nov 2009, 10:21)=-=- Low Level Design modified. --- /tmp/wklog.40.old.6734 2009-11-04 10:21:20.000000000 +0200 +++ /tmp/wklog.40.new.6734 2009-11-04 10:21:20.000000000 +0200 @@ -21,9 +21,9 @@ slave_proxy_id 4 exec_time 4 db_len 1 -+ query_len 2 (see Note 1) error_code 2 status_vars_len 2 ++ query_len 2 (see Note 1) + tables_info_len 2 (see Note 2) --------------------------------- BODY: -=-=(Alexi - Tue, 03 Nov 2009, 11:19)=-=- Low Level Design modified. --- /tmp/wklog.40.old.7187 2009-11-03 11:19:22.000000000 +0200 +++ /tmp/wklog.40.new.7187 2009-11-03 11:19:22.000000000 +0200 @@ -1 +1,132 @@ +OPTION: 2.5 Extend Query Events With Tables Info +================================================ +1. Query_log_event Binary Format +******************************** +Changes to be done: + + Query_log_event binary format + --------------------------------- + Name Size (bytes) + --------------------------------- + COMMON HEADER: + timestamp 4 + type 1 + server_id 4 + total_size 4 + master_position 4 + flags 2 + --------------------------------- + POST HEADER: + slave_proxy_id 4 + exec_time 4 + db_len 1 ++ query_len 2 (see Note 1) + error_code 2 + status_vars_len 2 ++ tables_info_len 2 (see Note 2) + --------------------------------- + BODY: + status_vars status_vars_len +- db db_len + 1 ++ db db_len (see Note 3) + query query_len ++ tables_info + + tables_info binary format + --------------------------------- + Name Size (bytes) + --------------------------------- + db_len 1 (see Note 4) + db db_len + table_name_len 1 + table_name table_name_len + ... + db_len 1 + db db_len + table_name_len 1 + table_name table_name_len + +NOTES +1. Currently Query_log_event format doesn't include 'query_len' because + it considers the query to extent to the end of the event. +2. If tables_info is not included in the event (--binlog-with-tables-info + option), tables_info_len = 0. +3. The trailing zero is redundant since the length is already known. +4. In case of db = current db, db_len = 0 and db = empty, because + current db is already included in the current event format. + +2. Where to get tables info from? +********************************* + +2.1. Case study: CREATE TABLE +****************************** + +*** CREATE TABLE table [SELECT ...] + + bool mysql_create_table_no_lock( + THD *thd, + const char *db, + const char *table_name, ...) + { + ... + // ------------------------------------- + // WL40: To be included in tables_info: + // * db, table_name + // * thd->lex->query_tables (tables refered to in + // the select-part; empty if no select-part) + // ------------------------------------- + write_bin_log(thd, TRUE, thd->query, thd->query_length); + } + +*** CREATE TABLE table LIKE src-table + + bool mysql_create_like_table( + ... + TABLE_LIST *table, + TABLE_LIST *src_table, + ...) + { + ... + if (thd->current_stmt_binlog_row_based) + { // RBR: In this case we don't replicate temp tables + if (!(create_info->options & HA_LEX_CREATE_TMP_TABLE)) + { + if (src_table->table->s->tmp_table) + { // CREATE normal-table LIKE temp-table: + + // Generate new query without LIKE-part + store_create_info(thd, table, &query, create_info, FALSE); + + // ------------------------------------- + // WL40: To include to tables_info: + // * table (src_table is not included) + // ------------------------------------- + write_bin_log(thd, TRUE, query.ptr(), query.length()); + } + else + { // CREATE normal-table LIKE normal-table + + // ------------------------------------- + // WL40: To include to log_tables_info: + // * table + // * src_table + // ------------------------------------- + write_bin_log(thd, TRUE, thd->query, thd->query_length); + } + } + // CREATE temp-table LIKE ... + // This case is not replicated + } + else + { // SBR: + // ------------------------------------- + // WL40: To include to tables_info: + // * table + // * src_table + // ------------------------------------- + write_bin_log(thd, TRUE, thd->query, thd->query_length); + } + } + +To be continued ------------------------------------------------------------ -=-=(View All Progress Notes, 17 total)=-=- http://askmonty.org/worklog/index.pl?tid=40&nolimit=1 DESCRIPTION: Replication slave can be set to filter updates to certain tables with --replicate-[wild-]{do,ignore}-table options. This task is about adding similar functionality to mysqlbinlog. HIGH-LEVEL SPECIFICATION: 1. Context ---------- (See http://askmonty.org/wiki/index.php/Scratch/ReplicationOptions for global overview) At the moment, the server has these replication slave options: --replicate-do-table=db.tbl --replicate-ignore-table=db.tbl --replicate-wild-do-table=pattern.pattern --replicate-wild-ignore-table=pattern.pattern They affect both RBR and SBR events. SBR events are checked after the statement has been parsed, the server iterates over list of used tables and checks them againist --replicate instructions. What is interesting is that this scheme still allows to update the ignored table through a VIEW. 2. Table filtering in mysqlbinlog --------------------------------- Per-table filtering of RBR events is easy (as it is relatively easy to extract the name of the table that the event applies to). Per-table filtering of SBR events is hard, as generally it is not apparent which tables the statement refers to. This opens possible options: 2.1 Put the parser into mysqlbinlog ----------------------------------- Once we have a full parser in mysqlbinlog, we'll be able to check which tables are used by a statement, and will allow to show behaviour identical to those that one obtains when using --replicate-* slave options. (It is not clear how much effort is needed to put the parser into mysqlbinlog. Any guesses?) 2.2 Use dumb regexp match ------------------------- Use a really dumb approach. A query is considered to be modifying table X if it matches an expression CREATE TABLE $tablename DROP $tablename UPDATE ...$tablename ... SET // here '...' can't contain the word 'SET' DELETE ...$tablename ... WHERE // same as above ALTER TABLE $tablename .. etc (go get from the grammar) .. The advantage over doing the same in awk is that mysqlbinlog will also process RBR statements, and together with that will provide a working solution for those who are careful with their table names not mixing with string constants and such. (TODO: string constants are of particular concern as they come from [potentially hostile] users, unlike e.g. table aliases which come from [not hostile] developers. Remove also all string constants before attempting to do match?) 2.3 Have the master put annotations ----------------------------------- We could add a master option so that it injects into query a mark that tells which tables the query will affect, e.g. for the query UPDATE t1 LEFT JOIN db3.t2 ON ... WHERE ... the binlog will have /* !mysqlbinlog: updates t1,db3.t2 */ UPDATE t1 LEFT JOIN ... and further processing in mysqlbinlog will be trivial. 2.4 Implement server functionality to ignore certain tables ----------------------------------------------------------- We could add a general facility in the server to ignore certain tables: SET SESSION ignored_tables = "db1.t1,db2.t2"; This would work similar to --replicate-ignore-table, but in a general way not restricted to the slave SQL thread. It would then be trivial for mysqlbinlog to add such statements at the start of the output, or probably the user could just do it manually with no need for additional options for mysqlbinlog. It might be useful to integrate this with the code that already handles --replicate-ignore-db and similar slave options. 2.5 Extend Query Events With Tables Info ---------------------------------------- We could extend query events structure with a tables info - a list of tables which the query refers to: <current query event structure> tables_info_len dbase_len dbase table_len table ... dbase_len dbase table_len table Note. In case of <dbase> = current data base, we can set dbase_len = 0 and dbase = empty because current query event structure already includes current data base name. Note. Possibly it is reasonable also to add a --binlog-with-tables-info option which defines whether tables info must be included to the query events. LOW-LEVEL DESIGN: OPTION: 2.5 Extend Query Events With Tables Info ================================================ 1. Adding --binlog-with-tables-info option ****************************************** GLOBAL, read-only option. When set, Query events are to be written in the extended binary format which contains tables_info. When not set, Query events are to be written in usual format (without any changes). 2. Query event extended binary format ************************************* When --binlog-with-tables-info is set, Query events are writen to binary log in the following (extended) format. Query_log_event binary format --------------------------------- Name Size (bytes) --------------------------------- COMMON HEADER: timestamp 4 type 1 server_id 4 total_size 4 master_position 4 flags 2 --------------------------------- POST HEADER: slave_proxy_id 4 exec_time 4 db_len 1 error_code 2 status_vars_len 2 + query_len 2 (see Note 1) + tables_info_len 2 --------------------------------- BODY: status_vars status_vars_len - db db_len + 1 + db db_len (see Note 2) query query_len + tables_info tables_info binary format --------------------------------- Name Size (bytes) --------------------------------- db_len 1 (see Note 3) db db_len table_name_len 1 table_name table_name_len ... db_len 1 db db_len table_name_len 1 table_name table_name_len NOTES 1. In usual format, Query_log_event doesn't include 'query_len' because it considers the query to extent to the end of the event. 2. For 'db' (current db) the trailing zero is redundant since the length is already known. 3. In tables_info, db_len = 0 means that this is the current db. When reading Query events from binary log, we can recognize its format by its post-header length: in extended case the post-header includes 4 additional bytes. #define QUERY_HEADER_LEN (QUERY_HEADER_MINIMAL_LEN + 4) + #define QUERY_HEADER_LEN_EXT (QUERY_HEADER_LEN + 4) ... #define Q_STATUS_VARS_LEN_OFFSET 11 + #define Q_QUERY_LEN_OFFSET Q_STATUS_VARS_LEN_OFFSET + 2 + #define Q_QUERY_TABLES_INFO_LEN_OFFSET Q_QUERY_LEN_OFFSET + 2 *********************************************************************** HELP NEEDED *********************************************************************** The QUERY_HEADER_LEN is used in the definition of MAX_LOG_EVENT_HEADER: log_event.h ~~~~~~~~~~~ #define MAX_LOG_EVENT_HEADER ( /* in order of Query_log_event::write */ \ LOG_EVENT_HEADER_LEN + /* write_header */ \ QUERY_HEADER_LEN + /* write_data */ \ EXECUTE_LOAD_QUERY_EXTRA_HEADER_LEN + /*write_post_header_for_derived */ \ MAX_SIZE_LOG_EVENT_STATUS + /* status */ \ NAME_LEN + 1) which is used only for setting thd->variables.max_allowed_packet mysql->net.max_packet_size Looks like (but I am not quite sure) that QUERY_HEADER_LEN can simply (without making any other changes) be substituted in this definition by QUERY_HEADER_LEN_EXT. Below I list all places where MAX_LOG_EVENT_HEADER is used: slave.cc ~~~~~~~~ static int init_slave_thread(...) { ... /* Adding MAX_LOG_EVENT_HEADER_LEN to the max_allowed_packet on all slave threads, since a replication event can become this much larger than the corresponding packet (query) sent from client to master. */ thd->variables.max_allowed_packet= global_system_variables.max_allowed_packet + MAX_LOG_EVENT_HEADER; /* note, incr over the global not session var */ ... } pthread_handler_t handle_slave_io(...) { ... /* Adding MAX_LOG_EVENT_HEADER_LEN to the max_packet_size on the I/O thread, since a replication event can become this much larger than the corresponding packet (query) sent from client to master. */ mysql->net.max_packet_size= thd->net.max_packet_size+= MAX_LOG_EVENT_HEADER; ... } sql_repl.cc ~~~~~~~~~~~ void mysql_binlog_send(...) { ... /* Adding MAX_LOG_EVENT_HEADER_LEN, since a binlog event can become this larger than the corresponding packet (query) sent from client to master. */ thd->variables.max_allowed_packet+= MAX_LOG_EVENT_HEADER; ... } bool mysql_show_binlog_events(...) { ... /* to account binlog event header size */ thd->variables.max_allowed_packet+= MAX_LOG_EVENT_HEADER; ... } 3. Changes in log events ************************ 3.1. Format description event ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Changes needed here concern setting post-header length for Query events. This setting is done in Format description event constructor which creates the event for writing to binary log: if (opt_binlog_with_tables_info) post_header_len[QUERY_EVENT - 1] = QUERY_HEADER_LEN_EXT; else post_header_len[QUERY_EVENT - 1] = QUERY_HEADER_LEN; This change is to be done only for case binlog_ver = 4. NOTE. The refered above constructor is allowed to be invoked in a client context for creating "artificial" Format description events in case of MySQL < 5.0 (e.g. see mysqlbinlog code). To avoid compilation problems (because of 'binlog_with_tables_info') and taking into account the "MySQL < 5.0" restriction, we have to #ifdef out the above code in following manner: switch (binlog_ver) { + #ifndef MYSQL_CLIENT case 4: /* MySQL 5.0 and higher */ ... break; + #endif case 1: case 3: ... } 3.2. Query event ~~~~~~~~~~~~~~~~ Changes needed here include adding tables_info and tables_info_len members (member for query length already exists) and modifying the following function-members: Query_log_event(buf) constructor -------------------------------- [Parses binary format written to the 'buf'] Getting post-header length from the Format description event (passed to the constructor as an argument), define whether buf contains an extended or usual Query event and parse the buf contents accordingly. NOTE. Defining Query event format here should be done with taking into account that this constructor can be called within a Query-derived event with the event_type argument != QUERY_EVENT. Query_log_event(thd) constructor -------------------------------- [Creates the event for binlogging] In case of opt_binlog_with_tables_info = TRUE, set additionally query_len, tables_info_len, and tables_info members (the constructor is to have an additional 'tables_info' argument). write() function ---------------- [Writes the event to binlog] In case of opt_binlog_with_tables_info = TRUE, write additional members (query_len, tables_info_len, and tables_info) to binary log. Also write corresponding whole event length to the common-header. <To be continued> 4. Where to get tables info from? ********************************* 4.1. Case study: CREATE TABLE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *** CREATE TABLE table [SELECT ...] bool mysql_create_table_no_lock( THD *thd, const char *db, const char *table_name, ...) { ... // ------------------------------------- // WL40: To be included in tables_info: // * db, table_name // * thd->lex->query_tables (tables refered to in // the select-part; empty if no select-part) // ------------------------------------- write_bin_log(thd, TRUE, thd->query, thd->query_length); } *** CREATE TABLE table LIKE src-table bool mysql_create_like_table( ... TABLE_LIST *table, TABLE_LIST *src_table, ...) { ... if (thd->current_stmt_binlog_row_based) { // RBR: In this case we don't replicate temp tables if (!(create_info->options & HA_LEX_CREATE_TMP_TABLE)) { if (src_table->table->s->tmp_table) { // CREATE normal-table LIKE temp-table: // Generate new query without LIKE-part store_create_info(thd, table, &query, create_info, FALSE); // ------------------------------------- // WL40: To include to tables_info: // * table (src_table is not included) // ------------------------------------- write_bin_log(thd, TRUE, query.ptr(), query.length()); } else { // CREATE normal-table LIKE normal-table // ------------------------------------- // WL40: To include to log_tables_info: // * table // * src_table // ------------------------------------- write_bin_log(thd, TRUE, thd->query, thd->query_length); } } // CREATE temp-table LIKE ... // This case is not replicated } else { // SBR: // ------------------------------------- // WL40: To include to tables_info: // * table // * src_table // ------------------------------------- write_bin_log(thd, TRUE, thd->query, thd->query_length); } } <To be continued> ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Rev 2758: Subquery optimization backport: in file:///home/psergey/dev/maria-5.3-subqueries-r6/
by Sergey Petrunya 15 Feb '10

15 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r6/ ------------------------------------------------------------ revno: 2758 revision-id: psergey(a)askmonty.org-20100215215306-hc0levm9ag1lv1b1 parent: psergey(a)askmonty.org-20100212181041-5rwekm1wpvwaikkx committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r6 timestamp: Tue 2010-02-16 00:53:06 +0300 message: Subquery optimization backport: - Factor out subquery code into sql/opt_subselect.{h,cc} - Stop using the term "confluent" (was used due to misreading the dictionary) Diff too large for email (8074 lines, the limit is 1000).

1 0

[Maria-developers] Rev 2734: Maria WL#61 in file:///Users/bell/maria/bzr/work-maria-5.2-engine/
by sanja＠askmonty.org 15 Feb '10

15 Feb '10

At file:///Users/bell/maria/bzr/work-maria-5.2-engine/ ------------------------------------------------------------ revno: 2734 revision-id: sanja(a)askmonty.org-20100215074703-tqcssnpbf43grygo parent: psergey(a)askmonty.org-20091202142609-18bp41q8mejxl47t committer: sanja(a)askmonty.org branch nick: work-maria-5.2-engine timestamp: Mon 2010-02-15 09:47:03 +0200 message: Maria WL#61 Interface for maria extensions. Additional information about plugins (maturity and string version) interface for maria extensions. === modified file 'CMakeLists.txt' --- a/CMakeLists.txt 2009-10-03 19:24:13 +0000 +++ b/CMakeLists.txt 2010-02-15 07:47:03 +0000 @@ -251,6 +251,7 @@ IF (ENGINE_BUILD_TYPE STREQUAL "STATIC") SET (mysql_plugin_defs "${mysql_plugin_defs},builtin_${PLUGIN_NAME}_plugin") + SET (mariadb_extra_plugin_defs "${mariadb_extra_plugin_defs},builtin_mariadb_${PLUGIN_NAME}_plugin") SET (MYSQLD_STATIC_ENGINE_LIBS ${MYSQLD_STATIC_ENGINE_LIBS} ${PLUGIN_NAME}) SET (STORAGE_ENGINE_DEFS "${STORAGE_ENGINE_DEFS} -DWITH_${ENGINE}_STORAGE_ENGINE") SET (WITH_${ENGINE}_STORAGE_ENGINE TRUE) @@ -269,6 +270,7 @@ IF(NOT WITHOUT_PARTITION_STORAGE_ENGINE) SET (STORAGE_ENGINE_DEFS "${STORAGE_ENGINE_DEFS} -DWITH_PARTITION_STORAGE_ENGINE") SET (mysql_plugin_defs "${mysql_plugin_defs},builtin_partition_plugin") + SET (mariadb_extra_plugin_defs "${mariadb_extra_plugin_defs},builtin_mariadb_partition_plugin") ENDIF(NOT WITHOUT_PARTITION_STORAGE_ENGINE) # Special handling for tmp tables with the maria engine === modified file 'config/ac-macros/plugins.m4' --- a/config/ac-macros/plugins.m4 2009-04-25 10:05:32 +0000 +++ b/config/ac-macros/plugins.m4 2010-02-15 07:47:03 +0000 @@ -461,6 +461,7 @@ ]) ]) mysql_plugin_defs="$mysql_plugin_defs, [builtin_]$2[_plugin]" + mariadb_extra_plugin_defs="$mariadb_extra_plugin_defs, [builtin_mariadb_]$2[_plugin]" [with_plugin_]$2=yes AC_MSG_RESULT([yes]) m4_ifdef([$11],[ === modified file 'configure.in' --- a/configure.in 2009-11-12 04:31:28 +0000 +++ b/configure.in 2010-02-15 07:47:03 +0000 @@ -2842,6 +2842,7 @@ AC_SUBST(mysql_plugin_dirs) AC_SUBST(mysql_plugin_libs) AC_SUBST(mysql_plugin_defs) +AC_SUBST(mariadb_extra_plugin_defs) # Now that sql_client_dirs and sql_server_dirs are stable, determine the union. === modified file 'include/mysql/plugin.h' --- a/include/mysql/plugin.h 2009-09-07 20:50:10 +0000 +++ b/include/mysql/plugin.h 2010-02-15 07:47:03 +0000 @@ -65,7 +65,10 @@ Plugin API. Common for all plugin types. */ +/* MySQL plugin interface version */ #define MYSQL_PLUGIN_INTERFACE_VERSION 0x0100 +/* MariaDB extentsion interface version */ +#define MARIAEXT_PLUGIN_INTERFACE_VERSION 0x0100 /* The allowable types of plugins @@ -86,6 +89,21 @@ #define PLUGIN_LICENSE_GPL_STRING "GPL" #define PLUGIN_LICENSE_BSD_STRING "BSD" +/* definitions of code maturity for plugins */ +#define PLUGIN_MATURITY_UNKNOWN 0 +#define PLUGIN_MATURITY_TEST 1 +#define PLUGIN_MATURITY_ALPHA 2 +#define PLUGIN_MATURITY_BETA 3 +#define PLUGIN_MATURITY_GAMMA 4 +#define PLUGIN_MATURITY_RELEASE 5 + +#define PLUGIN_MATURITY_UNKNOWN_STR "Unknown" +#define PLUGIN_MATURITY_TEST_STR "Test" +#define PLUGIN_MATURITY_ALPHA_STR "Alpha" +#define PLUGIN_MATURITY_BETA_STR "Beta" +#define PLUGIN_MATURITY_GAMMA_STR "Gamma" +#define PLUGIN_MATURITY_RELEASE_STR "Release" + /* Macros for beginning and ending plugin declarations. Between mysql_declare_plugin and mysql_declare_plugin_end there should @@ -94,15 +112,29 @@ #ifndef MYSQL_DYNAMIC_PLUGIN + #define __MYSQL_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ int VERSION= MYSQL_PLUGIN_INTERFACE_VERSION; \ int PSIZE= sizeof(struct st_mysql_plugin); \ struct st_mysql_plugin DECLS[]= { + +#define __MARIAEXT_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ +int VERSION= MARIAEXT_PLUGIN_INTERFACE_VERSION; \ +int PSIZE= sizeof(struct st_mariaext_plugin); \ +struct st_mariaext_plugin DECLS[]= { + #else + #define __MYSQL_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ MYSQL_PLUGIN_EXPORT int _mysql_plugin_interface_version_= MYSQL_PLUGIN_INTERFACE_VERSION; \ MYSQL_PLUGIN_EXPORT int _mysql_sizeof_struct_st_plugin_= sizeof(struct st_mysql_plugin); \ MYSQL_PLUGIN_EXPORT struct st_mysql_plugin _mysql_plugin_declarations_[]= { + +#define __MARIAEXT_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ +MYSQL_PLUGIN_EXPORT int _mariaext_plugin_interface_version_= MARIAEXT_PLUGIN_INTERFACE_VERSION; \ +MYSQL_PLUGIN_EXPORT int _mariaext_sizeof_struct_st_plugin_= sizeof(struct st_mariaext_plugin); \ +MYSQL_PLUGIN_EXPORT struct st_mariaext_plugin _mariaext_plugin_declarations_[]= { + #endif #define mysql_declare_plugin(NAME) \ @@ -111,7 +143,14 @@ builtin_ ## NAME ## _sizeof_struct_st_plugin, \ builtin_ ## NAME ## _plugin) +#define mariaext_declare_plugin(NAME) \ +__MARIAEXT_DECLARE_PLUGIN(NAME, \ + builtin_mariadb_ ## NAME ## _plugin_interface_version, \ + builtin_mariadb_ ## NAME ## _sizeof_struct_st_plugin, \ + builtin_mariadb_ ## NAME ## _plugin) + #define mysql_declare_plugin_end ,{0,0,0,0,0,0,0,0,0,0,0,0}} +#define mariaext_declare_plugin_end ,{0,0}} /* declarations for SHOW STATUS support in plugins @@ -407,6 +446,16 @@ void * __reserved1; /* reserved for dependency checking */ }; +/* + MariaDB extension for plugins declaration structure. +*/ + +struct st_mariaext_plugin +{ + const char *version_info; /* plugin version string */ + int maturity; /* HA_PLUGIN_MATURITY_XXX */ +}; + /************************************************************************* API for Full-text parser plugin. (MYSQL_FTPARSER_PLUGIN) */ === modified file 'include/mysql/plugin.h.pp' --- a/include/mysql/plugin.h.pp 2008-10-10 15:28:41 +0000 +++ b/include/mysql/plugin.h.pp 2010-02-15 07:47:03 +0000 @@ -46,6 +46,11 @@ struct st_mysql_sys_var **system_vars; void * __reserved1; }; +struct st_mariaext_plugin +{ + const char *version_info; + int maturity; +}; enum enum_ftparser_mode { MYSQL_FTPARSER_SIMPLE_MODE= 0, === modified file 'mysql-test/r/information_schema.result' --- a/mysql-test/r/information_schema.result 2009-10-19 17:14:48 +0000 +++ b/mysql-test/r/information_schema.result 2010-02-15 07:47:03 +0000 @@ -1175,7 +1175,7 @@ group by column_type order by num; column_type group_concat(table_schema, '.', table_name) num varchar(27) information_schema.COLUMNS 1 -varchar(7) information_schema.ROUTINES,information_schema.VIEWS 2 +varchar(7) information_schema.PLUGINS,information_schema.ROUTINES,information_schema.VIEWS 3 varchar(20) information_schema.FILES,information_schema.FILES,information_schema.PLUGINS,information_schema.PLUGINS,information_schema.PLUGINS,information_schema.PROFILING 6 create table t1(f1 char(1) not null, f2 char(9) not null) default character set utf8; === modified file 'plugin/daemon_example/daemon_example.cc' --- a/plugin/daemon_example/daemon_example.cc 2007-06-27 14:49:12 +0000 +++ b/plugin/daemon_example/daemon_example.cc 2010-02-15 07:47:03 +0000 @@ -200,3 +200,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(daemon_example) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_TEST /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'plugin/fulltext/plugin_example.c' --- a/plugin/fulltext/plugin_example.c 2007-04-26 19:26:04 +0000 +++ b/plugin/fulltext/plugin_example.c 2010-02-15 07:47:03 +0000 @@ -270,4 +270,10 @@ NULL } mysql_declare_plugin_end; +mariaext_declare_plugin(ftexample) +{ + "0.01", /* string version */ + PLUGIN_MATURITY_TEST /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'sql/ha_ndbcluster.cc' --- a/sql/ha_ndbcluster.cc 2009-09-07 20:50:10 +0000 +++ b/sql/ha_ndbcluster.cc 2010-02-15 07:47:03 +0000 @@ -10561,5 +10561,11 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(ndbcluster) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_BETA /* maturity */ +} +mariaext_declare_plugin_end; #endif === modified file 'sql/ha_partition.cc' --- a/sql/ha_partition.cc 2009-11-12 04:31:28 +0000 +++ b/sql/ha_partition.cc 2010-02-15 07:47:03 +0000 @@ -6510,5 +6510,11 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(partition) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; #endif === modified file 'sql/log.cc' --- a/sql/log.cc 2009-11-12 04:31:28 +0000 +++ b/sql/log.cc 2010-02-15 07:47:03 +0000 @@ -5795,3 +5795,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(binlog) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'sql/sql_builtin.cc.in' --- a/sql/sql_builtin.cc.in 2006-12-31 01:29:11 +0000 +++ b/sql/sql_builtin.cc.in 2010-02-15 07:47:03 +0000 @@ -16,6 +16,7 @@ #include <mysql/plugin.h> typedef struct st_mysql_plugin builtin_plugin[]; +typedef struct st_mariaext_plugin builtin_mariadb_plugin[]; extern builtin_plugin builtin_binlog_plugin@mysql_plugin_defs@; @@ -25,3 +26,10 @@ builtin_binlog_plugin@mysql_plugin_defs@,(struct st_mysql_plugin *)0 }; +extern builtin_mariadb_plugin + builtin_mariadb_binlog_plugin@mariadb_extra_plugin_defs@; + +struct st_mariaext_plugin *mysqld_bltnmexts[]= +{ + builtin_mariadb_binlog_plugin@mariadb_extra_plugin_defs@,(struct st_mariaext_plugin *)0 +}; === modified file 'sql/sql_plugin.cc' --- a/sql/sql_plugin.cc 2009-11-12 04:31:28 +0000 +++ b/sql/sql_plugin.cc 2010-02-15 07:47:03 +0000 @@ -28,6 +28,9 @@ #endif extern struct st_mysql_plugin *mysqld_builtins[]; +extern struct st_mariaext_plugin *mysqld_bltnmexts[]; +static st_mariaext_plugin no_mariaext[2]= {{"Unknown", 0}, {0, 0}}; +static st_mariaext_plugin *empty_mariaext= no_mariaext; /** @note The order of the enumeration is critical. @@ -82,6 +85,14 @@ "_mysql_sizeof_struct_st_plugin_"; static const char *plugin_declarations_sym= "_mysql_plugin_declarations_"; static int min_plugin_interface_version= MYSQL_PLUGIN_INTERFACE_VERSION & ~0xFF; +static const char *mariaext_plugin_interface_version_sym= + "_mariaext_plugin_interface_version_"; +static const char *mariaext_sizeof_st_plugin_sym= + "_mariaext_sizeof_struct_st_plugin_"; +static const char *mariaext_plugin_declarations_sym= + "_mariaext_plugin_declarations_"; +static int min_mariaext_plugin_interface_version= + MARIAEXT_PLUGIN_INTERFACE_VERSION & ~0xFF; #endif /* Note that 'int version' must be the first field of every plugin @@ -352,6 +363,7 @@ char dlpath[FN_REFLEN]; uint plugin_dir_len, dummy_errors, dlpathlen; struct st_plugin_dl *tmp, plugin_dl; + struct st_mariaext_plugin *mariaext= no_mariaext; void *sym; DBUG_ENTER("plugin_dl_add"); plugin_dir_len= strlen(opt_plugin_dir); @@ -507,6 +519,15 @@ files_charset_info, dl->str, dl->length, system_charset_info, &dummy_errors); plugin_dl.dl.str[plugin_dl.dl.length]= 0; + + if ((sym= dlsym(plugin_dl.handle, mariaext_plugin_interface_version_sym)) && + (*(int *)sym == MARIAEXT_PLUGIN_INTERFACE_VERSION) && + (sym= dlsym(plugin_dl.handle, mariaext_plugin_declarations_sym))) + { + mariaext= (struct st_mariaext_plugin *) sym; + } + plugin_dl.mariaext= mariaext; + /* Add this dll to array */ if (! (tmp= plugin_dl_insert_or_reuse(&plugin_dl))) { @@ -719,6 +740,7 @@ { struct st_plugin_int tmp; struct st_mysql_plugin *plugin; + struct st_mariaext_plugin *ext, *mariaext; DBUG_ENTER("plugin_add"); if (plugin_find_internal(name, MYSQL_ANY_PLUGIN)) { @@ -732,9 +754,22 @@ bzero((char*) &tmp, sizeof(tmp)); if (! (tmp.plugin_dl= plugin_dl_add(dl, report))) DBUG_RETURN(TRUE); + /* Find plugin by name */ - for (plugin= tmp.plugin_dl->plugins; plugin->info; plugin++) + for (plugin= tmp.plugin_dl->plugins, ext= tmp.plugin_dl->mariaext; + plugin->info; + plugin++, ext++) { + mariaext= ext; + if (!ext->version_info) + { + /* + Plugin didn't have any mariadb extensions; Use default one and + reset counter to do same for next internal plugin. + */ + mariaext= empty_mariaext; + ext--; + } uint name_len= strlen(plugin->name); if (plugin->type >= 0 && plugin->type < MYSQL_MAX_PLUGIN_TYPE_NUM && ! my_strnncoll(system_charset_info, @@ -759,6 +794,7 @@ goto err; } tmp.plugin= plugin; + tmp.mariaext= mariaext; tmp.name.str= (char *)plugin->name; tmp.name.length= name_len; tmp.ref_count= 0; @@ -1121,7 +1157,9 @@ uint i; bool is_myisam; struct st_mysql_plugin **builtins; + struct st_mariaext_plugin **bltnmexts; struct st_mysql_plugin *plugin; + struct st_mariaext_plugin *ext, *mariaext; struct st_plugin_int tmp, *plugin_ptr, **reap; MEM_ROOT tmp_root; bool reaped_mandatory_plugin= FALSE; @@ -1160,10 +1198,29 @@ /* First we register builtin plugins */ - for (builtins= mysqld_builtins; *builtins; builtins++) + for (builtins= mysqld_builtins, bltnmexts= mysqld_bltnmexts; + *builtins; + builtins++, bltnmexts++) { - for (plugin= *builtins; plugin->info; plugin++) + /* extensions should be the same numbers as static plugins) */ + DBUG_ASSERT(*bltnmexts); + for (plugin= *builtins, ext= *bltnmexts; + plugin->info; + plugin++, ext++) { + + /* in case if plugin describe less extensions then plugins */ + mariaext= ext; + if (!ext->version_info) + { + /* + Plugin didn't have any mariadb extensions; Use default one and + reset counter to do same for next internal plugin. + */ + mariaext= empty_mariaext; + ext--; + } + if (opt_ignore_builtin_innodb && !my_strnncoll(&my_charset_latin1, (const uchar*) plugin->name, 6, (const uchar*) "InnoDB", 6)) @@ -1186,6 +1243,7 @@ #endif bzero(&tmp, sizeof(tmp)); tmp.plugin= plugin; + tmp.mariaext= mariaext; tmp.name.str= (char *)plugin->name; tmp.name.length= strlen(plugin->name); tmp.state= 0; === modified file 'sql/sql_plugin.h' --- a/sql/sql_plugin.h 2009-05-14 12:03:33 +0000 +++ b/sql/sql_plugin.h 2010-02-15 07:47:03 +0000 @@ -63,6 +63,7 @@ LEX_STRING dl; void *handle; struct st_mysql_plugin *plugins; + struct st_mariaext_plugin *mariaext; int version; uint ref_count; /* number of plugins loaded from the library */ }; @@ -74,6 +75,7 @@ LEX_STRING name; struct st_mysql_plugin *plugin; struct st_plugin_dl *plugin_dl; + struct st_mariaext_plugin *mariaext; uint state; uint ref_count; /* number of threads using the plugin */ void *data; /* plugin type specific, e.g. handlerton */ @@ -95,6 +97,7 @@ #define plugin_name(pi) (&((pi)->name)) #define plugin_state(pi) ((pi)->state) #define plugin_equals(p1,p2) ((p1) == (p2)) +#define plugin_ext(pi) ((pi)->mariaext) #else typedef struct st_plugin_int **plugin_ref; #define plugin_decl(pi) ((pi)[0]->plugin) @@ -103,6 +106,8 @@ #define plugin_name(pi) (&((pi)[0]->name)) #define plugin_state(pi) ((pi)[0]->state) #define plugin_equals(p1,p2) ((p1) && (p2) && (p1)[0] == (p2)[0]) +#define plugin_ext(pi) ((pi)[0]->mariaext) + #endif typedef int (*plugin_type_init)(struct st_plugin_int *); === modified file 'sql/sql_show.cc' --- a/sql/sql_show.cc 2009-11-12 04:31:28 +0000 +++ b/sql/sql_show.cc 2010-02-15 07:47:03 +0000 @@ -94,12 +94,21 @@ return my_snprintf(buf, buf_length, "%d.%d", version>>8,version&0xff); } +static const LEX_STRING maturity_name[]={ + { C_STRING_WITH_LEN(PLUGIN_MATURITY_UNKNOWN_STR) }, + { C_STRING_WITH_LEN(PLUGIN_MATURITY_TEST_STR) }, + { C_STRING_WITH_LEN(PLUGIN_MATURITY_ALPHA_STR) }, + { C_STRING_WITH_LEN(PLUGIN_MATURITY_BETA_STR) }, + { C_STRING_WITH_LEN(PLUGIN_MATURITY_GAMMA_STR) }, + { C_STRING_WITH_LEN(PLUGIN_MATURITY_RELEASE_STR) }}; + static my_bool show_plugins(THD *thd, plugin_ref plugin, void *arg) { TABLE *table= (TABLE*) arg; struct st_mysql_plugin *plug= plugin_decl(plugin); struct st_plugin_dl *plugin_dl= plugin_dlib(plugin); + struct st_mariaext_plugin *mariaext= plugin_ext(plugin); CHARSET_INFO *cs= system_charset_info; char version_buf[20]; @@ -186,6 +195,26 @@ } table->field[9]->set_notnull(); + if ((uint) mariaext->maturity <= PLUGIN_MATURITY_RELEASE) + table->field[10]->store(maturity_name[mariaext->maturity].str, + maturity_name[mariaext->maturity].length, + cs); + else + { + DBUG_ASSERT(0); + table->field[10]->store("Unknown", 7, cs); + } + table->field[10]->set_notnull(); + + if (mariaext->version_info) + { + table->field[11]->store(mariaext->version_info, + strlen(mariaext->version_info), cs); + table->field[11]->set_notnull(); + } + else + table->field[11]->set_null(); + return schema_table_store_record(thd, table); } @@ -6990,6 +7019,8 @@ {"PLUGIN_AUTHOR", NAME_CHAR_LEN, MYSQL_TYPE_STRING, 0, 1, 0, SKIP_OPEN_TABLE}, {"PLUGIN_DESCRIPTION", 65535, MYSQL_TYPE_STRING, 0, 1, 0, SKIP_OPEN_TABLE}, {"PLUGIN_LICENSE", 80, MYSQL_TYPE_STRING, 0, 1, "License", SKIP_OPEN_TABLE}, + {"PLUGIN_MATURITY", 7, MYSQL_TYPE_STRING, 0, 1, 0, SKIP_OPEN_TABLE}, + {"PLUGIN_AUTH_VERSION", 80, MYSQL_TYPE_STRING, 0, 1, 0, SKIP_OPEN_TABLE}, {0, 0, MYSQL_TYPE_STRING, 0, 0, 0, SKIP_OPEN_TABLE} }; === modified file 'storage/archive/ha_archive.cc' --- a/storage/archive/ha_archive.cc 2009-09-07 20:50:10 +0000 +++ b/storage/archive/ha_archive.cc 2010-02-15 07:47:03 +0000 @@ -1642,4 +1642,10 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(archive) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/blackhole/ha_blackhole.cc' --- a/storage/blackhole/ha_blackhole.cc 2008-11-10 20:21:49 +0000 +++ b/storage/blackhole/ha_blackhole.cc 2010-02-15 07:47:03 +0000 @@ -369,3 +369,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(blackhole) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/csv/ha_tina.cc' --- a/storage/csv/ha_tina.cc 2009-04-25 10:05:32 +0000 +++ b/storage/csv/ha_tina.cc 2010-02-15 07:47:03 +0000 @@ -1636,4 +1636,9 @@ NULL /* config options */ } mysql_declare_plugin_end; - +mariaext_declare_plugin(csv) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/example/ha_example.cc' --- a/storage/example/ha_example.cc 2008-02-24 13:12:17 +0000 +++ b/storage/example/ha_example.cc 2010-02-15 07:47:03 +0000 @@ -906,3 +906,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(example) +{ + "0.1", /* string version */ + PLUGIN_MATURITY_TEST /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/federated/ha_federated.cc' --- a/storage/federated/ha_federated.cc 2009-09-07 20:50:10 +0000 +++ b/storage/federated/ha_federated.cc 2010-02-15 07:47:03 +0000 @@ -3379,3 +3379,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(federated) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_BETA /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/federatedx/ha_federatedx.cc' --- a/storage/federatedx/ha_federatedx.cc 2009-11-03 11:08:09 +0000 +++ b/storage/federatedx/ha_federatedx.cc 2010-02-15 07:47:03 +0000 @@ -3485,9 +3485,15 @@ PLUGIN_LICENSE_GPL, federatedx_db_init, /* Plugin Init */ federatedx_done, /* Plugin Deinit */ - 0x0100 /* 1.0 */, + 0x0200 /* 2.0 */, NULL, /* status variables */ NULL, /* system variables */ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(federated) +{ + "2.0", /* string version */ + PLUGIN_MATURITY_BETA /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/heap/ha_heap.cc' --- a/storage/heap/ha_heap.cc 2009-09-07 20:50:10 +0000 +++ b/storage/heap/ha_heap.cc 2010-02-15 07:47:03 +0000 @@ -767,3 +767,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(heap) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/ibmdb2i/ha_ibmdb2i.cc' --- a/storage/ibmdb2i/ha_ibmdb2i.cc 2009-07-08 09:10:01 +0000 +++ b/storage/ibmdb2i/ha_ibmdb2i.cc 2010-02-15 07:47:03 +0000 @@ -3357,3 +3357,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(ibmdb2i) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_UNKNOWN /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/innobase/handler/ha_innodb.cc' --- a/storage/innobase/handler/ha_innodb.cc 2009-10-16 22:57:48 +0000 +++ b/storage/innobase/handler/ha_innodb.cc 2010-02-15 07:47:03 +0000 @@ -8684,6 +8684,12 @@ NULL /* reserved */ } mysql_declare_plugin_end; +mariaext_declare_plugin(innobase) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; /** @brief Initialize the default value of innodb_commit_concurrency. === modified file 'storage/innodb_plugin/handler/ha_innodb.cc' --- a/storage/innodb_plugin/handler/ha_innodb.cc 2009-08-04 08:02:48 +0000 +++ b/storage/innodb_plugin/handler/ha_innodb.cc 2010-02-15 07:47:03 +0000 @@ -10032,6 +10032,12 @@ i_s_innodb_cmpmem, i_s_innodb_cmpmem_reset mysql_declare_plugin_end; +mariaext_declare_plugin(innodb_plugin) +{ + INNODB_VERSION_STR, /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; /** @brief Initialize the default value of innodb_commit_concurrency. === modified file 'storage/maria/ha_maria.cc' --- a/storage/maria/ha_maria.cc 2009-10-26 11:35:42 +0000 +++ b/storage/maria/ha_maria.cc 2010-02-15 07:47:03 +0000 @@ -3346,9 +3346,15 @@ PLUGIN_LICENSE_GPL, ha_maria_init, /* Plugin Init */ NULL, /* Plugin Deinit */ - 0x0100, /* 1.0 */ + 0x0105, /* 1.5 */ status_variables, /* status variables */ system_variables, /* system variables */ NULL } mysql_declare_plugin_end; +mariaext_declare_plugin(maria) +{ + "1.5", /* string version */ + PLUGIN_MATURITY_GAMMA /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/myisam/ha_myisam.cc' --- a/storage/myisam/ha_myisam.cc 2009-10-17 19:12:28 +0000 +++ b/storage/myisam/ha_myisam.cc 2010-02-15 07:47:03 +0000 @@ -2183,6 +2183,12 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(myisam) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; #ifdef HAVE_QUERY_CACHE === modified file 'storage/myisammrg/ha_myisammrg.cc' --- a/storage/myisammrg/ha_myisammrg.cc 2009-10-15 21:38:29 +0000 +++ b/storage/myisammrg/ha_myisammrg.cc 2010-02-15 07:47:03 +0000 @@ -1289,3 +1289,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(myisammrg) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/pbxt/src/ha_pbxt.cc' --- a/storage/pbxt/src/ha_pbxt.cc 2009-09-03 06:15:03 +0000 +++ b/storage/pbxt/src/ha_pbxt.cc 2010-02-15 07:47:03 +0000 @@ -5507,6 +5507,18 @@ drizzle_declare_plugin_end; #else mysql_declare_plugin_end; +#ifdef MARIADB_BASE_VERSION +mariaext_declare_plugin(pbxt) +{ /* PBXT */ + "1.0.09g RC3", /* string version */ + PLUGIN_MATURITY_GAMMA /* maturity */ +}, +{ /* PBXT_STATISTICS */ + "1.0.09g RC3", /* string version */ + PLUGIN_MATURITY_GAMMA /* maturity */ +} +mariaext_declare_plugin_end; +#endif #endif #if defined(XT_WIN) && defined(XT_COREDUMP) === modified file 'storage/xtradb/handler/ha_innodb.cc' --- a/storage/xtradb/handler/ha_innodb.cc 2009-10-16 22:57:48 +0000 +++ b/storage/xtradb/handler/ha_innodb.cc 2010-02-15 07:47:03 +0000 @@ -10540,6 +10540,69 @@ i_s_innodb_index_stats, i_s_innodb_patches mysql_declare_plugin_end; +mariaext_declare_plugin(innobase) +{ /* InnoDB */ + INNODB_VERSION_STR, /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +}, +{ /* INNODB_RSEG */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_BUFFER_POOL_PAGES */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_BUFFER_POOL_PAGES_INDEX */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_BUFFER_POOL_PAGES_BLOB */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_TRX */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_LOCKS */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_LOCK_WAITS */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_CMP */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_CMP_RESET */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_CMPMEM */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_CMPMEM_RESET */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_TABLE_STATS */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_INDEX_STATS */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* XTRADB_ENHANCEMENTS */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +} +mariaext_declare_plugin_end; + /** @brief Initialize the default value of innodb_commit_concurrency.

1 0

Re: [Maria-developers] New (by Igor): Partitioned Key Cache for MyISAM (85)
by Arjen Lentz 15 Feb '10

15 Feb '10

Hi Igor On 15/02/2010, at 10:55 AM, Igor Babaev wrote: >> Great idea, but something it needs accessible stats to make it >> usable. >> >> Right now we can already have multiple key caches, but there's no >> way to >> see the stats of them, so that makes it very difficult to use in a >> production environment - I prefer to not have to guess how well a >> cache >> is doing ;-) >> >> Please >> - FIRST figure out a way to make stats for multiple key cache visible >> - implement that for the current multi keycaches >> - then add the partitioned key cache feature > > Arjen, > > See the patch attached to the WL task. Ok so we'd get an INFORMATION_SCHEMA.KEYCACHE Or is it PERFORMANCE_SCHEMA ? Anyway that's good, will that also show the multiple keycaches (current implementation) ? That would be great! Regards, Arjen. -- Arjen Lentz, Exec.Director @ Open Query (http://openquery.com) Exceptional Services for MySQL at a fixed budget. Follow our blog at http://openquery.com/blog/ OurDelta: packages for MySQL and MariaDB @ http://ourdelta.org

1 0

[Maria-developers] New (by Igor): Partitioned Key Cache for MyISAM (85)
by worklog-noreply＠askmonty.org 15 Feb '10

15 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:10 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-BackLog TASK ID........: 85 (http://askmonty.org/worklog/?tid=85) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

2 1

[Maria-developers] Rev 2734: Maria WL#61 in file:///Users/bell/maria/bzr/work-maria-5.2-engine/
by sanja＠askmonty.org 15 Feb '10

15 Feb '10

At file:///Users/bell/maria/bzr/work-maria-5.2-engine/ ------------------------------------------------------------ revno: 2734 revision-id: sanja(a)askmonty.org-20100215001047-8cqnklgiv1pj3sa1 parent: psergey(a)askmonty.org-20091202142609-18bp41q8mejxl47t committer: sanja(a)askmonty.org branch nick: work-maria-5.2-engine timestamp: Mon 2010-02-15 02:10:47 +0200 message: Maria WL#61 Interface for maria extensions. Additional information about plugins (maturity and string version) interface for maria extensions. === modified file 'CMakeLists.txt' --- a/CMakeLists.txt 2009-10-03 19:24:13 +0000 +++ b/CMakeLists.txt 2010-02-15 00:10:47 +0000 @@ -251,6 +251,7 @@ IF (ENGINE_BUILD_TYPE STREQUAL "STATIC") SET (mysql_plugin_defs "${mysql_plugin_defs},builtin_${PLUGIN_NAME}_plugin") + SET (mariadb_extra_plugin_defs "${mariadb_extra_plugin_defs},builtin_mariadb_${PLUGIN_NAME}_plugin") SET (MYSQLD_STATIC_ENGINE_LIBS ${MYSQLD_STATIC_ENGINE_LIBS} ${PLUGIN_NAME}) SET (STORAGE_ENGINE_DEFS "${STORAGE_ENGINE_DEFS} -DWITH_${ENGINE}_STORAGE_ENGINE") SET (WITH_${ENGINE}_STORAGE_ENGINE TRUE) @@ -269,6 +270,7 @@ IF(NOT WITHOUT_PARTITION_STORAGE_ENGINE) SET (STORAGE_ENGINE_DEFS "${STORAGE_ENGINE_DEFS} -DWITH_PARTITION_STORAGE_ENGINE") SET (mysql_plugin_defs "${mysql_plugin_defs},builtin_partition_plugin") + SET (mariadb_extra_plugin_defs "${mariadb_extra_plugin_defs},builtin_mariadb_partition_plugin") ENDIF(NOT WITHOUT_PARTITION_STORAGE_ENGINE) # Special handling for tmp tables with the maria engine === modified file 'config/ac-macros/plugins.m4' --- a/config/ac-macros/plugins.m4 2009-04-25 10:05:32 +0000 +++ b/config/ac-macros/plugins.m4 2010-02-15 00:10:47 +0000 @@ -461,6 +461,7 @@ ]) ]) mysql_plugin_defs="$mysql_plugin_defs, [builtin_]$2[_plugin]" + mariadb_extra_plugin_defs="$mariadb_extra_plugin_defs, [builtin_mariadb_]$2[_plugin]" [with_plugin_]$2=yes AC_MSG_RESULT([yes]) m4_ifdef([$11],[ === modified file 'configure.in' --- a/configure.in 2009-11-12 04:31:28 +0000 +++ b/configure.in 2010-02-15 00:10:47 +0000 @@ -2842,6 +2842,7 @@ AC_SUBST(mysql_plugin_dirs) AC_SUBST(mysql_plugin_libs) AC_SUBST(mysql_plugin_defs) +AC_SUBST(mariadb_extra_plugin_defs) # Now that sql_client_dirs and sql_server_dirs are stable, determine the union. === modified file 'include/mysql/plugin.h' --- a/include/mysql/plugin.h 2009-09-07 20:50:10 +0000 +++ b/include/mysql/plugin.h 2010-02-15 00:10:47 +0000 @@ -65,7 +65,10 @@ Plugin API. Common for all plugin types. */ +/* MySQL plugin interface version */ #define MYSQL_PLUGIN_INTERFACE_VERSION 0x0100 +/* MariaDB extentsion interface version */ +#define MARIAEXT_PLUGIN_INTERFACE_VERSION 0x0100 /* The allowable types of plugins @@ -86,6 +89,21 @@ #define PLUGIN_LICENSE_GPL_STRING "GPL" #define PLUGIN_LICENSE_BSD_STRING "BSD" +/* definitions of code maturity for plugins */ +#define PLUGIN_MATURITY_UNKNOWN 0 +#define PLUGIN_MATURITY_TEST 1 +#define PLUGIN_MATURITY_ALPHA 2 +#define PLUGIN_MATURITY_BETA 3 +#define PLUGIN_MATURITY_GAMMA 4 +#define PLUGIN_MATURITY_RELEASE 5 + +#define PLUGIN_MATURITY_UNKNOWN_STR "Unknown" +#define PLUGIN_MATURITY_TEST_STR "Test" +#define PLUGIN_MATURITY_ALPHA_STR "Alpha" +#define PLUGIN_MATURITY_BETA_STR "Beta" +#define PLUGIN_MATURITY_GAMMA_STR "Gamma" +#define PLUGIN_MATURITY_RELEASE_STR "Release" + /* Macros for beginning and ending plugin declarations. Between mysql_declare_plugin and mysql_declare_plugin_end there should @@ -94,15 +112,29 @@ #ifndef MYSQL_DYNAMIC_PLUGIN + #define __MYSQL_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ int VERSION= MYSQL_PLUGIN_INTERFACE_VERSION; \ int PSIZE= sizeof(struct st_mysql_plugin); \ struct st_mysql_plugin DECLS[]= { + +#define __MARIAEXT_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ +int VERSION= MARIAEXT_PLUGIN_INTERFACE_VERSION; \ +int PSIZE= sizeof(struct st_mariaext_plugin); \ +struct st_mariaext_plugin DECLS[]= { + #else + #define __MYSQL_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ MYSQL_PLUGIN_EXPORT int _mysql_plugin_interface_version_= MYSQL_PLUGIN_INTERFACE_VERSION; \ MYSQL_PLUGIN_EXPORT int _mysql_sizeof_struct_st_plugin_= sizeof(struct st_mysql_plugin); \ MYSQL_PLUGIN_EXPORT struct st_mysql_plugin _mysql_plugin_declarations_[]= { + +#define __MARIAEXT_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ +MYSQL_PLUGIN_EXPORT int _mariaext_plugin_interface_version_= MARIAEXT_PLUGIN_INTERFACE_VERSION; \ +MYSQL_PLUGIN_EXPORT int _mariaext_sizeof_struct_st_plugin_= sizeof(struct st_mariaext_plugin); \ +MYSQL_PLUGIN_EXPORT struct st_mariaext_plugin _mariaext_plugin_declarations_[]= { + #endif #define mysql_declare_plugin(NAME) \ @@ -111,7 +143,14 @@ builtin_ ## NAME ## _sizeof_struct_st_plugin, \ builtin_ ## NAME ## _plugin) +#define mariaext_declare_plugin(NAME) \ +__MARIAEXT_DECLARE_PLUGIN(NAME, \ + builtin_mariadb_ ## NAME ## _plugin_interface_version, \ + builtin_mariadb_ ## NAME ## _sizeof_struct_st_plugin, \ + builtin_mariadb_ ## NAME ## _plugin) + #define mysql_declare_plugin_end ,{0,0,0,0,0,0,0,0,0,0,0,0}} +#define mariaext_declare_plugin_end ,{0,0}} /* declarations for SHOW STATUS support in plugins @@ -407,6 +446,16 @@ void * __reserved1; /* reserved for dependency checking */ }; +/* + MariaDB extension for plugins declaration structure. +*/ + +struct st_mariaext_plugin +{ + const char *version_info; /* plugin version string */ + int maturity; /* HA_PLUGIN_MATURITY_XXX */ +}; + /************************************************************************* API for Full-text parser plugin. (MYSQL_FTPARSER_PLUGIN) */ === modified file 'include/mysql/plugin.h.pp' --- a/include/mysql/plugin.h.pp 2008-10-10 15:28:41 +0000 +++ b/include/mysql/plugin.h.pp 2010-02-15 00:10:47 +0000 @@ -46,6 +46,11 @@ struct st_mysql_sys_var **system_vars; void * __reserved1; }; +struct st_mariaext_plugin +{ + const char *version_info; + int maturity; +}; enum enum_ftparser_mode { MYSQL_FTPARSER_SIMPLE_MODE= 0, === modified file 'plugin/daemon_example/daemon_example.cc' --- a/plugin/daemon_example/daemon_example.cc 2007-06-27 14:49:12 +0000 +++ b/plugin/daemon_example/daemon_example.cc 2010-02-15 00:10:47 +0000 @@ -200,3 +200,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(daemon_example) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_TEST /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'plugin/fulltext/plugin_example.c' --- a/plugin/fulltext/plugin_example.c 2007-04-26 19:26:04 +0000 +++ b/plugin/fulltext/plugin_example.c 2010-02-15 00:10:47 +0000 @@ -270,4 +270,10 @@ NULL } mysql_declare_plugin_end; +mariaext_declare_plugin(ftexample) +{ + "0.01", /* string version */ + PLUGIN_MATURITY_TEST /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'sql/ha_ndbcluster.cc' --- a/sql/ha_ndbcluster.cc 2009-09-07 20:50:10 +0000 +++ b/sql/ha_ndbcluster.cc 2010-02-15 00:10:47 +0000 @@ -10561,5 +10561,11 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(ndbcluster) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_BETA /* maturity */ +} +mariaext_declare_plugin_end; #endif === modified file 'sql/ha_partition.cc' --- a/sql/ha_partition.cc 2009-11-12 04:31:28 +0000 +++ b/sql/ha_partition.cc 2010-02-15 00:10:47 +0000 @@ -6510,5 +6510,11 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(partition) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; #endif === modified file 'sql/log.cc' --- a/sql/log.cc 2009-11-12 04:31:28 +0000 +++ b/sql/log.cc 2010-02-15 00:10:47 +0000 @@ -5795,3 +5795,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(binlog) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'sql/sql_builtin.cc.in' --- a/sql/sql_builtin.cc.in 2006-12-31 01:29:11 +0000 +++ b/sql/sql_builtin.cc.in 2010-02-15 00:10:47 +0000 @@ -16,6 +16,7 @@ #include <mysql/plugin.h> typedef struct st_mysql_plugin builtin_plugin[]; +typedef struct st_mariaext_plugin builtin_mariadb_plugin[]; extern builtin_plugin builtin_binlog_plugin@mysql_plugin_defs@; @@ -25,3 +26,10 @@ builtin_binlog_plugin@mysql_plugin_defs@,(struct st_mysql_plugin *)0 }; +extern builtin_mariadb_plugin + builtin_mariadb_binlog_plugin@mariadb_extra_plugin_defs@; + +struct st_mariaext_plugin *mysqld_bltnmexts[]= +{ + builtin_mariadb_binlog_plugin@mariadb_extra_plugin_defs@,(struct st_mariaext_plugin *)0 +}; === modified file 'sql/sql_plugin.cc' --- a/sql/sql_plugin.cc 2009-11-12 04:31:28 +0000 +++ b/sql/sql_plugin.cc 2010-02-15 00:10:47 +0000 @@ -28,6 +28,9 @@ #endif extern struct st_mysql_plugin *mysqld_builtins[]; +extern struct st_mariaext_plugin *mysqld_bltnmexts[]; +static st_mariaext_plugin no_mariaext[2]= {{"Unknown", 0}, {0, 0}}; +static st_mariaext_plugin *empty_mariaext= no_mariaext; /** @note The order of the enumeration is critical. @@ -82,6 +85,14 @@ "_mysql_sizeof_struct_st_plugin_"; static const char *plugin_declarations_sym= "_mysql_plugin_declarations_"; static int min_plugin_interface_version= MYSQL_PLUGIN_INTERFACE_VERSION & ~0xFF; +static const char *mariaext_plugin_interface_version_sym= + "_mariaext_plugin_interface_version_"; +static const char *mariaext_sizeof_st_plugin_sym= + "_mariaext_sizeof_struct_st_plugin_"; +static const char *mariaext_plugin_declarations_sym= + "_mariaext_plugin_declarations_"; +static int min_mariaext_plugin_interface_version= + MARIAEXT_PLUGIN_INTERFACE_VERSION & ~0xFF; #endif /* Note that 'int version' must be the first field of every plugin @@ -352,6 +363,7 @@ char dlpath[FN_REFLEN]; uint plugin_dir_len, dummy_errors, dlpathlen; struct st_plugin_dl *tmp, plugin_dl; + struct st_mariaext_plugin *mariaext= no_mariaext; void *sym; DBUG_ENTER("plugin_dl_add"); plugin_dir_len= strlen(opt_plugin_dir); @@ -507,6 +519,15 @@ files_charset_info, dl->str, dl->length, system_charset_info, &dummy_errors); plugin_dl.dl.str[plugin_dl.dl.length]= 0; + + if ((sym= dlsym(plugin_dl.handle, mariaext_plugin_interface_version_sym)) && + (*(int *)sym == MARIAEXT_PLUGIN_INTERFACE_VERSION) && + (sym= dlsym(plugin_dl.handle, mariaext_plugin_declarations_sym))) + { + mariaext= (struct st_mariaext_plugin *) sym; + } + plugin_dl.mariaext= mariaext; + /* Add this dll to array */ if (! (tmp= plugin_dl_insert_or_reuse(&plugin_dl))) { @@ -719,6 +740,7 @@ { struct st_plugin_int tmp; struct st_mysql_plugin *plugin; + struct st_mariaext_plugin *ext, *mariaext; DBUG_ENTER("plugin_add"); if (plugin_find_internal(name, MYSQL_ANY_PLUGIN)) { @@ -732,9 +754,22 @@ bzero((char*) &tmp, sizeof(tmp)); if (! (tmp.plugin_dl= plugin_dl_add(dl, report))) DBUG_RETURN(TRUE); + /* Find plugin by name */ - for (plugin= tmp.plugin_dl->plugins; plugin->info; plugin++) + for (plugin= tmp.plugin_dl->plugins, ext= tmp.plugin_dl->mariaext; + plugin->info; + plugin++, ext++) { + mariaext= ext; + if (!ext->version_info) + { + /* + Plugin didn't have any mariadb extensions; Use default one and + reset counter to do same for next internal plugin. + */ + mariaext= empty_mariaext; + ext--; + } uint name_len= strlen(plugin->name); if (plugin->type >= 0 && plugin->type < MYSQL_MAX_PLUGIN_TYPE_NUM && ! my_strnncoll(system_charset_info, @@ -759,6 +794,7 @@ goto err; } tmp.plugin= plugin; + tmp.mariaext= mariaext; tmp.name.str= (char *)plugin->name; tmp.name.length= name_len; tmp.ref_count= 0; @@ -1121,7 +1157,9 @@ uint i; bool is_myisam; struct st_mysql_plugin **builtins; + struct st_mariaext_plugin **bltnmexts; struct st_mysql_plugin *plugin; + struct st_mariaext_plugin *ext, *mariaext; struct st_plugin_int tmp, *plugin_ptr, **reap; MEM_ROOT tmp_root; bool reaped_mandatory_plugin= FALSE; @@ -1160,10 +1198,29 @@ /* First we register builtin plugins */ - for (builtins= mysqld_builtins; *builtins; builtins++) + for (builtins= mysqld_builtins, bltnmexts= mysqld_bltnmexts; + *builtins; + builtins++, bltnmexts++) { - for (plugin= *builtins; plugin->info; plugin++) + /* extensions should be the same numbers as static plugins) */ + DBUG_ASSERT(*bltnmexts); + for (plugin= *builtins, ext= *bltnmexts; + plugin->info; + plugin++, ext++) { + + /* in case if plugin describe less extensions then plugins */ + mariaext= ext; + if (!ext->version_info) + { + /* + Plugin didn't have any mariadb extensions; Use default one and + reset counter to do same for next internal plugin. + */ + mariaext= empty_mariaext; + ext--; + } + if (opt_ignore_builtin_innodb && !my_strnncoll(&my_charset_latin1, (const uchar*) plugin->name, 6, (const uchar*) "InnoDB", 6)) @@ -1186,6 +1243,7 @@ #endif bzero(&tmp, sizeof(tmp)); tmp.plugin= plugin; + tmp.mariaext= mariaext; tmp.name.str= (char *)plugin->name; tmp.name.length= strlen(plugin->name); tmp.state= 0; === modified file 'sql/sql_plugin.h' --- a/sql/sql_plugin.h 2009-05-14 12:03:33 +0000 +++ b/sql/sql_plugin.h 2010-02-15 00:10:47 +0000 @@ -63,6 +63,7 @@ LEX_STRING dl; void *handle; struct st_mysql_plugin *plugins; + struct st_mariaext_plugin *mariaext; int version; uint ref_count; /* number of plugins loaded from the library */ }; @@ -74,6 +75,7 @@ LEX_STRING name; struct st_mysql_plugin *plugin; struct st_plugin_dl *plugin_dl; + struct st_mariaext_plugin *mariaext; uint state; uint ref_count; /* number of threads using the plugin */ void *data; /* plugin type specific, e.g. handlerton */ @@ -95,6 +97,7 @@ #define plugin_name(pi) (&((pi)->name)) #define plugin_state(pi) ((pi)->state) #define plugin_equals(p1,p2) ((p1) == (p2)) +#define plugin_ext(pi) ((pi)->mariaext) #else typedef struct st_plugin_int **plugin_ref; #define plugin_decl(pi) ((pi)[0]->plugin) @@ -103,6 +106,8 @@ #define plugin_name(pi) (&((pi)[0]->name)) #define plugin_state(pi) ((pi)[0]->state) #define plugin_equals(p1,p2) ((p1) && (p2) && (p1)[0] == (p2)[0]) +#define plugin_ext(pi) ((pi)[0]->mariaext) + #endif typedef int (*plugin_type_init)(struct st_plugin_int *); === modified file 'sql/sql_show.cc' --- a/sql/sql_show.cc 2009-11-12 04:31:28 +0000 +++ b/sql/sql_show.cc 2010-02-15 00:10:47 +0000 @@ -94,12 +94,21 @@ return my_snprintf(buf, buf_length, "%d.%d", version>>8,version&0xff); } +static const LEX_STRING maturity_name[]={ + { C_STRING_WITH_LEN(PLUGIN_MATURITY_UNKNOWN_STR) }, + { C_STRING_WITH_LEN(PLUGIN_MATURITY_TEST_STR) }, + { C_STRING_WITH_LEN(PLUGIN_MATURITY_ALPHA_STR) }, + { C_STRING_WITH_LEN(PLUGIN_MATURITY_BETA_STR) }, + { C_STRING_WITH_LEN(PLUGIN_MATURITY_GAMMA_STR) }, + { C_STRING_WITH_LEN(PLUGIN_MATURITY_RELEASE_STR) }}; + static my_bool show_plugins(THD *thd, plugin_ref plugin, void *arg) { TABLE *table= (TABLE*) arg; struct st_mysql_plugin *plug= plugin_decl(plugin); struct st_plugin_dl *plugin_dl= plugin_dlib(plugin); + struct st_mariaext_plugin *mariaext= plugin_ext(plugin); CHARSET_INFO *cs= system_charset_info; char version_buf[20]; @@ -186,6 +195,26 @@ } table->field[9]->set_notnull(); + if ((uint) mariaext->maturity <= PLUGIN_MATURITY_RELEASE) + table->field[10]->store(maturity_name[mariaext->maturity].str, + maturity_name[mariaext->maturity].length, + cs); + else + { + DBUG_ASSERT(0); + table->field[10]->store("Unknown", 7, cs); + } + table->field[10]->set_notnull(); + + if (mariaext->version_info) + { + table->field[11]->store(mariaext->version_info, + strlen(mariaext->version_info), cs); + table->field[11]->set_notnull(); + } + else + table->field[11]->set_null(); + return schema_table_store_record(thd, table); } @@ -6990,6 +7019,8 @@ {"PLUGIN_AUTHOR", NAME_CHAR_LEN, MYSQL_TYPE_STRING, 0, 1, 0, SKIP_OPEN_TABLE}, {"PLUGIN_DESCRIPTION", 65535, MYSQL_TYPE_STRING, 0, 1, 0, SKIP_OPEN_TABLE}, {"PLUGIN_LICENSE", 80, MYSQL_TYPE_STRING, 0, 1, "License", SKIP_OPEN_TABLE}, + {"PLUGIN_MATURITY", 7, MYSQL_TYPE_STRING, 0, 1, 0, SKIP_OPEN_TABLE}, + {"PLUGIN_AUTH_VERSION", 80, MYSQL_TYPE_STRING, 0, 1, 0, SKIP_OPEN_TABLE}, {0, 0, MYSQL_TYPE_STRING, 0, 0, 0, SKIP_OPEN_TABLE} }; === modified file 'storage/archive/ha_archive.cc' --- a/storage/archive/ha_archive.cc 2009-09-07 20:50:10 +0000 +++ b/storage/archive/ha_archive.cc 2010-02-15 00:10:47 +0000 @@ -1642,4 +1642,10 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(archive) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/blackhole/ha_blackhole.cc' --- a/storage/blackhole/ha_blackhole.cc 2008-11-10 20:21:49 +0000 +++ b/storage/blackhole/ha_blackhole.cc 2010-02-15 00:10:47 +0000 @@ -369,3 +369,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(blackhole) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/csv/ha_tina.cc' --- a/storage/csv/ha_tina.cc 2009-04-25 10:05:32 +0000 +++ b/storage/csv/ha_tina.cc 2010-02-15 00:10:47 +0000 @@ -1636,4 +1636,9 @@ NULL /* config options */ } mysql_declare_plugin_end; - +mariaext_declare_plugin(csv) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/example/ha_example.cc' --- a/storage/example/ha_example.cc 2008-02-24 13:12:17 +0000 +++ b/storage/example/ha_example.cc 2010-02-15 00:10:47 +0000 @@ -906,3 +906,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(example) +{ + "0.1", /* string version */ + PLUGIN_MATURITY_TEST /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/federated/ha_federated.cc' --- a/storage/federated/ha_federated.cc 2009-09-07 20:50:10 +0000 +++ b/storage/federated/ha_federated.cc 2010-02-15 00:10:47 +0000 @@ -3379,3 +3379,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(federated) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_BETA /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/federatedx/ha_federatedx.cc' --- a/storage/federatedx/ha_federatedx.cc 2009-11-03 11:08:09 +0000 +++ b/storage/federatedx/ha_federatedx.cc 2010-02-15 00:10:47 +0000 @@ -3485,9 +3485,15 @@ PLUGIN_LICENSE_GPL, federatedx_db_init, /* Plugin Init */ federatedx_done, /* Plugin Deinit */ - 0x0100 /* 1.0 */, + 0x0200 /* 2.0 */, NULL, /* status variables */ NULL, /* system variables */ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(federated) +{ + "2.0", /* string version */ + PLUGIN_MATURITY_BETA /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/heap/ha_heap.cc' --- a/storage/heap/ha_heap.cc 2009-09-07 20:50:10 +0000 +++ b/storage/heap/ha_heap.cc 2010-02-15 00:10:47 +0000 @@ -767,3 +767,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(heap) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/ibmdb2i/ha_ibmdb2i.cc' --- a/storage/ibmdb2i/ha_ibmdb2i.cc 2009-07-08 09:10:01 +0000 +++ b/storage/ibmdb2i/ha_ibmdb2i.cc 2010-02-15 00:10:47 +0000 @@ -3357,3 +3357,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(ibmdb2i) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_UNKNOWN /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/innobase/handler/ha_innodb.cc' --- a/storage/innobase/handler/ha_innodb.cc 2009-10-16 22:57:48 +0000 +++ b/storage/innobase/handler/ha_innodb.cc 2010-02-15 00:10:47 +0000 @@ -8684,6 +8684,12 @@ NULL /* reserved */ } mysql_declare_plugin_end; +mariaext_declare_plugin(innobase) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; /** @brief Initialize the default value of innodb_commit_concurrency. === modified file 'storage/innodb_plugin/handler/ha_innodb.cc' --- a/storage/innodb_plugin/handler/ha_innodb.cc 2009-08-04 08:02:48 +0000 +++ b/storage/innodb_plugin/handler/ha_innodb.cc 2010-02-15 00:10:47 +0000 @@ -10032,6 +10032,12 @@ i_s_innodb_cmpmem, i_s_innodb_cmpmem_reset mysql_declare_plugin_end; +mariaext_declare_plugin(innodb_plugin) +{ + INNODB_VERSION_STR, /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; /** @brief Initialize the default value of innodb_commit_concurrency. === modified file 'storage/maria/ha_maria.cc' --- a/storage/maria/ha_maria.cc 2009-10-26 11:35:42 +0000 +++ b/storage/maria/ha_maria.cc 2010-02-15 00:10:47 +0000 @@ -3346,9 +3346,15 @@ PLUGIN_LICENSE_GPL, ha_maria_init, /* Plugin Init */ NULL, /* Plugin Deinit */ - 0x0100, /* 1.0 */ + 0x0105, /* 1.5 */ status_variables, /* status variables */ system_variables, /* system variables */ NULL } mysql_declare_plugin_end; +mariaext_declare_plugin(maria) +{ + "1.5", /* string version */ + PLUGIN_MATURITY_GAMMA /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/myisam/ha_myisam.cc' --- a/storage/myisam/ha_myisam.cc 2009-10-17 19:12:28 +0000 +++ b/storage/myisam/ha_myisam.cc 2010-02-15 00:10:47 +0000 @@ -2183,6 +2183,12 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(myisam) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; #ifdef HAVE_QUERY_CACHE === modified file 'storage/myisammrg/ha_myisammrg.cc' --- a/storage/myisammrg/ha_myisammrg.cc 2009-10-15 21:38:29 +0000 +++ b/storage/myisammrg/ha_myisammrg.cc 2010-02-15 00:10:47 +0000 @@ -1289,3 +1289,9 @@ NULL /* config options */ } mysql_declare_plugin_end; +mariaext_declare_plugin(myisammrg) +{ + "1.0", /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +} +mariaext_declare_plugin_end; === modified file 'storage/pbxt/src/ha_pbxt.cc' --- a/storage/pbxt/src/ha_pbxt.cc 2009-09-03 06:15:03 +0000 +++ b/storage/pbxt/src/ha_pbxt.cc 2010-02-15 00:10:47 +0000 @@ -5507,6 +5507,18 @@ drizzle_declare_plugin_end; #else mysql_declare_plugin_end; +#ifdef MARIADB_BASE_VERSION +mariaext_declare_plugin(pbxt) +{ /* PBXT */ + "1.0.09g RC3", /* string version */ + PLUGIN_MATURITY_GAMMA /* maturity */ +}, +{ /* PBXT_STATISTICS */ + "1.0.09g RC3", /* string version */ + PLUGIN_MATURITY_GAMMA /* maturity */ +} +mariaext_declare_plugin_end; +#endif #endif #if defined(XT_WIN) && defined(XT_COREDUMP) === modified file 'storage/xtradb/handler/ha_innodb.cc' --- a/storage/xtradb/handler/ha_innodb.cc 2009-10-16 22:57:48 +0000 +++ b/storage/xtradb/handler/ha_innodb.cc 2010-02-15 00:10:47 +0000 @@ -10540,6 +10540,69 @@ i_s_innodb_index_stats, i_s_innodb_patches mysql_declare_plugin_end; +mariaext_declare_plugin(innobase) +{ /* InnoDB */ + INNODB_VERSION_STR, /* string version */ + PLUGIN_MATURITY_RELEASE /* maturity */ +}, +{ /* INNODB_RSEG */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_BUFFER_POOL_PAGES */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_BUFFER_POOL_PAGES_INDEX */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_BUFFER_POOL_PAGES_BLOB */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_TRX */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_LOCKS */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_LOCK_WAITS */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_CMP */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_CMP_RESET */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_CMPMEM */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_CMPMEM_RESET */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_TABLE_STATS */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* INNODB_INDEX_STATS */ + "1.0", + PLUGIN_MATURITY_RELEASE +}, +{ /* XTRADB_ENHANCEMENTS */ + INNODB_VERSION_STR, + PLUGIN_MATURITY_RELEASE +} +mariaext_declare_plugin_end; + /** @brief Initialize the default value of innodb_commit_concurrency.

1 0

[Maria-developers] Updated (by Igor): Partitioned Key Cache for MyISAM (86)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:17 SUPERVISOR.....: Monty IMPLEMENTOR....: Igor COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-Sprint TASK ID........: 86 (http://askmonty.org/worklog/?tid=86) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Igor - Sun, 14 Feb 2010, 00:19)=-=- Privacy level updated. --- /tmp/wklog.86.old.10092 2010-02-13 22:19:03.000000000 +0000 +++ /tmp/wklog.86.new.10092 2010-02-13 22:19:03.000000000 +0000 @@ -1 +1 @@ -y +n -=-=(Igor - Sun, 14 Feb 2010, 00:19)=-=- Category updated. --- /tmp/wklog.86.old.10092 2010-02-13 22:19:03.000000000 +0000 +++ /tmp/wklog.86.new.10092 2010-02-13 22:19:03.000000000 +0000 @@ -1 +1 @@ -Server-BackLog +Server-Sprint -=-=(Igor - Sun, 14 Feb 2010, 00:18)=-=- Version updated. --- /tmp/wklog.86.old.10044 2010-02-14 00:18:31.000000000 +0200 +++ /tmp/wklog.86.new.10044 2010-02-14 00:18:31.000000000 +0200 @@ -1 +1 @@ -Benchmarks-3.0 +Server-5.2 DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers (see the attached file segmented_keycache_v2.diff with the original patch from the contributor). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Igor): Partitioned Key Cache for MyISAM (86)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:17 SUPERVISOR.....: Monty IMPLEMENTOR....: Igor COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-Sprint TASK ID........: 86 (http://askmonty.org/worklog/?tid=86) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Igor - Sun, 14 Feb 2010, 00:19)=-=- Privacy level updated. --- /tmp/wklog.86.old.10092 2010-02-13 22:19:03.000000000 +0000 +++ /tmp/wklog.86.new.10092 2010-02-13 22:19:03.000000000 +0000 @@ -1 +1 @@ -y +n -=-=(Igor - Sun, 14 Feb 2010, 00:19)=-=- Category updated. --- /tmp/wklog.86.old.10092 2010-02-13 22:19:03.000000000 +0000 +++ /tmp/wklog.86.new.10092 2010-02-13 22:19:03.000000000 +0000 @@ -1 +1 @@ -Server-BackLog +Server-Sprint -=-=(Igor - Sun, 14 Feb 2010, 00:18)=-=- Version updated. --- /tmp/wklog.86.old.10044 2010-02-14 00:18:31.000000000 +0200 +++ /tmp/wklog.86.new.10044 2010-02-14 00:18:31.000000000 +0200 @@ -1 +1 @@ -Benchmarks-3.0 +Server-5.2 DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers (see the attached file segmented_keycache_v2.diff with the original patch from the contributor). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Igor): Partitioned Key Cache for MyISAM (86)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:17 SUPERVISOR.....: Monty IMPLEMENTOR....: Igor COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-Sprint TASK ID........: 86 (http://askmonty.org/worklog/?tid=86) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Igor - Sun, 14 Feb 2010, 00:19)=-=- Privacy level updated. --- /tmp/wklog.86.old.10092 2010-02-13 22:19:03.000000000 +0000 +++ /tmp/wklog.86.new.10092 2010-02-13 22:19:03.000000000 +0000 @@ -1 +1 @@ -y +n -=-=(Igor - Sun, 14 Feb 2010, 00:19)=-=- Category updated. --- /tmp/wklog.86.old.10092 2010-02-13 22:19:03.000000000 +0000 +++ /tmp/wklog.86.new.10092 2010-02-13 22:19:03.000000000 +0000 @@ -1 +1 @@ -Server-BackLog +Server-Sprint -=-=(Igor - Sun, 14 Feb 2010, 00:18)=-=- Version updated. --- /tmp/wklog.86.old.10044 2010-02-14 00:18:31.000000000 +0200 +++ /tmp/wklog.86.new.10044 2010-02-14 00:18:31.000000000 +0200 @@ -1 +1 @@ -Benchmarks-3.0 +Server-5.2 DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers (see the attached file segmented_keycache_v2.diff with the original patch from the contributor). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Igor): Partitioned Key Cache for MyISAM (86)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:17 SUPERVISOR.....: Monty IMPLEMENTOR....: Igor COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-BackLog TASK ID........: 86 (http://askmonty.org/worklog/?tid=86) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Igor - Sun, 14 Feb 2010, 00:18)=-=- Version updated. --- /tmp/wklog.86.old.10044 2010-02-14 00:18:31.000000000 +0200 +++ /tmp/wklog.86.new.10044 2010-02-14 00:18:31.000000000 +0200 @@ -1 +1 @@ -Benchmarks-3.0 +Server-5.2 DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers (see the attached file segmented_keycache_v2.diff with the original patch from the contributor). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Igor): Partitioned Key Cache for MyISAM (86)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:17 SUPERVISOR.....: Monty IMPLEMENTOR....: Igor COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-BackLog TASK ID........: 86 (http://askmonty.org/worklog/?tid=86) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Igor - Sun, 14 Feb 2010, 00:18)=-=- Version updated. --- /tmp/wklog.86.old.10044 2010-02-14 00:18:31.000000000 +0200 +++ /tmp/wklog.86.new.10044 2010-02-14 00:18:31.000000000 +0200 @@ -1 +1 @@ -Benchmarks-3.0 +Server-5.2 DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers (see the attached file segmented_keycache_v2.diff with the original patch from the contributor). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Igor): Partitioned Key Cache for MyISAM (86)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:17 SUPERVISOR.....: Monty IMPLEMENTOR....: Igor COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-BackLog TASK ID........: 86 (http://askmonty.org/worklog/?tid=86) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Igor - Sun, 14 Feb 2010, 00:18)=-=- Version updated. --- /tmp/wklog.86.old.10044 2010-02-14 00:18:31.000000000 +0200 +++ /tmp/wklog.86.new.10044 2010-02-14 00:18:31.000000000 +0200 @@ -1 +1 @@ -Benchmarks-3.0 +Server-5.2 DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers (see the attached file segmented_keycache_v2.diff with the original patch from the contributor). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Igor): Partitioned Key Cache for MyISAM (86)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:17 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-BackLog TASK ID........: 86 (http://askmonty.org/worklog/?tid=86) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers (see the attached file segmented_keycache_v2.diff with the original patch from the contributor). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Igor): Partitioned Key Cache for MyISAM (86)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:17 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-BackLog TASK ID........: 86 (http://askmonty.org/worklog/?tid=86) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers (see the attached file segmented_keycache_v2.diff with the original patch from the contributor). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Igor): Partitioned Key Cache for MyISAM (86)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:17 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-BackLog TASK ID........: 86 (http://askmonty.org/worklog/?tid=86) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers (see the attached file segmented_keycache_v2.diff with the original patch from the contributor). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Igor): Partitioned Key Cache for MyISAM (85)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:10 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-Sprint TASK ID........: 85 (http://askmonty.org/worklog/?tid=85) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Igor - Sun, 14 Feb 2010, 00:15)=-=- Category updated. --- /tmp/wklog.85.old.9810 2010-02-13 22:15:43.000000000 +0000 +++ /tmp/wklog.85.new.9810 2010-02-13 22:15:43.000000000 +0000 @@ -1 +1 @@ -Server-BackLog +Server-Sprint -=-=(Igor - Sun, 14 Feb 2010, 00:15)=-=- Version updated. --- /tmp/wklog.85.old.9810 2010-02-13 22:15:43.000000000 +0000 +++ /tmp/wklog.85.new.9810 2010-02-13 22:15:43.000000000 +0000 @@ -1 +1 @@ -Benchmarks-3.0 +Server-5.2 -=-=(Igor - Sun, 14 Feb 2010, 00:12)=-=- New attachment: 'segmented_keycache_v2.diff' DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Igor): Partitioned Key Cache for MyISAM (85)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:10 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-Sprint TASK ID........: 85 (http://askmonty.org/worklog/?tid=85) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Igor - Sun, 14 Feb 2010, 00:15)=-=- Category updated. --- /tmp/wklog.85.old.9810 2010-02-13 22:15:43.000000000 +0000 +++ /tmp/wklog.85.new.9810 2010-02-13 22:15:43.000000000 +0000 @@ -1 +1 @@ -Server-BackLog +Server-Sprint -=-=(Igor - Sun, 14 Feb 2010, 00:15)=-=- Version updated. --- /tmp/wklog.85.old.9810 2010-02-13 22:15:43.000000000 +0000 +++ /tmp/wklog.85.new.9810 2010-02-13 22:15:43.000000000 +0000 @@ -1 +1 @@ -Benchmarks-3.0 +Server-5.2 -=-=(Igor - Sun, 14 Feb 2010, 00:12)=-=- New attachment: 'segmented_keycache_v2.diff' DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Igor): Partitioned Key Cache for MyISAM (85)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:10 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-Sprint TASK ID........: 85 (http://askmonty.org/worklog/?tid=85) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Igor - Sun, 14 Feb 2010, 00:15)=-=- Category updated. --- /tmp/wklog.85.old.9810 2010-02-13 22:15:43.000000000 +0000 +++ /tmp/wklog.85.new.9810 2010-02-13 22:15:43.000000000 +0000 @@ -1 +1 @@ -Server-BackLog +Server-Sprint -=-=(Igor - Sun, 14 Feb 2010, 00:15)=-=- Version updated. --- /tmp/wklog.85.old.9810 2010-02-13 22:15:43.000000000 +0000 +++ /tmp/wklog.85.new.9810 2010-02-13 22:15:43.000000000 +0000 @@ -1 +1 @@ -Benchmarks-3.0 +Server-5.2 -=-=(Igor - Sun, 14 Feb 2010, 00:12)=-=- New attachment: 'segmented_keycache_v2.diff' DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Igor): Partitioned Key Cache for MyISAM (85)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:10 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-BackLog TASK ID........: 85 (http://askmonty.org/worklog/?tid=85) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Igor - Sun, 14 Feb 2010, 00:12)=-=- New attachment: 'segmented_keycache_v2.diff' DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Igor): Partitioned Key Cache for MyISAM (85)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:10 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-BackLog TASK ID........: 85 (http://askmonty.org/worklog/?tid=85) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Igor - Sun, 14 Feb 2010, 00:12)=-=- New attachment: 'segmented_keycache_v2.diff' DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Igor): Partitioned Key Cache for MyISAM (85)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:10 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-BackLog TASK ID........: 85 (http://askmonty.org/worklog/?tid=85) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Igor - Sun, 14 Feb 2010, 00:12)=-=- New attachment: 'segmented_keycache_v2.diff' DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Igor): Partitioned Key Cache for MyISAM (85)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:10 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-BackLog TASK ID........: 85 (http://askmonty.org/worklog/?tid=85) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Igor): Partitioned Key Cache for MyISAM (85)
by worklog-noreply＠askmonty.org 13 Feb '10

13 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Partitioned Key Cache for MyISAM CREATION DATE..: Sun, 14 Feb 2010, 00:10 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Monty, Sergei CATEGORY.......: Server-BackLog TASK ID........: 85 (http://askmonty.org/worklog/?tid=85) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: A partitioned key cache is a collection of structures for regular MyiSAM key caches called key cache partitions. Any page from a file can be placed into a buffer of only one partition. The number of the partition is calculated from the file number and the position of the page in the file, and it's always the same for the page. The function that maps pages into partitions takes care of even distribution of pages among partitions. Partition key cache mitigate one of the major problem of simple key cache: thread contention for key cache lock (mutex). Every call of a key cache interface function must acquire this lock. So threads compete for this lock even in the case when they have acquired shared locks for the file and pages they want read from are in the key cache buffers. When working with a partitioned key cache any key cache interface function that needs only one page has to acquire the key cache lock only for the partition the page is ascribed to. This makes the chances for threads not compete for the same key cache lock better. The idea and the original of the partitioned key cache was provided by one of our external contributers. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Rev 2757: Fix for previous cset in file:///home/psergey/dev/maria-5.3-subqueries-r6/
by Sergey Petrunya 12 Feb '10

12 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r6/ ------------------------------------------------------------ revno: 2757 revision-id: psergey(a)askmonty.org-20100212181041-5rwekm1wpvwaikkx parent: psergey(a)askmonty.org-20100211235958-p11o4e80dlrn2bsq committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r6 timestamp: Fri 2010-02-12 21:10:41 +0300 message: Fix for previous cset === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-02-11 23:59:58 +0000 +++ b/sql/item_subselect.cc 2010-02-12 18:10:41 +0000 @@ -1316,6 +1316,7 @@ (char *)in_left_expr_name); master_unit->uncacheable|= UNCACHEABLE_DEPENDENT; + select_lex->uncacheable|= UNCACHEABLE_DEPENDENT; } if (!abort_on_null && left_expr->maybe_null && !pushed_cond_guards)

1 0

[Maria-developers] Rev 3770: MWL#68 Subquery optimization: Efficient NOT IN execution with NULLs in file:///home/tsk/mprog/src/mysql-6.0-mwl68/
by timour＠askmonty.org 12 Feb '10

12 Feb '10

At file:///home/tsk/mprog/src/mysql-6.0-mwl68/ ------------------------------------------------------------ revno: 3770 revision-id: timour(a)askmonty.org-20100212143343-l0pjascssuqedfk6 parent: timour(a)askmonty.org-20100201120948-mdt7gtwcz50q1dzp committer: timour(a)askmonty.org branch nick: mysql-6.0-mwl68 timestamp: Fri 2010-02-12 16:33:43 +0200 message: MWL#68 Subquery optimization: Efficient NOT IN execution with NULLs This patch implements working partial matching for materialized subqueries. The code passes the full regression test, except differences in EXPLAIN. There are no other known test failures. Diff too large for email (1593 lines, the limit is 1000).

1 0

[Maria-developers] bzr commit into file:///home/tsk/mprog/src/mysql-6.0-mwl68/ branch (timour:3770)
by timour＠askmonty.org 12 Feb '10

12 Feb '10

#At file:///home/tsk/mprog/src/mysql-6.0-mwl68/ based on revid:timour@askmonty.org-20100201120948-mdt7gtwcz50q1dzp 3770 timour(a)askmonty.org 2010-02-12 MWL#68 Subquery optimization: Efficient NOT IN execution with NULLs This patch implements working partial matching for materialized subqueries. The code passes the full regression test, except differences in EXPLAIN. There are no other known test failures. modified: sql/item_subselect.cc sql/item_subselect.h sql/sql_class.cc sql/sql_class.h sql/sql_select.cc === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-02-01 12:09:48 +0000 +++ b/sql/item_subselect.cc 2010-02-12 14:33:43 +0000 @@ -2436,6 +2436,17 @@ int subselect_uniquesubquery_engine::sca for (;;) { error=table->file->rnd_next(table->record[0]); + /* + TODO: The below tests are wrong, Monty's proposal: + if (error) { + if (error == HA_ERR_RECORD_DELETED) + continue; + if (error = HA_ERR_END_OF_FILE) + break; + else + report error; + break; + */ if (error && error != HA_ERR_END_OF_FILE) { error= report_error(table, error); @@ -2453,6 +2464,11 @@ int subselect_uniquesubquery_engine::sca } table->file->ha_rnd_end(); + /* + TODO: it seems to be an error to return TRUE when the error was + HA_ERR_END_OF_FILE which is perfectly fine. HA_ERR_END_OF_FILE + only means we didn't find a match. + */ DBUG_RETURN(error != 0); } @@ -2517,6 +2533,10 @@ bool subselect_uniquesubquery_engine::co See also the comment for the subselect_uniquesubquery_engine::exec() function. */ + /* + TODO: If not all outer cols are NULL, how we know the result is NULL, + and not FALSE? Even on top-level. + */ null_keypart= (*copy)->null_key; if (null_keypart) { @@ -2556,6 +2576,59 @@ bool subselect_uniquesubquery_engine::co /* + @retval 1 A NULL was found in the outer reference, index lookup is + not applicable, the outer ref is unsusable as a lookup key, + use some other method to find a match. + @retval 0 The outer ref was copied into an index lookup key. + @retval -1 The outer ref cannot possibly match any row, IN is FALSE. +*/ + +int subselect_uniquesubquery_engine::copy_ref_key_simple() +{ + for (store_key **copy= tab->ref.key_copy ; *copy ; copy++) + { + enum store_key::store_key_result store_res; + store_res= (*copy)->copy(); + tab->ref.key_err= store_res; + + /* + When there is a NULL part in the key we don't need to make index + lookup for such key thus we don't need to copy whole key. + If we later should do a sequential scan return OK. Fail otherwise. + + See also the comment for the subselect_uniquesubquery_engine::exec() + function. + */ + /* + TODO: If not all outer cols are NULL, how we know the result is NULL, + and not FALSE? Even on top-level. + */ + null_keypart= (*copy)->null_key; + if (null_keypart) + return 1; + + /* + Check if the error is equal to STORE_KEY_FATAL. This is not expressed + using the store_key::store_key_result enum because ref.key_err is a + boolean and we want to detect both TRUE and STORE_KEY_FATAL from the + space of the union of the values of [TRUE, FALSE] and + store_key::store_key_result. + TODO: fix the variable an return types. + */ + if (store_res == store_key::STORE_KEY_FATAL) + { + /* + Error converting the left IN operand to the column type of the right + IN operand. + */ + return -1; + } + } + return 0; +} + + +/* Execute subselect SYNOPSIS @@ -2595,7 +2668,10 @@ int subselect_uniquesubquery_engine::exe /* TODO: change to use of 'full_scan' here? */ if (copy_ref_key()) + { + /* TODO: copy_ref_key() == 1 means NULL result, not error, why return 1? */ DBUG_RETURN(1); + } if (table->status) { /* @@ -2637,6 +2713,52 @@ int subselect_uniquesubquery_engine::exe /* + TODO: this needs more thinking, as exec() is a bit wrong IMO. + - we don't need empty_result_set, as it is == 1 <=> when + item->value == 0 + - scan_table() returns >0 even when there was no actuall error, + but we only found EOF while scanning. + - scan_table should not check table->status, but it should check + HA_ERR_END_OF_FILE +*/ + +int subselect_uniquesubquery_engine::index_lookup() +{ + DBUG_ENTER("subselect_uniquesubquery_engine::index_lookup"); + int error; + TABLE *table= tab->table; + empty_result_set= TRUE; + table->status= 0; + + if (!table->file->inited) + table->file->ha_index_init(tab->ref.key, 0); + error= table->file->index_read_map(table->record[0], + tab->ref.key_buff, + make_prev_keypart_map(tab->ref.key_parts), + HA_READ_KEY_EXACT); + DBUG_PRINT("info", ("lookup result: %i", error)); + if (error && + error != HA_ERR_KEY_NOT_FOUND && error != HA_ERR_END_OF_FILE) + error= report_error(table, error); + else + { + error= 0; + table->null_row= 0; + if (!table->status && (!cond || cond->val_int())) + { + ((Item_in_subselect *) item)->value= 1; + empty_result_set= FALSE; + } + else + ((Item_in_subselect *) item)->value= 0; + } + + DBUG_RETURN(error); +} + + + +/* Index-lookup subselect 'engine' - run the subquery SYNOPSIS @@ -3136,6 +3258,7 @@ void subselect_hash_sj_engine::set_strat Item_in_subselect *item_in= (Item_in_subselect *) item; select_materialize_with_stats *result_sink= (select_materialize_with_stats *) result; + Item *outer_col; DBUG_ENTER("subselect_hash_sj_engine::set_strategy_using_data"); @@ -3146,13 +3269,20 @@ void subselect_hash_sj_engine::set_strat { if (!bitmap_is_set(&partial_match_key_parts, i)) continue; - - if (result_sink->get_null_count_of_col(i) == 0) + outer_col= item_in->left_expr->element_index(i); + /* + If column 'i' doesn't contain NULLs, and the corresponding outer reference + cannot have a NULL value, then 'i' is a non-nullable column. + */ + if (result_sink->get_null_count_of_col(i) == 0 && !outer_col->maybe_null) { bitmap_clear_bit(&partial_match_key_parts, i); bitmap_set_bit(&non_null_key_parts, i); --count_partial_match_columns; } + if (result_sink->get_null_count_of_col(i) == + tmp_table->file->stats.records) + ++count_null_only_columns; } /* If no column contains NULLs use regular hash index lookups. */ @@ -3177,6 +3307,7 @@ bitmap_init_memroot(MY_BITMAP *map, uint bitmap_buffer_size(n_bits))) || bitmap_init(map, bitmap_buf, n_bits, FALSE)) return TRUE; + bitmap_clear_all(map); return FALSE; } @@ -3209,10 +3340,10 @@ bool subselect_hash_sj_engine::init_perm DBUG_ENTER("subselect_hash_sj_engine::init_permanent"); - if (!(bitmap_init_memroot(&non_null_key_parts, tmp_columns->elements, - thd->mem_root)) || - !(bitmap_init_memroot(&partial_match_key_parts, tmp_columns->elements, - thd->mem_root))) + if (bitmap_init_memroot(&non_null_key_parts, tmp_columns->elements, + thd->mem_root) || + bitmap_init_memroot(&partial_match_key_parts, tmp_columns->elements, + thd->mem_root)) DBUG_RETURN(TRUE); set_strategy_using_schema(); @@ -3548,33 +3679,45 @@ int subselect_hash_sj_engine::exec() if (strategy == PARTIAL_MATCH) { - subselect_rowid_merge_engine *new_lookup_engine; + subselect_rowid_merge_engine *rowid_merge_engine; uint count_pm_keys; MY_BITMAP *nn_key_parts; + bool has_covering_null_row; + select_materialize_with_stats *result_sink= + (select_materialize_with_stats *) result; + /* Total number of keys needed for partial matching. */ - if (count_partial_match_columns < tmp_table->s->fields) - { - count_pm_keys= count_partial_match_columns + 1; - nn_key_parts= &non_null_key_parts; - } + nn_key_parts= (count_partial_match_columns < tmp_table->s->fields) ? + &non_null_key_parts : NULL; + + has_covering_null_row= (result_sink->get_max_nulls_in_row() == + tmp_table->s->fields - + (nn_key_parts ? bitmap_bits_set(nn_key_parts) : 0)); + + if (has_covering_null_row) + count_pm_keys= nn_key_parts ? 1 : 0; else - { - count_pm_keys= count_partial_match_columns; - nn_key_parts= NULL; - } + count_pm_keys= count_partial_match_columns - count_null_only_columns + + (nn_key_parts ? 1 : 0); - if (!(new_lookup_engine= - new subselect_rowid_merge_engine(lookup_engine, + if (!(rowid_merge_engine= + new subselect_rowid_merge_engine((subselect_uniquesubquery_engine*) + lookup_engine, tmp_table, count_pm_keys, + has_covering_null_row, item, result)) || - new_lookup_engine->init(nn_key_parts, &partial_match_key_parts)) + rowid_merge_engine->init(nn_key_parts, &partial_match_key_parts)) { - delete new_lookup_engine; strategy= PARTIAL_MATCH_SCAN; + delete rowid_merge_engine; /* TODO: setup execution structures for partial match via scanning. */ } - strategy= PARTIAL_MATCH_INDEX; + else + { + strategy= PARTIAL_MATCH_INDEX; + lookup_engine= rowid_merge_engine; + } } item_in->change_engine(lookup_engine); @@ -3632,15 +3775,49 @@ Ordered_key::Ordered_key(uint key_idx_ar ha_rows min_null_row_arg, ha_rows max_null_row_arg, uchar *row_num_to_rowid_arg) : key_idx(key_idx_arg), tbl(tbl_arg), search_key(search_key_arg), - row_num_to_rowid(row_num_to_rowid_arg), null_count(null_count_arg), - min_null_row(min_null_row_arg), max_null_row(max_null_row_arg) + row_num_to_rowid(row_num_to_rowid_arg), null_count(null_count_arg) +{ + DBUG_ASSERT(tbl->file->stats.records > null_count); + key_buff_elements= tbl->file->stats.records - null_count; + cur_key_idx= HA_POS_ERROR; + + DBUG_ASSERT((null_count && min_null_row_arg && max_null_row_arg) || + (!null_count && !min_null_row_arg && !max_null_row_arg)); + if (null_count) + { + /* The counters are 1-based, for key access we need 0-based indexes. */ + min_null_row= min_null_row_arg - 1; + max_null_row= max_null_row_arg - 1; + } + else + min_null_row= max_null_row= 0; +} + + +Ordered_key::~Ordered_key() { - key_column_count= search_key->cols(); - cur_row= HA_POS_ERROR; + /* + All data structures are allocated on thd->mem_root, thus we don't + free them here. + */ } /* + Cleanup that needs to be done for each PS (re)execution. +*/ + +void Ordered_key::cleanup() +{ + /* + Currently these keys are recreated for each PS re-execution, thus + there is nothing to cleanup, the whole object goes away after execution + is over. All handler related initialization/deinitialization is done by + the parent subselect_rowid_merge_engine object. + */ +} + +/* Initialize a multi-column index. */ @@ -3648,10 +3825,10 @@ bool Ordered_key::init(MY_BITMAP *column { THD *thd= tbl->in_use; uint cur_key_col= 0; + Item_field *cur_tmp_field; + Item_func_lt *fn_less_than; - DBUG_ENTER("Ordered_key::init"); - - DBUG_ASSERT(key_column_count == bitmap_bits_set(columns_to_index)); + key_column_count= bitmap_bits_set(columns_to_index); // TODO: check for mem allocation err, revert to scan @@ -3660,22 +3837,26 @@ bool Ordered_key::init(MY_BITMAP *column compare_pred= (Item_func_lt**) thd->alloc(key_column_count * sizeof(Item_func_lt*)); - for (uint i= 0; i < columns_to_index->n_bits; i++, cur_key_col++) + for (uint i= 0; i < columns_to_index->n_bits; i++) { if (!bitmap_is_set(columns_to_index, i)) continue; - key_columns[cur_key_col]= new Item_field(tbl->field[i]); + cur_tmp_field= new Item_field(tbl->field[i]); /* Create the predicate (tmp_column[i] < outer_ref[i]). */ - compare_pred[cur_key_col]= new Item_func_lt(key_columns[cur_key_col], - search_key->element_index(i)); + fn_less_than= new Item_func_lt(cur_tmp_field, + search_key->element_index(i)); + fn_less_than->fix_fields(thd, (Item**) &fn_less_than); + key_columns[cur_key_col]= cur_tmp_field; + compare_pred[cur_key_col]= fn_less_than; + ++cur_key_col; } if (alloc_keys_buffers()) { /* TODO revert to partial match via table scan. */ - DBUG_RETURN(TRUE); + return TRUE; } - DBUG_RETURN(FALSE); + return FALSE; } @@ -3687,9 +3868,7 @@ bool Ordered_key::init(int col_idx) { THD *thd= tbl->in_use; - DBUG_ENTER("Ordered_key::init"); - - DBUG_ASSERT(key_column_count == 1); + key_column_count= 1; // TODO: check for mem allocation err, revert to scan @@ -3700,23 +3879,25 @@ bool Ordered_key::init(int col_idx) /* Create the predicate (tmp_column[i] < outer_ref[i]). */ compare_pred[0]= new Item_func_lt(key_columns[0], search_key->element_index(col_idx)); + compare_pred[0]->fix_fields(thd, (Item**)&compare_pred[0]); if (alloc_keys_buffers()) { /* TODO revert to partial match via table scan. */ - DBUG_RETURN(TRUE); + return TRUE; } - DBUG_RETURN(FALSE); + return FALSE; } bool Ordered_key::alloc_keys_buffers() { THD *thd= tbl->in_use; - ha_rows row_count= tbl->file->stats.records; - if (!(row_index= (ha_rows*) thd->alloc((row_count - null_count) * - sizeof(ha_rows)))) + DBUG_ASSERT(key_buff_elements > 0); + + if (!(key_buff= (rownum_t*) thd->alloc(key_buff_elements * + sizeof(rownum_t)))) return TRUE; /* @@ -3724,10 +3905,14 @@ bool Ordered_key::alloc_keys_buffers() (max_null_row - min_null_row), and then use min_null_row as lookup offset. */ - if (!(bitmap_init_memroot(&null_key, max_null_row, - thd->mem_root))) + if (bitmap_init_memroot(&null_key, + /* this is max array index, we need count, so +1. */ + max_null_row + 1, + thd->mem_root)) return TRUE; + cur_key_idx= HA_POS_ERROR; + return FALSE; } @@ -3735,66 +3920,88 @@ bool Ordered_key::alloc_keys_buffers() /* Quick sort comparison function that compares two rows of the same table indentfied with their row numbers. + + @retval -1 + @retval 0 + @retval +1 */ -int Ordered_key::cmp_rows_by_rownum(Ordered_key *key, ha_rows *a, ha_rows *b) +int +Ordered_key::cmp_keys_by_row_data(ha_rows a, ha_rows b) { uchar *rowid_a, *rowid_b; int error, cmp_res; - TABLE *tbl= key->tbl; /* The length in bytes of the rowids (positions) of tmp_table. */ uint rowid_length= tbl->file->ref_length; - DBUG_ENTER("Ordered_key::cmp_rows_by_rownum"); if (a == b) - DBUG_RETURN(0); + return 0; /* Get the corresponding rowids. */ - rowid_a= key->row_num_to_rowid + (*a) * rowid_length; - rowid_b= key->row_num_to_rowid + (*b) * rowid_length; + rowid_a= row_num_to_rowid + a * rowid_length; + rowid_b= row_num_to_rowid + b * rowid_length; /* Fetch the rows for comparison. */ error= tbl->file->rnd_pos(tbl->record[0], rowid_a); DBUG_ASSERT(!error); error= tbl->file->rnd_pos(tbl->record[1], rowid_b); DBUG_ASSERT(!error); - /* Compare the two rows. */ - for (Field **f_ptr= tbl->field; *f_ptr; f_ptr++) + /* + Compare the two rows by the corresponding values of the indexed + columns. + */ + for (uint i= 0; i < key_column_count; i++) { - if ((cmp_res= (*f_ptr)->cmp_offset(tbl->s->rec_buff_length))) - DBUG_RETURN(cmp_res); + Field *cur_field= key_columns[i]->field; + if ((cmp_res= cur_field->cmp_offset(tbl->s->rec_buff_length))) + return (cmp_res > 0 ? 1 : -1); } - DBUG_RETURN(0); + return 0; +} + + +int +Ordered_key::cmp_keys_by_row_data_and_rownum(Ordered_key *key, + rownum_t* a, rownum_t* b) +{ + /* The result of comparing the two keys according to their row data. */ + int cmp_row_res= key->cmp_keys_by_row_data(*a, *b); + if (cmp_row_res) + return cmp_row_res; + return (*a < *b) ? -1 : (*a > *b) ? 1 : 0; } void Ordered_key::sort_keys() { - my_qsort(row_index, tbl->file->stats.records, sizeof(ha_rows), - (qsort_cmp) &cmp_rows_by_rownum); + my_qsort2(key_buff, key_buff_elements, sizeof(rownum_t), + (qsort2_cmp) &cmp_keys_by_row_data_and_rownum, (void*) this); + /* Invalidate the current row position. */ + cur_key_idx= HA_POS_ERROR; } /* Compare the value(s) of the current key in 'search_key' with the - data of the current table record accessible via 'key_columns'. + data of the current table record. @notes The comparison result follows from the way compare_pred is created in Ordered_key::init. Currently compare_pred compares a field in of the current row with the corresponding Item that contains the search key. + @param row_num Number of the row (not index in the key_buff array) + @retval -1 if (current row < search_key) @retval 0 if (current row == search_key) @retval +1 if (current row > search_key) */ -int Ordered_key::compare_row_with_key(ha_rows row_num) +int Ordered_key::cmp_key_with_search_key(rownum_t row_num) { /* The length in bytes of the rowids (positions) of tmp_table. */ uint rowid_length= tbl->file->ref_length; uchar *cur_rowid= row_num_to_rowid + row_num * rowid_length; int error, cmp_res; - DBUG_ENTER("Ordered_key::compare"); error= tbl->file->rnd_pos(tbl->record[0], cur_rowid); DBUG_ASSERT(!error); @@ -3804,9 +4011,9 @@ int Ordered_key::compare_row_with_key(ha /* Unlike Arg_comparator::compare_row() here there should be no NULLs. */ DBUG_ASSERT(!compare_pred[i]->null_value); if (cmp_res) - DBUG_RETURN(cmp_res); + return (cmp_res > 0 ? 1 : -1); } - DBUG_RETURN(0); + return 0; } @@ -3818,17 +4025,24 @@ int Ordered_key::compare_row_with_key(ha bool Ordered_key::lookup() { - DBUG_ENTER("Ordered_key::lookup"); + DBUG_ASSERT(key_buff_elements); ha_rows lo= 0; - ha_rows hi= tbl->file->stats.records - 1; + ha_rows hi= key_buff_elements - 1; ha_rows mid; int cmp_res; while (lo <= hi) { mid= lo + (hi - lo) / 2; - cmp_res= compare_row_with_key(mid); + cmp_res= cmp_key_with_search_key(key_buff[mid]); + /* + In order to find the minimum match, check if the pevious element is + equal or smaller than the found one. If equal, we need to search further + to the left. + */ + if (!cmp_res && mid > 0) + cmp_res= !cmp_key_with_search_key(key_buff[mid - 1]) ? 1 : 0; if (cmp_res == -1) { @@ -3838,17 +4052,48 @@ bool Ordered_key::lookup() else if (cmp_res == 1) { /* row[mid] > search_key */ + if (!mid) + goto not_found; hi= mid - 1; } else { /* row[mid] == search_key */ - cur_row= mid; - DBUG_RETURN(TRUE); + cur_key_idx= mid; + return TRUE; } } +not_found: + cur_key_idx= HA_POS_ERROR; + return FALSE; +} - DBUG_RETURN(FALSE); + +/* + Move the current index pointer to the next key with the same column + values as the current key. Since the index is sorted, all such keys + are contiguous. +*/ + +bool Ordered_key::next_same() +{ + DBUG_ASSERT(key_buff_elements); + + if (cur_key_idx < key_buff_elements - 1) + { + /* + TODO: + The below is quite inefficient, since as a result we will fetch every + row (except the last one) twice. There must be a more efficient way, + e.g. swapping record[0] and record[1], and reading only the new record. + */ + if (!cmp_keys_by_row_data(key_buff[cur_key_idx], key_buff[cur_key_idx + 1])) + { + ++cur_key_idx; + return TRUE; + } + } + return FALSE; } @@ -3865,56 +4110,147 @@ subselect_rowid_merge_engine::init(MY_BI /* The length in bytes of the rowids (positions) of tmp_table. */ uint rowid_length= tmp_table->file->ref_length; ha_rows row_count= tmp_table->file->stats.records; + rownum_t cur_rownum= 0; select_materialize_with_stats *result_sink= (select_materialize_with_stats *) result; uint cur_key= 0; + Item_in_subselect *item_in= (Item_in_subselect*) item; + int error; - if (!(row_num_to_rowid= (uchar*) thd->alloc(row_count * rowid_length * - sizeof(uchar)))) - return TRUE; + if (keys_count == 0) + { + /* There is nothing to initialize, we will only do regular lookups. */ + return FALSE; + } - if (!(bitmap_init_memroot(&matching_keys, keys_count, thd->mem_root))) + DBUG_ASSERT(!has_covering_null_row || (has_covering_null_row && + keys_count == 1 && + non_null_key_parts)); + + if (!(merge_keys= (Ordered_key**) thd->alloc(keys_count * + sizeof(Ordered_key*))) || + !(row_num_to_rowid= (uchar*) thd->alloc(row_count * rowid_length * + sizeof(uchar)))) return TRUE; - merge_keys= (Ordered_key**) thd->alloc(keys_count * sizeof(Ordered_key*)); /* Create the only non-NULL key if there is any. */ if (non_null_key_parts) { - non_null_key= new Ordered_key(cur_key, tmp_table, item, 0, 0, 0, - row_num_to_rowid); + non_null_key= new Ordered_key(cur_key, tmp_table, item_in->left_expr, + 0, 0, 0, row_num_to_rowid); if (non_null_key->init(non_null_key_parts)) { // TODO: revert to partial matching via scanning return TRUE; } merge_keys[cur_key]= non_null_key; - non_null_key->sort_keys(); + merge_keys[cur_key]->first(); ++cur_key; } + /* - Create one single-column NULL-key for each column in - partial_match_key_parts. + If there is a covering NULL row, the only key that is needed is the + only non-NULL key that is already created above. */ - for (uint i= 0; i < partial_match_key_parts->n_bits; i++, cur_key++) + if (!has_covering_null_row) + { + if (bitmap_init_memroot(&matching_keys, keys_count, thd->mem_root) || + bitmap_init_memroot(&matching_outer_cols, keys_count, thd->mem_root) || + bitmap_init_memroot(&null_only_columns, keys_count, thd->mem_root)) + return TRUE; + + /* + Create one single-column NULL-key for each column in + partial_match_key_parts. + */ + for (uint i= 0; i < partial_match_key_parts->n_bits; i++) + { + if (!bitmap_is_set(partial_match_key_parts, i)) + continue; + + if (result_sink->get_null_count_of_col(i) == row_count) + bitmap_set_bit(&null_only_columns, cur_key); + else + { + merge_keys[cur_key]= new Ordered_key(cur_key, tmp_table, + item_in->left_expr->element_index(i), + result_sink->get_null_count_of_col(i), + result_sink->get_min_null_of_col(i), + result_sink->get_max_null_of_col(i), + row_num_to_rowid); + if (merge_keys[cur_key]->init(i)) + { + // TODO: revert to partial matching via scanning + return TRUE; + } + merge_keys[cur_key]->first(); + } + ++cur_key; + } + } + + /* Populate the indexes with data from the temporary table. */ + tmp_table->file->ha_rnd_init(1); + tmp_table->file->extra_opt(HA_EXTRA_CACHE, + current_thd->variables.read_buff_size); + tmp_table->null_row= 0; + while (TRUE) { - if (!bitmap_is_set(partial_match_key_parts, i)) + error= tmp_table->file->rnd_next(tmp_table->record[0]); + if (error == HA_ERR_RECORD_DELETED) + { + /* We get this for duplicate records that should not be in tmp_table. */ continue; + } + /* + This is a temp table that we fully own, there should be no other + cause to stop the iteration than EOF. + */ + DBUG_ASSERT(!error || error == HA_ERR_END_OF_FILE); + if (error == HA_ERR_END_OF_FILE) + { + DBUG_ASSERT(cur_rownum == tmp_table->file->stats.records); + break; + } - merge_keys[cur_key]= new Ordered_key(cur_key, tmp_table, item, - result_sink->get_null_count_of_col(i), - result_sink->get_min_null_of_col(i), - result_sink->get_max_null_of_col(i), - row_num_to_rowid); - if (merge_keys[cur_key]->init(i)) + /* + Save the position of this record in the row_num -> rowid mapping. + */ + tmp_table->file->position(tmp_table->record[0]); + memcpy(row_num_to_rowid + cur_rownum * rowid_length, + tmp_table->file->ref, rowid_length); + + /* Add the current row number to the corresponding keys. */ + if (non_null_key) { - // TODO: revert to partial matching via scanning - return TRUE; + /* By definition there are no NULLs in the non-NULL key. */ + non_null_key->add_key(cur_rownum); } - merge_keys[cur_key]->sort_keys(); + + for (uint i= (non_null_key ? 1 : 0); i < keys_count; i++) + { + /* + Check if the first and only indexed column contains NULL in the curent + row, and add the row number to the corresponding key. + */ + if (tmp_table->field[merge_keys[i]->get_field_idx(0)]->is_null()) + merge_keys[i]->set_null(cur_rownum); + else + merge_keys[i]->add_key(cur_rownum); + } + ++cur_rownum; } + tmp_table->file->ha_rnd_end(); + + /* Sort the keys in each of the indexes. */ + for (uint i= 0; i < keys_count; i++) + merge_keys[i]->sort_keys(); + + // TODO: sort all the keys by NULL selectivity + if (init_queue(&pq, keys_count, 0, FALSE, - subselect_rowid_merge_engine::cmp_key_by_cur_row, NULL)) + subselect_rowid_merge_engine::cmp_keys_by_cur_rownum, NULL)) { // TODO: revert to partial matching via scanning return TRUE; @@ -3924,9 +4260,19 @@ subselect_rowid_merge_engine::init(MY_BI } +subselect_rowid_merge_engine::~subselect_rowid_merge_engine() +{ + delete_queue(&pq); +} + + void subselect_rowid_merge_engine::cleanup() { - // TODO + lookup_engine->cleanup(); + /* Tell handler we don't need the index anymore */ + if (tmp_table->file->inited) + tmp_table->file->ha_rnd_end(); + queue_remove_all(&pq); } @@ -3934,8 +4280,8 @@ void subselect_rowid_merge_engine::clean */ int -subselect_rowid_merge_engine::cmp_key_by_null_selectivity(Ordered_key *a, - Ordered_key *b) +subselect_rowid_merge_engine::cmp_keys_by_null_selectivity(Ordered_key *a, + Ordered_key *b) { double a_sel= a->null_selectivity(); double b_sel= b->null_selectivity(); @@ -3951,37 +4297,26 @@ subselect_rowid_merge_engine::cmp_key_by */ int -subselect_rowid_merge_engine::cmp_key_by_cur_row(void *arg, - uchar *k1, uchar *k2) +subselect_rowid_merge_engine::cmp_keys_by_cur_rownum(void *arg, + uchar *k1, uchar *k2) { - ha_rows row1= ((Ordered_key*) k1)->current(); - ha_rows row2= ((Ordered_key*) k2)->current(); + rownum_t r1= ((Ordered_key*) k1)->current(); + rownum_t r2= ((Ordered_key*) k2)->current(); - if (row1 > row2) - return 1; - if (row1 == row2) - return 0; - return -1; + return (r1 < r2) ? -1 : (r1 > r2) ? 1 : 0; } /* - Check if certain table row contains a NULL in all columns in all columns for - which there is no value match. - - @details Notice that if a column is not in the set 'keys', we assume that has - been checked otherwise that there is a partial or complete match for this - column. This allows to encode columns that consist of only NULLs as simply - missing in the set 'keys', because such columns match any value in any row. + Check if certain table row contains a NULL in all columns for which there is + no match in the corresponding value index. @retval TRUE if a NULL row exists @retval FALSE otherwise */ -bool subselect_rowid_merge_engine::test_null_row(ha_rows row_num) +bool subselect_rowid_merge_engine::test_null_row(rownum_t row_num) { - DBUG_ENTER("subselect_rowid_merge_engine::test_null_row"); - for (uint i = 0; i < keys_count; i++) { if (bitmap_is_set(&matching_keys, i)) @@ -3993,9 +4328,9 @@ bool subselect_rowid_merge_engine::test_ continue; } if (!merge_keys[i]->is_null(row_num)) - DBUG_RETURN(FALSE); + return FALSE; } - DBUG_RETURN(TRUE); + return TRUE; } @@ -4007,88 +4342,120 @@ bool subselect_rowid_merge_engine::test_ bool subselect_rowid_merge_engine::partial_match() { Ordered_key *min_key; /* Key that contains the current minimum position. */ - ha_rows min_row; /* Current row number of min_key. */ + rownum_t min_row_num; /* Current row number of min_key. */ Ordered_key *cur_key; - ha_rows cur_row; - - DBUG_ENTER("subselect_rowid_merge_engine::partial_match"); + rownum_t cur_row_num; + uint count_nulls_in_search_key= 0; /* If there is a non-NULL key, it must be the first key in the keys array. */ - DBUG_ASSERT(non_null_key && merge_keys[0] == non_null_key); + DBUG_ASSERT(!non_null_key || (non_null_key && merge_keys[0] == non_null_key)); /* Check if there is a match for the columns of the only non-NULL key. */ if (non_null_key && !non_null_key->lookup()) - DBUG_RETURN(FALSE); + return FALSE; + + /* + If there is a NULL (sub)row that covers all NULL-able columns, + then there is a guranteed partial match, and we don't need to search + for the matching row. + */ + if (has_covering_null_row) + return TRUE; + if (non_null_key) queue_insert(&pq, (uchar *) non_null_key); - /* - Add all non-empty value keys to the priority queue. Do not process the - non_null_key, since it was already processed above. + Do not add the non_null_key, since it was already processed above. */ - uint i= non_null_key ? 1 : 0; /* Skip the non-NULL key, already processed. */ - for (; i < keys_count; i++) + bitmap_clear_all(&matching_outer_cols); + for (uint i= test(non_null_key); i < keys_count; i++) { - if (merge_keys[i]->lookup()) + DBUG_ASSERT(merge_keys[i]->get_column_count() == 1); + if (merge_keys[i]->get_search_key(0)->is_null()) + { + ++count_nulls_in_search_key; + bitmap_set_bit(&matching_outer_cols, merge_keys[i]->get_key_idx()); + } + else if (merge_keys[i]->lookup()) queue_insert(&pq, (uchar *) merge_keys[i]); } + /* - Not all value keys are empty, thus we don't have only NULL keys. If we had, - the only possible match is a NULL row, and we cheked there is no such row, - therefore the result is known to be FALSE. In fact this algorithm makes - sense for at least two non-NULL columns. + If the outer reference consists of only NULLs, or if it has NULLs in all + nullable columns, the result is UNKNOWN. */ - DBUG_ASSERT(pq.elements > 1); + if (count_nulls_in_search_key == + ((Item_in_subselect *) item)->left_expr->cols() - + (non_null_key ? non_null_key->get_column_count() : 0)) + return TRUE; + + /* + If there is no NULL (sub)row that covers all NULL columns, and there is no + single match for any of the NULL columns, the result is FALSE. + */ + if (pq.elements - test(non_null_key) == 0) + return FALSE; + + DBUG_ASSERT(pq.elements); + min_key= (Ordered_key*) queue_remove(&pq, 0); - min_row= min_key->current(); - bitmap_clear_all(&matching_keys); + min_row_num= min_key->current(); + bitmap_copy(&matching_keys, &null_only_columns); bitmap_set_bit(&matching_keys, min_key->get_key_idx()); - min_key->next(); - if (!min_key->is_eof()) + bitmap_union(&matching_keys, &matching_outer_cols); + if (min_key->next_same()) queue_insert(&pq, (uchar *) min_key); + if (pq.elements == 0) + { + /* + Check the only matching row of the only key min_key for NULL matches + in the other columns. + */ + if (test_null_row(min_row_num)) + return TRUE; + else + return FALSE; + } + while (TRUE) { cur_key= (Ordered_key*) queue_remove(&pq, 0); - cur_row= min_key->current(); + cur_row_num= cur_key->current(); - if (cur_row == min_row) - { + if (cur_row_num == min_row_num) bitmap_set_bit(&matching_keys, cur_key->get_key_idx()); - /* There cannot be a complete match, as we already checked for one. */ - DBUG_ASSERT(bitmap_bits_set(&matching_keys) < matching_keys.n_bits); - } else { /* Follows from the correct use of priority queue. */ - DBUG_ASSERT(cur_row > min_row); - if (test_null_row(min_row)) - DBUG_RETURN(TRUE); + DBUG_ASSERT(cur_row_num > min_row_num); + if (test_null_row(min_row_num)) + return TRUE; else { min_key= cur_key; - min_row= cur_row; - bitmap_clear_all(&matching_keys); + min_row_num= cur_row_num; + bitmap_copy(&matching_keys, &null_only_columns); bitmap_set_bit(&matching_keys, min_key->get_key_idx()); + bitmap_union(&matching_keys, &matching_outer_cols); } } - cur_key->next(); - if (!cur_key->is_eof()) + if (cur_key->next_same()) queue_insert(&pq, (uchar *) cur_key); if (pq.elements == 0) { /* Check the last row of the last column in PQ for NULL matches. */ - if (test_null_row(min_row)) - DBUG_RETURN(TRUE); + if (test_null_row(min_row_num)) + return TRUE; else - DBUG_RETURN(FALSE); + return FALSE; } } /* We should never get here. */ DBUG_ASSERT(FALSE); - DBUG_RETURN(FALSE); + return FALSE; } @@ -4097,22 +4464,54 @@ int subselect_rowid_merge_engine::exec() Item_in_subselect *item_in= (Item_in_subselect *) item; int res; - DBUG_ENTER("subselect_rowid_merge_engine::exec"); - - if ((res= lookup_engine->exec())) + /* Try to find a matching row by index lookup. */ + res= lookup_engine->copy_ref_key_simple(); + if (res == -1) + { + /* The result is FALSE based on the outer reference. */ + item_in->value= 0; + item_in->null_value= 0; + return 0; + } + else if (res == 0) { - /* An error occured during exec(). */ - DBUG_RETURN(res); + if ((res= lookup_engine->index_lookup())) + { + /* An error occured during lookup(). */ + item_in->value= 0; + item_in->null_value= 0; + return res; + } + else if (item_in->value) + { + /* + A complete match was found, the result of IN is TRUE. + Notice: (this->item == lookup_engine->item) + */ + return 0; + } } - else if (item_in->value == 1) + + if (has_covering_null_row && !keys_count) { /* - A complete match was found, the result of IN is TRUE. - Notice: (this->item == lookup_engine->item) + If there is a NULL-only row that coveres all columns the result of IN + is UNKNOWN. */ - DBUG_RETURN(0); + item_in->value= 0; + /* + TODO: which one is the right way to propagate an UNKNOWN result? + Should we also set empty_result_set= FALSE; ??? + */ + //item_in->was_null= 1; + item_in->null_value= 1; + return 0; } + /* All data accesses during execution are via handler::rnd_pos() */ + if (tmp_table->file->inited) + tmp_table->file->ha_index_end(); + tmp_table->file->ha_rnd_init(0); /* There is no complete match. Look for a partial match (UNKNOWN result), or no match (FALSE). @@ -4121,18 +4520,25 @@ int subselect_rowid_merge_engine::exec() { /* The result of IN is UNKNOWN. */ item_in->value= 0; - /* TODO: which one is the right way to propagate an UNKNOWN result? */ - item_in->was_null= 1; + /* + TODO: which one is the right way to propagate an UNKNOWN result? + Should we also set empty_result_set= FALSE; ??? + */ + //item_in->was_null= 1; item_in->null_value= 1; } else { /* The result of IN is FALSE. */ item_in->value= 0; - /* TODO: which one is the right way to propagate an UNKNOWN result? */ - item_in->was_null= 0; + /* + TODO: which one is the right way to propagate an UNKNOWN result? + Should we also set empty_result_set= FALSE; ??? + */ + //item_in->was_null= 0; item_in->null_value= 0; } + tmp_table->file->ha_rnd_end(); - DBUG_RETURN(0); + return 0; } === modified file 'sql/item_subselect.h' --- a/sql/item_subselect.h 2010-02-01 12:09:48 +0000 +++ b/sql/item_subselect.h 2010-02-12 14:33:43 +0000 @@ -610,8 +610,10 @@ public: virtual void print (String *str, enum_query_type query_type); bool change_result(Item_subselect *si, select_result_interceptor *result); bool no_tables(); + int index_lookup(); int scan_table(); bool copy_ref_key(); + int copy_ref_key_simple(); bool no_rows() { return empty_result_set; } virtual enum_engine_type engine_type() { return UNIQUESUBQUERY_ENGINE; } }; @@ -678,6 +680,34 @@ inline bool Item_subselect::is_uncacheab return engine->uncacheable(); } +/* + Distinguish the type od (0-based) row numbers from the type of the index into + an array of row numbers. +*/ +typedef ha_rows rownum_t; + + +/* + An Ordered_key is an in-memory table index that allows O(log(N)) time + lookups of a multi-part key. + + If the index is over a single column, then this column may contain NULLs, and + the NULLs are stored and tested separately for NULL in O(1) via is_null(). + Multi-part indexes assume that the indexed columns do not contain NULLs. + + TODO: + = Due to the unnatural assymetry between single and multi-part indexes, it + makes sense to somehow refactor or extend the class. + + = This class can be refactored into a base abstract interface, and two + subclasses: + - one to represent single-column indexes, and + - another to represent multi-column indexes. + Such separation would allow slightly more efficient implementation of + the single-column indexes. + = The current design requires such indexes to be fully recreated for each + PS (re)execution, however most of the comprising objects can be reused. +*/ class Ordered_key { @@ -701,11 +731,12 @@ protected: /* Value index related members. */ /* The actual value index, consists of a sorted sequence of row numbers. - There are tbl->file->stats.records elements in this array. */ - ha_rows *row_index; - /* Current element in 'row_index'. */ - ha_rows cur_row; + rownum_t *key_buff; + /* Number of elements in key_buff. */ + ha_rows key_buff_elements; + /* Current element in 'key_buff'. */ + ha_rows cur_key_idx; /* Mapping from row numbers to row ids. The element row_num_to_rowid[i] contains a buffer with the rowid for the row numbered 'i'. @@ -734,15 +765,21 @@ protected: Quick sort comparison function that compares two rows of the same table indentfied with their row numbers. */ - static int cmp_rows_by_rownum(Ordered_key *key, ha_rows* a, ha_rows* b); + int cmp_keys_by_row_data(rownum_t a, rownum_t b); + static int cmp_keys_by_row_data_and_rownum(Ordered_key *key, + rownum_t* a, rownum_t* b); - int compare_row_with_key(ha_rows row_num); + int cmp_key_with_search_key(rownum_t row_num); public: + static void *operator new(size_t size) throw () + { return sql_alloc(size); } Ordered_key(uint key_idx_arg, TABLE *tbl_arg, Item *search_key_arg, ha_rows null_count_arg, ha_rows min_null_row_arg, ha_rows max_null_row_arg, uchar *row_num_to_rowid_arg); + ~Ordered_key(); + void cleanup(); /* Initialize a multi-column index. */ bool init(MY_BITMAP *columns_to_index); /* Initialize a single-column index. */ @@ -750,10 +787,21 @@ public: uint get_column_count() { return key_column_count; } uint get_key_idx() { return key_idx; } - void add_key(ha_rows row_num) + uint get_field_idx(uint i) + { + DBUG_ASSERT(i < key_column_count); + return key_columns[i]->field->field_index; + } + Item *get_search_key(uint i) { - row_index[cur_row]= row_num; - ++cur_row; + return search_key->element_index(key_columns[i]->field->field_index); + } + void add_key(rownum_t row_num) + { + /* The caller must know how many elements to add. */ + DBUG_ASSERT(key_buff_elements && cur_key_idx < key_buff_elements); + key_buff[cur_key_idx]= row_num; + ++cur_key_idx; } void sort_keys(); @@ -766,28 +814,38 @@ public: this->search_key. */ bool lookup(); - /* Return the current index element. */ - ha_rows current() { return row_index[cur_row]; } - /* Move the current index cursor at the next match. */ + /* Move the current index cursor to the first key. */ + void first() + { + DBUG_ASSERT(key_buff_elements); + cur_key_idx= 0; + } + /* TODO */ + bool next_same(); + /* Move the current index cursor to the next key. */ bool next() { - if (cur_row < tbl->file->stats.records) + DBUG_ASSERT(key_buff_elements); + if (cur_key_idx < key_buff_elements - 1) { - ++cur_row; + ++cur_key_idx; return TRUE; } return FALSE; }; - /* Return false if all matches are exhausted, true otherwise. */ - bool is_eof() { return cur_row == tbl->file->stats.records; } + /* Return the current index element. */ + rownum_t current() + { + DBUG_ASSERT(key_buff_elements && cur_key_idx < key_buff_elements); + return key_buff[cur_key_idx]; + } - void set_null(ha_rows row_num) + void set_null(rownum_t row_num) { bitmap_set_bit(&null_key, row_num); } - bool is_null(ha_rows row_num) + bool is_null(rownum_t row_num) { - DBUG_ENTER("Ordered_key::is_null"); /* Indexes consisting of only NULLs do not have a bitmap buffer at all. Their only initialized member is 'n_bits', which is equal to the number @@ -796,11 +854,11 @@ public: if (null_count == tbl->file->stats.records) { DBUG_ASSERT(tbl->file->stats.records == null_key.n_bits); - DBUG_RETURN(TRUE); + return TRUE; } if (row_num > max_null_row || row_num < min_null_row) - DBUG_RETURN(FALSE); - DBUG_RETURN(bitmap_is_set(&null_key, row_num)); + return FALSE; + return bitmap_is_set(&null_key, row_num); } }; @@ -815,18 +873,28 @@ protected: TRUE, then subselect_rowid_merge_engine further distinguishes between FALSE and UNKNOWN. */ - subselect_engine *lookup_engine; + subselect_uniquesubquery_engine *lookup_engine; /* - Mapping from row numbers to row ids. The element row_num_to_rowid[i] - contains a buffer with the rowid for the row numbered 'i'. + Mapping from row numbers to row ids. The rowids are stored sequentially + in the array - rowid[i] is located in row_num_to_rowid + i * rowid_length. */ uchar *row_num_to_rowid; /* A subset of all the keys for which there is a match for the same row. - Used during execution. Computed for each call to exec(). + Used during execution. Computed for each outer reference */ MY_BITMAP matching_keys; /* + The columns of the outer reference that are NULL. Computed for each + outer reference. + */ + MY_BITMAP matching_outer_cols; + /* + Columns that consist of only NULLs. Such columns match any value. + Computed once per query execution. + */ + MY_BITMAP null_only_columns; + /* Indexes of row numbers, sorted by <column_value, row_number>. If an index may contain NULLs, the NULLs are stored efficiently in a bitmap. @@ -849,44 +917,59 @@ protected: This queue is used by the partial match algorithm in method exec(). */ QUEUE pq; + /* True if there is a NULL (sub)row that covers all NULLable columns. */ + bool has_covering_null_row; protected: /* Comparison function to compare keys in order of increasing bitmap selectivity. */ - static int cmp_key_by_null_selectivity(Ordered_key *a, Ordered_key *b); + static int cmp_keys_by_null_selectivity(Ordered_key *a, Ordered_key *b); /* Comparison function used by the priority queue pq, the 'smaller' key is the one with the smaller current row number. */ - static int cmp_key_by_cur_row(void *arg, uchar *k1, uchar *k2); + static int cmp_keys_by_cur_rownum(void *arg, uchar *k1, uchar *k2); - bool test_null_row(ha_rows row_num); + bool test_null_row(rownum_t row_num); bool partial_match(); public: - subselect_rowid_merge_engine(subselect_engine *lookup_engine_arg, + subselect_rowid_merge_engine(subselect_uniquesubquery_engine *engine_arg, TABLE *tmp_table_arg, uint keys_count_arg, + uint has_covering_null_row_arg, Item_subselect *item_arg, select_result_interceptor *result_arg) :subselect_engine(item_arg, result_arg), - tmp_table(tmp_table_arg), lookup_engine(lookup_engine_arg), - keys_count(keys_count_arg) - {} - + tmp_table(tmp_table_arg), lookup_engine(engine_arg), + keys_count(keys_count_arg), non_null_key(NULL), + has_covering_null_row(has_covering_null_row_arg) + { + thd= lookup_engine->get_thd(); + } + ~subselect_rowid_merge_engine(); bool init(MY_BITMAP *non_null_key_parts, MY_BITMAP *partial_match_key_parts); void cleanup(); int prepare() { return 0; } void fix_length_and_dec(Item_cache**) {} int exec(); - uint cols() { return 0; } + uint cols() { /* TODO: what is the correct value? */ return 1; } uint8 uncacheable() { return UNCACHEABLE_DEPENDENT; } void exclude() {} table_map upper_select_const_tables() { return 0; } void print(String*, enum_query_type) {} bool change_result(Item_subselect*, select_result_interceptor*) - { return false; } + { DBUG_ASSERT(FALSE); return false; } bool no_tables() { return false; } - bool no_rows() {return false; } + bool no_rows() + { + /* + TODO: It is completely unclear what is the semantics of this + method. The current result is computed so that the call to no_rows() + from Item_in_optimizer::val_int() sets Item_in_optimizer::null_value + correctly. + */ + return !(((Item_in_subselect *) item)->null_value); + } }; @@ -933,6 +1016,7 @@ protected: /* Keyparts of the single column indexes with NULL, one keypart per index. */ MY_BITMAP partial_match_key_parts; uint count_partial_match_columns; + uint count_null_only_columns; /* A conjunction of all the equality condtions between all pairs of expressions that are arguments of an IN predicate. We need these to post-filter some @@ -962,7 +1046,7 @@ public: :subselect_engine(in_predicate, NULL), tmp_table(NULL), is_materialized(FALSE), materialize_engine(old_engine), lookup_engine(NULL), materialize_join(NULL), count_partial_match_columns(0), - semi_join_conds(NULL) + count_null_only_columns(0), semi_join_conds(NULL) { set_thd(thd); } === modified file 'sql/sql_class.cc' --- a/sql/sql_class.cc 2010-01-22 16:18:05 +0000 +++ b/sql/sql_class.cc 2010-02-12 14:33:43 +0000 @@ -2931,12 +2931,13 @@ create_result_table(THD *thd_arg, List<I options, HA_POS_ERROR, (char*) table_alias))) return TRUE; - /* TODO: if/where/when to free this buffer? */ - col_stat= (Column_statistics*) table->in_use->calloc(table->s->fields * - sizeof(Column_statistics)); + col_stat= (Column_statistics*) table->in_use->alloc(table->s->fields * + sizeof(Column_statistics)); if (!stat) return TRUE; + cleanup(); + table->file->extra(HA_EXTRA_WRITE_CACHE); table->file->extra(HA_EXTRA_IGNORE_DUP_KEY); return FALSE; @@ -2966,14 +2967,14 @@ bool select_materialize_with_stats::send { ++cur_col_stat->null_count; cur_col_stat->max_null_row= count_rows; - if (cur_col_stat->min_null_row == 0) + if (!cur_col_stat->min_null_row) cur_col_stat->min_null_row= count_rows; ++nulls_in_row; } ++cur_col_stat; } - if (nulls_in_row == items.elements) - ++null_record_count; + if (nulls_in_row > max_nulls_in_row) + max_nulls_in_row= nulls_in_row; return select_union::send_data(items); } === modified file 'sql/sql_class.h' --- a/sql/sql_class.h 2010-02-01 12:09:48 +0000 +++ b/sql/sql_class.h 2010-02-12 14:33:43 +0000 @@ -3044,17 +3044,20 @@ protected: public: /* Count of NULLs per column. */ ha_rows null_count; - /* The row number that contains the last NULL in a column. */ - ha_rows max_null_row; /* The row number that contains the first NULL in a column. */ ha_rows min_null_row; + /* The row number that contains the last NULL in a column. */ + ha_rows max_null_row; }; /* Array of statistics data per column. */ Column_statistics* col_stat; - /* The number of rows that consist only of NULL values. */ - ha_rows null_record_count; + /* + The number of columns in the biggest sub-row that consists of only + NULL values. + */ + ha_rows max_nulls_in_row; /* Count of rows writtent to the temp table. This is redundant as it is already stored in handler::stats.records, however that one is relatively @@ -3063,11 +3066,7 @@ protected: ha_rows count_rows; public: - select_materialize_with_stats() - { - null_record_count= 0; - count_rows= 0; - } + select_materialize_with_stats() {} virtual bool create_result_table(THD *thd, List<Item> *column_types, bool is_distinct, ulonglong options, const char *alias, bool bit_fields_as_long); @@ -3075,9 +3074,9 @@ public: bool send_data(List<Item> &items); void cleanup() { - null_record_count= 0; - count_rows= 0; memset(col_stat, 0, table->s->fields * sizeof(Column_statistics)); + max_nulls_in_row= 0; + count_rows= 0; } ha_rows get_null_count_of_col(uint idx) { @@ -3094,7 +3093,7 @@ public: DBUG_ASSERT(idx < table->s->fields); return col_stat[idx].min_null_row; } - ha_rows get_null_record_count() { return null_record_count; } + ha_rows get_max_nulls_in_row() { return max_nulls_in_row; } }; === modified file 'sql/sql_select.cc' --- a/sql/sql_select.cc 2010-01-22 16:18:05 +0000 +++ b/sql/sql_select.cc 2010-02-12 14:33:43 +0000 @@ -707,7 +707,7 @@ JOIN::prepare(Item ***rref_pointer_array subquery_types_allow_materialization(in_subs)) { // psergey-todo: duplicated_subselect_card_check: where it's done? - if (in_subs->is_top_level_item() && // 4 + if (//in_subs->is_top_level_item() && // 4 !in_subs->is_correlated && // 5 in_subs->exec_method == Item_in_subselect::NOT_TRANSFORMED) // 6 in_subs->exec_method= Item_in_subselect::MATERIALIZATION;

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2817: Fix for LPBUG#516148 Test maria.maria3 fails when --without-maria-tmp-tables is set
by noreply＠launchpad.net 12 Feb '10

12 Feb '10

------------------------------------------------------------ revno: 2817 committer: Michael Widenius <monty(a)askmonty.org> branch nick: maria-5.1 timestamp: Fri 2010-02-12 16:21:13 +0200 message: Fix for LPBUG#516148 Test maria.maria3 fails when --without-maria-tmp-tables is set modified: mysql-test/suite/maria/r/maria3.result mysql-test/suite/maria/t/maria3.test storage/maria/ha_maria.cc -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2817)
by Michael Widenius 12 Feb '10

12 Feb '10

#At lp:maria based on revid:monty@askmonty.org-20100211191524-rbd8pfcchi9ewm4a 2817 Michael Widenius 2010-02-12 Fix for LPBUG#516148 Test maria.maria3 fails when --without-maria-tmp-tables is set modified: mysql-test/suite/maria/r/maria3.result mysql-test/suite/maria/t/maria3.test storage/maria/ha_maria.cc per-file messages: mysql-test/suite/maria/r/maria3.result Updated test results mysql-test/suite/maria/t/maria3.test Don't show maria_used_for_temp_tables, as it's value is depending on configure options === modified file 'mysql-test/suite/maria/r/maria3.result' --- a/mysql-test/suite/maria/r/maria3.result 2009-09-18 01:04:43 +0000 +++ b/mysql-test/suite/maria/r/maria3.result 2010-02-12 14:21:13 +0000 @@ -301,7 +301,7 @@ check table t1 extended; Table Op Msg_type Msg_text test.t1 check status OK drop table t1; -show variables like 'maria%'; +select lower(variable_name) as Variable_name, Variable_value as Value from information_schema.session_variables where variable_name like "maria%" and variable_name not like "maria_used_for_temp_tables" order by 1; Variable_name Value maria_block_size 8192 maria_checkpoint_interval 30 @@ -309,16 +309,15 @@ maria_force_start_after_recovery_failure maria_log_file_size 4294959104 maria_log_purge_type immediate maria_max_sort_file_size 9223372036853727232 -maria_page_checksum OFF maria_pagecache_age_threshold 300 maria_pagecache_buffer_size 8384512 maria_pagecache_division_limit 100 +maria_page_checksum OFF maria_recover OFF maria_repair_threads 1 maria_sort_buffer_size 8388608 maria_stats_method nulls_unequal maria_sync_log_dir NEWFILE -maria_used_for_temp_tables ON show status like 'maria%'; Variable_name Value Maria_pagecache_blocks_not_flushed # === modified file 'mysql-test/suite/maria/t/maria3.test' --- a/mysql-test/suite/maria/t/maria3.test 2009-06-02 09:58:27 +0000 +++ b/mysql-test/suite/maria/t/maria3.test 2010-02-12 14:21:13 +0000 @@ -259,7 +259,7 @@ drop table t1; # Fix if we are using safemalloc --replace_result 8388572 8388600 -show variables like 'maria%'; +select lower(variable_name) as Variable_name, Variable_value as Value from information_schema.session_variables where variable_name like "maria%" and variable_name not like "maria_used_for_temp_tables" order by 1; --replace_column 2 # show status like 'maria%'; === modified file 'storage/maria/ha_maria.cc' --- a/storage/maria/ha_maria.cc 2010-02-10 19:06:24 +0000 +++ b/storage/maria/ha_maria.cc 2010-02-12 14:21:13 +0000 @@ -3278,11 +3278,11 @@ static struct st_mysql_sys_var* system_v MYSQL_SYSVAR(block_size), MYSQL_SYSVAR(checkpoint_interval), MYSQL_SYSVAR(force_start_after_recovery_failures), - MYSQL_SYSVAR(page_checksum), MYSQL_SYSVAR(log_dir_path), MYSQL_SYSVAR(log_file_size), MYSQL_SYSVAR(log_purge_type), MYSQL_SYSVAR(max_sort_file_size), + MYSQL_SYSVAR(page_checksum), MYSQL_SYSVAR(pagecache_age_threshold), MYSQL_SYSVAR(pagecache_buffer_size), MYSQL_SYSVAR(pagecache_division_limit),

1 0

[Maria-developers] Rev 2741: Group commit for maria engine. in file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit2/
by sanja＠askmonty.org 12 Feb '10

12 Feb '10

At file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit2/ ------------------------------------------------------------ revno: 2741 revision-id: sanja(a)askmonty.org-20100212131228-bgxli0wfybhjkvg9 parent: sergii(a)pisem.net-20100212084731-b5jst7oxhzp251pg committer: sanja(a)askmonty.org branch nick: work-maria-5.2-groupcommit2 timestamp: Fri 2010-02-12 15:12:28 +0200 message: Group commit for maria engine. === added file 'mysql-test/suite/maria/r/group_commit.result' --- a/mysql-test/suite/maria/r/group_commit.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/maria/r/group_commit.result 2010-02-12 13:12:28 +0000 @@ -0,0 +1,17 @@ +drop table if exists t1; +create table t1 (a int); +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +drop table t1; === modified file 'mysql-test/suite/maria/r/maria3.result' --- a/mysql-test/suite/maria/r/maria3.result 2009-09-18 01:04:43 +0000 +++ b/mysql-test/suite/maria/r/maria3.result 2010-02-12 13:12:28 +0000 @@ -306,6 +306,8 @@ maria_block_size 8192 maria_checkpoint_interval 30 maria_force_start_after_recovery_failures 0 +maria_group_commit none +maria_group_commit_interval 0 maria_log_file_size 4294959104 maria_log_purge_type immediate maria_max_sort_file_size 9223372036853727232 @@ -328,6 +330,7 @@ Maria_pagecache_reads # Maria_pagecache_write_requests # Maria_pagecache_writes # +Maria_transaction_log_syncs # create table t1 (b char(0)); insert into t1 values(NULL),(""); select length(b) from t1; === added file 'mysql-test/suite/maria/t/group_commit.test' --- a/mysql-test/suite/maria/t/group_commit.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/maria/t/group_commit.test 2010-02-12 13:12:28 +0000 @@ -0,0 +1,71 @@ +# Test different ways of syncing (mostly syntax) + +--disable_warnings +drop table if exists t1; +--enable_warnings + +create table t1 (a int); + +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +drop table t1; === added directory 'randgen' === added directory 'randgen/conf' === added file 'randgen/conf/maria_group_commit.yy' --- a/randgen/conf/maria_group_commit.yy 1970-01-01 00:00:00 +0000 +++ b/randgen/conf/maria_group_commit.yy 2010-02-12 13:12:28 +0000 @@ -0,0 +1,181 @@ +# test of group commit switching + +query: + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + change_group_commit | change_interval; + + +select: + SELECT select_item FROM join where order_by limit; + +select_item: + * | X . _field ; + +join: + _table AS X | + _table AS X LEFT JOIN _table AS Y ON ( X . _field = Y . _field ) ; + +where: + | + WHERE X . _field < value | + WHERE X . _field > value | + WHERE X . _field = value ; + +where_delete: + | + WHERE _field < value | + WHERE _field > value | + WHERE _field = value ; + +order_by: + | ORDER BY X . _field ; + +limit: + | LIMIT _digit ; + +insert: + INSERT INTO _table ( _field , _field ) VALUES ( value , value ) ; + +update: + UPDATE _table AS X SET _field = value where order_by limit ; + +delete: + DELETE FROM _table where_delete LIMIT _digit ; + +value: + ' _letter ' | _digit | _date | _datetime | _time | _english ; + +change_group_commit: + SET GLOBAL MARIA_GROUP_COMMIT=none_soft_hard; + +none_soft_hard: + NONE | SOFT | HARD; + +change_interval: + set_interval | set_interval | set_interval | set_interval | + drop_interval; + +set_interval: + SET GLOBAL MARIA_GROUP_COMMIT_INTERVAL=_tinyint_unsigned; + +drop_interval: + SET GLOBAL MARIA_GROUP_COMMIT_INTERVAL=0; === modified file 'storage/maria/ha_maria.cc' --- a/storage/maria/ha_maria.cc 2010-02-10 19:06:24 +0000 +++ b/storage/maria/ha_maria.cc 2010-02-12 13:12:28 +0000 @@ -102,22 +102,40 @@ array_elements(maria_translog_purge_type_names) - 1, "", maria_translog_purge_type_names, NULL }; + +/* transactional log directory sync */ const char *maria_sync_log_dir_names[]= { "NEVER", "NEWFILE", "ALWAYS", NullS }; - TYPELIB maria_sync_log_dir_typelib= { array_elements(maria_sync_log_dir_names) - 1, "", maria_sync_log_dir_names, NULL }; +/* transactional log group commit */ +const char *maria_group_commit_names[]= +{ + "none", "hard", "soft", NullS +}; +TYPELIB maria_group_commit_typelib= +{ + array_elements(maria_group_commit_names) - 1, "", + maria_group_commit_names, NULL +}; + /** Interval between background checkpoints in seconds */ static ulong checkpoint_interval; static void update_checkpoint_interval(MYSQL_THD thd, struct st_mysql_sys_var *var, void *var_ptr, const void *save); +static void update_maria_group_commit(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save); +static void update_maria_group_commit_interval(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save); /** After that many consecutive recovery failures, remove logs */ static ulong force_start_after_recovery_failures; static void update_log_file_size(MYSQL_THD thd, @@ -164,6 +182,24 @@ NULL, update_log_file_size, TRANSLOG_FILE_SIZE, TRANSLOG_MIN_FILE_SIZE, 0xffffffffL, TRANSLOG_PAGE_SIZE); +static MYSQL_SYSVAR_ENUM(group_commit, maria_group_commit, + PLUGIN_VAR_RQCMDARG, + "Specifies maria group commit mode. " + "Possible values are \"none\" (no group commit), " + "\"hard\" (with waiting to actual commit), " + "\"soft\" (no wait for commit (DANGEROUS!!!))", + NULL, update_maria_group_commit, + TRANSLOG_GCOMMIT_NONE, &maria_group_commit_typelib); + +static MYSQL_SYSVAR_ULONG(group_commit_interval, maria_group_commit_interval, + PLUGIN_VAR_RQCMDARG, + "Interval between commite in microseconds (1/1000000c)." + " 0 stands for no waiting" + " for other threads to come and do a commit in \"hard\" mode and no" + " sync()/commit at all in \"soft\" mode. Option has only an effect" + " if maria_group_commit is used", + NULL, update_maria_group_commit_interval, 0, 0, UINT_MAX, 1); + static MYSQL_SYSVAR_ENUM(log_purge_type, log_purge_type, PLUGIN_VAR_RQCMDARG, "Specifies how maria transactional log will be purged. " @@ -3278,6 +3314,8 @@ MYSQL_SYSVAR(block_size), MYSQL_SYSVAR(checkpoint_interval), MYSQL_SYSVAR(force_start_after_recovery_failures), + MYSQL_SYSVAR(group_commit), + MYSQL_SYSVAR(group_commit_interval), MYSQL_SYSVAR(page_checksum), MYSQL_SYSVAR(log_dir_path), MYSQL_SYSVAR(log_file_size), @@ -3309,6 +3347,92 @@ } /** + @brief Updates group commit mode +*/ + +static void update_maria_group_commit(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save) +{ + ulong value= (ulong)*((long *)var_ptr); + DBUG_ENTER("update_maria_group_commit"); + DBUG_PRINT("enter", ("old value: %lu new value %lu rate %lu", + value, (ulong)(*(long *)save), + maria_group_commit_interval)); + /* old value */ + switch (value) { + case TRANSLOG_GCOMMIT_NONE: + break; + case TRANSLOG_GCOMMIT_HARD: + translog_hard_group_commit(FALSE); + break; + case TRANSLOG_GCOMMIT_SOFT: + translog_soft_sync(FALSE); + if (maria_group_commit_interval) + translog_soft_sync_end(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + value= *(ulong *)var_ptr= (ulong)(*(long *)save); + translog_sync(); + /* new value */ + switch (value) { + case TRANSLOG_GCOMMIT_NONE: + break; + case TRANSLOG_GCOMMIT_HARD: + translog_hard_group_commit(TRUE); + break; + case TRANSLOG_GCOMMIT_SOFT: + translog_soft_sync(TRUE); + /* variable change made under global lock so we can just read it */ + if (maria_group_commit_interval) + translog_soft_sync_start(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + DBUG_VOID_RETURN; +} + +/** + @brief Updates group commit interval +*/ + +static void update_maria_group_commit_interval(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save) +{ + ulong new_value= (ulong)*((long *)save); + ulong *value_ptr= (ulong*) var_ptr; + DBUG_ENTER("update_maria_group_commit_interval"); + DBUG_PRINT("enter", ("old value: %lu new value %lu group commit %lu", + *value_ptr, new_value, maria_group_commit)); + + /* variable change made under global lock so we can just read it */ + switch (maria_group_commit) { + case TRANSLOG_GCOMMIT_NONE: + *value_ptr= new_value; + translog_set_group_commit_interval(new_value); + break; + case TRANSLOG_GCOMMIT_HARD: + *value_ptr= new_value; + translog_set_group_commit_interval(new_value); + break; + case TRANSLOG_GCOMMIT_SOFT: + if (*value_ptr) + translog_soft_sync_end(); + translog_set_group_commit_interval(new_value); + if ((*value_ptr= new_value)) + translog_soft_sync_start(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + DBUG_VOID_RETURN; +} + +/** @brief Updates the transaction log file limit. */ @@ -3330,6 +3454,7 @@ {"Maria_pagecache_reads", (char*) &maria_pagecache_var.global_cache_read, SHOW_LONGLONG}, {"Maria_pagecache_write_requests", (char*) &maria_pagecache_var.global_cache_w_requests, SHOW_LONGLONG}, {"Maria_pagecache_writes", (char*) &maria_pagecache_var.global_cache_write, SHOW_LONGLONG}, + {"Maria_transaction_log_syncs", (char*) &translog_syncs, SHOW_LONGLONG}, {NullS, NullS, SHOW_LONG} }; === modified file 'storage/maria/ma_init.c' --- a/storage/maria/ma_init.c 2008-10-09 20:03:54 +0000 +++ b/storage/maria/ma_init.c 2010-02-12 13:12:28 +0000 @@ -82,6 +82,11 @@ maria_inited= maria_multi_threaded= FALSE; ft_free_stopwords(); ma_checkpoint_end(); + if (translog_status == TRANSLOG_OK) + { + translog_soft_sync_end(); + translog_sync(); + } if ((trid= trnman_get_max_trid()) > max_trid_in_control_file) { /* === modified file 'storage/maria/ma_loghandler.c' --- a/storage/maria/ma_loghandler.c 2010-01-06 21:27:53 +0000 +++ b/storage/maria/ma_loghandler.c 2010-02-12 13:12:28 +0000 @@ -18,6 +18,7 @@ #include "ma_blockrec.h" /* for some constants and in-write hooks */ #include "ma_key_recover.h" /* For some in-write hooks */ #include "ma_checkpoint.h" +#include "ma_servicethread.h" /* On Windows, neither my_open() nor my_sync() work for directories. @@ -47,6 +48,15 @@ #include <m_ctype.h> #endif +/** @brief protects checkpoint_in_progress */ +static pthread_mutex_t LOCK_soft_sync; +/** @brief for killing the background checkpoint thread */ +static pthread_cond_t COND_soft_sync; +/** @brief control structure for checkpoint background thread */ +static MA_SERVICE_THREAD_CONTROL soft_sync_control= + {THREAD_DEAD, FALSE, &LOCK_soft_sync, &COND_soft_sync}; + + /* transaction log file descriptor */ typedef struct st_translog_file { @@ -124,10 +134,24 @@ /* Previous buffer offset to detect it flush finish */ TRANSLOG_ADDRESS prev_buffer_offset; /* + If the buffer was forced to close it save value of its horizon + otherwise LSN_IMPOSSIBLE + */ + TRANSLOG_ADDRESS pre_force_close_horizon; + /* How much is written (or will be written when copy_to_buffer_in_progress become 0) to this buffer */ translog_size_t size; + /* + When moving from one log buffer to another, we write the last of the + previous buffer to file and then move to start using the new log + buffer. In the case of a part filed last page, this page is not moved + to the start of the new buffer but instead we set the 'skip_data' + variable to tell us how much data at the beginning of the buffer is not + relevant. + */ + uint skipped_data; /* File handler for this buffer */ TRANSLOG_FILE *file; /* Threads which are waiting for buffer filling/freeing */ @@ -304,6 +328,7 @@ */ pthread_mutex_t log_flush_lock; pthread_cond_t log_flush_cond; + pthread_cond_t new_goal_cond; /* Protects changing of headers of finished files (max_lsn) */ pthread_mutex_t file_header_lock; @@ -344,13 +369,39 @@ ulong log_purge_type= TRANSLOG_PURGE_IMMIDIATE; ulong log_file_size= TRANSLOG_FILE_SIZE; +/* sync() of log files directory mode */ ulong sync_log_dir= TRANSLOG_SYNC_DIR_NEWFILE; +ulong maria_group_commit= TRANSLOG_GCOMMIT_NONE; +ulong maria_group_commit_interval= 0; /* Marker for end of log */ static uchar end_of_log= 0; #define END_OF_LOG &end_of_log +/** + Switch for "soft" sync (no real sync() but periodical sync by service + thread) +*/ +static volatile my_bool soft_sync= FALSE; +/** + Switch for "hard" group commit mode +*/ +static volatile my_bool hard_group_commit= FALSE; +/** + File numbers interval which have to be sync() +*/ +static uint32 soft_sync_min= 0; +static uint32 soft_sync_max= 0; +static uint32 soft_need_sync= 1; +/** + stores interval in microseconds +*/ +static uint32 group_commit_wait= 0; enum enum_translog_status translog_status= TRANSLOG_UNINITED; +ulonglong translog_syncs= 0; /* Number of sync()s */ + +/* time of last flush */ +static ulonglong flush_start= 0; /* chunk types */ #define TRANSLOG_CHUNK_LSN 0x00 /* 0 chunk refer as LSN (head or tail */ @@ -980,12 +1031,17 @@ static TRANSLOG_FILE *get_current_logfile() { TRANSLOG_FILE *file; + DBUG_ENTER("get_current_logfile"); rw_rdlock(&log_descriptor.open_files_lock); + DBUG_PRINT("info", ("max_file: %lu min_file: %lu open_files: %lu", + (ulong) log_descriptor.max_file, + (ulong) log_descriptor.min_file, + (ulong) log_descriptor.open_files.elements)); DBUG_ASSERT(log_descriptor.max_file - log_descriptor.min_file + 1 == log_descriptor.open_files.elements); file= *dynamic_element(&log_descriptor.open_files, 0, TRANSLOG_FILE **); rw_unlock(&log_descriptor.open_files_lock); - return (file); + DBUG_RETURN(file); } uchar NEAR maria_trans_file_magic[]= @@ -1069,6 +1125,7 @@ static my_bool translog_max_lsn_to_header(File file, LSN lsn) { uchar lsn_buff[LSN_STORE_SIZE]; + my_bool rc; DBUG_ENTER("translog_max_lsn_to_header"); DBUG_PRINT("enter", ("File descriptor: %ld " "lsn: (%lu,0x%lx)", @@ -1077,11 +1134,17 @@ lsn_store(lsn_buff, lsn); - DBUG_RETURN(my_pwrite(file, lsn_buff, - LSN_STORE_SIZE, - (LOG_HEADER_DATA_SIZE - LSN_STORE_SIZE), - log_write_flags) != 0 || - my_sync(file, MYF(MY_WME)) != 0); + rc= (my_pwrite(file, lsn_buff, + LSN_STORE_SIZE, + (LOG_HEADER_DATA_SIZE - LSN_STORE_SIZE), + log_write_flags) != 0 || + my_sync(file, MYF(MY_WME)) != 0); + /* + We should not increase counter in case of error above, but it is so + unlikely that we can ignore this case + */ + translog_syncs++; + DBUG_RETURN(rc); } @@ -1423,7 +1486,9 @@ static my_bool translog_buffer_init(struct st_translog_buffer *buffer, int num) { DBUG_ENTER("translog_buffer_init"); - buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; + buffer->pre_force_close_horizon= + buffer->prev_last_lsn= buffer->last_lsn= + LSN_IMPOSSIBLE; DBUG_PRINT("info", ("last_lsn and prev_last_lsn set to 0 buffer: 0x%lx", (ulong) buffer)); @@ -1435,6 +1500,7 @@ memset(buffer->buffer, TRANSLOG_FILLER, TRANSLOG_WRITE_BUFFER); /* Buffer size */ buffer->size= 0; + buffer->skipped_data= 0; /* cond of thread which is waiting for buffer filling */ if (pthread_cond_init(&buffer->waiting_filling_buffer, 0)) DBUG_RETURN(1); @@ -1489,7 +1555,10 @@ TODO: sync only we have changed the log */ if (!file->is_sync) + { rc= my_sync(file->handler.file, MYF(MY_WME)); + translog_syncs++; + } rc|= my_close(file->handler.file, MYF(MY_WME)); my_free(file, MYF(0)); return test(rc); @@ -2044,7 +2113,8 @@ (ulong) LSN_OFFSET(log_descriptor.horizon), (ulong) LSN_OFFSET(log_descriptor.horizon))); DBUG_ASSERT(buffer_no == buffer->buffer_no); - buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; + buffer->pre_force_close_horizon= + buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; DBUG_PRINT("info", ("last_lsn and prev_last_lsn set to 0 buffer: 0x%lx", (ulong) buffer)); buffer->offset= log_descriptor.horizon; @@ -2052,6 +2122,7 @@ buffer->file= get_current_logfile(); buffer->overlay= 0; buffer->size= 0; + buffer->skipped_data= 0; translog_cursor_init(cursor, buffer, buffer_no); DBUG_PRINT("info", ("file: #%ld (%d) init cursor #%u: 0x%lx " "chaser: %d Size: %lu (%lu)", @@ -2523,6 +2594,7 @@ TRANSLOG_ADDRESS offset= buffer->offset; TRANSLOG_FILE *file= buffer->file; uint8 ver= buffer->ver; + uint skipped_data; DBUG_ENTER("translog_buffer_flush"); DBUG_PRINT("enter", ("Buffer: #%u 0x%lx file: %d offset: (%lu,0x%lx) size: %lu", @@ -2557,6 +2629,8 @@ disk */ file= buffer->file; + skipped_data= buffer->skipped_data; + DBUG_ASSERT(skipped_data < TRANSLOG_PAGE_SIZE); for (i= 0, pg= LSN_OFFSET(buffer->offset) / TRANSLOG_PAGE_SIZE; i < buffer->size; i+= TRANSLOG_PAGE_SIZE, pg++) @@ -2573,13 +2647,16 @@ DBUG_ASSERT(i + TRANSLOG_PAGE_SIZE <= buffer->size); if (translog_status != TRANSLOG_OK && translog_status != TRANSLOG_SHUTDOWN) DBUG_RETURN(1); - if (pagecache_inject(log_descriptor.pagecache, + if (pagecache_write_part(log_descriptor.pagecache, &file->handler, pg, 3, buffer->buffer + i, PAGECACHE_PLAIN_PAGE, PAGECACHE_LOCK_LEFT_UNLOCKED, - PAGECACHE_PIN_LEFT_UNPINNED, 0, - LSN_IMPOSSIBLE)) + PAGECACHE_PIN_LEFT_UNPINNED, + PAGECACHE_WRITE_DONE, 0, + LSN_IMPOSSIBLE, + skipped_data, + TRANSLOG_PAGE_SIZE - skipped_data)) { DBUG_PRINT("error", ("Can't write page (%lu,0x%lx) to pagecache, error: %d", @@ -2589,10 +2666,12 @@ translog_stop_writing(); DBUG_RETURN(1); } + skipped_data= 0; } file->is_sync= 0; - if (my_pwrite(file->handler.file, buffer->buffer, - buffer->size, LSN_OFFSET(buffer->offset), + if (my_pwrite(file->handler.file, buffer->buffer + buffer->skipped_data, + buffer->size - buffer->skipped_data, + LSN_OFFSET(buffer->offset) + buffer->skipped_data, log_write_flags)) { DBUG_PRINT("error", ("Can't write buffer (%lu,0x%lx) size %lu " @@ -2985,6 +3064,7 @@ uchar *from, *table= NULL; int is_last_unfinished_page; uint last_protected_sector= 0; + uint skipped_data= curr_buffer->skipped_data; TRANSLOG_FILE file_copy; uint8 ver= curr_buffer->ver; translog_wait_for_writers(curr_buffer); @@ -2997,7 +3077,38 @@ } DBUG_ASSERT(LSN_FILE_NO(addr) == LSN_FILE_NO(curr_buffer->offset)); from= curr_buffer->buffer + (addr - curr_buffer->offset); - memcpy(buffer, from, TRANSLOG_PAGE_SIZE); + if (skipped_data && addr == curr_buffer->offset) + { + /* + We read page part of which is not present in buffer, + so we should read absent part from file (page cache actually) + */ + file= get_logfile_by_number(file_no); + DBUG_ASSERT(file != NULL); + /* + it's ok to not lock the page because: + - The log handler has it's own page cache. + - There is only one thread that can access the log + cache at a time + */ + if (!(buffer= pagecache_read(log_descriptor.pagecache, + &file->handler, + LSN_OFFSET(addr) / TRANSLOG_PAGE_SIZE, + 3, buffer, + PAGECACHE_PLAIN_PAGE, + PAGECACHE_LOCK_LEFT_UNLOCKED, + NULL))) + DBUG_RETURN(NULL); + } + else + skipped_data= 0; /* Read after skipped in buffer data */ + /* + Now we have correct data in buffer up to 'skipped_data'. The + following memcpy() will move the data from the internal buffer + that was not yet on disk. + */ + memcpy(buffer + skipped_data, from + skipped_data, + TRANSLOG_PAGE_SIZE - skipped_data); /* We can use copy then in translog_page_validator() because it do not put it permanently somewhere. @@ -3291,6 +3402,7 @@ uint32 next_page_offset, page_rest; uint32 i; File fd; + int rc; TRANSLOG_VALIDATOR_DATA data; char path[FN_REFLEN]; uchar page_buff[TRANSLOG_PAGE_SIZE]; @@ -3316,14 +3428,19 @@ TRANSLOG_PAGE_SIZE); page_rest= next_page_offset - LSN_OFFSET(addr); memset(page_buff, TRANSLOG_FILLER, page_rest); - if ((fd= open_logfile_by_number_no_cache(LSN_FILE_NO(addr))) < 0 || - ((my_chsize(fd, next_page_offset, TRANSLOG_FILLER, MYF(MY_WME)) || - (page_rest && my_pwrite(fd, page_buff, page_rest, LSN_OFFSET(addr), - log_write_flags)) || - my_sync(fd, MYF(MY_WME))) | - my_close(fd, MYF(MY_WME))) || - (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && - sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)))) + rc= ((fd= open_logfile_by_number_no_cache(LSN_FILE_NO(addr))) < 0 || + ((my_chsize(fd, next_page_offset, TRANSLOG_FILLER, MYF(MY_WME)) || + (page_rest && my_pwrite(fd, page_buff, page_rest, LSN_OFFSET(addr), + log_write_flags)) || + my_sync(fd, MYF(MY_WME))))); + translog_syncs++; + rc|= (fd > 0 && my_close(fd, MYF(MY_WME))); + if (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS) + { + rc|= sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)); + translog_syncs++; + } + if (rc) DBUG_RETURN(1); /* fix the horizon */ @@ -3483,7 +3600,10 @@ my_bool version_changed= 0; DBUG_ENTER("translog_init_with_table"); + translog_syncs= 0; + flush_start= 0; id_to_share= NULL; + log_descriptor.directory_fd= -1; log_descriptor.is_everything_flushed= 1; log_descriptor.flush_in_progress= 0; @@ -3511,6 +3631,7 @@ pthread_mutex_init(&log_descriptor.dirty_buffer_mask_lock, MY_MUTEX_INIT_FAST) || pthread_cond_init(&log_descriptor.log_flush_cond, 0) || + pthread_cond_init(&log_descriptor.new_goal_cond, 0) || my_rwlock_init(&log_descriptor.open_files_lock, NULL) || my_init_dynamic_array(&log_descriptor.open_files, @@ -3912,7 +4033,6 @@ log_descriptor.flushed= log_descriptor.horizon; log_descriptor.in_buffers_only= log_descriptor.bc.buffer->offset; log_descriptor.max_lsn= LSN_IMPOSSIBLE; /* set to 0 */ - log_descriptor.previous_flush_horizon= log_descriptor.horizon; /* Now 'flushed' is set to 'horizon' value, but 'horizon' is (potentially) address of the next LSN and we want indicate that all LSNs that are @@ -3995,6 +4115,10 @@ It is beginning of the log => there is no LSNs in the log => There is no harm in leaving it "as-is". */ + log_descriptor.previous_flush_horizon= log_descriptor.horizon; + DBUG_PRINT("info", ("previous_flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor. + previous_flush_horizon))); DBUG_RETURN(0); } file_no--; @@ -4070,6 +4194,9 @@ translog_free_record_header(&rec); } } + log_descriptor.previous_flush_horizon= log_descriptor.horizon; + DBUG_PRINT("info", ("previous_flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor.previous_flush_horizon))); DBUG_RETURN(0); err: ma_message_no_user(0, "log initialization failed"); @@ -4157,6 +4284,7 @@ pthread_mutex_destroy(&log_descriptor.log_flush_lock); pthread_mutex_destroy(&log_descriptor.dirty_buffer_mask_lock); pthread_cond_destroy(&log_descriptor.log_flush_cond); + pthread_cond_destroy(&log_descriptor.new_goal_cond); rwlock_destroy(&log_descriptor.open_files_lock); delete_dynamic(&log_descriptor.open_files); delete_dynamic(&log_descriptor.unfinished_files); @@ -6885,11 +7013,11 @@ { translog_size_t res; DBUG_ENTER("translog_read_record_header_from_buffer"); - DBUG_ASSERT(translog_is_LSN_chunk(page[page_offset])); - DBUG_ASSERT(translog_status == TRANSLOG_OK || - translog_status == TRANSLOG_READONLY); DBUG_PRINT("info", ("page byte: 0x%x offset: %u", (uint) page[page_offset], (uint) page_offset)); + DBUG_ASSERT(translog_is_LSN_chunk(page[page_offset])); + DBUG_ASSERT(translog_status == TRANSLOG_OK || + translog_status == TRANSLOG_READONLY); buff->type= (page[page_offset] & TRANSLOG_REC_TYPE); buff->short_trid= uint2korr(page + page_offset + 1); DBUG_PRINT("info", ("Type %u, Short TrID %u, LSN (%lu,0x%lx)", @@ -7356,27 +7484,27 @@ "Buffer addr: (%lu,0x%lx) " "Page addr: (%lu,0x%lx) " "size: %lu (%lu) Pg: %u left: %u in progress %u", - (uint) log_descriptor.bc.buffer_no, - (ulong) log_descriptor.bc.buffer, - LSN_IN_PARTS(log_descriptor.bc.buffer->offset), + (uint) old_buffer_no, + (ulong) old_buffer, + LSN_IN_PARTS(old_buffer->offset), (ulong) LSN_FILE_NO(log_descriptor.horizon), (ulong) (LSN_OFFSET(log_descriptor.horizon) - log_descriptor.bc.current_page_fill), - (ulong) log_descriptor.bc.buffer->size, + (ulong) old_buffer->size, (ulong) (log_descriptor.bc.ptr -log_descriptor.bc. buffer->buffer), (uint) log_descriptor.bc.current_page_fill, (uint) left, - (uint) log_descriptor.bc.buffer-> + (uint) old_buffer-> copy_to_buffer_in_progress)); translog_lock_assert_owner(); LINT_INIT(current_page_fill); - new_buff_beginning= log_descriptor.bc.buffer->offset; - new_buff_beginning+= log_descriptor.bc.buffer->size; /* increase offset */ + new_buff_beginning= old_buffer->offset; + new_buff_beginning+= old_buffer->size; /* increase offset */ DBUG_ASSERT(log_descriptor.bc.ptr !=NULL); DBUG_ASSERT(LSN_FILE_NO(log_descriptor.horizon) == - LSN_FILE_NO(log_descriptor.bc.buffer->offset)); + LSN_FILE_NO(old_buffer->offset)); translog_check_cursor(&log_descriptor.bc); DBUG_ASSERT(left < TRANSLOG_PAGE_SIZE); if (left) @@ -7387,18 +7515,20 @@ */ DBUG_PRINT("info", ("left: %u", (uint) left)); + old_buffer->pre_force_close_horizon= + old_buffer->offset + old_buffer->size; /* decrease offset */ new_buff_beginning-= log_descriptor.bc.current_page_fill; current_page_fill= log_descriptor.bc.current_page_fill; memset(log_descriptor.bc.ptr, TRANSLOG_FILLER, left); - log_descriptor.bc.buffer->size+= left; + old_buffer->size+= left; DBUG_PRINT("info", ("Finish Page buffer #%u: 0x%lx " "Size: %lu", - (uint) log_descriptor.bc.buffer->buffer_no, - (ulong) log_descriptor.bc.buffer, - (ulong) log_descriptor.bc.buffer->size)); - DBUG_ASSERT(log_descriptor.bc.buffer->buffer_no == + (uint) old_buffer->buffer_no, + (ulong) old_buffer, + (ulong) old_buffer->size)); + DBUG_ASSERT(old_buffer->buffer_no == log_descriptor.bc.buffer_no); } else @@ -7509,11 +7639,21 @@ if (left) { - /* - TODO: do not copy beginning of the page if we have no CRC or sector - checks on - */ - memcpy(new_buffer->buffer, data, current_page_fill); + if (log_descriptor.flags & + (TRANSLOG_PAGE_CRC | TRANSLOG_SECTOR_PROTECTION)) + memcpy(new_buffer->buffer, data, current_page_fill); + else + { + /* + This page header does not change if we add more data to the page so + we can not copy it and will not overwrite later + */ + new_buffer->skipped_data= current_page_fill; +#ifndef DBUG_OFF + memset(new_buffer->buffer, 0xa5, current_page_fill); +#endif + DBUG_ASSERT(new_buffer->skipped_data < TRANSLOG_PAGE_SIZE); + } } old_buffer->next_buffer_offset= new_buffer->offset; translog_buffer_lock(new_buffer); @@ -7561,6 +7701,7 @@ { log_descriptor.next_pass_max_lsn= lsn; log_descriptor.max_lsn_requester= pthread_self(); + pthread_cond_broadcast(&log_descriptor.new_goal_cond); } while (flush_no == log_descriptor.flush_no) { @@ -7572,66 +7713,78 @@ /** - @brief Flush the log up to given LSN (included) - - @param lsn log record serial number up to which (inclusive) - the log has to be flushed - - @return Operation status + @brief sync() range of files (inclusive) and directory (by request) + + @param min min internal file number to flush + @param max max internal file number to flush + @param sync_dir need sync directory + + return Operation status @retval 0 OK @retval 1 Error - -*/ - -my_bool translog_flush(TRANSLOG_ADDRESS lsn) -{ - LSN sent_to_disk= LSN_IMPOSSIBLE; - TRANSLOG_ADDRESS flush_horizon; - uint fn, i; +*/ + +static my_bool translog_sync_files(uint32 min, uint32 max, + my_bool sync_dir) +{ + uint fn; + my_bool rc= 0; + ulonglong flush_interval; + DBUG_ENTER("translog_sync_files"); + DBUG_PRINT("info", ("min: %lu max: %lu sync dir: %d", + (ulong) min, (ulong) max, (int) sync_dir)); + DBUG_ASSERT(min <= max); + + flush_interval= group_commit_wait; + if (flush_interval) + flush_start= my_micro_time(); + for (fn= min; fn <= max; fn++) + { + TRANSLOG_FILE *file= get_logfile_by_number(fn); + DBUG_ASSERT(file != NULL); + if (!file->is_sync) + { + if (my_sync(file->handler.file, MYF(MY_WME))) + { + rc= 1; + translog_stop_writing(); + DBUG_RETURN(rc); + } + translog_syncs++; + file->is_sync= 1; + } + } + + if (sync_dir) + { + if (!(rc= sync_dir(log_descriptor.directory_fd, + MYF(MY_WME | MY_IGNORE_BADFD)))) + translog_syncs++; + } + + DBUG_RETURN(rc); +} + + +/* + @brief Flushes buffers with LSNs in them less or equal address <lsn> + + @param lsn address up to which all LSNs should be flushed, + can be reset to real last LSN address + @parem sent_to_disk returns 'sent to disk' position + @param flush_horizon returns horizon of the flush + + @note About terminology see comment to translog_flush(). +*/ + +void translog_flush_buffers(TRANSLOG_ADDRESS *lsn, + TRANSLOG_ADDRESS *sent_to_disk, + TRANSLOG_ADDRESS *flush_horizon) +{ dirty_buffer_mask_t dirty_buffer_mask; + uint i; uint8 last_buffer_no, start_buffer_no; - my_bool rc= 0; - DBUG_ENTER("translog_flush"); - DBUG_PRINT("enter", ("Flush up to LSN: (%lu,0x%lx)", LSN_IN_PARTS(lsn))); - DBUG_ASSERT(translog_status == TRANSLOG_OK || - translog_status == TRANSLOG_READONLY); - LINT_INIT(sent_to_disk); - - pthread_mutex_lock(&log_descriptor.log_flush_lock); - DBUG_PRINT("info", ("Everything is flushed up to (%lu,0x%lx)", - LSN_IN_PARTS(log_descriptor.flushed))); - if (cmp_translog_addr(log_descriptor.flushed, lsn) >= 0) - { - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - DBUG_RETURN(0); - } - if (log_descriptor.flush_in_progress) - { - translog_flush_set_new_goal_and_wait(lsn); - if (!pthread_equal(log_descriptor.max_lsn_requester, pthread_self())) - { - /* fix lsn if it was horizon */ - if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->last_lsn) > 0) - lsn= BUFFER_MAX_LSN(log_descriptor.bc.buffer); - translog_flush_wait_for_end(lsn); - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - DBUG_RETURN(0); - } - log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; - } - log_descriptor.flush_in_progress= 1; - flush_horizon= log_descriptor.previous_flush_horizon; - DBUG_PRINT("info", ("flush_in_progress is set")); - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - - translog_lock(); - if (log_descriptor.is_everything_flushed) - { - DBUG_PRINT("info", ("everything is flushed")); - rc= (translog_status == TRANSLOG_READONLY); - translog_unlock(); - goto out; - } + DBUG_ENTER("translog_flush_buffers"); /* We will recheck information when will lock buffers one by @@ -7656,15 +7809,15 @@ /* if LSN up to which we have to flush bigger then maximum LSN of previous buffer and at least one LSN was saved in the current buffer (last_lsn != - LSN_IMPOSSIBLE) then we better finish the current buffer. + LSN_IMPOSSIBLE) then we have to close the current buffer. */ - if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->prev_last_lsn) > 0 && + if (cmp_translog_addr(*lsn, log_descriptor.bc.buffer->prev_last_lsn) > 0 && log_descriptor.bc.buffer->last_lsn != LSN_IMPOSSIBLE) { struct st_translog_buffer *buffer= log_descriptor.bc.buffer; - lsn= log_descriptor.bc.buffer->last_lsn; /* fix lsn if it was horizon */ + *lsn= log_descriptor.bc.buffer->last_lsn; /* fix lsn if it was horizon */ DBUG_PRINT("info", ("LSN to flush fixed to last lsn: (%lu,0x%lx)", - LSN_IN_PARTS(log_descriptor.bc.buffer->last_lsn))); + LSN_IN_PARTS(log_descriptor.bc.buffer->last_lsn))); last_buffer_no= log_descriptor.bc.buffer_no; log_descriptor.is_everything_flushed= 1; translog_force_current_buffer_to_finish(); @@ -7676,8 +7829,10 @@ TRANSLOG_BUFFERS_NO); translog_unlock(); } - sent_to_disk= translog_get_sent_to_disk(); - if (cmp_translog_addr(lsn, sent_to_disk) > 0) + + /* flush buffers */ + *sent_to_disk= translog_get_sent_to_disk(); + if (cmp_translog_addr(*lsn, *sent_to_disk) > 0) { DBUG_PRINT("info", ("Start buffer #: %u last buffer #: %u", @@ -7697,53 +7852,238 @@ LSN_IN_PARTS(buffer->last_lsn), (buffer->file ? "dirty" : "closed"))); - if (buffer->prev_last_lsn <= lsn && + if (buffer->prev_last_lsn <= *lsn && buffer->file != NULL) { - DBUG_ASSERT(flush_horizon <= buffer->offset + buffer->size); - flush_horizon= buffer->offset + buffer->size; + DBUG_ASSERT(*flush_horizon <= buffer->offset + buffer->size); + *flush_horizon= (buffer->pre_force_close_horizon != LSN_IMPOSSIBLE ? + buffer->pre_force_close_horizon : + buffer->offset + buffer->size); + /* pre_force_close_horizon is reset during new buffer start */ + DBUG_PRINT("info", ("flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(*flush_horizon))); + DBUG_ASSERT(*flush_horizon <= log_descriptor.horizon); + translog_buffer_flush(buffer); } translog_buffer_unlock(buffer); i= (i + 1) % TRANSLOG_BUFFERS_NO; } while (i != last_buffer_no); - sent_to_disk= translog_get_sent_to_disk(); - } - - /* sync files from previous flush till current one */ - for (fn= LSN_FILE_NO(log_descriptor.flushed); fn <= LSN_FILE_NO(lsn); fn++) - { - TRANSLOG_FILE *file= get_logfile_by_number(fn); - DBUG_ASSERT(file != NULL); - if (!file->is_sync) - { - if (my_sync(file->handler.file, MYF(MY_WME))) + *sent_to_disk= translog_get_sent_to_disk(); + } + + DBUG_VOID_RETURN; +} + +/** + @brief Flush the log up to given LSN (included) + + @param lsn log record serial number up to which (inclusive) + the log has to be flushed + + @return Operation status + @retval 0 OK + @retval 1 Error + + @note + + - Non group commit logic: Commits made in passes. Thread which started + flush first is performing actual flush, other threads sets new goal (LSN) + of the next pass (if it is maximum) and waits for the pass end or just + wait for the pass end. + + - If hard group commit enabled and rate set to zero: + The first thread sends all changed buffers to disk. This is repeated + as long as there are new LSNs added. The process can not loop + forever because we have limited number of threads and they will wait + for the data to be synced. + Pseudo code: + + do + send changed buffers to disk + while new_goal + sync + + - If hard group commit switched ON and less than rate microseconds has + passed from last sync, then after buffers have been sent to disk + wait until rate microseconds has passed since last sync, do sync and return. + This ensures that if we call sync infrequently we don't do any waits. + + - If soft group commit enabled everything works as with 'non group commit' + but the thread doesn't do any real sync(). If rate is not zero the + sync() will be performed by a service thread with the given rate + when needed (new LSN appears). + + @note Terminology: + 'sent to disk' means written to disk but not sync()ed, + 'flushed' mean sent to disk and synced(). +*/ + +my_bool translog_flush(TRANSLOG_ADDRESS lsn) +{ + struct timespec abstime; + ulonglong flush_interval; + ulonglong time_spent; + LSN sent_to_disk= LSN_IMPOSSIBLE; + TRANSLOG_ADDRESS flush_horizon; + my_bool rc= 0; + my_bool hgroup_commit_at_start; + DBUG_ENTER("translog_flush"); + DBUG_PRINT("enter", ("Flush up to LSN: (%lu,0x%lx)", LSN_IN_PARTS(lsn))); + DBUG_ASSERT(translog_status == TRANSLOG_OK || + translog_status == TRANSLOG_READONLY); + LINT_INIT(sent_to_disk); + + pthread_mutex_lock(&log_descriptor.log_flush_lock); + DBUG_PRINT("info", ("Everything is flushed up to (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor.flushed))); + if (cmp_translog_addr(log_descriptor.flushed, lsn) >= 0) + { + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_RETURN(0); + } + if (log_descriptor.flush_in_progress) + { + translog_lock(); + /* fix lsn if it was horizon */ + if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->last_lsn) > 0) + lsn= BUFFER_MAX_LSN(log_descriptor.bc.buffer); + translog_unlock(); + translog_flush_set_new_goal_and_wait(lsn); + if (!pthread_equal(log_descriptor.max_lsn_requester, pthread_self())) + { + /* + translog_flush_wait_for_end() release log_flush_lock while is + waiting then acquire it again + */ + translog_flush_wait_for_end(lsn); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_RETURN(0); + } + log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; + } + log_descriptor.flush_in_progress= 1; + flush_horizon= log_descriptor.previous_flush_horizon; + DBUG_PRINT("info", ("flush_in_progress is set, flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(flush_horizon))); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + + hgroup_commit_at_start= hard_group_commit; + if (hgroup_commit_at_start) + flush_interval= group_commit_wait; + + translog_lock(); + if (log_descriptor.is_everything_flushed) + { + DBUG_PRINT("info", ("everything is flushed")); + translog_unlock(); + pthread_mutex_lock(&log_descriptor.log_flush_lock); + goto out; + } + + for (;;) + { + /* Following function flushes buffers and makes translog_unlock() */ + translog_flush_buffers(&lsn, &sent_to_disk, &flush_horizon); + + if (!hgroup_commit_at_start) + break; /* flush pass is ended */ + +retest: + /* + We do not check time here because pthread_mutex_lock rarely takes + a lot of time so we can sacrifice a bit precision to performance + (taking into account that my_micro_time() might be expensive call). + */ + if (flush_interval == 0) + break; /* flush pass is ended */ + + pthread_mutex_lock(&log_descriptor.log_flush_lock); + if (log_descriptor.next_pass_max_lsn == LSN_IMPOSSIBLE) + { + if (flush_interval == 0 || + (time_spent= (my_micro_time() - flush_start)) >= flush_interval) { - rc= 1; - translog_stop_writing(); - sent_to_disk= LSN_IMPOSSIBLE; - goto out; + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + break; } - file->is_sync= 1; - } - } - - if (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && - (LSN_FILE_NO(log_descriptor.previous_flush_horizon) != - LSN_FILE_NO(flush_horizon) || - ((LSN_OFFSET(log_descriptor.previous_flush_horizon) - 1) / - TRANSLOG_PAGE_SIZE) != - ((LSN_OFFSET(flush_horizon) - 1) / TRANSLOG_PAGE_SIZE))) - rc|= sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)); + DBUG_PRINT("info", ("flush waits: %llu interval: %llu spent: %llu", + flush_interval - time_spent, + flush_interval, time_spent)); + /* wait time or next goal */ + set_timespec_nsec(abstime, flush_interval - time_spent); + pthread_cond_timedwait(&log_descriptor.new_goal_cond, + &log_descriptor.log_flush_lock, + &abstime); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_PRINT("info", ("retest conditions")); + goto retest; + } + + /* take next goal */ + lsn= log_descriptor.next_pass_max_lsn; + log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; + /* prevent other thread from continue */ + log_descriptor.max_lsn_requester= pthread_self(); + DBUG_PRINT("info", ("flush took next goal: (%lu,0x%lx)", + LSN_IN_PARTS(lsn))); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + + /* next flush pass */ + DBUG_PRINT("info", ("next flush pass")); + translog_lock(); + } + + /* + sync() files from previous flush till current one + */ + if (!soft_sync || hgroup_commit_at_start) + { + if ((rc= + translog_sync_files(LSN_FILE_NO(log_descriptor.flushed), + LSN_FILE_NO(lsn), + sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && + (LSN_FILE_NO(log_descriptor. + previous_flush_horizon) != + LSN_FILE_NO(flush_horizon) || + (LSN_OFFSET(log_descriptor. + previous_flush_horizon) / + TRANSLOG_PAGE_SIZE) != + (LSN_OFFSET(flush_horizon) / + TRANSLOG_PAGE_SIZE))))) + { + sent_to_disk= LSN_IMPOSSIBLE; + pthread_mutex_lock(&log_descriptor.log_flush_lock); + goto out; + } + /* keep values for soft sync() and forced sync() actual */ + { + uint32 fileno= LSN_FILE_NO(lsn); + my_atomic_rwlock_wrlock(&soft_sync_rwl); + my_atomic_store32(&soft_sync_min, fileno); + my_atomic_store32(&soft_sync_max, fileno); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + } + } + else + { + my_atomic_rwlock_wrlock(&soft_sync_rwl); + my_atomic_store32(&soft_sync_max, LSN_FILE_NO(lsn)); + my_atomic_store32(&soft_need_sync, 1); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + } + + DBUG_ASSERT(flush_horizon <= log_descriptor.horizon); + + pthread_mutex_lock(&log_descriptor.log_flush_lock); log_descriptor.previous_flush_horizon= flush_horizon; out: - pthread_mutex_lock(&log_descriptor.log_flush_lock); if (sent_to_disk != LSN_IMPOSSIBLE) log_descriptor.flushed= sent_to_disk; log_descriptor.flush_in_progress= 0; log_descriptor.flush_no++; DBUG_PRINT("info", ("flush_in_progress is dropped")); - pthread_mutex_unlock(&log_descriptor.log_flush_lock);\ + pthread_mutex_unlock(&log_descriptor.log_flush_lock); pthread_cond_broadcast(&log_descriptor.log_flush_cond); DBUG_RETURN(rc); } @@ -8113,6 +8453,8 @@ my_bool translog_purge(TRANSLOG_ADDRESS low) { uint32 last_need_file= LSN_FILE_NO(low); + uint32 min_unsync; + int soft; TRANSLOG_ADDRESS horizon= translog_get_horizon(); int rc= 0; DBUG_ENTER("translog_purge"); @@ -8120,12 +8462,26 @@ DBUG_ASSERT(translog_status == TRANSLOG_OK || translog_status == TRANSLOG_READONLY); + soft= soft_sync; + my_atomic_rwlock_wrlock(&soft_sync_rwl); + min_unsync= my_atomic_load32(&soft_sync_min); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + DBUG_PRINT("info", ("min_unsync: %lu", (ulong) min_unsync)); + if (soft && min_unsync < last_need_file) + { + last_need_file= min_unsync; + DBUG_PRINT("info", ("last_need_file set to %lu", (ulong)last_need_file)); + } + pthread_mutex_lock(&log_descriptor.purger_lock); + DBUG_PRINT("info", ("last_lsn_checked file: %lu:", + (ulong) log_descriptor.last_lsn_checked)); if (LSN_FILE_NO(log_descriptor.last_lsn_checked) < last_need_file) { uint32 i; uint32 min_file= translog_first_file(horizon, 1); DBUG_ASSERT(min_file != 0); /* log is already started */ + DBUG_PRINT("info", ("min_file: %lu:",(ulong) min_file)); for(i= min_file; i < last_need_file && rc == 0; i++) { LSN lsn= translog_get_file_max_lsn_stored(i); @@ -8356,6 +8712,159 @@ } + +/** + Sets soft sync mode + + @param mode TRUE if we need switch soft sync on else off +*/ + +void translog_soft_sync(my_bool mode) +{ + soft_sync= mode; +} + + +/** + Sets hard group commit + + @param mode TRUE if we need switch hard group commit on else off +*/ + +void translog_hard_group_commit(my_bool mode) +{ + hard_group_commit= mode; +} + + +/** + @brief forced log sync (used when we are switching modes) +*/ + +void translog_sync() +{ + uint32 max= get_current_logfile()->number; + uint32 min; + DBUG_ENTER("ma_translog_sync"); + + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + if (!min) + min= max; + + translog_sync_files(min, max, sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS); + + DBUG_VOID_RETURN; +} + + +/** + @brief set rate for group commit + + @param interval interval to set. + + @note We use this function with additional variable because have to + restart service thread with new value which we can't make inside changing + variable routine (update_maria_group_commit_interval) +*/ + +void translog_set_group_commit_interval(uint32 interval) +{ + DBUG_ENTER("translog_set_group_commit_interval"); + group_commit_wait= interval; + DBUG_PRINT("info", ("wait: %llu", + (ulonglong)group_commit_wait)); + DBUG_VOID_RETURN; +} + + +/** + @brief syncing service thread +*/ + +static pthread_handler_t +ma_soft_sync_background( void *arg __attribute__((unused))) +{ + + my_thread_init(); + { + DBUG_ENTER("ma_soft_sync_background"); + for(;;) + { + ulonglong prev_loop= my_micro_time(); + ulonglong time, sleep; + uint32 min, max, sync_request; + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + max= my_atomic_load32(&soft_sync_max); + sync_request= my_atomic_load32(&soft_need_sync); + my_atomic_store32(&soft_sync_min, max); + my_atomic_store32(&soft_need_sync, 0); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + + sleep= group_commit_wait; + if (sync_request) + translog_sync_files(min, max, FALSE); + time= my_micro_time() - prev_loop; + if (time > sleep) + sleep= 0; + else + sleep-= time; + if (my_service_thread_sleep(&soft_sync_control, sleep)) + break; + } + my_service_thread_signal_end(&soft_sync_control); + my_thread_end(); + DBUG_RETURN(0); + } +} + + +/** + @brief Starts syncing thread +*/ + +int translog_soft_sync_start(void) +{ + pthread_t th; + int res= 0; + uint32 min, max; + DBUG_ENTER("translog_soft_sync_start"); + + /* check and init variables */ + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + max= my_atomic_load32(&soft_sync_max); + if (!max) + my_atomic_store32(&soft_sync_max, (max= get_current_logfile()->number)); + if (!min) + my_atomic_store32(&soft_sync_min, max); + my_atomic_store32(&soft_need_sync, 1); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + + if (!(res= ma_service_thread_control_init(&soft_sync_control))) + if (!(res= pthread_create(&th, NULL, ma_soft_sync_background, NULL))) + soft_sync_control.status= THREAD_RUNNING; + DBUG_RETURN(res); +} + + +/** + @brief Stops syncing thread +*/ + +void translog_soft_sync_end(void) +{ + DBUG_ENTER("translog_soft_sync_end"); + if (soft_sync_control.inited) + { + ma_service_thread_control_end(&soft_sync_control); + } + DBUG_VOID_RETURN; +} + + #ifdef MARIA_DUMP_LOG #include <my_getopt.h> extern void translog_example_table_init(); === modified file 'storage/maria/ma_loghandler.h' --- a/storage/maria/ma_loghandler.h 2009-01-15 22:25:53 +0000 +++ b/storage/maria/ma_loghandler.h 2010-02-12 13:12:28 +0000 @@ -342,6 +342,14 @@ TRANSLOG_SHUTDOWN /* going to shutdown the loghandler */ }; extern enum enum_translog_status translog_status; +extern ulonglong translog_syncs; /* Number of sync()s */ + +void translog_soft_sync(my_bool mode); +void translog_hard_group_commit(my_bool mode); +int translog_soft_sync_start(void); +void translog_soft_sync_end(void); +void translog_sync(); +void translog_set_group_commit_interval(uint32 interval); /* all the rest added because of recovery; should we make @@ -441,6 +449,14 @@ typedef enum { + TRANSLOG_GCOMMIT_NONE, + TRANSLOG_GCOMMIT_HARD, + TRANSLOG_GCOMMIT_SOFT +} enum_maria_group_commit; +extern ulong maria_group_commit; +extern ulong maria_group_commit_interval; +typedef enum +{ TRANSLOG_PURGE_IMMIDIATE, TRANSLOG_PURGE_EXTERNAL, TRANSLOG_PURGE_ONDEMAND

1 0

[Maria-developers] Rev 2740: Group commit for maria storage engine. in file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/
by sanja＠askmonty.org 12 Feb '10

12 Feb '10

At file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/ ------------------------------------------------------------ revno: 2740 revision-id: sanja(a)askmonty.org-20100212091325-sluwoeo04cvmjewk parent: knielsen(a)knielsen-hq.org-20100201190519-b9uktnn90rwwiile committer: sanja(a)askmonty.org branch nick: work-maria-5.2-groupcommit timestamp: Fri 2010-02-12 11:13:25 +0200 message: Group commit for maria storage engine. === added file 'mysql-test/suite/maria/r/group_commit.result' --- a/mysql-test/suite/maria/r/group_commit.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/maria/r/group_commit.result 2010-02-12 09:13:25 +0000 @@ -0,0 +1,17 @@ +drop table if exists t1; +create table t1 (a int); +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +drop table t1; === modified file 'mysql-test/suite/maria/r/maria3.result' --- a/mysql-test/suite/maria/r/maria3.result 2009-09-18 01:04:43 +0000 +++ b/mysql-test/suite/maria/r/maria3.result 2010-02-12 09:13:25 +0000 @@ -306,6 +306,8 @@ maria_block_size 8192 maria_checkpoint_interval 30 maria_force_start_after_recovery_failures 0 +maria_group_commit none +maria_group_commit_interval 0 maria_log_file_size 4294959104 maria_log_purge_type immediate maria_max_sort_file_size 9223372036853727232 @@ -328,6 +330,7 @@ Maria_pagecache_reads # Maria_pagecache_write_requests # Maria_pagecache_writes # +Maria_transaction_log_syncs # create table t1 (b char(0)); insert into t1 values(NULL),(""); select length(b) from t1; === added file 'mysql-test/suite/maria/t/group_commit.test' --- a/mysql-test/suite/maria/t/group_commit.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/maria/t/group_commit.test 2010-02-12 09:13:25 +0000 @@ -0,0 +1,71 @@ +# Test different ways of syncing (mostly syntax) + +--disable_warnings +drop table if exists t1; +--enable_warnings + +create table t1 (a int); + +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +drop table t1; === added directory 'randgen' === added directory 'randgen/conf' === added file 'randgen/conf/maria_group_commit.yy' --- a/randgen/conf/maria_group_commit.yy 1970-01-01 00:00:00 +0000 +++ b/randgen/conf/maria_group_commit.yy 2010-02-12 09:13:25 +0000 @@ -0,0 +1,181 @@ +# test of group commit switching + +query: + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + select | insert | update| delete | + change_group_commit | change_interval; + + +select: + SELECT select_item FROM join where order_by limit; + +select_item: + * | X . _field ; + +join: + _table AS X | + _table AS X LEFT JOIN _table AS Y ON ( X . _field = Y . _field ) ; + +where: + | + WHERE X . _field < value | + WHERE X . _field > value | + WHERE X . _field = value ; + +where_delete: + | + WHERE _field < value | + WHERE _field > value | + WHERE _field = value ; + +order_by: + | ORDER BY X . _field ; + +limit: + | LIMIT _digit ; + +insert: + INSERT INTO _table ( _field , _field ) VALUES ( value , value ) ; + +update: + UPDATE _table AS X SET _field = value where order_by limit ; + +delete: + DELETE FROM _table where_delete LIMIT _digit ; + +value: + ' _letter ' | _digit | _date | _datetime | _time | _english ; + +change_group_commit: + SET GLOBAL MARIA_GROUP_COMMIT=none_soft_hard; + +none_soft_hard: + NONE | SOFT | HARD; + +change_interval: + set_interval | set_interval | set_interval | set_interval | + drop_interval; + +set_interval: + SET GLOBAL MARIA_GROUP_COMMIT_INTERVAL=_tinyint_unsigned; + +drop_interval: + SET GLOBAL MARIA_GROUP_COMMIT_INTERVAL=0; === modified file 'storage/maria/ha_maria.cc' --- a/storage/maria/ha_maria.cc 2009-12-03 11:34:11 +0000 +++ b/storage/maria/ha_maria.cc 2010-02-12 09:13:25 +0000 @@ -102,22 +102,40 @@ array_elements(maria_translog_purge_type_names) - 1, "", maria_translog_purge_type_names, NULL }; + +/* transactional log directory sync */ const char *maria_sync_log_dir_names[]= { "NEVER", "NEWFILE", "ALWAYS", NullS }; - TYPELIB maria_sync_log_dir_typelib= { array_elements(maria_sync_log_dir_names) - 1, "", maria_sync_log_dir_names, NULL }; +/* transactional log group commit */ +const char *maria_group_commit_names[]= +{ + "none", "hard", "soft", NullS +}; +TYPELIB maria_group_commit_typelib= +{ + array_elements(maria_group_commit_names) - 1, "", + maria_group_commit_names, NULL +}; + /** Interval between background checkpoints in seconds */ static ulong checkpoint_interval; static void update_checkpoint_interval(MYSQL_THD thd, struct st_mysql_sys_var *var, void *var_ptr, const void *save); +static void update_maria_group_commit(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save); +static void update_maria_group_commit_interval(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save); /** After that many consecutive recovery failures, remove logs */ static ulong force_start_after_recovery_failures; static void update_log_file_size(MYSQL_THD thd, @@ -164,6 +182,24 @@ NULL, update_log_file_size, TRANSLOG_FILE_SIZE, TRANSLOG_MIN_FILE_SIZE, 0xffffffffL, TRANSLOG_PAGE_SIZE); +static MYSQL_SYSVAR_ENUM(group_commit, maria_group_commit, + PLUGIN_VAR_RQCMDARG, + "Specifies maria group commit mode. " + "Possible values are \"none\" (no group commit), " + "\"hard\" (with waiting to actual commit), " + "\"soft\" (no wait for commit (DANGEROUS!!!))", + NULL, update_maria_group_commit, + TRANSLOG_GCOMMIT_NONE, &maria_group_commit_typelib); + +static MYSQL_SYSVAR_ULONG(group_commit_interval, maria_group_commit_interval, + PLUGIN_VAR_RQCMDARG, + "Interval between commite in microseconds (1/1000000c)." + " 0 stands for no waiting" + " for other threads to come and do a commit in \"hard\" mode and no" + " sync()/commit at all in \"soft\" mode. Option has only an effect" + " if maria_group_commit is used", + NULL, update_maria_group_commit_interval, 0, 0, UINT_MAX, 1); + static MYSQL_SYSVAR_ENUM(log_purge_type, log_purge_type, PLUGIN_VAR_RQCMDARG, "Specifies how maria transactional log will be purged. " @@ -3275,6 +3311,8 @@ MYSQL_SYSVAR(block_size), MYSQL_SYSVAR(checkpoint_interval), MYSQL_SYSVAR(force_start_after_recovery_failures), + MYSQL_SYSVAR(group_commit), + MYSQL_SYSVAR(group_commit_interval), MYSQL_SYSVAR(page_checksum), MYSQL_SYSVAR(log_dir_path), MYSQL_SYSVAR(log_file_size), @@ -3306,6 +3344,92 @@ } /** + @brief Updates group commit mode +*/ + +static void update_maria_group_commit(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save) +{ + ulong value= (ulong)*((long *)var_ptr); + DBUG_ENTER("update_maria_group_commit"); + DBUG_PRINT("enter", ("old value: %lu new value %lu rate %lu", + value, (ulong)(*(long *)save), + maria_group_commit_interval)); + /* old value */ + switch (value) { + case TRANSLOG_GCOMMIT_NONE: + break; + case TRANSLOG_GCOMMIT_HARD: + translog_hard_group_commit(FALSE); + break; + case TRANSLOG_GCOMMIT_SOFT: + translog_soft_sync(FALSE); + if (maria_group_commit_interval) + translog_soft_sync_end(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + value= *(ulong *)var_ptr= (ulong)(*(long *)save); + translog_sync(); + /* new value */ + switch (value) { + case TRANSLOG_GCOMMIT_NONE: + break; + case TRANSLOG_GCOMMIT_HARD: + translog_hard_group_commit(TRUE); + break; + case TRANSLOG_GCOMMIT_SOFT: + translog_soft_sync(TRUE); + /* variable change made under global lock so we can just read it */ + if (maria_group_commit_interval) + translog_soft_sync_start(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + DBUG_VOID_RETURN; +} + +/** + @brief Updates group commit interval +*/ + +static void update_maria_group_commit_interval(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save) +{ + ulong new_value= (ulong)*((long *)save); + ulong *value_ptr= (ulong*) var_ptr; + DBUG_ENTER("update_maria_group_commit_interval"); + DBUG_PRINT("enter", ("old value: %lu new value %lu group commit %lu", + *value_ptr, new_value, maria_group_commit)); + + /* variable change made under global lock so we can just read it */ + switch (maria_group_commit) { + case TRANSLOG_GCOMMIT_NONE: + *value_ptr= new_value; + translog_set_group_commit_interval(new_value); + break; + case TRANSLOG_GCOMMIT_HARD: + *value_ptr= new_value; + translog_set_group_commit_interval(new_value); + break; + case TRANSLOG_GCOMMIT_SOFT: + if (*value_ptr) + translog_soft_sync_end(); + translog_set_group_commit_interval(new_value); + if ((*value_ptr= new_value)) + translog_soft_sync_start(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + DBUG_VOID_RETURN; +} + +/** @brief Updates the transaction log file limit. */ @@ -3327,6 +3451,7 @@ {"Maria_pagecache_reads", (char*) &maria_pagecache_var.global_cache_read, SHOW_LONGLONG}, {"Maria_pagecache_write_requests", (char*) &maria_pagecache_var.global_cache_w_requests, SHOW_LONGLONG}, {"Maria_pagecache_writes", (char*) &maria_pagecache_var.global_cache_write, SHOW_LONGLONG}, + {"Maria_transaction_log_syncs", (char*) &translog_syncs, SHOW_LONGLONG}, {NullS, NullS, SHOW_LONG} }; === modified file 'storage/maria/ma_init.c' --- a/storage/maria/ma_init.c 2008-10-09 20:03:54 +0000 +++ b/storage/maria/ma_init.c 2010-02-12 09:13:25 +0000 @@ -82,6 +82,11 @@ maria_inited= maria_multi_threaded= FALSE; ft_free_stopwords(); ma_checkpoint_end(); + if (translog_status == TRANSLOG_OK) + { + translog_soft_sync_end(); + translog_sync(); + } if ((trid= trnman_get_max_trid()) > max_trid_in_control_file) { /* === modified file 'storage/maria/ma_loghandler.c' --- a/storage/maria/ma_loghandler.c 2010-01-06 21:27:53 +0000 +++ b/storage/maria/ma_loghandler.c 2010-02-12 09:13:25 +0000 @@ -18,6 +18,7 @@ #include "ma_blockrec.h" /* for some constants and in-write hooks */ #include "ma_key_recover.h" /* For some in-write hooks */ #include "ma_checkpoint.h" +#include "ma_servicethread.h" /* On Windows, neither my_open() nor my_sync() work for directories. @@ -47,6 +48,15 @@ #include <m_ctype.h> #endif +/** @brief protects checkpoint_in_progress */ +static pthread_mutex_t LOCK_soft_sync; +/** @brief for killing the background checkpoint thread */ +static pthread_cond_t COND_soft_sync; +/** @brief control structure for checkpoint background thread */ +static MA_SERVICE_THREAD_CONTROL soft_sync_control= + {THREAD_DEAD, FALSE, &LOCK_soft_sync, &COND_soft_sync}; + + /* transaction log file descriptor */ typedef struct st_translog_file { @@ -124,10 +134,24 @@ /* Previous buffer offset to detect it flush finish */ TRANSLOG_ADDRESS prev_buffer_offset; /* + If the buffer was forced to close it save value of its horizon + otherwise LSN_IMPOSSIBLE + */ + TRANSLOG_ADDRESS pre_force_close_horizon; + /* How much is written (or will be written when copy_to_buffer_in_progress become 0) to this buffer */ translog_size_t size; + /* + When moving from one log buffer to another, we write the last of the + previous buffer to file and then move to start using the new log + buffer. In the case of a part filed last page, this page is not moved + to the start of the new buffer but instead we set the 'skip_data' + variable to tell us how much data at the beginning of the buffer is not + relevant. + */ + uint skipped_data; /* File handler for this buffer */ TRANSLOG_FILE *file; /* Threads which are waiting for buffer filling/freeing */ @@ -304,6 +328,7 @@ */ pthread_mutex_t log_flush_lock; pthread_cond_t log_flush_cond; + pthread_cond_t new_goal_cond; /* Protects changing of headers of finished files (max_lsn) */ pthread_mutex_t file_header_lock; @@ -344,13 +369,39 @@ ulong log_purge_type= TRANSLOG_PURGE_IMMIDIATE; ulong log_file_size= TRANSLOG_FILE_SIZE; +/* sync() of log files directory mode */ ulong sync_log_dir= TRANSLOG_SYNC_DIR_NEWFILE; +ulong maria_group_commit= TRANSLOG_GCOMMIT_NONE; +ulong maria_group_commit_interval= 0; /* Marker for end of log */ static uchar end_of_log= 0; #define END_OF_LOG &end_of_log +/** + Switch for "soft" sync (no real sync() but periodical sync by service + thread) +*/ +static volatile my_bool soft_sync= FALSE; +/** + Switch for "hard" group commit mode +*/ +static volatile my_bool hard_group_commit= FALSE; +/** + File numbers interval which have to be sync() +*/ +static uint32 soft_sync_min= 0; +static uint32 soft_sync_max= 0; +static uint32 soft_need_sync= 1; +/** + stores interval in microseconds +*/ +static uint32 group_commit_wait= 0; enum enum_translog_status translog_status= TRANSLOG_UNINITED; +ulonglong translog_syncs= 0; /* Number of sync()s */ + +/* time of last flush */ +static ulonglong flush_start= 0; /* chunk types */ #define TRANSLOG_CHUNK_LSN 0x00 /* 0 chunk refer as LSN (head or tail */ @@ -980,12 +1031,17 @@ static TRANSLOG_FILE *get_current_logfile() { TRANSLOG_FILE *file; + DBUG_ENTER("get_current_logfile"); rw_rdlock(&log_descriptor.open_files_lock); + DBUG_PRINT("info", ("max_file: %lu min_file: %lu open_files: %lu", + (ulong) log_descriptor.max_file, + (ulong) log_descriptor.min_file, + (ulong) log_descriptor.open_files.elements)); DBUG_ASSERT(log_descriptor.max_file - log_descriptor.min_file + 1 == log_descriptor.open_files.elements); file= *dynamic_element(&log_descriptor.open_files, 0, TRANSLOG_FILE **); rw_unlock(&log_descriptor.open_files_lock); - return (file); + DBUG_RETURN(file); } uchar NEAR maria_trans_file_magic[]= @@ -1069,6 +1125,7 @@ static my_bool translog_max_lsn_to_header(File file, LSN lsn) { uchar lsn_buff[LSN_STORE_SIZE]; + my_bool rc; DBUG_ENTER("translog_max_lsn_to_header"); DBUG_PRINT("enter", ("File descriptor: %ld " "lsn: (%lu,0x%lx)", @@ -1077,11 +1134,17 @@ lsn_store(lsn_buff, lsn); - DBUG_RETURN(my_pwrite(file, lsn_buff, - LSN_STORE_SIZE, - (LOG_HEADER_DATA_SIZE - LSN_STORE_SIZE), - log_write_flags) != 0 || - my_sync(file, MYF(MY_WME)) != 0); + rc= (my_pwrite(file, lsn_buff, + LSN_STORE_SIZE, + (LOG_HEADER_DATA_SIZE - LSN_STORE_SIZE), + log_write_flags) != 0 || + my_sync(file, MYF(MY_WME)) != 0); + /* + We should not increase counter in case of error above, but it is so + unlikely that we can ignore this case + */ + translog_syncs++; + DBUG_RETURN(rc); } @@ -1423,7 +1486,9 @@ static my_bool translog_buffer_init(struct st_translog_buffer *buffer, int num) { DBUG_ENTER("translog_buffer_init"); - buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; + buffer->pre_force_close_horizon= + buffer->prev_last_lsn= buffer->last_lsn= + LSN_IMPOSSIBLE; DBUG_PRINT("info", ("last_lsn and prev_last_lsn set to 0 buffer: 0x%lx", (ulong) buffer)); @@ -1435,6 +1500,7 @@ memset(buffer->buffer, TRANSLOG_FILLER, TRANSLOG_WRITE_BUFFER); /* Buffer size */ buffer->size= 0; + buffer->skipped_data= 0; /* cond of thread which is waiting for buffer filling */ if (pthread_cond_init(&buffer->waiting_filling_buffer, 0)) DBUG_RETURN(1); @@ -1489,7 +1555,10 @@ TODO: sync only we have changed the log */ if (!file->is_sync) + { rc= my_sync(file->handler.file, MYF(MY_WME)); + translog_syncs++; + } rc|= my_close(file->handler.file, MYF(MY_WME)); my_free(file, MYF(0)); return test(rc); @@ -2044,7 +2113,8 @@ (ulong) LSN_OFFSET(log_descriptor.horizon), (ulong) LSN_OFFSET(log_descriptor.horizon))); DBUG_ASSERT(buffer_no == buffer->buffer_no); - buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; + buffer->pre_force_close_horizon= + buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; DBUG_PRINT("info", ("last_lsn and prev_last_lsn set to 0 buffer: 0x%lx", (ulong) buffer)); buffer->offset= log_descriptor.horizon; @@ -2052,6 +2122,7 @@ buffer->file= get_current_logfile(); buffer->overlay= 0; buffer->size= 0; + buffer->skipped_data= 0; translog_cursor_init(cursor, buffer, buffer_no); DBUG_PRINT("info", ("file: #%ld (%d) init cursor #%u: 0x%lx " "chaser: %d Size: %lu (%lu)", @@ -2523,6 +2594,7 @@ TRANSLOG_ADDRESS offset= buffer->offset; TRANSLOG_FILE *file= buffer->file; uint8 ver= buffer->ver; + uint skipped_data; DBUG_ENTER("translog_buffer_flush"); DBUG_PRINT("enter", ("Buffer: #%u 0x%lx file: %d offset: (%lu,0x%lx) size: %lu", @@ -2557,6 +2629,8 @@ disk */ file= buffer->file; + skipped_data= buffer->skipped_data; + DBUG_ASSERT(skipped_data < TRANSLOG_PAGE_SIZE); for (i= 0, pg= LSN_OFFSET(buffer->offset) / TRANSLOG_PAGE_SIZE; i < buffer->size; i+= TRANSLOG_PAGE_SIZE, pg++) @@ -2573,13 +2647,16 @@ DBUG_ASSERT(i + TRANSLOG_PAGE_SIZE <= buffer->size); if (translog_status != TRANSLOG_OK && translog_status != TRANSLOG_SHUTDOWN) DBUG_RETURN(1); - if (pagecache_inject(log_descriptor.pagecache, + if (pagecache_write_part(log_descriptor.pagecache, &file->handler, pg, 3, buffer->buffer + i, PAGECACHE_PLAIN_PAGE, PAGECACHE_LOCK_LEFT_UNLOCKED, - PAGECACHE_PIN_LEFT_UNPINNED, 0, - LSN_IMPOSSIBLE)) + PAGECACHE_PIN_LEFT_UNPINNED, + PAGECACHE_WRITE_DONE, 0, + LSN_IMPOSSIBLE, + skipped_data, + TRANSLOG_PAGE_SIZE - skipped_data)) { DBUG_PRINT("error", ("Can't write page (%lu,0x%lx) to pagecache, error: %d", @@ -2589,10 +2666,12 @@ translog_stop_writing(); DBUG_RETURN(1); } + skipped_data= 0; } file->is_sync= 0; - if (my_pwrite(file->handler.file, buffer->buffer, - buffer->size, LSN_OFFSET(buffer->offset), + if (my_pwrite(file->handler.file, buffer->buffer + buffer->skipped_data, + buffer->size - buffer->skipped_data, + LSN_OFFSET(buffer->offset) + buffer->skipped_data, log_write_flags)) { DBUG_PRINT("error", ("Can't write buffer (%lu,0x%lx) size %lu " @@ -2985,6 +3064,7 @@ uchar *from, *table= NULL; int is_last_unfinished_page; uint last_protected_sector= 0; + uint skipped_data= curr_buffer->skipped_data; TRANSLOG_FILE file_copy; uint8 ver= curr_buffer->ver; translog_wait_for_writers(curr_buffer); @@ -2997,7 +3077,38 @@ } DBUG_ASSERT(LSN_FILE_NO(addr) == LSN_FILE_NO(curr_buffer->offset)); from= curr_buffer->buffer + (addr - curr_buffer->offset); - memcpy(buffer, from, TRANSLOG_PAGE_SIZE); + if (skipped_data && addr == curr_buffer->offset) + { + /* + We read page part of which is not present in buffer, + so we should read absent part from file (page cache actually) + */ + file= get_logfile_by_number(file_no); + DBUG_ASSERT(file != NULL); + /* + it's ok to not lock the page because: + - The log handler has it's own page cache. + - There is only one thread that can access the log + cache at a time + */ + if (!(buffer= pagecache_read(log_descriptor.pagecache, + &file->handler, + LSN_OFFSET(addr) / TRANSLOG_PAGE_SIZE, + 3, buffer, + PAGECACHE_PLAIN_PAGE, + PAGECACHE_LOCK_LEFT_UNLOCKED, + NULL))) + DBUG_RETURN(NULL); + } + else + skipped_data= 0; /* Read after skipped in buffer data */ + /* + Now we have correct data in buffer up to 'skipped_data'. The + following memcpy() will move the data from the internal buffer + that was not yet on disk. + */ + memcpy(buffer + skipped_data, from + skipped_data, + TRANSLOG_PAGE_SIZE - skipped_data); /* We can use copy then in translog_page_validator() because it do not put it permanently somewhere. @@ -3291,6 +3402,7 @@ uint32 next_page_offset, page_rest; uint32 i; File fd; + int rc; TRANSLOG_VALIDATOR_DATA data; char path[FN_REFLEN]; uchar page_buff[TRANSLOG_PAGE_SIZE]; @@ -3316,14 +3428,19 @@ TRANSLOG_PAGE_SIZE); page_rest= next_page_offset - LSN_OFFSET(addr); memset(page_buff, TRANSLOG_FILLER, page_rest); - if ((fd= open_logfile_by_number_no_cache(LSN_FILE_NO(addr))) < 0 || - ((my_chsize(fd, next_page_offset, TRANSLOG_FILLER, MYF(MY_WME)) || - (page_rest && my_pwrite(fd, page_buff, page_rest, LSN_OFFSET(addr), - log_write_flags)) || - my_sync(fd, MYF(MY_WME))) | - my_close(fd, MYF(MY_WME))) || - (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && - sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)))) + rc= ((fd= open_logfile_by_number_no_cache(LSN_FILE_NO(addr))) < 0 || + ((my_chsize(fd, next_page_offset, TRANSLOG_FILLER, MYF(MY_WME)) || + (page_rest && my_pwrite(fd, page_buff, page_rest, LSN_OFFSET(addr), + log_write_flags)) || + my_sync(fd, MYF(MY_WME))))); + translog_syncs++; + rc|= (fd > 0 && my_close(fd, MYF(MY_WME))); + if (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS) + { + rc|= sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)); + translog_syncs++; + } + if (rc) DBUG_RETURN(1); /* fix the horizon */ @@ -3483,7 +3600,10 @@ my_bool version_changed= 0; DBUG_ENTER("translog_init_with_table"); + translog_syncs= 0; + flush_start= 0; id_to_share= NULL; + log_descriptor.directory_fd= -1; log_descriptor.is_everything_flushed= 1; log_descriptor.flush_in_progress= 0; @@ -3511,6 +3631,7 @@ pthread_mutex_init(&log_descriptor.dirty_buffer_mask_lock, MY_MUTEX_INIT_FAST) || pthread_cond_init(&log_descriptor.log_flush_cond, 0) || + pthread_cond_init(&log_descriptor.new_goal_cond, 0) || my_rwlock_init(&log_descriptor.open_files_lock, NULL) || my_init_dynamic_array(&log_descriptor.open_files, @@ -3912,7 +4033,6 @@ log_descriptor.flushed= log_descriptor.horizon; log_descriptor.in_buffers_only= log_descriptor.bc.buffer->offset; log_descriptor.max_lsn= LSN_IMPOSSIBLE; /* set to 0 */ - log_descriptor.previous_flush_horizon= log_descriptor.horizon; /* Now 'flushed' is set to 'horizon' value, but 'horizon' is (potentially) address of the next LSN and we want indicate that all LSNs that are @@ -3995,6 +4115,10 @@ It is beginning of the log => there is no LSNs in the log => There is no harm in leaving it "as-is". */ + log_descriptor.previous_flush_horizon= log_descriptor.horizon; + DBUG_PRINT("info", ("previous_flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor. + previous_flush_horizon))); DBUG_RETURN(0); } file_no--; @@ -4070,6 +4194,9 @@ translog_free_record_header(&rec); } } + log_descriptor.previous_flush_horizon= log_descriptor.horizon; + DBUG_PRINT("info", ("previous_flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor.previous_flush_horizon))); DBUG_RETURN(0); err: ma_message_no_user(0, "log initialization failed"); @@ -4157,6 +4284,7 @@ pthread_mutex_destroy(&log_descriptor.log_flush_lock); pthread_mutex_destroy(&log_descriptor.dirty_buffer_mask_lock); pthread_cond_destroy(&log_descriptor.log_flush_cond); + pthread_cond_destroy(&log_descriptor.new_goal_cond); rwlock_destroy(&log_descriptor.open_files_lock); delete_dynamic(&log_descriptor.open_files); delete_dynamic(&log_descriptor.unfinished_files); @@ -6885,11 +7013,11 @@ { translog_size_t res; DBUG_ENTER("translog_read_record_header_from_buffer"); - DBUG_ASSERT(translog_is_LSN_chunk(page[page_offset])); - DBUG_ASSERT(translog_status == TRANSLOG_OK || - translog_status == TRANSLOG_READONLY); DBUG_PRINT("info", ("page byte: 0x%x offset: %u", (uint) page[page_offset], (uint) page_offset)); + DBUG_ASSERT(translog_is_LSN_chunk(page[page_offset])); + DBUG_ASSERT(translog_status == TRANSLOG_OK || + translog_status == TRANSLOG_READONLY); buff->type= (page[page_offset] & TRANSLOG_REC_TYPE); buff->short_trid= uint2korr(page + page_offset + 1); DBUG_PRINT("info", ("Type %u, Short TrID %u, LSN (%lu,0x%lx)", @@ -7356,27 +7484,27 @@ "Buffer addr: (%lu,0x%lx) " "Page addr: (%lu,0x%lx) " "size: %lu (%lu) Pg: %u left: %u in progress %u", - (uint) log_descriptor.bc.buffer_no, - (ulong) log_descriptor.bc.buffer, - LSN_IN_PARTS(log_descriptor.bc.buffer->offset), + (uint) old_buffer_no, + (ulong) old_buffer, + LSN_IN_PARTS(old_buffer->offset), (ulong) LSN_FILE_NO(log_descriptor.horizon), (ulong) (LSN_OFFSET(log_descriptor.horizon) - log_descriptor.bc.current_page_fill), - (ulong) log_descriptor.bc.buffer->size, + (ulong) old_buffer->size, (ulong) (log_descriptor.bc.ptr -log_descriptor.bc. buffer->buffer), (uint) log_descriptor.bc.current_page_fill, (uint) left, - (uint) log_descriptor.bc.buffer-> + (uint) old_buffer-> copy_to_buffer_in_progress)); translog_lock_assert_owner(); LINT_INIT(current_page_fill); - new_buff_beginning= log_descriptor.bc.buffer->offset; - new_buff_beginning+= log_descriptor.bc.buffer->size; /* increase offset */ + new_buff_beginning= old_buffer->offset; + new_buff_beginning+= old_buffer->size; /* increase offset */ DBUG_ASSERT(log_descriptor.bc.ptr !=NULL); DBUG_ASSERT(LSN_FILE_NO(log_descriptor.horizon) == - LSN_FILE_NO(log_descriptor.bc.buffer->offset)); + LSN_FILE_NO(old_buffer->offset)); translog_check_cursor(&log_descriptor.bc); DBUG_ASSERT(left < TRANSLOG_PAGE_SIZE); if (left) @@ -7387,18 +7515,20 @@ */ DBUG_PRINT("info", ("left: %u", (uint) left)); + old_buffer->pre_force_close_horizon= + old_buffer->offset + old_buffer->size; /* decrease offset */ new_buff_beginning-= log_descriptor.bc.current_page_fill; current_page_fill= log_descriptor.bc.current_page_fill; memset(log_descriptor.bc.ptr, TRANSLOG_FILLER, left); - log_descriptor.bc.buffer->size+= left; + old_buffer->size+= left; DBUG_PRINT("info", ("Finish Page buffer #%u: 0x%lx " "Size: %lu", - (uint) log_descriptor.bc.buffer->buffer_no, - (ulong) log_descriptor.bc.buffer, - (ulong) log_descriptor.bc.buffer->size)); - DBUG_ASSERT(log_descriptor.bc.buffer->buffer_no == + (uint) old_buffer->buffer_no, + (ulong) old_buffer, + (ulong) old_buffer->size)); + DBUG_ASSERT(old_buffer->buffer_no == log_descriptor.bc.buffer_no); } else @@ -7509,11 +7639,21 @@ if (left) { - /* - TODO: do not copy beginning of the page if we have no CRC or sector - checks on - */ - memcpy(new_buffer->buffer, data, current_page_fill); + if (log_descriptor.flags & + (TRANSLOG_PAGE_CRC | TRANSLOG_SECTOR_PROTECTION)) + memcpy(new_buffer->buffer, data, current_page_fill); + else + { + /* + This page header does not change if we add more data to the page so + we can not copy it and will not overwrite later + */ + new_buffer->skipped_data= current_page_fill; +#ifndef DBUG_OFF + memset(new_buffer->buffer, 0xa5, current_page_fill); +#endif + DBUG_ASSERT(new_buffer->skipped_data < TRANSLOG_PAGE_SIZE); + } } old_buffer->next_buffer_offset= new_buffer->offset; translog_buffer_lock(new_buffer); @@ -7561,6 +7701,7 @@ { log_descriptor.next_pass_max_lsn= lsn; log_descriptor.max_lsn_requester= pthread_self(); + pthread_cond_broadcast(&log_descriptor.new_goal_cond); } while (flush_no == log_descriptor.flush_no) { @@ -7572,66 +7713,78 @@ /** - @brief Flush the log up to given LSN (included) - - @param lsn log record serial number up to which (inclusive) - the log has to be flushed - - @return Operation status + @brief sync() range of files (inclusive) and directory (by request) + + @param min min internal file number to flush + @param max max internal file number to flush + @param sync_dir need sync directory + + return Operation status @retval 0 OK @retval 1 Error - -*/ - -my_bool translog_flush(TRANSLOG_ADDRESS lsn) -{ - LSN sent_to_disk= LSN_IMPOSSIBLE; - TRANSLOG_ADDRESS flush_horizon; - uint fn, i; +*/ + +static my_bool translog_sync_files(uint32 min, uint32 max, + my_bool sync_dir) +{ + uint fn; + my_bool rc= 0; + ulonglong flush_interval; + DBUG_ENTER("translog_sync_files"); + DBUG_PRINT("info", ("min: %lu max: %lu sync dir: %d", + (ulong) min, (ulong) max, (int) sync_dir)); + DBUG_ASSERT(min <= max); + + flush_interval= group_commit_wait; + if (flush_interval) + flush_start= my_micro_time(); + for (fn= min; fn <= max; fn++) + { + TRANSLOG_FILE *file= get_logfile_by_number(fn); + DBUG_ASSERT(file != NULL); + if (!file->is_sync) + { + if (my_sync(file->handler.file, MYF(MY_WME))) + { + rc= 1; + translog_stop_writing(); + DBUG_RETURN(rc); + } + translog_syncs++; + file->is_sync= 1; + } + } + + if (sync_dir) + { + if (!(rc= sync_dir(log_descriptor.directory_fd, + MYF(MY_WME | MY_IGNORE_BADFD)))) + translog_syncs++; + } + + DBUG_RETURN(rc); +} + + +/* + @brief Flushes buffers with LSNs in them less or equal address <lsn> + + @param lsn address up to which all LSNs should be flushed, + can be reset to real last LSN address + @parem sent_to_disk returns 'sent to disk' position + @param flush_horizon returns horizon of the flush + + @note About terminology see comment to translog_flush(). +*/ + +void translog_flush_buffers(TRANSLOG_ADDRESS *lsn, + TRANSLOG_ADDRESS *sent_to_disk, + TRANSLOG_ADDRESS *flush_horizon) +{ dirty_buffer_mask_t dirty_buffer_mask; + uint i; uint8 last_buffer_no, start_buffer_no; - my_bool rc= 0; - DBUG_ENTER("translog_flush"); - DBUG_PRINT("enter", ("Flush up to LSN: (%lu,0x%lx)", LSN_IN_PARTS(lsn))); - DBUG_ASSERT(translog_status == TRANSLOG_OK || - translog_status == TRANSLOG_READONLY); - LINT_INIT(sent_to_disk); - - pthread_mutex_lock(&log_descriptor.log_flush_lock); - DBUG_PRINT("info", ("Everything is flushed up to (%lu,0x%lx)", - LSN_IN_PARTS(log_descriptor.flushed))); - if (cmp_translog_addr(log_descriptor.flushed, lsn) >= 0) - { - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - DBUG_RETURN(0); - } - if (log_descriptor.flush_in_progress) - { - translog_flush_set_new_goal_and_wait(lsn); - if (!pthread_equal(log_descriptor.max_lsn_requester, pthread_self())) - { - /* fix lsn if it was horizon */ - if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->last_lsn) > 0) - lsn= BUFFER_MAX_LSN(log_descriptor.bc.buffer); - translog_flush_wait_for_end(lsn); - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - DBUG_RETURN(0); - } - log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; - } - log_descriptor.flush_in_progress= 1; - flush_horizon= log_descriptor.previous_flush_horizon; - DBUG_PRINT("info", ("flush_in_progress is set")); - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - - translog_lock(); - if (log_descriptor.is_everything_flushed) - { - DBUG_PRINT("info", ("everything is flushed")); - rc= (translog_status == TRANSLOG_READONLY); - translog_unlock(); - goto out; - } + DBUG_ENTER("translog_flush_buffers"); /* We will recheck information when will lock buffers one by @@ -7656,15 +7809,15 @@ /* if LSN up to which we have to flush bigger then maximum LSN of previous buffer and at least one LSN was saved in the current buffer (last_lsn != - LSN_IMPOSSIBLE) then we better finish the current buffer. + LSN_IMPOSSIBLE) then we have to close the current buffer. */ - if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->prev_last_lsn) > 0 && + if (cmp_translog_addr(*lsn, log_descriptor.bc.buffer->prev_last_lsn) > 0 && log_descriptor.bc.buffer->last_lsn != LSN_IMPOSSIBLE) { struct st_translog_buffer *buffer= log_descriptor.bc.buffer; - lsn= log_descriptor.bc.buffer->last_lsn; /* fix lsn if it was horizon */ + *lsn= log_descriptor.bc.buffer->last_lsn; /* fix lsn if it was horizon */ DBUG_PRINT("info", ("LSN to flush fixed to last lsn: (%lu,0x%lx)", - LSN_IN_PARTS(log_descriptor.bc.buffer->last_lsn))); + LSN_IN_PARTS(log_descriptor.bc.buffer->last_lsn))); last_buffer_no= log_descriptor.bc.buffer_no; log_descriptor.is_everything_flushed= 1; translog_force_current_buffer_to_finish(); @@ -7676,8 +7829,10 @@ TRANSLOG_BUFFERS_NO); translog_unlock(); } - sent_to_disk= translog_get_sent_to_disk(); - if (cmp_translog_addr(lsn, sent_to_disk) > 0) + + /* flush buffers */ + *sent_to_disk= translog_get_sent_to_disk(); + if (cmp_translog_addr(*lsn, *sent_to_disk) > 0) { DBUG_PRINT("info", ("Start buffer #: %u last buffer #: %u", @@ -7697,53 +7852,238 @@ LSN_IN_PARTS(buffer->last_lsn), (buffer->file ? "dirty" : "closed"))); - if (buffer->prev_last_lsn <= lsn && + if (buffer->prev_last_lsn <= *lsn && buffer->file != NULL) { - DBUG_ASSERT(flush_horizon <= buffer->offset + buffer->size); - flush_horizon= buffer->offset + buffer->size; + DBUG_ASSERT(*flush_horizon <= buffer->offset + buffer->size); + *flush_horizon= (buffer->pre_force_close_horizon != LSN_IMPOSSIBLE ? + buffer->pre_force_close_horizon : + buffer->offset + buffer->size); + /* pre_force_close_horizon is reset during new buffer start */ + DBUG_PRINT("info", ("flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(*flush_horizon))); + DBUG_ASSERT(*flush_horizon <= log_descriptor.horizon); + translog_buffer_flush(buffer); } translog_buffer_unlock(buffer); i= (i + 1) % TRANSLOG_BUFFERS_NO; } while (i != last_buffer_no); - sent_to_disk= translog_get_sent_to_disk(); - } - - /* sync files from previous flush till current one */ - for (fn= LSN_FILE_NO(log_descriptor.flushed); fn <= LSN_FILE_NO(lsn); fn++) - { - TRANSLOG_FILE *file= get_logfile_by_number(fn); - DBUG_ASSERT(file != NULL); - if (!file->is_sync) - { - if (my_sync(file->handler.file, MYF(MY_WME))) + *sent_to_disk= translog_get_sent_to_disk(); + } + + DBUG_VOID_RETURN; +} + +/** + @brief Flush the log up to given LSN (included) + + @param lsn log record serial number up to which (inclusive) + the log has to be flushed + + @return Operation status + @retval 0 OK + @retval 1 Error + + @note + + - Non group commit logic: Commits made in passes. Thread which started + flush first is performing actual flush, other threads sets new goal (LSN) + of the next pass (if it is maximum) and waits for the pass end or just + wait for the pass end. + + - If hard group commit enabled and rate set to zero: + The first thread sends all changed buffers to disk. This is repeated + as long as there are new LSNs added. The process can not loop + forever because we have limited number of threads and they will wait + for the data to be synced. + Pseudo code: + + do + send changed buffers to disk + while new_goal + sync + + - If hard group commit switched ON and less than rate microseconds has + passed from last sync, then after buffers have been sent to disk + wait until rate microseconds has passed since last sync, do sync and return. + This ensures that if we call sync infrequently we don't do any waits. + + - If soft group commit enabled everything works as with 'non group commit' + but the thread doesn't do any real sync(). If rate is not zero the + sync() will be performed by a service thread with the given rate + when needed (new LSN appears). + + @note Terminology: + 'sent to disk' means written to disk but not sync()ed, + 'flushed' mean sent to disk and synced(). +*/ + +my_bool translog_flush(TRANSLOG_ADDRESS lsn) +{ + struct timespec abstime; + ulonglong flush_interval; + ulonglong time_spent; + LSN sent_to_disk= LSN_IMPOSSIBLE; + TRANSLOG_ADDRESS flush_horizon; + my_bool rc= 0; + my_bool hgroup_commit_at_start; + DBUG_ENTER("translog_flush"); + DBUG_PRINT("enter", ("Flush up to LSN: (%lu,0x%lx)", LSN_IN_PARTS(lsn))); + DBUG_ASSERT(translog_status == TRANSLOG_OK || + translog_status == TRANSLOG_READONLY); + LINT_INIT(sent_to_disk); + + pthread_mutex_lock(&log_descriptor.log_flush_lock); + DBUG_PRINT("info", ("Everything is flushed up to (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor.flushed))); + if (cmp_translog_addr(log_descriptor.flushed, lsn) >= 0) + { + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_RETURN(0); + } + if (log_descriptor.flush_in_progress) + { + translog_lock(); + /* fix lsn if it was horizon */ + if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->last_lsn) > 0) + lsn= BUFFER_MAX_LSN(log_descriptor.bc.buffer); + translog_unlock(); + translog_flush_set_new_goal_and_wait(lsn); + if (!pthread_equal(log_descriptor.max_lsn_requester, pthread_self())) + { + /* + translog_flush_wait_for_end() release log_flush_lock while is + waiting then acquire it again + */ + translog_flush_wait_for_end(lsn); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_RETURN(0); + } + log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; + } + log_descriptor.flush_in_progress= 1; + flush_horizon= log_descriptor.previous_flush_horizon; + DBUG_PRINT("info", ("flush_in_progress is set, flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(flush_horizon))); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + + hgroup_commit_at_start= hard_group_commit; + if (hgroup_commit_at_start) + flush_interval= group_commit_wait; + + translog_lock(); + if (log_descriptor.is_everything_flushed) + { + DBUG_PRINT("info", ("everything is flushed")); + translog_unlock(); + pthread_mutex_lock(&log_descriptor.log_flush_lock); + goto out; + } + + for (;;) + { + /* Following function flushes buffers and makes translog_unlock() */ + translog_flush_buffers(&lsn, &sent_to_disk, &flush_horizon); + + if (!hgroup_commit_at_start) + break; /* flush pass is ended */ + +retest: + /* + We do not check time here because pthread_mutex_lock rarely takes + a lot of time so we can sacrifice a bit precision to performance + (taking into account that my_micro_time() might be expensive call). + */ + if (flush_interval == 0) + break; /* flush pass is ended */ + + pthread_mutex_lock(&log_descriptor.log_flush_lock); + if (log_descriptor.next_pass_max_lsn == LSN_IMPOSSIBLE) + { + if (flush_interval == 0 || + (time_spent= (my_micro_time() - flush_start)) >= flush_interval) { - rc= 1; - translog_stop_writing(); - sent_to_disk= LSN_IMPOSSIBLE; - goto out; + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + break; } - file->is_sync= 1; - } - } - - if (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && - (LSN_FILE_NO(log_descriptor.previous_flush_horizon) != - LSN_FILE_NO(flush_horizon) || - ((LSN_OFFSET(log_descriptor.previous_flush_horizon) - 1) / - TRANSLOG_PAGE_SIZE) != - ((LSN_OFFSET(flush_horizon) - 1) / TRANSLOG_PAGE_SIZE))) - rc|= sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)); + DBUG_PRINT("info", ("flush waits: %llu interval: %llu spent: %llu", + flush_interval - time_spent, + flush_interval, time_spent)); + /* wait time or next goal */ + set_timespec_nsec(abstime, flush_interval - time_spent); + pthread_cond_timedwait(&log_descriptor.new_goal_cond, + &log_descriptor.log_flush_lock, + &abstime); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_PRINT("info", ("retest conditions")); + goto retest; + } + + /* take next goal */ + lsn= log_descriptor.next_pass_max_lsn; + log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; + /* prevent other thread from continue */ + log_descriptor.max_lsn_requester= pthread_self(); + DBUG_PRINT("info", ("flush took next goal: (%lu,0x%lx)", + LSN_IN_PARTS(lsn))); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + + /* next flush pass */ + DBUG_PRINT("info", ("next flush pass")); + translog_lock(); + } + + /* + sync() files from previous flush till current one + */ + if (!soft_sync || hgroup_commit_at_start) + { + if ((rc= + translog_sync_files(LSN_FILE_NO(log_descriptor.flushed), + LSN_FILE_NO(lsn), + sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && + (LSN_FILE_NO(log_descriptor. + previous_flush_horizon) != + LSN_FILE_NO(flush_horizon) || + (LSN_OFFSET(log_descriptor. + previous_flush_horizon) / + TRANSLOG_PAGE_SIZE) != + (LSN_OFFSET(flush_horizon) / + TRANSLOG_PAGE_SIZE))))) + { + sent_to_disk= LSN_IMPOSSIBLE; + pthread_mutex_lock(&log_descriptor.log_flush_lock); + goto out; + } + /* keep values for soft sync() and forced sync() actual */ + { + uint32 fileno= LSN_FILE_NO(lsn); + my_atomic_rwlock_wrlock(&soft_sync_rwl); + my_atomic_store32(&soft_sync_min, fileno); + my_atomic_store32(&soft_sync_max, fileno); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + } + } + else + { + my_atomic_rwlock_wrlock(&soft_sync_rwl); + my_atomic_store32(&soft_sync_max, LSN_FILE_NO(lsn)); + my_atomic_store32(&soft_need_sync, 1); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + } + + DBUG_ASSERT(flush_horizon <= log_descriptor.horizon); + + pthread_mutex_lock(&log_descriptor.log_flush_lock); log_descriptor.previous_flush_horizon= flush_horizon; out: - pthread_mutex_lock(&log_descriptor.log_flush_lock); if (sent_to_disk != LSN_IMPOSSIBLE) log_descriptor.flushed= sent_to_disk; log_descriptor.flush_in_progress= 0; log_descriptor.flush_no++; DBUG_PRINT("info", ("flush_in_progress is dropped")); - pthread_mutex_unlock(&log_descriptor.log_flush_lock);\ + pthread_mutex_unlock(&log_descriptor.log_flush_lock); pthread_cond_broadcast(&log_descriptor.log_flush_cond); DBUG_RETURN(rc); } @@ -8113,6 +8453,8 @@ my_bool translog_purge(TRANSLOG_ADDRESS low) { uint32 last_need_file= LSN_FILE_NO(low); + uint32 min_unsync; + int soft; TRANSLOG_ADDRESS horizon= translog_get_horizon(); int rc= 0; DBUG_ENTER("translog_purge"); @@ -8120,12 +8462,26 @@ DBUG_ASSERT(translog_status == TRANSLOG_OK || translog_status == TRANSLOG_READONLY); + soft= soft_sync; + my_atomic_rwlock_wrlock(&soft_sync_rwl); + min_unsync= my_atomic_load32(&soft_sync_min); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + DBUG_PRINT("info", ("min_unsync: %lu", (ulong) min_unsync)); + if (soft && min_unsync < last_need_file) + { + last_need_file= min_unsync; + DBUG_PRINT("info", ("last_need_file set to %lu", (ulong)last_need_file)); + } + pthread_mutex_lock(&log_descriptor.purger_lock); + DBUG_PRINT("info", ("last_lsn_checked file: %lu:", + (ulong) log_descriptor.last_lsn_checked)); if (LSN_FILE_NO(log_descriptor.last_lsn_checked) < last_need_file) { uint32 i; uint32 min_file= translog_first_file(horizon, 1); DBUG_ASSERT(min_file != 0); /* log is already started */ + DBUG_PRINT("info", ("min_file: %lu:",(ulong) min_file)); for(i= min_file; i < last_need_file && rc == 0; i++) { LSN lsn= translog_get_file_max_lsn_stored(i); @@ -8356,6 +8712,159 @@ } + +/** + Sets soft sync mode + + @param mode TRUE if we need switch soft sync on else off +*/ + +void translog_soft_sync(my_bool mode) +{ + soft_sync= mode; +} + + +/** + Sets hard group commit + + @param mode TRUE if we need switch hard group commit on else off +*/ + +void translog_hard_group_commit(my_bool mode) +{ + hard_group_commit= mode; +} + + +/** + @brief forced log sync (used when we are switching modes) +*/ + +void translog_sync() +{ + uint32 max= get_current_logfile()->number; + uint32 min; + DBUG_ENTER("ma_translog_sync"); + + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + if (!min) + min= max; + + translog_sync_files(min, max, sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS); + + DBUG_VOID_RETURN; +} + + +/** + @brief set rate for group commit + + @param interval interval to set. + + @note We use this function with additional variable because have to + restart service thread with new value which we can't make inside changing + variable routine (update_maria_group_commit_interval) +*/ + +void translog_set_group_commit_interval(uint32 interval) +{ + DBUG_ENTER("translog_set_group_commit_interval"); + group_commit_wait= interval; + DBUG_PRINT("info", ("wait: %llu", + (ulonglong)group_commit_wait)); + DBUG_VOID_RETURN; +} + + +/** + @brief syncing service thread +*/ + +static pthread_handler_t +ma_soft_sync_background( void *arg __attribute__((unused))) +{ + + my_thread_init(); + { + DBUG_ENTER("ma_soft_sync_background"); + for(;;) + { + ulonglong prev_loop= my_micro_time(); + ulonglong time, sleep; + uint32 min, max, sync_request; + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + max= my_atomic_load32(&soft_sync_max); + sync_request= my_atomic_load32(&soft_need_sync); + my_atomic_store32(&soft_sync_min, max); + my_atomic_store32(&soft_need_sync, 0); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + + sleep= group_commit_wait; + if (sync_request) + translog_sync_files(min, max, FALSE); + time= my_micro_time() - prev_loop; + if (time > sleep) + sleep= 0; + else + sleep-= time; + if (my_service_thread_sleep(&soft_sync_control, sleep)) + break; + } + my_service_thread_signal_end(&soft_sync_control); + my_thread_end(); + DBUG_RETURN(0); + } +} + + +/** + @brief Starts syncing thread +*/ + +int translog_soft_sync_start(void) +{ + pthread_t th; + int res= 0; + uint32 min, max; + DBUG_ENTER("translog_soft_sync_start"); + + /* check and init variables */ + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + max= my_atomic_load32(&soft_sync_max); + if (!max) + my_atomic_store32(&soft_sync_max, (max= get_current_logfile()->number)); + if (!min) + my_atomic_store32(&soft_sync_min, max); + my_atomic_store32(&soft_need_sync, 1); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + + if (!(res= ma_service_thread_control_init(&soft_sync_control))) + if (!(res= pthread_create(&th, NULL, ma_soft_sync_background, NULL))) + soft_sync_control.status= THREAD_RUNNING; + DBUG_RETURN(res); +} + + +/** + @brief Stops syncing thread +*/ + +void translog_soft_sync_end(void) +{ + DBUG_ENTER("translog_soft_sync_end"); + if (soft_sync_control.inited) + { + ma_service_thread_control_end(&soft_sync_control); + } + DBUG_VOID_RETURN; +} + + #ifdef MARIA_DUMP_LOG #include <my_getopt.h> extern void translog_example_table_init(); === modified file 'storage/maria/ma_loghandler.h' --- a/storage/maria/ma_loghandler.h 2009-01-15 22:25:53 +0000 +++ b/storage/maria/ma_loghandler.h 2010-02-12 09:13:25 +0000 @@ -342,6 +342,14 @@ TRANSLOG_SHUTDOWN /* going to shutdown the loghandler */ }; extern enum enum_translog_status translog_status; +extern ulonglong translog_syncs; /* Number of sync()s */ + +void translog_soft_sync(my_bool mode); +void translog_hard_group_commit(my_bool mode); +int translog_soft_sync_start(void); +void translog_soft_sync_end(void); +void translog_sync(); +void translog_set_group_commit_interval(uint32 interval); /* all the rest added because of recovery; should we make @@ -441,6 +449,14 @@ typedef enum { + TRANSLOG_GCOMMIT_NONE, + TRANSLOG_GCOMMIT_HARD, + TRANSLOG_GCOMMIT_SOFT +} enum_maria_group_commit; +extern ulong maria_group_commit; +extern ulong maria_group_commit_interval; +typedef enum +{ TRANSLOG_PURGE_IMMIDIATE, TRANSLOG_PURGE_EXTERNAL, TRANSLOG_PURGE_ONDEMAND

1 0

[Maria-developers] Rev 2740: Group commit for maria storage engine. in file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/
by sanja＠askmonty.org 12 Feb '10

12 Feb '10

At file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/ ------------------------------------------------------------ revno: 2740 revision-id: sanja(a)askmonty.org-20100212065247-vnhehxm6snm32c1j parent: knielsen(a)knielsen-hq.org-20100201190519-b9uktnn90rwwiile committer: sanja(a)askmonty.org branch nick: work-maria-5.2-groupcommit timestamp: Fri 2010-02-12 08:52:47 +0200 message: Group commit for maria storage engine. === added file 'mysql-test/suite/maria/r/group_commit.result' --- a/mysql-test/suite/maria/r/group_commit.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/maria/r/group_commit.result 2010-02-12 06:52:47 +0000 @@ -0,0 +1,17 @@ +drop table if exists t1; +create table t1 (a int); +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +drop table t1; === modified file 'mysql-test/suite/maria/r/maria3.result' --- a/mysql-test/suite/maria/r/maria3.result 2009-09-18 01:04:43 +0000 +++ b/mysql-test/suite/maria/r/maria3.result 2010-02-12 06:52:47 +0000 @@ -306,6 +306,8 @@ maria_block_size 8192 maria_checkpoint_interval 30 maria_force_start_after_recovery_failures 0 +maria_group_commit none +maria_group_commit_interval 0 maria_log_file_size 4294959104 maria_log_purge_type immediate maria_max_sort_file_size 9223372036853727232 @@ -328,6 +330,7 @@ Maria_pagecache_reads # Maria_pagecache_write_requests # Maria_pagecache_writes # +Maria_transaction_log_syncs # create table t1 (b char(0)); insert into t1 values(NULL),(""); select length(b) from t1; === added file 'mysql-test/suite/maria/t/group_commit.test' --- a/mysql-test/suite/maria/t/group_commit.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/maria/t/group_commit.test 2010-02-12 06:52:47 +0000 @@ -0,0 +1,71 @@ +# Test different ways of syncing (mostly syntax) + +--disable_warnings +drop table if exists t1; +--enable_warnings + +create table t1 (a int); + +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +drop table t1; === modified file 'storage/maria/ha_maria.cc' --- a/storage/maria/ha_maria.cc 2009-12-03 11:34:11 +0000 +++ b/storage/maria/ha_maria.cc 2010-02-12 06:52:47 +0000 @@ -102,22 +102,40 @@ array_elements(maria_translog_purge_type_names) - 1, "", maria_translog_purge_type_names, NULL }; + +/* transactional log directory sync */ const char *maria_sync_log_dir_names[]= { "NEVER", "NEWFILE", "ALWAYS", NullS }; - TYPELIB maria_sync_log_dir_typelib= { array_elements(maria_sync_log_dir_names) - 1, "", maria_sync_log_dir_names, NULL }; +/* transactional log group commit */ +const char *maria_group_commit_names[]= +{ + "none", "hard", "soft", NullS +}; +TYPELIB maria_group_commit_typelib= +{ + array_elements(maria_group_commit_names) - 1, "", + maria_group_commit_names, NULL +}; + /** Interval between background checkpoints in seconds */ static ulong checkpoint_interval; static void update_checkpoint_interval(MYSQL_THD thd, struct st_mysql_sys_var *var, void *var_ptr, const void *save); +static void update_maria_group_commit(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save); +static void update_maria_group_commit_interval(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save); /** After that many consecutive recovery failures, remove logs */ static ulong force_start_after_recovery_failures; static void update_log_file_size(MYSQL_THD thd, @@ -164,6 +182,24 @@ NULL, update_log_file_size, TRANSLOG_FILE_SIZE, TRANSLOG_MIN_FILE_SIZE, 0xffffffffL, TRANSLOG_PAGE_SIZE); +static MYSQL_SYSVAR_ENUM(group_commit, maria_group_commit, + PLUGIN_VAR_RQCMDARG, + "Specifies maria group commit mode. " + "Possible values are \"none\" (no group commit), " + "\"hard\" (with waiting to actual commit), " + "\"soft\" (no wait for commit (DANGEROUS!!!))", + NULL, update_maria_group_commit, + TRANSLOG_GCOMMIT_NONE, &maria_group_commit_typelib); + +static MYSQL_SYSVAR_ULONG(group_commit_interval, maria_group_commit_interval, + PLUGIN_VAR_RQCMDARG, + "Interval between commite in microseconds (1/1000000c)." + " 0 stands for no waiting" + " for other threads to come and do a commit in \"hard\" mode and no" + " sync()/commit at all in \"soft\" mode. Option has only an effect" + " if maria_group_commit is used", + NULL, update_maria_group_commit_interval, 0, 0, UINT_MAX, 1); + static MYSQL_SYSVAR_ENUM(log_purge_type, log_purge_type, PLUGIN_VAR_RQCMDARG, "Specifies how maria transactional log will be purged. " @@ -3275,6 +3311,8 @@ MYSQL_SYSVAR(block_size), MYSQL_SYSVAR(checkpoint_interval), MYSQL_SYSVAR(force_start_after_recovery_failures), + MYSQL_SYSVAR(group_commit), + MYSQL_SYSVAR(group_commit_interval), MYSQL_SYSVAR(page_checksum), MYSQL_SYSVAR(log_dir_path), MYSQL_SYSVAR(log_file_size), @@ -3306,6 +3344,92 @@ } /** + @brief Updates group commit mode +*/ + +static void update_maria_group_commit(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save) +{ + ulong value= (ulong)*((long *)var_ptr); + DBUG_ENTER("update_maria_group_commit"); + DBUG_PRINT("enter", ("old value: %lu new value %lu rate %lu", + value, (ulong)(*(long *)save), + maria_group_commit_interval)); + /* old value */ + switch (value) { + case TRANSLOG_GCOMMIT_NONE: + break; + case TRANSLOG_GCOMMIT_HARD: + translog_hard_group_commit(FALSE); + break; + case TRANSLOG_GCOMMIT_SOFT: + translog_soft_sync(FALSE); + if (maria_group_commit_interval) + translog_soft_sync_end(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + value= *(ulong *)var_ptr= (ulong)(*(long *)save); + translog_sync(); + /* new value */ + switch (value) { + case TRANSLOG_GCOMMIT_NONE: + break; + case TRANSLOG_GCOMMIT_HARD: + translog_hard_group_commit(TRUE); + break; + case TRANSLOG_GCOMMIT_SOFT: + translog_soft_sync(TRUE); + /* variable change made under global lock so we can just read it */ + if (maria_group_commit_interval) + translog_soft_sync_start(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + DBUG_VOID_RETURN; +} + +/** + @brief Updates group commit interval +*/ + +static void update_maria_group_commit_interval(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save) +{ + ulong new_value= (ulong)*((long *)save); + ulong *value_ptr= (ulong*) var_ptr; + DBUG_ENTER("update_maria_group_commit_interval"); + DBUG_PRINT("enter", ("old value: %lu new value %lu group commit %lu", + *value_ptr, new_value, maria_group_commit)); + + /* variable change made under global lock so we can just read it */ + switch (maria_group_commit) { + case TRANSLOG_GCOMMIT_NONE: + *value_ptr= new_value; + translog_set_group_commit_interval(new_value); + break; + case TRANSLOG_GCOMMIT_HARD: + *value_ptr= new_value; + translog_set_group_commit_interval(new_value); + break; + case TRANSLOG_GCOMMIT_SOFT: + if (*value_ptr) + translog_soft_sync_end(); + translog_set_group_commit_interval(new_value); + if ((*value_ptr= new_value)) + translog_soft_sync_start(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + DBUG_VOID_RETURN; +} + +/** @brief Updates the transaction log file limit. */ @@ -3327,6 +3451,7 @@ {"Maria_pagecache_reads", (char*) &maria_pagecache_var.global_cache_read, SHOW_LONGLONG}, {"Maria_pagecache_write_requests", (char*) &maria_pagecache_var.global_cache_w_requests, SHOW_LONGLONG}, {"Maria_pagecache_writes", (char*) &maria_pagecache_var.global_cache_write, SHOW_LONGLONG}, + {"Maria_transaction_log_syncs", (char*) &translog_syncs, SHOW_LONGLONG}, {NullS, NullS, SHOW_LONG} }; === modified file 'storage/maria/ma_init.c' --- a/storage/maria/ma_init.c 2008-10-09 20:03:54 +0000 +++ b/storage/maria/ma_init.c 2010-02-12 06:52:47 +0000 @@ -82,6 +82,11 @@ maria_inited= maria_multi_threaded= FALSE; ft_free_stopwords(); ma_checkpoint_end(); + if (translog_status == TRANSLOG_OK) + { + translog_soft_sync_end(); + translog_sync(); + } if ((trid= trnman_get_max_trid()) > max_trid_in_control_file) { /* === modified file 'storage/maria/ma_loghandler.c' --- a/storage/maria/ma_loghandler.c 2010-01-06 21:27:53 +0000 +++ b/storage/maria/ma_loghandler.c 2010-02-12 06:52:47 +0000 @@ -18,6 +18,7 @@ #include "ma_blockrec.h" /* for some constants and in-write hooks */ #include "ma_key_recover.h" /* For some in-write hooks */ #include "ma_checkpoint.h" +#include "ma_servicethread.h" /* On Windows, neither my_open() nor my_sync() work for directories. @@ -47,6 +48,15 @@ #include <m_ctype.h> #endif +/** @brief protects checkpoint_in_progress */ +static pthread_mutex_t LOCK_soft_sync; +/** @brief for killing the background checkpoint thread */ +static pthread_cond_t COND_soft_sync; +/** @brief control structure for checkpoint background thread */ +static MA_SERVICE_THREAD_CONTROL soft_sync_control= + {THREAD_DEAD, FALSE, &LOCK_soft_sync, &COND_soft_sync}; + + /* transaction log file descriptor */ typedef struct st_translog_file { @@ -124,10 +134,24 @@ /* Previous buffer offset to detect it flush finish */ TRANSLOG_ADDRESS prev_buffer_offset; /* + If the buffer was forced to close it save value of its horizon + otherwise LSN_IMPOSSIBLE + */ + TRANSLOG_ADDRESS pre_force_close_horizon; + /* How much is written (or will be written when copy_to_buffer_in_progress become 0) to this buffer */ translog_size_t size; + /* + When moving from one log buffer to another, we write the last of the + previous buffer to file and then move to start using the new log + buffer. In the case of a part filed last page, this page is not moved + to the start of the new buffer but instead we set the 'skip_data' + variable to tell us how much data at the beginning of the buffer is not + relevant. + */ + uint skipped_data; /* File handler for this buffer */ TRANSLOG_FILE *file; /* Threads which are waiting for buffer filling/freeing */ @@ -304,6 +328,7 @@ */ pthread_mutex_t log_flush_lock; pthread_cond_t log_flush_cond; + pthread_cond_t new_goal_cond; /* Protects changing of headers of finished files (max_lsn) */ pthread_mutex_t file_header_lock; @@ -344,13 +369,39 @@ ulong log_purge_type= TRANSLOG_PURGE_IMMIDIATE; ulong log_file_size= TRANSLOG_FILE_SIZE; +/* sync() of log files directory mode */ ulong sync_log_dir= TRANSLOG_SYNC_DIR_NEWFILE; +ulong maria_group_commit= TRANSLOG_GCOMMIT_NONE; +ulong maria_group_commit_interval= 0; /* Marker for end of log */ static uchar end_of_log= 0; #define END_OF_LOG &end_of_log +/** + Switch for "soft" sync (no real sync() but periodical sync by service + thread) +*/ +static volatile my_bool soft_sync= FALSE; +/** + Switch for "hard" group commit mode +*/ +static volatile my_bool hard_group_commit= FALSE; +/** + File numbers interval which have to be sync() +*/ +static uint32 soft_sync_min= 0; +static uint32 soft_sync_max= 0; +static uint32 soft_need_sync= 1; +/** + stores interval in microseconds +*/ +static uint32 group_commit_wait= 0; enum enum_translog_status translog_status= TRANSLOG_UNINITED; +ulonglong translog_syncs= 0; /* Number of sync()s */ + +/* time of last flush */ +static ulonglong flush_start= 0; /* chunk types */ #define TRANSLOG_CHUNK_LSN 0x00 /* 0 chunk refer as LSN (head or tail */ @@ -980,12 +1031,17 @@ static TRANSLOG_FILE *get_current_logfile() { TRANSLOG_FILE *file; + DBUG_ENTER("get_current_logfile"); rw_rdlock(&log_descriptor.open_files_lock); + DBUG_PRINT("info", ("max_file: %lu min_file: %lu open_files: %lu", + (ulong) log_descriptor.max_file, + (ulong) log_descriptor.min_file, + (ulong) log_descriptor.open_files.elements)); DBUG_ASSERT(log_descriptor.max_file - log_descriptor.min_file + 1 == log_descriptor.open_files.elements); file= *dynamic_element(&log_descriptor.open_files, 0, TRANSLOG_FILE **); rw_unlock(&log_descriptor.open_files_lock); - return (file); + DBUG_RETURN(file); } uchar NEAR maria_trans_file_magic[]= @@ -1069,6 +1125,7 @@ static my_bool translog_max_lsn_to_header(File file, LSN lsn) { uchar lsn_buff[LSN_STORE_SIZE]; + my_bool rc; DBUG_ENTER("translog_max_lsn_to_header"); DBUG_PRINT("enter", ("File descriptor: %ld " "lsn: (%lu,0x%lx)", @@ -1077,11 +1134,17 @@ lsn_store(lsn_buff, lsn); - DBUG_RETURN(my_pwrite(file, lsn_buff, - LSN_STORE_SIZE, - (LOG_HEADER_DATA_SIZE - LSN_STORE_SIZE), - log_write_flags) != 0 || - my_sync(file, MYF(MY_WME)) != 0); + rc= (my_pwrite(file, lsn_buff, + LSN_STORE_SIZE, + (LOG_HEADER_DATA_SIZE - LSN_STORE_SIZE), + log_write_flags) != 0 || + my_sync(file, MYF(MY_WME)) != 0); + /* + We should not increase counter in case of error above, but it is so + unlikely that we can ignore this case + */ + translog_syncs++; + DBUG_RETURN(rc); } @@ -1423,7 +1486,9 @@ static my_bool translog_buffer_init(struct st_translog_buffer *buffer, int num) { DBUG_ENTER("translog_buffer_init"); - buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; + buffer->pre_force_close_horizon= + buffer->prev_last_lsn= buffer->last_lsn= + LSN_IMPOSSIBLE; DBUG_PRINT("info", ("last_lsn and prev_last_lsn set to 0 buffer: 0x%lx", (ulong) buffer)); @@ -1435,6 +1500,7 @@ memset(buffer->buffer, TRANSLOG_FILLER, TRANSLOG_WRITE_BUFFER); /* Buffer size */ buffer->size= 0; + buffer->skipped_data= 0; /* cond of thread which is waiting for buffer filling */ if (pthread_cond_init(&buffer->waiting_filling_buffer, 0)) DBUG_RETURN(1); @@ -1489,7 +1555,10 @@ TODO: sync only we have changed the log */ if (!file->is_sync) + { rc= my_sync(file->handler.file, MYF(MY_WME)); + translog_syncs++; + } rc|= my_close(file->handler.file, MYF(MY_WME)); my_free(file, MYF(0)); return test(rc); @@ -2044,7 +2113,8 @@ (ulong) LSN_OFFSET(log_descriptor.horizon), (ulong) LSN_OFFSET(log_descriptor.horizon))); DBUG_ASSERT(buffer_no == buffer->buffer_no); - buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; + buffer->pre_force_close_horizon= + buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; DBUG_PRINT("info", ("last_lsn and prev_last_lsn set to 0 buffer: 0x%lx", (ulong) buffer)); buffer->offset= log_descriptor.horizon; @@ -2052,6 +2122,7 @@ buffer->file= get_current_logfile(); buffer->overlay= 0; buffer->size= 0; + buffer->skipped_data= 0; translog_cursor_init(cursor, buffer, buffer_no); DBUG_PRINT("info", ("file: #%ld (%d) init cursor #%u: 0x%lx " "chaser: %d Size: %lu (%lu)", @@ -2523,6 +2594,7 @@ TRANSLOG_ADDRESS offset= buffer->offset; TRANSLOG_FILE *file= buffer->file; uint8 ver= buffer->ver; + uint skipped_data; DBUG_ENTER("translog_buffer_flush"); DBUG_PRINT("enter", ("Buffer: #%u 0x%lx file: %d offset: (%lu,0x%lx) size: %lu", @@ -2557,6 +2629,8 @@ disk */ file= buffer->file; + skipped_data= buffer->skipped_data; + DBUG_ASSERT(skipped_data < TRANSLOG_PAGE_SIZE); for (i= 0, pg= LSN_OFFSET(buffer->offset) / TRANSLOG_PAGE_SIZE; i < buffer->size; i+= TRANSLOG_PAGE_SIZE, pg++) @@ -2573,13 +2647,16 @@ DBUG_ASSERT(i + TRANSLOG_PAGE_SIZE <= buffer->size); if (translog_status != TRANSLOG_OK && translog_status != TRANSLOG_SHUTDOWN) DBUG_RETURN(1); - if (pagecache_inject(log_descriptor.pagecache, + if (pagecache_write_part(log_descriptor.pagecache, &file->handler, pg, 3, buffer->buffer + i, PAGECACHE_PLAIN_PAGE, PAGECACHE_LOCK_LEFT_UNLOCKED, - PAGECACHE_PIN_LEFT_UNPINNED, 0, - LSN_IMPOSSIBLE)) + PAGECACHE_PIN_LEFT_UNPINNED, + PAGECACHE_WRITE_DONE, 0, + LSN_IMPOSSIBLE, + skipped_data, + TRANSLOG_PAGE_SIZE - skipped_data)) { DBUG_PRINT("error", ("Can't write page (%lu,0x%lx) to pagecache, error: %d", @@ -2589,10 +2666,12 @@ translog_stop_writing(); DBUG_RETURN(1); } + skipped_data= 0; } file->is_sync= 0; - if (my_pwrite(file->handler.file, buffer->buffer, - buffer->size, LSN_OFFSET(buffer->offset), + if (my_pwrite(file->handler.file, buffer->buffer + buffer->skipped_data, + buffer->size - buffer->skipped_data, + LSN_OFFSET(buffer->offset) + buffer->skipped_data, log_write_flags)) { DBUG_PRINT("error", ("Can't write buffer (%lu,0x%lx) size %lu " @@ -2985,6 +3064,7 @@ uchar *from, *table= NULL; int is_last_unfinished_page; uint last_protected_sector= 0; + uint skipped_data= curr_buffer->skipped_data; TRANSLOG_FILE file_copy; uint8 ver= curr_buffer->ver; translog_wait_for_writers(curr_buffer); @@ -2997,7 +3077,38 @@ } DBUG_ASSERT(LSN_FILE_NO(addr) == LSN_FILE_NO(curr_buffer->offset)); from= curr_buffer->buffer + (addr - curr_buffer->offset); - memcpy(buffer, from, TRANSLOG_PAGE_SIZE); + if (skipped_data && addr == curr_buffer->offset) + { + /* + We read page part of which is not present in buffer, + so we should read absent part from file (page cache actually) + */ + file= get_logfile_by_number(file_no); + DBUG_ASSERT(file != NULL); + /* + it's ok to not lock the page because: + - The log handler has it's own page cache. + - There is only one thread that can access the log + cache at a time + */ + if (!(buffer= pagecache_read(log_descriptor.pagecache, + &file->handler, + LSN_OFFSET(addr) / TRANSLOG_PAGE_SIZE, + 3, buffer, + PAGECACHE_PLAIN_PAGE, + PAGECACHE_LOCK_LEFT_UNLOCKED, + NULL))) + DBUG_RETURN(NULL); + } + else + skipped_data= 0; /* Read after skipped in buffer data */ + /* + Now we have correct data in buffer up to 'skipped_data'. The + following memcpy() will move the data from the internal buffer + that was not yet on disk. + */ + memcpy(buffer + skipped_data, from + skipped_data, + TRANSLOG_PAGE_SIZE - skipped_data); /* We can use copy then in translog_page_validator() because it do not put it permanently somewhere. @@ -3291,6 +3402,7 @@ uint32 next_page_offset, page_rest; uint32 i; File fd; + int rc; TRANSLOG_VALIDATOR_DATA data; char path[FN_REFLEN]; uchar page_buff[TRANSLOG_PAGE_SIZE]; @@ -3316,14 +3428,19 @@ TRANSLOG_PAGE_SIZE); page_rest= next_page_offset - LSN_OFFSET(addr); memset(page_buff, TRANSLOG_FILLER, page_rest); - if ((fd= open_logfile_by_number_no_cache(LSN_FILE_NO(addr))) < 0 || - ((my_chsize(fd, next_page_offset, TRANSLOG_FILLER, MYF(MY_WME)) || - (page_rest && my_pwrite(fd, page_buff, page_rest, LSN_OFFSET(addr), - log_write_flags)) || - my_sync(fd, MYF(MY_WME))) | - my_close(fd, MYF(MY_WME))) || - (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && - sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)))) + rc= ((fd= open_logfile_by_number_no_cache(LSN_FILE_NO(addr))) < 0 || + ((my_chsize(fd, next_page_offset, TRANSLOG_FILLER, MYF(MY_WME)) || + (page_rest && my_pwrite(fd, page_buff, page_rest, LSN_OFFSET(addr), + log_write_flags)) || + my_sync(fd, MYF(MY_WME))))); + translog_syncs++; + rc|= (fd > 0 && my_close(fd, MYF(MY_WME))); + if (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS) + { + rc|= sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)); + translog_syncs++; + } + if (rc) DBUG_RETURN(1); /* fix the horizon */ @@ -3483,7 +3600,10 @@ my_bool version_changed= 0; DBUG_ENTER("translog_init_with_table"); + translog_syncs= 0; + flush_start= 0; id_to_share= NULL; + log_descriptor.directory_fd= -1; log_descriptor.is_everything_flushed= 1; log_descriptor.flush_in_progress= 0; @@ -3511,6 +3631,7 @@ pthread_mutex_init(&log_descriptor.dirty_buffer_mask_lock, MY_MUTEX_INIT_FAST) || pthread_cond_init(&log_descriptor.log_flush_cond, 0) || + pthread_cond_init(&log_descriptor.new_goal_cond, 0) || my_rwlock_init(&log_descriptor.open_files_lock, NULL) || my_init_dynamic_array(&log_descriptor.open_files, @@ -3912,7 +4033,6 @@ log_descriptor.flushed= log_descriptor.horizon; log_descriptor.in_buffers_only= log_descriptor.bc.buffer->offset; log_descriptor.max_lsn= LSN_IMPOSSIBLE; /* set to 0 */ - log_descriptor.previous_flush_horizon= log_descriptor.horizon; /* Now 'flushed' is set to 'horizon' value, but 'horizon' is (potentially) address of the next LSN and we want indicate that all LSNs that are @@ -3995,6 +4115,10 @@ It is beginning of the log => there is no LSNs in the log => There is no harm in leaving it "as-is". */ + log_descriptor.previous_flush_horizon= log_descriptor.horizon; + DBUG_PRINT("info", ("previous_flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor. + previous_flush_horizon))); DBUG_RETURN(0); } file_no--; @@ -4070,6 +4194,9 @@ translog_free_record_header(&rec); } } + log_descriptor.previous_flush_horizon= log_descriptor.horizon; + DBUG_PRINT("info", ("previous_flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor.previous_flush_horizon))); DBUG_RETURN(0); err: ma_message_no_user(0, "log initialization failed"); @@ -4157,6 +4284,7 @@ pthread_mutex_destroy(&log_descriptor.log_flush_lock); pthread_mutex_destroy(&log_descriptor.dirty_buffer_mask_lock); pthread_cond_destroy(&log_descriptor.log_flush_cond); + pthread_cond_destroy(&log_descriptor.new_goal_cond); rwlock_destroy(&log_descriptor.open_files_lock); delete_dynamic(&log_descriptor.open_files); delete_dynamic(&log_descriptor.unfinished_files); @@ -6885,11 +7013,11 @@ { translog_size_t res; DBUG_ENTER("translog_read_record_header_from_buffer"); - DBUG_ASSERT(translog_is_LSN_chunk(page[page_offset])); - DBUG_ASSERT(translog_status == TRANSLOG_OK || - translog_status == TRANSLOG_READONLY); DBUG_PRINT("info", ("page byte: 0x%x offset: %u", (uint) page[page_offset], (uint) page_offset)); + DBUG_ASSERT(translog_is_LSN_chunk(page[page_offset])); + DBUG_ASSERT(translog_status == TRANSLOG_OK || + translog_status == TRANSLOG_READONLY); buff->type= (page[page_offset] & TRANSLOG_REC_TYPE); buff->short_trid= uint2korr(page + page_offset + 1); DBUG_PRINT("info", ("Type %u, Short TrID %u, LSN (%lu,0x%lx)", @@ -7356,27 +7484,27 @@ "Buffer addr: (%lu,0x%lx) " "Page addr: (%lu,0x%lx) " "size: %lu (%lu) Pg: %u left: %u in progress %u", - (uint) log_descriptor.bc.buffer_no, - (ulong) log_descriptor.bc.buffer, - LSN_IN_PARTS(log_descriptor.bc.buffer->offset), + (uint) old_buffer_no, + (ulong) old_buffer, + LSN_IN_PARTS(old_buffer->offset), (ulong) LSN_FILE_NO(log_descriptor.horizon), (ulong) (LSN_OFFSET(log_descriptor.horizon) - log_descriptor.bc.current_page_fill), - (ulong) log_descriptor.bc.buffer->size, + (ulong) old_buffer->size, (ulong) (log_descriptor.bc.ptr -log_descriptor.bc. buffer->buffer), (uint) log_descriptor.bc.current_page_fill, (uint) left, - (uint) log_descriptor.bc.buffer-> + (uint) old_buffer-> copy_to_buffer_in_progress)); translog_lock_assert_owner(); LINT_INIT(current_page_fill); - new_buff_beginning= log_descriptor.bc.buffer->offset; - new_buff_beginning+= log_descriptor.bc.buffer->size; /* increase offset */ + new_buff_beginning= old_buffer->offset; + new_buff_beginning+= old_buffer->size; /* increase offset */ DBUG_ASSERT(log_descriptor.bc.ptr !=NULL); DBUG_ASSERT(LSN_FILE_NO(log_descriptor.horizon) == - LSN_FILE_NO(log_descriptor.bc.buffer->offset)); + LSN_FILE_NO(old_buffer->offset)); translog_check_cursor(&log_descriptor.bc); DBUG_ASSERT(left < TRANSLOG_PAGE_SIZE); if (left) @@ -7387,18 +7515,20 @@ */ DBUG_PRINT("info", ("left: %u", (uint) left)); + old_buffer->pre_force_close_horizon= + old_buffer->offset + old_buffer->size; /* decrease offset */ new_buff_beginning-= log_descriptor.bc.current_page_fill; current_page_fill= log_descriptor.bc.current_page_fill; memset(log_descriptor.bc.ptr, TRANSLOG_FILLER, left); - log_descriptor.bc.buffer->size+= left; + old_buffer->size+= left; DBUG_PRINT("info", ("Finish Page buffer #%u: 0x%lx " "Size: %lu", - (uint) log_descriptor.bc.buffer->buffer_no, - (ulong) log_descriptor.bc.buffer, - (ulong) log_descriptor.bc.buffer->size)); - DBUG_ASSERT(log_descriptor.bc.buffer->buffer_no == + (uint) old_buffer->buffer_no, + (ulong) old_buffer, + (ulong) old_buffer->size)); + DBUG_ASSERT(old_buffer->buffer_no == log_descriptor.bc.buffer_no); } else @@ -7509,11 +7639,21 @@ if (left) { - /* - TODO: do not copy beginning of the page if we have no CRC or sector - checks on - */ - memcpy(new_buffer->buffer, data, current_page_fill); + if (log_descriptor.flags & + (TRANSLOG_PAGE_CRC | TRANSLOG_SECTOR_PROTECTION)) + memcpy(new_buffer->buffer, data, current_page_fill); + else + { + /* + This page header does not change if we add more data to the page so + we can not copy it and will not overwrite later + */ + new_buffer->skipped_data= current_page_fill; +#ifndef DBUG_OFF + memset(new_buffer->buffer, 0xa5, current_page_fill); +#endif + DBUG_ASSERT(new_buffer->skipped_data < TRANSLOG_PAGE_SIZE); + } } old_buffer->next_buffer_offset= new_buffer->offset; translog_buffer_lock(new_buffer); @@ -7561,6 +7701,7 @@ { log_descriptor.next_pass_max_lsn= lsn; log_descriptor.max_lsn_requester= pthread_self(); + pthread_cond_broadcast(&log_descriptor.new_goal_cond); } while (flush_no == log_descriptor.flush_no) { @@ -7572,66 +7713,78 @@ /** - @brief Flush the log up to given LSN (included) - - @param lsn log record serial number up to which (inclusive) - the log has to be flushed - - @return Operation status + @brief sync() range of files (inclusive) and directory (by request) + + @param min min internal file number to flush + @param max max internal file number to flush + @param sync_dir need sync directory + + return Operation status @retval 0 OK @retval 1 Error - -*/ - -my_bool translog_flush(TRANSLOG_ADDRESS lsn) -{ - LSN sent_to_disk= LSN_IMPOSSIBLE; - TRANSLOG_ADDRESS flush_horizon; - uint fn, i; +*/ + +static my_bool translog_sync_files(uint32 min, uint32 max, + my_bool sync_dir) +{ + uint fn; + my_bool rc= 0; + ulonglong flush_interval; + DBUG_ENTER("translog_sync_files"); + DBUG_PRINT("info", ("min: %lu max: %lu sync dir: %d", + (ulong) min, (ulong) max, (int) sync_dir)); + DBUG_ASSERT(min <= max); + + flush_interval= group_commit_wait; + if (flush_interval) + flush_start= my_micro_time(); + for (fn= min; fn <= max; fn++) + { + TRANSLOG_FILE *file= get_logfile_by_number(fn); + DBUG_ASSERT(file != NULL); + if (!file->is_sync) + { + if (my_sync(file->handler.file, MYF(MY_WME))) + { + rc= 1; + translog_stop_writing(); + DBUG_RETURN(rc); + } + translog_syncs++; + file->is_sync= 1; + } + } + + if (sync_dir) + { + if (!(rc= sync_dir(log_descriptor.directory_fd, + MYF(MY_WME | MY_IGNORE_BADFD)))) + translog_syncs++; + } + + DBUG_RETURN(rc); +} + + +/* + @brief Flushes buffers with LSNs in them less or equal address <lsn> + + @param lsn address up to which all LSNs should be flushed, + can be reset to real last LSN address + @parem sent_to_disk returns 'sent to disk' position + @param flush_horizon returns horizon of the flush + + @note About terminology see comment to translog_flush(). +*/ + +void translog_flush_buffers(TRANSLOG_ADDRESS *lsn, + TRANSLOG_ADDRESS *sent_to_disk, + TRANSLOG_ADDRESS *flush_horizon) +{ dirty_buffer_mask_t dirty_buffer_mask; + uint i; uint8 last_buffer_no, start_buffer_no; - my_bool rc= 0; - DBUG_ENTER("translog_flush"); - DBUG_PRINT("enter", ("Flush up to LSN: (%lu,0x%lx)", LSN_IN_PARTS(lsn))); - DBUG_ASSERT(translog_status == TRANSLOG_OK || - translog_status == TRANSLOG_READONLY); - LINT_INIT(sent_to_disk); - - pthread_mutex_lock(&log_descriptor.log_flush_lock); - DBUG_PRINT("info", ("Everything is flushed up to (%lu,0x%lx)", - LSN_IN_PARTS(log_descriptor.flushed))); - if (cmp_translog_addr(log_descriptor.flushed, lsn) >= 0) - { - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - DBUG_RETURN(0); - } - if (log_descriptor.flush_in_progress) - { - translog_flush_set_new_goal_and_wait(lsn); - if (!pthread_equal(log_descriptor.max_lsn_requester, pthread_self())) - { - /* fix lsn if it was horizon */ - if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->last_lsn) > 0) - lsn= BUFFER_MAX_LSN(log_descriptor.bc.buffer); - translog_flush_wait_for_end(lsn); - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - DBUG_RETURN(0); - } - log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; - } - log_descriptor.flush_in_progress= 1; - flush_horizon= log_descriptor.previous_flush_horizon; - DBUG_PRINT("info", ("flush_in_progress is set")); - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - - translog_lock(); - if (log_descriptor.is_everything_flushed) - { - DBUG_PRINT("info", ("everything is flushed")); - rc= (translog_status == TRANSLOG_READONLY); - translog_unlock(); - goto out; - } + DBUG_ENTER("translog_flush_buffers"); /* We will recheck information when will lock buffers one by @@ -7656,15 +7809,15 @@ /* if LSN up to which we have to flush bigger then maximum LSN of previous buffer and at least one LSN was saved in the current buffer (last_lsn != - LSN_IMPOSSIBLE) then we better finish the current buffer. + LSN_IMPOSSIBLE) then we have to close the current buffer. */ - if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->prev_last_lsn) > 0 && + if (cmp_translog_addr(*lsn, log_descriptor.bc.buffer->prev_last_lsn) > 0 && log_descriptor.bc.buffer->last_lsn != LSN_IMPOSSIBLE) { struct st_translog_buffer *buffer= log_descriptor.bc.buffer; - lsn= log_descriptor.bc.buffer->last_lsn; /* fix lsn if it was horizon */ + *lsn= log_descriptor.bc.buffer->last_lsn; /* fix lsn if it was horizon */ DBUG_PRINT("info", ("LSN to flush fixed to last lsn: (%lu,0x%lx)", - LSN_IN_PARTS(log_descriptor.bc.buffer->last_lsn))); + LSN_IN_PARTS(log_descriptor.bc.buffer->last_lsn))); last_buffer_no= log_descriptor.bc.buffer_no; log_descriptor.is_everything_flushed= 1; translog_force_current_buffer_to_finish(); @@ -7676,8 +7829,10 @@ TRANSLOG_BUFFERS_NO); translog_unlock(); } - sent_to_disk= translog_get_sent_to_disk(); - if (cmp_translog_addr(lsn, sent_to_disk) > 0) + + /* flush buffers */ + *sent_to_disk= translog_get_sent_to_disk(); + if (cmp_translog_addr(*lsn, *sent_to_disk) > 0) { DBUG_PRINT("info", ("Start buffer #: %u last buffer #: %u", @@ -7697,53 +7852,238 @@ LSN_IN_PARTS(buffer->last_lsn), (buffer->file ? "dirty" : "closed"))); - if (buffer->prev_last_lsn <= lsn && + if (buffer->prev_last_lsn <= *lsn && buffer->file != NULL) { - DBUG_ASSERT(flush_horizon <= buffer->offset + buffer->size); - flush_horizon= buffer->offset + buffer->size; + DBUG_ASSERT(*flush_horizon <= buffer->offset + buffer->size); + *flush_horizon= (buffer->pre_force_close_horizon != LSN_IMPOSSIBLE ? + buffer->pre_force_close_horizon : + buffer->offset + buffer->size); + /* pre_force_close_horizon is reset during new buffer start */ + DBUG_PRINT("info", ("flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(*flush_horizon))); + DBUG_ASSERT(*flush_horizon <= log_descriptor.horizon); + translog_buffer_flush(buffer); } translog_buffer_unlock(buffer); i= (i + 1) % TRANSLOG_BUFFERS_NO; } while (i != last_buffer_no); - sent_to_disk= translog_get_sent_to_disk(); - } - - /* sync files from previous flush till current one */ - for (fn= LSN_FILE_NO(log_descriptor.flushed); fn <= LSN_FILE_NO(lsn); fn++) - { - TRANSLOG_FILE *file= get_logfile_by_number(fn); - DBUG_ASSERT(file != NULL); - if (!file->is_sync) - { - if (my_sync(file->handler.file, MYF(MY_WME))) + *sent_to_disk= translog_get_sent_to_disk(); + } + + DBUG_VOID_RETURN; +} + +/** + @brief Flush the log up to given LSN (included) + + @param lsn log record serial number up to which (inclusive) + the log has to be flushed + + @return Operation status + @retval 0 OK + @retval 1 Error + + @note + + - Non group commit logic: Commits made in passes. Thread which started + flush first is performing actual flush, other threads sets new goal (LSN) + of the next pass (if it is maximum) and waits for the pass end or just + wait for the pass end. + + - If hard group commit enabled and rate set to zero: + The first thread sends all changed buffers to disk. This is repeated + as long as there are new LSNs added. The process can not loop + forever because we have limited number of threads and they will wait + for the data to be synced. + Pseudo code: + + do + send changed buffers to disk + while new_goal + sync + + - If hard group commit switched ON and less than rate microseconds has + passed from last sync, then after buffers have been sent to disk + wait until rate microseconds has passed since last sync, do sync and return. + This ensures that if we call sync infrequently we don't do any waits. + + - If soft group commit enabled everything works as with 'non group commit' + but the thread doesn't do any real sync(). If rate is not zero the + sync() will be performed by a service thread with the given rate + when needed (new LSN appears). + + @note Terminology: + 'sent to disk' means written to disk but not sync()ed, + 'flushed' mean sent to disk and synced(). +*/ + +my_bool translog_flush(TRANSLOG_ADDRESS lsn) +{ + struct timespec abstime; + ulonglong flush_interval; + ulonglong time_spent; + LSN sent_to_disk= LSN_IMPOSSIBLE; + TRANSLOG_ADDRESS flush_horizon; + my_bool rc= 0; + my_bool hgroup_commit_at_start; + DBUG_ENTER("translog_flush"); + DBUG_PRINT("enter", ("Flush up to LSN: (%lu,0x%lx)", LSN_IN_PARTS(lsn))); + DBUG_ASSERT(translog_status == TRANSLOG_OK || + translog_status == TRANSLOG_READONLY); + LINT_INIT(sent_to_disk); + + pthread_mutex_lock(&log_descriptor.log_flush_lock); + DBUG_PRINT("info", ("Everything is flushed up to (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor.flushed))); + if (cmp_translog_addr(log_descriptor.flushed, lsn) >= 0) + { + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_RETURN(0); + } + if (log_descriptor.flush_in_progress) + { + translog_lock(); + /* fix lsn if it was horizon */ + if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->last_lsn) > 0) + lsn= BUFFER_MAX_LSN(log_descriptor.bc.buffer); + translog_unlock(); + translog_flush_set_new_goal_and_wait(lsn); + if (!pthread_equal(log_descriptor.max_lsn_requester, pthread_self())) + { + /* + translog_flush_wait_for_end() release log_flush_lock while is + waiting then acquire it again + */ + translog_flush_wait_for_end(lsn); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_RETURN(0); + } + log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; + } + log_descriptor.flush_in_progress= 1; + flush_horizon= log_descriptor.previous_flush_horizon; + DBUG_PRINT("info", ("flush_in_progress is set, flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(flush_horizon))); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + + hgroup_commit_at_start= hard_group_commit; + if (hgroup_commit_at_start) + flush_interval= group_commit_wait; + + translog_lock(); + if (log_descriptor.is_everything_flushed) + { + DBUG_PRINT("info", ("everything is flushed")); + translog_unlock(); + pthread_mutex_lock(&log_descriptor.log_flush_lock); + goto out; + } + + for (;;) + { + /* Following function flushes buffers and makes translog_unlock() */ + translog_flush_buffers(&lsn, &sent_to_disk, &flush_horizon); + + if (!hgroup_commit_at_start) + break; /* flush pass is ended */ + +retest: + /* + We do not check time here because pthread_mutex_lock rarely takes + a lot of time so we can sacrifice a bit precision to performance + (taking into account that my_micro_time() might be expensive call). + */ + if (flush_interval == 0) + break; /* flush pass is ended */ + + pthread_mutex_lock(&log_descriptor.log_flush_lock); + if (log_descriptor.next_pass_max_lsn == LSN_IMPOSSIBLE) + { + if (flush_interval == 0 || + (time_spent= (my_micro_time() - flush_start)) >= flush_interval) { - rc= 1; - translog_stop_writing(); - sent_to_disk= LSN_IMPOSSIBLE; - goto out; + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + break; } - file->is_sync= 1; - } - } - - if (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && - (LSN_FILE_NO(log_descriptor.previous_flush_horizon) != - LSN_FILE_NO(flush_horizon) || - ((LSN_OFFSET(log_descriptor.previous_flush_horizon) - 1) / - TRANSLOG_PAGE_SIZE) != - ((LSN_OFFSET(flush_horizon) - 1) / TRANSLOG_PAGE_SIZE))) - rc|= sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)); + DBUG_PRINT("info", ("flush waits: %llu interval: %llu spent: %llu", + flush_interval - time_spent, + flush_interval, time_spent)); + /* wait time or next goal */ + set_timespec_nsec(abstime, flush_interval - time_spent); + pthread_cond_timedwait(&log_descriptor.new_goal_cond, + &log_descriptor.log_flush_lock, + &abstime); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_PRINT("info", ("retest conditions")); + goto retest; + } + + /* take next goal */ + lsn= log_descriptor.next_pass_max_lsn; + log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; + /* prevent other thread from continue */ + log_descriptor.max_lsn_requester= pthread_self(); + DBUG_PRINT("info", ("flush took next goal: (%lu,0x%lx)", + LSN_IN_PARTS(lsn))); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + + /* next flush pass */ + DBUG_PRINT("info", ("next flush pass")); + translog_lock(); + } + + /* + sync() files from previous flush till current one + */ + if (!soft_sync || hgroup_commit_at_start) + { + if ((rc= + translog_sync_files(LSN_FILE_NO(log_descriptor.flushed), + LSN_FILE_NO(lsn), + sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && + (LSN_FILE_NO(log_descriptor. + previous_flush_horizon) != + LSN_FILE_NO(flush_horizon) || + (LSN_OFFSET(log_descriptor. + previous_flush_horizon) / + TRANSLOG_PAGE_SIZE) != + (LSN_OFFSET(flush_horizon) / + TRANSLOG_PAGE_SIZE))))) + { + sent_to_disk= LSN_IMPOSSIBLE; + pthread_mutex_lock(&log_descriptor.log_flush_lock); + goto out; + } + /* keep values for soft sync() and forced sync() actual */ + { + uint32 fileno= LSN_FILE_NO(lsn); + my_atomic_rwlock_wrlock(&soft_sync_rwl); + my_atomic_store32(&soft_sync_min, fileno); + my_atomic_store32(&soft_sync_max, fileno); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + } + } + else + { + my_atomic_rwlock_wrlock(&soft_sync_rwl); + my_atomic_store32(&soft_sync_max, LSN_FILE_NO(lsn)); + my_atomic_store32(&soft_need_sync, 1); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + } + + DBUG_ASSERT(flush_horizon <= log_descriptor.horizon); + + pthread_mutex_lock(&log_descriptor.log_flush_lock); log_descriptor.previous_flush_horizon= flush_horizon; out: - pthread_mutex_lock(&log_descriptor.log_flush_lock); if (sent_to_disk != LSN_IMPOSSIBLE) log_descriptor.flushed= sent_to_disk; log_descriptor.flush_in_progress= 0; log_descriptor.flush_no++; DBUG_PRINT("info", ("flush_in_progress is dropped")); - pthread_mutex_unlock(&log_descriptor.log_flush_lock);\ + pthread_mutex_unlock(&log_descriptor.log_flush_lock); pthread_cond_broadcast(&log_descriptor.log_flush_cond); DBUG_RETURN(rc); } @@ -8113,6 +8453,8 @@ my_bool translog_purge(TRANSLOG_ADDRESS low) { uint32 last_need_file= LSN_FILE_NO(low); + uint32 min_unsync; + int soft; TRANSLOG_ADDRESS horizon= translog_get_horizon(); int rc= 0; DBUG_ENTER("translog_purge"); @@ -8120,12 +8462,26 @@ DBUG_ASSERT(translog_status == TRANSLOG_OK || translog_status == TRANSLOG_READONLY); + soft= soft_sync; + my_atomic_rwlock_wrlock(&soft_sync_rwl); + min_unsync= my_atomic_load32(&soft_sync_min); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + DBUG_PRINT("info", ("min_unsync: %lu", (ulong) min_unsync)); + if (soft && min_unsync < last_need_file) + { + last_need_file= min_unsync; + DBUG_PRINT("info", ("last_need_file set to %lu", (ulong)last_need_file)); + } + pthread_mutex_lock(&log_descriptor.purger_lock); + DBUG_PRINT("info", ("last_lsn_checked file: %lu:", + (ulong) log_descriptor.last_lsn_checked)); if (LSN_FILE_NO(log_descriptor.last_lsn_checked) < last_need_file) { uint32 i; uint32 min_file= translog_first_file(horizon, 1); DBUG_ASSERT(min_file != 0); /* log is already started */ + DBUG_PRINT("info", ("min_file: %lu:",(ulong) min_file)); for(i= min_file; i < last_need_file && rc == 0; i++) { LSN lsn= translog_get_file_max_lsn_stored(i); @@ -8356,6 +8712,159 @@ } + +/** + Sets soft sync mode + + @param mode TRUE if we need switch soft sync on else off +*/ + +void translog_soft_sync(my_bool mode) +{ + soft_sync= mode; +} + + +/** + Sets hard group commit + + @param mode TRUE if we need switch hard group commit on else off +*/ + +void translog_hard_group_commit(my_bool mode) +{ + hard_group_commit= mode; +} + + +/** + @brief forced log sync (used when we are switching modes) +*/ + +void translog_sync() +{ + uint32 max= get_current_logfile()->number; + uint32 min; + DBUG_ENTER("ma_translog_sync"); + + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + if (!min) + min= max; + + translog_sync_files(min, max, sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS); + + DBUG_VOID_RETURN; +} + + +/** + @brief set rate for group commit + + @param interval interval to set. + + @note We use this function with additional variable because have to + restart service thread with new value which we can't make inside changing + variable routine (update_maria_group_commit_interval) +*/ + +void translog_set_group_commit_interval(uint32 interval) +{ + DBUG_ENTER("translog_set_group_commit_interval"); + group_commit_wait= interval; + DBUG_PRINT("info", ("wait: %llu", + (ulonglong)group_commit_wait)); + DBUG_VOID_RETURN; +} + + +/** + @brief syncing service thread +*/ + +static pthread_handler_t +ma_soft_sync_background( void *arg __attribute__((unused))) +{ + + my_thread_init(); + { + DBUG_ENTER("ma_soft_sync_background"); + for(;;) + { + ulonglong prev_loop= my_micro_time(); + ulonglong time, sleep; + uint32 min, max, sync_request; + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + max= my_atomic_load32(&soft_sync_max); + sync_request= my_atomic_load32(&soft_need_sync); + my_atomic_store32(&soft_sync_min, max); + my_atomic_store32(&soft_need_sync, 0); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + + sleep= group_commit_wait; + if (sync_request) + translog_sync_files(min, max, FALSE); + time= my_micro_time() - prev_loop; + if (time > sleep) + sleep= 0; + else + sleep-= time; + if (my_service_thread_sleep(&soft_sync_control, sleep)) + break; + } + my_service_thread_signal_end(&soft_sync_control); + my_thread_end(); + DBUG_RETURN(0); + } +} + + +/** + @brief Starts syncing thread +*/ + +int translog_soft_sync_start(void) +{ + pthread_t th; + int res= 0; + uint32 min, max; + DBUG_ENTER("translog_soft_sync_start"); + + /* check and init variables */ + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + max= my_atomic_load32(&soft_sync_max); + if (!max) + my_atomic_store32(&soft_sync_max, (max= get_current_logfile()->number)); + if (!min) + my_atomic_store32(&soft_sync_min, max); + my_atomic_store32(&soft_need_sync, 1); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + + if (!(res= ma_service_thread_control_init(&soft_sync_control))) + if (!(res= pthread_create(&th, NULL, ma_soft_sync_background, NULL))) + soft_sync_control.status= THREAD_RUNNING; + DBUG_RETURN(res); +} + + +/** + @brief Stops syncing thread +*/ + +void translog_soft_sync_end(void) +{ + DBUG_ENTER("translog_soft_sync_end"); + if (soft_sync_control.inited) + { + ma_service_thread_control_end(&soft_sync_control); + } + DBUG_VOID_RETURN; +} + + #ifdef MARIA_DUMP_LOG #include <my_getopt.h> extern void translog_example_table_init(); === modified file 'storage/maria/ma_loghandler.h' --- a/storage/maria/ma_loghandler.h 2009-01-15 22:25:53 +0000 +++ b/storage/maria/ma_loghandler.h 2010-02-12 06:52:47 +0000 @@ -342,6 +342,14 @@ TRANSLOG_SHUTDOWN /* going to shutdown the loghandler */ }; extern enum enum_translog_status translog_status; +extern ulonglong translog_syncs; /* Number of sync()s */ + +void translog_soft_sync(my_bool mode); +void translog_hard_group_commit(my_bool mode); +int translog_soft_sync_start(void); +void translog_soft_sync_end(void); +void translog_sync(); +void translog_set_group_commit_interval(uint32 interval); /* all the rest added because of recovery; should we make @@ -441,6 +449,14 @@ typedef enum { + TRANSLOG_GCOMMIT_NONE, + TRANSLOG_GCOMMIT_HARD, + TRANSLOG_GCOMMIT_SOFT +} enum_maria_group_commit; +extern ulong maria_group_commit; +extern ulong maria_group_commit_interval; +typedef enum +{ TRANSLOG_PURGE_IMMIDIATE, TRANSLOG_PURGE_EXTERNAL, TRANSLOG_PURGE_ONDEMAND

1 0

[Maria-developers] Rev 2756: BUG#31480: Incorrect result for nested subquery when executed via semi join in file:///home/psergey/dev/maria-5.3-subqueries-r6/
by Sergey Petrunya 12 Feb '10

12 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r6/ ------------------------------------------------------------ revno: 2756 revision-id: psergey(a)askmonty.org-20100211235958-p11o4e80dlrn2bsq parent: psergey(a)askmonty.org-20100211223118-5fzuidow1pkubpzl committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r6 timestamp: Fri 2010-02-12 02:59:58 +0300 message: BUG#31480: Incorrect result for nested subquery when executed via semi join - Variant #3 of the fix. It also = Unifies code with table elimination's = is able to handle FROM-subquery pullout. === modified file 'mysql-test/r/subselect_sj.result' --- a/mysql-test/r/subselect_sj.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect_sj.result 2010-02-11 23:59:58 +0000 @@ -779,3 +779,48 @@ 1 PRIMARY it2 ALL NULL NULL NULL NULL 20 Using where; End temporary DROP TABLE ot1, it1, it2; # End of BUG#38075 +# +# BUG#31480: Incorrect result for nested subquery when executed via semi join +# +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +id select_type table type possible_keys key key_len ref rows filtered Extra +1 PRIMARY t2 ALL NULL NULL NULL NULL 6 100.00 Start temporary +1 PRIMARY t1 ALL NULL NULL NULL NULL 7 100.00 Using where; End temporary; Using join buffer +3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where +Warnings: +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +show warnings; +Level Code Message +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +a +2 +2 +3 +2 +drop table t1, t2, t3; === modified file 'mysql-test/r/subselect_sj_jcl6.result' --- a/mysql-test/r/subselect_sj_jcl6.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect_sj_jcl6.result 2010-02-11 23:59:58 +0000 @@ -783,6 +783,51 @@ 1 PRIMARY it2 ALL NULL NULL NULL NULL 20 Using where; End temporary; Using join buffer DROP TABLE ot1, it1, it2; # End of BUG#38075 +# +# BUG#31480: Incorrect result for nested subquery when executed via semi join +# +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +id select_type table type possible_keys key key_len ref rows filtered Extra +1 PRIMARY t2 ALL NULL NULL NULL NULL 6 100.00 Start temporary +1 PRIMARY t1 ALL NULL NULL NULL NULL 7 100.00 Using where; End temporary; Using join buffer +3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where +Warnings: +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +show warnings; +Level Code Message +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +a +2 +2 +3 +2 +drop table t1, t2, t3; set join_cache_level=default; show variables like 'join_cache_level'; Variable_name Value === modified file 'mysql-test/t/subselect_sj.test' --- a/mysql-test/t/subselect_sj.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_sj.test 2010-02-11 23:59:58 +0000 @@ -681,3 +681,41 @@ DROP TABLE ot1, it1, it2; --echo # End of BUG#38075 + +--echo # +--echo # BUG#31480: Incorrect result for nested subquery when executed via semi join +--echo # +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); + +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); + +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); + +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); + +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +show warnings; + +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); + +drop table t1, t2, t3; + === modified file 'sql/item.cc' --- a/sql/item.cc 2010-02-11 22:00:36 +0000 +++ b/sql/item.cc 2010-02-11 23:59:58 +0000 @@ -3647,7 +3647,7 @@ substitution) */ -static void mark_as_dependent(THD *thd, SELECT_LEX *last, SELECT_LEX *current, +static bool mark_as_dependent(THD *thd, SELECT_LEX *last, SELECT_LEX *current, Item_ident *resolved_item, Item_ident *mark_item) { @@ -3658,7 +3658,9 @@ /* store pointer on SELECT_LEX from which item is dependent */ if (mark_item) mark_item->depended_from= last; - current->mark_as_dependent(last, resolved_item); + if (current->mark_as_dependent(thd, last, /** resolved_item psergey-thu + **/mark_item)) + return TRUE; if (thd->lex->describe & DESCRIBE_EXTENDED) { push_warning_printf(thd, MYSQL_ERROR::WARN_LEVEL_NOTE, @@ -3668,6 +3670,7 @@ resolved_item->field_name, current->select_number, last->select_number); } + return FALSE; } @@ -4119,6 +4122,7 @@ ((ref_type == REF_ITEM || ref_type == FIELD_ITEM) ? (Item_ident*) (*reference) : 0)); + /* A reference to a view field had been found and we substituted it instead of this Item (find_field_in_tables @@ -4218,7 +4222,7 @@ return -1; mark_as_dependent(thd, last_checked_context->select_lex, - context->select_lex, this, + context->select_lex, rf, rf); return 0; } @@ -5998,7 +6002,7 @@ goto error; thd->change_item_tree(reference, fld); mark_as_dependent(thd, last_checked_context->select_lex, - thd->lex->current_select, this, fld); + thd->lex->current_select, fld, fld); /* A reference is resolved to a nest level that's outer or the same as the nest level of the enclosing set function : adjust the value of @@ -6438,7 +6442,7 @@ if (depended_from == new_parent) { *ref= outer_ref; - outer_ref->fix_after_pullout(new_parent, ref); + (*ref)->fix_after_pullout(new_parent, ref); } } === modified file 'sql/item.h' --- a/sql/item.h 2010-02-11 22:00:36 +0000 +++ b/sql/item.h 2010-02-11 23:59:58 +0000 @@ -1115,7 +1115,9 @@ /* - Class to be used to enumerate all field references in an item tree. + Class to be used to enumerate all field references in an item tree. This + includes references to outside but not fields of the tables within a + subquery. Suggested usage: class My_enumerator : public Field_enumerator @@ -2377,6 +2379,8 @@ } bool walk(Item_processor processor, bool walk_subquery, uchar *arg) { return (*ref)->walk(processor, walk_subquery, arg); } + bool enumerate_field_refs_processor(uchar *arg) + { return (*ref)->enumerate_field_refs_processor(arg); } virtual void print(String *str, enum_query_type query_type); bool result_as_longlong() { === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-01-28 13:48:33 +0000 +++ b/sql/item_subselect.cc 2010-02-11 23:59:58 +0000 @@ -39,8 +39,8 @@ Item_subselect::Item_subselect(): Item_result_field(), value_assigned(0), thd(0), substitution(0), engine(0), old_engine(0), used_tables_cache(0), have_to_be_excluded(0), - const_item_cache(1), in_fix_fields(0), engine_changed(0), changed(0), - is_correlated(FALSE) + const_item_cache(1), inside_first_fix_fields(0), done_first_fix_fields(FALSE), + engine_changed(0), changed(0), is_correlated(FALSE) { with_subselect= 1; reset(); @@ -167,18 +167,23 @@ DBUG_ASSERT(fixed == 0); engine->set_thd((thd= thd_param)); - if (!in_fix_fields) - refers_to.empty(); + if (!done_first_fix_fields) + { + done_first_fix_fields= TRUE; + inside_first_fix_fields= TRUE; + } + eliminated= FALSE; + parent_select= thd_param->lex->current_select; if (check_stack_overrun(thd, STACK_MIN_SIZE, (uchar*)&res)) return TRUE; - in_fix_fields++; res= engine->prepare(); // all transformation is done (used by prepared statements) changed= 1; + inside_first_fix_fields= FALSE; if (!res) { @@ -210,14 +215,12 @@ if (!(*ref)->fixed) ret= (*ref)->fix_fields(thd, ref); thd->where= save_where; - in_fix_fields--; return ret; } // Is it one field subselect? if (engine->cols() > max_columns) { my_error(ER_OPERAND_COLUMNS, MYF(0), 1); - in_fix_fields--; return TRUE; } fix_length_and_dec(); @@ -234,7 +237,6 @@ fixed= 1; err: - in_fix_fields--; thd->where= save_where; return res; } @@ -242,11 +244,12 @@ bool Item_subselect::enumerate_field_refs_processor(uchar *arg) { - List_iterator<Item> it(refers_to); - Item *item; - while ((item= it++)) + List_iterator<Ref_to_outside> it(upper_refs); + Ref_to_outside *upper; + + while ((upper= it++)) { - if (item->walk(&Item::enumerate_field_refs_processor, FALSE, arg)) + if (upper->item->walk(&Item::enumerate_field_refs_processor, FALSE, arg)) return TRUE; } return FALSE; @@ -258,6 +261,142 @@ return FALSE; } + +bool Item_subselect::mark_as_dependent(THD *thd, st_select_lex *select, + Item *item) +{ + if (inside_first_fix_fields) + { + is_correlated= TRUE; + Ref_to_outside *upper; + if (!(upper= new (thd->stmt_arena->mem_root) Ref_to_outside())) + return TRUE; + upper->select= select; + upper->item= item; + if (upper_refs.push_back(upper, thd->stmt_arena->mem_root)) + return TRUE; + } + return FALSE; +} + +/* + Adjust attributes after our parent select has been merged into grandparent + + DESCRIPTION + Subquery is a composite object which may be correlated, that is, it may + have + 1. references to tables of the parent select (i.e. one that has the clause + with the subquery predicate) + 2. references to tables of the grandparent select + 3. references to tables of further ancestors. + + Before the pullout, this item indicates: + - #1 with table bits in used_tables() + - #2 and #3 with OUTER_REF_TABLE_BIT. + + After parent has been merged with grandparent: + - references to parent and grandparent tables should be indicated with + table bits. + - references to greatgrandparent and further ancestors - with + OUTER_REF_TABLE_BIT. +*/ + +void Item_subselect::fix_after_pullout(st_select_lex *new_parent, Item **ref) +{ + recalc_used_tables(new_parent, TRUE); + parent_select= new_parent; +} + +class Field_fixer: public Field_enumerator +{ +public: + table_map used_tables; /* Collect used_tables here */ + st_select_lex *new_parent; /* Select we're in */ + virtual void visit_field(Field *field) + { + //for (TABLE_LIST *tbl= new_parent->leaf_tables; tbl; tbl= tbl->next_local) + //{ + // if (tbl->table == field->table) + // { + used_tables|= field->table->map; + // return; + // } + //} + //used_tables |= OUTER_REF_TABLE_BIT; + } +}; + + +/* + Recalculate used_tables_cache +*/ + +void Item_subselect::recalc_used_tables(st_select_lex *new_parent, + bool after_pullout) +{ + List_iterator<Ref_to_outside> it(upper_refs); + Ref_to_outside *upper; + + used_tables_cache= 0; + while ((upper= it++)) + { + bool found= FALSE; + /* + Check if + 1. the upper reference refers to the new immediate parent select, or + 2. one of the further ancestors. + + We rely on the fact that the tree of selects is modified by some kind of + 'flattening', i.e. a process where child selects are merged into their + parents. + The merged selects are removed from the select tree but keep pointers to + their parents. + */ + for (st_select_lex *sel= upper->select; sel; sel= sel->outer_select()) + { + /* + If we've reached the new parent select by walking upwards from + reference's original select, this means that the reference is now + referring to the direct parent: + */ + if (sel == new_parent) + { + found= TRUE; + /* + upper->item may be NULL when we've referred to a grouping function, + in which case we don't care about what it's table_map really is, + because item->with_sum_func==1 will ensure correct placement of the + item. + */ + if (upper->item) + { + // Now, iterate over fields and collect used_tables() attribute: + Field_fixer fixer; + fixer.used_tables= 0; + fixer.new_parent= new_parent; + upper->item->walk(&Item::enumerate_field_refs_processor, FALSE, + (uchar*)&fixer); + used_tables_cache |= fixer.used_tables; + /* + if (after_pullout) + upper->item->fix_after_pullout(new_parent, &(upper->item)); + upper->item->update_used_tables(); + used_tables_cache |= upper->item->used_tables(); + */ + } + } + } + if (!found) + used_tables_cache|= OUTER_REF_TABLE_BIT; + } + /* + Don't update const_tables_cache yet as we don't yet know which of the + parent's tables are constant. Parent will call update_used_tables() after + he has done const table detection, and that will be our chance to update + const_tables_cache. + */ +} + bool Item_subselect::walk(Item_processor processor, bool walk_subquery, uchar *argument) { @@ -397,6 +536,7 @@ void Item_subselect::update_used_tables() { + recalc_used_tables(parent_select, FALSE); if (!engine->uncacheable()) { // did all used tables become static? @@ -1843,6 +1983,18 @@ return result || Item_subselect::fix_fields(thd_arg, ref); } +void Item_in_subselect::fix_after_pullout(st_select_lex *new_parent, Item **ref) +{ + left_expr->fix_after_pullout(new_parent, &left_expr); + Item_subselect::fix_after_pullout(new_parent, ref); +} + +void Item_in_subselect::update_used_tables() +{ + Item_subselect::update_used_tables(); + left_expr->update_used_tables(); + used_tables_cache |= left_expr->used_tables(); +} /** Try to create an engine to compute the subselect via materialization, === modified file 'sql/item_subselect.h' --- a/sql/item_subselect.h 2010-01-28 13:48:33 +0000 +++ b/sql/item_subselect.h 2010-02-11 23:59:58 +0000 @@ -67,14 +67,32 @@ bool have_to_be_excluded; /* cache of constant state */ bool const_item_cache; - + + bool inside_first_fix_fields; + bool done_first_fix_fields; public: - /* - References from inside the subquery to the select that this predicate is - in. References to parent selects not included. + /* A reference from inside subquery predicate to somewhere outside of it */ + class Ref_to_outside : public Sql_alloc + { + public: + st_select_lex *select; /* Select where the reference is pointing to */ + /* + What is being referred. This may be NULL when we're referring to an + aggregate function. + */ + Item *item; + }; + /* + References from within this subquery to somewhere outside of it (i.e. to + parent select, grandparent select, etc) */ - List<Item> refers_to; - int in_fix_fields; + List<Ref_to_outside> upper_refs; + st_select_lex *parent_select; + + /* + TRUE<=>Table Elimination has made it redundant to evaluate this select + (and so it is not part of QEP, etc) + */ bool eliminated; /* changed engine indicator */ @@ -117,6 +135,9 @@ return null_value; } bool fix_fields(THD *thd, Item **ref); + bool mark_as_dependent(THD *thd, st_select_lex *select, Item *item); + void fix_after_pullout(st_select_lex *new_parent, Item **ref); + void recalc_used_tables(st_select_lex *new_parent, bool after_pullout); virtual bool exec(); virtual void fix_length_and_dec(); table_map used_tables() const; @@ -396,6 +417,8 @@ bool test_limit(st_select_lex_unit *unit); virtual void print(String *str, enum_query_type query_type); bool fix_fields(THD *thd, Item **ref); + void fix_after_pullout(st_select_lex *new_parent, Item **ref); + void update_used_tables(); bool setup_engine(); bool init_left_expr_cache(); bool is_expensive_processor(uchar *arg); === modified file 'sql/item_sum.cc' --- a/sql/item_sum.cc 2009-10-15 21:38:29 +0000 +++ b/sql/item_sum.cc 2010-02-11 23:59:58 +0000 @@ -350,7 +350,7 @@ sl= sl->master_unit()->outer_select() ) sl->master_unit()->item->with_sum_func= 1; } - thd->lex->current_select->mark_as_dependent(aggr_sel, NULL); + thd->lex->current_select->mark_as_dependent(thd, aggr_sel, NULL); return FALSE; } === modified file 'sql/sql_lex.cc' --- a/sql/sql_lex.cc 2010-01-28 13:48:33 +0000 +++ b/sql/sql_lex.cc 2010-02-11 23:59:58 +0000 @@ -1841,9 +1841,8 @@ 'last' should be reachable from this st_select_lex_node */ -void st_select_lex::mark_as_dependent(st_select_lex *last, Item *dependency) +bool st_select_lex::mark_as_dependent(THD *thd, st_select_lex *last, Item *dependency) { - SELECT_LEX *next_to_last; /* Mark all selects from resolved to 1 before select where was found table as depended (of select where was found table) @@ -1867,12 +1866,15 @@ sl->uncacheable|= UNCACHEABLE_UNITED; } } - next_to_last= s; + + Item_subselect *subquery_expr= s->master_unit()->item; + if (subquery_expr && subquery_expr->mark_as_dependent(thd, last, + dependency)) + return TRUE; } is_correlated= TRUE; this->master_unit()->item->is_correlated= TRUE; - if (dependency) - next_to_last->master_unit()->item->refers_to.push_back(dependency); + return FALSE; } bool st_select_lex_node::set_braces(bool value) { return 1; } === modified file 'sql/sql_lex.h' --- a/sql/sql_lex.h 2010-01-28 13:48:33 +0000 +++ b/sql/sql_lex.h 2010-02-11 23:59:58 +0000 @@ -747,7 +747,7 @@ return master_unit()->return_after_parsing(); } - void mark_as_dependent(st_select_lex *last, Item *dependency); + bool mark_as_dependent(THD *thd, st_select_lex *last, Item *dependency); bool set_braces(bool value); bool inc_in_sum_expr();

1 0

[Maria-developers] Rev 2755: Subquery optimizations: backport: enable disabled subquery code in BKA in file:///home/psergey/dev/maria-5.3-subqueries-r6/
by Sergey Petrunya 11 Feb '10

11 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r6/ ------------------------------------------------------------ revno: 2755 revision-id: psergey(a)askmonty.org-20100211223118-5fzuidow1pkubpzl parent: psergey(a)askmonty.org-20100211220036-qh3iw743tbgwpzax committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r6 timestamp: Fri 2010-02-12 01:31:18 +0300 message: Subquery optimizations: backport: enable disabled subquery code in BKA === modified file 'sql/sql_select.h' --- a/sql/sql_select.h 2010-01-28 13:48:33 +0000 +++ b/sql/sql_select.h 2010-02-11 22:31:18 +0000 @@ -282,13 +282,11 @@ } bool check_rowid_field() { -/* !!!NB igor: enable the code in this comment after backporting the SJ code if (keep_current_rowid && !used_rowid_fields) { used_rowid_fields= 1; used_fieldlength+= table->file->ref_length; } -*/ return test(used_rowid_fields); } bool is_inner_table_of_semi_join_with_first_match()

1 0

[Maria-developers] Rev 2754: Subquery optimizations: backport in file:///home/psergey/dev/maria-5.3-subqueries-r6/
by Sergey Petrunya 11 Feb '10

11 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r6/ ------------------------------------------------------------ revno: 2754 revision-id: psergey(a)askmonty.org-20100211220036-qh3iw743tbgwpzax parent: psergey(a)askmonty.org-20100211215932-qi36vl0i3zkl86bv committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r6 timestamp: Fri 2010-02-12 01:00:36 +0300 message: Subquery optimizations: backport - Fix valgrind failure: do initialize Item::is_expensive_cache. === modified file 'sql/item.cc' --- a/sql/item.cc 2010-01-17 14:55:08 +0000 +++ b/sql/item.cc 2010-02-11 22:00:36 +0000 @@ -373,8 +373,8 @@ Item::Item(): - rsize(0), name(0), orig_name(0), name_length(0), fixed(0), - is_autogenerated_name(TRUE), + is_expensive_cache(-1), rsize(0), name(0), orig_name(0), name_length(0), + fixed(0), is_autogenerated_name(TRUE), collation(&my_charset_bin, DERIVATION_COERCIBLE) { marker= 0; @@ -410,6 +410,7 @@ tables. */ Item::Item(THD *thd, Item *item): + is_expensive_cache(-1), rsize(0), str_value(item->str_value), name(item->name), === modified file 'sql/item.h' --- a/sql/item.h 2010-01-28 13:48:33 +0000 +++ b/sql/item.h 2010-02-11 22:00:36 +0000 @@ -513,6 +513,9 @@ enum traverse_order { POSTFIX, PREFIX }; + /* Cache of the result of is_expensive(). */ + int8 is_expensive_cache; + /* Reuse size, only used by SP local variable assignment, otherwize 0 */ uint rsize; @@ -878,9 +881,6 @@ static CHARSET_INFO *default_charset(); virtual CHARSET_INFO *compare_collation() { return NULL; } - /* Cache of the result of is_expensive(). */ - int8 is_expensive_cache; - virtual bool walk(Item_processor processor, bool walk_subquery, uchar *arg) { return (this->*processor)(arg);

1 0

[Maria-developers] Rev 2753: Subquery optimizations backport: Update test results (checked) in file:///home/psergey/dev/maria-5.3-subqueries-r6/
by Sergey Petrunya 11 Feb '10

11 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r6/ ------------------------------------------------------------ revno: 2753 revision-id: psergey(a)askmonty.org-20100211215932-qi36vl0i3zkl86bv parent: psergey(a)askmonty.org-20100211215823-63ikirl70ztmlk05 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r6 timestamp: Fri 2010-02-12 00:59:32 +0300 message: Subquery optimizations backport: Update test results (checked) === modified file 'mysql-test/r/join_cache.result' --- a/mysql-test/r/join_cache.result 2009-12-21 02:26:15 +0000 +++ b/mysql-test/r/join_cache.result 2010-02-11 21:59:32 +0000 @@ -1028,8 +1028,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -1343,8 +1343,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -1658,8 +1658,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -1973,8 +1973,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -2292,8 +2292,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -2514,8 +2514,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -2736,8 +2736,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -2958,8 +2958,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; === modified file 'mysql-test/r/type_datetime.result' --- a/mysql-test/r/type_datetime.result 2009-02-13 18:07:03 +0000 +++ b/mysql-test/r/type_datetime.result 2010-02-11 21:59:32 +0000 @@ -514,10 +514,9 @@ where id in (select id from t1 as x1 where (t1.cur_date is null)); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables -2 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE Warnings: Note 1276 Field or reference 'test.t1.cur_date' of SELECT #2 was resolved in SELECT #1 -Note 1003 select '1' AS `id`,'2007-04-25 18:30:22' AS `cur_date` from `test`.`t1` where <in_optimizer>('1',<exists>(select 1 AS `Not_used` from `test`.`t1` `x1` where 0)) +Note 1003 select '1' AS `id`,'2007-04-25 18:30:22' AS `cur_date` from `test`.`t1` `x1` join `test`.`t1` where (('2007-04-25 18:30:22' = 0)) select * from t1 where id in (select id from t1 as x1 where (t1.cur_date is null)); id cur_date @@ -526,10 +525,9 @@ where id in (select id from t2 as x1 where (t2.cur_date is null)); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables -2 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE Warnings: Note 1276 Field or reference 'test.t2.cur_date' of SELECT #2 was resolved in SELECT #1 -Note 1003 select '1' AS `id`,'2007-04-25' AS `cur_date` from `test`.`t2` where <in_optimizer>('1',<exists>(select 1 AS `Not_used` from `test`.`t2` `x1` where 0)) +Note 1003 select '1' AS `id`,'2007-04-25' AS `cur_date` from `test`.`t2` `x1` join `test`.`t2` where (('2007-04-25' = 0)) select * from t2 where id in (select id from t2 as x1 where (t2.cur_date is null)); id cur_date @@ -540,10 +538,10 @@ where id in (select id from t1 as x1 where (t1.cur_date is null)); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 ALL NULL NULL NULL NULL 2 100.00 Using where -2 DEPENDENT SUBQUERY x1 ALL NULL NULL NULL NULL 2 100.00 Using where +1 PRIMARY x1 ALL NULL NULL NULL NULL 2 100.00 Using where; FirstMatch(t1) Warnings: Note 1276 Field or reference 'test.t1.cur_date' of SELECT #2 was resolved in SELECT #1 -Note 1003 select `test`.`t1`.`id` AS `id`,`test`.`t1`.`cur_date` AS `cur_date` from `test`.`t1` where <in_optimizer>(`test`.`t1`.`id`,<exists>(select 1 AS `Not_used` from `test`.`t1` `x1` where ((`test`.`t1`.`cur_date` = 0) and (<cache>(`test`.`t1`.`id`) = `test`.`x1`.`id`)))) +Note 1003 select `test`.`t1`.`id` AS `id`,`test`.`t1`.`cur_date` AS `cur_date` from `test`.`t1` semi join (`test`.`t1` `x1`) where ((`test`.`x1`.`id` = `test`.`t1`.`id`) and (`test`.`t1`.`cur_date` = 0)) select * from t1 where id in (select id from t1 as x1 where (t1.cur_date is null)); id cur_date @@ -552,10 +550,10 @@ where id in (select id from t2 as x1 where (t2.cur_date is null)); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t2 ALL NULL NULL NULL NULL 2 100.00 Using where -2 DEPENDENT SUBQUERY x1 ALL NULL NULL NULL NULL 2 100.00 Using where +1 PRIMARY x1 ALL NULL NULL NULL NULL 2 100.00 Using where; FirstMatch(t2) Warnings: Note 1276 Field or reference 'test.t2.cur_date' of SELECT #2 was resolved in SELECT #1 -Note 1003 select `test`.`t2`.`id` AS `id`,`test`.`t2`.`cur_date` AS `cur_date` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`id`,<exists>(select 1 AS `Not_used` from `test`.`t2` `x1` where ((`test`.`t2`.`cur_date` = 0) and (<cache>(`test`.`t2`.`id`) = `test`.`x1`.`id`)))) +Note 1003 select `test`.`t2`.`id` AS `id`,`test`.`t2`.`cur_date` AS `cur_date` from `test`.`t2` semi join (`test`.`t2` `x1`) where ((`test`.`x1`.`id` = `test`.`t2`.`id`) and (`test`.`t2`.`cur_date` = 0)) select * from t2 where id in (select id from t2 as x1 where (t2.cur_date is null)); id cur_date

1 0

[Maria-developers] Rev 2752: Apply Jorgen Loland's fix: Bug#45221: Query "SELECT pk FROM C WHERE pk IN (SELECT int_key)" failing in file:///home/psergey/dev/maria-5.3-subqueries-r6/
by Sergey Petrunya 11 Feb '10

11 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r6/ ------------------------------------------------------------ revno: 2752 revision-id: psergey(a)askmonty.org-20100211215823-63ikirl70ztmlk05 parent: psergey(a)askmonty.org-20100211215602-irdyu314ddwew1xd committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r6 timestamp: Fri 2010-02-12 00:58:23 +0300 message: Apply Jorgen Loland's fix: Bug#45221: Query "SELECT pk FROM C WHERE pk IN (SELECT int_key)" failing XOR conditions are not optimized, and Item_cond_xor therefore acts like type Func_item even though it inherits from Item_cond. A subtle difference between Item_func and Item_cond is that you can get the children Items from the former by calling arguments(), and from the latter by calling argument_list(). However, since Item_cond_xor inherits from Item_cond, arguments() did not return any Items. The fact that Item_cond_xor::arguments() did not return it's children items lead to a problem for make_cond_for_index(); the method accepted that XOR items on unindexed columns were pushed using ICP. ICP evaluation of non-indexed columns does not (and should not) work. The fix for this bug is to make Item_cond_xor return it's children items when the arguments() method is used. This makes Item_cond_xor behave more like Item_func and in turn allows make_cond_for_index() to discover any conflicting children Items. This is a temporary fix and should be removed when Item_cond_xor is optimized. === modified file 'mysql-test/r/group_by.result' --- a/mysql-test/r/group_by.result 2009-02-26 17:17:06 +0000 +++ b/mysql-test/r/group_by.result 2010-02-11 21:58:23 +0000 @@ -1542,8 +1542,8 @@ EXPLAIN SELECT 1 FROM t1 WHERE a IN (SELECT a FROM t1 USE INDEX (i2) IGNORE INDEX (i2)); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 index NULL PRIMARY 4 NULL 144 Using where; Using index -2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 144 Using where +1 PRIMARY t1 index PRIMARY,i2 PRIMARY 4 NULL 144 Using index +1 PRIMARY t1 ALL NULL NULL NULL NULL 144 Using where; FirstMatch(t1) CREATE TABLE t2 (a INT, b INT, KEY(a)); INSERT INTO t2 VALUES (1, 1), (2, 2), (3,3), (4,4); EXPLAIN SELECT a, SUM(b) FROM t2 GROUP BY a LIMIT 2; @@ -1555,8 +1555,8 @@ EXPLAIN SELECT 1 FROM t2 WHERE a IN (SELECT a FROM t1 USE INDEX (i2) IGNORE INDEX (i2)); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t2 index NULL a 5 NULL 4 Using where; Using index -2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 144 Using where +1 PRIMARY t2 index a a 5 NULL 4 Using index +1 PRIMARY t1 ALL NULL NULL NULL NULL 144 Using where; FirstMatch(t2) SHOW VARIABLES LIKE 'old'; Variable_name Value old OFF === modified file 'sql/item_cmpfunc.h' --- a/sql/item_cmpfunc.h 2010-01-17 14:55:08 +0000 +++ b/sql/item_cmpfunc.h 2010-02-11 21:58:23 +0000 @@ -1715,14 +1715,34 @@ class Item_cond_xor :public Item_cond { public: - Item_cond_xor() :Item_cond() {} - Item_cond_xor(Item *i1,Item *i2) :Item_cond(i1,i2) {} + Item_cond_xor(Item *i1,Item *i2) :Item_cond(i1,i2) + { + /* + Items must be stored in args[] as well because this Item_cond is + treated as a FUNC_ITEM (see type()). I.e., users of it will get + it's children by calling arguments(), not argument_list(). This + is a temporary solution until XOR is optimized and treated like + a full Item_cond citizen. + */ + arg_count= 2; + args= tmp_arg; + args[0]= i1; + args[1]= i2; + } enum Functype functype() const { return COND_XOR_FUNC; } /* TODO: remove the next line when implementing XOR optimization */ enum Type type() const { return FUNC_ITEM; } longlong val_int(); const char *func_name() const { return "xor"; } void top_level_item() {} + /* Since child Items are stored in args[], Items cannot be added. + However, since Item_cond_xor is treated as a FUNC_ITEM (see + type()), the methods below should never be called. + */ + bool add(Item *item) { DBUG_ASSERT(FALSE); return FALSE; } + bool add_at_head(Item *item) { DBUG_ASSERT(FALSE); return FALSE; } + bool add_at_head(List<Item> *nlist) { DBUG_ASSERT(FALSE); return FALSE; } + void copy_andor_arguments(THD *thd, Item_cond *item) { DBUG_ASSERT(FALSE); } };

1 0

[Maria-developers] Rev 2751: Subquery backport: Update test results (checked) in file:///home/psergey/dev/maria-5.3-subqueries-r6/
by Sergey Petrunya 11 Feb '10

11 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r6/ ------------------------------------------------------------ revno: 2751 revision-id: psergey(a)askmonty.org-20100211215602-irdyu314ddwew1xd parent: psergey(a)askmonty.org-20100211215456-u85owf67gwqkkss5 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r6 timestamp: Fri 2010-02-12 00:56:02 +0300 message: Subquery backport: Update test results (checked) === modified file 'mysql-test/r/explain.result' --- a/mysql-test/r/explain.result 2009-12-15 07:16:46 +0000 +++ b/mysql-test/r/explain.result 2010-02-11 21:56:02 +0000 @@ -171,7 +171,7 @@ EXPLAIN SELECT OUTR.dt FROM t1 AS OUTR WHERE OUTR.dt IN (SELECT INNR.dt FROM t2 AS INNR WHERE OUTR.dt IS NULL ); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY OUTR ALL NULL NULL NULL NULL 2 Using where -2 DEPENDENT SUBQUERY INNR ALL NULL NULL NULL NULL 2 Using where +1 PRIMARY INNR ALL NULL NULL NULL NULL 2 Using where; FirstMatch(OUTR) flush tables; SELECT OUTR.dt FROM t1 AS OUTR WHERE OUTR.dt IN (SELECT INNR.dt FROM t2 AS INNR WHERE OUTR.dt IS NULL ); dt @@ -179,7 +179,7 @@ EXPLAIN SELECT OUTR.dt FROM t1 AS OUTR WHERE OUTR.dt IN ( SELECT INNR.dt FROM t2 AS INNR WHERE OUTR.t < '2005-11-13 7:41:31' ); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY OUTR ALL NULL NULL NULL NULL 2 Using where -2 DEPENDENT SUBQUERY INNR ALL NULL NULL NULL NULL 2 Using where +1 PRIMARY INNR ALL NULL NULL NULL NULL 2 Using where; FirstMatch(OUTR) flush tables; SELECT OUTR.dt FROM t1 AS OUTR WHERE OUTR.dt IN ( SELECT INNR.dt FROM t2 AS INNR WHERE OUTR.t < '2005-11-13 7:41:31' ); dt === modified file 'mysql-test/r/group_min_max.result' --- a/mysql-test/r/group_min_max.result 2009-08-30 07:03:37 +0000 +++ b/mysql-test/r/group_min_max.result 2010-02-11 21:56:02 +0000 @@ -2256,7 +2256,7 @@ a IN (SELECT max(b) FROM t1 GROUP BY a HAVING a < 2); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t1_outer index NULL a 10 NULL 15 Using where; Using index -2 DEPENDENT SUBQUERY t1 index NULL a 10 NULL 1 Using index +2 SUBQUERY t1 range NULL a 5 NULL 8 Using index for group-by EXPLAIN SELECT 1 FROM t1 AS t1_outer GROUP BY a HAVING a > (SELECT max(b) FROM t1 GROUP BY a HAVING a < 2); id select_type table type possible_keys key key_len ref rows Extra === modified file 'mysql-test/r/subselect3_jcl6.result' --- a/mysql-test/r/subselect3_jcl6.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect3_jcl6.result 2010-02-11 21:56:02 +0000 @@ -1140,7 +1140,7 @@ flush status; select count(*) from t0 A, t0 B, t0 C, t0 D where D.a in (select a from t1 E); count(*) -4999 +5000 show status like 'Created_tmp_disk_tables'; Variable_name Value Created_tmp_disk_tables 1

1 0

[Maria-developers] Rev 2750: Subquery optimization backport: Duplicate Elimination: in file:///home/psergey/dev/maria-5.3-subqueries-r6/
by Sergey Petrunya 11 Feb '10

11 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r6/ ------------------------------------------------------------ revno: 2750 revision-id: psergey(a)askmonty.org-20100211215456-u85owf67gwqkkss5 parent: psergey(a)askmonty.org-20100128134833-9000udjp5wa3tsff committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r6 timestamp: Fri 2010-02-12 00:54:56 +0300 message: Subquery optimization backport: Duplicate Elimination: process temporary table overflow correctly. === modified file 'sql/sql_select.cc' --- a/sql/sql_select.cc 2010-01-28 13:48:33 +0000 +++ b/sql/sql_select.cc 2010-02-11 21:54:56 +0000 @@ -16250,12 +16250,12 @@ if (error) { /* create_internal_tmp_table_from_heap will generate error if needed */ - if (sjtbl->tmp_table->file->is_fatal_error(error, HA_CHECK_DUP) && - create_internal_tmp_table_from_heap(thd, sjtbl->tmp_table, + if (!sjtbl->tmp_table->file->is_fatal_error(error, HA_CHECK_DUP)) + DBUG_RETURN(1); /* Duplicate */ + if (create_internal_tmp_table_from_heap(thd, sjtbl->tmp_table, sjtbl->start_recinfo, &sjtbl->recinfo, error, 1)) DBUG_RETURN(-1); - DBUG_RETURN(1); } DBUG_RETURN(0); }

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2816: Fix for LPBug#520243: useability bug of thread pool configuration
by noreply＠launchpad.net 11 Feb '10

11 Feb '10

------------------------------------------------------------ revno: 2816 committer: Michael Widenius <monty(a)askmonty.org> branch nick: maria-5.1 timestamp: Thu 2010-02-11 21:15:24 +0200 message: Fix for LPBug#520243: useability bug of thread pool configuration Now mysqld --help --verbose shows the value for thread-handling Fixed also that mysqld --one-thread works as expected. modified: sql/mysqld.cc -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2816)
by Michael Widenius 11 Feb '10

11 Feb '10

#At lp:maria based on revid:monty@askmonty.org-20100210212606-xj84sp2fhbrf5epc 2816 Michael Widenius 2010-02-11 Fix for LPBug#520243: useability bug of thread pool configuration Now mysqld --help --verbose shows the value for thread-handling Fixed also that mysqld --one-thread works as expected. modified: sql/mysqld.cc === modified file 'sql/mysqld.cc' --- a/sql/mysqld.cc 2010-02-10 19:06:24 +0000 +++ b/sql/mysqld.cc 2010-02-11 19:15:24 +0000 @@ -598,6 +598,7 @@ char *mysqld_unix_port, *opt_mysql_tmpdi const char **errmesg; /**< Error messages */ const char *myisam_recover_options_str="OFF"; const char *myisam_stats_method_str="nulls_unequal"; +const char *opt_thread_handling= thread_handling_typelib.type_names[0]; /** name of reference on left espression in rewritten IN subquery */ const char *in_left_expr_name= "<left expr>"; @@ -7290,7 +7291,8 @@ The minimum value for this variable is 4 1024, 0}, {"thread_handling", OPT_THREAD_HANDLING, "Define threads usage for handling queries: " - "one-thread-per-connection or no-threads", 0, 0, + "one-thread-per-connection or no-threads", + (uchar**) &opt_thread_handling, (uchar**) &opt_thread_handling, 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, {"updatable_views_with_limit", OPT_UPDATABLE_VIEWS_WITH_LIMIT, "1 = YES = Don't issue an error message (warning only) if a VIEW without presence of a key of the underlying table is used in queries with a LIMIT clause for updating. 0 = NO = Prohibit update of a VIEW, which does not contain a key of the underlying table and the query uses a LIMIT clause (usually get from GUI tools).", @@ -8721,14 +8723,15 @@ mysqld_get_one_option(int optid, break; } case OPT_ONE_THREAD: - global_system_variables.thread_handling= - SCHEDULER_ONE_THREAD_PER_CONNECTION; + global_system_variables.thread_handling= SCHEDULER_NO_THREADS; + opt_thread_handling= thread_handling_typelib.type_names[global_system_variables.thread_handling]; break; case OPT_THREAD_HANDLING: { int id; if (!find_opt_type(argument, &thread_handling_typelib, opt->name, &id)) global_system_variables.thread_handling= id - 1; + opt_thread_handling= thread_handling_typelib.type_names[global_system_variables.thread_handling]; break; } case OPT_FT_BOOLEAN_SYNTAX:

1 0

[Maria-developers] Updated (by Monty): Add support for dynamic columns (via google protocol buffers) (34)
by worklog-noreply＠askmonty.org 11 Feb '10

11 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add support for dynamic columns (via google protocol buffers) CREATION DATE..: Tue, 21 Jul 2009, 21:11 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 34 (http://askmonty.org/worklog/?tid=34) VERSION........: Server-5.3 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Monty - Thu, 11 Feb 2010, 20:04)=-=- Version updated. --- /tmp/wklog.34.old.18409 2010-02-11 18:04:35.000000000 +0000 +++ /tmp/wklog.34.new.18409 2010-02-11 18:04:35.000000000 +0000 @@ -1 +1 @@ -WorkLog-3.4 +Server-5.3 -=-=(Monty - Thu, 11 Feb 2010, 20:04)=-=- Status updated. --- /tmp/wklog.34.old.18409 2010-02-11 18:04:35.000000000 +0000 +++ /tmp/wklog.34.new.18409 2010-02-11 18:04:35.000000000 +0000 @@ -1 +1 @@ -Un-Assigned +Assigned -=-=(Monty - Thu, 11 Feb 2010, 20:04)=-=- Title modified. --- /tmp/wklog.34.old.18409 2010-02-11 18:04:35.000000000 +0000 +++ /tmp/wklog.34.new.18409 2010-02-11 18:04:35.000000000 +0000 @@ -1 +1 @@ -Add support for google protocol buffers +Add support for dynamic columns (via google protocol buffers) -=-=(Monty - Thu, 11 Feb 2010, 20:03)=-=- High Level Description modified. --- /tmp/wklog.34.old.18329 2010-02-11 18:03:42.000000000 +0000 +++ /tmp/wklog.34.new.18329 2010-02-11 18:03:42.000000000 +0000 @@ -19,3 +19,14 @@ Any support for indexing GPB data is outside of scope of this WL entry. +Example usage: + +SELECT proto_get(blob, 1, varchar) from table_with_proto; + +UPDATE table_with_proto SET blob=proto_add(blob, 2, "hello") where id=1; + +UPDATE table_with_proto SET blob=proto_del(blob,4) where id=5; + +Note that 'proto_add()' will replace any old value with the given proto_id. + + -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.17962 2010-02-11 19:59:45.000000000 +0200 +++ /tmp/wklog.34.new.17962 2010-02-11 19:59:45.000000000 +0200 @@ -1,13 +1,8 @@ - -<contents> 1. GPB Encoding overview 2. GPB in an SQL database -2.1 Informing server about GPB field names and types -2.2 Addressing GPB fields -2.2.1 Option1: SQL Function -2.2.2 Option2: SQL columns -</contents> - +3. Encoding to use for dynamic columns +4. How to store and access data in a protocol buffer from SQL +5. Extensions for the future 1. GPB Encoding overview ======================== @@ -37,42 +32,50 @@ traffic right away, and will open path to getting the best possible performance. -2.1 Informing server about GPB field names and types ----------------------------------------------------- -User-friendly/meaningful access to GPB fields requires knowledge of GPB field -names and types, which are not available from GPB message itself (see "GPB -encoding overview" section). - -So the first issue to be addressed is to get the server to know the definition -of stored messages. We intend to assume that all records have GPB messages -that conform to a certain single definition, which gives one definition per -GPB field. +3. Encoding to use for dynamic columns +====================================== -DecisionToMake: How to pass the server the GPB definition? -First idea: add a CREATE TABLE parameter which will specify either the -definition itself or path to .proto file with the definition. +The data should be coded into the proto buffer in the following format: + +<field_number><value_type><value>[<field_number><value_type><value>...] + +Where field_number is a number between 0-65536 that identifes the field +<value_type> is a enum of type 'Item_result' +<value> is the value coded in proto format. + +In other words, we should have no nested or complex structure. + +4. How to store and access data in a protocol buffer from SQL +============================================================ + +User-friendly/meaningful access to GPB fields requires knowledge of +GPB field names and types, which are not available from GPB message +itself (see "GPB encoding overview" section). + +To make things easy for the user, we will at first stage provide SQL +functions to manipulate a string that is actually in proto format. -2.2 Addressing GPB fields -------------------------- -We'll need to provide a way to access GPB fields. This can be complicated as -structures that are encoded in GPB message can be nested and recursive. - -2.2.1 Option1: SQL Function -~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Introduce an SQL function GPB_FIELD(path) which will return contents of the -field. -- Return type of the function will be determined from GPB message definition. -- For path, we can use XPath selector (a subset of XPath) syntax. - -(TODO ^ the above needs to be specified in more detail. is the selector as -simple as filesystem path or we allow quantifiers (with predicates?)?) - -2.2.2 Option2: SQL columns -~~~~~~~~~~~~~~~~~~~~~~~~~~ -Make GPB columns to be accessible as SQL columns. -This approach has problems: -- It might be hard to implement code-wise - - (TODO will Virtual columns patch help??) -- It is not clear how to access fields from nested structures. Should we allow - quoted names like `foo/bar[2]/baz' ? +The functions we should provde are: +proto_get(gpb, field_number, type) + +This return the field tagged with 'field_number' from the 'gpb' buffer. + +Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar + +proto_put(gpb, field_number, value) + +This returns a new gbp buffer with the new value appended. + +Example: proto_put(proto_put(blob, 1, 1), 2, "hello") + +5. Extension for future +======================= + +In the future we may want to access data based on name and get MariaDB to +automaticly know the correct type. To do this we need to be able to +store a definition for the content of the proto buffer somewhere. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify the +definition itself. -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High Level Description modified. --- /tmp/wklog.34.old.17915 2010-02-11 17:59:17.000000000 +0000 +++ /tmp/wklog.34.new.17915 2010-02-11 17:59:17.000000000 +0000 @@ -1,5 +1,21 @@ -Add support for Google Protocol Buffers (further GPB). It should be possible -to have columns that store GPB-encoded data, as well as use SQL constructs to +Add support for dynamic columns: + +- A column that can hold information from many columns +- One can instantly add or remove column data + +This is a useful feature for any store type of application, where you want to +store different type of information for different kind of items. + +For example, for shoes you want to store: material, size, colour, maker +For a computer you want to store ram, hard disk size etc... + +In a normal 'relational' system you would need to a table for each type. +With dynamic columns you have all common items as fixed fields (like +product_code, manufacturer, price) and the rest stored in a dynamic column. + +The proposed idea is to store the dynamic information in a blob in +Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. + Any support for indexing GPB data is outside of scope of this WL entry. -=-=(Knielsen - Fri, 22 Jan 2010, 11:38)=-=- Low Level Design modified. --- /tmp/wklog.34.old.29965 2010-01-22 11:38:57.000000000 +0200 +++ /tmp/wklog.34.new.29965 2010-01-22 11:38:57.000000000 +0200 @@ -2,3 +2,12 @@ and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. +* We should have both server-side support and client-side support (client side + means functions in libmysqlclient so that user can select the full BLOB and + extract fields in the application). + +* Add some kind of header to the GPB blob to support versioning and future + extensibility. + +* Add complete syntax description (update, add, drop, exists, ...). + -=-=(Psergey - Tue, 21 Jul 2009, 21:13)=-=- Low Level Design modified. --- /tmp/wklog.34.old.6462 2009-07-21 21:13:13.000000000 +0300 +++ /tmp/wklog.34.new.6462 2009-07-21 21:13:13.000000000 +0300 @@ -1 +1,4 @@ +* GPB tarball contains a protocol definition for .proto file structure itself + and a parser for text form of .proto file which then exposes the parsed + file via standard GPB message navigation API. -=-=(Psergey - Tue, 21 Jul 2009, 21:12)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.6399 2009-07-21 21:12:23.000000000 +0300 +++ /tmp/wklog.34.new.6399 2009-07-21 21:12:23.000000000 +0300 @@ -1 +1,78 @@ +<contents> +1. GPB Encoding overview +2. GPB in an SQL database +2.1 Informing server about GPB field names and types +2.2 Addressing GPB fields +2.2.1 Option1: SQL Function +2.2.2 Option2: SQL columns +</contents> + + +1. GPB Encoding overview +======================== + +GBB is a compact encoding for structured and typed data. A unit of GPB data +(it is called message) is only partially self-describing: it's possible to +iterate over its parts, but, quoting the spec + +http://code.google.com/apis/protocolbuffers/docs/encoding.html: + " the name and declared type for each field can only be determined on the + decoding end by referencing the message type's definition (i.e. the .proto + file). " + +2. GPB in an SQL database +========================= + +It is possible to store GPB data in MariaDB today - one can declare a binary +blob column and use it to store GPB messages. Storing and retrieving entire +messages will be the only available operations, though, as the server has no +idea about the GPB format. +It is apparent that ability to peek inside GPB data from SQL layer would be of +great advantage: one would be able to +- select only certain fields or parts of GPB messages +- filter records based on the values of GPB fields +- etc +performing such operations at SQL layer will allow to reduce client<->server +traffic right away, and will open path to getting the best possible +performance. + +2.1 Informing server about GPB field names and types +---------------------------------------------------- +User-friendly/meaningful access to GPB fields requires knowledge of GPB field +names and types, which are not available from GPB message itself (see "GPB +encoding overview" section). + +So the first issue to be addressed is to get the server to know the definition +of stored messages. We intend to assume that all records have GPB messages +that conform to a certain single definition, which gives one definition per +GPB field. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify either the +definition itself or path to .proto file with the definition. + +2.2 Addressing GPB fields +------------------------- +We'll need to provide a way to access GPB fields. This can be complicated as +structures that are encoded in GPB message can be nested and recursive. + +2.2.1 Option1: SQL Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Introduce an SQL function GPB_FIELD(path) which will return contents of the +field. +- Return type of the function will be determined from GPB message definition. +- For path, we can use XPath selector (a subset of XPath) syntax. + +(TODO ^ the above needs to be specified in more detail. is the selector as +simple as filesystem path or we allow quantifiers (with predicates?)?) + +2.2.2 Option2: SQL columns +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make GPB columns to be accessible as SQL columns. +This approach has problems: +- It might be hard to implement code-wise + - (TODO will Virtual columns patch help??) +- It is not clear how to access fields from nested structures. Should we allow + quoted names like `foo/bar[2]/baz' ? + DESCRIPTION: Add support for dynamic columns: - A column that can hold information from many columns - One can instantly add or remove column data This is a useful feature for any store type of application, where you want to store different type of information for different kind of items. For example, for shoes you want to store: material, size, colour, maker For a computer you want to store ram, hard disk size etc... In a normal 'relational' system you would need to a table for each type. With dynamic columns you have all common items as fixed fields (like product_code, manufacturer, price) and the rest stored in a dynamic column. The proposed idea is to store the dynamic information in a blob in Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. Any support for indexing GPB data is outside of scope of this WL entry. Example usage: SELECT proto_get(blob, 1, varchar) from table_with_proto; UPDATE table_with_proto SET blob=proto_add(blob, 2, "hello") where id=1; UPDATE table_with_proto SET blob=proto_del(blob,4) where id=5; Note that 'proto_add()' will replace any old value with the given proto_id. HIGH-LEVEL SPECIFICATION: 1. GPB Encoding overview 2. GPB in an SQL database 3. Encoding to use for dynamic columns 4. How to store and access data in a protocol buffer from SQL 5. Extensions for the future 1. GPB Encoding overview ======================== GBB is a compact encoding for structured and typed data. A unit of GPB data (it is called message) is only partially self-describing: it's possible to iterate over its parts, but, quoting the spec http://code.google.com/apis/protocolbuffers/docs/encoding.html: " the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file). " 2. GPB in an SQL database ========================= It is possible to store GPB data in MariaDB today - one can declare a binary blob column and use it to store GPB messages. Storing and retrieving entire messages will be the only available operations, though, as the server has no idea about the GPB format. It is apparent that ability to peek inside GPB data from SQL layer would be of great advantage: one would be able to - select only certain fields or parts of GPB messages - filter records based on the values of GPB fields - etc performing such operations at SQL layer will allow to reduce client<->server traffic right away, and will open path to getting the best possible performance. 3. Encoding to use for dynamic columns ====================================== The data should be coded into the proto buffer in the following format: <field_number><value_type><value>[<field_number><value_type><value>...] Where field_number is a number between 0-65536 that identifes the field <value_type> is a enum of type 'Item_result' <value> is the value coded in proto format. In other words, we should have no nested or complex structure. 4. How to store and access data in a protocol buffer from SQL ============================================================ User-friendly/meaningful access to GPB fields requires knowledge of GPB field names and types, which are not available from GPB message itself (see "GPB encoding overview" section). To make things easy for the user, we will at first stage provide SQL functions to manipulate a string that is actually in proto format. The functions we should provde are: proto_get(gpb, field_number, type) This return the field tagged with 'field_number' from the 'gpb' buffer. Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar proto_put(gpb, field_number, value) This returns a new gbp buffer with the new value appended. Example: proto_put(proto_put(blob, 1, 1), 2, "hello") 5. Extension for future ======================= In the future we may want to access data based on name and get MariaDB to automaticly know the correct type. To do this we need to be able to store a definition for the content of the proto buffer somewhere. DecisionToMake: How to pass the server the GPB definition? First idea: add a CREATE TABLE parameter which will specify the definition itself. LOW-LEVEL DESIGN: * GPB tarball contains a protocol definition for .proto file structure itself and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. * We should have both server-side support and client-side support (client side means functions in libmysqlclient so that user can select the full BLOB and extract fields in the application). * Add some kind of header to the GPB blob to support versioning and future extensibility. * Add complete syntax description (update, add, drop, exists, ...). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Monty): Add support for dynamic columns (via google protocol buffers) (34)
by worklog-noreply＠askmonty.org 11 Feb '10

11 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add support for dynamic columns (via google protocol buffers) CREATION DATE..: Tue, 21 Jul 2009, 21:11 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 34 (http://askmonty.org/worklog/?tid=34) VERSION........: Server-5.3 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Monty - Thu, 11 Feb 2010, 20:04)=-=- Version updated. --- /tmp/wklog.34.old.18409 2010-02-11 18:04:35.000000000 +0000 +++ /tmp/wklog.34.new.18409 2010-02-11 18:04:35.000000000 +0000 @@ -1 +1 @@ -WorkLog-3.4 +Server-5.3 -=-=(Monty - Thu, 11 Feb 2010, 20:04)=-=- Status updated. --- /tmp/wklog.34.old.18409 2010-02-11 18:04:35.000000000 +0000 +++ /tmp/wklog.34.new.18409 2010-02-11 18:04:35.000000000 +0000 @@ -1 +1 @@ -Un-Assigned +Assigned -=-=(Monty - Thu, 11 Feb 2010, 20:04)=-=- Title modified. --- /tmp/wklog.34.old.18409 2010-02-11 18:04:35.000000000 +0000 +++ /tmp/wklog.34.new.18409 2010-02-11 18:04:35.000000000 +0000 @@ -1 +1 @@ -Add support for google protocol buffers +Add support for dynamic columns (via google protocol buffers) -=-=(Monty - Thu, 11 Feb 2010, 20:03)=-=- High Level Description modified. --- /tmp/wklog.34.old.18329 2010-02-11 18:03:42.000000000 +0000 +++ /tmp/wklog.34.new.18329 2010-02-11 18:03:42.000000000 +0000 @@ -19,3 +19,14 @@ Any support for indexing GPB data is outside of scope of this WL entry. +Example usage: + +SELECT proto_get(blob, 1, varchar) from table_with_proto; + +UPDATE table_with_proto SET blob=proto_add(blob, 2, "hello") where id=1; + +UPDATE table_with_proto SET blob=proto_del(blob,4) where id=5; + +Note that 'proto_add()' will replace any old value with the given proto_id. + + -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.17962 2010-02-11 19:59:45.000000000 +0200 +++ /tmp/wklog.34.new.17962 2010-02-11 19:59:45.000000000 +0200 @@ -1,13 +1,8 @@ - -<contents> 1. GPB Encoding overview 2. GPB in an SQL database -2.1 Informing server about GPB field names and types -2.2 Addressing GPB fields -2.2.1 Option1: SQL Function -2.2.2 Option2: SQL columns -</contents> - +3. Encoding to use for dynamic columns +4. How to store and access data in a protocol buffer from SQL +5. Extensions for the future 1. GPB Encoding overview ======================== @@ -37,42 +32,50 @@ traffic right away, and will open path to getting the best possible performance. -2.1 Informing server about GPB field names and types ----------------------------------------------------- -User-friendly/meaningful access to GPB fields requires knowledge of GPB field -names and types, which are not available from GPB message itself (see "GPB -encoding overview" section). - -So the first issue to be addressed is to get the server to know the definition -of stored messages. We intend to assume that all records have GPB messages -that conform to a certain single definition, which gives one definition per -GPB field. +3. Encoding to use for dynamic columns +====================================== -DecisionToMake: How to pass the server the GPB definition? -First idea: add a CREATE TABLE parameter which will specify either the -definition itself or path to .proto file with the definition. +The data should be coded into the proto buffer in the following format: + +<field_number><value_type><value>[<field_number><value_type><value>...] + +Where field_number is a number between 0-65536 that identifes the field +<value_type> is a enum of type 'Item_result' +<value> is the value coded in proto format. + +In other words, we should have no nested or complex structure. + +4. How to store and access data in a protocol buffer from SQL +============================================================ + +User-friendly/meaningful access to GPB fields requires knowledge of +GPB field names and types, which are not available from GPB message +itself (see "GPB encoding overview" section). + +To make things easy for the user, we will at first stage provide SQL +functions to manipulate a string that is actually in proto format. -2.2 Addressing GPB fields -------------------------- -We'll need to provide a way to access GPB fields. This can be complicated as -structures that are encoded in GPB message can be nested and recursive. - -2.2.1 Option1: SQL Function -~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Introduce an SQL function GPB_FIELD(path) which will return contents of the -field. -- Return type of the function will be determined from GPB message definition. -- For path, we can use XPath selector (a subset of XPath) syntax. - -(TODO ^ the above needs to be specified in more detail. is the selector as -simple as filesystem path or we allow quantifiers (with predicates?)?) - -2.2.2 Option2: SQL columns -~~~~~~~~~~~~~~~~~~~~~~~~~~ -Make GPB columns to be accessible as SQL columns. -This approach has problems: -- It might be hard to implement code-wise - - (TODO will Virtual columns patch help??) -- It is not clear how to access fields from nested structures. Should we allow - quoted names like `foo/bar[2]/baz' ? +The functions we should provde are: +proto_get(gpb, field_number, type) + +This return the field tagged with 'field_number' from the 'gpb' buffer. + +Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar + +proto_put(gpb, field_number, value) + +This returns a new gbp buffer with the new value appended. + +Example: proto_put(proto_put(blob, 1, 1), 2, "hello") + +5. Extension for future +======================= + +In the future we may want to access data based on name and get MariaDB to +automaticly know the correct type. To do this we need to be able to +store a definition for the content of the proto buffer somewhere. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify the +definition itself. -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High Level Description modified. --- /tmp/wklog.34.old.17915 2010-02-11 17:59:17.000000000 +0000 +++ /tmp/wklog.34.new.17915 2010-02-11 17:59:17.000000000 +0000 @@ -1,5 +1,21 @@ -Add support for Google Protocol Buffers (further GPB). It should be possible -to have columns that store GPB-encoded data, as well as use SQL constructs to +Add support for dynamic columns: + +- A column that can hold information from many columns +- One can instantly add or remove column data + +This is a useful feature for any store type of application, where you want to +store different type of information for different kind of items. + +For example, for shoes you want to store: material, size, colour, maker +For a computer you want to store ram, hard disk size etc... + +In a normal 'relational' system you would need to a table for each type. +With dynamic columns you have all common items as fixed fields (like +product_code, manufacturer, price) and the rest stored in a dynamic column. + +The proposed idea is to store the dynamic information in a blob in +Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. + Any support for indexing GPB data is outside of scope of this WL entry. -=-=(Knielsen - Fri, 22 Jan 2010, 11:38)=-=- Low Level Design modified. --- /tmp/wklog.34.old.29965 2010-01-22 11:38:57.000000000 +0200 +++ /tmp/wklog.34.new.29965 2010-01-22 11:38:57.000000000 +0200 @@ -2,3 +2,12 @@ and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. +* We should have both server-side support and client-side support (client side + means functions in libmysqlclient so that user can select the full BLOB and + extract fields in the application). + +* Add some kind of header to the GPB blob to support versioning and future + extensibility. + +* Add complete syntax description (update, add, drop, exists, ...). + -=-=(Psergey - Tue, 21 Jul 2009, 21:13)=-=- Low Level Design modified. --- /tmp/wklog.34.old.6462 2009-07-21 21:13:13.000000000 +0300 +++ /tmp/wklog.34.new.6462 2009-07-21 21:13:13.000000000 +0300 @@ -1 +1,4 @@ +* GPB tarball contains a protocol definition for .proto file structure itself + and a parser for text form of .proto file which then exposes the parsed + file via standard GPB message navigation API. -=-=(Psergey - Tue, 21 Jul 2009, 21:12)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.6399 2009-07-21 21:12:23.000000000 +0300 +++ /tmp/wklog.34.new.6399 2009-07-21 21:12:23.000000000 +0300 @@ -1 +1,78 @@ +<contents> +1. GPB Encoding overview +2. GPB in an SQL database +2.1 Informing server about GPB field names and types +2.2 Addressing GPB fields +2.2.1 Option1: SQL Function +2.2.2 Option2: SQL columns +</contents> + + +1. GPB Encoding overview +======================== + +GBB is a compact encoding for structured and typed data. A unit of GPB data +(it is called message) is only partially self-describing: it's possible to +iterate over its parts, but, quoting the spec + +http://code.google.com/apis/protocolbuffers/docs/encoding.html: + " the name and declared type for each field can only be determined on the + decoding end by referencing the message type's definition (i.e. the .proto + file). " + +2. GPB in an SQL database +========================= + +It is possible to store GPB data in MariaDB today - one can declare a binary +blob column and use it to store GPB messages. Storing and retrieving entire +messages will be the only available operations, though, as the server has no +idea about the GPB format. +It is apparent that ability to peek inside GPB data from SQL layer would be of +great advantage: one would be able to +- select only certain fields or parts of GPB messages +- filter records based on the values of GPB fields +- etc +performing such operations at SQL layer will allow to reduce client<->server +traffic right away, and will open path to getting the best possible +performance. + +2.1 Informing server about GPB field names and types +---------------------------------------------------- +User-friendly/meaningful access to GPB fields requires knowledge of GPB field +names and types, which are not available from GPB message itself (see "GPB +encoding overview" section). + +So the first issue to be addressed is to get the server to know the definition +of stored messages. We intend to assume that all records have GPB messages +that conform to a certain single definition, which gives one definition per +GPB field. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify either the +definition itself or path to .proto file with the definition. + +2.2 Addressing GPB fields +------------------------- +We'll need to provide a way to access GPB fields. This can be complicated as +structures that are encoded in GPB message can be nested and recursive. + +2.2.1 Option1: SQL Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Introduce an SQL function GPB_FIELD(path) which will return contents of the +field. +- Return type of the function will be determined from GPB message definition. +- For path, we can use XPath selector (a subset of XPath) syntax. + +(TODO ^ the above needs to be specified in more detail. is the selector as +simple as filesystem path or we allow quantifiers (with predicates?)?) + +2.2.2 Option2: SQL columns +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make GPB columns to be accessible as SQL columns. +This approach has problems: +- It might be hard to implement code-wise + - (TODO will Virtual columns patch help??) +- It is not clear how to access fields from nested structures. Should we allow + quoted names like `foo/bar[2]/baz' ? + DESCRIPTION: Add support for dynamic columns: - A column that can hold information from many columns - One can instantly add or remove column data This is a useful feature for any store type of application, where you want to store different type of information for different kind of items. For example, for shoes you want to store: material, size, colour, maker For a computer you want to store ram, hard disk size etc... In a normal 'relational' system you would need to a table for each type. With dynamic columns you have all common items as fixed fields (like product_code, manufacturer, price) and the rest stored in a dynamic column. The proposed idea is to store the dynamic information in a blob in Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. Any support for indexing GPB data is outside of scope of this WL entry. Example usage: SELECT proto_get(blob, 1, varchar) from table_with_proto; UPDATE table_with_proto SET blob=proto_add(blob, 2, "hello") where id=1; UPDATE table_with_proto SET blob=proto_del(blob,4) where id=5; Note that 'proto_add()' will replace any old value with the given proto_id. HIGH-LEVEL SPECIFICATION: 1. GPB Encoding overview 2. GPB in an SQL database 3. Encoding to use for dynamic columns 4. How to store and access data in a protocol buffer from SQL 5. Extensions for the future 1. GPB Encoding overview ======================== GBB is a compact encoding for structured and typed data. A unit of GPB data (it is called message) is only partially self-describing: it's possible to iterate over its parts, but, quoting the spec http://code.google.com/apis/protocolbuffers/docs/encoding.html: " the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file). " 2. GPB in an SQL database ========================= It is possible to store GPB data in MariaDB today - one can declare a binary blob column and use it to store GPB messages. Storing and retrieving entire messages will be the only available operations, though, as the server has no idea about the GPB format. It is apparent that ability to peek inside GPB data from SQL layer would be of great advantage: one would be able to - select only certain fields or parts of GPB messages - filter records based on the values of GPB fields - etc performing such operations at SQL layer will allow to reduce client<->server traffic right away, and will open path to getting the best possible performance. 3. Encoding to use for dynamic columns ====================================== The data should be coded into the proto buffer in the following format: <field_number><value_type><value>[<field_number><value_type><value>...] Where field_number is a number between 0-65536 that identifes the field <value_type> is a enum of type 'Item_result' <value> is the value coded in proto format. In other words, we should have no nested or complex structure. 4. How to store and access data in a protocol buffer from SQL ============================================================ User-friendly/meaningful access to GPB fields requires knowledge of GPB field names and types, which are not available from GPB message itself (see "GPB encoding overview" section). To make things easy for the user, we will at first stage provide SQL functions to manipulate a string that is actually in proto format. The functions we should provde are: proto_get(gpb, field_number, type) This return the field tagged with 'field_number' from the 'gpb' buffer. Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar proto_put(gpb, field_number, value) This returns a new gbp buffer with the new value appended. Example: proto_put(proto_put(blob, 1, 1), 2, "hello") 5. Extension for future ======================= In the future we may want to access data based on name and get MariaDB to automaticly know the correct type. To do this we need to be able to store a definition for the content of the proto buffer somewhere. DecisionToMake: How to pass the server the GPB definition? First idea: add a CREATE TABLE parameter which will specify the definition itself. LOW-LEVEL DESIGN: * GPB tarball contains a protocol definition for .proto file structure itself and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. * We should have both server-side support and client-side support (client side means functions in libmysqlclient so that user can select the full BLOB and extract fields in the application). * Add some kind of header to the GPB blob to support versioning and future extensibility. * Add complete syntax description (update, add, drop, exists, ...). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Monty): Add support for dynamic columns (via google protocol buffers) (34)
by worklog-noreply＠askmonty.org 11 Feb '10

11 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add support for dynamic columns (via google protocol buffers) CREATION DATE..: Tue, 21 Jul 2009, 21:11 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 34 (http://askmonty.org/worklog/?tid=34) VERSION........: Server-5.3 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Monty - Thu, 11 Feb 2010, 20:04)=-=- Version updated. --- /tmp/wklog.34.old.18409 2010-02-11 18:04:35.000000000 +0000 +++ /tmp/wklog.34.new.18409 2010-02-11 18:04:35.000000000 +0000 @@ -1 +1 @@ -WorkLog-3.4 +Server-5.3 -=-=(Monty - Thu, 11 Feb 2010, 20:04)=-=- Status updated. --- /tmp/wklog.34.old.18409 2010-02-11 18:04:35.000000000 +0000 +++ /tmp/wklog.34.new.18409 2010-02-11 18:04:35.000000000 +0000 @@ -1 +1 @@ -Un-Assigned +Assigned -=-=(Monty - Thu, 11 Feb 2010, 20:04)=-=- Title modified. --- /tmp/wklog.34.old.18409 2010-02-11 18:04:35.000000000 +0000 +++ /tmp/wklog.34.new.18409 2010-02-11 18:04:35.000000000 +0000 @@ -1 +1 @@ -Add support for google protocol buffers +Add support for dynamic columns (via google protocol buffers) -=-=(Monty - Thu, 11 Feb 2010, 20:03)=-=- High Level Description modified. --- /tmp/wklog.34.old.18329 2010-02-11 18:03:42.000000000 +0000 +++ /tmp/wklog.34.new.18329 2010-02-11 18:03:42.000000000 +0000 @@ -19,3 +19,14 @@ Any support for indexing GPB data is outside of scope of this WL entry. +Example usage: + +SELECT proto_get(blob, 1, varchar) from table_with_proto; + +UPDATE table_with_proto SET blob=proto_add(blob, 2, "hello") where id=1; + +UPDATE table_with_proto SET blob=proto_del(blob,4) where id=5; + +Note that 'proto_add()' will replace any old value with the given proto_id. + + -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.17962 2010-02-11 19:59:45.000000000 +0200 +++ /tmp/wklog.34.new.17962 2010-02-11 19:59:45.000000000 +0200 @@ -1,13 +1,8 @@ - -<contents> 1. GPB Encoding overview 2. GPB in an SQL database -2.1 Informing server about GPB field names and types -2.2 Addressing GPB fields -2.2.1 Option1: SQL Function -2.2.2 Option2: SQL columns -</contents> - +3. Encoding to use for dynamic columns +4. How to store and access data in a protocol buffer from SQL +5. Extensions for the future 1. GPB Encoding overview ======================== @@ -37,42 +32,50 @@ traffic right away, and will open path to getting the best possible performance. -2.1 Informing server about GPB field names and types ----------------------------------------------------- -User-friendly/meaningful access to GPB fields requires knowledge of GPB field -names and types, which are not available from GPB message itself (see "GPB -encoding overview" section). - -So the first issue to be addressed is to get the server to know the definition -of stored messages. We intend to assume that all records have GPB messages -that conform to a certain single definition, which gives one definition per -GPB field. +3. Encoding to use for dynamic columns +====================================== -DecisionToMake: How to pass the server the GPB definition? -First idea: add a CREATE TABLE parameter which will specify either the -definition itself or path to .proto file with the definition. +The data should be coded into the proto buffer in the following format: + +<field_number><value_type><value>[<field_number><value_type><value>...] + +Where field_number is a number between 0-65536 that identifes the field +<value_type> is a enum of type 'Item_result' +<value> is the value coded in proto format. + +In other words, we should have no nested or complex structure. + +4. How to store and access data in a protocol buffer from SQL +============================================================ + +User-friendly/meaningful access to GPB fields requires knowledge of +GPB field names and types, which are not available from GPB message +itself (see "GPB encoding overview" section). + +To make things easy for the user, we will at first stage provide SQL +functions to manipulate a string that is actually in proto format. -2.2 Addressing GPB fields -------------------------- -We'll need to provide a way to access GPB fields. This can be complicated as -structures that are encoded in GPB message can be nested and recursive. - -2.2.1 Option1: SQL Function -~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Introduce an SQL function GPB_FIELD(path) which will return contents of the -field. -- Return type of the function will be determined from GPB message definition. -- For path, we can use XPath selector (a subset of XPath) syntax. - -(TODO ^ the above needs to be specified in more detail. is the selector as -simple as filesystem path or we allow quantifiers (with predicates?)?) - -2.2.2 Option2: SQL columns -~~~~~~~~~~~~~~~~~~~~~~~~~~ -Make GPB columns to be accessible as SQL columns. -This approach has problems: -- It might be hard to implement code-wise - - (TODO will Virtual columns patch help??) -- It is not clear how to access fields from nested structures. Should we allow - quoted names like `foo/bar[2]/baz' ? +The functions we should provde are: +proto_get(gpb, field_number, type) + +This return the field tagged with 'field_number' from the 'gpb' buffer. + +Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar + +proto_put(gpb, field_number, value) + +This returns a new gbp buffer with the new value appended. + +Example: proto_put(proto_put(blob, 1, 1), 2, "hello") + +5. Extension for future +======================= + +In the future we may want to access data based on name and get MariaDB to +automaticly know the correct type. To do this we need to be able to +store a definition for the content of the proto buffer somewhere. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify the +definition itself. -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High Level Description modified. --- /tmp/wklog.34.old.17915 2010-02-11 17:59:17.000000000 +0000 +++ /tmp/wklog.34.new.17915 2010-02-11 17:59:17.000000000 +0000 @@ -1,5 +1,21 @@ -Add support for Google Protocol Buffers (further GPB). It should be possible -to have columns that store GPB-encoded data, as well as use SQL constructs to +Add support for dynamic columns: + +- A column that can hold information from many columns +- One can instantly add or remove column data + +This is a useful feature for any store type of application, where you want to +store different type of information for different kind of items. + +For example, for shoes you want to store: material, size, colour, maker +For a computer you want to store ram, hard disk size etc... + +In a normal 'relational' system you would need to a table for each type. +With dynamic columns you have all common items as fixed fields (like +product_code, manufacturer, price) and the rest stored in a dynamic column. + +The proposed idea is to store the dynamic information in a blob in +Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. + Any support for indexing GPB data is outside of scope of this WL entry. -=-=(Knielsen - Fri, 22 Jan 2010, 11:38)=-=- Low Level Design modified. --- /tmp/wklog.34.old.29965 2010-01-22 11:38:57.000000000 +0200 +++ /tmp/wklog.34.new.29965 2010-01-22 11:38:57.000000000 +0200 @@ -2,3 +2,12 @@ and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. +* We should have both server-side support and client-side support (client side + means functions in libmysqlclient so that user can select the full BLOB and + extract fields in the application). + +* Add some kind of header to the GPB blob to support versioning and future + extensibility. + +* Add complete syntax description (update, add, drop, exists, ...). + -=-=(Psergey - Tue, 21 Jul 2009, 21:13)=-=- Low Level Design modified. --- /tmp/wklog.34.old.6462 2009-07-21 21:13:13.000000000 +0300 +++ /tmp/wklog.34.new.6462 2009-07-21 21:13:13.000000000 +0300 @@ -1 +1,4 @@ +* GPB tarball contains a protocol definition for .proto file structure itself + and a parser for text form of .proto file which then exposes the parsed + file via standard GPB message navigation API. -=-=(Psergey - Tue, 21 Jul 2009, 21:12)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.6399 2009-07-21 21:12:23.000000000 +0300 +++ /tmp/wklog.34.new.6399 2009-07-21 21:12:23.000000000 +0300 @@ -1 +1,78 @@ +<contents> +1. GPB Encoding overview +2. GPB in an SQL database +2.1 Informing server about GPB field names and types +2.2 Addressing GPB fields +2.2.1 Option1: SQL Function +2.2.2 Option2: SQL columns +</contents> + + +1. GPB Encoding overview +======================== + +GBB is a compact encoding for structured and typed data. A unit of GPB data +(it is called message) is only partially self-describing: it's possible to +iterate over its parts, but, quoting the spec + +http://code.google.com/apis/protocolbuffers/docs/encoding.html: + " the name and declared type for each field can only be determined on the + decoding end by referencing the message type's definition (i.e. the .proto + file). " + +2. GPB in an SQL database +========================= + +It is possible to store GPB data in MariaDB today - one can declare a binary +blob column and use it to store GPB messages. Storing and retrieving entire +messages will be the only available operations, though, as the server has no +idea about the GPB format. +It is apparent that ability to peek inside GPB data from SQL layer would be of +great advantage: one would be able to +- select only certain fields or parts of GPB messages +- filter records based on the values of GPB fields +- etc +performing such operations at SQL layer will allow to reduce client<->server +traffic right away, and will open path to getting the best possible +performance. + +2.1 Informing server about GPB field names and types +---------------------------------------------------- +User-friendly/meaningful access to GPB fields requires knowledge of GPB field +names and types, which are not available from GPB message itself (see "GPB +encoding overview" section). + +So the first issue to be addressed is to get the server to know the definition +of stored messages. We intend to assume that all records have GPB messages +that conform to a certain single definition, which gives one definition per +GPB field. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify either the +definition itself or path to .proto file with the definition. + +2.2 Addressing GPB fields +------------------------- +We'll need to provide a way to access GPB fields. This can be complicated as +structures that are encoded in GPB message can be nested and recursive. + +2.2.1 Option1: SQL Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Introduce an SQL function GPB_FIELD(path) which will return contents of the +field. +- Return type of the function will be determined from GPB message definition. +- For path, we can use XPath selector (a subset of XPath) syntax. + +(TODO ^ the above needs to be specified in more detail. is the selector as +simple as filesystem path or we allow quantifiers (with predicates?)?) + +2.2.2 Option2: SQL columns +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make GPB columns to be accessible as SQL columns. +This approach has problems: +- It might be hard to implement code-wise + - (TODO will Virtual columns patch help??) +- It is not clear how to access fields from nested structures. Should we allow + quoted names like `foo/bar[2]/baz' ? + DESCRIPTION: Add support for dynamic columns: - A column that can hold information from many columns - One can instantly add or remove column data This is a useful feature for any store type of application, where you want to store different type of information for different kind of items. For example, for shoes you want to store: material, size, colour, maker For a computer you want to store ram, hard disk size etc... In a normal 'relational' system you would need to a table for each type. With dynamic columns you have all common items as fixed fields (like product_code, manufacturer, price) and the rest stored in a dynamic column. The proposed idea is to store the dynamic information in a blob in Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. Any support for indexing GPB data is outside of scope of this WL entry. Example usage: SELECT proto_get(blob, 1, varchar) from table_with_proto; UPDATE table_with_proto SET blob=proto_add(blob, 2, "hello") where id=1; UPDATE table_with_proto SET blob=proto_del(blob,4) where id=5; Note that 'proto_add()' will replace any old value with the given proto_id. HIGH-LEVEL SPECIFICATION: 1. GPB Encoding overview 2. GPB in an SQL database 3. Encoding to use for dynamic columns 4. How to store and access data in a protocol buffer from SQL 5. Extensions for the future 1. GPB Encoding overview ======================== GBB is a compact encoding for structured and typed data. A unit of GPB data (it is called message) is only partially self-describing: it's possible to iterate over its parts, but, quoting the spec http://code.google.com/apis/protocolbuffers/docs/encoding.html: " the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file). " 2. GPB in an SQL database ========================= It is possible to store GPB data in MariaDB today - one can declare a binary blob column and use it to store GPB messages. Storing and retrieving entire messages will be the only available operations, though, as the server has no idea about the GPB format. It is apparent that ability to peek inside GPB data from SQL layer would be of great advantage: one would be able to - select only certain fields or parts of GPB messages - filter records based on the values of GPB fields - etc performing such operations at SQL layer will allow to reduce client<->server traffic right away, and will open path to getting the best possible performance. 3. Encoding to use for dynamic columns ====================================== The data should be coded into the proto buffer in the following format: <field_number><value_type><value>[<field_number><value_type><value>...] Where field_number is a number between 0-65536 that identifes the field <value_type> is a enum of type 'Item_result' <value> is the value coded in proto format. In other words, we should have no nested or complex structure. 4. How to store and access data in a protocol buffer from SQL ============================================================ User-friendly/meaningful access to GPB fields requires knowledge of GPB field names and types, which are not available from GPB message itself (see "GPB encoding overview" section). To make things easy for the user, we will at first stage provide SQL functions to manipulate a string that is actually in proto format. The functions we should provde are: proto_get(gpb, field_number, type) This return the field tagged with 'field_number' from the 'gpb' buffer. Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar proto_put(gpb, field_number, value) This returns a new gbp buffer with the new value appended. Example: proto_put(proto_put(blob, 1, 1), 2, "hello") 5. Extension for future ======================= In the future we may want to access data based on name and get MariaDB to automaticly know the correct type. To do this we need to be able to store a definition for the content of the proto buffer somewhere. DecisionToMake: How to pass the server the GPB definition? First idea: add a CREATE TABLE parameter which will specify the definition itself. LOW-LEVEL DESIGN: * GPB tarball contains a protocol definition for .proto file structure itself and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. * We should have both server-side support and client-side support (client side means functions in libmysqlclient so that user can select the full BLOB and extract fields in the application). * Add some kind of header to the GPB blob to support versioning and future extensibility. * Add complete syntax description (update, add, drop, exists, ...). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Monty): Add support for google protocol buffers (34)
by worklog-noreply＠askmonty.org 11 Feb '10

11 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add support for google protocol buffers CREATION DATE..: Tue, 21 Jul 2009, 21:11 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 34 (http://askmonty.org/worklog/?tid=34) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Monty - Thu, 11 Feb 2010, 20:03)=-=- High Level Description modified. --- /tmp/wklog.34.old.18329 2010-02-11 18:03:42.000000000 +0000 +++ /tmp/wklog.34.new.18329 2010-02-11 18:03:42.000000000 +0000 @@ -19,3 +19,14 @@ Any support for indexing GPB data is outside of scope of this WL entry. +Example usage: + +SELECT proto_get(blob, 1, varchar) from table_with_proto; + +UPDATE table_with_proto SET blob=proto_add(blob, 2, "hello") where id=1; + +UPDATE table_with_proto SET blob=proto_del(blob,4) where id=5; + +Note that 'proto_add()' will replace any old value with the given proto_id. + + -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.17962 2010-02-11 19:59:45.000000000 +0200 +++ /tmp/wklog.34.new.17962 2010-02-11 19:59:45.000000000 +0200 @@ -1,13 +1,8 @@ - -<contents> 1. GPB Encoding overview 2. GPB in an SQL database -2.1 Informing server about GPB field names and types -2.2 Addressing GPB fields -2.2.1 Option1: SQL Function -2.2.2 Option2: SQL columns -</contents> - +3. Encoding to use for dynamic columns +4. How to store and access data in a protocol buffer from SQL +5. Extensions for the future 1. GPB Encoding overview ======================== @@ -37,42 +32,50 @@ traffic right away, and will open path to getting the best possible performance. -2.1 Informing server about GPB field names and types ----------------------------------------------------- -User-friendly/meaningful access to GPB fields requires knowledge of GPB field -names and types, which are not available from GPB message itself (see "GPB -encoding overview" section). - -So the first issue to be addressed is to get the server to know the definition -of stored messages. We intend to assume that all records have GPB messages -that conform to a certain single definition, which gives one definition per -GPB field. +3. Encoding to use for dynamic columns +====================================== -DecisionToMake: How to pass the server the GPB definition? -First idea: add a CREATE TABLE parameter which will specify either the -definition itself or path to .proto file with the definition. +The data should be coded into the proto buffer in the following format: + +<field_number><value_type><value>[<field_number><value_type><value>...] + +Where field_number is a number between 0-65536 that identifes the field +<value_type> is a enum of type 'Item_result' +<value> is the value coded in proto format. + +In other words, we should have no nested or complex structure. + +4. How to store and access data in a protocol buffer from SQL +============================================================ + +User-friendly/meaningful access to GPB fields requires knowledge of +GPB field names and types, which are not available from GPB message +itself (see "GPB encoding overview" section). + +To make things easy for the user, we will at first stage provide SQL +functions to manipulate a string that is actually in proto format. -2.2 Addressing GPB fields -------------------------- -We'll need to provide a way to access GPB fields. This can be complicated as -structures that are encoded in GPB message can be nested and recursive. - -2.2.1 Option1: SQL Function -~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Introduce an SQL function GPB_FIELD(path) which will return contents of the -field. -- Return type of the function will be determined from GPB message definition. -- For path, we can use XPath selector (a subset of XPath) syntax. - -(TODO ^ the above needs to be specified in more detail. is the selector as -simple as filesystem path or we allow quantifiers (with predicates?)?) - -2.2.2 Option2: SQL columns -~~~~~~~~~~~~~~~~~~~~~~~~~~ -Make GPB columns to be accessible as SQL columns. -This approach has problems: -- It might be hard to implement code-wise - - (TODO will Virtual columns patch help??) -- It is not clear how to access fields from nested structures. Should we allow - quoted names like `foo/bar[2]/baz' ? +The functions we should provde are: +proto_get(gpb, field_number, type) + +This return the field tagged with 'field_number' from the 'gpb' buffer. + +Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar + +proto_put(gpb, field_number, value) + +This returns a new gbp buffer with the new value appended. + +Example: proto_put(proto_put(blob, 1, 1), 2, "hello") + +5. Extension for future +======================= + +In the future we may want to access data based on name and get MariaDB to +automaticly know the correct type. To do this we need to be able to +store a definition for the content of the proto buffer somewhere. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify the +definition itself. -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High Level Description modified. --- /tmp/wklog.34.old.17915 2010-02-11 17:59:17.000000000 +0000 +++ /tmp/wklog.34.new.17915 2010-02-11 17:59:17.000000000 +0000 @@ -1,5 +1,21 @@ -Add support for Google Protocol Buffers (further GPB). It should be possible -to have columns that store GPB-encoded data, as well as use SQL constructs to +Add support for dynamic columns: + +- A column that can hold information from many columns +- One can instantly add or remove column data + +This is a useful feature for any store type of application, where you want to +store different type of information for different kind of items. + +For example, for shoes you want to store: material, size, colour, maker +For a computer you want to store ram, hard disk size etc... + +In a normal 'relational' system you would need to a table for each type. +With dynamic columns you have all common items as fixed fields (like +product_code, manufacturer, price) and the rest stored in a dynamic column. + +The proposed idea is to store the dynamic information in a blob in +Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. + Any support for indexing GPB data is outside of scope of this WL entry. -=-=(Knielsen - Fri, 22 Jan 2010, 11:38)=-=- Low Level Design modified. --- /tmp/wklog.34.old.29965 2010-01-22 11:38:57.000000000 +0200 +++ /tmp/wklog.34.new.29965 2010-01-22 11:38:57.000000000 +0200 @@ -2,3 +2,12 @@ and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. +* We should have both server-side support and client-side support (client side + means functions in libmysqlclient so that user can select the full BLOB and + extract fields in the application). + +* Add some kind of header to the GPB blob to support versioning and future + extensibility. + +* Add complete syntax description (update, add, drop, exists, ...). + -=-=(Psergey - Tue, 21 Jul 2009, 21:13)=-=- Low Level Design modified. --- /tmp/wklog.34.old.6462 2009-07-21 21:13:13.000000000 +0300 +++ /tmp/wklog.34.new.6462 2009-07-21 21:13:13.000000000 +0300 @@ -1 +1,4 @@ +* GPB tarball contains a protocol definition for .proto file structure itself + and a parser for text form of .proto file which then exposes the parsed + file via standard GPB message navigation API. -=-=(Psergey - Tue, 21 Jul 2009, 21:12)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.6399 2009-07-21 21:12:23.000000000 +0300 +++ /tmp/wklog.34.new.6399 2009-07-21 21:12:23.000000000 +0300 @@ -1 +1,78 @@ +<contents> +1. GPB Encoding overview +2. GPB in an SQL database +2.1 Informing server about GPB field names and types +2.2 Addressing GPB fields +2.2.1 Option1: SQL Function +2.2.2 Option2: SQL columns +</contents> + + +1. GPB Encoding overview +======================== + +GBB is a compact encoding for structured and typed data. A unit of GPB data +(it is called message) is only partially self-describing: it's possible to +iterate over its parts, but, quoting the spec + +http://code.google.com/apis/protocolbuffers/docs/encoding.html: + " the name and declared type for each field can only be determined on the + decoding end by referencing the message type's definition (i.e. the .proto + file). " + +2. GPB in an SQL database +========================= + +It is possible to store GPB data in MariaDB today - one can declare a binary +blob column and use it to store GPB messages. Storing and retrieving entire +messages will be the only available operations, though, as the server has no +idea about the GPB format. +It is apparent that ability to peek inside GPB data from SQL layer would be of +great advantage: one would be able to +- select only certain fields or parts of GPB messages +- filter records based on the values of GPB fields +- etc +performing such operations at SQL layer will allow to reduce client<->server +traffic right away, and will open path to getting the best possible +performance. + +2.1 Informing server about GPB field names and types +---------------------------------------------------- +User-friendly/meaningful access to GPB fields requires knowledge of GPB field +names and types, which are not available from GPB message itself (see "GPB +encoding overview" section). + +So the first issue to be addressed is to get the server to know the definition +of stored messages. We intend to assume that all records have GPB messages +that conform to a certain single definition, which gives one definition per +GPB field. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify either the +definition itself or path to .proto file with the definition. + +2.2 Addressing GPB fields +------------------------- +We'll need to provide a way to access GPB fields. This can be complicated as +structures that are encoded in GPB message can be nested and recursive. + +2.2.1 Option1: SQL Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Introduce an SQL function GPB_FIELD(path) which will return contents of the +field. +- Return type of the function will be determined from GPB message definition. +- For path, we can use XPath selector (a subset of XPath) syntax. + +(TODO ^ the above needs to be specified in more detail. is the selector as +simple as filesystem path or we allow quantifiers (with predicates?)?) + +2.2.2 Option2: SQL columns +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make GPB columns to be accessible as SQL columns. +This approach has problems: +- It might be hard to implement code-wise + - (TODO will Virtual columns patch help??) +- It is not clear how to access fields from nested structures. Should we allow + quoted names like `foo/bar[2]/baz' ? + DESCRIPTION: Add support for dynamic columns: - A column that can hold information from many columns - One can instantly add or remove column data This is a useful feature for any store type of application, where you want to store different type of information for different kind of items. For example, for shoes you want to store: material, size, colour, maker For a computer you want to store ram, hard disk size etc... In a normal 'relational' system you would need to a table for each type. With dynamic columns you have all common items as fixed fields (like product_code, manufacturer, price) and the rest stored in a dynamic column. The proposed idea is to store the dynamic information in a blob in Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. Any support for indexing GPB data is outside of scope of this WL entry. Example usage: SELECT proto_get(blob, 1, varchar) from table_with_proto; UPDATE table_with_proto SET blob=proto_add(blob, 2, "hello") where id=1; UPDATE table_with_proto SET blob=proto_del(blob,4) where id=5; Note that 'proto_add()' will replace any old value with the given proto_id. HIGH-LEVEL SPECIFICATION: 1. GPB Encoding overview 2. GPB in an SQL database 3. Encoding to use for dynamic columns 4. How to store and access data in a protocol buffer from SQL 5. Extensions for the future 1. GPB Encoding overview ======================== GBB is a compact encoding for structured and typed data. A unit of GPB data (it is called message) is only partially self-describing: it's possible to iterate over its parts, but, quoting the spec http://code.google.com/apis/protocolbuffers/docs/encoding.html: " the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file). " 2. GPB in an SQL database ========================= It is possible to store GPB data in MariaDB today - one can declare a binary blob column and use it to store GPB messages. Storing and retrieving entire messages will be the only available operations, though, as the server has no idea about the GPB format. It is apparent that ability to peek inside GPB data from SQL layer would be of great advantage: one would be able to - select only certain fields or parts of GPB messages - filter records based on the values of GPB fields - etc performing such operations at SQL layer will allow to reduce client<->server traffic right away, and will open path to getting the best possible performance. 3. Encoding to use for dynamic columns ====================================== The data should be coded into the proto buffer in the following format: <field_number><value_type><value>[<field_number><value_type><value>...] Where field_number is a number between 0-65536 that identifes the field <value_type> is a enum of type 'Item_result' <value> is the value coded in proto format. In other words, we should have no nested or complex structure. 4. How to store and access data in a protocol buffer from SQL ============================================================ User-friendly/meaningful access to GPB fields requires knowledge of GPB field names and types, which are not available from GPB message itself (see "GPB encoding overview" section). To make things easy for the user, we will at first stage provide SQL functions to manipulate a string that is actually in proto format. The functions we should provde are: proto_get(gpb, field_number, type) This return the field tagged with 'field_number' from the 'gpb' buffer. Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar proto_put(gpb, field_number, value) This returns a new gbp buffer with the new value appended. Example: proto_put(proto_put(blob, 1, 1), 2, "hello") 5. Extension for future ======================= In the future we may want to access data based on name and get MariaDB to automaticly know the correct type. To do this we need to be able to store a definition for the content of the proto buffer somewhere. DecisionToMake: How to pass the server the GPB definition? First idea: add a CREATE TABLE parameter which will specify the definition itself. LOW-LEVEL DESIGN: * GPB tarball contains a protocol definition for .proto file structure itself and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. * We should have both server-side support and client-side support (client side means functions in libmysqlclient so that user can select the full BLOB and extract fields in the application). * Add some kind of header to the GPB blob to support versioning and future extensibility. * Add complete syntax description (update, add, drop, exists, ...). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Monty): Add support for google protocol buffers (34)
by worklog-noreply＠askmonty.org 11 Feb '10

11 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add support for google protocol buffers CREATION DATE..: Tue, 21 Jul 2009, 21:11 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 34 (http://askmonty.org/worklog/?tid=34) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Monty - Thu, 11 Feb 2010, 20:03)=-=- High Level Description modified. --- /tmp/wklog.34.old.18329 2010-02-11 18:03:42.000000000 +0000 +++ /tmp/wklog.34.new.18329 2010-02-11 18:03:42.000000000 +0000 @@ -19,3 +19,14 @@ Any support for indexing GPB data is outside of scope of this WL entry. +Example usage: + +SELECT proto_get(blob, 1, varchar) from table_with_proto; + +UPDATE table_with_proto SET blob=proto_add(blob, 2, "hello") where id=1; + +UPDATE table_with_proto SET blob=proto_del(blob,4) where id=5; + +Note that 'proto_add()' will replace any old value with the given proto_id. + + -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.17962 2010-02-11 19:59:45.000000000 +0200 +++ /tmp/wklog.34.new.17962 2010-02-11 19:59:45.000000000 +0200 @@ -1,13 +1,8 @@ - -<contents> 1. GPB Encoding overview 2. GPB in an SQL database -2.1 Informing server about GPB field names and types -2.2 Addressing GPB fields -2.2.1 Option1: SQL Function -2.2.2 Option2: SQL columns -</contents> - +3. Encoding to use for dynamic columns +4. How to store and access data in a protocol buffer from SQL +5. Extensions for the future 1. GPB Encoding overview ======================== @@ -37,42 +32,50 @@ traffic right away, and will open path to getting the best possible performance. -2.1 Informing server about GPB field names and types ----------------------------------------------------- -User-friendly/meaningful access to GPB fields requires knowledge of GPB field -names and types, which are not available from GPB message itself (see "GPB -encoding overview" section). - -So the first issue to be addressed is to get the server to know the definition -of stored messages. We intend to assume that all records have GPB messages -that conform to a certain single definition, which gives one definition per -GPB field. +3. Encoding to use for dynamic columns +====================================== -DecisionToMake: How to pass the server the GPB definition? -First idea: add a CREATE TABLE parameter which will specify either the -definition itself or path to .proto file with the definition. +The data should be coded into the proto buffer in the following format: + +<field_number><value_type><value>[<field_number><value_type><value>...] + +Where field_number is a number between 0-65536 that identifes the field +<value_type> is a enum of type 'Item_result' +<value> is the value coded in proto format. + +In other words, we should have no nested or complex structure. + +4. How to store and access data in a protocol buffer from SQL +============================================================ + +User-friendly/meaningful access to GPB fields requires knowledge of +GPB field names and types, which are not available from GPB message +itself (see "GPB encoding overview" section). + +To make things easy for the user, we will at first stage provide SQL +functions to manipulate a string that is actually in proto format. -2.2 Addressing GPB fields -------------------------- -We'll need to provide a way to access GPB fields. This can be complicated as -structures that are encoded in GPB message can be nested and recursive. - -2.2.1 Option1: SQL Function -~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Introduce an SQL function GPB_FIELD(path) which will return contents of the -field. -- Return type of the function will be determined from GPB message definition. -- For path, we can use XPath selector (a subset of XPath) syntax. - -(TODO ^ the above needs to be specified in more detail. is the selector as -simple as filesystem path or we allow quantifiers (with predicates?)?) - -2.2.2 Option2: SQL columns -~~~~~~~~~~~~~~~~~~~~~~~~~~ -Make GPB columns to be accessible as SQL columns. -This approach has problems: -- It might be hard to implement code-wise - - (TODO will Virtual columns patch help??) -- It is not clear how to access fields from nested structures. Should we allow - quoted names like `foo/bar[2]/baz' ? +The functions we should provde are: +proto_get(gpb, field_number, type) + +This return the field tagged with 'field_number' from the 'gpb' buffer. + +Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar + +proto_put(gpb, field_number, value) + +This returns a new gbp buffer with the new value appended. + +Example: proto_put(proto_put(blob, 1, 1), 2, "hello") + +5. Extension for future +======================= + +In the future we may want to access data based on name and get MariaDB to +automaticly know the correct type. To do this we need to be able to +store a definition for the content of the proto buffer somewhere. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify the +definition itself. -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High Level Description modified. --- /tmp/wklog.34.old.17915 2010-02-11 17:59:17.000000000 +0000 +++ /tmp/wklog.34.new.17915 2010-02-11 17:59:17.000000000 +0000 @@ -1,5 +1,21 @@ -Add support for Google Protocol Buffers (further GPB). It should be possible -to have columns that store GPB-encoded data, as well as use SQL constructs to +Add support for dynamic columns: + +- A column that can hold information from many columns +- One can instantly add or remove column data + +This is a useful feature for any store type of application, where you want to +store different type of information for different kind of items. + +For example, for shoes you want to store: material, size, colour, maker +For a computer you want to store ram, hard disk size etc... + +In a normal 'relational' system you would need to a table for each type. +With dynamic columns you have all common items as fixed fields (like +product_code, manufacturer, price) and the rest stored in a dynamic column. + +The proposed idea is to store the dynamic information in a blob in +Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. + Any support for indexing GPB data is outside of scope of this WL entry. -=-=(Knielsen - Fri, 22 Jan 2010, 11:38)=-=- Low Level Design modified. --- /tmp/wklog.34.old.29965 2010-01-22 11:38:57.000000000 +0200 +++ /tmp/wklog.34.new.29965 2010-01-22 11:38:57.000000000 +0200 @@ -2,3 +2,12 @@ and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. +* We should have both server-side support and client-side support (client side + means functions in libmysqlclient so that user can select the full BLOB and + extract fields in the application). + +* Add some kind of header to the GPB blob to support versioning and future + extensibility. + +* Add complete syntax description (update, add, drop, exists, ...). + -=-=(Psergey - Tue, 21 Jul 2009, 21:13)=-=- Low Level Design modified. --- /tmp/wklog.34.old.6462 2009-07-21 21:13:13.000000000 +0300 +++ /tmp/wklog.34.new.6462 2009-07-21 21:13:13.000000000 +0300 @@ -1 +1,4 @@ +* GPB tarball contains a protocol definition for .proto file structure itself + and a parser for text form of .proto file which then exposes the parsed + file via standard GPB message navigation API. -=-=(Psergey - Tue, 21 Jul 2009, 21:12)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.6399 2009-07-21 21:12:23.000000000 +0300 +++ /tmp/wklog.34.new.6399 2009-07-21 21:12:23.000000000 +0300 @@ -1 +1,78 @@ +<contents> +1. GPB Encoding overview +2. GPB in an SQL database +2.1 Informing server about GPB field names and types +2.2 Addressing GPB fields +2.2.1 Option1: SQL Function +2.2.2 Option2: SQL columns +</contents> + + +1. GPB Encoding overview +======================== + +GBB is a compact encoding for structured and typed data. A unit of GPB data +(it is called message) is only partially self-describing: it's possible to +iterate over its parts, but, quoting the spec + +http://code.google.com/apis/protocolbuffers/docs/encoding.html: + " the name and declared type for each field can only be determined on the + decoding end by referencing the message type's definition (i.e. the .proto + file). " + +2. GPB in an SQL database +========================= + +It is possible to store GPB data in MariaDB today - one can declare a binary +blob column and use it to store GPB messages. Storing and retrieving entire +messages will be the only available operations, though, as the server has no +idea about the GPB format. +It is apparent that ability to peek inside GPB data from SQL layer would be of +great advantage: one would be able to +- select only certain fields or parts of GPB messages +- filter records based on the values of GPB fields +- etc +performing such operations at SQL layer will allow to reduce client<->server +traffic right away, and will open path to getting the best possible +performance. + +2.1 Informing server about GPB field names and types +---------------------------------------------------- +User-friendly/meaningful access to GPB fields requires knowledge of GPB field +names and types, which are not available from GPB message itself (see "GPB +encoding overview" section). + +So the first issue to be addressed is to get the server to know the definition +of stored messages. We intend to assume that all records have GPB messages +that conform to a certain single definition, which gives one definition per +GPB field. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify either the +definition itself or path to .proto file with the definition. + +2.2 Addressing GPB fields +------------------------- +We'll need to provide a way to access GPB fields. This can be complicated as +structures that are encoded in GPB message can be nested and recursive. + +2.2.1 Option1: SQL Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Introduce an SQL function GPB_FIELD(path) which will return contents of the +field. +- Return type of the function will be determined from GPB message definition. +- For path, we can use XPath selector (a subset of XPath) syntax. + +(TODO ^ the above needs to be specified in more detail. is the selector as +simple as filesystem path or we allow quantifiers (with predicates?)?) + +2.2.2 Option2: SQL columns +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make GPB columns to be accessible as SQL columns. +This approach has problems: +- It might be hard to implement code-wise + - (TODO will Virtual columns patch help??) +- It is not clear how to access fields from nested structures. Should we allow + quoted names like `foo/bar[2]/baz' ? + DESCRIPTION: Add support for dynamic columns: - A column that can hold information from many columns - One can instantly add or remove column data This is a useful feature for any store type of application, where you want to store different type of information for different kind of items. For example, for shoes you want to store: material, size, colour, maker For a computer you want to store ram, hard disk size etc... In a normal 'relational' system you would need to a table for each type. With dynamic columns you have all common items as fixed fields (like product_code, manufacturer, price) and the rest stored in a dynamic column. The proposed idea is to store the dynamic information in a blob in Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. Any support for indexing GPB data is outside of scope of this WL entry. Example usage: SELECT proto_get(blob, 1, varchar) from table_with_proto; UPDATE table_with_proto SET blob=proto_add(blob, 2, "hello") where id=1; UPDATE table_with_proto SET blob=proto_del(blob,4) where id=5; Note that 'proto_add()' will replace any old value with the given proto_id. HIGH-LEVEL SPECIFICATION: 1. GPB Encoding overview 2. GPB in an SQL database 3. Encoding to use for dynamic columns 4. How to store and access data in a protocol buffer from SQL 5. Extensions for the future 1. GPB Encoding overview ======================== GBB is a compact encoding for structured and typed data. A unit of GPB data (it is called message) is only partially self-describing: it's possible to iterate over its parts, but, quoting the spec http://code.google.com/apis/protocolbuffers/docs/encoding.html: " the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file). " 2. GPB in an SQL database ========================= It is possible to store GPB data in MariaDB today - one can declare a binary blob column and use it to store GPB messages. Storing and retrieving entire messages will be the only available operations, though, as the server has no idea about the GPB format. It is apparent that ability to peek inside GPB data from SQL layer would be of great advantage: one would be able to - select only certain fields or parts of GPB messages - filter records based on the values of GPB fields - etc performing such operations at SQL layer will allow to reduce client<->server traffic right away, and will open path to getting the best possible performance. 3. Encoding to use for dynamic columns ====================================== The data should be coded into the proto buffer in the following format: <field_number><value_type><value>[<field_number><value_type><value>...] Where field_number is a number between 0-65536 that identifes the field <value_type> is a enum of type 'Item_result' <value> is the value coded in proto format. In other words, we should have no nested or complex structure. 4. How to store and access data in a protocol buffer from SQL ============================================================ User-friendly/meaningful access to GPB fields requires knowledge of GPB field names and types, which are not available from GPB message itself (see "GPB encoding overview" section). To make things easy for the user, we will at first stage provide SQL functions to manipulate a string that is actually in proto format. The functions we should provde are: proto_get(gpb, field_number, type) This return the field tagged with 'field_number' from the 'gpb' buffer. Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar proto_put(gpb, field_number, value) This returns a new gbp buffer with the new value appended. Example: proto_put(proto_put(blob, 1, 1), 2, "hello") 5. Extension for future ======================= In the future we may want to access data based on name and get MariaDB to automaticly know the correct type. To do this we need to be able to store a definition for the content of the proto buffer somewhere. DecisionToMake: How to pass the server the GPB definition? First idea: add a CREATE TABLE parameter which will specify the definition itself. LOW-LEVEL DESIGN: * GPB tarball contains a protocol definition for .proto file structure itself and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. * We should have both server-side support and client-side support (client side means functions in libmysqlclient so that user can select the full BLOB and extract fields in the application). * Add some kind of header to the GPB blob to support versioning and future extensibility. * Add complete syntax description (update, add, drop, exists, ...). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Monty): Add support for google protocol buffers (34)
by worklog-noreply＠askmonty.org 11 Feb '10

11 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add support for google protocol buffers CREATION DATE..: Tue, 21 Jul 2009, 21:11 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 34 (http://askmonty.org/worklog/?tid=34) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.17962 2010-02-11 19:59:45.000000000 +0200 +++ /tmp/wklog.34.new.17962 2010-02-11 19:59:45.000000000 +0200 @@ -1,13 +1,8 @@ - -<contents> 1. GPB Encoding overview 2. GPB in an SQL database -2.1 Informing server about GPB field names and types -2.2 Addressing GPB fields -2.2.1 Option1: SQL Function -2.2.2 Option2: SQL columns -</contents> - +3. Encoding to use for dynamic columns +4. How to store and access data in a protocol buffer from SQL +5. Extensions for the future 1. GPB Encoding overview ======================== @@ -37,42 +32,50 @@ traffic right away, and will open path to getting the best possible performance. -2.1 Informing server about GPB field names and types ----------------------------------------------------- -User-friendly/meaningful access to GPB fields requires knowledge of GPB field -names and types, which are not available from GPB message itself (see "GPB -encoding overview" section). - -So the first issue to be addressed is to get the server to know the definition -of stored messages. We intend to assume that all records have GPB messages -that conform to a certain single definition, which gives one definition per -GPB field. +3. Encoding to use for dynamic columns +====================================== -DecisionToMake: How to pass the server the GPB definition? -First idea: add a CREATE TABLE parameter which will specify either the -definition itself or path to .proto file with the definition. +The data should be coded into the proto buffer in the following format: + +<field_number><value_type><value>[<field_number><value_type><value>...] + +Where field_number is a number between 0-65536 that identifes the field +<value_type> is a enum of type 'Item_result' +<value> is the value coded in proto format. + +In other words, we should have no nested or complex structure. + +4. How to store and access data in a protocol buffer from SQL +============================================================ + +User-friendly/meaningful access to GPB fields requires knowledge of +GPB field names and types, which are not available from GPB message +itself (see "GPB encoding overview" section). + +To make things easy for the user, we will at first stage provide SQL +functions to manipulate a string that is actually in proto format. -2.2 Addressing GPB fields -------------------------- -We'll need to provide a way to access GPB fields. This can be complicated as -structures that are encoded in GPB message can be nested and recursive. - -2.2.1 Option1: SQL Function -~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Introduce an SQL function GPB_FIELD(path) which will return contents of the -field. -- Return type of the function will be determined from GPB message definition. -- For path, we can use XPath selector (a subset of XPath) syntax. - -(TODO ^ the above needs to be specified in more detail. is the selector as -simple as filesystem path or we allow quantifiers (with predicates?)?) - -2.2.2 Option2: SQL columns -~~~~~~~~~~~~~~~~~~~~~~~~~~ -Make GPB columns to be accessible as SQL columns. -This approach has problems: -- It might be hard to implement code-wise - - (TODO will Virtual columns patch help??) -- It is not clear how to access fields from nested structures. Should we allow - quoted names like `foo/bar[2]/baz' ? +The functions we should provde are: +proto_get(gpb, field_number, type) + +This return the field tagged with 'field_number' from the 'gpb' buffer. + +Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar + +proto_put(gpb, field_number, value) + +This returns a new gbp buffer with the new value appended. + +Example: proto_put(proto_put(blob, 1, 1), 2, "hello") + +5. Extension for future +======================= + +In the future we may want to access data based on name and get MariaDB to +automaticly know the correct type. To do this we need to be able to +store a definition for the content of the proto buffer somewhere. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify the +definition itself. -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High Level Description modified. --- /tmp/wklog.34.old.17915 2010-02-11 17:59:17.000000000 +0000 +++ /tmp/wklog.34.new.17915 2010-02-11 17:59:17.000000000 +0000 @@ -1,5 +1,21 @@ -Add support for Google Protocol Buffers (further GPB). It should be possible -to have columns that store GPB-encoded data, as well as use SQL constructs to +Add support for dynamic columns: + +- A column that can hold information from many columns +- One can instantly add or remove column data + +This is a useful feature for any store type of application, where you want to +store different type of information for different kind of items. + +For example, for shoes you want to store: material, size, colour, maker +For a computer you want to store ram, hard disk size etc... + +In a normal 'relational' system you would need to a table for each type. +With dynamic columns you have all common items as fixed fields (like +product_code, manufacturer, price) and the rest stored in a dynamic column. + +The proposed idea is to store the dynamic information in a blob in +Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. + Any support for indexing GPB data is outside of scope of this WL entry. -=-=(Knielsen - Fri, 22 Jan 2010, 11:38)=-=- Low Level Design modified. --- /tmp/wklog.34.old.29965 2010-01-22 11:38:57.000000000 +0200 +++ /tmp/wklog.34.new.29965 2010-01-22 11:38:57.000000000 +0200 @@ -2,3 +2,12 @@ and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. +* We should have both server-side support and client-side support (client side + means functions in libmysqlclient so that user can select the full BLOB and + extract fields in the application). + +* Add some kind of header to the GPB blob to support versioning and future + extensibility. + +* Add complete syntax description (update, add, drop, exists, ...). + -=-=(Psergey - Tue, 21 Jul 2009, 21:13)=-=- Low Level Design modified. --- /tmp/wklog.34.old.6462 2009-07-21 21:13:13.000000000 +0300 +++ /tmp/wklog.34.new.6462 2009-07-21 21:13:13.000000000 +0300 @@ -1 +1,4 @@ +* GPB tarball contains a protocol definition for .proto file structure itself + and a parser for text form of .proto file which then exposes the parsed + file via standard GPB message navigation API. -=-=(Psergey - Tue, 21 Jul 2009, 21:12)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.6399 2009-07-21 21:12:23.000000000 +0300 +++ /tmp/wklog.34.new.6399 2009-07-21 21:12:23.000000000 +0300 @@ -1 +1,78 @@ +<contents> +1. GPB Encoding overview +2. GPB in an SQL database +2.1 Informing server about GPB field names and types +2.2 Addressing GPB fields +2.2.1 Option1: SQL Function +2.2.2 Option2: SQL columns +</contents> + + +1. GPB Encoding overview +======================== + +GBB is a compact encoding for structured and typed data. A unit of GPB data +(it is called message) is only partially self-describing: it's possible to +iterate over its parts, but, quoting the spec + +http://code.google.com/apis/protocolbuffers/docs/encoding.html: + " the name and declared type for each field can only be determined on the + decoding end by referencing the message type's definition (i.e. the .proto + file). " + +2. GPB in an SQL database +========================= + +It is possible to store GPB data in MariaDB today - one can declare a binary +blob column and use it to store GPB messages. Storing and retrieving entire +messages will be the only available operations, though, as the server has no +idea about the GPB format. +It is apparent that ability to peek inside GPB data from SQL layer would be of +great advantage: one would be able to +- select only certain fields or parts of GPB messages +- filter records based on the values of GPB fields +- etc +performing such operations at SQL layer will allow to reduce client<->server +traffic right away, and will open path to getting the best possible +performance. + +2.1 Informing server about GPB field names and types +---------------------------------------------------- +User-friendly/meaningful access to GPB fields requires knowledge of GPB field +names and types, which are not available from GPB message itself (see "GPB +encoding overview" section). + +So the first issue to be addressed is to get the server to know the definition +of stored messages. We intend to assume that all records have GPB messages +that conform to a certain single definition, which gives one definition per +GPB field. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify either the +definition itself or path to .proto file with the definition. + +2.2 Addressing GPB fields +------------------------- +We'll need to provide a way to access GPB fields. This can be complicated as +structures that are encoded in GPB message can be nested and recursive. + +2.2.1 Option1: SQL Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Introduce an SQL function GPB_FIELD(path) which will return contents of the +field. +- Return type of the function will be determined from GPB message definition. +- For path, we can use XPath selector (a subset of XPath) syntax. + +(TODO ^ the above needs to be specified in more detail. is the selector as +simple as filesystem path or we allow quantifiers (with predicates?)?) + +2.2.2 Option2: SQL columns +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make GPB columns to be accessible as SQL columns. +This approach has problems: +- It might be hard to implement code-wise + - (TODO will Virtual columns patch help??) +- It is not clear how to access fields from nested structures. Should we allow + quoted names like `foo/bar[2]/baz' ? + DESCRIPTION: Add support for dynamic columns: - A column that can hold information from many columns - One can instantly add or remove column data This is a useful feature for any store type of application, where you want to store different type of information for different kind of items. For example, for shoes you want to store: material, size, colour, maker For a computer you want to store ram, hard disk size etc... In a normal 'relational' system you would need to a table for each type. With dynamic columns you have all common items as fixed fields (like product_code, manufacturer, price) and the rest stored in a dynamic column. The proposed idea is to store the dynamic information in a blob in Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. Any support for indexing GPB data is outside of scope of this WL entry. HIGH-LEVEL SPECIFICATION: 1. GPB Encoding overview 2. GPB in an SQL database 3. Encoding to use for dynamic columns 4. How to store and access data in a protocol buffer from SQL 5. Extensions for the future 1. GPB Encoding overview ======================== GBB is a compact encoding for structured and typed data. A unit of GPB data (it is called message) is only partially self-describing: it's possible to iterate over its parts, but, quoting the spec http://code.google.com/apis/protocolbuffers/docs/encoding.html: " the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file). " 2. GPB in an SQL database ========================= It is possible to store GPB data in MariaDB today - one can declare a binary blob column and use it to store GPB messages. Storing and retrieving entire messages will be the only available operations, though, as the server has no idea about the GPB format. It is apparent that ability to peek inside GPB data from SQL layer would be of great advantage: one would be able to - select only certain fields or parts of GPB messages - filter records based on the values of GPB fields - etc performing such operations at SQL layer will allow to reduce client<->server traffic right away, and will open path to getting the best possible performance. 3. Encoding to use for dynamic columns ====================================== The data should be coded into the proto buffer in the following format: <field_number><value_type><value>[<field_number><value_type><value>...] Where field_number is a number between 0-65536 that identifes the field <value_type> is a enum of type 'Item_result' <value> is the value coded in proto format. In other words, we should have no nested or complex structure. 4. How to store and access data in a protocol buffer from SQL ============================================================ User-friendly/meaningful access to GPB fields requires knowledge of GPB field names and types, which are not available from GPB message itself (see "GPB encoding overview" section). To make things easy for the user, we will at first stage provide SQL functions to manipulate a string that is actually in proto format. The functions we should provde are: proto_get(gpb, field_number, type) This return the field tagged with 'field_number' from the 'gpb' buffer. Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar proto_put(gpb, field_number, value) This returns a new gbp buffer with the new value appended. Example: proto_put(proto_put(blob, 1, 1), 2, "hello") 5. Extension for future ======================= In the future we may want to access data based on name and get MariaDB to automaticly know the correct type. To do this we need to be able to store a definition for the content of the proto buffer somewhere. DecisionToMake: How to pass the server the GPB definition? First idea: add a CREATE TABLE parameter which will specify the definition itself. LOW-LEVEL DESIGN: * GPB tarball contains a protocol definition for .proto file structure itself and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. * We should have both server-side support and client-side support (client side means functions in libmysqlclient so that user can select the full BLOB and extract fields in the application). * Add some kind of header to the GPB blob to support versioning and future extensibility. * Add complete syntax description (update, add, drop, exists, ...). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Monty): Add support for google protocol buffers (34)
by worklog-noreply＠askmonty.org 11 Feb '10

11 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add support for google protocol buffers CREATION DATE..: Tue, 21 Jul 2009, 21:11 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 34 (http://askmonty.org/worklog/?tid=34) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.17962 2010-02-11 19:59:45.000000000 +0200 +++ /tmp/wklog.34.new.17962 2010-02-11 19:59:45.000000000 +0200 @@ -1,13 +1,8 @@ - -<contents> 1. GPB Encoding overview 2. GPB in an SQL database -2.1 Informing server about GPB field names and types -2.2 Addressing GPB fields -2.2.1 Option1: SQL Function -2.2.2 Option2: SQL columns -</contents> - +3. Encoding to use for dynamic columns +4. How to store and access data in a protocol buffer from SQL +5. Extensions for the future 1. GPB Encoding overview ======================== @@ -37,42 +32,50 @@ traffic right away, and will open path to getting the best possible performance. -2.1 Informing server about GPB field names and types ----------------------------------------------------- -User-friendly/meaningful access to GPB fields requires knowledge of GPB field -names and types, which are not available from GPB message itself (see "GPB -encoding overview" section). - -So the first issue to be addressed is to get the server to know the definition -of stored messages. We intend to assume that all records have GPB messages -that conform to a certain single definition, which gives one definition per -GPB field. +3. Encoding to use for dynamic columns +====================================== -DecisionToMake: How to pass the server the GPB definition? -First idea: add a CREATE TABLE parameter which will specify either the -definition itself or path to .proto file with the definition. +The data should be coded into the proto buffer in the following format: + +<field_number><value_type><value>[<field_number><value_type><value>...] + +Where field_number is a number between 0-65536 that identifes the field +<value_type> is a enum of type 'Item_result' +<value> is the value coded in proto format. + +In other words, we should have no nested or complex structure. + +4. How to store and access data in a protocol buffer from SQL +============================================================ + +User-friendly/meaningful access to GPB fields requires knowledge of +GPB field names and types, which are not available from GPB message +itself (see "GPB encoding overview" section). + +To make things easy for the user, we will at first stage provide SQL +functions to manipulate a string that is actually in proto format. -2.2 Addressing GPB fields -------------------------- -We'll need to provide a way to access GPB fields. This can be complicated as -structures that are encoded in GPB message can be nested and recursive. - -2.2.1 Option1: SQL Function -~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Introduce an SQL function GPB_FIELD(path) which will return contents of the -field. -- Return type of the function will be determined from GPB message definition. -- For path, we can use XPath selector (a subset of XPath) syntax. - -(TODO ^ the above needs to be specified in more detail. is the selector as -simple as filesystem path or we allow quantifiers (with predicates?)?) - -2.2.2 Option2: SQL columns -~~~~~~~~~~~~~~~~~~~~~~~~~~ -Make GPB columns to be accessible as SQL columns. -This approach has problems: -- It might be hard to implement code-wise - - (TODO will Virtual columns patch help??) -- It is not clear how to access fields from nested structures. Should we allow - quoted names like `foo/bar[2]/baz' ? +The functions we should provde are: +proto_get(gpb, field_number, type) + +This return the field tagged with 'field_number' from the 'gpb' buffer. + +Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar + +proto_put(gpb, field_number, value) + +This returns a new gbp buffer with the new value appended. + +Example: proto_put(proto_put(blob, 1, 1), 2, "hello") + +5. Extension for future +======================= + +In the future we may want to access data based on name and get MariaDB to +automaticly know the correct type. To do this we need to be able to +store a definition for the content of the proto buffer somewhere. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify the +definition itself. -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High Level Description modified. --- /tmp/wklog.34.old.17915 2010-02-11 17:59:17.000000000 +0000 +++ /tmp/wklog.34.new.17915 2010-02-11 17:59:17.000000000 +0000 @@ -1,5 +1,21 @@ -Add support for Google Protocol Buffers (further GPB). It should be possible -to have columns that store GPB-encoded data, as well as use SQL constructs to +Add support for dynamic columns: + +- A column that can hold information from many columns +- One can instantly add or remove column data + +This is a useful feature for any store type of application, where you want to +store different type of information for different kind of items. + +For example, for shoes you want to store: material, size, colour, maker +For a computer you want to store ram, hard disk size etc... + +In a normal 'relational' system you would need to a table for each type. +With dynamic columns you have all common items as fixed fields (like +product_code, manufacturer, price) and the rest stored in a dynamic column. + +The proposed idea is to store the dynamic information in a blob in +Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. + Any support for indexing GPB data is outside of scope of this WL entry. -=-=(Knielsen - Fri, 22 Jan 2010, 11:38)=-=- Low Level Design modified. --- /tmp/wklog.34.old.29965 2010-01-22 11:38:57.000000000 +0200 +++ /tmp/wklog.34.new.29965 2010-01-22 11:38:57.000000000 +0200 @@ -2,3 +2,12 @@ and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. +* We should have both server-side support and client-side support (client side + means functions in libmysqlclient so that user can select the full BLOB and + extract fields in the application). + +* Add some kind of header to the GPB blob to support versioning and future + extensibility. + +* Add complete syntax description (update, add, drop, exists, ...). + -=-=(Psergey - Tue, 21 Jul 2009, 21:13)=-=- Low Level Design modified. --- /tmp/wklog.34.old.6462 2009-07-21 21:13:13.000000000 +0300 +++ /tmp/wklog.34.new.6462 2009-07-21 21:13:13.000000000 +0300 @@ -1 +1,4 @@ +* GPB tarball contains a protocol definition for .proto file structure itself + and a parser for text form of .proto file which then exposes the parsed + file via standard GPB message navigation API. -=-=(Psergey - Tue, 21 Jul 2009, 21:12)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.6399 2009-07-21 21:12:23.000000000 +0300 +++ /tmp/wklog.34.new.6399 2009-07-21 21:12:23.000000000 +0300 @@ -1 +1,78 @@ +<contents> +1. GPB Encoding overview +2. GPB in an SQL database +2.1 Informing server about GPB field names and types +2.2 Addressing GPB fields +2.2.1 Option1: SQL Function +2.2.2 Option2: SQL columns +</contents> + + +1. GPB Encoding overview +======================== + +GBB is a compact encoding for structured and typed data. A unit of GPB data +(it is called message) is only partially self-describing: it's possible to +iterate over its parts, but, quoting the spec + +http://code.google.com/apis/protocolbuffers/docs/encoding.html: + " the name and declared type for each field can only be determined on the + decoding end by referencing the message type's definition (i.e. the .proto + file). " + +2. GPB in an SQL database +========================= + +It is possible to store GPB data in MariaDB today - one can declare a binary +blob column and use it to store GPB messages. Storing and retrieving entire +messages will be the only available operations, though, as the server has no +idea about the GPB format. +It is apparent that ability to peek inside GPB data from SQL layer would be of +great advantage: one would be able to +- select only certain fields or parts of GPB messages +- filter records based on the values of GPB fields +- etc +performing such operations at SQL layer will allow to reduce client<->server +traffic right away, and will open path to getting the best possible +performance. + +2.1 Informing server about GPB field names and types +---------------------------------------------------- +User-friendly/meaningful access to GPB fields requires knowledge of GPB field +names and types, which are not available from GPB message itself (see "GPB +encoding overview" section). + +So the first issue to be addressed is to get the server to know the definition +of stored messages. We intend to assume that all records have GPB messages +that conform to a certain single definition, which gives one definition per +GPB field. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify either the +definition itself or path to .proto file with the definition. + +2.2 Addressing GPB fields +------------------------- +We'll need to provide a way to access GPB fields. This can be complicated as +structures that are encoded in GPB message can be nested and recursive. + +2.2.1 Option1: SQL Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Introduce an SQL function GPB_FIELD(path) which will return contents of the +field. +- Return type of the function will be determined from GPB message definition. +- For path, we can use XPath selector (a subset of XPath) syntax. + +(TODO ^ the above needs to be specified in more detail. is the selector as +simple as filesystem path or we allow quantifiers (with predicates?)?) + +2.2.2 Option2: SQL columns +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make GPB columns to be accessible as SQL columns. +This approach has problems: +- It might be hard to implement code-wise + - (TODO will Virtual columns patch help??) +- It is not clear how to access fields from nested structures. Should we allow + quoted names like `foo/bar[2]/baz' ? + DESCRIPTION: Add support for dynamic columns: - A column that can hold information from many columns - One can instantly add or remove column data This is a useful feature for any store type of application, where you want to store different type of information for different kind of items. For example, for shoes you want to store: material, size, colour, maker For a computer you want to store ram, hard disk size etc... In a normal 'relational' system you would need to a table for each type. With dynamic columns you have all common items as fixed fields (like product_code, manufacturer, price) and the rest stored in a dynamic column. The proposed idea is to store the dynamic information in a blob in Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. Any support for indexing GPB data is outside of scope of this WL entry. HIGH-LEVEL SPECIFICATION: 1. GPB Encoding overview 2. GPB in an SQL database 3. Encoding to use for dynamic columns 4. How to store and access data in a protocol buffer from SQL 5. Extensions for the future 1. GPB Encoding overview ======================== GBB is a compact encoding for structured and typed data. A unit of GPB data (it is called message) is only partially self-describing: it's possible to iterate over its parts, but, quoting the spec http://code.google.com/apis/protocolbuffers/docs/encoding.html: " the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file). " 2. GPB in an SQL database ========================= It is possible to store GPB data in MariaDB today - one can declare a binary blob column and use it to store GPB messages. Storing and retrieving entire messages will be the only available operations, though, as the server has no idea about the GPB format. It is apparent that ability to peek inside GPB data from SQL layer would be of great advantage: one would be able to - select only certain fields or parts of GPB messages - filter records based on the values of GPB fields - etc performing such operations at SQL layer will allow to reduce client<->server traffic right away, and will open path to getting the best possible performance. 3. Encoding to use for dynamic columns ====================================== The data should be coded into the proto buffer in the following format: <field_number><value_type><value>[<field_number><value_type><value>...] Where field_number is a number between 0-65536 that identifes the field <value_type> is a enum of type 'Item_result' <value> is the value coded in proto format. In other words, we should have no nested or complex structure. 4. How to store and access data in a protocol buffer from SQL ============================================================ User-friendly/meaningful access to GPB fields requires knowledge of GPB field names and types, which are not available from GPB message itself (see "GPB encoding overview" section). To make things easy for the user, we will at first stage provide SQL functions to manipulate a string that is actually in proto format. The functions we should provde are: proto_get(gpb, field_number, type) This return the field tagged with 'field_number' from the 'gpb' buffer. Example: proto_get(blob, 1, varchar) -> Returns field number 1 as varchar proto_put(gpb, field_number, value) This returns a new gbp buffer with the new value appended. Example: proto_put(proto_put(blob, 1, 1), 2, "hello") 5. Extension for future ======================= In the future we may want to access data based on name and get MariaDB to automaticly know the correct type. To do this we need to be able to store a definition for the content of the proto buffer somewhere. DecisionToMake: How to pass the server the GPB definition? First idea: add a CREATE TABLE parameter which will specify the definition itself. LOW-LEVEL DESIGN: * GPB tarball contains a protocol definition for .proto file structure itself and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. * We should have both server-side support and client-side support (client side means functions in libmysqlclient so that user can select the full BLOB and extract fields in the application). * Add some kind of header to the GPB blob to support versioning and future extensibility. * Add complete syntax description (update, add, drop, exists, ...). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Monty): Add support for google protocol buffers (34)
by worklog-noreply＠askmonty.org 11 Feb '10

11 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add support for google protocol buffers CREATION DATE..: Tue, 21 Jul 2009, 21:11 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 34 (http://askmonty.org/worklog/?tid=34) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High Level Description modified. --- /tmp/wklog.34.old.17915 2010-02-11 17:59:17.000000000 +0000 +++ /tmp/wklog.34.new.17915 2010-02-11 17:59:17.000000000 +0000 @@ -1,5 +1,21 @@ -Add support for Google Protocol Buffers (further GPB). It should be possible -to have columns that store GPB-encoded data, as well as use SQL constructs to +Add support for dynamic columns: + +- A column that can hold information from many columns +- One can instantly add or remove column data + +This is a useful feature for any store type of application, where you want to +store different type of information for different kind of items. + +For example, for shoes you want to store: material, size, colour, maker +For a computer you want to store ram, hard disk size etc... + +In a normal 'relational' system you would need to a table for each type. +With dynamic columns you have all common items as fixed fields (like +product_code, manufacturer, price) and the rest stored in a dynamic column. + +The proposed idea is to store the dynamic information in a blob in +Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. + Any support for indexing GPB data is outside of scope of this WL entry. -=-=(Knielsen - Fri, 22 Jan 2010, 11:38)=-=- Low Level Design modified. --- /tmp/wklog.34.old.29965 2010-01-22 11:38:57.000000000 +0200 +++ /tmp/wklog.34.new.29965 2010-01-22 11:38:57.000000000 +0200 @@ -2,3 +2,12 @@ and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. +* We should have both server-side support and client-side support (client side + means functions in libmysqlclient so that user can select the full BLOB and + extract fields in the application). + +* Add some kind of header to the GPB blob to support versioning and future + extensibility. + +* Add complete syntax description (update, add, drop, exists, ...). + -=-=(Psergey - Tue, 21 Jul 2009, 21:13)=-=- Low Level Design modified. --- /tmp/wklog.34.old.6462 2009-07-21 21:13:13.000000000 +0300 +++ /tmp/wklog.34.new.6462 2009-07-21 21:13:13.000000000 +0300 @@ -1 +1,4 @@ +* GPB tarball contains a protocol definition for .proto file structure itself + and a parser for text form of .proto file which then exposes the parsed + file via standard GPB message navigation API. -=-=(Psergey - Tue, 21 Jul 2009, 21:12)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.6399 2009-07-21 21:12:23.000000000 +0300 +++ /tmp/wklog.34.new.6399 2009-07-21 21:12:23.000000000 +0300 @@ -1 +1,78 @@ +<contents> +1. GPB Encoding overview +2. GPB in an SQL database +2.1 Informing server about GPB field names and types +2.2 Addressing GPB fields +2.2.1 Option1: SQL Function +2.2.2 Option2: SQL columns +</contents> + + +1. GPB Encoding overview +======================== + +GBB is a compact encoding for structured and typed data. A unit of GPB data +(it is called message) is only partially self-describing: it's possible to +iterate over its parts, but, quoting the spec + +http://code.google.com/apis/protocolbuffers/docs/encoding.html: + " the name and declared type for each field can only be determined on the + decoding end by referencing the message type's definition (i.e. the .proto + file). " + +2. GPB in an SQL database +========================= + +It is possible to store GPB data in MariaDB today - one can declare a binary +blob column and use it to store GPB messages. Storing and retrieving entire +messages will be the only available operations, though, as the server has no +idea about the GPB format. +It is apparent that ability to peek inside GPB data from SQL layer would be of +great advantage: one would be able to +- select only certain fields or parts of GPB messages +- filter records based on the values of GPB fields +- etc +performing such operations at SQL layer will allow to reduce client<->server +traffic right away, and will open path to getting the best possible +performance. + +2.1 Informing server about GPB field names and types +---------------------------------------------------- +User-friendly/meaningful access to GPB fields requires knowledge of GPB field +names and types, which are not available from GPB message itself (see "GPB +encoding overview" section). + +So the first issue to be addressed is to get the server to know the definition +of stored messages. We intend to assume that all records have GPB messages +that conform to a certain single definition, which gives one definition per +GPB field. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify either the +definition itself or path to .proto file with the definition. + +2.2 Addressing GPB fields +------------------------- +We'll need to provide a way to access GPB fields. This can be complicated as +structures that are encoded in GPB message can be nested and recursive. + +2.2.1 Option1: SQL Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Introduce an SQL function GPB_FIELD(path) which will return contents of the +field. +- Return type of the function will be determined from GPB message definition. +- For path, we can use XPath selector (a subset of XPath) syntax. + +(TODO ^ the above needs to be specified in more detail. is the selector as +simple as filesystem path or we allow quantifiers (with predicates?)?) + +2.2.2 Option2: SQL columns +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make GPB columns to be accessible as SQL columns. +This approach has problems: +- It might be hard to implement code-wise + - (TODO will Virtual columns patch help??) +- It is not clear how to access fields from nested structures. Should we allow + quoted names like `foo/bar[2]/baz' ? + DESCRIPTION: Add support for dynamic columns: - A column that can hold information from many columns - One can instantly add or remove column data This is a useful feature for any store type of application, where you want to store different type of information for different kind of items. For example, for shoes you want to store: material, size, colour, maker For a computer you want to store ram, hard disk size etc... In a normal 'relational' system you would need to a table for each type. With dynamic columns you have all common items as fixed fields (like product_code, manufacturer, price) and the rest stored in a dynamic column. The proposed idea is to store the dynamic information in a blob in Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. Any support for indexing GPB data is outside of scope of this WL entry. HIGH-LEVEL SPECIFICATION: <contents> 1. GPB Encoding overview 2. GPB in an SQL database 2.1 Informing server about GPB field names and types 2.2 Addressing GPB fields 2.2.1 Option1: SQL Function 2.2.2 Option2: SQL columns </contents> 1. GPB Encoding overview ======================== GBB is a compact encoding for structured and typed data. A unit of GPB data (it is called message) is only partially self-describing: it's possible to iterate over its parts, but, quoting the spec http://code.google.com/apis/protocolbuffers/docs/encoding.html: " the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file). " 2. GPB in an SQL database ========================= It is possible to store GPB data in MariaDB today - one can declare a binary blob column and use it to store GPB messages. Storing and retrieving entire messages will be the only available operations, though, as the server has no idea about the GPB format. It is apparent that ability to peek inside GPB data from SQL layer would be of great advantage: one would be able to - select only certain fields or parts of GPB messages - filter records based on the values of GPB fields - etc performing such operations at SQL layer will allow to reduce client<->server traffic right away, and will open path to getting the best possible performance. 2.1 Informing server about GPB field names and types ---------------------------------------------------- User-friendly/meaningful access to GPB fields requires knowledge of GPB field names and types, which are not available from GPB message itself (see "GPB encoding overview" section). So the first issue to be addressed is to get the server to know the definition of stored messages. We intend to assume that all records have GPB messages that conform to a certain single definition, which gives one definition per GPB field. DecisionToMake: How to pass the server the GPB definition? First idea: add a CREATE TABLE parameter which will specify either the definition itself or path to .proto file with the definition. 2.2 Addressing GPB fields ------------------------- We'll need to provide a way to access GPB fields. This can be complicated as structures that are encoded in GPB message can be nested and recursive. 2.2.1 Option1: SQL Function ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Introduce an SQL function GPB_FIELD(path) which will return contents of the field. - Return type of the function will be determined from GPB message definition. - For path, we can use XPath selector (a subset of XPath) syntax. (TODO ^ the above needs to be specified in more detail. is the selector as simple as filesystem path or we allow quantifiers (with predicates?)?) 2.2.2 Option2: SQL columns ~~~~~~~~~~~~~~~~~~~~~~~~~~ Make GPB columns to be accessible as SQL columns. This approach has problems: - It might be hard to implement code-wise - (TODO will Virtual columns patch help??) - It is not clear how to access fields from nested structures. Should we allow quoted names like `foo/bar[2]/baz' ? LOW-LEVEL DESIGN: * GPB tarball contains a protocol definition for .proto file structure itself and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. * We should have both server-side support and client-side support (client side means functions in libmysqlclient so that user can select the full BLOB and extract fields in the application). * Add some kind of header to the GPB blob to support versioning and future extensibility. * Add complete syntax description (update, add, drop, exists, ...). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Monty): Add support for google protocol buffers (34)
by worklog-noreply＠askmonty.org 11 Feb '10

11 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add support for google protocol buffers CREATION DATE..: Tue, 21 Jul 2009, 21:11 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 34 (http://askmonty.org/worklog/?tid=34) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Monty - Thu, 11 Feb 2010, 19:59)=-=- High Level Description modified. --- /tmp/wklog.34.old.17915 2010-02-11 17:59:17.000000000 +0000 +++ /tmp/wklog.34.new.17915 2010-02-11 17:59:17.000000000 +0000 @@ -1,5 +1,21 @@ -Add support for Google Protocol Buffers (further GPB). It should be possible -to have columns that store GPB-encoded data, as well as use SQL constructs to +Add support for dynamic columns: + +- A column that can hold information from many columns +- One can instantly add or remove column data + +This is a useful feature for any store type of application, where you want to +store different type of information for different kind of items. + +For example, for shoes you want to store: material, size, colour, maker +For a computer you want to store ram, hard disk size etc... + +In a normal 'relational' system you would need to a table for each type. +With dynamic columns you have all common items as fixed fields (like +product_code, manufacturer, price) and the rest stored in a dynamic column. + +The proposed idea is to store the dynamic information in a blob in +Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. + Any support for indexing GPB data is outside of scope of this WL entry. -=-=(Knielsen - Fri, 22 Jan 2010, 11:38)=-=- Low Level Design modified. --- /tmp/wklog.34.old.29965 2010-01-22 11:38:57.000000000 +0200 +++ /tmp/wklog.34.new.29965 2010-01-22 11:38:57.000000000 +0200 @@ -2,3 +2,12 @@ and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. +* We should have both server-side support and client-side support (client side + means functions in libmysqlclient so that user can select the full BLOB and + extract fields in the application). + +* Add some kind of header to the GPB blob to support versioning and future + extensibility. + +* Add complete syntax description (update, add, drop, exists, ...). + -=-=(Psergey - Tue, 21 Jul 2009, 21:13)=-=- Low Level Design modified. --- /tmp/wklog.34.old.6462 2009-07-21 21:13:13.000000000 +0300 +++ /tmp/wklog.34.new.6462 2009-07-21 21:13:13.000000000 +0300 @@ -1 +1,4 @@ +* GPB tarball contains a protocol definition for .proto file structure itself + and a parser for text form of .proto file which then exposes the parsed + file via standard GPB message navigation API. -=-=(Psergey - Tue, 21 Jul 2009, 21:12)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.6399 2009-07-21 21:12:23.000000000 +0300 +++ /tmp/wklog.34.new.6399 2009-07-21 21:12:23.000000000 +0300 @@ -1 +1,78 @@ +<contents> +1. GPB Encoding overview +2. GPB in an SQL database +2.1 Informing server about GPB field names and types +2.2 Addressing GPB fields +2.2.1 Option1: SQL Function +2.2.2 Option2: SQL columns +</contents> + + +1. GPB Encoding overview +======================== + +GBB is a compact encoding for structured and typed data. A unit of GPB data +(it is called message) is only partially self-describing: it's possible to +iterate over its parts, but, quoting the spec + +http://code.google.com/apis/protocolbuffers/docs/encoding.html: + " the name and declared type for each field can only be determined on the + decoding end by referencing the message type's definition (i.e. the .proto + file). " + +2. GPB in an SQL database +========================= + +It is possible to store GPB data in MariaDB today - one can declare a binary +blob column and use it to store GPB messages. Storing and retrieving entire +messages will be the only available operations, though, as the server has no +idea about the GPB format. +It is apparent that ability to peek inside GPB data from SQL layer would be of +great advantage: one would be able to +- select only certain fields or parts of GPB messages +- filter records based on the values of GPB fields +- etc +performing such operations at SQL layer will allow to reduce client<->server +traffic right away, and will open path to getting the best possible +performance. + +2.1 Informing server about GPB field names and types +---------------------------------------------------- +User-friendly/meaningful access to GPB fields requires knowledge of GPB field +names and types, which are not available from GPB message itself (see "GPB +encoding overview" section). + +So the first issue to be addressed is to get the server to know the definition +of stored messages. We intend to assume that all records have GPB messages +that conform to a certain single definition, which gives one definition per +GPB field. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify either the +definition itself or path to .proto file with the definition. + +2.2 Addressing GPB fields +------------------------- +We'll need to provide a way to access GPB fields. This can be complicated as +structures that are encoded in GPB message can be nested and recursive. + +2.2.1 Option1: SQL Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Introduce an SQL function GPB_FIELD(path) which will return contents of the +field. +- Return type of the function will be determined from GPB message definition. +- For path, we can use XPath selector (a subset of XPath) syntax. + +(TODO ^ the above needs to be specified in more detail. is the selector as +simple as filesystem path or we allow quantifiers (with predicates?)?) + +2.2.2 Option2: SQL columns +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make GPB columns to be accessible as SQL columns. +This approach has problems: +- It might be hard to implement code-wise + - (TODO will Virtual columns patch help??) +- It is not clear how to access fields from nested structures. Should we allow + quoted names like `foo/bar[2]/baz' ? + DESCRIPTION: Add support for dynamic columns: - A column that can hold information from many columns - One can instantly add or remove column data This is a useful feature for any store type of application, where you want to store different type of information for different kind of items. For example, for shoes you want to store: material, size, colour, maker For a computer you want to store ram, hard disk size etc... In a normal 'relational' system you would need to a table for each type. With dynamic columns you have all common items as fixed fields (like product_code, manufacturer, price) and the rest stored in a dynamic column. The proposed idea is to store the dynamic information in a blob in Google Protocol Buffers (further GPB) format and use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. Any support for indexing GPB data is outside of scope of this WL entry. HIGH-LEVEL SPECIFICATION: <contents> 1. GPB Encoding overview 2. GPB in an SQL database 2.1 Informing server about GPB field names and types 2.2 Addressing GPB fields 2.2.1 Option1: SQL Function 2.2.2 Option2: SQL columns </contents> 1. GPB Encoding overview ======================== GBB is a compact encoding for structured and typed data. A unit of GPB data (it is called message) is only partially self-describing: it's possible to iterate over its parts, but, quoting the spec http://code.google.com/apis/protocolbuffers/docs/encoding.html: " the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file). " 2. GPB in an SQL database ========================= It is possible to store GPB data in MariaDB today - one can declare a binary blob column and use it to store GPB messages. Storing and retrieving entire messages will be the only available operations, though, as the server has no idea about the GPB format. It is apparent that ability to peek inside GPB data from SQL layer would be of great advantage: one would be able to - select only certain fields or parts of GPB messages - filter records based on the values of GPB fields - etc performing such operations at SQL layer will allow to reduce client<->server traffic right away, and will open path to getting the best possible performance. 2.1 Informing server about GPB field names and types ---------------------------------------------------- User-friendly/meaningful access to GPB fields requires knowledge of GPB field names and types, which are not available from GPB message itself (see "GPB encoding overview" section). So the first issue to be addressed is to get the server to know the definition of stored messages. We intend to assume that all records have GPB messages that conform to a certain single definition, which gives one definition per GPB field. DecisionToMake: How to pass the server the GPB definition? First idea: add a CREATE TABLE parameter which will specify either the definition itself or path to .proto file with the definition. 2.2 Addressing GPB fields ------------------------- We'll need to provide a way to access GPB fields. This can be complicated as structures that are encoded in GPB message can be nested and recursive. 2.2.1 Option1: SQL Function ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Introduce an SQL function GPB_FIELD(path) which will return contents of the field. - Return type of the function will be determined from GPB message definition. - For path, we can use XPath selector (a subset of XPath) syntax. (TODO ^ the above needs to be specified in more detail. is the selector as simple as filesystem path or we allow quantifiers (with predicates?)?) 2.2.2 Option2: SQL columns ~~~~~~~~~~~~~~~~~~~~~~~~~~ Make GPB columns to be accessible as SQL columns. This approach has problems: - It might be hard to implement code-wise - (TODO will Virtual columns patch help??) - It is not clear how to access fields from nested structures. Should we allow quoted names like `foo/bar[2]/baz' ? LOW-LEVEL DESIGN: * GPB tarball contains a protocol definition for .proto file structure itself and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. * We should have both server-side support and client-side support (client side means functions in libmysqlclient so that user can select the full BLOB and extract fields in the application). * Add some kind of header to the GPB blob to support versioning and future extensibility. * Add complete syntax description (update, add, drop, exists, ...). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

Re: [Maria-developers] Rev 2740: Group commit for maria storage engine. in file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/
by Oleksandr Byelkin 11 Feb '10

11 Feb '10

Hi! 10 февр. 2010, в 21:38, Sergei Golubchik написал(а): [skip] >>> Why use my_atomic_store32 ? >> >> As I understood idea of atomic operation it is guaranted that we will >> read consistent value (not one byte from one value and other one from >> other). Yes I remember your statement that on modern 32bit system >> you >> always get it consistent, then why we made atomic operations at all? > > Because my_atomic_store32() also adds a full memory barrier to the > atomic store operation. That is, if you do > > my_atomic_store32(&a, 1); > my_atomic_store32(&b, 2); > > and then in another thread > > if (my_atomic_load32(&b) == 2) > { > ... > here you can be sure that a==1, because a=1 was executed before > b=2. And neither compiler nor the cpu swapped two assignments. > } > In other words it is real current value of the variable in all threads. It looks like what I need.

2 1

[Maria-developers] Rev 2758: Subquery optimizations: backport in file:///home/psergey/dev/maria-5.3-subqueries-r3/
by Sergey Petrunya 11 Feb '10

11 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r3/ ------------------------------------------------------------ revno: 2758 revision-id: psergey(a)askmonty.org-20100211120315-o1hpcxl5lkbrbl25 parent: psergey(a)askmonty.org-20100209203217-al1k9h50zrlphy5d committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r3 timestamp: Thu 2010-02-11 15:03:15 +0300 message: Subquery optimizations: backport - Fix valgrind failure: do initialize Item::is_expensive_cache. === modified file 'sql/item.cc' --- a/sql/item.cc 2010-02-08 13:10:19 +0000 +++ b/sql/item.cc 2010-02-11 12:03:15 +0000 @@ -373,8 +373,8 @@ Item::Item(): - rsize(0), name(0), orig_name(0), name_length(0), fixed(0), - is_autogenerated_name(TRUE), + is_expensive_cache(-1), rsize(0), name(0), orig_name(0), name_length(0), + fixed(0), is_autogenerated_name(TRUE), collation(&my_charset_bin, DERIVATION_COERCIBLE) { marker= 0; @@ -410,6 +410,7 @@ tables. */ Item::Item(THD *thd, Item *item): + is_expensive_cache(-1), rsize(0), str_value(item->str_value), name(item->name), === modified file 'sql/item.h' --- a/sql/item.h 2010-01-28 13:48:33 +0000 +++ b/sql/item.h 2010-02-11 12:03:15 +0000 @@ -513,6 +513,9 @@ enum traverse_order { POSTFIX, PREFIX }; + /* Cache of the result of is_expensive(). */ + int8 is_expensive_cache; + /* Reuse size, only used by SP local variable assignment, otherwize 0 */ uint rsize; @@ -878,9 +881,6 @@ static CHARSET_INFO *default_charset(); virtual CHARSET_INFO *compare_collation() { return NULL; } - /* Cache of the result of is_expensive(). */ - int8 is_expensive_cache; - virtual bool walk(Item_processor processor, bool walk_subquery, uchar *arg) { return (this->*processor)(arg);

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2815: Added option --temporary-tables to test speed of temporary tables
by noreply＠launchpad.net 10 Feb '10

10 Feb '10

------------------------------------------------------------ revno: 2815 committer: Michael Widenius <monty(a)askmonty.org> branch nick: maria-5.1 timestamp: Wed 2010-02-10 23:26:06 +0200 message: Added option --temporary-tables to test speed of temporary tables added: mysql-test/suite/parts/t/partition_repair_myisam-master.opt modified: sql-bench/bench-init.pl.sh sql-bench/server-cfg.sh sql-bench/test-connect.sh sql-bench/test-create.sh -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2814: When one does a drop table, the indexes are not flushed to disk before drop anymore (with MyISAM/...
by noreply＠launchpad.net 10 Feb '10

10 Feb '10

------------------------------------------------------------ revno: 2814 committer: Michael Widenius <monty(a)askmonty.org> branch nick: maria-5.1 timestamp: Wed 2010-02-10 21:06:24 +0200 message: When one does a drop table, the indexes are not flushed to disk before drop anymore (with MyISAM/Maria) myisam-recover options changed from OFF to 'DEFAULT' to get less change of data loss when using MyISAM. (The disadvantage is that changed MyISAM tables will be checked at access time; Use --myisam-recover=OFF for old behavior) Don't call extra(HA_EXTRA_FORCE_REOPEN) in ALTER TABLE if table is locked as this will mark table as crashed! Added assert to detect if we accidently would use MyISAM versioning in MySQL modified: include/my_base.h mysql-test/mysql-test-run.pl mysql-test/r/sp-destruct.result mysql-test/r/variables.result mysql-test/r/view.result mysql-test/suite/maria/t/maria-recovery2-master.opt mysql-test/t/sp-destruct.test mysql-test/t/view.test sql/lock.cc sql/mysql_priv.h sql/mysqld.cc sql/sql_base.cc sql/sql_delete.cc sql/sql_table.cc sql/table.cc sql/table.h storage/maria/ha_maria.cc storage/maria/ma_blockrec.c storage/maria/ma_close.c storage/maria/ma_extra.c storage/maria/ma_locking.c storage/maria/ma_recovery.c storage/maria/maria_def.h storage/myisam/mi_close.c storage/myisam/mi_extra.c storage/myisam/mi_open.c storage/myisam/myisamdef.h -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2815)
by Michael Widenius 10 Feb '10

10 Feb '10

#At lp:maria based on revid:monty@askmonty.org-20100210190624-38ucdn8y98k1v1zd 2815 Michael Widenius 2010-02-10 Added option --temporary-tables to test speed of temporary tables added: mysql-test/suite/parts/t/partition_repair_myisam-master.opt modified: sql-bench/bench-init.pl.sh sql-bench/server-cfg.sh sql-bench/test-connect.sh sql-bench/test-create.sh per-file messages: mysql-test/suite/parts/t/partition_repair_myisam-master.opt Added missing file from last push sql-bench/bench-init.pl.sh Added options: --temporary-tables to test speed of temporary tables sql-bench/server-cfg.sh Added limit for number of temporary tables one can create sql-bench/test-connect.sh Skip test that doesn't work with temporary tables. sql-bench/test-create.sh Added limit for number of temporary tables one can create === added file 'mysql-test/suite/parts/t/partition_repair_myisam-master.opt' --- a/mysql-test/suite/parts/t/partition_repair_myisam-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/parts/t/partition_repair_myisam-master.opt 2010-02-10 21:26:06 +0000 @@ -0,0 +1 @@ +--myisam-recover=off === modified file 'sql-bench/bench-init.pl.sh' --- a/sql-bench/bench-init.pl.sh 2010-02-09 17:17:04 +0000 +++ b/sql-bench/bench-init.pl.sh 2010-02-10 21:26:06 +0000 @@ -39,7 +39,7 @@ require "$pwd/server-cfg" || die "Can't $|=1; # Output data immediately -$opt_skip_test=$opt_skip_create=$opt_skip_delete=$opt_verbose=$opt_fast_insert=$opt_lock_tables=$opt_debug=$opt_skip_delete=$opt_fast=$opt_force=$opt_log=$opt_use_old_results=$opt_help=$opt_odbc=$opt_small_test=$opt_small_tables=$opt_samll_key_tables=$opt_stage=$opt_old_headers=$opt_die_on_errors=$opt_tcpip=$opt_random=$opt_only_missing_tests=0; +$opt_skip_test=$opt_skip_create=$opt_skip_delete=$opt_verbose=$opt_fast_insert=$opt_lock_tables=$opt_debug=$opt_skip_delete=$opt_fast=$opt_force=$opt_log=$opt_use_old_results=$opt_help=$opt_odbc=$opt_small_test=$opt_small_tables=$opt_samll_key_tables=$opt_stage=$opt_old_headers=$opt_die_on_errors=$opt_tcpip=$opt_random=$opt_only_missing_tests=$opt_temporary_tables=0; $opt_cmp=$opt_user=$opt_password=$opt_connect_options=$opt_connect_command= ""; $opt_server="mysql"; $opt_dir="output"; $opt_host="localhost";$opt_database="test"; @@ -59,7 +59,7 @@ $log_prog_args=join(" ", skip_arguments( "use-old-results","skip-test", "optimization","hw", "machine", "dir", "suffix", "log")); -GetOptions("skip-test=s","comments=s","cmp=s","server=s","user=s","host=s","database=s","password=s","loop-count=i","row-count=i","skip-create","skip-delete","verbose","fast-insert","lock-tables","debug","fast","force","field-count=i","regions=i","groups=i","time-limit=i","log","use-old-results","machine=s","dir=s","suffix=s","help","odbc","small-test","small-tables","small-key-tables","stage=i","threads=i","random","old-headers","die-on-errors","create-options=s","hires","tcpip","silent","optimization=s","hw=s","socket=s","connect-options=s","connect-command=s","only-missing-tests") || usage(); +GetOptions("skip-test=s","comments=s","cmp=s","server=s","user=s","host=s","database=s","password=s","loop-count=i","row-count=i","skip-create","skip-delete","verbose","fast-insert","lock-tables","debug","fast","force","field-count=i","regions=i","groups=i","time-limit=i","log","use-old-results","machine=s","dir=s","suffix=s","help","odbc","small-test","small-tables","small-key-tables","stage=i","threads=i","random","old-headers","die-on-errors","create-options=s","hires","tcpip","silent","optimization=s","hw=s","socket=s","connect-options=s","connect-command=s","only-missing-tests","temporary-tables") || usage(); usage() if ($opt_help); $server=get_server($opt_server,$opt_host,$opt_database,$opt_odbc, @@ -454,6 +454,9 @@ All benchmarks takes the following optio create all MySQL tables as InnoDB tables use: --create-options=ENGINE=InnoDB +--temporary-tables + Use temporary tables for all tests. + --database (Default $opt_database) In which database the test tables are created. === modified file 'sql-bench/server-cfg.sh' --- a/sql-bench/server-cfg.sh 2010-02-09 17:17:04 +0000 +++ b/sql-bench/server-cfg.sh 2010-02-10 21:26:06 +0000 @@ -159,6 +159,7 @@ sub new $limits{'max_index'} = 16; # Max number of keys $limits{'max_index_parts'} = 16; # Max segments/key $limits{'max_tables'} = (($machine || '') =~ "^win") ? 5000 : 65000; + $limits{'max_temporary_tables'}= 400; $limits{'max_text_size'} = 1000000; # Good enough for tests $limits{'multi_drop'} = 1; # Drop table can take many tables $limits{'order_by_position'} = 1; # Can use 'ORDER BY 1' @@ -189,6 +190,7 @@ sub new $self->{'transactions'} = 1; # Transactions enabled $limits{'max_columns'} = 90; # Max number of columns in table $limits{'max_tables'} = 32; # No comments + $limits{'max_temporary_tables'}= $limits{"max_tables"}; } if (defined($main::opt_create_options) && $main::opt_create_options =~ /engine=bdb/i) @@ -200,6 +202,7 @@ sub new { $limits{'working_blobs'} = 0; # Blobs not implemented yet $limits{'max_tables'} = 500; + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $self->{'transactions'} = 1; # Transactions enabled } @@ -270,7 +273,14 @@ sub create my($self,$table_name,$fields,$index,$options) = @_; my($query,@queries); - $query="create table $table_name ("; + if ($main::opt_temporary_tables) + { + $query="create temporary table $table_name ("; + } + else + { + $query="create table $table_name ("; + } foreach $field (@$fields) { # $field =~ s/ decimal/ double(10,2)/i; @@ -393,6 +403,7 @@ sub new $limits{'max_conditions'} = 74; $limits{'max_columns'} = 75; $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $limits{'max_text_size'} = 32000; $limits{'query_size'} = 65535; $limits{'max_index'} = 5; @@ -622,7 +633,9 @@ sub new $limits{'max_conditions'} = 9999; # This makes Pg real slow $limits{'max_index'} = 64; # Big enough $limits{'max_index_parts'} = 16; - $limits{'max_tables'} = 5000; # 10000 crashes pg 7.0.2 + $limits{'max_tables'} = 65000; + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 65000; # Good enough for test $limits{'multi_drop'} = 1; $limits{'order_by_position'} = 1; @@ -873,6 +886,8 @@ sub new $limits{'max_conditions'} = 9999; # Probably big enough $limits{'max_columns'} = 2000; # From crash-me $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 65492; # According to tests $limits{'query_size'} = 65535; # Probably a limit $limits{'max_index'} = 64; # Probably big enough @@ -1104,6 +1119,7 @@ sub new # above this value .... but can handle 2419 columns # maybe something for crash-me ... but how to check ??? $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $limits{'max_text_size'} = 4095; # max returned .... $limits{'query_size'} = 65535; # Not a limit, big enough $limits{'max_index'} = 64; # Big enough @@ -1374,6 +1390,8 @@ sub new $limits{'max_conditions'} = 9999; # (Actually not a limit) $limits{'max_columns'} = 254; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 2000; # Limit for blob test-connect $limits{'query_size'} = 65525; # Max size with default buffers. $limits{'max_index'} = 16; # Max number of keys @@ -1647,6 +1665,8 @@ sub new $limits{'max_column_name'} = 18; # max table and column name $limits{'max_columns'} = 994; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_index'} = 64; # Max number of keys $limits{'max_index_parts'} = 15; # Max segments/key $limits{'max_text_size'} = 65535; # Max size with default buffers. ?? @@ -1835,6 +1855,8 @@ sub new $limits{'max_conditions'} = 97; # We get 'Query is too complex' $limits{'max_columns'} = 255; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 255; # Max size with default buffers. $limits{'query_size'} = 65535; # Not a limit, big enough $limits{'max_index'} = 32; # Max number of keys @@ -2020,6 +2042,8 @@ sub new $limits{'max_conditions'} = 1030; # We get 'Query is too complex' $limits{'max_columns'} = 250; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 9830; # Max size with default buffers. $limits{'query_size'} = 9830; # Max size with default buffers. $limits{'max_index'} = 64; # Max number of keys @@ -2216,6 +2240,8 @@ sub new $limits{'max_conditions'} = 1030; # We get 'Query is too complex' $limits{'max_columns'} = 250; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 9830; # Max size with default buffers. $limits{'query_size'} = 9830; # Max size with default buffers. $limits{'max_index'} = 64; # Max number of keys @@ -2448,6 +2474,8 @@ sub new $limits{'max_conditions'} = 50; # (Actually not a limit) $limits{'max_columns'} = 254; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 2000; # Limit for blob test-connect $limits{'query_size'} = 65525; # Max size with default buffers. $limits{'max_index'} = 16; # Max number of keys @@ -2652,6 +2680,8 @@ sub new $limits{'max_conditions'} = 418; # We get 'Query is too complex' $limits{'max_columns'} = 500; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; + $limits{'max_text_size'} = 254; # Max size with default buffers. $limits{'query_size'} = 254; # Max size with default buffers. $limits{'max_index'} = 48; # Max number of keys @@ -2830,6 +2860,7 @@ sub new $limits{'max_conditions'} = 9999; # (Actually not a limit) $limits{'max_columns'} = 252; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $limits{'max_text_size'} = 15000; # Max size with default buffers. $limits{'query_size'} = 1000000; # Max size with default buffers. $limits{'max_index'} = 32; # Max number of keys @@ -3032,6 +3063,7 @@ sub new $limits{'max_conditions'} = 9999; # (Actually not a limit) $limits{'max_columns'} = 252; # Max number of columns in table $limits{'max_tables'} = 65000; # Should be big enough + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $limits{'max_text_size'} = 15000; # Max size with default buffers. $limits{'query_size'} = 1000000; # Max size with default buffers. $limits{'max_index'} = 65000; # Max number of keys @@ -3228,6 +3260,7 @@ sub new # The following should be 8192, but is smaller because Frontbase crashes.. $limits{'max_columns'} = 150; # Max number of columns in table $limits{'max_tables'} = 5000; # 10000 crashed FrontBase + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $limits{'max_text_size'} = 65000; # Max size with default buffers. $limits{'query_size'} = 8000000; # Max size with default buffers. $limits{'max_index'} = 38; # Max number of keys @@ -3440,6 +3473,7 @@ sub new $limits{'max_conditions'} = 9999; # (Actually not a limit) * $limits{'max_columns'} = 1023; # Max number of columns in table * $limits{'max_tables'} = 65000; # Should be big enough * unlimited actually + $limits{'max_temporary_tables'}= $limits{"max_tables"}; $limits{'max_text_size'} = 15000; # Max size with default buffers. $limits{'query_size'} = 64*1024; # Max size with default buffers. *64 kb by default. May be set by system variable $limits{'max_index'} = 510; # Max number of keys * === modified file 'sql-bench/test-connect.sh' --- a/sql-bench/test-connect.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/test-connect.sh 2010-02-10 21:26:06 +0000 @@ -161,41 +161,48 @@ if ($opt_fast && defined($server->{vacuu { $server->vacuum(0,\$dbh); } -$dbh->disconnect; +if (!$main::opt_temporary_tables) +{ + $dbh->disconnect; +} # # First test connect/select/disconnect # -print "Testing connect/select 1 row from table/disconnect\n"; +if (!$main::opt_temporary_tables) +{ + print "Testing connect/select 1 row from table/disconnect\n"; -$loop_time=new Benchmark; -$errors=0; + $loop_time=new Benchmark; + $errors=0; -for ($i=0 ; $i < $small_loop_count ; $i++) -{ - for ($j=0; $j < $max_test ; $j++) + for ($i=0 ; $i < $small_loop_count ; $i++) { - last if ($dbh = DBI->connect($server->{'data_source'}, $opt_user, $opt_password)); - $errors++; - } - die $DBI::errstr if ($j == $max_test); + for ($j=0; $j < $max_test ; $j++) + { + last if ($dbh = DBI->connect($server->{'data_source'}, $opt_user, $opt_password)); + $errors++; + } + die $DBI::errstr if ($j == $max_test); - $sth = $dbh->do("select a,i,s,$i from bench1") # Select * from table with 1 record + $sth = $dbh->do("select a,i,s,$i from bench1") # Select * from table with 1 record or die $DBI::errstr; - $dbh->disconnect; -} + $dbh->disconnect; + } -$end_time=new Benchmark; -print "Warning: $errors connections didn't work without a time delay\n" if ($errors); -print "Time to connect+select_1_row ($small_loop_count): " . + $end_time=new Benchmark; + print "Warning: $errors connections didn't work without a time delay\n" if ($errors); + print "Time to connect+select_1_row ($small_loop_count): " . timestr(timediff($end_time, $loop_time),"all") . "\n\n"; + $dbh = $server->connect(); +} + # # The same test, but without connect/disconnect # print "Testing select 1 row from table\n"; -$dbh = $server->connect(); $loop_time=new Benchmark; for ($i=0 ; $i < $opt_loop_count ; $i++) === modified file 'sql-bench/test-create.sh' --- a/sql-bench/test-create.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/test-create.sh 2010-02-10 21:26:06 +0000 @@ -47,7 +47,15 @@ if ($opt_small_test) $create_loop_count/=1000; } -$max_tables=min($limits->{'max_tables'},$opt_loop_count); +if ($opt_temporary_tables) +{ + $max_tables=min($limits->{'max_tables'},$opt_loop_count); +} +else +{ + $max_tables=min($limits->{'max_tables'},$opt_loop_count); + $max_tables=400; +} if ($opt_small_test) { @@ -71,7 +79,7 @@ $dbh = $server->connect(); if ($opt_force) # If tables used in this test exist, drop 'em { print "Okay..Let's make sure that our tables don't exist yet.\n\n"; - for ($i=1 ; $i <= $max_tables ; $i++) + for ($i=1 ; $i <= max($max_tables, $create_loop_count) ; $i++) { $dbh->do("drop table bench_$i" . $server->{'drop_attr'}); } @@ -245,7 +253,7 @@ for ($i=2 ; $i <= $keys ; $i++) } $loop_time=new Benchmark; -for ($i=1 ; $i <= $opt_loop_count ; $i++) +for ($i=1 ; $i <= $create_loop_count ; $i++) { do_many($dbh,$server->create("bench_$i", \@fields, \@keys)); $dbh->do("drop table bench_$i" . $server->{'drop_attr'}) or die $DBI::errstr;

1 0

[Maria-developers] Rev 2740: Group commit for maria storage engine. in file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/
by sanja＠askmonty.org 10 Feb '10

10 Feb '10

At file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/ ------------------------------------------------------------ revno: 2740 revision-id: sanja(a)askmonty.org-20100210205026-8l8veoi8dbon5cwl parent: knielsen(a)knielsen-hq.org-20100201190519-b9uktnn90rwwiile committer: sanja(a)askmonty.org branch nick: work-maria-5.2-groupcommit timestamp: Wed 2010-02-10 22:50:26 +0200 message: Group commit for maria storage engine. === added file 'mysql-test/suite/maria/r/group_commit.result' --- a/mysql-test/suite/maria/r/group_commit.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/maria/r/group_commit.result 2010-02-10 20:50:26 +0000 @@ -0,0 +1,17 @@ +drop table if exists t1; +create table t1 (a int); +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +drop table t1; === modified file 'mysql-test/suite/maria/r/maria3.result' --- a/mysql-test/suite/maria/r/maria3.result 2009-09-18 01:04:43 +0000 +++ b/mysql-test/suite/maria/r/maria3.result 2010-02-10 20:50:26 +0000 @@ -306,6 +306,8 @@ maria_block_size 8192 maria_checkpoint_interval 30 maria_force_start_after_recovery_failures 0 +maria_group_commit none +maria_group_commit_interval 0 maria_log_file_size 4294959104 maria_log_purge_type immediate maria_max_sort_file_size 9223372036853727232 @@ -328,6 +330,7 @@ Maria_pagecache_reads # Maria_pagecache_write_requests # Maria_pagecache_writes # +Maria_transaction_log_syncs # create table t1 (b char(0)); insert into t1 values(NULL),(""); select length(b) from t1; === added file 'mysql-test/suite/maria/t/group_commit.test' --- a/mysql-test/suite/maria/t/group_commit.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/maria/t/group_commit.test 2010-02-10 20:50:26 +0000 @@ -0,0 +1,71 @@ +# Test different ways of syncing (mostly syntax) + +--disable_warnings +drop table if exists t1; +--enable_warnings + +create table t1 (a int); + +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +drop table t1; === modified file 'storage/maria/ha_maria.cc' --- a/storage/maria/ha_maria.cc 2009-12-03 11:34:11 +0000 +++ b/storage/maria/ha_maria.cc 2010-02-10 20:50:26 +0000 @@ -102,22 +102,40 @@ array_elements(maria_translog_purge_type_names) - 1, "", maria_translog_purge_type_names, NULL }; + +/* transactional log directory sync */ const char *maria_sync_log_dir_names[]= { "NEVER", "NEWFILE", "ALWAYS", NullS }; - TYPELIB maria_sync_log_dir_typelib= { array_elements(maria_sync_log_dir_names) - 1, "", maria_sync_log_dir_names, NULL }; +/* transactional log group commit */ +const char *maria_group_commit_names[]= +{ + "none", "hard", "soft", NullS +}; +TYPELIB maria_group_commit_typelib= +{ + array_elements(maria_group_commit_names) - 1, "", + maria_group_commit_names, NULL +}; + /** Interval between background checkpoints in seconds */ static ulong checkpoint_interval; static void update_checkpoint_interval(MYSQL_THD thd, struct st_mysql_sys_var *var, void *var_ptr, const void *save); +static void update_maria_group_commit(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save); +static void update_maria_group_commit_interval(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save); /** After that many consecutive recovery failures, remove logs */ static ulong force_start_after_recovery_failures; static void update_log_file_size(MYSQL_THD thd, @@ -164,6 +182,24 @@ NULL, update_log_file_size, TRANSLOG_FILE_SIZE, TRANSLOG_MIN_FILE_SIZE, 0xffffffffL, TRANSLOG_PAGE_SIZE); +static MYSQL_SYSVAR_ENUM(group_commit, maria_group_commit, + PLUGIN_VAR_RQCMDARG, + "Specifies maria group commit mode. " + "Possible values are \"none\" (no group commit), " + "\"hard\" (with waiting to actual commit), " + "\"soft\" (no wait for commit (DANGEROUS!!!))", + NULL, update_maria_group_commit, + TRANSLOG_GCOMMIT_NONE, &maria_group_commit_typelib); + +static MYSQL_SYSVAR_ULONG(group_commit_interval, maria_group_commit_interval, + PLUGIN_VAR_RQCMDARG, + "Interval between commite in microseconds (1/1000000c)." + " 0 stands for no waiting" + " for other threads to come and do a commit in \"hard\" mode and no" + " sync()/commit at all in \"soft\" mode. Option has only an effect" + " if maria_group_commit is used", + NULL, update_maria_group_commit_interval, 0, 0, UINT_MAX, 1); + static MYSQL_SYSVAR_ENUM(log_purge_type, log_purge_type, PLUGIN_VAR_RQCMDARG, "Specifies how maria transactional log will be purged. " @@ -3275,6 +3311,8 @@ MYSQL_SYSVAR(block_size), MYSQL_SYSVAR(checkpoint_interval), MYSQL_SYSVAR(force_start_after_recovery_failures), + MYSQL_SYSVAR(group_commit), + MYSQL_SYSVAR(group_commit_interval), MYSQL_SYSVAR(page_checksum), MYSQL_SYSVAR(log_dir_path), MYSQL_SYSVAR(log_file_size), @@ -3306,6 +3344,92 @@ } /** + @brief Updates group commit mode +*/ + +static void update_maria_group_commit(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save) +{ + ulong value= (ulong)*((long *)var_ptr); + DBUG_ENTER("update_maria_group_commit"); + DBUG_PRINT("enter", ("old value: %lu new value %lu rate %lu", + value, (ulong)(*(long *)save), + maria_group_commit_interval)); + /* old value */ + switch (value) { + case TRANSLOG_GCOMMIT_NONE: + break; + case TRANSLOG_GCOMMIT_HARD: + translog_hard_group_commit(FALSE); + break; + case TRANSLOG_GCOMMIT_SOFT: + translog_soft_sync(FALSE); + if (maria_group_commit_interval) + translog_soft_sync_end(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + value= *(ulong *)var_ptr= (ulong)(*(long *)save); + translog_sync(); + /* new value */ + switch (value) { + case TRANSLOG_GCOMMIT_NONE: + break; + case TRANSLOG_GCOMMIT_HARD: + translog_hard_group_commit(TRUE); + break; + case TRANSLOG_GCOMMIT_SOFT: + translog_soft_sync(TRUE); + /* variable change made under global lock so we can just read it */ + if (maria_group_commit_interval) + translog_soft_sync_start(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + DBUG_VOID_RETURN; +} + +/** + @brief Updates group commit interval +*/ + +static void update_maria_group_commit_interval(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save) +{ + ulong new_value= (ulong)*((long *)save); + ulong *value_ptr= (ulong*) var_ptr; + DBUG_ENTER("update_maria_group_commit_interval"); + DBUG_PRINT("enter", ("old value: %lu new value %lu group commit %lu", + *value_ptr, new_value, maria_group_commit)); + + /* variable change made under global lock so we can just read it */ + switch (maria_group_commit) { + case TRANSLOG_GCOMMIT_NONE: + *value_ptr= new_value; + translog_set_group_commit_interval(new_value); + break; + case TRANSLOG_GCOMMIT_HARD: + *value_ptr= new_value; + translog_set_group_commit_interval(new_value); + break; + case TRANSLOG_GCOMMIT_SOFT: + if (*value_ptr) + translog_soft_sync_end(); + translog_set_group_commit_interval(new_value); + if ((*value_ptr= new_value)) + translog_soft_sync_start(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + DBUG_VOID_RETURN; +} + +/** @brief Updates the transaction log file limit. */ @@ -3327,6 +3451,7 @@ {"Maria_pagecache_reads", (char*) &maria_pagecache_var.global_cache_read, SHOW_LONGLONG}, {"Maria_pagecache_write_requests", (char*) &maria_pagecache_var.global_cache_w_requests, SHOW_LONGLONG}, {"Maria_pagecache_writes", (char*) &maria_pagecache_var.global_cache_write, SHOW_LONGLONG}, + {"Maria_transaction_log_syncs", (char*) &translog_syncs, SHOW_LONGLONG}, {NullS, NullS, SHOW_LONG} }; === modified file 'storage/maria/ma_init.c' --- a/storage/maria/ma_init.c 2008-10-09 20:03:54 +0000 +++ b/storage/maria/ma_init.c 2010-02-10 20:50:26 +0000 @@ -82,6 +82,11 @@ maria_inited= maria_multi_threaded= FALSE; ft_free_stopwords(); ma_checkpoint_end(); + if (translog_status == TRANSLOG_OK) + { + translog_soft_sync_end(); + translog_sync(); + } if ((trid= trnman_get_max_trid()) > max_trid_in_control_file) { /* === modified file 'storage/maria/ma_loghandler.c' --- a/storage/maria/ma_loghandler.c 2010-01-06 21:27:53 +0000 +++ b/storage/maria/ma_loghandler.c 2010-02-10 20:50:26 +0000 @@ -18,6 +18,7 @@ #include "ma_blockrec.h" /* for some constants and in-write hooks */ #include "ma_key_recover.h" /* For some in-write hooks */ #include "ma_checkpoint.h" +#include "ma_servicethread.h" /* On Windows, neither my_open() nor my_sync() work for directories. @@ -47,6 +48,15 @@ #include <m_ctype.h> #endif +/** @brief protects checkpoint_in_progress */ +static pthread_mutex_t LOCK_soft_sync; +/** @brief for killing the background checkpoint thread */ +static pthread_cond_t COND_soft_sync; +/** @brief control structure for checkpoint background thread */ +static MA_SERVICE_THREAD_CONTROL soft_sync_control= + {THREAD_DEAD, FALSE, &LOCK_soft_sync, &COND_soft_sync}; + + /* transaction log file descriptor */ typedef struct st_translog_file { @@ -124,10 +134,24 @@ /* Previous buffer offset to detect it flush finish */ TRANSLOG_ADDRESS prev_buffer_offset; /* + If the buffer was forced to close it save value of its horizon + otherwise LSN_IMPOSSIBLE + */ + TRANSLOG_ADDRESS pre_force_close_horizon; + /* How much is written (or will be written when copy_to_buffer_in_progress become 0) to this buffer */ translog_size_t size; + /* + When moving from one log buffer to another, we write the last of the + previous buffer to file and then move to start using the new log + buffer. In the case of a part filed last page, this page is not moved + to the start of the new buffer but instead we set the 'skip_data' + variable to tell us how much data at the beginning of the buffer is not + relevant. + */ + uint skipped_data; /* File handler for this buffer */ TRANSLOG_FILE *file; /* Threads which are waiting for buffer filling/freeing */ @@ -304,6 +328,7 @@ */ pthread_mutex_t log_flush_lock; pthread_cond_t log_flush_cond; + pthread_cond_t new_goal_cond; /* Protects changing of headers of finished files (max_lsn) */ pthread_mutex_t file_header_lock; @@ -344,13 +369,39 @@ ulong log_purge_type= TRANSLOG_PURGE_IMMIDIATE; ulong log_file_size= TRANSLOG_FILE_SIZE; +/* sync() of log files directory mode */ ulong sync_log_dir= TRANSLOG_SYNC_DIR_NEWFILE; +ulong maria_group_commit= TRANSLOG_GCOMMIT_NONE; +ulong maria_group_commit_interval= 0; /* Marker for end of log */ static uchar end_of_log= 0; #define END_OF_LOG &end_of_log +/** + Switch for "soft" sync (no real sync() but periodical sync by service + thread) +*/ +static volatile my_bool soft_sync= FALSE; +/** + Switch for "hard" group commit mode +*/ +static volatile my_bool hard_group_commit= FALSE; +/** + File numbers interval which have to be sync() +*/ +static uint32 soft_sync_min= 0; +static uint32 soft_sync_max= 0; +static uint32 soft_need_sync= 1; +/** + stores interval in microseconds +*/ +static uint32 group_commit_wait= 0; enum enum_translog_status translog_status= TRANSLOG_UNINITED; +ulonglong translog_syncs= 0; /* Number of sync()s */ + +/* time of last flush */ +static ulonglong flush_start= 0; /* chunk types */ #define TRANSLOG_CHUNK_LSN 0x00 /* 0 chunk refer as LSN (head or tail */ @@ -980,12 +1031,17 @@ static TRANSLOG_FILE *get_current_logfile() { TRANSLOG_FILE *file; + DBUG_ENTER("get_current_logfile"); rw_rdlock(&log_descriptor.open_files_lock); + DBUG_PRINT("info", ("max_file: %lu min_file: %lu open_files: %lu", + (ulong) log_descriptor.max_file, + (ulong) log_descriptor.min_file, + (ulong) log_descriptor.open_files.elements)); DBUG_ASSERT(log_descriptor.max_file - log_descriptor.min_file + 1 == log_descriptor.open_files.elements); file= *dynamic_element(&log_descriptor.open_files, 0, TRANSLOG_FILE **); rw_unlock(&log_descriptor.open_files_lock); - return (file); + DBUG_RETURN(file); } uchar NEAR maria_trans_file_magic[]= @@ -1069,6 +1125,7 @@ static my_bool translog_max_lsn_to_header(File file, LSN lsn) { uchar lsn_buff[LSN_STORE_SIZE]; + my_bool rc; DBUG_ENTER("translog_max_lsn_to_header"); DBUG_PRINT("enter", ("File descriptor: %ld " "lsn: (%lu,0x%lx)", @@ -1077,11 +1134,17 @@ lsn_store(lsn_buff, lsn); - DBUG_RETURN(my_pwrite(file, lsn_buff, - LSN_STORE_SIZE, - (LOG_HEADER_DATA_SIZE - LSN_STORE_SIZE), - log_write_flags) != 0 || - my_sync(file, MYF(MY_WME)) != 0); + rc= (my_pwrite(file, lsn_buff, + LSN_STORE_SIZE, + (LOG_HEADER_DATA_SIZE - LSN_STORE_SIZE), + log_write_flags) != 0 || + my_sync(file, MYF(MY_WME)) != 0); + /* + We should not increase counter in case of error above, but it is so + unlikely that we can ignore this case + */ + translog_syncs++; + DBUG_RETURN(rc); } @@ -1423,7 +1486,9 @@ static my_bool translog_buffer_init(struct st_translog_buffer *buffer, int num) { DBUG_ENTER("translog_buffer_init"); - buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; + buffer->pre_force_close_horizon= + buffer->prev_last_lsn= buffer->last_lsn= + LSN_IMPOSSIBLE; DBUG_PRINT("info", ("last_lsn and prev_last_lsn set to 0 buffer: 0x%lx", (ulong) buffer)); @@ -1435,6 +1500,7 @@ memset(buffer->buffer, TRANSLOG_FILLER, TRANSLOG_WRITE_BUFFER); /* Buffer size */ buffer->size= 0; + buffer->skipped_data= 0; /* cond of thread which is waiting for buffer filling */ if (pthread_cond_init(&buffer->waiting_filling_buffer, 0)) DBUG_RETURN(1); @@ -1489,7 +1555,10 @@ TODO: sync only we have changed the log */ if (!file->is_sync) + { rc= my_sync(file->handler.file, MYF(MY_WME)); + translog_syncs++; + } rc|= my_close(file->handler.file, MYF(MY_WME)); my_free(file, MYF(0)); return test(rc); @@ -2044,7 +2113,8 @@ (ulong) LSN_OFFSET(log_descriptor.horizon), (ulong) LSN_OFFSET(log_descriptor.horizon))); DBUG_ASSERT(buffer_no == buffer->buffer_no); - buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; + buffer->pre_force_close_horizon= + buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; DBUG_PRINT("info", ("last_lsn and prev_last_lsn set to 0 buffer: 0x%lx", (ulong) buffer)); buffer->offset= log_descriptor.horizon; @@ -2052,6 +2122,7 @@ buffer->file= get_current_logfile(); buffer->overlay= 0; buffer->size= 0; + buffer->skipped_data= 0; translog_cursor_init(cursor, buffer, buffer_no); DBUG_PRINT("info", ("file: #%ld (%d) init cursor #%u: 0x%lx " "chaser: %d Size: %lu (%lu)", @@ -2523,6 +2594,7 @@ TRANSLOG_ADDRESS offset= buffer->offset; TRANSLOG_FILE *file= buffer->file; uint8 ver= buffer->ver; + uint skipped_data; DBUG_ENTER("translog_buffer_flush"); DBUG_PRINT("enter", ("Buffer: #%u 0x%lx file: %d offset: (%lu,0x%lx) size: %lu", @@ -2557,6 +2629,8 @@ disk */ file= buffer->file; + skipped_data= buffer->skipped_data; + DBUG_ASSERT(skipped_data < TRANSLOG_PAGE_SIZE); for (i= 0, pg= LSN_OFFSET(buffer->offset) / TRANSLOG_PAGE_SIZE; i < buffer->size; i+= TRANSLOG_PAGE_SIZE, pg++) @@ -2573,13 +2647,16 @@ DBUG_ASSERT(i + TRANSLOG_PAGE_SIZE <= buffer->size); if (translog_status != TRANSLOG_OK && translog_status != TRANSLOG_SHUTDOWN) DBUG_RETURN(1); - if (pagecache_inject(log_descriptor.pagecache, + if (pagecache_write_part(log_descriptor.pagecache, &file->handler, pg, 3, buffer->buffer + i, PAGECACHE_PLAIN_PAGE, PAGECACHE_LOCK_LEFT_UNLOCKED, - PAGECACHE_PIN_LEFT_UNPINNED, 0, - LSN_IMPOSSIBLE)) + PAGECACHE_PIN_LEFT_UNPINNED, + PAGECACHE_WRITE_DONE, 0, + LSN_IMPOSSIBLE, + skipped_data, + TRANSLOG_PAGE_SIZE - skipped_data)) { DBUG_PRINT("error", ("Can't write page (%lu,0x%lx) to pagecache, error: %d", @@ -2589,10 +2666,12 @@ translog_stop_writing(); DBUG_RETURN(1); } + skipped_data= 0; } file->is_sync= 0; - if (my_pwrite(file->handler.file, buffer->buffer, - buffer->size, LSN_OFFSET(buffer->offset), + if (my_pwrite(file->handler.file, buffer->buffer + buffer->skipped_data, + buffer->size - buffer->skipped_data, + LSN_OFFSET(buffer->offset) + buffer->skipped_data, log_write_flags)) { DBUG_PRINT("error", ("Can't write buffer (%lu,0x%lx) size %lu " @@ -2985,6 +3064,7 @@ uchar *from, *table= NULL; int is_last_unfinished_page; uint last_protected_sector= 0; + uint skipped_data= curr_buffer->skipped_data; TRANSLOG_FILE file_copy; uint8 ver= curr_buffer->ver; translog_wait_for_writers(curr_buffer); @@ -2997,7 +3077,38 @@ } DBUG_ASSERT(LSN_FILE_NO(addr) == LSN_FILE_NO(curr_buffer->offset)); from= curr_buffer->buffer + (addr - curr_buffer->offset); - memcpy(buffer, from, TRANSLOG_PAGE_SIZE); + if (skipped_data && addr == curr_buffer->offset) + { + /* + We read page part of which is not present in buffer, + so we should read absent part from file (page cache actually) + */ + file= get_logfile_by_number(file_no); + DBUG_ASSERT(file != NULL); + /* + it's ok to not lock the page because: + - The log handler has it's own page cache. + - There is only one thread that can access the log + cache at a time + */ + if (!(buffer= pagecache_read(log_descriptor.pagecache, + &file->handler, + LSN_OFFSET(addr) / TRANSLOG_PAGE_SIZE, + 3, buffer, + PAGECACHE_PLAIN_PAGE, + PAGECACHE_LOCK_LEFT_UNLOCKED, + NULL))) + DBUG_RETURN(NULL); + } + else + skipped_data= 0; /* Read after skipped in buffer data */ + /* + Now we have correct data in buffer up to 'skipped_data'. The + following memcpy() will move the data from the internal buffer + that was not yet on disk. + */ + memcpy(buffer + skipped_data, from + skipped_data, + TRANSLOG_PAGE_SIZE - skipped_data); /* We can use copy then in translog_page_validator() because it do not put it permanently somewhere. @@ -3291,6 +3402,7 @@ uint32 next_page_offset, page_rest; uint32 i; File fd; + int rc; TRANSLOG_VALIDATOR_DATA data; char path[FN_REFLEN]; uchar page_buff[TRANSLOG_PAGE_SIZE]; @@ -3316,14 +3428,19 @@ TRANSLOG_PAGE_SIZE); page_rest= next_page_offset - LSN_OFFSET(addr); memset(page_buff, TRANSLOG_FILLER, page_rest); - if ((fd= open_logfile_by_number_no_cache(LSN_FILE_NO(addr))) < 0 || - ((my_chsize(fd, next_page_offset, TRANSLOG_FILLER, MYF(MY_WME)) || - (page_rest && my_pwrite(fd, page_buff, page_rest, LSN_OFFSET(addr), - log_write_flags)) || - my_sync(fd, MYF(MY_WME))) | - my_close(fd, MYF(MY_WME))) || - (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && - sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)))) + rc= ((fd= open_logfile_by_number_no_cache(LSN_FILE_NO(addr))) < 0 || + ((my_chsize(fd, next_page_offset, TRANSLOG_FILLER, MYF(MY_WME)) || + (page_rest && my_pwrite(fd, page_buff, page_rest, LSN_OFFSET(addr), + log_write_flags)) || + my_sync(fd, MYF(MY_WME))))); + translog_syncs++; + rc|= (fd > 0 && my_close(fd, MYF(MY_WME))); + if (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS) + { + rc|= sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)); + translog_syncs++; + } + if (rc) DBUG_RETURN(1); /* fix the horizon */ @@ -3511,6 +3628,7 @@ pthread_mutex_init(&log_descriptor.dirty_buffer_mask_lock, MY_MUTEX_INIT_FAST) || pthread_cond_init(&log_descriptor.log_flush_cond, 0) || + pthread_cond_init(&log_descriptor.new_goal_cond, 0) || my_rwlock_init(&log_descriptor.open_files_lock, NULL) || my_init_dynamic_array(&log_descriptor.open_files, @@ -3912,7 +4030,6 @@ log_descriptor.flushed= log_descriptor.horizon; log_descriptor.in_buffers_only= log_descriptor.bc.buffer->offset; log_descriptor.max_lsn= LSN_IMPOSSIBLE; /* set to 0 */ - log_descriptor.previous_flush_horizon= log_descriptor.horizon; /* Now 'flushed' is set to 'horizon' value, but 'horizon' is (potentially) address of the next LSN and we want indicate that all LSNs that are @@ -3995,6 +4112,10 @@ It is beginning of the log => there is no LSNs in the log => There is no harm in leaving it "as-is". */ + log_descriptor.previous_flush_horizon= log_descriptor.horizon; + DBUG_PRINT("info", ("previous_flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor. + previous_flush_horizon))); DBUG_RETURN(0); } file_no--; @@ -4070,6 +4191,9 @@ translog_free_record_header(&rec); } } + log_descriptor.previous_flush_horizon= log_descriptor.horizon; + DBUG_PRINT("info", ("previous_flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor.previous_flush_horizon))); DBUG_RETURN(0); err: ma_message_no_user(0, "log initialization failed"); @@ -4157,6 +4281,7 @@ pthread_mutex_destroy(&log_descriptor.log_flush_lock); pthread_mutex_destroy(&log_descriptor.dirty_buffer_mask_lock); pthread_cond_destroy(&log_descriptor.log_flush_cond); + pthread_cond_destroy(&log_descriptor.new_goal_cond); rwlock_destroy(&log_descriptor.open_files_lock); delete_dynamic(&log_descriptor.open_files); delete_dynamic(&log_descriptor.unfinished_files); @@ -6885,11 +7010,11 @@ { translog_size_t res; DBUG_ENTER("translog_read_record_header_from_buffer"); - DBUG_ASSERT(translog_is_LSN_chunk(page[page_offset])); - DBUG_ASSERT(translog_status == TRANSLOG_OK || - translog_status == TRANSLOG_READONLY); DBUG_PRINT("info", ("page byte: 0x%x offset: %u", (uint) page[page_offset], (uint) page_offset)); + DBUG_ASSERT(translog_is_LSN_chunk(page[page_offset])); + DBUG_ASSERT(translog_status == TRANSLOG_OK || + translog_status == TRANSLOG_READONLY); buff->type= (page[page_offset] & TRANSLOG_REC_TYPE); buff->short_trid= uint2korr(page + page_offset + 1); DBUG_PRINT("info", ("Type %u, Short TrID %u, LSN (%lu,0x%lx)", @@ -7356,27 +7481,27 @@ "Buffer addr: (%lu,0x%lx) " "Page addr: (%lu,0x%lx) " "size: %lu (%lu) Pg: %u left: %u in progress %u", - (uint) log_descriptor.bc.buffer_no, - (ulong) log_descriptor.bc.buffer, - LSN_IN_PARTS(log_descriptor.bc.buffer->offset), + (uint) old_buffer_no, + (ulong) old_buffer, + LSN_IN_PARTS(old_buffer->offset), (ulong) LSN_FILE_NO(log_descriptor.horizon), (ulong) (LSN_OFFSET(log_descriptor.horizon) - log_descriptor.bc.current_page_fill), - (ulong) log_descriptor.bc.buffer->size, + (ulong) old_buffer->size, (ulong) (log_descriptor.bc.ptr -log_descriptor.bc. buffer->buffer), (uint) log_descriptor.bc.current_page_fill, (uint) left, - (uint) log_descriptor.bc.buffer-> + (uint) old_buffer-> copy_to_buffer_in_progress)); translog_lock_assert_owner(); LINT_INIT(current_page_fill); - new_buff_beginning= log_descriptor.bc.buffer->offset; - new_buff_beginning+= log_descriptor.bc.buffer->size; /* increase offset */ + new_buff_beginning= old_buffer->offset; + new_buff_beginning+= old_buffer->size; /* increase offset */ DBUG_ASSERT(log_descriptor.bc.ptr !=NULL); DBUG_ASSERT(LSN_FILE_NO(log_descriptor.horizon) == - LSN_FILE_NO(log_descriptor.bc.buffer->offset)); + LSN_FILE_NO(old_buffer->offset)); translog_check_cursor(&log_descriptor.bc); DBUG_ASSERT(left < TRANSLOG_PAGE_SIZE); if (left) @@ -7387,18 +7512,20 @@ */ DBUG_PRINT("info", ("left: %u", (uint) left)); + old_buffer->pre_force_close_horizon= + old_buffer->offset + old_buffer->size; /* decrease offset */ new_buff_beginning-= log_descriptor.bc.current_page_fill; current_page_fill= log_descriptor.bc.current_page_fill; memset(log_descriptor.bc.ptr, TRANSLOG_FILLER, left); - log_descriptor.bc.buffer->size+= left; + old_buffer->size+= left; DBUG_PRINT("info", ("Finish Page buffer #%u: 0x%lx " "Size: %lu", - (uint) log_descriptor.bc.buffer->buffer_no, - (ulong) log_descriptor.bc.buffer, - (ulong) log_descriptor.bc.buffer->size)); - DBUG_ASSERT(log_descriptor.bc.buffer->buffer_no == + (uint) old_buffer->buffer_no, + (ulong) old_buffer, + (ulong) old_buffer->size)); + DBUG_ASSERT(old_buffer->buffer_no == log_descriptor.bc.buffer_no); } else @@ -7509,11 +7636,21 @@ if (left) { - /* - TODO: do not copy beginning of the page if we have no CRC or sector - checks on - */ - memcpy(new_buffer->buffer, data, current_page_fill); + if (log_descriptor.flags & + (TRANSLOG_PAGE_CRC | TRANSLOG_SECTOR_PROTECTION)) + memcpy(new_buffer->buffer, data, current_page_fill); + else + { + /* + This page header does not change if we add more data to the page so + we can not copy it and will not overwrite later + */ + new_buffer->skipped_data= current_page_fill; +#ifndef DBUG_OFF + memset(new_buffer->buffer, 0xa5, current_page_fill); +#endif + DBUG_ASSERT(new_buffer->skipped_data < TRANSLOG_PAGE_SIZE); + } } old_buffer->next_buffer_offset= new_buffer->offset; translog_buffer_lock(new_buffer); @@ -7561,6 +7698,7 @@ { log_descriptor.next_pass_max_lsn= lsn; log_descriptor.max_lsn_requester= pthread_self(); + pthread_cond_broadcast(&log_descriptor.new_goal_cond); } while (flush_no == log_descriptor.flush_no) { @@ -7572,66 +7710,78 @@ /** - @brief Flush the log up to given LSN (included) - - @param lsn log record serial number up to which (inclusive) - the log has to be flushed - - @return Operation status + @brief sync() range of files (inclusive) and directory (by request) + + @param min min internal file number to flush + @param max max internal file number to flush + @param sync_dir need sync directory + + return Operation status @retval 0 OK @retval 1 Error - -*/ - -my_bool translog_flush(TRANSLOG_ADDRESS lsn) -{ - LSN sent_to_disk= LSN_IMPOSSIBLE; - TRANSLOG_ADDRESS flush_horizon; - uint fn, i; +*/ + +static my_bool translog_sync_files(uint32 min, uint32 max, + my_bool sync_dir) +{ + uint fn; + my_bool rc= 0; + ulonglong flush_interval; + DBUG_ENTER("translog_sync_files"); + DBUG_PRINT("info", ("min: %lu max: %lu sync dir: %d", + (ulong) min, (ulong) max, (int) sync_dir)); + DBUG_ASSERT(min <= max); + + flush_interval= group_commit_wait; + if (flush_interval) + flush_start= my_micro_time(); + for (fn= min; fn <= max; fn++) + { + TRANSLOG_FILE *file= get_logfile_by_number(fn); + DBUG_ASSERT(file != NULL); + if (!file->is_sync) + { + if (my_sync(file->handler.file, MYF(MY_WME))) + { + rc= 1; + translog_stop_writing(); + DBUG_RETURN(rc); + } + translog_syncs++; + file->is_sync= 1; + } + } + + if (sync_dir) + { + if (!(rc= sync_dir(log_descriptor.directory_fd, + MYF(MY_WME | MY_IGNORE_BADFD)))) + translog_syncs++; + } + + DBUG_RETURN(rc); +} + + +/* + @brief Flushes buffers with LSNs in them less or equal address <lsn> + + @param lsn address up to which all LSNs should be flushed, + can be reset to real last LSN address + @parem sent_to_disk returns 'sent to disk' position + @param flush_horizon returns horizon of the flush + + @note About terminology see comment to translog_flush(). +*/ + +void translog_flush_buffers(TRANSLOG_ADDRESS *lsn, + TRANSLOG_ADDRESS *sent_to_disk, + TRANSLOG_ADDRESS *flush_horizon) +{ dirty_buffer_mask_t dirty_buffer_mask; + uint i; uint8 last_buffer_no, start_buffer_no; - my_bool rc= 0; - DBUG_ENTER("translog_flush"); - DBUG_PRINT("enter", ("Flush up to LSN: (%lu,0x%lx)", LSN_IN_PARTS(lsn))); - DBUG_ASSERT(translog_status == TRANSLOG_OK || - translog_status == TRANSLOG_READONLY); - LINT_INIT(sent_to_disk); - - pthread_mutex_lock(&log_descriptor.log_flush_lock); - DBUG_PRINT("info", ("Everything is flushed up to (%lu,0x%lx)", - LSN_IN_PARTS(log_descriptor.flushed))); - if (cmp_translog_addr(log_descriptor.flushed, lsn) >= 0) - { - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - DBUG_RETURN(0); - } - if (log_descriptor.flush_in_progress) - { - translog_flush_set_new_goal_and_wait(lsn); - if (!pthread_equal(log_descriptor.max_lsn_requester, pthread_self())) - { - /* fix lsn if it was horizon */ - if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->last_lsn) > 0) - lsn= BUFFER_MAX_LSN(log_descriptor.bc.buffer); - translog_flush_wait_for_end(lsn); - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - DBUG_RETURN(0); - } - log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; - } - log_descriptor.flush_in_progress= 1; - flush_horizon= log_descriptor.previous_flush_horizon; - DBUG_PRINT("info", ("flush_in_progress is set")); - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - - translog_lock(); - if (log_descriptor.is_everything_flushed) - { - DBUG_PRINT("info", ("everything is flushed")); - rc= (translog_status == TRANSLOG_READONLY); - translog_unlock(); - goto out; - } + DBUG_ENTER("translog_flush_buffers"); /* We will recheck information when will lock buffers one by @@ -7656,15 +7806,15 @@ /* if LSN up to which we have to flush bigger then maximum LSN of previous buffer and at least one LSN was saved in the current buffer (last_lsn != - LSN_IMPOSSIBLE) then we better finish the current buffer. + LSN_IMPOSSIBLE) then we have to close the current buffer. */ - if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->prev_last_lsn) > 0 && + if (cmp_translog_addr(*lsn, log_descriptor.bc.buffer->prev_last_lsn) > 0 && log_descriptor.bc.buffer->last_lsn != LSN_IMPOSSIBLE) { struct st_translog_buffer *buffer= log_descriptor.bc.buffer; - lsn= log_descriptor.bc.buffer->last_lsn; /* fix lsn if it was horizon */ + *lsn= log_descriptor.bc.buffer->last_lsn; /* fix lsn if it was horizon */ DBUG_PRINT("info", ("LSN to flush fixed to last lsn: (%lu,0x%lx)", - LSN_IN_PARTS(log_descriptor.bc.buffer->last_lsn))); + LSN_IN_PARTS(log_descriptor.bc.buffer->last_lsn))); last_buffer_no= log_descriptor.bc.buffer_no; log_descriptor.is_everything_flushed= 1; translog_force_current_buffer_to_finish(); @@ -7676,8 +7826,10 @@ TRANSLOG_BUFFERS_NO); translog_unlock(); } - sent_to_disk= translog_get_sent_to_disk(); - if (cmp_translog_addr(lsn, sent_to_disk) > 0) + + /* flush buffers */ + *sent_to_disk= translog_get_sent_to_disk(); + if (cmp_translog_addr(*lsn, *sent_to_disk) > 0) { DBUG_PRINT("info", ("Start buffer #: %u last buffer #: %u", @@ -7697,53 +7849,238 @@ LSN_IN_PARTS(buffer->last_lsn), (buffer->file ? "dirty" : "closed"))); - if (buffer->prev_last_lsn <= lsn && + if (buffer->prev_last_lsn <= *lsn && buffer->file != NULL) { - DBUG_ASSERT(flush_horizon <= buffer->offset + buffer->size); - flush_horizon= buffer->offset + buffer->size; + DBUG_ASSERT(*flush_horizon <= buffer->offset + buffer->size); + *flush_horizon= (buffer->pre_force_close_horizon != LSN_IMPOSSIBLE ? + buffer->pre_force_close_horizon : + buffer->offset + buffer->size); + /* pre_force_close_horizon is reset during new buffer start */ + DBUG_PRINT("info", ("flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(*flush_horizon))); + DBUG_ASSERT(*flush_horizon <= log_descriptor.horizon); + translog_buffer_flush(buffer); } translog_buffer_unlock(buffer); i= (i + 1) % TRANSLOG_BUFFERS_NO; } while (i != last_buffer_no); - sent_to_disk= translog_get_sent_to_disk(); - } - - /* sync files from previous flush till current one */ - for (fn= LSN_FILE_NO(log_descriptor.flushed); fn <= LSN_FILE_NO(lsn); fn++) - { - TRANSLOG_FILE *file= get_logfile_by_number(fn); - DBUG_ASSERT(file != NULL); - if (!file->is_sync) - { - if (my_sync(file->handler.file, MYF(MY_WME))) + *sent_to_disk= translog_get_sent_to_disk(); + } + + DBUG_VOID_RETURN; +} + +/** + @brief Flush the log up to given LSN (included) + + @param lsn log record serial number up to which (inclusive) + the log has to be flushed + + @return Operation status + @retval 0 OK + @retval 1 Error + + @note + + - Non group commit logic: Commits made in passes. Thread which started + flush first is performing actual flush, other threads sets new goal (LSN) + of the next pass (if it is maximum) and waits for the pass end or just + wait for the pass end. + + - If hard group commit enabled and rate set to zero: + The first thread sends all changed buffers to disk. This is repeated + as long as there are new LSNs added. The process can not loop + forever because we have limited number of threads and they will wait + for the data to be synced. + Pseudo code: + + do + send changed buffers to disk + while new_goal + sync + + - If hard group commit switched ON and less than rate microseconds has + passed from last sync, then after buffers have been sent to disk + wait until rate microseconds has passed since last sync, do sync and return. + This ensures that if we call sync infrequently we don't do any waits. + + - If soft group commit enabled everything works as with 'non group commit' + but the thread doesn't do any real sync(). If rate is not zero the + sync() will be performed by a service thread with the given rate + when needed (new LSN appears). + + @note Terminology: + 'sent to disk' means written to disk but not sync()ed, + 'flushed' mean sent to disk and synced(). +*/ + +my_bool translog_flush(TRANSLOG_ADDRESS lsn) +{ + struct timespec abstime; + ulonglong flush_interval; + ulonglong time_spent; + LSN sent_to_disk= LSN_IMPOSSIBLE; + TRANSLOG_ADDRESS flush_horizon; + my_bool rc= 0; + my_bool hgroup_commit_at_start; + DBUG_ENTER("translog_flush"); + DBUG_PRINT("enter", ("Flush up to LSN: (%lu,0x%lx)", LSN_IN_PARTS(lsn))); + DBUG_ASSERT(translog_status == TRANSLOG_OK || + translog_status == TRANSLOG_READONLY); + LINT_INIT(sent_to_disk); + + pthread_mutex_lock(&log_descriptor.log_flush_lock); + DBUG_PRINT("info", ("Everything is flushed up to (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor.flushed))); + if (cmp_translog_addr(log_descriptor.flushed, lsn) >= 0) + { + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_RETURN(0); + } + if (log_descriptor.flush_in_progress) + { + translog_lock(); + /* fix lsn if it was horizon */ + if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->last_lsn) > 0) + lsn= BUFFER_MAX_LSN(log_descriptor.bc.buffer); + translog_unlock(); + translog_flush_set_new_goal_and_wait(lsn); + if (!pthread_equal(log_descriptor.max_lsn_requester, pthread_self())) + { + /* + translog_flush_wait_for_end() release log_flush_lock while is + waiting then acquire it again + */ + translog_flush_wait_for_end(lsn); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_RETURN(0); + } + log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; + } + log_descriptor.flush_in_progress= 1; + flush_horizon= log_descriptor.previous_flush_horizon; + DBUG_PRINT("info", ("flush_in_progress is set, flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(flush_horizon))); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + + hgroup_commit_at_start= hard_group_commit; + if (hgroup_commit_at_start) + flush_interval= group_commit_wait; + + translog_lock(); + if (log_descriptor.is_everything_flushed) + { + DBUG_PRINT("info", ("everything is flushed")); + translog_unlock(); + pthread_mutex_lock(&log_descriptor.log_flush_lock); + goto out; + } + + for (;;) + { + /* Following function flushes buffers and makes translog_unlock() */ + translog_flush_buffers(&lsn, &sent_to_disk, &flush_horizon); + + if (!hgroup_commit_at_start) + break; /* flush pass is ended */ + +retest: + /* + We do not check time here because pthread_mutex_lock rarely takes + a lot of time so we can sacrifice a bit precision to performance + (taking into account that my_micro_time() might be expensive call). + */ + if (flush_interval == 0) + break; /* flush pass is ended */ + + pthread_mutex_lock(&log_descriptor.log_flush_lock); + if (log_descriptor.next_pass_max_lsn == LSN_IMPOSSIBLE) + { + if (flush_interval == 0 || + (time_spent= (my_micro_time() - flush_start)) >= flush_interval) { - rc= 1; - translog_stop_writing(); - sent_to_disk= LSN_IMPOSSIBLE; - goto out; + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + break; } - file->is_sync= 1; - } - } - - if (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && - (LSN_FILE_NO(log_descriptor.previous_flush_horizon) != - LSN_FILE_NO(flush_horizon) || - ((LSN_OFFSET(log_descriptor.previous_flush_horizon) - 1) / - TRANSLOG_PAGE_SIZE) != - ((LSN_OFFSET(flush_horizon) - 1) / TRANSLOG_PAGE_SIZE))) - rc|= sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)); + DBUG_PRINT("info", ("flush waits: %llu interval: %llu spent: %llu", + flush_interval - time_spent, + flush_interval, time_spent)); + /* wait time or next goal */ + set_timespec_nsec(abstime, flush_interval - time_spent); + pthread_cond_timedwait(&log_descriptor.new_goal_cond, + &log_descriptor.log_flush_lock, + &abstime); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_PRINT("info", ("retest conditions")); + goto retest; + } + + /* take next goal */ + lsn= log_descriptor.next_pass_max_lsn; + log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; + /* prevent other thread from continue */ + log_descriptor.max_lsn_requester= pthread_self(); + DBUG_PRINT("info", ("flush took next goal: (%lu,0x%lx)", + LSN_IN_PARTS(lsn))); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + + /* next flush pass */ + DBUG_PRINT("info", ("next flush pass")); + translog_lock(); + } + + /* + sync() files from previous flush till current one + */ + if (!soft_sync || hgroup_commit_at_start) + { + if ((rc= + translog_sync_files(LSN_FILE_NO(log_descriptor.flushed), + LSN_FILE_NO(lsn), + sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && + (LSN_FILE_NO(log_descriptor. + previous_flush_horizon) != + LSN_FILE_NO(flush_horizon) || + (LSN_OFFSET(log_descriptor. + previous_flush_horizon) / + TRANSLOG_PAGE_SIZE) != + (LSN_OFFSET(flush_horizon) / + TRANSLOG_PAGE_SIZE))))) + { + sent_to_disk= LSN_IMPOSSIBLE; + pthread_mutex_lock(&log_descriptor.log_flush_lock); + goto out; + } + /* keep values for soft sync() and forced sync() actual */ + { + uint32 fileno= LSN_FILE_NO(lsn); + my_atomic_rwlock_wrlock(&soft_sync_rwl); + my_atomic_store32(&soft_sync_min, fileno); + my_atomic_store32(&soft_sync_max, fileno); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + } + } + else + { + my_atomic_rwlock_wrlock(&soft_sync_rwl); + my_atomic_store32(&soft_sync_max, LSN_FILE_NO(lsn)); + my_atomic_store32(&soft_need_sync, 1); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + } + + DBUG_ASSERT(flush_horizon <= log_descriptor.horizon); + + pthread_mutex_lock(&log_descriptor.log_flush_lock); log_descriptor.previous_flush_horizon= flush_horizon; out: - pthread_mutex_lock(&log_descriptor.log_flush_lock); if (sent_to_disk != LSN_IMPOSSIBLE) log_descriptor.flushed= sent_to_disk; log_descriptor.flush_in_progress= 0; log_descriptor.flush_no++; DBUG_PRINT("info", ("flush_in_progress is dropped")); - pthread_mutex_unlock(&log_descriptor.log_flush_lock);\ + pthread_mutex_unlock(&log_descriptor.log_flush_lock); pthread_cond_broadcast(&log_descriptor.log_flush_cond); DBUG_RETURN(rc); } @@ -8113,6 +8450,8 @@ my_bool translog_purge(TRANSLOG_ADDRESS low) { uint32 last_need_file= LSN_FILE_NO(low); + uint32 min_unsync; + int soft; TRANSLOG_ADDRESS horizon= translog_get_horizon(); int rc= 0; DBUG_ENTER("translog_purge"); @@ -8120,12 +8459,26 @@ DBUG_ASSERT(translog_status == TRANSLOG_OK || translog_status == TRANSLOG_READONLY); + soft= soft_sync; + my_atomic_rwlock_wrlock(&soft_sync_rwl); + min_unsync= my_atomic_load32(&soft_sync_min); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + DBUG_PRINT("info", ("min_unsync: %lu", (ulong) min_unsync)); + if (soft && min_unsync < last_need_file) + { + last_need_file= min_unsync; + DBUG_PRINT("info", ("last_need_file set to %lu", (ulong)last_need_file)); + } + pthread_mutex_lock(&log_descriptor.purger_lock); + DBUG_PRINT("info", ("last_lsn_checked file: %lu:", + (ulong) log_descriptor.last_lsn_checked)); if (LSN_FILE_NO(log_descriptor.last_lsn_checked) < last_need_file) { uint32 i; uint32 min_file= translog_first_file(horizon, 1); DBUG_ASSERT(min_file != 0); /* log is already started */ + DBUG_PRINT("info", ("min_file: %lu:",(ulong) min_file)); for(i= min_file; i < last_need_file && rc == 0; i++) { LSN lsn= translog_get_file_max_lsn_stored(i); @@ -8356,6 +8709,159 @@ } + +/** + Sets soft sync mode + + @param mode TRUE if we need switch soft sync on else off +*/ + +void translog_soft_sync(my_bool mode) +{ + soft_sync= mode; +} + + +/** + Sets hard group commit + + @param mode TRUE if we need switch hard group commit on else off +*/ + +void translog_hard_group_commit(my_bool mode) +{ + hard_group_commit= mode; +} + + +/** + @brief forced log sync (used when we are switching modes) +*/ + +void translog_sync() +{ + uint32 max= get_current_logfile()->number; + uint32 min; + DBUG_ENTER("ma_translog_sync"); + + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + if (!min) + min= max; + + translog_sync_files(min, max, sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS); + + DBUG_VOID_RETURN; +} + + +/** + @brief set rate for group commit + + @param interval interval to set. + + @note We use this function with additional variable because have to + restart service thread with new value which we can't make inside changing + variable routine (update_maria_group_commit_interval) +*/ + +void translog_set_group_commit_interval(uint32 interval) +{ + DBUG_ENTER("translog_set_group_commit_interval"); + group_commit_wait= interval; + DBUG_PRINT("info", ("wait: %llu", + (ulonglong)group_commit_wait)); + DBUG_VOID_RETURN; +} + + +/** + @brief syncing service thread +*/ + +static pthread_handler_t +ma_soft_sync_background( void *arg __attribute__((unused))) +{ + + my_thread_init(); + { + DBUG_ENTER("ma_soft_sync_background"); + for(;;) + { + ulonglong prev_loop= my_micro_time(); + ulonglong time, sleep; + uint32 min, max, sync_request; + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + max= my_atomic_load32(&soft_sync_max); + sync_request= my_atomic_load32(&soft_need_sync); + my_atomic_store32(&soft_sync_min, max); + my_atomic_store32(&soft_need_sync, 0); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + + sleep= group_commit_wait; + if (sync_request) + translog_sync_files(min, max, FALSE); + time= my_micro_time() - prev_loop; + if (time > sleep) + sleep= 0; + else + sleep-= time; + if (my_service_thread_sleep(&soft_sync_control, sleep)) + break; + } + my_service_thread_signal_end(&soft_sync_control); + my_thread_end(); + DBUG_RETURN(0); + } +} + + +/** + @brief Starts syncing thread +*/ + +int translog_soft_sync_start(void) +{ + pthread_t th; + int res= 0; + uint32 min, max; + DBUG_ENTER("translog_soft_sync_start"); + + /* check and init variables */ + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + max= my_atomic_load32(&soft_sync_max); + if (!max) + my_atomic_store32(&soft_sync_max, (max= get_current_logfile()->number)); + if (!min) + my_atomic_store32(&soft_sync_min, max); + my_atomic_store32(&soft_need_sync, 1); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + + if (!(res= ma_service_thread_control_init(&soft_sync_control))) + if (!(res= pthread_create(&th, NULL, ma_soft_sync_background, NULL))) + soft_sync_control.status= THREAD_RUNNING; + DBUG_RETURN(res); +} + + +/** + @brief Stops syncing thread +*/ + +void translog_soft_sync_end(void) +{ + DBUG_ENTER("translog_soft_sync_end"); + if (soft_sync_control.inited) + { + ma_service_thread_control_end(&soft_sync_control); + } + DBUG_VOID_RETURN; +} + + #ifdef MARIA_DUMP_LOG #include <my_getopt.h> extern void translog_example_table_init(); === modified file 'storage/maria/ma_loghandler.h' --- a/storage/maria/ma_loghandler.h 2009-01-15 22:25:53 +0000 +++ b/storage/maria/ma_loghandler.h 2010-02-10 20:50:26 +0000 @@ -342,6 +342,14 @@ TRANSLOG_SHUTDOWN /* going to shutdown the loghandler */ }; extern enum enum_translog_status translog_status; +extern ulonglong translog_syncs; /* Number of sync()s */ + +void translog_soft_sync(my_bool mode); +void translog_hard_group_commit(my_bool mode); +int translog_soft_sync_start(void); +void translog_soft_sync_end(void); +void translog_sync(); +void translog_set_group_commit_interval(uint32 interval); /* all the rest added because of recovery; should we make @@ -441,6 +449,14 @@ typedef enum { + TRANSLOG_GCOMMIT_NONE, + TRANSLOG_GCOMMIT_HARD, + TRANSLOG_GCOMMIT_SOFT +} enum_maria_group_commit; +extern ulong maria_group_commit; +extern ulong maria_group_commit_interval; +typedef enum +{ TRANSLOG_PURGE_IMMIDIATE, TRANSLOG_PURGE_EXTERNAL, TRANSLOG_PURGE_ONDEMAND

1 0

[Maria-developers] Rev 2740: Group commit for maria storage engine. in file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/
by sanja＠askmonty.org 10 Feb '10

10 Feb '10

At file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/ ------------------------------------------------------------ revno: 2740 revision-id: sanja(a)askmonty.org-20100209083259-ekki5zw4hbaeqpwh parent: knielsen(a)knielsen-hq.org-20100201190519-b9uktnn90rwwiile committer: sanja(a)askmonty.org branch nick: work-maria-5.2-groupcommit timestamp: Tue 2010-02-09 10:32:59 +0200 message: Group commit for maria storage engine. === added file 'mysql-test/suite/maria/r/group_commit.result' --- a/mysql-test/suite/maria/r/group_commit.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/maria/r/group_commit.result 2010-02-09 08:32:59 +0000 @@ -0,0 +1,17 @@ +drop table if exists t1; +create table t1 (a int); +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 0; +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 100; +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +drop table t1; === modified file 'mysql-test/suite/maria/r/maria3.result' --- a/mysql-test/suite/maria/r/maria3.result 2009-09-18 01:04:43 +0000 +++ b/mysql-test/suite/maria/r/maria3.result 2010-02-09 08:32:59 +0000 @@ -306,6 +306,8 @@ maria_block_size 8192 maria_checkpoint_interval 30 maria_force_start_after_recovery_failures 0 +maria_group_commit none +maria_group_commit_interval 0 maria_log_file_size 4294959104 maria_log_purge_type immediate maria_max_sort_file_size 9223372036853727232 @@ -328,6 +330,7 @@ Maria_pagecache_reads # Maria_pagecache_write_requests # Maria_pagecache_writes # +Maria_transaction_log_syncs # create table t1 (b char(0)); insert into t1 values(NULL),(""); select length(b) from t1; === added file 'mysql-test/suite/maria/t/group_commit.test' --- a/mysql-test/suite/maria/t/group_commit.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/maria/t/group_commit.test 2010-02-09 08:32:59 +0000 @@ -0,0 +1,71 @@ +# Test different ways of syncing (mostly syntax) + +--disable_warnings +drop table if exists t1; +--enable_warnings + +create table t1 (a int); + +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="HARD"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 0; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="SOFT"; +SET GLOBAL maria_group_commit_interval= 100; +--disable_query_log +let $num = 5000; +while ($num) +{ + insert into t1 values (1); + dec $num; +} +--enable_query_log +SET GLOBAL maria_group_commit="NONE"; +SET GLOBAL maria_group_commit_interval= 0; +drop table t1; === modified file 'storage/maria/ha_maria.cc' --- a/storage/maria/ha_maria.cc 2009-12-03 11:34:11 +0000 +++ b/storage/maria/ha_maria.cc 2010-02-09 08:32:59 +0000 @@ -102,22 +102,40 @@ array_elements(maria_translog_purge_type_names) - 1, "", maria_translog_purge_type_names, NULL }; + +/* transactional log directory sync */ const char *maria_sync_log_dir_names[]= { "NEVER", "NEWFILE", "ALWAYS", NullS }; - TYPELIB maria_sync_log_dir_typelib= { array_elements(maria_sync_log_dir_names) - 1, "", maria_sync_log_dir_names, NULL }; +/* transactional log group commit */ +const char *maria_group_commit_names[]= +{ + "none", "hard", "soft", NullS +}; +TYPELIB maria_group_commit_typelib= +{ + array_elements(maria_group_commit_names) - 1, "", + maria_group_commit_names, NULL +}; + /** Interval between background checkpoints in seconds */ static ulong checkpoint_interval; static void update_checkpoint_interval(MYSQL_THD thd, struct st_mysql_sys_var *var, void *var_ptr, const void *save); +static void update_maria_group_commit(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save); +static void update_maria_group_commit_interval(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save); /** After that many consecutive recovery failures, remove logs */ static ulong force_start_after_recovery_failures; static void update_log_file_size(MYSQL_THD thd, @@ -164,6 +182,24 @@ NULL, update_log_file_size, TRANSLOG_FILE_SIZE, TRANSLOG_MIN_FILE_SIZE, 0xffffffffL, TRANSLOG_PAGE_SIZE); +static MYSQL_SYSVAR_ENUM(group_commit, maria_group_commit, + PLUGIN_VAR_RQCMDARG, + "Specifies maria group commit mode. " + "Possible values are \"none\" (no group commit), " + "\"hard\" (with waiting to actual commit), " + "\"soft\" (no wait for commit (DANGEROUS!!!))", + NULL, update_maria_group_commit, + TRANSLOG_GCOMMIT_NONE, &maria_group_commit_typelib); + +static MYSQL_SYSVAR_ULONG(group_commit_interval, maria_group_commit_interval, + PLUGIN_VAR_RQCMDARG, + "Interval between commite in microseconds (1/1000000c)." + " 0 stands for no waiting" + "for other threads to come and do a commit in \"hard\" mode and no" + " sync()/commit at all in \"soft\" mode. Option has only an effect" + "if maria_group_commit is used", + NULL, update_maria_group_commit_interval, 0, 0, UINT_MAX, 1); + static MYSQL_SYSVAR_ENUM(log_purge_type, log_purge_type, PLUGIN_VAR_RQCMDARG, "Specifies how maria transactional log will be purged. " @@ -3275,6 +3311,8 @@ MYSQL_SYSVAR(block_size), MYSQL_SYSVAR(checkpoint_interval), MYSQL_SYSVAR(force_start_after_recovery_failures), + MYSQL_SYSVAR(group_commit), + MYSQL_SYSVAR(group_commit_interval), MYSQL_SYSVAR(page_checksum), MYSQL_SYSVAR(log_dir_path), MYSQL_SYSVAR(log_file_size), @@ -3306,6 +3344,92 @@ } /** + @brief Updates group commit mode +*/ + +static void update_maria_group_commit(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save) +{ + ulong value= (ulong)*((long *)var_ptr); + DBUG_ENTER("update_maria_group_commit"); + DBUG_PRINT("enter", ("old value: %lu new value %lu rate %lu", + value, (ulong)(*(long *)save), + maria_group_commit_interval)); + /* old value */ + switch (value) { + case TRANSLOG_GCOMMIT_NONE: + break; + case TRANSLOG_GCOMMIT_HARD: + translog_hard_group_commit(FALSE); + break; + case TRANSLOG_GCOMMIT_SOFT: + translog_soft_sync(FALSE); + if (maria_group_commit_interval) + translog_soft_sync_end(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + value= *(ulong *)var_ptr= (ulong)(*(long *)save); + translog_sync(); + /* new value */ + switch (value) { + case TRANSLOG_GCOMMIT_NONE: + break; + case TRANSLOG_GCOMMIT_HARD: + translog_hard_group_commit(TRUE); + break; + case TRANSLOG_GCOMMIT_SOFT: + translog_soft_sync(TRUE); + /* variable change made under global lock so we can just read it */ + if (maria_group_commit_interval) + translog_soft_sync_start(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + DBUG_VOID_RETURN; +} + +/** + @brief Updates group commit interval +*/ + +static void update_maria_group_commit_interval(MYSQL_THD thd, + struct st_mysql_sys_var *var, + void *var_ptr, const void *save) +{ + ulong new_value= (ulong)*((long *)save); + ulong *value_ptr= (ulong*) var_ptr; + DBUG_ENTER("update_maria_group_commit_interval"); + DBUG_PRINT("enter", ("old value: %lu new value %lu group commit %lu", + *value_ptr, new_value, maria_group_commit)); + + /* variable change made under global lock so we can just read it */ + switch (maria_group_commit) { + case TRANSLOG_GCOMMIT_NONE: + *value_ptr= new_value; + translog_set_group_commit_interval(new_value); + break; + case TRANSLOG_GCOMMIT_HARD: + *value_ptr= new_value; + translog_set_group_commit_interval(new_value); + break; + case TRANSLOG_GCOMMIT_SOFT: + if (*value_ptr) + translog_soft_sync_end(); + translog_set_group_commit_interval(new_value); + if ((*value_ptr= new_value)) + translog_soft_sync_start(); + break; + default: + DBUG_ASSERT(0); /* impossible */ + } + DBUG_VOID_RETURN; +} + +/** @brief Updates the transaction log file limit. */ @@ -3327,6 +3451,7 @@ {"Maria_pagecache_reads", (char*) &maria_pagecache_var.global_cache_read, SHOW_LONGLONG}, {"Maria_pagecache_write_requests", (char*) &maria_pagecache_var.global_cache_w_requests, SHOW_LONGLONG}, {"Maria_pagecache_writes", (char*) &maria_pagecache_var.global_cache_write, SHOW_LONGLONG}, + {"Maria_transaction_log_syncs", (char*) &translog_syncs, SHOW_LONGLONG}, {NullS, NullS, SHOW_LONG} }; === modified file 'storage/maria/ma_init.c' --- a/storage/maria/ma_init.c 2008-10-09 20:03:54 +0000 +++ b/storage/maria/ma_init.c 2010-02-09 08:32:59 +0000 @@ -82,6 +82,11 @@ maria_inited= maria_multi_threaded= FALSE; ft_free_stopwords(); ma_checkpoint_end(); + if (translog_status == TRANSLOG_OK) + { + translog_soft_sync_end(); + translog_sync(); + } if ((trid= trnman_get_max_trid()) > max_trid_in_control_file) { /* === modified file 'storage/maria/ma_loghandler.c' --- a/storage/maria/ma_loghandler.c 2010-01-06 21:27:53 +0000 +++ b/storage/maria/ma_loghandler.c 2010-02-09 08:32:59 +0000 @@ -18,6 +18,7 @@ #include "ma_blockrec.h" /* for some constants and in-write hooks */ #include "ma_key_recover.h" /* For some in-write hooks */ #include "ma_checkpoint.h" +#include "ma_servicethread.h" /* On Windows, neither my_open() nor my_sync() work for directories. @@ -47,6 +48,15 @@ #include <m_ctype.h> #endif +/** @brief protects checkpoint_in_progress */ +static pthread_mutex_t LOCK_soft_sync; +/** @brief for killing the background checkpoint thread */ +static pthread_cond_t COND_soft_sync; +/** @brief control structure for checkpoint background thread */ +static MA_SERVICE_THREAD_CONTROL soft_sync_control= + {THREAD_DEAD, FALSE, &LOCK_soft_sync, &COND_soft_sync}; + + /* transaction log file descriptor */ typedef struct st_translog_file { @@ -124,10 +134,20 @@ /* Previous buffer offset to detect it flush finish */ TRANSLOG_ADDRESS prev_buffer_offset; /* + If the buffer was forced to close it save value of its horizon + otherwise LSN_IMPOSSIBLE + */ + TRANSLOG_ADDRESS pre_force_close_horizon; + /* How much is written (or will be written when copy_to_buffer_in_progress become 0) to this buffer */ translog_size_t size; + /* + How much data was skipped during moving page from previous buffer + to this one (it is optimisation of forcing buffer to finish + */ + uint skipped_data; /* File handler for this buffer */ TRANSLOG_FILE *file; /* Threads which are waiting for buffer filling/freeing */ @@ -304,6 +324,7 @@ */ pthread_mutex_t log_flush_lock; pthread_cond_t log_flush_cond; + pthread_cond_t new_goal_cond; /* Protects changing of headers of finished files (max_lsn) */ pthread_mutex_t file_header_lock; @@ -344,13 +365,38 @@ ulong log_purge_type= TRANSLOG_PURGE_IMMIDIATE; ulong log_file_size= TRANSLOG_FILE_SIZE; +/* sync() of log files directory mode */ ulong sync_log_dir= TRANSLOG_SYNC_DIR_NEWFILE; +ulong maria_group_commit= TRANSLOG_GCOMMIT_NONE; +ulong maria_group_commit_interval= 0; /* Marker for end of log */ static uchar end_of_log= 0; #define END_OF_LOG &end_of_log +/** + Switch for "soft" sync (no real sync() but periodical sync by service + thread) +*/ +static volatile my_bool soft_sync= FALSE; +/** + Switch for "hard" group commit mode +*/ +static volatile my_bool hard_group_commit= FALSE; +/** + File numbers interval which have to be sync() +*/ +static uint32 soft_sync_min= 0; +static uint32 soft_sync_max= 0; +/** + stores interval in microseconds +*/ +static uint32 group_commit_wait= 0; enum enum_translog_status translog_status= TRANSLOG_UNINITED; +ulonglong translog_syncs= 0; /* Number of sync()s */ + +/* time of last flush */ +static ulonglong flush_start= 0; /* chunk types */ #define TRANSLOG_CHUNK_LSN 0x00 /* 0 chunk refer as LSN (head or tail */ @@ -980,12 +1026,17 @@ static TRANSLOG_FILE *get_current_logfile() { TRANSLOG_FILE *file; + DBUG_ENTER("get_current_logfile"); rw_rdlock(&log_descriptor.open_files_lock); + DBUG_PRINT("info", ("max_file: %lu min_file: %lu open_files: %lu", + (ulong) log_descriptor.max_file, + (ulong) log_descriptor.min_file, + (ulong) log_descriptor.open_files.elements)); DBUG_ASSERT(log_descriptor.max_file - log_descriptor.min_file + 1 == log_descriptor.open_files.elements); file= *dynamic_element(&log_descriptor.open_files, 0, TRANSLOG_FILE **); rw_unlock(&log_descriptor.open_files_lock); - return (file); + DBUG_RETURN(file); } uchar NEAR maria_trans_file_magic[]= @@ -1069,6 +1120,7 @@ static my_bool translog_max_lsn_to_header(File file, LSN lsn) { uchar lsn_buff[LSN_STORE_SIZE]; + my_bool rc; DBUG_ENTER("translog_max_lsn_to_header"); DBUG_PRINT("enter", ("File descriptor: %ld " "lsn: (%lu,0x%lx)", @@ -1077,11 +1129,13 @@ lsn_store(lsn_buff, lsn); - DBUG_RETURN(my_pwrite(file, lsn_buff, - LSN_STORE_SIZE, - (LOG_HEADER_DATA_SIZE - LSN_STORE_SIZE), - log_write_flags) != 0 || - my_sync(file, MYF(MY_WME)) != 0); + if (!(rc= (my_pwrite(file, lsn_buff, + LSN_STORE_SIZE, + (LOG_HEADER_DATA_SIZE - LSN_STORE_SIZE), + log_write_flags) != 0 || + my_sync(file, MYF(MY_WME)) != 0))) + translog_syncs++; + DBUG_RETURN(rc); } @@ -1423,7 +1477,9 @@ static my_bool translog_buffer_init(struct st_translog_buffer *buffer, int num) { DBUG_ENTER("translog_buffer_init"); - buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; + buffer->pre_force_close_horizon= + buffer->prev_last_lsn= buffer->last_lsn= + LSN_IMPOSSIBLE; DBUG_PRINT("info", ("last_lsn and prev_last_lsn set to 0 buffer: 0x%lx", (ulong) buffer)); @@ -1435,6 +1491,7 @@ memset(buffer->buffer, TRANSLOG_FILLER, TRANSLOG_WRITE_BUFFER); /* Buffer size */ buffer->size= 0; + buffer->skipped_data= 0; /* cond of thread which is waiting for buffer filling */ if (pthread_cond_init(&buffer->waiting_filling_buffer, 0)) DBUG_RETURN(1); @@ -1489,7 +1546,10 @@ TODO: sync only we have changed the log */ if (!file->is_sync) + { rc= my_sync(file->handler.file, MYF(MY_WME)); + translog_syncs++; + } rc|= my_close(file->handler.file, MYF(MY_WME)); my_free(file, MYF(0)); return test(rc); @@ -2044,7 +2104,8 @@ (ulong) LSN_OFFSET(log_descriptor.horizon), (ulong) LSN_OFFSET(log_descriptor.horizon))); DBUG_ASSERT(buffer_no == buffer->buffer_no); - buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; + buffer->pre_force_close_horizon= + buffer->prev_last_lsn= buffer->last_lsn= LSN_IMPOSSIBLE; DBUG_PRINT("info", ("last_lsn and prev_last_lsn set to 0 buffer: 0x%lx", (ulong) buffer)); buffer->offset= log_descriptor.horizon; @@ -2052,6 +2113,7 @@ buffer->file= get_current_logfile(); buffer->overlay= 0; buffer->size= 0; + buffer->skipped_data= 0; translog_cursor_init(cursor, buffer, buffer_no); DBUG_PRINT("info", ("file: #%ld (%d) init cursor #%u: 0x%lx " "chaser: %d Size: %lu (%lu)", @@ -2523,6 +2585,7 @@ TRANSLOG_ADDRESS offset= buffer->offset; TRANSLOG_FILE *file= buffer->file; uint8 ver= buffer->ver; + uint skipped_data; DBUG_ENTER("translog_buffer_flush"); DBUG_PRINT("enter", ("Buffer: #%u 0x%lx file: %d offset: (%lu,0x%lx) size: %lu", @@ -2557,6 +2620,8 @@ disk */ file= buffer->file; + skipped_data= buffer->skipped_data; + DBUG_ASSERT(skipped_data < TRANSLOG_PAGE_SIZE); for (i= 0, pg= LSN_OFFSET(buffer->offset) / TRANSLOG_PAGE_SIZE; i < buffer->size; i+= TRANSLOG_PAGE_SIZE, pg++) @@ -2573,13 +2638,16 @@ DBUG_ASSERT(i + TRANSLOG_PAGE_SIZE <= buffer->size); if (translog_status != TRANSLOG_OK && translog_status != TRANSLOG_SHUTDOWN) DBUG_RETURN(1); - if (pagecache_inject(log_descriptor.pagecache, + if (pagecache_write_part(log_descriptor.pagecache, &file->handler, pg, 3, buffer->buffer + i, PAGECACHE_PLAIN_PAGE, PAGECACHE_LOCK_LEFT_UNLOCKED, - PAGECACHE_PIN_LEFT_UNPINNED, 0, - LSN_IMPOSSIBLE)) + PAGECACHE_PIN_LEFT_UNPINNED, + PAGECACHE_WRITE_DONE, 0, + LSN_IMPOSSIBLE, + skipped_data, + TRANSLOG_PAGE_SIZE - skipped_data)) { DBUG_PRINT("error", ("Can't write page (%lu,0x%lx) to pagecache, error: %d", @@ -2589,10 +2657,12 @@ translog_stop_writing(); DBUG_RETURN(1); } + skipped_data= 0; } file->is_sync= 0; - if (my_pwrite(file->handler.file, buffer->buffer, - buffer->size, LSN_OFFSET(buffer->offset), + if (my_pwrite(file->handler.file, buffer->buffer + buffer->skipped_data, + buffer->size - buffer->skipped_data, + LSN_OFFSET(buffer->offset) + buffer->skipped_data, log_write_flags)) { DBUG_PRINT("error", ("Can't write buffer (%lu,0x%lx) size %lu " @@ -2985,6 +3055,7 @@ uchar *from, *table= NULL; int is_last_unfinished_page; uint last_protected_sector= 0; + uint skipped_data= curr_buffer->skipped_data; TRANSLOG_FILE file_copy; uint8 ver= curr_buffer->ver; translog_wait_for_writers(curr_buffer); @@ -2997,7 +3068,25 @@ } DBUG_ASSERT(LSN_FILE_NO(addr) == LSN_FILE_NO(curr_buffer->offset)); from= curr_buffer->buffer + (addr - curr_buffer->offset); - memcpy(buffer, from, TRANSLOG_PAGE_SIZE); + if (skipped_data > (addr - curr_buffer->offset)) + { + /* + We read page part of which is not present in buffer, + so we should read absent part from file (page cache actually) + */ + file= get_logfile_by_number(file_no); + DBUG_ASSERT(file != NULL); + buffer= pagecache_read(log_descriptor.pagecache, &file->handler, + LSN_OFFSET(addr) / TRANSLOG_PAGE_SIZE, + 3, buffer, + PAGECACHE_PLAIN_PAGE, + PAGECACHE_LOCK_LEFT_UNLOCKED, + NULL); + } + else + skipped_data= 0; /* Read after skipped in buffer data */ + memcpy(buffer + skipped_data, from + skipped_data, + TRANSLOG_PAGE_SIZE - skipped_data); /* We can use copy then in translog_page_validator() because it do not put it permanently somewhere. @@ -3291,6 +3380,7 @@ uint32 next_page_offset, page_rest; uint32 i; File fd; + int rc; TRANSLOG_VALIDATOR_DATA data; char path[FN_REFLEN]; uchar page_buff[TRANSLOG_PAGE_SIZE]; @@ -3316,14 +3406,19 @@ TRANSLOG_PAGE_SIZE); page_rest= next_page_offset - LSN_OFFSET(addr); memset(page_buff, TRANSLOG_FILLER, page_rest); - if ((fd= open_logfile_by_number_no_cache(LSN_FILE_NO(addr))) < 0 || - ((my_chsize(fd, next_page_offset, TRANSLOG_FILLER, MYF(MY_WME)) || - (page_rest && my_pwrite(fd, page_buff, page_rest, LSN_OFFSET(addr), - log_write_flags)) || - my_sync(fd, MYF(MY_WME))) | - my_close(fd, MYF(MY_WME))) || - (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && - sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)))) + rc= ((fd= open_logfile_by_number_no_cache(LSN_FILE_NO(addr))) < 0 || + ((my_chsize(fd, next_page_offset, TRANSLOG_FILLER, MYF(MY_WME)) || + (page_rest && my_pwrite(fd, page_buff, page_rest, LSN_OFFSET(addr), + log_write_flags)) || + my_sync(fd, MYF(MY_WME))))); + translog_syncs++; + rc|= (fd > 0 && my_close(fd, MYF(MY_WME))); + if (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS) + { + rc|= sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)); + translog_syncs++; + } + if (rc) DBUG_RETURN(1); /* fix the horizon */ @@ -3511,6 +3606,7 @@ pthread_mutex_init(&log_descriptor.dirty_buffer_mask_lock, MY_MUTEX_INIT_FAST) || pthread_cond_init(&log_descriptor.log_flush_cond, 0) || + pthread_cond_init(&log_descriptor.new_goal_cond, 0) || my_rwlock_init(&log_descriptor.open_files_lock, NULL) || my_init_dynamic_array(&log_descriptor.open_files, @@ -3912,7 +4008,6 @@ log_descriptor.flushed= log_descriptor.horizon; log_descriptor.in_buffers_only= log_descriptor.bc.buffer->offset; log_descriptor.max_lsn= LSN_IMPOSSIBLE; /* set to 0 */ - log_descriptor.previous_flush_horizon= log_descriptor.horizon; /* Now 'flushed' is set to 'horizon' value, but 'horizon' is (potentially) address of the next LSN and we want indicate that all LSNs that are @@ -3995,6 +4090,10 @@ It is beginning of the log => there is no LSNs in the log => There is no harm in leaving it "as-is". */ + log_descriptor.previous_flush_horizon= log_descriptor.horizon; + DBUG_PRINT("info", ("previous_flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor. + previous_flush_horizon))); DBUG_RETURN(0); } file_no--; @@ -4070,6 +4169,9 @@ translog_free_record_header(&rec); } } + log_descriptor.previous_flush_horizon= log_descriptor.horizon; + DBUG_PRINT("info", ("previous_flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor.previous_flush_horizon))); DBUG_RETURN(0); err: ma_message_no_user(0, "log initialization failed"); @@ -4157,6 +4259,7 @@ pthread_mutex_destroy(&log_descriptor.log_flush_lock); pthread_mutex_destroy(&log_descriptor.dirty_buffer_mask_lock); pthread_cond_destroy(&log_descriptor.log_flush_cond); + pthread_cond_destroy(&log_descriptor.new_goal_cond); rwlock_destroy(&log_descriptor.open_files_lock); delete_dynamic(&log_descriptor.open_files); delete_dynamic(&log_descriptor.unfinished_files); @@ -6885,11 +6988,11 @@ { translog_size_t res; DBUG_ENTER("translog_read_record_header_from_buffer"); - DBUG_ASSERT(translog_is_LSN_chunk(page[page_offset])); - DBUG_ASSERT(translog_status == TRANSLOG_OK || - translog_status == TRANSLOG_READONLY); DBUG_PRINT("info", ("page byte: 0x%x offset: %u", (uint) page[page_offset], (uint) page_offset)); + DBUG_ASSERT(translog_is_LSN_chunk(page[page_offset])); + DBUG_ASSERT(translog_status == TRANSLOG_OK || + translog_status == TRANSLOG_READONLY); buff->type= (page[page_offset] & TRANSLOG_REC_TYPE); buff->short_trid= uint2korr(page + page_offset + 1); DBUG_PRINT("info", ("Type %u, Short TrID %u, LSN (%lu,0x%lx)", @@ -7356,27 +7459,27 @@ "Buffer addr: (%lu,0x%lx) " "Page addr: (%lu,0x%lx) " "size: %lu (%lu) Pg: %u left: %u in progress %u", - (uint) log_descriptor.bc.buffer_no, - (ulong) log_descriptor.bc.buffer, - LSN_IN_PARTS(log_descriptor.bc.buffer->offset), + (uint) old_buffer_no, + (ulong) old_buffer, + LSN_IN_PARTS(old_buffer->offset), (ulong) LSN_FILE_NO(log_descriptor.horizon), (ulong) (LSN_OFFSET(log_descriptor.horizon) - log_descriptor.bc.current_page_fill), - (ulong) log_descriptor.bc.buffer->size, + (ulong) old_buffer->size, (ulong) (log_descriptor.bc.ptr -log_descriptor.bc. buffer->buffer), (uint) log_descriptor.bc.current_page_fill, (uint) left, - (uint) log_descriptor.bc.buffer-> + (uint) old_buffer-> copy_to_buffer_in_progress)); translog_lock_assert_owner(); LINT_INIT(current_page_fill); - new_buff_beginning= log_descriptor.bc.buffer->offset; - new_buff_beginning+= log_descriptor.bc.buffer->size; /* increase offset */ + new_buff_beginning= old_buffer->offset; + new_buff_beginning+= old_buffer->size; /* increase offset */ DBUG_ASSERT(log_descriptor.bc.ptr !=NULL); DBUG_ASSERT(LSN_FILE_NO(log_descriptor.horizon) == - LSN_FILE_NO(log_descriptor.bc.buffer->offset)); + LSN_FILE_NO(old_buffer->offset)); translog_check_cursor(&log_descriptor.bc); DBUG_ASSERT(left < TRANSLOG_PAGE_SIZE); if (left) @@ -7387,18 +7490,20 @@ */ DBUG_PRINT("info", ("left: %u", (uint) left)); + old_buffer->pre_force_close_horizon= + old_buffer->offset + old_buffer->size; /* decrease offset */ new_buff_beginning-= log_descriptor.bc.current_page_fill; current_page_fill= log_descriptor.bc.current_page_fill; memset(log_descriptor.bc.ptr, TRANSLOG_FILLER, left); - log_descriptor.bc.buffer->size+= left; + old_buffer->size+= left; DBUG_PRINT("info", ("Finish Page buffer #%u: 0x%lx " "Size: %lu", - (uint) log_descriptor.bc.buffer->buffer_no, - (ulong) log_descriptor.bc.buffer, - (ulong) log_descriptor.bc.buffer->size)); - DBUG_ASSERT(log_descriptor.bc.buffer->buffer_no == + (uint) old_buffer->buffer_no, + (ulong) old_buffer, + (ulong) old_buffer->size)); + DBUG_ASSERT(old_buffer->buffer_no == log_descriptor.bc.buffer_no); } else @@ -7509,11 +7614,21 @@ if (left) { - /* - TODO: do not copy beginning of the page if we have no CRC or sector - checks on - */ - memcpy(new_buffer->buffer, data, current_page_fill); + if (log_descriptor.flags & + (TRANSLOG_PAGE_CRC | TRANSLOG_SECTOR_PROTECTION)) + memcpy(new_buffer->buffer, data, current_page_fill); + else + { + /* + This page header does not change if we add more data to the page so + we can not copy it and will not overwrite later + */ + new_buffer->skipped_data= current_page_fill; +#ifndef DBUG_OFF + memset(new_buffer->buffer, 0xa5, current_page_fill); +#endif + DBUG_ASSERT(new_buffer->skipped_data < TRANSLOG_PAGE_SIZE); + } } old_buffer->next_buffer_offset= new_buffer->offset; translog_buffer_lock(new_buffer); @@ -7561,6 +7676,7 @@ { log_descriptor.next_pass_max_lsn= lsn; log_descriptor.max_lsn_requester= pthread_self(); + pthread_cond_broadcast(&log_descriptor.new_goal_cond); } while (flush_no == log_descriptor.flush_no) { @@ -7572,66 +7688,78 @@ /** - @brief Flush the log up to given LSN (included) - - @param lsn log record serial number up to which (inclusive) - the log has to be flushed - - @return Operation status + @brief sync() range of files (inclusive) and directory (by request) + + @param min min internal file number to flush + @param max max internal file number to flush + @param sync_dir need sync directory + + return Operation status @retval 0 OK @retval 1 Error - -*/ - -my_bool translog_flush(TRANSLOG_ADDRESS lsn) -{ - LSN sent_to_disk= LSN_IMPOSSIBLE; - TRANSLOG_ADDRESS flush_horizon; - uint fn, i; +*/ + +static my_bool translog_sync_files(uint32 min, uint32 max, + my_bool sync_dir) +{ + uint fn; + my_bool rc= 0; + ulonglong flush_interval; + DBUG_ENTER("translog_sync_files"); + DBUG_PRINT("info", ("min: %lu max: %lu sync dir: %d", + (ulong) min, (ulong) max, (int) sync_dir)); + DBUG_ASSERT(min <= max); + + flush_interval= group_commit_wait; + if (flush_interval) + flush_start= my_micro_time(); + for (fn= min; fn <= max; fn++) + { + TRANSLOG_FILE *file= get_logfile_by_number(fn); + DBUG_ASSERT(file != NULL); + if (!file->is_sync) + { + if (my_sync(file->handler.file, MYF(MY_WME))) + { + rc= 1; + translog_stop_writing(); + DBUG_RETURN(rc); + } + translog_syncs++; + file->is_sync= 1; + } + } + + if (sync_dir) + { + if (!(rc= sync_dir(log_descriptor.directory_fd, + MYF(MY_WME | MY_IGNORE_BADFD)))) + translog_syncs++; + } + + DBUG_RETURN(rc); +} + + +/* + @brief Flushes buffers with LSNs in them less or equal address <lsn> + + @param lsn address up to which all LSNs should be flushed, + can be reset to real last LSN address + @parem sent_to_disk returns 'sent to disk' position + @param flush_horizon returns horizon of the flush + + @note About terminology see comment to translog_flush(). +*/ + +void translog_flush_buffers(TRANSLOG_ADDRESS *lsn, + TRANSLOG_ADDRESS *sent_to_disk, + TRANSLOG_ADDRESS *flush_horizon) +{ dirty_buffer_mask_t dirty_buffer_mask; + uint i; uint8 last_buffer_no, start_buffer_no; - my_bool rc= 0; - DBUG_ENTER("translog_flush"); - DBUG_PRINT("enter", ("Flush up to LSN: (%lu,0x%lx)", LSN_IN_PARTS(lsn))); - DBUG_ASSERT(translog_status == TRANSLOG_OK || - translog_status == TRANSLOG_READONLY); - LINT_INIT(sent_to_disk); - - pthread_mutex_lock(&log_descriptor.log_flush_lock); - DBUG_PRINT("info", ("Everything is flushed up to (%lu,0x%lx)", - LSN_IN_PARTS(log_descriptor.flushed))); - if (cmp_translog_addr(log_descriptor.flushed, lsn) >= 0) - { - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - DBUG_RETURN(0); - } - if (log_descriptor.flush_in_progress) - { - translog_flush_set_new_goal_and_wait(lsn); - if (!pthread_equal(log_descriptor.max_lsn_requester, pthread_self())) - { - /* fix lsn if it was horizon */ - if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->last_lsn) > 0) - lsn= BUFFER_MAX_LSN(log_descriptor.bc.buffer); - translog_flush_wait_for_end(lsn); - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - DBUG_RETURN(0); - } - log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; - } - log_descriptor.flush_in_progress= 1; - flush_horizon= log_descriptor.previous_flush_horizon; - DBUG_PRINT("info", ("flush_in_progress is set")); - pthread_mutex_unlock(&log_descriptor.log_flush_lock); - - translog_lock(); - if (log_descriptor.is_everything_flushed) - { - DBUG_PRINT("info", ("everything is flushed")); - rc= (translog_status == TRANSLOG_READONLY); - translog_unlock(); - goto out; - } + DBUG_ENTER("translog_flush_buffers"); /* We will recheck information when will lock buffers one by @@ -7656,15 +7784,15 @@ /* if LSN up to which we have to flush bigger then maximum LSN of previous buffer and at least one LSN was saved in the current buffer (last_lsn != - LSN_IMPOSSIBLE) then we better finish the current buffer. + LSN_IMPOSSIBLE) then we have to close the current buffer. */ - if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->prev_last_lsn) > 0 && + if (cmp_translog_addr(*lsn, log_descriptor.bc.buffer->prev_last_lsn) > 0 && log_descriptor.bc.buffer->last_lsn != LSN_IMPOSSIBLE) { struct st_translog_buffer *buffer= log_descriptor.bc.buffer; - lsn= log_descriptor.bc.buffer->last_lsn; /* fix lsn if it was horizon */ + *lsn= log_descriptor.bc.buffer->last_lsn; /* fix lsn if it was horizon */ DBUG_PRINT("info", ("LSN to flush fixed to last lsn: (%lu,0x%lx)", - LSN_IN_PARTS(log_descriptor.bc.buffer->last_lsn))); + LSN_IN_PARTS(log_descriptor.bc.buffer->last_lsn))); last_buffer_no= log_descriptor.bc.buffer_no; log_descriptor.is_everything_flushed= 1; translog_force_current_buffer_to_finish(); @@ -7676,8 +7804,10 @@ TRANSLOG_BUFFERS_NO); translog_unlock(); } - sent_to_disk= translog_get_sent_to_disk(); - if (cmp_translog_addr(lsn, sent_to_disk) > 0) + + /* flush buffers */ + *sent_to_disk= translog_get_sent_to_disk(); + if (cmp_translog_addr(*lsn, *sent_to_disk) > 0) { DBUG_PRINT("info", ("Start buffer #: %u last buffer #: %u", @@ -7697,53 +7827,237 @@ LSN_IN_PARTS(buffer->last_lsn), (buffer->file ? "dirty" : "closed"))); - if (buffer->prev_last_lsn <= lsn && + if (buffer->prev_last_lsn <= *lsn && buffer->file != NULL) { - DBUG_ASSERT(flush_horizon <= buffer->offset + buffer->size); - flush_horizon= buffer->offset + buffer->size; + DBUG_ASSERT(*flush_horizon <= buffer->offset + buffer->size); + *flush_horizon= (buffer->pre_force_close_horizon != LSN_IMPOSSIBLE ? + buffer->pre_force_close_horizon : + buffer->offset + buffer->size); + /* pre_force_close_horizon is reset during new buffer start */ + DBUG_PRINT("info", ("flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(*flush_horizon))); + DBUG_ASSERT(*flush_horizon <= log_descriptor.horizon); + translog_buffer_flush(buffer); } translog_buffer_unlock(buffer); i= (i + 1) % TRANSLOG_BUFFERS_NO; } while (i != last_buffer_no); - sent_to_disk= translog_get_sent_to_disk(); - } - - /* sync files from previous flush till current one */ - for (fn= LSN_FILE_NO(log_descriptor.flushed); fn <= LSN_FILE_NO(lsn); fn++) - { - TRANSLOG_FILE *file= get_logfile_by_number(fn); - DBUG_ASSERT(file != NULL); - if (!file->is_sync) - { - if (my_sync(file->handler.file, MYF(MY_WME))) + *sent_to_disk= translog_get_sent_to_disk(); + } + + DBUG_VOID_RETURN; +} + +/** + @brief Flush the log up to given LSN (included) + + @param lsn log record serial number up to which (inclusive) + the log has to be flushed + + @return Operation status + @retval 0 OK + @retval 1 Error + + @note + + - Non group commit logic: Commits made in passes. Thread which started + flush first is performing actual flush, other threads sets new goal (LSN) + of the next pass (if it is maximum) and waits for the pass end or just + wait for the pass end. + + - If hard group commit enabled and rate set to zero: + The first thread sends all changed buffers to disk. This is repeated + as long as there are new LSNs added. The process can not loop + forever because we have limited number of threads and they will wait + for the data to be synced. + Pseudo code: + + do + send changed buffers to disk + while new_goal + sync + + - If hard group commit switched ON and less than rate microseconds has + passed from last sync, then after buffers have been sent to disk + wait until rate microseconds has passed since last sync, do sync and return. + This ensures that if we call sync infrequently we don't do any waits. + + - If soft group commit enabled everything works as with 'non group commit' + but the thread doesn't do any real sync(). If rate is not zero the + sync() will be performed by a service thread with the given rate + when needed (new LSN appears). + + @note Terminology: + 'sent to disk' means written to disk but not sync()ed, + 'flushed' mean sent to disk and synced(). +*/ + +my_bool translog_flush(TRANSLOG_ADDRESS lsn) +{ + struct timespec abstime; + ulonglong flush_interval; + ulonglong time_spent; + LSN sent_to_disk= LSN_IMPOSSIBLE; + TRANSLOG_ADDRESS flush_horizon; + my_bool rc= 0; + my_bool hgroup_commit_at_start; + DBUG_ENTER("translog_flush"); + DBUG_PRINT("enter", ("Flush up to LSN: (%lu,0x%lx)", LSN_IN_PARTS(lsn))); + DBUG_ASSERT(translog_status == TRANSLOG_OK || + translog_status == TRANSLOG_READONLY); + LINT_INIT(sent_to_disk); + + pthread_mutex_lock(&log_descriptor.log_flush_lock); + DBUG_PRINT("info", ("Everything is flushed up to (%lu,0x%lx)", + LSN_IN_PARTS(log_descriptor.flushed))); + if (cmp_translog_addr(log_descriptor.flushed, lsn) >= 0) + + + { + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_RETURN(0); + } + if (log_descriptor.flush_in_progress) + { + translog_lock(); + /* fix lsn if it was horizon */ + if (cmp_translog_addr(lsn, log_descriptor.bc.buffer->last_lsn) > 0) + lsn= BUFFER_MAX_LSN(log_descriptor.bc.buffer); + translog_unlock(); + translog_flush_set_new_goal_and_wait(lsn); + if (!pthread_equal(log_descriptor.max_lsn_requester, pthread_self())) + { + /* + translog_flush_wait_for_end() release log_flush_lock while is + waiting then acquire it again + */ + translog_flush_wait_for_end(lsn); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_RETURN(0); + } + log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; + } + log_descriptor.flush_in_progress= 1; + flush_horizon= log_descriptor.previous_flush_horizon; + DBUG_PRINT("info", ("flush_in_progress is set, flush_horizon: (%lu,0x%lx)", + LSN_IN_PARTS(flush_horizon))); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + + hgroup_commit_at_start= hard_group_commit; + if (hgroup_commit_at_start) + flush_interval= group_commit_wait; + + translog_lock(); + if (log_descriptor.is_everything_flushed) + { + DBUG_PRINT("info", ("everything is flushed")); + translog_unlock(); + pthread_mutex_lock(&log_descriptor.log_flush_lock); + goto out; + } + + for (;;) + { + /* Following function flushes buffers and makes translog_unlock() */ + translog_flush_buffers(&lsn, &sent_to_disk, &flush_horizon); + + if (!hgroup_commit_at_start) + break; /* flush pass is ended */ + +retest: + if (flush_interval != 0 && + (my_micro_time() - flush_start) >= flush_interval) + break; /* flush pass is ended */ + + pthread_mutex_lock(&log_descriptor.log_flush_lock); + if (log_descriptor.next_pass_max_lsn != LSN_IMPOSSIBLE) + { + /* take next goal */ + lsn= log_descriptor.next_pass_max_lsn; + log_descriptor.next_pass_max_lsn= LSN_IMPOSSIBLE; + /* prevent other thread from continue */ + log_descriptor.max_lsn_requester= pthread_self(); + DBUG_PRINT("info", ("flush took next goal: (%lu,0x%lx)", + LSN_IN_PARTS(lsn))); + } + else + { + if (flush_interval == 0 || + (time_spent= (my_micro_time() - flush_start)) >= flush_interval) { - rc= 1; - translog_stop_writing(); - sent_to_disk= LSN_IMPOSSIBLE; - goto out; + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + break; } - file->is_sync= 1; - } - } - - if (sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && - (LSN_FILE_NO(log_descriptor.previous_flush_horizon) != - LSN_FILE_NO(flush_horizon) || - ((LSN_OFFSET(log_descriptor.previous_flush_horizon) - 1) / - TRANSLOG_PAGE_SIZE) != - ((LSN_OFFSET(flush_horizon) - 1) / TRANSLOG_PAGE_SIZE))) - rc|= sync_dir(log_descriptor.directory_fd, MYF(MY_WME | MY_IGNORE_BADFD)); + DBUG_PRINT("info", ("flush waits: %llu interval: %llu spent: %llu", + flush_interval - time_spent, + flush_interval, time_spent)); + /* wait time or next goal */ + set_timespec_nsec(abstime, flush_interval - time_spent); + pthread_cond_timedwait(&log_descriptor.new_goal_cond, + &log_descriptor.log_flush_lock, + &abstime); + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + DBUG_PRINT("info", ("retest conditions")); + goto retest; + } + pthread_mutex_unlock(&log_descriptor.log_flush_lock); + + /* next flush pass */ + DBUG_PRINT("info", ("next flush pass")); + translog_lock(); + } + + /* + sync() files from previous flush till current one + */ + if (!soft_sync || hgroup_commit_at_start) + { + if ((rc= + translog_sync_files(LSN_FILE_NO(log_descriptor.flushed), + LSN_FILE_NO(lsn), + sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS && + (LSN_FILE_NO(log_descriptor. + previous_flush_horizon) != + LSN_FILE_NO(flush_horizon) || + (LSN_OFFSET(log_descriptor. + previous_flush_horizon) / + TRANSLOG_PAGE_SIZE) != + (LSN_OFFSET(flush_horizon) / + TRANSLOG_PAGE_SIZE))))) + { + sent_to_disk= LSN_IMPOSSIBLE; + pthread_mutex_lock(&log_descriptor.log_flush_lock); + goto out; + } + /* keep values for soft sync() and forced sync() actual */ + { + uint32 fileno= LSN_FILE_NO(lsn); + my_atomic_rwlock_wrlock(&soft_sync_rwl); + my_atomic_store32(&soft_sync_min, fileno); + my_atomic_store32(&soft_sync_max, fileno); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + } + } + else + { + my_atomic_rwlock_wrlock(&soft_sync_rwl); + my_atomic_store32(&soft_sync_max, LSN_FILE_NO(lsn)); + my_atomic_rwlock_wrunlock(&soft_sync_rwl); + } + + DBUG_ASSERT(flush_horizon <= log_descriptor.horizon); + + pthread_mutex_lock(&log_descriptor.log_flush_lock); log_descriptor.previous_flush_horizon= flush_horizon; out: - pthread_mutex_lock(&log_descriptor.log_flush_lock); if (sent_to_disk != LSN_IMPOSSIBLE) log_descriptor.flushed= sent_to_disk; log_descriptor.flush_in_progress= 0; log_descriptor.flush_no++; DBUG_PRINT("info", ("flush_in_progress is dropped")); - pthread_mutex_unlock(&log_descriptor.log_flush_lock);\ + pthread_mutex_unlock(&log_descriptor.log_flush_lock); pthread_cond_broadcast(&log_descriptor.log_flush_cond); DBUG_RETURN(rc); } @@ -8113,6 +8427,8 @@ my_bool translog_purge(TRANSLOG_ADDRESS low) { uint32 last_need_file= LSN_FILE_NO(low); + uint32 min_unsync; + int soft; TRANSLOG_ADDRESS horizon= translog_get_horizon(); int rc= 0; DBUG_ENTER("translog_purge"); @@ -8120,12 +8436,23 @@ DBUG_ASSERT(translog_status == TRANSLOG_OK || translog_status == TRANSLOG_READONLY); + soft= soft_sync; + DBUG_PRINT("info", ("min_unsync: %lu", (ulong) min_unsync)); + if (soft && min_unsync < last_need_file) + { + last_need_file= min_unsync; + DBUG_PRINT("info", ("last_need_file set to %lu", (ulong)last_need_file)); + } + pthread_mutex_lock(&log_descriptor.purger_lock); + DBUG_PRINT("info", ("last_lsn_checked file: %lu:", + (ulong) log_descriptor.last_lsn_checked)); if (LSN_FILE_NO(log_descriptor.last_lsn_checked) < last_need_file) { uint32 i; uint32 min_file= translog_first_file(horizon, 1); DBUG_ASSERT(min_file != 0); /* log is already started */ + DBUG_PRINT("info", ("min_file: %lu:",(ulong) min_file)); for(i= min_file; i < last_need_file && rc == 0; i++) { LSN lsn= translog_get_file_max_lsn_stored(i); @@ -8356,6 +8683,155 @@ } + +/** + Sets soft sync mode + + @param mode TRUE if we need switch soft sync on else off +*/ + +void translog_soft_sync(my_bool mode) +{ + soft_sync= mode; +} + + +/** + Sets hard group commit + + @param mode TRUE if we need switch hard group commit on else off +*/ + +void translog_hard_group_commit(my_bool mode) +{ + hard_group_commit= mode; +} + + +/** + @brief forced log sync (used when we are switching modes) +*/ + +void translog_sync() +{ + uint32 max= get_current_logfile()->number; + uint32 min; + DBUG_ENTER("ma_translog_sync"); + + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + if (!min) + min= max; + + translog_sync_files(min, max, sync_log_dir >= TRANSLOG_SYNC_DIR_ALWAYS); + + DBUG_VOID_RETURN; +} + + +/** + @brief set rate for group commit + + @param interval interval to set. + + @note We use this function with additional variable because have to + restart service thread with new value which we can't make inside changing + variable routine (update_maria_group_commit_interval) +*/ + +void translog_set_group_commit_interval(uint32 interval) +{ + DBUG_ENTER("translog_set_group_commit_interval"); + group_commit_wait= interval; + DBUG_PRINT("info", ("wait: %llu", + (ulonglong)group_commit_wait)); + DBUG_VOID_RETURN; +} + + +/** + @brief syncing service thread +*/ + +static pthread_handler_t +ma_soft_sync_background( void *arg __attribute__((unused))) +{ + + my_thread_init(); + { + DBUG_ENTER("ma_soft_sync_background"); + for(;;) + { + ulonglong prev_loop= my_micro_time(); + ulonglong time, sleep; + uint32 min, max; + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + max= my_atomic_load32(&soft_sync_max); + my_atomic_store32(&soft_sync_min, max); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + + sleep= group_commit_wait; + translog_sync_files(min, max, FALSE); + time= my_micro_time() - prev_loop; + if (time > sleep) + sleep= 0; + else + sleep-= time; + if (my_service_thread_sleep(&soft_sync_control, sleep)) + break; + } + my_service_thread_signal_end(&soft_sync_control); + my_thread_end(); + DBUG_RETURN(0); + } +} + + +/** + @brief Starts syncing thread +*/ + +int translog_soft_sync_start(void) +{ + pthread_t th; + int res= 0; + uint32 min, max; + DBUG_ENTER("translog_soft_sync_start"); + + /* check and init variables */ + my_atomic_rwlock_rdlock(&soft_sync_rwl); + min= my_atomic_load32(&soft_sync_min); + max= my_atomic_load32(&soft_sync_max); + if (!max) + my_atomic_store32(&soft_sync_max, (max= get_current_logfile()->number)); + if (!min) + my_atomic_store32(&soft_sync_min, max); + my_atomic_rwlock_rdunlock(&soft_sync_rwl); + + if (!(res= ma_service_thread_control_init(&soft_sync_control))) + if (!(res= pthread_create(&th, NULL, ma_soft_sync_background, NULL))) + soft_sync_control.status= THREAD_RUNNING; + DBUG_RETURN(res); +} + + +/** + @brief Stops syncing thread +*/ + +void translog_soft_sync_end(void) +{ + DBUG_ENTER("translog_soft_sync_end"); + if (soft_sync_control.inited) + { + ma_service_thread_control_end(&soft_sync_control); + } + DBUG_VOID_RETURN; +} + + #ifdef MARIA_DUMP_LOG #include <my_getopt.h> extern void translog_example_table_init(); === modified file 'storage/maria/ma_loghandler.h' --- a/storage/maria/ma_loghandler.h 2009-01-15 22:25:53 +0000 +++ b/storage/maria/ma_loghandler.h 2010-02-09 08:32:59 +0000 @@ -342,6 +342,14 @@ TRANSLOG_SHUTDOWN /* going to shutdown the loghandler */ }; extern enum enum_translog_status translog_status; +extern ulonglong translog_syncs; /* Number of sync()s */ + +void translog_soft_sync(my_bool mode); +void translog_hard_group_commit(my_bool mode); +int translog_soft_sync_start(void); +void translog_soft_sync_end(void); +void translog_sync(); +void translog_set_group_commit_interval(uint32 interval); /* all the rest added because of recovery; should we make @@ -441,6 +449,14 @@ typedef enum { + TRANSLOG_GCOMMIT_NONE, + TRANSLOG_GCOMMIT_HARD, + TRANSLOG_GCOMMIT_SOFT +} enum_maria_group_commit; +extern ulong maria_group_commit; +extern ulong maria_group_commit_interval; +typedef enum +{ TRANSLOG_PURGE_IMMIDIATE, TRANSLOG_PURGE_EXTERNAL, TRANSLOG_PURGE_ONDEMAND

3 2

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2814)
by Michael Widenius 10 Feb '10

10 Feb '10

#At lp:maria based on revid:monty@askmonty.org-20100209171704-h7stfhbh94k54tbf 2814 Michael Widenius 2010-02-10 When one does a drop table, the indexes are not flushed to disk before drop anymore (with MyISAM/Maria) myisam-recover options changed from OFF to 'DEFAULT' to get less change of data loss when using MyISAM. (The disadvantage is that changed MyISAM tables will be checked at access time; Use --myisam-recover=OFF for old behavior) Don't call extra(HA_EXTRA_FORCE_REOPEN) in ALTER TABLE if table is locked as this will mark table as crashed! Added assert to detect if we accidently would use MyISAM versioning in MySQL modified: include/my_base.h mysql-test/mysql-test-run.pl mysql-test/r/sp-destruct.result mysql-test/r/variables.result mysql-test/r/view.result mysql-test/suite/maria/t/maria-recovery2-master.opt mysql-test/t/sp-destruct.test mysql-test/t/view.test sql/lock.cc sql/mysql_priv.h sql/mysqld.cc sql/sql_base.cc sql/sql_delete.cc sql/sql_table.cc sql/table.cc sql/table.h storage/maria/ha_maria.cc storage/maria/ma_blockrec.c storage/maria/ma_close.c storage/maria/ma_extra.c storage/maria/ma_locking.c storage/maria/ma_recovery.c storage/maria/maria_def.h storage/myisam/mi_close.c storage/myisam/mi_extra.c storage/myisam/mi_open.c storage/myisam/myisamdef.h per-file messages: include/my_base.h Mark NOT_USED as USED, as we now use this as a flag to not call extra() mysql-test/mysql-test-run.pl Don't write all options when there is something wrong with the arguments mysql-test/r/sp-destruct.result Add missing flush of mysql.proc (as the test copied live tables) mysql-test/r/variables.result myisam-recover options changed to 'default' mysql-test/r/view.result Don't show create time in result mysql-test/suite/maria/t/maria-recovery2-master.opt Don't run test with myisam-recover (as this produces extra warnings during simulated death) mysql-test/t/sp-destruct.test Add missing flush of mysql.proc (as the test copied live tables) mysql-test/t/view.test Don't show create time in result sql/lock.cc Added marker if table was deleted to argument list sql/mysql_priv.h Added marker if table was deleted to argument list sql/mysqld.cc myisam-recover options changed from OFF to 'DEFAULT' to get less change of data loss when using MyISAM Allow one to specify OFF as argument to myisam-recover (was default before but one couldn't specify it) sql/sql_base.cc Mark if table is going to be deleted sql/sql_delete.cc Mark if table is going to be deleted sql/sql_table.cc Mark if table is going to be deleted Don't call extra(HA_EXTRA_FORCE_REOPEN) in ALTER TABLE if table is locked as this will mark table as crashed! sql/table.cc Signal to handler if table is getting deleted as part of getting droped from table cache. sql/table.h Added marker if table is going to be deleted. storage/maria/ha_maria.cc Don't search for transaction handler if file is not transactional or outside of transaction (Fixed possible core dump) storage/maria/ma_blockrec.c Don't write changed information if table is going to be deleted. storage/maria/ma_close.c Don't write changed information if table is going to be deleted. storage/maria/ma_extra.c Mark tables that are deleted as crased, to ensure good behavior on restart if we suddenly crash. storage/maria/ma_locking.c Cleanup storage/maria/ma_recovery.c We need trnman to be inited during redo phase (to be able to open tables checked with maria_chk) storage/maria/maria_def.h Added marker if table is going to be deleted. storage/myisam/mi_close.c Don't write changed information if table is going to be deleted. storage/myisam/mi_extra.c Mark tables that are deleted as crased, to ensure good behavior on restart if we suddenly crash. storage/myisam/mi_open.c Added assert to detect if we accidently would use MyISAM versioning in MySQL storage/myisam/myisamdef.h Added marker if table is going to be deleted. === modified file 'include/my_base.h' --- a/include/my_base.h 2009-09-07 20:50:10 +0000 +++ b/include/my_base.h 2010-02-10 19:06:24 +0000 @@ -111,7 +111,7 @@ enum ha_storage_media { enum ha_extra_function { HA_EXTRA_NORMAL=0, /* Optimize for space (def) */ HA_EXTRA_QUICK=1, /* Optimize for speed */ - HA_EXTRA_NOT_USED=2, + HA_EXTRA_NOT_USED=2, /* Should be ignored by handler */ HA_EXTRA_CACHE=3, /* Cache record in HA_rrnd() */ HA_EXTRA_NO_CACHE=4, /* End caching of records (def) */ HA_EXTRA_NO_READCHECK=5, /* No readcheck on update */ === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2010-01-29 10:42:31 +0000 +++ b/mysql-test/mysql-test-run.pl 2010-02-10 19:06:24 +0000 @@ -5542,6 +5542,8 @@ sub usage ($) { if ( $message ) { print STDERR "$message\n"; + print STDERR "For full list of options, use $0 --help\n"; + exit; } print <<HERE; === modified file 'mysql-test/r/sp-destruct.result' --- a/mysql-test/r/sp-destruct.result 2009-11-21 11:18:21 +0000 +++ b/mysql-test/r/sp-destruct.result 2010-02-10 19:06:24 +0000 @@ -1,4 +1,5 @@ call mtr.add_suppression("Column count of mysql.proc is wrong. Expected 20, found 19. The table is probably corrupted"); +flush table mysql.proc; use test; drop procedure if exists bug14233; drop function if exists bug14233; === modified file 'mysql-test/r/variables.result' --- a/mysql-test/r/variables.result 2010-01-11 13:15:28 +0000 +++ b/mysql-test/r/variables.result 2010-02-10 19:06:24 +0000 @@ -1261,12 +1261,12 @@ ERROR HY000: Variable 'lower_case_table_ # SHOW VARIABLES like 'myisam_recover_options'; Variable_name Value -myisam_recover_options OFF +myisam_recover_options DEFAULT SELECT @@session.myisam_recover_options; ERROR HY000: Variable 'myisam_recover_options' is a GLOBAL variable SELECT @@global.myisam_recover_options; @@global.myisam_recover_options -OFF +DEFAULT SET @@session.myisam_recover_options= 'x'; ERROR HY000: Variable 'myisam_recover_options' is a read only variable SET @@global.myisam_recover_options= 'x'; === modified file 'mysql-test/r/view.result' --- a/mysql-test/r/view.result 2009-10-15 21:38:29 +0000 +++ b/mysql-test/r/view.result 2010-02-10 19:06:24 +0000 @@ -155,13 +155,13 @@ v5 VIEW v6 VIEW show table status; Name Engine Version Row_format Rows Avg_row_length Data_length Max_data_length Index_length Data_free Auto_increment Create_time Update_time Check_time Collation Checksum Create_options Comment -t1 MyISAM 10 Fixed 5 9 45 # 1024 0 NULL # # NULL latin1_swedish_ci NULL -v1 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # NULL NULL NULL NULL VIEW -v2 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # NULL NULL NULL NULL VIEW -v3 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # NULL NULL NULL NULL VIEW -v4 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # NULL NULL NULL NULL VIEW -v5 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # NULL NULL NULL NULL VIEW -v6 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # NULL NULL NULL NULL VIEW +t1 MyISAM 10 Fixed 5 9 45 # 1024 0 NULL # # # latin1_swedish_ci NULL +v1 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # # NULL NULL NULL VIEW +v2 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # # NULL NULL NULL VIEW +v3 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # # NULL NULL NULL VIEW +v4 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # # NULL NULL NULL VIEW +v5 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # # NULL NULL NULL VIEW +v6 NULL NULL NULL NULL NULL NULL # NULL NULL NULL # # # NULL NULL NULL VIEW drop view v1,v2,v3,v4,v5,v6; create view v1 (c,d,e,f) as select a,b, a in (select a+2 from t1), a = all (select a from t1) from t1; === modified file 'mysql-test/suite/maria/t/maria-recovery2-master.opt' --- a/mysql-test/suite/maria/t/maria-recovery2-master.opt 2009-01-15 14:29:14 +0000 +++ b/mysql-test/suite/maria/t/maria-recovery2-master.opt 2010-02-10 19:06:24 +0000 @@ -1 +1 @@ ---skip-stack-trace --skip-core-file --loose-maria-log-dir-path=$MYSQLTEST_VARDIR/tmp +--skip-stack-trace --skip-core-file --loose-maria-log-dir-path=$MYSQLTEST_VARDIR/tmp --myisam-recover= === modified file 'mysql-test/t/sp-destruct.test' --- a/mysql-test/t/sp-destruct.test 2009-11-21 11:18:21 +0000 +++ b/mysql-test/t/sp-destruct.test 2010-02-10 19:06:24 +0000 @@ -17,6 +17,7 @@ call mtr.add_suppression("Column count o # Backup proc table let $MYSQLD_DATADIR= `select @@datadir`; +flush table mysql.proc; --copy_file $MYSQLD_DATADIR/mysql/proc.frm $MYSQLTEST_VARDIR/tmp/proc.frm --copy_file $MYSQLD_DATADIR/mysql/proc.MYD $MYSQLTEST_VARDIR/tmp/proc.MYD --copy_file $MYSQLD_DATADIR/mysql/proc.MYI $MYSQLTEST_VARDIR/tmp/proc.MYI === modified file 'mysql-test/t/view.test' --- a/mysql-test/t/view.test 2009-10-15 21:38:29 +0000 +++ b/mysql-test/t/view.test 2010-02-10 19:06:24 +0000 @@ -87,7 +87,7 @@ explain extended select c from v6; # show table/table status test show tables; show full tables; ---replace_column 8 # 12 # 13 # +--replace_column 8 # 12 # 13 # 14 # show table status; drop view v1,v2,v3,v4,v5,v6; === modified file 'sql/lock.cc' --- a/sql/lock.cc 2009-10-15 21:38:29 +0000 +++ b/sql/lock.cc 2010-02-10 19:06:24 +0000 @@ -1049,10 +1049,14 @@ int lock_table_name(THD *thd, TABLE_LIST DBUG_RETURN(-1); table_list->table=table; + table->s->deleting= table_list->deleting; /* Return 1 if table is in use */ DBUG_RETURN(test(remove_table_from_cache(thd, db, table_list->table_name, - check_in_use ? RTFC_NO_FLAG : RTFC_WAIT_OTHER_THREAD_FLAG))); + (check_in_use ? + RTFC_NO_FLAG : + RTFC_WAIT_OTHER_THREAD_FLAG), + table_list->deleting))); } === modified file 'sql/mysql_priv.h' --- a/sql/mysql_priv.h 2009-12-03 11:19:05 +0000 +++ b/sql/mysql_priv.h 2010-02-10 19:06:24 +0000 @@ -1636,7 +1636,7 @@ uint prep_alter_part_table(THD *thd, TAB #define RTFC_WAIT_OTHER_THREAD_FLAG 0x0002 #define RTFC_CHECK_KILLED_FLAG 0x0004 bool remove_table_from_cache(THD *thd, const char *db, const char *table, - uint flags); + uint flags, my_bool deleting); #define NORMAL_PART_NAME 0 #define TEMP_PART_NAME 1 === modified file 'sql/mysqld.cc' --- a/sql/mysqld.cc 2010-01-29 18:42:22 +0000 +++ b/sql/mysqld.cc 2010-02-10 19:06:24 +0000 @@ -7962,7 +7962,13 @@ static int mysql_init_variables(void) refresh_version= 1L; /* Increments on each reload */ global_query_id= thread_id= 1L; strmov(server_version, MYSQL_SERVER_VERSION); - myisam_recover_options_str= sql_mode_str= "OFF"; + sql_mode_str= ""; + + /* By default, auto-repair MyISAM tables after crash */ + myisam_recover_options_str= "DEFAULT"; + myisam_recover_options= HA_RECOVER_DEFAULT; + ha_open_options|= HA_OPEN_ABORT_IF_CRASHED; + myisam_stats_method_str= "nulls_unequal"; my_bind_addr = htonl(INADDR_ANY); threads.empty(); @@ -8616,26 +8622,31 @@ mysqld_get_one_option(int optid, #endif case OPT_MYISAM_RECOVER: { - if (!argument) - { - myisam_recover_options= HA_RECOVER_DEFAULT; - myisam_recover_options_str= myisam_recover_typelib.type_names[0]; - } - else if (!argument[0]) + if (argument && (!argument[0] || + my_strcasecmp(system_charset_info, argument, "OFF") == 0)) { myisam_recover_options= HA_RECOVER_NONE; myisam_recover_options_str= "OFF"; + ha_open_options&= ~HA_OPEN_ABORT_IF_CRASHED; } else { - myisam_recover_options_str=argument; - myisam_recover_options= - find_bit_type_or_exit(argument, &myisam_recover_typelib, opt->name, - &error); - if (error) - return 1; + if (!argument) + { + myisam_recover_options= HA_RECOVER_DEFAULT; + myisam_recover_options_str= myisam_recover_typelib.type_names[0]; + } + else + { + myisam_recover_options_str=argument; + myisam_recover_options= + find_bit_type_or_exit(argument, &myisam_recover_typelib, opt->name, + &error); + if (error) + return 1; + } + ha_open_options|=HA_OPEN_ABORT_IF_CRASHED; } - ha_open_options|=HA_OPEN_ABORT_IF_CRASHED; break; } case OPT_CONCURRENT_INSERT: === modified file 'sql/sql_base.cc' --- a/sql/sql_base.cc 2010-01-15 15:27:55 +0000 +++ b/sql/sql_base.cc 2010-02-10 19:06:24 +0000 @@ -930,7 +930,7 @@ bool close_cached_tables(THD *thd, TABLE for (TABLE_LIST *table= tables; table; table= table->next_local) { if (remove_table_from_cache(thd, table->db, table->table_name, - RTFC_OWNED_BY_THD_FLAG)) + RTFC_OWNED_BY_THD_FLAG, table->deleting)) found=1; } if (!found) @@ -8404,6 +8404,11 @@ void remove_db_from_cache(const char *db if (!strcmp(table->s->db.str, db)) { table->s->version= 0L; /* Free when thread is ready */ + /* + This functions only called from DROP DATABASE code, so we are going + to drop all tables so we mark them as deleting + */ + table->s->deleting= TRUE; if (!table->in_use) relink_unused(table); } @@ -8446,7 +8451,7 @@ void flush_tables() */ bool remove_table_from_cache(THD *thd, const char *db, const char *table_name, - uint flags) + uint flags, my_bool deleting) { char key[MAX_DBKEY_LENGTH]; uint key_length; @@ -8540,7 +8545,10 @@ bool remove_table_from_cache(THD *thd, c } } while (unused_tables && !unused_tables->s->version) + { + unused_tables->s->deleting= deleting; VOID(hash_delete(&open_cache,(uchar*) unused_tables)); + } DBUG_PRINT("info", ("Removing table from table_def_cache")); /* Remove table from table definition cache if it's not in use */ @@ -8734,7 +8742,8 @@ int abort_and_upgrade_lock(ALTER_PARTITI /* If MERGE child, forward lock handling to parent. */ mysql_lock_abort(lpt->thd, lpt->table->parent ? lpt->table->parent : lpt->table, TRUE); - VOID(remove_table_from_cache(lpt->thd, lpt->db, lpt->table_name, flags)); + VOID(remove_table_from_cache(lpt->thd, lpt->db, lpt->table_name, flags, + FALSE)); VOID(pthread_mutex_unlock(&LOCK_open)); DBUG_RETURN(0); } @@ -8759,7 +8768,7 @@ void close_open_tables_and_downgrade(ALT { VOID(pthread_mutex_lock(&LOCK_open)); remove_table_from_cache(lpt->thd, lpt->db, lpt->table_name, - RTFC_WAIT_OTHER_THREAD_FLAG); + RTFC_WAIT_OTHER_THREAD_FLAG, FALSE); VOID(pthread_mutex_unlock(&LOCK_open)); /* If MERGE child, forward lock handling to parent. */ mysql_lock_downgrade_write(lpt->thd, lpt->table->parent ? lpt->table->parent : === modified file 'sql/sql_delete.cc' --- a/sql/sql_delete.cc 2010-01-15 15:27:55 +0000 +++ b/sql/sql_delete.cc 2010-02-10 19:06:24 +0000 @@ -1088,6 +1088,7 @@ bool mysql_truncate(THD *thd, TABLE_LIST HA_CREATE_INFO create_info; char path[FN_REFLEN + 1]; TABLE *table; + TABLE_LIST *tbl; bool error; uint path_length; bool is_temporary_table= false; @@ -1108,6 +1109,9 @@ bool mysql_truncate(THD *thd, TABLE_LIST if (!ha_check_storage_engine_flag(table_type, HTON_CAN_RECREATE)) goto trunc_by_del; + for (tbl= table_list; tbl; tbl= tbl->next_local) + tbl->deleting= TRUE; /* to trigger HA_PREPARE_FOR_DROP */ + table->file->info(HA_STATUS_AUTO | HA_STATUS_NO_LOCK); create_info.options|= HA_LEX_CREATE_TMP_TABLE; === modified file 'sql/sql_table.cc' --- a/sql/sql_table.cc 2010-01-15 15:27:55 +0000 +++ b/sql/sql_table.cc 2010-02-10 19:06:24 +0000 @@ -1880,6 +1880,7 @@ int mysql_rm_table_part2(THD *thd, TABLE { TABLE_SHARE *share; table->db_type= NULL; + if ((share= get_cached_table_share(table->db, table->table_name))) table->db_type= share->db_type(); @@ -1974,9 +1975,10 @@ int mysql_rm_table_part2(THD *thd, TABLE { TABLE *locked_table; abort_locked_tables(thd, db, table->table_name); + table->deleting= TRUE; remove_table_from_cache(thd, db, table->table_name, RTFC_WAIT_OTHER_THREAD_FLAG | - RTFC_CHECK_KILLED_FLAG); + RTFC_CHECK_KILLED_FLAG, FALSE); /* If the table was used in lock tables, remember it so that unlock_table_names can free it @@ -4213,9 +4215,10 @@ void wait_while_table_is_used(THD *thd,T /* Wait until all there are no other threads that has this table open */ remove_table_from_cache(thd, table->s->db.str, table->s->table_name.str, - RTFC_WAIT_OTHER_THREAD_FLAG); + RTFC_WAIT_OTHER_THREAD_FLAG, FALSE); /* extra() call must come only after all instances above are closed */ - VOID(table->file->extra(function)); + if (function != HA_EXTRA_NOT_USED) + VOID(table->file->extra(function)); DBUG_VOID_RETURN; } @@ -4717,7 +4720,7 @@ static bool mysql_admin_table(THD* thd, remove_table_from_cache(thd, table->table->s->db.str, table->table->s->table_name.str, RTFC_WAIT_OTHER_THREAD_FLAG | - RTFC_CHECK_KILLED_FLAG); + RTFC_CHECK_KILLED_FLAG, FALSE); thd->exit_cond(old_message); DBUG_EXECUTE_IF("wait_in_mysql_admin_table", wait_for_kill_signal(thd);); if (thd->killed) @@ -4975,7 +4978,8 @@ send_result_message: { pthread_mutex_lock(&LOCK_open); remove_table_from_cache(thd, table->table->s->db.str, - table->table->s->table_name.str, RTFC_NO_FLAG); + table->table->s->table_name.str, + RTFC_NO_FLAG, FALSE); pthread_mutex_unlock(&LOCK_open); } /* May be something modified consequently we have to invalidate cache */ @@ -6738,7 +6742,9 @@ view_err: from concurrent DDL statements. */ VOID(pthread_mutex_lock(&LOCK_open)); - wait_while_table_is_used(thd, table, HA_EXTRA_FORCE_REOPEN); + wait_while_table_is_used(thd, table, + thd->locked_tables ? HA_EXTRA_NOT_USED : + HA_EXTRA_FORCE_REOPEN); VOID(pthread_mutex_unlock(&LOCK_open)); DBUG_EXECUTE_IF("sleep_alter_enable_indexes", my_sleep(6000000);); error= table->file->ha_enable_indexes(HA_KEY_SWITCH_NONUNIQ_SAVE); @@ -6746,7 +6752,9 @@ view_err: break; case DISABLE: VOID(pthread_mutex_lock(&LOCK_open)); - wait_while_table_is_used(thd, table, HA_EXTRA_FORCE_REOPEN); + wait_while_table_is_used(thd, table, + thd->locked_tables ? HA_EXTRA_NOT_USED : + HA_EXTRA_FORCE_REOPEN); VOID(pthread_mutex_unlock(&LOCK_open)); error=table->file->ha_disable_indexes(HA_KEY_SWITCH_NONUNIQ_SAVE); /* COND_refresh will be signaled in close_thread_tables() */ @@ -7192,7 +7200,9 @@ view_err: else { VOID(pthread_mutex_lock(&LOCK_open)); - wait_while_table_is_used(thd, table, HA_EXTRA_FORCE_REOPEN); + wait_while_table_is_used(thd, table, + thd->locked_tables ? HA_EXTRA_NOT_USED : + HA_EXTRA_FORCE_REOPEN); VOID(pthread_mutex_unlock(&LOCK_open)); thd_proc_info(thd, "manage keys"); alter_table_manage_keys(table, table->file->indexes_are_disabled(), === modified file 'sql/table.cc' --- a/sql/table.cc 2010-01-15 15:27:55 +0000 +++ b/sql/table.cc 2010-02-10 19:06:24 +0000 @@ -1977,7 +1977,11 @@ int closefrm(register TABLE *table, bool DBUG_PRINT("enter", ("table: 0x%lx", (long) table)); if (table->db_stat) + { + if (table->s->deleting) + table->file->extra(HA_EXTRA_PREPARE_FOR_DROP); error=table->file->close(); + } my_free((char*) table->alias, MYF(MY_ALLOW_ZERO_PTR)); table->alias= 0; if (table->field) === modified file 'sql/table.h' --- a/sql/table.h 2010-01-15 15:27:55 +0000 +++ b/sql/table.h 2010-02-10 19:06:24 +0000 @@ -431,6 +431,7 @@ typedef struct st_table_share bool is_view; bool name_lock, replace_with_name_lock; bool waiting_on_cond; /* Protection against free */ + bool deleting; /* going to delete this table */ ulong table_map_id; /* for row-based replication */ ulonglong table_map_version; @@ -1379,7 +1380,7 @@ struct TABLE_LIST */ bool create; bool internal_tmp_table; - + bool deleting; /* going to delete this table */ /* View creation context. */ === modified file 'storage/maria/ha_maria.cc' --- a/storage/maria/ha_maria.cc 2009-12-03 11:34:11 +0000 +++ b/storage/maria/ha_maria.cc 2010-02-10 19:06:24 +0000 @@ -2255,9 +2255,12 @@ int ha_maria::extra(enum ha_extra_functi extern_lock(F_UNLOCK) (which resets file->trn) followed by maria_close() without calling commit/rollback in between. If file->trn is not set we can't remove file->share from the transaction list in the extra() call. + + table->in_use is not set in the case this is a done as part of closefrm() + as part of drop table. */ - if (!file->trn && + if (file->s->now_transactional && !file->trn && table->in_use && (operation == HA_EXTRA_PREPARE_FOR_DROP || operation == HA_EXTRA_PREPARE_FOR_RENAME)) { === modified file 'storage/maria/ma_blockrec.c' --- a/storage/maria/ma_blockrec.c 2010-01-28 11:35:10 +0000 +++ b/storage/maria/ma_blockrec.c 2010-02-10 19:06:24 +0000 @@ -430,8 +430,9 @@ my_bool _ma_once_end_block_record(MARIA_ if (share->bitmap.file.file >= 0) { if (flush_pagecache_blocks(share->pagecache, &share->bitmap.file, - share->temporary ? FLUSH_IGNORE_CHANGED : - FLUSH_RELEASE)) + ((share->temporary || share->deleting) ? + FLUSH_IGNORE_CHANGED : + FLUSH_RELEASE))) res= 1; /* File must be synced as it is going out of the maria_open_list and so === modified file 'storage/maria/ma_close.c' --- a/storage/maria/ma_close.c 2010-01-29 18:42:22 +0000 +++ b/storage/maria/ma_close.c 2010-02-10 19:06:24 +0000 @@ -79,7 +79,7 @@ int maria_close(register MARIA_HA *info) if ((*share->once_end)(share)) error= my_errno; if (flush_pagecache_blocks(share->pagecache, &share->kfile, - (share->temporary ? + ((share->temporary || share->deleting) ? FLUSH_IGNORE_CHANGED : FLUSH_RELEASE))) error= my_errno; === modified file 'storage/maria/ma_extra.c' --- a/storage/maria/ma_extra.c 2009-10-06 06:13:56 +0000 +++ b/storage/maria/ma_extra.c 2010-02-10 19:06:24 +0000 @@ -305,6 +305,12 @@ int maria_extra(MARIA_HA *info, enum ha_ pthread_mutex_unlock(&THR_LOCK_maria); break; case HA_EXTRA_PREPARE_FOR_DROP: + /* Signals about intent to delete this table */ + share->deleting= TRUE; + share->global_changed= FALSE; /* force writing changed flag */ + /* To force repair if reopened */ + _ma_mark_file_changed(info); + /* Fall trough */ case HA_EXTRA_PREPARE_FOR_RENAME: { my_bool do_flush= test(function != HA_EXTRA_PREPARE_FOR_DROP); === modified file 'storage/maria/ma_locking.c' --- a/storage/maria/ma_locking.c 2009-10-06 06:13:56 +0000 +++ b/storage/maria/ma_locking.c 2010-02-10 19:06:24 +0000 @@ -387,6 +387,9 @@ int _ma_test_if_changed(register MARIA_H open_count is not maintained on disk for temporary tables. */ +#define _MA_ALREADY_MARKED_FILE_CHANGED \ + ((share->state.changed & STATE_CHANGED) && share->global_changed) + int _ma_mark_file_changed(MARIA_HA *info) { uchar buff[3]; @@ -394,8 +397,6 @@ int _ma_mark_file_changed(MARIA_HA *info int error= 1; DBUG_ENTER("_ma_mark_file_changed"); -#define _MA_ALREADY_MARKED_FILE_CHANGED \ - ((share->state.changed & STATE_CHANGED) && share->global_changed) if (_MA_ALREADY_MARKED_FILE_CHANGED) DBUG_RETURN(0); pthread_mutex_lock(&share->intern_lock); /* recheck under mutex */ === modified file 'storage/maria/ma_recovery.c' --- a/storage/maria/ma_recovery.c 2009-10-26 11:35:42 +0000 +++ b/storage/maria/ma_recovery.c 2010-02-10 19:06:24 +0000 @@ -312,11 +312,14 @@ int maria_apply_log(LSN from_lsn, enum m now= my_getsystime(); in_redo_phase= TRUE; + trnman_init(max_trid_in_control_file); if (run_redo_phase(from_lsn, apply)) { ma_message_no_user(0, "Redo phase failed"); + trnman_destroy(); goto err; } + trnman_destroy(); if ((uncommitted_trans= end_of_redo_phase(should_run_undo_phase)) == (uint)-1) === modified file 'storage/maria/maria_def.h' --- a/storage/maria/maria_def.h 2009-11-29 23:08:56 +0000 +++ b/storage/maria/maria_def.h 2010-02-10 19:06:24 +0000 @@ -390,6 +390,7 @@ typedef struct st_maria_share my_bool now_transactional; my_bool have_versioning; my_bool key_del_used; /* != 0 if key_del is locked */ + my_bool deleting; /* we are going to delete this table */ #ifdef THREAD THR_LOCK lock; void (*lock_restore_status)(void *); === modified file 'storage/myisam/mi_close.c' --- a/storage/myisam/mi_close.c 2009-09-07 20:50:10 +0000 +++ b/storage/myisam/mi_close.c 2010-02-10 19:06:24 +0000 @@ -64,8 +64,9 @@ int mi_close(register MI_INFO *info) if (share->kfile >= 0) abort();); if (share->kfile >= 0 && flush_key_blocks(share->key_cache, share->kfile, - share->temporary ? FLUSH_IGNORE_CHANGED : - FLUSH_RELEASE)) + ((share->temporary || share->deleting) ? + FLUSH_IGNORE_CHANGED : + FLUSH_RELEASE))) error=my_errno; if (share->kfile >= 0) { === modified file 'storage/myisam/mi_extra.c' --- a/storage/myisam/mi_extra.c 2009-10-06 06:13:56 +0000 +++ b/storage/myisam/mi_extra.c 2010-02-10 19:06:24 +0000 @@ -256,8 +256,13 @@ int mi_extra(MI_INFO *info, enum ha_extr share->last_version= 0L; /* Impossible version */ pthread_mutex_unlock(&THR_LOCK_myisam); break; - case HA_EXTRA_PREPARE_FOR_RENAME: case HA_EXTRA_PREPARE_FOR_DROP: + /* Signals about intent to delete this table */ + share->deleting= TRUE; + share->global_changed= FALSE; /* force writing changed flag */ + _mi_mark_file_changed(info); + /* Fall trough */ + case HA_EXTRA_PREPARE_FOR_RENAME: pthread_mutex_lock(&THR_LOCK_myisam); share->last_version= 0L; /* Impossible version */ pthread_mutex_lock(&share->intern_lock); === modified file 'storage/myisam/mi_open.c' --- a/storage/myisam/mi_open.c 2009-12-03 11:19:05 +0000 +++ b/storage/myisam/mi_open.c 2010-02-10 19:06:24 +0000 @@ -58,6 +58,8 @@ MI_INFO *test_if_reopen(char *filename) { MI_INFO *info=(MI_INFO*) pos->data; MYISAM_SHARE *share=info->s; + DBUG_ASSERT(strcmp(share->unique_file_name,filename) || + share->last_version); if (!strcmp(share->unique_file_name,filename) && share->last_version) return info; } === modified file 'storage/myisam/myisamdef.h' --- a/storage/myisam/myisamdef.h 2009-12-03 11:34:11 +0000 +++ b/storage/myisam/myisamdef.h 2010-02-10 19:06:24 +0000 @@ -221,6 +221,7 @@ typedef struct st_mi_isam_share my_bool changed, /* If changed since lock */ global_changed, /* If changed since open */ not_flushed, temporary, delay_key_write, concurrent_insert; + my_bool deleting; /* we are going to delete this table */ #ifdef THREAD THR_LOCK lock; pthread_mutex_t intern_lock; /* Locking for use with _locking */

1 0

[Maria-developers] Rev 2757: Subquery optimizations backport: Update test results (checked) in file:///home/psergey/dev/maria-5.3-subqueries-r3/
by Sergey Petrunya 09 Feb '10

09 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r3/ ------------------------------------------------------------ revno: 2757 revision-id: psergey(a)askmonty.org-20100209203217-al1k9h50zrlphy5d parent: psergey(a)askmonty.org-20100208133030-e4zjy15b7o14ud8c committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r3 timestamp: Tue 2010-02-09 23:32:17 +0300 message: Subquery optimizations backport: Update test results (checked) === modified file 'mysql-test/r/join_cache.result' --- a/mysql-test/r/join_cache.result 2009-12-21 02:26:15 +0000 +++ b/mysql-test/r/join_cache.result 2010-02-09 20:32:17 +0000 @@ -1028,8 +1028,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -1343,8 +1343,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -1658,8 +1658,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -1973,8 +1973,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -2292,8 +2292,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -2514,8 +2514,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -2736,8 +2736,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; @@ -2958,8 +2958,8 @@ WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY City ALL Population NULL NULL NULL 4079 Using where -2 DEPENDENT SUBQUERY Country unique_subquery PRIMARY,Name PRIMARY 3 func 1 Using where +1 PRIMARY Country range PRIMARY,Name Name 52 NULL 10 Using index condition; Using MRR +1 PRIMARY City ref Population,Country Country 3 world.Country.Code 18 Using where; Using join buffer SELECT Name FROM City WHERE City.Country IN (SELECT Code FROM Country WHERE Country.Name LIKE 'L%') AND City.Population > 100000; === modified file 'mysql-test/r/type_datetime.result' --- a/mysql-test/r/type_datetime.result 2009-02-13 18:07:03 +0000 +++ b/mysql-test/r/type_datetime.result 2010-02-09 20:32:17 +0000 @@ -514,10 +514,9 @@ where id in (select id from t1 as x1 where (t1.cur_date is null)); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables -2 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE Warnings: Note 1276 Field or reference 'test.t1.cur_date' of SELECT #2 was resolved in SELECT #1 -Note 1003 select '1' AS `id`,'2007-04-25 18:30:22' AS `cur_date` from `test`.`t1` where <in_optimizer>('1',<exists>(select 1 AS `Not_used` from `test`.`t1` `x1` where 0)) +Note 1003 select '1' AS `id`,'2007-04-25 18:30:22' AS `cur_date` from `test`.`t1` `x1` join `test`.`t1` where (('2007-04-25 18:30:22' = 0)) select * from t1 where id in (select id from t1 as x1 where (t1.cur_date is null)); id cur_date @@ -526,10 +525,9 @@ where id in (select id from t2 as x1 where (t2.cur_date is null)); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables -2 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL NULL Impossible WHERE Warnings: Note 1276 Field or reference 'test.t2.cur_date' of SELECT #2 was resolved in SELECT #1 -Note 1003 select '1' AS `id`,'2007-04-25' AS `cur_date` from `test`.`t2` where <in_optimizer>('1',<exists>(select 1 AS `Not_used` from `test`.`t2` `x1` where 0)) +Note 1003 select '1' AS `id`,'2007-04-25' AS `cur_date` from `test`.`t2` `x1` join `test`.`t2` where (('2007-04-25' = 0)) select * from t2 where id in (select id from t2 as x1 where (t2.cur_date is null)); id cur_date @@ -540,10 +538,10 @@ where id in (select id from t1 as x1 where (t1.cur_date is null)); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t1 ALL NULL NULL NULL NULL 2 100.00 Using where -2 DEPENDENT SUBQUERY x1 ALL NULL NULL NULL NULL 2 100.00 Using where +1 PRIMARY x1 ALL NULL NULL NULL NULL 2 100.00 Using where; FirstMatch(t1) Warnings: Note 1276 Field or reference 'test.t1.cur_date' of SELECT #2 was resolved in SELECT #1 -Note 1003 select `test`.`t1`.`id` AS `id`,`test`.`t1`.`cur_date` AS `cur_date` from `test`.`t1` where <in_optimizer>(`test`.`t1`.`id`,<exists>(select 1 AS `Not_used` from `test`.`t1` `x1` where ((`test`.`t1`.`cur_date` = 0) and (<cache>(`test`.`t1`.`id`) = `test`.`x1`.`id`)))) +Note 1003 select `test`.`t1`.`id` AS `id`,`test`.`t1`.`cur_date` AS `cur_date` from `test`.`t1` semi join (`test`.`t1` `x1`) where ((`test`.`x1`.`id` = `test`.`t1`.`id`) and (`test`.`t1`.`cur_date` = 0)) select * from t1 where id in (select id from t1 as x1 where (t1.cur_date is null)); id cur_date @@ -552,10 +550,10 @@ where id in (select id from t2 as x1 where (t2.cur_date is null)); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t2 ALL NULL NULL NULL NULL 2 100.00 Using where -2 DEPENDENT SUBQUERY x1 ALL NULL NULL NULL NULL 2 100.00 Using where +1 PRIMARY x1 ALL NULL NULL NULL NULL 2 100.00 Using where; FirstMatch(t2) Warnings: Note 1276 Field or reference 'test.t2.cur_date' of SELECT #2 was resolved in SELECT #1 -Note 1003 select `test`.`t2`.`id` AS `id`,`test`.`t2`.`cur_date` AS `cur_date` from `test`.`t2` where <in_optimizer>(`test`.`t2`.`id`,<exists>(select 1 AS `Not_used` from `test`.`t2` `x1` where ((`test`.`t2`.`cur_date` = 0) and (<cache>(`test`.`t2`.`id`) = `test`.`x1`.`id`)))) +Note 1003 select `test`.`t2`.`id` AS `id`,`test`.`t2`.`cur_date` AS `cur_date` from `test`.`t2` semi join (`test`.`t2` `x1`) where ((`test`.`x1`.`id` = `test`.`t2`.`id`) and (`test`.`t2`.`cur_date` = 0)) select * from t2 where id in (select id from t2 as x1 where (t2.cur_date is null)); id cur_date

1 0

[Maria-developers] We need instructions for manually installing .deb
by Kristian Nielsen 09 Feb '10

09 Feb '10

Hi Daniel, We really need to put some instructions prominently on the download page for installing the .debs. I have seen users having big problems installing them. Just something like `dpkg --install *.deb` does not work at all :-( I actually had to spend like 15 minutes sorting out exactly how to install it, many users would have given up. Here are the commands that I found will work on Debian 5 amd64 (hopefully other .deb distros will be similar): sudo apt-get update sudo apt-get install libdbi-perl libdbd-mysql-perl psmisc sudo dpkg --install mysql-common_5.1.42-mariadb73_all.deb sudo dpkg --install libmariadbclient16_5.1.42-mariadb73_amd64.deb libmysqlclient16_5.1.42-mariadb73_amd64.deb mariadb-client_5.1.42-mariadb73_all.deb mariadb-client-5.1_5.1.42-mariadb73_amd64.deb mariadb-server_5.1.42-mariadb73_all.deb mariadb-server-5.1_5.1.42-mariadb73_amd64.deb Also, as soon as OurDelta has the 5.1.42 repositories up, we need a strong warning against manually installing the .debs, strongly suggesting the OurDelta repositories instead, as it will work much better for users. (With the repositories, just `apt-get install mariadb-server` will work, as will the Synaptic package manager GUI). I hope you will be able to sort out some instructions, else ask me and I will try to help. This is made somewhat more urgent since after FOSDEM a number of people will probably want to try out the .debs ASAP, as we advertised them ... - Kristian.

2 1

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2813: Added --connect-command="sql-string" to sql-bench test suite.
by noreply＠launchpad.net 09 Feb '10

09 Feb '10

------------------------------------------------------------ revno: 2813 committer: Michael Widenius <monty(a)askmonty.org> branch nick: maria-5.1 timestamp: Tue 2010-02-09 19:17:04 +0200 message: Added --connect-command="sql-string" to sql-bench test suite. This allows one to send an extra command to the mysql server to setup the environment before starting tests. modified: sql-bench/bench-init.pl.sh sql-bench/server-cfg.sh -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2813)
by Michael Widenius 09 Feb '10

09 Feb '10

#At lp:maria based on revid:knielsen@knielsen-hq.org-20100131153603-9uo859vt0kra7tbz 2813 Michael Widenius 2010-02-09 Added --connect-command="sql-string" to sql-bench test suite. This allows one to send an extra command to the mysql server to setup the environment before starting tests. modified: sql-bench/bench-init.pl.sh sql-bench/server-cfg.sh === modified file 'sql-bench/bench-init.pl.sh' --- a/sql-bench/bench-init.pl.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/bench-init.pl.sh 2010-02-09 17:17:04 +0000 @@ -40,7 +40,7 @@ require "$pwd/server-cfg" || die "Can't $|=1; # Output data immediately $opt_skip_test=$opt_skip_create=$opt_skip_delete=$opt_verbose=$opt_fast_insert=$opt_lock_tables=$opt_debug=$opt_skip_delete=$opt_fast=$opt_force=$opt_log=$opt_use_old_results=$opt_help=$opt_odbc=$opt_small_test=$opt_small_tables=$opt_samll_key_tables=$opt_stage=$opt_old_headers=$opt_die_on_errors=$opt_tcpip=$opt_random=$opt_only_missing_tests=0; -$opt_cmp=$opt_user=$opt_password=$opt_connect_options=""; +$opt_cmp=$opt_user=$opt_password=$opt_connect_options=$opt_connect_command= ""; $opt_server="mysql"; $opt_dir="output"; $opt_host="localhost";$opt_database="test"; $opt_machine=""; $opt_suffix=""; @@ -59,7 +59,7 @@ $log_prog_args=join(" ", skip_arguments( "use-old-results","skip-test", "optimization","hw", "machine", "dir", "suffix", "log")); -GetOptions("skip-test=s","comments=s","cmp=s","server=s","user=s","host=s","database=s","password=s","loop-count=i","row-count=i","skip-create","skip-delete","verbose","fast-insert","lock-tables","debug","fast","force","field-count=i","regions=i","groups=i","time-limit=i","log","use-old-results","machine=s","dir=s","suffix=s","help","odbc","small-test","small-tables","small-key-tables","stage=i","threads=i","random","old-headers","die-on-errors","create-options=s","hires","tcpip","silent","optimization=s","hw=s","socket=s","connect-options=s","only-missing-tests") || usage(); +GetOptions("skip-test=s","comments=s","cmp=s","server=s","user=s","host=s","database=s","password=s","loop-count=i","row-count=i","skip-create","skip-delete","verbose","fast-insert","lock-tables","debug","fast","force","field-count=i","regions=i","groups=i","time-limit=i","log","use-old-results","machine=s","dir=s","suffix=s","help","odbc","small-test","small-tables","small-key-tables","stage=i","threads=i","random","old-headers","die-on-errors","create-options=s","hires","tcpip","silent","optimization=s","hw=s","socket=s","connect-options=s","connect-command=s","only-missing-tests") || usage(); usage() if ($opt_help); $server=get_server($opt_server,$opt_host,$opt_database,$opt_odbc, @@ -595,6 +595,10 @@ All benchmarks takes the following optio Add options, which uses at DBI connect. For example --connect-options=mysql_read_default_file=/etc/my.cnf. +--connect-command='SQL command' + Initialization command to execute when logged in. Useful for setting + up the environment. + EOF exit(0); } === modified file 'sql-bench/server-cfg.sh' --- a/sql-bench/server-cfg.sh 2009-05-29 13:40:55 +0000 +++ b/sql-bench/server-cfg.sh 2010-02-09 17:17:04 +0000 @@ -249,6 +249,11 @@ sub connect die "Got error: '$DBI::errstr' when connecting to " . $self->{'data_source'} ." with user: '$main::opt_user' password: '$main::opt_password'\n"; $dbh->do("SET OPTION LOG_OFF=1,UPDATE_LOG=0"); + if ($main::opt_connect_command ne "") + { + $dbh->do($main::opt_connect_command) or + die "Can't execute connect_command: $main::opt_connect_command error: $DBI::errstr\n"; + } return $dbh; }

1 0

[Maria-developers] New (by Serg): mutex/condition service (83)
by worklog-noreply＠askmonty.org 09 Feb '10

09 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: mutex/condition service CREATION DATE..: Tue, 09 Feb 2010, 18:20 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 83 (http://askmonty.org/worklog/?tid=83) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: DESCRIPTION: create a service that provides access to server implementation of mutexes, conditions, and rwlocks. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Serg): mutex/condition service (83)
by worklog-noreply＠askmonty.org 09 Feb '10

09 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: mutex/condition service CREATION DATE..: Tue, 09 Feb 2010, 18:20 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 83 (http://askmonty.org/worklog/?tid=83) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: DESCRIPTION: create a service that provides access to server implementation of mutexes, conditions, and rwlocks. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Serg): add charset support to ftparser plugins (82)
by worklog-noreply＠askmonty.org 09 Feb '10

09 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: add charset support to ftparser plugins CREATION DATE..: Tue, 09 Feb 2010, 18:19 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 82 (http://askmonty.org/worklog/?tid=82) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 20 (hours remain) ORIG. ESTIMATE.: 20 PROGRESS NOTES: DESCRIPTION: fulltext parser plugins don't know the character set of the text they parse and have no access to the charset support functionality in the server. extend the ftparser api to provide charset of the parsed text ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Serg): add charset support to ftparser plugins (82)
by worklog-noreply＠askmonty.org 09 Feb '10

09 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: add charset support to ftparser plugins CREATION DATE..: Tue, 09 Feb 2010, 18:19 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 82 (http://askmonty.org/worklog/?tid=82) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 20 (hours remain) ORIG. ESTIMATE.: 20 PROGRESS NOTES: DESCRIPTION: fulltext parser plugins don't know the character set of the text they parse and have no access to the charset support functionality in the server. extend the ftparser api to provide charset of the parsed text ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Serg): create a charset service (81)
by worklog-noreply＠askmonty.org 09 Feb '10

09 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: create a charset service CREATION DATE..: Tue, 09 Feb 2010, 18:17 SUPERVISOR.....: Bothorsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 81 (http://askmonty.org/worklog/?tid=81) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 20 (hours remain) ORIG. ESTIMATE.: 20 PROGRESS NOTES: DESCRIPTION: create a Service for CHARSET_INFO and related functionality ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Serg): create a charset service (81)
by worklog-noreply＠askmonty.org 09 Feb '10

09 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: create a charset service CREATION DATE..: Tue, 09 Feb 2010, 18:17 SUPERVISOR.....: Bothorsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 81 (http://askmonty.org/worklog/?tid=81) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 20 (hours remain) ORIG. ESTIMATE.: 20 PROGRESS NOTES: DESCRIPTION: create a Service for CHARSET_INFO and related functionality ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Serg): show plugins soname ... (80)
by worklog-noreply＠askmonty.org 09 Feb '10

09 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: show plugins soname ... CREATION DATE..: Tue, 09 Feb 2010, 18:16 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 80 (http://askmonty.org/worklog/?tid=80) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: DESCRIPTION: To install a plugin one needs to know its name in advance. There is no way to examine a .so to see what it contains. SHOW PLUGINS SONAME ... command will list all - including uninstalled - plugins from a given .so ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Serg): show plugins soname ... (80)
by worklog-noreply＠askmonty.org 09 Feb '10

09 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: show plugins soname ... CREATION DATE..: Tue, 09 Feb 2010, 18:16 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 80 (http://askmonty.org/worklog/?tid=80) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: DESCRIPTION: To install a plugin one needs to know its name in advance. There is no way to examine a .so to see what it contains. SHOW PLUGINS SONAME ... command will list all - including uninstalled - plugins from a given .so ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Serg): duplicate plugin names (79)
by worklog-noreply＠askmonty.org 09 Feb '10

09 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: duplicate plugin names CREATION DATE..: Tue, 09 Feb 2010, 18:14 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 79 (http://askmonty.org/worklog/?tid=79) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: DESCRIPTION: currently plugin names should be globally unique in the server. In fact, they only need to be unique within a plugin type. There is no reason why there can not be a fulltext parser plugin "csv" and storage engine plugin "csv". This task allows plugins with the same name to be installed as long as they are of different plugin type. It also adds INSTALL plugin_type PLUGIN name SONAME "path"; command (e.g. INSTALL FTPARSER PLUGIN csv SONAME "csv.so"), to resolve the case when one .so has two plugins with the same name. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Serg): duplicate plugin names (79)
by worklog-noreply＠askmonty.org 09 Feb '10

09 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: duplicate plugin names CREATION DATE..: Tue, 09 Feb 2010, 18:14 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 79 (http://askmonty.org/worklog/?tid=79) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: DESCRIPTION: currently plugin names should be globally unique in the server. In fact, they only need to be unique within a plugin type. There is no reason why there can not be a fulltext parser plugin "csv" and storage engine plugin "csv". This task allows plugins with the same name to be installed as long as they are of different plugin type. It also adds INSTALL plugin_type PLUGIN name SONAME "path"; command (e.g. INSTALL FTPARSER PLUGIN csv SONAME "csv.so"), to resolve the case when one .so has two plugins with the same name. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Serg): INSTALL PLUGIN * (78)
by worklog-noreply＠askmonty.org 09 Feb '10

09 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: INSTALL PLUGIN * CREATION DATE..: Tue, 09 Feb 2010, 18:10 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 78 (http://askmonty.org/worklog/?tid=78) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 10 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: DESCRIPTION: InnoDB, XtraDB, PBXT (at least) come with a storage engine plugin and many information_schema plugins in one .so file. Currently one needs to install them all one by one. INSTALL PLUGIN * SONAME xxx would be a more convenient way to install everything at once. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Serg): INSTALL PLUGIN * (78)
by worklog-noreply＠askmonty.org 09 Feb '10

09 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: INSTALL PLUGIN * CREATION DATE..: Tue, 09 Feb 2010, 18:10 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 78 (http://askmonty.org/worklog/?tid=78) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 10 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: DESCRIPTION: InnoDB, XtraDB, PBXT (at least) come with a storage engine plugin and many information_schema plugins in one .so file. Currently one needs to install them all one by one. INSTALL PLUGIN * SONAME xxx would be a more convenient way to install everything at once. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Rev 2740: Group commit for maria storage engine. in file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/
by sanja＠askmonty.org 09 Feb '10

09 Feb '10

At file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/ ------------------------------------------------------------ revno: 2740 revision-id: sanja(a)askmonty.org-20100209082807-om89773tey55ok66 parent: knielsen(a)knielsen-hq.org-20100201190519-b9uktnn90rwwiile committer: sanja(a)askmonty.org branch nick: work-maria-5.2-groupcommit timestamp: Tue 2010-02-09 10:28:07 +0200 message: Group commit for maria storage engine. Diff too large for email (1447 lines, the limit is 1000).

1 0

[Maria-developers] Rev 2740: Group commit for maria storage engine. in file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/
by sanja＠askmonty.org 09 Feb '10

09 Feb '10

At file:///Users/bell/maria/bzr/work-maria-5.2-groupcommit/ ------------------------------------------------------------ revno: 2740 revision-id: sanja(a)askmonty.org-20100209081347-si2jhe5q4olk5a08 parent: knielsen(a)knielsen-hq.org-20100201190519-b9uktnn90rwwiile committer: sanja(a)askmonty.org branch nick: work-maria-5.2-groupcommit timestamp: Tue 2010-02-09 10:13:47 +0200 message: Group commit for maria storage engine. Diff too large for email (1444 lines, the limit is 1000).

1 0

[Maria-developers] Rev 2756: Apply Jorgen Loland's fix: in file:///home/psergey/dev/maria-5.3-subqueries-r3/
by Sergey Petrunya 08 Feb '10

08 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r3/ ------------------------------------------------------------ revno: 2756 revision-id: psergey(a)askmonty.org-20100208133030-e4zjy15b7o14ud8c parent: psergey(a)askmonty.org-20100208132741-nj9zq7z8nwlfwchq committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r3 timestamp: Mon 2010-02-08 15:30:30 +0200 message: Apply Jorgen Loland's fix: Bug#45221: Query "SELECT pk FROM C WHERE pk IN (SELECT int_key)" failing XOR conditions are not optimized, and Item_cond_xor therefore acts like type Func_item even though it inherits from Item_cond. A subtle difference between Item_func and Item_cond is that you can get the children Items from the former by calling arguments(), and from the latter by calling argument_list(). However, since Item_cond_xor inherits from Item_cond, arguments() did not return any Items. The fact that Item_cond_xor::arguments() did not return it's children items lead to a problem for make_cond_for_index(); the method accepted that XOR items on unindexed columns were pushed using ICP. ICP evaluation of non-indexed columns does not (and should not) work. The fix for this bug is to make Item_cond_xor return it's children items when the arguments() method is used. This makes Item_cond_xor behave more like Item_func and in turn allows make_cond_for_index() to discover any conflicting children Items. This is a temporary fix and should be removed when Item_cond_xor is optimized. === modified file 'sql/item_cmpfunc.h' --- a/sql/item_cmpfunc.h 2010-01-17 14:55:08 +0000 +++ b/sql/item_cmpfunc.h 2010-02-08 13:30:30 +0000 @@ -1715,14 +1715,34 @@ class Item_cond_xor :public Item_cond { public: - Item_cond_xor() :Item_cond() {} - Item_cond_xor(Item *i1,Item *i2) :Item_cond(i1,i2) {} + Item_cond_xor(Item *i1,Item *i2) :Item_cond(i1,i2) + { + /* + Items must be stored in args[] as well because this Item_cond is + treated as a FUNC_ITEM (see type()). I.e., users of it will get + it's children by calling arguments(), not argument_list(). This + is a temporary solution until XOR is optimized and treated like + a full Item_cond citizen. + */ + arg_count= 2; + args= tmp_arg; + args[0]= i1; + args[1]= i2; + } enum Functype functype() const { return COND_XOR_FUNC; } /* TODO: remove the next line when implementing XOR optimization */ enum Type type() const { return FUNC_ITEM; } longlong val_int(); const char *func_name() const { return "xor"; } void top_level_item() {} + /* Since child Items are stored in args[], Items cannot be added. + However, since Item_cond_xor is treated as a FUNC_ITEM (see + type()), the methods below should never be called. + */ + bool add(Item *item) { DBUG_ASSERT(FALSE); return FALSE; } + bool add_at_head(Item *item) { DBUG_ASSERT(FALSE); return FALSE; } + bool add_at_head(List<Item> *nlist) { DBUG_ASSERT(FALSE); return FALSE; } + void copy_andor_arguments(THD *thd, Item_cond *item) { DBUG_ASSERT(FALSE); } };

1 0

[Maria-developers] Rev 2755: Subquery optimizations: backport in file:///home/psergey/dev/maria-5.3-subqueries-r3/
by Sergey Petrunya 08 Feb '10

08 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r3/ ------------------------------------------------------------ revno: 2755 revision-id: psergey(a)askmonty.org-20100208132741-nj9zq7z8nwlfwchq parent: psergey(a)askmonty.org-20100208131019-q3vmltpo7vu9ihz6 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r3 timestamp: Mon 2010-02-08 15:27:41 +0200 message: Subquery optimizations: backport - Test result updates (checked) === modified file 'mysql-test/r/group_by.result' --- a/mysql-test/r/group_by.result 2009-02-26 17:17:06 +0000 +++ b/mysql-test/r/group_by.result 2010-02-08 13:27:41 +0000 @@ -1542,8 +1542,8 @@ EXPLAIN SELECT 1 FROM t1 WHERE a IN (SELECT a FROM t1 USE INDEX (i2) IGNORE INDEX (i2)); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t1 index NULL PRIMARY 4 NULL 144 Using where; Using index -2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 144 Using where +1 PRIMARY t1 index PRIMARY,i2 PRIMARY 4 NULL 144 Using index +1 PRIMARY t1 ALL NULL NULL NULL NULL 144 Using where; FirstMatch(t1) CREATE TABLE t2 (a INT, b INT, KEY(a)); INSERT INTO t2 VALUES (1, 1), (2, 2), (3,3), (4,4); EXPLAIN SELECT a, SUM(b) FROM t2 GROUP BY a LIMIT 2; @@ -1555,8 +1555,8 @@ EXPLAIN SELECT 1 FROM t2 WHERE a IN (SELECT a FROM t1 USE INDEX (i2) IGNORE INDEX (i2)); id select_type table type possible_keys key key_len ref rows Extra -1 PRIMARY t2 index NULL a 5 NULL 4 Using where; Using index -2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 144 Using where +1 PRIMARY t2 index a a 5 NULL 4 Using index +1 PRIMARY t1 ALL NULL NULL NULL NULL 144 Using where; FirstMatch(t2) SHOW VARIABLES LIKE 'old'; Variable_name Value old OFF

1 0

[Maria-developers] Rev 2754: Subquery backport: pass the correct item (the one that was resolved in in file:///home/psergey/dev/maria-5.3-subqueries-r3/
by Sergey Petrunya 08 Feb '10

08 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r3/ ------------------------------------------------------------ revno: 2754 revision-id: psergey(a)askmonty.org-20100208131019-q3vmltpo7vu9ihz6 parent: psergey(a)askmonty.org-20100208130923-g38q4uiyu90g60w9 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r3 timestamp: Mon 2010-02-08 15:10:19 +0200 message: Subquery backport: pass the correct item (the one that was resolved in ancestor select and has item->fixed==1) to mark_as_dependent === modified file 'sql/item.cc' --- a/sql/item.cc 2010-02-08 09:56:16 +0000 +++ b/sql/item.cc 2010-02-08 13:10:19 +0000 @@ -3643,7 +3643,9 @@ @param current current select @param resolved_item item which was resolved in outer SELECT(for warning) @param mark_item item which should be marked (can be differ in case of - substitution) + substitution) And also different when accessing VIEW + columns: in that case its Item_..._view_ref while + resolved_item is Item_field. */ static bool mark_as_dependent(THD *thd, SELECT_LEX *last, SELECT_LEX *current, @@ -3657,7 +3659,7 @@ /* store pointer on SELECT_LEX from which item is dependent */ if (mark_item) mark_item->depended_from= last; - if (current->mark_as_dependent(thd, last, resolved_item)) + if (current->mark_as_dependent(thd, last, /*resolved_item*/ mark_item)) //psergey-fix2 return TRUE; if (thd->lex->describe & DESCRIBE_EXTENDED) {

1 0

[Maria-developers] Rev 2753: Subquery backport: Update test results (checked) in file:///home/psergey/dev/maria-5.3-subqueries-r3/
by Sergey Petrunya 08 Feb '10

08 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r3/ ------------------------------------------------------------ revno: 2753 revision-id: psergey(a)askmonty.org-20100208130923-g38q4uiyu90g60w9 parent: psergey(a)askmonty.org-20100208125530-l0o01n6lxk95cn78 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r3 timestamp: Mon 2010-02-08 15:09:23 +0200 message: Subquery backport: Update test results (checked) === modified file 'mysql-test/r/explain.result' --- a/mysql-test/r/explain.result 2009-12-15 07:16:46 +0000 +++ b/mysql-test/r/explain.result 2010-02-08 13:09:23 +0000 @@ -171,7 +171,7 @@ EXPLAIN SELECT OUTR.dt FROM t1 AS OUTR WHERE OUTR.dt IN (SELECT INNR.dt FROM t2 AS INNR WHERE OUTR.dt IS NULL ); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY OUTR ALL NULL NULL NULL NULL 2 Using where -2 DEPENDENT SUBQUERY INNR ALL NULL NULL NULL NULL 2 Using where +1 PRIMARY INNR ALL NULL NULL NULL NULL 2 Using where; FirstMatch(OUTR) flush tables; SELECT OUTR.dt FROM t1 AS OUTR WHERE OUTR.dt IN (SELECT INNR.dt FROM t2 AS INNR WHERE OUTR.dt IS NULL ); dt @@ -179,7 +179,7 @@ EXPLAIN SELECT OUTR.dt FROM t1 AS OUTR WHERE OUTR.dt IN ( SELECT INNR.dt FROM t2 AS INNR WHERE OUTR.t < '2005-11-13 7:41:31' ); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY OUTR ALL NULL NULL NULL NULL 2 Using where -2 DEPENDENT SUBQUERY INNR ALL NULL NULL NULL NULL 2 Using where +1 PRIMARY INNR ALL NULL NULL NULL NULL 2 Using where; FirstMatch(OUTR) flush tables; SELECT OUTR.dt FROM t1 AS OUTR WHERE OUTR.dt IN ( SELECT INNR.dt FROM t2 AS INNR WHERE OUTR.t < '2005-11-13 7:41:31' ); dt === modified file 'mysql-test/r/group_min_max.result' --- a/mysql-test/r/group_min_max.result 2009-08-30 07:03:37 +0000 +++ b/mysql-test/r/group_min_max.result 2010-02-08 13:09:23 +0000 @@ -2256,7 +2256,7 @@ a IN (SELECT max(b) FROM t1 GROUP BY a HAVING a < 2); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t1_outer index NULL a 10 NULL 15 Using where; Using index -2 DEPENDENT SUBQUERY t1 index NULL a 10 NULL 1 Using index +2 SUBQUERY t1 range NULL a 5 NULL 8 Using index for group-by EXPLAIN SELECT 1 FROM t1 AS t1_outer GROUP BY a HAVING a > (SELECT max(b) FROM t1 GROUP BY a HAVING a < 2); id select_type table type possible_keys key key_len ref rows Extra === modified file 'mysql-test/r/subselect3_jcl6.result' --- a/mysql-test/r/subselect3_jcl6.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect3_jcl6.result 2010-02-08 13:09:23 +0000 @@ -1140,7 +1140,7 @@ flush status; select count(*) from t0 A, t0 B, t0 C, t0 D where D.a in (select a from t1 E); count(*) -4999 +5000 show status like 'Created_tmp_disk_tables'; Variable_name Value Created_tmp_disk_tables 1

1 0

[Maria-developers] Rev 2740: options for CREATE TABLE (MWL#43) (version after first review). in file:///home/bell/maria/bzr/work-maria-5.2-createoptions/
by sanja＠askmonty.org 05 Feb '10

05 Feb '10

At file:///home/bell/maria/bzr/work-maria-5.2-createoptions/ ------------------------------------------------------------ revno: 2740 revision-id: sanja(a)askmonty.org-20100205170316-gg4nio1p81cpmjop parent: knielsen(a)knielsen-hq.org-20100201190519-b9uktnn90rwwiile committer: sanja(a)askmonty.org branch nick: work-maria-5.2-createoptions timestamp: Fri 2010-02-05 19:03:16 +0200 message: options for CREATE TABLE (MWL#43) (version after first review). Diff too large for email (1280 lines, the limit is 1000).

2 1

[Maria-developers] Updated (by Serg): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 05 Feb '10

05 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen, Serg CATEGORY.......: Server-Sprint TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 20 ESTIMATE.......: 35 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Serg - Fri, 05 Feb 2010, 14:04)=-=- Observers changed: Knielsen,Serg -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Category updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Status updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +In-Progress -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 -=-=(Alexi - Sat, 19 Dec 2009, 15:58)=-=- Low Level Design modified. --- /tmp/wklog.47.old.15437 2009-12-19 13:58:12.000000000 +0000 +++ /tmp/wklog.47.new.15437 2009-12-19 13:58:12.000000000 +0000 @@ -1 +1,337 @@ +Content +~~~~~~~ + 1. Annotate_rows event number + 2. Outline of Annotate_rows event behavior + 3. How Master writes Annotate_rows events to the binary log + 4. How slave treats replicate-annotate-rows-events option + 5. How slave IO thread requests Annotate_rows events + 6. How master executes the request + 7. How slave SQL thread processes Annotate_rows events + 8. General remarks + +1. Annotate_rows event number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To avoid possible event numbers conflict with MySQL/Sun, we leave a gap +between the last MySQL event number and the Annotate_rows event number: + + enum Log_event_type + { ... + INCIDENT_EVENT= 26, + // New MySQL event numbers are to be added here + MYSQL_EVENTS_END, + + MARIA_EVENTS_BEGIN= 51, + // New Maria event numbers start from here + ANNOTATE_ROWS_EVENT= 51, + + ENUM_END_EVENT + }; + +together with the corresponding extension of 'post_header_len' array in the +Format description event. (This extension does not affect the compatibility +of the binary log). Here is how Format description event looks like with +this extension: + + ************************ + FORMAT_DESCRIPTION_EVENT + ************************ + 00000004 | A1 A0 2C 4B | time_when = 1261215905 + 00000008 | 0F | event_type = 15 + 00000009 | 64 00 00 00 | server_id = 100 + 0000000D | 7F 00 00 00 | event_len = 127 + 00000011 | 83 00 00 00 | log_pos = 00000083 + 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F + ------------------------ + 00000017 | 04 00 | binlog_ver = 4 + 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log + ..... ... + 0000004B | A1 A0 2C 4B | time_created = 1261215905 + 0000004F | 13 | common_header_len = 19 + ------------------------ + post_header_len + ------------------------ + 00000050 | 38 | 56 - START_EVENT_V3 [1] + ..... ... + 00000069 | 02 | 2 - INCIDENT_EVENT [26] + 0000006A | 00 | 0 - RESERVED [27] + ..... ... + 00000081 | 00 | 0 - RESERVED [50] + 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + ************************ + +2. Outline of Annotate_rows event behavior +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Each Annotate_rows_log_event object has two private members describing the +corresponding query: + + char *m_query_txt; + uint m_query_len; + +When the object is created for writing to a binary log, this query is taken +from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix +as well as other implementation details): + + Annotate_rows_log_event(THD *thd) + { + m_query_txt = thd->query(); + m_query_len = thd->query_length(); + } + +When the object is read from a binary log, the query is taken from the buffer +containing the binary log representation of the event (this buffer is allocated +in Log_event object from which all Log events are derived): + + Annotate_rows_log_event(char *buf, uint event_len, + Format_description_log_event *desc) + { + m_query_len = event_len - desc->common_header_len; + m_query_txt = buf + desc->common_header_len; + } + +The events are written to the binary log by the Log_event::write() member +which calls virtual write_data_header() and write_data_body() members +("data header" and "post header" are synonym in replication terminology). +In our case, data header is empty and data body is just the query: + + bool write_data_body(IO_CACHE *file) + { + return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); + } + +Printing the event is just printing the query: + + void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) + { + my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); + } + +3. How Master writes Annotate_rows events to the binary log +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The event is written to the binary log just before the group of Table_map +events which precede corresponding Rows events (one query may generate +several Table map events in the binary log, but the corresponding +Annotate_rows event must be written only once before the first Table map +event; hence the boolean variable 'with_annotate' below): + + int write_locked_table_maps(THD *thd) + { ... + bool with_annotate= thd->variables.binlog_annotate_rows_events; + ... + for (uint i= 0; i < ... <number of tables> ...; ++i) + { ... + thd->binlog_write_table_map(table, ..., with_annotate); + with_annotate= 0; // write Annotate_event not more than once + ... + } + ... + } + + int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) + { ... + Table_map_log_event the_event(...); + ... + if (with_annotate) + { + Annotate_rows_log_event anno(this); + mysql_bin_log.write(&anno); + } + + mysql_bin_log.write(&the_event); + ... + } + +4. How slave treats replicate-annotate-rows-events option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The replicate-annotate-rows-events option is treated just as the session +value of the binlog_annotate_rows_events variable for the slave IO and +SQL threads. This setting is done during initialization of these threads: + + pthread_handler_t handle_slave_io(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_IO); + ... + } + + pthread_handler_t handle_slave_sql(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_SQL); + ... + } + + int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) + { ... + thd->variables.binlog_annotate_rows_events= + opt_replicate_annotate_rows_events; + ... + } + +5. How slave IO thread requests Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When requesting an event, the slave should inform the master whether +it should send Annotate_rows events or not. To that end we add a new +BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: + + #define BINLOG_DUMP_NON_BLOCK 1 + #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 + + pthread_handler_t handle_slave_io(void *arg) + { ... + request_dump(mysql, ...); + ... + } + + int request_dump(MYSQL* mysql, ...) + { ... + if (opt_log_slave_updates && + mi->io_thd->variables.binlog_annotate_rows_events) + binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; + ... + int2store(buf + 4, binlog_flags); + ... + simple_command(mysql, COM_BINLOG_DUMP, buf, ...); + ... + } + +6. How master executes the request +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + case COM_BINLOG_DUMP: + { ... + flags= uint2korr(packet + 4); + ... + mysql_binlog_send(thd, ..., flags); + ... + } + + void mysql_binlog_send(THD* thd, ..., ushort flags) + { ... + Log_event::read_log_event(&log, packet, ...); + ... + if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || + thd->server_id == 0 /* slave == mysqlbinlog */ ) + { + my_net_write(net, packet->ptr(), packet->length()); + } + ... + } + +7. How slave SQL thread processes Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The slave processes each recieved event by "applying" it, i.e. by +calling the Log_event::apply_event() function which in turn calls +the virtual do_apply_event() member specific for each type of the +event. + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev = next_event(rli); + ... + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + + int apply_event_and_update_pos(Log_event *ev, ...) + { ... + ev->apply_event(...); + ... + } + + int Log_event::apply_event(...) + { + return do_apply_event(...); + } + +What does it mean to "apply" an Annotate_rows event? It means to set current +thd query to that of the described by the event, i.e. to the query which +caused the subsequent Rows events (see "How Master writes Annotate_rows +events to the binary log" to follow what happens further when the subsequent +Rows events is applied): + + int Annotate_rows_log_event::do_apply_event(...) + { + thd->set_query(m_query_txt, m_query_len); + } + +NOTE. I am not sure, but possibly current values of thd->query and +thd->query_length should be saved before calling set_query() and to be +restored on the Annotate_rows_log_event object deletion. +Is it really needed ? + +After calling this do_apply_event() function we may not delete the +Annotate_rows_log_event object immediatedly (see exec_relay_log_event() +above) because thd->query now points to the string inside this object. +We may keep the pointer to this object in the Relay_log_info: + + class Relay_log_info + { + public: + ... + void set_annotate_event(Annotate_rows_log_event*); + Annotate_rows_log_event* get_annotate_event(); + void free_annotate_event(); + ... + private: + Annotate_rows_log_event* m_annotate_event; + }; + +When the saved Annotate_rows object may be deleted? When all corresponding +Rows events will be processed, i.e. before processing the first non-Rows +event (note that Annotate_rows object resides in the binary log *after* +the (possible) 'BEGIN' Query event which accompanies the rows events; note +also that this deletion is adjusted with the case when some or all +corresponding Rows events are filtered out by replicate filter rules): + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev= next_event(rli); + ... + if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) + rli->free_annotate_event(); + + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + rli->set_annotate_event((Annotate_rows_log_event*) ev); + else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + +where + + #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ + (type) == WRITE_ROWS_EVENT || \ + (type) == UPDATE_ROWS_EVENT || \ + (type) == DELETE_ROWS_EVENT ) + +8. General remarks +~~~~~~~~~~~~~~~~~~ +Kristian noticed that introducing new log event type should be coordinated +somehow with MySQL/Sun: + + Kristian: The numeric code for this event must be assigned carefully. + It should be coordinated with MySQL/Sun, otherwise we can get into a + situation where MySQL uses the same numeric code for one event that + MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two + impossible. + Alex: I reserved about 20 numbers not to have possible conflicts + with MySQL. + Kristian: Still, I think it would be appropriate to send a polite email + to internals(a)lists.mysql.com about this and suggesting to reserve the + event number. + +Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT +flag taking into account that MySQL/Sun may also introduce a flag with the +same value to be used in the request_dump-mysql_binlog_send interface. +But this is mainly the question of merging: if a conflict concerning this +flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value +(this does not require additional changes in the code). ------------------------------------------------------------ -=-=(View All Progress Notes, 23 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Serg): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 05 Feb '10

05 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen, Serg CATEGORY.......: Server-Sprint TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 20 ESTIMATE.......: 35 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Serg - Fri, 05 Feb 2010, 14:04)=-=- Observers changed: Knielsen,Serg -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Category updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Status updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +In-Progress -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 -=-=(Alexi - Sat, 19 Dec 2009, 15:58)=-=- Low Level Design modified. --- /tmp/wklog.47.old.15437 2009-12-19 13:58:12.000000000 +0000 +++ /tmp/wklog.47.new.15437 2009-12-19 13:58:12.000000000 +0000 @@ -1 +1,337 @@ +Content +~~~~~~~ + 1. Annotate_rows event number + 2. Outline of Annotate_rows event behavior + 3. How Master writes Annotate_rows events to the binary log + 4. How slave treats replicate-annotate-rows-events option + 5. How slave IO thread requests Annotate_rows events + 6. How master executes the request + 7. How slave SQL thread processes Annotate_rows events + 8. General remarks + +1. Annotate_rows event number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To avoid possible event numbers conflict with MySQL/Sun, we leave a gap +between the last MySQL event number and the Annotate_rows event number: + + enum Log_event_type + { ... + INCIDENT_EVENT= 26, + // New MySQL event numbers are to be added here + MYSQL_EVENTS_END, + + MARIA_EVENTS_BEGIN= 51, + // New Maria event numbers start from here + ANNOTATE_ROWS_EVENT= 51, + + ENUM_END_EVENT + }; + +together with the corresponding extension of 'post_header_len' array in the +Format description event. (This extension does not affect the compatibility +of the binary log). Here is how Format description event looks like with +this extension: + + ************************ + FORMAT_DESCRIPTION_EVENT + ************************ + 00000004 | A1 A0 2C 4B | time_when = 1261215905 + 00000008 | 0F | event_type = 15 + 00000009 | 64 00 00 00 | server_id = 100 + 0000000D | 7F 00 00 00 | event_len = 127 + 00000011 | 83 00 00 00 | log_pos = 00000083 + 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F + ------------------------ + 00000017 | 04 00 | binlog_ver = 4 + 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log + ..... ... + 0000004B | A1 A0 2C 4B | time_created = 1261215905 + 0000004F | 13 | common_header_len = 19 + ------------------------ + post_header_len + ------------------------ + 00000050 | 38 | 56 - START_EVENT_V3 [1] + ..... ... + 00000069 | 02 | 2 - INCIDENT_EVENT [26] + 0000006A | 00 | 0 - RESERVED [27] + ..... ... + 00000081 | 00 | 0 - RESERVED [50] + 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + ************************ + +2. Outline of Annotate_rows event behavior +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Each Annotate_rows_log_event object has two private members describing the +corresponding query: + + char *m_query_txt; + uint m_query_len; + +When the object is created for writing to a binary log, this query is taken +from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix +as well as other implementation details): + + Annotate_rows_log_event(THD *thd) + { + m_query_txt = thd->query(); + m_query_len = thd->query_length(); + } + +When the object is read from a binary log, the query is taken from the buffer +containing the binary log representation of the event (this buffer is allocated +in Log_event object from which all Log events are derived): + + Annotate_rows_log_event(char *buf, uint event_len, + Format_description_log_event *desc) + { + m_query_len = event_len - desc->common_header_len; + m_query_txt = buf + desc->common_header_len; + } + +The events are written to the binary log by the Log_event::write() member +which calls virtual write_data_header() and write_data_body() members +("data header" and "post header" are synonym in replication terminology). +In our case, data header is empty and data body is just the query: + + bool write_data_body(IO_CACHE *file) + { + return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); + } + +Printing the event is just printing the query: + + void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) + { + my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); + } + +3. How Master writes Annotate_rows events to the binary log +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The event is written to the binary log just before the group of Table_map +events which precede corresponding Rows events (one query may generate +several Table map events in the binary log, but the corresponding +Annotate_rows event must be written only once before the first Table map +event; hence the boolean variable 'with_annotate' below): + + int write_locked_table_maps(THD *thd) + { ... + bool with_annotate= thd->variables.binlog_annotate_rows_events; + ... + for (uint i= 0; i < ... <number of tables> ...; ++i) + { ... + thd->binlog_write_table_map(table, ..., with_annotate); + with_annotate= 0; // write Annotate_event not more than once + ... + } + ... + } + + int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) + { ... + Table_map_log_event the_event(...); + ... + if (with_annotate) + { + Annotate_rows_log_event anno(this); + mysql_bin_log.write(&anno); + } + + mysql_bin_log.write(&the_event); + ... + } + +4. How slave treats replicate-annotate-rows-events option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The replicate-annotate-rows-events option is treated just as the session +value of the binlog_annotate_rows_events variable for the slave IO and +SQL threads. This setting is done during initialization of these threads: + + pthread_handler_t handle_slave_io(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_IO); + ... + } + + pthread_handler_t handle_slave_sql(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_SQL); + ... + } + + int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) + { ... + thd->variables.binlog_annotate_rows_events= + opt_replicate_annotate_rows_events; + ... + } + +5. How slave IO thread requests Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When requesting an event, the slave should inform the master whether +it should send Annotate_rows events or not. To that end we add a new +BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: + + #define BINLOG_DUMP_NON_BLOCK 1 + #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 + + pthread_handler_t handle_slave_io(void *arg) + { ... + request_dump(mysql, ...); + ... + } + + int request_dump(MYSQL* mysql, ...) + { ... + if (opt_log_slave_updates && + mi->io_thd->variables.binlog_annotate_rows_events) + binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; + ... + int2store(buf + 4, binlog_flags); + ... + simple_command(mysql, COM_BINLOG_DUMP, buf, ...); + ... + } + +6. How master executes the request +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + case COM_BINLOG_DUMP: + { ... + flags= uint2korr(packet + 4); + ... + mysql_binlog_send(thd, ..., flags); + ... + } + + void mysql_binlog_send(THD* thd, ..., ushort flags) + { ... + Log_event::read_log_event(&log, packet, ...); + ... + if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || + thd->server_id == 0 /* slave == mysqlbinlog */ ) + { + my_net_write(net, packet->ptr(), packet->length()); + } + ... + } + +7. How slave SQL thread processes Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The slave processes each recieved event by "applying" it, i.e. by +calling the Log_event::apply_event() function which in turn calls +the virtual do_apply_event() member specific for each type of the +event. + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev = next_event(rli); + ... + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + + int apply_event_and_update_pos(Log_event *ev, ...) + { ... + ev->apply_event(...); + ... + } + + int Log_event::apply_event(...) + { + return do_apply_event(...); + } + +What does it mean to "apply" an Annotate_rows event? It means to set current +thd query to that of the described by the event, i.e. to the query which +caused the subsequent Rows events (see "How Master writes Annotate_rows +events to the binary log" to follow what happens further when the subsequent +Rows events is applied): + + int Annotate_rows_log_event::do_apply_event(...) + { + thd->set_query(m_query_txt, m_query_len); + } + +NOTE. I am not sure, but possibly current values of thd->query and +thd->query_length should be saved before calling set_query() and to be +restored on the Annotate_rows_log_event object deletion. +Is it really needed ? + +After calling this do_apply_event() function we may not delete the +Annotate_rows_log_event object immediatedly (see exec_relay_log_event() +above) because thd->query now points to the string inside this object. +We may keep the pointer to this object in the Relay_log_info: + + class Relay_log_info + { + public: + ... + void set_annotate_event(Annotate_rows_log_event*); + Annotate_rows_log_event* get_annotate_event(); + void free_annotate_event(); + ... + private: + Annotate_rows_log_event* m_annotate_event; + }; + +When the saved Annotate_rows object may be deleted? When all corresponding +Rows events will be processed, i.e. before processing the first non-Rows +event (note that Annotate_rows object resides in the binary log *after* +the (possible) 'BEGIN' Query event which accompanies the rows events; note +also that this deletion is adjusted with the case when some or all +corresponding Rows events are filtered out by replicate filter rules): + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev= next_event(rli); + ... + if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) + rli->free_annotate_event(); + + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + rli->set_annotate_event((Annotate_rows_log_event*) ev); + else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + +where + + #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ + (type) == WRITE_ROWS_EVENT || \ + (type) == UPDATE_ROWS_EVENT || \ + (type) == DELETE_ROWS_EVENT ) + +8. General remarks +~~~~~~~~~~~~~~~~~~ +Kristian noticed that introducing new log event type should be coordinated +somehow with MySQL/Sun: + + Kristian: The numeric code for this event must be assigned carefully. + It should be coordinated with MySQL/Sun, otherwise we can get into a + situation where MySQL uses the same numeric code for one event that + MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two + impossible. + Alex: I reserved about 20 numbers not to have possible conflicts + with MySQL. + Kristian: Still, I think it would be appropriate to send a polite email + to internals(a)lists.mysql.com about this and suggesting to reserve the + event number. + +Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT +flag taking into account that MySQL/Sun may also introduce a flag with the +same value to be used in the request_dump-mysql_binlog_send interface. +But this is mainly the question of merging: if a conflict concerning this +flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value +(this does not require additional changes in the code). ------------------------------------------------------------ -=-=(View All Progress Notes, 23 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Serg): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 05 Feb '10

05 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen, Serg CATEGORY.......: Server-Sprint TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 20 ESTIMATE.......: 35 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Serg - Fri, 05 Feb 2010, 14:04)=-=- Observers changed: Knielsen,Serg -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Category updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Status updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +In-Progress -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 -=-=(Alexi - Sat, 19 Dec 2009, 15:58)=-=- Low Level Design modified. --- /tmp/wklog.47.old.15437 2009-12-19 13:58:12.000000000 +0000 +++ /tmp/wklog.47.new.15437 2009-12-19 13:58:12.000000000 +0000 @@ -1 +1,337 @@ +Content +~~~~~~~ + 1. Annotate_rows event number + 2. Outline of Annotate_rows event behavior + 3. How Master writes Annotate_rows events to the binary log + 4. How slave treats replicate-annotate-rows-events option + 5. How slave IO thread requests Annotate_rows events + 6. How master executes the request + 7. How slave SQL thread processes Annotate_rows events + 8. General remarks + +1. Annotate_rows event number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To avoid possible event numbers conflict with MySQL/Sun, we leave a gap +between the last MySQL event number and the Annotate_rows event number: + + enum Log_event_type + { ... + INCIDENT_EVENT= 26, + // New MySQL event numbers are to be added here + MYSQL_EVENTS_END, + + MARIA_EVENTS_BEGIN= 51, + // New Maria event numbers start from here + ANNOTATE_ROWS_EVENT= 51, + + ENUM_END_EVENT + }; + +together with the corresponding extension of 'post_header_len' array in the +Format description event. (This extension does not affect the compatibility +of the binary log). Here is how Format description event looks like with +this extension: + + ************************ + FORMAT_DESCRIPTION_EVENT + ************************ + 00000004 | A1 A0 2C 4B | time_when = 1261215905 + 00000008 | 0F | event_type = 15 + 00000009 | 64 00 00 00 | server_id = 100 + 0000000D | 7F 00 00 00 | event_len = 127 + 00000011 | 83 00 00 00 | log_pos = 00000083 + 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F + ------------------------ + 00000017 | 04 00 | binlog_ver = 4 + 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log + ..... ... + 0000004B | A1 A0 2C 4B | time_created = 1261215905 + 0000004F | 13 | common_header_len = 19 + ------------------------ + post_header_len + ------------------------ + 00000050 | 38 | 56 - START_EVENT_V3 [1] + ..... ... + 00000069 | 02 | 2 - INCIDENT_EVENT [26] + 0000006A | 00 | 0 - RESERVED [27] + ..... ... + 00000081 | 00 | 0 - RESERVED [50] + 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + ************************ + +2. Outline of Annotate_rows event behavior +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Each Annotate_rows_log_event object has two private members describing the +corresponding query: + + char *m_query_txt; + uint m_query_len; + +When the object is created for writing to a binary log, this query is taken +from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix +as well as other implementation details): + + Annotate_rows_log_event(THD *thd) + { + m_query_txt = thd->query(); + m_query_len = thd->query_length(); + } + +When the object is read from a binary log, the query is taken from the buffer +containing the binary log representation of the event (this buffer is allocated +in Log_event object from which all Log events are derived): + + Annotate_rows_log_event(char *buf, uint event_len, + Format_description_log_event *desc) + { + m_query_len = event_len - desc->common_header_len; + m_query_txt = buf + desc->common_header_len; + } + +The events are written to the binary log by the Log_event::write() member +which calls virtual write_data_header() and write_data_body() members +("data header" and "post header" are synonym in replication terminology). +In our case, data header is empty and data body is just the query: + + bool write_data_body(IO_CACHE *file) + { + return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); + } + +Printing the event is just printing the query: + + void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) + { + my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); + } + +3. How Master writes Annotate_rows events to the binary log +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The event is written to the binary log just before the group of Table_map +events which precede corresponding Rows events (one query may generate +several Table map events in the binary log, but the corresponding +Annotate_rows event must be written only once before the first Table map +event; hence the boolean variable 'with_annotate' below): + + int write_locked_table_maps(THD *thd) + { ... + bool with_annotate= thd->variables.binlog_annotate_rows_events; + ... + for (uint i= 0; i < ... <number of tables> ...; ++i) + { ... + thd->binlog_write_table_map(table, ..., with_annotate); + with_annotate= 0; // write Annotate_event not more than once + ... + } + ... + } + + int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) + { ... + Table_map_log_event the_event(...); + ... + if (with_annotate) + { + Annotate_rows_log_event anno(this); + mysql_bin_log.write(&anno); + } + + mysql_bin_log.write(&the_event); + ... + } + +4. How slave treats replicate-annotate-rows-events option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The replicate-annotate-rows-events option is treated just as the session +value of the binlog_annotate_rows_events variable for the slave IO and +SQL threads. This setting is done during initialization of these threads: + + pthread_handler_t handle_slave_io(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_IO); + ... + } + + pthread_handler_t handle_slave_sql(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_SQL); + ... + } + + int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) + { ... + thd->variables.binlog_annotate_rows_events= + opt_replicate_annotate_rows_events; + ... + } + +5. How slave IO thread requests Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When requesting an event, the slave should inform the master whether +it should send Annotate_rows events or not. To that end we add a new +BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: + + #define BINLOG_DUMP_NON_BLOCK 1 + #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 + + pthread_handler_t handle_slave_io(void *arg) + { ... + request_dump(mysql, ...); + ... + } + + int request_dump(MYSQL* mysql, ...) + { ... + if (opt_log_slave_updates && + mi->io_thd->variables.binlog_annotate_rows_events) + binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; + ... + int2store(buf + 4, binlog_flags); + ... + simple_command(mysql, COM_BINLOG_DUMP, buf, ...); + ... + } + +6. How master executes the request +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + case COM_BINLOG_DUMP: + { ... + flags= uint2korr(packet + 4); + ... + mysql_binlog_send(thd, ..., flags); + ... + } + + void mysql_binlog_send(THD* thd, ..., ushort flags) + { ... + Log_event::read_log_event(&log, packet, ...); + ... + if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || + thd->server_id == 0 /* slave == mysqlbinlog */ ) + { + my_net_write(net, packet->ptr(), packet->length()); + } + ... + } + +7. How slave SQL thread processes Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The slave processes each recieved event by "applying" it, i.e. by +calling the Log_event::apply_event() function which in turn calls +the virtual do_apply_event() member specific for each type of the +event. + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev = next_event(rli); + ... + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + + int apply_event_and_update_pos(Log_event *ev, ...) + { ... + ev->apply_event(...); + ... + } + + int Log_event::apply_event(...) + { + return do_apply_event(...); + } + +What does it mean to "apply" an Annotate_rows event? It means to set current +thd query to that of the described by the event, i.e. to the query which +caused the subsequent Rows events (see "How Master writes Annotate_rows +events to the binary log" to follow what happens further when the subsequent +Rows events is applied): + + int Annotate_rows_log_event::do_apply_event(...) + { + thd->set_query(m_query_txt, m_query_len); + } + +NOTE. I am not sure, but possibly current values of thd->query and +thd->query_length should be saved before calling set_query() and to be +restored on the Annotate_rows_log_event object deletion. +Is it really needed ? + +After calling this do_apply_event() function we may not delete the +Annotate_rows_log_event object immediatedly (see exec_relay_log_event() +above) because thd->query now points to the string inside this object. +We may keep the pointer to this object in the Relay_log_info: + + class Relay_log_info + { + public: + ... + void set_annotate_event(Annotate_rows_log_event*); + Annotate_rows_log_event* get_annotate_event(); + void free_annotate_event(); + ... + private: + Annotate_rows_log_event* m_annotate_event; + }; + +When the saved Annotate_rows object may be deleted? When all corresponding +Rows events will be processed, i.e. before processing the first non-Rows +event (note that Annotate_rows object resides in the binary log *after* +the (possible) 'BEGIN' Query event which accompanies the rows events; note +also that this deletion is adjusted with the case when some or all +corresponding Rows events are filtered out by replicate filter rules): + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev= next_event(rli); + ... + if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) + rli->free_annotate_event(); + + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + rli->set_annotate_event((Annotate_rows_log_event*) ev); + else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + +where + + #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ + (type) == WRITE_ROWS_EVENT || \ + (type) == UPDATE_ROWS_EVENT || \ + (type) == DELETE_ROWS_EVENT ) + +8. General remarks +~~~~~~~~~~~~~~~~~~ +Kristian noticed that introducing new log event type should be coordinated +somehow with MySQL/Sun: + + Kristian: The numeric code for this event must be assigned carefully. + It should be coordinated with MySQL/Sun, otherwise we can get into a + situation where MySQL uses the same numeric code for one event that + MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two + impossible. + Alex: I reserved about 20 numbers not to have possible conflicts + with MySQL. + Kristian: Still, I think it would be appropriate to send a polite email + to internals(a)lists.mysql.com about this and suggesting to reserve the + event number. + +Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT +flag taking into account that MySQL/Sun may also introduce a flag with the +same value to be used in the request_dump-mysql_binlog_send interface. +But this is mainly the question of merging: if a conflict concerning this +flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value +(this does not require additional changes in the code). ------------------------------------------------------------ -=-=(View All Progress Notes, 23 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Serg): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 05 Feb '10

05 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen, Serg CATEGORY.......: Server-Sprint TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 20 ESTIMATE.......: 35 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Serg - Fri, 05 Feb 2010, 14:04)=-=- Observers changed: Knielsen,Serg -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Category updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Status updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +In-Progress -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 -=-=(Alexi - Sat, 19 Dec 2009, 15:58)=-=- Low Level Design modified. --- /tmp/wklog.47.old.15437 2009-12-19 13:58:12.000000000 +0000 +++ /tmp/wklog.47.new.15437 2009-12-19 13:58:12.000000000 +0000 @@ -1 +1,337 @@ +Content +~~~~~~~ + 1. Annotate_rows event number + 2. Outline of Annotate_rows event behavior + 3. How Master writes Annotate_rows events to the binary log + 4. How slave treats replicate-annotate-rows-events option + 5. How slave IO thread requests Annotate_rows events + 6. How master executes the request + 7. How slave SQL thread processes Annotate_rows events + 8. General remarks + +1. Annotate_rows event number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To avoid possible event numbers conflict with MySQL/Sun, we leave a gap +between the last MySQL event number and the Annotate_rows event number: + + enum Log_event_type + { ... + INCIDENT_EVENT= 26, + // New MySQL event numbers are to be added here + MYSQL_EVENTS_END, + + MARIA_EVENTS_BEGIN= 51, + // New Maria event numbers start from here + ANNOTATE_ROWS_EVENT= 51, + + ENUM_END_EVENT + }; + +together with the corresponding extension of 'post_header_len' array in the +Format description event. (This extension does not affect the compatibility +of the binary log). Here is how Format description event looks like with +this extension: + + ************************ + FORMAT_DESCRIPTION_EVENT + ************************ + 00000004 | A1 A0 2C 4B | time_when = 1261215905 + 00000008 | 0F | event_type = 15 + 00000009 | 64 00 00 00 | server_id = 100 + 0000000D | 7F 00 00 00 | event_len = 127 + 00000011 | 83 00 00 00 | log_pos = 00000083 + 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F + ------------------------ + 00000017 | 04 00 | binlog_ver = 4 + 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log + ..... ... + 0000004B | A1 A0 2C 4B | time_created = 1261215905 + 0000004F | 13 | common_header_len = 19 + ------------------------ + post_header_len + ------------------------ + 00000050 | 38 | 56 - START_EVENT_V3 [1] + ..... ... + 00000069 | 02 | 2 - INCIDENT_EVENT [26] + 0000006A | 00 | 0 - RESERVED [27] + ..... ... + 00000081 | 00 | 0 - RESERVED [50] + 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + ************************ + +2. Outline of Annotate_rows event behavior +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Each Annotate_rows_log_event object has two private members describing the +corresponding query: + + char *m_query_txt; + uint m_query_len; + +When the object is created for writing to a binary log, this query is taken +from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix +as well as other implementation details): + + Annotate_rows_log_event(THD *thd) + { + m_query_txt = thd->query(); + m_query_len = thd->query_length(); + } + +When the object is read from a binary log, the query is taken from the buffer +containing the binary log representation of the event (this buffer is allocated +in Log_event object from which all Log events are derived): + + Annotate_rows_log_event(char *buf, uint event_len, + Format_description_log_event *desc) + { + m_query_len = event_len - desc->common_header_len; + m_query_txt = buf + desc->common_header_len; + } + +The events are written to the binary log by the Log_event::write() member +which calls virtual write_data_header() and write_data_body() members +("data header" and "post header" are synonym in replication terminology). +In our case, data header is empty and data body is just the query: + + bool write_data_body(IO_CACHE *file) + { + return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); + } + +Printing the event is just printing the query: + + void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) + { + my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); + } + +3. How Master writes Annotate_rows events to the binary log +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The event is written to the binary log just before the group of Table_map +events which precede corresponding Rows events (one query may generate +several Table map events in the binary log, but the corresponding +Annotate_rows event must be written only once before the first Table map +event; hence the boolean variable 'with_annotate' below): + + int write_locked_table_maps(THD *thd) + { ... + bool with_annotate= thd->variables.binlog_annotate_rows_events; + ... + for (uint i= 0; i < ... <number of tables> ...; ++i) + { ... + thd->binlog_write_table_map(table, ..., with_annotate); + with_annotate= 0; // write Annotate_event not more than once + ... + } + ... + } + + int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) + { ... + Table_map_log_event the_event(...); + ... + if (with_annotate) + { + Annotate_rows_log_event anno(this); + mysql_bin_log.write(&anno); + } + + mysql_bin_log.write(&the_event); + ... + } + +4. How slave treats replicate-annotate-rows-events option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The replicate-annotate-rows-events option is treated just as the session +value of the binlog_annotate_rows_events variable for the slave IO and +SQL threads. This setting is done during initialization of these threads: + + pthread_handler_t handle_slave_io(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_IO); + ... + } + + pthread_handler_t handle_slave_sql(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_SQL); + ... + } + + int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) + { ... + thd->variables.binlog_annotate_rows_events= + opt_replicate_annotate_rows_events; + ... + } + +5. How slave IO thread requests Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When requesting an event, the slave should inform the master whether +it should send Annotate_rows events or not. To that end we add a new +BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: + + #define BINLOG_DUMP_NON_BLOCK 1 + #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 + + pthread_handler_t handle_slave_io(void *arg) + { ... + request_dump(mysql, ...); + ... + } + + int request_dump(MYSQL* mysql, ...) + { ... + if (opt_log_slave_updates && + mi->io_thd->variables.binlog_annotate_rows_events) + binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; + ... + int2store(buf + 4, binlog_flags); + ... + simple_command(mysql, COM_BINLOG_DUMP, buf, ...); + ... + } + +6. How master executes the request +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + case COM_BINLOG_DUMP: + { ... + flags= uint2korr(packet + 4); + ... + mysql_binlog_send(thd, ..., flags); + ... + } + + void mysql_binlog_send(THD* thd, ..., ushort flags) + { ... + Log_event::read_log_event(&log, packet, ...); + ... + if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || + thd->server_id == 0 /* slave == mysqlbinlog */ ) + { + my_net_write(net, packet->ptr(), packet->length()); + } + ... + } + +7. How slave SQL thread processes Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The slave processes each recieved event by "applying" it, i.e. by +calling the Log_event::apply_event() function which in turn calls +the virtual do_apply_event() member specific for each type of the +event. + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev = next_event(rli); + ... + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + + int apply_event_and_update_pos(Log_event *ev, ...) + { ... + ev->apply_event(...); + ... + } + + int Log_event::apply_event(...) + { + return do_apply_event(...); + } + +What does it mean to "apply" an Annotate_rows event? It means to set current +thd query to that of the described by the event, i.e. to the query which +caused the subsequent Rows events (see "How Master writes Annotate_rows +events to the binary log" to follow what happens further when the subsequent +Rows events is applied): + + int Annotate_rows_log_event::do_apply_event(...) + { + thd->set_query(m_query_txt, m_query_len); + } + +NOTE. I am not sure, but possibly current values of thd->query and +thd->query_length should be saved before calling set_query() and to be +restored on the Annotate_rows_log_event object deletion. +Is it really needed ? + +After calling this do_apply_event() function we may not delete the +Annotate_rows_log_event object immediatedly (see exec_relay_log_event() +above) because thd->query now points to the string inside this object. +We may keep the pointer to this object in the Relay_log_info: + + class Relay_log_info + { + public: + ... + void set_annotate_event(Annotate_rows_log_event*); + Annotate_rows_log_event* get_annotate_event(); + void free_annotate_event(); + ... + private: + Annotate_rows_log_event* m_annotate_event; + }; + +When the saved Annotate_rows object may be deleted? When all corresponding +Rows events will be processed, i.e. before processing the first non-Rows +event (note that Annotate_rows object resides in the binary log *after* +the (possible) 'BEGIN' Query event which accompanies the rows events; note +also that this deletion is adjusted with the case when some or all +corresponding Rows events are filtered out by replicate filter rules): + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev= next_event(rli); + ... + if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) + rli->free_annotate_event(); + + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + rli->set_annotate_event((Annotate_rows_log_event*) ev); + else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + +where + + #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ + (type) == WRITE_ROWS_EVENT || \ + (type) == UPDATE_ROWS_EVENT || \ + (type) == DELETE_ROWS_EVENT ) + +8. General remarks +~~~~~~~~~~~~~~~~~~ +Kristian noticed that introducing new log event type should be coordinated +somehow with MySQL/Sun: + + Kristian: The numeric code for this event must be assigned carefully. + It should be coordinated with MySQL/Sun, otherwise we can get into a + situation where MySQL uses the same numeric code for one event that + MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two + impossible. + Alex: I reserved about 20 numbers not to have possible conflicts + with MySQL. + Kristian: Still, I think it would be appropriate to send a polite email + to internals(a)lists.mysql.com about this and suggesting to reserve the + event number. + +Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT +flag taking into account that MySQL/Sun may also introduce a flag with the +same value to be used in the request_dump-mysql_binlog_send interface. +But this is mainly the question of merging: if a conflict concerning this +flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value +(this does not require additional changes in the code). ------------------------------------------------------------ -=-=(View All Progress Notes, 23 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Guest): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 05 Feb '10

05 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen CATEGORY.......: Server-Sprint TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 20 ESTIMATE.......: 35 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Category updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Status updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +In-Progress -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 -=-=(Alexi - Sat, 19 Dec 2009, 15:58)=-=- Low Level Design modified. --- /tmp/wklog.47.old.15437 2009-12-19 13:58:12.000000000 +0000 +++ /tmp/wklog.47.new.15437 2009-12-19 13:58:12.000000000 +0000 @@ -1 +1,337 @@ +Content +~~~~~~~ + 1. Annotate_rows event number + 2. Outline of Annotate_rows event behavior + 3. How Master writes Annotate_rows events to the binary log + 4. How slave treats replicate-annotate-rows-events option + 5. How slave IO thread requests Annotate_rows events + 6. How master executes the request + 7. How slave SQL thread processes Annotate_rows events + 8. General remarks + +1. Annotate_rows event number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To avoid possible event numbers conflict with MySQL/Sun, we leave a gap +between the last MySQL event number and the Annotate_rows event number: + + enum Log_event_type + { ... + INCIDENT_EVENT= 26, + // New MySQL event numbers are to be added here + MYSQL_EVENTS_END, + + MARIA_EVENTS_BEGIN= 51, + // New Maria event numbers start from here + ANNOTATE_ROWS_EVENT= 51, + + ENUM_END_EVENT + }; + +together with the corresponding extension of 'post_header_len' array in the +Format description event. (This extension does not affect the compatibility +of the binary log). Here is how Format description event looks like with +this extension: + + ************************ + FORMAT_DESCRIPTION_EVENT + ************************ + 00000004 | A1 A0 2C 4B | time_when = 1261215905 + 00000008 | 0F | event_type = 15 + 00000009 | 64 00 00 00 | server_id = 100 + 0000000D | 7F 00 00 00 | event_len = 127 + 00000011 | 83 00 00 00 | log_pos = 00000083 + 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F + ------------------------ + 00000017 | 04 00 | binlog_ver = 4 + 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log + ..... ... + 0000004B | A1 A0 2C 4B | time_created = 1261215905 + 0000004F | 13 | common_header_len = 19 + ------------------------ + post_header_len + ------------------------ + 00000050 | 38 | 56 - START_EVENT_V3 [1] + ..... ... + 00000069 | 02 | 2 - INCIDENT_EVENT [26] + 0000006A | 00 | 0 - RESERVED [27] + ..... ... + 00000081 | 00 | 0 - RESERVED [50] + 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + ************************ + +2. Outline of Annotate_rows event behavior +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Each Annotate_rows_log_event object has two private members describing the +corresponding query: + + char *m_query_txt; + uint m_query_len; + +When the object is created for writing to a binary log, this query is taken +from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix +as well as other implementation details): + + Annotate_rows_log_event(THD *thd) + { + m_query_txt = thd->query(); + m_query_len = thd->query_length(); + } + +When the object is read from a binary log, the query is taken from the buffer +containing the binary log representation of the event (this buffer is allocated +in Log_event object from which all Log events are derived): + + Annotate_rows_log_event(char *buf, uint event_len, + Format_description_log_event *desc) + { + m_query_len = event_len - desc->common_header_len; + m_query_txt = buf + desc->common_header_len; + } + +The events are written to the binary log by the Log_event::write() member +which calls virtual write_data_header() and write_data_body() members +("data header" and "post header" are synonym in replication terminology). +In our case, data header is empty and data body is just the query: + + bool write_data_body(IO_CACHE *file) + { + return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); + } + +Printing the event is just printing the query: + + void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) + { + my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); + } + +3. How Master writes Annotate_rows events to the binary log +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The event is written to the binary log just before the group of Table_map +events which precede corresponding Rows events (one query may generate +several Table map events in the binary log, but the corresponding +Annotate_rows event must be written only once before the first Table map +event; hence the boolean variable 'with_annotate' below): + + int write_locked_table_maps(THD *thd) + { ... + bool with_annotate= thd->variables.binlog_annotate_rows_events; + ... + for (uint i= 0; i < ... <number of tables> ...; ++i) + { ... + thd->binlog_write_table_map(table, ..., with_annotate); + with_annotate= 0; // write Annotate_event not more than once + ... + } + ... + } + + int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) + { ... + Table_map_log_event the_event(...); + ... + if (with_annotate) + { + Annotate_rows_log_event anno(this); + mysql_bin_log.write(&anno); + } + + mysql_bin_log.write(&the_event); + ... + } + +4. How slave treats replicate-annotate-rows-events option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The replicate-annotate-rows-events option is treated just as the session +value of the binlog_annotate_rows_events variable for the slave IO and +SQL threads. This setting is done during initialization of these threads: + + pthread_handler_t handle_slave_io(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_IO); + ... + } + + pthread_handler_t handle_slave_sql(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_SQL); + ... + } + + int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) + { ... + thd->variables.binlog_annotate_rows_events= + opt_replicate_annotate_rows_events; + ... + } + +5. How slave IO thread requests Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When requesting an event, the slave should inform the master whether +it should send Annotate_rows events or not. To that end we add a new +BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: + + #define BINLOG_DUMP_NON_BLOCK 1 + #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 + + pthread_handler_t handle_slave_io(void *arg) + { ... + request_dump(mysql, ...); + ... + } + + int request_dump(MYSQL* mysql, ...) + { ... + if (opt_log_slave_updates && + mi->io_thd->variables.binlog_annotate_rows_events) + binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; + ... + int2store(buf + 4, binlog_flags); + ... + simple_command(mysql, COM_BINLOG_DUMP, buf, ...); + ... + } + +6. How master executes the request +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + case COM_BINLOG_DUMP: + { ... + flags= uint2korr(packet + 4); + ... + mysql_binlog_send(thd, ..., flags); + ... + } + + void mysql_binlog_send(THD* thd, ..., ushort flags) + { ... + Log_event::read_log_event(&log, packet, ...); + ... + if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || + thd->server_id == 0 /* slave == mysqlbinlog */ ) + { + my_net_write(net, packet->ptr(), packet->length()); + } + ... + } + +7. How slave SQL thread processes Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The slave processes each recieved event by "applying" it, i.e. by +calling the Log_event::apply_event() function which in turn calls +the virtual do_apply_event() member specific for each type of the +event. + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev = next_event(rli); + ... + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + + int apply_event_and_update_pos(Log_event *ev, ...) + { ... + ev->apply_event(...); + ... + } + + int Log_event::apply_event(...) + { + return do_apply_event(...); + } + +What does it mean to "apply" an Annotate_rows event? It means to set current +thd query to that of the described by the event, i.e. to the query which +caused the subsequent Rows events (see "How Master writes Annotate_rows +events to the binary log" to follow what happens further when the subsequent +Rows events is applied): + + int Annotate_rows_log_event::do_apply_event(...) + { + thd->set_query(m_query_txt, m_query_len); + } + +NOTE. I am not sure, but possibly current values of thd->query and +thd->query_length should be saved before calling set_query() and to be +restored on the Annotate_rows_log_event object deletion. +Is it really needed ? + +After calling this do_apply_event() function we may not delete the +Annotate_rows_log_event object immediatedly (see exec_relay_log_event() +above) because thd->query now points to the string inside this object. +We may keep the pointer to this object in the Relay_log_info: + + class Relay_log_info + { + public: + ... + void set_annotate_event(Annotate_rows_log_event*); + Annotate_rows_log_event* get_annotate_event(); + void free_annotate_event(); + ... + private: + Annotate_rows_log_event* m_annotate_event; + }; + +When the saved Annotate_rows object may be deleted? When all corresponding +Rows events will be processed, i.e. before processing the first non-Rows +event (note that Annotate_rows object resides in the binary log *after* +the (possible) 'BEGIN' Query event which accompanies the rows events; note +also that this deletion is adjusted with the case when some or all +corresponding Rows events are filtered out by replicate filter rules): + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev= next_event(rli); + ... + if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) + rli->free_annotate_event(); + + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + rli->set_annotate_event((Annotate_rows_log_event*) ev); + else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + +where + + #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ + (type) == WRITE_ROWS_EVENT || \ + (type) == UPDATE_ROWS_EVENT || \ + (type) == DELETE_ROWS_EVENT ) + +8. General remarks +~~~~~~~~~~~~~~~~~~ +Kristian noticed that introducing new log event type should be coordinated +somehow with MySQL/Sun: + + Kristian: The numeric code for this event must be assigned carefully. + It should be coordinated with MySQL/Sun, otherwise we can get into a + situation where MySQL uses the same numeric code for one event that + MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two + impossible. + Alex: I reserved about 20 numbers not to have possible conflicts + with MySQL. + Kristian: Still, I think it would be appropriate to send a polite email + to internals(a)lists.mysql.com about this and suggesting to reserve the + event number. + +Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT +flag taking into account that MySQL/Sun may also introduce a flag with the +same value to be used in the request_dump-mysql_binlog_send interface. +But this is mainly the question of merging: if a conflict concerning this +flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value +(this does not require additional changes in the code). -=-=(Alexi - Sat, 19 Dec 2009, 15:41)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.14545 2009-12-19 15:41:21.000000000 +0200 +++ /tmp/wklog.47.new.14545 2009-12-19 15:41:21.000000000 +0200 @@ -1,122 +1,107 @@ -First suggestion: - -> I think for this we would actually need a new binlog event type -> (Comment_log_event?). Unless we want to log an empty statement Query_log_event -> containing only a comment (a bit of a hack). - -New server option -~~~~~~~~~~~~~~~~~ - --binlog-annotate-rows-events - -Setting this option makes RBR (rows-) events in the binary log to be -preceded by Annotate rows events (see below). The corresponding -'binlog_annotate_rows_events' system variable is dynamic and has both -global and session values. Default global value is OFF. - -Note. Session values are usefull to make it possible to annotate only - some selected statements: +Content +~~~~~~~ + 1. Annotate_rows_log_event + 2. Server option: --binlog-annotate-rows-events + 3. Server option: --replicate-annotate-rows-events + 4. mysqlbinlog option: --print-annotate-rows-events + 5. mysqlbinlog output + +1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Describes the query which caused the corresponding rows events. Has empty +post-header and contains the query text in its data part. Example: + + ************************ + ANNOTATE_RBR_EVENT + ************************ + 00000220 | B6 A0 2C 4B | time_when = 1261215926 + 00000224 | 33 | event_type = 51 + 00000225 | 64 00 00 00 | server_id = 100 + 00000229 | 36 00 00 00 | event_len = 54 + 0000022D | 56 02 00 00 | log_pos = 00000256 + 00000231 | 00 00 | flags = <none> + ------------------------ + 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" + 00000237 | 52 54 20 49 | + 0000023B | 4E 54 4F 20 | + 0000023F | 74 31 20 56 | + 00000243 | 41 4C 55 45 | + 00000247 | 53 20 28 31 | + 0000024B | 29 2C 20 28 | + 0000024F | 32 29 2C 20 | + 00000253 | 28 33 29 | + ************************ + +In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event +and precedes the first of Table map events which accompany the corresponding +rows events. (See example in the "mysqlbinlog output" section below.) + +2. Server option: --binlog-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Tells the master to write Annotate_rows events to the binary log. + + * Variable Name: binlog_annotate_rows_events + * Scope: Global & Session + * Access Type: Dynamic + * Data Type: bool + * Default Value: OFF +NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... -New binlog event type -~~~~~~~~~~~~~~~~~~~~~ - Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] - -Describes the query which caused the corresponding rows event. In binary log, -precedes each Table_map_log_event. Contains empty post-header and the query -text in its data part. - -The numeric code for this event must be assigned carefully. It should be -coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL -uses the same numeric code for one event that MariaDB uses for -ANNOTATE_ROWS_EVENT, which would make merging the two impossible. - -Example: - - ... - ************************ - ANNOTATE_ROWS_EVENT [51] - ************************ - 000000C7 | 54 1B 12 4B | time_when = 1259477844 - 000000CB | 33 | event_type = 51 - 000000CC | 64 00 00 00 | server_id = 100 - 000000D0 | 2C 00 00 00 | event_len = 44 - 000000D4 | F3 00 00 00 | log_pos = 000000F3 - 000000D8 | 00 00 | flags = <none> - ------------------------ - 000000DA | 69 6E 73 65 | query = "insert into t1 values (1)" - 000000DE | 72 74 20 69 | - 000000E2 | 6E 74 6F 20 | - 000000E6 | 74 31 20 76 | - 000000EA | 61 6C 75 65 | - 000000EE | 73 20 28 31 | - 000000F2 | 29 | - ************************ - TABLE_MAP_EVENT [19] - ************************ - 000000F3 | 54 1B 12 4B | time_when = 1259477844 - 000000F7 | 13 | event_type = 19 - 000000F8 | 64 00 00 00 | server_id = 100 - 000000FC | 29 00 00 00 | event_len = 41 - 00000100 | 1C 01 00 00 | log_pos = 0000011C - 00000104 | 00 00 | flags = <none> - ------------------------ - ... - ************************ - WRITE_ROWS_EVENT [23] - ************************ - 0000011C | 54 1B 12 4B | time_when = 1259477844 - 00000120 | 17 | event_type = 23 - 00000121 | 64 00 00 00 | server_id = 100 - 00000125 | 22 00 00 00 | event_len = 34 - 00000129 | 3E 01 00 00 | log_pos = 0000013E - 0000012D | 10 00 | flags = LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F - ------------------------ - 0000012F | 0F 00 00 00 | table_id = 15 - ... - -New mysqlbinlog option -~~~~~~~~~~~~~~~~~~~~~~ - --print-annotate-rows-events - -With this option, mysqlbinlog prints the content of Annotate-rows -events (if the binary log does contain them). Without this option -(i.e. by default), mysqlbinlog skips Annotate rows events. - - -mysqlbinlog output -~~~~~~~~~~~~~~~~~~ -Something like this: +3. Server option: --replicate-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Tells the slave to reproduce Annotate_rows events recieved from the master +in its own binary log (sensible only in pair with log-slave-updates option). + + * Variable Name: replicate_annotate_rows_events + * Scope: Global + * Access Type: Read only + * Data Type: bool + * Default Value: OFF + +NOTE. Why do we additionally need this 'replicate' option? Why not to make +the slave to reproduce this events when its binlog-annotate-rows-events +global value is ON? Well, because, for example, we may want to configure +the slave which should reproduce Annotate_rows events but has global +binlog-annotate-rows-events = OFF meaning this to be the default value for +the client threads (see also "How slave treats replicate-annotate-rows-events +option" in LLD part). + +4. mysqlbinlog option: --print-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +With this option, mysqlbinlog prints the content of Annotate_rows events (if +the binary log does contain them). Without this option (i.e. by default), +mysqlbinlog skips Annotate_rows events. +5. mysqlbinlog output +~~~~~~~~~~~~~~~~~~~~~ +With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events +in a form like this: ... - # at 199 - # at 243 - # at 284 - #091129 9:57:24 server id 100 end_log_pos 243 Query: `insert into t1 values -(1)` - #091129 9:57:24 server id 100 end_log_pos 284 Table_map: `test`.`t1` mapped -to number 15 - #091129 9:57:24 server id 100 end_log_pos 318 Write_rows: table id 15 + # at 1646 + #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 +exec_time=0 error_code=0 + SET TIMESTAMP=1261215926/*!*/; + BEGIN + /*!*/; + # at 1714 + # at 1812 + # at 1853 + # at 1894 + # at 1938 + #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM +t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` + #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` +mapped to number 16 + #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` +mapped to number 17 + #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 + #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F - - BINLOG ' - VBsSSzNkAAAALAAAAPMAAAAAAGluc2VydCBpbnRvIHQxIHZhbHVlcyAoMSk= - VBsSSxNkAAAAKQAAABwBAAAAAA8AAAAAAAAABHRlc3QAAnQxAAEDAAE= - VBsSSxdkAAAAIgAAAD4BAAAQAA8AAAAAAAEAAf/+AQAAAA== - '/*!*/; - ### INSERT INTO test.t1 - ### SET - ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ... -When master sends Annotate rows events -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -1. Master always sends Annotate_rows events to mysqlbinlog (in - remote case). -2. Master sends Annotate_rows events to a slave only if the slave has - both log-slave-updates and binlog-annotate-rows-events options set. - ------------------------------------------------------------ -=-=(View All Progress Notes, 22 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Guest): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 05 Feb '10

05 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen CATEGORY.......: Server-Sprint TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 20 ESTIMATE.......: 35 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Category updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Status updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +In-Progress -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 -=-=(Alexi - Sat, 19 Dec 2009, 15:58)=-=- Low Level Design modified. --- /tmp/wklog.47.old.15437 2009-12-19 13:58:12.000000000 +0000 +++ /tmp/wklog.47.new.15437 2009-12-19 13:58:12.000000000 +0000 @@ -1 +1,337 @@ +Content +~~~~~~~ + 1. Annotate_rows event number + 2. Outline of Annotate_rows event behavior + 3. How Master writes Annotate_rows events to the binary log + 4. How slave treats replicate-annotate-rows-events option + 5. How slave IO thread requests Annotate_rows events + 6. How master executes the request + 7. How slave SQL thread processes Annotate_rows events + 8. General remarks + +1. Annotate_rows event number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To avoid possible event numbers conflict with MySQL/Sun, we leave a gap +between the last MySQL event number and the Annotate_rows event number: + + enum Log_event_type + { ... + INCIDENT_EVENT= 26, + // New MySQL event numbers are to be added here + MYSQL_EVENTS_END, + + MARIA_EVENTS_BEGIN= 51, + // New Maria event numbers start from here + ANNOTATE_ROWS_EVENT= 51, + + ENUM_END_EVENT + }; + +together with the corresponding extension of 'post_header_len' array in the +Format description event. (This extension does not affect the compatibility +of the binary log). Here is how Format description event looks like with +this extension: + + ************************ + FORMAT_DESCRIPTION_EVENT + ************************ + 00000004 | A1 A0 2C 4B | time_when = 1261215905 + 00000008 | 0F | event_type = 15 + 00000009 | 64 00 00 00 | server_id = 100 + 0000000D | 7F 00 00 00 | event_len = 127 + 00000011 | 83 00 00 00 | log_pos = 00000083 + 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F + ------------------------ + 00000017 | 04 00 | binlog_ver = 4 + 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log + ..... ... + 0000004B | A1 A0 2C 4B | time_created = 1261215905 + 0000004F | 13 | common_header_len = 19 + ------------------------ + post_header_len + ------------------------ + 00000050 | 38 | 56 - START_EVENT_V3 [1] + ..... ... + 00000069 | 02 | 2 - INCIDENT_EVENT [26] + 0000006A | 00 | 0 - RESERVED [27] + ..... ... + 00000081 | 00 | 0 - RESERVED [50] + 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + ************************ + +2. Outline of Annotate_rows event behavior +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Each Annotate_rows_log_event object has two private members describing the +corresponding query: + + char *m_query_txt; + uint m_query_len; + +When the object is created for writing to a binary log, this query is taken +from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix +as well as other implementation details): + + Annotate_rows_log_event(THD *thd) + { + m_query_txt = thd->query(); + m_query_len = thd->query_length(); + } + +When the object is read from a binary log, the query is taken from the buffer +containing the binary log representation of the event (this buffer is allocated +in Log_event object from which all Log events are derived): + + Annotate_rows_log_event(char *buf, uint event_len, + Format_description_log_event *desc) + { + m_query_len = event_len - desc->common_header_len; + m_query_txt = buf + desc->common_header_len; + } + +The events are written to the binary log by the Log_event::write() member +which calls virtual write_data_header() and write_data_body() members +("data header" and "post header" are synonym in replication terminology). +In our case, data header is empty and data body is just the query: + + bool write_data_body(IO_CACHE *file) + { + return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); + } + +Printing the event is just printing the query: + + void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) + { + my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); + } + +3. How Master writes Annotate_rows events to the binary log +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The event is written to the binary log just before the group of Table_map +events which precede corresponding Rows events (one query may generate +several Table map events in the binary log, but the corresponding +Annotate_rows event must be written only once before the first Table map +event; hence the boolean variable 'with_annotate' below): + + int write_locked_table_maps(THD *thd) + { ... + bool with_annotate= thd->variables.binlog_annotate_rows_events; + ... + for (uint i= 0; i < ... <number of tables> ...; ++i) + { ... + thd->binlog_write_table_map(table, ..., with_annotate); + with_annotate= 0; // write Annotate_event not more than once + ... + } + ... + } + + int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) + { ... + Table_map_log_event the_event(...); + ... + if (with_annotate) + { + Annotate_rows_log_event anno(this); + mysql_bin_log.write(&anno); + } + + mysql_bin_log.write(&the_event); + ... + } + +4. How slave treats replicate-annotate-rows-events option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The replicate-annotate-rows-events option is treated just as the session +value of the binlog_annotate_rows_events variable for the slave IO and +SQL threads. This setting is done during initialization of these threads: + + pthread_handler_t handle_slave_io(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_IO); + ... + } + + pthread_handler_t handle_slave_sql(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_SQL); + ... + } + + int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) + { ... + thd->variables.binlog_annotate_rows_events= + opt_replicate_annotate_rows_events; + ... + } + +5. How slave IO thread requests Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When requesting an event, the slave should inform the master whether +it should send Annotate_rows events or not. To that end we add a new +BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: + + #define BINLOG_DUMP_NON_BLOCK 1 + #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 + + pthread_handler_t handle_slave_io(void *arg) + { ... + request_dump(mysql, ...); + ... + } + + int request_dump(MYSQL* mysql, ...) + { ... + if (opt_log_slave_updates && + mi->io_thd->variables.binlog_annotate_rows_events) + binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; + ... + int2store(buf + 4, binlog_flags); + ... + simple_command(mysql, COM_BINLOG_DUMP, buf, ...); + ... + } + +6. How master executes the request +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + case COM_BINLOG_DUMP: + { ... + flags= uint2korr(packet + 4); + ... + mysql_binlog_send(thd, ..., flags); + ... + } + + void mysql_binlog_send(THD* thd, ..., ushort flags) + { ... + Log_event::read_log_event(&log, packet, ...); + ... + if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || + thd->server_id == 0 /* slave == mysqlbinlog */ ) + { + my_net_write(net, packet->ptr(), packet->length()); + } + ... + } + +7. How slave SQL thread processes Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The slave processes each recieved event by "applying" it, i.e. by +calling the Log_event::apply_event() function which in turn calls +the virtual do_apply_event() member specific for each type of the +event. + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev = next_event(rli); + ... + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + + int apply_event_and_update_pos(Log_event *ev, ...) + { ... + ev->apply_event(...); + ... + } + + int Log_event::apply_event(...) + { + return do_apply_event(...); + } + +What does it mean to "apply" an Annotate_rows event? It means to set current +thd query to that of the described by the event, i.e. to the query which +caused the subsequent Rows events (see "How Master writes Annotate_rows +events to the binary log" to follow what happens further when the subsequent +Rows events is applied): + + int Annotate_rows_log_event::do_apply_event(...) + { + thd->set_query(m_query_txt, m_query_len); + } + +NOTE. I am not sure, but possibly current values of thd->query and +thd->query_length should be saved before calling set_query() and to be +restored on the Annotate_rows_log_event object deletion. +Is it really needed ? + +After calling this do_apply_event() function we may not delete the +Annotate_rows_log_event object immediatedly (see exec_relay_log_event() +above) because thd->query now points to the string inside this object. +We may keep the pointer to this object in the Relay_log_info: + + class Relay_log_info + { + public: + ... + void set_annotate_event(Annotate_rows_log_event*); + Annotate_rows_log_event* get_annotate_event(); + void free_annotate_event(); + ... + private: + Annotate_rows_log_event* m_annotate_event; + }; + +When the saved Annotate_rows object may be deleted? When all corresponding +Rows events will be processed, i.e. before processing the first non-Rows +event (note that Annotate_rows object resides in the binary log *after* +the (possible) 'BEGIN' Query event which accompanies the rows events; note +also that this deletion is adjusted with the case when some or all +corresponding Rows events are filtered out by replicate filter rules): + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev= next_event(rli); + ... + if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) + rli->free_annotate_event(); + + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + rli->set_annotate_event((Annotate_rows_log_event*) ev); + else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + +where + + #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ + (type) == WRITE_ROWS_EVENT || \ + (type) == UPDATE_ROWS_EVENT || \ + (type) == DELETE_ROWS_EVENT ) + +8. General remarks +~~~~~~~~~~~~~~~~~~ +Kristian noticed that introducing new log event type should be coordinated +somehow with MySQL/Sun: + + Kristian: The numeric code for this event must be assigned carefully. + It should be coordinated with MySQL/Sun, otherwise we can get into a + situation where MySQL uses the same numeric code for one event that + MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two + impossible. + Alex: I reserved about 20 numbers not to have possible conflicts + with MySQL. + Kristian: Still, I think it would be appropriate to send a polite email + to internals(a)lists.mysql.com about this and suggesting to reserve the + event number. + +Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT +flag taking into account that MySQL/Sun may also introduce a flag with the +same value to be used in the request_dump-mysql_binlog_send interface. +But this is mainly the question of merging: if a conflict concerning this +flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value +(this does not require additional changes in the code). -=-=(Alexi - Sat, 19 Dec 2009, 15:41)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.14545 2009-12-19 15:41:21.000000000 +0200 +++ /tmp/wklog.47.new.14545 2009-12-19 15:41:21.000000000 +0200 @@ -1,122 +1,107 @@ -First suggestion: - -> I think for this we would actually need a new binlog event type -> (Comment_log_event?). Unless we want to log an empty statement Query_log_event -> containing only a comment (a bit of a hack). - -New server option -~~~~~~~~~~~~~~~~~ - --binlog-annotate-rows-events - -Setting this option makes RBR (rows-) events in the binary log to be -preceded by Annotate rows events (see below). The corresponding -'binlog_annotate_rows_events' system variable is dynamic and has both -global and session values. Default global value is OFF. - -Note. Session values are usefull to make it possible to annotate only - some selected statements: +Content +~~~~~~~ + 1. Annotate_rows_log_event + 2. Server option: --binlog-annotate-rows-events + 3. Server option: --replicate-annotate-rows-events + 4. mysqlbinlog option: --print-annotate-rows-events + 5. mysqlbinlog output + +1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Describes the query which caused the corresponding rows events. Has empty +post-header and contains the query text in its data part. Example: + + ************************ + ANNOTATE_RBR_EVENT + ************************ + 00000220 | B6 A0 2C 4B | time_when = 1261215926 + 00000224 | 33 | event_type = 51 + 00000225 | 64 00 00 00 | server_id = 100 + 00000229 | 36 00 00 00 | event_len = 54 + 0000022D | 56 02 00 00 | log_pos = 00000256 + 00000231 | 00 00 | flags = <none> + ------------------------ + 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" + 00000237 | 52 54 20 49 | + 0000023B | 4E 54 4F 20 | + 0000023F | 74 31 20 56 | + 00000243 | 41 4C 55 45 | + 00000247 | 53 20 28 31 | + 0000024B | 29 2C 20 28 | + 0000024F | 32 29 2C 20 | + 00000253 | 28 33 29 | + ************************ + +In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event +and precedes the first of Table map events which accompany the corresponding +rows events. (See example in the "mysqlbinlog output" section below.) + +2. Server option: --binlog-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Tells the master to write Annotate_rows events to the binary log. + + * Variable Name: binlog_annotate_rows_events + * Scope: Global & Session + * Access Type: Dynamic + * Data Type: bool + * Default Value: OFF +NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... -New binlog event type -~~~~~~~~~~~~~~~~~~~~~ - Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] - -Describes the query which caused the corresponding rows event. In binary log, -precedes each Table_map_log_event. Contains empty post-header and the query -text in its data part. - -The numeric code for this event must be assigned carefully. It should be -coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL -uses the same numeric code for one event that MariaDB uses for -ANNOTATE_ROWS_EVENT, which would make merging the two impossible. - -Example: - - ... - ************************ - ANNOTATE_ROWS_EVENT [51] - ************************ - 000000C7 | 54 1B 12 4B | time_when = 1259477844 - 000000CB | 33 | event_type = 51 - 000000CC | 64 00 00 00 | server_id = 100 - 000000D0 | 2C 00 00 00 | event_len = 44 - 000000D4 | F3 00 00 00 | log_pos = 000000F3 - 000000D8 | 00 00 | flags = <none> - ------------------------ - 000000DA | 69 6E 73 65 | query = "insert into t1 values (1)" - 000000DE | 72 74 20 69 | - 000000E2 | 6E 74 6F 20 | - 000000E6 | 74 31 20 76 | - 000000EA | 61 6C 75 65 | - 000000EE | 73 20 28 31 | - 000000F2 | 29 | - ************************ - TABLE_MAP_EVENT [19] - ************************ - 000000F3 | 54 1B 12 4B | time_when = 1259477844 - 000000F7 | 13 | event_type = 19 - 000000F8 | 64 00 00 00 | server_id = 100 - 000000FC | 29 00 00 00 | event_len = 41 - 00000100 | 1C 01 00 00 | log_pos = 0000011C - 00000104 | 00 00 | flags = <none> - ------------------------ - ... - ************************ - WRITE_ROWS_EVENT [23] - ************************ - 0000011C | 54 1B 12 4B | time_when = 1259477844 - 00000120 | 17 | event_type = 23 - 00000121 | 64 00 00 00 | server_id = 100 - 00000125 | 22 00 00 00 | event_len = 34 - 00000129 | 3E 01 00 00 | log_pos = 0000013E - 0000012D | 10 00 | flags = LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F - ------------------------ - 0000012F | 0F 00 00 00 | table_id = 15 - ... - -New mysqlbinlog option -~~~~~~~~~~~~~~~~~~~~~~ - --print-annotate-rows-events - -With this option, mysqlbinlog prints the content of Annotate-rows -events (if the binary log does contain them). Without this option -(i.e. by default), mysqlbinlog skips Annotate rows events. - - -mysqlbinlog output -~~~~~~~~~~~~~~~~~~ -Something like this: +3. Server option: --replicate-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Tells the slave to reproduce Annotate_rows events recieved from the master +in its own binary log (sensible only in pair with log-slave-updates option). + + * Variable Name: replicate_annotate_rows_events + * Scope: Global + * Access Type: Read only + * Data Type: bool + * Default Value: OFF + +NOTE. Why do we additionally need this 'replicate' option? Why not to make +the slave to reproduce this events when its binlog-annotate-rows-events +global value is ON? Well, because, for example, we may want to configure +the slave which should reproduce Annotate_rows events but has global +binlog-annotate-rows-events = OFF meaning this to be the default value for +the client threads (see also "How slave treats replicate-annotate-rows-events +option" in LLD part). + +4. mysqlbinlog option: --print-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +With this option, mysqlbinlog prints the content of Annotate_rows events (if +the binary log does contain them). Without this option (i.e. by default), +mysqlbinlog skips Annotate_rows events. +5. mysqlbinlog output +~~~~~~~~~~~~~~~~~~~~~ +With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events +in a form like this: ... - # at 199 - # at 243 - # at 284 - #091129 9:57:24 server id 100 end_log_pos 243 Query: `insert into t1 values -(1)` - #091129 9:57:24 server id 100 end_log_pos 284 Table_map: `test`.`t1` mapped -to number 15 - #091129 9:57:24 server id 100 end_log_pos 318 Write_rows: table id 15 + # at 1646 + #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 +exec_time=0 error_code=0 + SET TIMESTAMP=1261215926/*!*/; + BEGIN + /*!*/; + # at 1714 + # at 1812 + # at 1853 + # at 1894 + # at 1938 + #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM +t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` + #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` +mapped to number 16 + #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` +mapped to number 17 + #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 + #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F - - BINLOG ' - VBsSSzNkAAAALAAAAPMAAAAAAGluc2VydCBpbnRvIHQxIHZhbHVlcyAoMSk= - VBsSSxNkAAAAKQAAABwBAAAAAA8AAAAAAAAABHRlc3QAAnQxAAEDAAE= - VBsSSxdkAAAAIgAAAD4BAAAQAA8AAAAAAAEAAf/+AQAAAA== - '/*!*/; - ### INSERT INTO test.t1 - ### SET - ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ... -When master sends Annotate rows events -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -1. Master always sends Annotate_rows events to mysqlbinlog (in - remote case). -2. Master sends Annotate_rows events to a slave only if the slave has - both log-slave-updates and binlog-annotate-rows-events options set. - ------------------------------------------------------------ -=-=(View All Progress Notes, 22 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Guest): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 05 Feb '10

05 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen CATEGORY.......: Server-Sprint TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: In-Progress PRIORITY.......: 60 WORKED HOURS...: 20 ESTIMATE.......: 35 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Category updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-Sprint -=-=(Guest - Fri, 05 Feb 2010, 13:40)=-=- Status updated. --- /tmp/wklog.47.old.9197 2010-02-05 13:40:36.000000000 +0200 +++ /tmp/wklog.47.new.9197 2010-02-05 13:40:36.000000000 +0200 @@ -1 +1 @@ -Un-Assigned +In-Progress -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 -=-=(Alexi - Sat, 19 Dec 2009, 15:58)=-=- Low Level Design modified. --- /tmp/wklog.47.old.15437 2009-12-19 13:58:12.000000000 +0000 +++ /tmp/wklog.47.new.15437 2009-12-19 13:58:12.000000000 +0000 @@ -1 +1,337 @@ +Content +~~~~~~~ + 1. Annotate_rows event number + 2. Outline of Annotate_rows event behavior + 3. How Master writes Annotate_rows events to the binary log + 4. How slave treats replicate-annotate-rows-events option + 5. How slave IO thread requests Annotate_rows events + 6. How master executes the request + 7. How slave SQL thread processes Annotate_rows events + 8. General remarks + +1. Annotate_rows event number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To avoid possible event numbers conflict with MySQL/Sun, we leave a gap +between the last MySQL event number and the Annotate_rows event number: + + enum Log_event_type + { ... + INCIDENT_EVENT= 26, + // New MySQL event numbers are to be added here + MYSQL_EVENTS_END, + + MARIA_EVENTS_BEGIN= 51, + // New Maria event numbers start from here + ANNOTATE_ROWS_EVENT= 51, + + ENUM_END_EVENT + }; + +together with the corresponding extension of 'post_header_len' array in the +Format description event. (This extension does not affect the compatibility +of the binary log). Here is how Format description event looks like with +this extension: + + ************************ + FORMAT_DESCRIPTION_EVENT + ************************ + 00000004 | A1 A0 2C 4B | time_when = 1261215905 + 00000008 | 0F | event_type = 15 + 00000009 | 64 00 00 00 | server_id = 100 + 0000000D | 7F 00 00 00 | event_len = 127 + 00000011 | 83 00 00 00 | log_pos = 00000083 + 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F + ------------------------ + 00000017 | 04 00 | binlog_ver = 4 + 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log + ..... ... + 0000004B | A1 A0 2C 4B | time_created = 1261215905 + 0000004F | 13 | common_header_len = 19 + ------------------------ + post_header_len + ------------------------ + 00000050 | 38 | 56 - START_EVENT_V3 [1] + ..... ... + 00000069 | 02 | 2 - INCIDENT_EVENT [26] + 0000006A | 00 | 0 - RESERVED [27] + ..... ... + 00000081 | 00 | 0 - RESERVED [50] + 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + ************************ + +2. Outline of Annotate_rows event behavior +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Each Annotate_rows_log_event object has two private members describing the +corresponding query: + + char *m_query_txt; + uint m_query_len; + +When the object is created for writing to a binary log, this query is taken +from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix +as well as other implementation details): + + Annotate_rows_log_event(THD *thd) + { + m_query_txt = thd->query(); + m_query_len = thd->query_length(); + } + +When the object is read from a binary log, the query is taken from the buffer +containing the binary log representation of the event (this buffer is allocated +in Log_event object from which all Log events are derived): + + Annotate_rows_log_event(char *buf, uint event_len, + Format_description_log_event *desc) + { + m_query_len = event_len - desc->common_header_len; + m_query_txt = buf + desc->common_header_len; + } + +The events are written to the binary log by the Log_event::write() member +which calls virtual write_data_header() and write_data_body() members +("data header" and "post header" are synonym in replication terminology). +In our case, data header is empty and data body is just the query: + + bool write_data_body(IO_CACHE *file) + { + return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); + } + +Printing the event is just printing the query: + + void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) + { + my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); + } + +3. How Master writes Annotate_rows events to the binary log +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The event is written to the binary log just before the group of Table_map +events which precede corresponding Rows events (one query may generate +several Table map events in the binary log, but the corresponding +Annotate_rows event must be written only once before the first Table map +event; hence the boolean variable 'with_annotate' below): + + int write_locked_table_maps(THD *thd) + { ... + bool with_annotate= thd->variables.binlog_annotate_rows_events; + ... + for (uint i= 0; i < ... <number of tables> ...; ++i) + { ... + thd->binlog_write_table_map(table, ..., with_annotate); + with_annotate= 0; // write Annotate_event not more than once + ... + } + ... + } + + int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) + { ... + Table_map_log_event the_event(...); + ... + if (with_annotate) + { + Annotate_rows_log_event anno(this); + mysql_bin_log.write(&anno); + } + + mysql_bin_log.write(&the_event); + ... + } + +4. How slave treats replicate-annotate-rows-events option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The replicate-annotate-rows-events option is treated just as the session +value of the binlog_annotate_rows_events variable for the slave IO and +SQL threads. This setting is done during initialization of these threads: + + pthread_handler_t handle_slave_io(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_IO); + ... + } + + pthread_handler_t handle_slave_sql(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_SQL); + ... + } + + int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) + { ... + thd->variables.binlog_annotate_rows_events= + opt_replicate_annotate_rows_events; + ... + } + +5. How slave IO thread requests Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When requesting an event, the slave should inform the master whether +it should send Annotate_rows events or not. To that end we add a new +BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: + + #define BINLOG_DUMP_NON_BLOCK 1 + #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 + + pthread_handler_t handle_slave_io(void *arg) + { ... + request_dump(mysql, ...); + ... + } + + int request_dump(MYSQL* mysql, ...) + { ... + if (opt_log_slave_updates && + mi->io_thd->variables.binlog_annotate_rows_events) + binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; + ... + int2store(buf + 4, binlog_flags); + ... + simple_command(mysql, COM_BINLOG_DUMP, buf, ...); + ... + } + +6. How master executes the request +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + case COM_BINLOG_DUMP: + { ... + flags= uint2korr(packet + 4); + ... + mysql_binlog_send(thd, ..., flags); + ... + } + + void mysql_binlog_send(THD* thd, ..., ushort flags) + { ... + Log_event::read_log_event(&log, packet, ...); + ... + if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || + thd->server_id == 0 /* slave == mysqlbinlog */ ) + { + my_net_write(net, packet->ptr(), packet->length()); + } + ... + } + +7. How slave SQL thread processes Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The slave processes each recieved event by "applying" it, i.e. by +calling the Log_event::apply_event() function which in turn calls +the virtual do_apply_event() member specific for each type of the +event. + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev = next_event(rli); + ... + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + + int apply_event_and_update_pos(Log_event *ev, ...) + { ... + ev->apply_event(...); + ... + } + + int Log_event::apply_event(...) + { + return do_apply_event(...); + } + +What does it mean to "apply" an Annotate_rows event? It means to set current +thd query to that of the described by the event, i.e. to the query which +caused the subsequent Rows events (see "How Master writes Annotate_rows +events to the binary log" to follow what happens further when the subsequent +Rows events is applied): + + int Annotate_rows_log_event::do_apply_event(...) + { + thd->set_query(m_query_txt, m_query_len); + } + +NOTE. I am not sure, but possibly current values of thd->query and +thd->query_length should be saved before calling set_query() and to be +restored on the Annotate_rows_log_event object deletion. +Is it really needed ? + +After calling this do_apply_event() function we may not delete the +Annotate_rows_log_event object immediatedly (see exec_relay_log_event() +above) because thd->query now points to the string inside this object. +We may keep the pointer to this object in the Relay_log_info: + + class Relay_log_info + { + public: + ... + void set_annotate_event(Annotate_rows_log_event*); + Annotate_rows_log_event* get_annotate_event(); + void free_annotate_event(); + ... + private: + Annotate_rows_log_event* m_annotate_event; + }; + +When the saved Annotate_rows object may be deleted? When all corresponding +Rows events will be processed, i.e. before processing the first non-Rows +event (note that Annotate_rows object resides in the binary log *after* +the (possible) 'BEGIN' Query event which accompanies the rows events; note +also that this deletion is adjusted with the case when some or all +corresponding Rows events are filtered out by replicate filter rules): + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev= next_event(rli); + ... + if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) + rli->free_annotate_event(); + + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + rli->set_annotate_event((Annotate_rows_log_event*) ev); + else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + +where + + #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ + (type) == WRITE_ROWS_EVENT || \ + (type) == UPDATE_ROWS_EVENT || \ + (type) == DELETE_ROWS_EVENT ) + +8. General remarks +~~~~~~~~~~~~~~~~~~ +Kristian noticed that introducing new log event type should be coordinated +somehow with MySQL/Sun: + + Kristian: The numeric code for this event must be assigned carefully. + It should be coordinated with MySQL/Sun, otherwise we can get into a + situation where MySQL uses the same numeric code for one event that + MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two + impossible. + Alex: I reserved about 20 numbers not to have possible conflicts + with MySQL. + Kristian: Still, I think it would be appropriate to send a polite email + to internals(a)lists.mysql.com about this and suggesting to reserve the + event number. + +Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT +flag taking into account that MySQL/Sun may also introduce a flag with the +same value to be used in the request_dump-mysql_binlog_send interface. +But this is mainly the question of merging: if a conflict concerning this +flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value +(this does not require additional changes in the code). -=-=(Alexi - Sat, 19 Dec 2009, 15:41)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.14545 2009-12-19 15:41:21.000000000 +0200 +++ /tmp/wklog.47.new.14545 2009-12-19 15:41:21.000000000 +0200 @@ -1,122 +1,107 @@ -First suggestion: - -> I think for this we would actually need a new binlog event type -> (Comment_log_event?). Unless we want to log an empty statement Query_log_event -> containing only a comment (a bit of a hack). - -New server option -~~~~~~~~~~~~~~~~~ - --binlog-annotate-rows-events - -Setting this option makes RBR (rows-) events in the binary log to be -preceded by Annotate rows events (see below). The corresponding -'binlog_annotate_rows_events' system variable is dynamic and has both -global and session values. Default global value is OFF. - -Note. Session values are usefull to make it possible to annotate only - some selected statements: +Content +~~~~~~~ + 1. Annotate_rows_log_event + 2. Server option: --binlog-annotate-rows-events + 3. Server option: --replicate-annotate-rows-events + 4. mysqlbinlog option: --print-annotate-rows-events + 5. mysqlbinlog output + +1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Describes the query which caused the corresponding rows events. Has empty +post-header and contains the query text in its data part. Example: + + ************************ + ANNOTATE_RBR_EVENT + ************************ + 00000220 | B6 A0 2C 4B | time_when = 1261215926 + 00000224 | 33 | event_type = 51 + 00000225 | 64 00 00 00 | server_id = 100 + 00000229 | 36 00 00 00 | event_len = 54 + 0000022D | 56 02 00 00 | log_pos = 00000256 + 00000231 | 00 00 | flags = <none> + ------------------------ + 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" + 00000237 | 52 54 20 49 | + 0000023B | 4E 54 4F 20 | + 0000023F | 74 31 20 56 | + 00000243 | 41 4C 55 45 | + 00000247 | 53 20 28 31 | + 0000024B | 29 2C 20 28 | + 0000024F | 32 29 2C 20 | + 00000253 | 28 33 29 | + ************************ + +In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event +and precedes the first of Table map events which accompany the corresponding +rows events. (See example in the "mysqlbinlog output" section below.) + +2. Server option: --binlog-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Tells the master to write Annotate_rows events to the binary log. + + * Variable Name: binlog_annotate_rows_events + * Scope: Global & Session + * Access Type: Dynamic + * Data Type: bool + * Default Value: OFF +NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... -New binlog event type -~~~~~~~~~~~~~~~~~~~~~ - Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] - -Describes the query which caused the corresponding rows event. In binary log, -precedes each Table_map_log_event. Contains empty post-header and the query -text in its data part. - -The numeric code for this event must be assigned carefully. It should be -coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL -uses the same numeric code for one event that MariaDB uses for -ANNOTATE_ROWS_EVENT, which would make merging the two impossible. - -Example: - - ... - ************************ - ANNOTATE_ROWS_EVENT [51] - ************************ - 000000C7 | 54 1B 12 4B | time_when = 1259477844 - 000000CB | 33 | event_type = 51 - 000000CC | 64 00 00 00 | server_id = 100 - 000000D0 | 2C 00 00 00 | event_len = 44 - 000000D4 | F3 00 00 00 | log_pos = 000000F3 - 000000D8 | 00 00 | flags = <none> - ------------------------ - 000000DA | 69 6E 73 65 | query = "insert into t1 values (1)" - 000000DE | 72 74 20 69 | - 000000E2 | 6E 74 6F 20 | - 000000E6 | 74 31 20 76 | - 000000EA | 61 6C 75 65 | - 000000EE | 73 20 28 31 | - 000000F2 | 29 | - ************************ - TABLE_MAP_EVENT [19] - ************************ - 000000F3 | 54 1B 12 4B | time_when = 1259477844 - 000000F7 | 13 | event_type = 19 - 000000F8 | 64 00 00 00 | server_id = 100 - 000000FC | 29 00 00 00 | event_len = 41 - 00000100 | 1C 01 00 00 | log_pos = 0000011C - 00000104 | 00 00 | flags = <none> - ------------------------ - ... - ************************ - WRITE_ROWS_EVENT [23] - ************************ - 0000011C | 54 1B 12 4B | time_when = 1259477844 - 00000120 | 17 | event_type = 23 - 00000121 | 64 00 00 00 | server_id = 100 - 00000125 | 22 00 00 00 | event_len = 34 - 00000129 | 3E 01 00 00 | log_pos = 0000013E - 0000012D | 10 00 | flags = LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F - ------------------------ - 0000012F | 0F 00 00 00 | table_id = 15 - ... - -New mysqlbinlog option -~~~~~~~~~~~~~~~~~~~~~~ - --print-annotate-rows-events - -With this option, mysqlbinlog prints the content of Annotate-rows -events (if the binary log does contain them). Without this option -(i.e. by default), mysqlbinlog skips Annotate rows events. - - -mysqlbinlog output -~~~~~~~~~~~~~~~~~~ -Something like this: +3. Server option: --replicate-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Tells the slave to reproduce Annotate_rows events recieved from the master +in its own binary log (sensible only in pair with log-slave-updates option). + + * Variable Name: replicate_annotate_rows_events + * Scope: Global + * Access Type: Read only + * Data Type: bool + * Default Value: OFF + +NOTE. Why do we additionally need this 'replicate' option? Why not to make +the slave to reproduce this events when its binlog-annotate-rows-events +global value is ON? Well, because, for example, we may want to configure +the slave which should reproduce Annotate_rows events but has global +binlog-annotate-rows-events = OFF meaning this to be the default value for +the client threads (see also "How slave treats replicate-annotate-rows-events +option" in LLD part). + +4. mysqlbinlog option: --print-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +With this option, mysqlbinlog prints the content of Annotate_rows events (if +the binary log does contain them). Without this option (i.e. by default), +mysqlbinlog skips Annotate_rows events. +5. mysqlbinlog output +~~~~~~~~~~~~~~~~~~~~~ +With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events +in a form like this: ... - # at 199 - # at 243 - # at 284 - #091129 9:57:24 server id 100 end_log_pos 243 Query: `insert into t1 values -(1)` - #091129 9:57:24 server id 100 end_log_pos 284 Table_map: `test`.`t1` mapped -to number 15 - #091129 9:57:24 server id 100 end_log_pos 318 Write_rows: table id 15 + # at 1646 + #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 +exec_time=0 error_code=0 + SET TIMESTAMP=1261215926/*!*/; + BEGIN + /*!*/; + # at 1714 + # at 1812 + # at 1853 + # at 1894 + # at 1938 + #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM +t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` + #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` +mapped to number 16 + #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` +mapped to number 17 + #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 + #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F - - BINLOG ' - VBsSSzNkAAAALAAAAPMAAAAAAGluc2VydCBpbnRvIHQxIHZhbHVlcyAoMSk= - VBsSSxNkAAAAKQAAABwBAAAAAA8AAAAAAAAABHRlc3QAAnQxAAEDAAE= - VBsSSxdkAAAAIgAAAD4BAAAQAA8AAAAAAAEAAf/+AQAAAA== - '/*!*/; - ### INSERT INTO test.t1 - ### SET - ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ... -When master sends Annotate rows events -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -1. Master always sends Annotate_rows events to mysqlbinlog (in - remote case). -2. Master sends Annotate_rows events to a slave only if the slave has - both log-slave-updates and binlog-annotate-rows-events options set. - ------------------------------------------------------------ -=-=(View All Progress Notes, 22 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] FYI: askmonty.org/buildbot is moving to buildbot.askmonty.org/buildbot
by Daniel Bartholomew 04 Feb '10

04 Feb '10

All, This is an FYI for everyone that uses our buildbot web interface: Due to an upcoming server move I am in the process of moving our buildbot web interface from http://askmonty.org/buildbot to http://buildbot.askmonty.org/buildbot Both of the above currently work (and point at the same data), but as of Friday 5 Feb 2010 at approx 09:00 US Eastern time, http://askmonty.org/buildbot will turn into a redirect. The redirect will handle all of your old bookmarks (i.e. they should still work), but I thought I should at least let everyone know so that you aren't surprised by the move. I've already updated all of the links I could find on the askmonty.org wiki. And if you are wondering why I kept the /buildbot subdirectory in place on the new subdomain, well, it was easier and safer to keep it in place, so I did. If you connect directly to http://buildbot.askmonty.org you will automatically be redirected to the /buildbot subdir. Thanks! -- Daniel Bartholomew Monty Program - http://askmonty.org

1 0

[Maria-developers] Rev 157: saving uncommitted changes in /etc prior to apt run in file:///etc/
by timour＠askmonty.org 04 Feb '10

04 Feb '10

At file:///etc/ ------------------------------------------------------------ revno: 157 revision-id: timour(a)askmonty.org-20100204112537-3a6fyd3ibqzi73wc parent: root@lamia-20100203064317-whagbeui3pyl1bck committer: timour(a)askmonty.org branch nick: lamia /etc repository timestamp: Thu 2010-02-04 13:25:37 +0200 message: saving uncommitted changes in /etc prior to apt run === modified file 'cups/subscriptions.conf' --- a/cups/subscriptions.conf 2010-02-03 06:43:17 +0000 +++ b/cups/subscriptions.conf 2010-02-04 11:25:37 +0000 @@ -1,11 +1,11 @@ # Subscription configuration file for CUPS v1.4.1 -# Written by cupsd on 2010-02-02 08:18 -NextSubscriptionId 101 -<Subscription 100> +# Written by cupsd on 2010-02-04 13:19 +NextSubscriptionId 102 +<Subscription 101> Events printer-state-changed printer-restarted printer-shutdown printer-stopped printer-added printer-deleted job-state-changed job-created job-completed job-stopped job-progress Owner tsk LeaseDuration 86400 Interval 0 -ExpirationTime 1265177812 -NextEventId 8 +ExpirationTime 1265368739 +NextEventId 1 </Subscription> === modified file 'cups/subscriptions.conf.O' --- a/cups/subscriptions.conf.O 2010-02-03 06:43:17 +0000 +++ b/cups/subscriptions.conf.O 2010-02-04 11:25:37 +0000 @@ -1,11 +1,3 @@ # Subscription configuration file for CUPS v1.4.1 -# Written by cupsd on 2010-02-02 08:17 +# Written by cupsd on 2010-02-03 08:43 NextSubscriptionId 101 -<Subscription 100> -Events printer-state-changed printer-restarted printer-shutdown printer-stopped printer-added printer-deleted job-state-changed job-created job-completed job-stopped job-progress -Owner tsk -LeaseDuration 86400 -Interval 0 -ExpirationTime 1265177812 -NextEventId 7 -</Subscription> === modified file 'resolv.conf' --- a/resolv.conf 2010-02-01 11:56:20 +0000 +++ b/resolv.conf 2010-02-04 11:25:37 +0000 @@ -1,2 +1,3 @@ # Generated by NetworkManager -nameserver 192.168.2.1 +nameserver 212.50.0.10 +nameserver 212.50.10.50

1 0

[Maria-developers] bzr commit into file:///etc/ branch (timour:157)
by timour＠askmonty.org 04 Feb '10

04 Feb '10

#At file:///etc/ based on revid:root@lamia-20100203064317-whagbeui3pyl1bck 157 timour(a)askmonty.org 2010-02-04 saving uncommitted changes in /etc prior to apt run modified: cups/subscriptions.conf cups/subscriptions.conf.O resolv.conf === modified file 'cups/subscriptions.conf' --- a/cups/subscriptions.conf 2010-02-03 06:43:17 +0000 +++ b/cups/subscriptions.conf 2010-02-04 11:25:37 +0000 @@ -1,11 +1,11 @@ # Subscription configuration file for CUPS v1.4.1 -# Written by cupsd on 2010-02-02 08:18 -NextSubscriptionId 101 -<Subscription 100> +# Written by cupsd on 2010-02-04 13:19 +NextSubscriptionId 102 +<Subscription 101> Events printer-state-changed printer-restarted printer-shutdown printer-stopped printer-added printer-deleted job-state-changed job-created job-completed job-stopped job-progress Owner tsk LeaseDuration 86400 Interval 0 -ExpirationTime 1265177812 -NextEventId 8 +ExpirationTime 1265368739 +NextEventId 1 </Subscription> === modified file 'cups/subscriptions.conf.O' --- a/cups/subscriptions.conf.O 2010-02-03 06:43:17 +0000 +++ b/cups/subscriptions.conf.O 2010-02-04 11:25:37 +0000 @@ -1,11 +1,3 @@ # Subscription configuration file for CUPS v1.4.1 -# Written by cupsd on 2010-02-02 08:17 +# Written by cupsd on 2010-02-03 08:43 NextSubscriptionId 101 -<Subscription 100> -Events printer-state-changed printer-restarted printer-shutdown printer-stopped printer-added printer-deleted job-state-changed job-created job-completed job-stopped job-progress -Owner tsk -LeaseDuration 86400 -Interval 0 -ExpirationTime 1265177812 -NextEventId 7 -</Subscription> === modified file 'resolv.conf' --- a/resolv.conf 2010-02-01 11:56:20 +0000 +++ b/resolv.conf 2010-02-04 11:25:37 +0000 @@ -1,2 +1,3 @@ # Generated by NetworkManager -nameserver 192.168.2.1 +nameserver 212.50.0.10 +nameserver 212.50.10.50

1 0

[Maria-developers] Rev 8: Merge. in file:///Users/hakan/work/monty_program/mariadb-tools/
by Hakan Kuecuekyilmaz 04 Feb '10

04 Feb '10

At file:///Users/hakan/work/monty_program/mariadb-tools/ ------------------------------------------------------------ revno: 8 [merge] revision-id: hakan(a)askmonty.org-20100204111840-0d9u3nyio2h8tavi parent: hakan(a)askmonty.org-20100204011956-u74a4es6oogvd50w parent: knielsen@hasky-20100202151012-pjcblb2v3bqsp5eo committer: Hakan Kuecuekyilmaz <hakan(a)askmonty.org> branch nick: mariadb-tools timestamp: Thu 2010-02-04 12:18:40 +0100 message: Merge. modified: buildbot/maria-master.cfg mariamaster.cfg-20091218103450-cvifjz3i70oerkej-1 buildbot/runvm runvm-20091218082659-ept0jpsqa5e8jno6-2 === modified file 'buildbot/maria-master.cfg' --- a/buildbot/maria-master.cfg 2010-01-02 21:17:58 +0000 +++ b/buildbot/maria-master.cfg 2010-02-02 15:10:12 +0000 @@ -59,7 +59,6 @@ ,mkSlave("psergey-pylon-amd64", max_builds=1) ,mkSlave("psergey-foxhole-x86", max_builds=1) ,mkSlave("psergey-pslp2-x86", max_builds=1) - ,mkSlave("hakan-mac-g5", max_builds=1) ,mkSlave("work-opensuse-amd64", max_builds=1) ,mkSlave("psergey-win32box", max_builds=1) ,mkSlave("vm-win-1", max_builds=1) @@ -114,7 +113,7 @@ "lp:~maria-captains/maria/maria-5.2-merge-5.1" : "maria-5.2-merge-5.1", "lp:~maria-captains/maria/5.2-dsmrr" : "5.2-dsmrr", "lp:~maria-captains/maria/5.3" : "5.3", - "lp:~maria-captains/maria/5.3" : "5.3-sj-subqueries" + "lp:~maria-captains/maria/5.3-sj-subqueries" : "5.3-sj-subqueries" } mailSource = mail.BzrLaunchpadEmailMaildirSource("/var/lib/buildbot/Maildir", branchMap=myBranchMap) @@ -163,7 +162,7 @@ # "lenny-amd64-dbg", "adutko-alpha", "debian5-i386-fulltest", "jaunty-x86-valgrind", "jaunty-amd64-rel", - "gentoo-x86-dbg", "ubuntu-x86-dbg", "macosx-g5-dbg", + "gentoo-x86-dbg", "ubuntu-x86-dbg", "macosx-x86-bld", "work-amd64-valgrind", "winxp-x86-nmake", "win32-rel-nmake", "hardy-amd64-fulltest", "gentoo-amd64-sanja", "opensolaris-511-x86", @@ -180,6 +179,7 @@ branches=["mariadb-5.1-knielsen", "5.1-release", "5.1", "5.1.39-oqgraph", "5.1-merge", "mariadb-5.1-monty", + "5.2", "mariadb-5.2-monty", ], treeStableTimer=1, # 1 sec for bzr properties= {"bakebranch": "lp:~maria-captains/ourdelta/ourdelta-montyprogram-fixes"}, @@ -197,7 +197,9 @@ "kvm-deb-jaunty-amd64", "kvm-deb-jaunty-x86", "kvm-deb-karmic-amd64", "kvm-deb-karmic-x86", "kvm-deb-lucid-amd64", "kvm-deb-lucid-x86", - "kvm-bintar-hardy-amd64", "kvm-bintar-hardy-x86",])) + "kvm-bintar-hardy-amd64", "kvm-bintar-hardy-x86", + "opensolaris-511-bintar", + ])) ####### BUILDERS @@ -529,7 +531,7 @@ "slavename": "adutko-centos5-amd64", "builddir": "centos5-amd64-minimal", "factory": f_minimal, - "category": "experimental", + "category": "main", } f_win32_rel_nmake = factory.BuildFactory() @@ -699,7 +701,7 @@ "slavename": "adutko-ultrasparc3", "builddir": "adutko-ultrasparc3", "factory": f_sol_sparc_32, - "category": "new", + "category": "main", } f_dbg_alpha = factory.BuildFactory() @@ -771,23 +773,6 @@ "category": "experimental", } -f_dbg_g5 = factory.BuildFactory() -f_dbg_g5.addStep(bzr_shared_repo) -f_dbg_g5.addStep(bzr_checkout) -f_dbg_g5.addStep(getCompileStep(["BUILD/compile-ppc-max"])) -f_dbg_g5.addStep(getMTR( - test_type="nm", - test_info="Normal run, no --ps-protocol", - command=["sh", "-c", "cd mysql-test && exec perl mysql-test-run.pl --force --retry=3 --skip-ndb --testcase-timeout=45 --suite-timeout=1080 --mysqld=--skip-safemalloc"], - timeout=2700)) - -bld_mac_g5 = {'name': "macosx-g5-dbg", - 'slavename': "hakan-mac-g5", - 'builddir': "hakan-mac-g5", - 'factory': f_dbg_g5, - "category": "experimental", - } - f_mac_x86 = factory.BuildFactory() f_mac_x86.addStep(bzr_shared_repo) f_mac_x86.addStep(bzr_checkout) @@ -878,6 +863,36 @@ "category": "experimental", } +# The trees for which we save binary packages. +savedPackageBranches= ["5.1-release", "5.2-release", "5.3-release"] + +# Get a build step that will archive binary packages (or source tarball). +# Only the newest 3 builds are saved for each (branch, builder) combination. +# The packages are saved under the build number of the tarbake step, so it is +# easy to locate all packages for a given release. +def getPackageArchiveStep(source, tarbuildnum): + cmdText= ("TARBUILDNUM='" + tarbuildnum + "'\n" + + "SRC='" + source + "'\n" + """ +BUILDERNAME='%(buildername)s' +BRANCH='%(branch)s' +BASE="/archive/pack/$BRANCH" +DST="$BASE/build-$TARBUILDNUM/$BUILDERNAME" +set -ex +mkdir -p "$BASE" +rm -Rf "$DST" +(ls -td $BASE/build-*/"$BUILDERNAME" || : ) | (read DUMMY || exit 0; read DUMMY || exit 0; while read VICTIM; do rm -Rf "$VICTIM"; done) +rmdir $BASE/build-* 2>/dev/null || : +mkdir -p "$DST" +cp -r "$SRC" "$DST/" +""") + return ShellCommand( + doStepIf=(lambda(step): step.getProperty("branch") in savedPackageBranches), + description=["archiving"], + descriptionDone=["archive"], + command=["sh", "-c", WithProperties(cmdText)], + ) + + f_kvm_tarbake_jaunty_x86= factory.BuildFactory() f_kvm_tarbake_jaunty_x86.addStep(Compile( description=["making", "dist"], @@ -906,35 +921,38 @@ """ set -ex cd buildbot/build/$(cat buildbot/build/bakery.txt)/ -echo mariadb-*.tar.gz > ../distname.txt -mv $(cat ../distname.txt) ../ +basename mariadb-*.tar.gz .tar.gz > ../distdirname.txt +mv "$(cat ../distdirname.txt).tar.gz" ../ """, - "= scp -P 2223 buildbot@localhost:buildbot/build/distname.txt .", + "= scp -P 2223 buildbot@localhost:buildbot/build/distdirname.txt .", "= scp -P 2223 buildbot@localhost:buildbot/build/bakery.txt .", "= scp -P 2223 'buildbot@localhost:buildbot/build/mariadb-*.tar.gz' .", "= scp -P 2223 'buildbot@localhost:buildbot/build/bakery-*.tar.gz' .", ], )) f_kvm_tarbake_jaunty_x86.addStep(SetProperty( - property="distname", - command=["cat", "distname.txt"], + property="distdirname", + command=["cat", "distdirname.txt"], )) f_kvm_tarbake_jaunty_x86.addStep(SetProperty( property="bakery", command=["cat", "bakery.txt"], )) -f_kvm_tarbake_jaunty_x86.addStep(FileUpload(slavesrc=WithProperties("%(distname)s"), - masterdest=WithProperties("/var/lib/buildbot/OQ-tarballs/%(buildnumber)s:%(distname)s"))) +f_kvm_tarbake_jaunty_x86.addStep(getPackageArchiveStep("%(distdirname)s.tar.gz", "%(buildnumber)s")) +f_kvm_tarbake_jaunty_x86.addStep(FileUpload(slavesrc=WithProperties("%(distdirname)s.tar.gz"), + masterdest=WithProperties("/var/lib/buildbot/OQ-tarballs/%(buildnumber)s:%(distdirname)s.tar.gz"))) f_kvm_tarbake_jaunty_x86.addStep(FileUpload(slavesrc=WithProperties("%(bakery)s.tar.gz"), masterdest=WithProperties("/var/lib/buildbot/OQ-bakeries/%(buildnumber)s:%(bakery)s.tar.gz"))) f_kvm_tarbake_jaunty_x86.addStep(Trigger( schedulerNames=["kvm-ourdelta-sched"], waitForFinish=False, updateSourceStamp=True, - set_properties={ "tarballpath": WithProperties("/var/lib/buildbot/OQ-tarballs/%(buildnumber)s:%(distname)s"), + set_properties={ "tarballpath": WithProperties("/var/lib/buildbot/OQ-tarballs/%(buildnumber)s:%(distdirname)s.tar.gz"), "bakerypath": WithProperties("/var/lib/buildbot/OQ-bakeries/%(buildnumber)s:%(bakery)s.tar.gz"), + "distname": WithProperties("%(distdirname)s.tar.gz"), + "tarbuildnum": WithProperties("%(buildnumber)s"), }, - copy_properties=[ "distname", "bakery" ])) + copy_properties=[ "distdirname", "bakery" ])) bld_kvm_tarbake_jaunty_x86 = {"name": "kvm-tarbake-jaunty-x86", "slavename": "knielsen-kvm-x86", @@ -977,6 +995,7 @@ "= rm -Rf rpms && mkdir rpms", "= scp -P "+port+" 'buildbot@localhost:/usr/src/redhat/RPMS/"+arch+"/*.rpm' rpms/", ])) + rpm_fact.addStep(getPackageArchiveStep("rpms", "%(tarbuildnum)s")) rpm_fact.addStep(Test( description=["testing", "install"], descriptionDone=["test", "install"], @@ -1013,11 +1032,22 @@ deb_fact.addStep(FileDownload( mastersrc=WithProperties("%(bakerypath)s"), slavedest=WithProperties("%(bakery)s.tar.gz"))) + # Extract the compiler warning suppressions file from the source tarball. + deb_fact.addStep(ShellCommand( + description=["getting", ".supp"], + descriptionDone=["get", ".supp"], + command=["sh", "-c", WithProperties(""" +rm -f compiler_warnings.supp +tar zxf "%(distname)s" --strip 2 "$(basename %(distname)s .tar.gz)/support-files/compiler_warnings.supp" +exit 0 # best-effort, not fatal if no suppression file +""")])) deb_fact.addStep(Compile( description=["making", "debs"], descriptionDone=["make", "debs"], logfiles={"kernel": "kernel_"+port+".log"}, warningPattern=gccWarningPattern, + warningExtractor=Compile.warnExtractFromRegexpGroups, + suppressionFile=WithProperties("compiler_warnings.supp"), command=["runvm", "--base-image=/kvm/vms/"+kvm_image+"-build.qcow2"] + args +["--logfile=kernel_"+port+".log", "vm-tmp-"+port+".qcow2", "rm -Rf buildbot && mkdir buildbot", WithProperties("= scp -P "+port+" %(distname)s buildbot@localhost:buildbot/"), @@ -1039,6 +1069,7 @@ "= rm -Rf debs", "= scp -r -P "+port+" buildbot@localhost:buildbot/debs .", ])) + deb_fact.addStep(getPackageArchiveStep("debs", "%(tarbuildnum)s")) deb_fact.addStep(Test( description=["testing", "install"], descriptionDone=["test", "install"], @@ -1144,6 +1175,7 @@ property="bindistname", command=["cat", "bindistname.txt"], )) + bin_fact.addStep(getPackageArchiveStep("%(bindistname)s.tar.gz", "%(tarbuildnum)s")) bin_fact.addStep(Test( description=["testing", "bintar"], descriptionDone=["test", "bintar"], @@ -1178,14 +1210,55 @@ 'slavename': 'opensolaris-x86', 'builddir': 'opensolaris-511-x86', 'factory': f_dbg_sol_64, - "category": "new", + "category": "main", } +f_sol_64_bintar = factory.BuildFactory() +f_sol_64_bintar.addStep(FileDownload(mastersrc=WithProperties("%(tarballpath)s"), + slavedest=WithProperties("%(distname)s"))) +f_sol_64_bintar.addStep(ShellCommand( + description="untarring", descriptionDone="untar", + command=["sh", "-c", WithProperties("rm -Rf %(distdirname)s && tar zxf %(distname)s")])) +f_sol_64_bintar.addStep(getCompileStep(["sh", "-c", WithProperties("cd %(distdirname)s && BUILD/compile-solaris-amd64-debug")], + subdir="%(distdirname)s")) +f_sol_64_bintar.addStep(ShellCommand( + description="packaging", descriptionDone="package", + command=["sh", "-c", WithProperties(""" +set -ex +cd %(distdirname)s +scripts/make_binary_distribution +ORIG_TAR=$(echo mariadb-*.tar.gz) +ORIG_DIR=$(basename ${ORIG_TAR} .tar.gz) +NEW_DIR="$(echo "${ORIG_DIR}" | sed -e 's/-MariaDB//')" +NEW_TAR="${NEW_DIR}.tar.gz" +rm -Rf $ORIG_DIR +tar zxf $ORIG_TAR +mv $ORIG_DIR "${NEW_DIR}" +tar zcf "${NEW_TAR}" "${NEW_DIR}/" +echo "${NEW_TAR}" > ../bindistname.txt +""")])) +f_sol_64_bintar.addStep(SetProperty( + property="bindistname", + command=["cat", "bindistname.txt"], + )) +f_sol_64_bintar.addStep(getPackageArchiveStep("%(distdirname)s/%(bindistname)s", "%(tarbuildnum)s")) +f_sol_64_bintar.addStep(getMTR( + test_type="nm", + test_info="Normal run, no --ps-protocol", + command=["sh", "-c", WithProperties("cd %(distdirname)s/mysql-test && exec perl mysql-test-run.pl --force --retry=3 --skip-ndb --parallel=2")], + mtr_subdir=WithProperties("%(distdirname)s/mysql-test"))) + +bld_opensolaris_511_bintar = {'name': 'opensolaris-511-bintar', + 'slavename': 'opensolaris-x86', + 'builddir': 'opensolaris-511-bintar', + 'factory': f_sol_64_bintar, + "category": "package", + } + c['builders'] = [bld_fulltest, bld_fulltest2, bld_work_valgrind, bld_valgrind_32, bld_rel_amd64, bld_dbg_x86, bld_makedist, bld_archivist_amd64, bld_archivist_cnc, - bld_mac_g5, bld_dbg2_x86, bld_psergey_win32box, bld_win32_tarball, bld_win32_zip, bld_win32_rel_nmake, bld_mac_x86, @@ -1205,6 +1278,7 @@ bld_kvm_deb_karmic_amd64, bld_kvm_deb_karmic_x86, bld_kvm_deb_lucid_amd64, bld_kvm_deb_lucid_x86, bld_kvm_bintar_hardy_amd64, bld_kvm_bintar_hardy_x86, + bld_opensolaris_511_bintar, bld_opensolaris_511_x86, bld_sol_sparc_32, ] === modified file 'buildbot/runvm' --- a/buildbot/runvm 2010-01-14 13:43:51 +0000 +++ b/buildbot/runvm 2010-02-01 12:44:19 +0000 @@ -52,6 +52,19 @@ my $opt_baseimage= undef; my @user_cmd_opt; +# Disable host key checking for ssh. +# This is a bit convoluted due to OpenSSH's slight security-paranoia. +# Without this, we would get a login failure if using another VM image +# (with different host key) on the same port, which is annoying. +# An alternative would be to use CheckHostIP=no and HostKeyAlias=<img.qcow2> +# to get ssh to check a different key for each image. But that would still +# cause an error if re-generating an image (with new ssh host key), and it +# doesn't really give any additional security. +my @ssh_cmd_prefix= ($ssh_exec, + '-o', 'UserKnownHostsFile=/dev/null', + '-o', 'StrictHostKeyChecking=no', + '-o', 'LogLevel=ERROR'); + my $image; my $pidfile; @@ -177,17 +190,6 @@ return $res; } -sub exec_guest_cmd { - my ($cmd_and_args)= @_; - if (scalar(@$cmd_and_args) == 0 || $cmd_and_args->[0] eq '') - { - # Otherwise ssh will do an interactive login, which we do not want. - exit 0; - } - - exec_guest_cmd $ssh_exec, '-p', $opt_port, @user_cmd_opt, 'localhost', @$cmd_and_args; -} - sub is_port_used { socket(SOCK, PF_INET, SOCK_STREAM, getprotobyname('tcp')) or die "socket() failed: $!\n"; @@ -348,7 +350,7 @@ die "Fatal error: Cannot fork(): $!\n"; } elsif (!$pid) { # Child. - exec_with_print($ssh_exec, '-o', 'ConnectTimeout=4', '-p', $opt_port, + exec_with_print(@ssh_cmd_prefix, '-o', 'ConnectTimeout=4', '-p', $opt_port, @user_cmd_opt, 'localhost', 'sudo', '/sbin/shutdown', '-h', 'now'); } else { @@ -410,7 +412,7 @@ die "Fatal error: Cannot fork(): $!\n"; } elsif (!$pid) { # Child. - exec_with_print($ssh_exec, '-o', 'ConnectTimeout=4', '-p', $opt_port, + exec_with_print(@ssh_cmd_prefix, '-o', 'ConnectTimeout=4', '-p', $opt_port, @user_cmd_opt, 'localhost', '/bin/true'); } else { # Parent. @@ -506,7 +508,7 @@ $res= system(substr($arg, 1)); } else { print STDERR "+ $arg\n"; - $res= system($ssh_exec, '-p', $opt_port, @user_cmd_opt, 'localhost', $arg); + $res= system(@ssh_cmd_prefix, '-p', $opt_port, @user_cmd_opt, 'localhost', $arg); } if ($res < 0) { print STDERR "Could not spawn command: $!\n";

1 0

[Maria-developers] Updated (by Alexi): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 04 Feb '10

04 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen CATEGORY.......: Server-RawIdeaBin TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 20 ESTIMATE.......: 35 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 -=-=(Alexi - Sat, 19 Dec 2009, 15:58)=-=- Low Level Design modified. --- /tmp/wklog.47.old.15437 2009-12-19 13:58:12.000000000 +0000 +++ /tmp/wklog.47.new.15437 2009-12-19 13:58:12.000000000 +0000 @@ -1 +1,337 @@ +Content +~~~~~~~ + 1. Annotate_rows event number + 2. Outline of Annotate_rows event behavior + 3. How Master writes Annotate_rows events to the binary log + 4. How slave treats replicate-annotate-rows-events option + 5. How slave IO thread requests Annotate_rows events + 6. How master executes the request + 7. How slave SQL thread processes Annotate_rows events + 8. General remarks + +1. Annotate_rows event number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To avoid possible event numbers conflict with MySQL/Sun, we leave a gap +between the last MySQL event number and the Annotate_rows event number: + + enum Log_event_type + { ... + INCIDENT_EVENT= 26, + // New MySQL event numbers are to be added here + MYSQL_EVENTS_END, + + MARIA_EVENTS_BEGIN= 51, + // New Maria event numbers start from here + ANNOTATE_ROWS_EVENT= 51, + + ENUM_END_EVENT + }; + +together with the corresponding extension of 'post_header_len' array in the +Format description event. (This extension does not affect the compatibility +of the binary log). Here is how Format description event looks like with +this extension: + + ************************ + FORMAT_DESCRIPTION_EVENT + ************************ + 00000004 | A1 A0 2C 4B | time_when = 1261215905 + 00000008 | 0F | event_type = 15 + 00000009 | 64 00 00 00 | server_id = 100 + 0000000D | 7F 00 00 00 | event_len = 127 + 00000011 | 83 00 00 00 | log_pos = 00000083 + 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F + ------------------------ + 00000017 | 04 00 | binlog_ver = 4 + 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log + ..... ... + 0000004B | A1 A0 2C 4B | time_created = 1261215905 + 0000004F | 13 | common_header_len = 19 + ------------------------ + post_header_len + ------------------------ + 00000050 | 38 | 56 - START_EVENT_V3 [1] + ..... ... + 00000069 | 02 | 2 - INCIDENT_EVENT [26] + 0000006A | 00 | 0 - RESERVED [27] + ..... ... + 00000081 | 00 | 0 - RESERVED [50] + 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + ************************ + +2. Outline of Annotate_rows event behavior +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Each Annotate_rows_log_event object has two private members describing the +corresponding query: + + char *m_query_txt; + uint m_query_len; + +When the object is created for writing to a binary log, this query is taken +from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix +as well as other implementation details): + + Annotate_rows_log_event(THD *thd) + { + m_query_txt = thd->query(); + m_query_len = thd->query_length(); + } + +When the object is read from a binary log, the query is taken from the buffer +containing the binary log representation of the event (this buffer is allocated +in Log_event object from which all Log events are derived): + + Annotate_rows_log_event(char *buf, uint event_len, + Format_description_log_event *desc) + { + m_query_len = event_len - desc->common_header_len; + m_query_txt = buf + desc->common_header_len; + } + +The events are written to the binary log by the Log_event::write() member +which calls virtual write_data_header() and write_data_body() members +("data header" and "post header" are synonym in replication terminology). +In our case, data header is empty and data body is just the query: + + bool write_data_body(IO_CACHE *file) + { + return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); + } + +Printing the event is just printing the query: + + void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) + { + my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); + } + +3. How Master writes Annotate_rows events to the binary log +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The event is written to the binary log just before the group of Table_map +events which precede corresponding Rows events (one query may generate +several Table map events in the binary log, but the corresponding +Annotate_rows event must be written only once before the first Table map +event; hence the boolean variable 'with_annotate' below): + + int write_locked_table_maps(THD *thd) + { ... + bool with_annotate= thd->variables.binlog_annotate_rows_events; + ... + for (uint i= 0; i < ... <number of tables> ...; ++i) + { ... + thd->binlog_write_table_map(table, ..., with_annotate); + with_annotate= 0; // write Annotate_event not more than once + ... + } + ... + } + + int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) + { ... + Table_map_log_event the_event(...); + ... + if (with_annotate) + { + Annotate_rows_log_event anno(this); + mysql_bin_log.write(&anno); + } + + mysql_bin_log.write(&the_event); + ... + } + +4. How slave treats replicate-annotate-rows-events option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The replicate-annotate-rows-events option is treated just as the session +value of the binlog_annotate_rows_events variable for the slave IO and +SQL threads. This setting is done during initialization of these threads: + + pthread_handler_t handle_slave_io(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_IO); + ... + } + + pthread_handler_t handle_slave_sql(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_SQL); + ... + } + + int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) + { ... + thd->variables.binlog_annotate_rows_events= + opt_replicate_annotate_rows_events; + ... + } + +5. How slave IO thread requests Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When requesting an event, the slave should inform the master whether +it should send Annotate_rows events or not. To that end we add a new +BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: + + #define BINLOG_DUMP_NON_BLOCK 1 + #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 + + pthread_handler_t handle_slave_io(void *arg) + { ... + request_dump(mysql, ...); + ... + } + + int request_dump(MYSQL* mysql, ...) + { ... + if (opt_log_slave_updates && + mi->io_thd->variables.binlog_annotate_rows_events) + binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; + ... + int2store(buf + 4, binlog_flags); + ... + simple_command(mysql, COM_BINLOG_DUMP, buf, ...); + ... + } + +6. How master executes the request +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + case COM_BINLOG_DUMP: + { ... + flags= uint2korr(packet + 4); + ... + mysql_binlog_send(thd, ..., flags); + ... + } + + void mysql_binlog_send(THD* thd, ..., ushort flags) + { ... + Log_event::read_log_event(&log, packet, ...); + ... + if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || + thd->server_id == 0 /* slave == mysqlbinlog */ ) + { + my_net_write(net, packet->ptr(), packet->length()); + } + ... + } + +7. How slave SQL thread processes Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The slave processes each recieved event by "applying" it, i.e. by +calling the Log_event::apply_event() function which in turn calls +the virtual do_apply_event() member specific for each type of the +event. + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev = next_event(rli); + ... + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + + int apply_event_and_update_pos(Log_event *ev, ...) + { ... + ev->apply_event(...); + ... + } + + int Log_event::apply_event(...) + { + return do_apply_event(...); + } + +What does it mean to "apply" an Annotate_rows event? It means to set current +thd query to that of the described by the event, i.e. to the query which +caused the subsequent Rows events (see "How Master writes Annotate_rows +events to the binary log" to follow what happens further when the subsequent +Rows events is applied): + + int Annotate_rows_log_event::do_apply_event(...) + { + thd->set_query(m_query_txt, m_query_len); + } + +NOTE. I am not sure, but possibly current values of thd->query and +thd->query_length should be saved before calling set_query() and to be +restored on the Annotate_rows_log_event object deletion. +Is it really needed ? + +After calling this do_apply_event() function we may not delete the +Annotate_rows_log_event object immediatedly (see exec_relay_log_event() +above) because thd->query now points to the string inside this object. +We may keep the pointer to this object in the Relay_log_info: + + class Relay_log_info + { + public: + ... + void set_annotate_event(Annotate_rows_log_event*); + Annotate_rows_log_event* get_annotate_event(); + void free_annotate_event(); + ... + private: + Annotate_rows_log_event* m_annotate_event; + }; + +When the saved Annotate_rows object may be deleted? When all corresponding +Rows events will be processed, i.e. before processing the first non-Rows +event (note that Annotate_rows object resides in the binary log *after* +the (possible) 'BEGIN' Query event which accompanies the rows events; note +also that this deletion is adjusted with the case when some or all +corresponding Rows events are filtered out by replicate filter rules): + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev= next_event(rli); + ... + if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) + rli->free_annotate_event(); + + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + rli->set_annotate_event((Annotate_rows_log_event*) ev); + else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + +where + + #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ + (type) == WRITE_ROWS_EVENT || \ + (type) == UPDATE_ROWS_EVENT || \ + (type) == DELETE_ROWS_EVENT ) + +8. General remarks +~~~~~~~~~~~~~~~~~~ +Kristian noticed that introducing new log event type should be coordinated +somehow with MySQL/Sun: + + Kristian: The numeric code for this event must be assigned carefully. + It should be coordinated with MySQL/Sun, otherwise we can get into a + situation where MySQL uses the same numeric code for one event that + MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two + impossible. + Alex: I reserved about 20 numbers not to have possible conflicts + with MySQL. + Kristian: Still, I think it would be appropriate to send a polite email + to internals(a)lists.mysql.com about this and suggesting to reserve the + event number. + +Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT +flag taking into account that MySQL/Sun may also introduce a flag with the +same value to be used in the request_dump-mysql_binlog_send interface. +But this is mainly the question of merging: if a conflict concerning this +flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value +(this does not require additional changes in the code). -=-=(Alexi - Sat, 19 Dec 2009, 15:41)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.14545 2009-12-19 15:41:21.000000000 +0200 +++ /tmp/wklog.47.new.14545 2009-12-19 15:41:21.000000000 +0200 @@ -1,122 +1,107 @@ -First suggestion: - -> I think for this we would actually need a new binlog event type -> (Comment_log_event?). Unless we want to log an empty statement Query_log_event -> containing only a comment (a bit of a hack). - -New server option -~~~~~~~~~~~~~~~~~ - --binlog-annotate-rows-events - -Setting this option makes RBR (rows-) events in the binary log to be -preceded by Annotate rows events (see below). The corresponding -'binlog_annotate_rows_events' system variable is dynamic and has both -global and session values. Default global value is OFF. - -Note. Session values are usefull to make it possible to annotate only - some selected statements: +Content +~~~~~~~ + 1. Annotate_rows_log_event + 2. Server option: --binlog-annotate-rows-events + 3. Server option: --replicate-annotate-rows-events + 4. mysqlbinlog option: --print-annotate-rows-events + 5. mysqlbinlog output + +1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Describes the query which caused the corresponding rows events. Has empty +post-header and contains the query text in its data part. Example: + + ************************ + ANNOTATE_RBR_EVENT + ************************ + 00000220 | B6 A0 2C 4B | time_when = 1261215926 + 00000224 | 33 | event_type = 51 + 00000225 | 64 00 00 00 | server_id = 100 + 00000229 | 36 00 00 00 | event_len = 54 + 0000022D | 56 02 00 00 | log_pos = 00000256 + 00000231 | 00 00 | flags = <none> + ------------------------ + 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" + 00000237 | 52 54 20 49 | + 0000023B | 4E 54 4F 20 | + 0000023F | 74 31 20 56 | + 00000243 | 41 4C 55 45 | + 00000247 | 53 20 28 31 | + 0000024B | 29 2C 20 28 | + 0000024F | 32 29 2C 20 | + 00000253 | 28 33 29 | + ************************ + +In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event +and precedes the first of Table map events which accompany the corresponding +rows events. (See example in the "mysqlbinlog output" section below.) + +2. Server option: --binlog-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Tells the master to write Annotate_rows events to the binary log. + + * Variable Name: binlog_annotate_rows_events + * Scope: Global & Session + * Access Type: Dynamic + * Data Type: bool + * Default Value: OFF +NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... -New binlog event type -~~~~~~~~~~~~~~~~~~~~~ - Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] - -Describes the query which caused the corresponding rows event. In binary log, -precedes each Table_map_log_event. Contains empty post-header and the query -text in its data part. - -The numeric code for this event must be assigned carefully. It should be -coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL -uses the same numeric code for one event that MariaDB uses for -ANNOTATE_ROWS_EVENT, which would make merging the two impossible. - -Example: - - ... - ************************ - ANNOTATE_ROWS_EVENT [51] - ************************ - 000000C7 | 54 1B 12 4B | time_when = 1259477844 - 000000CB | 33 | event_type = 51 - 000000CC | 64 00 00 00 | server_id = 100 - 000000D0 | 2C 00 00 00 | event_len = 44 - 000000D4 | F3 00 00 00 | log_pos = 000000F3 - 000000D8 | 00 00 | flags = <none> - ------------------------ - 000000DA | 69 6E 73 65 | query = "insert into t1 values (1)" - 000000DE | 72 74 20 69 | - 000000E2 | 6E 74 6F 20 | - 000000E6 | 74 31 20 76 | - 000000EA | 61 6C 75 65 | - 000000EE | 73 20 28 31 | - 000000F2 | 29 | - ************************ - TABLE_MAP_EVENT [19] - ************************ - 000000F3 | 54 1B 12 4B | time_when = 1259477844 - 000000F7 | 13 | event_type = 19 - 000000F8 | 64 00 00 00 | server_id = 100 - 000000FC | 29 00 00 00 | event_len = 41 - 00000100 | 1C 01 00 00 | log_pos = 0000011C - 00000104 | 00 00 | flags = <none> - ------------------------ - ... - ************************ - WRITE_ROWS_EVENT [23] - ************************ - 0000011C | 54 1B 12 4B | time_when = 1259477844 - 00000120 | 17 | event_type = 23 - 00000121 | 64 00 00 00 | server_id = 100 - 00000125 | 22 00 00 00 | event_len = 34 - 00000129 | 3E 01 00 00 | log_pos = 0000013E - 0000012D | 10 00 | flags = LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F - ------------------------ - 0000012F | 0F 00 00 00 | table_id = 15 - ... - -New mysqlbinlog option -~~~~~~~~~~~~~~~~~~~~~~ - --print-annotate-rows-events - -With this option, mysqlbinlog prints the content of Annotate-rows -events (if the binary log does contain them). Without this option -(i.e. by default), mysqlbinlog skips Annotate rows events. - - -mysqlbinlog output -~~~~~~~~~~~~~~~~~~ -Something like this: +3. Server option: --replicate-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Tells the slave to reproduce Annotate_rows events recieved from the master +in its own binary log (sensible only in pair with log-slave-updates option). + + * Variable Name: replicate_annotate_rows_events + * Scope: Global + * Access Type: Read only + * Data Type: bool + * Default Value: OFF + +NOTE. Why do we additionally need this 'replicate' option? Why not to make +the slave to reproduce this events when its binlog-annotate-rows-events +global value is ON? Well, because, for example, we may want to configure +the slave which should reproduce Annotate_rows events but has global +binlog-annotate-rows-events = OFF meaning this to be the default value for +the client threads (see also "How slave treats replicate-annotate-rows-events +option" in LLD part). + +4. mysqlbinlog option: --print-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +With this option, mysqlbinlog prints the content of Annotate_rows events (if +the binary log does contain them). Without this option (i.e. by default), +mysqlbinlog skips Annotate_rows events. +5. mysqlbinlog output +~~~~~~~~~~~~~~~~~~~~~ +With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events +in a form like this: ... - # at 199 - # at 243 - # at 284 - #091129 9:57:24 server id 100 end_log_pos 243 Query: `insert into t1 values -(1)` - #091129 9:57:24 server id 100 end_log_pos 284 Table_map: `test`.`t1` mapped -to number 15 - #091129 9:57:24 server id 100 end_log_pos 318 Write_rows: table id 15 + # at 1646 + #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 +exec_time=0 error_code=0 + SET TIMESTAMP=1261215926/*!*/; + BEGIN + /*!*/; + # at 1714 + # at 1812 + # at 1853 + # at 1894 + # at 1938 + #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM +t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` + #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` +mapped to number 16 + #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` +mapped to number 17 + #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 + #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F - - BINLOG ' - VBsSSzNkAAAALAAAAPMAAAAAAGluc2VydCBpbnRvIHQxIHZhbHVlcyAoMSk= - VBsSSxNkAAAAKQAAABwBAAAAAA8AAAAAAAAABHRlc3QAAnQxAAEDAAE= - VBsSSxdkAAAAIgAAAD4BAAAQAA8AAAAAAAEAAf/+AQAAAA== - '/*!*/; - ### INSERT INTO test.t1 - ### SET - ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ... -When master sends Annotate rows events -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -1. Master always sends Annotate_rows events to mysqlbinlog (in - remote case). -2. Master sends Annotate_rows events to a slave only if the slave has - both log-slave-updates and binlog-annotate-rows-events options set. - -=-=(Bothorsen - Fri, 18 Dec 2009, 16:22)=-=- Add estimation time. Worked 5 hours and estimate 35 hours remain (original estimate increased by 5 hours). -=-=(Bothorsen - Fri, 18 Dec 2009, 16:16)=-=- This is the work done on this patch so far. Most of it done by Alex. Worked 15 hours and estimate 035 hours remain (original estimate increased by 50 hours). ------------------------------------------------------------ -=-=(View All Progress Notes, 20 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Alexi): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 04 Feb '10

04 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen CATEGORY.......: Server-RawIdeaBin TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 20 ESTIMATE.......: 35 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 -=-=(Alexi - Sat, 19 Dec 2009, 15:58)=-=- Low Level Design modified. --- /tmp/wklog.47.old.15437 2009-12-19 13:58:12.000000000 +0000 +++ /tmp/wklog.47.new.15437 2009-12-19 13:58:12.000000000 +0000 @@ -1 +1,337 @@ +Content +~~~~~~~ + 1. Annotate_rows event number + 2. Outline of Annotate_rows event behavior + 3. How Master writes Annotate_rows events to the binary log + 4. How slave treats replicate-annotate-rows-events option + 5. How slave IO thread requests Annotate_rows events + 6. How master executes the request + 7. How slave SQL thread processes Annotate_rows events + 8. General remarks + +1. Annotate_rows event number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To avoid possible event numbers conflict with MySQL/Sun, we leave a gap +between the last MySQL event number and the Annotate_rows event number: + + enum Log_event_type + { ... + INCIDENT_EVENT= 26, + // New MySQL event numbers are to be added here + MYSQL_EVENTS_END, + + MARIA_EVENTS_BEGIN= 51, + // New Maria event numbers start from here + ANNOTATE_ROWS_EVENT= 51, + + ENUM_END_EVENT + }; + +together with the corresponding extension of 'post_header_len' array in the +Format description event. (This extension does not affect the compatibility +of the binary log). Here is how Format description event looks like with +this extension: + + ************************ + FORMAT_DESCRIPTION_EVENT + ************************ + 00000004 | A1 A0 2C 4B | time_when = 1261215905 + 00000008 | 0F | event_type = 15 + 00000009 | 64 00 00 00 | server_id = 100 + 0000000D | 7F 00 00 00 | event_len = 127 + 00000011 | 83 00 00 00 | log_pos = 00000083 + 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F + ------------------------ + 00000017 | 04 00 | binlog_ver = 4 + 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log + ..... ... + 0000004B | A1 A0 2C 4B | time_created = 1261215905 + 0000004F | 13 | common_header_len = 19 + ------------------------ + post_header_len + ------------------------ + 00000050 | 38 | 56 - START_EVENT_V3 [1] + ..... ... + 00000069 | 02 | 2 - INCIDENT_EVENT [26] + 0000006A | 00 | 0 - RESERVED [27] + ..... ... + 00000081 | 00 | 0 - RESERVED [50] + 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + ************************ + +2. Outline of Annotate_rows event behavior +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Each Annotate_rows_log_event object has two private members describing the +corresponding query: + + char *m_query_txt; + uint m_query_len; + +When the object is created for writing to a binary log, this query is taken +from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix +as well as other implementation details): + + Annotate_rows_log_event(THD *thd) + { + m_query_txt = thd->query(); + m_query_len = thd->query_length(); + } + +When the object is read from a binary log, the query is taken from the buffer +containing the binary log representation of the event (this buffer is allocated +in Log_event object from which all Log events are derived): + + Annotate_rows_log_event(char *buf, uint event_len, + Format_description_log_event *desc) + { + m_query_len = event_len - desc->common_header_len; + m_query_txt = buf + desc->common_header_len; + } + +The events are written to the binary log by the Log_event::write() member +which calls virtual write_data_header() and write_data_body() members +("data header" and "post header" are synonym in replication terminology). +In our case, data header is empty and data body is just the query: + + bool write_data_body(IO_CACHE *file) + { + return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); + } + +Printing the event is just printing the query: + + void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) + { + my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); + } + +3. How Master writes Annotate_rows events to the binary log +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The event is written to the binary log just before the group of Table_map +events which precede corresponding Rows events (one query may generate +several Table map events in the binary log, but the corresponding +Annotate_rows event must be written only once before the first Table map +event; hence the boolean variable 'with_annotate' below): + + int write_locked_table_maps(THD *thd) + { ... + bool with_annotate= thd->variables.binlog_annotate_rows_events; + ... + for (uint i= 0; i < ... <number of tables> ...; ++i) + { ... + thd->binlog_write_table_map(table, ..., with_annotate); + with_annotate= 0; // write Annotate_event not more than once + ... + } + ... + } + + int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) + { ... + Table_map_log_event the_event(...); + ... + if (with_annotate) + { + Annotate_rows_log_event anno(this); + mysql_bin_log.write(&anno); + } + + mysql_bin_log.write(&the_event); + ... + } + +4. How slave treats replicate-annotate-rows-events option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The replicate-annotate-rows-events option is treated just as the session +value of the binlog_annotate_rows_events variable for the slave IO and +SQL threads. This setting is done during initialization of these threads: + + pthread_handler_t handle_slave_io(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_IO); + ... + } + + pthread_handler_t handle_slave_sql(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_SQL); + ... + } + + int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) + { ... + thd->variables.binlog_annotate_rows_events= + opt_replicate_annotate_rows_events; + ... + } + +5. How slave IO thread requests Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When requesting an event, the slave should inform the master whether +it should send Annotate_rows events or not. To that end we add a new +BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: + + #define BINLOG_DUMP_NON_BLOCK 1 + #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 + + pthread_handler_t handle_slave_io(void *arg) + { ... + request_dump(mysql, ...); + ... + } + + int request_dump(MYSQL* mysql, ...) + { ... + if (opt_log_slave_updates && + mi->io_thd->variables.binlog_annotate_rows_events) + binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; + ... + int2store(buf + 4, binlog_flags); + ... + simple_command(mysql, COM_BINLOG_DUMP, buf, ...); + ... + } + +6. How master executes the request +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + case COM_BINLOG_DUMP: + { ... + flags= uint2korr(packet + 4); + ... + mysql_binlog_send(thd, ..., flags); + ... + } + + void mysql_binlog_send(THD* thd, ..., ushort flags) + { ... + Log_event::read_log_event(&log, packet, ...); + ... + if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || + thd->server_id == 0 /* slave == mysqlbinlog */ ) + { + my_net_write(net, packet->ptr(), packet->length()); + } + ... + } + +7. How slave SQL thread processes Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The slave processes each recieved event by "applying" it, i.e. by +calling the Log_event::apply_event() function which in turn calls +the virtual do_apply_event() member specific for each type of the +event. + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev = next_event(rli); + ... + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + + int apply_event_and_update_pos(Log_event *ev, ...) + { ... + ev->apply_event(...); + ... + } + + int Log_event::apply_event(...) + { + return do_apply_event(...); + } + +What does it mean to "apply" an Annotate_rows event? It means to set current +thd query to that of the described by the event, i.e. to the query which +caused the subsequent Rows events (see "How Master writes Annotate_rows +events to the binary log" to follow what happens further when the subsequent +Rows events is applied): + + int Annotate_rows_log_event::do_apply_event(...) + { + thd->set_query(m_query_txt, m_query_len); + } + +NOTE. I am not sure, but possibly current values of thd->query and +thd->query_length should be saved before calling set_query() and to be +restored on the Annotate_rows_log_event object deletion. +Is it really needed ? + +After calling this do_apply_event() function we may not delete the +Annotate_rows_log_event object immediatedly (see exec_relay_log_event() +above) because thd->query now points to the string inside this object. +We may keep the pointer to this object in the Relay_log_info: + + class Relay_log_info + { + public: + ... + void set_annotate_event(Annotate_rows_log_event*); + Annotate_rows_log_event* get_annotate_event(); + void free_annotate_event(); + ... + private: + Annotate_rows_log_event* m_annotate_event; + }; + +When the saved Annotate_rows object may be deleted? When all corresponding +Rows events will be processed, i.e. before processing the first non-Rows +event (note that Annotate_rows object resides in the binary log *after* +the (possible) 'BEGIN' Query event which accompanies the rows events; note +also that this deletion is adjusted with the case when some or all +corresponding Rows events are filtered out by replicate filter rules): + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev= next_event(rli); + ... + if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) + rli->free_annotate_event(); + + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + rli->set_annotate_event((Annotate_rows_log_event*) ev); + else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + +where + + #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ + (type) == WRITE_ROWS_EVENT || \ + (type) == UPDATE_ROWS_EVENT || \ + (type) == DELETE_ROWS_EVENT ) + +8. General remarks +~~~~~~~~~~~~~~~~~~ +Kristian noticed that introducing new log event type should be coordinated +somehow with MySQL/Sun: + + Kristian: The numeric code for this event must be assigned carefully. + It should be coordinated with MySQL/Sun, otherwise we can get into a + situation where MySQL uses the same numeric code for one event that + MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two + impossible. + Alex: I reserved about 20 numbers not to have possible conflicts + with MySQL. + Kristian: Still, I think it would be appropriate to send a polite email + to internals(a)lists.mysql.com about this and suggesting to reserve the + event number. + +Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT +flag taking into account that MySQL/Sun may also introduce a flag with the +same value to be used in the request_dump-mysql_binlog_send interface. +But this is mainly the question of merging: if a conflict concerning this +flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value +(this does not require additional changes in the code). -=-=(Alexi - Sat, 19 Dec 2009, 15:41)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.14545 2009-12-19 15:41:21.000000000 +0200 +++ /tmp/wklog.47.new.14545 2009-12-19 15:41:21.000000000 +0200 @@ -1,122 +1,107 @@ -First suggestion: - -> I think for this we would actually need a new binlog event type -> (Comment_log_event?). Unless we want to log an empty statement Query_log_event -> containing only a comment (a bit of a hack). - -New server option -~~~~~~~~~~~~~~~~~ - --binlog-annotate-rows-events - -Setting this option makes RBR (rows-) events in the binary log to be -preceded by Annotate rows events (see below). The corresponding -'binlog_annotate_rows_events' system variable is dynamic and has both -global and session values. Default global value is OFF. - -Note. Session values are usefull to make it possible to annotate only - some selected statements: +Content +~~~~~~~ + 1. Annotate_rows_log_event + 2. Server option: --binlog-annotate-rows-events + 3. Server option: --replicate-annotate-rows-events + 4. mysqlbinlog option: --print-annotate-rows-events + 5. mysqlbinlog output + +1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Describes the query which caused the corresponding rows events. Has empty +post-header and contains the query text in its data part. Example: + + ************************ + ANNOTATE_RBR_EVENT + ************************ + 00000220 | B6 A0 2C 4B | time_when = 1261215926 + 00000224 | 33 | event_type = 51 + 00000225 | 64 00 00 00 | server_id = 100 + 00000229 | 36 00 00 00 | event_len = 54 + 0000022D | 56 02 00 00 | log_pos = 00000256 + 00000231 | 00 00 | flags = <none> + ------------------------ + 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" + 00000237 | 52 54 20 49 | + 0000023B | 4E 54 4F 20 | + 0000023F | 74 31 20 56 | + 00000243 | 41 4C 55 45 | + 00000247 | 53 20 28 31 | + 0000024B | 29 2C 20 28 | + 0000024F | 32 29 2C 20 | + 00000253 | 28 33 29 | + ************************ + +In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event +and precedes the first of Table map events which accompany the corresponding +rows events. (See example in the "mysqlbinlog output" section below.) + +2. Server option: --binlog-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Tells the master to write Annotate_rows events to the binary log. + + * Variable Name: binlog_annotate_rows_events + * Scope: Global & Session + * Access Type: Dynamic + * Data Type: bool + * Default Value: OFF +NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... -New binlog event type -~~~~~~~~~~~~~~~~~~~~~ - Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] - -Describes the query which caused the corresponding rows event. In binary log, -precedes each Table_map_log_event. Contains empty post-header and the query -text in its data part. - -The numeric code for this event must be assigned carefully. It should be -coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL -uses the same numeric code for one event that MariaDB uses for -ANNOTATE_ROWS_EVENT, which would make merging the two impossible. - -Example: - - ... - ************************ - ANNOTATE_ROWS_EVENT [51] - ************************ - 000000C7 | 54 1B 12 4B | time_when = 1259477844 - 000000CB | 33 | event_type = 51 - 000000CC | 64 00 00 00 | server_id = 100 - 000000D0 | 2C 00 00 00 | event_len = 44 - 000000D4 | F3 00 00 00 | log_pos = 000000F3 - 000000D8 | 00 00 | flags = <none> - ------------------------ - 000000DA | 69 6E 73 65 | query = "insert into t1 values (1)" - 000000DE | 72 74 20 69 | - 000000E2 | 6E 74 6F 20 | - 000000E6 | 74 31 20 76 | - 000000EA | 61 6C 75 65 | - 000000EE | 73 20 28 31 | - 000000F2 | 29 | - ************************ - TABLE_MAP_EVENT [19] - ************************ - 000000F3 | 54 1B 12 4B | time_when = 1259477844 - 000000F7 | 13 | event_type = 19 - 000000F8 | 64 00 00 00 | server_id = 100 - 000000FC | 29 00 00 00 | event_len = 41 - 00000100 | 1C 01 00 00 | log_pos = 0000011C - 00000104 | 00 00 | flags = <none> - ------------------------ - ... - ************************ - WRITE_ROWS_EVENT [23] - ************************ - 0000011C | 54 1B 12 4B | time_when = 1259477844 - 00000120 | 17 | event_type = 23 - 00000121 | 64 00 00 00 | server_id = 100 - 00000125 | 22 00 00 00 | event_len = 34 - 00000129 | 3E 01 00 00 | log_pos = 0000013E - 0000012D | 10 00 | flags = LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F - ------------------------ - 0000012F | 0F 00 00 00 | table_id = 15 - ... - -New mysqlbinlog option -~~~~~~~~~~~~~~~~~~~~~~ - --print-annotate-rows-events - -With this option, mysqlbinlog prints the content of Annotate-rows -events (if the binary log does contain them). Without this option -(i.e. by default), mysqlbinlog skips Annotate rows events. - - -mysqlbinlog output -~~~~~~~~~~~~~~~~~~ -Something like this: +3. Server option: --replicate-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Tells the slave to reproduce Annotate_rows events recieved from the master +in its own binary log (sensible only in pair with log-slave-updates option). + + * Variable Name: replicate_annotate_rows_events + * Scope: Global + * Access Type: Read only + * Data Type: bool + * Default Value: OFF + +NOTE. Why do we additionally need this 'replicate' option? Why not to make +the slave to reproduce this events when its binlog-annotate-rows-events +global value is ON? Well, because, for example, we may want to configure +the slave which should reproduce Annotate_rows events but has global +binlog-annotate-rows-events = OFF meaning this to be the default value for +the client threads (see also "How slave treats replicate-annotate-rows-events +option" in LLD part). + +4. mysqlbinlog option: --print-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +With this option, mysqlbinlog prints the content of Annotate_rows events (if +the binary log does contain them). Without this option (i.e. by default), +mysqlbinlog skips Annotate_rows events. +5. mysqlbinlog output +~~~~~~~~~~~~~~~~~~~~~ +With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events +in a form like this: ... - # at 199 - # at 243 - # at 284 - #091129 9:57:24 server id 100 end_log_pos 243 Query: `insert into t1 values -(1)` - #091129 9:57:24 server id 100 end_log_pos 284 Table_map: `test`.`t1` mapped -to number 15 - #091129 9:57:24 server id 100 end_log_pos 318 Write_rows: table id 15 + # at 1646 + #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 +exec_time=0 error_code=0 + SET TIMESTAMP=1261215926/*!*/; + BEGIN + /*!*/; + # at 1714 + # at 1812 + # at 1853 + # at 1894 + # at 1938 + #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM +t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` + #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` +mapped to number 16 + #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` +mapped to number 17 + #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 + #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F - - BINLOG ' - VBsSSzNkAAAALAAAAPMAAAAAAGluc2VydCBpbnRvIHQxIHZhbHVlcyAoMSk= - VBsSSxNkAAAAKQAAABwBAAAAAA8AAAAAAAAABHRlc3QAAnQxAAEDAAE= - VBsSSxdkAAAAIgAAAD4BAAAQAA8AAAAAAAEAAf/+AQAAAA== - '/*!*/; - ### INSERT INTO test.t1 - ### SET - ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ... -When master sends Annotate rows events -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -1. Master always sends Annotate_rows events to mysqlbinlog (in - remote case). -2. Master sends Annotate_rows events to a slave only if the slave has - both log-slave-updates and binlog-annotate-rows-events options set. - -=-=(Bothorsen - Fri, 18 Dec 2009, 16:22)=-=- Add estimation time. Worked 5 hours and estimate 35 hours remain (original estimate increased by 5 hours). -=-=(Bothorsen - Fri, 18 Dec 2009, 16:16)=-=- This is the work done on this patch so far. Most of it done by Alex. Worked 15 hours and estimate 035 hours remain (original estimate increased by 50 hours). ------------------------------------------------------------ -=-=(View All Progress Notes, 20 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Alexi): Store in binlog text of statements that caused RBR events (47)
by worklog-noreply＠askmonty.org 04 Feb '10

04 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Store in binlog text of statements that caused RBR events CREATION DATE..: Sat, 15 Aug 2009, 23:48 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Knielsen CATEGORY.......: Server-RawIdeaBin TASK ID........: 47 (http://askmonty.org/worklog/?tid=47) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 20 ESTIMATE.......: 35 (hours remain) ORIG. ESTIMATE.: 35 PROGRESS NOTES: -=-=(Alexi - Thu, 04 Feb 2010, 09:54)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16174 2010-02-04 09:54:13.000000000 +0200 +++ /tmp/wklog.47.new.16174 2010-02-04 09:54:13.000000000 +0200 @@ -171,35 +171,20 @@ 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When requesting an event, the slave should inform the master whether -it should send Annotate_rows events or not. To that end we add a new -BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: +If the replicate-annotate-rows-events option is not set on a slave, there +is no need for master to send Annotate_rows events to this slave. The slave +(or mysqlbinlog in remote case), before requesting binlog dump via the +COM_BINLOG_DUMP command, informs the master whether it should send these +events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server +command: + + case COM_BINLOG_DUMP_OPTIONS_EXT: + thd->binlog_dump_flags_ext= packet[0]; + my_ok(thd); + break; - #define BINLOG_DUMP_NON_BLOCK 1 - #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 - - pthread_handler_t handle_slave_io(void *arg) - { ... - request_dump(mysql, ...); - ... - } - - int request_dump(MYSQL* mysql, ...) - { ... - if (opt_log_slave_updates && - mi->io_thd->variables.binlog_annotate_rows_events) - binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; - ... - int2store(buf + 4, binlog_flags); - ... - simple_command(mysql, COM_BINLOG_DUMP, buf, ...); - ... - } - -NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the -simple_command() function, should also use this flag if it wants (in case -of the --print-annotate-rows-events option set) to recieve Annotate_rows -events. +Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible +conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -338,10 +323,4 @@ to internals(a)lists.mysql.com about this and suggesting to reserve the event number. -Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT -flag taking into account that MySQL/Sun may also introduce a flag with the -same value to be used in the request_dump-mysql_binlog_send interface. -But this is mainly the question of merging: if a conflict concerning this -flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value -(this does not require additional changes in the code). -=-=(Alexi - Sun, 20 Dec 2009, 16:00)=-=- Low Level Design modified. --- /tmp/wklog.47.old.19667 2009-12-20 14:00:56.000000000 +0000 +++ /tmp/wklog.47.new.19667 2009-12-20 14:00:56.000000000 +0000 @@ -196,6 +196,11 @@ ... } +NOTE. mysqlbinlog, when remotely requesting BINLOG_DUMP by calling the +simple_command() function, should also use this flag if it wants (in case +of the --print-annotate-rows-events option set) to recieve Annotate_rows +events. + 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -212,8 +217,7 @@ Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || - flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || - thd->server_id == 0 /* slave == mysqlbinlog */ ) + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } -=-=(Alexi - Sun, 20 Dec 2009, 13:14)=-=- Low Level Design modified. --- /tmp/wklog.47.old.11350 2009-12-20 13:14:04.000000000 +0200 +++ /tmp/wklog.47.new.11350 2009-12-20 13:14:04.000000000 +0200 @@ -282,23 +282,18 @@ Annotate_rows_log_event* m_annotate_event; }; -When the saved Annotate_rows object may be deleted? When all corresponding -Rows events will be processed, i.e. before processing the first non-Rows -event (note that Annotate_rows object resides in the binary log *after* -the (possible) 'BEGIN' Query event which accompanies the rows events; note -also that this deletion is adjusted with the case when some or all -corresponding Rows events are filtered out by replicate filter rules): +The saved Annotate_rows object should be deleted when all corresponding +Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... - if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) - rli->free_annotate_event(); - apply_event_and_update_pos(ev, ...); - if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + if (rli->get_annotate_event() && is_last_rows_event(ev)) + rli->free_annotate_event(); + else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; @@ -307,10 +302,21 @@ where - #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ - (type) == WRITE_ROWS_EVENT || \ + bool is_last_rows_event(Log_event* ev) + { + Log_event_type type= ev->get_type_code(); + if (IS_ROWS_EVENT_TYPE(type)) + { + Rows_log_event* rows= (Rows_log_event*)ev; + return rows->get_flags(Rows_log_event::STMT_END_F); + } + + return 0; + } + + #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ - (type) == DELETE_ROWS_EVENT ) + (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ -=-=(Alexi - Sun, 20 Dec 2009, 09:29)=-=- Low Level Design modified. --- /tmp/wklog.47.old.32726 2009-12-20 07:29:56.000000000 +0000 +++ /tmp/wklog.47.new.32726 2009-12-20 07:29:56.000000000 +0000 @@ -56,7 +56,7 @@ 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] - 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior -=-=(Alexi - Sat, 19 Dec 2009, 16:10)=-=- Low Level Design modified. --- /tmp/wklog.47.old.16051 2009-12-19 16:10:48.000000000 +0200 +++ /tmp/wklog.47.new.16051 2009-12-19 16:10:48.000000000 +0200 @@ -253,7 +253,7 @@ thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent -Rows events is applied): +Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { -=-=(Alexi - Sat, 19 Dec 2009, 16:02)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.15695 2009-12-19 16:02:33.000000000 +0200 +++ /tmp/wklog.47.new.15695 2009-12-19 16:02:33.000000000 +0200 @@ -12,7 +12,7 @@ post-header and contains the query text in its data part. Example: ************************ - ANNOTATE_RBR_EVENT + ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 -=-=(Alexi - Sat, 19 Dec 2009, 15:58)=-=- Low Level Design modified. --- /tmp/wklog.47.old.15437 2009-12-19 13:58:12.000000000 +0000 +++ /tmp/wklog.47.new.15437 2009-12-19 13:58:12.000000000 +0000 @@ -1 +1,337 @@ +Content +~~~~~~~ + 1. Annotate_rows event number + 2. Outline of Annotate_rows event behavior + 3. How Master writes Annotate_rows events to the binary log + 4. How slave treats replicate-annotate-rows-events option + 5. How slave IO thread requests Annotate_rows events + 6. How master executes the request + 7. How slave SQL thread processes Annotate_rows events + 8. General remarks + +1. Annotate_rows event number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To avoid possible event numbers conflict with MySQL/Sun, we leave a gap +between the last MySQL event number and the Annotate_rows event number: + + enum Log_event_type + { ... + INCIDENT_EVENT= 26, + // New MySQL event numbers are to be added here + MYSQL_EVENTS_END, + + MARIA_EVENTS_BEGIN= 51, + // New Maria event numbers start from here + ANNOTATE_ROWS_EVENT= 51, + + ENUM_END_EVENT + }; + +together with the corresponding extension of 'post_header_len' array in the +Format description event. (This extension does not affect the compatibility +of the binary log). Here is how Format description event looks like with +this extension: + + ************************ + FORMAT_DESCRIPTION_EVENT + ************************ + 00000004 | A1 A0 2C 4B | time_when = 1261215905 + 00000008 | 0F | event_type = 15 + 00000009 | 64 00 00 00 | server_id = 100 + 0000000D | 7F 00 00 00 | event_len = 127 + 00000011 | 83 00 00 00 | log_pos = 00000083 + 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F + ------------------------ + 00000017 | 04 00 | binlog_ver = 4 + 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log + ..... ... + 0000004B | A1 A0 2C 4B | time_created = 1261215905 + 0000004F | 13 | common_header_len = 19 + ------------------------ + post_header_len + ------------------------ + 00000050 | 38 | 56 - START_EVENT_V3 [1] + ..... ... + 00000069 | 02 | 2 - INCIDENT_EVENT [26] + 0000006A | 00 | 0 - RESERVED [27] + ..... ... + 00000081 | 00 | 0 - RESERVED [50] + 00000082 | 00 | 0 - ANNOTATE_RBR_EVENT [51] + ************************ + +2. Outline of Annotate_rows event behavior +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Each Annotate_rows_log_event object has two private members describing the +corresponding query: + + char *m_query_txt; + uint m_query_len; + +When the object is created for writing to a binary log, this query is taken +from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix +as well as other implementation details): + + Annotate_rows_log_event(THD *thd) + { + m_query_txt = thd->query(); + m_query_len = thd->query_length(); + } + +When the object is read from a binary log, the query is taken from the buffer +containing the binary log representation of the event (this buffer is allocated +in Log_event object from which all Log events are derived): + + Annotate_rows_log_event(char *buf, uint event_len, + Format_description_log_event *desc) + { + m_query_len = event_len - desc->common_header_len; + m_query_txt = buf + desc->common_header_len; + } + +The events are written to the binary log by the Log_event::write() member +which calls virtual write_data_header() and write_data_body() members +("data header" and "post header" are synonym in replication terminology). +In our case, data header is empty and data body is just the query: + + bool write_data_body(IO_CACHE *file) + { + return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); + } + +Printing the event is just printing the query: + + void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) + { + my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); + } + +3. How Master writes Annotate_rows events to the binary log +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The event is written to the binary log just before the group of Table_map +events which precede corresponding Rows events (one query may generate +several Table map events in the binary log, but the corresponding +Annotate_rows event must be written only once before the first Table map +event; hence the boolean variable 'with_annotate' below): + + int write_locked_table_maps(THD *thd) + { ... + bool with_annotate= thd->variables.binlog_annotate_rows_events; + ... + for (uint i= 0; i < ... <number of tables> ...; ++i) + { ... + thd->binlog_write_table_map(table, ..., with_annotate); + with_annotate= 0; // write Annotate_event not more than once + ... + } + ... + } + + int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) + { ... + Table_map_log_event the_event(...); + ... + if (with_annotate) + { + Annotate_rows_log_event anno(this); + mysql_bin_log.write(&anno); + } + + mysql_bin_log.write(&the_event); + ... + } + +4. How slave treats replicate-annotate-rows-events option +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The replicate-annotate-rows-events option is treated just as the session +value of the binlog_annotate_rows_events variable for the slave IO and +SQL threads. This setting is done during initialization of these threads: + + pthread_handler_t handle_slave_io(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_IO); + ... + } + + pthread_handler_t handle_slave_sql(void *arg) + { + THD *thd= new THD; + ... + init_slave_thread(thd, SLAVE_THD_SQL); + ... + } + + int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) + { ... + thd->variables.binlog_annotate_rows_events= + opt_replicate_annotate_rows_events; + ... + } + +5. How slave IO thread requests Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When requesting an event, the slave should inform the master whether +it should send Annotate_rows events or not. To that end we add a new +BINLOG_SEND_ANNOTATE_ROWS_EVENT flag used when requesting an event: + + #define BINLOG_DUMP_NON_BLOCK 1 + #define BINLOG_SEND_ANNOTATE_ROWS_EVENT 2 + + pthread_handler_t handle_slave_io(void *arg) + { ... + request_dump(mysql, ...); + ... + } + + int request_dump(MYSQL* mysql, ...) + { ... + if (opt_log_slave_updates && + mi->io_thd->variables.binlog_annotate_rows_events) + binlog_flags|= BINLOG_SEND_ANNOTATE_ROWS_EVENT; + ... + int2store(buf + 4, binlog_flags); + ... + simple_command(mysql, COM_BINLOG_DUMP, buf, ...); + ... + } + +6. How master executes the request +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + case COM_BINLOG_DUMP: + { ... + flags= uint2korr(packet + 4); + ... + mysql_binlog_send(thd, ..., flags); + ... + } + + void mysql_binlog_send(THD* thd, ..., ushort flags) + { ... + Log_event::read_log_event(&log, packet, ...); + ... + if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || + flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT || + thd->server_id == 0 /* slave == mysqlbinlog */ ) + { + my_net_write(net, packet->ptr(), packet->length()); + } + ... + } + +7. How slave SQL thread processes Annotate_rows events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The slave processes each recieved event by "applying" it, i.e. by +calling the Log_event::apply_event() function which in turn calls +the virtual do_apply_event() member specific for each type of the +event. + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev = next_event(rli); + ... + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + + int apply_event_and_update_pos(Log_event *ev, ...) + { ... + ev->apply_event(...); + ... + } + + int Log_event::apply_event(...) + { + return do_apply_event(...); + } + +What does it mean to "apply" an Annotate_rows event? It means to set current +thd query to that of the described by the event, i.e. to the query which +caused the subsequent Rows events (see "How Master writes Annotate_rows +events to the binary log" to follow what happens further when the subsequent +Rows events is applied): + + int Annotate_rows_log_event::do_apply_event(...) + { + thd->set_query(m_query_txt, m_query_len); + } + +NOTE. I am not sure, but possibly current values of thd->query and +thd->query_length should be saved before calling set_query() and to be +restored on the Annotate_rows_log_event object deletion. +Is it really needed ? + +After calling this do_apply_event() function we may not delete the +Annotate_rows_log_event object immediatedly (see exec_relay_log_event() +above) because thd->query now points to the string inside this object. +We may keep the pointer to this object in the Relay_log_info: + + class Relay_log_info + { + public: + ... + void set_annotate_event(Annotate_rows_log_event*); + Annotate_rows_log_event* get_annotate_event(); + void free_annotate_event(); + ... + private: + Annotate_rows_log_event* m_annotate_event; + }; + +When the saved Annotate_rows object may be deleted? When all corresponding +Rows events will be processed, i.e. before processing the first non-Rows +event (note that Annotate_rows object resides in the binary log *after* +the (possible) 'BEGIN' Query event which accompanies the rows events; note +also that this deletion is adjusted with the case when some or all +corresponding Rows events are filtered out by replicate filter rules): + + int exec_relay_log_event(THD* thd, Relay_log_info* rli) + { ... + Log_event *ev= next_event(rli); + ... + if (rli->get_annotate_event() && !IS_RBR_EVENT_TYPE(ev->get_type_code())) + rli->free_annotate_event(); + + apply_event_and_update_pos(ev, ...); + + if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) + rli->set_annotate_event((Annotate_rows_log_event*) ev); + else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) + delete ev; + ... + } + +where + + #define IS_RBR_EVENT_TYPE(type) ( (type) == TABLE_MAP_EVENT || \ + (type) == WRITE_ROWS_EVENT || \ + (type) == UPDATE_ROWS_EVENT || \ + (type) == DELETE_ROWS_EVENT ) + +8. General remarks +~~~~~~~~~~~~~~~~~~ +Kristian noticed that introducing new log event type should be coordinated +somehow with MySQL/Sun: + + Kristian: The numeric code for this event must be assigned carefully. + It should be coordinated with MySQL/Sun, otherwise we can get into a + situation where MySQL uses the same numeric code for one event that + MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two + impossible. + Alex: I reserved about 20 numbers not to have possible conflicts + with MySQL. + Kristian: Still, I think it would be appropriate to send a polite email + to internals(a)lists.mysql.com about this and suggesting to reserve the + event number. + +Also we should notice the introduction of the BINLOG_SEND_ANNOTATE_ROWS_EVENT +flag taking into account that MySQL/Sun may also introduce a flag with the +same value to be used in the request_dump-mysql_binlog_send interface. +But this is mainly the question of merging: if a conflict concerning this +flag occur, we may simply change the BINLOG_SEND_ANNOTATE_ROWS_EVENT value +(this does not require additional changes in the code). -=-=(Alexi - Sat, 19 Dec 2009, 15:41)=-=- High-Level Specification modified. --- /tmp/wklog.47.old.14545 2009-12-19 15:41:21.000000000 +0200 +++ /tmp/wklog.47.new.14545 2009-12-19 15:41:21.000000000 +0200 @@ -1,122 +1,107 @@ -First suggestion: - -> I think for this we would actually need a new binlog event type -> (Comment_log_event?). Unless we want to log an empty statement Query_log_event -> containing only a comment (a bit of a hack). - -New server option -~~~~~~~~~~~~~~~~~ - --binlog-annotate-rows-events - -Setting this option makes RBR (rows-) events in the binary log to be -preceded by Annotate rows events (see below). The corresponding -'binlog_annotate_rows_events' system variable is dynamic and has both -global and session values. Default global value is OFF. - -Note. Session values are usefull to make it possible to annotate only - some selected statements: +Content +~~~~~~~ + 1. Annotate_rows_log_event + 2. Server option: --binlog-annotate-rows-events + 3. Server option: --replicate-annotate-rows-events + 4. mysqlbinlog option: --print-annotate-rows-events + 5. mysqlbinlog output + +1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Describes the query which caused the corresponding rows events. Has empty +post-header and contains the query text in its data part. Example: + + ************************ + ANNOTATE_RBR_EVENT + ************************ + 00000220 | B6 A0 2C 4B | time_when = 1261215926 + 00000224 | 33 | event_type = 51 + 00000225 | 64 00 00 00 | server_id = 100 + 00000229 | 36 00 00 00 | event_len = 54 + 0000022D | 56 02 00 00 | log_pos = 00000256 + 00000231 | 00 00 | flags = <none> + ------------------------ + 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" + 00000237 | 52 54 20 49 | + 0000023B | 4E 54 4F 20 | + 0000023F | 74 31 20 56 | + 00000243 | 41 4C 55 45 | + 00000247 | 53 20 28 31 | + 0000024B | 29 2C 20 28 | + 0000024F | 32 29 2C 20 | + 00000253 | 28 33 29 | + ************************ + +In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event +and precedes the first of Table map events which accompany the corresponding +rows events. (See example in the "mysqlbinlog output" section below.) + +2. Server option: --binlog-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Tells the master to write Annotate_rows events to the binary log. + + * Variable Name: binlog_annotate_rows_events + * Scope: Global & Session + * Access Type: Dynamic + * Data Type: bool + * Default Value: OFF +NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... -New binlog event type -~~~~~~~~~~~~~~~~~~~~~ - Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] - -Describes the query which caused the corresponding rows event. In binary log, -precedes each Table_map_log_event. Contains empty post-header and the query -text in its data part. - -The numeric code for this event must be assigned carefully. It should be -coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL -uses the same numeric code for one event that MariaDB uses for -ANNOTATE_ROWS_EVENT, which would make merging the two impossible. - -Example: - - ... - ************************ - ANNOTATE_ROWS_EVENT [51] - ************************ - 000000C7 | 54 1B 12 4B | time_when = 1259477844 - 000000CB | 33 | event_type = 51 - 000000CC | 64 00 00 00 | server_id = 100 - 000000D0 | 2C 00 00 00 | event_len = 44 - 000000D4 | F3 00 00 00 | log_pos = 000000F3 - 000000D8 | 00 00 | flags = <none> - ------------------------ - 000000DA | 69 6E 73 65 | query = "insert into t1 values (1)" - 000000DE | 72 74 20 69 | - 000000E2 | 6E 74 6F 20 | - 000000E6 | 74 31 20 76 | - 000000EA | 61 6C 75 65 | - 000000EE | 73 20 28 31 | - 000000F2 | 29 | - ************************ - TABLE_MAP_EVENT [19] - ************************ - 000000F3 | 54 1B 12 4B | time_when = 1259477844 - 000000F7 | 13 | event_type = 19 - 000000F8 | 64 00 00 00 | server_id = 100 - 000000FC | 29 00 00 00 | event_len = 41 - 00000100 | 1C 01 00 00 | log_pos = 0000011C - 00000104 | 00 00 | flags = <none> - ------------------------ - ... - ************************ - WRITE_ROWS_EVENT [23] - ************************ - 0000011C | 54 1B 12 4B | time_when = 1259477844 - 00000120 | 17 | event_type = 23 - 00000121 | 64 00 00 00 | server_id = 100 - 00000125 | 22 00 00 00 | event_len = 34 - 00000129 | 3E 01 00 00 | log_pos = 0000013E - 0000012D | 10 00 | flags = LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F - ------------------------ - 0000012F | 0F 00 00 00 | table_id = 15 - ... - -New mysqlbinlog option -~~~~~~~~~~~~~~~~~~~~~~ - --print-annotate-rows-events - -With this option, mysqlbinlog prints the content of Annotate-rows -events (if the binary log does contain them). Without this option -(i.e. by default), mysqlbinlog skips Annotate rows events. - - -mysqlbinlog output -~~~~~~~~~~~~~~~~~~ -Something like this: +3. Server option: --replicate-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Tells the slave to reproduce Annotate_rows events recieved from the master +in its own binary log (sensible only in pair with log-slave-updates option). + + * Variable Name: replicate_annotate_rows_events + * Scope: Global + * Access Type: Read only + * Data Type: bool + * Default Value: OFF + +NOTE. Why do we additionally need this 'replicate' option? Why not to make +the slave to reproduce this events when its binlog-annotate-rows-events +global value is ON? Well, because, for example, we may want to configure +the slave which should reproduce Annotate_rows events but has global +binlog-annotate-rows-events = OFF meaning this to be the default value for +the client threads (see also "How slave treats replicate-annotate-rows-events +option" in LLD part). + +4. mysqlbinlog option: --print-annotate-rows-events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +With this option, mysqlbinlog prints the content of Annotate_rows events (if +the binary log does contain them). Without this option (i.e. by default), +mysqlbinlog skips Annotate_rows events. +5. mysqlbinlog output +~~~~~~~~~~~~~~~~~~~~~ +With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events +in a form like this: ... - # at 199 - # at 243 - # at 284 - #091129 9:57:24 server id 100 end_log_pos 243 Query: `insert into t1 values -(1)` - #091129 9:57:24 server id 100 end_log_pos 284 Table_map: `test`.`t1` mapped -to number 15 - #091129 9:57:24 server id 100 end_log_pos 318 Write_rows: table id 15 + # at 1646 + #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 +exec_time=0 error_code=0 + SET TIMESTAMP=1261215926/*!*/; + BEGIN + /*!*/; + # at 1714 + # at 1812 + # at 1853 + # at 1894 + # at 1938 + #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM +t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` + #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` +mapped to number 16 + #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` +mapped to number 17 + #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 + #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F - - BINLOG ' - VBsSSzNkAAAALAAAAPMAAAAAAGluc2VydCBpbnRvIHQxIHZhbHVlcyAoMSk= - VBsSSxNkAAAAKQAAABwBAAAAAA8AAAAAAAAABHRlc3QAAnQxAAEDAAE= - VBsSSxdkAAAAIgAAAD4BAAAQAA8AAAAAAAEAAf/+AQAAAA== - '/*!*/; - ### INSERT INTO test.t1 - ### SET - ### @1=1 /* INT meta=0 nullable=1 is_null=0 */ ... -When master sends Annotate rows events -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -1. Master always sends Annotate_rows events to mysqlbinlog (in - remote case). -2. Master sends Annotate_rows events to a slave only if the slave has - both log-slave-updates and binlog-annotate-rows-events options set. - -=-=(Bothorsen - Fri, 18 Dec 2009, 16:22)=-=- Add estimation time. Worked 5 hours and estimate 35 hours remain (original estimate increased by 5 hours). -=-=(Bothorsen - Fri, 18 Dec 2009, 16:16)=-=- This is the work done on this patch so far. Most of it done by Alex. Worked 15 hours and estimate 035 hours remain (original estimate increased by 50 hours). ------------------------------------------------------------ -=-=(View All Progress Notes, 20 total)=-=- http://askmonty.org/worklog/index.pl?tid=47&nolimit=1 DESCRIPTION: Store in binlog (and show in mysqlbinlog output) texts of statements that caused RBR events This is needed for (list from Monty): - Easier to understand why updates happened - Would make it easier to find out where in application things went wrong (as you can search for exact strings) - Allow one to filter things based on comments in the statement. The cost of this can be that the binlog will be approximately 2x in size (especially insert of big blob's would be a bit painful), so this should be an optional feature. HIGH-LEVEL SPECIFICATION: Content ~~~~~~~ 1. Annotate_rows_log_event 2. Server option: --binlog-annotate-rows-events 3. Server option: --replicate-annotate-rows-events 4. mysqlbinlog option: --print-annotate-rows-events 5. mysqlbinlog output 1. Annotate_rows_log_event [ ANNOTATE_ROWS_EVENT ] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Describes the query which caused the corresponding rows events. Has empty post-header and contains the query text in its data part. Example: ************************ ANNOTATE_ROWS_EVENT ************************ 00000220 | B6 A0 2C 4B | time_when = 1261215926 00000224 | 33 | event_type = 51 00000225 | 64 00 00 00 | server_id = 100 00000229 | 36 00 00 00 | event_len = 54 0000022D | 56 02 00 00 | log_pos = 00000256 00000231 | 00 00 | flags = <none> ------------------------ 00000233 | 49 4E 53 45 | query = "INSERT INTO t1 VALUES (1), (2), (3)" 00000237 | 52 54 20 49 | 0000023B | 4E 54 4F 20 | 0000023F | 74 31 20 56 | 00000243 | 41 4C 55 45 | 00000247 | 53 20 28 31 | 0000024B | 29 2C 20 28 | 0000024F | 32 29 2C 20 | 00000253 | 28 33 29 | ************************ In binary log, Annotate_rows event follows the (possible) 'BEGIN' Query event and precedes the first of Table map events which accompany the corresponding rows events. (See example in the "mysqlbinlog output" section below.) 2. Server option: --binlog-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the master to write Annotate_rows events to the binary log. * Variable Name: binlog_annotate_rows_events * Scope: Global & Session * Access Type: Dynamic * Data Type: bool * Default Value: OFF NOTE. Session values allows to annotate only some selected statements: ... SET SESSION binlog_annotate_rows_events=ON; ... statements to be annotated ... SET SESSION binlog_annotate_rows_events=OFF; ... statements not to be annotated ... 3. Server option: --replicate-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tells the slave to reproduce Annotate_rows events recieved from the master in its own binary log (sensible only in pair with log-slave-updates option). * Variable Name: replicate_annotate_rows_events * Scope: Global * Access Type: Read only * Data Type: bool * Default Value: OFF NOTE. Why do we additionally need this 'replicate' option? Why not to make the slave to reproduce this events when its binlog-annotate-rows-events global value is ON? Well, because, for example, we may want to configure the slave which should reproduce Annotate_rows events but has global binlog-annotate-rows-events = OFF meaning this to be the default value for the client threads (see also "How slave treats replicate-annotate-rows-events option" in LLD part). 4. mysqlbinlog option: --print-annotate-rows-events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With this option, mysqlbinlog prints the content of Annotate_rows events (if the binary log does contain them). Without this option (i.e. by default), mysqlbinlog skips Annotate_rows events. 5. mysqlbinlog output ~~~~~~~~~~~~~~~~~~~~~ With --print-annotate-rows-events, mysqlbinlog outputs Annotate_rows events in a form like this: ... # at 1646 #091219 12:45:26 server id 100 end_log_pos 1714 Query thread_id=1 exec_time=0 error_code=0 SET TIMESTAMP=1261215926/*!*/; BEGIN /*!*/; # at 1714 # at 1812 # at 1853 # at 1894 # at 1938 #091219 12:45:26 server id 100 end_log_pos 1812 Query: `DELETE t1, t2 FROM t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.a=t2.a AND t2.a=t3.a` #091219 12:45:26 server id 100 end_log_pos 1853 Table_map: `test`.`t1` mapped to number 16 #091219 12:45:26 server id 100 end_log_pos 1894 Table_map: `test`.`t2` mapped to number 17 #091219 12:45:26 server id 100 end_log_pos 1938 Delete_rows: table id 16 #091219 12:45:26 server id 100 end_log_pos 1982 Delete_rows: table id 17 flags: STMT_END_F ... LOW-LEVEL DESIGN: Content ~~~~~~~ 1. Annotate_rows event number 2. Outline of Annotate_rows event behavior 3. How Master writes Annotate_rows events to the binary log 4. How slave treats replicate-annotate-rows-events option 5. How slave IO thread requests Annotate_rows events 6. How master executes the request 7. How slave SQL thread processes Annotate_rows events 8. General remarks 1. Annotate_rows event number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To avoid possible event numbers conflict with MySQL/Sun, we leave a gap between the last MySQL event number and the Annotate_rows event number: enum Log_event_type { ... INCIDENT_EVENT= 26, // New MySQL event numbers are to be added here MYSQL_EVENTS_END, MARIA_EVENTS_BEGIN= 51, // New Maria event numbers start from here ANNOTATE_ROWS_EVENT= 51, ENUM_END_EVENT }; together with the corresponding extension of 'post_header_len' array in the Format description event. (This extension does not affect the compatibility of the binary log). Here is how Format description event looks like with this extension: ************************ FORMAT_DESCRIPTION_EVENT ************************ 00000004 | A1 A0 2C 4B | time_when = 1261215905 00000008 | 0F | event_type = 15 00000009 | 64 00 00 00 | server_id = 100 0000000D | 7F 00 00 00 | event_len = 127 00000011 | 83 00 00 00 | log_pos = 00000083 00000015 | 01 00 | flags = LOG_EVENT_BINLOG_IN_USE_F ------------------------ 00000017 | 04 00 | binlog_ver = 4 00000019 | 35 2E 32 2E | server_ver = 5.2.0-MariaDB-alpha-debug-log ..... ... 0000004B | A1 A0 2C 4B | time_created = 1261215905 0000004F | 13 | common_header_len = 19 ------------------------ post_header_len ------------------------ 00000050 | 38 | 56 - START_EVENT_V3 [1] ..... ... 00000069 | 02 | 2 - INCIDENT_EVENT [26] 0000006A | 00 | 0 - RESERVED [27] ..... ... 00000081 | 00 | 0 - RESERVED [50] 00000082 | 00 | 0 - ANNOTATE_ROWS_EVENT [51] ************************ 2. Outline of Annotate_rows event behavior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each Annotate_rows_log_event object has two private members describing the corresponding query: char *m_query_txt; uint m_query_len; When the object is created for writing to a binary log, this query is taken from 'thd' (for short, below we omit the 'Annotate_rows_log_event::' prefix as well as other implementation details): Annotate_rows_log_event(THD *thd) { m_query_txt = thd->query(); m_query_len = thd->query_length(); } When the object is read from a binary log, the query is taken from the buffer containing the binary log representation of the event (this buffer is allocated in Log_event object from which all Log events are derived): Annotate_rows_log_event(char *buf, uint event_len, Format_description_log_event *desc) { m_query_len = event_len - desc->common_header_len; m_query_txt = buf + desc->common_header_len; } The events are written to the binary log by the Log_event::write() member which calls virtual write_data_header() and write_data_body() members ("data header" and "post header" are synonym in replication terminology). In our case, data header is empty and data body is just the query: bool write_data_body(IO_CACHE *file) { return my_b_safe_write(file, (uchar*) m_query_txt, m_query_len); } Printing the event is just printing the query: void Annotate_rows_log_event::print(FILE *file, PRINT_EVENT_INFO *pinfo) { my_b_printf(&pinfo->head_cache, "\tQuery: `%s`\n", m_query_txt); } 3. How Master writes Annotate_rows events to the binary log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The event is written to the binary log just before the group of Table_map events which precede corresponding Rows events (one query may generate several Table map events in the binary log, but the corresponding Annotate_rows event must be written only once before the first Table map event; hence the boolean variable 'with_annotate' below): int write_locked_table_maps(THD *thd) { ... bool with_annotate= thd->variables.binlog_annotate_rows_events; ... for (uint i= 0; i < ... <number of tables> ...; ++i) { ... thd->binlog_write_table_map(table, ..., with_annotate); with_annotate= 0; // write Annotate_event not more than once ... } ... } int THD::binlog_write_table_map(TABLE *table, ..., bool with_annotate) { ... Table_map_log_event the_event(...); ... if (with_annotate) { Annotate_rows_log_event anno(this); mysql_bin_log.write(&anno); } mysql_bin_log.write(&the_event); ... } 4. How slave treats replicate-annotate-rows-events option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The replicate-annotate-rows-events option is treated just as the session value of the binlog_annotate_rows_events variable for the slave IO and SQL threads. This setting is done during initialization of these threads: pthread_handler_t handle_slave_io(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_IO); ... } pthread_handler_t handle_slave_sql(void *arg) { THD *thd= new THD; ... init_slave_thread(thd, SLAVE_THD_SQL); ... } int init_slave_thread(THD* thd, SLAVE_THD_TYPE thd_type) { ... thd->variables.binlog_annotate_rows_events= opt_replicate_annotate_rows_events; ... } 5. How slave IO thread requests Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the replicate-annotate-rows-events option is not set on a slave, there is no need for master to send Annotate_rows events to this slave. The slave (or mysqlbinlog in remote case), before requesting binlog dump via the COM_BINLOG_DUMP command, informs the master whether it should send these events by executing the newly added COM_BINLOG_DUMP_OPTIONS_EXT server command: case COM_BINLOG_DUMP_OPTIONS_EXT: thd->binlog_dump_flags_ext= packet[0]; my_ok(thd); break; Note. We add this new command and don't use COM_BINLOG_DUMP to avoid possible conflicts with MySQL/Sun. 6. How master executes the request ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ case COM_BINLOG_DUMP: { ... flags= uint2korr(packet + 4); ... mysql_binlog_send(thd, ..., flags); ... } void mysql_binlog_send(THD* thd, ..., ushort flags) { ... Log_event::read_log_event(&log, packet, ...); ... if ((*packet)[EVENT_TYPE_OFFSET + 1] != ANNOTATE_ROWS_EVENT || flags & BINLOG_SEND_ANNOTATE_ROWS_EVENT) { my_net_write(net, packet->ptr(), packet->length()); } ... } 7. How slave SQL thread processes Annotate_rows events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The slave processes each recieved event by "applying" it, i.e. by calling the Log_event::apply_event() function which in turn calls the virtual do_apply_event() member specific for each type of the event. int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev = next_event(rli); ... apply_event_and_update_pos(ev, ...); if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } int apply_event_and_update_pos(Log_event *ev, ...) { ... ev->apply_event(...); ... } int Log_event::apply_event(...) { return do_apply_event(...); } What does it mean to "apply" an Annotate_rows event? It means to set current thd query to that of the described by the event, i.e. to the query which caused the subsequent Rows events (see "How Master writes Annotate_rows events to the binary log" to follow what happens further when the subsequent Rows events are applied): int Annotate_rows_log_event::do_apply_event(...) { thd->set_query(m_query_txt, m_query_len); } NOTE. I am not sure, but possibly current values of thd->query and thd->query_length should be saved before calling set_query() and to be restored on the Annotate_rows_log_event object deletion. Is it really needed ? After calling this do_apply_event() function we may not delete the Annotate_rows_log_event object immediatedly (see exec_relay_log_event() above) because thd->query now points to the string inside this object. We may keep the pointer to this object in the Relay_log_info: class Relay_log_info { public: ... void set_annotate_event(Annotate_rows_log_event*); Annotate_rows_log_event* get_annotate_event(); void free_annotate_event(); ... private: Annotate_rows_log_event* m_annotate_event; }; The saved Annotate_rows object should be deleted when all corresponding Rows events will be processed: int exec_relay_log_event(THD* thd, Relay_log_info* rli) { ... Log_event *ev= next_event(rli); ... apply_event_and_update_pos(ev, ...); if (rli->get_annotate_event() && is_last_rows_event(ev)) rli->free_annotate_event(); else if (ev->get_type_code() == ANNOTATE_ROWS_EVENT) rli->set_annotate_event((Annotate_rows_log_event*) ev); else if (ev->get_type_code() != FORMAT_DESCRIPTION_EVENT) delete ev; ... } where bool is_last_rows_event(Log_event* ev) { Log_event_type type= ev->get_type_code(); if (IS_ROWS_EVENT_TYPE(type)) { Rows_log_event* rows= (Rows_log_event*)ev; return rows->get_flags(Rows_log_event::STMT_END_F); } return 0; } #define IS_ROWS_EVENT_TYPE(type) ((type) == WRITE_ROWS_EVENT || \ (type) == UPDATE_ROWS_EVENT || \ (type) == DELETE_ROWS_EVENT) 8. General remarks ~~~~~~~~~~~~~~~~~~ Kristian noticed that introducing new log event type should be coordinated somehow with MySQL/Sun: Kristian: The numeric code for this event must be assigned carefully. It should be coordinated with MySQL/Sun, otherwise we can get into a situation where MySQL uses the same numeric code for one event that MariaDB uses for ANNOTATE_ROWS_EVENT, which would make merging the two impossible. Alex: I reserved about 20 numbers not to have possible conflicts with MySQL. Kristian: Still, I think it would be appropriate to send a polite email to internals(a)lists.mysql.com about this and suggesting to reserve the event number. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] SHOW PROFILE enhancements for Windows
by Alex Budovski 04 Feb '10

04 Feb '10

Hi, I've implemented a few things for the Windows port of SHOW PROFILE ( IO read/writes, user/kernel times, page faults) which you may want to consider looking at. There's also a few miscellaneous fixes in there. The commit log has the details. See: https://launchpad.net/~abudovski/maria/robust Thanks.

2 7

[Maria-developers] Rev 7: Added make install and enabled system tables as requested by Monty. in file:///Users/hakan/work/monty_program/mariadb-tools/
by Hakan Kuecuekyilmaz 04 Feb '10

04 Feb '10

At file:///Users/hakan/work/monty_program/mariadb-tools/ ------------------------------------------------------------ revno: 7 revision-id: hakan(a)askmonty.org-20100204011956-u74a4es6oogvd50w parent: hakan(a)askmonty.org-20100115142922-bgc032k180y2i29r committer: Hakan Kuecuekyilmaz <hakan(a)askmonty.org> branch nick: mariadb-tools timestamp: Thu 2010-02-04 02:19:56 +0100 message: Added make install and enabled system tables as requested by Monty. === modified file 'sql-bench/run-sql-bench.sh' --- a/sql-bench/run-sql-bench.sh 2010-01-12 12:42:59 +0000 +++ b/sql-bench/run-sql-bench.sh 2010-02-04 01:19:56 +0000 @@ -51,7 +51,8 @@ # Binaries. # BZR='/usr/local/bin/bzr' -MYSQLADMIN='client/mysqladmin' +#BZR='/usr/bin/bzr' +MYSQLADMIN='bin/mysqladmin' # # Check system. @@ -139,7 +140,9 @@ exit 1 fi - ./configure $MARIADB_CONFIG + # We need --prefix for running make install. Otherwise + # mysql_install_db does not work properly. + ./configure $MARIADB_CONFIG --prefix=${TEMP_DIR}/install if [ $? != 0 ]; then echo "[ERROR]: ./configure $MARIADB_CONFIG failed." echo " Please check your MARIADB_CONFIG in $i." @@ -157,10 +160,24 @@ exit 1 fi + make install + if [ $? != 0 ]; then + echo '[ERROR]: make install.' + echo ' Please check your build logs.' + echo 'Exiting.' + + exit 1 + fi + + cd ${TEMP_DIR}/install + + # Install system tables. + bin/mysql_install_db --no-defaults --basedir=${TEMP_DIR}/install --datadir=${TEMP_DIR}/data + # Start mysqld. MARIADB_SOCKET="${TEMP_DIR}/mysql.sock" MARIADB_OPTIONS="$MARIADB_OPTIONS \ - --datadir=$TEMP_DIR \ + --datadir=${TEMP_DIR}/data \ --tmpdir=$TEMP_DIR \ --socket=$MARIADB_SOCKET" @@ -168,10 +185,10 @@ --socket=$MARIADB_SOCKET" # Determine mysqld version for result file naming. - MARIADB_VERSION=$(sql/mysqld --version | awk '{ print $3 }') + MARIADB_VERSION=$(libexec/mysqld --version | awk '{ print $3 }') SUFFIX="$SUFFIX"-"$MARIADB_VERSION" - sql/mysqld $MARIADB_OPTIONS & + libexec/mysqld $MARIADB_OPTIONS & j=0 STARTED=-1 @@ -196,15 +213,6 @@ exit 1 fi - $MYSQLADMIN $MYSQLADMIN_OPTIONS create test - if [ $? != 0 ]; then - echo '[ERROR]: Create schema test failed.' - echo ' Please check your mysqld error log.' - echo 'Exiting.' - - exit 1 - fi - # Run sql-bench. cd sql-bench COMMENTS="Revision used: $REVISION_ID \ === modified file 'sql-bench/sql-bench-configurations/sql-bench-default-config.inc' --- a/sql-bench/sql-bench-configurations/sql-bench-default-config.inc 2009-12-29 13:42:53 +0000 +++ b/sql-bench/sql-bench-configurations/sql-bench-default-config.inc 2010-02-04 01:19:56 +0000 @@ -16,11 +16,9 @@ # # Options for starting mysqld. # -# Attention: Do not set --datadir, --tmpdir, and --socket. +# Attention: Do not set --datadir, --tmpdir, nor --socket. # -export MARIADB_OPTIONS="--no-defaults \ - --skip-grant-tables \ - --language=./sql/share/english" +export MARIADB_OPTIONS="--no-defaults" # # Options for sql-bench.

1 0

[Maria-developers] Rev 7: Added make install and enabled system tables. in file:///Users/hakan/work/monty_program/mariadb-tools/
by Hakan Kuecuekyilmaz 04 Feb '10

04 Feb '10

At file:///Users/hakan/work/monty_program/mariadb-tools/ ------------------------------------------------------------ revno: 7 revision-id: hakan(a)askmonty.org-20100204011810-rqj1ikb41773p0au parent: hakan(a)askmonty.org-20100115142922-bgc032k180y2i29r committer: Hakan Kuecuekyilmaz <hakan(a)askmonty.org> branch nick: mariadb-tools timestamp: Thu 2010-02-04 02:18:10 +0100 message: Added make install and enabled system tables. === modified file 'sql-bench/run-sql-bench.sh' --- a/sql-bench/run-sql-bench.sh 2010-01-12 12:42:59 +0000 +++ b/sql-bench/run-sql-bench.sh 2010-02-04 01:18:10 +0000 @@ -50,8 +50,9 @@ # # Binaries. # -BZR='/usr/local/bin/bzr' -MYSQLADMIN='client/mysqladmin' +#BZR='/usr/local/bin/bzr' +BZR='/usr/bin/bzr' +MYSQLADMIN='bin/mysqladmin' # # Check system. @@ -139,7 +140,9 @@ exit 1 fi - ./configure $MARIADB_CONFIG + # We need --prefix for running make install. Otherwise + # mysql_install_db does not work properly. + ./configure $MARIADB_CONFIG --prefix=${TEMP_DIR}/install if [ $? != 0 ]; then echo "[ERROR]: ./configure $MARIADB_CONFIG failed." echo " Please check your MARIADB_CONFIG in $i." @@ -157,10 +160,24 @@ exit 1 fi + make install + if [ $? != 0 ]; then + echo '[ERROR]: make install.' + echo ' Please check your build logs.' + echo 'Exiting.' + + exit 1 + fi + + cd ${TEMP_DIR}/install + + # Install system tables. + bin/mysql_install_db --no-defaults --basedir=${TEMP_DIR}/install --datadir=${TEMP_DIR}/data + # Start mysqld. MARIADB_SOCKET="${TEMP_DIR}/mysql.sock" MARIADB_OPTIONS="$MARIADB_OPTIONS \ - --datadir=$TEMP_DIR \ + --datadir=${TEMP_DIR}/data \ --tmpdir=$TEMP_DIR \ --socket=$MARIADB_SOCKET" @@ -168,10 +185,10 @@ --socket=$MARIADB_SOCKET" # Determine mysqld version for result file naming. - MARIADB_VERSION=$(sql/mysqld --version | awk '{ print $3 }') + MARIADB_VERSION=$(libexec/mysqld --version | awk '{ print $3 }') SUFFIX="$SUFFIX"-"$MARIADB_VERSION" - sql/mysqld $MARIADB_OPTIONS & + libexec/mysqld $MARIADB_OPTIONS & j=0 STARTED=-1 @@ -196,15 +213,6 @@ exit 1 fi - $MYSQLADMIN $MYSQLADMIN_OPTIONS create test - if [ $? != 0 ]; then - echo '[ERROR]: Create schema test failed.' - echo ' Please check your mysqld error log.' - echo 'Exiting.' - - exit 1 - fi - # Run sql-bench. cd sql-bench COMMENTS="Revision used: $REVISION_ID \ === modified file 'sql-bench/sql-bench-configurations/sql-bench-default-config.inc' --- a/sql-bench/sql-bench-configurations/sql-bench-default-config.inc 2009-12-29 13:42:53 +0000 +++ b/sql-bench/sql-bench-configurations/sql-bench-default-config.inc 2010-02-04 01:18:10 +0000 @@ -16,11 +16,9 @@ # # Options for starting mysqld. # -# Attention: Do not set --datadir, --tmpdir, and --socket. +# Attention: Do not set --datadir, --tmpdir, nor --socket. # -export MARIADB_OPTIONS="--no-defaults \ - --skip-grant-tables \ - --language=./sql/share/english" +export MARIADB_OPTIONS="--no-defaults" # # Options for sql-bench.

1 0

[Maria-developers] [psergey@askmonty.org: Rev 2750: BUG#31480: Incorrect result for nested subquery when executed via semi join: in file:///home/psergey/dev/maria-5.3-subqueries-r3/]
by Sergey Petrunya 02 Feb '10

02 Feb '10

Hello Igor, I believe that the following patch has the property that item_subselect->update_used_tables() will handle table-bit re-assignments, i.e. it is what we have discussed earlier today. I've also pushed this into 5.3-sj-subqueries tree. ----- Forwarded message from Sergey Petrunya <psergey(a)askmonty.org> ----- From: Sergey Petrunya <psergey(a)askmonty.org> To: maria-developers(a)lists.launchpad.net X-Mailer: mail (GNU Mailutils 1.2) Date: Tue, 2 Feb 2010 23:00:49 +0300 (MSK) Subject: [Maria-developers] Rev 2750: BUG#31480: Incorrect result for nested subquery when executed via semi join: in file:///home/psergey/dev/maria-5.3-subqueries-r3/ At file:///home/psergey/dev/maria-5.3-subqueries-r3/ ------------------------------------------------------------ revno: 2750 revision-id: psergey(a)askmonty.org-20100202200045-13q0nb5dwzm739j6 parent: psergey(a)askmonty.org-20100128134833-9000udjp5wa3tsff committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r3 timestamp: Tue 2010-02-02 23:00:45 +0300 message: BUG#31480: Incorrect result for nested subquery when executed via semi join: A mark-2 fix that can survive FROM subquery handling and has some code unification with table elimination: Each subquery predicate now stores a (flat) list of all references from inside to outside the subquery. We actually store (select, referred_item) pairs which allows Item_subselect::fix_after_pullout() to recalculate subquery predicate's attributes after a broad range of FROM- and IN-subselect flattening operations. === modified file 'mysql-test/r/subselect_sj.result' --- a/mysql-test/r/subselect_sj.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect_sj.result 2010-02-02 20:00:45 +0000 @@ -779,3 +779,48 @@ 1 PRIMARY it2 ALL NULL NULL NULL NULL 20 Using where; End temporary DROP TABLE ot1, it1, it2; # End of BUG#38075 +# +# BUG#31480: Incorrect result for nested subquery when executed via semi join +# +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +id select_type table type possible_keys key key_len ref rows filtered Extra +1 PRIMARY t2 ALL NULL NULL NULL NULL 6 100.00 Start temporary +1 PRIMARY t1 ALL NULL NULL NULL NULL 7 100.00 Using where; End temporary; Using join buffer +3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where +Warnings: +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +show warnings; +Level Code Message +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +a +2 +2 +3 +2 +drop table t1, t2, t3; === modified file 'mysql-test/r/subselect_sj_jcl6.result' --- a/mysql-test/r/subselect_sj_jcl6.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect_sj_jcl6.result 2010-02-02 20:00:45 +0000 @@ -783,6 +783,51 @@ 1 PRIMARY it2 ALL NULL NULL NULL NULL 20 Using where; End temporary; Using join buffer DROP TABLE ot1, it1, it2; # End of BUG#38075 +# +# BUG#31480: Incorrect result for nested subquery when executed via semi join +# +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +id select_type table type possible_keys key key_len ref rows filtered Extra +1 PRIMARY t2 ALL NULL NULL NULL NULL 6 100.00 Start temporary +1 PRIMARY t1 ALL NULL NULL NULL NULL 7 100.00 Using where; End temporary; Using join buffer +3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where +Warnings: +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +show warnings; +Level Code Message +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +a +2 +2 +3 +2 +drop table t1, t2, t3; set join_cache_level=default; show variables like 'join_cache_level'; Variable_name Value === modified file 'mysql-test/t/subselect_sj.test' --- a/mysql-test/t/subselect_sj.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_sj.test 2010-02-02 20:00:45 +0000 @@ -681,3 +681,41 @@ DROP TABLE ot1, it1, it2; --echo # End of BUG#38075 + +--echo # +--echo # BUG#31480: Incorrect result for nested subquery when executed via semi join +--echo # +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); + +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); + +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); + +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); + +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +show warnings; + +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); + +drop table t1, t2, t3; + === modified file 'sql/item.cc' --- a/sql/item.cc 2010-01-17 14:55:08 +0000 +++ b/sql/item.cc 2010-02-02 20:00:45 +0000 @@ -3646,7 +3646,7 @@ substitution) */ -static void mark_as_dependent(THD *thd, SELECT_LEX *last, SELECT_LEX *current, +static bool mark_as_dependent(THD *thd, SELECT_LEX *last, SELECT_LEX *current, Item_ident *resolved_item, Item_ident *mark_item) { @@ -3657,7 +3657,8 @@ /* store pointer on SELECT_LEX from which item is dependent */ if (mark_item) mark_item->depended_from= last; - current->mark_as_dependent(last, resolved_item); + if (current->mark_as_dependent(thd, last, resolved_item)) + return TRUE; if (thd->lex->describe & DESCRIBE_EXTENDED) { push_warning_printf(thd, MYSQL_ERROR::WARN_LEVEL_NOTE, @@ -3667,6 +3668,7 @@ resolved_item->field_name, current->select_number, last->select_number); } + return FALSE; } @@ -4118,6 +4120,7 @@ ((ref_type == REF_ITEM || ref_type == FIELD_ITEM) ? (Item_ident*) (*reference) : 0)); + /* A reference to a view field had been found and we substituted it instead of this Item (find_field_in_tables @@ -6437,7 +6440,7 @@ if (depended_from == new_parent) { *ref= outer_ref; - outer_ref->fix_after_pullout(new_parent, ref); + (*ref)->fix_after_pullout(new_parent, ref); } } === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-01-28 13:48:33 +0000 +++ b/sql/item_subselect.cc 2010-02-02 20:00:45 +0000 @@ -39,8 +39,8 @@ Item_subselect::Item_subselect(): Item_result_field(), value_assigned(0), thd(0), substitution(0), engine(0), old_engine(0), used_tables_cache(0), have_to_be_excluded(0), - const_item_cache(1), in_fix_fields(0), engine_changed(0), changed(0), - is_correlated(FALSE) + const_item_cache(1), inside_first_fix_fields(0), done_first_fix_fields(FALSE), + engine_changed(0), changed(0), is_correlated(FALSE) { with_subselect= 1; reset(); @@ -167,18 +167,23 @@ DBUG_ASSERT(fixed == 0); engine->set_thd((thd= thd_param)); - if (!in_fix_fields) - refers_to.empty(); + if (!done_first_fix_fields) + { + done_first_fix_fields= TRUE; + inside_first_fix_fields= TRUE; + } + eliminated= FALSE; + parent_select= thd_param->lex->current_select; if (check_stack_overrun(thd, STACK_MIN_SIZE, (uchar*)&res)) return TRUE; - in_fix_fields++; res= engine->prepare(); // all transformation is done (used by prepared statements) changed= 1; + inside_first_fix_fields= FALSE; if (!res) { @@ -210,14 +215,12 @@ if (!(*ref)->fixed) ret= (*ref)->fix_fields(thd, ref); thd->where= save_where; - in_fix_fields--; return ret; } // Is it one field subselect? if (engine->cols() > max_columns) { my_error(ER_OPERAND_COLUMNS, MYF(0), 1); - in_fix_fields--; return TRUE; } fix_length_and_dec(); @@ -234,7 +237,6 @@ fixed= 1; err: - in_fix_fields--; thd->where= save_where; return res; } @@ -242,11 +244,12 @@ bool Item_subselect::enumerate_field_refs_processor(uchar *arg) { - List_iterator<Item> it(refers_to); - Item *item; - while ((item= it++)) + List_iterator<Ref_to_outside> it(upper_refs); + Ref_to_outside *upper; + + while ((upper= it++)) { - if (item->walk(&Item::enumerate_field_refs_processor, FALSE, arg)) + if (upper->item->walk(&Item::enumerate_field_refs_processor, FALSE, arg)) return TRUE; } return FALSE; @@ -258,6 +261,115 @@ return FALSE; } + +bool Item_subselect::mark_as_dependent(THD *thd, st_select_lex *select, + Item *item) +{ + if (inside_first_fix_fields) + { + is_correlated= TRUE; + Ref_to_outside *upper; + if (!(upper= new (thd->stmt_arena->mem_root) Ref_to_outside())) + return TRUE; + upper->select= select; + upper->item= item; + if (upper_refs.push_back(upper, thd->stmt_arena->mem_root)) + return TRUE; + } + return FALSE; +} + +/* + Adjust attributes after our parent select has been merged into grandparent + + DESCRIPTION + Subquery is a composite object which may be correlated, that is, it may + have + 1. references to tables of the parent select (i.e. one that has the clause + with the subquery predicate) + 2. references to tables of the grandparent select + 3. references to tables of further ancestors. + + Before the pullout, this item indicates: + - #1 with table bits in used_tables() + - #2 and #3 with OUTER_REF_TABLE_BIT. + + After parent has been merged with grandparent: + - references to parent and grandparent tables should be indicated with + table bits. + - references to greatgrandparent and further ancestors - with + OUTER_REF_TABLE_BIT. +*/ + +void Item_subselect::fix_after_pullout(st_select_lex *new_parent, Item **ref) +{ + recalc_used_tables(new_parent, TRUE); + parent_select= new_parent; +} + + +/* + Recalculate used_tables_cache +*/ + +void Item_subselect::recalc_used_tables(st_select_lex *new_parent, + bool after_pullout) +{ + List_iterator<Ref_to_outside> it(upper_refs); + Ref_to_outside *upper; + + used_tables_cache= 0; + while ((upper= it++)) + { + bool found= FALSE; + /* + Check if + 1. the upper reference refers to the new immediate parent select, or + 2. one of the further ancestors. + + We rely on the fact that the tree of selects is modified by some kind of + 'flattening', i.e. a process where child selects are merged into their + parents. + The merged selects are removed from the select tree but keep pointers to + their parents. + */ + for (st_select_lex *sel= upper->select; sel; sel= sel->outer_select()) + { + /* + If we've reached the new parent select by walking upwards from + reference's original select, this means that the reference is now + referring to the direct parent: + */ + if (sel == new_parent) + { + found= TRUE; + /* + upper->item may be NULL when we've referred to a grouping function, + in which case we don't care about what it's table_map really is, + because item->with_sum_func==1 will ensure correct placement of the + item. + */ + if (upper->item) + { + if (after_pullout) + upper->item->fix_after_pullout(new_parent, &(upper->item)); + upper->item->update_used_tables(); + used_tables_cache |= upper->item->used_tables(); + } + } + } + if (!found) + used_tables_cache|= OUTER_REF_TABLE_BIT; + } + /* + Don't update const_tables_cache yet as we don't yet know which of the + parent's tables are constant. Parent will call update_used_tables() after + he has done const table detection, and that will be our chance to update + const_tables_cache. + */ +} + + bool Item_subselect::walk(Item_processor processor, bool walk_subquery, uchar *argument) { @@ -397,6 +509,7 @@ void Item_subselect::update_used_tables() { + recalc_used_tables(parent_select, FALSE); if (!engine->uncacheable()) { // did all used tables become static? @@ -1843,6 +1956,18 @@ return result || Item_subselect::fix_fields(thd_arg, ref); } +void Item_in_subselect::fix_after_pullout(st_select_lex *new_parent, Item **ref) +{ + left_expr->fix_after_pullout(new_parent, &left_expr); + Item_subselect::fix_after_pullout(new_parent, ref); +} + +void Item_in_subselect::update_used_tables() +{ + Item_subselect::update_used_tables(); + left_expr->update_used_tables(); + used_tables_cache |= left_expr->used_tables(); +} /** Try to create an engine to compute the subselect via materialization, === modified file 'sql/item_subselect.h' --- a/sql/item_subselect.h 2010-01-28 13:48:33 +0000 +++ b/sql/item_subselect.h 2010-02-02 20:00:45 +0000 @@ -67,14 +67,32 @@ bool have_to_be_excluded; /* cache of constant state */ bool const_item_cache; - + + bool inside_first_fix_fields; + bool done_first_fix_fields; public: - /* - References from inside the subquery to the select that this predicate is - in. References to parent selects not included. + /* A reference from inside subquery predicate to somewhere outside of it */ + class Ref_to_outside : public Sql_alloc + { + public: + st_select_lex *select; /* Select where the reference is pointing to */ + /* + What is being referred. This may be NULL when we're referring to an + aggregate function. + */ + Item *item; + }; + /* + References from within this subquery to somewhere outside of it (i.e. to + parent select, grandparent select, etc) */ - List<Item> refers_to; - int in_fix_fields; + List<Ref_to_outside> upper_refs; + st_select_lex *parent_select; + + /* + TRUE<=>Table Elimination has made it redundant to evaluate this select + (and so it is not part of QEP, etc) + */ bool eliminated; /* changed engine indicator */ @@ -117,6 +135,9 @@ return null_value; } bool fix_fields(THD *thd, Item **ref); + bool mark_as_dependent(THD *thd, st_select_lex *select, Item *item); + void fix_after_pullout(st_select_lex *new_parent, Item **ref); + void recalc_used_tables(st_select_lex *new_parent, bool after_pullout); virtual bool exec(); virtual void fix_length_and_dec(); table_map used_tables() const; @@ -396,6 +417,8 @@ bool test_limit(st_select_lex_unit *unit); virtual void print(String *str, enum_query_type query_type); bool fix_fields(THD *thd, Item **ref); + void fix_after_pullout(st_select_lex *new_parent, Item **ref); + void update_used_tables(); bool setup_engine(); bool init_left_expr_cache(); bool is_expensive_processor(uchar *arg); === modified file 'sql/item_sum.cc' --- a/sql/item_sum.cc 2009-10-15 21:38:29 +0000 +++ b/sql/item_sum.cc 2010-02-02 20:00:45 +0000 @@ -350,7 +350,7 @@ sl= sl->master_unit()->outer_select() ) sl->master_unit()->item->with_sum_func= 1; } - thd->lex->current_select->mark_as_dependent(aggr_sel, NULL); + thd->lex->current_select->mark_as_dependent(thd, aggr_sel, NULL); return FALSE; } === modified file 'sql/sql_lex.cc' --- a/sql/sql_lex.cc 2010-01-28 13:48:33 +0000 +++ b/sql/sql_lex.cc 2010-02-02 20:00:45 +0000 @@ -1841,9 +1841,8 @@ 'last' should be reachable from this st_select_lex_node */ -void st_select_lex::mark_as_dependent(st_select_lex *last, Item *dependency) +bool st_select_lex::mark_as_dependent(THD *thd, st_select_lex *last, Item *dependency) { - SELECT_LEX *next_to_last; /* Mark all selects from resolved to 1 before select where was found table as depended (of select where was found table) @@ -1867,12 +1866,15 @@ sl->uncacheable|= UNCACHEABLE_UNITED; } } - next_to_last= s; + + Item_subselect *subquery_expr= s->master_unit()->item; + if (subquery_expr && subquery_expr->mark_as_dependent(thd, last, + dependency)) + return TRUE; } is_correlated= TRUE; this->master_unit()->item->is_correlated= TRUE; - if (dependency) - next_to_last->master_unit()->item->refers_to.push_back(dependency); + return FALSE; } bool st_select_lex_node::set_braces(bool value) { return 1; } === modified file 'sql/sql_lex.h' --- a/sql/sql_lex.h 2010-01-28 13:48:33 +0000 +++ b/sql/sql_lex.h 2010-02-02 20:00:45 +0000 @@ -747,7 +747,7 @@ return master_unit()->return_after_parsing(); } - void mark_as_dependent(st_select_lex *last, Item *dependency); + bool mark_as_dependent(THD *thd, st_select_lex *last, Item *dependency); bool set_braces(bool value); bool inc_in_sum_expr(); === modified file 'sql/sql_select.h' --- a/sql/sql_select.h 2010-01-28 13:48:33 +0000 +++ b/sql/sql_select.h 2010-02-02 20:00:45 +0000 @@ -282,13 +282,11 @@ } bool check_rowid_field() { -/* !!!NB igor: enable the code in this comment after backporting the SJ code if (keep_current_rowid && !used_rowid_fields) { used_rowid_fields= 1; used_fieldlength+= table->file->ref_length; } -*/ return test(used_rowid_fields); } bool is_inner_table_of_semi_join_with_first_match() _______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers(a)lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp ----- End forwarded message ----- -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Rev 2750: BUG#31480: Incorrect result for nested subquery when executed via semi join: in file:///home/psergey/dev/maria-5.3-subqueries-r3/
by Sergey Petrunya 02 Feb '10

02 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r3/ ------------------------------------------------------------ revno: 2750 revision-id: psergey(a)askmonty.org-20100202200045-13q0nb5dwzm739j6 parent: psergey(a)askmonty.org-20100128134833-9000udjp5wa3tsff committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r3 timestamp: Tue 2010-02-02 23:00:45 +0300 message: BUG#31480: Incorrect result for nested subquery when executed via semi join: A mark-2 fix that can survive FROM subquery handling and has some code unification with table elimination: Each subquery predicate now stores a (flat) list of all references from inside to outside the subquery. We actually store (select, referred_item) pairs which allows Item_subselect::fix_after_pullout() to recalculate subquery predicate's attributes after a broad range of FROM- and IN-subselect flattening operations. === modified file 'mysql-test/r/subselect_sj.result' --- a/mysql-test/r/subselect_sj.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect_sj.result 2010-02-02 20:00:45 +0000 @@ -779,3 +779,48 @@ 1 PRIMARY it2 ALL NULL NULL NULL NULL 20 Using where; End temporary DROP TABLE ot1, it1, it2; # End of BUG#38075 +# +# BUG#31480: Incorrect result for nested subquery when executed via semi join +# +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +id select_type table type possible_keys key key_len ref rows filtered Extra +1 PRIMARY t2 ALL NULL NULL NULL NULL 6 100.00 Start temporary +1 PRIMARY t1 ALL NULL NULL NULL NULL 7 100.00 Using where; End temporary; Using join buffer +3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where +Warnings: +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +show warnings; +Level Code Message +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +a +2 +2 +3 +2 +drop table t1, t2, t3; === modified file 'mysql-test/r/subselect_sj_jcl6.result' --- a/mysql-test/r/subselect_sj_jcl6.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect_sj_jcl6.result 2010-02-02 20:00:45 +0000 @@ -783,6 +783,51 @@ 1 PRIMARY it2 ALL NULL NULL NULL NULL 20 Using where; End temporary; Using join buffer DROP TABLE ot1, it1, it2; # End of BUG#38075 +# +# BUG#31480: Incorrect result for nested subquery when executed via semi join +# +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +id select_type table type possible_keys key key_len ref rows filtered Extra +1 PRIMARY t2 ALL NULL NULL NULL NULL 6 100.00 Start temporary +1 PRIMARY t1 ALL NULL NULL NULL NULL 7 100.00 Using where; End temporary; Using join buffer +3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where +Warnings: +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +show warnings; +Level Code Message +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +a +2 +2 +3 +2 +drop table t1, t2, t3; set join_cache_level=default; show variables like 'join_cache_level'; Variable_name Value === modified file 'mysql-test/t/subselect_sj.test' --- a/mysql-test/t/subselect_sj.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_sj.test 2010-02-02 20:00:45 +0000 @@ -681,3 +681,41 @@ DROP TABLE ot1, it1, it2; --echo # End of BUG#38075 + +--echo # +--echo # BUG#31480: Incorrect result for nested subquery when executed via semi join +--echo # +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); + +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); + +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); + +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); + +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +show warnings; + +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); + +drop table t1, t2, t3; + === modified file 'sql/item.cc' --- a/sql/item.cc 2010-01-17 14:55:08 +0000 +++ b/sql/item.cc 2010-02-02 20:00:45 +0000 @@ -3646,7 +3646,7 @@ substitution) */ -static void mark_as_dependent(THD *thd, SELECT_LEX *last, SELECT_LEX *current, +static bool mark_as_dependent(THD *thd, SELECT_LEX *last, SELECT_LEX *current, Item_ident *resolved_item, Item_ident *mark_item) { @@ -3657,7 +3657,8 @@ /* store pointer on SELECT_LEX from which item is dependent */ if (mark_item) mark_item->depended_from= last; - current->mark_as_dependent(last, resolved_item); + if (current->mark_as_dependent(thd, last, resolved_item)) + return TRUE; if (thd->lex->describe & DESCRIBE_EXTENDED) { push_warning_printf(thd, MYSQL_ERROR::WARN_LEVEL_NOTE, @@ -3667,6 +3668,7 @@ resolved_item->field_name, current->select_number, last->select_number); } + return FALSE; } @@ -4118,6 +4120,7 @@ ((ref_type == REF_ITEM || ref_type == FIELD_ITEM) ? (Item_ident*) (*reference) : 0)); + /* A reference to a view field had been found and we substituted it instead of this Item (find_field_in_tables @@ -6437,7 +6440,7 @@ if (depended_from == new_parent) { *ref= outer_ref; - outer_ref->fix_after_pullout(new_parent, ref); + (*ref)->fix_after_pullout(new_parent, ref); } } === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-01-28 13:48:33 +0000 +++ b/sql/item_subselect.cc 2010-02-02 20:00:45 +0000 @@ -39,8 +39,8 @@ Item_subselect::Item_subselect(): Item_result_field(), value_assigned(0), thd(0), substitution(0), engine(0), old_engine(0), used_tables_cache(0), have_to_be_excluded(0), - const_item_cache(1), in_fix_fields(0), engine_changed(0), changed(0), - is_correlated(FALSE) + const_item_cache(1), inside_first_fix_fields(0), done_first_fix_fields(FALSE), + engine_changed(0), changed(0), is_correlated(FALSE) { with_subselect= 1; reset(); @@ -167,18 +167,23 @@ DBUG_ASSERT(fixed == 0); engine->set_thd((thd= thd_param)); - if (!in_fix_fields) - refers_to.empty(); + if (!done_first_fix_fields) + { + done_first_fix_fields= TRUE; + inside_first_fix_fields= TRUE; + } + eliminated= FALSE; + parent_select= thd_param->lex->current_select; if (check_stack_overrun(thd, STACK_MIN_SIZE, (uchar*)&res)) return TRUE; - in_fix_fields++; res= engine->prepare(); // all transformation is done (used by prepared statements) changed= 1; + inside_first_fix_fields= FALSE; if (!res) { @@ -210,14 +215,12 @@ if (!(*ref)->fixed) ret= (*ref)->fix_fields(thd, ref); thd->where= save_where; - in_fix_fields--; return ret; } // Is it one field subselect? if (engine->cols() > max_columns) { my_error(ER_OPERAND_COLUMNS, MYF(0), 1); - in_fix_fields--; return TRUE; } fix_length_and_dec(); @@ -234,7 +237,6 @@ fixed= 1; err: - in_fix_fields--; thd->where= save_where; return res; } @@ -242,11 +244,12 @@ bool Item_subselect::enumerate_field_refs_processor(uchar *arg) { - List_iterator<Item> it(refers_to); - Item *item; - while ((item= it++)) + List_iterator<Ref_to_outside> it(upper_refs); + Ref_to_outside *upper; + + while ((upper= it++)) { - if (item->walk(&Item::enumerate_field_refs_processor, FALSE, arg)) + if (upper->item->walk(&Item::enumerate_field_refs_processor, FALSE, arg)) return TRUE; } return FALSE; @@ -258,6 +261,115 @@ return FALSE; } + +bool Item_subselect::mark_as_dependent(THD *thd, st_select_lex *select, + Item *item) +{ + if (inside_first_fix_fields) + { + is_correlated= TRUE; + Ref_to_outside *upper; + if (!(upper= new (thd->stmt_arena->mem_root) Ref_to_outside())) + return TRUE; + upper->select= select; + upper->item= item; + if (upper_refs.push_back(upper, thd->stmt_arena->mem_root)) + return TRUE; + } + return FALSE; +} + +/* + Adjust attributes after our parent select has been merged into grandparent + + DESCRIPTION + Subquery is a composite object which may be correlated, that is, it may + have + 1. references to tables of the parent select (i.e. one that has the clause + with the subquery predicate) + 2. references to tables of the grandparent select + 3. references to tables of further ancestors. + + Before the pullout, this item indicates: + - #1 with table bits in used_tables() + - #2 and #3 with OUTER_REF_TABLE_BIT. + + After parent has been merged with grandparent: + - references to parent and grandparent tables should be indicated with + table bits. + - references to greatgrandparent and further ancestors - with + OUTER_REF_TABLE_BIT. +*/ + +void Item_subselect::fix_after_pullout(st_select_lex *new_parent, Item **ref) +{ + recalc_used_tables(new_parent, TRUE); + parent_select= new_parent; +} + + +/* + Recalculate used_tables_cache +*/ + +void Item_subselect::recalc_used_tables(st_select_lex *new_parent, + bool after_pullout) +{ + List_iterator<Ref_to_outside> it(upper_refs); + Ref_to_outside *upper; + + used_tables_cache= 0; + while ((upper= it++)) + { + bool found= FALSE; + /* + Check if + 1. the upper reference refers to the new immediate parent select, or + 2. one of the further ancestors. + + We rely on the fact that the tree of selects is modified by some kind of + 'flattening', i.e. a process where child selects are merged into their + parents. + The merged selects are removed from the select tree but keep pointers to + their parents. + */ + for (st_select_lex *sel= upper->select; sel; sel= sel->outer_select()) + { + /* + If we've reached the new parent select by walking upwards from + reference's original select, this means that the reference is now + referring to the direct parent: + */ + if (sel == new_parent) + { + found= TRUE; + /* + upper->item may be NULL when we've referred to a grouping function, + in which case we don't care about what it's table_map really is, + because item->with_sum_func==1 will ensure correct placement of the + item. + */ + if (upper->item) + { + if (after_pullout) + upper->item->fix_after_pullout(new_parent, &(upper->item)); + upper->item->update_used_tables(); + used_tables_cache |= upper->item->used_tables(); + } + } + } + if (!found) + used_tables_cache|= OUTER_REF_TABLE_BIT; + } + /* + Don't update const_tables_cache yet as we don't yet know which of the + parent's tables are constant. Parent will call update_used_tables() after + he has done const table detection, and that will be our chance to update + const_tables_cache. + */ +} + + bool Item_subselect::walk(Item_processor processor, bool walk_subquery, uchar *argument) { @@ -397,6 +509,7 @@ void Item_subselect::update_used_tables() { + recalc_used_tables(parent_select, FALSE); if (!engine->uncacheable()) { // did all used tables become static? @@ -1843,6 +1956,18 @@ return result || Item_subselect::fix_fields(thd_arg, ref); } +void Item_in_subselect::fix_after_pullout(st_select_lex *new_parent, Item **ref) +{ + left_expr->fix_after_pullout(new_parent, &left_expr); + Item_subselect::fix_after_pullout(new_parent, ref); +} + +void Item_in_subselect::update_used_tables() +{ + Item_subselect::update_used_tables(); + left_expr->update_used_tables(); + used_tables_cache |= left_expr->used_tables(); +} /** Try to create an engine to compute the subselect via materialization, === modified file 'sql/item_subselect.h' --- a/sql/item_subselect.h 2010-01-28 13:48:33 +0000 +++ b/sql/item_subselect.h 2010-02-02 20:00:45 +0000 @@ -67,14 +67,32 @@ bool have_to_be_excluded; /* cache of constant state */ bool const_item_cache; - + + bool inside_first_fix_fields; + bool done_first_fix_fields; public: - /* - References from inside the subquery to the select that this predicate is - in. References to parent selects not included. + /* A reference from inside subquery predicate to somewhere outside of it */ + class Ref_to_outside : public Sql_alloc + { + public: + st_select_lex *select; /* Select where the reference is pointing to */ + /* + What is being referred. This may be NULL when we're referring to an + aggregate function. + */ + Item *item; + }; + /* + References from within this subquery to somewhere outside of it (i.e. to + parent select, grandparent select, etc) */ - List<Item> refers_to; - int in_fix_fields; + List<Ref_to_outside> upper_refs; + st_select_lex *parent_select; + + /* + TRUE<=>Table Elimination has made it redundant to evaluate this select + (and so it is not part of QEP, etc) + */ bool eliminated; /* changed engine indicator */ @@ -117,6 +135,9 @@ return null_value; } bool fix_fields(THD *thd, Item **ref); + bool mark_as_dependent(THD *thd, st_select_lex *select, Item *item); + void fix_after_pullout(st_select_lex *new_parent, Item **ref); + void recalc_used_tables(st_select_lex *new_parent, bool after_pullout); virtual bool exec(); virtual void fix_length_and_dec(); table_map used_tables() const; @@ -396,6 +417,8 @@ bool test_limit(st_select_lex_unit *unit); virtual void print(String *str, enum_query_type query_type); bool fix_fields(THD *thd, Item **ref); + void fix_after_pullout(st_select_lex *new_parent, Item **ref); + void update_used_tables(); bool setup_engine(); bool init_left_expr_cache(); bool is_expensive_processor(uchar *arg); === modified file 'sql/item_sum.cc' --- a/sql/item_sum.cc 2009-10-15 21:38:29 +0000 +++ b/sql/item_sum.cc 2010-02-02 20:00:45 +0000 @@ -350,7 +350,7 @@ sl= sl->master_unit()->outer_select() ) sl->master_unit()->item->with_sum_func= 1; } - thd->lex->current_select->mark_as_dependent(aggr_sel, NULL); + thd->lex->current_select->mark_as_dependent(thd, aggr_sel, NULL); return FALSE; } === modified file 'sql/sql_lex.cc' --- a/sql/sql_lex.cc 2010-01-28 13:48:33 +0000 +++ b/sql/sql_lex.cc 2010-02-02 20:00:45 +0000 @@ -1841,9 +1841,8 @@ 'last' should be reachable from this st_select_lex_node */ -void st_select_lex::mark_as_dependent(st_select_lex *last, Item *dependency) +bool st_select_lex::mark_as_dependent(THD *thd, st_select_lex *last, Item *dependency) { - SELECT_LEX *next_to_last; /* Mark all selects from resolved to 1 before select where was found table as depended (of select where was found table) @@ -1867,12 +1866,15 @@ sl->uncacheable|= UNCACHEABLE_UNITED; } } - next_to_last= s; + + Item_subselect *subquery_expr= s->master_unit()->item; + if (subquery_expr && subquery_expr->mark_as_dependent(thd, last, + dependency)) + return TRUE; } is_correlated= TRUE; this->master_unit()->item->is_correlated= TRUE; - if (dependency) - next_to_last->master_unit()->item->refers_to.push_back(dependency); + return FALSE; } bool st_select_lex_node::set_braces(bool value) { return 1; } === modified file 'sql/sql_lex.h' --- a/sql/sql_lex.h 2010-01-28 13:48:33 +0000 +++ b/sql/sql_lex.h 2010-02-02 20:00:45 +0000 @@ -747,7 +747,7 @@ return master_unit()->return_after_parsing(); } - void mark_as_dependent(st_select_lex *last, Item *dependency); + bool mark_as_dependent(THD *thd, st_select_lex *last, Item *dependency); bool set_braces(bool value); bool inc_in_sum_expr(); === modified file 'sql/sql_select.h' --- a/sql/sql_select.h 2010-01-28 13:48:33 +0000 +++ b/sql/sql_select.h 2010-02-02 20:00:45 +0000 @@ -282,13 +282,11 @@ } bool check_rowid_field() { -/* !!!NB igor: enable the code in this comment after backporting the SJ code if (keep_current_rowid && !used_rowid_fields) { used_rowid_fields= 1; used_fieldlength+= table->file->ref_length; } -*/ return test(used_rowid_fields); } bool is_inner_table_of_semi_join_with_first_match()

1 0

Re: [Maria-developers] Ideas for improving MariaDB/MySQL replication
by Alex Yurchenko 02 Feb '10

02 Feb '10

Hi! On Mon, 1 Feb 2010 11:06:22 +0100, Sergei Golubchik <serg(a)askmonty.org> wrote: > Hi, Alex! > > On Jan 27, Alex Yurchenko wrote: >> >> I'll take this opportunity to put forth some theory behind the global >> transaction IDs as we see it at Codership. >> >> 1. We have an abstract set of data subject to replication/logging. It >> can be a whole database, a schema, a table, a row. Lets call it a >> Replication Set (RS). >> >> 2. RS is undergoing changes in time which can be represented as a >> series of atomic changes. Let's call it RS History. That it is a >> _series_ is trivial but important - otherwise we can't reproduce >> historical RS state evolution. Each RS change is represented by a >> changeset. Since it is a series, RS changesets can be enumerated with >> a sequence of natural numbers without gaps within a given RS History. >> Here comes the first component of a global transaction ID: sequence >> number (seqno). > > Why should it be a sequence of natural numbers without gaps ? 1) Well, to begin with I didn't say that it "should", I said that it "can" ;). I was proposing a definition, so it depends on what we want to achieve. Obviously, "gaplessness" is a useful requirement. It allows global transaction IDs to not only be globally unique, but also unambiguously indicates the position of a changeset in the history of changes. By relaxing it we are loosing a natural way to check for gaps in the stream of events. E.g. you can't say if you can concatenate binlog files one of which ends at 10 and another starts at 12. When a node at position 100 joins a cluster, how will it know that the next event to process is 113? "No gaps" requirement allows us to take a single changeset, carry it around all we like and then apply consistently elsewhere without the need of any other context. Indeed, there are other ways to address this, but gapless seqno is obviously the simplest of them all. 2) It is not a limiting requirement at all. If we agree, that RS undergoes a _series_ of changes - change 1, change 2, change 3, change 4, change 5, etc...- nothing prevents us from enumerating them without gaps. Moreover, why and according to what algorithm are you going to introduce gaps in sequence numbers? I mean, there is additional work to be done to achieve gaps when enumerating a sequence. 3) It simplifies testing and debugging. Robert from Continuent also raised this question, but I didn't have a chance to respond to it in time. No, gapless seqnos is not a caprice of Galera developers ;), it is just a proposal for global trx ID based on our experience. Galera can maintain it internally anyways, but I believe everyone would benefit from it. >> 3. However there can be more than one RS. Moreover, the same RS can >> end up in different clusters and undergo different changes. So, to >> achieve truly global unambiguity each changeset, in addition to seqno, >> should be marked with a RS History ID. Obviously seqnos from different >> histories are logically incomparable. Therefore RS History ID can be >> any globally unique identifier, with no need for < or > operations. >> This is the second component of global transaction ID. > > Assuming we want to replicate just one table, do you mean that in a > replication cluster this Logical Table is a Replication Set, that is > all copies of this table on all nodes belong to the same RS ? Yes, with a small clarification, that strictly speaking, copies of the table don't _belong_ to RS, they _are_ RS, that is in your example Replication Set consists of a single table, not of multiple copies the table. It is probably more correct to speak about RS replicas on nodes, rather than copies of the table. Regards, Alex -- Alexey Yurchenko, Codership Oy, www.codership.com Skype: alexey.yurchenko, Phone: +358-400-516-011

1 0

[Maria-developers] Rev 2750: BUG#31480: Incorrect result for nested subquery when executed via semi join: in file:///home/psergey/dev/maria-5.3-subqueries-r3/
by Sergey Petrunya 02 Feb '10

02 Feb '10

At file:///home/psergey/dev/maria-5.3-subqueries-r3/ ------------------------------------------------------------ revno: 2750 revision-id: psergey(a)askmonty.org-20100202121522-fw6ckzcdfis4d21p parent: psergey(a)askmonty.org-20100128134833-9000udjp5wa3tsff committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r3 timestamp: Tue 2010-02-02 15:15:22 +0300 message: BUG#31480: Incorrect result for nested subquery when executed via semi join: A mark-2 fix that can survive FROM subquery handling and has some code unification with table elimination: Each subquery predicate now stores a (flat) list of all references from inside to outside the subquery. We actually store (select, referred_item) pairs which allows Item_subselect::fix_after_pullout() to recalculate subquery predicate's attributes after a broad range of FROM- and IN-subselect flattening operations. === modified file 'mysql-test/r/subselect_sj.result' --- a/mysql-test/r/subselect_sj.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect_sj.result 2010-02-02 12:15:22 +0000 @@ -779,3 +779,48 @@ 1 PRIMARY it2 ALL NULL NULL NULL NULL 20 Using where; End temporary DROP TABLE ot1, it1, it2; # End of BUG#38075 +# +# BUG#31480: Incorrect result for nested subquery when executed via semi join +# +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +id select_type table type possible_keys key key_len ref rows filtered Extra +1 PRIMARY t2 ALL NULL NULL NULL NULL 6 100.00 Start temporary +1 PRIMARY t1 ALL NULL NULL NULL NULL 7 100.00 Using where; End temporary; Using join buffer +3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where +Warnings: +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +show warnings; +Level Code Message +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +a +2 +2 +3 +2 +drop table t1, t2, t3; === modified file 'mysql-test/r/subselect_sj_jcl6.result' --- a/mysql-test/r/subselect_sj_jcl6.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect_sj_jcl6.result 2010-02-02 12:15:22 +0000 @@ -783,6 +783,51 @@ 1 PRIMARY it2 ALL NULL NULL NULL NULL 20 Using where; End temporary; Using join buffer DROP TABLE ot1, it1, it2; # End of BUG#38075 +# +# BUG#31480: Incorrect result for nested subquery when executed via semi join +# +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +id select_type table type possible_keys key key_len ref rows filtered Extra +1 PRIMARY t2 ALL NULL NULL NULL NULL 6 100.00 Start temporary +1 PRIMARY t1 ALL NULL NULL NULL NULL 7 100.00 Using where; End temporary; Using join buffer +3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where +Warnings: +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +show warnings; +Level Code Message +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +a +2 +2 +3 +2 +drop table t1, t2, t3; set join_cache_level=default; show variables like 'join_cache_level'; Variable_name Value === modified file 'mysql-test/t/subselect_sj.test' --- a/mysql-test/t/subselect_sj.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_sj.test 2010-02-02 12:15:22 +0000 @@ -681,3 +681,41 @@ DROP TABLE ot1, it1, it2; --echo # End of BUG#38075 + +--echo # +--echo # BUG#31480: Incorrect result for nested subquery when executed via semi join +--echo # +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); + +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); + +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); + +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); + +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +show warnings; + +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); + +drop table t1, t2, t3; + === modified file 'sql/item.cc' --- a/sql/item.cc 2010-01-17 14:55:08 +0000 +++ b/sql/item.cc 2010-02-02 12:15:22 +0000 @@ -3646,7 +3646,7 @@ substitution) */ -static void mark_as_dependent(THD *thd, SELECT_LEX *last, SELECT_LEX *current, +static bool mark_as_dependent(THD *thd, SELECT_LEX *last, SELECT_LEX *current, Item_ident *resolved_item, Item_ident *mark_item) { @@ -3657,7 +3657,8 @@ /* store pointer on SELECT_LEX from which item is dependent */ if (mark_item) mark_item->depended_from= last; - current->mark_as_dependent(last, resolved_item); + if (current->mark_as_dependent(thd, last, resolved_item)) + return TRUE; if (thd->lex->describe & DESCRIBE_EXTENDED) { push_warning_printf(thd, MYSQL_ERROR::WARN_LEVEL_NOTE, @@ -3667,6 +3668,7 @@ resolved_item->field_name, current->select_number, last->select_number); } + return FALSE; } @@ -4118,6 +4120,7 @@ ((ref_type == REF_ITEM || ref_type == FIELD_ITEM) ? (Item_ident*) (*reference) : 0)); + /* A reference to a view field had been found and we substituted it instead of this Item (find_field_in_tables @@ -6437,7 +6440,7 @@ if (depended_from == new_parent) { *ref= outer_ref; - outer_ref->fix_after_pullout(new_parent, ref); + (*ref)->fix_after_pullout(new_parent, ref); } } === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-01-28 13:48:33 +0000 +++ b/sql/item_subselect.cc 2010-02-02 12:15:22 +0000 @@ -39,8 +39,8 @@ Item_subselect::Item_subselect(): Item_result_field(), value_assigned(0), thd(0), substitution(0), engine(0), old_engine(0), used_tables_cache(0), have_to_be_excluded(0), - const_item_cache(1), in_fix_fields(0), engine_changed(0), changed(0), - is_correlated(FALSE) + const_item_cache(1), inside_first_fix_fields(0), done_first_fix_fields(FALSE), + engine_changed(0), changed(0), is_correlated(FALSE) { with_subselect= 1; reset(); @@ -167,18 +167,22 @@ DBUG_ASSERT(fixed == 0); engine->set_thd((thd= thd_param)); - if (!in_fix_fields) - refers_to.empty(); + if (!done_first_fix_fields) + { + done_first_fix_fields= TRUE; + inside_first_fix_fields= TRUE; + } + eliminated= FALSE; if (check_stack_overrun(thd, STACK_MIN_SIZE, (uchar*)&res)) return TRUE; - in_fix_fields++; res= engine->prepare(); // all transformation is done (used by prepared statements) changed= 1; + inside_first_fix_fields= FALSE; if (!res) { @@ -210,14 +214,12 @@ if (!(*ref)->fixed) ret= (*ref)->fix_fields(thd, ref); thd->where= save_where; - in_fix_fields--; return ret; } // Is it one field subselect? if (engine->cols() > max_columns) { my_error(ER_OPERAND_COLUMNS, MYF(0), 1); - in_fix_fields--; return TRUE; } fix_length_and_dec(); @@ -234,7 +236,6 @@ fixed= 1; err: - in_fix_fields--; thd->where= save_where; return res; } @@ -242,11 +243,12 @@ bool Item_subselect::enumerate_field_refs_processor(uchar *arg) { - List_iterator<Item> it(refers_to); - Item *item; - while ((item= it++)) + List_iterator<Ref_to_outside> it(upper_refs); + Ref_to_outside *upper; + + while ((upper= it++)) { - if (item->walk(&Item::enumerate_field_refs_processor, FALSE, arg)) + if (upper->item->walk(&Item::enumerate_field_refs_processor, FALSE, arg)) return TRUE; } return FALSE; @@ -258,6 +260,102 @@ return FALSE; } + +bool Item_subselect::mark_as_dependent(THD *thd, st_select_lex *select, + Item *item) +{ + if (inside_first_fix_fields) + { + is_correlated= TRUE; + Ref_to_outside *upper; + if (!(upper= new (thd->stmt_arena->mem_root) Ref_to_outside())) + return TRUE; + upper->select= select; + upper->item= item; + if (upper_refs.push_back(upper, thd->stmt_arena->mem_root)) + return TRUE; + } + return FALSE; +} + +/* + Adjust attributes after our parent select has been merged into grandparent + + DESCRIPTION + Subquery is a composite object which may be correlated, that is, it may + have + 1. references to tables of the parent select (i.e. one that has the clause + with the subquery predicate) + 2. references to tables of the grandparent select + 3. references to tables of further ancestors. + + Before the pullout, this item indicates: + - #1 with table bits in used_tables() + - #2 and #3 with OUTER_REF_TABLE_BIT. + + After parent has been merged with grandparent: + - references to parent and grandparent tables should be indicated with + table bits. + - references to greatgrandparent and further ancestors - with + OUTER_REF_TABLE_BIT. +*/ + +void Item_subselect::fix_after_pullout(st_select_lex *new_parent, Item **ref) +{ + List_iterator<Ref_to_outside> it(upper_refs); + Ref_to_outside *upper; + + used_tables_cache= 0; + while ((upper= it++)) + { + bool found= FALSE; + /* + Check if + 1. the upper reference refers to the new immediate parent select, or + 2. one of the further ancestors. + + We rely on the fact that the tree of selects is modified by some kind of + 'flattening', i.e. a process where child selects are merged into their + parents. + The merged selects are removed from the select tree but keep pointers to + their parents. + */ + for (st_select_lex *sel= upper->select; sel; sel= sel->outer_select()) + { + /* + If we've reached the new parent select by walking upwards from + reference's original select, this means that the reference is now + referring to the direct parent: + */ + if (sel == new_parent) + { + found= TRUE; + /* + upper->item may be NULL when we've referred to a grouping function, + in which case we don't care about what it's table_map really is, + because item->with_sum_func would ensure correct placement of the + item. + */ + if (upper->item) + { + upper->item->update_used_tables(); + upper->item->fix_after_pullout(new_parent, &(upper->item)); + used_tables_cache |= upper->item->used_tables(); + } + } + } + if (!found) + used_tables_cache|= OUTER_REF_TABLE_BIT; + } + /* + Don't update const_tables_cache yet as we don't yet know which of the + parent's tables are constant. Parent will call update_used_tables() after + he has done const table detection, and that will be our chance to update + const_tables_cache. + */ +} + + bool Item_subselect::walk(Item_processor processor, bool walk_subquery, uchar *argument) { === modified file 'sql/item_subselect.h' --- a/sql/item_subselect.h 2010-01-28 13:48:33 +0000 +++ b/sql/item_subselect.h 2010-02-02 12:15:22 +0000 @@ -67,14 +67,31 @@ bool have_to_be_excluded; /* cache of constant state */ bool const_item_cache; - + + bool inside_first_fix_fields; + bool done_first_fix_fields; public: - /* - References from inside the subquery to the select that this predicate is - in. References to parent selects not included. + /* A reference from inside subquery predicate to somewhere outside of it */ + class Ref_to_outside : public Sql_alloc + { + public: + st_select_lex *select; /* Select where the reference is pointing to */ + /* + What is being referred. This may be NULL when we're referring to an + aggregate function. + */ + Item *item; + }; + /* + References from within this subquery to somewhere outside of it (i.e. to + parent select, grandparent select, etc) */ - List<Item> refers_to; - int in_fix_fields; + List<Ref_to_outside> upper_refs; + + /* + TRUE<=>Table Elimination has made it redundant to evaluate this select + (and so it is not part of QEP, etc) + */ bool eliminated; /* changed engine indicator */ @@ -117,6 +134,8 @@ return null_value; } bool fix_fields(THD *thd, Item **ref); + bool mark_as_dependent(THD *thd, st_select_lex *select, Item *item); + void fix_after_pullout(st_select_lex *new_parent, Item **ref); virtual bool exec(); virtual void fix_length_and_dec(); table_map used_tables() const; === modified file 'sql/item_sum.cc' --- a/sql/item_sum.cc 2009-10-15 21:38:29 +0000 +++ b/sql/item_sum.cc 2010-02-02 12:15:22 +0000 @@ -350,7 +350,7 @@ sl= sl->master_unit()->outer_select() ) sl->master_unit()->item->with_sum_func= 1; } - thd->lex->current_select->mark_as_dependent(aggr_sel, NULL); + thd->lex->current_select->mark_as_dependent(thd, aggr_sel, NULL); return FALSE; } === modified file 'sql/sql_lex.cc' --- a/sql/sql_lex.cc 2010-01-28 13:48:33 +0000 +++ b/sql/sql_lex.cc 2010-02-02 12:15:22 +0000 @@ -1841,9 +1841,8 @@ 'last' should be reachable from this st_select_lex_node */ -void st_select_lex::mark_as_dependent(st_select_lex *last, Item *dependency) +bool st_select_lex::mark_as_dependent(THD *thd, st_select_lex *last, Item *dependency) { - SELECT_LEX *next_to_last; /* Mark all selects from resolved to 1 before select where was found table as depended (of select where was found table) @@ -1867,12 +1866,15 @@ sl->uncacheable|= UNCACHEABLE_UNITED; } } - next_to_last= s; + + Item_subselect *subquery_expr= s->master_unit()->item; + if (subquery_expr && subquery_expr->mark_as_dependent(thd, last, + dependency)) + return TRUE; } is_correlated= TRUE; this->master_unit()->item->is_correlated= TRUE; - if (dependency) - next_to_last->master_unit()->item->refers_to.push_back(dependency); + return FALSE; } bool st_select_lex_node::set_braces(bool value) { return 1; } === modified file 'sql/sql_lex.h' --- a/sql/sql_lex.h 2010-01-28 13:48:33 +0000 +++ b/sql/sql_lex.h 2010-02-02 12:15:22 +0000 @@ -747,7 +747,7 @@ return master_unit()->return_after_parsing(); } - void mark_as_dependent(st_select_lex *last, Item *dependency); + bool mark_as_dependent(THD *thd, st_select_lex *last, Item *dependency); bool set_braces(bool value); bool inc_in_sum_expr();

1 0

Re: [Maria-developers] really need call my_thread_init() in libmysqlclient_r?
by xiaobing jiang 02 Feb '10

02 Feb '10

hi, Sergei: > You cannot easily remove my_thread_init/my_thread_end, besides debugging > THR_KEY_mysys is also needed for my_errno, which is uses all over mysys. > thanks, but have another question: what about in one thread, I call mysql_init() twice to create 2 connection. they will share the THR_KEY_mysys thread local variable, right? the two connections will share my_error, is this will cause some 'bug'?

1 1

[Maria-developers] bzr commit into Mariadb 5.2, with Maria 2.0:maria/5.2 branch (knielsen:2739)
by knielsen＠knielsen-hq.org 01 Feb '10

01 Feb '10

#At lp:maria/5.2 2739 knielsen(a)knielsen-hq.org 2010-02-01 Fix bad 5.1->5.2 merge: timezone must now be set explicitly for test cases that depend on it. added: mysql-test/suite/binlog/t/binlog_row_mysqlbinlog_options-master.opt === added file 'mysql-test/suite/binlog/t/binlog_row_mysqlbinlog_options-master.opt' --- a/mysql-test/suite/binlog/t/binlog_row_mysqlbinlog_options-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/binlog/t/binlog_row_mysqlbinlog_options-master.opt 2010-02-01 19:05:19 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3

1 0

Re: [Maria-developers] Support building with system libevent
by Kristian Nielsen 01 Feb '10

01 Feb '10

Sergei Golubchik <serg(a)askmonty.org> writes: > Surely, it is possible to install a libevent 1.4 deb on Debian <= 5 and > Ubuntu <= 9.04, right ? They are not in the default package repository - > is that a big deal ? MariaDB is not either. Possible yes, but only if one knows where to obtain such a .deb package. I don't know anywhere such a package is available for older Ubuntu/Debian? (One could try installing the package from a newer Ubuntu or Debian unstable, may or may not work. But in general in .deb world people expect to not have to manually go hunt for dependencies. Maybe .rpm is different?) MariaDB is in the "default" repository list if users put the OurDelta repository there, which is the recommended way to install .deb. - Kristian.

1 0

[Maria-developers] Rev 3769: MWL#68 Subquery optimization: Efficient NOT IN execution with NULLs in file:///home/tsk/mprog/src/mysql-6.0-mwl68/
by timour＠askmonty.org 01 Feb '10

01 Feb '10

At file:///home/tsk/mprog/src/mysql-6.0-mwl68/ ------------------------------------------------------------ revno: 3769 revision-id: timour(a)askmonty.org-20100201120948-mdt7gtwcz50q1dzp parent: timour(a)sun.com-20100122161805-8lgrisqabrlvc3nc committer: timour(a)askmonty.org branch nick: mysql-6.0-mwl68 timestamp: Mon 2010-02-01 14:09:48 +0200 message: MWL#68 Subquery optimization: Efficient NOT IN execution with NULLs Completed main coding of partial matching. The code compiles, but cannot run. Changes compared to the previous commit: - Completed creation and initialization of all objects needed for partial matching. - Adjusted the interfaces of multiple methods in order to pass the correct information needed for creation/initialization. - Added comparion functions needed for binary search and index sorting. - Fixed binary search in the value index. - Exposed the Arg_comparator of comparison predicates. Diff too large for email (1030 lines, the limit is 1000).

1 0

[Maria-developers] bzr commit into file:///home/tsk/mprog/src/mysql-6.0-mwl68/ branch (timour:3769)
by timour＠askmonty.org 01 Feb '10

01 Feb '10

#At file:///home/tsk/mprog/src/mysql-6.0-mwl68/ based on revid:timour@sun.com-20100122161805-8lgrisqabrlvc3nc 3769 timour(a)askmonty.org 2010-02-01 MWL#68 Subquery optimization: Efficient NOT IN execution with NULLs Completed main coding of partial matching. The code compiles, but cannot run. Changes compared to the previous commit: - Completed creation and initialization of all objects needed for partial matching. - Adjusted the interfaces of multiple methods in order to pass the correct information needed for creation/initialization. - Added comparion functions needed for binary search and index sorting. - Fixed binary search in the value index. - Exposed the Arg_comparator of comparison predicates. @ sql/item_cmpfunc.h Expose the Arg_comparator of a comparison predicate so that it is possible to get the comparison result. @ sql/item_subselect.cc - Completed creation and initialization of all objects needed for partial matching. - Adjusted the interfaces of multiple methods in order to pass the correct information needed for creation/initialization. - Added comparion functions needed for binary search and index sorting. - Fixed binary search in the value index. @ sql/item_subselect.h Completed creation and initialization of all objects needed for partial matching. @ sql/sql_class.h - added accessors for NULL statistics modified: sql/item_cmpfunc.h sql/item_subselect.cc sql/item_subselect.h sql/sql_class.h === modified file 'sql/item_cmpfunc.h' --- a/sql/item_cmpfunc.h 2009-12-04 07:48:05 +0000 +++ b/sql/item_cmpfunc.h 2010-02-01 12:09:48 +0000 @@ -355,6 +355,7 @@ public: CHARSET_INFO *compare_collation() { return cmp.cmp_collation.collation; } uint decimal_precision() const { return 1; } void top_level_item() { abort_on_null= TRUE; } + Arg_comparator *get_comparator() { return &cmp; } friend class Arg_comparator; }; === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-01-22 16:18:05 +0000 +++ b/sql/item_subselect.cc 2010-02-01 12:09:48 +0000 @@ -3102,32 +3102,21 @@ void subselect_hash_sj_engine::set_strat outer_col= item_in->left_expr->element_index(i); inner_col= inner_col_it++; - if (!inner_col->maybe_null) - { - if (!outer_col->maybe_null) - { - non_null_outer_cols.push_back(outer_col); - non_null_key_parts |= 1 << i; - } - else - { - non_null_late_key_parts |= 1 << i; - ++count_non_null_late_cols; - } - } + if (!inner_col->maybe_null && !outer_col->maybe_null) + bitmap_set_bit(&non_null_key_parts, i); else { - partial_match_key_parts |= 1 << i; - ++count_partial_match_cols; + bitmap_set_bit(&partial_match_key_parts, i); + ++count_partial_match_columns; } } } /* If no column contains NULLs use regular hash index lookups. */ - if (!(non_null_late_key_parts || partial_match_key_parts)) - strategy= COMPLETE_MATCH; - else + if (count_partial_match_columns) strategy= PARTIAL_MATCH; + else + strategy= COMPLETE_MATCH; DBUG_VOID_RETURN; } @@ -3145,7 +3134,6 @@ void subselect_hash_sj_engine::set_strat void subselect_hash_sj_engine::set_strategy_using_data() { Item_in_subselect *item_in= (Item_in_subselect *) item; - key_part_map cur_col= 0; select_materialize_with_stats *result_sink= (select_materialize_with_stats *) result; @@ -3154,77 +3142,45 @@ void subselect_hash_sj_engine::set_strat /* Call this procedure only if already selected partial matching. */ DBUG_ASSERT(strategy == PARTIAL_MATCH); - /* - TODO: uncomment after enabling index creation after materialization. - List_iterator<Item> inner_col_it(*item_in->unit->get_unit_column_types()); - Item *inner_col, *outer_col; - */ - for (uint i= 0; i < item_in->left_expr->cols(); i++) { - /* - TODO: uncomment after enabling index creation after materialization. - outer_col= item_in->left_expr->element_index(i); - inner_col= inner_col_it++; - */ - cur_col= 1 << i; - - if (!(cur_col & partial_match_key_parts)) + if (!bitmap_is_set(&partial_match_key_parts, i)) continue; - if (result_sink->get_column_null_count(i) == - tmp_table->file->stats.records) + if (result_sink->get_null_count_of_col(i) == 0) { - /* Column i of the temp table consists only of NULLs. */ - --count_partial_match_cols; - inner_partial_match= TRUE; - partial_match_key_parts &= ~cur_col; /* unset bit 'i' */ - } - else if (result_sink->get_column_null_count(i) == 0) - { - --count_partial_match_cols; - partial_match_key_parts &= ~cur_col; - /* - TODO - Column i of the temp table doesn't contain any NULLs. Currently we - cannot create/alter an index on an already populated internal - temporary table. As a result even if we detect that a column should - belong to the NON_NULL index, it is too late to alter that index. The - only thing we can do is change it from PARTIAL_MATCH to NON_NULL_LATE, - thus removing the "OR NULL" predicate during lookup. Once this - limitation is removed, use the commented code below instead of the - following two lines. - */ - ++count_non_null_late_cols; - non_null_late_key_parts |= cur_col; - /* - if (!inner_col->maybe_null) - { - non_null_key_parts |= cur_col; - non_null_outer_cols.push_back(outer_col); - } - else - { - ++count_non_null_late_cols; - non_null_late_key_parts |= cur_col; - } - */ + bitmap_clear_bit(&partial_match_key_parts, i); + bitmap_set_bit(&non_null_key_parts, i); + --count_partial_match_columns; } } - /* - if (non_null_outer_cols.elements > max number of key parts) - DBUG_RETURN(TRUE); - */ - /* If no column contains NULLs use regular hash index lookups. */ - if (!(non_null_late_key_parts || partial_match_key_parts)) + if (!count_partial_match_columns) strategy= COMPLETE_MATCH; DBUG_VOID_RETURN; } +/* + Initialize a MY_BITMAP with a buffer allocated on the current + memory root. +*/ + +static my_bool +bitmap_init_memroot(MY_BITMAP *map, uint n_bits, MEM_ROOT *mem_root) +{ + my_bitmap_map *bitmap_buf; + + if (!(bitmap_buf= (my_bitmap_map*) alloc_root(mem_root, + bitmap_buffer_size(n_bits))) || + bitmap_init(map, bitmap_buf, n_bits, FALSE)) + return TRUE; + return FALSE; +} + + /** Create all structures needed for IN execution that can live between PS reexecution. @@ -3253,6 +3209,11 @@ bool subselect_hash_sj_engine::init_perm DBUG_ENTER("subselect_hash_sj_engine::init_permanent"); + if (!(bitmap_init_memroot(&non_null_key_parts, tmp_columns->elements, + thd->mem_root)) || + !(bitmap_init_memroot(&partial_match_key_parts, tmp_columns->elements, + thd->mem_root))) + DBUG_RETURN(TRUE); set_strategy_using_schema(); /* @@ -3261,6 +3222,10 @@ bool subselect_hash_sj_engine::init_perm managed (created/filled/etc) internally by the interceptor. */ /* + TODO: + Select a more efficient result sink when we know there is no need to collect + data statistics. + if (strategy == COMPLETE_MATCH) { if (!(result= new select_union)) @@ -3314,13 +3279,6 @@ bool subselect_hash_sj_engine::init_perm if (make_semi_join_conds()) DBUG_RETURN(TRUE); - /* - A complete match is the best we can get, so we can immediately - create the enginte to be used for lookup. - */ - if (strategy == COMPLETE_MATCH && - !(lookup_engine= make_unique_engine())) - DBUG_RETURN(TRUE); DBUG_RETURN(FALSE); } @@ -3582,7 +3540,7 @@ int subselect_hash_sj_engine::exec() set_strategy_using_data(); /* A unique_engine is used both for complete and partial matching. */ - if (!lookup_engine && !(lookup_engine= make_unique_engine())) + if (!(lookup_engine= make_unique_engine())) { res= 1; goto err; @@ -3590,12 +3548,33 @@ int subselect_hash_sj_engine::exec() if (strategy == PARTIAL_MATCH) { - if (!(lookup_engine= new subselect_rowid_merge_engine(lookup_engine, - tmp_table))) + subselect_rowid_merge_engine *new_lookup_engine; + uint count_pm_keys; + MY_BITMAP *nn_key_parts; + /* Total number of keys needed for partial matching. */ + if (count_partial_match_columns < tmp_table->s->fields) { - res= 1; - goto err; + count_pm_keys= count_partial_match_columns + 1; + nn_key_parts= &non_null_key_parts; } + else + { + count_pm_keys= count_partial_match_columns; + nn_key_parts= NULL; + } + + if (!(new_lookup_engine= + new subselect_rowid_merge_engine(lookup_engine, + tmp_table, + count_pm_keys, + item, result)) || + new_lookup_engine->init(nn_key_parts, &partial_match_key_parts)) + { + delete new_lookup_engine; + strategy= PARTIAL_MATCH_SCAN; + /* TODO: setup execution structures for partial match via scanning. */ + } + strategy= PARTIAL_MATCH_INDEX; } item_in->change_engine(lookup_engine); @@ -3648,37 +3627,299 @@ bool subselect_hash_sj_engine::change_re } -bool Ordered_key::sort_keys() +Ordered_key::Ordered_key(uint key_idx_arg, TABLE *tbl_arg, + Item *search_key_arg, ha_rows null_count_arg, + ha_rows min_null_row_arg, ha_rows max_null_row_arg, + uchar *row_num_to_rowid_arg) + : key_idx(key_idx_arg), tbl(tbl_arg), search_key(search_key_arg), + row_num_to_rowid(row_num_to_rowid_arg), null_count(null_count_arg), + min_null_row(min_null_row_arg), max_null_row(max_null_row_arg) { - return TRUE; + key_column_count= search_key->cols(); + cur_row= HA_POS_ERROR; +} + + +/* + Initialize a multi-column index. +*/ + +bool Ordered_key::init(MY_BITMAP *columns_to_index) +{ + THD *thd= tbl->in_use; + uint cur_key_col= 0; + + DBUG_ENTER("Ordered_key::init"); + + DBUG_ASSERT(key_column_count == bitmap_bits_set(columns_to_index)); + + // TODO: check for mem allocation err, revert to scan + + key_columns= (Item_field**) thd->alloc(key_column_count * + sizeof(Item_field*)); + compare_pred= (Item_func_lt**) thd->alloc(key_column_count * + sizeof(Item_func_lt*)); + + for (uint i= 0; i < columns_to_index->n_bits; i++, cur_key_col++) + { + if (!bitmap_is_set(columns_to_index, i)) + continue; + key_columns[cur_key_col]= new Item_field(tbl->field[i]); + /* Create the predicate (tmp_column[i] < outer_ref[i]). */ + compare_pred[cur_key_col]= new Item_func_lt(key_columns[cur_key_col], + search_key->element_index(i)); + } + + if (alloc_keys_buffers()) + { + /* TODO revert to partial match via table scan. */ + DBUG_RETURN(TRUE); + } + DBUG_RETURN(FALSE); +} + + +/* + Initialize a single-column index. +*/ + +bool Ordered_key::init(int col_idx) +{ + THD *thd= tbl->in_use; + + DBUG_ENTER("Ordered_key::init"); + + DBUG_ASSERT(key_column_count == 1); + + // TODO: check for mem allocation err, revert to scan + + key_columns= (Item_field**) thd->alloc(sizeof(Item_field*)); + compare_pred= (Item_func_lt**) thd->alloc(sizeof(Item_func_lt*)); + + key_columns[0]= new Item_field(tbl->field[col_idx]); + /* Create the predicate (tmp_column[i] < outer_ref[i]). */ + compare_pred[0]= new Item_func_lt(key_columns[0], + search_key->element_index(col_idx)); + + if (alloc_keys_buffers()) + { + /* TODO revert to partial match via table scan. */ + DBUG_RETURN(TRUE); + } + DBUG_RETURN(FALSE); +} + + +bool Ordered_key::alloc_keys_buffers() +{ + THD *thd= tbl->in_use; + ha_rows row_count= tbl->file->stats.records; + + if (!(row_index= (ha_rows*) thd->alloc((row_count - null_count) * + sizeof(ha_rows)))) + return TRUE; + + /* + TODO: it is enough to create bitmaps with size + (max_null_row - min_null_row), and then use min_null_row as + lookup offset. + */ + if (!(bitmap_init_memroot(&null_key, max_null_row, + thd->mem_root))) + return TRUE; + + return FALSE; +} + + +/* + Quick sort comparison function that compares two rows of the same table + indentfied with their row numbers. +*/ + +int Ordered_key::cmp_rows_by_rownum(Ordered_key *key, ha_rows *a, ha_rows *b) +{ + uchar *rowid_a, *rowid_b; + int error, cmp_res; + TABLE *tbl= key->tbl; + /* The length in bytes of the rowids (positions) of tmp_table. */ + uint rowid_length= tbl->file->ref_length; + + DBUG_ENTER("Ordered_key::cmp_rows_by_rownum"); + if (a == b) + DBUG_RETURN(0); + /* Get the corresponding rowids. */ + rowid_a= key->row_num_to_rowid + (*a) * rowid_length; + rowid_b= key->row_num_to_rowid + (*b) * rowid_length; + /* Fetch the rows for comparison. */ + error= tbl->file->rnd_pos(tbl->record[0], rowid_a); + DBUG_ASSERT(!error); + error= tbl->file->rnd_pos(tbl->record[1], rowid_b); + DBUG_ASSERT(!error); + /* Compare the two rows. */ + for (Field **f_ptr= tbl->field; *f_ptr; f_ptr++) + { + if ((cmp_res= (*f_ptr)->cmp_offset(tbl->s->rec_buff_length))) + DBUG_RETURN(cmp_res); + } + DBUG_RETURN(0); +} + + +void Ordered_key::sort_keys() +{ + my_qsort(row_index, tbl->file->stats.records, sizeof(ha_rows), + (qsort_cmp) &cmp_rows_by_rownum); +} + + +/* + Compare the value(s) of the current key in 'search_key' with the + data of the current table record accessible via 'key_columns'. + + @notes The comparison result follows from the way compare_pred + is created in Ordered_key::init. Currently compare_pred compares + a field in of the current row with the corresponding Item that + contains the search key. + + @retval -1 if (current row < search_key) + @retval 0 if (current row == search_key) + @retval +1 if (current row > search_key) +*/ + +int Ordered_key::compare_row_with_key(ha_rows row_num) +{ + /* The length in bytes of the rowids (positions) of tmp_table. */ + uint rowid_length= tbl->file->ref_length; + uchar *cur_rowid= row_num_to_rowid + row_num * rowid_length; + int error, cmp_res; + + DBUG_ENTER("Ordered_key::compare"); + error= tbl->file->rnd_pos(tbl->record[0], cur_rowid); + DBUG_ASSERT(!error); + + for (uint i= 0; i < key_column_count; i++) + { + cmp_res= compare_pred[i]->get_comparator()->compare(); + /* Unlike Arg_comparator::compare_row() here there should be no NULLs. */ + DBUG_ASSERT(!compare_pred[i]->null_value); + if (cmp_res) + DBUG_RETURN(cmp_res); + } + DBUG_RETURN(0); } /* + Find a key in a sorted array of keys via binary search. + see create_subq_in_equalities() */ -bool Ordered_key::lookup(Item *search_key) +bool Ordered_key::lookup() { DBUG_ENTER("Ordered_key::lookup"); - DBUG_ASSERT(search_key->cols() == key_column_count); - for (uint i= 0; i < key_column_count; i++) + ha_rows lo= 0; + ha_rows hi= tbl->file->stats.records - 1; + ha_rows mid; + int cmp_res; + + while (lo <= hi) { - // j = corresponding colum at pos i - // compare(search_key->element_index(i), key_columns(j)) - ; + mid= lo + (hi - lo) / 2; + cmp_res= compare_row_with_key(mid); + + if (cmp_res == -1) + { + /* row[mid] < search_key */ + lo= mid + 1; + } + else if (cmp_res == 1) + { + /* row[mid] > search_key */ + hi= mid - 1; + } + else + { + /* row[mid] == search_key */ + cur_row= mid; + DBUG_RETURN(TRUE); + } } - DBUG_RETURN(TRUE); + + DBUG_RETURN(FALSE); } /* + @param non_null_key_parts + @param partial_match_key_parts A union of all single-column NULL key parts. + @param count_partial_match_columns Number of NULL keyparts (set bits above). */ -bool subselect_rowid_merge_engine::init() +bool +subselect_rowid_merge_engine::init(MY_BITMAP *non_null_key_parts, + MY_BITMAP *partial_match_key_parts) { - // TODO + /* The length in bytes of the rowids (positions) of tmp_table. */ + uint rowid_length= tmp_table->file->ref_length; + ha_rows row_count= tmp_table->file->stats.records; + select_materialize_with_stats *result_sink= + (select_materialize_with_stats *) result; + uint cur_key= 0; + + if (!(row_num_to_rowid= (uchar*) thd->alloc(row_count * rowid_length * + sizeof(uchar)))) + return TRUE; + + if (!(bitmap_init_memroot(&matching_keys, keys_count, thd->mem_root))) + return TRUE; + + merge_keys= (Ordered_key**) thd->alloc(keys_count * sizeof(Ordered_key*)); + /* Create the only non-NULL key if there is any. */ + if (non_null_key_parts) + { + non_null_key= new Ordered_key(cur_key, tmp_table, item, 0, 0, 0, + row_num_to_rowid); + if (non_null_key->init(non_null_key_parts)) + { + // TODO: revert to partial matching via scanning + return TRUE; + } + merge_keys[cur_key]= non_null_key; + non_null_key->sort_keys(); + ++cur_key; + } + /* + Create one single-column NULL-key for each column in + partial_match_key_parts. + */ + for (uint i= 0; i < partial_match_key_parts->n_bits; i++, cur_key++) + { + if (!bitmap_is_set(partial_match_key_parts, i)) + continue; + + merge_keys[cur_key]= new Ordered_key(cur_key, tmp_table, item, + result_sink->get_null_count_of_col(i), + result_sink->get_min_null_of_col(i), + result_sink->get_max_null_of_col(i), + row_num_to_rowid); + if (merge_keys[cur_key]->init(i)) + { + // TODO: revert to partial matching via scanning + return TRUE; + } + merge_keys[cur_key]->sort_keys(); + } + + if (init_queue(&pq, keys_count, 0, FALSE, + subselect_rowid_merge_engine::cmp_key_by_cur_row, NULL)) + { + // TODO: revert to partial matching via scanning + return TRUE; + } + return FALSE; } @@ -3690,6 +3931,41 @@ void subselect_rowid_merge_engine::clean /* +*/ + +int +subselect_rowid_merge_engine::cmp_key_by_null_selectivity(Ordered_key *a, + Ordered_key *b) +{ + double a_sel= a->null_selectivity(); + double b_sel= b->null_selectivity(); + if (a_sel == b_sel) + return 0; + if (a_sel > b_sel) + return 1; + return -1; +} + + +/* +*/ + +int +subselect_rowid_merge_engine::cmp_key_by_cur_row(void *arg, + uchar *k1, uchar *k2) +{ + ha_rows row1= ((Ordered_key*) k1)->current(); + ha_rows row2= ((Ordered_key*) k2)->current(); + + if (row1 > row2) + return 1; + if (row1 == row2) + return 0; + return -1; +} + + +/* Check if certain table row contains a NULL in all columns in all columns for which there is no value match. @@ -3704,13 +3980,11 @@ void subselect_rowid_merge_engine::clean bool subselect_rowid_merge_engine::test_null_row(ha_rows row_num) { - Ordered_key *cur_key= keys; - DBUG_ENTER("subselect_rowid_merge_engine::test_null_row"); - for (uint i = 0; i < keys_count; i++, cur_key++) + for (uint i = 0; i < keys_count; i++) { - if (bitmap_is_set(matching_keys, i)) + if (bitmap_is_set(&matching_keys, i)) { /* The key 'i' already matches a value in row 'row_num', thus we @@ -3718,7 +3992,7 @@ bool subselect_rowid_merge_engine::test_ */ continue; } - if (!cur_key->is_null(row_num)) + if (!merge_keys[i]->is_null(row_num)) DBUG_RETURN(FALSE); } DBUG_RETURN(TRUE); @@ -3736,14 +4010,13 @@ bool subselect_rowid_merge_engine::parti ha_rows min_row; /* Current row number of min_key. */ Ordered_key *cur_key; ha_rows cur_row; - Item_in_subselect *item_in= (Item_in_subselect *) item; DBUG_ENTER("subselect_rowid_merge_engine::partial_match"); /* If there is a non-NULL key, it must be the first key in the keys array. */ - DBUG_ASSERT(non_null_key && keys == non_null_key); + DBUG_ASSERT(non_null_key && merge_keys[0] == non_null_key); /* Check if there is a match for the columns of the only non-NULL key. */ - if (non_null_key && !non_null_key->lookup(item_in->left_expr)) + if (non_null_key && !non_null_key->lookup()) DBUG_RETURN(FALSE); if (non_null_key) queue_insert(&pq, (uchar *) non_null_key); @@ -3753,10 +4026,10 @@ bool subselect_rowid_merge_engine::parti non_null_key, since it was already processed above. */ uint i= non_null_key ? 1 : 0; /* Skip the non-NULL key, already processed. */ - for (cur_key= keys; i < keys_count; i++, cur_key++) + for (; i < keys_count; i++) { - if (cur_key->lookup(item_in->left_expr)) - queue_insert(&pq, (uchar *) cur_key); + if (merge_keys[i]->lookup()) + queue_insert(&pq, (uchar *) merge_keys[i]); } /* Not all value keys are empty, thus we don't have only NULL keys. If we had, @@ -3767,8 +4040,8 @@ bool subselect_rowid_merge_engine::parti DBUG_ASSERT(pq.elements > 1); min_key= (Ordered_key*) queue_remove(&pq, 0); min_row= min_key->current(); - bitmap_clear_all(matching_keys); - bitmap_set_bit(matching_keys, min_key->get_key_idx()); + bitmap_clear_all(&matching_keys); + bitmap_set_bit(&matching_keys, min_key->get_key_idx()); min_key->next(); if (!min_key->is_eof()) queue_insert(&pq, (uchar *) min_key); @@ -3780,9 +4053,9 @@ bool subselect_rowid_merge_engine::parti if (cur_row == min_row) { - bitmap_set_bit(matching_keys, cur_key->get_key_idx()); + bitmap_set_bit(&matching_keys, cur_key->get_key_idx()); /* There cannot be a complete match, as we already checked for one. */ - DBUG_ASSERT(bitmap_bits_set(matching_keys) < matching_keys->n_bits); + DBUG_ASSERT(bitmap_bits_set(&matching_keys) < matching_keys.n_bits); } else { @@ -3794,8 +4067,8 @@ bool subselect_rowid_merge_engine::parti { min_key= cur_key; min_row= cur_row; - bitmap_clear_all(matching_keys); - bitmap_set_bit(matching_keys, min_key->get_key_idx()); + bitmap_clear_all(&matching_keys); + bitmap_set_bit(&matching_keys, min_key->get_key_idx()); } } === modified file 'sql/item_subselect.h' --- a/sql/item_subselect.h 2010-01-22 16:18:05 +0000 +++ b/sql/item_subselect.h 2010-02-01 12:09:48 +0000 @@ -683,68 +683,70 @@ class Ordered_key { protected: /* - Index of the key in some array of keys. This index allows to + Index of the key in an array of keys. This index allows to construct (sub)sets of keys represented by bitmaps. */ uint key_idx; + /* The table being indexed. */ + TABLE *tbl; /* The columns being indexed. */ Item_field **key_columns; /* Number of elements in 'key_columns' (number of key parts). */ uint key_column_count; + /* + An expression, or sequence of expressions that forms the search key. + */ + Item *search_key; /* Value index related members. */ - /* The actual value index, consists of a sorted sequence of row numbers. */ + /* + The actual value index, consists of a sorted sequence of row numbers. + There are tbl->file->stats.records elements in this array. + */ ha_rows *row_index; /* Current element in 'row_index'. */ ha_rows cur_row; /* - TODO: define a quick sort comparison function. + Mapping from row numbers to row ids. The element row_num_to_rowid[i] + contains a buffer with the rowid for the row numbered 'i'. + The memory for this member is not maintanined by this class because + all Ordered_key indexes of the same table share the same mapping. + */ + uchar *row_num_to_rowid; + /* + A sequence of predicates to compare the search key with the corresponding + columns of a table row from the index. */ + Item_func_lt **compare_pred; /* Null index related members. */ MY_BITMAP null_key; /* Count of NULLs per column. */ ha_rows null_count; - /* The row number that contains the last NULL in a column. */ - ha_rows max_null_row; /* The row number that contains the first NULL in a column. */ ha_rows min_null_row; - /* - TODO: define a qsort comparison function to compare keys in order of - increasing bitmap selectivity. - */ + /* The row number that contains the last NULL in a column. */ + ha_rows max_null_row; protected: - ha_rows get_row_count() - { - /* Assume that file->info(HA_STATUS_VARIABLE) has been called. */ - return key_columns[0]->field->table->file->stats.records; - } + bool alloc_keys_buffers(); /* - Compute the index (position) of an indexed column in the table definition. - - @param i index in the 'key_columns' array. - - @returns The index of the corresponding indexed column in the TABLE::field - array of all table fields. + Quick sort comparison function that compares two rows of the same table + indentfied with their row numbers. */ - uint get_column_idx(uint i) - { - DBUG_ENTER("get_column_idx"); - DBUG_ASSERT(i < key_column_count); - /* All key_columns must be from the same table, so any one is fine. */ - //TABLE *tab= key_columns[0]->field->table; - //Field *col_i= columns->field + i; - //DBUG_RETURN(col_i - tab->field); - DBUG_RETURN(0); - } + static int cmp_rows_by_rownum(Ordered_key *key, ha_rows* a, ha_rows* b); + + int compare_row_with_key(ha_rows row_num); public: - Ordered_key(TABLE *tab) - { - /* TODO: init all Item_fields from the table columns. */ - } - bool init(ha_rows row_count); + Ordered_key(uint key_idx_arg, TABLE *tbl_arg, + Item *search_key_arg, ha_rows null_count_arg, + ha_rows min_null_row_arg, ha_rows max_null_row_arg, + uchar *row_num_to_rowid_arg); + /* Initialize a multi-column index. */ + bool init(MY_BITMAP *columns_to_index); + /* Initialize a single-column index. */ + bool init(int col_idx); uint get_column_count() { return key_column_count; } uint get_key_idx() { return key_idx; } @@ -753,21 +755,23 @@ public: row_index[cur_row]= row_num; ++cur_row; } - bool sort_keys(); + + void sort_keys(); + + double null_selectivity() { return (1 - null_count / null_key.n_bits); } + /* Position the current element at the first row that matches the key. - TODO: the argument here is the left IN argument, which is a sequence - of Items. We have to compare these Items with the corresponding Fields - of the temp table. To do this wrap each field in an Item_field, then - compare. See how it is done in create_subq_in_equalities(). + The key itself is propagated by evaluating the current value(s) of + this->search_key. */ - bool lookup(Item *search_key); + bool lookup(); /* Return the current index element. */ ha_rows current() { return row_index[cur_row]; } /* Move the current index cursor at the next match. */ bool next() { - if (cur_row < get_row_count()) + if (cur_row < tbl->file->stats.records) { ++cur_row; return TRUE; @@ -775,7 +779,7 @@ public: return FALSE; }; /* Return false if all matches are exhausted, true otherwise. */ - bool is_eof() { return cur_row == get_row_count(); } + bool is_eof() { return cur_row == tbl->file->stats.records; } void set_null(ha_rows row_num) { @@ -789,9 +793,9 @@ public: Their only initialized member is 'n_bits', which is equal to the number of temp table rows. */ - if (null_count == get_row_count()) + if (null_count == tbl->file->stats.records) { - DBUG_ASSERT(get_row_count() == null_key.n_bits); + DBUG_ASSERT(tbl->file->stats.records == null_key.n_bits); DBUG_RETURN(TRUE); } if (row_num > max_null_row || row_num < min_null_row) @@ -812,20 +816,16 @@ protected: FALSE and UNKNOWN. */ subselect_engine *lookup_engine; - - /* The length in bytes of the rowids (positions) of tmp_table. */ - uint rowid_length; /* - Mapping from row numbers to row ids. The element 'i' with lenght - 'rowid_length' - (row_num_to_rowid + i*rowid_length) contains - the rowid of row numbered 'i'. + Mapping from row numbers to row ids. The element row_num_to_rowid[i] + contains a buffer with the rowid for the row numbered 'i'. */ uchar *row_num_to_rowid; /* A subset of all the keys for which there is a match for the same row. Used during execution. Computed for each call to exec(). */ - MY_BITMAP *matching_keys; + MY_BITMAP matching_keys; /* Indexes of row numbers, sorted by <column_value, row_number>. If an index may contain NULLs, the NULLs are stored efficiently in a bitmap. @@ -834,7 +834,7 @@ protected: one with the fewer NULLs is first. Thus, if there is any index on non-NULL columns, it is contained in keys[0]. */ - Ordered_key *keys; + Ordered_key **merge_keys; /* The number of elements in keys. */ uint keys_count; /* @@ -849,33 +849,31 @@ protected: This queue is used by the partial match algorithm in method exec(). */ QUEUE pq; - +protected: /* - True if some column in the temp table consist of only NULLs. Then - any match is a partial match. + Comparison function to compare keys in order of increasing bitmap + selectivity. */ - bool inner_partial_match; - bool null_keypart; /* TRUE <=> constructed search tuple has a NULL */ + static int cmp_key_by_null_selectivity(Ordered_key *a, Ordered_key *b); /* - A conjunction of all the equality condtions between all pairs of expressions - that are arguments of an IN predicate. We need these to post-filter some - IN results because index lookups sometimes match values that are actually - not equal to the search key in SQL terms. + Comparison function used by the priority queue pq, the 'smaller' key + is the one with the smaller current row number. */ - Item_cond_and *semi_join_conds; -protected: + static int cmp_key_by_cur_row(void *arg, uchar *k1, uchar *k2); + bool test_null_row(ha_rows row_num); bool partial_match(); public: subselect_rowid_merge_engine(subselect_engine *lookup_engine_arg, - TABLE *tmp_table_arg) - :subselect_engine(NULL, NULL) - { - lookup_engine= lookup_engine_arg; - tmp_table= tmp_table_arg; - rowid_length= tmp_table->file->ref_length; - } - bool init(); + TABLE *tmp_table_arg, uint keys_count_arg, + Item_subselect *item_arg, + select_result_interceptor *result_arg) + :subselect_engine(item_arg, result_arg), + tmp_table(tmp_table_arg), lookup_engine(lookup_engine_arg), + keys_count(keys_count_arg) + {} + + bool init(MY_BITMAP *non_null_key_parts, MY_BITMAP *partial_match_key_parts); void cleanup(); int prepare() { return 0; } void fix_length_and_dec(Item_cache**) {} @@ -930,15 +928,11 @@ protected: */ bool has_null_row; - /* Keyparts of the only non-NULL composite index in a ror_intersect. */ - key_part_map non_null_key_parts; - List<Item> non_null_outer_cols; /* Corresponding non-NULL outer columns. */ - /* keyparts of the non-NULL single column indexes, one keypart per index. */ - key_part_map non_null_late_key_parts; - /* keyparts of the single column indexes with NULL, one keypart per index. */ - key_part_map partial_match_key_parts; - uint count_non_null_late_cols, count_partial_match_cols; - + /* Keyparts of the only non-NULL composite index in a rowid merge. */ + MY_BITMAP non_null_key_parts; + /* Keyparts of the single column indexes with NULL, one keypart per index. */ + MY_BITMAP partial_match_key_parts; + uint count_partial_match_columns; /* A conjunction of all the equality condtions between all pairs of expressions that are arguments of an IN predicate. We need these to post-filter some @@ -948,8 +942,10 @@ protected: Item *semi_join_conds; /* Possible execution strategies that can be used to compute hash semi-join.*/ enum exec_strategy { - COMPLETE_MATCH, /* Use plain index lookups. */ - PARTIAL_MATCH, /* Use partial matching. */ + COMPLETE_MATCH, /* Use regular index lookups. */ + PARTIAL_MATCH, /* Use some partial matching strategy. */ + PARTIAL_MATCH_INDEX, /* Use partial matching through index merging. */ + PARTIAL_MATCH_SCAN, /* Use partial matching through table scan. */ IMPOSSIBLE /* Subquery materialization is not applicable. */ }; /* The chosen execution strategy. Computed after materialization. */ @@ -965,14 +961,10 @@ public: subselect_single_select_engine *old_engine) :subselect_engine(in_predicate, NULL), tmp_table(NULL), is_materialized(FALSE), materialize_engine(old_engine), lookup_engine(NULL), - materialize_join(NULL), inner_partial_match(FALSE), - count_non_null_late_cols(0), count_partial_match_cols(0), + materialize_join(NULL), count_partial_match_columns(0), semi_join_conds(NULL) { set_thd(thd); - non_null_key_parts= (key_part_map) 0; - non_null_late_key_parts= (key_part_map) 0; - partial_match_key_parts= (key_part_map) 0; } ~subselect_hash_sj_engine(); === modified file 'sql/sql_class.h' --- a/sql/sql_class.h 2010-01-22 16:18:05 +0000 +++ b/sql/sql_class.h 2010-02-01 12:09:48 +0000 @@ -3079,11 +3079,21 @@ public: count_rows= 0; memset(col_stat, 0, table->s->fields * sizeof(Column_statistics)); } - ha_rows get_column_null_count(uint idx) + ha_rows get_null_count_of_col(uint idx) { DBUG_ASSERT(idx < table->s->fields); return col_stat[idx].null_count; } + ha_rows get_max_null_of_col(uint idx) + { + DBUG_ASSERT(idx < table->s->fields); + return col_stat[idx].max_null_row; + } + ha_rows get_min_null_of_col(uint idx) + { + DBUG_ASSERT(idx < table->s->fields); + return col_stat[idx].min_null_row; + } ha_rows get_null_record_count() { return null_record_count; } };

1 0

Re: [Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2809: Support building with system libevent
by Kristian Nielsen 01 Feb '10

01 Feb '10

Sergei Golubchik <serg(a)askmonty.org> writes: >> Then I also need to decide what to do for packages when system >> libevent is too old. I checked, and Debian 4+5 and Ubuntu up to 9.04 >> have only libevent 1.3. Ubuntu >= 9.10 has libevent 1.4. >> >> I think the best is to use bundled libevent on those older distros. We >> will then still suffer from >> https://bugs.launchpad.net/maria/+bug/509521 on those, but hopefully >> most people using plugins which require libevent will build mariadb >> themselves and/or use a newer distro. > > What about specifying a deb dependency that mariadb needs > libevent >= 1.4 ? > > Then package manager will handle the rest I don't understand what you mean? On Ubuntu 9.10 and up specifying a dependency on libevent >=1.4 will work, yes. But on Debian <= 5 and Ubuntu <= 9.04, a dependency on libevent >= 1.4 would make the package uninstallable, as libevent 1.4 is not available in those distros. Or did you mean we should provide libevent 1.4 as a package in the OurDelta repository? I think using system libevent on Ubuntu >= 9.10 and bundled on Ubuntu <= 9.04 and Debian should work ok. Most people installing something like the Pinba storage engine (which uses libevent) will probably in any case recompile mariadb. Whatever packages do will not affect them, so the important thing is that with your patch, they can now get MariaDB to use libevent 1.4 if they install it. - Kristian.

1 0

[Maria-developers] really need call my_thread_init() in libmysqlclient_r?
by xiaobing jiang 01 Feb '10

01 Feb '10

HI, all: There is a bug about mysql-python, https://sourceforge.net/tracker/?func=detail&aid=1464563&group_id=22307&ati…. After review the code of libmysqlclient_r , I find for every new thread to create connection, it will call my_thread_init to init some thread local variable and other things. but the client api seems not using this thread local variable almost, just for some debug information. Can using macro remove this when the code using by libmysqlclient_r.so? If can do that, to fix the bug is easy. sorry for my poor engish, and this maybe should post in mysql mail-list! Thanks!

1 0

[Maria-developers] Rev 2738: merge 5.1->5.2 in file:///home/bell/maria/bzr/work-maria-5.2-merge-5.1/
by sanja＠askmonty.org 01 Feb '10

01 Feb '10

At file:///home/bell/maria/bzr/work-maria-5.2-merge-5.1/ ------------------------------------------------------------ revno: 2738 [merge] revision-id: sanja(a)askmonty.org-20100201061412-xx7312ytg9d6qnbk parent: monty(a)askmonty.org-20100106192016-46to56kdmp1frlgl parent: monty(a)askmonty.org-20100129184222-45b3lr7hp2joa5mp committer: sanja(a)askmonty.org branch nick: work-maria-5.2-merge-5.1 timestamp: Mon 2010-02-01 08:14:12 +0200 message: merge 5.1->5.2 removed: mysql-test/suite/maria/t/maria2-master.opt maria2master.opt-20090215104617-4fkcge2h3ssmgmv0-1 storage/xtradb/handler/handler0vars.h handler0vars.h-20081203050234-edoolglm28lyejuc-3 storage/xtradb/handler/win_delay_loader.cc win_delay_loader.cc-20081203050234-edoolglm28lyejuc-4 storage/xtradb/win-plugin/ winplugin-20081203050234-edoolglm28lyejuc-2 storage/xtradb/win-plugin/README readme-20081203050234-edoolglm28lyejuc-15 storage/xtradb/win-plugin/win-plugin.diff winplugin.diff-20081203050234-edoolglm28lyejuc-16 added: BUILD/compile-bintar compilebintar-20100107101810-lelof47hh40zljzw-1 BUILD/util.sh util.sh-20100107105306-5523083hapn1b4n5-1 mysql-test/extra/rpl_tests/rpl_not_null.test rpl_not_null.test-20090929140808-ln24vq9he9qkn20x-1 mysql-test/r/bug47671.result bug47671.result-20091125061811-9f1u8eyhc4lhli89-1 mysql-test/r/innodb-consistent.result innodbconsistent.res-20100106115838-xm7yncpgr8fsekjt-2 mysql-test/r/innodb_bug44571.result innodb_bug44571.resu-20100106115838-xm7yncpgr8fsekjt-6 mysql-test/r/innodb_bug46676.result innodb_bug46676.resu-20100106115838-xm7yncpgr8fsekjt-10 mysql-test/r/innodb_bug47167.result innodb_bug47167.resu-20100106115838-xm7yncpgr8fsekjt-12 mysql-test/r/udf_query_cache.result udf_query_cache.resu-20100111125806-7lkv720m352b58ia-1 mysql-test/std_data/bug47012.ARM bug47012.arm-20091110122641-iqp1lul6n2ekz7m1-3 mysql-test/std_data/bug47012.ARZ bug47012.arz-20091110122641-iqp1lul6n2ekz7m1-2 mysql-test/std_data/bug47012.frm bug47012.frm-20091110122641-iqp1lul6n2ekz7m1-1 mysql-test/suite/innodb/r/innodb_bug46676.result innodb_bug46676.resu-20091130121246-w0etrydh59vqrhfk-1 mysql-test/suite/innodb/r/innodb_bug47167.result innodb_bug47167.resu-20091130114932-z1l0gqbz3zjhelcn-1 mysql-test/suite/innodb/t/innodb_bug46676.test innodb_bug46676.test-20091130121246-w0etrydh59vqrhfk-2 mysql-test/suite/innodb/t/innodb_bug47167.test innodb_bug47167.test-20091130114932-z1l0gqbz3zjhelcn-2 mysql-test/suite/rpl/r/rpl_loaddata_symlink.result rpl_loaddata_symlink-20091112200612-x5ek042xipzbdluy-1 mysql-test/suite/rpl/r/rpl_nondeterministic_functions.result rpl_nondeterministic-20091118142013-9zx8msk45pqp0kcn-1 mysql-test/suite/rpl/r/rpl_not_null_innodb.result rpl_not_null_innodb.-20090929140808-ln24vq9he9qkn20x-2 mysql-test/suite/rpl/r/rpl_not_null_myisam.result rpl_not_null_myisam.-20090929140808-ln24vq9he9qkn20x-3 mysql-test/suite/rpl/r/rpl_row_trunc_temp.result rpl_row_trunc_temp.r-20091117145658-biyfd3a42vxttw4g-1 mysql-test/suite/rpl/t/rpl_loaddata_symlink-master.opt rpl_loaddata_symlink-20091112200645-33lzxlcjtrz4ekex-1 mysql-test/suite/rpl/t/rpl_loaddata_symlink-master.sh rpl_loaddata_symlink-20091120052332-4iio0f9yuk8srq37-1 mysql-test/suite/rpl/t/rpl_loaddata_symlink-slave.opt rpl_loaddata_symlink-20091120052318-rey16d8e27esu4af-1 mysql-test/suite/rpl/t/rpl_loaddata_symlink-slave.sh rpl_loaddata_symlink-20091120052322-exs2kyw12kmns1tw-1 mysql-test/suite/rpl/t/rpl_loaddata_symlink.test rpl_loaddata_symlink-20091112200628-bv7esgx1vr7rhcky-1 mysql-test/suite/rpl/t/rpl_nondeterministic_functions.test rpl_nondeterministic-20091118141837-p0896pkizwxx60yu-1 mysql-test/suite/rpl/t/rpl_not_null_innodb.test rpl_not_null_innodb.-20090929140808-ln24vq9he9qkn20x-4 mysql-test/suite/rpl/t/rpl_not_null_myisam.test rpl_not_null_myisam.-20090929140808-ln24vq9he9qkn20x-5 mysql-test/suite/rpl/t/rpl_row_trunc_temp.test rpl_row_trunc_temp.t-20091117145658-biyfd3a42vxttw4g-2 mysql-test/t/bug47671-master.opt bug47671master.opt-20091125061855-277zbp2qtbff4akr-1 mysql-test/t/bug47671.test bug47671.test-20091125061804-u0cz6u7eeeta2ao3-1 mysql-test/t/innodb-consistent-master.opt innodbconsistentmast-20100106115838-xm7yncpgr8fsekjt-1 mysql-test/t/innodb-consistent.test innodbconsistent.tes-20100106115838-xm7yncpgr8fsekjt-3 mysql-test/t/innodb_bug44571.test innodb_bug44571.test-20100106115838-xm7yncpgr8fsekjt-7 mysql-test/t/innodb_bug46676.test innodb_bug46676.test-20100106115838-xm7yncpgr8fsekjt-11 mysql-test/t/innodb_bug47167.test innodb_bug47167.test-20100106115838-xm7yncpgr8fsekjt-13 mysql-test/t/mysqlbinlog2-master.opt mysqlbinlog2master.o-20100119103451-7nkhltk4tgr4fegm-1 mysql-test/t/mysqlbinlog_row-master.opt mysqlbinlog_rowmaste-20100127131139-b8fh6zh2p2j13cfj-1 mysql-test/t/mysqlbinlog_row_innodb-master.opt mysqlbinlog_row_inno-20100119125405-22esbb4wolnwm5h3-1 mysql-test/t/mysqlbinlog_row_myisam-master.opt mysqlbinlog_row_myis-20100119125405-22esbb4wolnwm5h3-2 mysql-test/t/mysqlbinlog_row_trans-master.opt mysqlbinlog_row_tran-20100127113200-cas1q0skxod0t0xi-2 mysql-test/t/udf_query_cache-master.opt udf_query_cachemaste-20100111125806-7lkv720m352b58ia-2 mysql-test/t/udf_query_cache.test udf_query_cache.test-20100111125806-7lkv720m352b58ia-3 storage/xtradb/ut/ut0auxconf_have_gcc_atomics.c ut0auxconf_have_gcc_-20100106115838-xm7yncpgr8fsekjt-16 renamed: mysql-test/suite/pbxt/t/load_unique_error1.inc => mysql-test/std_data/pbxt_load_unique_error1.inc load_unique_error1.i-20090407105731-jrdzpnlb2nlsfdp1-1 modified: .bzrignore sp1f-ignore-20001018235455-q4gxfbritt5f42nwix354ufpsvrf5ebj BUILD/Makefile.am sp1f-makefile.am-20020102192940-dza66ux2yxyklupzjz4q3km3hvye5rnj BUILD/SETUP.sh sp1f-setup.sh-20001218212418-itvzddls4bsqffggcsjklbawdmaxdhde BUILD/compile-solaris-amd64-debug-forte* compilesolarisamd64d-20090707110736-p2i53hs87u5tkgxs-1 BUILD/compile-solaris-x86-32* compilesolarisx8632-20090707110736-p2i53hs87u5tkgxs-2 BUILD/compile-solaris-x86-32-debug* compilesolarisx8632d-20090707110736-p2i53hs87u5tkgxs-3 BUILD/compile-solaris-x86-32-debug-forte* compilesolarisx8632d-20090707110736-p2i53hs87u5tkgxs-4 BUILD/compile-solaris-x86-forte-32* compilesolarisx86for-20090707110736-p2i53hs87u5tkgxs-5 Makefile.am sp1f-makefile.am-19700101030959-jbbpiygwpgybyqknlavdxxupbrjonu7h client/mysql.cc sp1f-mysql.cc-19700101030959-5sipizk7ehvbsi3tywrkdords5qy5zdl client/mysqlbinlog.cc sp1f-mysqlbinlog.cc-19700101030959-b3vgyo47ljent5mhbyj6ik33bi4bukad client/mysqlslap.c sp1f-mysqlslap.c-20051130000206-7t375hf5mtlqof5xd4nj76yckxvxykhv client/mysqltest.cc sp1f-mysqltest.c-20001010065317-ix4zw26srlev7yugcz455ux22zwyynyf config/ac-macros/plugins.m4 sp1f-plugins.m4-20060413204924-cltp6uagmyygsgdno6od3mamfizdhk3m configure.in sp1f-configure.in-19700101030959-mgdpoxtnh2ewmvusvfpkreuhwvffkcjw extra/comp_err.c sp1f-comp_err.c-19700101030959-xhnod5xbbwq5dckoic5y65at66d3sgik extra/libevent/devpoll.c devpoll.c-20090312215838-41pxaswf0zgarxu3-30 extra/libevent/evbuffer.c evbuffer.c-20090312215838-41pxaswf0zgarxu3-33 extra/yassl/src/yassl_error.cpp sp1f-yassl_error.cpp-20050428132311-uwd5s2khyebi5wkzp66p3hhvr4sh44f3 extra/yassl/taocrypt/include/asn.hpp sp1f-asn.hpp-20050428132312-5ijcjgxj7cy3t67jcpi4rg3rbr4nnfmn extra/yassl/taocrypt/src/asn.cpp sp1f-asn.cpp-20050428132318-okq6hllvtur6rcfg4gc5pbxebunf764v include/ft_global.h sp1f-ft_global.h-19700101030959-qzez255ofrojrptdc5z2oi3sfi3bemf7 include/maria.h sp1f-maria.h-20060411134400-ylx7cem3pcdf2jg6it2tuutxyzoljzvv include/my_pthread.h sp1f-my_pthread.h-19700101030959-z4yp3kljwx5fgmhlyvumtwxuw73xgrjn include/mysql.h sp1f-mysql.h-19700101030959-soi7hu6ji273nui3fm25jjf4m4362pcw include/mysql.h.pp mysql.h.pp-20080613094407-2m1760u4zdzt4dc7-1 include/violite.h sp1f-violite.h-19700101030959-jfyqeh5pmto4ncgcdcdf36bl5ininiqx libmysql/libmysql.c sp1f-libmysql.c-19700101030959-ba4gwsjdmik5puh2qyrfpvoflwer257l libmysql/libmysql.def sp1f-libmysql.def-20011019010014-dtnxyq3opddoys6ptxjmzhmaa2rlowmq libmysqld/libmysqld.def sp1f-libmysqld.def-20020109073846-c3x4ovkgky3uok5h6uiaxaevadsluq6k mysql-test/collections/default.experimental default.experimental-20090224104813-e52mxw708penxv44-1 mysql-test/extra/rpl_tests/rpl_extraSlave_Col.test sp1f-rpl_extraslave_col.t-20061103140340-egmkull7owd2wp7d4egg6itzef6p7g23 mysql-test/extra/rpl_tests/rpl_row_tabledefs.test sp1f-rpl_row_tabledefs.te-20051222053451-cr6a33nz4a4knerv7ws3ffszjgkqfet3 mysql-test/extra/rpl_tests/rpl_stm_000001.test sp1f-rpl000001.test-20001118063528-ailyrmllkfzwjx3qfvmu555ijzuk5yur mysql-test/include/mtr_warnings.sql sp1f-mtr_warnings.sql-20080408145123-lhtlr627ins6hwi3hxjrcytx4t27nyjr mysql-test/include/ps_conv.inc sp1f-ps_conv.inc-20040925150736-7yq4rnzrahaz656cmry5skpgvu5fjbet mysql-test/lib/mtr_cases.pm sp1f-mtr_cases.pl-20050203205008-rrteoawyobvgq6u7zeyce4tmuu334ayg mysql-test/mysql-test-run.pl sp1f-mysqltestrun.pl-20041230152716-xjnn5ndv4rr4by6ijmj5a4ysubxc7qh3 mysql-test/r/archive.result sp1f-archive.result-20040525194738-teb7vr2fyyav2vmvw55tdwgvu3h65flc mysql-test/r/delayed.result sp1f-delayed.result-20001228015633-d5brh5c3ulnb2qshtfvxu5cvvvxf4lsr mysql-test/r/delete.result sp1f-delete.result-20010928050551-vf5sxtd554vuepifylwowaaq7k3mbilw mysql-test/r/fulltext.result sp1f-fulltext.result-20001228015633-fi5pm63lvhgn665dsef6fjihfkijbrvt mysql-test/r/func_group.result sp1f-func_group.result-20001228015633-oe57bieiww3s6erojiyha7p26m5ul5ql mysql-test/r/func_misc.result sp1f-func_misc.result-20001228015633-4sy6dzzt7xcs4ubzcxloyguc7zhougbr mysql-test/r/grant.result sp1f-grant.result-20020905131705-2gfwpyej777fcllxzcvadzd6tqdxfho3 mysql-test/r/grant2.result sp1f-grant2.result-20030722200047-flh2uaxcbwah7yfj5uohcoxndutgaced mysql-test/r/group_min_max.result sp1f-group_min_max.result-20040827133611-aqzadxttbw23mkanmvdsiaambv2pcy27 mysql-test/r/information_schema.result sp1f-information_schema.r-20041113105544-waoxa2fjjsicturpothmjmi6jc3yrovn mysql-test/r/information_schema_all_engines.result information_schema_a-20090408133348-au36idguotknighe-2 mysql-test/r/innodb-autoinc.result innodbautoinc.result-20081201061010-zymrrwrczns2vrex-280 mysql-test/r/innodb-index.result innodbindex.result-20081201061010-zymrrwrczns2vrex-284 mysql-test/r/innodb-timeout.result innodbtimeout.result-20081203050234-edoolglm28lyejuc-7 mysql-test/r/innodb-zip.result innodbzip.result-20081201061010-zymrrwrczns2vrex-296 mysql-test/r/innodb.result innodb.result-20081201061010-zymrrwrczns2vrex-298 mysql-test/r/innodb_bug36169.result innodb_bug36169.resu-20081201061010-zymrrwrczns2vrex-306 mysql-test/r/innodb_bug44369.result innodb_bug44369.resu-20091005111405-nbp5t33h95jrqha2-1 mysql-test/r/innodb_file_format.result innodb_file_format.r-20090923000535-ke95wdd4zn27df71-18 mysql-test/r/innodb_lock_wait_timeout_1.result bug40113.result-20090619150423-w3im08cym6tyzn8f-3 mysql-test/r/innodb_mysql.result sp1f-innodb_mysql.result-20060426055153-bychbbfnqtvmvrwccwhn24i6yi46uqjv mysql-test/r/innodb_xtradb_bug317074.result innodb_xtradb_bug317-20090326061054-ylrdb8libxw6u7e9-8 mysql-test/r/mysql.result sp1f-mysql.result-20050517191330-5ywsflw7k6pttof273om5l2mb7pyiu22 mysql-test/r/mysqltest.result sp1f-mysqltest.result-20041022024801-dfor5httbrm4yhbhqtfjzpkst5hoejym mysql-test/r/olap.result sp1f-olap.result-20020720115150-egx2d46xkyxi5dgcpyjexyj4ri6wlcvb mysql-test/r/openssl_1.result sp1f-ssl.result-20010831211351-xa6w74zno32dlg3iwugerlalsvrsq5hn mysql-test/r/order_by.result sp1f-order_by.result-20001228015634-omkoitbok7pbz53pkfmplnhbifnrebic mysql-test/r/partition.result sp1f-partition.result-20050718113029-xlmjyugiq5h2b5wjp236ipsmkmej7i62 mysql-test/r/query_cache.result sp1f-query_cache.result-20011205230530-qf3qzwsqsgfi67vv5ijruxeci6cbkwjl mysql-test/r/query_cache_notembedded.result sp1f-query_cache_notembed-20050729121335-enxz2r7srcrudvsmkq357ior3n4nlqpq mysql-test/r/range.result sp1f-range.result-20001228015634-6hpoyn74lnc7irf4gop2jbowgpazbbae mysql-test/r/select.result sp1f-select.result-20010103001548-znkoalxem6wchsbxizfosjhpfmhfyxuk mysql-test/r/show_check.result sp1f-show_check.result-20001228015634-5hf7elb3nj3zmuz6tosvytmbu52bploi mysql-test/r/sp-destruct.result sp1f-spdestruct.result-20051026133447-inwbkiot3w72y54qgbh3r3qkm7io632d mysql-test/r/sp-security.result sp1f-spsecurity.result-20031213154048-xglie74lizlzappe5papku3ysbvrzg75 mysql-test/r/sp.result sp1f-sp.result-20030117133802-duandg3yzagzyv7zhhbbt2kcomcegpc7 mysql-test/r/sp_notembedded.result sp1f-sp_notembedded.resul-20060224163410-okgh2uh6w7jxcoszw5y4sk6pq3ngt5n6 mysql-test/r/trigger.result sp1f-trigger.result-20040907122911-6m6f5d2ijohoqspgy53ybn6kavj4zefi mysql-test/r/trigger_notembedded.result sp1f-triggergrant.result-20051110192455-2zus7d4a7l2y7ldnokefkk6ibykyn46y mysql-test/r/type_newdecimal.result sp1f-type_newdecimal.resu-20050208224936-kj6dxol5i7zbbfohybib53ffje5i63mk mysql-test/r/type_year.result sp1f-type_year.result-20001228015634-qnsjzaaz3ams6pb2etie5s3eleghfgp5 mysql-test/r/udf.result sp1f-udf.result-20060215161120-pm5l3nyny5gbznc2egfu4bhwgxbuc6wz mysql-test/r/variables.result sp1f-variables.result-20001228015635-w5m2doorn7gzhyyhpqrlqsupnwn6f6xh mysql-test/std_data/cacert.pem sp1f-cacert.pem-20010724060723-mzsvdgy4lyvjjx62aqycz65bca4q4ien mysql-test/std_data/client-cert.pem sp1f-clientcert.pem-20010724060723-x4ijt4gnmldkenng3te7iebn6qfarrjj mysql-test/std_data/client-key.pem sp1f-clientkey.pem-20010822071134-2mmefinjcnjzc2gu6vh3ord32xvhpbch mysql-test/std_data/server-cert.pem sp1f-servercert.pem-20010724060723-w7e2s2asnomtwtus3ncffn53f2qjifu6 mysql-test/std_data/server-key.pem sp1f-serverkey.pem-20010822071134-trmhdwb2jmf3nfahr76457fjthitvdm7 mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result sp1f-mix_innodb_myisam_bi-20030822133916-l72xhg2oxjthj6ichxowk55lrbjebfxa mysql-test/suite/binlog/r/binlog_stm_row.result binlog_stm_row.resul-20080929134451-58v46d7hr9wzyo6f-1 mysql-test/suite/binlog/r/binlog_unsafe.result sp1f-binlog_unsafe.result-20070514124535-jqttmp2p2jfelgeewle4swp5rb4j7pbi mysql-test/suite/binlog/t/binlog_killed.test sp1f-binlog_killed.test-20070528192019-wgkf3lpghurbledmqfyi43fg3mlrhxby mysql-test/suite/binlog/t/binlog_stm_mix_innodb_myisam.test sp1f-binlog_stm_mix_innod-20051222053459-semsiuc62fcxnhjqz663pgpzwfjexwci mysql-test/suite/binlog/t/binlog_stm_row.test binlog_stm_row.test-20080929134444-cku0pcqzy8apdoac-1 mysql-test/suite/binlog/t/binlog_unsafe.test sp1f-binlog_unsafe.test-20070514124535-pau2ov4yta3qsk5zbdwkywce3vhd54xr mysql-test/suite/funcs_1/datadict/processlist_val.inc sp1f-processlist_val.inc-20070815194641-3hfsmyh3jr2gjhvgms52iydpkamdbnoz mysql-test/suite/funcs_1/r/is_columns_is.result sp1f-is_columns_is.result-20080307163304-7bd6seaxklddmff6f3bb54inlyw6unpw mysql-test/suite/funcs_1/r/is_tables_is.result sp1f-is_tables_is.result-20080307163304-6xl5vbegso6wet3dzoehxb645vntxpig mysql-test/suite/innodb/r/innodb-index.result innodbindex.result-20090610132748-q9m60aph2eqy8zr6-10 mysql-test/suite/innodb/t/innodb-consistent-master.opt innodbconsistentmast-20091009132511-05q1yxchk8rz94rf-1 mysql-test/suite/innodb/t/innodb-index.test innodbindex.test-20090610132748-q9m60aph2eqy8zr6-24 mysql-test/suite/maria/r/maria-recover.result mariarecover.result-20080602174048-lgw3ipowzkym118b-1 mysql-test/suite/maria/t/maria-recover.test mariarecover.test-20080602174033-rnr5wg8wn2bqarwk-1 mysql-test/suite/parts/inc/partition_timestamp.inc sp1f-partition_timestamp.-20070206122236-ukxavk6zxnrewb3yuohuvwqpnkf3pfne mysql-test/suite/parts/r/partition_recover_myisam.result partition_repair_myi-20080609121315-mjya2e9ekn7bunzm-3 mysql-test/suite/parts/t/partition_alter1_2_innodb.test sp1f-partition_alter1_2_i-20080513231047-g6yx4e6tufcbky2wq463xgngz66cuqnu mysql-test/suite/parts/t/partition_alter2_1_innodb.test sp1f-partition_alter2_inn-20070206122238-btax2l7djjymz6hlnkmuvwdt2nbqbxed mysql-test/suite/parts/t/partition_alter2_2_innodb.test partition_alter2_2_i-20080908140941-884mge0s10lxgki2-3 mysql-test/suite/parts/t/partition_alter4_innodb.test sp1f-partition_alter4_inn-20070206122238-wzdnxx4co5gutrm6mnbees2dkz7bwsii mysql-test/suite/parts/t/partition_recover_myisam.test partition_repair_myi-20080609121315-mjya2e9ekn7bunzm-2 mysql-test/suite/pbxt/r/func_group.result func_group.result-20090402100035-4ilk9i91sh65vjcb-50 mysql-test/suite/pbxt/r/join_nested.result join_nested.result-20090402100035-4ilk9i91sh65vjcb-81 mysql-test/suite/pbxt/r/lock_multi.result lock_multi.result-20090402100035-4ilk9i91sh65vjcb-90 mysql-test/suite/pbxt/r/mysqlshow.result mysqlshow.result-20090402100035-4ilk9i91sh65vjcb-101 mysql-test/suite/pbxt/r/pbxt_bugs.result pbxt_bugs.result-20090402100035-4ilk9i91sh65vjcb-120 mysql-test/suite/pbxt/t/join_nested.test join_nested.test-20090402100035-4ilk9i91sh65vjcb-248 mysql-test/suite/pbxt/t/lock_multi.test lock_multi.test-20090402100035-4ilk9i91sh65vjcb-257 mysql-test/suite/pbxt/t/pbxt_bugs.test pbxt_bugs.test-20090402100035-4ilk9i91sh65vjcb-287 mysql-test/suite/pbxt/t/pbxt_locking.test pbxt_locking.test-20090402100035-4ilk9i91sh65vjcb-288 mysql-test/suite/pbxt/t/pbxt_transactions.test pbxt_transactions.te-20090402100035-4ilk9i91sh65vjcb-291 mysql-test/suite/pbxt/t/ps_1general.test ps_1general.test-20090402100035-4ilk9i91sh65vjcb-297 mysql-test/suite/rpl/r/rpl_err_ignoredtable.result sp1f-rpl_error_ignored_ta-20030708095945-wtbf3wai2clqedywdvwntfdfwmloumec mysql-test/suite/rpl/r/rpl_extraCol_innodb.result sp1f-rpl_extracol_innodb.-20061103140439-oyaqsdcl3ymjfl5y2wvwjz3cgb36dbj3 mysql-test/suite/rpl/r/rpl_extraCol_myisam.result sp1f-rpl_extracol_myisam.-20061103140439-ipxcnvlavhkichgny6fvkejbdgnvudtd mysql-test/suite/rpl/r/rpl_get_lock.result sp1f-rpl_get_lock.result-20010831215549-yggpnug7jmur7hftd3ln6ytpungqckpy mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result sp1f-rpl000002.result-20001118063528-dp4vigctbaz5p7s7r7cqtgabk25a5j3m mysql-test/suite/rpl/r/rpl_relayspace.result sp1f-rpl_relayspace.resul-20030317215153-kx422hojs2xkiqciwgt7jps2hdk376fb mysql-test/suite/rpl/r/rpl_row_create_table.result sp1f-rpl_row_create_table-20051222053452-uud3ktz3erqptqb64rkh7ftoo7bdbf6c mysql-test/suite/rpl/r/rpl_row_tabledefs_2myisam.result sp1f-rpl_row_tabledefs.re-20051222053457-perubbeq3fwsqe5phfwcpsstjqltqrnj mysql-test/suite/rpl/r/rpl_row_tabledefs_3innodb.result sp1f-rpl_row_tabledefs_3i-20060508180502-wvvscuvjv34fiuhqjuhi6yvk6cwbmrpz mysql-test/suite/rpl/r/rpl_stm_000001.result sp1f-rpl000001.result-20010116163624-seoa5zygxq5ibscm6kld7cneoimbmer4 mysql-test/suite/rpl/r/rpl_temporary.result sp1f-rpl_temporary.result-20021229214238-uwyas6jaay7ygaqsdwolzlcec6reyckw mysql-test/suite/rpl/r/rpl_trigger.result sp1f-rpl_trigger.result-20050815151505-durmghxr6fqgzgolaezf6gfotoeptwhz mysql-test/suite/rpl/t/disabled.def sp1f-disabled.def-20070627122758-vdqevwzhnizicdrxrmfy4w4afgprx46x mysql-test/suite/rpl/t/rpl_err_ignoredtable.test sp1f-rpl_error_ignored_ta-20030708095933-nrriw3pbfsfrugbgvjpriczjb3dwm4mn mysql-test/suite/rpl/t/rpl_get_lock.test sp1f-rpl_get_lock.test-20010831215549-mxwhygd7dfgxcx4dnnjjomunk7oojsj5 mysql-test/suite/rpl/t/rpl_mixed_ddl_dml.test sp1f-rpl000002.test-20001118063528-xtihamqla2qxwkn544mamd5mlt5pev33 mysql-test/suite/rpl/t/rpl_relayspace.test sp1f-rpl_relayspace.test-20030317215153-aincxws3k2fb4ojvtowjzgandznovi7b mysql-test/suite/rpl/t/rpl_rotate_logs.test sp1f-rpl000016.test-20001215004309-uqid5ejphbyjwielf3t6nd7523ynp353 mysql-test/suite/rpl/t/rpl_row_create_table.test sp1f-rpl_row_create_table-20051222053501-6a6pxustyjnj6swfd4bbsptzllxov5ao mysql-test/suite/rpl/t/rpl_temporary.test sp1f-rpl_temporary.test-20021229214239-nxqbr5fvrk5sm3d5xf2gwzhfim3xdy7k mysql-test/suite/rpl/t/rpl_trigger.test sp1f-rpl_trigger.test-20050815151506-5wtqt7aazfxwebrvi37du2prbglo4lmq mysql-test/suite/rpl_ndb/r/rpl_ndb_extraCol.result sp1f-rpl_ndb_extracol.res-20061103140449-ucihyswq7mtsamyjm2whggyjxyfekxeo mysql-test/t/archive.test sp1f-archive.test-20040525194738-qla5yawytktcj3tlbgrlhvf3thbo6ghq mysql-test/t/delayed.test sp1f-delayed.test-20001228015635-nfs6w3ic7qt55pnm6uld7wmmq4p73afq mysql-test/t/delete.test sp1f-delete.test-20001228015635-7lhk263y3s3wild7htgoaesssx5wdy4s mysql-test/t/disabled.def sp1f-disabled.def-20050315184020-inpdp4hiogithilv62snllppjz2dcing mysql-test/t/fulltext.test sp1f-fulltext.test-20001228015635-snfzkwn2snrqit5pagdg2vzhcoa56eea mysql-test/t/func_group.test sp1f-func_group.test-20001228015635-wkz277djccbddkitm63hibutxp7o4rb7 mysql-test/t/func_misc.test sp1f-func_misc.test-20001228015635-kayguwcdgtjnekzavvdzbsnqcdwfm36c mysql-test/t/grant.test sp1f-grant.test-20020905131705-iadu5zcjshnxgtjx7qpmfrs77bl75suy mysql-test/t/grant2.test sp1f-grant2.test-20030722200048-galnas2hib5h2ygo4rzcnpblby7awdow mysql-test/t/group_min_max.test sp1f-group_min_max.test-20040827133612-bbe7hj6l7byvtyxsg4iicylzflsgy6vj mysql-test/t/innodb-analyze.test innodbanalyze.test-20081203050234-edoolglm28lyejuc-6 mysql-test/t/innodb-autoinc.test innodbautoinc.test-20081201061010-zymrrwrczns2vrex-281 mysql-test/t/innodb-index.test innodbindex.test-20081201061010-zymrrwrczns2vrex-285 mysql-test/t/innodb-master.opt innodbmaster.opt-20081201061010-zymrrwrczns2vrex-290 mysql-test/t/innodb-semi-consistent-master.opt innodbsemiconsistent-20081201061010-zymrrwrczns2vrex-293 mysql-test/t/innodb-timeout.test innodbtimeout.test-20081203050234-edoolglm28lyejuc-8 mysql-test/t/innodb-use-sys-malloc-master.opt innodbusesysmallocma-20090326061054-ylrdb8libxw6u7e9-3 mysql-test/t/innodb-zip.test innodbzip.test-20081201061010-zymrrwrczns2vrex-297 mysql-test/t/innodb.test innodb.test-20081201061010-zymrrwrczns2vrex-299 mysql-test/t/innodb_bug34300.test innodb_bug34300.test-20081201061010-zymrrwrczns2vrex-303 mysql-test/t/innodb_bug36169.test innodb_bug36169.test-20081201061010-zymrrwrczns2vrex-307 mysql-test/t/innodb_bug36172.test innodb_bug36172.test-20081203050234-edoolglm28lyejuc-10 mysql-test/t/innodb_bug42101-nonzero-master.opt innodb_bug42101nonze-20090519075917-c0hbhca1f80pmx80-4 mysql-test/t/innodb_bug44369.test innodb_bug44369.test-20091005111405-nbp5t33h95jrqha2-2 mysql-test/t/innodb_file_format.test innodb_file_format.t-20090923000535-ke95wdd4zn27df71-19 mysql-test/t/innodb_information_schema.test innodb_information_s-20081201061010-zymrrwrczns2vrex-309 mysql-test/t/innodb_lock_wait_timeout_1.test bug40113.test-20090619150423-w3im08cym6tyzn8f-2 mysql-test/t/innodb_mysql.test sp1f-innodb_mysql.test-20060816102624-6ymo37d3nyhvbqyzqn5ohsfuydwo426k mysql-test/t/innodb_xtradb_bug317074.test innodb_xtradb_bug317-20090326061054-ylrdb8libxw6u7e9-9 mysql-test/t/mysql.test sp1f-mysql.test-20050517191330-gc7zxd3q7cgw4g3pdnswpnxwqnvqhwks mysql-test/t/mysqltest.test sp1f-mysqltest.test-20041022024800-v3hvkzs4236l6rpunai7xttdltot7rvz mysql-test/t/olap.test sp1f-olap.test-20020720115151-u3y5qjyyz4c7hufu5vftj74rijkr7rf2 mysql-test/t/openssl_1.test sp1f-ssl.test-20010831211355-mk47pipvythsqcor32yidzoopgdewdo6 mysql-test/t/order_by.test sp1f-order_by.test-20001228015636-nr7aml75ra7mdlruhoqo5dgbfv5tcesc mysql-test/t/partition.test sp1f-partition.test-20050718113034-pbo3ht3bf4gfa3mz44on3sqafyctwo35 mysql-test/t/partition_innodb_semi_consistent.test partition_innodb_sem-20081216114001-2cqkultf4k3xhbvc-2 mysql-test/t/query_cache.test sp1f-query_cache.test-20011205230530-yfwho76ujeasygr3magwlmssnvwsukio mysql-test/t/query_cache_debug.test sp1f-query_cache_debug.te-20080107200614-idvgytisf3mqftabyk43v42cynhijq5h mysql-test/t/query_cache_notembedded.test sp1f-query_cache_notembed-20050729121335-367lhbc36drodp262lkuott3pk25wcdt mysql-test/t/range.test sp1f-range.test-20001228015636-xfak6bsaw5p3ek36np7bznadjb3boh2q mysql-test/t/select.test sp1f-select.test-20010103001548-tbl2ff7qehzh43qnsmf4ejhjqe66f46n mysql-test/t/show_check-master.opt sp1f-show_checkmaster.opt-20061004031826-m2pj2wv7l6njctrnpaenfdqxhckyfxpz mysql-test/t/show_check.test sp1f-show_check.test-20001228015637-uv35wm2ryvpkyrr6ojhmi2nq6x6jgdod mysql-test/t/sp-destruct.test sp1f-spdestruct.test-20051026133448-bt2vu42upsulrlap6vytgz7hygx6a2hj mysql-test/t/sp-security.test sp1f-spsecurity.test-20031213154048-snbqkvepvo4c45wtxld2qrc3h35ap4ty mysql-test/t/sp.test sp1f-sp.test-20030117133803-b6pcfv2yscbqkur5fszep7acmdg7nf5k mysql-test/t/sp_notembedded.test sp1f-sp_notembedded.test-20060224163411-4bxzhibgkpu3fm3zoyvknrqo3zudvvfa mysql-test/t/trigger.test sp1f-trigger.test-20040907122911-eamsjnplirl554ohkncdnwi765xm2hbk mysql-test/t/trigger_notembedded.test sp1f-triggergrant.test-20051110192456-j6hwzoi4loitpk57ccqotlhkzrm6ucsv mysql-test/t/type_newdecimal.test sp1f-type_newdecimal.test-20050208224936-e244l5ugrk3oditjqp53n6izptkrteq2 mysql-test/t/type_year.test sp1f-type_year.test-20001228015637-j547qmpytndiwdwgn35oq34jgjduzo6l mysql-test/t/udf.test sp1f-udf.test-20060215161120-inrv7ph3327gnzcvcqk25vmihneybyhk mysql-test/t/variables.test sp1f-variables.test-20001228015637-u4toadkin7aellpwwz75e5h5zuutteid mysql-test/valgrind.supp sp1f-valgrind.supp-20050406142216-yg7xhezklqhgqlc3inx36vbghodhbovy mysys/my_file.c sp1f-my_file.c-20040219173304-xky3nl63gme3w2apldzfpufwgcfnq23x mysys/my_getopt.c sp1f-my_getopt.c-20020125212008-5ppwsdqmfhny46gxkjxph22zh3phetir mysys/my_sync.c sp1f-my_sync.c-20031102135456-o4s6sunug6w2ch4bok2p3auq37qgqzox mysys/my_thr_init.c sp1f-my_thr_init.c-19700101030959-siwg2eavxsdwdc4kkmwxvs42rp6ntkrm scripts/make_win_bin_dist sp1f-make_win_bin_dist-20060901123056-xnusgszvkfrrcxkqidb7zszax2ezpyto scripts/mysql_secure_installation.pl.in sp1f-mysql_secure_install-20071228215050-nnco3kgp33fxs7ja6zdy6xh56zszi2cc scripts/mysql_secure_installation.sh sp1f-mysql_secure_install-20020609041239-b4zztmtqycjs24aubuhwun6np5wuuesy sql-common/client.c sp1f-client.c-20030502160736-oraaciqy6jkfwygs6tqfoaxgjbi65yo7 sql/CMakeLists.txt sp1f-cmakelists.txt-20060831175237-esoeu5kpdtwjvehkghwy6fzbleniq2wy sql/event_db_repository.cc sp1f-event_db_repository.-20060627064838-k6rpjg72omnihtxhbubu6ht7wjvxggb7 sql/field.cc sp1f-field.cc-19700101030959-f4imaofclsea3n4fj4ow5m7havmyxa2r sql/field.h sp1f-field.h-19700101030959-3n6smzxcwkjl7bikm3wg4hfkjn66uvvp sql/ha_ndbcluster.cc sp1f-ha_ndbcluster.cc-20040414175836-rvqnoxrkqexyhfu3d62s4t345ip7rez2 sql/handler.h sp1f-handler.h-19700101030959-mumq2hpilkpgxuf22ftyv5kbilysnzvn sql/item.cc sp1f-item.cc-19700101030959-u7hxqopwpfly4kf5ctlyk2dvrq4l3dhn sql/item.h sp1f-item.h-19700101030959-rrkb43htudd62batmoteashkebcwykpa sql/item_cmpfunc.cc sp1f-item_cmpfunc.cc-19700101030959-hrk7pi2n6qpwxauufnkizirsoucdcx2e sql/item_cmpfunc.h sp1f-item_cmpfunc.h-19700101030959-pcvbjplo4e4ng7ibynfhcd6pjyem57gr sql/item_create.cc sp1f-item_create.cc-19700101030959-zdsezbi5r5xu5syntjdzqs2d2dswsojn sql/item_func.cc sp1f-item_func.cc-19700101030959-3wmsx76yvc25sroqpfrx2n77kqdxxn3y sql/item_func.h sp1f-item_func.h-19700101030959-fbjcbwkg66qubbzptqwh5w5evhnpukze sql/item_geofunc.cc sp1f-item_geofunc.cc-20030530102226-vdbf2bd6tpkrzoy6q2wdibkzd3bkv2io sql/item_strfunc.cc sp1f-item_strfunc.cc-19700101030959-yl2pwnrngmla3nmlgiuiwrztx3iu4ffl sql/item_subselect.cc sp1f-item_subselect.cc-20020512204640-qep43aqhsfrwkqmrobni6czc3fqj36oo sql/item_subselect.h sp1f-item_subselect.h-20020512204640-qdg77wil56cxyhtc2bjjdrppxq3wqgh3 sql/item_sum.cc sp1f-item_sum.cc-19700101030959-4woo23bi3am2t2zvsddqbpxk7xbttdkm sql/item_sum.h sp1f-item_sum.h-19700101030959-ecgohlekwm355wxl5fv4zzq3alalbwyl sql/item_timefunc.cc sp1f-item_timefunc.cc-19700101030959-rvvlgmw5b4ewpuuxuntrkiqimyrr5sw2 sql/item_xmlfunc.cc sp1f-item_xmlfunc.cc-20051221130500-wo5dgojvjjm6mmra7fay3ri7ud5ow3yl sql/log.cc sp1f-log.cc-19700101030959-r3hdfovek4kl6nd64ovoaknmirota6bq sql/log_event.cc sp1f-log_event.cc-19700101030959-msmqlflsngxosswid2hpzxly5vfqdddc sql/log_event.h sp1f-log_event.h-19700101030959-clq6ett55tcqbpys2i4cpfrdccq7j4om sql/mysqld.cc sp1f-mysqld.cc-19700101030959-zpswdvekpvixxzxf7gdtofzel7nywtfj sql/net_serv.cc sp1f-net_serv.cc-19700101030959-dp4s27g5nk64sph4g6g54dghekqozzmy sql/opt_range.cc sp1f-opt_range.cc-19700101030959-afe3wtevb7zwrg4xyibt35uamov5r7ds sql/repl_failsafe.cc sp1f-repl_failsafe.cc-20011010025623-k7zhoyc3smc7tbliyp7vaf3f4idq22so sql/rpl_record.cc sp1f-rpl_record.cc-20070413125523-wuthuk5jk7uxikuioz6esll6xakdngs4 sql/rpl_record.h sp1f-rpl_record.h-20070413125523-xvn32ub2xcvqged7y6ayilghjetpvkvg sql/rpl_rli.cc sp1f-rpl_rli.cc-20061031112305-25t7pxjrjm24qo5h65c7rml66xu3uw4p sql/rpl_tblmap.cc sp1f-rpl_tblmap.cc-20051222053448-sgowtys7fb4tdpjvmzwktjxmb5krm3cc sql/set_var.cc sp1f-set_var.cc-20020723153119-nwbpg2pwpz55pfw7yfzaxt7hsszzy7y3 sql/slave.cc sp1f-slave.cc-19700101030959-a636aj3mjxgu7fnznrg5kt77p3u2bvhh sql/sp.cc sp1f-sp.cc-20021212121421-6xwuvxq5bku2b4yv655kp2e5gsvautd5 sql/sp.h sp1f-sp.h-20021212121421-eh5y7kpcb3hkgy4wjuh3q3non36itye5 sql/sp_cache.cc sp1f-sp_cache.cc-20030703140129-ugsn54s2jpxh7hdznsgxn6ubwvbtj5hw sql/sp_head.cc sp1f-sp_head.cc-20021208185920-jtgc5wvyqdnu2gvcdus3gazrfhxbofxd sql/sp_head.h sp1f-sp_head.h-20021208185920-yrolg3rzamehfoejkbiai4q7njg5w6cd sql/sp_rcontext.cc sp1f-sp_rcontext.cc-20030916122605-vg62h2qkjnbj54b4zlijgqswudijiyaf sql/sql_acl.cc sp1f-sql_acl.cc-19700101030959-c4hku3uqxzujthqnndeprbrhamqy6a4i sql/sql_acl.h sp1f-sql_acl.h-19700101030959-byf4bn7yfbxu6wa6z76kqcuspjl67msj sql/sql_base.cc sp1f-sql_base.cc-19700101030959-w7tul2gb2n4jzayjwlslj3ybmf3uhk6a sql/sql_cache.cc sp1f-sql_cache.cc-19700101030959-74bsqwcnhboovijsogcenqana5inu6wo sql/sql_cache.h sp1f-sql_cache.h-20011202123401-gegktsz2a3er7fqwpmpoejydzpkeadeo sql/sql_class.cc sp1f-sql_class.cc-19700101030959-rpotnweaff2pikkozh3butrf7mv3oero sql/sql_connect.cc sp1f-sql_connect.cc-20070223111352-fhh5znxdfvzxuca7da3uu4olnwgkrm4n sql/sql_delete.cc sp1f-sql_delete.cc-19700101030959-ch2a6r6ushvc2vfwxt7ehcjuplelwthr sql/sql_insert.cc sp1f-sql_insert.cc-19700101030959-xgwqe5svnimxudzdcuitauljzz2zjk5g sql/sql_load.cc sp1f-sql_load.cc-19700101030959-hoqlay5we4yslrw23xqedulkejw6a3o5 sql/sql_parse.cc sp1f-sql_parse.cc-19700101030959-ehcre3rwhv5l3mlxqhaxg36ujenxnrcd sql/sql_partition.cc sp1f-sql_partition.cc-20050718113038-57h5bzswps6cel2y7k7qideue3ghbg3u sql/sql_plugin.cc sp1f-sql_plugin.cc-20051105112032-hrm64p6xfjq33ud6zy3uivpo7azm75a2 sql/sql_profile.cc sp1f-sql_profile.cc-20070222150305-yv5grcusm3k2b6rrcx3kkqggtm33i3z4 sql/sql_profile.h sp1f-sql_profile.h-20070222150305-tbdpljkisvi3e657yunhkvsnjw6ifjru sql/sql_select.cc sp1f-sql_select.cc-19700101030959-egb7whpkh76zzvikycs5nsnuviu4fdlb sql/sql_show.cc sp1f-sql_show.cc-19700101030959-umlljfnpplg452h7reeyqr4xnbmlkvfj sql/sql_table.cc sp1f-sql_table.cc-19700101030959-tzdkvgigezpuaxnldqh3fx2h7h2ggslu sql/sql_yacc.yy sp1f-sql_yacc.yy-19700101030959-wvn4qyy2drpmge7kaq3dysprbhlrv27j sql/table.cc sp1f-table.cc-19700101030959-nsxtem2adyqzwe6nz4cgrpcmts3o54v7 sql/table.h sp1f-table.h-19700101030959-dv72bajftxj5fbdjuajquappanuv2ija sql/udf_example.def sp1f-udf_example.def-20060922124240-fxmt4egcapnpdlbg5u4xlrq4bppsjcnw storage/archive/CMakeLists.txt* sp1f-cmakelists.txt-20060324221904-eeglvbzhfjgzaeenavdxsdvjzfjatvre storage/archive/azio.c sp1f-azio.c-20051223034959-gjinfrr75cpes7iyw4uasbps2bkn65rc storage/archive/azlib.h sp1f-azlib.h-20051223035000-23fs3st2s6whj3xl4hjzfxuzsjyzgnf6 storage/archive/ha_archive.cc sp1f-ha_archive.cc-20040521001938-uy57z43drkjeirpjafdzdpvfxruqho4q storage/federated/CMakeLists.txt* sp1f-cmakelists.txt-20060819155453-lvbabhbwyj6n6qzjroyxasefis7wczs5 storage/innobase/btr/btr0btr.c sp1f-btr0btr.c-20010217121858-mwrlor7vioqhw732pmqhbjjhpajttjyp storage/innobase/data/data0type.c sp1f-data0type.c-20010217121859-zfpknsetjkfoyjnhomi2z2ek3jda2wji storage/innobase/handler/ha_innodb.cc sp1f-ha_innobase.cc-20001205235417-rlet3ei56gdrss673dssnrqgug67lwno storage/innobase/include/ha_prototypes.h sp1f-ha_prototypes.h-20060601063356-7fsbop6gnnzwnq7pndelgmztdm2uygg2 storage/innobase/include/mach0data.h sp1f-mach0data.h-20010217121903-66grfy3b6skgudvdli4owdshxnvcjo2o storage/innobase/include/mach0data.ic sp1f-mach0data.ic-20010217121903-n2kjcumh2n3j5ydrwt5cllgns7c4kpyw storage/innobase/include/os0file.h sp1f-os0file.h-20010217121904-5jy6i4kwdivoji2tzlssz6tj3hmntl4c storage/innobase/include/trx0trx.h sp1f-trx0trx.h-20010217121908-in6xvc2qnxbrz74ytwthpviclfo53zls storage/innobase/lock/lock0lock.c sp1f-lock0lock.c-20010217121909-slawfq55wuitbo2ujz7h5doebhwxfmhe storage/innobase/os/os0file.c sp1f-os0file.c-20010217121911-3wbmvsjzaw7z42arqr46leh46lbamvcp storage/innobase/row/row0sel.c sp1f-row0sel.c-20010217121914-c6o7vqncdgzrorm4pko5tpdlfeyujhvq storage/innobase/trx/trx0trx.c sp1f-trx0trx.c-20010217121916-b5g7pmqxezfo2mktfpeiapibgdhogrv2 storage/innobase/ut/ut0ut.c sp1f-ut0ut.c-20010217121917-7nqugyaqeu7yecvgv73lm354njkx74zq storage/innodb_plugin/CMakeLists.txt cmakelists.txt-20090527093836-7v4wb2xxka10h4d0-2 storage/innodb_plugin/ChangeLog changelog-20090527093836-7v4wb2xxka10h4d0-4 storage/innodb_plugin/btr/btr0btr.c btr0btr.c-20090527093836-7v4wb2xxka10h4d0-46 storage/innodb_plugin/btr/btr0sea.c btr0sea.c-20090527093836-7v4wb2xxka10h4d0-49 storage/innodb_plugin/buf/buf0buf.c buf0buf.c-20090527093836-7v4wb2xxka10h4d0-51 storage/innodb_plugin/data/data0type.c data0type.c-20090527093836-7v4wb2xxka10h4d0-56 storage/innodb_plugin/dict/dict0dict.c dict0dict.c-20090527093836-7v4wb2xxka10h4d0-59 storage/innodb_plugin/fil/fil0fil.c fil0fil.c-20090527093836-7v4wb2xxka10h4d0-65 storage/innodb_plugin/handler/ha_innodb.cc ha_innodb.cc-20090527093836-7v4wb2xxka10h4d0-72 storage/innodb_plugin/handler/ha_innodb.h ha_innodb.h-20090527093836-7v4wb2xxka10h4d0-73 storage/innodb_plugin/handler/handler0alter.cc handler0alter.cc-20090527093836-7v4wb2xxka10h4d0-74 storage/innodb_plugin/ibuf/ibuf0ibuf.c ibuf0ibuf.c-20090527093836-7v4wb2xxka10h4d0-80 storage/innodb_plugin/include/btr0sea.h btr0sea.h-20090527093836-7v4wb2xxka10h4d0-87 storage/innodb_plugin/include/db0err.h db0err.h-20090527093836-7v4wb2xxka10h4d0-105 storage/innodb_plugin/include/dict0dict.h dict0dict.h-20090527093836-7v4wb2xxka10h4d0-110 storage/innodb_plugin/include/fil0fil.h fil0fil.h-20090527093836-7v4wb2xxka10h4d0-123 storage/innodb_plugin/include/ha_prototypes.h ha_prototypes.h-20090527093836-7v4wb2xxka10h4d0-134 storage/innodb_plugin/include/ibuf0ibuf.h ibuf0ibuf.h-20090527093836-7v4wb2xxka10h4d0-138 storage/innodb_plugin/include/lock0lock.h lock0lock.h-20090527093836-7v4wb2xxka10h4d0-142 storage/innodb_plugin/include/log0log.h log0log.h-20090527093836-7v4wb2xxka10h4d0-147 storage/innodb_plugin/include/log0recv.h log0recv.h-20090527093836-7v4wb2xxka10h4d0-149 storage/innodb_plugin/include/mem0mem.h mem0mem.h-20090527093836-7v4wb2xxka10h4d0-155 storage/innodb_plugin/include/mem0pool.h mem0pool.h-20090527093836-7v4wb2xxka10h4d0-157 storage/innodb_plugin/include/os0file.h os0file.h-20090527093836-7v4wb2xxka10h4d0-165 storage/innodb_plugin/include/pars0pars.h pars0pars.h-20090527093836-7v4wb2xxka10h4d0-182 storage/innodb_plugin/include/srv0srv.h srv0srv.h-20090527093836-7v4wb2xxka10h4d0-223 storage/innodb_plugin/include/thr0loc.h thr0loc.h-20090527093836-7v4wb2xxka10h4d0-233 storage/innodb_plugin/include/trx0i_s.h trx0i_s.h-20090527093836-7v4wb2xxka10h4d0-235 storage/innodb_plugin/include/trx0purge.h trx0purge.h-20090527093836-7v4wb2xxka10h4d0-236 storage/innodb_plugin/include/trx0rseg.h trx0rseg.h-20090527093836-7v4wb2xxka10h4d0-242 storage/innodb_plugin/include/trx0sys.h trx0sys.h-20090527093836-7v4wb2xxka10h4d0-244 storage/innodb_plugin/include/trx0trx.h trx0trx.h-20090527093836-7v4wb2xxka10h4d0-246 storage/innodb_plugin/include/trx0undo.h trx0undo.h-20090527093836-7v4wb2xxka10h4d0-249 storage/innodb_plugin/include/univ.i univ.i-20090527093836-7v4wb2xxka10h4d0-252 storage/innodb_plugin/include/usr0sess.h usr0sess.h-20090527093836-7v4wb2xxka10h4d0-253 storage/innodb_plugin/lock/lock0lock.c lock0lock.c-20090527093836-7v4wb2xxka10h4d0-274 storage/innodb_plugin/log/log0log.c log0log.c-20090527093836-7v4wb2xxka10h4d0-275 storage/innodb_plugin/log/log0recv.c log0recv.c-20090527093836-7v4wb2xxka10h4d0-276 storage/innodb_plugin/mem/mem0dbg.c mem0dbg.c-20090527093836-7v4wb2xxka10h4d0-278 storage/innodb_plugin/mem/mem0pool.c mem0pool.c-20090527093836-7v4wb2xxka10h4d0-280 storage/innodb_plugin/os/os0file.c os0file.c-20090527093836-7v4wb2xxka10h4d0-338 storage/innodb_plugin/os/os0sync.c os0sync.c-20090527093836-7v4wb2xxka10h4d0-340 storage/innodb_plugin/os/os0thread.c os0thread.c-20090527093836-7v4wb2xxka10h4d0-341 storage/innodb_plugin/pars/lexyy.c lexyy.c-20090527093836-7v4wb2xxka10h4d0-345 storage/innodb_plugin/pars/pars0lex.l pars0lex.l-20090527093836-7v4wb2xxka10h4d0-350 storage/innodb_plugin/que/que0que.c que0que.c-20090527093836-7v4wb2xxka10h4d0-354 storage/innodb_plugin/row/row0merge.c row0merge.c-20090527093836-7v4wb2xxka10h4d0-360 storage/innodb_plugin/row/row0mysql.c row0mysql.c-20090527093836-7v4wb2xxka10h4d0-361 storage/innodb_plugin/srv/srv0srv.c srv0srv.c-20090527093836-7v4wb2xxka10h4d0-373 storage/innodb_plugin/srv/srv0start.c srv0start.c-20090527093836-7v4wb2xxka10h4d0-374 storage/innodb_plugin/sync/sync0arr.c sync0arr.c-20090527093836-7v4wb2xxka10h4d0-375 storage/innodb_plugin/sync/sync0sync.c sync0sync.c-20090527093836-7v4wb2xxka10h4d0-377 storage/innodb_plugin/thr/thr0loc.c thr0loc.c-20090527093836-7v4wb2xxka10h4d0-378 storage/innodb_plugin/trx/trx0i_s.c trx0i_s.c-20090527093836-7v4wb2xxka10h4d0-379 storage/innodb_plugin/trx/trx0purge.c trx0purge.c-20090527093836-7v4wb2xxka10h4d0-380 storage/innodb_plugin/trx/trx0rseg.c trx0rseg.c-20090527093836-7v4wb2xxka10h4d0-383 storage/innodb_plugin/trx/trx0sys.c trx0sys.c-20090527093836-7v4wb2xxka10h4d0-384 storage/innodb_plugin/trx/trx0trx.c trx0trx.c-20090527093836-7v4wb2xxka10h4d0-385 storage/innodb_plugin/trx/trx0undo.c trx0undo.c-20090527093836-7v4wb2xxka10h4d0-386 storage/innodb_plugin/usr/usr0sess.c usr0sess.c-20090527093836-7v4wb2xxka10h4d0-387 storage/innodb_plugin/ut/ut0mem.c ut0mem.c-20090527093836-7v4wb2xxka10h4d0-395 storage/maria/ft_maria.c sp1f-ft_maria.c-20060411134407-c7zixlxmx36vm37l35blmgkjeq5e2zgv storage/maria/ma_blockrec.c sp1f-ma_blockrec.c-20070118193810-5wtbfa4irhu4voa3diiuus5km2j6jvlv storage/maria/ma_check.c sp1f-ma_check.c-20060411134408-m5d5jao4sr32xsjjkig2uhdndqm5cgba storage/maria/ma_close.c sp1f-ma_close.c-20060411134409-5c3eq7j6oloex4c4hrvcqrsuvz7xohev storage/maria/ma_create.c sp1f-ma_create.c-20060411134410-ozzigempkjj2kdgxfbasiwfjzwjejevd storage/maria/ma_loghandler.c sp1f-ma_loghandler.c-20070202074129-utpzp3km4lrxldm2tdhejae2zy6zlmhq storage/maria/ma_test3.c sp1f-ma_test3.c-20060411134447-llbsdlhu2zyxbt6taoa2lsts7snaic2j storage/myisam/ft_boolean_search.c sp1f-ft_boolean_search.c-20010411110351-pu6lfsyiumvnnewko2oqbyjz6g3q4xm3 storage/myisam/ft_myisam.c sp1f-ft_myisam.c-20060411134458-uct2l3bly6nrej2hilx4explav2i3kgv storage/myisam/ft_nlq_search.c sp1f-ft_nlq_search.c-20010411110351-a7dhoojgfpsydi5k4qawswaatmakqe7b storage/myisam/ft_parser.c sp1f-ft_parser.c-19700101030959-goim35zn24ujo7rbznobwhhw5r3lemab storage/myisam/ft_stopwords.c sp1f-ft_stopwords.c-19700101030959-vgask5ebyzpaoa7j37ybfnjhx4rkzm63 storage/myisam/ftdefs.h sp1f-ftdefs.h-19700101030959-c5sgpgnpbutzv5fvbe6a63x6up2niz2p storage/myisam/mi_check.c sp1f-mi_check.c-19700101030959-yzbhnjgzcmqdyj4zz5codhkkw5eedp6f storage/myisam/mi_create.c sp1f-mi_create.c-19700101030959-i6lazhpsyf7ggr2yjukf6xxybhraxup3 storage/myisam/mi_test3.c sp1f-mi_test3.c-19700101030959-3yn4wc53noyuhc4nphffr4hxesodgfyf storage/pbxt/ChangeLog changelog-20090326121724-x683v32twzr3fi0y-3 storage/pbxt/plug.in plug.in-20090326121724-x683v32twzr3fi0y-9 storage/pbxt/src/Makefile.am makefile.am-20090326121724-x683v32twzr3fi0y-13 storage/pbxt/src/discover_xt.cc discover_xt.cc-20090326121724-x683v32twzr3fi0y-26 storage/pbxt/src/ha_pbxt.cc ha_pbxt.cc-20090326121724-x683v32twzr3fi0y-30 storage/pbxt/src/restart_xt.cc restart_xt.cc-20090326121724-x683v32twzr3fi0y-53 storage/pbxt/src/strutil_xt.cc strutil_xt.cc-20090326121724-x683v32twzr3fi0y-59 storage/pbxt/src/table_xt.cc table_xt.cc-20090326121724-x683v32twzr3fi0y-65 storage/pbxt/src/thread_xt.cc thread_xt.cc-20090326121724-x683v32twzr3fi0y-67 storage/pbxt/src/thread_xt.h thread_xt.h-20090326121724-x683v32twzr3fi0y-68 storage/pbxt/src/trace_xt.cc trace_xt.cc-20090326121724-x683v32twzr3fi0y-69 storage/xtradb/CMakeLists.txt cmakelists.txt-20081201061010-zymrrwrczns2vrex-1 storage/xtradb/ChangeLog changelog-20081201061010-zymrrwrczns2vrex-3 storage/xtradb/Makefile.am makefile.am-20081201061010-zymrrwrczns2vrex-4 storage/xtradb/btr/btr0btr.c btr0btr.c-20081201061010-zymrrwrczns2vrex-44 storage/xtradb/btr/btr0sea.c btr0sea.c-20081201061010-zymrrwrczns2vrex-47 storage/xtradb/buf/buf0buddy.c buf0buddy.c-20081201061010-zymrrwrczns2vrex-48 storage/xtradb/buf/buf0buf.c buf0buf.c-20081201061010-zymrrwrczns2vrex-49 storage/xtradb/buf/buf0flu.c buf0flu.c-20081201061010-zymrrwrczns2vrex-50 storage/xtradb/buf/buf0lru.c buf0lru.c-20081201061010-zymrrwrczns2vrex-51 storage/xtradb/buf/buf0rea.c buf0rea.c-20081201061010-zymrrwrczns2vrex-52 storage/xtradb/data/data0type.c data0type.c-20081201061010-zymrrwrczns2vrex-54 storage/xtradb/dict/dict0crea.c dict0crea.c-20081201061010-zymrrwrczns2vrex-56 storage/xtradb/dict/dict0dict.c dict0dict.c-20081201061010-zymrrwrczns2vrex-57 storage/xtradb/fil/fil0fil.c fil0fil.c-20081201061010-zymrrwrczns2vrex-63 storage/xtradb/fsp/fsp0fsp.c fsp0fsp.c-20081201061010-zymrrwrczns2vrex-64 storage/xtradb/handler/ha_innodb.cc ha_innodb.cc-20081201061010-zymrrwrczns2vrex-70 storage/xtradb/handler/ha_innodb.h ha_innodb.h-20081201061010-zymrrwrczns2vrex-71 storage/xtradb/handler/handler0alter.cc handler0alter.cc-20081201061010-zymrrwrczns2vrex-72 storage/xtradb/handler/i_s.cc i_s.cc-20081201061010-zymrrwrczns2vrex-73 storage/xtradb/handler/i_s.h i_s.h-20081201061010-zymrrwrczns2vrex-74 storage/xtradb/handler/innodb_patch_info.h innodb_patch_info.h-20081206234022-hep6ryfeacyr0572-1 storage/xtradb/ibuf/ibuf0ibuf.c ibuf0ibuf.c-20081201061010-zymrrwrczns2vrex-76 storage/xtradb/include/btr0cur.h btr0cur.h-20081201061010-zymrrwrczns2vrex-79 storage/xtradb/include/btr0sea.h btr0sea.h-20081201061010-zymrrwrczns2vrex-83 storage/xtradb/include/buf0buf.h buf0buf.h-20081201061010-zymrrwrczns2vrex-88 storage/xtradb/include/buf0buf.ic buf0buf.ic-20081201061010-zymrrwrczns2vrex-89 storage/xtradb/include/buf0lru.h buf0lru.h-20081201061010-zymrrwrczns2vrex-92 storage/xtradb/include/buf0rea.h buf0rea.h-20081201061010-zymrrwrczns2vrex-94 storage/xtradb/include/buf0types.h buf0types.h-20081201061010-zymrrwrczns2vrex-95 storage/xtradb/include/db0err.h db0err.h-20081201061010-zymrrwrczns2vrex-101 storage/xtradb/include/dict0crea.h dict0crea.h-20081201061010-zymrrwrczns2vrex-104 storage/xtradb/include/dict0dict.h dict0dict.h-20081201061010-zymrrwrczns2vrex-106 storage/xtradb/include/dict0mem.h dict0mem.h-20081201061010-zymrrwrczns2vrex-110 storage/xtradb/include/fil0fil.h fil0fil.h-20081201061010-zymrrwrczns2vrex-119 storage/xtradb/include/fsp0fsp.h fsp0fsp.h-20081201061010-zymrrwrczns2vrex-120 storage/xtradb/include/ibuf0ibuf.h ibuf0ibuf.h-20081201061010-zymrrwrczns2vrex-134 storage/xtradb/include/lock0lock.h lock0lock.h-20081201061010-zymrrwrczns2vrex-138 storage/xtradb/include/log0log.h log0log.h-20081201061010-zymrrwrczns2vrex-143 storage/xtradb/include/log0log.ic log0log.ic-20081201061010-zymrrwrczns2vrex-144 storage/xtradb/include/log0recv.h log0recv.h-20081201061010-zymrrwrczns2vrex-145 storage/xtradb/include/mem0mem.h mem0mem.h-20081201061010-zymrrwrczns2vrex-151 storage/xtradb/include/mem0pool.h mem0pool.h-20081201061010-zymrrwrczns2vrex-153 storage/xtradb/include/mtr0mtr.h mtr0mtr.h-20081201061010-zymrrwrczns2vrex-157 storage/xtradb/include/os0file.h os0file.h-20081201061010-zymrrwrczns2vrex-161 storage/xtradb/include/os0sync.h os0sync.h-20081201061010-zymrrwrczns2vrex-164 storage/xtradb/include/page0page.h page0page.h-20081201061010-zymrrwrczns2vrex-170 storage/xtradb/include/page0page.ic page0page.ic-20081201061010-zymrrwrczns2vrex-171 storage/xtradb/include/page0zip.h page0zip.h-20081201061010-zymrrwrczns2vrex-173 storage/xtradb/include/pars0pars.h pars0pars.h-20081201061010-zymrrwrczns2vrex-178 storage/xtradb/include/rem0cmp.h rem0cmp.h-20081201061010-zymrrwrczns2vrex-189 storage/xtradb/include/rem0rec.ic rem0rec.ic-20081201061010-zymrrwrczns2vrex-192 storage/xtradb/include/row0ins.h row0ins.h-20081201061010-zymrrwrczns2vrex-196 storage/xtradb/include/row0mysql.h row0mysql.h-20081201061010-zymrrwrczns2vrex-199 storage/xtradb/include/srv0srv.h srv0srv.h-20081201061010-zymrrwrczns2vrex-219 storage/xtradb/include/sync0rw.h sync0rw.h-20081201061010-zymrrwrczns2vrex-224 storage/xtradb/include/sync0sync.h sync0sync.h-20081201061010-zymrrwrczns2vrex-226 storage/xtradb/include/thr0loc.h thr0loc.h-20081201061010-zymrrwrczns2vrex-229 storage/xtradb/include/trx0i_s.h trx0i_s.h-20081201061010-zymrrwrczns2vrex-231 storage/xtradb/include/trx0purge.h trx0purge.h-20081201061010-zymrrwrczns2vrex-232 storage/xtradb/include/trx0rec.h trx0rec.h-20081201061010-zymrrwrczns2vrex-234 storage/xtradb/include/trx0rec.ic trx0rec.ic-20081201061010-zymrrwrczns2vrex-235 storage/xtradb/include/trx0roll.h trx0roll.h-20081201061010-zymrrwrczns2vrex-236 storage/xtradb/include/trx0rseg.h trx0rseg.h-20081201061010-zymrrwrczns2vrex-238 storage/xtradb/include/trx0sys.h trx0sys.h-20081201061010-zymrrwrczns2vrex-240 storage/xtradb/include/trx0sys.ic trx0sys.ic-20081201061010-zymrrwrczns2vrex-241 storage/xtradb/include/trx0trx.h trx0trx.h-20081201061010-zymrrwrczns2vrex-242 storage/xtradb/include/trx0types.h trx0types.h-20081201061010-zymrrwrczns2vrex-244 storage/xtradb/include/trx0undo.h trx0undo.h-20081201061010-zymrrwrczns2vrex-245 storage/xtradb/include/univ.i univ.i-20081201061010-zymrrwrczns2vrex-248 storage/xtradb/include/usr0sess.h usr0sess.h-20081201061010-zymrrwrczns2vrex-249 storage/xtradb/include/ut0auxconf.h ut0auxconf.h-20090326061054-ylrdb8libxw6u7e9-2 storage/xtradb/include/ut0byte.h ut0byte.h-20081201061010-zymrrwrczns2vrex-252 storage/xtradb/include/ut0byte.ic ut0byte.ic-20081201061010-zymrrwrczns2vrex-253 storage/xtradb/include/ut0ut.h ut0ut.h-20081201061010-zymrrwrczns2vrex-263 storage/xtradb/lock/lock0lock.c lock0lock.c-20081201061010-zymrrwrczns2vrex-269 storage/xtradb/log/log0log.c log0log.c-20081201061010-zymrrwrczns2vrex-270 storage/xtradb/log/log0recv.c log0recv.c-20081201061010-zymrrwrczns2vrex-271 storage/xtradb/mem/mem0dbg.c mem0dbg.c-20081201061010-zymrrwrczns2vrex-273 storage/xtradb/mem/mem0mem.c mem0mem.c-20081201061010-zymrrwrczns2vrex-274 storage/xtradb/mem/mem0pool.c mem0pool.c-20081201061010-zymrrwrczns2vrex-275 storage/xtradb/mtr/mtr0mtr.c mtr0mtr.c-20081201061010-zymrrwrczns2vrex-277 storage/xtradb/os/os0file.c os0file.c-20081201061010-zymrrwrczns2vrex-313 storage/xtradb/os/os0proc.c os0proc.c-20081201061010-zymrrwrczns2vrex-314 storage/xtradb/os/os0sync.c os0sync.c-20081201061010-zymrrwrczns2vrex-315 storage/xtradb/os/os0thread.c os0thread.c-20081201061010-zymrrwrczns2vrex-316 storage/xtradb/page/page0cur.c page0cur.c-20081201061010-zymrrwrczns2vrex-317 storage/xtradb/page/page0page.c page0page.c-20081201061010-zymrrwrczns2vrex-318 storage/xtradb/page/page0zip.c page0zip.c-20081201061010-zymrrwrczns2vrex-319 storage/xtradb/pars/lexyy.c lexyy.c-20081201061010-zymrrwrczns2vrex-320 storage/xtradb/pars/pars0lex.l pars0lex.l-20081201061010-zymrrwrczns2vrex-325 storage/xtradb/plug.in plug.in-20081201061010-zymrrwrczns2vrex-31 storage/xtradb/que/que0que.c que0que.c-20081201061010-zymrrwrczns2vrex-329 storage/xtradb/rem/rem0cmp.c rem0cmp.c-20081201061010-zymrrwrczns2vrex-331 storage/xtradb/row/row0ins.c row0ins.c-20081201061010-zymrrwrczns2vrex-334 storage/xtradb/row/row0merge.c row0merge.c-20081201061010-zymrrwrczns2vrex-335 storage/xtradb/row/row0mysql.c row0mysql.c-20081201061010-zymrrwrczns2vrex-336 storage/xtradb/row/row0sel.c row0sel.c-20081201061010-zymrrwrczns2vrex-339 storage/xtradb/scripts/install_innodb_plugins.sql install_innodb_plugi-20081201061010-zymrrwrczns2vrex-345 storage/xtradb/scripts/install_innodb_plugins_win.sql install_innodb_plugi-20081203050234-edoolglm28lyejuc-14 storage/xtradb/srv/srv0srv.c srv0srv.c-20081201061010-zymrrwrczns2vrex-347 storage/xtradb/srv/srv0start.c srv0start.c-20081201061010-zymrrwrczns2vrex-348 storage/xtradb/sync/sync0arr.c sync0arr.c-20081201061010-zymrrwrczns2vrex-349 storage/xtradb/sync/sync0rw.c sync0rw.c-20081201061010-zymrrwrczns2vrex-350 storage/xtradb/sync/sync0sync.c sync0sync.c-20081201061010-zymrrwrczns2vrex-351 storage/xtradb/thr/thr0loc.c thr0loc.c-20081201061010-zymrrwrczns2vrex-352 storage/xtradb/trx/trx0i_s.c trx0i_s.c-20081201061010-zymrrwrczns2vrex-353 storage/xtradb/trx/trx0purge.c trx0purge.c-20081201061010-zymrrwrczns2vrex-354 storage/xtradb/trx/trx0rec.c trx0rec.c-20081201061010-zymrrwrczns2vrex-355 storage/xtradb/trx/trx0roll.c trx0roll.c-20081201061010-zymrrwrczns2vrex-356 storage/xtradb/trx/trx0rseg.c trx0rseg.c-20081201061010-zymrrwrczns2vrex-357 storage/xtradb/trx/trx0sys.c trx0sys.c-20081201061010-zymrrwrczns2vrex-358 storage/xtradb/trx/trx0trx.c trx0trx.c-20081201061010-zymrrwrczns2vrex-359 storage/xtradb/trx/trx0undo.c trx0undo.c-20081201061010-zymrrwrczns2vrex-360 storage/xtradb/usr/usr0sess.c usr0sess.c-20081201061010-zymrrwrczns2vrex-361 storage/xtradb/ut/ut0auxconf_atomic_pthread_t_solaris.c ut0auxconf_atomic_pt-20090923000535-ke95wdd4zn27df71-21 storage/xtradb/ut/ut0mem.c ut0mem.c-20081201061010-zymrrwrczns2vrex-365 storage/xtradb/ut/ut0ut.c ut0ut.c-20081201061010-zymrrwrczns2vrex-367 support-files/compiler_warnings.supp sp1f-disabled_compiler_wa-20070110170439-wzgdkamsch2nrkgvcp2hytmquqeorohi tests/mysql_client_test.c sp1f-client_test.c-20020614002636-eqy2zzksgelocknwbbogfuwxfwqy7q5x unittest/mysys/waiting_threads-t.c waiting_threadst.c-20080623170213-r8baqa2porlpxzq1-5 vio/vio.c sp1f-vio.c-20010520120430-aw76h22ssarmssof7rplhty5elqiexku vio/viosocket.c sp1f-viotcpip.c-20010520120437-u3pbzbt3fdfbclbmusalnzmuqh2y4nav vio/viossl.c sp1f-viossl.c-20010520120431-amywaj3niiokylabjhaly7w33kgdifl6 Diff too large for email (36932 lines, the limit is 1000).

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2812: Automerge MariaDB 5.1.42 release branch into main.
by noreply＠launchpad.net 31 Jan '10

31 Jan '10

Merge authors: Kristian Nielsen (knielsen) ------------------------------------------------------------ revno: 2812 [merge] committer: knielsen(a)knielsen-hq.org branch nick: work-5.1 timestamp: Sun 2010-01-31 16:36:03 +0100 message: Automerge MariaDB 5.1.42 release branch into main. modified: libmysqld/CMakeLists.txt -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2812)
by knielsen＠knielsen-hq.org 31 Jan '10

31 Jan '10

#At lp:maria 2812 knielsen(a)knielsen-hq.org 2010-01-31 [merge] Automerge MariaDB 5.1.42 release branch into main. modified: libmysqld/CMakeLists.txt === modified file 'libmysqld/CMakeLists.txt' --- a/libmysqld/CMakeLists.txt 2009-12-03 11:19:05 +0000 +++ b/libmysqld/CMakeLists.txt 2010-01-31 09:13:21 +0000 @@ -153,7 +153,7 @@ FILE(WRITE cmake_dummy.c " ") # custom targets to "sql/CMakeLists.txt" and reference them here. ADD_LIBRARY(mysqlserver STATIC ${LIBMYSQLD_SOURCES}) ADD_DEPENDENCIES(mysqlserver GenServerSource GenError) -TARGET_LINK_LIBRARIES(mysqlserver) +TARGET_LINK_LIBRARIES(mysqlserver psapi.lib) # Add any additional libraries requested by engine(s) FOREACH (ENGINE_LIB ${MYSQLD_STATIC_ENGINE_LIBS})

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2792)
by knielsen＠knielsen-hq.org 31 Jan '10

31 Jan '10

#At lp:maria 2792 knielsen(a)knielsen-hq.org 2010-01-31 Fix windows build of embedded server (forgotten dependency). modified: libmysqld/CMakeLists.txt === modified file 'libmysqld/CMakeLists.txt' --- a/libmysqld/CMakeLists.txt 2009-12-03 11:19:05 +0000 +++ b/libmysqld/CMakeLists.txt 2010-01-31 09:13:21 +0000 @@ -153,7 +153,7 @@ FILE(WRITE cmake_dummy.c " ") # custom targets to "sql/CMakeLists.txt" and reference them here. ADD_LIBRARY(mysqlserver STATIC ${LIBMYSQLD_SOURCES}) ADD_DEPENDENCIES(mysqlserver GenServerSource GenError) -TARGET_LINK_LIBRARIES(mysqlserver) +TARGET_LINK_LIBRARIES(mysqlserver psapi.lib) # Add any additional libraries requested by engine(s) FOREACH (ENGINE_LIB ${MYSQLD_STATIC_ENGINE_LIBS})

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2811: now we force at least libevent-1.4
by noreply＠launchpad.net 30 Jan '10

30 Jan '10

------------------------------------------------------------ revno: 2811 committer: Sergei Golubchik <sergii(a)pisem.net> branch nick: maria-5.1 timestamp: Sat 2010-01-30 13:39:39 +0100 message: now we force at least libevent-1.4 modified: config/ac-macros/libevent.m4 -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2809: Support building with system libevent
by noreply＠launchpad.net 30 Jan '10

30 Jan '10

------------------------------------------------------------ revno: 2809 committer: Sergei Golubchik <sergii(a)pisem.net> branch nick: maria-5.1 timestamp: Fri 2010-01-29 21:35:56 +0100 message: Support building with system libevent modified: config/ac-macros/libevent.m4 -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

2 2

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2791)
by knielsen＠knielsen-hq.org 30 Jan '10

30 Jan '10

#At lp:maria 2791 knielsen(a)knielsen-hq.org 2010-01-30 [merge] Automerge from main into MariaDB 5.1.42 release branch. removed: mysql-test/suite/maria/t/maria2-master.opt storage/xtradb/handler/handler0vars.h storage/xtradb/handler/win_delay_loader.cc storage/xtradb/win-plugin/ storage/xtradb/win-plugin/README storage/xtradb/win-plugin/win-plugin.diff added: mysql-test/extra/rpl_tests/rpl_not_null.test mysql-test/r/bug47671.result mysql-test/r/innodb-consistent.result mysql-test/r/innodb_bug44571.result mysql-test/r/innodb_bug46676.result mysql-test/r/innodb_bug47167.result mysql-test/r/udf_query_cache.result mysql-test/std_data/bug47012.ARM mysql-test/std_data/bug47012.ARZ mysql-test/std_data/bug47012.frm mysql-test/suite/innodb/r/innodb_bug46676.result mysql-test/suite/innodb/r/innodb_bug47167.result mysql-test/suite/innodb/t/innodb_bug46676.test mysql-test/suite/innodb/t/innodb_bug47167.test mysql-test/suite/rpl/r/rpl_loaddata_symlink.result mysql-test/suite/rpl/r/rpl_nondeterministic_functions.result mysql-test/suite/rpl/r/rpl_not_null_innodb.result mysql-test/suite/rpl/r/rpl_not_null_myisam.result mysql-test/suite/rpl/r/rpl_row_trunc_temp.result mysql-test/suite/rpl/t/rpl_loaddata_symlink-master.opt mysql-test/suite/rpl/t/rpl_loaddata_symlink-master.sh mysql-test/suite/rpl/t/rpl_loaddata_symlink-slave.opt mysql-test/suite/rpl/t/rpl_loaddata_symlink-slave.sh mysql-test/suite/rpl/t/rpl_loaddata_symlink.test mysql-test/suite/rpl/t/rpl_nondeterministic_functions.test mysql-test/suite/rpl/t/rpl_not_null_innodb.test mysql-test/suite/rpl/t/rpl_not_null_myisam.test mysql-test/suite/rpl/t/rpl_row_trunc_temp.test mysql-test/t/bug47671-master.opt mysql-test/t/bug47671.test mysql-test/t/innodb-consistent-master.opt mysql-test/t/innodb-consistent.test mysql-test/t/innodb_bug44571.test mysql-test/t/innodb_bug46676.test mysql-test/t/innodb_bug47167.test mysql-test/t/mysqlbinlog2-master.opt mysql-test/t/mysqlbinlog_row-master.opt mysql-test/t/mysqlbinlog_row_innodb-master.opt mysql-test/t/mysqlbinlog_row_myisam-master.opt mysql-test/t/mysqlbinlog_row_trans-master.opt mysql-test/t/udf_query_cache-master.opt mysql-test/t/udf_query_cache.test storage/xtradb/ut/ut0auxconf_have_gcc_atomics.c modified: .bzrignore BUILD/util.sh Makefile.am client/mysql.cc client/mysqlbinlog.cc client/mysqlslap.c client/mysqltest.cc configure.in extra/comp_err.c extra/libevent/devpoll.c extra/yassl/src/yassl_error.cpp extra/yassl/taocrypt/include/asn.hpp extra/yassl/taocrypt/src/asn.cpp include/ft_global.h include/maria.h include/my_pthread.h include/mysql.h include/mysql.h.pp include/violite.h libmysql/libmysql.c libmysql/libmysql.def libmysqld/libmysqld.def mysql-test/collections/default.experimental mysql-test/extra/rpl_tests/rpl_extraSlave_Col.test mysql-test/extra/rpl_tests/rpl_row_tabledefs.test mysql-test/extra/rpl_tests/rpl_stm_000001.test mysql-test/include/mtr_warnings.sql mysql-test/include/ps_conv.inc mysql-test/lib/mtr_cases.pm mysql-test/mysql-test-run.pl mysql-test/r/archive.result mysql-test/r/delayed.result mysql-test/r/delete.result mysql-test/r/fulltext.result mysql-test/r/func_group.result mysql-test/r/func_misc.result mysql-test/r/grant.result mysql-test/r/grant2.result mysql-test/r/group_min_max.result mysql-test/r/information_schema.result mysql-test/r/information_schema_all_engines.result mysql-test/r/innodb-autoinc.result mysql-test/r/innodb-index.result mysql-test/r/innodb-timeout.result mysql-test/r/innodb-zip.result mysql-test/r/innodb.result mysql-test/r/innodb_bug36169.result mysql-test/r/innodb_bug44369.result mysql-test/r/innodb_file_format.result mysql-test/r/innodb_lock_wait_timeout_1.result mysql-test/r/innodb_mysql.result mysql-test/r/innodb_xtradb_bug317074.result mysql-test/r/mysql.result mysql-test/r/mysqltest.result mysql-test/r/olap.result mysql-test/r/openssl_1.result mysql-test/r/order_by.result mysql-test/r/partition.result mysql-test/r/query_cache.result mysql-test/r/query_cache_notembedded.result mysql-test/r/range.result mysql-test/r/select.result mysql-test/r/show_check.result mysql-test/r/sp-destruct.result mysql-test/r/sp-security.result mysql-test/r/sp.result mysql-test/r/sp_notembedded.result mysql-test/r/trigger.result mysql-test/r/trigger_notembedded.result mysql-test/r/type_newdecimal.result mysql-test/r/type_year.result mysql-test/r/udf.result mysql-test/r/variables.result mysql-test/std_data/cacert.pem mysql-test/std_data/client-cert.pem mysql-test/std_data/client-key.pem mysql-test/std_data/server-cert.pem mysql-test/std_data/server-key.pem mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result mysql-test/suite/binlog/r/binlog_stm_row.result mysql-test/suite/binlog/r/binlog_unsafe.result mysql-test/suite/binlog/t/binlog_killed.test mysql-test/suite/binlog/t/binlog_stm_mix_innodb_myisam.test mysql-test/suite/binlog/t/binlog_stm_row.test mysql-test/suite/binlog/t/binlog_unsafe.test mysql-test/suite/funcs_1/datadict/processlist_val.inc mysql-test/suite/funcs_1/r/is_columns_is.result mysql-test/suite/funcs_1/r/is_tables_is.result mysql-test/suite/innodb/r/innodb-index.result mysql-test/suite/innodb/t/innodb-consistent-master.opt mysql-test/suite/innodb/t/innodb-index.test mysql-test/suite/maria/r/maria-recover.result mysql-test/suite/maria/t/maria-recover.test mysql-test/suite/parts/inc/partition_timestamp.inc mysql-test/suite/parts/r/partition_recover_myisam.result mysql-test/suite/parts/t/partition_alter1_2_innodb.test mysql-test/suite/parts/t/partition_alter2_1_innodb.test mysql-test/suite/parts/t/partition_alter2_2_innodb.test mysql-test/suite/parts/t/partition_alter4_innodb.test mysql-test/suite/parts/t/partition_recover_myisam.test mysql-test/suite/pbxt/r/func_group.result mysql-test/suite/pbxt/r/mysqlshow.result mysql-test/suite/rpl/r/rpl_err_ignoredtable.result mysql-test/suite/rpl/r/rpl_extraCol_innodb.result mysql-test/suite/rpl/r/rpl_extraCol_myisam.result mysql-test/suite/rpl/r/rpl_get_lock.result mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result mysql-test/suite/rpl/r/rpl_relayspace.result mysql-test/suite/rpl/r/rpl_row_create_table.result mysql-test/suite/rpl/r/rpl_row_tabledefs_2myisam.result mysql-test/suite/rpl/r/rpl_row_tabledefs_3innodb.result mysql-test/suite/rpl/r/rpl_stm_000001.result mysql-test/suite/rpl/r/rpl_temporary.result mysql-test/suite/rpl/r/rpl_trigger.result mysql-test/suite/rpl/t/disabled.def mysql-test/suite/rpl/t/rpl_err_ignoredtable.test mysql-test/suite/rpl/t/rpl_get_lock.test mysql-test/suite/rpl/t/rpl_mixed_ddl_dml.test mysql-test/suite/rpl/t/rpl_relayspace.test mysql-test/suite/rpl/t/rpl_rotate_logs.test mysql-test/suite/rpl/t/rpl_row_create_table.test mysql-test/suite/rpl/t/rpl_temporary.test mysql-test/suite/rpl/t/rpl_trigger.test mysql-test/suite/rpl_ndb/r/rpl_ndb_extraCol.result mysql-test/t/archive.test mysql-test/t/delayed.test mysql-test/t/delete.test mysql-test/t/disabled.def mysql-test/t/fulltext.test mysql-test/t/func_group.test mysql-test/t/func_misc.test mysql-test/t/grant.test mysql-test/t/grant2.test mysql-test/t/group_min_max.test mysql-test/t/innodb-analyze.test mysql-test/t/innodb-autoinc.test mysql-test/t/innodb-index.test mysql-test/t/innodb-master.opt mysql-test/t/innodb-semi-consistent-master.opt mysql-test/t/innodb-timeout.test mysql-test/t/innodb-use-sys-malloc-master.opt mysql-test/t/innodb-zip.test mysql-test/t/innodb.test mysql-test/t/innodb_bug34300.test mysql-test/t/innodb_bug36169.test mysql-test/t/innodb_bug36172.test mysql-test/t/innodb_bug42101-nonzero-master.opt mysql-test/t/innodb_bug44369.test mysql-test/t/innodb_file_format.test mysql-test/t/innodb_information_schema.test mysql-test/t/innodb_lock_wait_timeout_1.test mysql-test/t/innodb_mysql.test mysql-test/t/innodb_xtradb_bug317074.test mysql-test/t/mysql.test mysql-test/t/mysqltest.test mysql-test/t/olap.test mysql-test/t/openssl_1.test mysql-test/t/order_by.test mysql-test/t/partition.test mysql-test/t/partition_innodb_semi_consistent.test mysql-test/t/query_cache.test mysql-test/t/query_cache_debug.test mysql-test/t/query_cache_notembedded.test mysql-test/t/range.test mysql-test/t/select.test mysql-test/t/show_check-master.opt mysql-test/t/show_check.test mysql-test/t/sp-destruct.test mysql-test/t/sp-security.test mysql-test/t/sp.test mysql-test/t/sp_notembedded.test mysql-test/t/trigger.test mysql-test/t/trigger_notembedded.test mysql-test/t/type_newdecimal.test mysql-test/t/type_year.test mysql-test/t/udf.test mysql-test/t/variables.test mysql-test/valgrind.supp mysys/my_file.c mysys/my_getopt.c mysys/my_sync.c mysys/my_thr_init.c scripts/make_win_bin_dist scripts/mysql_secure_installation.pl.in scripts/mysql_secure_installation.sh sql-common/client.c sql/CMakeLists.txt sql/event_db_repository.cc sql/field.cc sql/field.h sql/handler.h sql/item.cc sql/item.h sql/item_cmpfunc.cc sql/item_cmpfunc.h sql/item_create.cc sql/item_func.cc sql/item_func.h sql/item_geofunc.cc sql/item_strfunc.cc sql/item_subselect.cc sql/item_subselect.h sql/item_sum.cc sql/item_sum.h sql/item_timefunc.cc sql/item_xmlfunc.cc sql/log.cc sql/log_event.cc sql/log_event.h sql/mysqld.cc sql/net_serv.cc sql/opt_range.cc sql/repl_failsafe.cc sql/rpl_record.cc sql/rpl_record.h sql/rpl_rli.cc sql/rpl_tblmap.cc sql/set_var.cc sql/slave.cc sql/sp.cc sql/sp.h sql/sp_cache.cc sql/sp_head.cc sql/sp_head.h sql/sp_rcontext.cc sql/sql_acl.cc sql/sql_acl.h sql/sql_base.cc sql/sql_cache.cc sql/sql_cache.h sql/sql_class.cc sql/sql_connect.cc sql/sql_delete.cc sql/sql_insert.cc sql/sql_load.cc sql/sql_parse.cc sql/sql_partition.cc sql/sql_profile.cc sql/sql_profile.h sql/sql_select.cc sql/sql_show.cc sql/sql_table.cc sql/sql_yacc.yy sql/table.cc sql/table.h sql/udf_example.def storage/archive/CMakeLists.txt* storage/archive/azio.c storage/archive/ha_archive.cc storage/federated/CMakeLists.txt* storage/innobase/btr/btr0btr.c storage/innobase/data/data0type.c storage/innobase/handler/ha_innodb.cc storage/innobase/include/ha_prototypes.h storage/innobase/include/mach0data.h storage/innobase/include/mach0data.ic storage/innobase/include/os0file.h storage/innobase/include/trx0trx.h storage/innobase/lock/lock0lock.c storage/innobase/os/os0file.c storage/innobase/row/row0sel.c storage/innobase/trx/trx0trx.c storage/innobase/ut/ut0ut.c storage/innodb_plugin/CMakeLists.txt storage/innodb_plugin/ChangeLog storage/innodb_plugin/btr/btr0btr.c storage/innodb_plugin/btr/btr0sea.c storage/innodb_plugin/buf/buf0buf.c storage/innodb_plugin/data/data0type.c storage/innodb_plugin/dict/dict0dict.c storage/innodb_plugin/fil/fil0fil.c storage/innodb_plugin/handler/ha_innodb.cc storage/innodb_plugin/handler/ha_innodb.h storage/innodb_plugin/handler/handler0alter.cc storage/innodb_plugin/ibuf/ibuf0ibuf.c storage/innodb_plugin/include/btr0sea.h storage/innodb_plugin/include/db0err.h storage/innodb_plugin/include/dict0dict.h storage/innodb_plugin/include/fil0fil.h storage/innodb_plugin/include/ha_prototypes.h storage/innodb_plugin/include/ibuf0ibuf.h storage/innodb_plugin/include/lock0lock.h storage/innodb_plugin/include/log0log.h storage/innodb_plugin/include/log0recv.h storage/innodb_plugin/include/mem0mem.h storage/innodb_plugin/include/mem0pool.h storage/innodb_plugin/include/os0file.h storage/innodb_plugin/include/pars0pars.h storage/innodb_plugin/include/srv0srv.h storage/innodb_plugin/include/thr0loc.h storage/innodb_plugin/include/trx0i_s.h storage/innodb_plugin/include/trx0purge.h storage/innodb_plugin/include/trx0rseg.h storage/innodb_plugin/include/trx0sys.h storage/innodb_plugin/include/trx0trx.h storage/innodb_plugin/include/trx0undo.h storage/innodb_plugin/include/univ.i storage/innodb_plugin/include/usr0sess.h storage/innodb_plugin/lock/lock0lock.c storage/innodb_plugin/log/log0log.c storage/innodb_plugin/log/log0recv.c storage/innodb_plugin/mem/mem0dbg.c storage/innodb_plugin/mem/mem0pool.c storage/innodb_plugin/os/os0file.c storage/innodb_plugin/os/os0sync.c storage/innodb_plugin/os/os0thread.c storage/innodb_plugin/pars/lexyy.c storage/innodb_plugin/pars/pars0lex.l storage/innodb_plugin/que/que0que.c storage/innodb_plugin/row/row0merge.c storage/innodb_plugin/row/row0mysql.c storage/innodb_plugin/srv/srv0srv.c storage/innodb_plugin/srv/srv0start.c storage/innodb_plugin/sync/sync0arr.c storage/innodb_plugin/sync/sync0sync.c storage/innodb_plugin/thr/thr0loc.c storage/innodb_plugin/trx/trx0i_s.c storage/innodb_plugin/trx/trx0purge.c storage/innodb_plugin/trx/trx0rseg.c storage/innodb_plugin/trx/trx0sys.c storage/innodb_plugin/trx/trx0trx.c storage/innodb_plugin/trx/trx0undo.c storage/innodb_plugin/usr/usr0sess.c storage/innodb_plugin/ut/ut0mem.c storage/maria/ft_maria.c storage/maria/ma_blockrec.c storage/maria/ma_check.c storage/maria/ma_close.c storage/maria/ma_create.c storage/myisam/ft_boolean_search.c storage/myisam/ft_myisam.c storage/myisam/ft_nlq_search.c storage/myisam/ft_parser.c storage/myisam/ft_stopwords.c storage/myisam/ftdefs.h storage/myisam/mi_check.c storage/myisam/mi_create.c storage/xtradb/CMakeLists.txt storage/xtradb/ChangeLog storage/xtradb/Makefile.am storage/xtradb/btr/btr0btr.c storage/xtradb/btr/btr0sea.c storage/xtradb/buf/buf0buddy.c storage/xtradb/buf/buf0buf.c storage/xtradb/buf/buf0flu.c storage/xtradb/buf/buf0lru.c storage/xtradb/buf/buf0rea.c storage/xtradb/data/data0type.c storage/xtradb/dict/dict0crea.c storage/xtradb/dict/dict0dict.c storage/xtradb/fil/fil0fil.c storage/xtradb/fsp/fsp0fsp.c storage/xtradb/handler/ha_innodb.cc storage/xtradb/handler/ha_innodb.h storage/xtradb/handler/handler0alter.cc storage/xtradb/handler/i_s.cc storage/xtradb/handler/i_s.h storage/xtradb/handler/innodb_patch_info.h storage/xtradb/ibuf/ibuf0ibuf.c storage/xtradb/include/btr0cur.h storage/xtradb/include/btr0sea.h storage/xtradb/include/buf0buf.h storage/xtradb/include/buf0buf.ic storage/xtradb/include/buf0lru.h storage/xtradb/include/buf0rea.h storage/xtradb/include/buf0types.h storage/xtradb/include/db0err.h storage/xtradb/include/dict0crea.h storage/xtradb/include/dict0dict.h storage/xtradb/include/dict0mem.h storage/xtradb/include/fil0fil.h storage/xtradb/include/fsp0fsp.h storage/xtradb/include/ibuf0ibuf.h storage/xtradb/include/lock0lock.h storage/xtradb/include/log0log.h storage/xtradb/include/log0log.ic storage/xtradb/include/log0recv.h storage/xtradb/include/mem0mem.h storage/xtradb/include/mem0pool.h storage/xtradb/include/mtr0mtr.h storage/xtradb/include/os0file.h storage/xtradb/include/os0sync.h storage/xtradb/include/page0page.h storage/xtradb/include/page0page.ic storage/xtradb/include/page0zip.h storage/xtradb/include/pars0pars.h storage/xtradb/include/rem0cmp.h storage/xtradb/include/rem0rec.ic storage/xtradb/include/row0ins.h storage/xtradb/include/row0mysql.h storage/xtradb/include/srv0srv.h storage/xtradb/include/sync0rw.h storage/xtradb/include/sync0sync.h storage/xtradb/include/thr0loc.h storage/xtradb/include/trx0i_s.h storage/xtradb/include/trx0purge.h storage/xtradb/include/trx0rec.h storage/xtradb/include/trx0rec.ic storage/xtradb/include/trx0roll.h storage/xtradb/include/trx0rseg.h storage/xtradb/include/trx0sys.h storage/xtradb/include/trx0sys.ic storage/xtradb/include/trx0trx.h storage/xtradb/include/trx0types.h storage/xtradb/include/trx0undo.h storage/xtradb/include/univ.i storage/xtradb/include/usr0sess.h storage/xtradb/include/ut0auxconf.h storage/xtradb/include/ut0byte.h storage/xtradb/include/ut0byte.ic storage/xtradb/include/ut0ut.h storage/xtradb/lock/lock0lock.c storage/xtradb/log/log0log.c storage/xtradb/log/log0recv.c storage/xtradb/mem/mem0dbg.c storage/xtradb/mem/mem0mem.c storage/xtradb/mem/mem0pool.c storage/xtradb/mtr/mtr0mtr.c storage/xtradb/os/os0file.c storage/xtradb/os/os0proc.c storage/xtradb/os/os0sync.c storage/xtradb/os/os0thread.c storage/xtradb/page/page0cur.c storage/xtradb/page/page0page.c storage/xtradb/page/page0zip.c storage/xtradb/pars/lexyy.c storage/xtradb/pars/pars0lex.l storage/xtradb/plug.in storage/xtradb/que/que0que.c storage/xtradb/rem/rem0cmp.c storage/xtradb/row/row0ins.c storage/xtradb/row/row0merge.c storage/xtradb/row/row0mysql.c storage/xtradb/row/row0sel.c storage/xtradb/scripts/install_innodb_plugins.sql storage/xtradb/scripts/install_innodb_plugins_win.sql storage/xtradb/srv/srv0srv.c storage/xtradb/srv/srv0start.c storage/xtradb/sync/sync0arr.c storage/xtradb/sync/sync0rw.c storage/xtradb/sync/sync0sync.c storage/xtradb/thr/thr0loc.c storage/xtradb/trx/trx0i_s.c storage/xtradb/trx/trx0purge.c storage/xtradb/trx/trx0rec.c storage/xtradb/trx/trx0roll.c storage/xtradb/trx/trx0rseg.c storage/xtradb/trx/trx0sys.c storage/xtradb/trx/trx0trx.c storage/xtradb/trx/trx0undo.c storage/xtradb/usr/usr0sess.c storage/xtradb/ut/ut0auxconf_atomic_pthread_t_solaris.c storage/xtradb/ut/ut0mem.c storage/xtradb/ut/ut0ut.c support-files/compiler_warnings.supp tests/mysql_client_test.c unittest/mysys/waiting_threads-t.c vio/vio.c vio/viosocket.c vio/viossl.c === modified file '.bzrignore' --- a/.bzrignore 2009-12-22 13:50:20 +0000 +++ b/.bzrignore 2010-01-29 18:42:22 +0000 @@ -40,15 +40,15 @@ *.dsp *.Po *.Plo -*/*.dir/* +*.dir/ */*_pure_*warnings */.deps */.libs/* */.pure -*/debug/* -*/minsizerel/* -*/release/* -*/relwithdebinfo/* +debug/ +MinSizeRel/ +Release/ +RelWithDebInfo/ *~ .*.swp ./CMakeCache.txt @@ -96,7 +96,7 @@ BitKeeper/tmp/gone BitKeeper/tmp BitKeeper/log BitKeeper/etc/SCCS -CMakeFiles/* +CMakeFiles/ COPYING COPYING.LIB Docs/#manual.texi# === modified file 'BUILD/util.sh' --- a/BUILD/util.sh 2010-01-07 12:02:18 +0000 +++ b/BUILD/util.sh 2010-01-28 22:34:22 +0000 @@ -16,6 +16,8 @@ # Setting cpu options. get_cpuopt () { + case "$(uname -o)" in + *Linux*) case "$(gcc -dumpmachine)" in x86_64-*) # gcc barfs on -march=... on x64 @@ -26,7 +28,13 @@ get_cpuopt () { CPUOPT="-m32 -march=i586 -mtune=generic" ;; esac - return 0 + ;; + *Solaris*) + # ToDo: handle 32-bit build? For now default to 64-bit. + CPUOPT="-D__sun -m64 -mtune=athlon64" + ;; + esac + return 0 } # Default to a parallel build, but only if AM_MAKEFLAGS is not set. === modified file 'Makefile.am' --- a/Makefile.am 2009-12-03 11:19:05 +0000 +++ b/Makefile.am 2010-01-15 15:27:55 +0000 @@ -208,10 +208,6 @@ test-bt-fast: -cd mysql-test ; MTR_BUILD_THREAD=auto \ @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --comment=stress --suite=stress -test-bt-fast2: - -cd mysql-test ; MTR_BUILD_THREAD=auto \ - @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --force --comment=ps --ps-protocol --report-features - test-bt-debug: -cd mysql-test ; MTR_BUILD_THREAD=auto \ @PERL@ ./mysql-test-run.pl $(MTR_EXTRA_OPTIONS) --comment=debug --force --timer \ === modified file 'client/mysql.cc' --- a/client/mysql.cc 2009-12-03 11:34:11 +0000 +++ b/client/mysql.cc 2010-01-15 15:27:55 +0000 @@ -4356,7 +4356,7 @@ com_status(String *buffer __attribute__( Don't remove "limit 1", it is protection againts SQL_SELECT_LIMIT=0 */ - if (mysql_store_result_for_lazy(&result)) + if (!mysql_store_result_for_lazy(&result)) { MYSQL_ROW cur=mysql_fetch_row(result); if (cur) @@ -4401,7 +4401,7 @@ com_status(String *buffer __attribute__( if (mysql_errno(&mysql) == CR_SERVER_GONE_ERROR) return 0; } - if (mysql_store_result_for_lazy(&result)) + if (!mysql_store_result_for_lazy(&result)) { MYSQL_ROW cur=mysql_fetch_row(result); if (cur) @@ -4496,9 +4496,7 @@ server_version_string(MYSQL *con) */ if (server_version == NULL) - { - server_version= strdup(mysql_get_server_info(con)); - } + server_version= my_strdup(mysql_get_server_info(con), MYF(MY_WME)); } return server_version ? server_version : ""; === modified file 'client/mysqlbinlog.cc' --- a/client/mysqlbinlog.cc 2009-12-03 11:19:05 +0000 +++ b/client/mysqlbinlog.cc 2010-01-09 09:04:51 +0000 @@ -1378,6 +1378,10 @@ static int parse_args(int *argc, char*** */ static Exit_status safe_connect() { + /* Close and old connections to MySQL */ + if (mysql) + mysql_close(mysql); + mysql= mysql_init(NULL); if (!mysql) === modified file 'client/mysqlslap.c' --- a/client/mysqlslap.c 2009-12-03 11:34:11 +0000 +++ b/client/mysqlslap.c 2010-01-29 18:42:22 +0000 @@ -292,6 +292,25 @@ static int gettimeofday(struct timeval * } #endif +void set_mysql_connect_options(MYSQL *mysql) +{ + if (opt_compress) + mysql_options(mysql,MYSQL_OPT_COMPRESS,NullS); +#ifdef HAVE_OPENSSL + if (opt_use_ssl) + mysql_ssl_set(mysql, opt_ssl_key, opt_ssl_cert, opt_ssl_ca, + opt_ssl_capath, opt_ssl_cipher); +#endif + if (opt_protocol) + mysql_options(mysql,MYSQL_OPT_PROTOCOL,(char*)&opt_protocol); +#ifdef HAVE_SMEM + if (shared_memory_base_name) + mysql_options(mysql,MYSQL_SHARED_MEMORY_BASE_NAME,shared_memory_base_name); +#endif + mysql_options(mysql, MYSQL_SET_CHARSET_NAME, default_charset); +} + + int main(int argc, char **argv) { MYSQL mysql; @@ -323,20 +342,7 @@ int main(int argc, char **argv) exit(1); } mysql_init(&mysql); - if (opt_compress) - mysql_options(&mysql,MYSQL_OPT_COMPRESS,NullS); -#ifdef HAVE_OPENSSL - if (opt_use_ssl) - mysql_ssl_set(&mysql, opt_ssl_key, opt_ssl_cert, opt_ssl_ca, - opt_ssl_capath, opt_ssl_cipher); -#endif - if (opt_protocol) - mysql_options(&mysql,MYSQL_OPT_PROTOCOL,(char*)&opt_protocol); -#ifdef HAVE_SMEM - if (shared_memory_base_name) - mysql_options(&mysql,MYSQL_SHARED_MEMORY_BASE_NAME,shared_memory_base_name); -#endif - mysql_options(&mysql, MYSQL_SET_CHARSET_NAME, default_charset); + set_mysql_connect_options(&mysql); if (!opt_only_print) { @@ -1815,6 +1821,7 @@ pthread_handler_t run_task(void *p) my_progname, mysql_error(mysql)); exit(0); } + set_mysql_connect_options(mysql); if (mysql_thread_init()) { @@ -1855,7 +1862,6 @@ limit_not_met: my_progname, mysql_error(mysql)); exit(0); } - if (slap_connect(mysql)) goto end; } @@ -2223,6 +2229,7 @@ slap_connect(MYSQL *mysql) int x, connect_error= 1; for (x= 0; x < 10; x++) { + set_mysql_connect_options(mysql); if (mysql_real_connect(mysql, host, user, opt_password, create_schema_string, opt_mysql_port, === modified file 'client/mysqltest.cc' --- a/client/mysqltest.cc 2009-12-27 13:54:41 +0000 +++ b/client/mysqltest.cc 2010-01-28 14:49:14 +0000 @@ -657,7 +657,9 @@ public: } while ((bytes= fread(buf, 1, sizeof(buf), m_file)) > 0) - fwrite(buf, 1, bytes, stderr); + if (fwrite(buf, 1, bytes, stderr)) + die("Failed to write to '%s', errno: %d", + m_file_name, errno); if (!lines) { === modified file 'configure.in' --- a/configure.in 2009-12-23 08:32:14 +0000 +++ b/configure.in 2010-01-29 10:42:31 +0000 @@ -9,15 +9,16 @@ AC_CANONICAL_SYSTEM # remember to also update version.c in ndb # # When changing major version number please also check switch statement -# in mysqlbinlog.cc / check_master_version(). -# -# When merging new MySQL releases, update the version number to match the -# MySQL version number. -# -# Note: the following line must be parseable by win/configure.js:GetVersion() -AM_INIT_AUTOMAKE(mysql, 5.1.41-MariaDB-rc) +# in mysqlbinlog::check_master_version(). +AM_INIT_AUTOMAKE(mysql, 5.1.42-MariaDB) AM_CONFIG_HEADER([include/config.h:config.h.in]) +# Request support for automake silent-rules if available. +# Default to verbose output. One can use the configure-time +# option --enable-silent-rules or make V=0 to activate +# silent rules. +m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([no])]) + PROTOCOL_VERSION=10 DOT_FRM_VERSION=6 # See the libtool docs for information on how to do shared lib versions. @@ -224,14 +225,6 @@ then GXX="no" fi -if test "$ac_cv_prog_gcc" = "yes" -then - AS="$CC -c" - AC_SUBST(AS) -else - AC_PATH_PROG(AS, as, as) -fi - # Still need ranlib for readline; local static use only so no libtool. AC_PROG_RANLIB # We use libtool @@ -688,7 +681,7 @@ AC_ARG_ENABLE(assembler, AC_MSG_CHECKING(if we should use assembler functions) # For now we only support assembler on i386 and sparc systems -AM_CONDITIONAL(ASSEMBLER_x86, test "$ENABLE_ASSEMBLER" = "yes" -a "$BASE_MACHINE_TYPE" = "i386" && $AS strings/strings-x86.s -o checkassembler >/dev/null 2>&1 && test -f checkassembler && (rm -f checkassembler; exit 0;)) +AM_CONDITIONAL(ASSEMBLER_x86, test "$ENABLE_ASSEMBLER" = "yes" -a "$BASE_MACHINE_TYPE" = "i386" && $CCAS $CCASFLAGS -c strings/strings-x86.s -o checkassembler >/dev/null 2>&1 && test -f checkassembler && (rm -f checkassembler; exit 0;)) AM_CONDITIONAL(ASSEMBLER_sparc32, test "$ENABLE_ASSEMBLER" = "yes" -a "$BASE_MACHINE_TYPE" = "sparc") AM_CONDITIONAL(ASSEMBLER_sparc64, test "$ENABLE_ASSEMBLER" = "yes" -a "$BASE_MACHINE_TYPE" = "sparcv9") AM_CONDITIONAL(ASSEMBLER, test "$ASSEMBLER_x86_TRUE" = "" -o "$ASSEMBLER_sparc32_TRUE" = "") === modified file 'extra/comp_err.c' --- a/extra/comp_err.c 2009-02-13 16:41:47 +0000 +++ b/extra/comp_err.c 2009-11-20 10:11:31 +0000 @@ -660,7 +660,7 @@ static ha_checksum checksum_format_speci case 'u': case 'x': case 's': - chksum= my_checksum(chksum, start, (uint) (p - start)); + chksum= my_checksum(chksum, start, (uint) (p + 1 - start)); start= 0; /* Not in format specifier anymore */ break; @@ -1030,8 +1030,10 @@ static char *parse_text_line(char *pos) { int i, nr; char *row= pos; + size_t len; DBUG_ENTER("parse_text_line"); + len= strlen (pos); while (*pos) { if (*pos == '\\') @@ -1039,11 +1041,11 @@ static char *parse_text_line(char *pos) switch (*++pos) { case '\\': case '"': - VOID(strmov(pos - 1, pos)); + VOID(memmove (pos - 1, pos, len - (row - pos))); break; case 'n': pos[-1]= '\n'; - VOID(strmov(pos, pos + 1)); + VOID(memmove (pos, pos + 1, len - (row - pos))); break; default: if (*pos >= '0' && *pos < '8') @@ -1053,10 +1055,10 @@ static char *parse_text_line(char *pos) nr= nr * 8 + (*(pos++) - '0'); pos -= i; pos[-1]= nr; - VOID(strmov(pos, pos + i)); + VOID(memmove (pos, pos + i, len - (row - pos))); } else if (*pos) - VOID(strmov(pos - 1, pos)); /* Remove '\' */ + VOID(memmove (pos - 1, pos, len - (row - pos))); /* Remove '\' */ } } else === modified file 'extra/libevent/devpoll.c' --- a/extra/libevent/devpoll.c 2010-01-07 13:00:06 +0000 +++ b/extra/libevent/devpoll.c 2010-01-28 11:35:10 +0000 @@ -140,7 +140,7 @@ devpoll_init(struct event_base *base) return (NULL); if (getrlimit(RLIMIT_NOFILE, &rl) == 0 && - rl.rlim_cur != RLIM_INFINITY) + (unsigned long long) rl.rlim_cur != (unsigned long long) RLIM_INFINITY) nfiles = rl.rlim_cur - 1; /* Initialize the kernel queue */ === modified file 'extra/yassl/src/yassl_error.cpp' --- a/extra/yassl/src/yassl_error.cpp 2008-11-18 16:45:44 +0000 +++ b/extra/yassl/src/yassl_error.cpp 2010-01-29 10:42:31 +0000 @@ -60,7 +60,7 @@ void SetErrorString(YasslError error, ch using namespace TaoCrypt; const int max = MAX_ERROR_SZ; // shorthand - switch (error) { + switch ((int) error) { // yaSSL proper errors case range_error : === modified file 'extra/yassl/taocrypt/include/asn.hpp' --- a/extra/yassl/taocrypt/include/asn.hpp 2007-01-29 15:54:40 +0000 +++ b/extra/yassl/taocrypt/include/asn.hpp 2010-01-27 10:38:29 +0000 @@ -305,6 +305,7 @@ private: bool ValidateSignature(SignerList*); bool ConfirmSignature(Source&); void GetKey(); + char* AddTag(char*, const char*, const char*, word32, word32); void GetName(NameType); void GetValidity(); void GetDate(DateType); === modified file 'extra/yassl/taocrypt/src/asn.cpp' --- a/extra/yassl/taocrypt/src/asn.cpp 2009-09-15 11:22:39 +0000 +++ b/extra/yassl/taocrypt/src/asn.cpp 2010-01-27 10:38:29 +0000 @@ -652,6 +652,25 @@ word32 CertDecoder::GetDigest() } +// memory length checked add tag to buffer +char* CertDecoder::AddTag(char* ptr, const char* buf_end, const char* tag_name, + word32 tag_name_length, word32 tag_value_length) +{ + if (ptr + tag_name_length + tag_value_length > buf_end) { + source_.SetError(CONTENT_E); + return 0; + } + + memcpy(ptr, tag_name, tag_name_length); + ptr += tag_name_length; + + memcpy(ptr, source_.get_current(), tag_value_length); + ptr += tag_value_length; + + return ptr; +} + + // process NAME, either issuer or subject void CertDecoder::GetName(NameType nt) { @@ -659,11 +678,22 @@ void CertDecoder::GetName(NameType nt) SHA sha; word32 length = GetSequence(); // length of all distinguished names - assert (length < ASN_NAME_MAX); + + if (length >= ASN_NAME_MAX) + return; length += source_.get_index(); - char* ptr = (nt == ISSUER) ? issuer_ : subject_; - word32 idx = 0; + char* ptr; + char* buf_end; + + if (nt == ISSUER) { + ptr = issuer_; + buf_end = ptr + sizeof(issuer_) - 1; // 1 byte for trailing 0 + } + else { + ptr = subject_; + buf_end = ptr + sizeof(subject_) - 1; // 1 byte for trailing 0 + } while (source_.get_index() < length) { GetSet(); @@ -685,47 +715,36 @@ void CertDecoder::GetName(NameType nt) byte id = source_.next(); b = source_.next(); // strType word32 strLen = GetLength(source_); - bool copy = false; - - if (id == COMMON_NAME) { - memcpy(&ptr[idx], "/CN=", 4); - idx += 4; - copy = true; - } - else if (id == SUR_NAME) { - memcpy(&ptr[idx], "/SN=", 4); - idx += 4; - copy = true; - } - else if (id == COUNTRY_NAME) { - memcpy(&ptr[idx], "/C=", 3); - idx += 3; - copy = true; - } - else if (id == LOCALITY_NAME) { - memcpy(&ptr[idx], "/L=", 3); - idx += 3; - copy = true; - } - else if (id == STATE_NAME) { - memcpy(&ptr[idx], "/ST=", 4); - idx += 4; - copy = true; - } - else if (id == ORG_NAME) { - memcpy(&ptr[idx], "/O=", 3); - idx += 3; - copy = true; - } - else if (id == ORGUNIT_NAME) { - memcpy(&ptr[idx], "/OU=", 4); - idx += 4; - copy = true; - } - if (copy) { - memcpy(&ptr[idx], source_.get_current(), strLen); - idx += strLen; + switch (id) { + case COMMON_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/CN=", 4, strLen))) + return; + break; + case SUR_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/SN=", 4, strLen))) + return; + break; + case COUNTRY_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/C=", 3, strLen))) + return; + break; + case LOCALITY_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/L=", 3, strLen))) + return; + break; + case STATE_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/ST=", 4, strLen))) + return; + break; + case ORG_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/O=", 3, strLen))) + return; + break; + case ORGUNIT_NAME: + if (!(ptr = AddTag(ptr, buf_end, "/OU=", 4, strLen))) + return; + break; } sha.Update(source_.get_current(), strLen); @@ -740,17 +759,15 @@ void CertDecoder::GetName(NameType nt) word32 length = GetLength(source_); if (email) { - memcpy(&ptr[idx], "/emailAddress=", 14); - idx += 14; - - memcpy(&ptr[idx], source_.get_current(), length); - idx += length; + if (!(ptr = AddTag(ptr, buf_end, "/emailAddress=", 14, length))) + return; } source_.advance(length); } } - ptr[idx++] = 0; + + *ptr = 0; if (nt == ISSUER) sha.Final(issuerHash_); === modified file 'include/ft_global.h' --- a/include/ft_global.h 2009-11-30 13:36:06 +0000 +++ b/include/ft_global.h 2010-01-27 21:53:08 +0000 @@ -62,7 +62,8 @@ void ft_free_stopwords(void); #define FT_SORTED 2 #define FT_EXPAND 4 /* query expansion */ -FT_INFO *ft_init_search(uint,void *, uint, uchar *, uint,CHARSET_INFO *, uchar *); +FT_INFO *ft_init_search(uint,void *, uint, uchar *, size_t, + CHARSET_INFO *, uchar *); my_bool ft_boolean_check_syntax_string(const uchar *); /* Internal symbols for fulltext between maria and MyISAM */ === modified file 'include/maria.h' --- a/include/maria.h 2009-02-19 09:01:25 +0000 +++ b/include/maria.h 2010-01-27 21:53:08 +0000 @@ -462,7 +462,7 @@ void maria_versioning(MARIA_HA *info, my void maria_ignore_trids(MARIA_HA *info); /* fulltext functions */ -FT_INFO *maria_ft_init_search(uint,void *, uint, uchar *, uint, +FT_INFO *maria_ft_init_search(uint,void *, uint, uchar *, size_t, CHARSET_INFO *, uchar *); /* 'Almost-internal' Maria functions */ === modified file 'include/my_pthread.h' --- a/include/my_pthread.h 2009-06-30 12:01:29 +0000 +++ b/include/my_pthread.h 2010-01-14 16:51:00 +0000 @@ -543,9 +543,9 @@ void safe_mutex_free_deadlock_data(safe_ #else #define my_pthread_mutex_init(A,B,C,D) pthread_mutex_init((A),(B)) #define my_pthread_mutex_lock(A,B) pthread_mutex_lock(A) -#define safe_mutex_assert_owner(mp) -#define safe_mutex_assert_not_owner(mp) -#define safe_mutex_free_deadlock_data(mp) +#define safe_mutex_assert_owner(mp) do {} while(0) +#define safe_mutex_assert_not_owner(mp) do {} while(0) +#define safe_mutex_free_deadlock_data(mp) do {} while(0) #endif /* SAFE_MUTEX */ #if defined(MY_PTHREAD_FASTMUTEX) && !defined(SAFE_MUTEX) === modified file 'include/mysql.h' --- a/include/mysql.h 2009-12-03 11:19:05 +0000 +++ b/include/mysql.h 2010-01-15 15:27:55 +0000 @@ -558,16 +558,6 @@ unsigned long STDCALL mysql_real_escape_ char *to,const char *from, unsigned long length); void STDCALL mysql_debug(const char *debug); -char * STDCALL mysql_odbc_escape_string(MYSQL *mysql, - char *to, - unsigned long to_length, - const char *from, - unsigned long from_length, - void *param, - char * - (*extend_buffer) - (void *, char *to, - unsigned long *length)); void STDCALL myodbc_remove_escape(MYSQL *mysql,char *name); unsigned int STDCALL mysql_thread_safe(void); my_bool STDCALL mysql_embedded(void); === modified file 'include/mysql.h.pp' --- a/include/mysql.h.pp 2009-12-03 11:19:05 +0000 +++ b/include/mysql.h.pp 2010-01-15 15:27:55 +0000 @@ -518,16 +518,6 @@ unsigned long mysql_real_escape_string(M char *to,const char *from, unsigned long length); void mysql_debug(const char *debug); -char * mysql_odbc_escape_string(MYSQL *mysql, - char *to, - unsigned long to_length, - const char *from, - unsigned long from_length, - void *param, - char * - (*extend_buffer) - (void *, char *to, - unsigned long *length)); void myodbc_remove_escape(MYSQL *mysql,char *name); unsigned int mysql_thread_safe(void); my_bool mysql_embedded(void); === modified file 'include/violite.h' --- a/include/violite.h 2009-12-03 11:19:05 +0000 +++ b/include/violite.h 2010-01-29 10:42:31 +0000 @@ -123,8 +123,8 @@ struct st_VioSSLFd SSL_CTX *ssl_context; }; -int sslaccept(struct st_VioSSLFd*, Vio *, long timeout); -int sslconnect(struct st_VioSSLFd*, Vio *, long timeout); +int sslaccept(struct st_VioSSLFd*, Vio *, long timeout, char *error_string); +int sslconnect(struct st_VioSSLFd*, Vio *, long timeout, char *error_string); struct st_VioSSLFd *new_VioSSLConnectorFd(const char *key_file, const char *cert_file, @@ -225,8 +225,8 @@ struct st_vio #endif /* HAVE_SMEM */ #ifdef _WIN32 OVERLAPPED pipe_overlapped; - DWORD read_timeout_millis; - DWORD write_timeout_millis; + DWORD read_timeout_ms; + DWORD write_timeout_ms; #endif }; #endif /* vio_violite_h_ */ === modified file 'libmysql/libmysql.c' --- a/libmysql/libmysql.c 2009-12-03 11:34:11 +0000 +++ b/libmysql/libmysql.c 2010-01-15 15:27:55 +0000 @@ -1642,20 +1642,6 @@ mysql_real_escape_string(MYSQL *mysql, c return (uint) escape_string_for_mysql(mysql->charset, to, 0, from, length); } - -char * STDCALL -mysql_odbc_escape_string(MYSQL *mysql __attribute__((unused)), - char *to __attribute__((unused)), - ulong to_length __attribute__((unused)), - const char *from __attribute__((unused)), - ulong from_length __attribute__((unused)), - void *param __attribute__((unused)), - char * (*extend_buffer)(void *, char *, ulong *) - __attribute__((unused))) -{ - return NULL; -} - void STDCALL myodbc_remove_escape(MYSQL *mysql,char *name) { === modified file 'libmysql/libmysql.def' --- a/libmysql/libmysql.def 2009-12-03 11:19:05 +0000 +++ b/libmysql/libmysql.def 2010-01-15 15:27:55 +0000 @@ -78,7 +78,6 @@ EXPORTS mysql_next_result mysql_num_fields mysql_num_rows - mysql_odbc_escape_string mysql_options mysql_stmt_param_count mysql_stmt_param_metadata === modified file 'libmysqld/libmysqld.def' --- a/libmysqld/libmysqld.def 2009-12-03 11:19:05 +0000 +++ b/libmysqld/libmysqld.def 2010-01-15 15:27:55 +0000 @@ -50,7 +50,6 @@ EXPORTS mysql_next_result mysql_num_fields mysql_num_rows - mysql_odbc_escape_string mysql_options mysql_ping mysql_query === modified file 'mysql-test/collections/default.experimental' --- a/mysql-test/collections/default.experimental 2009-10-26 12:33:03 +0000 +++ b/mysql-test/collections/default.experimental 2009-12-02 09:47:49 +0000 @@ -13,15 +13,13 @@ funcs_1.ndb* funcs_2.ndb_charset # joro : NDB tests marked as experimental as agreed with bochklin main.ctype_gbk_binlog @solaris # Bug#46010: main.ctype_gbk_binlog fails sporadically : Table 't2' already exists -main.innodb-autoinc* # Bug#47809 2009-10-04 joro innodb-autoinc.test fails with valgrind errors with the innodb plugin main.plugin_load @solaris # Bug#42144 ndb.* # joro : NDB tests marked as experimental as agreed with bochklin -rpl.rpl_cross_version* # Bug #43913 2009-10-26 joro rpl_cross_version can't pass on conflicts complainig clash with --slave-load-tm -rpl.rpl_get_master_version_and_clock* # Bug#46931 2009-08-26 alik rpl.rpl_get_master_version_and_clock fails on hpux11.31 +rpl.rpl_cross_version* # Bug#48340 2009-12-01 Daogang rpl_cross_version: Found warnings/errors in server log file! +rpl.rpl_get_master_version_and_clock* # Bug #49191 2009-12-01 Daogang rpl_get_master_version_and_clock failed on PB2: COM_REGISTER_SLAVE failed rpl.rpl_innodb_bug28430* @solaris # Bug#46029 -rpl.rpl_row_create_table* # Bug#45576: rpl_row_create_table fails on PB2 rpl.rpl_trigger* # Bug#47810 2009-10-04 joro rpl.rpl_trigger.test fails with valgrind errors with the innodb plugin rpl_ndb.* # joro : NDB tests marked as experimental as agreed with bochklin === modified file 'mysql-test/extra/rpl_tests/rpl_extraSlave_Col.test' --- a/mysql-test/extra/rpl_tests/rpl_extraSlave_Col.test 2009-08-28 14:13:27 +0000 +++ b/mysql-test/extra/rpl_tests/rpl_extraSlave_Col.test 2009-10-22 00:21:50 +0000 @@ -407,37 +407,57 @@ sync_slave_with_master; ########################################### # Bug#22234, Bug#23907 Extra Slave Col is not # erroring on extra col with no default values. -######################################################## +############################################################### +# Error reaction is up to sql_mode of the slave sql (bug#38173) #--echo *** Create t9 on slave *** -STOP SLAVE; -RESET SLAVE; -eval CREATE TABLE t9 (a INT KEY, b BLOB, c CHAR(5), - d TIMESTAMP, - e INT NOT NULL) ENGINE=$engine_type; - ---echo *** Create t9 on Master *** -connection master; -eval CREATE TABLE t9 (a INT PRIMARY KEY, b BLOB, c CHAR(5) +# Please, check BUG#47741 to see why you are not testing NDB. +if (`SELECT $engine_type != 'NDB'`) +{ + STOP SLAVE; + RESET SLAVE; + eval CREATE TABLE t9 (a INT KEY, b BLOB, c CHAR(5), + d TIMESTAMP, + e INT NOT NULL, + f text not null, + g text, + h blob not null, + i blob) ENGINE=$engine_type; + + --echo *** Create t9 on Master *** + connection master; + eval CREATE TABLE t9 (a INT PRIMARY KEY, b BLOB, c CHAR(5) ) ENGINE=$engine_type; -RESET MASTER; + RESET MASTER; ---echo *** Start Slave *** -connection slave; -START SLAVE; - ---echo *** Master Data Insert *** -connection master; -set @b1 = 'b1b1b1b1'; -set @b1 = concat(@b1,@b1); -INSERT INTO t9 () VALUES(1,@b1,'Kyle'),(2,@b1,'JOE'),(3,@b1,'QA'); - -connection slave; ---source include/wait_for_slave_sql_to_stop.inc ---replace_result $MASTER_MYPORT MASTER_PORT ---replace_column 1 # 4 # 7 # 8 # 9 # 16 # 22 # 23 # 33 # 35 # 36 # ---query_vertical SHOW SLAVE STATUS -SET GLOBAL SQL_SLAVE_SKIP_COUNTER=2; -START SLAVE; + --echo *** Start Slave *** + connection slave; + START SLAVE; + + --echo *** Master Data Insert *** + connection master; + set @b1 = 'b1b1b1b1'; + + set @b1 = concat(@b1,@b1); + INSERT INTO t9 () VALUES(1,@b1,'Kyle'),(2,@b1,'JOE'),(3,@b1,'QA'); + + # the test would stop slave if @@sql_mode for the sql thread + # was set to strict. Otherwise, as with this tests setup, + # the implicit defaults will be inserted into fields even though + # they are declared without DEFAULT clause. + + sync_slave_with_master; + select * from t9; + + # todo: fix Bug #43992 slave sql thread can't tune own sql_mode ... + # and add/restore waiting for stop test + + #--source include/wait_for_slave_sql_to_stop.inc + #--replace_result $MASTER_MYPORT MASTER_PORT + #--replace_column 1 # 4 # 7 # 8 # 9 # 16 # 22 # 23 # 33 # 35 # 36 # + #--query_vertical SHOW SLAVE STATUS + #SET GLOBAL SQL_SLAVE_SKIP_COUNTER=2; + #START SLAVE; +} #--echo *** Drop t9 *** #connection master; === added file 'mysql-test/extra/rpl_tests/rpl_not_null.test' --- a/mysql-test/extra/rpl_tests/rpl_not_null.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/extra/rpl_tests/rpl_not_null.test 2009-10-22 00:19:52 +0000 @@ -0,0 +1,364 @@ +################################################################################# +# This test checks if the replication between "null" fields to either "null" +# fields or "not null" fields works properly. In the first case, the execution +# should work fine. In the second case, it may fail according to the sql_mode +# being used. +# +# The test is devided in three main parts: +# +# 1 - NULL --> NULL (no failures) +# 2 - NULL --> NOT NULL ( sql-mode = STRICT and failures) +# 3 - NULL --> NOT NULL ( sql-mode != STRICT and no failures) +# +################################################################################# +connection master; + +SET SQL_LOG_BIN= 0; +eval CREATE TABLE t1(`a` INT, `b` DATE DEFAULT NULL, +`c` INT DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; + +eval CREATE TABLE t2(`a` INT, `b` DATE DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; + +eval CREATE TABLE t3(`a` INT, `b` DATE DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; + +eval CREATE TABLE t4(`a` INT, `b` DATE DEFAULT NULL, +`c` INT DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; +SET SQL_LOG_BIN= 1; + +connection slave; + +eval CREATE TABLE t1(`a` INT, `b` DATE DEFAULT NULL, +`c` INT DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; + +eval CREATE TABLE t2(`a` INT, `b` DATE DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; + +eval CREATE TABLE t3(`a` INT, `b` DATE DEFAULT '0000-00-00', +`c` INT DEFAULT 500, +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; + +eval CREATE TABLE t4(`a` INT, `b` DATE DEFAULT '0000-00-00', +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; + +--echo ************* EXECUTION WITH INSERTS ************* +connection master; +INSERT INTO t1(a,b,c) VALUES (1, null, 1); +INSERT INTO t1(a,b,c) VALUES (2,'1111-11-11', 2); +INSERT INTO t1(a,b) VALUES (3, null); +INSERT INTO t1(a,c) VALUES (4, 4); +INSERT INTO t1(a) VALUES (5); + +INSERT INTO t2(a,b) VALUES (1, null); +INSERT INTO t2(a,b) VALUES (2,'1111-11-11'); +INSERT INTO t2(a) VALUES (3); + +INSERT INTO t3(a,b) VALUES (1, null); +INSERT INTO t3(a,b) VALUES (2,'1111-11-11'); +INSERT INTO t3(a) VALUES (3); + +INSERT INTO t4(a,b,c) VALUES (1, null, 1); +INSERT INTO t4(a,b,c) VALUES (2,'1111-11-11', 2); +INSERT INTO t4(a,b) VALUES (3, null); +INSERT INTO t4(a,c) VALUES (4, 4); +INSERT INTO t4(a) VALUES (5); + +--echo ************* SHOWING THE RESULT SETS WITH INSERTS ************* +sync_slave_with_master; + +--echo TABLES t1 and t2 must be equal otherwise an error will be thrown. +let $diff_table_1=master:test.t1; +let $diff_table_2=slave:test.t1; +source include/diff_tables.inc; + +let $diff_table_1=master:test.t2; +let $diff_table_2=slave:test.t2; +source include/diff_tables.inc; + +--echo TABLES t2 and t3 must be different. +connection master; +SELECT * FROM t3 ORDER BY a; +connection slave; +SELECT * FROM t3 ORDER BY a; +connection master; +SELECT * FROM t4 ORDER BY a; +connection slave; +SELECT * FROM t4 ORDER BY a; + +--echo ************* EXECUTION WITH UPDATES and REPLACES ************* +connection master; +DELETE FROM t1; +INSERT INTO t1(a,b,c) VALUES (1,'1111-11-11', 1); +REPLACE INTO t1(a,b,c) VALUES (2,'1111-11-11', 2); +UPDATE t1 set b= NULL, c= 300 where a= 1; +REPLACE INTO t1(a,b,c) VALUES (2, NULL, 300); + +--echo ************* SHOWING THE RESULT SETS WITH UPDATES and REPLACES ************* +sync_slave_with_master; + +--echo TABLES t1 and t2 must be equal otherwise an error will be thrown. +let $diff_table_1=master:test.t1; +let $diff_table_2=slave:test.t1; +source include/diff_tables.inc; + +--echo ************* CLEANING ************* +connection master; + +DROP TABLE t1; +DROP TABLE t2; +DROP TABLE t3; +DROP TABLE t4; + +sync_slave_with_master; + +connection master; + +SET SQL_LOG_BIN= 0; +eval CREATE TABLE t1 (`a` INT, `b` BIT DEFAULT NULL, `c` BIT DEFAULT NULL, +PRIMARY KEY (`a`)) ENGINE= $engine; +SET SQL_LOG_BIN= 1; + +connection slave; + +eval CREATE TABLE t1 (`a` INT, `b` BIT DEFAULT b'01', `c` BIT DEFAULT NULL, +PRIMARY KEY (`a`)) ENGINE= $engine; + +--echo ************* EXECUTION WITH INSERTS ************* +connection master; +INSERT INTO t1(a,b,c) VALUES (1, null, b'01'); +INSERT INTO t1(a,b,c) VALUES (2,b'00', b'01'); +INSERT INTO t1(a,b) VALUES (3, null); +INSERT INTO t1(a,c) VALUES (4, b'01'); +INSERT INTO t1(a) VALUES (5); + +--echo ************* SHOWING THE RESULT SETS WITH INSERTS ************* +--echo TABLES t1 and t2 must be different. +sync_slave_with_master; +connection master; +SELECT a,b+0,c+0 FROM t1 ORDER BY a; +connection slave; +SELECT a,b+0,c+0 FROM t1 ORDER BY a; + +--echo ************* EXECUTION WITH UPDATES and REPLACES ************* +connection master; +DELETE FROM t1; +INSERT INTO t1(a,b,c) VALUES (1,b'00', b'01'); +REPLACE INTO t1(a,b,c) VALUES (2,b'00',b'01'); +UPDATE t1 set b= NULL, c= b'00' where a= 1; +REPLACE INTO t1(a,b,c) VALUES (2, NULL, b'00'); + +--echo ************* SHOWING THE RESULT SETS WITH UPDATES and REPLACES ************* +--echo TABLES t1 and t2 must be equal otherwise an error will be thrown. +sync_slave_with_master; +let $diff_table_1=master:test.t1; +let $diff_table_2=slave:test.t1; +source include/diff_tables.inc; + +connection master; + +DROP TABLE t1; + +sync_slave_with_master; + +--echo ################################################################################ +--echo # NULL ---> NOT NULL (STRICT MODE) +--echo # UNCOMMENT THIS AFTER FIXING BUG#43992 +--echo ################################################################################ +#connection slave; +#SET GLOBAL sql_mode="TRADITIONAL"; +# +#STOP SLAVE; +#--source include/wait_for_slave_to_stop.inc +#START SLAVE; +#--source include/wait_for_slave_to_start.inc +# +#let $y=0; +#while (`select $y < 6`) +#{ +# connection master; +# +# SET SQL_LOG_BIN= 0; +# eval CREATE TABLE t1(`a` INT NOT NULL, `b` INT, +# PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; +# eval CREATE TABLE t2(`a` INT NOT NULL, `b` INT, +# PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; +# eval CREATE TABLE t3(`a` INT NOT NULL, `b` INT, +# PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; +# SET SQL_LOG_BIN= 1; +# +# connection slave; +# +# eval CREATE TABLE t1(`a` INT NOT NULL, `b` INT NOT NULL, +# `c` INT NOT NULL, +# PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; +# eval CREATE TABLE t2(`a` INT NOT NULL, `b` INT NOT NULL, +# `c` INT, +# PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; +# eval CREATE TABLE t3(`a` INT NOT NULL, `b` INT NOT NULL, +# `c` INT DEFAULT 500, +# PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; +# +# if (`select $y=0`) +# { +# --echo ************* EXECUTION WITH INSERTS ************* +# connection master; +# INSERT INTO t1(a) VALUES (1); +# } +# +# if (`select $y=1`) +# { +# --echo ************* EXECUTION WITH INSERTS ************* +# connection master; +# INSERT INTO t1(a, b) VALUES (1, NULL); +# } +# +# if (`select $y=2`) +# { +# --echo ************* EXECUTION WITH UPDATES ************* +# connection master; +# INSERT INTO t3(a, b) VALUES (1, 1); +# INSERT INTO t3(a, b) VALUES (2, 1); +# UPDATE t3 SET b = NULL where a= 1; +# } +# +# if (`select $y=3`) +# { +# --echo ************* EXECUTION WITH INSERTS/REPLACES ************* +# connection master; +# REPLACE INTO t3(a, b) VALUES (1, null); +# } +# +# if (`select $y=4`) +# { +# --echo ************* EXECUTION WITH UPDATES/REPLACES ************* +# connection master; +# INSERT INTO t3(a, b) VALUES (1, 1); +# REPLACE INTO t3(a, b) VALUES (1, null); +# } +# +# if (`select $y=5`) +# { +# --echo ************* EXECUTION WITH MULTI-ROW INSERTS ************* +# connection master; +# +# SET SQL_LOG_BIN= 0; +# INSERT INTO t2(a, b) VALUES (1, 1); +# INSERT INTO t2(a, b) VALUES (2, 1); +# INSERT INTO t2(a, b) VALUES (3, null); +# INSERT INTO t2(a, b) VALUES (4, 1); +# INSERT INTO t2(a, b) VALUES (5, 1); +# SET SQL_LOG_BIN= 1; +# +# INSERT INTO t2 SELECT a + 10, b from t2; +# --echo The statement below is just executed to stop processing +# INSERT INTO t1(a) VALUES (1); +# } +# +# --echo ************* SHOWING THE RESULT SETS ************* +# connection slave; +# --source include/wait_for_slave_sql_to_stop.inc +# connection master; +# SELECT * FROM t1 ORDER BY a; +# connection slave; +# SELECT * FROM t1 ORDER BY a; +# connection master; +# SELECT * FROM t2 ORDER BY a; +# connection slave; +# SELECT * FROM t2 ORDER BY a; +# connection master; +# SELECT * FROM t3 ORDER BY a; +# connection slave; +# SELECT * FROM t3 ORDER BY a; +# --source include/reset_master_and_slave.inc +# +# connection master; +# +# DROP TABLE t1; +# DROP TABLE t2; +# DROP TABLE t3; +# +# sync_slave_with_master; +# +# inc $y; +#} +#connection slave; +#SET GLOBAL sql_mode=""; +# +#STOP SLAVE; +#source include/wait_for_slave_to_stop.inc; +#START SLAVE; +#--source include/wait_for_slave_to_start.inc + +--echo ################################################################################ +--echo # NULL ---> NOT NULL (NON-STRICT MODE) +--echo ################################################################################ +connection master; + +SET SQL_LOG_BIN= 0; +eval CREATE TABLE t1(`a` INT NOT NULL, `b` INT, +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; +eval CREATE TABLE t2(`a` INT NOT NULL, `b` INT, +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; +eval CREATE TABLE t3(`a` INT NOT NULL, `b` INT, +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; +SET SQL_LOG_BIN= 1; + +connection slave; + +eval CREATE TABLE t1(`a` INT NOT NULL, `b` INT NOT NULL, +`c` INT NOT NULL, +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; +eval CREATE TABLE t2(`a` INT NOT NULL, `b` INT NOT NULL, +`c` INT, +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; +eval CREATE TABLE t3(`a` INT NOT NULL, `b` INT NOT NULL, +`c` INT DEFAULT 500, +PRIMARY KEY(`a`)) ENGINE=$engine DEFAULT CHARSET=LATIN1; + +--echo ************* EXECUTION WITH INSERTS ************* +connection master; +INSERT INTO t1(a) VALUES (1); +INSERT INTO t1(a, b) VALUES (2, NULL); +INSERT INTO t1(a, b) VALUES (3, 1); + +INSERT INTO t2(a) VALUES (1); +INSERT INTO t2(a, b) VALUES (2, NULL); +INSERT INTO t2(a, b) VALUES (3, 1); + +INSERT INTO t3(a) VALUES (1); +INSERT INTO t3(a, b) VALUES (2, NULL); +INSERT INTO t3(a, b) VALUES (3, 1); +INSERT INTO t3(a, b) VALUES (4, 1); +REPLACE INTO t3(a, b) VALUES (5, null); + +REPLACE INTO t3(a, b) VALUES (3, null); +UPDATE t3 SET b = NULL where a = 4; + +--echo ************* SHOWING THE RESULT SETS ************* +connection master; +sync_slave_with_master; + +connection master; +SELECT * FROM t1 ORDER BY a; +connection slave; +SELECT * FROM t1 ORDER BY a; +connection master; +SELECT * FROM t2 ORDER BY a; +connection slave; +SELECT * FROM t2 ORDER BY a; +connection master; +SELECT * FROM t3 ORDER BY a; +connection slave; +SELECT * FROM t3 ORDER BY a; + +connection master; + +DROP TABLE t1; +DROP TABLE t2; +DROP TABLE t3; + +sync_slave_with_master; === modified file 'mysql-test/extra/rpl_tests/rpl_row_tabledefs.test' --- a/mysql-test/extra/rpl_tests/rpl_row_tabledefs.test 2008-03-14 20:02:52 +0000 +++ b/mysql-test/extra/rpl_tests/rpl_row_tabledefs.test 2009-10-22 00:10:42 +0000 @@ -111,21 +111,18 @@ SELECT a,b,x FROM t1_int ORDER BY a; SELECT a,b,HEX(x),HEX(y),HEX(z) FROM t1_bit ORDER BY a; SELECT a,b,x FROM t1_char ORDER BY a; -# Each of these inserts should generate an error and stop the slave - connection master; INSERT INTO t9 VALUES (2); sync_slave_with_master; # Now slave is guaranteed to be running connection master; INSERT INTO t1_nodef VALUES (1,2); -connection slave; ---source include/wait_for_slave_sql_to_stop.inc ---replace_result $MASTER_MYPORT MASTER_PORT ---replace_column 1 # 4 # 7 # 8 # 9 # 20 <Last_Error> 22 # 23 # 33 # 35 <Last_IO_Errno> 36 <Last_IO_Error> 38 <Last_SQL_Error> ---query_vertical SHOW SLAVE STATUS -SET GLOBAL SQL_SLAVE_SKIP_COUNTER=2; -START SLAVE; + +# Last insert on wider slave table succeeds while slave sql sql_mode permits. +# The previous version of the above test expected slave sql to stop. +# bug#38173 relaxed conditions to stop only with the strict mode. +sync_slave_with_master; +select count(*) from t1_nodef; # # Replicating to tables with fewer columns at the end works as of WL#3228 === modified file 'mysql-test/extra/rpl_tests/rpl_stm_000001.test' --- a/mysql-test/extra/rpl_tests/rpl_stm_000001.test 2009-10-20 18:00:07 +0000 +++ b/mysql-test/extra/rpl_tests/rpl_stm_000001.test 2009-11-18 14:50:31 +0000 @@ -1,6 +1,11 @@ --- source include/have_binlog_format_mixed_or_statement.inc +# Requires binlog_format=statement format since query involving +# get_lock() is logged in row format if binlog_format=mixed or row. +-- source include/have_binlog_format_statement.inc -- source include/master-slave.inc +CALL mtr.add_suppression("Statement may not be safe to log in statement format."); + +# Load some data into t1 create table t1 (word char(20) not null); load data infile '../../std_data/words.dat' into table t1; --replace_result $MYSQL_TEST_DIR MYSQL_TEST_DIR @@ -10,9 +15,7 @@ select * from t1 limit 10; # # Test slave with wrong password # -save_master_pos; -connection slave; -sync_with_master; +sync_slave_with_master; stop slave; connection master; set password for root@"localhost" = password('foo'); @@ -29,16 +32,12 @@ sleep 2; create table t3(n int); insert into t3 values(1),(2); -save_master_pos; -connection slave; -sync_with_master; +sync_slave_with_master; select * from t3; select sum(length(word)) from t1; connection master; drop table t1,t3; -save_master_pos; -connection slave; -sync_with_master; +sync_slave_with_master; # Test if the slave SQL thread can be more than 16K behind the slave # I/O thread (> IO_SIZE) @@ -77,12 +76,13 @@ unlock tables; connection master; create table t2(id int); insert into t2 values(connection_id()); -save_master_pos; connection master1; # Avoid generating result create temporary table t3(n int); +--disable_warnings insert into t3 select get_lock('crash_lock%20C', 1) from t2; +--enable_warnings connection master; send update t1 set n = n + get_lock('crash_lock%20C', 2); @@ -93,8 +93,11 @@ kill @id; # We don't drop t3 as this is a temporary table drop table t2; connection master; +# The get_lock function causes warning for unsafe statement. +--disable_warnings --error 1317,2013 reap; +--enable_warnings connection slave; # The SQL slave thread should now have stopped because the query was killed on # the master (so it has a non-zero error code in the binlog). @@ -117,16 +120,12 @@ insert into mysql.user (Host, User, Pass select select_priv,user from mysql.user where user = _binary'blafasel2'; update mysql.user set Select_priv = "Y" where User= _binary"blafasel2"; select select_priv,user from mysql.user where user = _binary'blafasel2'; -save_master_pos; -connection slave; -sync_with_master; +sync_slave_with_master; select n from t1; select select_priv,user from mysql.user where user = _binary'blafasel2'; connection master1; drop table t1; delete from mysql.user where user="blafasel2"; -save_master_pos; -connection slave; -sync_with_master; +sync_slave_with_master; # End of 4.1 tests === modified file 'mysql-test/include/mtr_warnings.sql' --- a/mysql-test/include/mtr_warnings.sql 2009-12-03 11:19:05 +0000 +++ b/mysql-test/include/mtr_warnings.sql 2010-01-15 15:27:55 +0000 @@ -175,6 +175,8 @@ INSERT INTO global_suppressions VALUES ("Can't find file: '.\\\\test\\\\\\?{8}.frm'"), ("Slave: Unknown table 't1' Error_code: 1051"), + /* Maria storage engine dependent tests */ + /* maria-recovery.test has warning about missing log file */ ("File '.*maria_log.000.*' not found \$Errcode: 2\$"), /* and about marked-corrupted table */ @@ -184,6 +186,14 @@ INSERT INTO global_suppressions VALUES ("Table '..mysqltest.t_corrupted2' is marked as crashed and should be"), ("Incorrect key file for table '..mysqltest.t_corrupted2.MAI'"), + /* + Transient network failures that cause warnings on reconnect. + BUG#47743 and BUG#47983. + */ + ("Slave I/O: Get master SERVER_ID failed with error:.*"), + ("Slave I/O: Get master clock failed with error:.*"), + ("Slave I/O: Get master COLLATION_SERVER failed with error:.*"), + ("Slave I/O: Get master TIME_ZONE failed with error:.*"), ("THE_LAST_SUPPRESSION")|| === modified file 'mysql-test/include/ps_conv.inc' --- a/mysql-test/include/ps_conv.inc 2007-07-04 09:20:53 +0000 +++ b/mysql-test/include/ps_conv.inc 2010-01-19 10:36:52 +0000 @@ -30,6 +30,9 @@ --disable_warnings drop table if exists t5 ; --enable_warnings +--disable_query_log +SET TIME_ZONE= '+03:00'; +--enable_query_log set @arg01= 8; set @arg02= 8.0; set @arg03= 80.00000000000e-1; === modified file 'mysql-test/lib/mtr_cases.pm' --- a/mysql-test/lib/mtr_cases.pm 2009-12-06 17:34:54 +0000 +++ b/mysql-test/lib/mtr_cases.pm 2010-01-15 15:27:55 +0000 @@ -524,6 +524,10 @@ sub collect_one_suite next if ($test->{'name'} eq 'sys_vars.innodb_thread_concurrency_basic'); # Can't work with InnoPlug. Test framework needs to be re-designed. next if ($test->{'name'} eq 'main.innodb_bug46000'); + # Fails with innodb plugin + next if ($test->{'name'} eq 'main.innodb-autoinc'); + # Fails with innodb plugin: r6185 Testcases changes not included + next if ($test->{'name'} eq 'main.innodb_bug44369'); # Copy test options my $new_test= My::Test->new(); while (my ($key, $value) = each(%$test)) === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2010-01-06 21:27:53 +0000 +++ b/mysql-test/mysql-test-run.pl 2010-01-29 10:42:31 +0000 @@ -413,7 +413,6 @@ sub main { # Not all tests completed, failure mtr_report(); mtr_report("Only ", int(@$completed), " of $num_tests completed."); - mtr_error("Not all tests completed"); } mtr_print_line(); @@ -425,6 +424,10 @@ sub main { mtr_report_stats($fail, $completed, $extra_warnings); + if ( @$completed != $num_tests) + { + mtr_error("Not all tests completed"); + } exit(0); } @@ -3425,7 +3428,7 @@ sub restart_forced_by_test # Return timezone value of tinfo or default value sub timezone { my ($tinfo)= @_; - return $tinfo->{timezone} || "GMT-3"; + return $tinfo->{timezone} || "DEFAULT"; } @@ -3455,7 +3458,11 @@ sub run_testcase ($$) { # Init variables that can change between each test case # ------------------------------------------------------- my $timezone= timezone($tinfo); - $ENV{'TZ'}= $timezone; + if ($timezone ne 'DEFAULT') { + $ENV{'TZ'}= $timezone; + } else { + delete($ENV{'TZ'}); + } mtr_verbose("Setting timezone: $timezone"); if ( ! using_extern() ) === modified file 'mysql-test/r/archive.result' --- a/mysql-test/r/archive.result 2009-09-10 06:58:13 +0000 +++ b/mysql-test/r/archive.result 2009-11-11 08:03:29 +0000 @@ -12717,3 +12717,14 @@ COUNT(t1.a) 729 DROP TABLE t1; SET @@join_buffer_size= @save_join_buffer_size; +SHOW CREATE TABLE t1; +ERROR HY000: Table upgrade required. Please do "REPAIR TABLE `t1`" or dump/reload to fix it! +SELECT * FROM t1; +ERROR HY000: Table upgrade required. Please do "REPAIR TABLE `t1`" or dump/reload to fix it! +INSERT INTO t1 (col1, col2) VALUES (1, "value"); +ERROR HY000: Table upgrade required. Please do "REPAIR TABLE `t1`" or dump/reload to fix it! +REPAIR TABLE t1; +Table Op Msg_type Msg_text +test.t1 repair Error Table upgrade required. Please do "REPAIR TABLE `t1`" or dump/reload to fix it! +test.t1 repair error Corrupt +DROP TABLE t1; === added file 'mysql-test/r/bug47671.result' --- a/mysql-test/r/bug47671.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/r/bug47671.result 2010-01-15 15:27:55 +0000 @@ -0,0 +1,14 @@ +# +# Bug#47671 - wrong character-set after upgrade from 5.1.34 to 5.1.39 +# +# Extract only charset information from 'status' command output using regex +-------------- + +Server: MariaDB +Server characterset: utf8 +Db characterset: utf8 +Client characterset: utf8 +Conn. characterset: utf8 + +-------------- + === modified file 'mysql-test/r/delayed.result' --- a/mysql-test/r/delayed.result 2009-03-11 15:32:42 +0000 +++ b/mysql-test/r/delayed.result 2010-01-15 15:27:55 +0000 @@ -314,4 +314,16 @@ a b 2 2 drop table t1; set global low_priority_updates = @old_delayed_updates; +# +# Bug #47682 strange behaviour of INSERT DELAYED +# +DROP TABLE IF EXISTS t1, t2; +CREATE TABLE t1 (f1 integer); +CREATE TABLE t2 (f1 integer); +FLUSH TABLES WITH READ LOCK; +LOCK TABLES t1 READ; +INSERT DELAYED INTO t2 VALUES (1); +Got one of the listed errors +UNLOCK TABLES; +DROP TABLE t1, t2; End of 5.1 tests === modified file 'mysql-test/r/delete.result' --- a/mysql-test/r/delete.result 2009-09-28 10:48:52 +0000 +++ b/mysql-test/r/delete.result 2009-11-18 09:32:03 +0000 @@ -324,3 +324,16 @@ a 1 2 DROP TABLE t1, t2, t3; +# +# Bug #46425 crash in Diagnostics_area::set_ok_status, +# empty statement, DELETE IGNORE +# +CREATE table t1 (i INTEGER); +INSERT INTO t1 VALUES (1); +CREATE TRIGGER tr1 AFTER DELETE ON t1 FOR EACH ROW +BEGIN +INSERT INTO t1 SELECT * FROM t1 AS A; +END | +DELETE IGNORE FROM t1; +ERROR HY000: Can't update table 't1' in stored function/trigger because it is already used by statement which invoked this stored function/trigger. +DROP TABLE t1; === modified file 'mysql-test/r/fulltext.result' --- a/mysql-test/r/fulltext.result 2009-09-07 20:50:10 +0000 +++ b/mysql-test/r/fulltext.result 2010-01-15 15:27:55 +0000 @@ -559,3 +559,42 @@ EXECUTE s; MATCH (col) AGAINST('findme') DEALLOCATE PREPARE s; DROP TABLE t1; +# +# Bug #47930: MATCH IN BOOLEAN MODE returns too many results +# inside subquery +# +CREATE TABLE t1 (a int); +INSERT INTO t1 VALUES (1), (2); +CREATE TABLE t2 (a int, b2 char(10), FULLTEXT KEY b2 (b2)); +INSERT INTO t2 VALUES (1,'Scargill'); +CREATE TABLE t3 (a int, b int); +INSERT INTO t3 VALUES (1,1), (2,1); +# t2 should use full text index +EXPLAIN +SELECT count(*) FROM t1 WHERE +not exists( +SELECT 1 FROM t2, t3 +WHERE t3.a=t1.a AND MATCH(b2) AGAINST('scargill' IN BOOLEAN MODE) +); +id select_type table type possible_keys key key_len ref rows Extra +1 PRIMARY t1 ALL NULL NULL NULL NULL 2 Using where +2 DEPENDENT SUBQUERY t2 fulltext b2 b2 0 1 Using where +2 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 2 Using where +# should return 0 +SELECT count(*) FROM t1 WHERE +not exists( +SELECT 1 FROM t2, t3 +WHERE t3.a=t1.a AND MATCH(b2) AGAINST('scargill' IN BOOLEAN MODE) +); +count(*) +0 +# should return 0 +SELECT count(*) FROM t1 WHERE +not exists( +SELECT 1 FROM t2 IGNORE INDEX (b2), t3 +WHERE t3.a=t1.a AND MATCH(b2) AGAINST('scargill' IN BOOLEAN MODE) +); +count(*) +0 +DROP TABLE t1,t2,t3; +End of 5.1 tests === modified file 'mysql-test/r/func_group.result' --- a/mysql-test/r/func_group.result 2009-10-14 08:46:50 +0000 +++ b/mysql-test/r/func_group.result 2009-11-24 15:26:13 +0000 @@ -885,7 +885,7 @@ cast(sum(distinct df) as signed) 3 select cast(min(df) as signed) from t1; cast(min(df) as signed) -0 +1 select 1e8 * sum(distinct df) from t1; 1e8 * sum(distinct df) 330000000 @@ -1520,4 +1520,197 @@ max i # Cleanup # DROP TABLE t1; +# +# Bug#43668: Wrong comparison and MIN/MAX for YEAR(2) +# +create table t1 (f1 year(2), f2 year(4), f3 date, f4 datetime); +insert into t1 values +(98,1998,19980101,"1998-01-01 00:00:00"), +(00,2000,20000101,"2000-01-01 00:00:01"), +(02,2002,20020101,"2002-01-01 23:59:59"), +(60,2060,20600101,"2060-01-01 11:11:11"), +(70,1970,19700101,"1970-11-11 22:22:22"), +(NULL,NULL,NULL,NULL); +select min(f1),max(f1) from t1; +min(f1) max(f1) +70 60 +select min(f2),max(f2) from t1; +min(f2) max(f2) +1970 2060 +select min(f3),max(f3) from t1; +min(f3) max(f3) +1970-01-01 2060-01-01 +select min(f4),max(f4) from t1; +min(f4) max(f4) +1970-11-11 22:22:22 2060-01-01 11:11:11 +select a.f1 as a, b.f1 as b, a.f1 > b.f1 as gt, +a.f1 < b.f1 as lt, a.f1<=>b.f1 as eq +from t1 a, t1 b; +a b gt lt eq +98 98 0 0 1 +00 98 1 0 0 +02 98 1 0 0 +60 98 1 0 0 +70 98 0 1 0 +NULL 98 NULL NULL 0 +98 00 0 1 0 +00 00 0 0 1 +02 00 1 0 0 +60 00 1 0 0 +70 00 0 1 0 +NULL 00 NULL NULL 0 +98 02 0 1 0 +00 02 0 1 0 +02 02 0 0 1 +60 02 1 0 0 +70 02 0 1 0 +NULL 02 NULL NULL 0 +98 60 0 1 0 +00 60 0 1 0 +02 60 0 1 0 +60 60 0 0 1 +70 60 0 1 0 +NULL 60 NULL NULL 0 +98 70 1 0 0 +00 70 1 0 0 +02 70 1 0 0 +60 70 1 0 0 +70 70 0 0 1 +NULL 70 NULL NULL 0 +98 NULL NULL NULL 0 +00 NULL NULL NULL 0 +02 NULL NULL NULL 0 +60 NULL NULL NULL 0 +70 NULL NULL NULL 0 +NULL NULL NULL NULL 1 +select a.f1 as a, b.f2 as b, a.f1 > b.f2 as gt, +a.f1 < b.f2 as lt, a.f1<=>b.f2 as eq +from t1 a, t1 b; +a b gt lt eq +98 1998 0 0 1 +00 1998 1 0 0 +02 1998 1 0 0 +60 1998 1 0 0 +70 1998 0 1 0 +NULL 1998 NULL NULL 0 +98 2000 0 1 0 +00 2000 0 0 1 +02 2000 1 0 0 +60 2000 1 0 0 +70 2000 0 1 0 +NULL 2000 NULL NULL 0 +98 2002 0 1 0 +00 2002 0 1 0 +02 2002 0 0 1 +60 2002 1 0 0 +70 2002 0 1 0 +NULL 2002 NULL NULL 0 +98 2060 0 1 0 +00 2060 0 1 0 +02 2060 0 1 0 +60 2060 0 0 1 +70 2060 0 1 0 +NULL 2060 NULL NULL 0 +98 1970 1 0 0 +00 1970 1 0 0 +02 1970 1 0 0 +60 1970 1 0 0 +70 1970 0 0 1 +NULL 1970 NULL NULL 0 +98 NULL NULL NULL 0 +00 NULL NULL NULL 0 +02 NULL NULL NULL 0 +60 NULL NULL NULL 0 +70 NULL NULL NULL 0 +NULL NULL NULL NULL 1 +select a.f1 as a, b.f3 as b, a.f1 > b.f3 as gt, +a.f1 < b.f3 as lt, a.f1<=>b.f3 as eq +from t1 a, t1 b; +a b gt lt eq +98 1998-01-01 0 1 0 +00 1998-01-01 1 0 0 +02 1998-01-01 1 0 0 +60 1998-01-01 1 0 0 +70 1998-01-01 0 1 0 +NULL 1998-01-01 NULL NULL 0 +98 2000-01-01 0 1 0 +00 2000-01-01 0 1 0 +02 2000-01-01 1 0 0 +60 2000-01-01 1 0 0 +70 2000-01-01 0 1 0 +NULL 2000-01-01 NULL NULL 0 +98 2002-01-01 0 1 0 +00 2002-01-01 0 1 0 +02 2002-01-01 0 1 0 +60 2002-01-01 1 0 0 +70 2002-01-01 0 1 0 +NULL 2002-01-01 NULL NULL 0 +98 2060-01-01 0 1 0 +00 2060-01-01 0 1 0 +02 2060-01-01 0 1 0 +60 2060-01-01 0 1 0 +70 2060-01-01 0 1 0 +NULL 2060-01-01 NULL NULL 0 +98 1970-01-01 1 0 0 +00 1970-01-01 1 0 0 +02 1970-01-01 1 0 0 +60 1970-01-01 1 0 0 +70 1970-01-01 0 1 0 +NULL 1970-01-01 NULL NULL 0 +98 NULL NULL NULL 0 +00 NULL NULL NULL 0 +02 NULL NULL NULL 0 +60 NULL NULL NULL 0 +70 NULL NULL NULL 0 +NULL NULL NULL NULL 1 +select a.f1 as a, b.f4 as b, a.f1 > b.f4 as gt, +a.f1 < b.f4 as lt, a.f1<=>b.f4 as eq +from t1 a, t1 b; +a b gt lt eq +98 1998-01-01 00:00:00 0 1 0 +00 1998-01-01 00:00:00 1 0 0 +02 1998-01-01 00:00:00 1 0 0 +60 1998-01-01 00:00:00 1 0 0 +70 1998-01-01 00:00:00 0 1 0 +NULL 1998-01-01 00:00:00 NULL NULL 0 +98 2000-01-01 00:00:01 0 1 0 +00 2000-01-01 00:00:01 0 1 0 +02 2000-01-01 00:00:01 1 0 0 +60 2000-01-01 00:00:01 1 0 0 +70 2000-01-01 00:00:01 0 1 0 +NULL 2000-01-01 00:00:01 NULL NULL 0 +98 2002-01-01 23:59:59 0 1 0 +00 2002-01-01 23:59:59 0 1 0 +02 2002-01-01 23:59:59 0 1 0 +60 2002-01-01 23:59:59 1 0 0 +70 2002-01-01 23:59:59 0 1 0 +NULL 2002-01-01 23:59:59 NULL NULL 0 +98 2060-01-01 11:11:11 0 1 0 +00 2060-01-01 11:11:11 0 1 0 +02 2060-01-01 11:11:11 0 1 0 +60 2060-01-01 11:11:11 0 1 0 +70 2060-01-01 11:11:11 0 1 0 +NULL 2060-01-01 11:11:11 NULL NULL 0 +98 1970-11-11 22:22:22 1 0 0 +00 1970-11-11 22:22:22 1 0 0 +02 1970-11-11 22:22:22 1 0 0 +60 1970-11-11 22:22:22 1 0 0 +70 1970-11-11 22:22:22 0 1 0 +NULL 1970-11-11 22:22:22 NULL NULL 0 +98 NULL NULL NULL 0 +00 NULL NULL NULL 0 +02 NULL NULL NULL 0 +60 NULL NULL NULL 0 +70 NULL NULL NULL 0 +NULL NULL NULL NULL 1 +select *, f1 = f2 from t1; +f1 f2 f3 f4 f1 = f2 +98 1998 1998-01-01 1998-01-01 00:00:00 1 +00 2000 2000-01-01 2000-01-01 00:00:01 1 +02 2002 2002-01-01 2002-01-01 23:59:59 1 +60 2060 2060-01-01 2060-01-01 11:11:11 1 +70 1970 1970-01-01 1970-11-11 22:22:22 1 +NULL NULL NULL NULL NULL +drop table t1; +# End of 5.1 tests === modified file 'mysql-test/r/func_misc.result' --- a/mysql-test/r/func_misc.result 2009-10-28 07:52:34 +0000 +++ b/mysql-test/r/func_misc.result 2010-01-11 13:15:28 +0000 @@ -104,95 +104,6 @@ t1 CREATE TABLE `t1` ( `length(uuid())` int(10) NOT NULL DEFAULT '0' ) ENGINE=MyISAM DEFAULT CHARSET=latin1 drop table t1; -#------------------------------------------------------------------------ -# Tests for Bug#6760 and Bug#12689 -SET @row_count = 4; -SET @sleep_time_per_result_row = 1; -SET @max_acceptable_delay = 2; -SET @@global.query_cache_size = 1024 * 64; -DROP TEMPORARY TABLE IF EXISTS t_history; -DROP TABLE IF EXISTS t1; -CREATE TEMPORARY TABLE t_history (attempt SMALLINT, -start_ts DATETIME, end_ts DATETIME, -start_cached INTEGER, end_cached INTEGER); -CREATE TABLE t1 (f1 BIGINT); -INSERT INTO t_history -SET attempt = 4 - 4 + 1, start_ts = NOW(), -start_cached = 0; -SELECT *, SLEEP(@sleep_time_per_result_row) FROM t1; -f1 SLEEP(@sleep_time_per_result_row) -1 0 -1 0 -1 0 -1 0 -UPDATE t_history SET end_ts = NOW() -WHERE attempt = 4 - 4 + 1; -UPDATE t_history SET end_cached = 0 -WHERE attempt = 4 - 4 + 1; -INSERT INTO t_history -SET attempt = 4 - 3 + 1, start_ts = NOW(), -start_cached = 0; -SELECT *, SLEEP(@sleep_time_per_result_row) FROM t1; -f1 SLEEP(@sleep_time_per_result_row) -1 0 -1 0 -1 0 -1 0 -UPDATE t_history SET end_ts = NOW() -WHERE attempt = 4 - 3 + 1; -UPDATE t_history SET end_cached = 0 -WHERE attempt = 4 - 3 + 1; -INSERT INTO t_history -SET attempt = 4 - 2 + 1, start_ts = NOW(), -start_cached = 0; -SELECT *, SLEEP(@sleep_time_per_result_row) FROM t1; -f1 SLEEP(@sleep_time_per_result_row) -1 0 -1 0 -1 0 -1 0 -UPDATE t_history SET end_ts = NOW() -WHERE attempt = 4 - 2 + 1; -UPDATE t_history SET end_cached = 0 -WHERE attempt = 4 - 2 + 1; -INSERT INTO t_history -SET attempt = 4 - 1 + 1, start_ts = NOW(), -start_cached = 0; -SELECT *, SLEEP(@sleep_time_per_result_row) FROM t1; -f1 SLEEP(@sleep_time_per_result_row) -1 0 -1 0 -1 0 -1 0 -UPDATE t_history SET end_ts = NOW() -WHERE attempt = 4 - 1 + 1; -UPDATE t_history SET end_cached = 0 -WHERE attempt = 4 - 1 + 1; -# Test 1: Does the query with SLEEP need a reasonable time? -SELECT COUNT(*) >= 4 - 1 INTO @aux1 FROM t_history -WHERE TIMEDIFF(end_ts,start_ts) - @sleep_time_per_result_row * @row_count -BETWEEN 0 AND @max_acceptable_delay; -SELECT @aux1 AS "Expect 1"; -Expect 1 -1 -# Test 2: Does the query with SLEEP need a reasonable time even in case -# of the non first execution? -SELECT COUNT(*) >= 4 - 1 - 1 INTO @aux2 FROM t_history -WHERE TIMEDIFF(end_ts,start_ts) - @sleep_time_per_result_row * @row_count -BETWEEN 0 AND @max_acceptable_delay -AND attempt > 1; -SELECT @aux2 AS "Expect 1"; -Expect 1 -1 -# Test 3: The query with SLEEP must be not cached. -SELECT COUNT(*) = 4 INTO @aux3 FROM t_history -WHERE end_cached = start_cached; -SELECT @aux3 AS "Expect 1"; -Expect 1 -1 -DROP TABLE t1; -DROP TEMPORARY TABLE t_history; -SET @@global.query_cache_size = default; create table t1 select INET_ATON('255.255.0.1') as `a`; show create table t1; Table Create Table === modified file 'mysql-test/r/grant.result' --- a/mysql-test/r/grant.result 2009-10-27 10:09:36 +0000 +++ b/mysql-test/r/grant.result 2010-01-29 10:42:31 +0000 @@ -29,15 +29,15 @@ show grants for mysqltest_1@localhost; Grants for mysqltest_1@localhost GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' GRANT SELECT ON `mysqltest`.* TO 'mysqltest_1'@'localhost' -grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "MySQL AB"; +grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "Monty Program Ab"; show grants for mysqltest_1@localhost; Grants for mysqltest_1@localhost -GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'MySQL AB' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' +GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'Monty Program Ab' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' GRANT SELECT ON `mysqltest`.* TO 'mysqltest_1'@'localhost' revoke all privileges on mysqltest.* from mysqltest_1@localhost; show grants for mysqltest_1@localhost; Grants for mysqltest_1@localhost -GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'MySQL AB' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' +GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'Monty Program Ab' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' delete from mysql.user where user='mysqltest_1'; flush privileges; delete from mysql.user where user='mysqltest_1'; === modified file 'mysql-test/r/grant2.result' --- a/mysql-test/r/grant2.result 2009-02-27 08:03:47 +0000 +++ b/mysql-test/r/grant2.result 2009-10-30 05:06:10 +0000 @@ -443,3 +443,30 @@ DROP TABLE db1.t1, db1.t2; DROP USER mysqltest1@localhost; DROP DATABASE db1; End of 5.0 tests +USE mysql; +SELECT LEFT(CURRENT_USER(),INSTR(CURRENT_USER(),'@')-1) INTO @u; +SELECT MID(CURRENT_USER(),INSTR(CURRENT_USER(),'@')+1) INTO @h; +SELECT password FROM user WHERE user=@u AND host=@h INTO @pwd; +SELECT user,host,password,insert_priv FROM user WHERE user=@u AND host=@h; +user host password insert_priv +root localhost Y +UPDATE user SET insert_priv='N' WHERE user=@u AND host=@h; +SELECT user,host,password,insert_priv FROM user WHERE user=@u AND host=@h; +user host password insert_priv +root localhost N +GRANT INSERT ON *.* TO CURRENT_USER(); +SELECT user,host,password,insert_priv FROM user WHERE user=@u AND host=@h; +user host password insert_priv +root localhost Y +UPDATE user SET insert_priv='N' WHERE user=@u AND host=@h; +GRANT INSERT ON *.* TO CURRENT_USER() IDENTIFIED BY 'keksdose'; +SELECT user,host,password,insert_priv FROM user WHERE user=@u AND host=@h; +user host password insert_priv +root localhost *0BB7188CF0DE9B403BA66E9DD810D82652D002EB Y +UPDATE user SET password=@pwd WHERE user=@u AND host=@h; +SELECT user,host,password,insert_priv FROM user WHERE user=@u AND host=@h; +user host password insert_priv +root localhost Y +FLUSH PRIVILEGES; +USE test; +End of 5.1 tests === modified file 'mysql-test/r/group_min_max.result' --- a/mysql-test/r/group_min_max.result 2009-10-09 09:30:40 +0000 +++ b/mysql-test/r/group_min_max.result 2009-11-23 10:04:17 +0000 @@ -2501,6 +2501,17 @@ SELECT a, MAX(b) FROM t WHERE b > 0 AND a MAX(b) 2 1 DROP TABLE t; +# +# Bug #48472: Loose index scan inappropriately chosen for some WHERE +# conditions +# +CREATE TABLE t (a INT, b INT, INDEX (a,b)); +INSERT INTO t VALUES (2,0), (2,0), (2,1), (2,1); +INSERT INTO t SELECT * FROM t; +SELECT a, MAX(b) FROM t WHERE 0=b+0 GROUP BY a; +a MAX(b) +2 0 +DROP TABLE t; End of 5.0 tests # # Bug #46607: Assertion failed: (cond_type == Item::FUNC_ITEM) results in === modified file 'mysql-test/r/information_schema.result' --- a/mysql-test/r/information_schema.result 2009-09-29 20:19:43 +0000 +++ b/mysql-test/r/information_schema.result 2010-01-15 15:58:25 +0000 @@ -85,6 +85,7 @@ TABLE_PRIVILEGES TRIGGERS USER_PRIVILEGES VIEWS +XTRADB_ADMIN_COMMAND XTRADB_ENHANCEMENTS columns_priv db @@ -865,8 +866,8 @@ TABLE_CONSTRAINTS TABLE_NAME select TABLE_PRIVILEGES TABLE_NAME select VIEWS TABLE_NAME select INNODB_BUFFER_POOL_PAGES_INDEX table_name select -INNODB_INDEX_STATS table_name select INNODB_TABLE_STATS table_name select +INNODB_INDEX_STATS table_name select delete from mysql.user where user='mysqltest_4'; delete from mysql.db where user='mysqltest_4'; flush privileges; === modified file 'mysql-test/r/information_schema_all_engines.result' --- a/mysql-test/r/information_schema_all_engines.result 2009-08-03 20:09:53 +0000 +++ b/mysql-test/r/information_schema_all_engines.result 2010-01-15 15:58:25 +0000 @@ -35,7 +35,7 @@ INNODB_CMP INNODB_RSEG XTRADB_ENHANCEMENTS INNODB_BUFFER_POOL_PAGES_INDEX -INNODB_INDEX_STATS +XTRADB_ADMIN_COMMAND INNODB_TRX INNODB_CMP_RESET INNODB_LOCK_WAITS @@ -44,6 +44,7 @@ INNODB_LOCKS INNODB_CMPMEM INNODB_TABLE_STATS INNODB_BUFFER_POOL_PAGES_BLOB +INNODB_INDEX_STATS SELECT t.table_name, c1.column_name FROM information_schema.tables t INNER JOIN @@ -93,7 +94,7 @@ INNODB_CMP page_size INNODB_RSEG rseg_id XTRADB_ENHANCEMENTS name INNODB_BUFFER_POOL_PAGES_INDEX schema_name -INNODB_INDEX_STATS table_name +XTRADB_ADMIN_COMMAND result_message INNODB_TRX trx_id INNODB_CMP_RESET page_size INNODB_LOCK_WAITS requesting_trx_id @@ -102,6 +103,7 @@ INNODB_LOCKS lock_id INNODB_CMPMEM page_size INNODB_TABLE_STATS table_name INNODB_BUFFER_POOL_PAGES_BLOB space_id +INNODB_INDEX_STATS table_name SELECT t.table_name, c1.column_name FROM information_schema.tables t INNER JOIN @@ -151,7 +153,7 @@ INNODB_CMP page_size INNODB_RSEG rseg_id XTRADB_ENHANCEMENTS name INNODB_BUFFER_POOL_PAGES_INDEX schema_name -INNODB_INDEX_STATS table_name +XTRADB_ADMIN_COMMAND result_message INNODB_TRX trx_id INNODB_CMP_RESET page_size INNODB_LOCK_WAITS requesting_trx_id @@ -160,6 +162,7 @@ INNODB_LOCKS lock_id INNODB_CMPMEM page_size INNODB_TABLE_STATS table_name INNODB_BUFFER_POOL_PAGES_BLOB space_id +INNODB_INDEX_STATS table_name select 1 as f1 from information_schema.tables where "CHARACTER_SETS"= (select cast(table_name as char) from information_schema.tables order by table_name limit 1) limit 1; @@ -262,7 +265,7 @@ Database: information_schema | INNODB_RSEG | | XTRADB_ENHANCEMENTS | | INNODB_BUFFER_POOL_PAGES_INDEX | -| INNODB_INDEX_STATS | +| XTRADB_ADMIN_COMMAND | | INNODB_TRX | | INNODB_CMP_RESET | | INNODB_LOCK_WAITS | @@ -271,6 +274,7 @@ Database: information_schema | INNODB_CMPMEM | | INNODB_TABLE_STATS | | INNODB_BUFFER_POOL_PAGES_BLOB | +| INNODB_INDEX_STATS | +---------------------------------------+ Database: INFORMATION_SCHEMA +---------------------------------------+ @@ -310,7 +314,7 @@ Database: INFORMATION_SCHEMA | INNODB_RSEG | | XTRADB_ENHANCEMENTS | | INNODB_BUFFER_POOL_PAGES_INDEX | -| INNODB_INDEX_STATS | +| XTRADB_ADMIN_COMMAND | | INNODB_TRX | | INNODB_CMP_RESET | | INNODB_LOCK_WAITS | @@ -319,6 +323,7 @@ Database: INFORMATION_SCHEMA | INNODB_CMPMEM | | INNODB_TABLE_STATS | | INNODB_BUFFER_POOL_PAGES_BLOB | +| INNODB_INDEX_STATS | +---------------------------------------+ Wildcard: inf_rmation_schema +--------------------+ @@ -328,5 +333,5 @@ Wildcard: inf_rmation_schema +--------------------+ SELECT table_schema, count(*) FROM information_schema.TABLES WHERE table_schema IN ('mysql', 'INFORMATION_SCHEMA', 'test', 'mysqltest') AND table_name<>'ndb_binlog_index' AND table_name<>'ndb_apply_status' GROUP BY TABLE_SCHEMA; table_schema count(*) -information_schema 43 +information_schema 44 mysql 22 === modified file 'mysql-test/r/innodb-autoinc.result' --- a/mysql-test/r/innodb-autoinc.result 2009-12-03 11:34:11 +0000 +++ b/mysql-test/r/innodb-autoinc.result 2010-01-15 17:02:57 +0000 @@ -197,7 +197,7 @@ c1 c2 5 9 DROP TABLE t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=100, @@SESSION.AUTO_INCREMENT_OFFSET=10; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 100 auto_increment_offset 10 @@ -230,7 +230,7 @@ c1 DROP TABLE t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 1 auto_increment_offset 1 @@ -269,7 +269,7 @@ c1 DROP TABLE t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 1 auto_increment_offset 1 @@ -282,7 +282,7 @@ SELECT * FROM t1; c1 -1 SET @@SESSION.AUTO_INCREMENT_INCREMENT=100, @@SESSION.AUTO_INCREMENT_OFFSET=10; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 100 auto_increment_offset 10 @@ -315,7 +315,7 @@ c1 DROP TABLE t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 1 auto_increment_offset 1 @@ -330,7 +330,7 @@ SELECT * FROM t1; c1 1 SET @@SESSION.AUTO_INCREMENT_INCREMENT=100, @@SESSION.AUTO_INCREMENT_OFFSET=10; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 100 auto_increment_offset 10 @@ -370,7 +370,7 @@ c1 DROP TABLE t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 1 auto_increment_offset 1 @@ -385,7 +385,7 @@ SELECT * FROM t1; c1 1 SET @@SESSION.AUTO_INCREMENT_INCREMENT=100, @@SESSION.AUTO_INCREMENT_OFFSET=10; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 100 auto_increment_offset 10 @@ -419,7 +419,7 @@ c1 DROP TABLE t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 1 auto_increment_offset 1 @@ -434,7 +434,7 @@ c1 1 9223372036854775794 SET @@SESSION.AUTO_INCREMENT_INCREMENT=2, @@SESSION.AUTO_INCREMENT_OFFSET=10; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 2 auto_increment_offset 10 @@ -452,7 +452,7 @@ c1 DROP TABLE t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 1 auto_increment_offset 1 @@ -467,7 +467,7 @@ c1 1 18446744073709551603 SET @@SESSION.AUTO_INCREMENT_INCREMENT=2, @@SESSION.AUTO_INCREMENT_OFFSET=10; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 2 auto_increment_offset 10 @@ -485,7 +485,7 @@ c1 DROP TABLE t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 1 auto_increment_offset 1 @@ -500,7 +500,7 @@ c1 1 18446744073709551603 SET @@SESSION.AUTO_INCREMENT_INCREMENT=5, @@SESSION.AUTO_INCREMENT_OFFSET=7; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 5 auto_increment_offset 7 @@ -514,7 +514,7 @@ c1 DROP TABLE t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 1 auto_increment_offset 1 @@ -533,7 +533,7 @@ c1 -9223372036854775806 1 SET @@SESSION.AUTO_INCREMENT_INCREMENT=3, @@SESSION.AUTO_INCREMENT_OFFSET=3; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 3 auto_increment_offset 3 @@ -550,7 +550,7 @@ c1 DROP TABLE t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 1 auto_increment_offset 1 @@ -568,7 +568,7 @@ SET @@SESSION.AUTO_INCREMENT_INCREMENT=1 Warnings: Warning 1292 Truncated incorrect auto_increment_increment value: '1152921504606846976' Warning 1292 Truncated incorrect auto_increment_offset value: '1152921504606846976' -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 65535 auto_increment_offset 65535 @@ -581,7 +581,7 @@ c1 DROP TABLE t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 1 auto_increment_offset 1 @@ -875,11 +875,11 @@ ALTER TABLE t1 CHANGE c1 d1 INT NOT NULL SELECT * FROM t1; d1 1 -3 +2 SELECT * FROM t1; d1 1 -3 +2 INSERT INTO t1 VALUES(null); Got one of the listed errors ALTER TABLE t1 AUTO_INCREMENT = 3; @@ -888,16 +888,16 @@ Table Create Table t1 CREATE TABLE `t1` ( `d1` int(11) NOT NULL AUTO_INCREMENT, PRIMARY KEY (`d1`) -) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1 +) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1 INSERT INTO t1 VALUES(null); SELECT * FROM t1; d1 1 +2 3 -4 DROP TABLE t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; Variable_name Value auto_increment_increment 1 auto_increment_offset 1 @@ -1126,3 +1126,61 @@ SELECT * FROM T1; c1 c2 10 0 DROP TABLE T1; +CREATE TABLE T1(C1 DOUBLE AUTO_INCREMENT KEY, C2 CHAR(10)) ENGINE=InnoDB; +INSERT INTO T1(C1, C2) VALUES (1, 'innodb'), (3, 'innodb'); +INSERT INTO T1(C2) VALUES ('innodb'); +SHOW CREATE TABLE T1; +Table Create Table +T1 CREATE TABLE `T1` ( + `C1` double NOT NULL AUTO_INCREMENT, + `C2` char(10) DEFAULT NULL, + PRIMARY KEY (`C1`) +) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=latin1 +DROP TABLE T1; +CREATE TABLE T1(C1 FLOAT AUTO_INCREMENT KEY, C2 CHAR(10)) ENGINE=InnoDB; +INSERT INTO T1(C1, C2) VALUES (1, 'innodb'), (3, 'innodb'); +INSERT INTO T1(C2) VALUES ('innodb'); +SHOW CREATE TABLE T1; +Table Create Table +T1 CREATE TABLE `T1` ( + `C1` float NOT NULL AUTO_INCREMENT, + `C2` char(10) DEFAULT NULL, + PRIMARY KEY (`C1`) +) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=latin1 +DROP TABLE T1; +CREATE TABLE t1 (c1 INT AUTO_INCREMENT PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 SET c1 = 1; +SHOW CREATE TABLE t1; +Table Create Table +t1 CREATE TABLE `t1` ( + `c1` int(11) NOT NULL AUTO_INCREMENT, + PRIMARY KEY (`c1`) +) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=latin1 +INSERT INTO t1 SET c1 = 2; +INSERT INTO t1 SET c1 = -1; +SELECT * FROM t1; +c1 +-1 +1 +2 +INSERT INTO t1 SET c1 = -1; +Got one of the listed errors +SHOW CREATE TABLE t1; +Table Create Table +t1 CREATE TABLE `t1` ( + `c1` int(11) NOT NULL AUTO_INCREMENT, + PRIMARY KEY (`c1`) +) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1 +REPLACE INTO t1 VALUES (-1); +SELECT * FROM t1; +c1 +-1 +1 +2 +SHOW CREATE TABLE t1; +Table Create Table +t1 CREATE TABLE `t1` ( + `c1` int(11) NOT NULL AUTO_INCREMENT, + PRIMARY KEY (`c1`) +) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1 +DROP TABLE t1; === added file 'mysql-test/r/innodb-consistent.result' --- a/mysql-test/r/innodb-consistent.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/r/innodb-consistent.result 2010-01-15 15:58:25 +0000 @@ -0,0 +1,35 @@ +drop table if exists t1; +set session transaction isolation level read committed; +create table t1(a int not null) engine=innodb DEFAULT CHARSET=latin1; +create table t2 like t1; +insert into t2 values (1),(2),(3),(4),(5),(6),(7); +set autocommit=0; +begin; +replace into t1 select * from t2; +set session transaction isolation level read committed; +set autocommit=0; +delete from t2 where a=5; +commit; +delete from t2; +commit; +commit; +begin; +insert into t1 select * from t2; +set session transaction isolation level read committed; +set autocommit=0; +delete from t2 where a=5; +commit; +delete from t2; +commit; +commit; +select * from t1; +a +1 +2 +3 +4 +5 +6 +7 +drop table t1; +drop table t2; === modified file 'mysql-test/r/innodb-index.result' --- a/mysql-test/r/innodb-index.result 2009-11-30 21:37:27 +0000 +++ b/mysql-test/r/innodb-index.result 2010-01-15 15:58:25 +0000 @@ -1,4 +1,3 @@ -SET @save_innodb_file_format_check=@@global.innodb_file_format_check; create table t1(a int not null, b int, c char(10) not null, d varchar(20)) engine = innodb; insert into t1 values (5,5,'oo','oo'),(4,4,'tr','tr'),(3,4,'ad','ad'),(2,3,'ak','ak'); commit; @@ -629,7 +628,7 @@ drop table t1; create table t1(a int not null, b int, c char(10), d varchar(20), primary key (a)) engine = innodb; insert into t1 values (1,1,'ab','ab'),(2,2,'ac','ac'),(3,3,'ac','ac'),(4,4,'afe','afe'),(5,4,'affe','affe'); alter table t1 add unique index (b), add unique index (c), add unique index (d); -ERROR 23000: Duplicate entry '4' for key 'b' +ERROR 23000: Duplicate entry 'ac' for key 'c' alter table t1 add unique index (c), add unique index (b), add index (d); ERROR 23000: Duplicate entry 'ac' for key 'c' show create table t1; @@ -970,6 +969,7 @@ create index t1u on t1 (u(1)); drop table t1; set global innodb_file_per_table=0; set global innodb_file_format=Antelope; +set global innodb_file_format_check=Antelope; SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0; SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0; CREATE TABLE t1( @@ -1171,4 +1171,3 @@ a b 3 a 3 b DROP TABLE t1; -SET GLOBAL innodb_file_format_check=@save_innodb_file_format_check; === modified file 'mysql-test/r/innodb-timeout.result' --- a/mysql-test/r/innodb-timeout.result 2009-06-09 13:19:13 +0000 +++ b/mysql-test/r/innodb-timeout.result 2010-01-18 12:56:10 +0000 @@ -23,6 +23,7 @@ select @@innodb_lock_wait_timeout; create table t1(a int primary key)engine=innodb; begin; insert into t1 values(1),(2),(3); +set innodb_lock_wait_timeout=5; select * from t1 for update; commit; a @@ -31,8 +32,15 @@ a 3 begin; insert into t1 values(4); +set innodb_lock_wait_timeout=2; +set @a= current_timestamp(); select * from t1 for update; -commit; ERROR HY000: Lock wait timeout exceeded; try restarting transaction +set @b= current_timestamp(); +set @c= timestampdiff(SECOND, @a, @b); +select if(@c >= 1 and @c <= 10, 'OK', concat("NOT OK, time passed=", @c)); +if(@c >= 1 and @c <= 10, 'OK', concat("NOT OK, time passed=", @c)) +OK +commit; drop table t1; set global innodb_lock_wait_timeout=50; === modified file 'mysql-test/r/innodb-zip.result' --- a/mysql-test/r/innodb-zip.result 2009-06-09 15:08:46 +0000 +++ b/mysql-test/r/innodb-zip.result 2010-01-15 15:58:25 +0000 @@ -196,15 +196,15 @@ drop table t1; set innodb_strict_mode = on; create table t1 (id int primary key) engine = innodb key_block_size = 0; ERROR HY000: Can't create table 'test.t1' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: invalid KEY_BLOCK_SIZE = 0. Valid values are [1, 2, 4, 8, 16] +Warning 1478 InnoDB: invalid KEY_BLOCK_SIZE = 0. Valid values are [1, 2, 4, 8, 16] Error 1005 Can't create table 'test.t1' (errno: 1478) create table t2 (id int primary key) engine = innodb key_block_size = 9; ERROR HY000: Can't create table 'test.t2' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: invalid KEY_BLOCK_SIZE = 9. Valid values are [1, 2, 4, 8, 16] +Warning 1478 InnoDB: invalid KEY_BLOCK_SIZE = 9. Valid values are [1, 2, 4, 8, 16] Error 1005 Can't create table 'test.t2' (errno: 1478) create table t3 (id int primary key) engine = innodb key_block_size = 1; create table t4 (id int primary key) engine = innodb key_block_size = 2; @@ -233,30 +233,30 @@ key_block_size = 8 row_format = compress create table t2 (id int primary key) engine = innodb key_block_size = 8 row_format = redundant; ERROR HY000: Can't create table 'test.t2' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: cannot specify ROW_FORMAT = REDUNDANT with KEY_BLOCK_SIZE. +Warning 1478 InnoDB: cannot specify ROW_FORMAT = REDUNDANT with KEY_BLOCK_SIZE. Error 1005 Can't create table 'test.t2' (errno: 1478) create table t3 (id int primary key) engine = innodb key_block_size = 8 row_format = compact; ERROR HY000: Can't create table 'test.t3' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: cannot specify ROW_FORMAT = COMPACT with KEY_BLOCK_SIZE. +Warning 1478 InnoDB: cannot specify ROW_FORMAT = COMPACT with KEY_BLOCK_SIZE. Error 1005 Can't create table 'test.t3' (errno: 1478) create table t4 (id int primary key) engine = innodb key_block_size = 8 row_format = dynamic; ERROR HY000: Can't create table 'test.t4' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: cannot specify ROW_FORMAT = DYNAMIC with KEY_BLOCK_SIZE. +Warning 1478 InnoDB: cannot specify ROW_FORMAT = DYNAMIC with KEY_BLOCK_SIZE. Error 1005 Can't create table 'test.t4' (errno: 1478) create table t5 (id int primary key) engine = innodb key_block_size = 8 row_format = default; ERROR HY000: Can't create table 'test.t5' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: cannot specify ROW_FORMAT = COMPACT with KEY_BLOCK_SIZE. +Warning 1478 InnoDB: cannot specify ROW_FORMAT = COMPACT with KEY_BLOCK_SIZE. Error 1005 Can't create table 'test.t5' (errno: 1478) SELECT table_schema, table_name, row_format FROM information_schema.tables WHERE engine='innodb'; @@ -266,26 +266,26 @@ drop table t1; create table t1 (id int primary key) engine = innodb key_block_size = 9 row_format = redundant; ERROR HY000: Can't create table 'test.t1' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: invalid KEY_BLOCK_SIZE = 9. Valid values are [1, 2, 4, 8, 16] -Error 1478 InnoDB: cannot specify ROW_FORMAT = REDUNDANT with KEY_BLOCK_SIZE. +Warning 1478 InnoDB: invalid KEY_BLOCK_SIZE = 9. Valid values are [1, 2, 4, 8, 16] +Warning 1478 InnoDB: cannot specify ROW_FORMAT = REDUNDANT with KEY_BLOCK_SIZE. Error 1005 Can't create table 'test.t1' (errno: 1478) create table t2 (id int primary key) engine = innodb key_block_size = 9 row_format = compact; ERROR HY000: Can't create table 'test.t2' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: invalid KEY_BLOCK_SIZE = 9. Valid values are [1, 2, 4, 8, 16] -Error 1478 InnoDB: cannot specify ROW_FORMAT = COMPACT with KEY_BLOCK_SIZE. +Warning 1478 InnoDB: invalid KEY_BLOCK_SIZE = 9. Valid values are [1, 2, 4, 8, 16] +Warning 1478 InnoDB: cannot specify ROW_FORMAT = COMPACT with KEY_BLOCK_SIZE. Error 1005 Can't create table 'test.t2' (errno: 1478) create table t2 (id int primary key) engine = innodb key_block_size = 9 row_format = dynamic; ERROR HY000: Can't create table 'test.t2' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: invalid KEY_BLOCK_SIZE = 9. Valid values are [1, 2, 4, 8, 16] -Error 1478 InnoDB: cannot specify ROW_FORMAT = DYNAMIC with KEY_BLOCK_SIZE. +Warning 1478 InnoDB: invalid KEY_BLOCK_SIZE = 9. Valid values are [1, 2, 4, 8, 16] +Warning 1478 InnoDB: cannot specify ROW_FORMAT = DYNAMIC with KEY_BLOCK_SIZE. Error 1005 Can't create table 'test.t2' (errno: 1478) SELECT table_schema, table_name, row_format FROM information_schema.tables WHERE engine='innodb'; @@ -293,45 +293,45 @@ table_schema table_name row_format set global innodb_file_per_table = off; create table t1 (id int primary key) engine = innodb key_block_size = 1; ERROR HY000: Can't create table 'test.t1' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_per_table. +Warning 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_per_table. Error 1005 Can't create table 'test.t1' (errno: 1478) create table t2 (id int primary key) engine = innodb key_block_size = 2; ERROR HY000: Can't create table 'test.t2' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_per_table. +Warning 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_per_table. Error 1005 Can't create table 'test.t2' (errno: 1478) create table t3 (id int primary key) engine = innodb key_block_size = 4; ERROR HY000: Can't create table 'test.t3' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_per_table. +Warning 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_per_table. Error 1005 Can't create table 'test.t3' (errno: 1478) create table t4 (id int primary key) engine = innodb key_block_size = 8; ERROR HY000: Can't create table 'test.t4' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_per_table. +Warning 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_per_table. Error 1005 Can't create table 'test.t4' (errno: 1478) create table t5 (id int primary key) engine = innodb key_block_size = 16; ERROR HY000: Can't create table 'test.t5' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_per_table. +Warning 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_per_table. Error 1005 Can't create table 'test.t5' (errno: 1478) create table t6 (id int primary key) engine = innodb row_format = compressed; ERROR HY000: Can't create table 'test.t6' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: ROW_FORMAT=COMPRESSED requires innodb_file_per_table. +Warning 1478 InnoDB: ROW_FORMAT=COMPRESSED requires innodb_file_per_table. Error 1005 Can't create table 'test.t6' (errno: 1478) create table t7 (id int primary key) engine = innodb row_format = dynamic; ERROR HY000: Can't create table 'test.t7' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: ROW_FORMAT=DYNAMIC requires innodb_file_per_table. +Warning 1478 InnoDB: ROW_FORMAT=DYNAMIC requires innodb_file_per_table. Error 1005 Can't create table 'test.t7' (errno: 1478) create table t8 (id int primary key) engine = innodb row_format = compact; create table t9 (id int primary key) engine = innodb row_format = redundant; @@ -345,45 +345,45 @@ set global innodb_file_per_table = on; set global innodb_file_format = `0`; create table t1 (id int primary key) engine = innodb key_block_size = 1; ERROR HY000: Can't create table 'test.t1' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_format > Antelope. +Warning 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_format > Antelope. Error 1005 Can't create table 'test.t1' (errno: 1478) create table t2 (id int primary key) engine = innodb key_block_size = 2; ERROR HY000: Can't create table 'test.t2' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_format > Antelope. +Warning 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_format > Antelope. Error 1005 Can't create table 'test.t2' (errno: 1478) create table t3 (id int primary key) engine = innodb key_block_size = 4; ERROR HY000: Can't create table 'test.t3' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_format > Antelope. +Warning 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_format > Antelope. Error 1005 Can't create table 'test.t3' (errno: 1478) create table t4 (id int primary key) engine = innodb key_block_size = 8; ERROR HY000: Can't create table 'test.t4' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_format > Antelope. +Warning 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_format > Antelope. Error 1005 Can't create table 'test.t4' (errno: 1478) create table t5 (id int primary key) engine = innodb key_block_size = 16; ERROR HY000: Can't create table 'test.t5' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_format > Antelope. +Warning 1478 InnoDB: KEY_BLOCK_SIZE requires innodb_file_format > Antelope. Error 1005 Can't create table 'test.t5' (errno: 1478) create table t6 (id int primary key) engine = innodb row_format = compressed; ERROR HY000: Can't create table 'test.t6' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: ROW_FORMAT=COMPRESSED requires innodb_file_format > Antelope. +Warning 1478 InnoDB: ROW_FORMAT=COMPRESSED requires innodb_file_format > Antelope. Error 1005 Can't create table 'test.t6' (errno: 1478) create table t7 (id int primary key) engine = innodb row_format = dynamic; ERROR HY000: Can't create table 'test.t7' (errno: 1478) -show errors; +show warnings; Level Code Message -Error 1478 InnoDB: ROW_FORMAT=DYNAMIC requires innodb_file_format > Antelope. +Warning 1478 InnoDB: ROW_FORMAT=DYNAMIC requires innodb_file_format > Antelope. Error 1005 Can't create table 'test.t7' (errno: 1478) create table t8 (id int primary key) engine = innodb row_format = compact; create table t9 (id int primary key) engine = innodb row_format = redundant; === modified file 'mysql-test/r/innodb.result' --- a/mysql-test/r/innodb.result 2009-12-27 13:54:41 +0000 +++ b/mysql-test/r/innodb.result 2010-01-15 15:58:25 +0000 @@ -3090,7 +3090,7 @@ ERROR HY000: Lock wait timeout exceeded; commit; drop table t1, t2, t3, t5, t6, t8, t9; CREATE TABLE t1 (DB_ROW_ID int) engine=innodb; -ERROR HY000: Can't create table 'test.t1' (errno: -1) +ERROR 42000: Incorrect column name 'DB_ROW_ID' CREATE TABLE t1 ( a BIGINT(20) NOT NULL, PRIMARY KEY (a) === modified file 'mysql-test/r/innodb_bug36169.result' --- a/mysql-test/r/innodb_bug36169.result 2009-11-13 21:26:08 +0000 +++ b/mysql-test/r/innodb_bug36169.result 2010-01-15 15:58:25 +0000 @@ -1,4 +1,2 @@ -set @old_innodb_file_format=@@innodb_file_format; -set @old_innodb_file_per_table=@@innodb_file_per_table; SET GLOBAL innodb_file_format='Barracuda'; SET GLOBAL innodb_file_per_table=ON; === modified file 'mysql-test/r/innodb_bug44369.result' --- a/mysql-test/r/innodb_bug44369.result 2009-11-02 14:59:44 +0000 +++ b/mysql-test/r/innodb_bug44369.result 2010-01-15 15:58:25 +0000 @@ -1,14 +1,6 @@ create table bug44369 (DB_ROW_ID int) engine=innodb; -ERROR HY000: Can't create table 'test.bug44369' (errno: -1) +ERROR 42000: Incorrect column name 'DB_ROW_ID' create table bug44369 (db_row_id int) engine=innodb; -ERROR HY000: Can't create table 'test.bug44369' (errno: -1) -show warnings; -Level Code Message -Warning 1005 Error creating table 'test/bug44369' with column name 'db_row_id'. 'db_row_id' is a reserved name. Please try to re-create the table with a different column name. -Error 1005 Can't create table 'test.bug44369' (errno: -1) +ERROR 42000: Incorrect column name 'db_row_id' create table bug44369 (db_TRX_Id int) engine=innodb; -ERROR HY000: Can't create table 'test.bug44369' (errno: -1) -show warnings; -Level Code Message -Warning 1005 Error creating table 'test/bug44369' with column name 'db_TRX_Id'. 'db_TRX_Id' is a reserved name. Please try to re-create the table with a different column name. -Error 1005 Can't create table 'test.bug44369' (errno: -1) +ERROR 42000: Incorrect column name 'db_TRX_Id' === added file 'mysql-test/r/innodb_bug44571.result' --- a/mysql-test/r/innodb_bug44571.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/r/innodb_bug44571.result 2010-01-15 15:58:25 +0000 @@ -0,0 +1,7 @@ +CREATE TABLE bug44571 (foo INT) ENGINE=InnoDB; +ALTER TABLE bug44571 CHANGE foo bar INT; +ALTER TABLE bug44571 ADD INDEX bug44571b (foo); +ERROR 42000: Key column 'foo' doesn't exist in table +ALTER TABLE bug44571 ADD INDEX bug44571b (bar); +CREATE INDEX bug44571c ON bug44571 (bar); +DROP TABLE bug44571; === added file 'mysql-test/r/innodb_bug46676.result' --- a/mysql-test/r/innodb_bug46676.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/r/innodb_bug46676.result 2010-01-15 15:58:25 +0000 @@ -0,0 +1,9 @@ +SET foreign_key_checks=0; +CREATE TABLE t1 (id int, foreign key (id) references t2(id)) ENGINE=INNODB; +CREATE TABLE t2 (id int, foreign key (id) references t1(id)) ENGINE=INNODB; +SET foreign_key_checks=1; +SELECT COUNT(*) FROM information_schema.key_column_usage WHERE REFERENCED_TABLE_NAME in ('t1', 't2'); +COUNT(*) +2 +SET foreign_key_checks=0; +DROP TABLE t1, t2; === added file 'mysql-test/r/innodb_bug47167.result' --- a/mysql-test/r/innodb_bug47167.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/r/innodb_bug47167.result 2010-01-15 15:58:25 +0000 @@ -0,0 +1,24 @@ +set @old_innodb_file_format_check=@@innodb_file_format_check; +select @old_innodb_file_format_check; +@old_innodb_file_format_check +Antelope +set global innodb_file_format_check = Barracuda; +select @@innodb_file_format_check; +@@innodb_file_format_check +Barracuda +set global innodb_file_format_check = DEFAULT; +select @@innodb_file_format_check; +@@innodb_file_format_check +Barracuda +set global innodb_file_format_check = @old_innodb_file_format_check; +select @@innodb_file_format_check; +@@innodb_file_format_check +Antelope +set global innodb_file_format_check = cheetah; +ERROR HY000: Incorrect arguments to SET +set global innodb_file_format_check = Bear; +ERROR HY000: Incorrect arguments to SET +set global innodb_file_format_check = on; +ERROR HY000: Incorrect arguments to SET +set global innodb_file_format_check = off; +ERROR HY000: Incorrect arguments to SET === modified file 'mysql-test/r/innodb_file_format.result' --- a/mysql-test/r/innodb_file_format.result 2009-11-30 21:37:27 +0000 +++ b/mysql-test/r/innodb_file_format.result 2010-01-15 15:58:25 +0000 @@ -1,4 +1,3 @@ -set @old_innodb_file_format=@@innodb_file_format; call mtr.add_suppression("InnoDB: invalid innodb_file_format_check value"); select @@innodb_file_format; @@innodb_file_format @@ -32,8 +31,6 @@ select @@innodb_file_format_check; @@innodb_file_format_check Barracuda set global innodb_file_format_check=default; -Warnings: -Warning 1210 Ignoring SET innodb_file_format=on select @@innodb_file_format_check; @@innodb_file_format_check Barracuda @@ -44,5 +41,4 @@ ERROR HY000: Incorrect arguments to SET select @@innodb_file_format_check; @@innodb_file_format_check Barracuda -set global innodb_file_format=@old_innodb_file_format; -set global innodb_file_format_check=Antelope; +set global innodb_file_format_check=antelope; === modified file 'mysql-test/r/innodb_lock_wait_timeout_1.result' --- a/mysql-test/r/innodb_lock_wait_timeout_1.result 2009-11-03 17:45:52 +0000 +++ b/mysql-test/r/innodb_lock_wait_timeout_1.result 2009-11-12 11:43:33 +0000 @@ -48,6 +48,24 @@ commit; set autocommit=default; drop table t1; # +# Bug #37183 insert ignore into .. select ... hangs +# after deadlock was encountered +# +create table t1(id int primary key,v int)engine=innodb; +insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5),(6,6),(7,7); +create table t2 like t1; +begin; +update t1 set v=id*2 where id=1; +begin; +update t1 set v=id*2 where id=2; +update t1 set v=id*2 where id=2; +ERROR HY000: Lock wait timeout exceeded; try restarting transaction +insert ignore into t2 select * from t1 where id=1; +ERROR HY000: Lock wait timeout exceeded; try restarting transaction +rollback; +rollback; +drop table t1, t2; +# # Bug#41756 Strange error messages about locks from InnoDB # drop table if exists t1; === modified file 'mysql-test/r/innodb_mysql.result' --- a/mysql-test/r/innodb_mysql.result 2009-12-03 11:19:05 +0000 +++ b/mysql-test/r/innodb_mysql.result 2010-01-15 15:27:55 +0000 @@ -2251,4 +2251,26 @@ c >= '2009-10-09 00:00:00.001' AND c <= id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables DROP TABLE t1; +# +# Bug #46175: NULL read_view and consistent read assertion +# +CREATE TABLE t1(a CHAR(13),KEY(a)) ENGINE=innodb; +CREATE TABLE t2(b DATETIME,KEY(b)) ENGINE=innodb; +INSERT INTO t1 VALUES (),(); +INSERT INTO t2 VALUES (),(); +CREATE OR REPLACE VIEW v1 AS SELECT 1 FROM t2 +WHERE b =(SELECT a FROM t1 LIMIT 1); +CREATE PROCEDURE p1(num INT) +BEGIN +DECLARE i INT DEFAULT 0; +REPEAT +SHOW CREATE VIEW v1; +SET i:=i+1; +UNTIL i>num END REPEAT; +END| +# Should not crash +# Should not crash +DROP PROCEDURE p1; +DROP VIEW v1; +DROP TABLE t1,t2; End of 5.1 tests === modified file 'mysql-test/r/innodb_xtradb_bug317074.result' --- a/mysql-test/r/innodb_xtradb_bug317074.result 2009-10-28 07:52:34 +0000 +++ b/mysql-test/r/innodb_xtradb_bug317074.result 2010-01-15 15:58:25 +0000 @@ -1,6 +1,5 @@ SET @old_innodb_file_format=@@innodb_file_format; SET @old_innodb_file_per_table=@@innodb_file_per_table; -SET @old_innodb_file_format_check=@@innodb_file_format_check; SET GLOBAL innodb_file_format='Barracuda'; SET GLOBAL innodb_file_per_table=ON; DROP TABLE IF EXISTS `test1`; @@ -29,4 +28,4 @@ ALTER TABLE test1 ENGINE=MyISAM; DROP TABLE test1; SET GLOBAL innodb_file_format=@old_innodb_file_format; SET GLOBAL innodb_file_per_table=@old_innodb_file_per_table; -SET GLOBAL innodb_file_format_check=@old_innodb_file_format_check; +set global innodb_file_format_check=Antelope; === modified file 'mysql-test/r/mysql.result' --- a/mysql-test/r/mysql.result 2009-07-31 23:43:46 +0000 +++ b/mysql-test/r/mysql.result 2009-11-27 14:41:45 +0000 @@ -229,5 +229,4 @@ a: b </row> </resultset> drop table t1; - -End of tests +End of 5.0 tests === modified file 'mysql-test/r/mysqltest.result' --- a/mysql-test/r/mysqltest.result 2009-10-08 09:30:03 +0000 +++ b/mysql-test/r/mysqltest.result 2010-01-11 13:15:28 +0000 @@ -1,3 +1,4 @@ +SET GLOBAL max_connections = 1000; select 0 as "before_use_test" ; before_use_test 0 === modified file 'mysql-test/r/olap.result' --- a/mysql-test/r/olap.result 2009-10-30 15:59:06 +0000 +++ b/mysql-test/r/olap.result 2009-12-08 09:26:11 +0000 @@ -753,4 +753,16 @@ b 100 NULL DROP TABLE t1, t2; +# +# Bug #48475: DISTINCT is ignored with GROUP BY WITH ROLLUP +# and only const tables +CREATE TABLE t1 (a INT); +CREATE TABLE t2 (b INT); +INSERT INTO t1 VALUES (1); +INSERT INTO t2 VALUES (1); +SELECT DISTINCT b FROM t1, t2 GROUP BY a, b WITH ROLLUP; +b +1 +NULL +DROP TABLE t1, t2; End of 5.0 tests === modified file 'mysql-test/r/openssl_1.result' --- a/mysql-test/r/openssl_1.result 2009-05-25 13:00:18 +0000 +++ b/mysql-test/r/openssl_1.result 2010-01-29 10:42:31 +0000 @@ -3,8 +3,10 @@ create table t1(f1 int); insert into t1 values (5); grant select on test.* to ssl_user1@localhost require SSL; grant select on test.* to ssl_user2@localhost require cipher "DHE-RSA-AES256-SHA"; -grant select on test.* to ssl_user3@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com"; -grant select on test.* to ssl_user4@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com" ISSUER "/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB"; +grant select on test.* to ssl_user3@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; +grant select on test.* to ssl_user4@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org" ISSUER "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; grant select on test.* to ssl_user5@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "xxx"; flush privileges; connect(localhost,ssl_user5,,test,MASTER_PORT,MASTER_SOCKET); @@ -44,9 +46,9 @@ ERROR 42000: DELETE command denied to us drop user ssl_user1@localhost, ssl_user2@localhost, ssl_user3@localhost, ssl_user4@localhost, ssl_user5@localhost; drop table t1; -mysqltest: Could not open connection 'default': 2026 SSL connection error -mysqltest: Could not open connection 'default': 2026 SSL connection error -mysqltest: Could not open connection 'default': 2026 SSL connection error +mysqltest: Could not open connection 'default': 2026 SSL error: ASN: bad other signature confirmation +mysqltest: Could not open connection 'default': 2026 SSL error: ASN: bad other signature confirmation +mysqltest: Could not open connection 'default': 2026 SSL error: ASN: bad other signature confirmation SSL error: Unable to get private key from '' mysqltest: Could not open connection 'default': 2026 SSL connection error SSL error: Unable to get certificate from '' === modified file 'mysql-test/r/order_by.result' --- a/mysql-test/r/order_by.result 2009-10-14 14:30:39 +0000 +++ b/mysql-test/r/order_by.result 2009-11-10 08:58:43 +0000 @@ -1444,6 +1444,27 @@ FROM t3; 2 NULL DROP TABLE t1, t2, t3; +# +# Bug #42760: Select doesn't return desired results when we have null +# values +# +CREATE TABLE t1 ( +a INT, +c INT, +UNIQUE KEY a_c (a,c), +KEY (a)); +INSERT INTO t1 VALUES (1, 10), (2, NULL); +# Must use ref-or-null on the a_c index +EXPLAIN +SELECT 1 AS col FROM t1 WHERE a=2 AND (c=10 OR c IS NULL) ORDER BY c; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 ref_or_null a_c,a a_c 10 const,const 1 Using where +# Must return 1 row +SELECT 1 AS col FROM t1 WHERE a=2 AND (c=10 OR c IS NULL) ORDER BY c; +col +1 +DROP TABLE t1; +End of 5.0 tests CREATE TABLE t2 (a varchar(32), b int(11), c float, d double, UNIQUE KEY a (a,b,c), KEY b (b), KEY c (c)); CREATE TABLE t1 (a varchar(32), b char(3), UNIQUE KEY a (a,b), KEY b (b)); === modified file 'mysql-test/r/partition.result' --- a/mysql-test/r/partition.result 2009-12-03 11:19:05 +0000 +++ b/mysql-test/r/partition.result 2010-01-15 15:27:55 +0000 @@ -1,4 +1,10 @@ drop table if exists t1, t2; +CREATE TABLE t1 (a INT, b INT) +PARTITION BY LIST (a) +SUBPARTITION BY HASH (b) +(PARTITION p1 VALUES IN (1)); +ALTER TABLE t1 ADD COLUMN c INT; +DROP TABLE t1; CREATE TABLE t1 ( a int NOT NULL, b int NOT NULL); @@ -50,6 +56,13 @@ t1 CREATE TABLE `t1` ( PARTITION p3 VALUES LESS THAN (733969) ENGINE = MyISAM, PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = MyISAM) */ DROP TABLE t1; +create table t1 (a int NOT NULL, b varchar(5) NOT NULL) +default charset=utf8 +partition by list (a) +subpartition by key (b) +(partition p0 values in (1), +partition p1 values in (2)); +drop table t1; create table t1 (a int, b int, key(a)) partition by list (a) ( partition p0 values in (1), @@ -2045,10 +2058,15 @@ DROP TABLE t1; # # Bug #45807: crash accessing partitioned table and sql_mode # contains ONLY_FULL_GROUP_BY +# Bug#46923: select count(*) from partitioned table fails with +# ONLY_FULL_GROUP_BY # SET SESSION SQL_MODE='ONLY_FULL_GROUP_BY'; CREATE TABLE t1(id INT,KEY(id)) ENGINE=MYISAM PARTITION BY HASH(id) PARTITIONS 2; +SELECT COUNT(*) FROM t1; +COUNT(*) +0 DROP TABLE t1; SET SESSION SQL_MODE=DEFAULT; # === modified file 'mysql-test/r/query_cache.result' --- a/mysql-test/r/query_cache.result 2009-12-03 11:19:05 +0000 +++ b/mysql-test/r/query_cache.result 2010-01-11 13:15:28 +0000 @@ -1302,6 +1302,15 @@ drop procedure f3; drop procedure f4; drop table t1; set GLOBAL query_cache_size=0; +set GLOBAL query_cache_size=100000; +set SESSION query_cache_size=10000; +ERROR HY000: Variable 'query_cache_size' is a GLOBAL variable and should be set with SET GLOBAL +set global query_cache_limit=100; +set global query_cache_size=100; +set global query_cache_type=demand; +set GLOBAL query_cache_type=default; +set GLOBAL query_cache_limit=default; +set GLOBAL query_cache_size=default; End of 4.1 tests SET GLOBAL query_cache_size=102400; create table t1(a int); @@ -1707,6 +1716,95 @@ Variable_name Value Qcache_hits 2 DROP TABLE t1; SET GLOBAL query_cache_size= default; +#------------------------------------------------------------------------ +# Tests for Bug#6760 and Bug#12689 +SET @row_count = 4; +SET @sleep_time_per_result_row = 1; +SET @max_acceptable_delay = 2; +SET @@global.query_cache_size = 1024 * 64; +DROP TEMPORARY TABLE IF EXISTS t_history; +DROP TABLE IF EXISTS t1; +CREATE TEMPORARY TABLE t_history (attempt SMALLINT, +start_ts DATETIME, end_ts DATETIME, +start_cached INTEGER, end_cached INTEGER); +CREATE TABLE t1 (f1 BIGINT); +INSERT INTO t_history +SET attempt = 4 - 4 + 1, start_ts = NOW(), +start_cached = 0; +SELECT *, SLEEP(@sleep_time_per_result_row) FROM t1; +f1 SLEEP(@sleep_time_per_result_row) +1 0 +1 0 +1 0 +1 0 +UPDATE t_history SET end_ts = NOW() +WHERE attempt = 4 - 4 + 1; +UPDATE t_history SET end_cached = 0 +WHERE attempt = 4 - 4 + 1; +INSERT INTO t_history +SET attempt = 4 - 3 + 1, start_ts = NOW(), +start_cached = 0; +SELECT *, SLEEP(@sleep_time_per_result_row) FROM t1; +f1 SLEEP(@sleep_time_per_result_row) +1 0 +1 0 +1 0 +1 0 +UPDATE t_history SET end_ts = NOW() +WHERE attempt = 4 - 3 + 1; +UPDATE t_history SET end_cached = 0 +WHERE attempt = 4 - 3 + 1; +INSERT INTO t_history +SET attempt = 4 - 2 + 1, start_ts = NOW(), +start_cached = 0; +SELECT *, SLEEP(@sleep_time_per_result_row) FROM t1; +f1 SLEEP(@sleep_time_per_result_row) +1 0 +1 0 +1 0 +1 0 +UPDATE t_history SET end_ts = NOW() +WHERE attempt = 4 - 2 + 1; +UPDATE t_history SET end_cached = 0 +WHERE attempt = 4 - 2 + 1; +INSERT INTO t_history +SET attempt = 4 - 1 + 1, start_ts = NOW(), +start_cached = 0; +SELECT *, SLEEP(@sleep_time_per_result_row) FROM t1; +f1 SLEEP(@sleep_time_per_result_row) +1 0 +1 0 +1 0 +1 0 +UPDATE t_history SET end_ts = NOW() +WHERE attempt = 4 - 1 + 1; +UPDATE t_history SET end_cached = 0 +WHERE attempt = 4 - 1 + 1; +# Test 1: Does the query with SLEEP need a reasonable time? +SELECT COUNT(*) >= 4 - 1 INTO @aux1 FROM t_history +WHERE TIMEDIFF(end_ts,start_ts) - @sleep_time_per_result_row * @row_count +BETWEEN 0 AND @max_acceptable_delay; +SELECT @aux1 AS "Expect 1"; +Expect 1 +1 +# Test 2: Does the query with SLEEP need a reasonable time even in case +# of the non first execution? +SELECT COUNT(*) >= 4 - 1 - 1 INTO @aux2 FROM t_history +WHERE TIMEDIFF(end_ts,start_ts) - @sleep_time_per_result_row * @row_count +BETWEEN 0 AND @max_acceptable_delay +AND attempt > 1; +SELECT @aux2 AS "Expect 1"; +Expect 1 +1 +# Test 3: The query with SLEEP must be not cached. +SELECT COUNT(*) = 4 INTO @aux3 FROM t_history +WHERE end_cached = start_cached; +SELECT @aux3 AS "Expect 1"; +Expect 1 +1 +DROP TABLE t1; +DROP TEMPORARY TABLE t_history; +SET @@global.query_cache_size = default; End of 5.0 tests SET GLOBAL query_cache_size=1024*1024*512; CREATE TABLE t1 (a ENUM('rainbow')); === modified file 'mysql-test/r/query_cache_notembedded.result' --- a/mysql-test/r/query_cache_notembedded.result 2009-02-13 19:32:24 +0000 +++ b/mysql-test/r/query_cache_notembedded.result 2010-01-11 13:15:28 +0000 @@ -382,3 +382,55 @@ set GLOBAL query_cache_type=default; set GLOBAL query_cache_limit=default; set GLOBAL query_cache_min_res_unit=default; set GLOBAL query_cache_size=default; +drop table if exists t1| +create table t1 ( +id char(16) not null default '', +data int not null +)| +drop procedure if exists bug3583| +drop procedure if exists bug3583| +create procedure bug3583() +begin +declare c int; +select * from t1; +select count(*) into c from t1; +select c; +end| +insert into t1 values ("x", 3), ("y", 5)| +set @x = @@query_cache_size| +set global query_cache_size = 10*1024*1024| +flush status| +flush query cache| +show status like 'Qcache_hits'| +Variable_name Value +Qcache_hits 0 +call bug3583()| +id data +x 3 +y 5 +c +2 +show status like 'Qcache_hits'| +Variable_name Value +Qcache_hits 0 +call bug3583()| +id data +x 3 +y 5 +c +2 +call bug3583()| +id data +x 3 +y 5 +c +2 +show status like 'Qcache_hits'| +Variable_name Value +Qcache_hits 2 +set global query_cache_size = @x| +flush status| +flush query cache| +delete from t1| +drop procedure bug3583| +drop table t1| === modified file 'mysql-test/r/range.result' --- a/mysql-test/r/range.result 2009-11-02 12:24:07 +0000 +++ b/mysql-test/r/range.result 2009-12-08 09:26:11 +0000 @@ -1603,4 +1603,54 @@ SELECT str_to_date('', '%Y-%m-%d'); str_to_date('', '%Y-%m-%d') 0000-00-00 DROP TABLE t1, t2; +# +# Bug#48459: valgrind errors with query using 'Range checked for each +# record' +# +CREATE TABLE t1 ( +a INT, +b CHAR(2), +c INT, +d INT, +KEY ( c ), +KEY ( d, a, b ( 2 ) ), +KEY ( b ( 1 ) ) +); +INSERT INTO t1 VALUES ( NULL, 'a', 1, 2 ), ( NULL, 'a', 1, 2 ), +( 1, 'a', 1, 2 ), ( 1, 'a', 1, 2 ); +CREATE TABLE t2 ( +a INT, +c INT, +e INT, +KEY ( e ) +); +INSERT INTO t2 VALUES ( 1, 1, NULL ), ( 1, 1, NULL ); +# Should not give Valgrind warnings +SELECT 1 +FROM t1, t2 +WHERE t1.d <> '1' AND t1.b > '1' +AND t1.a = t2.a AND t1.c = t2.c; +1 +1 +1 +1 +1 +DROP TABLE t1, t2; +# +# Bug #48665: sql-bench's insert test fails due to wrong result +# +CREATE TABLE t1 (a INT, b INT, PRIMARY KEY (a)); +INSERT INTO t1 VALUES (0,0), (1,1); +EXPLAIN +SELECT * FROM t1 FORCE INDEX (PRIMARY) +WHERE (a>=1 AND a<=2) OR (a>=4 AND a<=5) OR (a>=0 AND a <=10); +id select_type table type possible_keys key key_len ref rows Extra +@ @ @ range @ @ @ @ @ @ +# Should return 2 rows +SELECT * FROM t1 FORCE INDEX (PRIMARY) +WHERE (a>=1 AND a<=2) OR (a>=4 AND a<=5) OR (a>=0 AND a <=10); +a b +0 0 +1 1 +DROP TABLE t1; End of 5.1 tests === modified file 'mysql-test/r/select.result' --- a/mysql-test/r/select.result 2009-12-03 11:19:05 +0000 +++ b/mysql-test/r/select.result 2010-01-15 15:27:55 +0000 @@ -4426,6 +4426,20 @@ ROW(a,a) <=> ROW((SELECT 1 FROM t1 WHERE INTO @var0; ERROR 21000: Subquery returns more than 1 row DROP TABLE t1; +# +# Bug #48458: simple query tries to allocate enormous amount of +# memory +# +CREATE TABLE t1(a INT NOT NULL, b YEAR); +INSERT INTO t1 VALUES (); +Warnings: +Warning 1364 Field 'a' doesn't have a default value +CREATE TABLE t2(c INT); +# Should not err out because of out-of-memory +SELECT 1 FROM t2 JOIN t1 ON 1=1 +WHERE a != '1' AND NOT a >= b OR NOT ROW(b,a )<> ROW(a,a); +1 +DROP TABLE t1,t2; End of 5.0 tests create table t1(a INT, KEY (a)); INSERT INTO t1 VALUES (1),(2),(3),(4),(5); @@ -4576,4 +4590,47 @@ field2 15:13:38 drop table A,AA,B,BB; #end of test for bug#45266 +# +# BUG#48052: Valgrind warning - uninitialized value in init_read_record() +# +CREATE TABLE t1 ( +pk int(11) NOT NULL, +i int(11) DEFAULT NULL, +v varchar(1) DEFAULT NULL, +PRIMARY KEY (pk) +); +INSERT INTO t1 VALUES (2,7,'m'); +INSERT INTO t1 VALUES (3,9,'m'); +SELECT v +FROM t1 +WHERE NOT pk > 0 +HAVING v <= 't' +ORDER BY pk; +v +DROP TABLE t1; +# +# Bug#49489 Uninitialized cache led to a wrong result. +# +CREATE TABLE t1(c1 DOUBLE(5,4)); +INSERT INTO t1 VALUES (9.1234); +SELECT * FROM t1 WHERE c1 < 9.12345; +c1 +9.1234 +DROP TABLE t1; +# End of test for bug#49489. +# +# Bug #49517: Inconsistent behavior while using +# NULLable BIGINT and INT columns in comparison +# +CREATE TABLE t1(a BIGINT UNSIGNED NOT NULL, b BIGINT NULL, c INT NULL); +INSERT INTO t1 VALUES(105, NULL, NULL); +SELECT * FROM t1 WHERE b < 102; +a b c +SELECT * FROM t1 WHERE c < 102; +a b c +SELECT * FROM t1 WHERE 102 < b; +a b c +SELECT * FROM t1 WHERE 102 < c; +a b c +DROP TABLE t1; End of 5.1 tests === modified file 'mysql-test/r/show_check.result' --- a/mysql-test/r/show_check.result 2009-03-06 14:56:17 +0000 +++ b/mysql-test/r/show_check.result 2010-01-27 21:53:08 +0000 @@ -252,13 +252,11 @@ drop table t1; flush tables; show open tables; Database Table In_use Name_locked -mysql general_log 0 0 create table t1(n int); insert into t1 values (1); show open tables; Database Table In_use Name_locked test t1 0 0 -mysql general_log 0 0 drop table t1; create table t1 (a int not null, b VARCHAR(10), INDEX (b) ) AVG_ROW_LENGTH=10 CHECKSUM=1 COMMENT="test" ENGINE=MYISAM MIN_ROWS=10 MAX_ROWS=100 PACK_KEYS=1 DELAY_KEY_WRITE=1 ROW_FORMAT=fixed; show create table t1; @@ -1454,4 +1452,10 @@ GRANT PROCESS ON *.* TO test_u@localhost SHOW ENGINE MYISAM MUTEX; SHOW ENGINE MYISAM STATUS; DROP USER test_u@localhost; +# +# Bug #48985: show create table crashes if previous access to the table +# was killed +# +SHOW CREATE TABLE non_existent; +ERROR 70100: Query execution was interrupted End of 5.1 tests === modified file 'mysql-test/r/sp-destruct.result' --- a/mysql-test/r/sp-destruct.result 2008-04-08 14:51:26 +0000 +++ b/mysql-test/r/sp-destruct.result 2009-11-21 11:18:21 +0000 @@ -1,3 +1,4 @@ +call mtr.add_suppression("Column count of mysql.proc is wrong. Expected 20, found 19. The table is probably corrupted"); use test; drop procedure if exists bug14233; drop function if exists bug14233; @@ -11,11 +12,13 @@ create table t1 (id int); create trigger t1_ai after insert on t1 for each row call bug14233(); alter table mysql.proc drop type; call bug14233(); -ERROR HY000: Failed to load routine test.bug14233. The table mysql.proc is missing, corrupt, or contains bad data (internal code -5) +ERROR HY000: Column count of mysql.proc is wrong. Expected 20, found 19. The table is probably corrupted create view v1 as select bug14233_f(); -ERROR HY000: Failed to load routine test.bug14233_f. The table mysql.proc is missing, corrupt, or contains bad data (internal code -5) +ERROR HY000: Column count of mysql.proc is wrong. Expected 20, found 19. The table is probably corrupted insert into t1 values (0); -ERROR HY000: Failed to load routine test.bug14233. The table mysql.proc is missing, corrupt, or contains bad data (internal code -5) +ERROR HY000: Column count of mysql.proc is wrong. Expected 20, found 19. The table is probably corrupted +show procedure status; +ERROR HY000: Column count of mysql.proc is wrong. Expected 20, found 19. The table is probably corrupted flush table mysql.proc; call bug14233(); ERROR HY000: Incorrect information in file: './mysql/proc.frm' @@ -88,3 +91,28 @@ show procedure status where db=DATABASE( Db Name Type Definer Modified Created Security_type Comment character_set_client collation_connection Database Collation show function status where db=DATABASE(); Db Name Type Definer Modified Created Security_type Comment character_set_client collation_connection Database Collation +DROP TABLE IF EXISTS proc_backup; +DROP PROCEDURE IF EXISTS p1; +# Backup the proc table +RENAME TABLE mysql.proc TO proc_backup; +CREATE TABLE mysql.proc LIKE proc_backup; +FLUSH TABLE mysql.proc; +# Test with a valid table. +CREATE PROCEDURE p1() +SET @foo = 10; +CALL p1(); +SHOW PROCEDURE STATUS; +Db Name Type Definer Modified Created Security_type Comment character_set_client collation_connection Database Collation +test p1 PROCEDURE root@localhost 0000-00-00 00:00:00 0000-00-00 00:00:00 DEFINER latin1 latin1_swedish_ci latin1_swedish_ci +# Modify a field of the table. +ALTER TABLE mysql.proc MODIFY comment CHAR (32); +CREATE PROCEDURE p2() +SET @foo = 10; +ERROR HY000: Cannot load from mysql.proc. The table is probably corrupted +# Procedure loaded from the cache +CALL p1(); +SHOW PROCEDURE STATUS; +ERROR HY000: Cannot load from mysql.proc. The table is probably corrupted +DROP TABLE mysql.proc; +RENAME TABLE proc_backup TO mysql.proc; +FLUSH TABLE mysql.proc; === modified file 'mysql-test/r/sp-security.result' --- a/mysql-test/r/sp-security.result 2009-03-06 14:56:17 +0000 +++ b/mysql-test/r/sp-security.result 2009-11-27 16:10:28 +0000 @@ -510,4 +510,60 @@ DROP USER mysqltest_u1@localhost; DROP PROCEDURE p_suid; DROP FUNCTION f_suid; DROP TABLE t1; +# +# Bug #48872 : Privileges for stored functions ignored if function name +# is mixed case +# +CREATE DATABASE B48872; +USE B48872; +CREATE TABLE `TestTab` (id INT); +INSERT INTO `TestTab` VALUES (1),(2); +CREATE FUNCTION `f_Test`() RETURNS INT RETURN 123; +CREATE FUNCTION `f_Test_denied`() RETURNS INT RETURN 123; +CREATE USER 'tester'; +CREATE USER 'Tester'; +GRANT SELECT ON TABLE `TestTab` TO 'tester'; +GRANT EXECUTE ON FUNCTION `f_Test` TO 'tester'; +GRANT EXECUTE ON FUNCTION `f_Test_denied` TO 'Tester'; +SELECT f_Test(); +f_Test() +123 +SELECT * FROM TestTab; +id +1 +2 +SELECT * FROM TestTab; +id +1 +2 +SELECT `f_Test`(); +`f_Test`() +123 +SELECT `F_TEST`(); +`F_TEST`() +123 +SELECT f_Test(); +f_Test() +123 +SELECT F_TEST(); +F_TEST() +123 +SELECT * FROM TestTab; +SELECT `f_Test`(); +SELECT `F_TEST`(); +SELECT f_Test(); +SELECT F_TEST(); +SELECT `f_Test_denied`(); +`f_Test_denied`() +123 +SELECT `F_TEST_DENIED`(); +`F_TEST_DENIED`() +123 +DROP TABLE `TestTab`; +DROP FUNCTION `f_Test`; +DROP FUNCTION `f_Test_denied`; +USE test; +DROP USER 'tester'; +DROP USER 'Tester'; +DROP DATABASE B48872; End of 5.0 tests. === modified file 'mysql-test/r/sp.result' --- a/mysql-test/r/sp.result 2009-10-23 13:54:58 +0000 +++ b/mysql-test/r/sp.result 2009-11-13 01:03:26 +0000 @@ -6979,6 +6979,64 @@ CALL p1; ERROR 42S22: Unknown column 'A.b' in 'IN/ALL/ANY subquery' DROP PROCEDURE p1; DROP TABLE t1, t2; +# +# Bug#47627: SET @@{global.session}.local_variable in stored routine causes crash +# Bug#48626: Crash or lost connection using SET for declared variables with @@ +# +DROP PROCEDURE IF EXISTS p1; +DROP PROCEDURE IF EXISTS p2; +DROP PROCEDURE IF EXISTS p3; +CREATE PROCEDURE p1() +BEGIN +DECLARE v INT DEFAULT 0; +SET @@SESSION.v= 10; +END// +ERROR HY000: Unknown system variable 'v' +CREATE PROCEDURE p2() +BEGIN +DECLARE v INT DEFAULT 0; +SET v= 10; +END// +call p2()// +CREATE PROCEDURE p3() +BEGIN +DECLARE v INT DEFAULT 0; +SELECT @@SESSION.v; +END// +ERROR HY000: Unknown system variable 'v' +CREATE PROCEDURE p4() +BEGIN +DECLARE v INT DEFAULT 0; +SET @@GLOBAL.v= 10; +END// +ERROR HY000: Unknown system variable 'v' +CREATE PROCEDURE p5() +BEGIN +DECLARE init_connect INT DEFAULT 0; +SET init_connect= 10; +SET @@GLOBAL.init_connect= 'SELECT 1'; +SET @@SESSION.IDENTITY= 1; +SELECT @@SESSION.IDENTITY; +SELECT @@GLOBAL.init_connect; +SELECT init_connect; +END// +CREATE PROCEDURE p6() +BEGIN +DECLARE v INT DEFAULT 0; +SET @@v= 0; +END// +ERROR HY000: Unknown system variable 'v' +SET @old_init_connect= @@GLOBAL.init_connect; +CALL p5(); +@@SESSION.IDENTITY +1 +@@GLOBAL.init_connect +SELECT 1 +init_connect +10 +SET @@GLOBAL.init_connect= @old_init_connect; +DROP PROCEDURE p2; +DROP PROCEDURE p5; # ------------------------------------------------------------------ # -- End of 5.1 tests # ------------------------------------------------------------------ === modified file 'mysql-test/r/sp_notembedded.result' --- a/mysql-test/r/sp_notembedded.result 2009-10-13 18:21:42 +0000 +++ b/mysql-test/r/sp_notembedded.result 2010-01-11 13:15:28 +0000 @@ -25,58 +25,6 @@ call bug4902_2()| show warnings| Level Code Message drop procedure bug4902_2| -drop table if exists t1| -create table t1 ( -id char(16) not null default '', -data int not null -)| -drop procedure if exists bug3583| -drop procedure if exists bug3583| -create procedure bug3583() -begin -declare c int; -select * from t1; -select count(*) into c from t1; -select c; -end| -insert into t1 values ("x", 3), ("y", 5)| -set @x = @@query_cache_size| -set global query_cache_size = 10*1024*1024| -flush status| -flush query cache| -show status like 'Qcache_hits'| -Variable_name Value -Qcache_hits 0 -call bug3583()| -id data -x 3 -y 5 -c -2 -show status like 'Qcache_hits'| -Variable_name Value -Qcache_hits 0 -call bug3583()| -id data -x 3 -y 5 -c -2 -call bug3583()| -id data -x 3 -y 5 -c -2 -show status like 'Qcache_hits'| -Variable_name Value -Qcache_hits 2 -set global query_cache_size = @x| -flush status| -flush query cache| -delete from t1| -drop procedure bug3583| -drop table t1| drop procedure if exists bug6807| create procedure bug6807() begin === modified file 'mysql-test/r/trigger.result' --- a/mysql-test/r/trigger.result 2009-06-22 12:51:33 +0000 +++ b/mysql-test/r/trigger.result 2010-01-12 08:19:48 +0000 @@ -1448,33 +1448,6 @@ isave 1 2 drop table t1, t2, t3; -CREATE TABLE t1 (id INTEGER); -CREATE TABLE t2 (id INTEGER); -INSERT INTO t2 VALUES (1),(2); -CREATE TRIGGER t1_test AFTER INSERT ON t1 FOR EACH ROW -INSERT INTO t2 VALUES (new.id); -SELECT GET_LOCK('B26162',120); -GET_LOCK('B26162',120) -1 -SELECT 'rl_acquirer', GET_LOCK('B26162',120), id FROM t2 WHERE id = 1; -SET SESSION LOW_PRIORITY_UPDATES=1; -SET GLOBAL LOW_PRIORITY_UPDATES=1; -INSERT INTO t1 VALUES (5); -SELECT 'rl_contender', id FROM t2 WHERE id > 1; -SELECT RELEASE_LOCK('B26162'); -RELEASE_LOCK('B26162') -1 -rl_acquirer GET_LOCK('B26162',120) id -rl_acquirer 1 1 -SELECT RELEASE_LOCK('B26162'); -RELEASE_LOCK('B26162') -1 -rl_contender id -rl_contender 2 -DROP TRIGGER t1_test; -DROP TABLE t1,t2; -SET SESSION LOW_PRIORITY_UPDATES=DEFAULT; -SET GLOBAL LOW_PRIORITY_UPDATES=DEFAULT; Bug#28502 Triggers that update another innodb table will block on X lock unnecessarily === modified file 'mysql-test/r/trigger_notembedded.result' --- a/mysql-test/r/trigger_notembedded.result 2009-09-17 11:33:23 +0000 +++ b/mysql-test/r/trigger_notembedded.result 2010-01-12 08:19:48 +0000 @@ -445,6 +445,33 @@ DROP TABLE t2; DROP TABLE t1; DROP DATABASE mysqltest_db1; USE test; +CREATE TABLE t1 (id INTEGER); +CREATE TABLE t2 (id INTEGER); +INSERT INTO t2 VALUES (1),(2); +CREATE TRIGGER t1_test AFTER INSERT ON t1 FOR EACH ROW +INSERT INTO t2 VALUES (new.id); +SELECT GET_LOCK('B26162',120); +GET_LOCK('B26162',120) +1 +SELECT 'rl_acquirer', GET_LOCK('B26162',120), id FROM t2 WHERE id = 1; +SET SESSION LOW_PRIORITY_UPDATES=1; +SET GLOBAL LOW_PRIORITY_UPDATES=1; +INSERT INTO t1 VALUES (5); +SELECT 'rl_contender', id FROM t2 WHERE id > 1; +SELECT RELEASE_LOCK('B26162'); +RELEASE_LOCK('B26162') +1 +rl_acquirer GET_LOCK('B26162',120) id +rl_acquirer 1 1 +SELECT RELEASE_LOCK('B26162'); +RELEASE_LOCK('B26162') +1 +rl_contender id +rl_contender 2 +DROP TRIGGER t1_test; +DROP TABLE t1,t2; +SET SESSION LOW_PRIORITY_UPDATES=DEFAULT; +SET GLOBAL LOW_PRIORITY_UPDATES=DEFAULT; End of 5.0 tests. drop table if exists t1; create table t1 (i int); === modified file 'mysql-test/r/type_newdecimal.result' --- a/mysql-test/r/type_newdecimal.result 2009-11-02 11:21:39 +0000 +++ b/mysql-test/r/type_newdecimal.result 2009-12-08 09:26:11 +0000 @@ -1630,3 +1630,287 @@ SELECT my_col FROM t1; my_col 0.012345687012345687012345687012 DROP TABLE t1; +# +# Bug#45261: Crash, stored procedure + decimal +# +DROP TABLE IF EXISTS t1; +CREATE TABLE t1 SELECT +/* 81 */ 100000000000000000000000000000000000000000000000000000000000000000000000000000001 +AS c1; +Warnings: +Warning 1264 Out of range value for column 'c1' at row 1 +DESC t1; +Field Type Null Key Default Extra +c1 decimal(65,0) NO 0 +SELECT * FROM t1; +c1 +99999999999999999999999999999999999999999999999999999999999999999 +DROP TABLE t1; +CREATE TABLE t1 SELECT +/* 81 */ 100000000000000000000000000000000000000000000000000000000000000000000000000000001. +AS c1; +Warnings: +Warning 1264 Out of range value for column 'c1' at row 1 +DESC t1; +Field Type Null Key Default Extra +c1 decimal(65,0) NO 0 +SELECT * FROM t1; +c1 +99999999999999999999999999999999999999999999999999999999999999999 +DROP TABLE t1; +CREATE TABLE t1 SELECT +/* 81 */ 100000000000000000000000000000000000000000000000000000000000000000000000000000001.1 /* 1 */ +AS c1; +Warnings: +Warning 1264 Out of range value for column 'c1' at row 1 +DESC t1; +Field Type Null Key Default Extra +c1 decimal(65,0) NO 0 +SELECT * FROM t1; +c1 +99999999999999999999999999999999999999999999999999999999999999999 +DROP TABLE t1; +CREATE TABLE t1 SELECT +/* 82 */ 1000000000000000000000000000000000000000000000000000000000000000000000000000000001 +AS c1; +Warnings: +Error 1292 Truncated incorrect DECIMAL value: '' +DESC t1; +Field Type Null Key Default Extra +c1 decimal(65,0) NO 0 +SELECT * FROM t1; +c1 +99999999999999999999999999999999999999999999999999999999999999999 +DROP TABLE t1; +CREATE TABLE t1 SELECT +/* 40 */ 1000000000000000000000000000000000000001.1000000000000000000000000000000000000001 /* 40 */ +AS c1; +Warnings: +Warning 1264 Out of range value for column 'c1' at row 1 +DESC t1; +Field Type Null Key Default Extra +c1 decimal(65,30) NO 0.000000000000000000000000000000 +SELECT * FROM t1; +c1 +99999999999999999999999999999999999.999999999999999999999999999999 +DROP TABLE t1; +CREATE TABLE t1 SELECT +/* 1 */ 1.10000000000000000000000000000000000000000000000000000000000000000000000000000001 /* 80 */ +AS c1; +DESC t1; +Field Type Null Key Default Extra +c1 decimal(31,30) NO 0.000000000000000000000000000000 +SELECT * FROM t1; +c1 +1.100000000000000000000000000000 +DROP TABLE t1; +CREATE TABLE t1 SELECT +/* 1 */ 1.100000000000000000000000000000000000000000000000000000000000000000000000000000001 /* 81 */ +AS c1; +DESC t1; +Field Type Null Key Default Extra +c1 decimal(31,30) NO 0.000000000000000000000000000000 +SELECT * FROM t1; +c1 +1.100000000000000000000000000000 +DROP TABLE t1; +CREATE TABLE t1 SELECT +.100000000000000000000000000000000000000000000000000000000000000000000000000000001 /* 81 */ +AS c1; +Warnings: +Note 1265 Data truncated for column 'c1' at row 1 +DESC t1; +Field Type Null Key Default Extra +c1 decimal(30,30) NO 0.000000000000000000000000000000 +SELECT * FROM t1; +c1 +0.100000000000000000000000000000 +DROP TABLE t1; +CREATE TABLE t1 SELECT +/* 45 */ 123456789012345678901234567890123456789012345.123456789012345678901234567890123456789012345 /* 45 */ +AS c1; +Warnings: +Warning 1264 Out of range value for column 'c1' at row 1 +DESC t1; +Field Type Null Key Default Extra +c1 decimal(65,30) NO 0.000000000000000000000000000000 +SELECT * FROM t1; +c1 +99999999999999999999999999999999999.999999999999999999999999999999 +DROP TABLE t1; +CREATE TABLE t1 SELECT +/* 65 */ 12345678901234567890123456789012345678901234567890123456789012345.1 /* 1 */ +AS c1; +Warnings: +Warning 1264 Out of range value for column 'c1' at row 1 +DESC t1; +Field Type Null Key Default Extra +c1 decimal(65,1) NO 0.0 +SELECT * FROM t1; +c1 +9999999999999999999999999999999999999999999999999999999999999999.9 +DROP TABLE t1; +CREATE TABLE t1 SELECT +/* 66 */ 123456789012345678901234567890123456789012345678901234567890123456.1 /* 1 */ +AS c1; +Warnings: +Warning 1264 Out of range value for column 'c1' at row 1 +DESC t1; +Field Type Null Key Default Extra +c1 decimal(65,1) NO 0.0 +SELECT * FROM t1; +c1 +9999999999999999999999999999999999999999999999999999999999999999.9 +DROP TABLE t1; +CREATE TABLE t1 SELECT +.123456789012345678901234567890123456789012345678901234567890123456 /* 66 */ +AS c1; +Warnings: +Note 1265 Data truncated for column 'c1' at row 1 +DESC t1; +Field Type Null Key Default Extra +c1 decimal(30,30) NO 0.000000000000000000000000000000 +SELECT * FROM t1; +c1 +0.123456789012345678901234567890 +DROP TABLE t1; +CREATE TABLE t1 AS SELECT 123.1234567890123456789012345678901 /* 31 */ AS c1; +Warnings: +Note 1265 Data truncated for column 'c1' at row 1 +DESC t1; +Field Type Null Key Default Extra +c1 decimal(33,30) NO 0.000000000000000000000000000000 +SELECT * FROM t1; +c1 +123.123456789012345678901234567890 +DROP TABLE t1; +CREATE TABLE t1 SELECT 1.1 + CAST(1 AS DECIMAL(65,30)) AS c1; +DESC t1; +Field Type Null Key Default Extra +c1 decimal(65,30) NO 0.000000000000000000000000000000 +SELECT * FROM t1; +c1 +2.100000000000000000000000000000 +DROP TABLE t1; +# +# Test that the integer and decimal parts are properly calculated. +# +CREATE TABLE t1 (a DECIMAL(30,30)); +INSERT INTO t1 VALUES (0.1),(0.2),(0.3); +CREATE TABLE t2 SELECT MIN(a + 0.0000000000000000000000000000001) AS c1 FROM t1; +Warnings: +Note 1265 Data truncated for column 'c1' at row 3 +DESC t2; +Field Type Null Key Default Extra +c1 decimal(32,30) YES NULL +DROP TABLE t1,t2; +CREATE TABLE t1 (a DECIMAL(30,30)); +INSERT INTO t1 VALUES (0.1),(0.2),(0.3); +CREATE TABLE t2 SELECT IFNULL(a + 0.0000000000000000000000000000001, NULL) AS c1 FROM t1; +Warnings: +Note 1265 Data truncated for column 'c1' at row 1 +Note 1265 Data truncated for column 'c1' at row 2 +Note 1265 Data truncated for column 'c1' at row 3 +DESC t2; +Field Type Null Key Default Extra +c1 decimal(34,0) YES NULL +DROP TABLE t1,t2; +CREATE TABLE t1 (a DECIMAL(30,30)); +INSERT INTO t1 VALUES (0.1),(0.2),(0.3); +CREATE TABLE t2 SELECT CASE a WHEN 0.1 THEN 0.0000000000000000000000000000000000000000000000000000000000000000001 END AS c1 FROM t1; +Warnings: +Note 1265 Data truncated for column 'c1' at row 1 +DESC t2; +Field Type Null Key Default Extra +c1 decimal(65,30) YES NULL +DROP TABLE t1,t2; +# +# Test that variables get maximum precision. +# +SET @decimal= 1.1; +CREATE TABLE t1 SELECT @decimal AS c1; +DESC t1; +Field Type Null Key Default Extra +c1 decimal(65,30) YES NULL +SELECT * FROM t1; +c1 +1.100000000000000000000000000000 +DROP TABLE t1; +# +# Bug #45261 : Crash, stored procedure + decimal +# Original test by the reporter. +# +# should not crash +CREATE TABLE t1 +SELECT .123456789012345678901234567890123456789012345678901234567890123456 AS a; +Warnings: +Note 1265 Data truncated for column 'a' at row 1 +DROP TABLE t1; +CREATE PROCEDURE test_proc() +BEGIN +# The las non critical CUSER definition is: +# DECLARE mycursor CURSOR FOR SELECT 1 % +# .12345678912345678912345678912345678912345678912345678912345678912 AS my_col; +DECLARE mycursor CURSOR FOR +SELECT 1 % +.123456789123456789123456789123456789123456789123456789123456789123456789123456789 +AS my_col; +OPEN mycursor; +CLOSE mycursor; +END| +# should not crash +CALL test_proc(); +DROP PROCEDURE test_proc; +# +# Bug #48370 Absolutely wrong calculations with GROUP BY and +# decimal fields when using IF +# +CREATE TABLE currencies (id int, rate decimal(16,4), +PRIMARY KEY (id), KEY (rate)); +INSERT INTO currencies VALUES (11,0.7028); +INSERT INTO currencies VALUES (1,1); +CREATE TABLE payments ( +id int, +supplier_id int, +status int, +currency_id int, +vat decimal(7,4), +PRIMARY KEY (id), +KEY currency_id (currency_id), +KEY supplier_id (supplier_id) +); +INSERT INTO payments (id,status,vat,supplier_id,currency_id) VALUES +(3001,2,0.0000,344,11), (1,2,0.0000,1,1); +CREATE TABLE sub_tasks ( +id int, +currency_id int, +price decimal(16,4), +discount decimal(10,4), +payment_id int, +PRIMARY KEY (id), +KEY currency_id (currency_id), +KEY payment_id (payment_id) +) ; +INSERT INTO sub_tasks (id, price, discount, payment_id, currency_id) VALUES +(52, 12.60, 0, 3001, 11), (56, 14.58, 0, 3001, 11); +# should return 1 and the same values in col 2 and 3 +select STRAIGHT_JOIN +(1 + PAY.vat) AS mult, +SUM(ROUND((SUB.price - ROUND(ROUND(SUB.price, 2) * SUB.discount, 2)) * +CUR.rate / CUR.rate, 2) +) v_net_with_discount, +SUM(ROUND((SUB.price - ROUND(ROUND(SUB.price, 2) * SUB.discount, 1)) * +CUR.rate / CUR.rate , 2) +* (1 + PAY.vat) +) v_total +from +currencies CUR, payments PAY, sub_tasks SUB +where +SUB.payment_id = PAY.id and +PAY.currency_id = CUR.id and +PAY.id > 2 +group by PAY.id + 1; +mult v_net_with_discount v_total +1.0000 27.18 27.180000 +DROP TABLE currencies, payments, sub_tasks; +End of 5.1 tests === modified file 'mysql-test/r/type_year.result' --- a/mysql-test/r/type_year.result 2007-03-29 04:08:30 +0000 +++ b/mysql-test/r/type_year.result 2009-12-15 08:37:10 +0000 @@ -46,3 +46,267 @@ a 2001 drop table t1; End of 5.0 tests +# +# Bug #49480: WHERE using YEAR columns returns unexpected results +# +CREATE TABLE t2(yy YEAR(2), c2 CHAR(4)); +CREATE TABLE t4(yyyy YEAR(4), c4 CHAR(4)); +INSERT INTO t2 (c2) VALUES (NULL),(1970),(1999),(2000),(2001),(2069); +INSERT INTO t4 (c4) SELECT c2 FROM t2; +UPDATE t2 SET yy = c2; +UPDATE t4 SET yyyy = c4; +SELECT * FROM t2; +yy c2 +NULL NULL +70 1970 +99 1999 +00 2000 +01 2001 +69 2069 +SELECT * FROM t4; +yyyy c4 +NULL NULL +1970 1970 +1999 1999 +2000 2000 +2001 2001 +2069 2069 +# Comparison of YEAR(2) with YEAR(4) +SELECT * FROM t2, t4 WHERE yy = yyyy; +yy c2 yyyy c4 +70 1970 1970 1970 +99 1999 1999 1999 +00 2000 2000 2000 +01 2001 2001 2001 +69 2069 2069 2069 +SELECT * FROM t2, t4 WHERE yy <=> yyyy; +yy c2 yyyy c4 +NULL NULL NULL NULL +70 1970 1970 1970 +99 1999 1999 1999 +00 2000 2000 2000 +01 2001 2001 2001 +69 2069 2069 2069 +SELECT * FROM t2, t4 WHERE yy < yyyy; +yy c2 yyyy c4 +70 1970 1999 1999 +70 1970 2000 2000 +99 1999 2000 2000 +70 1970 2001 2001 +99 1999 2001 2001 +00 2000 2001 2001 +70 1970 2069 2069 +99 1999 2069 2069 +00 2000 2069 2069 +01 2001 2069 2069 +SELECT * FROM t2, t4 WHERE yy > yyyy; +yy c2 yyyy c4 +99 1999 1970 1970 +00 2000 1970 1970 +01 2001 1970 1970 +69 2069 1970 1970 +00 2000 1999 1999 +01 2001 1999 1999 +69 2069 1999 1999 +01 2001 2000 2000 +69 2069 2000 2000 +69 2069 2001 2001 +# Comparison of YEAR(2) with YEAR(2) +SELECT * FROM t2 a, t2 b WHERE a.yy = b.yy; +yy c2 yy c2 +70 1970 70 1970 +99 1999 99 1999 +00 2000 00 2000 +01 2001 01 2001 +69 2069 69 2069 +SELECT * FROM t2 a, t2 b WHERE a.yy <=> b.yy; +yy c2 yy c2 +NULL NULL NULL NULL +70 1970 70 1970 +99 1999 99 1999 +00 2000 00 2000 +01 2001 01 2001 +69 2069 69 2069 +SELECT * FROM t2 a, t2 b WHERE a.yy < b.yy; +yy c2 yy c2 +70 1970 99 1999 +70 1970 00 2000 +99 1999 00 2000 +70 1970 01 2001 +99 1999 01 2001 +00 2000 01 2001 +70 1970 69 2069 +99 1999 69 2069 +00 2000 69 2069 +01 2001 69 2069 +# Comparison of YEAR(4) with YEAR(4) +SELECT * FROM t4 a, t4 b WHERE a.yyyy = b.yyyy; +yyyy c4 yyyy c4 +1970 1970 1970 1970 +1999 1999 1999 1999 +2000 2000 2000 2000 +2001 2001 2001 2001 +2069 2069 2069 2069 +SELECT * FROM t4 a, t4 b WHERE a.yyyy <=> b.yyyy; +yyyy c4 yyyy c4 +NULL NULL NULL NULL +1970 1970 1970 1970 +1999 1999 1999 1999 +2000 2000 2000 2000 +2001 2001 2001 2001 +2069 2069 2069 2069 +SELECT * FROM t4 a, t4 b WHERE a.yyyy < b.yyyy; +yyyy c4 yyyy c4 +1970 1970 1999 1999 +1970 1970 2000 2000 +1999 1999 2000 2000 +1970 1970 2001 2001 +1999 1999 2001 2001 +2000 2000 2001 2001 +1970 1970 2069 2069 +1999 1999 2069 2069 +2000 2000 2069 2069 +2001 2001 2069 2069 +# Comparison with constants: +SELECT * FROM t2 WHERE yy = NULL; +yy c2 +SELECT * FROM t4 WHERE yyyy = NULL; +yyyy c4 +SELECT * FROM t2 WHERE yy <=> NULL; +yy c2 +NULL NULL +SELECT * FROM t4 WHERE yyyy <=> NULL; +yyyy c4 +NULL NULL +SELECT * FROM t2 WHERE yy < NULL; +yy c2 +SELECT * FROM t2 WHERE yy > NULL; +yy c2 +SELECT * FROM t2 WHERE yy = NOW(); +yy c2 +SELECT * FROM t4 WHERE yyyy = NOW(); +yyyy c4 +SELECT * FROM t2 WHERE yy = 99; +yy c2 +99 1999 +SELECT * FROM t2 WHERE 99 = yy; +yy c2 +99 1999 +SELECT * FROM t4 WHERE yyyy = 99; +yyyy c4 +1999 1999 +SELECT * FROM t2 WHERE yy = 'test'; +yy c2 +00 2000 +Warnings: +Warning 1292 Truncated incorrect DOUBLE value: 'test' +SELECT * FROM t4 WHERE yyyy = 'test'; +yyyy c4 +Warnings: +Warning 1292 Truncated incorrect DOUBLE value: 'test' +SELECT * FROM t2 WHERE yy = '1999'; +yy c2 +99 1999 +SELECT * FROM t4 WHERE yyyy = '1999'; +yyyy c4 +1999 1999 +SELECT * FROM t2 WHERE yy = 1999; +yy c2 +99 1999 +SELECT * FROM t4 WHERE yyyy = 1999; +yyyy c4 +1999 1999 +SELECT * FROM t2 WHERE yy = 1999.1; +yy c2 +99 1999 +SELECT * FROM t4 WHERE yyyy = 1999.1; +yyyy c4 +1999 1999 +SELECT * FROM t2 WHERE yy = 1998.9; +yy c2 +99 1999 +SELECT * FROM t4 WHERE yyyy = 1998.9; +yyyy c4 +1999 1999 +# Coverage tests for YEAR with zero/2000 constants: +SELECT * FROM t2 WHERE yy = 0; +yy c2 +00 2000 +SELECT * FROM t2 WHERE yy = '0'; +yy c2 +00 2000 +SELECT * FROM t2 WHERE yy = '0000'; +yy c2 +00 2000 +SELECT * FROM t2 WHERE yy = '2000'; +yy c2 +00 2000 +SELECT * FROM t2 WHERE yy = 2000; +yy c2 +00 2000 +SELECT * FROM t4 WHERE yyyy = 0; +yyyy c4 +SELECT * FROM t4 WHERE yyyy = '0'; +yyyy c4 +2000 2000 +SELECT * FROM t4 WHERE yyyy = '0000'; +yyyy c4 +SELECT * FROM t4 WHERE yyyy = '2000'; +yyyy c4 +2000 2000 +SELECT * FROM t4 WHERE yyyy = 2000; +yyyy c4 +2000 2000 +# Comparison with constants those are out of YEAR range +# (coverage test for backward compatibility) +SELECT COUNT(yy) FROM t2; +COUNT(yy) +5 +SELECT COUNT(yyyy) FROM t4; +COUNT(yyyy) +5 +SELECT COUNT(*) FROM t2 WHERE yy = -1; +COUNT(*) +0 +SELECT COUNT(*) FROM t4 WHERE yyyy > -1; +COUNT(*) +5 +SELECT COUNT(*) FROM t2 WHERE yy > -1000000000000000000; +COUNT(*) +5 +SELECT COUNT(*) FROM t4 WHERE yyyy > -1000000000000000000; +COUNT(*) +5 +SELECT COUNT(*) FROM t2 WHERE yy < 2156; +COUNT(*) +5 +SELECT COUNT(*) FROM t4 WHERE yyyy < 2156; +COUNT(*) +5 +SELECT COUNT(*) FROM t2 WHERE yy < 1000000000000000000; +COUNT(*) +5 +SELECT COUNT(*) FROM t4 WHERE yyyy < 1000000000000000000; +COUNT(*) +5 +SELECT * FROM t2 WHERE yy < 123; +yy c2 +70 1970 +99 1999 +00 2000 +01 2001 +69 2069 +SELECT * FROM t2 WHERE yy > 123; +yy c2 +SELECT * FROM t4 WHERE yyyy < 123; +yyyy c4 +SELECT * FROM t4 WHERE yyyy > 123; +yyyy c4 +1970 1970 +1999 1999 +2000 2000 +2001 2001 +2069 2069 +DROP TABLE t2, t4; +# +End of 5.1 tests === modified file 'mysql-test/r/udf.result' --- a/mysql-test/r/udf.result 2009-09-07 09:57:22 +0000 +++ b/mysql-test/r/udf.result 2010-01-22 21:19:21 +0000 @@ -38,8 +38,6 @@ ERROR HY000: Can't initialize function ' select reverse_lookup("127.0.0.1"); select reverse_lookup(127,0,0,1); select reverse_lookup("localhost"); -reverse_lookup("localhost") -NULL select avgcost(); ERROR HY000: Can't initialize function 'avgcost'; wrong number of arguments: AVGCOST() requires two arguments select avgcost(100,23.76); @@ -311,29 +309,6 @@ drop function f3; drop function metaphon; drop function myfunc_double; drop function myfunc_int; -CREATE FUNCTION metaphon RETURNS STRING SONAME "UDF_EXAMPLE_LIB"; -create table t1 (a char); -set GLOBAL query_cache_size=1355776; -reset query cache; -select metaphon('MySQL') from t1; -metaphon('MySQL') -show status like "Qcache_hits"; -Variable_name Value -Qcache_hits 0 -show status like "Qcache_queries_in_cache"; -Variable_name Value -Qcache_queries_in_cache 0 -select metaphon('MySQL') from t1; -metaphon('MySQL') -show status like "Qcache_hits"; -Variable_name Value -Qcache_hits 0 -show status like "Qcache_queries_in_cache"; -Variable_name Value -Qcache_queries_in_cache 0 -drop table t1; -drop function metaphon; -set GLOBAL query_cache_size=default; DROP DATABASE IF EXISTS mysqltest; CREATE DATABASE mysqltest; USE mysqltest; === added file 'mysql-test/r/udf_query_cache.result' --- a/mysql-test/r/udf_query_cache.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/r/udf_query_cache.result 2010-01-11 13:15:28 +0000 @@ -0,0 +1,24 @@ +drop table if exists t1; +CREATE FUNCTION metaphon RETURNS STRING SONAME "UDF_EXAMPLE_LIB"; +create table t1 (a char); +set GLOBAL query_cache_size=1355776; +reset query cache; +select metaphon('MySQL') from t1; +metaphon('MySQL') +show status like "Qcache_hits"; +Variable_name Value +Qcache_hits 0 +show status like "Qcache_queries_in_cache"; +Variable_name Value +Qcache_queries_in_cache 0 +select metaphon('MySQL') from t1; +metaphon('MySQL') +show status like "Qcache_hits"; +Variable_name Value +Qcache_hits 0 +show status like "Qcache_queries_in_cache"; +Variable_name Value +Qcache_queries_in_cache 0 +drop table t1; +drop function metaphon; +set GLOBAL query_cache_size=default; === modified file 'mysql-test/r/variables.result' --- a/mysql-test/r/variables.result 2009-09-15 10:46:35 +0000 +++ b/mysql-test/r/variables.result 2010-01-11 13:15:28 +0000 @@ -19,8 +19,6 @@ set @my_myisam_max_sort_file_size =@@glo set @my_net_buffer_length =@@global.net_buffer_length; set @my_net_write_timeout =@@global.net_write_timeout; set @my_net_read_timeout =@@global.net_read_timeout; -set @my_query_cache_limit =@@global.query_cache_limit; -set @my_query_cache_type =@@global.query_cache_type; set @my_rpl_recovery_rank =@@global.rpl_recovery_rank; set @my_server_id =@@global.server_id; set @my_slow_launch_time =@@global.slow_launch_time; @@ -215,7 +213,6 @@ storage_engine MRG_MYISAM select * from information_schema.global_variables where variable_name like 'storage_engine'; VARIABLE_NAME VARIABLE_VALUE STORAGE_ENGINE MRG_MYISAM -set GLOBAL query_cache_size=100000; set GLOBAL myisam_max_sort_file_size=2000000; show global variables like 'myisam_max_sort_file_size'; Variable_name Value @@ -423,8 +420,6 @@ ERROR 42000: Variable 'big_tables' can't show local variables like 'storage_engine'; Variable_name Value storage_engine MEMORY -set SESSION query_cache_size=10000; -ERROR HY000: Variable 'query_cache_size' is a GLOBAL variable and should be set with SET GLOBAL set GLOBAL storage_engine=DEFAULT; ERROR 42000: Variable 'storage_engine' doesn't have a default value set character_set_client=UNKNOWN_CHARACTER_SET; @@ -529,9 +524,6 @@ Warnings: Warning 1292 Truncated incorrect net_buffer_length value: '100' set net_read_timeout=100; set net_write_timeout=100; -set global query_cache_limit=100; -set global query_cache_size=100; -set global query_cache_type=demand; set read_buffer_size=100; Warnings: Warning 1292 Truncated incorrect read_buffer_size value: '100' @@ -1047,8 +1039,6 @@ set global myisam_max_sort_file_size =@m set global net_buffer_length =@my_net_buffer_length; set global net_write_timeout =@my_net_write_timeout; set global net_read_timeout =@my_net_read_timeout; -set global query_cache_limit =@my_query_cache_limit; -set global query_cache_type =@my_query_cache_type; set global rpl_recovery_rank =@my_rpl_recovery_rank; set global server_id =@my_server_id; set global slow_launch_time =@my_slow_launch_time; === added file 'mysql-test/std_data/bug47012.ARM' Binary files a/mysql-test/std_data/bug47012.ARM 1970-01-01 00:00:00 +0000 and b/mysql-test/std_data/bug47012.ARM 2009-11-11 08:03:29 +0000 differ === added file 'mysql-test/std_data/bug47012.ARZ' Binary files a/mysql-test/std_data/bug47012.ARZ 1970-01-01 00:00:00 +0000 and b/mysql-test/std_data/bug47012.ARZ 2009-11-11 08:03:29 +0000 differ === added file 'mysql-test/std_data/bug47012.frm' Binary files a/mysql-test/std_data/bug47012.frm 1970-01-01 00:00:00 +0000 and b/mysql-test/std_data/bug47012.frm 2009-11-11 08:03:29 +0000 differ === modified file 'mysql-test/std_data/cacert.pem' --- a/mysql-test/std_data/cacert.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/cacert.pem 2010-01-29 10:42:31 +0000 @@ -1,17 +1,19 @@ -----BEGIN CERTIFICATE----- -MIICrTCCAhagAwIBAgIJAJXpePU0UOTVMA0GCSqGSIb3DQEBBQUAMEQxCzAJBgNV -BAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxhMRAwDgYDVQQHEwdVcHBzYWxhMREwDwYD -VQQKEwhNeVNRTCBBQjAeFw0wOTAxMjgxMDQ5NDZaFw0xNDAxMjcxMDQ5NDZaMEQx -CzAJBgNVBAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxhMRAwDgYDVQQHEwdVcHBzYWxh -MREwDwYDVQQKEwhNeVNRTCBBQjCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA -4XQHAe5R1+TXC8noZtWf+d5E0v1C59FWpn9SWEUCBjE5UiIwuJvi4Y+7xWGOXLAI -/JzJx5gNXLBiTsE/zh0uX9fKlajLhxB0GN+QU0ZlpQ1BeYipEcNXeI/7cT499f6v -XWabnTflivdCgHSWUOQ20/Lzs6kP6/e6OoZd/DPSjPECAwEAAaOBpjCBozAdBgNV -HQ4EFgQU8uLqVWWkmuKsnZf1RWz294wRrd8wdAYDVR0jBG0wa4AU8uLqVWWkmuKs -nZf1RWz294wRrd+hSKRGMEQxCzAJBgNVBAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxh -MRAwDgYDVQQHEwdVcHBzYWxhMREwDwYDVQQKEwhNeVNRTCBBQoIJAJXpePU0UOTV -MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADgYEAMMTE5sDN+Z0ZlV7KvH3g -6+aKvql8dTpRT3hYukeQlWua0nq74WPGVw0c4e/M/vbiMwmJcCYpB9pd4+dHqzSw -aPyoenjY6UF8n7B4quWy3SIUk2LSHeJLW+kzJn2afN9gvipFhdVh/uU2TIyLGOur -Z/vmJX2W7hF1uqPnbfa8Lrw= +MIIDIjCCAougAwIBAgIJAJhuvLP+2mGwMA0GCSqGSIb3DQEBBQUAMGoxCzAJBgNV +BAYTAkZJMRAwDgYDVQQIEwdUdXVzdWxhMRkwFwYDVQQKExBNb250eSBQcm9ncmFt +IEFiMS4wLAYJKoZIhvcNAQkBFh9hYnN0cmFjdC5kZXZlbG9wZXJAYXNrbW9udHku +b3JnMB4XDTEwMDEyODIxNTcyNVoXDTEwMDIyNzIxNTcyNVowajELMAkGA1UEBhMC +RkkxEDAOBgNVBAgTB1R1dXN1bGExGTAXBgNVBAoTEE1vbnR5IFByb2dyYW0gQWIx +LjAsBgkqhkiG9w0BCQEWH2Fic3RyYWN0LmRldmVsb3BlckBhc2ttb250eS5vcmcw +gZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMIaY4pwbst50S32xJH3bSXhPGep +6gx1AWwZKsHTXL3VeMO6PHmC8zu5HM0zbOcrIJcXL3YVnpmE4b9OQxIiMSx1Yd+U +u8/sTkxgpsEKhCbIzECIwPhppyT/JP5aSXCadEvg+PSjikv8dOVkD68wVG4CcFIX +MFttsPebBVzEokZZAgMBAAGjgc8wgcwwHQYDVR0OBBYEFOCKaNHFFPrju8AwzWxS +f96IKfRwMIGcBgNVHSMEgZQwgZGAFOCKaNHFFPrju8AwzWxSf96IKfRwoW6kbDBq +MQswCQYDVQQGEwJGSTEQMA4GA1UECBMHVHV1c3VsYTEZMBcGA1UEChMQTW9udHkg +UHJvZ3JhbSBBYjEuMCwGCSqGSIb3DQEJARYfYWJzdHJhY3QuZGV2ZWxvcGVyQGFz +a21vbnR5Lm9yZ4IJAJhuvLP+2mGwMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEF +BQADgYEAsmCX2/k9AInq2qhXtnkLip6cB0iOerLTNAzEijZc/aVf4wUjkL3cqhmC +kSTCwAHIOxp+ICwh6ky3xghXjoI9QnPFDVkRkzPT2tV0IoBaeQuI4e0CU2EY7L3P +XoDqp3oq1XtVcr9ZZdP68fBYUG/qcrWcXWk45ZFaBmBv3TotsGk= -----END CERTIFICATE----- === modified file 'mysql-test/std_data/client-cert.pem' --- a/mysql-test/std_data/client-cert.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/client-cert.pem 2010-01-29 10:42:31 +0000 @@ -1,55 +1,60 @@ Certificate: Data: Version: 3 (0x2) - Serial Number: 3 (0x3) + Serial Number: 2 (0x2) Signature Algorithm: sha1WithRSAEncryption - Issuer: C=SE, ST=Uppsala, L=Uppsala, O=MySQL AB + Issuer: C=FI, ST=Tuusula, O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org Validity - Not Before: Jan 28 11:04:39 2009 GMT - Not After : Jan 28 11:04:39 2010 GMT - Subject: C=SE, ST=Uppsala, O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com + Not Before: Jan 28 22:01:38 2010 GMT + Not After : Dec 7 22:01:38 2019 GMT + Subject: C=FI, ST=Tuusula, O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org Subject Public Key Info: Public Key Algorithm: rsaEncryption - RSA Public Key: (512 bit) - Modulus (512 bit): - 00:e1:52:30:2c:d9:be:64:28:91:5d:7a:fd:d9:e9: - 14:35:7a:d2:94:4e:91:46:e0:db:9f:6b:79:f4:4c: - ac:6e:07:61:34:86:74:62:a7:a8:44:af:fa:87:87: - a8:7d:42:61:ff:ab:50:d4:7b:bf:75:fa:d5:d5:b3: - 74:fb:56:1e:37 + RSA Public Key: (1024 bit) + Modulus (1024 bit): + 00:bd:dc:3d:f8:3c:0b:d4:d2:c0:a3:9c:34:2d:e7: + 11:ff:4d:43:35:17:f6:0d:91:01:92:9e:4f:4d:c0: + 38:d5:62:03:55:33:db:66:a2:91:d3:f2:b6:23:34: + 95:53:50:3d:4f:e3:0c:d7:76:fd:f6:54:64:e6:f6: + dc:70:74:7c:6b:74:41:59:b0:19:5d:62:90:3c:a7: + c8:5e:21:8f:2b:22:6b:c7:43:9b:be:79:84:60:da: + 16:c9:ce:ee:fd:66:cb:54:81:e2:b5:1c:cf:f9:74: + de:38:2b:28:d4:31:33:55:d2:30:1c:a3:e4:c2:c7: + 31:46:43:d5:33:3d:8a:0b:47 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE + Netscape Comment: + OpenSSL Generated Certificate X509v3 Subject Key Identifier: - 58:30:B5:9B:2C:05:94:06:BA:3D:3C:F0:B2:CD:1D:67:65:E3:7F:85 + BE:E6:DB:19:8D:DB:72:9A:85:EE:B2:B8:5D:E7:FF:61:DF:09:08:AF X509v3 Authority Key Identifier: - keyid:F2:E2:EA:55:65:A4:9A:E2:AC:9D:97:F5:45:6C:F6:F7:8C:11:AD:DF - DirName:/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB - serial:95:E9:78:F5:34:50:E4:D5 + keyid:E0:8A:68:D1:C5:14:FA:E3:BB:C0:30:CD:6C:52:7F:DE:88:29:F4:70 Signature Algorithm: sha1WithRSAEncryption - 05:19:e3:13:14:fc:c5:28:bf:69:f8:00:b3:25:cb:bd:ca:9f: - 2f:4c:b3:a8:04:11:f0:74:27:bd:82:2c:b4:49:9b:a7:59:f0: - f7:87:d1:e0:ba:99:a2:fe:4b:1d:10:6f:e4:a2:b3:cd:7f:8b: - 68:31:46:ee:cd:9e:e2:47:e1:4c:fa:74:d1:e2:8b:cc:a0:4b: - a8:24:d1:a4:c3:6b:2a:c6:28:cd:41:e0:06:48:e6:cf:f2:3c: - ca:37:95:d7:29:64:6b:91:91:83:e7:ac:c8:0b:87:bc:da:a6: - aa:f1:44:43:c8:74:7b:15:26:91:2e:03:c4:71:50:6c:f8:68: - dc:8c + 41:95:6d:0a:a4:ee:af:68:cd:94:26:59:9a:18:b7:75:3c:c5: + 0f:22:d3:5c:31:9b:85:a0:93:b3:f0:50:29:ba:1e:d3:5a:43: + 0b:77:2d:98:87:a7:a7:39:0f:40:8d:03:d3:b3:67:43:77:bc: + 3c:51:c2:f9:9e:7a:2d:39:c4:5c:16:d7:70:d6:74:d1:6c:e1: + 6a:4d:fd:1f:10:af:64:3b:f4:64:e9:b2:b3:fb:c8:cd:c5:41: + cd:99:e0:ac:83:1d:81:2c:6b:99:ba:80:02:12:72:f7:3b:bb: + 93:72:00:da:ff:d3:87:75:d2:3a:a4:ca:4d:c1:8b:c1:21:50: + cb:57 -----BEGIN CERTIFICATE----- -MIICfzCCAeigAwIBAgIBAzANBgkqhkiG9w0BAQUFADBEMQswCQYDVQQGEwJTRTEQ -MA4GA1UECBMHVXBwc2FsYTEQMA4GA1UEBxMHVXBwc2FsYTERMA8GA1UEChMITXlT -UUwgQUIwHhcNMDkwMTI4MTEwNDM5WhcNMTAwMTI4MTEwNDM5WjBlMQswCQYDVQQG -EwJTRTEQMA4GA1UECBMHVXBwc2FsYTERMA8GA1UEChMITXlTUUwgQUIxMTAvBgkq -hkiG9w0BCQEWImFic3RyYWN0Lm15c3FsLmRldmVsb3BlckBteXNxbC5jb20wXDAN -BgkqhkiG9w0BAQEFAANLADBIAkEA4VIwLNm+ZCiRXXr92ekUNXrSlE6RRuDbn2t5 -9EysbgdhNIZ0YqeoRK/6h4eofUJh/6tQ1Hu/dfrV1bN0+1YeNwIDAQABo4GjMIGg -MAkGA1UdEwQCMAAwHQYDVR0OBBYEFFgwtZssBZQGuj088LLNHWdl43+FMHQGA1Ud -IwRtMGuAFPLi6lVlpJrirJ2X9UVs9veMEa3foUikRjBEMQswCQYDVQQGEwJTRTEQ -MA4GA1UECBMHVXBwc2FsYTEQMA4GA1UEBxMHVXBwc2FsYTERMA8GA1UEChMITXlT -UUwgQUKCCQCV6Xj1NFDk1TANBgkqhkiG9w0BAQUFAAOBgQAFGeMTFPzFKL9p+ACz -Jcu9yp8vTLOoBBHwdCe9giy0SZunWfD3h9Hgupmi/ksdEG/korPNf4toMUbuzZ7i -R+FM+nTR4ovMoEuoJNGkw2sqxijNQeAGSObP8jzKN5XXKWRrkZGD56zIC4e82qaq -8URDyHR7FSaRLgPEcVBs+GjcjA== +MIICxTCCAi6gAwIBAgIBAjANBgkqhkiG9w0BAQUFADBqMQswCQYDVQQGEwJGSTEQ +MA4GA1UECBMHVHV1c3VsYTEZMBcGA1UEChMQTW9udHkgUHJvZ3JhbSBBYjEuMCwG +CSqGSIb3DQEJARYfYWJzdHJhY3QuZGV2ZWxvcGVyQGFza21vbnR5Lm9yZzAeFw0x +MDAxMjgyMjAxMzhaFw0xOTEyMDcyMjAxMzhaMGoxCzAJBgNVBAYTAkZJMRAwDgYD +VQQIEwdUdXVzdWxhMRkwFwYDVQQKExBNb250eSBQcm9ncmFtIEFiMS4wLAYJKoZI +hvcNAQkBFh9hYnN0cmFjdC5kZXZlbG9wZXJAYXNrbW9udHkub3JnMIGfMA0GCSqG +SIb3DQEBAQUAA4GNADCBiQKBgQC93D34PAvU0sCjnDQt5xH/TUM1F/YNkQGSnk9N +wDjVYgNVM9tmopHT8rYjNJVTUD1P4wzXdv32VGTm9txwdHxrdEFZsBldYpA8p8he +IY8rImvHQ5u+eYRg2hbJzu79ZstUgeK1HM/5dN44KyjUMTNV0jAco+TCxzFGQ9Uz +PYoLRwIDAQABo3sweTAJBgNVHRMEAjAAMCwGCWCGSAGG+EIBDQQfFh1PcGVuU1NM +IEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUvubbGY3bcpqF7rK4Xef/ +Yd8JCK8wHwYDVR0jBBgwFoAU4Ipo0cUU+uO7wDDNbFJ/3ogp9HAwDQYJKoZIhvcN +AQEFBQADgYEAQZVtCqTur2jNlCZZmhi3dTzFDyLTXDGbhaCTs/BQKboe01pDC3ct +mIenpzkPQI0D07NnQ3e8PFHC+Z56LTnEXBbXcNZ00Wzhak39HxCvZDv0ZOmys/vI +zcVBzZngrIMdgSxrmbqAAhJy9zu7k3IA2v/Th3XSOqTKTcGLwSFQy1c= -----END CERTIFICATE----- === modified file 'mysql-test/std_data/client-key.pem' --- a/mysql-test/std_data/client-key.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/client-key.pem 2010-01-29 10:42:31 +0000 @@ -1,9 +1,15 @@ -----BEGIN RSA PRIVATE KEY----- -MIIBOQIBAAJBAOFSMCzZvmQokV16/dnpFDV60pROkUbg259refRMrG4HYTSGdGKn -qESv+oeHqH1CYf+rUNR7v3X61dWzdPtWHjcCAwEAAQJAXYooM8ZlcuEgj+VKU1ee -qyEFIMqJJxqcMk+E/nWCM96WxCP3zHNSrqNfSpI3ld7QzMwhdRz+gFLxT2gGNpIw -MQIhAPxzM/lDihe67X3ADYtDl9ZjA8Pm430x9sXlcxI17tCZAiEA5H1SyFl4mUee -9VnfSC2XGW7lwz72ZygfVX+b7tLWF08CIEh40gzW5MfXM+KLxdea+fXjyursV5ZT -R6KcMiKiNQLRAiAcmHqlzFzFgisotai2Fc6VRkXHG7gmzOSvBJt1VjmpDQIge6jf -2N7whTdvC4ferB+zUlgWQdyvx1c3T4gnt6PYdaY= +MIICXgIBAAKBgQC93D34PAvU0sCjnDQt5xH/TUM1F/YNkQGSnk9NwDjVYgNVM9tm +opHT8rYjNJVTUD1P4wzXdv32VGTm9txwdHxrdEFZsBldYpA8p8heIY8rImvHQ5u+ +eYRg2hbJzu79ZstUgeK1HM/5dN44KyjUMTNV0jAco+TCxzFGQ9UzPYoLRwIDAQAB +AoGBAJa2lprPT7UJ99Ho1aL6ota/RnKHKtNqII17DgjyZis9OtgP6kJ3GrvdF6iq +vT79my4nVrJTyxYXuGF/5U1/qqNjuPPBE1Xbu1ubQlFv8CT0kKYynQ7Z3ls8fAHC +B3VJXnUVlG+GHtUEFFG4FQVX1fn/Sga67ioJ6ivAiBlHKaPBAkEA5f2ToWlj4u9O +KgfRkN54wdIp4yu2c40pbhMfKGjGGsBAHk92+qSBpzEmxLcI6Ay+4/QysSR4jYmK +jCJuxiTu1QJBANNU3Hx8Il2SF/2BqGLcIh2SHxzKQIT5wAyD2jb+P2cHvbk6pKGR +VTmw5bibxXmYMS6J/L2zUF2xtFe+Svwz96sCQEnKYSqBqOWvyBFeLtPfPTlal8vm +Q4SxfuBtTCrn6t+8XRYcgt0KGPsunvSwkS/6nuh+eiExxWgMACLUDVyPjv0CQQC4 +sJJc7LOv6Oy0bWr2swHRrBEqvQsz63zOszCzHPHWHirNxZV5aiT8XT/2XZRwlvRs +gsVyGFLk/1fn0vN/g/8vAkEAxUdzUKvC1ZwjzGhgcz2bQU0tEZN4C9jBCiwOI2ud +BpAsPG0xAGGL2+hz0B0n88XiTHobiTZ1bg4Z41i4pXx2ZA== -----END RSA PRIVATE KEY----- === modified file 'mysql-test/std_data/server-cert.pem' --- a/mysql-test/std_data/server-cert.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/server-cert.pem 2010-01-29 10:42:31 +0000 @@ -3,53 +3,59 @@ Certificate: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha1WithRSAEncryption - Issuer: C=SE, ST=Uppsala, L=Uppsala, O=MySQL AB + Issuer: C=FI, ST=Tuusula, O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org Validity - Not Before: Jan 28 10:55:13 2009 GMT - Not After : Jan 28 10:55:13 2010 GMT - Subject: C=SE, ST=Uppsala, O=MySQL AB, CN=localhost/emailAddress=abstract.mysql.developer(a)mysql.com + Not Before: Jan 28 21:59:14 2010 GMT + Not After : Dec 7 21:59:14 2019 GMT + Subject: C=FI, ST=Tuusula, O=Monty Program Ab, CN=localhost/emailAddress=abstract.developer(a)askmonty.org Subject Public Key Info: Public Key Algorithm: rsaEncryption - RSA Public Key: (512 bit) - Modulus (512 bit): - 00:b6:8f:e5:b7:b4:86:83:13:8a:f9:bf:63:cb:64: - 2d:b9:51:d1:de:ab:7b:45:1f:aa:b5:66:73:13:f9: - a6:07:d5:ba:7c:fa:92:bd:37:e2:ad:87:db:3e:b6: - 6a:12:64:f8:ee:17:e3:15:06:2f:a8:82:68:bf:57: - 8d:c3:04:98:27 + RSA Public Key: (1024 bit) + Modulus (1024 bit): + 00:cc:79:74:2a:14:7e:77:06:b0:ec:1b:b6:da:70: + 4c:4f:0e:94:04:8a:e7:69:f4:4c:9c:67:22:41:56: + 3a:96:53:9e:95:9d:17:55:80:41:c0:13:d6:94:0f: + cd:2c:51:fe:a4:6e:f2:74:d8:5d:3b:3a:80:e3:85: + 5b:a5:bc:7d:5c:25:55:e5:40:77:fe:f3:cb:5b:cd: + da:a5:f4:36:64:30:a2:a0:7f:93:b3:c4:56:75:2a: + c0:f7:65:2a:d7:e6:ff:88:25:03:e0:b4:40:2e:74: + 4c:cb:46:95:73:cb:25:5c:87:0e:ec:0f:5a:19:c2: + b6:dc:9e:e8:f0:30:b1:9c:99 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE + Netscape Comment: + OpenSSL Generated Certificate X509v3 Subject Key Identifier: - D9:9A:B8:5F:22:EA:04:10:C8:25:7D:82:57:E6:2E:FD:19:29:E7:DA + 6D:13:3B:40:52:3C:AF:18:EA:33:D1:B7:56:21:1B:05:FE:0B:9E:38 X509v3 Authority Key Identifier: - keyid:F2:E2:EA:55:65:A4:9A:E2:AC:9D:97:F5:45:6C:F6:F7:8C:11:AD:DF - DirName:/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB - serial:95:E9:78:F5:34:50:E4:D5 + keyid:E0:8A:68:D1:C5:14:FA:E3:BB:C0:30:CD:6C:52:7F:DE:88:29:F4:70 Signature Algorithm: sha1WithRSAEncryption - 54:07:2d:21:0b:a5:af:3b:58:23:32:5e:56:7f:ab:58:63:48: - 91:aa:38:90:89:16:f9:cc:bf:a4:0e:78:2b:9f:c5:1b:58:a6: - e6:08:8f:2e:ae:97:03:21:9b:f1:cd:c0:26:8f:1d:d7:28:27: - a0:8e:81:09:1b:1c:0f:c9:a5:41:3a:2d:44:3f:9c:fa:87:ff: - c8:4c:2b:44:f7:1b:c1:3e:4f:01:7f:e9:26:cc:9f:1c:06:b5: - 0b:27:d1:10:90:be:93:0c:9c:e7:b0:d1:ea:27:99:4e:06:14: - 0c:7a:e9:c1:52:c5:33:68:bc:61:0d:db:81:3b:57:48:57:bf: - 42:9a + 97:db:65:23:7f:f1:15:3c:1e:83:ac:0e:0a:50:a0:0c:22:b8: + 45:d4:ca:21:05:47:3b:3d:03:b5:6c:4b:8d:bb:5f:57:c3:c7: + 4e:71:23:cf:33:a3:7f:a0:3d:bd:58:75:b8:37:22:16:2f:e9: + ed:ae:9b:94:29:81:6e:34:79:cf:41:bd:3d:8d:17:d7:22:1c: + 1b:58:c7:0f:79:13:56:1d:e8:d8:4e:e5:07:3f:79:1b:dd:c4: + 06:9b:c5:b6:02:34:43:c5:bf:e5:87:ad:f1:c1:8a:f2:be:c2: + 00:1d:d4:27:1f:87:c8:80:31:ec:6e:97:95:b4:84:40:d1:73: + 42:71 -----BEGIN CERTIFICATE----- -MIICkzCCAfygAwIBAgIBATANBgkqhkiG9w0BAQUFADBEMQswCQYDVQQGEwJTRTEQ -MA4GA1UECBMHVXBwc2FsYTEQMA4GA1UEBxMHVXBwc2FsYTERMA8GA1UEChMITXlT -UUwgQUIwHhcNMDkwMTI4MTA1NTEzWhcNMTAwMTI4MTA1NTEzWjB5MQswCQYDVQQG -EwJTRTEQMA4GA1UECBMHVXBwc2FsYTERMA8GA1UEChMITXlTUUwgQUIxEjAQBgNV -BAMTCWxvY2FsaG9zdDExMC8GCSqGSIb3DQEJARYiYWJzdHJhY3QubXlzcWwuZGV2 -ZWxvcGVyQG15c3FsLmNvbTBcMA0GCSqGSIb3DQEBAQUAA0sAMEgCQQC2j+W3tIaD -E4r5v2PLZC25UdHeq3tFH6q1ZnMT+aYH1bp8+pK9N+Kth9s+tmoSZPjuF+MVBi+o -gmi/V43DBJgnAgMBAAGjgaMwgaAwCQYDVR0TBAIwADAdBgNVHQ4EFgQU2Zq4XyLq -BBDIJX2CV+Yu/Rkp59owdAYDVR0jBG0wa4AU8uLqVWWkmuKsnZf1RWz294wRrd+h -SKRGMEQxCzAJBgNVBAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxhMRAwDgYDVQQHEwdV -cHBzYWxhMREwDwYDVQQKEwhNeVNRTCBBQoIJAJXpePU0UOTVMA0GCSqGSIb3DQEB -BQUAA4GBAFQHLSELpa87WCMyXlZ/q1hjSJGqOJCJFvnMv6QOeCufxRtYpuYIjy6u -lwMhm/HNwCaPHdcoJ6COgQkbHA/JpUE6LUQ/nPqH/8hMK0T3G8E+TwF/6SbMnxwG -tQsn0RCQvpMMnOew0eonmU4GFAx66cFSxTNovGEN24E7V0hXv0Ka +MIIC2TCCAkKgAwIBAgIBATANBgkqhkiG9w0BAQUFADBqMQswCQYDVQQGEwJGSTEQ +MA4GA1UECBMHVHV1c3VsYTEZMBcGA1UEChMQTW9udHkgUHJvZ3JhbSBBYjEuMCwG +CSqGSIb3DQEJARYfYWJzdHJhY3QuZGV2ZWxvcGVyQGFza21vbnR5Lm9yZzAeFw0x +MDAxMjgyMTU5MTRaFw0xOTEyMDcyMTU5MTRaMH4xCzAJBgNVBAYTAkZJMRAwDgYD +VQQIEwdUdXVzdWxhMRkwFwYDVQQKExBNb250eSBQcm9ncmFtIEFiMRIwEAYDVQQD +Ewlsb2NhbGhvc3QxLjAsBgkqhkiG9w0BCQEWH2Fic3RyYWN0LmRldmVsb3BlckBh +c2ttb250eS5vcmcwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMx5dCoUfncG +sOwbttpwTE8OlASK52n0TJxnIkFWOpZTnpWdF1WAQcAT1pQPzSxR/qRu8nTYXTs6 +gOOFW6W8fVwlVeVAd/7zy1vN2qX0NmQwoqB/k7PEVnUqwPdlKtfm/4glA+C0QC50 +TMtGlXPLJVyHDuwPWhnCttye6PAwsZyZAgMBAAGjezB5MAkGA1UdEwQCMAAwLAYJ +YIZIAYb4QgENBB8WHU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1Ud +DgQWBBRtEztAUjyvGOoz0bdWIRsF/gueODAfBgNVHSMEGDAWgBTgimjRxRT647vA +MM1sUn/eiCn0cDANBgkqhkiG9w0BAQUFAAOBgQCX22Ujf/EVPB6DrA4KUKAMIrhF +1MohBUc7PQO1bEuNu19Xw8dOcSPPM6N/oD29WHW4NyIWL+ntrpuUKYFuNHnPQb09 +jRfXIhwbWMcPeRNWHejYTuUHP3kb3cQGm8W2AjRDxb/lh63xwYryvsIAHdQnH4fI +gDHsbpeVtIRA0XNCcQ== -----END CERTIFICATE----- === modified file 'mysql-test/std_data/server-key.pem' --- a/mysql-test/std_data/server-key.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/server-key.pem 2010-01-29 10:42:31 +0000 @@ -1,9 +1,15 @@ -----BEGIN RSA PRIVATE KEY----- -MIIBOQIBAAJBALaP5be0hoMTivm/Y8tkLblR0d6re0UfqrVmcxP5pgfVunz6kr03 -4q2H2z62ahJk+O4X4xUGL6iCaL9XjcMEmCcCAwEAAQJASA5VwgNb0CKHiPm0ntOk -hG+54SRX3DmafEy6gRjZIl/bZ/asSLhXUZ+CeohyrQh7BZgYWvykd8pRISL9eKsU -GQIhAOXkUrOtP/EtjyqNluEqZdG+RZi/7p61JS3Ce13Myu+LAiEAy0uMlV34AJpM -b40FPKqlHxw8DD/Dt1iKhNVAg8+LDVUCIFjv7fbJDbW2VG63/Cj8CAwOukoP5rbL -iaicVrHBKrllAiB9+MiaXeopZXNrxDS0jQFYr8Q9yt1aJVFgUkxx4Q9HKQIgZPs0 -KlF3NNNWw78INaAEkyf0IEssnLMsuoCWw0DIOak= +MIICXgIBAAKBgQDMeXQqFH53BrDsG7bacExPDpQEiudp9EycZyJBVjqWU56VnRdV +gEHAE9aUD80sUf6kbvJ02F07OoDjhVulvH1cJVXlQHf+88tbzdql9DZkMKKgf5Oz +xFZ1KsD3ZSrX5v+IJQPgtEAudEzLRpVzyyVchw7sD1oZwrbcnujwMLGcmQIDAQAB +AoGBAMdMYkNZsmJFbVDVOobzCg3Mgc1jrmeBrOKNS8AvUe+QFXRyp3m5B102eOHb +/PmD+hU/5qao9UZzoYkiRM/oRq45jrqJEYwWrX007bKK0F9hnErtC1ImM1nBFVhx +6+6cr+ShUkvtj8+wJ2d5bIccUzGCUfFR5tb5BnePTXK8IVoBAkEA7WGNxHAVKgjS +AzlpHr5fvpivA07hNVJizTwZdWGGYeETilZhkkuMRwREceeohF6ILMf0FTZdFSa/ +8EeLa3icIQJBANyDKFjynKwWy5pyRSz75mVwrEi+4eTQPsCPNWLkbpbEPwqPLYWJ +2VSFkISXF7b7Od48JkQWgiB8/kXqMDEdsXkCQQCzZvj3ryWvoP7nhOoXXBWMPGR4 +gZLe86bMKVGsTsp7CtnzwRj4sbQQr/7yfvvzHmaYQX4M0gtDQwfolomd7YdBAkEA +y24ETuqjNu9grf81aiaJipPDnOjcJOcovSRgr/blPxmUvv0Pld5yLNN7W5a4PgrO +fAMpmi7ZpXcqbP17sBQgoQJAWTDFKAmfHPVdDGZuCw4yceP5d+Tv7ABglZUvpPKx +kAvGN1WBASUuCQJDOIgzl6gvYX07S5p147i9mv7UBWOpvw== -----END RSA PRIVATE KEY----- === modified file 'mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result' --- a/mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result 2009-09-28 12:41:10 +0000 +++ b/mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result 2009-11-18 14:50:31 +0000 @@ -1,3 +1,4 @@ +CALL mtr.add_suppression("Statement may not be safe to log in statement format."); drop table if exists t1, t2; create table t1 (a int) engine=innodb; create table t2 (a int) engine=myisam; @@ -224,6 +225,8 @@ create table t0 (n int); insert t0 select * from t1; set autocommit=1; insert into t0 select GET_LOCK("lock1",null); +Warnings: +Note 1592 Statement may not be safe to log in statement format. set autocommit=0; create table t2 (n int) engine=innodb; insert into t2 values (3); === modified file 'mysql-test/suite/binlog/r/binlog_stm_row.result' --- a/mysql-test/suite/binlog/r/binlog_stm_row.result 2009-09-07 20:50:10 +0000 +++ b/mysql-test/suite/binlog/r/binlog_stm_row.result 2010-01-15 15:27:55 +0000 @@ -1,3 +1,4 @@ +CALL mtr.add_suppression("Statement may not be safe to log in statement format."); DROP TABLE IF EXISTS t1; DROP TABLE IF EXISTS t2; set @saved_global_binlog_format = @@global.binlog_format; @@ -29,6 +30,8 @@ SELECT RELEASE_LOCK('Bug#34306'); RELEASE_LOCK('Bug#34306') 1 # con2 +Warnings: +Note 1592 Statement may not be safe to log in statement format. SELECT RELEASE_LOCK('Bug#34306'); RELEASE_LOCK('Bug#34306') 1 === modified file 'mysql-test/suite/binlog/r/binlog_unsafe.result' --- a/mysql-test/suite/binlog/r/binlog_unsafe.result 2009-09-07 20:50:10 +0000 +++ b/mysql-test/suite/binlog/r/binlog_unsafe.result 2010-01-19 10:36:52 +0000 @@ -327,4 +327,87 @@ Warnings: Note 1592 Statement may not be safe to log in statement format. DROP TABLE t1, t2; SET @@SESSION.SQL_MODE = @save_sql_mode; +CREATE TABLE t1 (a VARCHAR(1000)); +INSERT INTO t1 VALUES (CURRENT_USER()); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (FOUND_ROWS()); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (GET_LOCK('tmp', 1)); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (IS_FREE_LOCK('tmp')); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (IS_USED_LOCK('tmp')); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (LOAD_FILE('../../std_data/words2.dat')); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (MASTER_POS_WAIT('dummy arg', 4711, 1)); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (RELEASE_LOCK('tmp')); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (ROW_COUNT()); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (SESSION_USER()); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (SLEEP(1)); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (SYSDATE()); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (SYSTEM_USER()); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (USER()); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (UUID()); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (UUID_SHORT()); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +INSERT INTO t1 VALUES (VERSION()); +Warnings: +Note 1592 Statement may not be safe to log in statement format. +DELETE FROM t1; +SET TIME_ZONE= '+03:00'; +SET TIMESTAMP=1000000; +INSERT INTO t1 VALUES +(CURDATE()), +(CURRENT_DATE()), +(CURRENT_TIME()), +(CURRENT_TIMESTAMP()), +(CURTIME()), +(LOCALTIME()), +(LOCALTIMESTAMP()), +(NOW()), +(UNIX_TIMESTAMP()), +(UTC_DATE()), +(UTC_TIME()), +(UTC_TIMESTAMP()); +SELECT * FROM t1; +a +1970-01-12 +1970-01-12 +16:46:40 +1970-01-12 16:46:40 +16:46:40 +1970-01-12 16:46:40 +1970-01-12 16:46:40 +1970-01-12 16:46:40 +1000000 +1970-01-12 +13:46:40 +1970-01-12 13:46:40 +DROP TABLE t1; "End of tests" === modified file 'mysql-test/suite/binlog/t/binlog_killed.test' --- a/mysql-test/suite/binlog/t/binlog_killed.test 2008-10-23 19:27:09 +0000 +++ b/mysql-test/suite/binlog/t/binlog_killed.test 2009-11-18 14:50:31 +0000 @@ -1,5 +1,5 @@ -- source include/have_innodb.inc --- source include/have_binlog_format_mixed_or_statement.inc +-- source include/have_binlog_format_statement.inc # You cannot use `KILL' with the Embedded MySQL Server library, # because the embedded server merely runs inside the threads of the host === modified file 'mysql-test/suite/binlog/t/binlog_stm_mix_innodb_myisam.test' --- a/mysql-test/suite/binlog/t/binlog_stm_mix_innodb_myisam.test 2008-02-28 11:21:44 +0000 +++ b/mysql-test/suite/binlog/t/binlog_stm_mix_innodb_myisam.test 2009-11-18 14:50:31 +0000 @@ -2,6 +2,9 @@ # For both statement and row based bin logs 9/19/2005 [jbm] -- source include/have_binlog_format_statement.inc + +CALL mtr.add_suppression("Statement may not be safe to log in statement format."); + -- source extra/binlog_tests/mix_innodb_myisam_binlog.test set @@session.binlog_format=statement; === modified file 'mysql-test/suite/binlog/t/binlog_stm_row.test' --- a/mysql-test/suite/binlog/t/binlog_stm_row.test 2009-02-19 09:01:25 +0000 +++ b/mysql-test/suite/binlog/t/binlog_stm_row.test 2010-01-15 15:27:55 +0000 @@ -1,5 +1,8 @@ --source include/have_log_bin.inc ---source include/have_binlog_format_row_or_statement.inc +# Test sets its own binlog_format, so we restrict it to run only once +--source include/have_binlog_format_row.inc + +CALL mtr.add_suppression("Statement may not be safe to log in statement format."); # Get rid of previous tests binlog --disable_query_log === modified file 'mysql-test/suite/binlog/t/binlog_unsafe.test' --- a/mysql-test/suite/binlog/t/binlog_unsafe.test 2009-09-07 20:50:10 +0000 +++ b/mysql-test/suite/binlog/t/binlog_unsafe.test 2010-01-19 10:36:52 +0000 @@ -388,4 +388,57 @@ DELETE FROM t1 LIMIT 1; DROP TABLE t1, t2; SET @@SESSION.SQL_MODE = @save_sql_mode; + +# +# BUG#47995: Mark user functions as unsafe +# +# Test that the system functions that are supposed to be marked unsafe +# generate a warning. Each INSERT statement below should generate a +# warning. +# + +CREATE TABLE t1 (a VARCHAR(1000)); +INSERT INTO t1 VALUES (CURRENT_USER()); #marked unsafe before BUG#47995 +INSERT INTO t1 VALUES (FOUND_ROWS()); #marked unsafe before BUG#47995 +INSERT INTO t1 VALUES (GET_LOCK('tmp', 1)); +INSERT INTO t1 VALUES (IS_FREE_LOCK('tmp')); +INSERT INTO t1 VALUES (IS_USED_LOCK('tmp')); +INSERT INTO t1 VALUES (LOAD_FILE('../../std_data/words2.dat')); #marked unsafe before BUG#47995 +INSERT INTO t1 VALUES (MASTER_POS_WAIT('dummy arg', 4711, 1)); +INSERT INTO t1 VALUES (RELEASE_LOCK('tmp')); +INSERT INTO t1 VALUES (ROW_COUNT()); #marked unsafe before BUG#47995 +INSERT INTO t1 VALUES (SESSION_USER()); #marked unsafe before BUG#47995 +INSERT INTO t1 VALUES (SLEEP(1)); +INSERT INTO t1 VALUES (SYSDATE()); +INSERT INTO t1 VALUES (SYSTEM_USER()); #marked unsafe before BUG#47995 +INSERT INTO t1 VALUES (USER()); #marked unsafe before BUG#47995 +INSERT INTO t1 VALUES (UUID()); #marked unsafe before BUG#47995 +INSERT INTO t1 VALUES (UUID_SHORT()); #marked unsafe before BUG#47995 +INSERT INTO t1 VALUES (VERSION()); +DELETE FROM t1; + +# Since we replicate the TIMESTAMP variable, functions affected by the +# TIMESTAMP variable are safe to replicate. So we check that the +# following following functions depend on the TIMESTAMP variable and +# don't generate a warning. + +SET TIME_ZONE= '+03:00'; +SET TIMESTAMP=1000000; +INSERT INTO t1 VALUES + (CURDATE()), + (CURRENT_DATE()), + (CURRENT_TIME()), + (CURRENT_TIMESTAMP()), + (CURTIME()), + (LOCALTIME()), + (LOCALTIMESTAMP()), + (NOW()), + (UNIX_TIMESTAMP()), + (UTC_DATE()), + (UTC_TIME()), + (UTC_TIMESTAMP()); +SELECT * FROM t1; + +DROP TABLE t1; + --echo "End of tests" === modified file 'mysql-test/suite/funcs_1/datadict/processlist_val.inc' --- a/mysql-test/suite/funcs_1/datadict/processlist_val.inc 2009-10-10 09:59:06 +0000 +++ b/mysql-test/suite/funcs_1/datadict/processlist_val.inc 2010-01-11 13:15:28 +0000 @@ -238,7 +238,7 @@ echo # Poll till all connections of 'test_user' are in a state with COMMAND = 'Sleep' ; let $wait_condition= SELECT COUNT(*) = 2 FROM INFORMATION_SCHEMA.PROCESSLIST - WHERE USER = 'test_user' AND COMMAND = 'Sleep'; + WHERE USER = 'test_user' AND COMMAND = 'Sleep' AND STATE = ''; --source include/wait_condition.inc echo # ----- switch to connection con2 (user = test_user) ----- === modified file 'mysql-test/suite/funcs_1/r/is_columns_is.result' --- a/mysql-test/suite/funcs_1/r/is_columns_is.result 2009-10-10 09:59:06 +0000 +++ b/mysql-test/suite/funcs_1/r/is_columns_is.result 2010-01-16 05:12:57 +0000 @@ -127,7 +127,7 @@ NULL information_schema INNODB_BUFFER_PO NULL information_schema INNODB_BUFFER_POOL_PAGES_BLOB page_no 2 0 NO bigint NULL NULL 19 0 NULL NULL bigint(21) unsigned select NULL information_schema INNODB_BUFFER_POOL_PAGES_BLOB part_len 4 0 NO bigint NULL NULL 19 0 NULL NULL bigint(21) unsigned select NULL information_schema INNODB_BUFFER_POOL_PAGES_BLOB space_id 1 0 NO bigint NULL NULL 19 0 NULL NULL bigint(21) unsigned select -NULL information_schema INNODB_BUFFER_POOL_PAGES_INDEX accessed 9 0 NO bigint NULL NULL 19 0 NULL NULL bigint(21) unsigned select +NULL information_schema INNODB_BUFFER_POOL_PAGES_INDEX access_time 9 0 NO bigint NULL NULL 19 0 NULL NULL bigint(21) unsigned select NULL information_schema INNODB_BUFFER_POOL_PAGES_INDEX data_size 7 0 NO bigint NULL NULL 19 0 NULL NULL bigint(21) unsigned select NULL information_schema INNODB_BUFFER_POOL_PAGES_INDEX dirty 11 0 NO bigint NULL NULL 19 0 NULL NULL bigint(21) unsigned select NULL information_schema INNODB_BUFFER_POOL_PAGES_INDEX fix_count 14 0 NO bigint NULL NULL 19 0 NULL NULL bigint(21) unsigned select @@ -394,6 +394,7 @@ NULL information_schema VIEWS TABLE_CATA NULL information_schema VIEWS TABLE_NAME 3 NO varchar 64 192 NULL NULL utf8 utf8_general_ci varchar(64) select NULL information_schema VIEWS TABLE_SCHEMA 2 NO varchar 64 192 NULL NULL utf8 utf8_general_ci varchar(64) select NULL information_schema VIEWS VIEW_DEFINITION 4 NULL NO longtext 4294967295 4294967295 NULL NULL utf8 utf8_general_ci longtext select +NULL information_schema XTRADB_ADMIN_COMMAND result_message 1 NO varchar 1024 3072 NULL NULL utf8 utf8_general_ci varchar(1024) select NULL information_schema XTRADB_ENHANCEMENTS comment 3 NO varchar 100 300 NULL NULL utf8 utf8_general_ci varchar(100) select NULL information_schema XTRADB_ENHANCEMENTS description 2 NO varchar 255 765 NULL NULL utf8 utf8_general_ci varchar(255) select NULL information_schema XTRADB_ENHANCEMENTS link 4 NO varchar 255 765 NULL NULL utf8 utf8_general_ci varchar(255) select @@ -589,7 +590,7 @@ NULL information_schema INNODB_BUFFER_PO NULL information_schema INNODB_BUFFER_POOL_PAGES_INDEX n_recs bigint NULL NULL NULL NULL bigint(21) unsigned NULL information_schema INNODB_BUFFER_POOL_PAGES_INDEX data_size bigint NULL NULL NULL NULL bigint(21) unsigned NULL information_schema INNODB_BUFFER_POOL_PAGES_INDEX hashed bigint NULL NULL NULL NULL bigint(21) unsigned -NULL information_schema INNODB_BUFFER_POOL_PAGES_INDEX accessed bigint NULL NULL NULL NULL bigint(21) unsigned +NULL information_schema INNODB_BUFFER_POOL_PAGES_INDEX access_time bigint NULL NULL NULL NULL bigint(21) unsigned NULL information_schema INNODB_BUFFER_POOL_PAGES_INDEX modified bigint NULL NULL NULL NULL bigint(21) unsigned NULL information_schema INNODB_BUFFER_POOL_PAGES_INDEX dirty bigint NULL NULL NULL NULL bigint(21) unsigned NULL information_schema INNODB_BUFFER_POOL_PAGES_INDEX old bigint NULL NULL NULL NULL bigint(21) unsigned @@ -848,6 +849,7 @@ NULL information_schema TRIGGERS CREATED 3.0000 information_schema VIEWS SECURITY_TYPE varchar 7 21 utf8 utf8_general_ci varchar(7) 3.0000 information_schema VIEWS CHARACTER_SET_CLIENT varchar 32 96 utf8 utf8_general_ci varchar(32) 3.0000 information_schema VIEWS COLLATION_CONNECTION varchar 32 96 utf8 utf8_general_ci varchar(32) +3.0000 information_schema XTRADB_ADMIN_COMMAND result_message varchar 1024 3072 utf8 utf8_general_ci varchar(1024) 3.0000 information_schema XTRADB_ENHANCEMENTS name varchar 255 765 utf8 utf8_general_ci varchar(255) 3.0000 information_schema XTRADB_ENHANCEMENTS description varchar 255 765 utf8 utf8_general_ci varchar(255) 3.0000 information_schema XTRADB_ENHANCEMENTS comment varchar 100 300 utf8 utf8_general_ci varchar(100) === modified file 'mysql-test/suite/funcs_1/r/is_tables_is.result' --- a/mysql-test/suite/funcs_1/r/is_tables_is.result 2009-10-10 09:59:06 +0000 +++ b/mysql-test/suite/funcs_1/r/is_tables_is.result 2010-01-16 05:12:57 +0000 @@ -958,6 +958,29 @@ user_comment Separator ----------------------------------------------------- TABLE_CATALOG NULL TABLE_SCHEMA information_schema +TABLE_NAME XTRADB_ADMIN_COMMAND +TABLE_TYPE SYSTEM VIEW +ENGINE MEMORY +VERSION 10 +ROW_FORMAT Fixed +TABLE_ROWS #TBLR# +AVG_ROW_LENGTH #ARL# +DATA_LENGTH #DL# +MAX_DATA_LENGTH #MDL# +INDEX_LENGTH #IL# +DATA_FREE #DF# +AUTO_INCREMENT NULL +CREATE_TIME #CRT# +UPDATE_TIME #UT# +CHECK_TIME #CT# +TABLE_COLLATION utf8_general_ci +CHECKSUM NULL +CREATE_OPTIONS #CO# +TABLE_COMMENT #TC# +user_comment +Separator ----------------------------------------------------- +TABLE_CATALOG NULL +TABLE_SCHEMA information_schema TABLE_NAME XTRADB_ENHANCEMENTS TABLE_TYPE SYSTEM VIEW ENGINE MEMORY @@ -1941,6 +1964,29 @@ user_comment Separator ----------------------------------------------------- TABLE_CATALOG NULL TABLE_SCHEMA information_schema +TABLE_NAME XTRADB_ADMIN_COMMAND +TABLE_TYPE SYSTEM VIEW +ENGINE MEMORY +VERSION 10 +ROW_FORMAT Fixed +TABLE_ROWS #TBLR# +AVG_ROW_LENGTH #ARL# +DATA_LENGTH #DL# +MAX_DATA_LENGTH #MDL# +INDEX_LENGTH #IL# +DATA_FREE #DF# +AUTO_INCREMENT NULL +CREATE_TIME #CRT# +UPDATE_TIME #UT# +CHECK_TIME #CT# +TABLE_COLLATION utf8_general_ci +CHECKSUM NULL +CREATE_OPTIONS #CO# +TABLE_COMMENT #TC# +user_comment +Separator ----------------------------------------------------- +TABLE_CATALOG NULL +TABLE_SCHEMA information_schema TABLE_NAME XTRADB_ENHANCEMENTS TABLE_TYPE SYSTEM VIEW ENGINE MEMORY === modified file 'mysql-test/suite/innodb/r/innodb-index.result' --- a/mysql-test/suite/innodb/r/innodb-index.result 2009-06-10 13:51:20 +0000 +++ b/mysql-test/suite/innodb/r/innodb-index.result 2009-11-30 12:49:13 +0000 @@ -968,6 +968,7 @@ create index t1u on t1 (u(1)); drop table t1; set global innodb_file_per_table=0; set global innodb_file_format=Antelope; +set global innodb_file_format_check=Antelope; SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0; SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0; CREATE TABLE t1( === added file 'mysql-test/suite/innodb/r/innodb_bug46676.result' --- a/mysql-test/suite/innodb/r/innodb_bug46676.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/innodb/r/innodb_bug46676.result 2009-11-30 12:24:54 +0000 @@ -0,0 +1,9 @@ +SET foreign_key_checks=0; +CREATE TABLE t1 (id int, foreign key (id) references t2(id)) ENGINE=INNODB; +CREATE TABLE t2 (id int, foreign key (id) references t1(id)) ENGINE=INNODB; +SET foreign_key_checks=1; +SELECT COUNT(*) FROM information_schema.key_column_usage WHERE REFERENCED_TABLE_NAME in ('t1', 't2'); +COUNT(*) +2 +SET foreign_key_checks=0; +DROP TABLE t1, t2; === added file 'mysql-test/suite/innodb/r/innodb_bug47167.result' --- a/mysql-test/suite/innodb/r/innodb_bug47167.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/innodb/r/innodb_bug47167.result 2009-11-30 11:56:21 +0000 @@ -0,0 +1,24 @@ +set @old_innodb_file_format_check=@@innodb_file_format_check; +select @old_innodb_file_format_check; +@old_innodb_file_format_check +Antelope +set global innodb_file_format_check = Barracuda; +select @@innodb_file_format_check; +@@innodb_file_format_check +Barracuda +set global innodb_file_format_check = DEFAULT; +select @@innodb_file_format_check; +@@innodb_file_format_check +Barracuda +set global innodb_file_format_check = @old_innodb_file_format_check; +select @@innodb_file_format_check; +@@innodb_file_format_check +Antelope +set global innodb_file_format_check = cheetah; +ERROR HY000: Incorrect arguments to SET +set global innodb_file_format_check = Bear; +ERROR HY000: Incorrect arguments to SET +set global innodb_file_format_check = on; +ERROR HY000: Incorrect arguments to SET +set global innodb_file_format_check = off; +ERROR HY000: Incorrect arguments to SET === modified file 'mysql-test/suite/innodb/t/innodb-consistent-master.opt' --- a/mysql-test/suite/innodb/t/innodb-consistent-master.opt 2009-10-09 13:37:47 +0000 +++ b/mysql-test/suite/innodb/t/innodb-consistent-master.opt 2009-11-30 12:49:13 +0000 @@ -1 +1 @@ ---innodb_lock_wait_timeout=2 +--loose-innodb_lock_wait_timeout=2 === modified file 'mysql-test/suite/innodb/t/innodb-index.test' --- a/mysql-test/suite/innodb/t/innodb-index.test 2009-06-11 12:57:44 +0000 +++ b/mysql-test/suite/innodb/t/innodb-index.test 2009-11-30 12:49:13 +0000 @@ -1,5 +1,7 @@ -- source include/have_innodb.inc +let $innodb_file_format_check_orig=`select @@innodb_file_format_check`; + create table t1(a int not null, b int, c char(10) not null, d varchar(20)) engine = innodb; insert into t1 values (5,5,'oo','oo'),(4,4,'tr','tr'),(3,4,'ad','ad'),(2,3,'ak','ak'); commit; @@ -398,6 +400,7 @@ create index t1u on t1 (u(1)); drop table t1; eval set global innodb_file_per_table=$per_table; eval set global innodb_file_format=$format; +eval set global innodb_file_format_check=$format; # # Test to check whether CREATE INDEX handles implicit foreign key @@ -532,3 +535,10 @@ disconnect a; disconnect b; DROP TABLE t1; + +# +# restore environment to the state it was before this test execution +# + +-- disable_query_log +eval SET GLOBAL innodb_file_format_check=$innodb_file_format_check_orig; === added file 'mysql-test/suite/innodb/t/innodb_bug46676.test' --- a/mysql-test/suite/innodb/t/innodb_bug46676.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/innodb/t/innodb_bug46676.test 2009-11-30 12:24:54 +0000 @@ -0,0 +1,16 @@ +# This is the test for bug 46676: mysqld got exception 0xc0000005 +# It is reproducible with InnoDB plugin 1.0.4 + MySQL 5.1.37. +# But no longer reproducible after MySQL 5.1.38 (with plugin 1.0.5). + +--source include/have_innodb.inc + +SET foreign_key_checks=0; +CREATE TABLE t1 (id int, foreign key (id) references t2(id)) ENGINE=INNODB; +CREATE TABLE t2 (id int, foreign key (id) references t1(id)) ENGINE=INNODB; +SET foreign_key_checks=1; + +# Server crashes +SELECT COUNT(*) FROM information_schema.key_column_usage WHERE REFERENCED_TABLE_NAME in ('t1', 't2'); + +SET foreign_key_checks=0; +DROP TABLE t1, t2; === added file 'mysql-test/suite/innodb/t/innodb_bug47167.test' --- a/mysql-test/suite/innodb/t/innodb_bug47167.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/innodb/t/innodb_bug47167.test 2009-11-30 11:56:21 +0000 @@ -0,0 +1,46 @@ +# This is the unit test for bug *47167. +# It tests setting the global variable +# "innodb_file_format_check" with a +# user-Defined Variable. + +--source include/have_innodb.inc +-- source suite/innodb/include/have_innodb_plugin.inc + +# Save the value (Antelope) in 'innodb_file_format_check' to +# 'old_innodb_file_format_check' +set @old_innodb_file_format_check=@@innodb_file_format_check; + +# @old_innodb_file_format_check shall have the value of 'Antelope' +select @old_innodb_file_format_check; + +# Reset the value in 'innodb_file_format_check' to 'Barracuda' +set global innodb_file_format_check = Barracuda; + +select @@innodb_file_format_check; + +# Set 'innodb_file_format_check' to its default value, which +# is the latest file format supported in the current release. +set global innodb_file_format_check = DEFAULT; + +select @@innodb_file_format_check; + +# Put the saved value back to 'innodb_file_format_check' +set global innodb_file_format_check = @old_innodb_file_format_check; + +# Check whether 'innodb_file_format_check' get its original value. +select @@innodb_file_format_check; + +# Following are negative tests, all should fail. +--disable_warnings +--error ER_WRONG_ARGUMENTS +set global innodb_file_format_check = cheetah; + +--error ER_WRONG_ARGUMENTS +set global innodb_file_format_check = Bear; + +--error ER_WRONG_ARGUMENTS +set global innodb_file_format_check = on; + +--error ER_WRONG_ARGUMENTS +set global innodb_file_format_check = off; +--enable_warnings === modified file 'mysql-test/suite/maria/r/maria-recover.result' --- a/mysql-test/suite/maria/r/maria-recover.result 2009-02-19 09:01:25 +0000 +++ b/mysql-test/suite/maria/r/maria-recover.result 2010-01-27 17:41:05 +0000 @@ -1,6 +1,3 @@ -call mtr.add_suppression("Checking table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Recovering table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Table '.\/mysqltest\/t_corrupted2' is marked as crashed and should be repaired"); select @@global.maria_recover; @@global.maria_recover BACKUP === modified file 'mysql-test/suite/maria/t/maria-recover.test' --- a/mysql-test/suite/maria/t/maria-recover.test 2009-10-09 08:09:24 +0000 +++ b/mysql-test/suite/maria/t/maria-recover.test 2010-01-27 17:41:05 +0000 @@ -2,9 +2,13 @@ --source include/have_maria.inc -call mtr.add_suppression("Checking table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Recovering table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Table '.\/mysqltest\/t_corrupted2' is marked as crashed and should be repaired"); +--disable_query_log +# Note: \\. matches a single period. We use '.' as directory separator to +# account for Unix and Windows variation. +call mtr.add_suppression("Checking table: '\\..mysqltest.t_corrupted2'"); +call mtr.add_suppression("Recovering table: '\\..mysqltest.t_corrupted2'"); +call mtr.add_suppression("Table '\\..mysqltest.t_corrupted2' is marked as crashed and should be repaired"); +--enable_query_log # Note: we're setting an environment variable (not prefixing it by $), # so that the perl code below can access it. === removed file 'mysql-test/suite/maria/t/maria2-master.opt' --- a/mysql-test/suite/maria/t/maria2-master.opt 2009-02-15 10:58:34 +0000 +++ b/mysql-test/suite/maria/t/maria2-master.opt 1970-01-01 00:00:00 +0000 @@ -1,2 +0,0 @@ ---secure-file-priv="" - === modified file 'mysql-test/suite/parts/inc/partition_timestamp.inc' --- a/mysql-test/suite/parts/inc/partition_timestamp.inc 2009-10-28 07:52:34 +0000 +++ b/mysql-test/suite/parts/inc/partition_timestamp.inc 2010-01-27 17:41:05 +0000 @@ -48,6 +48,7 @@ show create table t3; let $count=12; --echo $count inserts; --disable_query_log +SET TIME_ZONE= '+03:00'; begin; while ($count) { === modified file 'mysql-test/suite/parts/r/partition_recover_myisam.result' --- a/mysql-test/suite/parts/r/partition_recover_myisam.result 2009-08-29 21:29:47 +0000 +++ b/mysql-test/suite/parts/r/partition_recover_myisam.result 2010-01-27 17:41:05 +0000 @@ -1,5 +1,3 @@ -call mtr.add_suppression("./test/t1_will_crash"); -call mtr.add_suppression("Got an error from unknown thread, ha_myisam.cc"); CREATE TABLE t1_will_crash (a INT, KEY (a)) ENGINE=MyISAM; INSERT INTO t1_will_crash VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10), (11); FLUSH TABLES; === modified file 'mysql-test/suite/parts/t/partition_alter1_2_innodb.test' --- a/mysql-test/suite/parts/t/partition_alter1_2_innodb.test 2009-10-09 13:08:09 +0000 +++ b/mysql-test/suite/parts/t/partition_alter1_2_innodb.test 2010-01-15 15:27:55 +0000 @@ -28,6 +28,8 @@ #------------------------------------------------------------------------------# # General not engine specific settings and requirements +--source include/big_test.inc + ##### Options, for debugging support ##### let $debug= 0; let $with_partitioning= 1; === modified file 'mysql-test/suite/parts/t/partition_alter2_1_innodb.test' --- a/mysql-test/suite/parts/t/partition_alter2_1_innodb.test 2009-10-09 13:08:09 +0000 +++ b/mysql-test/suite/parts/t/partition_alter2_1_innodb.test 2010-01-15 15:27:55 +0000 @@ -22,6 +22,8 @@ # any of the variables. # +--source include/big_test.inc + #------------------------------------------------------------------------------# # General not engine specific settings and requirements === modified file 'mysql-test/suite/parts/t/partition_alter2_2_innodb.test' --- a/mysql-test/suite/parts/t/partition_alter2_2_innodb.test 2009-10-09 13:08:09 +0000 +++ b/mysql-test/suite/parts/t/partition_alter2_2_innodb.test 2010-01-15 15:27:55 +0000 @@ -22,6 +22,8 @@ # any of the variables. # +--source include/big_test.inc + #------------------------------------------------------------------------------# # General not engine specific settings and requirements === modified file 'mysql-test/suite/parts/t/partition_alter4_innodb.test' --- a/mysql-test/suite/parts/t/partition_alter4_innodb.test 2009-10-09 13:08:09 +0000 +++ b/mysql-test/suite/parts/t/partition_alter4_innodb.test 2010-01-15 15:27:55 +0000 @@ -22,6 +22,8 @@ # any of the variables. # +--source include/big_test.inc + #------------------------------------------------------------------------------# # General not engine specific settings and requirements === modified file 'mysql-test/suite/parts/t/partition_recover_myisam.test' --- a/mysql-test/suite/parts/t/partition_recover_myisam.test 2009-12-03 11:19:05 +0000 +++ b/mysql-test/suite/parts/t/partition_recover_myisam.test 2010-01-27 17:41:05 +0000 @@ -1,7 +1,9 @@ # test the auto-recover (--myisam-recover) of partitioned myisam tables -call mtr.add_suppression("./test/t1_will_crash"); +--disable_query_log +call mtr.add_suppression("..test.t1_will_crash"); call mtr.add_suppression("Got an error from unknown thread, ha_myisam.cc"); +--enable_query_log --source include/have_partition.inc --disable_warnings === modified file 'mysql-test/suite/pbxt/r/func_group.result' --- a/mysql-test/suite/pbxt/r/func_group.result 2009-11-24 10:19:08 +0000 +++ b/mysql-test/suite/pbxt/r/func_group.result 2010-01-16 05:12:57 +0000 @@ -885,7 +885,7 @@ cast(sum(distinct df) as signed) 3 select cast(min(df) as signed) from t1; cast(min(df) as signed) -0 +1 select 1e8 * sum(distinct df) from t1; 1e8 * sum(distinct df) 330000000 === modified file 'mysql-test/suite/pbxt/r/mysqlshow.result' --- a/mysql-test/suite/pbxt/r/mysqlshow.result 2009-08-17 15:57:58 +0000 +++ b/mysql-test/suite/pbxt/r/mysqlshow.result 2010-01-16 05:12:57 +0000 @@ -113,7 +113,7 @@ Database: information_schema | INNODB_RSEG | | XTRADB_ENHANCEMENTS | | INNODB_BUFFER_POOL_PAGES_INDEX | -| INNODB_INDEX_STATS | +| XTRADB_ADMIN_COMMAND | | INNODB_TRX | | INNODB_CMP_RESET | | INNODB_LOCK_WAITS | @@ -122,6 +122,7 @@ Database: information_schema | INNODB_CMPMEM | | INNODB_TABLE_STATS | | INNODB_BUFFER_POOL_PAGES_BLOB | +| INNODB_INDEX_STATS | +---------------------------------------+ Database: INFORMATION_SCHEMA +---------------------------------------+ @@ -161,7 +162,7 @@ Database: INFORMATION_SCHEMA | INNODB_RSEG | | XTRADB_ENHANCEMENTS | | INNODB_BUFFER_POOL_PAGES_INDEX | -| INNODB_INDEX_STATS | +| XTRADB_ADMIN_COMMAND | | INNODB_TRX | | INNODB_CMP_RESET | | INNODB_LOCK_WAITS | @@ -170,6 +171,7 @@ Database: INFORMATION_SCHEMA | INNODB_CMPMEM | | INNODB_TABLE_STATS | | INNODB_BUFFER_POOL_PAGES_BLOB | +| INNODB_INDEX_STATS | +---------------------------------------+ Wildcard: inf_rmation_schema +--------------------+ === modified file 'mysql-test/suite/rpl/r/rpl_err_ignoredtable.result' --- a/mysql-test/suite/rpl/r/rpl_err_ignoredtable.result 2007-06-27 12:28:02 +0000 +++ b/mysql-test/suite/rpl/r/rpl_err_ignoredtable.result 2009-11-18 14:50:31 +0000 @@ -4,6 +4,7 @@ reset master; reset slave; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; start slave; +CALL mtr.add_suppression("Statement may not be safe to log in statement format."); create table t1 (a int primary key); create table t4 (a int primary key); insert into t1 values (1),(1); === modified file 'mysql-test/suite/rpl/r/rpl_extraCol_innodb.result' --- a/mysql-test/suite/rpl/r/rpl_extraCol_innodb.result 2009-08-28 14:13:27 +0000 +++ b/mysql-test/suite/rpl/r/rpl_extraCol_innodb.result 2009-10-22 00:10:42 +0000 @@ -404,7 +404,11 @@ STOP SLAVE; RESET SLAVE; CREATE TABLE t9 (a INT KEY, b BLOB, c CHAR(5), d TIMESTAMP, -e INT NOT NULL) ENGINE='InnoDB'; +e INT NOT NULL, +f text not null, +g text, +h blob not null, +i blob) ENGINE='InnoDB'; *** Create t9 on Master *** CREATE TABLE t9 (a INT PRIMARY KEY, b BLOB, c CHAR(5) ) ENGINE='InnoDB'; @@ -415,47 +419,11 @@ START SLAVE; set @b1 = 'b1b1b1b1'; set @b1 = concat(@b1,@b1); INSERT INTO t9 () VALUES(1,@b1,'Kyle'),(2,@b1,'JOE'),(3,@b1,'QA'); -SHOW SLAVE STATUS; -Slave_IO_State # -Master_Host 127.0.0.1 -Master_User root -Master_Port # -Connect_Retry 1 -Master_Log_File master-bin.000001 -Read_Master_Log_Pos # -Relay_Log_File # -Relay_Log_Pos # -Relay_Master_Log_File master-bin.000001 -Slave_IO_Running Yes -Slave_SQL_Running No -Replicate_Do_DB -Replicate_Ignore_DB -Replicate_Do_Table -Replicate_Ignore_Table # -Replicate_Wild_Do_Table -Replicate_Wild_Ignore_Table -Last_Errno 1364 -Last_Error Could not execute Write_rows event on table test.t9; Field 'e' doesn't have a default value, Error_code: 1364; handler error HA_ERR_ROWS_EVENT_APPLY; the event's master log master-bin.000001, end_log_pos 330 -Skip_Counter 0 -Exec_Master_Log_Pos # -Relay_Log_Space # -Until_Condition None -Until_Log_File -Until_Log_Pos 0 -Master_SSL_Allowed No -Master_SSL_CA_File -Master_SSL_CA_Path -Master_SSL_Cert -Master_SSL_Cipher -Master_SSL_Key -Seconds_Behind_Master # -Master_SSL_Verify_Server_Cert No -Last_IO_Errno # -Last_IO_Error # -Last_SQL_Errno 1364 -Last_SQL_Error Could not execute Write_rows event on table test.t9; Field 'e' doesn't have a default value, Error_code: 1364; handler error HA_ERR_ROWS_EVENT_APPLY; the event's master log master-bin.000001, end_log_pos 330 -SET GLOBAL SQL_SLAVE_SKIP_COUNTER=2; -START SLAVE; +select * from t9; +a b c d e f g h i +1 b1b1b1b1b1b1b1b1 Kyle 0000-00-00 00:00:00 0 NULL NULL +2 b1b1b1b1b1b1b1b1 JOE 0000-00-00 00:00:00 0 NULL NULL +3 b1b1b1b1b1b1b1b1 QA 0000-00-00 00:00:00 0 NULL NULL *** Create t10 on slave *** STOP SLAVE; RESET SLAVE; === modified file 'mysql-test/suite/rpl/r/rpl_extraCol_myisam.result' --- a/mysql-test/suite/rpl/r/rpl_extraCol_myisam.result 2009-08-28 14:13:27 +0000 +++ b/mysql-test/suite/rpl/r/rpl_extraCol_myisam.result 2009-10-22 00:10:42 +0000 @@ -404,7 +404,11 @@ STOP SLAVE; RESET SLAVE; CREATE TABLE t9 (a INT KEY, b BLOB, c CHAR(5), d TIMESTAMP, -e INT NOT NULL) ENGINE='MyISAM'; +e INT NOT NULL, +f text not null, +g text, +h blob not null, +i blob) ENGINE='MyISAM'; *** Create t9 on Master *** CREATE TABLE t9 (a INT PRIMARY KEY, b BLOB, c CHAR(5) ) ENGINE='MyISAM'; @@ -415,47 +419,11 @@ START SLAVE; set @b1 = 'b1b1b1b1'; set @b1 = concat(@b1,@b1); INSERT INTO t9 () VALUES(1,@b1,'Kyle'),(2,@b1,'JOE'),(3,@b1,'QA'); -SHOW SLAVE STATUS; -Slave_IO_State # -Master_Host 127.0.0.1 -Master_User root -Master_Port # -Connect_Retry 1 -Master_Log_File master-bin.000001 -Read_Master_Log_Pos # -Relay_Log_File # -Relay_Log_Pos # -Relay_Master_Log_File master-bin.000001 -Slave_IO_Running Yes -Slave_SQL_Running No -Replicate_Do_DB -Replicate_Ignore_DB -Replicate_Do_Table -Replicate_Ignore_Table # -Replicate_Wild_Do_Table -Replicate_Wild_Ignore_Table -Last_Errno 1364 -Last_Error Could not execute Write_rows event on table test.t9; Field 'e' doesn't have a default value, Error_code: 1364; handler error HA_ERR_ROWS_EVENT_APPLY; the event's master log master-bin.000001, end_log_pos 330 -Skip_Counter 0 -Exec_Master_Log_Pos # -Relay_Log_Space # -Until_Condition None -Until_Log_File -Until_Log_Pos 0 -Master_SSL_Allowed No -Master_SSL_CA_File -Master_SSL_CA_Path -Master_SSL_Cert -Master_SSL_Cipher -Master_SSL_Key -Seconds_Behind_Master # -Master_SSL_Verify_Server_Cert No -Last_IO_Errno # -Last_IO_Error # -Last_SQL_Errno 1364 -Last_SQL_Error Could not execute Write_rows event on table test.t9; Field 'e' doesn't have a default value, Error_code: 1364; handler error HA_ERR_ROWS_EVENT_APPLY; the event's master log master-bin.000001, end_log_pos 330 -SET GLOBAL SQL_SLAVE_SKIP_COUNTER=2; -START SLAVE; +select * from t9; +a b c d e f g h i +1 b1b1b1b1b1b1b1b1 Kyle 0000-00-00 00:00:00 0 NULL NULL +2 b1b1b1b1b1b1b1b1 JOE 0000-00-00 00:00:00 0 NULL NULL +3 b1b1b1b1b1b1b1b1 QA 0000-00-00 00:00:00 0 NULL NULL *** Create t10 on slave *** STOP SLAVE; RESET SLAVE; === modified file 'mysql-test/suite/rpl/r/rpl_get_lock.result' --- a/mysql-test/suite/rpl/r/rpl_get_lock.result 2008-02-12 19:09:16 +0000 +++ b/mysql-test/suite/rpl/r/rpl_get_lock.result 2009-11-18 14:50:31 +0000 @@ -4,6 +4,7 @@ reset master; reset slave; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; start slave; +CALL mtr.add_suppression("Statement may not be safe to log in statement format."); create table t1(n int); insert into t1 values(get_lock("lock",2)); select get_lock("lock",2); === added file 'mysql-test/suite/rpl/r/rpl_loaddata_symlink.result' --- a/mysql-test/suite/rpl/r/rpl_loaddata_symlink.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/r/rpl_loaddata_symlink.result 2009-11-28 04:43:16 +0000 @@ -0,0 +1,17 @@ +stop slave; +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; +reset master; +reset slave; +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; +start slave; +create table t1(a int not null auto_increment, b int, primary key(a) ); +load data infile '../../std_data/rpl_loaddata.dat' into table t1; +select * from t1; +a b +1 10 +2 15 +select * from t1; +a b +1 10 +2 15 +drop table t1; === modified file 'mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result' --- a/mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result 2008-11-04 17:07:14 +0000 +++ b/mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result 2010-01-19 10:36:52 +0000 @@ -18,6 +18,7 @@ Server_id Host Port Rpl_recovery_rank Ma drop table t1; stop slave; create table t2(id int auto_increment primary key, created datetime); +SET TIME_ZONE= '+03:00'; set timestamp=12345; insert into t2 set created=now(); select * from t2; === added file 'mysql-test/suite/rpl/r/rpl_nondeterministic_functions.result' --- a/mysql-test/suite/rpl/r/rpl_nondeterministic_functions.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/r/rpl_nondeterministic_functions.result 2009-11-18 14:50:31 +0000 @@ -0,0 +1,26 @@ +stop slave; +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; +reset master; +reset slave; +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; +start slave; +CREATE TABLE t1 (a VARCHAR(1000)); +INSERT INTO t1 VALUES (CONNECTION_ID()); +INSERT INTO t1 VALUES (CONNECTION_ID()); +INSERT INTO t1 VALUES +(CURDATE()), +(CURRENT_DATE()), +(CURRENT_TIME()), +(CURRENT_TIMESTAMP()), +(CURTIME()), +(LOCALTIME()), +(LOCALTIMESTAMP()), +(NOW()), +(UNIX_TIMESTAMP()), +(UTC_DATE()), +(UTC_TIME()), +(UTC_TIMESTAMP()); +INSERT INTO t1 VALUES (RAND()); +INSERT INTO t1 VALUES (LAST_INSERT_ID()); +Comparing tables master:test.t1 and slave:test.t1 +DROP TABLE t1; === added file 'mysql-test/suite/rpl/r/rpl_not_null_innodb.result' --- a/mysql-test/suite/rpl/r/rpl_not_null_innodb.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/r/rpl_not_null_innodb.result 2009-10-22 00:19:52 +0000 @@ -0,0 +1,202 @@ +stop slave; +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; +reset master; +reset slave; +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; +start slave; +SET SQL_LOG_BIN= 0; +CREATE TABLE t1(`a` INT, `b` DATE DEFAULT NULL, +`c` INT DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +CREATE TABLE t2(`a` INT, `b` DATE DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +CREATE TABLE t3(`a` INT, `b` DATE DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +CREATE TABLE t4(`a` INT, `b` DATE DEFAULT NULL, +`c` INT DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +SET SQL_LOG_BIN= 1; +CREATE TABLE t1(`a` INT, `b` DATE DEFAULT NULL, +`c` INT DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +CREATE TABLE t2(`a` INT, `b` DATE DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +CREATE TABLE t3(`a` INT, `b` DATE DEFAULT '0000-00-00', +`c` INT DEFAULT 500, +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +CREATE TABLE t4(`a` INT, `b` DATE DEFAULT '0000-00-00', +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +************* EXECUTION WITH INSERTS ************* +INSERT INTO t1(a,b,c) VALUES (1, null, 1); +INSERT INTO t1(a,b,c) VALUES (2,'1111-11-11', 2); +INSERT INTO t1(a,b) VALUES (3, null); +INSERT INTO t1(a,c) VALUES (4, 4); +INSERT INTO t1(a) VALUES (5); +INSERT INTO t2(a,b) VALUES (1, null); +INSERT INTO t2(a,b) VALUES (2,'1111-11-11'); +INSERT INTO t2(a) VALUES (3); +INSERT INTO t3(a,b) VALUES (1, null); +INSERT INTO t3(a,b) VALUES (2,'1111-11-11'); +INSERT INTO t3(a) VALUES (3); +INSERT INTO t4(a,b,c) VALUES (1, null, 1); +INSERT INTO t4(a,b,c) VALUES (2,'1111-11-11', 2); +INSERT INTO t4(a,b) VALUES (3, null); +INSERT INTO t4(a,c) VALUES (4, 4); +INSERT INTO t4(a) VALUES (5); +************* SHOWING THE RESULT SETS WITH INSERTS ************* +TABLES t1 and t2 must be equal otherwise an error will be thrown. +Comparing tables master:test.t1 and slave:test.t1 +Comparing tables master:test.t2 and slave:test.t2 +TABLES t2 and t3 must be different. +SELECT * FROM t3 ORDER BY a; +a b +1 NULL +2 1111-11-11 +3 NULL +SELECT * FROM t3 ORDER BY a; +a b c +1 NULL 500 +2 1111-11-11 500 +3 NULL 500 +SELECT * FROM t4 ORDER BY a; +a b c +1 NULL 1 +2 1111-11-11 2 +3 NULL NULL +4 NULL 4 +5 NULL NULL +SELECT * FROM t4 ORDER BY a; +a b +1 NULL +2 1111-11-11 +3 NULL +4 NULL +5 NULL +************* EXECUTION WITH UPDATES and REPLACES ************* +DELETE FROM t1; +INSERT INTO t1(a,b,c) VALUES (1,'1111-11-11', 1); +REPLACE INTO t1(a,b,c) VALUES (2,'1111-11-11', 2); +UPDATE t1 set b= NULL, c= 300 where a= 1; +REPLACE INTO t1(a,b,c) VALUES (2, NULL, 300); +************* SHOWING THE RESULT SETS WITH UPDATES and REPLACES ************* +TABLES t1 and t2 must be equal otherwise an error will be thrown. +Comparing tables master:test.t1 and slave:test.t1 +************* CLEANING ************* +DROP TABLE t1; +DROP TABLE t2; +DROP TABLE t3; +DROP TABLE t4; +SET SQL_LOG_BIN= 0; +CREATE TABLE t1 (`a` INT, `b` BIT DEFAULT NULL, `c` BIT DEFAULT NULL, +PRIMARY KEY (`a`)) ENGINE= Innodb; +SET SQL_LOG_BIN= 1; +CREATE TABLE t1 (`a` INT, `b` BIT DEFAULT b'01', `c` BIT DEFAULT NULL, +PRIMARY KEY (`a`)) ENGINE= Innodb; +************* EXECUTION WITH INSERTS ************* +INSERT INTO t1(a,b,c) VALUES (1, null, b'01'); +INSERT INTO t1(a,b,c) VALUES (2,b'00', b'01'); +INSERT INTO t1(a,b) VALUES (3, null); +INSERT INTO t1(a,c) VALUES (4, b'01'); +INSERT INTO t1(a) VALUES (5); +************* SHOWING THE RESULT SETS WITH INSERTS ************* +TABLES t1 and t2 must be different. +SELECT a,b+0,c+0 FROM t1 ORDER BY a; +a b+0 c+0 +1 NULL 1 +2 0 1 +3 NULL NULL +4 NULL 1 +5 NULL NULL +SELECT a,b+0,c+0 FROM t1 ORDER BY a; +a b+0 c+0 +1 NULL 1 +2 0 1 +3 NULL NULL +4 NULL 1 +5 NULL NULL +************* EXECUTION WITH UPDATES and REPLACES ************* +DELETE FROM t1; +INSERT INTO t1(a,b,c) VALUES (1,b'00', b'01'); +REPLACE INTO t1(a,b,c) VALUES (2,b'00',b'01'); +UPDATE t1 set b= NULL, c= b'00' where a= 1; +REPLACE INTO t1(a,b,c) VALUES (2, NULL, b'00'); +************* SHOWING THE RESULT SETS WITH UPDATES and REPLACES ************* +TABLES t1 and t2 must be equal otherwise an error will be thrown. +Comparing tables master:test.t1 and slave:test.t1 +DROP TABLE t1; +################################################################################ +# NULL ---> NOT NULL (STRICT MODE) +# UNCOMMENT THIS AFTER FIXING BUG#43992 +################################################################################ +################################################################################ +# NULL ---> NOT NULL (NON-STRICT MODE) +################################################################################ +SET SQL_LOG_BIN= 0; +CREATE TABLE t1(`a` INT NOT NULL, `b` INT, +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +CREATE TABLE t2(`a` INT NOT NULL, `b` INT, +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +CREATE TABLE t3(`a` INT NOT NULL, `b` INT, +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +SET SQL_LOG_BIN= 1; +CREATE TABLE t1(`a` INT NOT NULL, `b` INT NOT NULL, +`c` INT NOT NULL, +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +CREATE TABLE t2(`a` INT NOT NULL, `b` INT NOT NULL, +`c` INT, +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +CREATE TABLE t3(`a` INT NOT NULL, `b` INT NOT NULL, +`c` INT DEFAULT 500, +PRIMARY KEY(`a`)) ENGINE=Innodb DEFAULT CHARSET=LATIN1; +************* EXECUTION WITH INSERTS ************* +INSERT INTO t1(a) VALUES (1); +INSERT INTO t1(a, b) VALUES (2, NULL); +INSERT INTO t1(a, b) VALUES (3, 1); +INSERT INTO t2(a) VALUES (1); +INSERT INTO t2(a, b) VALUES (2, NULL); +INSERT INTO t2(a, b) VALUES (3, 1); +INSERT INTO t3(a) VALUES (1); +INSERT INTO t3(a, b) VALUES (2, NULL); +INSERT INTO t3(a, b) VALUES (3, 1); +INSERT INTO t3(a, b) VALUES (4, 1); +REPLACE INTO t3(a, b) VALUES (5, null); +REPLACE INTO t3(a, b) VALUES (3, null); +UPDATE t3 SET b = NULL where a = 4; +************* SHOWING THE RESULT SETS ************* +SELECT * FROM t1 ORDER BY a; +a b +1 NULL +2 NULL +3 1 +SELECT * FROM t1 ORDER BY a; +a b c +1 0 0 +2 0 0 +3 1 0 +SELECT * FROM t2 ORDER BY a; +a b +1 NULL +2 NULL +3 1 +SELECT * FROM t2 ORDER BY a; +a b c +1 0 NULL +2 0 NULL +3 1 NULL +SELECT * FROM t3 ORDER BY a; +a b +1 NULL +2 NULL +3 NULL +4 NULL +5 NULL +SELECT * FROM t3 ORDER BY a; +a b c +1 0 500 +2 0 500 +3 0 500 +4 0 500 +5 0 500 +DROP TABLE t1; +DROP TABLE t2; +DROP TABLE t3; === added file 'mysql-test/suite/rpl/r/rpl_not_null_myisam.result' --- a/mysql-test/suite/rpl/r/rpl_not_null_myisam.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/r/rpl_not_null_myisam.result 2009-10-22 00:19:52 +0000 @@ -0,0 +1,202 @@ +stop slave; +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; +reset master; +reset slave; +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; +start slave; +SET SQL_LOG_BIN= 0; +CREATE TABLE t1(`a` INT, `b` DATE DEFAULT NULL, +`c` INT DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +CREATE TABLE t2(`a` INT, `b` DATE DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +CREATE TABLE t3(`a` INT, `b` DATE DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +CREATE TABLE t4(`a` INT, `b` DATE DEFAULT NULL, +`c` INT DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +SET SQL_LOG_BIN= 1; +CREATE TABLE t1(`a` INT, `b` DATE DEFAULT NULL, +`c` INT DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +CREATE TABLE t2(`a` INT, `b` DATE DEFAULT NULL, +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +CREATE TABLE t3(`a` INT, `b` DATE DEFAULT '0000-00-00', +`c` INT DEFAULT 500, +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +CREATE TABLE t4(`a` INT, `b` DATE DEFAULT '0000-00-00', +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +************* EXECUTION WITH INSERTS ************* +INSERT INTO t1(a,b,c) VALUES (1, null, 1); +INSERT INTO t1(a,b,c) VALUES (2,'1111-11-11', 2); +INSERT INTO t1(a,b) VALUES (3, null); +INSERT INTO t1(a,c) VALUES (4, 4); +INSERT INTO t1(a) VALUES (5); +INSERT INTO t2(a,b) VALUES (1, null); +INSERT INTO t2(a,b) VALUES (2,'1111-11-11'); +INSERT INTO t2(a) VALUES (3); +INSERT INTO t3(a,b) VALUES (1, null); +INSERT INTO t3(a,b) VALUES (2,'1111-11-11'); +INSERT INTO t3(a) VALUES (3); +INSERT INTO t4(a,b,c) VALUES (1, null, 1); +INSERT INTO t4(a,b,c) VALUES (2,'1111-11-11', 2); +INSERT INTO t4(a,b) VALUES (3, null); +INSERT INTO t4(a,c) VALUES (4, 4); +INSERT INTO t4(a) VALUES (5); +************* SHOWING THE RESULT SETS WITH INSERTS ************* +TABLES t1 and t2 must be equal otherwise an error will be thrown. +Comparing tables master:test.t1 and slave:test.t1 +Comparing tables master:test.t2 and slave:test.t2 +TABLES t2 and t3 must be different. +SELECT * FROM t3 ORDER BY a; +a b +1 NULL +2 1111-11-11 +3 NULL +SELECT * FROM t3 ORDER BY a; +a b c +1 NULL 500 +2 1111-11-11 500 +3 NULL 500 +SELECT * FROM t4 ORDER BY a; +a b c +1 NULL 1 +2 1111-11-11 2 +3 NULL NULL +4 NULL 4 +5 NULL NULL +SELECT * FROM t4 ORDER BY a; +a b +1 NULL +2 1111-11-11 +3 NULL +4 NULL +5 NULL +************* EXECUTION WITH UPDATES and REPLACES ************* +DELETE FROM t1; +INSERT INTO t1(a,b,c) VALUES (1,'1111-11-11', 1); +REPLACE INTO t1(a,b,c) VALUES (2,'1111-11-11', 2); +UPDATE t1 set b= NULL, c= 300 where a= 1; +REPLACE INTO t1(a,b,c) VALUES (2, NULL, 300); +************* SHOWING THE RESULT SETS WITH UPDATES and REPLACES ************* +TABLES t1 and t2 must be equal otherwise an error will be thrown. +Comparing tables master:test.t1 and slave:test.t1 +************* CLEANING ************* +DROP TABLE t1; +DROP TABLE t2; +DROP TABLE t3; +DROP TABLE t4; +SET SQL_LOG_BIN= 0; +CREATE TABLE t1 (`a` INT, `b` BIT DEFAULT NULL, `c` BIT DEFAULT NULL, +PRIMARY KEY (`a`)) ENGINE= MyISAM; +SET SQL_LOG_BIN= 1; +CREATE TABLE t1 (`a` INT, `b` BIT DEFAULT b'01', `c` BIT DEFAULT NULL, +PRIMARY KEY (`a`)) ENGINE= MyISAM; +************* EXECUTION WITH INSERTS ************* +INSERT INTO t1(a,b,c) VALUES (1, null, b'01'); +INSERT INTO t1(a,b,c) VALUES (2,b'00', b'01'); +INSERT INTO t1(a,b) VALUES (3, null); +INSERT INTO t1(a,c) VALUES (4, b'01'); +INSERT INTO t1(a) VALUES (5); +************* SHOWING THE RESULT SETS WITH INSERTS ************* +TABLES t1 and t2 must be different. +SELECT a,b+0,c+0 FROM t1 ORDER BY a; +a b+0 c+0 +1 NULL 1 +2 0 1 +3 NULL NULL +4 NULL 1 +5 NULL NULL +SELECT a,b+0,c+0 FROM t1 ORDER BY a; +a b+0 c+0 +1 NULL 1 +2 0 1 +3 NULL NULL +4 NULL 1 +5 NULL NULL +************* EXECUTION WITH UPDATES and REPLACES ************* +DELETE FROM t1; +INSERT INTO t1(a,b,c) VALUES (1,b'00', b'01'); +REPLACE INTO t1(a,b,c) VALUES (2,b'00',b'01'); +UPDATE t1 set b= NULL, c= b'00' where a= 1; +REPLACE INTO t1(a,b,c) VALUES (2, NULL, b'00'); +************* SHOWING THE RESULT SETS WITH UPDATES and REPLACES ************* +TABLES t1 and t2 must be equal otherwise an error will be thrown. +Comparing tables master:test.t1 and slave:test.t1 +DROP TABLE t1; +################################################################################ +# NULL ---> NOT NULL (STRICT MODE) +# UNCOMMENT THIS AFTER FIXING BUG#43992 +################################################################################ +################################################################################ +# NULL ---> NOT NULL (NON-STRICT MODE) +################################################################################ +SET SQL_LOG_BIN= 0; +CREATE TABLE t1(`a` INT NOT NULL, `b` INT, +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +CREATE TABLE t2(`a` INT NOT NULL, `b` INT, +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +CREATE TABLE t3(`a` INT NOT NULL, `b` INT, +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +SET SQL_LOG_BIN= 1; +CREATE TABLE t1(`a` INT NOT NULL, `b` INT NOT NULL, +`c` INT NOT NULL, +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +CREATE TABLE t2(`a` INT NOT NULL, `b` INT NOT NULL, +`c` INT, +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +CREATE TABLE t3(`a` INT NOT NULL, `b` INT NOT NULL, +`c` INT DEFAULT 500, +PRIMARY KEY(`a`)) ENGINE=MyISAM DEFAULT CHARSET=LATIN1; +************* EXECUTION WITH INSERTS ************* +INSERT INTO t1(a) VALUES (1); +INSERT INTO t1(a, b) VALUES (2, NULL); +INSERT INTO t1(a, b) VALUES (3, 1); +INSERT INTO t2(a) VALUES (1); +INSERT INTO t2(a, b) VALUES (2, NULL); +INSERT INTO t2(a, b) VALUES (3, 1); +INSERT INTO t3(a) VALUES (1); +INSERT INTO t3(a, b) VALUES (2, NULL); +INSERT INTO t3(a, b) VALUES (3, 1); +INSERT INTO t3(a, b) VALUES (4, 1); +REPLACE INTO t3(a, b) VALUES (5, null); +REPLACE INTO t3(a, b) VALUES (3, null); +UPDATE t3 SET b = NULL where a = 4; +************* SHOWING THE RESULT SETS ************* +SELECT * FROM t1 ORDER BY a; +a b +1 NULL +2 NULL +3 1 +SELECT * FROM t1 ORDER BY a; +a b c +1 0 0 +2 0 0 +3 1 0 +SELECT * FROM t2 ORDER BY a; +a b +1 NULL +2 NULL +3 1 +SELECT * FROM t2 ORDER BY a; +a b c +1 0 NULL +2 0 NULL +3 1 NULL +SELECT * FROM t3 ORDER BY a; +a b +1 NULL +2 NULL +3 NULL +4 NULL +5 NULL +SELECT * FROM t3 ORDER BY a; +a b c +1 0 500 +2 0 500 +3 0 500 +4 0 500 +5 0 500 +DROP TABLE t1; +DROP TABLE t2; +DROP TABLE t3; === modified file 'mysql-test/suite/rpl/r/rpl_relayspace.result' --- a/mysql-test/suite/rpl/r/rpl_relayspace.result 2007-06-27 12:28:02 +0000 +++ b/mysql-test/suite/rpl/r/rpl_relayspace.result 2010-01-18 12:56:10 +0000 @@ -14,6 +14,6 @@ start slave io_thread; stop slave io_thread; reset slave; start slave; -select master_pos_wait('master-bin.001',200,6)=-1; -master_pos_wait('master-bin.001',200,6)=-1 +select master_pos_wait('MASTER_LOG_FILE',200,30)=-1; +master_pos_wait('MASTER_LOG_FILE',200,30)=-1 0 === modified file 'mysql-test/suite/rpl/r/rpl_row_create_table.result' --- a/mysql-test/suite/rpl/r/rpl_row_create_table.result 2009-10-06 00:54:00 +0000 +++ b/mysql-test/suite/rpl/r/rpl_row_create_table.result 2009-11-27 13:34:39 +0000 @@ -476,4 +476,30 @@ master-bin.000001 # Table_map # # table_ master-bin.000001 # Write_rows # # table_id: # flags: STMT_END_F master-bin.000001 # Query # # COMMIT DROP DATABASE mysqltest1; +stop slave; +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; +reset master; +reset slave; +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; +start slave; +CREATE TEMPORARY TABLE t7(c1 INT); +CREATE TABLE t5(c1 INT); +CREATE TABLE t4(c1 INT); +CREATE VIEW bug48506_t1 AS SELECT 1; +CREATE VIEW bug48506_t2 AS SELECT * FROM t4; +CREATE VIEW bug48506_t3 AS SELECT t5.c1 AS A, t4.c1 AS B FROM t5, t4; +CREATE TABLE bug48506_t4(c1 INT); +DROP VIEW bug48506_t1, bug48506_t2, bug48506_t3; +DROP TABLE bug48506_t4; +CREATE TABLE IF NOT EXISTS bug48506_t1 LIKE t7; +CREATE TABLE IF NOT EXISTS bug48506_t2 LIKE t7; +CREATE TABLE IF NOT EXISTS bug48506_t3 LIKE t7; +CREATE TABLE IF NOT EXISTS bug48506_t4 LIKE t7; +SHOW TABLES LIKE 'bug48506%'; +Tables_in_test (bug48506%) +bug48506_t4 +DROP VIEW IF EXISTS bug48506_t1, bug48506_t2, bug48506_t3; +DROP TEMPORARY TABLES t7; +DROP TABLES t4, t5; +DROP TABLES IF EXISTS bug48506_t4; end of the tests === modified file 'mysql-test/suite/rpl/r/rpl_row_tabledefs_2myisam.result' --- a/mysql-test/suite/rpl/r/rpl_row_tabledefs_2myisam.result 2008-03-14 20:02:52 +0000 +++ b/mysql-test/suite/rpl/r/rpl_row_tabledefs_2myisam.result 2009-10-22 00:10:42 +0000 @@ -105,47 +105,9 @@ a b x 2 10 Foo is a bar INSERT INTO t9 VALUES (2); INSERT INTO t1_nodef VALUES (1,2); -SHOW SLAVE STATUS; -Slave_IO_State # -Master_Host 127.0.0.1 -Master_User root -Master_Port # -Connect_Retry 1 -Master_Log_File master-bin.000001 -Read_Master_Log_Pos # -Relay_Log_File # -Relay_Log_Pos # -Relay_Master_Log_File master-bin.000001 -Slave_IO_Running Yes -Slave_SQL_Running No -Replicate_Do_DB -Replicate_Ignore_DB -Replicate_Do_Table -Replicate_Ignore_Table -Replicate_Wild_Do_Table -Replicate_Wild_Ignore_Table -Last_Errno 1364 -Last_Error <Last_Error> -Skip_Counter 0 -Exec_Master_Log_Pos # -Relay_Log_Space # -Until_Condition None -Until_Log_File -Until_Log_Pos 0 -Master_SSL_Allowed No -Master_SSL_CA_File -Master_SSL_CA_Path -Master_SSL_Cert -Master_SSL_Cipher -Master_SSL_Key -Seconds_Behind_Master # -Master_SSL_Verify_Server_Cert No -Last_IO_Errno <Last_IO_Errno> -Last_IO_Error <Last_IO_Error> -Last_SQL_Errno 1364 -Last_SQL_Error <Last_SQL_Error> -SET GLOBAL SQL_SLAVE_SKIP_COUNTER=2; -START SLAVE; +select count(*) from t1_nodef; +count(*) +1 INSERT INTO t9 VALUES (2); **** On Master **** INSERT INTO t2 VALUES (2,4); === modified file 'mysql-test/suite/rpl/r/rpl_row_tabledefs_3innodb.result' --- a/mysql-test/suite/rpl/r/rpl_row_tabledefs_3innodb.result 2008-03-14 20:02:52 +0000 +++ b/mysql-test/suite/rpl/r/rpl_row_tabledefs_3innodb.result 2009-10-22 00:10:42 +0000 @@ -105,47 +105,9 @@ a b x 2 10 Foo is a bar INSERT INTO t9 VALUES (2); INSERT INTO t1_nodef VALUES (1,2); -SHOW SLAVE STATUS; -Slave_IO_State # -Master_Host 127.0.0.1 -Master_User root -Master_Port # -Connect_Retry 1 -Master_Log_File master-bin.000001 -Read_Master_Log_Pos # -Relay_Log_File # -Relay_Log_Pos # -Relay_Master_Log_File master-bin.000001 -Slave_IO_Running Yes -Slave_SQL_Running No -Replicate_Do_DB -Replicate_Ignore_DB -Replicate_Do_Table -Replicate_Ignore_Table -Replicate_Wild_Do_Table -Replicate_Wild_Ignore_Table -Last_Errno 1364 -Last_Error <Last_Error> -Skip_Counter 0 -Exec_Master_Log_Pos # -Relay_Log_Space # -Until_Condition None -Until_Log_File -Until_Log_Pos 0 -Master_SSL_Allowed No -Master_SSL_CA_File -Master_SSL_CA_Path -Master_SSL_Cert -Master_SSL_Cipher -Master_SSL_Key -Seconds_Behind_Master # -Master_SSL_Verify_Server_Cert No -Last_IO_Errno <Last_IO_Errno> -Last_IO_Error <Last_IO_Error> -Last_SQL_Errno 1364 -Last_SQL_Error <Last_SQL_Error> -SET GLOBAL SQL_SLAVE_SKIP_COUNTER=2; -START SLAVE; +select count(*) from t1_nodef; +count(*) +1 INSERT INTO t9 VALUES (2); **** On Master **** INSERT INTO t2 VALUES (2,4); === added file 'mysql-test/suite/rpl/r/rpl_row_trunc_temp.result' --- a/mysql-test/suite/rpl/r/rpl_row_trunc_temp.result 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/r/rpl_row_trunc_temp.result 2009-11-22 05:10:33 +0000 @@ -0,0 +1,29 @@ +stop slave; +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; +reset master; +reset slave; +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; +start slave; +CREATE TEMPORARY TABLE t1(c1 INTEGER); +CREATE TABLE t2(c1 INTEGER); +CREATE TABLE t1(c1 INTEGER); +INSERT INTO t1 VALUES(1), (2); +INSERT INTO t2 VALUES(1), (2); +SELECT * FROM t1; +c1 +1 +2 +SELECT * FROM t2; +c1 +1 +2 +TRUNCATE t1; +TRUNCATE t2; +SELECT * FROM t1; +c1 +1 +2 +SELECT * FROM t2; +c1 +DROP TABLE t1; +DROP TABLE t2; === modified file 'mysql-test/suite/rpl/r/rpl_stm_000001.result' --- a/mysql-test/suite/rpl/r/rpl_stm_000001.result 2007-12-12 17:19:24 +0000 +++ b/mysql-test/suite/rpl/r/rpl_stm_000001.result 2009-11-18 14:50:31 +0000 @@ -4,6 +4,7 @@ reset master; reset slave; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; start slave; +CALL mtr.add_suppression("Statement may not be safe to log in statement format."); create table t1 (word char(20) not null); load data infile '../../std_data/words.dat' into table t1; load data local infile 'MYSQL_TEST_DIR/std_data/words.dat' into table t1; === modified file 'mysql-test/suite/rpl/r/rpl_temporary.result' --- a/mysql-test/suite/rpl/r/rpl_temporary.result 2009-05-22 23:29:41 +0000 +++ b/mysql-test/suite/rpl/r/rpl_temporary.result 2010-01-11 13:15:28 +0000 @@ -4,7 +4,8 @@ reset master; reset slave; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; start slave; -call mtr.add_suppression("Slave: Can\'t find record in \'user\' Error_code: 1032"); +SET sql_log_bin = 0; +SET sql_log_bin = 1; reset master; DROP TABLE IF EXISTS t1; CREATE TEMPORARY TABLE t1 (a char(1)); @@ -127,6 +128,8 @@ select * from t1; a 1 drop table t1; +SET sql_log_bin = 0; +SET sql_log_bin = 1; -- Bug#43748 -- make a user on the slave that can list but not kill system threads. FLUSH PRIVILEGES; === modified file 'mysql-test/suite/rpl/r/rpl_trigger.result' --- a/mysql-test/suite/rpl/r/rpl_trigger.result 2009-08-03 09:47:45 +0000 +++ b/mysql-test/suite/rpl/r/rpl_trigger.result 2009-11-18 14:50:31 +0000 @@ -4,6 +4,7 @@ reset master; reset slave; drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9; start slave; +CALL mtr.add_suppression("Statement may not be safe to log in statement format."); DROP TABLE IF EXISTS t1; DROP TABLE IF EXISTS t2; DROP TABLE IF EXISTS t3; === modified file 'mysql-test/suite/rpl/t/disabled.def' --- a/mysql-test/suite/rpl/t/disabled.def 2009-09-27 10:12:58 +0000 +++ b/mysql-test/suite/rpl/t/disabled.def 2009-12-01 09:21:15 +0000 @@ -10,3 +10,4 @@ # ############################################################################## +rpl_row_create_table : Bug#45576 2009-12-01 joro rpl_row_create_table fails on PB2 === modified file 'mysql-test/suite/rpl/t/rpl_err_ignoredtable.test' --- a/mysql-test/suite/rpl/t/rpl_err_ignoredtable.test 2009-10-20 18:00:07 +0000 +++ b/mysql-test/suite/rpl/t/rpl_err_ignoredtable.test 2009-11-18 14:50:31 +0000 @@ -7,6 +7,8 @@ -- source include/master-slave.inc +CALL mtr.add_suppression("Statement may not be safe to log in statement format."); + connection master; create table t1 (a int primary key); create table t4 (a int primary key); @@ -14,19 +16,15 @@ create table t4 (a int primary key); --error 1022, ER_DUP_ENTRY insert into t1 values (1),(1); insert into t4 values (1),(2); -save_master_pos; -connection slave; # as the t1 table is ignored on the slave, the slave should be able to sync -sync_with_master; +sync_slave_with_master; # check that the table has been ignored, because otherwise the test is nonsense show tables like 't1'; show tables like 't4'; SELECT * FROM test.t4 ORDER BY a; connection master; drop table t1; -save_master_pos; -connection slave; -sync_with_master; +sync_slave_with_master; # Now test that even critical errors (connection killed) # are ignored if rules allow it. @@ -50,18 +48,17 @@ kill @id; drop table t2,t3; insert into t4 values (3),(4); connection master; +# The get_lock function causes warning for unsafe statement. +--disable_warnings --error 0,1317,2013 reap; +--enable_warnings connection master1; -save_master_pos; -connection slave; -sync_with_master; +sync_slave_with_master; SELECT * FROM test.t4 ORDER BY a; connection master1; DROP TABLE test.t4; -save_master_pos; -connection slave; -sync_with_master; +sync_slave_with_master; # End of 4.1 tests # Adding comment for force manual merge 5.0 -> wl1012. delete me if needed === modified file 'mysql-test/suite/rpl/t/rpl_get_lock.test' --- a/mysql-test/suite/rpl/t/rpl_get_lock.test 2007-06-27 12:28:02 +0000 +++ b/mysql-test/suite/rpl/t/rpl_get_lock.test 2009-11-18 14:50:31 +0000 @@ -1,7 +1,12 @@ source include/master-slave.inc; +CALL mtr.add_suppression("Statement may not be safe to log in statement format."); + create table t1(n int); +# Use of get_lock gives a warning for unsafeness if binlog_format=statement +--disable_warnings insert into t1 values(get_lock("lock",2)); +--enable_warnings dirty_close master; connection master1; select get_lock("lock",2); === added file 'mysql-test/suite/rpl/t/rpl_loaddata_symlink-master.opt' --- a/mysql-test/suite/rpl/t/rpl_loaddata_symlink-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/t/rpl_loaddata_symlink-master.opt 2009-11-28 04:43:16 +0000 @@ -0,0 +1 @@ +--secure-file-priv=$MYSQLTEST_VARDIR/std_data_master_link === added file 'mysql-test/suite/rpl/t/rpl_loaddata_symlink-master.sh' --- a/mysql-test/suite/rpl/t/rpl_loaddata_symlink-master.sh 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/t/rpl_loaddata_symlink-master.sh 2009-11-28 04:43:16 +0000 @@ -0,0 +1 @@ +ln -s $MYSQLTEST_VARDIR/std_data $MYSQLTEST_VARDIR/std_data_master_link === added file 'mysql-test/suite/rpl/t/rpl_loaddata_symlink-slave.opt' --- a/mysql-test/suite/rpl/t/rpl_loaddata_symlink-slave.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/t/rpl_loaddata_symlink-slave.opt 2009-11-28 04:43:16 +0000 @@ -0,0 +1 @@ +--slave-load-tmpdir=$MYSQLTEST_VARDIR/std_data_slave_link === added file 'mysql-test/suite/rpl/t/rpl_loaddata_symlink-slave.sh' --- a/mysql-test/suite/rpl/t/rpl_loaddata_symlink-slave.sh 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/t/rpl_loaddata_symlink-slave.sh 2009-11-28 04:43:16 +0000 @@ -0,0 +1 @@ +ln -s $MYSQLTEST_VARDIR/std_data $MYSQLTEST_VARDIR/std_data_slave_link === added file 'mysql-test/suite/rpl/t/rpl_loaddata_symlink.test' --- a/mysql-test/suite/rpl/t/rpl_loaddata_symlink.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/t/rpl_loaddata_symlink.test 2009-11-28 04:43:16 +0000 @@ -0,0 +1,20 @@ +# +# BUG#43913 +# This test verifies if loading data infile will work fine +# if the path of the load data file is a symbolic link. +# +--source include/master-slave.inc +--source include/have_binlog_format_statement.inc + +create table t1(a int not null auto_increment, b int, primary key(a) ); +load data infile '../../std_data/rpl_loaddata.dat' into table t1; +select * from t1; + +sync_slave_with_master; +connection slave; +select * from t1; + +connection master; +drop table t1; +sync_slave_with_master; + === modified file 'mysql-test/suite/rpl/t/rpl_mixed_ddl_dml.test' --- a/mysql-test/suite/rpl/t/rpl_mixed_ddl_dml.test 2007-06-27 12:28:02 +0000 +++ b/mysql-test/suite/rpl/t/rpl_mixed_ddl_dml.test 2010-01-19 10:36:52 +0000 @@ -23,6 +23,7 @@ connection master; # Test replication of timestamp create table t2(id int auto_increment primary key, created datetime); +SET TIME_ZONE= '+03:00'; set timestamp=12345; insert into t2 set created=now(); select * from t2; === added file 'mysql-test/suite/rpl/t/rpl_nondeterministic_functions.test' --- a/mysql-test/suite/rpl/t/rpl_nondeterministic_functions.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/t/rpl_nondeterministic_functions.test 2009-11-18 14:50:31 +0000 @@ -0,0 +1,53 @@ +# ==== Purpose ==== +# +# Test that nondeterministic system functions are correctly replicated. +# +# (Some functions are only correctly replicated if binlog_format=MIXED +# or ROW. See binlog_unsafe.test for a test that those variables are +# indeed unsafe.) +# +# ==== Implementation ==== +# +# We insert the values of each unsafe function into a table. Then we +# replicate and check that the table is identical on slave. +# +# ==== Related bugs ==== +# +# BUG#47995 + +--source include/master-slave.inc + +CREATE TABLE t1 (a VARCHAR(1000)); + +# We replicate the connection_id in the query_log_event +INSERT INTO t1 VALUES (CONNECTION_ID()); +--connection master1 +INSERT INTO t1 VALUES (CONNECTION_ID()); + +# We replicate the TIMESTAMP variable, so the following functions that +# are affected by the TIMESTAMP variable should be safe to replicate. +INSERT INTO t1 VALUES + (CURDATE()), + (CURRENT_DATE()), + (CURRENT_TIME()), + (CURRENT_TIMESTAMP()), + (CURTIME()), + (LOCALTIME()), + (LOCALTIMESTAMP()), + (NOW()), + (UNIX_TIMESTAMP()), + (UTC_DATE()), + (UTC_TIME()), + (UTC_TIMESTAMP()); + +# We replicate the random seed in a rand_log_event +INSERT INTO t1 VALUES (RAND()); +# We replicate the last_insert_id in an intvar_log_event +INSERT INTO t1 VALUES (LAST_INSERT_ID()); + +--sync_slave_with_master +--let $diff_table_1= master:test.t1 +--let $diff_table_2= slave:test.t1 +--source include/diff_tables.inc + +DROP TABLE t1; === added file 'mysql-test/suite/rpl/t/rpl_not_null_innodb.test' --- a/mysql-test/suite/rpl/t/rpl_not_null_innodb.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/t/rpl_not_null_innodb.test 2009-10-22 00:15:45 +0000 @@ -0,0 +1,19 @@ +################################################################################# +# This test checks if the replication between "null" fields to either "null" +# fields or "not null" fields works properly. In the first case, the execution +# should work fine. In the second case, it may fail according to the sql_mode +# being used. +# +# The test is devided in three main parts: +# +# 1 - NULL --> NULL (no failures) +# 2 - NULL --> NOT NULL ( sql-mode = STRICT and failures) +# 3 - NULL --> NOT NULL ( sql-mode != STRICT and no failures) +# +################################################################################# +--source include/master-slave.inc +--source include/have_innodb.inc +--source include/have_binlog_format_row.inc + +let $engine=Innodb; +--source extra/rpl_tests/rpl_not_null.test === added file 'mysql-test/suite/rpl/t/rpl_not_null_myisam.test' --- a/mysql-test/suite/rpl/t/rpl_not_null_myisam.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/t/rpl_not_null_myisam.test 2009-10-22 00:15:45 +0000 @@ -0,0 +1,18 @@ +################################################################################# +# This test checks if the replication between "null" fields to either "null" +# fields or "not null" fields works properly. In the first case, the execution +# should work fine. In the second case, it may fail according to the sql_mode +# being used. +# +# The test is devided in three main parts: +# +# 1 - NULL --> NULL (no failures) +# 2 - NULL --> NOT NULL ( sql-mode = STRICT and failures) +# 3 - NULL --> NOT NULL ( sql-mode != STRICT and no failures) +# +################################################################################# +--source include/master-slave.inc +--source include/have_binlog_format_row.inc + +let $engine=MyISAM; +--source extra/rpl_tests/rpl_not_null.test === modified file 'mysql-test/suite/rpl/t/rpl_relayspace.test' --- a/mysql-test/suite/rpl/t/rpl_relayspace.test 2008-02-28 11:36:14 +0000 +++ b/mysql-test/suite/rpl/t/rpl_relayspace.test 2010-01-18 12:56:10 +0000 @@ -2,8 +2,10 @@ # to force the deadlock after one event. source include/master-slave.inc; +let $master_log_file= query_get_value(SHOW MASTER STATUS, File, 1); connection slave; stop slave; +source include/wait_for_slave_to_stop.inc; connection master; # This will generate a master's binlog > 10 bytes create table t1 (a int); @@ -20,6 +22,7 @@ source include/wait_for_slave_param.inc; # A bug caused the I/O thread to refuse stopping. stop slave io_thread; +source include/wait_for_slave_io_to_stop.inc; reset slave; start slave; # The I/O thread stops filling the relay log when @@ -29,9 +32,11 @@ start slave; # So we should have a deadlock. # if it is not resolved automatically we'll detect # it with master_pos_wait that waits for farther than 1Ob; -# it will timeout after 10 seconds; +# it will timeout; # also the slave will probably not cooperate to shutdown # (as 2 threads are locked) -select master_pos_wait('master-bin.001',200,6)=-1; +--replace_result $master_log_file MASTER_LOG_FILE +eval select master_pos_wait('$master_log_file',200,30)=-1; + # End of 4.1 tests === modified file 'mysql-test/suite/rpl/t/rpl_rotate_logs.test' --- a/mysql-test/suite/rpl/t/rpl_rotate_logs.test 2009-03-12 09:48:41 +0000 +++ b/mysql-test/suite/rpl/t/rpl_rotate_logs.test 2010-01-27 21:53:08 +0000 @@ -153,7 +153,7 @@ disable_query_log; while ($1) { #eval means expand $ expressions - eval insert into t3 values($1 + 4); + eval insert HIGH_PRIORITY into t3 values($1 + 4); dec $1; } enable_query_log; === modified file 'mysql-test/suite/rpl/t/rpl_row_create_table.test' --- a/mysql-test/suite/rpl/t/rpl_row_create_table.test 2009-01-23 12:22:05 +0000 +++ b/mysql-test/suite/rpl/t/rpl_row_create_table.test 2009-11-27 13:34:39 +0000 @@ -292,4 +292,40 @@ connection master; DROP DATABASE mysqltest1; sync_slave_with_master; +# +# BUG#48506: crash in CREATE TABLE <existing_view> IF NOT EXISTS LIKE +# <tmp_tbl> with RBL +# + +source include/master-slave-reset.inc; + +connection master; +CREATE TEMPORARY TABLE t7(c1 INT); +CREATE TABLE t5(c1 INT); +CREATE TABLE t4(c1 INT); +CREATE VIEW bug48506_t1 AS SELECT 1; +CREATE VIEW bug48506_t2 AS SELECT * FROM t4; +CREATE VIEW bug48506_t3 AS SELECT t5.c1 AS A, t4.c1 AS B FROM t5, t4; +CREATE TABLE bug48506_t4(c1 INT); +--disable_warnings +sync_slave_with_master; +DROP VIEW bug48506_t1, bug48506_t2, bug48506_t3; +DROP TABLE bug48506_t4; + +connection master; +CREATE TABLE IF NOT EXISTS bug48506_t1 LIKE t7; +CREATE TABLE IF NOT EXISTS bug48506_t2 LIKE t7; +CREATE TABLE IF NOT EXISTS bug48506_t3 LIKE t7; +CREATE TABLE IF NOT EXISTS bug48506_t4 LIKE t7; +--enable_warnings +sync_slave_with_master; + +SHOW TABLES LIKE 'bug48506%'; + +connection master; +DROP VIEW IF EXISTS bug48506_t1, bug48506_t2, bug48506_t3; +DROP TEMPORARY TABLES t7; +DROP TABLES t4, t5; +DROP TABLES IF EXISTS bug48506_t4; +source include/master-slave-end.inc; --echo end of the tests === added file 'mysql-test/suite/rpl/t/rpl_row_trunc_temp.test' --- a/mysql-test/suite/rpl/t/rpl_row_trunc_temp.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/suite/rpl/t/rpl_row_trunc_temp.test 2009-11-22 05:10:33 +0000 @@ -0,0 +1,35 @@ +# +# Bug#48350 truncate temporary table crashes replication +# +# All statements operating on temporary tables should not be binlogged in RBR. +# However, before fix of bug#48350, 'TRUNCATE ...' statement on a temporary +# table was binlogged in RBR. +# + +--source include/master-slave.inc +--source include/have_binlog_format_row.inc + +#This statement is not binlogged in RBR. +CREATE TEMPORARY TABLE t1(c1 INTEGER); +CREATE TABLE t2(c1 INTEGER); +sync_slave_with_master; + +CREATE TABLE t1(c1 INTEGER); +INSERT INTO t1 VALUES(1), (2); +INSERT INTO t2 VALUES(1), (2); +SELECT * FROM t1; +SELECT * FROM t2; + +connection master; +TRUNCATE t1; +TRUNCATE t2; +sync_slave_with_master; +# t1 will have nothing, if 'TRUNCATE t1' has been replicate from master to +# slave. +SELECT * FROM t1; +SELECT * FROM t2; + +DROP TABLE t1; +connection master; +DROP TABLE t2; +--source include/master-slave-end.inc === modified file 'mysql-test/suite/rpl/t/rpl_temporary.test' --- a/mysql-test/suite/rpl/t/rpl_temporary.test 2009-05-22 23:29:41 +0000 +++ b/mysql-test/suite/rpl/t/rpl_temporary.test 2010-01-11 13:15:28 +0000 @@ -1,7 +1,10 @@ +-- source include/master-slave.inc + # Test need anonymous user when connection are made as "zedjzlcsjhd" +# But we only need it on the master, not the slave. +SET sql_log_bin = 0; source include/add_anonymous_users.inc; - --- source include/master-slave.inc +SET sql_log_bin = 1; # Clean up old slave's binlogs. # The slave is started with --log-slave-updates @@ -17,9 +20,6 @@ source include/add_anonymous_users.inc; save_master_pos; connection slave; -# Add suppression for expected warning(s) in slaves error log -call mtr.add_suppression("Slave: Can\'t find record in \'user\' Error_code: 1032"); - sync_with_master; reset master; @@ -291,7 +291,9 @@ drop table t1; --remove_file $MYSQLTEST_VARDIR/tmp/bug14157.sql # Delete the anonymous users +SET sql_log_bin = 0; source include/delete_anonymous_users.inc; +SET sql_log_bin = 1; === modified file 'mysql-test/suite/rpl/t/rpl_trigger.test' --- a/mysql-test/suite/rpl/t/rpl_trigger.test 2009-08-03 15:01:06 +0000 +++ b/mysql-test/suite/rpl/t/rpl_trigger.test 2009-11-18 14:50:31 +0000 @@ -5,6 +5,8 @@ --source include/have_binlog_format_mixed_or_statement.inc --source include/master-slave.inc +CALL mtr.add_suppression("Statement may not be safe to log in statement format."); + --disable_warnings DROP TABLE IF EXISTS t1; DROP TABLE IF EXISTS t2; @@ -89,7 +91,11 @@ end | delimiter ;| +# The trigger causes a warning for unsafe statement when +# binlog_format=statement since it uses get_lock. +--disable_warnings insert into t1 set a = now(); +--enable_warnings select a=b && a=c from t1; let $time=`select a from t1`; @@ -135,7 +141,11 @@ disconnect con2; truncate table t1; drop trigger t1_first; +# The trigger causes a warning for unsafe statement when +# binlog_format=statement since it uses get_lock. +--disable_warnings insert into t1 values ("2003-03-03","2003-03-03","2003-03-03"),(bug12480(),bug12480(),bug12480()),(now(),now(),now()); +--enable_warnings select a=b && a=c from t1; drop function bug12480; === modified file 'mysql-test/suite/rpl_ndb/r/rpl_ndb_extraCol.result' --- a/mysql-test/suite/rpl_ndb/r/rpl_ndb_extraCol.result 2009-08-29 08:30:59 +0000 +++ b/mysql-test/suite/rpl_ndb/r/rpl_ndb_extraCol.result 2009-10-22 00:21:50 +0000 @@ -400,62 +400,6 @@ set @b1 = concat(@b1,@b1); INSERT INTO t8 () VALUES(1,@b1,'Kyle'),(2,@b1,'JOE'),(3,@b1,'QA'); *** Drop t8 *** DROP TABLE t8; -STOP SLAVE; -RESET SLAVE; -CREATE TABLE t9 (a INT KEY, b BLOB, c CHAR(5), -d TIMESTAMP, -e INT NOT NULL) ENGINE='NDB'; -*** Create t9 on Master *** -CREATE TABLE t9 (a INT PRIMARY KEY, b BLOB, c CHAR(5) -) ENGINE='NDB'; -RESET MASTER; -*** Start Slave *** -START SLAVE; -*** Master Data Insert *** -set @b1 = 'b1b1b1b1'; -set @b1 = concat(@b1,@b1); -INSERT INTO t9 () VALUES(1,@b1,'Kyle'),(2,@b1,'JOE'),(3,@b1,'QA'); -SHOW SLAVE STATUS; -Slave_IO_State # -Master_Host 127.0.0.1 -Master_User root -Master_Port # -Connect_Retry 1 -Master_Log_File master-bin.000001 -Read_Master_Log_Pos # -Relay_Log_File # -Relay_Log_Pos # -Relay_Master_Log_File master-bin.000001 -Slave_IO_Running Yes -Slave_SQL_Running No -Replicate_Do_DB -Replicate_Ignore_DB -Replicate_Do_Table -Replicate_Ignore_Table # -Replicate_Wild_Do_Table -Replicate_Wild_Ignore_Table -Last_Errno 1364 -Last_Error Could not execute Write_rows event on table test.t9; Field 'e' doesn't have a default value, Error_code: 1364; handler error HA_ERR_ROWS_EVENT_APPLY; the event's master log master-bin.000001, end_log_pos 447 -Skip_Counter 0 -Exec_Master_Log_Pos # -Relay_Log_Space # -Until_Condition None -Until_Log_File -Until_Log_Pos 0 -Master_SSL_Allowed No -Master_SSL_CA_File -Master_SSL_CA_Path -Master_SSL_Cert -Master_SSL_Cipher -Master_SSL_Key -Seconds_Behind_Master # -Master_SSL_Verify_Server_Cert No -Last_IO_Errno # -Last_IO_Error # -Last_SQL_Errno 1364 -Last_SQL_Error Could not execute Write_rows event on table test.t9; Field 'e' doesn't have a default value, Error_code: 1364; handler error HA_ERR_ROWS_EVENT_APPLY; the event's master log master-bin.000001, end_log_pos 447 -SET GLOBAL SQL_SLAVE_SKIP_COUNTER=2; -START SLAVE; *** Create t10 on slave *** STOP SLAVE; RESET SLAVE; === modified file 'mysql-test/t/archive.test' --- a/mysql-test/t/archive.test 2009-12-03 11:19:05 +0000 +++ b/mysql-test/t/archive.test 2010-01-15 15:27:55 +0000 @@ -1625,3 +1625,24 @@ INSERT INTO t1 VALUES('aaaaaaaaaaaaaaaaa SELECT COUNT(t1.a) FROM t1, t1 a, t1 b, t1 c, t1 d, t1 e; DROP TABLE t1; SET @@join_buffer_size= @save_join_buffer_size; + +# +# BUG#47012 archive tables are not upgradeable, and server crashes on any access +# +let $MYSQLD_DATADIR= `SELECT @@datadir`; +copy_file std_data/bug47012.frm $MYSQLD_DATADIR/test/t1.frm; +copy_file std_data/bug47012.ARZ $MYSQLD_DATADIR/test/t1.ARZ; +copy_file std_data/bug47012.ARM $MYSQLD_DATADIR/test/t1.ARM; + +--error ER_TABLE_NEEDS_UPGRADE +SHOW CREATE TABLE t1; + +--error ER_TABLE_NEEDS_UPGRADE +SELECT * FROM t1; + +--error ER_TABLE_NEEDS_UPGRADE +INSERT INTO t1 (col1, col2) VALUES (1, "value"); + +REPAIR TABLE t1; +DROP TABLE t1; +remove_file $MYSQLD_DATADIR/test/t1.ARM; === added file 'mysql-test/t/bug47671-master.opt' --- a/mysql-test/t/bug47671-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/bug47671-master.opt 2009-11-25 06:55:49 +0000 @@ -0,0 +1 @@ +--default-character-set=utf8 --skip-character-set-client-handshake === added file 'mysql-test/t/bug47671.test' --- a/mysql-test/t/bug47671.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/bug47671.test 2009-11-30 05:24:26 +0000 @@ -0,0 +1,9 @@ +# Embedded server doesn't support external clients +--source include/not_embedded.inc + +--echo # +--echo # Bug#47671 - wrong character-set after upgrade from 5.1.34 to 5.1.39 +--echo # +--echo # Extract only charset information from 'status' command output using regex +--replace_regex /.*mysql.*// /Connection.*// /Current.*// /SSL.*// /Using.*// /Server version.*// /Protocol.*// /UNIX.*// /Uptime.*// /Threads.*// /TCP.*// +--exec $MYSQL -e "status"; === modified file 'mysql-test/t/delayed.test' --- a/mysql-test/t/delayed.test 2009-03-11 15:32:42 +0000 +++ b/mysql-test/t/delayed.test 2010-01-15 15:27:55 +0000 @@ -341,4 +341,28 @@ drop table t1; set global low_priority_updates = @old_delayed_updates; + +--echo # +--echo # Bug #47682 strange behaviour of INSERT DELAYED +--echo # + +--disable_warnings +DROP TABLE IF EXISTS t1, t2; +--enable_warnings + +CREATE TABLE t1 (f1 integer); +CREATE TABLE t2 (f1 integer); + +FLUSH TABLES WITH READ LOCK; +LOCK TABLES t1 READ; + +# ER_CANT_UPDATE_WITH_READLOCK with normal execution +# ER_TABLE_NOT_LOCKED when executed as prepared statement +--error ER_CANT_UPDATE_WITH_READLOCK, ER_TABLE_NOT_LOCKED +INSERT DELAYED INTO t2 VALUES (1); + +UNLOCK TABLES; +DROP TABLE t1, t2; + + --echo End of 5.1 tests === modified file 'mysql-test/t/delete.test' --- a/mysql-test/t/delete.test 2009-09-28 10:48:52 +0000 +++ b/mysql-test/t/delete.test 2009-11-18 09:32:03 +0000 @@ -336,3 +336,25 @@ SELECT * FROM t2; SELECT * FROM t3; DROP TABLE t1, t2, t3; + +--echo # +--echo # Bug #46425 crash in Diagnostics_area::set_ok_status, +--echo # empty statement, DELETE IGNORE +--echo # + +CREATE table t1 (i INTEGER); + +INSERT INTO t1 VALUES (1); + +--delimiter | + +CREATE TRIGGER tr1 AFTER DELETE ON t1 FOR EACH ROW +BEGIN + INSERT INTO t1 SELECT * FROM t1 AS A; +END | + +--delimiter ; +--error ER_CANT_UPDATE_USED_TABLE_IN_SF_OR_TRG +DELETE IGNORE FROM t1; + +DROP TABLE t1; \ No newline at end of file === modified file 'mysql-test/t/disabled.def' --- a/mysql-test/t/disabled.def 2009-12-03 11:19:05 +0000 +++ b/mysql-test/t/disabled.def 2010-01-15 17:02:57 +0000 @@ -11,7 +11,4 @@ ############################################################################## kill : Bug#37780 2008-12-03 HHunger need some changes to be robust enough for pushbuild. query_cache_28249 : Bug#43861 2009-03-25 main.query_cache_28249 fails sporadically -partition_innodb_builtin : Bug#32430 2009-09-25 mattiasj Waiting for push of Innodb changes -partition_innodb_plugin : Bug#32430 2009-09-25 mattiasj Waiting for push of Innodb changes -innodb-autoinc : Bug#48482 2009-11-02 svoj innodb-autoinc.test fails with results difference rpl_killed_ddl : Bug#45520: rpl_killed_ddl fails sporadically in pb2 === modified file 'mysql-test/t/fulltext.test' --- a/mysql-test/t/fulltext.test 2009-12-27 13:54:41 +0000 +++ b/mysql-test/t/fulltext.test 2010-01-15 15:27:55 +0000 @@ -496,3 +496,44 @@ PREPARE s FROM EXECUTE s; DEALLOCATE PREPARE s; DROP TABLE t1; + +--echo # +--echo # Bug #47930: MATCH IN BOOLEAN MODE returns too many results +--echo # inside subquery +--echo # + +CREATE TABLE t1 (a int); +INSERT INTO t1 VALUES (1), (2); + +CREATE TABLE t2 (a int, b2 char(10), FULLTEXT KEY b2 (b2)); +INSERT INTO t2 VALUES (1,'Scargill'); + +CREATE TABLE t3 (a int, b int); +INSERT INTO t3 VALUES (1,1), (2,1); + +--echo # t2 should use full text index +EXPLAIN +SELECT count(*) FROM t1 WHERE + not exists( + SELECT 1 FROM t2, t3 + WHERE t3.a=t1.a AND MATCH(b2) AGAINST('scargill' IN BOOLEAN MODE) + ); + +--echo # should return 0 +SELECT count(*) FROM t1 WHERE + not exists( + SELECT 1 FROM t2, t3 + WHERE t3.a=t1.a AND MATCH(b2) AGAINST('scargill' IN BOOLEAN MODE) + ); + +--echo # should return 0 +SELECT count(*) FROM t1 WHERE + not exists( + SELECT 1 FROM t2 IGNORE INDEX (b2), t3 + WHERE t3.a=t1.a AND MATCH(b2) AGAINST('scargill' IN BOOLEAN MODE) + ); + +DROP TABLE t1,t2,t3; + + +--echo End of 5.1 tests === modified file 'mysql-test/t/func_group.test' --- a/mysql-test/t/func_group.test 2009-10-14 08:46:50 +0000 +++ b/mysql-test/t/func_group.test 2009-11-24 15:26:13 +0000 @@ -1053,4 +1053,35 @@ ORDER BY max; --echo # DROP TABLE t1; +--echo # +--echo # Bug#43668: Wrong comparison and MIN/MAX for YEAR(2) +--echo # +create table t1 (f1 year(2), f2 year(4), f3 date, f4 datetime); +insert into t1 values + (98,1998,19980101,"1998-01-01 00:00:00"), + (00,2000,20000101,"2000-01-01 00:00:01"), + (02,2002,20020101,"2002-01-01 23:59:59"), + (60,2060,20600101,"2060-01-01 11:11:11"), + (70,1970,19700101,"1970-11-11 22:22:22"), + (NULL,NULL,NULL,NULL); +select min(f1),max(f1) from t1; +select min(f2),max(f2) from t1; +select min(f3),max(f3) from t1; +select min(f4),max(f4) from t1; +select a.f1 as a, b.f1 as b, a.f1 > b.f1 as gt, + a.f1 < b.f1 as lt, a.f1<=>b.f1 as eq +from t1 a, t1 b; +select a.f1 as a, b.f2 as b, a.f1 > b.f2 as gt, + a.f1 < b.f2 as lt, a.f1<=>b.f2 as eq +from t1 a, t1 b; +select a.f1 as a, b.f3 as b, a.f1 > b.f3 as gt, + a.f1 < b.f3 as lt, a.f1<=>b.f3 as eq +from t1 a, t1 b; +select a.f1 as a, b.f4 as b, a.f1 > b.f4 as gt, + a.f1 < b.f4 as lt, a.f1<=>b.f4 as eq +from t1 a, t1 b; +select *, f1 = f2 from t1; +drop table t1; +--echo # --echo End of 5.1 tests + === modified file 'mysql-test/t/func_misc.test' --- a/mysql-test/t/func_misc.test 2009-10-28 07:52:34 +0000 +++ b/mysql-test/t/func_misc.test 2010-01-11 13:15:28 +0000 @@ -103,198 +103,6 @@ show create table t1; drop table t1; # -# Bug#6760: Add SLEEP() function (feature request) -# -# Logics of original test: -# Reveal that a query with SLEEP does not need less time than estimated. -# -# Bug#12689: SLEEP() gets incorrectly cached/optimized-away -# -# Description from bug report (slightly modified) -# -# Bug 1 (happened all time): -# SELECT * FROM t1 WHERE SLEEP(1) will only result in a sleep of 1 -# second, regardless of the number of rows in t1. -# Bug 2 (happened all time): -# Such a query will also get cached by the query cache, but should not. -# -# Notes (mleich, 2008-05) -# ======================= -# -# Experiments around -# Bug#36345 Test 'func_misc' fails on RHAS3 x86_64 -# showed that the tests for both bugs could produce in case of parallel -# artificial system time (like via ntpd) -# - decreases false alarm -# - increases false success -# -# We try here to circumvent these issues by reimplementation of the tests -# and sophisticated scripting, although the cause of the problems is a massive -# error within the setup of the testing environment. -# Tests relying on or checking derivates of the system time must never meet -# parallel manipulations of system time. -# -# Results of experiments with/without manipulation of system time, -# information_schema.processlist content, high load on testing box -# ---------------------------------------------------------------- -# Definition: Predicted_cumulative_sleep_time = -# #_of_result_rows * sleep_time_per_result_row -# -# 1. Total (real sleep time) ~= predicted_cumulative_sleep_time !! -# 2. The state of a session within the PROCESSLIST changes to 'User sleep' -# if the sessions runs a statement containing the sleep function and the -# processing of the statement is just within the phase where the sleep -# is done. (*) -# 3. NOW() and processlist.time behave "synchronous" to system time and -# show also the "jumps" caused by system time manipulations. (*) -# 4. processlist.time is unsigned, the "next" value below 0 is ~ 4G (*) -# 5. Current processlist.time ~= current real sleep time if the system time -# was not manipulated. (*) -# 6. High system load can cause delays of <= 2 seconds. -# 7. Thanks to Davi for excellent hints and ideas. -# -# (*) -# - information_schema.processlist is not available before MySQL 5.1. -# - Observation of processlist content requires a -# - "worker" session sending the query with "send" and pulling results -# with "reap" -# - session observing the processlist parallel to the worker session -# "send" and "reap" do not work in case of an embedded server. -# Conclusion: Tests based on processlist have too many restrictions. -# -# Solutions for subtests based on TIMEDIFF of values filled via NOW() -# ------------------------------------------------------------------- -# Run the following sequence three times -# 1. SELECT <start_time> -# 2. Query with SLEEP -# 3. SELECT <end_time> -# If TIMEDIFF(<end_time>,<start_time>) is at least two times within a -# reasonable range assume that we did not met errors we were looking for. -# -# It is extreme unlikely that we have two system time changes within the -# < 30 seconds runtime. Even if the unlikely happens, there are so -# frequent runs of this test on this or another testing box which will -# catch the problem. -# - ---echo #------------------------------------------------------------------------ ---echo # Tests for Bug#6760 and Bug#12689 -# Number of rows within the intended result set. -SET @row_count = 4; -# Parameter within SLEEP function -SET @sleep_time_per_result_row = 1; -# Maximum acceptable delay caused by high load on testing box -SET @max_acceptable_delay = 2; -# TIMEDIFF = time for query with sleep (mostly the time caused by SLEEP) -# + time for delays caused by high load on testing box -# Ensure that at least a reasonable fraction of TIMEDIFF belongs to the SLEEP -# by appropriate setting of variables. -# Ensure that any "judging" has a base of minimum three attempts. -# (Test 2 uses all attempts except the first one.) -if (!` SELECT (@sleep_time_per_result_row * @row_count - @max_acceptable_delay > - @sleep_time_per_result_row) AND (@row_count - 1 >= 3)`) -{ - --echo # Have to abort because of error in plausibility check - --echo ###################################################### - --vertical_results - SELECT @sleep_time_per_result_row * @row_count - @max_acceptable_delay > - @sleep_time_per_result_row AS must_be_1, - @row_count - 1 >= 3 AS must_be_also_1, - @sleep_time_per_result_row, @row_count, @max_acceptable_delay; - exit; -} -SET @@global.query_cache_size = 1024 * 64; ---disable_warnings -DROP TEMPORARY TABLE IF EXISTS t_history; -DROP TABLE IF EXISTS t1; ---enable_warnings -CREATE TEMPORARY TABLE t_history (attempt SMALLINT, -start_ts DATETIME, end_ts DATETIME, -start_cached INTEGER, end_cached INTEGER); -CREATE TABLE t1 (f1 BIGINT); -let $num = `SELECT @row_count`; ---disable_query_log -begin; -while ($num) -{ - INSERT INTO t1 VALUES (1); - dec $num; -} -commit; ---enable_query_log - -let $loops = 4; -let $num = $loops; -while ($num) -{ - let $Qcache_queries_in_cache = - query_get_value(SHOW STATUS LIKE 'Qcache_queries_in_cache', Value, 1); - eval - INSERT INTO t_history - SET attempt = $loops - $num + 1, start_ts = NOW(), - start_cached = $Qcache_queries_in_cache; - SELECT *, SLEEP(@sleep_time_per_result_row) FROM t1; - # - # Do not determine Qcache_queries_in_cache before updating end_ts. The SHOW - # might cost too much time on an overloaded box. - eval - UPDATE t_history SET end_ts = NOW() - WHERE attempt = $loops - $num + 1; - let $Qcache_queries_in_cache = - query_get_value(SHOW STATUS LIKE 'Qcache_queries_in_cache', Value, 1); - eval - UPDATE t_history SET end_cached = $Qcache_queries_in_cache - WHERE attempt = $loops - $num + 1; - # DEBUG eval SELECT * FROM t_history WHERE attempt = $loops - $num + 1; - dec $num; -} - -# 1. The majority of queries with SLEEP must need a reasonable time -# -> SLEEP has an impact on runtime -# = Replacement for original Bug#6760 test -# -> total runtime is clear more needed than for one result row needed -# = Replacement for one of the original Bug#12689 tests ---echo # Test 1: Does the query with SLEEP need a reasonable time? -eval SELECT COUNT(*) >= $loops - 1 INTO @aux1 FROM t_history -WHERE TIMEDIFF(end_ts,start_ts) - @sleep_time_per_result_row * @row_count - BETWEEN 0 AND @max_acceptable_delay; -SELECT @aux1 AS "Expect 1"; -# -# 2. The majority of queries (the first one must be ignored) with SLEEP must -# need a reasonable time -# -> If we assume that the result of a cached query will be sent back -# immediate, without any sleep, than the query with SLEEP cannot be cached -# (current and intended behaviour for queries with SLEEP). -# -> It could be also not excluded that the query was cached but the server -# honoured somehow the SLEEP. Such a behaviour would be also acceptable. -# = Replacement for one of the original Bug#12689 tests ---echo # Test 2: Does the query with SLEEP need a reasonable time even in case ---echo # of the non first execution? -eval SELECT COUNT(*) >= $loops - 1 - 1 INTO @aux2 FROM t_history -WHERE TIMEDIFF(end_ts,start_ts) - @sleep_time_per_result_row * @row_count - BETWEEN 0 AND @max_acceptable_delay - AND attempt > 1; -SELECT @aux2 AS "Expect 1"; -# -# 3. The query with SLEEP should be not cached. -# -> SHOW STATUS Qcache_queries_in_cache must be not incremented after -# the execution of the query with SLEEP ---echo # Test 3: The query with SLEEP must be not cached. -eval SELECT COUNT(*) = $loops INTO @aux3 FROM t_history -WHERE end_cached = start_cached; -SELECT @aux3 AS "Expect 1"; -# -# Dump the content of t_history if one of the tests failed. -if (`SELECT @aux1 + @aux2 + @aux3 <> 3`) -{ - --echo # Some tests failed, dumping the content of t_history - SELECT * FROM t_history; -} -DROP TABLE t1; -DROP TEMPORARY TABLE t_history; -SET @@global.query_cache_size = default; - -# # Bug #21466: INET_ATON() returns signed, not unsigned # === modified file 'mysql-test/t/grant.test' --- a/mysql-test/t/grant.test 2009-06-15 15:53:45 +0000 +++ b/mysql-test/t/grant.test 2010-01-29 10:42:31 +0000 @@ -35,7 +35,7 @@ revoke delete on mysqltest.* from mysqlt show grants for mysqltest_1@localhost; grant select on mysqltest.* to mysqltest_1@localhost require NONE; show grants for mysqltest_1@localhost; -grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "MySQL AB"; +grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "Monty Program Ab"; show grants for mysqltest_1@localhost; revoke all privileges on mysqltest.* from mysqltest_1@localhost; show grants for mysqltest_1@localhost; === modified file 'mysql-test/t/grant2.test' --- a/mysql-test/t/grant2.test 2009-02-27 08:03:47 +0000 +++ b/mysql-test/t/grant2.test 2009-10-30 05:06:10 +0000 @@ -632,5 +632,40 @@ DROP DATABASE db1; --echo End of 5.0 tests +# +# Bug #48319: Server crashes on "GRANT/REVOKE ... TO CURRENT_USER" +# + +# work out who we are. +USE mysql; +SELECT LEFT(CURRENT_USER(),INSTR(CURRENT_USER(),'@')-1) INTO @u; +SELECT MID(CURRENT_USER(),INSTR(CURRENT_USER(),'@')+1) INTO @h; +SELECT password FROM user WHERE user=@u AND host=@h INTO @pwd; + +# show current privs. +SELECT user,host,password,insert_priv FROM user WHERE user=@u AND host=@h; + +# toggle INSERT +UPDATE user SET insert_priv='N' WHERE user=@u AND host=@h; +SELECT user,host,password,insert_priv FROM user WHERE user=@u AND host=@h; + +# show that GRANT ... TO CURRENT_USER() no longer crashes +GRANT INSERT ON *.* TO CURRENT_USER(); +SELECT user,host,password,insert_priv FROM user WHERE user=@u AND host=@h; +UPDATE user SET insert_priv='N' WHERE user=@u AND host=@h; + +# show that GRANT ... TO CURRENT_USER() IDENTIFIED BY ... works now +GRANT INSERT ON *.* TO CURRENT_USER() IDENTIFIED BY 'keksdose'; +SELECT user,host,password,insert_priv FROM user WHERE user=@u AND host=@h; + +UPDATE user SET password=@pwd WHERE user=@u AND host=@h; +SELECT user,host,password,insert_priv FROM user WHERE user=@u AND host=@h; + +FLUSH PRIVILEGES; + +USE test; + +--echo End of 5.1 tests + # Wait till we reached the initial number of concurrent sessions --source include/wait_until_count_sessions.inc === modified file 'mysql-test/t/group_min_max.test' --- a/mysql-test/t/group_min_max.test 2009-08-30 07:03:37 +0000 +++ b/mysql-test/t/group_min_max.test 2009-11-23 10:04:17 +0000 @@ -1016,6 +1016,18 @@ SELECT a, MAX(b) FROM t WHERE b > 0 AND DROP TABLE t; +--echo # +--echo # Bug #48472: Loose index scan inappropriately chosen for some WHERE +--echo # conditions +--echo # + +CREATE TABLE t (a INT, b INT, INDEX (a,b)); +INSERT INTO t VALUES (2,0), (2,0), (2,1), (2,1); +INSERT INTO t SELECT * FROM t; + +SELECT a, MAX(b) FROM t WHERE 0=b+0 GROUP BY a; + +DROP TABLE t; --echo End of 5.0 tests === modified file 'mysql-test/t/innodb-analyze.test' --- a/mysql-test/t/innodb-analyze.test 2009-11-13 21:26:08 +0000 +++ b/mysql-test/t/innodb-analyze.test 2010-01-15 15:58:25 +0000 @@ -11,7 +11,7 @@ -- disable_result_log -- enable_warnings -SET @old_innodb_stats_sample_pages=@@innodb_stats_sample_pages; +let $sample_pages=`select @@innodb_stats_sample_pages`; SET GLOBAL innodb_stats_sample_pages=0; # check that the value has been adjusted to 1 @@ -62,4 +62,4 @@ SET GLOBAL innodb_stats_sample_pages=16; ANALYZE TABLE innodb_analyze; DROP TABLE innodb_analyze; -SET GLOBAL innodb_stats_sample_pages=@old_innodb_stats_sample_pages; +EVAL SET GLOBAL innodb_stats_sample_pages=$sample_pages; === modified file 'mysql-test/t/innodb-autoinc.test' --- a/mysql-test/t/innodb-autoinc.test 2009-12-03 11:34:11 +0000 +++ b/mysql-test/t/innodb-autoinc.test 2010-01-15 21:12:30 +0000 @@ -2,6 +2,8 @@ # embedded server ignores 'delayed', so skip this -- source include/not_embedded.inc +let $file_format_check=`select @@innodb_file_format_check`; + --disable_warnings drop table if exists t1; --enable_warnings @@ -156,7 +158,7 @@ DROP TABLE t1; # # Test changes to AUTOINC next value calculation SET @@SESSION.AUTO_INCREMENT_INCREMENT=100, @@SESSION.AUTO_INCREMENT_OFFSET=10; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (c1 INT AUTO_INCREMENT, PRIMARY KEY(c1)) ENGINE=InnoDB; INSERT INTO t1 VALUES (NULL),(5),(NULL); @@ -173,7 +175,7 @@ DROP TABLE t1; # Reset the AUTOINC session variables SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (c1 INT AUTO_INCREMENT, PRIMARY KEY(c1)) ENGINE=InnoDB; INSERT INTO t1 VALUES(0); @@ -193,13 +195,13 @@ DROP TABLE t1; # Reset the AUTOINC session variables SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (c1 INT AUTO_INCREMENT, PRIMARY KEY(c1)) ENGINE=InnoDB; INSERT INTO t1 VALUES(-1); SELECT * FROM t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=100, @@SESSION.AUTO_INCREMENT_OFFSET=10; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; INSERT INTO t1 VALUES (-2), (NULL),(2),(NULL); INSERT INTO t1 VALUES (250),(NULL); SELECT * FROM t1; @@ -214,13 +216,13 @@ DROP TABLE t1; # Reset the AUTOINC session variables SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (c1 INT UNSIGNED AUTO_INCREMENT, PRIMARY KEY(c1)) ENGINE=InnoDB; INSERT INTO t1 VALUES(-1); SELECT * FROM t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=100, @@SESSION.AUTO_INCREMENT_OFFSET=10; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; INSERT INTO t1 VALUES (-2); INSERT INTO t1 VALUES (NULL); INSERT INTO t1 VALUES (2); @@ -240,13 +242,13 @@ DROP TABLE t1; # Reset the AUTOINC session variables SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (c1 INT UNSIGNED AUTO_INCREMENT, PRIMARY KEY(c1)) ENGINE=InnoDB; INSERT INTO t1 VALUES(-1); SELECT * FROM t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=100, @@SESSION.AUTO_INCREMENT_OFFSET=10; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; INSERT INTO t1 VALUES (-2),(NULL),(2),(NULL); INSERT INTO t1 VALUES (250),(NULL); SELECT * FROM t1; @@ -262,7 +264,7 @@ DROP TABLE t1; # Check for overflow handling when increment is > 1 SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (c1 BIGINT AUTO_INCREMENT, PRIMARY KEY(c1)) ENGINE=InnoDB; # TODO: Fix the autoinc init code @@ -271,7 +273,7 @@ INSERT INTO t1 VALUES(NULL); INSERT INTO t1 VALUES (9223372036854775794); #-- 2^63 - 14 SELECT * FROM t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=2, @@SESSION.AUTO_INCREMENT_OFFSET=10; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; # This should just fit INSERT INTO t1 VALUES (NULL),(NULL),(NULL),(NULL),(NULL),(NULL); SELECT * FROM t1; @@ -281,7 +283,7 @@ DROP TABLE t1; # Check for overflow handling when increment and offser are > 1 SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (c1 BIGINT UNSIGNED AUTO_INCREMENT, PRIMARY KEY(c1)) ENGINE=InnoDB; # TODO: Fix the autoinc init code @@ -290,7 +292,7 @@ INSERT INTO t1 VALUES(NULL); INSERT INTO t1 VALUES (18446744073709551603); #-- 2^64 - 13 SELECT * FROM t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=2, @@SESSION.AUTO_INCREMENT_OFFSET=10; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; # This should fail because of overflow but it doesn't, it seems to be # a MySQL server bug. It wraps around to 0 for the last value. # See MySQL Bug# 39828 @@ -313,7 +315,7 @@ DROP TABLE t1; # Check for overflow handling when increment and offset are odd numbers SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (c1 BIGINT UNSIGNED AUTO_INCREMENT, PRIMARY KEY(c1)) ENGINE=InnoDB; # TODO: Fix the autoinc init code @@ -322,7 +324,7 @@ INSERT INTO t1 VALUES(NULL); INSERT INTO t1 VALUES (18446744073709551603); #-- 2^64 - 13 SELECT * FROM t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=5, @@SESSION.AUTO_INCREMENT_OFFSET=7; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; # This should fail because of overflow but it doesn't. It fails with # a duplicate entry message because of a MySQL server bug, it wraps # around. See MySQL Bug# 39828, once MySQL fix the bug we can replace @@ -344,7 +346,7 @@ DROP TABLE t1; # and check for large -ve numbers SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (c1 BIGINT AUTO_INCREMENT, PRIMARY KEY(c1)) ENGINE=InnoDB; # TODO: Fix the autoinc init code @@ -355,7 +357,7 @@ INSERT INTO t1 VALUES(-92233720368547758 INSERT INTO t1 VALUES(-9223372036854775808); #-- -2^63 SELECT * FROM t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=3, @@SESSION.AUTO_INCREMENT_OFFSET=3; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; INSERT INTO t1 VALUES (NULL),(NULL), (NULL); SELECT * FROM t1; DROP TABLE t1; @@ -364,7 +366,7 @@ DROP TABLE t1; # large numbers 2^60 SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (c1 BIGINT UNSIGNED AUTO_INCREMENT, PRIMARY KEY(c1)) ENGINE=InnoDB; # TODO: Fix the autoinc init code @@ -373,7 +375,7 @@ INSERT INTO t1 VALUES(NULL); INSERT INTO t1 VALUES (18446744073709551610); #-- 2^64 - 2 SELECT * FROM t1; SET @@SESSION.AUTO_INCREMENT_INCREMENT=1152921504606846976, @@SESSION.AUTO_INCREMENT_OFFSET=1152921504606846976; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; # This should fail because of overflow but it doesn't. It wraps around # and the autoinc values look bogus too. # See MySQL Bug# 39828, once MySQL fix the bug we can enable the error @@ -396,7 +398,7 @@ DROP TABLE t1; # SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; SET @@INSERT_ID=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; CREATE TABLE t1 (c1 DOUBLE NOT NULL AUTO_INCREMENT, c2 INT, PRIMARY KEY (c1)) ENGINE=InnoDB; INSERT INTO t1 VALUES(NULL, 1); INSERT INTO t1 VALUES(NULL, 2); @@ -508,7 +510,7 @@ DROP TABLE t1; # If the user has specified negative values for an AUTOINC column then # InnoDB should ignore those values when setting the table's max value. SET @@SESSION.AUTO_INCREMENT_INCREMENT=1, @@SESSION.AUTO_INCREMENT_OFFSET=1; -SHOW VARIABLES LIKE "%auto_inc%"; +SHOW VARIABLES LIKE "auto_inc%"; # TINYINT CREATE TABLE t1 (c1 TINYINT PRIMARY KEY AUTO_INCREMENT, c2 VARCHAR(10)) ENGINE=InnoDB; INSERT INTO t1 VALUES (1, NULL); @@ -620,3 +622,42 @@ SHOW CREATE TABLE T1; INSERT INTO T1 (c2) values (0); SELECT * FROM T1; DROP TABLE T1; + +## +# 49032: Use the correct function to read the AUTOINC column value +# +CREATE TABLE T1(C1 DOUBLE AUTO_INCREMENT KEY, C2 CHAR(10)) ENGINE=InnoDB; +INSERT INTO T1(C1, C2) VALUES (1, 'innodb'), (3, 'innodb'); +# Restart the server +-- source include/restart_mysqld.inc +INSERT INTO T1(C2) VALUES ('innodb'); +SHOW CREATE TABLE T1; +DROP TABLE T1; +CREATE TABLE T1(C1 FLOAT AUTO_INCREMENT KEY, C2 CHAR(10)) ENGINE=InnoDB; +INSERT INTO T1(C1, C2) VALUES (1, 'innodb'), (3, 'innodb'); +# Restart the server +-- source include/restart_mysqld.inc +INSERT INTO T1(C2) VALUES ('innodb'); +SHOW CREATE TABLE T1; +DROP TABLE T1; + +## +# 47720: REPLACE INTO Autoincrement column with negative values +# +CREATE TABLE t1 (c1 INT AUTO_INCREMENT PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 SET c1 = 1; +SHOW CREATE TABLE t1; +INSERT INTO t1 SET c1 = 2; +INSERT INTO t1 SET c1 = -1; +SELECT * FROM t1; +-- error ER_DUP_ENTRY,1062 +INSERT INTO t1 SET c1 = -1; +SHOW CREATE TABLE t1; +REPLACE INTO t1 VALUES (-1); +SELECT * FROM t1; +SHOW CREATE TABLE t1; +DROP TABLE t1; + +--disable_query_log +EVAL SET GLOBAL innodb_file_format_check=$file_format_check; +--enable_query_log === added file 'mysql-test/t/innodb-consistent-master.opt' --- a/mysql-test/t/innodb-consistent-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/innodb-consistent-master.opt 2010-01-15 15:58:25 +0000 @@ -0,0 +1 @@ +--loose-innodb_lock_wait_timeout=2 === added file 'mysql-test/t/innodb-consistent.test' --- a/mysql-test/t/innodb-consistent.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/innodb-consistent.test 2010-01-15 15:58:25 +0000 @@ -0,0 +1,58 @@ +-- source include/not_embedded.inc +-- source include/have_innodb.inc + +--disable_warnings +drop table if exists t1; +--enable_warnings + +# REPLACE INTO ... SELECT and INSERT INTO ... SELECT should do +# a consistent read of the source table. + +connect (a,localhost,root,,); +connect (b,localhost,root,,); +connection a; +set session transaction isolation level read committed; +create table t1(a int not null) engine=innodb DEFAULT CHARSET=latin1; +create table t2 like t1; +insert into t2 values (1),(2),(3),(4),(5),(6),(7); +set autocommit=0; + +# REPLACE INTO ... SELECT case +begin; +# this should not result in any locks on t2. +replace into t1 select * from t2; + +connection b; +set session transaction isolation level read committed; +set autocommit=0; +# should not cuase a lock wait. +delete from t2 where a=5; +commit; +delete from t2; +commit; +connection a; +commit; + +# INSERT INTO ... SELECT case +begin; +# this should not result in any locks on t2. +insert into t1 select * from t2; + +connection b; +set session transaction isolation level read committed; +set autocommit=0; +# should not cuase a lock wait. +delete from t2 where a=5; +commit; +delete from t2; +commit; +connection a; +commit; + +select * from t1; +drop table t1; +drop table t2; + +connection default; +disconnect a; +disconnect b; === modified file 'mysql-test/t/innodb-index.test' --- a/mysql-test/t/innodb-index.test 2009-11-30 21:37:27 +0000 +++ b/mysql-test/t/innodb-index.test 2010-01-15 15:58:25 +0000 @@ -1,6 +1,6 @@ -- source include/have_innodb.inc -SET @save_innodb_file_format_check=@@global.innodb_file_format_check; +let $innodb_file_format_check_orig=`select @@innodb_file_format_check`; create table t1(a int not null, b int, c char(10) not null, d varchar(20)) engine = innodb; insert into t1 values (5,5,'oo','oo'),(4,4,'tr','tr'),(3,4,'ad','ad'),(2,3,'ak','ak'); @@ -404,6 +404,7 @@ create index t1u on t1 (u(1)); drop table t1; eval set global innodb_file_per_table=$per_table; eval set global innodb_file_format=$format; +eval set global innodb_file_format_check=$format; # # Test to check whether CREATE INDEX handles implicit foreign key @@ -541,4 +542,9 @@ disconnect b; DROP TABLE t1; -SET GLOBAL innodb_file_format_check=@save_innodb_file_format_check; +# +# restore environment to the state it was before this test execution +# + +-- disable_query_log +eval SET GLOBAL innodb_file_format_check=$innodb_file_format_check_orig; === modified file 'mysql-test/t/innodb-master.opt' --- a/mysql-test/t/innodb-master.opt 2009-06-09 13:19:13 +0000 +++ b/mysql-test/t/innodb-master.opt 2010-01-15 15:58:25 +0000 @@ -1 +1 @@ ---binlog_cache_size=32768 --innodb_lock_wait_timeout=1 +--binlog_cache_size=32768 --loose_innodb_lock_wait_timeout=1 === modified file 'mysql-test/t/innodb-semi-consistent-master.opt' --- a/mysql-test/t/innodb-semi-consistent-master.opt 2009-06-09 13:19:13 +0000 +++ b/mysql-test/t/innodb-semi-consistent-master.opt 2010-01-15 15:58:25 +0000 @@ -1 +1 @@ ---innodb_lock_wait_timeout=2 +--loose-innodb_lock_wait_timeout=2 === modified file 'mysql-test/t/innodb-timeout.test' --- a/mysql-test/t/innodb-timeout.test 2009-06-09 13:19:13 +0000 +++ b/mysql-test/t/innodb-timeout.test 2010-01-18 12:56:10 +0000 @@ -30,6 +30,7 @@ begin; insert into t1 values(1),(2),(3); connection b; +set innodb_lock_wait_timeout=5; --send select * from t1 for update; @@ -44,16 +45,20 @@ begin; insert into t1 values(4); connection b; ---send +# Test that we get a lock timeout. +# We cannot reliably test that the timeout is exactly 1 seconds due to +# process scheduling differences on the host running the test suite. But we +# can test that we are within reasonable range. +set innodb_lock_wait_timeout=2; +set @a= current_timestamp(); +--error ER_LOCK_WAIT_TIMEOUT select * from t1 for update; +set @b= current_timestamp(); +set @c= timestampdiff(SECOND, @a, @b); +select if(@c >= 1 and @c <= 10, 'OK', concat("NOT OK, time passed=", @c)); connection a; -sleep 2; commit; - -connection b; ---error ER_LOCK_WAIT_TIMEOUT -reap; drop table t1; connection default; === modified file 'mysql-test/t/innodb-use-sys-malloc-master.opt' --- a/mysql-test/t/innodb-use-sys-malloc-master.opt 2009-06-09 13:19:13 +0000 +++ b/mysql-test/t/innodb-use-sys-malloc-master.opt 2010-01-15 15:58:25 +0000 @@ -1,2 +1 @@ ---innodb-use-sys-malloc=true ---innodb-use-sys-malloc=true +--loose-innodb-use-sys-malloc=true === modified file 'mysql-test/t/innodb-zip.test' --- a/mysql-test/t/innodb-zip.test 2009-06-09 15:08:46 +0000 +++ b/mysql-test/t/innodb-zip.test 2010-01-15 15:58:25 +0000 @@ -178,11 +178,11 @@ set innodb_strict_mode = on; --error ER_CANT_CREATE_TABLE create table t1 (id int primary key) engine = innodb key_block_size = 0; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t2 (id int primary key) engine = innodb key_block_size = 9; -show errors; +show warnings; create table t3 (id int primary key) engine = innodb key_block_size = 1; @@ -208,22 +208,22 @@ key_block_size = 8 row_format = compress --error ER_CANT_CREATE_TABLE create table t2 (id int primary key) engine = innodb key_block_size = 8 row_format = redundant; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t3 (id int primary key) engine = innodb key_block_size = 8 row_format = compact; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t4 (id int primary key) engine = innodb key_block_size = 8 row_format = dynamic; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t5 (id int primary key) engine = innodb key_block_size = 8 row_format = default; -show errors; +show warnings; SELECT table_schema, table_name, row_format FROM information_schema.tables WHERE engine='innodb'; @@ -233,17 +233,17 @@ drop table t1; --error ER_CANT_CREATE_TABLE create table t1 (id int primary key) engine = innodb key_block_size = 9 row_format = redundant; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t2 (id int primary key) engine = innodb key_block_size = 9 row_format = compact; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t2 (id int primary key) engine = innodb key_block_size = 9 row_format = dynamic; -show errors; +show warnings; SELECT table_schema, table_name, row_format FROM information_schema.tables WHERE engine='innodb'; @@ -253,25 +253,25 @@ set global innodb_file_per_table = off; --error ER_CANT_CREATE_TABLE create table t1 (id int primary key) engine = innodb key_block_size = 1; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t2 (id int primary key) engine = innodb key_block_size = 2; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t3 (id int primary key) engine = innodb key_block_size = 4; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t4 (id int primary key) engine = innodb key_block_size = 8; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t5 (id int primary key) engine = innodb key_block_size = 16; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t6 (id int primary key) engine = innodb row_format = compressed; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t7 (id int primary key) engine = innodb row_format = dynamic; -show errors; +show warnings; create table t8 (id int primary key) engine = innodb row_format = compact; create table t9 (id int primary key) engine = innodb row_format = redundant; @@ -285,25 +285,25 @@ set global innodb_file_format = `0`; --error ER_CANT_CREATE_TABLE create table t1 (id int primary key) engine = innodb key_block_size = 1; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t2 (id int primary key) engine = innodb key_block_size = 2; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t3 (id int primary key) engine = innodb key_block_size = 4; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t4 (id int primary key) engine = innodb key_block_size = 8; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t5 (id int primary key) engine = innodb key_block_size = 16; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t6 (id int primary key) engine = innodb row_format = compressed; -show errors; +show warnings; --error ER_CANT_CREATE_TABLE create table t7 (id int primary key) engine = innodb row_format = dynamic; -show errors; +show warnings; create table t8 (id int primary key) engine = innodb row_format = compact; create table t9 (id int primary key) engine = innodb row_format = redundant; === modified file 'mysql-test/t/innodb.test' --- a/mysql-test/t/innodb.test 2009-12-27 13:54:41 +0000 +++ b/mysql-test/t/innodb.test 2010-01-15 15:58:25 +0000 @@ -2270,7 +2270,7 @@ disconnect j; drop table t1, t2, t3, t5, t6, t8, t9; # bug 18934, "InnoDB crashes when table uses column names like DB_ROW_ID" ---error 1005 +--error ER_WRONG_COLUMN_NAME CREATE TABLE t1 (DB_ROW_ID int) engine=innodb; # === modified file 'mysql-test/t/innodb_bug34300.test' --- a/mysql-test/t/innodb_bug34300.test 2009-11-13 21:26:08 +0000 +++ b/mysql-test/t/innodb_bug34300.test 2010-01-15 15:58:25 +0000 @@ -9,7 +9,7 @@ -- disable_result_log # set packet size and reconnect -SET @save_max_allowed_packet=@@global.max_allowed_packet; +let $max_packet=`select @@global.max_allowed_packet`; SET @@global.max_allowed_packet=16777216; --connect (newconn, localhost, root,,) @@ -33,4 +33,4 @@ SELECT f4, f8 FROM bug34300; DROP TABLE bug34300; disconnect newconn; connection default; -SET @@global.max_allowed_packet=@save_max_allowed_packet; +EVAL SET @@global.max_allowed_packet=$max_packet; === modified file 'mysql-test/t/innodb_bug36169.test' --- a/mysql-test/t/innodb_bug36169.test 2009-11-13 21:26:08 +0000 +++ b/mysql-test/t/innodb_bug36169.test 2010-01-15 15:58:25 +0000 @@ -4,9 +4,9 @@ # -- source include/have_innodb.inc -set @old_innodb_file_format=@@innodb_file_format; -set @old_innodb_file_per_table=@@innodb_file_per_table; +let $file_format=`select @@innodb_file_format`; +let $file_per_table=`select @@innodb_file_per_table`; SET GLOBAL innodb_file_format='Barracuda'; SET GLOBAL innodb_file_per_table=ON; @@ -1155,5 +1155,5 @@ DROP TABLE IF EXISTS table4; DROP TABLE IF EXISTS table5; DROP TABLE IF EXISTS table6; -set global innodb_file_format=@old_innodb_file_format; -set global innodb_file_per_table=@old_innodb_file_per_table; +EVAL SET GLOBAL innodb_file_format=$file_format; +EVAL SET GLOBAL innodb_file_per_table=$file_per_table; === modified file 'mysql-test/t/innodb_bug36172.test' --- a/mysql-test/t/innodb_bug36172.test 2009-11-13 21:26:08 +0000 +++ b/mysql-test/t/innodb_bug36172.test 2010-01-15 15:58:25 +0000 @@ -13,9 +13,10 @@ SET storage_engine=InnoDB; -- disable_query_log -- disable_result_log -set @old_innodb_file_per_table=@@innodb_file_per_table; -set @old_innodb_file_format=@@innodb_file_format; +let $file_format=`select @@innodb_file_format`; +let $file_format_check=`select @@innodb_file_format_check`; +let $file_per_table=`select @@innodb_file_per_table`; SET GLOBAL innodb_file_format='Barracuda'; SET GLOBAL innodb_file_per_table=on; @@ -27,6 +28,6 @@ INSERT IGNORE INTO `table0` SET `col19` CHECK TABLE table0 EXTENDED; DROP TABLE table0; -set global innodb_file_per_table=@old_innodb_file_per_table; -set global innodb_file_format=@old_innodb_file_format; -set global innodb_file_format_check=Antelope; +EVAL SET GLOBAL innodb_file_format=$file_format; +EVAL SET GLOBAL innodb_file_format_check=$file_format_check; +EVAL SET GLOBAL innodb_file_per_table=$file_per_table; === modified file 'mysql-test/t/innodb_bug42101-nonzero-master.opt' --- a/mysql-test/t/innodb_bug42101-nonzero-master.opt 2009-05-19 08:20:28 +0000 +++ b/mysql-test/t/innodb_bug42101-nonzero-master.opt 2010-01-15 15:58:25 +0000 @@ -1 +1 @@ ---innodb_commit_concurrency=1 +--loose_innodb_commit_concurrency=1 === modified file 'mysql-test/t/innodb_bug44369.test' --- a/mysql-test/t/innodb_bug44369.test 2009-11-02 14:59:44 +0000 +++ b/mysql-test/t/innodb_bug44369.test 2010-01-15 15:58:25 +0000 @@ -6,16 +6,12 @@ --source include/have_innodb.inc # This create table operation should fail. ---error ER_CANT_CREATE_TABLE +--error ER_WRONG_COLUMN_NAME create table bug44369 (DB_ROW_ID int) engine=innodb; # This create should fail as well ---error ER_CANT_CREATE_TABLE +--error ER_WRONG_COLUMN_NAME create table bug44369 (db_row_id int) engine=innodb; -show warnings; - ---error ER_CANT_CREATE_TABLE +--error ER_WRONG_COLUMN_NAME create table bug44369 (db_TRX_Id int) engine=innodb; - -show warnings; === added file 'mysql-test/t/innodb_bug44571.test' --- a/mysql-test/t/innodb_bug44571.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/innodb_bug44571.test 2010-01-15 15:58:25 +0000 @@ -0,0 +1,13 @@ +# +# Bug#44571 InnoDB Plugin crashes on ADD INDEX +# http://bugs.mysql.com/44571 +# +-- source include/have_innodb.inc + +CREATE TABLE bug44571 (foo INT) ENGINE=InnoDB; +ALTER TABLE bug44571 CHANGE foo bar INT; +-- error ER_KEY_COLUMN_DOES_NOT_EXITS +ALTER TABLE bug44571 ADD INDEX bug44571b (foo); +ALTER TABLE bug44571 ADD INDEX bug44571b (bar); +CREATE INDEX bug44571c ON bug44571 (bar); +DROP TABLE bug44571; === added file 'mysql-test/t/innodb_bug46676.test' --- a/mysql-test/t/innodb_bug46676.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/innodb_bug46676.test 2010-01-15 15:58:25 +0000 @@ -0,0 +1,16 @@ +# This is the test for bug 46676: mysqld got exception 0xc0000005 +# It is reproducible with InnoDB plugin 1.0.4 + MySQL 5.1.37. +# But no longer reproducible after MySQL 5.1.38 (with plugin 1.0.5). + +--source include/have_innodb.inc + +SET foreign_key_checks=0; +CREATE TABLE t1 (id int, foreign key (id) references t2(id)) ENGINE=INNODB; +CREATE TABLE t2 (id int, foreign key (id) references t1(id)) ENGINE=INNODB; +SET foreign_key_checks=1; + +# Server crashes +SELECT COUNT(*) FROM information_schema.key_column_usage WHERE REFERENCED_TABLE_NAME in ('t1', 't2'); + +SET foreign_key_checks=0; +DROP TABLE t1, t2; === added file 'mysql-test/t/innodb_bug47167.test' --- a/mysql-test/t/innodb_bug47167.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/innodb_bug47167.test 2010-01-15 15:58:25 +0000 @@ -0,0 +1,45 @@ +# This is the unit test for bug *47167. +# It tests setting the global variable +# "innodb_file_format_check" with a +# user-Defined Variable. + +--source include/have_innodb.inc + +# Save the value (Antelope) in 'innodb_file_format_check' to +# 'old_innodb_file_format_check' +set @old_innodb_file_format_check=@@innodb_file_format_check; + +# @old_innodb_file_format_check shall have the value of 'Antelope' +select @old_innodb_file_format_check; + +# Reset the value in 'innodb_file_format_check' to 'Barracuda' +set global innodb_file_format_check = Barracuda; + +select @@innodb_file_format_check; + +# Set 'innodb_file_format_check' to its default value, which +# is the latest file format supported in the current release. +set global innodb_file_format_check = DEFAULT; + +select @@innodb_file_format_check; + +# Put the saved value back to 'innodb_file_format_check' +set global innodb_file_format_check = @old_innodb_file_format_check; + +# Check whether 'innodb_file_format_check' get its original value. +select @@innodb_file_format_check; + +# Following are negative tests, all should fail. +--disable_warnings +--error ER_WRONG_ARGUMENTS +set global innodb_file_format_check = cheetah; + +--error ER_WRONG_ARGUMENTS +set global innodb_file_format_check = Bear; + +--error ER_WRONG_ARGUMENTS +set global innodb_file_format_check = on; + +--error ER_WRONG_ARGUMENTS +set global innodb_file_format_check = off; +--enable_warnings === modified file 'mysql-test/t/innodb_file_format.test' --- a/mysql-test/t/innodb_file_format.test 2009-11-30 21:37:27 +0000 +++ b/mysql-test/t/innodb_file_format.test 2010-01-15 15:58:25 +0000 @@ -1,5 +1,4 @@ -- source include/have_innodb.inc -set @old_innodb_file_format=@@innodb_file_format; call mtr.add_suppression("InnoDB: invalid innodb_file_format_check value"); @@ -29,6 +28,4 @@ set global innodb_file_format=on; --error ER_WRONG_ARGUMENTS set global innodb_file_format=off; select @@innodb_file_format_check; - -set global innodb_file_format=@old_innodb_file_format; -set global innodb_file_format_check=Antelope; +set global innodb_file_format_check=antelope; === modified file 'mysql-test/t/innodb_information_schema.test' --- a/mysql-test/t/innodb_information_schema.test 2009-06-23 12:00:24 +0000 +++ b/mysql-test/t/innodb_information_schema.test 2010-01-15 15:58:25 +0000 @@ -109,19 +109,18 @@ SELECT * FROM ```t'\"_str` WHERE c1 = '3 -- send SELECT * FROM ```t'\"_str` WHERE c1 = '4' FOR UPDATE; +-- enable_result_log -- connection con_verify_innodb_locks - -# Loop, giving time for the above 2 queries to execute before continuing. -# Without this, it sometimes happens that the SELECT FROM innodb_locks +# Wait for the above queries to execute before continuing. +# Without this, it sometimes happens that the SELECT from innodb_locks # executes before some of them, resulting in less than expected number -# of rows being selected from innodb_locks. -SET @counter := 0; -while (`SELECT (@counter := @counter + 1) <= 50 AND COUNT(*) != 14 FROM INFORMATION_SCHEMA.INNODB_LOCKS`) -{ - sleep 0.1; -} - --- enable_result_log +# of rows being selected from innodb_locks. If there is a bug and there +# are no 14 rows in innodb_locks then this test will fail with timeout. +let $count = 14; +let $table = INFORMATION_SCHEMA.INNODB_LOCKS; +-- source include/wait_until_rows_count.inc +# the above enables the query log, re-disable it +-- disable_query_log SELECT lock_mode, lock_type, lock_table, lock_index, lock_rec, lock_data FROM INFORMATION_SCHEMA.INNODB_LOCKS ORDER BY lock_data; === modified file 'mysql-test/t/innodb_lock_wait_timeout_1.test' --- a/mysql-test/t/innodb_lock_wait_timeout_1.test 2009-11-03 17:45:52 +0000 +++ b/mysql-test/t/innodb_lock_wait_timeout_1.test 2009-11-12 11:43:33 +0000 @@ -71,6 +71,40 @@ set autocommit=default; drop table t1; --echo # +--echo # Bug #37183 insert ignore into .. select ... hangs +--echo # after deadlock was encountered +--echo # +connect (con1,localhost,root,,); +create table t1(id int primary key,v int)engine=innodb; +insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5),(6,6),(7,7); +create table t2 like t1; + +--connection con1 +begin; +update t1 set v=id*2 where id=1; + +--connection default +begin; +update t1 set v=id*2 where id=2; + +--connection con1 +--error 1205 +update t1 set v=id*2 where id=2; + +--connection default +--error 1205 +insert ignore into t2 select * from t1 where id=1; +rollback; + +--connection con1 +rollback; + +--connection default +disconnect con1; +drop table t1, t2; + + +--echo # --echo # Bug#41756 Strange error messages about locks from InnoDB --echo # --disable_warnings === modified file 'mysql-test/t/innodb_mysql.test' --- a/mysql-test/t/innodb_mysql.test 2009-12-03 11:19:05 +0000 +++ b/mysql-test/t/innodb_mysql.test 2010-01-15 15:27:55 +0000 @@ -491,5 +491,51 @@ EXPLAIN SELECT * FROM t1 WHERE a = 'TEST c >= '2009-10-09 00:00:00.001' AND c <= '2009-10-09 00:00:00.00'; DROP TABLE t1; +--echo # +--echo # Bug #46175: NULL read_view and consistent read assertion +--echo # + +CREATE TABLE t1(a CHAR(13),KEY(a)) ENGINE=innodb; +CREATE TABLE t2(b DATETIME,KEY(b)) ENGINE=innodb; +INSERT INTO t1 VALUES (),(); +INSERT INTO t2 VALUES (),(); +CREATE OR REPLACE VIEW v1 AS SELECT 1 FROM t2 + WHERE b =(SELECT a FROM t1 LIMIT 1); + +--disable_query_log +--disable_result_log +CONNECT (con1, localhost, root,,); +--enable_query_log +--enable_result_log +CONNECTION default; + +DELIMITER |; +CREATE PROCEDURE p1(num INT) +BEGIN + DECLARE i INT DEFAULT 0; + REPEAT + SHOW CREATE VIEW v1; + SET i:=i+1; + UNTIL i>num END REPEAT; +END| +DELIMITER ;| + +--echo # Should not crash +--disable_query_log +--disable_result_log +--send CALL p1(1000) +CONNECTION con1; +--echo # Should not crash +CALL p1(1000); + +CONNECTION default; +--reap +--enable_query_log +--enable_result_log + +DISCONNECT con1; +DROP PROCEDURE p1; +DROP VIEW v1; +DROP TABLE t1,t2; --echo End of 5.1 tests === modified file 'mysql-test/t/innodb_xtradb_bug317074.test' --- a/mysql-test/t/innodb_xtradb_bug317074.test 2009-10-28 07:52:34 +0000 +++ b/mysql-test/t/innodb_xtradb_bug317074.test 2010-01-15 15:58:25 +0000 @@ -2,7 +2,7 @@ SET @old_innodb_file_format=@@innodb_file_format; SET @old_innodb_file_per_table=@@innodb_file_per_table; -SET @old_innodb_file_format_check=@@innodb_file_format_check; +let $innodb_file_format_check_orig=`select @@innodb_file_format_check`; SET GLOBAL innodb_file_format='Barracuda'; SET GLOBAL innodb_file_per_table=ON; @@ -45,4 +45,4 @@ ALTER TABLE test1 ENGINE=MyISAM; DROP TABLE test1; SET GLOBAL innodb_file_format=@old_innodb_file_format; SET GLOBAL innodb_file_per_table=@old_innodb_file_per_table; -SET GLOBAL innodb_file_format_check=@old_innodb_file_format_check; +eval set global innodb_file_format_check=$innodb_file_format_check_orig; === modified file 'mysql-test/t/mysql.test' --- a/mysql-test/t/mysql.test 2009-10-05 13:22:23 +0000 +++ b/mysql-test/t/mysql.test 2010-01-15 15:27:55 +0000 @@ -386,10 +386,16 @@ drop tables t1, t2; # # Bug #27884: mysql --html does not quote HTML special characters in output # ---exec $MYSQL --html test -e "select '< & >' as '<'" +--write_file $MYSQLTEST_VARDIR/tmp/bug27884.sql +SELECT '< & >' AS `<`; +EOF +--exec $MYSQL --html test < $MYSQLTEST_VARDIR/tmp/bug27884.sql + +remove_file $MYSQLTEST_VARDIR/tmp/bug27884.sql; + # -# Bug #27884: mysql client + null byte +# Bug #28203: mysql client + null byte # create table t1 (a char(5)); insert into t1 values ('\0b\0'); @@ -402,5 +408,5 @@ insert into t1 values ('\0b\0'); --exec $MYSQL --xml test -e "select a from t1" drop table t1; ---echo ---echo End of tests + +--echo End of 5.0 tests === added file 'mysql-test/t/mysqlbinlog2-master.opt' --- a/mysql-test/t/mysqlbinlog2-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog2-master.opt 2010-01-19 10:36:52 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row-master.opt' --- a/mysql-test/t/mysqlbinlog_row-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_innodb-master.opt' --- a/mysql-test/t/mysqlbinlog_row_innodb-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_innodb-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_myisam-master.opt' --- a/mysql-test/t/mysqlbinlog_row_myisam-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_myisam-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_trans-master.opt' --- a/mysql-test/t/mysqlbinlog_row_trans-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_trans-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === modified file 'mysql-test/t/mysqltest.test' --- a/mysql-test/t/mysqltest.test 2009-10-08 09:30:03 +0000 +++ b/mysql-test/t/mysqltest.test 2010-01-11 13:15:28 +0000 @@ -9,6 +9,14 @@ # Save the initial number of concurrent sessions --source include/count_sessions.inc +# Some tests below connect/disconnect rapidly in a loop. This causes a race +# where mysqld may not have time to register the previous disconnects before +# new connects, and eventually we run out of connections. So we need to +# increase the maximum. +let $saved_max_connections = `SELECT @@global.max_connections`; +SET GLOBAL max_connections = 1000; + + # ============================================================================ # # Test of mysqltest itself @@ -2319,3 +2327,7 @@ disconnect $y; connection default; # Wait till we reached the initial number of concurrent sessions --source include/wait_until_count_sessions.inc + +--disable_query_log +--eval SET GLOBAL max_connections = $saved_max_connections +--enable_query_log === modified file 'mysql-test/t/olap.test' --- a/mysql-test/t/olap.test 2009-10-30 15:54:53 +0000 +++ b/mysql-test/t/olap.test 2009-12-08 09:26:11 +0000 @@ -390,4 +390,17 @@ SELECT DISTINCT b FROM t1, t2 GROUP BY a DROP TABLE t1, t2; +--echo # +--echo # Bug #48475: DISTINCT is ignored with GROUP BY WITH ROLLUP +--echo # and only const tables + +CREATE TABLE t1 (a INT); +CREATE TABLE t2 (b INT); +INSERT INTO t1 VALUES (1); +INSERT INTO t2 VALUES (1); + +SELECT DISTINCT b FROM t1, t2 GROUP BY a, b WITH ROLLUP; + +DROP TABLE t1, t2; + --echo End of 5.0 tests === modified file 'mysql-test/t/openssl_1.test' --- a/mysql-test/t/openssl_1.test 2009-05-25 13:00:18 +0000 +++ b/mysql-test/t/openssl_1.test 2010-01-29 10:42:31 +0000 @@ -15,8 +15,10 @@ insert into t1 values (5); grant select on test.* to ssl_user1@localhost require SSL; grant select on test.* to ssl_user2@localhost require cipher "DHE-RSA-AES256-SHA"; -grant select on test.* to ssl_user3@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com"; -grant select on test.* to ssl_user4@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com" ISSUER "/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB"; +grant select on test.* to ssl_user3@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; +grant select on test.* to ssl_user4@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org" ISSUER "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; grant select on test.* to ssl_user5@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "xxx"; flush privileges; === modified file 'mysql-test/t/order_by.test' --- a/mysql-test/t/order_by.test 2009-12-03 11:19:05 +0000 +++ b/mysql-test/t/order_by.test 2010-01-15 15:27:55 +0000 @@ -869,6 +869,31 @@ SELECT DROP TABLE t1, t2, t3; +--echo # +--echo # Bug #42760: Select doesn't return desired results when we have null +--echo # values +--echo # + +CREATE TABLE t1 ( + a INT, + c INT, + UNIQUE KEY a_c (a,c), + KEY (a)); + +INSERT INTO t1 VALUES (1, 10), (2, NULL); + +--echo # Must use ref-or-null on the a_c index +EXPLAIN +SELECT 1 AS col FROM t1 WHERE a=2 AND (c=10 OR c IS NULL) ORDER BY c; +--echo # Must return 1 row +SELECT 1 AS col FROM t1 WHERE a=2 AND (c=10 OR c IS NULL) ORDER BY c; + +DROP TABLE t1; + + +--echo End of 5.0 tests + + # # Bug #35206: select query result different if the key is indexed or not # === modified file 'mysql-test/t/partition.test' --- a/mysql-test/t/partition.test 2009-12-03 11:19:05 +0000 +++ b/mysql-test/t/partition.test 2010-01-15 15:27:55 +0000 @@ -15,6 +15,15 @@ drop table if exists t1, t2; --enable_warnings # +# Bug#48276: can't add column if subpartition exists +CREATE TABLE t1 (a INT, b INT) +PARTITION BY LIST (a) +SUBPARTITION BY HASH (b) +(PARTITION p1 VALUES IN (1)); +ALTER TABLE t1 ADD COLUMN c INT; +DROP TABLE t1; + +# # Bug#46639: 1030 (HY000): Got error 124 from storage engine on # INSERT ... SELECT ... CREATE TABLE t1 ( @@ -62,6 +71,17 @@ SHOW CREATE TABLE t1; DROP TABLE t1; # +# Bug#45904: Error when CHARSET=utf8 and subpartitioning +# +create table t1 (a int NOT NULL, b varchar(5) NOT NULL) +default charset=utf8 +partition by list (a) +subpartition by key (b) +(partition p0 values in (1), + partition p1 values in (2)); +drop table t1; + +# # Bug#44059: rec_per_key on empty partition gives weird optimiser results # create table t1 (a int, b int, key(a)) @@ -2035,11 +2055,14 @@ DROP TABLE t1; --echo # --echo # Bug #45807: crash accessing partitioned table and sql_mode --echo # contains ONLY_FULL_GROUP_BY +--echo # Bug#46923: select count(*) from partitioned table fails with +--echo # ONLY_FULL_GROUP_BY --echo # SET SESSION SQL_MODE='ONLY_FULL_GROUP_BY'; CREATE TABLE t1(id INT,KEY(id)) ENGINE=MYISAM PARTITION BY HASH(id) PARTITIONS 2; +SELECT COUNT(*) FROM t1; DROP TABLE t1; SET SESSION SQL_MODE=DEFAULT; === modified file 'mysql-test/t/partition_innodb_semi_consistent.test' --- a/mysql-test/t/partition_innodb_semi_consistent.test 2009-01-13 22:12:16 +0000 +++ b/mysql-test/t/partition_innodb_semi_consistent.test 2010-01-28 14:49:14 +0000 @@ -187,6 +187,7 @@ SELECT * FROM t1; --echo # Switch to connection con2 connection con2; +--reap SELECT * FROM t1; connection default; === modified file 'mysql-test/t/query_cache.test' --- a/mysql-test/t/query_cache.test 2009-12-03 11:19:05 +0000 +++ b/mysql-test/t/query_cache.test 2010-01-11 13:15:28 +0000 @@ -882,6 +882,19 @@ drop procedure f4; drop table t1; set GLOBAL query_cache_size=0; +# Tests moved from main.variables due to needing query cache in server. +set GLOBAL query_cache_size=100000; +--error ER_GLOBAL_VARIABLE +set SESSION query_cache_size=10000; +set global query_cache_limit=100; +set global query_cache_size=100; +set global query_cache_type=demand; + +set GLOBAL query_cache_type=default; +set GLOBAL query_cache_limit=default; +set GLOBAL query_cache_size=default; + + --echo End of 4.1 tests # @@ -1288,6 +1301,198 @@ SHOW STATUS LIKE "Qcache_hits"; DROP TABLE t1; SET GLOBAL query_cache_size= default; +# +# Bug#6760: Add SLEEP() function (feature request) +# +# Logics of original test: +# Reveal that a query with SLEEP does not need less time than estimated. +# +# Bug#12689: SLEEP() gets incorrectly cached/optimized-away +# +# Description from bug report (slightly modified) +# +# Bug 1 (happened all time): +# SELECT * FROM t1 WHERE SLEEP(1) will only result in a sleep of 1 +# second, regardless of the number of rows in t1. +# Bug 2 (happened all time): +# Such a query will also get cached by the query cache, but should not. +# +# Notes (mleich, 2008-05) +# ======================= +# +# Experiments around +# Bug#36345 Test 'func_misc' fails on RHAS3 x86_64 +# showed that the tests for both bugs could produce in case of parallel +# artificial system time (like via ntpd) +# - decreases false alarm +# - increases false success +# +# We try here to circumvent these issues by reimplementation of the tests +# and sophisticated scripting, although the cause of the problems is a massive +# error within the setup of the testing environment. +# Tests relying on or checking derivates of the system time must never meet +# parallel manipulations of system time. +# +# Results of experiments with/without manipulation of system time, +# information_schema.processlist content, high load on testing box +# ---------------------------------------------------------------- +# Definition: Predicted_cumulative_sleep_time = +# #_of_result_rows * sleep_time_per_result_row +# +# 1. Total (real sleep time) ~= predicted_cumulative_sleep_time !! +# 2. The state of a session within the PROCESSLIST changes to 'User sleep' +# if the sessions runs a statement containing the sleep function and the +# processing of the statement is just within the phase where the sleep +# is done. (*) +# 3. NOW() and processlist.time behave "synchronous" to system time and +# show also the "jumps" caused by system time manipulations. (*) +# 4. processlist.time is unsigned, the "next" value below 0 is ~ 4G (*) +# 5. Current processlist.time ~= current real sleep time if the system time +# was not manipulated. (*) +# 6. High system load can cause delays of <= 2 seconds. +# 7. Thanks to Davi for excellent hints and ideas. +# +# (*) +# - information_schema.processlist is not available before MySQL 5.1. +# - Observation of processlist content requires a +# - "worker" session sending the query with "send" and pulling results +# with "reap" +# - session observing the processlist parallel to the worker session +# "send" and "reap" do not work in case of an embedded server. +# Conclusion: Tests based on processlist have too many restrictions. +# +# Solutions for subtests based on TIMEDIFF of values filled via NOW() +# ------------------------------------------------------------------- +# Run the following sequence three times +# 1. SELECT <start_time> +# 2. Query with SLEEP +# 3. SELECT <end_time> +# If TIMEDIFF(<end_time>,<start_time>) is at least two times within a +# reasonable range assume that we did not met errors we were looking for. +# +# It is extreme unlikely that we have two system time changes within the +# < 30 seconds runtime. Even if the unlikely happens, there are so +# frequent runs of this test on this or another testing box which will +# catch the problem. +# + +--echo #------------------------------------------------------------------------ +--echo # Tests for Bug#6760 and Bug#12689 +# Number of rows within the intended result set. +SET @row_count = 4; +# Parameter within SLEEP function +SET @sleep_time_per_result_row = 1; +# Maximum acceptable delay caused by high load on testing box +SET @max_acceptable_delay = 2; +# TIMEDIFF = time for query with sleep (mostly the time caused by SLEEP) +# + time for delays caused by high load on testing box +# Ensure that at least a reasonable fraction of TIMEDIFF belongs to the SLEEP +# by appropriate setting of variables. +# Ensure that any "judging" has a base of minimum three attempts. +# (Test 2 uses all attempts except the first one.) +if (!` SELECT (@sleep_time_per_result_row * @row_count - @max_acceptable_delay > + @sleep_time_per_result_row) AND (@row_count - 1 >= 3)`) +{ + --echo # Have to abort because of error in plausibility check + --echo ###################################################### + --vertical_results + SELECT @sleep_time_per_result_row * @row_count - @max_acceptable_delay > + @sleep_time_per_result_row AS must_be_1, + @row_count - 1 >= 3 AS must_be_also_1, + @sleep_time_per_result_row, @row_count, @max_acceptable_delay; + exit; +} +SET @@global.query_cache_size = 1024 * 64; +--disable_warnings +DROP TEMPORARY TABLE IF EXISTS t_history; +DROP TABLE IF EXISTS t1; +--enable_warnings +CREATE TEMPORARY TABLE t_history (attempt SMALLINT, +start_ts DATETIME, end_ts DATETIME, +start_cached INTEGER, end_cached INTEGER); +CREATE TABLE t1 (f1 BIGINT); +let $num = `SELECT @row_count`; +--disable_query_log +begin; +while ($num) +{ + INSERT INTO t1 VALUES (1); + dec $num; +} +commit; +--enable_query_log + +let $loops = 4; +let $num = $loops; +while ($num) +{ + let $Qcache_queries_in_cache = + query_get_value(SHOW STATUS LIKE 'Qcache_queries_in_cache', Value, 1); + eval + INSERT INTO t_history + SET attempt = $loops - $num + 1, start_ts = NOW(), + start_cached = $Qcache_queries_in_cache; + SELECT *, SLEEP(@sleep_time_per_result_row) FROM t1; + # + # Do not determine Qcache_queries_in_cache before updating end_ts. The SHOW + # might cost too much time on an overloaded box. + eval + UPDATE t_history SET end_ts = NOW() + WHERE attempt = $loops - $num + 1; + let $Qcache_queries_in_cache = + query_get_value(SHOW STATUS LIKE 'Qcache_queries_in_cache', Value, 1); + eval + UPDATE t_history SET end_cached = $Qcache_queries_in_cache + WHERE attempt = $loops - $num + 1; + # DEBUG eval SELECT * FROM t_history WHERE attempt = $loops - $num + 1; + dec $num; +} + +# 1. The majority of queries with SLEEP must need a reasonable time +# -> SLEEP has an impact on runtime +# = Replacement for original Bug#6760 test +# -> total runtime is clear more needed than for one result row needed +# = Replacement for one of the original Bug#12689 tests +--echo # Test 1: Does the query with SLEEP need a reasonable time? +eval SELECT COUNT(*) >= $loops - 1 INTO @aux1 FROM t_history +WHERE TIMEDIFF(end_ts,start_ts) - @sleep_time_per_result_row * @row_count + BETWEEN 0 AND @max_acceptable_delay; +SELECT @aux1 AS "Expect 1"; +# +# 2. The majority of queries (the first one must be ignored) with SLEEP must +# need a reasonable time +# -> If we assume that the result of a cached query will be sent back +# immediate, without any sleep, than the query with SLEEP cannot be cached +# (current and intended behaviour for queries with SLEEP). +# -> It could be also not excluded that the query was cached but the server +# honoured somehow the SLEEP. Such a behaviour would be also acceptable. +# = Replacement for one of the original Bug#12689 tests +--echo # Test 2: Does the query with SLEEP need a reasonable time even in case +--echo # of the non first execution? +eval SELECT COUNT(*) >= $loops - 1 - 1 INTO @aux2 FROM t_history +WHERE TIMEDIFF(end_ts,start_ts) - @sleep_time_per_result_row * @row_count + BETWEEN 0 AND @max_acceptable_delay + AND attempt > 1; +SELECT @aux2 AS "Expect 1"; +# +# 3. The query with SLEEP should be not cached. +# -> SHOW STATUS Qcache_queries_in_cache must be not incremented after +# the execution of the query with SLEEP +--echo # Test 3: The query with SLEEP must be not cached. +eval SELECT COUNT(*) = $loops INTO @aux3 FROM t_history +WHERE end_cached = start_cached; +SELECT @aux3 AS "Expect 1"; +# +# Dump the content of t_history if one of the tests failed. +if (`SELECT @aux1 + @aux2 + @aux3 <> 3`) +{ + --echo # Some tests failed, dumping the content of t_history + SELECT * FROM t_history; +} +DROP TABLE t1; +DROP TEMPORARY TABLE t_history; +SET @@global.query_cache_size = default; + --echo End of 5.0 tests # === modified file 'mysql-test/t/query_cache_debug.test' --- a/mysql-test/t/query_cache_debug.test 2009-09-07 20:50:10 +0000 +++ b/mysql-test/t/query_cache_debug.test 2010-01-29 10:42:31 +0000 @@ -1,5 +1,4 @@ --source include/not_embedded.inc ---source include/big_test.inc --source include/have_query_cache.inc --source include/have_debug.inc === modified file 'mysql-test/t/query_cache_notembedded.test' --- a/mysql-test/t/query_cache_notembedded.test 2009-04-25 09:04:38 +0000 +++ b/mysql-test/t/query_cache_notembedded.test 2010-01-11 13:15:28 +0000 @@ -274,5 +274,52 @@ set GLOBAL query_cache_limit=default; set GLOBAL query_cache_min_res_unit=default; set GLOBAL query_cache_size=default; +# +# Bug#3583 query cache doesn't work for stored procedures +# +delimiter |; +--disable_warnings +drop table if exists t1| +--enable_warnings +create table t1 ( + id char(16) not null default '', + data int not null +)| +--disable_warnings +drop procedure if exists bug3583| +--enable_warnings +--disable_warnings +drop procedure if exists bug3583| +--enable_warnings +create procedure bug3583() +begin + declare c int; + + select * from t1; + select count(*) into c from t1; + select c; +end| + +insert into t1 values ("x", 3), ("y", 5)| +set @x = @@query_cache_size| +set global query_cache_size = 10*1024*1024| + +flush status| +flush query cache| +show status like 'Qcache_hits'| +call bug3583()| +show status like 'Qcache_hits'| +call bug3583()| +call bug3583()| +show status like 'Qcache_hits'| + +set global query_cache_size = @x| +flush status| +flush query cache| +delete from t1| +drop procedure bug3583| +drop table t1| +delimiter ;| + # Wait till we reached the initial number of concurrent sessions --source include/wait_until_count_sessions.inc === modified file 'mysql-test/t/range.test' --- a/mysql-test/t/range.test 2009-11-02 12:24:07 +0000 +++ b/mysql-test/t/range.test 2009-12-08 09:26:11 +0000 @@ -1260,4 +1260,57 @@ SELECT str_to_date('', '%Y-%m-%d'); DROP TABLE t1, t2; +--echo # +--echo # Bug#48459: valgrind errors with query using 'Range checked for each +--echo # record' +--echo # +CREATE TABLE t1 ( + a INT, + b CHAR(2), + c INT, + d INT, + KEY ( c ), + KEY ( d, a, b ( 2 ) ), + KEY ( b ( 1 ) ) +); + +INSERT INTO t1 VALUES ( NULL, 'a', 1, 2 ), ( NULL, 'a', 1, 2 ), + ( 1, 'a', 1, 2 ), ( 1, 'a', 1, 2 ); + +CREATE TABLE t2 ( + a INT, + c INT, + e INT, + KEY ( e ) +); + +INSERT INTO t2 VALUES ( 1, 1, NULL ), ( 1, 1, NULL ); + +--echo # Should not give Valgrind warnings +SELECT 1 +FROM t1, t2 +WHERE t1.d <> '1' AND t1.b > '1' +AND t1.a = t2.a AND t1.c = t2.c; + +DROP TABLE t1, t2; + +--echo # +--echo # Bug #48665: sql-bench's insert test fails due to wrong result +--echo # + +CREATE TABLE t1 (a INT, b INT, PRIMARY KEY (a)); + +INSERT INTO t1 VALUES (0,0), (1,1); + +--replace_column 1 @ 2 @ 3 @ 5 @ 6 @ 7 @ 8 @ 9 @ 10 @ +EXPLAIN +SELECT * FROM t1 FORCE INDEX (PRIMARY) + WHERE (a>=1 AND a<=2) OR (a>=4 AND a<=5) OR (a>=0 AND a <=10); + +--echo # Should return 2 rows +SELECT * FROM t1 FORCE INDEX (PRIMARY) + WHERE (a>=1 AND a<=2) OR (a>=4 AND a<=5) OR (a>=0 AND a <=10); + +DROP TABLE t1; + --echo End of 5.1 tests === modified file 'mysql-test/t/select.test' --- a/mysql-test/t/select.test 2009-10-30 14:13:13 +0000 +++ b/mysql-test/t/select.test 2009-12-15 17:08:21 +0000 @@ -3772,6 +3772,19 @@ INTO @var0; DROP TABLE t1; +--echo # +--echo # Bug #48458: simple query tries to allocate enormous amount of +--echo # memory +--echo # + +CREATE TABLE t1(a INT NOT NULL, b YEAR); +INSERT INTO t1 VALUES (); +CREATE TABLE t2(c INT); +--echo # Should not err out because of out-of-memory +SELECT 1 FROM t2 JOIN t1 ON 1=1 + WHERE a != '1' AND NOT a >= b OR NOT ROW(b,a )<> ROW(a,a); +DROP TABLE t1,t2; + --echo End of 5.0 tests @@ -3918,4 +3931,60 @@ SELECT table1 .`time_key` field2 FROM B drop table A,AA,B,BB; --echo #end of test for bug#45266 + +--echo # +--echo # BUG#48052: Valgrind warning - uninitialized value in init_read_record() +--echo # + +# Needed in 6.0 codebase +#--echo # Disable Index condition pushdown +#--replace_column 1 # +#SELECT @old_icp:=@@engine_condition_pushdown; +#SET SESSION engine_condition_pushdown = 'OFF'; + +CREATE TABLE t1 ( + pk int(11) NOT NULL, + i int(11) DEFAULT NULL, + v varchar(1) DEFAULT NULL, + PRIMARY KEY (pk) +); + +INSERT INTO t1 VALUES (2,7,'m'); +INSERT INTO t1 VALUES (3,9,'m'); + +SELECT v +FROM t1 +WHERE NOT pk > 0 +HAVING v <= 't' +ORDER BY pk; + +# Needed in 6.0 codebase +#--echo # Restore old value for Index condition pushdown +#SET SESSION engine_condition_pushdown=@old_icp; + +DROP TABLE t1; + +--echo # +--echo # Bug#49489 Uninitialized cache led to a wrong result. +--echo # +CREATE TABLE t1(c1 DOUBLE(5,4)); +INSERT INTO t1 VALUES (9.1234); +SELECT * FROM t1 WHERE c1 < 9.12345; +DROP TABLE t1; +--echo # End of test for bug#49489. + + +--echo # +--echo # Bug #49517: Inconsistent behavior while using +--echo # NULLable BIGINT and INT columns in comparison +--echo # +CREATE TABLE t1(a BIGINT UNSIGNED NOT NULL, b BIGINT NULL, c INT NULL); +INSERT INTO t1 VALUES(105, NULL, NULL); +SELECT * FROM t1 WHERE b < 102; +SELECT * FROM t1 WHERE c < 102; +SELECT * FROM t1 WHERE 102 < b; +SELECT * FROM t1 WHERE 102 < c; +DROP TABLE t1; + + --echo End of 5.1 tests === modified file 'mysql-test/t/show_check-master.opt' --- a/mysql-test/t/show_check-master.opt 2009-12-03 11:19:05 +0000 +++ b/mysql-test/t/show_check-master.opt 2010-01-27 21:53:08 +0000 @@ -1 +1 @@ ---log-output=table --slow-query-log --log-long-format --log-queries-not-using-indexes --myisam-recover="" --general-log --general-log-file="" --slow-query-log-file="" +--log-output=file --slow-query-log --log-long-format --log-queries-not-using-indexes --myisam-recover="" --general-log --general-log-file="foo" --slow-query-log-file="" === modified file 'mysql-test/t/show_check.test' --- a/mysql-test/t/show_check.test 2009-03-06 14:56:17 +0000 +++ b/mysql-test/t/show_check.test 2009-12-15 09:03:24 +0000 @@ -1207,6 +1207,28 @@ connection default; DROP USER test_u@localhost; +--echo # +--echo # Bug #48985: show create table crashes if previous access to the table +--echo # was killed +--echo # + +connect(con1,localhost,root,,); +CONNECTION con1; +LET $ID= `SELECT connection_id()`; + +CONNECTION default; +--disable_query_log +eval KILL QUERY $ID; +--enable_query_log + +CONNECTION con1; +--error ER_QUERY_INTERRUPTED +SHOW CREATE TABLE non_existent; + +CONNECTION default; +DISCONNECT con1; + + --echo End of 5.1 tests # Wait till all disconnects are completed === modified file 'mysql-test/t/sp-destruct.test' --- a/mysql-test/t/sp-destruct.test 2008-04-08 14:51:26 +0000 +++ b/mysql-test/t/sp-destruct.test 2009-11-21 11:18:21 +0000 @@ -12,6 +12,9 @@ # mysqltest should be fixed to allow REPLACE_RESULT in error message -- source include/not_embedded.inc +# Supress warnings written to the log file +call mtr.add_suppression("Column count of mysql.proc is wrong. Expected 20, found 19. The table is probably corrupted"); + # Backup proc table let $MYSQLD_DATADIR= `select @@datadir`; --copy_file $MYSQLD_DATADIR/mysql/proc.frm $MYSQLTEST_VARDIR/tmp/proc.frm @@ -38,15 +41,14 @@ create trigger t1_ai after insert on t1 # Unsupported tampering with the mysql.proc definition alter table mysql.proc drop type; ---replace_result $MYSQL_TEST_DIR . ---error ER_SP_PROC_TABLE_CORRUPT +--error ER_COL_COUNT_DOESNT_MATCH_CORRUPTED call bug14233(); ---replace_result $MYSQL_TEST_DIR . ---error ER_SP_PROC_TABLE_CORRUPT +--error ER_COL_COUNT_DOESNT_MATCH_CORRUPTED create view v1 as select bug14233_f(); ---replace_result $MYSQL_TEST_DIR . ---error ER_SP_PROC_TABLE_CORRUPT +--error ER_COL_COUNT_DOESNT_MATCH_CORRUPTED insert into t1 values (0); +--error ER_COL_COUNT_DOESNT_MATCH_CORRUPTED +show procedure status; flush table mysql.proc; @@ -155,3 +157,43 @@ drop procedure bug14233_3; # Assert: These should show nothing. show procedure status where db=DATABASE(); show function status where db=DATABASE(); + +# +# Bug#41726 upgrade from 5.0 to 5.1.30 crashes if you didn't run mysql_upgrade +# + + +--disable_warnings +DROP TABLE IF EXISTS proc_backup; +DROP PROCEDURE IF EXISTS p1; +--enable_warnings + +--echo # Backup the proc table + +RENAME TABLE mysql.proc TO proc_backup; +CREATE TABLE mysql.proc LIKE proc_backup; +FLUSH TABLE mysql.proc; + +--echo # Test with a valid table. + +CREATE PROCEDURE p1() + SET @foo = 10; +CALL p1(); +--replace_column 5 '0000-00-00 00:00:00' 6 '0000-00-00 00:00:00' +SHOW PROCEDURE STATUS; + +--echo # Modify a field of the table. + +ALTER TABLE mysql.proc MODIFY comment CHAR (32); + +--error ER_CANNOT_LOAD_FROM_TABLE +CREATE PROCEDURE p2() + SET @foo = 10; +--echo # Procedure loaded from the cache +CALL p1(); +--error ER_CANNOT_LOAD_FROM_TABLE +SHOW PROCEDURE STATUS; + +DROP TABLE mysql.proc; +RENAME TABLE proc_backup TO mysql.proc; +FLUSH TABLE mysql.proc; === modified file 'mysql-test/t/sp-security.test' --- a/mysql-test/t/sp-security.test 2009-03-06 14:56:17 +0000 +++ b/mysql-test/t/sp-security.test 2009-11-27 16:10:28 +0000 @@ -865,6 +865,65 @@ DROP PROCEDURE p_suid; DROP FUNCTION f_suid; DROP TABLE t1; +--echo # +--echo # Bug #48872 : Privileges for stored functions ignored if function name +--echo # is mixed case +--echo # + +CREATE DATABASE B48872; +USE B48872; +CREATE TABLE `TestTab` (id INT); +INSERT INTO `TestTab` VALUES (1),(2); +CREATE FUNCTION `f_Test`() RETURNS INT RETURN 123; +CREATE FUNCTION `f_Test_denied`() RETURNS INT RETURN 123; +CREATE USER 'tester'; +CREATE USER 'Tester'; +GRANT SELECT ON TABLE `TestTab` TO 'tester'; +GRANT EXECUTE ON FUNCTION `f_Test` TO 'tester'; +GRANT EXECUTE ON FUNCTION `f_Test_denied` TO 'Tester'; + +SELECT f_Test(); +SELECT * FROM TestTab; + +CONNECT (con_tester,localhost,tester,,B48872); +CONNECT (con_tester_denied,localhost,Tester,,B48872); +CONNECTION con_tester; + +SELECT * FROM TestTab; +SELECT `f_Test`(); +SELECT `F_TEST`(); +SELECT f_Test(); +SELECT F_TEST(); + +CONNECTION con_tester_denied; + +--disable_result_log +--error ER_TABLEACCESS_DENIED_ERROR +SELECT * FROM TestTab; +--error ER_PROCACCESS_DENIED_ERROR +SELECT `f_Test`(); +--error ER_PROCACCESS_DENIED_ERROR +SELECT `F_TEST`(); +--error ER_PROCACCESS_DENIED_ERROR +SELECT f_Test(); +--error ER_PROCACCESS_DENIED_ERROR +SELECT F_TEST(); +--enable_result_log +SELECT `f_Test_denied`(); +SELECT `F_TEST_DENIED`(); + +CONNECTION default; +DISCONNECT con_tester; +DISCONNECT con_tester_denied; +DROP TABLE `TestTab`; +DROP FUNCTION `f_Test`; +DROP FUNCTION `f_Test_denied`; + +USE test; +DROP USER 'tester'; +DROP USER 'Tester'; +DROP DATABASE B48872; + --echo End of 5.0 tests. # Wait till all disconnects are completed === modified file 'mysql-test/t/sp.test' --- a/mysql-test/t/sp.test 2009-10-23 13:54:58 +0000 +++ b/mysql-test/t/sp.test 2009-11-13 01:03:26 +0000 @@ -8263,6 +8263,73 @@ CALL p1; DROP PROCEDURE p1; DROP TABLE t1, t2; +--echo # +--echo # Bug#47627: SET @@{global.session}.local_variable in stored routine causes crash +--echo # Bug#48626: Crash or lost connection using SET for declared variables with @@ +--echo # + +--disable_warnings +DROP PROCEDURE IF EXISTS p1; +DROP PROCEDURE IF EXISTS p2; +DROP PROCEDURE IF EXISTS p3; +--enable_warnings + +delimiter //; + +--error ER_UNKNOWN_SYSTEM_VARIABLE +CREATE PROCEDURE p1() +BEGIN + DECLARE v INT DEFAULT 0; + SET @@SESSION.v= 10; +END// + +CREATE PROCEDURE p2() +BEGIN + DECLARE v INT DEFAULT 0; + SET v= 10; +END// +call p2()// + +--error ER_UNKNOWN_SYSTEM_VARIABLE +CREATE PROCEDURE p3() +BEGIN + DECLARE v INT DEFAULT 0; + SELECT @@SESSION.v; +END// + +--error ER_UNKNOWN_SYSTEM_VARIABLE +CREATE PROCEDURE p4() +BEGIN + DECLARE v INT DEFAULT 0; + SET @@GLOBAL.v= 10; +END// + +CREATE PROCEDURE p5() +BEGIN + DECLARE init_connect INT DEFAULT 0; + SET init_connect= 10; + SET @@GLOBAL.init_connect= 'SELECT 1'; + SET @@SESSION.IDENTITY= 1; + SELECT @@SESSION.IDENTITY; + SELECT @@GLOBAL.init_connect; + SELECT init_connect; +END// + +--error ER_UNKNOWN_SYSTEM_VARIABLE +CREATE PROCEDURE p6() +BEGIN + DECLARE v INT DEFAULT 0; + SET @@v= 0; +END// + +delimiter ;// + +SET @old_init_connect= @@GLOBAL.init_connect; +CALL p5(); +SET @@GLOBAL.init_connect= @old_init_connect; + +DROP PROCEDURE p2; +DROP PROCEDURE p5; --echo # ------------------------------------------------------------------ --echo # -- End of 5.1 tests === modified file 'mysql-test/t/sp_notembedded.test' --- a/mysql-test/t/sp_notembedded.test 2009-10-13 18:21:42 +0000 +++ b/mysql-test/t/sp_notembedded.test 2010-01-11 13:15:28 +0000 @@ -56,52 +56,6 @@ show warnings| drop procedure bug4902_2| # -# Bug#3583 query cache doesn't work for stored procedures -# ---disable_warnings -drop table if exists t1| ---enable_warnings -create table t1 ( - id char(16) not null default '', - data int not null -)| ---disable_warnings -drop procedure if exists bug3583| ---enable_warnings ---disable_warnings -drop procedure if exists bug3583| ---enable_warnings -create procedure bug3583() -begin - declare c int; - - select * from t1; - select count(*) into c from t1; - select c; -end| - -insert into t1 values ("x", 3), ("y", 5)| -set @x = @@query_cache_size| -set global query_cache_size = 10*1024*1024| - -flush status| -flush query cache| -show status like 'Qcache_hits'| -call bug3583()| -show status like 'Qcache_hits'| -call bug3583()| -call bug3583()| -show status like 'Qcache_hits'| - -set global query_cache_size = @x| -flush status| -flush query cache| -delete from t1| -drop procedure bug3583| -drop table t1| - - -# # Bug#6807 Stored procedure crash if CREATE PROCEDURE ... KILL QUERY # --disable_warnings === modified file 'mysql-test/t/trigger.test' --- a/mysql-test/t/trigger.test 2009-06-22 12:51:33 +0000 +++ b/mysql-test/t/trigger.test 2010-01-12 08:19:48 +0000 @@ -1767,68 +1767,6 @@ drop table t1, t2, t3; disconnect addconroot1; disconnect addconroot2; disconnect addconwithoutdb; -# -# Bug #26162: Trigger DML ignores low_priority_updates setting -# -CREATE TABLE t1 (id INTEGER); -CREATE TABLE t2 (id INTEGER); - -INSERT INTO t2 VALUES (1),(2); - -# trigger that produces the high priority insert, but should be low, adding -# LOW_PRIORITY fixes this -CREATE TRIGGER t1_test AFTER INSERT ON t1 FOR EACH ROW - INSERT INTO t2 VALUES (new.id); - -CONNECT (rl_holder, localhost, root,,); -CONNECT (rl_acquirer, localhost, root,,); -CONNECT (wl_acquirer, localhost, root,,); -CONNECT (rl_contender, localhost, root,,); - -CONNECTION rl_holder; -SELECT GET_LOCK('B26162',120); - -CONNECTION rl_acquirer; ---send -SELECT 'rl_acquirer', GET_LOCK('B26162',120), id FROM t2 WHERE id = 1; - -CONNECTION wl_acquirer; -SET SESSION LOW_PRIORITY_UPDATES=1; -SET GLOBAL LOW_PRIORITY_UPDATES=1; -#need to wait for rl_acquirer to lock on the B26162 lock -sleep 2; ---send -INSERT INTO t1 VALUES (5); - -CONNECTION rl_contender; -# must not "see" the row inserted by the INSERT (as it must run before the -# INSERT) ---send -SELECT 'rl_contender', id FROM t2 WHERE id > 1; - -CONNECTION rl_holder; -#need to wait for wl_acquirer and rl_contender to lock on t2 -sleep 2; -SELECT RELEASE_LOCK('B26162'); - -CONNECTION rl_acquirer; ---reap -SELECT RELEASE_LOCK('B26162'); -CONNECTION wl_acquirer; ---reap -CONNECTION rl_contender; ---reap - -CONNECTION default; -DISCONNECT rl_acquirer; -DISCONNECT wl_acquirer; -DISCONNECT rl_contender; -DISCONNECT rl_holder; - -DROP TRIGGER t1_test; -DROP TABLE t1,t2; -SET SESSION LOW_PRIORITY_UPDATES=DEFAULT; -SET GLOBAL LOW_PRIORITY_UPDATES=DEFAULT; --echo --echo Bug#28502 Triggers that update another innodb table will block --echo on X lock unnecessarily === modified file 'mysql-test/t/trigger_notembedded.test' --- a/mysql-test/t/trigger_notembedded.test 2009-06-25 10:52:50 +0000 +++ b/mysql-test/t/trigger_notembedded.test 2010-01-12 08:19:48 +0000 @@ -875,6 +875,79 @@ DROP TABLE t1; DROP DATABASE mysqltest_db1; USE test; +# +# Bug #26162: Trigger DML ignores low_priority_updates setting +# +CREATE TABLE t1 (id INTEGER); +CREATE TABLE t2 (id INTEGER); + +INSERT INTO t2 VALUES (1),(2); + +# trigger that produces the high priority insert, but should be low, adding +# LOW_PRIORITY fixes this +CREATE TRIGGER t1_test AFTER INSERT ON t1 FOR EACH ROW + INSERT INTO t2 VALUES (new.id); + +CONNECT (rl_holder, localhost, root,,); +CONNECT (rl_acquirer, localhost, root,,); +CONNECT (wl_acquirer, localhost, root,,); +CONNECT (rl_contender, localhost, root,,); + +CONNECTION rl_holder; +SELECT GET_LOCK('B26162',120); + +CONNECTION rl_acquirer; +let $rl_acquirer_thread_id = `SELECT @@pseudo_thread_id`; +--send +SELECT 'rl_acquirer', GET_LOCK('B26162',120), id FROM t2 WHERE id = 1; + +CONNECTION wl_acquirer; +let $wl_acquirer_thread_id = `SELECT @@pseudo_thread_id`; +SET SESSION LOW_PRIORITY_UPDATES=1; +SET GLOBAL LOW_PRIORITY_UPDATES=1; +#need to wait for rl_acquirer to lock on the B26162 lock +let $wait_condition= + SELECT STATE = 'User lock' FROM INFORMATION_SCHEMA.PROCESSLIST + WHERE ID = $rl_acquirer_thread_id; +--source include/wait_condition.inc +--send +INSERT INTO t1 VALUES (5); + +CONNECTION rl_contender; +# Wait until wl_acquirer is waiting for the read lock on t2 to be released. +let $wait_condition= + SELECT STATE = 'Locked' FROM INFORMATION_SCHEMA.PROCESSLIST + WHERE ID = $wl_acquirer_thread_id; +--source include/wait_condition.inc +# must not "see" the row inserted by the INSERT (as it must run before the +# INSERT) +--send +SELECT 'rl_contender', id FROM t2 WHERE id > 1; + +CONNECTION rl_holder; +#need to wait for wl_acquirer and rl_contender to lock on t2 +sleep 2; +SELECT RELEASE_LOCK('B26162'); + +CONNECTION rl_acquirer; +--reap +SELECT RELEASE_LOCK('B26162'); +CONNECTION wl_acquirer; +--reap +CONNECTION rl_contender; +--reap + +CONNECTION default; +DISCONNECT rl_acquirer; +DISCONNECT wl_acquirer; +DISCONNECT rl_contender; +DISCONNECT rl_holder; + +DROP TRIGGER t1_test; +DROP TABLE t1,t2; +SET SESSION LOW_PRIORITY_UPDATES=DEFAULT; +SET GLOBAL LOW_PRIORITY_UPDATES=DEFAULT; + --echo End of 5.0 tests. # === modified file 'mysql-test/t/type_newdecimal.test' --- a/mysql-test/t/type_newdecimal.test 2009-11-02 11:21:39 +0000 +++ b/mysql-test/t/type_newdecimal.test 2009-12-08 09:26:11 +0000 @@ -1286,3 +1286,229 @@ CREATE TABLE t1 SELECT 1 % .123456789123 DESCRIBE t1; SELECT my_col FROM t1; DROP TABLE t1; + +--echo # +--echo # Bug#45261: Crash, stored procedure + decimal +--echo # + +--disable_warnings +DROP TABLE IF EXISTS t1; +--enable_warnings + +CREATE TABLE t1 SELECT + /* 81 */ 100000000000000000000000000000000000000000000000000000000000000000000000000000001 + AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 SELECT + /* 81 */ 100000000000000000000000000000000000000000000000000000000000000000000000000000001. + AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 SELECT + /* 81 */ 100000000000000000000000000000000000000000000000000000000000000000000000000000001.1 /* 1 */ + AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 SELECT + /* 82 */ 1000000000000000000000000000000000000000000000000000000000000000000000000000000001 + AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 SELECT + /* 40 */ 1000000000000000000000000000000000000001.1000000000000000000000000000000000000001 /* 40 */ + AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 SELECT + /* 1 */ 1.10000000000000000000000000000000000000000000000000000000000000000000000000000001 /* 80 */ + AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 SELECT + /* 1 */ 1.100000000000000000000000000000000000000000000000000000000000000000000000000000001 /* 81 */ + AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 SELECT + .100000000000000000000000000000000000000000000000000000000000000000000000000000001 /* 81 */ + AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 SELECT + /* 45 */ 123456789012345678901234567890123456789012345.123456789012345678901234567890123456789012345 /* 45 */ + AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 SELECT + /* 65 */ 12345678901234567890123456789012345678901234567890123456789012345.1 /* 1 */ + AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 SELECT + /* 66 */ 123456789012345678901234567890123456789012345678901234567890123456.1 /* 1 */ + AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 SELECT + .123456789012345678901234567890123456789012345678901234567890123456 /* 66 */ + AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 AS SELECT 123.1234567890123456789012345678901 /* 31 */ AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 SELECT 1.1 + CAST(1 AS DECIMAL(65,30)) AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +--echo # +--echo # Test that the integer and decimal parts are properly calculated. +--echo # + +CREATE TABLE t1 (a DECIMAL(30,30)); +INSERT INTO t1 VALUES (0.1),(0.2),(0.3); +CREATE TABLE t2 SELECT MIN(a + 0.0000000000000000000000000000001) AS c1 FROM t1; +DESC t2; +DROP TABLE t1,t2; + +CREATE TABLE t1 (a DECIMAL(30,30)); +INSERT INTO t1 VALUES (0.1),(0.2),(0.3); +CREATE TABLE t2 SELECT IFNULL(a + 0.0000000000000000000000000000001, NULL) AS c1 FROM t1; +DESC t2; +DROP TABLE t1,t2; + +CREATE TABLE t1 (a DECIMAL(30,30)); +INSERT INTO t1 VALUES (0.1),(0.2),(0.3); +CREATE TABLE t2 SELECT CASE a WHEN 0.1 THEN 0.0000000000000000000000000000000000000000000000000000000000000000001 END AS c1 FROM t1; +DESC t2; +DROP TABLE t1,t2; + +--echo # +--echo # Test that variables get maximum precision. +--echo # + +SET @decimal= 1.1; +CREATE TABLE t1 SELECT @decimal AS c1; +DESC t1; +SELECT * FROM t1; +DROP TABLE t1; + +--echo # +--echo # Bug #45261 : Crash, stored procedure + decimal +--echo # Original test by the reporter. +--echo # + +--echo # should not crash +CREATE TABLE t1 +SELECT .123456789012345678901234567890123456789012345678901234567890123456 AS a; +DROP TABLE t1; + +delimiter |; +CREATE PROCEDURE test_proc() +BEGIN + # The las non critical CUSER definition is: + # DECLARE mycursor CURSOR FOR SELECT 1 % + # .12345678912345678912345678912345678912345678912345678912345678912 AS my_col; + DECLARE mycursor CURSOR FOR +SELECT 1 % +.123456789123456789123456789123456789123456789123456789123456789123456789123456789 + AS my_col; + + OPEN mycursor; + CLOSE mycursor; +END| +delimiter ;| +--echo # should not crash +CALL test_proc(); +DROP PROCEDURE test_proc; + +--echo # +--echo # Bug #48370 Absolutely wrong calculations with GROUP BY and +--echo # decimal fields when using IF +--echo # + +CREATE TABLE currencies (id int, rate decimal(16,4), + PRIMARY KEY (id), KEY (rate)); + +INSERT INTO currencies VALUES (11,0.7028); +INSERT INTO currencies VALUES (1,1); + +CREATE TABLE payments ( + id int, + supplier_id int, + status int, + currency_id int, + vat decimal(7,4), + PRIMARY KEY (id), + KEY currency_id (currency_id), + KEY supplier_id (supplier_id) +); + +INSERT INTO payments (id,status,vat,supplier_id,currency_id) VALUES +(3001,2,0.0000,344,11), (1,2,0.0000,1,1); + +CREATE TABLE sub_tasks ( + id int, + currency_id int, + price decimal(16,4), + discount decimal(10,4), + payment_id int, + PRIMARY KEY (id), + KEY currency_id (currency_id), + KEY payment_id (payment_id) +) ; + +INSERT INTO sub_tasks (id, price, discount, payment_id, currency_id) VALUES +(52, 12.60, 0, 3001, 11), (56, 14.58, 0, 3001, 11); + +--echo # should return 1 and the same values in col 2 and 3 +select STRAIGHT_JOIN + (1 + PAY.vat) AS mult, + SUM(ROUND((SUB.price - ROUND(ROUND(SUB.price, 2) * SUB.discount, 2)) * + CUR.rate / CUR.rate, 2) + ) v_net_with_discount, + + SUM(ROUND((SUB.price - ROUND(ROUND(SUB.price, 2) * SUB.discount, 1)) * + CUR.rate / CUR.rate , 2) + * (1 + PAY.vat) + ) v_total +from + currencies CUR, payments PAY, sub_tasks SUB +where + SUB.payment_id = PAY.id and + PAY.currency_id = CUR.id and + PAY.id > 2 +group by PAY.id + 1; + +DROP TABLE currencies, payments, sub_tasks; + + +--echo End of 5.1 tests === modified file 'mysql-test/t/type_year.test' --- a/mysql-test/t/type_year.test 2007-03-29 04:08:30 +0000 +++ b/mysql-test/t/type_year.test 2009-12-15 08:37:10 +0000 @@ -30,3 +30,109 @@ select * from t1; drop table t1; --echo End of 5.0 tests + +--echo # +--echo # Bug #49480: WHERE using YEAR columns returns unexpected results +--echo # + +CREATE TABLE t2(yy YEAR(2), c2 CHAR(4)); +CREATE TABLE t4(yyyy YEAR(4), c4 CHAR(4)); + +INSERT INTO t2 (c2) VALUES (NULL),(1970),(1999),(2000),(2001),(2069); +INSERT INTO t4 (c4) SELECT c2 FROM t2; +UPDATE t2 SET yy = c2; +UPDATE t4 SET yyyy = c4; + +SELECT * FROM t2; +SELECT * FROM t4; + +--echo # Comparison of YEAR(2) with YEAR(4) + +SELECT * FROM t2, t4 WHERE yy = yyyy; +SELECT * FROM t2, t4 WHERE yy <=> yyyy; +SELECT * FROM t2, t4 WHERE yy < yyyy; +SELECT * FROM t2, t4 WHERE yy > yyyy; + +--echo # Comparison of YEAR(2) with YEAR(2) + +SELECT * FROM t2 a, t2 b WHERE a.yy = b.yy; +SELECT * FROM t2 a, t2 b WHERE a.yy <=> b.yy; +SELECT * FROM t2 a, t2 b WHERE a.yy < b.yy; + +--echo # Comparison of YEAR(4) with YEAR(4) + +SELECT * FROM t4 a, t4 b WHERE a.yyyy = b.yyyy; +SELECT * FROM t4 a, t4 b WHERE a.yyyy <=> b.yyyy; +SELECT * FROM t4 a, t4 b WHERE a.yyyy < b.yyyy; + +--echo # Comparison with constants: + +SELECT * FROM t2 WHERE yy = NULL; +SELECT * FROM t4 WHERE yyyy = NULL; +SELECT * FROM t2 WHERE yy <=> NULL; +SELECT * FROM t4 WHERE yyyy <=> NULL; +SELECT * FROM t2 WHERE yy < NULL; +SELECT * FROM t2 WHERE yy > NULL; + +SELECT * FROM t2 WHERE yy = NOW(); +SELECT * FROM t4 WHERE yyyy = NOW(); + +SELECT * FROM t2 WHERE yy = 99; +SELECT * FROM t2 WHERE 99 = yy; +SELECT * FROM t4 WHERE yyyy = 99; + +SELECT * FROM t2 WHERE yy = 'test'; +SELECT * FROM t4 WHERE yyyy = 'test'; + +SELECT * FROM t2 WHERE yy = '1999'; +SELECT * FROM t4 WHERE yyyy = '1999'; + +SELECT * FROM t2 WHERE yy = 1999; +SELECT * FROM t4 WHERE yyyy = 1999; + +SELECT * FROM t2 WHERE yy = 1999.1; +SELECT * FROM t4 WHERE yyyy = 1999.1; + +SELECT * FROM t2 WHERE yy = 1998.9; +SELECT * FROM t4 WHERE yyyy = 1998.9; + +--echo # Coverage tests for YEAR with zero/2000 constants: + +SELECT * FROM t2 WHERE yy = 0; +SELECT * FROM t2 WHERE yy = '0'; +SELECT * FROM t2 WHERE yy = '0000'; +SELECT * FROM t2 WHERE yy = '2000'; +SELECT * FROM t2 WHERE yy = 2000; + +SELECT * FROM t4 WHERE yyyy = 0; +SELECT * FROM t4 WHERE yyyy = '0'; +SELECT * FROM t4 WHERE yyyy = '0000'; +SELECT * FROM t4 WHERE yyyy = '2000'; +SELECT * FROM t4 WHERE yyyy = 2000; + +--echo # Comparison with constants those are out of YEAR range +--echo # (coverage test for backward compatibility) + +SELECT COUNT(yy) FROM t2; +SELECT COUNT(yyyy) FROM t4; + +SELECT COUNT(*) FROM t2 WHERE yy = -1; +SELECT COUNT(*) FROM t4 WHERE yyyy > -1; +SELECT COUNT(*) FROM t2 WHERE yy > -1000000000000000000; +SELECT COUNT(*) FROM t4 WHERE yyyy > -1000000000000000000; + +SELECT COUNT(*) FROM t2 WHERE yy < 2156; +SELECT COUNT(*) FROM t4 WHERE yyyy < 2156; +SELECT COUNT(*) FROM t2 WHERE yy < 1000000000000000000; +SELECT COUNT(*) FROM t4 WHERE yyyy < 1000000000000000000; + +SELECT * FROM t2 WHERE yy < 123; +SELECT * FROM t2 WHERE yy > 123; +SELECT * FROM t4 WHERE yyyy < 123; +SELECT * FROM t4 WHERE yyyy > 123; + +DROP TABLE t2, t4; + +--echo # + +--echo End of 5.1 tests === modified file 'mysql-test/t/udf.test' --- a/mysql-test/t/udf.test 2009-09-07 09:57:22 +0000 +++ b/mysql-test/t/udf.test 2010-01-22 21:19:21 +0000 @@ -56,9 +56,9 @@ select reverse_lookup(); --disable_result_log select reverse_lookup("127.0.0.1"); select reverse_lookup(127,0,0,1); +select reverse_lookup("localhost"); --enable_result_log -select reverse_lookup("localhost"); --error ER_CANT_INITIALIZE_UDF select avgcost(); --error ER_CANT_INITIALIZE_UDF @@ -342,29 +342,6 @@ drop function myfunc_double; drop function myfunc_int; # -# Bug #28921: Queries containing UDF functions are cached -# - ---replace_result $UDF_EXAMPLE_LIB UDF_EXAMPLE_LIB -eval CREATE FUNCTION metaphon RETURNS STRING SONAME "$UDF_EXAMPLE_LIB"; -create table t1 (a char); - -set GLOBAL query_cache_size=1355776; -reset query cache; - -select metaphon('MySQL') from t1; -show status like "Qcache_hits"; -show status like "Qcache_queries_in_cache"; - -select metaphon('MySQL') from t1; -show status like "Qcache_hits"; -show status like "Qcache_queries_in_cache"; - -drop table t1; -drop function metaphon; -set GLOBAL query_cache_size=default; - -# # Bug#28318 CREATE FUNCTION (UDF) requires a schema # === added file 'mysql-test/t/udf_query_cache-master.opt' --- a/mysql-test/t/udf_query_cache-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/udf_query_cache-master.opt 2010-01-11 13:15:28 +0000 @@ -0,0 +1 @@ +$UDF_EXAMPLE_LIB_OPT === added file 'mysql-test/t/udf_query_cache.test' --- a/mysql-test/t/udf_query_cache.test 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/udf_query_cache.test 2010-01-11 13:15:28 +0000 @@ -0,0 +1,35 @@ +--source include/have_udf.inc +--source include/have_query_cache.inc +# +# To run this tests the "sql/udf_example.c" need to be compiled into +# udf_example.so and LD_LIBRARY_PATH should be setup to point out where +# the library are. +# + +--disable_warnings +drop table if exists t1; +--enable_warnings + +# +# Bug #28921: Queries containing UDF functions are cached +# + +--replace_result $UDF_EXAMPLE_LIB UDF_EXAMPLE_LIB +eval CREATE FUNCTION metaphon RETURNS STRING SONAME "$UDF_EXAMPLE_LIB"; +create table t1 (a char); + +set GLOBAL query_cache_size=1355776; +reset query cache; + +select metaphon('MySQL') from t1; +show status like "Qcache_hits"; +show status like "Qcache_queries_in_cache"; + +select metaphon('MySQL') from t1; +show status like "Qcache_hits"; +show status like "Qcache_queries_in_cache"; + +drop table t1; +drop function metaphon; +set GLOBAL query_cache_size=default; + === modified file 'mysql-test/t/variables.test' --- a/mysql-test/t/variables.test 2009-09-07 20:50:10 +0000 +++ b/mysql-test/t/variables.test 2010-01-11 13:15:28 +0000 @@ -28,8 +28,6 @@ set @my_myisam_max_sort_file_size =@@glo set @my_net_buffer_length =@@global.net_buffer_length; set @my_net_write_timeout =@@global.net_write_timeout; set @my_net_read_timeout =@@global.net_read_timeout; -set @my_query_cache_limit =@@global.query_cache_limit; -set @my_query_cache_type =@@global.query_cache_type; set @my_rpl_recovery_rank =@@global.rpl_recovery_rank; set @my_server_id =@@global.server_id; set @my_slow_launch_time =@@global.slow_launch_time; @@ -138,7 +136,6 @@ show local variables like 'storage_engin select * from information_schema.session_variables where variable_name like 'storage_engine'; show global variables like 'storage_engine'; select * from information_schema.global_variables where variable_name like 'storage_engine'; -set GLOBAL query_cache_size=100000; set GLOBAL myisam_max_sort_file_size=2000000; show global variables like 'myisam_max_sort_file_size'; @@ -255,8 +252,6 @@ set storage_engine=UNKNOWN_TABLE_TYPE; --error ER_WRONG_VALUE_FOR_VAR set storage_engine=MERGE, big_tables=2; show local variables like 'storage_engine'; ---error ER_GLOBAL_VARIABLE -set SESSION query_cache_size=10000; --error ER_NO_DEFAULT set GLOBAL storage_engine=DEFAULT; --error ER_UNKNOWN_CHARACTER_SET @@ -334,9 +329,6 @@ set myisam_sort_buffer_size=100; set global net_buffer_length=100; set net_read_timeout=100; set net_write_timeout=100; -set global query_cache_limit=100; -set global query_cache_size=100; -set global query_cache_type=demand; set read_buffer_size=100; set read_rnd_buffer_size=100; set global rpl_recovery_rank=100; @@ -822,8 +814,6 @@ set global myisam_max_sort_file_size =@m set global net_buffer_length =@my_net_buffer_length; set global net_write_timeout =@my_net_write_timeout; set global net_read_timeout =@my_net_read_timeout; -set global query_cache_limit =@my_query_cache_limit; -set global query_cache_type =@my_query_cache_type; set global rpl_recovery_rank =@my_rpl_recovery_rank; set global server_id =@my_server_id; set global slow_launch_time =@my_slow_launch_time; === modified file 'mysql-test/valgrind.supp' --- a/mysql-test/valgrind.supp 2009-12-03 11:19:05 +0000 +++ b/mysql-test/valgrind.supp 2010-01-18 12:56:10 +0000 @@ -991,3 +991,15 @@ fun:_dl_allocate_tls fun:pthread_create* } + +# +# Bug in Glibc 2.9: http://sourceware.org/bugzilla/show_bug.cgi?id=10391 +# Fixed in latest Glibc, but suppressed here for running tests on hosts +# with older Glibc version. +# +{ + Glibc bug in __libc_res_nsend + Memcheck:Cond + fun:__libc_res_nsend + fun:__libc_res_nquery +} === modified file 'mysys/my_file.c' --- a/mysys/my_file.c 2006-12-23 19:20:40 +0000 +++ b/mysys/my_file.c 2010-01-28 11:35:10 +0000 @@ -51,7 +51,7 @@ static uint set_max_open_files(uint max_ DBUG_PRINT("info", ("rlim_cur: %u rlim_max: %u", (uint) rlimit.rlim_cur, (uint) rlimit.rlim_max)); - if (rlimit.rlim_cur == RLIM_INFINITY) + if ((ulonglong) rlimit.rlim_cur == (ulonglong) RLIM_INFINITY) rlimit.rlim_cur = max_file_limit; if (rlimit.rlim_cur >= max_file_limit) DBUG_RETURN(rlimit.rlim_cur); /* purecov: inspected */ === modified file 'mysys/my_getopt.c' --- a/mysys/my_getopt.c 2009-12-03 11:19:05 +0000 +++ b/mysys/my_getopt.c 2010-01-15 15:27:55 +0000 @@ -414,17 +414,11 @@ invalid value '%s'", (optp->var_type & GET_TYPE_MASK) == GET_ENUM)) { if (optend == disabled_my_option) - if ((optp->var_type & GET_TYPE_MASK) == GET_BOOL) - *((my_bool*) value)= (my_bool) 0; - else - *((ulong*) value)= (ulong) 0; + init_one_value(optp, value, 0); else { if (!optend) /* No argument -> enable option */ - if ((optp->var_type & GET_TYPE_MASK) == GET_BOOL) - *((my_bool*) value)= (my_bool) 1; - else - *((ulong*) value)= (ulong) 1; + init_one_value(optp, value, 1); else argument= optend; } === modified file 'mysys/my_sync.c' --- a/mysys/my_sync.c 2010-01-06 21:27:53 +0000 +++ b/mysys/my_sync.c 2010-01-15 15:27:55 +0000 @@ -104,11 +104,11 @@ int my_sync_dir(const char *dir_name __a myf my_flags __attribute__((unused))) { #ifdef NEED_EXPLICIT_SYNC_DIR - DBUG_ENTER("my_sync_dir"); - DBUG_PRINT("my",("Dir: '%s' my_flags: %d", dir_name, my_flags)); File dir_fd; int res= 0; const char *correct_dir_name; + DBUG_ENTER("my_sync_dir"); + DBUG_PRINT("my",("Dir: '%s' my_flags: %d", dir_name, my_flags)); /* Sometimes the path does not contain an explicit directory */ correct_dir_name= (dir_name[0] == 0) ? cur_dir_name : dir_name; /* === modified file 'mysys/my_thr_init.c' --- a/mysys/my_thr_init.c 2009-12-03 11:19:05 +0000 +++ b/mysys/my_thr_init.c 2010-01-29 18:42:22 +0000 @@ -317,7 +317,7 @@ my_bool my_thread_init(void) /* Skip initialization if the thread specific variable is already initialized */ - if (THR_KEY_mysys.id) + if (THR_KEY_mysys.init) goto end; tmp= &THR_KEY_mysys; #endif === modified file 'scripts/make_win_bin_dist' --- a/scripts/make_win_bin_dist 2009-12-03 11:19:05 +0000 +++ b/scripts/make_win_bin_dist 2010-01-15 15:27:55 +0000 @@ -352,7 +352,7 @@ mkdir $DESTDIR/mysql-test cp mysql-test/mysql-test-run.pl $DESTDIR/mysql-test/ cp mysql-test/mysql-stress-test.pl $DESTDIR/mysql-test/ cp mysql-test/README $DESTDIR/mysql-test/ -cp -R mysql-test/{t,r,include,suite,std_data,lib} $DESTDIR/mysql-test/ +cp -R mysql-test/{t,r,include,suite,std_data,lib,collections} $DESTDIR/mysql-test/ rm -rf $DESTDIR/mysql-test/lib/My/SafeProcess/my_safe_kill.{dir,vcproj} rm -rf $DESTDIR/mysql-test/lib/My/SafeProcess/my_safe_process.{dir,vcproj} === modified file 'scripts/mysql_secure_installation.pl.in' --- a/scripts/mysql_secure_installation.pl.in 2007-12-28 21:58:54 +0000 +++ b/scripts/mysql_secure_installation.pl.in 2009-11-03 21:34:01 +0000 @@ -17,16 +17,41 @@ # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA use Fcntl; +use File::Spec; +use if $^O eq 'MSWin32', 'Term::ReadKey' => qw/ReadMode/; use strict; my $config = ".my.cnf.$$"; my $command = ".mysql.$$"; my $hadpass = 0; +my $mysql; # How to call the mysql client +my $rootpass = ""; -# FIXME -# trap "interrupt" 2 -my $rootpass = ""; +$SIG{QUIT} = $SIG{INT} = sub { + print "\nAborting!\n\n"; + echo_on(); + cleanup(); + exit 1; +}; + + +END { + # Remove temporary files, even if exiting via die(), etc. + cleanup(); +} + + +sub read_without_echo { + my ($prompt) = @_; + print $prompt; + echo_off(); + my $answer = <STDIN>; + echo_on(); + print "\n"; + chomp($answer); + return $answer; +} sub echo_on { if ($^O eq 'MSWin32') { @@ -55,6 +80,25 @@ sub write_file { } sub prepare { + # Locate the mysql client; look in current directory first, then + # in path + our $SAVEERR; # Suppress Perl warning message + open SAVEERR, ">& STDERR"; + close STDERR; + for my $m (File::Spec->catfile('bin', 'mysql'), 'mysql') { + # mysql --version should always work + qx($m --no-defaults --version); + next unless $? == 0; + + $mysql = $m; + last; + } + open STDERR, ">& SAVEERR"; + + die "Can't find a 'mysql' client in PATH or ./bin\n" + unless $mysql; + + # Create safe files to avoid leaking info to other users foreach my $file ( $config, $command ) { next if -f $file; # Already exists local *FILE; @@ -64,30 +108,50 @@ sub prepare { } } +# Simple escape mechanism (\-escape any ' and \), suitable for two contexts: +# - single-quoted SQL strings +# - single-quoted option values on the right hand side of = in my.cnf +# +# These two contexts don't handle escapes identically. SQL strings allow +# quoting any character (\C => C, for any C), but my.cnf parsing allows +# quoting only \, ' or ". For example, password='a\b' quotes a 3-character +# string in my.cnf, but a 2-character string in SQL. +# +# This simple escape works correctly in both places. +sub basic_single_escape { + my ($str) = @_; + # Inside a character class, \ is not special; this escapes both \ and ' + $str =~ s/([\'])/\\$1/g; + return $str; +} + sub do_query { my $query = shift; write_file($command, $query); - system("mysql --defaults-file=$config < $command"); - return $?; + my $rv = system("$mysql --defaults-file=$config < $command"); + # system() returns -1 if exec fails (e.g., command not found, etc.); die + # in this case because nothing is going to work + die "Failed to execute mysql client '$mysql'\n" if $rv == -1; + # Return true if query executed OK, or false if there was some problem + # (for example, SQL error or wrong password) + return ($rv == 0 ? 1 : undef); } sub make_config { my $password = shift; + my $esc_pass = basic_single_escape($rootpass); write_file($config, "# mysql_secure_installation config file", "[mysql]", "user=root", - "password=$rootpass"); + "password='$esc_pass'"); } sub get_root_password { - my $status = 1; - while ( $status == 1 ) { - echo_off(); - print "Enter current password for root (enter for none): "; - my $password = <STDIN>; - echo_on(); + my $attempts = 3; + for (;;) { + my $password = read_without_echo("Enter current password for root (enter for none): "); if ( $password ) { $hadpass = 1; } else { @@ -95,64 +159,56 @@ sub get_root_password { } $rootpass = $password; make_config($rootpass); - do_query(""); - $status = $?; + last if do_query(""); + + die "Unable to connect to the server as root user, giving up.\n" + if --$attempts == 0; } print "OK, successfully used password, moving on...\n\n"; } sub set_root_password { - echo_off(); - print "New password: "; - my $password1 = <STDIN>; - print "\nRe-enter new password: "; - my $password2 = <STDIN>; - print "\n"; - echo_on(); - - if ( $password1 eq $password2 ) { - print "Sorry, passwords do not match.\n\n"; - return 1; - } + my $password1; + for (;;) { + $password1 = read_without_echo("New password: "); + + if ( !$password1 ) { + print "Sorry, you can't use an empty password here.\n\n"; + next; + } - if ( !$password1 ) { - print "Sorry, you can't use an empty password here.\n\n"; - return 1; - } + my $password2 = read_without_echo("Re-enter new password: "); - do_query("UPDATE mysql.user SET Password=PASSWORD('$password1') WHERE User='root';"); - if ( $? == 0 ) { - print "Password updated successfully!\n"; - print "Reloading privilege tables..\n"; - if ( !reload_privilege_tables() ) { - exit 1; + if ( $password1 ne $password2 ) { + print "Sorry, passwords do not match.\n\n"; + next; } - print "\n"; - $rootpass = $password1; - make_config($rootpass); - } else { - print "Password update failed!\n"; - exit 1; + + last; } - return 0; + my $esc_pass = basic_single_escape($password1); + do_query("UPDATE mysql.user SET Password=PASSWORD('$esc_pass') WHERE User='root';") + or die "Password update failed!\n"; + + print "Password updated successfully!\n"; + print "Reloading privilege tables..\n"; + reload_privilege_tables() + or die "Can not continue.\n"; + + print "\n"; + $rootpass = $password1; + make_config($rootpass); } sub remove_anonymous_users { - do_query("DELETE FROM mysql.user WHERE User='';"); - if ( $? == 0 ) { - print " ... Success!\n"; - } else { - print " ... Failed!\n"; - exit 1; - } - - return 0; + do_query("DELETE FROM mysql.user WHERE User='';") + or die print " ... Failed!\n"; + print " ... Success!\n"; } sub remove_remote_root { - do_query("DELETE FROM mysql.user WHERE User='root' AND Host!='localhost';"); - if ( $? == 0 ) { + if (do_query("DELETE FROM mysql.user WHERE User='root' AND Host!='localhost';")) { print " ... Success!\n"; } else { print " ... Failed!\n"; @@ -161,44 +217,31 @@ sub remove_remote_root { sub remove_test_database { print " - Dropping test database...\n"; - do_query("DROP DATABASE test;"); - if ( $? == 0 ) { + if (do_query("DROP DATABASE test;")) { print " ... Success!\n"; } else { print " ... Failed! Not critical, keep moving...\n"; } print " - Removing privileges on test database...\n"; - do_query("DELETE FROM mysql.db WHERE Db='test' OR Db='test\\_%'"); - if ( $? == 0 ) { + if (do_query("DELETE FROM mysql.db WHERE Db='test' OR Db='test\\_%'")) { print " ... Success!\n"; } else { print " ... Failed! Not critical, keep moving...\n"; } - - return 0; } sub reload_privilege_tables { - do_query("FLUSH PRIVILEGES;"); - if ( $? == 0 ) { + if (do_query("FLUSH PRIVILEGES;")) { print " ... Success!\n"; - return 0; + return 1; } else { print " ... Failed!\n"; - return 1; + return undef; } } -sub interrupt { - print "\nAborting!\n\n"; - cleanup(); - echo_on(); - exit 1; -} - sub cleanup { - print "Cleaning up...\n"; unlink($config,$command); } @@ -242,11 +285,7 @@ my $reply = <STDIN>; if ( $reply =~ /n/i ) { print " ... skipping.\n"; } else { - my $status = 1; - while ( $status == 1 ) { - set_root_password(); - $status = $?; - } + set_root_password(); } print "\n"; @@ -334,8 +373,6 @@ if ( $reply =~ /n/i ) { } print "\n"; -cleanup(); - print <<HERE; === modified file 'scripts/mysql_secure_installation.sh' --- a/scripts/mysql_secure_installation.sh 2009-10-23 16:48:54 +0000 +++ b/scripts/mysql_secure_installation.sh 2010-01-15 15:27:55 +0000 @@ -189,16 +189,39 @@ prepare() { } do_query() { - echo $1 >$command + echo "$1" >$command + #sed 's,^,> ,' < $command # Debugging $bindir/mysql --defaults-file=$config <$command return $? } +# Simple escape mechanism (\-escape any ' and \), suitable for two contexts: +# - single-quoted SQL strings +# - single-quoted option values on the right hand side of = in my.cnf +# +# These two contexts don't handle escapes identically. SQL strings allow +# quoting any character (\C => C, for any C), but my.cnf parsing allows +# quoting only \, ' or ". For example, password='a\b' quotes a 3-character +# string in my.cnf, but a 2-character string in SQL. +# +# This simple escape works correctly in both places. +basic_single_escape () { + # The quoting on this sed command is a bit complex. Single-quoted strings + # don't allow *any* escape mechanism, so they cannot contain a single + # quote. The string sed gets (as argv[1]) is: s/$['\]$/\\\1/g + # + # Inside a character class, \ and ' are not special, so the ['\] character + # class is balanced and contains two characters. + echo "$1" | sed 's/$['"'"'\]$/\\\1/g' +} + make_config() { echo "# mysql_secure_installation config file" >$config echo "[mysql]" >>$config echo "user=root" >>$config - echo "password=$rootpass" >>$config + esc_pass=`basic_single_escape "$rootpass"` + echo "password='$esc_pass'" >>$config + #sed 's,^,> ,' < $config # Debugging } get_root_password() { @@ -245,13 +268,12 @@ set_root_password() { return 1 fi - do_query "UPDATE mysql.user SET Password=PASSWORD('$password1') WHERE User='root';" + esc_pass=`basic_single_escape "$password1"` + do_query "UPDATE mysql.user SET Password=PASSWORD('$esc_pass') WHERE User='root';" if [ $? -eq 0 ]; then echo "Password updated successfully!" echo "Reloading privilege tables.." - if ! reload_privilege_tables; then - exit 1 - fi + reload_privilege_tables || exit 1 echo rootpass=$password1 make_config === modified file 'sql-common/client.c' --- a/sql-common/client.c 2009-12-03 15:26:54 +0000 +++ b/sql-common/client.c 2010-01-29 18:42:22 +0000 @@ -1863,6 +1863,7 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons uint port, const char *unix_socket,ulong client_flag) { char buff[NAME_LEN+USERNAME_LENGTH+100]; + char error_string[1024]; char *end,*host_info= NULL; my_socket sock; in_addr_t ip_addr; @@ -1939,7 +1940,8 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons #if defined(HAVE_SMEM) if ((!mysql->options.protocol || mysql->options.protocol == MYSQL_PROTOCOL_MEMORY) && - (!host || !strcmp(host,LOCAL_HOST))) + (!host || !strcmp(host,LOCAL_HOST)) && + mysql->options.shared_memory_base_name) { if ((create_shared_memory(mysql,net, mysql->options.connect_timeout)) == INVALID_HANDLE_VALUE) @@ -1948,7 +1950,7 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons ("host: '%s' socket: '%s' shared memory: %s have_tcpip: %d", host ? host : "<null>", unix_socket ? unix_socket : "<null>", - (int) mysql->options.shared_memory_base_name, + mysql->options.shared_memory_base_name, (int) have_tcpip)); if (mysql->options.protocol == MYSQL_PROTOCOL_MEMORY) goto error; @@ -2331,9 +2333,14 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons /* Connect to the server */ DBUG_PRINT("info", ("IO layer change in progress...")); if (sslconnect(ssl_fd, mysql->net.vio, - (long) (mysql->options.connect_timeout))) + (long) (mysql->options.connect_timeout), + error_string)) { - set_mysql_error(mysql, CR_SSL_CONNECTION_ERROR, unknown_sqlstate); + set_mysql_extended_error(mysql, CR_SSL_CONNECTION_ERROR, + unknown_sqlstate, + "SSL error: %s", + error_string[0] ? error_string : + ER(CR_SSL_CONNECTION_ERROR)); goto error; } DBUG_PRINT("info", ("IO layer change done!")); @@ -2746,6 +2753,13 @@ void mysql_detach_stmt_list(LIST **stmt_ } +/* + Close a MySQL connection and free all resources attached to it. + + This function is coded in such that it can be called multiple times + (As some clients call this after mysql_real_connect() fails) +*/ + void STDCALL mysql_close(MYSQL *mysql) { DBUG_ENTER("mysql_close"); @@ -2779,10 +2793,16 @@ void STDCALL mysql_close(MYSQL *mysql) } #endif if (mysql != mysql->master) + { mysql_close(mysql->master); + mysql->master= 0; + } #ifndef MYSQL_SERVER if (mysql->thd) + { (*mysql->methods->free_embedded_thd)(mysql); + mysql->thd= 0; + } #endif if (mysql->free_me) my_free((uchar*) mysql,MYF(0)); === modified file 'sql/CMakeLists.txt' --- a/sql/CMakeLists.txt 2009-12-03 11:19:05 +0000 +++ b/sql/CMakeLists.txt 2010-01-29 18:42:22 +0000 @@ -97,7 +97,7 @@ SET_TARGET_PROPERTIES(mysqld PROPERTIES SET (MYSQLD_CORE_LIBS mysys zlib dbug strings yassl taocrypt vio regex sql libevent) TARGET_LINK_LIBRARIES(mysqld ${MYSQLD_CORE_LIBS} ${MYSQLD_STATIC_ENGINE_LIBS}) -TARGET_LINK_LIBRARIES(mysqld ws2_32.lib) +TARGET_LINK_LIBRARIES(mysqld ws2_32.lib psapi.lib) IF(MSVC AND NOT WITHOUT_DYNAMIC_PLUGINS) === modified file 'sql/event_db_repository.cc' --- a/sql/event_db_repository.cc 2009-12-03 11:19:05 +0000 +++ b/sql/event_db_repository.cc 2010-01-15 15:27:55 +0000 @@ -26,7 +26,7 @@ */ static -const TABLE_FIELD_W_TYPE event_table_fields[ET_FIELD_COUNT] = +const TABLE_FIELD_TYPE event_table_fields[ET_FIELD_COUNT] = { { { C_STRING_WITH_LEN("db") }, @@ -151,6 +151,24 @@ const TABLE_FIELD_W_TYPE event_table_fie } }; +static const TABLE_FIELD_DEF + event_table_def= {ET_FIELD_COUNT, event_table_fields}; + +class Event_db_intact : public Table_check_intact +{ +protected: + void report_error(uint, const char *fmt, ...) + { + va_list args; + va_start(args, fmt); + error_log_print(ERROR_LEVEL, fmt, args); + va_end(args); + } +}; + +/** In case of an error, a message is printed to the error log. */ +static Event_db_intact table_intact; + /** Puts some data common to CREATE and ALTER EVENT into a row. @@ -1117,10 +1135,8 @@ Event_db_repository::check_system_tables } else { - if (table_check_intact(tables.table, MYSQL_DB_FIELD_COUNT, - mysql_db_table_fields)) + if (table_intact.check(tables.table, &mysql_db_table_def)) ret= 1; - /* in case of an error, the message is printed inside table_check_intact */ close_thread_tables(thd); } @@ -1154,9 +1170,8 @@ Event_db_repository::check_system_tables } else { - if (table_check_intact(tables.table, ET_FIELD_COUNT, event_table_fields)) + if (table_intact.check(tables.table, &event_table_def)) ret= 1; - /* in case of an error, the message is printed inside table_check_intact */ close_thread_tables(thd); } === modified file 'sql/field.cc' --- a/sql/field.cc 2009-12-03 11:19:05 +0000 +++ b/sql/field.cc 2010-01-15 15:27:55 +0000 @@ -2487,6 +2487,50 @@ Field_new_decimal::Field_new_decimal(uin } +Field *Field_new_decimal::create_from_item (Item *item) +{ + uint8 dec= item->decimals; + uint8 intg= item->decimal_precision() - dec; + uint32 len= item->max_length; + + DBUG_ASSERT (item->result_type() == DECIMAL_RESULT); + + /* + Trying to put too many digits overall in a DECIMAL(prec,dec) + will always throw a warning. We must limit dec to + DECIMAL_MAX_SCALE however to prevent an assert() later. + */ + + if (dec > 0) + { + signed int overflow; + + dec= min(dec, DECIMAL_MAX_SCALE); + + /* + If the value still overflows the field with the corrected dec, + we'll throw out decimals rather than integers. This is still + bad and of course throws a truncation warning. + +1: for decimal point + */ + + const int required_length= + my_decimal_precision_to_length(intg + dec, dec, + item->unsigned_flag); + + overflow= required_length - len; + + if (overflow > 0) + dec= max(0, dec - overflow); // too long, discard fract + else + /* Corrected value fits. */ + len= required_length; + } + return new Field_new_decimal(len, item->maybe_null, item->name, + dec, item->unsigned_flag); +} + + int Field_new_decimal::reset(void) { store_value(&decimal_zero); === modified file 'sql/field.h' --- a/sql/field.h 2009-12-03 11:19:05 +0000 +++ b/sql/field.h 2010-01-15 15:27:55 +0000 @@ -807,6 +807,7 @@ public: uint is_equal(Create_field *new_field); virtual const uchar *unpack(uchar* to, const uchar *from, uint param_data, bool low_byte_first); + static Field *create_from_item (Item *); }; === modified file 'sql/handler.h' --- a/sql/handler.h 2010-01-04 13:12:53 +0000 +++ b/sql/handler.h 2010-01-14 16:51:00 +0000 @@ -891,9 +891,9 @@ typedef struct { ulonglong delete_length; ha_rows records; ulong mean_rec_length; - ulong create_time; - ulong check_time; - ulong update_time; + time_t create_time; + time_t check_time; + time_t update_time; ulonglong check_sum; } PARTITION_INFO; @@ -1060,9 +1060,9 @@ public: ha_rows records; ha_rows deleted; /* Deleted records */ ulong mean_rec_length; /* physical reclength */ - ulong create_time; /* When table was created */ - ulong check_time; - ulong update_time; + time_t create_time; /* When table was created */ + time_t check_time; + time_t update_time; uint block_size; /* index block size */ ha_statistics(): === modified file 'sql/item.cc' --- a/sql/item.cc 2009-12-03 11:19:05 +0000 +++ b/sql/item.cc 2010-01-27 21:53:08 +0000 @@ -4908,9 +4908,7 @@ Field *Item::tmp_table_field_from_field_ switch (field_type()) { case MYSQL_TYPE_DECIMAL: case MYSQL_TYPE_NEWDECIMAL: - field= new Field_new_decimal((uchar*) 0, max_length, null_ptr, 0, - Field::NONE, name, decimals, 0, - unsigned_flag); + field= Field_new_decimal::create_from_item(this); break; case MYSQL_TYPE_TINY: field= new Field_tiny((uchar*) 0, max_length, null_ptr, 0, Field::NONE, @@ -6949,9 +6947,24 @@ int stored_field_cmp_to_item(THD *thd, F Item_cache* Item_cache::get_cache(const Item *item) { - switch (item->result_type()) { + return get_cache(item, item->result_type()); +} + + +/** + Get a cache item of given type. + + @param item value to be cached + @param type required type of cache + + @return cache item +*/ + +Item_cache* Item_cache::get_cache(const Item *item, const Item_result type) +{ + switch (type) { case INT_RESULT: - return new Item_cache_int(); + return new Item_cache_int(item->field_type()); case REAL_RESULT: return new Item_cache_real(); case DECIMAL_RESULT: @@ -6967,6 +6980,13 @@ Item_cache* Item_cache::get_cache(const } } +void Item_cache::store(Item *item) +{ + example= item; + if (!item) + null_value= TRUE; + value_cached= FALSE; +} void Item_cache::print(String *str, enum_query_type query_type) { @@ -6978,17 +6998,22 @@ void Item_cache::print(String *str, enum str->append(')'); } - -void Item_cache_int::store(Item *item) +bool Item_cache_int::cache_value() { - value= item->val_int_result(); - null_value= item->null_value; - unsigned_flag= item->unsigned_flag; + if (!example) + return FALSE; + value_cached= TRUE; + value= example->val_int_result(); + null_value= example->null_value; + unsigned_flag= example->unsigned_flag; + return TRUE; } void Item_cache_int::store(Item *item, longlong val_arg) { + /* An explicit values is given, save it. */ + value_cached= TRUE; value= val_arg; null_value= item->null_value; unsigned_flag= item->unsigned_flag; @@ -6998,6 +7023,8 @@ void Item_cache_int::store(Item *item, l String *Item_cache_int::val_str(String *str) { DBUG_ASSERT(fixed == 1); + if (!value_cached && !cache_value()) + return NULL; str->set(value, default_charset()); return str; } @@ -7006,21 +7033,52 @@ String *Item_cache_int::val_str(String * my_decimal *Item_cache_int::val_decimal(my_decimal *decimal_val) { DBUG_ASSERT(fixed == 1); + if (!value_cached && !cache_value()) + return NULL; int2my_decimal(E_DEC_FATAL_ERROR, value, unsigned_flag, decimal_val); return decimal_val; } +double Item_cache_int::val_real() +{ + DBUG_ASSERT(fixed == 1); + if (!value_cached && !cache_value()) + return 0.0; + return (double) value; +} -void Item_cache_real::store(Item *item) +longlong Item_cache_int::val_int() { - value= item->val_result(); - null_value= item->null_value; + DBUG_ASSERT(fixed == 1); + if (!value_cached && !cache_value()) + return 0; + return value; } +bool Item_cache_real::cache_value() +{ + if (!example) + return FALSE; + value_cached= TRUE; + value= example->val_result(); + null_value= example->null_value; + return TRUE; +} + + +double Item_cache_real::val_real() +{ + DBUG_ASSERT(fixed == 1); + if (!value_cached && !cache_value()) + return 0.0; + return value; +} longlong Item_cache_real::val_int() { DBUG_ASSERT(fixed == 1); + if (!value_cached && !cache_value()) + return 0; return (longlong) rint(value); } @@ -7028,6 +7086,8 @@ longlong Item_cache_real::val_int() String* Item_cache_real::val_str(String *str) { DBUG_ASSERT(fixed == 1); + if (!value_cached && !cache_value()) + return NULL; str->set_real(value, decimals, default_charset()); return str; } @@ -7036,22 +7096,30 @@ String* Item_cache_real::val_str(String my_decimal *Item_cache_real::val_decimal(my_decimal *decimal_val) { DBUG_ASSERT(fixed == 1); + if (!value_cached && !cache_value()) + return NULL; double2my_decimal(E_DEC_FATAL_ERROR, value, decimal_val); return decimal_val; } -void Item_cache_decimal::store(Item *item) +bool Item_cache_decimal::cache_value() { - my_decimal *val= item->val_decimal_result(&decimal_value); - if (!(null_value= item->null_value) && val != &decimal_value) + if (!example) + return FALSE; + value_cached= TRUE; + my_decimal *val= example->val_decimal_result(&decimal_value); + if (!(null_value= example->null_value) && val != &decimal_value) my_decimal2decimal(val, &decimal_value); + return TRUE; } double Item_cache_decimal::val_real() { DBUG_ASSERT(fixed); double res; + if (!value_cached && !cache_value()) + return 0.0; my_decimal2double(E_DEC_FATAL_ERROR, &decimal_value, &res); return res; } @@ -7060,6 +7128,8 @@ longlong Item_cache_decimal::val_int() { DBUG_ASSERT(fixed); longlong res; + if (!value_cached && !cache_value()) + return 0; my_decimal2int(E_DEC_FATAL_ERROR, &decimal_value, unsigned_flag, &res); return res; } @@ -7067,6 +7137,8 @@ longlong Item_cache_decimal::val_int() String* Item_cache_decimal::val_str(String *str) { DBUG_ASSERT(fixed); + if (!value_cached && !cache_value()) + return NULL; my_decimal_round(E_DEC_FATAL_ERROR, &decimal_value, decimals, FALSE, &decimal_value); my_decimal2string(E_DEC_FATAL_ERROR, &decimal_value, 0, 0, 0, str); @@ -7076,15 +7148,20 @@ String* Item_cache_decimal::val_str(Stri my_decimal *Item_cache_decimal::val_decimal(my_decimal *val) { DBUG_ASSERT(fixed); + if (!value_cached && !cache_value()) + return NULL; return &decimal_value; } -void Item_cache_str::store(Item *item) +bool Item_cache_str::cache_value() { - value_buff.set(buffer, sizeof(buffer), item->collation.collation); - value= item->str_result(&value_buff); - if ((null_value= item->null_value)) + if (!example) + return FALSE; + value_cached= TRUE; + value_buff.set(buffer, sizeof(buffer), example->collation.collation); + value= example->str_result(&value_buff); + if ((null_value= example->null_value)) value= 0; else if (value != &value_buff) { @@ -7099,6 +7176,7 @@ void Item_cache_str::store(Item *item) value_buff.copy(*value); value= &value_buff; } + return TRUE; } double Item_cache_str::val_real() @@ -7106,6 +7184,8 @@ double Item_cache_str::val_real() DBUG_ASSERT(fixed == 1); int err_not_used; char *end_not_used; + if (!value_cached && !cache_value()) + return 0.0; if (value) return my_strntod(value->charset(), (char*) value->ptr(), value->length(), &end_not_used, &err_not_used); @@ -7117,6 +7197,8 @@ longlong Item_cache_str::val_int() { DBUG_ASSERT(fixed == 1); int err; + if (!value_cached && !cache_value()) + return 0; if (value) return my_strntoll(value->charset(), value->ptr(), value->length(), 10, (char**) 0, &err); @@ -7124,9 +7206,21 @@ longlong Item_cache_str::val_int() return (longlong)0; } + +String* Item_cache_str::val_str(String *str) +{ + DBUG_ASSERT(fixed == 1); + if (!value_cached && !cache_value()) + return 0; + return value; +} + + my_decimal *Item_cache_str::val_decimal(my_decimal *decimal_val) { DBUG_ASSERT(fixed == 1); + if (!value_cached && !cache_value()) + return NULL; if (value) string2my_decimal(E_DEC_FATAL_ERROR, value, decimal_val); else @@ -7137,6 +7231,8 @@ my_decimal *Item_cache_str::val_decimal( int Item_cache_str::save_in_field(Field *field, bool no_conversions) { + if (!value_cached && !cache_value()) + return 0; int res= Item_cache::save_in_field(field, no_conversions); return (is_varbinary && field->type() == MYSQL_TYPE_STRING && value->length() < field->field_length) ? 1 : res; @@ -7171,13 +7267,30 @@ bool Item_cache_row::setup(Item * item) void Item_cache_row::store(Item * item) { + example= item; + if (!item) + { + null_value= TRUE; + return; + } + for (uint i= 0; i < item_count; i++) + values[i]->store(item->element_index(i)); +} + + +bool Item_cache_row::cache_value() +{ + if (!example) + return FALSE; + value_cached= TRUE; null_value= 0; - item->bring_value(); + example->bring_value(); for (uint i= 0; i < item_count; i++) { - values[i]->store(item->element_index(i)); + values[i]->cache_value(); null_value|= values[i]->null_value; } + return TRUE; } === modified file 'sql/item.h' --- a/sql/item.h 2009-12-03 11:19:05 +0000 +++ b/sql/item.h 2010-01-15 15:27:55 +0000 @@ -1053,7 +1053,11 @@ class sp_head; class Item_basic_constant :public Item { + table_map used_table_map; public: + Item_basic_constant(): Item(), used_table_map(0) {}; + void set_used_tables(table_map map) { used_table_map= map; } + table_map used_tables() const { return used_table_map; } /* to prevent drop fixed flag (no need parent cleanup call) */ void cleanup() { @@ -1065,7 +1069,6 @@ public: if (orig_name) name= orig_name; } - Item_basic_constant() {} /* Remove gcc warning */ }; @@ -2165,6 +2168,23 @@ public: save_in_field(result_field, no_conversions); } void cleanup(); + /* + This method is used for debug purposes to print the name of an + item to the debug log. The second use of this method is as + a helper function of print() and error messages, where it is + applicable. To suit both goals it should return a meaningful, + distinguishable and sintactically correct string. This method + should not be used for runtime type identification, use enum + {Sum}Functype and Item_func::functype()/Item_sum::sum_func() + instead. + Added here, to the parent class of both Item_func and Item_sum_func. + + NOTE: for Items inherited from Item_sum, func_name() return part of + function name till first argument (including '(') to make difference in + names for functions with 'distinct' clause and without 'distinct' and + also to make printing of items inherited from Item_sum uniform. + */ + virtual const char *func_name() const= 0; }; @@ -2924,15 +2944,25 @@ protected: */ Field *cached_field; enum enum_field_types cached_field_type; -public: - Item_cache(): - example(0), used_table_map(0), cached_field(0), cached_field_type(MYSQL_TYPE_STRING) + /* + TRUE <=> cache holds value of the last stored item (i.e actual value). + store() stores item to be cached and sets this flag to FALSE. + On the first call of val_xxx function if this flag is set to FALSE the + cache_value() will be called to actually cache value of saved item. + cache_value() will set this flag to TRUE. + */ + bool value_cached; +public: + Item_cache(): + example(0), used_table_map(0), cached_field(0), cached_field_type(MYSQL_TYPE_STRING), + value_cached(0) { fixed= 1; null_value= 1; } Item_cache(enum_field_types field_type_arg): - example(0), used_table_map(0), cached_field(0), cached_field_type(field_type_arg) + example(0), used_table_map(0), cached_field(0), cached_field_type(field_type_arg), + value_cached(0) { fixed= 1; null_value= 1; @@ -2952,10 +2982,10 @@ public: cached_field= ((Item_field *)item)->field; return 0; }; - virtual void store(Item *)= 0; enum Type type() const { return CACHE_ITEM; } enum_field_types field_type() const { return cached_field_type; } static Item_cache* get_cache(const Item *item); + static Item_cache* get_cache(const Item* item, const Item_result type); table_map used_tables() const { return used_table_map; } virtual void keep_array() {} virtual void print(String *str, enum_query_type query_type); @@ -2967,6 +2997,8 @@ public: { return this == item; } + virtual void store(Item *item); + virtual bool cache_value()= 0; }; @@ -2975,18 +3007,19 @@ class Item_cache_int: public Item_cache protected: longlong value; public: - Item_cache_int(): Item_cache(), value(0) {} + Item_cache_int(): Item_cache(), + value(0) {} Item_cache_int(enum_field_types field_type_arg): Item_cache(field_type_arg), value(0) {} - void store(Item *item); void store(Item *item, longlong val_arg); - double val_real() { DBUG_ASSERT(fixed == 1); return (double) value; } - longlong val_int() { DBUG_ASSERT(fixed == 1); return value; } + double val_real(); + longlong val_int(); String* val_str(String *str); my_decimal *val_decimal(my_decimal *); enum Item_result result_type() const { return INT_RESULT; } bool result_as_longlong() { return TRUE; } + bool cache_value(); }; @@ -2994,14 +3027,15 @@ class Item_cache_real: public Item_cache { double value; public: - Item_cache_real(): Item_cache(), value(0) {} + Item_cache_real(): Item_cache(), + value(0) {} - void store(Item *item); - double val_real() { DBUG_ASSERT(fixed == 1); return value; } + double val_real(); longlong val_int(); String* val_str(String *str); my_decimal *val_decimal(my_decimal *); enum Item_result result_type() const { return REAL_RESULT; } + bool cache_value(); }; @@ -3012,12 +3046,12 @@ protected: public: Item_cache_decimal(): Item_cache() {} - void store(Item *item); double val_real(); longlong val_int(); String* val_str(String *str); my_decimal *val_decimal(my_decimal *); enum Item_result result_type() const { return DECIMAL_RESULT; } + bool cache_value(); }; @@ -3035,14 +3069,14 @@ public: MYSQL_TYPE_VARCHAR && !((const Item_field *) item)->field->has_charset()) {} - void store(Item *item); double val_real(); longlong val_int(); - String* val_str(String *) { DBUG_ASSERT(fixed == 1); return value; } + String* val_str(String *); my_decimal *val_decimal(my_decimal *); enum Item_result result_type() const { return STRING_RESULT; } CHARSET_INFO *charset() const { return value->charset(); }; int save_in_field(Field *field, bool no_conversions); + bool cache_value(); }; class Item_cache_row: public Item_cache @@ -3052,7 +3086,8 @@ class Item_cache_row: public Item_cache bool save_array; public: Item_cache_row() - :Item_cache(), values(0), item_count(2), save_array(0) {} + :Item_cache(), values(0), item_count(2), + save_array(0) {} /* 'allocate' used only in row transformer, to preallocate space for row @@ -3110,6 +3145,7 @@ public: values= 0; DBUG_VOID_RETURN; } + bool cache_value(); }; === modified file 'sql/item_cmpfunc.cc' --- a/sql/item_cmpfunc.cc 2009-12-03 11:19:05 +0000 +++ b/sql/item_cmpfunc.cc 2010-01-15 15:27:55 +0000 @@ -30,6 +30,9 @@ #include "sql_select.h" static bool convert_constant_item(THD *, Item_field *, Item **); +static longlong +get_year_value(THD *thd, Item ***item_arg, Item **cache_arg, + Item *warn_item, bool *is_null); static Item_result item_store_type(Item_result a, Item *item, my_bool unsigned_flag) @@ -533,11 +536,12 @@ void Item_bool_func2::fix_length_and_dec } -int Arg_comparator::set_compare_func(Item_bool_func2 *item, Item_result type) +int Arg_comparator::set_compare_func(Item_result_field *item, Item_result type) { owner= item; func= comparator_matrix[type] - [test(owner->functype() == Item_func::EQUAL_FUNC)]; + [is_owner_equal_func()]; + switch (type) { case ROW_RESULT: { @@ -557,7 +561,8 @@ int Arg_comparator::set_compare_func(Ite my_error(ER_OPERAND_COLUMNS, MYF(0), (*a)->element_index(i)->cols()); return 1; } - if (comparators[i].set_cmp_func(owner, (*a)->addr(i), (*b)->addr(i))) + if (comparators[i].set_cmp_func(owner, (*a)->addr(i), (*b)->addr(i), + set_null)) return 1; } break; @@ -571,7 +576,8 @@ int Arg_comparator::set_compare_func(Ite if (cmp_collation.set((*a)->collation, (*b)->collation) || cmp_collation.derivation == DERIVATION_NONE) { - my_coll_agg_error((*a)->collation, (*b)->collation, owner->func_name()); + my_coll_agg_error((*a)->collation, (*b)->collation, + owner->func_name()); return 1; } if (cmp_collation.collation == &my_charset_bin) @@ -785,15 +791,21 @@ Arg_comparator::can_compare_as_dates(Ite if (cmp_type != CMP_DATE_DFLT) { + THD *thd= current_thd; /* Do not cache GET_USER_VAR() function as its const_item() may return TRUE for the current thread but it still may change during the execution. + Don't use cache while in the context analysis mode only (i.e. for + EXPLAIN/CREATE VIEW and similar queries). Cache is useless in such + cases and can cause problems. For example evaluating subqueries can + confuse storage engines since in context analysis mode tables + aren't locked. */ - if (cmp_type != CMP_DATE_WITH_DATE && str_arg->const_item() && + if (!thd->is_context_analysis_only() && + cmp_type != CMP_DATE_WITH_DATE && str_arg->const_item() && (str_arg->type() != Item::FUNC_ITEM || ((Item_func*)str_arg)->functype() != Item_func::GUSERVAR_FUNC)) { - THD *thd= current_thd; ulonglong value; bool error; String tmp, *str_val= 0; @@ -875,18 +887,20 @@ get_time_value(THD *thd, Item ***item_ar } -int Arg_comparator::set_cmp_func(Item_bool_func2 *owner_arg, +int Arg_comparator::set_cmp_func(Item_result_field *owner_arg, Item **a1, Item **a2, Item_result type) { ulonglong const_value= (ulonglong)-1; + thd= current_thd; + owner= owner_arg; + set_null= set_null && owner_arg; a= a1; b= a2; + thd= current_thd; if (can_compare_as_dates(*a, *b, &const_value)) { - thd= current_thd; - owner= owner_arg; a_type= (*a)->field_type(); b_type= (*b)->field_type(); a_cache= 0; @@ -894,6 +908,10 @@ int Arg_comparator::set_cmp_func(Item_bo if (const_value != (ulonglong)-1) { + /* + cache_converted_constant can't be used here because it can't + correctly convert a DATETIME value from string to int representation. + */ Item_cache_int *cache= new Item_cache_int(); /* Mark the cache as non-const to prevent re-caching. */ cache->set_used_tables(1); @@ -910,22 +928,22 @@ int Arg_comparator::set_cmp_func(Item_bo b= (Item **)&b_cache; } } - is_nulls_eq= test(owner && owner->functype() == Item_func::EQUAL_FUNC); + is_nulls_eq= is_owner_equal_func(); func= &Arg_comparator::compare_datetime; - get_value_func= &get_datetime_value; + get_value_a_func= &get_datetime_value; + get_value_b_func= &get_datetime_value; return 0; } else if (type == STRING_RESULT && (*a)->field_type() == MYSQL_TYPE_TIME && (*b)->field_type() == MYSQL_TYPE_TIME) { /* Compare TIME values as integers. */ - thd= current_thd; - owner= owner_arg; a_cache= 0; b_cache= 0; - is_nulls_eq= test(owner && owner->functype() == Item_func::EQUAL_FUNC); + is_nulls_eq= is_owner_equal_func(); func= &Arg_comparator::compare_datetime; - get_value_func= &get_time_value; + get_value_a_func= &get_time_value; + get_value_b_func= &get_time_value; return 0; } else if (type == STRING_RESULT && @@ -934,20 +952,97 @@ int Arg_comparator::set_cmp_func(Item_bo { DTCollation coll; coll.set((*a)->collation.collation); - if (agg_item_set_converter(coll, owner_arg->func_name(), + if (agg_item_set_converter(coll, owner->func_name(), b, 1, MY_COLL_CMP_CONV, 1)) return 1; } + else if (try_year_cmp_func(type)) + return 0; + a= cache_converted_constant(thd, a, &a_cache, type); + b= cache_converted_constant(thd, b, &b_cache, type); return set_compare_func(owner_arg, type); } -void Arg_comparator::set_datetime_cmp_func(Item **a1, Item **b1) +/* + Helper function to call from Arg_comparator::set_cmp_func() +*/ + +bool Arg_comparator::try_year_cmp_func(Item_result type) +{ + if (type == ROW_RESULT) + return FALSE; + + bool a_is_year= (*a)->field_type() == MYSQL_TYPE_YEAR; + bool b_is_year= (*b)->field_type() == MYSQL_TYPE_YEAR; + + if (!a_is_year && !b_is_year) + return FALSE; + + if (a_is_year && b_is_year) + { + get_value_a_func= &get_year_value; + get_value_b_func= &get_year_value; + } + else if (a_is_year && (*b)->is_datetime()) + { + get_value_a_func= &get_year_value; + get_value_b_func= &get_datetime_value; + } + else if (b_is_year && (*a)->is_datetime()) + { + get_value_b_func= &get_year_value; + get_value_a_func= &get_datetime_value; + } + else + return FALSE; + + is_nulls_eq= is_owner_equal_func(); + func= &Arg_comparator::compare_datetime; + + return TRUE; +} + +/** + Convert and cache a constant. + + @param value [in] An item to cache + @param cache_item [out] Placeholder for the cache item + @param type [in] Comparison type + + @details + When given item is a constant and its type differs from comparison type + then cache its value to avoid type conversion of this constant on each + evaluation. In this case the value is cached and the reference to the cache + is returned. + Original value is returned otherwise. + + @return cache item or original value. +*/ + +Item** Arg_comparator::cache_converted_constant(THD *thd, Item **value, + Item **cache_item, + Item_result type) +{ + /* Don't need cache if doing context analysis only. */ + if (!thd->is_context_analysis_only() && + (*value)->const_item() && type != (*value)->result_type()) + { + Item_cache *cache= Item_cache::get_cache(*value, type); + cache->setup(*value); + *cache_item= cache; + return cache_item; + } + return value; +} + + +void Arg_comparator::set_datetime_cmp_func(Item_result_field *owner_arg, + Item **a1, Item **b1) { thd= current_thd; - /* A caller will handle null values by itself. */ - owner= NULL; + owner= owner_arg; a= a1; b= b1; a_type= (*a)->field_type(); @@ -956,7 +1051,8 @@ void Arg_comparator::set_datetime_cmp_fu b_cache= 0; is_nulls_eq= FALSE; func= &Arg_comparator::compare_datetime; - get_value_func= &get_datetime_value; + get_value_a_func= &get_datetime_value; + get_value_b_func= &get_datetime_value; } @@ -1056,6 +1152,56 @@ get_datetime_value(THD *thd, Item ***ite return value; } + +/* + Retrieves YEAR value of 19XX-00-00 00:00:00 form from given item. + + SYNOPSIS + get_year_value() + thd thread handle + item_arg [in/out] item to retrieve YEAR value from + cache_arg [in/out] pointer to place to store the caching item to + warn_item [in] item for issuing the conversion warning + is_null [out] TRUE <=> the item_arg is null + + DESCRIPTION + Retrieves the YEAR value of 19XX form from given item for comparison by the + compare_datetime() function. + Converts year to DATETIME of form YYYY-00-00 00:00:00 for the compatibility + with the get_datetime_value function result. + + RETURN + obtained value +*/ + +static longlong +get_year_value(THD *thd, Item ***item_arg, Item **cache_arg, + Item *warn_item, bool *is_null) +{ + longlong value= 0; + Item *item= **item_arg; + + value= item->val_int(); + *is_null= item->null_value; + if (*is_null) + return ~(ulonglong) 0; + + /* + Coerce value to the 19XX form in order to correctly compare + YEAR(2) & YEAR(4) types. + */ + if (value < 70) + value+= 100; + if (value <= 1900) + value+= 1900; + + /* Convert year to DATETIME of form YYYY-00-00 00:00:00 (YYYY0000000000). */ + value*= 10000000000LL; + + return value; +} + + /* Compare items values as dates. @@ -1088,25 +1234,25 @@ int Arg_comparator::compare_datetime() longlong a_value, b_value; /* Get DATE/DATETIME/TIME value of the 'a' item. */ - a_value= (*get_value_func)(thd, &a, &a_cache, *b, &a_is_null); + a_value= (*get_value_a_func)(thd, &a, &a_cache, *b, &a_is_null); if (!is_nulls_eq && a_is_null) { - if (owner) + if (set_null) owner->null_value= 1; return -1; } /* Get DATE/DATETIME/TIME value of the 'b' item. */ - b_value= (*get_value_func)(thd, &b, &b_cache, *a, &b_is_null); + b_value= (*get_value_b_func)(thd, &b, &b_cache, *a, &b_is_null); if (a_is_null || b_is_null) { - if (owner) + if (set_null) owner->null_value= is_nulls_eq ? 0 : 1; return is_nulls_eq ? (a_is_null == b_is_null) : -1; } /* Here we have two not-NULL values. */ - if (owner) + if (set_null) owner->null_value= 0; /* Compare values. */ @@ -1119,15 +1265,17 @@ int Arg_comparator::compare_datetime() int Arg_comparator::compare_string() { String *res1,*res2; - if ((res1= (*a)->val_str(&owner->tmp_value1))) + if ((res1= (*a)->val_str(&value1))) { - if ((res2= (*b)->val_str(&owner->tmp_value2))) + if ((res2= (*b)->val_str(&value2))) { - owner->null_value= 0; + if (set_null) + owner->null_value= 0; return sortcmp(res1,res2,cmp_collation.collation); } } - owner->null_value= 1; + if (set_null) + owner->null_value= 1; return -1; } @@ -1146,18 +1294,20 @@ int Arg_comparator::compare_string() int Arg_comparator::compare_binary_string() { String *res1,*res2; - if ((res1= (*a)->val_str(&owner->tmp_value1))) + if ((res1= (*a)->val_str(&value1))) { - if ((res2= (*b)->val_str(&owner->tmp_value2))) + if ((res2= (*b)->val_str(&value2))) { - owner->null_value= 0; + if (set_null) + owner->null_value= 0; uint res1_length= res1->length(); uint res2_length= res2->length(); int cmp= memcmp(res1->ptr(), res2->ptr(), min(res1_length,res2_length)); return cmp ? cmp : (int) (res1_length - res2_length); } } - owner->null_value= 1; + if (set_null) + owner->null_value= 1; return -1; } @@ -1170,8 +1320,8 @@ int Arg_comparator::compare_binary_strin int Arg_comparator::compare_e_string() { String *res1,*res2; - res1= (*a)->val_str(&owner->tmp_value1); - res2= (*b)->val_str(&owner->tmp_value2); + res1= (*a)->val_str(&value1); + res2= (*b)->val_str(&value2); if (!res1 || !res2) return test(res1 == res2); return test(sortcmp(res1, res2, cmp_collation.collation) == 0); @@ -1181,8 +1331,8 @@ int Arg_comparator::compare_e_string() int Arg_comparator::compare_e_binary_string() { String *res1,*res2; - res1= (*a)->val_str(&owner->tmp_value1); - res2= (*b)->val_str(&owner->tmp_value2); + res1= (*a)->val_str(&value1); + res2= (*b)->val_str(&value2); if (!res1 || !res2) return test(res1 == res2); return test(stringcmp(res1, res2) == 0); @@ -1203,13 +1353,15 @@ int Arg_comparator::compare_real() val2= (*b)->val_real(); if (!(*b)->null_value) { - owner->null_value= 0; + if (set_null) + owner->null_value= 0; if (val1 < val2) return -1; if (val1 == val2) return 0; return 1; } } - owner->null_value= 1; + if (set_null) + owner->null_value= 1; return -1; } @@ -1223,11 +1375,13 @@ int Arg_comparator::compare_decimal() my_decimal *val2= (*b)->val_decimal(&value2); if (!(*b)->null_value) { - owner->null_value= 0; + if (set_null) + owner->null_value= 0; return my_decimal_cmp(val1, val2); } } - owner->null_value= 1; + if (set_null) + owner->null_value= 1; return -1; } @@ -1265,7 +1419,8 @@ int Arg_comparator::compare_real_fixed() val2= (*b)->val_real(); if (!(*b)->null_value) { - owner->null_value= 0; + if (set_null) + owner->null_value= 0; if (val1 == val2 || fabs(val1 - val2) < precision) return 0; if (val1 < val2) @@ -1273,7 +1428,8 @@ int Arg_comparator::compare_real_fixed() return 1; } } - owner->null_value= 1; + if (set_null) + owner->null_value= 1; return -1; } @@ -1296,13 +1452,15 @@ int Arg_comparator::compare_int_signed() longlong val2= (*b)->val_int(); if (!(*b)->null_value) { - owner->null_value= 0; + if (set_null) + owner->null_value= 0; if (val1 < val2) return -1; if (val1 == val2) return 0; return 1; } } - owner->null_value= 1; + if (set_null) + owner->null_value= 1; return -1; } @@ -1319,13 +1477,15 @@ int Arg_comparator::compare_int_unsigned ulonglong val2= (*b)->val_int(); if (!(*b)->null_value) { - owner->null_value= 0; + if (set_null) + owner->null_value= 0; if (val1 < val2) return -1; if (val1 == val2) return 0; return 1; } } - owner->null_value= 1; + if (set_null) + owner->null_value= 1; return -1; } @@ -1342,7 +1502,8 @@ int Arg_comparator::compare_int_signed_u ulonglong uval2= (ulonglong)(*b)->val_int(); if (!(*b)->null_value) { - owner->null_value= 0; + if (set_null) + owner->null_value= 0; if (sval1 < 0 || (ulonglong)sval1 < uval2) return -1; if ((ulonglong)sval1 == uval2) @@ -1350,7 +1511,8 @@ int Arg_comparator::compare_int_signed_u return 1; } } - owner->null_value= 1; + if (set_null) + owner->null_value= 1; return -1; } @@ -1367,7 +1529,8 @@ int Arg_comparator::compare_int_unsigned longlong sval2= (*b)->val_int(); if (!(*b)->null_value) { - owner->null_value= 0; + if (set_null) + owner->null_value= 0; if (sval2 < 0) return 1; if (uval1 < (ulonglong)sval2) @@ -1377,7 +1540,8 @@ int Arg_comparator::compare_int_unsigned return 1; } } - owner->null_value= 1; + if (set_null) + owner->null_value= 1; return -1; } @@ -1413,10 +1577,11 @@ int Arg_comparator::compare_row() for (uint i= 0; i<n; i++) { res= comparators[i].compare(); - if (owner->null_value) + /* Aggregate functions don't need special null handling. */ + if (owner->null_value && owner->type() == Item::FUNC_ITEM) { // NULL was compared - switch (owner->functype()) { + switch (((Item_func*)owner)->functype()) { case Item_func::NE_FUNC: break; // NE never aborts on NULL even if abort_on_null is set case Item_func::LT_FUNC: @@ -1425,7 +1590,7 @@ int Arg_comparator::compare_row() case Item_func::GE_FUNC: return -1; // <, <=, > and >= always fail on NULL default: // EQ_FUNC - if (owner->abort_on_null) + if (((Item_bool_func2*)owner)->abort_on_null) return -1; // We do not need correct NULL returning } was_null= 1; @@ -1581,6 +1746,7 @@ longlong Item_in_optimizer::val_int() bool tmp; DBUG_ASSERT(fixed == 1); cache->store(args[0]); + cache->cache_value(); if (cache->null_value) { @@ -1748,8 +1914,8 @@ longlong Item_func_lt::val_int() longlong Item_func_strcmp::val_int() { DBUG_ASSERT(fixed == 1); - String *a=args[0]->val_str(&tmp_value1); - String *b=args[1]->val_str(&tmp_value2); + String *a=args[0]->val_str(&cmp.value1); + String *b=args[1]->val_str(&cmp.value2); if (!a || !b) { null_value=1; @@ -2032,8 +2198,8 @@ void Item_func_between::fix_length_and_d if (compare_as_dates) { - ge_cmp.set_datetime_cmp_func(args, args + 1); - le_cmp.set_datetime_cmp_func(args, args + 2); + ge_cmp.set_datetime_cmp_func(this, args, args + 1); + le_cmp.set_datetime_cmp_func(this, args, args + 2); } else if (time_items_found == 3) { @@ -4370,13 +4536,13 @@ void Item_func_isnotnull::print(String * longlong Item_func_like::val_int() { DBUG_ASSERT(fixed == 1); - String* res = args[0]->val_str(&tmp_value1); + String* res = args[0]->val_str(&cmp.value1); if (args[0]->null_value) { null_value=1; return 0; } - String* res2 = args[1]->val_str(&tmp_value2); + String* res2 = args[1]->val_str(&cmp.value2); if (args[1]->null_value) { null_value=1; @@ -4400,7 +4566,7 @@ Item_func::optimize_type Item_func_like: { if (args[1]->const_item()) { - String* res2= args[1]->val_str((String *)&tmp_value2); + String* res2= args[1]->val_str((String *)&cmp.value2); if (!res2) return OPTIMIZE_NONE; @@ -4431,7 +4597,7 @@ bool Item_func_like::fix_fields(THD *thd if (escape_item->const_item()) { /* If we are on execution stage */ - String *escape_str= escape_item->val_str(&tmp_value1); + String *escape_str= escape_item->val_str(&cmp.value1); if (escape_str) { if (escape_used_in_parsing && ( @@ -4486,7 +4652,7 @@ bool Item_func_like::fix_fields(THD *thd if (args[1]->const_item() && !use_strnxfrm(collation.collation) && !(specialflag & SPECIAL_NO_NEW_FUNC)) { - String* res2 = args[1]->val_str(&tmp_value2); + String* res2 = args[1]->val_str(&cmp.value2); if (!res2) return FALSE; // Null argument === modified file 'sql/item_cmpfunc.h' --- a/sql/item_cmpfunc.h 2009-12-03 11:19:05 +0000 +++ b/sql/item_cmpfunc.h 2010-01-15 15:27:55 +0000 @@ -32,7 +32,7 @@ class Arg_comparator: public Sql_alloc { Item **a, **b; arg_cmp_func func; - Item_bool_func2 *owner; + Item_result_field *owner; Arg_comparator *comparators; // used only for compare_row() double precision; /* Fields used in DATE/DATETIME comparison. */ @@ -40,30 +40,40 @@ class Arg_comparator: public Sql_alloc enum_field_types a_type, b_type; // Types of a and b items Item *a_cache, *b_cache; // Cached values of a and b items bool is_nulls_eq; // TRUE <=> compare for the EQUAL_FUNC + bool set_null; // TRUE <=> set owner->null_value + // when one of arguments is NULL. enum enum_date_cmp_type { CMP_DATE_DFLT= 0, CMP_DATE_WITH_DATE, CMP_DATE_WITH_STR, CMP_STR_WITH_DATE }; - longlong (*get_value_func)(THD *thd, Item ***item_arg, Item **cache_arg, - Item *warn_item, bool *is_null); + longlong (*get_value_a_func)(THD *thd, Item ***item_arg, Item **cache_arg, + Item *warn_item, bool *is_null); + longlong (*get_value_b_func)(THD *thd, Item ***item_arg, Item **cache_arg, + Item *warn_item, bool *is_null); + bool try_year_cmp_func(Item_result type); public: DTCollation cmp_collation; + /* Allow owner function to use string buffers. */ + String value1, value2; - Arg_comparator(): thd(0), a_cache(0), b_cache(0) {}; + Arg_comparator(): thd(0), a_cache(0), b_cache(0), set_null(TRUE), + get_value_a_func(0), get_value_b_func(0) {}; Arg_comparator(Item **a1, Item **a2): a(a1), b(a2), thd(0), - a_cache(0), b_cache(0) {}; + a_cache(0), b_cache(0), set_null(TRUE), + get_value_a_func(0), get_value_b_func(0) {}; - int set_compare_func(Item_bool_func2 *owner, Item_result type); - inline int set_compare_func(Item_bool_func2 *owner_arg) + int set_compare_func(Item_result_field *owner, Item_result type); + inline int set_compare_func(Item_result_field *owner_arg) { return set_compare_func(owner_arg, item_cmp_type((*a)->result_type(), (*b)->result_type())); } - int set_cmp_func(Item_bool_func2 *owner_arg, + int set_cmp_func(Item_result_field *owner_arg, Item **a1, Item **a2, Item_result type); - inline int set_cmp_func(Item_bool_func2 *owner_arg, - Item **a1, Item **a2) + inline int set_cmp_func(Item_result_field *owner_arg, + Item **a1, Item **a2, bool set_null_arg) { + set_null= set_null_arg; return set_cmp_func(owner_arg, a1, a2, item_cmp_type((*a1)->result_type(), (*a2)->result_type())); @@ -93,8 +103,15 @@ public: static enum enum_date_cmp_type can_compare_as_dates(Item *a, Item *b, ulonglong *const_val_arg); - void set_datetime_cmp_func(Item **a1, Item **b1); + Item** cache_converted_constant(THD *thd, Item **value, Item **cache, + Item_result type); + void set_datetime_cmp_func(Item_result_field *owner_arg, Item **a1, Item **b1); static arg_cmp_func comparator_matrix [5][2]; + inline bool is_owner_equal_func() + { + return (owner->type() == Item::FUNC_ITEM && + ((Item_func*)owner)->functype() == Item_func::EQUAL_FUNC); + } friend class Item_func; }; @@ -324,7 +341,6 @@ class Item_bool_func2 :public Item_int_f { /* Bool with 2 string args */ protected: Arg_comparator cmp; - String tmp_value1,tmp_value2; bool abort_on_null; public: @@ -333,7 +349,7 @@ public: void fix_length_and_dec(); void set_cmp_func() { - cmp.set_cmp_func(this, tmp_arg, tmp_arg+1); + cmp.set_cmp_func(this, tmp_arg, tmp_arg+1, TRUE); } optimize_type select_optimize() const { return OPTIMIZE_OP; } virtual enum Functype rev_functype() const { return UNKNOWN_FUNC; } === modified file 'sql/item_create.cc' --- a/sql/item_create.cc 2009-10-15 21:38:29 +0000 +++ b/sql/item_create.cc 2010-01-15 15:27:55 +0000 @@ -3524,6 +3524,7 @@ Create_func_get_lock Create_func_get_loc Item* Create_func_get_lock::create(THD *thd, Item *arg1, Item *arg2) { + thd->lex->set_stmt_unsafe(); thd->lex->uncacheable(UNCACHEABLE_SIDEEFFECT); return new (thd->mem_root) Item_func_get_lock(arg1, arg2); } @@ -3635,6 +3636,7 @@ Create_func_is_free_lock Create_func_is_ Item* Create_func_is_free_lock::create(THD *thd, Item *arg1) { + thd->lex->set_stmt_unsafe(); thd->lex->uncacheable(UNCACHEABLE_SIDEEFFECT); return new (thd->mem_root) Item_func_is_free_lock(arg1); } @@ -3645,6 +3647,7 @@ Create_func_is_used_lock Create_func_is_ Item* Create_func_is_used_lock::create(THD *thd, Item *arg1) { + thd->lex->set_stmt_unsafe(); thd->lex->uncacheable(UNCACHEABLE_SIDEEFFECT); return new (thd->mem_root) Item_func_is_used_lock(arg1); } @@ -3961,6 +3964,8 @@ Create_func_master_pos_wait::create_nati Item *func= NULL; int arg_count= 0; + thd->lex->set_stmt_unsafe(); + if (item_list != NULL) arg_count= item_list->elements; @@ -4203,6 +4208,7 @@ Create_func_release_lock Create_func_rel Item* Create_func_release_lock::create(THD *thd, Item *arg1) { + thd->lex->set_stmt_unsafe(); thd->lex->uncacheable(UNCACHEABLE_SIDEEFFECT); return new (thd->mem_root) Item_func_release_lock(arg1); } @@ -4325,6 +4331,7 @@ Create_func_sleep Create_func_sleep::s_s Item* Create_func_sleep::create(THD *thd, Item *arg1) { + thd->lex->set_stmt_unsafe(); thd->lex->uncacheable(UNCACHEABLE_SIDEEFFECT); return new (thd->mem_root) Item_func_sleep(arg1); } @@ -4591,6 +4598,7 @@ Create_func_version Create_func_version: Item* Create_func_version::create(THD *thd) { + thd->lex->set_stmt_unsafe(); return new (thd->mem_root) Item_static_string_func("version()", server_version, (uint) strlen(server_version), === modified file 'sql/item_func.cc' --- a/sql/item_func.cc 2009-12-03 11:19:05 +0000 +++ b/sql/item_func.cc 2010-01-15 15:27:55 +0000 @@ -450,45 +450,8 @@ Field *Item_func::tmp_table_field(TABLE case STRING_RESULT: return make_string_field(table); case DECIMAL_RESULT: - { - uint8 dec= decimals; - uint8 intg= decimal_precision() - dec; - uint32 len= max_length; - - /* - Trying to put too many digits overall in a DECIMAL(prec,dec) - will always throw a warning. We must limit dec to - DECIMAL_MAX_SCALE however to prevent an assert() later. - */ - - if (dec > 0) - { - int overflow; - - dec= min(dec, DECIMAL_MAX_SCALE); - - /* - If the value still overflows the field with the corrected dec, - we'll throw out decimals rather than integers. This is still - bad and of course throws a truncation warning. - */ - - const int required_length= - my_decimal_precision_to_length(intg + dec, dec, - unsigned_flag); - - overflow= required_length - len; - - if (overflow > 0) - dec= max(0, dec - overflow); // too long, discard fract - else - /* Corrected value fits. */ - len= required_length; - } - - field= new Field_new_decimal(len, maybe_null, name, dec, unsigned_flag); + field= Field_new_decimal::create_from_item(this); break; - } case ROW_RESULT: default: // This case should never be chosen === modified file 'sql/item_func.h' --- a/sql/item_func.h 2009-12-03 11:19:05 +0000 +++ b/sql/item_func.h 2010-01-15 15:27:55 +0000 @@ -124,17 +124,6 @@ public: virtual optimize_type select_optimize() const { return OPTIMIZE_NONE; } virtual bool have_rev_func() const { return 0; } virtual Item *key_item() const { return args[0]; } - /* - This method is used for debug purposes to print the name of an - item to the debug log. The second use of this method is as - a helper function of print(), where it is applicable. - To suit both goals it should return a meaningful, - distinguishable and sintactically correct string. This method - should not be used for runtime type identification, use enum - {Sum}Functype and Item_func::functype()/Item_sum::sum_func() - instead. - */ - virtual const char *func_name() const= 0; virtual bool const_item() const { return const_item_cache; } inline Item **arguments() const { return args; } void set_arguments(List<Item> &list); === modified file 'sql/item_geofunc.cc' --- a/sql/item_geofunc.cc 2009-10-24 06:57:31 +0000 +++ b/sql/item_geofunc.cc 2009-12-08 09:26:11 +0000 @@ -511,8 +511,8 @@ err: longlong Item_func_spatial_rel::val_int() { DBUG_ASSERT(fixed == 1); - String *res1= args[0]->val_str(&tmp_value1); - String *res2= args[1]->val_str(&tmp_value2); + String *res1= args[0]->val_str(&cmp.value1); + String *res2= args[1]->val_str(&cmp.value2); Geometry_buffer buffer1, buffer2; Geometry *g1, *g2; MBR mbr1, mbr2; === modified file 'sql/item_strfunc.cc' --- a/sql/item_strfunc.cc 2009-12-03 11:19:05 +0000 +++ b/sql/item_strfunc.cc 2010-01-15 15:27:55 +0000 @@ -1828,8 +1828,9 @@ String *Item_func_database::val_str(Stri /** - @todo - make USER() replicate properly (currently it is replicated to "") + @note USER() is replicated correctly if binlog_format=ROW or (as of + BUG#28086) binlog_format=MIXED, but is incorrectly replicated to '' + if binlog_format=STATEMENT. */ bool Item_func_user::init(const char *user, const char *host) { === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2009-12-03 11:19:05 +0000 +++ b/sql/item_subselect.cc 2010-01-15 15:27:55 +0000 @@ -503,6 +503,7 @@ Item_singlerow_subselect::select_transfo void Item_singlerow_subselect::store(uint i, Item *item) { row[i]->store(item); + row[i]->cache_value(); } enum Item_result Item_singlerow_subselect::result_type() const @@ -1854,6 +1855,7 @@ void subselect_engine::set_row(List<Item if (!(row[i]= Item_cache::get_cache(sel_item))) return; row[i]->setup(sel_item); + row[i]->store(sel_item); } if (item_list.elements > 1) res_type= ROW_RESULT; === modified file 'sql/item_subselect.h' --- a/sql/item_subselect.h 2009-08-31 20:02:09 +0000 +++ b/sql/item_subselect.h 2010-01-15 15:27:55 +0000 @@ -142,6 +142,7 @@ public: @return the SELECT_LEX structure associated with this Item */ st_select_lex* get_select_lex(); + const char *func_name() const { DBUG_ASSERT(0); return "subselect"; } friend class select_subselect; friend class Item_in_optimizer; === modified file 'sql/item_sum.cc' --- a/sql/item_sum.cc 2009-12-03 11:19:05 +0000 +++ b/sql/item_sum.cc 2010-01-15 15:27:55 +0000 @@ -517,8 +517,7 @@ Field *Item_sum::create_tmp_field(bool g name, table->s, collation.collation); break; case DECIMAL_RESULT: - field= new Field_new_decimal(max_length, maybe_null, name, - decimals, unsigned_flag); + field= Field_new_decimal::create_from_item(this); break; case ROW_RESULT: default: @@ -610,35 +609,6 @@ Item_sum_num::fix_fields(THD *thd, Item } -Item_sum_hybrid::Item_sum_hybrid(THD *thd, Item_sum_hybrid *item) - :Item_sum(thd, item), value(item->value), hybrid_type(item->hybrid_type), - hybrid_field_type(item->hybrid_field_type), cmp_sign(item->cmp_sign), - was_values(item->was_values) -{ - /* copy results from old value */ - switch (hybrid_type) { - case INT_RESULT: - sum_int= item->sum_int; - break; - case DECIMAL_RESULT: - my_decimal2decimal(&item->sum_dec, &sum_dec); - break; - case REAL_RESULT: - sum= item->sum; - break; - case STRING_RESULT: - /* - This can happen with ROLLUP. Note that the value is already - copied at function call. - */ - break; - case ROW_RESULT: - default: - DBUG_ASSERT(0); - } - collation.set(item->collation); -} - bool Item_sum_hybrid::fix_fields(THD *thd, Item **ref) { @@ -658,15 +628,12 @@ Item_sum_hybrid::fix_fields(THD *thd, It switch (hybrid_type= item->result_type()) { case INT_RESULT: max_length= 20; - sum_int= 0; break; case DECIMAL_RESULT: max_length= item->max_length; - my_decimal_set_zero(&sum_dec); break; case REAL_RESULT: max_length= float_length(decimals); - sum= 0.0; break; case STRING_RESULT: max_length= item->max_length; @@ -675,10 +642,10 @@ Item_sum_hybrid::fix_fields(THD *thd, It default: DBUG_ASSERT(0); }; + setup(args[0], NULL); /* MIN/MAX can return NULL for empty set indepedent of the used column */ maybe_null= 1; unsigned_flag=item->unsigned_flag; - collation.set(item->collation); result_field=0; null_value=1; fix_length_and_dec(); @@ -696,6 +663,30 @@ Item_sum_hybrid::fix_fields(THD *thd, It return FALSE; } + +/** + MIN/MAX function setup. + + @param item argument of MIN/MAX function + @param value_arg calculated value of MIN/MAX function + + @details + Setup cache/comparator of MIN/MAX functions. When called by the + copy_or_same function value_arg parameter contains calculated value + of the original MIN/MAX object and it is saved in this object's cache. +*/ + +void Item_sum_hybrid::setup(Item *item, Item *value_arg) +{ + value= Item_cache::get_cache(item); + value->setup(item); + value->store(value_arg); + cmp= new Arg_comparator(); + cmp->set_cmp_func(this, args, (Item**)&value, FALSE); + collation.set(item->collation); +} + + Field *Item_sum_hybrid::create_tmp_field(bool group, TABLE *table, uint convert_blob_length) { @@ -1265,8 +1256,7 @@ Field *Item_sum_avg::create_tmp_field(bo 0, name, &my_charset_bin); } else if (hybrid_type == DECIMAL_RESULT) - field= new Field_new_decimal(max_length, maybe_null, name, - decimals, unsigned_flag); + field= Field_new_decimal::create_from_item(this); else field= new Field_double(max_length, maybe_null, name, decimals, TRUE); if (field) @@ -1587,19 +1577,7 @@ void Item_sum_variance::update_field() void Item_sum_hybrid::clear() { - switch (hybrid_type) { - case INT_RESULT: - sum_int= 0; - break; - case DECIMAL_RESULT: - my_decimal_set_zero(&sum_dec); - break; - case REAL_RESULT: - sum= 0.0; - break; - default: - value.length(0); - } + value->null_value= 1; null_value= 1; } @@ -1608,30 +1586,7 @@ double Item_sum_hybrid::val_real() DBUG_ASSERT(fixed == 1); if (null_value) return 0.0; - switch (hybrid_type) { - case STRING_RESULT: - { - char *end_not_used; - int err_not_used; - String *res; res=val_str(&str_value); - return (res ? my_strntod(res->charset(), (char*) res->ptr(), res->length(), - &end_not_used, &err_not_used) : 0.0); - } - case INT_RESULT: - if (unsigned_flag) - return ulonglong2double(sum_int); - return (double) sum_int; - case DECIMAL_RESULT: - my_decimal2double(E_DEC_FATAL_ERROR, &sum_dec, &sum); - return sum; - case REAL_RESULT: - return sum; - case ROW_RESULT: - default: - // This case should never be choosen - DBUG_ASSERT(0); - return 0; - } + return value->val_real(); } longlong Item_sum_hybrid::val_int() @@ -1639,18 +1594,7 @@ longlong Item_sum_hybrid::val_int() DBUG_ASSERT(fixed == 1); if (null_value) return 0; - switch (hybrid_type) { - case INT_RESULT: - return sum_int; - case DECIMAL_RESULT: - { - longlong result; - my_decimal2int(E_DEC_FATAL_ERROR, &sum_dec, unsigned_flag, &result); - return sum_int; - } - default: - return (longlong) rint(Item_sum_hybrid::val_real()); - } + return value->val_int(); } @@ -1659,26 +1603,7 @@ my_decimal *Item_sum_hybrid::val_decimal DBUG_ASSERT(fixed == 1); if (null_value) return 0; - switch (hybrid_type) { - case STRING_RESULT: - string2my_decimal(E_DEC_FATAL_ERROR, &value, val); - break; - case REAL_RESULT: - double2my_decimal(E_DEC_FATAL_ERROR, sum, val); - break; - case DECIMAL_RESULT: - val= &sum_dec; - break; - case INT_RESULT: - int2my_decimal(E_DEC_FATAL_ERROR, sum_int, unsigned_flag, val); - break; - case ROW_RESULT: - default: - // This case should never be choosen - DBUG_ASSERT(0); - break; - } - return val; // Keep compiler happy + return value->val_decimal(val); } @@ -1688,25 +1613,7 @@ Item_sum_hybrid::val_str(String *str) DBUG_ASSERT(fixed == 1); if (null_value) return 0; - switch (hybrid_type) { - case STRING_RESULT: - return &value; - case REAL_RESULT: - str->set_real(sum,decimals, &my_charset_bin); - break; - case DECIMAL_RESULT: - my_decimal2string(E_DEC_FATAL_ERROR, &sum_dec, 0, 0, 0, str); - return str; - case INT_RESULT: - str->set_int(sum_int, unsigned_flag, &my_charset_bin); - break; - case ROW_RESULT: - default: - // This case should never be choosen - DBUG_ASSERT(0); - break; - } - return str; // Keep compiler happy + return value->val_str(str); } @@ -1715,7 +1622,9 @@ void Item_sum_hybrid::cleanup() DBUG_ENTER("Item_sum_hybrid::cleanup"); Item_sum::cleanup(); forced_const= FALSE; - + if (cmp) + delete cmp; + cmp= 0; /* by default it is TRUE to avoid TRUE reporting by Item_func_not_all/Item_func_nop_all if this item was never called. @@ -1736,63 +1645,22 @@ void Item_sum_hybrid::no_rows_in_result( Item *Item_sum_min::copy_or_same(THD* thd) { - return new (thd->mem_root) Item_sum_min(thd, this); + Item_sum_min *item= new (thd->mem_root) Item_sum_min(thd, this); + item->setup(args[0], value); + return item; } bool Item_sum_min::add() { - switch (hybrid_type) { - case STRING_RESULT: + /* args[0] < value */ + int res= cmp->compare(); + if (!args[0]->null_value && + (null_value || res < 0)) { - String *result=args[0]->val_str(&tmp_value); - if (!args[0]->null_value && - (null_value || sortcmp(&value,result,collation.collation) > 0)) - { - value.copy(*result); - null_value=0; - } - } - break; - case INT_RESULT: - { - longlong nr=args[0]->val_int(); - if (!args[0]->null_value && (null_value || - (unsigned_flag && - (ulonglong) nr < (ulonglong) sum_int) || - (!unsigned_flag && nr < sum_int))) - { - sum_int=nr; - null_value=0; - } - } - break; - case DECIMAL_RESULT: - { - my_decimal value_buff, *val= args[0]->val_decimal(&value_buff); - if (!args[0]->null_value && - (null_value || (my_decimal_cmp(&sum_dec, val) > 0))) - { - my_decimal2decimal(val, &sum_dec); - null_value= 0; - } - } - break; - case REAL_RESULT: - { - double nr= args[0]->val_real(); - if (!args[0]->null_value && (null_value || nr < sum)) - { - sum=nr; - null_value=0; - } - } - break; - case ROW_RESULT: - default: - // This case should never be choosen - DBUG_ASSERT(0); - break; + value->store(args[0]); + value->cache_value(); + null_value= 0; } return 0; } @@ -1800,63 +1668,22 @@ bool Item_sum_min::add() Item *Item_sum_max::copy_or_same(THD* thd) { - return new (thd->mem_root) Item_sum_max(thd, this); + Item_sum_max *item= new (thd->mem_root) Item_sum_max(thd, this); + item->setup(args[0], value); + return item; } bool Item_sum_max::add() { - switch (hybrid_type) { - case STRING_RESULT: + /* args[0] > value */ + int res= cmp->compare(); + if (!args[0]->null_value && + (null_value || res > 0)) { - String *result=args[0]->val_str(&tmp_value); - if (!args[0]->null_value && - (null_value || sortcmp(&value,result,collation.collation) < 0)) - { - value.copy(*result); - null_value=0; - } - } - break; - case INT_RESULT: - { - longlong nr=args[0]->val_int(); - if (!args[0]->null_value && (null_value || - (unsigned_flag && - (ulonglong) nr > (ulonglong) sum_int) || - (!unsigned_flag && nr > sum_int))) - { - sum_int=nr; - null_value=0; - } - } - break; - case DECIMAL_RESULT: - { - my_decimal value_buff, *val= args[0]->val_decimal(&value_buff); - if (!args[0]->null_value && - (null_value || (my_decimal_cmp(val, &sum_dec) > 0))) - { - my_decimal2decimal(val, &sum_dec); - null_value= 0; - } - } - break; - case REAL_RESULT: - { - double nr= args[0]->val_real(); - if (!args[0]->null_value && (null_value || nr > sum)) - { - sum=nr; - null_value=0; - } - } - break; - case ROW_RESULT: - default: - // This case should never be choosen - DBUG_ASSERT(0); - break; + value->store(args[0]); + value->cache_value(); + null_value= 0; } return 0; } @@ -2221,14 +2048,15 @@ void Item_sum_hybrid::update_field() void Item_sum_hybrid::min_max_update_str_field() { - String *res_str=args[0]->val_str(&value); + DBUG_ASSERT(cmp); + String *res_str=args[0]->val_str(&cmp->value1); if (!args[0]->null_value) { - result_field->val_str(&tmp_value); + result_field->val_str(&cmp->value2); if (result_field->is_null() || - (cmp_sign * sortcmp(res_str,&tmp_value,collation.collation)) < 0) + (cmp_sign * sortcmp(res_str,&cmp->value2,collation.collation)) < 0) result_field->store(res_str->ptr(),res_str->length(),res_str->charset()); result_field->set_notnull(); } === modified file 'sql/item_sum.h' --- a/sql/item_sum.h 2009-09-15 10:46:35 +0000 +++ b/sql/item_sum.h 2010-01-15 15:27:55 +0000 @@ -329,22 +329,6 @@ public: virtual void update_field()=0; virtual bool keep_field_type(void) const { return 0; } virtual void fix_length_and_dec() { maybe_null=1; null_value=1; } - /* - This method is used for debug purposes to print the name of an - item to the debug log. The second use of this method is as - a helper function of print(), where it is applicable. - To suit both goals it should return a meaningful, - distinguishable and sintactically correct string. This method - should not be used for runtime type identification, use enum - {Sum}Functype and Item_func::functype()/Item_sum::sum_func() - instead. - - NOTE: for Items inherited from Item_sum, func_name() return part of - function name till first argument (including '(') to make difference in - names for functions with 'distinct' clause and without 'distinct' and - also to make printing of items inherited from Item_sum uniform. - */ - virtual const char *func_name() const= 0; virtual Item *result_item(Field *field) { return new Item_field(field); } /* @@ -679,6 +663,7 @@ public: } void fix_length_and_dec() {} enum Item_result result_type () const { return hybrid_type; } + const char *func_name() const { DBUG_ASSERT(0); return "avg_field"; } }; @@ -747,6 +732,7 @@ public: } void fix_length_and_dec() {} enum Item_result result_type () const { return hybrid_type; } + const char *func_name() const { DBUG_ASSERT(0); return "variance_field"; } }; @@ -822,6 +808,7 @@ public: my_decimal *val_decimal(my_decimal *); enum Item_result result_type () const { return REAL_RESULT; } enum_field_types field_type() const { return MYSQL_TYPE_DOUBLE;} + const char *func_name() const { DBUG_ASSERT(0); return "std_field"; } }; /* @@ -847,14 +834,13 @@ class Item_sum_std :public Item_sum_vari }; // This class is a string or number function depending on num_func - +class Arg_comparator; +class Item_cache; class Item_sum_hybrid :public Item_sum { protected: - String value,tmp_value; - double sum; - longlong sum_int; - my_decimal sum_dec; + Item_cache *value; + Arg_comparator *cmp; Item_result hybrid_type; enum_field_types hybrid_field_type; int cmp_sign; @@ -862,12 +848,17 @@ protected: public: Item_sum_hybrid(Item *item_par,int sign) - :Item_sum(item_par), sum(0.0), sum_int(0), + :Item_sum(item_par), value(0), cmp(0), hybrid_type(INT_RESULT), hybrid_field_type(MYSQL_TYPE_LONGLONG), cmp_sign(sign), was_values(TRUE) { collation.set(&my_charset_bin); } - Item_sum_hybrid(THD *thd, Item_sum_hybrid *item); + Item_sum_hybrid(THD *thd, Item_sum_hybrid *item) + :Item_sum(thd, item), value(item->value), hybrid_type(item->hybrid_type), + hybrid_field_type(item->hybrid_field_type), cmp_sign(item->cmp_sign), + was_values(item->was_values) + { } bool fix_fields(THD *, Item **); + void setup(Item *item, Item *value_arg); void clear(); double val_real(); longlong val_int(); === modified file 'sql/item_timefunc.cc' --- a/sql/item_timefunc.cc 2009-12-03 11:19:05 +0000 +++ b/sql/item_timefunc.cc 2010-01-15 15:27:55 +0000 @@ -391,7 +391,7 @@ static bool extract_date_time(DATE_TIME_ if (tmp - val > 6) tmp= (char*) val + 6; l_time->second_part= (int) my_strtoll10(val, &tmp, &error); - frac_part= 6 - (uint) (tmp - val); + frac_part= 6 - (int) (tmp - val); if (frac_part > 0) l_time->second_part*= (ulong) log_10_int[frac_part]; val= tmp; @@ -882,9 +882,9 @@ static bool get_interval_info(const char value= value*LL(10) + (longlong) (*str - '0'); if (transform_msec && i == count - 1) // microseconds always last { - long msec_length= 6 - (uint) (str - start); + int msec_length= 6 - (int)(str - start); if (msec_length > 0) - value*= (long) log_10_int[msec_length]; + value*= (long)log_10_int[msec_length]; } values[i]= value; while (str != end && !my_isdigit(cs,*str)) === modified file 'sql/item_xmlfunc.cc' --- a/sql/item_xmlfunc.cc 2009-12-03 11:19:05 +0000 +++ b/sql/item_xmlfunc.cc 2010-01-15 15:27:55 +0000 @@ -941,14 +941,16 @@ static Item *create_comparator(MY_XPATH in a loop through all of the nodes in the node set. */ - Item *fake= new Item_string("", 0, xpath->cs); + Item_string *fake= new Item_string("", 0, xpath->cs); + /* Don't cache fake because its value will be changed during comparison.*/ + fake->set_used_tables(RAND_TABLE_BIT); Item_nodeset_func *nodeset; Item *scalar, *comp; if (a->type() == Item::XPATH_NODESET) { nodeset= (Item_nodeset_func*) a; scalar= b; - comp= eq_func(oper, fake, scalar); + comp= eq_func(oper, (Item*)fake, scalar); } else { === modified file 'sql/log.cc' --- a/sql/log.cc 2010-01-04 18:25:29 +0000 +++ b/sql/log.cc 2010-01-15 15:27:55 +0000 @@ -5691,9 +5691,8 @@ int TC_LOG_BINLOG::recover(IO_CACHE *log Xid_log_event *xev=(Xid_log_event *)ev; uchar *x= (uchar *) memdup_root(&mem_root, (uchar*) &xev->xid, sizeof(xev->xid)); - if (! x) + if (!x || my_hash_insert(&xids, x)) goto err2; - my_hash_insert(&xids, x); } delete ev; } === modified file 'sql/log_event.cc' --- a/sql/log_event.cc 2009-12-03 11:34:11 +0000 +++ b/sql/log_event.cc 2010-01-15 15:27:55 +0000 @@ -8453,13 +8453,17 @@ Rows_log_event::write_row(const Relay_lo auto_afree_ptr<char> key(NULL); /* fill table->record[0] with default values */ - + bool abort_on_warnings= (rli->sql_thd->variables.sql_mode & + (MODE_STRICT_TRANS_TABLES | MODE_STRICT_ALL_TABLES)); if ((error= prepare_record(table, m_width, - TRUE /* check if columns have def. values */))) + table->file->ht->db_type != DB_TYPE_NDBCLUSTER, + abort_on_warnings, m_curr_row == m_rows_buf))) DBUG_RETURN(error); /* unpack row into table->record[0] */ - error= unpack_current_row(rli); // TODO: how to handle errors? + if ((error= unpack_current_row(rli, abort_on_warnings))) + DBUG_RETURN(error); + if (m_curr_row == m_rows_buf) { /* this is the first row to be inserted, we estimate the rows with @@ -9256,8 +9260,12 @@ Update_rows_log_event::do_exec_row(const store_record(m_table,record[1]); + bool abort_on_warnings= (rli->sql_thd->variables.sql_mode & + (MODE_STRICT_TRANS_TABLES | MODE_STRICT_ALL_TABLES)); m_curr_row= m_curr_row_end; - error= unpack_current_row(rli); // this also updates m_curr_row_end + /* this also updates m_curr_row_end */ + if ((error= unpack_current_row(rli, abort_on_warnings))) + return error; /* Now we have the right row to update. The old row (the one we're === modified file 'sql/log_event.h' --- a/sql/log_event.h 2009-12-03 11:19:05 +0000 +++ b/sql/log_event.h 2010-01-15 15:27:55 +0000 @@ -3541,12 +3541,16 @@ protected: int write_row(const Relay_log_info *const, const bool); // Unpack the current row into m_table->record[0] - int unpack_current_row(const Relay_log_info *const rli) + int unpack_current_row(const Relay_log_info *const rli, + const bool abort_on_warning= TRUE) { DBUG_ASSERT(m_table); + + bool first_row= (m_curr_row == m_rows_buf); ASSERT_OR_RETURN_ERROR(m_curr_row < m_rows_end, HA_ERR_CORRUPT_EVENT); int const result= ::unpack_row(rli, m_table, m_width, m_curr_row, &m_cols, - &m_curr_row_end, &m_master_reclength); + &m_curr_row_end, &m_master_reclength, + abort_on_warning, first_row); if (m_curr_row_end > m_rows_end) my_error(ER_SLAVE_CORRUPT_EVENT, MYF(0)); ASSERT_OR_RETURN_ERROR(m_curr_row_end <= m_rows_end, HA_ERR_CORRUPT_EVENT); === modified file 'sql/mysqld.cc' --- a/sql/mysqld.cc 2009-12-04 15:12:22 +0000 +++ b/sql/mysqld.cc 2010-01-29 18:42:22 +0000 @@ -1173,11 +1173,14 @@ static void __cdecl kill_server(int sig_ /* Send event to smem_event_connect_request for aborting */ - if (!SetEvent(smem_event_connect_request)) + if (opt_enable_shared_memory) { - DBUG_PRINT("error", - ("Got error: %ld from SetEvent of smem_event_connect_request", - GetLastError())); + if (!SetEvent(smem_event_connect_request)) + { + DBUG_PRINT("error", + ("Got error: %ld from SetEvent of smem_event_connect_request", + GetLastError())); + } } #endif @@ -2704,7 +2707,7 @@ static void init_signals(void) { /* Change limits so that we will get a core file */ STRUCT_RLIMIT rl; - rl.rlim_cur = rl.rlim_max = RLIM_INFINITY; + rl.rlim_cur = rl.rlim_max = (rlim_t) RLIM_INFINITY; if (setrlimit(RLIMIT_CORE, &rl) && global_system_variables.log_warnings) sql_print_warning("setrlimit could not change the size of core files to 'infinity'; We may not be able to generate a core file on signals"); } @@ -4005,6 +4008,27 @@ server."); if (opt_bin_log) { + /* Reports an error and aborts, if the --log-bin's path + is a directory.*/ + if (opt_bin_logname && + opt_bin_logname[strlen(opt_bin_logname) - 1] == FN_LIBCHAR) + { + sql_print_error("Path '%s' is a directory name, please specify \ +a file name for --log-bin option", opt_bin_logname); + unireg_abort(1); + } + + /* Reports an error and aborts, if the --log-bin-index's path + is a directory.*/ + if (opt_binlog_index_name && + opt_binlog_index_name[strlen(opt_binlog_index_name) - 1] + == FN_LIBCHAR) + { + sql_print_error("Path '%s' is a directory name, please specify \ +a file name for --log-bin-index option", opt_binlog_index_name); + unireg_abort(1); + } + char buf[FN_REFLEN]; const char *ln; ln= mysql_bin_log.generate_name(opt_bin_logname, "-bin", 1, buf); @@ -5355,12 +5379,16 @@ pthread_handler_t handle_connections_soc pthread_handler_t handle_connections_namedpipes(void *arg) { HANDLE hConnectedPipe; - OVERLAPPED connectOverlapped = {0}; + OVERLAPPED connectOverlapped= {0}; THD *thd; my_thread_init(); DBUG_ENTER("handle_connections_namedpipes"); - connectOverlapped.hEvent = CreateEvent(NULL, TRUE, FALSE, NULL); - + connectOverlapped.hEvent= CreateEvent(NULL, TRUE, FALSE, NULL); + if (!connectOverlapped.hEvent) + { + sql_print_error("Can't create event, last error=%u", GetLastError()); + unireg_abort(1); + } DBUG_PRINT("general",("Waiting for named pipe connections.")); while (!abort_loop) { @@ -5383,7 +5411,8 @@ pthread_handler_t handle_connections_nam { CloseHandle(hPipe); if ((hPipe= CreateNamedPipe(pipe_name, - PIPE_ACCESS_DUPLEX|FILE_FLAG_OVERLAPPED, + PIPE_ACCESS_DUPLEX | + FILE_FLAG_OVERLAPPED, PIPE_TYPE_BYTE | PIPE_READMODE_BYTE | PIPE_WAIT, @@ -5403,7 +5432,8 @@ pthread_handler_t handle_connections_nam hConnectedPipe = hPipe; /* create new pipe for new connection */ if ((hPipe = CreateNamedPipe(pipe_name, - PIPE_ACCESS_DUPLEX|FILE_FLAG_OVERLAPPED, + PIPE_ACCESS_DUPLEX | + FILE_FLAG_OVERLAPPED, PIPE_TYPE_BYTE | PIPE_READMODE_BYTE | PIPE_WAIT, @@ -8974,14 +9004,8 @@ static int fix_paths(void) pos[0]= FN_LIBCHAR; pos[1]= 0; } - convert_dirname(mysql_real_data_home,mysql_real_data_home,NullS); - my_realpath(mysql_unpacked_real_data_home, mysql_real_data_home, MYF(0)); - mysql_unpacked_real_data_home_len= strlen(mysql_unpacked_real_data_home); - if (mysql_unpacked_real_data_home[mysql_unpacked_real_data_home_len-1] == FN_LIBCHAR) - --mysql_unpacked_real_data_home_len; - - convert_dirname(language,language,NullS); + convert_dirname(mysql_real_data_home,mysql_real_data_home,NullS); (void) my_load_path(mysql_home,mysql_home,""); // Resolve current dir (void) my_load_path(mysql_real_data_home,mysql_real_data_home,mysql_home); (void) my_load_path(pidfile_name,pidfile_name,mysql_real_data_home); @@ -8989,6 +9013,12 @@ static int fix_paths(void) get_relative_path(PLUGINDIR), mysql_home); opt_plugin_dir_ptr= opt_plugin_dir; + my_realpath(mysql_unpacked_real_data_home, mysql_real_data_home, MYF(0)); + mysql_unpacked_real_data_home_len= + (int) strlen(mysql_unpacked_real_data_home); + if (mysql_unpacked_real_data_home[mysql_unpacked_real_data_home_len-1] == FN_LIBCHAR) + --mysql_unpacked_real_data_home_len; + char *sharedir=get_relative_path(SHAREDIR); if (test_if_hard_path(sharedir)) strmake(buff,sharedir,sizeof(buff)-1); /* purecov: tested */ @@ -9019,8 +9049,8 @@ static int fix_paths(void) /* Convert the secure-file-priv option to system format, allowing a quick strcmp to check if read or write is in an allowed dir - */ - if (opt_secure_file_priv) + */ + if (opt_secure_file_priv && opt_secure_file_priv[0]) { convert_dirname(buff, opt_secure_file_priv, NullS); my_free(opt_secure_file_priv, MYF(0)); === modified file 'sql/net_serv.cc' --- a/sql/net_serv.cc 2009-09-07 20:50:10 +0000 +++ b/sql/net_serv.cc 2010-01-29 10:42:31 +0000 @@ -219,7 +219,7 @@ my_bool net_realloc(NET *net, size_t len -1 Don't know if data is ready or not */ -#if !defined(EMBEDDED_LIBRARY) +#if !defined(EMBEDDED_LIBRARY) && defined(DBUG_OFF) static int net_data_is_ready(my_socket sd) { @@ -280,13 +280,18 @@ static int net_data_is_ready(my_socket s void net_clear(NET *net, my_bool clear_buffer) { -#if !defined(EMBEDDED_LIBRARY) +#if !defined(EMBEDDED_LIBRARY) && defined(DBUG_OFF) size_t count; int ready; #endif DBUG_ENTER("net_clear"); -#if !defined(EMBEDDED_LIBRARY) +/* + We don't do a clear in case of DBUG_OFF to catch bugs + in the protocol handling +*/ + +#if !defined(EMBEDDED_LIBRARY) && defined(DBUG_OFF) if (clear_buffer) { while ((ready= net_data_is_ready(net->vio->sd)) > 0) === modified file 'sql/opt_range.cc' --- a/sql/opt_range.cc 2009-12-03 11:19:05 +0000 +++ b/sql/opt_range.cc 2010-01-15 15:27:55 +0000 @@ -446,9 +446,9 @@ public: range_key, *range_key_flag); *range_key_flag|= key_tree->min_flag; if (key_tree->next_key_part && + key_tree->next_key_part->type == SEL_ARG::KEY_RANGE && key_tree->next_key_part->part == key_tree->part+1 && - !(*range_key_flag & (NO_MIN_RANGE | NEAR_MIN)) && - key_tree->next_key_part->type == SEL_ARG::KEY_RANGE) + !(*range_key_flag & (NO_MIN_RANGE | NEAR_MIN))) res+= key_tree->next_key_part->store_min_key(key, range_key, range_key_flag); return res; @@ -462,9 +462,9 @@ public: range_key, *range_key_flag); (*range_key_flag)|= key_tree->max_flag; if (key_tree->next_key_part && + key_tree->next_key_part->type == SEL_ARG::KEY_RANGE && key_tree->next_key_part->part == key_tree->part+1 && - !(*range_key_flag & (NO_MAX_RANGE | NEAR_MAX)) && - key_tree->next_key_part->type == SEL_ARG::KEY_RANGE) + !(*range_key_flag & (NO_MAX_RANGE | NEAR_MAX))) res+= key_tree->next_key_part->store_max_key(key, range_key, range_key_flag); return res; @@ -1700,6 +1700,7 @@ SEL_ARG *SEL_ARG::clone(RANGE_OPT_PARAM tmp->prev= *next_arg; // Link into next/prev chain (*next_arg)->next=tmp; (*next_arg)= tmp; + tmp->part= this->part; } else { @@ -6672,6 +6673,7 @@ key_or(RANGE_OPT_PARAM *param, SEL_ARG * else if ((cmp=tmp->cmp_max_to_min(key2)) < 0) { // Found tmp.max < key2.min SEL_ARG *next=tmp->next; + /* key1 on the left of key2 non-overlapping */ if (cmp == -2 && eq_tree(tmp->next_key_part,key2->next_key_part)) { // Join near ranges like tmp.max < 0 and key2.min >= 0 @@ -6700,6 +6702,7 @@ key_or(RANGE_OPT_PARAM *param, SEL_ARG * int tmp_cmp; if ((tmp_cmp=tmp->cmp_min_to_max(key2)) > 0) // if tmp.min > key2.max { + /* tmp is on the right of key2 non-overlapping */ if (tmp_cmp == 2 && eq_tree(tmp->next_key_part,key2->next_key_part)) { // ranges are connected tmp->copy_min_to_min(key2); @@ -6734,25 +6737,52 @@ key_or(RANGE_OPT_PARAM *param, SEL_ARG * } } - // tmp.max >= key2.min && tmp.min <= key.max (overlapping ranges) + /* + tmp.min >= key2.min && tmp.min <= key.max (overlapping ranges) + key2.min <= tmp.min <= key2.max + */ if (eq_tree(tmp->next_key_part,key2->next_key_part)) { if (tmp->is_same(key2)) { + /* + Found exact match of key2 inside key1. + Use the relevant range in key1. + */ tmp->merge_flags(key2); // Copy maybe flags key2->increment_use_count(-1); // Free not used tree } else { SEL_ARG *last=tmp; + SEL_ARG *first=tmp; + /* + Find the last range in tmp that overlaps key2 and has the same + condition on the rest of the keyparts. + */ while (last->next && last->next->cmp_min_to_max(key2) <= 0 && eq_tree(last->next->next_key_part,key2->next_key_part)) { + /* + We've found the last overlapping key1 range in last. + This means that the ranges between (and including) the + first overlapping range (tmp) and the last overlapping range + (last) are fully nested into the current range of key2 + and can safely be discarded. We just need the minimum endpoint + of the first overlapping range (tmp) so we can compare it with + the minimum endpoint of the enclosing key2 range. + */ SEL_ARG *save=last; last=last->next; key1=key1->tree_delete(save); } - last->copy_min(tmp); + /* + The tmp range (the first overlapping range) could have been discarded + by the previous loop. We should re-direct tmp to the new united range + that's taking its place. + */ + tmp= last; + last->copy_min(first); bool full_range= last->copy_min(key2); if (!full_range) { @@ -7262,27 +7292,25 @@ int test_rb_tree(SEL_ARG *element,SEL_AR } -/* - Count how many times SEL_ARG graph "root" refers to its part "key" +/** + Count how many times SEL_ARG graph "root" refers to its part "key" via + transitive closure. - SYNOPSIS - count_key_part_usage() - root An RB-Root node in a SEL_ARG graph. - key Another RB-Root node in that SEL_ARG graph. + @param root An RB-Root node in a SEL_ARG graph. + @param key Another RB-Root node in that SEL_ARG graph. - DESCRIPTION - The passed "root" node may refer to "key" node via root->next_key_part, - root->next->n + The passed "root" node may refer to "key" node via root->next_key_part, + root->next->n - This function counts how many times the node "key" is referred (via - SEL_ARG::next_key_part) by - - intervals of RB-tree pointed by "root", - - intervals of RB-trees that are pointed by SEL_ARG::next_key_part from - intervals of RB-tree pointed by "root", - - and so on. + This function counts how many times the node "key" is referred (via + SEL_ARG::next_key_part) by + - intervals of RB-tree pointed by "root", + - intervals of RB-trees that are pointed by SEL_ARG::next_key_part from + intervals of RB-tree pointed by "root", + - and so on. - Here is an example (horizontal links represent next_key_part pointers, - vertical links - next/prev prev pointers): + Here is an example (horizontal links represent next_key_part pointers, + vertical links - next/prev prev pointers): +----+ $ |root|-----------------+ @@ -7302,8 +7330,8 @@ int test_rb_tree(SEL_ARG *element,SEL_AR ... +---+ $ | | |------------+ +---+ $ - RETURN - Number of links to "key" from nodes reachable from "root". + @return + Number of links to "key" from nodes reachable from "root". */ static ulong count_key_part_usage(SEL_ARG *root, SEL_ARG *key) @@ -7558,8 +7586,8 @@ check_quick_keys(PARAM *param, uint idx, param->first_null_comp= key_tree->part+1; if (key_tree->next_key_part && - key_tree->next_key_part->part == key_tree->part+1 && - key_tree->next_key_part->type == SEL_ARG::KEY_RANGE) + key_tree->next_key_part->type == SEL_ARG::KEY_RANGE && + key_tree->next_key_part->part == key_tree->part+1) { // const key as prefix if (min_key_length == max_key_length && !memcmp(min_key, max_key, (uint) (tmp_max_key - max_key)) && @@ -7840,8 +7868,8 @@ get_quick_keys(PARAM *param,QUICK_RANGE_ &tmp_max_key,max_key_flag); if (key_tree->next_key_part && - key_tree->next_key_part->part == key_tree->part+1 && - key_tree->next_key_part->type == SEL_ARG::KEY_RANGE) + key_tree->next_key_part->type == SEL_ARG::KEY_RANGE && + key_tree->next_key_part->part == key_tree->part+1) { // const key as prefix if ((tmp_min_key - min_key) == (tmp_max_key - max_key) && memcmp(min_key, max_key, (uint)(tmp_max_key - max_key))==0 && @@ -9823,7 +9851,11 @@ check_group_min_max_predicates(COND *con } else if (cur_arg->const_item()) { - DBUG_RETURN(TRUE); + /* + For predicates of the form "const OP expr" we also have to check 'expr' + to make a decision. + */ + continue; } else DBUG_RETURN(FALSE); === modified file 'sql/repl_failsafe.cc' --- a/sql/repl_failsafe.cc 2009-09-23 13:10:23 +0000 +++ b/sql/repl_failsafe.cc 2009-11-20 15:18:01 +0000 @@ -559,7 +559,12 @@ HOSTS"; goto err; } si->server_id = log_server_id; - my_hash_insert(&slave_list, (uchar*)si); + if (my_hash_insert(&slave_list, (uchar*)si)) + { + error= "the slave is out of memory"; + pthread_mutex_unlock(&LOCK_slave_list); + goto err; + } } strmake(si->host, row[1], sizeof(si->host)-1); si->port = atoi(row[port_ind]); === modified file 'sql/rpl_record.cc' --- a/sql/rpl_record.cc 2009-03-05 19:54:53 +0000 +++ b/sql/rpl_record.cc 2010-01-28 11:35:10 +0000 @@ -180,7 +180,8 @@ int unpack_row(Relay_log_info const *rli, TABLE *table, uint const colcnt, uchar const *const row_data, MY_BITMAP const *cols, - uchar const **const row_end, ulong *const master_reclength) + uchar const **const row_end, ulong *const master_reclength, + const bool abort_on_warning, const bool first_row) { DBUG_ENTER("unpack_row"); DBUG_ASSERT(row_data); @@ -224,8 +225,35 @@ unpack_row(Relay_log_info const *rli, /* Field...::unpack() cannot return 0 */ DBUG_ASSERT(pack_ptr != NULL); - if ((null_bits & null_mask) && f->maybe_null()) - f->set_null(); + if (null_bits & null_mask) + { + if (f->maybe_null()) + { + DBUG_PRINT("debug", ("Was NULL; null mask: 0x%x; null bits: 0x%x", + null_mask, null_bits)); + f->set_null(); + } + else + { + MYSQL_ERROR::enum_warning_level error_type= + MYSQL_ERROR::WARN_LEVEL_NOTE; + if (abort_on_warning && (table->file->has_transactions() || + first_row)) + { + error = HA_ERR_ROWS_EVENT_APPLY; + error_type= MYSQL_ERROR::WARN_LEVEL_ERROR; + } + else + { + f->set_default(); + error_type= MYSQL_ERROR::WARN_LEVEL_WARN; + } + push_warning_printf(current_thd, error_type, + ER_BAD_NULL_ERROR, + ER(ER_BAD_NULL_ERROR), + f->field_name); + } + } else { f->set_notnull(); @@ -305,13 +333,17 @@ unpack_row(Relay_log_info const *rli, @param table Table whose record[0] buffer is prepared. @param skip Number of columns for which default/nullable check should be skipped. - @param check Indicates if errors should be raised when checking - default/nullable field properties. + @param check Specifies if lack of default error needs checking. + @param abort_on_warning + Controls how to react on lack of a field's default. + The parameter mimics the master side one for + @c check_that_all_fields_are_given_values. @returns 0 on success or a handler level error code */ int prepare_record(TABLE *const table, - const uint skip, const bool check) + const uint skip, const bool check, + const bool abort_on_warning, const bool first_row) { DBUG_ENTER("prepare_record"); @@ -326,17 +358,36 @@ int prepare_record(TABLE *const table, if (skip >= table->s->fields || !check) DBUG_RETURN(0); - /* Checking if exists default/nullable fields in the default values. */ - - for (Field **field_ptr= table->field+skip ; *field_ptr ; ++field_ptr) + /* + For fields the extra fields on the slave, we check if they have a default. + The check follows the same rules as the INSERT query without specifying an + explicit value for a field not having the explicit default + (@c check_that_all_fields_are_given_values()). + */ + for (Field **field_ptr= table->field+skip; *field_ptr; ++field_ptr) { - uint32 const mask= NOT_NULL_FLAG | NO_DEFAULT_VALUE_FLAG; Field *const f= *field_ptr; - - if (((f->flags & mask) == mask)) + if ((f->flags & NO_DEFAULT_VALUE_FLAG) && + (f->real_type() != MYSQL_TYPE_ENUM)) { - my_error(ER_NO_DEFAULT_FOR_FIELD, MYF(0), f->field_name); - error = HA_ERR_ROWS_EVENT_APPLY; + + MYSQL_ERROR::enum_warning_level error_type= + MYSQL_ERROR::WARN_LEVEL_NOTE; + if (abort_on_warning && (table->file->has_transactions() || + first_row)) + { + error= HA_ERR_ROWS_EVENT_APPLY; + error_type= MYSQL_ERROR::WARN_LEVEL_ERROR; + } + else + { + f->set_default(); + error_type= MYSQL_ERROR::WARN_LEVEL_WARN; + } + push_warning_printf(current_thd, error_type, + ER_NO_DEFAULT_FOR_FIELD, + ER(ER_NO_DEFAULT_FOR_FIELD), + f->field_name); } } === modified file 'sql/rpl_record.h' --- a/sql/rpl_record.h 2008-01-31 12:54:03 +0000 +++ b/sql/rpl_record.h 2009-10-22 00:15:45 +0000 @@ -27,10 +27,13 @@ size_t pack_row(TABLE* table, MY_BITMAP int unpack_row(Relay_log_info const *rli, TABLE *table, uint const colcnt, uchar const *const row_data, MY_BITMAP const *cols, - uchar const **const row_end, ulong *const master_reclength); + uchar const **const row_end, ulong *const master_reclength, + const bool abort_on_warning= TRUE, const bool first_row= TRUE); // Fill table's record[0] with default values. -int prepare_record(TABLE *const, const uint =0, const bool =FALSE); +int prepare_record(TABLE *const table, const uint skip, const bool check, + const bool abort_on_warning= TRUE, + const bool first_row= TRUE); #endif #endif === modified file 'sql/rpl_rli.cc' --- a/sql/rpl_rli.cc 2009-12-03 11:19:05 +0000 +++ b/sql/rpl_rli.cc 2010-01-15 15:27:55 +0000 @@ -100,7 +100,8 @@ int init_relay_log_info(Relay_log_info* rli->tables_to_lock_count= 0; char pattern[FN_REFLEN]; - if (fn_format(pattern, PREFIX_SQL_LOAD, slave_load_tmpdir, "", + (void) my_realpath(pattern, slave_load_tmpdir, 0); + if (fn_format(pattern, PREFIX_SQL_LOAD, pattern, "", MY_SAFE_PATH | MY_RETURN_REAL_PATH) == NullS) { pthread_mutex_unlock(&rli->data_lock); @@ -127,6 +128,29 @@ int init_relay_log_info(Relay_log_info* rli->relay_log.max_size (and mysql_bin_log.max_size). */ { + /* Reports an error and returns, if the --relay-log's path + is a directory.*/ + if (opt_relay_logname && + opt_relay_logname[strlen(opt_relay_logname) - 1] == FN_LIBCHAR) + { + pthread_mutex_unlock(&rli->data_lock); + sql_print_error("Path '%s' is a directory name, please specify \ +a file name for --relay-log option", opt_relay_logname); + DBUG_RETURN(1); + } + + /* Reports an error and returns, if the --relay-log-index's path + is a directory.*/ + if (opt_relaylog_index_name && + opt_relaylog_index_name[strlen(opt_relaylog_index_name) - 1] + == FN_LIBCHAR) + { + pthread_mutex_unlock(&rli->data_lock); + sql_print_error("Path '%s' is a directory name, please specify \ +a file name for --relay-log-index option", opt_relaylog_index_name); + DBUG_RETURN(1); + } + char buf[FN_REFLEN]; const char *ln; static bool name_warning_sent= 0; === modified file 'sql/rpl_tblmap.cc' --- a/sql/rpl_tblmap.cc 2008-08-20 14:06:31 +0000 +++ b/sql/rpl_tblmap.cc 2009-11-20 15:18:01 +0000 @@ -119,7 +119,13 @@ int table_mapping::set_table(ulong table } e->table_id= table_id; e->table= table; - my_hash_insert(&m_table_ids,(uchar *)e); + if (my_hash_insert(&m_table_ids,(uchar *)e)) + { + /* we add this entry to the chain of free (free for use) entries */ + e->next= m_free; + m_free= e; + DBUG_RETURN(ERR_MEMORY_ALLOCATION); + } DBUG_PRINT("info", ("tid %lu -> table 0x%lx (%s)", table_id, (long) e->table, === modified file 'sql/set_var.cc' --- a/sql/set_var.cc 2009-12-03 11:19:05 +0000 +++ b/sql/set_var.cc 2010-01-14 16:32:41 +0000 @@ -58,6 +58,9 @@ #include <my_getopt.h> #include <thr_alarm.h> #include <myisam.h> +#ifdef WITH_MARIA_STORAGE_ENGINE +#include <maria.h> +#endif #include <my_dir.h> #include <waiting_threads.h> #include "events.h" === modified file 'sql/slave.cc' --- a/sql/slave.cc 2009-12-06 17:51:48 +0000 +++ b/sql/slave.cc 2010-01-29 10:42:31 +0000 @@ -3803,10 +3803,11 @@ static int connect_to_master(THD* thd, M suppress_warnings= 0; mi->report(ERROR_LEVEL, last_errno, "error %s to master '%s@%s:%d'" - " - retry-time: %d retries: %lu", + " - retry-time: %d retries: %lu message: %s", (reconnect ? "reconnecting" : "connecting"), mi->user, mi->host, mi->port, - mi->connect_retry, master_retry_count); + mi->connect_retry, master_retry_count, + mysql_error(mysql)); } /* By default we try forever. The reason is that failure will trigger === modified file 'sql/sp.cc' --- a/sql/sp.cc 2009-10-16 10:29:42 +0000 +++ b/sql/sp.cc 2009-11-21 11:18:21 +0000 @@ -70,6 +70,122 @@ enum MYSQL_PROC_FIELD_COUNT }; +static const +TABLE_FIELD_TYPE proc_table_fields[MYSQL_PROC_FIELD_COUNT] = +{ + { + { C_STRING_WITH_LEN("db") }, + { C_STRING_WITH_LEN("char(64)") }, + { C_STRING_WITH_LEN("utf8") } + }, + { + { C_STRING_WITH_LEN("name") }, + { C_STRING_WITH_LEN("char(64)") }, + { C_STRING_WITH_LEN("utf8") } + }, + { + { C_STRING_WITH_LEN("type") }, + { C_STRING_WITH_LEN("enum('FUNCTION','PROCEDURE')") }, + { NULL, 0 } + }, + { + { C_STRING_WITH_LEN("specific_name") }, + { C_STRING_WITH_LEN("char(64)") }, + { C_STRING_WITH_LEN("utf8") } + }, + { + { C_STRING_WITH_LEN("language") }, + { C_STRING_WITH_LEN("enum('SQL')") }, + { NULL, 0 } + }, + { + { C_STRING_WITH_LEN("sql_data_access") }, + { C_STRING_WITH_LEN("enum('CONTAINS_SQL','NO_SQL','READS_SQL_DATA','MODIFIES_SQL_DATA')") }, + { NULL, 0 } + }, + { + { C_STRING_WITH_LEN("is_deterministic") }, + { C_STRING_WITH_LEN("enum('YES','NO')") }, + { NULL, 0 } + }, + { + { C_STRING_WITH_LEN("security_type") }, + { C_STRING_WITH_LEN("enum('INVOKER','DEFINER')") }, + { NULL, 0 } + }, + { + { C_STRING_WITH_LEN("param_list") }, + { C_STRING_WITH_LEN("blob") }, + { NULL, 0 } + }, + + { + { C_STRING_WITH_LEN("returns") }, + { C_STRING_WITH_LEN("longblob") }, + { NULL, 0 } + }, + { + { C_STRING_WITH_LEN("body") }, + { C_STRING_WITH_LEN("longblob") }, + { NULL, 0 } + }, + { + { C_STRING_WITH_LEN("definer") }, + { C_STRING_WITH_LEN("char(77)") }, + { C_STRING_WITH_LEN("utf8") } + }, + { + { C_STRING_WITH_LEN("created") }, + { C_STRING_WITH_LEN("timestamp") }, + { NULL, 0 } + }, + { + { C_STRING_WITH_LEN("modified") }, + { C_STRING_WITH_LEN("timestamp") }, + { NULL, 0 } + }, + { + { C_STRING_WITH_LEN("sql_mode") }, + { C_STRING_WITH_LEN("set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES'," + "'IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION'," + "'NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB'," + "'NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40'," + "'ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES'," + "'STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES'," + "'ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER'," + "'HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH')") }, + { NULL, 0 } + }, + { + { C_STRING_WITH_LEN("comment") }, + { C_STRING_WITH_LEN("char(64)") }, + { C_STRING_WITH_LEN("utf8") } + }, + { + { C_STRING_WITH_LEN("character_set_client") }, + { C_STRING_WITH_LEN("char(32)") }, + { C_STRING_WITH_LEN("utf8") } + }, + { + { C_STRING_WITH_LEN("collation_connection") }, + { C_STRING_WITH_LEN("char(32)") }, + { C_STRING_WITH_LEN("utf8") } + }, + { + { C_STRING_WITH_LEN("db_collation") }, + { C_STRING_WITH_LEN("char(32)") }, + { C_STRING_WITH_LEN("utf8") } + }, + { + { C_STRING_WITH_LEN("body_utf8") }, + { C_STRING_WITH_LEN("longblob") }, + { NULL, 0 } + } +}; + +static const TABLE_FIELD_DEF + proc_table_def= {MYSQL_PROC_FIELD_COUNT, proc_table_fields}; + /*************************************************************************/ /** @@ -247,6 +363,50 @@ Stored_routine_creation_ctx::load_from_d /*************************************************************************/ +class Proc_table_intact : public Table_check_intact +{ +private: + bool m_print_once; + +public: + Proc_table_intact() : m_print_once(TRUE) {} + +protected: + void report_error(uint code, const char *fmt, ...); +}; + + +/** + Report failure to validate the mysql.proc table definition. + Print a message to the error log only once. +*/ + +void Proc_table_intact::report_error(uint code, const char *fmt, ...) +{ + va_list args; + char buf[512]; + + va_start(args, fmt); + my_vsnprintf(buf, sizeof(buf), fmt, args); + va_end(args); + + if (code) + my_message(code, buf, MYF(0)); + else + my_error(ER_CANNOT_LOAD_FROM_TABLE, MYF(0), "proc"); + + if (m_print_once) + { + m_print_once= FALSE; + sql_print_error("%s", buf); + } +}; + + +/** Single instance used to control printing to the error log. */ +static Proc_table_intact proc_table_intact; + + /** Open the mysql.proc table for read. @@ -266,15 +426,17 @@ TABLE *open_proc_table_for_read(THD *thd DBUG_ENTER("open_proc_table_for_read"); TABLE_LIST table; - bzero((char*) &table, sizeof(table)); - table.db= (char*) "mysql"; - table.table_name= table.alias= (char*)"proc"; - table.lock_type= TL_READ; + table.init_one_table("mysql", "proc", TL_READ); + + if (open_system_tables_for_read(thd, &table, backup)) + DBUG_RETURN(NULL); - if (!open_system_tables_for_read(thd, &table, backup)) + if (!proc_table_intact.check(table.table, &proc_table_def)) DBUG_RETURN(table.table); - else - DBUG_RETURN(0); + + close_system_tables(thd, backup); + + DBUG_RETURN(NULL); } @@ -296,13 +458,19 @@ static TABLE *open_proc_table_for_update { DBUG_ENTER("open_proc_table_for_update"); - TABLE_LIST table; - bzero((char*) &table, sizeof(table)); - table.db= (char*) "mysql"; - table.table_name= table.alias= (char*)"proc"; - table.lock_type= TL_WRITE; + TABLE *table; + TABLE_LIST table_list; + table_list.init_one_table("mysql", "proc", TL_WRITE); + + if (!(table= open_system_table_for_update(thd, &table_list))) + DBUG_RETURN(NULL); + + if (!proc_table_intact.check(table, &proc_table_def)) + DBUG_RETURN(table); + + close_thread_tables(thd); - DBUG_RETURN(open_system_table_for_update(thd, &table)); + DBUG_RETURN(NULL); } @@ -1506,7 +1674,8 @@ static bool add_used_routine(LEX *lex, Q rn->key.length= key->length; rn->key.str= (char *)rn + sizeof(Sroutine_hash_entry); memcpy(rn->key.str, key->str, key->length + 1); - my_hash_insert(&lex->sroutines, (uchar *)rn); + if (my_hash_insert(&lex->sroutines, (uchar *)rn)) + return FALSE; lex->sroutines_list.link_in_list((uchar *)rn, (uchar **)&rn->next); rn->belong_to_view= belong_to_view; return TRUE; @@ -1584,16 +1753,24 @@ void sp_remove_not_own_routines(LEX *lex dependant on time of life of elements from source hash. It also won't touch lists linking elements in source and destination hashes. + + @returns + @return TRUE Failure + @return FALSE Success */ -void sp_update_sp_used_routines(HASH *dst, HASH *src) +bool sp_update_sp_used_routines(HASH *dst, HASH *src) { for (uint i=0 ; i < src->records ; i++) { Sroutine_hash_entry *rt= (Sroutine_hash_entry *)hash_element(src, i); if (!hash_search(dst, (uchar *)rt->key.str, rt->key.length)) - my_hash_insert(dst, (uchar *)rt); + { + if (my_hash_insert(dst, (uchar *)rt)) + return TRUE; + } } + return FALSE; } === modified file 'sql/sp.h' --- a/sql/sp.h 2009-07-28 17:44:38 +0000 +++ b/sql/sp.h 2009-11-20 15:18:01 +0000 @@ -69,7 +69,7 @@ void sp_get_prelocking_info(THD *thd, bo void sp_add_used_routine(LEX *lex, Query_arena *arena, sp_name *rt, char rt_type); void sp_remove_not_own_routines(LEX *lex); -void sp_update_sp_used_routines(HASH *dst, HASH *src); +bool sp_update_sp_used_routines(HASH *dst, HASH *src); int sp_cache_routines_and_add_tables(THD *thd, LEX *lex, bool first_no_prelock); int sp_cache_routines_and_add_tables_for_view(THD *thd, LEX *lex, === modified file 'sql/sp_cache.cc' --- a/sql/sp_cache.cc 2008-12-02 22:02:52 +0000 +++ b/sql/sp_cache.cc 2010-01-15 15:27:55 +0000 @@ -36,10 +36,16 @@ public: sp_cache(); ~sp_cache(); - inline void insert(sp_head *sp) + /** + Inserts a sp_head object into a hash table. + + @returns Success status + @return TRUE Failure + @return FALSE Success + */ + inline bool insert(sp_head *sp) { - /* TODO: why don't we check return value? */ - my_hash_insert(&m_hashtable, (const uchar *)sp); + return my_hash_insert(&m_hashtable, (const uchar *)sp); } inline sp_head *lookup(char *name, uint namelen) === modified file 'sql/sp_head.cc' --- a/sql/sp_head.cc 2009-12-03 11:34:11 +0000 +++ b/sql/sp_head.cc 2010-01-15 15:27:55 +0000 @@ -2090,8 +2090,18 @@ sp_head::reset_lex(THD *thd) DBUG_RETURN(FALSE); } -/// Restore lex during parsing, after we have parsed a sub statement. -void + +/** + Restore lex during parsing, after we have parsed a sub statement. + + @param thd Thread handle + + @return + @retval TRUE failure + @retval FALSE success +*/ + +bool sp_head::restore_lex(THD *thd) { DBUG_ENTER("sp_head::restore_lex"); @@ -2102,7 +2112,7 @@ sp_head::restore_lex(THD *thd) oldlex= (LEX *)m_lex.pop(); if (! oldlex) - return; // Nothing to restore + DBUG_RETURN(FALSE); // Nothing to restore oldlex->trg_table_fields.push_back(&sublex->trg_table_fields); @@ -2118,7 +2128,8 @@ sp_head::restore_lex(THD *thd) Add routines which are used by statement to respective set for this routine. */ - sp_update_sp_used_routines(&m_sroutines, &sublex->sroutines); + if (sp_update_sp_used_routines(&m_sroutines, &sublex->sroutines)) + DBUG_RETURN(TRUE); /* Merge tables used by this statement (but not by its functions or procedures) to multiset of tables used by this routine. @@ -2130,7 +2141,7 @@ sp_head::restore_lex(THD *thd) delete sublex; } thd->lex= oldlex; - DBUG_VOID_RETURN; + DBUG_RETURN(FALSE); } /** @@ -3867,7 +3878,8 @@ sp_head::merge_table_list(THD *thd, TABL tab->lock_type= table->lock_type; tab->lock_count= tab->query_lock_count= 1; tab->trg_event_map= table->trg_event_map; - my_hash_insert(&m_sptabs, (uchar *)tab); + if (my_hash_insert(&m_sptabs, (uchar *)tab)) + return FALSE; } } return TRUE; === modified file 'sql/sp_head.h' --- a/sql/sp_head.h 2009-04-29 02:59:10 +0000 +++ b/sql/sp_head.h 2009-11-20 15:18:01 +0000 @@ -340,7 +340,7 @@ public: @todo Conflicting comment in sp_head.cc */ - void + bool restore_lex(THD *thd); /// Put the instruction on the backpatch list, associated with the label. === modified file 'sql/sp_rcontext.cc' --- a/sql/sp_rcontext.cc 2008-01-23 22:36:57 +0000 +++ b/sql/sp_rcontext.cc 2009-11-06 19:34:25 +0000 @@ -617,7 +617,7 @@ sp_rcontext::set_case_expr(THD *thd, int } m_case_expr_holders[case_expr_id]->store(case_expr_item); - + m_case_expr_holders[case_expr_id]->cache_value(); return FALSE; } === modified file 'sql/sql_acl.cc' --- a/sql/sql_acl.cc 2009-12-03 11:19:05 +0000 +++ b/sql/sql_acl.cc 2010-01-15 15:27:55 +0000 @@ -31,9 +31,8 @@ #include "sp_head.h" #include "sp.h" -time_t mysql_db_table_last_check= 0L; - -TABLE_FIELD_W_TYPE mysql_db_table_fields[MYSQL_DB_FIELD_COUNT] = { +static const +TABLE_FIELD_TYPE mysql_db_table_fields[MYSQL_DB_FIELD_COUNT] = { { { C_STRING_WITH_LEN("Host") }, { C_STRING_WITH_LEN("char(60)") }, @@ -146,6 +145,8 @@ TABLE_FIELD_W_TYPE mysql_db_table_fields } }; +const TABLE_FIELD_DEF + mysql_db_table_def= {MYSQL_DB_FIELD_COUNT, mysql_db_table_fields}; #ifndef NO_EMBEDDED_ACCESS_CHECKS @@ -2405,7 +2406,12 @@ GRANT_TABLE::GRANT_TABLE(TABLE *form, TA privs = cols = 0; /* purecov: deadcode */ return; /* purecov: deadcode */ } - my_hash_insert(&hash_columns, (uchar *) mem_check); + if (my_hash_insert(&hash_columns, (uchar *) mem_check)) + { + /* Invalidate this entry */ + privs= cols= 0; + return; + } } while (!col_privs->file->index_next(col_privs->record[0]) && !key_cmp_if_same(col_privs,key,0,key_prefix_len)); col_privs->file->ha_index_end(); @@ -2439,14 +2445,17 @@ static GRANT_NAME *name_hash_search(HASH const char *host,const char* ip, const char *db, const char *user, const char *tname, - bool exact) + bool exact, bool name_tolower) { - char helping [NAME_LEN*2+USERNAME_LENGTH+3]; + char helping [NAME_LEN*2+USERNAME_LENGTH+3], *name_ptr; uint len; GRANT_NAME *grant_name,*found=0; HASH_SEARCH_STATE state; - len = (uint) (strmov(strmov(strmov(helping,user)+1,db)+1,tname)-helping)+ 1; + name_ptr= strmov(strmov(helping, user) + 1, db) + 1; + len = (uint) (strmov(name_ptr, tname) - helping) + 1; + if (name_tolower) + my_casedn_str(files_charset_info, name_ptr); for (grant_name= (GRANT_NAME*) hash_first(name_hash, (uchar*) helping, len, &state); grant_name ; @@ -2479,7 +2488,7 @@ routine_hash_search(const char *host, co { return (GRANT_TABLE*) name_hash_search(proc ? &proc_priv_hash : &func_priv_hash, - host, ip, db, user, tname, exact); + host, ip, db, user, tname, exact, TRUE); } @@ -2488,7 +2497,7 @@ table_hash_search(const char *host, cons const char *user, const char *tname, bool exact) { return (GRANT_TABLE*) name_hash_search(&column_priv_hash, host, ip, db, - user, tname, exact); + user, tname, exact, FALSE); } @@ -2610,7 +2619,11 @@ static int replace_column_table(GRANT_TA goto end; /* purecov: inspected */ } grant_column= new GRANT_COLUMN(column->column,privileges); - my_hash_insert(&g_t->hash_columns,(uchar*) grant_column); + if (my_hash_insert(&g_t->hash_columns,(uchar*) grant_column)) + { + result= -1; + goto end; + } } } @@ -3135,12 +3148,12 @@ int mysql_table_grant(THD *thd, TABLE_LI Str->user.str, table_name, rights, column_priv); - if (!grant_table) // end of memory + if (!grant_table || + my_hash_insert(&column_priv_hash,(uchar*) grant_table)) { result= TRUE; /* purecov: deadcode */ continue; /* purecov: deadcode */ } - my_hash_insert(&column_priv_hash,(uchar*) grant_table); } /* If revoke_grant, calculate the new column privilege for tables_priv */ @@ -3344,12 +3357,13 @@ bool mysql_routine_grant(THD *thd, TABLE grant_name= new GRANT_NAME(Str->host.str, db_name, Str->user.str, table_name, rights, TRUE); - if (!grant_name) + if (!grant_name || + my_hash_insert(is_proc ? + &proc_priv_hash : &func_priv_hash,(uchar*) grant_name)) { result= TRUE; continue; } - my_hash_insert(is_proc ? &proc_priv_hash : &func_priv_hash,(uchar*) grant_name); } if (replace_routine_table(thd, grant_name, tables[1].table, *Str, @@ -3452,6 +3466,13 @@ bool mysql_grant(THD *thd, const char *d result= TRUE; continue; } + /* + No User, but a password? + They did GRANT ... TO CURRENT_USER() IDENTIFIED BY ... ! + Get the current user, and shallow-copy the new password to them! + */ + if (!tmp_Str->user.str && tmp_Str->password.str) + Str->password= tmp_Str->password; if (replace_user_table(thd, tables[0].table, *Str, (!db ? rights : 0), revoke_grant, create_new_users, test(thd->variables.sql_mode & === modified file 'sql/sql_acl.h' --- a/sql/sql_acl.h 2009-05-29 13:37:54 +0000 +++ b/sql/sql_acl.h 2009-11-21 11:18:21 +0000 @@ -159,8 +159,7 @@ enum mysql_db_table_field MYSQL_DB_FIELD_COUNT }; -extern TABLE_FIELD_W_TYPE mysql_db_table_fields[]; -extern time_t mysql_db_table_last_check; +extern const TABLE_FIELD_DEF mysql_db_table_def; /* Classes */ === modified file 'sql/sql_base.cc' --- a/sql/sql_base.cc 2009-12-04 15:12:22 +0000 +++ b/sql/sql_base.cc 2010-01-15 15:27:55 +0000 @@ -2938,7 +2938,12 @@ TABLE *open_table(THD *thd, TABLE_LIST * DBUG_PRINT("info", ("inserting table '%s'.'%s' 0x%lx into the cache", table->s->db.str, table->s->table_name.str, (long) table)); - VOID(my_hash_insert(&open_cache,(uchar*) table)); + if (my_hash_insert(&open_cache,(uchar*) table)) + { + my_free(table, MYF(0)); + VOID(pthread_mutex_unlock(&LOCK_open)); + DBUG_RETURN(NULL); + } } check_unused(); // Debugging call === modified file 'sql/sql_cache.cc' --- a/sql/sql_cache.cc 2009-12-03 11:19:05 +0000 +++ b/sql/sql_cache.cc 2010-01-29 10:42:31 +0000 @@ -377,9 +377,11 @@ TODO list: static void debug_wait_for_kill(const char *info) { - DBUG_ENTER("debug_wait_for_kill"); const char *prev_info; THD *thd; + char buff[1024]; + DBUG_ENTER("debug_wait_for_kill"); + thd= current_thd; prev_info= thd->proc_info; thd->proc_info= info; @@ -387,8 +389,16 @@ static void debug_wait_for_kill(const ch while(!thd->killed) my_sleep(1000); thd->killed= THD::NOT_KILLED; + /* + Remove the set debug variable, to ensure we don't get stuck on it again + This is needed as for MyISAM, invalidate_table() may be called twice + (Once from mysql_delete() and once from mi_update_status()) + */ + sprintf(buff, "-d,%s", info); + DBUG_SET(buff); sql_print_information("Exit debug_wait_for_kill"); thd->proc_info= prev_info; + DBUG_VOID_RETURN; } @@ -421,12 +431,16 @@ TYPELIB query_cache_type_typelib= effect by another thread. This enables a quick path in execution to skip waits when the outcome is known. + @param use_timeout TRUE if the lock can abort because of a timeout. + + @note use_timeout is optional and default value is FALSE. + @return @retval FALSE An exclusive lock was taken @retval TRUE The locking attempt failed */ -bool Query_cache::try_lock(void) +bool Query_cache::try_lock(bool use_timeout) { bool interrupt= FALSE; DBUG_ENTER("Query_cache::try_lock"); @@ -456,7 +470,26 @@ bool Query_cache::try_lock(void) else { DBUG_ASSERT(m_cache_lock_status == Query_cache::LOCKED); - pthread_cond_wait(&COND_cache_status_changed, &structure_guard_mutex); + /* + To prevent send_result_to_client() and query_cache_insert() from + blocking execution for too long a timeout is put on the lock. + */ + if (use_timeout) + { + struct timespec waittime; + set_timespec_nsec(waittime,(ulong)(50000000L)); /* Wait for 50 msec */ + int res= pthread_cond_timedwait(&COND_cache_status_changed, + &structure_guard_mutex,&waittime); + if (res == ETIMEDOUT) + { + interrupt= TRUE; + break; + } + } + else + { + pthread_cond_wait(&COND_cache_status_changed, &structure_guard_mutex); + } } } pthread_mutex_unlock(&structure_guard_mutex); @@ -891,15 +924,18 @@ void query_cache_insert(NET *net, const void query_cache_abort(NET *net) { + THD *thd; DBUG_ENTER("query_cache_abort"); - THD *thd= current_thd; /* See the comment on double-check locking usage above. */ if (net->query_cache_query == 0) DBUG_VOID_RETURN; if (query_cache.try_lock()) + { + net->query_cache_query = 0; DBUG_VOID_RETURN; + } /* While we were waiting another thread might have changed the status @@ -909,6 +945,7 @@ void query_cache_abort(NET *net) net->query_cache_query); if (query_block) { + thd= current_thd; thd_proc_info(thd, "storing result in query cache"); DUMP(&query_cache); BLOCK_LOCK_WR(query_block); @@ -918,6 +955,7 @@ void query_cache_abort(NET *net) DBUG_EXECUTE("check_querycache",query_cache.check_integrity(1);); } + DBUG_ASSERT(!net->query_cache_query); query_cache.unlock(); DBUG_VOID_RETURN; } @@ -947,8 +985,12 @@ void query_cache_end_of_result(THD *thd) #endif if (query_cache.try_lock()) + { + thd->net.query_cache_query= 0; DBUG_VOID_RETURN; + } + /* thd->net.query_cache_query may have changed during resize */ query_block= ((Query_cache_block*) thd->net.query_cache_query); if (query_block) { @@ -974,8 +1016,8 @@ void query_cache_end_of_result(THD *thd) to this function. In the release version that query should be ignored and removed from QC. */ - DBUG_ASSERT(0); query_cache.free_query(query_block); + thd->net.query_cache_query= 0; query_cache.unlock(); DBUG_VOID_RETURN; } @@ -1190,8 +1232,14 @@ def_week_frmt: %lu, in_trans: %d, autoco A table- or a full flush operation can potentially take a long time to finish. We choose not to wait for them and skip caching statements instead. + + In case the wait time can't be determined there is an upper limit which + causes try_lock() to abort with a time out. + + The 'TRUE' parameter indicate that the lock is allowed to timeout + */ - if (try_lock()) + if (try_lock(TRUE)) DBUG_VOID_RETURN; if (query_cache_size == 0) { @@ -1306,8 +1354,8 @@ end: to the user. RESULTS - 1 Query was not cached. - 0 The query was cached and user was sent the result. + 0 Query was not cached. + 1 The query was cached and user was sent the result. -1 The query was cached but we didn't have rights to use it. No error is sent to the client yet. @@ -1388,8 +1436,10 @@ Query_cache::send_result_to_client(THD * Try to obtain an exclusive lock on the query cache. If the cache is disabled or if a full cache flush is in progress, the attempt to get the lock is aborted. + + The 'TRUE' parameter indicate that the lock is allowed to timeout */ - if (try_lock()) + if (try_lock(TRUE)) goto err; if (query_cache_size == 0) === modified file 'sql/sql_cache.h' --- a/sql/sql_cache.h 2009-06-16 08:34:47 +0000 +++ b/sql/sql_cache.h 2009-11-20 12:49:06 +0000 @@ -485,7 +485,7 @@ protected: const char *name); my_bool in_blocks(Query_cache_block * point); - bool try_lock(void); + bool try_lock(bool use_timeout= FALSE); void lock(void); void lock_and_suspend(void); void unlock(void); === modified file 'sql/sql_class.cc' --- a/sql/sql_class.cc 2009-12-04 15:12:22 +0000 +++ b/sql/sql_class.cc 2010-01-15 15:27:55 +0000 @@ -379,6 +379,8 @@ char *thd_security_context(THD *thd, cha str.append(proc_info); } + pthread_mutex_lock(&thd->LOCK_thd_data); + if (thd->query()) { if (max_query_len < 1) @@ -388,6 +390,9 @@ char *thd_security_context(THD *thd, cha str.append('\n'); str.append(thd->query(), len); } + + pthread_mutex_unlock(&thd->LOCK_thd_data); + if (str.c_ptr_safe() == buffer) return buffer; === modified file 'sql/sql_connect.cc' --- a/sql/sql_connect.cc 2009-12-03 11:19:05 +0000 +++ b/sql/sql_connect.cc 2010-01-29 10:42:31 +0000 @@ -798,6 +798,7 @@ static int check_connection(THD *thd) DBUG_PRINT("info", ("client capabilities: %lu", thd->client_capabilities)); if (thd->client_capabilities & CLIENT_SSL) { + char error_string[1024]; /* Do the SSL layering. */ if (!ssl_acceptor_fd) { @@ -806,7 +807,7 @@ static int check_connection(THD *thd) return 1; } DBUG_PRINT("info", ("IO layer change in progress...")); - if (sslaccept(ssl_acceptor_fd, net->vio, net->read_timeout)) + if (sslaccept(ssl_acceptor_fd, net->vio, net->read_timeout, error_string)) { DBUG_PRINT("error", ("Failed to accept new SSL connection")); inc_host_errors(&thd->remote.sin_addr); === modified file 'sql/sql_delete.cc' --- a/sql/sql_delete.cc 2009-12-03 11:34:11 +0000 +++ b/sql/sql_delete.cc 2010-01-15 15:27:55 +0000 @@ -426,7 +426,8 @@ cleanup: } DBUG_ASSERT(transactional_table || !deleted || thd->transaction.stmt.modified_non_trans_table); free_underlaid_joins(thd, select_lex); - if (error < 0 || (thd->lex->ignore && !thd->is_fatal_error)) + if (error < 0 || + (thd->lex->ignore && !thd->is_error() && !thd->is_fatal_error)) { /* If a TRUNCATE TABLE was issued, the number of rows should be reported as @@ -1089,6 +1090,7 @@ bool mysql_truncate(THD *thd, TABLE_LIST TABLE *table; bool error; uint path_length; + bool is_temporary_table= false; DBUG_ENTER("mysql_truncate"); bzero((char*) &create_info,sizeof(create_info)); @@ -1101,6 +1103,8 @@ bool mysql_truncate(THD *thd, TABLE_LIST { TABLE_SHARE *share= table->s; handlerton *table_type= share->db_type(); + is_temporary_table= true; + if (!ha_check_storage_engine_flag(table_type, HTON_CAN_RECREATE)) goto trunc_by_del; @@ -1166,11 +1170,9 @@ end: { if (!error) { - /* - TRUNCATE must always be statement-based binlogged (not row-based) so - we don't test current_stmt_binlog_row_based. - */ - write_bin_log(thd, TRUE, thd->query(), thd->query_length()); + /* In RBR, the statement is not binlogged if the table is temporary. */ + if (!is_temporary_table || !thd->current_stmt_binlog_row_based) + write_bin_log(thd, TRUE, thd->query(), thd->query_length()); my_ok(thd); // This should return record count } VOID(pthread_mutex_lock(&LOCK_open)); === modified file 'sql/sql_insert.cc' --- a/sql/sql_insert.cc 2009-12-04 15:12:22 +0000 +++ b/sql/sql_insert.cc 2010-01-15 15:27:55 +0000 @@ -521,6 +521,22 @@ bool open_and_lock_for_insert_delayed(TH DBUG_ENTER("open_and_lock_for_insert_delayed"); #ifndef EMBEDDED_LIBRARY + if (thd->locked_tables && thd->global_read_lock) + { + /* + If this connection has the global read lock, the handler thread + will not be able to lock the table. It will wait for the global + read lock to go away, but this will never happen since the + connection thread will be stuck waiting for the handler thread + to open and lock the table. + If we are not in locked tables mode, INSERT will seek protection + against the global read lock (and fail), thus we will only get + to this point in locked tables mode. + */ + my_error(ER_CANT_UPDATE_WITH_READLOCK, MYF(0)); + DBUG_RETURN(TRUE); + } + if (delayed_get_table(thd, table_list)) DBUG_RETURN(TRUE); === modified file 'sql/sql_load.cc' --- a/sql/sql_load.cc 2009-12-03 11:19:05 +0000 +++ b/sql/sql_load.cc 2010-01-15 15:27:55 +0000 @@ -304,7 +304,8 @@ int mysql_load(THD *thd,sql_exchange *ex else { (void) fn_format(name, ex->file_name, mysql_real_data_home, "", - MY_RELATIVE_PATH | MY_UNPACK_FILENAME); + MY_RELATIVE_PATH | MY_UNPACK_FILENAME | + MY_RETURN_REAL_PATH); #if !defined(__WIN__) && ! defined(__NETWARE__) MY_STAT stat_info; if (!my_stat(name,&stat_info,MYF(MY_WME))) @@ -347,12 +348,16 @@ int mysql_load(THD *thd,sql_exchange *ex DBUG_ASSERT(FALSE); #endif } - else if (opt_secure_file_priv && - strncmp(opt_secure_file_priv, name, strlen(opt_secure_file_priv))) + else if (opt_secure_file_priv) { - /* Read only allowed from within dir specified by secure_file_priv */ - my_error(ER_OPTION_PREVENTS_STATEMENT, MYF(0), "--secure-file-priv"); - DBUG_RETURN(TRUE); + char secure_file_real_path[FN_REFLEN]; + (void) my_realpath(secure_file_real_path, opt_secure_file_priv, 0); + if (strncmp(secure_file_real_path, name, strlen(secure_file_real_path))) + { + /* Read only allowed from within dir specified by secure_file_priv */ + my_error(ER_OPTION_PREVENTS_STATEMENT, MYF(0), "--secure-file-priv"); + DBUG_RETURN(TRUE); + } } } === modified file 'sql/sql_parse.cc' --- a/sql/sql_parse.cc 2009-12-03 11:34:11 +0000 +++ b/sql/sql_parse.cc 2010-01-29 10:42:31 +0000 @@ -1252,12 +1252,14 @@ bool dispatch_command(enum enum_server_c ha_maria::implicit_commit(thd, FALSE); #endif - net_end_statement(thd); - query_cache_end_of_result(thd); /* Multiple queries exits, execute them individually */ close_thread_tables(thd); + + net_end_statement(thd); + query_cache_end_of_result(thd); + ulong length= (ulong)(packet_end - beginning_of_next_stmt); log_slow_statement(thd); @@ -1645,13 +1647,22 @@ bool dispatch_command(enum enum_server_c ha_maria::implicit_commit(thd, FALSE); #endif - net_end_statement(thd); - query_cache_end_of_result(thd); - + if (!(sql_command_flags[thd->lex->sql_command] & CF_CHANGES_DATA)) + { + /* No changes in data; We can send ok at once to the client */ + net_end_statement(thd); + query_cache_end_of_result(thd); + } thd->proc_info= "closing tables"; /* Free tables */ close_thread_tables(thd); + if (sql_command_flags[thd->lex->sql_command] & CF_CHANGES_DATA) + { + net_end_statement(thd); + query_cache_end_of_result(thd); + } + log_slow_statement(thd); thd_proc_info(thd, "cleaning up"); @@ -7632,6 +7643,9 @@ void get_default_definer(THD *thd, LEX_U definer->host.str= (char *) sctx->priv_host; definer->host.length= strlen(definer->host.str); + + definer->password.str= NULL; + definer->password.length= 0; } @@ -7683,6 +7697,8 @@ LEX_USER *create_definer(THD *thd, LEX_S definer->user= *user_name; definer->host= *host_name; + definer->password.str= NULL; + definer->password.length= 0; return definer; } === modified file 'sql/sql_partition.cc' --- a/sql/sql_partition.cc 2009-12-03 11:19:05 +0000 +++ b/sql/sql_partition.cc 2010-01-15 15:27:55 +0000 @@ -196,26 +196,27 @@ bool partition_default_handling(TABLE *t { DBUG_ENTER("partition_default_handling"); - if (part_info->use_default_no_partitions) + if (!is_create_table_ind) { - if (!is_create_table_ind && - table->file->get_no_parts(normalized_path, &part_info->no_parts)) + if (part_info->use_default_no_partitions) { - DBUG_RETURN(TRUE); + if (table->file->get_no_parts(normalized_path, &part_info->no_parts)) + { + DBUG_RETURN(TRUE); + } } - } - else if (part_info->is_sub_partitioned() && - part_info->use_default_no_subpartitions) - { - uint no_parts; - if (!is_create_table_ind && - (table->file->get_no_parts(normalized_path, &no_parts))) + else if (part_info->is_sub_partitioned() && + part_info->use_default_no_subpartitions) { - DBUG_RETURN(TRUE); + uint no_parts; + if (table->file->get_no_parts(normalized_path, &no_parts)) + { + DBUG_RETURN(TRUE); + } + DBUG_ASSERT(part_info->no_parts > 0); + DBUG_ASSERT((no_parts % part_info->no_parts) == 0); + part_info->no_subparts= no_parts / part_info->no_parts; } - DBUG_ASSERT(part_info->no_parts > 0); - part_info->no_subparts= no_parts / part_info->no_parts; - DBUG_ASSERT((no_parts % part_info->no_parts) == 0); } part_info->set_up_defaults_for_partitioning(table->file, (ulonglong)0, (uint)0); @@ -905,6 +906,8 @@ bool fix_fields_part_func(THD *thd, Item char* db_name; char db_name_string[FN_REFLEN]; bool save_use_only_table_context; + uint8 saved_full_group_by_flag; + nesting_map saved_allow_sum_func; DBUG_ENTER("fix_fields_part_func"); if (part_info->fixed) @@ -974,9 +977,19 @@ bool fix_fields_part_func(THD *thd, Item save_use_only_table_context= thd->lex->use_only_table_context; thd->lex->use_only_table_context= TRUE; thd->lex->current_select->cur_pos_in_select_list= UNDEF_POS; + saved_full_group_by_flag= thd->lex->current_select->full_group_by_flag; + saved_allow_sum_func= thd->lex->allow_sum_func; + thd->lex->allow_sum_func= 0; error= func_expr->fix_fields(thd, (Item**)&func_expr); + /* + Restore full_group_by_flag and allow_sum_func, + fix_fields should not affect mysql_select later, see Bug#46923. + */ + thd->lex->current_select->full_group_by_flag= saved_full_group_by_flag; + thd->lex->allow_sum_func= saved_allow_sum_func; + thd->lex->use_only_table_context= save_use_only_table_context; context->table_list= save_table_list; @@ -1679,7 +1692,7 @@ bool fix_partition_func(THD *thd, TABLE if (((part_info->part_type != HASH_PARTITION || part_info->list_of_part_fields == FALSE) && check_part_func_fields(part_info->part_field_array, TRUE)) || - (part_info->list_of_part_fields == FALSE && + (part_info->list_of_subpart_fields == FALSE && part_info->is_sub_partitioned() && check_part_func_fields(part_info->subpart_field_array, TRUE))) { === modified file 'sql/sql_profile.cc' --- a/sql/sql_profile.cc 2009-10-15 21:38:29 +0000 +++ b/sql/sql_profile.cc 2010-01-29 18:42:22 +0000 @@ -131,6 +131,23 @@ int make_profile_table_for_show(THD *thd #define RUSAGE_USEC(tv) ((tv).tv_sec*1000*1000 + (tv).tv_usec) #define RUSAGE_DIFF_USEC(tv1, tv2) (RUSAGE_USEC((tv1))-RUSAGE_USEC((tv2))) +#ifdef __WIN__ +inline ULONGLONG FileTimeToQuadWord(FILETIME *ft) +{ + ULONGLONG nrv = 0; + nrv |= ft->dwHighDateTime; + nrv <<= 32; + nrv |= ft->dwLowDateTime; + return nrv; +} + + +// Get time difference between to FILETIME objects in seconds. +inline double GetTimeDiffInSeconds(FILETIME *a, FILETIME *b) +{ + return ((FileTimeToQuadWord(a) - FileTimeToQuadWord(b)) / 1e7); +} +#endif /* __WIN__ */ PROF_MEASUREMENT::PROF_MEASUREMENT(QUERY_PROFILE *profile_arg, const char *status_arg) @@ -221,6 +238,11 @@ void PROF_MEASUREMENT::collect() time_usecs= (double) my_getsystime() / 10.0; /* 1 sec was 1e7, now is 1e6 */ #ifdef HAVE_GETRUSAGE getrusage(RUSAGE_SELF, &rusage); +#elif defined(__WIN__) + FILETIME ftDummy; + GetProcessTimes(GetCurrentProcess(), &ftDummy, &ftDummy, &ftKernel, &ftUser); + GetProcessIoCounters(GetCurrentProcess(), &io_count); + GetProcessMemoryInfo(GetCurrentProcess(), &mem_count, sizeof(mem_count)); #endif } @@ -590,6 +612,23 @@ int PROFILING::fill_statistics_info(THD table->field[5]->store_decimal(&cpu_stime_decimal); table->field[4]->set_notnull(); table->field[5]->set_notnull(); +#elif defined(__WIN__) + my_decimal cpu_utime_decimal, cpu_stime_decimal; + + double2my_decimal(E_DEC_FATAL_ERROR, + GetTimeDiffInSeconds(&entry->ftUser, + &previous->ftUser), + &cpu_utime_decimal); + double2my_decimal(E_DEC_FATAL_ERROR, + GetTimeDiffInSeconds(&entry->ftKernel, + &previous->ftKernel), + &cpu_stime_decimal); + + // Store the result. + table->field[4]->store_decimal(&cpu_utime_decimal); + table->field[5]->store_decimal(&cpu_stime_decimal); + table->field[4]->set_notnull(); + table->field[5]->set_notnull(); #else /* TODO: Add CPU-usage info for non-BSD systems */ #endif @@ -612,6 +651,17 @@ int PROFILING::fill_statistics_info(THD table->field[9]->store((uint32)(entry->rusage.ru_oublock - previous->rusage.ru_oublock)); table->field[9]->set_notnull(); +#elif defined(__WIN__) + ULONGLONG reads_delta = entry->io_count.ReadOperationCount - + previous->io_count.ReadOperationCount; + ULONGLONG writes_delta = entry->io_count.WriteOperationCount - + previous->io_count.WriteOperationCount; + + table->field[8]->store((uint32)reads_delta); + table->field[8]->set_notnull(); + + table->field[9]->store((uint32)writes_delta); + table->field[9]->set_notnull(); #else /* TODO: Add block IO info for non-BSD systems */ #endif @@ -634,6 +684,13 @@ int PROFILING::fill_statistics_info(THD table->field[13]->store((uint32)(entry->rusage.ru_minflt - previous->rusage.ru_minflt), true); table->field[13]->set_notnull(); +#elif defined(__WIN__) + /* Windows APIs don't easily distinguish between hard and soft page + faults, so we just fill the 'major' column and leave the second NULL. + */ + table->field[12]->store((uint32)(entry->mem_count.PageFaultCount - + previous->mem_count.PageFaultCount), true); + table->field[12]->set_notnull(); #else /* TODO: Add page fault info for non-BSD systems */ #endif === modified file 'sql/sql_profile.h' --- a/sql/sql_profile.h 2009-09-07 20:50:10 +0000 +++ b/sql/sql_profile.h 2010-01-29 18:42:22 +0000 @@ -36,6 +36,10 @@ int make_profile_table_for_show(THD *thd #if defined(ENABLED_PROFILING) && defined(COMMUNITY_SERVER) #include "mysql_priv.h" +#ifdef __WIN__ +#include <psapi.h> +#endif + #ifdef HAVE_SYS_RESOURCE_H #include <sys/resource.h> #endif @@ -165,6 +169,10 @@ private: char *status; #ifdef HAVE_GETRUSAGE struct rusage rusage; +#elif defined(__WIN__) + FILETIME ftKernel, ftUser; + IO_COUNTERS io_count; + PROCESS_MEMORY_COUNTERS mem_count; #endif char *function; === modified file 'sql/sql_select.cc' --- a/sql/sql_select.cc 2009-12-03 11:34:11 +0000 +++ b/sql/sql_select.cc 2010-01-15 15:27:55 +0000 @@ -992,14 +992,20 @@ JOIN::optimize() DBUG_RETURN(1); } - if (select_lex->olap == ROLLUP_TYPE && rollup_process_const_fields()) + if (rollup.state != ROLLUP::STATE_NONE) { - DBUG_PRINT("error", ("Error: rollup_process_fields() failed")); - DBUG_RETURN(1); + if (rollup_process_const_fields()) + { + DBUG_PRINT("error", ("Error: rollup_process_fields() failed")); + DBUG_RETURN(1); + } + } + else + { + /* Remove distinct if only const tables */ + select_distinct= select_distinct && (const_tables != tables); } - /* Remove distinct if only const tables */ - select_distinct= select_distinct && (const_tables != tables); thd_proc_info(thd, "preparing"); if (result->initialize_tables(this)) { @@ -1298,11 +1304,14 @@ JOIN::optimize() - We are using an ORDER BY or GROUP BY on fields not in the first table - We are using different ORDER BY and GROUP BY orders - The user wants us to buffer the result. + When the WITH ROLLUP modifier is present, we cannot skip temporary table + creation for the DISTINCT clause just because there are only const tables. */ - need_tmp= (const_tables != tables && + need_tmp= ((const_tables != tables && ((select_distinct || !simple_order || !simple_group) || (group_list && order) || - test(select_options & OPTION_BUFFER_RESULT))); + test(select_options & OPTION_BUFFER_RESULT))) || + (rollup.state != ROLLUP::STATE_NONE && select_distinct)); // No cache for MATCH make_join_readinfo(this, @@ -2144,17 +2153,13 @@ JOIN::exec() DBUG_VOID_RETURN; if (!curr_table->select->cond) curr_table->select->cond= sort_table_cond; - else // This should never happen + else { if (!(curr_table->select->cond= new Item_cond_and(curr_table->select->cond, sort_table_cond))) DBUG_VOID_RETURN; - /* - Item_cond_and do not need fix_fields for execution, its parameters - are fixed or do not need fix_fields, too - */ - curr_table->select->cond->quick_fix_field(); + curr_table->select->cond->fix_fields(thd, 0); } curr_table->select_cond= curr_table->select->cond; curr_table->select_cond->top_level_item(); @@ -6565,6 +6570,56 @@ void rr_unlock_row(st_join_table *tab) +/** + Pick the appropriate access method functions + + Sets the functions for the selected table access method + + @param tab Table reference to put access method +*/ + +static void +pick_table_access_method(JOIN_TAB *tab) +{ + switch (tab->type) + { + case JT_REF: + tab->read_first_record= join_read_always_key; + tab->read_record.read_record= join_read_next_same; + break; + + case JT_REF_OR_NULL: + tab->read_first_record= join_read_always_key_or_null; + tab->read_record.read_record= join_read_next_same_or_null; + break; + + case JT_CONST: + tab->read_first_record= join_read_const; + tab->read_record.read_record= join_no_more_records; + break; + + case JT_EQ_REF: + tab->read_first_record= join_read_key; + tab->read_record.read_record= join_no_more_records; + break; + + case JT_FT: + tab->read_first_record= join_ft_read_first; + tab->read_record.read_record= join_ft_read_next; + break; + + case JT_SYSTEM: + tab->read_first_record= join_read_system; + tab->read_record.read_record= join_no_more_records; + break; + + /* keep gcc happy */ + default: + break; + } +} + + static void make_join_readinfo(JOIN *join, ulonglong options) { @@ -6599,45 +6654,15 @@ make_join_readinfo(JOIN *join, ulonglong tab->sorted= sorted; sorted= 0; // only first must be sorted + table->status=STATUS_NO_RECORD; + pick_table_access_method (tab); + switch (tab->type) { - case JT_SYSTEM: // Only happens with left join - table->status=STATUS_NO_RECORD; - tab->read_first_record= join_read_system; - tab->read_record.read_record= join_no_more_records; - break; - case JT_CONST: // Only happens with left join - table->status=STATUS_NO_RECORD; - tab->read_first_record= join_read_const; - tab->read_record.read_record= join_no_more_records; - if (table->covering_keys.is_set(tab->ref.key) && - !table->no_keyread) - { - table->key_read=1; - table->file->extra(HA_EXTRA_KEYREAD); - } - break; case JT_EQ_REF: - table->status=STATUS_NO_RECORD; - if (tab->select) - { - delete tab->select->quick; - tab->select->quick=0; - } - delete tab->quick; - tab->quick=0; - tab->read_first_record= join_read_key; tab->read_record.unlock_row= join_read_key_unlock_row; - tab->read_record.read_record= join_no_more_records; - if (table->covering_keys.is_set(tab->ref.key) && - !table->no_keyread) - { - table->key_read=1; - table->file->extra(HA_EXTRA_KEYREAD); - } - break; + /* fall through */ case JT_REF_OR_NULL: case JT_REF: - table->status=STATUS_NO_RECORD; if (tab->select) { delete tab->select->quick; @@ -6645,34 +6670,20 @@ make_join_readinfo(JOIN *join, ulonglong } delete tab->quick; tab->quick=0; + /* fall through */ + case JT_CONST: // Only happens with left join if (table->covering_keys.is_set(tab->ref.key) && !table->no_keyread) { table->key_read=1; table->file->extra(HA_EXTRA_KEYREAD); } - if (tab->type == JT_REF) - { - tab->read_first_record= join_read_always_key; - tab->read_record.read_record= join_read_next_same; - } - else - { - tab->read_first_record= join_read_always_key_or_null; - tab->read_record.read_record= join_read_next_same_or_null; - } - break; - case JT_FT: - table->status=STATUS_NO_RECORD; - tab->read_first_record= join_ft_read_first; - tab->read_record.read_record= join_ft_read_next; break; case JT_ALL: /* If previous table use cache If the incoming data set is already sorted don't use cache. */ - table->status=STATUS_NO_RECORD; if (i != join->const_tables && !(options & SELECT_NO_JOIN_CACHE) && tab->use_quick != 2 && !tab->first_inner && !ordered_set) { @@ -6758,6 +6769,9 @@ make_join_readinfo(JOIN *join, ulonglong } } break; + case JT_FT: + case JT_SYSTEM: + break; default: DBUG_PRINT("error",("Table type %d found",tab->type)); /* purecov: deadcode */ break; /* purecov: deadcode */ @@ -7909,12 +7923,12 @@ static COND *build_equal_items_for_cond( { item_equal->fix_length_and_dec(); item_equal->update_used_tables(); + set_if_bigger(thd->lex->current_select->max_equal_elems, + item_equal->members()); + return item_equal; } - else - item_equal= (Item_equal *) eq_list.pop(); - set_if_bigger(thd->lex->current_select->max_equal_elems, - item_equal->members()); - return item_equal; + + return eq_list.pop(); } else { @@ -9552,47 +9566,8 @@ static Field *create_tmp_field_from_item new_field->set_derivation(item->collation.derivation); break; case DECIMAL_RESULT: - { - uint8 dec= item->decimals; - uint8 intg= ((Item_decimal *) item)->decimal_precision() - dec; - uint32 len= item->max_length; - - /* - Trying to put too many digits overall in a DECIMAL(prec,dec) - will always throw a warning. We must limit dec to - DECIMAL_MAX_SCALE however to prevent an assert() later. - */ - - if (dec > 0) - { - signed int overflow; - - dec= min(dec, DECIMAL_MAX_SCALE); - - /* - If the value still overflows the field with the corrected dec, - we'll throw out decimals rather than integers. This is still - bad and of course throws a truncation warning. - +1: for decimal point - */ - - const int required_length= - my_decimal_precision_to_length(intg + dec, dec, - item->unsigned_flag); - - overflow= required_length - len; - - if (overflow > 0) - dec= max(0, dec - overflow); // too long, discard fract - else - /* Corrected value fits. */ - len= required_length; - } - - new_field= new Field_new_decimal(len, maybe_null, item->name, - dec, item->unsigned_flag); + new_field= Field_new_decimal::create_from_item(item); break; - } case ROW_RESULT: default: // This case should never be choosen @@ -13496,6 +13471,8 @@ test_if_skip_sort_order(JOIN_TAB *tab,OR if (create_ref_for_key(tab->join, tab, keyuse, tab->join->const_table_map)) DBUG_RETURN(0); + + pick_table_access_method(tab); } else { @@ -14305,7 +14282,10 @@ static int remove_dup_with_hash_index(TH goto err; } else - (void) my_hash_insert(&hash, org_key_pos); + { + if (my_hash_insert(&hash, org_key_pos)) + goto err; + } key_pos+=extra_length; } my_free((char*) key_buffer,MYF(0)); === modified file 'sql/sql_show.cc' --- a/sql/sql_show.cc 2009-12-03 11:34:11 +0000 +++ b/sql/sql_show.cc 2010-01-15 15:27:55 +0000 @@ -721,7 +721,7 @@ mysqld_show_create(THD *thd, TABLE_LIST thd->push_internal_handler(&view_error_suppressor); bool error= open_normal_and_derived_tables(thd, table_list, 0); thd->pop_internal_handler(); - if (error && thd->main_da.is_error()) + if (error && (thd->killed || thd->main_da.is_error())) DBUG_RETURN(TRUE); } === modified file 'sql/sql_table.cc' --- a/sql/sql_table.cc 2009-12-03 11:19:05 +0000 +++ b/sql/sql_table.cc 2010-01-15 15:27:55 +0000 @@ -5428,12 +5428,20 @@ binlog: } VOID(pthread_mutex_unlock(&LOCK_open)); - IF_DBUG(int result=) - store_create_info(thd, table, &query, - create_info, FALSE /* show_database */); + /* + The condition avoids a crash as described in BUG#48506. Other + binlogging problems related to CREATE TABLE IF NOT EXISTS LIKE + when the existing object is a view will be solved by BUG 47442. + */ + if (!table->view) + { + IF_DBUG(int result=) + store_create_info(thd, table, &query, + create_info, FALSE /* show_database */); - DBUG_ASSERT(result == 0); // store_create_info() always return 0 - write_bin_log(thd, TRUE, query.ptr(), query.length()); + DBUG_ASSERT(result == 0); // store_create_info() always return 0 + write_bin_log(thd, TRUE, query.ptr(), query.length()); + } } else // Case 1 write_bin_log(thd, TRUE, thd->query(), thd->query_length()); === modified file 'sql/sql_yacc.yy' --- a/sql/sql_yacc.yy 2009-12-03 11:19:05 +0000 +++ b/sql/sql_yacc.yy 2010-01-17 17:22:46 +0000 @@ -389,6 +389,138 @@ void case_stmt_action_end_case(LEX *lex, lex->sphead->do_cont_backpatch(); } + +static bool +find_sys_var_null_base(THD *thd, struct sys_var_with_base *tmp) +{ + tmp->var= find_sys_var(thd, tmp->base_name.str, tmp->base_name.length); + + if (tmp->var == NULL) + my_error(ER_UNKNOWN_SYSTEM_VARIABLE, MYF(0), tmp->base_name.str); + else + tmp->base_name= null_lex_str; + + return thd->is_error(); +} + + +/** + Helper action for a SET statement. + Used to push a system variable into the assignment list. + + @param thd the current thread + @param tmp the system variable with base name + @param var_type the scope of the variable + @param val the value being assigned to the variable + + @return TRUE if error, FALSE otherwise. +*/ + +static bool +set_system_variable(THD *thd, struct sys_var_with_base *tmp, + enum enum_var_type var_type, Item *val) +{ + set_var *var; + LEX *lex= thd->lex; + + /* No AUTOCOMMIT from a stored function or trigger. */ + if (lex->spcont && tmp->var == &sys_autocommit) + lex->sphead->m_flags|= sp_head::HAS_SET_AUTOCOMMIT_STMT; + + if (! (var= new set_var(var_type, tmp->var, &tmp->base_name, val))) + return TRUE; + + return lex->var_list.push_back(var); +} + + +/** + Helper action for a SET statement. + Used to push a SP local variable into the assignment list. + + @param thd the current thread + @param var_type the SP local variable + @param val the value being assigned to the variable + + @return TRUE if error, FALSE otherwise. +*/ + +static bool +set_local_variable(THD *thd, sp_variable_t *spv, Item *val) +{ + Item *it; + LEX *lex= thd->lex; + sp_instr_set *sp_set; + + if (val) + it= val; + else if (spv->dflt) + it= spv->dflt; + else + { + it= new (thd->mem_root) Item_null(); + if (it == NULL) + return TRUE; + } + + sp_set= new sp_instr_set(lex->sphead->instructions(), lex->spcont, + spv->offset, it, spv->type, lex, TRUE); + + return (sp_set == NULL || lex->sphead->add_instr(sp_set)); +} + + +/** + Helper action for a SET statement. + Used to SET a field of NEW row. + + @param thd the current thread + @param name the field name + @param val the value being assigned to the row + + @return TRUE if error, FALSE otherwise. +*/ + +static bool +set_trigger_new_row(THD *thd, LEX_STRING *name, Item *val) +{ + LEX *lex= thd->lex; + Item_trigger_field *trg_fld; + sp_instr_set_trigger_field *sp_fld; + + /* QQ: Shouldn't this be field's default value ? */ + if (! val) + val= new Item_null(); + + DBUG_ASSERT(lex->trg_chistics.action_time == TRG_ACTION_BEFORE && + (lex->trg_chistics.event == TRG_EVENT_INSERT || + lex->trg_chistics.event == TRG_EVENT_UPDATE)); + + trg_fld= new (thd->mem_root) + Item_trigger_field(lex->current_context(), + Item_trigger_field::NEW_ROW, + name->str, UPDATE_ACL, FALSE); + + if (trg_fld == NULL) + return TRUE; + + sp_fld= new sp_instr_set_trigger_field(lex->sphead->instructions(), + lex->spcont, trg_fld, val, lex); + + if (sp_fld == NULL) + return TRUE; + + /* + Let us add this item to list of all Item_trigger_field + objects in trigger. + */ + lex->trg_table_fields.link_in_list((uchar *) trg_fld, + (uchar **) &trg_fld->next_trg_field); + + return lex->sphead->add_instr(sp_fld); +} + + /** Helper to resolve the SQL:2003 Syntax exception 1) in <in predicate>. See SQL:2003, Part 2, section 8.4 <in predicate>, Note 184, page 383. @@ -2335,8 +2467,8 @@ sp_decl: } pctx->declare_var_boundary(0); - lex->sphead->restore_lex(YYTHD); - + if (lex->sphead->restore_lex(YYTHD)) + MYSQL_YYABORT; $$.vars= $2; $$.conds= $$.hndlrs= $$.curs= 0; } @@ -2446,7 +2578,8 @@ sp_cursor_stmt: } lex->sp_lex_in_use= TRUE; $$= lex; - lex->sphead->restore_lex(YYTHD); + if (lex->sphead->restore_lex(YYTHD)) + MYSQL_YYABORT; } ; @@ -2665,7 +2798,8 @@ sp_proc_stmt_statement: sp->add_instr(i)) MYSQL_YYABORT; } - sp->restore_lex(thd); + if (sp->restore_lex(thd)) + MYSQL_YYABORT; } ; @@ -2693,7 +2827,8 @@ sp_proc_stmt_return: MYSQL_YYABORT; sp->m_flags|= sp_head::HAS_RETURN; } - sp->restore_lex(YYTHD); + if (sp->restore_lex(YYTHD)) + MYSQL_YYABORT; } ; @@ -2933,7 +3068,8 @@ sp_if: sp->add_cont_backpatch(i) || sp->add_instr(i)) MYSQL_YYABORT; - sp->restore_lex(YYTHD); + if (sp->restore_lex(YYTHD)) + MYSQL_YYABORT; } sp_proc_stmts1 { @@ -2979,7 +3115,9 @@ simple_case_stmt: if (case_stmt_action_expr(lex, $3)) MYSQL_YYABORT; - lex->sphead->restore_lex(YYTHD); /* For expr $3 */ + /* For expr $3 */ + if (lex->sphead->restore_lex(YYTHD)) + MYSQL_YYABORT; } simple_when_clause_list else_clause_opt @@ -3029,7 +3167,9 @@ simple_when_clause: LEX *lex= Lex; if (case_stmt_action_when(lex, $3, true)) MYSQL_YYABORT; - lex->sphead->restore_lex(YYTHD); /* For expr $3 */ + /* For expr $3 */ + if (lex->sphead->restore_lex(YYTHD)) + MYSQL_YYABORT; } THEN_SYM sp_proc_stmts1 @@ -3050,7 +3190,9 @@ searched_when_clause: LEX *lex= Lex; if (case_stmt_action_when(lex, $3, false)) MYSQL_YYABORT; - lex->sphead->restore_lex(YYTHD); /* For expr $3 */ + /* For expr $3 */ + if (lex->sphead->restore_lex(YYTHD)) + MYSQL_YYABORT; } THEN_SYM sp_proc_stmts1 @@ -3227,7 +3369,8 @@ sp_unlabeled_control: sp->new_cont_backpatch(i) || sp->add_instr(i)) MYSQL_YYABORT; - sp->restore_lex(YYTHD); + if (sp->restore_lex(YYTHD)) + MYSQL_YYABORT; } sp_proc_stmts1 END WHILE_SYM { @@ -3253,7 +3396,8 @@ sp_unlabeled_control: if (i == NULL || lex->sphead->add_instr(i)) MYSQL_YYABORT; - lex->sphead->restore_lex(YYTHD); + if (lex->sphead->restore_lex(YYTHD)) + MYSQL_YYABORT; /* We can shortcut the cont_backpatch here */ i->m_cont_dest= ip+1; } @@ -7539,6 +7683,14 @@ function_call_nonkeyword: } | SYSDATE optional_braces { + /* + Unlike other time-related functions, SYSDATE() is + replication-unsafe because it is not affected by the + TIMESTAMP variable. It is unsafe even if + sysdate_is_now=1, because the slave may have + sysdate_is_now=0. + */ + Lex->set_stmt_unsafe(); if (global_system_variables.sysdate_is_now == 0) $$= new (YYTHD->mem_root) Item_func_sysdate_local(); else @@ -9483,16 +9635,12 @@ replace: insert_lock_option: /* empty */ { -#ifdef HAVE_QUERY_CACHE /* - If it is SP we do not allow insert optimisation whan result of + If it is SP we do not allow insert optimisation when result of insert visible only after the table unlocking but everyone can read table. */ $$= (Lex->sphead ? TL_WRITE_DEFAULT : TL_WRITE_CONCURRENT_INSERT); -#else - $$= TL_WRITE_CONCURRENT_INSERT; -#endif } | LOW_PRIORITY { $$= TL_WRITE_LOW_PRIORITY; } | DELAYED_SYM { $$= TL_WRITE_DELAYED; } @@ -10515,15 +10663,11 @@ load_data_lock: /* empty */ { $$= TL_WRITE_DEFAULT; } | CONCURRENT { -#ifdef HAVE_QUERY_CACHE /* - Ignore this option in SP to avoid problem with query cache + Ignore this option in SP to avoid problem with query cache and + triggers with non default priority locks */ - if (Lex->sphead != 0) - $$= TL_WRITE_DEFAULT; - else -#endif - $$= TL_WRITE_CONCURRENT_INSERT; + $$= (Lex->sphead ? TL_WRITE_DEFAULT : TL_WRITE_CONCURRENT_INSERT); } | LOW_PRIORITY { $$= TL_WRITE_LOW_PRIORITY; } ; @@ -11791,7 +11935,8 @@ option_type_value: if (sp->add_instr(i)) MYSQL_YYABORT; } - lex->sphead->restore_lex(thd); + if (lex->sphead->restore_lex(thd)) + MYSQL_YYABORT; } } ; @@ -11831,98 +11976,42 @@ sys_option_value: option_type internal_variable_name equal set_expr_or_default { THD *thd= YYTHD; - LEX *lex=Lex; + LEX *lex= Lex; + LEX_STRING *name= &$2.base_name; if ($2.var == trg_new_row_fake_var) { /* We are in trigger and assigning value to field of new row */ - Item *it; - Item_trigger_field *trg_fld; - sp_instr_set_trigger_field *sp_fld; - LINT_INIT(sp_fld); if ($1) { my_parse_error(ER(ER_SYNTAX_ERROR)); MYSQL_YYABORT; } - if ($4) - it= $4; - else - { - /* QQ: Shouldn't this be field's default value ? */ - it= new Item_null(); - } - - DBUG_ASSERT(lex->trg_chistics.action_time == TRG_ACTION_BEFORE && - (lex->trg_chistics.event == TRG_EVENT_INSERT || - lex->trg_chistics.event == TRG_EVENT_UPDATE)); - - trg_fld= new (thd->mem_root) - Item_trigger_field(Lex->current_context(), - Item_trigger_field::NEW_ROW, - $2.base_name.str, - UPDATE_ACL, FALSE); - if (trg_fld == NULL) - MYSQL_YYABORT; - - sp_fld= new sp_instr_set_trigger_field(lex->sphead-> - instructions(), - lex->spcont, - trg_fld, - it, lex); - if (sp_fld == NULL) - MYSQL_YYABORT; - - /* - Let us add this item to list of all Item_trigger_field - objects in trigger. - */ - lex->trg_table_fields.link_in_list((uchar *)trg_fld, - (uchar **) &trg_fld-> - next_trg_field); - - if (lex->sphead->add_instr(sp_fld)) + if (set_trigger_new_row(YYTHD, name, $4)) MYSQL_YYABORT; } else if ($2.var) - { /* System variable */ + { if ($1) lex->option_type= $1; - set_var *var= new set_var(lex->option_type, $2.var, - &$2.base_name, $4); - if (var == NULL) + + /* It is a system variable. */ + if (set_system_variable(thd, &$2, lex->option_type, $4)) MYSQL_YYABORT; - lex->var_list.push_back(var); } else { - /* An SP local variable */ - sp_pcontext *ctx= lex->spcont; - sp_variable_t *spv; - sp_instr_set *sp_set; - Item *it; + sp_pcontext *spc= lex->spcont; + sp_variable_t *spv= spc->find_variable(name); + if ($1) { my_parse_error(ER(ER_SYNTAX_ERROR)); MYSQL_YYABORT; } - spv= ctx->find_variable(&$2.base_name); - - if ($4) - it= $4; - else if (spv->dflt) - it= spv->dflt; - else - { - it= new (thd->mem_root) Item_null(); - if (it == NULL) - MYSQL_YYABORT; - } - sp_set= new sp_instr_set(lex->sphead->instructions(), ctx, - spv->offset, it, spv->type, lex, TRUE); - if (sp_set == NULL || - lex->sphead->add_instr(sp_set)) + /* It is a local variable. */ + if (set_local_variable(thd, spv, $4)) MYSQL_YYABORT; } } @@ -11958,11 +12047,16 @@ option_value: } | '@' '@' opt_var_ident_type internal_variable_name equal set_expr_or_default { - LEX *lex=Lex; - set_var *var= new set_var($3, $4.var, &$4.base_name, $6); - if (var == NULL) + THD *thd= YYTHD; + struct sys_var_with_base tmp= $4; + /* Lookup if necessary: must be a system variable. */ + if (tmp.var == NULL) + { + if (find_sys_var_null_base(thd, &tmp)) + MYSQL_YYABORT; + } + if (set_system_variable(thd, &tmp, $3, $6)) MYSQL_YYABORT; - lex->var_list.push_back(var); } | charset old_or_new_charset_name_or_default { @@ -12055,31 +12149,26 @@ internal_variable_name: ident { THD *thd= YYTHD; - LEX *lex= thd->lex; - sp_pcontext *spc= lex->spcont; + sp_pcontext *spc= thd->lex->spcont; sp_variable_t *spv; - /* We have to lookup here since local vars can shadow sysvars */ + /* Best effort lookup for system variable. */ if (!spc || !(spv = spc->find_variable(&$1))) { + struct sys_var_with_base tmp= {NULL, $1}; + /* Not an SP local variable */ - sys_var *tmp=find_sys_var(thd, $1.str, $1.length); - if (!tmp) + if (find_sys_var_null_base(thd, &tmp)) MYSQL_YYABORT; - $$.var= tmp; - $$.base_name= null_lex_str; - if (spc && tmp == &sys_autocommit) - { - /* - We don't allow setting AUTOCOMMIT from a stored function - or trigger. - */ - lex->sphead->m_flags|= sp_head::HAS_SET_AUTOCOMMIT_STMT; - } + + $$= tmp; } else { - /* An SP local variable */ + /* + Possibly an SP local variable (or a shadowed sysvar). + Will depend on the context of the SET statement. + */ $$.var= NULL; $$.base_name= $1; } @@ -12237,12 +12326,7 @@ lock_option: | WRITE_SYM { $$= TL_WRITE_DEFAULT; } | WRITE_SYM CONCURRENT { -#ifdef HAVE_QUERY_CACHE - if (Lex->sphead != 0) - $$= TL_WRITE_DEFAULT; - else -#endif - $$= TL_WRITE_CONCURRENT_INSERT; + $$= (Lex->sphead ? TL_WRITE_DEFAULT : TL_WRITE_CONCURRENT_INSERT); } | LOW_PRIORITY WRITE_SYM { $$= TL_WRITE_LOW_PRIORITY; } === modified file 'sql/table.cc' --- a/sql/table.cc 2009-12-03 11:34:11 +0000 +++ b/sql/table.cc 2010-01-15 15:27:55 +0000 @@ -1316,8 +1316,16 @@ static int open_binary_frm(THD *thd, TAB share->timestamp_field_offset= i; if (use_hash) - (void) my_hash_insert(&share->name_hash, - (uchar*) field_ptr); // never fail + if (my_hash_insert(&share->name_hash, (uchar*) field_ptr) ) + { + /* + Set return code 8 here to indicate that an error has + occurred but that the error message already has been + sent (OOM). + */ + error= 8; + goto err; + } } *field_ptr=0; // End marker @@ -2804,34 +2812,38 @@ bool check_column_name(const char *name) and such errors never reach the user. */ -my_bool -table_check_intact(TABLE *table, const uint table_f_count, - const TABLE_FIELD_W_TYPE *table_def) +bool +Table_check_intact::check(TABLE *table, const TABLE_FIELD_DEF *table_def) { uint i; my_bool error= FALSE; - my_bool fields_diff_count; + const TABLE_FIELD_TYPE *field_def= table_def->field; DBUG_ENTER("table_check_intact"); DBUG_PRINT("info",("table: %s expected_count: %d", - table->alias, table_f_count)); + table->alias, table_def->count)); - fields_diff_count= (table->s->fields != table_f_count); - if (fields_diff_count) + /* Whether the table definition has already been validated. */ + if (table->s->table_field_def_cache == table_def) + DBUG_RETURN(FALSE); + + if (table->s->fields != table_def->count) { DBUG_PRINT("info", ("Column count has changed, checking the definition")); /* previous MySQL version */ if (MYSQL_VERSION_ID > table->s->mysql_version) { - sql_print_error(ER(ER_COL_COUNT_DOESNT_MATCH_PLEASE_UPDATE), - table->alias, table_f_count, table->s->fields, - table->s->mysql_version, MYSQL_VERSION_ID); + report_error(ER_COL_COUNT_DOESNT_MATCH_PLEASE_UPDATE, + ER(ER_COL_COUNT_DOESNT_MATCH_PLEASE_UPDATE), + table->alias, table_def->count, table->s->fields, + table->s->mysql_version, MYSQL_VERSION_ID); DBUG_RETURN(TRUE); } else if (MYSQL_VERSION_ID == table->s->mysql_version) { - sql_print_error(ER(ER_COL_COUNT_DOESNT_MATCH_CORRUPTED), table->alias, - table_f_count, table->s->fields); + report_error(ER_COL_COUNT_DOESNT_MATCH_CORRUPTED, + ER(ER_COL_COUNT_DOESNT_MATCH_CORRUPTED), table->alias, + table_def->count, table->s->fields); DBUG_RETURN(TRUE); } /* @@ -2843,7 +2855,7 @@ table_check_intact(TABLE *table, const u */ } char buffer[STRING_BUFFER_USUAL_SIZE]; - for (i=0 ; i < table_f_count; i++, table_def++) + for (i=0 ; i < table_def->count; i++, field_def++) { String sql_type(buffer, sizeof(buffer), system_charset_info); sql_type.length(0); @@ -2851,18 +2863,18 @@ table_check_intact(TABLE *table, const u { Field *field= table->field[i]; - if (strncmp(field->field_name, table_def->name.str, - table_def->name.length)) + if (strncmp(field->field_name, field_def->name.str, + field_def->name.length)) { /* Name changes are not fatal, we use ordinal numbers to access columns. Still this can be a sign of a tampered table, output an error to the error log. */ - sql_print_error("Incorrect definition of table %s.%s: " - "expected column '%s' at position %d, found '%s'.", - table->s->db.str, table->alias, table_def->name.str, i, - field->field_name); + report_error(0, "Incorrect definition of table %s.%s: " + "expected column '%s' at position %d, found '%s'.", + table->s->db.str, table->alias, field_def->name.str, i, + field->field_name); } field->sql_type(sql_type); /* @@ -2882,47 +2894,51 @@ table_check_intact(TABLE *table, const u the new table definition is backward compatible with the original one. */ - if (strncmp(sql_type.c_ptr_safe(), table_def->type.str, - table_def->type.length - 1)) + if (strncmp(sql_type.c_ptr_safe(), field_def->type.str, + field_def->type.length - 1)) { - sql_print_error("Incorrect definition of table %s.%s: " - "expected column '%s' at position %d to have type " - "%s, found type %s.", table->s->db.str, table->alias, - table_def->name.str, i, table_def->type.str, - sql_type.c_ptr_safe()); + report_error(0, "Incorrect definition of table %s.%s: " + "expected column '%s' at position %d to have type " + "%s, found type %s.", table->s->db.str, table->alias, + field_def->name.str, i, field_def->type.str, + sql_type.c_ptr_safe()); error= TRUE; } - else if (table_def->cset.str && !field->has_charset()) + else if (field_def->cset.str && !field->has_charset()) { - sql_print_error("Incorrect definition of table %s.%s: " - "expected the type of column '%s' at position %d " - "to have character set '%s' but the type has no " - "character set.", table->s->db.str, table->alias, - table_def->name.str, i, table_def->cset.str); + report_error(0, "Incorrect definition of table %s.%s: " + "expected the type of column '%s' at position %d " + "to have character set '%s' but the type has no " + "character set.", table->s->db.str, table->alias, + field_def->name.str, i, field_def->cset.str); error= TRUE; } - else if (table_def->cset.str && - strcmp(field->charset()->csname, table_def->cset.str)) + else if (field_def->cset.str && + strcmp(field->charset()->csname, field_def->cset.str)) { - sql_print_error("Incorrect definition of table %s.%s: " - "expected the type of column '%s' at position %d " - "to have character set '%s' but found " - "character set '%s'.", table->s->db.str, table->alias, - table_def->name.str, i, table_def->cset.str, - field->charset()->csname); + report_error(0, "Incorrect definition of table %s.%s: " + "expected the type of column '%s' at position %d " + "to have character set '%s' but found " + "character set '%s'.", table->s->db.str, table->alias, + field_def->name.str, i, field_def->cset.str, + field->charset()->csname); error= TRUE; } } else { - sql_print_error("Incorrect definition of table %s.%s: " - "expected column '%s' at position %d to have type %s " - " but the column is not found.", - table->s->db.str, table->alias, - table_def->name.str, i, table_def->type.str); + report_error(0, "Incorrect definition of table %s.%s: " + "expected column '%s' at position %d to have type %s " + " but the column is not found.", + table->s->db.str, table->alias, + field_def->name.str, i, field_def->type.str); error= TRUE; } } + + if (! error) + table->s->table_field_def_cache= table_def; + DBUG_RETURN(error); } === modified file 'sql/table.h' --- a/sql/table.h 2009-12-03 11:19:05 +0000 +++ b/sql/table.h 2010-01-15 15:27:55 +0000 @@ -285,6 +285,36 @@ typedef enum enum_table_category TABLE_C TABLE_CATEGORY get_table_category(const LEX_STRING *db, const LEX_STRING *name); + +typedef struct st_table_field_type +{ + LEX_STRING name; + LEX_STRING type; + LEX_STRING cset; +} TABLE_FIELD_TYPE; + + +typedef struct st_table_field_def +{ + uint count; + const TABLE_FIELD_TYPE *field; +} TABLE_FIELD_DEF; + + +class Table_check_intact +{ +protected: + virtual void report_error(uint code, const char *fmt, ...)= 0; + +public: + Table_check_intact() {} + virtual ~Table_check_intact() {} + + /** Checks whether a table is intact. */ + bool check(TABLE *table, const TABLE_FIELD_DEF *table_def); +}; + + /* This structure is shared between different table objects. There is one instance of table share per one table in the database. @@ -423,6 +453,18 @@ typedef struct st_table_share handlerton *default_part_db_type; #endif + /** + Cache the checked structure of this table. + + The pointer data is used to describe the structure that + a instance of the table must have. Each element of the + array specifies a field that must exist on the table. + + The pointer is cached in order to perform the check only + once -- when the table is loaded from the disk. + */ + const TABLE_FIELD_DEF *table_field_def_cache; + /** place to store storage engine specific data */ void *ha_data; @@ -1674,17 +1716,6 @@ typedef struct st_open_table_list{ uint32 in_use,locked; } OPEN_TABLE_LIST; -typedef struct st_table_field_w_type -{ - LEX_STRING name; - LEX_STRING type; - LEX_STRING cset; -} TABLE_FIELD_W_TYPE; - - -my_bool -table_check_intact(TABLE *table, const uint table_f_count, - const TABLE_FIELD_W_TYPE *table_def); static inline my_bitmap_map *tmp_use_all_columns(TABLE *table, MY_BITMAP *bitmap) === modified file 'sql/udf_example.def' --- a/sql/udf_example.def 2007-10-29 14:01:40 +0000 +++ b/sql/udf_example.def 2010-01-29 18:42:22 +0000 @@ -3,8 +3,10 @@ VERSION 1.0 EXPORTS lookup lookup_init + lookup_deinit reverse_lookup reverse_lookup_init + reverse_lookup_deinit metaphon_init metaphon_deinit metaphon === modified file 'storage/archive/CMakeLists.txt' (properties changed: -x to +x) --- a/storage/archive/CMakeLists.txt 2009-06-10 08:59:49 +0000 +++ b/storage/archive/CMakeLists.txt 2009-11-10 19:41:43 +0000 @@ -13,6 +13,9 @@ # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +SET(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -DSAFEMALLOC -DSAFE_MUTEX") +SET(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} -DSAFEMALLOC -DSAFE_MUTEX") + INCLUDE("${PROJECT_SOURCE_DIR}/storage/mysql_storage_engine.cmake") SET(ARCHIVE_SOURCES azio.c ha_archive.cc ha_archive.h) MYSQL_STORAGE_ENGINE(ARCHIVE) === modified file 'storage/archive/azio.c' --- a/storage/archive/azio.c 2009-05-22 12:38:50 +0000 +++ b/storage/archive/azio.c 2010-01-15 15:27:55 +0000 @@ -71,7 +71,8 @@ int az_open (azio_stream *s, const char s->transparent = 0; s->mode = 'r'; s->version = (unsigned char)az_magic[1]; /* this needs to be a define to version */ - s->version = (unsigned char)az_magic[2]; /* minor version */ + s->minor_version= (unsigned char) az_magic[2]; /* minor version */ + s->dirty= AZ_STATE_CLEAN; /* We do our own version of append by nature. @@ -354,10 +355,19 @@ void read_header(azio_stream *s, unsigne s->comment_length= (unsigned int)uint4korr(buffer + AZ_COMMENT_LENGTH_POS); s->dirty= (unsigned int)buffer[AZ_DIRTY_POS]; } - else + else if (buffer[0] == gz_magic[0] && buffer[1] == gz_magic[1]) { - DBUG_ASSERT(buffer[0] == az_magic[0] && buffer[1] == az_magic[1]); - return; + /* + Set version number to previous version (2). + */ + s->version= (unsigned char) 2; + } else { + /* + Unknown version. + Most probably due to a corrupt archive. + */ + s->dirty= AZ_STATE_DIRTY; + s->z_err= Z_VERSION_ERROR; } } === modified file 'storage/archive/ha_archive.cc' --- a/storage/archive/ha_archive.cc 2009-12-03 11:19:05 +0000 +++ b/storage/archive/ha_archive.cc 2010-01-15 15:27:55 +0000 @@ -360,6 +360,12 @@ ARCHIVE_SHARE *ha_archive::get_share(con stats.auto_increment_value= archive_tmp.auto_increment + 1; share->rows_recorded= (ha_rows)archive_tmp.rows; share->crashed= archive_tmp.dirty; + /* + If archive version is less than 3, It should be upgraded before + use. + */ + if (archive_tmp.version < ARCHIVE_VERSION) + *rc= HA_ERR_TABLE_NEEDS_UPGRADE; azclose(&archive_tmp); VOID(my_hash_insert(&archive_open_tables, (uchar*) share)); @@ -491,7 +497,15 @@ int ha_archive::open(const char *name, i (open_options & HA_OPEN_FOR_REPAIR) ? "yes" : "no")); share= get_share(name, &rc); - if (rc == HA_ERR_CRASHED_ON_USAGE && !(open_options & HA_OPEN_FOR_REPAIR)) + /* + Allow open on crashed table in repair mode only. + Block open on 5.0 ARCHIVE table. Though we have almost all + routines to access these tables, they were not well tested. + For now we have to refuse to open such table to avoid + potential data loss. + */ + if ((rc == HA_ERR_CRASHED_ON_USAGE && !(open_options & HA_OPEN_FOR_REPAIR)) + || rc == HA_ERR_TABLE_NEEDS_UPGRADE) { /* purecov: begin inspected */ free_share(); === modified file 'storage/federated/CMakeLists.txt' (properties changed: -x to +x) --- a/storage/federated/CMakeLists.txt 2009-06-10 08:59:49 +0000 +++ b/storage/federated/CMakeLists.txt 2009-11-10 19:41:43 +0000 @@ -1,18 +1,21 @@ # Copyright (C) 2006 MySQL AB -# +# # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; version 2 of the License. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +SET(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -DSAFEMALLOC -DSAFE_MUTEX") +SET(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} -DSAFEMALLOC -DSAFE_MUTEX") + INCLUDE("${PROJECT_SOURCE_DIR}/storage/mysql_storage_engine.cmake") SET(FEDERATED_SOURCES ha_federated.cc) MYSQL_STORAGE_ENGINE(FEDERATED) === modified file 'storage/innobase/btr/btr0btr.c' --- a/storage/innobase/btr/btr0btr.c 2007-07-10 14:34:21 +0000 +++ b/storage/innobase/btr/btr0btr.c 2009-11-30 08:50:08 +0000 @@ -709,8 +709,15 @@ btr_create( } else { /* It is a non-ibuf tree: create a file segment for leaf pages */ - fseg_create(space, page_no, PAGE_HEADER + PAGE_BTR_SEG_LEAF, - mtr); + if (!fseg_create(space, page_no, + PAGE_HEADER + PAGE_BTR_SEG_LEAF, mtr)) { + /* Not enough space for new segment, free root + segment before return. */ + btr_free_root(space, page_no, mtr); + + return(FIL_NULL); + } + /* The fseg create acquires a second latch on the page, therefore we must declare it: */ #ifdef UNIV_SYNC_DEBUG === modified file 'storage/innobase/data/data0type.c' --- a/storage/innobase/data/data0type.c 2007-07-10 11:37:43 +0000 +++ b/storage/innobase/data/data0type.c 2009-11-30 08:53:52 +0000 @@ -252,6 +252,22 @@ dtype_print( fputs("DATA_SYS", stderr); break; + case DATA_FLOAT: + fputs("DATA_FLOAT", stderr); + break; + + case DATA_DOUBLE: + fputs("DATA_DOUBLE", stderr); + break; + + case DATA_DECIMAL: + fputs("DATA_DECIMAL", stderr); + break; + + case DATA_VARMYSQL: + fputs("DATA_VARMYSQL", stderr); + break; + default: fprintf(stderr, "type %lu", (ulong) mtype); break; === modified file 'storage/innobase/handler/ha_innodb.cc' --- a/storage/innobase/handler/ha_innodb.cc 2009-12-03 11:19:05 +0000 +++ b/storage/innobase/handler/ha_innodb.cc 2010-01-15 15:27:55 +0000 @@ -662,6 +662,12 @@ convert_error_code_to_mysql( } else if (error == (int) DB_DUPLICATE_KEY) { + /* Be cautious with returning this error, since + mysql could re-enter the storage layer to get + duplicated key info, the operation requires a + valid table handle and/or transaction information, + which might not always be available in the error + handling stage. */ return(HA_ERR_FOUND_DUPP_KEY); } else if (error == (int) DB_FOREIGN_DUPLICATE_KEY) { @@ -765,35 +771,6 @@ convert_error_code_to_mysql( } /***************************************************************** -If you want to print a thd that is not associated with the current thread, -you must call this function before reserving the InnoDB kernel_mutex, to -protect MySQL from setting thd->query NULL. If you print a thd of the current -thread, we know that MySQL cannot modify thd->query, and it is not necessary -to call this. Call innobase_mysql_end_print_arbitrary_thd() after you release -the kernel_mutex. -NOTE that /mysql/innobase/lock/lock0lock.c must contain the prototype for this -function! */ -extern "C" -void -innobase_mysql_prepare_print_arbitrary_thd(void) -/*============================================*/ -{ - VOID(pthread_mutex_lock(&LOCK_thread_count)); -} - -/***************************************************************** -Releases the mutex reserved by innobase_mysql_prepare_print_arbitrary_thd(). -NOTE that /mysql/innobase/lock/lock0lock.c must contain the prototype for this -function! */ -extern "C" -void -innobase_mysql_end_print_arbitrary_thd(void) -/*========================================*/ -{ - VOID(pthread_mutex_unlock(&LOCK_thread_count)); -} - -/***************************************************************** Prints info of a THD object (== user session thread) to the given file. NOTE that /mysql/innobase/trx/trx0trx.c must contain the prototype for this function! */ @@ -1499,70 +1476,148 @@ innobase_invalidate_query_cache( #endif } -/********************************************************************* -Display an SQL identifier. */ -extern "C" -void -innobase_print_identifier( -/*======================*/ - FILE* f, /* in: output stream */ - trx_t* trx, /* in: transaction */ - ibool table_id,/* in: TRUE=print a table name, - FALSE=print other identifier */ - const char* name, /* in: name to print */ - ulint namelen)/* in: length of name */ -{ - const char* s = name; - char* qname = NULL; +/*****************************************************************//** +Convert an SQL identifier to the MySQL system_charset_info (UTF-8) +and quote it if needed. +@return pointer to the end of buf */ +static +char* +innobase_convert_identifier( +/*========================*/ + char* buf, /*!< out: buffer for converted identifier */ + ulint buflen, /*!< in: length of buf, in bytes */ + const char* id, /*!< in: identifier to convert */ + ulint idlen, /*!< in: length of id, in bytes */ + void* thd, /*!< in: MySQL connection thread, or NULL */ + ibool file_id)/*!< in: TRUE=id is a table or database name; + FALSE=id is an UTF-8 string */ +{ + char nz[NAME_LEN + 1]; +#if MYSQL_VERSION_ID >= 50141 + char nz2[NAME_LEN + 1 + EXPLAIN_FILENAME_MAX_EXTRA_LENGTH]; +#else /* MYSQL_VERSION_ID >= 50141 */ + char nz2[NAME_LEN + 1 + sizeof srv_mysql50_table_name_prefix]; +#endif /* MYSQL_VERSION_ID >= 50141 */ + + const char* s = id; int q; - if (table_id) { - /* Decode the table name. The filename_to_tablename() - function expects a NUL-terminated string. The input and - output strings buffers must not be shared. The function - only produces more output when the name contains other - characters than [0-9A-Z_a-z]. */ - char* temp_name = (char*) my_malloc((uint) namelen + 1, MYF(MY_WME)); - uint qnamelen = (uint) (namelen - + (1 + sizeof srv_mysql50_table_name_prefix)); - - if (temp_name) { - qname = (char*) my_malloc(qnamelen, MYF(MY_WME)); - if (qname) { - memcpy(temp_name, name, namelen); - temp_name[namelen] = 0; - s = qname; - namelen = filename_to_tablename(temp_name, - qname, qnamelen); - } - my_free(temp_name, MYF(0)); - } + if (file_id) { + /* Decode the table name. The MySQL function expects + a NUL-terminated string. The input and output strings + buffers must not be shared. */ + + if (UNIV_UNLIKELY(idlen > (sizeof nz) - 1)) { + idlen = (sizeof nz) - 1; + } + + memcpy(nz, id, idlen); + nz[idlen] = 0; + + s = nz2; +#if MYSQL_VERSION_ID >= 50141 + idlen = explain_filename((THD*) thd, nz, nz2, sizeof nz2, + EXPLAIN_PARTITIONS_AS_COMMENT); + goto no_quote; +#else /* MYSQL_VERSION_ID >= 50141 */ + idlen = filename_to_tablename(nz, nz2, sizeof nz2); +#endif /* MYSQL_VERSION_ID >= 50141 */ } - if (!trx || !trx->mysql_thd) { - + /* See if the identifier needs to be quoted. */ + if (UNIV_UNLIKELY(!thd)) { q = '"'; } else { - q = get_quote_char_for_identifier((THD*) trx->mysql_thd, - s, (int) namelen); + q = get_quote_char_for_identifier((THD*) thd, s, (int) idlen); } if (q == EOF) { - fwrite(s, 1, namelen, f); - } else { - const char* e = s + namelen; - putc(q, f); - while (s < e) { - int c = *s++; - if (c == q) { - putc(c, f); +#if MYSQL_VERSION_ID >= 50141 +no_quote: +#endif /* MYSQL_VERSION_ID >= 50141 */ + if (UNIV_UNLIKELY(idlen > buflen)) { + idlen = buflen; + } + memcpy(buf, s, idlen); + return(buf + idlen); + } + + /* Quote the identifier. */ + if (buflen < 2) { + return(buf); + } + + *buf++ = q; + buflen--; + + for (; idlen; idlen--) { + int c = *s++; + if (UNIV_UNLIKELY(c == q)) { + if (UNIV_UNLIKELY(buflen < 3)) { + break; + } + + *buf++ = c; + *buf++ = c; + buflen -= 2; + } else { + if (UNIV_UNLIKELY(buflen < 2)) { + break; } - putc(c, f); + + *buf++ = c; + buflen--; + } + } + + *buf++ = q; + return(buf); +} + +/*****************************************************************//** +Convert a table or index name to the MySQL system_charset_info (UTF-8) +and quote it if needed. +@return pointer to the end of buf */ +extern "C" +char* +innobase_convert_name( +/*==================*/ + char* buf, /*!< out: buffer for converted identifier */ + ulint buflen, /*!< in: length of buf, in bytes */ + const char* id, /*!< in: identifier to convert */ + ulint idlen, /*!< in: length of id, in bytes */ + void* thd, /*!< in: MySQL connection thread, or NULL */ + ibool table_id)/*!< in: TRUE=id is a table or database name; + FALSE=id is an index name */ +{ + char* s = buf; + const char* bufend = buf + buflen; + + if (table_id) { + const char* slash = (const char*) memchr(id, '/', idlen); + if (!slash) { + + goto no_db_name; } - putc(q, f); + + /* Print the database name and table name separately. */ + s = innobase_convert_identifier(s, bufend - s, id, slash - id, + thd, TRUE); + if (UNIV_LIKELY(s < bufend)) { + *s++ = '.'; + s = innobase_convert_identifier(s, bufend - s, + slash + 1, idlen + - (slash - id) - 1, + thd, TRUE); + } + } else { +no_db_name: + s = innobase_convert_identifier(buf, buflen, id, idlen, + thd, table_id); } - my_free(qname, MYF(MY_ALLOW_ZERO_PTR)); + return(s); + } /************************************************************************** @@ -3986,24 +4041,29 @@ no_commit: update the table upper limit. Note: last_value will be 0 if get_auto_increment() was not called.*/ - if (auto_inc <= col_max_value - && auto_inc >= prebuilt->autoinc_last_value) { + if (auto_inc >= prebuilt->autoinc_last_value) { set_max_autoinc: - ut_a(prebuilt->autoinc_increment > 0); - - ulonglong need; - ulonglong offset; - - offset = prebuilt->autoinc_offset; - need = prebuilt->autoinc_increment; - - auto_inc = innobase_next_autoinc( - auto_inc, need, offset, col_max_value); - - err = innobase_set_max_autoinc(auto_inc); - - if (err != DB_SUCCESS) { - error = err; + /* This should filter out the negative + values set explicitly by the user. */ + if (auto_inc <= col_max_value) { + ut_a(prebuilt->autoinc_increment > 0); + + ulonglong need; + ulonglong offset; + + offset = prebuilt->autoinc_offset; + need = prebuilt->autoinc_increment; + + auto_inc = innobase_next_autoinc( + auto_inc, + need, offset, col_max_value); + + err = innobase_set_max_autoinc( + auto_inc); + + if (err != DB_SUCCESS) { + error = err; + } } } break; @@ -5970,6 +6030,24 @@ ha_innobase::rename_table( innobase_commit_low(trx); trx_free_for_mysql(trx); + /* Add a special case to handle the Duplicated Key error + and return DB_ERROR instead. + This is to avoid a possible SIGSEGV error from mysql error + handling code. Currently, mysql handles the Duplicated Key + error by re-entering the storage layer and getting dup key + info by calling get_dup_key(). This operation requires a valid + table handle ('row_prebuilt_t' structure) which could no + longer be available in the error handling stage. The suggested + solution is to report a 'table exists' error message (since + the dup key error here is due to an existing table whose name + is the one we are trying to rename to) and return the generic + error code. */ + if (error == (int) DB_DUPLICATE_KEY) { + my_error(ER_TABLE_EXISTS_ERROR, MYF(0), to); + + error = DB_ERROR; + } + error = convert_error_code_to_mysql(error, NULL); DBUG_RETURN(error); @@ -8204,8 +8282,7 @@ innobase_xa_prepare( executing XA PREPARE and XA COMMIT commands. In this case we cannot know how many minutes or hours will be between XA PREPARE and XA COMMIT, and we don't want - to block for undefined period of time. - */ + to block for undefined period of time. */ pthread_mutex_lock(&prepare_commit_mutex); trx->active_trans = 2; } === modified file 'storage/innobase/include/ha_prototypes.h' --- a/storage/innobase/include/ha_prototypes.h 2008-12-14 19:28:19 +0000 +++ b/storage/innobase/include/ha_prototypes.h 2009-11-30 08:26:45 +0000 @@ -24,18 +24,21 @@ innobase_convert_string( CHARSET_INFO* from_cs, uint* errors); -/********************************************************************* -Display an SQL identifier. */ - -void -innobase_print_identifier( -/*======================*/ - FILE* f, /* in: output stream */ - trx_t* trx, /* in: transaction */ - ibool table_id,/* in: TRUE=print a table name, - FALSE=print other identifier */ - const char* name, /* in: name to print */ - ulint namelen);/* in: length of name */ +/*****************************************************************//** +Convert a table or index name to the MySQL system_charset_info (UTF-8) +and quote it if needed. +@return pointer to the end of buf */ + +char* +innobase_convert_name( +/*==================*/ + char* buf, /*!< out: buffer for converted identifier */ + ulint buflen, /*!< in: length of buf, in bytes */ + const char* id, /*!< in: identifier to convert */ + ulint idlen, /*!< in: length of id, in bytes */ + void* thd, /*!< in: MySQL connection thread, or NULL */ + ibool table_id);/*!< in: TRUE=id is a table or database name; + FALSE=id is an index name */ /********************************************************************** Returns true if the thread is the replication thread on the slave === modified file 'storage/innobase/include/mach0data.h' --- a/storage/innobase/include/mach0data.h 2008-08-20 00:37:41 +0000 +++ b/storage/innobase/include/mach0data.h 2009-11-30 09:41:38 +0000 @@ -266,8 +266,8 @@ UNIV_INLINE double mach_double_read( /*=============*/ - /* out: double read */ - byte* b); /* in: pointer to memory from where to read */ + /* out: double read */ + const byte* b); /* in: pointer to memory from where to read */ /************************************************************* Writes a double. It is stored in a little-endian format. */ UNIV_INLINE @@ -282,8 +282,8 @@ UNIV_INLINE float mach_float_read( /*============*/ - /* out: float read */ - byte* b); /* in: pointer to memory from where to read */ + /* out: float read */ + const byte* b); /* in: pointer to memory from where to read */ /************************************************************* Writes a float. It is stored in a little-endian format. */ UNIV_INLINE === modified file 'storage/innobase/include/mach0data.ic' --- a/storage/innobase/include/mach0data.ic 2008-08-20 00:37:41 +0000 +++ b/storage/innobase/include/mach0data.ic 2009-11-30 09:41:38 +0000 @@ -504,8 +504,8 @@ UNIV_INLINE double mach_double_read( /*=============*/ - /* out: double read */ - byte* b) /* in: pointer to memory from where to read */ + /* out: double read */ + const byte* b) /* in: pointer to memory from where to read */ { double d; ulint i; @@ -553,8 +553,8 @@ UNIV_INLINE float mach_float_read( /*============*/ - /* out: float read */ - byte* b) /* in: pointer to memory from where to read */ + /* out: float read */ + const byte* b) /* in: pointer to memory from where to read */ { float d; ulint i; === modified file 'storage/innobase/include/os0file.h' --- a/storage/innobase/include/os0file.h 2007-07-10 14:34:21 +0000 +++ b/storage/innobase/include/os0file.h 2009-11-30 08:40:31 +0000 @@ -96,6 +96,8 @@ log. */ to become available again */ #define OS_FILE_SHARING_VIOLATION 76 #define OS_FILE_ERROR_NOT_SPECIFIED 77 + /* 78 is used in the plugin */ +#define OS_FILE_OPERATION_ABORTED 79 /* Types for aio operations */ #define OS_FILE_READ 10 === modified file 'storage/innobase/include/trx0trx.h' --- a/storage/innobase/include/trx0trx.h 2009-07-10 23:12:13 +0000 +++ b/storage/innobase/include/trx0trx.h 2009-12-01 10:38:40 +0000 @@ -318,9 +318,7 @@ trx_commit_step( /************************************************************************** Prints info about a transaction to the given file. The caller must own the -kernel mutex and must have called -innobase_mysql_prepare_print_arbitrary_thd(), unless he knows that MySQL -or InnoDB cannot meanwhile change the info printed here. */ +kernel mutex. */ void trx_print( === modified file 'storage/innobase/lock/lock0lock.c' --- a/storage/innobase/lock/lock0lock.c 2009-07-10 23:12:13 +0000 +++ b/storage/innobase/lock/lock0lock.c 2009-12-01 10:38:40 +0000 @@ -22,31 +22,6 @@ Created 5/7/1996 Heikki Tuuri #include "trx0sys.h" -/* 2 function prototypes copied from ha_innodb.cc: */ - -/***************************************************************** -If you want to print a thd that is not associated with the current thread, -you must call this function before reserving the InnoDB kernel_mutex, to -protect MySQL from setting thd->query NULL. If you print a thd of the current -thread, we know that MySQL cannot modify thd->query, and it is not necessary -to call this. Call innobase_mysql_end_print_arbitrary_thd() after you release -the kernel_mutex. -NOTE that /mysql/innobase/lock/lock0lock.c must contain the prototype for this -function! */ - -void -innobase_mysql_prepare_print_arbitrary_thd(void); -/*============================================*/ - -/***************************************************************** -Relases the mutex reserved by innobase_mysql_prepare_print_arbitrary_thd(). -NOTE that /mysql/innobase/lock/lock0lock.c must contain the prototype for this -function! */ - -void -innobase_mysql_end_print_arbitrary_thd(void); -/*========================================*/ - /* Restricts the length of search we will do in the waits-for graph of transactions */ #define LOCK_MAX_N_STEPS_IN_DEADLOCK_CHECK 1000000 @@ -4222,11 +4197,6 @@ lock_print_info_summary( /*====================*/ FILE* file) /* in: file where to print */ { - /* We must protect the MySQL thd->query field with a MySQL mutex, and - because the MySQL mutex must be reserved before the kernel_mutex of - InnoDB, we call innobase_mysql_prepare_print_arbitrary_thd() here. */ - - innobase_mysql_prepare_print_arbitrary_thd(); lock_mutex_enter_kernel(); if (lock_deadlock_found) { @@ -4314,7 +4284,6 @@ loop: if (trx == NULL) { lock_mutex_exit_kernel(); - innobase_mysql_end_print_arbitrary_thd(); ut_ad(lock_validate()); @@ -4386,7 +4355,6 @@ loop: if (load_page_first) { lock_mutex_exit_kernel(); - innobase_mysql_end_print_arbitrary_thd(); mtr_start(&mtr); @@ -4397,7 +4365,6 @@ loop: load_page_first = FALSE; - innobase_mysql_prepare_print_arbitrary_thd(); lock_mutex_enter_kernel(); goto loop; === modified file 'storage/innobase/os/os0file.c' --- a/storage/innobase/os/os0file.c 2008-12-14 19:15:12 +0000 +++ b/storage/innobase/os/os0file.c 2009-11-30 08:40:31 +0000 @@ -257,6 +257,13 @@ os_file_get_last_error( " software or another instance\n" "InnoDB: of MySQL." " Please close it to get rid of this error.\n"); + } else if (err == ERROR_OPERATION_ABORTED) { + fprintf(stderr, + "InnoDB: The error means that the I/O" + " operation has been aborted\n" + "InnoDB: because of either a thread exit" + " or an application request.\n" + "InnoDB: Retry attempt is made.\n"); } else { fprintf(stderr, "InnoDB: Some operating system error numbers" @@ -278,6 +285,8 @@ os_file_get_last_error( } else if (err == ERROR_SHARING_VIOLATION || err == ERROR_LOCK_VIOLATION) { return(OS_FILE_SHARING_VIOLATION); + } else if (err == ERROR_OPERATION_ABORTED) { + return(OS_FILE_OPERATION_ABORTED); } else { return(100 + err); } @@ -402,6 +411,10 @@ os_file_handle_error_cond_exit( os_thread_sleep(10000000); /* 10 sec */ return(TRUE); + } else if (err == OS_FILE_OPERATION_ABORTED) { + + os_thread_sleep(100000); /* 100 ms */ + return(TRUE); } else { if (name) { fprintf(stderr, "InnoDB: File name %s\n", name); @@ -3692,6 +3705,7 @@ os_aio_windows_handle( ibool ret_val; BOOL ret; DWORD len; + BOOL retry = FALSE; if (segment == ULINT_UNDEFINED) { array = os_aio_sync_array; @@ -3745,14 +3759,52 @@ os_aio_windows_handle( ut_a(TRUE == os_file_flush(slot->file)); } # endif /* UNIV_DO_FLUSH */ + } else if (os_file_handle_error(slot->name, "Windows aio")) { + + retry = TRUE; } else { - os_file_handle_error(slot->name, "Windows aio"); ret_val = FALSE; } os_mutex_exit(array->mutex); + if (retry) { + /* retry failed read/write operation synchronously. + No need to hold array->mutex. */ + + switch (slot->type) { + case OS_FILE_WRITE: + ret = WriteFile(slot->file, slot->buf, + slot->len, &len, + &(slot->control)); + + break; + case OS_FILE_READ: + ret = ReadFile(slot->file, slot->buf, + slot->len, &len, + &(slot->control)); + + break; + default: + ut_error; + } + + if (!ret && GetLastError() == ERROR_IO_PENDING) { + /* aio was queued successfully! + We want a synchronous i/o operation on a + file where we also use async i/o: in Windows + we must use the same wait mechanism as for + async i/o */ + + ret = GetOverlappedResult(slot->file, + &(slot->control), + &len, TRUE); + } + + ret_val = ret && len == slot->len; + } + os_aio_array_free_slot(array, slot); return(ret_val); === modified file 'storage/innobase/row/row0sel.c' --- a/storage/innobase/row/row0sel.c 2009-07-10 23:12:13 +0000 +++ b/storage/innobase/row/row0sel.c 2009-11-30 09:41:38 +0000 @@ -4514,6 +4514,7 @@ row_search_autoinc_read_column( dict_index_t* index, /* in: index to read from */ const rec_t* rec, /* in: current rec */ ulint col_no, /* in: column number */ + ulint mtype, /*!< in: column main type */ ibool unsigned_type) /* in: signed or unsigned flag */ { ulint len; @@ -4535,9 +4536,26 @@ row_search_autoinc_read_column( data = rec_get_nth_field((rec_t*)rec, offsets, col_no, &len); ut_a(len != UNIV_SQL_NULL); - ut_a(len <= sizeof value); - value = mach_read_int_type(data, len, unsigned_type); + switch (mtype) { + case DATA_INT: + ut_a(len <= sizeof value); + value = mach_read_int_type(data, len, unsigned_type); + break; + + case DATA_FLOAT: + ut_a(len == sizeof(float)); + value = mach_float_read(data); + break; + + case DATA_DOUBLE: + ut_a(len == sizeof(double)); + value = mach_double_read(data); + break; + + default: + ut_error; + } if (UNIV_LIKELY_NULL(heap)) { mem_heap_free(heap); @@ -4625,7 +4643,8 @@ row_search_max_autoinc( dfield->col->prtype & DATA_UNSIGNED); *value = row_search_autoinc_read_column( - index, rec, i, unsigned_type); + index, rec, i, + dfield->col->mtype, unsigned_type); } } === modified file 'storage/innobase/trx/trx0trx.c' --- a/storage/innobase/trx/trx0trx.c 2009-07-10 23:12:13 +0000 +++ b/storage/innobase/trx/trx0trx.c 2009-12-01 10:38:40 +0000 @@ -1652,9 +1652,7 @@ trx_mark_sql_stat_end( /************************************************************************** Prints info about a transaction to the given file. The caller must own the -kernel mutex and must have called -innobase_mysql_prepare_print_arbitrary_thd(), unless he knows that MySQL -or InnoDB cannot meanwhile change the info printed here. */ +kernel mutex. */ void trx_print( === modified file 'storage/innobase/ut/ut0ut.c' --- a/storage/innobase/ut/ut0ut.c 2008-12-14 19:18:59 +0000 +++ b/storage/innobase/ut/ut0ut.c 2009-11-30 08:26:45 +0000 @@ -19,6 +19,7 @@ Created 5/11/1994 Heikki Tuuri #include "ut0sort.h" #include "trx0trx.h" #include "ha_prototypes.h" +#include "mysql_com.h" /* NAME_LEN */ ibool ut_always_false = FALSE; @@ -484,26 +485,17 @@ ut_print_namel( const char* name, /* in: name to print */ ulint namelen)/* in: length of name */ { -#ifdef UNIV_HOTBACKUP - fwrite(name, 1, namelen, f); -#else - if (table_id) { - char* slash = memchr(name, '/', namelen); - if (!slash) { - - goto no_db_name; - } + /* 2 * NAME_LEN for database and table name, + and some slack for the #mysql50# prefix and quotes */ + char buf[3 * NAME_LEN]; + const char* bufend; + + bufend = innobase_convert_name(buf, sizeof buf, + name, namelen, + trx ? trx->mysql_thd : NULL, + table_id); - /* Print the database name and table name separately. */ - innobase_print_identifier(f, trx, TRUE, name, slash - name); - putc('.', f); - innobase_print_identifier(f, trx, TRUE, slash + 1, - namelen - (slash - name) - 1); - } else { -no_db_name: - innobase_print_identifier(f, trx, table_id, name, namelen); - } -#endif + fwrite(buf, 1, bufend - buf, f); } /************************************************************************** === modified file 'storage/innodb_plugin/CMakeLists.txt' --- a/storage/innodb_plugin/CMakeLists.txt 2009-12-03 11:19:05 +0000 +++ b/storage/innodb_plugin/CMakeLists.txt 2010-01-15 15:27:55 +0000 @@ -83,4 +83,4 @@ SET(INNODB_PLUGIN_SOURCES btr/btr0btr.c ADD_DEFINITIONS(-DHAVE_WINDOWS_ATOMICS -DIB_HAVE_PAUSE_INSTRUCTION) #Disable storage engine, as we are using XtraDB -#MYSQL_STORAGE_ENGINE(INNODB_PLUGIN) +#MYSQL_STORAGE_ENGINE(INNOBASE) === modified file 'storage/innodb_plugin/ChangeLog' --- a/storage/innodb_plugin/ChangeLog 2009-11-03 10:34:03 +0000 +++ b/storage/innodb_plugin/ChangeLog 2009-11-30 13:42:26 +0000 @@ -1,3 +1,87 @@ +2009-11-20 The InnoDB Team + + * handler/ha_innodb.cc: + Add a workaround to prevent a crash due to Bug#45961 DDL on + partitioned innodb tables leaves data dictionary in an inconsistent + state + +2009-11-19 The InnoDB Team + + * btr/btr0btr.c: + Fix Bug#48469 when innodb tablespace is configured too small, crash + and corruption! + +2009-11-19 The InnoDB Team + + * data/data0type.c: + Fix Bug#48526 Data type for float and double is incorrectly reported + in InnoDB table monitor + +2009-11-19 The InnoDB Team + + * CMakeLists.txt: + Fix Bug#48317 cannot build innodb as static library + +2009-11-18 The InnoDB Team + + * handler/handler0alter.cc: + Fix Bug#48782 On lock wait timeout, CREATE INDEX (creating primary key) + attempts DROP TABLE + +2009-11-17 The InnoDB Team + + * handler/ha_innodb.cc, mysql-test/innodb.result, + mysql-test/innodb.test, mysql-test/innodb_bug44369.result, + mysql-test/innodb_bug44369.test, mysql-test/patches/innodb-index.diff, + row/row0mysql.c: + Report duplicate table names to the client connection, not to the + error log. + +2009-11-12 The InnoDB Team + + * handler/ha_innodb.cc, include/db0err.h, row/row0merge.c, + row/row0mysql.c: + Allow CREATE INDEX to be interrupted. + Also, when CHECK TABLE is interrupted, report ER_QUERY_INTERRUPTED. + +2009-11-11 The InnoDB Team + + * handler/ha_innodb.cc, mysql-test/innodb_bug47167.result, + mysql-test/innodb_bug47167.test, mysql-test/innodb_file_format.result: + Fix Bug#47167 "set global innodb_file_format_check" cannot set value + by User-Defined Variable + +2009-11-11 The InnoDB Team + + * include/os0file.h, os/os0file.c: + Fix Bug#3139 Mysql crashes: 'windows error 995' after several selects + on a large DB + +2009-11-04 The InnoDB Team + + * handler/ha_innodb.cc: + Fix Bug#32430 'show innodb status' causes errors + Invalid (old?) table or database name in logs + +2009-11-02 The InnoDB Team + + * btr/btr0sea.c, buf/buf0buf.c, dict/dict0dict.c, fil/fil0fil.c, + ibuf/ibuf0ibuf.c, include/btr0sea.h, include/dict0dict.h, + include/fil0fil.h, include/ibuf0ibuf.h, include/lock0lock.h, + include/log0log.h, include/log0recv.h, include/mem0mem.h, + include/mem0pool.h, include/os0file.h, include/pars0pars.h, + include/srv0srv.h, include/thr0loc.h, include/trx0i_s.h, + include/trx0purge.h, include/trx0rseg.h, include/trx0sys.h, + include/trx0undo.h, include/usr0sess.h, lock/lock0lock.c, + log/log0log.c, log/log0recv.c, mem/mem0dbg.c, mem/mem0pool.c, + os/os0file.c, os/os0sync.c, os/os0thread.c, pars/lexyy.c, + pars/pars0lex.l, que/que0que.c, srv/srv0srv.c, srv/srv0start.c, + sync/sync0arr.c, sync/sync0sync.c, thr/thr0loc.c, trx/trx0i_s.c, + trx/trx0purge.c, trx/trx0rseg.c, trx/trx0sys.c, trx/trx0undo.c, + usr/usr0sess.c, ut/ut0mem.c: + Fix Bug #45992 innodb memory not freed after shutdown + Fix Bug #46656 InnoDB plugin: memory leaks (Valgrind) + 2009-10-29 The InnoDB Team * handler/ha_innodb.cc, mysql-test/innodb-autoinc.result, @@ -66,6 +150,12 @@ Fix Bug#47058 Failure to compile innodb_plugin on solaris 10u7 + spro cc/CC 5.10 +2009-10-13 The InnoDB Team + + * buf/buf0flu.c: + Call fsync() on datafiles after a batch of pages is written to disk + even when skip_innodb_doublewrite is set. + 2009-10-05 The InnoDB Team * buf/buf0buf.c: === modified file 'storage/innodb_plugin/btr/btr0btr.c' --- a/storage/innodb_plugin/btr/btr0btr.c 2009-10-12 12:00:56 +0000 +++ b/storage/innodb_plugin/btr/btr0btr.c 2009-11-30 13:42:26 +0000 @@ -790,8 +790,15 @@ btr_create( } else { /* It is a non-ibuf tree: create a file segment for leaf pages */ - fseg_create(space, page_no, - PAGE_HEADER + PAGE_BTR_SEG_LEAF, mtr); + if (!fseg_create(space, page_no, + PAGE_HEADER + PAGE_BTR_SEG_LEAF, mtr)) { + /* Not enough space for new segment, free root + segment before return. */ + btr_free_root(space, zip_size, page_no, mtr); + + return(FIL_NULL); + } + /* The fseg create acquires a second latch on the page, therefore we must declare it: */ buf_block_dbg_add_level(block, SYNC_TREE_NODE_NEW); === modified file 'storage/innodb_plugin/btr/btr0sea.c' --- a/storage/innodb_plugin/btr/btr0sea.c 2009-10-08 11:28:37 +0000 +++ b/storage/innodb_plugin/btr/btr0sea.c 2009-11-30 11:32:05 +0000 @@ -175,6 +175,21 @@ btr_search_sys_create( btr_search_sys->hash_index = ha_create(hash_size, 0, 0); } +/*****************************************************************//** +Frees the adaptive search system at a database shutdown. */ +UNIV_INTERN +void +btr_search_sys_free(void) +/*=====================*/ +{ + mem_free(btr_search_latch_temp); + btr_search_latch_temp = NULL; + mem_heap_free(btr_search_sys->hash_index->heap); + hash_table_free(btr_search_sys->hash_index); + mem_free(btr_search_sys); + btr_search_sys = NULL; +} + /********************************************************************//** Disable the adaptive hash search system and empty the index. */ UNIV_INTERN === modified file 'storage/innodb_plugin/buf/buf0buf.c' --- a/storage/innodb_plugin/buf/buf0buf.c 2009-11-03 10:26:07 +0000 +++ b/storage/innodb_plugin/buf/buf0buf.c 2009-11-30 11:32:05 +0000 @@ -1020,7 +1020,11 @@ buf_pool_free(void) os_mem_free_large(chunk->mem, chunk->mem_size); } - buf_pool->n_chunks = 0; + mem_free(buf_pool->chunks); + hash_table_free(buf_pool->page_hash); + hash_table_free(buf_pool->zip_hash); + mem_free(buf_pool); + buf_pool = NULL; } /********************************************************************//** === modified file 'storage/innodb_plugin/data/data0type.c' --- a/storage/innodb_plugin/data/data0type.c 2009-07-30 12:42:56 +0000 +++ b/storage/innodb_plugin/data/data0type.c 2009-11-30 13:35:20 +0000 @@ -237,6 +237,22 @@ dtype_print( fputs("DATA_SYS", stderr); break; + case DATA_FLOAT: + fputs("DATA_FLOAT", stderr); + break; + + case DATA_DOUBLE: + fputs("DATA_DOUBLE", stderr); + break; + + case DATA_DECIMAL: + fputs("DATA_DECIMAL", stderr); + break; + + case DATA_VARMYSQL: + fputs("DATA_VARMYSQL", stderr); + break; + default: fprintf(stderr, "type %lu", (ulong) mtype); break; === modified file 'storage/innodb_plugin/dict/dict0dict.c' --- a/storage/innodb_plugin/dict/dict0dict.c 2009-10-09 12:52:18 +0000 +++ b/storage/innodb_plugin/dict/dict0dict.c 2009-11-30 11:42:51 +0000 @@ -1200,7 +1200,7 @@ dict_index_too_big_for_undo( = TRX_UNDO_PAGE_HDR - TRX_UNDO_PAGE_HDR_SIZE + 2 /* next record pointer */ + 1 /* type_cmpl */ - + 11 /* trx->undo_no */ - 11 /* table->id */ + + 11 /* trx->undo_no */ + 11 /* table->id */ + 1 /* rec_get_info_bits() */ + 11 /* DB_TRX_ID */ + 11 /* DB_ROLL_PTR */ @@ -4652,6 +4652,26 @@ dict_ind_init(void) dict_ind_redundant->cached = dict_ind_compact->cached = TRUE; } +/**********************************************************************//** +Frees dict_ind_redundant and dict_ind_compact. */ +static +void +dict_ind_free(void) +/*===============*/ +{ + dict_table_t* table; + + table = dict_ind_compact->table; + dict_mem_index_free(dict_ind_compact); + dict_ind_compact = NULL; + dict_mem_table_free(table); + + table = dict_ind_redundant->table; + dict_mem_index_free(dict_ind_redundant); + dict_ind_redundant = NULL; + dict_mem_table_free(table); +} + #ifndef UNIV_HOTBACKUP /**********************************************************************//** Get index by name @@ -4777,4 +4797,55 @@ dict_table_check_for_dup_indexes( } } #endif /* UNIV_DEBUG */ + +/************************************************************************** +Closes the data dictionary module. */ +UNIV_INTERN +void +dict_close(void) +/*============*/ +{ + ulint i; + + /* Free the hash elements. We don't remove them from the table + because we are going to destroy the table anyway. */ + for (i = 0; i < hash_get_n_cells(dict_sys->table_hash); i++) { + dict_table_t* table; + + table = HASH_GET_FIRST(dict_sys->table_hash, i); + + while (table) { + dict_table_t* prev_table = table; + + table = HASH_GET_NEXT(name_hash, prev_table); +#ifdef UNIV_DEBUG + ut_a(prev_table->magic_n == DICT_TABLE_MAGIC_N); +#endif + /* Acquire only because it's a pre-condition. */ + mutex_enter(&dict_sys->mutex); + + dict_table_remove_from_cache(prev_table); + + mutex_exit(&dict_sys->mutex); + } + } + + hash_table_free(dict_sys->table_hash); + + /* The elements are the same instance as in dict_sys->table_hash, + therefore we don't delete the individual elements. */ + hash_table_free(dict_sys->table_id_hash); + + dict_ind_free(); + + mutex_free(&dict_sys->mutex); + + rw_lock_free(&dict_operation_lock); + memset(&dict_operation_lock, 0x0, sizeof(dict_operation_lock)); + + mutex_free(&dict_foreign_err_mutex); + + mem_free(dict_sys); + dict_sys = NULL; +} #endif /* !UNIV_HOTBACKUP */ === modified file 'storage/innodb_plugin/fil/fil0fil.c' --- a/storage/innodb_plugin/fil/fil0fil.c 2009-11-03 10:24:21 +0000 +++ b/storage/innodb_plugin/fil/fil0fil.c 2009-11-30 11:32:05 +0000 @@ -321,6 +321,17 @@ fil_get_space_id_for_table( /*=======================*/ const char* name); /*!< in: table name in the standard 'databasename/tablename' format */ +/*******************************************************************//** +Frees a space object from the tablespace memory cache. Closes the files in +the chain but does not delete them. There must not be any pending i/o's or +flushes on the files. */ +static +ibool +fil_space_free( +/*===========*/ + /* out: TRUE if success */ + ulint id, /* in: space id */ + ibool own_mutex);/* in: TRUE if own system->mutex */ /********************************************************************//** Reads data from a space to a buffer. Remember that the possible incomplete blocks at the end of file are ignored: they are not taken into account when @@ -1144,7 +1155,7 @@ try_again: mutex_exit(&fil_system->mutex); - fil_space_free(namesake_id); + fil_space_free(namesake_id, FALSE); goto try_again; } @@ -1269,17 +1280,21 @@ Frees a space object from the tablespace the chain but does not delete them. There must not be any pending i/o's or flushes on the files. @return TRUE if success */ -UNIV_INTERN +static ibool fil_space_free( /*===========*/ - ulint id) /*!< in: space id */ + /* out: TRUE if success */ + ulint id, /* in: space id */ + ibool own_mutex) /* in: TRUE if own system->mutex */ { fil_space_t* space; fil_space_t* namespace; fil_node_t* fil_node; - mutex_enter(&fil_system->mutex); + if (!own_mutex) { + mutex_enter(&fil_system->mutex); + } space = fil_space_get_by_id(id); @@ -1326,7 +1341,9 @@ fil_space_free( ut_a(0 == UT_LIST_GET_LEN(space->chain)); - mutex_exit(&fil_system->mutex); + if (!own_mutex) { + mutex_exit(&fil_system->mutex); + } rw_lock_free(&(space->latch)); @@ -1586,6 +1603,8 @@ fil_close_all_files(void) space = UT_LIST_GET_FIRST(fil_system->space_list); while (space != NULL) { + fil_space_t* prev_space = space; + node = UT_LIST_GET_FIRST(space->chain); while (node != NULL) { @@ -1595,6 +1614,7 @@ fil_close_all_files(void) node = UT_LIST_GET_NEXT(chain, node); } space = UT_LIST_GET_NEXT(space_list, space); + fil_space_free(prev_space->id, TRUE); } mutex_exit(&fil_system->mutex); @@ -2226,7 +2246,7 @@ try_again: #endif /* printf("Deleting tablespace %s id %lu\n", space->name, id); */ - success = fil_space_free(id); + success = fil_space_free(id, FALSE); if (success) { success = os_file_delete(path); @@ -4753,3 +4773,26 @@ fil_page_get_type( return(mach_read_from_2(page + FIL_PAGE_TYPE)); } + +/******************************************************************** +Initializes the tablespace memory cache. */ +UNIV_INTERN +void +fil_close(void) +/*===========*/ +{ + /* The mutex should already have been freed. */ + ut_ad(fil_system->mutex.magic_n == 0); + + hash_table_free(fil_system->spaces); + + hash_table_free(fil_system->name_hash); + + ut_a(UT_LIST_GET_LEN(fil_system->LRU) == 0); + ut_a(UT_LIST_GET_LEN(fil_system->unflushed_spaces) == 0); + ut_a(UT_LIST_GET_LEN(fil_system->space_list) == 0); + + mem_free(fil_system); + + fil_system = NULL; +} === modified file 'storage/innodb_plugin/handler/ha_innodb.cc' --- a/storage/innodb_plugin/handler/ha_innodb.cc 2009-11-03 10:34:38 +0000 +++ b/storage/innodb_plugin/handler/ha_innodb.cc 2009-12-08 09:26:11 +0000 @@ -269,10 +269,10 @@ innobase_file_format_check_on_off( /************************************************************//** Validate the file format check config parameters, as a side effect it sets the srv_check_file_format_at_startup variable. -@return true if valid config value */ +@return the format_id if valid config value, otherwise, return -1 */ static -bool -innobase_file_format_check_validate( +int +innobase_file_format_validate_and_set( /*================================*/ const char* format_check); /*!< in: parameter value */ /****************************************************************//** @@ -785,11 +785,20 @@ convert_error_code_to_mysql( case DB_SUCCESS: return(0); + case DB_INTERRUPTED: + my_error(ER_QUERY_INTERRUPTED, MYF(0)); + /* fall through */ case DB_ERROR: default: return(-1); /* unspecified error */ case DB_DUPLICATE_KEY: + /* Be cautious with returning this error, since + mysql could re-enter the storage layer to get + duplicated key info, the operation requires a + valid table handle and/or transaction information, + which might not always be available in the error + handling stage. */ return(HA_ERR_FOUND_DUPP_KEY); case DB_FOREIGN_DUPLICATE_KEY: @@ -890,36 +899,6 @@ convert_error_code_to_mysql( } /*************************************************************//** -If you want to print a thd that is not associated with the current thread, -you must call this function before reserving the InnoDB kernel_mutex, to -protect MySQL from setting thd->query NULL. If you print a thd of the current -thread, we know that MySQL cannot modify thd->query, and it is not necessary -to call this. Call innobase_mysql_end_print_arbitrary_thd() after you release -the kernel_mutex. */ -extern "C" UNIV_INTERN -void -innobase_mysql_prepare_print_arbitrary_thd(void) -/*============================================*/ -{ - ut_ad(!mutex_own(&kernel_mutex)); - VOID(pthread_mutex_lock(&LOCK_thread_count)); -} - -/*************************************************************//** -Releases the mutex reserved by innobase_mysql_prepare_print_arbitrary_thd(). -In the InnoDB latching order, the mutex sits right above the -kernel_mutex. In debug builds, we assert that the kernel_mutex is -released before this function is invoked. */ -extern "C" UNIV_INTERN -void -innobase_mysql_end_print_arbitrary_thd(void) -/*========================================*/ -{ - ut_ad(!mutex_own(&kernel_mutex)); - VOID(pthread_mutex_unlock(&LOCK_thread_count)); -} - -/*************************************************************//** Prints info of a THD object (== user session thread) to the given file. */ extern "C" UNIV_INTERN void @@ -1707,15 +1686,19 @@ innobase_convert_identifier( FALSE=id is an UTF-8 string */ { char nz[NAME_LEN + 1]; +#if MYSQL_VERSION_ID >= 50141 + char nz2[NAME_LEN + 1 + EXPLAIN_FILENAME_MAX_EXTRA_LENGTH]; +#else /* MYSQL_VERSION_ID >= 50141 */ char nz2[NAME_LEN + 1 + sizeof srv_mysql50_table_name_prefix]; +#endif /* MYSQL_VERSION_ID >= 50141 */ const char* s = id; int q; if (file_id) { - /* Decode the table name. The filename_to_tablename() - function expects a NUL-terminated string. The input and - output strings buffers must not be shared. */ + /* Decode the table name. The MySQL function expects + a NUL-terminated string. The input and output strings + buffers must not be shared. */ if (UNIV_UNLIKELY(idlen > (sizeof nz) - 1)) { idlen = (sizeof nz) - 1; @@ -1725,7 +1708,13 @@ innobase_convert_identifier( nz[idlen] = 0; s = nz2; +#if MYSQL_VERSION_ID >= 50141 + idlen = explain_filename((THD*) thd, nz, nz2, sizeof nz2, + EXPLAIN_PARTITIONS_AS_COMMENT); + goto no_quote; +#else /* MYSQL_VERSION_ID >= 50141 */ idlen = filename_to_tablename(nz, nz2, sizeof nz2); +#endif /* MYSQL_VERSION_ID >= 50141 */ } /* See if the identifier needs to be quoted. */ @@ -1736,6 +1725,9 @@ innobase_convert_identifier( } if (q == EOF) { +#if MYSQL_VERSION_ID >= 50141 +no_quote: +#endif /* MYSQL_VERSION_ID >= 50141 */ if (UNIV_UNLIKELY(idlen > buflen)) { idlen = buflen; } @@ -2133,8 +2125,8 @@ mem_free_and_error: /* Did the user specify a format name that we support ? As a side effect it will update the variable srv_check_file_format_at_startup */ - if (!innobase_file_format_check_validate( - innobase_file_format_check)) { + if (innobase_file_format_validate_and_set( + innobase_file_format_check) < 0) { sql_print_error("InnoDB: invalid " "innodb_file_format_check value: " @@ -5225,8 +5217,10 @@ ha_innobase::change_active_index( prebuilt->index); if (UNIV_UNLIKELY(!prebuilt->index_usable)) { - sql_print_warning("InnoDB: insufficient history for index %u", - keynr); + push_warning_printf(user_thd, MYSQL_ERROR::WARN_LEVEL_WARN, + HA_ERR_TABLE_DEF_CHANGED, + "InnoDB: insufficient history for index %u", + keynr); /* The caller seems to ignore this. Thus, we must check this again in row_search_for_mysql(). */ DBUG_RETURN(2); @@ -5713,17 +5707,8 @@ create_table_def( /* First check whether the column to be added has a system reserved name. */ if (dict_col_name_is_reserved(field->field_name)){ - push_warning_printf( - (THD*) trx->mysql_thd, - MYSQL_ERROR::WARN_LEVEL_WARN, - ER_CANT_CREATE_TABLE, - "Error creating table '%s' with " - "column name '%s'. '%s' is a " - "reserved name. Please try to " - "re-create the table with a " - "different column name.", - table->name, (char*) field->field_name, - (char*) field->field_name); + my_error(ER_WRONG_COLUMN_NAME, MYF(0), + field->field_name); dict_mem_table_free(table); trx_commit_for_mysql(trx); @@ -5745,6 +5730,14 @@ create_table_def( error = row_create_table_for_mysql(table, trx); + if (error == DB_DUPLICATE_KEY) { + char buf[100]; + innobase_convert_identifier(buf, sizeof buf, + table_name, strlen(table_name), + trx->mysql_thd, TRUE); + my_error(ER_TABLE_EXISTS_ERROR, MYF(0), buf); + } + error_ret: error = convert_error_code_to_mysql(error, flags, NULL); @@ -6798,6 +6791,24 @@ ha_innobase::rename_table( innobase_commit_low(trx); trx_free_for_mysql(trx); + /* Add a special case to handle the Duplicated Key error + and return DB_ERROR instead. + This is to avoid a possible SIGSEGV error from mysql error + handling code. Currently, mysql handles the Duplicated Key + error by re-entering the storage layer and getting dup key + info by calling get_dup_key(). This operation requires a valid + table handle ('row_prebuilt_t' structure) which could no + longer be available in the error handling stage. The suggested + solution is to report a 'table exists' error message (since + the dup key error here is due to an existing table whose name + is the one we are trying to rename to) and return the generic + error code. */ + if (error == (int) DB_DUPLICATE_KEY) { + my_error(ER_TABLE_EXISTS_ERROR, MYF(0), to); + + error = DB_ERROR; + } + error = convert_error_code_to_mysql(error, 0, NULL); DBUG_RETURN(error); @@ -7348,11 +7359,15 @@ ha_innobase::check( ret = row_check_table_for_mysql(prebuilt); - if (ret == DB_SUCCESS) { + switch (ret) { + case DB_SUCCESS: return(HA_ADMIN_OK); + case DB_INTERRUPTED: + my_error(ER_QUERY_INTERRUPTED, MYF(0)); + return(-1); + default: + return(HA_ADMIN_CORRUPT); } - - return(HA_ADMIN_CORRUPT); } /*************************************************************//** @@ -7899,7 +7914,10 @@ ha_innobase::external_lock( ulong const tx_isolation = thd_tx_isolation(ha_thd()); if (tx_isolation <= ISO_READ_COMMITTED && binlog_format == BINLOG_FORMAT_STMT - && thd_binlog_filter_ok(thd)) +#if MYSQL_VERSION_ID > 50140 + && thd_binlog_filter_ok(thd) +#endif /* MYSQL_VERSION_ID > 50140 */ + ) { char buf[256]; my_snprintf(buf, sizeof(buf), @@ -9148,8 +9166,7 @@ innobase_xa_prepare( executing XA PREPARE and XA COMMIT commands. In this case we cannot know how many minutes or hours will be between XA PREPARE and XA COMMIT, and we don't want - to block for undefined period of time. - */ + to block for undefined period of time. */ pthread_mutex_lock(&prepare_commit_mutex); trx->active_trans = 2; } @@ -9491,25 +9508,24 @@ innobase_file_format_check_on_off( /************************************************************//** Validate the file format check config parameters, as a side effect it sets the srv_check_file_format_at_startup variable. -@return true if valid config value */ +@return the format_id if valid config value, otherwise, return -1 */ static -bool -innobase_file_format_check_validate( +int +innobase_file_format_validate_and_set( /*================================*/ const char* format_check) /*!< in: parameter value */ { uint format_id; - bool ret = true; format_id = innobase_file_format_name_lookup(format_check); if (format_id < DICT_TF_FORMAT_MAX + 1) { srv_check_file_format_at_startup = format_id; + + return((int) format_id); } else { - ret = false; + return(-1); } - - return(ret); } /*************************************************************//** @@ -9544,7 +9560,11 @@ innodb_file_format_name_validate( if (format_id <= DICT_TF_FORMAT_MAX) { - *static_cast<const char**>(save) = file_format_input; + /* Save a pointer to the name in the + 'file_format_name_map' constant array. */ + *static_cast<const char**>(save) = + trx_sys_file_format_id_to_name(format_id); + return(0); } } @@ -9607,6 +9627,7 @@ innodb_file_format_check_validate( const char* file_format_input; char buff[STRING_BUFFER_USUAL_SIZE]; int len = sizeof(buff); + int format_id; ut_a(save != NULL); ut_a(value != NULL); @@ -9619,24 +9640,35 @@ innodb_file_format_check_validate( message if they did so. */ if (innobase_file_format_check_on_off(file_format_input)) { - sql_print_warning( + push_warning_printf(thd, + MYSQL_ERROR::WARN_LEVEL_WARN, + ER_WRONG_ARGUMENTS, "InnoDB: invalid innodb_file_format_check " "value; on/off can only be set at startup or " "in the configuration file"); - } else if (innobase_file_format_check_validate( - file_format_input)) { + } else { + format_id = innobase_file_format_validate_and_set( + file_format_input); - *static_cast<const char**>(save) = file_format_input; + if (format_id >= 0) { + /* Save a pointer to the name in the + 'file_format_name_map' constant array. */ + *static_cast<const char**>(save) = + trx_sys_file_format_id_to_name( + (uint)format_id); - return(0); + return(0); - } else { - sql_print_warning( - "InnoDB: invalid innodb_file_format_check " - "value; can be any format up to %s " - "or its equivalent numeric id", - trx_sys_file_format_id_to_name( - DICT_TF_FORMAT_MAX)); + } else { + push_warning_printf(thd, + MYSQL_ERROR::WARN_LEVEL_WARN, + ER_WRONG_ARGUMENTS, + "InnoDB: invalid innodb_file_format_check " + "value; can be any format up to %s " + "or its equivalent numeric id", + trx_sys_file_format_id_to_name( + DICT_TF_FORMAT_MAX)); + } } } @@ -9906,12 +9938,15 @@ static MYSQL_SYSVAR_STR(file_format, inn innodb_file_format_name_validate, innodb_file_format_name_update, "Antelope"); +/* If a new file format is introduced, the file format +name needs to be updated accordingly. Please refer to +file_format_name_map[] defined in trx0sys.c for the next +file format name. */ static MYSQL_SYSVAR_STR(file_format_check, innobase_file_format_check, PLUGIN_VAR_OPCMDARG, "The highest file format in the tablespace.", innodb_file_format_check_validate, - innodb_file_format_check_update, - "on"); + innodb_file_format_check_update, "Barracuda"); static MYSQL_SYSVAR_ULONG(flush_log_at_trx_commit, srv_flush_log_at_trx_commit, PLUGIN_VAR_OPCMDARG, === modified file 'storage/innodb_plugin/handler/ha_innodb.h' --- a/storage/innodb_plugin/handler/ha_innodb.h 2009-11-03 10:07:51 +0000 +++ b/storage/innodb_plugin/handler/ha_innodb.h 2009-11-30 12:11:36 +0000 @@ -258,12 +258,14 @@ int thd_binlog_format(const MYSQL_THD th */ void thd_mark_transaction_to_rollback(MYSQL_THD thd, bool all); +#if MYSQL_VERSION_ID > 50140 /** Check if binary logging is filtered for thread's current db. @param thd Thread handle @retval 1 the query is not filtered, 0 otherwise. */ bool thd_binlog_filter_ok(const MYSQL_THD thd); +#endif /* MYSQL_VERSION_ID > 50140 */ } typedef struct trx_struct trx_t; === modified file 'storage/innodb_plugin/handler/handler0alter.cc' --- a/storage/innodb_plugin/handler/handler0alter.cc 2009-11-03 10:07:51 +0000 +++ b/storage/innodb_plugin/handler/handler0alter.cc 2009-11-30 13:42:26 +0000 @@ -765,10 +765,11 @@ err_exit: ut_ad(error == DB_SUCCESS); /* Commit the data dictionary transaction in order to release - the table locks on the system tables. Unfortunately, this - means that if MySQL crashes while creating a new primary key - inside row_merge_build_indexes(), indexed_table will not be - dropped on crash recovery. Thus, it will become orphaned. */ + the table locks on the system tables. This means that if + MySQL crashes while creating a new primary key inside + row_merge_build_indexes(), indexed_table will not be dropped + by trx_rollback_active(). It will have to be recovered or + dropped by the database administrator. */ trx_commit_for_mysql(trx); row_mysql_unlock_data_dictionary(trx); @@ -882,7 +883,9 @@ error: /* fall through */ default: if (new_primary) { - row_merge_drop_table(trx, indexed_table); + if (indexed_table != innodb_table) { + row_merge_drop_table(trx, indexed_table); + } } else { if (!dict_locked) { row_mysql_lock_data_dictionary(trx); === modified file 'storage/innodb_plugin/ibuf/ibuf0ibuf.c' --- a/storage/innodb_plugin/ibuf/ibuf0ibuf.c 2009-07-30 12:42:56 +0000 +++ b/storage/innodb_plugin/ibuf/ibuf0ibuf.c 2009-11-30 11:32:05 +0000 @@ -390,6 +390,27 @@ ibuf_count_set( #endif /******************************************************************//** +Closes insert buffer and frees the data structures. */ +UNIV_INTERN +void +ibuf_close(void) +/*============*/ +{ + mutex_free(&ibuf_pessimistic_insert_mutex); + memset(&ibuf_pessimistic_insert_mutex, + 0x0, sizeof(ibuf_pessimistic_insert_mutex)); + + mutex_free(&ibuf_mutex); + memset(&ibuf_mutex, 0x0, sizeof(ibuf_mutex)); + + mutex_free(&ibuf_bitmap_mutex); + memset(&ibuf_bitmap_mutex, 0x0, sizeof(ibuf_mutex)); + + mem_free(ibuf); + ibuf = NULL; +} + +/******************************************************************//** Updates the size information of the ibuf, assuming the segment size has not changed. */ static === modified file 'storage/innodb_plugin/include/btr0sea.h' --- a/storage/innodb_plugin/include/btr0sea.h 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/include/btr0sea.h 2009-11-30 11:32:05 +0000 @@ -41,6 +41,12 @@ void btr_search_sys_create( /*==================*/ ulint hash_size); /*!< in: hash index hash table size */ +/*****************************************************************//** +Frees the adaptive search system at a database shutdown. */ +UNIV_INTERN +void +btr_search_sys_free(void); +/*=====================*/ /********************************************************************//** Disable the adaptive hash search system and empty the index. */ === modified file 'storage/innodb_plugin/include/db0err.h' --- a/storage/innodb_plugin/include/db0err.h 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/include/db0err.h 2009-11-30 12:24:54 +0000 @@ -32,6 +32,7 @@ enum db_err { /* The following are error codes */ DB_ERROR, + DB_INTERRUPTED, DB_OUT_OF_MEMORY, DB_OUT_OF_FILE_SPACE, DB_LOCK_WAIT, === modified file 'storage/innodb_plugin/include/dict0dict.h' --- a/storage/innodb_plugin/include/dict0dict.h 2009-10-08 11:28:37 +0000 +++ b/storage/innodb_plugin/include/dict0dict.h 2009-11-30 11:32:05 +0000 @@ -1151,6 +1151,13 @@ void dict_ind_init(void); /*===============*/ +/**********************************************************************//** +Closes the data dictionary module. */ +UNIV_INTERN +void +dict_close(void); +/*============*/ + #ifndef UNIV_NONINL #include "dict0dict.ic" #endif === modified file 'storage/innodb_plugin/include/fil0fil.h' --- a/storage/innodb_plugin/include/fil0fil.h 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/include/fil0fil.h 2009-11-30 11:32:05 +0000 @@ -224,15 +224,6 @@ fil_space_create( 0 for uncompressed tablespaces */ ulint purpose);/*!< in: FIL_TABLESPACE, or FIL_LOG if log */ /*******************************************************************//** -Frees a space object from a the tablespace memory cache. Closes the files in -the chain but does not delete them. -@return TRUE if success */ -UNIV_INTERN -ibool -fil_space_free( -/*===========*/ - ulint id); /*!< in: space id */ -/*******************************************************************//** Returns the size of the space in pages. The tablespace must be cached in the memory cache. @return space size, 0 if space not found */ @@ -278,6 +269,12 @@ fil_init( ulint hash_size, /*!< in: hash table size */ ulint max_n_open); /*!< in: max number of open files */ /*******************************************************************//** +Initializes the tablespace memory cache. */ +UNIV_INTERN +void +fil_close(void); +/*===========*/ +/*******************************************************************//** Opens all log files and system tablespace data files. They stay open until the database server shutdown. This should be called at a server startup after the space objects for the log and the system tablespace have been created. The === modified file 'storage/innodb_plugin/include/ha_prototypes.h' --- a/storage/innodb_plugin/include/ha_prototypes.h 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/include/ha_prototypes.h 2009-12-01 10:38:40 +0000 @@ -153,28 +153,6 @@ get_innobase_type_from_mysql_type( const void* field) /*!< in: MySQL Field */ __attribute__((nonnull)); -/*************************************************************//** -If you want to print a thd that is not associated with the current thread, -you must call this function before reserving the InnoDB kernel_mutex, to -protect MySQL from setting thd->query NULL. If you print a thd of the current -thread, we know that MySQL cannot modify thd->query, and it is not necessary -to call this. Call innobase_mysql_end_print_arbitrary_thd() after you release -the kernel_mutex. */ -UNIV_INTERN -void -innobase_mysql_prepare_print_arbitrary_thd(void); -/*============================================*/ - -/*************************************************************//** -Releases the mutex reserved by innobase_mysql_prepare_print_arbitrary_thd(). -In the InnoDB latching order, the mutex sits right above the -kernel_mutex. In debug builds, we assert that the kernel_mutex is -released before this function is invoked. */ -UNIV_INTERN -void -innobase_mysql_end_print_arbitrary_thd(void); -/*========================================*/ - /******************************************************************//** Get the variable length bounds of the given character set. */ UNIV_INTERN === modified file 'storage/innodb_plugin/include/ibuf0ibuf.h' --- a/storage/innodb_plugin/include/ibuf0ibuf.h 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/include/ibuf0ibuf.h 2009-11-30 11:32:05 +0000 @@ -356,6 +356,12 @@ void ibuf_print( /*=======*/ FILE* file); /*!< in: file where to print */ +/******************************************************************//** +Closes insert buffer and frees the data structures. */ +UNIV_INTERN +void +ibuf_close(void); +/*============*/ #define IBUF_HEADER_PAGE_NO FSP_IBUF_HEADER_PAGE_NO #define IBUF_TREE_ROOT_PAGE_NO FSP_IBUF_TREE_ROOT_PAGE_NO === modified file 'storage/innodb_plugin/include/lock0lock.h' --- a/storage/innodb_plugin/include/lock0lock.h 2009-10-08 12:18:19 +0000 +++ b/storage/innodb_plugin/include/lock0lock.h 2009-11-30 11:32:05 +0000 @@ -59,6 +59,12 @@ lock_sys_create( /*============*/ ulint n_cells); /*!< in: number of slots in lock hash table */ /*********************************************************************//** +Closes the lock system at database shutdown. */ +UNIV_INTERN +void +lock_sys_close(void); +/*================*/ +/*********************************************************************//** Checks if some transaction has an implicit x-lock on a record in a clustered index. @return transaction which has the x-lock, or NULL */ === modified file 'storage/innodb_plugin/include/log0log.h' --- a/storage/innodb_plugin/include/log0log.h 2009-10-08 12:18:19 +0000 +++ b/storage/innodb_plugin/include/log0log.h 2009-11-30 11:32:05 +0000 @@ -572,6 +572,18 @@ UNIV_INTERN void log_refresh_stats(void); /*===================*/ +/********************************************************** +Shutdown the log system but do not release all the memory. */ +UNIV_INTERN +void +log_shutdown(void); +/*==============*/ +/********************************************************** +Free the log system data structures. */ +UNIV_INTERN +void +log_mem_free(void); +/*==============*/ extern log_t* log_sys; @@ -584,7 +596,7 @@ extern log_t* log_sys; #define LOG_RECOVER 98887331 /* The counting of lsn's starts from this value: this must be non-zero */ -#define LOG_START_LSN ((ib_uint64_t) (16 * OS_FILE_LOG_BLOCK_SIZE)) +#define LOG_START_LSN ((ib_uint64_t) (16 * OS_FILE_LOG_BLOCK_SIZE)) #define LOG_BUFFER_SIZE (srv_log_buffer_size * UNIV_PAGE_SIZE) #define LOG_ARCHIVE_BUF_SIZE (srv_log_buffer_size * UNIV_PAGE_SIZE / 4) @@ -721,9 +733,12 @@ struct log_group_struct{ ulint lsn_offset; /*!< the offset of the above lsn */ ulint n_pending_writes;/*!< number of currently pending flush writes for this log group */ + byte** file_header_bufs_ptr;/*!< unaligned buffers */ byte** file_header_bufs;/*!< buffers for each file header in the group */ +#ifdef UNIV_LOG_ARCHIVE /*-----------------------------*/ + byte** archive_file_header_bufs_ptr;/*!< unaligned buffers */ byte** archive_file_header_bufs;/*!< buffers for each file header in the group */ ulint archive_space_id;/*!< file space which @@ -742,10 +757,12 @@ struct log_group_struct{ completion function then sets the new value to ..._file_no */ ulint next_archived_offset; /*!< like the preceding field */ +#endif /* UNIV_LOG_ARCHIVE */ /*-----------------------------*/ ib_uint64_t scanned_lsn; /*!< used only in recovery: recovery scan succeeded up to this lsn in this log group */ + byte* checkpoint_buf_ptr;/*!< unaligned checkpoint header */ byte* checkpoint_buf; /*!< checkpoint header is written from this buffer to the group */ UT_LIST_NODE_T(log_group_t) @@ -763,6 +780,7 @@ struct log_struct{ #ifndef UNIV_HOTBACKUP mutex_t mutex; /*!< mutex protecting the log */ #endif /* !UNIV_HOTBACKUP */ + byte* buf_ptr; /* unaligned log buffer */ byte* buf; /*!< log buffer */ ulint buf_size; /*!< log buffer size in bytes */ ulint max_buf_free; /*!< recommended maximum value of @@ -899,6 +917,7 @@ struct log_struct{ should wait for this without owning the log mutex */ #endif /* !UNIV_HOTBACKUP */ + byte* checkpoint_buf_ptr;/* unaligned checkpoint header */ byte* checkpoint_buf; /*!< checkpoint header is read to this buffer */ /* @} */ === modified file 'storage/innodb_plugin/include/log0recv.h' --- a/storage/innodb_plugin/include/log0recv.h 2009-10-09 14:13:15 +0000 +++ b/storage/innodb_plugin/include/log0recv.h 2009-11-30 11:32:05 +0000 @@ -239,6 +239,18 @@ UNIV_INTERN void recv_sys_create(void); /*=================*/ +/**********************************************************//** +Release recovery system mutexes. */ +UNIV_INTERN +void +recv_sys_close(void); +/*================*/ +/********************************************************//** +Frees the recovery system memory. */ +UNIV_INTERN +void +recv_sys_mem_free(void); +/*===================*/ /********************************************************//** Inits the recovery system for a recovery operation. */ UNIV_INTERN @@ -246,6 +258,12 @@ void recv_sys_init( /*==========*/ ulint available_memory); /*!< in: available memory in bytes */ +/********************************************************//** +Reset the state of the recovery system variables. */ +UNIV_INTERN +void +recv_sys_var_init(void); +/*===================*/ /*******************************************************************//** Empties the hash table of stored log records, applying them to appropriate pages. */ === modified file 'storage/innodb_plugin/include/mem0mem.h' --- a/storage/innodb_plugin/include/mem0mem.h 2009-07-30 12:42:56 +0000 +++ b/storage/innodb_plugin/include/mem0mem.h 2009-11-30 11:32:05 +0000 @@ -82,6 +82,13 @@ void mem_init( /*=====*/ ulint size); /*!< in: common pool size in bytes */ +/******************************************************************//** +Closes the memory system. */ +UNIV_INTERN +void +mem_close(void); +/*===========*/ + /**************************************************************//** Use this macro instead of the corresponding function! Macro for memory heap creation. */ === modified file 'storage/innodb_plugin/include/mem0pool.h' --- a/storage/innodb_plugin/include/mem0pool.h 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/include/mem0pool.h 2009-11-30 11:32:05 +0000 @@ -62,6 +62,13 @@ mem_pool_create( /*============*/ ulint size); /*!< in: pool size in bytes */ /********************************************************************//** +Frees a memory pool. */ +UNIV_INTERN +void +mem_pool_free( +/*==========*/ + mem_pool_t* pool); /*!< in, own: memory pool */ +/********************************************************************//** Allocates memory from a pool. NOTE: This low-level function should only be used in mem0mem.*! @return own: allocated memory buffer */ === modified file 'storage/innodb_plugin/include/os0file.h' --- a/storage/innodb_plugin/include/os0file.h 2009-11-03 09:59:06 +0000 +++ b/storage/innodb_plugin/include/os0file.h 2009-11-30 12:04:09 +0000 @@ -158,6 +158,7 @@ log. */ #define OS_FILE_SHARING_VIOLATION 76 #define OS_FILE_ERROR_NOT_SPECIFIED 77 #define OS_FILE_INSUFFICIENT_RESOURCE 78 +#define OS_FILE_OPERATION_ABORTED 79 /* @} */ /** Types for aio operations @{ */ @@ -620,6 +621,13 @@ os_aio_init( ulint n_write_segs, /*<! in: number of writer threads */ ulint n_slots_sync); /*<! in: number of slots in the sync aio array */ +/*********************************************************************** +Frees the asynchronous io system. */ +UNIV_INTERN +void +os_aio_free(void); +/*=============*/ + /*******************************************************************//** Requests an asynchronous i/o operation. @return TRUE if request was queued successfully, FALSE if fail */ === modified file 'storage/innodb_plugin/include/pars0pars.h' --- a/storage/innodb_plugin/include/pars0pars.h 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/include/pars0pars.h 2009-11-30 11:32:05 +0000 @@ -583,6 +583,12 @@ pars_info_get_bound_id( pars_info_t* info, /*!< in: info struct */ const char* name); /*!< in: bound id name to find */ +/******************************************************************//** +Release any resources used by the lexer. */ +UNIV_INTERN +void +pars_lexer_close(void); +/*==================*/ /** Extra information supplied for pars_sql(). */ struct pars_info_struct { === modified file 'storage/innodb_plugin/include/srv0srv.h' --- a/storage/innodb_plugin/include/srv0srv.h 2009-10-08 11:28:37 +0000 +++ b/storage/innodb_plugin/include/srv0srv.h 2009-11-30 11:32:05 +0000 @@ -411,7 +411,7 @@ void srv_init(void); /*==========*/ /*********************************************************************//** -Frees the OS fast mutex created in srv_boot(). */ +Frees the data structures created in srv_init(). */ UNIV_INTERN void srv_free(void); === modified file 'storage/innodb_plugin/include/thr0loc.h' --- a/storage/innodb_plugin/include/thr0loc.h 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/include/thr0loc.h 2009-11-30 11:32:05 +0000 @@ -39,6 +39,12 @@ UNIV_INTERN void thr_local_init(void); /*================*/ + /****************************************************************//** +Close the thread local storage module. */ +UNIV_INTERN +void +thr_local_close(void); +/*=================*/ /*******************************************************************//** Creates a local storage struct for the calling new thread. */ UNIV_INTERN === modified file 'storage/innodb_plugin/include/trx0i_s.h' --- a/storage/innodb_plugin/include/trx0i_s.h 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/include/trx0i_s.h 2009-11-30 11:32:05 +0000 @@ -141,6 +141,13 @@ void trx_i_s_cache_init( /*===============*/ trx_i_s_cache_t* cache); /*!< out: cache to init */ +/*******************************************************************//** +Free the INFORMATION SCHEMA trx related cache. */ +UNIV_INTERN +void +trx_i_s_cache_free( +/*===============*/ + trx_i_s_cache_t* cache); /*!< in/out: cache to free */ /*******************************************************************//** Issue a shared/read lock on the tables cache. */ === modified file 'storage/innodb_plugin/include/trx0purge.h' --- a/storage/innodb_plugin/include/trx0purge.h 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/include/trx0purge.h 2009-11-30 11:32:05 +0000 @@ -71,6 +71,12 @@ void trx_purge_sys_create(void); /*======================*/ /********************************************************************//** +Frees the global purge system control structure. */ +UNIV_INTERN +void +trx_purge_sys_close(void); +/*======================*/ +/************************************************************************ Adds the update undo log as the first log in the history list. Removes the update undo log segment from the rseg slot if it is too big for reuse. */ UNIV_INTERN === modified file 'storage/innodb_plugin/include/trx0rseg.h' --- a/storage/innodb_plugin/include/trx0rseg.h 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/include/trx0rseg.h 2009-11-30 11:32:05 +0000 @@ -125,6 +125,13 @@ trx_rseg_create( ulint max_size, /*!< in: max size in pages */ ulint* id, /*!< out: rseg id */ mtr_t* mtr); /*!< in: mtr */ +/*************************************************************************** +Free's an instance of the rollback segment in memory. */ +UNIV_INTERN +void +trx_rseg_mem_free( +/*==============*/ + trx_rseg_t* rseg); /* in, own: instance to free */ /* Number of undo log slots in a rollback segment file copy */ === modified file 'storage/innodb_plugin/include/trx0sys.h' --- a/storage/innodb_plugin/include/trx0sys.h 2009-07-30 12:42:56 +0000 +++ b/storage/innodb_plugin/include/trx0sys.h 2009-11-30 11:32:05 +0000 @@ -334,6 +334,12 @@ void trx_sys_file_format_tag_init(void); /*==============================*/ /*****************************************************************//** +Shutdown/Close the transaction system. */ +UNIV_INTERN +void +trx_sys_close(void); +/*===============*/ +/*****************************************************************//** Get the name representation of the file format from its id. @return pointer to the name */ UNIV_INTERN === modified file 'storage/innodb_plugin/include/trx0trx.h' --- a/storage/innodb_plugin/include/trx0trx.h 2009-10-08 13:05:59 +0000 +++ b/storage/innodb_plugin/include/trx0trx.h 2009-12-01 10:38:40 +0000 @@ -338,9 +338,7 @@ trx_commit_step( /**********************************************************************//** Prints info about a transaction to the given file. The caller must own the -kernel mutex and must have called -innobase_mysql_prepare_print_arbitrary_thd(), unless he knows that MySQL -or InnoDB cannot meanwhile change the info printed here. */ +kernel mutex. */ UNIV_INTERN void trx_print( === modified file 'storage/innodb_plugin/include/trx0undo.h' --- a/storage/innodb_plugin/include/trx0undo.h 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/include/trx0undo.h 2009-11-30 11:32:05 +0000 @@ -333,6 +333,13 @@ trx_undo_parse_discard_latest( byte* end_ptr,/*!< in: buffer end */ page_t* page, /*!< in: page or NULL */ mtr_t* mtr); /*!< in: mtr or NULL */ +/************************************************************************ +Frees an undo log memory copy. */ +UNIV_INTERN +void +trx_undo_mem_free( +/*==============*/ + trx_undo_t* undo); /* in: the undo object to be freed */ /* Types of an undo log segment */ #define TRX_UNDO_INSERT 1 /* contains undo entries for inserts */ === modified file 'storage/innodb_plugin/include/univ.i' --- a/storage/innodb_plugin/include/univ.i 2009-11-03 10:26:39 +0000 +++ b/storage/innodb_plugin/include/univ.i 2009-11-30 13:13:34 +0000 @@ -46,7 +46,7 @@ Created 1/20/1994 Heikki Tuuri #define INNODB_VERSION_MAJOR 1 #define INNODB_VERSION_MINOR 0 -#define INNODB_VERSION_BUGFIX 5 +#define INNODB_VERSION_BUGFIX 6 /* The following is the InnoDB version as shown in SELECT plugin_version FROM information_schema.plugins; === modified file 'storage/innodb_plugin/include/usr0sess.h' --- a/storage/innodb_plugin/include/usr0sess.h 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/include/usr0sess.h 2009-11-30 11:32:05 +0000 @@ -44,14 +44,12 @@ sess_t* sess_open(void); /*============*/ /*********************************************************************//** -Closes a session, freeing the memory occupied by it, if it is in a state -where it should be closed. -@return TRUE if closed */ +Closes a session, freeing the memory occupied by it. */ UNIV_INTERN -ibool -sess_try_close( -/*===========*/ - sess_t* sess); /*!< in, own: session object */ +void +sess_close( +/*=======*/ + sess_t* sess); /* in, own: session object */ /* The session handle. All fields are protected by the kernel mutex */ struct sess_struct{ === modified file 'storage/innodb_plugin/lock/lock0lock.c' --- a/storage/innodb_plugin/lock/lock0lock.c 2009-10-09 14:13:15 +0000 +++ b/storage/innodb_plugin/lock/lock0lock.c 2009-12-01 10:38:40 +0000 @@ -578,6 +578,23 @@ lock_sys_create( } /*********************************************************************//** +Closes the lock system at database shutdown. */ +UNIV_INTERN +void +lock_sys_close(void) +/*================*/ +{ + if (lock_latest_err_file != NULL) { + fclose(lock_latest_err_file); + lock_latest_err_file = NULL; + } + + hash_table_free(lock_sys->rec_hash); + mem_free(lock_sys); + lock_sys = NULL; +} + +/*********************************************************************//** Gets the size of a lock struct. @return size in bytes */ UNIV_INTERN @@ -4307,11 +4324,6 @@ lock_print_info_summary( /*====================*/ FILE* file) /*!< in: file where to print */ { - /* We must protect the MySQL thd->query field with a MySQL mutex, and - because the MySQL mutex must be reserved before the kernel_mutex of - InnoDB, we call innobase_mysql_prepare_print_arbitrary_thd() here. */ - - innobase_mysql_prepare_print_arbitrary_thd(); lock_mutex_enter_kernel(); if (lock_deadlock_found) { @@ -4394,7 +4406,6 @@ loop: if (trx == NULL) { lock_mutex_exit_kernel(); - innobase_mysql_end_print_arbitrary_thd(); ut_ad(lock_validate()); @@ -4478,7 +4489,6 @@ loop: } lock_mutex_exit_kernel(); - innobase_mysql_end_print_arbitrary_thd(); mtr_start(&mtr); @@ -4489,7 +4499,6 @@ loop: load_page_first = FALSE; - innobase_mysql_prepare_print_arbitrary_thd(); lock_mutex_enter_kernel(); goto loop; === modified file 'storage/innodb_plugin/log/log0log.c' --- a/storage/innodb_plugin/log/log0log.c 2009-10-09 14:13:15 +0000 +++ b/storage/innodb_plugin/log/log0log.c 2009-11-30 11:32:05 +0000 @@ -771,8 +771,6 @@ void log_init(void) /*==========*/ { - byte* buf; - log_sys = mem_alloc(sizeof(log_t)); mutex_create(&log_sys->mutex, SYNC_LOG); @@ -787,8 +785,8 @@ log_init(void) ut_a(LOG_BUFFER_SIZE >= 16 * OS_FILE_LOG_BLOCK_SIZE); ut_a(LOG_BUFFER_SIZE >= 4 * UNIV_PAGE_SIZE); - buf = mem_alloc(LOG_BUFFER_SIZE + OS_FILE_LOG_BLOCK_SIZE); - log_sys->buf = ut_align(buf, OS_FILE_LOG_BLOCK_SIZE); + log_sys->buf_ptr = mem_alloc(LOG_BUFFER_SIZE + OS_FILE_LOG_BLOCK_SIZE); + log_sys->buf = ut_align(log_sys->buf_ptr, OS_FILE_LOG_BLOCK_SIZE); log_sys->buf_size = LOG_BUFFER_SIZE; @@ -833,9 +831,9 @@ log_init(void) rw_lock_create(&log_sys->checkpoint_lock, SYNC_NO_ORDER_CHECK); - log_sys->checkpoint_buf - = ut_align(mem_alloc(2 * OS_FILE_LOG_BLOCK_SIZE), - OS_FILE_LOG_BLOCK_SIZE); + log_sys->checkpoint_buf_ptr = mem_alloc(2 * OS_FILE_LOG_BLOCK_SIZE); + log_sys->checkpoint_buf = ut_align(log_sys->checkpoint_buf_ptr, + OS_FILE_LOG_BLOCK_SIZE); memset(log_sys->checkpoint_buf, '\0', OS_FILE_LOG_BLOCK_SIZE); /*----------------------------*/ @@ -918,23 +916,33 @@ log_group_init( group->lsn_offset = LOG_FILE_HDR_SIZE; group->n_pending_writes = 0; + group->file_header_bufs_ptr = mem_alloc(sizeof(byte*) * n_files); group->file_header_bufs = mem_alloc(sizeof(byte*) * n_files); #ifdef UNIV_LOG_ARCHIVE + group->archive_file_header_bufs_ptr = mem_alloc( + sizeof(byte*) * n_files); group->archive_file_header_bufs = mem_alloc(sizeof(byte*) * n_files); #endif /* UNIV_LOG_ARCHIVE */ for (i = 0; i < n_files; i++) { - *(group->file_header_bufs + i) = ut_align( - mem_alloc(LOG_FILE_HDR_SIZE + OS_FILE_LOG_BLOCK_SIZE), + group->file_header_bufs_ptr[i] = mem_alloc( + LOG_FILE_HDR_SIZE + OS_FILE_LOG_BLOCK_SIZE); + + group->file_header_bufs[i] = ut_align( + group->file_header_bufs_ptr[i], OS_FILE_LOG_BLOCK_SIZE); memset(*(group->file_header_bufs + i), '\0', LOG_FILE_HDR_SIZE); #ifdef UNIV_LOG_ARCHIVE - *(group->archive_file_header_bufs + i) = ut_align( - mem_alloc(LOG_FILE_HDR_SIZE + OS_FILE_LOG_BLOCK_SIZE), + group->archive_file_header_bufs_ptr[i] = mem_alloc( + LOG_FILE_HDR_SIZE + OS_FILE_LOG_BLOCK_SIZE); + + group->archive_file_header_bufs[i] = ut_align( + group->archive_file_header_bufs_ptr[i], OS_FILE_LOG_BLOCK_SIZE); + memset(*(group->archive_file_header_bufs + i), '\0', LOG_FILE_HDR_SIZE); #endif /* UNIV_LOG_ARCHIVE */ @@ -947,8 +955,9 @@ log_group_init( group->archived_offset = 0; #endif /* UNIV_LOG_ARCHIVE */ - group->checkpoint_buf = ut_align( - mem_alloc(2 * OS_FILE_LOG_BLOCK_SIZE), OS_FILE_LOG_BLOCK_SIZE); + group->checkpoint_buf_ptr = mem_alloc(2 * OS_FILE_LOG_BLOCK_SIZE); + group->checkpoint_buf = ut_align(group->checkpoint_buf_ptr, + OS_FILE_LOG_BLOCK_SIZE); memset(group->checkpoint_buf, '\0', OS_FILE_LOG_BLOCK_SIZE); @@ -3364,4 +3373,95 @@ log_refresh_stats(void) log_sys->n_log_ios_old = log_sys->n_log_ios; log_sys->last_printout_time = time(NULL); } + +/********************************************************************** +Closes a log group. */ +static +void +log_group_close( +/*===========*/ + log_group_t* group) /* in,own: log group to close */ +{ + ulint i; + + for (i = 0; i < group->n_files; i++) { + mem_free(group->file_header_bufs_ptr[i]); +#ifdef UNIV_LOG_ARCHIVE + mem_free(group->archive_file_header_bufs_ptr[i]); +#endif /* UNIV_LOG_ARCHIVE */ + } + + mem_free(group->file_header_bufs_ptr); + mem_free(group->file_header_bufs); + +#ifdef UNIV_LOG_ARCHIVE + mem_free(group->archive_file_header_bufs_ptr); + mem_free(group->archive_file_header_bufs); +#endif /* UNIV_LOG_ARCHIVE */ + + mem_free(group->checkpoint_buf_ptr); + + mem_free(group); +} + +/********************************************************** +Shutdown the log system but do not release all the memory. */ +UNIV_INTERN +void +log_shutdown(void) +/*==============*/ +{ + log_group_t* group; + + group = UT_LIST_GET_FIRST(log_sys->log_groups); + + while (UT_LIST_GET_LEN(log_sys->log_groups) > 0) { + log_group_t* prev_group = group; + + group = UT_LIST_GET_NEXT(log_groups, group); + UT_LIST_REMOVE(log_groups, log_sys->log_groups, prev_group); + + log_group_close(prev_group); + } + + mem_free(log_sys->buf_ptr); + log_sys->buf_ptr = NULL; + log_sys->buf = NULL; + mem_free(log_sys->checkpoint_buf_ptr); + log_sys->checkpoint_buf_ptr = NULL; + log_sys->checkpoint_buf = NULL; + + os_event_free(log_sys->no_flush_event); + os_event_free(log_sys->one_flushed_event); + + rw_lock_free(&log_sys->checkpoint_lock); + + mutex_free(&log_sys->mutex); + +#ifdef UNIV_LOG_ARCHIVE + rw_lock_free(&log_sys->archive_lock); + os_event_create(log_sys->archiving_on); +#endif /* UNIV_LOG_ARCHIVE */ + +#ifdef UNIV_LOG_DEBUG + recv_sys_debug_free(); +#endif + + recv_sys_close(); +} + +/********************************************************** +Free the log system data structures. */ +UNIV_INTERN +void +log_mem_free(void) +/*==============*/ +{ + if (log_sys != NULL) { + recv_sys_mem_free(); + mem_free(log_sys); + + log_sys = NULL; + } +} #endif /* !UNIV_HOTBACKUP */ === modified file 'storage/innodb_plugin/log/log0recv.c' --- a/storage/innodb_plugin/log/log0recv.c 2009-10-09 14:13:15 +0000 +++ b/storage/innodb_plugin/log/log0recv.c 2009-11-30 11:32:05 +0000 @@ -69,15 +69,15 @@ UNIV_INTERN recv_sys_t* recv_sys = NULL; /** TRUE when applying redo log records during crash recovery; FALSE otherwise. Note that this is FALSE while a background thread is rolling back incomplete transactions. */ -UNIV_INTERN ibool recv_recovery_on = FALSE; +UNIV_INTERN ibool recv_recovery_on; #ifdef UNIV_LOG_ARCHIVE /** TRUE when applying redo log records from an archived log file */ -UNIV_INTERN ibool recv_recovery_from_backup_on = FALSE; +UNIV_INTERN ibool recv_recovery_from_backup_on; #endif /* UNIV_LOG_ARCHIVE */ #ifndef UNIV_HOTBACKUP /** TRUE when recv_init_crash_recovery() has been called. */ -UNIV_INTERN ibool recv_needed_recovery = FALSE; +UNIV_INTERN ibool recv_needed_recovery; # ifdef UNIV_DEBUG /** TRUE if writing to the redo log (mtr_commit) is forbidden. Protected by log_sys->mutex. */ @@ -87,7 +87,7 @@ UNIV_INTERN ibool recv_no_log_write = FA /** TRUE if buf_page_is_corrupted() should check if the log sequence number (FIL_PAGE_LSN) is in the future. Initially FALSE, and set by recv_recovery_from_checkpoint_start_func(). */ -UNIV_INTERN ibool recv_lsn_checks_on = FALSE; +UNIV_INTERN ibool recv_lsn_checks_on; /** There are two conditions under which we scan the logs, the first is normal startup and the second is when we do a recovery from an @@ -97,7 +97,7 @@ startup. If we find log entries that wer we know that the server was not cleanly shutdown. We must then initialize the crash recovery environment before attempting to store these entries in the log hash table. */ -static ibool recv_log_scan_is_startup_type = FALSE; +static ibool recv_log_scan_is_startup_type; /** If the following is TRUE, the buffer pool file pages must be invalidated after recovery and no ibuf operations are allowed; this becomes TRUE if @@ -108,7 +108,7 @@ buffer pool before the pages have been r TRUE means that recovery is running and no operations on the log files are allowed yet: the variable name is misleading. */ -UNIV_INTERN ibool recv_no_ibuf_operations = FALSE; +UNIV_INTERN ibool recv_no_ibuf_operations; /** TRUE when the redo log is being backed up */ # define recv_is_making_a_backup FALSE /** TRUE when recovering from a backed up redo log file */ @@ -116,30 +116,30 @@ UNIV_INTERN ibool recv_no_ibuf_operation #else /* !UNIV_HOTBACKUP */ # define recv_needed_recovery FALSE /** TRUE when the redo log is being backed up */ -UNIV_INTERN ibool recv_is_making_a_backup = FALSE; +UNIV_INTERN ibool recv_is_making_a_backup = FALSE; /** TRUE when recovering from a backed up redo log file */ UNIV_INTERN ibool recv_is_from_backup = FALSE; # define buf_pool_get_curr_size() (5 * 1024 * 1024) #endif /* !UNIV_HOTBACKUP */ /** The following counter is used to decide when to print info on log scan */ -static ulint recv_scan_print_counter = 0; +static ulint recv_scan_print_counter; /** The type of the previous parsed redo log record */ -static ulint recv_previous_parsed_rec_type = 999999; +static ulint recv_previous_parsed_rec_type; /** The offset of the previous parsed redo log record */ -static ulint recv_previous_parsed_rec_offset = 0; +static ulint recv_previous_parsed_rec_offset; /** The 'multi' flag of the previous parsed redo log record */ -static ulint recv_previous_parsed_rec_is_multi = 0; +static ulint recv_previous_parsed_rec_is_multi; /** Maximum page number encountered in the redo log */ -UNIV_INTERN ulint recv_max_parsed_page_no = 0; +UNIV_INTERN ulint recv_max_parsed_page_no; /** This many frames must be left free in the buffer pool when we scan the log and store the scanned log records in the buffer pool: we will use these free frames to read in pages when we start applying the log records to the database. */ -UNIV_INTERN ulint recv_n_pool_free_frames = 256; +UNIV_INTERN ulint recv_n_pool_free_frames; /** The maximum lsn we see for a page during the recovery process. If this is bigger than the lsn we are able to scan up to, that is an indication that @@ -170,7 +170,8 @@ recv_sys_create(void) return; } - recv_sys = mem_alloc(sizeof(recv_sys_t)); + recv_sys = mem_alloc(sizeof(*recv_sys)); + memset(recv_sys, 0x0, sizeof(*recv_sys)); mutex_create(&recv_sys->mutex, SYNC_RECV); @@ -179,6 +180,106 @@ recv_sys_create(void) } /********************************************************//** +Release recovery system mutexes. */ +UNIV_INTERN +void +recv_sys_close(void) +/*================*/ +{ + if (recv_sys != NULL) { + if (recv_sys->addr_hash != NULL) { + hash_table_free(recv_sys->addr_hash); + } + + if (recv_sys->heap != NULL) { + mem_heap_free(recv_sys->heap); + } + + if (recv_sys->buf != NULL) { + ut_free(recv_sys->buf); + } + + if (recv_sys->last_block_buf_start != NULL) { + mem_free(recv_sys->last_block_buf_start); + } + + mutex_free(&recv_sys->mutex); + + mem_free(recv_sys); + recv_sys = NULL; + } +} + +/********************************************************//** +Frees the recovery system memory. */ +UNIV_INTERN +void +recv_sys_mem_free(void) +/*===================*/ +{ + if (recv_sys != NULL) { + if (recv_sys->addr_hash != NULL) { + hash_table_free(recv_sys->addr_hash); + } + + if (recv_sys->heap != NULL) { + mem_heap_free(recv_sys->heap); + } + + if (recv_sys->buf != NULL) { + ut_free(recv_sys->buf); + } + + if (recv_sys->last_block_buf_start != NULL) { + mem_free(recv_sys->last_block_buf_start); + } + + mem_free(recv_sys); + recv_sys = NULL; + } +} + +/************************************************************ +Reset the state of the recovery system variables. */ +UNIV_INTERN +void +recv_sys_var_init(void) +/*===================*/ +{ + recv_lsn_checks_on = FALSE; + + recv_n_pool_free_frames = 256; + + recv_recovery_on = FALSE; + +#ifdef UNIV_LOG_ARCHIVE + recv_recovery_from_backup_on = FALSE; +#endif /* UNIV_LOG_ARCHIVE */ + + recv_needed_recovery = FALSE; + + recv_lsn_checks_on = FALSE; + + recv_log_scan_is_startup_type = FALSE; + + recv_no_ibuf_operations = FALSE; + + recv_scan_print_counter = 0; + + recv_previous_parsed_rec_type = 999999; + + recv_previous_parsed_rec_offset = 0; + + recv_previous_parsed_rec_is_multi = 0; + + recv_max_parsed_page_no = 0; + + recv_n_pool_free_frames = 256; + + recv_max_page_lsn = 0; +} + +/************************************************************ Inits the recovery system for a recovery operation. */ UNIV_INTERN void @@ -253,8 +354,8 @@ recv_sys_empty_hash(void) Frees the recovery system. */ static void -recv_sys_free(void) -/*===============*/ +recv_sys_debug_free(void) +/*=====================*/ { mutex_enter(&(recv_sys->mutex)); @@ -263,8 +364,10 @@ recv_sys_free(void) ut_free(recv_sys->buf); mem_free(recv_sys->last_block_buf_start); - recv_sys->addr_hash = NULL; + recv_sys->buf = NULL; recv_sys->heap = NULL; + recv_sys->addr_hash = NULL; + recv_sys->last_block_buf_start = NULL; mutex_exit(&(recv_sys->mutex)); } @@ -3149,7 +3252,7 @@ recv_recovery_from_checkpoint_finish(voi recv_recovery_on = FALSE; #ifndef UNIV_LOG_DEBUG - recv_sys_free(); + recv_sys_debug_free(); #endif /* Roll back any recovered data dictionary transactions, so that the data dictionary tables will be free of any locks. === modified file 'storage/innodb_plugin/mem/mem0dbg.c' --- a/storage/innodb_plugin/mem/mem0dbg.c 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/mem/mem0dbg.c 2009-11-30 11:32:05 +0000 @@ -170,6 +170,17 @@ mem_init( mem_comm_pool = mem_pool_create(size); } + +/******************************************************************//** +Closes the memory system. */ +UNIV_INTERN +void +mem_close(void) +/*===========*/ +{ + mem_pool_free(mem_comm_pool); + mem_comm_pool = NULL; +} #endif /* !UNIV_HOTBACKUP */ #ifdef UNIV_MEM_DEBUG === modified file 'storage/innodb_plugin/mem/mem0pool.c' --- a/storage/innodb_plugin/mem/mem0pool.c 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/mem/mem0pool.c 2009-11-30 11:32:05 +0000 @@ -261,6 +261,18 @@ mem_pool_create( } /********************************************************************//** +Frees a memory pool. */ +UNIV_INTERN +void +mem_pool_free( +/*==========*/ + mem_pool_t* pool) /*!< in, own: memory pool */ +{ + ut_free(pool->buf); + ut_free(pool); +} + +/********************************************************************//** Fills the specified free list. @return TRUE if we were able to insert a block to the free list */ static === modified file 'storage/innodb_plugin/os/os0file.c' --- a/storage/innodb_plugin/os/os0file.c 2009-11-03 09:59:31 +0000 +++ b/storage/innodb_plugin/os/os0file.c 2009-11-30 12:04:09 +0000 @@ -323,6 +323,13 @@ os_file_get_last_error( "InnoDB: The error means that there are no" " sufficient system resources or quota to" " complete the operation.\n"); + } else if (err == ERROR_OPERATION_ABORTED) { + fprintf(stderr, + "InnoDB: The error means that the I/O" + " operation has been aborted\n" + "InnoDB: because of either a thread exit" + " or an application request.\n" + "InnoDB: Retry attempt is made.\n"); } else { fprintf(stderr, "InnoDB: Some operating system error numbers" @@ -347,6 +354,8 @@ os_file_get_last_error( } else if (err == ERROR_WORKING_SET_QUOTA || err == ERROR_NO_SYSTEM_RESOURCES) { return(OS_FILE_INSUFFICIENT_RESOURCE); + } else if (err == ERROR_OPERATION_ABORTED) { + return(OS_FILE_OPERATION_ABORTED); } else { return(100 + err); } @@ -469,6 +478,10 @@ os_file_handle_error_cond_exit( os_thread_sleep(100000); /* 100 ms */ return(TRUE); + } else if (err == OS_FILE_OPERATION_ABORTED) { + + os_thread_sleep(100000); /* 100 ms */ + return(TRUE); } else { if (name) { fprintf(stderr, "InnoDB: File name %s\n", name); @@ -3029,6 +3042,34 @@ os_aio_array_create( return(array); } +/************************************************************************//** +Frees an aio wait array. */ +static +void +os_aio_array_free( +/*==============*/ + os_aio_array_t* array) /*!< in, own: array to free */ +{ +#ifdef WIN_ASYNC_IO + ulint i; + + for (i = 0; i < array->n_slots; i++) { + os_aio_slot_t* slot = os_aio_array_get_nth_slot(array, i); + os_event_free(slot->event); + } +#endif /* WIN_ASYNC_IO */ + +#ifdef __WIN__ + ut_free(array->native_events); +#endif /* __WIN__ */ + os_mutex_free(array->mutex); + os_event_free(array->not_full); + os_event_free(array->is_empty); + + ut_free(array->slots); + ut_free(array); +} + /*********************************************************************** Initializes the asynchronous io system. Creates one array each for ibuf and log i/o. Also creates one array each for read and write where each @@ -3099,6 +3140,35 @@ os_aio_init( } +/*********************************************************************** +Frees the asynchronous io system. */ +UNIV_INTERN +void +os_aio_free(void) +/*=============*/ +{ + ulint i; + + os_aio_array_free(os_aio_ibuf_array); + os_aio_ibuf_array = NULL; + os_aio_array_free(os_aio_log_array); + os_aio_log_array = NULL; + os_aio_array_free(os_aio_read_array); + os_aio_read_array = NULL; + os_aio_array_free(os_aio_write_array); + os_aio_write_array = NULL; + os_aio_array_free(os_aio_sync_array); + os_aio_sync_array = NULL; + + for (i = 0; i < os_aio_n_segments; i++) { + os_event_free(os_aio_segment_wait_events[i]); + } + + ut_free(os_aio_segment_wait_events); + os_aio_segment_wait_events = 0; + os_aio_n_segments = 0; +} + #ifdef WIN_ASYNC_IO /************************************************************************//** Wakes up all async i/o threads in the array in Windows async i/o at @@ -3709,6 +3779,7 @@ os_aio_windows_handle( ibool ret_val; BOOL ret; DWORD len; + BOOL retry = FALSE; if (segment == ULINT_UNDEFINED) { array = os_aio_sync_array; @@ -3762,14 +3833,52 @@ os_aio_windows_handle( ut_a(TRUE == os_file_flush(slot->file)); } #endif /* UNIV_DO_FLUSH */ + } else if (os_file_handle_error(slot->name, "Windows aio")) { + + retry = TRUE; } else { - os_file_handle_error(slot->name, "Windows aio"); ret_val = FALSE; } os_mutex_exit(array->mutex); + if (retry) { + /* retry failed read/write operation synchronously. + No need to hold array->mutex. */ + + switch (slot->type) { + case OS_FILE_WRITE: + ret = WriteFile(slot->file, slot->buf, + slot->len, &len, + &(slot->control)); + + break; + case OS_FILE_READ: + ret = ReadFile(slot->file, slot->buf, + slot->len, &len, + &(slot->control)); + + break; + default: + ut_error; + } + + if (!ret && GetLastError() == ERROR_IO_PENDING) { + /* aio was queued successfully! + We want a synchronous i/o operation on a + file where we also use async i/o: in Windows + we must use the same wait mechanism as for + async i/o */ + + ret = GetOverlappedResult(slot->file, + &(slot->control), + &len, TRUE); + } + + ret_val = ret && len == slot->len; + } + os_aio_array_free_slot(array, slot); return(ret_val); === modified file 'storage/innodb_plugin/os/os0sync.c' --- a/storage/innodb_plugin/os/os0sync.c 2009-07-30 12:42:56 +0000 +++ b/storage/innodb_plugin/os/os0sync.c 2009-11-30 11:32:05 +0000 @@ -86,6 +86,9 @@ os_sync_init(void) UT_LIST_INIT(os_event_list); UT_LIST_INIT(os_mutex_list); + os_sync_mutex = NULL; + os_sync_mutex_inited = FALSE; + os_sync_mutex = os_mutex_create(NULL); os_sync_mutex_inited = TRUE; @@ -713,6 +716,7 @@ os_fast_mutex_free( os_mutex_enter(os_sync_mutex); } + ut_ad(os_fast_mutex_count > 0); os_fast_mutex_count--; if (UNIV_LIKELY(os_sync_mutex_inited)) { === modified file 'storage/innodb_plugin/os/os0thread.c' --- a/storage/innodb_plugin/os/os0thread.c 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/os/os0thread.c 2009-11-30 11:32:05 +0000 @@ -233,6 +233,7 @@ os_thread_exit( #ifdef __WIN__ ExitThread((DWORD)exit_value); #else + pthread_detach(pthread_self()); pthread_exit(exit_value); #endif } === modified file 'storage/innodb_plugin/pars/lexyy.c' --- a/storage/innodb_plugin/pars/lexyy.c 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/pars/lexyy.c 2009-11-30 11:32:05 +0000 @@ -2778,3 +2778,16 @@ static void yyfree (void * ptr ) + +/********************************************************************** +Release any resources used by the lexer. */ +UNIV_INTERN +void +pars_lexer_close(void) +/*==================*/ +{ + yylex_destroy(); + free(stringbuf); + stringbuf = NULL; + stringbuf_len_alloc = stringbuf_len = 0; +} === modified file 'storage/innodb_plugin/pars/pars0lex.l' --- a/storage/innodb_plugin/pars/pars0lex.l 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/pars/pars0lex.l 2009-11-30 11:32:05 +0000 @@ -661,3 +661,16 @@ In the state 'id', only two actions are } %% + +/********************************************************************** +Release any resources used by the lexer. */ +UNIV_INTERN +void +pars_lexer_close(void) +/*==================*/ +{ + yylex_destroy(); + free(stringbuf); + stringbuf = NULL; + stringbuf_len_alloc = stringbuf_len = 0; +} === modified file 'storage/innodb_plugin/que/que0que.c' --- a/storage/innodb_plugin/que/que0que.c 2009-07-30 12:42:56 +0000 +++ b/storage/innodb_plugin/que/que0que.c 2009-11-30 11:32:05 +0000 @@ -518,6 +518,7 @@ que_graph_free_recursive( upd_node_t* upd; tab_node_t* cre_tab; ind_node_t* cre_ind; + purge_node_t* purge; if (node == NULL) { @@ -579,6 +580,13 @@ que_graph_free_recursive( mem_heap_free(ins->entry_sys_heap); break; + case QUE_NODE_PURGE: + purge = node; + + mem_heap_free(purge->heap); + + break; + case QUE_NODE_UPDATE: upd = node; === modified file 'storage/innodb_plugin/row/row0merge.c' --- a/storage/innodb_plugin/row/row0merge.c 2009-10-09 14:13:15 +0000 +++ b/storage/innodb_plugin/row/row0merge.c 2009-11-30 12:24:54 +0000 @@ -1200,6 +1200,12 @@ row_merge_read_clustered_index( in order to release the latch on the old page. */ if (btr_pcur_is_after_last_on_page(&pcur)) { + if (UNIV_UNLIKELY(trx_is_interrupted(trx))) { + i = 0; + err = DB_INTERRUPTED; + goto err_exit; + } + btr_pcur_store_position(&pcur, &mtr); mtr_commit(&mtr); mtr_start(&mtr); @@ -1557,6 +1563,7 @@ static __attribute__((nonnull)) ulint row_merge( /*======*/ + trx_t* trx, /*!< in: transaction */ const dict_index_t* index, /*!< in: index being created */ merge_file_t* file, /*!< in/out: file containing index entries */ @@ -1590,6 +1597,10 @@ row_merge( for (; foffs0 < ihalf && foffs1 < file->offset; foffs0++, foffs1++) { ulint ahalf; /*!< arithmetic half the input file */ + if (UNIV_UNLIKELY(trx_is_interrupted(trx))) { + return(DB_INTERRUPTED); + } + error = row_merge_blocks(index, file, block, &foffs0, &foffs1, &of, table); @@ -1617,6 +1628,10 @@ row_merge( /* Copy the last blocks, if there are any. */ while (foffs0 < ihalf) { + if (UNIV_UNLIKELY(trx_is_interrupted(trx))) { + return(DB_INTERRUPTED); + } + if (!row_merge_blocks_copy(index, file, block, &foffs0, &of)) { return(DB_CORRUPTION); } @@ -1625,6 +1640,10 @@ row_merge( ut_ad(foffs0 == ihalf); while (foffs1 < file->offset) { + if (UNIV_UNLIKELY(trx_is_interrupted(trx))) { + return(DB_INTERRUPTED); + } + if (!row_merge_blocks_copy(index, file, block, &foffs1, &of)) { return(DB_CORRUPTION); } @@ -1653,6 +1672,7 @@ static ulint row_merge_sort( /*===========*/ + trx_t* trx, /*!< in: transaction */ const dict_index_t* index, /*!< in: index being created */ merge_file_t* file, /*!< in/out: file containing index entries */ @@ -1671,7 +1691,8 @@ row_merge_sort( do { ulint error; - error = row_merge(index, file, &half, block, tmpfd, table); + error = row_merge(trx, index, file, &half, + block, tmpfd, table); if (error != DB_SUCCESS) { return(error); @@ -2490,7 +2511,7 @@ row_merge_build_indexes( sorting and inserting. */ for (i = 0; i < n_indexes; i++) { - error = row_merge_sort(indexes[i], &merge_files[i], + error = row_merge_sort(trx, indexes[i], &merge_files[i], block, &tmpfd, table); if (error == DB_SUCCESS) { === modified file 'storage/innodb_plugin/row/row0mysql.c' --- a/storage/innodb_plugin/row/row0mysql.c 2009-11-03 10:32:33 +0000 +++ b/storage/innodb_plugin/row/row0mysql.c 2009-11-30 13:13:34 +0000 @@ -1880,6 +1880,8 @@ err_exit: if (UNIV_UNLIKELY(err != DB_SUCCESS)) { trx->error_state = DB_SUCCESS; trx_general_rollback_for_mysql(trx, NULL); + /* TO DO: free table? The code below will dereference + table->name, though. */ } switch (err) { @@ -1898,31 +1900,6 @@ err_exit: break; case DB_DUPLICATE_KEY: - ut_print_timestamp(stderr); - fputs(" InnoDB: Error: table ", stderr); - ut_print_name(stderr, trx, TRUE, table->name); - fputs(" already exists in InnoDB internal\n" - "InnoDB: data dictionary. Have you deleted" - " the .frm file\n" - "InnoDB: and not used DROP TABLE?" - " Have you used DROP DATABASE\n" - "InnoDB: for InnoDB tables in" - " MySQL version <= 3.23.43?\n" - "InnoDB: See the Restrictions section" - " of the InnoDB manual.\n" - "InnoDB: You can drop the orphaned table" - " inside InnoDB by\n" - "InnoDB: creating an InnoDB table with" - " the same name in another\n" - "InnoDB: database and copying the .frm file" - " to the current database.\n" - "InnoDB: Then MySQL thinks the table exists," - " and DROP TABLE will\n" - "InnoDB: succeed.\n" - "InnoDB: You can look for further help from\n" - "InnoDB: " REFMAN "innodb-troubleshooting.html\n", - stderr); - /* We may also get err == DB_ERROR if the .ibd file for the table already exists */ @@ -4157,6 +4134,7 @@ row_check_table_for_mysql( } if (trx_is_interrupted(prebuilt->trx)) { + ret = DB_INTERRUPTED; break; } === modified file 'storage/innodb_plugin/srv/srv0srv.c' --- a/storage/innodb_plugin/srv/srv0srv.c 2009-10-09 12:19:13 +0000 +++ b/storage/innodb_plugin/srv/srv0srv.c 2009-11-30 11:32:05 +0000 @@ -1006,13 +1006,26 @@ srv_init(void) } /*********************************************************************//** -Frees the OS fast mutex created in srv_init(). */ +Frees the data structures created in srv_init(). */ UNIV_INTERN void srv_free(void) /*==========*/ { os_fast_mutex_free(&srv_conc_mutex); + mem_free(srv_conc_slots); + srv_conc_slots = NULL; + + mem_free(srv_sys->threads); + mem_free(srv_sys); + srv_sys = NULL; + + mem_free(kernel_mutex_temp); + kernel_mutex_temp = NULL; + mem_free(srv_mysql_table); + srv_mysql_table = NULL; + + trx_i_s_cache_free(trx_i_s_cache); } /*********************************************************************//** @@ -1024,6 +1037,8 @@ srv_general_init(void) /*==================*/ { ut_mem_init(); + /* Reset the system variables in the recovery module. */ + recv_sys_var_init(); os_sync_init(); sync_init(); mem_init(srv_mem_pool_size); === modified file 'storage/innodb_plugin/srv/srv0start.c' --- a/storage/innodb_plugin/srv/srv0start.c 2009-11-03 10:23:22 +0000 +++ b/storage/innodb_plugin/srv/srv0start.c 2009-11-30 11:32:05 +0000 @@ -103,6 +103,7 @@ Created 2/16/1996 Heikki Tuuri # include "row0row.h" # include "row0mysql.h" # include "btr0pcur.h" +# include "thr0loc.h" # include "os0sync.h" /* for INNODB_RW_LOCKS_USE_ATOMICS */ /** Log sequence number immediately after startup */ @@ -495,6 +496,8 @@ io_handler_thread( mutex_exit(&ios_mutex); } + thr_local_free(os_thread_get_curr_id()); + /* We count the number of threads in os_thread_exit(). A created thread should always use that to exit and not use return() to exit. The thread actually never comes here because it is exited in an @@ -531,32 +534,6 @@ srv_normalize_path_for_win( #endif } -/*********************************************************************//** -Adds a slash or a backslash to the end of a string if it is missing -and the string is not empty. -@return string which has the separator if the string is not empty */ -UNIV_INTERN -char* -srv_add_path_separator_if_needed( -/*=============================*/ - char* str) /*!< in: null-terminated character string */ -{ - char* out_str; - ulint len = ut_strlen(str); - - if (len == 0 || str[len - 1] == SRV_PATH_SEPARATOR) { - - return(str); - } - - out_str = ut_malloc(len + 2); - memcpy(out_str, str, len); - out_str[len] = SRV_PATH_SEPARATOR; - out_str[len + 1] = 0; - - return(out_str); -} - #ifndef UNIV_HOTBACKUP /*********************************************************************//** Calculates the low 32 bits when a file size which is given as a number @@ -605,19 +582,24 @@ open_or_create_log_file( ulint size; ulint size_high; char name[10000]; + ulint dirnamelen; UT_NOT_USED(create_new_db); *log_file_created = FALSE; srv_normalize_path_for_win(srv_log_group_home_dirs[k]); - srv_log_group_home_dirs[k] = srv_add_path_separator_if_needed( - srv_log_group_home_dirs[k]); - ut_a(strlen(srv_log_group_home_dirs[k]) - < (sizeof name) - 10 - sizeof "ib_logfile"); - sprintf(name, "%s%s%lu", srv_log_group_home_dirs[k], - "ib_logfile", (ulong) i); + dirnamelen = strlen(srv_log_group_home_dirs[k]); + ut_a(dirnamelen < (sizeof name) - 10 - sizeof "ib_logfile"); + memcpy(name, srv_log_group_home_dirs[k], dirnamelen); + + /* Add a path separator if needed. */ + if (dirnamelen && name[dirnamelen - 1] != SRV_PATH_SEPARATOR) { + name[dirnamelen++] = SRV_PATH_SEPARATOR; + } + + sprintf(name + dirnamelen, "%s%lu", "ib_logfile", (ulong) i); files[i] = os_file_create(name, OS_FILE_CREATE, OS_FILE_NORMAL, OS_LOG_FILE, &ret); @@ -780,14 +762,22 @@ open_or_create_data_files( *create_new_db = FALSE; srv_normalize_path_for_win(srv_data_home); - srv_data_home = srv_add_path_separator_if_needed(srv_data_home); for (i = 0; i < srv_n_data_files; i++) { + ulint dirnamelen; + srv_normalize_path_for_win(srv_data_file_names[i]); + dirnamelen = strlen(srv_data_home); - ut_a(strlen(srv_data_home) + strlen(srv_data_file_names[i]) + ut_a(dirnamelen + strlen(srv_data_file_names[i]) < (sizeof name) - 1); - sprintf(name, "%s%s", srv_data_home, srv_data_file_names[i]); + memcpy(name, srv_data_home, dirnamelen); + /* Add a path separator if needed. */ + if (dirnamelen && name[dirnamelen - 1] != SRV_PATH_SEPARATOR) { + name[dirnamelen++] = SRV_PATH_SEPARATOR; + } + + strcpy(name + dirnamelen, srv_data_file_names[i]); if (srv_data_file_is_raw_partition[i] == 0) { @@ -1009,7 +999,7 @@ skip_size_check: return(DB_SUCCESS); } -/****************************************************************//** +/******************************************************************** Starts InnoDB and creates a new database if database files are not found and the user wants. @return DB_SUCCESS or error code */ @@ -1120,7 +1110,7 @@ innobase_start_or_create_for_mysql(void) if (srv_start_has_been_called) { fprintf(stderr, - "InnoDB: Error:startup called second time" + "InnoDB: Error: startup called second time" " during the process lifetime.\n" "InnoDB: In the MySQL Embedded Server Library" " you cannot call server_init()\n" @@ -1959,8 +1949,10 @@ innobase_shutdown_for_mysql(void) /* All the threads have exited or are just exiting; NOTE that the threads may not have completed their exit yet. Should we use pthread_join() to make sure - they have exited? Now we just sleep 0.1 seconds and - hope that is enough! */ + they have exited? If we did, we would have to + remove the pthread_detach() from + os_thread_exit(). Now we just sleep 0.1 + seconds and hope that is enough! */ os_mutex_exit(os_sync_mutex); @@ -1999,37 +1991,41 @@ innobase_shutdown_for_mysql(void) srv_misc_tmpfile = 0; } + /* This must be disabled before closing the buffer pool + and closing the data dictionary. */ + btr_search_disable(); + + ibuf_close(); + log_shutdown(); + lock_sys_close(); + thr_local_close(); trx_sys_file_format_close(); + trx_sys_close(); mutex_free(&srv_monitor_file_mutex); mutex_free(&srv_dict_tmpfile_mutex); mutex_free(&srv_misc_tmpfile_mutex); + dict_close(); + btr_search_sys_free(); /* 3. Free all InnoDB's own mutexes and the os_fast_mutexes inside them */ + os_aio_free(); sync_close(); + srv_free(); + fil_close(); /* 4. Free the os_conc_mutex and all os_events and os_mutexes */ - srv_free(); os_sync_free(); - /* Check that all read views are closed except read view owned - by a purge. */ - - if (UT_LIST_GET_LEN(trx_sys->view_list) > 1) { - fprintf(stderr, - "InnoDB: Error: all read views were not closed" - " before shutdown:\n" - "InnoDB: %lu read views open \n", - UT_LIST_GET_LEN(trx_sys->view_list) - 1); - } - - /* 5. Free all allocated memory and the os_fast_mutex created in - ut0mem.c */ + /* 5. Free all allocated memory */ + pars_lexer_close(); + log_mem_free(); buf_pool_free(); ut_free_all_mem(); + mem_close(); if (os_thread_count != 0 || os_event_count != 0 @@ -2060,6 +2056,7 @@ innobase_shutdown_for_mysql(void) } srv_was_started = FALSE; + srv_start_has_been_called = FALSE; return((int) DB_SUCCESS); } === modified file 'storage/innodb_plugin/sync/sync0arr.c' --- a/storage/innodb_plugin/sync/sync0arr.c 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/sync/sync0arr.c 2009-11-30 11:32:05 +0000 @@ -227,24 +227,21 @@ sync_array_create( SYNC_ARRAY_MUTEX: determines the type of mutex protecting the data structure */ { + ulint sz; sync_array_t* arr; - sync_cell_t* cell_array; - sync_cell_t* cell; - ulint i; ut_a(n_cells > 0); /* Allocate memory for the data structures */ arr = ut_malloc(sizeof(sync_array_t)); + memset(arr, 0x0, sizeof(*arr)); - cell_array = ut_malloc(sizeof(sync_cell_t) * n_cells); + sz = sizeof(sync_cell_t) * n_cells; + arr->array = ut_malloc(sz); + memset(arr->array, 0x0, sz); arr->n_cells = n_cells; - arr->n_reserved = 0; - arr->array = cell_array; arr->protection = protection; - arr->sg_count = 0; - arr->res_count = 0; /* Then create the mutex to protect the wait array complex */ if (protection == SYNC_ARRAY_OS_MUTEX) { @@ -255,13 +252,6 @@ sync_array_create( ut_error; } - for (i = 0; i < n_cells; i++) { - cell = sync_array_get_nth_cell(arr, i); - cell->wait_object = NULL; - cell->waiting = FALSE; - cell->signal_count = 0; - } - return(arr); } === modified file 'storage/innodb_plugin/sync/sync0sync.c' --- a/storage/innodb_plugin/sync/sync0sync.c 2009-10-12 12:00:56 +0000 +++ b/storage/innodb_plugin/sync/sync0sync.c 2009-11-30 11:32:05 +0000 @@ -1377,7 +1377,12 @@ sync_close(void) mutex_free(&mutex_list_mutex); #ifdef UNIV_SYNC_DEBUG mutex_free(&sync_thread_mutex); + + /* Switch latching order checks on in sync0sync.c */ + sync_order_checks_on = FALSE; #endif /* UNIV_SYNC_DEBUG */ + + sync_initialized = FALSE; } /*******************************************************************//** === modified file 'storage/innodb_plugin/thr/thr0loc.c' --- a/storage/innodb_plugin/thr/thr0loc.c 2009-10-08 10:00:49 +0000 +++ b/storage/innodb_plugin/thr/thr0loc.c 2009-11-30 11:32:05 +0000 @@ -246,3 +246,34 @@ thr_local_init(void) mutex_create(&thr_local_mutex, SYNC_THR_LOCAL); } + +/******************************************************************** +Close the thread local storage module. */ +UNIV_INTERN +void +thr_local_close(void) +/*=================*/ +{ + ulint i; + + ut_a(thr_local_hash != NULL); + + /* Free the hash elements. We don't remove them from the table + because we are going to destroy the table anyway. */ + for (i = 0; i < hash_get_n_cells(thr_local_hash); i++) { + thr_local_t* local; + + local = HASH_GET_FIRST(thr_local_hash, i); + + while (local) { + thr_local_t* prev_local = local; + + local = HASH_GET_NEXT(hash, prev_local); + ut_a(prev_local->magic_n == THR_LOCAL_MAGIC_N); + mem_free(prev_local); + } + } + + hash_table_free(thr_local_hash); + thr_local_hash = NULL; +} === modified file 'storage/innodb_plugin/trx/trx0i_s.c' --- a/storage/innodb_plugin/trx/trx0i_s.c 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/trx/trx0i_s.c 2009-12-01 10:38:40 +0000 @@ -60,7 +60,7 @@ Created July 17, 2007 Vasil Dimov /** @brief The maximum number of chunks to allocate for a table cache. The rows of a table cache are stored in a set of chunks. When a new -row is added a new chunk is allocated if necessary. Assuming that the +row is added a new chunk is allocated if necessary. Assuming that the first one is 1024 rows (TABLE_CACHE_INITIAL_ROWSNUM) and each subsequent is N/2 where N is the number of rows we have allocated till now, then 39th chunk would accommodate 1677416425 rows and all chunks @@ -238,6 +238,27 @@ table_cache_init( } /*******************************************************************//** +Frees a table cache. */ +static +void +table_cache_free( +/*=============*/ + i_s_table_cache_t* table_cache) /*!< in/out: table cache */ +{ + ulint i; + + for (i = 0; i < MEM_CHUNKS_IN_TABLE_CACHE; i++) { + + /* the memory is actually allocated in + table_cache_create_empty_row() */ + if (table_cache->chunks[i].base) { + mem_free(table_cache->chunks[i].base); + table_cache->chunks[i].base = NULL; + } + } +} + +/*******************************************************************//** Returns an empty row from a table cache. The row is allocated if no more empty rows are available. The number of used rows is incremented. If the memory limit is hit then NULL is returned and nothing is @@ -1184,9 +1205,6 @@ trx_i_s_possibly_fetch_data_into_cache( return(1); } - /* We are going to access trx->query in all transactions */ - innobase_mysql_prepare_print_arbitrary_thd(); - /* We need to read trx_sys and record/table lock queues */ mutex_enter(&kernel_mutex); @@ -1194,8 +1212,6 @@ trx_i_s_possibly_fetch_data_into_cache( mutex_exit(&kernel_mutex); - innobase_mysql_end_print_arbitrary_thd(); - return(0); } @@ -1252,6 +1268,22 @@ trx_i_s_cache_init( } /*******************************************************************//** +Free the INFORMATION SCHEMA trx related cache. */ +UNIV_INTERN +void +trx_i_s_cache_free( +/*===============*/ + trx_i_s_cache_t* cache) /*!< in, own: cache to free */ +{ + hash_table_free(cache->locks_hash); + ha_storage_free(cache->storage); + table_cache_free(&cache->innodb_trx); + table_cache_free(&cache->innodb_locks); + table_cache_free(&cache->innodb_lock_waits); + memset(cache, 0, sizeof *cache); +} + +/*******************************************************************//** Issue a shared/read lock on the tables cache. */ UNIV_INTERN void === modified file 'storage/innodb_plugin/trx/trx0purge.c' --- a/storage/innodb_plugin/trx/trx0purge.c 2009-07-30 12:42:56 +0000 +++ b/storage/innodb_plugin/trx/trx0purge.c 2009-11-30 11:32:05 +0000 @@ -249,6 +249,44 @@ trx_purge_sys_create(void) purge_sys->heap); } +/************************************************************************ +Frees the global purge system control structure. */ +UNIV_INTERN +void +trx_purge_sys_close(void) +/*======================*/ +{ + ut_ad(!mutex_own(&kernel_mutex)); + + que_graph_free(purge_sys->query); + + ut_a(purge_sys->sess->trx->is_purge); + purge_sys->sess->trx->conc_state = TRX_NOT_STARTED; + sess_close(purge_sys->sess); + purge_sys->sess = NULL; + + if (purge_sys->view != NULL) { + /* Because acquiring the kernel mutex is a pre-condition + of read_view_close(). We don't really need it here. */ + mutex_enter(&kernel_mutex); + + read_view_close(purge_sys->view); + purge_sys->view = NULL; + + mutex_exit(&kernel_mutex); + } + + trx_undo_arr_free(purge_sys->arr); + + rw_lock_free(&purge_sys->latch); + mutex_free(&purge_sys->mutex); + + mem_heap_free(purge_sys->heap); + mem_free(purge_sys); + + purge_sys = NULL; +} + /*================ UNDO LOG HISTORY LIST =============================*/ /********************************************************************//** === modified file 'storage/innodb_plugin/trx/trx0rseg.c' --- a/storage/innodb_plugin/trx/trx0rseg.c 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/trx/trx0rseg.c 2009-11-30 11:32:05 +0000 @@ -132,6 +132,49 @@ trx_rseg_header_create( } /***********************************************************************//** +Free's an instance of the rollback segment in memory. */ +UNIV_INTERN +void +trx_rseg_mem_free( +/*==============*/ + trx_rseg_t* rseg) /* in, own: instance to free */ +{ + trx_undo_t* undo; + + mutex_free(&rseg->mutex); + + /* There can't be any active transactions. */ + ut_a(UT_LIST_GET_LEN(rseg->update_undo_list) == 0); + ut_a(UT_LIST_GET_LEN(rseg->insert_undo_list) == 0); + + undo = UT_LIST_GET_FIRST(rseg->update_undo_cached); + + while (undo != NULL) { + trx_undo_t* prev_undo = undo; + + undo = UT_LIST_GET_NEXT(undo_list, undo); + UT_LIST_REMOVE(undo_list, rseg->update_undo_cached, prev_undo); + + trx_undo_mem_free(prev_undo); + } + + undo = UT_LIST_GET_FIRST(rseg->insert_undo_cached); + + while (undo != NULL) { + trx_undo_t* prev_undo = undo; + + undo = UT_LIST_GET_NEXT(undo_list, undo); + UT_LIST_REMOVE(undo_list, rseg->insert_undo_cached, prev_undo); + + trx_undo_mem_free(prev_undo); + } + + trx_sys_set_nth_rseg(trx_sys, rseg->id, NULL); + + mem_free(rseg); +} + +/*************************************************************************** Creates and initializes a rollback segment object. The values for the fields are read from the header. The object is inserted to the rseg list of the trx system object and a pointer is inserted in the rseg === modified file 'storage/innodb_plugin/trx/trx0sys.c' --- a/storage/innodb_plugin/trx/trx0sys.c 2009-07-30 12:42:56 +0000 +++ b/storage/innodb_plugin/trx/trx0sys.c 2009-11-30 11:32:05 +0000 @@ -40,6 +40,7 @@ Created 3/26/1996 Heikki Tuuri #include "trx0purge.h" #include "log0log.h" #include "os0file.h" +#include "read0read.h" /** The file format tag structure with id and name. */ struct file_format_struct { @@ -1533,3 +1534,80 @@ trx_sys_file_format_id_to_name( } #endif /* !UNIV_HOTBACKUP */ + +/********************************************************************* +Shutdown/Close the transaction system. */ +UNIV_INTERN +void +trx_sys_close(void) +/*===============*/ +{ + trx_rseg_t* rseg; + read_view_t* view; + + ut_ad(trx_sys != NULL); + + /* Check that all read views are closed except read view owned + by a purge. */ + + if (UT_LIST_GET_LEN(trx_sys->view_list) > 1) { + fprintf(stderr, + "InnoDB: Error: all read views were not closed" + " before shutdown:\n" + "InnoDB: %lu read views open \n", + UT_LIST_GET_LEN(trx_sys->view_list) - 1); + } + + sess_close(trx_dummy_sess); + trx_dummy_sess = NULL; + + trx_purge_sys_close(); + + mutex_enter(&kernel_mutex); + + /* Free the double write data structures. */ + ut_a(trx_doublewrite != NULL); + ut_free(trx_doublewrite->write_buf_unaligned); + trx_doublewrite->write_buf_unaligned = NULL; + + mem_free(trx_doublewrite->buf_block_arr); + trx_doublewrite->buf_block_arr = NULL; + + mutex_free(&trx_doublewrite->mutex); + mem_free(trx_doublewrite); + trx_doublewrite = NULL; + + /* There can't be any active transactions. */ + rseg = UT_LIST_GET_FIRST(trx_sys->rseg_list); + + while (rseg != NULL) { + trx_rseg_t* prev_rseg = rseg; + + rseg = UT_LIST_GET_NEXT(rseg_list, prev_rseg); + UT_LIST_REMOVE(rseg_list, trx_sys->rseg_list, prev_rseg); + + trx_rseg_mem_free(prev_rseg); + } + + view = UT_LIST_GET_FIRST(trx_sys->view_list); + + while (view != NULL) { + read_view_t* prev_view = view; + + view = UT_LIST_GET_NEXT(view_list, prev_view); + + /* Views are allocated from the trx_sys->global_read_view_heap. + So, we simply remove the element here. */ + UT_LIST_REMOVE(view_list, trx_sys->view_list, prev_view); + } + + ut_a(UT_LIST_GET_LEN(trx_sys->trx_list) == 0); + ut_a(UT_LIST_GET_LEN(trx_sys->rseg_list) == 0); + ut_a(UT_LIST_GET_LEN(trx_sys->view_list) == 0); + ut_a(UT_LIST_GET_LEN(trx_sys->mysql_trx_list) == 0); + + mem_free(trx_sys); + + trx_sys = NULL; + mutex_exit(&kernel_mutex); +} === modified file 'storage/innodb_plugin/trx/trx0trx.c' --- a/storage/innodb_plugin/trx/trx0trx.c 2009-10-09 14:13:15 +0000 +++ b/storage/innodb_plugin/trx/trx0trx.c 2009-12-01 10:38:40 +0000 @@ -1636,9 +1636,7 @@ trx_mark_sql_stat_end( /**********************************************************************//** Prints info about a transaction to the given file. The caller must own the -kernel mutex and must have called -innobase_mysql_prepare_print_arbitrary_thd(), unless he knows that MySQL -or InnoDB cannot meanwhile change the info printed here. */ +kernel mutex. */ UNIV_INTERN void trx_print( === modified file 'storage/innodb_plugin/trx/trx0undo.c' --- a/storage/innodb_plugin/trx/trx0undo.c 2009-07-30 12:42:56 +0000 +++ b/storage/innodb_plugin/trx/trx0undo.c 2009-11-30 11:32:05 +0000 @@ -1522,7 +1522,7 @@ trx_undo_mem_init_for_reuse( /********************************************************************//** Frees an undo log memory copy. */ -static +UNIV_INTERN void trx_undo_mem_free( /*==============*/ === modified file 'storage/innodb_plugin/usr/usr0sess.c' --- a/storage/innodb_plugin/usr/usr0sess.c 2009-05-27 09:45:59 +0000 +++ b/storage/innodb_plugin/usr/usr0sess.c 2009-11-30 11:32:05 +0000 @@ -32,14 +32,6 @@ Created 6/25/1996 Heikki Tuuri #include "trx0trx.h" /*********************************************************************//** -Closes a session, freeing the memory occupied by it. */ -static -void -sess_close( -/*=======*/ - sess_t* sess); /*!< in, own: session object */ - -/*********************************************************************//** Opens a session. @return own: session object */ UNIV_INTERN @@ -64,35 +56,16 @@ sess_open(void) /*********************************************************************//** Closes a session, freeing the memory occupied by it. */ -static +UNIV_INTERN void sess_close( /*=======*/ sess_t* sess) /*!< in, own: session object */ { - ut_ad(mutex_own(&kernel_mutex)); - ut_ad(sess->trx == NULL); - - mem_free(sess); -} - -/*********************************************************************//** -Closes a session, freeing the memory occupied by it, if it is in a state -where it should be closed. -@return TRUE if closed */ -UNIV_INTERN -ibool -sess_try_close( -/*===========*/ - sess_t* sess) /*!< in, own: session object */ -{ - ut_ad(mutex_own(&kernel_mutex)); + ut_ad(!mutex_own(&kernel_mutex)); - if (UT_LIST_GET_LEN(sess->graphs) == 0) { - sess_close(sess); + ut_a(UT_LIST_GET_LEN(sess->graphs) == 0); - return(TRUE); - } - - return(FALSE); + trx_free_for_background(sess->trx); + mem_free(sess); } === modified file 'storage/innodb_plugin/ut/ut0mem.c' --- a/storage/innodb_plugin/ut/ut0mem.c 2009-07-30 12:42:56 +0000 +++ b/storage/innodb_plugin/ut/ut0mem.c 2009-11-30 11:32:05 +0000 @@ -433,6 +433,8 @@ ut_free_all_mem(void) " total allocated memory is %lu\n", (ulong) ut_total_allocated_memory); } + + ut_mem_block_list_inited = FALSE; } #endif /* !UNIV_HOTBACKUP */ === modified file 'storage/maria/ft_maria.c' --- a/storage/maria/ft_maria.c 2007-07-02 17:45:15 +0000 +++ b/storage/maria/ft_maria.c 2010-01-27 21:53:08 +0000 @@ -22,8 +22,8 @@ #include "ma_ftdefs.h" FT_INFO *maria_ft_init_search(uint flags, void *info, uint keynr, - uchar *query, uint query_len, CHARSET_INFO *cs, - uchar *record) + uchar *query, size_t query_len, + CHARSET_INFO *cs, uchar *record) { FT_INFO *res; if (flags & FT_BOOL) === modified file 'storage/maria/ma_blockrec.c' --- a/storage/maria/ma_blockrec.c 2010-01-06 21:27:53 +0000 +++ b/storage/maria/ma_blockrec.c 2010-01-28 11:35:10 +0000 @@ -1688,7 +1688,8 @@ static my_bool get_head_or_tail_page(MAR if (!page_link.changed) goto crashed; - DBUG_ASSERT((res->buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) == page_type); + DBUG_ASSERT((uint) (res->buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) == + page_type); if (!(dir= find_free_position(page_type == HEAD_PAGE ? info : 0, res->buff, block_size, &res->rownr, &res->length, &res->empty_space))) === modified file 'storage/maria/ma_check.c' --- a/storage/maria/ma_check.c 2009-11-29 23:08:56 +0000 +++ b/storage/maria/ma_check.c 2010-01-14 16:51:00 +0000 @@ -6018,7 +6018,7 @@ int maria_update_state_info(HA_CHECK *pa { if (update & UPDATE_TIME) { - share->state.check_time= (long) time((time_t*) 0); + share->state.check_time= time((time_t*) 0); if (!share->state.create_time) share->state.create_time= share->state.check_time; } === modified file 'storage/maria/ma_close.c' --- a/storage/maria/ma_close.c 2009-02-05 22:38:30 +0000 +++ b/storage/maria/ma_close.c 2010-01-29 18:42:22 +0000 @@ -177,6 +177,7 @@ int maria_close(register MARIA_HA *info) { (void) pthread_mutex_destroy(&share->intern_lock); (void) pthread_mutex_destroy(&share->close_lock); + (void) pthread_cond_destroy(&share->key_del_cond); my_free((uchar *)share, MYF(0)); /* If share cannot be freed, it's because checkpoint has previously === modified file 'storage/maria/ma_create.c' --- a/storage/maria/ma_create.c 2009-02-19 09:01:25 +0000 +++ b/storage/maria/ma_create.c 2010-01-14 16:51:00 +0000 @@ -772,7 +772,7 @@ int maria_create(const char *name, enum share.base.min_block_length= share.base.pack_reclength; if (! (flags & HA_DONT_TOUCH_DATA)) - share.state.create_time= (long) time((time_t*) 0); + share.state.create_time= time((time_t*) 0); pthread_mutex_lock(&THR_LOCK_maria); === modified file 'storage/myisam/ft_boolean_search.c' --- a/storage/myisam/ft_boolean_search.c 2009-11-30 13:36:06 +0000 +++ b/storage/myisam/ft_boolean_search.c 2010-01-28 14:49:14 +0000 @@ -299,7 +299,7 @@ static int ftb_parse_query_internal(MYSQ } -static int _ftb_parse_query(FTB *ftb, uchar *query, uint len, +static int _ftb_parse_query(FTB *ftb, uchar *query, mysql_ft_size_t len, struct st_mysql_ftparser *parser) { MYSQL_FTPARSER_PARAM *param; @@ -321,7 +321,7 @@ static int _ftb_parse_query(FTB *ftb, uc param->mysql_add_word= ftb_query_add_word; param->mysql_ftparam= (void *)&ftb_param; param->cs= ftb->charset; - param->doc= (char*) query; + param->doc= query; param->length= len; param->flags= 0; param->mode= MYSQL_FTPARSER_FULL_BOOLEAN_INFO; @@ -475,8 +475,7 @@ static void _ftb_init_index_search(FT_IN int i; FTB_WORD *ftbw; - if ((ftb->state != READY && ftb->state !=INDEX_DONE) || - ftb->keynr == NO_SUCH_KEY) + if (ftb->state == UNINITIALIZED || ftb->keynr == NO_SUCH_KEY) return; ftb->state=INDEX_SEARCH; @@ -541,7 +540,7 @@ static void _ftb_init_index_search(FT_IN FT_INFO * ft_init_boolean_search(MI_INFO *info, uint keynr, uchar *query, - uint query_len, CHARSET_INFO *cs) + mysql_ft_size_t query_len, CHARSET_INFO *cs) { FTB *ftb; FTB_EXPR *ftbe; @@ -680,8 +679,9 @@ static int ftb_check_phrase_internal(MYS -1 is returned if error occurs. */ -static int _ftb_check_phrase(FTB *ftb, const uchar *document, uint len, - FTB_EXPR *ftbe, struct st_mysql_ftparser *parser) +static int _ftb_check_phrase(FTB *ftb, const uchar *document, + mysql_ft_size_t len, + FTB_EXPR *ftbe, struct st_mysql_ftparser *parser) { MY_FTB_PHRASE_PARAM ftb_param; MYSQL_FTPARSER_PARAM *param; @@ -702,7 +702,7 @@ static int _ftb_check_phrase(FTB *ftb, c param->mysql_add_word= ftb_phrase_add_word; param->mysql_ftparam= (void *)&ftb_param; param->cs= ftb->charset; - param->doc= (char *) document; + param->doc= document; param->length= len; param->flags= 0; param->mode= MYSQL_FTPARSER_WITH_STOPWORDS; @@ -1000,7 +1000,7 @@ float ft_boolean_find_relevance(FT_INFO { if (!ftsi.pos) continue; - param->doc= (char *)ftsi.pos; + param->doc= ftsi.pos; param->length= ftsi.len; if (unlikely(parser->parse(param))) return 0; === modified file 'storage/myisam/ft_myisam.c' --- a/storage/myisam/ft_myisam.c 2007-07-02 17:45:15 +0000 +++ b/storage/myisam/ft_myisam.c 2010-01-27 21:53:08 +0000 @@ -23,8 +23,8 @@ #include "ftdefs.h" FT_INFO *ft_init_search(uint flags, void *info, uint keynr, - uchar *query, uint query_len, CHARSET_INFO *cs, - uchar *record) + uchar *query, size_t query_len, + CHARSET_INFO *cs, uchar *record) { FT_INFO *res; if (flags & FT_BOOL) === modified file 'storage/myisam/ft_nlq_search.c' --- a/storage/myisam/ft_nlq_search.c 2009-11-30 13:36:06 +0000 +++ b/storage/myisam/ft_nlq_search.c 2010-01-27 21:53:08 +0000 @@ -204,7 +204,8 @@ static int FT_DOC_cmp(void *unused __att FT_INFO *ft_init_nlq_search(MI_INFO *info, uint keynr, uchar *query, - uint query_len, uint flags, uchar *record) + mysql_ft_size_t query_len, uint flags, + uchar *record) { TREE wtree; ALL_IN_ONE aio; === modified file 'storage/myisam/ft_parser.c' --- a/storage/myisam/ft_parser.c 2009-11-30 13:36:06 +0000 +++ b/storage/myisam/ft_parser.c 2010-01-28 14:49:14 +0000 @@ -175,7 +175,7 @@ uchar ft_get_word(CHARSET_INFO *cs, cons if ((param->trunc=(doc<end && *doc == FTB_TRUNC))) doc++; - if (((length >= ft_min_word_len && !is_stopword((char*) word->pos, + if (((length >= ft_min_word_len && !is_stopword(word->pos, word->len)) || param->trunc) && length < ft_max_word_len) { @@ -298,13 +298,13 @@ static int ft_parse_internal(MYSQL_FTPAR DBUG_ENTER("ft_parse_internal"); while (ft_simple_get_word(wtree->custom_arg, &doc, end, &w, TRUE)) - if (param->mysql_add_word(param, (char*) w.pos, w.len, 0)) + if (param->mysql_add_word(param, w.pos, w.len, 0)) DBUG_RETURN(1); DBUG_RETURN(0); } -int ft_parse(TREE *wtree, const uchar *doc, int doclen, +int ft_parse(TREE *wtree, const uchar *doc, mysql_ft_size_t doclen, struct st_mysql_ftparser *parser, MYSQL_FTPARSER_PARAM *param, MEM_ROOT *mem_root) { @@ -319,7 +319,7 @@ int ft_parse(TREE *wtree, const uchar *d param->mysql_add_word= ft_add_word; param->mysql_ftparam= &my_param; param->cs= wtree->custom_arg; - param->doc= (char*) doc; + param->doc= doc; param->length= doclen; param->mode= MYSQL_FTPARSER_SIMPLE_MODE; DBUG_RETURN(parser->parse(param)); === modified file 'storage/myisam/ft_stopwords.c' --- a/storage/myisam/ft_stopwords.c 2009-11-29 23:08:56 +0000 +++ b/storage/myisam/ft_stopwords.c 2010-01-28 14:49:14 +0000 @@ -20,8 +20,8 @@ typedef struct st_ft_stopwords { - const char * pos; - uint len; + const uchar* pos; + size_t len; } FT_STOPWORD; static TREE *stopwords3=NULL; @@ -30,15 +30,15 @@ static int FT_STOPWORD_cmp(void* cmp_arg FT_STOPWORD *w1, FT_STOPWORD *w2) { return ha_compare_text(default_charset_info, - (uchar *)w1->pos,w1->len, - (uchar *)w2->pos,w2->len,0,0); + w1->pos, w1->len, + w2->pos, w2->len, 0, 0); } static void FT_STOPWORD_free(FT_STOPWORD *w, TREE_FREE action, void *arg __attribute__((unused))) { if (action == free_free) - my_free((uchar*) w->pos, MYF(0)); + my_free((void*) w->pos, MYF(0)); } static int ft_add_stopword(const char *w) === modified file 'storage/myisam/ftdefs.h' --- a/storage/myisam/ftdefs.h 2009-11-29 23:08:56 +0000 +++ b/storage/myisam/ftdefs.h 2010-01-27 21:53:08 +0000 @@ -97,8 +97,8 @@ typedef struct st_ft_word { const uchar *pos; - uint len; double weight; + size_t len; } FT_WORD; int is_stopword(const uchar *word, size_t len); @@ -111,7 +111,8 @@ uchar ft_simple_get_word(CHARSET_INFO *, FT_WORD *, my_bool); typedef struct _st_ft_seg_iterator { - uint num, len; + uint num; + mysql_ft_size_t len; HA_KEYSEG *seg; const uchar *rec, *pos; } FT_SEG_ITERATOR; @@ -128,8 +129,9 @@ FT_WORD * _mi_ft_parserecord(MI_INFO *, uint _mi_ft_parse(TREE *, MI_INFO *, uint, const uchar *, MYSQL_FTPARSER_PARAM *, MEM_ROOT *); -FT_INFO *ft_init_nlq_search(MI_INFO *, uint, uchar *, uint, uint, uchar *); -FT_INFO *ft_init_boolean_search(MI_INFO *, uint, uchar *, uint, CHARSET_INFO *); +FT_INFO *ft_init_nlq_search(MI_INFO *, uint, uchar *, mysql_ft_size_t, uint, + uchar *); +FT_INFO *ft_init_boolean_search(MI_INFO *, uint, uchar *, mysql_ft_size_t, CHARSET_INFO *); extern const struct _ft_vft _ft_vft_nlq; int ft_nlq_read_next(FT_INFO *, char *); === modified file 'storage/myisam/mi_check.c' --- a/storage/myisam/mi_check.c 2009-12-03 11:34:11 +0000 +++ b/storage/myisam/mi_check.c 2010-01-14 16:51:00 +0000 @@ -4447,7 +4447,7 @@ int update_state_info(HA_CHECK *param, M { if (update & UPDATE_TIME) { - share->state.check_time= (long) time((time_t*) 0); + share->state.check_time= time((time_t*) 0); if (!share->state.create_time) share->state.create_time=share->state.check_time; } === modified file 'storage/myisam/mi_create.c' --- a/storage/myisam/mi_create.c 2009-10-15 21:38:29 +0000 +++ b/storage/myisam/mi_create.c 2010-01-14 16:51:00 +0000 @@ -575,7 +575,7 @@ int mi_create(const char *name,uint keys max(share.base.pack_reclength,MI_MIN_BLOCK_LENGTH) : MI_EXTEND_BLOCK_LENGTH; if (! (flags & HA_DONT_TOUCH_DATA)) - share.state.create_time= (long) time((time_t*) 0); + share.state.create_time= time((time_t*) 0); pthread_mutex_lock(&THR_LOCK_myisam); === modified file 'storage/xtradb/CMakeLists.txt' --- a/storage/xtradb/CMakeLists.txt 2009-11-14 09:53:18 +0000 +++ b/storage/xtradb/CMakeLists.txt 2010-01-15 15:58:25 +0000 @@ -13,36 +13,41 @@ # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA -IF (CMAKE_SIZEOF_VOID_P MATCHES 8) - SET(WIN64 TRUE) -ENDIF (CMAKE_SIZEOF_VOID_P MATCHES 8) +# This is the CMakeLists for InnoDB Plugin -# Check type sizes -include(CheckTypeSize) - -# Currently, the checked results are not used. -CHECK_TYPE_SIZE(int SIZEOF_INT) -CHECK_TYPE_SIZE(long SIZEOF_LONG) -CHECK_TYPE_SIZE(void* SIZEOF_VOID_P) SET(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -DSAFEMALLOC -DSAFE_MUTEX") SET(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} -DSAFEMALLOC -DSAFE_MUTEX") -INCLUDE("${PROJECT_SOURCE_DIR}/storage/mysql_storage_engine.cmake") + +# Starting at 5.1.38, MySQL CMake files are simplified. But the plugin +# CMakeLists.txt still needs to work with previous versions of MySQL. +IF (MYSQL_VERSION_ID GREATER "50137") + INCLUDE("${PROJECT_SOURCE_DIR}/storage/mysql_storage_engine.cmake") +ENDIF (MYSQL_VERSION_ID GREATER "50137") + +IF (CMAKE_SIZEOF_VOID_P MATCHES 8) + SET(WIN64 TRUE) +ENDIF (CMAKE_SIZEOF_VOID_P MATCHES 8) + ADD_DEFINITIONS(-D_WIN32 -D_LIB -DMYSQL_SERVER) -INCLUDE_DIRECTORIES(${CMAKE_SOURCE_DIR}/include ${CMAKE_SOURCE_DIR}/zlib - ${CMAKE_SOURCE_DIR}/storage/xtradb/include - ${CMAKE_SOURCE_DIR}/storage/xtradb/handler - ${CMAKE_SOURCE_DIR}/sql - ${CMAKE_SOURCE_DIR}/regex - ${CMAKE_SOURCE_DIR}/extra/yassl/include) +# Include directories under innobase +INCLUDE_DIRECTORIES(${CMAKE_SOURCE_DIR}/storage/xtradb/include + ${CMAKE_SOURCE_DIR}/storage/xtradb/handler) + +# Include directories under mysql +INCLUDE_DIRECTORIES(${CMAKE_SOURCE_DIR}/include + ${CMAKE_SOURCE_DIR}/sql + ${CMAKE_SOURCE_DIR}/regex + ${CMAKE_SOURCE_DIR}/zlib + ${CMAKE_SOURCE_DIR}/extra/yassl/include) # Removing compiler optimizations for innodb/mem/* files on 64-bit Windows # due to 64-bit compiler error, See MySQL Bug #19424, #36366, #34297 -IF(MSVC AND $(WIN64)) +IF (MSVC AND $(WIN64)) SET_SOURCE_FILES_PROPERTIES(mem/mem0mem.c mem/mem0pool.c PROPERTIES COMPILE_FLAGS -Od) -ENDIF(MSVC AND $(WIN64)) +ENDIF (MSVC AND $(WIN64)) SET(INNOBASE_SOURCES btr/btr0btr.c btr/btr0cur.c btr/btr0pcur.c btr/btr0sea.c buf/buf0buddy.c buf/buf0buf.c buf/buf0flu.c buf/buf0lru.c buf/buf0rea.c @@ -77,5 +82,20 @@ SET(INNOBASE_SOURCES btr/btr0btr.c btr/b usr/usr0sess.c ut/ut0byte.c ut/ut0dbg.c ut/ut0mem.c ut/ut0rnd.c ut/ut0ut.c ut/ut0vec.c ut/ut0list.c ut/ut0wqueue.c) +ADD_DEFINITIONS(-DHAVE_WINDOWS_ATOMICS -DIB_HAVE_PAUSE_INSTRUCTION) -MYSQL_STORAGE_ENGINE(INNOBASE) +IF (MYSQL_VERSION_ID GREATER "50137") + MYSQL_STORAGE_ENGINE(INNOBASE) + # Use ha_innodb for plugin name, if plugin is built + GET_TARGET_PROPERTY(LIB_LOCATION ha_innobase LOCATION) + IF(LIB_LOCATION) + SET_TARGET_PROPERTIES(ha_innobase PROPERTIES OUTPUT_NAME ha_innodb) + ENDIF(LIB_LOCATION) +ELSE (MYSQL_VERSION_ID GREATER "50137") + IF (NOT SOURCE_SUBLIBS) + ADD_DEFINITIONS(-D_WIN32 -DMYSQL_SERVER) + ADD_LIBRARY(innobase STATIC ${INNOBASE_SOURCES}) + # Require mysqld_error.h, which is built as part of the GenError + ADD_DEPENDENCIES(innobase GenError) + ENDIF (NOT SOURCE_SUBLIBS) +ENDIF (MYSQL_VERSION_ID GREATER "50137") === modified file 'storage/xtradb/ChangeLog' --- a/storage/xtradb/ChangeLog 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/ChangeLog 2010-01-15 15:58:25 +0000 @@ -1,3 +1,319 @@ +2009-11-20 The InnoDB Team + + * handler/ha_innodb.cc: + Add a workaround to prevent a crash due to Bug#45961 DDL on + partitioned innodb tables leaves data dictionary in an inconsistent + state + +2009-11-19 The InnoDB Team + + * btr/btr0btr.c: + Fix Bug#48469 when innodb tablespace is configured too small, crash + and corruption! + +2009-11-19 The InnoDB Team + + * data/data0type.c: + Fix Bug#48526 Data type for float and double is incorrectly reported + in InnoDB table monitor + +2009-11-19 The InnoDB Team + + * CMakeLists.txt: + Fix Bug#48317 cannot build innodb as static library + +2009-11-18 The InnoDB Team + + * handler/handler0alter.cc: + Fix Bug#48782 On lock wait timeout, CREATE INDEX (creating primary key) + attempts DROP TABLE + +2009-11-17 The InnoDB Team + + * handler/ha_innodb.cc, mysql-test/innodb.result, + mysql-test/innodb.test, mysql-test/innodb_bug44369.result, + mysql-test/innodb_bug44369.test, mysql-test/patches/innodb-index.diff, + row/row0mysql.c: + Report duplicate table names to the client connection, not to the + error log. + +2009-11-12 The InnoDB Team + + * handler/ha_innodb.cc, include/db0err.h, row/row0merge.c, + row/row0mysql.c: + Allow CREATE INDEX to be interrupted. + Also, when CHECK TABLE is interrupted, report ER_QUERY_INTERRUPTED. + +2009-11-11 The InnoDB Team + + * handler/ha_innodb.cc, mysql-test/innodb_bug47167.result, + mysql-test/innodb_bug47167.test, mysql-test/innodb_file_format.result: + Fix Bug#47167 "set global innodb_file_format_check" cannot set value + by User-Defined Variable + +2009-11-11 The InnoDB Team + + * include/os0file.h, os/os0file.c: + Fix Bug#3139 Mysql crashes: 'windows error 995' after several selects + on a large DB + +2009-11-04 The InnoDB Team + + * handler/ha_innodb.cc: + Fix Bug#32430 'show innodb status' causes errors + Invalid (old?) table or database name in logs + +2009-11-02 The InnoDB Team + + * btr/btr0sea.c, buf/buf0buf.c, dict/dict0dict.c, fil/fil0fil.c, + ibuf/ibuf0ibuf.c, include/btr0sea.h, include/dict0dict.h, + include/fil0fil.h, include/ibuf0ibuf.h, include/lock0lock.h, + include/log0log.h, include/log0recv.h, include/mem0mem.h, + include/mem0pool.h, include/os0file.h, include/pars0pars.h, + include/srv0srv.h, include/thr0loc.h, include/trx0i_s.h, + include/trx0purge.h, include/trx0rseg.h, include/trx0sys.h, + include/trx0undo.h, include/usr0sess.h, lock/lock0lock.c, + log/log0log.c, log/log0recv.c, mem/mem0dbg.c, mem/mem0pool.c, + os/os0file.c, os/os0sync.c, os/os0thread.c, pars/lexyy.c, + pars/pars0lex.l, que/que0que.c, srv/srv0srv.c, srv/srv0start.c, + sync/sync0arr.c, sync/sync0sync.c, thr/thr0loc.c, trx/trx0i_s.c, + trx/trx0purge.c, trx/trx0rseg.c, trx/trx0sys.c, trx/trx0undo.c, + usr/usr0sess.c, ut/ut0mem.c: + Fix Bug #45992 innodb memory not freed after shutdown + Fix Bug #46656 InnoDB plugin: memory leaks (Valgrind) + +2009-10-29 The InnoDB Team + + * handler/ha_innodb.cc, mysql-test/innodb-autoinc.result, + mysql-test/innodb-autoinc.test: + Fix Bug#47125 auto_increment start value is ignored if an index is + created and engine=innodb + +2009-10-29 The InnoDB Team + + * handler/ha_innodb.cc, mysql-test/innodb_bug47777.result, + mysql-test/innodb_bug47777.test: + Fix Bug#47777 innodb dies with spatial pk: Failing assertion: buf <= + original_buf + buf_len + +2009-10-29 The InnoDB Team + + * handler/ha_innodb.cc: + Fix Bug#38996 Race condition in ANALYZE TABLE + +2009-10-29 The InnoDB Team + + * handler/ha_innodb.cc: + Fix bug#42383: Can't create table 'test.bug39438' + +2009-10-29 The InnoDB Team + + * os/os0proc.c: + Fix Bug#48237 Error handling in os_mem_alloc_large appears to + be incorrect + +2009-10-29 The InnoDB Team + + * buf/buf0buf.c, buf/buf0lru.c, include/buf0buf.h, include/buf0buf.ic: + Fix corruption of the buf_pool->LRU_old list and improve debug + assertions. + +2009-10-28 The InnoDB Team + + * srv/srv0start.c: + Fix Bug#41490 After enlargement of InnoDB page size, the error message + become inaccurate + +2009-10-26 The InnoDB Team + + * row/row0ins.c: + When allocating a data tuple, zero out the system fields in order + to avoid Valgrind warnings about uninitialized fields in + dtuple_validate(). + +2009-10-22 The InnoDB Team + + * handler/ha_innodb.cc, mysql-test/innodb-zip.result, + mysql-test/innodb-zip.test, mysql-test/innodb_bug44369.result, + mysql-test/innodb_bug44369.test: + Fix Bug#47233 Innodb calls push_warning(MYSQL_ERROR::WARN_LEVEL_ERROR) + +2009-10-19 The InnoDB Team + + * mysql-test/innodb_information_schema.test: + Fix Bug#47808 innodb_information_schema.test fails when run under + valgrind + +2009-10-15 The InnoDB Team + + * include/page0page.ic: + Fix Bug#47058 Failure to compile innodb_plugin on solaris 10u7 + spro + cc/CC 5.10 + +2009-10-13 The InnoDB Team + + * buf/buf0flu.c: + Call fsync() on datafiles after a batch of pages is written to disk + even when skip_innodb_doublewrite is set. + +2009-10-05 The InnoDB Team + + * buf/buf0buf.c: + Do not invalidate buffer pool while an LRU batch is active. Added code + to buf_pool_invalidate() to wait for the running batches to finish. + +2009-10-01 The InnoDB Team + + * handler/ha_innodb.cc: + Fix Bug#47763 typo in error message: Failed to open table %s after %lu + attemtps. + +2009-10-01 The InnoDB Team + + * fsp/fsp0fsp.c, row/row0merge.c: + Clean up after a crash during DROP INDEX. When InnoDB crashes + while dropping an index, ensure that the index will be completely + dropped during crash recovery. The MySQL .frm file may still + contain the dropped index, but there is little that we can do + about it. + +2009-09-28 The InnoDB Team + + * handler/ha_innodb.cc: + When a secondary index exists in the MySQL .frm file but not in + the InnoDB data dictionary, return an error instead of letting an + assertion fail in index_read. + +2009-09-28 The InnoDB Team + + * btr/btr0btr.c, buf/buf0buf.c, include/page0page.h, + include/page0zip.h, page/page0cur.c, page/page0page.c, + page/page0zip.c: + Do not write to PAGE_INDEX_ID when restoring an uncompressed page + after a compression failure. The field should only be written + when creating a B-tree page. This fix addresses a race condition + in a debug assertion. + +2009-09-28 The InnoDB Team + + * fil/fil0fil.c: + Try to prevent the reuse of tablespace identifiers after InnoDB + has crashed during table creation. Also, refuse to start if files + with duplicate tablespace identifiers are encountered. + +2009-09-25 The InnoDB Team + + * include/os0file.h, os/os0file.c: + Fix Bug#47055 unconditional exit(1) on ERROR_WORKING_SET_QUOTA + 1453 (0x5AD) for InnoDB backend + +2009-09-19 The InnoDB Team + + * handler/ha_innodb.cc, mysql-test/innodb-consistent-master.opt, + mysql-test/innodb-consistent.result, + mysql-test/innodb-consistent.test: + Fix Bug#37232 Innodb might get too many read locks for DML with + repeatable-read + +2009-09-19 The InnoDB Team + + * fsp/fsp0fsp.c: + Fix Bug#31183 Tablespace full problems not reported in error log, + error message unclear + +2009-09-17 The InnoDB Team + + * mysql-test/innodb-zip.result, mysql-test/innodb-zip.test: + Make the test pass with zlib 1.2.3.3. Apparently, the definition + of compressBound() has changed between zlib versions, and the + maximum record size of a table with 1K compressed page size has + been reduced by one byte. This is an arbitrary test. In practical + applications, for good write performance, the compressed page size + should be chosen to be bigger than the absolute minimum. + +2009-09-16 The InnoDB Team + + * handler/ha_innodb.cc: + Fix Bug#46256 drop table with unknown collation crashes innodb + +2009-09-16 The InnoDB Team + + * dict/dict0dict.c, handler/ha_innodb.cc, + mysql-test/innodb_bug44369.result, mysql-test/innodb_bug44369.test, + row/row0mysql.c: + Fix Bug#44369 InnoDB: Does not uniformly disallow disallowed column + names + +2009-09-16 The InnoDB Team + + * handler/ha_innodb.cc, include/db0err.h, + mysql-test/innodb_bug46000.result, mysql-test/innodb_bug46000.test: + Fix Bug#46000 using index called GEN_CLUST_INDEX crashes server + +2009-09-02 The InnoDB Team + + * include/lock0lock.h, include/row0mysql.h, lock/lock0lock.c, + row/row0mysql.c: + Fix a regression introduced by the fix for MySQL bug#26316. We check + whether a transaction holds any AUTOINC locks before we acquire + the kernel mutex and release those locks. + +2009-08-27 The InnoDB Team + + * dict/dict0dict.c, include/dict0dict.h, + mysql-test/innodb_bug44571.result, mysql-test/innodb_bug44571.test: + Fix Bug#44571 InnoDB Plugin crashes on ADD INDEX + +2009-08-27 The InnoDB Team + + * row/row0merge.c: + Fix a bug in the merge sort that can corrupt indexes in fast index + creation. Add some consistency checks. Check that the number of + records remains constant in every merge sort pass. + +2009-08-27 The InnoDB Team + + * buf/buf0buf.c, buf/buf0lru.c, buf/buf0rea.c, handler/ha_innodb.cc, + include/buf0buf.h, include/buf0buf.ic, include/buf0lru.h, + include/ut0ut.h, ut/ut0ut.c: + Make it possible to tune the buffer pool LRU eviction policy to be + more resistant against index scans. Introduce the settable global + variables innodb_old_blocks_pct and innodb_old_blocks_time for + controlling the buffer pool eviction policy. The parameter + innodb_old_blocks_pct (5..95) controls the desired amount of "old" + blocks in the LRU list. The default is 37, corresponding to the + old fixed ratio of 3/8. Each time a block is accessed, it will be + moved to the "new" blocks if its first access was at least + innodb_old_blocks_time milliseconds ago (default 0, meaning every + block). The idea is that in index scans, blocks will be accessed + a few times within innodb_old_blocks_time, and they will remain in + the "old" section of the LRU list. Thus, when innodb_old_blocks_time + is nonzero, blocks retrieved for one-time index scans will be more + likely candidates for eviction than blocks that are accessed in + random patterns. + +2009-08-26 The InnoDB Team + + * handler/ha_innodb.cc, os/os0file.c: + Fix Bug#42885 buf_read_ahead_random, buf_read_ahead_linear counters, + thread wakeups + +2009-08-20 The InnoDB Team + + * lock/lock0lock.c: + Fix Bug#46650 Innodb assertion autoinc_lock == lock in + lock_table_remove_low on INSERT SELECT + +2009-08-13 The InnoDB Team + + * handler/handler0alter.cc: + Fix Bug#46657 InnoDB plugin: invalid read in index_merge_innodb test + (Valgrind) + +2009-08-11 The InnoDB Team + + InnoDB Plugin 1.0.4 released + 2009-07-20 The InnoDB Team * buf/buf0rea.c, handler/ha_innodb.cc, include/srv0srv.h, === modified file 'storage/xtradb/Makefile.am' --- a/storage/xtradb/Makefile.am 2009-11-13 22:53:04 +0000 +++ b/storage/xtradb/Makefile.am 2010-01-15 15:58:25 +0000 @@ -22,7 +22,7 @@ MYSQLLIBdir= $(pkglibdir) pkgplugindir= $(pkglibdir)/plugin INCLUDES= -I$(top_srcdir)/include -I$(top_builddir)/include \ -I$(top_srcdir)/regex \ - -I$(top_srcdir)/storage/xtradb/include \ + -I$(srcdir)/include \ -I$(top_srcdir)/sql \ -I$(srcdir) @ZLIB_INCLUDES@ @@ -31,7 +31,6 @@ DEFS= @DEFS@ noinst_HEADERS= \ handler/ha_innodb.h \ - handler/handler0vars.h \ handler/i_s.h \ include/btr0btr.h \ include/btr0btr.ic \ === modified file 'storage/xtradb/btr/btr0btr.c' --- a/storage/xtradb/btr/btr0btr.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/btr/btr0btr.c 2010-01-06 12:00:14 +0000 @@ -790,14 +790,21 @@ btr_create( } else { /* It is a non-ibuf tree: create a file segment for leaf pages */ - fseg_create(space, page_no, - PAGE_HEADER + PAGE_BTR_SEG_LEAF, mtr); + if (!fseg_create(space, page_no, + PAGE_HEADER + PAGE_BTR_SEG_LEAF, mtr)) { + /* Not enough space for new segment, free root + segment before return. */ + btr_free_root(space, zip_size, page_no, mtr); + + return(FIL_NULL); + } + /* The fseg create acquires a second latch on the page, therefore we must declare it: */ buf_block_dbg_add_level(block, SYNC_TREE_NODE_NEW); } - /* Create a new index page on the the allocated segment page */ + /* Create a new index page on the allocated segment page */ page_zip = buf_block_get_page_zip(block); if (UNIV_LIKELY_NULL(page_zip)) { @@ -1011,7 +1018,26 @@ btr_page_reorganize_low( (!page_zip_compress(page_zip, page, index, NULL))) { /* Restore the old page and exit. */ - buf_frame_copy(page, temp_page); + +#if defined UNIV_DEBUG || defined UNIV_ZIP_DEBUG + /* Check that the bytes that we skip are identical. */ + ut_a(!memcmp(page, temp_page, PAGE_HEADER)); + ut_a(!memcmp(PAGE_HEADER + PAGE_N_RECS + page, + PAGE_HEADER + PAGE_N_RECS + temp_page, + PAGE_DATA - (PAGE_HEADER + PAGE_N_RECS))); + ut_a(!memcmp(UNIV_PAGE_SIZE - FIL_PAGE_DATA_END + page, + UNIV_PAGE_SIZE - FIL_PAGE_DATA_END + temp_page, + FIL_PAGE_DATA_END)); +#endif /* UNIV_DEBUG || UNIV_ZIP_DEBUG */ + + memcpy(PAGE_HEADER + page, PAGE_HEADER + temp_page, + PAGE_N_RECS - PAGE_N_DIR_SLOTS); + memcpy(PAGE_DATA + page, PAGE_DATA + temp_page, + UNIV_PAGE_SIZE - PAGE_DATA - FIL_PAGE_DATA_END); + +#if defined UNIV_DEBUG || defined UNIV_ZIP_DEBUG + ut_a(!memcmp(page, temp_page, UNIV_PAGE_SIZE)); +#endif /* UNIV_DEBUG || UNIV_ZIP_DEBUG */ goto func_exit; } @@ -1902,7 +1928,7 @@ func_start: n_uniq, &heap); /* If the new record is less than the existing record - the the split in the middle will copy the existing + the split in the middle will copy the existing record to the new node. */ if (cmp_dtuple_rec(tuple, first_rec, offsets) < 0) { split_rec = page_get_middle_rec(page); === modified file 'storage/xtradb/btr/btr0sea.c' --- a/storage/xtradb/btr/btr0sea.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/btr/btr0sea.c 2010-01-06 12:00:14 +0000 @@ -175,6 +175,21 @@ btr_search_sys_create( btr_search_sys->hash_index = ha_create(hash_size, 0, 0); } +/*****************************************************************//** +Frees the adaptive search system at a database shutdown. */ +UNIV_INTERN +void +btr_search_sys_free(void) +/*=====================*/ +{ + mem_free(btr_search_latch_temp); + btr_search_latch_temp = NULL; + mem_heap_free(btr_search_sys->hash_index->heap); + hash_table_free(btr_search_sys->hash_index); + mem_free(btr_search_sys); + btr_search_sys = NULL; +} + /********************************************************************//** Disable the adaptive hash search system and empty the index. */ UNIV_INTERN @@ -957,7 +972,7 @@ btr_search_guess_on_hash( /* Increment the page get statistics though we did not really fix the page: for user info only */ - buf_pool->n_page_gets++; + buf_pool->stat.n_page_gets++; return(TRUE); === modified file 'storage/xtradb/buf/buf0buddy.c' --- a/storage/xtradb/buf/buf0buddy.c 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/buf/buf0buddy.c 2010-01-15 15:58:25 +0000 @@ -531,11 +531,10 @@ buf_buddy_relocate( UNIV_MEM_ASSERT_W(src, size); mutex = buf_page_get_mutex_enter(bpage); - ut_a(mutex); mutex_enter(&zip_free_mutex); - if (buf_page_can_relocate(bpage)) { + if (mutex && buf_page_can_relocate(bpage)) { /* Relocate the compressed page. */ ut_a(bpage->zip.data == src); memcpy(dst, src, size); @@ -563,7 +562,9 @@ success: rw_lock_x_unlock(&page_hash_latch); } - mutex_exit(mutex); + if (mutex) { + mutex_exit(mutex); + } } else if (i == buf_buddy_get_slot(sizeof(buf_page_t))) { /* This must be a buf_page_t object. */ UNIV_MEM_ASSERT_RW(src, size); === modified file 'storage/xtradb/buf/buf0buf.c' --- a/storage/xtradb/buf/buf0buf.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/buf/buf0buf.c 2010-01-28 11:35:10 +0000 @@ -51,6 +51,40 @@ Created 11/5/1995 Heikki Tuuri #include "dict0dict.h" #include "log0recv.h" #include "page0zip.h" +#include "trx0trx.h" + +/* prototypes for new functions added to ha_innodb.cc */ +trx_t* innobase_get_trx(); + +inline void _increment_page_get_statistics(buf_block_t* block, trx_t* trx) +{ + ulint block_hash; + ulint block_hash_byte; + byte block_hash_offset; + + ut_ad(block); + + if (!innobase_get_slow_log() || !trx || !trx->take_stats) + return; + + if (!trx->distinct_page_access_hash) { + trx->distinct_page_access_hash = mem_alloc(DPAH_SIZE); + memset(trx->distinct_page_access_hash, 0, DPAH_SIZE); + } + + block_hash = ut_hash_ulint((block->page.space << 20) + block->page.space + + block->page.offset, DPAH_SIZE << 3); + block_hash_byte = block_hash >> 3; + block_hash_offset = (byte) block_hash & 0x07; + if (block_hash_byte >= DPAH_SIZE) + fprintf(stderr, "!!! block_hash_byte = %lu block_hash_offset = %lu !!!\n", (unsigned long) block_hash_byte, (unsigned long) block_hash_offset); + if (block_hash_offset > 7) + fprintf(stderr, "!!! block_hash_byte = %lu block_hash_offset = %lu !!!\n", (unsigned long) block_hash_byte, (unsigned long) block_hash_offset); + if ((trx->distinct_page_access_hash[block_hash_byte] & ((byte) 0x01 << block_hash_offset)) == 0) + trx->distinct_page_access++; + trx->distinct_page_access_hash[block_hash_byte] |= (byte) 0x01 << block_hash_offset; + return; +} /* IMPLEMENTATION OF THE BUFFER POOL @@ -845,16 +879,35 @@ buf_chunk_not_freed( block = chunk->blocks; for (i = chunk->size; i--; block++) { - mutex_enter(&block->mutex); - - if (buf_block_get_state(block) == BUF_BLOCK_FILE_PAGE - && !buf_flush_ready_for_replace(&block->page)) { + ibool ready; + switch (buf_block_get_state(block)) { + case BUF_BLOCK_ZIP_FREE: + case BUF_BLOCK_ZIP_PAGE: + case BUF_BLOCK_ZIP_DIRTY: + /* The uncompressed buffer pool should never + contain compressed block descriptors. */ + ut_error; + break; + case BUF_BLOCK_NOT_USED: + case BUF_BLOCK_READY_FOR_USE: + case BUF_BLOCK_MEMORY: + case BUF_BLOCK_REMOVE_HASH: + /* Skip blocks that are not being used for + file pages. */ + break; + case BUF_BLOCK_FILE_PAGE: + mutex_enter(&block->mutex); + ready = buf_flush_ready_for_replace(&block->page); mutex_exit(&block->mutex); - return(block); - } - mutex_exit(&block->mutex); + if (!ready) { + + return(block); + } + + break; + } } return(NULL); @@ -985,8 +1038,6 @@ buf_pool_init(void) buf_pool->no_flush[i] = os_event_create(NULL); } - buf_pool->ulint_clock = 1; - /* 3. Initialize LRU fields --------------------------- */ /* All fields are initialized by mem_zalloc(). */ @@ -1024,7 +1075,11 @@ buf_pool_free(void) os_mem_free_large(chunk->mem, chunk->mem_size); } - buf_pool->n_chunks = 0; + mem_free(buf_pool->chunks); + hash_table_free(buf_pool->page_hash); + hash_table_free(buf_pool->zip_hash); + mem_free(buf_pool); + buf_pool = NULL; } /********************************************************************//** @@ -1171,10 +1226,15 @@ buf_relocate( #ifdef UNIV_LRU_DEBUG /* buf_pool->LRU_old must be the first item in the LRU list whose "old" flag is set. */ + ut_a(buf_pool->LRU_old->old); ut_a(!UT_LIST_GET_PREV(LRU, buf_pool->LRU_old) || !UT_LIST_GET_PREV(LRU, buf_pool->LRU_old)->old); ut_a(!UT_LIST_GET_NEXT(LRU, buf_pool->LRU_old) || UT_LIST_GET_NEXT(LRU, buf_pool->LRU_old)->old); + } else { + /* Check that the "old" flag is consistent in + the block and its neighbours. */ + buf_page_set_old(dpage, buf_page_is_old(dpage)); #endif /* UNIV_LRU_DEBUG */ } @@ -1510,35 +1570,8 @@ buf_pool_resize(void) } /********************************************************************//** -Moves the block to the start of the LRU list if there is a danger -that the block would drift out of the buffer pool. */ -UNIV_INLINE -void -buf_block_make_young( -/*=================*/ - buf_page_t* bpage) /*!< in: block to make younger */ -{ - ut_ad(!buf_pool_mutex_own()); - - /* Note that we read freed_page_clock's without holding any mutex: - this is allowed since the result is used only in heuristics */ - - if (buf_page_peek_if_too_old(bpage)) { - - //buf_pool_mutex_enter(); - mutex_enter(&LRU_list_mutex); - /* There has been freeing activity in the LRU list: - best to move to the head of the LRU list */ - - buf_LRU_make_block_young(bpage); - //buf_pool_mutex_exit(); - mutex_exit(&LRU_list_mutex); - } -} - -/********************************************************************//** Moves a page to the start of the buffer pool LRU list. This high-level -function can be used to prevent an important page from from slipping out of +function can be used to prevent an important page from slipping out of the buffer pool. */ UNIV_INTERN void @@ -1558,6 +1591,42 @@ buf_page_make_young( } /********************************************************************//** +Sets the time of the first access of a page and moves a page to the +start of the buffer pool LRU list if it is too old. This high-level +function can be used to prevent an important page from slipping +out of the buffer pool. */ +static +void +buf_page_set_accessed_make_young( +/*=============================*/ + buf_page_t* bpage, /*!< in/out: buffer block of a + file page */ + unsigned access_time) /*!< in: bpage->access_time + read under mutex protection, + or 0 if unknown */ +{ + ut_ad(!buf_pool_mutex_own()); + ut_a(buf_page_in_file(bpage)); + + if (buf_page_peek_if_too_old(bpage)) { + //buf_pool_mutex_enter(); + mutex_enter(&LRU_list_mutex); + buf_LRU_make_block_young(bpage); + //buf_pool_mutex_exit(); + mutex_exit(&LRU_list_mutex); + } else if (!access_time) { + ulint time_ms = ut_time_ms(); + mutex_t* block_mutex = buf_page_get_mutex_enter(bpage); + //buf_pool_mutex_enter(); + if (block_mutex) { + buf_page_set_accessed(bpage, time_ms); + mutex_exit(block_mutex); + } + //buf_pool_mutex_exit(); + } +} + +/********************************************************************//** Resets the check_index_page_at_flush field of a page if found in the buffer pool. */ UNIV_INTERN @@ -1696,11 +1765,20 @@ buf_page_get_zip( buf_page_t* bpage; mutex_t* block_mutex; ibool must_read; + unsigned access_time; + trx_t* trx = NULL; + ulint sec; + ulint ms; + ib_uint64_t start_time; + ib_uint64_t finish_time; #ifndef UNIV_LOG_DEBUG ut_ad(!ibuf_inside()); #endif - buf_pool->n_page_gets++; + if (innobase_get_slow_log()) { + trx = innobase_get_trx(); + } + buf_pool->stat.n_page_gets++; for (;;) { //buf_pool_mutex_enter(); @@ -1716,7 +1794,7 @@ lookup: //buf_pool_mutex_exit(); rw_lock_s_unlock(&page_hash_latch); - buf_read_page(space, zip_size, offset); + buf_read_page(space, zip_size, offset, trx); #if defined UNIV_DEBUG || defined UNIV_BUF_DEBUG ut_a(++buf_dbg_counter % 37 || buf_validate()); @@ -1770,14 +1848,13 @@ err_exit: got_block: must_read = buf_page_get_io_fix(bpage) == BUF_IO_READ; + access_time = buf_page_is_accessed(bpage); //buf_pool_mutex_exit(); - buf_page_set_accessed(bpage, TRUE); - mutex_exit(block_mutex); - buf_block_make_young(bpage); + buf_page_set_accessed_make_young(bpage, access_time); #ifdef UNIV_DEBUG_FILE_ACCESSES ut_a(!bpage->file_page_was_freed); @@ -1793,6 +1870,13 @@ got_block: /* Let us wait until the read operation completes */ + if (innobase_get_slow_log() && trx && trx->take_stats) + { + ut_usectime(&sec, &ms); + start_time = (ib_uint64_t)sec * 1000000 + ms; + } else { + start_time = 0; + } for (;;) { enum buf_io_fix io_fix; @@ -1807,6 +1891,12 @@ got_block: break; } } + if (innobase_get_slow_log() && trx && trx->take_stats && start_time) + { + ut_usectime(&sec, &ms); + finish_time = (ib_uint64_t)sec * 1000000 + ms; + trx->io_reads_wait_timer += (ulint)(finish_time - start_time); + } } #ifdef UNIV_IBUF_COUNT_DEBUG @@ -1870,7 +1960,7 @@ buf_zip_decompress( switch (fil_page_get_type(frame)) { case FIL_PAGE_INDEX: if (page_zip_decompress(&block->page.zip, - block->frame)) { + block->frame, TRUE)) { return(TRUE); } @@ -2058,10 +2148,15 @@ buf_page_get_gen( mtr_t* mtr) /*!< in: mini-transaction */ { buf_block_t* block; - ibool accessed; + unsigned access_time; ulint fix_type; ibool must_read; mutex_t* block_mutex; + trx_t* trx = NULL; + ulint sec; + ulint ms; + ib_uint64_t start_time; + ib_uint64_t finish_time; ut_ad(mtr); ut_ad((rw_latch == RW_S_LATCH) @@ -2075,14 +2170,16 @@ buf_page_get_gen( #ifndef UNIV_LOG_DEBUG ut_ad(!ibuf_inside() || ibuf_page(space, zip_size, offset, NULL)); #endif - buf_pool->n_page_gets++; + if (innobase_get_slow_log()) { + trx = innobase_get_trx(); + } + buf_pool->stat.n_page_gets++; loop: block = guess; //buf_pool_mutex_enter(); if (block) { block_mutex = buf_page_get_mutex_enter((buf_page_t*)block); - ut_a(block_mutex); /* If the guess is a compressed page descriptor that has been allocated by buf_buddy_alloc(), it may have @@ -2092,7 +2189,9 @@ loop: the guess may be pointing to a buffer pool chunk that has been released when resizing the buffer pool. */ - if (!buf_block_is_uncompressed(block) + if (!block_mutex) { + block = guess = NULL; + } else if (!buf_block_is_uncompressed(block) || offset != block->page.offset || space != block->page.space || buf_block_get_state(block) != BUF_BLOCK_FILE_PAGE) { @@ -2127,7 +2226,7 @@ loop2: return(NULL); } - buf_read_page(space, zip_size, offset); + buf_read_page(space, zip_size, offset, trx); #if defined UNIV_DEBUG || defined UNIV_BUF_DEBUG ut_a(++buf_dbg_counter % 37 || buf_validate()); @@ -2351,17 +2450,17 @@ wait_until_unfixed: UNIV_MEM_ASSERT_RW(&block->page, sizeof block->page); buf_block_buf_fix_inc(block, file, line); - //buf_pool_mutex_exit(); - /* Check if this is the first access to the page */ + //mutex_exit(&block->mutex); - accessed = buf_page_is_accessed(&block->page); + /* Check if this is the first access to the page */ - buf_page_set_accessed(&block->page, TRUE); + access_time = buf_page_is_accessed(&block->page); + //buf_pool_mutex_exit(); mutex_exit(block_mutex); - buf_block_make_young(&block->page); + buf_page_set_accessed_make_young(&block->page, access_time); #ifdef UNIV_DEBUG_FILE_ACCESSES ut_a(!block->page.file_page_was_freed); @@ -2379,6 +2478,13 @@ wait_until_unfixed: /* Let us wait until the read operation completes */ + if (innobase_get_slow_log() && trx && trx->take_stats) + { + ut_usectime(&sec, &ms); + start_time = (ib_uint64_t)sec * 1000000 + ms; + } else { + start_time = 0; + } for (;;) { enum buf_io_fix io_fix; @@ -2393,6 +2499,12 @@ wait_until_unfixed: break; } } + if (innobase_get_slow_log() && trx && trx->take_stats && start_time) + { + ut_usectime(&sec, &ms); + finish_time = (ib_uint64_t)sec * 1000000 + ms; + trx->io_reads_wait_timer += (ulint)(finish_time - start_time); + } } fix_type = MTR_MEMO_BUF_FIX; @@ -2414,17 +2526,21 @@ wait_until_unfixed: mtr_memo_push(mtr, block, fix_type); - if (!accessed) { + if (!access_time) { /* In the case of a first access, try to apply linear read-ahead */ - buf_read_ahead_linear(space, zip_size, offset); + buf_read_ahead_linear(space, zip_size, offset, trx); } #ifdef UNIV_IBUF_COUNT_DEBUG ut_a(ibuf_count_get(buf_block_get_space(block), buf_block_get_page_no(block)) == 0); #endif + if (innobase_get_slow_log()) { + _increment_page_get_statistics(block, trx); + } + return(block); } @@ -2444,9 +2560,10 @@ buf_page_optimistic_get_func( ulint line, /*!< in: line where called */ mtr_t* mtr) /*!< in: mini-transaction */ { - ibool accessed; + unsigned access_time; ibool success; ulint fix_type; + trx_t* trx = NULL; ut_ad(mtr && block); ut_ad((rw_latch == RW_S_LATCH) || (rw_latch == RW_X_LATCH)); @@ -2461,14 +2578,16 @@ buf_page_optimistic_get_func( } buf_block_buf_fix_inc(block, file, line); - accessed = buf_page_is_accessed(&block->page); - buf_page_set_accessed(&block->page, TRUE); mutex_exit(&block->mutex); - buf_block_make_young(&block->page); + /* Check if this is the first access to the page. + We do a dirty read on purpose, to avoid mutex contention. + This field is only used for heuristic purposes; it does not + affect correctness. */ - /* Check if this is the first access to the page */ + access_time = buf_page_is_accessed(&block->page); + buf_page_set_accessed_make_young(&block->page, access_time); ut_ad(!ibuf_inside() || ibuf_page(buf_block_get_space(block), @@ -2520,21 +2639,28 @@ buf_page_optimistic_get_func( #ifdef UNIV_DEBUG_FILE_ACCESSES ut_a(block->page.file_page_was_freed == FALSE); #endif - if (UNIV_UNLIKELY(!accessed)) { + if (innobase_get_slow_log()) { + trx = innobase_get_trx(); + } + + if (UNIV_UNLIKELY(!access_time)) { /* In the case of a first access, try to apply linear read-ahead */ buf_read_ahead_linear(buf_block_get_space(block), buf_block_get_zip_size(block), - buf_block_get_page_no(block)); + buf_block_get_page_no(block), trx); } #ifdef UNIV_IBUF_COUNT_DEBUG ut_a(ibuf_count_get(buf_block_get_space(block), buf_block_get_page_no(block)) == 0); #endif - buf_pool->n_page_gets++; + buf_pool->stat.n_page_gets++; + if (innobase_get_slow_log()) { + _increment_page_get_statistics(block, trx); + } return(TRUE); } @@ -2556,6 +2682,7 @@ buf_page_get_known_nowait( { ibool success; ulint fix_type; + trx_t* trx = NULL; ut_ad(mtr); ut_ad((rw_latch == RW_S_LATCH) || (rw_latch == RW_X_LATCH)); @@ -2581,8 +2708,24 @@ buf_page_get_known_nowait( mutex_exit(&block->mutex); - if (mode == BUF_MAKE_YOUNG) { - buf_block_make_young(&block->page); + if (mode == BUF_MAKE_YOUNG && buf_page_peek_if_too_old(&block->page)) { + //buf_pool_mutex_enter(); + mutex_enter(&LRU_list_mutex); + buf_LRU_make_block_young(&block->page); + //buf_pool_mutex_exit(); + mutex_exit(&LRU_list_mutex); + } else if (!buf_page_is_accessed(&block->page)) { + /* Above, we do a dirty read on purpose, to avoid + mutex contention. The field buf_page_t::access_time + is only used for heuristic purposes. Writes to the + field must be protected by mutex, however. */ + ulint time_ms = ut_time_ms(); + + //buf_pool_mutex_enter(); + mutex_enter(&block->mutex); + buf_page_set_accessed(&block->page, time_ms); + //buf_pool_mutex_exit(); + mutex_exit(&block->mutex); } ut_ad(!ibuf_inside() || (mode == BUF_KEEP_OLD)); @@ -2621,7 +2764,12 @@ buf_page_get_known_nowait( || (ibuf_count_get(buf_block_get_space(block), buf_block_get_page_no(block)) == 0)); #endif - buf_pool->n_page_gets++; + buf_pool->stat.n_page_gets++; + + if (innobase_get_slow_log()) { + trx = innobase_get_trx(); + _increment_page_get_statistics(block, trx); + } return(TRUE); } @@ -2700,7 +2848,7 @@ buf_page_try_get_func( #endif /* UNIV_DEBUG_FILE_ACCESSES */ buf_block_dbg_add_level(block, SYNC_NO_ORDER_CHECK); - buf_pool->n_page_gets++; + buf_pool->stat.n_page_gets++; #ifdef UNIV_IBUF_COUNT_DEBUG ut_a(ibuf_count_get(buf_block_get_space(block), @@ -2719,10 +2867,10 @@ buf_page_init_low( buf_page_t* bpage) /*!< in: block to init */ { bpage->flush_type = BUF_FLUSH_LRU; - bpage->accessed = FALSE; bpage->io_fix = BUF_IO_NONE; bpage->buf_fix_count = 0; bpage->freed_page_clock = 0; + bpage->access_time = 0; bpage->newest_modification = 0; bpage->oldest_modification = 0; HASH_INVALIDATE(bpage, hash); @@ -3044,6 +3192,7 @@ buf_page_create( buf_frame_t* frame; buf_block_t* block; buf_block_t* free_block = NULL; + ulint time_ms = ut_time_ms(); ut_ad(mtr); ut_ad(space || !zip_size); @@ -3095,7 +3244,7 @@ buf_page_create( buf_LRU_add_block(&block->page, FALSE); buf_block_buf_fix_inc(block, __FILE__, __LINE__); - buf_pool->n_pages_created++; + buf_pool->stat.n_pages_created++; if (zip_size) { void* data; @@ -3132,13 +3281,13 @@ buf_page_create( rw_lock_x_unlock(&block->lock); } + buf_page_set_accessed(&block->page, time_ms); + //buf_pool_mutex_exit(); mutex_exit(&LRU_list_mutex); mtr_memo_push(mtr, block, MTR_MEMO_BUF_FIX); - buf_page_set_accessed(&block->page, TRUE); - mutex_exit(&block->mutex); /* Delete possible entries for the page from the insert buffer: @@ -3355,7 +3504,7 @@ corrupt: ut_ad(buf_pool->n_pend_reads > 0); buf_pool->n_pend_reads--; - buf_pool->n_pages_read++; + buf_pool->stat.n_pages_read++; if (uncompressed) { rw_lock_x_unlock_gen(&((buf_block_t*) bpage)->lock, @@ -3380,7 +3529,7 @@ corrupt: BUF_IO_WRITE); } - buf_pool->n_pages_written++; + buf_pool->stat.n_pages_written++; break; @@ -3411,7 +3560,32 @@ void buf_pool_invalidate(void) /*=====================*/ { - ibool freed; + ibool freed; + enum buf_flush i; + + buf_pool_mutex_enter(); + + for (i = BUF_FLUSH_LRU; i < BUF_FLUSH_N_TYPES; i++) { + + /* As this function is called during startup and + during redo application phase during recovery, InnoDB + is single threaded (apart from IO helper threads) at + this stage. No new write batch can be in intialization + stage at this point. */ + ut_ad(buf_pool->init_flush[i] == FALSE); + + /* However, it is possible that a write batch that has + been posted earlier is still not complete. For buffer + pool invalidation to proceed we must ensure there is NO + write activity happening. */ + if (buf_pool->n_flush[i] > 0) { + buf_pool_mutex_exit(); + buf_flush_wait_batch_end(i); + buf_pool_mutex_enter(); + } + } + + buf_pool_mutex_exit(); ut_ad(buf_all_freed()); @@ -3427,6 +3601,14 @@ buf_pool_invalidate(void) ut_ad(UT_LIST_GET_LEN(buf_pool->LRU) == 0); ut_ad(UT_LIST_GET_LEN(buf_pool->unzip_LRU) == 0); + buf_pool->freed_page_clock = 0; + buf_pool->LRU_old = NULL; + buf_pool->LRU_old_len = 0; + buf_pool->LRU_flush_ended = 0; + + memset(&buf_pool->stat, 0x00, sizeof(buf_pool->stat)); + buf_refresh_io_stats(); + //buf_pool_mutex_exit(); mutex_exit(&LRU_list_mutex); } @@ -3706,6 +3888,7 @@ buf_print(void) "n pending decompressions %lu\n" "n pending reads %lu\n" "n pending flush LRU %lu list %lu single page %lu\n" + "pages made young %lu, not young %lu\n" "pages read %lu, created %lu, written %lu\n", (ulong) size, (ulong) UT_LIST_GET_LEN(buf_pool->LRU), @@ -3716,8 +3899,11 @@ buf_print(void) (ulong) buf_pool->n_flush[BUF_FLUSH_LRU], (ulong) buf_pool->n_flush[BUF_FLUSH_LIST], (ulong) buf_pool->n_flush[BUF_FLUSH_SINGLE_PAGE], - (ulong) buf_pool->n_pages_read, buf_pool->n_pages_created, - (ulong) buf_pool->n_pages_written); + (ulong) buf_pool->stat.n_pages_made_young, + (ulong) buf_pool->stat.n_pages_not_made_young, + (ulong) buf_pool->stat.n_pages_read, + (ulong) buf_pool->stat.n_pages_created, + (ulong) buf_pool->stat.n_pages_written); /* Count the number of blocks belonging to each index in the buffer */ @@ -3927,10 +4113,9 @@ buf_print_io( { time_t current_time; double time_elapsed; - ulint size; + ulint n_gets_diff; ut_ad(buf_pool); - size = buf_pool->curr_size; //buf_pool_mutex_enter(); mutex_enter(&LRU_list_mutex); @@ -3943,13 +4128,15 @@ buf_print_io( "Buffer pool size, bytes %lu\n" "Free buffers %lu\n" "Database pages %lu\n" + "Old database pages %lu\n" "Modified db pages %lu\n" "Pending reads %lu\n" "Pending writes: LRU %lu, flush list %lu, single page %lu\n", - (ulong) size, - (ulong) size * UNIV_PAGE_SIZE, + (ulong) buf_pool->curr_size, + (ulong) buf_pool->curr_size * UNIV_PAGE_SIZE, (ulong) UT_LIST_GET_LEN(buf_pool->free), (ulong) UT_LIST_GET_LEN(buf_pool->LRU), + (ulong) buf_pool->LRU_old_len, (ulong) UT_LIST_GET_LEN(buf_pool->flush_list), (ulong) buf_pool->n_pend_reads, (ulong) buf_pool->n_flush[BUF_FLUSH_LRU] @@ -3961,37 +4148,66 @@ buf_print_io( current_time = time(NULL); time_elapsed = 0.001 + difftime(current_time, buf_pool->last_printout_time); - buf_pool->last_printout_time = current_time; fprintf(file, + "Pages made young %lu, not young %lu\n" + "%.2f youngs/s, %.2f non-youngs/s\n" "Pages read %lu, created %lu, written %lu\n" "%.2f reads/s, %.2f creates/s, %.2f writes/s\n", - (ulong) buf_pool->n_pages_read, - (ulong) buf_pool->n_pages_created, - (ulong) buf_pool->n_pages_written, - (buf_pool->n_pages_read - buf_pool->n_pages_read_old) + (ulong) buf_pool->stat.n_pages_made_young, + (ulong) buf_pool->stat.n_pages_not_made_young, + (buf_pool->stat.n_pages_made_young + - buf_pool->old_stat.n_pages_made_young) + / time_elapsed, + (buf_pool->stat.n_pages_not_made_young + - buf_pool->old_stat.n_pages_not_made_young) / time_elapsed, - (buf_pool->n_pages_created - buf_pool->n_pages_created_old) + (ulong) buf_pool->stat.n_pages_read, + (ulong) buf_pool->stat.n_pages_created, + (ulong) buf_pool->stat.n_pages_written, + (buf_pool->stat.n_pages_read + - buf_pool->old_stat.n_pages_read) / time_elapsed, - (buf_pool->n_pages_written - buf_pool->n_pages_written_old) + (buf_pool->stat.n_pages_created + - buf_pool->old_stat.n_pages_created) + / time_elapsed, + (buf_pool->stat.n_pages_written + - buf_pool->old_stat.n_pages_written) / time_elapsed); - if (buf_pool->n_page_gets > buf_pool->n_page_gets_old) { - fprintf(file, "Buffer pool hit rate %lu / 1000\n", + n_gets_diff = buf_pool->stat.n_page_gets - buf_pool->old_stat.n_page_gets; + + if (n_gets_diff) { + fprintf(file, + "Buffer pool hit rate %lu / 1000," + " young-making rate %lu / 1000 not %lu / 1000\n", (ulong) - (1000 - ((1000 * (buf_pool->n_pages_read - - buf_pool->n_pages_read_old)) - / (buf_pool->n_page_gets - - buf_pool->n_page_gets_old)))); + (1000 - ((1000 * (buf_pool->stat.n_pages_read + - buf_pool->old_stat.n_pages_read)) + / (buf_pool->stat.n_page_gets + - buf_pool->old_stat.n_page_gets))), + (ulong) + (1000 * (buf_pool->stat.n_pages_made_young + - buf_pool->old_stat.n_pages_made_young) + / n_gets_diff), + (ulong) + (1000 * (buf_pool->stat.n_pages_not_made_young + - buf_pool->old_stat.n_pages_not_made_young) + / n_gets_diff)); } else { fputs("No buffer pool page gets since the last printout\n", file); } - buf_pool->n_page_gets_old = buf_pool->n_page_gets; - buf_pool->n_pages_read_old = buf_pool->n_pages_read; - buf_pool->n_pages_created_old = buf_pool->n_pages_created; - buf_pool->n_pages_written_old = buf_pool->n_pages_written; + /* Statistics about read ahead algorithm */ + fprintf(file, "Pages read ahead %.2f/s," + " evicted without access %.2f/s\n", + (buf_pool->stat.n_ra_pages_read + - buf_pool->old_stat.n_ra_pages_read) + / time_elapsed, + (buf_pool->stat.n_ra_pages_evicted + - buf_pool->old_stat.n_ra_pages_evicted) + / time_elapsed); /* Print some values to help us with visualizing what is happening with LRU eviction. */ @@ -4003,6 +4219,7 @@ buf_print_io( buf_LRU_stat_sum.io, buf_LRU_stat_cur.io, buf_LRU_stat_sum.unzip, buf_LRU_stat_cur.unzip); + buf_refresh_io_stats(); //buf_pool_mutex_exit(); mutex_exit(&LRU_list_mutex); mutex_exit(&free_list_mutex); @@ -4018,10 +4235,7 @@ buf_refresh_io_stats(void) /*======================*/ { buf_pool->last_printout_time = time(NULL); - buf_pool->n_page_gets_old = buf_pool->n_page_gets; - buf_pool->n_pages_read_old = buf_pool->n_pages_read; - buf_pool->n_pages_created_old = buf_pool->n_pages_created; - buf_pool->n_pages_written_old = buf_pool->n_pages_written; + buf_pool->old_stat = buf_pool->stat; } /*********************************************************************//** === modified file 'storage/xtradb/buf/buf0flu.c' --- a/storage/xtradb/buf/buf0flu.c 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/buf/buf0flu.c 2010-01-15 15:58:25 +0000 @@ -331,6 +331,28 @@ buf_flush_write_complete( } /********************************************************************//** +Flush a batch of writes to the datafiles that have already been +written by the OS. */ +static +void +buf_flush_sync_datafiles(void) +/*==========================*/ +{ + /* Wake possible simulated aio thread to actually post the + writes to the operating system */ + os_aio_simulated_wake_handler_threads(); + + /* Wait that all async writes to tablespaces have been posted to + the OS */ + os_aio_wait_until_no_pending_writes(); + + /* Now we flush the data to disk (for example, with fsync) */ + fil_flush_file_spaces(FIL_TABLESPACE); + + return; +} + +/********************************************************************//** Flushes possible buffered writes from the doublewrite memory buffer to disk, and also wakes up the aio thread if simulated aio is used. It is very important to call this function after a batch of writes has been posted, @@ -347,8 +369,8 @@ buf_flush_buffered_writes(void) ulint i; if (!srv_use_doublewrite_buf || trx_doublewrite == NULL) { - os_aio_simulated_wake_handler_threads(); - + /* Sync the writes to the disk. */ + buf_flush_sync_datafiles(); return; } @@ -556,22 +578,10 @@ flush: buf_LRU_stat_inc_io(); } - /* Wake possible simulated aio thread to actually post the - writes to the operating system */ - - os_aio_simulated_wake_handler_threads(); - - /* Wait that all async writes to tablespaces have been posted to - the OS */ - - os_aio_wait_until_no_pending_writes(); - - /* Now we flush the data to disk (for example, with fsync) */ - - fil_flush_file_spaces(FIL_TABLESPACE); + /* Sync the writes to the disk. */ + buf_flush_sync_datafiles(); /* We can now reuse the doublewrite memory buffer: */ - trx_doublewrite->first_free = 0; mutex_exit(&(trx_doublewrite->mutex)); @@ -994,9 +1004,7 @@ buf_flush_try_neighbors( || buf_page_is_old(bpage)) { mutex_t* block_mutex = buf_page_get_mutex_enter(bpage); - ut_a(block_mutex); - - if (buf_flush_ready_for_flush(bpage, flush_type) + if (block_mutex && buf_flush_ready_for_flush(bpage, flush_type) && (i == offset || !bpage->buf_fix_count)) { /* We only try to flush those neighbors != offset where the buf fix count is @@ -1012,7 +1020,7 @@ buf_flush_try_neighbors( //buf_pool_mutex_enter(); rw_lock_s_lock(&page_hash_latch); - } else { + } else if (block_mutex) { mutex_exit(block_mutex); } } @@ -1050,6 +1058,7 @@ buf_flush_batch( min_n), otherwise ignored */ { buf_page_t* bpage; + buf_page_t* prev_bpage = NULL; ulint page_count = 0; ulint old_page_count; ulint space; @@ -1103,6 +1112,9 @@ flush_next: mutex_enter(&flush_list_mutex); remaining = UT_LIST_GET_LEN(buf_pool->flush_list); bpage = UT_LIST_GET_LAST(buf_pool->flush_list); + if (bpage) { + prev_bpage = UT_LIST_GET_PREV(flush_list, bpage); + } mutex_exit(&flush_list_mutex); if (!bpage || bpage->oldest_modification >= lsn_limit) { @@ -1123,11 +1135,14 @@ flush_next: mutex_t*block_mutex = buf_page_get_mutex_enter(bpage); ibool ready; - ut_a(buf_page_in_file(bpage)); + //ut_a(buf_page_in_file(bpage)); - ut_a(block_mutex); - ready = buf_flush_ready_for_flush(bpage, flush_type); - mutex_exit(block_mutex); + if (block_mutex) { + ready = buf_flush_ready_for_flush(bpage, flush_type); + mutex_exit(block_mutex); + } else { + ready = FALSE; + } if (ready) { space = buf_page_get_space(bpage); @@ -1162,6 +1177,13 @@ flush_next: mutex_enter(&flush_list_mutex); bpage = UT_LIST_GET_PREV(flush_list, bpage); //ut_ad(!bpage || bpage->in_flush_list); /* optimistic */ + if (bpage != prev_bpage) { + /* the search may warp.. retrying */ + bpage = NULL; + } + if (bpage) { + prev_bpage = UT_LIST_GET_PREV(flush_list, bpage); + } mutex_exit(&flush_list_mutex); remaining--; } @@ -1271,13 +1293,13 @@ buf_flush_LRU_recommendation(void) } block_mutex = buf_page_get_mutex_enter(bpage); - ut_a(block_mutex); - - if (buf_flush_ready_for_replace(bpage)) { + if (block_mutex && buf_flush_ready_for_replace(bpage)) { n_replaceable++; } - mutex_exit(block_mutex); + if (block_mutex) { + mutex_exit(block_mutex); + } distance++; === modified file 'storage/xtradb/buf/buf0lru.c' --- a/storage/xtradb/buf/buf0lru.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/buf/buf0lru.c 2010-01-06 12:00:14 +0000 @@ -49,18 +49,22 @@ Created 11/5/1995 Heikki Tuuri #include "log0recv.h" #include "srv0srv.h" -/** The number of blocks from the LRU_old pointer onward, including the block -pointed to, must be 3/8 of the whole LRU list length, except that the -tolerance defined below is allowed. Note that the tolerance must be small -enough such that for even the BUF_LRU_OLD_MIN_LEN long LRU list, the -LRU_old pointer is not allowed to point to either end of the LRU list. */ +/** The number of blocks from the LRU_old pointer onward, including +the block pointed to, must be buf_LRU_old_ratio/BUF_LRU_OLD_RATIO_DIV +of the whole LRU list length, except that the tolerance defined below +is allowed. Note that the tolerance must be small enough such that for +even the BUF_LRU_OLD_MIN_LEN long LRU list, the LRU_old pointer is not +allowed to point to either end of the LRU list. */ #define BUF_LRU_OLD_TOLERANCE 20 -/** The whole LRU list length is divided by this number to determine an -initial segment in buf_LRU_get_recent_limit */ - -#define BUF_LRU_INITIAL_RATIO 8 +/** The minimum amount of non-old blocks when the LRU_old list exists +(that is, when there are more than BUF_LRU_OLD_MIN_LEN blocks). +@see buf_LRU_old_adjust_len */ +#define BUF_LRU_NON_OLD_MIN_LEN 5 +#if BUF_LRU_NON_OLD_MIN_LEN >= BUF_LRU_OLD_MIN_LEN +# error "BUF_LRU_NON_OLD_MIN_LEN >= BUF_LRU_OLD_MIN_LEN" +#endif /** When dropping the search hash index entries before deleting an ibd file, we build a local array of pages belonging to that tablespace @@ -107,6 +111,15 @@ UNIV_INTERN buf_LRU_stat_t buf_LRU_stat_ /* @} */ +/** @name Heuristics for detecting index scan @{ */ +/** Reserve this much/BUF_LRU_OLD_RATIO_DIV of the buffer pool for +"old" blocks. Protected by buf_pool_mutex. */ +UNIV_INTERN uint buf_LRU_old_ratio; +/** Move blocks to "new" LRU list only if the first access was at +least this many milliseconds ago. Not protected by any mutex or latch. */ +UNIV_INTERN uint buf_LRU_old_threshold_ms; +/* @} */ + /******************************************************************//** Takes a block out of the LRU list and page hash table. If the block is compressed-only (BUF_BLOCK_ZIP_PAGE), @@ -255,9 +268,12 @@ scan_again: mutex_t* block_mutex = buf_page_get_mutex_enter(bpage); buf_page_t* prev_bpage; - ut_a(block_mutex); prev_bpage = UT_LIST_GET_PREV(LRU, bpage); + if (!block_mutex) { + goto next_page; + } + ut_a(buf_page_in_file(bpage)); if (buf_page_get_state(bpage) != BUF_BLOCK_FILE_PAGE @@ -360,9 +376,13 @@ scan_again: ut_a(buf_page_in_file(bpage)); - ut_a(block_mutex); prev_bpage = UT_LIST_GET_PREV(LRU, bpage); + if (!block_mutex) { + bpage = prev_bpage; + continue; + } + if (buf_page_get_space(bpage) == id) { if (bpage->buf_fix_count > 0 || buf_page_get_io_fix(bpage) != BUF_IO_NONE) { @@ -450,45 +470,6 @@ next_page: } } -/******************************************************************//** -Gets the minimum LRU_position field for the blocks in an initial segment -(determined by BUF_LRU_INITIAL_RATIO) of the LRU list. The limit is not -guaranteed to be precise, because the ulint_clock may wrap around. -@return the limit; zero if could not determine it */ -UNIV_INTERN -ulint -buf_LRU_get_recent_limit(void) -/*==========================*/ -{ - const buf_page_t* bpage; - ulint len; - ulint limit; - - //buf_pool_mutex_enter(); - mutex_enter(&LRU_list_mutex); - - len = UT_LIST_GET_LEN(buf_pool->LRU); - - if (len < BUF_LRU_OLD_MIN_LEN) { - /* The LRU list is too short to do read-ahead */ - - //buf_pool_mutex_exit(); - mutex_exit(&LRU_list_mutex); - - return(0); - } - - bpage = UT_LIST_GET_FIRST(buf_pool->LRU); - - limit = buf_page_get_LRU_position(bpage); - len /= BUF_LRU_INITIAL_RATIO; - - //buf_pool_mutex_exit(); - mutex_exit(&LRU_list_mutex); - - return(limit > len ? (limit - len) : 0); -} - /********************************************************************//** Insert a compressed block into buf_pool->zip_clean in the LRU order. */ UNIV_INTERN @@ -631,10 +612,13 @@ restart: bpage = UT_LIST_GET_PREV(LRU, bpage), distance--) { enum buf_lru_free_block_status freed; + unsigned accessed; mutex_t* block_mutex = buf_page_get_mutex_enter(bpage); - ut_a(block_mutex); + if (!block_mutex) { + goto restart; + } if (!bpage->in_LRU_list || !buf_page_in_file(bpage)) { @@ -645,11 +629,18 @@ restart: ut_ad(buf_page_in_file(bpage)); ut_ad(bpage->in_LRU_list); + accessed = buf_page_is_accessed(bpage); freed = buf_LRU_free_block(bpage, TRUE, NULL, have_LRU_mutex); mutex_exit(block_mutex); switch (freed) { case BUF_LRU_FREED: + /* Keep track of pages that are evicted without + ever being accessed. This gives us a measure of + the effectiveness of readahead */ + if (!accessed) { + ++buf_pool->stat.n_ra_pages_evicted; + } return(TRUE); case BUF_LRU_NOT_FREED: @@ -1027,8 +1018,10 @@ buf_LRU_old_adjust_len(void) ut_a(buf_pool->LRU_old); //ut_ad(buf_pool_mutex_own()); ut_ad(mutex_own(&LRU_list_mutex)); -#if 3 * (BUF_LRU_OLD_MIN_LEN / 8) <= BUF_LRU_OLD_TOLERANCE + 5 -# error "3 * (BUF_LRU_OLD_MIN_LEN / 8) <= BUF_LRU_OLD_TOLERANCE + 5" + ut_ad(buf_LRU_old_ratio >= BUF_LRU_OLD_RATIO_MIN); + ut_ad(buf_LRU_old_ratio <= BUF_LRU_OLD_RATIO_MAX); +#if BUF_LRU_OLD_RATIO_MIN * BUF_LRU_OLD_MIN_LEN <= BUF_LRU_OLD_RATIO_DIV * (BUF_LRU_OLD_TOLERANCE + 5) +# error "BUF_LRU_OLD_RATIO_MIN * BUF_LRU_OLD_MIN_LEN <= BUF_LRU_OLD_RATIO_DIV * (BUF_LRU_OLD_TOLERANCE + 5)" #endif #ifdef UNIV_LRU_DEBUG /* buf_pool->LRU_old must be the first item in the LRU list @@ -1040,34 +1033,39 @@ buf_LRU_old_adjust_len(void) || UT_LIST_GET_NEXT(LRU, buf_pool->LRU_old)->old); #endif /* UNIV_LRU_DEBUG */ + old_len = buf_pool->LRU_old_len; + new_len = ut_min(UT_LIST_GET_LEN(buf_pool->LRU) + * buf_LRU_old_ratio / BUF_LRU_OLD_RATIO_DIV, + UT_LIST_GET_LEN(buf_pool->LRU) + - (BUF_LRU_OLD_TOLERANCE + + BUF_LRU_NON_OLD_MIN_LEN)); + for (;;) { - old_len = buf_pool->LRU_old_len; - new_len = 3 * (UT_LIST_GET_LEN(buf_pool->LRU) / 8); + buf_page_t* LRU_old = buf_pool->LRU_old; - ut_ad(buf_pool->LRU_old->in_LRU_list); - ut_a(buf_pool->LRU_old); + ut_a(LRU_old); + ut_ad(LRU_old->in_LRU_list); #ifdef UNIV_LRU_DEBUG - ut_a(buf_pool->LRU_old->old); + ut_a(LRU_old->old); #endif /* UNIV_LRU_DEBUG */ /* Update the LRU_old pointer if necessary */ - if (old_len < new_len - BUF_LRU_OLD_TOLERANCE) { + if (old_len + BUF_LRU_OLD_TOLERANCE < new_len) { - buf_pool->LRU_old = UT_LIST_GET_PREV( - LRU, buf_pool->LRU_old); + buf_pool->LRU_old = LRU_old = UT_LIST_GET_PREV( + LRU, LRU_old); #ifdef UNIV_LRU_DEBUG - ut_a(!buf_pool->LRU_old->old); + ut_a(!LRU_old->old); #endif /* UNIV_LRU_DEBUG */ - buf_page_set_old(buf_pool->LRU_old, TRUE); - buf_pool->LRU_old_len++; + old_len = ++buf_pool->LRU_old_len; + buf_page_set_old(LRU_old, TRUE); } else if (old_len > new_len + BUF_LRU_OLD_TOLERANCE) { - buf_page_set_old(buf_pool->LRU_old, FALSE); - buf_pool->LRU_old = UT_LIST_GET_NEXT( - LRU, buf_pool->LRU_old); - buf_pool->LRU_old_len--; + buf_pool->LRU_old = UT_LIST_GET_NEXT(LRU, LRU_old); + old_len = --buf_pool->LRU_old_len; + buf_page_set_old(LRU_old, FALSE); } else { return; } @@ -1092,12 +1090,13 @@ buf_LRU_old_init(void) the adjust function to move the LRU_old pointer to the right position */ - bpage = UT_LIST_GET_FIRST(buf_pool->LRU); - - while (bpage != NULL) { + for (bpage = UT_LIST_GET_LAST(buf_pool->LRU); bpage != NULL; + bpage = UT_LIST_GET_PREV(LRU, bpage)) { ut_ad(bpage->in_LRU_list); - buf_page_set_old(bpage, TRUE); - bpage = UT_LIST_GET_NEXT(LRU, bpage); + ut_ad(buf_page_in_file(bpage)); + /* This loop temporarily violates the + assertions of buf_page_set_old(). */ + bpage->old = TRUE; } buf_pool->LRU_old = UT_LIST_GET_FIRST(buf_pool->LRU); @@ -1152,16 +1151,19 @@ buf_LRU_remove_block( if (UNIV_UNLIKELY(bpage == buf_pool->LRU_old)) { - /* Below: the previous block is guaranteed to exist, because - the LRU_old pointer is only allowed to differ by the - tolerance value from strict 3/8 of the LRU list length. */ + /* Below: the previous block is guaranteed to exist, + because the LRU_old pointer is only allowed to differ + by BUF_LRU_OLD_TOLERANCE from strict + buf_LRU_old_ratio/BUF_LRU_OLD_RATIO_DIV of the LRU + list length. */ + buf_page_t* prev_bpage = UT_LIST_GET_PREV(LRU, bpage); - buf_pool->LRU_old = UT_LIST_GET_PREV(LRU, bpage); - ut_a(buf_pool->LRU_old); + ut_a(prev_bpage); #ifdef UNIV_LRU_DEBUG - ut_a(!buf_pool->LRU_old->old); + ut_a(!prev_bpage->old); #endif /* UNIV_LRU_DEBUG */ - buf_page_set_old(buf_pool->LRU_old, TRUE); + buf_pool->LRU_old = prev_bpage; + buf_page_set_old(prev_bpage, TRUE); buf_pool->LRU_old_len++; } @@ -1172,10 +1174,19 @@ buf_LRU_remove_block( buf_unzip_LRU_remove_block_if_needed(bpage); - /* If the LRU list is so short that LRU_old not defined, return */ + /* If the LRU list is so short that LRU_old is not defined, + clear the "old" flags and return */ if (UT_LIST_GET_LEN(buf_pool->LRU) < BUF_LRU_OLD_MIN_LEN) { + for (bpage = UT_LIST_GET_FIRST(buf_pool->LRU); bpage != NULL; + bpage = UT_LIST_GET_NEXT(LRU, bpage)) { + /* This loop temporarily violates the + assertions of buf_page_set_old(). */ + bpage->old = FALSE; + } + buf_pool->LRU_old = NULL; + buf_pool->LRU_old_len = 0; return; } @@ -1227,8 +1238,6 @@ buf_LRU_add_block_to_end_low( /*=========================*/ buf_page_t* bpage) /*!< in: control block */ { - buf_page_t* last_bpage; - ut_ad(buf_pool); ut_ad(bpage); //ut_ad(buf_pool_mutex_own()); @@ -1236,31 +1245,18 @@ buf_LRU_add_block_to_end_low( ut_a(buf_page_in_file(bpage)); - last_bpage = UT_LIST_GET_LAST(buf_pool->LRU); - - if (last_bpage) { - bpage->LRU_position = last_bpage->LRU_position; - } else { - bpage->LRU_position = buf_pool_clock_tic(); - } - ut_ad(!bpage->in_LRU_list); UT_LIST_ADD_LAST(LRU, buf_pool->LRU, bpage); bpage->in_LRU_list = TRUE; - buf_page_set_old(bpage, TRUE); - - if (UT_LIST_GET_LEN(buf_pool->LRU) >= BUF_LRU_OLD_MIN_LEN) { - - buf_pool->LRU_old_len++; - } - if (UT_LIST_GET_LEN(buf_pool->LRU) > BUF_LRU_OLD_MIN_LEN) { ut_ad(buf_pool->LRU_old); /* Adjust the length of the old block list if necessary */ + buf_page_set_old(bpage, TRUE); + buf_pool->LRU_old_len++; buf_LRU_old_adjust_len(); } else if (UT_LIST_GET_LEN(buf_pool->LRU) == BUF_LRU_OLD_MIN_LEN) { @@ -1269,6 +1265,8 @@ buf_LRU_add_block_to_end_low( defined: init it */ buf_LRU_old_init(); + } else { + buf_page_set_old(bpage, buf_pool->LRU_old != NULL); } /* If this is a zipped block with decompressed frame as well @@ -1302,7 +1300,6 @@ buf_LRU_add_block_low( UT_LIST_ADD_FIRST(LRU, buf_pool->LRU, bpage); - bpage->LRU_position = buf_pool_clock_tic(); bpage->freed_page_clock = buf_pool->freed_page_clock; } else { #ifdef UNIV_LRU_DEBUG @@ -1317,23 +1314,17 @@ buf_LRU_add_block_low( UT_LIST_INSERT_AFTER(LRU, buf_pool->LRU, buf_pool->LRU_old, bpage); buf_pool->LRU_old_len++; - - /* We copy the LRU position field of the previous block - to the new block */ - - bpage->LRU_position = (buf_pool->LRU_old)->LRU_position; } bpage->in_LRU_list = TRUE; - buf_page_set_old(bpage, old); - if (UT_LIST_GET_LEN(buf_pool->LRU) > BUF_LRU_OLD_MIN_LEN) { ut_ad(buf_pool->LRU_old); /* Adjust the length of the old block list if necessary */ + buf_page_set_old(bpage, old); buf_LRU_old_adjust_len(); } else if (UT_LIST_GET_LEN(buf_pool->LRU) == BUF_LRU_OLD_MIN_LEN) { @@ -1342,6 +1333,8 @@ buf_LRU_add_block_low( defined: init it */ buf_LRU_old_init(); + } else { + buf_page_set_old(bpage, buf_pool->LRU_old != NULL); } /* If this is a zipped block with decompressed frame as well @@ -1375,6 +1368,13 @@ buf_LRU_make_block_young( /*=====================*/ buf_page_t* bpage) /*!< in: control block */ { + //ut_ad(buf_pool_mutex_own()); + ut_ad(mutex_own(&LRU_list_mutex)); + + if (bpage->old) { + buf_pool->stat.n_pages_made_young++; + } + buf_LRU_remove_block(bpage); buf_LRU_add_block_low(bpage, FALSE); } @@ -1571,15 +1571,6 @@ not_freed: buf_pool->LRU_old = b; } -#ifdef UNIV_LRU_DEBUG - ut_a(prev_b->old - || !UT_LIST_GET_NEXT(LRU, b) - || UT_LIST_GET_NEXT(LRU, b)->old); - } else { - ut_a(!prev_b->old - || !UT_LIST_GET_NEXT(LRU, b) - || !UT_LIST_GET_NEXT(LRU, b)->old); -#endif /* UNIV_LRU_DEBUG */ } lru_len = UT_LIST_GET_LEN(buf_pool->LRU); @@ -1595,6 +1586,11 @@ not_freed: defined: init it */ buf_LRU_old_init(); } +#ifdef UNIV_LRU_DEBUG + /* Check that the "old" flag is consistent + in the block and its neighbours. */ + buf_page_set_old(b, buf_page_is_old(b)); +#endif /* UNIV_LRU_DEBUG */ } else { b->in_LRU_list = FALSE; buf_LRU_add_block_low(b, buf_page_is_old(b)); @@ -1985,6 +1981,52 @@ buf_LRU_block_free_hashed_page( buf_LRU_block_free_non_file_page(block, have_page_hash_mutex); } +/**********************************************************************//** +Updates buf_LRU_old_ratio. +@return updated old_pct */ +UNIV_INTERN +uint +buf_LRU_old_ratio_update( +/*=====================*/ + uint old_pct,/*!< in: Reserve this percentage of + the buffer pool for "old" blocks. */ + ibool adjust) /*!< in: TRUE=adjust the LRU list; + FALSE=just assign buf_LRU_old_ratio + during the initialization of InnoDB */ +{ + uint ratio; + + ratio = old_pct * BUF_LRU_OLD_RATIO_DIV / 100; + if (ratio < BUF_LRU_OLD_RATIO_MIN) { + ratio = BUF_LRU_OLD_RATIO_MIN; + } else if (ratio > BUF_LRU_OLD_RATIO_MAX) { + ratio = BUF_LRU_OLD_RATIO_MAX; + } + + if (adjust) { + //buf_pool_mutex_enter(); + mutex_enter(&LRU_list_mutex); + + if (ratio != buf_LRU_old_ratio) { + buf_LRU_old_ratio = ratio; + + if (UT_LIST_GET_LEN(buf_pool->LRU) + >= BUF_LRU_OLD_MIN_LEN) { + buf_LRU_old_adjust_len(); + } + } + + //buf_pool_mutex_exit(); + mutex_exit(&LRU_list_mutex); + } else { + buf_LRU_old_ratio = ratio; + } + + /* the reverse of + ratio = old_pct * BUF_LRU_OLD_RATIO_DIV / 100 */ + return((uint) (ratio * 100 / (double) BUF_LRU_OLD_RATIO_DIV + 0.5)); +} + /********************************************************************//** Update the historical stats that we are collecting for LRU eviction policy at the end of each interval. */ @@ -2023,6 +2065,218 @@ func_exit: memset(&buf_LRU_stat_cur, 0, sizeof buf_LRU_stat_cur); } +/********************************************************************//** +Dump the LRU page list to the specific file. */ +#define LRU_DUMP_FILE "ib_lru_dump" + +UNIV_INTERN +ibool +buf_LRU_file_dump(void) +/*===================*/ +{ + os_file_t dump_file = -1; + ibool success; + byte* buffer_base = NULL; + byte* buffer = NULL; + buf_page_t* bpage; + ulint buffers; + ulint offset; + ibool ret = FALSE; + ulint i; + + for (i = 0; i < srv_n_data_files; i++) { + if (strstr(srv_data_file_names[i], LRU_DUMP_FILE) != NULL) { + fprintf(stderr, + " InnoDB: The name '%s' seems to be used for" + " innodb_data_file_path. Dumping LRU list is not" + " done for safeness.\n", LRU_DUMP_FILE); + goto end; + } + } + + buffer_base = ut_malloc(2 * UNIV_PAGE_SIZE); + buffer = ut_align(buffer_base, UNIV_PAGE_SIZE); + if (!buffer) { + fprintf(stderr, + " InnoDB: cannot allocate buffer.\n"); + goto end; + } + + dump_file = os_file_create(LRU_DUMP_FILE, OS_FILE_OVERWRITE, + OS_FILE_NORMAL, OS_DATA_FILE, &success); + if (!success) { + os_file_get_last_error(TRUE); + fprintf(stderr, + " InnoDB: cannot open %s\n", LRU_DUMP_FILE); + goto end; + } + + mutex_enter(&LRU_list_mutex); + bpage = UT_LIST_GET_LAST(buf_pool->LRU); + + buffers = offset = 0; + while (bpage != NULL) { + if (offset == 0) { + memset(buffer, 0, UNIV_PAGE_SIZE); + } + + mach_write_to_4(buffer + offset * 4, bpage->space); + offset++; + mach_write_to_4(buffer + offset * 4, bpage->offset); + offset++; + + if (offset == UNIV_PAGE_SIZE/4) { + success = os_file_write(LRU_DUMP_FILE, dump_file, buffer, + (buffers << UNIV_PAGE_SIZE_SHIFT) & 0xFFFFFFFFUL, + (buffers >> (32 - UNIV_PAGE_SIZE_SHIFT)), + UNIV_PAGE_SIZE); + if (!success) { + mutex_exit(&LRU_list_mutex); + fprintf(stderr, + " InnoDB: cannot write page %lu of %s\n", + buffers, LRU_DUMP_FILE); + goto end; + } + buffers++; + offset = 0; + } + + bpage = UT_LIST_GET_PREV(LRU, bpage); + } + mutex_exit(&LRU_list_mutex); + + if (offset == 0) { + memset(buffer, 0, UNIV_PAGE_SIZE); + } + + mach_write_to_4(buffer + offset * 4, 0xFFFFFFFFUL); + offset++; + mach_write_to_4(buffer + offset * 4, 0xFFFFFFFFUL); + offset++; + + success = os_file_write(LRU_DUMP_FILE, dump_file, buffer, + (buffers << UNIV_PAGE_SIZE_SHIFT) & 0xFFFFFFFFUL, + (buffers >> (32 - UNIV_PAGE_SIZE_SHIFT)), + UNIV_PAGE_SIZE); + if (!success) { + goto end; + } + + ret = TRUE; +end: + if (dump_file != -1) + os_file_close(dump_file); + if (buffer_base) + ut_free(buffer_base); + + return(ret); +} +/********************************************************************//** +Read the pages based on the specific file.*/ +UNIV_INTERN +ibool +buf_LRU_file_restore(void) +/*======================*/ +{ + os_file_t dump_file = -1; + ibool success; + byte* buffer_base = NULL; + byte* buffer = NULL; + ulint buffers; + ulint offset; + ulint reads = 0; + ulint req = 0; + ibool terminated = FALSE; + ibool ret = FALSE; + + buffer_base = ut_malloc(2 * UNIV_PAGE_SIZE); + buffer = ut_align(buffer_base, UNIV_PAGE_SIZE); + if (!buffer) { + fprintf(stderr, + " InnoDB: cannot allocate buffer.\n"); + goto end; + } + + dump_file = os_file_create_simple_no_error_handling( + LRU_DUMP_FILE, OS_FILE_OPEN, OS_FILE_READ_ONLY, &success); + if (!success) { + os_file_get_last_error(TRUE); + fprintf(stderr, + " InnoDB: cannot open %s\n", LRU_DUMP_FILE); + goto end; + } + + buffers = 0; + while (!terminated) { + success = os_file_read(dump_file, buffer, + (buffers << UNIV_PAGE_SIZE_SHIFT) & 0xFFFFFFFFUL, + (buffers >> (32 - UNIV_PAGE_SIZE_SHIFT)), + UNIV_PAGE_SIZE); + if (!success) { + fprintf(stderr, + " InnoDB: cannot read page %lu of %s," + " or meet unexpected terminal.", + buffers, LRU_DUMP_FILE); + goto end; + } + + for (offset = 0; offset < UNIV_PAGE_SIZE/4; offset += 2) { + ulint space_id, zip_size, page_no; + ulint err; + ib_int64_t tablespace_version; + + space_id = mach_read_from_4(buffer + offset * 4); + page_no = mach_read_from_4(buffer + (offset + 1) * 4); + if (space_id == 0xFFFFFFFFUL + || page_no == 0xFFFFFFFFUL) { + terminated = TRUE; + break; + } + + if (offset % 16 == 15) { + os_aio_simulated_wake_handler_threads(); + buf_flush_free_margin(FALSE); + } + + zip_size = fil_space_get_zip_size(space_id); + if (UNIV_UNLIKELY(zip_size == ULINT_UNDEFINED)) { + continue; + } + + if (fil_area_is_exist(space_id, zip_size, page_no, 0, + zip_size ? zip_size : UNIV_PAGE_SIZE)) { + + tablespace_version = fil_space_get_version(space_id); + + req++; + reads += buf_read_page_low(&err, FALSE, BUF_READ_ANY_PAGE + | OS_AIO_SIMULATED_WAKE_LATER, + space_id, zip_size, TRUE, + tablespace_version, page_no, NULL); + buf_LRU_stat_inc_io(); + } + } + + buffers++; + } + + os_aio_simulated_wake_handler_threads(); + buf_flush_free_margin(FALSE); + + ut_print_timestamp(stderr); + fprintf(stderr, + " InnoDB: reading pages based on the dumped LRU list was done." + " (requested: %lu, read: %lu)\n", req, reads); + ret = TRUE; +end: + if (dump_file != -1) + os_file_close(dump_file); + if (buffer_base) + ut_free(buffer_base); + + return(ret); +} + #if defined UNIV_DEBUG || defined UNIV_BUF_DEBUG /**********************************************************************//** Validates the LRU list. @@ -2036,7 +2290,6 @@ buf_LRU_validate(void) buf_block_t* block; ulint old_len; ulint new_len; - ulint LRU_pos; ut_ad(buf_pool); //buf_pool_mutex_enter(); @@ -2046,7 +2299,11 @@ buf_LRU_validate(void) ut_a(buf_pool->LRU_old); old_len = buf_pool->LRU_old_len; - new_len = 3 * (UT_LIST_GET_LEN(buf_pool->LRU) / 8); + new_len = ut_min(UT_LIST_GET_LEN(buf_pool->LRU) + * buf_LRU_old_ratio / BUF_LRU_OLD_RATIO_DIV, + UT_LIST_GET_LEN(buf_pool->LRU) + - (BUF_LRU_OLD_TOLERANCE + + BUF_LRU_NON_OLD_MIN_LEN)); ut_a(old_len >= new_len - BUF_LRU_OLD_TOLERANCE); ut_a(old_len <= new_len + BUF_LRU_OLD_TOLERANCE); } @@ -2077,28 +2334,24 @@ buf_LRU_validate(void) } if (buf_page_is_old(bpage)) { - old_len++; - } + const buf_page_t* prev + = UT_LIST_GET_PREV(LRU, bpage); + const buf_page_t* next + = UT_LIST_GET_NEXT(LRU, bpage); - if (buf_pool->LRU_old && (old_len == 1)) { - ut_a(buf_pool->LRU_old == bpage); - } + if (!old_len++) { + ut_a(buf_pool->LRU_old == bpage); + } else { + ut_a(!prev || buf_page_is_old(prev)); + } - LRU_pos = buf_page_get_LRU_position(bpage); + ut_a(!next || buf_page_is_old(next)); + } bpage = UT_LIST_GET_NEXT(LRU, bpage); - - if (bpage) { - /* If the following assert fails, it may - not be an error: just the buf_pool clock - has wrapped around */ - ut_a(LRU_pos >= buf_page_get_LRU_position(bpage)); - } } - if (buf_pool->LRU_old) { - ut_a(buf_pool->LRU_old_len == old_len); - } + ut_a(buf_pool->LRU_old_len == old_len); mutex_exit(&LRU_list_mutex); mutex_enter(&free_list_mutex); @@ -2149,9 +2402,6 @@ buf_LRU_print(void) //buf_pool_mutex_enter(); mutex_enter(&LRU_list_mutex); - fprintf(stderr, "Pool ulint clock %lu\n", - (ulong) buf_pool->ulint_clock); - bpage = UT_LIST_GET_FIRST(buf_pool->LRU); while (bpage != NULL) { @@ -2182,18 +2432,16 @@ buf_LRU_print(void) const byte* frame; case BUF_BLOCK_FILE_PAGE: frame = buf_block_get_frame((buf_block_t*) bpage); - fprintf(stderr, "\nLRU pos %lu type %lu" + fprintf(stderr, "\ntype %lu" " index id %lu\n", - (ulong) buf_page_get_LRU_position(bpage), (ulong) fil_page_get_type(frame), (ulong) ut_dulint_get_low( btr_page_get_index_id(frame))); break; case BUF_BLOCK_ZIP_PAGE: frame = bpage->zip.data; - fprintf(stderr, "\nLRU pos %lu type %lu size %lu" + fprintf(stderr, "\ntype %lu size %lu" " index id %lu\n", - (ulong) buf_page_get_LRU_position(bpage), (ulong) fil_page_get_type(frame), (ulong) buf_page_get_zip_size(bpage), (ulong) ut_dulint_get_low( @@ -2201,8 +2449,7 @@ buf_LRU_print(void) break; default: - fprintf(stderr, "\nLRU pos %lu !state %lu!\n", - (ulong) buf_page_get_LRU_position(bpage), + fprintf(stderr, "\n!state %lu!\n", (ulong) buf_page_get_state(bpage)); break; } === modified file 'storage/xtradb/buf/buf0rea.c' --- a/storage/xtradb/buf/buf0rea.c 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/buf/buf0rea.c 2010-01-15 15:58:25 +0000 @@ -38,14 +38,6 @@ Created 11/5/1995 Heikki Tuuri #include "srv0start.h" #include "srv0srv.h" -/** The size in blocks of the area where the random read-ahead algorithm counts -the accessed pages when deciding whether to read-ahead */ -#define BUF_READ_AHEAD_RANDOM_AREA BUF_READ_AHEAD_AREA - -/** There must be at least this many pages in buf_pool in the area to start -a random read-ahead */ -#define BUF_READ_AHEAD_RANDOM_THRESHOLD (1 + BUF_READ_AHEAD_RANDOM_AREA / 2) - /** The linear read-ahead area size */ #define BUF_READ_AHEAD_LINEAR_AREA BUF_READ_AHEAD_AREA @@ -62,8 +54,9 @@ flag is cleared and the x-lock released @return 1 if a read request was queued, 0 if the page already resided in buf_pool, or if the page is in the doublewrite buffer blocks in which case it is never read into the pool, or if the tablespace does -not exist or is being dropped */ -static +not exist or is being dropped +@return 1 if read request is issued. 0 if it is not */ +UNIV_INTERN ulint buf_read_page_low( /*==============*/ @@ -82,7 +75,8 @@ buf_read_page_low( treat the tablespace as dropped; this is a timestamp we use to stop dangling page reads from a tablespace which we have DISCARDed + IMPORTed back */ - ulint offset) /*!< in: page number */ + ulint offset, /*!< in: page number */ + trx_t* trx) { buf_page_t* bpage; ulint wake_later; @@ -183,15 +177,15 @@ not_to_recover: ut_ad(buf_page_in_file(bpage)); if (zip_size) { - *err = fil_io(OS_FILE_READ | wake_later, + *err = _fil_io(OS_FILE_READ | wake_later, sync, space, zip_size, offset, 0, zip_size, - bpage->zip.data, bpage); + bpage->zip.data, bpage, trx); } else { ut_a(buf_page_get_state(bpage) == BUF_BLOCK_FILE_PAGE); - *err = fil_io(OS_FILE_READ | wake_later, + *err = _fil_io(OS_FILE_READ | wake_later, sync, space, 0, offset, 0, UNIV_PAGE_SIZE, - ((buf_block_t*) bpage)->frame, bpage); + ((buf_block_t*) bpage)->frame, bpage, trx); } ut_a(*err == DB_SUCCESS); @@ -205,206 +199,33 @@ not_to_recover: } /********************************************************************//** -Applies a random read-ahead in buf_pool if there are at least a threshold -value of accessed pages from the random read-ahead area. Does not read any -page, not even the one at the position (space, offset), if the read-ahead -mechanism is not activated. NOTE 1: the calling thread may own latches on -pages: to avoid deadlocks this function must be written such that it cannot -end up waiting for these latches! NOTE 2: the calling thread must want -access to the page given: this rule is set to prevent unintended read-aheads -performed by ibuf routines, a situation which could result in a deadlock if -the OS does not support asynchronous i/o. -@return number of page read requests issued; NOTE that if we read ibuf -pages, it may happen that the page at the given page number does not -get read even if we return a positive value! */ -static -ulint -buf_read_ahead_random( -/*==================*/ - ulint space, /*!< in: space id */ - ulint zip_size,/*!< in: compressed page size in bytes, or 0 */ - ulint offset) /*!< in: page number of a page which the current thread - wants to access */ -{ - ib_int64_t tablespace_version; - ulint recent_blocks = 0; - ulint count; - ulint LRU_recent_limit; - ulint ibuf_mode; - ulint low, high; - ulint err; - ulint i; - ulint buf_read_ahead_random_area; - -// /* We have currently disabled random readahead */ -// return(0); - - if (!(srv_read_ahead & 1)) { - return(0); - } - - if (srv_startup_is_before_trx_rollback_phase) { - /* No read-ahead to avoid thread deadlocks */ - return(0); - } - - if (ibuf_bitmap_page(zip_size, offset) - || trx_sys_hdr_page(space, offset)) { - - /* If it is an ibuf bitmap page or trx sys hdr, we do - no read-ahead, as that could break the ibuf page access - order */ - - return(0); - } - - /* Remember the tablespace version before we ask te tablespace size - below: if DISCARD + IMPORT changes the actual .ibd file meanwhile, we - do not try to read outside the bounds of the tablespace! */ - - tablespace_version = fil_space_get_version(space); - - buf_read_ahead_random_area = BUF_READ_AHEAD_RANDOM_AREA; - - low = (offset / buf_read_ahead_random_area) - * buf_read_ahead_random_area; - high = (offset / buf_read_ahead_random_area + 1) - * buf_read_ahead_random_area; - if (high > fil_space_get_size(space)) { - - high = fil_space_get_size(space); - } - - /* Get the minimum LRU_position field value for an initial segment - of the LRU list, to determine which blocks have recently been added - to the start of the list. */ - - LRU_recent_limit = buf_LRU_get_recent_limit(); - - //buf_pool_mutex_enter(); - mutex_enter(&buf_pool_mutex); - - if (buf_pool->n_pend_reads - > buf_pool->curr_size / BUF_READ_AHEAD_PEND_LIMIT) { - //buf_pool_mutex_exit(); - mutex_exit(&buf_pool_mutex); - - return(0); - } - mutex_exit(&buf_pool_mutex); - - /* Count how many blocks in the area have been recently accessed, - that is, reside near the start of the LRU list. */ - - rw_lock_s_lock(&page_hash_latch); - for (i = low; i < high; i++) { - const buf_page_t* bpage = buf_page_hash_get(space, i); - - if (bpage - && buf_page_is_accessed(bpage) - && (buf_page_get_LRU_position(bpage) > LRU_recent_limit)) { - - recent_blocks++; - - if (recent_blocks >= BUF_READ_AHEAD_RANDOM_THRESHOLD) { - - //buf_pool_mutex_exit(); - rw_lock_s_unlock(&page_hash_latch); - goto read_ahead; - } - } - } - - //buf_pool_mutex_exit(); - rw_lock_s_unlock(&page_hash_latch); - /* Do nothing */ - return(0); - -read_ahead: - /* Read all the suitable blocks within the area */ - - if (ibuf_inside()) { - ibuf_mode = BUF_READ_IBUF_PAGES_ONLY; - } else { - ibuf_mode = BUF_READ_ANY_PAGE; - } - - count = 0; - - for (i = low; i < high; i++) { - /* It is only sensible to do read-ahead in the non-sync aio - mode: hence FALSE as the first parameter */ - - if (!ibuf_bitmap_page(zip_size, i)) { - count += buf_read_page_low( - &err, FALSE, - ibuf_mode | OS_AIO_SIMULATED_WAKE_LATER, - space, zip_size, FALSE, - tablespace_version, i); - if (err == DB_TABLESPACE_DELETED) { - ut_print_timestamp(stderr); - fprintf(stderr, - " InnoDB: Warning: in random" - " readahead trying to access\n" - "InnoDB: tablespace %lu page %lu,\n" - "InnoDB: but the tablespace does not" - " exist or is just being dropped.\n", - (ulong) space, (ulong) i); - } - } - } - - /* In simulated aio we wake the aio handler threads only after - queuing all aio requests, in native aio the following call does - nothing: */ - - os_aio_simulated_wake_handler_threads(); - -#ifdef UNIV_DEBUG - if (buf_debug_prints && (count > 0)) { - fprintf(stderr, - "Random read-ahead space %lu offset %lu pages %lu\n", - (ulong) space, (ulong) offset, - (ulong) count); - } -#endif /* UNIV_DEBUG */ - - ++srv_read_ahead_rnd; - return(count); -} - -/********************************************************************//** High-level function which reads a page asynchronously from a file to the buffer buf_pool if it is not already there. Sets the io_fix flag and sets an exclusive lock on the buffer frame. The flag is cleared and the x-lock -released by the i/o-handler thread. Does a random read-ahead if it seems -sensible. -@return number of page read requests issued: this can be greater than -1 if read-ahead occurred */ +released by the i/o-handler thread. +@return TRUE if page has been read in, FALSE in case of failure */ UNIV_INTERN -ulint +ibool buf_read_page( /*==========*/ ulint space, /*!< in: space id */ ulint zip_size,/*!< in: compressed page size in bytes, or 0 */ - ulint offset) /*!< in: page number */ + ulint offset, /*!< in: page number */ + trx_t* trx) { ib_int64_t tablespace_version; ulint count; - ulint count2; ulint err; tablespace_version = fil_space_get_version(space); - count = buf_read_ahead_random(space, zip_size, offset); - /* We do the i/o in the synchronous aio mode to save thread switches: hence TRUE */ - count2 = buf_read_page_low(&err, TRUE, BUF_READ_ANY_PAGE, space, - zip_size, FALSE, - tablespace_version, offset); - srv_buf_pool_reads+= count2; + count = buf_read_page_low(&err, TRUE, BUF_READ_ANY_PAGE, space, + zip_size, FALSE, + tablespace_version, offset, trx); + srv_buf_pool_reads += count; if (err == DB_TABLESPACE_DELETED) { ut_print_timestamp(stderr); fprintf(stderr, @@ -421,14 +242,14 @@ buf_read_page( /* Increment number of I/O operations used for LRU policy. */ buf_LRU_stat_inc_io(); - return(count + count2); + return(count > 0); } /********************************************************************//** Applies linear read-ahead if in the buf_pool the page is a border page of a linear read-ahead area and all the pages in the area have been accessed. Does not read any page if the read-ahead mechanism is not activated. Note -that the the algorithm looks at the 'natural' adjacent successor and +that the algorithm looks at the 'natural' adjacent successor and predecessor of the page, which on the leaf level of a B-tree are the next and previous page in the chain of leaves. To know these, the page specified in (space, offset) must already be present in the buf_pool. Thus, the @@ -454,8 +275,9 @@ buf_read_ahead_linear( /*==================*/ ulint space, /*!< in: space id */ ulint zip_size,/*!< in: compressed page size in bytes, or 0 */ - ulint offset) /*!< in: page number of a page; NOTE: the current thread + ulint offset, /*!< in: page number of a page; NOTE: the current thread must want access to this page (see NOTE 3 above) */ + trx_t* trx) { ib_int64_t tablespace_version; buf_page_t* bpage; @@ -557,9 +379,17 @@ buf_read_ahead_linear( fail_count++; } else if (pred_bpage) { - int res = (ut_ulint_cmp( - buf_page_get_LRU_position(bpage), - buf_page_get_LRU_position(pred_bpage))); + /* Note that buf_page_is_accessed() returns + the time of the first access. If some blocks + of the extent existed in the buffer pool at + the time of a linear access pattern, the first + access times may be nonmonotonic, even though + the latest access times were linear. The + threshold (srv_read_ahead_factor) should help + a little against this. */ + int res = ut_ulint_cmp( + buf_page_is_accessed(bpage), + buf_page_is_accessed(pred_bpage)); /* Accesses not in the right order */ if (res != 0 && res != asc_or_desc) { fail_count++; @@ -670,7 +500,7 @@ buf_read_ahead_linear( count += buf_read_page_low( &err, FALSE, ibuf_mode | OS_AIO_SIMULATED_WAKE_LATER, - space, zip_size, FALSE, tablespace_version, i); + space, zip_size, FALSE, tablespace_version, i, trx); if (err == DB_TABLESPACE_DELETED) { ut_print_timestamp(stderr); fprintf(stderr, @@ -705,7 +535,7 @@ buf_read_ahead_linear( LRU policy decision. */ buf_LRU_stat_inc_io(); - ++srv_read_ahead_seq; + buf_pool->stat.n_ra_pages_read += count; return(count); } @@ -760,7 +590,7 @@ buf_read_ibuf_merge_pages( buf_read_page_low(&err, sync && (i + 1 == n_stored), BUF_READ_ANY_PAGE, space_ids[i], zip_size, TRUE, space_versions[i], - page_nos[i]); + page_nos[i], NULL); if (UNIV_UNLIKELY(err == DB_TABLESPACE_DELETED)) { tablespace_deleted: @@ -857,12 +687,12 @@ buf_read_recv_pages( if ((i + 1 == n_stored) && sync) { buf_read_page_low(&err, TRUE, BUF_READ_ANY_PAGE, space, zip_size, TRUE, tablespace_version, - page_nos[i]); + page_nos[i], NULL); } else { buf_read_page_low(&err, FALSE, BUF_READ_ANY_PAGE | OS_AIO_SIMULATED_WAKE_LATER, space, zip_size, TRUE, - tablespace_version, page_nos[i]); + tablespace_version, page_nos[i], NULL); } } === modified file 'storage/xtradb/data/data0type.c' --- a/storage/xtradb/data/data0type.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/data/data0type.c 2010-01-06 12:00:14 +0000 @@ -237,6 +237,22 @@ dtype_print( fputs("DATA_SYS", stderr); break; + case DATA_FLOAT: + fputs("DATA_FLOAT", stderr); + break; + + case DATA_DOUBLE: + fputs("DATA_DOUBLE", stderr); + break; + + case DATA_DECIMAL: + fputs("DATA_DECIMAL", stderr); + break; + + case DATA_VARMYSQL: + fputs("DATA_VARMYSQL", stderr); + break; + default: fprintf(stderr, "type %lu", (ulong) mtype); break; === modified file 'storage/xtradb/dict/dict0crea.c' --- a/storage/xtradb/dict/dict0crea.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/dict/dict0crea.c 2010-01-06 12:00:14 +0000 @@ -1387,7 +1387,7 @@ dict_create_add_foreign_field_to_diction Add a single foreign key definition to the data dictionary tables in the database. We also generate names to constraints that were not named by the user. A generated constraint has a name of the format -databasename/tablename_ibfk_<number>, where the numbers start from 1, and +databasename/tablename_ibfk_NUMBER, where the numbers start from 1, and are given locally for this table, that is, the number is not global, as in the old format constraints < 4.0.18 it used to be. @return error code or DB_SUCCESS */ === modified file 'storage/xtradb/dict/dict0dict.c' --- a/storage/xtradb/dict/dict0dict.c 2009-12-03 11:34:11 +0000 +++ b/storage/xtradb/dict/dict0dict.c 2010-01-15 15:58:25 +0000 @@ -82,9 +82,10 @@ static char dict_ibfk[] = "_ibfk_"; /*******************************************************************//** Tries to find column names for the index and sets the col field of the -index. */ +index. +@return TRUE if the column names were found */ static -void +ibool dict_index_find_cols( /*=================*/ dict_table_t* table, /*!< in: table */ @@ -1261,7 +1262,7 @@ dict_index_too_big_for_undo( = TRX_UNDO_PAGE_HDR - TRX_UNDO_PAGE_HDR_SIZE + 2 /* next record pointer */ + 1 /* type_cmpl */ - + 11 /* trx->undo_no */ - 11 /* table->id */ + + 11 /* trx->undo_no */ + 11 /* table->id */ + 1 /* rec_get_info_bits() */ + 11 /* DB_TRX_ID */ + 11 /* DB_ROLL_PTR */ @@ -1440,7 +1441,11 @@ dict_index_too_big_for_tree( goto add_field_size; } + if (srv_relax_table_creation) { + field_max_size = dict_col_get_min_size(col); + } else { field_max_size = dict_col_get_max_size(col); + } field_ext_max_size = field_max_size < 256 ? 1 : 2; if (field->prefix_len) { @@ -1493,7 +1498,7 @@ add_field_size: /**********************************************************************//** Adds an index to the dictionary cache. -@return DB_SUCCESS or DB_TOO_BIG_RECORD */ +@return DB_SUCCESS, DB_TOO_BIG_RECORD, or DB_CORRUPTION */ UNIV_INTERN ulint dict_index_add_to_cache( @@ -1519,7 +1524,10 @@ dict_index_add_to_cache( ut_a(!dict_index_is_clust(index) || UT_LIST_GET_LEN(table->indexes) == 0); - dict_index_find_cols(table, index); + if (!dict_index_find_cols(table, index)) { + + return(DB_CORRUPTION); + } /* Build the cache internal representation of the index, containing also the added system fields */ @@ -1732,9 +1740,10 @@ dict_index_remove_from_cache( /*******************************************************************//** Tries to find column names for the index and sets the col field of the -index. */ +index. +@return TRUE if the column names were found */ static -void +ibool dict_index_find_cols( /*=================*/ dict_table_t* table, /*!< in: table */ @@ -1759,17 +1768,21 @@ dict_index_find_cols( } } +#ifdef UNIV_DEBUG /* It is an error not to find a matching column. */ fputs("InnoDB: Error: no matching column for ", stderr); ut_print_name(stderr, NULL, FALSE, field->name); fputs(" in ", stderr); dict_index_name_print(stderr, NULL, index); fputs("!\n", stderr); - ut_error; +#endif /* UNIV_DEBUG */ + return(FALSE); found: ; } + + return(TRUE); } #endif /* !UNIV_HOTBACKUP */ @@ -4711,6 +4724,26 @@ dict_ind_init(void) dict_ind_redundant->cached = dict_ind_compact->cached = TRUE; } +/**********************************************************************//** +Frees dict_ind_redundant and dict_ind_compact. */ +static +void +dict_ind_free(void) +/*===============*/ +{ + dict_table_t* table; + + table = dict_ind_compact->table; + dict_mem_index_free(dict_ind_compact); + dict_ind_compact = NULL; + dict_mem_table_free(table); + + table = dict_ind_redundant->table; + dict_mem_index_free(dict_ind_redundant); + dict_ind_redundant = NULL; + dict_mem_table_free(table); +} + #ifndef UNIV_HOTBACKUP /**********************************************************************//** Get index by name @@ -4836,4 +4869,55 @@ dict_table_check_for_dup_indexes( } } #endif /* UNIV_DEBUG */ + +/************************************************************************** +Closes the data dictionary module. */ +UNIV_INTERN +void +dict_close(void) +/*============*/ +{ + ulint i; + + /* Free the hash elements. We don't remove them from the table + because we are going to destroy the table anyway. */ + for (i = 0; i < hash_get_n_cells(dict_sys->table_hash); i++) { + dict_table_t* table; + + table = HASH_GET_FIRST(dict_sys->table_hash, i); + + while (table) { + dict_table_t* prev_table = table; + + table = HASH_GET_NEXT(name_hash, prev_table); +#ifdef UNIV_DEBUG + ut_a(prev_table->magic_n == DICT_TABLE_MAGIC_N); +#endif + /* Acquire only because it's a pre-condition. */ + mutex_enter(&dict_sys->mutex); + + dict_table_remove_from_cache(prev_table); + + mutex_exit(&dict_sys->mutex); + } + } + + hash_table_free(dict_sys->table_hash); + + /* The elements are the same instance as in dict_sys->table_hash, + therefore we don't delete the individual elements. */ + hash_table_free(dict_sys->table_id_hash); + + dict_ind_free(); + + mutex_free(&dict_sys->mutex); + + rw_lock_free(&dict_operation_lock); + memset(&dict_operation_lock, 0x0, sizeof(dict_operation_lock)); + + mutex_free(&dict_foreign_err_mutex); + + mem_free(dict_sys); + dict_sys = NULL; +} #endif /* !UNIV_HOTBACKUP */ === modified file 'storage/xtradb/fil/fil0fil.c' --- a/storage/xtradb/fil/fil0fil.c 2009-11-29 23:08:56 +0000 +++ b/storage/xtradb/fil/fil0fil.c 2010-01-15 15:58:25 +0000 @@ -327,6 +327,17 @@ fil_get_space_id_for_table( /*=======================*/ const char* name); /*!< in: table name in the standard 'databasename/tablename' format */ +/*******************************************************************//** +Frees a space object from the tablespace memory cache. Closes the files in +the chain but does not delete them. There must not be any pending i/o's or +flushes on the files. */ +static +ibool +fil_space_free( +/*===========*/ + /* out: TRUE if success */ + ulint id, /* in: space id */ + ibool own_mutex);/* in: TRUE if own system->mutex */ /********************************************************************//** Reads data from a space to a buffer. Remember that the possible incomplete blocks at the end of file are ignored: they are not taken into account when @@ -600,6 +611,11 @@ fil_node_create( UT_LIST_ADD_LAST(chain, space->chain, node); + if (id < SRV_LOG_SPACE_FIRST_ID && fil_system->max_assigned_id < id) { + + fil_system->max_assigned_id = id; + } + mutex_exit(&fil_system->mutex); } @@ -619,12 +635,10 @@ fil_node_open_file( ulint size_high; ibool ret; ibool success; -#ifndef UNIV_HOTBACKUP byte* buf2; byte* page; ulint space_id; ulint flags; -#endif /* !UNIV_HOTBACKUP */ ut_ad(mutex_own(&(system->mutex))); ut_a(node->n_pending == 0); @@ -660,9 +674,12 @@ fil_node_open_file( size_bytes = (((ib_int64_t)size_high) << 32) + (ib_int64_t)size_low; #ifdef UNIV_HOTBACKUP - node->size = (ulint) (size_bytes / UNIV_PAGE_SIZE); - /* TODO: adjust to zip_size, like below? */ -#else + if (space->id == 0) { + node->size = (ulint) (size_bytes / UNIV_PAGE_SIZE); + os_file_close(node->handle); + goto add_size; + } +#endif /* UNIV_HOTBACKUP */ ut_a(space->purpose != FIL_LOG); ut_a(space->id != 0); @@ -741,7 +758,10 @@ fil_node_open_file( (size_bytes / dict_table_flags_to_zip_size(flags)); } -#endif + +#ifdef UNIV_HOTBACKUP +add_size: +#endif /* UNIV_HOTBACKUP */ space->size += node->size; } @@ -961,7 +981,7 @@ close_more: " while the maximum\n" "InnoDB: allowed value would be %lu.\n" "InnoDB: You may need to raise the value of" - " innodb_max_files_open in\n" + " innodb_open_files in\n" "InnoDB: my.cnf.\n", (ulong) fil_system->n_open, (ulong) fil_system->max_n_open); @@ -1141,7 +1161,7 @@ try_again: mutex_exit(&fil_system->mutex); - fil_space_free(namesake_id); + fil_space_free(namesake_id, FALSE); goto try_again; } @@ -1266,17 +1286,21 @@ Frees a space object from the tablespace the chain but does not delete them. There must not be any pending i/o's or flushes on the files. @return TRUE if success */ -UNIV_INTERN +static ibool fil_space_free( /*===========*/ - ulint id) /*!< in: space id */ + /* out: TRUE if success */ + ulint id, /* in: space id */ + ibool own_mutex) /* in: TRUE if own system->mutex */ { fil_space_t* space; fil_space_t* namespace; fil_node_t* fil_node; - mutex_enter(&fil_system->mutex); + if (!own_mutex) { + mutex_enter(&fil_system->mutex); + } space = fil_space_get_by_id(id); @@ -1323,7 +1347,9 @@ fil_space_free( ut_a(0 == UT_LIST_GET_LEN(space->chain)); - mutex_exit(&fil_system->mutex); + if (!own_mutex) { + mutex_exit(&fil_system->mutex); + } rw_lock_free(&(space->latch)); @@ -1541,7 +1567,7 @@ fil_open_log_and_system_tablespace_files fprintf(stderr, "InnoDB: Warning: you must" " raise the value of" - " innodb_max_open_files in\n" + " innodb_open_files in\n" "InnoDB: my.cnf! Remember that" " InnoDB keeps all log files" " and all system\n" @@ -1583,6 +1609,8 @@ fil_close_all_files(void) space = UT_LIST_GET_FIRST(fil_system->space_list); while (space != NULL) { + fil_space_t* prev_space = space; + node = UT_LIST_GET_FIRST(space->chain); while (node != NULL) { @@ -1592,6 +1620,7 @@ fil_close_all_files(void) node = UT_LIST_GET_NEXT(chain, node); } space = UT_LIST_GET_NEXT(space_list, space); + fil_space_free(prev_space->id, TRUE); } mutex_exit(&fil_system->mutex); @@ -2223,7 +2252,7 @@ try_again: #endif /* printf("Deleting tablespace %s id %lu\n", space->name, id); */ - success = fil_space_free(id); + success = fil_space_free(id, FALSE); if (success) { success = os_file_delete(path); @@ -2929,7 +2958,6 @@ fil_open_single_table_tablespace( byte* page; ulint space_id; ulint space_flags; - ibool ret = TRUE; filepath = fil_make_ibd_name(name, FALSE); @@ -3330,7 +3358,7 @@ skip_write: (ulong) space_id, (ulong) space_flags, (ulong) id, (ulong) flags); - ret = FALSE; + success = FALSE; goto func_exit; } @@ -3350,7 +3378,7 @@ func_exit: os_file_close(file); mem_free(filepath); - return(ret); + return(success); } #endif /* !UNIV_HOTBACKUP */ @@ -3566,7 +3594,7 @@ fil_load_single_table_tablespace( fprintf(stderr, "InnoDB: Renaming tablespace %s of id %lu,\n" "InnoDB: to %s_ibbackup_old_vers_<timestamp>\n" - "InnoDB: because its size %lld is too small" + "InnoDB: because its size %" PRId64 " is too small" " (< 4 pages 16 kB each),\n" "InnoDB: or the space id in the file header" " is not sensible.\n" @@ -3628,7 +3656,17 @@ fil_load_single_table_tablespace( if (!success) { - goto func_exit; + if (srv_force_recovery > 0) { + fprintf(stderr, + "InnoDB: innodb_force_recovery" + " was set to %lu. Continuing crash recovery\n" + "InnoDB: even though the tablespace creation" + " of this table failed.\n", + srv_force_recovery); + goto func_exit; + } + + exit(1); } /* We do not use the size information we have about the file, because @@ -4163,7 +4201,7 @@ fil_extend_space_to_desired_size( node->name, node->handle, buf, offset_low, offset_high, page_size * n_pages, - NULL, NULL); + NULL, NULL, NULL); #endif if (success) { node->size += n_pages; @@ -4490,7 +4528,7 @@ Reads or writes data. This operation is i/o on a tablespace which does not exist */ UNIV_INTERN ulint -fil_io( +_fil_io( /*===*/ ulint type, /*!< in: OS_FILE_READ or OS_FILE_WRITE, ORed to OS_FILE_LOG, if a log i/o @@ -4515,8 +4553,9 @@ fil_io( void* buf, /*!< in/out: buffer where to store read data or from where to write; in aio this must be appropriately aligned */ - void* message) /*!< in: message for aio handler if non-sync + void* message, /*!< in: message for aio handler if non-sync aio used, else ignored */ + trx_t* trx) { ulint mode; fil_space_t* space; @@ -4686,7 +4725,7 @@ fil_io( #else /* Queue the aio request */ ret = os_aio(type, mode | wake_later, node->name, node->handle, buf, - offset_low, offset_high, len, node, message); + offset_low, offset_high, len, node, message, trx); #endif ut_a(ret); @@ -4706,6 +4745,78 @@ fil_io( return(DB_SUCCESS); } +/********************************************************************//** +Confirm whether the parameters are valid or not */ +UNIV_INTERN +ibool +fil_area_is_exist( +/*==============*/ + ulint space_id, /*!< in: space id */ + ulint zip_size, /*!< in: compressed page size in bytes; + 0 for uncompressed pages */ + ulint block_offset, /*!< in: offset in number of blocks */ + ulint byte_offset, /*!< in: remainder of offset in bytes; in + aio this must be divisible by the OS block + size */ + ulint len) /*!< in: how many bytes to read or write; this + must not cross a file boundary; in aio this + must be a block size multiple */ +{ + fil_space_t* space; + fil_node_t* node; + + /* Reserve the fil_system mutex and make sure that we can open at + least one file while holding it, if the file is not already open */ + + fil_mutex_enter_and_prepare_for_io(space_id); + + space = fil_space_get_by_id(space_id); + + if (!space) { + mutex_exit(&fil_system->mutex); + return(FALSE); + } + + node = UT_LIST_GET_FIRST(space->chain); + + for (;;) { + if (UNIV_UNLIKELY(node == NULL)) { + mutex_exit(&fil_system->mutex); + return(FALSE); + } + + if (space->id != 0 && node->size == 0) { + /* We do not know the size of a single-table tablespace + before we open the file */ + + break; + } + + if (node->size > block_offset) { + /* Found! */ + break; + } else { + block_offset -= node->size; + node = UT_LIST_GET_NEXT(chain, node); + } + } + + /* Open file if closed */ + fil_node_prepare_for_io(node, fil_system, space); + fil_node_complete_io(node, fil_system, OS_FILE_READ); + + /* Check that at least the start offset is within the bounds of a + single-table tablespace */ + if (UNIV_UNLIKELY(node->size <= block_offset) + && space->id != 0 && space->purpose == FIL_TABLESPACE) { + mutex_exit(&fil_system->mutex); + return(FALSE); + } + + mutex_exit(&fil_system->mutex); + return(TRUE); +} + #ifndef UNIV_HOTBACKUP /**********************************************************************//** Waits for an aio operation to complete. This function is used to write the @@ -5065,6 +5176,29 @@ fil_page_get_type( return(mach_read_from_2(page + FIL_PAGE_TYPE)); } +/******************************************************************** +Initializes the tablespace memory cache. */ +UNIV_INTERN +void +fil_close(void) +/*===========*/ +{ + /* The mutex should already have been freed. */ + ut_ad(fil_system->mutex.magic_n == 0); + + hash_table_free(fil_system->spaces); + + hash_table_free(fil_system->name_hash); + + ut_a(UT_LIST_GET_LEN(fil_system->LRU) == 0); + ut_a(UT_LIST_GET_LEN(fil_system->unflushed_spaces) == 0); + ut_a(UT_LIST_GET_LEN(fil_system->space_list) == 0); + + mem_free(fil_system); + + fil_system = NULL; +} + /************************************************************************* Return local hash table informations. */ === modified file 'storage/xtradb/fsp/fsp0fsp.c' --- a/storage/xtradb/fsp/fsp0fsp.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/fsp/fsp0fsp.c 2010-01-06 12:00:14 +0000 @@ -232,6 +232,9 @@ the extent are free and which contain ol #define XDES_ARR_OFFSET (FSP_HEADER_OFFSET + FSP_HEADER_SIZE) #ifndef UNIV_HOTBACKUP +/* Flag to indicate if we have printed the tablespace full error. */ +static ibool fsp_tbs_full_error_printed = FALSE; + /**********************************************************************//** Returns an extent to the free list of a space. */ static @@ -1099,7 +1102,7 @@ fsp_header_inc_size( /**********************************************************************//** Gets the current free limit of the system tablespace. The free limit -means the place of the first page which has never been put to the the +means the place of the first page which has never been put to the free list for allocation. The space above that address is initialized to zero. Sets also the global variable log_fsp_current_free_limit. @return free limit in megabytes */ @@ -1218,6 +1221,19 @@ fsp_try_extend_data_file( if (space == 0 && !srv_auto_extend_last_data_file) { + /* We print the error message only once to avoid + spamming the error log. Note that we don't need + to reset the flag to FALSE as dealing with this + error requires server restart. */ + if (fsp_tbs_full_error_printed == FALSE) { + fprintf(stderr, + "InnoDB: Error: Data file(s) ran" + " out of space.\n" + "Please add another data file or" + " use \'autoextend\' for the last" + " data file.\n"); + fsp_tbs_full_error_printed = TRUE; + } return(FALSE); } @@ -1832,6 +1848,8 @@ fsp_seg_inode_page_find_used( if (!ut_dulint_is_zero(mach_read_from_8(inode + FSEG_ID))) { /* This is used */ + ut_ad(mach_read_from_4(inode + FSEG_MAGIC_N) + == FSEG_MAGIC_N_VALUE); return(i); } } @@ -1863,6 +1881,9 @@ fsp_seg_inode_page_find_free( return(i); } + + ut_ad(mach_read_from_4(inode + FSEG_MAGIC_N) + == FSEG_MAGIC_N_VALUE); } return(ULINT_UNDEFINED); @@ -1981,6 +2002,8 @@ fsp_alloc_seg_inode( page + FSEG_INODE_PAGE_NODE, mtr); } + ut_ad(ut_dulint_is_zero(mach_read_from_8(inode + FSEG_ID)) + || mach_read_from_4(inode + FSEG_MAGIC_N) == FSEG_MAGIC_N_VALUE); return(inode); } @@ -2018,7 +2041,7 @@ fsp_free_seg_inode( } mlog_write_dulint(inode + FSEG_ID, ut_dulint_zero, mtr); - mlog_write_ulint(inode + FSEG_MAGIC_N, 0, MLOG_4BYTES, mtr); + mlog_write_ulint(inode + FSEG_MAGIC_N, 0xfa051ce3, MLOG_4BYTES, mtr); if (ULINT_UNDEFINED == fsp_seg_inode_page_find_used(page, zip_size, mtr)) { @@ -2034,11 +2057,11 @@ fsp_free_seg_inode( /**********************************************************************//** Returns the file segment inode, page x-latched. -@return segment inode, page x-latched */ +@return segment inode, page x-latched; NULL if the inode is free */ static fseg_inode_t* -fseg_inode_get( -/*===========*/ +fseg_inode_try_get( +/*===============*/ fseg_header_t* header, /*!< in: segment header */ ulint space, /*!< in: space id */ ulint zip_size,/*!< in: compressed page size in bytes @@ -2054,12 +2077,38 @@ fseg_inode_get( inode = fut_get_ptr(space, zip_size, inode_addr, RW_X_LATCH, mtr); - ut_ad(mach_read_from_4(inode + FSEG_MAGIC_N) == FSEG_MAGIC_N_VALUE); + if (UNIV_UNLIKELY + (ut_dulint_is_zero(mach_read_from_8(inode + FSEG_ID)))) { + + inode = NULL; + } else { + ut_ad(mach_read_from_4(inode + FSEG_MAGIC_N) + == FSEG_MAGIC_N_VALUE); + } return(inode); } /**********************************************************************//** +Returns the file segment inode, page x-latched. +@return segment inode, page x-latched */ +static +fseg_inode_t* +fseg_inode_get( +/*===========*/ + fseg_header_t* header, /*!< in: segment header */ + ulint space, /*!< in: space id */ + ulint zip_size,/*!< in: compressed page size in bytes + or 0 for uncompressed pages */ + mtr_t* mtr) /*!< in: mtr handle */ +{ + fseg_inode_t* inode + = fseg_inode_try_get(header, space, zip_size, mtr); + ut_a(inode); + return(inode); +} + +/**********************************************************************//** Gets the page number from the nth fragment page slot. @return page number, FIL_NULL if not in use */ UNIV_INLINE @@ -2073,6 +2122,7 @@ fseg_get_nth_frag_page_no( ut_ad(inode && mtr); ut_ad(n < FSEG_FRAG_ARR_N_SLOTS); ut_ad(mtr_memo_contains_page(mtr, inode, MTR_MEMO_PAGE_X_FIX)); + ut_ad(mach_read_from_4(inode + FSEG_MAGIC_N) == FSEG_MAGIC_N_VALUE); return(mach_read_from_4(inode + FSEG_FRAG_ARR + n * FSEG_FRAG_SLOT_SIZE)); } @@ -2091,6 +2141,7 @@ fseg_set_nth_frag_page_no( ut_ad(inode && mtr); ut_ad(n < FSEG_FRAG_ARR_N_SLOTS); ut_ad(mtr_memo_contains_page(mtr, inode, MTR_MEMO_PAGE_X_FIX)); + ut_ad(mach_read_from_4(inode + FSEG_MAGIC_N) == FSEG_MAGIC_N_VALUE); mlog_write_ulint(inode + FSEG_FRAG_ARR + n * FSEG_FRAG_SLOT_SIZE, page_no, MLOG_4BYTES, mtr); @@ -2451,6 +2502,8 @@ fseg_fill_free_list( xdes_set_state(descr, XDES_FSEG, mtr); seg_id = mtr_read_dulint(inode + FSEG_ID, mtr); + ut_ad(mach_read_from_4(inode + FSEG_MAGIC_N) + == FSEG_MAGIC_N_VALUE); mlog_write_dulint(descr + XDES_ID, seg_id, mtr); flst_add_last(inode + FSEG_FREE, descr + XDES_FLST_NODE, mtr); @@ -2479,6 +2532,7 @@ fseg_alloc_free_extent( fil_addr_t first; ut_ad(!((page_offset(inode) - FSEG_ARR_OFFSET) % FSEG_INODE_SIZE)); + ut_ad(mach_read_from_4(inode + FSEG_MAGIC_N) == FSEG_MAGIC_N_VALUE); if (flst_get_len(inode + FSEG_FREE, mtr) > 0) { /* Segment free list is not empty, allocate from it */ @@ -3136,6 +3190,8 @@ fseg_mark_page_used( ut_ad(seg_inode && mtr); ut_ad(!((page_offset(seg_inode) - FSEG_ARR_OFFSET) % FSEG_INODE_SIZE)); + ut_ad(mach_read_from_4(seg_inode + FSEG_MAGIC_N) + == FSEG_MAGIC_N_VALUE); descr = xdes_get_descriptor(space, zip_size, page, mtr); @@ -3373,6 +3429,8 @@ fseg_free_extent( ut_a(xdes_get_state(descr, mtr) == XDES_FSEG); ut_a(0 == ut_dulint_cmp(mtr_read_dulint(descr + XDES_ID, mtr), mtr_read_dulint(seg_inode + FSEG_ID, mtr))); + ut_ad(mach_read_from_4(seg_inode + FSEG_MAGIC_N) + == FSEG_MAGIC_N_VALUE); first_page_in_extent = page - (page % FSP_EXTENT_SIZE); @@ -3463,7 +3521,13 @@ fseg_free_step( ut_a(descr); ut_a(xdes_get_bit(descr, XDES_FREE_BIT, header_page % FSP_EXTENT_SIZE, mtr) == FALSE); - inode = fseg_inode_get(header, space, zip_size, mtr); + inode = fseg_inode_try_get(header, space, zip_size, mtr); + + if (UNIV_UNLIKELY(inode == NULL)) { + fprintf(stderr, "double free of inode from %u:%u\n", + (unsigned) space, (unsigned) header_page); + return(TRUE); + } descr = fseg_get_first_extent(inode, space, zip_size, mtr); @@ -3587,6 +3651,7 @@ fseg_get_first_extent( ut_ad(inode && mtr); ut_ad(space == page_get_space_id(page_align(inode))); + ut_ad(mach_read_from_4(inode + FSEG_MAGIC_N) == FSEG_MAGIC_N_VALUE); first = fil_addr_null; @@ -3801,6 +3866,7 @@ fseg_print_low( (ulong) reserved, (ulong) used, (ulong) n_full, (ulong) n_frag, (ulong) n_free, (ulong) n_not_full, (ulong) n_used); + ut_ad(mach_read_from_4(inode + FSEG_MAGIC_N) == FSEG_MAGIC_N_VALUE); } #ifdef UNIV_BTR_PRINT === modified file 'storage/xtradb/handler/ha_innodb.cc' --- a/storage/xtradb/handler/ha_innodb.cc 2009-12-03 11:34:11 +0000 +++ b/storage/xtradb/handler/ha_innodb.cc 2010-01-15 21:12:30 +0000 @@ -61,7 +61,9 @@ with this program; if not, write to the #pragma implementation // gcc: Class implementation #endif +#ifndef MYSQL_SERVER #define MYSQL_SERVER +#endif #include <mysql_priv.h> #ifdef MYSQL_SERVER @@ -77,6 +79,7 @@ with this program; if not, write to the /* Include necessary InnoDB headers */ extern "C" { #include "univ.i" +#include "buf0lru.h" #include "btr0sea.h" #include "os0file.h" #include "os0thread.h" @@ -109,7 +112,6 @@ extern "C" { #include "ha_innodb.h" #include "i_s.h" -#include "handler0vars.h" #ifdef MYSQL_SERVER // Defined in trx0sys.c @@ -120,9 +122,12 @@ extern ib_int64_t trx_sys_mysql_relay_lo #endif /* MYSQL_SERVER */ #ifndef MYSQL_SERVER -/* This is needed because of Bug #3596. Let us hope that pthread_mutex_t +# ifndef MYSQL_PLUGIN_IMPORT +# define MYSQL_PLUGIN_IMPORT /* nothing */ +# endif /* MYSQL_PLUGIN_IMPORT */ +/* This is needed because of Bug #3596. Let us hope that pthread_mutex_t is defined the same in both builds: the MySQL server and the InnoDB plugin. */ -extern pthread_mutex_t LOCK_thread_count; +extern MYSQL_PLUGIN_IMPORT pthread_mutex_t LOCK_thread_count; #if MYSQL_VERSION_ID < 50124 /* this is defined in mysql_priv.h inside #ifdef MYSQL_SERVER @@ -146,13 +151,9 @@ static bool innodb_inited = 0; /* In the Windows plugin, the return value of current_thd is undefined. Map it to NULL. */ -#if defined MYSQL_DYNAMIC_PLUGIN && defined __WIN__ -# undef current_thd -# define current_thd NULL -# define EQ_CURRENT_THD(thd) TRUE -#else /* MYSQL_DYNAMIC_PLUGIN && __WIN__ */ -# define EQ_CURRENT_THD(thd) ((thd) == current_thd) -#endif /* MYSQL_DYNAMIC_PLUGIN && __WIN__ */ + +#define EQ_CURRENT_THD(thd) ((thd) == current_thd) + static struct handlerton* innodb_hton_ptr; @@ -172,6 +173,10 @@ static ulong innobase_write_io_threads; static my_bool innobase_thread_concurrency_timer_based; static long long innobase_buffer_pool_size, innobase_log_file_size; +/** Percentage of the buffer pool to reserve for 'old' blocks. +Connected to buf_LRU_old_ratio. */ +static uint innobase_old_blocks_pct; + /* The default values for the following char* start-up parameters are determined in innobase_init below: */ @@ -186,9 +191,7 @@ file formats in the configuration file, of the supported file formats during runtime. */ static char* innobase_file_format_check = NULL; -/* The following has a misleading name: starting from 4.0.5, this also -affects Windows: */ -static char* innobase_unix_file_flush_method = NULL; +static char* innobase_file_flush_method = NULL; /* Below we have boolean-valued start-up parameters, and their default values */ @@ -202,7 +205,7 @@ static my_bool innobase_use_doublewrite static my_bool innobase_use_checksums = TRUE; static my_bool innobase_extra_undoslots = FALSE; static my_bool innobase_fast_recovery = FALSE; -static my_bool innobase_use_purge_thread = FALSE; +static my_bool innobase_recovery_stats = TRUE; static my_bool innobase_locks_unsafe_for_binlog = FALSE; static my_bool innobase_overwrite_relay_log_info = FALSE; static my_bool innobase_rollback_on_timeout = FALSE; @@ -238,10 +241,10 @@ static void free_share(INNOBASE_SHARE *s static int innobase_close_connection(handlerton *hton, THD* thd); static int innobase_commit(handlerton *hton, THD* thd, bool all); static int innobase_rollback(handlerton *hton, THD* thd, bool all); -static int innobase_rollback_to_savepoint(handlerton *hton, THD* thd, +static int innobase_rollback_to_savepoint(handlerton *hton, THD* thd, void *savepoint); static int innobase_savepoint(handlerton *hton, THD* thd, void *savepoint); -static int innobase_release_savepoint(handlerton *hton, THD* thd, +static int innobase_release_savepoint(handlerton *hton, THD* thd, void *savepoint); static handler *innobase_create_handler(handlerton *hton, TABLE_SHARE *table, @@ -285,10 +288,10 @@ innobase_file_format_check_on_off( /************************************************************//** Validate the file format check config parameters, as a side effect it sets the srv_check_file_format_at_startup variable. -@return true if valid config value */ +@return the format_id if valid config value, otherwise, return -1 */ static -bool -innobase_file_format_check_validate( +int +innobase_file_format_validate_and_set( /*================================*/ const char* format_check); /*!< in: parameter value */ /****************************************************************//** @@ -519,10 +522,10 @@ static SHOW_VAR innodb_status_variables[ (char*) &export_vars.innodb_buffer_pool_pages_misc, SHOW_LONG}, {"buffer_pool_pages_total", (char*) &export_vars.innodb_buffer_pool_pages_total, SHOW_LONG}, - {"buffer_pool_read_ahead_rnd", - (char*) &export_vars.innodb_buffer_pool_read_ahead_rnd, SHOW_LONG}, - {"buffer_pool_read_ahead_seq", - (char*) &export_vars.innodb_buffer_pool_read_ahead_seq, SHOW_LONG}, + {"buffer_pool_read_ahead", + (char*) &export_vars.innodb_buffer_pool_read_ahead, SHOW_LONG}, + {"buffer_pool_read_ahead_evicted", + (char*) &export_vars.innodb_buffer_pool_read_ahead_evicted, SHOW_LONG}, {"buffer_pool_read_requests", (char*) &export_vars.innodb_buffer_pool_read_requests, SHOW_LONG}, {"buffer_pool_reads", @@ -803,11 +806,20 @@ convert_error_code_to_mysql( case DB_SUCCESS: return(0); + case DB_INTERRUPTED: + my_error(ER_QUERY_INTERRUPTED, MYF(0)); + /* fall through */ case DB_ERROR: default: return(-1); /* unspecified error */ case DB_DUPLICATE_KEY: + /* Be cautious with returning this error, since + mysql could re-enter the storage layer to get + duplicated key info, the operation requires a + valid table handle and/or transaction information, + which might not always be available in the error + handling stage. */ return(HA_ERR_FOUND_DUPP_KEY); case DB_FOREIGN_DUPLICATE_KEY: @@ -894,17 +906,14 @@ convert_error_code_to_mysql( return(ER_PRIMARY_CANT_HAVE_NULL); case DB_TOO_MANY_CONCURRENT_TRXS: - /* Once MySQL add the appropriate code to errmsg.txt then - we can get rid of this #ifdef. NOTE: The code checked by - the #ifdef is the suggested name for the error condition - and the actual error code name could very well be different. - This will require some monitoring, ie. the status - of this request on our part.*/ -#ifdef ER_TOO_MANY_CONCURRENT_TRXS - return(ER_TOO_MANY_CONCURRENT_TRXS); -#else + /* New error code HA_ERR_TOO_MANY_CONCURRENT_TRXS is only + available in 5.1.38 and later, but the plugin should still + work with previous versions of MySQL. */ +#ifdef HA_ERR_TOO_MANY_CONCURRENT_TRXS + return(HA_ERR_TOO_MANY_CONCURRENT_TRXS); +#else /* HA_ERR_TOO_MANY_CONCURRENT_TRXS */ return(HA_ERR_RECORD_FILE_FULL); -#endif +#endif /* HA_ERR_TOO_MANY_CONCURRENT_TRXS */ case DB_UNSUPPORTED: return(HA_ERR_UNSUPPORTED); } @@ -978,7 +987,23 @@ innobase_get_cset_width( *mbminlen = cs->mbminlen; *mbmaxlen = cs->mbmaxlen; } else { - ut_a(cset == 0); + THD* thd = current_thd; + + if (thd && thd_sql_command(thd) == SQLCOM_DROP_TABLE) { + + /* Fix bug#46256: allow tables to be dropped if the + collation is not found, but issue a warning. */ + if ((global_system_variables.log_warnings) + && (cset != 0)){ + + sql_print_warning( + "Unknown collation #%lu.", cset); + } + } else { + + ut_a(cset == 0); + } + *mbminlen = *mbmaxlen = 0; } } @@ -1052,6 +1077,7 @@ innobase_get_charset( } #if defined (__WIN__) && defined (MYSQL_DYNAMIC_PLUGIN) +extern MYSQL_PLUGIN_IMPORT MY_TMPDIR mysql_tmpdir_list; /*******************************************************************//** Map an OS error to an errno value. The OS error number is stored in _doserrno and the mapped value is stored in errno) */ @@ -1339,6 +1365,16 @@ innobase_trx_init( trx->check_unique_secondary = !thd_test_options( thd, OPTION_RELAXED_UNIQUE_CHECKS); +#ifdef EXTENDED_SLOWLOG + if (thd_log_slow_verbosity(thd) & SLOG_V_INNODB) { + trx->take_stats = TRUE; + } else { + trx->take_stats = FALSE; + } +#else + trx->take_stats = FALSE; +#endif + DBUG_VOID_RETURN; } @@ -1395,6 +1431,32 @@ check_trx_exists( } +/************************************************************************* +Gets current trx. */ +extern "C" +trx_t* +innobase_get_trx() +{ + THD *thd=current_thd; + if (likely(thd != 0)) { + trx_t*& trx = thd_to_trx(thd); + return(trx); + } else { + return(NULL); + } +} + +extern "C" +ibool +innobase_get_slow_log() +{ +#ifdef EXTENDED_SLOWLOG + return((ibool) thd_opt_slow_log()); +#else + return(FALSE); +#endif +} + /*********************************************************************//** Construct ha_innobase handler. */ UNIV_INTERN @@ -1711,15 +1773,19 @@ innobase_convert_identifier( FALSE=id is an UTF-8 string */ { char nz[NAME_LEN + 1]; +#if MYSQL_VERSION_ID >= 50141 + char nz2[NAME_LEN + 1 + EXPLAIN_FILENAME_MAX_EXTRA_LENGTH]; +#else /* MYSQL_VERSION_ID >= 50141 */ char nz2[NAME_LEN + 1 + sizeof srv_mysql50_table_name_prefix]; +#endif /* MYSQL_VERSION_ID >= 50141 */ const char* s = id; int q; if (file_id) { - /* Decode the table name. The filename_to_tablename() - function expects a NUL-terminated string. The input and - output strings buffers must not be shared. */ + /* Decode the table name. The MySQL function expects + a NUL-terminated string. The input and output strings + buffers must not be shared. */ if (UNIV_UNLIKELY(idlen > (sizeof nz) - 1)) { idlen = (sizeof nz) - 1; @@ -1729,7 +1795,13 @@ innobase_convert_identifier( nz[idlen] = 0; s = nz2; +#if MYSQL_VERSION_ID >= 50141 + idlen = explain_filename((THD*) thd, nz, nz2, sizeof nz2, + EXPLAIN_PARTITIONS_AS_COMMENT); + goto no_quote; +#else /* MYSQL_VERSION_ID >= 50141 */ idlen = filename_to_tablename(nz, nz2, sizeof nz2); +#endif /* MYSQL_VERSION_ID >= 50141 */ } /* See if the identifier needs to be quoted. */ @@ -1740,6 +1812,9 @@ innobase_convert_identifier( } if (q == EOF) { +#if MYSQL_VERSION_ID >= 50141 +no_quote: +#endif /* MYSQL_VERSION_ID >= 50141 */ if (UNIV_UNLIKELY(idlen > buflen)) { idlen = buflen; } @@ -2222,8 +2297,8 @@ mem_free_and_error: /* Did the user specify a format name that we support ? As a side effect it will update the variable srv_check_file_format_at_startup */ - if (!innobase_file_format_check_validate( - innobase_file_format_check)) { + if (innobase_file_format_validate_and_set( + innobase_file_format_check) < 0) { sql_print_error("InnoDB: invalid " "innodb_file_format_check value: " @@ -2264,7 +2339,7 @@ innobase_change_buffering_inited_ok: /* --------------------------------------------------*/ - srv_file_flush_method_str = innobase_unix_file_flush_method; + srv_file_flush_method_str = innobase_file_flush_method; srv_n_log_groups = (ulint) innobase_mirrored_log_groups; srv_n_log_files = (ulint) innobase_log_files_in_group; @@ -2292,8 +2367,7 @@ innobase_change_buffering_inited_ok: srv_force_recovery = (ulint) innobase_force_recovery; srv_fast_recovery = (ibool) innobase_fast_recovery; - - srv_use_purge_thread = (ibool) innobase_use_purge_thread; + srv_recovery_stats = (ibool) innobase_recovery_stats; srv_use_doublewrite_buf = (ibool) innobase_use_doublewrite; srv_use_checksums = (ibool) innobase_use_checksums; @@ -2329,6 +2403,9 @@ innobase_change_buffering_inited_ok: ut_a(0 == strcmp(my_charset_latin1.name, "latin1_swedish_ci")); srv_latin1_ordering = my_charset_latin1.sort_order; + innobase_old_blocks_pct = buf_LRU_old_ratio_update( + innobase_old_blocks_pct, FALSE); + innobase_commit_concurrency_init_default(); /* Since we in this module access directly the fields of a trx @@ -2510,7 +2587,10 @@ innobase_alter_table_flags( { return(HA_ONLINE_ADD_INDEX_NO_WRITES | HA_ONLINE_DROP_INDEX_NO_WRITES - | HA_ONLINE_ADD_UNIQUE_INDEX_NO_WRITES + /* Current InnoDB doesn't sort unique indexes along mysqld's order + It is dangerous to use index. So it is disabled until + the bug http://bugs.mysql.com/47622 */ + /* | HA_ONLINE_ADD_UNIQUE_INDEX_NO_WRITES */ | HA_ONLINE_DROP_UNIQUE_INDEX_NO_WRITES | HA_ONLINE_ADD_PK_INDEX_NO_WRITES); } @@ -2676,6 +2756,19 @@ retry: } } + /* The following calls to read the MySQL binary log + file name and the position return consistent results: + 1) Other InnoDB transactions cannot intervene between + these calls as we are holding prepare_commit_mutex. + 2) Binary logging of other engines is not relevant + to InnoDB as all InnoDB requires is that committing + InnoDB transactions appear in the same order in the + MySQL binary log as they appear in InnoDB logs. + 3) A MySQL log file rotation cannot happen because + MySQL protects against this by having a counter of + transactions in prepared state and it only allows + a rotation when the counter drops to zero. See + LOCK_prep_xids and COND_prep_xids in log.cc. */ trx->mysql_log_file_name = mysql_bin_log_file_name(); trx->mysql_log_offset = (ib_int64_t) mysql_bin_log_file_pos(); @@ -2761,6 +2854,8 @@ innobase_rollback( innobase_release_stat_resources(trx); + trx->n_autoinc_rows = 0; /* Reset the number AUTO-INC rows required */ + /* If we had reserved the auto-inc lock for some table (if we come here to roll back the latest SQL statement) we release it now before a possibly lengthy rollback */ @@ -3322,7 +3417,7 @@ retry: if (is_part) { sql_print_error("Failed to open table %s after " - "%lu attemtps.\n", norm_name, + "%lu attempts.\n", norm_name, retries); } @@ -3926,7 +4021,6 @@ ha_innobase::store_key_val_for_row( as BLOB data in innodb. */ || mysql_type == MYSQL_TYPE_GEOMETRY) { - CHARSET_INFO* cs; ulint key_len; ulint true_len; @@ -4648,24 +4742,29 @@ no_commit: update the table upper limit. Note: last_value will be 0 if get_auto_increment() was not called.*/ - if (auto_inc <= col_max_value - && auto_inc >= prebuilt->autoinc_last_value) { + if (auto_inc >= prebuilt->autoinc_last_value) { set_max_autoinc: - ut_a(prebuilt->autoinc_increment > 0); - - ulonglong need; - ulonglong offset; - - offset = prebuilt->autoinc_offset; - need = prebuilt->autoinc_increment; - - auto_inc = innobase_next_autoinc( - auto_inc, need, offset, col_max_value); - - err = innobase_set_max_autoinc(auto_inc); - - if (err != DB_SUCCESS) { - error = err; + /* This should filter out the negative + values set explicitly by the user. */ + if (auto_inc <= col_max_value) { + ut_a(prebuilt->autoinc_increment > 0); + + ulonglong need; + ulonglong offset; + + offset = prebuilt->autoinc_offset; + need = prebuilt->autoinc_increment; + + auto_inc = innobase_next_autoinc( + auto_inc, + need, offset, col_max_value); + + err = innobase_set_max_autoinc( + auto_inc); + + if (err != DB_SUCCESS) { + error = err; + } } } break; @@ -5227,6 +5326,11 @@ ha_innobase::index_read( index = prebuilt->index; + if (UNIV_UNLIKELY(index == NULL)) { + prebuilt->index_usable = FALSE; + DBUG_RETURN(HA_ERR_CRASHED); + } + /* Note that if the index for which the search template is built is not necessarily prebuilt->index, but can also be the clustered index */ @@ -5386,6 +5490,7 @@ ha_innobase::change_active_index( if (UNIV_UNLIKELY(!prebuilt->index)) { sql_print_warning("InnoDB: change_active_index(%u) failed", keynr); + prebuilt->index_usable = FALSE; DBUG_RETURN(1); } @@ -5393,8 +5498,10 @@ ha_innobase::change_active_index( prebuilt->index); if (UNIV_UNLIKELY(!prebuilt->index_usable)) { - sql_print_warning("InnoDB: insufficient history for index %u", - keynr); + push_warning_printf(user_thd, MYSQL_ERROR::WARN_LEVEL_WARN, + HA_ERR_TABLE_DEF_CHANGED, + "InnoDB: insufficient history for index %u", + keynr); /* The caller seems to ignore this. Thus, we must check this again in row_search_for_mysql(). */ DBUG_RETURN(convert_error_code_to_mysql(DB_MISSING_HISTORY, @@ -5850,7 +5957,7 @@ create_table_def( number fits in one byte in prtype */ push_warning_printf( (THD*) trx->mysql_thd, - MYSQL_ERROR::WARN_LEVEL_ERROR, + MYSQL_ERROR::WARN_LEVEL_WARN, ER_CANT_CREATE_TABLE, "In InnoDB, charset-collation codes" " must be below 256." @@ -5882,17 +5989,8 @@ create_table_def( /* First check whether the column to be added has a system reserved name. */ if (dict_col_name_is_reserved(field->field_name)){ - push_warning_printf( - (THD*) trx->mysql_thd, - MYSQL_ERROR::WARN_LEVEL_WARN, - ER_CANT_CREATE_TABLE, - "Error creating table '%s' with " - "column name '%s'. '%s' is a " - "reserved name. Please try to " - "re-create the table with a " - "different column name.", - table->name, (char*) field->field_name, - (char*) field->field_name); + my_error(ER_WRONG_COLUMN_NAME, MYF(0), + field->field_name); dict_mem_table_free(table); trx_commit_for_mysql(trx); @@ -5914,6 +6012,14 @@ create_table_def( error = row_create_table_for_mysql(table, trx); + if (error == DB_DUPLICATE_KEY) { + char buf[100]; + innobase_convert_identifier(buf, sizeof buf, + table_name, strlen(table_name), + trx->mysql_thd, TRUE); + my_error(ER_TABLE_EXISTS_ERROR, MYF(0), buf); + } + error_ret: error = convert_error_code_to_mysql(error, flags, NULL); @@ -6064,7 +6170,6 @@ create_clustered_index_when_no_primary( /* We pass 0 as the space id, and determine at a lower level the space id where to store the table */ - index = dict_mem_index_create(table_name, innobase_index_reserve_name, 0, DICT_CLUSTERED, 0); @@ -6119,7 +6224,7 @@ create_options_are_valid( /* Valid value. */ break; default: - push_warning_printf(thd, MYSQL_ERROR::WARN_LEVEL_ERROR, + push_warning_printf(thd, MYSQL_ERROR::WARN_LEVEL_WARN, ER_ILLEGAL_HA_CREATE_OPTION, "InnoDB: invalid" " KEY_BLOCK_SIZE = %lu." @@ -6133,7 +6238,7 @@ create_options_are_valid( /* If KEY_BLOCK_SIZE was specified, check for its dependencies. */ if (kbs_specified && !srv_file_per_table) { - push_warning(thd, MYSQL_ERROR::WARN_LEVEL_ERROR, + push_warning(thd, MYSQL_ERROR::WARN_LEVEL_WARN, ER_ILLEGAL_HA_CREATE_OPTION, "InnoDB: KEY_BLOCK_SIZE" " requires innodb_file_per_table."); @@ -6141,7 +6246,7 @@ create_options_are_valid( } if (kbs_specified && srv_file_format < DICT_TF_FORMAT_ZIP) { - push_warning(thd, MYSQL_ERROR::WARN_LEVEL_ERROR, + push_warning(thd, MYSQL_ERROR::WARN_LEVEL_WARN, ER_ILLEGAL_HA_CREATE_OPTION, "InnoDB: KEY_BLOCK_SIZE" " requires innodb_file_format >" @@ -6165,7 +6270,7 @@ create_options_are_valid( if (!srv_file_per_table) { push_warning_printf( thd, - MYSQL_ERROR::WARN_LEVEL_ERROR, + MYSQL_ERROR::WARN_LEVEL_WARN, ER_ILLEGAL_HA_CREATE_OPTION, "InnoDB: ROW_FORMAT=%s" " requires innodb_file_per_table.", @@ -6177,7 +6282,7 @@ create_options_are_valid( if (srv_file_format < DICT_TF_FORMAT_ZIP) { push_warning_printf( thd, - MYSQL_ERROR::WARN_LEVEL_ERROR, + MYSQL_ERROR::WARN_LEVEL_WARN, ER_ILLEGAL_HA_CREATE_OPTION, "InnoDB: ROW_FORMAT=%s" " requires innodb_file_format >" @@ -6194,7 +6299,7 @@ create_options_are_valid( && form->s->row_type == ROW_TYPE_DYNAMIC) { push_warning_printf( thd, - MYSQL_ERROR::WARN_LEVEL_ERROR, + MYSQL_ERROR::WARN_LEVEL_WARN, ER_ILLEGAL_HA_CREATE_OPTION, "InnoDB: cannot specify" " ROW_FORMAT = DYNAMIC with" @@ -6218,7 +6323,7 @@ create_options_are_valid( if (kbs_specified) { push_warning_printf( thd, - MYSQL_ERROR::WARN_LEVEL_ERROR, + MYSQL_ERROR::WARN_LEVEL_WARN, ER_ILLEGAL_HA_CREATE_OPTION, "InnoDB: cannot specify" " ROW_FORMAT = %s with" @@ -6231,7 +6336,7 @@ create_options_are_valid( default: push_warning(thd, - MYSQL_ERROR::WARN_LEVEL_ERROR, + MYSQL_ERROR::WARN_LEVEL_WARN, ER_ILLEGAL_HA_CREATE_OPTION, "InnoDB: invalid ROW_FORMAT specifier."); ret = FALSE; @@ -6295,13 +6400,15 @@ ha_innobase::create( 1. <database_name>/<table_name>: for normal table creation 2. full path: for temp table creation, or sym link - When srv_file_per_table is on, check for full path pattern, i.e. + When srv_file_per_table is on and mysqld_embedded is off, + check for full path pattern, i.e. X:\dir\..., X is a driver letter, or \\dir1\dir2\..., UNC path returns error if it is in full path format, but not creating a temp. table. Currently InnoDB does not support symbolic link on Windows. */ if (srv_file_per_table + && !mysqld_embedded && (!create_info->options & HA_LEX_CREATE_TMP_TABLE)) { if ((name[1] == ':') @@ -6519,6 +6626,7 @@ ha_innobase::create( goto cleanup; } + /* Create the keys */ if (form->s->keys == 0 || primary_key_no == -1) { @@ -6967,6 +7075,24 @@ ha_innobase::rename_table( innobase_commit_low(trx); trx_free_for_mysql(trx); + /* Add a special case to handle the Duplicated Key error + and return DB_ERROR instead. + This is to avoid a possible SIGSEGV error from mysql error + handling code. Currently, mysql handles the Duplicated Key + error by re-entering the storage layer and getting dup key + info by calling get_dup_key(). This operation requires a valid + table handle ('row_prebuilt_t' structure) which could no + longer be available in the error handling stage. The suggested + solution is to report a 'table exists' error message (since + the dup key error here is due to an existing table whose name + is the one we are trying to rename to) and return the generic + error code. */ + if (error == (int) DB_DUPLICATE_KEY) { + my_error(ER_TABLE_EXISTS_ERROR, MYF(0), to); + + error = DB_ERROR; + } + error = convert_error_code_to_mysql(error, 0, NULL); DBUG_RETURN(error); @@ -7519,11 +7645,15 @@ ha_innobase::check( ret = row_check_table_for_mysql(prebuilt); - if (ret == DB_SUCCESS) { + switch (ret) { + case DB_SUCCESS: return(HA_ADMIN_OK); + case DB_INTERRUPTED: + my_error(ER_QUERY_INTERRUPTED, MYF(0)); + return(-1); + default: + return(HA_ADMIN_CORRUPT); } - - return(HA_ADMIN_CORRUPT); } /*************************************************************//** @@ -8069,8 +8199,11 @@ ha_innobase::external_lock( ulong const binlog_format= thd_binlog_format(thd); ulong const tx_isolation = thd_tx_isolation(ha_thd()); if (tx_isolation <= ISO_READ_COMMITTED - && binlog_format == BINLOG_FORMAT_STMT - && thd_binlog_filter_ok(thd)) + && binlog_format == BINLOG_FORMAT_STMT +#if MYSQL_VERSION_ID > 50140 + && thd_binlog_filter_ok(thd) +#endif /* MYSQL_VERSION_ID > 50140 */ + ) { char buf[256]; my_snprintf(buf, sizeof(buf), @@ -8183,6 +8316,23 @@ ha_innobase::external_lock( statement has ended */ if (trx->n_mysql_tables_in_use == 0) { +#ifdef EXTENDED_SLOWLOG + increment_thd_innodb_stats(thd, trx->io_reads, + trx->io_read, + trx->io_reads_wait_timer, + trx->lock_que_wait_timer, + trx->innodb_que_wait_timer, + trx->distinct_page_access); + + trx->io_reads = 0; + trx->io_read = 0; + trx->io_reads_wait_timer = 0; + trx->lock_que_wait_timer = 0; + trx->innodb_que_wait_timer = 0; + trx->distinct_page_access = 0; + if (trx->distinct_page_access_hash) + memset(trx->distinct_page_access_hash, 0, DPAH_SIZE); +#endif trx->mysql_n_tables_locked = 0; prebuilt->used_in_HANDLER = FALSE; @@ -8470,8 +8620,8 @@ innodb_mutex_show_status( rw_lock_wait_time += mutex->lspent_time; } #else /* UNIV_DEBUG */ - buf1len= (uint) my_snprintf(buf1, sizeof(buf1), "%s:%lu", - mutex->cfile_name, (ulong) mutex->cline); + buf1len= (uint) my_snprintf(buf1, sizeof(buf1), "%s", + mutex->cmutex_name); buf2len= (uint) my_snprintf(buf2, sizeof(buf2), "os_waits=%lu", mutex->count_os_wait); @@ -8496,8 +8646,8 @@ next_mutex: while (lock != NULL) { if (lock->count_os_wait && !buf_pool_is_block_lock(lock)) { - buf1len= my_snprintf(buf1, sizeof(buf1), "%s:%lu", - lock->cfile_name, (ulong) lock->cline); + buf1len= my_snprintf(buf1, sizeof(buf1), "%s", + lock->lock_name); buf2len= my_snprintf(buf2, sizeof(buf2), "os_waits=%lu", lock->count_os_wait); @@ -8718,6 +8868,7 @@ ha_innobase::store_lock( && isolation_level != TRX_ISO_SERIALIZABLE && (lock_type == TL_READ || lock_type == TL_READ_NO_INSERT) && (sql_command == SQLCOM_INSERT_SELECT + || sql_command == SQLCOM_REPLACE_SELECT || sql_command == SQLCOM_UPDATE || sql_command == SQLCOM_CREATE_TABLE)) { @@ -8725,10 +8876,11 @@ ha_innobase::store_lock( option set or this session is using READ COMMITTED isolation level and isolation level of the transaction is not set to serializable and MySQL is doing - INSERT INTO...SELECT or UPDATE ... = (SELECT ...) or - CREATE ... SELECT... without FOR UPDATE or - IN SHARE MODE in select, then we use consistent - read for select. */ + INSERT INTO...SELECT or REPLACE INTO...SELECT + or UPDATE ... = (SELECT ...) or CREATE ... + SELECT... without FOR UPDATE or IN SHARE + MODE in select, then we use consistent read + for select. */ prebuilt->select_lock_type = LOCK_NONE; prebuilt->stored_select_lock_type = LOCK_NONE; @@ -8957,8 +9109,7 @@ ha_innobase::get_auto_increment( col_max_value = innobase_get_int_col_max_value( table->next_number_field); - current = *first_value > col_max_value ? autoinc : *first_value; - + current = *first_value > col_max_value ? autoinc : *first_value; need = *nb_reserved_values * increment; /* Compute the last value in the interval */ @@ -9322,8 +9473,7 @@ innobase_xa_prepare( executing XA PREPARE and XA COMMIT commands. In this case we cannot know how many minutes or hours will be between XA PREPARE and XA COMMIT, and we don't want - to block for undefined period of time. - */ + to block for undefined period of time. */ pthread_mutex_lock(&prepare_commit_mutex); trx->active_trans = 2; } @@ -9575,6 +9725,20 @@ ha_innobase::check_if_incompatible_data( DBUG_RETURN(COMPATIBLE_DATA_NO); } + /* Renaming column asynchronizes dictionary between mysqld and InnoDB... + If not synchronized, treat as COMPATIBLE_DATA_NO + until the bug http://bugs.mysql.com/47621 is fixed officialily */ + { + uint i; + for (i = 0; i < table->s->fields; i++) { + if (table->field[i]->flags & FIELD_IN_ADD_INDEX + && innobase_strcasecmp(table->field[i]->field_name, + dict_table_get_col_name(prebuilt->table, i))) { + DBUG_RETURN(COMPATIBLE_DATA_NO); + } + } + } + /* Check if a column participating in a foreign key is being renamed. There is no mechanism for updating InnoDB foreign key definitions. */ if (foreign_key_column_is_being_renamed(prebuilt, table)) { @@ -9683,25 +9847,24 @@ innobase_file_format_check_on_off( /************************************************************//** Validate the file format check config parameters, as a side effect it sets the srv_check_file_format_at_startup variable. -@return true if valid config value */ +@return the format_id if valid config value, otherwise, return -1 */ static -bool -innobase_file_format_check_validate( +int +innobase_file_format_validate_and_set( /*================================*/ const char* format_check) /*!< in: parameter value */ { uint format_id; - bool ret = true; format_id = innobase_file_format_name_lookup(format_check); if (format_id < DICT_TF_FORMAT_MAX + 1) { srv_check_file_format_at_startup = format_id; + + return((int) format_id); } else { - ret = false; + return(-1); } - - return(ret); } /*************************************************************//** @@ -9720,7 +9883,6 @@ innodb_file_format_name_validate( struct st_mysql_value* value) /*!< in: incoming string */ { const char* file_format_input; - char* file_format_input_strdup; char buff[STRING_BUFFER_USUAL_SIZE]; int len = sizeof(buff); @@ -9737,18 +9899,12 @@ innodb_file_format_name_validate( if (format_id <= DICT_TF_FORMAT_MAX) { - /* Copy out from stack-allocated memory (which will not - survive return from this function). The memory will be - freed in innodb_file_format_check_update(). */ - file_format_input_strdup = thd_strmake(thd, file_format_input, len); + /* Save a pointer to the name in the + 'file_format_name_map' constant array. */ + *static_cast<const char**>(save) = + trx_sys_file_format_id_to_name(format_id); - *static_cast<char**>(save) = file_format_input_strdup; - - if (file_format_input_strdup == NULL) { - return(1); - } else { - return(0); - } + return(0); } } @@ -9808,9 +9964,9 @@ innodb_file_format_check_validate( struct st_mysql_value* value) /*!< in: incoming string */ { const char* file_format_input; - char* file_format_input_strdup; char buff[STRING_BUFFER_USUAL_SIZE]; int len = sizeof(buff); + int format_id; ut_a(save != NULL); ut_a(value != NULL); @@ -9823,33 +9979,35 @@ innodb_file_format_check_validate( message if they did so. */ if (innobase_file_format_check_on_off(file_format_input)) { - sql_print_warning( + push_warning_printf(thd, + MYSQL_ERROR::WARN_LEVEL_WARN, + ER_WRONG_ARGUMENTS, "InnoDB: invalid innodb_file_format_check " "value; on/off can only be set at startup or " "in the configuration file"); - } else if (innobase_file_format_check_validate( - file_format_input)) { + } else { + format_id = innobase_file_format_validate_and_set( + file_format_input); - /* Copy out from stack-allocated memory (which will not - survive return from this function). The memory will be - freed in innodb_file_format_check_update(). */ - file_format_input_strdup = thd_strmake(thd, file_format_input, len); + if (format_id >= 0) { + /* Save a pointer to the name in the + 'file_format_name_map' constant array. */ + *static_cast<const char**>(save) = + trx_sys_file_format_id_to_name( + (uint)format_id); - *static_cast<char**>(save) = file_format_input_strdup; + return(0); - if (file_format_input_strdup == NULL) { - return(1); } else { - return(0); + push_warning_printf(thd, + MYSQL_ERROR::WARN_LEVEL_WARN, + ER_WRONG_ARGUMENTS, + "InnoDB: invalid innodb_file_format_check " + "value; can be any format up to %s " + "or its equivalent numeric id", + trx_sys_file_format_id_to_name( + DICT_TF_FORMAT_MAX)); } - - } else { - sql_print_warning( - "InnoDB: invalid innodb_file_format_check " - "value; can be any format up to %s " - "or its equivalent numeric id", - trx_sys_file_format_id_to_name( - DICT_TF_FORMAT_MAX)); } } @@ -9880,6 +10038,7 @@ innodb_file_format_check_update( ut_a(var_ptr != NULL); format_name_in = *static_cast<const char*const*>(save); + if (!format_name_in) { return; @@ -9929,6 +10088,25 @@ innodb_adaptive_hash_index_update( } } +/****************************************************************//** +Update the system variable innodb_old_blocks_pct using the "saved" +value. This function is registered as a callback with MySQL. */ +static +void +innodb_old_blocks_pct_update( +/*=========================*/ + THD* thd, /*!< in: thread handle */ + struct st_mysql_sys_var* var, /*!< in: pointer to + system variable */ + void* var_ptr,/*!< out: where the + formal string goes */ + const void* save) /*!< in: immediate result + from check function */ +{ + innobase_old_blocks_pct = buf_LRU_old_ratio_update( + *static_cast<const uint*>(save), TRUE); +} + /*************************************************************//** Check if it is a valid value of innodb_change_buffering. This function is registered as a callback with MySQL. @@ -10075,13 +10253,18 @@ static MYSQL_SYSVAR_BOOL(extra_undoslots static MYSQL_SYSVAR_BOOL(fast_recovery, innobase_fast_recovery, PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY, "Enable to use speed hack of recovery avoiding flush list sorting.", - NULL, NULL, FALSE); + NULL, NULL, TRUE); -static MYSQL_SYSVAR_BOOL(use_purge_thread, innobase_use_purge_thread, +static MYSQL_SYSVAR_BOOL(recovery_stats, innobase_recovery_stats, PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY, - "Enable to use purge devoted thread.", + "Output statistics of recovery process after it.", NULL, NULL, FALSE); +static MYSQL_SYSVAR_ULONG(use_purge_thread, srv_use_purge_thread, + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY, + "Number of purge devoted threads. #### over 1 is EXPERIMENTAL ####", + NULL, NULL, 1, 0, 64, 0); + static MYSQL_SYSVAR_BOOL(overwrite_relay_log_info, innobase_overwrite_relay_log_info, PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY, "During InnoDB crash recovery on slave overwrite relay-log.info " @@ -10122,12 +10305,15 @@ static MYSQL_SYSVAR_STR(file_format, inn innodb_file_format_name_validate, innodb_file_format_name_update, "Antelope"); +/* If a new file format is introduced, the file format +name needs to be updated accordingly. Please refer to +file_format_name_map[] defined in trx0sys.c for the next +file format name. */ static MYSQL_SYSVAR_STR(file_format_check, innobase_file_format_check, PLUGIN_VAR_OPCMDARG, "The highest file format in the tablespace.", innodb_file_format_check_validate, - innodb_file_format_check_update, - "on"); + innodb_file_format_check_update, "Barracuda"); static MYSQL_SYSVAR_ULONG(flush_log_at_trx_commit, srv_flush_log_at_trx_commit, PLUGIN_VAR_OPCMDARG, @@ -10136,7 +10322,7 @@ static MYSQL_SYSVAR_ULONG(flush_log_at_t " or 2 (write at commit, flush once per second).", NULL, NULL, 1, 0, 2, 0); -static MYSQL_SYSVAR_STR(flush_method, innobase_unix_file_flush_method, +static MYSQL_SYSVAR_STR(flush_method, innobase_file_flush_method, PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY, "With which method to flush data.", NULL, NULL, NULL); @@ -10177,7 +10363,7 @@ static MYSQL_SYSVAR_ULONG(max_dirty_page static MYSQL_SYSVAR_BOOL(adaptive_flushing, srv_adaptive_flushing, PLUGIN_VAR_NOCMDARG, "Attempt flushing dirty pages to avoid IO bursts at checkpoints.", - NULL, NULL, TRUE); + NULL, NULL, FALSE); static MYSQL_SYSVAR_ULONG(max_purge_lag, srv_max_purge_lag, PLUGIN_VAR_RQCMDARG, @@ -10273,7 +10459,7 @@ static MYSQL_SYSVAR_ULONG(concurrency_ti NULL, NULL, 500L, 1L, ~0L, 0); static MYSQL_SYSVAR_LONG(file_io_threads, innobase_file_io_threads, - PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY, + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY | PLUGIN_VAR_NOSYSVAR, "Number of file I/O threads in InnoDB.", NULL, NULL, 4, 4, 64, 0); @@ -10312,6 +10498,18 @@ static MYSQL_SYSVAR_LONG(mirrored_log_gr "Number of identical copies of log groups we keep for the database. Currently this should be set to 1.", NULL, NULL, 1, 1, 10, 0); +static MYSQL_SYSVAR_UINT(old_blocks_pct, innobase_old_blocks_pct, + PLUGIN_VAR_RQCMDARG, + "Percentage of the buffer pool to reserve for 'old' blocks.", + NULL, innodb_old_blocks_pct_update, 100 * 3 / 8, 5, 95, 0); + +static MYSQL_SYSVAR_UINT(old_blocks_time, buf_LRU_old_threshold_ms, + PLUGIN_VAR_RQCMDARG, + "Move blocks to the 'new' end of the buffer pool if the first access" + " was at least this many milliseconds ago." + " The timeout is disabled if 0 (the default).", + NULL, NULL, 0, 0, UINT_MAX32, 0); + static MYSQL_SYSVAR_LONG(open_files, innobase_open_files, PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY, "How many files at the maximum InnoDB keeps open at the same time.", @@ -10390,13 +10588,18 @@ static MYSQL_SYSVAR_LONGLONG(ibuf_max_si static MYSQL_SYSVAR_ULONG(ibuf_active_contract, srv_ibuf_active_contract, PLUGIN_VAR_RQCMDARG, "Enable/Disable active_contract of insert buffer. 0:disable 1:enable", - NULL, NULL, 0, 0, 1, 0); + NULL, NULL, 1, 0, 1, 0); static MYSQL_SYSVAR_ULONG(ibuf_accel_rate, srv_ibuf_accel_rate, PLUGIN_VAR_RQCMDARG, "Tunes amount of insert buffer processing of background, in addition to innodb_io_capacity. (in percentage)", NULL, NULL, 100, 100, 999999999, 0); +static MYSQL_SYSVAR_ULONG(checkpoint_age_target, srv_checkpoint_age_target, + PLUGIN_VAR_RQCMDARG, + "Control soft limit of checkpoint age. (0 : not control)", + NULL, NULL, 0, 0, ~0UL, 0); + static MYSQL_SYSVAR_ULONG(flush_neighbor_pages, srv_flush_neighbor_pages, PLUGIN_VAR_RQCMDARG, "Enable/Disable flushing also neighbor pages. 0:disable 1:enable", @@ -10432,7 +10635,7 @@ TYPELIB read_ahead_typelib= }; static MYSQL_SYSVAR_ENUM(read_ahead, srv_read_ahead, PLUGIN_VAR_RQCMDARG, - "Control read ahead activity. (none, random, [linear], both)", + "Control read ahead activity (none, random, [linear], both). [from 1.0.5: random read ahead is ignored]", NULL, innodb_read_ahead_update, 2, &read_ahead_typelib); static @@ -10463,8 +10666,8 @@ TYPELIB adaptive_checkpoint_typelib= }; static MYSQL_SYSVAR_ENUM(adaptive_checkpoint, srv_adaptive_checkpoint, PLUGIN_VAR_RQCMDARG, - "Enable/Disable flushing along modified age. ([none], reflex, estimate)", - NULL, innodb_adaptive_checkpoint_update, 0, &adaptive_checkpoint_typelib); + "Enable/Disable flushing along modified age. (none, reflex, [estimate])", + NULL, innodb_adaptive_checkpoint_update, 2, &adaptive_checkpoint_typelib); static MYSQL_SYSVAR_ULONG(enable_unsafe_group_commit, srv_enable_unsafe_group_commit, PLUGIN_VAR_RQCMDARG, @@ -10486,6 +10689,11 @@ static MYSQL_SYSVAR_ULONG(dict_size_limi "Limit the allocated memory for dictionary cache. (0: unlimited)", NULL, NULL, 0, 0, LONG_MAX, 0); +static MYSQL_SYSVAR_ULONG(relax_table_creation, srv_relax_table_creation, + PLUGIN_VAR_RQCMDARG, + "Relax limitation of column size at table creation as builtin InnoDB.", + NULL, NULL, 0, 0, 1, 0); + static struct st_mysql_sys_var* innobase_system_variables[]= { MYSQL_SYSVAR(additional_mem_pool_size), MYSQL_SYSVAR(autoextend_increment), @@ -10498,6 +10706,7 @@ static struct st_mysql_sys_var* innobase MYSQL_SYSVAR(doublewrite), MYSQL_SYSVAR(extra_undoslots), MYSQL_SYSVAR(fast_recovery), + MYSQL_SYSVAR(recovery_stats), MYSQL_SYSVAR(fast_shutdown), MYSQL_SYSVAR(file_io_threads), MYSQL_SYSVAR(read_io_threads), @@ -10522,6 +10731,8 @@ static struct st_mysql_sys_var* innobase MYSQL_SYSVAR(adaptive_flushing), MYSQL_SYSVAR(max_purge_lag), MYSQL_SYSVAR(mirrored_log_groups), + MYSQL_SYSVAR(old_blocks_pct), + MYSQL_SYSVAR(old_blocks_time), MYSQL_SYSVAR(open_files), MYSQL_SYSVAR(overwrite_relay_log_info), MYSQL_SYSVAR(rollback_on_timeout), @@ -10548,6 +10759,7 @@ static struct st_mysql_sys_var* innobase MYSQL_SYSVAR(ibuf_max_size), MYSQL_SYSVAR(ibuf_active_contract), MYSQL_SYSVAR(ibuf_accel_rate), + MYSQL_SYSVAR(checkpoint_age_target), MYSQL_SYSVAR(flush_neighbor_pages), MYSQL_SYSVAR(read_ahead), MYSQL_SYSVAR(adaptive_checkpoint), @@ -10560,6 +10772,7 @@ static struct st_mysql_sys_var* innobase MYSQL_SYSVAR(read_ahead_threshold), MYSQL_SYSVAR(io_capacity), MYSQL_SYSVAR(use_purge_thread), + MYSQL_SYSVAR(relax_table_creation), NULL }; @@ -10591,6 +10804,7 @@ i_s_innodb_cmpmem, i_s_innodb_cmpmem_reset, i_s_innodb_table_stats, i_s_innodb_index_stats, +i_s_innodb_admin_command, i_s_innodb_patches mysql_declare_plugin_end; === modified file 'storage/xtradb/handler/ha_innodb.h' --- a/storage/xtradb/handler/ha_innodb.h 2009-12-03 11:34:11 +0000 +++ b/storage/xtradb/handler/ha_innodb.h 2010-01-15 15:58:25 +0000 @@ -258,12 +258,14 @@ int thd_binlog_format(const MYSQL_THD th */ void thd_mark_transaction_to_rollback(MYSQL_THD thd, bool all); +#if MYSQL_VERSION_ID > 50140 /** Check if binary logging is filtered for thread's current db. @param thd Thread handle @retval 1 the query is not filtered, 0 otherwise. */ bool thd_binlog_filter_ok(const MYSQL_THD thd); +#endif /* MYSQL_VERSION_ID > 50140 */ } typedef struct trx_struct trx_t; @@ -289,6 +291,8 @@ trx_t* innobase_trx_allocate( /*==================*/ MYSQL_THD thd); /*!< in: user thread handle */ + + /*********************************************************************//** This function checks each index name for a table against reserved system default primary index name 'GEN_CLUST_INDEX'. If a name === modified file 'storage/xtradb/handler/handler0alter.cc' --- a/storage/xtradb/handler/handler0alter.cc 2009-12-03 11:34:11 +0000 +++ b/storage/xtradb/handler/handler0alter.cc 2010-01-15 15:58:25 +0000 @@ -35,7 +35,6 @@ extern "C" { } #include "ha_innodb.h" -#include "handler0vars.h" /*************************************************************//** Copies an InnoDB column to a MySQL field. This function is @@ -629,7 +628,7 @@ ha_innobase::add_index( ulint num_created = 0; ibool dict_locked = FALSE; ulint new_primary; - ulint error; + int error; DBUG_ENTER("ha_innobase::add_index"); ut_a(table); @@ -668,7 +667,7 @@ ha_innobase::add_index( if (UNIV_UNLIKELY(error)) { err_exit: mem_heap_free(heap); - trx_general_rollback_for_mysql(trx, FALSE, NULL); + trx_general_rollback_for_mysql(trx, NULL); trx_free_for_mysql(trx); trx_commit_for_mysql(prebuilt->trx); DBUG_RETURN(error); @@ -766,10 +765,11 @@ err_exit: ut_ad(error == DB_SUCCESS); /* Commit the data dictionary transaction in order to release - the table locks on the system tables. Unfortunately, this - means that if MySQL crashes while creating a new primary key - inside row_merge_build_indexes(), indexed_table will not be - dropped on crash recovery. Thus, it will become orphaned. */ + the table locks on the system tables. This means that if + MySQL crashes while creating a new primary key inside + row_merge_build_indexes(), indexed_table will not be dropped + by trx_rollback_active(). It will have to be recovered or + dropped by the database administrator. */ trx_commit_for_mysql(trx); row_mysql_unlock_data_dictionary(trx); @@ -806,7 +806,7 @@ error_handling: alter table t drop index b, add index (b); The fix will have to parse the SQL and note that the index - being added has the same name as the the one being dropped and + being added has the same name as the one being dropped and ignore that in the dup index check.*/ //dict_table_check_for_dup_indexes(prebuilt->table); #endif @@ -868,6 +868,7 @@ error_handling: indexed_table->n_mysql_handles_opened++; error = row_merge_drop_table(trx, innodb_table); + innodb_table = indexed_table; goto convert_error; case DB_TOO_BIG_RECORD: @@ -882,7 +883,9 @@ error: /* fall through */ default: if (new_primary) { - row_merge_drop_table(trx, indexed_table); + if (indexed_table != innodb_table) { + row_merge_drop_table(trx, indexed_table); + } } else { if (!dict_locked) { row_mysql_lock_data_dictionary(trx); === removed file 'storage/xtradb/handler/handler0vars.h' --- a/storage/xtradb/handler/handler0vars.h 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/handler/handler0vars.h 1970-01-01 00:00:00 +0000 @@ -1,73 +0,0 @@ -/***************************************************************************** - -Copyright (c) 2008, 2009, Innobase Oy. All Rights Reserved. - -This program is free software; you can redistribute it and/or modify it under -the terms of the GNU General Public License as published by the Free Software -Foundation; version 2 of the License. - -This program is distributed in the hope that it will be useful, but WITHOUT -ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS -FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - -You should have received a copy of the GNU General Public License along with -this program; if not, write to the Free Software Foundation, Inc., 59 Temple -Place, Suite 330, Boston, MA 02111-1307 USA - -*****************************************************************************/ - -/*******************************************************************//** -@file handler/handler0vars.h -This file contains accessor functions for dynamic plugin on Windows. -***********************************************************************/ - -#if defined __WIN__ && defined MYSQL_DYNAMIC_PLUGIN -/*******************************************************************//** -This is a list of externals that can not be resolved by delay loading. -They have to be resolved indirectly via their addresses in the .map file. -All of them are external variables. */ -extern MYSQL_PLUGIN_IMPORT CHARSET_INFO my_charset_bin; -extern MYSQL_PLUGIN_IMPORT CHARSET_INFO my_charset_latin1; -extern MYSQL_PLUGIN_IMPORT CHARSET_INFO my_charset_filename; -extern MYSQL_PLUGIN_IMPORT CHARSET_INFO* system_charset_info; -extern MYSQL_PLUGIN_IMPORT CHARSET_INFO* default_charset_info; -//extern MYSQL_PLUGIN_IMPORT CHARSET_INFO** all_charsets; -extern MYSQL_PLUGIN_IMPORT system_variables global_system_variables; -//extern MYSQL_PLUGIN_IMPORT char* mysql_real_data_home; -extern MYSQL_PLUGIN_IMPORT char* mysql_data_home; -//extern MYSQL_PLUGIN_IMPORT char** tx_isolation_names; -//extern MYSQL_PLUGIN_IMPORT char** binlog_format_names; -//extern MYSQL_PLUGIN_IMPORT char reg_ext; -extern MYSQL_PLUGIN_IMPORT pthread_mutex_t LOCK_thread_count; -extern MYSQL_PLUGIN_IMPORT key_map key_map_full; -extern MYSQL_PLUGIN_IMPORT MY_TMPDIR mysql_tmpdir_list; -extern MYSQL_PLUGIN_IMPORT bool mysqld_embedded; -extern MYSQL_PLUGIN_IMPORT uint lower_case_table_names; -extern MYSQL_PLUGIN_IMPORT ulong specialflag; -extern MYSQL_PLUGIN_IMPORT int my_umask; - -extern MYSQL_PLUGIN_IMPORT char *relay_log_info_file; - -/* -#define my_charset_bin (*wdl_my_charset_bin) -#define my_charset_latin1 (*wdl_my_charset_latin1) -#define my_charset_filename (*wdl_my_charset_filename) -#define system_charset_info (*wdl_system_charset_info) -#define default_charset_info (*wdl_default_charset_info) -#define all_charsets (wdl_all_charsets) -#define global_system_variables (*wdl_global_system_variables) -#define mysql_real_data_home (wdl_mysql_real_data_home) -#define mysql_data_home (*wdl_mysql_data_home) -#define tx_isolation_names (wdl_tx_isolation_names) -#define binlog_format_names (wdl_binlog_format_names) -#define reg_ext (wdl_reg_ext) -#define LOCK_thread_count (*wdl_LOCK_thread_count) -#define key_map_full (*wdl_key_map_full) -#define mysql_tmpdir_list (*wdl_mysql_tmpdir_list) -#define mysqld_embedded (*wdl_mysqld_embedded) -*/ -//#define lower_case_table_names (*wdl_lower_case_table_names) -//#define specialflag (*wdl_specialflag) -//#define my_umask (*wdl_my_umask) - -#endif === modified file 'storage/xtradb/handler/i_s.cc' --- a/storage/xtradb/handler/i_s.cc 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/handler/i_s.cc 2010-01-28 11:35:10 +0000 @@ -47,6 +47,7 @@ extern "C" { #include "trx0rseg.h" /* for trx_rseg_struct */ #include "trx0sys.h" /* for trx_sys */ #include "dict0dict.h" /* for dict_sys */ +#include "buf0lru.h" /* for XTRA_LRU_[DUMP/RESTORE] */ /* from buf0buf.c */ struct buf_chunk_struct{ ulint mem_size; /* allocated size of the chunk */ @@ -56,7 +57,6 @@ struct buf_chunk_struct{ buf_block_t* blocks; /* array of buffer control blocks */ }; } -#include "handler0vars.h" static const char plugin_author[] = "Innobase Oy"; @@ -84,14 +84,16 @@ do { \ #define STRUCT_FLD(name, value) value #endif -static const ST_FIELD_INFO END_OF_ST_FIELD_INFO = - {STRUCT_FLD(field_name, NULL), - STRUCT_FLD(field_length, 0), - STRUCT_FLD(field_type, MYSQL_TYPE_NULL), - STRUCT_FLD(value, 0), - STRUCT_FLD(field_flags, 0), - STRUCT_FLD(old_name, ""), - STRUCT_FLD(open_method, SKIP_OPEN_TABLE)}; +/* Don't use a static const variable here, as some C++ compilers (notably +HPUX aCC: HP ANSI C++ B3910B A.03.65) can't handle it. */ +#define END_OF_ST_FIELD_INFO \ + {STRUCT_FLD(field_name, NULL), \ + STRUCT_FLD(field_length, 0), \ + STRUCT_FLD(field_type, MYSQL_TYPE_NULL), \ + STRUCT_FLD(value, 0), \ + STRUCT_FLD(field_flags, 0), \ + STRUCT_FLD(old_name, ""), \ + STRUCT_FLD(open_method, SKIP_OPEN_TABLE)} /* Use the following types mapping: @@ -511,7 +513,7 @@ static ST_FIELD_INFO i_s_innodb_buffer_p STRUCT_FLD(old_name, ""), STRUCT_FLD(open_method, SKIP_OPEN_TABLE)}, - {STRUCT_FLD(field_name, "accessed"), + {STRUCT_FLD(field_name, "access_time"), STRUCT_FLD(field_length, MY_INT64_NUM_DECIMAL_DIGITS), STRUCT_FLD(field_type, MYSQL_TYPE_LONGLONG), STRUCT_FLD(value, 0), @@ -728,7 +730,7 @@ i_s_innodb_buffer_pool_pages_fill( field_store_string(table->field[0], page_type); table->field[1]->store(block->page.space); table->field[2]->store(block->page.offset); - table->field[3]->store(block->page.LRU_position); + table->field[3]->store(0); table->field[4]->store(block->page.buf_fix_count); table->field[5]->store(block->page.flush_type); @@ -817,11 +819,11 @@ i_s_innodb_buffer_pool_pages_index_fill( table->field[5]->store(page_get_n_recs(frame)); table->field[6]->store(page_get_data_size(frame)); table->field[7]->store(block->is_hashed); - table->field[8]->store(block->page.accessed); + table->field[8]->store(block->page.access_time); table->field[9]->store(block->page.newest_modification != 0); table->field[10]->store(block->page.oldest_modification != 0); table->field[11]->store(block->page.old); - table->field[12]->store(block->page.LRU_position); + table->field[12]->store(0); table->field[13]->store(block->page.buf_fix_count); table->field[14]->store(block->page.flush_type); @@ -915,7 +917,7 @@ i_s_innodb_buffer_pool_pages_blob_fill( table->field[4]->store(block->page.offset); } - table->field[5]->store(block->page.LRU_position); + table->field[5]->store(0); table->field[6]->store(block->page.buf_fix_count); table->field[7]->store(block->page.flush_type); @@ -2953,3 +2955,170 @@ UNIV_INTERN struct st_mysql_plugin i_s_i STRUCT_FLD(system_vars, NULL), STRUCT_FLD(__reserved1, NULL) }; + +/*********************************************************************** +*/ +static ST_FIELD_INFO i_s_innodb_admin_command_info[] = +{ + {STRUCT_FLD(field_name, "result_message"), + STRUCT_FLD(field_length, 1024), + STRUCT_FLD(field_type, MYSQL_TYPE_STRING), + STRUCT_FLD(value, 0), + STRUCT_FLD(field_flags, 0), + STRUCT_FLD(old_name, ""), + STRUCT_FLD(open_method, SKIP_OPEN_TABLE)}, + + END_OF_ST_FIELD_INFO +}; + +#ifndef INNODB_COMPATIBILITY_HOOKS +#error InnoDB needs MySQL to be built with #define INNODB_COMPATIBILITY_HOOKS +#endif + +extern "C" { +char **thd_query(MYSQL_THD thd); +} + +static +int +i_s_innodb_admin_command_fill( +/*==========================*/ + THD* thd, + TABLE_LIST* tables, + COND* cond) +{ + TABLE* i_s_table = (TABLE *) tables->table; + CHARSET_INFO *cs= system_charset_info; + char** query_str; + char* ptr; + char quote = '\0'; + const char* command_head = "XTRA_"; + + DBUG_ENTER("i_s_innodb_admin_command_fill"); + + /* deny access to non-superusers */ + if (check_global_access(thd, PROCESS_ACL)) { + DBUG_RETURN(0); + } + + if(thd_sql_command(thd) != SQLCOM_SELECT) { + field_store_string(i_s_table->field[0], + "SELECT command is only accepted."); + goto end_func; + } + + query_str = thd_query(thd); + ptr = *query_str; + + for (; *ptr; ptr++) { + if (*ptr == quote) { + quote = '\0'; + } else if (quote) { + } else if (*ptr == '`' || *ptr == '"') { + quote = *ptr; + } else { + long i; + for (i = 0; command_head[i]; i++) { + if (toupper((int)(unsigned char)(ptr[i])) + != toupper((int)(unsigned char) + (command_head[i]))) { + goto nomatch; + } + } + break; +nomatch: + ; + } + } + + if (!*ptr) { + field_store_string(i_s_table->field[0], + "No XTRA_* command in the SQL statement." + " Please add /*!XTRA_xxxx*/ to the SQL."); + goto end_func; + } + + if (!strncasecmp("XTRA_HELLO", ptr, 10)) { + /* This is example command XTRA_HELLO */ + + ut_print_timestamp(stderr); + fprintf(stderr, " InnoDB: administration command test for XtraDB" + " 'XTRA_HELLO' was detected.\n"); + + field_store_string(i_s_table->field[0], + "Hello!"); + goto end_func; + } + else if (!strncasecmp("XTRA_LRU_DUMP", ptr, 13)) { + ut_print_timestamp(stderr); + fprintf(stderr, " InnoDB: administration command 'XTRA_LRU_DUMP'" + " was detected.\n"); + + if (buf_LRU_file_dump()) { + field_store_string(i_s_table->field[0], + "XTRA_LRU_DUMP was succeeded."); + } else { + field_store_string(i_s_table->field[0], + "XTRA_LRU_DUMP was failed."); + } + + goto end_func; + } + else if (!strncasecmp("XTRA_LRU_RESTORE", ptr, 16)) { + ut_print_timestamp(stderr); + fprintf(stderr, " InnoDB: administration command 'XTRA_LRU_RESTORE'" + " was detected.\n"); + + if (buf_LRU_file_restore()) { + field_store_string(i_s_table->field[0], + "XTRA_LRU_RESTORE was succeeded."); + } else { + field_store_string(i_s_table->field[0], + "XTRA_LRU_RESTORE was failed."); + } + + goto end_func; + } + + field_store_string(i_s_table->field[0], + "Undefined XTRA_* command."); + goto end_func; + +end_func: + if (schema_table_store_record(thd, i_s_table)) { + DBUG_RETURN(1); + } else { + DBUG_RETURN(0); + } +} + +static +int +i_s_innodb_admin_command_init( +/*==========================*/ + void* p) +{ + DBUG_ENTER("i_s_innodb_admin_command_init"); + ST_SCHEMA_TABLE* schema = (ST_SCHEMA_TABLE*) p; + + schema->fields_info = i_s_innodb_admin_command_info; + schema->fill_table = i_s_innodb_admin_command_fill; + + DBUG_RETURN(0); +} + +UNIV_INTERN struct st_mysql_plugin i_s_innodb_admin_command = +{ + STRUCT_FLD(type, MYSQL_INFORMATION_SCHEMA_PLUGIN), + STRUCT_FLD(info, &i_s_info), + STRUCT_FLD(name, "XTRADB_ADMIN_COMMAND"), + STRUCT_FLD(author, plugin_author), + STRUCT_FLD(descr, "XtraDB specific command acceptor"), + STRUCT_FLD(license, PLUGIN_LICENSE_GPL), + STRUCT_FLD(init, i_s_innodb_admin_command_init), + STRUCT_FLD(deinit, i_s_common_deinit), + STRUCT_FLD(version, 0x0100 /* 1.0 */), + STRUCT_FLD(status_vars, NULL), + STRUCT_FLD(system_vars, NULL), + STRUCT_FLD(__reserved1, NULL) +}; === modified file 'storage/xtradb/handler/i_s.h' --- a/storage/xtradb/handler/i_s.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/handler/i_s.h 2009-11-04 20:11:12 +0000 @@ -40,5 +40,6 @@ extern struct st_mysql_plugin i_s_innodb extern struct st_mysql_plugin i_s_innodb_rseg; extern struct st_mysql_plugin i_s_innodb_table_stats; extern struct st_mysql_plugin i_s_innodb_index_stats; +extern struct st_mysql_plugin i_s_innodb_admin_command; #endif /* i_s_h */ === modified file 'storage/xtradb/handler/innodb_patch_info.h' --- a/storage/xtradb/handler/innodb_patch_info.h 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/handler/innodb_patch_info.h 2010-01-15 15:58:25 +0000 @@ -38,5 +38,10 @@ struct innodb_enhancement { {"innodb_stats","Additional features about InnoDB statistics/optimizer","","http://www.percona.com/docs/wiki/percona-xtradb"}, {"innodb_recovery_patches","Bugfixes and adjustments about recovery process","","http://www.percona.com/docs/wiki/percona-xtradb"}, {"innodb_purge_thread","Enable to use purge devoted thread","","http://www.percona.com/docs/wiki/percona-xtradb"}, +{"innodb_admin_command_base","XtraDB specific command interface through i_s","","http://www.percona.com/docs/wiki/percona-xtradb"}, +{"innodb_show_lock_name","Show mutex/lock name instead of crated file/line","","http://www.percona.com/docs/wiki/percona-xtradb"}, +{"innodb_extend_slow","Extended statistics in slow.log","It is InnoDB-part only. It needs to patch also to mysqld.","http://www.percona.com/docs/wiki/percona-xtradb"}, +{"innodb_relax_table_creation","Relax limitation of column size at table creation as builtin InnoDB.","","http://www.percona.com/docs/wiki/percona-xtradb"}, +{"innodb_lru_dump_restore","Dump and restore command for content of buffer pool","","http://www.percona.com/docs/wiki/percona-xtradb"}, {NULL, NULL, NULL, NULL} }; === removed file 'storage/xtradb/handler/win_delay_loader.cc' --- a/storage/xtradb/handler/win_delay_loader.cc 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/handler/win_delay_loader.cc 1970-01-01 00:00:00 +0000 @@ -1,1024 +0,0 @@ -/***************************************************************************** - -Copyright (c) 2008, 2009, Innobase Oy. All Rights Reserved. - -This program is free software; you can redistribute it and/or modify it under -the terms of the GNU General Public License as published by the Free Software -Foundation; version 2 of the License. - -This program is distributed in the hope that it will be useful, but WITHOUT -ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS -FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - -You should have received a copy of the GNU General Public License along with -this program; if not, write to the Free Software Foundation, Inc., 59 Temple -Place, Suite 330, Boston, MA 02111-1307 USA - -*****************************************************************************/ - -/*******************************************************************//** -@file handler/win_delay_loader.cc -This file contains functions that implement the delay loader on Windows. - -This is a customized version of delay loader with limited functionalities. -It does not support: - -* (manual) unloading -* multiple delay loaded DLLs -* multiple loading of the same DLL - -This delay loader is used only by the InnoDB plugin. Other components (DLLs) -can still use the default delay loader, provided by MSVC. - -Several acronyms used by Microsoft: - * IAT: import address table - * INT: import name table - * RVA: Relative Virtual Address - -See http://msdn.microsoft.com/en-us/magazine/bb985992.aspx for details of -PE format. -***********************************************************************/ -#if defined (__WIN__) && defined (MYSQL_DYNAMIC_PLUGIN) -# define WIN32_LEAN_AND_MEAN -# include <windows.h> -# include <delayimp.h> -# include <mysql_priv.h> - -extern "C" { -# include "univ.i" -# include "hash0hash.h" -} - -/*******************************************************************//** -This following contains a list of externals that can not be resolved by -delay loading. They have to be resolved indirectly via their addresses -in the .map file. All of them are external variables. */ -CHARSET_INFO* wdl_my_charset_bin; -CHARSET_INFO* wdl_my_charset_latin1; -CHARSET_INFO* wdl_my_charset_filename; -CHARSET_INFO** wdl_system_charset_info; -CHARSET_INFO** wdl_default_charset_info; -CHARSET_INFO** wdl_all_charsets; -system_variables* wdl_global_system_variables; -char* wdl_mysql_real_data_home; -char** wdl_mysql_data_home; -char** wdl_tx_isolation_names; -char** wdl_binlog_format_names; -char* wdl_reg_ext; -pthread_mutex_t* wdl_LOCK_thread_count; -key_map* wdl_key_map_full; -MY_TMPDIR* wdl_mysql_tmpdir_list; -bool* wdl_mysqld_embedded; -uint* wdl_lower_case_table_names; -ulong* wdl_specialflag; -int* wdl_my_umask; - -/*******************************************************************//** -The preferred load-address defined in PE (portable executable format). */ -#if defined(_M_IA64) -#pragma section(".base", long, read) -extern "C" -__declspec(allocate(".base")) -const IMAGE_DOS_HEADER __ImageBase; -#else -extern "C" -const IMAGE_DOS_HEADER __ImageBase; -#endif - -/*******************************************************************//** -A template function for converting a relative address (RVA) to an -absolute address (VA). This is due to the pointers in the delay -descriptor (ImgDelayDescr in delayimp.h) have been changed from -VAs to RVAs to work on both 32- and 64-bit platforms. -@return absolute virtual address */ -template <class X> -X PFromRva( -/*=======*/ - RVA rva) /*!< in: relative virtual address */ -{ - return X(PBYTE(&__ImageBase) + rva); -} - -/*******************************************************************//** -Convert to the old format for convenience. The structure as well as its -element names follow the definition of ImgDelayDescr in delayimp.h. */ -struct InternalImgDelayDescr -{ - DWORD grAttrs; /*!< attributes */ - LPCSTR szName; /*!< pointer to dll name */ - HMODULE* phmod; /*!< address of module handle */ - PImgThunkData pIAT; /*!< address of the IAT */ - PCImgThunkData pINT; /*!< address of the INT */ - PCImgThunkData pBoundIAT; /*!< address of the optional bound IAT */ - PCImgThunkData pUnloadIAT; /*!< address of optional copy of - original IAT */ - DWORD dwTimeStamp; /*!< 0 if not bound, - otherwise date/time stamp of DLL - bound to (Old BIND) */ -}; - -typedef struct map_hash_chain_struct map_hash_chain_t; - -struct map_hash_chain_struct { - char* symbol; /*!< pointer to a symbol */ - ulint value; /*!< address of the symbol */ - map_hash_chain_t* next; /*!< pointer to the next cell - in the same folder. */ - map_hash_chain_t* chain; /*!< a linear chain used for - cleanup. */ -}; - -static HMODULE my_hmod = 0; -static struct hash_table_struct* m_htbl = NULL ; -static map_hash_chain_t* chain_header = NULL; -static ibool wdl_init = FALSE; -const ulint MAP_HASH_CELLS_NUM = 10000; - -#ifndef DBUG_OFF -/*******************************************************************//** -In the dynamic plugin, it is required to call the following dbug functions -in the server: - _db_pargs_ - _db_doprnt_ - _db_enter_ - _db_return_ - _db_dump_ - -The plugin will get those function pointers during the initialization. */ -typedef void (__cdecl* pfn_db_enter_)( - const char* _func_, - const char* _file_, - uint _line_, - const char** _sfunc_, - const char** _sfile_, - uint* _slevel_, - char***); - -typedef void (__cdecl* pfn_db_return_)( - uint _line_, - const char** _sfunc_, - const char** _sfile_, - uint* _slevel_); - -typedef void (__cdecl* pfn_db_pargs_)( - uint _line_, - const char* keyword); - -typedef void (__cdecl* pfn_db_doprnt_)( - const char* format, - ...); - -typedef void (__cdecl* pfn_db_dump_)( - uint _line_, - const char* keyword, - const unsigned char* memory, - size_t length); - -static pfn_db_enter_ wdl_db_enter_; -static pfn_db_return_ wdl_db_return_; -static pfn_db_pargs_ wdl_db_pargs_; -static pfn_db_doprnt_ wdl_db_doprnt_; -static pfn_db_dump_ wdl_db_dump_; -#endif /* !DBUG_OFF */ - -/*************************************************************//** -Creates a hash table with >= n array cells. The actual number of cells is -chosen to be a prime number slightly bigger than n. - -This is the same function as hash_create in hash0hash.c, except the -memory allocation. This function is invoked before the engine is -initialized, and buffer pools are not ready yet. -@return own: created hash table */ -static -hash_table_t* -wdl_hash_create( -/*============*/ - ulint n) /*!< in: number of array cells */ -{ - hash_cell_t* array; - ulint prime; - hash_table_t* table; - - prime = ut_find_prime(n); - - table = (hash_table_t*) malloc(sizeof(hash_table_t)); - if (table == NULL) { - return(NULL); - } - - array = (hash_cell_t*) malloc(sizeof(hash_cell_t) * prime); - if (array == NULL) { - free(table); - return(NULL); - } - - table->array = array; - table->n_cells = prime; - table->n_mutexes = 0; - table->mutexes = NULL; - table->heaps = NULL; - table->heap = NULL; - table->magic_n = HASH_TABLE_MAGIC_N; - - /* Initialize the cell array */ - hash_table_clear(table); - - return(table); -} - -/*************************************************************//** -Frees a hash table. */ -static -void -wdl_hash_table_free( -/*================*/ - hash_table_t* table) /*!< in, own: hash table */ -{ - ut_a(table != NULL); - ut_a(table->mutexes == NULL); - - free(table->array); - free(table); -} - -/*******************************************************************//** -Function for calculating the count of imports given the base of the IAT. -@return number of imports */ -static -ulint -wdl_import_count( -/*=============*/ - PCImgThunkData pitd_base) /*!< in: base of the IAT */ -{ - ulint ret = 0; - PCImgThunkData pitd = pitd_base; - - while (pitd->u1.Function) { - pitd++; - ret++; - } - - return(ret); -} - -/*******************************************************************//** -Read Mapfile to a hashtable for faster access -@return TRUE if the mapfile is loaded successfully. */ -static -ibool -wdl_load_mapfile( -/*=============*/ - const char* filename) /*!< in: name of the mapfile. */ -{ - FILE* fp; - const size_t nSize = 256; - char tmp_buf[nSize]; - char* func_name; - char* func_addr; - ulint load_addr = 0; - ibool valid_load_addr = FALSE; -#ifdef _WIN64 - const char* tmp_string = " Preferred load address is %16llx"; -#else - const char* tmp_string = " Preferred load address is %08x"; -#endif - - fp = fopen(filename, "r"); - if (fp == NULL) { - - return(FALSE); - } - - /* Check whether to create the hashtable */ - if (m_htbl == NULL) { - - m_htbl = wdl_hash_create(MAP_HASH_CELLS_NUM); - - if (m_htbl == NULL) { - - fclose(fp); - return(FALSE); - } - } - - /* Search start of symbol list and get the preferred load address */ - while (fgets(tmp_buf, sizeof(tmp_buf), fp)) { - - if (sscanf(tmp_buf, tmp_string, &load_addr) == 1) { - - valid_load_addr = TRUE; - } - - if (strstr(tmp_buf, "Rva+Base") != NULL) { - - break; - } - } - - if (valid_load_addr == FALSE) { - - /* No "Preferred load address", the map file is wrong. */ - fclose(fp); - return(FALSE); - } - - /* Read symbol list */ - while (fgets(tmp_buf, sizeof(tmp_buf), fp)) - { - map_hash_chain_t* map_cell; - ulint map_fold; - - if (*tmp_buf == 0) { - - continue; - } - - func_name = strtok(tmp_buf, " "); - func_name = strtok(NULL, " "); - func_addr = strtok(NULL, " "); - - if (func_name && func_addr) { - - ut_snprintf(tmp_buf, nSize, "0x%s", func_addr); - if (*func_name == '_') { - - func_name++; - } - - map_cell = (map_hash_chain_t*) - malloc(sizeof(map_hash_chain_t)); - if (map_cell == NULL) { - return(FALSE); - } - - /* Chain all cells together */ - map_cell->chain = chain_header; - chain_header = map_cell; - - map_cell->symbol = strdup(func_name); - map_cell->value = (ulint) _strtoui64(tmp_buf, NULL, 0) - - load_addr; - map_fold = ut_fold_string(map_cell->symbol); - - HASH_INSERT(map_hash_chain_t, - next, - m_htbl, - map_fold, - map_cell); - } - } - - fclose(fp); - - return(TRUE); -} - -/*************************************************************//** -Cleanup.during DLL unload */ -static -void -wdl_cleanup(void) -/*=============*/ -{ - while (chain_header != NULL) { - map_hash_chain_t* tmp; - - tmp = chain_header->chain; - free(chain_header->symbol); - free(chain_header); - chain_header = tmp; - } - - if (m_htbl != NULL) { - - wdl_hash_table_free(m_htbl); - } -} - -/*******************************************************************//** -Load the mapfile mysqld.map. -@return the module handle */ -static -HMODULE -wdl_get_mysqld_mapfile(void) -/*========================*/ -{ - char file_name[MAX_PATH]; - char* ext; - ulint err; - - if (my_hmod == 0) { - - size_t nSize = MAX_PATH - strlen(".map") -1; - - /* First find out the name of current executable */ - my_hmod = GetModuleHandle(NULL); - if (my_hmod == 0) { - - return(my_hmod); - } - - err = GetModuleFileName(my_hmod, file_name, nSize); - if (err == 0) { - - my_hmod = 0; - return(my_hmod); - } - - ext = strrchr(file_name, '.'); - if (ext != NULL) { - - *ext = 0; - strcat(file_name, ".map"); - - err = wdl_load_mapfile(file_name); - if (err == 0) { - - my_hmod = 0; - } - } else { - - my_hmod = 0; - } - } - - return(my_hmod); -} - -/*******************************************************************//** -Retrieves the address of an exported function. It follows the convention -of GetProcAddress(). -@return address of exported function. */ -static -FARPROC -wdl_get_procaddr_from_map( -/*======================*/ - HANDLE m_handle, /*!< in: module handle */ - const char* import_proc) /*!< in: procedure name */ -{ - map_hash_chain_t* hash_chain; - ulint map_fold; - - map_fold = ut_fold_string(import_proc); - HASH_SEARCH( - next, - m_htbl, - map_fold, - map_hash_chain_t*, - hash_chain, - , - (ut_strcmp(hash_chain->symbol, import_proc) == 0)); - - if (hash_chain == NULL) { - -#ifdef _WIN64 - /* On Win64, the leading '_' may not be taken out. In this - case, search again without the leading '_'. */ - if (*import_proc == '_') { - - import_proc++; - } - - map_fold = ut_fold_string(import_proc); - HASH_SEARCH( - next, - m_htbl, - map_fold, - map_hash_chain_t*, - hash_chain, - , - (ut_strcmp(hash_chain->symbol, import_proc) == 0)); - - if (hash_chain == NULL) { -#endif - if (wdl_init == TRUE) { - - sql_print_error( - "InnoDB: the procedure pointer of %s" - " is not found.", - import_proc); - } - - return(0); -#ifdef _WIN64 - } -#endif - } - - return((FARPROC) ((ulint) m_handle + hash_chain->value)); -} - -/*******************************************************************//** -Retrieves the address of an exported variable. -Note: It does not follow the Windows call convention FARPROC. -@return address of exported variable. */ -static -void* -wdl_get_varaddr_from_map( -/*=====================*/ - HANDLE m_handle, /*!< in: module handle */ - const char* import_variable) /*!< in: variable name */ -{ - map_hash_chain_t* hash_chain; - ulint map_fold; - - map_fold = ut_fold_string(import_variable); - HASH_SEARCH( - next, - m_htbl, - map_fold, - map_hash_chain_t*, - hash_chain, - , - (ut_strcmp(hash_chain->symbol, import_variable) == 0)); - - if (hash_chain == NULL) { - -#ifdef _WIN64 - /* On Win64, the leading '_' may not be taken out. In this - case, search again without the leading '_'. */ - if (*import_variable == '_') { - - import_variable++; - } - - map_fold = ut_fold_string(import_variable); - HASH_SEARCH( - next, - m_htbl, - map_fold, - map_hash_chain_t*, - hash_chain, - , - (ut_strcmp(hash_chain->symbol, import_variable) == 0)); - - if (hash_chain == NULL) { -#endif - if (wdl_init == TRUE) { - - sql_print_error( - "InnoDB: the variable address of %s" - " is not found.", - import_variable); - } - - return(0); -#ifdef _WIN64 - } -#endif - } - - return((void*) ((ulint) m_handle + hash_chain->value)); -} - -/*******************************************************************//** -Bind all unresolved external variables from the MySQL executable. -@return TRUE if successful */ -static -bool -wdl_get_external_variables(void) -/*============================*/ -{ - HMODULE hmod = wdl_get_mysqld_mapfile(); - - if (hmod == 0) { - - return(FALSE); - } - -#define GET_SYM(sym, var, type) \ - var = (type*) wdl_get_varaddr_from_map(hmod, sym); \ - if (var == NULL) return(FALSE) -#ifdef _WIN64 -#define GET_SYM2(sym1, sym2, var, type) \ - var = (type*) wdl_get_varaddr_from_map(hmod, sym1); \ - if (var == NULL) return(FALSE) -#else -#define GET_SYM2(sym1, sym2, var, type) \ - var = (type*) wdl_get_varaddr_from_map(hmod, sym2); \ - if (var == NULL) return(FALSE) -#endif // (_WIN64) -#define GET_C_SYM(sym, type) GET_SYM(#sym, wdl_##sym, type) -#define GET_PROC_ADDR(sym) \ - wdl##sym = (pfn##sym) wdl_get_procaddr_from_map(hmod, #sym) - - GET_C_SYM(my_charset_bin, CHARSET_INFO); - GET_C_SYM(my_charset_latin1, CHARSET_INFO); - GET_C_SYM(my_charset_filename, CHARSET_INFO); - GET_C_SYM(default_charset_info, CHARSET_INFO*); - GET_C_SYM(all_charsets, CHARSET_INFO*); - GET_C_SYM(my_umask, int); - - GET_SYM("?global_system_variables@@3Usystem_variables@@A", - wdl_global_system_variables, struct system_variables); - GET_SYM("?mysql_real_data_home@@3PADA", - wdl_mysql_real_data_home, char); - GET_SYM("?reg_ext@@3PADA", wdl_reg_ext, char); - GET_SYM("?LOCK_thread_count@@3U_RTL_CRITICAL_SECTION@@A", - wdl_LOCK_thread_count, pthread_mutex_t); - GET_SYM("?key_map_full@@3V?$Bitmap@$0EA@@@A", - wdl_key_map_full, key_map); - GET_SYM("?mysql_tmpdir_list@@3Ust_my_tmpdir@@A", - wdl_mysql_tmpdir_list, MY_TMPDIR); - GET_SYM("?mysqld_embedded@@3_NA", - wdl_mysqld_embedded, bool); - GET_SYM("?lower_case_table_names@@3IA", - wdl_lower_case_table_names, uint); - GET_SYM("?specialflag@@3KA", wdl_specialflag, ulong); - - GET_SYM2("?system_charset_info@@3PEAUcharset_info_st@@EA", - "?system_charset_info@@3PAUcharset_info_st@@A", - wdl_system_charset_info, CHARSET_INFO*); - GET_SYM2("?mysql_data_home@@3PEADEA", - "?mysql_data_home@@3PADA", - wdl_mysql_data_home, char*); - GET_SYM2("?tx_isolation_names@@3PAPEBDA", - "?tx_isolation_names@@3PAPBDA", - wdl_tx_isolation_names, char*); - GET_SYM2("?binlog_format_names@@3PAPEBDA", - "?binlog_format_names@@3PAPBDA", - wdl_binlog_format_names, char*); - -#ifndef DBUG_OFF - GET_PROC_ADDR(_db_enter_); - GET_PROC_ADDR(_db_return_); - GET_PROC_ADDR(_db_pargs_); - GET_PROC_ADDR(_db_doprnt_); - GET_PROC_ADDR(_db_dump_); - - /* If any of the dbug functions is not available, just make them - all invalid. This is the case when working with a non-debug - version of the server. */ - if (wdl_db_enter_ == NULL || wdl_db_return_ == NULL - || wdl_db_pargs_ == NULL || wdl_db_doprnt_ == NULL - || wdl_db_dump_ == NULL) { - - wdl_db_enter_ = NULL; - wdl_db_return_ = NULL; - wdl_db_pargs_ = NULL; - wdl_db_doprnt_ = NULL; - wdl_db_dump_ = NULL; - } -#endif /* !DBUG_OFF */ - - wdl_init = TRUE; - return(TRUE); - -#undef GET_SYM -#undef GET_SYM2 -#undef GET_C_SYM -#undef GET_PROC_ADDR -} - -/*******************************************************************//** -The DLL Delayed Loading Helper Function for resolving externals. - -The function may fail due to one of the three reasons: - -* Invalid parameter, which happens if the attributes in pidd aren't - specified correctly. -* Failed to load the map file mysqld.map. -* Failed to find an external name in the map file mysqld.map. - -Note: this function is called by run-time as well as __HrLoadAllImportsForDll. -So, it has to follow Windows call convention. -@return the address of the imported function */ -extern "C" -FARPROC WINAPI -__delayLoadHelper2( -/*===============*/ - PCImgDelayDescr pidd, /*!< in: a const pointer to a - ImgDelayDescr, see delayimp.h. */ - FARPROC* iat_entry) /*!< in/out: A pointer to the slot in - the delay load import address table - to be updated with the address of the - imported function. */ -{ - ulint iIAT, iINT; - HMODULE hmod; - PCImgThunkData pitd; - FARPROC fun = NULL; - - /* Set up data used for the hook procs */ - InternalImgDelayDescr idd = { - pidd->grAttrs, - PFromRva<LPCSTR>(pidd->rvaDLLName), - PFromRva<HMODULE*>(pidd->rvaHmod), - PFromRva<PImgThunkData>(pidd->rvaIAT), - PFromRva<PCImgThunkData>(pidd->rvaINT), - PFromRva<PCImgThunkData>(pidd->rvaBoundIAT), - PFromRva<PCImgThunkData>(pidd->rvaUnloadIAT), - pidd->dwTimeStamp - }; - - DelayLoadInfo dli = { - sizeof(DelayLoadInfo), - pidd, - iat_entry, - idd.szName, - {0}, - 0, - 0, - 0 - }; - - /* Check the Delay Load Attributes, log an error of invalid - parameter, which happens if the attributes in pidd are not - specified correctly. */ - if ((idd.grAttrs & dlattrRva) == 0) { - - sql_print_error("InnoDB: invalid parameter for delay loader."); - return(0); - } - - hmod = *idd.phmod; - - /* Calculate the index for the IAT entry in the import address table. - The INT entries are ordered the same as the IAT entries so the - calculation can be done on the IAT side. */ - iIAT = (PCImgThunkData) iat_entry - idd.pIAT; - iINT = iIAT; - - pitd = &(idd.pINT[iINT]); - - dli.dlp.fImportByName = !IMAGE_SNAP_BY_ORDINAL(pitd->u1.Ordinal); - - if (dli.dlp.fImportByName) { - - dli.dlp.szProcName = (LPCSTR) (PFromRva<PIMAGE_IMPORT_BY_NAME> - ((RVA) ((UINT_PTR) pitd->u1.AddressOfData))->Name); - } else { - - dli.dlp.dwOrdinal = (ulint) IMAGE_ORDINAL(pitd->u1.Ordinal); - } - - /* Now, load the mapfile, if it has not been done yet */ - if (hmod == 0) { - - hmod = wdl_get_mysqld_mapfile(); - } - - if (hmod == 0) { - /* LoadLibrary failed. */ - PDelayLoadInfo rgpdli[1] = {&dli}; - - dli.dwLastError = ::GetLastError(); - - sql_print_error( - "InnoDB: failed to load mysqld.map with error %d.", - dli.dwLastError); - - return(0); - } - - /* Store the library handle. */ - idd.phmod = &hmod; - - /* Go for the procedure now. */ - dli.hmodCur = hmod; - - if (pidd->rvaBoundIAT && pidd->dwTimeStamp) { - - /* Bound imports exist, check the timestamp from the target - image */ - PIMAGE_NT_HEADERS pinh; - - pinh = (PIMAGE_NT_HEADERS) ((byte*) hmod - + ((PIMAGE_DOS_HEADER) hmod)->e_lfanew); - - if (pinh->Signature == IMAGE_NT_SIGNATURE - && pinh->FileHeader.TimeDateStamp == idd.dwTimeStamp - && (DWORD) hmod == pinh->OptionalHeader.ImageBase) { - - /* We have a decent address in the bound IAT. */ - fun = (FARPROC) (UINT_PTR) - idd.pBoundIAT[iIAT].u1.Function; - - if (fun) { - - *iat_entry = fun; - return(fun); - } - } - } - - fun = wdl_get_procaddr_from_map(hmod, dli.dlp.szProcName); - - if (fun == 0) { - - return(0); - } - - *iat_entry = fun; - return(fun); -} - -/*******************************************************************//** -Unload a DLL that was delay loaded. This function is called by run-time. -@return TRUE is returned if the DLL is found and the IAT matches the -original one. */ -extern "C" -BOOL WINAPI -__FUnloadDelayLoadedDLL2( -/*=====================*/ - LPCSTR module_name) /*!< in: DLL name */ -{ - return(TRUE); -} - -/**************************************************************//** -Load all imports from a DLL that was specified with the /delayload linker -option. -Note: this function is called by run-time. So, it has to follow Windows call -convention. -@return S_OK if the DLL matches, otherwise ERROR_MOD_NOT_FOUND is returned. */ -extern "C" -HRESULT WINAPI -__HrLoadAllImportsForDll( -/*=====================*/ - LPCSTR module_name) /*!< in: DLL name */ -{ - PIMAGE_NT_HEADERS img; - PCImgDelayDescr pidd; - IMAGE_DATA_DIRECTORY* image_data; - LPCSTR current_module; - HRESULT ret = ERROR_MOD_NOT_FOUND; - HMODULE hmod = (HMODULE) &__ImageBase; - - img = (PIMAGE_NT_HEADERS) ((byte*) hmod - + ((PIMAGE_DOS_HEADER) hmod)->e_lfanew); - image_data = - &img->OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_DELAY_IMPORT]; - - /* Scan the delay load IAT/INT for the DLL */ - if (image_data->Size) { - - pidd = PFromRva<PCImgDelayDescr>(image_data->VirtualAddress); - - /* Check all of the listed DLLs we want to load. */ - while (pidd->rvaDLLName) { - - current_module = PFromRva<LPCSTR>(pidd->rvaDLLName); - - if (stricmp(module_name, current_module) == 0) { - - /* Found it, break out with pidd and - current_module set appropriately */ - break; - } - - /* To the next delay import descriptor */ - pidd++; - } - - if (pidd->rvaDLLName) { - - /* Found a matching DLL, now process it. */ - FARPROC* iat_entry; - size_t count; - - iat_entry = PFromRva<FARPROC*>(pidd->rvaIAT); - count = wdl_import_count((PCImgThunkData) iat_entry); - - /* now load all the imports from the DLL */ - while (count > 0) { - - /* No need to check the return value */ - __delayLoadHelper2(pidd, iat_entry); - iat_entry++; - count--; - } - - ret = S_OK; - } - } - - return ret; -} - -/**************************************************************//** -The main function of a DLL -@return TRUE if the call succeeds */ -BOOL -WINAPI -DllMain( -/*====*/ - HINSTANCE hinstDLL, /*!< in: handle to the DLL module */ - DWORD fdwReason, /*!< Reason code that indicates why the - DLL entry-point function is being - called.*/ - LPVOID lpvReserved) /*!< in: additional parameter based on - fdwReason */ -{ - BOOL success = TRUE; - - switch (fdwReason) { - - case DLL_PROCESS_ATTACH: - success = wdl_get_external_variables(); - break; - - case DLL_PROCESS_DETACH: - wdl_cleanup(); - break; - } - - return(success); -} - -#ifndef DBUG_OFF -/**************************************************************//** -Process entry point to user function. It makes the call to _db_enter_ -in mysqld.exe. The DBUG functions are defined in my_dbug.h. */ -extern "C" UNIV_INTERN -void -_db_enter_( - const char* _func_, /*!< in: current function name */ - const char* _file_, /*!< in: current file name */ - uint _line_, /*!< in: current source line number */ - const char** _sfunc_, /*!< out: previous _func_ */ - const char** _sfile_, /*!< out: previous _file_ */ - uint* _slevel_, /*!< out: previous nesting level */ - char*** _sframep_) /*!< out: previous frame pointer */ -{ - if (wdl_db_enter_ != NULL) { - - wdl_db_enter_(_func_, _file_, _line_, _sfunc_, _sfile_, - _slevel_, _sframep_); - } -} - -/**************************************************************//** -Process exit from user function. It makes the call to _db_return_() -in the server. */ -extern "C" UNIV_INTERN -void -_db_return_( - uint _line_, /*!< in: current source line number */ - const char** _sfunc_, /*!< out: previous _func_ */ - const char** _sfile_, /*!< out: previous _file_ */ - uint* _slevel_) /*!< out: previous level */ -{ - if (wdl_db_return_ != NULL) { - - wdl_db_return_(_line_, _sfunc_, _sfile_, _slevel_); - } -} - -/**************************************************************//** -Log arguments for subsequent use. It makes the call to _db_pargs_() -in the server. */ -extern "C" UNIV_INTERN -void -_db_pargs_( - uint _line_, /*!< in: current source line number */ - const char* keyword) /*!< in: keyword for current macro */ -{ - if (wdl_db_pargs_ != NULL) { - - wdl_db_pargs_(_line_, keyword); - } -} - -/**************************************************************//** -Handle print of debug lines. It saves the text into a buffer first, -then makes the call to _db_doprnt_() in the server. The text is -truncated to the size of buffer. */ -extern "C" UNIV_INTERN -void -_db_doprnt_( - const char* format, /*!< in: the format string */ - ...) /*!< in: list of arguments */ -{ - va_list argp; - char buffer[512]; - - if (wdl_db_doprnt_ != NULL) { - - va_start(argp, format); - /* it is ok to ignore the trunction. */ - _vsnprintf(buffer, sizeof(buffer), format, argp); - wdl_db_doprnt_(buffer); - va_end(argp); - } -} - -/**************************************************************//** -Dump a string in hex. It makes the call to _db_dump_() in the server. */ -extern "C" UNIV_INTERN -void -_db_dump_( - uint _line_, /*!< in: current source line - number */ - const char* keyword, /*!< in: keyword list */ - const unsigned char* memory, /*!< in: memory to dump */ - size_t length) /*!< in: bytes to dump */ -{ - if (wdl_db_dump_ != NULL) { - - wdl_db_dump_(_line_, keyword, memory, length); - } -} - -#endif /* !DBUG_OFF */ -#endif /* defined (__WIN__) && defined (MYSQL_DYNAMIC_PLUGIN) */ === modified file 'storage/xtradb/ibuf/ibuf0ibuf.c' --- a/storage/xtradb/ibuf/ibuf0ibuf.c 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/ibuf/ibuf0ibuf.c 2010-01-15 15:58:25 +0000 @@ -390,6 +390,27 @@ ibuf_count_set( #endif /******************************************************************//** +Closes insert buffer and frees the data structures. */ +UNIV_INTERN +void +ibuf_close(void) +/*============*/ +{ + mutex_free(&ibuf_pessimistic_insert_mutex); + memset(&ibuf_pessimistic_insert_mutex, + 0x0, sizeof(ibuf_pessimistic_insert_mutex)); + + mutex_free(&ibuf_mutex); + memset(&ibuf_mutex, 0x0, sizeof(ibuf_mutex)); + + mutex_free(&ibuf_bitmap_mutex); + memset(&ibuf_bitmap_mutex, 0x0, sizeof(ibuf_mutex)); + + mem_free(ibuf); + ibuf = NULL; +} + +/******************************************************************//** Updates the size information of the ibuf, assuming the segment size has not changed. */ static === modified file 'storage/xtradb/include/btr0cur.h' --- a/storage/xtradb/include/btr0cur.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/btr0cur.h 2010-01-06 12:00:14 +0000 @@ -618,7 +618,7 @@ enum btr_cur_method { hash_node, and might be necessary to update */ BTR_CUR_BINARY, /*!< success using the binary search */ - BTR_CUR_INSERT_TO_IBUF, /*!< performed the intended insert to + BTR_CUR_INSERT_TO_IBUF /*!< performed the intended insert to the insert buffer */ }; === modified file 'storage/xtradb/include/btr0sea.h' --- a/storage/xtradb/include/btr0sea.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/btr0sea.h 2010-01-06 12:00:14 +0000 @@ -41,6 +41,12 @@ void btr_search_sys_create( /*==================*/ ulint hash_size); /*!< in: hash index hash table size */ +/*****************************************************************//** +Frees the adaptive search system at a database shutdown. */ +UNIV_INTERN +void +btr_search_sys_free(void); +/*=====================*/ /********************************************************************//** Disable the adaptive hash search system and empty the index. */ === modified file 'storage/xtradb/include/buf0buf.h' --- a/storage/xtradb/include/buf0buf.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/buf0buf.h 2010-01-06 12:00:14 +0000 @@ -346,7 +346,7 @@ buf_page_release( mtr_t* mtr); /*!< in: mtr */ /********************************************************************//** Moves a page to the start of the buffer pool LRU list. This high-level -function can be used to prevent an important page from from slipping out of +function can be used to prevent an important page from slipping out of the buffer pool. */ UNIV_INTERN void @@ -707,15 +707,6 @@ buf_page_belongs_to_unzip_LRU( /*==========================*/ const buf_page_t* bpage) /*!< in: pointer to control block */ __attribute__((pure)); -/*********************************************************************//** -Determine the approximate LRU list position of a block. -@return LRU list position */ -UNIV_INLINE -ulint -buf_page_get_LRU_position( -/*======================*/ - const buf_page_t* bpage) /*!< in: control block */ - __attribute__((pure)); /*********************************************************************//** Gets the mutex of a block. @@ -825,14 +816,14 @@ buf_page_set_old( buf_page_t* bpage, /*!< in/out: control block */ ibool old); /*!< in: old */ /*********************************************************************//** -Determine if a block has been accessed in the buffer pool. -@return TRUE if accessed */ +Determine the time of first access of a block in the buffer pool. +@return ut_time_ms() at the time of first access, 0 if not accessed */ UNIV_INLINE -ibool +unsigned buf_page_is_accessed( /*=================*/ const buf_page_t* bpage) /*!< in: control block */ - __attribute__((pure)); + __attribute__((nonnull, pure)); /*********************************************************************//** Flag a block accessed. */ UNIV_INLINE @@ -840,7 +831,8 @@ void buf_page_set_accessed( /*==================*/ buf_page_t* bpage, /*!< in/out: control block */ - ibool accessed); /*!< in: accessed */ + ulint time_ms) /*!< in: ut_time_ms() */ + __attribute__((nonnull)); /*********************************************************************//** Gets the buf_block_t handle of a buffered file block if an uncompressed page frame exists, or NULL. @@ -1026,14 +1018,6 @@ buf_block_hash_get( /*===============*/ ulint space, /*!< in: space id */ ulint offset);/*!< in: offset of the page within space */ -/*******************************************************************//** -Increments the pool clock by one and returns its new value. Remember that -in the 32 bit version the clock wraps around at 4 billion! -@return new clock value */ -UNIV_INLINE -ulint -buf_pool_clock_tic(void); -/*====================*/ /*********************************************************************//** Gets the current length of the free list of buffer blocks. @return length of the free list */ @@ -1073,16 +1057,10 @@ struct buf_page_struct{ flushed to disk, this tells the flush_type. @see enum buf_flush */ - unsigned accessed:1; /*!< TRUE if the page has been accessed - while in the buffer pool: read-ahead - may read in pages which have not been - accessed yet; a thread is allowed to - read this for heuristic purposes - without holding any mutex or latch */ unsigned io_fix:2; /*!< type of pending I/O operation; also protected by buf_pool_mutex @see enum buf_io_fix */ - unsigned buf_fix_count:24;/*!< count of how manyfold this block + unsigned buf_fix_count:25;/*!< count of how manyfold this block is currently bufferfixed */ /* @} */ #endif /* !UNIV_HOTBACKUP */ @@ -1112,7 +1090,16 @@ struct buf_page_struct{ - BUF_BLOCK_FILE_PAGE: flush_list - BUF_BLOCK_ZIP_DIRTY: flush_list - BUF_BLOCK_ZIP_PAGE: zip_clean - - BUF_BLOCK_ZIP_FREE: zip_free[] */ + - BUF_BLOCK_ZIP_FREE: zip_free[] + + The contents of the list node + is undefined if !in_flush_list + && state == BUF_BLOCK_FILE_PAGE, + or if state is one of + BUF_BLOCK_MEMORY, + BUF_BLOCK_REMOVE_HASH or + BUF_BLOCK_READY_IN_USE. */ + /* resplit for optimistic use */ UT_LIST_NODE_T(buf_page_t) free; UT_LIST_NODE_T(buf_page_t) flush_list; @@ -1155,18 +1142,8 @@ struct buf_page_struct{ debugging */ //#endif /* UNIV_DEBUG */ unsigned old:1; /*!< TRUE if the block is in the old - blocks in the LRU list */ - unsigned LRU_position:31;/*!< value which monotonically - decreases (or may stay - constant if old==TRUE) toward - the end of the LRU list, if - buf_pool->ulint_clock has not - wrapped around: NOTE that this - value can only be used in - heuristic algorithms, because - of the possibility of a - wrap-around! */ - unsigned freed_page_clock:32;/*!< the value of + blocks in buf_pool->LRU_old */ + unsigned freed_page_clock:31;/*!< the value of buf_pool->freed_page_clock when this block was the last time put to the head of the @@ -1174,6 +1151,9 @@ struct buf_page_struct{ to read this for heuristic purposes without holding any mutex or latch */ + unsigned access_time:32; /*!< time of first access, or + 0 if the block was never accessed + in the buffer pool */ /* @} */ # ifdef UNIV_DEBUG_FILE_ACCESSES ibool file_page_was_freed; @@ -1318,6 +1298,31 @@ Compute the hash fold value for blocks i #define BUF_POOL_ZIP_FOLD_BPAGE(b) BUF_POOL_ZIP_FOLD((buf_block_t*) (b)) /* @} */ +/** @brief The buffer pool statistics structure. */ +struct buf_pool_stat_struct{ + ulint n_page_gets; /*!< number of page gets performed; + also successful searches through + the adaptive hash index are + counted as page gets; this field + is NOT protected by the buffer + pool mutex */ + ulint n_pages_read; /*!< number read operations */ + ulint n_pages_written;/*!< number write operations */ + ulint n_pages_created;/*!< number of pages created + in the pool with no read */ + ulint n_ra_pages_read;/*!< number of pages read in + as part of read ahead */ + ulint n_ra_pages_evicted;/*!< number of read ahead + pages that are evicted without + being accessed */ + ulint n_pages_made_young; /*!< number of pages made young, in + calls to buf_LRU_make_block_young() */ + ulint n_pages_not_made_young; /*!< number of pages not made + young because the first access + was not long enough ago, in + buf_page_peek_if_too_old() */ +}; + /** @brief The buffer pool structure. NOTE! The definition appears here only for other modules of this @@ -1342,28 +1347,16 @@ struct buf_pool_struct{ ulint n_pend_reads; /*!< number of pending read operations */ ulint n_pend_unzip; /*!< number of pending decompressions */ - time_t last_printout_time; /*!< when buf_print was last time + time_t last_printout_time; + /*!< when buf_print_io was last time called */ - ulint n_pages_read; /*!< number read operations */ - ulint n_pages_written;/*!< number write operations */ - ulint n_pages_created;/*!< number of pages created - in the pool with no read */ - ulint n_page_gets; /*!< number of page gets performed; - also successful searches through - the adaptive hash index are - counted as page gets; this field - is NOT protected by the buffer - pool mutex */ - ulint n_page_gets_old;/*!< n_page_gets when buf_print was - last time called: used to calculate - hit rate */ - ulint n_pages_read_old;/*!< n_pages_read when buf_print was - last time called */ - ulint n_pages_written_old;/*!< number write operations */ - ulint n_pages_created_old;/*!< number of pages created in - the pool with no read */ + buf_pool_stat_t stat; /*!< current statistics */ + buf_pool_stat_t old_stat; /*!< old statistics */ + /* @} */ + /** @name Page flushing algorithm fields */ + /* @{ */ UT_LIST_BASE_NODE_T(buf_page_t) flush_list; @@ -1379,10 +1372,6 @@ struct buf_pool_struct{ /*!< this is in the set state when there is no flush batch of the given type running */ - ulint ulint_clock; /*!< a sequence number used to count - time. NOTE! This counter wraps - around at 4 billion (if ulint == - 32 bits)! */ ulint freed_page_clock;/*!< a sequence number used to count the number of buffer blocks removed from the end of @@ -1406,17 +1395,18 @@ struct buf_pool_struct{ block list */ UT_LIST_BASE_NODE_T(buf_page_t) LRU; /*!< base node of the LRU list */ - buf_page_t* LRU_old; /*!< pointer to the about 3/8 oldest - blocks in the LRU list; NULL if LRU - length less than BUF_LRU_OLD_MIN_LEN; + buf_page_t* LRU_old; /*!< pointer to the about + buf_LRU_old_ratio/BUF_LRU_OLD_RATIO_DIV + oldest blocks in the LRU list; + NULL if LRU length less than + BUF_LRU_OLD_MIN_LEN; NOTE: when LRU_old != NULL, its length should always equal LRU_old_len */ ulint LRU_old_len; /*!< length of the LRU list from the block to which LRU_old points onward, including that block; see buf0lru.c for the restrictions - on this value; not defined if - LRU_old == NULL; + on this value; 0 if LRU_old == NULL; NOTE: LRU_old_len must be adjusted whenever LRU_old shrinks or grows! */ === modified file 'storage/xtradb/include/buf0buf.ic' --- a/storage/xtradb/include/buf0buf.ic 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/include/buf0buf.ic 2010-01-15 15:58:25 +0000 @@ -72,9 +72,30 @@ buf_page_peek_if_too_old( /*=====================*/ const buf_page_t* bpage) /*!< in: block to make younger */ { - return(buf_pool->freed_page_clock - >= buf_page_get_freed_page_clock(bpage) - + 1 + (buf_pool->curr_size / 4)); + if (UNIV_UNLIKELY(buf_pool->freed_page_clock == 0)) { + /* If eviction has not started yet, do not update the + statistics or move blocks in the LRU list. This is + either the warm-up phase or an in-memory workload. */ + return(FALSE); + } else if (buf_LRU_old_threshold_ms && bpage->old) { + unsigned access_time = buf_page_is_accessed(bpage); + + if (access_time > 0 + && (ut_time_ms() - access_time) + >= buf_LRU_old_threshold_ms) { + return(TRUE); + } + + buf_pool->stat.n_pages_not_made_young++; + return(FALSE); + } else { + /* FIXME: bpage->freed_page_clock is 31 bits */ + return((buf_pool->freed_page_clock & ((1UL << 31) - 1)) + > ((ulint) bpage->freed_page_clock + + (buf_pool->curr_size + * (BUF_LRU_OLD_RATIO_DIV - buf_LRU_old_ratio) + / (BUF_LRU_OLD_RATIO_DIV * 4)))); + } } /*********************************************************************//** @@ -125,23 +146,6 @@ try_again: return(lsn); } - -/*******************************************************************//** -Increments the buf_pool clock by one and returns its new value. Remember -that in the 32 bit version the clock wraps around at 4 billion! -@return new clock value */ -UNIV_INLINE -ulint -buf_pool_clock_tic(void) -/*====================*/ -{ - //ut_ad(buf_pool_mutex_own()); - ut_ad(mutex_own(&LRU_list_mutex)); - - buf_pool->ulint_clock++; - - return(buf_pool->ulint_clock); -} #endif /* !UNIV_HOTBACKUP */ /*********************************************************************//** @@ -288,21 +292,6 @@ buf_page_belongs_to_unzip_LRU( } /*********************************************************************//** -Determine the approximate LRU list position of a block. -@return LRU list position */ -UNIV_INLINE -ulint -buf_page_get_LRU_position( -/*======================*/ - const buf_page_t* bpage) /*!< in: control block */ -{ - ut_ad(buf_page_in_file(bpage)); - //ut_ad(buf_pool_mutex_own()); /* This is used in optimistic */ - - return(bpage->LRU_position); -} - -/*********************************************************************//** Gets the mutex of a block. @return pointer to mutex protecting bpage */ UNIV_INLINE @@ -508,10 +497,19 @@ buf_page_set_old( ut_ad(bpage->in_LRU_list); #ifdef UNIV_LRU_DEBUG - if (UT_LIST_GET_PREV(LRU, bpage) && UT_LIST_GET_NEXT(LRU, bpage) - && UT_LIST_GET_PREV(LRU, bpage)->old - == UT_LIST_GET_NEXT(LRU, bpage)->old) { - ut_a(UT_LIST_GET_PREV(LRU, bpage)->old == old); + ut_a((buf_pool->LRU_old_len == 0) == (buf_pool->LRU_old == NULL)); + /* If a block is flagged "old", the LRU_old list must exist. */ + ut_a(!old || buf_pool->LRU_old); + + if (UT_LIST_GET_PREV(LRU, bpage) && UT_LIST_GET_NEXT(LRU, bpage)) { + const buf_page_t* prev = UT_LIST_GET_PREV(LRU, bpage); + const buf_page_t* next = UT_LIST_GET_NEXT(LRU, bpage); + if (prev->old == next->old) { + ut_a(prev->old == old); + } else { + ut_a(!prev->old); + ut_a(buf_pool->LRU_old == (old ? bpage : next)); + } } #endif /* UNIV_LRU_DEBUG */ @@ -519,17 +517,17 @@ buf_page_set_old( } /*********************************************************************//** -Determine if a block has been accessed in the buffer pool. -@return TRUE if accessed */ +Determine the time of first access of a block in the buffer pool. +@return ut_time_ms() at the time of first access, 0 if not accessed */ UNIV_INLINE -ibool +unsigned buf_page_is_accessed( /*=================*/ const buf_page_t* bpage) /*!< in: control block */ { ut_ad(buf_page_in_file(bpage)); - return(bpage->accessed); + return(bpage->access_time); } /*********************************************************************//** @@ -539,12 +537,16 @@ void buf_page_set_accessed( /*==================*/ buf_page_t* bpage, /*!< in/out: control block */ - ibool accessed) /*!< in: accessed */ + ulint time_ms) /*!< in: ut_time_ms() */ { ut_a(buf_page_in_file(bpage)); + //ut_ad(buf_pool_mutex_own()); ut_ad(mutex_own(buf_page_get_mutex(bpage))); - bpage->accessed = accessed; + if (!bpage->access_time) { + /* Make this the time of the first access. */ + bpage->access_time = time_ms; + } } /*********************************************************************//** @@ -825,15 +827,15 @@ buf_page_get_newest_modification( ib_uint64_t lsn; mutex_t* block_mutex = buf_page_get_mutex_enter(bpage); - ut_a(block_mutex); - - if (buf_page_in_file(bpage)) { + if (block_mutex && buf_page_in_file(bpage)) { lsn = bpage->newest_modification; } else { lsn = 0; } - mutex_exit(block_mutex); + if (block_mutex) { + mutex_exit(block_mutex); + } return(lsn); } === modified file 'storage/xtradb/include/buf0lru.h' --- a/storage/xtradb/include/buf0lru.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/buf0lru.h 2010-01-06 12:00:14 +0000 @@ -69,7 +69,7 @@ These are low-level functions #########################################################################*/ /** Minimum LRU list length for which the LRU_old pointer is defined */ -#define BUF_LRU_OLD_MIN_LEN 80 +#define BUF_LRU_OLD_MIN_LEN 512 /* 8 megabytes of 16k pages */ /** Maximum LRU list search length in buf_flush_LRU_recommendation() */ #define BUF_LRU_FREE_SEARCH_LEN (5 + 2 * BUF_READ_AHEAD_AREA) @@ -84,15 +84,6 @@ void buf_LRU_invalidate_tablespace( /*==========================*/ ulint id); /*!< in: space id */ -/******************************************************************//** -Gets the minimum LRU_position field for the blocks in an initial segment -(determined by BUF_LRU_INITIAL_RATIO) of the LRU list. The limit is not -guaranteed to be precise, because the ulint_clock may wrap around. -@return the limit; zero if could not determine it */ -UNIV_INTERN -ulint -buf_LRU_get_recent_limit(void); -/*==========================*/ /********************************************************************//** Insert a compressed block into buf_pool->zip_clean in the LRU order. */ UNIV_INTERN @@ -203,6 +194,18 @@ void buf_LRU_make_block_old( /*===================*/ buf_page_t* bpage); /*!< in: control block */ +/**********************************************************************//** +Updates buf_LRU_old_ratio. +@return updated old_pct */ +UNIV_INTERN +uint +buf_LRU_old_ratio_update( +/*=====================*/ + uint old_pct,/*!< in: Reserve this percentage of + the buffer pool for "old" blocks. */ + ibool adjust);/*!< in: TRUE=adjust the LRU list; + FALSE=just assign buf_LRU_old_ratio + during the initialization of InnoDB */ /********************************************************************//** Update the historical stats that we are collecting for LRU eviction policy at the end of each interval. */ @@ -210,6 +213,18 @@ UNIV_INTERN void buf_LRU_stat_update(void); /*=====================*/ +/********************************************************************//** +Dump the LRU page list to the specific file. */ +UNIV_INTERN +ibool +buf_LRU_file_dump(void); +/*===================*/ +/********************************************************************//** +Read the pages based on the specific file.*/ +UNIV_INTERN +ibool +buf_LRU_file_restore(void); +/*======================*/ #if defined UNIV_DEBUG || defined UNIV_BUF_DEBUG /**********************************************************************//** @@ -229,6 +244,35 @@ buf_LRU_print(void); /*===============*/ #endif /* UNIV_DEBUG_PRINT || UNIV_DEBUG || UNIV_BUF_DEBUG */ +/** @name Heuristics for detecting index scan @{ */ +/** Reserve this much/BUF_LRU_OLD_RATIO_DIV of the buffer pool for +"old" blocks. Protected by buf_pool_mutex. */ +extern uint buf_LRU_old_ratio; +/** The denominator of buf_LRU_old_ratio. */ +#define BUF_LRU_OLD_RATIO_DIV 1024 +/** Maximum value of buf_LRU_old_ratio. +@see buf_LRU_old_adjust_len +@see buf_LRU_old_ratio_update */ +#define BUF_LRU_OLD_RATIO_MAX BUF_LRU_OLD_RATIO_DIV +/** Minimum value of buf_LRU_old_ratio. +@see buf_LRU_old_adjust_len +@see buf_LRU_old_ratio_update +The minimum must exceed +(BUF_LRU_OLD_TOLERANCE + 5) * BUF_LRU_OLD_RATIO_DIV / BUF_LRU_OLD_MIN_LEN. */ +#define BUF_LRU_OLD_RATIO_MIN 51 + +#if BUF_LRU_OLD_RATIO_MIN >= BUF_LRU_OLD_RATIO_MAX +# error "BUF_LRU_OLD_RATIO_MIN >= BUF_LRU_OLD_RATIO_MAX" +#endif +#if BUF_LRU_OLD_RATIO_MAX > BUF_LRU_OLD_RATIO_DIV +# error "BUF_LRU_OLD_RATIO_MAX > BUF_LRU_OLD_RATIO_DIV" +#endif + +/** Move blocks to "new" LRU list only if the first access was at +least this many milliseconds ago. Not protected by any mutex or latch. */ +extern uint buf_LRU_old_threshold_ms; +/* @} */ + /** @brief Statistics for selecting the LRU list for eviction. These statistics are not 'of' LRU but 'for' LRU. We keep count of I/O === modified file 'storage/xtradb/include/buf0rea.h' --- a/storage/xtradb/include/buf0rea.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/buf0rea.h 2010-01-06 12:00:14 +0000 @@ -27,28 +27,59 @@ Created 11/5/1995 Heikki Tuuri #define buf0rea_h #include "univ.i" +#include "trx0types.h" #include "buf0types.h" /********************************************************************//** +Low-level function which reads a page asynchronously from a file to the +buffer buf_pool if it is not already there, in which case does nothing. +Sets the io_fix flag and sets an exclusive lock on the buffer frame. The +flag is cleared and the x-lock released by an i/o-handler thread. +@return 1 if a read request was queued, 0 if the page already resided +in buf_pool, or if the page is in the doublewrite buffer blocks in +which case it is never read into the pool, or if the tablespace does +not exist or is being dropped +@return 1 if read request is issued. 0 if it is not */ +UNIV_INTERN +ulint +buf_read_page_low( +/*==============*/ + ulint* err, /*!< out: DB_SUCCESS or DB_TABLESPACE_DELETED if we are + trying to read from a non-existent tablespace, or a + tablespace which is just now being dropped */ + ibool sync, /*!< in: TRUE if synchronous aio is desired */ + ulint mode, /*!< in: BUF_READ_IBUF_PAGES_ONLY, ..., + ORed to OS_AIO_SIMULATED_WAKE_LATER (see below + at read-ahead functions) */ + ulint space, /*!< in: space id */ + ulint zip_size,/*!< in: compressed page size, or 0 */ + ibool unzip, /*!< in: TRUE=request uncompressed page */ + ib_int64_t tablespace_version, /*!< in: if the space memory object has + this timestamp different from what we are giving here, + treat the tablespace as dropped; this is a timestamp we + use to stop dangling page reads from a tablespace + which we have DISCARDed + IMPORTed back */ + ulint offset, /*!< in: page number */ + trx_t* trx); +/********************************************************************//** High-level function which reads a page asynchronously from a file to the buffer buf_pool if it is not already there. Sets the io_fix flag and sets an exclusive lock on the buffer frame. The flag is cleared and the x-lock -released by the i/o-handler thread. Does a random read-ahead if it seems -sensible. -@return number of page read requests issued: this can be greater than -1 if read-ahead occurred */ +released by the i/o-handler thread. +@return TRUE if page has been read in, FALSE in case of failure */ UNIV_INTERN -ulint +ibool buf_read_page( /*==========*/ ulint space, /*!< in: space id */ ulint zip_size,/*!< in: compressed page size in bytes, or 0 */ - ulint offset);/*!< in: page number */ + ulint offset, /*!< in: page number */ + trx_t* trx); /********************************************************************//** Applies linear read-ahead if in the buf_pool the page is a border page of a linear read-ahead area and all the pages in the area have been accessed. Does not read any page if the read-ahead mechanism is not activated. Note -that the the algorithm looks at the 'natural' adjacent successor and +that the algorithm looks at the 'natural' adjacent successor and predecessor of the page, which on the leaf level of a B-tree are the next and previous page in the chain of leaves. To know these, the page specified in (space, offset) must already be present in the buf_pool. Thus, the @@ -74,8 +105,9 @@ buf_read_ahead_linear( /*==================*/ ulint space, /*!< in: space id */ ulint zip_size,/*!< in: compressed page size in bytes, or 0 */ - ulint offset);/*!< in: page number of a page; NOTE: the current thread + ulint offset, /*!< in: page number of a page; NOTE: the current thread must want access to this page (see NOTE 3 above) */ + trx_t* trx); /********************************************************************//** Issues read requests for pages which the ibuf module wants to read in, in order to contract the insert buffer tree. Technically, this function is like === modified file 'storage/xtradb/include/buf0types.h' --- a/storage/xtradb/include/buf0types.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/buf0types.h 2010-01-06 12:00:14 +0000 @@ -34,6 +34,8 @@ typedef struct buf_block_struct buf_blo typedef struct buf_chunk_struct buf_chunk_t; /** Buffer pool comprising buf_chunk_t */ typedef struct buf_pool_struct buf_pool_t; +/** Buffer pool statistics struct */ +typedef struct buf_pool_stat_struct buf_pool_stat_t; /** A buffer frame. @see page_t */ typedef byte buf_frame_t; === modified file 'storage/xtradb/include/db0err.h' --- a/storage/xtradb/include/db0err.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/db0err.h 2010-01-06 12:00:14 +0000 @@ -32,6 +32,7 @@ enum db_err { /* The following are error codes */ DB_ERROR, + DB_INTERRUPTED, DB_OUT_OF_MEMORY, DB_OUT_OF_FILE_SPACE, DB_LOCK_WAIT, === modified file 'storage/xtradb/include/dict0crea.h' --- a/storage/xtradb/include/dict0crea.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/dict0crea.h 2010-01-06 12:00:14 +0000 @@ -110,7 +110,7 @@ dict_create_or_check_foreign_constraint_ Adds foreign key definitions to data dictionary tables in the database. We look at table->foreign_list, and also generate names to constraints that were not named by the user. A generated constraint has a name of the format -databasename/tablename_ibfk_<number>, where the numbers start from 1, and are +databasename/tablename_ibfk_NUMBER, where the numbers start from 1, and are given locally for this table, that is, the number is not global, as in the old format constraints < 4.0.18 it used to be. @return error code or DB_SUCCESS */ === modified file 'storage/xtradb/include/dict0dict.h' --- a/storage/xtradb/include/dict0dict.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/dict0dict.h 2010-01-06 12:00:14 +0000 @@ -712,7 +712,7 @@ dict_index_find_on_id_low( dulint id); /*!< in: index id */ /**********************************************************************//** Adds an index to the dictionary cache. -@return DB_SUCCESS or error code */ +@return DB_SUCCESS, DB_TOO_BIG_RECORD, or DB_CORRUPTION */ UNIV_INTERN ulint dict_index_add_to_cache( @@ -1157,6 +1157,13 @@ void dict_ind_init(void); /*===============*/ +/**********************************************************************//** +Closes the data dictionary module. */ +UNIV_INTERN +void +dict_close(void); +/*============*/ + #ifndef UNIV_NONINL #include "dict0dict.ic" #endif === modified file 'storage/xtradb/include/dict0mem.h' --- a/storage/xtradb/include/dict0mem.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/dict0mem.h 2010-01-06 12:00:14 +0000 @@ -317,7 +317,7 @@ struct dict_foreign_struct{ char* id; /*!< id of the constraint as a null-terminated string */ unsigned n_fields:10; /*!< number of indexes' first fields - for which the the foreign key + for which the foreign key constraint is defined: we allow the indexes to contain more fields than mentioned in the constraint, as long === modified file 'storage/xtradb/include/fil0fil.h' --- a/storage/xtradb/include/fil0fil.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/fil0fil.h 2010-01-06 12:00:14 +0000 @@ -224,15 +224,6 @@ fil_space_create( 0 for uncompressed tablespaces */ ulint purpose);/*!< in: FIL_TABLESPACE, or FIL_LOG if log */ /*******************************************************************//** -Frees a space object from a the tablespace memory cache. Closes the files in -the chain but does not delete them. -@return TRUE if success */ -UNIV_INTERN -ibool -fil_space_free( -/*===========*/ - ulint id); /*!< in: space id */ -/*******************************************************************//** Returns the size of the space in pages. The tablespace must be cached in the memory cache. @return space size, 0 if space not found */ @@ -278,6 +269,12 @@ fil_init( ulint hash_size, /*!< in: hash table size */ ulint max_n_open); /*!< in: max number of open files */ /*******************************************************************//** +Initializes the tablespace memory cache. */ +UNIV_INTERN +void +fil_close(void); +/*===========*/ +/*******************************************************************//** Opens all log files and system tablespace data files. They stay open until the database server shutdown. This should be called at a server startup after the space objects for the log and the system tablespace have been created. The @@ -614,9 +611,12 @@ fil_space_get_n_reserved_extents( Reads or writes data. This operation is asynchronous (aio). @return DB_SUCCESS, or DB_TABLESPACE_DELETED if we are trying to do i/o on a tablespace which does not exist */ +#define fil_io(type, sync, space_id, zip_size, block_offset, byte_offset, len, buf, message) \ + _fil_io(type, sync, space_id, zip_size, block_offset, byte_offset, len, buf, message, NULL) + UNIV_INTERN ulint -fil_io( +_fil_io( /*===*/ ulint type, /*!< in: OS_FILE_READ or OS_FILE_WRITE, ORed to OS_FILE_LOG, if a log i/o @@ -641,8 +641,25 @@ fil_io( void* buf, /*!< in/out: buffer where to store read data or from where to write; in aio this must be appropriately aligned */ - void* message); /*!< in: message for aio handler if non-sync + void* message, /*!< in: message for aio handler if non-sync aio used, else ignored */ + trx_t* trx); +/********************************************************************//** +Confirm whether the parameters are valid or not */ +UNIV_INTERN +ibool +fil_area_is_exist( +/*==============*/ + ulint space_id, /*!< in: space id */ + ulint zip_size, /*!< in: compressed page size in bytes; + 0 for uncompressed pages */ + ulint block_offset, /*!< in: offset in number of blocks */ + ulint byte_offset, /*!< in: remainder of offset in bytes; in + aio this must be divisible by the OS block + size */ + ulint len); /*!< in: how many bytes to read or write; this + must not cross a file boundary; in aio this + must be a block size multiple */ /**********************************************************************//** Waits for an aio operation to complete. This function is used to write the handler for completed requests. The aio array of pending requests is divided === modified file 'storage/xtradb/include/fsp0fsp.h' --- a/storage/xtradb/include/fsp0fsp.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/fsp0fsp.h 2010-01-06 12:00:14 +0000 @@ -42,7 +42,7 @@ fsp_init(void); /*==========*/ /**********************************************************************//** Gets the current free limit of the system tablespace. The free limit -means the place of the first page which has never been put to the the +means the place of the first page which has never been put to the free list for allocation. The space above that address is initialized to zero. Sets also the global variable log_fsp_current_free_limit. @return free limit in megabytes */ === modified file 'storage/xtradb/include/ibuf0ibuf.h' --- a/storage/xtradb/include/ibuf0ibuf.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/ibuf0ibuf.h 2010-01-06 12:00:14 +0000 @@ -356,6 +356,12 @@ void ibuf_print( /*=======*/ FILE* file); /*!< in: file where to print */ +/******************************************************************//** +Closes insert buffer and frees the data structures. */ +UNIV_INTERN +void +ibuf_close(void); +/*============*/ #define IBUF_HEADER_PAGE_NO FSP_IBUF_HEADER_PAGE_NO #define IBUF_TREE_ROOT_PAGE_NO FSP_IBUF_TREE_ROOT_PAGE_NO === modified file 'storage/xtradb/include/lock0lock.h' --- a/storage/xtradb/include/lock0lock.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/lock0lock.h 2010-01-06 12:00:14 +0000 @@ -59,6 +59,12 @@ lock_sys_create( /*============*/ ulint n_cells); /*!< in: number of slots in lock hash table */ /*********************************************************************//** +Closes the lock system at database shutdown. */ +UNIV_INTERN +void +lock_sys_close(void); +/*================*/ +/*********************************************************************//** Checks if some transaction has an implicit x-lock on a record in a clustered index. @return transaction which has the x-lock, or NULL */ @@ -630,6 +636,14 @@ lock_number_of_rows_locked( /*=======================*/ trx_t* trx); /*!< in: transaction */ /*******************************************************************//** +Check if a transaction holds any autoinc locks. +@return TRUE if the transaction holds any AUTOINC locks. */ +UNIV_INTERN +ibool +lock_trx_holds_autoinc_locks( +/*=========================*/ + const trx_t* trx); /*!< in: transaction */ +/*******************************************************************//** Release all the transaction's autoinc locks. */ UNIV_INTERN void === modified file 'storage/xtradb/include/log0log.h' --- a/storage/xtradb/include/log0log.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/log0log.h 2010-01-06 12:00:14 +0000 @@ -118,10 +118,9 @@ UNIV_INLINE ib_uint64_t log_reserve_and_write_fast( /*=======================*/ - byte* str, /*!< in: string */ + const void* str, /*!< in: string */ ulint len, /*!< in: string length */ - ib_uint64_t* start_lsn,/*!< out: start lsn of the log record */ - ibool* success);/*!< out: TRUE if success */ + ib_uint64_t* start_lsn);/*!< out: start lsn of the log record */ /***********************************************************************//** Releases the log mutex. */ UNIV_INLINE @@ -283,7 +282,7 @@ log_make_checkpoint_at( later lsn, if IB_ULONGLONG_MAX, makes a checkpoint at the latest lsn */ ibool write_always); /*!< in: the function normally checks if - the the new checkpoint would have a + the new checkpoint would have a greater lsn than the previous one: if not, then no physical write is done; by setting this parameter TRUE, a @@ -573,6 +572,18 @@ UNIV_INTERN void log_refresh_stats(void); /*===================*/ +/********************************************************** +Shutdown the log system but do not release all the memory. */ +UNIV_INTERN +void +log_shutdown(void); +/*==============*/ +/********************************************************** +Free the log system data structures. */ +UNIV_INTERN +void +log_mem_free(void); +/*==============*/ extern log_t* log_sys; @@ -585,7 +596,7 @@ extern log_t* log_sys; #define LOG_RECOVER 98887331 /* The counting of lsn's starts from this value: this must be non-zero */ -#define LOG_START_LSN ((ib_uint64_t) (16 * OS_FILE_LOG_BLOCK_SIZE)) +#define LOG_START_LSN ((ib_uint64_t) (16 * OS_FILE_LOG_BLOCK_SIZE)) #define LOG_BUFFER_SIZE (srv_log_buffer_size * UNIV_PAGE_SIZE) #define LOG_ARCHIVE_BUF_SIZE (srv_log_buffer_size * UNIV_PAGE_SIZE / 4) @@ -722,9 +733,12 @@ struct log_group_struct{ ulint lsn_offset; /*!< the offset of the above lsn */ ulint n_pending_writes;/*!< number of currently pending flush writes for this log group */ + byte** file_header_bufs_ptr;/*!< unaligned buffers */ byte** file_header_bufs;/*!< buffers for each file header in the group */ +#ifdef UNIV_LOG_ARCHIVE /*-----------------------------*/ + byte** archive_file_header_bufs_ptr;/*!< unaligned buffers */ byte** archive_file_header_bufs;/*!< buffers for each file header in the group */ ulint archive_space_id;/*!< file space which @@ -743,10 +757,12 @@ struct log_group_struct{ completion function then sets the new value to ..._file_no */ ulint next_archived_offset; /*!< like the preceding field */ +#endif /* UNIV_LOG_ARCHIVE */ /*-----------------------------*/ ib_uint64_t scanned_lsn; /*!< used only in recovery: recovery scan succeeded up to this lsn in this log group */ + byte* checkpoint_buf_ptr;/*!< unaligned checkpoint header */ byte* checkpoint_buf; /*!< checkpoint header is written from this buffer to the group */ UT_LIST_NODE_T(log_group_t) @@ -764,6 +780,7 @@ struct log_struct{ #ifndef UNIV_HOTBACKUP mutex_t mutex; /*!< mutex protecting the log */ #endif /* !UNIV_HOTBACKUP */ + byte* buf_ptr; /* unaligned log buffer */ byte* buf; /*!< log buffer */ ulint buf_size; /*!< log buffer size in bytes */ ulint max_buf_free; /*!< recommended maximum value of @@ -900,6 +917,7 @@ struct log_struct{ should wait for this without owning the log mutex */ #endif /* !UNIV_HOTBACKUP */ + byte* checkpoint_buf_ptr;/* unaligned checkpoint header */ byte* checkpoint_buf; /*!< checkpoint header is read to this buffer */ /* @} */ === modified file 'storage/xtradb/include/log0log.ic' --- a/storage/xtradb/include/log0log.ic 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/log0log.ic 2010-01-06 12:00:14 +0000 @@ -27,6 +27,7 @@ Created 12/9/1995 Heikki Tuuri #include "mach0data.h" #include "mtr0mtr.h" +#ifdef UNIV_LOG_DEBUG /******************************************************//** Checks by parsing that the catenated log segment for a single mtr is consistent. */ @@ -34,11 +35,12 @@ UNIV_INTERN ibool log_check_log_recs( /*===============*/ - byte* buf, /*!< in: pointer to the start of + const byte* buf, /*!< in: pointer to the start of the log segment in the log_sys->buf log buffer */ ulint len, /*!< in: segment length in bytes */ ib_uint64_t buf_start_lsn); /*!< in: buffer start lsn */ +#endif /* UNIV_LOG_DEBUG */ /************************************************************//** Gets a log block flush bit. @@ -305,55 +307,76 @@ UNIV_INLINE ib_uint64_t log_reserve_and_write_fast( /*=======================*/ - byte* str, /*!< in: string */ + const void* str, /*!< in: string */ ulint len, /*!< in: string length */ - ib_uint64_t* start_lsn,/*!< out: start lsn of the log record */ - ibool* success)/*!< out: TRUE if success */ + ib_uint64_t* start_lsn)/*!< out: start lsn of the log record */ { - log_t* log = log_sys; ulint data_len; - ib_uint64_t lsn; - - *success = TRUE; - - mutex_enter(&(log->mutex)); - - data_len = len + log->buf_free % OS_FILE_LOG_BLOCK_SIZE; +#ifdef UNIV_LOG_LSN_DEBUG + /* length of the LSN pseudo-record */ + ulint lsn_len = 1 + + mach_get_compressed_size(log_sys->lsn >> 32) + + mach_get_compressed_size(log_sys->lsn & 0xFFFFFFFFUL); +#endif /* UNIV_LOG_LSN_DEBUG */ + + mutex_enter(&log_sys->mutex); + + data_len = len +#ifdef UNIV_LOG_LSN_DEBUG + + lsn_len +#endif /* UNIV_LOG_LSN_DEBUG */ + + log_sys->buf_free % OS_FILE_LOG_BLOCK_SIZE; if (data_len >= OS_FILE_LOG_BLOCK_SIZE - LOG_BLOCK_TRL_SIZE) { /* The string does not fit within the current log block or the log block would become full */ - *success = FALSE; - - mutex_exit(&(log->mutex)); + mutex_exit(&log_sys->mutex); return(0); } - *start_lsn = log->lsn; + *start_lsn = log_sys->lsn; + +#ifdef UNIV_LOG_LSN_DEBUG + { + /* Write the LSN pseudo-record. */ + byte* b = &log_sys->buf[log_sys->buf_free]; + *b++ = MLOG_LSN | (MLOG_SINGLE_REC_FLAG & *(const byte*) str); + /* Write the LSN in two parts, + as a pseudo page number and space id. */ + b += mach_write_compressed(b, log_sys->lsn >> 32); + b += mach_write_compressed(b, log_sys->lsn & 0xFFFFFFFFUL); + ut_a(b - lsn_len == &log_sys->buf[log_sys->buf_free]); - ut_memcpy(log->buf + log->buf_free, str, len); + memcpy(b, str, len); + len += lsn_len; + } +#else /* UNIV_LOG_LSN_DEBUG */ + memcpy(log_sys->buf + log_sys->buf_free, str, len); +#endif /* UNIV_LOG_LSN_DEBUG */ - log_block_set_data_len((byte*) ut_align_down(log->buf + log->buf_free, + log_block_set_data_len((byte*) ut_align_down(log_sys->buf + + log_sys->buf_free, OS_FILE_LOG_BLOCK_SIZE), data_len); #ifdef UNIV_LOG_DEBUG - log->old_buf_free = log->buf_free; - log->old_lsn = log->lsn; + log_sys->old_buf_free = log_sys->buf_free; + log_sys->old_lsn = log_sys->lsn; #endif - log->buf_free += len; + log_sys->buf_free += len; - ut_ad(log->buf_free <= log->buf_size); + ut_ad(log_sys->buf_free <= log_sys->buf_size); - lsn = log->lsn += len; + log_sys->lsn += len; #ifdef UNIV_LOG_DEBUG - log_check_log_recs(log->buf + log->old_buf_free, - log->buf_free - log->old_buf_free, log->old_lsn); + log_check_log_recs(log_sys->buf + log_sys->old_buf_free, + log_sys->buf_free - log_sys->old_buf_free, + log_sys->old_lsn); #endif - return(lsn); + return(log_sys->lsn); } /***********************************************************************//** === modified file 'storage/xtradb/include/log0recv.h' --- a/storage/xtradb/include/log0recv.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/log0recv.h 2010-01-06 12:00:14 +0000 @@ -239,6 +239,18 @@ UNIV_INTERN void recv_sys_create(void); /*=================*/ +/**********************************************************//** +Release recovery system mutexes. */ +UNIV_INTERN +void +recv_sys_close(void); +/*================*/ +/********************************************************//** +Frees the recovery system memory. */ +UNIV_INTERN +void +recv_sys_mem_free(void); +/*===================*/ /********************************************************//** Inits the recovery system for a recovery operation. */ UNIV_INTERN @@ -246,6 +258,12 @@ void recv_sys_init( /*==========*/ ulint available_memory); /*!< in: available memory in bytes */ +/********************************************************//** +Reset the state of the recovery system variables. */ +UNIV_INTERN +void +recv_sys_var_init(void); +/*===================*/ /*******************************************************************//** Empties the hash table of stored log records, applying them to appropriate pages. */ @@ -412,6 +430,39 @@ struct recv_sys_struct{ hash_table_t* addr_hash;/*!< hash table of file addresses of pages */ ulint n_addrs;/*!< number of not processed hashed file addresses in the hash table */ + +/* If you modified the following defines at original file, + You should also modify them. */ +/* defined in os0file.c */ +#define OS_AIO_MERGE_N_CONSECUTIVE 64 +/* defined in log0recv.c */ +#define RECV_READ_AHEAD_AREA 32 + time_t stats_recv_start_time; + ulint stats_recv_turns; + + ulint stats_read_requested_pages; + ulint stats_read_in_area[RECV_READ_AHEAD_AREA]; + + ulint stats_read_io_pages; + ulint stats_read_io_consecutive[OS_AIO_MERGE_N_CONSECUTIVE]; + ulint stats_write_io_pages; + ulint stats_write_io_consecutive[OS_AIO_MERGE_N_CONSECUTIVE]; + + ulint stats_doublewrite_check_pages; + ulint stats_doublewrite_overwrite_pages; + + ulint stats_recover_pages_with_read; + ulint stats_recover_pages_without_read; + + ulint stats_log_recs; + ulint stats_log_len_sum; + + ulint stats_applied_log_recs; + ulint stats_applied_log_len_sum; + ulint stats_pages_already_new; + + ib_uint64_t stats_oldest_modified_lsn; + ib_uint64_t stats_newest_modified_lsn; }; /** The recovery system */ @@ -433,6 +484,11 @@ are allowed yet: the variable name is mi extern ibool recv_no_ibuf_operations; /** TRUE when recv_init_crash_recovery() has been called. */ extern ibool recv_needed_recovery; +#ifdef UNIV_DEBUG +/** TRUE if writing to the redo log (mtr_commit) is forbidden. +Protected by log_sys->mutex. */ +extern ibool recv_no_log_write; +#endif /* UNIV_DEBUG */ /** TRUE if buf_page_is_corrupted() should check if the log sequence number (FIL_PAGE_LSN) is in the future. Initially FALSE, and set by === modified file 'storage/xtradb/include/mem0mem.h' --- a/storage/xtradb/include/mem0mem.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/mem0mem.h 2010-01-06 12:00:14 +0000 @@ -82,6 +82,13 @@ void mem_init( /*=====*/ ulint size); /*!< in: common pool size in bytes */ +/******************************************************************//** +Closes the memory system. */ +UNIV_INTERN +void +mem_close(void); +/*===========*/ + /**************************************************************//** Use this macro instead of the corresponding function! Macro for memory heap creation. */ === modified file 'storage/xtradb/include/mem0pool.h' --- a/storage/xtradb/include/mem0pool.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/mem0pool.h 2010-01-06 12:00:14 +0000 @@ -62,6 +62,13 @@ mem_pool_create( /*============*/ ulint size); /*!< in: pool size in bytes */ /********************************************************************//** +Frees a memory pool. */ +UNIV_INTERN +void +mem_pool_free( +/*==========*/ + mem_pool_t* pool); /*!< in, own: memory pool */ +/********************************************************************//** Allocates memory from a pool. NOTE: This low-level function should only be used in mem0mem.*! @return own: allocated memory buffer */ === modified file 'storage/xtradb/include/mtr0mtr.h' --- a/storage/xtradb/include/mtr0mtr.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/mtr0mtr.h 2010-01-06 12:00:14 +0000 @@ -106,6 +106,9 @@ For 1 - 8 bytes, the flag value must giv #define MLOG_IBUF_BITMAP_INIT ((byte)27) /*!< initialize an ibuf bitmap page */ /*#define MLOG_FULL_PAGE ((byte)28) full contents of a page */ +#ifdef UNIV_LOG_LSN_DEBUG +# define MLOG_LSN ((byte)28) /* current LSN */ +#endif #define MLOG_INIT_FILE_PAGE ((byte)29) /*!< this means that a file page is taken into use and the prior @@ -118,7 +121,7 @@ For 1 - 8 bytes, the flag value must giv #define MLOG_WRITE_STRING ((byte)30) /*!< write a string to a page */ #define MLOG_MULTI_REC_END ((byte)31) /*!< if a single mtr writes - log records for several pages, + several log records, this log record ends the sequence of these records */ #define MLOG_DUMMY_RECORD ((byte)32) /*!< dummy log record used to === modified file 'storage/xtradb/include/os0file.h' --- a/storage/xtradb/include/os0file.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/os0file.h 2010-01-06 12:00:14 +0000 @@ -53,6 +53,7 @@ Created 10/21/1995 Heikki Tuuri #define os0file_h #include "univ.i" +#include "trx0types.h" #ifndef __WIN__ #include <dirent.h> @@ -157,6 +158,8 @@ log. */ to become available again */ #define OS_FILE_SHARING_VIOLATION 76 #define OS_FILE_ERROR_NOT_SPECIFIED 77 +#define OS_FILE_INSUFFICIENT_RESOURCE 78 +#define OS_FILE_OPERATION_ABORTED 79 /* @} */ /** Types for aio operations @{ */ @@ -497,9 +500,12 @@ os_file_get_last_error( /*******************************************************************//** Requests a synchronous read operation. @return TRUE if request was successful, FALSE if fail */ +#define os_file_read(file, buf, offset, offset_high, n) \ + _os_file_read(file, buf, offset, offset_high, n, NULL) + UNIV_INTERN ibool -os_file_read( +_os_file_read( /*=========*/ os_file_t file, /*!< in: handle to a file */ void* buf, /*!< in: buffer where to read */ @@ -507,7 +513,8 @@ os_file_read( offset where to read */ ulint offset_high,/*!< in: most significant 32 bits of offset */ - ulint n); /*!< in: number of bytes to read */ + ulint n, /*!< in: number of bytes to read */ + trx_t* trx); /*******************************************************************//** Rewind file to its start, read at most size - 1 bytes from it to str, and NUL-terminate str. All errors are silently ignored. This function is @@ -619,6 +626,13 @@ os_aio_init( ulint n_write_segs, /*<! in: number of writer threads */ ulint n_slots_sync); /*<! in: number of slots in the sync aio array */ +/*********************************************************************** +Frees the asynchronous io system. */ +UNIV_INTERN +void +os_aio_free(void); +/*=============*/ + /*******************************************************************//** Requests an asynchronous i/o operation. @return TRUE if request was queued successfully, FALSE if fail */ @@ -654,10 +668,11 @@ os_aio( (can be used to identify a completed aio operation); ignored if mode is OS_AIO_SYNC */ - void* message2);/*!< in: message for the aio handler + void* message2,/*!< in: message for the aio handler (can be used to identify a completed aio operation); ignored if mode is OS_AIO_SYNC */ + trx_t* trx); /************************************************************************//** Wakes up all async i/o threads so that they know to exit themselves in shutdown. */ === modified file 'storage/xtradb/include/os0sync.h' --- a/storage/xtradb/include/os0sync.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/os0sync.h 2010-01-06 12:00:14 +0000 @@ -285,44 +285,74 @@ os_fast_mutex_free( /**********************************************************//** Atomic compare-and-swap and increment for InnoDB. */ -#ifdef HAVE_GCC_ATOMIC_BUILTINS +#if defined(HAVE_IB_GCC_ATOMIC_BUILTINS) + +#define HAVE_ATOMIC_BUILTINS + /**********************************************************//** Returns true if swapped, ptr is pointer to target, old_val is value to compare to, new_val is the value to swap in. */ + # define os_compare_and_swap(ptr, old_val, new_val) \ __sync_bool_compare_and_swap(ptr, old_val, new_val) + # define os_compare_and_swap_ulint(ptr, old_val, new_val) \ os_compare_and_swap(ptr, old_val, new_val) + # define os_compare_and_swap_lint(ptr, old_val, new_val) \ os_compare_and_swap(ptr, old_val, new_val) -# define os_compare_and_swap_thread_id(ptr, old_val, new_val) \ + +# ifdef HAVE_IB_ATOMIC_PTHREAD_T_GCC +# define os_compare_and_swap_thread_id(ptr, old_val, new_val) \ os_compare_and_swap(ptr, old_val, new_val) +# define INNODB_RW_LOCKS_USE_ATOMICS +# define IB_ATOMICS_STARTUP_MSG \ + "Mutexes and rw_locks use GCC atomic builtins" +# else /* HAVE_IB_ATOMIC_PTHREAD_T_GCC */ +# define IB_ATOMICS_STARTUP_MSG \ + "Mutexes use GCC atomic builtins, rw_locks do not" +# endif /* HAVE_IB_ATOMIC_PTHREAD_T_GCC */ + /**********************************************************//** Returns the resulting value, ptr is pointer to target, amount is the amount of increment. */ + # define os_atomic_increment(ptr, amount) \ __sync_add_and_fetch(ptr, amount) + # define os_atomic_increment_lint(ptr, amount) \ os_atomic_increment(ptr, amount) + # define os_atomic_increment_ulint(ptr, amount) \ os_atomic_increment(ptr, amount) + /**********************************************************//** Returns the old value of *ptr, atomically sets *ptr to new_val */ + # define os_atomic_test_and_set_byte(ptr, new_val) \ __sync_lock_test_and_set(ptr, new_val) + +#elif defined(HAVE_IB_SOLARIS_ATOMICS) + +#define HAVE_ATOMIC_BUILTINS + /* If not compiling with GCC or GCC doesn't support the atomic intrinsics and running on Solaris >= 10 use Solaris atomics */ -#elif defined(HAVE_SOLARIS_ATOMICS) + #include <atomic.h> + /**********************************************************//** Returns true if swapped, ptr is pointer to target, old_val is value to compare to, new_val is the value to swap in. */ + # define os_compare_and_swap_ulint(ptr, old_val, new_val) \ (atomic_cas_ulong(ptr, old_val, new_val) == old_val) + # define os_compare_and_swap_lint(ptr, old_val, new_val) \ ((lint)atomic_cas_ulong((ulong_t*) ptr, old_val, new_val) == old_val) -# ifdef INNODB_RW_LOCKS_USE_ATOMICS -# if SIZEOF_PTHREAD_T == 4 + +# ifdef HAVE_IB_ATOMIC_PTHREAD_T_SOLARIS +# if SIZEOF_PTHREAD_T == 4 # define os_compare_and_swap_thread_id(ptr, old_val, new_val) \ ((pthread_t)atomic_cas_32(ptr, old_val, new_val) == old_val) # elif SIZEOF_PTHREAD_T == 8 @@ -331,21 +361,35 @@ compare to, new_val is the value to swap # else # error "SIZEOF_PTHREAD_T != 4 or 8" # endif /* SIZEOF_PTHREAD_T CHECK */ -# endif /* INNODB_RW_LOCKS_USE_ATOMICS */ +# define INNODB_RW_LOCKS_USE_ATOMICS +# define IB_ATOMICS_STARTUP_MSG \ + "Mutexes and rw_locks use Solaris atomic functions" +# else /* HAVE_IB_ATOMIC_PTHREAD_T_SOLARIS */ +# define IB_ATOMICS_STARTUP_MSG \ + "Mutexes use Solaris atomic functions, rw_locks do not" +# endif /* HAVE_IB_ATOMIC_PTHREAD_T_SOLARIS */ /**********************************************************//** Returns the resulting value, ptr is pointer to target, amount is the amount of increment. */ + # define os_atomic_increment_lint(ptr, amount) \ atomic_add_long_nv((ulong_t*) ptr, amount) + # define os_atomic_increment_ulint(ptr, amount) \ atomic_add_long_nv(ptr, amount) + /**********************************************************//** Returns the old value of *ptr, atomically sets *ptr to new_val */ + # define os_atomic_test_and_set_byte(ptr, new_val) \ atomic_swap_uchar(ptr, new_val) -/* On Windows, use Windows atomics / interlocked */ + #elif defined(HAVE_WINDOWS_ATOMICS) + +#define HAVE_ATOMIC_BUILTINS + +/* On Windows, use Windows atomics / interlocked */ # ifdef _WIN64 # define win_cmp_and_xchg InterlockedCompareExchange64 # define win_xchg_and_add InterlockedExchangeAdd64 @@ -353,31 +397,46 @@ Returns the old value of *ptr, atomicall # define win_cmp_and_xchg InterlockedCompareExchange # define win_xchg_and_add InterlockedExchangeAdd # endif + /**********************************************************//** Returns true if swapped, ptr is pointer to target, old_val is value to compare to, new_val is the value to swap in. */ + # define os_compare_and_swap_ulint(ptr, old_val, new_val) \ (win_cmp_and_xchg(ptr, new_val, old_val) == old_val) + # define os_compare_and_swap_lint(ptr, old_val, new_val) \ (win_cmp_and_xchg(ptr, new_val, old_val) == old_val) -# ifdef INNODB_RW_LOCKS_USE_ATOMICS -# define os_compare_and_swap_thread_id(ptr, old_val, new_val) \ + +/* windows thread objects can always be passed to windows atomic functions */ +# define os_compare_and_swap_thread_id(ptr, old_val, new_val) \ (InterlockedCompareExchange(ptr, new_val, old_val) == old_val) -# endif /* INNODB_RW_LOCKS_USE_ATOMICS */ +# define INNODB_RW_LOCKS_USE_ATOMICS +# define IB_ATOMICS_STARTUP_MSG \ + "Mutexes and rw_locks use Windows interlocked functions" + /**********************************************************//** Returns the resulting value, ptr is pointer to target, amount is the amount of increment. */ + # define os_atomic_increment_lint(ptr, amount) \ (win_xchg_and_add(ptr, amount) + amount) + # define os_atomic_increment_ulint(ptr, amount) \ ((ulint) (win_xchg_and_add(ptr, amount) + amount)) + /**********************************************************//** Returns the old value of *ptr, atomically sets *ptr to new_val. InterlockedExchange() operates on LONG, and the LONG will be clobbered */ + # define os_atomic_test_and_set_byte(ptr, new_val) \ ((byte) InterlockedExchange(ptr, new_val)) -#endif /* HAVE_GCC_ATOMIC_BUILTINS */ + +#else +# define IB_ATOMICS_STARTUP_MSG \ + "Mutexes and rw_locks use InnoDB's own implementation" +#endif #ifndef UNIV_NONINL #include "os0sync.ic" === modified file 'storage/xtradb/include/page0page.h' --- a/storage/xtradb/include/page0page.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/page0page.h 2010-01-06 12:00:14 +0000 @@ -76,8 +76,11 @@ typedef byte page_header_t; header which are set in a page create */ /*----*/ #define PAGE_LEVEL 26 /* level of the node in an index tree; the - leaf level is the level 0 */ -#define PAGE_INDEX_ID 28 /* index id where the page belongs */ + leaf level is the level 0. This field should + not be written to after page creation. */ +#define PAGE_INDEX_ID 28 /* index id where the page belongs. + This field should not be written to after + page creation. */ #define PAGE_BTR_SEG_LEAF 36 /* file segment header for the leaf pages in a B-tree: defined only on the root page of a B-tree, but not in the root of an ibuf tree */ === modified file 'storage/xtradb/include/page0page.ic' --- a/storage/xtradb/include/page0page.ic 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/page0page.ic 2010-01-06 12:00:14 +0000 @@ -907,7 +907,7 @@ page_get_data_size( /************************************************************//** Allocates a block of memory from the free list of an index page. */ -UNIV_INTERN +UNIV_INLINE void page_mem_alloc_free( /*================*/ === modified file 'storage/xtradb/include/page0zip.h' --- a/storage/xtradb/include/page0zip.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/page0zip.h 2010-01-06 12:00:14 +0000 @@ -127,8 +127,12 @@ page_zip_decompress( /*================*/ page_zip_des_t* page_zip,/*!< in: data, ssize; out: m_start, m_end, m_nonempty, n_blobs */ - page_t* page) /*!< out: uncompressed page, may be trashed */ - __attribute__((nonnull)); + page_t* page, /*!< out: uncompressed page, may be trashed */ + ibool all) /*!< in: TRUE=decompress the whole page; + FALSE=verify but do not copy some + page header fields that should not change + after page creation */ + __attribute__((nonnull(1,2))); #ifdef UNIV_DEBUG /**********************************************************************//** @@ -385,8 +389,8 @@ IMPORTANT: if page_zip_reorganize() is i non-clustered index, the caller must update the insert buffer free bits in the same mini-transaction in such a way that the modification will be redo-logged. -@return TRUE on success, FALSE on failure; page and page_zip will be -left intact on failure. */ +@return TRUE on success, FALSE on failure; page_zip will be left +intact on failure, but page will be overwritten. */ UNIV_INTERN ibool page_zip_reorganize( === modified file 'storage/xtradb/include/pars0pars.h' --- a/storage/xtradb/include/pars0pars.h 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/include/pars0pars.h 2010-01-15 15:58:25 +0000 @@ -583,6 +583,12 @@ pars_info_get_bound_id( pars_info_t* info, /*!< in: info struct */ const char* name); /*!< in: bound id name to find */ +/******************************************************************//** +Release any resources used by the lexer. */ +UNIV_INTERN +void +pars_lexer_close(void); +/*==================*/ /** Extra information supplied for pars_sql(). */ struct pars_info_struct { === modified file 'storage/xtradb/include/rem0cmp.h' --- a/storage/xtradb/include/rem0cmp.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/rem0cmp.h 2010-01-06 12:00:14 +0000 @@ -89,7 +89,7 @@ cmp_dfield_dfield( /*************************************************************//** This function is used to compare a data tuple to a physical record. Only dtuple->n_fields_cmp first fields are taken into account for -the the data tuple! If we denote by n = n_fields_cmp, then rec must +the data tuple! If we denote by n = n_fields_cmp, then rec must have either m >= n fields, or it must differ from dtuple in some of the m fields rec has. If rec has an externally stored field we do not compare it but return with value 0 if such a comparison should be === modified file 'storage/xtradb/include/rem0rec.ic' --- a/storage/xtradb/include/rem0rec.ic 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/rem0rec.ic 2010-01-06 12:00:14 +0000 @@ -65,7 +65,7 @@ most significant bytes and bits are writ - offset_of_this_record) mod 64Ki, where mod is the modulo as a non-negative number; - we can calculate the the offset of the next + we can calculate the offset of the next record with the formula: relative_offset + offset_of_this_record mod UNIV_PAGE_SIZE === modified file 'storage/xtradb/include/row0ins.h' --- a/storage/xtradb/include/row0ins.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/row0ins.h 2010-01-06 12:00:14 +0000 @@ -45,7 +45,7 @@ row_ins_check_foreign_constraint( /*=============================*/ ibool check_ref,/*!< in: TRUE If we want to check that the referenced table is ok, FALSE if we - want to to check the foreign key table */ + want to check the foreign key table */ dict_foreign_t* foreign,/*!< in: foreign constraint; NOTE that the tables mentioned in it must be in the dictionary cache if they exist at all */ === modified file 'storage/xtradb/include/row0mysql.h' --- a/storage/xtradb/include/row0mysql.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/row0mysql.h 2010-01-06 12:00:14 +0000 @@ -177,7 +177,9 @@ row_update_prebuilt_trx( in MySQL handle */ trx_t* trx); /*!< in: transaction handle */ /*********************************************************************//** -Unlocks AUTO_INC type locks that were possibly reserved by a trx. */ +Unlocks AUTO_INC type locks that were possibly reserved by a trx. This +function should be called at the the end of an SQL statement, by the +connection thread that owns the transaction (trx->mysql_thd). */ UNIV_INTERN void row_unlock_table_autoinc_for_mysql( @@ -754,8 +756,6 @@ struct row_prebuilt_struct { store it here so that we can return it to MySQL */ /*----------------------*/ - UT_LIST_NODE_T(row_prebuilt_t) prebuilts; - /*!< list node of table->prebuilts */ ulint magic_n2; /*!< this should be the same as magic_n */ }; === modified file 'storage/xtradb/include/srv0srv.h' --- a/storage/xtradb/include/srv0srv.h 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/include/srv0srv.h 2010-01-15 15:58:25 +0000 @@ -80,6 +80,9 @@ at a time */ #define SRV_AUTO_EXTEND_INCREMENT \ (srv_auto_extend_increment * ((1024 * 1024) / UNIV_PAGE_SIZE)) +/* prototypes for new functions added to ha_innodb.cc */ +ibool innobase_get_slow_log(); + /* This is set to TRUE if the MySQL user has set it in MySQL */ extern ibool srv_lower_case_table_names; @@ -133,8 +136,9 @@ extern ulint* srv_data_file_is_raw_parti extern ibool srv_extra_undoslots; extern ibool srv_fast_recovery; +extern ibool srv_recovery_stats; -extern ibool srv_use_purge_thread; +extern ulint srv_use_purge_thread; extern ibool srv_auto_extend_last_data_file; extern ulint srv_last_file_size_max; @@ -235,12 +239,14 @@ extern ulong srv_replication_delay; extern long long srv_ibuf_max_size; extern ulong srv_ibuf_active_contract; extern ulong srv_ibuf_accel_rate; +extern ulint srv_checkpoint_age_target; extern ulong srv_flush_neighbor_pages; extern ulong srv_enable_unsafe_group_commit; extern ulong srv_read_ahead; extern ulong srv_adaptive_checkpoint; extern ulong srv_expand_import; +extern ulint srv_relax_table_creation; extern ulong srv_extra_rsegments; extern ulong srv_dict_size_limit; @@ -345,10 +351,6 @@ extern ulint srv_buf_pool_flushed; /** Number of buffer pool reads that led to the reading of a disk page */ extern ulint srv_buf_pool_reads; -/** Number of sequential read-aheads */ -extern ulint srv_read_ahead_seq; -/** Number of random read-aheads */ -extern ulint srv_read_ahead_rnd; /** Status variables to be passed to MySQL */ typedef struct export_var_struct export_struc; @@ -428,6 +430,7 @@ enum srv_thread_type { SRV_INSERT, /**< thread flushing the insert buffer to disk */ #endif SRV_PURGE, /* thread purging undo records */ + SRV_PURGE_WORKER, /* thread purging undo records */ SRV_MASTER /**< the master thread, (whose type number must be biggest) */ }; @@ -446,7 +449,7 @@ void srv_init(void); /*==========*/ /*********************************************************************//** -Frees the OS fast mutex created in srv_boot(). */ +Frees the data structures created in srv_init(). */ UNIV_INTERN void srv_free(void); @@ -509,6 +512,13 @@ srv_purge_thread( /*=============*/ void* arg); /* in: a dummy parameter required by os_thread_create */ +/************************************************************************* +The undo purge thread. */ +UNIV_INTERN +os_thread_ret_t +srv_purge_worker_thread( +/*====================*/ + void* arg); /*******************************************************************//** Tells the Innobase server that there has been activity in the database and wakes up the master thread if it is suspended (not sleeping). Used @@ -645,13 +655,13 @@ struct export_var_struct{ #ifdef UNIV_DEBUG ulint innodb_buffer_pool_pages_latched; /*!< Latched pages */ #endif /* UNIV_DEBUG */ - ulint innodb_buffer_pool_read_requests; /*!< buf_pool->n_page_gets */ + ulint innodb_buffer_pool_read_requests; /*!< buf_pool->stat.n_page_gets */ ulint innodb_buffer_pool_reads; /*!< srv_buf_pool_reads */ ulint innodb_buffer_pool_wait_free; /*!< srv_buf_pool_wait_free */ ulint innodb_buffer_pool_pages_flushed; /*!< srv_buf_pool_flushed */ ulint innodb_buffer_pool_write_requests;/*!< srv_buf_pool_write_requests */ - ulint innodb_buffer_pool_read_ahead_seq;/*!< srv_read_ahead_seq */ - ulint innodb_buffer_pool_read_ahead_rnd;/*!< srv_read_ahead_rnd */ + ulint innodb_buffer_pool_read_ahead; /*!< srv_read_ahead */ + ulint innodb_buffer_pool_read_ahead_evicted;/*!< srv_read_ahead evicted*/ ulint innodb_dblwr_pages_written; /*!< srv_dblwr_pages_written */ ulint innodb_dblwr_writes; /*!< srv_dblwr_writes */ ibool innodb_have_atomic_builtins; /*!< HAVE_ATOMIC_BUILTINS */ @@ -663,9 +673,9 @@ struct export_var_struct{ ulint innodb_os_log_pending_writes; /*!< srv_os_log_pending_writes */ ulint innodb_os_log_pending_fsyncs; /*!< fil_n_pending_log_flushes */ ulint innodb_page_size; /*!< UNIV_PAGE_SIZE */ - ulint innodb_pages_created; /*!< buf_pool->n_pages_created */ - ulint innodb_pages_read; /*!< buf_pool->n_pages_read */ - ulint innodb_pages_written; /*!< buf_pool->n_pages_written */ + ulint innodb_pages_created; /*!< buf_pool->stat.n_pages_created */ + ulint innodb_pages_read; /*!< buf_pool->stat.n_pages_read */ + ulint innodb_pages_written; /*!< buf_pool->stat.n_pages_written */ ulint innodb_row_lock_waits; /*!< srv_n_lock_wait_count */ ulint innodb_row_lock_current_waits; /*!< srv_n_lock_wait_current_count */ ib_int64_t innodb_row_lock_time; /*!< srv_n_lock_wait_time === modified file 'storage/xtradb/include/sync0rw.h' --- a/storage/xtradb/include/sync0rw.h 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/include/sync0rw.h 2010-01-15 15:58:25 +0000 @@ -120,7 +120,7 @@ is necessary only if the memory block co # endif /* UNIV_SYNC_DEBUG */ #else /* UNIV_DEBUG */ # define rw_lock_create(L, level) \ - rw_lock_create_func((L), __FILE__, __LINE__) + rw_lock_create_func((L), #L, NULL, 0) #endif /* UNIV_DEBUG */ /******************************************************************//** @@ -137,8 +137,8 @@ rw_lock_create_func( # ifdef UNIV_SYNC_DEBUG ulint level, /*!< in: level */ # endif /* UNIV_SYNC_DEBUG */ - const char* cmutex_name, /*!< in: mutex name */ #endif /* UNIV_DEBUG */ + const char* cmutex_name, /*!< in: mutex name */ const char* cfile_name, /*!< in: file name where created */ ulint cline); /*!< in: file line where created */ /******************************************************************//** @@ -540,7 +540,8 @@ struct rw_lock_struct { ulint level; /*!< Level in the global latching order. */ #endif /* UNIV_SYNC_DEBUG */ ulint count_os_wait; /*!< Count of os_waits. May not be accurate */ - const char* cfile_name;/*!< File name where lock created */ + //const char* cfile_name;/*!< File name where lock created */ + const char* lock_name;/*!< lock name */ /* last s-lock file/line is not guaranteed to be correct */ const char* last_s_file_name;/*!< File name where last s-locked */ const char* last_x_file_name;/*!< File name where last x-locked */ @@ -551,7 +552,7 @@ struct rw_lock_struct { are at the start of this struct, thus we can peek this field without causing much memory bus traffic */ - unsigned cline:14; /*!< Line where created */ + //unsigned cline:14; /*!< Line where created */ unsigned last_s_line:14; /*!< Line number where last time s-locked */ unsigned last_x_line:14; /*!< Line number where last time x-locked */ ulint magic_n; /*!< RW_LOCK_MAGIC_N */ === modified file 'storage/xtradb/include/sync0sync.h' --- a/storage/xtradb/include/sync0sync.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/sync0sync.h 2010-01-06 12:00:14 +0000 @@ -80,7 +80,7 @@ necessary only if the memory block conta # endif #else # define mutex_create(M, level) \ - mutex_create_func((M), __FILE__, __LINE__) + mutex_create_func((M), #M, NULL, 0) #endif /******************************************************************//** @@ -93,8 +93,8 @@ void mutex_create_func( /*==============*/ mutex_t* mutex, /*!< in: pointer to memory */ -#ifdef UNIV_DEBUG const char* cmutex_name, /*!< in: mutex name */ +#ifdef UNIV_DEBUG # ifdef UNIV_SYNC_DEBUG ulint level, /*!< in: level */ # endif /* UNIV_SYNC_DEBUG */ @@ -513,7 +513,7 @@ struct mutex_struct { os_fast_mutex; /*!< We use this OS mutex in place of lock_word when atomic operations are not enabled */ #endif - ulint waiters; /*!< This ulint is set to 1 if there are (or + volatile ulint waiters; /*!< This ulint is set to 1 if there are (or may be) threads waiting in the global wait array for this mutex to be released. Otherwise, this is 0. */ @@ -524,9 +524,9 @@ struct mutex_struct { ulint line; /*!< Line where the mutex was locked */ ulint level; /*!< Level in the global latching order */ #endif /* UNIV_SYNC_DEBUG */ +#ifdef UNIV_DEBUG const char* cfile_name;/*!< File name where mutex created */ ulint cline; /*!< Line where created */ -#ifdef UNIV_DEBUG os_thread_id_t thread_id; /*!< The thread id of the thread which locked the mutex. */ ulint magic_n; /*!< MUTEX_MAGIC_N */ @@ -541,9 +541,9 @@ struct mutex_struct { ulong count_os_yield; /*!< count of os_wait */ ulonglong lspent_time; /*!< mutex os_wait timer msec */ ulonglong lmax_spent_time;/*!< mutex os_wait timer msec */ - const char* cmutex_name; /*!< mutex name */ ulint mutex_type; /*!< 0=usual mutex, 1=rw_lock mutex */ #endif /* UNIV_DEBUG */ + const char* cmutex_name; /*!< mutex name */ }; /** The global array of wait cells for implementation of the databases own === modified file 'storage/xtradb/include/thr0loc.h' --- a/storage/xtradb/include/thr0loc.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/thr0loc.h 2010-01-06 12:00:14 +0000 @@ -39,6 +39,12 @@ UNIV_INTERN void thr_local_init(void); /*================*/ + /****************************************************************//** +Close the thread local storage module. */ +UNIV_INTERN +void +thr_local_close(void); +/*=================*/ /*******************************************************************//** Creates a local storage struct for the calling new thread. */ UNIV_INTERN === modified file 'storage/xtradb/include/trx0i_s.h' --- a/storage/xtradb/include/trx0i_s.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/trx0i_s.h 2010-01-06 12:00:14 +0000 @@ -141,6 +141,13 @@ void trx_i_s_cache_init( /*===============*/ trx_i_s_cache_t* cache); /*!< out: cache to init */ +/*******************************************************************//** +Free the INFORMATION SCHEMA trx related cache. */ +UNIV_INTERN +void +trx_i_s_cache_free( +/*===============*/ + trx_i_s_cache_t* cache); /*!< in/out: cache to free */ /*******************************************************************//** Issue a shared/read lock on the tables cache. */ === modified file 'storage/xtradb/include/trx0purge.h' --- a/storage/xtradb/include/trx0purge.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/trx0purge.h 2010-01-06 12:00:14 +0000 @@ -71,6 +71,12 @@ void trx_purge_sys_create(void); /*======================*/ /********************************************************************//** +Frees the global purge system control structure. */ +UNIV_INTERN +void +trx_purge_sys_close(void); +/*======================*/ +/************************************************************************ Adds the update undo log as the first log in the history list. Removes the update undo log segment from the rseg slot if it is too big for reuse. */ UNIV_INTERN @@ -108,6 +114,25 @@ UNIV_INTERN ulint trx_purge(void); /*===========*/ +/********************************************************************** +This function runs a purge worker batch */ +UNIV_INTERN +void +trx_purge_worker( +/*=============*/ + ulint worker_id); +/********************************************************************** +This function waits the event for worker batch */ +UNIV_INTERN +void +trx_purge_worker_wait(void); +/*========================*/ +/********************************************************************** +This function wakes the waiting worker batch */ +UNIV_INTERN +void +trx_purge_worker_wake(void); +/*========================*/ /******************************************************************//** Prints information of the purge system to stderr. */ UNIV_INTERN @@ -125,6 +150,11 @@ struct trx_purge_struct{ of the trx system and it never ends */ que_t* query; /*!< The query graph which will do the parallelized purge operation */ + ulint n_worker; + os_event_t worker_event; + sess_t** sess_arr; + trx_t** trx_arr; + que_t** query_arr; rw_lock_t latch; /*!< The latch protecting the purge view. A purge operation must acquire an x-latch here for the instant at which === modified file 'storage/xtradb/include/trx0rec.h' --- a/storage/xtradb/include/trx0rec.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/trx0rec.h 2010-01-06 12:00:14 +0000 @@ -44,8 +44,8 @@ UNIV_INLINE trx_undo_rec_t* trx_undo_rec_copy( /*==============*/ - trx_undo_rec_t* undo_rec, /*!< in: undo log record */ - mem_heap_t* heap); /*!< in: heap where copied */ + const trx_undo_rec_t* undo_rec, /*!< in: undo log record */ + mem_heap_t* heap); /*!< in: heap where copied */ /**********************************************************************//** Reads the undo log record type. @return record type */ === modified file 'storage/xtradb/include/trx0rec.ic' --- a/storage/xtradb/include/trx0rec.ic 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/trx0rec.ic 2010-01-06 12:00:14 +0000 @@ -100,8 +100,8 @@ UNIV_INLINE trx_undo_rec_t* trx_undo_rec_copy( /*==============*/ - trx_undo_rec_t* undo_rec, /*!< in: undo log record */ - mem_heap_t* heap) /*!< in: heap where copied */ + const trx_undo_rec_t* undo_rec, /*!< in: undo log record */ + mem_heap_t* heap) /*!< in: heap where copied */ { ulint len; === modified file 'storage/xtradb/include/trx0roll.h' --- a/storage/xtradb/include/trx0roll.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/trx0roll.h 2010-01-06 12:00:14 +0000 @@ -133,6 +133,17 @@ trx_rollback( Rollback or clean up any incomplete transactions which were encountered in crash recovery. If the transaction already was committed, then we clean up a possible insert undo log. If the +transaction was not yet committed, then we roll it back. */ +UNIV_INTERN +void +trx_rollback_or_clean_recovered( +/*============================*/ + ibool all); /*!< in: FALSE=roll back dictionary transactions; + TRUE=roll back all non-PREPARED transactions */ +/*******************************************************************//** +Rollback or clean up any incomplete transactions which were +encountered in crash recovery. If the transaction already was +committed, then we clean up a possible insert undo log. If the transaction was not yet committed, then we roll it back. Note: this is done in a background thread. @return a dummy parameter */ @@ -208,9 +219,9 @@ int trx_general_rollback_for_mysql( /*===========================*/ trx_t* trx, /*!< in: transaction handle */ - ibool partial,/*!< in: TRUE if partial rollback requested */ trx_savept_t* savept);/*!< in: pointer to savepoint undo number, if - partial rollback requested */ + partial rollback requested, or NULL for + complete rollback */ /*******************************************************************//** Rolls back a transaction back to a named savepoint. Modifications after the savepoint are undone but InnoDB does NOT release the corresponding locks === modified file 'storage/xtradb/include/trx0rseg.h' --- a/storage/xtradb/include/trx0rseg.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/trx0rseg.h 2010-01-06 12:00:14 +0000 @@ -125,6 +125,13 @@ trx_rseg_create( ulint max_size, /*!< in: max size in pages */ ulint* id, /*!< out: rseg id */ mtr_t* mtr); /*!< in: mtr */ +/*************************************************************************** +Free's an instance of the rollback segment in memory. */ +UNIV_INTERN +void +trx_rseg_mem_free( +/*==============*/ + trx_rseg_t* rseg); /* in, own: instance to free */ /* Real max value may be 4076 in usual. But reserve 4 slot for safety or etc... */ === modified file 'storage/xtradb/include/trx0sys.h' --- a/storage/xtradb/include/trx0sys.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/trx0sys.h 2010-01-06 12:00:14 +0000 @@ -344,6 +344,12 @@ void trx_sys_file_format_tag_init(void); /*==============================*/ /*****************************************************************//** +Shutdown/Close the transaction system. */ +UNIV_INTERN +void +trx_sys_close(void); +/*===============*/ +/*****************************************************************//** Get the name representation of the file format from its id. @return pointer to the name */ UNIV_INTERN === modified file 'storage/xtradb/include/trx0sys.ic' --- a/storage/xtradb/include/trx0sys.ic 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/trx0sys.ic 2010-01-06 12:00:14 +0000 @@ -34,11 +34,11 @@ typedef byte trx_sysf_rseg_t; /* Rollback segment specification slot offsets */ /*-------------------------------------------------------------*/ -#define TRX_SYS_RSEG_SPACE 0 /* space where the the segment +#define TRX_SYS_RSEG_SPACE 0 /* space where the segment header is placed; starting with MySQL/InnoDB 5.1.7, this is UNIV_UNDEFINED if the slot is unused */ -#define TRX_SYS_RSEG_PAGE_NO 4 /* page number where the the segment +#define TRX_SYS_RSEG_PAGE_NO 4 /* page number where the segment header is placed; this is FIL_NULL if the slot is unused */ /*-------------------------------------------------------------*/ === modified file 'storage/xtradb/include/trx0trx.h' --- a/storage/xtradb/include/trx0trx.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/trx0trx.h 2010-01-06 12:00:14 +0000 @@ -179,7 +179,7 @@ trx_commit_off_kernel( /****************************************************************//** Cleans up a transaction at database startup. The cleanup is needed if the transaction already got to the middle of a commit when the database -crashed, andf we cannot roll it back. */ +crashed, and we cannot roll it back. */ UNIV_INTERN void trx_cleanup_at_db_startup( @@ -360,7 +360,7 @@ enum trx_dict_op { operation modes in crash recovery. */ TRX_DICT_OP_TABLE = 1, /** The transaction is creating or dropping an index in an - existing table. In crash recovery, the the data dictionary + existing table. In crash recovery, the data dictionary must be locked, but the table must not be dropped. */ TRX_DICT_OP_INDEX = 2 }; @@ -729,6 +729,17 @@ struct trx_struct{ /*------------------------------*/ char detailed_error[256]; /*!< detailed error message for last error, or empty. */ + /*------------------------------*/ + ulint io_reads; + ib_uint64_t io_read; + ulint io_reads_wait_timer; + ib_uint64_t lock_que_wait_ustarted; + ulint lock_que_wait_timer; + ulint innodb_que_wait_timer; + ulint distinct_page_access; +#define DPAH_SIZE 8192 + byte* distinct_page_access_hash; + ibool take_stats; }; #define TRX_MAX_N_THREADS 32 /* maximum number of === modified file 'storage/xtradb/include/trx0types.h' --- a/storage/xtradb/include/trx0types.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/trx0types.h 2010-01-06 12:00:14 +0000 @@ -70,7 +70,7 @@ typedef struct trx_named_savept_struct t enum trx_rb_ctx { RB_NONE = 0, /*!< no rollback */ RB_NORMAL, /*!< normal rollback */ - RB_RECOVERY, /*!< rolling back an incomplete transaction, + RB_RECOVERY /*!< rolling back an incomplete transaction, in crash recovery */ }; === modified file 'storage/xtradb/include/trx0undo.h' --- a/storage/xtradb/include/trx0undo.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/trx0undo.h 2010-01-06 12:00:14 +0000 @@ -333,6 +333,13 @@ trx_undo_parse_discard_latest( byte* end_ptr,/*!< in: buffer end */ page_t* page, /*!< in: page or NULL */ mtr_t* mtr); /*!< in: mtr or NULL */ +/************************************************************************ +Frees an undo log memory copy. */ +UNIV_INTERN +void +trx_undo_mem_free( +/*==============*/ + trx_undo_t* undo); /* in: the undo object to be freed */ /* Types of an undo log segment */ #define TRX_UNDO_INSERT 1 /* contains undo entries for inserts */ === modified file 'storage/xtradb/include/univ.i' --- a/storage/xtradb/include/univ.i 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/include/univ.i 2010-01-15 15:58:25 +0000 @@ -46,12 +46,12 @@ Created 1/20/1994 Heikki Tuuri #define INNODB_VERSION_MAJOR 1 #define INNODB_VERSION_MINOR 0 -#define INNODB_VERSION_BUGFIX 4 -#define PERCONA_INNODB_VERSION 8 +#define INNODB_VERSION_BUGFIX 6 +#define PERCONA_INNODB_VERSION 9 /* The following is the InnoDB version as shown in SELECT plugin_version FROM information_schema.plugins; -calculated in in make_version_string() in sql/sql_show.cc like this: +calculated in make_version_string() in sql/sql_show.cc like this: "version >> 8" . "version & 0xff" because the version is shown with only one dot, we skip the last component, i.e. we show M.N.P as M.N */ @@ -59,13 +59,14 @@ component, i.e. we show M.N.P as M.N */ (INNODB_VERSION_MAJOR << 8 | INNODB_VERSION_MINOR) /* auxiliary macros to help creating the version as string */ -#define __INNODB_VERSION(a, b, c, d) (#a "." #b "." #c "-" #d) -#define _INNODB_VERSION(a, b, c, d) __INNODB_VERSION(a, b, c, d) +#define __INNODB_VERSION(a, b, c, d) (#a "." #b "." #c "-" #d) +#define _INNODB_VERSION(a, b, c, d) __INNODB_VERSION(a, b, c, d) + #define INNODB_VERSION_STR \ _INNODB_VERSION(INNODB_VERSION_MAJOR, \ INNODB_VERSION_MINOR, \ - INNODB_VERSION_BUGFIX, \ + INNODB_VERSION_BUGFIX, \ PERCONA_INNODB_VERSION) #define REFMAN "http://dev.mysql.com/doc/refman/5.1/en/" @@ -80,17 +81,25 @@ the virtual method table (vtable) in GCC # define ha_innobase ha_innodb #endif /* MYSQL_DYNAMIC_PLUGIN */ +/* if any of the following macros is defined at this point this means +that the code from the "right" plug.in was executed and we do not +need to include ut0auxconf.h which would either define the same macros +or will be empty */ +#if !defined(HAVE_IB_GCC_ATOMIC_BUILTINS) \ + && !defined(HAVE_IB_ATOMIC_PTHREAD_T_GCC) \ + && !defined(HAVE_IB_SOLARIS_ATOMICS) \ + && !defined(HAVE_IB_ATOMIC_PTHREAD_T_SOLARIS) \ + && !defined(SIZEOF_PTHREAD_T) \ + && !defined(HAVE_IB_PAUSE_INSTRUCTION) +# include "ut0auxconf.h" +#endif + #if (defined(WIN32) || defined(_WIN32) || defined(WIN64) || defined(_WIN64)) && !defined(MYSQL_SERVER) && !defined(__WIN__) # undef __WIN__ # define __WIN__ # include <windows.h> -# if defined(HAVE_WINDOWS_ATOMICS) -/* If atomics are defined we use them in InnoDB mutex implementation */ -# define HAVE_ATOMIC_BUILTINS -# endif /* HAVE_WINDOWS_ATOMICS */ - # ifdef _NT_ # define __NT__ # endif @@ -113,45 +122,17 @@ if we are compiling on Windows. */ # include <sys/mman.h> /* mmap() for os0proc.c */ # endif -# undef PACKAGE -# undef VERSION - /* Include the header file generated by GNU autoconf */ # ifndef __WIN__ -#ifndef UNIV_HOTBACKUP -# include "config.h" -#endif /* UNIV_HOTBACKUP */ +# ifndef UNIV_HOTBACKUP +# include "config.h" +# endif /* UNIV_HOTBACKUP */ # endif # ifdef HAVE_SCHED_H # include <sched.h> # endif -# if defined(HAVE_GCC_ATOMIC_BUILTINS) || defined(HAVE_SOLARIS_ATOMICS) \ - || defined(HAVE_WINDOWS_ATOMICS) -/* If atomics are defined we use them in InnoDB mutex implementation */ -# define HAVE_ATOMIC_BUILTINS -# endif /* (HAVE_GCC_ATOMIC_BUILTINS) || (HAVE_SOLARIS_ATOMICS) - || (HAVE_WINDOWS_ATOMICS) */ - -/* For InnoDB rw_locks to work with atomics we need the thread_id -to be no more than machine word wide. The following enables using -atomics for InnoDB rw_locks where these conditions are met. */ -#ifdef HAVE_ATOMIC_BUILTINS -/* if HAVE_ATOMIC_PTHREAD_T is defined at this point that means that -the code from plug.in has defined it and we do not need to include -ut0auxconf.h which would either define HAVE_ATOMIC_PTHREAD_T or will -be empty */ -# ifndef HAVE_ATOMIC_PTHREAD_T -# include "ut0auxconf.h" -# endif /* HAVE_ATOMIC_PTHREAD_T */ -/* now HAVE_ATOMIC_PTHREAD_T is eventually defined either by plug.in or -from Makefile.in->ut0auxconf.h */ -# ifdef HAVE_ATOMIC_PTHREAD_T -# define INNODB_RW_LOCKS_USE_ATOMICS -# endif /* HAVE_ATOMIC_PTHREAD_T */ -#endif /* HAVE_ATOMIC_BUILTINS */ - /* We only try to do explicit inlining of functions with gcc and Sun Studio */ @@ -198,12 +179,18 @@ command. Not tested on Windows. */ debugging without UNIV_DEBUG */ #define UNIV_DEBUG /* Enable ut_ad() assertions and disable UNIV_INLINE */ +#define UNIV_DEBUG_LOCK_VALIDATE /* Enable + ut_ad(lock_rec_validate_page()) + assertions. */ #define UNIV_DEBUG_FILE_ACCESSES /* Debug .ibd file access (field file_page_was_freed in buf_page_t) */ #define UNIV_LRU_DEBUG /* debug the buffer pool LRU */ #define UNIV_HASH_DEBUG /* debug HASH_ macros */ #define UNIV_LIST_DEBUG /* debug UT_LIST_ macros */ +#define UNIV_LOG_LSN_DEBUG /* write LSN to the redo log; +this will break redo log file compatibility, but it may be useful when +debugging redo log application problems. */ #define UNIV_MEM_DEBUG /* detect memory leaks etc */ #define UNIV_IBUF_DEBUG /* debug the insert buffer */ #define UNIV_IBUF_COUNT_DEBUG /* debug the insert buffer; @@ -253,7 +240,7 @@ by one. */ /* Linkage specifier for non-static InnoDB symbols (variables and functions) that are only referenced from within InnoDB, not from MySQL */ -#if defined(__GNUC__) && (__GNUC__ >= 4) && !defined(UNIV_HOTBACKUP) +#if defined(__GNUC__) && (__GNUC__ >= 4) || defined(__INTEL_COMPILER) # define UNIV_INTERN __attribute__((visibility ("hidden"))) #else # define UNIV_INTERN @@ -410,7 +397,9 @@ it is read. */ /* Minimize cache-miss latency by moving data at addr into a cache before it is read or written. */ # define UNIV_PREFETCH_RW(addr) __builtin_prefetch(addr, 1, 3) -#elif defined(__SUNPRO_C) || defined(__SUNPRO_CC) +/* Sun Studio includes sun_prefetch.h as of version 5.9 */ +#elif (defined(__SUNPRO_C) && __SUNPRO_C >= 0x590) \ + || (defined(__SUNPRO_CC) && __SUNPRO_CC >= 0x590) # include <sun_prefetch.h> #if __SUNPRO_C >= 0x550 # undef UNIV_INTERN === modified file 'storage/xtradb/include/usr0sess.h' --- a/storage/xtradb/include/usr0sess.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/usr0sess.h 2010-01-06 12:00:14 +0000 @@ -44,14 +44,12 @@ sess_t* sess_open(void); /*============*/ /*********************************************************************//** -Closes a session, freeing the memory occupied by it, if it is in a state -where it should be closed. -@return TRUE if closed */ +Closes a session, freeing the memory occupied by it. */ UNIV_INTERN -ibool -sess_try_close( -/*===========*/ - sess_t* sess); /*!< in, own: session object */ +void +sess_close( +/*=======*/ + sess_t* sess); /* in, own: session object */ /* The session handle. All fields are protected by the kernel mutex */ struct sess_struct{ === modified file 'storage/xtradb/include/ut0auxconf.h' --- a/storage/xtradb/include/ut0auxconf.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/ut0auxconf.h 2010-01-06 12:00:14 +0000 @@ -1,14 +1,14 @@ /* Do not remove this file even though it is empty. This file is included in univ.i and will cause compilation failure if not present. -A custom check has been added in the generated +A custom checks have been added in the generated storage/innobase/Makefile.in that is shipped with the InnoDB Plugin -source archive. This check tries to compile a test program and if -successful then adds "#define HAVE_ATOMIC_PTHREAD_T" to this file. -This is a hack that has been developed in order to check for pthread_t -atomicity without the need to regenerate the ./configure script that is +source archive. These checks eventually define some macros and put +them in this file. +This is a hack that has been developed in order to deploy new compile +time checks without the need to regenerate the ./configure script that is distributed in the MySQL 5.1 official source archives. If by any chance Makefile.in and ./configure are regenerated and thus -the hack from Makefile.in wiped away then the "real" check from plug.in +the hack from Makefile.in wiped away then the "real" checks from plug.in will take over. */ === modified file 'storage/xtradb/include/ut0byte.h' --- a/storage/xtradb/include/ut0byte.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/ut0byte.h 2010-01-06 12:00:14 +0000 @@ -219,8 +219,8 @@ UNIV_INLINE void* ut_align( /*=====*/ - void* ptr, /*!< in: pointer */ - ulint align_no); /*!< in: align by this number */ + const void* ptr, /*!< in: pointer */ + ulint align_no); /*!< in: align by this number */ /*********************************************************//** The following function rounds down a pointer to the nearest aligned address. === modified file 'storage/xtradb/include/ut0byte.ic' --- a/storage/xtradb/include/ut0byte.ic 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/ut0byte.ic 2010-01-06 12:00:14 +0000 @@ -319,8 +319,8 @@ UNIV_INLINE void* ut_align( /*=====*/ - void* ptr, /*!< in: pointer */ - ulint align_no) /*!< in: align by this number */ + const void* ptr, /*!< in: pointer */ + ulint align_no) /*!< in: align by this number */ { ut_ad(align_no > 0); ut_ad(((align_no - 1) & align_no) == 0); === modified file 'storage/xtradb/include/ut0ut.h' --- a/storage/xtradb/include/ut0ut.h 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/include/ut0ut.h 2010-01-06 12:00:14 +0000 @@ -34,6 +34,11 @@ Created 1/20/1994 Heikki Tuuri #define ut0ut_h #include "univ.i" + +#ifndef UNIV_HOTBACKUP +# include "os0sync.h" /* for HAVE_ATOMIC_BUILTINS */ +#endif /* UNIV_HOTBACKUP */ + #include <time.h> #ifndef MYSQL_SERVER #include <ctype.h> @@ -47,7 +52,8 @@ Created 1/20/1994 Heikki Tuuri /** Time stamp */ typedef time_t ib_time_t; -#if defined(IB_HAVE_PAUSE_INSTRUCTION) +#ifndef UNIV_HOTBACKUP +#if defined(HAVE_IB_PAUSE_INSTRUCTION) # ifdef WIN32 /* In the Win32 API, the x86 PAUSE instruction is executed by calling the YieldProcessor macro defined in WinNT.h. It is a CPU architecture- @@ -84,6 +90,7 @@ do { \ os_thread_sleep(2000 /* 2 ms */); \ } \ } while (0) +#endif /* !UNIV_HOTBACKUP */ /********************************************************//** Gets the high 32 bits in a ulint. That is makes a shift >> 32, @@ -216,6 +223,7 @@ UNIV_INTERN ib_time_t ut_time(void); /*=========*/ +#ifndef UNIV_HOTBACKUP /**********************************************************//** Returns system time. Upon successful completion, the value 0 is returned; otherwise the @@ -239,6 +247,16 @@ ullint ut_time_us( /*=======*/ ullint* tloc); /*!< out: us since epoch, if non-NULL */ +/**********************************************************//** +Returns the number of milliseconds since some epoch. The +value may wrap around. It should only be used for heuristic +purposes. +@return ms since epoch */ +UNIV_INTERN +ulint +ut_time_ms(void); +/*============*/ +#endif /* !UNIV_HOTBACKUP */ /**********************************************************//** Returns the difference of two times in seconds. === modified file 'storage/xtradb/lock/lock0lock.c' --- a/storage/xtradb/lock/lock0lock.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/lock/lock0lock.c 2010-01-06 12:00:14 +0000 @@ -214,7 +214,7 @@ a waiting s-lock request on the next rec by a read cursor moving in the ascending order in the index, we cannot do the insert immediately, because when we finally commit our transaction, the read cursor should see also the new inserted record. So we should -move the read cursor backward from the the next record for it to pass over +move the read cursor backward from the next record for it to pass over the new inserted record. This move backward may be too cumbersome to implement. If we in this situation just enqueue a second x-lock request for our transaction on the next record, then the deadlock mechanism @@ -360,10 +360,9 @@ ibool lock_rec_validate_page( /*===================*/ ulint space, /*!< in: space id */ + ulint zip_size,/*!< in: compressed page size in bytes + or 0 for uncompressed pages */ ulint page_no);/*!< in: page number */ - -/* Define the following in order to enable lock_rec_validate_page() checks. */ -# undef UNIV_DEBUG_LOCK_VALIDATE #endif /* UNIV_DEBUG */ /* The lock system */ @@ -579,6 +578,23 @@ lock_sys_create( } /*********************************************************************//** +Closes the lock system at database shutdown. */ +UNIV_INTERN +void +lock_sys_close(void) +/*================*/ +{ + if (lock_latest_err_file != NULL) { + fclose(lock_latest_err_file); + lock_latest_err_file = NULL; + } + + hash_table_free(lock_sys->rec_hash); + mem_free(lock_sys); + lock_sys = NULL; +} + +/*********************************************************************//** Gets the size of a lock struct. @return size in bytes */ UNIV_INTERN @@ -1739,6 +1755,8 @@ lock_rec_enqueue_waiting( { lock_t* lock; trx_t* trx; + ulint sec; + ulint ms; ut_ad(mutex_own(&kernel_mutex)); @@ -1797,6 +1815,10 @@ lock_rec_enqueue_waiting( trx->que_state = TRX_QUE_LOCK_WAIT; trx->was_chosen_as_deadlock_victim = FALSE; trx->wait_started = time(NULL); + if (innobase_get_slow_log() && trx->take_stats) { + ut_usectime(&sec, &ms); + trx->lock_que_wait_ustarted = (ib_uint64_t)sec * 1000000 + ms; + } ut_a(que_thr_stop(thr)); @@ -2622,6 +2644,7 @@ lock_move_reorganize_page( #ifdef UNIV_DEBUG_LOCK_VALIDATE ut_ad(lock_rec_validate_page(buf_block_get_space(block), + buf_block_get_zip_size(block), buf_block_get_page_no(block))); #endif } @@ -2711,8 +2734,10 @@ lock_move_rec_list_end( #ifdef UNIV_DEBUG_LOCK_VALIDATE ut_ad(lock_rec_validate_page(buf_block_get_space(block), + buf_block_get_zip_size(block), buf_block_get_page_no(block))); ut_ad(lock_rec_validate_page(buf_block_get_space(new_block), + buf_block_get_zip_size(block), buf_block_get_page_no(new_block))); #endif } @@ -2822,6 +2847,7 @@ lock_move_rec_list_start( #ifdef UNIV_DEBUG_LOCK_VALIDATE ut_ad(lock_rec_validate_page(buf_block_get_space(block), + buf_block_get_zip_size(block), buf_block_get_page_no(block))); #endif } @@ -3574,7 +3600,8 @@ lock_table_remove_low( and lock_grant()). Therefore it can be empty and we need to check for that. */ - if (!ib_vector_is_empty(trx->autoinc_locks)) { + if (!lock_get_wait(lock) + && !ib_vector_is_empty(trx->autoinc_locks)) { lock_t* autoinc_lock; autoinc_lock = ib_vector_pop(trx->autoinc_locks); @@ -3607,6 +3634,8 @@ lock_table_enqueue_waiting( { lock_t* lock; trx_t* trx; + ulint sec; + ulint ms; ut_ad(mutex_own(&kernel_mutex)); @@ -3647,8 +3676,10 @@ lock_table_enqueue_waiting( if (lock_deadlock_occurs(lock, trx)) { - lock_reset_lock_and_trx_wait(lock); + /* The order here is important, we don't want to + lose the state of the lock before calling remove. */ lock_table_remove_low(lock); + lock_reset_lock_and_trx_wait(lock); return(DB_DEADLOCK); } @@ -3660,6 +3691,10 @@ lock_table_enqueue_waiting( return(DB_SUCCESS); } + if (innobase_get_slow_log() && trx->take_stats) { + ut_usectime(&sec, &ms); + trx->lock_que_wait_ustarted = (ib_uint64_t)sec * 1000000 + ms; + } trx->que_state = TRX_QUE_LOCK_WAIT; trx->was_chosen_as_deadlock_victim = FALSE; trx->wait_started = time(NULL); @@ -4627,6 +4662,10 @@ lock_rec_queue_validate( next function call: we have to release lock table mutex to obey the latching order */ + /* If this thread is holding the file space latch + (fil_space_t::latch), the following check WILL break + latching order and may cause a deadlock of threads. */ + impl_trx = lock_sec_rec_some_has_impl_off_kernel( rec, index, offsets); @@ -4684,6 +4723,8 @@ ibool lock_rec_validate_page( /*===================*/ ulint space, /*!< in: space id */ + ulint zip_size,/*!< in: compressed page size in bytes + or 0 for uncompressed pages */ ulint page_no)/*!< in: page number */ { dict_index_t* index; @@ -4694,7 +4735,6 @@ lock_rec_validate_page( ulint nth_lock = 0; ulint nth_bit = 0; ulint i; - ulint zip_size; mtr_t mtr; mem_heap_t* heap = NULL; ulint offsets_[REC_OFFS_NORMAL_SIZE]; @@ -4705,7 +4745,6 @@ lock_rec_validate_page( mtr_start(&mtr); - zip_size = fil_space_get_zip_size(space); ut_ad(zip_size != ULINT_UNDEFINED); block = buf_page_get(space, zip_size, page_no, RW_X_LATCH, &mtr); buf_block_dbg_add_level(block, SYNC_NO_ORDER_CHECK); @@ -4750,6 +4789,11 @@ loop: lock_mutex_exit_kernel(); + /* If this thread is holding the file space + latch (fil_space_t::latch), the following + check WILL break the latching order and may + cause a deadlock of threads. */ + lock_rec_queue_validate(block, rec, index, offsets); lock_mutex_enter_kernel(); @@ -4840,7 +4884,9 @@ lock_validate(void) lock_mutex_exit_kernel(); - lock_rec_validate_page(space, page_no); + lock_rec_validate_page(space, + fil_space_get_zip_size(space), + page_no); lock_mutex_enter_kernel(); @@ -5364,6 +5410,20 @@ lock_release_autoinc_last_lock( } /*******************************************************************//** +Check if a transaction holds any autoinc locks. +@return TRUE if the transaction holds any AUTOINC locks. */ +UNIV_INTERN +ibool +lock_trx_holds_autoinc_locks( +/*=========================*/ + const trx_t* trx) /*!< in: transaction */ +{ + ut_a(trx->autoinc_locks != NULL); + + return(!ib_vector_is_empty(trx->autoinc_locks)); +} + +/*******************************************************************//** Release all the transaction's autoinc locks. */ UNIV_INTERN void === modified file 'storage/xtradb/log/log0log.c' --- a/storage/xtradb/log/log0log.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/log/log0log.c 2010-01-06 12:00:14 +0000 @@ -241,6 +241,7 @@ log_reserve_and_open( ut_a(len < log->buf_size / 2); loop: mutex_enter(&(log->mutex)); + ut_ad(!recv_no_log_write); /* Calculate an upper limit for the space the string may take in the log buffer */ @@ -309,6 +310,7 @@ log_write_low( ut_ad(mutex_own(&(log->mutex))); part_loop: + ut_ad(!recv_no_log_write); /* Calculate a part length */ data_len = (log->buf_free % OS_FILE_LOG_BLOCK_SIZE) + str_len; @@ -362,6 +364,33 @@ part_loop: } /************************************************************//** +*/ +UNIV_INLINE +ulint +log_max_modified_age_async() +{ + if (srv_checkpoint_age_target) { + return(ut_min(log_sys->max_modified_age_async, + srv_checkpoint_age_target + - srv_checkpoint_age_target / 8)); + } else { + return(log_sys->max_modified_age_async); + } +} + +UNIV_INLINE +ulint +log_max_checkpoint_age_async() +{ + if (srv_checkpoint_age_target) { + return(ut_min(log_sys->max_checkpoint_age_async, + srv_checkpoint_age_target)); + } else { + return(log_sys->max_checkpoint_age_async); + } +} + +/************************************************************//** Closes the log. @return lsn */ UNIV_INTERN @@ -377,6 +406,7 @@ log_close(void) ib_uint64_t checkpoint_age; ut_ad(mutex_own(&(log->mutex))); + ut_ad(!recv_no_log_write); lsn = log->lsn; @@ -429,7 +459,7 @@ log_close(void) } } - if (checkpoint_age <= log->max_modified_age_async) { + if (checkpoint_age <= log_max_modified_age_async()) { goto function_exit; } @@ -437,8 +467,8 @@ log_close(void) oldest_lsn = buf_pool_get_oldest_modification(); if (!oldest_lsn - || lsn - oldest_lsn > log->max_modified_age_async - || checkpoint_age > log->max_checkpoint_age_async) { + || lsn - oldest_lsn > log_max_modified_age_async() + || checkpoint_age > log_max_checkpoint_age_async()) { log->check_flush_or_checkpoint = TRUE; } @@ -668,8 +698,6 @@ log_calc_max_ages(void) ulint archive_margin; ulint smallest_archive_margin; - ut_ad(!mutex_own(&(log_sys->mutex))); - mutex_enter(&(log_sys->mutex)); group = UT_LIST_GET_FIRST(log_sys->log_groups); @@ -770,8 +798,6 @@ void log_init(void) /*==========*/ { - byte* buf; - log_sys = mem_alloc(sizeof(log_t)); mutex_create(&log_sys->mutex, SYNC_LOG); @@ -786,8 +812,8 @@ log_init(void) ut_a(LOG_BUFFER_SIZE >= 16 * OS_FILE_LOG_BLOCK_SIZE); ut_a(LOG_BUFFER_SIZE >= 4 * UNIV_PAGE_SIZE); - buf = mem_alloc(LOG_BUFFER_SIZE + OS_FILE_LOG_BLOCK_SIZE); - log_sys->buf = ut_align(buf, OS_FILE_LOG_BLOCK_SIZE); + log_sys->buf_ptr = mem_alloc(LOG_BUFFER_SIZE + OS_FILE_LOG_BLOCK_SIZE); + log_sys->buf = ut_align(log_sys->buf_ptr, OS_FILE_LOG_BLOCK_SIZE); log_sys->buf_size = LOG_BUFFER_SIZE; @@ -832,9 +858,9 @@ log_init(void) rw_lock_create(&log_sys->checkpoint_lock, SYNC_NO_ORDER_CHECK); - log_sys->checkpoint_buf - = ut_align(mem_alloc(2 * OS_FILE_LOG_BLOCK_SIZE), - OS_FILE_LOG_BLOCK_SIZE); + log_sys->checkpoint_buf_ptr = mem_alloc(2 * OS_FILE_LOG_BLOCK_SIZE); + log_sys->checkpoint_buf = ut_align(log_sys->checkpoint_buf_ptr, + OS_FILE_LOG_BLOCK_SIZE); memset(log_sys->checkpoint_buf, '\0', OS_FILE_LOG_BLOCK_SIZE); /*----------------------------*/ @@ -917,23 +943,33 @@ log_group_init( group->lsn_offset = LOG_FILE_HDR_SIZE; group->n_pending_writes = 0; + group->file_header_bufs_ptr = mem_alloc(sizeof(byte*) * n_files); group->file_header_bufs = mem_alloc(sizeof(byte*) * n_files); #ifdef UNIV_LOG_ARCHIVE + group->archive_file_header_bufs_ptr = mem_alloc( + sizeof(byte*) * n_files); group->archive_file_header_bufs = mem_alloc(sizeof(byte*) * n_files); #endif /* UNIV_LOG_ARCHIVE */ for (i = 0; i < n_files; i++) { - *(group->file_header_bufs + i) = ut_align( - mem_alloc(LOG_FILE_HDR_SIZE + OS_FILE_LOG_BLOCK_SIZE), + group->file_header_bufs_ptr[i] = mem_alloc( + LOG_FILE_HDR_SIZE + OS_FILE_LOG_BLOCK_SIZE); + + group->file_header_bufs[i] = ut_align( + group->file_header_bufs_ptr[i], OS_FILE_LOG_BLOCK_SIZE); memset(*(group->file_header_bufs + i), '\0', LOG_FILE_HDR_SIZE); #ifdef UNIV_LOG_ARCHIVE - *(group->archive_file_header_bufs + i) = ut_align( - mem_alloc(LOG_FILE_HDR_SIZE + OS_FILE_LOG_BLOCK_SIZE), + group->archive_file_header_bufs_ptr[i] = mem_alloc( + LOG_FILE_HDR_SIZE + OS_FILE_LOG_BLOCK_SIZE); + + group->archive_file_header_bufs[i] = ut_align( + group->archive_file_header_bufs_ptr[i], OS_FILE_LOG_BLOCK_SIZE); + memset(*(group->archive_file_header_bufs + i), '\0', LOG_FILE_HDR_SIZE); #endif /* UNIV_LOG_ARCHIVE */ @@ -946,8 +982,9 @@ log_group_init( group->archived_offset = 0; #endif /* UNIV_LOG_ARCHIVE */ - group->checkpoint_buf = ut_align( - mem_alloc(2 * OS_FILE_LOG_BLOCK_SIZE), OS_FILE_LOG_BLOCK_SIZE); + group->checkpoint_buf_ptr = mem_alloc(2 * OS_FILE_LOG_BLOCK_SIZE); + group->checkpoint_buf = ut_align(group->checkpoint_buf_ptr, + OS_FILE_LOG_BLOCK_SIZE); memset(group->checkpoint_buf, '\0', OS_FILE_LOG_BLOCK_SIZE); @@ -1117,6 +1154,7 @@ log_io_complete( } mutex_enter(&(log_sys->mutex)); + ut_ad(!recv_no_log_write); ut_a(group->n_pending_writes > 0); ut_a(log_sys->n_pending_writes > 0); @@ -1148,6 +1186,7 @@ log_group_file_header_flush( ulint dest_offset; ut_ad(mutex_own(&(log_sys->mutex))); + ut_ad(!recv_no_log_write); ut_a(nth_file < group->n_files); buf = *(group->file_header_bufs + nth_file); @@ -1219,6 +1258,7 @@ log_group_write_buf( ulint i; ut_ad(mutex_own(&(log_sys->mutex))); + ut_ad(!recv_no_log_write); ut_a(len % OS_FILE_LOG_BLOCK_SIZE == 0); ut_a(((ulint) start_lsn) % OS_FILE_LOG_BLOCK_SIZE == 0); @@ -1361,6 +1401,7 @@ loop: #endif mutex_enter(&(log_sys->mutex)); + ut_ad(!recv_no_log_write); if (flush_to_disk && log_sys->flushed_to_disk_lsn >= lsn) { @@ -1974,6 +2015,7 @@ log_checkpoint( mutex_enter(&(log_sys->mutex)); + ut_ad(!recv_no_log_write); oldest_lsn = log_buf_pool_get_oldest_modification(); mutex_exit(&(log_sys->mutex)); @@ -2047,7 +2089,7 @@ log_make_checkpoint_at( later lsn, if IB_ULONGLONG_MAX, makes a checkpoint at the latest lsn */ ibool write_always) /*!< in: the function normally checks if - the the new checkpoint would have a + the new checkpoint would have a greater lsn than the previous one: if not, then no physical write is done; by setting this parameter TRUE, a @@ -2086,6 +2128,7 @@ loop: do_checkpoint = FALSE; mutex_enter(&(log->mutex)); + ut_ad(!recv_no_log_write); if (log->check_flush_or_checkpoint == FALSE) { mutex_exit(&(log->mutex)); @@ -2103,10 +2146,10 @@ loop: sync = TRUE; advance = 2 * (age - log->max_modified_age_sync); - } else if (age > log->max_modified_age_async) { + } else if (age > log_max_modified_age_async()) { /* A flush is not urgent: we do an asynchronous preflush */ - advance = age - log->max_modified_age_async; + advance = age - log_max_modified_age_async(); } else { advance = 0; } @@ -2120,7 +2163,7 @@ loop: do_checkpoint = TRUE; - } else if (checkpoint_age > log->max_checkpoint_age_async) { + } else if (checkpoint_age > log_max_checkpoint_age_async()) { /* A checkpoint is not urgent: do it asynchronously */ do_checkpoint = TRUE; @@ -3035,6 +3078,7 @@ loop: #endif /* UNIV_LOG_ARCHIVE */ mutex_enter(&(log_sys->mutex)); + ut_ad(!recv_no_log_write); if (log_sys->check_flush_or_checkpoint) { @@ -3120,6 +3164,16 @@ loop: goto loop; } + /* Check that the purge threads ended */ + if (srv_use_purge_thread + && (srv_n_threads_active[SRV_PURGE] != 0 + || srv_n_threads_active[SRV_PURGE_WORKER] != 0)) { + + mutex_exit(&kernel_mutex); + + goto loop; + } + mutex_exit(&kernel_mutex); mutex_enter(&(log_sys->mutex)); @@ -3234,6 +3288,7 @@ loop: ut_a(lsn == log_sys->lsn); } +#ifdef UNIV_LOG_DEBUG /******************************************************//** Checks by parsing that the catenated log segment for a single mtr is consistent. */ @@ -3241,7 +3296,7 @@ UNIV_INTERN ibool log_check_log_recs( /*===============*/ - byte* buf, /*!< in: pointer to the start of + const byte* buf, /*!< in: pointer to the start of the log segment in the log_sys->buf log buffer */ ulint len, /*!< in: segment length in bytes */ @@ -3249,8 +3304,8 @@ log_check_log_recs( { ib_uint64_t contiguous_lsn; ib_uint64_t scanned_lsn; - byte* start; - byte* end; + const byte* start; + const byte* end; byte* buf1; byte* scan_buf; @@ -3283,6 +3338,7 @@ log_check_log_recs( return(TRUE); } +#endif /* UNIV_LOG_DEBUG */ /******************************************************//** Peeks the current lsn. @@ -3326,10 +3382,12 @@ log_print( log_sys->last_checkpoint_lsn); fprintf(file, - "Max checkpoint age %lu\n" - "Modified age %lu\n" - "Checkpoint age %lu\n", + "Max checkpoint age %lu\n" + "Checkpoint age target %lu\n" + "Modified age %lu\n" + "Checkpoint age %lu\n", (ulong) log_sys->max_checkpoint_age, + (ulong) log_max_checkpoint_age_async(), (ulong) (log_sys->lsn - log_buf_pool_get_oldest_modification()), (ulong) (log_sys->lsn - log_sys->last_checkpoint_lsn)); @@ -3363,4 +3421,95 @@ log_refresh_stats(void) log_sys->n_log_ios_old = log_sys->n_log_ios; log_sys->last_printout_time = time(NULL); } + +/********************************************************************** +Closes a log group. */ +static +void +log_group_close( +/*===========*/ + log_group_t* group) /* in,own: log group to close */ +{ + ulint i; + + for (i = 0; i < group->n_files; i++) { + mem_free(group->file_header_bufs_ptr[i]); +#ifdef UNIV_LOG_ARCHIVE + mem_free(group->archive_file_header_bufs_ptr[i]); +#endif /* UNIV_LOG_ARCHIVE */ + } + + mem_free(group->file_header_bufs_ptr); + mem_free(group->file_header_bufs); + +#ifdef UNIV_LOG_ARCHIVE + mem_free(group->archive_file_header_bufs_ptr); + mem_free(group->archive_file_header_bufs); +#endif /* UNIV_LOG_ARCHIVE */ + + mem_free(group->checkpoint_buf_ptr); + + mem_free(group); +} + +/********************************************************** +Shutdown the log system but do not release all the memory. */ +UNIV_INTERN +void +log_shutdown(void) +/*==============*/ +{ + log_group_t* group; + + group = UT_LIST_GET_FIRST(log_sys->log_groups); + + while (UT_LIST_GET_LEN(log_sys->log_groups) > 0) { + log_group_t* prev_group = group; + + group = UT_LIST_GET_NEXT(log_groups, group); + UT_LIST_REMOVE(log_groups, log_sys->log_groups, prev_group); + + log_group_close(prev_group); + } + + mem_free(log_sys->buf_ptr); + log_sys->buf_ptr = NULL; + log_sys->buf = NULL; + mem_free(log_sys->checkpoint_buf_ptr); + log_sys->checkpoint_buf_ptr = NULL; + log_sys->checkpoint_buf = NULL; + + os_event_free(log_sys->no_flush_event); + os_event_free(log_sys->one_flushed_event); + + rw_lock_free(&log_sys->checkpoint_lock); + + mutex_free(&log_sys->mutex); + +#ifdef UNIV_LOG_ARCHIVE + rw_lock_free(&log_sys->archive_lock); + os_event_create(log_sys->archiving_on); +#endif /* UNIV_LOG_ARCHIVE */ + +#ifdef UNIV_LOG_DEBUG + recv_sys_debug_free(); +#endif + + recv_sys_close(); +} + +/********************************************************** +Free the log system data structures. */ +UNIV_INTERN +void +log_mem_free(void) +/*==============*/ +{ + if (log_sys != NULL) { + recv_sys_mem_free(); + mem_free(log_sys); + + log_sys = NULL; + } +} #endif /* !UNIV_HOTBACKUP */ === modified file 'storage/xtradb/log/log0recv.c' --- a/storage/xtradb/log/log0recv.c 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/log/log0recv.c 2010-01-15 15:58:25 +0000 @@ -69,20 +69,25 @@ UNIV_INTERN recv_sys_t* recv_sys = NULL; /** TRUE when applying redo log records during crash recovery; FALSE otherwise. Note that this is FALSE while a background thread is rolling back incomplete transactions. */ -UNIV_INTERN ibool recv_recovery_on = FALSE; +UNIV_INTERN ibool recv_recovery_on; #ifdef UNIV_LOG_ARCHIVE /** TRUE when applying redo log records from an archived log file */ -UNIV_INTERN ibool recv_recovery_from_backup_on = FALSE; +UNIV_INTERN ibool recv_recovery_from_backup_on; #endif /* UNIV_LOG_ARCHIVE */ #ifndef UNIV_HOTBACKUP /** TRUE when recv_init_crash_recovery() has been called. */ -UNIV_INTERN ibool recv_needed_recovery = FALSE; +UNIV_INTERN ibool recv_needed_recovery; +# ifdef UNIV_DEBUG +/** TRUE if writing to the redo log (mtr_commit) is forbidden. +Protected by log_sys->mutex. */ +UNIV_INTERN ibool recv_no_log_write = FALSE; +# endif /* UNIV_DEBUG */ /** TRUE if buf_page_is_corrupted() should check if the log sequence number (FIL_PAGE_LSN) is in the future. Initially FALSE, and set by recv_recovery_from_checkpoint_start_func(). */ -UNIV_INTERN ibool recv_lsn_checks_on = FALSE; +UNIV_INTERN ibool recv_lsn_checks_on; /** There are two conditions under which we scan the logs, the first is normal startup and the second is when we do a recovery from an @@ -92,7 +97,7 @@ startup. If we find log entries that wer we know that the server was not cleanly shutdown. We must then initialize the crash recovery environment before attempting to store these entries in the log hash table. */ -static ibool recv_log_scan_is_startup_type = FALSE; +static ibool recv_log_scan_is_startup_type; /** If the following is TRUE, the buffer pool file pages must be invalidated after recovery and no ibuf operations are allowed; this becomes TRUE if @@ -103,7 +108,7 @@ buffer pool before the pages have been r TRUE means that recovery is running and no operations on the log files are allowed yet: the variable name is misleading. */ -UNIV_INTERN ibool recv_no_ibuf_operations = FALSE; +UNIV_INTERN ibool recv_no_ibuf_operations; /** TRUE when the redo log is being backed up */ # define recv_is_making_a_backup FALSE /** TRUE when recovering from a backed up redo log file */ @@ -111,30 +116,30 @@ UNIV_INTERN ibool recv_no_ibuf_operation #else /* !UNIV_HOTBACKUP */ # define recv_needed_recovery FALSE /** TRUE when the redo log is being backed up */ -UNIV_INTERN ibool recv_is_making_a_backup = FALSE; +UNIV_INTERN ibool recv_is_making_a_backup = FALSE; /** TRUE when recovering from a backed up redo log file */ UNIV_INTERN ibool recv_is_from_backup = FALSE; # define buf_pool_get_curr_size() (5 * 1024 * 1024) #endif /* !UNIV_HOTBACKUP */ /** The following counter is used to decide when to print info on log scan */ -static ulint recv_scan_print_counter = 0; +static ulint recv_scan_print_counter; /** The type of the previous parsed redo log record */ -static ulint recv_previous_parsed_rec_type = 999999; +static ulint recv_previous_parsed_rec_type; /** The offset of the previous parsed redo log record */ -static ulint recv_previous_parsed_rec_offset = 0; +static ulint recv_previous_parsed_rec_offset; /** The 'multi' flag of the previous parsed redo log record */ -static ulint recv_previous_parsed_rec_is_multi = 0; +static ulint recv_previous_parsed_rec_is_multi; /** Maximum page number encountered in the redo log */ -UNIV_INTERN ulint recv_max_parsed_page_no = 0; +UNIV_INTERN ulint recv_max_parsed_page_no; /** This many frames must be left free in the buffer pool when we scan the log and store the scanned log records in the buffer pool: we will use these free frames to read in pages when we start applying the log records to the database. */ -UNIV_INTERN ulint recv_n_pool_free_frames = 1024; +UNIV_INTERN ulint recv_n_pool_free_frames; /** The maximum lsn we see for a page during the recovery process. If this is bigger than the lsn we are able to scan up to, that is an indication that @@ -165,15 +170,119 @@ recv_sys_create(void) return; } - recv_sys = mem_alloc(sizeof(recv_sys_t)); + recv_sys = mem_alloc(sizeof(*recv_sys)); + memset(recv_sys, 0x0, sizeof(*recv_sys)); mutex_create(&recv_sys->mutex, SYNC_RECV); recv_sys->heap = NULL; recv_sys->addr_hash = NULL; + + recv_sys->stats_recv_start_time = time(NULL); + recv_sys->stats_oldest_modified_lsn = IB_ULONGLONG_MAX; } /********************************************************//** +Release recovery system mutexes. */ +UNIV_INTERN +void +recv_sys_close(void) +/*================*/ +{ + if (recv_sys != NULL) { + if (recv_sys->addr_hash != NULL) { + hash_table_free(recv_sys->addr_hash); + } + + if (recv_sys->heap != NULL) { + mem_heap_free(recv_sys->heap); + } + + if (recv_sys->buf != NULL) { + ut_free(recv_sys->buf); + } + + if (recv_sys->last_block_buf_start != NULL) { + mem_free(recv_sys->last_block_buf_start); + } + + mutex_free(&recv_sys->mutex); + + mem_free(recv_sys); + recv_sys = NULL; + } +} + +/********************************************************//** +Frees the recovery system memory. */ +UNIV_INTERN +void +recv_sys_mem_free(void) +/*===================*/ +{ + if (recv_sys != NULL) { + if (recv_sys->addr_hash != NULL) { + hash_table_free(recv_sys->addr_hash); + } + + if (recv_sys->heap != NULL) { + mem_heap_free(recv_sys->heap); + } + + if (recv_sys->buf != NULL) { + ut_free(recv_sys->buf); + } + + if (recv_sys->last_block_buf_start != NULL) { + mem_free(recv_sys->last_block_buf_start); + } + + mem_free(recv_sys); + recv_sys = NULL; + } +} + +/************************************************************ +Reset the state of the recovery system variables. */ +UNIV_INTERN +void +recv_sys_var_init(void) +/*===================*/ +{ + recv_lsn_checks_on = FALSE; + + recv_n_pool_free_frames = 1024; + + recv_recovery_on = FALSE; + +#ifdef UNIV_LOG_ARCHIVE + recv_recovery_from_backup_on = FALSE; +#endif /* UNIV_LOG_ARCHIVE */ + + recv_needed_recovery = FALSE; + + recv_lsn_checks_on = FALSE; + + recv_log_scan_is_startup_type = FALSE; + + recv_no_ibuf_operations = FALSE; + + recv_scan_print_counter = 0; + + recv_previous_parsed_rec_type = 999999; + + recv_previous_parsed_rec_offset = 0; + + recv_previous_parsed_rec_is_multi = 0; + + recv_max_parsed_page_no = 0; + + recv_n_pool_free_frames = 1024; + + recv_max_page_lsn = 0; +} + +/************************************************************ Inits the recovery system for a recovery operation. */ UNIV_INTERN void @@ -248,8 +357,8 @@ recv_sys_empty_hash(void) Frees the recovery system. */ static void -recv_sys_free(void) -/*===============*/ +recv_sys_debug_free(void) +/*=====================*/ { mutex_enter(&(recv_sys->mutex)); @@ -258,8 +367,10 @@ recv_sys_free(void) ut_free(recv_sys->buf); mem_free(recv_sys->last_block_buf_start); - recv_sys->addr_hash = NULL; + recv_sys->buf = NULL; recv_sys->heap = NULL; + recv_sys->addr_hash = NULL; + recv_sys->last_block_buf_start = NULL; mutex_exit(&(recv_sys->mutex)); } @@ -853,6 +964,11 @@ recv_parse_or_apply_log_rec_body( } switch (type) { +#ifdef UNIV_LOG_LSN_DEBUG + case MLOG_LSN: + /* The LSN is checked in recv_parse_log_rec(). */ + break; +#endif /* UNIV_LOG_LSN_DEBUG */ case MLOG_1BYTE: case MLOG_2BYTES: case MLOG_4BYTES: case MLOG_8BYTES: #ifdef UNIV_DEBUG if (page && page_type == FIL_PAGE_TYPE_ALLOCATED @@ -1223,6 +1339,11 @@ recv_add_to_hash_table( len = rec_end - body; + if (srv_recovery_stats) { + recv_sys->stats_log_recs++; + recv_sys->stats_log_len_sum += len; + } + recv = mem_heap_alloc(recv_sys->heap, sizeof(recv_t)); recv->type = type; recv->len = rec_end - body; @@ -1269,7 +1390,7 @@ recv_add_to_hash_table( sizeof(recv_data_t) + len); *prev_field = recv_data; - ut_memcpy(((byte*)recv_data) + sizeof(recv_data_t), body, len); + memcpy(recv_data + 1, body, len); prev_field = &(recv_data->next); @@ -1327,12 +1448,14 @@ recv_recover_page_func( buf_block_t* block) /*!< in/out: buffer block */ { page_t* page; + page_zip_des_t* page_zip; recv_addr_t* recv_addr; recv_t* recv; byte* buf; ib_uint64_t start_lsn; ib_uint64_t end_lsn; ib_uint64_t page_lsn; + ib_uint64_t page_lsn_orig; ib_uint64_t page_newest_lsn; ibool modification_to_page; #ifndef UNIV_HOTBACKUP @@ -1372,12 +1495,21 @@ recv_recover_page_func( recv_addr->state = RECV_BEING_PROCESSED; + if (srv_recovery_stats) { + if (just_read_in) { + recv_sys->stats_recover_pages_with_read++; + } else { + recv_sys->stats_recover_pages_without_read++; + } + } + mutex_exit(&(recv_sys->mutex)); mtr_start(&mtr); mtr_set_log_mode(&mtr, MTR_LOG_NONE); page = block->frame; + page_zip = buf_block_get_page_zip(block); #ifndef UNIV_HOTBACKUP if (just_read_in) { @@ -1400,6 +1532,7 @@ recv_recover_page_func( /* Read the newest modification lsn from the page */ page_lsn = mach_read_ull(page + FIL_PAGE_LSN); + page_lsn_orig = page_lsn; #ifndef UNIV_HOTBACKUP /* It may be that the page has been modified in the buffer @@ -1419,6 +1552,21 @@ recv_recover_page_func( modification_to_page = FALSE; start_lsn = end_lsn = 0; + if (srv_recovery_stats) { + mutex_enter(&(recv_sys->mutex)); + if (page_lsn_orig && recv_sys->stats_oldest_modified_lsn > page_lsn_orig) { + recv_sys->stats_oldest_modified_lsn = page_lsn_orig; + } + if (page_lsn_orig && recv_sys->stats_newest_modified_lsn < page_lsn_orig) { + recv_sys->stats_newest_modified_lsn = page_lsn_orig; + } + if (UT_LIST_GET_LAST(recv_addr->rec_list)->start_lsn + < page_lsn_orig) { + recv_sys->stats_pages_already_new++; + } + mutex_exit(&(recv_sys->mutex)); + } + recv = UT_LIST_GET_FIRST(recv_addr->rec_list); while (recv) { @@ -1438,13 +1586,19 @@ recv_recover_page_func( if (recv->type == MLOG_INIT_FILE_PAGE) { page_lsn = page_newest_lsn; - mach_write_ull(page + UNIV_PAGE_SIZE - - FIL_PAGE_END_LSN_OLD_CHKSUM, 0); - mach_write_ull(page + FIL_PAGE_LSN, 0); + memset(FIL_PAGE_LSN + page, 0, 8); + memset(UNIV_PAGE_SIZE - FIL_PAGE_END_LSN_OLD_CHKSUM + + page, 0, 8); + + if (page_zip) { + memset(FIL_PAGE_LSN + page_zip->data, 0, 8); + } } if (recv->start_lsn >= page_lsn) { + ib_uint64_t end_lsn; + if (!modification_to_page) { modification_to_page = TRUE; @@ -1466,11 +1620,24 @@ recv_recover_page_func( recv_parse_or_apply_log_rec_body(recv->type, buf, buf + recv->len, block, &mtr); - mach_write_ull(page + UNIV_PAGE_SIZE - - FIL_PAGE_END_LSN_OLD_CHKSUM, - recv->start_lsn + recv->len); - mach_write_ull(page + FIL_PAGE_LSN, - recv->start_lsn + recv->len); + + if (srv_recovery_stats) { + mutex_enter(&(recv_sys->mutex)); + recv_sys->stats_applied_log_recs++; + recv_sys->stats_applied_log_len_sum += recv->len; + mutex_exit(&(recv_sys->mutex)); + } + + end_lsn = recv->start_lsn + recv->len; + mach_write_ull(FIL_PAGE_LSN + page, end_lsn); + mach_write_ull(UNIV_PAGE_SIZE + - FIL_PAGE_END_LSN_OLD_CHKSUM + + page, end_lsn); + + if (page_zip) { + mach_write_ull(FIL_PAGE_LSN + + page_zip->data, end_lsn); + } } if (recv->len > RECV_DATA_BLOCK_SIZE) { @@ -1561,6 +1728,13 @@ recv_read_in_area( } } + if (srv_recovery_stats && n) { + mutex_enter(&(recv_sys->mutex)); + recv_sys->stats_read_requested_pages += n; + recv_sys->stats_read_in_area[n - 1]++; + mutex_exit(&(recv_sys->mutex)); + } + buf_read_recv_pages(FALSE, space, zip_size, page_nos, n); /* fprintf(stderr, "Recv pages at %lu n %lu\n", page_nos[0], n); @@ -1688,6 +1862,7 @@ loop: /* Flush all the file pages to disk and invalidate them in the buffer pool */ + ut_d(recv_no_log_write = TRUE); mutex_exit(&(recv_sys->mutex)); mutex_exit(&(log_sys->mutex)); @@ -1701,6 +1876,7 @@ loop: mutex_enter(&(log_sys->mutex)); mutex_enter(&(recv_sys->mutex)); + ut_d(recv_no_log_write = FALSE); recv_no_ibuf_operations = FALSE; } @@ -1712,6 +1888,10 @@ loop: if (has_printed) { fprintf(stderr, "InnoDB: Apply batch completed\n"); + + if (srv_recovery_stats) { + recv_sys->stats_recv_turns++; + } } mutex_exit(&(recv_sys->mutex)); @@ -1912,6 +2092,17 @@ recv_parse_log_rec( return(0); } +#ifdef UNIV_LOG_LSN_DEBUG + if (*type == MLOG_LSN) { + ib_uint64_t lsn = (ib_uint64_t) *space << 32 | *page_no; +# ifdef UNIV_LOG_DEBUG + ut_a(lsn == log_sys->old_lsn); +# else /* UNIV_LOG_DEBUG */ + ut_a(lsn == recv_sys->recovered_lsn); +# endif /* UNIV_LOG_DEBUG */ + } +#endif /* UNIV_LOG_LSN_DEBUG */ + /* Check that page_no is sensible */ if (UNIV_UNLIKELY(*page_no > 0x8FFFFFFFUL)) { @@ -2169,6 +2360,12 @@ loop: #endif /* In normal mysqld crash recovery we do not try to replay file operations */ +#ifdef UNIV_LOG_LSN_DEBUG + } else if (type == MLOG_LSN) { + /* Do not add these records to the hash table. + The page number and space id fields are misused + for something else. */ +#endif /* UNIV_LOG_LSN_DEBUG */ } else { recv_add_to_hash_table(type, space, page_no, body, ptr + len, old_lsn, @@ -2200,11 +2397,11 @@ loop: = recv_sys->recovered_offset + total_len; recv_previous_parsed_rec_is_multi = 1; - if ((!store_to_hash) && (type != MLOG_MULTI_REC_END)) { #ifdef UNIV_LOG_DEBUG + if ((!store_to_hash) && (type != MLOG_MULTI_REC_END)) { recv_check_incomplete_log_recs(ptr, len); -#endif /* UNIV_LOG_DEBUG */ } +#endif /* UNIV_LOG_DEBUG */ #ifdef UNIV_DEBUG if (log_debug_writes) { @@ -2268,7 +2465,11 @@ loop: break; } - if (store_to_hash) { + if (store_to_hash +#ifdef UNIV_LOG_LSN_DEBUG + && type != MLOG_LSN +#endif /* UNIV_LOG_LSN_DEBUG */ + ) { recv_add_to_hash_table(type, space, page_no, body, ptr + len, old_lsn, @@ -2417,8 +2618,7 @@ recv_scan_log_recs( scanned_lsn = start_lsn; more_data = FALSE; - while (log_block < buf + len && !finished) { - + do { no = log_block_get_hdr_no(log_block); /* fprintf(stderr, "Log block header no %lu\n", no); @@ -2548,10 +2748,11 @@ recv_scan_log_recs( /* Log data for this group ends here */ finished = TRUE; + break; } else { log_block += OS_FILE_LOG_BLOCK_SIZE; } - } + } while (log_block < buf + len && !finished); *group_scanned_lsn = scanned_lsn; @@ -3078,6 +3279,84 @@ recv_recovery_from_checkpoint_finish(voi } #endif /* UNIV_DEBUG */ + if (recv_needed_recovery && srv_recovery_stats) { + FILE* file = stderr; + ulint i; + + fprintf(stderr, + "InnoDB: Applying log records was done. Its statistics are followings.\n"); + + fprintf(stderr, + "============================================================\n" + "-------------------\n" + "RECOVERY STATISTICS\n" + "-------------------\n"); + fprintf(stderr, + "Recovery time: %g sec. (%lu turns)\n", + difftime(time(NULL), recv_sys->stats_recv_start_time), + recv_sys->stats_recv_turns); + + fprintf(stderr, + "\n" + "Data page IO statistics\n" + " Requested pages: %lu\n" + " Read pages: %lu\n" + " Written pages: %lu\n" + " (Dirty blocks): %lu\n", + recv_sys->stats_read_requested_pages, + recv_sys->stats_read_io_pages, + recv_sys->stats_write_io_pages, + UT_LIST_GET_LEN(buf_pool->flush_list)); + + fprintf(stderr, + " Grouping IO [times]:\n" + "\tnumber of pages,\n" + "\t\tread request neighbors (in %d pages chunk),\n" + "\t\t\tcombined read IO,\n" + "\t\t\t\tcombined write IO\n", + RECV_READ_AHEAD_AREA); + for (i = 0; i < ut_max(RECV_READ_AHEAD_AREA, + OS_AIO_MERGE_N_CONSECUTIVE); i++) { + fprintf(stderr, + "\t%3lu,\t%lu,\t%lu,\t%lu\n", i + 1, + (i < RECV_READ_AHEAD_AREA) ? + recv_sys->stats_read_in_area[i] : 0, + (i < OS_AIO_MERGE_N_CONSECUTIVE) ? + recv_sys->stats_read_io_consecutive[i] : 0, + (i < OS_AIO_MERGE_N_CONSECUTIVE) ? + recv_sys->stats_write_io_consecutive[i] : 0); + } + + fprintf(stderr, + "\n" + "Recovery process statistics\n" + " Checked pages by doublewrite buffer: %lu\n" + " Overwritten pages from doublewrite: %lu\n" + " Recovered pages by io_thread: %lu\n" + " Recovered pages by main thread: %lu\n" + " Parsed log records to apply: %lu\n" + " Sum of the length: %lu\n" + " Applied log records: %lu\n" + " Sum of the length: %lu\n" + " Pages which are already new enough: %lu (It may not be accurate, if turns > 1)\n" + " Oldest page's LSN: %llu\n" + " Newest page's LSN: %llu\n", + recv_sys->stats_doublewrite_check_pages, + recv_sys->stats_doublewrite_overwrite_pages, + recv_sys->stats_recover_pages_with_read, + recv_sys->stats_recover_pages_without_read, + recv_sys->stats_log_recs, + recv_sys->stats_log_len_sum, + recv_sys->stats_applied_log_recs, + recv_sys->stats_applied_log_len_sum, + recv_sys->stats_pages_already_new, + recv_sys->stats_oldest_modified_lsn, + recv_sys->stats_newest_modified_lsn); + + fprintf(stderr, + "============================================================\n"); + } + if (recv_needed_recovery) { trx_sys_print_mysql_master_log_pos(); trx_sys_print_mysql_binlog_offset(); @@ -3104,8 +3383,13 @@ recv_recovery_from_checkpoint_finish(voi recv_recovery_on = FALSE; #ifndef UNIV_LOG_DEBUG - recv_sys_free(); + recv_sys_debug_free(); #endif + /* Roll back any recovered data dictionary transactions, so + that the data dictionary tables will be free of any locks. + The data dictionary latch should guarantee that there is at + most one data dictionary transaction active at a time. */ + trx_rollback_or_clean_recovered(FALSE); /* Drop partially created indexes. */ row_merge_drop_temp_indexes(); === modified file 'storage/xtradb/mem/mem0dbg.c' --- a/storage/xtradb/mem/mem0dbg.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/mem/mem0dbg.c 2010-01-06 12:00:14 +0000 @@ -170,6 +170,17 @@ mem_init( mem_comm_pool = mem_pool_create(size); } + +/******************************************************************//** +Closes the memory system. */ +UNIV_INTERN +void +mem_close(void) +/*===========*/ +{ + mem_pool_free(mem_comm_pool); + mem_comm_pool = NULL; +} #endif /* !UNIV_HOTBACKUP */ #ifdef UNIV_MEM_DEBUG === modified file 'storage/xtradb/mem/mem0mem.c' --- a/storage/xtradb/mem/mem0mem.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/mem/mem0mem.c 2010-01-06 12:00:14 +0000 @@ -475,16 +475,18 @@ mem_heap_block_free( len = block->len; block->magic_n = MEM_FREED_BLOCK_MAGIC_N; +#ifndef UNIV_HOTBACKUP + if (!srv_use_sys_malloc) { #ifdef UNIV_MEM_DEBUG - /* In the debug version we set the memory to a random combination - of hex 0xDE and 0xAD. */ + /* In the debug version we set the memory to a random + combination of hex 0xDE and 0xAD. */ - mem_erase_buf((byte*)block, len); + mem_erase_buf((byte*)block, len); #else /* UNIV_MEM_DEBUG */ - UNIV_MEM_ASSERT_AND_FREE(block, len); + UNIV_MEM_ASSERT_AND_FREE(block, len); #endif /* UNIV_MEM_DEBUG */ -#ifndef UNIV_HOTBACKUP + } if (type == MEM_HEAP_DYNAMIC || len < UNIV_PAGE_SIZE / 2) { ut_ad(!buf_block); @@ -495,6 +497,14 @@ mem_heap_block_free( buf_block_free(buf_block); } #else /* !UNIV_HOTBACKUP */ +#ifdef UNIV_MEM_DEBUG + /* In the debug version we set the memory to a random + combination of hex 0xDE and 0xAD. */ + + mem_erase_buf((byte*)block, len); +#else /* UNIV_MEM_DEBUG */ + UNIV_MEM_ASSERT_AND_FREE(block, len); +#endif /* UNIV_MEM_DEBUG */ ut_free(block); #endif /* !UNIV_HOTBACKUP */ } === modified file 'storage/xtradb/mem/mem0pool.c' --- a/storage/xtradb/mem/mem0pool.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/mem/mem0pool.c 2010-01-06 12:00:14 +0000 @@ -261,6 +261,18 @@ mem_pool_create( } /********************************************************************//** +Frees a memory pool. */ +UNIV_INTERN +void +mem_pool_free( +/*==========*/ + mem_pool_t* pool) /*!< in, own: memory pool */ +{ + ut_free(pool->buf); + ut_free(pool); +} + +/********************************************************************//** Fills the specified free list. @return TRUE if we were able to insert a block to the free list */ static === modified file 'storage/xtradb/mtr/mtr0mtr.c' --- a/storage/xtradb/mtr/mtr0mtr.c 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/mtr/mtr0mtr.c 2010-01-15 15:58:25 +0000 @@ -36,6 +36,7 @@ Created 11/26/1995 Heikki Tuuri #include "buf0flu.h" #ifndef UNIV_HOTBACKUP +# include "log0recv.h" /*****************************************************************//** Releases the item in the slot given. */ UNIV_INLINE @@ -148,7 +149,6 @@ mtr_log_reserve_and_write( dyn_array_t* mlog; dyn_block_t* block; ulint data_size; - ibool success; byte* first_data; ut_ad(mtr); @@ -167,8 +167,8 @@ mtr_log_reserve_and_write( if (mlog->heap == NULL) { mtr->end_lsn = log_reserve_and_write_fast( first_data, dyn_block_get_used(mlog), - &(mtr->start_lsn), &success); - if (success) { + &mtr->start_lsn); + if (mtr->end_lsn) { return; } @@ -215,6 +215,8 @@ mtr_commit( ut_d(mtr->state = MTR_COMMITTING); #ifndef UNIV_HOTBACKUP + /* This is a dirty read, for debugging. */ + ut_ad(!recv_no_log_write); write_log = mtr->modifications && mtr->n_log_recs; if (write_log) { === modified file 'storage/xtradb/os/os0file.c' --- a/storage/xtradb/os/os0file.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/os/os0file.c 2010-01-06 12:00:14 +0000 @@ -55,6 +55,9 @@ Created 10/21/1995 Heikki Tuuri #include "srv0start.h" #include "fil0fil.h" #include "buf0buf.h" +#include "trx0sys.h" +#include "trx0trx.h" +#include "log0recv.h" #ifndef UNIV_HOTBACKUP # include "os0sync.h" # include "os0thread.h" @@ -88,7 +91,9 @@ UNIV_INTERN ibool os_do_not_call_flush_a /* We do not call os_file_flush in every os_file_write. */ #endif /* UNIV_DO_FLUSH */ -#ifndef UNIV_HOTBACKUP +#ifdef UNIV_HOTBACKUP +# define os_aio_use_native_aio FALSE +#else /* UNIV_HOTBACKUP */ /* We use these mutexes to protect lseek + file i/o operation, if the OS does not provide an atomic pread or pwrite, or similar */ #define OS_FILE_N_SEEK_MUTEXES 16 @@ -235,7 +240,7 @@ static ulint os_aio_n_segments = ULINT_U /** If the following is TRUE, read i/o handler threads try to wait until a batch of new read requests have been posted */ static volatile ibool os_aio_recommend_sleep_for_read_threads = FALSE; -#endif /* !UNIV_HOTBACKUP */ +#endif /* UNIV_HOTBACKUP */ UNIV_INTERN ulint os_n_file_reads = 0; UNIV_INTERN ulint os_bytes_read_since_printout = 0; @@ -352,6 +357,19 @@ os_file_get_last_error( " software or another instance\n" "InnoDB: of MySQL." " Please close it to get rid of this error.\n"); + } else if (err == ERROR_WORKING_SET_QUOTA + || err == ERROR_NO_SYSTEM_RESOURCES) { + fprintf(stderr, + "InnoDB: The error means that there are no" + " sufficient system resources or quota to" + " complete the operation.\n"); + } else if (err == ERROR_OPERATION_ABORTED) { + fprintf(stderr, + "InnoDB: The error means that the I/O" + " operation has been aborted\n" + "InnoDB: because of either a thread exit" + " or an application request.\n" + "InnoDB: Retry attempt is made.\n"); } else { fprintf(stderr, "InnoDB: Some operating system error numbers" @@ -373,6 +391,11 @@ os_file_get_last_error( } else if (err == ERROR_SHARING_VIOLATION || err == ERROR_LOCK_VIOLATION) { return(OS_FILE_SHARING_VIOLATION); + } else if (err == ERROR_WORKING_SET_QUOTA + || err == ERROR_NO_SYSTEM_RESOURCES) { + return(OS_FILE_INSUFFICIENT_RESOURCE); + } else if (err == ERROR_OPERATION_ABORTED) { + return(OS_FILE_OPERATION_ABORTED); } else { return(100 + err); } @@ -491,6 +514,14 @@ os_file_handle_error_cond_exit( os_thread_sleep(10000000); /* 10 sec */ return(TRUE); + } else if (err == OS_FILE_INSUFFICIENT_RESOURCE) { + + os_thread_sleep(100000); /* 100 ms */ + return(TRUE); + } else if (err == OS_FILE_OPERATION_ABORTED) { + + os_thread_sleep(100000); /* 100 ms */ + return(TRUE); } else { if (name) { fprintf(stderr, "InnoDB: File name %s\n", name); @@ -854,6 +885,23 @@ next_file: ret = stat(full_path, &statinfo); if (ret) { + + if (errno == ENOENT) { + /* readdir() returned a file that does not exist, + it must have been deleted in the meantime. Do what + would have happened if the file was deleted before + readdir() - ignore and go to the next entry. + If this is the last entry then info->name will still + contain the name of the deleted file when this + function returns, but this is not an issue since the + caller shouldn't be looking at info when end of + directory is returned. */ + + ut_free(full_path); + + goto next_file; + } + os_file_handle_error_no_exit(full_path, "stat"); ut_free(full_path); @@ -1282,6 +1330,7 @@ try_again: } #endif #ifdef UNIV_NON_BUFFERED_IO +# ifndef UNIV_HOTBACKUP if (type == OS_LOG_FILE && srv_flush_log_at_trx_commit == 2) { /* Do not use unbuffered i/o to log files because value 2 denotes that we do not flush the log at every @@ -1290,10 +1339,14 @@ try_again: == SRV_WIN_IO_UNBUFFERED) { attributes = attributes | FILE_FLAG_NO_BUFFERING; } -#endif +# else /* !UNIV_HOTBACKUP */ + attributes = attributes | FILE_FLAG_NO_BUFFERING; +# endif /* !UNIV_HOTBACKUP */ +#endif /* UNIV_NON_BUFFERED_IO */ } else if (purpose == OS_FILE_NORMAL) { attributes = 0; #ifdef UNIV_NON_BUFFERED_IO +# ifndef UNIV_HOTBACKUP if (type == OS_LOG_FILE && srv_flush_log_at_trx_commit == 2) { /* Do not use unbuffered i/o to log files because value 2 denotes that we do not flush the log at every @@ -1302,7 +1355,10 @@ try_again: == SRV_WIN_IO_UNBUFFERED) { attributes = attributes | FILE_FLAG_NO_BUFFERING; } -#endif +# else /* !UNIV_HOTBACKUP */ + attributes = attributes | FILE_FLAG_NO_BUFFERING; +# endif /* !UNIV_HOTBACKUP */ +#endif /* UNIV_NON_BUFFERED_IO */ } else { attributes = 0; ut_error; @@ -2046,20 +2102,30 @@ os_file_flush( /*******************************************************************//** Does a synchronous read operation in Posix. @return number of bytes read, -1 if error */ +#define os_file_pread(file, buf, n, offset, offset_high) \ + _os_file_pread(file, buf, n, offset, offset_high, NULL); + static ssize_t -os_file_pread( +_os_file_pread( /*==========*/ os_file_t file, /*!< in: handle to a file */ void* buf, /*!< in: buffer where to read */ ulint n, /*!< in: number of bytes to read */ ulint offset, /*!< in: least significant 32 bits of file offset from where to read */ - ulint offset_high) /*!< in: most significant 32 bits of + ulint offset_high, /*!< in: most significant 32 bits of offset */ + trx_t* trx) { off_t offs; +#if defined(HAVE_PREAD) && !defined(HAVE_BROKEN_PREAD) ssize_t n_bytes; +#endif /* HAVE_PREAD && !HAVE_BROKEN_PREAD */ + ulint sec; + ulint ms; + ib_uint64_t start_time; + ib_uint64_t finish_time; ut_a((offset & 0xFFFFFFFFUL) == offset); @@ -2080,6 +2146,15 @@ os_file_pread( os_n_file_reads++; + if (innobase_get_slow_log() && trx && trx->take_stats) + { + trx->io_reads++; + trx->io_read += n; + ut_usectime(&sec, &ms); + start_time = (ib_uint64_t)sec * 1000000 + ms; + } else { + start_time = 0; + } #if defined(HAVE_PREAD) && !defined(HAVE_BROKEN_PREAD) os_mutex_enter(os_file_count_mutex); os_file_n_pending_preads++; @@ -2093,21 +2168,32 @@ os_file_pread( os_n_pending_reads--; os_mutex_exit(os_file_count_mutex); + if (innobase_get_slow_log() && trx && trx->take_stats && start_time) + { + ut_usectime(&sec, &ms); + finish_time = (ib_uint64_t)sec * 1000000 + ms; + trx->io_reads_wait_timer += (ulint)(finish_time - start_time); + } + return(n_bytes); #else { off_t ret_offset; ssize_t ret; +#ifndef UNIV_HOTBACKUP ulint i; +#endif /* !UNIV_HOTBACKUP */ os_mutex_enter(os_file_count_mutex); os_n_pending_reads++; os_mutex_exit(os_file_count_mutex); +#ifndef UNIV_HOTBACKUP /* Protect the seek / read operation with a mutex */ i = ((ulint) file) % OS_FILE_N_SEEK_MUTEXES; os_mutex_enter(os_file_seek_mutexes[i]); +#endif /* !UNIV_HOTBACKUP */ ret_offset = lseek(file, offs, SEEK_SET); @@ -2117,12 +2203,21 @@ os_file_pread( ret = read(file, buf, (ssize_t)n); } +#ifndef UNIV_HOTBACKUP os_mutex_exit(os_file_seek_mutexes[i]); +#endif /* !UNIV_HOTBACKUP */ os_mutex_enter(os_file_count_mutex); os_n_pending_reads--; os_mutex_exit(os_file_count_mutex); + if (innobase_get_slow_log() && trx && trx->take_stats && start_time) + { + ut_usectime(&sec, &ms); + finish_time = (ib_uint64_t)sec * 1000000 + ms; + trx->io_reads_wait_timer += (ulint)(finish_time - start_time); + } + return(ret); } #endif @@ -2195,16 +2290,20 @@ os_file_pwrite( #else { off_t ret_offset; +# ifndef UNIV_HOTBACKUP ulint i; +# endif /* !UNIV_HOTBACKUP */ os_mutex_enter(os_file_count_mutex); os_n_pending_writes++; os_mutex_exit(os_file_count_mutex); +# ifndef UNIV_HOTBACKUP /* Protect the seek / write operation with a mutex */ i = ((ulint) file) % OS_FILE_N_SEEK_MUTEXES; os_mutex_enter(os_file_seek_mutexes[i]); +# endif /* UNIV_HOTBACKUP */ ret_offset = lseek(file, offs, SEEK_SET); @@ -2230,7 +2329,9 @@ os_file_pwrite( # endif /* UNIV_DO_FLUSH */ func_exit: +# ifndef UNIV_HOTBACKUP os_mutex_exit(os_file_seek_mutexes[i]); +# endif /* !UNIV_HOTBACKUP */ os_mutex_enter(os_file_count_mutex); os_n_pending_writes--; @@ -2247,7 +2348,7 @@ Requests a synchronous positioned read o @return TRUE if request was successful, FALSE if fail */ UNIV_INTERN ibool -os_file_read( +_os_file_read( /*=========*/ os_file_t file, /*!< in: handle to a file */ void* buf, /*!< in: buffer where to read */ @@ -2255,7 +2356,8 @@ os_file_read( offset where to read */ ulint offset_high, /*!< in: most significant 32 bits of offset */ - ulint n) /*!< in: number of bytes to read */ + ulint n, /*!< in: number of bytes to read */ + trx_t* trx) { #ifdef __WIN__ BOOL ret; @@ -2264,7 +2366,9 @@ os_file_read( DWORD low; DWORD high; ibool retry; +#ifndef UNIV_HOTBACKUP ulint i; +#endif /* !UNIV_HOTBACKUP */ ut_a((offset & 0xFFFFFFFFUL) == offset); @@ -2283,16 +2387,20 @@ try_again: os_n_pending_reads++; os_mutex_exit(os_file_count_mutex); +#ifndef UNIV_HOTBACKUP /* Protect the seek / read operation with a mutex */ i = ((ulint) file) % OS_FILE_N_SEEK_MUTEXES; os_mutex_enter(os_file_seek_mutexes[i]); +#endif /* !UNIV_HOTBACKUP */ ret2 = SetFilePointer(file, low, &high, FILE_BEGIN); if (ret2 == 0xFFFFFFFF && GetLastError() != NO_ERROR) { +#ifndef UNIV_HOTBACKUP os_mutex_exit(os_file_seek_mutexes[i]); +#endif /* !UNIV_HOTBACKUP */ os_mutex_enter(os_file_count_mutex); os_n_pending_reads--; @@ -2303,7 +2411,9 @@ try_again: ret = ReadFile(file, buf, (DWORD) n, &len, NULL); +#ifndef UNIV_HOTBACKUP os_mutex_exit(os_file_seek_mutexes[i]); +#endif /* !UNIV_HOTBACKUP */ os_mutex_enter(os_file_count_mutex); os_n_pending_reads--; @@ -2312,14 +2422,14 @@ try_again: if (ret && len == n) { return(TRUE); } -#else +#else /* __WIN__ */ ibool retry; ssize_t ret; os_bytes_read_since_printout += n; try_again: - ret = os_file_pread(file, buf, n, offset, offset_high); + ret = _os_file_pread(file, buf, n, offset, offset_high, trx); if ((ulint)ret == n) { @@ -2331,7 +2441,7 @@ try_again: "InnoDB: Was only able to read %ld.\n", (ulong)n, (ulong)offset_high, (ulong)offset, (long)ret); -#endif +#endif /* __WIN__ */ #ifdef __WIN__ error_handling: #endif @@ -2380,7 +2490,9 @@ os_file_read_no_error_handling( DWORD low; DWORD high; ibool retry; +#ifndef UNIV_HOTBACKUP ulint i; +#endif /* !UNIV_HOTBACKUP */ ut_a((offset & 0xFFFFFFFFUL) == offset); @@ -2399,16 +2511,20 @@ try_again: os_n_pending_reads++; os_mutex_exit(os_file_count_mutex); +#ifndef UNIV_HOTBACKUP /* Protect the seek / read operation with a mutex */ i = ((ulint) file) % OS_FILE_N_SEEK_MUTEXES; os_mutex_enter(os_file_seek_mutexes[i]); +#endif /* !UNIV_HOTBACKUP */ ret2 = SetFilePointer(file, low, &high, FILE_BEGIN); if (ret2 == 0xFFFFFFFF && GetLastError() != NO_ERROR) { +#ifndef UNIV_HOTBACKUP os_mutex_exit(os_file_seek_mutexes[i]); +#endif /* !UNIV_HOTBACKUP */ os_mutex_enter(os_file_count_mutex); os_n_pending_reads--; @@ -2419,7 +2535,9 @@ try_again: ret = ReadFile(file, buf, (DWORD) n, &len, NULL); +#ifndef UNIV_HOTBACKUP os_mutex_exit(os_file_seek_mutexes[i]); +#endif /* !UNIV_HOTBACKUP */ os_mutex_enter(os_file_count_mutex); os_n_pending_reads--; @@ -2428,7 +2546,7 @@ try_again: if (ret && len == n) { return(TRUE); } -#else +#else /* __WIN__ */ ibool retry; ssize_t ret; @@ -2441,7 +2559,7 @@ try_again: return(TRUE); } -#endif +#endif /* __WIN__ */ #ifdef __WIN__ error_handling: #endif @@ -2500,9 +2618,11 @@ os_file_write( DWORD ret2; DWORD low; DWORD high; - ulint i; ulint n_retries = 0; ulint err; +#ifndef UNIV_HOTBACKUP + ulint i; +#endif /* !UNIV_HOTBACKUP */ ut_a((offset & 0xFFFFFFFF) == offset); @@ -2519,16 +2639,20 @@ retry: os_n_pending_writes++; os_mutex_exit(os_file_count_mutex); +#ifndef UNIV_HOTBACKUP /* Protect the seek / write operation with a mutex */ i = ((ulint) file) % OS_FILE_N_SEEK_MUTEXES; os_mutex_enter(os_file_seek_mutexes[i]); +#endif /* !UNIV_HOTBACKUP */ ret2 = SetFilePointer(file, low, &high, FILE_BEGIN); if (ret2 == 0xFFFFFFFF && GetLastError() != NO_ERROR) { +#ifndef UNIV_HOTBACKUP os_mutex_exit(os_file_seek_mutexes[i]); +#endif /* !UNIV_HOTBACKUP */ os_mutex_enter(os_file_count_mutex); os_n_pending_writes--; @@ -2562,7 +2686,9 @@ retry: } # endif /* UNIV_DO_FLUSH */ +#ifndef UNIV_HOTBACKUP os_mutex_exit(os_file_seek_mutexes[i]); +#endif /* !UNIV_HOTBACKUP */ os_mutex_enter(os_file_count_mutex); os_n_pending_writes--; @@ -2988,6 +3114,34 @@ os_aio_array_create( return(array); } +/************************************************************************//** +Frees an aio wait array. */ +static +void +os_aio_array_free( +/*==============*/ + os_aio_array_t* array) /*!< in, own: array to free */ +{ +#ifdef WIN_ASYNC_IO + ulint i; + + for (i = 0; i < array->n_slots; i++) { + os_aio_slot_t* slot = os_aio_array_get_nth_slot(array, i); + os_event_free(slot->event); + } +#endif /* WIN_ASYNC_IO */ + +#ifdef __WIN__ + ut_free(array->native_events); +#endif /* __WIN__ */ + os_mutex_free(array->mutex); + os_event_free(array->not_full); + os_event_free(array->is_empty); + + ut_free(array->slots); + ut_free(array); +} + /*********************************************************************** Initializes the asynchronous io system. Creates one array each for ibuf and log i/o. Also creates one array each for read and write where each @@ -3061,6 +3215,35 @@ os_aio_init( } +/*********************************************************************** +Frees the asynchronous io system. */ +UNIV_INTERN +void +os_aio_free(void) +/*=============*/ +{ + ulint i; + + os_aio_array_free(os_aio_ibuf_array); + os_aio_ibuf_array = NULL; + os_aio_array_free(os_aio_log_array); + os_aio_log_array = NULL; + os_aio_array_free(os_aio_read_array); + os_aio_read_array = NULL; + os_aio_array_free(os_aio_write_array); + os_aio_write_array = NULL; + os_aio_array_free(os_aio_sync_array); + os_aio_sync_array = NULL; + + for (i = 0; i < os_aio_n_segments; i++) { + os_event_free(os_aio_segment_wait_events[i]); + } + + ut_free(os_aio_segment_wait_events); + os_aio_segment_wait_events = 0; + os_aio_n_segments = 0; +} + #ifdef WIN_ASYNC_IO /************************************************************************//** Wakes up all async i/o threads in the array in Windows async i/o at @@ -3211,7 +3394,8 @@ os_aio_array_reserve_slot( offset */ ulint offset_high, /*!< in: most significant 32 bits of offset */ - ulint len) /*!< in: length of the block to read or write */ + ulint len, /*!< in: length of the block to read or write */ + trx_t* trx) { os_aio_slot_t* slot; #ifdef WIN_ASYNC_IO @@ -3432,9 +3616,21 @@ void os_aio_simulated_put_read_threads_to_sleep(void) /*============================================*/ { + +/* The idea of putting background IO threads to sleep is only for +Windows when using simulated AIO. Windows XP seems to schedule +background threads too eagerly to allow for coalescing during +readahead requests. */ +#ifdef __WIN__ os_aio_array_t* array; ulint g; + if (os_aio_use_native_aio) { + /* We do not use simulated aio: do nothing */ + + return; + } + os_aio_recommend_sleep_for_read_threads = TRUE; for (g = 0; g < os_aio_n_segments; g++) { @@ -3445,6 +3641,7 @@ os_aio_simulated_put_read_threads_to_sle os_event_reset(os_aio_segment_wait_events[g]); } } +#endif /* __WIN__ */ } /*******************************************************************//** @@ -3482,10 +3679,11 @@ os_aio( (can be used to identify a completed aio operation); ignored if mode is OS_AIO_SYNC */ - void* message2)/*!< in: message for the aio handler + void* message2,/*!< in: message for the aio handler (can be used to identify a completed aio operation); ignored if mode is OS_AIO_SYNC */ + trx_t* trx) { os_aio_array_t* array; os_aio_slot_t* slot; @@ -3524,8 +3722,8 @@ os_aio( wait in the Windows case. */ if (type == OS_FILE_READ) { - return(os_file_read(file, buf, offset, - offset_high, n)); + return(_os_file_read(file, buf, offset, + offset_high, n, trx)); } ut_a(type == OS_FILE_WRITE); @@ -3558,8 +3756,13 @@ try_again: ut_error; } + if (trx && type == OS_FILE_READ) + { + trx->io_reads++; + trx->io_read += n; + } slot = os_aio_array_reserve_slot(type, array, message1, message2, file, - name, buf, offset, offset_high, n); + name, buf, offset, offset_high, n, trx); if (type == OS_FILE_READ) { if (os_aio_use_native_aio) { #ifdef WIN_ASYNC_IO @@ -3679,6 +3882,7 @@ os_aio_windows_handle( ibool ret_val; BOOL ret; DWORD len; + BOOL retry = FALSE; if (segment == ULINT_UNDEFINED) { array = os_aio_sync_array; @@ -3732,14 +3936,52 @@ os_aio_windows_handle( ut_a(TRUE == os_file_flush(slot->file)); } #endif /* UNIV_DO_FLUSH */ + } else if (os_file_handle_error(slot->name, "Windows aio")) { + + retry = TRUE; } else { - os_file_handle_error(slot->name, "Windows aio"); ret_val = FALSE; } os_mutex_exit(array->mutex); + if (retry) { + /* retry failed read/write operation synchronously. + No need to hold array->mutex. */ + + switch (slot->type) { + case OS_FILE_WRITE: + ret = WriteFile(slot->file, slot->buf, + slot->len, &len, + &(slot->control)); + + break; + case OS_FILE_READ: + ret = ReadFile(slot->file, slot->buf, + slot->len, &len, + &(slot->control)); + + break; + default: + ut_error; + } + + if (!ret && GetLastError() == ERROR_IO_PENDING) { + /* aio was queued successfully! + We want a synchronous i/o operation on a + file where we also use async i/o: in Windows + we must use the same wait mechanism as for + async i/o */ + + ret = GetOverlappedResult(slot->file, + &(slot->control), + &len, TRUE); + } + + ret_val = ret && len == slot->len; + } + os_aio_array_free_slot(array, slot); return(ret_val); @@ -4018,6 +4260,18 @@ consecutive_loop: } } + if (srv_recovery_stats && recv_recovery_is_on() && n_consecutive) { + mutex_enter(&(recv_sys->mutex)); + if (slot->type == OS_FILE_READ) { + recv_sys->stats_read_io_pages += n_consecutive; + recv_sys->stats_read_io_consecutive[n_consecutive - 1]++; + } else if (slot->type == OS_FILE_WRITE) { + recv_sys->stats_write_io_pages += n_consecutive; + recv_sys->stats_write_io_consecutive[n_consecutive - 1]++; + } + mutex_exit(&(recv_sys->mutex)); + } + os_mutex_enter(array->mutex); /* Mark the i/os done in slots */ === modified file 'storage/xtradb/os/os0proc.c' --- a/storage/xtradb/os/os0proc.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/os/os0proc.c 2010-01-06 12:00:14 +0000 @@ -97,6 +97,7 @@ os_mem_alloc_large( fprintf(stderr, "InnoDB: HugeTLB: Warning: Failed to" " attach shared memory segment, errno %d\n", errno); + ptr = NULL; } /* Remove the shared memory segment so that it will be === modified file 'storage/xtradb/os/os0sync.c' --- a/storage/xtradb/os/os0sync.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/os/os0sync.c 2010-01-06 12:00:14 +0000 @@ -86,6 +86,9 @@ os_sync_init(void) UT_LIST_INIT(os_event_list); UT_LIST_INIT(os_mutex_list); + os_sync_mutex = NULL; + os_sync_mutex_inited = FALSE; + os_sync_mutex = os_mutex_create(NULL); os_sync_mutex_inited = TRUE; @@ -713,6 +716,7 @@ os_fast_mutex_free( os_mutex_enter(os_sync_mutex); } + ut_ad(os_fast_mutex_count > 0); os_fast_mutex_count--; if (UNIV_LIKELY(os_sync_mutex_inited)) { === modified file 'storage/xtradb/os/os0thread.c' --- a/storage/xtradb/os/os0thread.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/os/os0thread.c 2010-01-06 12:00:14 +0000 @@ -233,6 +233,7 @@ os_thread_exit( #ifdef __WIN__ ExitThread((DWORD)exit_value); #else + pthread_detach(pthread_self()); pthread_exit(exit_value); #endif } === modified file 'storage/xtradb/page/page0cur.c' --- a/storage/xtradb/page/page0cur.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/page/page0cur.c 2010-01-06 12:00:14 +0000 @@ -1195,7 +1195,7 @@ page_cur_insert_rec_zip_reorg( } /* Out of space: restore the page */ - if (!page_zip_decompress(page_zip, page)) { + if (!page_zip_decompress(page_zip, page, FALSE)) { ut_error; /* Memory corrupted? */ } ut_ad(page_validate(page, index)); === modified file 'storage/xtradb/page/page0page.c' --- a/storage/xtradb/page/page0page.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/page/page0page.c 2010-01-06 12:00:14 +0000 @@ -45,7 +45,7 @@ Created 2/2/1994 Heikki Tuuri ============== The index page consists of a page header which contains the page's -id and other information. On top of it are the the index records +id and other information. On top of it are the index records in a heap linked into a one way linear list according to alphabetic order. Just below page end is an array of pointers which we call page directory, @@ -679,7 +679,7 @@ page_copy_rec_list_end( if (UNIV_UNLIKELY (!page_zip_decompress(new_page_zip, - new_page))) { + new_page, FALSE))) { ut_error; } ut_ad(page_validate(new_page, index)); @@ -792,7 +792,7 @@ page_copy_rec_list_start( if (UNIV_UNLIKELY (!page_zip_decompress(new_page_zip, - new_page))) { + new_page, FALSE))) { ut_error; } ut_ad(page_validate(new_page, index)); === modified file 'storage/xtradb/page/page0zip.c' --- a/storage/xtradb/page/page0zip.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/page/page0zip.c 2010-01-06 12:00:14 +0000 @@ -47,8 +47,10 @@ Created June 2005 by Marko Makela # define buf_LRU_stat_inc_unzip() ((void) 0) #endif /* !UNIV_HOTBACKUP */ +#ifndef UNIV_HOTBACKUP /** Statistics on compression, indexed by page_zip_des_t::ssize - 1 */ UNIV_INTERN page_zip_stat_t page_zip_stat[PAGE_ZIP_NUM_SSIZE - 1]; +#endif /* !UNIV_HOTBACKUP */ /* Please refer to ../include/page0zip.ic for a description of the compressed page format. */ @@ -1144,7 +1146,9 @@ page_zip_compress( ulint* offsets = NULL; ulint n_blobs = 0; byte* storage;/* storage of uncompressed columns */ +#ifndef UNIV_HOTBACKUP ullint usec = ut_time_us(NULL); +#endif /* !UNIV_HOTBACKUP */ #ifdef PAGE_ZIP_COMPRESS_DBG FILE* logfile = NULL; #endif @@ -1208,7 +1212,9 @@ page_zip_compress( } } #endif /* PAGE_ZIP_COMPRESS_DBG */ +#ifndef UNIV_HOTBACKUP page_zip_stat[page_zip->ssize - 1].compressed++; +#endif /* !UNIV_HOTBACKUP */ if (UNIV_UNLIKELY(n_dense * PAGE_ZIP_DIR_SLOT_SIZE >= page_zip_get_size(page_zip))) { @@ -1345,8 +1351,10 @@ err_exit: fclose(logfile); } #endif /* PAGE_ZIP_COMPRESS_DBG */ +#ifndef UNIV_HOTBACKUP page_zip_stat[page_zip->ssize - 1].compressed_usec += ut_time_us(NULL) - usec; +#endif /* !UNIV_HOTBACKUP */ return(FALSE); } @@ -1404,12 +1412,14 @@ err_exit: fclose(logfile); } #endif /* PAGE_ZIP_COMPRESS_DBG */ +#ifndef UNIV_HOTBACKUP { page_zip_stat_t* zip_stat = &page_zip_stat[page_zip->ssize - 1]; zip_stat->compressed_ok++; zip_stat->compressed_usec += ut_time_us(NULL) - usec; } +#endif /* !UNIV_HOTBACKUP */ return(TRUE); } @@ -2811,7 +2821,11 @@ page_zip_decompress( /*================*/ page_zip_des_t* page_zip,/*!< in: data, ssize; out: m_start, m_end, m_nonempty, n_blobs */ - page_t* page) /*!< out: uncompressed page, may be trashed */ + page_t* page, /*!< out: uncompressed page, may be trashed */ + ibool all) /*!< in: TRUE=decompress the whole page; + FALSE=verify but do not copy some + page header fields that should not change + after page creation */ { z_stream d_stream; dict_index_t* index = NULL; @@ -2820,7 +2834,9 @@ page_zip_decompress( ulint trx_id_col = ULINT_UNDEFINED; mem_heap_t* heap; ulint* offsets; +#ifndef UNIV_HOTBACKUP ullint usec = ut_time_us(NULL); +#endif /* !UNIV_HOTBACKUP */ ut_ad(page_zip_simple_validate(page_zip)); UNIV_MEM_ASSERT_W(page, UNIV_PAGE_SIZE); @@ -2839,13 +2855,36 @@ page_zip_decompress( heap = mem_heap_create(n_dense * (3 * sizeof *recs) + UNIV_PAGE_SIZE); recs = mem_heap_alloc(heap, n_dense * (2 * sizeof *recs)); + if (all) { + /* Copy the page header. */ + memcpy(page, page_zip->data, PAGE_DATA); + } else { + /* Check that the bytes that we skip are identical. */ +#if defined UNIV_DEBUG || defined UNIV_ZIP_DEBUG + ut_a(!memcmp(FIL_PAGE_TYPE + page, + FIL_PAGE_TYPE + page_zip->data, + PAGE_HEADER - FIL_PAGE_TYPE)); + ut_a(!memcmp(PAGE_HEADER + PAGE_LEVEL + page, + PAGE_HEADER + PAGE_LEVEL + page_zip->data, + PAGE_DATA - (PAGE_HEADER + PAGE_LEVEL))); +#endif /* UNIV_DEBUG || UNIV_ZIP_DEBUG */ + + /* Copy the mutable parts of the page header. */ + memcpy(page, page_zip->data, FIL_PAGE_TYPE); + memcpy(PAGE_HEADER + page, PAGE_HEADER + page_zip->data, + PAGE_LEVEL - PAGE_N_DIR_SLOTS); + +#if defined UNIV_DEBUG || defined UNIV_ZIP_DEBUG + /* Check that the page headers match after copying. */ + ut_a(!memcmp(page, page_zip->data, PAGE_DATA)); +#endif /* UNIV_DEBUG || UNIV_ZIP_DEBUG */ + } + #ifdef UNIV_ZIP_DEBUG - /* Clear the page. */ - memset(page, 0x55, UNIV_PAGE_SIZE); + /* Clear the uncompressed page, except the header. */ + memset(PAGE_DATA + page, 0x55, UNIV_PAGE_SIZE - PAGE_DATA); #endif /* UNIV_ZIP_DEBUG */ - UNIV_MEM_INVALID(page, UNIV_PAGE_SIZE); - /* Copy the page header. */ - memcpy(page, page_zip->data, PAGE_DATA); + UNIV_MEM_INVALID(PAGE_DATA + page, UNIV_PAGE_SIZE - PAGE_DATA); /* Copy the page directory. */ if (UNIV_UNLIKELY(!page_zip_dir_decode(page_zip, page, recs, @@ -2976,12 +3015,14 @@ err_exit: page_zip_fields_free(index); mem_heap_free(heap); +#ifndef UNIV_HOTBACKUP { page_zip_stat_t* zip_stat = &page_zip_stat[page_zip->ssize - 1]; zip_stat->decompressed++; zip_stat->decompressed_usec += ut_time_us(NULL) - usec; } +#endif /* !UNIV_HOTBACKUP */ /* Update the stat counter for LRU policy. */ buf_LRU_stat_inc_unzip(); @@ -3084,7 +3125,7 @@ page_zip_validate_low( #endif /* UNIV_DEBUG_VALGRIND */ temp_page_zip = *page_zip; - valid = page_zip_decompress(&temp_page_zip, temp_page); + valid = page_zip_decompress(&temp_page_zip, temp_page, TRUE); if (!valid) { fputs("page_zip_validate(): failed to decompress\n", stderr); goto func_exit; @@ -4362,8 +4403,8 @@ IMPORTANT: if page_zip_reorganize() is i non-clustered index, the caller must update the insert buffer free bits in the same mini-transaction in such a way that the modification will be redo-logged. -@return TRUE on success, FALSE on failure; page and page_zip will be -left intact on failure. */ +@return TRUE on success, FALSE on failure; page_zip will be left +intact on failure, but page will be overwritten. */ UNIV_INTERN ibool page_zip_reorganize( @@ -4428,9 +4469,6 @@ page_zip_reorganize( if (UNIV_UNLIKELY(!page_zip_compress(page_zip, page, index, mtr))) { - /* Restore the old page and exit. */ - buf_frame_copy(page, temp_page); - #ifndef UNIV_HOTBACKUP buf_block_free(temp_block); #endif /* !UNIV_HOTBACKUP */ @@ -4591,7 +4629,8 @@ corrupt: memcpy(page_zip->data + page_zip_get_size(page_zip) - trailer_size, ptr + 8 + size, trailer_size); - if (UNIV_UNLIKELY(!page_zip_decompress(page_zip, page))) { + if (UNIV_UNLIKELY(!page_zip_decompress(page_zip, page, + TRUE))) { goto corrupt; } === modified file 'storage/xtradb/pars/lexyy.c' --- a/storage/xtradb/pars/lexyy.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/pars/lexyy.c 2010-01-17 11:41:32 +0000 @@ -2778,3 +2778,18 @@ static void yyfree (void * ptr ) + +/********************************************************************** +Release any resources used by the lexer. */ +UNIV_INTERN +void +pars_lexer_close(void) +/*==================*/ +{ + if (yy_buffer_stack) + yylex_destroy(); + if (stringbuf) + free(stringbuf); + stringbuf = NULL; + stringbuf_len_alloc = stringbuf_len = 0; +} === modified file 'storage/xtradb/pars/pars0lex.l' --- a/storage/xtradb/pars/pars0lex.l 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/pars/pars0lex.l 2010-01-17 11:41:32 +0000 @@ -661,3 +661,18 @@ In the state 'id', only two actions are } %% + +/********************************************************************** +Release any resources used by the lexer. */ +UNIV_INTERN +void +pars_lexer_close(void) +/*==================*/ +{ + if (yy_buffer_stack) + yylex_destroy(); + if (stringbuf) + free(stringbuf); + stringbuf = NULL; + stringbuf_len_alloc = stringbuf_len = 0; +} === modified file 'storage/xtradb/plug.in' --- a/storage/xtradb/plug.in 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/plug.in 2010-01-15 15:58:25 +0000 @@ -40,19 +40,11 @@ MYSQL_PLUGIN_ACTIONS(innobase, [ irix*|osf*|sysv5uw7*|openbsd*) CFLAGS="$CFLAGS -DUNIV_MUST_NOT_INLINE";; *solaris*|*SunOS*) - # Begin Solaris atomic function checks - AC_CHECK_FUNCS(atomic_cas_ulong atomic_cas_32 \ - atomic_cas_64 atomic_add_long, - AC_DEFINE( - [HAVE_SOLARIS_ATOMICS], - [1], - [Define to 1 if Solaris supports \ - atomic functions.])) - ### End Solaris atomic function checks - CFLAGS="$CFLAGS -DUNIV_SOLARIS";; esac + INNODB_DYNAMIC_CFLAGS="-DMYSQL_DYNAMIC_PLUGIN" + case "$target_cpu" in x86_64) # The AMD64 ABI forbids absolute addresses in shared libraries @@ -63,7 +55,60 @@ MYSQL_PLUGIN_ACTIONS(innobase, [ ;; esac AC_SUBST(INNODB_DYNAMIC_CFLAGS) + + AC_MSG_CHECKING(whether GCC atomic builtins are available) + # either define HAVE_IB_GCC_ATOMIC_BUILTINS or not + AC_TRY_RUN( + [ + int main() + { + long x; + long y; + long res; + char c; + + x = 10; + y = 123; + res = __sync_bool_compare_and_swap(&x, x, y); + if (!res || x != y) { + return(1); + } + + x = 10; + y = 123; + res = __sync_bool_compare_and_swap(&x, x + 1, y); + if (res || x != 10) { + return(1); + } + + x = 10; + y = 123; + res = __sync_add_and_fetch(&x, y); + if (res != 123 + 10 || x != 123 + 10) { + return(1); + } + + c = 10; + res = __sync_lock_test_and_set(&c, 123); + if (res != 10 || c != 123) { + return(1); + } + + return(0); + } + ], + [ + AC_DEFINE([HAVE_IB_GCC_ATOMIC_BUILTINS], [1], + [GCC atomic builtins are available]) + AC_MSG_RESULT(yes) + ], + [ + AC_MSG_RESULT(no) + ] + ) + AC_MSG_CHECKING(whether pthread_t can be used by GCC atomic builtins) + # either define HAVE_IB_ATOMIC_PTHREAD_T_GCC or not AC_TRY_RUN( [ #include <pthread.h> @@ -84,47 +129,73 @@ MYSQL_PLUGIN_ACTIONS(innobase, [ } ], [ - AC_DEFINE([HAVE_ATOMIC_PTHREAD_T], [1], + AC_DEFINE([HAVE_IB_ATOMIC_PTHREAD_T_GCC], [1], [pthread_t can be used by GCC atomic builtins]) AC_MSG_RESULT(yes) ], [ AC_MSG_RESULT(no) ] - ) + ) + + AC_MSG_CHECKING(whether Solaris libc atomic functions are available) + # either define HAVE_IB_SOLARIS_ATOMICS or not + AC_CHECK_FUNCS(atomic_add_long \ + atomic_cas_32 \ + atomic_cas_64 \ + atomic_cas_ulong, + + AC_DEFINE([HAVE_IB_SOLARIS_ATOMICS], [1], + [Define to 1 if Solaris libc atomic functions \ + are available]) + ) + + AC_MSG_CHECKING(whether pthread_t can be used by Solaris libc atomic functions) + # either define HAVE_IB_ATOMIC_PTHREAD_T_SOLARIS or not + AC_TRY_RUN( + [ + #include <pthread.h> + #include <string.h> - # Try using solaris atomics on SunOS if GCC atomics are not available - AC_CHECK_DECLS( - [HAVE_ATOMIC_PTHREAD_T], - [ - AC_MSG_NOTICE(no need to check pthread_t size) - ], - [ - AC_CHECK_DECLS( - [HAVE_SOLARIS_ATOMICS], - [ - AC_MSG_CHECKING(checking if pthread_t size is integral) - AC_TRY_RUN( - [ - #include <pthread.h> - int main() - { - pthread_t x = 0; - return(0); - } - ], - [ - AC_DEFINE([HAVE_ATOMIC_PTHREAD_T], [1], + int main(int argc, char** argv) { + pthread_t x1; + pthread_t x2; + pthread_t x3; + + memset(&x1, 0x0, sizeof(x1)); + memset(&x2, 0x0, sizeof(x2)); + memset(&x3, 0x0, sizeof(x3)); + + if (sizeof(pthread_t) == 4) { + + atomic_cas_32(&x1, x2, x3); + + } else if (sizeof(pthread_t) == 8) { + + atomic_cas_64(&x1, x2, x3); + + } else { + + return(1); + } + + return(0); + } + ], + [ + AC_DEFINE([HAVE_IB_ATOMIC_PTHREAD_T_SOLARIS], [1], [pthread_t can be used by solaris atomics]) - AC_MSG_RESULT(yes) - # size of pthread_t is needed for typed solaris atomics - AC_CHECK_SIZEOF([pthread_t], [], [#include <pthread.h>]) - ], - [ - AC_MSG_RESULT(no) - ]) - ]) - ]) + AC_MSG_RESULT(yes) + ], + [ + AC_MSG_RESULT(no) + ] + ) + + # this is needed to know which one of atomic_cas_32() or atomic_cas_64() + # to use in the source + AC_CHECK_SIZEOF([pthread_t], [], [#include <pthread.h>]) + # Check for x86 PAUSE instruction AC_MSG_CHECKING(for x86 PAUSE instruction) # We have to actually try running the test program, because of a bug @@ -141,7 +212,7 @@ MYSQL_PLUGIN_ACTIONS(innobase, [ } ], [ - AC_DEFINE([IB_HAVE_PAUSE_INSTRUCTION], [1], [Does x86 PAUSE instruction exist]) + AC_DEFINE([HAVE_IB_PAUSE_INSTRUCTION], [1], [Does x86 PAUSE instruction exist]) AC_MSG_RESULT(yes) ], [ === modified file 'storage/xtradb/que/que0que.c' --- a/storage/xtradb/que/que0que.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/que/que0que.c 2010-01-06 12:00:14 +0000 @@ -518,6 +518,7 @@ que_graph_free_recursive( upd_node_t* upd; tab_node_t* cre_tab; ind_node_t* cre_ind; + purge_node_t* purge; if (node == NULL) { @@ -579,6 +580,13 @@ que_graph_free_recursive( mem_heap_free(ins->entry_sys_heap); break; + case QUE_NODE_PURGE: + purge = node; + + mem_heap_free(purge->heap); + + break; + case QUE_NODE_UPDATE: upd = node; === modified file 'storage/xtradb/rem/rem0cmp.c' --- a/storage/xtradb/rem/rem0cmp.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/rem/rem0cmp.c 2010-01-06 12:00:14 +0000 @@ -36,7 +36,7 @@ Created 7/1/1994 Heikki Tuuri The records are put into alphabetical order in the following way: let F be the first field where two records disagree. -If there is a character in some position n where the the +If there is a character in some position n where the records disagree, the order is determined by comparison of the characters at position n, possibly after collating transformation. If there is no such character, @@ -76,7 +76,7 @@ cmp_debug_dtuple_rec_with_match( /*************************************************************//** This function is used to compare two data fields for which the data type is such that we must use MySQL code to compare them. The prototype here -must be a copy of the the one in ha_innobase.cc! +must be a copy of the one in ha_innobase.cc! @return 1, 0, -1, if a is greater, equal, less than b, respectively */ extern int @@ -399,7 +399,7 @@ next_byte: /*************************************************************//** This function is used to compare a data tuple to a physical record. Only dtuple->n_fields_cmp first fields are taken into account for -the the data tuple! If we denote by n = n_fields_cmp, then rec must +the data tuple! If we denote by n = n_fields_cmp, then rec must have either m >= n fields, or it must differ from dtuple in some of the m fields rec has. If rec has an externally stored field we do not compare it but return with value 0 if such a comparison should be === modified file 'storage/xtradb/row/row0ins.c' --- a/storage/xtradb/row/row0ins.c 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/row/row0ins.c 2010-01-15 15:58:25 +0000 @@ -141,7 +141,7 @@ row_ins_alloc_sys_fields( dfield = dtuple_get_nth_field(row, dict_col_get_no(col)); - ptr = mem_heap_alloc(heap, DATA_ROW_ID_LEN); + ptr = mem_heap_zalloc(heap, DATA_ROW_ID_LEN); dfield_set_data(dfield, ptr, DATA_ROW_ID_LEN); @@ -152,7 +152,7 @@ row_ins_alloc_sys_fields( col = dict_table_get_sys_col(table, DATA_TRX_ID); dfield = dtuple_get_nth_field(row, dict_col_get_no(col)); - ptr = mem_heap_alloc(heap, DATA_TRX_ID_LEN); + ptr = mem_heap_zalloc(heap, DATA_TRX_ID_LEN); dfield_set_data(dfield, ptr, DATA_TRX_ID_LEN); @@ -163,7 +163,7 @@ row_ins_alloc_sys_fields( col = dict_table_get_sys_col(table, DATA_ROLL_PTR); dfield = dtuple_get_nth_field(row, dict_col_get_no(col)); - ptr = mem_heap_alloc(heap, DATA_ROLL_PTR_LEN); + ptr = mem_heap_zalloc(heap, DATA_ROLL_PTR_LEN); dfield_set_data(dfield, ptr, DATA_ROLL_PTR_LEN); } @@ -1191,7 +1191,7 @@ row_ins_check_foreign_constraint( /*=============================*/ ibool check_ref,/*!< in: TRUE if we want to check that the referenced table is ok, FALSE if we - want to to check the foreign key table */ + want to check the foreign key table */ dict_foreign_t* foreign,/*!< in: foreign constraint; NOTE that the tables mentioned in it must be in the dictionary cache if they exist at all */ === modified file 'storage/xtradb/row/row0merge.c' --- a/storage/xtradb/row/row0merge.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/row/row0merge.c 2010-01-06 12:00:14 +0000 @@ -60,9 +60,19 @@ Completed by Sunny Bains and Marko Makel #ifdef UNIV_DEBUG /** Set these in order ot enable debug printout. */ /* @{ */ +/** Log the outcome of each row_merge_cmp() call, comparing records. */ static ibool row_merge_print_cmp; +/** Log each record read from temporary file. */ static ibool row_merge_print_read; +/** Log each record write to temporary file. */ static ibool row_merge_print_write; +/** Log each row_merge_blocks() call, merging two blocks of records to +a bigger one. */ +static ibool row_merge_print_block; +/** Log each block read from temporary file. */ +static ibool row_merge_print_block_read; +/** Log each block read from temporary file. */ +static ibool row_merge_print_block_write; /* @} */ #endif /* UNIV_DEBUG */ @@ -109,8 +119,9 @@ typedef struct row_merge_buf_struct row_ /** Information about temporary files used in merge sort */ struct merge_file_struct { - int fd; /*!< file descriptor */ - ulint offset; /*!< file offset */ + int fd; /*!< file descriptor */ + ulint offset; /*!< file offset (end of file) */ + ib_uint64_t n_rec; /*!< number of records in the file */ }; /** Information about temporary files used in merge sort */ @@ -682,6 +693,13 @@ row_merge_read( ib_uint64_t ofs = ((ib_uint64_t) offset) * sizeof *buf; ibool success; +#ifdef UNIV_DEBUG + if (row_merge_print_block_read) { + fprintf(stderr, "row_merge_read fd=%d ofs=%lu\n", + fd, (ulong) offset); + } +#endif /* UNIV_DEBUG */ + success = os_file_read_no_error_handling(OS_FILE_FROM_FD(fd), buf, (ulint) (ofs & 0xFFFFFFFF), (ulint) (ofs >> 32), @@ -709,6 +727,13 @@ row_merge_write( ib_uint64_t ofs = ((ib_uint64_t) offset) * sizeof(row_merge_block_t); +#ifdef UNIV_DEBUG + if (row_merge_print_block_write) { + fprintf(stderr, "row_merge_write fd=%d ofs=%lu\n", + fd, (ulong) offset); + } +#endif /* UNIV_DEBUG */ + return(UNIV_LIKELY(os_file_write("(merge)", OS_FILE_FROM_FD(fd), buf, (ulint) (ofs & 0xFFFFFFFF), (ulint) (ofs >> 32), @@ -718,7 +743,7 @@ row_merge_write( /********************************************************************//** Read a merge record. @return pointer to next record, or NULL on I/O error or end of list */ -static +static __attribute__((nonnull)) const byte* row_merge_read_rec( /*===============*/ @@ -1070,7 +1095,7 @@ row_merge_cmp( Reads clustered index of the table and create temporary files containing the index entries for the indexes to be built. @return DB_SUCCESS or error */ -static +static __attribute__((nonnull)) ulint row_merge_read_clustered_index( /*===========================*/ @@ -1175,6 +1200,12 @@ row_merge_read_clustered_index( in order to release the latch on the old page. */ if (btr_pcur_is_after_last_on_page(&pcur)) { + if (UNIV_UNLIKELY(trx_is_interrupted(trx))) { + i = 0; + err = DB_INTERRUPTED; + goto err_exit; + } + btr_pcur_store_position(&pcur, &mtr); mtr_commit(&mtr); mtr_start(&mtr); @@ -1233,6 +1264,7 @@ row_merge_read_clustered_index( if (UNIV_LIKELY (row && row_merge_buf_add(buf, row, ext))) { + file->n_rec++; continue; } @@ -1274,14 +1306,19 @@ err_exit: UNIV_MEM_INVALID(block[0], sizeof block[0]); merge_buf[i] = row_merge_buf_empty(buf); - /* Try writing the record again, now that - the buffer has been written out and emptied. */ + if (UNIV_LIKELY(row != NULL)) { + /* Try writing the record again, now + that the buffer has been written out + and emptied. */ + + if (UNIV_UNLIKELY + (!row_merge_buf_add(buf, row, ext))) { + /* An empty buffer should have enough + room for at least one record. */ + ut_error; + } - if (UNIV_UNLIKELY - (row && !row_merge_buf_add(buf, row, ext))) { - /* An empty buffer should have enough - room for at least one record. */ - ut_error; + file->n_rec++; } } @@ -1320,7 +1357,7 @@ func_exit: b2 = row_merge_write_rec(&block[2], &buf[2], b2, \ of->fd, &of->offset, \ mrec##N, offsets##N); \ - if (UNIV_UNLIKELY(!b2)) { \ + if (UNIV_UNLIKELY(!b2 || ++of->n_rec > file->n_rec)) { \ goto corrupt; \ } \ b##N = row_merge_read_rec(&block[N], &buf[N], \ @@ -1336,14 +1373,14 @@ func_exit: } while (0) /*************************************************************//** -Merge two blocks of linked lists on disk and write a bigger block. +Merge two blocks of records on disk and write a bigger block. @return DB_SUCCESS or error code */ static ulint row_merge_blocks( /*=============*/ const dict_index_t* index, /*!< in: index being created */ - merge_file_t* file, /*!< in/out: file containing + const merge_file_t* file, /*!< in: file containing index entries */ row_merge_block_t* block, /*!< in/out: 3 buffers */ ulint* foffs0, /*!< in/out: offset of first @@ -1366,6 +1403,17 @@ row_merge_blocks( ulint* offsets0;/* offsets of mrec0 */ ulint* offsets1;/* offsets of mrec1 */ +#ifdef UNIV_DEBUG + if (row_merge_print_block) { + fprintf(stderr, + "row_merge_blocks fd=%d ofs=%lu + fd=%d ofs=%lu" + " = fd=%d ofs=%lu\n", + file->fd, (ulong) *foffs0, + file->fd, (ulong) *foffs1, + of->fd, (ulong) of->offset); + } +#endif /* UNIV_DEBUG */ + heap = row_merge_heap_create(index, &offsets0, &offsets1); /* Write a record and read the next record. Split the output @@ -1438,16 +1486,88 @@ done1: } /*************************************************************//** +Copy a block of index entries. +@return TRUE on success, FALSE on failure */ +static __attribute__((nonnull)) +ibool +row_merge_blocks_copy( +/*==================*/ + const dict_index_t* index, /*!< in: index being created */ + const merge_file_t* file, /*!< in: input file */ + row_merge_block_t* block, /*!< in/out: 3 buffers */ + ulint* foffs0, /*!< in/out: input file offset */ + merge_file_t* of) /*!< in/out: output file */ +{ + mem_heap_t* heap; /*!< memory heap for offsets0, offsets1 */ + + mrec_buf_t buf[3]; /*!< buffer for handling + split mrec in block[] */ + const byte* b0; /*!< pointer to block[0] */ + byte* b2; /*!< pointer to block[2] */ + const mrec_t* mrec0; /*!< merge rec, points to block[0] */ + ulint* offsets0;/* offsets of mrec0 */ + ulint* offsets1;/* dummy offsets */ + +#ifdef UNIV_DEBUG + if (row_merge_print_block) { + fprintf(stderr, + "row_merge_blocks_copy fd=%d ofs=%lu" + " = fd=%d ofs=%lu\n", + file->fd, (ulong) foffs0, + of->fd, (ulong) of->offset); + } +#endif /* UNIV_DEBUG */ + + heap = row_merge_heap_create(index, &offsets0, &offsets1); + + /* Write a record and read the next record. Split the output + file in two halves, which can be merged on the following pass. */ + + if (!row_merge_read(file->fd, *foffs0, &block[0])) { +corrupt: + mem_heap_free(heap); + return(FALSE); + } + + b0 = block[0]; + b2 = block[2]; + + b0 = row_merge_read_rec(&block[0], &buf[0], b0, index, file->fd, + foffs0, &mrec0, offsets0); + if (UNIV_UNLIKELY(!b0 && mrec0)) { + + goto corrupt; + } + + if (mrec0) { + /* append all mrec0 to output */ + for (;;) { + ROW_MERGE_WRITE_GET_NEXT(0, goto done0); + } + } +done0: + + /* The file offset points to the beginning of the last page + that has been read. Update it to point to the next block. */ + (*foffs0)++; + + mem_heap_free(heap); + return(row_merge_write_eof(&block[2], b2, of->fd, &of->offset) + != NULL); +} + +/*************************************************************//** Merge disk files. @return DB_SUCCESS or error code */ -static +static __attribute__((nonnull)) ulint row_merge( /*======*/ + trx_t* trx, /*!< in: transaction */ const dict_index_t* index, /*!< in: index being created */ merge_file_t* file, /*!< in/out: file containing index entries */ - ulint half, /*!< in: half the file */ + ulint* half, /*!< in/out: half the file */ row_merge_block_t* block, /*!< in/out: 3 buffers */ int* tmpfd, /*!< in/out: temporary file handle */ TABLE* table) /*!< in/out: MySQL table, for @@ -1458,43 +1578,87 @@ row_merge( ulint foffs1; /*!< second input offset */ ulint error; /*!< error code */ merge_file_t of; /*!< output file */ + const ulint ihalf = *half; + /*!< half the input file */ + ulint ohalf; /*!< half the output file */ UNIV_MEM_ASSERT_W(block[0], 3 * sizeof block[0]); - ut_ad(half > 0); + ut_ad(ihalf < file->offset); of.fd = *tmpfd; of.offset = 0; + of.n_rec = 0; /* Merge blocks to the output file. */ + ohalf = 0; foffs0 = 0; - foffs1 = half; + foffs1 = ihalf; + + for (; foffs0 < ihalf && foffs1 < file->offset; foffs0++, foffs1++) { + ulint ahalf; /*!< arithmetic half the input file */ + + if (UNIV_UNLIKELY(trx_is_interrupted(trx))) { + return(DB_INTERRUPTED); + } - for (; foffs0 < half && foffs1 < file->offset; foffs0++, foffs1++) { error = row_merge_blocks(index, file, block, &foffs0, &foffs1, &of, table); if (error != DB_SUCCESS) { return(error); } + + /* Record the offset of the output file when + approximately half the output has been generated. In + this way, the next invocation of row_merge() will + spend most of the time in this loop. The initial + estimate is ohalf==0. */ + ahalf = file->offset / 2; + ut_ad(ohalf <= of.offset); + + /* Improve the estimate until reaching half the input + file size, or we can not get any closer to it. All + comparands should be non-negative when !(ohalf < ahalf) + because ohalf <= of.offset. */ + if (ohalf < ahalf || of.offset - ahalf < ohalf - ahalf) { + ohalf = of.offset; + } } - /* Copy the last block, if there is one. */ - while (foffs0 < half) { - if (!row_merge_read(file->fd, foffs0++, block) - || !row_merge_write(of.fd, of.offset++, block)) { + /* Copy the last blocks, if there are any. */ + + while (foffs0 < ihalf) { + if (UNIV_UNLIKELY(trx_is_interrupted(trx))) { + return(DB_INTERRUPTED); + } + + if (!row_merge_blocks_copy(index, file, block, &foffs0, &of)) { return(DB_CORRUPTION); } } + + ut_ad(foffs0 == ihalf); + while (foffs1 < file->offset) { - if (!row_merge_read(file->fd, foffs1++, block) - || !row_merge_write(of.fd, of.offset++, block)) { + if (UNIV_UNLIKELY(trx_is_interrupted(trx))) { + return(DB_INTERRUPTED); + } + + if (!row_merge_blocks_copy(index, file, block, &foffs1, &of)) { return(DB_CORRUPTION); } } + ut_ad(foffs1 == file->offset); + + if (UNIV_UNLIKELY(of.n_rec != file->n_rec)) { + return(DB_CORRUPTION); + } + /* Swap file descriptors for the next pass. */ *tmpfd = file->fd; *file = of; + *half = ohalf; UNIV_MEM_INVALID(block[0], 3 * sizeof block[0]); @@ -1508,6 +1672,7 @@ static ulint row_merge_sort( /*===========*/ + trx_t* trx, /*!< in: transaction */ const dict_index_t* index, /*!< in: index being created */ merge_file_t* file, /*!< in/out: file containing index entries */ @@ -1517,20 +1682,26 @@ row_merge_sort( reporting erroneous key value if applicable */ { - ulint blksz; /*!< block size */ + ulint half = file->offset / 2; + + /* The file should always contain at least one byte (the end + of file marker). Thus, it must be at least one block. */ + ut_ad(file->offset > 0); - for (blksz = 1; blksz < file->offset; blksz *= 2) { - ulint half; + do { ulint error; - ut_ad(ut_is_2pow(blksz)); - half = ut_2pow_round((file->offset + (blksz - 1)) / 2, blksz); - error = row_merge(index, file, half, block, tmpfd, table); + error = row_merge(trx, index, file, &half, + block, tmpfd, table); if (error != DB_SUCCESS) { return(error); } - } + + /* half > 0 should hold except when the file consists + of one block. No need to merge further then. */ + ut_ad(half > 0 || file->offset == 1); + } while (half < file->offset && half > 0); return(DB_SUCCESS); } @@ -1797,7 +1968,15 @@ row_merge_drop_index( static const char str1[] = "PROCEDURE DROP_INDEX_PROC () IS\n" "BEGIN\n" + /* Rename the index, so that it will be dropped by + row_merge_drop_temp_indexes() at crash recovery + if the server crashes before this trx is committed. */ + "UPDATE SYS_INDEXES SET NAME=CONCAT('" + TEMP_INDEX_PREFIX_STR "', NAME) WHERE ID = :indexid;\n" + "COMMIT WORK;\n" + /* Drop the field definitions of the index. */ "DELETE FROM SYS_FIELDS WHERE INDEX_ID = :indexid;\n" + /* Drop the index definition and the B-tree. */ "DELETE FROM SYS_INDEXES WHERE ID = :indexid\n" " AND TABLE_ID = :tableid;\n" "END;\n"; @@ -1909,6 +2088,7 @@ row_merge_file_create( { merge_file->fd = innobase_mysql_tmpfile(); merge_file->offset = 0; + merge_file->n_rec = 0; } /*********************************************************************//** @@ -2129,7 +2309,7 @@ row_merge_rename_tables( if (err != DB_SUCCESS) { err_exit: trx->error_state = DB_SUCCESS; - trx_general_rollback_for_mysql(trx, FALSE, NULL); + trx_general_rollback_for_mysql(trx, NULL); trx->error_state = DB_SUCCESS; } @@ -2331,7 +2511,7 @@ row_merge_build_indexes( sorting and inserting. */ for (i = 0; i < n_indexes; i++) { - error = row_merge_sort(indexes[i], &merge_files[i], + error = row_merge_sort(trx, indexes[i], &merge_files[i], block, &tmpfd, table); if (error == DB_SUCCESS) { === modified file 'storage/xtradb/row/row0mysql.c' --- a/storage/xtradb/row/row0mysql.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/row/row0mysql.c 2010-01-06 12:00:14 +0000 @@ -510,7 +510,7 @@ handle_new_error: switch (err) { case DB_LOCK_WAIT_TIMEOUT: if (row_rollback_on_timeout) { - trx_general_rollback_for_mysql(trx, FALSE, NULL); + trx_general_rollback_for_mysql(trx, NULL); break; } /* fall through */ @@ -526,7 +526,7 @@ handle_new_error: /* Roll back the latest, possibly incomplete insertion or update */ - trx_general_rollback_for_mysql(trx, TRUE, savept); + trx_general_rollback_for_mysql(trx, savept); } /* MySQL will roll back the latest SQL statement */ break; @@ -548,7 +548,7 @@ handle_new_error: /* Roll back the whole transaction; this resolution was added to version 3.23.43 */ - trx_general_rollback_for_mysql(trx, FALSE, NULL); + trx_general_rollback_for_mysql(trx, NULL); break; case DB_MUST_GET_MORE_FILE_SPACE: @@ -869,18 +869,22 @@ row_update_statistics_if_needed( } /*********************************************************************//** -Unlocks AUTO_INC type locks that were possibly reserved by a trx. */ +Unlocks AUTO_INC type locks that were possibly reserved by a trx. This +function should be called at the the end of an SQL statement, by the +connection thread that owns the transaction (trx->mysql_thd). */ UNIV_INTERN void row_unlock_table_autoinc_for_mysql( /*===============================*/ trx_t* trx) /*!< in/out: transaction */ { - mutex_enter(&kernel_mutex); + if (lock_trx_holds_autoinc_locks(trx)) { + mutex_enter(&kernel_mutex); - lock_release_autoinc_locks(trx); + lock_release_autoinc_locks(trx); - mutex_exit(&kernel_mutex); + mutex_exit(&kernel_mutex); + } } /*********************************************************************//** @@ -1770,7 +1774,6 @@ row_create_table_for_mysql( const char* table_name; ulint table_name_len; ulint err; - ulint i; ut_ad(trx->mysql_thread_id == os_thread_get_curr_id()); #ifdef UNIV_SYNC_DEBUG @@ -1805,15 +1808,6 @@ err_exit: goto err_exit; } - /* Check that no reserved column names are used. */ - for (i = 0; i < dict_table_get_n_user_cols(table); i++) { - if (dict_col_name_is_reserved( - dict_table_get_col_name(table, i))) { - - goto err_exit; - } - } - trx_start_if_not_started(trx); /* The table name is prefixed with the database name and a '/'. @@ -1888,7 +1882,9 @@ err_exit: if (UNIV_UNLIKELY(err != DB_SUCCESS)) { trx->error_state = DB_SUCCESS; - trx_general_rollback_for_mysql(trx, FALSE, NULL); + trx_general_rollback_for_mysql(trx, NULL); + /* TO DO: free table? The code below will dereference + table->name, though. */ } switch (err) { @@ -1907,31 +1903,6 @@ err_exit: break; case DB_DUPLICATE_KEY: - ut_print_timestamp(stderr); - fputs(" InnoDB: Error: table ", stderr); - ut_print_name(stderr, trx, TRUE, table->name); - fputs(" already exists in InnoDB internal\n" - "InnoDB: data dictionary. Have you deleted" - " the .frm file\n" - "InnoDB: and not used DROP TABLE?" - " Have you used DROP DATABASE\n" - "InnoDB: for InnoDB tables in" - " MySQL version <= 3.23.43?\n" - "InnoDB: See the Restrictions section" - " of the InnoDB manual.\n" - "InnoDB: You can drop the orphaned table" - " inside InnoDB by\n" - "InnoDB: creating an InnoDB table with" - " the same name in another\n" - "InnoDB: database and copying the .frm file" - " to the current database.\n" - "InnoDB: Then MySQL thinks the table exists," - " and DROP TABLE will\n" - "InnoDB: succeed.\n" - "InnoDB: You can look for further help from\n" - "InnoDB: " REFMAN "innodb-troubleshooting.html\n", - stderr); - /* We may also get err == DB_ERROR if the .ibd file for the table already exists */ @@ -2056,7 +2027,7 @@ error_handling: trx->error_state = DB_SUCCESS; - trx_general_rollback_for_mysql(trx, FALSE, NULL); + trx_general_rollback_for_mysql(trx, NULL); row_drop_table_for_mysql(table_name, trx, FALSE); @@ -2077,7 +2048,7 @@ Scans a table create SQL string and adds the foreign key constraints declared in the string. This function should be called after the indexes for a table have been created. Each foreign key constraint must be accompanied with indexes in -bot participating tables. The indexes are allowed to contain more +both participating tables. The indexes are allowed to contain more fields than mentioned in the constraint. Check also that foreign key constraints which reference this table are ok. @return error code or DB_SUCCESS */ @@ -2124,7 +2095,7 @@ row_table_add_foreign_constraints( trx->error_state = DB_SUCCESS; - trx_general_rollback_for_mysql(trx, FALSE, NULL); + trx_general_rollback_for_mysql(trx, NULL); row_drop_table_for_mysql(name, trx, FALSE); @@ -2491,7 +2462,7 @@ row_discard_tablespace_for_mysql( if (err != DB_SUCCESS) { trx->error_state = DB_SUCCESS; - trx_general_rollback_for_mysql(trx, FALSE, NULL); + trx_general_rollback_for_mysql(trx, NULL); trx->error_state = DB_SUCCESS; } else { dict_table_change_id_in_cache(table, new_id); @@ -2500,7 +2471,7 @@ row_discard_tablespace_for_mysql( if (!success) { trx->error_state = DB_SUCCESS; - trx_general_rollback_for_mysql(trx, FALSE, NULL); + trx_general_rollback_for_mysql(trx, NULL); trx->error_state = DB_SUCCESS; err = DB_ERROR; @@ -2952,7 +2923,7 @@ next_rec: if (err != DB_SUCCESS) { trx->error_state = DB_SUCCESS; - trx_general_rollback_for_mysql(trx, FALSE, NULL); + trx_general_rollback_for_mysql(trx, NULL); trx->error_state = DB_SUCCESS; ut_print_timestamp(stderr); fputs(" InnoDB: Unable to assign a new identifier to table ", @@ -3593,7 +3564,7 @@ row_delete_constraint( if ((err == DB_SUCCESS) && !strchr(id, '/')) { /* Old format < 4.0.18 constraints have constraint ids - <number>_<number>. We only try deleting them if the + NUMBER_NUMBER. We only try deleting them if the constraint name does not contain a '/' character, otherwise deleting a new format constraint named 'foo/bar' from database 'baz' would remove constraint 'bar' from database @@ -3857,7 +3828,7 @@ end: "InnoDB: succeed.\n", stderr); } trx->error_state = DB_SUCCESS; - trx_general_rollback_for_mysql(trx, FALSE, NULL); + trx_general_rollback_for_mysql(trx, NULL); trx->error_state = DB_SUCCESS; } else { /* The following call will also rename the .ibd data file if @@ -3866,7 +3837,7 @@ end: if (!dict_table_rename_in_cache(table, new_name, !new_is_tmp)) { trx->error_state = DB_SUCCESS; - trx_general_rollback_for_mysql(trx, FALSE, NULL); + trx_general_rollback_for_mysql(trx, NULL); trx->error_state = DB_SUCCESS; goto funct_exit; } @@ -3906,7 +3877,7 @@ end: ut_a(dict_table_rename_in_cache(table, old_name, FALSE)); trx->error_state = DB_SUCCESS; - trx_general_rollback_for_mysql(trx, FALSE, NULL); + trx_general_rollback_for_mysql(trx, NULL); trx->error_state = DB_SUCCESS; } } @@ -4166,6 +4137,7 @@ row_check_table_for_mysql( } if (trx_is_interrupted(prebuilt->trx)) { + ret = DB_INTERRUPTED; break; } === modified file 'storage/xtradb/row/row0sel.c' --- a/storage/xtradb/row/row0sel.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/row/row0sel.c 2010-01-15 21:12:30 +0000 @@ -4616,6 +4616,7 @@ row_search_autoinc_read_column( dict_index_t* index, /*!< in: index to read from */ const rec_t* rec, /*!< in: current rec */ ulint col_no, /*!< in: column number */ + ulint mtype, /*!< in: column main type */ ibool unsigned_type) /*!< in: signed or unsigned flag */ { ulint len; @@ -4632,10 +4633,27 @@ row_search_autoinc_read_column( data = rec_get_nth_field(rec, offsets, col_no, &len); ut_a(len != UNIV_SQL_NULL); - ut_a(len <= sizeof value); /* we assume AUTOINC value cannot be negative */ - value = mach_read_int_type(data, len, unsigned_type); + switch (mtype) { + case DATA_INT: + ut_a(len <= sizeof value); + value = mach_read_int_type(data, len, unsigned_type); + break; + + case DATA_FLOAT: + ut_a(len == sizeof(float)); + value = mach_float_read(data); + break; + + case DATA_DOUBLE: + ut_a(len == sizeof(double)); + value = mach_double_read(data); + break; + + default: + ut_error; + } if (UNIV_LIKELY_NULL(heap)) { mem_heap_free(heap); @@ -4721,7 +4739,8 @@ row_search_max_autoinc( dfield->col->prtype & DATA_UNSIGNED); *value = row_search_autoinc_read_column( - index, rec, i, unsigned_type); + index, rec, i, + dfield->col->mtype, unsigned_type); } } === modified file 'storage/xtradb/scripts/install_innodb_plugins.sql' --- a/storage/xtradb/scripts/install_innodb_plugins.sql 2009-06-25 01:43:25 +0000 +++ b/storage/xtradb/scripts/install_innodb_plugins.sql 2010-01-06 12:00:14 +0000 @@ -14,3 +14,4 @@ INSTALL PLUGIN INNODB_BUFFER_POOL_PAGES_ INSTALL PLUGIN innodb_rseg SONAME 'ha_innodb.so'; INSTALL PLUGIN innodb_table_stats SONAME 'ha_innodb.so'; INSTALL PLUGIN innodb_index_stats SONAME 'ha_innodb.so'; +INSTALL PLUGIN xtradb_admin_command SONAME 'ha_innodb.so'; === modified file 'storage/xtradb/scripts/install_innodb_plugins_win.sql' --- a/storage/xtradb/scripts/install_innodb_plugins_win.sql 2008-12-03 05:06:00 +0000 +++ b/storage/xtradb/scripts/install_innodb_plugins_win.sql 2010-01-06 12:00:14 +0000 @@ -7,3 +7,11 @@ INSTALL PLUGIN innodb_cmp SONAME 'ha_inn INSTALL PLUGIN innodb_cmp_reset SONAME 'ha_innodb.dll'; INSTALL PLUGIN innodb_cmpmem SONAME 'ha_innodb.dll'; INSTALL PLUGIN innodb_cmpmem_reset SONAME 'ha_innodb.dll'; +INSTALL PLUGIN XTRADB_ENHANCEMENTS SONAME 'ha_innodb.dll'; +INSTALL PLUGIN INNODB_BUFFER_POOL_PAGES SONAME 'ha_innodb.dll'; +INSTALL PLUGIN INNODB_BUFFER_POOL_PAGES_BLOB SONAME 'ha_innodb.dll'; +INSTALL PLUGIN INNODB_BUFFER_POOL_PAGES_INDEX SONAME 'ha_innodb.dll'; +INSTALL PLUGIN innodb_rseg SONAME 'ha_innodb.dll'; +INSTALL PLUGIN innodb_table_stats SONAME 'ha_innodb.dll'; +INSTALL PLUGIN innodb_index_stats SONAME 'ha_innodb.dll'; +INSTALL PLUGIN xtradb_admin_command SONAME 'ha_innodb.dll'; === modified file 'storage/xtradb/srv/srv0srv.c' --- a/storage/xtradb/srv/srv0srv.c 2010-01-06 21:27:53 +0000 +++ b/storage/xtradb/srv/srv0srv.c 2010-01-15 19:48:33 +0000 @@ -81,7 +81,6 @@ Created 10/8/1995 Heikki Tuuri #include "ut0mem.h" #include "ut0ut.h" #include "os0proc.h" -#include "os0sync.h" #include "mem0mem.h" #include "mem0pool.h" #include "sync0sync.h" @@ -103,6 +102,10 @@ Created 10/8/1995 Heikki Tuuri #include "row0mysql.h" #include "ha_prototypes.h" #include "trx0i_s.h" +#include "os0sync.h" /* for HAVE_ATOMIC_BUILTINS */ + +/* prototypes for new functions added to ha_innodb.cc */ +ibool innobase_get_slow_log(); /* This is set to TRUE if the MySQL user has set it in MySQL; currently affects only FOREIGN KEY definition parsing */ @@ -162,8 +165,9 @@ UNIV_INTERN ulint* srv_data_file_sizes = UNIV_INTERN ibool srv_extra_undoslots = FALSE; UNIV_INTERN ibool srv_fast_recovery = FALSE; +UNIV_INTERN ibool srv_recovery_stats = FALSE; -UNIV_INTERN ibool srv_use_purge_thread = FALSE; +UNIV_INTERN ulint srv_use_purge_thread = 0; /* if TRUE, then we auto-extend the last data file */ UNIV_INTERN ibool srv_auto_extend_last_data_file = FALSE; @@ -303,12 +307,6 @@ UNIV_INTERN ulint srv_buf_pool_flushed = reading of a disk page */ UNIV_INTERN ulint srv_buf_pool_reads = 0; -/** Number of sequential read-aheads */ -UNIV_INTERN ulint srv_read_ahead_seq = 0; - -/** Number of random read-aheads */ -UNIV_INTERN ulint srv_read_ahead_rnd = 0; - /* structure to pass status variables to MySQL */ UNIV_INTERN export_struc export_vars; @@ -399,6 +397,7 @@ UNIV_INTERN ulong srv_ibuf_active_contra UNIV_INTERN ulong srv_ibuf_accel_rate = 100; #define PCT_IBUF_IO(pct) ((ulint) (srv_io_capacity * srv_ibuf_accel_rate * ((double) pct / 10000.0))) +UNIV_INTERN ulint srv_checkpoint_age_target = 0; UNIV_INTERN ulong srv_flush_neighbor_pages = 1; /* 0:disable 1:enable */ UNIV_INTERN ulong srv_enable_unsafe_group_commit = 0; /* 0:disable 1:enable */ @@ -406,6 +405,7 @@ UNIV_INTERN ulong srv_read_ahead = 3; /* UNIV_INTERN ulong srv_adaptive_checkpoint = 0; /* 0: none 1: reflex 2: estimate */ UNIV_INTERN ulong srv_expand_import = 0; /* 0:disable 1:enable */ +UNIV_INTERN ulint srv_relax_table_creation = 0; /* 0:disable 1:enable */ UNIV_INTERN ulong srv_extra_rsegments = 0; /* extra rseg for users */ UNIV_INTERN ulong srv_dict_size_limit = 0; @@ -494,8 +494,6 @@ static ulint srv_main_background_loops static ulint srv_main_flush_loops = 0; /* Log writes involving flush. */ static ulint srv_log_writes_and_flush = 0; -/* Log writes not including flush. */ -static ulint srv_log_buffer_writes = 0; /* This is only ever touched by the master thread. It records the time when the last flush of log file has happened. The master @@ -644,7 +642,7 @@ future, but at the moment we plan to imp which could be called a global priority inheritance. If a thread has to wait for a long time, say 300 milliseconds, for a resource, we just guess that it may be waiting for a resource owned by a background -thread, and boost the the priority of all runnable background threads +thread, and boost the priority of all runnable background threads to the normal level. The background threads then themselves adjust their fixed priority back to background after releasing all resources they had (or, at some fixed points in their program code). @@ -744,9 +742,8 @@ srv_print_master_thread_info( srv_main_1_second_loops, srv_main_sleeps, srv_main_10_second_loops, srv_main_background_loops, srv_main_flush_loops); - fprintf(file, "srv_master_thread log flush and writes: %lu " - " log writes only: %lu\n", - srv_log_writes_and_flush, srv_log_buffer_writes); + fprintf(file, "srv_master_thread log flush and writes: %lu\n", + srv_log_writes_and_flush); } /*********************************************************************//** @@ -1044,13 +1041,26 @@ srv_init(void) } /*********************************************************************//** -Frees the OS fast mutex created in srv_init(). */ +Frees the data structures created in srv_init(). */ UNIV_INTERN void srv_free(void) /*==========*/ { os_fast_mutex_free(&srv_conc_mutex); + mem_free(srv_conc_slots); + srv_conc_slots = NULL; + + mem_free(srv_sys->threads); + mem_free(srv_sys); + srv_sys = NULL; + + mem_free(kernel_mutex_temp); + kernel_mutex_temp = NULL; + mem_free(srv_mysql_table); + srv_mysql_table = NULL; + + trx_i_s_cache_free(trx_i_s_cache); } /*********************************************************************//** @@ -1062,6 +1072,8 @@ srv_general_init(void) /*==================*/ { ut_mem_init(); + /* Reset the system variables in the recovery module. */ + recv_sys_var_init(); os_sync_init(); sync_init(); mem_init(srv_mem_pool_size); @@ -1077,7 +1089,7 @@ UNIV_INTERN ulong srv_max_purge_lag = 0 Puts an OS thread to wait if there are too many concurrent threads (>= srv_thread_concurrency) inside InnoDB. The threads wait in a FIFO queue. */ -#ifdef INNODB_RW_LOCKS_USE_ATOMICS +#ifdef HAVE_ATOMIC_BUILTINS static void enter_innodb_with_tickets(trx_t* trx) { @@ -1155,6 +1167,10 @@ srv_conc_enter_innodb( ibool has_slept = FALSE; srv_conc_slot_t* slot = NULL; ulint i; + ib_uint64_t start_time = 0L; + ib_uint64_t finish_time = 0L; + ulint sec; + ulint ms; if (trx->mysql_thd != NULL && thd_is_replication_slave_thread(trx->mysql_thd)) { @@ -1175,7 +1191,7 @@ srv_conc_enter_innodb( return; } -#ifdef INNODB_RW_LOCKS_USE_ATOMICS +#ifdef HAVE_ATOMIC_BUILTINS if (srv_thread_concurrency_timer_based) { srv_conc_enter_innodb_timer_based(trx); return; @@ -1231,6 +1247,7 @@ retry: switches. */ if (SRV_THREAD_SLEEP_DELAY > 0) { os_thread_sleep(SRV_THREAD_SLEEP_DELAY); + trx->innodb_que_wait_timer += SRV_THREAD_SLEEP_DELAY; } trx->op_info = ""; @@ -1286,12 +1303,25 @@ retry: /* Go to wait for the event; when a thread leaves InnoDB it will release this thread */ + if (innobase_get_slow_log() && trx->take_stats) { + ut_usectime(&sec, &ms); + start_time = (ib_uint64_t)sec * 1000000 + ms; + } else { + start_time = 0; + } + trx->op_info = "waiting in InnoDB queue"; os_event_wait(slot->event); trx->op_info = ""; + if (innobase_get_slow_log() && trx->take_stats && start_time) { + ut_usectime(&sec, &ms); + finish_time = (ib_uint64_t)sec * 1000000 + ms; + trx->innodb_que_wait_timer += (ulint)(finish_time - start_time); + } + os_fast_mutex_lock(&srv_conc_mutex); srv_conc_n_waiting_threads--; @@ -1325,7 +1355,7 @@ srv_conc_force_enter_innodb( } ut_ad(srv_conc_n_threads >= 0); -#ifdef INNODB_RW_LOCKS_USE_ATOMICS +#ifdef HAVE_ATOMIC_BUILTINS if (srv_thread_concurrency_timer_based) { os_atomic_increment_lint(&srv_conc_n_threads, 1); trx->declared_to_be_inside_innodb = TRUE; @@ -1366,7 +1396,7 @@ srv_conc_force_exit_innodb( return; } -#ifdef INNODB_RW_LOCKS_USE_ATOMICS +#ifdef HAVE_ATOMIC_BUILTINS if (srv_thread_concurrency_timer_based) { srv_conc_exit_innodb_timer_based(trx); return; @@ -2081,14 +2111,16 @@ srv_export_innodb_status(void) export_vars.innodb_data_writes = os_n_file_writes; export_vars.innodb_data_written = srv_data_written; export_vars.innodb_dict_tables= (dict_sys ? UT_LIST_GET_LEN(dict_sys->table_LRU) : 0); - export_vars.innodb_buffer_pool_read_requests = buf_pool->n_page_gets; + export_vars.innodb_buffer_pool_read_requests = buf_pool->stat.n_page_gets; export_vars.innodb_buffer_pool_write_requests = srv_buf_pool_write_requests; export_vars.innodb_buffer_pool_wait_free = srv_buf_pool_wait_free; export_vars.innodb_buffer_pool_pages_flushed = srv_buf_pool_flushed; export_vars.innodb_buffer_pool_reads = srv_buf_pool_reads; - export_vars.innodb_buffer_pool_read_ahead_rnd = srv_read_ahead_rnd; - export_vars.innodb_buffer_pool_read_ahead_seq = srv_read_ahead_seq; + export_vars.innodb_buffer_pool_read_ahead + = buf_pool->stat.n_ra_pages_read; + export_vars.innodb_buffer_pool_read_ahead_evicted + = buf_pool->stat.n_ra_pages_evicted; export_vars.innodb_buffer_pool_pages_data = UT_LIST_GET_LEN(buf_pool->LRU); export_vars.innodb_buffer_pool_pages_dirty @@ -2119,9 +2151,9 @@ srv_export_innodb_status(void) export_vars.innodb_log_writes = srv_log_writes; export_vars.innodb_dblwr_pages_written = srv_dblwr_pages_written; export_vars.innodb_dblwr_writes = srv_dblwr_writes; - export_vars.innodb_pages_created = buf_pool->n_pages_created; - export_vars.innodb_pages_read = buf_pool->n_pages_read; - export_vars.innodb_pages_written = buf_pool->n_pages_written; + export_vars.innodb_pages_created = buf_pool->stat.n_pages_created; + export_vars.innodb_pages_read = buf_pool->stat.n_pages_read; + export_vars.innodb_pages_written = buf_pool->stat.n_pages_written; export_vars.innodb_row_lock_waits = srv_n_lock_wait_count; export_vars.innodb_row_lock_current_waits = srv_n_lock_wait_current_count; @@ -2488,12 +2520,6 @@ srv_sync_log_buffer_in_background(void) log_buffer_sync_in_background(TRUE); srv_last_log_flush_time = current_time; srv_log_writes_and_flush++; - } else { - /* Actually we don't need to write logs here. - We are just being extra safe here by forcing - the log buffer to log file. */ - log_buffer_sync_in_background(FALSE); - srv_log_buffer_writes++; } } @@ -2533,10 +2559,9 @@ srv_master_thread( srv_main_thread_process_no = os_proc_get_number(); srv_main_thread_id = os_thread_pf(os_thread_get_curr_id()); - srv_table_reserve_slot(SRV_MASTER); - mutex_enter(&kernel_mutex); + srv_table_reserve_slot(SRV_MASTER); srv_n_threads_active[SRV_MASTER]++; mutex_exit(&kernel_mutex); @@ -2551,8 +2576,8 @@ loop: srv_main_thread_op_info = "reserving kernel mutex"; - n_ios_very_old = log_sys->n_log_ios + buf_pool->n_pages_read - + buf_pool->n_pages_written; + n_ios_very_old = log_sys->n_log_ios + buf_pool->stat.n_pages_read + + buf_pool->stat.n_pages_written; mutex_enter(&kernel_mutex); /* Store the user activity counter at the start of this loop */ @@ -2572,8 +2597,8 @@ loop: skip_sleep = FALSE; for (i = 0; i < 10; i++) { - n_ios_old = log_sys->n_log_ios + buf_pool->n_pages_read - + buf_pool->n_pages_written; + n_ios_old = log_sys->n_log_ios + buf_pool->stat.n_pages_read + + buf_pool->stat.n_pages_written; srv_main_thread_op_info = "sleeping"; srv_main_1_second_loops++; @@ -2625,8 +2650,8 @@ loop: n_pend_ios = buf_get_n_pending_ios() + log_sys->n_pending_writes; - n_ios = log_sys->n_log_ios + buf_pool->n_pages_read - + buf_pool->n_pages_written; + n_ios = log_sys->n_log_ios + buf_pool->stat.n_pages_read + + buf_pool->stat.n_pages_written; if (n_pend_ios < SRV_PEND_IO_THRESHOLD && (n_ios - n_ios_old < SRV_RECENT_IO_ACTIVITY)) { srv_main_thread_op_info = "doing insert buffer merge"; @@ -2642,6 +2667,8 @@ loop: /* Try to keep the number of modified pages in the buffer pool under the limit wished by the user */ + srv_main_thread_op_info = + "flushing buffer pool pages"; n_pages_flushed = buf_flush_batch(BUF_FLUSH_LIST, PCT_IO(100), IB_ULONGLONG_MAX); @@ -2664,6 +2691,8 @@ loop: ulint n_flush = buf_flush_get_desired_flush_rate(); if (n_flush) { + srv_main_thread_op_info = + "flushing buffer pool pages"; n_flush = ut_min(PCT_IO(100), n_flush); n_pages_flushed = buf_flush_batch( @@ -2835,8 +2864,8 @@ retry_flush_batch: are not required, and may be disabled. */ n_pend_ios = buf_get_n_pending_ios() + log_sys->n_pending_writes; - n_ios = log_sys->n_log_ios + buf_pool->n_pages_read - + buf_pool->n_pages_written; + n_ios = log_sys->n_log_ios + buf_pool->stat.n_pages_read + + buf_pool->stat.n_pages_written; srv_main_10_second_loops++; if (n_pend_ios < SRV_PEND_IO_THRESHOLD @@ -3130,20 +3159,34 @@ srv_purge_thread( ulint n_pages_purged_sum = 1; /* dummy */ ulint history_len; ulint sleep_ms= 10000; /* initial: 10 sec. */ + ibool can_be_last = FALSE; #ifdef UNIV_DEBUG_THREAD_CREATION fprintf(stderr, "Purge thread starts, id %lu\n", os_thread_pf(os_thread_get_curr_id())); #endif - srv_table_reserve_slot(SRV_PURGE); mutex_enter(&kernel_mutex); + srv_table_reserve_slot(SRV_PURGE); srv_n_threads_active[SRV_PURGE]++; mutex_exit(&kernel_mutex); loop: - if (srv_fast_shutdown && srv_shutdown_state > 0) { - goto exit_func; + if (srv_shutdown_state > 0) { + if (srv_fast_shutdown) { + /* someone other should wait the end of the workers */ + goto exit_func; + } + + mutex_enter(&kernel_mutex); + if (srv_n_threads_active[SRV_PURGE_WORKER]) { + can_be_last = FALSE; + } else { + can_be_last = TRUE; + } + mutex_exit(&kernel_mutex); + + sleep_ms = 10; } os_thread_sleep( sleep_ms * 1000 ); @@ -3164,6 +3207,15 @@ loop: n_pages_purged_sum += n_pages_purged; } while (n_pages_purged); + if (srv_shutdown_state > 0 && can_be_last) { + /* the last trx_purge() is executed without workers */ + goto exit_func; + } + + if (n_pages_purged_sum) { + srv_active_wake_master_thread(); + } + if (n_pages_purged_sum == 0) sleep_ms *= 10; if (sleep_ms > 10000) @@ -3172,9 +3224,62 @@ loop: goto loop; exit_func: - /* We count the number of threads in os_thread_exit(). A created - thread should always use that to exit and not use return() to exit. */ + trx_purge_worker_wake(); /* It may not make sense. for safety only */ + /* wake master thread to flush the pages */ + srv_wake_master_thread(); + + mutex_enter(&kernel_mutex); + srv_n_threads_active[SRV_PURGE]--; + mutex_exit(&kernel_mutex); + os_thread_exit(NULL); + + OS_THREAD_DUMMY_RETURN; +} + +/************************************************************************* +A thread which is devoted to purge, for take over the master thread's +purging */ +UNIV_INTERN +os_thread_ret_t +srv_purge_worker_thread( +/*====================*/ + void* arg) +{ + ulint worker_id; /* index for array */ + + worker_id = *((ulint*)arg); + +#ifdef UNIV_DEBUG_THREAD_CREATION + fprintf(stderr, "Purge worker thread starts, id %lu\n", + os_thread_pf(os_thread_get_curr_id())); +#endif + mutex_enter(&kernel_mutex); + srv_table_reserve_slot(SRV_PURGE_WORKER); + srv_n_threads_active[SRV_PURGE_WORKER]++; + mutex_exit(&kernel_mutex); + +loop: + /* purge worker threads only works when srv_shutdown_state==0 */ + /* for safety and exactness. */ + if (srv_shutdown_state > 0) { + goto exit_func; + } + + trx_purge_worker_wait(); + + if (srv_shutdown_state > 0) { + goto exit_func; + } + + trx_purge_worker(worker_id); + + goto loop; + +exit_func: + mutex_enter(&kernel_mutex); + srv_n_threads_active[SRV_PURGE_WORKER]--; + mutex_exit(&kernel_mutex); os_thread_exit(NULL); OS_THREAD_DUMMY_RETURN; === modified file 'storage/xtradb/srv/srv0start.c' --- a/storage/xtradb/srv/srv0start.c 2009-11-13 21:26:08 +0000 +++ b/storage/xtradb/srv/srv0start.c 2010-01-17 08:41:43 +0000 @@ -103,6 +103,8 @@ Created 2/16/1996 Heikki Tuuri # include "row0row.h" # include "row0mysql.h" # include "btr0pcur.h" +# include "thr0loc.h" +# include "os0sync.h" /* for INNODB_RW_LOCKS_USE_ATOMICS */ /** Log sequence number immediately after startup */ UNIV_INTERN ib_uint64_t srv_start_lsn; @@ -141,9 +143,9 @@ static mutex_t ios_mutex; static ulint ios; /** io_handler_thread parameters for thread identification */ -static ulint n[SRV_MAX_N_IO_THREADS + 5]; +static ulint n[SRV_MAX_N_IO_THREADS + 5 + 64]; /** io_handler_thread identifiers */ -static os_thread_id_t thread_ids[SRV_MAX_N_IO_THREADS + 5]; +static os_thread_id_t thread_ids[SRV_MAX_N_IO_THREADS + 5 + 64]; /** We use this mutex to test the return value of pthread_mutex_trylock on successful locking. HP-UX does NOT return 0, though Linux et al do. */ @@ -494,6 +496,8 @@ io_handler_thread( mutex_exit(&ios_mutex); } + thr_local_free(os_thread_get_curr_id()); + /* We count the number of threads in os_thread_exit(). A created thread should always use that to exit and not use return() to exit. The thread actually never comes here because it is exited in an @@ -530,32 +534,6 @@ srv_normalize_path_for_win( #endif } -/*********************************************************************//** -Adds a slash or a backslash to the end of a string if it is missing -and the string is not empty. -@return string which has the separator if the string is not empty */ -UNIV_INTERN -char* -srv_add_path_separator_if_needed( -/*=============================*/ - char* str) /*!< in: null-terminated character string */ -{ - char* out_str; - ulint len = ut_strlen(str); - - if (len == 0 || str[len - 1] == SRV_PATH_SEPARATOR) { - - return(str); - } - - out_str = ut_malloc(len + 2); - memcpy(out_str, str, len); - out_str[len] = SRV_PATH_SEPARATOR; - out_str[len + 1] = 0; - - return(out_str); -} - #ifndef UNIV_HOTBACKUP /*********************************************************************//** Calculates the low 32 bits when a file size which is given as a number @@ -604,19 +582,24 @@ open_or_create_log_file( ulint size; ulint size_high; char name[10000]; + ulint dirnamelen; UT_NOT_USED(create_new_db); *log_file_created = FALSE; srv_normalize_path_for_win(srv_log_group_home_dirs[k]); - srv_log_group_home_dirs[k] = srv_add_path_separator_if_needed( - srv_log_group_home_dirs[k]); - ut_a(strlen(srv_log_group_home_dirs[k]) - < (sizeof name) - 10 - sizeof "ib_logfile"); - sprintf(name, "%s%s%lu", srv_log_group_home_dirs[k], - "ib_logfile", (ulong) i); + dirnamelen = strlen(srv_log_group_home_dirs[k]); + ut_a(dirnamelen < (sizeof name) - 10 - sizeof "ib_logfile"); + memcpy(name, srv_log_group_home_dirs[k], dirnamelen); + + /* Add a path separator if needed. */ + if (dirnamelen && name[dirnamelen - 1] != SRV_PATH_SEPARATOR) { + name[dirnamelen++] = SRV_PATH_SEPARATOR; + } + + sprintf(name + dirnamelen, "%s%lu", "ib_logfile", (ulong) i); files[i] = os_file_create(name, OS_FILE_CREATE, OS_FILE_NORMAL, OS_LOG_FILE, &ret); @@ -779,14 +762,22 @@ open_or_create_data_files( *create_new_db = FALSE; srv_normalize_path_for_win(srv_data_home); - srv_data_home = srv_add_path_separator_if_needed(srv_data_home); for (i = 0; i < srv_n_data_files; i++) { + ulint dirnamelen; + srv_normalize_path_for_win(srv_data_file_names[i]); + dirnamelen = strlen(srv_data_home); - ut_a(strlen(srv_data_home) + strlen(srv_data_file_names[i]) + ut_a(dirnamelen + strlen(srv_data_file_names[i]) < (sizeof name) - 1); - sprintf(name, "%s%s", srv_data_home, srv_data_file_names[i]); + memcpy(name, srv_data_home, dirnamelen); + /* Add a path separator if needed. */ + if (dirnamelen && name[dirnamelen - 1] != SRV_PATH_SEPARATOR) { + name[dirnamelen++] = SRV_PATH_SEPARATOR; + } + + strcpy(name + dirnamelen, srv_data_file_names[i]); if (srv_data_file_is_raw_partition[i] == 0) { @@ -1008,7 +999,7 @@ skip_size_check: return(DB_SUCCESS); } -/****************************************************************//** +/******************************************************************** Starts InnoDB and creates a new database if database files are not found and the user wants. @return DB_SUCCESS or error code */ @@ -1096,6 +1087,10 @@ innobase_start_or_create_for_mysql(void) "InnoDB: !!!!!!!! UNIV_SEARCH_DEBUG switched on !!!!!!!!!\n"); #endif +#ifdef UNIV_LOG_LSN_DEBUG + fprintf(stderr, + "InnoDB: !!!!!!!! UNIV_LOG_LSN_DEBUG switched on !!!!!!!!!\n"); +#endif /* UNIV_LOG_LSN_DEBUG */ #ifdef UNIV_MEM_DEBUG fprintf(stderr, "InnoDB: !!!!!!!! UNIV_MEM_DEBUG switched on !!!!!!!!!\n"); @@ -1106,34 +1101,7 @@ innobase_start_or_create_for_mysql(void) "InnoDB: The InnoDB memory heap is disabled\n"); } -#ifdef HAVE_GCC_ATOMIC_BUILTINS -# ifdef INNODB_RW_LOCKS_USE_ATOMICS - fprintf(stderr, - "InnoDB: Mutexes and rw_locks use GCC atomic builtins.\n"); -# else /* INNODB_RW_LOCKS_USE_ATOMICS */ - fprintf(stderr, - "InnoDB: Mutexes use GCC atomic builtins, rw_locks do not.\n"); -# endif /* INNODB_RW_LOCKS_USE_ATOMICS */ -#elif defined(HAVE_SOLARIS_ATOMICS) -# ifdef INNODB_RW_LOCKS_USE_ATOMICS - fprintf(stderr, - "InnoDB: Mutexes and rw_locks use Solaris atomic functions.\n"); -# else - fprintf(stderr, - "InnoDB: Mutexes use Solaris atomic functions.\n"); -# endif /* INNODB_RW_LOCKS_USE_ATOMICS */ -#elif HAVE_WINDOWS_ATOMICS -# ifdef INNODB_RW_LOCKS_USE_ATOMICS - fprintf(stderr, - "InnoDB: Mutexes and rw_locks use Windows interlocked functions.\n"); -# else - fprintf(stderr, - "InnoDB: Mutexes use Windows interlocked functions.\n"); -# endif /* INNODB_RW_LOCKS_USE_ATOMICS */ -#else /* HAVE_GCC_ATOMIC_BUILTINS */ - fprintf(stderr, - "InnoDB: Neither mutexes nor rw_locks use GCC atomic builtins.\n"); -#endif /* HAVE_GCC_ATOMIC_BUILTINS */ + fprintf(stderr, "InnoDB: %s\n", IB_ATOMICS_STARTUP_MSG); /* Since InnoDB does not currently clean up all its internal data structures in MySQL Embedded Server Library server_end(), we @@ -1142,7 +1110,7 @@ innobase_start_or_create_for_mysql(void) if (srv_start_has_been_called) { fprintf(stderr, - "InnoDB: Error:startup called second time" + "InnoDB: Error: startup called second time" " during the process lifetime.\n" "InnoDB: In the MySQL Embedded Server Library" " you cannot call server_init()\n" @@ -1409,7 +1377,7 @@ innobase_start_or_create_for_mysql(void) sum_of_new_sizes += srv_data_file_sizes[i]; } - if (sum_of_new_sizes < 640) { + if (sum_of_new_sizes < 10485760 / UNIV_PAGE_SIZE) { fprintf(stderr, "InnoDB: Error: tablespace size must be" " at least 10 MB\n"); @@ -1739,8 +1707,17 @@ innobase_start_or_create_for_mysql(void) + (1 + SRV_MAX_N_IO_THREADS)); if (srv_use_purge_thread) { + ulint i; + os_thread_create(&srv_purge_thread, NULL, thread_ids + (4 + SRV_MAX_N_IO_THREADS)); + + for (i = 0; i < srv_use_purge_thread - 1; i++) { + n[5 + i + SRV_MAX_N_IO_THREADS] = i; /* using as index for arrays in purge_sys */ + os_thread_create(&srv_purge_worker_thread, + n + (5 + i + SRV_MAX_N_IO_THREADS), + thread_ids + (5 + i + SRV_MAX_N_IO_THREADS)); + } } #ifdef UNIV_DEBUG /* buf_debug_prints = TRUE; */ @@ -1853,7 +1830,7 @@ innobase_start_or_create_for_mysql(void) /* Actually, we did not change the undo log format between 4.0 and 4.1.1, and we would not need to run purge to completion. Note also that the purge algorithm in 4.1.1 - can process the the history list again even after a full + can process the history list again even after a full purge, because our algorithm does not cut the end of the history list in all cases so that it would become empty after a full purge. That mean that we may purge 4.0 type @@ -2005,8 +1982,10 @@ innobase_shutdown_for_mysql(void) /* All the threads have exited or are just exiting; NOTE that the threads may not have completed their exit yet. Should we use pthread_join() to make sure - they have exited? Now we just sleep 0.1 seconds and - hope that is enough! */ + they have exited? If we did, we would have to + remove the pthread_detach() from + os_thread_exit(). Now we just sleep 0.1 + seconds and hope that is enough! */ os_mutex_exit(os_sync_mutex); @@ -2045,36 +2024,40 @@ innobase_shutdown_for_mysql(void) srv_misc_tmpfile = 0; } + /* This must be disabled before closing the buffer pool + and closing the data dictionary. */ + btr_search_disable(); + + ibuf_close(); + log_shutdown(); + lock_sys_close(); + thr_local_close(); trx_sys_file_format_close(); + trx_sys_close(); mutex_free(&srv_monitor_file_mutex); mutex_free(&srv_dict_tmpfile_mutex); mutex_free(&srv_misc_tmpfile_mutex); + dict_close(); + btr_search_sys_free(); /* 3. Free all InnoDB's own mutexes and the os_fast_mutexes inside them */ + os_aio_free(); sync_close(); + srv_free(); + fil_close(); /* 4. Free the os_conc_mutex and all os_events and os_mutexes */ - srv_free(); os_sync_free(); - /* Check that all read views are closed except read view owned - by a purge. */ - - if (UT_LIST_GET_LEN(trx_sys->view_list) > 1) { - fprintf(stderr, - "InnoDB: Error: all read views were not closed" - " before shutdown:\n" - "InnoDB: %lu read views open \n", - UT_LIST_GET_LEN(trx_sys->view_list) - 1); - } - - /* 5. Free all allocated memory and the os_fast_mutex created in - ut0mem.c */ + /* 5. Free all allocated memory */ + pars_lexer_close(); + log_mem_free(); buf_pool_free(); + mem_close(); ut_free_all_mem(); if (os_thread_count != 0 @@ -2106,6 +2089,7 @@ innobase_shutdown_for_mysql(void) } srv_was_started = FALSE; + srv_start_has_been_called = FALSE; return((int) DB_SUCCESS); } === modified file 'storage/xtradb/sync/sync0arr.c' --- a/storage/xtradb/sync/sync0arr.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/sync/sync0arr.c 2010-01-06 12:00:14 +0000 @@ -227,24 +227,21 @@ sync_array_create( SYNC_ARRAY_MUTEX: determines the type of mutex protecting the data structure */ { + ulint sz; sync_array_t* arr; - sync_cell_t* cell_array; - sync_cell_t* cell; - ulint i; ut_a(n_cells > 0); /* Allocate memory for the data structures */ arr = ut_malloc(sizeof(sync_array_t)); + memset(arr, 0x0, sizeof(*arr)); - cell_array = ut_malloc(sizeof(sync_cell_t) * n_cells); + sz = sizeof(sync_cell_t) * n_cells; + arr->array = ut_malloc(sz); + memset(arr->array, 0x0, sz); arr->n_cells = n_cells; - arr->n_reserved = 0; - arr->array = cell_array; arr->protection = protection; - arr->sg_count = 0; - arr->res_count = 0; /* Then create the mutex to protect the wait array complex */ if (protection == SYNC_ARRAY_OS_MUTEX) { @@ -255,13 +252,6 @@ sync_array_create( ut_error; } - for (i = 0; i < n_cells; i++) { - cell = sync_array_get_nth_cell(arr, i); - cell->wait_object = NULL; - cell->waiting = FALSE; - cell->signal_count = 0; - } - return(arr); } @@ -492,12 +482,12 @@ sync_array_cell_print( mutex = cell->old_wait_mutex; fprintf(file, - "Mutex at %p created file %s line %lu, lock var %lu\n" + "Mutex at %p '%s', lock var %lu\n" #ifdef UNIV_SYNC_DEBUG "Last time reserved in file %s line %lu, " #endif /* UNIV_SYNC_DEBUG */ "waiters flag %lu\n", - (void*) mutex, mutex->cfile_name, (ulong) mutex->cline, + (void*) mutex, mutex->cmutex_name, (ulong) mutex->lock_word, #ifdef UNIV_SYNC_DEBUG mutex->file_name, (ulong) mutex->line, @@ -513,9 +503,8 @@ sync_array_cell_print( rwlock = cell->old_wait_rw_lock; fprintf(file, - " RW-latch at %p created in file %s line %lu\n", - (void*) rwlock, rwlock->cfile_name, - (ulong) rwlock->cline); + " RW-latch at %p '%s'\n", + (void*) rwlock, rwlock->lock_name); writer = rw_lock_get_writer(rwlock); if (writer != RW_LOCK_NOT_LOCKED) { fprintf(file, === modified file 'storage/xtradb/sync/sync0rw.c' --- a/storage/xtradb/sync/sync0rw.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/sync/sync0rw.c 2010-01-06 12:00:14 +0000 @@ -38,6 +38,7 @@ Created 9/11/1995 Heikki Tuuri #include "os0thread.h" #include "mem0mem.h" #include "srv0srv.h" +#include "os0sync.h" /* for INNODB_RW_LOCKS_USE_ATOMICS */ /* IMPLEMENTATION OF THE RW_LOCK @@ -230,8 +231,8 @@ rw_lock_create_func( # ifdef UNIV_SYNC_DEBUG ulint level, /*!< in: level */ # endif /* UNIV_SYNC_DEBUG */ - const char* cmutex_name, /*!< in: mutex name */ #endif /* UNIV_DEBUG */ + const char* cmutex_name, /*!< in: mutex name */ const char* cfile_name, /*!< in: file name where created */ ulint cline) /*!< in: file line where created */ { @@ -241,14 +242,15 @@ rw_lock_create_func( #ifndef INNODB_RW_LOCKS_USE_ATOMICS mutex_create(rw_lock_get_mutex(lock), SYNC_NO_ORDER_CHECK); - lock->mutex.cfile_name = cfile_name; - lock->mutex.cline = cline; + ut_d(lock->mutex.cfile_name = cfile_name); + ut_d(lock->mutex.cline = cline); - ut_d(lock->mutex.cmutex_name = cmutex_name); + lock->mutex.cmutex_name = cmutex_name; ut_d(lock->mutex.mutex_type = 1); #else /* INNODB_RW_LOCKS_USE_ATOMICS */ # ifdef UNIV_DEBUG - UT_NOT_USED(cmutex_name); + UT_NOT_USED(cfile_name); + UT_NOT_USED(cline); # endif #endif /* INNODB_RW_LOCKS_USE_ATOMICS */ @@ -268,8 +270,7 @@ rw_lock_create_func( lock->magic_n = RW_LOCK_MAGIC_N; - lock->cfile_name = cfile_name; - lock->cline = (unsigned int) cline; + lock->lock_name = cmutex_name; lock->count_os_wait = 0; lock->last_s_file_name = "not yet reserved"; @@ -304,8 +305,6 @@ rw_lock_free( ut_ad(rw_lock_validate(lock)); ut_a(lock->lock_word == X_LOCK_DECR); - lock->magic_n = 0; - #ifndef INNODB_RW_LOCKS_USE_ATOMICS mutex_free(rw_lock_get_mutex(lock)); #endif /* INNODB_RW_LOCKS_USE_ATOMICS */ @@ -325,6 +324,8 @@ rw_lock_free( UT_LIST_REMOVE(list, rw_lock_list, lock); mutex_exit(&rw_lock_list_mutex); + + lock->magic_n = 0; } #ifdef UNIV_DEBUG @@ -390,10 +391,10 @@ lock_loop: if (srv_print_latch_waits) { fprintf(stderr, "Thread %lu spin wait rw-s-lock at %p" - " cfile %s cline %lu rnds %lu\n", + " '%s' rnds %lu\n", (ulong) os_thread_pf(os_thread_get_curr_id()), (void*) lock, - lock->cfile_name, (ulong) lock->cline, (ulong) i); + lock->lock_name, (ulong) i); } /* We try once again to obtain the lock */ @@ -426,10 +427,9 @@ lock_loop: if (srv_print_latch_waits) { fprintf(stderr, "Thread %lu OS wait rw-s-lock at %p" - " cfile %s cline %lu\n", + " '%s'\n", os_thread_pf(os_thread_get_curr_id()), - (void*) lock, lock->cfile_name, - (ulong) lock->cline); + (void*) lock, lock->lock_name); } /* these stats may not be accurate */ @@ -648,9 +648,9 @@ lock_loop: if (srv_print_latch_waits) { fprintf(stderr, "Thread %lu spin wait rw-x-lock at %p" - " cfile %s cline %lu rnds %lu\n", + " '%s' rnds %lu\n", os_thread_pf(os_thread_get_curr_id()), (void*) lock, - lock->cfile_name, (ulong) lock->cline, (ulong) i); + lock->lock_name, (ulong) i); } sync_array_reserve_cell(sync_primary_wait_array, @@ -671,9 +671,9 @@ lock_loop: if (srv_print_latch_waits) { fprintf(stderr, "Thread %lu OS wait for rw-x-lock at %p" - " cfile %s cline %lu\n", + " '%s'\n", os_thread_pf(os_thread_get_curr_id()), (void*) lock, - lock->cfile_name, (ulong) lock->cline); + lock->lock_name); } /* these stats may not be accurate */ === modified file 'storage/xtradb/sync/sync0sync.c' --- a/storage/xtradb/sync/sync0sync.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/sync/sync0sync.c 2010-01-06 12:00:14 +0000 @@ -39,6 +39,7 @@ Created 9/5/1995 Heikki Tuuri #include "buf0buf.h" #include "srv0srv.h" #include "buf0types.h" +#include "os0sync.h" /* for HAVE_ATOMIC_BUILTINS */ /* REASONS FOR IMPLEMENTING THE SPIN LOCK MUTEX @@ -237,8 +238,8 @@ void mutex_create_func( /*==============*/ mutex_t* mutex, /*!< in: pointer to memory */ -#ifdef UNIV_DEBUG const char* cmutex_name, /*!< in: mutex name */ +#ifdef UNIV_DEBUG # ifdef UNIV_SYNC_DEBUG ulint level, /*!< in: level */ # endif /* UNIV_SYNC_DEBUG */ @@ -253,7 +254,7 @@ mutex_create_func( mutex->lock_word = 0; #endif mutex->event = os_event_create(NULL); - mutex_set_waiters(mutex, 0); + mutex->waiters = 0; #ifdef UNIV_DEBUG mutex->magic_n = MUTEX_MAGIC_N; #endif /* UNIV_DEBUG */ @@ -262,11 +263,13 @@ mutex_create_func( mutex->file_name = "not yet reserved"; mutex->level = level; #endif /* UNIV_SYNC_DEBUG */ +#ifdef UNIV_DEBUG mutex->cfile_name = cfile_name; mutex->cline = cline; +#endif /* UNIV_DEBUG */ mutex->count_os_wait = 0; -#ifdef UNIV_DEBUG mutex->cmutex_name= cmutex_name; +#ifdef UNIV_DEBUG mutex->count_using= 0; mutex->mutex_type= 0; mutex->lspent_time= 0; @@ -424,10 +427,18 @@ mutex_set_waiters( the value is stored to memory */ ut_ad(mutex); +#ifdef INNODB_RW_LOCKS_USE_ATOMICS + if (n) { + os_compare_and_swap_ulint(&mutex->waiters, 0, 1); + } else { + os_compare_and_swap_ulint(&mutex->waiters, 1, 0); + } +#else ptr = &(mutex->waiters); *ptr = n; /* Here we assume that the write of a single word in memory is atomic */ +#endif } /******************************************************************//** @@ -498,9 +509,9 @@ spin_loop: #ifdef UNIV_SRV_PRINT_LATCH_WAITS fprintf(stderr, "Thread %lu spin wait mutex at %p" - " cfile %s cline %lu rnds %lu\n", + " '%s' rnds %lu\n", (ulong) os_thread_pf(os_thread_get_curr_id()), (void*) mutex, - mutex->cfile_name, (ulong) mutex->cline, (ulong) i); + mutex->cmutex_name, (ulong) i); #endif mutex_spin_round_count += i; @@ -575,9 +586,9 @@ spin_loop: #ifdef UNIV_SRV_PRINT_LATCH_WAITS fprintf(stderr, - "Thread %lu OS wait mutex at %p cfile %s cline %lu rnds %lu\n", + "Thread %lu OS wait mutex at %p '%s' rnds %lu\n", (ulong) os_thread_pf(os_thread_get_curr_id()), (void*) mutex, - mutex->cfile_name, (ulong) mutex->cline, (ulong) i); + mutex->cmutex_name, (ulong) i); #endif mutex_os_wait_count++; @@ -849,7 +860,8 @@ sync_thread_levels_g( /*=================*/ sync_level_t* arr, /*!< in: pointer to level array for an OS thread */ - ulint limit) /*!< in: level limit */ + ulint limit, /*!< in: level limit */ + ulint warn) /*!< in: TRUE=display a diagnostic message */ { sync_level_t* slot; rw_lock_t* lock; @@ -863,6 +875,11 @@ sync_thread_levels_g( if (slot->latch != NULL) { if (slot->level <= limit) { + if (!warn) { + + return(FALSE); + } + lock = slot->latch; mutex = slot->latch; @@ -873,9 +890,8 @@ sync_thread_levels_g( if (mutex->magic_n == MUTEX_MAGIC_N) { fprintf(stderr, - "Mutex created at %s %lu\n", - mutex->cfile_name, - (ulong) mutex->cline); + "Mutex '%s'\n", + mutex->cmutex_name); if (mutex_get_lock_word(mutex) != 0) { const char* file_name; @@ -1106,7 +1122,7 @@ sync_thread_add_level( case SYNC_DICT_HEADER: case SYNC_TRX_I_S_RWLOCK: case SYNC_TRX_I_S_LAST_READ: - if (!sync_thread_levels_g(array, level)) { + if (!sync_thread_levels_g(array, level, TRUE)) { fprintf(stderr, "InnoDB: sync_thread_levels_g(array, %lu)" " does not hold!\n", level); @@ -1117,36 +1133,44 @@ sync_thread_add_level( /* Either the thread must own the buffer pool mutex (buf_pool_mutex), or it is allowed to latch only ONE buffer block (block->mutex or buf_pool_zip_mutex). */ - if (!sync_thread_levels_g(array, level)) { - ut_a(sync_thread_levels_g(array, level - 1)); + if (!sync_thread_levels_g(array, level, FALSE)) { + ut_a(sync_thread_levels_g(array, level - 1, TRUE)); ut_a(sync_thread_levels_contain(array, SYNC_BUF_LRU_LIST)); } break; case SYNC_REC_LOCK: - ut_a((sync_thread_levels_contain(array, SYNC_KERNEL) - && sync_thread_levels_g(array, SYNC_REC_LOCK - 1)) - || sync_thread_levels_g(array, SYNC_REC_LOCK)); + if (sync_thread_levels_contain(array, SYNC_KERNEL)) { + ut_a(sync_thread_levels_g(array, SYNC_REC_LOCK - 1, + TRUE)); + } else { + ut_a(sync_thread_levels_g(array, SYNC_REC_LOCK, TRUE)); + } break; case SYNC_IBUF_BITMAP: /* Either the thread must own the master mutex to all the bitmap pages, or it is allowed to latch only ONE bitmap page. */ - ut_a((sync_thread_levels_contain(array, SYNC_IBUF_BITMAP_MUTEX) - && sync_thread_levels_g(array, SYNC_IBUF_BITMAP - 1)) - || sync_thread_levels_g(array, SYNC_IBUF_BITMAP)); + if (sync_thread_levels_contain(array, + SYNC_IBUF_BITMAP_MUTEX)) { + ut_a(sync_thread_levels_g(array, SYNC_IBUF_BITMAP - 1, + TRUE)); + } else { + ut_a(sync_thread_levels_g(array, SYNC_IBUF_BITMAP, + TRUE)); + } break; case SYNC_FSP_PAGE: ut_a(sync_thread_levels_contain(array, SYNC_FSP)); break; case SYNC_FSP: ut_a(sync_thread_levels_contain(array, SYNC_FSP) - || sync_thread_levels_g(array, SYNC_FSP)); + || sync_thread_levels_g(array, SYNC_FSP, TRUE)); break; case SYNC_TRX_UNDO_PAGE: ut_a(sync_thread_levels_contain(array, SYNC_TRX_UNDO) || sync_thread_levels_contain(array, SYNC_RSEG) || sync_thread_levels_contain(array, SYNC_PURGE_SYS) - || sync_thread_levels_g(array, SYNC_TRX_UNDO_PAGE)); + || sync_thread_levels_g(array, SYNC_TRX_UNDO_PAGE, TRUE)); break; case SYNC_RSEG_HEADER: ut_a(sync_thread_levels_contain(array, SYNC_RSEG)); @@ -1158,37 +1182,41 @@ sync_thread_add_level( case SYNC_TREE_NODE: ut_a(sync_thread_levels_contain(array, SYNC_INDEX_TREE) || sync_thread_levels_contain(array, SYNC_DICT_OPERATION) - || sync_thread_levels_g(array, SYNC_TREE_NODE - 1)); + || sync_thread_levels_g(array, SYNC_TREE_NODE - 1, TRUE)); break; case SYNC_TREE_NODE_NEW: ut_a(sync_thread_levels_contain(array, SYNC_FSP_PAGE) || sync_thread_levels_contain(array, SYNC_IBUF_MUTEX)); break; case SYNC_INDEX_TREE: - ut_a((sync_thread_levels_contain(array, SYNC_IBUF_MUTEX) - && sync_thread_levels_contain(array, SYNC_FSP) - && sync_thread_levels_g(array, SYNC_FSP_PAGE - 1)) - || sync_thread_levels_g(array, SYNC_TREE_NODE - 1)); + if (sync_thread_levels_contain(array, SYNC_IBUF_MUTEX) + && sync_thread_levels_contain(array, SYNC_FSP)) { + ut_a(sync_thread_levels_g(array, SYNC_FSP_PAGE - 1, + TRUE)); + } else { + ut_a(sync_thread_levels_g(array, SYNC_TREE_NODE - 1, + TRUE)); + } break; case SYNC_IBUF_MUTEX: - ut_a(sync_thread_levels_g(array, SYNC_FSP_PAGE - 1)); + ut_a(sync_thread_levels_g(array, SYNC_FSP_PAGE - 1, TRUE)); break; case SYNC_IBUF_PESS_INSERT_MUTEX: - ut_a(sync_thread_levels_g(array, SYNC_FSP - 1) - && !sync_thread_levels_contain(array, SYNC_IBUF_MUTEX)); + ut_a(sync_thread_levels_g(array, SYNC_FSP - 1, TRUE)); + ut_a(!sync_thread_levels_contain(array, SYNC_IBUF_MUTEX)); break; case SYNC_IBUF_HEADER: - ut_a(sync_thread_levels_g(array, SYNC_FSP - 1) - && !sync_thread_levels_contain(array, SYNC_IBUF_MUTEX) - && !sync_thread_levels_contain( - array, SYNC_IBUF_PESS_INSERT_MUTEX)); + ut_a(sync_thread_levels_g(array, SYNC_FSP - 1, TRUE)); + ut_a(!sync_thread_levels_contain(array, SYNC_IBUF_MUTEX)); + ut_a(!sync_thread_levels_contain(array, + SYNC_IBUF_PESS_INSERT_MUTEX)); break; case SYNC_DICT: #ifdef UNIV_DEBUG ut_a(buf_debug_prints - || sync_thread_levels_g(array, SYNC_DICT)); + || sync_thread_levels_g(array, SYNC_DICT, TRUE)); #else /* UNIV_DEBUG */ - ut_a(sync_thread_levels_g(array, SYNC_DICT)); + ut_a(sync_thread_levels_g(array, SYNC_DICT, TRUE)); #endif /* UNIV_DEBUG */ break; default: @@ -1364,7 +1392,12 @@ sync_close(void) mutex_free(&mutex_list_mutex); #ifdef UNIV_SYNC_DEBUG mutex_free(&sync_thread_mutex); + + /* Switch latching order checks on in sync0sync.c */ + sync_order_checks_on = FALSE; #endif /* UNIV_SYNC_DEBUG */ + + sync_initialized = FALSE; } /*******************************************************************//** === modified file 'storage/xtradb/thr/thr0loc.c' --- a/storage/xtradb/thr/thr0loc.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/thr/thr0loc.c 2010-01-06 12:00:14 +0000 @@ -63,7 +63,7 @@ struct thr_local_struct{ os_thread_t handle; /*!< operating system handle to the thread */ ulint slot_no;/*!< the index of the slot in the thread table for this thread */ - ibool in_ibuf;/*!< TRUE if the the thread is doing an ibuf + ibool in_ibuf;/*!< TRUE if the thread is doing an ibuf operation */ hash_node_t hash; /*!< hash chain node */ ulint magic_n;/*!< magic number (THR_LOCAL_MAGIC_N) */ @@ -250,6 +250,37 @@ thr_local_init(void) mutex_create(&thr_local_mutex, SYNC_THR_LOCAL); } +/******************************************************************** +Close the thread local storage module. */ +UNIV_INTERN +void +thr_local_close(void) +/*=================*/ +{ + ulint i; + + ut_a(thr_local_hash != NULL); + + /* Free the hash elements. We don't remove them from the table + because we are going to destroy the table anyway. */ + for (i = 0; i < hash_get_n_cells(thr_local_hash); i++) { + thr_local_t* local; + + local = HASH_GET_FIRST(thr_local_hash, i); + + while (local) { + thr_local_t* prev_local = local; + + local = HASH_GET_NEXT(hash, prev_local); + ut_a(prev_local->magic_n == THR_LOCAL_MAGIC_N); + mem_free(prev_local); + } + } + + hash_table_free(thr_local_hash); + thr_local_hash = NULL; +} + /************************************************************************* Return local hash table informations. */ === modified file 'storage/xtradb/trx/trx0i_s.c' --- a/storage/xtradb/trx/trx0i_s.c 2009-11-29 23:08:56 +0000 +++ b/storage/xtradb/trx/trx0i_s.c 2010-01-15 15:58:25 +0000 @@ -60,7 +60,7 @@ Created July 17, 2007 Vasil Dimov /** @brief The maximum number of chunks to allocate for a table cache. The rows of a table cache are stored in a set of chunks. When a new -row is added a new chunk is allocated if necessary. Assuming that the +row is added a new chunk is allocated if necessary. Assuming that the first one is 1024 rows (TABLE_CACHE_INITIAL_ROWSNUM) and each subsequent is N/2 where N is the number of rows we have allocated till now, then 39th chunk would accommodate 1677416425 rows and all chunks @@ -238,6 +238,27 @@ table_cache_init( } /*******************************************************************//** +Frees a table cache. */ +static +void +table_cache_free( +/*=============*/ + i_s_table_cache_t* table_cache) /*!< in/out: table cache */ +{ + ulint i; + + for (i = 0; i < MEM_CHUNKS_IN_TABLE_CACHE; i++) { + + /* the memory is actually allocated in + table_cache_create_empty_row() */ + if (table_cache->chunks[i].base) { + mem_free(table_cache->chunks[i].base); + table_cache->chunks[i].base = NULL; + } + } +} + +/*******************************************************************//** Returns an empty row from a table cache. The row is allocated if no more empty rows are available. The number of used rows is incremented. If the memory limit is hit then NULL is returned and nothing is @@ -1252,6 +1273,22 @@ trx_i_s_cache_init( } /*******************************************************************//** +Free the INFORMATION SCHEMA trx related cache. */ +UNIV_INTERN +void +trx_i_s_cache_free( +/*===============*/ + trx_i_s_cache_t* cache) /*!< in, own: cache to free */ +{ + hash_table_free(cache->locks_hash); + ha_storage_free(cache->storage); + table_cache_free(&cache->innodb_trx); + table_cache_free(&cache->innodb_locks); + table_cache_free(&cache->innodb_lock_waits); + memset(cache, 0, sizeof *cache); +} + +/*******************************************************************//** Issue a shared/read lock on the tables cache. */ UNIV_INTERN void === modified file 'storage/xtradb/trx/trx0purge.c' --- a/storage/xtradb/trx/trx0purge.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/trx/trx0purge.c 2010-01-06 12:00:14 +0000 @@ -184,8 +184,9 @@ this query graph. @return own: the query graph */ static que_t* -trx_purge_graph_build(void) +trx_purge_graph_build( /*=======================*/ + trx_t* trx) { mem_heap_t* heap; que_fork_t* fork; @@ -194,7 +195,7 @@ trx_purge_graph_build(void) heap = mem_heap_create(512); fork = que_fork_create(NULL, NULL, QUE_FORK_PURGE, heap); - fork->trx = purge_sys->trx; + fork->trx = trx; thr = que_thr_create(fork, heap); @@ -243,10 +244,73 @@ trx_purge_sys_create(void) ut_a(trx_start_low(purge_sys->trx, ULINT_UNDEFINED)); - purge_sys->query = trx_purge_graph_build(); + purge_sys->query = trx_purge_graph_build(purge_sys->trx); purge_sys->view = read_view_oldest_copy_or_open_new(ut_dulint_zero, purge_sys->heap); + + purge_sys->n_worker = 0; + if (srv_use_purge_thread > 1) { + /* Use worker threads */ + ulint i; + + purge_sys->n_worker = srv_use_purge_thread - 1; + + purge_sys->sess_arr = mem_alloc(sizeof(sess_t*) * purge_sys->n_worker); + purge_sys->trx_arr = mem_alloc(sizeof(trx_t*) * purge_sys->n_worker); + purge_sys->query_arr = mem_alloc(sizeof(que_t*) * purge_sys->n_worker); + + purge_sys->worker_event = os_event_create(NULL); + os_event_reset(purge_sys->worker_event); + + for (i = 0; i < purge_sys->n_worker; i++) { + purge_sys->sess_arr[i] = sess_open(); + + purge_sys->trx_arr[i] = purge_sys->sess_arr[i]->trx; + purge_sys->trx_arr[i]->is_purge = 1; + ut_a(trx_start_low(purge_sys->trx_arr[i], ULINT_UNDEFINED)); + + purge_sys->query_arr[i] = trx_purge_graph_build(purge_sys->trx_arr[i]); + } + } +} + +/************************************************************************ +Frees the global purge system control structure. */ +UNIV_INTERN +void +trx_purge_sys_close(void) +/*======================*/ +{ + ut_ad(!mutex_own(&kernel_mutex)); + + que_graph_free(purge_sys->query); + + ut_a(purge_sys->sess->trx->is_purge); + purge_sys->sess->trx->conc_state = TRX_NOT_STARTED; + sess_close(purge_sys->sess); + purge_sys->sess = NULL; + + if (purge_sys->view != NULL) { + /* Because acquiring the kernel mutex is a pre-condition + of read_view_close(). We don't really need it here. */ + mutex_enter(&kernel_mutex); + + read_view_close(purge_sys->view); + purge_sys->view = NULL; + + mutex_exit(&kernel_mutex); + } + + trx_undo_arr_free(purge_sys->arr); + + rw_lock_free(&purge_sys->latch); + mutex_free(&purge_sys->mutex); + + mem_heap_free(purge_sys->heap); + mem_free(purge_sys); + + purge_sys = NULL; } /*================ UNDO LOG HISTORY LIST =============================*/ @@ -1110,7 +1174,7 @@ trx_purge(void) /* Handle at most 20 undo log pages in one purge batch */ - purge_sys->handle_limit = purge_sys->n_pages_handled + 20; + purge_sys->handle_limit = purge_sys->n_pages_handled + 20 * (srv_use_purge_thread + 1); old_pages_handled = purge_sys->n_pages_handled; @@ -1129,6 +1193,9 @@ trx_purge(void) mutex_exit(&kernel_mutex); + if (purge_sys->n_worker) + os_event_set(purge_sys->worker_event); + /* srv_que_task_enqueue(thr2); */ if (srv_print_thread_releases) { @@ -1138,6 +1205,9 @@ trx_purge(void) que_run_threads(thr); + if (purge_sys->n_worker) + os_event_reset(purge_sys->worker_event); + if (srv_print_thread_releases) { fprintf(stderr, @@ -1148,6 +1218,52 @@ trx_purge(void) return(purge_sys->n_pages_handled - old_pages_handled); } +/********************************************************************** +This function runs a purge worker batch */ +UNIV_INTERN +void +trx_purge_worker( +/*=============*/ + ulint worker_id) +{ + que_thr_t* thr; + + mutex_enter(&kernel_mutex); + + thr = que_fork_start_command(purge_sys->query_arr[worker_id]); + + ut_ad(thr); + + mutex_exit(&kernel_mutex); + + que_run_threads(thr); + + if (purge_sys->state == TRX_STOP_PURGE) { /* optimistic */ + os_event_reset(purge_sys->worker_event); + } +} + +/********************************************************************** +This function waits the event for worker batch */ +UNIV_INTERN +void +trx_purge_worker_wait(void) +/*=======================*/ +{ + os_event_wait(purge_sys->worker_event); +} + +/********************************************************************** +This function wakes the waiting worker batch */ +UNIV_INTERN +void +trx_purge_worker_wake(void) +/*=======================*/ +{ + if (purge_sys->n_worker) + os_event_set(purge_sys->worker_event); +} + /******************************************************************//** Prints information of the purge system to stderr. */ UNIV_INTERN === modified file 'storage/xtradb/trx/trx0rec.c' --- a/storage/xtradb/trx/trx0rec.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/trx/trx0rec.c 2010-01-06 12:00:14 +0000 @@ -1333,7 +1333,7 @@ trx_undo_get_undo_rec_low( ulint rseg_id; ulint page_no; ulint offset; - page_t* undo_page; + const page_t* undo_page; trx_rseg_t* rseg; ibool is_insert; mtr_t mtr; @@ -1572,7 +1572,7 @@ trx_undo_prev_version_build( /* We have to set the appropriate extern storage bits in the old version of the record: the extern bits in rec for those - fields that update does NOT update, as well as the the bits for + fields that update does NOT update, as well as the bits for those fields that update updates to become externally stored fields. Store the info: */ === modified file 'storage/xtradb/trx/trx0roll.c' --- a/storage/xtradb/trx/trx0roll.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/trx/trx0roll.c 2010-01-06 12:00:14 +0000 @@ -66,9 +66,9 @@ int trx_general_rollback_for_mysql( /*===========================*/ trx_t* trx, /*!< in: transaction handle */ - ibool partial,/*!< in: TRUE if partial rollback requested */ trx_savept_t* savept) /*!< in: pointer to savepoint undo number, if - partial rollback requested */ + partial rollback requested, or NULL for + complete rollback */ { mem_heap_t* heap; que_thr_t* thr; @@ -85,9 +85,8 @@ trx_general_rollback_for_mysql( roll_node = roll_node_create(heap); - roll_node->partial = partial; - - if (partial) { + if (savept) { + roll_node->partial = TRUE; roll_node->savept = *savept; } @@ -145,7 +144,7 @@ trx_rollback_for_mysql( the transaction object does not have an InnoDB session object, and we set a dummy session that we use for all MySQL transactions. */ - err = trx_general_rollback_for_mysql(trx, FALSE, NULL); + err = trx_general_rollback_for_mysql(trx, NULL); trx->op_info = ""; @@ -170,8 +169,7 @@ trx_rollback_last_sql_stat_for_mysql( trx->op_info = "rollback of SQL statement"; - err = trx_general_rollback_for_mysql(trx, TRUE, - &(trx->last_sql_stat_start)); + err = trx_general_rollback_for_mysql(trx, &trx->last_sql_stat_start); /* The following call should not be needed, but we play safe: */ trx_mark_sql_stat_end(trx); @@ -282,7 +280,7 @@ trx_rollback_to_savepoint_for_mysql( trx->op_info = "rollback to a savepoint"; - err = trx_general_rollback_for_mysql(trx, TRUE, &(savep->savept)); + err = trx_general_rollback_for_mysql(trx, &savep->savept); /* Store the current undo_no of the transaction so that we know where to roll back if we have to roll back the next SQL statement: */ @@ -534,28 +532,26 @@ trx_rollback_active( Rollback or clean up any incomplete transactions which were encountered in crash recovery. If the transaction already was committed, then we clean up a possible insert undo log. If the -transaction was not yet committed, then we roll it back. -Note: this is done in a background thread. -@return a dummy parameter */ +transaction was not yet committed, then we roll it back. */ UNIV_INTERN -os_thread_ret_t -trx_rollback_or_clean_all_recovered( -/*================================*/ - void* arg __attribute__((unused))) - /*!< in: a dummy parameter required by - os_thread_create */ +void +trx_rollback_or_clean_recovered( +/*============================*/ + ibool all) /*!< in: FALSE=roll back dictionary transactions; + TRUE=roll back all non-PREPARED transactions */ { trx_t* trx; mutex_enter(&kernel_mutex); - if (UT_LIST_GET_FIRST(trx_sys->trx_list)) { + if (!UT_LIST_GET_FIRST(trx_sys->trx_list)) { + goto leave_function; + } + if (all) { fprintf(stderr, "InnoDB: Starting in background the rollback" " of uncommitted transactions\n"); - } else { - goto leave_function; } mutex_exit(&kernel_mutex); @@ -584,18 +580,42 @@ loop: goto loop; case TRX_ACTIVE: - mutex_exit(&kernel_mutex); - trx_rollback_active(trx); - goto loop; + if (all || trx_get_dict_operation(trx) + != TRX_DICT_OP_NONE) { + mutex_exit(&kernel_mutex); + trx_rollback_active(trx); + goto loop; + } } } - ut_print_timestamp(stderr); - fprintf(stderr, - " InnoDB: Rollback of non-prepared transactions completed\n"); + if (all) { + ut_print_timestamp(stderr); + fprintf(stderr, + " InnoDB: Rollback of non-prepared" + " transactions completed\n"); + } leave_function: mutex_exit(&kernel_mutex); +} + +/*******************************************************************//** +Rollback or clean up any incomplete transactions which were +encountered in crash recovery. If the transaction already was +committed, then we clean up a possible insert undo log. If the +transaction was not yet committed, then we roll it back. +Note: this is done in a background thread. +@return a dummy parameter */ +UNIV_INTERN +os_thread_ret_t +trx_rollback_or_clean_all_recovered( +/*================================*/ + void* arg __attribute__((unused))) + /*!< in: a dummy parameter required by + os_thread_create */ +{ + trx_rollback_or_clean_recovered(TRUE); /* We count the number of threads in os_thread_exit(). A created thread should always use that to exit and not use return() to exit. */ === modified file 'storage/xtradb/trx/trx0rseg.c' --- a/storage/xtradb/trx/trx0rseg.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/trx/trx0rseg.c 2010-01-06 12:00:14 +0000 @@ -132,6 +132,49 @@ trx_rseg_header_create( } /***********************************************************************//** +Free's an instance of the rollback segment in memory. */ +UNIV_INTERN +void +trx_rseg_mem_free( +/*==============*/ + trx_rseg_t* rseg) /* in, own: instance to free */ +{ + trx_undo_t* undo; + + mutex_free(&rseg->mutex); + + /* There can't be any active transactions. */ + ut_a(UT_LIST_GET_LEN(rseg->update_undo_list) == 0); + ut_a(UT_LIST_GET_LEN(rseg->insert_undo_list) == 0); + + undo = UT_LIST_GET_FIRST(rseg->update_undo_cached); + + while (undo != NULL) { + trx_undo_t* prev_undo = undo; + + undo = UT_LIST_GET_NEXT(undo_list, undo); + UT_LIST_REMOVE(undo_list, rseg->update_undo_cached, prev_undo); + + trx_undo_mem_free(prev_undo); + } + + undo = UT_LIST_GET_FIRST(rseg->insert_undo_cached); + + while (undo != NULL) { + trx_undo_t* prev_undo = undo; + + undo = UT_LIST_GET_NEXT(undo_list, undo); + UT_LIST_REMOVE(undo_list, rseg->insert_undo_cached, prev_undo); + + trx_undo_mem_free(prev_undo); + } + + trx_sys_set_nth_rseg(trx_sys, rseg->id, NULL); + + mem_free(rseg); +} + +/*************************************************************************** Creates and initializes a rollback segment object. The values for the fields are read from the header. The object is inserted to the rseg list of the trx system object and a pointer is inserted in the rseg === modified file 'storage/xtradb/trx/trx0sys.c' --- a/storage/xtradb/trx/trx0sys.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/trx/trx0sys.c 2010-01-06 12:00:14 +0000 @@ -39,7 +39,9 @@ Created 3/26/1996 Heikki Tuuri #include "srv0srv.h" #include "trx0purge.h" #include "log0log.h" +#include "log0recv.h" #include "os0file.h" +#include "read0read.h" /** The file format tag structure with id and name. */ struct file_format_struct { @@ -552,6 +554,12 @@ trx_sys_doublewrite_init_or_restore_page zip_size ? zip_size : UNIV_PAGE_SIZE, read_buf, NULL); + if (srv_recovery_stats && recv_recovery_is_on()) { + mutex_enter(&(recv_sys->mutex)); + recv_sys->stats_doublewrite_check_pages++; + mutex_exit(&(recv_sys->mutex)); + } + /* Check if the page is corrupt */ if (UNIV_UNLIKELY @@ -599,6 +607,13 @@ trx_sys_doublewrite_init_or_restore_page zip_size, page_no, 0, zip_size ? zip_size : UNIV_PAGE_SIZE, page, NULL); + + if (srv_recovery_stats && recv_recovery_is_on()) { + mutex_enter(&(recv_sys->mutex)); + recv_sys->stats_doublewrite_overwrite_pages++; + mutex_exit(&(recv_sys->mutex)); + } + fprintf(stderr, "InnoDB: Recovered the page from" " the doublewrite buffer.\n"); @@ -1592,3 +1607,80 @@ trx_sys_file_format_id_to_name( } #endif /* !UNIV_HOTBACKUP */ + +/********************************************************************* +Shutdown/Close the transaction system. */ +UNIV_INTERN +void +trx_sys_close(void) +/*===============*/ +{ + trx_rseg_t* rseg; + read_view_t* view; + + ut_ad(trx_sys != NULL); + + /* Check that all read views are closed except read view owned + by a purge. */ + + if (UT_LIST_GET_LEN(trx_sys->view_list) > 1) { + fprintf(stderr, + "InnoDB: Error: all read views were not closed" + " before shutdown:\n" + "InnoDB: %lu read views open \n", + UT_LIST_GET_LEN(trx_sys->view_list) - 1); + } + + sess_close(trx_dummy_sess); + trx_dummy_sess = NULL; + + trx_purge_sys_close(); + + mutex_enter(&kernel_mutex); + + /* Free the double write data structures. */ + ut_a(trx_doublewrite != NULL); + ut_free(trx_doublewrite->write_buf_unaligned); + trx_doublewrite->write_buf_unaligned = NULL; + + mem_free(trx_doublewrite->buf_block_arr); + trx_doublewrite->buf_block_arr = NULL; + + mutex_free(&trx_doublewrite->mutex); + mem_free(trx_doublewrite); + trx_doublewrite = NULL; + + /* There can't be any active transactions. */ + rseg = UT_LIST_GET_FIRST(trx_sys->rseg_list); + + while (rseg != NULL) { + trx_rseg_t* prev_rseg = rseg; + + rseg = UT_LIST_GET_NEXT(rseg_list, prev_rseg); + UT_LIST_REMOVE(rseg_list, trx_sys->rseg_list, prev_rseg); + + trx_rseg_mem_free(prev_rseg); + } + + view = UT_LIST_GET_FIRST(trx_sys->view_list); + + while (view != NULL) { + read_view_t* prev_view = view; + + view = UT_LIST_GET_NEXT(view_list, prev_view); + + /* Views are allocated from the trx_sys->global_read_view_heap. + So, we simply remove the element here. */ + UT_LIST_REMOVE(view_list, trx_sys->view_list, prev_view); + } + + ut_a(UT_LIST_GET_LEN(trx_sys->trx_list) == 0); + ut_a(UT_LIST_GET_LEN(trx_sys->rseg_list) == 0); + ut_a(UT_LIST_GET_LEN(trx_sys->view_list) == 0); + ut_a(UT_LIST_GET_LEN(trx_sys->mysql_trx_list) == 0); + + mem_free(trx_sys); + + trx_sys = NULL; + mutex_exit(&kernel_mutex); +} === modified file 'storage/xtradb/trx/trx0trx.c' --- a/storage/xtradb/trx/trx0trx.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/trx/trx0trx.c 2010-01-06 12:00:14 +0000 @@ -178,6 +178,15 @@ trx_create( trx->global_read_view = NULL; trx->read_view = NULL; + trx->io_reads = 0; + trx->io_read = 0; + trx->io_reads_wait_timer = 0; + trx->lock_que_wait_timer = 0; + trx->innodb_que_wait_timer = 0; + trx->distinct_page_access = 0; + trx->distinct_page_access_hash = NULL; + trx->take_stats = FALSE; + /* Set X/Open XA transaction identification to NULL */ memset(&trx->xid, 0, sizeof(trx->xid)); trx->xid.formatID = -1; @@ -215,6 +224,11 @@ trx_allocate_for_mysql(void) trx->mysql_process_no = os_proc_get_number(); + if (innobase_get_slow_log() && trx->take_stats) { + trx->distinct_page_access_hash = mem_alloc(DPAH_SIZE); + memset(trx->distinct_page_access_hash, 0, DPAH_SIZE); + } + return(trx); } @@ -346,6 +360,12 @@ trx_free_for_mysql( /*===============*/ trx_t* trx) /*!< in, own: trx object */ { + if (trx->distinct_page_access_hash) + { + mem_free(trx->distinct_page_access_hash); + trx->distinct_page_access_hash= NULL; + } + mutex_enter(&kernel_mutex); UT_LIST_REMOVE(mysql_trx_list, trx_sys->mysql_trx_list, trx); @@ -367,6 +387,12 @@ trx_free_for_background( /*====================*/ trx_t* trx) /*!< in, own: trx object */ { + if (trx->distinct_page_access_hash) + { + mem_free(trx->distinct_page_access_hash); + trx->distinct_page_access_hash= NULL; + } + mutex_enter(&kernel_mutex); trx_free(trx); @@ -820,7 +846,7 @@ trx_commit_off_kernel( in exactly the same order as commit lsn's, if the transactions have different rollback segments. To get exactly the same order we should hold the kernel mutex up to this point, - adding to to the contention of the kernel mutex. However, if + adding to the contention of the kernel mutex. However, if a transaction T2 is able to see modifications made by a transaction T1, T2 will always get a bigger transaction number and a bigger commit lsn than T1. */ @@ -967,7 +993,7 @@ trx_commit_off_kernel( /****************************************************************//** Cleans up a transaction at database startup. The cleanup is needed if the transaction already got to the middle of a commit when the database -crashed, andf we cannot roll it back. */ +crashed, and we cannot roll it back. */ UNIV_INTERN void trx_cleanup_at_db_startup( @@ -1072,6 +1098,9 @@ trx_end_lock_wait( trx_t* trx) /*!< in: transaction */ { que_thr_t* thr; + ulint sec; + ulint ms; + ib_uint64_t now; ut_ad(mutex_own(&kernel_mutex)); ut_ad(trx->que_state == TRX_QUE_LOCK_WAIT); @@ -1086,6 +1115,11 @@ trx_end_lock_wait( thr = UT_LIST_GET_FIRST(trx->wait_thrs); } + if (innobase_get_slow_log() && trx->take_stats) { + ut_usectime(&sec, &ms); + now = (ib_uint64_t)sec * 1000000 + ms; + trx->lock_que_wait_timer += (ulint)(now - trx->lock_que_wait_ustarted); + } trx->que_state = TRX_QUE_RUNNING; } @@ -1099,6 +1133,9 @@ trx_lock_wait_to_suspended( trx_t* trx) /*!< in: transaction in the TRX_QUE_LOCK_WAIT state */ { que_thr_t* thr; + ulint sec; + ulint ms; + ib_uint64_t now; ut_ad(mutex_own(&kernel_mutex)); ut_ad(trx->que_state == TRX_QUE_LOCK_WAIT); @@ -1113,6 +1150,11 @@ trx_lock_wait_to_suspended( thr = UT_LIST_GET_FIRST(trx->wait_thrs); } + if (innobase_get_slow_log() && trx->take_stats) { + ut_usectime(&sec, &ms); + now = (ib_uint64_t)sec * 1000000 + ms; + trx->lock_que_wait_timer += (ulint)(now - trx->lock_que_wait_ustarted); + } trx->que_state = TRX_QUE_RUNNING; } === modified file 'storage/xtradb/trx/trx0undo.c' --- a/storage/xtradb/trx/trx0undo.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/trx/trx0undo.c 2010-01-06 12:00:14 +0000 @@ -1560,7 +1560,7 @@ trx_undo_mem_init_for_reuse( /********************************************************************//** Frees an undo log memory copy. */ -static +UNIV_INTERN void trx_undo_mem_free( /*==============*/ === modified file 'storage/xtradb/usr/usr0sess.c' --- a/storage/xtradb/usr/usr0sess.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/usr/usr0sess.c 2010-01-06 12:00:14 +0000 @@ -32,14 +32,6 @@ Created 6/25/1996 Heikki Tuuri #include "trx0trx.h" /*********************************************************************//** -Closes a session, freeing the memory occupied by it. */ -static -void -sess_close( -/*=======*/ - sess_t* sess); /*!< in, own: session object */ - -/*********************************************************************//** Opens a session. @return own: session object */ UNIV_INTERN @@ -64,35 +56,16 @@ sess_open(void) /*********************************************************************//** Closes a session, freeing the memory occupied by it. */ -static +UNIV_INTERN void sess_close( /*=======*/ sess_t* sess) /*!< in, own: session object */ { - ut_ad(mutex_own(&kernel_mutex)); - ut_ad(sess->trx == NULL); - - mem_free(sess); -} - -/*********************************************************************//** -Closes a session, freeing the memory occupied by it, if it is in a state -where it should be closed. -@return TRUE if closed */ -UNIV_INTERN -ibool -sess_try_close( -/*===========*/ - sess_t* sess) /*!< in, own: session object */ -{ - ut_ad(mutex_own(&kernel_mutex)); + ut_ad(!mutex_own(&kernel_mutex)); - if (UT_LIST_GET_LEN(sess->graphs) == 0) { - sess_close(sess); + ut_a(UT_LIST_GET_LEN(sess->graphs) == 0); - return(TRUE); - } - - return(FALSE); + trx_free_for_background(sess->trx); + mem_free(sess); } === modified file 'storage/xtradb/ut/ut0auxconf_atomic_pthread_t_solaris.c' --- a/storage/xtradb/ut/ut0auxconf_atomic_pthread_t_solaris.c 2009-09-23 00:06:02 +0000 +++ b/storage/xtradb/ut/ut0auxconf_atomic_pthread_t_solaris.c 2010-01-06 12:00:14 +0000 @@ -17,18 +17,38 @@ Place, Suite 330, Boston, MA 02111-1307 *****************************************************************************/ /***************************************************************************** -If this program compiles, then pthread_t objects can be used as arguments -to Solaris libc atomic functions. +If this program compiles and returns 0, then pthread_t objects can be used as +arguments to Solaris libc atomic functions. Created April 18, 2009 Vasil Dimov *****************************************************************************/ #include <pthread.h> +#include <string.h> int main(int argc, char** argv) { - pthread_t x = 0; + pthread_t x1; + pthread_t x2; + pthread_t x3; + + memset(&x1, 0x0, sizeof(x1)); + memset(&x2, 0x0, sizeof(x2)); + memset(&x3, 0x0, sizeof(x3)); + + if (sizeof(pthread_t) == 4) { + + atomic_cas_32(&x1, x2, x3); + + } else if (sizeof(pthread_t) == 8) { + + atomic_cas_64(&x1, x2, x3); + + } else { + + return(1); + } return(0); } === added file 'storage/xtradb/ut/ut0auxconf_have_gcc_atomics.c' --- a/storage/xtradb/ut/ut0auxconf_have_gcc_atomics.c 1970-01-01 00:00:00 +0000 +++ b/storage/xtradb/ut/ut0auxconf_have_gcc_atomics.c 2010-01-06 12:00:14 +0000 @@ -0,0 +1,61 @@ +/***************************************************************************** + +Copyright (c) 2009, Innobase Oy. All Rights Reserved. + +This program is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free Software +Foundation; version 2 of the License. + +This program is distributed in the hope that it will be useful, but WITHOUT +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. + +You should have received a copy of the GNU General Public License along with +this program; if not, write to the Free Software Foundation, Inc., 59 Temple +Place, Suite 330, Boston, MA 02111-1307 USA + +*****************************************************************************/ + +/***************************************************************************** +If this program compiles and returns 0, then GCC atomic funcions are available. + +Created September 12, 2009 Vasil Dimov +*****************************************************************************/ + +int +main(int argc, char** argv) +{ + long x; + long y; + long res; + char c; + + x = 10; + y = 123; + res = __sync_bool_compare_and_swap(&x, x, y); + if (!res || x != y) { + return(1); + } + + x = 10; + y = 123; + res = __sync_bool_compare_and_swap(&x, x + 1, y); + if (res || x != 10) { + return(1); + } + + x = 10; + y = 123; + res = __sync_add_and_fetch(&x, y); + if (res != 123 + 10 || x != 123 + 10) { + return(1); + } + + c = 10; + res = __sync_lock_test_and_set(&c, 123); + if (res != 10 || c != 123) { + return(1); + } + + return(0); +} === modified file 'storage/xtradb/ut/ut0mem.c' --- a/storage/xtradb/ut/ut0mem.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/ut/ut0mem.c 2010-01-06 12:00:14 +0000 @@ -433,6 +433,8 @@ ut_free_all_mem(void) " total allocated memory is %lu\n", (ulong) ut_total_allocated_memory); } + + ut_mem_block_list_inited = FALSE; } #endif /* !UNIV_HOTBACKUP */ === modified file 'storage/xtradb/ut/ut0ut.c' --- a/storage/xtradb/ut/ut0ut.c 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/ut/ut0ut.c 2010-01-06 12:00:14 +0000 @@ -132,6 +132,7 @@ ut_time(void) return(time(NULL)); } +#ifndef UNIV_HOTBACKUP /**********************************************************//** Returns system time. Upon successful completion, the value 0 is returned; otherwise the @@ -200,6 +201,24 @@ ut_time_us( } /**********************************************************//** +Returns the number of milliseconds since some epoch. The +value may wrap around. It should only be used for heuristic +purposes. +@return ms since epoch */ +UNIV_INTERN +ulint +ut_time_ms(void) +/*============*/ +{ + struct timeval tv; + + ut_gettimeofday(&tv, NULL); + + return((ulint) tv.tv_sec * 1000 + tv.tv_usec / 1000); +} +#endif /* !UNIV_HOTBACKUP */ + +/**********************************************************//** Returns the difference of two times in seconds. @return time2 - time1 expressed in seconds */ UNIV_INTERN === removed directory 'storage/xtradb/win-plugin' === removed file 'storage/xtradb/win-plugin/README' --- a/storage/xtradb/win-plugin/README 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/win-plugin/README 1970-01-01 00:00:00 +0000 @@ -1,22 +0,0 @@ -This directory contains patches that need to be applied to the MySQL -source tree in order to build the dynamic plugin on Windows -- -HA_INNODB.DLL. Please note the followings when adding the patches: - -* The patch must be applied from the mysql top-level source directory. - patch -p0 < win-plugin.diff -* The patch filenames end in ".diff". -* All patches here are expected to apply cleanly to the latest MySQL 5.1 - tree when storage/innobase is replaced with this InnoDB branch. - -When applying the patch, the following files will be modified: - - * CMakeLists.txt - * sql/CMakeLists.txt - * win/configure.js - -Also, two new files will be added: - - * sql/mysqld.def - * sql/mysqld_x64.def - -You can get "patch" utility for Windows from http://unxutils.sourceforge.net/ === removed file 'storage/xtradb/win-plugin/win-plugin.diff' --- a/storage/xtradb/win-plugin/win-plugin.diff 2009-09-07 10:22:53 +0000 +++ b/storage/xtradb/win-plugin/win-plugin.diff 1970-01-01 00:00:00 +0000 @@ -1,279 +0,0 @@ -diff -Nur CMakeLists.txt.orig CMakeLists.txt ---- CMakeLists.txt.orig 2008-10-03 12:25:41 -05:00 -+++ CMakeLists.txt 2008-09-26 17:32:51 -05:00 -@@ -254,9 +254,9 @@ - IF(WITH_FEDERATED_STORAGE_ENGINE) - ADD_SUBDIRECTORY(storage/federated) - ENDIF(WITH_FEDERATED_STORAGE_ENGINE) --IF(WITH_INNOBASE_STORAGE_ENGINE) -+IF(WITH_INNOBASE_STORAGE_ENGINE OR INNODB_DYNAMIC_PLUGIN) - ADD_SUBDIRECTORY(storage/innobase) --ENDIF(WITH_INNOBASE_STORAGE_ENGINE) -+ENDIF(WITH_INNOBASE_STORAGE_ENGINE OR INNODB_DYNAMIC_PLUGIN) - ADD_SUBDIRECTORY(sql) - ADD_SUBDIRECTORY(server-tools/instance-manager) - ADD_SUBDIRECTORY(libmysql) - -diff -Nur sql/CMakeLists.txt.orig sql/CMakeLists.txt ---- sql/CMakeLists.txt.orig 2008-10-03 12:25:41 -05:00 -+++ sql/CMakeLists.txt 2008-09-24 03:58:19 -05:00 -@@ -98,6 +98,15 @@ - LINK_FLAGS "/PDB:${CMAKE_CFG_INTDIR}/mysqld${MYSQLD_EXE_SUFFIX}.pdb") - ENDIF(cmake_version EQUAL 20406) - -+# Checks for 64-bit version -+IF(CMAKE_SIZEOF_VOID_P MATCHES 8) -+SET_TARGET_PROPERTIES(mysqld PROPERTIES -+ LINK_FLAGS "/def:\"${PROJECT_SOURCE_DIR}/sql/mysqld_x64.def\"") -+ELSE(CMAKE_SIZEOF_VOID_P MATCHES 8) -+SET_TARGET_PROPERTIES(mysqld PROPERTIES -+ LINK_FLAGS "/def:\"${PROJECT_SOURCE_DIR}/sql/mysqld.def\"") -+ENDIF(CMAKE_SIZEOF_VOID_P MATCHES 8) -+ - IF(EMBED_MANIFESTS) - MYSQL_EMBED_MANIFEST("mysqld" "asInvoker") - ENDIF(EMBED_MANIFESTS) - -diff -Nur sql/mysqld.def.orig sql/mysqld.def ---- sql/mysqld.def.orig 1969-12-31 18:00:00 -06:00 -+++ sql/mysqld.def 2009-04-09 02:20:32 -05:00 -@@ -0,0 +1,111 @@ -+EXPORTS -+ ?use_hidden_primary_key@handler@@UAEXXZ -+ ?get_dynamic_partition_info@handler@@UAEXPAUPARTITION_INFO@@I@Z -+ ?read_first_row@handler@@UAEHPAEI@Z -+ ?read_range_next@handler@@UAEHXZ -+ ?read_range_first@handler@@UAEHPBUst_key_range@@0_N1@Z -+ ?read_multi_range_first@handler@@UAEHPAPAUst_key_multi_range@@PAU2@I_NPAUst_handler_buffer@@@Z -+ ?read_multi_range_next@handler@@UAEHPAPAUst_key_multi_range@@@Z -+ ?index_read_idx_map@handler@@UAEHPAEIPBEKW4ha_rkey_function@@@Z -+ ?print_error@handler@@UAEXHH@Z -+ ?clone@handler@@UAEPAV1@PAUst_mem_root@@@Z -+ ?get_auto_increment@handler@@UAEX_K00PA_K1@Z -+ ?index_next_same@handler@@UAEHPAEPBEI@Z -+ ?get_error_message@handler@@UAE_NHPAVString@@@Z -+ ?ha_thd@handler@@IBEPAVTHD@@XZ -+ ?update_auto_increment@handler@@QAEHXZ -+ ?ha_statistic_increment@handler@@IBEXPQsystem_status_var@@K@Z -+ ?trans_register_ha@@YAXPAVTHD@@_NPAUhandlerton@@@Z -+ ?cmp@Field_blob@@QAEHPBEI0I@Z -+ ?set_time@Field_timestamp@@QAEXXZ -+ ?sql_print_error@@YAXPBDZZ -+ ?sql_print_warning@@YAXPBDZZ -+ ?check_global_access@@YA_NPAVTHD@@K@Z -+ ?schema_table_store_record@@YA_NPAVTHD@@PAUst_table@@@Z -+ ?get_quote_char_for_identifier@@YAHPAVTHD@@PBDI@Z -+ ?copy@String@@QAE_NXZ -+ ?copy@String@@QAE_NABV1@@Z -+ ?copy@String@@QAE_NPBDIPAUcharset_info_st@@@Z -+ ?copy_and_convert@@YAIPADIPAUcharset_info_st@@PBDI1PAI@Z -+ ?filename_to_tablename@@YAIPBDPADI@Z -+ ?strconvert@@YAIPAUcharset_info_st@@PBD0PADIPAI@Z -+ ?calculate_key_len@@YAIPAUst_table@@IPBEK@Z -+ ?sql_alloc@@YAPAXI@Z -+ ?localtime_to_TIME@@YAXPAUst_mysql_time@@PAUtm@@@Z -+ ?push_warning@@YAPAVMYSQL_ERROR@@PAVTHD@@W4enum_warning_level@1@IPBD@Z -+ ?push_warning_printf@@YAXPAVTHD@@W4enum_warning_level@MYSQL_ERROR@@IPBDZZ -+ ?drop_table@handler@@EAEXPBD@Z -+ ?column_bitmaps_signal@handler@@UAEXXZ -+ ?delete_table@handler@@MAEHPBD@Z -+ ?rename_table@handler@@MAEHPBD0@Z -+ ?key_map_empty@@3V?$Bitmap@$0EA@@@B -+ ?THR_THD@@3PAVTHD@@A -+ ?end_of_list@@3Ulist_node@@A -+ ?mysql_tmpdir_list@@3Ust_my_tmpdir@@A -+ mysql_query_cache_invalidate4 -+ thd_query -+ thd_sql_command -+ thd_get_thread_id -+ thd_get_xid -+ thd_slave_thread -+ thd_non_transactional_update -+ thd_mark_transaction_to_rollback -+ thd_security_context -+ thd_charset -+ thd_test_options -+ thd_ha_data -+ thd_killed -+ thd_tx_isolation -+ thd_tablespace_op -+ thd_sql_command -+ thd_memdup -+ thd_make_lex_string -+ thd_in_lock_tables -+ thd_binlog_format -+ _my_hash_init -+ my_hash_free -+ my_tmpdir -+ check_if_legal_filename -+ my_filename -+ my_sync_dir_by_file -+ alloc_root -+ thr_lock_data_init -+ thr_lock_init -+ thr_lock_delete -+ my_multi_malloc -+ get_charset -+ unpack_filename -+ my_hash_insert -+ my_hash_search -+ my_hash_delete -+ mysql_bin_log_file_pos -+ mysql_bin_log_file_name -+ mysqld_embedded -+ my_thread_name -+ my_malloc -+ my_no_flags_free -+ _sanity -+ _mymalloc -+ _myfree -+ _my_strdup -+ _my_thread_var -+ my_error -+ pthread_cond_init -+ pthread_cond_signal -+ pthread_cond_wait -+ pthread_cond_destroy -+ localtime_r -+ my_strdup -+ deflate -+ deflateEnd -+ deflateReset -+ deflateInit2_ -+ inflateEnd -+ inflateInit_ -+ inflate -+ compressBound -+ inflateInit2_ -+ adler32 -+ longlong2str -+ strend -+ my_snprintf - -diff -Nur sql/mysqld_x64.def.orig sql/mysqld_x64.def ---- sql/mysqld_x64.def.orig 1969-12-31 18:00:00 -06:00 -+++ sql/mysqld_x64.def 2009-04-09 02:22:04 -05:00 -@@ -0,0 +1,111 @@ -+EXPORTS -+ ?use_hidden_primary_key@handler@@UEAAXXZ -+ ?get_dynamic_partition_info@handler@@UEAAXPEAUPARTITION_INFO@@I@Z -+ ?read_first_row@handler@@UEAAHPEAEI@Z -+ ?read_range_next@handler@@UEAAHXZ -+ ?read_range_first@handler@@UEAAHPEBUst_key_range@@0_N1@Z -+ ?read_multi_range_first@handler@@UEAAHPEAPEAUst_key_multi_range@@PEAU2@I_NPEAUst_handler_buffer@@@Z -+ ?read_multi_range_next@handler@@UEAAHPEAPEAUst_key_multi_range@@@Z -+ ?index_read_idx_map@handler@@UEAAHPEAEIPEBEKW4ha_rkey_function@@@Z -+ ?print_error@handler@@UEAAXHH@Z -+ ?clone@handler@@UEAAPEAV1@PEAUst_mem_root@@@Z -+ ?get_auto_increment@handler@@UEAAX_K00PEA_K1@Z -+ ?index_next_same@handler@@UEAAHPEAEPEBEI@Z -+ ?get_error_message@handler@@UEAA_NHPEAVString@@@Z -+ ?ha_thd@handler@@IEBAPEAVTHD@@XZ -+ ?update_auto_increment@handler@@QEAAHXZ -+ ?ha_statistic_increment@handler@@IEBAXPEQsystem_status_var@@K@Z -+ ?trans_register_ha@@YAXPEAVTHD@@_NPEAUhandlerton@@@Z -+ ?cmp@Field_blob@@QEAAHPEBEI0I@Z -+ ?set_time@Field_timestamp@@QEAAXXZ -+ ?sql_print_error@@YAXPEBDZZ -+ ?sql_print_warning@@YAXPEBDZZ -+ ?check_global_access@@YA_NPEAVTHD@@K@Z -+ ?schema_table_store_record@@YA_NPEAVTHD@@PEAUst_table@@@Z -+ ?get_quote_char_for_identifier@@YAHPEAVTHD@@PEBDI@Z -+ ?copy@String@@QEAA_NXZ -+ ?copy@String@@QEAA_NAEBV1@@Z -+ ?copy@String@@QEAA_NPEBDIPEAUcharset_info_st@@@Z -+ ?copy_and_convert@@YAIPEADIPEAUcharset_info_st@@PEBDI1PEAI@Z -+ ?filename_to_tablename@@YAIPEBDPEADI@Z -+ ?strconvert@@YAIPEAUcharset_info_st@@PEBD0PEADIPEAI@Z -+ ?calculate_key_len@@YAIPEAUst_table@@IPEBEK@Z -+ ?sql_alloc@@YAPEAX_K@Z -+ ?localtime_to_TIME@@YAXPEAUst_mysql_time@@PEAUtm@@@Z -+ ?push_warning@@YAPEAVMYSQL_ERROR@@PEAVTHD@@W4enum_warning_level@1@IPEBD@Z -+ ?push_warning_printf@@YAXPEAVTHD@@W4enum_warning_level@MYSQL_ERROR@@IPEBDZZ -+ ?drop_table@handler@@EEAAXPEBD@Z -+ ?column_bitmaps_signal@handler@@UEAAXXZ -+ ?delete_table@handler@@MEAAHPEBD@Z -+ ?rename_table@handler@@MEAAHPEBD0@Z -+ ?key_map_empty@@3V?$Bitmap@$0EA@@@B -+ ?THR_THD@@3PEAVTHD@@EA -+ ?end_of_list@@3Ulist_node@@A -+ ?mysql_tmpdir_list@@3Ust_my_tmpdir@@A -+ mysql_query_cache_invalidate4 -+ thd_query -+ thd_sql_command -+ thd_get_thread_id -+ thd_get_xid -+ thd_slave_thread -+ thd_non_transactional_update -+ thd_mark_transaction_to_rollback -+ thd_security_context -+ thd_charset -+ thd_test_options -+ thd_ha_data -+ thd_killed -+ thd_tx_isolation -+ thd_tablespace_op -+ thd_sql_command -+ thd_memdup -+ thd_make_lex_string -+ thd_in_lock_tables -+ thd_binlog_format -+ _my_hash_init -+ my_hash_free -+ my_tmpdir -+ check_if_legal_filename -+ my_filename -+ my_sync_dir_by_file -+ alloc_root -+ thr_lock_data_init -+ thr_lock_init -+ thr_lock_delete -+ my_multi_malloc -+ get_charset -+ unpack_filename -+ my_hash_insert -+ my_hash_search -+ my_hash_delete -+ mysql_bin_log_file_pos -+ mysql_bin_log_file_name -+ mysqld_embedded -+ my_thread_name -+ my_malloc -+ my_no_flags_free -+ _sanity -+ _mymalloc -+ _myfree -+ _my_strdup -+ _my_thread_var -+ my_error -+ pthread_cond_init -+ pthread_cond_signal -+ pthread_cond_wait -+ pthread_cond_destroy -+ localtime_r -+ my_strdup -+ deflate -+ deflateEnd -+ deflateReset -+ deflateInit2_ -+ inflateEnd -+ inflateInit_ -+ inflate -+ compressBound -+ inflateInit2_ -+ adler32 -+ longlong2str -+ strend -+ my_snprintf - -diff -Nur win/configure.js.orig win/configure.js ---- win/configure.js.orig 2008-09-26 21:18:37 -05:00 -+++ win/configure.js 2008-10-01 11:21:27 -05:00 -@@ -50,6 +50,7 @@ - case "EMBED_MANIFESTS": - case "EXTRA_DEBUG": - case "WITH_EMBEDDED_SERVER": -+ case "INNODB_DYNAMIC_PLUGIN": - configfile.WriteLine("SET (" + args.Item(i) + " TRUE)"); - break; - case "MYSQL_SERVER_SUFFIX": === modified file 'support-files/compiler_warnings.supp' --- a/support-files/compiler_warnings.supp 2010-01-06 21:27:53 +0000 +++ b/support-files/compiler_warnings.supp 2010-01-28 14:49:14 +0000 @@ -4,6 +4,7 @@ # integer.cpp: .*control reaches end of non-void function.*: 1288-1427 +integer.cpp: .*no return statement in function returning non-void.*: 1288-1427 DictTabInfo.cpp : .*invalid access to non-static.* DictTabInfo.cpp : .*macro was used incorrectly.* DbdihMain.cpp : .*unused variable.* : 6666-7013 @@ -27,7 +28,17 @@ _flex_tmp.c: .*not enough actual paramet pars0lex.l: .*conversion from 'ulint' to 'int', possible loss of data.* btr/btr0cur\.c: .*value computed is not used.*: 3175-3375 include/buf0buf\.ic: unused parameter ‘mtr’ +fil/fil0fil\.c: pointer targets in passing argument.*differ in signedness fil/fil0fil\.c: comparison between signed and unsigned : 3100-3199 +fil/fil0fil\.c: unused parameter +log/log0recv\.c: unused variable +os/os0file\.c: unused parameter +os/os0file\.c: pointer targets in assignment differ in signedness +handler/i_s\.cc: unused variable +sync/sync0rw\.c: unused parameter +sync/sync0sync\.c: unused parameter +sync/sync0sync\.c: unused variable +ut/ut0ut\.c: ignoring return value of # # bdb is not critical to keep up to date @@ -50,6 +61,11 @@ db_vrfy.c : .*comparison is always false .*/extra/libevent/select\.c : .*comparison between signed and unsigned.* : 270-280 # +# Ignore warnings from system libraries +# +/usr/share/aclocal/audiofile.m4 : .* + +# # Ignore all conversion warnings on windows 64 # (Is safe as we are not yet supporting strings >= 2G) # @@ -87,10 +103,15 @@ storage/maria/ma_pagecache.c: .*'info_ch # I think these are due to mix of C and C++. # storage/pbxt/ : typedef.*was ignored in this declaration +ha_pbxt\.cc : variable.*might be clobbered by.*longjmp # # Yassl include/runtime.hpp: .*pure_error.* +.*/extra/yassl/taocrypt/.*: comparison with string literal +.*/extra/yassl/taocrypt/src/blowfish\.cpp: array subscript is above array bounds +.*/extra/yassl/taocrypt/src/file\.cpp: ignoring return value +.*/extra/yassl/taocrypt/src/integer\.cpp: control reaches end of non-void function # # Groff warnings on OpenSUSE. @@ -102,6 +123,8 @@ include/runtime.hpp: .*pure_error.* # listener.cc : .*conversion from 'SOCKET' to 'int'.* net_serv.cc : .*conversion from 'SOCKET' to 'int'.* +set_var.cc: right-hand operand of comma has no effect : 1000-1400 + # allow a little moving space for the warning below mi_packrec.c : .*result of 32-bit shift implicitly converted to 64 bits.* : 560-600 === modified file 'tests/mysql_client_test.c' --- a/tests/mysql_client_test.c 2009-12-03 11:19:05 +0000 +++ b/tests/mysql_client_test.c 2010-01-11 13:15:28 +0000 @@ -2465,6 +2465,34 @@ static uint query_cache_hits(MYSQL *conn /* + Check that query cache is available in server. +*/ +static my_bool is_query_cache_available() +{ + int rc; + MYSQL_RES *result; + MYSQL_ROW row; + int res= -1; + + rc= mysql_query(mysql, "SHOW VARIABLES LIKE 'have_query_cache'"); + myquery(rc); + + result= mysql_store_result(mysql); + DIE_UNLESS(result); + + row= mysql_fetch_row(result); + DIE_UNLESS(row != NULL); + if (strcmp(row[1], "YES") == 0) + res= 1; + else if (strcmp(row[1], "NO") == 0) + res= 0; + mysql_free_result(result); + + DIE_UNLESS(res == 0 || res == 1); + return res; +} + +/* Test that prepared statements make use of the query cache just as normal statements (BUG#735). */ @@ -2508,6 +2536,12 @@ static void test_ps_query_cache() myheader("test_ps_query_cache"); + if (! is_query_cache_available()) + { + fprintf(stdout, "Skipping test_ps_query_cache: Query cache not available.\n"); + return; + } + rc= mysql_query(mysql, "SET SQL_MODE=''"); myquery(rc); @@ -17863,8 +17897,6 @@ static void test_bug43560(void) Bug#36326: nested transaction and select */ -#ifdef HAVE_QUERY_CACHE - static void test_bug36326() { int rc; @@ -17872,6 +17904,12 @@ static void test_bug36326() DBUG_ENTER("test_bug36326"); myheader("test_bug36326"); + if (! is_query_cache_available()) + { + fprintf(stdout, "Skipping test_bug36326: Query cache not available.\n"); + DBUG_VOID_RETURN; + } + rc= mysql_autocommit(mysql, TRUE); myquery(rc); rc= mysql_query(mysql, "DROP TABLE IF EXISTS t1"); @@ -17911,8 +17949,6 @@ static void test_bug36326() DBUG_VOID_RETURN; } -#endif - /** Bug#41078: With CURSOR_TYPE_READ_ONLY mysql_stmt_fetch() returns short string value. @@ -18373,9 +18409,7 @@ static struct my_tests_st my_tests[]= { { "test_bug38486", test_bug38486 }, { "test_bug40365", test_bug40365 }, { "test_bug43560", test_bug43560 }, -#ifdef HAVE_QUERY_CACHE { "test_bug36326", test_bug36326 }, -#endif { "test_bug41078", test_bug41078 }, { "test_bug44495", test_bug44495 }, { 0, 0 } === modified file 'unittest/mysys/waiting_threads-t.c' --- a/unittest/mysys/waiting_threads-t.c 2009-09-15 10:46:35 +0000 +++ b/unittest/mysys/waiting_threads-t.c 2010-01-28 14:49:14 +0000 @@ -258,7 +258,7 @@ void do_tests() #define test_kill_strategy(X) \ diag("kill strategy: " #X); \ DBUG_EXECUTE("reset_file", \ - { rewind(DBUG_FILE); ftruncate(fileno(DBUG_FILE), 0); }); \ + { rewind(DBUG_FILE); (void) ftruncate(fileno(DBUG_FILE), 0); }); \ DBUG_PRINT("info", ("kill strategy: " #X)); \ kill_strategy=X; \ do_one_test(); === modified file 'vio/vio.c' --- a/vio/vio.c 2009-11-02 22:19:58 +0000 +++ b/vio/vio.c 2009-11-20 12:09:50 +0000 @@ -62,10 +62,8 @@ static void vio_init(Vio* vio, enum enum vio->timeout=vio_win32_timeout; /* Set default timeout */ - vio->read_timeout_millis = INFINITE; - vio->write_timeout_millis = INFINITE; - - memset(&(vio->pipe_overlapped), 0, sizeof(OVERLAPPED)); + vio->read_timeout_ms= INFINITE; + vio->write_timeout_ms= INFINITE; vio->pipe_overlapped.hEvent= CreateEvent(NULL, TRUE, FALSE, NULL); DBUG_VOID_RETURN; } @@ -90,8 +88,8 @@ static void vio_init(Vio* vio, enum enum /* Currently, shared memory is on Windows only, hence the below is ok*/ vio->timeout= vio_win32_timeout; /* Set default timeout */ - vio->read_timeout_millis= INFINITE; - vio->write_timeout_millis= INFINITE; + vio->read_timeout_ms= INFINITE; + vio->write_timeout_ms= INFINITE; DBUG_VOID_RETURN; } #endif @@ -115,22 +113,20 @@ static void vio_init(Vio* vio, enum enum DBUG_VOID_RETURN; } #endif /* HAVE_OPENSSL */ - { - vio->viodelete =vio_delete; - vio->vioerrno =vio_errno; - vio->read= (flags & VIO_BUFFERED_READ) ? vio_read_buff : vio_read; - vio->write =vio_write; - vio->fastsend =vio_fastsend; - vio->viokeepalive =vio_keepalive; - vio->should_retry =vio_should_retry; - vio->was_interrupted=vio_was_interrupted; - vio->vioclose =vio_close; - vio->peer_addr =vio_peer_addr; - vio->in_addr =vio_in_addr; - vio->vioblocking =vio_blocking; - vio->is_blocking =vio_is_blocking; - vio->timeout =vio_timeout; - } + vio->viodelete =vio_delete; + vio->vioerrno =vio_errno; + vio->read= (flags & VIO_BUFFERED_READ) ? vio_read_buff : vio_read; + vio->write =vio_write; + vio->fastsend =vio_fastsend; + vio->viokeepalive =vio_keepalive; + vio->should_retry =vio_should_retry; + vio->was_interrupted=vio_was_interrupted; + vio->vioclose =vio_close; + vio->peer_addr =vio_peer_addr; + vio->in_addr =vio_in_addr; + vio->vioblocking =vio_blocking; + vio->is_blocking =vio_is_blocking; + vio->timeout =vio_timeout; DBUG_VOID_RETURN; } === modified file 'vio/viosocket.c' --- a/vio/viosocket.c 2009-12-03 11:19:05 +0000 +++ b/vio/viosocket.c 2010-01-15 15:27:55 +0000 @@ -428,14 +428,14 @@ void vio_timeout(Vio *vio, uint which, u /* Finish pending IO on pipe. Honor wait timeout */ -static int pipe_complete_io(Vio* vio, char* buf, size_t size, DWORD timeout_millis) +static size_t pipe_complete_io(Vio* vio, char* buf, size_t size, DWORD timeout_ms) { DWORD length; DWORD ret; DBUG_ENTER("pipe_complete_io"); - ret= WaitForSingleObject(vio->pipe_overlapped.hEvent, timeout_millis); + ret= WaitForSingleObject(vio->pipe_overlapped.hEvent, timeout_ms); /* WaitForSingleObjects will normally return WAIT_OBJECT_O (success, IO completed) or WAIT_TIMEOUT. @@ -444,14 +444,14 @@ static int pipe_complete_io(Vio* vio, ch { CancelIo(vio->hPipe); DBUG_PRINT("error",("WaitForSingleObject() returned %d", ret)); - DBUG_RETURN(-1); + DBUG_RETURN((size_t)-1); } if (!GetOverlappedResult(vio->hPipe,&(vio->pipe_overlapped),&length, FALSE)) { DBUG_PRINT("error",("GetOverlappedResult() returned last error %d", GetLastError())); - DBUG_RETURN(-1); + DBUG_RETURN((size_t)-1); } DBUG_RETURN(length); @@ -461,49 +461,58 @@ static int pipe_complete_io(Vio* vio, ch size_t vio_read_pipe(Vio * vio, uchar *buf, size_t size) { DWORD bytes_read; + size_t retval; DBUG_ENTER("vio_read_pipe"); DBUG_PRINT("enter", ("sd: %d buf: 0x%lx size: %u", vio->sd, (long) buf, (uint) size)); - if (!ReadFile(vio->hPipe, buf, (DWORD)size, &bytes_read, + if (ReadFile(vio->hPipe, buf, (DWORD)size, &bytes_read, &(vio->pipe_overlapped))) { + retval= bytes_read; + } + else + { if (GetLastError() != ERROR_IO_PENDING) { DBUG_PRINT("error",("ReadFile() returned last error %d", GetLastError())); DBUG_RETURN((size_t)-1); } - bytes_read= pipe_complete_io(vio, buf, size,vio->read_timeout_millis); + retval= pipe_complete_io(vio, buf, size,vio->read_timeout_ms); } - DBUG_PRINT("exit", ("%d", bytes_read)); - DBUG_RETURN(bytes_read); + DBUG_PRINT("exit", ("%lld", (longlong)retval)); + DBUG_RETURN(retval); } size_t vio_write_pipe(Vio * vio, const uchar* buf, size_t size) { DWORD bytes_written; + size_t retval; DBUG_ENTER("vio_write_pipe"); DBUG_PRINT("enter", ("sd: %d buf: 0x%lx size: %u", vio->sd, (long) buf, (uint) size)); - if (!WriteFile(vio->hPipe, buf, (DWORD)size, &bytes_written, + if (WriteFile(vio->hPipe, buf, (DWORD)size, &bytes_written, &(vio->pipe_overlapped))) { + retval= bytes_written; + } + else + { if (GetLastError() != ERROR_IO_PENDING) { DBUG_PRINT("vio_error",("WriteFile() returned last error %d", GetLastError())); DBUG_RETURN((size_t)-1); } - bytes_written = pipe_complete_io(vio, (char *)buf, size, - vio->write_timeout_millis); + retval= pipe_complete_io(vio, (char *)buf, size, vio->write_timeout_ms); } - DBUG_PRINT("exit", ("%d", bytes_written)); - DBUG_RETURN(bytes_written); + DBUG_PRINT("exit", ("%lld", (longlong)retval)); + DBUG_RETURN(retval); } @@ -528,21 +537,21 @@ int vio_close_pipe(Vio * vio) void vio_win32_timeout(Vio *vio, uint which , uint timeout_sec) { - DWORD timeout_millis; + DWORD timeout_ms; /* Windows is measuring timeouts in milliseconds. Check for possible int overflow. */ if (timeout_sec > UINT_MAX/1000) - timeout_millis= INFINITE; + timeout_ms= INFINITE; else - timeout_millis= timeout_sec * 1000; + timeout_ms= timeout_sec * 1000; /* which == 1 means "write", which == 0 means "read".*/ if(which) - vio->write_timeout_millis= timeout_millis; + vio->write_timeout_ms= timeout_ms; else - vio->read_timeout_millis= timeout_millis; + vio->read_timeout_ms= timeout_ms; } @@ -577,7 +586,7 @@ size_t vio_read_shared_memory(Vio * vio, WAIT_ABANDONED_0 and WAIT_TIMEOUT - fail. We can't read anything */ if (WaitForMultipleObjects(array_elements(events), events, FALSE, - vio->read_timeout_millis) != WAIT_OBJECT_0) + vio->read_timeout_ms) != WAIT_OBJECT_0) { DBUG_RETURN(-1); }; @@ -634,7 +643,7 @@ size_t vio_write_shared_memory(Vio * vio while (remain != 0) { if (WaitForMultipleObjects(array_elements(events), events, FALSE, - vio->write_timeout_millis) != WAIT_OBJECT_0) + vio->write_timeout_ms) != WAIT_OBJECT_0) { DBUG_RETURN((size_t) -1); } === modified file 'vio/viossl.c' --- a/vio/viossl.c 2008-02-07 08:55:49 +0000 +++ b/vio/viossl.c 2010-01-29 10:42:31 +0000 @@ -74,8 +74,11 @@ report_errors(SSL* ssl) } if (ssl) - DBUG_PRINT("error", ("error: %s", - ERR_error_string(SSL_get_error(ssl, l), buf))); + { + int error= SSL_get_error(ssl, l); + DBUG_PRINT("error", ("error: %s (%d)", + ERR_error_string(error, buf), error)); + } DBUG_PRINT("info", ("socket_errno: %d", socket_errno)); DBUG_VOID_RETURN; @@ -174,16 +177,17 @@ void vio_ssl_delete(Vio *vio) static int ssl_do(struct st_VioSSLFd *ptr, Vio *vio, long timeout, - int (*connect_accept_func)(SSL*)) + int (*connect_accept_func)(SSL*), char *error_string) { SSL *ssl; my_bool unused; my_bool was_blocking; - DBUG_ENTER("ssl_do"); DBUG_PRINT("enter", ("ptr: 0x%lx, sd: %d ctx: 0x%lx", (long) ptr, vio->sd, (long) ptr->ssl_context)); + error_string[0]= 0; + /* Set socket to blocking if not already set */ vio_blocking(vio, 1, &was_blocking); @@ -191,6 +195,7 @@ static int ssl_do(struct st_VioSSLFd *pt { DBUG_PRINT("error", ("SSL_new failure")); report_errors(ssl); + strmov(error_string, "SSL_new failed"); vio_blocking(vio, was_blocking, &unused); DBUG_RETURN(1); } @@ -203,6 +208,7 @@ static int ssl_do(struct st_VioSSLFd *pt { DBUG_PRINT("error", ("SSL_connect/accept failure")); report_errors(ssl); + ERR_error_string(SSL_get_error(ssl, 0), error_string); SSL_free(ssl); vio_blocking(vio, was_blocking, &unused); DBUG_RETURN(1); @@ -250,17 +256,19 @@ static int ssl_do(struct st_VioSSLFd *pt } -int sslaccept(struct st_VioSSLFd *ptr, Vio *vio, long timeout) +int sslaccept(struct st_VioSSLFd *ptr, Vio *vio, long timeout, + char *error_string) { DBUG_ENTER("sslaccept"); - DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_accept)); + DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_accept, error_string)); } -int sslconnect(struct st_VioSSLFd *ptr, Vio *vio, long timeout) +int sslconnect(struct st_VioSSLFd *ptr, Vio *vio, long timeout, + char *error_string) { DBUG_ENTER("sslconnect"); - DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_connect)); + DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_connect, error_string)); }

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2810: hide nm warnings in configure
by noreply＠launchpad.net 30 Jan '10

30 Jan '10

------------------------------------------------------------ revno: 2810 committer: Sergei Golubchik <sergii(a)pisem.net> branch nick: maria-5.1 timestamp: Fri 2010-01-29 21:37:22 +0100 message: hide nm warnings in configure modified: configure.in -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2808) Bug#31173
by Michael Widenius 29 Jan '10

29 Jan '10

#At lp:maria based on revid:monty@askmonty.org-20100129105202-5pyhpgwqcvxawxhh 2808 Michael Widenius 2010-01-29 Patch set contributed by Alex Budovski (MCA) Fix for Bug#31173: mysqlslap.exe crashes if called without any parameters modified: .bzrignore client/mysqlslap.c mysys/my_thr_init.c sql-common/client.c sql/CMakeLists.txt sql/mysqld.cc sql/sql_profile.cc sql/sql_profile.h sql/udf_example.def storage/maria/ma_close.c per-file messages: .bzrignore Fixed .bzrignore rules. Many were simply not ignoring what they were meant to. client/mysqlslap.c Fixed bug for Bug#31173: mysqlslap.exe crashes if called without any parameters The original patch could cause memory leaks and odd problems depending on how connection was made. This code ensures that all mysql_options() are set for each mysql_real_connect(). (This patch by Monty) mysys/my_thr_init.c Fixed multiply-initialized critical section on Windows, due to code incorrectly checking the wrong field in an attempt to prevent multiple-initialization. sql-common/client.c Don't use shared memory if it's not set (for example after failed mysql_real_connect). Ensure that mysql_close() resets all resources so that it's safe to call it twice. (Patch by monty, related to Bug#31173: mysqlslap.exe crashes if called without any parameters) sql/CMakeLists.txt Added page fault counters for SHOW PROFILE on Windows. sql/mysqld.cc Fixed attempt to set a NULL event. The code now only sets the event if appropriate (i.e. shared memory is being used) sql/sql_profile.cc Added page fault counters for SHOW PROFILE on Windows. sql/sql_profile.h Added page fault counters for SHOW PROFILE on Windows. sql/udf_example.def Some cleanup functions were not exported from udf_example.dll, causing them to never be executed, and as a result multiple-initialization of kernel objects occurred and resources were not being freed correctly. storage/maria/ma_close.c Condition variable share->key_del_cond was never being destroyed, while its containing heap block was being freed in maria_close(), leaking kernel resources. === modified file '.bzrignore' --- a/.bzrignore 2009-12-22 13:50:20 +0000 +++ b/.bzrignore 2010-01-29 18:42:22 +0000 @@ -40,15 +40,15 @@ *.dsp *.Po *.Plo -*/*.dir/* +*.dir/ */*_pure_*warnings */.deps */.libs/* */.pure -*/debug/* -*/minsizerel/* -*/release/* -*/relwithdebinfo/* +debug/ +MinSizeRel/ +Release/ +RelWithDebInfo/ *~ .*.swp ./CMakeCache.txt @@ -96,7 +96,7 @@ BitKeeper/tmp/gone BitKeeper/tmp BitKeeper/log BitKeeper/etc/SCCS -CMakeFiles/* +CMakeFiles/ COPYING COPYING.LIB Docs/#manual.texi# === modified file 'client/mysqlslap.c' --- a/client/mysqlslap.c 2009-12-03 11:34:11 +0000 +++ b/client/mysqlslap.c 2010-01-29 18:42:22 +0000 @@ -292,6 +292,25 @@ static int gettimeofday(struct timeval * } #endif +void set_mysql_connect_options(MYSQL *mysql) +{ + if (opt_compress) + mysql_options(mysql,MYSQL_OPT_COMPRESS,NullS); +#ifdef HAVE_OPENSSL + if (opt_use_ssl) + mysql_ssl_set(mysql, opt_ssl_key, opt_ssl_cert, opt_ssl_ca, + opt_ssl_capath, opt_ssl_cipher); +#endif + if (opt_protocol) + mysql_options(mysql,MYSQL_OPT_PROTOCOL,(char*)&opt_protocol); +#ifdef HAVE_SMEM + if (shared_memory_base_name) + mysql_options(mysql,MYSQL_SHARED_MEMORY_BASE_NAME,shared_memory_base_name); +#endif + mysql_options(mysql, MYSQL_SET_CHARSET_NAME, default_charset); +} + + int main(int argc, char **argv) { MYSQL mysql; @@ -323,20 +342,7 @@ int main(int argc, char **argv) exit(1); } mysql_init(&mysql); - if (opt_compress) - mysql_options(&mysql,MYSQL_OPT_COMPRESS,NullS); -#ifdef HAVE_OPENSSL - if (opt_use_ssl) - mysql_ssl_set(&mysql, opt_ssl_key, opt_ssl_cert, opt_ssl_ca, - opt_ssl_capath, opt_ssl_cipher); -#endif - if (opt_protocol) - mysql_options(&mysql,MYSQL_OPT_PROTOCOL,(char*)&opt_protocol); -#ifdef HAVE_SMEM - if (shared_memory_base_name) - mysql_options(&mysql,MYSQL_SHARED_MEMORY_BASE_NAME,shared_memory_base_name); -#endif - mysql_options(&mysql, MYSQL_SET_CHARSET_NAME, default_charset); + set_mysql_connect_options(&mysql); if (!opt_only_print) { @@ -1815,6 +1821,7 @@ pthread_handler_t run_task(void *p) my_progname, mysql_error(mysql)); exit(0); } + set_mysql_connect_options(mysql); if (mysql_thread_init()) { @@ -1855,7 +1862,6 @@ limit_not_met: my_progname, mysql_error(mysql)); exit(0); } - if (slap_connect(mysql)) goto end; } @@ -2223,6 +2229,7 @@ slap_connect(MYSQL *mysql) int x, connect_error= 1; for (x= 0; x < 10; x++) { + set_mysql_connect_options(mysql); if (mysql_real_connect(mysql, host, user, opt_password, create_schema_string, opt_mysql_port, === modified file 'mysys/my_thr_init.c' --- a/mysys/my_thr_init.c 2009-12-03 11:19:05 +0000 +++ b/mysys/my_thr_init.c 2010-01-29 18:42:22 +0000 @@ -317,7 +317,7 @@ my_bool my_thread_init(void) /* Skip initialization if the thread specific variable is already initialized */ - if (THR_KEY_mysys.id) + if (THR_KEY_mysys.init) goto end; tmp= &THR_KEY_mysys; #endif === modified file 'sql-common/client.c' --- a/sql-common/client.c 2010-01-29 10:42:31 +0000 +++ b/sql-common/client.c 2010-01-29 18:42:22 +0000 @@ -1940,7 +1940,8 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons #if defined(HAVE_SMEM) if ((!mysql->options.protocol || mysql->options.protocol == MYSQL_PROTOCOL_MEMORY) && - (!host || !strcmp(host,LOCAL_HOST))) + (!host || !strcmp(host,LOCAL_HOST)) && + mysql->options.shared_memory_base_name) { if ((create_shared_memory(mysql,net, mysql->options.connect_timeout)) == INVALID_HANDLE_VALUE) @@ -1949,7 +1950,7 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons ("host: '%s' socket: '%s' shared memory: %s have_tcpip: %d", host ? host : "<null>", unix_socket ? unix_socket : "<null>", - (int) mysql->options.shared_memory_base_name, + mysql->options.shared_memory_base_name, (int) have_tcpip)); if (mysql->options.protocol == MYSQL_PROTOCOL_MEMORY) goto error; @@ -2752,6 +2753,13 @@ void mysql_detach_stmt_list(LIST **stmt_ } +/* + Close a MySQL connection and free all resources attached to it. + + This function is coded in such that it can be called multiple times + (As some clients call this after mysql_real_connect() fails) +*/ + void STDCALL mysql_close(MYSQL *mysql) { DBUG_ENTER("mysql_close"); @@ -2785,10 +2793,16 @@ void STDCALL mysql_close(MYSQL *mysql) } #endif if (mysql != mysql->master) + { mysql_close(mysql->master); + mysql->master= 0; + } #ifndef MYSQL_SERVER if (mysql->thd) + { (*mysql->methods->free_embedded_thd)(mysql); + mysql->thd= 0; + } #endif if (mysql->free_me) my_free((uchar*) mysql,MYF(0)); === modified file 'sql/CMakeLists.txt' --- a/sql/CMakeLists.txt 2009-12-03 11:19:05 +0000 +++ b/sql/CMakeLists.txt 2010-01-29 18:42:22 +0000 @@ -97,7 +97,7 @@ SET_TARGET_PROPERTIES(mysqld PROPERTIES SET (MYSQLD_CORE_LIBS mysys zlib dbug strings yassl taocrypt vio regex sql libevent) TARGET_LINK_LIBRARIES(mysqld ${MYSQLD_CORE_LIBS} ${MYSQLD_STATIC_ENGINE_LIBS}) -TARGET_LINK_LIBRARIES(mysqld ws2_32.lib) +TARGET_LINK_LIBRARIES(mysqld ws2_32.lib psapi.lib) IF(MSVC AND NOT WITHOUT_DYNAMIC_PLUGINS) === modified file 'sql/mysqld.cc' --- a/sql/mysqld.cc 2010-01-28 11:35:10 +0000 +++ b/sql/mysqld.cc 2010-01-29 18:42:22 +0000 @@ -1173,11 +1173,14 @@ static void __cdecl kill_server(int sig_ /* Send event to smem_event_connect_request for aborting */ - if (!SetEvent(smem_event_connect_request)) + if (opt_enable_shared_memory) { - DBUG_PRINT("error", - ("Got error: %ld from SetEvent of smem_event_connect_request", - GetLastError())); + if (!SetEvent(smem_event_connect_request)) + { + DBUG_PRINT("error", + ("Got error: %ld from SetEvent of smem_event_connect_request", + GetLastError())); + } } #endif === modified file 'sql/sql_profile.cc' --- a/sql/sql_profile.cc 2009-10-15 21:38:29 +0000 +++ b/sql/sql_profile.cc 2010-01-29 18:42:22 +0000 @@ -131,6 +131,23 @@ int make_profile_table_for_show(THD *thd #define RUSAGE_USEC(tv) ((tv).tv_sec*1000*1000 + (tv).tv_usec) #define RUSAGE_DIFF_USEC(tv1, tv2) (RUSAGE_USEC((tv1))-RUSAGE_USEC((tv2))) +#ifdef __WIN__ +inline ULONGLONG FileTimeToQuadWord(FILETIME *ft) +{ + ULONGLONG nrv = 0; + nrv |= ft->dwHighDateTime; + nrv <<= 32; + nrv |= ft->dwLowDateTime; + return nrv; +} + + +// Get time difference between to FILETIME objects in seconds. +inline double GetTimeDiffInSeconds(FILETIME *a, FILETIME *b) +{ + return ((FileTimeToQuadWord(a) - FileTimeToQuadWord(b)) / 1e7); +} +#endif /* __WIN__ */ PROF_MEASUREMENT::PROF_MEASUREMENT(QUERY_PROFILE *profile_arg, const char *status_arg) @@ -221,6 +238,11 @@ void PROF_MEASUREMENT::collect() time_usecs= (double) my_getsystime() / 10.0; /* 1 sec was 1e7, now is 1e6 */ #ifdef HAVE_GETRUSAGE getrusage(RUSAGE_SELF, &rusage); +#elif defined(__WIN__) + FILETIME ftDummy; + GetProcessTimes(GetCurrentProcess(), &ftDummy, &ftDummy, &ftKernel, &ftUser); + GetProcessIoCounters(GetCurrentProcess(), &io_count); + GetProcessMemoryInfo(GetCurrentProcess(), &mem_count, sizeof(mem_count)); #endif } @@ -590,6 +612,23 @@ int PROFILING::fill_statistics_info(THD table->field[5]->store_decimal(&cpu_stime_decimal); table->field[4]->set_notnull(); table->field[5]->set_notnull(); +#elif defined(__WIN__) + my_decimal cpu_utime_decimal, cpu_stime_decimal; + + double2my_decimal(E_DEC_FATAL_ERROR, + GetTimeDiffInSeconds(&entry->ftUser, + &previous->ftUser), + &cpu_utime_decimal); + double2my_decimal(E_DEC_FATAL_ERROR, + GetTimeDiffInSeconds(&entry->ftKernel, + &previous->ftKernel), + &cpu_stime_decimal); + + // Store the result. + table->field[4]->store_decimal(&cpu_utime_decimal); + table->field[5]->store_decimal(&cpu_stime_decimal); + table->field[4]->set_notnull(); + table->field[5]->set_notnull(); #else /* TODO: Add CPU-usage info for non-BSD systems */ #endif @@ -612,6 +651,17 @@ int PROFILING::fill_statistics_info(THD table->field[9]->store((uint32)(entry->rusage.ru_oublock - previous->rusage.ru_oublock)); table->field[9]->set_notnull(); +#elif defined(__WIN__) + ULONGLONG reads_delta = entry->io_count.ReadOperationCount - + previous->io_count.ReadOperationCount; + ULONGLONG writes_delta = entry->io_count.WriteOperationCount - + previous->io_count.WriteOperationCount; + + table->field[8]->store((uint32)reads_delta); + table->field[8]->set_notnull(); + + table->field[9]->store((uint32)writes_delta); + table->field[9]->set_notnull(); #else /* TODO: Add block IO info for non-BSD systems */ #endif @@ -634,6 +684,13 @@ int PROFILING::fill_statistics_info(THD table->field[13]->store((uint32)(entry->rusage.ru_minflt - previous->rusage.ru_minflt), true); table->field[13]->set_notnull(); +#elif defined(__WIN__) + /* Windows APIs don't easily distinguish between hard and soft page + faults, so we just fill the 'major' column and leave the second NULL. + */ + table->field[12]->store((uint32)(entry->mem_count.PageFaultCount - + previous->mem_count.PageFaultCount), true); + table->field[12]->set_notnull(); #else /* TODO: Add page fault info for non-BSD systems */ #endif === modified file 'sql/sql_profile.h' --- a/sql/sql_profile.h 2009-09-07 20:50:10 +0000 +++ b/sql/sql_profile.h 2010-01-29 18:42:22 +0000 @@ -36,6 +36,10 @@ int make_profile_table_for_show(THD *thd #if defined(ENABLED_PROFILING) && defined(COMMUNITY_SERVER) #include "mysql_priv.h" +#ifdef __WIN__ +#include <psapi.h> +#endif + #ifdef HAVE_SYS_RESOURCE_H #include <sys/resource.h> #endif @@ -165,6 +169,10 @@ private: char *status; #ifdef HAVE_GETRUSAGE struct rusage rusage; +#elif defined(__WIN__) + FILETIME ftKernel, ftUser; + IO_COUNTERS io_count; + PROCESS_MEMORY_COUNTERS mem_count; #endif char *function; === modified file 'sql/udf_example.def' --- a/sql/udf_example.def 2007-10-29 14:01:40 +0000 +++ b/sql/udf_example.def 2010-01-29 18:42:22 +0000 @@ -3,8 +3,10 @@ VERSION 1.0 EXPORTS lookup lookup_init + lookup_deinit reverse_lookup reverse_lookup_init + reverse_lookup_deinit metaphon_init metaphon_deinit metaphon === modified file 'storage/maria/ma_close.c' --- a/storage/maria/ma_close.c 2009-02-05 22:38:30 +0000 +++ b/storage/maria/ma_close.c 2010-01-29 18:42:22 +0000 @@ -177,6 +177,7 @@ int maria_close(register MARIA_HA *info) { (void) pthread_mutex_destroy(&share->intern_lock); (void) pthread_mutex_destroy(&share->close_lock); + (void) pthread_cond_destroy(&share->key_del_cond); my_free((uchar *)share, MYF(0)); /* If share cannot be freed, it's because checkpoint has previously

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2808: Patch set contributed by Alex Budovski (MCA)
by noreply＠launchpad.net 29 Jan '10

29 Jan '10

------------------------------------------------------------ revno: 2808 committer: Michael Widenius <monty(a)askmonty.org> branch nick: maria-5.1 timestamp: Fri 2010-01-29 20:42:22 +0200 message: Patch set contributed by Alex Budovski (MCA) Fix for Bug#31173: mysqlslap.exe crashes if called without any parameters modified: .bzrignore client/mysqlslap.c mysys/my_thr_init.c sql-common/client.c sql/CMakeLists.txt sql/mysqld.cc sql/sql_profile.cc sql/sql_profile.h sql/udf_example.def storage/maria/ma_close.c -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] Fix for 2 failing tests on Windows
by Alex Budovski 29 Jan '10

29 Jan '10

Hello, Some tests that call mtr.add_suppression to ignore expected warnings hard-code the forward slash '/' character as the expected path delimiter. This breaks tests on Windows as the regex does not match as expected. E.g. Recovering table: './mysqltest/t_corrupted2' fails to match 100123 23:48:26 [Warning] Recovering table: '.\mysqltest\t_corrupted2' and hence the test fails spuriously. Attached is a simple patch that fixes this issue while maintaining compatibility with other platforms. Let me know what you think. - Alex PS. From the Windows buildbot test logs[1], only these two tests are observed to fail in this way. [1] http://askmonty.org/buildbot/builders/win32-tarball/builds/238/steps/test/l…

3 4

[Maria-developers] Updated (by Monty): Subquery optimization: Avoid recalculating subquery if external fields values found in subquery cache (66)
by worklog-noreply＠askmonty.org 29 Jan '10

29 Jan '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subquery optimization: Avoid recalculating subquery if external fields values found in subquery cache CREATION DATE..: Wed, 25 Nov 2009, 22:25 SUPERVISOR.....: Monty IMPLEMENTOR....: Sanja COPIES TO......: CATEGORY.......: Server-BackLog TASK ID........: 66 (http://askmonty.org/worklog/?tid=66) VERSION........: Server-5.3 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Monty - Fri, 29 Jan 2010, 19:07)=-=- Version updated. --- /tmp/wklog.66.old.5893 2010-01-29 19:07:10.000000000 +0200 +++ /tmp/wklog.66.new.5893 2010-01-29 19:07:10.000000000 +0200 @@ -1 +1 @@ -Server-5.2 +Server-5.3 -=-=(Psergey - Wed, 20 Jan 2010, 14:50)=-=- High-Level Specification modified. --- /tmp/wklog.66.old.26873 2010-01-20 14:50:41.000000000 +0200 +++ /tmp/wklog.66.new.26873 2010-01-20 14:50:41.000000000 +0200 @@ -4,7 +4,6 @@ To check/discuss: ----------------- -* Do we put subquery cache on all levels of subqueries or on highest level only * Will there be any means to measure subquery cache hit rate? * MySQL-6.0 has a one-element predicate result cache. It is called "left expression cache", grep for left_expr_cache in sql/item_subselect.* @@ -41,7 +40,12 @@ - subquery_item_result is 'bool' for subquery predicates, and is of some scalar or ROW(scalar1,...scalarN) type for scalar-context subquery. -We dont support cases when outer_expr or correlation_references are blobs. +We don't support cases when outer_expr or correlation_references are blobs. + +All subquery predicates are cached. That is, if one subquery predicate is +located within another, both of them will have caches (one option to reduce +cache memory usage was to use cache only for the upper-most select. we decided +against it). 2. Data structure used for the cache ------------------------------------ -=-=(Psergey - Wed, 20 Jan 2010, 13:07)=-=- High-Level Specification modified. --- /tmp/wklog.66.old.17649 2010-01-20 13:07:07.000000000 +0200 +++ /tmp/wklog.66.new.17649 2010-01-20 13:07:07.000000000 +0200 @@ -3,7 +3,13 @@ To check/discuss: - To put subquery cache on all levels of subqueries or on highest level only. +----------------- +* Do we put subquery cache on all levels of subqueries or on highest level only +* Will there be any means to measure subquery cache hit rate? +* MySQL-6.0 has a one-element predicate result cache. It is called "left + expression cache", grep for left_expr_cache in sql/item_subselect.* + When this WL is merged with 6.0's optimizations, these two caches will + need to be unified somehow. <contents> -=-=(Psergey - Mon, 18 Jan 2010, 16:40)=-=- Low Level Design modified. --- /tmp/wklog.66.old.24899 2010-01-18 16:40:16.000000000 +0200 +++ /tmp/wklog.66.new.24899 2010-01-18 16:40:16.000000000 +0200 @@ -1,3 +1,5 @@ +* Target version: base on mysql-5.2 code + All items on which subquery depend could be collected in st_select_lex::mark_as_dependent (direct of indirect reference?) -=-=(Psergey - Mon, 18 Jan 2010, 16:37)=-=- Low Level Design modified. --- /tmp/wklog.66.old.24586 2010-01-18 16:37:07.000000000 +0200 +++ /tmp/wklog.66.new.24586 2010-01-18 16:37:07.000000000 +0200 @@ -4,6 +4,11 @@ Temporary table index should be created by all fields except result field (TMP_TABLE_PARAM::keyinfo). +How to fill the temptable +------------------------- +Can reuse approach from SJ-Materialization. Its code is in end_sj_materialize() +and is supposed to be quite trivial. + How to make lookups into temptable ---------------------------------- We'll reuse approach used by SJ-Materialization in 6.0. -=-=(Psergey - Mon, 18 Jan 2010, 16:34)=-=- Low Level Design modified. --- /tmp/wklog.66.old.24328 2010-01-18 16:34:19.000000000 +0200 +++ /tmp/wklog.66.new.24328 2010-01-18 16:34:19.000000000 +0200 @@ -32,8 +32,8 @@ Question: or perhaps that is not necessarry? </questionable> -Execution process -~~~~~~~~~~~~~~~~~ +Doing the lookup +~~~~~~~~~~~~~~~~ SJ-Materialization does lookup in sub_select_sjm(), with this code: /* Do index lookup in the materialized table */ @@ -42,4 +42,12 @@ if (res || !sjm->in_equality->val_int()) DBUG_RETURN(NESTED_LOOP_NO_MORE_ROWS); +The code in this WL will use the same approach +Extracting the value of the subquery predicate +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The goal of making the lookup is to get the value of subquery predicate. +This is done by creating an Item_field $I which refers to appropriate +temporary table's field and then subquery_predicate->val_int() will invoke +$I->val_int(), subquery_predicate->val_str() will invoke $I->val_str() and so +forth. -=-=(Psergey - Mon, 18 Jan 2010, 16:23)=-=- Low Level Design modified. --- /tmp/wklog.66.old.23203 2010-01-18 16:23:18.000000000 +0200 +++ /tmp/wklog.66.new.23203 2010-01-18 16:23:18.000000000 +0200 @@ -31,3 +31,15 @@ Question: or perhaps that is not necessarry? </questionable> + +Execution process +~~~~~~~~~~~~~~~~~ +SJ-Materialization does lookup in sub_select_sjm(), with this code: + + /* Do index lookup in the materialized table */ + if ((res= join_read_key2(join_tab, sjm->table, sjm->tab_ref)) == 1) + DBUG_RETURN(NESTED_LOOP_ERROR); /* purecov: inspected */ + if (res || !sjm->in_equality->val_int()) + DBUG_RETURN(NESTED_LOOP_NO_MORE_ROWS); + + -=-=(Psergey - Mon, 18 Jan 2010, 16:22)=-=- Low Level Design modified. --- /tmp/wklog.66.old.23076 2010-01-18 16:22:07.000000000 +0200 +++ /tmp/wklog.66.new.23076 2010-01-18 16:22:07.000000000 +0200 @@ -4,3 +4,30 @@ Temporary table index should be created by all fields except result field (TMP_TABLE_PARAM::keyinfo). +How to make lookups into temptable +---------------------------------- +We'll reuse approach used by SJ-Materialization in 6.0. + +Setup process +~~~~~~~~~~~~~ +Setup is performed in the same way as in setup_sj_materialization(), +see the code that starts these lines: + + /* + Create/initialize everything we will need to index lookups into the + temptable. + */ + +and ends at this line: + + Remove the injected semi-join IN-equalities from join_tab conds. This + +<questionable> +We'll also need to check equalities, i.e. do an equivalent of this: + + if (!(sjm->in_equality= create_subq_in_equalities(thd, sjm, + emb_sj_nest->sj_subq_pred))) + DBUG_RETURN(TRUE); /* purecov: inspected */ + +Question: or perhaps that is not necessarry? +</questionable> -=-=(Psergey - Tue, 12 Jan 2010, 18:39)=-=- High-Level Specification modified. --- /tmp/wklog.66.old.31666 2010-01-12 18:39:43.000000000 +0200 +++ /tmp/wklog.66.new.31666 2010-01-12 18:39:43.000000000 +0200 @@ -4,3 +4,99 @@ To check/discuss: To put subquery cache on all levels of subqueries or on highest level only. + + +<contents> +1. Scope of the task +2. Data structure used for the cache +3. Cache size +4. Interplay with other subquery optimizations +5. User interface +</contents> + +1. Scope of the task +-------------------- +This WL should handle all subquery predicates, i.e. it should handle these +cases: + + outer_expr IN (SELECT correlated_select) + outer_expr $CMP$ ALL/ANY (SELECT correlated_select) + EXISTS (SELECT correlated_select) + scalar-context subquery: (SELECT correlated_select) + +The cache will maintain + + (outer_expr, correlation_references)-> subquery_item_result + +mapping, where +- correlation_references is a list of tablename.column_name that are referred + from the correlated_select but tablename is a table that is ouside the + subquery. +- subquery_item_result is 'bool' for subquery predicates, and is of +some scalar or ROW(scalar1,...scalarN) type for scalar-context subquery. + +We dont support cases when outer_expr or correlation_references are blobs. + +2. Data structure used for the cache +------------------------------------ +There are two data structures available in the codebase that will allow fast +equality lookups: + +1. HASH (mysys/hash.c) tables +2. Temporary tables (the ones that are used for e.g. GROUP BY) + +None of them has any support for element eviction on overflow (using LRU or +some other policy). + +Query cache and MyISAM/Maria's key/page cache ought to support some eviction +mechanism, but code-wise it is not readily reusable, one will need to factor +it out (or copy it). + +We choose to use #2, and not to have any eviction policy. See subsequent +sections for details and reasoning behind the decision. + +3. Cache size +------------- +Typically, a cache has some maximum size and a policy which is used to +select a cache entry for removal when the cache becomes full (e.g. find +and remove the least [recently] used entry) + +For this WL entry we will use a cache of infinite size. The reasoning behind +this is that: +- is is easy to do: we have temporary tables that can grow to arbitrarily + large size while still providing the same insert/lookup interface. +- it suits us: unless the subquery is resolved with one index lookup, + hitting the cache would be many times cheaper than re-running the + subquery, so cache is worth having. + +4. Interplay with other subquery optimizations +---------------------------------------------- +* This WL entry should not care about IN->EXISTS transformation: caching for + IN subquery and result of its conversion to EXISTS would work in the same + way. + +* This optimization is orthogonal to <=>ANY -> MIN/MAX rewrite (it will + work/be useful irrespectively of whether the rewrite has been performed or + not) + +* TODO: compare this with materialization for uncorrelated IN-subqueries. Is + this basically the same? + A: no, it is not: + - IN-Materialization has to perform full materialization before it can + do the first subquery evaluation. This WL's code has almost no startup + costs. + - This optimization has temp.table of (corr_reference, predicate_value), + while IN-materialization will have (corr_reference) only. + +5. User interface +----------------- +* There will be an @@optimizer_switch flag to turn this optimization on and + off (TODO: name of the flag?) + +* TODO: how do we show this in EXPLAIN [EXTENDED]? The most easiest is to + print something in the warning text of EXPLAIN EXTEDED that would indicate + use of cache. + +* temporary table sizing (max size for heap table, whether to use MyISAM or + Maria) will be controlled with common temp.table control variables. + -=-=(Psergey - Mon, 11 Jan 2010, 13:25)=-=- As of today, there is code that - collects outside references - creates a temporary table with index that would allow for fast lookups. there is no code to - fill the temporary table - make lookups into it Reported zero hours worked. Estimate unchanged. ------------------------------------------------------------ -=-=(View All Progress Notes, 13 total)=-=- http://askmonty.org/worklog/index.pl?tid=66&nolimit=1 DESCRIPTION: Collect all outer items/references (left part of the subquiery and outer references inside the subquery) in key string. Compare the string (which represents certain value set of the references) against values in hash table and return cached result of subquery if the reference values combination has already been used. For example in the following subquery: (L1, L2) IN (SELECT A, B FROM T WHERE T.F1>OTER_FIELD) set of references to look into the subquery cache is (L1, L2, OTER_FIELD). The subquery cache should be implemented as simple LRU connected to the subquery. Size of the subquery cache (in number of results (but maybe in used memory amount)) is limited by session variable (query parameter?). HIGH-LEVEL SPECIFICATION: Attach subquery cache to each Item_subquery. Interface should allow to use hash or temporary table inside. To check/discuss: ----------------- * Will there be any means to measure subquery cache hit rate? * MySQL-6.0 has a one-element predicate result cache. It is called "left expression cache", grep for left_expr_cache in sql/item_subselect.* When this WL is merged with 6.0's optimizations, these two caches will need to be unified somehow. <contents> 1. Scope of the task 2. Data structure used for the cache 3. Cache size 4. Interplay with other subquery optimizations 5. User interface </contents> 1. Scope of the task -------------------- This WL should handle all subquery predicates, i.e. it should handle these cases: outer_expr IN (SELECT correlated_select) outer_expr $CMP$ ALL/ANY (SELECT correlated_select) EXISTS (SELECT correlated_select) scalar-context subquery: (SELECT correlated_select) The cache will maintain (outer_expr, correlation_references)-> subquery_item_result mapping, where - correlation_references is a list of tablename.column_name that are referred from the correlated_select but tablename is a table that is ouside the subquery. - subquery_item_result is 'bool' for subquery predicates, and is of some scalar or ROW(scalar1,...scalarN) type for scalar-context subquery. We don't support cases when outer_expr or correlation_references are blobs. All subquery predicates are cached. That is, if one subquery predicate is located within another, both of them will have caches (one option to reduce cache memory usage was to use cache only for the upper-most select. we decided against it). 2. Data structure used for the cache ------------------------------------ There are two data structures available in the codebase that will allow fast equality lookups: 1. HASH (mysys/hash.c) tables 2. Temporary tables (the ones that are used for e.g. GROUP BY) None of them has any support for element eviction on overflow (using LRU or some other policy). Query cache and MyISAM/Maria's key/page cache ought to support some eviction mechanism, but code-wise it is not readily reusable, one will need to factor it out (or copy it). We choose to use #2, and not to have any eviction policy. See subsequent sections for details and reasoning behind the decision. 3. Cache size ------------- Typically, a cache has some maximum size and a policy which is used to select a cache entry for removal when the cache becomes full (e.g. find and remove the least [recently] used entry) For this WL entry we will use a cache of infinite size. The reasoning behind this is that: - is is easy to do: we have temporary tables that can grow to arbitrarily large size while still providing the same insert/lookup interface. - it suits us: unless the subquery is resolved with one index lookup, hitting the cache would be many times cheaper than re-running the subquery, so cache is worth having. 4. Interplay with other subquery optimizations ---------------------------------------------- * This WL entry should not care about IN->EXISTS transformation: caching for IN subquery and result of its conversion to EXISTS would work in the same way. * This optimization is orthogonal to <=>ANY -> MIN/MAX rewrite (it will work/be useful irrespectively of whether the rewrite has been performed or not) * TODO: compare this with materialization for uncorrelated IN-subqueries. Is this basically the same? A: no, it is not: - IN-Materialization has to perform full materialization before it can do the first subquery evaluation. This WL's code has almost no startup costs. - This optimization has temp.table of (corr_reference, predicate_value), while IN-materialization will have (corr_reference) only. 5. User interface ----------------- * There will be an @@optimizer_switch flag to turn this optimization on and off (TODO: name of the flag?) * TODO: how do we show this in EXPLAIN [EXTENDED]? The most easiest is to print something in the warning text of EXPLAIN EXTEDED that would indicate use of cache. * temporary table sizing (max size for heap table, whether to use MyISAM or Maria) will be controlled with common temp.table control variables. LOW-LEVEL DESIGN: * Target version: base on mysql-5.2 code All items on which subquery depend could be collected in st_select_lex::mark_as_dependent (direct of indirect reference?) Temporary table index should be created by all fields except result field (TMP_TABLE_PARAM::keyinfo). How to fill the temptable ------------------------- Can reuse approach from SJ-Materialization. Its code is in end_sj_materialize() and is supposed to be quite trivial. How to make lookups into temptable ---------------------------------- We'll reuse approach used by SJ-Materialization in 6.0. Setup process ~~~~~~~~~~~~~ Setup is performed in the same way as in setup_sj_materialization(), see the code that starts these lines: /* Create/initialize everything we will need to index lookups into the temptable. */ and ends at this line: Remove the injected semi-join IN-equalities from join_tab conds. This <questionable> We'll also need to check equalities, i.e. do an equivalent of this: if (!(sjm->in_equality= create_subq_in_equalities(thd, sjm, emb_sj_nest->sj_subq_pred))) DBUG_RETURN(TRUE); /* purecov: inspected */ Question: or perhaps that is not necessarry? </questionable> Doing the lookup ~~~~~~~~~~~~~~~~ SJ-Materialization does lookup in sub_select_sjm(), with this code: /* Do index lookup in the materialized table */ if ((res= join_read_key2(join_tab, sjm->table, sjm->tab_ref)) == 1) DBUG_RETURN(NESTED_LOOP_ERROR); /* purecov: inspected */ if (res || !sjm->in_equality->val_int()) DBUG_RETURN(NESTED_LOOP_NO_MORE_ROWS); The code in this WL will use the same approach Extracting the value of the subquery predicate ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The goal of making the lookup is to get the value of subquery predicate. This is done by creating an Item_field $I which refers to appropriate temporary table's field and then subquery_predicate->val_int() will invoke $I->val_int(), subquery_predicate->val_str() will invoke $I->val_str() and so forth. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Monty): Subquery optimization: Avoid recalculating subquery if external fields values found in subquery cache (66)
by worklog-noreply＠askmonty.org 29 Jan '10

29 Jan '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subquery optimization: Avoid recalculating subquery if external fields values found in subquery cache CREATION DATE..: Wed, 25 Nov 2009, 22:25 SUPERVISOR.....: Monty IMPLEMENTOR....: Sanja COPIES TO......: CATEGORY.......: Server-BackLog TASK ID........: 66 (http://askmonty.org/worklog/?tid=66) VERSION........: Server-5.3 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Monty - Fri, 29 Jan 2010, 19:07)=-=- Version updated. --- /tmp/wklog.66.old.5893 2010-01-29 19:07:10.000000000 +0200 +++ /tmp/wklog.66.new.5893 2010-01-29 19:07:10.000000000 +0200 @@ -1 +1 @@ -Server-5.2 +Server-5.3 -=-=(Psergey - Wed, 20 Jan 2010, 14:50)=-=- High-Level Specification modified. --- /tmp/wklog.66.old.26873 2010-01-20 14:50:41.000000000 +0200 +++ /tmp/wklog.66.new.26873 2010-01-20 14:50:41.000000000 +0200 @@ -4,7 +4,6 @@ To check/discuss: ----------------- -* Do we put subquery cache on all levels of subqueries or on highest level only * Will there be any means to measure subquery cache hit rate? * MySQL-6.0 has a one-element predicate result cache. It is called "left expression cache", grep for left_expr_cache in sql/item_subselect.* @@ -41,7 +40,12 @@ - subquery_item_result is 'bool' for subquery predicates, and is of some scalar or ROW(scalar1,...scalarN) type for scalar-context subquery. -We dont support cases when outer_expr or correlation_references are blobs. +We don't support cases when outer_expr or correlation_references are blobs. + +All subquery predicates are cached. That is, if one subquery predicate is +located within another, both of them will have caches (one option to reduce +cache memory usage was to use cache only for the upper-most select. we decided +against it). 2. Data structure used for the cache ------------------------------------ -=-=(Psergey - Wed, 20 Jan 2010, 13:07)=-=- High-Level Specification modified. --- /tmp/wklog.66.old.17649 2010-01-20 13:07:07.000000000 +0200 +++ /tmp/wklog.66.new.17649 2010-01-20 13:07:07.000000000 +0200 @@ -3,7 +3,13 @@ To check/discuss: - To put subquery cache on all levels of subqueries or on highest level only. +----------------- +* Do we put subquery cache on all levels of subqueries or on highest level only +* Will there be any means to measure subquery cache hit rate? +* MySQL-6.0 has a one-element predicate result cache. It is called "left + expression cache", grep for left_expr_cache in sql/item_subselect.* + When this WL is merged with 6.0's optimizations, these two caches will + need to be unified somehow. <contents> -=-=(Psergey - Mon, 18 Jan 2010, 16:40)=-=- Low Level Design modified. --- /tmp/wklog.66.old.24899 2010-01-18 16:40:16.000000000 +0200 +++ /tmp/wklog.66.new.24899 2010-01-18 16:40:16.000000000 +0200 @@ -1,3 +1,5 @@ +* Target version: base on mysql-5.2 code + All items on which subquery depend could be collected in st_select_lex::mark_as_dependent (direct of indirect reference?) -=-=(Psergey - Mon, 18 Jan 2010, 16:37)=-=- Low Level Design modified. --- /tmp/wklog.66.old.24586 2010-01-18 16:37:07.000000000 +0200 +++ /tmp/wklog.66.new.24586 2010-01-18 16:37:07.000000000 +0200 @@ -4,6 +4,11 @@ Temporary table index should be created by all fields except result field (TMP_TABLE_PARAM::keyinfo). +How to fill the temptable +------------------------- +Can reuse approach from SJ-Materialization. Its code is in end_sj_materialize() +and is supposed to be quite trivial. + How to make lookups into temptable ---------------------------------- We'll reuse approach used by SJ-Materialization in 6.0. -=-=(Psergey - Mon, 18 Jan 2010, 16:34)=-=- Low Level Design modified. --- /tmp/wklog.66.old.24328 2010-01-18 16:34:19.000000000 +0200 +++ /tmp/wklog.66.new.24328 2010-01-18 16:34:19.000000000 +0200 @@ -32,8 +32,8 @@ Question: or perhaps that is not necessarry? </questionable> -Execution process -~~~~~~~~~~~~~~~~~ +Doing the lookup +~~~~~~~~~~~~~~~~ SJ-Materialization does lookup in sub_select_sjm(), with this code: /* Do index lookup in the materialized table */ @@ -42,4 +42,12 @@ if (res || !sjm->in_equality->val_int()) DBUG_RETURN(NESTED_LOOP_NO_MORE_ROWS); +The code in this WL will use the same approach +Extracting the value of the subquery predicate +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The goal of making the lookup is to get the value of subquery predicate. +This is done by creating an Item_field $I which refers to appropriate +temporary table's field and then subquery_predicate->val_int() will invoke +$I->val_int(), subquery_predicate->val_str() will invoke $I->val_str() and so +forth. -=-=(Psergey - Mon, 18 Jan 2010, 16:23)=-=- Low Level Design modified. --- /tmp/wklog.66.old.23203 2010-01-18 16:23:18.000000000 +0200 +++ /tmp/wklog.66.new.23203 2010-01-18 16:23:18.000000000 +0200 @@ -31,3 +31,15 @@ Question: or perhaps that is not necessarry? </questionable> + +Execution process +~~~~~~~~~~~~~~~~~ +SJ-Materialization does lookup in sub_select_sjm(), with this code: + + /* Do index lookup in the materialized table */ + if ((res= join_read_key2(join_tab, sjm->table, sjm->tab_ref)) == 1) + DBUG_RETURN(NESTED_LOOP_ERROR); /* purecov: inspected */ + if (res || !sjm->in_equality->val_int()) + DBUG_RETURN(NESTED_LOOP_NO_MORE_ROWS); + + -=-=(Psergey - Mon, 18 Jan 2010, 16:22)=-=- Low Level Design modified. --- /tmp/wklog.66.old.23076 2010-01-18 16:22:07.000000000 +0200 +++ /tmp/wklog.66.new.23076 2010-01-18 16:22:07.000000000 +0200 @@ -4,3 +4,30 @@ Temporary table index should be created by all fields except result field (TMP_TABLE_PARAM::keyinfo). +How to make lookups into temptable +---------------------------------- +We'll reuse approach used by SJ-Materialization in 6.0. + +Setup process +~~~~~~~~~~~~~ +Setup is performed in the same way as in setup_sj_materialization(), +see the code that starts these lines: + + /* + Create/initialize everything we will need to index lookups into the + temptable. + */ + +and ends at this line: + + Remove the injected semi-join IN-equalities from join_tab conds. This + +<questionable> +We'll also need to check equalities, i.e. do an equivalent of this: + + if (!(sjm->in_equality= create_subq_in_equalities(thd, sjm, + emb_sj_nest->sj_subq_pred))) + DBUG_RETURN(TRUE); /* purecov: inspected */ + +Question: or perhaps that is not necessarry? +</questionable> -=-=(Psergey - Tue, 12 Jan 2010, 18:39)=-=- High-Level Specification modified. --- /tmp/wklog.66.old.31666 2010-01-12 18:39:43.000000000 +0200 +++ /tmp/wklog.66.new.31666 2010-01-12 18:39:43.000000000 +0200 @@ -4,3 +4,99 @@ To check/discuss: To put subquery cache on all levels of subqueries or on highest level only. + + +<contents> +1. Scope of the task +2. Data structure used for the cache +3. Cache size +4. Interplay with other subquery optimizations +5. User interface +</contents> + +1. Scope of the task +-------------------- +This WL should handle all subquery predicates, i.e. it should handle these +cases: + + outer_expr IN (SELECT correlated_select) + outer_expr $CMP$ ALL/ANY (SELECT correlated_select) + EXISTS (SELECT correlated_select) + scalar-context subquery: (SELECT correlated_select) + +The cache will maintain + + (outer_expr, correlation_references)-> subquery_item_result + +mapping, where +- correlation_references is a list of tablename.column_name that are referred + from the correlated_select but tablename is a table that is ouside the + subquery. +- subquery_item_result is 'bool' for subquery predicates, and is of +some scalar or ROW(scalar1,...scalarN) type for scalar-context subquery. + +We dont support cases when outer_expr or correlation_references are blobs. + +2. Data structure used for the cache +------------------------------------ +There are two data structures available in the codebase that will allow fast +equality lookups: + +1. HASH (mysys/hash.c) tables +2. Temporary tables (the ones that are used for e.g. GROUP BY) + +None of them has any support for element eviction on overflow (using LRU or +some other policy). + +Query cache and MyISAM/Maria's key/page cache ought to support some eviction +mechanism, but code-wise it is not readily reusable, one will need to factor +it out (or copy it). + +We choose to use #2, and not to have any eviction policy. See subsequent +sections for details and reasoning behind the decision. + +3. Cache size +------------- +Typically, a cache has some maximum size and a policy which is used to +select a cache entry for removal when the cache becomes full (e.g. find +and remove the least [recently] used entry) + +For this WL entry we will use a cache of infinite size. The reasoning behind +this is that: +- is is easy to do: we have temporary tables that can grow to arbitrarily + large size while still providing the same insert/lookup interface. +- it suits us: unless the subquery is resolved with one index lookup, + hitting the cache would be many times cheaper than re-running the + subquery, so cache is worth having. + +4. Interplay with other subquery optimizations +---------------------------------------------- +* This WL entry should not care about IN->EXISTS transformation: caching for + IN subquery and result of its conversion to EXISTS would work in the same + way. + +* This optimization is orthogonal to <=>ANY -> MIN/MAX rewrite (it will + work/be useful irrespectively of whether the rewrite has been performed or + not) + +* TODO: compare this with materialization for uncorrelated IN-subqueries. Is + this basically the same? + A: no, it is not: + - IN-Materialization has to perform full materialization before it can + do the first subquery evaluation. This WL's code has almost no startup + costs. + - This optimization has temp.table of (corr_reference, predicate_value), + while IN-materialization will have (corr_reference) only. + +5. User interface +----------------- +* There will be an @@optimizer_switch flag to turn this optimization on and + off (TODO: name of the flag?) + +* TODO: how do we show this in EXPLAIN [EXTENDED]? The most easiest is to + print something in the warning text of EXPLAIN EXTEDED that would indicate + use of cache. + +* temporary table sizing (max size for heap table, whether to use MyISAM or + Maria) will be controlled with common temp.table control variables. + -=-=(Psergey - Mon, 11 Jan 2010, 13:25)=-=- As of today, there is code that - collects outside references - creates a temporary table with index that would allow for fast lookups. there is no code to - fill the temporary table - make lookups into it Reported zero hours worked. Estimate unchanged. ------------------------------------------------------------ -=-=(View All Progress Notes, 13 total)=-=- http://askmonty.org/worklog/index.pl?tid=66&nolimit=1 DESCRIPTION: Collect all outer items/references (left part of the subquiery and outer references inside the subquery) in key string. Compare the string (which represents certain value set of the references) against values in hash table and return cached result of subquery if the reference values combination has already been used. For example in the following subquery: (L1, L2) IN (SELECT A, B FROM T WHERE T.F1>OTER_FIELD) set of references to look into the subquery cache is (L1, L2, OTER_FIELD). The subquery cache should be implemented as simple LRU connected to the subquery. Size of the subquery cache (in number of results (but maybe in used memory amount)) is limited by session variable (query parameter?). HIGH-LEVEL SPECIFICATION: Attach subquery cache to each Item_subquery. Interface should allow to use hash or temporary table inside. To check/discuss: ----------------- * Will there be any means to measure subquery cache hit rate? * MySQL-6.0 has a one-element predicate result cache. It is called "left expression cache", grep for left_expr_cache in sql/item_subselect.* When this WL is merged with 6.0's optimizations, these two caches will need to be unified somehow. <contents> 1. Scope of the task 2. Data structure used for the cache 3. Cache size 4. Interplay with other subquery optimizations 5. User interface </contents> 1. Scope of the task -------------------- This WL should handle all subquery predicates, i.e. it should handle these cases: outer_expr IN (SELECT correlated_select) outer_expr $CMP$ ALL/ANY (SELECT correlated_select) EXISTS (SELECT correlated_select) scalar-context subquery: (SELECT correlated_select) The cache will maintain (outer_expr, correlation_references)-> subquery_item_result mapping, where - correlation_references is a list of tablename.column_name that are referred from the correlated_select but tablename is a table that is ouside the subquery. - subquery_item_result is 'bool' for subquery predicates, and is of some scalar or ROW(scalar1,...scalarN) type for scalar-context subquery. We don't support cases when outer_expr or correlation_references are blobs. All subquery predicates are cached. That is, if one subquery predicate is located within another, both of them will have caches (one option to reduce cache memory usage was to use cache only for the upper-most select. we decided against it). 2. Data structure used for the cache ------------------------------------ There are two data structures available in the codebase that will allow fast equality lookups: 1. HASH (mysys/hash.c) tables 2. Temporary tables (the ones that are used for e.g. GROUP BY) None of them has any support for element eviction on overflow (using LRU or some other policy). Query cache and MyISAM/Maria's key/page cache ought to support some eviction mechanism, but code-wise it is not readily reusable, one will need to factor it out (or copy it). We choose to use #2, and not to have any eviction policy. See subsequent sections for details and reasoning behind the decision. 3. Cache size ------------- Typically, a cache has some maximum size and a policy which is used to select a cache entry for removal when the cache becomes full (e.g. find and remove the least [recently] used entry) For this WL entry we will use a cache of infinite size. The reasoning behind this is that: - is is easy to do: we have temporary tables that can grow to arbitrarily large size while still providing the same insert/lookup interface. - it suits us: unless the subquery is resolved with one index lookup, hitting the cache would be many times cheaper than re-running the subquery, so cache is worth having. 4. Interplay with other subquery optimizations ---------------------------------------------- * This WL entry should not care about IN->EXISTS transformation: caching for IN subquery and result of its conversion to EXISTS would work in the same way. * This optimization is orthogonal to <=>ANY -> MIN/MAX rewrite (it will work/be useful irrespectively of whether the rewrite has been performed or not) * TODO: compare this with materialization for uncorrelated IN-subqueries. Is this basically the same? A: no, it is not: - IN-Materialization has to perform full materialization before it can do the first subquery evaluation. This WL's code has almost no startup costs. - This optimization has temp.table of (corr_reference, predicate_value), while IN-materialization will have (corr_reference) only. 5. User interface ----------------- * There will be an @@optimizer_switch flag to turn this optimization on and off (TODO: name of the flag?) * TODO: how do we show this in EXPLAIN [EXTENDED]? The most easiest is to print something in the warning text of EXPLAIN EXTEDED that would indicate use of cache. * temporary table sizing (max size for heap table, whether to use MyISAM or Maria) will be controlled with common temp.table control variables. LOW-LEVEL DESIGN: * Target version: base on mysql-5.2 code All items on which subquery depend could be collected in st_select_lex::mark_as_dependent (direct of indirect reference?) Temporary table index should be created by all fields except result field (TMP_TABLE_PARAM::keyinfo). How to fill the temptable ------------------------- Can reuse approach from SJ-Materialization. Its code is in end_sj_materialize() and is supposed to be quite trivial. How to make lookups into temptable ---------------------------------- We'll reuse approach used by SJ-Materialization in 6.0. Setup process ~~~~~~~~~~~~~ Setup is performed in the same way as in setup_sj_materialization(), see the code that starts these lines: /* Create/initialize everything we will need to index lookups into the temptable. */ and ends at this line: Remove the injected semi-join IN-equalities from join_tab conds. This <questionable> We'll also need to check equalities, i.e. do an equivalent of this: if (!(sjm->in_equality= create_subq_in_equalities(thd, sjm, emb_sj_nest->sj_subq_pred))) DBUG_RETURN(TRUE); /* purecov: inspected */ Question: or perhaps that is not necessarry? </questionable> Doing the lookup ~~~~~~~~~~~~~~~~ SJ-Materialization does lookup in sub_select_sjm(), with this code: /* Do index lookup in the materialized table */ if ((res= join_read_key2(join_tab, sjm->table, sjm->tab_ref)) == 1) DBUG_RETURN(NESTED_LOOP_ERROR); /* purecov: inspected */ if (res || !sjm->in_equality->val_int()) DBUG_RETURN(NESTED_LOOP_NO_MORE_ROWS); The code in this WL will use the same approach Extracting the value of the subquery predicate ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The goal of making the lookup is to get the value of subquery predicate. This is done by creating an Item_field $I which refers to appropriate temporary table's field and then subquery_predicate->val_int() will invoke $I->val_int(), subquery_predicate->val_str() will invoke $I->val_str() and so forth. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Monty): Microseconds (10)
by worklog-noreply＠askmonty.org 29 Jan '10

29 Jan '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Microseconds CREATION DATE..: Thu, 26 Mar 2009, 00:29 SUPERVISOR.....: Monty IMPLEMENTOR....: Bothorsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 10 (http://askmonty.org/worklog/?tid=10) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Monty - Fri, 29 Jan 2010, 19:05)=-=- Version updated. --- /tmp/wklog.10.old.5698 2010-01-29 19:05:42.000000000 +0200 +++ /tmp/wklog.10.new.5698 2010-01-29 19:05:42.000000000 +0200 @@ -1 +1 @@ -Server-5.2 +Server-5.3 DESCRIPTION: Add microsecond precision to NOW() Add new field types for time and datetime with microprecision ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Monty): Microseconds (10)
by worklog-noreply＠askmonty.org 29 Jan '10

29 Jan '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Microseconds CREATION DATE..: Thu, 26 Mar 2009, 00:29 SUPERVISOR.....: Monty IMPLEMENTOR....: Bothorsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 10 (http://askmonty.org/worklog/?tid=10) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: -=-=(Monty - Fri, 29 Jan 2010, 19:05)=-=- Version updated. --- /tmp/wklog.10.old.5698 2010-01-29 19:05:42.000000000 +0200 +++ /tmp/wklog.10.new.5698 2010-01-29 19:05:42.000000000 +0200 @@ -1 +1 @@ -Server-5.2 +Server-5.3 DESCRIPTION: Add microsecond precision to NOW() Add new field types for time and datetime with microprecision ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2807)
by Michael Widenius 29 Jan '10

29 Jan '10

#At lp:maria based on revid:knielsen@knielsen-hq.org-20100128223422-t6apa3py61dgbjjw 2807 Michael Widenius 2010-01-29 [merge] Auto merge modified: client/mysqltest.cc configure.in extra/yassl/src/yassl_error.cpp include/violite.h mysql-test/mysql-test-run.pl mysql-test/r/grant.result mysql-test/r/openssl_1.result mysql-test/std_data/cacert.pem mysql-test/std_data/client-cert.pem mysql-test/std_data/client-key.pem mysql-test/std_data/server-cert.pem mysql-test/std_data/server-key.pem mysql-test/t/grant.test mysql-test/t/openssl_1.test mysql-test/t/partition_innodb_semi_consistent.test mysql-test/t/query_cache_debug.test sql-common/client.c sql/net_serv.cc sql/slave.cc sql/sql_cache.cc sql/sql_connect.cc sql/sql_parse.cc storage/myisam/ft_boolean_search.c storage/myisam/ft_parser.c storage/myisam/ft_stopwords.c support-files/compiler_warnings.supp unittest/mysys/waiting_threads-t.c vio/viossl.c === modified file 'client/mysqltest.cc' --- a/client/mysqltest.cc 2010-01-15 15:27:55 +0000 +++ b/client/mysqltest.cc 2010-01-28 14:49:14 +0000 @@ -657,7 +657,9 @@ public: } while ((bytes= fread(buf, 1, sizeof(buf), m_file)) > 0) - fwrite(buf, 1, bytes, stderr); + if (fwrite(buf, 1, bytes, stderr)) + die("Failed to write to '%s', errno: %d", + m_file_name, errno); if (!lines) { === modified file 'configure.in' --- a/configure.in 2010-01-17 17:22:46 +0000 +++ b/configure.in 2010-01-29 10:42:31 +0000 @@ -10,7 +10,7 @@ AC_CANONICAL_SYSTEM # # When changing major version number please also check switch statement # in mysqlbinlog::check_master_version(). -AM_INIT_AUTOMAKE(mysql, 5.1.42-MariaDB-rc) +AM_INIT_AUTOMAKE(mysql, 5.1.42-MariaDB) AM_CONFIG_HEADER([include/config.h:config.h.in]) # Request support for automake silent-rules if available. === modified file 'extra/yassl/src/yassl_error.cpp' --- a/extra/yassl/src/yassl_error.cpp 2008-11-18 16:45:44 +0000 +++ b/extra/yassl/src/yassl_error.cpp 2010-01-29 10:42:31 +0000 @@ -60,7 +60,7 @@ void SetErrorString(YasslError error, ch using namespace TaoCrypt; const int max = MAX_ERROR_SZ; // shorthand - switch (error) { + switch ((int) error) { // yaSSL proper errors case range_error : === modified file 'include/violite.h' --- a/include/violite.h 2010-01-15 15:27:55 +0000 +++ b/include/violite.h 2010-01-29 10:42:31 +0000 @@ -123,8 +123,8 @@ struct st_VioSSLFd SSL_CTX *ssl_context; }; -int sslaccept(struct st_VioSSLFd*, Vio *, long timeout); -int sslconnect(struct st_VioSSLFd*, Vio *, long timeout); +int sslaccept(struct st_VioSSLFd*, Vio *, long timeout, char *error_string); +int sslconnect(struct st_VioSSLFd*, Vio *, long timeout, char *error_string); struct st_VioSSLFd *new_VioSSLConnectorFd(const char *key_file, const char *cert_file, === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2010-01-27 17:41:05 +0000 +++ b/mysql-test/mysql-test-run.pl 2010-01-29 10:42:31 +0000 @@ -413,7 +413,6 @@ sub main { # Not all tests completed, failure mtr_report(); mtr_report("Only ", int(@$completed), " of $num_tests completed."); - mtr_error("Not all tests completed"); } mtr_print_line(); @@ -425,6 +424,10 @@ sub main { mtr_report_stats($fail, $completed, $extra_warnings); + if ( @$completed != $num_tests) + { + mtr_error("Not all tests completed"); + } exit(0); } === modified file 'mysql-test/r/grant.result' --- a/mysql-test/r/grant.result 2009-10-27 10:09:36 +0000 +++ b/mysql-test/r/grant.result 2010-01-29 10:42:31 +0000 @@ -29,15 +29,15 @@ show grants for mysqltest_1@localhost; Grants for mysqltest_1@localhost GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' GRANT SELECT ON `mysqltest`.* TO 'mysqltest_1'@'localhost' -grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "MySQL AB"; +grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "Monty Program Ab"; show grants for mysqltest_1@localhost; Grants for mysqltest_1@localhost -GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'MySQL AB' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' +GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'Monty Program Ab' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' GRANT SELECT ON `mysqltest`.* TO 'mysqltest_1'@'localhost' revoke all privileges on mysqltest.* from mysqltest_1@localhost; show grants for mysqltest_1@localhost; Grants for mysqltest_1@localhost -GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'MySQL AB' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' +GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'Monty Program Ab' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' delete from mysql.user where user='mysqltest_1'; flush privileges; delete from mysql.user where user='mysqltest_1'; === modified file 'mysql-test/r/openssl_1.result' --- a/mysql-test/r/openssl_1.result 2009-05-25 13:00:18 +0000 +++ b/mysql-test/r/openssl_1.result 2010-01-29 10:42:31 +0000 @@ -3,8 +3,10 @@ create table t1(f1 int); insert into t1 values (5); grant select on test.* to ssl_user1@localhost require SSL; grant select on test.* to ssl_user2@localhost require cipher "DHE-RSA-AES256-SHA"; -grant select on test.* to ssl_user3@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com"; -grant select on test.* to ssl_user4@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com" ISSUER "/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB"; +grant select on test.* to ssl_user3@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; +grant select on test.* to ssl_user4@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org" ISSUER "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; grant select on test.* to ssl_user5@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "xxx"; flush privileges; connect(localhost,ssl_user5,,test,MASTER_PORT,MASTER_SOCKET); @@ -44,9 +46,9 @@ ERROR 42000: DELETE command denied to us drop user ssl_user1@localhost, ssl_user2@localhost, ssl_user3@localhost, ssl_user4@localhost, ssl_user5@localhost; drop table t1; -mysqltest: Could not open connection 'default': 2026 SSL connection error -mysqltest: Could not open connection 'default': 2026 SSL connection error -mysqltest: Could not open connection 'default': 2026 SSL connection error +mysqltest: Could not open connection 'default': 2026 SSL error: ASN: bad other signature confirmation +mysqltest: Could not open connection 'default': 2026 SSL error: ASN: bad other signature confirmation +mysqltest: Could not open connection 'default': 2026 SSL error: ASN: bad other signature confirmation SSL error: Unable to get private key from '' mysqltest: Could not open connection 'default': 2026 SSL connection error SSL error: Unable to get certificate from '' === modified file 'mysql-test/std_data/cacert.pem' --- a/mysql-test/std_data/cacert.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/cacert.pem 2010-01-29 10:42:31 +0000 @@ -1,17 +1,19 @@ -----BEGIN CERTIFICATE----- -MIICrTCCAhagAwIBAgIJAJXpePU0UOTVMA0GCSqGSIb3DQEBBQUAMEQxCzAJBgNV -BAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxhMRAwDgYDVQQHEwdVcHBzYWxhMREwDwYD -VQQKEwhNeVNRTCBBQjAeFw0wOTAxMjgxMDQ5NDZaFw0xNDAxMjcxMDQ5NDZaMEQx -CzAJBgNVBAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxhMRAwDgYDVQQHEwdVcHBzYWxh -MREwDwYDVQQKEwhNeVNRTCBBQjCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA -4XQHAe5R1+TXC8noZtWf+d5E0v1C59FWpn9SWEUCBjE5UiIwuJvi4Y+7xWGOXLAI -/JzJx5gNXLBiTsE/zh0uX9fKlajLhxB0GN+QU0ZlpQ1BeYipEcNXeI/7cT499f6v -XWabnTflivdCgHSWUOQ20/Lzs6kP6/e6OoZd/DPSjPECAwEAAaOBpjCBozAdBgNV -HQ4EFgQU8uLqVWWkmuKsnZf1RWz294wRrd8wdAYDVR0jBG0wa4AU8uLqVWWkmuKs -nZf1RWz294wRrd+hSKRGMEQxCzAJBgNVBAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxh -MRAwDgYDVQQHEwdVcHBzYWxhMREwDwYDVQQKEwhNeVNRTCBBQoIJAJXpePU0UOTV -MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADgYEAMMTE5sDN+Z0ZlV7KvH3g -6+aKvql8dTpRT3hYukeQlWua0nq74WPGVw0c4e/M/vbiMwmJcCYpB9pd4+dHqzSw -aPyoenjY6UF8n7B4quWy3SIUk2LSHeJLW+kzJn2afN9gvipFhdVh/uU2TIyLGOur -Z/vmJX2W7hF1uqPnbfa8Lrw= +MIIDIjCCAougAwIBAgIJAJhuvLP+2mGwMA0GCSqGSIb3DQEBBQUAMGoxCzAJBgNV +BAYTAkZJMRAwDgYDVQQIEwdUdXVzdWxhMRkwFwYDVQQKExBNb250eSBQcm9ncmFt +IEFiMS4wLAYJKoZIhvcNAQkBFh9hYnN0cmFjdC5kZXZlbG9wZXJAYXNrbW9udHku +b3JnMB4XDTEwMDEyODIxNTcyNVoXDTEwMDIyNzIxNTcyNVowajELMAkGA1UEBhMC +RkkxEDAOBgNVBAgTB1R1dXN1bGExGTAXBgNVBAoTEE1vbnR5IFByb2dyYW0gQWIx +LjAsBgkqhkiG9w0BCQEWH2Fic3RyYWN0LmRldmVsb3BlckBhc2ttb250eS5vcmcw +gZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMIaY4pwbst50S32xJH3bSXhPGep +6gx1AWwZKsHTXL3VeMO6PHmC8zu5HM0zbOcrIJcXL3YVnpmE4b9OQxIiMSx1Yd+U +u8/sTkxgpsEKhCbIzECIwPhppyT/JP5aSXCadEvg+PSjikv8dOVkD68wVG4CcFIX +MFttsPebBVzEokZZAgMBAAGjgc8wgcwwHQYDVR0OBBYEFOCKaNHFFPrju8AwzWxS +f96IKfRwMIGcBgNVHSMEgZQwgZGAFOCKaNHFFPrju8AwzWxSf96IKfRwoW6kbDBq +MQswCQYDVQQGEwJGSTEQMA4GA1UECBMHVHV1c3VsYTEZMBcGA1UEChMQTW9udHkg +UHJvZ3JhbSBBYjEuMCwGCSqGSIb3DQEJARYfYWJzdHJhY3QuZGV2ZWxvcGVyQGFz +a21vbnR5Lm9yZ4IJAJhuvLP+2mGwMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEF +BQADgYEAsmCX2/k9AInq2qhXtnkLip6cB0iOerLTNAzEijZc/aVf4wUjkL3cqhmC +kSTCwAHIOxp+ICwh6ky3xghXjoI9QnPFDVkRkzPT2tV0IoBaeQuI4e0CU2EY7L3P +XoDqp3oq1XtVcr9ZZdP68fBYUG/qcrWcXWk45ZFaBmBv3TotsGk= -----END CERTIFICATE----- === modified file 'mysql-test/std_data/client-cert.pem' --- a/mysql-test/std_data/client-cert.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/client-cert.pem 2010-01-29 10:42:31 +0000 @@ -1,55 +1,60 @@ Certificate: Data: Version: 3 (0x2) - Serial Number: 3 (0x3) + Serial Number: 2 (0x2) Signature Algorithm: sha1WithRSAEncryption - Issuer: C=SE, ST=Uppsala, L=Uppsala, O=MySQL AB + Issuer: C=FI, ST=Tuusula, O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org Validity - Not Before: Jan 28 11:04:39 2009 GMT - Not After : Jan 28 11:04:39 2010 GMT - Subject: C=SE, ST=Uppsala, O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com + Not Before: Jan 28 22:01:38 2010 GMT + Not After : Dec 7 22:01:38 2019 GMT + Subject: C=FI, ST=Tuusula, O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org Subject Public Key Info: Public Key Algorithm: rsaEncryption - RSA Public Key: (512 bit) - Modulus (512 bit): - 00:e1:52:30:2c:d9:be:64:28:91:5d:7a:fd:d9:e9: - 14:35:7a:d2:94:4e:91:46:e0:db:9f:6b:79:f4:4c: - ac:6e:07:61:34:86:74:62:a7:a8:44:af:fa:87:87: - a8:7d:42:61:ff:ab:50:d4:7b:bf:75:fa:d5:d5:b3: - 74:fb:56:1e:37 + RSA Public Key: (1024 bit) + Modulus (1024 bit): + 00:bd:dc:3d:f8:3c:0b:d4:d2:c0:a3:9c:34:2d:e7: + 11:ff:4d:43:35:17:f6:0d:91:01:92:9e:4f:4d:c0: + 38:d5:62:03:55:33:db:66:a2:91:d3:f2:b6:23:34: + 95:53:50:3d:4f:e3:0c:d7:76:fd:f6:54:64:e6:f6: + dc:70:74:7c:6b:74:41:59:b0:19:5d:62:90:3c:a7: + c8:5e:21:8f:2b:22:6b:c7:43:9b:be:79:84:60:da: + 16:c9:ce:ee:fd:66:cb:54:81:e2:b5:1c:cf:f9:74: + de:38:2b:28:d4:31:33:55:d2:30:1c:a3:e4:c2:c7: + 31:46:43:d5:33:3d:8a:0b:47 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE + Netscape Comment: + OpenSSL Generated Certificate X509v3 Subject Key Identifier: - 58:30:B5:9B:2C:05:94:06:BA:3D:3C:F0:B2:CD:1D:67:65:E3:7F:85 + BE:E6:DB:19:8D:DB:72:9A:85:EE:B2:B8:5D:E7:FF:61:DF:09:08:AF X509v3 Authority Key Identifier: - keyid:F2:E2:EA:55:65:A4:9A:E2:AC:9D:97:F5:45:6C:F6:F7:8C:11:AD:DF - DirName:/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB - serial:95:E9:78:F5:34:50:E4:D5 + keyid:E0:8A:68:D1:C5:14:FA:E3:BB:C0:30:CD:6C:52:7F:DE:88:29:F4:70 Signature Algorithm: sha1WithRSAEncryption - 05:19:e3:13:14:fc:c5:28:bf:69:f8:00:b3:25:cb:bd:ca:9f: - 2f:4c:b3:a8:04:11:f0:74:27:bd:82:2c:b4:49:9b:a7:59:f0: - f7:87:d1:e0:ba:99:a2:fe:4b:1d:10:6f:e4:a2:b3:cd:7f:8b: - 68:31:46:ee:cd:9e:e2:47:e1:4c:fa:74:d1:e2:8b:cc:a0:4b: - a8:24:d1:a4:c3:6b:2a:c6:28:cd:41:e0:06:48:e6:cf:f2:3c: - ca:37:95:d7:29:64:6b:91:91:83:e7:ac:c8:0b:87:bc:da:a6: - aa:f1:44:43:c8:74:7b:15:26:91:2e:03:c4:71:50:6c:f8:68: - dc:8c + 41:95:6d:0a:a4:ee:af:68:cd:94:26:59:9a:18:b7:75:3c:c5: + 0f:22:d3:5c:31:9b:85:a0:93:b3:f0:50:29:ba:1e:d3:5a:43: + 0b:77:2d:98:87:a7:a7:39:0f:40:8d:03:d3:b3:67:43:77:bc: + 3c:51:c2:f9:9e:7a:2d:39:c4:5c:16:d7:70:d6:74:d1:6c:e1: + 6a:4d:fd:1f:10:af:64:3b:f4:64:e9:b2:b3:fb:c8:cd:c5:41: + cd:99:e0:ac:83:1d:81:2c:6b:99:ba:80:02:12:72:f7:3b:bb: + 93:72:00:da:ff:d3:87:75:d2:3a:a4:ca:4d:c1:8b:c1:21:50: + cb:57 -----BEGIN CERTIFICATE----- -MIICfzCCAeigAwIBAgIBAzANBgkqhkiG9w0BAQUFADBEMQswCQYDVQQGEwJTRTEQ -MA4GA1UECBMHVXBwc2FsYTEQMA4GA1UEBxMHVXBwc2FsYTERMA8GA1UEChMITXlT -UUwgQUIwHhcNMDkwMTI4MTEwNDM5WhcNMTAwMTI4MTEwNDM5WjBlMQswCQYDVQQG -EwJTRTEQMA4GA1UECBMHVXBwc2FsYTERMA8GA1UEChMITXlTUUwgQUIxMTAvBgkq -hkiG9w0BCQEWImFic3RyYWN0Lm15c3FsLmRldmVsb3BlckBteXNxbC5jb20wXDAN -BgkqhkiG9w0BAQEFAANLADBIAkEA4VIwLNm+ZCiRXXr92ekUNXrSlE6RRuDbn2t5 -9EysbgdhNIZ0YqeoRK/6h4eofUJh/6tQ1Hu/dfrV1bN0+1YeNwIDAQABo4GjMIGg -MAkGA1UdEwQCMAAwHQYDVR0OBBYEFFgwtZssBZQGuj088LLNHWdl43+FMHQGA1Ud -IwRtMGuAFPLi6lVlpJrirJ2X9UVs9veMEa3foUikRjBEMQswCQYDVQQGEwJTRTEQ -MA4GA1UECBMHVXBwc2FsYTEQMA4GA1UEBxMHVXBwc2FsYTERMA8GA1UEChMITXlT -UUwgQUKCCQCV6Xj1NFDk1TANBgkqhkiG9w0BAQUFAAOBgQAFGeMTFPzFKL9p+ACz -Jcu9yp8vTLOoBBHwdCe9giy0SZunWfD3h9Hgupmi/ksdEG/korPNf4toMUbuzZ7i -R+FM+nTR4ovMoEuoJNGkw2sqxijNQeAGSObP8jzKN5XXKWRrkZGD56zIC4e82qaq -8URDyHR7FSaRLgPEcVBs+GjcjA== +MIICxTCCAi6gAwIBAgIBAjANBgkqhkiG9w0BAQUFADBqMQswCQYDVQQGEwJGSTEQ +MA4GA1UECBMHVHV1c3VsYTEZMBcGA1UEChMQTW9udHkgUHJvZ3JhbSBBYjEuMCwG +CSqGSIb3DQEJARYfYWJzdHJhY3QuZGV2ZWxvcGVyQGFza21vbnR5Lm9yZzAeFw0x +MDAxMjgyMjAxMzhaFw0xOTEyMDcyMjAxMzhaMGoxCzAJBgNVBAYTAkZJMRAwDgYD +VQQIEwdUdXVzdWxhMRkwFwYDVQQKExBNb250eSBQcm9ncmFtIEFiMS4wLAYJKoZI +hvcNAQkBFh9hYnN0cmFjdC5kZXZlbG9wZXJAYXNrbW9udHkub3JnMIGfMA0GCSqG +SIb3DQEBAQUAA4GNADCBiQKBgQC93D34PAvU0sCjnDQt5xH/TUM1F/YNkQGSnk9N +wDjVYgNVM9tmopHT8rYjNJVTUD1P4wzXdv32VGTm9txwdHxrdEFZsBldYpA8p8he +IY8rImvHQ5u+eYRg2hbJzu79ZstUgeK1HM/5dN44KyjUMTNV0jAco+TCxzFGQ9Uz +PYoLRwIDAQABo3sweTAJBgNVHRMEAjAAMCwGCWCGSAGG+EIBDQQfFh1PcGVuU1NM +IEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUvubbGY3bcpqF7rK4Xef/ +Yd8JCK8wHwYDVR0jBBgwFoAU4Ipo0cUU+uO7wDDNbFJ/3ogp9HAwDQYJKoZIhvcN +AQEFBQADgYEAQZVtCqTur2jNlCZZmhi3dTzFDyLTXDGbhaCTs/BQKboe01pDC3ct +mIenpzkPQI0D07NnQ3e8PFHC+Z56LTnEXBbXcNZ00Wzhak39HxCvZDv0ZOmys/vI +zcVBzZngrIMdgSxrmbqAAhJy9zu7k3IA2v/Th3XSOqTKTcGLwSFQy1c= -----END CERTIFICATE----- === modified file 'mysql-test/std_data/client-key.pem' --- a/mysql-test/std_data/client-key.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/client-key.pem 2010-01-29 10:42:31 +0000 @@ -1,9 +1,15 @@ -----BEGIN RSA PRIVATE KEY----- -MIIBOQIBAAJBAOFSMCzZvmQokV16/dnpFDV60pROkUbg259refRMrG4HYTSGdGKn -qESv+oeHqH1CYf+rUNR7v3X61dWzdPtWHjcCAwEAAQJAXYooM8ZlcuEgj+VKU1ee -qyEFIMqJJxqcMk+E/nWCM96WxCP3zHNSrqNfSpI3ld7QzMwhdRz+gFLxT2gGNpIw -MQIhAPxzM/lDihe67X3ADYtDl9ZjA8Pm430x9sXlcxI17tCZAiEA5H1SyFl4mUee -9VnfSC2XGW7lwz72ZygfVX+b7tLWF08CIEh40gzW5MfXM+KLxdea+fXjyursV5ZT -R6KcMiKiNQLRAiAcmHqlzFzFgisotai2Fc6VRkXHG7gmzOSvBJt1VjmpDQIge6jf -2N7whTdvC4ferB+zUlgWQdyvx1c3T4gnt6PYdaY= +MIICXgIBAAKBgQC93D34PAvU0sCjnDQt5xH/TUM1F/YNkQGSnk9NwDjVYgNVM9tm +opHT8rYjNJVTUD1P4wzXdv32VGTm9txwdHxrdEFZsBldYpA8p8heIY8rImvHQ5u+ +eYRg2hbJzu79ZstUgeK1HM/5dN44KyjUMTNV0jAco+TCxzFGQ9UzPYoLRwIDAQAB +AoGBAJa2lprPT7UJ99Ho1aL6ota/RnKHKtNqII17DgjyZis9OtgP6kJ3GrvdF6iq +vT79my4nVrJTyxYXuGF/5U1/qqNjuPPBE1Xbu1ubQlFv8CT0kKYynQ7Z3ls8fAHC +B3VJXnUVlG+GHtUEFFG4FQVX1fn/Sga67ioJ6ivAiBlHKaPBAkEA5f2ToWlj4u9O +KgfRkN54wdIp4yu2c40pbhMfKGjGGsBAHk92+qSBpzEmxLcI6Ay+4/QysSR4jYmK +jCJuxiTu1QJBANNU3Hx8Il2SF/2BqGLcIh2SHxzKQIT5wAyD2jb+P2cHvbk6pKGR +VTmw5bibxXmYMS6J/L2zUF2xtFe+Svwz96sCQEnKYSqBqOWvyBFeLtPfPTlal8vm +Q4SxfuBtTCrn6t+8XRYcgt0KGPsunvSwkS/6nuh+eiExxWgMACLUDVyPjv0CQQC4 +sJJc7LOv6Oy0bWr2swHRrBEqvQsz63zOszCzHPHWHirNxZV5aiT8XT/2XZRwlvRs +gsVyGFLk/1fn0vN/g/8vAkEAxUdzUKvC1ZwjzGhgcz2bQU0tEZN4C9jBCiwOI2ud +BpAsPG0xAGGL2+hz0B0n88XiTHobiTZ1bg4Z41i4pXx2ZA== -----END RSA PRIVATE KEY----- === modified file 'mysql-test/std_data/server-cert.pem' --- a/mysql-test/std_data/server-cert.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/server-cert.pem 2010-01-29 10:42:31 +0000 @@ -3,53 +3,59 @@ Certificate: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha1WithRSAEncryption - Issuer: C=SE, ST=Uppsala, L=Uppsala, O=MySQL AB + Issuer: C=FI, ST=Tuusula, O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org Validity - Not Before: Jan 28 10:55:13 2009 GMT - Not After : Jan 28 10:55:13 2010 GMT - Subject: C=SE, ST=Uppsala, O=MySQL AB, CN=localhost/emailAddress=abstract.mysql.developer(a)mysql.com + Not Before: Jan 28 21:59:14 2010 GMT + Not After : Dec 7 21:59:14 2019 GMT + Subject: C=FI, ST=Tuusula, O=Monty Program Ab, CN=localhost/emailAddress=abstract.developer(a)askmonty.org Subject Public Key Info: Public Key Algorithm: rsaEncryption - RSA Public Key: (512 bit) - Modulus (512 bit): - 00:b6:8f:e5:b7:b4:86:83:13:8a:f9:bf:63:cb:64: - 2d:b9:51:d1:de:ab:7b:45:1f:aa:b5:66:73:13:f9: - a6:07:d5:ba:7c:fa:92:bd:37:e2:ad:87:db:3e:b6: - 6a:12:64:f8:ee:17:e3:15:06:2f:a8:82:68:bf:57: - 8d:c3:04:98:27 + RSA Public Key: (1024 bit) + Modulus (1024 bit): + 00:cc:79:74:2a:14:7e:77:06:b0:ec:1b:b6:da:70: + 4c:4f:0e:94:04:8a:e7:69:f4:4c:9c:67:22:41:56: + 3a:96:53:9e:95:9d:17:55:80:41:c0:13:d6:94:0f: + cd:2c:51:fe:a4:6e:f2:74:d8:5d:3b:3a:80:e3:85: + 5b:a5:bc:7d:5c:25:55:e5:40:77:fe:f3:cb:5b:cd: + da:a5:f4:36:64:30:a2:a0:7f:93:b3:c4:56:75:2a: + c0:f7:65:2a:d7:e6:ff:88:25:03:e0:b4:40:2e:74: + 4c:cb:46:95:73:cb:25:5c:87:0e:ec:0f:5a:19:c2: + b6:dc:9e:e8:f0:30:b1:9c:99 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE + Netscape Comment: + OpenSSL Generated Certificate X509v3 Subject Key Identifier: - D9:9A:B8:5F:22:EA:04:10:C8:25:7D:82:57:E6:2E:FD:19:29:E7:DA + 6D:13:3B:40:52:3C:AF:18:EA:33:D1:B7:56:21:1B:05:FE:0B:9E:38 X509v3 Authority Key Identifier: - keyid:F2:E2:EA:55:65:A4:9A:E2:AC:9D:97:F5:45:6C:F6:F7:8C:11:AD:DF - DirName:/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB - serial:95:E9:78:F5:34:50:E4:D5 + keyid:E0:8A:68:D1:C5:14:FA:E3:BB:C0:30:CD:6C:52:7F:DE:88:29:F4:70 Signature Algorithm: sha1WithRSAEncryption - 54:07:2d:21:0b:a5:af:3b:58:23:32:5e:56:7f:ab:58:63:48: - 91:aa:38:90:89:16:f9:cc:bf:a4:0e:78:2b:9f:c5:1b:58:a6: - e6:08:8f:2e:ae:97:03:21:9b:f1:cd:c0:26:8f:1d:d7:28:27: - a0:8e:81:09:1b:1c:0f:c9:a5:41:3a:2d:44:3f:9c:fa:87:ff: - c8:4c:2b:44:f7:1b:c1:3e:4f:01:7f:e9:26:cc:9f:1c:06:b5: - 0b:27:d1:10:90:be:93:0c:9c:e7:b0:d1:ea:27:99:4e:06:14: - 0c:7a:e9:c1:52:c5:33:68:bc:61:0d:db:81:3b:57:48:57:bf: - 42:9a + 97:db:65:23:7f:f1:15:3c:1e:83:ac:0e:0a:50:a0:0c:22:b8: + 45:d4:ca:21:05:47:3b:3d:03:b5:6c:4b:8d:bb:5f:57:c3:c7: + 4e:71:23:cf:33:a3:7f:a0:3d:bd:58:75:b8:37:22:16:2f:e9: + ed:ae:9b:94:29:81:6e:34:79:cf:41:bd:3d:8d:17:d7:22:1c: + 1b:58:c7:0f:79:13:56:1d:e8:d8:4e:e5:07:3f:79:1b:dd:c4: + 06:9b:c5:b6:02:34:43:c5:bf:e5:87:ad:f1:c1:8a:f2:be:c2: + 00:1d:d4:27:1f:87:c8:80:31:ec:6e:97:95:b4:84:40:d1:73: + 42:71 -----BEGIN CERTIFICATE----- -MIICkzCCAfygAwIBAgIBATANBgkqhkiG9w0BAQUFADBEMQswCQYDVQQGEwJTRTEQ -MA4GA1UECBMHVXBwc2FsYTEQMA4GA1UEBxMHVXBwc2FsYTERMA8GA1UEChMITXlT -UUwgQUIwHhcNMDkwMTI4MTA1NTEzWhcNMTAwMTI4MTA1NTEzWjB5MQswCQYDVQQG -EwJTRTEQMA4GA1UECBMHVXBwc2FsYTERMA8GA1UEChMITXlTUUwgQUIxEjAQBgNV -BAMTCWxvY2FsaG9zdDExMC8GCSqGSIb3DQEJARYiYWJzdHJhY3QubXlzcWwuZGV2 -ZWxvcGVyQG15c3FsLmNvbTBcMA0GCSqGSIb3DQEBAQUAA0sAMEgCQQC2j+W3tIaD -E4r5v2PLZC25UdHeq3tFH6q1ZnMT+aYH1bp8+pK9N+Kth9s+tmoSZPjuF+MVBi+o -gmi/V43DBJgnAgMBAAGjgaMwgaAwCQYDVR0TBAIwADAdBgNVHQ4EFgQU2Zq4XyLq -BBDIJX2CV+Yu/Rkp59owdAYDVR0jBG0wa4AU8uLqVWWkmuKsnZf1RWz294wRrd+h -SKRGMEQxCzAJBgNVBAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxhMRAwDgYDVQQHEwdV -cHBzYWxhMREwDwYDVQQKEwhNeVNRTCBBQoIJAJXpePU0UOTVMA0GCSqGSIb3DQEB -BQUAA4GBAFQHLSELpa87WCMyXlZ/q1hjSJGqOJCJFvnMv6QOeCufxRtYpuYIjy6u -lwMhm/HNwCaPHdcoJ6COgQkbHA/JpUE6LUQ/nPqH/8hMK0T3G8E+TwF/6SbMnxwG -tQsn0RCQvpMMnOew0eonmU4GFAx66cFSxTNovGEN24E7V0hXv0Ka +MIIC2TCCAkKgAwIBAgIBATANBgkqhkiG9w0BAQUFADBqMQswCQYDVQQGEwJGSTEQ +MA4GA1UECBMHVHV1c3VsYTEZMBcGA1UEChMQTW9udHkgUHJvZ3JhbSBBYjEuMCwG +CSqGSIb3DQEJARYfYWJzdHJhY3QuZGV2ZWxvcGVyQGFza21vbnR5Lm9yZzAeFw0x +MDAxMjgyMTU5MTRaFw0xOTEyMDcyMTU5MTRaMH4xCzAJBgNVBAYTAkZJMRAwDgYD +VQQIEwdUdXVzdWxhMRkwFwYDVQQKExBNb250eSBQcm9ncmFtIEFiMRIwEAYDVQQD +Ewlsb2NhbGhvc3QxLjAsBgkqhkiG9w0BCQEWH2Fic3RyYWN0LmRldmVsb3BlckBh +c2ttb250eS5vcmcwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMx5dCoUfncG +sOwbttpwTE8OlASK52n0TJxnIkFWOpZTnpWdF1WAQcAT1pQPzSxR/qRu8nTYXTs6 +gOOFW6W8fVwlVeVAd/7zy1vN2qX0NmQwoqB/k7PEVnUqwPdlKtfm/4glA+C0QC50 +TMtGlXPLJVyHDuwPWhnCttye6PAwsZyZAgMBAAGjezB5MAkGA1UdEwQCMAAwLAYJ +YIZIAYb4QgENBB8WHU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1Ud +DgQWBBRtEztAUjyvGOoz0bdWIRsF/gueODAfBgNVHSMEGDAWgBTgimjRxRT647vA +MM1sUn/eiCn0cDANBgkqhkiG9w0BAQUFAAOBgQCX22Ujf/EVPB6DrA4KUKAMIrhF +1MohBUc7PQO1bEuNu19Xw8dOcSPPM6N/oD29WHW4NyIWL+ntrpuUKYFuNHnPQb09 +jRfXIhwbWMcPeRNWHejYTuUHP3kb3cQGm8W2AjRDxb/lh63xwYryvsIAHdQnH4fI +gDHsbpeVtIRA0XNCcQ== -----END CERTIFICATE----- === modified file 'mysql-test/std_data/server-key.pem' --- a/mysql-test/std_data/server-key.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/server-key.pem 2010-01-29 10:42:31 +0000 @@ -1,9 +1,15 @@ -----BEGIN RSA PRIVATE KEY----- -MIIBOQIBAAJBALaP5be0hoMTivm/Y8tkLblR0d6re0UfqrVmcxP5pgfVunz6kr03 -4q2H2z62ahJk+O4X4xUGL6iCaL9XjcMEmCcCAwEAAQJASA5VwgNb0CKHiPm0ntOk -hG+54SRX3DmafEy6gRjZIl/bZ/asSLhXUZ+CeohyrQh7BZgYWvykd8pRISL9eKsU -GQIhAOXkUrOtP/EtjyqNluEqZdG+RZi/7p61JS3Ce13Myu+LAiEAy0uMlV34AJpM -b40FPKqlHxw8DD/Dt1iKhNVAg8+LDVUCIFjv7fbJDbW2VG63/Cj8CAwOukoP5rbL -iaicVrHBKrllAiB9+MiaXeopZXNrxDS0jQFYr8Q9yt1aJVFgUkxx4Q9HKQIgZPs0 -KlF3NNNWw78INaAEkyf0IEssnLMsuoCWw0DIOak= +MIICXgIBAAKBgQDMeXQqFH53BrDsG7bacExPDpQEiudp9EycZyJBVjqWU56VnRdV +gEHAE9aUD80sUf6kbvJ02F07OoDjhVulvH1cJVXlQHf+88tbzdql9DZkMKKgf5Oz +xFZ1KsD3ZSrX5v+IJQPgtEAudEzLRpVzyyVchw7sD1oZwrbcnujwMLGcmQIDAQAB +AoGBAMdMYkNZsmJFbVDVOobzCg3Mgc1jrmeBrOKNS8AvUe+QFXRyp3m5B102eOHb +/PmD+hU/5qao9UZzoYkiRM/oRq45jrqJEYwWrX007bKK0F9hnErtC1ImM1nBFVhx +6+6cr+ShUkvtj8+wJ2d5bIccUzGCUfFR5tb5BnePTXK8IVoBAkEA7WGNxHAVKgjS +AzlpHr5fvpivA07hNVJizTwZdWGGYeETilZhkkuMRwREceeohF6ILMf0FTZdFSa/ +8EeLa3icIQJBANyDKFjynKwWy5pyRSz75mVwrEi+4eTQPsCPNWLkbpbEPwqPLYWJ +2VSFkISXF7b7Od48JkQWgiB8/kXqMDEdsXkCQQCzZvj3ryWvoP7nhOoXXBWMPGR4 +gZLe86bMKVGsTsp7CtnzwRj4sbQQr/7yfvvzHmaYQX4M0gtDQwfolomd7YdBAkEA +y24ETuqjNu9grf81aiaJipPDnOjcJOcovSRgr/blPxmUvv0Pld5yLNN7W5a4PgrO +fAMpmi7ZpXcqbP17sBQgoQJAWTDFKAmfHPVdDGZuCw4yceP5d+Tv7ABglZUvpPKx +kAvGN1WBASUuCQJDOIgzl6gvYX07S5p147i9mv7UBWOpvw== -----END RSA PRIVATE KEY----- === modified file 'mysql-test/t/grant.test' --- a/mysql-test/t/grant.test 2009-06-15 15:53:45 +0000 +++ b/mysql-test/t/grant.test 2010-01-29 10:42:31 +0000 @@ -35,7 +35,7 @@ revoke delete on mysqltest.* from mysqlt show grants for mysqltest_1@localhost; grant select on mysqltest.* to mysqltest_1@localhost require NONE; show grants for mysqltest_1@localhost; -grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "MySQL AB"; +grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "Monty Program Ab"; show grants for mysqltest_1@localhost; revoke all privileges on mysqltest.* from mysqltest_1@localhost; show grants for mysqltest_1@localhost; === modified file 'mysql-test/t/openssl_1.test' --- a/mysql-test/t/openssl_1.test 2009-05-25 13:00:18 +0000 +++ b/mysql-test/t/openssl_1.test 2010-01-29 10:42:31 +0000 @@ -15,8 +15,10 @@ insert into t1 values (5); grant select on test.* to ssl_user1@localhost require SSL; grant select on test.* to ssl_user2@localhost require cipher "DHE-RSA-AES256-SHA"; -grant select on test.* to ssl_user3@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com"; -grant select on test.* to ssl_user4@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com" ISSUER "/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB"; +grant select on test.* to ssl_user3@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; +grant select on test.* to ssl_user4@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org" ISSUER "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; grant select on test.* to ssl_user5@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "xxx"; flush privileges; === modified file 'mysql-test/t/partition_innodb_semi_consistent.test' --- a/mysql-test/t/partition_innodb_semi_consistent.test 2009-01-13 22:12:16 +0000 +++ b/mysql-test/t/partition_innodb_semi_consistent.test 2010-01-28 14:49:14 +0000 @@ -187,6 +187,7 @@ SELECT * FROM t1; --echo # Switch to connection con2 connection con2; +--reap SELECT * FROM t1; connection default; === modified file 'mysql-test/t/query_cache_debug.test' --- a/mysql-test/t/query_cache_debug.test 2009-09-07 20:50:10 +0000 +++ b/mysql-test/t/query_cache_debug.test 2010-01-29 10:42:31 +0000 @@ -1,5 +1,4 @@ --source include/not_embedded.inc ---source include/big_test.inc --source include/have_query_cache.inc --source include/have_debug.inc === modified file 'sql-common/client.c' --- a/sql-common/client.c 2009-12-03 15:26:54 +0000 +++ b/sql-common/client.c 2010-01-29 10:42:31 +0000 @@ -1863,6 +1863,7 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons uint port, const char *unix_socket,ulong client_flag) { char buff[NAME_LEN+USERNAME_LENGTH+100]; + char error_string[1024]; char *end,*host_info= NULL; my_socket sock; in_addr_t ip_addr; @@ -2331,9 +2332,14 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons /* Connect to the server */ DBUG_PRINT("info", ("IO layer change in progress...")); if (sslconnect(ssl_fd, mysql->net.vio, - (long) (mysql->options.connect_timeout))) + (long) (mysql->options.connect_timeout), + error_string)) { - set_mysql_error(mysql, CR_SSL_CONNECTION_ERROR, unknown_sqlstate); + set_mysql_extended_error(mysql, CR_SSL_CONNECTION_ERROR, + unknown_sqlstate, + "SSL error: %s", + error_string[0] ? error_string : + ER(CR_SSL_CONNECTION_ERROR)); goto error; } DBUG_PRINT("info", ("IO layer change done!")); === modified file 'sql/net_serv.cc' --- a/sql/net_serv.cc 2009-09-07 20:50:10 +0000 +++ b/sql/net_serv.cc 2010-01-29 10:42:31 +0000 @@ -219,7 +219,7 @@ my_bool net_realloc(NET *net, size_t len -1 Don't know if data is ready or not */ -#if !defined(EMBEDDED_LIBRARY) +#if !defined(EMBEDDED_LIBRARY) && defined(DBUG_OFF) static int net_data_is_ready(my_socket sd) { @@ -280,13 +280,18 @@ static int net_data_is_ready(my_socket s void net_clear(NET *net, my_bool clear_buffer) { -#if !defined(EMBEDDED_LIBRARY) +#if !defined(EMBEDDED_LIBRARY) && defined(DBUG_OFF) size_t count; int ready; #endif DBUG_ENTER("net_clear"); -#if !defined(EMBEDDED_LIBRARY) +/* + We don't do a clear in case of DBUG_OFF to catch bugs + in the protocol handling +*/ + +#if !defined(EMBEDDED_LIBRARY) && defined(DBUG_OFF) if (clear_buffer) { while ((ready= net_data_is_ready(net->vio->sd)) > 0) === modified file 'sql/slave.cc' --- a/sql/slave.cc 2009-12-06 17:51:48 +0000 +++ b/sql/slave.cc 2010-01-29 10:42:31 +0000 @@ -3803,10 +3803,11 @@ static int connect_to_master(THD* thd, M suppress_warnings= 0; mi->report(ERROR_LEVEL, last_errno, "error %s to master '%s@%s:%d'" - " - retry-time: %d retries: %lu", + " - retry-time: %d retries: %lu message: %s", (reconnect ? "reconnecting" : "connecting"), mi->user, mi->host, mi->port, - mi->connect_retry, master_retry_count); + mi->connect_retry, master_retry_count, + mysql_error(mysql)); } /* By default we try forever. The reason is that failure will trigger === modified file 'sql/sql_cache.cc' --- a/sql/sql_cache.cc 2010-01-15 15:27:55 +0000 +++ b/sql/sql_cache.cc 2010-01-29 10:42:31 +0000 @@ -377,9 +377,11 @@ TODO list: static void debug_wait_for_kill(const char *info) { - DBUG_ENTER("debug_wait_for_kill"); const char *prev_info; THD *thd; + char buff[1024]; + DBUG_ENTER("debug_wait_for_kill"); + thd= current_thd; prev_info= thd->proc_info; thd->proc_info= info; @@ -387,8 +389,16 @@ static void debug_wait_for_kill(const ch while(!thd->killed) my_sleep(1000); thd->killed= THD::NOT_KILLED; + /* + Remove the set debug variable, to ensure we don't get stuck on it again + This is needed as for MyISAM, invalidate_table() may be called twice + (Once from mysql_delete() and once from mi_update_status()) + */ + sprintf(buff, "-d,%s", info); + DBUG_SET(buff); sql_print_information("Exit debug_wait_for_kill"); thd->proc_info= prev_info; + DBUG_VOID_RETURN; } @@ -914,15 +924,18 @@ void query_cache_insert(NET *net, const void query_cache_abort(NET *net) { + THD *thd; DBUG_ENTER("query_cache_abort"); - THD *thd= current_thd; /* See the comment on double-check locking usage above. */ if (net->query_cache_query == 0) DBUG_VOID_RETURN; if (query_cache.try_lock()) + { + net->query_cache_query = 0; DBUG_VOID_RETURN; + } /* While we were waiting another thread might have changed the status @@ -932,6 +945,7 @@ void query_cache_abort(NET *net) net->query_cache_query); if (query_block) { + thd= current_thd; thd_proc_info(thd, "storing result in query cache"); DUMP(&query_cache); BLOCK_LOCK_WR(query_block); @@ -941,6 +955,7 @@ void query_cache_abort(NET *net) DBUG_EXECUTE("check_querycache",query_cache.check_integrity(1);); } + DBUG_ASSERT(!net->query_cache_query); query_cache.unlock(); DBUG_VOID_RETURN; } @@ -970,8 +985,12 @@ void query_cache_end_of_result(THD *thd) #endif if (query_cache.try_lock()) + { + thd->net.query_cache_query= 0; DBUG_VOID_RETURN; + } + /* thd->net.query_cache_query may have changed during resize */ query_block= ((Query_cache_block*) thd->net.query_cache_query); if (query_block) { @@ -997,8 +1016,8 @@ void query_cache_end_of_result(THD *thd) to this function. In the release version that query should be ignored and removed from QC. */ - DBUG_ASSERT(0); query_cache.free_query(query_block); + thd->net.query_cache_query= 0; query_cache.unlock(); DBUG_VOID_RETURN; } === modified file 'sql/sql_connect.cc' --- a/sql/sql_connect.cc 2009-12-03 11:19:05 +0000 +++ b/sql/sql_connect.cc 2010-01-29 10:42:31 +0000 @@ -798,6 +798,7 @@ static int check_connection(THD *thd) DBUG_PRINT("info", ("client capabilities: %lu", thd->client_capabilities)); if (thd->client_capabilities & CLIENT_SSL) { + char error_string[1024]; /* Do the SSL layering. */ if (!ssl_acceptor_fd) { @@ -806,7 +807,7 @@ static int check_connection(THD *thd) return 1; } DBUG_PRINT("info", ("IO layer change in progress...")); - if (sslaccept(ssl_acceptor_fd, net->vio, net->read_timeout)) + if (sslaccept(ssl_acceptor_fd, net->vio, net->read_timeout, error_string)) { DBUG_PRINT("error", ("Failed to accept new SSL connection")); inc_host_errors(&thd->remote.sin_addr); === modified file 'sql/sql_parse.cc' --- a/sql/sql_parse.cc 2010-01-15 15:27:55 +0000 +++ b/sql/sql_parse.cc 2010-01-29 10:42:31 +0000 @@ -1252,12 +1252,14 @@ bool dispatch_command(enum enum_server_c ha_maria::implicit_commit(thd, FALSE); #endif - net_end_statement(thd); - query_cache_end_of_result(thd); /* Multiple queries exits, execute them individually */ close_thread_tables(thd); + + net_end_statement(thd); + query_cache_end_of_result(thd); + ulong length= (ulong)(packet_end - beginning_of_next_stmt); log_slow_statement(thd); @@ -1645,13 +1647,22 @@ bool dispatch_command(enum enum_server_c ha_maria::implicit_commit(thd, FALSE); #endif - net_end_statement(thd); - query_cache_end_of_result(thd); - + if (!(sql_command_flags[thd->lex->sql_command] & CF_CHANGES_DATA)) + { + /* No changes in data; We can send ok at once to the client */ + net_end_statement(thd); + query_cache_end_of_result(thd); + } thd->proc_info= "closing tables"; /* Free tables */ close_thread_tables(thd); + if (sql_command_flags[thd->lex->sql_command] & CF_CHANGES_DATA) + { + net_end_statement(thd); + query_cache_end_of_result(thd); + } + log_slow_statement(thd); thd_proc_info(thd, "cleaning up"); === modified file 'storage/myisam/ft_boolean_search.c' --- a/storage/myisam/ft_boolean_search.c 2010-01-27 21:53:08 +0000 +++ b/storage/myisam/ft_boolean_search.c 2010-01-28 14:49:14 +0000 @@ -321,7 +321,7 @@ static int _ftb_parse_query(FTB *ftb, uc param->mysql_add_word= ftb_query_add_word; param->mysql_ftparam= (void *)&ftb_param; param->cs= ftb->charset; - param->doc= (char*) query; + param->doc= query; param->length= len; param->flags= 0; param->mode= MYSQL_FTPARSER_FULL_BOOLEAN_INFO; @@ -702,7 +702,7 @@ static int _ftb_check_phrase(FTB *ftb, c param->mysql_add_word= ftb_phrase_add_word; param->mysql_ftparam= (void *)&ftb_param; param->cs= ftb->charset; - param->doc= (char *) document; + param->doc= document; param->length= len; param->flags= 0; param->mode= MYSQL_FTPARSER_WITH_STOPWORDS; @@ -1000,7 +1000,7 @@ float ft_boolean_find_relevance(FT_INFO { if (!ftsi.pos) continue; - param->doc= (char *)ftsi.pos; + param->doc= ftsi.pos; param->length= ftsi.len; if (unlikely(parser->parse(param))) return 0; === modified file 'storage/myisam/ft_parser.c' --- a/storage/myisam/ft_parser.c 2010-01-27 21:53:08 +0000 +++ b/storage/myisam/ft_parser.c 2010-01-28 14:49:14 +0000 @@ -175,7 +175,7 @@ uchar ft_get_word(CHARSET_INFO *cs, cons if ((param->trunc=(doc<end && *doc == FTB_TRUNC))) doc++; - if (((length >= ft_min_word_len && !is_stopword((char*) word->pos, + if (((length >= ft_min_word_len && !is_stopword(word->pos, word->len)) || param->trunc) && length < ft_max_word_len) { @@ -298,7 +298,7 @@ static int ft_parse_internal(MYSQL_FTPAR DBUG_ENTER("ft_parse_internal"); while (ft_simple_get_word(wtree->custom_arg, &doc, end, &w, TRUE)) - if (param->mysql_add_word(param, (char*) w.pos, w.len, 0)) + if (param->mysql_add_word(param, w.pos, w.len, 0)) DBUG_RETURN(1); DBUG_RETURN(0); } @@ -319,7 +319,7 @@ int ft_parse(TREE *wtree, const uchar *d param->mysql_add_word= ft_add_word; param->mysql_ftparam= &my_param; param->cs= wtree->custom_arg; - param->doc= (char*) doc; + param->doc= doc; param->length= doclen; param->mode= MYSQL_FTPARSER_SIMPLE_MODE; DBUG_RETURN(parser->parse(param)); === modified file 'storage/myisam/ft_stopwords.c' --- a/storage/myisam/ft_stopwords.c 2010-01-27 21:53:08 +0000 +++ b/storage/myisam/ft_stopwords.c 2010-01-28 14:49:14 +0000 @@ -20,7 +20,7 @@ typedef struct st_ft_stopwords { - const char * pos; + const uchar* pos; size_t len; } FT_STOPWORD; @@ -30,15 +30,15 @@ static int FT_STOPWORD_cmp(void* cmp_arg FT_STOPWORD *w1, FT_STOPWORD *w2) { return ha_compare_text(default_charset_info, - (uchar *)w1->pos,w1->len, - (uchar *)w2->pos,w2->len,0,0); + w1->pos, w1->len, + w2->pos, w2->len, 0, 0); } static void FT_STOPWORD_free(FT_STOPWORD *w, TREE_FREE action, void *arg __attribute__((unused))) { if (action == free_free) - my_free((uchar*) w->pos, MYF(0)); + my_free((void*) w->pos, MYF(0)); } static int ft_add_stopword(const char *w) === modified file 'support-files/compiler_warnings.supp' --- a/support-files/compiler_warnings.supp 2010-01-28 11:35:10 +0000 +++ b/support-files/compiler_warnings.supp 2010-01-28 14:49:14 +0000 @@ -4,6 +4,7 @@ # integer.cpp: .*control reaches end of non-void function.*: 1288-1427 +integer.cpp: .*no return statement in function returning non-void.*: 1288-1427 DictTabInfo.cpp : .*invalid access to non-static.* DictTabInfo.cpp : .*macro was used incorrectly.* DbdihMain.cpp : .*unused variable.* : 6666-7013 @@ -27,13 +28,17 @@ _flex_tmp.c: .*not enough actual paramet pars0lex.l: .*conversion from 'ulint' to 'int', possible loss of data.* btr/btr0cur\.c: .*value computed is not used.*: 3175-3375 include/buf0buf\.ic: unused parameter ��mtr�� +fil/fil0fil\.c: pointer targets in passing argument.*differ in signedness fil/fil0fil\.c: comparison between signed and unsigned : 3100-3199 fil/fil0fil\.c: unused parameter log/log0recv\.c: unused variable os/os0file\.c: unused parameter +os/os0file\.c: pointer targets in assignment differ in signedness handler/i_s\.cc: unused variable sync/sync0rw\.c: unused parameter sync/sync0sync\.c: unused parameter +sync/sync0sync\.c: unused variable +ut/ut0ut\.c: ignoring return value of # # bdb is not critical to keep up to date @@ -98,10 +103,15 @@ storage/maria/ma_pagecache.c: .*'info_ch # I think these are due to mix of C and C++. # storage/pbxt/ : typedef.*was ignored in this declaration +ha_pbxt\.cc : variable.*might be clobbered by.*longjmp # # Yassl include/runtime.hpp: .*pure_error.* +.*/extra/yassl/taocrypt/.*: comparison with string literal +.*/extra/yassl/taocrypt/src/blowfish\.cpp: array subscript is above array bounds +.*/extra/yassl/taocrypt/src/file\.cpp: ignoring return value +.*/extra/yassl/taocrypt/src/integer\.cpp: control reaches end of non-void function # # Groff warnings on OpenSUSE. === modified file 'unittest/mysys/waiting_threads-t.c' --- a/unittest/mysys/waiting_threads-t.c 2009-09-15 10:46:35 +0000 +++ b/unittest/mysys/waiting_threads-t.c 2010-01-28 14:49:14 +0000 @@ -258,7 +258,7 @@ void do_tests() #define test_kill_strategy(X) \ diag("kill strategy: " #X); \ DBUG_EXECUTE("reset_file", \ - { rewind(DBUG_FILE); ftruncate(fileno(DBUG_FILE), 0); }); \ + { rewind(DBUG_FILE); (void) ftruncate(fileno(DBUG_FILE), 0); }); \ DBUG_PRINT("info", ("kill strategy: " #X)); \ kill_strategy=X; \ do_one_test(); === modified file 'vio/viossl.c' --- a/vio/viossl.c 2008-02-07 08:55:49 +0000 +++ b/vio/viossl.c 2010-01-29 10:42:31 +0000 @@ -74,8 +74,11 @@ report_errors(SSL* ssl) } if (ssl) - DBUG_PRINT("error", ("error: %s", - ERR_error_string(SSL_get_error(ssl, l), buf))); + { + int error= SSL_get_error(ssl, l); + DBUG_PRINT("error", ("error: %s (%d)", + ERR_error_string(error, buf), error)); + } DBUG_PRINT("info", ("socket_errno: %d", socket_errno)); DBUG_VOID_RETURN; @@ -174,16 +177,17 @@ void vio_ssl_delete(Vio *vio) static int ssl_do(struct st_VioSSLFd *ptr, Vio *vio, long timeout, - int (*connect_accept_func)(SSL*)) + int (*connect_accept_func)(SSL*), char *error_string) { SSL *ssl; my_bool unused; my_bool was_blocking; - DBUG_ENTER("ssl_do"); DBUG_PRINT("enter", ("ptr: 0x%lx, sd: %d ctx: 0x%lx", (long) ptr, vio->sd, (long) ptr->ssl_context)); + error_string[0]= 0; + /* Set socket to blocking if not already set */ vio_blocking(vio, 1, &was_blocking); @@ -191,6 +195,7 @@ static int ssl_do(struct st_VioSSLFd *pt { DBUG_PRINT("error", ("SSL_new failure")); report_errors(ssl); + strmov(error_string, "SSL_new failed"); vio_blocking(vio, was_blocking, &unused); DBUG_RETURN(1); } @@ -203,6 +208,7 @@ static int ssl_do(struct st_VioSSLFd *pt { DBUG_PRINT("error", ("SSL_connect/accept failure")); report_errors(ssl); + ERR_error_string(SSL_get_error(ssl, 0), error_string); SSL_free(ssl); vio_blocking(vio, was_blocking, &unused); DBUG_RETURN(1); @@ -250,17 +256,19 @@ static int ssl_do(struct st_VioSSLFd *pt } -int sslaccept(struct st_VioSSLFd *ptr, Vio *vio, long timeout) +int sslaccept(struct st_VioSSLFd *ptr, Vio *vio, long timeout, + char *error_string) { DBUG_ENTER("sslaccept"); - DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_accept)); + DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_accept, error_string)); } -int sslconnect(struct st_VioSSLFd *ptr, Vio *vio, long timeout) +int sslconnect(struct st_VioSSLFd *ptr, Vio *vio, long timeout, + char *error_string) { DBUG_ENTER("sslconnect"); - DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_connect)); + DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_connect, error_string)); }

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2807: Auto merge
by noreply＠launchpad.net 29 Jan '10

29 Jan '10

Merge authors: Matthew Montgomery <mmontgomery(a)mysql.com> Michael Widenius (monty) ------------------------------------------------------------ revno: 2807 [merge] committer: Michael Widenius <monty(a)askmonty.org> branch nick: maria-5.1 timestamp: Fri 2010-01-29 12:52:02 +0200 message: Auto merge modified: client/mysqltest.cc configure.in extra/yassl/src/yassl_error.cpp include/violite.h mysql-test/mysql-test-run.pl mysql-test/r/grant.result mysql-test/r/openssl_1.result mysql-test/std_data/cacert.pem mysql-test/std_data/client-cert.pem mysql-test/std_data/client-key.pem mysql-test/std_data/server-cert.pem mysql-test/std_data/server-key.pem mysql-test/t/grant.test mysql-test/t/openssl_1.test mysql-test/t/partition_innodb_semi_consistent.test mysql-test/t/query_cache_debug.test sql-common/client.c sql/net_serv.cc sql/slave.cc sql/sql_cache.cc sql/sql_connect.cc sql/sql_parse.cc storage/myisam/ft_boolean_search.c storage/myisam/ft_parser.c storage/myisam/ft_stopwords.c support-files/compiler_warnings.supp unittest/mysys/waiting_threads-t.c vio/viossl.c -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2807)
by Michael Widenius 29 Jan '10

29 Jan '10

#At lp:maria based on revid:monty@askmonty.org-20100128144914-lqwgn4dphka023yz 2807 Michael Widenius 2010-01-29 [merge] Changed version number from RC to stable Fixed bug in Yassle to get correct error messages in case of errors Provide better error messages in case of ssl connect failure Updated out-of-date ssl certificates to fix failing mysql-test-system (certificates now active for 10 years) Fixed bug in query_cache that could cause asserts and hangs in DEBUG builds. Fixed bug where one connection did not see changes done by another connection. modified: configure.in extra/yassl/src/yassl_error.cpp include/violite.h mysql-test/mysql-test-run.pl mysql-test/r/grant.result mysql-test/r/openssl_1.result mysql-test/std_data/cacert.pem mysql-test/std_data/client-cert.pem mysql-test/std_data/client-key.pem mysql-test/std_data/server-cert.pem mysql-test/std_data/server-key.pem mysql-test/t/grant.test mysql-test/t/openssl_1.test mysql-test/t/query_cache_debug.test sql-common/client.c sql/net_serv.cc sql/slave.cc sql/sql_cache.cc sql/sql_connect.cc sql/sql_parse.cc vio/viossl.c per-file messages: configure.in Changed version number from RC to stable extra/yassl/src/yassl_error.cpp Fixed bug in Yassle to get correct error messages in case of errors - 'error' is an enum that hold more error numbers than the enum was defined for include/violite.h Added error output string for sslaccept() and sslconnect() to get reason for connect failure mysql-test/mysql-test-run.pl Write failed test cases if mysql-test-run fails because of too many errors mysql-test/r/grant.result Update results to reflect new certificates mysql-test/r/openssl_1.result Update results to reflect new certificates mysql-test/std_data/cacert.pem Update ssl certificate mysql-test/std_data/client-cert.pem Update ssl certificate mysql-test/std_data/client-key.pem Update ssl certificate mysql-test/std_data/server-cert.pem Update ssl certificate mysql-test/std_data/server-key.pem Update ssl certificate mysql-test/t/grant.test Update test to reflect new certificates mysql-test/t/openssl_1.test Update test to reflect new certificates mysql-test/t/query_cache_debug.test Remove 'big_test' as test is now fast sql-common/client.c Give a better error message if ssl connect fails sql/net_serv.cc Fixed compiler warnings sql/slave.cc Give a better error message in logs if ssl connect fails sql/sql_cache.cc debug_wait_for_kill() now removes the set watch variable after kill signal This is needed as invalidate_table() may be called twice for one query. Ensure that net->query_cache_query is reset after query. This fixes assert in query_cache_end_of_result() if query_cache_query holds results from previous query. Removed DBUG_ASSERT(0), as this code can be run by query_cache_debug.test sql/sql_connect.cc Give a better error message if ssl connect fails sql/sql_parse.cc Fixed bug where one connection did not see changes done by another connection. For statements that changes tables, close_thread_tables() MUST be called before sending OK as a table handler may not make the changes available for other connections before unlock_tables(). vio/viossl.c Give a better error message if ssl connect fails === modified file 'configure.in' --- a/configure.in 2010-01-17 17:22:46 +0000 +++ b/configure.in 2010-01-29 10:42:31 +0000 @@ -10,7 +10,7 @@ AC_CANONICAL_SYSTEM # # When changing major version number please also check switch statement # in mysqlbinlog::check_master_version(). -AM_INIT_AUTOMAKE(mysql, 5.1.42-MariaDB-rc) +AM_INIT_AUTOMAKE(mysql, 5.1.42-MariaDB) AM_CONFIG_HEADER([include/config.h:config.h.in]) # Request support for automake silent-rules if available. === modified file 'extra/yassl/src/yassl_error.cpp' --- a/extra/yassl/src/yassl_error.cpp 2008-11-18 16:45:44 +0000 +++ b/extra/yassl/src/yassl_error.cpp 2010-01-29 10:42:31 +0000 @@ -60,7 +60,7 @@ void SetErrorString(YasslError error, ch using namespace TaoCrypt; const int max = MAX_ERROR_SZ; // shorthand - switch (error) { + switch ((int) error) { // yaSSL proper errors case range_error : === modified file 'include/violite.h' --- a/include/violite.h 2010-01-15 15:27:55 +0000 +++ b/include/violite.h 2010-01-29 10:42:31 +0000 @@ -123,8 +123,8 @@ struct st_VioSSLFd SSL_CTX *ssl_context; }; -int sslaccept(struct st_VioSSLFd*, Vio *, long timeout); -int sslconnect(struct st_VioSSLFd*, Vio *, long timeout); +int sslaccept(struct st_VioSSLFd*, Vio *, long timeout, char *error_string); +int sslconnect(struct st_VioSSLFd*, Vio *, long timeout, char *error_string); struct st_VioSSLFd *new_VioSSLConnectorFd(const char *key_file, const char *cert_file, === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2010-01-27 17:41:05 +0000 +++ b/mysql-test/mysql-test-run.pl 2010-01-29 10:42:31 +0000 @@ -413,7 +413,6 @@ sub main { # Not all tests completed, failure mtr_report(); mtr_report("Only ", int(@$completed), " of $num_tests completed."); - mtr_error("Not all tests completed"); } mtr_print_line(); @@ -425,6 +424,10 @@ sub main { mtr_report_stats($fail, $completed, $extra_warnings); + if ( @$completed != $num_tests) + { + mtr_error("Not all tests completed"); + } exit(0); } === modified file 'mysql-test/r/grant.result' --- a/mysql-test/r/grant.result 2009-10-27 10:09:36 +0000 +++ b/mysql-test/r/grant.result 2010-01-29 10:42:31 +0000 @@ -29,15 +29,15 @@ show grants for mysqltest_1@localhost; Grants for mysqltest_1@localhost GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' GRANT SELECT ON `mysqltest`.* TO 'mysqltest_1'@'localhost' -grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "MySQL AB"; +grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "Monty Program Ab"; show grants for mysqltest_1@localhost; Grants for mysqltest_1@localhost -GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'MySQL AB' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' +GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'Monty Program Ab' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' GRANT SELECT ON `mysqltest`.* TO 'mysqltest_1'@'localhost' revoke all privileges on mysqltest.* from mysqltest_1@localhost; show grants for mysqltest_1@localhost; Grants for mysqltest_1@localhost -GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'MySQL AB' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' +GRANT USAGE ON *.* TO 'mysqltest_1'@'localhost' REQUIRE ISSUER 'Monty Program Ab' SUBJECT 'testsubject' CIPHER 'EDH-RSA-DES-CBC3-SHA' delete from mysql.user where user='mysqltest_1'; flush privileges; delete from mysql.user where user='mysqltest_1'; === modified file 'mysql-test/r/openssl_1.result' --- a/mysql-test/r/openssl_1.result 2009-05-25 13:00:18 +0000 +++ b/mysql-test/r/openssl_1.result 2010-01-29 10:42:31 +0000 @@ -3,8 +3,10 @@ create table t1(f1 int); insert into t1 values (5); grant select on test.* to ssl_user1@localhost require SSL; grant select on test.* to ssl_user2@localhost require cipher "DHE-RSA-AES256-SHA"; -grant select on test.* to ssl_user3@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com"; -grant select on test.* to ssl_user4@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com" ISSUER "/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB"; +grant select on test.* to ssl_user3@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; +grant select on test.* to ssl_user4@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org" ISSUER "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; grant select on test.* to ssl_user5@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "xxx"; flush privileges; connect(localhost,ssl_user5,,test,MASTER_PORT,MASTER_SOCKET); @@ -44,9 +46,9 @@ ERROR 42000: DELETE command denied to us drop user ssl_user1@localhost, ssl_user2@localhost, ssl_user3@localhost, ssl_user4@localhost, ssl_user5@localhost; drop table t1; -mysqltest: Could not open connection 'default': 2026 SSL connection error -mysqltest: Could not open connection 'default': 2026 SSL connection error -mysqltest: Could not open connection 'default': 2026 SSL connection error +mysqltest: Could not open connection 'default': 2026 SSL error: ASN: bad other signature confirmation +mysqltest: Could not open connection 'default': 2026 SSL error: ASN: bad other signature confirmation +mysqltest: Could not open connection 'default': 2026 SSL error: ASN: bad other signature confirmation SSL error: Unable to get private key from '' mysqltest: Could not open connection 'default': 2026 SSL connection error SSL error: Unable to get certificate from '' === modified file 'mysql-test/std_data/cacert.pem' --- a/mysql-test/std_data/cacert.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/cacert.pem 2010-01-29 10:42:31 +0000 @@ -1,17 +1,19 @@ -----BEGIN CERTIFICATE----- -MIICrTCCAhagAwIBAgIJAJXpePU0UOTVMA0GCSqGSIb3DQEBBQUAMEQxCzAJBgNV -BAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxhMRAwDgYDVQQHEwdVcHBzYWxhMREwDwYD -VQQKEwhNeVNRTCBBQjAeFw0wOTAxMjgxMDQ5NDZaFw0xNDAxMjcxMDQ5NDZaMEQx -CzAJBgNVBAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxhMRAwDgYDVQQHEwdVcHBzYWxh -MREwDwYDVQQKEwhNeVNRTCBBQjCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA -4XQHAe5R1+TXC8noZtWf+d5E0v1C59FWpn9SWEUCBjE5UiIwuJvi4Y+7xWGOXLAI -/JzJx5gNXLBiTsE/zh0uX9fKlajLhxB0GN+QU0ZlpQ1BeYipEcNXeI/7cT499f6v -XWabnTflivdCgHSWUOQ20/Lzs6kP6/e6OoZd/DPSjPECAwEAAaOBpjCBozAdBgNV -HQ4EFgQU8uLqVWWkmuKsnZf1RWz294wRrd8wdAYDVR0jBG0wa4AU8uLqVWWkmuKs -nZf1RWz294wRrd+hSKRGMEQxCzAJBgNVBAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxh -MRAwDgYDVQQHEwdVcHBzYWxhMREwDwYDVQQKEwhNeVNRTCBBQoIJAJXpePU0UOTV -MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADgYEAMMTE5sDN+Z0ZlV7KvH3g -6+aKvql8dTpRT3hYukeQlWua0nq74WPGVw0c4e/M/vbiMwmJcCYpB9pd4+dHqzSw -aPyoenjY6UF8n7B4quWy3SIUk2LSHeJLW+kzJn2afN9gvipFhdVh/uU2TIyLGOur -Z/vmJX2W7hF1uqPnbfa8Lrw= +MIIDIjCCAougAwIBAgIJAJhuvLP+2mGwMA0GCSqGSIb3DQEBBQUAMGoxCzAJBgNV +BAYTAkZJMRAwDgYDVQQIEwdUdXVzdWxhMRkwFwYDVQQKExBNb250eSBQcm9ncmFt +IEFiMS4wLAYJKoZIhvcNAQkBFh9hYnN0cmFjdC5kZXZlbG9wZXJAYXNrbW9udHku +b3JnMB4XDTEwMDEyODIxNTcyNVoXDTEwMDIyNzIxNTcyNVowajELMAkGA1UEBhMC +RkkxEDAOBgNVBAgTB1R1dXN1bGExGTAXBgNVBAoTEE1vbnR5IFByb2dyYW0gQWIx +LjAsBgkqhkiG9w0BCQEWH2Fic3RyYWN0LmRldmVsb3BlckBhc2ttb250eS5vcmcw +gZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMIaY4pwbst50S32xJH3bSXhPGep +6gx1AWwZKsHTXL3VeMO6PHmC8zu5HM0zbOcrIJcXL3YVnpmE4b9OQxIiMSx1Yd+U +u8/sTkxgpsEKhCbIzECIwPhppyT/JP5aSXCadEvg+PSjikv8dOVkD68wVG4CcFIX +MFttsPebBVzEokZZAgMBAAGjgc8wgcwwHQYDVR0OBBYEFOCKaNHFFPrju8AwzWxS +f96IKfRwMIGcBgNVHSMEgZQwgZGAFOCKaNHFFPrju8AwzWxSf96IKfRwoW6kbDBq +MQswCQYDVQQGEwJGSTEQMA4GA1UECBMHVHV1c3VsYTEZMBcGA1UEChMQTW9udHkg +UHJvZ3JhbSBBYjEuMCwGCSqGSIb3DQEJARYfYWJzdHJhY3QuZGV2ZWxvcGVyQGFz +a21vbnR5Lm9yZ4IJAJhuvLP+2mGwMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEF +BQADgYEAsmCX2/k9AInq2qhXtnkLip6cB0iOerLTNAzEijZc/aVf4wUjkL3cqhmC +kSTCwAHIOxp+ICwh6ky3xghXjoI9QnPFDVkRkzPT2tV0IoBaeQuI4e0CU2EY7L3P +XoDqp3oq1XtVcr9ZZdP68fBYUG/qcrWcXWk45ZFaBmBv3TotsGk= -----END CERTIFICATE----- === modified file 'mysql-test/std_data/client-cert.pem' --- a/mysql-test/std_data/client-cert.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/client-cert.pem 2010-01-29 10:42:31 +0000 @@ -1,55 +1,60 @@ Certificate: Data: Version: 3 (0x2) - Serial Number: 3 (0x3) + Serial Number: 2 (0x2) Signature Algorithm: sha1WithRSAEncryption - Issuer: C=SE, ST=Uppsala, L=Uppsala, O=MySQL AB + Issuer: C=FI, ST=Tuusula, O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org Validity - Not Before: Jan 28 11:04:39 2009 GMT - Not After : Jan 28 11:04:39 2010 GMT - Subject: C=SE, ST=Uppsala, O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com + Not Before: Jan 28 22:01:38 2010 GMT + Not After : Dec 7 22:01:38 2019 GMT + Subject: C=FI, ST=Tuusula, O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org Subject Public Key Info: Public Key Algorithm: rsaEncryption - RSA Public Key: (512 bit) - Modulus (512 bit): - 00:e1:52:30:2c:d9:be:64:28:91:5d:7a:fd:d9:e9: - 14:35:7a:d2:94:4e:91:46:e0:db:9f:6b:79:f4:4c: - ac:6e:07:61:34:86:74:62:a7:a8:44:af:fa:87:87: - a8:7d:42:61:ff:ab:50:d4:7b:bf:75:fa:d5:d5:b3: - 74:fb:56:1e:37 + RSA Public Key: (1024 bit) + Modulus (1024 bit): + 00:bd:dc:3d:f8:3c:0b:d4:d2:c0:a3:9c:34:2d:e7: + 11:ff:4d:43:35:17:f6:0d:91:01:92:9e:4f:4d:c0: + 38:d5:62:03:55:33:db:66:a2:91:d3:f2:b6:23:34: + 95:53:50:3d:4f:e3:0c:d7:76:fd:f6:54:64:e6:f6: + dc:70:74:7c:6b:74:41:59:b0:19:5d:62:90:3c:a7: + c8:5e:21:8f:2b:22:6b:c7:43:9b:be:79:84:60:da: + 16:c9:ce:ee:fd:66:cb:54:81:e2:b5:1c:cf:f9:74: + de:38:2b:28:d4:31:33:55:d2:30:1c:a3:e4:c2:c7: + 31:46:43:d5:33:3d:8a:0b:47 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE + Netscape Comment: + OpenSSL Generated Certificate X509v3 Subject Key Identifier: - 58:30:B5:9B:2C:05:94:06:BA:3D:3C:F0:B2:CD:1D:67:65:E3:7F:85 + BE:E6:DB:19:8D:DB:72:9A:85:EE:B2:B8:5D:E7:FF:61:DF:09:08:AF X509v3 Authority Key Identifier: - keyid:F2:E2:EA:55:65:A4:9A:E2:AC:9D:97:F5:45:6C:F6:F7:8C:11:AD:DF - DirName:/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB - serial:95:E9:78:F5:34:50:E4:D5 + keyid:E0:8A:68:D1:C5:14:FA:E3:BB:C0:30:CD:6C:52:7F:DE:88:29:F4:70 Signature Algorithm: sha1WithRSAEncryption - 05:19:e3:13:14:fc:c5:28:bf:69:f8:00:b3:25:cb:bd:ca:9f: - 2f:4c:b3:a8:04:11:f0:74:27:bd:82:2c:b4:49:9b:a7:59:f0: - f7:87:d1:e0:ba:99:a2:fe:4b:1d:10:6f:e4:a2:b3:cd:7f:8b: - 68:31:46:ee:cd:9e:e2:47:e1:4c:fa:74:d1:e2:8b:cc:a0:4b: - a8:24:d1:a4:c3:6b:2a:c6:28:cd:41:e0:06:48:e6:cf:f2:3c: - ca:37:95:d7:29:64:6b:91:91:83:e7:ac:c8:0b:87:bc:da:a6: - aa:f1:44:43:c8:74:7b:15:26:91:2e:03:c4:71:50:6c:f8:68: - dc:8c + 41:95:6d:0a:a4:ee:af:68:cd:94:26:59:9a:18:b7:75:3c:c5: + 0f:22:d3:5c:31:9b:85:a0:93:b3:f0:50:29:ba:1e:d3:5a:43: + 0b:77:2d:98:87:a7:a7:39:0f:40:8d:03:d3:b3:67:43:77:bc: + 3c:51:c2:f9:9e:7a:2d:39:c4:5c:16:d7:70:d6:74:d1:6c:e1: + 6a:4d:fd:1f:10:af:64:3b:f4:64:e9:b2:b3:fb:c8:cd:c5:41: + cd:99:e0:ac:83:1d:81:2c:6b:99:ba:80:02:12:72:f7:3b:bb: + 93:72:00:da:ff:d3:87:75:d2:3a:a4:ca:4d:c1:8b:c1:21:50: + cb:57 -----BEGIN CERTIFICATE----- -MIICfzCCAeigAwIBAgIBAzANBgkqhkiG9w0BAQUFADBEMQswCQYDVQQGEwJTRTEQ -MA4GA1UECBMHVXBwc2FsYTEQMA4GA1UEBxMHVXBwc2FsYTERMA8GA1UEChMITXlT -UUwgQUIwHhcNMDkwMTI4MTEwNDM5WhcNMTAwMTI4MTEwNDM5WjBlMQswCQYDVQQG -EwJTRTEQMA4GA1UECBMHVXBwc2FsYTERMA8GA1UEChMITXlTUUwgQUIxMTAvBgkq -hkiG9w0BCQEWImFic3RyYWN0Lm15c3FsLmRldmVsb3BlckBteXNxbC5jb20wXDAN -BgkqhkiG9w0BAQEFAANLADBIAkEA4VIwLNm+ZCiRXXr92ekUNXrSlE6RRuDbn2t5 -9EysbgdhNIZ0YqeoRK/6h4eofUJh/6tQ1Hu/dfrV1bN0+1YeNwIDAQABo4GjMIGg -MAkGA1UdEwQCMAAwHQYDVR0OBBYEFFgwtZssBZQGuj088LLNHWdl43+FMHQGA1Ud -IwRtMGuAFPLi6lVlpJrirJ2X9UVs9veMEa3foUikRjBEMQswCQYDVQQGEwJTRTEQ -MA4GA1UECBMHVXBwc2FsYTEQMA4GA1UEBxMHVXBwc2FsYTERMA8GA1UEChMITXlT -UUwgQUKCCQCV6Xj1NFDk1TANBgkqhkiG9w0BAQUFAAOBgQAFGeMTFPzFKL9p+ACz -Jcu9yp8vTLOoBBHwdCe9giy0SZunWfD3h9Hgupmi/ksdEG/korPNf4toMUbuzZ7i -R+FM+nTR4ovMoEuoJNGkw2sqxijNQeAGSObP8jzKN5XXKWRrkZGD56zIC4e82qaq -8URDyHR7FSaRLgPEcVBs+GjcjA== +MIICxTCCAi6gAwIBAgIBAjANBgkqhkiG9w0BAQUFADBqMQswCQYDVQQGEwJGSTEQ +MA4GA1UECBMHVHV1c3VsYTEZMBcGA1UEChMQTW9udHkgUHJvZ3JhbSBBYjEuMCwG +CSqGSIb3DQEJARYfYWJzdHJhY3QuZGV2ZWxvcGVyQGFza21vbnR5Lm9yZzAeFw0x +MDAxMjgyMjAxMzhaFw0xOTEyMDcyMjAxMzhaMGoxCzAJBgNVBAYTAkZJMRAwDgYD +VQQIEwdUdXVzdWxhMRkwFwYDVQQKExBNb250eSBQcm9ncmFtIEFiMS4wLAYJKoZI +hvcNAQkBFh9hYnN0cmFjdC5kZXZlbG9wZXJAYXNrbW9udHkub3JnMIGfMA0GCSqG +SIb3DQEBAQUAA4GNADCBiQKBgQC93D34PAvU0sCjnDQt5xH/TUM1F/YNkQGSnk9N +wDjVYgNVM9tmopHT8rYjNJVTUD1P4wzXdv32VGTm9txwdHxrdEFZsBldYpA8p8he +IY8rImvHQ5u+eYRg2hbJzu79ZstUgeK1HM/5dN44KyjUMTNV0jAco+TCxzFGQ9Uz +PYoLRwIDAQABo3sweTAJBgNVHRMEAjAAMCwGCWCGSAGG+EIBDQQfFh1PcGVuU1NM +IEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUvubbGY3bcpqF7rK4Xef/ +Yd8JCK8wHwYDVR0jBBgwFoAU4Ipo0cUU+uO7wDDNbFJ/3ogp9HAwDQYJKoZIhvcN +AQEFBQADgYEAQZVtCqTur2jNlCZZmhi3dTzFDyLTXDGbhaCTs/BQKboe01pDC3ct +mIenpzkPQI0D07NnQ3e8PFHC+Z56LTnEXBbXcNZ00Wzhak39HxCvZDv0ZOmys/vI +zcVBzZngrIMdgSxrmbqAAhJy9zu7k3IA2v/Th3XSOqTKTcGLwSFQy1c= -----END CERTIFICATE----- === modified file 'mysql-test/std_data/client-key.pem' --- a/mysql-test/std_data/client-key.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/client-key.pem 2010-01-29 10:42:31 +0000 @@ -1,9 +1,15 @@ -----BEGIN RSA PRIVATE KEY----- -MIIBOQIBAAJBAOFSMCzZvmQokV16/dnpFDV60pROkUbg259refRMrG4HYTSGdGKn -qESv+oeHqH1CYf+rUNR7v3X61dWzdPtWHjcCAwEAAQJAXYooM8ZlcuEgj+VKU1ee -qyEFIMqJJxqcMk+E/nWCM96WxCP3zHNSrqNfSpI3ld7QzMwhdRz+gFLxT2gGNpIw -MQIhAPxzM/lDihe67X3ADYtDl9ZjA8Pm430x9sXlcxI17tCZAiEA5H1SyFl4mUee -9VnfSC2XGW7lwz72ZygfVX+b7tLWF08CIEh40gzW5MfXM+KLxdea+fXjyursV5ZT -R6KcMiKiNQLRAiAcmHqlzFzFgisotai2Fc6VRkXHG7gmzOSvBJt1VjmpDQIge6jf -2N7whTdvC4ferB+zUlgWQdyvx1c3T4gnt6PYdaY= +MIICXgIBAAKBgQC93D34PAvU0sCjnDQt5xH/TUM1F/YNkQGSnk9NwDjVYgNVM9tm +opHT8rYjNJVTUD1P4wzXdv32VGTm9txwdHxrdEFZsBldYpA8p8heIY8rImvHQ5u+ +eYRg2hbJzu79ZstUgeK1HM/5dN44KyjUMTNV0jAco+TCxzFGQ9UzPYoLRwIDAQAB +AoGBAJa2lprPT7UJ99Ho1aL6ota/RnKHKtNqII17DgjyZis9OtgP6kJ3GrvdF6iq +vT79my4nVrJTyxYXuGF/5U1/qqNjuPPBE1Xbu1ubQlFv8CT0kKYynQ7Z3ls8fAHC +B3VJXnUVlG+GHtUEFFG4FQVX1fn/Sga67ioJ6ivAiBlHKaPBAkEA5f2ToWlj4u9O +KgfRkN54wdIp4yu2c40pbhMfKGjGGsBAHk92+qSBpzEmxLcI6Ay+4/QysSR4jYmK +jCJuxiTu1QJBANNU3Hx8Il2SF/2BqGLcIh2SHxzKQIT5wAyD2jb+P2cHvbk6pKGR +VTmw5bibxXmYMS6J/L2zUF2xtFe+Svwz96sCQEnKYSqBqOWvyBFeLtPfPTlal8vm +Q4SxfuBtTCrn6t+8XRYcgt0KGPsunvSwkS/6nuh+eiExxWgMACLUDVyPjv0CQQC4 +sJJc7LOv6Oy0bWr2swHRrBEqvQsz63zOszCzHPHWHirNxZV5aiT8XT/2XZRwlvRs +gsVyGFLk/1fn0vN/g/8vAkEAxUdzUKvC1ZwjzGhgcz2bQU0tEZN4C9jBCiwOI2ud +BpAsPG0xAGGL2+hz0B0n88XiTHobiTZ1bg4Z41i4pXx2ZA== -----END RSA PRIVATE KEY----- === modified file 'mysql-test/std_data/server-cert.pem' --- a/mysql-test/std_data/server-cert.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/server-cert.pem 2010-01-29 10:42:31 +0000 @@ -3,53 +3,59 @@ Certificate: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha1WithRSAEncryption - Issuer: C=SE, ST=Uppsala, L=Uppsala, O=MySQL AB + Issuer: C=FI, ST=Tuusula, O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org Validity - Not Before: Jan 28 10:55:13 2009 GMT - Not After : Jan 28 10:55:13 2010 GMT - Subject: C=SE, ST=Uppsala, O=MySQL AB, CN=localhost/emailAddress=abstract.mysql.developer(a)mysql.com + Not Before: Jan 28 21:59:14 2010 GMT + Not After : Dec 7 21:59:14 2019 GMT + Subject: C=FI, ST=Tuusula, O=Monty Program Ab, CN=localhost/emailAddress=abstract.developer(a)askmonty.org Subject Public Key Info: Public Key Algorithm: rsaEncryption - RSA Public Key: (512 bit) - Modulus (512 bit): - 00:b6:8f:e5:b7:b4:86:83:13:8a:f9:bf:63:cb:64: - 2d:b9:51:d1:de:ab:7b:45:1f:aa:b5:66:73:13:f9: - a6:07:d5:ba:7c:fa:92:bd:37:e2:ad:87:db:3e:b6: - 6a:12:64:f8:ee:17:e3:15:06:2f:a8:82:68:bf:57: - 8d:c3:04:98:27 + RSA Public Key: (1024 bit) + Modulus (1024 bit): + 00:cc:79:74:2a:14:7e:77:06:b0:ec:1b:b6:da:70: + 4c:4f:0e:94:04:8a:e7:69:f4:4c:9c:67:22:41:56: + 3a:96:53:9e:95:9d:17:55:80:41:c0:13:d6:94:0f: + cd:2c:51:fe:a4:6e:f2:74:d8:5d:3b:3a:80:e3:85: + 5b:a5:bc:7d:5c:25:55:e5:40:77:fe:f3:cb:5b:cd: + da:a5:f4:36:64:30:a2:a0:7f:93:b3:c4:56:75:2a: + c0:f7:65:2a:d7:e6:ff:88:25:03:e0:b4:40:2e:74: + 4c:cb:46:95:73:cb:25:5c:87:0e:ec:0f:5a:19:c2: + b6:dc:9e:e8:f0:30:b1:9c:99 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE + Netscape Comment: + OpenSSL Generated Certificate X509v3 Subject Key Identifier: - D9:9A:B8:5F:22:EA:04:10:C8:25:7D:82:57:E6:2E:FD:19:29:E7:DA + 6D:13:3B:40:52:3C:AF:18:EA:33:D1:B7:56:21:1B:05:FE:0B:9E:38 X509v3 Authority Key Identifier: - keyid:F2:E2:EA:55:65:A4:9A:E2:AC:9D:97:F5:45:6C:F6:F7:8C:11:AD:DF - DirName:/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB - serial:95:E9:78:F5:34:50:E4:D5 + keyid:E0:8A:68:D1:C5:14:FA:E3:BB:C0:30:CD:6C:52:7F:DE:88:29:F4:70 Signature Algorithm: sha1WithRSAEncryption - 54:07:2d:21:0b:a5:af:3b:58:23:32:5e:56:7f:ab:58:63:48: - 91:aa:38:90:89:16:f9:cc:bf:a4:0e:78:2b:9f:c5:1b:58:a6: - e6:08:8f:2e:ae:97:03:21:9b:f1:cd:c0:26:8f:1d:d7:28:27: - a0:8e:81:09:1b:1c:0f:c9:a5:41:3a:2d:44:3f:9c:fa:87:ff: - c8:4c:2b:44:f7:1b:c1:3e:4f:01:7f:e9:26:cc:9f:1c:06:b5: - 0b:27:d1:10:90:be:93:0c:9c:e7:b0:d1:ea:27:99:4e:06:14: - 0c:7a:e9:c1:52:c5:33:68:bc:61:0d:db:81:3b:57:48:57:bf: - 42:9a + 97:db:65:23:7f:f1:15:3c:1e:83:ac:0e:0a:50:a0:0c:22:b8: + 45:d4:ca:21:05:47:3b:3d:03:b5:6c:4b:8d:bb:5f:57:c3:c7: + 4e:71:23:cf:33:a3:7f:a0:3d:bd:58:75:b8:37:22:16:2f:e9: + ed:ae:9b:94:29:81:6e:34:79:cf:41:bd:3d:8d:17:d7:22:1c: + 1b:58:c7:0f:79:13:56:1d:e8:d8:4e:e5:07:3f:79:1b:dd:c4: + 06:9b:c5:b6:02:34:43:c5:bf:e5:87:ad:f1:c1:8a:f2:be:c2: + 00:1d:d4:27:1f:87:c8:80:31:ec:6e:97:95:b4:84:40:d1:73: + 42:71 -----BEGIN CERTIFICATE----- -MIICkzCCAfygAwIBAgIBATANBgkqhkiG9w0BAQUFADBEMQswCQYDVQQGEwJTRTEQ -MA4GA1UECBMHVXBwc2FsYTEQMA4GA1UEBxMHVXBwc2FsYTERMA8GA1UEChMITXlT -UUwgQUIwHhcNMDkwMTI4MTA1NTEzWhcNMTAwMTI4MTA1NTEzWjB5MQswCQYDVQQG -EwJTRTEQMA4GA1UECBMHVXBwc2FsYTERMA8GA1UEChMITXlTUUwgQUIxEjAQBgNV -BAMTCWxvY2FsaG9zdDExMC8GCSqGSIb3DQEJARYiYWJzdHJhY3QubXlzcWwuZGV2 -ZWxvcGVyQG15c3FsLmNvbTBcMA0GCSqGSIb3DQEBAQUAA0sAMEgCQQC2j+W3tIaD -E4r5v2PLZC25UdHeq3tFH6q1ZnMT+aYH1bp8+pK9N+Kth9s+tmoSZPjuF+MVBi+o -gmi/V43DBJgnAgMBAAGjgaMwgaAwCQYDVR0TBAIwADAdBgNVHQ4EFgQU2Zq4XyLq -BBDIJX2CV+Yu/Rkp59owdAYDVR0jBG0wa4AU8uLqVWWkmuKsnZf1RWz294wRrd+h -SKRGMEQxCzAJBgNVBAYTAlNFMRAwDgYDVQQIEwdVcHBzYWxhMRAwDgYDVQQHEwdV -cHBzYWxhMREwDwYDVQQKEwhNeVNRTCBBQoIJAJXpePU0UOTVMA0GCSqGSIb3DQEB -BQUAA4GBAFQHLSELpa87WCMyXlZ/q1hjSJGqOJCJFvnMv6QOeCufxRtYpuYIjy6u -lwMhm/HNwCaPHdcoJ6COgQkbHA/JpUE6LUQ/nPqH/8hMK0T3G8E+TwF/6SbMnxwG -tQsn0RCQvpMMnOew0eonmU4GFAx66cFSxTNovGEN24E7V0hXv0Ka +MIIC2TCCAkKgAwIBAgIBATANBgkqhkiG9w0BAQUFADBqMQswCQYDVQQGEwJGSTEQ +MA4GA1UECBMHVHV1c3VsYTEZMBcGA1UEChMQTW9udHkgUHJvZ3JhbSBBYjEuMCwG +CSqGSIb3DQEJARYfYWJzdHJhY3QuZGV2ZWxvcGVyQGFza21vbnR5Lm9yZzAeFw0x +MDAxMjgyMTU5MTRaFw0xOTEyMDcyMTU5MTRaMH4xCzAJBgNVBAYTAkZJMRAwDgYD +VQQIEwdUdXVzdWxhMRkwFwYDVQQKExBNb250eSBQcm9ncmFtIEFiMRIwEAYDVQQD +Ewlsb2NhbGhvc3QxLjAsBgkqhkiG9w0BCQEWH2Fic3RyYWN0LmRldmVsb3BlckBh +c2ttb250eS5vcmcwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMx5dCoUfncG +sOwbttpwTE8OlASK52n0TJxnIkFWOpZTnpWdF1WAQcAT1pQPzSxR/qRu8nTYXTs6 +gOOFW6W8fVwlVeVAd/7zy1vN2qX0NmQwoqB/k7PEVnUqwPdlKtfm/4glA+C0QC50 +TMtGlXPLJVyHDuwPWhnCttye6PAwsZyZAgMBAAGjezB5MAkGA1UdEwQCMAAwLAYJ +YIZIAYb4QgENBB8WHU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1Ud +DgQWBBRtEztAUjyvGOoz0bdWIRsF/gueODAfBgNVHSMEGDAWgBTgimjRxRT647vA +MM1sUn/eiCn0cDANBgkqhkiG9w0BAQUFAAOBgQCX22Ujf/EVPB6DrA4KUKAMIrhF +1MohBUc7PQO1bEuNu19Xw8dOcSPPM6N/oD29WHW4NyIWL+ntrpuUKYFuNHnPQb09 +jRfXIhwbWMcPeRNWHejYTuUHP3kb3cQGm8W2AjRDxb/lh63xwYryvsIAHdQnH4fI +gDHsbpeVtIRA0XNCcQ== -----END CERTIFICATE----- === modified file 'mysql-test/std_data/server-key.pem' --- a/mysql-test/std_data/server-key.pem 2009-06-11 16:21:32 +0000 +++ b/mysql-test/std_data/server-key.pem 2010-01-29 10:42:31 +0000 @@ -1,9 +1,15 @@ -----BEGIN RSA PRIVATE KEY----- -MIIBOQIBAAJBALaP5be0hoMTivm/Y8tkLblR0d6re0UfqrVmcxP5pgfVunz6kr03 -4q2H2z62ahJk+O4X4xUGL6iCaL9XjcMEmCcCAwEAAQJASA5VwgNb0CKHiPm0ntOk -hG+54SRX3DmafEy6gRjZIl/bZ/asSLhXUZ+CeohyrQh7BZgYWvykd8pRISL9eKsU -GQIhAOXkUrOtP/EtjyqNluEqZdG+RZi/7p61JS3Ce13Myu+LAiEAy0uMlV34AJpM -b40FPKqlHxw8DD/Dt1iKhNVAg8+LDVUCIFjv7fbJDbW2VG63/Cj8CAwOukoP5rbL -iaicVrHBKrllAiB9+MiaXeopZXNrxDS0jQFYr8Q9yt1aJVFgUkxx4Q9HKQIgZPs0 -KlF3NNNWw78INaAEkyf0IEssnLMsuoCWw0DIOak= +MIICXgIBAAKBgQDMeXQqFH53BrDsG7bacExPDpQEiudp9EycZyJBVjqWU56VnRdV +gEHAE9aUD80sUf6kbvJ02F07OoDjhVulvH1cJVXlQHf+88tbzdql9DZkMKKgf5Oz +xFZ1KsD3ZSrX5v+IJQPgtEAudEzLRpVzyyVchw7sD1oZwrbcnujwMLGcmQIDAQAB +AoGBAMdMYkNZsmJFbVDVOobzCg3Mgc1jrmeBrOKNS8AvUe+QFXRyp3m5B102eOHb +/PmD+hU/5qao9UZzoYkiRM/oRq45jrqJEYwWrX007bKK0F9hnErtC1ImM1nBFVhx +6+6cr+ShUkvtj8+wJ2d5bIccUzGCUfFR5tb5BnePTXK8IVoBAkEA7WGNxHAVKgjS +AzlpHr5fvpivA07hNVJizTwZdWGGYeETilZhkkuMRwREceeohF6ILMf0FTZdFSa/ +8EeLa3icIQJBANyDKFjynKwWy5pyRSz75mVwrEi+4eTQPsCPNWLkbpbEPwqPLYWJ +2VSFkISXF7b7Od48JkQWgiB8/kXqMDEdsXkCQQCzZvj3ryWvoP7nhOoXXBWMPGR4 +gZLe86bMKVGsTsp7CtnzwRj4sbQQr/7yfvvzHmaYQX4M0gtDQwfolomd7YdBAkEA +y24ETuqjNu9grf81aiaJipPDnOjcJOcovSRgr/blPxmUvv0Pld5yLNN7W5a4PgrO +fAMpmi7ZpXcqbP17sBQgoQJAWTDFKAmfHPVdDGZuCw4yceP5d+Tv7ABglZUvpPKx +kAvGN1WBASUuCQJDOIgzl6gvYX07S5p147i9mv7UBWOpvw== -----END RSA PRIVATE KEY----- === modified file 'mysql-test/t/grant.test' --- a/mysql-test/t/grant.test 2009-06-15 15:53:45 +0000 +++ b/mysql-test/t/grant.test 2010-01-29 10:42:31 +0000 @@ -35,7 +35,7 @@ revoke delete on mysqltest.* from mysqlt show grants for mysqltest_1@localhost; grant select on mysqltest.* to mysqltest_1@localhost require NONE; show grants for mysqltest_1@localhost; -grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "MySQL AB"; +grant USAGE on mysqltest.* to mysqltest_1@localhost require cipher "EDH-RSA-DES-CBC3-SHA" AND SUBJECT "testsubject" ISSUER "Monty Program Ab"; show grants for mysqltest_1@localhost; revoke all privileges on mysqltest.* from mysqltest_1@localhost; show grants for mysqltest_1@localhost; === modified file 'mysql-test/t/openssl_1.test' --- a/mysql-test/t/openssl_1.test 2009-05-25 13:00:18 +0000 +++ b/mysql-test/t/openssl_1.test 2010-01-29 10:42:31 +0000 @@ -15,8 +15,10 @@ insert into t1 values (5); grant select on test.* to ssl_user1@localhost require SSL; grant select on test.* to ssl_user2@localhost require cipher "DHE-RSA-AES256-SHA"; -grant select on test.* to ssl_user3@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com"; -grant select on test.* to ssl_user4@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/O=MySQL AB/emailAddress=abstract.mysql.developer(a)mysql.com" ISSUER "/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB"; +grant select on test.* to ssl_user3@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; +grant select on test.* to ssl_user4@localhost require cipher +"DHE-RSA-AES256-SHA" AND SUBJECT "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org" ISSUER "/C=FI/ST=Tuusula/O=Monty Program Ab/emailAddress=abstract.developer(a)askmonty.org"; grant select on test.* to ssl_user5@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "xxx"; flush privileges; === modified file 'mysql-test/t/query_cache_debug.test' --- a/mysql-test/t/query_cache_debug.test 2009-09-07 20:50:10 +0000 +++ b/mysql-test/t/query_cache_debug.test 2010-01-29 10:42:31 +0000 @@ -1,5 +1,4 @@ --source include/not_embedded.inc ---source include/big_test.inc --source include/have_query_cache.inc --source include/have_debug.inc === modified file 'sql-common/client.c' --- a/sql-common/client.c 2009-12-03 15:26:54 +0000 +++ b/sql-common/client.c 2010-01-29 10:42:31 +0000 @@ -1863,6 +1863,7 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons uint port, const char *unix_socket,ulong client_flag) { char buff[NAME_LEN+USERNAME_LENGTH+100]; + char error_string[1024]; char *end,*host_info= NULL; my_socket sock; in_addr_t ip_addr; @@ -2331,9 +2332,14 @@ CLI_MYSQL_REAL_CONNECT(MYSQL *mysql,cons /* Connect to the server */ DBUG_PRINT("info", ("IO layer change in progress...")); if (sslconnect(ssl_fd, mysql->net.vio, - (long) (mysql->options.connect_timeout))) + (long) (mysql->options.connect_timeout), + error_string)) { - set_mysql_error(mysql, CR_SSL_CONNECTION_ERROR, unknown_sqlstate); + set_mysql_extended_error(mysql, CR_SSL_CONNECTION_ERROR, + unknown_sqlstate, + "SSL error: %s", + error_string[0] ? error_string : + ER(CR_SSL_CONNECTION_ERROR)); goto error; } DBUG_PRINT("info", ("IO layer change done!")); === modified file 'sql/net_serv.cc' --- a/sql/net_serv.cc 2010-01-28 14:49:14 +0000 +++ b/sql/net_serv.cc 2010-01-29 10:42:31 +0000 @@ -219,7 +219,7 @@ my_bool net_realloc(NET *net, size_t len -1 Don't know if data is ready or not */ -#if !defined(EMBEDDED_LIBRARY) +#if !defined(EMBEDDED_LIBRARY) && defined(DBUG_OFF) static int net_data_is_ready(my_socket sd) { @@ -280,7 +280,7 @@ static int net_data_is_ready(my_socket s void net_clear(NET *net, my_bool clear_buffer) { -#if !defined(EMBEDDED_LIBRARY) +#if !defined(EMBEDDED_LIBRARY) && defined(DBUG_OFF) size_t count; int ready; #endif === modified file 'sql/slave.cc' --- a/sql/slave.cc 2009-12-06 17:51:48 +0000 +++ b/sql/slave.cc 2010-01-29 10:42:31 +0000 @@ -3803,10 +3803,11 @@ static int connect_to_master(THD* thd, M suppress_warnings= 0; mi->report(ERROR_LEVEL, last_errno, "error %s to master '%s@%s:%d'" - " - retry-time: %d retries: %lu", + " - retry-time: %d retries: %lu message: %s", (reconnect ? "reconnecting" : "connecting"), mi->user, mi->host, mi->port, - mi->connect_retry, master_retry_count); + mi->connect_retry, master_retry_count, + mysql_error(mysql)); } /* By default we try forever. The reason is that failure will trigger === modified file 'sql/sql_cache.cc' --- a/sql/sql_cache.cc 2010-01-15 15:27:55 +0000 +++ b/sql/sql_cache.cc 2010-01-29 10:42:31 +0000 @@ -377,9 +377,11 @@ TODO list: static void debug_wait_for_kill(const char *info) { - DBUG_ENTER("debug_wait_for_kill"); const char *prev_info; THD *thd; + char buff[1024]; + DBUG_ENTER("debug_wait_for_kill"); + thd= current_thd; prev_info= thd->proc_info; thd->proc_info= info; @@ -387,8 +389,16 @@ static void debug_wait_for_kill(const ch while(!thd->killed) my_sleep(1000); thd->killed= THD::NOT_KILLED; + /* + Remove the set debug variable, to ensure we don't get stuck on it again + This is needed as for MyISAM, invalidate_table() may be called twice + (Once from mysql_delete() and once from mi_update_status()) + */ + sprintf(buff, "-d,%s", info); + DBUG_SET(buff); sql_print_information("Exit debug_wait_for_kill"); thd->proc_info= prev_info; + DBUG_VOID_RETURN; } @@ -914,15 +924,18 @@ void query_cache_insert(NET *net, const void query_cache_abort(NET *net) { + THD *thd; DBUG_ENTER("query_cache_abort"); - THD *thd= current_thd; /* See the comment on double-check locking usage above. */ if (net->query_cache_query == 0) DBUG_VOID_RETURN; if (query_cache.try_lock()) + { + net->query_cache_query = 0; DBUG_VOID_RETURN; + } /* While we were waiting another thread might have changed the status @@ -932,6 +945,7 @@ void query_cache_abort(NET *net) net->query_cache_query); if (query_block) { + thd= current_thd; thd_proc_info(thd, "storing result in query cache"); DUMP(&query_cache); BLOCK_LOCK_WR(query_block); @@ -941,6 +955,7 @@ void query_cache_abort(NET *net) DBUG_EXECUTE("check_querycache",query_cache.check_integrity(1);); } + DBUG_ASSERT(!net->query_cache_query); query_cache.unlock(); DBUG_VOID_RETURN; } @@ -970,8 +985,12 @@ void query_cache_end_of_result(THD *thd) #endif if (query_cache.try_lock()) + { + thd->net.query_cache_query= 0; DBUG_VOID_RETURN; + } + /* thd->net.query_cache_query may have changed during resize */ query_block= ((Query_cache_block*) thd->net.query_cache_query); if (query_block) { @@ -997,8 +1016,8 @@ void query_cache_end_of_result(THD *thd) to this function. In the release version that query should be ignored and removed from QC. */ - DBUG_ASSERT(0); query_cache.free_query(query_block); + thd->net.query_cache_query= 0; query_cache.unlock(); DBUG_VOID_RETURN; } === modified file 'sql/sql_connect.cc' --- a/sql/sql_connect.cc 2009-12-03 11:19:05 +0000 +++ b/sql/sql_connect.cc 2010-01-29 10:42:31 +0000 @@ -798,6 +798,7 @@ static int check_connection(THD *thd) DBUG_PRINT("info", ("client capabilities: %lu", thd->client_capabilities)); if (thd->client_capabilities & CLIENT_SSL) { + char error_string[1024]; /* Do the SSL layering. */ if (!ssl_acceptor_fd) { @@ -806,7 +807,7 @@ static int check_connection(THD *thd) return 1; } DBUG_PRINT("info", ("IO layer change in progress...")); - if (sslaccept(ssl_acceptor_fd, net->vio, net->read_timeout)) + if (sslaccept(ssl_acceptor_fd, net->vio, net->read_timeout, error_string)) { DBUG_PRINT("error", ("Failed to accept new SSL connection")); inc_host_errors(&thd->remote.sin_addr); === modified file 'sql/sql_parse.cc' --- a/sql/sql_parse.cc 2010-01-15 15:27:55 +0000 +++ b/sql/sql_parse.cc 2010-01-29 10:42:31 +0000 @@ -1252,12 +1252,14 @@ bool dispatch_command(enum enum_server_c ha_maria::implicit_commit(thd, FALSE); #endif - net_end_statement(thd); - query_cache_end_of_result(thd); /* Multiple queries exits, execute them individually */ close_thread_tables(thd); + + net_end_statement(thd); + query_cache_end_of_result(thd); + ulong length= (ulong)(packet_end - beginning_of_next_stmt); log_slow_statement(thd); @@ -1645,13 +1647,22 @@ bool dispatch_command(enum enum_server_c ha_maria::implicit_commit(thd, FALSE); #endif - net_end_statement(thd); - query_cache_end_of_result(thd); - + if (!(sql_command_flags[thd->lex->sql_command] & CF_CHANGES_DATA)) + { + /* No changes in data; We can send ok at once to the client */ + net_end_statement(thd); + query_cache_end_of_result(thd); + } thd->proc_info= "closing tables"; /* Free tables */ close_thread_tables(thd); + if (sql_command_flags[thd->lex->sql_command] & CF_CHANGES_DATA) + { + net_end_statement(thd); + query_cache_end_of_result(thd); + } + log_slow_statement(thd); thd_proc_info(thd, "cleaning up"); === modified file 'vio/viossl.c' --- a/vio/viossl.c 2008-02-07 08:55:49 +0000 +++ b/vio/viossl.c 2010-01-29 10:42:31 +0000 @@ -74,8 +74,11 @@ report_errors(SSL* ssl) } if (ssl) - DBUG_PRINT("error", ("error: %s", - ERR_error_string(SSL_get_error(ssl, l), buf))); + { + int error= SSL_get_error(ssl, l); + DBUG_PRINT("error", ("error: %s (%d)", + ERR_error_string(error, buf), error)); + } DBUG_PRINT("info", ("socket_errno: %d", socket_errno)); DBUG_VOID_RETURN; @@ -174,16 +177,17 @@ void vio_ssl_delete(Vio *vio) static int ssl_do(struct st_VioSSLFd *ptr, Vio *vio, long timeout, - int (*connect_accept_func)(SSL*)) + int (*connect_accept_func)(SSL*), char *error_string) { SSL *ssl; my_bool unused; my_bool was_blocking; - DBUG_ENTER("ssl_do"); DBUG_PRINT("enter", ("ptr: 0x%lx, sd: %d ctx: 0x%lx", (long) ptr, vio->sd, (long) ptr->ssl_context)); + error_string[0]= 0; + /* Set socket to blocking if not already set */ vio_blocking(vio, 1, &was_blocking); @@ -191,6 +195,7 @@ static int ssl_do(struct st_VioSSLFd *pt { DBUG_PRINT("error", ("SSL_new failure")); report_errors(ssl); + strmov(error_string, "SSL_new failed"); vio_blocking(vio, was_blocking, &unused); DBUG_RETURN(1); } @@ -203,6 +208,7 @@ static int ssl_do(struct st_VioSSLFd *pt { DBUG_PRINT("error", ("SSL_connect/accept failure")); report_errors(ssl); + ERR_error_string(SSL_get_error(ssl, 0), error_string); SSL_free(ssl); vio_blocking(vio, was_blocking, &unused); DBUG_RETURN(1); @@ -250,17 +256,19 @@ static int ssl_do(struct st_VioSSLFd *pt } -int sslaccept(struct st_VioSSLFd *ptr, Vio *vio, long timeout) +int sslaccept(struct st_VioSSLFd *ptr, Vio *vio, long timeout, + char *error_string) { DBUG_ENTER("sslaccept"); - DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_accept)); + DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_accept, error_string)); } -int sslconnect(struct st_VioSSLFd *ptr, Vio *vio, long timeout) +int sslconnect(struct st_VioSSLFd *ptr, Vio *vio, long timeout, + char *error_string) { DBUG_ENTER("sslconnect"); - DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_connect)); + DBUG_RETURN(ssl_do(ptr, vio, timeout, SSL_connect, error_string)); }

1 0

[Maria-developers] Strange compiler error from pbxt
by Michael Widenius 29 Jan '10

29 Jan '10

Hi! On one of the buildbot systems We get the following compiler warning: ha_pbxt.cc:1563: warning: variable 'selfâ' might be clobbered by 'longjmp' I have now disabled this warning in buildbot, but you may want to take a look at this. Regards, Monty

2 1

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2806: Add support in bintar build script for OpenSolaris.
by noreply＠launchpad.net 28 Jan '10

28 Jan '10

------------------------------------------------------------ revno: 2806 committer: knielsen(a)knielsen-hq.org branch nick: work-5.1 timestamp: Thu 2010-01-28 23:34:22 +0100 message: Add support in bintar build script for OpenSolaris. modified: BUILD/util.sh -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2806)
by knielsen＠knielsen-hq.org 28 Jan '10

28 Jan '10

#At lp:maria 2806 knielsen(a)knielsen-hq.org 2010-01-28 Add support in bintar build script for OpenSolaris. modified: BUILD/util.sh === modified file 'BUILD/util.sh' --- a/BUILD/util.sh 2010-01-07 12:02:18 +0000 +++ b/BUILD/util.sh 2010-01-28 22:34:22 +0000 @@ -16,6 +16,8 @@ # Setting cpu options. get_cpuopt () { + case "$(uname -o)" in + *Linux*) case "$(gcc -dumpmachine)" in x86_64-*) # gcc barfs on -march=... on x64 @@ -26,7 +28,13 @@ get_cpuopt () { CPUOPT="-m32 -march=i586 -mtune=generic" ;; esac - return 0 + ;; + *Solaris*) + # ToDo: handle 32-bit build? For now default to 64-bit. + CPUOPT="-D__sun -m64 -mtune=athlon64" + ;; + esac + return 0 } # Default to a parallel build, but only if AM_MAKEFLAGS is not set.

1 0

[Maria-developers] Rev 2750: BUG#31480: Incorrect result for nested subquery when executed via semi join in file:///home/psergey/dev/maria-5.3-subqueries-r3/
by Sergey Petrunya 28 Jan '10

28 Jan '10

At file:///home/psergey/dev/maria-5.3-subqueries-r3/ ------------------------------------------------------------ revno: 2750 revision-id: psergey(a)askmonty.org-20100128152533-q0dicbcuu9fvol5p parent: psergey(a)askmonty.org-20100128134833-9000udjp5wa3tsff committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r3 timestamp: Thu 2010-01-28 18:25:33 +0300 message: BUG#31480: Incorrect result for nested subquery when executed via semi join - Make Item_subselect store its "depth", distance to furthest_correlated_ancestor, and bitmaps of tables that it depends on in ancestor tables. This allows to have Item_subselect to update its attributes when it has been pulled out from a select to its parent. === modified file 'mysql-test/r/subselect_sj.result' --- a/mysql-test/r/subselect_sj.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect_sj.result 2010-01-28 15:25:33 +0000 @@ -779,3 +779,48 @@ 1 PRIMARY it2 ALL NULL NULL NULL NULL 20 Using where; End temporary DROP TABLE ot1, it1, it2; # End of BUG#38075 +# +# BUG#31480: Incorrect result for nested subquery when executed via semi join +# +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +id select_type table type possible_keys key key_len ref rows filtered Extra +1 PRIMARY t2 ALL NULL NULL NULL NULL 6 100.00 Start temporary +1 PRIMARY t1 ALL NULL NULL NULL NULL 7 100.00 Using where; End temporary; Using join buffer +3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where +Warnings: +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +show warnings; +Level Code Message +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +a +2 +2 +3 +2 +drop table t1, t2, t3; === modified file 'mysql-test/r/subselect_sj_jcl6.result' --- a/mysql-test/r/subselect_sj_jcl6.result 2010-01-17 14:51:10 +0000 +++ b/mysql-test/r/subselect_sj_jcl6.result 2010-01-28 15:25:33 +0000 @@ -783,6 +783,51 @@ 1 PRIMARY it2 ALL NULL NULL NULL NULL 20 Using where; End temporary; Using join buffer DROP TABLE ot1, it1, it2; # End of BUG#38075 +# +# BUG#31480: Incorrect result for nested subquery when executed via semi join +# +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +id select_type table type possible_keys key key_len ref rows filtered Extra +1 PRIMARY t2 ALL NULL NULL NULL NULL 6 100.00 Start temporary +1 PRIMARY t1 ALL NULL NULL NULL NULL 7 100.00 Using where; End temporary; Using join buffer +3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where +Warnings: +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +show warnings; +Level Code Message +Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 +Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` semi join (`test`.`t2`) where ((`test`.`t1`.`a` = `test`.`t2`.`c`) and <nop>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select 1 AS `Not_used` from `test`.`t3` where ((`test`.`t1`.`b` = `test`.`t3`.`e`) and (<cache>(`test`.`t2`.`d`) >= `test`.`t3`.`e`)))))) +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +a +2 +2 +3 +2 +drop table t1, t2, t3; set join_cache_level=default; show variables like 'join_cache_level'; Variable_name Value === modified file 'mysql-test/t/subselect_sj.test' --- a/mysql-test/t/subselect_sj.test 2010-01-17 14:51:10 +0000 +++ b/mysql-test/t/subselect_sj.test 2010-01-28 15:25:33 +0000 @@ -681,3 +681,41 @@ DROP TABLE ot1, it1, it2; --echo # End of BUG#38075 + +--echo # +--echo # BUG#31480: Incorrect result for nested subquery when executed via semi join +--echo # +create table t1 (a int not null, b int not null); +create table t2 (c int not null, d int not null); +create table t3 (e int not null); + +insert into t1 values (1,10); +insert into t1 values (2,10); +insert into t1 values (1,20); +insert into t1 values (2,20); +insert into t1 values (3,20); +insert into t1 values (2,30); +insert into t1 values (4,40); + +insert into t2 values (2,10); +insert into t2 values (2,20); +insert into t2 values (4,10); +insert into t2 values (5,10); +insert into t2 values (3,20); +insert into t2 values (2,40); + +insert into t3 values (10); +insert into t3 values (30); +insert into t3 values (10); +insert into t3 values (20); + +explain extended +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); +show warnings; + +select a from t1 +where a in (select c from t2 where d >= some(select e from t3 where b=e)); + +drop table t1, t2, t3; + === modified file 'sql/item.cc' --- a/sql/item.cc 2010-01-17 14:55:08 +0000 +++ b/sql/item.cc 2010-01-28 15:25:33 +0000 @@ -2211,7 +2211,8 @@ } -void Item_field::fix_after_pullout(st_select_lex *new_parent, Item **ref) +void Item_field::fix_after_pullout(st_select_lex *new_parent, + uint parent_tables, Item **ref) { if (new_parent == depended_from) depended_from= NULL; @@ -3648,7 +3649,7 @@ static void mark_as_dependent(THD *thd, SELECT_LEX *last, SELECT_LEX *current, Item_ident *resolved_item, - Item_ident *mark_item) + Item_ident *mark_item, table_map dep_map) { const char *db_name= (resolved_item->db_name ? resolved_item->db_name : ""); @@ -3657,7 +3658,7 @@ /* store pointer on SELECT_LEX from which item is dependent */ if (mark_item) mark_item->depended_from= last; - current->mark_as_dependent(last, resolved_item); + current->mark_as_dependent(last, resolved_item, dep_map); if (thd->lex->describe & DESCRIBE_EXTENDED) { push_warning_printf(thd, MYSQL_ERROR::WARN_LEVEL_NOTE, @@ -3715,21 +3716,26 @@ Item_subselect *prev_subselect_item= previous_select->master_unit()->item; Item_ident *dependent= resolved_item; + table_map found_used_tables; if (found_field == view_ref_found) { Item::Type type= found_item->type(); + found_used_tables= found_item->used_tables(); prev_subselect_item->used_tables_cache|= - found_item->used_tables(); + found_used_tables; dependent= ((type == Item::REF_ITEM || type == Item::FIELD_ITEM) ? (Item_ident*) found_item : 0); } else + { + found_used_tables= found_field->table->map; prev_subselect_item->used_tables_cache|= found_field->table->map; + } prev_subselect_item->const_item_cache= 0; mark_as_dependent(thd, last_select, current_sel, resolved_item, - dependent); + dependent, found_used_tables); } } @@ -4010,6 +4016,7 @@ SELECT_LEX *current_sel= (SELECT_LEX *) thd->lex->current_select; Name_resolution_context *outer_context= 0; SELECT_LEX *select= 0; + uint n_levels= 0; /* Currently derived tables cannot be correlated */ if (current_sel->master_unit()->first_select()->linkage != DERIVED_TABLE_TYPE) @@ -4102,7 +4109,8 @@ context->select_lex, this, ((ref_type == REF_ITEM || ref_type == FIELD_ITEM) ? - (Item_ident*) (*reference) : 0)); + (Item_ident*) (*reference) : 0), + (*from_field)->table->map); return 0; } } @@ -4117,7 +4125,8 @@ context->select_lex, this, ((ref_type == REF_ITEM || ref_type == FIELD_ITEM) ? (Item_ident*) (*reference) : - 0)); + 0), + (*reference)->used_tables()); /* A reference to a view field had been found and we substituted it instead of this Item (find_field_in_tables @@ -4151,6 +4160,7 @@ */ prev_subselect_item->used_tables_cache|= OUTER_REF_TABLE_BIT; prev_subselect_item->const_item_cache= 0; + n_levels++; } DBUG_ASSERT(ref != 0); @@ -4218,14 +4228,15 @@ mark_as_dependent(thd, last_checked_context->select_lex, context->select_lex, this, - rf); + rf, rf->used_tables()); return 0; } else { mark_as_dependent(thd, last_checked_context->select_lex, context->select_lex, - this, (Item_ident*)*reference); + this, (Item_ident*)*reference, + (*reference)->used_tables()); if (last_checked_context->select_lex->having_fix_field) { Item_ref *rf; @@ -5947,7 +5958,8 @@ ((refer_type == REF_ITEM || refer_type == FIELD_ITEM) ? (Item_ident*) (*reference) : - 0)); + 0), + (*reference)->used_tables()); /* view reference found, we substituted it instead of this Item, so can quit @@ -5997,7 +6009,8 @@ goto error; thd->change_item_tree(reference, fld); mark_as_dependent(thd, last_checked_context->select_lex, - thd->lex->current_select, this, fld); + thd->lex->current_select, this, fld, ++ from_field->table->map); /* A reference is resolved to a nest level that's outer or the same as the nest level of the enclosing set function : adjust the value of @@ -6020,7 +6033,8 @@ /* Should be checked in resolve_ref_in_select_and_group(). */ DBUG_ASSERT(*ref && (*ref)->fixed); mark_as_dependent(thd, last_checked_context->select_lex, - context->select_lex, this, this); + context->select_lex, this, this, + (*ref)->used_tables()); /* A reference is resolved to a nest level that's outer or the same as the nest level of the enclosing set function : adjust the value of @@ -6432,20 +6446,22 @@ } -void Item_outer_ref::fix_after_pullout(st_select_lex *new_parent, Item **ref) +void Item_outer_ref::fix_after_pullout(st_select_lex *new_parent, + uint parent_tables, Item **ref) { if (depended_from == new_parent) { *ref= outer_ref; - outer_ref->fix_after_pullout(new_parent, ref); + (*ref)->fix_after_pullout(new_parent, parent_tables, ref); } } -void Item_ref::fix_after_pullout(st_select_lex *new_parent, Item **refptr) +void Item_ref::fix_after_pullout(st_select_lex *new_parent, + uint parent_tables, Item **refptr) { if (depended_from == new_parent) { - (*ref)->fix_after_pullout(new_parent, ref); + (*ref)->fix_after_pullout(new_parent, parent_tables, ref); depended_from= NULL; } } === modified file 'sql/item.h' --- a/sql/item.h 2010-01-28 13:48:33 +0000 +++ b/sql/item.h 2010-01-28 15:25:33 +0000 @@ -569,7 +569,8 @@ Fix after some tables has been pulled out. Basically re-calculate all attributes that are dependent on the tables. */ - virtual void fix_after_pullout(st_select_lex *new_parent, Item **ref) {}; + virtual void fix_after_pullout(st_select_lex *new_parent, uint parent_tables, + Item **ref) {}; /* should be used in case where we are sure that we do not need @@ -1574,7 +1575,8 @@ bool send(Protocol *protocol, String *str_arg); void reset_field(Field *f); bool fix_fields(THD *, Item **); - void fix_after_pullout(st_select_lex *new_parent, Item **ref); + void fix_after_pullout(st_select_lex *new_parent, uint parent_tables, + Item **ref); void make_field(Send_field *tmp_field); int save_in_field(Field *field,bool no_conversions); void save_org_in_field(Field *field); @@ -2343,7 +2345,8 @@ bool send(Protocol *prot, String *tmp); void make_field(Send_field *field); bool fix_fields(THD *, Item **); - void fix_after_pullout(st_select_lex *new_parent, Item **ref); + void fix_after_pullout(st_select_lex *new_parent, uint parent_tables, + Item **ref); int save_in_field(Field *field, bool no_conversions); void save_org_in_field(Field *field); enum Item_result result_type () const { return (*ref)->result_type(); } @@ -2520,7 +2523,8 @@ outer_ref->save_org_in_field(result_field); } bool fix_fields(THD *, Item **); - void fix_after_pullout(st_select_lex *new_parent, Item **ref); + void fix_after_pullout(st_select_lex *new_parent, uint parent_tables, + Item **ref); table_map used_tables() const { return (*ref)->const_item() ? 0 : OUTER_REF_TABLE_BIT; === modified file 'sql/item_cmpfunc.cc' --- a/sql/item_cmpfunc.cc 2010-01-17 14:55:08 +0000 +++ b/sql/item_cmpfunc.cc 2010-01-28 15:25:33 +0000 @@ -3959,7 +3959,8 @@ } -void Item_cond::fix_after_pullout(st_select_lex *new_parent, Item **ref) +void Item_cond::fix_after_pullout(st_select_lex *new_parent, + uint parent_tables, Item **ref) { List_iterator<Item> li(list); Item *item; @@ -3973,7 +3974,7 @@ while ((item=li++)) { table_map tmp_table_map; - item->fix_after_pullout(new_parent, li.ref()); + item->fix_after_pullout(new_parent, parent_tables, li.ref()); item= *li.ref(); used_tables_cache|= item->used_tables(); const_item_cache&= item->const_item(); === modified file 'sql/item_cmpfunc.h' --- a/sql/item_cmpfunc.h 2010-01-17 14:55:08 +0000 +++ b/sql/item_cmpfunc.h 2010-01-28 15:25:33 +0000 @@ -1472,7 +1472,8 @@ bool add_at_head(Item *item) { return list.push_front(item); } void add_at_head(List<Item> *nlist) { list.prepand(nlist); } bool fix_fields(THD *, Item **ref); - void fix_after_pullout(st_select_lex *new_parent, Item **ref); + void fix_after_pullout(st_select_lex *new_parent, uint parent_tables, + Item **ref); enum Type type() const { return COND_ITEM; } List<Item>* argument_list() { return &list; } === modified file 'sql/item_func.cc' --- a/sql/item_func.cc 2010-01-28 13:48:33 +0000 +++ b/sql/item_func.cc 2010-01-28 15:25:33 +0000 @@ -206,7 +206,8 @@ } -void Item_func::fix_after_pullout(st_select_lex *new_parent, Item **ref) +void Item_func::fix_after_pullout(st_select_lex *new_parent, + uint parent_tables, Item **ref) { Item **arg,**arg_end; @@ -217,7 +218,7 @@ { for (arg=args, arg_end=args+arg_count; arg != arg_end ; arg++) { - (*arg)->fix_after_pullout(new_parent, arg); + (*arg)->fix_after_pullout(new_parent, parent_tables, arg); Item *item= *arg; used_tables_cache|= item->used_tables(); === modified file 'sql/item_func.h' --- a/sql/item_func.h 2010-01-28 13:48:33 +0000 +++ b/sql/item_func.h 2010-01-28 15:25:33 +0000 @@ -117,7 +117,8 @@ // Constructor used for Item_cond_and/or (see Item comment) Item_func(THD *thd, Item_func *item); bool fix_fields(THD *, Item **ref); - void fix_after_pullout(st_select_lex *new_parent, Item **ref); + void fix_after_pullout(st_select_lex *new_parent, uint parent_tables, + Item **ref); table_map used_tables() const; table_map not_null_tables() const; void update_used_tables(); === modified file 'sql/item_row.cc' --- a/sql/item_row.cc 2010-01-17 14:55:08 +0000 +++ b/sql/item_row.cc 2010-01-28 15:25:33 +0000 @@ -125,13 +125,14 @@ } -void Item_row::fix_after_pullout(st_select_lex *new_parent, Item **ref) +void Item_row::fix_after_pullout(st_select_lex *new_parent, + uint parent_tables, Item **ref) { used_tables_cache= 0; const_item_cache= 1; for (uint i= 0; i < arg_count; i++) { - items[i]->fix_after_pullout(new_parent, &items[i]); + items[i]->fix_after_pullout(new_parent, parent_tables, &items[i]); used_tables_cache|= items[i]->used_tables(); const_item_cache&= items[i]->const_item(); } === modified file 'sql/item_row.h' --- a/sql/item_row.h 2010-01-17 14:55:08 +0000 +++ b/sql/item_row.h 2010-01-28 15:25:33 +0000 @@ -59,7 +59,8 @@ return 0; }; bool fix_fields(THD *thd, Item **ref); - void fix_after_pullout(st_select_lex *new_parent, Item **ref); + void fix_after_pullout(st_select_lex *new_parent, uint parent_tables, + Item **ref); void cleanup(); void split_sum_func(THD *thd, Item **ref_pointer_array, List<Item> &fields); table_map used_tables() const { return used_tables_cache; }; === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-01-28 13:48:33 +0000 +++ b/sql/item_subselect.cc 2010-01-28 15:25:33 +0000 @@ -39,8 +39,8 @@ Item_subselect::Item_subselect(): Item_result_field(), value_assigned(0), thd(0), substitution(0), engine(0), old_engine(0), used_tables_cache(0), have_to_be_excluded(0), - const_item_cache(1), in_fix_fields(0), engine_changed(0), changed(0), - is_correlated(FALSE) + const_item_cache(1), inside_first_fix_fields(0), in_fix_fields(0), + ancestor_used_tables(0), engine_changed(0), changed(0), is_correlated(FALSE) { with_subselect= 1; reset(); @@ -159,6 +159,15 @@ } +void Item_subselect::set_depth() +{ + uint n= 0; + for (SELECT_LEX *s= unit->first_select(); s; s= s->outer_select()) + n++; + this->depth= n - 1; +} + + bool Item_subselect::fix_fields(THD *thd_param, Item **ref) { char const *save_where= thd_param->where; @@ -169,6 +178,18 @@ engine->set_thd((thd= thd_param)); if (!in_fix_fields) refers_to.empty(); + if (!ancestor_used_tables) + { + set_depth(); + size_t size= (1+depth) * sizeof(table_map); + if (!(ancestor_used_tables= (table_map*) + alloc_root(thd->stmt_arena->mem_root, size))) + return TRUE; + bzero(ancestor_used_tables, size); + furthest_correlated_ancestor= 0; + inside_first_fix_fields= TRUE; + } + eliminated= FALSE; if (check_stack_overrun(thd, STACK_MIN_SIZE, (uchar*)&res)) @@ -179,6 +200,7 @@ // all transformation is done (used by prepared statements) changed= 1; + inside_first_fix_fields= FALSE; if (!res) { @@ -258,6 +280,65 @@ return FALSE; } +void Item_subselect::mark_as_dependent(uint n_levels, table_map dep_map) +{ + if (inside_first_fix_fields) + { + is_correlated= TRUE; + furthest_correlated_ancestor= max(furthest_correlated_ancestor, n_levels); + if (n_levels > 1) + ancestor_used_tables[n_levels - 2] |= dep_map; + } +} + + +/* + Adjust attributes after our parent select has been merged into grandparent + + DESCRIPTION + Subquery is a composite object which may be correlated, that is, it may + have + 1. references to tables of the parent select (i.e. one that has the clause + with the subquery predicate) + 2. references to tables of the grandparent select + 3. references to tables of further ancestors. + + Before the pullout, this item indicates: + - #1 with table bits in used_tables() + - #2 and #3 with OUTER_REF_TABLE_BIT. + + After parent has been merged with grandparent: + - references to parent and grandparent tables should be indicated with + table bits. + - references to greatgrandparent and further ancestors - with + OUTER_REF_TABLE_BIT. + + This is exactly what this function does, based on pre-collected info in + ancestor_used_tables and furthest_correlated_ancestor. +*/ + +void Item_subselect::fix_after_pullout(st_select_lex *new_parent, + uint parent_tables, Item **ref) +{ + used_tables_cache= (used_tables_cache << parent_tables) | + ancestor_used_tables[0]; + for (uint i=0; i < depth; i++) + ancestor_used_tables[i]= ancestor_used_tables[i+1]; + depth--; + + if (furthest_correlated_ancestor) + furthest_correlated_ancestor--; + used_tables_cache &= ~OUTER_REF_TABLE_BIT; + if (furthest_correlated_ancestor > 1) + used_tables_cache |= OUTER_REF_TABLE_BIT; + /* + Don't update const_tables_cache yet as we don't yet know which of the + parent's tables are constant. Parent will call update_used_tables() anyway, + and that will be our chance to update. + */ +} + + bool Item_subselect::walk(Item_processor processor, bool walk_subquery, uchar *argument) { === modified file 'sql/item_subselect.h' --- a/sql/item_subselect.h 2010-01-28 13:48:33 +0000 +++ b/sql/item_subselect.h 2010-01-28 15:25:33 +0000 @@ -68,6 +68,37 @@ /* cache of constant state */ bool const_item_cache; + int inside_first_fix_fields; +public: + /* + Depth of the subquery predicate. + If the subquery predicate is attatched to some clause of the top-level + select, depth will be 1 + If it is attached to a clause in a subquery of the top-level select, depth + will be 2 and so forth. + */ + uint depth; + + /* + Maximum correlation level of the select + - select that has no references to outside will have 0, + - select that references tables in the select it is located will have 1, + - select that has references to tables of its parent select will have 2, + - select that has references to tables of grandparent will have 3 + and so forth. + */ + uint furthest_correlated_ancestor; + /* + This is used_tables() for non-direct ancestors. That is, + - used_tables() shows which tables of the parent select are referred to + from within the subquery, + - ancestor_used_tables[0] shows which tables of the grandparent select are + referred to from within the subquery, + - ancestor_used_tables[1] shows which tables of the great grand parent + select... and so forth. + */ + table_map *ancestor_used_tables; + public: /* References from inside the subquery to the select that this predicate is @@ -92,6 +123,7 @@ Item_subselect(); virtual subs_type substype() { return UNKNOWN_SUBS; } + void set_depth(); /* We need this method, because some compilers do not allow 'this' @@ -117,6 +149,9 @@ return null_value; } bool fix_fields(THD *thd, Item **ref); + void mark_as_dependent(uint n_levels, table_map dep_map); + void fix_after_pullout(st_select_lex *new_parent, uint parent_tables, + Item **ref); virtual bool exec(); virtual void fix_length_and_dec(); table_map used_tables() const; === modified file 'sql/item_sum.cc' --- a/sql/item_sum.cc 2009-10-15 21:38:29 +0000 +++ b/sql/item_sum.cc 2010-01-28 15:25:33 +0000 @@ -350,7 +350,7 @@ sl= sl->master_unit()->outer_select() ) sl->master_unit()->item->with_sum_func= 1; } - thd->lex->current_select->mark_as_dependent(aggr_sel, NULL); + thd->lex->current_select->mark_as_dependent(aggr_sel, NULL, 0); return FALSE; } === modified file 'sql/sql_lex.cc' --- a/sql/sql_lex.cc 2010-01-28 13:48:33 +0000 +++ b/sql/sql_lex.cc 2010-01-28 15:25:33 +0000 @@ -1841,8 +1841,10 @@ 'last' should be reachable from this st_select_lex_node */ -void st_select_lex::mark_as_dependent(st_select_lex *last, Item *dependency) +void st_select_lex::mark_as_dependent(st_select_lex *last, Item *dependency, + table_map dep_map) { + uint n_levels= master_unit()->item->depth; SELECT_LEX *next_to_last; /* Mark all selects from resolved to 1 before select where was @@ -1868,6 +1870,11 @@ } } next_to_last= s; + Item_subselect *subquery_predicate= s->master_unit()->item; + if (subquery_predicate) + subquery_predicate->mark_as_dependent(n_levels, dep_map); + n_levels--; + //psergey-merge-todo: next_to_last should be removed?? } is_correlated= TRUE; this->master_unit()->item->is_correlated= TRUE; === modified file 'sql/sql_lex.h' --- a/sql/sql_lex.h 2010-01-28 13:48:33 +0000 +++ b/sql/sql_lex.h 2010-01-28 15:25:33 +0000 @@ -747,7 +747,8 @@ return master_unit()->return_after_parsing(); } - void mark_as_dependent(st_select_lex *last, Item *dependency); + void mark_as_dependent(st_select_lex *last, Item *dependency, + table_map dep_map); bool set_braces(bool value); bool inc_in_sum_expr(); === modified file 'sql/sql_select.cc' --- a/sql/sql_select.cc 2010-01-28 13:48:33 +0000 +++ b/sql/sql_select.cc 2010-01-28 15:25:33 +0000 @@ -3144,16 +3144,23 @@ } -void fix_list_after_tbl_changes(SELECT_LEX *new_parent, List<TABLE_LIST> *tlist) +void fix_list_after_tbl_changes(SELECT_LEX *new_parent, uint parent_tables, + List<TABLE_LIST> *tlist) { List_iterator<TABLE_LIST> it(*tlist); TABLE_LIST *table; while ((table= it++)) { if (table->on_expr) - table->on_expr->fix_after_pullout(new_parent, &table->on_expr); + { + table->on_expr->fix_after_pullout(new_parent, parent_tables, + &table->on_expr); + } if (table->nested_join) - fix_list_after_tbl_changes(new_parent, &table->nested_join->join_list); + { + fix_list_after_tbl_changes(new_parent, parent_tables, + &table->nested_join->join_list); + } } } @@ -3357,6 +3364,7 @@ /*TODO: also reset the 'with_subselect' there. */ /* n. Adjust the parent_join->tables counter */ + uint parent_tables= parent_join->tables; uint table_no= parent_join->tables; /* n. Walk through child's tables and adjust table->map */ for (tl= subq_lex->leaf_tables; tl; tl= tl->next_leaf, table_no++) @@ -3434,8 +3442,10 @@ Walk through sj nest's WHERE and ON expressions and call item->fix_table_changes() for all items. */ - sj_nest->sj_on_expr->fix_after_pullout(parent_lex, &sj_nest->sj_on_expr); - fix_list_after_tbl_changes(parent_lex, &sj_nest->nested_join->join_list); + sj_nest->sj_on_expr->fix_after_pullout(parent_lex, parent_join->tables, + &sj_nest->sj_on_expr); + fix_list_after_tbl_changes(parent_lex, parent_join->tables, + &sj_nest->nested_join->join_list); /* Unlink the child select_lex so it doesn't show up in EXPLAIN: */

1 1

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2806)
by Michael Widenius 28 Jan '10

28 Jan '10

#At lp:maria based on revid:monty@askmonty.org-20100128113510-bmn8nf14b76l3mof 2806 Michael Widenius 2010-01-28 Fixed compiler warnings & failing test cases When compiling with debug, don't clear buffer in 'net_clear()' - This allows us to easier find bugs in the protocol and also get repeatable test failures in test cases where someone forgot to do --reap modified: client/mysqltest.cc mysql-test/t/partition_innodb_semi_consistent.test sql/net_serv.cc storage/myisam/ft_boolean_search.c storage/myisam/ft_parser.c storage/myisam/ft_stopwords.c support-files/compiler_warnings.supp unittest/mysys/waiting_threads-t.c per-file messages: client/mysqltest.cc Fixed compiler warning mysql-test/t/partition_innodb_semi_consistent.test Added missing --reap (fixes random failure) sql/net_serv.cc When compiling with debug, don't clear buffer in 'net_clear()' - This allows us to easier find bugs in the protocol and also get repeatable test failures in test cases where someone forgot to do --reap storage/myisam/ft_boolean_search.c Fixed compiler warnings storage/myisam/ft_parser.c Fixed compiler warnings storage/myisam/ft_stopwords.c Fixed compiler warnings support-files/compiler_warnings.supp Added a lot of new suppression of not relevant warnings and warnings in systems we are not in charge of unittest/mysys/waiting_threads-t.c Fixed compiler warnings === modified file 'client/mysqltest.cc' --- a/client/mysqltest.cc 2010-01-15 15:27:55 +0000 +++ b/client/mysqltest.cc 2010-01-28 14:49:14 +0000 @@ -657,7 +657,9 @@ public: } while ((bytes= fread(buf, 1, sizeof(buf), m_file)) > 0) - fwrite(buf, 1, bytes, stderr); + if (fwrite(buf, 1, bytes, stderr)) + die("Failed to write to '%s', errno: %d", + m_file_name, errno); if (!lines) { === modified file 'mysql-test/t/partition_innodb_semi_consistent.test' --- a/mysql-test/t/partition_innodb_semi_consistent.test 2009-01-13 22:12:16 +0000 +++ b/mysql-test/t/partition_innodb_semi_consistent.test 2010-01-28 14:49:14 +0000 @@ -187,6 +187,7 @@ SELECT * FROM t1; --echo # Switch to connection con2 connection con2; +--reap SELECT * FROM t1; connection default; === modified file 'sql/net_serv.cc' --- a/sql/net_serv.cc 2009-09-07 20:50:10 +0000 +++ b/sql/net_serv.cc 2010-01-28 14:49:14 +0000 @@ -286,7 +286,12 @@ void net_clear(NET *net, my_bool clear_b #endif DBUG_ENTER("net_clear"); -#if !defined(EMBEDDED_LIBRARY) +/* + We don't do a clear in case of DBUG_OFF to catch bugs + in the protocol handling +*/ + +#if !defined(EMBEDDED_LIBRARY) && defined(DBUG_OFF) if (clear_buffer) { while ((ready= net_data_is_ready(net->vio->sd)) > 0) === modified file 'storage/myisam/ft_boolean_search.c' --- a/storage/myisam/ft_boolean_search.c 2010-01-27 21:53:08 +0000 +++ b/storage/myisam/ft_boolean_search.c 2010-01-28 14:49:14 +0000 @@ -321,7 +321,7 @@ static int _ftb_parse_query(FTB *ftb, uc param->mysql_add_word= ftb_query_add_word; param->mysql_ftparam= (void *)&ftb_param; param->cs= ftb->charset; - param->doc= (char*) query; + param->doc= query; param->length= len; param->flags= 0; param->mode= MYSQL_FTPARSER_FULL_BOOLEAN_INFO; @@ -702,7 +702,7 @@ static int _ftb_check_phrase(FTB *ftb, c param->mysql_add_word= ftb_phrase_add_word; param->mysql_ftparam= (void *)&ftb_param; param->cs= ftb->charset; - param->doc= (char *) document; + param->doc= document; param->length= len; param->flags= 0; param->mode= MYSQL_FTPARSER_WITH_STOPWORDS; @@ -1000,7 +1000,7 @@ float ft_boolean_find_relevance(FT_INFO { if (!ftsi.pos) continue; - param->doc= (char *)ftsi.pos; + param->doc= ftsi.pos; param->length= ftsi.len; if (unlikely(parser->parse(param))) return 0; === modified file 'storage/myisam/ft_parser.c' --- a/storage/myisam/ft_parser.c 2010-01-27 21:53:08 +0000 +++ b/storage/myisam/ft_parser.c 2010-01-28 14:49:14 +0000 @@ -175,7 +175,7 @@ uchar ft_get_word(CHARSET_INFO *cs, cons if ((param->trunc=(doc<end && *doc == FTB_TRUNC))) doc++; - if (((length >= ft_min_word_len && !is_stopword((char*) word->pos, + if (((length >= ft_min_word_len && !is_stopword(word->pos, word->len)) || param->trunc) && length < ft_max_word_len) { @@ -298,7 +298,7 @@ static int ft_parse_internal(MYSQL_FTPAR DBUG_ENTER("ft_parse_internal"); while (ft_simple_get_word(wtree->custom_arg, &doc, end, &w, TRUE)) - if (param->mysql_add_word(param, (char*) w.pos, w.len, 0)) + if (param->mysql_add_word(param, w.pos, w.len, 0)) DBUG_RETURN(1); DBUG_RETURN(0); } @@ -319,7 +319,7 @@ int ft_parse(TREE *wtree, const uchar *d param->mysql_add_word= ft_add_word; param->mysql_ftparam= &my_param; param->cs= wtree->custom_arg; - param->doc= (char*) doc; + param->doc= doc; param->length= doclen; param->mode= MYSQL_FTPARSER_SIMPLE_MODE; DBUG_RETURN(parser->parse(param)); === modified file 'storage/myisam/ft_stopwords.c' --- a/storage/myisam/ft_stopwords.c 2010-01-27 21:53:08 +0000 +++ b/storage/myisam/ft_stopwords.c 2010-01-28 14:49:14 +0000 @@ -20,7 +20,7 @@ typedef struct st_ft_stopwords { - const char * pos; + const uchar* pos; size_t len; } FT_STOPWORD; @@ -30,15 +30,15 @@ static int FT_STOPWORD_cmp(void* cmp_arg FT_STOPWORD *w1, FT_STOPWORD *w2) { return ha_compare_text(default_charset_info, - (uchar *)w1->pos,w1->len, - (uchar *)w2->pos,w2->len,0,0); + w1->pos, w1->len, + w2->pos, w2->len, 0, 0); } static void FT_STOPWORD_free(FT_STOPWORD *w, TREE_FREE action, void *arg __attribute__((unused))) { if (action == free_free) - my_free((uchar*) w->pos, MYF(0)); + my_free((void*) w->pos, MYF(0)); } static int ft_add_stopword(const char *w) === modified file 'support-files/compiler_warnings.supp' --- a/support-files/compiler_warnings.supp 2010-01-28 11:35:10 +0000 +++ b/support-files/compiler_warnings.supp 2010-01-28 14:49:14 +0000 @@ -4,6 +4,7 @@ # integer.cpp: .*control reaches end of non-void function.*: 1288-1427 +integer.cpp: .*no return statement in function returning non-void.*: 1288-1427 DictTabInfo.cpp : .*invalid access to non-static.* DictTabInfo.cpp : .*macro was used incorrectly.* DbdihMain.cpp : .*unused variable.* : 6666-7013 @@ -27,13 +28,17 @@ _flex_tmp.c: .*not enough actual paramet pars0lex.l: .*conversion from 'ulint' to 'int', possible loss of data.* btr/btr0cur\.c: .*value computed is not used.*: 3175-3375 include/buf0buf\.ic: unused parameter ��mtr�� +fil/fil0fil\.c: pointer targets in passing argument.*differ in signedness fil/fil0fil\.c: comparison between signed and unsigned : 3100-3199 fil/fil0fil\.c: unused parameter log/log0recv\.c: unused variable os/os0file\.c: unused parameter +os/os0file\.c: pointer targets in assignment differ in signedness handler/i_s\.cc: unused variable sync/sync0rw\.c: unused parameter sync/sync0sync\.c: unused parameter +sync/sync0sync\.c: unused variable +ut/ut0ut\.c: ignoring return value of # # bdb is not critical to keep up to date @@ -98,10 +103,15 @@ storage/maria/ma_pagecache.c: .*'info_ch # I think these are due to mix of C and C++. # storage/pbxt/ : typedef.*was ignored in this declaration +ha_pbxt\.cc : variable.*might be clobbered by.*longjmp # # Yassl include/runtime.hpp: .*pure_error.* +.*/extra/yassl/taocrypt/.*: comparison with string literal +.*/extra/yassl/taocrypt/src/blowfish\.cpp: array subscript is above array bounds +.*/extra/yassl/taocrypt/src/file\.cpp: ignoring return value +.*/extra/yassl/taocrypt/src/integer\.cpp: control reaches end of non-void function # # Groff warnings on OpenSUSE. === modified file 'unittest/mysys/waiting_threads-t.c' --- a/unittest/mysys/waiting_threads-t.c 2009-09-15 10:46:35 +0000 +++ b/unittest/mysys/waiting_threads-t.c 2010-01-28 14:49:14 +0000 @@ -258,7 +258,7 @@ void do_tests() #define test_kill_strategy(X) \ diag("kill strategy: " #X); \ DBUG_EXECUTE("reset_file", \ - { rewind(DBUG_FILE); ftruncate(fileno(DBUG_FILE), 0); }); \ + { rewind(DBUG_FILE); (void) ftruncate(fileno(DBUG_FILE), 0); }); \ DBUG_PRINT("info", ("kill strategy: " #X)); \ kill_strategy=X; \ do_one_test();

1 0

[Maria-developers] Rev 2749: Subquery optimizations: non-semijoin materialization in file:///home/psergey/dev/maria-5.3-subqueries-r3/
by Sergey Petrunya 28 Jan '10

28 Jan '10

At file:///home/psergey/dev/maria-5.3-subqueries-r3/ ------------------------------------------------------------ revno: 2749 revision-id: psergey(a)askmonty.org-20100128134833-9000udjp5wa3tsff parent: psergey(a)askmonty.org-20100118223223-0sxbi4py9i0guf1l committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: maria-5.3-subqueries-r3 timestamp: Thu 2010-01-28 16:48:33 +0300 message: Subquery optimizations: non-semijoin materialization - Backport into Maria DB 5.3, part 1 Diff too large for email (2209 lines, the limit is 1000).

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2805)
by Michael Widenius 28 Jan '10

28 Jan '10

#At lp:maria based on revid:knielsen@knielsen-hq.org-20100128072046-x28qsbhoudenjj0f 2805 Michael Widenius 2010-01-28 [merge] Merge with fixes for compiler warnings and 2 fixed test cases Fixed some additional compiler warnings from OpenSolaris build. modified: extra/libevent/devpoll.c include/ft_global.h include/maria.h mysql-test/r/show_check.result mysql-test/suite/rpl/t/rpl_rotate_logs.test mysql-test/t/show_check-master.opt mysys/my_file.c sql/item.cc sql/mysqld.cc sql/rpl_record.cc storage/maria/ft_maria.c storage/maria/ma_blockrec.c storage/myisam/ft_boolean_search.c storage/myisam/ft_myisam.c storage/myisam/ft_nlq_search.c storage/myisam/ft_parser.c storage/myisam/ft_stopwords.c storage/myisam/ftdefs.h storage/xtradb/buf/buf0buf.c storage/xtradb/handler/i_s.cc support-files/compiler_warnings.supp per-file messages: extra/libevent/devpoll.c Fixed compiler warning mysys/my_file.c Fixed compiler warning sql/mysqld.cc Fixed compiler warning sql/rpl_record.cc Removed not used variable storage/maria/ma_blockrec.c Fixed compiler warning storage/xtradb/buf/buf0buf.c Fixed compiler warning storage/xtradb/handler/i_s.cc Fixed compiler warning support-files/compiler_warnings.supp Added suppression of compiler warnings in InnoDB/XtraDB Added suppression of compiler warnings that can safely be ignored. === modified file 'extra/libevent/devpoll.c' --- a/extra/libevent/devpoll.c 2010-01-07 13:00:06 +0000 +++ b/extra/libevent/devpoll.c 2010-01-28 11:35:10 +0000 @@ -140,7 +140,7 @@ devpoll_init(struct event_base *base) return (NULL); if (getrlimit(RLIMIT_NOFILE, &rl) == 0 && - rl.rlim_cur != RLIM_INFINITY) + (unsigned long long) rl.rlim_cur != (unsigned long long) RLIM_INFINITY) nfiles = rl.rlim_cur - 1; /* Initialize the kernel queue */ === modified file 'include/ft_global.h' --- a/include/ft_global.h 2009-11-30 13:36:06 +0000 +++ b/include/ft_global.h 2010-01-27 21:53:08 +0000 @@ -62,7 +62,8 @@ void ft_free_stopwords(void); #define FT_SORTED 2 #define FT_EXPAND 4 /* query expansion */ -FT_INFO *ft_init_search(uint,void *, uint, uchar *, uint,CHARSET_INFO *, uchar *); +FT_INFO *ft_init_search(uint,void *, uint, uchar *, size_t, + CHARSET_INFO *, uchar *); my_bool ft_boolean_check_syntax_string(const uchar *); /* Internal symbols for fulltext between maria and MyISAM */ === modified file 'include/maria.h' --- a/include/maria.h 2009-02-19 09:01:25 +0000 +++ b/include/maria.h 2010-01-27 21:53:08 +0000 @@ -462,7 +462,7 @@ void maria_versioning(MARIA_HA *info, my void maria_ignore_trids(MARIA_HA *info); /* fulltext functions */ -FT_INFO *maria_ft_init_search(uint,void *, uint, uchar *, uint, +FT_INFO *maria_ft_init_search(uint,void *, uint, uchar *, size_t, CHARSET_INFO *, uchar *); /* 'Almost-internal' Maria functions */ === modified file 'mysql-test/r/show_check.result' --- a/mysql-test/r/show_check.result 2009-12-15 09:03:24 +0000 +++ b/mysql-test/r/show_check.result 2010-01-27 21:53:08 +0000 @@ -252,13 +252,11 @@ drop table t1; flush tables; show open tables; Database Table In_use Name_locked -mysql general_log 0 0 create table t1(n int); insert into t1 values (1); show open tables; Database Table In_use Name_locked test t1 0 0 -mysql general_log 0 0 drop table t1; create table t1 (a int not null, b VARCHAR(10), INDEX (b) ) AVG_ROW_LENGTH=10 CHECKSUM=1 COMMENT="test" ENGINE=MYISAM MIN_ROWS=10 MAX_ROWS=100 PACK_KEYS=1 DELAY_KEY_WRITE=1 ROW_FORMAT=fixed; show create table t1; === modified file 'mysql-test/suite/rpl/t/rpl_rotate_logs.test' --- a/mysql-test/suite/rpl/t/rpl_rotate_logs.test 2009-03-12 09:48:41 +0000 +++ b/mysql-test/suite/rpl/t/rpl_rotate_logs.test 2010-01-27 21:53:08 +0000 @@ -153,7 +153,7 @@ disable_query_log; while ($1) { #eval means expand $ expressions - eval insert into t3 values($1 + 4); + eval insert HIGH_PRIORITY into t3 values($1 + 4); dec $1; } enable_query_log; === modified file 'mysql-test/t/show_check-master.opt' --- a/mysql-test/t/show_check-master.opt 2009-12-03 11:19:05 +0000 +++ b/mysql-test/t/show_check-master.opt 2010-01-27 21:53:08 +0000 @@ -1 +1 @@ ---log-output=table --slow-query-log --log-long-format --log-queries-not-using-indexes --myisam-recover="" --general-log --general-log-file="" --slow-query-log-file="" +--log-output=file --slow-query-log --log-long-format --log-queries-not-using-indexes --myisam-recover="" --general-log --general-log-file="foo" --slow-query-log-file="" === modified file 'mysys/my_file.c' --- a/mysys/my_file.c 2006-12-23 19:20:40 +0000 +++ b/mysys/my_file.c 2010-01-28 11:35:10 +0000 @@ -51,7 +51,7 @@ static uint set_max_open_files(uint max_ DBUG_PRINT("info", ("rlim_cur: %u rlim_max: %u", (uint) rlimit.rlim_cur, (uint) rlimit.rlim_max)); - if (rlimit.rlim_cur == RLIM_INFINITY) + if ((ulonglong) rlimit.rlim_cur == (ulonglong) RLIM_INFINITY) rlimit.rlim_cur = max_file_limit; if (rlimit.rlim_cur >= max_file_limit) DBUG_RETURN(rlimit.rlim_cur); /* purecov: inspected */ === modified file 'sql/item.cc' --- a/sql/item.cc 2010-01-15 15:27:55 +0000 +++ b/sql/item.cc 2010-01-27 21:53:08 +0000 @@ -7119,7 +7119,7 @@ double Item_cache_decimal::val_real() DBUG_ASSERT(fixed); double res; if (!value_cached && !cache_value()) - return NULL; + return 0.0; my_decimal2double(E_DEC_FATAL_ERROR, &decimal_value, &res); return res; } === modified file 'sql/mysqld.cc' --- a/sql/mysqld.cc 2010-01-15 15:27:55 +0000 +++ b/sql/mysqld.cc 2010-01-28 11:35:10 +0000 @@ -2704,7 +2704,7 @@ static void init_signals(void) { /* Change limits so that we will get a core file */ STRUCT_RLIMIT rl; - rl.rlim_cur = rl.rlim_max = RLIM_INFINITY; + rl.rlim_cur = rl.rlim_max = (rlim_t) RLIM_INFINITY; if (setrlimit(RLIMIT_CORE, &rl) && global_system_variables.log_warnings) sql_print_warning("setrlimit could not change the size of core files to 'infinity'; We may not be able to generate a core file on signals"); } === modified file 'sql/rpl_record.cc' --- a/sql/rpl_record.cc 2009-10-22 00:15:45 +0000 +++ b/sql/rpl_record.cc 2010-01-28 11:35:10 +0000 @@ -366,7 +366,6 @@ int prepare_record(TABLE *const table, */ for (Field **field_ptr= table->field+skip; *field_ptr; ++field_ptr) { - uint32 const mask= NOT_NULL_FLAG | NO_DEFAULT_VALUE_FLAG; Field *const f= *field_ptr; if ((f->flags & NO_DEFAULT_VALUE_FLAG) && (f->real_type() != MYSQL_TYPE_ENUM)) === modified file 'storage/maria/ft_maria.c' --- a/storage/maria/ft_maria.c 2007-07-02 17:45:15 +0000 +++ b/storage/maria/ft_maria.c 2010-01-27 21:53:08 +0000 @@ -22,8 +22,8 @@ #include "ma_ftdefs.h" FT_INFO *maria_ft_init_search(uint flags, void *info, uint keynr, - uchar *query, uint query_len, CHARSET_INFO *cs, - uchar *record) + uchar *query, size_t query_len, + CHARSET_INFO *cs, uchar *record) { FT_INFO *res; if (flags & FT_BOOL) === modified file 'storage/maria/ma_blockrec.c' --- a/storage/maria/ma_blockrec.c 2010-01-06 21:27:53 +0000 +++ b/storage/maria/ma_blockrec.c 2010-01-28 11:35:10 +0000 @@ -1688,7 +1688,8 @@ static my_bool get_head_or_tail_page(MAR if (!page_link.changed) goto crashed; - DBUG_ASSERT((res->buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) == page_type); + DBUG_ASSERT((uint) (res->buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) == + page_type); if (!(dir= find_free_position(page_type == HEAD_PAGE ? info : 0, res->buff, block_size, &res->rownr, &res->length, &res->empty_space))) === modified file 'storage/myisam/ft_boolean_search.c' --- a/storage/myisam/ft_boolean_search.c 2010-01-15 15:27:55 +0000 +++ b/storage/myisam/ft_boolean_search.c 2010-01-27 21:53:08 +0000 @@ -299,7 +299,7 @@ static int ftb_parse_query_internal(MYSQ } -static int _ftb_parse_query(FTB *ftb, uchar *query, uint len, +static int _ftb_parse_query(FTB *ftb, uchar *query, mysql_ft_size_t len, struct st_mysql_ftparser *parser) { MYSQL_FTPARSER_PARAM *param; @@ -540,7 +540,7 @@ static void _ftb_init_index_search(FT_IN FT_INFO * ft_init_boolean_search(MI_INFO *info, uint keynr, uchar *query, - uint query_len, CHARSET_INFO *cs) + mysql_ft_size_t query_len, CHARSET_INFO *cs) { FTB *ftb; FTB_EXPR *ftbe; @@ -679,8 +679,9 @@ static int ftb_check_phrase_internal(MYS -1 is returned if error occurs. */ -static int _ftb_check_phrase(FTB *ftb, const uchar *document, uint len, - FTB_EXPR *ftbe, struct st_mysql_ftparser *parser) +static int _ftb_check_phrase(FTB *ftb, const uchar *document, + mysql_ft_size_t len, + FTB_EXPR *ftbe, struct st_mysql_ftparser *parser) { MY_FTB_PHRASE_PARAM ftb_param; MYSQL_FTPARSER_PARAM *param; === modified file 'storage/myisam/ft_myisam.c' --- a/storage/myisam/ft_myisam.c 2007-07-02 17:45:15 +0000 +++ b/storage/myisam/ft_myisam.c 2010-01-27 21:53:08 +0000 @@ -23,8 +23,8 @@ #include "ftdefs.h" FT_INFO *ft_init_search(uint flags, void *info, uint keynr, - uchar *query, uint query_len, CHARSET_INFO *cs, - uchar *record) + uchar *query, size_t query_len, + CHARSET_INFO *cs, uchar *record) { FT_INFO *res; if (flags & FT_BOOL) === modified file 'storage/myisam/ft_nlq_search.c' --- a/storage/myisam/ft_nlq_search.c 2009-11-30 13:36:06 +0000 +++ b/storage/myisam/ft_nlq_search.c 2010-01-27 21:53:08 +0000 @@ -204,7 +204,8 @@ static int FT_DOC_cmp(void *unused __att FT_INFO *ft_init_nlq_search(MI_INFO *info, uint keynr, uchar *query, - uint query_len, uint flags, uchar *record) + mysql_ft_size_t query_len, uint flags, + uchar *record) { TREE wtree; ALL_IN_ONE aio; === modified file 'storage/myisam/ft_parser.c' --- a/storage/myisam/ft_parser.c 2009-11-30 13:36:06 +0000 +++ b/storage/myisam/ft_parser.c 2010-01-27 21:53:08 +0000 @@ -304,7 +304,7 @@ static int ft_parse_internal(MYSQL_FTPAR } -int ft_parse(TREE *wtree, const uchar *doc, int doclen, +int ft_parse(TREE *wtree, const uchar *doc, mysql_ft_size_t doclen, struct st_mysql_ftparser *parser, MYSQL_FTPARSER_PARAM *param, MEM_ROOT *mem_root) { === modified file 'storage/myisam/ft_stopwords.c' --- a/storage/myisam/ft_stopwords.c 2009-11-29 23:08:56 +0000 +++ b/storage/myisam/ft_stopwords.c 2010-01-27 21:53:08 +0000 @@ -21,7 +21,7 @@ typedef struct st_ft_stopwords { const char * pos; - uint len; + size_t len; } FT_STOPWORD; static TREE *stopwords3=NULL; === modified file 'storage/myisam/ftdefs.h' --- a/storage/myisam/ftdefs.h 2009-11-29 23:08:56 +0000 +++ b/storage/myisam/ftdefs.h 2010-01-27 21:53:08 +0000 @@ -97,8 +97,8 @@ typedef struct st_ft_word { const uchar *pos; - uint len; double weight; + size_t len; } FT_WORD; int is_stopword(const uchar *word, size_t len); @@ -111,7 +111,8 @@ uchar ft_simple_get_word(CHARSET_INFO *, FT_WORD *, my_bool); typedef struct _st_ft_seg_iterator { - uint num, len; + uint num; + mysql_ft_size_t len; HA_KEYSEG *seg; const uchar *rec, *pos; } FT_SEG_ITERATOR; @@ -128,8 +129,9 @@ FT_WORD * _mi_ft_parserecord(MI_INFO *, uint _mi_ft_parse(TREE *, MI_INFO *, uint, const uchar *, MYSQL_FTPARSER_PARAM *, MEM_ROOT *); -FT_INFO *ft_init_nlq_search(MI_INFO *, uint, uchar *, uint, uint, uchar *); -FT_INFO *ft_init_boolean_search(MI_INFO *, uint, uchar *, uint, CHARSET_INFO *); +FT_INFO *ft_init_nlq_search(MI_INFO *, uint, uchar *, mysql_ft_size_t, uint, + uchar *); +FT_INFO *ft_init_boolean_search(MI_INFO *, uint, uchar *, mysql_ft_size_t, CHARSET_INFO *); extern const struct _ft_vft _ft_vft_nlq; int ft_nlq_read_next(FT_INFO *, char *); === modified file 'storage/xtradb/buf/buf0buf.c' --- a/storage/xtradb/buf/buf0buf.c 2010-01-06 12:00:14 +0000 +++ b/storage/xtradb/buf/buf0buf.c 2010-01-28 11:35:10 +0000 @@ -76,10 +76,10 @@ inline void _increment_page_get_statisti block->page.offset, DPAH_SIZE << 3); block_hash_byte = block_hash >> 3; block_hash_offset = (byte) block_hash & 0x07; - if (block_hash_byte < 0 || block_hash_byte >= DPAH_SIZE) - fprintf(stderr, "!!! block_hash_byte = %lu block_hash_offset = %lu !!!\n", block_hash_byte, block_hash_offset); - if (block_hash_offset < 0 || block_hash_offset > 7) - fprintf(stderr, "!!! block_hash_byte = %lu block_hash_offset = %lu !!!\n", block_hash_byte, block_hash_offset); + if (block_hash_byte >= DPAH_SIZE) + fprintf(stderr, "!!! block_hash_byte = %lu block_hash_offset = %lu !!!\n", (unsigned long) block_hash_byte, (unsigned long) block_hash_offset); + if (block_hash_offset > 7) + fprintf(stderr, "!!! block_hash_byte = %lu block_hash_offset = %lu !!!\n", (unsigned long) block_hash_byte, (unsigned long) block_hash_offset); if ((trx->distinct_page_access_hash[block_hash_byte] & ((byte) 0x01 << block_hash_offset)) == 0) trx->distinct_page_access++; trx->distinct_page_access_hash[block_hash_byte] |= (byte) 0x01 << block_hash_offset; === modified file 'storage/xtradb/handler/i_s.cc' --- a/storage/xtradb/handler/i_s.cc 2010-01-15 15:58:25 +0000 +++ b/storage/xtradb/handler/i_s.cc 2010-01-28 11:35:10 +0000 @@ -2992,7 +2992,7 @@ i_s_innodb_admin_command_fill( char** query_str; char* ptr; char quote = '\0'; - char* command_head = "XTRA_"; + const char* command_head = "XTRA_"; DBUG_ENTER("i_s_innodb_admin_command_fill"); === modified file 'support-files/compiler_warnings.supp' --- a/support-files/compiler_warnings.supp 2010-01-06 21:27:53 +0000 +++ b/support-files/compiler_warnings.supp 2010-01-28 11:35:10 +0000 @@ -28,6 +28,12 @@ pars0lex.l: .*conversion from 'ulint' to btr/btr0cur\.c: .*value computed is not used.*: 3175-3375 include/buf0buf\.ic: unused parameter ��mtr�� fil/fil0fil\.c: comparison between signed and unsigned : 3100-3199 +fil/fil0fil\.c: unused parameter +log/log0recv\.c: unused variable +os/os0file\.c: unused parameter +handler/i_s\.cc: unused variable +sync/sync0rw\.c: unused parameter +sync/sync0sync\.c: unused parameter # # bdb is not critical to keep up to date @@ -50,6 +56,11 @@ db_vrfy.c : .*comparison is always false .*/extra/libevent/select\.c : .*comparison between signed and unsigned.* : 270-280 # +# Ignore warnings from system libraries +# +/usr/share/aclocal/audiofile.m4 : .* + +# # Ignore all conversion warnings on windows 64 # (Is safe as we are not yet supporting strings >= 2G) # @@ -102,6 +113,8 @@ include/runtime.hpp: .*pure_error.* # listener.cc : .*conversion from 'SOCKET' to 'int'.* net_serv.cc : .*conversion from 'SOCKET' to 'int'.* +set_var.cc: right-hand operand of comma has no effect : 1000-1400 + # allow a little moving space for the warning below mi_packrec.c : .*result of 32-bit shift implicitly converted to 64 bits.* : 560-600

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2804)
by knielsen＠knielsen-hq.org 28 Jan '10

28 Jan '10

#At lp:maria 2804 knielsen(a)knielsen-hq.org 2010-01-28 [merge] automerge windows fixes. added: mysql-test/t/mysqlbinlog2-master.opt mysql-test/t/mysqlbinlog_row-master.opt mysql-test/t/mysqlbinlog_row_innodb-master.opt mysql-test/t/mysqlbinlog_row_myisam-master.opt mysql-test/t/mysqlbinlog_row_trans-master.opt modified: mysql-test/include/ps_conv.inc mysql-test/mysql-test-run.pl mysql-test/suite/binlog/r/binlog_unsafe.result mysql-test/suite/binlog/t/binlog_unsafe.test mysql-test/suite/maria/r/maria-recover.result mysql-test/suite/maria/t/maria-recover.test mysql-test/suite/parts/inc/partition_timestamp.inc mysql-test/suite/parts/r/partition_recover_myisam.result mysql-test/suite/parts/t/partition_recover_myisam.test mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result mysql-test/suite/rpl/t/rpl_mixed_ddl_dml.test === modified file 'mysql-test/include/ps_conv.inc' --- a/mysql-test/include/ps_conv.inc 2007-07-04 09:20:53 +0000 +++ b/mysql-test/include/ps_conv.inc 2010-01-19 10:36:52 +0000 @@ -30,6 +30,9 @@ --disable_warnings drop table if exists t5 ; --enable_warnings +--disable_query_log +SET TIME_ZONE= '+03:00'; +--enable_query_log set @arg01= 8; set @arg02= 8.0; set @arg03= 80.00000000000e-1; === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2010-01-06 21:27:53 +0000 +++ b/mysql-test/mysql-test-run.pl 2010-01-27 17:41:05 +0000 @@ -3425,7 +3425,7 @@ sub restart_forced_by_test # Return timezone value of tinfo or default value sub timezone { my ($tinfo)= @_; - return $tinfo->{timezone} || "GMT-3"; + return $tinfo->{timezone} || "DEFAULT"; } @@ -3455,7 +3455,11 @@ sub run_testcase ($$) { # Init variables that can change between each test case # ------------------------------------------------------- my $timezone= timezone($tinfo); - $ENV{'TZ'}= $timezone; + if ($timezone ne 'DEFAULT') { + $ENV{'TZ'}= $timezone; + } else { + delete($ENV{'TZ'}); + } mtr_verbose("Setting timezone: $timezone"); if ( ! using_extern() ) === modified file 'mysql-test/suite/binlog/r/binlog_unsafe.result' --- a/mysql-test/suite/binlog/r/binlog_unsafe.result 2010-01-15 15:27:55 +0000 +++ b/mysql-test/suite/binlog/r/binlog_unsafe.result 2010-01-19 10:36:52 +0000 @@ -380,6 +380,7 @@ INSERT INTO t1 VALUES (VERSION()); Warnings: Note 1592 Statement may not be safe to log in statement format. DELETE FROM t1; +SET TIME_ZONE= '+03:00'; SET TIMESTAMP=1000000; INSERT INTO t1 VALUES (CURDATE()), === modified file 'mysql-test/suite/binlog/t/binlog_unsafe.test' --- a/mysql-test/suite/binlog/t/binlog_unsafe.test 2010-01-15 15:27:55 +0000 +++ b/mysql-test/suite/binlog/t/binlog_unsafe.test 2010-01-19 10:36:52 +0000 @@ -422,6 +422,7 @@ DELETE FROM t1; # following following functions depend on the TIMESTAMP variable and # don't generate a warning. +SET TIME_ZONE= '+03:00'; SET TIMESTAMP=1000000; INSERT INTO t1 VALUES (CURDATE()), === modified file 'mysql-test/suite/maria/r/maria-recover.result' --- a/mysql-test/suite/maria/r/maria-recover.result 2009-02-19 09:01:25 +0000 +++ b/mysql-test/suite/maria/r/maria-recover.result 2010-01-27 17:41:05 +0000 @@ -1,6 +1,3 @@ -call mtr.add_suppression("Checking table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Recovering table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Table '.\/mysqltest\/t_corrupted2' is marked as crashed and should be repaired"); select @@global.maria_recover; @@global.maria_recover BACKUP === modified file 'mysql-test/suite/maria/t/maria-recover.test' --- a/mysql-test/suite/maria/t/maria-recover.test 2009-10-09 08:09:24 +0000 +++ b/mysql-test/suite/maria/t/maria-recover.test 2010-01-27 17:41:05 +0000 @@ -2,9 +2,13 @@ --source include/have_maria.inc -call mtr.add_suppression("Checking table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Recovering table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Table '.\/mysqltest\/t_corrupted2' is marked as crashed and should be repaired"); +--disable_query_log +# Note: \\. matches a single period. We use '.' as directory separator to +# account for Unix and Windows variation. +call mtr.add_suppression("Checking table: '\\..mysqltest.t_corrupted2'"); +call mtr.add_suppression("Recovering table: '\\..mysqltest.t_corrupted2'"); +call mtr.add_suppression("Table '\\..mysqltest.t_corrupted2' is marked as crashed and should be repaired"); +--enable_query_log # Note: we're setting an environment variable (not prefixing it by $), # so that the perl code below can access it. === modified file 'mysql-test/suite/parts/inc/partition_timestamp.inc' --- a/mysql-test/suite/parts/inc/partition_timestamp.inc 2009-10-28 07:52:34 +0000 +++ b/mysql-test/suite/parts/inc/partition_timestamp.inc 2010-01-27 17:41:05 +0000 @@ -48,6 +48,7 @@ show create table t3; let $count=12; --echo $count inserts; --disable_query_log +SET TIME_ZONE= '+03:00'; begin; while ($count) { === modified file 'mysql-test/suite/parts/r/partition_recover_myisam.result' --- a/mysql-test/suite/parts/r/partition_recover_myisam.result 2009-08-29 21:29:47 +0000 +++ b/mysql-test/suite/parts/r/partition_recover_myisam.result 2010-01-27 17:41:05 +0000 @@ -1,5 +1,3 @@ -call mtr.add_suppression("./test/t1_will_crash"); -call mtr.add_suppression("Got an error from unknown thread, ha_myisam.cc"); CREATE TABLE t1_will_crash (a INT, KEY (a)) ENGINE=MyISAM; INSERT INTO t1_will_crash VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10), (11); FLUSH TABLES; === modified file 'mysql-test/suite/parts/t/partition_recover_myisam.test' --- a/mysql-test/suite/parts/t/partition_recover_myisam.test 2009-12-03 11:19:05 +0000 +++ b/mysql-test/suite/parts/t/partition_recover_myisam.test 2010-01-27 17:41:05 +0000 @@ -1,7 +1,9 @@ # test the auto-recover (--myisam-recover) of partitioned myisam tables -call mtr.add_suppression("./test/t1_will_crash"); +--disable_query_log +call mtr.add_suppression("..test.t1_will_crash"); call mtr.add_suppression("Got an error from unknown thread, ha_myisam.cc"); +--enable_query_log --source include/have_partition.inc --disable_warnings === modified file 'mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result' --- a/mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result 2008-11-04 17:07:14 +0000 +++ b/mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result 2010-01-19 10:36:52 +0000 @@ -18,6 +18,7 @@ Server_id Host Port Rpl_recovery_rank Ma drop table t1; stop slave; create table t2(id int auto_increment primary key, created datetime); +SET TIME_ZONE= '+03:00'; set timestamp=12345; insert into t2 set created=now(); select * from t2; === modified file 'mysql-test/suite/rpl/t/rpl_mixed_ddl_dml.test' --- a/mysql-test/suite/rpl/t/rpl_mixed_ddl_dml.test 2007-06-27 12:28:02 +0000 +++ b/mysql-test/suite/rpl/t/rpl_mixed_ddl_dml.test 2010-01-19 10:36:52 +0000 @@ -23,6 +23,7 @@ connection master; # Test replication of timestamp create table t2(id int auto_increment primary key, created datetime); +SET TIME_ZONE= '+03:00'; set timestamp=12345; insert into t2 set created=now(); select * from t2; === added file 'mysql-test/t/mysqlbinlog2-master.opt' --- a/mysql-test/t/mysqlbinlog2-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog2-master.opt 2010-01-19 10:36:52 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row-master.opt' --- a/mysql-test/t/mysqlbinlog_row-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_innodb-master.opt' --- a/mysql-test/t/mysqlbinlog_row_innodb-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_innodb-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_myisam-master.opt' --- a/mysql-test/t/mysqlbinlog_row_myisam-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_myisam-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_trans-master.opt' --- a/mysql-test/t/mysqlbinlog_row_trans-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_trans-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (monty:2803)
by Michael Widenius 27 Jan '10

27 Jan '10

#At lp:maria based on revid:hakan@askmonty.org-20100122211921-e177xzarwy7l8j5y 2803 Michael Widenius 2010-01-27 Fixed some compiler warnings and errors in test suite found by buildbot modified: include/ft_global.h include/maria.h mysql-test/r/show_check.result mysql-test/suite/rpl/t/rpl_rotate_logs.test mysql-test/t/show_check-master.opt sql/item.cc storage/maria/ft_maria.c storage/myisam/ft_boolean_search.c storage/myisam/ft_myisam.c storage/myisam/ft_nlq_search.c storage/myisam/ft_parser.c storage/myisam/ft_stopwords.c storage/myisam/ftdefs.h per-file messages: include/ft_global.h Changed string size to 'size_t' (to avoid compiler warnings later) include/maria.h Changed string size to 'size_t' (to avoid compiler warnings later) mysql-test/r/show_check.result Don't use logging to tables, as there is no guarantee in which order tables are showed in 'show open tables' mysql-test/suite/rpl/t/rpl_rotate_logs.test Use HIGH_PRIORITY inserts to avoid problem with concurrent inserts on the slave. mysql-test/t/show_check-master.opt Don't use logging to tables, as there is no guarantee in which order tables are showed in 'show open tables' sql/item.cc Fixed compiler warning storage/maria/ft_maria.c Changed string size to 'size_t' (to avoid compiler warnings later) storage/myisam/ft_boolean_search.c Fixed compiler warnings storage/myisam/ft_myisam.c Changed string size to 'size_t' (to avoid compiler warnings later) storage/myisam/ft_nlq_search.c Changed string size to 'size_t' (to avoid compiler warnings later) storage/myisam/ft_parser.c Changed string size to 'size_t' (to avoid compiler warnings) storage/myisam/ft_stopwords.c Changed string size to 'size_t' (to avoid compiler warnings) storage/myisam/ftdefs.h Changed string size to 'size_t' (to avoid compiler warnings) === modified file 'include/ft_global.h' --- a/include/ft_global.h 2009-11-30 13:36:06 +0000 +++ b/include/ft_global.h 2010-01-27 21:53:08 +0000 @@ -62,7 +62,8 @@ void ft_free_stopwords(void); #define FT_SORTED 2 #define FT_EXPAND 4 /* query expansion */ -FT_INFO *ft_init_search(uint,void *, uint, uchar *, uint,CHARSET_INFO *, uchar *); +FT_INFO *ft_init_search(uint,void *, uint, uchar *, size_t, + CHARSET_INFO *, uchar *); my_bool ft_boolean_check_syntax_string(const uchar *); /* Internal symbols for fulltext between maria and MyISAM */ === modified file 'include/maria.h' --- a/include/maria.h 2009-02-19 09:01:25 +0000 +++ b/include/maria.h 2010-01-27 21:53:08 +0000 @@ -462,7 +462,7 @@ void maria_versioning(MARIA_HA *info, my void maria_ignore_trids(MARIA_HA *info); /* fulltext functions */ -FT_INFO *maria_ft_init_search(uint,void *, uint, uchar *, uint, +FT_INFO *maria_ft_init_search(uint,void *, uint, uchar *, size_t, CHARSET_INFO *, uchar *); /* 'Almost-internal' Maria functions */ === modified file 'mysql-test/r/show_check.result' --- a/mysql-test/r/show_check.result 2009-12-15 09:03:24 +0000 +++ b/mysql-test/r/show_check.result 2010-01-27 21:53:08 +0000 @@ -252,13 +252,11 @@ drop table t1; flush tables; show open tables; Database Table In_use Name_locked -mysql general_log 0 0 create table t1(n int); insert into t1 values (1); show open tables; Database Table In_use Name_locked test t1 0 0 -mysql general_log 0 0 drop table t1; create table t1 (a int not null, b VARCHAR(10), INDEX (b) ) AVG_ROW_LENGTH=10 CHECKSUM=1 COMMENT="test" ENGINE=MYISAM MIN_ROWS=10 MAX_ROWS=100 PACK_KEYS=1 DELAY_KEY_WRITE=1 ROW_FORMAT=fixed; show create table t1; === modified file 'mysql-test/suite/rpl/t/rpl_rotate_logs.test' --- a/mysql-test/suite/rpl/t/rpl_rotate_logs.test 2009-03-12 09:48:41 +0000 +++ b/mysql-test/suite/rpl/t/rpl_rotate_logs.test 2010-01-27 21:53:08 +0000 @@ -153,7 +153,7 @@ disable_query_log; while ($1) { #eval means expand $ expressions - eval insert into t3 values($1 + 4); + eval insert HIGH_PRIORITY into t3 values($1 + 4); dec $1; } enable_query_log; === modified file 'mysql-test/t/show_check-master.opt' --- a/mysql-test/t/show_check-master.opt 2009-12-03 11:19:05 +0000 +++ b/mysql-test/t/show_check-master.opt 2010-01-27 21:53:08 +0000 @@ -1 +1 @@ ---log-output=table --slow-query-log --log-long-format --log-queries-not-using-indexes --myisam-recover="" --general-log --general-log-file="" --slow-query-log-file="" +--log-output=file --slow-query-log --log-long-format --log-queries-not-using-indexes --myisam-recover="" --general-log --general-log-file="foo" --slow-query-log-file="" === modified file 'sql/item.cc' --- a/sql/item.cc 2010-01-15 15:27:55 +0000 +++ b/sql/item.cc 2010-01-27 21:53:08 +0000 @@ -7119,7 +7119,7 @@ double Item_cache_decimal::val_real() DBUG_ASSERT(fixed); double res; if (!value_cached && !cache_value()) - return NULL; + return 0.0; my_decimal2double(E_DEC_FATAL_ERROR, &decimal_value, &res); return res; } === modified file 'storage/maria/ft_maria.c' --- a/storage/maria/ft_maria.c 2007-07-02 17:45:15 +0000 +++ b/storage/maria/ft_maria.c 2010-01-27 21:53:08 +0000 @@ -22,8 +22,8 @@ #include "ma_ftdefs.h" FT_INFO *maria_ft_init_search(uint flags, void *info, uint keynr, - uchar *query, uint query_len, CHARSET_INFO *cs, - uchar *record) + uchar *query, size_t query_len, + CHARSET_INFO *cs, uchar *record) { FT_INFO *res; if (flags & FT_BOOL) === modified file 'storage/myisam/ft_boolean_search.c' --- a/storage/myisam/ft_boolean_search.c 2010-01-15 15:27:55 +0000 +++ b/storage/myisam/ft_boolean_search.c 2010-01-27 21:53:08 +0000 @@ -299,7 +299,7 @@ static int ftb_parse_query_internal(MYSQ } -static int _ftb_parse_query(FTB *ftb, uchar *query, uint len, +static int _ftb_parse_query(FTB *ftb, uchar *query, mysql_ft_size_t len, struct st_mysql_ftparser *parser) { MYSQL_FTPARSER_PARAM *param; @@ -540,7 +540,7 @@ static void _ftb_init_index_search(FT_IN FT_INFO * ft_init_boolean_search(MI_INFO *info, uint keynr, uchar *query, - uint query_len, CHARSET_INFO *cs) + mysql_ft_size_t query_len, CHARSET_INFO *cs) { FTB *ftb; FTB_EXPR *ftbe; @@ -679,8 +679,9 @@ static int ftb_check_phrase_internal(MYS -1 is returned if error occurs. */ -static int _ftb_check_phrase(FTB *ftb, const uchar *document, uint len, - FTB_EXPR *ftbe, struct st_mysql_ftparser *parser) +static int _ftb_check_phrase(FTB *ftb, const uchar *document, + mysql_ft_size_t len, + FTB_EXPR *ftbe, struct st_mysql_ftparser *parser) { MY_FTB_PHRASE_PARAM ftb_param; MYSQL_FTPARSER_PARAM *param; === modified file 'storage/myisam/ft_myisam.c' --- a/storage/myisam/ft_myisam.c 2007-07-02 17:45:15 +0000 +++ b/storage/myisam/ft_myisam.c 2010-01-27 21:53:08 +0000 @@ -23,8 +23,8 @@ #include "ftdefs.h" FT_INFO *ft_init_search(uint flags, void *info, uint keynr, - uchar *query, uint query_len, CHARSET_INFO *cs, - uchar *record) + uchar *query, size_t query_len, + CHARSET_INFO *cs, uchar *record) { FT_INFO *res; if (flags & FT_BOOL) === modified file 'storage/myisam/ft_nlq_search.c' --- a/storage/myisam/ft_nlq_search.c 2009-11-30 13:36:06 +0000 +++ b/storage/myisam/ft_nlq_search.c 2010-01-27 21:53:08 +0000 @@ -204,7 +204,8 @@ static int FT_DOC_cmp(void *unused __att FT_INFO *ft_init_nlq_search(MI_INFO *info, uint keynr, uchar *query, - uint query_len, uint flags, uchar *record) + mysql_ft_size_t query_len, uint flags, + uchar *record) { TREE wtree; ALL_IN_ONE aio; === modified file 'storage/myisam/ft_parser.c' --- a/storage/myisam/ft_parser.c 2009-11-30 13:36:06 +0000 +++ b/storage/myisam/ft_parser.c 2010-01-27 21:53:08 +0000 @@ -304,7 +304,7 @@ static int ft_parse_internal(MYSQL_FTPAR } -int ft_parse(TREE *wtree, const uchar *doc, int doclen, +int ft_parse(TREE *wtree, const uchar *doc, mysql_ft_size_t doclen, struct st_mysql_ftparser *parser, MYSQL_FTPARSER_PARAM *param, MEM_ROOT *mem_root) { === modified file 'storage/myisam/ft_stopwords.c' --- a/storage/myisam/ft_stopwords.c 2009-11-29 23:08:56 +0000 +++ b/storage/myisam/ft_stopwords.c 2010-01-27 21:53:08 +0000 @@ -21,7 +21,7 @@ typedef struct st_ft_stopwords { const char * pos; - uint len; + size_t len; } FT_STOPWORD; static TREE *stopwords3=NULL; === modified file 'storage/myisam/ftdefs.h' --- a/storage/myisam/ftdefs.h 2009-11-29 23:08:56 +0000 +++ b/storage/myisam/ftdefs.h 2010-01-27 21:53:08 +0000 @@ -97,8 +97,8 @@ typedef struct st_ft_word { const uchar *pos; - uint len; double weight; + size_t len; } FT_WORD; int is_stopword(const uchar *word, size_t len); @@ -111,7 +111,8 @@ uchar ft_simple_get_word(CHARSET_INFO *, FT_WORD *, my_bool); typedef struct _st_ft_seg_iterator { - uint num, len; + uint num; + mysql_ft_size_t len; HA_KEYSEG *seg; const uchar *rec, *pos; } FT_SEG_ITERATOR; @@ -128,8 +129,9 @@ FT_WORD * _mi_ft_parserecord(MI_INFO *, uint _mi_ft_parse(TREE *, MI_INFO *, uint, const uchar *, MYSQL_FTPARSER_PARAM *, MEM_ROOT *); -FT_INFO *ft_init_nlq_search(MI_INFO *, uint, uchar *, uint, uint, uchar *); -FT_INFO *ft_init_boolean_search(MI_INFO *, uint, uchar *, uint, CHARSET_INFO *); +FT_INFO *ft_init_nlq_search(MI_INFO *, uint, uchar *, mysql_ft_size_t, uint, + uchar *); +FT_INFO *ft_init_boolean_search(MI_INFO *, uint, uchar *, mysql_ft_size_t, CHARSET_INFO *); extern const struct _ft_vft _ft_vft_nlq; int ft_nlq_read_next(FT_INFO *, char *);

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2794)
by knielsen＠knielsen-hq.org 27 Jan '10

27 Jan '10

#At lp:maria 2794 knielsen(a)knielsen-hq.org 2010-01-27 [merge] automerge for buildbot test added: mysql-test/t/mysqlbinlog_row-master.opt mysql-test/t/mysqlbinlog_row_innodb-master.opt mysql-test/t/mysqlbinlog_row_myisam-master.opt mysql-test/t/mysqlbinlog_row_trans-master.opt modified: mysql-test/mysql-test-run.pl mysql-test/suite/maria/r/maria-recover.result mysql-test/suite/maria/t/maria-recover.test mysql-test/suite/parts/inc/partition_timestamp.inc mysql-test/suite/parts/r/partition_recover_myisam.result mysql-test/suite/parts/t/partition_recover_myisam.test === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2010-01-19 10:36:52 +0000 +++ b/mysql-test/mysql-test-run.pl 2010-01-27 17:41:05 +0000 @@ -3457,6 +3457,8 @@ sub run_testcase ($$) { my $timezone= timezone($tinfo); if ($timezone ne 'DEFAULT') { $ENV{'TZ'}= $timezone; + } else { + delete($ENV{'TZ'}); } mtr_verbose("Setting timezone: $timezone"); === modified file 'mysql-test/suite/maria/r/maria-recover.result' --- a/mysql-test/suite/maria/r/maria-recover.result 2009-02-19 09:01:25 +0000 +++ b/mysql-test/suite/maria/r/maria-recover.result 2010-01-27 17:41:05 +0000 @@ -1,6 +1,3 @@ -call mtr.add_suppression("Checking table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Recovering table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Table '.\/mysqltest\/t_corrupted2' is marked as crashed and should be repaired"); select @@global.maria_recover; @@global.maria_recover BACKUP === modified file 'mysql-test/suite/maria/t/maria-recover.test' --- a/mysql-test/suite/maria/t/maria-recover.test 2009-10-09 08:09:24 +0000 +++ b/mysql-test/suite/maria/t/maria-recover.test 2010-01-27 17:41:05 +0000 @@ -2,9 +2,13 @@ --source include/have_maria.inc -call mtr.add_suppression("Checking table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Recovering table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Table '.\/mysqltest\/t_corrupted2' is marked as crashed and should be repaired"); +--disable_query_log +# Note: \\. matches a single period. We use '.' as directory separator to +# account for Unix and Windows variation. +call mtr.add_suppression("Checking table: '\\..mysqltest.t_corrupted2'"); +call mtr.add_suppression("Recovering table: '\\..mysqltest.t_corrupted2'"); +call mtr.add_suppression("Table '\\..mysqltest.t_corrupted2' is marked as crashed and should be repaired"); +--enable_query_log # Note: we're setting an environment variable (not prefixing it by $), # so that the perl code below can access it. === modified file 'mysql-test/suite/parts/inc/partition_timestamp.inc' --- a/mysql-test/suite/parts/inc/partition_timestamp.inc 2009-10-28 07:52:34 +0000 +++ b/mysql-test/suite/parts/inc/partition_timestamp.inc 2010-01-27 17:41:05 +0000 @@ -48,6 +48,7 @@ show create table t3; let $count=12; --echo $count inserts; --disable_query_log +SET TIME_ZONE= '+03:00'; begin; while ($count) { === modified file 'mysql-test/suite/parts/r/partition_recover_myisam.result' --- a/mysql-test/suite/parts/r/partition_recover_myisam.result 2009-08-29 21:29:47 +0000 +++ b/mysql-test/suite/parts/r/partition_recover_myisam.result 2010-01-27 17:41:05 +0000 @@ -1,5 +1,3 @@ -call mtr.add_suppression("./test/t1_will_crash"); -call mtr.add_suppression("Got an error from unknown thread, ha_myisam.cc"); CREATE TABLE t1_will_crash (a INT, KEY (a)) ENGINE=MyISAM; INSERT INTO t1_will_crash VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10), (11); FLUSH TABLES; === modified file 'mysql-test/suite/parts/t/partition_recover_myisam.test' --- a/mysql-test/suite/parts/t/partition_recover_myisam.test 2009-12-03 11:19:05 +0000 +++ b/mysql-test/suite/parts/t/partition_recover_myisam.test 2010-01-27 17:41:05 +0000 @@ -1,7 +1,9 @@ # test the auto-recover (--myisam-recover) of partitioned myisam tables -call mtr.add_suppression("./test/t1_will_crash"); +--disable_query_log +call mtr.add_suppression("..test.t1_will_crash"); call mtr.add_suppression("Got an error from unknown thread, ha_myisam.cc"); +--enable_query_log --source include/have_partition.inc --disable_warnings === added file 'mysql-test/t/mysqlbinlog_row-master.opt' --- a/mysql-test/t/mysqlbinlog_row-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_innodb-master.opt' --- a/mysql-test/t/mysqlbinlog_row_innodb-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_innodb-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_myisam-master.opt' --- a/mysql-test/t/mysqlbinlog_row_myisam-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_myisam-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_trans-master.opt' --- a/mysql-test/t/mysqlbinlog_row_trans-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_trans-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2803)
by knielsen＠knielsen-hq.org 27 Jan '10

27 Jan '10

#At lp:maria 2803 knielsen(a)knielsen-hq.org 2010-01-27 Fix test failures due to previous change of not setting TZ by default. Fix by explicitly setting timezone for a few more tests that need it. (We avoid setting TZ everywhere by default as this breaks some tests on windows). Also add fix of two other windows failures due to non-portable suppressions, thanks to Alex Budovski. added: mysql-test/t/mysqlbinlog_row-master.opt mysql-test/t/mysqlbinlog_row_innodb-master.opt mysql-test/t/mysqlbinlog_row_myisam-master.opt mysql-test/t/mysqlbinlog_row_trans-master.opt modified: mysql-test/mysql-test-run.pl mysql-test/suite/maria/r/maria-recover.result mysql-test/suite/maria/t/maria-recover.test mysql-test/suite/parts/inc/partition_timestamp.inc mysql-test/suite/parts/r/partition_recover_myisam.result mysql-test/suite/parts/t/partition_recover_myisam.test per-file messages: mysql-test/mysql-test-run.pl Don't let --timezone option from one test stray into the next test (mostly to make results more deterministic rather than depend on whatever test happens to run just before). mysql-test/suite/maria/r/maria-recover.result Fix suppression pattern to also work with Windows \ path separator. mysql-test/suite/maria/t/maria-recover.test Fix suppression pattern to also work with Windows \ path separator. mysql-test/suite/parts/inc/partition_timestamp.inc Set timezone explicitly for test that needs it. mysql-test/suite/parts/r/partition_recover_myisam.result Fix suppression pattern to also work with Windows \ path separator. mysql-test/suite/parts/t/partition_recover_myisam.test Fix suppression pattern to also work with Windows \ path separator. mysql-test/t/mysqlbinlog_row-master.opt Set timezone explicitly for test that needs it. mysql-test/t/mysqlbinlog_row_innodb-master.opt Set timezone explicitly for test that needs it. mysql-test/t/mysqlbinlog_row_myisam-master.opt Set timezone explicitly for test that needs it. mysql-test/t/mysqlbinlog_row_trans-master.opt Set timezone explicitly for test that needs it. === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2010-01-19 10:36:52 +0000 +++ b/mysql-test/mysql-test-run.pl 2010-01-27 17:41:05 +0000 @@ -3457,6 +3457,8 @@ sub run_testcase ($$) { my $timezone= timezone($tinfo); if ($timezone ne 'DEFAULT') { $ENV{'TZ'}= $timezone; + } else { + delete($ENV{'TZ'}); } mtr_verbose("Setting timezone: $timezone"); === modified file 'mysql-test/suite/maria/r/maria-recover.result' --- a/mysql-test/suite/maria/r/maria-recover.result 2009-02-19 09:01:25 +0000 +++ b/mysql-test/suite/maria/r/maria-recover.result 2010-01-27 17:41:05 +0000 @@ -1,6 +1,3 @@ -call mtr.add_suppression("Checking table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Recovering table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Table '.\/mysqltest\/t_corrupted2' is marked as crashed and should be repaired"); select @@global.maria_recover; @@global.maria_recover BACKUP === modified file 'mysql-test/suite/maria/t/maria-recover.test' --- a/mysql-test/suite/maria/t/maria-recover.test 2009-10-09 08:09:24 +0000 +++ b/mysql-test/suite/maria/t/maria-recover.test 2010-01-27 17:41:05 +0000 @@ -2,9 +2,13 @@ --source include/have_maria.inc -call mtr.add_suppression("Checking table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Recovering table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Table '.\/mysqltest\/t_corrupted2' is marked as crashed and should be repaired"); +--disable_query_log +# Note: \\. matches a single period. We use '.' as directory separator to +# account for Unix and Windows variation. +call mtr.add_suppression("Checking table: '\\..mysqltest.t_corrupted2'"); +call mtr.add_suppression("Recovering table: '\\..mysqltest.t_corrupted2'"); +call mtr.add_suppression("Table '\\..mysqltest.t_corrupted2' is marked as crashed and should be repaired"); +--enable_query_log # Note: we're setting an environment variable (not prefixing it by $), # so that the perl code below can access it. === modified file 'mysql-test/suite/parts/inc/partition_timestamp.inc' --- a/mysql-test/suite/parts/inc/partition_timestamp.inc 2009-10-28 07:52:34 +0000 +++ b/mysql-test/suite/parts/inc/partition_timestamp.inc 2010-01-27 17:41:05 +0000 @@ -48,6 +48,7 @@ show create table t3; let $count=12; --echo $count inserts; --disable_query_log +SET TIME_ZONE= '+03:00'; begin; while ($count) { === modified file 'mysql-test/suite/parts/r/partition_recover_myisam.result' --- a/mysql-test/suite/parts/r/partition_recover_myisam.result 2009-08-29 21:29:47 +0000 +++ b/mysql-test/suite/parts/r/partition_recover_myisam.result 2010-01-27 17:41:05 +0000 @@ -1,5 +1,3 @@ -call mtr.add_suppression("./test/t1_will_crash"); -call mtr.add_suppression("Got an error from unknown thread, ha_myisam.cc"); CREATE TABLE t1_will_crash (a INT, KEY (a)) ENGINE=MyISAM; INSERT INTO t1_will_crash VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10), (11); FLUSH TABLES; === modified file 'mysql-test/suite/parts/t/partition_recover_myisam.test' --- a/mysql-test/suite/parts/t/partition_recover_myisam.test 2009-12-03 11:19:05 +0000 +++ b/mysql-test/suite/parts/t/partition_recover_myisam.test 2010-01-27 17:41:05 +0000 @@ -1,7 +1,9 @@ # test the auto-recover (--myisam-recover) of partitioned myisam tables -call mtr.add_suppression("./test/t1_will_crash"); +--disable_query_log +call mtr.add_suppression("..test.t1_will_crash"); call mtr.add_suppression("Got an error from unknown thread, ha_myisam.cc"); +--enable_query_log --source include/have_partition.inc --disable_warnings === added file 'mysql-test/t/mysqlbinlog_row-master.opt' --- a/mysql-test/t/mysqlbinlog_row-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_innodb-master.opt' --- a/mysql-test/t/mysqlbinlog_row_innodb-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_innodb-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_myisam-master.opt' --- a/mysql-test/t/mysqlbinlog_row_myisam-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_myisam-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_trans-master.opt' --- a/mysql-test/t/mysqlbinlog_row_trans-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_trans-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3

1 0