- developers - lists.mariadb.org

[Maria-developers] Reviews of parallel replications
by Michael Widenius 10 Dec '13

10 Dec '13

Hi! Now continuing with reviews of the parallel replication code. Will take one patch at a time with a target to have everything reviewed this week. Review of revno 3678: Parallel replication: error handling. ------------------------------------------------------------ revno: 3678 committer: knielsen(a)knielsen-hq.org branch nick: work-10.0-mdev4506 timestamp: Mon 2013-10-14 15:28:16 +0200 message: MDEV-4506: Parallel replication: error handling. Add an error code to the wait_for_commit facility. Now, when a transaction fails, it can signal the error to any subsequent transaction that is waiting for it to commit. The waiting transactions then receive the error code back from wait_for_prior_commit() and can handle the error appropriately. Also fix one race that could cause crash if @@slave_parallel_threads were changed several times quickly in succession. <cut> === modified file 'sql/handler.cc' --- sql/handler.cc 2013-10-13 21:24:05 +0000 +++ sql/handler.cc 2013-10-14 13:28:16 +0000 @@ -1458,10 +1458,11 @@ int ha_commit_one_phase(THD *thd, bool a transaction.all.ha_list, see why in trans_register_ha()). */ bool is_real_trans=all || thd->transaction.all.ha_list == 0; + int res; DBUG_ENTER("ha_commit_one_phase"); - if (is_real_trans) - thd->wait_for_prior_commit(); - int res= commit_one_phase_2(thd, all, trans, is_real_trans); + if (is_real_trans && (res= thd->wait_for_prior_commit())) + DBUG_RETURN(res); + res= commit_one_phase_2(thd, all, trans, is_real_trans); DBUG_RETURN(res); Another way to write the above is: if (is_real_trans && !(res= thd->wait_for_prior_commit())) res= commit_one_phase_2(thd, all, trans, is_real_trans); DBUG_RETURN(res); <cut> I was checking how ha_commit_one_phase() was used. In ha_commit_trans() or trans_xa_commit(), if ha_commit_one_phase() fails, it will not run rollback. Is this right? I would have assumed that on failure, ha_commit_one_phase() should run ha_rollback_trans() before returning. === modified file 'sql/log.cc' @@ -7844,7 +7844,8 @@ int TC_LOG_MMAP::log_and_order(THD *thd, mysql_mutex_unlock(&LOCK_prepare_ordered); } - thd->wait_for_prior_commit(); + if (thd->wait_for_prior_commit()) + return 0; What does the return 0 mean here? Is it an error? Is thd->is_fatal_error assumed to be set if this happens? A comment here would not be totally wrong... @@ -5633,7 +5634,7 @@ wait_for_commit::~wait_for_commit() void -wait_for_commit::wakeup() +wait_for_commit::wakeup(int wakeup_error) { /* We signal each waiter on their own condition and mutex (rather than using @@ -5649,6 +5650,7 @@ wait_for_commit::wakeup() */ mysql_mutex_lock(&LOCK_wait_commit); waiting_for_commit= false; + this->wakeup_error= wakeup_error; Consider adding an assert that 'this->wakeup_error' is 0. Good way to ensure we don't accidently call this function twice... (as things would be bad if we first would call it with an error and then with 0) mysql_mutex_unlock(&LOCK_wait_commit); mysql_cond_signal(&COND_wait_commit); } Regards, Monty

1 0

[Maria-developers] (MDEV-4403) Attempting to use cassandra storage engine causes "service 'my_snprintf_service' interface version mismatch"
by Sergei Golubchik 10 Dec '13

10 Dec '13

Hi. I don't see my commit emails on the list, so here you are: first patch is the bug fix itself. second - cleanup of the existing code after the bug fix. Regards, Sergei

2 1

[Maria-developers] Coverity
by Christian Convey 09 Dec '13

09 Dec '13

Hi guys, I'm afraid I won't have the time available that I'd hope to, for analyzing MariaDB with Coverity Prevent. Would anyone like to take over from me the Coverity MariaDB account? Thanks, Christian

2 1

[Maria-developers] plugin doesn't load
by AugustQ 06 Dec '13

06 Dec '13

Hi, just one question: I have a simple plugin (a minimal storage engine) that does not load. When I try to load it I get this error-message: MariaDB [(none)]> install plugin dummy soname 'ha_dummy'; ERROR 1126 (HY000): Can't open shared library '/home/august/MariaDB/pgm/lib/plugin/ha_dummy.so' (errno: 2 undefined symbol: _ZN9ha_myisam5cloneEPKcP11st_mem_root) Can you please give me a tip. Thanks. Environment is MariaDB 10.0.4. The same code works without any problems in MySQL (5.5.8 - just for a test). The code is a minimal engine that just does nothing, I want to use it as a template for further work. this is the code in the form that it is working (the relevant parts): in the header-file: class ha_dummy: public handler in the code: ha_dummy::ha_dummy(handlerton *hton, TABLE_SHARE *table_arg) :handler(hton, table_arg) now I change this by modifying the inheritance-order: in the header file: added: #include "../myisam/ha_myisam.h" #include "../myisam/myisamdef.h" modified: class ha_dummy: public ha_myisam in the code: /* Constructor */ ha_dummy::ha_dummy(handlerton *hton, TABLE_SHARE *table_arg) :ha_myisam(hton, table_arg) It compiles but I cannot load it. What's the problem with st_mem_root? Thanks AugustQ PS: as written before: both versions work with MySQL.

3 4

Re: [Maria-developers] [Commits] Rev 3732: Fixed bug mdev-5382 in file:///home/igor/maria/maria-5.3-mdev5382/
by Oleksandr Byelkin 05 Dec '13

05 Dec '13

Hi! 05.12.2013 21:13, Igor Babaev пишет: > At file:///home/igor/maria/maria-5.3-mdev5382/ > > ------------------------------------------------------------ > revno: 3732 > revision-id: igor(a)askmonty.org-20131205191320-pfja761po2x9p9ri > parent: bar(a)mnogosearch.org-20131203110843-1bsdu0hcrpzhie8w > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-mdev5382 > timestamp: Thu 2013-12-05 11:13:20 -0800 > message: > Fixed bug mdev-5382 > When marking used columns the function find_field_in_table_ref() erroneously > called the walk method for the real item behind a view/derived table field > with the second parameter set to TRUE. > This erroneous code was introduced in 2006. OK to push! > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] RQG crashes tokudb due to unexpected storage engine API call sequence
by Rich Prohaska 05 Dec '13

05 Dec '13

We are debugging a crash induced by RQG in the tokudb storage engine. The cause of the crash is a mismatch in assumptions about the sequence of storage engine API calls. We see: tokudb::external_lock creates a txn tokudb::index_init creates a cursor on a index db with the txn tokudb::index_read use the cursor to try to read a row, and returns an error tokudb_commit commits the txn (called from sql_parse.cc:4584) tokudb::index_end tries to close the cursor and crashes since the txn has been committed. (called from the unit.cleanup() call at sql_parse.cc:4592) The current tokudb software does not expect the commit to occur before index_end is called. Is this assumption correct? Since mariadb (and i suspect mysql) does this, it appears that the storage engine should expect commit before index_end. Any comments? Thanks Rich Prohaska

2 2

[Maria-developers] MDEV-5345 - Deadlock between mysql_change_user(), SHOW VARIABLES and INSTALL PLUGIN
by Sergey Vojtovich 04 Dec '13

04 Dec '13

Hi Sergei, I'm afraid I couldn't find easy fix for this deadlock. The question is if it is worth to spend time for more complex (and possibly less stable) solution. Details below. Locking profile --------------- INSTALL PLUGIN: LOCK_plugin -> LOCK_system_variables_hash (w) SHOW VARIABLES: LOCK_system_variables_hash (r) -> LOCK_global_system_variables change_user(): LOCK_global_system_variables -> LOCK_plugin INSTALL PLUGIN -------------- Needs LOCK_plugin to register new plugin. Needs LOCK_system_variables_hash to register plugin variables. Doesn't seem to need both locks at the same time. Similar lock order is used in a few places. SHOW VARIABLES -------------- Needs LOCK_system_variables_hash to iterate system_variable_hash. Needs LOCK_global_system_variables to read variable value. Does seem to need both locks at the same time. This lock order is not used in other places. change_user() ------------- Needs LOCK_global_system_variables to read global_system_variable.table_plugin. Needs LOCK_plugin to my_plugin_lock(table_plugin). Does seem to need both locks at the same time. Similar lock order is used in a few places. "SHOW VARIABLES" looks correct. So I have only two ideas: - Fix INSTALL PLUGIN so it releases LOCK_plugin before registering variables. Sounds like the best solution, but there are a few more things to fix, e.g. UNINTALL PLUGIN. - Store default storage engine as a string, not Sys_var_plugin. Thanks, Sergey

2 3

[Maria-developers] Build fail of MariaDB10 with TokuDB enabled
by darx＠sent.com 03 Dec '13

03 Dec '13

On the (eventual) road to successfully building MariaDB10 + TokuDB, Building cd src/mariadb bzr log | head ------------------------------------------------------------ revno: 3911 committer: Alexander Barkov <bar(a)mnogosearch.org> branch nick: maria-10.0 timestamp: Tue 2013-11-26 10:53:21 +0400 message: Fixing malformed data in mysql-test/std_data/Index.xml ------------------------------------------------------------ revno: 3910 committer: Sergey Vojtovich <svoj(a)mariadb.org> on uname -a Linux dev1 3.7.10-1.16-desktop #1 SMP PREEMPT Fri May 31 20:21:23 UTC 2013 (97c14ba) x86_64 x86_64 x86_64 GNU/Linux lsb_release -ar LSB Version: core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch Distributor ID: openSUSE project Description: openSUSE 12.3 (x86_64) Release: 12.3 Codename: Dartmouth with gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib64/gcc/x86_64-suse-linux/4.8/lto-wrapper Target: x86_64-suse-linux Configured with: ../configure --prefix=/usr --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,java,ada --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.8 --enable-ssp --disable-libssp --disable-plugin --with-bugurl=http://bugs.opensuse.org/ --with-pkgversion='SUSE Linux' --disable-libgcj --disable-libmudflap --with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit --enable-libstdcxx-allocator=new --disable-libstdcxx-pch --enable-version-specific-runtime-libs --enable-linker-build-id --enable-linux-futex --program-suffix=-4.8 --without-system-libunwind --with-arch-32=i586 --with-tune=generic --build=x86_64-suse-linux --host=x86_64-suse-linux Thread model: posix gcc version 4.8.2 20131016 [gcc-4_8-branch revision 203692] (SUSE Linux) cmake --version cmake version 2.8.12.1 applying a needed src patch cd /usr/local/src/mariadb # BUG @ http://www.linuxfromscratch.org/blfs/view/svn/server/mariadb.html sed -i "s/srv_buf_size/srv_sort_buf_size/" storage/innobase/row/row0log.cc after cmake, with gcc c++11 support NOT enabled, mkdir bld cd bld cmake .. \ ... \ -DWITHOUT_TOKUDB=1 build completes successfully, make ... ldd sql/mysqld linux-vdso.so.1 (0x00007fffb9dff000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9031e86000) libaio.so.1 => /lib64/libaio.so.1 (0x00007f9031c84000) libz.so.1 => /lib64/libz.so.1 (0x00007f9031a6e000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f9031832000) libssl.so.1.0.0 => /usr/local/ssl/lib64/libssl.so.1.0.0 (0x00007f90315c6000) libcrypto.so.1.0.0 => /usr/local/ssl/lib64/libcrypto.so.1.0.0 (0x00007f90311da000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f9030fd6000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f9030cce000) libm.so.6 => /lib64/libm.so.6 (0x00007f90309d0000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f90307b9000) libc.so.6 => /lib64/libc.so.6 (0x00007f903040b000) /lib64/ld-linux-x86-64.so.2 (0x00007f90320a2000) repeating, but with TokuDB build enabled, cd bld cmake .. \ ... \ - -DWITHOUT_TOKUDB=1 + -DWITHOUT_TOKUDB=0 build now fails, make ... [ 29%] Built target perfschema Scanning dependencies of target tokuportability_static_conv [ 29%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/huge_page_detection.cc.o [ 29%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/file.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/memory.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/os_malloc.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/portability.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_assert.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_crash.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_fair_rwlock.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_path.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_pthread.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_time.cc.o Linking CXX static library libtokuportability_static_conv.a [ 30%] Built target tokuportability_static_conv [ 30%] Generating tokuportability_static_depends.cc Scanning dependencies of target tokuportability_static [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static.dir/tokuportability_static_depends.cc.o Linking CXX static library libtokuportability_static.a /usr/bin/ar: creating /usr/local/src/mariadb/bld/storage/tokudb/ft-index/portability/libtokuportability_static.a [ 30%] Built target tokuportability_static Scanning dependencies of target tokuportability [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/huge_page_detection.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/file.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/memory.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/os_malloc.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/portability.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_assert.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_crash.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_fair_rwlock.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_path.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_pthread.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_time.cc.o Linking CXX shared library libtokuportability.so [ 31%] Built target tokuportability Scanning dependencies of target logformat [ 31%] Building CXX object storage/tokudb/ft-index/ft/CMakeFiles/logformat.dir/logformat.cc.o Linking CXX executable logformat [ 31%] Built target logformat Scanning dependencies of target generate_log_code [ 31%] Generating log_code.cc, log_print.cc, log_header.h ./logformat: error while loading shared libraries: libtokuportability.so: cannot open shared object file: No such file or directory make[2]: *** [storage/tokudb/ft-index/ft/log_code.cc] Error 127 make[1]: *** [storage/tokudb/ft-index/ft/CMakeFiles/generate_log_code.dir/all] Error 2 make: *** [all] Error 2 yes, I'm aware that opensuse is not a currently "supported platform" for TokuDB. If that's a dealbreaker, I'd like to know MariaDB project's position, if any, on getting that support in place. If it's not, I'd like to keep trying to get it working, here. darx

4 15

Re: [Maria-developers] MDEV-4956 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.used_tables
by Sergei Golubchik 03 Dec '13

03 Dec '13

Hi, Sergey! On Aug 27, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 3807 > revision-id: svoj(a)mariadb.org-20130827121233-xh1uyhgfwbhedqyf > parent: jplindst(a)mariadb.org-20130823060357-pww92qxla7o8iir7 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Tue 2013-08-27 16:12:33 +0400 > message: > MDEV-4956 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.used_tables > > - tc_acquire_table and tc_release_table do not access > TABLE_SHARE::tdc.used_tables anymore > - in tc_acquire_table(): release LOCK_tdc after we relase LOCK_open > (saves a few CPU cycles in critical section) > - in tc_release_table(): if we reached table cache threshold, evict > to-be-released table without moving it to unused_tables. unused_tables > must be empty at this point (added assertion). I don't understand what you're doing here, could you elaborate? E.g. explain in a changeset comment what you've done, why you introduced a new list for tables (all_share), what is its semantics, etc. Regards, Sergei

3 12

[Maria-developers] Building TOKUDB into 5.5.34 from sources
by md＠rpzdesign.com 02 Dec '13

02 Dec '13

I am trying to build 5.5.34 from source on Linux x64 (Fedora 14). I was looking at the readme files and could not find anything about building 5.5.34 from source and including TokuDB (which has the source code in the /storage/TokuDB/ directory) into the build. I was looking for something like: -DWITH_TOKUDB_STORAGE_ENGINE=ON but did not stumble across anything. Anybody got some ideas? Thanks, Marco Here is my build script: rm -f CMakeCache.txt make clean cmake \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/mysql \ -DENABLED_PROFILING=OFF \ -DENABLE_DEBUG_SYNC=ON \ -DENABLE_GCOV=OFF \ -DINSTALL_LAYOUT=STANDALONE \ -DMYSQL_DATADIR=/mysql/data \ -DINSTALL_INCLUDEDIR=include/mysql \ -DINSTALL_UNIX_ADDRDIR=/mysql/mysql.sock \ -DUSE_ARIA_FOR_TMP_TABLES=ON \ -DWITHOUT_PBXT_STORAGE_ENGINE=ON \ -DWITHOUT_SERVER=OFF \ -DWITH_ARCHIVE_STORAGE_ENGINE=OFF \ -DWITH_EMBEDDED_SERVER=OFF \ -DWITH_FAST_MUTEXES=OFF \ -DWITH_FEDERATEDX_STORAGE_ENGINE=OFF \ -DWITH_FEEDBACK=OFF \ -DWITH_LIBWRAP=OFF \ -DWITH_PIC=ON \ -DWITH_PARTITION_STORAGE_ENGINE=OFF \ -DWITH_PERFSCHEMA_STORAGE_ENGINE=ON \ -DWITH_SPHINX_STORAGE_ENGINE=OFF \ -DWITH_SSL=OFF \ -DWITH_UNIT_TESTS=OFF \ -DWITH_VALGRIND=OFF \ -DWITH_XTRADB_STORAGE_ENGINE=ON \ -DWITHOUT_OQGRAPH=ON \ -DWITH_ZLIB=bundled make (Sorry, I posted this question on maria.com knowledge base open questions also) https://mariadb.com/kb/en/cmake-settings-to-include-tokudb-in-5534/

3 2

[Maria-developers] Movable conditions and semi-joins
by Sergei Petrunia 27 Nov '13

27 Nov '13

Hi! The below is an extension of the skype discussion. I wrote down the limitation imposed by FirstMatch, and also covered other Semi-join strategies. <contents> 1. SJ-Materialization 2. FirstMatch 3. Duplicate Elimination 4. Loose Scan </contents> == 1. SJ-Materialization == The general rule is: If the condition is originally attached inside an SJM nest, one should not move it outside. Conditions that are originally outside should not be moved inisde the SJM nest. The reason for the rule is that field values for inner tables are only available inside the SJM nest. Correspondingly, values of fields of outer tables are not available at materialization step. The only possible exception is when the condition depends only on columns that are in the IN-equality. Then, it could be moved. I don't think this is worth handling for the first milestone. == 2. FirstMatch == Consider a query select ... from ot1, nt1, nt2 where expr(ot1) in (select ... from it1, it2, ... where ...) and a join order ot1 FirstMatch(it1, it2) nt1 nt2 It is possible to move cond(otX) to table itX or ntX. There is a problem with moving condition attached to itX (will call it SUBQ_COND. this ocndition depends on table itX and maybe on ot1) to table ntX. Consider join execution execution: ot1.row0 ot1.row1 -+- it1.row1 -- it2.row1 | +- it1.row2 -- it2.row2 ->- (F1)-->-- nt1.row1 (F2)--<-- FirstMatch check is at point (F2). At that point, we know that ot1.row1 has had a match in the subquery, and the execution returns to table ot1. If SUBQ_COND is moved to table ntX, then there may be a situation where we will get to point (F2) and assume that the subquery had a match, while actually it hasn't. A suggestion by Igor: at point (F2) we should check the last return value of SUBQ_COND (actually, of all conditions that were moved like SUBQ_COND was) It is actually more complicated: what if there is COND(ot1, nt1) such that COND(ot1.ro1, nt1.row1) = FALSE? Then, we will never evaluate SUBQ_COND(ot1.row1, ...). What if never evaluate SUBQ_COND(ot1.row1, ...) but before we have evaluated SUBQ_COND(ot1.row0, ...)? We will have to track what we have evaluated for the current row of ot1.row1. == 3. Duplicate Elimination == Duplicate Elimination removes duplicate matches using a temporary table. In order to work correctly, filtering needs to be performed before the weedout is done. Let's again use an example. select ... from ot1, nt1, nt2 where expr(ot1) in (select ... from it1, it2, ... where ...) and a join order (DUPS-W-START) it1 ot1 it2 (DUPS-W-WEEDOUT) nt1 nt2 at DUPS-W-START, all rows are deleted from the temp. table. at DUPS-W-WEEDOUT, we take ot1.rowid and attempt to write it into the temp table. Basic idea: If a condition SUBQ_COND(itX, ...) is moved to the right across the (DUPS-W-WEEDOUT) point, wrong query results will occur. Example: Suppose, SUBQ_COND(it2) is moved to nt1, and join execution proceeds as follows: it1.row1 --- ot1.row1 --- it2.row1 -- DUPS-W-WEEDOUT -- nt1 | - We reach DUPS-W-WEEDOUT with (it1.row1, ot1.row1, it2.row1). - The temp. table is empty, so rowid(ot1.row1) is written into temp.table, and Weedout doesn't weed it out. - At nt1, SUBQ_COND(it2.row1) is evaluated and found to be false. join execution returns back. - Suppose, tables it2 and ot1 have no more matching rows, so execution returns to it1. (the same diagram with a few more steps) it1.row1 --- ot1.row1 --- it2.row1 -- DUPS-W-WEEDOUT -- nt1 | it1.row2 --- ot1.row1 --- it2.row2 -- DUPS-W-WEEDOUT - We reach DUPS-W-WEEDOUT with (it1.row2, ot1.tow1, it2.rowX). - The temptable already has rowid(ot1.row1). The record combination is filtered out. if SUBQ_COND(it2.row2)=TRUE, then the result will miss a row. == 4. Loose Scan == Loose Scan is similar to First Match - it does short-cutting (but also makes forward jumps on the driving table). Short-cutting imposes a limitation similar to FirstMatch - conditions that depend on subquery's tables cannot be moved out of LooseScan's range. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] maria-tokudb-engine-10.0 deb package (raring) - if files are empty where are sources?
by Paul Hewson 27 Nov '13

27 Nov '13

Hello, I'm trying to follow the instructions on https://mariadb.com/kb/en/how-to-enable-tokudb-in-mariadb/ which imply there is more tuned version of tokudb for mariadb than I might get by following instructions at tokutek. However, I have ubuntu 13.04 (raring), and despite some mismatch in numbers (I have maria 10.0 rather than 5.5) I am getting an empty file when I try to apt-get install mariadb-tokudb-engine-10.0 I found a bit of discussion about this point, but got the impression it had been fixed. Does it mean it was fixed and the debs were removed? If so, do you mind if I ask where I get the mariadb tokutek sources. Again the webpage https://mariadb.com/kb/en/how-to-enable-tokudb-in-mariadb/ implies only some parts (most parts) of totudb have been merged into mariadb. Thanks, sorry for troubling you with naive issues like this. Paul ________________________________ [http://www.plymouth.ac.uk/images/email_footer.gif]<http://www.plymouth.ac.uk/worldclass> This email and any files with it are confidential and intended solely for the use of the recipient to whom it is addressed. If you are not the intended recipient then copying, distribution or other use of the information contained is strictly prohibited and you should not rely on it. If you have received this email in error please let the sender know immediately and delete it from your system(s). Internet emails are not necessarily secure. While we take every care, Plymouth University accepts no responsibility for viruses and it is your responsibility to scan emails and their attachments. Plymouth University does not accept responsibility for any changes made after it was sent. Nothing in this email or its attachments constitutes an order for goods or services unless accompanied by an official order form.

2 2

[Maria-developers] Malformed XML in mysql-test/std_data/Index.xml added in revision 3873
by Jeremy Cole 26 Nov '13

26 Nov '13

Bar, Regarding your change: http://bazaar.launchpad.net/~maria-captains/maria/10.0/revision/3873/mysql-… I was wondering if it's intentional that mysql-test/std_data/Index.xml now contains malformed XML. It wasn't mentioned in your commit message and the test nor XML files do not mention it specifically. Although utf8_5624_5_bad is supposed to be "bad" I didn't think it was supposed to be *that* "bad" (mismatched "p" vs "a" tags): $ xmllint --noout mysql-test/std_data/Index.xml mysql-test/std_data/Index.xml:126: parser error : Opening and ending tag mismatch: p line 126 and a <reset>a-a4</reset>xxx04</a> ^ mysql-test/std_data/Index.xml:127: parser error : Opening and ending tag mismatch: p line 127 and a <reset>a-aa5</reset>xxx05</a> ^ mysql-test/std_data/Index.xml:128: parser error : Opening and ending tag mismatch: p line 128 and a <reset>a-aaa6</reset>xxx06</a> ^ mysql-test/std_data/Index.xml:129: parser error : Opening and ending tag mismatch: p line 129 and a <reset>a-aaaa7</reset>xxx07</a> ^ mysql-test/std_data/Index.xml:130: parser error : Opening and ending tag mismatch: p line 130 and a <reset>a-aaaaa8</reset>xxx08</a> ^ mysql-test/std_data/Index.xml:131: parser error : Opening and ending tag mismatch: p line 131 and a <reset>a-aaaaaa9</reset>xxx09</a> ^ mysql-test/std_data/Index.xml:132: parser error : Opening and ending tag mismatch: p line 132 and a <reset>a-aaaaaa10</reset>xxx10</a> ^ Regards, Jeremy

2 1

[Maria-developers] Force PK option and global lock for DDLs
by erkan yanar 26 Nov '13

26 Nov '13

Ahoi, I would like to propagate two ideas (did it on #irc some times but still some love is missing) 1. Force PK option. I would like to have an option to force PK for (innodb) tables. Forcing is quite easy (a poc from 2011): http://linsenraum.de/erkules/2011/03/this-table-type-requires-a-primary-key… Why? RBR and so Galera would benefit from such an option. 2. Global DDL Lock Without Global DDL Lock i.e. ALTER TABLE, DROP TABLE etc. brings you in trouble while doing a --single-transaction dump. You can use flush tables with read lock, but then you need not to do a single-transaction dump anyway. Even xtrabackup got troubles: https://bugs.launchpad.net/percona-xtrabackup/+bug/722638 I got to confess I think ignoring tablespace ids is the right soltution. Even I think this are simple options. I believe they would be quite welcome to a lot of dba's out there. Regards Erkan -- über den grenzen muß die freiheit wohl wolkenlos sein

1 1

Re: [Maria-developers] [Commits] Rev 3908: MDEV-5277 - Ensure that all MySQL 5.6 options are supported
by Sergei Golubchik 25 Nov '13

25 Nov '13

Hi, Sergey! On Nov 25, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 3908 > revision-id: svoj(a)mariadb.org-20131125144940-nhdqjt2eisqaa7pb > parent: svoj(a)mariadb.org-20131114140000-5eq8gopqu5giwvqq > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0-mdev5277 > timestamp: Mon 2013-11-25 18:49:40 +0400 > message: > MDEV-5277 - Ensure that all MySQL 5.6 options are supported by the > MariaDB 10.0 server Looks ok to me, thanks! Regards, Sergei

1 0

Re: [Maria-developers] Rev 3985: MDEV-5321 Calling mysql_library_end accesses freed memory; dumps memory to display
by Sergei Golubchik 25 Nov '13

25 Nov '13

Hi, Holyfoot! On Nov 24, holyfoot(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 3985 > revision-id: holyfoot(a)askmonty.org-20131124184915-spkpy6qt8vdbluaz > parent: igor(a)askmonty.org-20131123023813-0mgl217lyyyysv5b > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5321 > timestamp: Sun 2013-11-24 22:49:15 +0400 > message: > MDEV-5321 Calling mysql_library_end accesses freed memory; dumps memory to display. > Don't call the vio_end() in the clean_up() in EMBEDDED mode. > Call vio_end() before the end_embedded_server(). ok to push Regards, Sergei

1 0

Re: [Maria-developers] Rev 3908: MDEV-5277 - Ensure that all MySQL 5.6 options are supported
by Sergei Golubchik 25 Nov '13

25 Nov '13

Hi, Sergey! On Nov 20, Sergey Vojtovich wrote: > revno: 3908 > revision-id: svoj(a)mariadb.org-20131120113753-vuxj3v4jt9cvccdy > parent: svoj(a)mariadb.org-20131114140000-5eq8gopqu5giwvqq > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0-mdev5277 > timestamp: Wed 2013-11-20 15:37:53 +0400 > message: > MDEV-5277 - Ensure that all MySQL 5.6 options are supported by the > MariaDB 10.0 server > === modified file 'mysql-test/r/mysqld--help.result' > --- a/mysql-test/r/mysqld--help.result 2013-11-03 20:26:44 +0000 > +++ b/mysql-test/r/mysqld--help.result 2013-11-20 11:37:53 +0000 > @@ -4,6 +4,9 @@ > --defaults-file=# Only read default options from the given file #. > --defaults-extra-file=# Read this file after the global files are read. > > + --abort-slave-event-count[=name] > + MySQL 5.6 compatibility option. Use > + '--debug-abort-slave-event-count' instead. > --allow-suspicious-udfs Do we really want all these do-nothing compatibility options to show up in --help? May be not? The goal of this MDEV was not to fail with an error when starting on mysqld's my.cnf file, no one needs --help for that. > Allows use of UDFs consisting of only one symbol xxx() > without corresponding xxx_init() or xxx_deinit(). That > @@ -136,10 +151,16 @@ > --deadlock-timeout-short=# > Short timeout for the two-step deadlock detection (in > microseconds) > + --default-authentication-plugin[=name] > + MySQL 5.6 compatibility option. To be implemented in > + later 10.x versions. Remove "10.x" please, just "in later versions". We don't want to promise that all these options will be implemented in 10.x series. > === modified file 'storage/innobase/handler/ha_innodb.cc' > --- a/storage/innobase/handler/ha_innodb.cc 2013-11-13 21:58:19 +0000 > +++ b/storage/innobase/handler/ha_innodb.cc 2013-11-20 11:37:53 +0000 > @@ -15750,6 +15750,12 @@ static MYSQL_SYSVAR_BOOL(purge_stop_now, > PLUGIN_VAR_OPCMDARG, > "Set purge state to STOP", > NULL, purge_stop_now_set, FALSE); > + > +static my_bool innodb_log_checkpoint_now; > +static MYSQL_SYSVAR_BOOL(log_checkpoint_now, innodb_log_checkpoint_now, > + PLUGIN_VAR_OPCMDARG | PLUGIN_VAR_NOSYSVAR, > + "MySQL 5.6 compatibility option. To be implemented in later 10.x versions.", > + NULL, NULL, FALSE); > #endif /* UNIV_DEBUG */ Why is that (and everything else in ha_innodb.cc) necessary? We'll merge with the latest InnoDB, we will have all its variables. Regards, Sergei

2 3

[Maria-developers] MariaDB-10.0-beta sysbench results
by Axel Schwenke 23 Nov '13

23 Nov '13

Hi, I have run another sysbench with in-memory workload to check where we are with respect to mutex contention. The benchmark is sysbench OLTP (using the bzr trunk of sysbench with LUA support). 10 mio rows in a single table (about 2.5G .ibd file). InnoDB buffer pool set to 32G and warmed up. I have run the test on two different machines. First on our old trusty facebook2 machine with 8 cores/16 threads in two NUMA zones. Second on the fat lizard2 machine with 32 cores/64 threads in 4 NUMA zones. As expected, the heavy contention of the LOCK_open mutex is more visible on the bigger machine (I have not done any detailed analysis in this benchmark, but before while doing benchmarks for MDEV-5081) I notice a visible performance gain in MariaDB-10.0 over MariaDB-5.5 at 32 threads. This is probably due to the work on the InnoDB kernel mutex (impossible to prove since 5.5 doesn't have mutex instrumentation). At higher concurrency however the LOCK_open mutex kicks in and performance is clearly dominated by that mutex alone (it flattens out for all 5.5 and 10.0 versions of MariaDB) The good news is, that 10.0.4, 10.0.5 and 10.0.6 are quite close. There is a small regression from 10.0.4 to 10.0.5, but IMHO this isn't worth investigation while the LOCK_open issue is not fixed. Attached: diagrams. OpenOffice spreadsheets. XL

6 7

[Maria-developers] get is now a reserved word in mariadb 10 ?
by Jocelyn Fournier 21 Nov '13

21 Nov '13

Hi, I have a query with a column named "get". i.e. SELECT get FROM my_table... It works perfectly in 5.5, but in 10.0.6 it fails : ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'get FROM my_table' at line 1 I assume "get" is a new reserved word, probably inherited from MySQL 5.6 memcached support, although memcached is not supported in MariaDB ? (it's mentionned here : http://dev.mysql.com/doc/mysqld-version-reference/en/mysqld-version-referen… ) Is there a specific KB entry for MariaDB 10.0.6 reserved words ? (I only see an entry for 5.5 here : https://mariadb.com/kb/en/reserved-words/ ) Thanks and regards, Jocelyn Fournier

4 4

[Maria-developers] Prep for MariaDB 5.5.34
by Daniel Bartholomew 20 Nov '13

20 Nov '13

I'm currently in the process of preparing for the MariaDB 5.5.34 release. Draft changelog and release notes are here: - https://mariadb.com/kb/en/mariadb-5534-release-notes/ - https://mariadb.com/kb/en/mariadb-5534-changelog/ As always, fixes, additions, and such are welcome. Buildbot is still working on building the packages and when it's finished I'll get them prepped, signed, and uploaded to the mirrors with the plan to release tomorrow morning US Eastern Time. Thanks. -- Daniel Bartholomew MariaDB

1 0

[Maria-developers] MariaDB 10.0.6 using more file descriptors than 5.5 ?
by Jocelyn Fournier 20 Nov '13

20 Nov '13

Hi, I've just migrated MariaDB (+tokudb) from 5.5.33a to 10.0.6, and I've encountered a '[ERROR] Error in accept: Too many open files' error during a backup. As this was not occuring with MariaDB 5.5, is this expected to have 10.0.6 using much more file descriptors ? (open_files_limit was set to about 8100 with 5.5, I raised it to 65535 now to fix this issue). Thanks, Jocelyn Fournier

3 3

[Maria-developers] MDEV-5277 - Ensure that all MySQL 5.6 options are supported by the MariaDB 10.0 server
by Sergey Vojtovich 20 Nov '13

20 Nov '13

Hi! I'm gathering 5.6 options that are not present in 10.0 and the way we want to handle them. Could you check options listed below and if you're aware of their destiny indicate if options: * not used or needed in MariaDB * to be implemented in later 10.x versions * tuned via different 10.x specific options The following options exist in 5.6 but not in 10.0: * default-tmp-storage-engine * log-raw * default_authentication_plugin * ignore-db-dir * binlog-max-flush-queue-time * binlog-row-image * explicit-defaults-for-timestamp * master-info-repository * relay-log-info-repository * binlog-rows-query-log-events * binlog-order-commits * log-throttle-queries-not-using-indexes * end-markers-in-json * optimizer-trace * optimizer-trace-features * optimizer-trace-offset * optimizer-trace-limit * optimizer-trace-max-mem-size * eq-range-index-dive-limit * server-uuid * server-id-bits * slave-rows-search-algorithms * table-open-cache-instances * slave-allow-batching * slave-checkpoint-period * slave-checkpoint-group * slave-parallel-workers * slave-pending-jobs-size-max * enforce-gtid-consistency * gtid-mode * disconnect-on-expired-password * sha256-password-private-key-path (sha256_password plugin) * sha256-password-public-key-path (sha256_password plugin) The following options exist in 5.5 and 5.6 but not in 10.0: * abort-slave-event-count => debug-abort-slave-event-count * disconnect-slave-event-count => debug-disconnect-slave-event-count * exit-info => debug-exit-info * max-binlog-dump-events => debug-max-binlog-dump-events * sporadic-binlog-dump-fail => debug-sporadic-binlog-dump-fail * new * skip-new * skip-stack-trace The following options were added after 5.6.10: * innodb-buffer-pool-evict // UNIV_DEBUG * innodb-ft-total-cache-size * innodb-ft-result-cache-limit * innodb-log-compressed-pages * innodb-log-checkpoint-now // UNIV_DEBUG * rpl-stop-slave-timeout * validate-user-plugins The following NO_CMD_LINE options exist in 5.6 but not in 10.0 (won't add them): * gtid-executed * gtid-next * gtid-next-list * gtid-owned * gtid-purged * log-bin-basename * relay-log-basename * transaction-allow-batching Thanks, Sergey

4 12

Re: [Maria-developers] Rev 3974: MDEV-5182 - build of 10.0.4/r3863 fails @ 'cmake'
by Sergei Golubchik 20 Nov '13

20 Nov '13

Hi, Sergey! On Nov 18, Sergey Vojtovich wrote: > ------------------------------------------------------------ > revno: 3974 > revision-id: svoj(a)mariadb.org-20131118114801-1kny2cvx8kpjsw7t > parent: jplindst(a)mariadb.org-20131115132442-5d4hqg5idjta2dni > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5-mdev5182 > timestamp: Mon 2013-11-18 15:48:01 +0400 > message: > MDEV-5182 - build of 10.0.4/r3863 fails @ 'cmake' with > -DINSTALL_SYSCONFDIR/-DDEFAULT_SYSCONFDIR specified > > The reason for a bug is in support-files/CMakeLists.txt that tries to > install files in INSTALL_SYSCONF2DIR, without checking whether this > variable is actually defined. > > Normally INSTALL_SYSCONF2DIR is always defined if INSTALL_SYSCONFDIR is > defined, and support-files/CMakeLists.txt assumes that. But when one > specifies INSTALL_SYSCONFDIR manually on the command line, this > assumption might be broken. Looks ok to me. An alternative solution would be to define INSTALL_SYSCONF2DIR if INSTALL_SYSCONFDIR is defined. Regards, Sergei

2 1

[Maria-developers] Open Questions in the Knowledge Base
by Ian Gilfillan 19 Nov '13

19 Nov '13

There are quite a few unanswered questions in the Knowledge Base https://mariadb.com/kb/en/+questions/ If anyone knowledgeable in those areas can help answer some of these, even just a request for more info, that would be great :) thanks, ian

2 1

Re: [Maria-developers] Accessing rpl_global_gtid_binlog_state is not consistently protected
by Kristian Nielsen 19 Nov '13

19 Nov '13

Pavel Ivanov <pivanof(a)google.com> writes: > sql/log.cc has highly inconsistent pattern of accessing > rpl_global_gtid_binlog_state. In some places it's protected by mutex > rpl_global_gtid_binlog_state.LOCK_binlog_state, in some places it's > protected by global mutex LOCK_rpl_gtid_state, and in some places it's > not protected by any mutex at all. This is a bug that should be fixed, > right? Yes, agree, thanks for finding this. > I can file this in JIRA if you want, just wanted to let you know faster. I've filed it as MDEV-5306. > I've noticed this problem after one of our AddressSanitizer testing > bots reported that rpl_binlog_state::bump_seq_no_if_needed (called > from MYSQL_BIN_LOG::bump_seq_no_counter_if_needed) was using memory > from hash after it was already freed by rpl_binlog_state::reset > (called by MYSQL_BIN_LOG::reset_logs). Can you test again with the latest patch I've just pushed to 10.0-base? I looked briefly into making an mtr test case to catch this, but I did not see any easy way to do that. I went through the code to try and check all uses of rpl_binlog_state for proper locking, but it would be nice to get a confirmation that I did not miss anything. - Kristian.

3 2

Re: [Maria-developers] MariaDB Galera replication
by Pavel Ivanov 18 Nov '13

18 Nov '13

I'm starting a new thread as this is already doesn't have anything to do with the original topic. On Fri, Nov 15, 2013 at 10:46 AM, Alex Yurchenko <alexey.yurchenko(a)codership.com> wrote: >>> Please pardon this arrogant interruption of your discussion and shameless >>> self-promotion, but I just could not help noticing that Galera >>> replication >>> was designed specifically with these goals in mind. And it does seem to >>> achieve them better than semi-sync plugin. Have you considered Galera? >>> What >>> makes you prefer semi-sync over Galera, if I may ask? >> >> >> To be honest I never looked at how Galera works before. I've looked at >> it now and I don't see how it can fit with us. The major disadvantages >> I immediately see: >> 1. Synchronous replication. That means client must wait while >> transaction is applied on all nodes which is unacceptably big latency >> of each transaction. And what if there's a network blip and some node >> becomes inaccessible? All writes will just freeze? I see the statement >> that "failed nodes automatically excluded from the cluster", but to do >> that cluster must wait for some timeout in case it's indeed a network >> blip and node will "quickly" reconnect. And every client must wait for >> cluster to decide what happened with that one node. >> 2. Let's say node fell out of the cluster for 5 minutes and then >> reconnected. I guess it will be treated as "new node", it will >> generate state transfer and the node will start downloading the whole >> database? And while it's trying to download say 500GB of data files >> all other nodes (or maybe just donor?) won't be able to change those >> files locally and thus will blow up its memory consumption. That means >> they could quickly run out of memory and "new node" won't be able to >> finish its "initialization"... >> 3. It looks like there's strong asymmetry in starting cluster nodes -- >> the first one should be started with empty wsrep_cluster_address and >> all others should be started with the address of the first node. So I >> can't start all nodes uniformly and then issue some commands to >> connect them to each other. That's bad. >> 4. What's the transition path? How do I upgrade MySQL/MariaDB >> replicating using usual replication to Galera? It looks like there's >> no such path and the solution is stop the world using regular >> replication and restart it using Galera. Sorry I can't do that with >> our production systems. >> >> I believe these problems are severe enough for us, so that we can't >> work with Galera. > > > Pavel, you seem to be terribly mistaken on almost all accounts: > > 1. *Replication* (i.e. data buffer copying) is indeed synchronous. But > nobody said that commit is. What Galera does is very similar to semi-sync, > except that it does it technically better. I would not dare to suggest > Galera replication if I didn't believe it to be superior to semi-sync in > every respect. Well, apparently we have a different understanding of what the term "synchronous replication" means. This term is all over the Galera doc, but I didn't find the detailed description of how actually Galera replication work. So I assumed that my understanding of the term (which actually seem to be in line with wiki's definitions http://en.wikipedia.org/wiki/Replication_(computing) ) is what was implied there. So I hope you'll be able to describe in detail how Galera replication works. > As an example here's an independent comparison of Galera vs. > semi-sync performance: > http://linsenraum.de/erkules/2011/06/momentum-galera.html. This is a nice blog post written in German and posted in 2011. And while Google Translate gave me an idea what post was about it would be nice to see something more recent and with better description of what was the actual testing set up. > In fact, majority > of Galera users migrated from the regular *asynchronous* MySQL replication, > which I think is a testimony to Galera performance. I don't mean to troll, but this can also mean that everyone who migrated didn't care much about performance and Galera's performance was within sane boundaries... BTW, just found here https://mariadb.com/kb/en/mariadb-galera-cluster-known-limitations/ : "by design performance of the cluster cannot be higher than performance of the slowest node; however, even if you have only one node, its performance can be considerably lower comparing to running the same server in a standalone mode". That contradicts your words. > 2. Node reconnecting to cluster will normally receive only events that it > missed while being disconnected. This seem to contradict to the docs. Again from https://mariadb.com/kb/en/mariadb-galera-cluster-known-limitations/ : "After a temporary split, if the 'good' part of the cluster was still reachable and its state was modified, resynchronization occurs". > 3. You are partially right about it, but isn't it much different from > regular MySQL replication where you first need to set up master and then > connect slaves (even if you have physically launched the servers at the same > time). Operation of setting up master and then connecting slaves consists of mostly only executing CHANGE MASTER TO and then START SLAVE on all slaves after all MySQL instances (including master) were started with the same set of command line flags. This is fundamentally different from starting instances with different arguments, especially when these arguments should be different depending on whether the replica is starting first or there's already some other replica running. > Yet, Galera nodes can be started simultaneously and then joined > together by setting wsrep_cluster_address from mysql client connection. This > is not advertised method, because in that case state snapshot transfer can > be done only by mysqldump. If you set the address in advance, rsync or > xtrabackup can be used to provision the fresh node. This is of course better because I can start all instances with the same command line arguments. But transferring snapshot of a very big database using mysqldump, and causing the node that creates mysqldump to blow up memory consumption during the process, that is still a big problem. > 4. Every Galera node can perfectly work as either master or slave to native > MySQL replication. So migration path is quite clear. Nope, not clear yet. So I'll be able to upgrade all my MySQL instances to a Galera-supporting binary while they are replicating using standard MySQL replication. That's good. Now, how the Galera replication is turned on after that? What will happen if I just set wsrep_cluster_address address on all replicas? What will replicas do, and what will happen with the standard MySQL replication? > It is very sad that you happen to have such gross misconceptions about > Galera. If those were true, how would MariaDB Galera Cluster get paying > customers? Care to share some numbers? Like what's the rough amount of those paying customers? What size is the biggest installation -- number of clusters, replicas, highest QPS load? I'm not asking to share any confidential information, but the rough ballpark of the numbers would be helpful. > May be my reply will convince you to have a second look at it. > (In addition to the above Galera is fully multi-master, does parallel > applying and works great in WAN) I hope your explanation of how Galera replication work will help me understand how great it works over WAN and how you could make full multi-master work without fully synchronous replication in my understanding of that term. Pavel

3 9

[Maria-developers] What is bundled libevent library for?
by Pavel Ivanov 18 Nov '13

18 Nov '13

Sergei, Do you know why MariaDB sources have libevent library bundled with them? I don't see it being used anywhere. I see it was added in this commit by Monty http://bazaar.launchpad.net/~maria-captains/maria/10.0/revision/3492.3.1, but even in that commit I can't find any usages of the library. So should it be removed from MariaDB? Thank you, Pavel

4 4

[Maria-developers] Why local modifications to PCRE library?
by Pavel Ivanov 16 Nov '13

16 Nov '13

Alexander, I see you've included the full source of PCRE library into MariaDB tree and even made some local modifications which are not included into PCRE upstream. Could you please explain why did you do that? Why not allow to build with system standard PCRE library? I understand that you could have needed some modifications and thus could want to include sources from PCRE trunk to allow users to build with modifications when their system-wide library is too old. But as I don't see your modifications on PCRE trunk that means you probably had something different in mind, and decided to incur the pain of constantly updating the included sources and merging security fixes. So why did you decide to do that? Thank you, Pavel

3 7

Re: [Maria-developers] Semisync plugin incompatibility
by Kristian Nielsen 15 Nov '13

15 Nov '13

Pavel Ivanov <pivanof(a)google.com> writes: > We've noticed recently that semisync_master plugin in MariaDB (which > apparently was fully inherited from MySQL) is seriously incompatible > with our understanding of the purpose of semi-sync replication. This > incompatibility was apparently introduced as a fix for > http://bugs.mysql.com/bug.php?id=45672. The "major no-no" that bug So as I understand it, this bug is about what should happen when semisync is enabled, but no slaves are connected. Apparently before the fix of Bug#45672, an error was thrown late during COMMIT. So the transaction was committed (locally on the master), but the client still got an error back. And if I understand correctly, after the fix of Bug#45672, no error is thrown in the case where no slave is connected. > talks about is in our opinion the whole purpose of semi-sync > replication -- if transaction is not replicated to at least one slave > client shouldn't get OK even if transaction is committed locally on > the master. Also master shouldn't just turn off semi-sync replication > whenever it wants. So with "just turn off semi-sync replication whenever it wants" - what are you refering to here? I seem to remember that semisync has a timeout, and it gets disabled if that timeout triggers? My guess is that this is what you have in mind, but I wanted to ask to make sure ... > We will fix this problem for us, but first I wanted to understand > what's your view of the purpose of semi-sync replication and how you > think it should work? I need to know your opinion to understand how I > should fix this issue... Well, personally, I never was much interested in semi-sync. But it is my understanding that there is some interest, so I will answer with what small opinion I have. I suppose the general idea is that when client sees its COMMIT complete, it can know that its transaction exists in at least two places (master binlog + at least one slave relay log). So there is no longer any single point of failure that can cause loss of the transaction. Another point of view I is that semi-sync provides some sort of throttle on how fast the master can generate events compared to how fast the slaves can receive them: http://www.mysqlperformanceblog.com/2012/01/19/how-does-semisynchronous-mys… There was also a suggestion (and a patch is floating around somewhere) for "enhanced semisync replication": https://mariadb.atlassian.net/browse/MDEV-162 This delays not only client acknowledge but also InnoDB commit until the ack from at least one slave, which means that transactions are not visible to other clients until they exist on at least one slave in addition to on the master. Since this is _semi_-sync, not real two-phase commit synchronous replication, the main problem is that there is way to ensure consistency in the general error case. The transaction is already fully committed on the master, it cannot be rolled back. So we are left with the choice of one of two evils: 1. Report an error to the client. Most clients would then probably wrongly assume that the transaction was _not_ committed. There also does not seem to be much the client can do about the error except perhaps log an incident to the monitoring system. On the other hand, then at least the problem is not silently ignored. 2. Report success to the client but complain loudly in the error log (I assume this is what happens in current code). This leaves the client unaware that there is a problem (but presumably the monitoring system will catch the message in the error log). >From this summary, I think I can see the logic of the current behaviour: - It preserves protection against single-point-of-failure. If all slaves are gone, then we already have one failure, and unless we experience a double failure (master also failing before slave recovers), the transaction will eventually be sent to a slave and no overall failure happens. - If the client can anyway not do anything about the problem except notify the monitoring system, the server may as well do the notification itself. But the opposite point of view also has merit. The client asked for semi-sync behavior, but did not get it, and it does not even have a way to know about the problem. That is not good. Does the client currently at least get a warning for the COMMIT? I think it should (eg. the fix for Bug#45672 should at least have been to turn the error into a warning, not remove the error completely). What I think could make sense is if the client got an error during the prepare phase if no slaves are connected. In this case we _can_ roll back the transaction and give an error to the client without any issue of consistency. But it still leaves a small window where the last slave can disappear between the prepare and the commit phase and leave us with the original problem. I hope this helps you ... Maybe you can describe your use-case, and how you need things to work for that case? Personally I have nothing against changing this behaviour to something more logical, I am just not sure what the most logical behaviour is ... - Kristian.

4 8

[Maria-developers] parallel replication monitoring
by Giuseppe Maxia 15 Nov '13

15 Nov '13

What can I use to see the status of parallel replication? I am looking for: * how many workers are used; * in which schema they are working; * what query they are running. So far, I only see mysql.gtid_slave_pos and information_schema.processlist. Anything else available? Thanks in advance Giuseppe

4 6

[Maria-developers] Slave can take a very long time to start replication
by Pavel Ivanov 15 Nov '13

15 Nov '13

Kristian, As I understand currently when slave connects to master and wants to start replicating it passes GTID to start from, master finds binlog file where the earliest GTID is located and then scans through that file to find the exact binlog position to start sending binlog events from. If this binlog file is pretty big then scanning can take a very long time. I guess especially long when several slaves try to start replicating roughly at the same time. We observed 60-90 seconds between initial connection by the slave and the first real binlog events starting to flow. In this period of time slave doesn't receive anything from master and thus it's very easy to confuse such situation with connection loss, hit slave_net_timeout, reopen connection to the master again and force it to start searching through binlog file from the very beginning... Putting aside the argument of what value is good enough for slave_net_timeout, I'd say in any case slave taking 60 seconds to just start receiving binlog events from master is unacceptable. Did you think about this problem before? Maybe you've even planned already to implement some solution for this? Thank you, Pavel

2 1

[Maria-developers] Review for maria/10.0-serg/revision/3899 (MDEV-5248 Serious incompatibility and data corruption of DATETIME and DATE types due to get_innobase_type_from_mysql_type refactor combined with InnoDB Online DDL)
by Jeremy Cole 15 Nov '13

15 Nov '13

Serg, Thanks for looking into this. General comments: - The approach overall looks sound. I am very much in favor of reverting back to Oracle MySQL's type codes as you have, by reverting the refactor of get_innobase_type_from_mysql_type as you did. - You've mixed tabs and spaces in a bunch of places. InnoDB code should use tabs to indent. - It would probably be worth visually comparing to MySQL 5.6's get_innobase_type_from_mysql_type to ensure there is nothing new there aside from my comments. - It is almost certainly worth creating test cases with data files from older MariaDB and newer MySQL versions to exercise the usage of the code paths that only occur during upgrade; that doesn't need to happen here necessarily. Comments on the diff inline below. Regards, Jeremy (I removed the .result file differences as any issues there will be resolved by fixes to the .test.) === renamed file 'mysql-test/suite/innodb/t/1byte_data_int.opt' => > 'mysql-test/suite/innodb/t/data_types.opt' > === renamed file 'mysql-test/suite/innodb/t/1byte_data_int.test' => > 'mysql-test/suite/innodb/t/data_types.test' > --- mysql-test/suite/innodb/t/1byte_data_int.test 2013-10-31 22:20:05 > +0000 > +++ mysql-test/suite/innodb/t/data_types.test 2013-11-11 20:54:09 +0000 > @@ -1,25 +1,76 @@ > --source include/have_innodb.inc > > I would strongly suggest a detailed explanatory comment and warning added with "--echo #" so that it appears in the result file as well, explaining the repercussions changing any of the results here. > ---echo # Create a table with a 1-byte ENUM, 1-byte SET, and TINYINT > UNSIGNED. > - > CREATE TABLE t1 > ( > - t1_enum ENUM("a", "b", "c"), > - t1_set SET("a", "b", "c"), > - t1_tinyint_s TINYINT, > - t1_tinyint_u TINYINT UNSIGNED > + t1_BIGINT BIGINT, > + t1_BIGINT_UNSIGNED BIGINT UNSIGNED, > + t1_BINARY_100 BINARY(100), > + t1_BIT_2 BIT(2), > + t1_BIT_20 BIT(20), > + t1_BLOB BLOB, > + t1_CHAR_100 CHAR(100), > + t1_CHAR_100_BINARY CHAR(100) BINARY, > + t1_DATE DATE, > + t1_DATETIME DATETIME, > + t1_DATETIME_6 DATETIME(6), > + t1_DECIMAL_10_3 DECIMAL(10,3), > + t1_DECIMAL_10_3_UNSIGNED DECIMAL(10,3) UNSIGNED, > + t1_DOUBLE DOUBLE, > + t1_DOUBLE_UNSIGNED DOUBLE UNSIGNED, > + t1_ENUM ENUM("a", "b", "c"), > + t1_ENUM_BINARY ENUM('a','b') BINARY, > Why the mix of quote styles? For completeness this should also use an ENUM with > 254 elements to create a 2-byte ENUM. > + t1_FLOAT FLOAT, > + t1_FLOAT_UNSIGNED FLOAT UNSIGNED, > + t1_INT INT, > + t1_INT_UNSIGNED INT UNSIGNED, > + t1_LONGBLOB LONGBLOB, > + t1_LONGTEXT LONGTEXT, > + t1_MEDIUMBLOB MEDIUMBLOB, > + t1_MEDIUMINT MEDIUMINT, > + t1_MEDIUMINT_UNSIGNED MEDIUMINT UNSIGNED, > + t1_MEDIUMTEXT MEDIUMTEXT, > + t1_SET SET("a", "b", "c"), > + t1_SET_BINARY SET('a','b') BINARY, > Same mix of quote styles. Also should create a SET with > 8 elements to create a 2-byte SET. > + t1_SMALLINT SMALLINT, > + t1_SMALLINT_UNSIGNED SMALLINT UNSIGNED, > + t1_TEXT TEXT, > + t1_TIME TIME, > + t1_TIMESTAMP TIMESTAMP, > + t1_TIMESTAMP_5 TIMESTAMP(5), > + t1_TIME_4 TIME(4), > Maybe move up to TIME above (I guess these are all alphabetically sorted, but it may be better to sort only by the type [or add e.g. _0 to the base one]). Why 4 and not 3 or 6 or some other value? > + t1_TINYBLOB TINYBLOB, > + t1_TINYINT TINYINT, > + t1_TINYINT_UNSIGNED TINYINT UNSIGNED, > + t1_TINYTEXT TINYTEXT, > + t1_VARBINARY_100 VARBINARY(100), > + t1_VARCHAR_10 VARCHAR(10), > + t1_VARCHAR_10_BINARY VARCHAR(10) BINARY, > + t1_VARCHAR_500 VARCHAR(500), > + t1_VARCHAR_500_BINARY VARCHAR(500) BINARY, > + t1_YEAR_2 YEAR(2), > + t1_YEAR_4 YEAR(4) > ) ENGINE=InnoDB; > > ---echo # All t1 fields' mtypes should be 6 (DATA_INT). > - > SELECT > - name, > - mtype, > - (prtype & 512) = 512 AS is_unsigned > + name, CASE mtype WHEN 1 THEN "DATA_VARCHAR" > Put CASE on its own line (distinct from "name") and move the first WHEN to its own line. Re-indent other WHEN's to match. > + WHEN 2 THEN "DATA_CHAR" > + WHEN 3 THEN "DATA_FIXBINARY" > + WHEN 4 THEN "DATA_BINARY" > + WHEN 5 THEN "DATA_BLOB" > + WHEN 6 THEN "DATA_INT" > + WHEN 7 THEN "DATA_SYS_CHILD" > + WHEN 8 THEN "DATA_SYS" > + WHEN 9 THEN "DATA_FLOAT" > + WHEN 10 THEN "DATA_DOUBLE" > + WHEN 11 THEN "DATA_DECIMAL" > + WHEN 12 THEN "DATA_VARMYSQL" > + WHEN 13 THEN "DATA_MYSQL" > + WHEN 63 THEN "DATA_MTYPE_MAX" > + ELSE mtype > + END, > Add an alias e.g. AS base_type so that the result's column name isn't a truncated version of this CASE. > + IF((prtype & 512) = 512,"UNSIGNED","") AS is_unsigned > It's a bit weird to have is_unsigned be "UNSIGNED" or empty. Maybe YES/NO instead? Not a big deal I suppose. > FROM information_schema.INNODB_SYS_COLUMNS > WHERE name LIKE "t1\_%" > ORDER BY name; > > ---echo # Cleanup > - > DROP TABLE t1; > === modified file 'storage/innobase/dict/dict0stats.cc' > --- storage/innobase/dict/dict0stats.cc 2013-06-22 15:47:12 +0000 > +++ storage/innobase/dict/dict0stats.cc 2013-11-11 20:54:09 +0000 > @@ -176,7 +176,7 @@ > DATA_NOT_NULL, 192}, > > {"last_update", DATA_INT, > - DATA_NOT_NULL | DATA_UNSIGNED, 4}, > + DATA_NOT_NULL, 4}, > > {"n_rows", DATA_INT, > DATA_NOT_NULL | DATA_UNSIGNED, 8}, > @@ -207,7 +207,7 @@ > DATA_NOT_NULL, 192}, > > {"last_update", DATA_INT, > - DATA_NOT_NULL | DATA_UNSIGNED, 4}, > + DATA_NOT_NULL, 4}, > > {"stat_name", DATA_VARMYSQL, > DATA_NOT_NULL, 64*3}, > === modified file 'storage/innobase/handler/ha_innodb.cc' > --- storage/innobase/handler/ha_innodb.cc 2013-11-03 23:45:27 +0000 > +++ storage/innobase/handler/ha_innodb.cc 2013-11-11 20:54:09 +0000 > @@ -5745,67 +5745,94 @@ > 8 bits: this is used in ibuf and also when DATA_NOT_NULL is ORed to > the type */ > > - compile_time_assert((ulint)MYSQL_TYPE_STRING < 256); > - compile_time_assert((ulint)MYSQL_TYPE_VAR_STRING < 256); > - compile_time_assert((ulint)MYSQL_TYPE_DOUBLE < 256); > - compile_time_assert((ulint)MYSQL_TYPE_FLOAT < 256); > - compile_time_assert((ulint)MYSQL_TYPE_DECIMAL < 256); > - > - *unsigned_flag = 0; > - > - switch (field->key_type()) { > - case HA_KEYTYPE_USHORT_INT: > - case HA_KEYTYPE_ULONG_INT: > - case HA_KEYTYPE_UINT24: > - case HA_KEYTYPE_ULONGLONG: > + DBUG_ASSERT((ulint)MYSQL_TYPE_STRING < 256); > + DBUG_ASSERT((ulint)MYSQL_TYPE_VAR_STRING < 256); > + DBUG_ASSERT((ulint)MYSQL_TYPE_DOUBLE < 256); > + DBUG_ASSERT((ulint)MYSQL_TYPE_FLOAT < 256); > + DBUG_ASSERT((ulint)MYSQL_TYPE_DECIMAL < 256); > + > + if (field->flags & UNSIGNED_FLAG) { > + > *unsigned_flag = DATA_UNSIGNED; > - /* fall through */ > - case HA_KEYTYPE_SHORT_INT: > - case HA_KEYTYPE_LONG_INT: > - case HA_KEYTYPE_INT24: > - case HA_KEYTYPE_INT8: > - case HA_KEYTYPE_LONGLONG: > + } else { > + *unsigned_flag = 0; > + } > + > + if (field->real_type() == MYSQL_TYPE_ENUM > + || field->real_type() == MYSQL_TYPE_SET) { > + > + /* MySQL has field->type() a string type for these, but the > + data is actually internally stored as an unsigned integer > + code! */ > + > + *unsigned_flag = DATA_UNSIGNED; /* MySQL has its own unsigned > + flag set to zero, even though > + internally this is an unsigned > + integer type */ > return(DATA_INT); > - case HA_KEYTYPE_FLOAT: > - return(DATA_FLOAT); > - case HA_KEYTYPE_DOUBLE: > - return(DATA_DOUBLE); > - case HA_KEYTYPE_BINARY: > - if (field->type() == MYSQL_TYPE_TINY || > - field->real_type() == MYSQL_TYPE_ENUM || > - field->real_type() == MYSQL_TYPE_SET > - ) > - { // compatibility workaround > - *unsigned_flag= DATA_UNSIGNED; > - return DATA_INT; > - } > - return(DATA_FIXBINARY); > - case HA_KEYTYPE_VARBINARY2: > - if (field->type() != MYSQL_TYPE_VARCHAR) > - return(DATA_BLOB); > - /* fall through */ > - case HA_KEYTYPE_VARBINARY1: > - return(DATA_BINARY); > - case HA_KEYTYPE_VARTEXT2: > - if (field->type() != MYSQL_TYPE_VARCHAR) > - return(DATA_BLOB); > - /* fall through */ > - case HA_KEYTYPE_VARTEXT1: > - if (field->charset() == &my_charset_latin1) { > + } > + > + switch (field->type()) { > + /* NOTE that we only allow string types in DATA_MYSQL and > + DATA_VARMYSQL */ > + case MYSQL_TYPE_VAR_STRING: /* old <= 4.1 VARCHAR */ > + case MYSQL_TYPE_VARCHAR: /* new >= 5.0.3 true VARCHAR */ > + if (field->binary()) { > + return(DATA_BINARY); > + } else if (strcmp(field->charset()->name, > + "latin1_swedish_ci") == 0) { > return(DATA_VARCHAR); > } else { > return(DATA_VARMYSQL); > } > - case HA_KEYTYPE_TEXT: > - if (field->charset() == &my_charset_latin1) { > + case MYSQL_TYPE_BIT: > + case MYSQL_TYPE_STRING: if (field->binary()) { > + > + return(DATA_FIXBINARY); > + } else if (strcmp(field->charset()->name, > + "latin1_swedish_ci") == 0) { > return(DATA_CHAR); > } else { > return(DATA_MYSQL); > } > - case HA_KEYTYPE_NUM: > + case MYSQL_TYPE_NEWDECIMAL: > + return(DATA_FIXBINARY); > + case MYSQL_TYPE_LONG: > + case MYSQL_TYPE_LONGLONG: > + case MYSQL_TYPE_TINY: > + case MYSQL_TYPE_SHORT: > + case MYSQL_TYPE_INT24: > + case MYSQL_TYPE_DATE: > + case MYSQL_TYPE_YEAR: > + case MYSQL_TYPE_NEWDATE: > + return(DATA_INT); > + case MYSQL_TYPE_TIMESTAMP: > + *unsigned_flag = 0; > + /* fall through */ > + case MYSQL_TYPE_TIME: > + case MYSQL_TYPE_DATETIME: > + if (field->key_type() == HA_KEYTYPE_BINARY) > + return(DATA_FIXBINARY); > + else > + return(DATA_INT); > You also need to support MYSQL_TYPE_{TIMESTAMP,TIME,DATETIME}2 for MySQL 5.6 here. > + case MYSQL_TYPE_FLOAT: > + return(DATA_FLOAT); > + case MYSQL_TYPE_DOUBLE: > + return(DATA_DOUBLE); > + case MYSQL_TYPE_DECIMAL: > return(DATA_DECIMAL); > - case HA_KEYTYPE_BIT: > - case HA_KEYTYPE_END: > + case MYSQL_TYPE_GEOMETRY: > + case MYSQL_TYPE_TINY_BLOB: > + case MYSQL_TYPE_MEDIUM_BLOB: > + case MYSQL_TYPE_BLOB: > + case MYSQL_TYPE_LONG_BLOB: > + return(DATA_BLOB); > + case MYSQL_TYPE_NULL: > + /* MySQL currently accepts "NULL" datatype, but will > + reject such datatype in the next release. We will cope > + with it and not trigger assertion failure in 5.1 */ > + break; > + default: > ut_error; > } > > === modified file 'storage/innobase/handler/handler0alter.cc' > --- storage/innobase/handler/handler0alter.cc 2013-09-21 08:14:42 +0000 > +++ storage/innobase/handler/handler0alter.cc 2013-11-11 20:54:09 +0000 > @@ -347,6 +347,32 @@ > } > } > > + /* > + InnoDB in different MariaDB versions was generating different > mtype > + codes for certain types. Also signed/unsigned bit was generated > + differently too. > It may be more clear to merge these two sentences into one. > + > + Online ALTER would change the mtype/unsigned_flag (to what the > + current code generates) without changing the underlying data > + represenation, and it might result in data corruption. > + > + Don't do online ALTER if mtype/unsigned_flag are wrong. > + */ > + for (ulint i = 0; i < table->s->fields; i++) { > + const Field* field = table->field[i]; > + const dict_col_t* col = dict_table_get_nth_col(prebuilt->table, i); > + ulint unsigned_flag; > + if (col->mtype != get_innobase_type_from_mysql_type(&unsigned_flag, > field)) { > + > + DBUG_RETURN(HA_ALTER_INPLACE_NOT_SUPPORTED); > + } > + > + if ((col->prtype & DATA_UNSIGNED) != unsigned_flag) { > + > + DBUG_RETURN(HA_ALTER_INPLACE_NOT_SUPPORTED); > + } > + } > + > This looks reasonable to me. > /* We should be able to do the operation in-place. > See if we can do it online (LOCK=NONE). */ > bool online = true;

2 4

Re: [Maria-developers] maria-tokudb-engine-10.0 deb package (raring) - if files are empty where are sources?
by Sergei Golubchik 14 Nov '13

14 Nov '13

Hi, Jocelyn! On Nov 14, Jocelyn Fournier wrote: > Hi, > > Perhaps it would be great to mention in the 10.0.5 download page it's > not yet compatible with TokuDB. > I've tried myself to upgrade my server using MariaDB 5.5.33a with tokudb > to 10.0.5 and it failed because of this issue. I believe, only deb packages are affected. TokuDB is present in rpms, in the source and in the binary tarballs. But I see you point, we'll probably add some warning, thanks! Regards, Sergei

2 1

Re: [Maria-developers] MDEV-5220 - [PATCH] MariaDB 10.0.4 doesn't compile without perfschema
by Sergei Golubchik 14 Nov '13

14 Nov '13

Hi, Sergey! Generally, it's ok, of course. But as a matter of style (up to you to fix it here, but keep in mind for the future) - see below On Nov 13, Sergey Vojtovich wrote: > === modified file 'sql/mysqld.cc' > --- a/sql/mysqld.cc 2013-11-01 11:00:11 +0000 > +++ b/sql/mysqld.cc 2013-11-13 07:07:55 +0000 > @@ -760,6 +760,7 @@ char **orig_argv; > > static struct my_option pfs_early_options[] __attribute__((unused)) = I suppose, we don't need __attribute__((unused)) here, if the array is used unconditionally. > { > +#ifdef WITH_PERFSCHEMA_STORAGE_ENGINE > {"performance_schema_instrument", OPT_PFS_INSTRUMENT, > "Default startup value for a performance schema instrument.", > &pfs_param.m_pfs_instrument, &pfs_param.m_pfs_instrument, 0, GET_STR, > @@ -824,6 +825,7 @@ static struct my_option pfs_early_option > &pfs_param.m_consumer_statement_digest_enabled, > &pfs_param.m_consumer_statement_digest_enabled, 0, > GET_BOOL, OPT_ARG, TRUE, 0, 0, 0, 0, 0} > +#endif /* WITH_PERFSCHEMA_STORAGE_ENGINE */ > }; > > #ifdef HAVE_PSI_INTERFACE > @@ -1146,7 +1148,6 @@ void net_after_header_psi(struct st_net > > void init_net_server_extension(THD *thd) > { > -#ifdef HAVE_PSI_INTERFACE > /* Start with a clean state for connection events. */ > thd->m_idle_psi= NULL; > thd->m_statement_psi= NULL; > @@ -1157,9 +1158,6 @@ void init_net_server_extension(THD *thd) > thd->m_net_server_extension.m_after_header= net_after_header_psi; > /* Activate this private extension for the mysqld server. */ > thd->net.extension= & thd->m_net_server_extension; > -#else > - thd->net.extension= NULL; > -#endif > } > #endif /* EMBEDDED_LIBRARY */ > > @@ -4397,7 +4395,9 @@ static int init_thread_environment() > #ifdef HAVE_EVENT_SCHEDULER > Events::init_mutexes(); > #endif > +#ifdef HAVE_PSI_INTERFACE > init_show_explain_psi_keys(); > +#endif I usually define an no-op function and use it unconditionally, to avoid scattering #ifdefs in the code. > /* Parameter for threads created for connections */ > (void) pthread_attr_init(&connection_attrib); > (void) pthread_attr_setdetachstate(&connection_attrib, > @@ -5137,9 +5137,12 @@ int mysqld_main(int argc, char **argv) > buffered_logs.init(); > my_getopt_error_reporter= buffered_option_error_reporter; > my_charset_error_reporter= buffered_option_error_reporter; > +#ifdef WITH_PERFSCHEMA_STORAGE_ENGINE > pfs_param.m_pfs_instrument= const_cast<char*>(""); > > - int ho_error= handle_early_options(); > + int ho_error= > +#endif /* WITH_PERFSCHEMA_STORAGE_ENGINE */ > + handle_early_options(); Please, don't #ifdef part of a statement. Here I'd use __attribute__((unused)) to suppress a compiler warning. > #ifdef WITH_PERFSCHEMA_STORAGE_ENGINE > if (ho_error == 0) > Regards, Sergei

2 1

[Maria-developers] Thoughts on a variable to control high resolution temporal type format and rounding behavior?
by Jeremy Cole 13 Nov '13

13 Nov '13

Sergei, In the process of working through the bugs I've reported re. time formats and InnoDB storage formats we noted that there isn't currently any way to cause MariaDB to create a *new* Oracle MySQL 5.6-format TIMESTAMP/TIME/DATETIME field, and thus there's not really very good test coverage of them. Additionally the existence of this difference relative to Oracle MySQL is not really desirable. I would propose adding a new sysvar to control the temporal time format e.g. temporal_microsecond_format={mysql|mariadb} which controls whether newly created columns are created as Field_*_hires or Field_*f. This would especially allow: 1. Tests to be run in both Oracle MySQL and MariaDB modes. 2. A single test to create a column in one format and request conversion to the other. 3. A user to convert the format of their existing tables for compatibility. Additionally I would suggest a new sysvar to control the rounding behavior of the same temporal types, e.g. temporal_microsecond_rounding={round|truncate} which controls whether to round (as Oracle MySQL 5.6 does) or to truncate (as MariaDB does) when faced with fitting a larger number of decimals in a value into a smaller number decimals in a field. (This of course assumes that the current MariaDB behavior and divergence from Oracle MySQL behavior is intentional, and not a bug. Even if it is considered a bug it could probably not be changed outright at this point and the sysvar would still be needed.) Any thoughts on either of these? We would of course be willing to do the work and provide the patches for them. I just wanted to get your general feelings on the ideas. Regards, Jeremy

3 6

[Maria-developers] EXPLAIN in the slow query log
by Sergei Petrunia 12 Nov '13

12 Nov '13

Hello, After recent changes in the [SHOW] EXPLAIN architecture in MariaDB, it became really easy to add an option to print EXPLAIN(*) output into the slow query log, so I went ahead and implemented it. @@log_slow_verbosity has got a 'explain' flag, and if it is on, the slow query log entries look like this: # Time: 131017 21:26:02 # User@Host: root[root] @ localhost [] # Thread_id: 2 Schema: dbt3sf1 QC_hit: No # Query_time: 5.535819 Lock_time: 0.000350 Rows_sent: 1 Rows_examined: 65633 ## <explain> ## id select_type table type possible_keys key key_len ref rows Extra ## 1 SIMPLE nation ref PRIMARY,n_name n_name 26 const 1 Using where; Using index ## 1 SIMPLE customer ref PRIMARY,i_c_nationkey i_c_nationkey 5 dbt3sf1.nation.n_nationkey 25124 Using index ## 1 SIMPLE orders ref i_o_custkey i_o_custkey 5 dbt3sf1.customer.c_custkey 7 Using index ## </explain> SET timestamp=1382030762; select count(*) from customer, orders, nation where c_custkey=o_custkey and c_nationkey=n_nationkey and n_name='GERMANY'; It seems, pt-slow-query-log has no problem parsing this. Open questions: - Is this syntax OK? - Currently, some statements (like SHOW KEYS) print a dumb explain that shows that full table scan is used. There is a suggestion to not print EXPLAIN for any SHOW statements. Is this ok? (*) something like EXPLAIN ANALYZE, with numbers from actual execution, would be better but we are not there yet. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

5 6

[Maria-developers] A newer InnoDB in 10.0 series?
by Colin Charles 12 Nov '13

12 Nov '13

Hi! I see that we are shipping InnoDB from 5.6.5 (at least that's what the plugin version tells me - 1.2.5). Are we going to pop XtraDB or a modern InnoDB into 10.0.6? Thanks -- Colin Charles, Chief Evangelist MariaDB | t: +6-012-204-3201 | Skype: colincharles

3 3

Re: [Maria-developers] [Commits] Rev 3886: MDEV-4439 ALTER TABLE .. [ADD|DROP] FOREIGN KEY IF [NOT] EXISTS does not work if constraint name is not used. in file:///home/hf/wmar/10exp/
by Jan Lindström 11 Nov '13

11 Nov '13

Hi, MySQL changes look good, InnoDB changes need some work that I will do, please do not push those. R: Jan > At file:///home/hf/wmar/10exp/ > > ------------------------------------------------------------ > revno: 3886 > revision-id: holyfoot(a)askmonty.org-20131108110551-j0rxpxrnfertay2q > parent: knielsen(a)knielsen-hq.org-20131107065240-gxxg4ilcs8o2v0q2 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: 10exp > timestamp: Fri 2013-11-08 15:05:51 +0400 > message: > MDEV-4439 ALTER TABLE .. [ADD|DROP] FOREIGN KEY IF [NOT] EXISTS does not work if constraint name is not used. > Patches for server and the Innodb engine. > Server is fixed so it does nothing if no indexes left to alter. > Innodb parser is fixed so it looks for the IF [NOT] EXISTS option, > another change is that it uses the index name for the internal dictionary. > Prior to that it only used the CONSTRAINT name for it. > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- -- Jan Lindström Principal Engineer MariaDB | MaxScale | skype: jan_p_lindstrom www.skysql.com <http://www.skysql.com/> Twitter <http://twitter.com/skysql> Blog <http://www.skysql.com/blog/> Facebook <http://www.facebook.com/skysql> LinkedIn <http://www.linkedin.com/company/1214250> Google+ <https://plus.google.com/117544963211695643458/posts>

1 0

Re: [Maria-developers] patch to upgrade collations
by Michael Widenius 08 Nov '13

08 Nov '13

Hi! >>>>> "Alexander" == Alexander Barkov <bar(a)mariadb.org> writes: Alexander> Hi Monty, Alexander> Please find the full patch attached. Alexander> Greetings. Here is a modified patch that also fixes that REPAIR works. It doesn't include the myisam files that you had in your patch, but it will of course need these. I have verified that all test cases you changed works. Regards, Monty

1 0

[Maria-developers] MDEV-4954
by Sergei Golubchik 08 Nov '13

08 Nov '13

Hi, Oden could you please take a look at MDEV-4954? https://mariadb.atlassian.net/browse/MDEV-4954 "RPM Packaging Removes Config File" The complain is that when one uninstalls MariaDB, /etc/my.cnf gets "removed" (actually, renamed to /etc/my.cnf.rpmsave). And MySQL packages don't do that (because they don't include my.cnf at all). Anything we can do to make the user happier? Regards, Sergei

1 0

[Maria-developers] Debian/Ubuntu packages
by Matt Coleman 07 Nov '13

07 Nov '13

Hello, I’m working on fixing the dependency issues with MariaDB 5.5.33a vs MySQL 5.5.34. When I’m ready to publish my fix, what version number should I give it? The current packages have the version 5.5.33a+maria-1~CODENAME, where CODENAME is the code name for the Debian or Ubuntu release (ie. “sid” or “precise”, etc). Since I am not changing any code, it seems like like 5.5.33a should remain the same. I expected that the fixed package would be 5.5.33a+maria-2~CODENAME, but autobake-deb.sh assembles that version string and the 1 is hardcoded, not set by a variable. The two variables RELEASE_NAME and RELEASE_EXTRA can insert text between “+maria-“ and “1”. What is this project’s versioning convention in regards to changes that only affect the packaging? Matt

2 2

Re: [Maria-developers] [Commits] Rev 3942: MDEV-5205 - MariaDB does not start if more than 128 cpu's are available
by Sergei Golubchik 06 Nov '13

06 Nov '13

Hi, Sergey! On Nov 06, Sergey Vojtovich wrote: > ------------------------------------------------------------ > revno: 3942 > revision-id: svoj(a)mariadb.org-20131106185339-wjz3xuru8ojnz00l > parent: monty(a)askmonty.org-20131105183036-051tak9a2fq9ii05 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5-mdev5205 > timestamp: Wed 2013-11-06 22:53:39 +0400 > message: > MDEV-5205 - MariaDB does not start if more than 128 cpu's are available > > An addition to fix for MDEV-5205, fixes server crash on shutdown. > > Thread groups are destroyed asynchronously, that is kill server > thread sends shutdown request to all thread groups without waiting > for compeltion. > > It means all_groups array must not be freed until all thread groups > are down. This patch suggests that all_groups is freed when last > thread group is destroyed. Looks reasonable. ok to push Regards, Sergei

1 0

[Maria-developers] MariaDB 10.0.5 Beta Release Prep
by Daniel Bartholomew 05 Nov '13

05 Nov '13

I'm currently in the process of preparing for the MariaDB 10.0.5 Beta release. Draft changelog and release notes are here: - https://mariadb.com/kb/en/mariadb-1005-release-notes/ - https://mariadb.com/kb/en/mariadb-1005-changelog/ The release notes page is not much more than a template at present and needs to be be filled in before the release. Please add any items that should be there ASAP. Current plans are to activate and announce the release tomorrow (6 Nov). Thanks. -- Daniel Bartholomew MariaDB

1 0

[Maria-developers] Opened_plugin_libraries status variable
by Ian Gilfillan 05 Nov '13

05 Nov '13

Hi I would like to document the Opened_plugin_libraries status variable, introduced in MariaDB 10.0, but can't understand what it's all about. I have looked at: http://lists.askmonty.org/pipermail/commits/2013-April/004595.html but it's not really making sense what it's testing. In my own setup, each time I query the INFORMATION_SCHEMA.ALL_PLUGINS table, Opened_plugin_libraries increments by 19 (no matter what conditions I apply). The only correlation I can find is that there are 19 records with the LOAD_OPTION = 'OFF'. Is this what Opened_plugin_libraries is recording? Why does it increment each time I query the table (19, 38, 57 etc) - how is this useful? Can someone clarify exactly what this variable records, and how it's useful? thanks, ian

2 1

[Maria-developers] On how to fix indentation and history in sql_join_cache.cc
by Sergei Petrunia 05 Nov '13

05 Nov '13

Hi Sanja, My experiment was successfull. It seems, one can fix identation in sql_join_cache.cc so in the following way: # Create a branch in a state "right after the problem" bzr branch -rsanja(a)askmonty.org-20121223213711-ac5vz2fspwapjvtr 10.0 work-maria-10.0-base-cassandra-look1 # The -r3457 branch shows the state "right before the merge that borked it" bzr branch -r3457 work-maria-10.0-base-cassandra-look1 work-maria-10.0-base-cassandra-look1-r3457 cd work-maria-10.0-base-cassandra-look1-r3457 # Do some dummy change to force a merge gvim sql/sql_join_cache.cc bzr gcommit # Prepare the branch where we will fix it cd .. bzr branch work-maria-10.0-base-cassandra-look1 work-maria-10.0-base-cassandra-look1-cp cd work-maria-10.0-base-cassandra-look1-cp # This should show that you're going to merge the dummy change bzr missing --theirs-only ../work-maria-10.0-base-cassandra-look1-r3457/ # # This will will succeed without conflicts. We will use this merge as an # excuse to fix the problem # bzr merge --pull ../work-maria-10.0-base-cassandra-look1-r3457/ # # Now, edit the "broken" file and the original file until sql/sql_join_cache.cc # is ok # gvimdiff ~/home/psergey/dev2/work-maria-10.0-base-cassandra-look1-r3457/sql/sql_join_cache.cc sql/sql_join_cache.cc # Commit bzr gcommit # # Merge the change into a recent 10.0-base. # cd 10.0-base bzr merge --pull ../work-maria-10.0-base-cassandra-look1-cp/ bzr gcommit # # Check the history. It MUST NOT show any of today's merges. # the code in bkah_skip_index_tuple() should show that lines belong to # # igor(a)askmonty.org-20100831163421-iiddbral4jr3q75q # The main patch for MWL#128: Implement Block Nested Loop Hash Join. # # sanja(a)montyprogram.com-20120216064910-k0ud8ja7ef1jnpx4 # Counters for Index Condition Pushdown added (MDEV-130). bzr gannotate sql/sql_join_cache.cc BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 2

[Maria-developers] Preparing MDEV-4506, Parallel replication, for 10.0.5 release
by Kristian Nielsen 04 Nov '13

04 Nov '13

Hi Monty, I was asked to prepare MDEV-4506, parallel replication, for release of 10.0.5 on Monday. For this, I did merge to 10.0-base and then to 10.0. I have pushed the results here: lp:~maria-captains/maria/mdev-4506-base # for 10.0-base merge lp:~maria-captains/maria/mdev-4506 # for 10.0 merge However, I had problems with a couple of your changes, to temporary tables and innodb log file size. The code that you modified in 10.0-base does not seem to exist in 10.0 (or at least I couldn't find it easily). Can you take a look at this and see if you can complete the merge? The missing bits are just the few hunks appended at the end of this mail, so I assume it should be easy for someone who knows what the patches are supposed to do. The tree lp:~maria-captains/maria/mdev-4506 currently has a lot of test failures, but it looks like most of them will be fixed by completing the last bits of the merge. - Kristian. ----------------------------------------------------------------------- sql/sql_base.cc: <<<<<<< TREE if (table_list->mdl_request.type >= MDL_SHARED_WRITE && thd->tx_read_only && !(flags & (MYSQL_LOCK_LOG_TABLE | MYSQL_OPEN_HAS_MDL_LOCK))) ||||||| BASE-REVISION if (table_list->open_type != OT_BASE_ONLY && ! (flags & MYSQL_OPEN_SKIP_TEMPORARY)) ======= if (table_list->open_type != OT_BASE_ONLY && ! (flags & MYSQL_OPEN_SKIP_TEMPORARY) && thd->have_temporary_tables()) >>>>>>> MERGE-SOURCE { <<<<<<< TREE my_error(ER_CANT_EXECUTE_IN_READ_ONLY_TRANSACTION, MYF(0)); DBUG_RETURN(true); ||||||| BASE-REVISION for (table= thd->temporary_tables; table ; table=table->next) { if (table->s->table_cache_key.length == key_length + TMP_TABLE_KEY_EXTRA && !memcmp(table->s->table_cache_key.str, key, key_length + TMP_TABLE_KEY_EXTRA)) { /* We're trying to use the same temporary table twice in a query. Right now we don't support this because a temporary table is always represented by only one TABLE object in THD, and it can not be cloned. Emit an error for an unsupported behaviour. */ if (table->query_id) { DBUG_PRINT("error", ("query_id: %lu server_id: %u pseudo_thread_id: %lu", (ulong) table->query_id, (uint) thd->variables.server_id, (ulong) thd->variables.pseudo_thread_id)); my_error(ER_CANT_REOPEN_TABLE, MYF(0), table->alias.c_ptr()); DBUG_RETURN(TRUE); } table->query_id= thd->query_id; thd->thread_specific_used= TRUE; DBUG_PRINT("info",("Using temporary table")); goto reset; } } ======= if ((table= find_temporary_table(thd, key, key_length + TMP_TABLE_KEY_EXTRA))) { /* Check if we're trying to use the same temporary table twice in a query. Right now we don't support this because a temporary table is always represented by only one TABLE object in THD, and it can not be cloned. Emit an error for an unsupported behaviour. */ if (table->query_id) { DBUG_PRINT("error", ("query_id: %lu server_id: %u pseudo_thread_id: %lu", (ulong) table->query_id, (uint) thd->variables.server_id, (ulong) thd->variables.pseudo_thread_id)); my_error(ER_CANT_REOPEN_TABLE, MYF(0), table->alias.c_ptr()); DBUG_RETURN(TRUE); } table->query_id= thd->query_id; thd->thread_specific_used= TRUE; DBUG_PRINT("info",("Using temporary table")); goto reset; } >>>>>>> MERGE-SOURCE sql/slave.cc: <<<<<<< TREE static void set_thd_in_use_temporary_tables(Relay_log_info *rli) { TABLE *table; for (table= rli->save_temporary_tables ; table ; table= table->next) { table->in_use= rli->sql_thd; if (table->file != NULL) { /* Since we are stealing opened temporary tables from one thread to another, we need to let the performance schema know that, for aggregates per thread to work properly. */ table->file->unbind_psi(); table->file->rebind_psi(); } } } ||||||| BASE-REVISION static void set_thd_in_use_temporary_tables(Relay_log_info *rli) { TABLE *table; for (table= rli->save_temporary_tables ; table ; table= table->next) table->in_use= rli->sql_thd; } ======= >>>>>>> MERGE-SOURCE === modified file 'storage/innobase/srv/srv0start.c' --- storage/innobase/srv/srv0start.c 2013-09-06 20:31:30 +0000 +++ storage/innobase/srv/srv0start.c 2013-11-01 08:17:06 +0000 @@ -621,7 +621,7 @@ open_or_create_log_file( || size_high != srv_calc_high32(srv_log_file_size)) { fprintf(stderr, - "InnoDB: Error: log file %s is" + "InnoDB: Warning: log file %s is" " of different size %lu %lu bytes\n" "InnoDB: than specified in the .cnf" " file %lu %lu bytes!\n", @@ -629,7 +629,8 @@ open_or_create_log_file( (ulong) srv_calc_high32(srv_log_file_size), (ulong) srv_calc_low32(srv_log_file_size)); - return(DB_ERROR); + srv_log_file_size= size + + (((longlong) size_high) << 32); } } else { *log_file_created = TRUE;

2 3

Re: [Maria-developers] MDEV-5205 - MariaDB does not start if more than 128 cpu's are available
by Sergei Golubchik 01 Nov '13

01 Nov '13

Hi, Sergey! On Nov 01, Sergey Vojtovich wrote: > revno: 3939 > revision-id: svoj(a)mariadb.org-20131101154841-egjkqc4e00mxc3ty > parent: sanja(a)montyprogram.com-20131029165036-f4wqcr2nrxzuwuuv > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5-mdev5205 > timestamp: Fri 2013-11-01 19:48:41 +0400 > message: > MDEV-5205 - MariaDB does not start if more than 128 cpu's are available > > === modified file 'sql/threadpool_unix.cc' > --- a/sql/threadpool_unix.cc 2013-06-22 12:02:03 +0000 > +++ b/sql/threadpool_unix.cc 2013-11-01 15:48:41 +0000 > @@ -907,6 +907,7 @@ int thread_group_init(thread_group_t *th > thread_group->pollfd= -1; > thread_group->shutdown_pipe[0]= -1; > thread_group->shutdown_pipe[1]= -1; > + thread_group->queue.empty(); ok, but please add to the definition of I_P_List::empty() a comment that it's used in many places in the code instead of a constructor, to initialize a bzero-ed I_P_List instance. > DBUG_RETURN(0); > } > Regards, Sergei

1 0

[Maria-developers] doubt, Got a packet bigger than 'max_allowed_packet' bytes - 1153
by Roberto Spadim 01 Nov '13

01 Nov '13

Hi guys, i'm with a doubt now... using the php extension of mysql i got this error Got a packet bigger than 'max_allowed_packet' bytes - 1153 the point is... should the client api change the packet to a size smaller than max limit? or should i set the max_allowed_packet at server side? this variable have some memory problem if changed to a big value? today i'm using 1MB, but i don't know if i change for example to 1024MB i could break the server, does the max_allowed_packet just a limit, or a malloc parameter to alloc memory at each packet? -- Roberto Spadim SPAEmpresarial

1 1

Re: [Maria-developers] MWL83: review
by Sergei Petrunia 31 Oct '13

31 Oct '13

Hi Timour, Found another possible issue: The code in best_extension_by_limited_search() calls join->static_pushdown_cost(). static_pushdown_cost() may modify join->position[$I].read_time for any $I. Now, consider an example: SELECT * FROM t1,t2,t3 WHERE subquery_conditions The join optimizer will consider join orders: S1: t1 S2: t1 t2 S3: t1 t2 t3 S4: t1 t3 t2 On step S3, we will get a complete join order, and the new code in best_extension_by_limited_search() is invoked. It will call join->static_pushdown_cost(...) which may change the value of join->position[0].read_time On step S4, we again will get a complete join order. The new code in best_extension_by_limited_search() is invoked, it will call join->static_pushdown_cost(...), which again may change join->position[0].read_time Now, that value will be meaningless. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] Rev 3711: MDEV-5104 crash in Item_field::used_tables with broken order by in file:///home/tsk/mprog/src/5.3/
by Oleksandr Byelkin 29 Oct '13

29 Oct '13

Hi! OK to push! Thank you! 29.10.2013 12:39, timour(a)askmonty.org пишет: > At file:///home/tsk/mprog/src/5.3/ > > ------------------------------------------------------------ > revno: 3711 > revision-id: timour(a)askmonty.org-20131029103903-bvjnf3b2aqnoyut5 > parent: sanja(a)askmonty.org-20131021104549-857lwynmy813qyd4 > fixes bug: https://mariadb.atlassian.net/browse/MDEV-5104 > committer: timour(a)askmonty.org <timour(a)askmonty.org> > branch nick: 5.3 > timestamp: Tue 2013-10-29 12:39:03 +0200 > message: > MDEV-5104 crash in Item_field::used_tables with broken order by > > Analysis: > st_select_lex_unit::prepare() computes can_skip_order_by as TRUE. > As a result join->prepare() gets called with order == NULL, and > doesn't do name resolution for the inner ORDER clause. Due to this > the prepare phase doesn't detect that the query references non-exiting > function and field. > > Later join->optimize() calls update_used_tables() for a non-resolved > Item_field, which understandably has no Field object. This call results > in a crash. > > Solution: > Resolve unnecessary ORDER BY clauses to detect if they reference non-exising > objects. Then remove such clauses from the JOIN object. > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] MWL83: review
by Sergei Petrunia 29 Oct '13

29 Oct '13

Hi Timour, This is the first portion of the feedback. A general note about EXPLAIN output: "Subqueries: 2" looks ugly. A human will read this as " there are two subqueries somewhere", which is clearly not the case. All other lists (indexes for index merge, ref components) are comma-separated, while list of subquery ids uses spaces. As far as I understand, @@optimizer_switch flags and other controls for dynamic condition are present, but the feature is not working? We can't push it in this state, because this will be very confusing for anybody trying to use the new features. Further comments below marked with 'psergey:' > === modified file 'sql/item.cc' > --- sql/item.cc 2013-08-20 11:48:29 +0000 > +++ sql/item.cc 2013-09-27 08:25:14 +0000 > @@ -457,7 +457,8 @@ int Item::save_str_value_in_field(Field > Item::Item(): > is_expensive_cache(-1), rsize(0), name(0), orig_name(0), name_length(0), > fixed(0), is_autogenerated_name(TRUE), > - collation(&my_charset_bin, DERIVATION_COERCIBLE) > + collation(&my_charset_bin, DERIVATION_COERCIBLE)/*TODO, > + pushdown_tables(0)*/ > { > marker= 0; > maybe_null=null_value=with_sum_func=with_field=unsigned_flag=0; > @@ -466,7 +467,7 @@ int Item::save_str_value_in_field(Field > with_subselect= 0; > cmp_context= IMPOSSIBLE_RESULT; > /* Initially this item is not attached to any JOIN_TAB. */ > - join_tab_idx= MAX_TABLES; > + join_tab_idx= -1; > > /* Put item in free list so that we can free all items at end */ > THD *thd= current_thd; > @@ -515,7 +516,8 @@ int Item::save_str_value_in_field(Field > is_autogenerated_name(item->is_autogenerated_name), > with_subselect(item->has_subquery()), > collation(item->collation), > - cmp_context(item->cmp_context) > + cmp_context(item->cmp_context)/*TODO, > + pushdown_tables(0)*/ > { > next= thd->free_list; // Put in free list > thd->free_list= this; > @@ -583,7 +585,7 @@ void Item::cleanup() > DBUG_PRINT("enter", ("this: %p", this)); > fixed=0; > marker= 0; > - join_tab_idx= MAX_TABLES; > + join_tab_idx= -1; > if (orig_name) > name= orig_name; > DBUG_VOID_RETURN; > @@ -605,6 +607,19 @@ bool Item::cleanup_processor(uchar *arg) > > > /** > + Add the cost of an item to the cost of previous items in the > + containing expression tree. > + > + @param arg the cost of the previously visited items in the exression tree > +*/ > + > +bool Item::sum_cost_processor(uchar *arg) > +{ > + *((double*)arg)= *((double*)arg) + get_cost(); > + return false; > +} > + > +/** > rename item (used for views, cleanup() return original name). > > @param new_name new name of item; > @@ -8719,6 +8734,17 @@ void Item_cache::set_null() > } > > > +void Item_cache::update_used_tables() > +{ > + example->update_used_tables(); > + if (example->const_item()) > + { > + used_table_map= 0; > + maybe_null= example->maybe_null; > + } > +} > + > + > bool Item_cache_int::cache_value() > { > if (!example) > @@ -9649,7 +9675,7 @@ table_map Item_ref_null_helper::used_tab > #ifndef DBUG_OFF > > /* Debugger help function */ > -static char dbug_item_print_buf[256]; > +static char dbug_item_print_buf[2048]; > > const char *dbug_print_item(Item *item) > { > > === modified file 'sql/item.h' > --- sql/item.h 2013-07-17 19:24:29 +0000 > +++ sql/item.h 2013-09-27 13:22:36 +0000 > @@ -578,10 +578,12 @@ class Item { > make_cond_for_table procedure. During query execution, this item is > evaluated when the join loop reaches the corresponding JOIN_TAB. > > - If the value of join_tab_idx >= MAX_TABLES, this means that there is no > - corresponding JOIN_TAB. > + If join_tab_idx < 0, this means that there is no corresponding JOIN_TAB. > + > + cur_join_tab_idx is used during join optimization to determine the best > + position where to push an expression. > */ > - uint join_tab_idx; > + int join_tab_idx; > > public: > static void *operator new(size_t size) throw () > @@ -658,6 +660,9 @@ class Item { > subselect */ > DTCollation collation; > Item_result cmp_context; /* Comparison context */ > + /* tables in the QEP where this predicate should be pushed to */ > + //table_map pushdown_tables; > + psergey: It seems, you've introduced the above member, but then decided not to use it? Please remove all leftover comments, then. > // alloc & destruct is done as start of select using sql_alloc > Item(); > /* > @@ -1006,6 +1011,7 @@ class Item { > have to be updated in update_used_tables() > */ > virtual table_map not_null_tables() const { return used_tables(); } > + > /* > Returns true if this is a simple constant item like an integer, not > a constant expression. Used in the optimizer to propagate basic constants. > @@ -1172,12 +1178,14 @@ class Item { > virtual bool remove_fixed(uchar * arg) { fixed= 0; return 0; } > virtual bool cleanup_processor(uchar *arg); > virtual bool collect_item_field_processor(uchar * arg) { return 0; } > + virtual bool collect_item_subselect_processor(uchar * arg) { return 0; } > virtual bool add_field_to_set_processor(uchar * arg) { return 0; } > virtual bool find_item_in_field_list_processor(uchar *arg) { return 0; } > virtual bool find_item_processor(uchar *arg); > virtual bool change_context_processor(uchar *context) { return 0; } > virtual bool reset_query_id_processor(uchar *query_id_arg) { return 0; } > virtual bool is_expensive_processor(uchar *arg) { return 0; } > + virtual bool sum_cost_processor(uchar *arg); > virtual bool register_field_in_read_map(uchar *arg) { return 0; } > virtual bool register_field_in_write_map(uchar *arg) { return 0; } > virtual bool enumerate_field_refs_processor(uchar *arg) { return 0; } > @@ -1196,6 +1204,22 @@ class Item { > virtual bool find_selective_predicates_list_processor(uchar *opt_arg) > { return 0; } > > + /** > + Set the join tab index to the minimal (left-most) JOIN_TAB to which this > + Item is attached. The number is an index in depth_first_tab() traversal > + order. > + */ > + virtual bool set_join_tab_idx_processor(uchar *join_tab_idx_arg) > + { > + /* > + Items can participate in several pushdown conditions attached to > + different join tabs. The join tab index property identifies the > + left-most join tab in the join plan. > + */ > + set_join_tab_idx(*((int*) join_tab_idx_arg)); > + return false; > + } > + > /* To call bool function for all arguments */ > struct bool_func_call_args > { > @@ -1450,6 +1474,32 @@ class Item { > is_expensive_cache= walk(&Item::is_expensive_processor, 0, (uchar*)0); > return test(is_expensive_cache); > } > + /** > + The default cost to evaluate a 'cheap' predicate. > + TODO: > + Notice that this also adds the cost of accessing Fields (via Item_field). > + Perhaps this cost should be smaller or 0. > + */ > + virtual double get_cost() { return (1 / (double)TIME_FOR_COMPARE); } > + /* The default execution-time cost of an Item is its compile-time cost. */ > + virtual double exec_cost() { return get_cost(); } psergey: I'm reading the above and I don't understand get_cost() and exec_cost(). > + > + > + /** > + Return true if this Item must be wrapped in a dynamic condition. > + */ > + bool needs_dynamic_cond() > + { > + /* > + @todo Do not check if a subquery is expensive, because even subqueries > + that the optimizer reports to be cheap, are actually quite expensive to > + re-evaluate multiple times. For instance the scalar SQ of DBT3-Q20 > + reports only ~7 examined rows, but still it is very beneficial to move > + it to later JOIN_TAB in the query plan. > + */ > + return (with_subselect && !const_item() /* && is_expensive() */); > + } > + > virtual Field::geometry_type get_geometry_type() const > { return Field::GEOM_GEOMETRY; }; > String *check_well_formed_result(String *str, bool send_error= 0); > @@ -1488,12 +1538,12 @@ class Item { > Item is attached. The number is an index is depth_first_tab() traversal > order. > */ > - virtual void set_join_tab_idx(uint join_tab_idx_arg) > + virtual void set_join_tab_idx(int join_tab_idx_arg) > { > - if (join_tab_idx_arg < join_tab_idx) > + if (join_tab_idx < 0 || join_tab_idx_arg < join_tab_idx) > join_tab_idx= join_tab_idx_arg; > } > - virtual uint get_join_tab_idx() { return join_tab_idx; } > + virtual int get_join_tab_idx() { return join_tab_idx; } > > table_map view_used_tables(TABLE_LIST *view) > { > @@ -4065,6 +4115,7 @@ class Item_cache: public Item_basic_cons > } > > void set_used_tables(table_map map) { used_table_map= map; } > + void update_used_tables(); > > virtual bool allocate(uint i) { return 0; } > virtual bool setup(Item *item) > @@ -4443,4 +4494,8 @@ class Item_iterator_row: public Item_ite > void close() {} > }; > > +#ifndef DBUG_OFF > +const char *dbug_print_item(Item *item); > +#endif /*DBUG_OFF*/ > + > #endif /* SQL_ITEM_INCLUDED */ > > === modified file 'sql/item_cmpfunc.cc' > --- sql/item_cmpfunc.cc 2013-08-20 11:48:29 +0000 > +++ sql/item_cmpfunc.cc 2013-09-27 08:25:14 +0000 > @@ -461,7 +461,17 @@ static bool convert_const_to_int(THD *th > field_item->field_type() != MYSQL_TYPE_YEAR) > return 1; > > - if ((*item)->const_item() && !(*item)->is_expensive()) > + /* > + An item can be converted to an int if: > + - is constant, and > + - it is not a subquery (or contains a subquery), and > + - it is cheap to compute. > + The reason for the second requirement is to prevent subquery > + optimization/execution during the JOIN::prepare phase, because this > + function is called during the prepare phase as well. > + */ > + if ((*item)->const_item() && > + !(*item)->with_subselect && !(*item)->is_expensive()) > { > TABLE *table= field->table; > ulonglong orig_sql_mode= thd->variables.sql_mode; > @@ -1478,7 +1488,8 @@ bool Item_in_optimizer::fix_left(THD *th > eval_not_null_tables(NULL); > with_sum_func= args[0]->with_sum_func; > with_field= args[0]->with_field; > - if ((const_item_cache= args[0]->const_item())) > + if (!(args[0]->with_subselect || args[0]->is_expensive()) && > + (const_item_cache= args[0]->const_item())) > { > cache->store(args[0]); > cache->cache_value(); > @@ -4685,6 +4696,17 @@ void Item_cond::neg_arguments(THD *thd) > } > > > +double Item_cond::exec_cost() > +{ > + List_iterator_fast<Item> li(list); > + Item *item; > + double cost= 0; > + while ((item= li++)) > + cost+= item->exec_cost(); > + return cost; > +} > + > + > void Item_cond_and::mark_as_condition_AND_part(TABLE_LIST *embedding) > { > List_iterator<Item> li(list); > @@ -6386,3 +6408,295 @@ longlong Item_func_dyncol_exists::val_in > null_value= TRUE; > return 0; > } > + > + > +double Item_func_collect_stats::exec_cost() > +{ > + return args[0]->exec_cost(); > +} > + > + > +double Item_func_collect_stats::exec_selectivity() > +{ > + return count_evals ? count_true / count_evals : 0; > +} > + > + > +#ifndef DBUG_OFF > +void Item_func_collect_stats::dbug_print_stats() > +{ > + char buff[1024]; > + String str(buff,(uint32) sizeof(buff), system_charset_info); > + str.length(0); > + str.extra_allocation(1024); > + print(&str, QT_ORDINARY); > + > + (void) fprintf(DBUG_FILE, " sel: %2.2f, cond_evals: %f, cond_true: %f (", > + exec_selectivity(), count_evals, count_true); > + (void) fputs(str.c_ptr_safe(),DBUG_FILE); > + (void) fprintf(DBUG_FILE, ")\n"); > + > + if (is_cond_and(args[0])) > + { > + List_iterator<Item> and_parts_it(*((Item_cond*) args[0])->argument_list()); > + Item *and_part; > + (void) fprintf(DBUG_FILE, " Individual conjunct selectivity: {\n"); > + while ((and_part= and_parts_it++)) > + { > + if (and_part->type() == Item::FUNC_ITEM && > + (((Item_func*)and_part)->functype() == Item_func::DYNAMIC_COND_FUNC || > + ((Item_func*)and_part)->functype() == Item_func::EXEC_STATS_FUNC)) > + { > + Item_func_collect_stats *stat_cond= (Item_func_collect_stats*) and_part; > + stat_cond->dbug_print_stats(); > + } > + } > + (void) fprintf(DBUG_FILE, "}\n"); > + } > +} > +#endif /*DBUG_OFF*/ > + > + > +Item_func_dynamic_cond::Item_func_dynamic_cond(Item *cond, JOIN_TAB *atab, JOIN_TAB **ctab) > + : Item_func_collect_stats(cond) > +{ > + JOIN *join= atab->join; > + JOIN_TAB *cur_tab; > + THD *thd= atab->table->in_use; > + > + active_tab= atab; > + exec_tab= ctab; > + max_evals= thd->variables.dynamic_pushdown_max_evals; > + cost_ratio= (double) thd->variables.dynamic_pushdown_cost_ratio / 100.0; > + sampling_probability= (double) thd->variables.dynamic_pushdown_sampling_probability / 100.0; > + max_selectivity= (double) thd->variables.dynamic_pushdown_max_selectivity / 100.0; > + > + /* Collect the set of all subqueries referenced inside 'cond'. */ > + cond->walk(&Item::collect_item_subselect_processor, 0, > + (uchar*) &subqueries); > + > + /* Find the boundaries between which the predicate can be moved. */ > + if (active_tab->bush_root_tab) > + { > + first_tab= active_tab->bush_root_tab->bush_children->start; > + last_tab= active_tab->bush_root_tab->bush_children->end; > + } > + else > + { > + first_tab= join->join_tab + join->const_tables; > + last_tab= join->join_tab + join->top_join_tab_count; > + } > + > + DBUG_ASSERT(active_tab >= first_tab && active_tab < last_tab); > + > + /* Precompute some cost information that is static at execution time. */ > + first_tab->exec_fanout= 1; > + for (cur_tab= first_tab; cur_tab < last_tab; cur_tab++) > + rel_access_cost+= cur_tab->read_time; > + rel_access_cost/= first_tab->records_read; > + > + /* Copied from Item_func_rand::seed_random, what are the right args? */ > + my_rnd_init(&rand, (uint32) (0x10001L+55555555L), > + (uint32) (0x10000001L)); > + last_res= -1; > +} > + > + > +longlong Item_func_dynamic_cond::val_int() > +{ > + longlong res= 1; > + THD *thd= active_tab->join->thd; > + > + DBUG_ASSERT(active_tab >= first_tab && active_tab < last_tab); > + DBUG_ASSERT(!(*exec_tab) || (active_tab->join == (*exec_tab)->join)); > + DBUG_ASSERT(!(*exec_tab) || (*exec_tab >= first_tab && *exec_tab < last_tab)); > + > + if (is_active()) > + { > + /* > + If the active tab is the same as the first tab, this method must > + re-evaluate on each call (that is, not use the cached last_res). > + */ > + DBUG_ASSERT(*exec_tab > first_tab || last_res == -1); > + /* > + If this dynamic condition is attached to the JOIN_TAB currently being > + executed, evaluate the wrapper condition. > + */ > + if (last_res == -1) > + { > + last_res= Item_func_collect_stats::val_int(); > + /* Choose the best JOIN_TAB where to attach this dynamic condition. */ > + if (optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_DYNAMIC_PUSHDOWN) && > + ((ulonglong) count_evals % max_evals == 0)) > + choose_best_join_tab(); > + } > + res= last_res; > + last_res= -1; > + } > + else if (optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_DYNAMIC_PUSHDOWN) && > + (*exec_tab == first_tab) && my_rnd(&rand) <= sampling_probability) > + { > + /* > + If dynamic pushdown is used, evaluate the condition to collect statistics > + about its selectivity. The result is returned so it can be used to > + filter join records during sampling. > + */ > + DBUG_ASSERT(last_res == -1); > + res= last_res= Item_func_collect_stats::val_int(); > + } > + return res; > +} > + > + > +/** > +*/ > + > +bool Item_func_dynamic_cond::walk(Item_processor processor, bool walk_subquery, > + uchar *argument) > +{ > + return is_active() ? > + Item_func::walk(processor, walk_subquery, argument) : false; > +} > + > + > +/** > + Compute the execution-time selectivity of the condition based on counters. > +*/ > + > +double Item_func_dynamic_cond::exec_selectivity() > +{ > + return is_active() ? Item_func_collect_stats::exec_selectivity() : 1; > +} > + > + > +/** > + Compute the total cost of all subqueries wrapped in this dynamic condition. > + > + @todo > + Currently the method returns the compile-time cost. It should instead > + recompute the cost in the same way as choose_best_join_tab() using execution > + time statistics. > +*/ > + > +double Item_func_dynamic_cond::exec_cost() > +{ > + if (is_active()) > + { > + double cost= 0; > + Item_subselect *subs; > + List_iterator_fast<Item_subselect> subs_it(subqueries); > + while ((subs= subs_it++)) > + cost+= subs->get_cost(); > + return cost; > + } > + else > + return 0; > +} > + > + > +/** > + Choose the JOIN_TAB in the query plan, where to attach this condition so that > + that the total plan cost is minimal. > +*/ > + > +void Item_func_dynamic_cond::choose_best_join_tab() > +{ > + JOIN_TAB *cur_tab, *best_tab; > + > + /* The current JOI_TAB where this dynamic cond is attached. */ > + JOIN_TAB *active_tab_save= active_tab; > + /* The JOIN_TAB that is being executed. */ > + JOIN_TAB *exec_tab_save= *exec_tab; > + double join_fanout; > + double cond_cost; > + double cond_sel; > + double rel_plan_cost; > + double best_plan_cost= DBL_MAX; > + > + /* > + Update the execution-time approximations of partial join fanouts. > + > + TODO: this is very inexact approach, because even if there is a fanout > + estimate for join_tab J_i, it will be incomparable to the estimate of the > + previous join tabs if not all rows of J_[i-1] were joined with J_i. > + */ > + first_tab->exec_fanout= first_tab->exec_partial_join_records ? > + /* use the execution-time counter */ > + first_tab->exec_partial_join_records : > + /* use the optimize-time estimate */ > + first_tab->records_read; > + for (cur_tab= first_tab + 1; cur_tab < last_tab; cur_tab++) > + { > + /* > + If an execution time counter is present, use it, otherwise use the > + optimize-time estimate. > + */ > + if (cur_tab->exec_partial_join_records) > + cur_tab->exec_fanout= cur_tab->exec_partial_join_records / > + (cur_tab-1)->exec_partial_join_records; > + else > + cur_tab->exec_fanout= cur_tab->records_read; > + } > + > + /* > + Compute the cost of each alternative plan where this dynamic condition is > + pushed to each possible JOIN_TAB. Currently we only consider moving the > + condition later in the join plan, that is, to any JOIN_TAB after active_tab. > + > + This condition is 'moved' from one JOIN_TAB to the next by looping through > + all possible 'active_tab' values, starting from the current one. > + */ > + for (active_tab= active_tab_save; active_tab < last_tab; active_tab++) > + { > + rel_plan_cost= 0; > + > + /* Recompute the plan cost for the current active_tab. */ > + for (cur_tab= first_tab; cur_tab < last_tab; cur_tab++) > + { > + Item_func_collect_stats *pushed_cond; > + cond_cost= 0; > + cond_sel= 0; > + join_fanout= cur_tab->exec_fanout; > + > + /* > + Set the current execution tab. This affects all dynamic conditions of > + the current JOIN. > + */ > + *exec_tab= cur_tab; > + > + if (cur_tab->select_cond) > + { > + /* > + If a top-level AND condition contains an expensive non-constant > + subquery, each of the AND terms is wrapped in an item that collectes > + execution statistics. > + */ > + DBUG_ASSERT(cur_tab->select_cond->type() == Item::FUNC_ITEM && > + (((Item_func*)cur_tab->select_cond)->functype() == Item_func::DYNAMIC_COND_FUNC || > + ((Item_func*)cur_tab->select_cond)->functype() == Item_func::EXEC_STATS_FUNC)); > + > + pushed_cond= (Item_func_collect_stats*) cur_tab->select_cond; > + cond_cost+= pushed_cond->exec_cost(); > + cond_sel*= pushed_cond->exec_selectivity(); > + } > + > + rel_plan_cost+= rel_access_cost + join_fanout * cond_sel * cond_cost; > + } > + > + if (rel_plan_cost / best_plan_cost < cost_ratio) > + { > + best_plan_cost= rel_plan_cost; > + best_tab= active_tab; > + } > + } > + > + active_tab= best_tab; > + *exec_tab= exec_tab_save; > +} > + > + > +void Item_func_dynamic_cond::print(String *str, enum_query_type query_type) > +{ > + if (is_active()) > + Item_bool_func::print(str, query_type); > +} > > === modified file 'sql/item_cmpfunc.h' > --- sql/item_cmpfunc.h 2013-07-17 19:24:29 +0000 > +++ sql/item_cmpfunc.h 2013-09-27 08:25:14 +0000 > @@ -125,6 +125,12 @@ class Item_bool_func :public Item_int_fu > bool is_bool_func() { return 1; } > void fix_length_and_dec() { decimals=0; max_length=1; } > uint decimal_precision() const { return 1; } > + /* > + The execution logic must guarantee that these methods are called only > + for classes that provide real implementations. In this way we can use > + polymorphism without implementing the methods for all subclasses. > + */ > + virtual double exec_selectivity() { DBUG_ASSERT(FALSE); return 0; } > }; > > > @@ -258,13 +264,12 @@ class Item_in_optimizer: public Item_boo > void cleanup(); > const char *func_name() const { return "<in_optimizer>"; } > Item_cache **get_cache() { return &cache; } > + Item *left_expr() { return args[0]; } > void keep_top_level_cache(); > Item *transform(Item_transformer transformer, uchar *arg); > virtual Item *expr_cache_insert_transformer(uchar *thd_arg); > bool is_expensive_processor(uchar *arg); > bool is_expensive(); > - void set_join_tab_idx(uint join_tab_idx_arg) > - { args[1]->set_join_tab_idx(join_tab_idx_arg); } > virtual void get_cache_parameters(List<Item> &parameters); > bool is_top_level_item(); > bool eval_not_null_tables(uchar *opt_arg); > @@ -492,8 +497,139 @@ class Item_func_trig_cond: public Item_b > const char *func_name() const { return "trigcond"; }; > bool const_item() const { return FALSE; } > bool *get_trig_var() { return trig_var; } > + void update_used_tables() > + { > + Item_bool_func::update_used_tables(); > + const_item_cache= false; > + used_tables_cache|= OUTER_REF_TABLE_BIT; > + } > +}; > + > + > +/** > + Item wrapper that collects execution-time statistics about its child Item. > + > + This class can be used only during execution. > +*/ > + > +class Item_func_collect_stats : public Item_bool_func > +{ > +protected: > + double count_true; /* The number of times the predicate was true. */ > + double count_evals; /* The number of times the predicate was evaluated. */ > +public: > + Item_func_collect_stats(Item *cond) > + : Item_bool_func(cond), count_true(0), count_evals(0) > + { > + DBUG_ASSERT(cond && cond->fixed); > + /* > + It doesn't make sense to wrap an item that is already capable of > + collecting execution-time statistitcs. > + */ > + DBUG_ASSERT(cond->type() != Item::FUNC_ITEM || > + (((Item_func*)cond)->functype() != Item_func::EXEC_STATS_FUNC && > + ((Item_func*)cond)->functype() != Item_func::DYNAMIC_COND_FUNC)); > + fixed= true; > + } > + enum Functype functype() const { return EXEC_STATS_FUNC; }; > + const char *func_name() const { return "exec_stats"; }; > + bool const_item() const { return FALSE; } > + longlong val_int() > + { > + longlong res= args[0]->val_int(); > + ++count_evals; > + if (res) > + ++count_true; > + return res; > + } > + virtual double exec_selectivity(); > + virtual double exec_cost(); > + > +#ifndef DBUG_OFF > + void dbug_print_stats(); > +#endif > +}; > + > + > +/** > + A wrapper class that allows to change dynamically (during execution) the > + JOIN_TAB that a condition is attached to. > +*/ > + > +class Item_func_dynamic_cond : public Item_func_collect_stats > +{ > +protected: > + List<Item_subselect> subqueries; > + /* The join_tab where the condition is currently 'pushed'.*/ > + st_join_table *active_tab; > + /* The join_tab currently being executed. */ > + st_join_table **exec_tab; > + /* The boundaries of the sub-plan where this predicate can be pushed. */ > + st_join_table *first_tab, *last_tab; > + /* > + The relative access cost based on optimizer estimates. This is the cost > + of all access methods per one record from the first table in the plan. > + */ > + double rel_access_cost; > + /* > + The number of times a dynamic condition must be evaluated in > + order to trigger dynamic pushdown". Cached copy of system variable > + dynamic_pushdown_max_evals. > + */ > + ha_rows max_evals; > + /* > + The ratio by which a new placement of the predicate in the plan > + must improve the plan cost in order to trigger dynamic pushdown. > + Derived from system variable dynamic_pushdown_cost_ratio. > + */ > + double cost_ratio; > + /* > + The probability with which to evaluate this condition in order to > + estimate its selectivity. > + */ > + double sampling_probability; > + /* > + If this predicate's selectivity is below this limit, it is considered > + sufficiently selective to re-filter all records in a partial join. > + The selectivity itself is estimated by sampling the predicate with > + 'sampling_probability'. > + */ > + double max_selectivity; > + /* > + Random number generator state. > + @todo Consider making it static. > + */ > + struct my_rnd_struct rand; > + /* > + The result of the last evaluation, -1 denotes an invalid result which > + requires a re-evaluation. This member is needed because the same Item > + may be evaluated more than once for the same partial join record due to: > + - sampling - then the item is "active" both for the left-most pushdown > + join_tab, and for the "active" join_tab. > + - moving the item to a later join_tab. > + */ > + longlong last_res; > + > +protected: > + bool is_active() > + { > + DBUG_ASSERT(*exec_tab); > + return (active_tab == *exec_tab); > + } > +public: > + Item_func_dynamic_cond(Item *cond, st_join_table *atab, st_join_table **ctab); > + longlong val_int(); > + enum Functype functype() const { return DYNAMIC_COND_FUNC; }; > + const char *func_name() const { return "dyncond"; }; > + bool walk(Item_processor processor, bool walk_subquery, uchar *argument); > + virtual double exec_selectivity(); > + virtual double exec_cost(); > + void choose_best_join_tab(); > + > + virtual void print(String *str, enum_query_type query_type); > }; > > + > class Item_func_not_all :public Item_func_not > { > /* allow to check presence of values in max/min optimization */ > @@ -1609,6 +1745,7 @@ class Item_cond :public Item_bool_func > Item *compile(Item_analyzer analyzer, uchar **arg_p, > Item_transformer transformer, uchar *arg_t); > bool eval_not_null_tables(uchar *opt_arg); > + virtual double exec_cost(); > }; > > template <template<class> class LI, class T> class Item_equal_iterator; > @@ -1922,6 +2059,9 @@ class Item_cond_and :public Item_cond > > inline bool is_cond_and(Item *item) > { > + if (!item) > + return FALSE; > + > if (item->type() != Item::COND_ITEM) > return FALSE; > > > === modified file 'sql/item_func.h' > --- sql/item_func.h 2013-08-20 11:48:29 +0000 > +++ sql/item_func.h 2013-09-27 08:25:14 +0000 > @@ -64,7 +64,7 @@ class Item_func :public Item_result_fiel > SP_STARTPOINT,SP_ENDPOINT,SP_EXTERIORRING, > SP_POINTN,SP_GEOMETRYN,SP_INTERIORRINGN, > NOT_FUNC, NOT_ALL_FUNC, > - NOW_FUNC, TRIG_COND_FUNC, > + NOW_FUNC, TRIG_COND_FUNC, DYNAMIC_COND_FUNC, EXEC_STATS_FUNC, > SUSERVAR_FUNC, GUSERVAR_FUNC, COLLATE_FUNC, > EXTRACT_FUNC, CHAR_TYPECAST_FUNC, FUNC_SP, UDF_FUNC, > NEG_FUNC, GSYSVAR_FUNC, DYNCOL_FUNC }; > > === modified file 'sql/item_subselect.cc' > --- sql/item_subselect.cc 2013-07-17 19:24:29 +0000 > +++ sql/item_subselect.cc 2013-09-23 15:47:28 +0000 > @@ -50,8 +50,8 @@ int check_and_do_in_subquery_rewrites(JO > Item_subselect::Item_subselect(): > Item_result_field(), value_assigned(0), own_engine(0), thd(0), old_engine(0), > used_tables_cache(0), have_to_be_excluded(0), const_item_cache(1), > - inside_first_fix_fields(0), done_first_fix_fields(FALSE), > - expr_cache(0), forced_const(FALSE), substitution(0), engine(0), eliminated(FALSE), > + inside_first_fix_fields(0), done_first_fix_fields(FALSE), expr_cache(0), > + forced_const(FALSE), substitution(0), engine(0), eliminated(FALSE), > changed(0), is_correlated(FALSE) > { > DBUG_ENTER("Item_subselect::Item_subselect"); > @@ -400,6 +400,21 @@ bool Item_subselect::set_fake_select_as_ > } > > > +bool Item_subselect::collect_item_subselect_processor(uchar * arg) > +{ > + List<Item_subselect> *subs_list= (List<Item_subselect>*) arg; > + List_iterator<Item_subselect> subs_list_it(*subs_list); > + Item_subselect *cur_subs; > + while ((cur_subs= subs_list_it++)) > + { > + if (cur_subs->eq(this, 1)) > + return false; /* Already in the set. */ > + } > + subs_list->push_back(this); > + return false; > +} > + > + > bool Item_subselect::mark_as_dependent(THD *thd, st_select_lex *select, > Item *item) > { > @@ -551,7 +566,7 @@ void Item_subselect::recalc_used_tables( > */ > bool Item_subselect::is_expensive() > { > - double examined_rows= 0; > + double examined_rows= 0, cur_examined_rows= 0; > > for (SELECT_LEX *sl= unit->first_select(); sl; sl= sl->next_select()) > { > @@ -560,19 +575,20 @@ bool Item_subselect::is_expensive() > continue; > > /* > - Subqueries whose result is known after optimization are not expensive. > - Such subqueries have all tables optimized away, thus have no join plan. > + The cost of a subquery is known if it was optimized, and it is either > + executable, or was already executed and cleaned up. > */ > - if (cur_join->optimized && > - (cur_join->zero_result_cause || !cur_join->tables_list)) > - return false; > + if (cur_join->have_query_plan != JOIN::QEP_EXECUTABLE && > + cur_join->have_query_plan != JOIN::QEP_DELETED) > + return true; > > /* > - If a subquery is not optimized we cannot estimate its cost. A subquery is > - considered optimized if it has a join plan. > + If JOIN::optimize determined that the subquery has an empty result, or > + this is a table-less subquery, then then subquery is cheap. > */ > - if (!(cur_join->optimized && cur_join->join_tab)) > - return true; > + if (!cur_join->table_count || !cur_join->tables_list || > + cur_join->zero_result_cause) > + return false; > > if (sl->first_inner_unit()) > { > @@ -583,13 +599,64 @@ bool Item_subselect::is_expensive() > return true; > } > > - examined_rows+= cur_join->get_examined_rows(); > + if ((cur_examined_rows= cur_join->get_examined_rows_estimate()) < 0) > + return true; psergey: This looks very weird. Perhaphs, it is better to add a constant: const ha_rows NO_EXAMINED_ROWS_ESTIMATE_YET=-1; > + examined_rows+= cur_examined_rows; > } > > return (examined_rows > thd->variables.expensive_subquery_limit); > } > > > +/** > + Compute the cost of a subquery and all its subqueries. > + > + @return total subquery cost in terms of JOIN::best_read > +*/ > + > +double Item_subselect::get_cost() > +{ > + double total_cost= 0; > + for (SELECT_LEX *sl= unit->first_select(); sl; sl= sl->next_select()) > + { > + JOIN *inner_join= sl->join; > + total_cost+= > + inner_join->best_read + /* The cost of this subquery. */ > + /* The cost of the subqueries of the subquery. */ > + inner_join->total_subquery_cost(); > + } > + return total_cost; > +} > + > + > +/** > + Check if the subquery predicate has only one possible execution strategy. > + > + @details > + If there is only one possible execution strategy for a subquery predicate, > + the optimizer doesn't need to know the number of times the subquery will be > + executed in order to choose the better strategy. In this case the underlying > + subquery can be optimized before optimizing the outer query. > + > + @retval FALSE more than one strategies are possible > + @retval TRUE only one strategy is possible > +*/ > + > +bool Item_subselect::has_single_strategy() const > +{ > + if (!is_in_predicate()) > + return true; > + else > + { > + Item_in_subselect *in_subs= (Item_in_subselect*) this; > + if (!in_subs->test_strategy(SUBS_MATERIALIZATION) || > + !in_subs->test_strategy(SUBS_IN_TO_EXISTS)) > + return true; > + } > + return false; > +} > + > + > bool Item_subselect::walk(Item_processor processor, bool walk_subquery, > uchar *argument) > { > @@ -835,6 +902,10 @@ bool Item_in_subselect::exec() > test_if_item_cache_changed(*left_expr_cache) < 0) > DBUG_RETURN(FALSE); > > + if (first_execution && test_strategy(SUBS_MATERIALIZATION) && > + engine->engine_type() == subselect_engine::SINGLE_SELECT_ENGINE) > + setup_mat_engine(); > + > /* > The exec() method below updates item::value, and item::null_value, thus if > we don't call it, the next call to item::val_int() will return whatever > @@ -888,6 +959,12 @@ void Item_subselect::update_used_tables( > if (!(used_tables_cache & ~engine->upper_select_const_tables())) > const_item_cache= 1; > } > + /* > + TODO: when merged with dynamic pushdown, pushdown_tables should not be > + added to the used tables. Instead, dynamic pushdown could use this table > + as the initial dynamic position, or it may ignore this estimate altogether. > + used_tables_cache|= pushdown_tables; > + */ > } > } > > @@ -896,6 +973,16 @@ void Item_subselect::print(String *str, > { > if (engine) > { > + /* > + For non-SJ materialization the engine is created during exec(), therefore > + it is not available during EXPLAIN. Create the materialization engine in > + order to print it. > + */ > + if (is_in_predicate() && > + ((Item_in_subselect*)this)->test_strategy(SUBS_MATERIALIZATION) && > + engine->engine_type() == subselect_engine::SINGLE_SELECT_ENGINE) > + ((Item_in_subselect*)this)->setup_mat_engine(); > + > str->append('('); > engine->print(str, query_type); > str->append(')'); > @@ -1921,7 +2008,7 @@ bool Item_allany_subselect::transform_in > The swap is needed for expressions of type 'f1 < ALL ( SELECT ....)' > where we want to evaluate the sub query even if f1 would be null. > */ > - subs= func->create_swap(*(optimizer->get_cache()), subs); > + subs= func->create_swap(optimizer->left_expr(), subs); > thd->change_item_tree(place, subs); > if (subs->fix_fields(thd, &subs)) > DBUG_RETURN(true); > @@ -3174,8 +3261,13 @@ void Item_in_subselect::update_used_tabl > { > Item_subselect::update_used_tables(); > left_expr->update_used_tables(); > - //used_tables_cache |= left_expr->used_tables(); > used_tables_cache= Item_subselect::used_tables() | left_expr->used_tables(); > + /* > + Constant optimization of the outer query may transform the left IN operand > + into a constant. The subquery is no longer uncacheable. > + */ > + if (!used_tables_cache) > + engine->make_cacheable(); > } > > > @@ -4128,12 +4220,24 @@ uint8 subselect_single_select_engine::un > } > > > +void subselect_single_select_engine::make_cacheable() > +{ > + select_lex->uncacheable= 0; > +} > + > + > uint8 subselect_union_engine::uncacheable() > { > return unit->uncacheable; > } > > > +void subselect_union_engine::make_cacheable() > +{ > + unit->uncacheable= 0; > +} > + > + > void subselect_single_select_engine::exclude() > { > select_lex->master_unit()->exclude_level(); > @@ -5045,7 +5149,7 @@ double get_fanout_with_deps(JOIN *join, > !tab->emb_sj_nest && > tab->records_read != 0) > { > - fanout *= rows2double(tab->records_read); > + fanout *= tab->records_read; > } > } > return fanout; > @@ -5234,7 +5338,7 @@ int subselect_hash_sj_engine::exec() > materialize_join->exec(); > if ((res= test(materialize_join->error || thd->is_fatal_error || > thd->is_error()))) > - goto err; > + goto end; > > /* > TODO: > @@ -5259,7 +5363,8 @@ int subselect_hash_sj_engine::exec() > item_in->reset(); > item_in->make_const(); > item_in->set_first_execution(); > - DBUG_RETURN(FALSE); > + res= 0; > + goto end; > } > > /* > @@ -5302,7 +5407,8 @@ int subselect_hash_sj_engine::exec() > item_in->null_value= 1; > item_in->make_const(); > item_in->set_first_execution(); > - DBUG_RETURN(FALSE); > + res= 0; > + goto end; > } > > if (has_covering_null_row) > @@ -5360,7 +5466,7 @@ int subselect_hash_sj_engine::exec() > { > /* This is an irrecoverable error. */ > res= 1; > - goto err; > + goto end; > } > } > } > @@ -5369,7 +5475,7 @@ int subselect_hash_sj_engine::exec() > lookup_engine= pm_engine; > item_in->change_engine(lookup_engine); > > -err: > +end: > thd->lex->current_select= save_select; > DBUG_RETURN(res); > } > > === modified file 'sql/item_subselect.h' > --- sql/item_subselect.h 2013-03-27 22:41:02 +0000 > +++ sql/item_subselect.h 2013-09-23 15:47:28 +0000 > @@ -133,13 +133,14 @@ class Item_subselect :public Item_result > > Item_subselect(); > > - virtual subs_type substype() { return UNKNOWN_SUBS; } > - bool is_in_predicate() > + virtual subs_type substype() const { return UNKNOWN_SUBS; } > + bool is_in_predicate() const > { > return (substype() == Item_subselect::IN_SUBS || > substype() == Item_subselect::ALL_SUBS || > substype() == Item_subselect::ANY_SUBS); > } > + bool has_single_strategy() const; > > /* > We need this method, because some compilers do not allow 'this' > @@ -182,6 +183,7 @@ class Item_subselect :public Item_result > void make_const() > { > used_tables_cache= 0; > + //pushdown_tables= 0; > const_item_cache= 0; > forced_const= TRUE; > } > @@ -234,6 +236,8 @@ class Item_subselect :public Item_result > @retval FALSE otherwise > */ > bool is_expensive_processor(uchar *arg) { return is_expensive(); } > + virtual double get_cost(); > + bool collect_item_subselect_processor(uchar * arg); > > /** > Get the SELECT_LEX structure associated with this Item. > @@ -261,6 +265,8 @@ class Item_subselect :public Item_result > friend bool convert_join_subqueries_to_semijoins(JOIN *join); > }; > > +typedef bool (Item_subselect::*subselect_filter)() const; > + > /* single value subselect */ > > class Item_cache; > @@ -274,7 +280,7 @@ class Item_singlerow_subselect :public I > {} > > void cleanup(); > - subs_type substype() { return SINGLEROW_SUBS; } > + subs_type substype() const { return SINGLEROW_SUBS; } > > void reset(); > void no_rows_in_result(); > @@ -370,7 +376,7 @@ class Item_exists_subselect :public Item > emb_on_expr_nest(NULL), optimizer(0), exists_transformed(0) > {} > > - subs_type substype() { return EXISTS_SUBS; } > + subs_type substype() const { return EXISTS_SUBS; } > void reset() > { > eliminated= FALSE; > @@ -577,7 +583,7 @@ class Item_in_subselect :public Item_exi > pushed_cond_guards(NULL), func(NULL), is_jtbm_merged(FALSE), > is_jtbm_const_tab(FALSE), upper_item(0) {} > void cleanup(); > - subs_type substype() { return IN_SUBS; } > + subs_type substype() const { return IN_SUBS; } > void reset() > { > eliminated= FALSE; > @@ -717,7 +723,7 @@ class Item_allany_subselect :public Item > > void cleanup(); > // only ALL subquery has upper not > - subs_type substype() { return all?ALL_SUBS:ANY_SUBS; } > + subs_type substype() const { return all?ALL_SUBS:ANY_SUBS; } > bool select_transformer(JOIN *join); > void create_comp_func(bool invert) { func= func_creator(invert); } > virtual void print(String *str, enum_query_type query_type); > @@ -788,6 +794,7 @@ class subselect_engine: public Sql_alloc > virtual int exec()= 0; > virtual uint cols()= 0; /* return number of columns in select */ > virtual uint8 uncacheable()= 0; /* query is uncacheable */ > + virtual void make_cacheable() {} > enum Item_result type() { return res_type; } > enum Item_result cmptype() { return cmp_type; } > enum_field_types field_type() { return res_field_type; } > @@ -828,6 +835,7 @@ class subselect_single_select_engine: pu > int exec(); > uint cols(); > uint8 uncacheable(); > + void make_cacheable(); > void exclude(); > table_map upper_select_const_tables(); > virtual void print (String *str, enum_query_type query_type); > @@ -862,6 +870,7 @@ class subselect_union_engine: public sub > int exec(); > uint cols(); > uint8 uncacheable(); > + void make_cacheable(); > void exclude(); > table_map upper_select_const_tables(); > virtual void print (String *str, enum_query_type query_type); > > === modified file 'sql/opt_subselect.cc' > --- sql/opt_subselect.cc 2013-07-17 19:24:29 +0000 > +++ sql/opt_subselect.cc 2013-05-28 12:22:32 +0000 > @@ -3620,7 +3620,10 @@ bool setup_sj_materialization_part2(JOIN > emb_sj_nest->sj_subq_pred))) > DBUG_RETURN(TRUE); /* purecov: inspected */ > sjm_tab->type= JT_EQ_REF; > - sjm_tab->select_cond= sjm->in_equality; > + remove_sj_conds(&sjm_tab->select_cond); > + sjm_tab->select_cond= and_items(sjm_tab->select_cond, sjm->in_equality); > + if (!sjm_tab->select_cond->fixed) > + sjm_tab->select_cond->fix_fields(thd, &sjm_tab->select_cond); > } > else > { > @@ -4907,7 +4910,7 @@ static void remove_subq_pushed_predicate > > > /** > - Optimize all subqueries of a query that were not flattened into a semijoin. > + Optimize all non-IN subqueries that were not flattened into a semijoin. > > @details > Optimize all immediate children subqueries of a query. > @@ -4924,7 +4927,12 @@ static void remove_subq_pushed_predicate > > bool JOIN::optimize_unflattened_subqueries() > { > - return select_lex->optimize_unflattened_subqueries(false); > + return select_lex->optimize_subqueries(&Item_subselect::has_single_strategy); > +} > + > +bool JOIN::optimize_in_subqueries() > +{ > + return select_lex->optimize_subqueries(NULL); > } > > /** > @@ -4958,7 +4966,7 @@ bool JOIN::optimize_constant_subqueries( > not for EXPLAIN. > */ > select_lex->options&= ~SELECT_DESCRIBE; > - res= select_lex->optimize_unflattened_subqueries(true); > + res= select_lex->optimize_subqueries(&Item_subselect::const_item); > select_lex->options= save_options; > return res; > } > @@ -5242,9 +5250,13 @@ bool setup_jtbm_semi_joins(JOIN *join, L > { > DBUG_ASSERT(subq_pred->test_set_strategy(SUBS_MATERIALIZATION)); > subq_pred->is_jtbm_const_tab= FALSE; > + > + DBUG_ASSERT(subq_pred->engine->engine_type() == subselect_engine::SINGLE_SELECT_ENGINE); > + subq_pred->setup_mat_engine(); > + DBUG_ASSERT(subq_pred->engine->engine_type() == subselect_engine::HASH_SJ_ENGINE); > subselect_hash_sj_engine *hash_sj_engine= > - ((subselect_hash_sj_engine*)item->engine); > - > + ((subselect_hash_sj_engine*) subq_pred->engine); > + > table->table= hash_sj_engine->tmp_table; > table->table->pos_in_table_list= table; > > @@ -5464,27 +5476,6 @@ bool JOIN::choose_subquery_plan(table_ma > DBUG_PRINT("info",("outer_lookup_keys: %.2f", outer_lookup_keys)); > } > > - /* > - If (1) materialization is a possible strategy based on semantic analysis > - during the prepare phase, then if > - (2) it is more expensive than the IN->EXISTS transformation, and > - (3) it is not possible to create usable indexes for the materialization > - strategy, > - fall back to IN->EXISTS. > - otherwise > - use materialization. > - */ > - if (in_subs->test_strategy(SUBS_MATERIALIZATION) && > - in_subs->setup_mat_engine()) > - { > - /* > - If materialization was the cheaper or the only user-selected strategy, > - but it is not possible to execute it due to limitations in the > - implementation, fall back to IN-TO-EXISTS. > - */ > - in_subs->set_strategy(SUBS_IN_TO_EXISTS); > - } > - > if (in_subs->test_strategy(SUBS_MATERIALIZATION)) > { > /* Restore the original query plan used for materialization. */ > > === modified file 'sql/sql_class.h' > --- sql/sql_class.h 2013-09-13 16:14:56 +0000 > +++ sql/sql_class.h 2013-09-27 08:25:14 +0000 > @@ -508,6 +508,10 @@ typedef struct system_variables > ulong optimizer_search_depth; > ulong optimizer_selectivity_sampling_limit; > ulong optimizer_use_condition_selectivity; > + ha_rows dynamic_pushdown_max_evals; > + uint dynamic_pushdown_cost_ratio; > + uint dynamic_pushdown_sampling_probability; > + uint dynamic_pushdown_max_selectivity; > ulong use_stat_tables; > ulong histogram_size; > ulong histogram_type; > > === modified file 'sql/sql_delete.cc' > --- sql/sql_delete.cc 2013-08-06 20:33:18 +0000 > +++ sql/sql_delete.cc 2013-04-01 10:36:05 +0000 > @@ -126,7 +126,7 @@ bool mysql_delete(THD *thd, TABLE_LIST * > } > > /* Apply the IN=>EXISTS transformation to all subqueries and optimize them. */ > - if (select_lex->optimize_unflattened_subqueries(false)) > + if (select_lex->optimize_subqueries(NULL)) > DBUG_RETURN(TRUE); > > const_cond= (!conds || conds->const_item()); > > === modified file 'sql/sql_join_cache.cc' > --- sql/sql_join_cache.cc 2013-05-05 18:39:31 +0000 > +++ sql/sql_join_cache.cc 2013-09-27 08:25:14 +0000 > @@ -2080,6 +2080,7 @@ enum_nested_loop_state JOIN_CACHE::join_ > if (outer_join_first_inner && !join_tab->first_unmatched) > join_tab->not_null_compl= TRUE; > > + *(join->cur_exec_tab)= join_tab; psergey: Do you still remember why you had to add this? It would be nice to put in a comment. > if (!join_tab->first_unmatched) > { > /* Find all records from join_tab that match records from join buffer */ > > === modified file 'sql/sql_lex.cc' > --- sql/sql_lex.cc 2013-08-18 19:29:06 +0000 > +++ sql/sql_lex.cc 2013-04-01 10:36:05 +0000 > @@ -3461,7 +3461,7 @@ bool st_select_lex::add_index_hint (THD > @retval TRUE error occurred. > */ > > -bool st_select_lex::optimize_unflattened_subqueries(bool const_only) > +bool st_select_lex::optimize_subqueries(subselect_filter filter) > { > for (SELECT_LEX_UNIT *un= first_inner_unit(); un; un= un->next_unit()) > { > @@ -3476,11 +3476,8 @@ bool st_select_lex::optimize_unflattened > continue; > } > > - if (const_only && !subquery_predicate->const_item()) > - { > - /* Skip non-constant subqueries if the caller asked so. */ > + if (filter && !((subquery_predicate->*filter)())) > continue; > - } > > bool empty_union_result= true; > bool is_correlated_unit= false; > @@ -4204,7 +4201,9 @@ int st_select_lex::print_explain(select_ > bool *printed_anything) > { > int res; > - if (join && join->have_query_plan == JOIN::QEP_AVAILABLE) > + if (join && > + (join->have_query_plan == JOIN::QEP_EXPLAINABLE || > + join->have_query_plan == JOIN::QEP_EXECUTABLE)) > { > /* > There is a number of reasons join can be marked as degenerate, so all > > === modified file 'sql/sql_lex.h' > --- sql/sql_lex.h 2013-07-17 19:24:29 +0000 > +++ sql/sql_lex.h 2013-04-01 10:36:05 +0000 > @@ -1026,7 +1026,7 @@ class st_select_lex: public st_select_le > > void clear_index_hints(void) { index_hints= NULL; } > bool is_part_of_union() { return master_unit()->is_union(); } > - bool optimize_unflattened_subqueries(bool const_only); > + bool optimize_subqueries(subselect_filter filter); > /* Set the EXPLAIN type for this subquery. */ > void set_explain_type(bool on_the_fly); > bool handle_derived(LEX *lex, uint phases); > > === modified file 'sql/sql_priv.h' > --- sql/sql_priv.h 2013-07-17 19:24:29 +0000 > +++ sql/sql_priv.h 2013-09-27 08:25:14 +0000 > @@ -228,7 +228,8 @@ template <class T> bool valid_buffer_ran > #define OPTIMIZER_SWITCH_TABLE_ELIMINATION (1ULL << 26) > #define OPTIMIZER_SWITCH_EXTENDED_KEYS (1ULL << 27) > #define OPTIMIZER_SWITCH_EXISTS_TO_IN (1ULL << 28) > -#define OPTIMIZER_SWITCH_USE_CONDITION_SELECTIVITY (1ULL << 29) > +#define OPTIMIZER_SWITCH_EXPENSIVE_PRED_STATIC_PUSHDOWN (1ULL << 29) > +#define OPTIMIZER_SWITCH_EXPENSIVE_PRED_DYNAMIC_PUSHDOWN (1ULL << 30) > > #define OPTIMIZER_SWITCH_DEFAULT (OPTIMIZER_SWITCH_INDEX_MERGE | \ > OPTIMIZER_SWITCH_INDEX_MERGE_UNION | \ > @@ -250,7 +251,7 @@ template <class T> bool valid_buffer_ran > OPTIMIZER_SWITCH_SUBQUERY_CACHE | \ > OPTIMIZER_SWITCH_SEMIJOIN | \ > OPTIMIZER_SWITCH_FIRSTMATCH | \ > - OPTIMIZER_SWITCH_LOOSE_SCAN ) > + OPTIMIZER_SWITCH_LOOSE_SCAN) > /* > Replication uses 8 bytes to store SQL_MODE in the binary log. The day you > use strictly more than 64 bits by adding one more define above, you should > > === modified file 'sql/sql_select.cc' > --- sql/sql_select.cc 2013-08-20 11:48:29 +0000 > +++ sql/sql_select.cc 2013-09-27 13:22:36 +0000 > @@ -200,14 +200,12 @@ int join_read_always_key_or_null(JOIN_TA > int join_read_next_same_or_null(READ_RECORD *info); > static COND *make_cond_for_table(THD *thd, Item *cond,table_map table, > table_map used_table, > - int join_tab_idx_arg, > bool exclude_expensive_cond, > bool retain_ref_cond); > static COND *make_cond_for_table_from_pred(THD *thd, Item *root_cond, > Item *cond, > table_map tables, > table_map used_table, > - int join_tab_idx_arg, > bool exclude_expensive_cond, > bool retain_ref_cond); > > @@ -334,6 +332,50 @@ bool dbug_user_var_equals_int(THD *thd, > } > return FALSE; > } > + > + > +/** > + Print query execution statistics. > +*/ > + > +void print_execution_stats(JOIN *join) > +{ > + JOIN_TAB *first_tab, *tab; > + > + DBUG_LOCK_FILE; > + > + fprintf(DBUG_FILE, "\nexecution_statistics {\n"); > + > + first_tab= first_breadth_first_tab(join, WALK_OPTIMIZATION_TABS); > + for (tab= first_tab; tab; > + tab= next_breadth_first_tab(join, WALK_OPTIMIZATION_TABS, tab)) > + { > + if (tab->bush_root_tab) > + continue; /* TODO: how to process nested tabs?*/ > + fprintf(DBUG_FILE, "%s: actual_card: %f, est_card: %g, est_sel: %f\n", > + tab->table->alias.c_ptr(), > + tab->exec_partial_join_records, > + tab->partial_join_cardinality, > + tab->cond_selectivity); > + > + if (!tab->select_cond) > + continue; > + > + fprintf(DBUG_FILE, " dynamic_cond_stats:\n"); > + if (tab->select_cond->type() == Item::FUNC_ITEM && > + (((Item_func*)tab->select_cond)->functype() == Item_func::DYNAMIC_COND_FUNC || > + ((Item_func*)tab->select_cond)->functype() == Item_func::EXEC_STATS_FUNC)) > + { > + Item_func_collect_stats *stat_cond= (Item_func_collect_stats*) tab->select_cond; > + stat_cond->dbug_print_stats(); > + } > + } > + fprintf(DBUG_FILE, "}\n"); > + fprintf(DBUG_FILE, "\n"); > + > + DBUG_UNLOCK_FILE; > +} > + > #endif > > > @@ -1013,9 +1055,16 @@ int JOIN::optimize() > short-circuit because optimized==TRUE. > */ > if (!res && have_query_plan != QEP_DELETED) > - have_query_plan= QEP_AVAILABLE; > + { > + if (have_query_plan == QEP_INCOMPLETE) > + have_query_plan= QEP_EXPLAINABLE; > + else > + have_query_plan= QEP_EXECUTABLE; > + } > return res; > } > + > + > /** > global select optimisation. > > @@ -1046,6 +1095,7 @@ JOIN::optimize_inner() > > set_allowed_join_cache_types(); > need_distinct= TRUE; > + examined_rows_estimate= -1; > > /* Run optimize phase for all derived tables/views used in this SELECT. */ > if (select_lex->handle_derived(thd->lex, DT_OPTIMIZE)) > @@ -1282,8 +1332,7 @@ TODO: make view to decide if it is possi > if (conds && !(thd->lex->describe & DESCRIBE_EXTENDED)) > { > COND *table_independent_conds= > - make_cond_for_table(thd, conds, PSEUDO_TABLE_BITS, 0, -1, > - FALSE, FALSE); > + make_cond_for_table(thd, conds, PSEUDO_TABLE_BITS, 0, FALSE, FALSE); > DBUG_EXECUTE("where", > print_where(table_independent_conds, > "where after opt_sum_query()", > @@ -1296,6 +1345,11 @@ TODO: make view to decide if it is possi > { > DBUG_PRINT("info",("No tables")); > error= 0; > + /* > + Even if there are no tables the query has some small cost. Use the default > + cost of all 'cheap' items. > + */ > + best_read= (1 / (double)TIME_FOR_COMPARE); > goto setup_subq_exit; > } > error= -1; // Error is sent to client > @@ -1730,12 +1784,18 @@ TODO: make view to decide if it is possi > if (!(select_options & SELECT_DESCRIBE)) > init_ftfuncs(thd, select_lex, test(order)); > > - if (optimize_unflattened_subqueries()) > + if (optimize_in_subqueries()) > DBUG_RETURN(1); > > int res; > if ((res= rewrite_to_index_subquery_engine(this)) != -1) > + { > + /* > + This subquery has been rewritten to direct index access via handler calls > + instead of going through JOIN::exec. > + */ > DBUG_RETURN(res); > + } > if (setup_subquery_caches()) > DBUG_RETURN(-1); > > @@ -1863,7 +1923,7 @@ TODO: make view to decide if it is possi > Even with zero matching rows, subqueries in the HAVING clause may > need to be evaluated if there are aggregate functions in the query. > */ > - if (optimize_unflattened_subqueries()) > + if (optimize_unflattened_subqueries() || optimize_in_subqueries()) > DBUG_RETURN(1); > error= 0; > > @@ -2015,6 +2075,10 @@ int JOIN::init_execution() > DBUG_RETURN(-1); /* purecov: inspected */ > } > > + > + if (setup_dynamic_conditions()) > + DBUG_RETURN(1); > + > DBUG_RETURN(0); > } > > @@ -2845,8 +2909,7 @@ void JOIN::exec_inner() > curr_table->table->map); > > Item* sort_table_cond= make_cond_for_table(thd, curr_join->tmp_having, > - used_tables, > - (table_map)0, -1, > + used_tables, (table_map)0, > FALSE, FALSE); > if (sort_table_cond) > { > @@ -2886,7 +2949,7 @@ void JOIN::exec_inner() > QT_ORDINARY);); > curr_join->tmp_having= make_cond_for_table(thd, curr_join->tmp_having, > ~ (table_map) 0, > - ~used_tables, -1, > + ~used_tables, > FALSE, FALSE); > DBUG_EXECUTE("where",print_where(curr_join->tmp_having, > "having after sort", > @@ -3331,6 +3394,7 @@ make_join_statistics(JOIN *join, List<TA > */ > join->best_positions= (POSITION*) join->thd->alloc(sizeof(POSITION)* > (table_count +1)); > + join->cur_exec_tab= (JOIN_TAB**) join->thd->calloc(sizeof(JOIN_TAB*)); > > if (join->thd->is_fatal_error) > DBUG_RETURN(1); // Eom /* purecov: inspected */ > @@ -3884,6 +3948,9 @@ make_join_statistics(JOIN *join, List<TA > > if (join->const_tables != join->table_count) > optimize_keyuse(join, keyuse_array); > + > + if (join->optimize_unflattened_subqueries()) > + DBUG_RETURN(1); > > if (optimize_semijoin_nests(join, all_table_map)) > DBUG_RETURN(TRUE); /* purecov: inspected */ > @@ -6879,28 +6946,57 @@ void JOIN::get_prefix_cost_and_fanout(ui > > > /** > + Compute the total cost of all subqueries of a (sub)query. > + > + @return the cost of all subqueries > +*/ > + > +double JOIN::total_subquery_cost() psergey: I think this function needs to be called get_total_subquery() cost, to follow the pattern set by other similar functions. When I read this function, I don't understand why it doesn't account for the fact that a subquery might be evaluated multiple times. Please add a comment about that. > +{ > + double total_cost= 0; > + for (SELECT_LEX_UNIT *un= select_lex->first_inner_unit(); > + un; un= un->next_unit()) > + { > + Item_subselect *subquery_predicate= un->item; > + if (!subquery_predicate) > + continue; > + > + total_cost+= subquery_predicate->get_cost(); > + } > + return total_cost; > +} > + > + > +/** > Estimate the number of rows that query execution will read. > > @todo This is a very pessimistic upper bound. Use join selectivity > when available to produce a more realistic number. > */ > > -double JOIN::get_examined_rows() > +double JOIN::get_examined_rows_estimate() > { > - ha_rows examined_rows; > - double prev_fanout= 1; > - JOIN_TAB *tab= first_breadth_first_tab(this, WALK_OPTIMIZATION_TABS); > - JOIN_TAB *prev_tab= tab; > + if (examined_rows_estimate >= 0) > + return examined_rows_estimate; > + else if (!(have_query_plan == QEP_EXPLAINABLE || > + have_query_plan == QEP_EXECUTABLE)) > + return -1; > + else > + { > + double prev_fanout= 1; > + JOIN_TAB *tab= first_breadth_first_tab(this, WALK_OPTIMIZATION_TABS); > + JOIN_TAB *prev_tab= tab; > > - examined_rows= tab->get_examined_rows(); > + examined_rows_estimate= tab->get_examined_rows_estimate(); > > - while ((tab= next_breadth_first_tab(this, WALK_OPTIMIZATION_TABS, tab))) > - { > - prev_fanout *= prev_tab->records_read; > - examined_rows+= (ha_rows) (tab->get_examined_rows() * prev_fanout); > - prev_tab= tab; > + while ((tab= next_breadth_first_tab(this, WALK_OPTIMIZATION_TABS, tab))) > + { > + prev_fanout *= prev_tab->records_read; > + examined_rows_estimate+= tab->get_examined_rows_estimate() * prev_fanout; > + prev_tab= tab; > + } > + return examined_rows_estimate; > } > - return examined_rows; > } > > > @@ -7374,15 +7470,15 @@ best_extension_by_limited_search(JOIN > remaining_tables & > ~real_table_bit); > join->positions[idx].cond_selectivity= pushdown_cond_selectivity; > - double partial_join_cardinality= current_record_count * > - pushdown_cond_selectivity; > + join->positions[idx].partial_join_cardinality= current_record_count * > + pushdown_cond_selectivity; > if ( (search_depth > 1) && (remaining_tables & ~real_table_bit) & allowed_tables ) > { /* Recursively expand the current partial plan */ > swap_variables(JOIN_TAB*, join->best_ref[idx], *pos); > if (best_extension_by_limited_search(join, > remaining_tables & ~real_table_bit, > idx + 1, > - partial_join_cardinality, > + join->positions[idx].partial_join_cardinality, > current_read_time, > search_depth - 1, > prune_level, > @@ -7400,13 +7496,68 @@ best_extension_by_limited_search(JOIN > join->positions[join->const_tables].table->table) > /* We have to make a temp table */ > current_read_time+= current_record_count; > + > + /* > + For each subquery find the partial join with the least cardinality > + in order to minimize the number of subquery evaluations. > + */ psergey: Please move this new piece of code into a separate function. It does one thing, its action can be easily separated from what the rest of the code in this function does, so it should be moved away. > + if (optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_STATIC_PUSHDOWN) && > + join->conds && join->conds->with_subselect > + /* @todo This is not a reliable estimate: > + && join->conds->is_expensive()*/) > + { > + bool may_benefit_from_static_pushdown= false; > + /* > + Find and store the POSITION with the least partial result cardinality > + between the current and last POSITION in the query plan. > + */ psergey: Does the loop below do what the comment says? AFAIU, the code sets "least_card_pos" for every position in the join->positions[] array. > + POSITION *min_pos= NULL; > + for (POSITION *p_pos= join->positions + idx, > + *p_end= join->positions + join->const_tables; > + p_pos >= p_end ; > + p_pos--) > + { > + if (!min_pos || > + p_pos->partial_join_cardinality <= min_pos->partial_join_cardinality) > + min_pos= p_pos; > + p_pos->least_card_pos= min_pos; > + if (p_pos->least_card_pos != p_pos) > + may_benefit_from_static_pushdown= true; > + } > + > + /* > + Make expensive predicates dependent on the table that is in the > + partial join with the least cardinality after the left-most join > + where the predicate can be pushed. > + */ psergey: ^ out of date comment? > + if (may_benefit_from_static_pushdown) > + { > + if (is_cond_and(join->conds)) > + { psergey: I find it weird that every time you need to analyze the WHERE condition for expensive items. I would have expected that you would - collect a list of AND-parts of the WHERE that can be moved - do some pre-processing (like, fetch their cost) and after that run join optimization that would use this data. > + List_iterator<Item> and_parts_it(*((Item_cond*) join->conds)->argument_list()); > + Item *and_part; > + while ((and_part= and_parts_it++)) > + { > + if (!and_part->with_subselect > + /* @todo This is not a reliable estimate: > + !and_part->is_expensive()*/) psergey: There are a lot of places in the code that check a condition whether a given Item object should be processed by cost-based condition pushdown. This means, the check should be factored out into a function. > + continue; > + current_read_time+= join->static_pushdown_cost(and_part, idx); > + } > + } > + else > + current_read_time+= join->static_pushdown_cost(join->conds, idx); > + } > + } > + > if (current_read_time < join->best_read) > { > memcpy((uchar*) join->best_positions, (uchar*) join->positions, > sizeof(POSITION) * (idx + 1)); > - join->record_count= partial_join_cardinality; > + join->record_count= join->positions[idx].partial_join_cardinality; > join->best_read= current_read_time - 0.001; > } > + > DBUG_EXECUTE("opt", print_plan(join, idx+1, > current_record_count, > read_time, > @@ -7605,7 +7756,7 @@ int JOIN_TAB::make_scan_filter() > if (cond && > (tmp= make_cond_for_table(join->thd, cond, > join->const_table_map | table->map, > - table->map, -1, FALSE, TRUE))) > + table->map, FALSE, TRUE))) > { > DBUG_EXECUTE("where",print_where(tmp,"cache", QT_ORDINARY);); > if (!(cache_select= > @@ -8048,7 +8199,8 @@ get_best_combination(JOIN *join) > sub-order > */ > SJ_MATERIALIZATION_INFO *sjm= cur_pos->table->emb_sj_nest->sj_mat_info; > - j->records= j->records_read= (ha_rows)(sjm->is_sj_scan? sjm->rows : 1); > + j->records_read= sjm->is_sj_scan? sjm->rows : 1; > + j->records= (ha_rows) j->records_read; > j->cond_selectivity= 1.0; > JOIN_TAB *jt; > JOIN_TAB_RANGE *jt_range; > @@ -8112,7 +8264,7 @@ get_best_combination(JOIN *join) > Save records_read in JOIN_TAB so that select_describe()/etc don't have > to access join->best_positions[]. > */ > - j->records_read= (ha_rows)join->best_positions[tablenr].records_read; > + j->records_read= join->best_positions[tablenr].records_read; > j->cond_selectivity= join->best_positions[tablenr].cond_selectivity; > join->map2table[j->table->tablenr]= j; > > @@ -8614,6 +8766,7 @@ JOIN::make_simple_join(JOIN *parent, TAB > bzero((char*) &join_tab->read_record,sizeof(join_tab->read_record)); > temp_table->status=0; > temp_table->null_row=0; > + *cur_exec_tab= *(parent->cur_exec_tab); > DBUG_RETURN(FALSE); > } > > @@ -8969,7 +9122,7 @@ make_join_select(JOIN *join,SQL_SELECT * > join->exec_const_cond= > make_cond_for_table(thd, cond, > join->const_table_map, > - (table_map) 0, -1, FALSE, FALSE); > + (table_map) 0, FALSE, FALSE); > /* Add conditions added by add_not_null_conds(). */ > for (uint i= 0 ; i < join->const_tables ; i++) > add_cond_and_fix(thd, &join->exec_const_cond, > @@ -8991,7 +9144,7 @@ make_join_select(JOIN *join,SQL_SELECT * > join->const_table_map | > OUTER_REF_TABLE_BIT, > OUTER_REF_TABLE_BIT, > - -1, FALSE, FALSE); > + FALSE, FALSE); > if (outer_ref_cond) > { > add_cond_and_fix(thd, &outer_ref_cond, join->outer_ref_cond); > @@ -9005,7 +9158,7 @@ make_join_select(JOIN *join,SQL_SELECT * > join->const_table_map | > PSEUDO_TABLE_BITS, > PSEUDO_TABLE_BITS, > - -1, FALSE, FALSE); > + FALSE, FALSE); > if (pseudo_bits_cond) > { > add_cond_and_fix(thd, &pseudo_bits_cond, > @@ -9102,8 +9255,10 @@ make_join_select(JOIN *join,SQL_SELECT * > } > else > { > - tmp= make_cond_for_table(thd, cond, used_tables, current_map, i, > + tmp= make_cond_for_table(thd, cond, used_tables, current_map, > FALSE, FALSE); > + if (tmp) > + tmp->walk(&Item::set_join_tab_idx_processor, 0, (uchar*) &i); > } > /* Add conditions added by add_not_null_conds(). */ > if (tab->select_cond) > @@ -9184,7 +9339,7 @@ make_join_select(JOIN *join,SQL_SELECT * > { > COND *push_cond= > make_cond_for_table(thd, tmp, current_map, current_map, > - -1, FALSE, FALSE); > + FALSE, FALSE); > if (push_cond) > { > /* Push condition to handler */ > @@ -9357,7 +9512,7 @@ make_join_select(JOIN *join,SQL_SELECT * > JOIN_TAB *cond_tab= join_tab->first_inner; > COND *tmp= make_cond_for_table(thd, *join_tab->on_expr_ref, > join->const_table_map, > - (table_map) 0, -1, FALSE, FALSE); > + (table_map) 0, FALSE, FALSE); > if (!tmp) > continue; > tmp= new Item_func_trig_cond(tmp, &cond_tab->not_null_compl); > @@ -9403,11 +9558,11 @@ make_join_select(JOIN *join,SQL_SELECT * > current_map= tab->table->map; > used_tables2|= current_map; > /* > - psergey: have put the -1 below. It's bad, will need to fix it. > + psergey: After pushdown, set the join_tab_idx of tmp_cond to the > + right join tab (tab - first_tab)? > */ > COND *tmp_cond= make_cond_for_table(thd, on_expr, used_tables2, > - current_map, /*(tab - first_tab)*/ -1, > - FALSE, FALSE); > + current_map, FALSE, FALSE); > bool is_sjm_lookup_tab= FALSE; > if (tab->bush_children) > { > @@ -10198,6 +10353,8 @@ uint check_join_cache_usage(JOIN_TAB *ta > if ((tab->cache= new JOIN_CACHE_BNL(join, tab, prev_cache)) && > ((options & SELECT_DESCRIBE) || !tab->cache->init())) > { > + if (options & SELECT_DESCRIBE) > + join->have_query_plan= JOIN::QEP_INCOMPLETE; > tab->icp_other_tables_ok= FALSE; > return (2-test(!prev_cache)); > } > @@ -10233,6 +10390,8 @@ uint check_join_cache_usage(JOIN_TAB *ta > if ((tab->cache= new JOIN_CACHE_BNLH(join, tab, prev_cache)) && > ((options & SELECT_DESCRIBE) || !tab->cache->init())) > { > + if (options & SELECT_DESCRIBE) > + join->have_query_plan= JOIN::QEP_INCOMPLETE; > tab->icp_other_tables_ok= FALSE; > return (4-test(!prev_cache)); > } > @@ -10253,7 +10412,11 @@ uint check_join_cache_usage(JOIN_TAB *ta > prev_cache= 0; > if ((tab->cache= new JOIN_CACHE_BKA(join, tab, flags, prev_cache)) && > ((options & SELECT_DESCRIBE) || !tab->cache->init())) > + { > + if (options & SELECT_DESCRIBE) > + join->have_query_plan= JOIN::QEP_INCOMPLETE; > return (6-test(!prev_cache)); > + } > goto no_join_cache; > } > else > @@ -10263,7 +10426,9 @@ uint check_join_cache_usage(JOIN_TAB *ta > if ((tab->cache= new JOIN_CACHE_BKAH(join, tab, flags, prev_cache)) && > ((options & SELECT_DESCRIBE) || !tab->cache->init())) > { > - tab->idx_cond_fact_out= FALSE; > + if (options & SELECT_DESCRIBE) > + join->have_query_plan= JOIN::QEP_INCOMPLETE; > + tab->idx_cond_fact_out= FALSE; > return (8-test(!prev_cache)); > } > goto no_join_cache; > @@ -10422,10 +10587,9 @@ make_join_readinfo(JOIN *join, ulonglong > for (tab= join->join_tab; tab != join->join_tab + join->const_tables; tab++) > tab->partial_join_cardinality= 1; > > - JOIN_TAB *prev_tab= NULL; > for (tab= first_linear_tab(join, WITHOUT_CONST_TABLES), i= join->const_tables; > tab; > - prev_tab=tab, tab= next_linear_tab(join, tab, WITH_BUSH_ROOTS)) > + tab= next_linear_tab(join, tab, WITH_BUSH_ROOTS)) > { > /* > The approximation below for partial join cardinality is not good because > @@ -10435,12 +10599,9 @@ make_join_readinfo(JOIN *join, ulonglong > Later it should be improved. > */ > > - if (tab->bush_root_tab && tab->bush_root_tab->bush_children->start == tab) > - prev_tab= NULL; > DBUG_ASSERT(tab->bush_children || tab->table == join->best_positions[i].table->table); > > - tab->partial_join_cardinality= join->best_positions[i].records_read * > - (prev_tab? prev_tab->partial_join_cardinality : 1); > + tab->partial_join_cardinality= join->best_positions[i].partial_join_cardinality; > if (!tab->bush_children) > i++; > } > @@ -10807,9 +10968,9 @@ double JOIN_TAB::scan_time() > @todo: why not use JOIN_TAB::found_records > */ > > -ha_rows JOIN_TAB::get_examined_rows() > +double JOIN_TAB::get_examined_rows_estimate() > { > - ha_rows examined_rows; > + double examined_rows; > > if (select && select->quick && use_quick != 2) > examined_rows= select->quick->records; > @@ -10839,7 +11000,7 @@ ha_rows JOIN_TAB::get_examined_rows() > } > } > else > - examined_rows= (ha_rows) records_read; > + examined_rows= records_read; > > return examined_rows; > } > @@ -16457,6 +16618,13 @@ do_select(JOIN *join,List<Item> *fields, > sufficient to check only the condition pseudo_bits_cond. > */ > DBUG_ASSERT(join->outer_ref_cond == NULL); > + DBUG_EXECUTE_IF("show_explain_probe_do_select_const", > + if (dbug_user_var_equals_int(join->thd, > + "show_explain_probe_select_id", > + join->select_lex->select_number)) > + dbug_serve_apcs(join->thd, 1); > + ); > + > if (!join->pseudo_bits_cond || join->pseudo_bits_cond->val_int()) > { > error= (*end_select)(join, 0, 0); > @@ -16512,6 +16680,23 @@ do_select(JOIN *join,List<Item> *fields, > if (error == NESTED_LOOP_NO_MORE_ROWS || join->thd->killed == ABORT_QUERY) > error= NESTED_LOOP_OK; > > + DBUG_EXECUTE("counts", > + join->is_top_level_join() ? > + print_execution_stats(join) : (void)0;); > + /* Reset execution-time counters. */ > + if (join->table_count) > + { > + JOIN_TAB *first_tab, *tab; > + first_tab= first_breadth_first_tab(join, WALK_OPTIMIZATION_TABS); > + for (tab= first_tab; tab; > + tab= next_breadth_first_tab(join, WALK_OPTIMIZATION_TABS, tab)) > + { > + if (tab->bush_root_tab) > + continue; /* TODO: how to process nested tabs?*/ > + tab->exec_partial_join_records= 0; > + } > + } > + > if (table) > { > int tmp, new_errno= 0; > @@ -16927,6 +17112,8 @@ evaluate_join_record(JOIN *join, JOIN_TA > if (join_tab->table->vfield) > update_virtual_fields(join->thd, join_tab->table); > > + *(join->cur_exec_tab)= join_tab; > + > if (select_cond) > { > select_cond_result= test(select_cond->val_int()); > @@ -17053,6 +17240,7 @@ evaluate_join_record(JOIN *join, JOIN_TA > if (found) > { > enum enum_nested_loop_state rc; > + join_tab->exec_partial_join_records++; > /* A match from join_tab is found for the current partial join. */ > rc= (*join_tab->next_select)(join, join_tab+1, 0); > join->thd->warning_info->inc_current_row_for_warning(); > @@ -17106,6 +17294,7 @@ evaluate_null_complemented_join_record(J > and no matches has been found for the current outer row. > */ > JOIN_TAB *last_inner_tab= join_tab->last_inner; > + *(join->cur_exec_tab)= join_tab; > /* Cache variables for faster loop */ > COND *select_cond; > for ( ; join_tab <= last_inner_tab ; join_tab++) > @@ -17169,6 +17358,8 @@ evaluate_null_complemented_join_record(J > join->return_tab= join_tab->do_firstmatch; > } > > + join_tab->exec_partial_join_records++; > + > /* > Send the row complemented by nulls to be joined with the > remaining tables. > @@ -18536,21 +18727,34 @@ bool test_if_ref(Item *root_cond, Item_f > static Item * > make_cond_for_table(THD *thd, Item *cond, table_map tables, > table_map used_table, > - int join_tab_idx_arg, > bool exclude_expensive_cond __attribute__((unused)), > bool retain_ref_cond) > { > return make_cond_for_table_from_pred(thd, cond, cond, tables, used_table, > - join_tab_idx_arg, > exclude_expensive_cond, > retain_ref_cond); > } > > > +static int compare_items_by_cost(Item* it1, Item* it2, void *arg) > +{ > + double cost1= 0, cost2= 0; > + > + it1->walk(&Item::sum_cost_processor, false, (uchar*) &cost1); > + it2->walk(&Item::sum_cost_processor, false, (uchar*) &cost2); > + > + if (cost1 == cost2) > + return 0; > + if (cost1 > cost2) > + return -1; > + else > + return 1; > +} > + > + > static Item * > make_cond_for_table_from_pred(THD *thd, Item *root_cond, Item *cond, > table_map tables, table_map used_table, > - int join_tab_idx_arg, > bool exclude_expensive_cond __attribute__ > ((unused)), > bool retain_ref_cond) > @@ -18573,7 +18777,6 @@ make_cond_for_table_from_pred(THD *thd, > { > Item *fix=make_cond_for_table_from_pred(thd, root_cond, item, > tables, used_table, > - join_tab_idx_arg, > exclude_expensive_cond, > retain_ref_cond); > if (fix) > @@ -18594,6 +18797,11 @@ make_cond_for_table_from_pred(THD *thd, > new_cond->used_tables_cache= > ((Item_cond_and*) cond)->used_tables_cache & > tables; > + /* > + Move expensive conditions to the end of the AND list to reduce the > + number of times they are evaluated. > + */ > + bubble_sort<Item>(new_cond->argument_list(), compare_items_by_cost, NULL); > return new_cond; > } > } > @@ -18608,7 +18816,6 @@ make_cond_for_table_from_pred(THD *thd, > { > Item *fix=make_cond_for_table_from_pred(thd, root_cond, item, > tables, 0L, > - join_tab_idx_arg, > exclude_expensive_cond, > retain_ref_cond); > if (!fix) > @@ -18623,6 +18830,11 @@ make_cond_for_table_from_pred(THD *thd, > new_cond->fix_fields(thd, 0); > new_cond->used_tables_cache= ((Item_cond_or*) cond)->used_tables_cache; > new_cond->top_level_item(); > + /* > + Move expensive conditions to the end of the AND list to reduce the > + number of times they are evaluated. > + */ > + bubble_sort<Item>(new_cond->argument_list(), compare_items_by_cost, NULL); > return new_cond; > } > } > @@ -18638,7 +18850,6 @@ make_cond_for_table_from_pred(THD *thd, > > if (cond->marker == 2 || cond->eq_cmp_result() == Item::COND_OK) > { > - cond->set_join_tab_idx(join_tab_idx_arg); > return cond; // Not boolean op > } > > @@ -18661,7 +18872,6 @@ make_cond_for_table_from_pred(THD *thd, > } > } > cond->marker=2; > - cond->set_join_tab_idx(join_tab_idx_arg); > return cond; > } > > @@ -19688,6 +19898,7 @@ create_sort_index(THD *thd, JOIN *join, > tab= join->join_tab + join->const_tables; > table= tab->table; > select= tab->select; > + *(join->cur_exec_tab)= tab; > > JOIN_TAB *save_pre_sort_join_tab= NULL; > if (join->pre_sort_join_tab) > @@ -22287,7 +22498,8 @@ int JOIN::print_explain(select_result_si > DBUG_PRINT("info", ("Select 0x%lx, type %s, message %s", > (ulong)join->select_lex, join->select_lex->type, > message ? message : "NULL")); > - DBUG_ASSERT(on_the_fly? have_query_plan == QEP_AVAILABLE: TRUE); > + DBUG_ASSERT(on_the_fly? (have_query_plan == QEP_EXPLAINABLE || > + have_query_plan == QEP_EXECUTABLE) : TRUE); > /* Don't log this into the slow query log */ > > if (!on_the_fly) > @@ -22326,6 +22538,9 @@ int JOIN::print_explain(select_result_si > uint select_id= join->select_lex->select_number; > JOIN_TAB* const first_top_tab= first_breadth_first_tab(join, WALK_OPTIMIZATION_TABS); > > + if (!join->initialized && join->setup_dynamic_conditions()) > + return 1; > + > for (JOIN_TAB *tab= first_breadth_first_tab(join, WALK_OPTIMIZATION_TABS); tab; > tab= next_breadth_first_tab(join, WALK_OPTIMIZATION_TABS, tab)) > { > @@ -22583,7 +22798,7 @@ int JOIN::print_explain(select_result_si > } > else > { > - ha_rows examined_rows= tab->get_examined_rows(); > + double examined_rows= tab->get_examined_rows_estimate(); > > item_list.push_back(new Item_int((longlong) (ulonglong) examined_rows, > MY_INT64_NUM_DECIMAL_DIGITS)); > @@ -22596,7 +22811,7 @@ int JOIN::print_explain(select_result_si > { > double pushdown_cond_selectivity= tab->cond_selectivity; > if (pushdown_cond_selectivity == 1.0) > - f= (float) (100.0 * tab->records_read / examined_rows); > + f= (100.0 * (float)tab->records_read) / examined_rows; > else > f= (float) (100.0 * pushdown_cond_selectivity); > } > @@ -22687,6 +22902,25 @@ int JOIN::print_explain(select_result_si > } > else > extra.append(STRING_WITH_LEN("; Using where")); > + /* Add the id's of the subselects to the table they are pushed to. */ > + if (explain_flags & DESCRIBE_EXTENDED) > + { > + List<Item_subselect> subs_list; > + *cur_exec_tab= tab; > + tab->select_cond->walk(&Item::collect_item_subselect_processor, 0, > + (uchar*) &subs_list); > + if (subs_list.elements) > + { > + List_iterator_fast<Item_subselect> subs_it(subs_list); > + Item_subselect *subs; > + extra.append(STRING_WITH_LEN("; Subqueries:")); > + while ((subs= subs_it++)) > + { > + extra.append(STRING_WITH_LEN(" ")); > + extra.append_ulonglong(subs->get_select_lex()->select_number); > + } > + } > + } > } > } > if (table_list /* SJM bushes don't have table_list */ && > @@ -23652,6 +23886,204 @@ void JOIN::cache_const_exprs() > } > > > +double JOIN::static_pushdown_cost(Item *pushed_cond, uint idx) > +{ psergey: This function needs a comment. I see that it modifies position opt_pos->read_time. What is the function that undoes this change? > + table_map cond_tables= pushed_cond->used_tables(); > + POSITION *last_pos= NULL; > + POSITION *opt_pos; > + double pushed_cond_cost; > + > + DBUG_ASSERT(pushed_cond->with_subselect > + /*pushed_cond->is_expensive()*/); > + > + if (!cond_tables || cond_tables == OUTER_REF_TABLE_BIT || idx <= const_tables) > + return 0; > + > + /* > + Find the last join_tab on which the subquery depends. This is the > + earliest join_tab where the condition can be pushed. > + */ > + for (POSITION *p_pos= positions + idx, *p_end= positions + const_tables; > + p_pos >= p_end ; p_pos--) > + { > + if (p_pos->table->table->map & cond_tables) > + { > + last_pos= p_pos; > + break; > + } > + } > + if (!last_pos) > + return 0; // pushed_cond depends only on constant tables > + > + opt_pos= last_pos->least_card_pos; > + > + /* > + Update the cost of the POSITION where the subquery is pushed to, > + and the total cost of the query plan. > + */ > + pushed_cond_cost= opt_pos->partial_join_cardinality * pushed_cond->get_cost(); > + opt_pos->read_time+= pushed_cond_cost; > + > + return pushed_cond_cost; > +} > + > + > +/** > + Make expensive subqueries dynamically pushdownable. > +*/ > + > +bool JOIN::setup_dynamic_conditions() > +{ > + List<Item_func_dynamic_cond> prev_dynamic_conds; > + List<Item_func_dynamic_cond> cur_dynamic_conds; > + bool do_static_pushdown= false, do_dynamic_pushdown= false; > + JOIN_TAB *first_tab, *last_tab, *min_tab, *tab; > + > + if (!(optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_STATIC_PUSHDOWN) || > + optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_DYNAMIC_PUSHDOWN)) || !conds || > + (table_count - const_tables < 2)) > + return false; > + > + first_tab= join_tab + const_tables; > + last_tab= join_tab + top_join_tab_count; > + > + /* > + Update the least cardinality join_tab for each tab for the chosen join order. > + Check if there are any conditions at all that need to be made dynamic. > + */ > + min_tab= NULL; > + for (tab= last_tab - 1; tab >= first_tab; tab--) > + { > + if (optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_STATIC_PUSHDOWN)) psergey: Is there any reason to have this check inside the loop? Please move it outside. > + { > + if (!min_tab || > + tab->partial_join_cardinality <= min_tab->partial_join_cardinality) > + min_tab= tab; > + tab->least_card_tab= min_tab; > + /* > + At least one join_tab must have lower cardinality than its preceeding > + join_tabs in order to use static pushdown. > + */ > + if (tab != tab->least_card_tab) > + do_static_pushdown= true; > + } > + else > + { > + /* Only dynamic pushdown, push conditions to the left-most possible tab. */ > + tab->least_card_tab= tab; > + } > + > + if (optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_DYNAMIC_PUSHDOWN) && > + tab->select_cond && tab->select_cond->needs_dynamic_cond()) > + do_dynamic_pushdown= true; > + } > + > + if (!do_static_pushdown && !do_dynamic_pushdown) > + return false; > + > + /* > + TODO: the following loop processes only the top-level JOIN_TABs. The loop > + skips semi-join nests. Each semi-join nest must be processed in the same > + way as the top-level joins. > + */ > + for (tab= first_tab; tab < last_tab; tab++) > + { > + Item_func_collect_stats *wrapped_cond; > + > + if (tab->select_cond) > + { > + if (is_cond_and(tab->select_cond)) > + { > + List_iterator<Item> and_parts_it(*((Item_cond*) tab->select_cond)->argument_list()); > + Item *and_part; > + while ((and_part= and_parts_it++)) > + { > + if (and_part->needs_dynamic_cond()) > + { > + /* > + TODO: > + If the leftmost join_tab where the dynamic condion can be pushed > + is in fact the last join table, do not wrap the condition since > + it has nowhere to move. > + */ > + if (!(wrapped_cond= new Item_func_dynamic_cond(and_part, > + tab->least_card_tab, > + cur_exec_tab))) > + return true; > + cur_dynamic_conds.push_back((Item_func_dynamic_cond*)wrapped_cond); > + } > + else > + { > + if (!(wrapped_cond= new Item_func_collect_stats(and_part))) > + return true; > + } > + and_parts_it.replace(wrapped_cond); > + } > + } > + else if (tab->select_cond->needs_dynamic_cond()) > + { > + /* > + TODO: > + If the leftmost join_tab where the dynamic condion can be pushed > + is in fact the last join table, do not wrap the condition since > + it has nowhere to move. > + */ > + if (!(wrapped_cond= new Item_func_dynamic_cond(tab->select_cond, > + tab->least_card_tab, > + cur_exec_tab))) > + return true; > + tab->set_select_cond(wrapped_cond, __LINE__); > + cur_dynamic_conds.push_back((Item_func_dynamic_cond*)wrapped_cond); > + } > + else > + { > + if (!(wrapped_cond= new Item_func_collect_stats(tab->select_cond))) > + return true; > + tab->set_select_cond(wrapped_cond, __LINE__); > + } > + } > + > + /* > + Add all dynamic conditions pushed to previous JOIN_TABs also to the > + current JOIN_TAB. This allows to "move" a dynamic condition > + from one tab to another by enabling it for the corrseponding tab. > + */ > + List_iterator_fast<Item_func_dynamic_cond> dyn_cond_it(prev_dynamic_conds); > + while ((wrapped_cond= dyn_cond_it++)) > + { > + if (is_cond_and(tab->select_cond)) > + ((Item_cond_and*)tab->select_cond)->add(wrapped_cond); > + else > + { > + tab->set_select_cond(and_items(tab->select_cond, wrapped_cond), __LINE__); > + tab->select_cond->quick_fix_field(); > + } > + } > + prev_dynamic_conds.concat(&cur_dynamic_conds); > + cur_dynamic_conds.empty(); > + > + /* > + If the top-level condition attached to this JOIN_TAB is not wrapped in a > + condition that can collect statistics, wrap it into one. > + */ > + if (tab->select_cond && > + !(tab->select_cond->type() == Item::FUNC_ITEM && > + (((Item_func*)tab->select_cond)->functype() == Item_func::DYNAMIC_COND_FUNC || > + ((Item_func*)tab->select_cond)->functype() == Item_func::EXEC_STATS_FUNC))) > + { > + if (!(wrapped_cond= new Item_func_collect_stats(tab->select_cond))) > + return true; > + tab->set_select_cond(wrapped_cond, __LINE__); > + } > + > + if (tab->select_cond) > + tab->select_cond->quick_fix_field(); > + } > + > + return false; > +} > + > + > /** > Find a cheaper access key than a given @a key > > > === modified file 'sql/sql_select.h' > --- sql/sql_select.h 2013-07-17 19:24:29 +0000 > +++ sql/sql_select.h 2013-09-27 08:25:14 +0000 > @@ -289,8 +289,8 @@ typedef struct st_join_table { > */ > double read_time; > > - /* psergey-todo: make the below have type double, like POSITION::records_read? */ > - ha_rows records_read; > + /* Copy of POSITION::records_read, set by get_best_combination() */ > + double records_read; > > /* The selectivity of the conditions that can be pushed to the table */ > double cond_selectivity; > @@ -300,6 +300,12 @@ typedef struct st_join_table { > > double partial_join_cardinality; > > + /* Execution-time counters of actual cardinality and selectivity. */ > + /* The cardinality of the partial join represented by this JOIN_TAB. */ > + double exec_partial_join_records; psergey: I understand this is a counter? Why use floating point type for it, then? > + /* Estimate of the fanout of this join based on execution-time counters. */ > + double exec_fanout; > + > table_map dependent,key_dependent; > /* > 1 - use quick select > @@ -377,6 +383,12 @@ typedef struct st_join_table { > NULL - Not doing a loose scan on this join tab. > */ > struct st_join_table *loosescan_match_tab; > + > + /* > + The JOIN_TAB after this one with the least partial join cardinality. > + Valid only after the join optimization phase. > + */ > + st_join_table *least_card_tab; > > /* TRUE <=> we are inside LooseScan range */ > bool inside_loosescan_range; > @@ -522,7 +534,7 @@ typedef struct st_join_table { > return (is_hash_join_key_no(key) ? hj_key : table->key_info+key); > } > double scan_time(); > - ha_rows get_examined_rows(); > + double get_examined_rows_estimate(); > bool preread_init(); > > bool is_sjm_nest() { return test(bush_children); } > @@ -767,6 +779,8 @@ typedef struct st_position :public Sql_a > { > /* The table that's put into join order */ > JOIN_TAB *table; > + /* The JOIN_TAB after this one with the least partial join cardinality */ > + st_position *least_card_pos; > > /* > The "fanout": number of output rows that will be produced (after > @@ -777,6 +791,7 @@ typedef struct st_position :public Sql_a > > /* The selectivity of the pushed down conditions */ > double cond_selectivity; > + double partial_join_cardinality; > > /* > Cost accessing the table in course of the entire complete join execution, > @@ -917,6 +932,7 @@ class JOIN :public Sql_alloc > void restore_query_plan(Join_plan_state *restore_from); > /* Choose a subquery plan for a table-less subquery. */ > bool choose_tableless_subquery_plan(); > + double examined_rows_estimate; > > public: > JOIN_TAB *join_tab, **best_ref; > @@ -1214,13 +1230,20 @@ class JOIN :public Sql_alloc > > bool union_part; ///< this subselect is part of union > > - enum join_optimization_state { NOT_OPTIMIZED=0, > - OPTIMIZATION_IN_PROGRESS=1, > - OPTIMIZATION_DONE=2}; > bool optimized; ///< flag to avoid double optimization in EXPLAIN > bool initialized; ///< flag to avoid double init_execution calls > > - enum { QEP_NOT_PRESENT_YET, QEP_AVAILABLE, QEP_DELETED} have_query_plan; > + enum {QEP_NOT_PRESENT_YET, > + /* A query plan exists. It is being optimized, but is still incomplete. */ > + QEP_INCOMPLETE, > + /* > + A query plan exists, but it is usable only for EXPLAIN because > + some of its structures are incomplete. > + */ > + QEP_EXPLAINABLE, > + /* A query plan exists, and is ready for execution. */ > + QEP_EXECUTABLE, > + QEP_DELETED} have_query_plan; > > /* > Additional WHERE and HAVING predicates to be considered for IN=>EXISTS > @@ -1248,6 +1271,10 @@ class JOIN :public Sql_alloc > TABLE *table_reexec[1]; // make_simple_join() > JOIN_TAB *join_tab_reexec; // make_simple_join() > /* end of allocation caching storage */ > + /* > + The index of the current partial join being processed during query execution. > + */ > + JOIN_TAB **cur_exec_tab; > > JOIN(THD *thd_arg, List<Item> &fields_arg, ulonglong select_options_arg, > select_result *result_arg) > @@ -1259,7 +1286,7 @@ class JOIN :public Sql_alloc > void init(THD *thd_arg, List<Item> &fields_arg, ulonglong select_options_arg, > select_result *result_arg) > { > - join_tab= join_tab_save= 0; > + join_tab= join_tab_save= table_access_tabs= 0; > table= 0; > table_count= 0; > top_join_tab_count= 0; > @@ -1274,6 +1301,7 @@ class JOIN :public Sql_alloc > found_records= 0; > fetch_limit= HA_POS_ERROR; > examined_rows= 0; > + examined_rows_estimate= -1; > exec_tmp_table1= 0; > exec_tmp_table2= 0; > sortorder= 0; > @@ -1334,6 +1362,7 @@ class JOIN :public Sql_alloc > query plan was produced > */ > table_access_tabs= NULL; > + cur_exec_tab= NULL; > } > > int prepare(Item ***rref_pointer_array, TABLE_LIST *tables, uint wind_num, > @@ -1352,6 +1381,7 @@ class JOIN :public Sql_alloc > bool alloc_func_list(); > bool flatten_subqueries(); > bool optimize_unflattened_subqueries(); > + bool optimize_in_subqueries(); > bool optimize_constant_subqueries(); > bool make_sum_func_list(List<Item> &all_fields, List<Item> &send_fields, > bool before_group_by, bool recompute= FALSE); > @@ -1452,9 +1482,13 @@ class JOIN :public Sql_alloc > void get_prefix_cost_and_fanout(uint n_tables, > double *read_time_arg, > double *record_count_arg); > - double get_examined_rows(); > + double get_examined_rows_estimate(); > + double total_subquery_cost(); > + > /* defined in opt_subselect.cc */ > bool transform_max_min_subquery(); > + double static_pushdown_cost(Item *cond, uint idx); > + bool setup_dynamic_conditions(); > /* True if this JOIN is a subquery under an IN predicate. */ > bool is_in_subquery() > { > > === modified file 'sql/sql_test.cc' > --- sql/sql_test.cc 2013-06-06 15:51:28 +0000 > +++ sql/sql_test.cc 2013-09-27 08:25:14 +0000 > @@ -365,7 +365,11 @@ print_plan(JOIN* join, uint idx, double > pos= join->best_positions[i]; > table= pos.table->table; > if (table) > + { > fputs(table->s->table_name.str, DBUG_FILE); > + fprintf(DBUG_FILE, "(records_read: %.2f, read_time: %.2f)", > + pos.records_read, pos.read_time); > + } > fputc(' ', DBUG_FILE); > } > } > @@ -377,10 +381,11 @@ print_plan(JOIN* join, uint idx, double > { > join_table= (*plan_nodes); > fputs(join_table->table->s->table_name.str, DBUG_FILE); > - fprintf(DBUG_FILE, "(%lu,%lu,%lu)", > - (ulong) join_table->found_records, > - (ulong) join_table->records, > - (ulong) join_table->read_time); > + fprintf(DBUG_FILE, "(found_records: %llu, records_read: %llu, records: %llu, read_time: %.2f)", > + (ulonglong) join_table->found_records, > + (ulonglong) join_table->records_read, > + (ulonglong) join_table->records, > + join_table->read_time); > fputc(' ', DBUG_FILE); > } > fputc('\n', DBUG_FILE); > > === modified file 'sql/sql_update.cc' > --- sql/sql_update.cc 2013-07-17 19:24:29 +0000 > +++ sql/sql_update.cc 2013-04-01 10:36:05 +0000 > @@ -358,7 +358,7 @@ int mysql_update(THD *thd, > } > > /* Apply the IN=>EXISTS transformation to all subqueries and optimize them. */ > - if (select_lex->optimize_unflattened_subqueries(false)) > + if (select_lex->optimize_subqueries(NULL)) > DBUG_RETURN(TRUE); > > if (select_lex->inner_refs_list.elements && > > === modified file 'sql/sys_vars.cc' > --- sql/sys_vars.cc 2013-08-26 11:26:21 +0000 > +++ sql/sys_vars.cc 2013-09-27 08:25:14 +0000 > @@ -1836,6 +1836,8 @@ export const char *optimizer_switch_name > "table_elimination", > "extended_keys", > "exists_to_in", > + "expensive_pred_static_pushdown", > + "expensive_pred_dynamic_pushdown", > "default", NullS > }; > /** propagates changes to @@engine_condition_pushdown */ > @@ -1878,7 +1880,9 @@ static Sys_var_flagset Sys_optimizer_swi > "subquery_cache, " > "table_elimination, " > "extended_keys, " > - "exists_to_in " > + "exists_to_in, " > + "expensive_pred_static_pushdown, " > + "expensive_pred_dynamic_pushdown" > "} and val is one of {on, off, default}", > SESSION_VAR(optimizer_switch), CMD_LINE(REQUIRED_ARG), > optimizer_switch_names, DEFAULT(OPTIMIZER_SWITCH_DEFAULT), > @@ -4243,6 +4247,36 @@ static Sys_var_harows Sys_expensive_subq > SESSION_VAR(expensive_subquery_limit), CMD_LINE(REQUIRED_ARG), > VALID_RANGE(0, HA_POS_ERROR), DEFAULT(100), BLOCK_SIZE(1)); > > +static Sys_var_harows Sys_dynamic_pushdown_max_evals( > + "dynamic_pushdown_max_evals", > + "The number of times a dynamic condition must be evaluated in " > + "order to trigger dynamic pushdown", > + SESSION_VAR(dynamic_pushdown_max_evals), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0, HA_POS_ERROR), DEFAULT(100), BLOCK_SIZE(1)); > + > +static Sys_var_uint Sys_dynamic_pushdown_cost_ratio( > + "dynamic_pushdown_cost_ratio", > + "The ratio (in %) by which a new placement of the predicate in the plan" > + "must improve the plan cost in order to trigger dynamic pushdown.", > + SESSION_VAR(dynamic_pushdown_cost_ratio), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0, 100), DEFAULT(25), BLOCK_SIZE(1)); > + > +static Sys_var_uint Sys_dynamic_pushdown_sampling_probability( > + "dynamic_pushdown_sampling_probability", > + "The probability (in %) with which to evaluate this condition in order" > + "to estimate its selectivity.", > + SESSION_VAR(dynamic_pushdown_sampling_probability), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0, 100), DEFAULT(3), BLOCK_SIZE(1)); > + > +static Sys_var_uint Sys_dynamic_pushdown_max_selectivity( > + "dynamic_pushdown_max_selectivity", > + "If a predicate's selectivity is below this limit, it is considered" > + "sufficiently selective to re-filter all records in a partial blocked join." > + "The selectivity itself is estimated by sampling the predicate with" > + "probability 'dynamic_pushdown_sampling_probability'.", > + SESSION_VAR(dynamic_pushdown_max_selectivity), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0, 100), DEFAULT(25), BLOCK_SIZE(1)); > + > static bool check_pseudo_slave_mode(sys_var *self, THD *thd, set_var *var) > { > longlong previous_val= thd->variables.pseudo_slave_mode; > BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

[Maria-developers] @@optimizer_search_depth default value - why 62 and not 0?
by Sergei Petrunia 25 Oct '13

25 Oct '13

Hi Timour, Me and Stephane were discussing a performance issue, and are wondering - why does the @@optimizer_search_depth has a value of 62? There is a vale of optimizer_search_depth=0 which tells the optimizer to chose depth automatically (exhaustive if less than 7 tables, greedy search if more than that). This value looks like it was intended to be the default. Yet, currently the default (in both MariaDB and MySQL) is 62, which tells the optimizer use the "new" (as in "new in 5.1") greedy search code, but sets the depth so high that it uses exhaustive search. Is there any reason behind this? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] Rev 3710: MariaDB made be compiled by gcc 4.8.1
by Sergei Golubchik 24 Oct '13

24 Oct '13

Hi, Sanja! On Oct 24, sanja(a)montyprogram.com wrote: > At file:///home/bell/maria/bzr/work-maria-5.3-test/ > > ------------------------------------------------------------ > revno: 3710 > revision-id: sanja(a)montyprogram.com-20131024170011-r5r6fd84y4x1yr09 > parent: bar(a)mnogosearch.org-20131021093629-0nyk7osjssern1sa > committer: sanja(a)montyprogram.com > branch nick: work-maria-5.3-test > timestamp: Thu 2013-10-24 20:00:11 +0300 > message: > MariaDB made be compiled by gcc 4.8.1 Okay, that looks much better. Please push it in a stage tree, just to check that it doesn't break on all other compilers that we use, and then push into main. Thanks! Regards, Sergei

1 0

Re: [Maria-developers] MariaDB made be compiled by gcc 4.8.1
by Sergei Golubchik 24 Oct '13

24 Oct '13

Hi, Sanja! > >>> === modified file 'storage/maria/ma_checksum.c' > >>> --- a/storage/maria/ma_checksum.c 2008-04-03 13:40:25 +0000 > >>> +++ b/storage/maria/ma_checksum.c 2013-10-22 07:56:05 +0000 > >>> @@ -58,7 +58,7 @@ ha_checksum _ma_checksum(MARIA_HA *info, > >>> length= _ma_calc_blob_length(blob_size_length, pos); > >>> if (length) > >>> { > >>> - memcpy((char*) &pos, pos + blob_size_length, sizeof(char*)); > >>> + memcpy((char*) &pos, pos + blob_size_length, (sizeof(char*))); > >> Why? > > ma_checksum.c: In function '_ma_checksum': > > ma_checksum.c:61:60: error: argument to 'sizeof' in 'memcpy' call is the > > same pointer type 'char *' as the destination; expected 'char' or an > > explicit length [-Werror=sizeof-pointer-memaccess] > > memcpy((char*) &pos, pos + blob_size_length, sizeof(char*)); > > ^ > > shouldn't it be: > memcpy((char**) &pos, pos + blob_size_length, sizeof(char*)); Sanja ^^^ That would be a better fix. The result looks more logical than with meaningless parentheses. Or, I suppose, you can try to drop the cast completely: memcpy(&pos, pos + blob_size_length, sizeof(pos)); > > BTW all other such fixes was done because this check. Yes, I've figured it out. Regards, Sergei

2 2

[Maria-developers] hash + like
by Roberto Spadim 24 Oct '13

24 Oct '13

Hi guys, how could i optimize a hash index using like? i was thinking about something like a index over first left bytes using hash, example: field LIKE "abc%" create index hash_index( field(3) ) using hash any idea if this could be possible? this could boost some queries here that i'm using btree today i'm using the LIKE, but i'm considering a virtual column + a index over virtual column + a ( AND virtual_column='abc') any other idea? or maybe optimize the LIKE function -- Roberto Spadim

2 1

Re: [Maria-developers] MariaDB made be compiled by gcc 4.8.1
by Sergei Golubchik 24 Oct '13

24 Oct '13

Hi, Sanja! See below On Oct 22, sanja(a)montyprogram.com wrote: > message: > MariaDB made be compiled by gcc 4.8.1 > === modified file 'extra/yassl/include/yassl_int.hpp' > --- a/extra/yassl/include/yassl_int.hpp 2012-04-05 08:49:38 +0000 > +++ b/extra/yassl/include/yassl_int.hpp 2013-10-22 07:56:05 +0000 > @@ -440,7 +440,7 @@ public: > const Ciphers& GetCiphers() const; > const DH_Parms& GetDH_Parms() const; > const Stats& GetStats() const; > - const VerifyCallback getVerifyCallback() const; > + VerifyCallback getVerifyCallback() const; Why? > === modified file 'storage/maria/ma_checksum.c' > --- a/storage/maria/ma_checksum.c 2008-04-03 13:40:25 +0000 > +++ b/storage/maria/ma_checksum.c 2013-10-22 07:56:05 +0000 > @@ -58,7 +58,7 @@ ha_checksum _ma_checksum(MARIA_HA *info, > length= _ma_calc_blob_length(blob_size_length, pos); > if (length) > { > - memcpy((char*) &pos, pos + blob_size_length, sizeof(char*)); > + memcpy((char*) &pos, pos + blob_size_length, (sizeof(char*))); Why? > crc= my_checksum(crc, pos, length); > } > continue; Regards, Sergei

2 1

[Maria-developers] gcc warning
by Roberto Spadim 20 Oct '13

20 Oct '13

hi guys! just a warning , bye :) [ 99%] Building C object tests/CMakeFiles/bug25714.dir/bug25714.c.o /home/src/mariadb/mariadb-5.5.33a/tests/bug25714.c: In function 'main': /home/src/mariadb/mariadb-5.5.33a/tests/bug25714.c:26:7: warning: variable 'OK' set but not used [-Wunused-but-set-variable] int OK; ^ -- Roberto Spadim

1 0

Re: [Maria-developers] Comments for parallel replication patch (MDEV-4506)
by Kristian Nielsen 18 Oct '13

18 Oct '13

Michael Widenius <michael.widenius(a)gmail.com> writes: >> What about Rotate_log_event::do_update_pos() and >> Stop_log_event::do_update_pos()? >> >> They currently access the flag in rli. I am not sure that is right, as > this >> could be far ahead of the event they are actually executing? > > Aren't the above events handled by the sql driver thread? No, by the worker threads. I do not understand how it could be done in the driver thread? do_update_pos updates the current position of the last event executed, it cannot happen until after events have been executed? And this happens in the worker threads. > My thought was that the driver thread was in charge of all reading of the > relay log and would thus hide any stop or rotate events from the sql > execution threads. The Rotate event is what updates the group_master_log_name to the name of the next binlog file on the master. This must not happen until previous events have been executed (otherwise they will be wrongly skipped if we stop and restart the slave). Or did I misunderstand? My best suggestion at the moment is to copy the required information from m_flags (and a couple of other fields from rli) into rgi, so that the right value gets associated with the right event in each worker thread. But let's discuss when you are back from travelling if you have a better idea. > The purgeing of log should be changed to check the position of the st > committed event. It's safe to delay any older logs. > The other I am not sure about as I can't look at the code just know. > However we need a solution that works also for masters that don't support > GTID. Yes, the non-GTID case is the main problem. (Using GTID, everything is updated automatically in parallel in the mysql.gtid_slave_pos and there is a sub_id field that ensures we can figure out which GTIDs are the most recent ones). - Kristian.

1 0

[Maria-developers] Parallel replication MDEV-4506
by Michael Widenius 14 Oct '13

14 Oct '13

Hi! Just a quick status update of parallel replication in MariaDB 10.0 I yesterday pushed to the 10.0-knielsen tree changes that makes many of the replication variables thread safe. I will today push a patch that should fix the rest of the variables. Here is the changelog entry for this: Fixes for parallel slave: - Made slaves temporary table multi-thread slave safe by adding mutex around save_temporary_table usage. - rli->save_temporary_tables is the active list of all used temporary tables - This is copied to THD->temporary_tables when temporary tables are opened and updated when temporary tables are closed - Added THD->lock_temporary_tables() and THD->unlock_temporary_tables() to simplify this. - Relay_log_info->sql_thd renamed to Relay_log_info->sql_driver_thd to avoid wrong usage for merged code. - Added is_part_of_group() to mark functions that are part of the next function. This replaces setting IN_STMT when events are executed. - Added is_begin(), is_commit() and is_rollback() functions to Query_log_event to simplify code. - If slave_skip_counter is set run things in single threaded mode. This simplifies code for skipping events. - Updating state of relay log (IN_STMT and IN_TRANSACTION) is moved to one single function: update_state_of_relay_log() We can't use OPTION_BEGIN to check for the state anymore as the sql_driver and sql execution threads may be different. Clear IN_STMT and IN_TRANSACTION in init_relay_log_pos() and Relay_log_info::cleanup_context() to ensure the flags doesn't survive slave restarts is_in_group() is now independent of state of executed transaction. - Reset thd->transaction.all.modified_non_trans_table() if we did set it for single table row events. This was mainly for keeping the flag as documented. - Changed slave_open_temp_tables to uint32 to be able to use atomic operators on it. - Relay_log_info::sleep_lock -> rpl_group_info::sleep_lock - Relay_log_info::sleep_cond -> rpl_group_info::sleep_cond - Changed some functions to take rpl_group_info instead of Relay_log_info to make them multi-slave safe and to simplify usage - do_shall_skip() - continue_group() - sql_slave_killed() - next_event() - Simplifed arguments to io_slave_killed(), check_io_slave_killed() and sql_slave_killed(); No reason to supply THD as this is part of the given structure. - set_thd_in_use_temporary_tables() removed as in_use is set on usage - Added information to thd_proc_info() which thread is waiting for slave mutex to exit. - In open_table() reuse code from find_temporary_table() Other things: - More DBUG statements - Fixed the rpl_incident.test can be run with --debug - More comments - Disabled not used function rpl_connect_master() The TODO for parallel replication is documented at top of rpl_parallel.cc. Here is the comment: ------- - Error handling. If we fail in one of multiple parallel executions, we need to make a best effort to complete prior transactions and roll back following transactions, so slave binlog position will be correct. And all the retry logic for temporary errors like deadlock. - Stopping the slave needs to handle stopping all parallel executions. And the logic in sql_slave_killed() that waits for current event group to complete needs to be extended appropriately... - Audit the use of Relay_log_info::data_lock. Make sure it is held correctly in all needed places also when using parallel replication. - We need some user-configurable limit on how far ahead the SQL thread will fetch and queue events for parallel execution (otherwise if slave gets behind we will fill up memory with pending malloc()'ed events). - Fix update of relay-log.info and master.info. In non-GTID replication, they must be serialised to preserve correctness. In GTID replication, we should not update them at all except at slave thread stop. - All the waits (eg. in struct wait_for_commit and in rpl_parallel_thread_pool::get_thread()) need to be killable. And on kill, everything needs to be correctly rolled back and stopped in all threads, to ensure a consistent slave replication state. - Handle the case of a partial event group. This occurs when the master crashes in the middle of writing the event group to the binlog. The slave rolls back the transaction; parallel execution needs to be able to deal with this wrt. commit_orderer and such. - We should notice if the master doesn't support GTID, and then run in single threaded mode against that master. This is needed to be able to support multi-master-replication with old and new masters. - Retry of failed transactions is not yet implemented for the parallel case. ---------- What this means is: In theory things should work, as long as there is no problems in the binary or relay log and we don't fill up memory with too many events. We will merge this branch to 10.0-base and then to 10.0 withing the next few days and start testing it. We also plan to release a beta of 10.0 with this code ASAP as 10.0 is now feature complete. During the beta phase we will fix the above outstanding issues. Kristian will this week start working on the error handling. After that we have to look at the memory consumption (ie, not give the sql execution threads more work than they can handle). I will shortly update the following article with information about how to use parallel replication. https://mariadb.com/kb/en/parallel-replication/ The task itself is documented at: https://mariadb.atlassian.net/browse/MDEV-4506 Regards, Monty

1 0

[Maria-developers] Wrong SQL thread position reporting after IO thread restart
by Pavel Ivanov 11 Oct '13

11 Oct '13

Krisitan, I found what I think is a bug in IO and SQL thread accounting. How to reproduce: 1) Set up two servers S1 and S2. S1 is a master, S2 is slave with master_using_gtid = current_pos. 2) Execute some transactions on the master, e.g. create database d; create table d.t (n int); insert into d.t values (1); 3) Both servers are at 0-1-3 now, SHOW SLAVE STATUS on S2 shows Read_Master_Log_Pos equals to Exec_Master_Log_Pos. 4) Execute STOP SLAVE IO_THREAD on S2. 5) S2 reports in the logs: "Slave I/O thread exiting, ... GTID position 0-1-2". So IO thread didn't realize that it received full transaction for 0-1-3 even though it didn't receive next GTID event. 6) Execute START SLAVE IO_THREAD on S2. 7) At this point SHOW SLAVE STATUS on S2 shows Read_Master_Log_Pos the same as in step 3, but Exec_Master_Log_Pos is now less than in step 3 as if SQL thread didn't catch up with IO thread yet. But despite both threads running and no more transactions executed on master Exec_Master_Log_Pos doesn't change and doesn't become equal to Read_Master_Log_Pos. This apparently happens because IO thread restarts from one transaction behind, adds to relay log Rotate event that master sends with the position of that transaction, but then doesn't add any events for the transaction because it knows they already were added into relay log. I think both problems are bugs. And although after fixing the first it would be really hard (if possible) to reproduce second, I'd think the reporting of SQL thread's position still should be fixed. Thank you, Pavel

2 2

Re: [Maria-developers] [Commits] Rev 3702: Fixed bug mdev-5105. in file:///home/igor/maria/maria-5.3-mdev5105/
by Oleksandr Byelkin 10 Oct '13

10 Oct '13

Hi! 10.10.2013 20:08, Igor Babaev пишет: > At file:///home/igor/maria/maria-5.3-mdev5105/ > > ------------------------------------------------------------ > revno: 3702 > revision-id: igor(a)askmonty.org-20131010170826-dl2xwc048hvnxmau > parent: igor(a)askmonty.org-20131004165107-0hdknrn9111ojupx > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-mdev5105 > timestamp: Thu 2013-10-10 10:08:26 -0700 > message: > Fixed bug mdev-5105. > The bug caused a memory overwrite in the function update_ref_and_keys() > It happened due to a wrong value of SELECT_LEX::cond_count. This value > historically was calculated by the fix_fields method. Now the logic of > calling this method became too complicated and, as a result, this value > is calculated not always correctly. > The patch changes the way how and when the values of SELECT_LEX::cond_count > and of SELECT_LEX::between_count are calculated. The new code does it just at > the beginning of update_ref_and_keys(). > > OK to push.

1 0

[Maria-developers] On predicate pushdown in presense of join buffering.
by Sergei Petrunia 10 Oct '13

10 Oct '13

Hello, Here is the result of discussion on Join Buffering + Static Predicate Pushdown. == Problem setting == Consider a query using join buffering +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 2 | | | 1 | SIMPLE | t2 | ALL | NULL | NULL | NULL | NULL | 2 | Using where; Using join buffer (flat, BNL join) | | 1 | SIMPLE | t3 | ALL | NULL | NULL | NULL | NULL | 2 | Using join buffer (flat, BNL join) | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ Consider a predicate "P(t2)". Join buffering works as follows: read records from table t1 and put them into buffer; for each record in table t2 { if (cond(t2)) { for each record in the join buffer { if (cond(t1, t2) { pass record for further processing } } } } cond(2) is stored in "join_tab->cache_select->cond" cond(t1, t2) is stored in join_tab->select_cond". The question is, how should Static Predicate Pushdown work with join buffering? Should the condition be put into cache_select->cond or into select_cond of some later join_tab? Currently, the server will put the predicate P(t2) into both join_tab[t2]->select_cond and join_tab[t2]->cache_select->cond. (And subquery cache will wrap them into different caches) == Solution == First observation: if the expensive subquery predicate depends on several tables (not counting the const tables) then it will never be put into cache_select->cond. join_tab[$TBL]->cache_select->cond can only have conditions that depend only on table $TBL. If a predicate P(T) depends only on table $T, it can be put into 1. join_tab[$T]->cache_select->cond 2. join_tab[$T]->select_cond 3. join_tab[$T+1]->select_cond 4. join_tab[$T+2]->select_cond ... Putting the predicate into both #1 and #2 does not make sense. We have decided that Timour will remove duplicate predicate pushdown from cache_select and select_cond. After that feature, the predicates in cache_select->cond will not have duplicates in cache_select. That way, the options for Static Predicate Pushdown will be: 1. join_tab[$T]->cache_select->cond 3. join_tab[$T+1]->select_cond 4. join_tab[$T+2]->select_cond ... === Needed adjustments in optimizer === Consider the EXPLAIN again: +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 2 | | | 1 | SIMPLE | t2 | ALL | NULL | NULL | NULL | NULL | 2 | Using where; Using join buffer (flat, BNL join) | | 1 | SIMPLE | t3 | ALL | NULL | NULL | NULL | NULL | 2 | Using join buffer (flat, BNL join) | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ and a predicate P(t2). Currently, Static Predicate Pushdown will consider putting the predicate at: t2->select_cond: fanout(t1) * fanout(t2) predicate invocations t3->select_cond: fanout(t1) * fanout(t2)* fanout(t3) predicate invocations (here, fanout takes into account selectivity, so it can be that fanout <1) If P(t2) is not attached to t2->select_cond but to t2->cache_select->cond instead, the number of invocations will change to: t2->cache_select->cond: fanout(t1) / number_of_t1_records_in_join_buffer * fanout(t2). The above is obvious for non-BKA case. ==== A note about subquery predicate cache ==== [A theoretical comment] We're not going to do this, but: if P(t2) was attached [ONLY] to t2->select_cond, execution of the query will cause these invocations: P(t2.row1) ---+ P(t2.row1) |-- number_of_t1_records_in_join_buffer times ... | P(t2.row1) ----+ P(t2.row2) ----+ P(t2.row2) |-- number_of_t1_records_in_join_buffer times ... | P(t2.row2) ----+ ... etc. That is, subquery predicate cache will reduce the number of times the predicate is executed. == Applicability for join algorithms == Basic idea: when join buffer is using BKA, join_tab->cache_select==NULL, there is no cache_select. Details: Join algorithms where join_tab->cache_selct->cond is used: - JOIN_CACHE_BNL - JOIN_CACHE_BNLH - JOIN_CACHE_BKAH BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Questions re MDEV-4736 and MDEV-4739 (was Re: Spider's installation sql file)
by Sergey Vojtovich 07 Oct '13

07 Oct '13

Hi Kentoku, I'm adding MariaDB developers to CC. On Thu, Sep 19, 2013 at 01:19:13AM +0900, kentoku wrote: > Hi Sergey, > > > But what kind of errors are possible in your case? Other storage engines > doesn't > seem to suffer from this API violation. > > Spider support bulk updating and deleting for avoiding network roundtrip > between data node. Some times, last bulk updating is executed in rnd_end() > function. So rnd_end() has possibility getting errors from data node. I'm afraid fixing rnd_end() callers in the server may stall for a long time. Is it accaptable for spider to use bulk updates and deletes API instead (see handler.h: start_bulk_update/start_bulk_delete). > By the way, about MDEV-4739. I get the following stack trace. > Program received signal SIGSEGV, Segmentation fault. > 0x00000000005eabf6 in thd_wait_begin (thd=0x29da060, > wait_type=10) > at /ssd1/mariadb-10.0.4/sql/sql_class.cc:4277 > 4277 MYSQL_CALLBACK(thd->scheduler, thd_wait_begin, (thd, wait_type)); > (gdb) print thd > $1 = (THD *) 0x29da060 > (gdb) bt > #0 0x00000000005eabf6 in thd_wait_begin ( > thd=0x29da060, wait_type=10) > at /ssd1/mariadb-10.0.4/sql/sql_class.cc:4277 > #1 0x000000000072a114 in scheduler_wait_sync_begin () > at /ssd1/mariadb-10.0.4/sql/scheduler.cc:59 > #2 0x0000000000d6dc20 in my_sync (fd=23, my_flags=0) > at /ssd1/mariadb-10.0.4/mysys/my_sync.c:76 > #3 0x0000000000d6b54f in my_msync (fd=23, > addr=0x7ffff7ff4000, len=4096, flags=4) > at /ssd1/mariadb-10.0.4/mysys/my_mmap.c:27 > #4 0x00000000008bea03 in TC_LOG_MMAP::open ( > this=0x16e6a00, opt_name=0xe19c87 "tc.log") > at /ssd1/mariadb-10.0.4/sql/log.cc:7735 > #5 0x00000000005751cb in init_server_components () > at /ssd1/mariadb-10.0.4/sql/mysqld.cc:4797 > #6 0x0000000000575a07 in mysqld_main (argc=30, > argv=0x1f204d0) > at /ssd1/mariadb-10.0.4/sql/mysqld.cc:5208 > #7 0x000000000056d884 in main (argc=11, > argv=0x7fffffffe3a8) > at /ssd1/mariadb-10.0.4/sql/main.cc:25 > (gdb) print thd->scheduler > $2 = (scheduler_functions *) 0x8f8f8f8f8f8f8f8f > (gdb) print thd_wait_begin > $3 = {void (THD *, > int)} 0x5eaba4 <thd_wait_begin(THD*, int)> > (gdb) print wait_type > $4 = 10 > > It is looks that "thd->scheduler" is not initialized. What do you think? > Must storage engine set it? Nope, it looks like a bug in thread pool. MDEV-4739 has different trace, how did you get this one? Just executed given test? Regards, Sergey

3 16

[Maria-developers] Comments for parallel replication patch (MDEV-4506)
by Kristian Nielsen 07 Oct '13

07 Oct '13

Hi Monty, I read through your latest patch for the MDEV-4506 task, parallel replication. Just to re-cap, this is a work-in-progress patch, so these are just some comments along the way that I hope will be useful. > -static bool sql_slave_killed(THD* thd, Relay_log_info* rli) > +static bool sql_slave_killed(rpl_group_info *rgi) > -static Log_event* next_event(Relay_log_info* rli) > +static Log_event* next_event(rpl_group_info *rgi) This does not look right to me. Can you explain the reason to use rgi here instead of rli? To me, rgi is about what happens in parallel in each worker thread. But these functions are about what happens in the driver SQL thread, fetching each event from the relay log and checking if the driver SQL thread is done and can exit. It looks from the patch like rgi is always passed the serial_rgi. I think it would be better if it did not use rgi at all, we could pass in any extra information needed directly. Note that serial_rgi should _only_ be used in the case where parallel replication is not activated and actual event execution happens inside the main driver SQL thread. Or did I mis something? > Log_event::enum_skip_reason > -Gtid_log_event::do_shall_skip(Relay_log_info *rli) > +Gtid_log_event::do_shall_skip(rpl_group_info *rgi) > { > + Relay_log_info *rli= rgi->rli; > /* > An event skipped due to @@skip_replication must not be counted towards the > number of events to be skipped due to @@sql_slave_skip_counter. > @@ -6315,10 +6319,13 @@ Gtid_log_event::do_shall_skip(Relay_log_ > if (rli->slave_skip_counter > 0) > { > if (!(flags2 & FL_STANDALONE)) > + { > thd->variables.option_bits|= OPTION_BEGIN; > - return Log_event::continue_group(rli); > + rgi->rli->set_flag(Relay_log_info::IN_TRANSACTION); > + } > + return Log_event::continue_group(rgi); > } > - return Log_event::do_shall_skip(rli); > + return Log_event::do_shall_skip(rgi); > } This looks to me like it cannot work. shall_skip() is called from apply_event_and_update_pos() which runs in each worker thread, in parallel. But rli->slave_skip_counter needs to be checked from the driver SQL thread, in the order that events are read from the relay log. Otherwise, if slave_skip_counter is 1 and we have transactions T1 and T2, T2 might run first and we would end up skipping T2 instead of T1 which is wrong. Besides, it seems better to in as many cases as possible to skip events by not queuing them for a worker thread in the first place. I am wondering if this change came as part of a mechanical change from rli to rgi when is_in_group() was moved from rli to rgi? That is why I had this ToDo comment: > - - Relay_log_info::is_in_group(). This needs to be handled correctly in all > - callers. I think it needs to be split into two, one version in > - Relay_log_info to be used from next_event() in slave.cc, one to be used in > - per-transaction stuff. I think what needs to be done here is that event skip handling needs to be moved, so that it is done in exec_relay_log_event() rather than in apply_event_and_update_pos(). Then it can be done always in the driver SQL thread, both in the parallel and in the non-parallel case. It is possible that we then may need the flag associated with is_in_group() both in rli and in rgi, we will see when we do the change. What do you think? > @@ -71,6 +66,7 @@ rpt_handle_event(rpl_parallel_thread::qu > > /* ToDo: Access to thd, and what about rli, split out a parallel part? */ > mysql_mutex_lock(&rli->data_lock); > + qev->ev->thd= thd; Indeed, this seems to be needed. Wow. I'm surprised that I have been able to get anything to work when this was missing?!? Or is it because in most cases, THD is passed into Log_event:: functions directly, rather than using Log_event::thd ? > @@ -1094,9 +1090,9 @@ bool Relay_log_info::is_until_satisfied( > !replicate_same_server_id) > DBUG_RETURN(FALSE); > log_name= group_master_log_name; > - log_pos= (!ev)? group_master_log_pos : > - ((thd->variables.option_bits & OPTION_BEGIN || !ev->log_pos) ? > - group_master_log_pos : ev->log_pos - ev->data_written); > + log_pos= ((!ev)? group_master_log_pos : > + (get_flag(IN_TRANSACTION) || !ev->log_pos) ? > + group_master_log_pos : ev->log_pos - ev->data_written); Yes, it is clear that we need something like this. You told me on IRC that you still needed to work on this. Eg. I see you started to add a part_of_next_statement() but did not implement this. I also am worried how to make sure that we do not miss any cases here, and agree it needs to be checked. I will just mention that for GTID I already added Log_event::is_part_of_group(), which seems related to your part_of_next_statement()? Maybe they could use the same mechanism. One outstanding issue with is_part_of_group is for row-based replication with multi-table update statements. In such multi-table updates there will be two (or more) Rows_log_event events, and there is a flag to tell which is the last one. They need to be considered part of the same statement. I think this is not yet correctly handled in the GTID code (MDEV-4985). Probably this case also needs to be handled correctly here? > - if ((rgi->thd->variables.option_bits & OPTION_BEGIN) && opt_using_transactions) > + if ((rgi->rli->get_flag(IN_TRANSACTION)) && opt_using_transactions) There is more to be done here than this. Because this runs in each parallel worker thread, in parallel. Wouldn't it be more correct actually to use rgi->thd here, actually? But in any case, we need to think about how to handle old-style replication position updates. Since we already ensure that transactions commit in-order, maybe it is ok to do the updates to position in parallel, as long as we do not decrement the position. So if we have transactions T1 and T2 running in parallel, T1 will commit before T2, but T2 may then race ahead and update replication position before T1. That is ok (both T1 and T2 are committed), but then we need in T1 to check and see that position is already ahead, so we do not set it back to T1 (that could cause duplication of T2 in case of slave restart). And in the GTID case, where the replication position is not used in case of slave restart, then it would be best not to update the master.info and relay-log.info file after every transaction (to save the cost of write + possibly fsync). But it would be best to still update it in-memory so that SHOW SLAVE STATUS shows something meaningful. > + Flags for the state of reading the relay log. Note that these are > + bits "Note that these are bit masks" is less ambiguous. > + This is updated when a temporary table is created or drooped by > + a replication thread. "drooped" -> "dropped". > + /** > + Is the replication inside a group? > + > + Replication is inside a group if either: > + - The IN_TRANSACTION flag is set, meaning we're inside a transaction > + - The IN_STMT flag is set, meaning we're inside a statement > + > + @retval true Replication thread is currently inside a group > + @retval false Replication thread is currently not inside a group > + */ > + bool is_in_group() const { > + return (m_flags & (IN_STMT | IN_TRANSACTION)); > + } Do I understand correctly that this refers to the last event fetched from the relay log, not the last event executed? These are different in the case of parallel replication. I think the comment should be clarified to explicitly say one way or the other. Hm, I see now that this is not your comment (you just moved it), so it is ok, but it would still be good if we could clarify the comment. > -static bool sql_slave_killed(THD* thd, Relay_log_info* rli) > +static bool sql_slave_killed(rpl_group_info *rgi) > { > /* > - The transaction should always be binlogged if OPTION_KEEP_LOG is set > - (it implies that something can not be rolled back). And such case > - should be regarded similarly as modifing a non-transactional table > - because retrying of the transaction will lead to an error or inconsistency > - as well. > - Example: OPTION_KEEP_LOG is set if a temporary table is created or dropped. > + The transaction should always be binlogged if OPTION_KEEP_LOG is > + set (it implies that something can not be rolled back). And such > + case should be regarded similarly as modifing a > + non-transactional table because retrying of the transaction will > + lead to an error or inconsistency as well. > + > + Example: OPTION_KEEP_LOG is set if a temporary table is created > + or dropped. > */ > if ((thd->transaction.all.modified_non_trans_table || > (thd->variables.option_bits & OPTION_KEEP_LOG)) So checking thd->transaction is not meaningful in case of parallel replication. However, this is related to stopping the slave, which as we discussed before is still work-in-progress. I think we can keep this code to handle the non-parallel case, and then add a similar check in the worker threads (maybe in a new function sql_worker_killed()). And maybe then add a check here that all worker threads have stopped. I'm looking into this. So this should be ok for now, just wanted to mention it. - Kristian.

1 0

Re: [Maria-developers] Rev 3847: Client attributes
by Sergei Golubchik 04 Oct '13

04 Oct '13

Hi, Sanja! On Oct 02, sanja(a)askmonty.org wrote: > === modified file 'client/mysqltest.cc' > --- a/client/mysqltest.cc 2013-07-21 14:39:19 +0000 > +++ b/client/mysqltest.cc 2013-10-02 13:54:55 +0000 > @@ -5648,6 +5648,10 @@ void safe_connect(MYSQL* mysql, const ch > verbose_msg("Connecting to server %s:%d (socket %s) as '%s'" > ", connection '%s', attempt %d ...", > host, port, sock, user, name, failed_attempts); > + > + mysql_options(mysql, MYSQL_OPT_CONNECT_ATTR_RESET, 0); > + mysql_options4(mysql, MYSQL_OPT_CONNECT_ATTR_ADD, > + "program_name", "mysqltest"); Did you test connecting without connection attributes? e.g. 5.5 client and 10.0 server? and mysql-5.5 client? And mysql-5.6 client? please do, at least manually > while(!mysql_real_connect(mysql, host,user, pass, db, port, sock, > CLIENT_MULTI_STATEMENTS | CLIENT_REMEMBER_OPTIONS)) > { > > === modified file 'include/mysql.h' > --- a/include/mysql.h 2013-07-18 14:46:57 +0000 > +++ b/include/mysql.h 2013-10-02 13:54:55 +0000 > @@ -557,6 +557,8 @@ int STDCALL mysql_list_proce > int status); > int STDCALL mysql_options(MYSQL *mysql,enum mysql_option option, > const void *arg); > +int STDCALL mysql_options4(MYSQL *mysql,enum mysql_option option, > + const void *arg1, const void *arg2); This is pretty silly, I'd rather have mysql_optionv(MYSQL *mysql, enum mysql_option option, ...) or even (redefining the existing function): mysql_option(MYSQL *mysql, enum mysql_option option, ...) but we're copying MySQL here, so we don't have much of a choice > void STDCALL mysql_free_result(MYSQL_RES *result); > int STDCALL mysql_free_result_start(MYSQL_RES *result); > int STDCALL mysql_free_result_cont(MYSQL_RES *result, int status); > > === modified file 'sql/sql_acl.cc' > --- a/sql/sql_acl.cc 2013-09-21 08:14:42 +0000 > +++ b/sql/sql_acl.cc 2013-10-02 13:54:55 +0000 > @@ -8570,7 +8628,13 @@ static ulong parse_client_handshake_pack > mpvio->acl_user->plugin= old_password_plugin_name; > } > } > - > + > + if (db && Why do you check for db being not NULL? > + (thd->client_capabilities & CLIENT_CONNECT_ATTRS) && > + read_client_connect_attrs(&next_field, ((char *)net->read_pos) + pkt_len, > + mpvio->thd->charset())) > + return packet_error; > + > /* > if the acl_user needs a different plugin to authenticate > (specified in GRANT ... AUTHENTICATED VIA plugin_name ..) > > === modified file 'sql/sys_vars.cc' > --- a/sql/sys_vars.cc 2013-09-21 08:14:42 +0000 > +++ b/sql/sys_vars.cc 2013-10-02 13:54:55 +0000 > @@ -71,6 +71,7 @@ > #define export /* not static */ > > #ifdef WITH_PERFSCHEMA_STORAGE_ENGINE > +#ifndef EMBEDDED_LIBRARY Why is this? > > static Sys_var_mybool Sys_pfs_enabled( > "performance_schema", Regards, Sergei

2 3

[Maria-developers] Adding versions to documentation
by Daniel Bartholomew 03 Oct '13

03 Oct '13

All, Whenever we document new features in the Knowledge Base, one thing we need to do is say when it was (or will be) introduced. For example, if I'm documenting a feature that was introduced in MariaDB 10.0.4, I would add something similar to the following to the top of the page: <<product mariadb from=10.0.4>> This feature was introduced in MariaDB 10.0.4 <</product>> If you're in a hurry and can't remember the <<product>> macro syntax, just use a sentence. We'll wrap it in a product macro later. Also, if you come across an article that doesn't have this information, please add it if you can, or send me an email so that I know it's missing. Thanks! -- Daniel Bartholomew, Sr Tech Writer / Sys Admin MariaDB

1 0

[Maria-developers] [Gsoc]Regexp enhancements License
by Sudheera Palihakkara 03 Oct '13

03 Oct '13

Hi all, This is to inform you that the code that I created for the task "Regexp enhancements" during "Google Summer Of Code 2013" is available to to MariaDB Foundation under terms of the New BSD License. thank you. -- *Sudheera Palihakkara.* Undergraduate Department of *Computer Science and Engineering, *Faculty of Engineering, *University of Moratuwa*, Sri Lanka.

1 0

[Maria-developers] Replicating same server_id problem
by Pavel Ivanov 01 Oct '13

01 Oct '13

Kristian, Currently MariaDB (as well as MySQL of all previous versions) has a very big problem related to replicating same server_id. There is --replicate-same-server-id flag which as I understand (when set to 0) controls two things: 1) It doesn't allow slave to connect to a master with the same server_id. 2) Slave ignores all binlog events in the replication stream that have the same server_id as slave. And this flag cannot be set to 1 when --log-slave-updates is used. And that is a big problem. Consider the following scenario: let's say we have two servers S1 (master) and S2 (slave). Let's say at some moment in time they are completely in sync and you bring down S2 to take cold backup (you can even include binlogs in it). Then you bring it back up, S1 is still master. Now you execute some transactions, then you do a failover, make S2 master and execute some more transactions. Then you bring down S1, restore it from the backup taken earlier and connect to replicate from S2 again. At this point S1 will have to reply some transaction that have the same server_id as S1 has, but S1 doesn't have those transactions yet, so it shouldn't skip them, but it will. What do you think about how this should be fixed? As I understand you explicitly wanted to support replication cycles, so you still want the skipping of transactions with the same server_id to exist. But the situation above is a valid production use case. Maybe in GTID world it can be solved better? E.g. if transaction has the same server_id, but the GTID wasn't applied yet then it shouldn't be skipped? Thank you, Pavel

4 8

[Maria-developers] Galera and aria
by Roberto Spadim 01 Oct '13

01 Oct '13

Hi again guys! =) I have a doubt, i was using a diff -u galera-5.33a mariadb-5.33a | grep -i "myisam" and found some lines... well could we add the aria storage to 'work' like myisam with galera cluster? i don't know if it's the right way (changing "== MYISAM_ENGINE" to something like (== myisam_engine OR == maria_engine)) if yes... i could make this patch and send? should i create a mdev to this task (allow aria engine with galera cluster)? today it's not a trasational engine, but it's better than myisam with the crash safe feature that's all :) thanks guys -- Roberto Spadim

2 4

[Maria-developers] regexp review
by Oleksandr Byelkin 30 Sep '13

30 Sep '13

Hi! Hi! Everything is OK. But there are some small issues: --- mysql-test/include/ctype_utf8mb4.inc 2010-03-05 08:17:19 +0000 +++ mysql-test/include/ctype_utf8mb4.inc 2013-09-26 14:02:17 +0000 @@ -234,15 +234,15 @@ set names utf8mb4; set names utf8mb4; # This should return TRUE -select 'вася' rlike '[[:<:]]вася[[:>:]]'; -select 'вася ' rlike '[[:<:]]вася[[:>:]]'; -select ' вася' rlike '[[:<:]]вася[[:>:]]'; -select ' вася ' rlike '[[:<:]]вася[[:>:]]'; +select 'вася' rlike '\\bвася\\b'; +select 'вася ' rlike '\\bвася\\b'; +select ' вася' rlike '\\bвася\\b'; +select ' вася ' rlike '\\bвася\\b'; Is above unsupported pattern? === modified file 'sql/mysqld.cc' --- sql/mysqld.cc 2013-09-18 11:07:31 +0000 +++ sql/mysqld.cc 2013-09-26 14:02:17 +0000 @@ -1898,7 +1898,7 @@ void clean_up(bool print_message) delete global_rpl_filter; end_ssl(); vio_end(); - my_regex_end(); + //my_regex_end(); #if defined(ENABLED_DEBUG_SYNC) /* End the debug sync facility. See debug_sync.cc. */ debug_sync_end(); @@ -3904,10 +3904,10 @@ static int init_common_variables() return 1; item_init(); #ifndef EMBEDDED_LIBRARY - my_regex_init(&my_charset_latin1, check_enough_stack_size); + //my_regex_init(&my_charset_latin1, check_enough_stack_size); my_string_stack_guard= check_enough_stack_size; #else - my_regex_init(&my_charset_latin1, NULL); + //my_regex_init(&my_charset_latin1, NULL); #endif /* Process a comma-separated character set list and choose Remove it please (I think it was just forgotten).

2 1

[Maria-developers] Empty directory in mariadb's datadir is not handled correctly
by Honza Horak 30 Sep '13

30 Sep '13

Hi guys, quite tricky problem came up after a directory has been created by something else than mysqld in the datadir. Suppose we use datadir=/var/lib/mysql. When we create any file in that directory by hand (or some program does it, which is not just imagination, it really happens), e.g. /var/lib/mysql/.local, MariaDB (Oracle's MySQL will behave the same probably) server thinks this directory is a database, which is wrong. Then we get some weird messages and outputs: # mysqlshow +--------------------+ | Databases | +--------------------+ | information_schema | | #mysql50#.local | | mysql | | performance_schema | | test | +--------------------+ # mysqlcheck -A mysqlcheck: Got error: 1102: Incorrect database name '#mysql50#.local' when selecting the database I believe this is not how the things should work. I'm not sure, but I'd expect there is still some information about existing databases in the information_schema table. So IMHO the server/tool should always fetch such info in addition to pure testing if the database directory exists. Or is such testing simply impossible? Regards, Honza

2 1

Re: [Maria-developers] [All] MariaDB's best (download) month ever
by Kristian Nielsen 30 Sep '13

30 Sep '13

Colin Charles <byte(a)mariadb.com> writes: > It's 17.09 today, and we've hit 50,160 total downloads (this number is > considered company-internal, we don't share this number externally). "company-internal"? What have you been smoking? Colin, you are supposed to be the community guy for crying out lout :-( This is *completely*, utterly unacceptable. MariaDB is a community project. We are about openness. We are about working together, all of us, to create a better world. We are *not* some individuals or companys playground. So no, the community's usage numbers of its own project is not "internal", thank you very much. So please cut out the crap already. - Kristian.

6 11

[Maria-developers] Prep for MariaDB Galera 5.5.33a
by Daniel Bartholomew 28 Sep '13

28 Sep '13

All, I've begun the preparations for the MariaDB Galera 5.5.33a release. Draft release notes and changelog are here: - https://mariadb.com/kb/en/mariadb-galera-5533a-release-notes/ - https://mariadb.com/kb/en/mariadb-galera-5533a-changelog/ Please make necessary changes/additions/etc... to the above pages (or let me know). Thanks. -- Daniel Bartholomew, Sr Tech Writer / Sys Admin MariaDB | t: +1 919-809-6069 | Skype: daniel-bartholomew

1 0

[Maria-developers] MySQL bug affects mariaDB
by Peter Laursen 27 Sep '13

27 Sep '13

I think the 'mysql' client with MariaDB has a similar problem as it has with MySQL - refer? http://bugs.mysql.com/bug.php?id=70443 MariaDB installer installs my.ini to /datadir - what with MariaDB is a subfolder to /basedir. The server will look for it correctly as the registry key defining the service has a --default-file specification. But the command line clients don't read the registry key. @Wlad? -- Peter -- Webyog

3 7

[Maria-developers] EXPLAIN INSERT in MySQL - not useful after all?
by Sergei Petrunia 27 Sep '13

27 Sep '13

4 4

[Maria-developers] [GSoC] MariaDB Contributor Agreement for Kerberos Authentication Plugin
by QIU Shuang 27 Sep '13

27 Sep '13

Hi community, I declare that all the work done so far (and future) on the project Kerberize MariaDB is licenced by the new BSD Lisence. Please contact me if any problems with the license. Thank you all and best regards! Sincerely, Shuang

1 0

[Maria-developers] [GSoC] Request for code review
by QIU Shuang 27 Sep '13

27 Sep '13

Hi Sergei, This is Shuang, a participant for GSoC 2013 Kerberize MariaDB project. I want to ask the community to review the code before requesting for pull and send the MCA. How could I send the codes? In a diff file? or from other approaches? Thank you very much and best regards! Sincerely, Shuang

2 2

[Maria-developers] Review of base64.diff and german2.diff
by Michael Widenius 27 Sep '13

27 Sep '13

Hi! Here is the review for the code that we should put into 10.0 First the base64: > === modified file 'mysql-test/t/func_str.test' > --- mysql-test/t/func_str.test 2013-05-07 11:05:09 +0000 > +++ mysql-test/t/func_str.test 2013-08-28 13:14:24 +0000 > @@ -1555,3 +1555,118 @@ drop table t1,t2; > --echo # End of 5.5 tests > --echo # > > + > +--echo # > +--echo # Start of 5.6 tests > +--echo # Shouldn't this be start of 10.0 tests ? (I know that this code is from MySQL 5.6, but still for us this is 10.0...) > === modified file 'mysys/base64.c' > --- mysys/base64.c 2011-06-30 15:46:53 +0000 > +++ mysys/base64.c 2013-03-09 06:22:59 +0000 > @@ -1,5 +1,4 @@ > -/* Copyright (c) 2003-2008 MySQL AB, 2009 Sun Microsystems, Inc. > - Use is subject to license terms. > +/* Copyright (c) 2003, 2010, Oracle and/or its affiliates. All rights reserved. Removed 'all rights reserved'. You can't have that for GPL code. If there is any new code from us, please add a copyright message for the MariaDB foundation too! > > This program is free software; you can redistribute it and/or modify > it under the terms of the GNU General Public License as published by > @@ -25,6 +24,28 @@ static char base64_table[] = "ABCDEFGHIJ > "abcdefghijklmnopqrstuvwxyz" > "0123456789+/"; > > +/** > + * Maximum length base64_needed_encoded_length() > + * can handle without overflow. > + */ > +int > +base64_encode_max_arg_length() > +{ > +#if (SIZEOF_INT == 8) > + /* > + 6827690988321067803 -> 9223372036854775805 > + 6827690988321067804 -> -9223372036854775807 > + */ Please describe from where the following number comes from (so that anyone can recalculate and check the number if needed): > + return 0x5EC0D4C77B03531BLL > +#else > + /* > + 1589695686 -> 2147483646 > + 1589695687 -> -2147483645 > + */ > + return 0x5EC0D4C6; > +#endif > +} > + > > int > base64_needed_encoded_length(int length_of_data) > @@ -39,6 +60,21 @@ base64_needed_encoded_length(int length_ > } > > > +/** > + * Maximum length base64_needed_decoded_length() > + * can handle without overflow. > + */ > +int > +base64_decode_max_arg_length() > +{ The following also need a comment. > +#if (SIZEOF_INT == 8) > + return 0x2AAAAAAAAAAAAAAALL; > +#else > + return 0x2AAAAAAA; > +#endif > +} > + > + <cut> > +/** > + * Get the next character from a base64 encoded stream. > + * Skip spaces, then scan the next base64 character or a pad character > + * and collect bits into decoder->c. > + * > + * @param decoder Pointer to MY_BASE64_DECODER > + * @return > + * FALSE on success (a valid base64 encoding character found) > + * TRUE on error (unexpected character or unexpected end-of-input found) > + */ Add an empty line here. It would also be good to have a good description of the base64 format in this file so that one can understand what the functions are supposed to do (or at least a link to the specification). Some thing I figured out by asking bar on IRC: - We never generate spaces - When decoding, we allow spaces anywhere. - The end of an 64 base decoded string is always padded with '=' so that the final length is dividable with 4. > +static inline my_bool > +my_base64_decoder_getch(MY_BASE64_DECODER *decoder) Remove inline from the above; It's likely to make the code slower, not faster as this is a big function with lot of if's. > { > - char b[3]; > - size_t i= 0; > - char *dst_base= (char *)dst; > - char const *src= src_base; > - char *d= dst_base; > - size_t j; > + if (my_base64_decoder_skip_spaces(decoder)) > + return TRUE; /* End-of-input */ > > - while (i < len) > + if (!my_base64_add(decoder)) /* Valid base64 character found */ > { > - unsigned c= 0; > - size_t mark= 0; > + if (decoder->mark) > + { > + /* If we have scanned '=' already, then only '=' is valid */ > + DBUG_ASSERT(decoder->state == 3); > + decoder->error= 1; > + decoder->src--; > + return TRUE; /* expected '=', but encoding character found */ > + } > + decoder->state++; > + return FALSE; > + } If you want to have the funtion inline, then move the following to another not inline function; We don't need to generate code for this for every function call: > + > + /* Process error */ > + switch (decoder->state) > + { > + case 0: > + case 1: > + decoder->src--; > + return TRUE; /* base64 character expected */ > + break; > + > + case 2: > + case 3: > + if (decoder->src[-1] == '=') > + { > + decoder->error= 0; /* Not an error - it's a pad character */ > + decoder->mark++; > + } > + else > + { > + decoder->src--; > + return TRUE; /* base64 character or '=' expected */ > + } > + break; > + default: > + DBUG_ASSERT(0); > + return TRUE; /* Wrong state, should not happen */ > + } <cut> Add to the comment for base64_decode that the 'dst' buffer has to be at least 2 character longer than what is needed to decode src_base. > +int > +base64_decode(const char *src_base, size_t len, > + void *dst, const char **end_ptr, int flags) > +{ > + if (my_base64_decoder_getch(&decoder) || > + my_base64_decoder_getch(&decoder) || > + my_base64_decoder_getch(&decoder) || > + my_base64_decoder_getch(&decoder)) > + break; Generating the error handling with a switch for every of the above is wasteful. See previous comment. > + /* Return error if there are more non-space characters */ > + decoder.state= 0; > + if (!my_base64_decoder_skip_spaces(&decoder)) > + decoder.error= 1; > + > if (end_ptr != NULL) Note that base64_needed_decoded_length() used ceil() when it's not needed. <cut> > === modified file 'sql/item_strfunc.cc' > @@ -451,6 +452,101 @@ void Item_func_aes_decrypt::fix_length_a > set_persist_maybe_null(1); > } > > + > +void Item_func_to_base64::fix_length_and_dec() > +{ > + maybe_null= args[0]->maybe_null; > + collation.set(default_charset(), DERIVATION_COERCIBLE, MY_REPERTOIRE_ASCII); Maybe better to cast both arguments to (ulonglong) as the rest of the code is depending on this. Also, why is base64_encode_max_arg_length() int instead of uint or even better size_t? > + if (args[0]->max_length > (uint) base64_encode_max_arg_length()) > + { > + maybe_null= 1; > + fix_char_length_ulonglong((ulonglong) base64_encode_max_arg_length()); > + } > + else > + { > + int length= base64_needed_encoded_length((int) args[0]->max_length); > + DBUG_ASSERT(length > 0); Don't think assert is needed as the function gurantees it already. > + fix_char_length_ulonglong((ulonglong) length - 1); > + } > +} <cut> > +String *Item_func_from_base64::val_str(String *str) > +{ > + String *res= args[0]->val_str_ascii(str); > + bool too_long= false; > + int length; > + const char *end_ptr; > + > + if (!res || > + res->length() > (uint) base64_decode_max_arg_length() || > + (too_long= > + ((uint) (length= base64_needed_decoded_length((int) res->length())) > > + current_thd->variables.max_allowed_packet)) || > + tmp_value.alloc((uint) length) || > + (length= base64_decode(res->ptr(), (int) res->length(), > + (char *) tmp_value.ptr(), &end_ptr, 0)) < 0 || > + end_ptr < res->ptr() + res->length()) > + { Shouldn't we get a readable error or warning if the string contained wrong characters? Now it looks like we will just return null. Something like 'Malformed base64 string. Error at position %d" would be nice. > + null_value= 1; // NULL input, too long input, OOM, or badly formed input > + if (too_long) > + { > + push_warning_printf(current_thd, Sql_condition::WARN_LEVEL_WARN, > + ER_WARN_ALLOWED_PACKET_OVERFLOWED, > + ER(ER_WARN_ALLOWED_PACKET_OVERFLOWED), func_name(), > + current_thd->variables.max_allowed_packet); > + } > + return 0; > + } > + tmp_value.length((uint) length); > + null_value= 0; > + return &tmp_value; > +} <cut> Review of german2.diff === modified file 'include/my_global.h' --- include/my_global.h 2013-07-21 14:39:19 +0000 +++ include/my_global.h 2013-08-29 08:27:09 +0000 @@ -1218,4 +1218,11 @@ static inline double rint(double x) #define HAVE_EXTERNAL_CLIENT #endif /* EMBEDDED_LIBRARY */ + +enum loglevel { + ERROR_LEVEL= 0, + WARNING_LEVEL= 1, + INFORMATION_LEVEL= 2 +}; + #endif /* my_global_h */ Not sure why the above, which is a my_sys construct, should be in my_global.h. Please add at least a comment about this. === modified file 'mysys/charset.c' --- mysys/charset.c 2012-08-14 14:23:34 +0000 +++ mysys/charset.c 2013-08-29 12:38:44 +0000 @@ -66,6 +66,9 @@ static my_bool init_state_maps(struct ch if (!(cs->ident_map= ident_map= (uchar*) my_once_alloc(256, MYF(MY_WME)))) return 1; + state_map= (uchar *) cs->state_map; + ident_map= (uchar *) cs->ident_map; Why is the above needed? They are already set one line above. (In addition cs->state_map and cs->ident_map are already uchar *) --- strings/ctype-simple.c 2013-07-21 14:39:19 +0000 +++ strings/ctype-simple.c 2013-08-29 09:05:07 +0000 @@ -1163,12 +1163,12 @@ static int pcmp(const void * f, const vo return res; } -static my_bool create_fromuni(struct charset_info_st *cs, - void *(*alloc)(size_t)) +static my_bool +create_fromuni(struct charset_info_st *cs, + MY_CHARSET_LOADER *loader) { uni_idx idx[PLANE_NUM]; int i,n; - struct my_uni_idx_st *tab_from_uni; /* Check that Unicode map is loaded. @@ -1209,18 +1209,18 @@ static my_bool create_fromuni(struct cha for (i=0; i < PLANE_NUM; i++) { int ch,numchars; - uchar *tab; /* Skip empty plane */ if (!idx[i].nchars) break; numchars=idx[i].uidx.to-idx[i].uidx.from+1; - if (!(idx[i].uidx.tab= tab= (uchar*) - alloc(numchars * sizeof(*idx[i].uidx.tab)))) + if (!(idx[i].uidx.tab= (uchar *) + (loader->once_alloc) (numchars * + sizeof(*idx[i].uidx.tab)))) return TRUE; - bzero(tab,numchars*sizeof(*tab)); + bzero((char*) idx[i].uidx.tab, numchars * sizeof(*idx[i].uidx.tab)); for (ch=1; ch < PLANE_SIZE; ch++) { @@ -1228,32 +1228,32 @@ static my_bool create_fromuni(struct cha if (wc >= idx[i].uidx.from && wc <= idx[i].uidx.to && wc) { int ofs= wc - idx[i].uidx.from; - tab[ofs]= ch; + ((char *) idx[i].uidx.tab)[ofs]= ch; } } } Why remove tab to be a shortcut for idx[i].uidx.from ? Shouldn't it be faster and shorter to use tab than using idx[i].uidx.from? (At least in this place) -static const uint16 page0FCdata[]= { /* FC00 (3 weights per char) */ +static uint16 page0FCdata[]= { /* FC00 (3 weights per char) */ 0x134F,0x135E,0x0000, 0x134F,0x1364,0x0000, 0x134F,0x13B0,0x0000, 0x134F,0x13C7,0x0000, 0x134F,0x13C8,0x0000, 0x1352,0x135E,0x0000, 0x1352,0x1364,0x0000, 0x1352,0x1365,0x0000, 0x1352,0x13B0,0x0000, @@ -6006,7 +6005,7 @@ static const uint16 page0FCdata[]= { /* 0x1381,0x13C8,0x0000, 0x1382,0x13C7,0x0000, 0x1382,0x13C8,0x0000, 0x1364,0x13C7,0x0000 }; If we are not going to change the above, better to keep these as const! They will then be in the const segment and you will have protection from anyone trying to change these! <cut> +my_uca_add_contraction(MY_CONTRACTIONS *list, my_wc_t *wc, size_t len, + my_bool with_context) { - MY_CONTRACTIONS *list= (MY_CONTRACTIONS*) cs->contractions; MY_CONTRACTION *next= &list->item[list->nitems]; - DBUG_ASSERT(len == 2); /* We currently support only contraction2 */ - next->ch[0]= wc[0]; - next->ch[1]= wc[1]; + size_t i; + /* + Contraction is always at least 2 characters. + Contraction is never longer than MY_UCA_MAX_CONTRACTION, + which is guaranteed by using my_coll_rule_expand() with proper limit. + */ + DBUG_ASSERT(len > 1 && len <= MY_UCA_MAX_CONTRACTION); + for (i= 0; i < len; i++) + { + /* + We don't support contractions with U+0000. + my_coll_rule_expand() guarantees there're no U+0000 in a contraction. + */ + DBUG_ASSERT(wc[i] != 0); + next->ch[i]= wc[i]; + } + if (i < MY_UCA_MAX_CONTRACTION) + next->ch[i]= 0; /* Add end-of-line marker */ Wouldn't it make life easier to always have the marker there? (Ie, always have place for the marker). At least you can remove one if from the code. If there is more than one if to remove, then it's a simple optimzation do to... <cut> +static int +my_coll_parser_too_long_error(MY_COLL_RULE_PARSER *p, const char *name) +{ + my_snprintf(p->errstr, sizeof(p->errstr), "%s is too long", name); + return 0; +} You should limit '%s' to 64 characters so that one gets the name in the output even if it's over sizeof()... %-.64s .... <cut> Regards, Monty

7 40

[Maria-developers] Review request : MDEV-4476 (mytop tracker)
by Jean Weisbuch 26 Sep '13

26 Sep '13

Hi, I have been doing modifications to the MariaDB mytop version (which is shipped on the /scripts directory), especially for bugfix, small display optimizations, use when available the I_S.PROCESSLIST instead of SHOW FULL PROCESSLIST and recently to show 1/10th of seconds on queries running time on servers having the TIME_MS column on the PROCESSLIST. A more detailed list : - Control chars on queries are not displayed anymore to avoid possible command executions/mytop misbehaving : when a query having control chars (ASCII 000 to 037) in it is shown in mytop, it can act like a key was pressed and end up doing strange/unwanted things. - Display cannot overlap the terminal size anymore when fullqueries is activated. - If not all the queries are shown on screen, a "-- Truncated query list --" is shown on the bottom of the query list. - It is now using the informations from INFORMATION_SCHEMA.PROCESSLIST when available instead of the SHOW FULL PROCESSLIST output for a better modularity is a similar maneer as Innotop does. - The query time is now shown with the 1/10th of second precision on servers having the TIME_MS patch (MariaDB and Percona) for queries running for less than 10000 seconds. - Command summary view fixed to retreive/show only the Com_* values and not the Com* (was showing a "Compression" stats row for this boolean variable). - Better have_query_cache test in case other values than ON/OFF/DEMAND would be introduced. Another detail that could be good to fix for MariaDB 10 GA is that the script is shipped as mytop.sh on MariaDB while its a Perl script (it is for example shipped as /usr/bin/mytop on Debian/Ubuntu official packages of mytop). The JIRA task : https://mariadb.atlassian.net/browse/MDEV-4476 The GIT repository where i put the latest revisions (i only have been using it lately but i commented all the changes and bugfixes along the time on the JIRA task and could provide older revisions/diff if required) : https://github.com/jb-boin/mariadb/blob/5.5.30/scripts/mytop.sh Feel free to contact me for more informations. Thanks in advance, Jean Weisbuch

1 0

[Maria-developers] Review of MDEV-4506, parallel replication, part 1
by Michael Widenius 26 Sep '13

26 Sep '13

Hi! Here is the first part of the review. I have 1/2 of one file left (rpl_parallel.cc) to review but I wanted you to have a chance to read what I have done so far. I plan to finish the review tomorrow and start working on the code during the weekend. >>>>> "Kristian" == Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: Kristian> Hi Monty, Kristian> Here is a copy of the status mail I sent earlier. The status is mostly still Kristian> current. Kristian> The code is in this tree: Kristian> lp:~maria-captains/maria/10.0-knielsen Kristian> In that tree, you can extract the parallel replication code as a single patch Kristian> with a command like this: Kristian> bzr diff -rrevid:knielsen@knielsen-hq.org-20130608103621-g91eaielv4n22aha Kristian> In the forwarded mail, I describe getting a 4.5 times speedup on a quad-core Kristian> laptop, so that's quite promising. === modified file 'mysql-test/r/mysqld--help.result' --- mysql-test/r/mysqld--help.result 2013-05-28 11:28:31 +0000 +++ mysql-test/r/mysqld--help.result 2013-08-29 08:18:22 +0000 <cut> @@ -783,6 +794,11 @@ --slave-net-timeout=# Number of seconds to wait for more data from any master/slave connection before aborting the read + --slave-parallel-threads=# + If non-zero, number of threads to spawn to apply in + parallel events on the slave that were group-committed on + the master or were logged with GTID in different + replication domains. What happens if --slave-parallel-threads is 0 ? Does this mean that there will be no parallel threads? Is this the same as 1 ? If same as 1, why not have the option to take values starting from 1 ? IRC: K> yes, if --slave-parallel-threads is 0, then the feature is disabled. The old code is used, and the SQL thread applies the events itself K> if --slave-parallel-threads is > 0, then a pool of threads are spawned. The SQL threads do not apply events themselves, they just put it in queues for the parallel threads to apply M> do we ever need to do this with only one thread? K> (we might want to give an error if user tries to do this, as it does not make much sense) M> So why not have the value from 1 to N M> where 1 is old code and > 1 is using new code ? K> That might make sense === modified file 'mysql-test/suite/perfschema/r/dml_setup_instruments.result' --- mysql-test/suite/perfschema/r/dml_setup_instruments.result 2013-03-26 09:35:34 +0000 +++ mysql-test/suite/perfschema/r/dml_setup_instruments.result 2013-08-29 08:18:22 +0000 @@ -38,14 +38,14 @@ order by name limit 10; NAME ENABLED TIMED wait/synch/cond/sql/COND_flush_thread_cache YES YES wait/synch/cond/sql/COND_manager YES YES +wait/synch/cond/sql/COND_parallel_entry YES YES +wait/synch/cond/sql/COND_prepare_ordered YES YES wait/synch/cond/sql/COND_queue_state YES YES wait/synch/cond/sql/COND_rpl_status YES YES +wait/synch/cond/sql/COND_rpl_thread YES YES +wait/synch/cond/sql/COND_rpl_thread_pool YES YES wait/synch/cond/sql/COND_server_started YES YES wait/synch/cond/sql/COND_thread_cache YES YES -wait/synch/cond/sql/COND_thread_count YES YES -wait/synch/cond/sql/Delayed_insert::cond YES YES -wait/synch/cond/sql/Delayed_insert::cond_client YES YES -wait/synch/cond/sql/Event_scheduler::COND_state YES YES Any idea why the above was deleted? Don't see anything in your patch that could affect that. <cut> === added file 'mysql-test/suite/rpl/t/rpl_parallel.test' --- mysql-test/suite/rpl/t/rpl_parallel.test 1970-01-01 00:00:00 +0000 +++ mysql-test/suite/rpl/t/rpl_parallel.test 2013-08-29 08:18:22 +0000 I didn't find any test results for rpl_parallel.test or rpl_paralell2 in your tree. Did you forget to commit these? I was also missing some comment in these files what they was testing. === modified file 'sql/log.cc' --- sql/log.cc 2013-06-06 15:51:28 +0000 +++ sql/log.cc 2013-08-29 08:18:22 +0000 @@ -6541,44 +6543,201 @@ MYSQL_BIN_LOG::write_transaction_to_binl } Could you please document in detail the arguments and return value for the following function. - I spent 3 hours to try to understand this function in detail and having a full description of the arguments would have helped a bit. bool -MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) +MYSQL_BIN_LOG::queue_for_group_commit(group_commit_entry *entry, + wait_for_commit *wfc) { + group_commit_entry *orig_queue; + wait_for_commit *list, *cur, *last; + /* To facilitate group commit for the binlog, we first queue up ourselves in the group commit queue. Then the first thread to enter the queue waits for the LOCK_log mutex, and commits for everyone in the queue once it gets the lock. Any other threads in the queue just wait for the first one to finish the commit and wake them up. Do you still use LOCK_log for this? It's a bit confusing to mention LOCK_log here when you take the LOCK_prepare_ordered in the function. If LOCK_log is required, shouldn't there be an assert for it in this function? + + To support in-order parallel replication with group commit, after we add + some transaction to the queue, we check if there were other transactions + already prepared to commit but just waiting for the first one to commit. + If so, we add those to the queue as well, transitively for all waiters. */ entry->thd->clear_wakeup_ready(); mysql_mutex_lock(&LOCK_prepare_ordered); <cut> + orig_queue= group_commit_queue; + + /* + Iteratively process everything added to the queue, looking for waiters, + and their waiters, and so on. If a waiter is ready to commit, we + immediately add it to the queue; if not we just wake it up. + + This would be natural to do with recursion, but we want to avoid + potentially unbounded recursion blowing the C stack, so we use the list + approach instead. Add here: cur->next_subsequent_commit is at this point an empty list. 'last' will always point to the last element of the list. We use this list to simulate recursion. I did find it confusing that here you threat next_subsequent_commit as a way to avoid recursion, but you use this list for other things in register_wait_for_prior_commit(). Suggestion: - If you want to reuse memory for the list, please use an union over next_subsequent_commit and have two names for it, so that you don't in the code use the same name for two different list. + */ + list= wfc; + cur= list; + last= list; + for (;;) + { + /* Add the entry to the group commit queue. */ + entry->next= group_commit_queue; + group_commit_queue= entry; + + if (entry->cache_mngr->using_xa) + { + DEBUG_SYNC(entry->thd, "commit_before_prepare_ordered"); + run_prepare_ordered(entry->thd, entry->all); + DEBUG_SYNC(entry->thd, "commit_after_prepare_ordered"); + } + + if (!cur) + break; // Can happen if initial entry has no wait_for_commit + Add a comment what the following if is testing (As there is several lists involved, the code is not that easy to understand) + if (cur->subsequent_commits_list) + { + bool have_lock; + wait_for_commit *waiter; + + mysql_mutex_lock(&cur->LOCK_wait_commit); + have_lock= true; + waiter= cur->subsequent_commits_list; + /* Check again, now safely under lock. */ + if (waiter) + { + /* Grab the list of waiters and process it. */ + cur->subsequent_commits_list= NULL; + do + { + wait_for_commit *next= waiter->next_subsequent_commit; + group_commit_entry *entry2= + (group_commit_entry *)waiter->opaque_pointer; + if (entry2) + { + /* + This is another transaction ready to be written to the binary + log. We can put it into the queue directly, without needing a + separate context switch to the other thread. We just set a flag + so that the other thread will know when it wakes up that it was + already processed. + + So put it at the end of the list to be processed in a subsequent + iteration of the outer loop. + */ + entry2->queued_by_other= true; + last->next_subsequent_commit= waiter; + last= waiter; + /* + As a small optimisation, we do not actually need to set + waiter->next_subsequent_commit to NULL, as we can use the + pointer `last' to check for end-of-list. + */ + } + else + { + /* + Wake up the waiting transaction. + + For this, we need to set the "wakeup running" flag and release + the waitee lock to avoid a deadlock, see comments on + THD::wakeup_subsequent_commits2() for details. + */ + if (have_lock) + { + cur->wakeup_subsequent_commits_running= true; + mysql_mutex_unlock(&cur->LOCK_wait_commit); + have_lock= false; Move have_lock= false to just after the 'if (have_lock)' - Easier to see that you are really reseting the variable. - Faster as there is no need to reload the variable from stack after mysql_mutex_unlock() function. - I try to nowadays in my code to always reset flag variables just after testing them to achive the above. + } + waiter->wakeup(); + } + waiter= next; + } while (waiter); + } You can optimze away one row, one 'if level' and one 'if' away by doing this: Move: 'cur->subsequent_commits_list= NULL;' before the if. Change 'if (waiter)' to 'while (waiter)' Remove 'do' and the corresponding 'while (waiter);' + if (have_lock) + mysql_mutex_unlock(&cur->LOCK_wait_commit); + } + if (cur == last) + break; Add a comment here of what is in next_subsequent_commit list. + cur= cur->next_subsequent_commit; + entry= (group_commit_entry *)cur->opaque_pointer; + DBUG_ASSERT(entry != NULL); } + + /* Now we need to clear the wakeup_subsequent_commits_running flags. */ + if (list) + { + for (;;) + { + if (list->wakeup_subsequent_commits_running) + { + mysql_mutex_lock(&list->LOCK_wait_commit); + list->wakeup_subsequent_commits_running= false; + mysql_mutex_unlock(&list->LOCK_wait_commit); Why do we need the mutex above? The only place where we test this variable, as far as I can find, is in wait_for_commit::register_wait_for_prior_commit() This thread doesn't know if the other thread is just before mysql_mutex_lock(&waitee->LOCK_wait_commit), inside the lock or after the lock. So when we reset the variable should not matter. But somehow this sounds strange. What happens if the above variable is reset just before mysql_mutex_lock(&waitee->LOCK_wait_commit) in wait_for_commit::register_wait_for_prior_commit() ? Isn't the waitee added to the list when it shouldn't? + } + if (list == last) + break; + list= list->next_subsequent_commit; + } + } + + if (opt_binlog_commit_wait_count > 0) + mysql_cond_signal(&COND_prepare_ordered); mysql_mutex_unlock(&LOCK_prepare_ordered); DEBUG_SYNC(entry->thd, "commit_after_release_LOCK_prepare_ordered"); + return orig_queue == NULL; +} + +bool +MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) +{ + wait_for_commit *wfc; + bool is_leader; + + wfc= entry->thd->wait_for_commit_ptr; + entry->queued_by_other= false; + if (wfc && wfc->waiting_for_commit) + { Add comment, something like: /* We have to wait for the threads in wfc->waiting_for_commit */ + mysql_mutex_lock(&wfc->LOCK_wait_commit); + /* Do an extra check here, this time safely under lock. */ + if (wfc->waiting_for_commit) + { + wfc->opaque_pointer= entry; + do + { + mysql_cond_wait(&wfc->COND_wait_commit, &wfc->LOCK_wait_commit); + } while (wfc->waiting_for_commit); + wfc->opaque_pointer= NULL; + } + mysql_mutex_unlock(&wfc->LOCK_wait_commit); + } <cut> @@ -6597,7 +6756,10 @@ MYSQL_BIN_LOG::write_transaction_to_binl if (next) { - next->thd->signal_wakeup_ready(); + if (next->queued_by_other) + next->thd->wait_for_commit_ptr->wakeup(); What does the above code mean ? (add a comment). Why do we wakeup the thread we where waiting for instead of ourselves? + else + next->thd->signal_wakeup_ready(); } else { <cut> @@ -6968,6 +7140,48 @@ MYSQL_BIN_LOG::write_transaction_or_stmt return 0; } + +void +MYSQL_BIN_LOG::wait_for_sufficient_commits() +{ + size_t count; + group_commit_entry *e; + group_commit_entry *last_head; + struct timespec wait_until; + + mysql_mutex_assert_owner(&LOCK_log); + mysql_mutex_assert_owner(&LOCK_prepare_ordered); + + count= 0; + for (e= last_head= group_commit_queue; e; e= e->next) + ++count; + if (count >= opt_binlog_commit_wait_count) + return; Assuming that the queue is bigger may be better: for (e= last_head= group_commit_queue, count=0; e ; e= e->next) if (++count >= opt_binlog_commit_wait_count) return; + mysql_mutex_unlock(&LOCK_log); Here you are going to take LOCK_log and LOCK_prepare_ordered in the wrong order. safe_mutex() should have noticed this. In other code you are first taking LOCK_log and then LOCK_prepare_ordered. If someone else is calling trx_group_commit_leader() while you are in this code you would get a mutex deadlock. + set_timespec_nsec(wait_until, (ulonglong)1000*opt_binlog_commit_wait_usec); + + for (;;) + { + int err; + group_commit_entry *head; + + err= mysql_cond_timedwait(&COND_prepare_ordered, &LOCK_prepare_ordered, + &wait_until); + if (err == ETIMEDOUT) + break; + head= group_commit_queue; + for (e= head; e && e != last_head; e= e->next) + ++count; + if (count >= opt_binlog_commit_wait_count) + break; + last_head= head; + } + + mysql_mutex_lock(&LOCK_log); +} <cut> === modified file 'sql/log_event.cc' --- sql/log_event.cc 2013-06-06 15:51:28 +0000 +++ sql/log_event.cc 2013-08-29 08:18:22 +0000 <cut> @@ -6101,6 +6110,18 @@ Gtid_log_event::Gtid_log_event(const cha domain_id= uint4korr(buf); buf+= 4; flags2= *buf; + if (flags2 & FL_GROUP_COMMIT_ID) + { + if (event_len < (uint)header_size + GTID_HEADER_LEN + 2) + { + seq_no= 0; // So is_valid() returns false + return; + } + ++buf; + commit_id= uint8korr(buf); + } + else + commit_id= 0; } Setting commit_id=0 first instead in the if will generate smaller and faster code (setting a variable is faster than an a jmp). === modified file 'sql/log_event.h' --- sql/log_event.h 2013-06-06 15:51:28 +0000 +++ sql/log_event.h 2013-08-29 08:18:22 +0000 @@ -1317,9 +1317,9 @@ class Log_event @see do_apply_event */ - int apply_event(Relay_log_info const *rli) + int apply_event(struct rpl_group_info *rgi) What is the simple rule to know when to use Relay_log_info and when to use rpl_group_info ? If I understand things correct, relay_log_info should only be used for direct manipulation of data in the relay log. By the way, you should probably change 'struct rpl_group_info' to just rpl_group_info; Now you are mixing both versions in the code. === modified file 'sql/log_event_old.cc' --- sql/log_event_old.cc 2013-02-19 10:45:29 +0000 +++ sql/log_event_old.cc 2013-08-29 08:18:22 +0000 @@ -67,7 +68,7 @@ Old_rows_log_event::do_apply_event(Old_r do_apply_event(). We still check here to prevent future coding errors. */ - DBUG_ASSERT(rli->sql_thd == ev_thd); + DBUG_ASSERT(rgi->thd == ev_thd); Should we change all test for rli->sql_thd to rgi->thd ? You have at least this test in log_event.c and slave.cc. I assume that rli should not have a sql_thd member anymore ? <cut> === added file 'sql/rpl_parallel.cc' --- sql/rpl_parallel.cc 1970-01-01 00:00:00 +0000 +++ sql/rpl_parallel.cc 2013-08-29 08:18:22 +0000 @@ -0,0 +1,699 @@ +#include "my_global.h" +#include "rpl_parallel.h" +#include "slave.h" +#include "rpl_mi.h" + + +/* + Code for optional parallel execution of replicated events on the slave. + + ToDo list: + + - Review every field in Relay_log_info, and all code that accesses it. + Split out the necessary parts into rpl_group_info, to avoid conflicts + between parallel execution of events. (Such as deferred events ...) + + - Error handling. If we fail in one of multiple parallel executions, we + need to make a best effort to complete prior transactions and roll back + following transactions, so slave binlog position will be correct. + And all the retry logic for temporary errors like deadlock. + + - Stopping the slave needs to handle stopping all parallel executions. And + the logic in sql_slave_killed() that waits for current event group to + complete needs to be extended appropriately... + + - Audit the use of Relay_log_info::data_lock. Make sure it is held + correctly in all needed places also when using parallel replication. + + - We need some user-configurable limit on how far ahead the SQL thread will + fetch and queue events for parallel execution (otherwise if slave gets + behind we will fill up memory with pending malloc()'ed events). + + - Fix update of relay-log.info and master.info. In non-GTID replication, + they must be serialised to preserve correctness. In GTID replication, we + should not update them at all except at slave thread stop. + + - All the waits (eg. in struct wait_for_commit and in + rpl_parallel_thread_pool::get_thread()) need to be killable. And on kill, + everything needs to be correctly rolled back and stopped in all threads, + to ensure a consistent slave replication state. I can add kill checking in all functions that waits for a mutex. Will do it with adding enter_cond()/exit_cond() in all functions waiting for mysql_cond_wait(). + + - Handle the case of a partial event group. This occurs when the master + crashes in the middle of writing the event group to the binlog. The + slave rolls back the transaction; parallel execution needs to be able + to deal with this wrt. commit_orderer and such. + + - Relay_log_info::is_in_group(). This needs to be handled correctly in all + callers. I think it needs to be split into two, one version in + Relay_log_info to be used from next_event() in slave.cc, one to be used in + per-transaction stuff. + + - We should fail if we connect to the master with opt_slave_parallel_threads + greater than zero and master does not support GTID. Just to avoid a bunch + of potential problems, we won't be able to do any parallel replication + in this case anyway. If the master doesn't support GTID, I think we should instead of stopping, just issue a warning in the log and continue with normal replication. Otherwise we would have problems when using multi-source replication if a single slave doesn't support GTID. +*/ + +struct rpl_parallel_thread_pool global_rpl_thread_pool; + + +static void +rpt_handle_event(rpl_parallel_thread::queued_event *qev, + struct rpl_parallel_thread *rpt) +{ + int err; + struct rpl_group_info *rgi= qev->rgi; + Relay_log_info *rli= rgi->rli; + THD *thd= rgi->thd; + + thd->rli_slave= rli; + thd->rpl_filter = rli->mi->rpl_filter; + /* ToDo: Get rid of rli->group_info, it is not thread safe. */ + rli->group_info= rgi; + + /* ToDo: Access to thd, and what about rli, split out a parallel part? */ + mysql_mutex_lock(&rli->data_lock); + err= apply_event_and_update_pos(qev->ev, thd, rgi, rpt); We should also get rid of the thd argument and change the function to be a bool so that we can return an error. + /* ToDo: error handling. */ +} + + +pthread_handler_t +handle_rpl_parallel_thread(void *arg) +{ <cut> + mysql_mutex_lock(&rpt->LOCK_rpl_thread); + rpt->thd= thd; + + while (rpt->delay_start) + mysql_cond_wait(&rpt->COND_rpl_thread, &rpt->LOCK_rpl_thread); A much easier way to achive the wait is to have the caller lock mysql_mutex_lock(&rpt->LOCK_rpl_thread) and then just release it when it's time for this thread to continue. - No need for signaling or checking if the thread got the signal - No need to have a cond wait here - No need to have a delay_start flag + rpt->running= true; + + while (!rpt->stop && !thd->killed) + { + rpl_parallel_thread *list; + + old_msg= thd->proc_info; + thd->enter_cond(&rpt->COND_rpl_thread, &rpt->LOCK_rpl_thread, + "Waiting for work from SQL thread"); + while (!rpt->stop && !thd->killed && !(events= rpt->event_queue)) + mysql_cond_wait(&rpt->COND_rpl_thread, &rpt->LOCK_rpl_thread); Add comment: /* Mark that this thread is now executing */ + rpt->free= false; + rpt->event_queue= rpt->last_in_queue= NULL; + thd->exit_cond(old_msg); + Note that events may be undefined here if thd->killed was set above (for the first loop) It would be good to check for thd->killed or rpt->stop here and then abort the loop. Another question: do we really need rpt->stop ? Isn't it enough with using the thd->killed for this ? Otherwise we would need to check for both rpt->top and thd->killed in every loop that may be aborted. + more_events: + while (events) + { + struct rpl_parallel_thread::queued_event *next= events->next; + Log_event_type event_type= events->ev->get_type_code(); + rpl_group_info *rgi= events->rgi; + rpl_parallel_entry *entry= rgi->parallel_entry; + uint64 wait_for_sub_id; + uint64 wait_start_sub_id; + bool end_of_group; + Add a comment why this is handled here and not in rpt_handle_event() + if (event_type == GTID_EVENT) + { + in_event_group= true; Add a comment what the following variable stands for + group_standalone= + (0 != (static_cast<Gtid_log_event *>(events->ev)->flags2 & + Gtid_log_event::FL_STANDALONE)); + Rest of this file will be reviewed tomorrow. <cut> === modified file 'sql/rpl_rli.cc' --- sql/rpl_rli.cc 2013-06-06 15:51:28 +0000 +++ sql/rpl_rli.cc 2013-08-29 08:18:22 +0000 @@ -1194,13 +1194,15 @@ bool Relay_log_info::cached_charset_comp void Relay_log_info::stmt_done(my_off_t event_master_log_pos, - time_t event_creation_time, THD *thd) + time_t event_creation_time, THD *thd, + struct rpl_group_info *rgi) Do we need thd anymore as this is in rgi->thd? If they are not the same, please add a comment about this. It's not otherwise clear when to use rgi->thd and thd as this function uses both. @@ -1265,7 +1267,7 @@ void Relay_log_info::cleanup_context(THD { DBUG_ENTER("Relay_log_info::cleanup_context"); - DBUG_ASSERT(sql_thd == thd); + DBUG_ASSERT(opt_slave_parallel_threads > 0 || sql_thd == thd); Add a comment which THD is passed to the function. <cut> + +int +event_group_new_gtid(rpl_group_info *rgi, Gtid_log_event *gev) +{ + uint64 sub_id= rpl_global_gtid_slave_state.next_subid(gev->domain_id); + if (!sub_id) + { Add a comment that this is an end of memory error. <cut> + +void +delete_or_keep_event_post_apply(Relay_log_info *rli, + Log_event_type typ, Log_event *ev) +{ + /* + ToDo: This needs to work on rpl_group_info, not Relay_log_info, to be + thread-safe for parallel replication. + */ + switch (typ) { + case FORMAT_DESCRIPTION_EVENT: + /* + Format_description_log_event should not be deleted because it + will be used to read info about the relay log's format; + it will be deleted when the SQL thread does not need it, + i.e. when this thread terminates. + */ + break; + case ANNOTATE_ROWS_EVENT: + /* + Annotate_rows event should not be deleted because after it has + been applied, thd->query points to the string inside this event. + The thd->query will be used to generate new Annotate_rows event + during applying the subsequent Rows events. + */ + rli->set_annotate_event((Annotate_rows_log_event*) ev); I assume we should create relay_group_info->set_annotate_event instead of the above. + break; + case DELETE_ROWS_EVENT: + case UPDATE_ROWS_EVENT: + case WRITE_ROWS_EVENT: + /* + After the last Rows event has been applied, the saved Annotate_rows + event (if any) is not needed anymore and can be deleted. + */ + if (((Rows_log_event*)ev)->get_flags(Rows_log_event::STMT_END_F)) + rli->free_annotate_event(); + /* fall through */ + default: + DBUG_PRINT("info", ("Deleting the event after it has been executed")); + if (!rli->is_deferred_event(ev)) + delete ev; I assume that is_deferred_event should also be moved to relay_group_info. + break; + } +} + #endif === modified file 'sql/rpl_rli.h' --- sql/rpl_rli.h 2013-06-06 15:51:28 +0000 +++ sql/rpl_rli.h 2013-08-29 08:18:22 +0000 @@ -22,6 +22,7 @@ #include "log.h" /* LOG_INFO, MYSQL_BIN_LOG */ #include "sql_class.h" /* THD */ #include "log_event.h" +#include "rpl_parallel.h" struct RPL_TABLE_LIST; class Master_info; @@ -52,6 +53,8 @@ class Master_info; *****************************************************************************/ +struct rpl_group_info; + class Relay_log_info : public Slave_reporting_capability { public: @@ -311,13 +314,9 @@ class Relay_log_info : public Slave_repo char slave_patternload_file[FN_REFLEN]; size_t slave_patternload_file_size; - /* - Current GTID being processed. - The sub_id gives the binlog order within one domain_id. A zero sub_id - means that there is no active GTID. - */ - uint64 gtid_sub_id; - rpl_gtid current_gtid; + /* ToDo: We need to remove this, always use the per-transaction one to work with parallel replication. */ + struct rpl_group_info *group_info; + rpl_parallel parallel; Relay_log_info(bool is_slave_recovery); ~Relay_log_info(); @@ -459,7 +458,8 @@ class Relay_log_info : public Slave_repo the <code>Seconds_behind_master</code> field. */ void stmt_done(my_off_t event_log_pos, - time_t event_creation_time, THD *thd); + time_t event_creation_time, THD *thd, + struct rpl_group_info *rgi); /** @@ -594,6 +594,60 @@ class Relay_log_info : public Slave_repo }; +/* + This is data for various state needed to be kept for the processing of + one event group in the SQL thread. + + For single-threaded replication it is linked from the RLI, for parallel + replication it is linked into each event group being executed in parallel. +*/ Could you please rewrite the above to make it more clear. After discussing with you, I think the following could be added: /* In single-threaded replication, there will be one global rpl_group_info and one global RPLI per master connection. They will be linked together. In parallel replication, there will be one rpl_group_info object for each running thd. All rpl_group_info will share the same RLI. */ +struct rpl_group_info +{ + Relay_log_info *rli; + THD *thd; <cut> === modified file 'sql/sql_class.cc' --- sql/sql_class.cc 2013-06-06 15:51:28 +0000 +++ sql/sql_class.cc 2013-08-29 08:18:22 +0000 +void +wait_for_commit::wakeup() +{ + /* + We signal each waiter on their own condition and mutex (rather than using + pthread_cond_broadcast() or something like that). + + Otherwise we would need to somehow ensure that they were done + waking up before we could allow this THD to be destroyed, which would + be annoying and unnecessary. + */ + mysql_mutex_lock(&LOCK_wait_commit); + waiting_for_commit= false; + mysql_cond_signal(&COND_wait_commit); + mysql_mutex_unlock(&LOCK_wait_commit); +} In this particular case it looks safe to move the cond signal out of the mutex. I don't see how anyone could miss the signal. + + +/* + Register that the next commit of this THD should wait to complete until + commit in another THD (the waitee) has completed. + + The wait may occur explicitly, with the waiter sitting in + wait_for_prior_commit() until the waitee calls wakeup_subsequent_commits(). + + Alternatively, the TC (eg. binlog) may do the commits of both waitee and + waiter at once during group commit, resolving both of them in the right + order. + + Only one waitee can be registered for a waiter; it must be removed by + wait_for_prior_commit() or unregister_wait_for_prior_commit() before a new + one is registered. But it is ok for several waiters to register a wait for + the same waitee. It is also permissible for one THD to be both a waiter and + a waitee at the same time. +*/ +void +wait_for_commit::register_wait_for_prior_commit(wait_for_commit *waitee) +{ + waiting_for_commit= true; + DBUG_ASSERT(!this->waitee /* No prior registration allowed */); + this->waitee= waitee; + + mysql_mutex_lock(&waitee->LOCK_wait_commit); + /* + If waitee is in the middle of wakeup, then there is nothing to wait for, + so we need not register. This is necessary to avoid a race in unregister, + see comments on wakeup_subsequent_commits2() for details. + */ + if (waitee->wakeup_subsequent_commits_running) + waiting_for_commit= false; + else + { Please add a comment here what the following code is doing. As this is two different lists involved, it's not trivial code. You also seam to use the list next_subsequent_commit for something different in queue_for_group_commit() code. + this->next_subsequent_commit= waitee->subsequent_commits_list; + waitee->subsequent_commits_list= this; + } + mysql_mutex_unlock(&waitee->LOCK_wait_commit); +} <cut> +/* + Wakeup anyone waiting for us to have committed. + + Note about locking: + + We have a potential race or deadlock between wakeup_subsequent_commits() in + the waitee and unregister_wait_for_prior_commit() in the waiter. + + Both waiter and waitee needs to take their own lock before it is safe to take + a lock on the other party - else the other party might disappear and invalid + memory data could be accessed. But if we take the two locks in different + order, we may end up in a deadlock. + + The waiter needs to lock the waitee to delete itself from the list in + unregister_wait_for_prior_commit(). Thus wakeup_subsequent_commits() can not + hold its own lock while locking waiters, lest we deadlock. lest ? + So we need to prevent unregister_wait_for_prior_commit() running while wakeup + is in progress - otherwise the unregister could complete before the wakeup, + leading to incorrect spurious wakeup or accessing invalid memory. + <cut> + Then also register_wait_for_prior_commit() needs to check if + wakeup_subsequent_commits() is running, and skip the registration if + so. This is needed in case a new waiter manages to register itself and + immediately try to unregister while wakeup_subsequent_commits() is + running. Else the new waiter would also wait rather than unregister, but it + would not be woken up until next wakeup, which could be potentially much + later than necessary. +*/ Add new line here +void +wait_for_commit::wakeup_subsequent_commits2() +{ <cut> === modified file 'sql/sql_class.h' --- sql/sql_class.h 2013-06-06 15:51:28 +0000 +++ sql/sql_class.h 2013-08-29 08:18:22 +0000 @@ -1553,6 +1553,116 @@ class Global_read_lock }; +/* + Class to facilitate the commit of one transactions waiting for the commit of + another transaction to complete first. + + This is used during (parallel) replication, to allow different transactions + to be applied in parallel, but still commit in order. + + The transaction that wants to wait for a prior commit must first register + to wait with register_wait_for_prior_commit(waitee). Such registration + must be done holding the waitee->LOCK_wait_commit, to prevent the other + THD from disappearing during the registration. + + Then during commit, if a THD is registered to wait, it will call + wait_for_prior_commit() as part of ha_commit_trans(). If no wait is + registered, or if the waitee for has already completed commit, then + wait_for_prior_commit() returns immediately. + + And when a THD that may be waited for has completed commit (more precisely + commit_ordered()), then it must call wakeup_subsequent_commits() to wake + up any waiters. Note that this must be done at a point that is guaranteed + to be later than any waiters registering themselves. It is safe to call + wakeup_subsequent_commits() multiple times, as waiters are removed from + registration as part of the wakeup. + + The reason for separate register and wait calls is that this allows to + register the wait early, at a point where the waited-for THD is known to + exist. And then the actual wait can be done much later, where the + waited-for THD may have been long gone. By registering early, the waitee + can signal before disappearing. +*/ Instead of taking locks to ensure that THD will disapper, why not keep all THD objects around until all transactions in a group has been executed? +struct wait_for_commit +{ <cut> + bool waiting_for_commit; + /* + Flag set when wakeup_subsequent_commits_running() is active, see commonts commonts -> comments === modified file 'sql/sys_vars.cc' --- sql/sys_vars.cc 2013-06-07 07:31:11 +0000 +++ sql/sys_vars.cc 2013-08-29 08:18:22 +0000 <cut> +static bool +check_slave_parallel_threads(sys_var *self, THD *thd, set_var *var) +{ + bool running; + + mysql_mutex_lock(&LOCK_active_mi); + running= master_info_index->give_error_if_slave_running(); + mysql_mutex_unlock(&LOCK_active_mi); + if (running) + return true; + + return false; +} + +static bool +fix_slave_parallel_threads(sys_var *self, THD *thd, enum_var_type type) +{ + bool running; + bool err= false; + + mysql_mutex_unlock(&LOCK_global_system_variables); + mysql_mutex_lock(&LOCK_active_mi); + running= master_info_index->give_error_if_slave_running(); + mysql_mutex_unlock(&LOCK_active_mi); You can call check_slave_parallel_threads() above to reuse code. + if (running || rpl_parallel_change_thread_count(&global_rpl_thread_pool, + opt_slave_parallel_threads)) + err= true; + mysql_mutex_lock(&LOCK_global_system_variables); + + return err; +} + + +static Sys_var_ulong Sys_slave_parallel_threads( + "slave_parallel_threads", + "If non-zero, number of threads to spawn to apply in parallel events " + "on the slave that were group-committed on the master or were logged " + "with GTID in different replication domains.", + GLOBAL_VAR(opt_slave_parallel_threads), CMD_LINE(REQUIRED_ARG), + VALID_RANGE(0,16383), DEFAULT(0), BLOCK_SIZE(1), NO_MUTEX_GUARD, + NOT_IN_BINLOG, ON_CHECK(check_slave_parallel_threads), + ON_UPDATE(fix_slave_parallel_threads)); #endif We have talked about having 1 to be the minimum, which would give the old behavior. <cut> Overal things looks very good and it should not be too much work to add the things you have written on the todo part.a Regards Monty

2 10

[Maria-developers] [Spatial] On current implementation approach
by Mateusz Loskot 25 Sep '13

25 Sep '13

Hi, I'm going to ask question about how the current Spatial Extensions are implemented. I have spent some time reading the source code in the current trunk (spatial.h|cc, gcal*.h|cc, related Field and Item definitions, etc.), so I have a rough understanding of the overall structure, how the geometry data types are implemented and exposed to SQL, how the spatial functions are defined and registered. I did not looked into details of implementation of geospatial algorithms, but that's too low level for the question I'm going to ask. Initially, I was going to ask very detailed question, listing all the relevant code definitions and asking separately about each of them, but that's not necessary at this stage, I think. Instead, I'm going to simplify and ask about the bigger picture, more about MariaDB extensions API: 1. Is it possible to implement MariaDB extensions like Spatial (custom type + set of functions) without such a tight coupling with the internal implementation of the type system (without messing Field class with geometry types directly, etc.)? 2. Is it possible to implement Spatial using User-Defined Functions (UDF) defined in shared binary? 3. What is the reason behind using Well-Known-Binary (WKB) stream of bytes to transport geometry values into/from functions? Is it due to limitations of MariaDB type system where String is the only universal carrier for complex data? This concern is related to necessity of encoding/decoding WKB when chaining spatial function calls, and possibilities to avoid it. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net "Participation in this whole process is a form of torture" ~~ Szalony

3 10

[Maria-developers] close mdev-4387
by Roberto Spadim 23 Sep '13

23 Sep '13

hi guys, please close https://mariadb.atlassian.net/browse/MDEV-4387 it's duplicated by alread commited https://mariadb.atlassian.net/browse/MDEV-5025 thanks =) -- Roberto Spadim

1 0

Re: [Maria-developers] [Roles] Final Status
by Vicentiu Ciorbaru 23 Sep '13

23 Sep '13

Hi Sergei! >> Even though I didn't manage to implement everything yet, I want to >> finish this project and get it ready for an upstream push, outside of >> GSoC. > 1. I'd need either from you in email (but then cc: it to > maria-developers) or in your blog (or both) a statement about the > license. A simple one would be to say that this your code is available > to us under BSD license. Another option is our MariaDB Contributor > Agreement: https://mariadb.com/kb/en/mca/ I hereby declare that all my work done so far (and future) by me on the Roles project, is released under the BSD licence. I will also make a statement on my blog in my wrap up post a bit later. If there are any more clarifications needed, let me know! Regards, Vicențiu

2 2

[Maria-developers] Review of MDEV-4506, parallel replication, part 3
by Michael Widenius 23 Sep '13

23 Sep '13

Hi! This is a review of the latest code pushed to your tree: bzr diff -r 3662.. Note that I will apply some of the trivial changes myself to the tree to speed up things (this is mostly about comments). === added file 'mysql-test/suite/rpl/r/rpl_parallel.result' --- mysql-test/suite/rpl/r/rpl_parallel.result 1970-01-01 00:00:00 +0000 +++ mysql-test/suite/rpl/r/rpl_parallel.result 2013-09-22 20:25:20 +0000 Great that you got these done! Will make my life much easier when I work the parallel replication code! <cut> === modified file 'sql/log.cc' --- sql/log.cc 2013-07-04 22:26:15 +0000 +++ sql/log.cc 2013-09-22 20:25:20 +0000 @@ -6542,27 +6542,93 @@ MYSQL_BIN_LOG::write_transaction_to_binl } } + +/* + Put a transaction that is ready to commit in the group commit queue. + The transaction is identified by the ENTRY object passed into this function. + + To facilitate group commit for the binlog, we first queue up ourselves in + this function. Then later the first thread to enter the queue waits for + the LOCK_log mutex, and commits for everyone in the queue once it gets the + lock. Any other threads in the queue just wait for the first one to finish + the commit and wake them up. This way, all transactions in the queue get + committed in a single disk operation. + + The return value of this function is TRUE if queued as the first entry in + the queue (meaning this is the leader), FALSE otherwise. I moved this last in the comment as: @retval TRUE If queued as the first entry in the queue (meaning this is the leader) @retval FALSE Otherwise + + The main work in this function is when the commit in one transaction has + been marked to wait for the commit of another transaction to happen + first. This is used to support in-order parallel replication, where + transactions can execute out-of-order but need to be committed in-order with + how they happened on the master. The waiting of one commit on another needs + to be integrated with the group commit queue, to ensure that the waiting + transaction can participate in the same group commit as the waited-for + transaction. + + So when we put a transaction in the queue, we check if there were other + transactions already prepared to commit but just waiting for the first one + to commit. If so, we add those to the queue as well, transitively for all + waiters. +*/ + <cut> @@ -6574,10 +6640,28 @@ MYSQL_BIN_LOG::queue_for_group_commit(gr This would be natural to do with recursion, but we want to avoid potentially unbounded recursion blowing the C stack, so we use the list approach instead. + + We keep a list of all the waiters that need to be processed in `list', + linked through the next_subsequent_commit pointer. Initially this list + contains only the entry passed into this function. + + We process entries in the list one by one. The element currently being + processed is pointed to by `cur`, and the element at the end of the list + is pointed to by `last` (we do not use NULL to terminate the list). + + As we process an element, it is first added to the group_commit_queue. + Then any waiters for that element are added at the end of the list, to + be processed in subsequent iterations. This continues until the list + is exhausted, with all elements ever added eventually processed. + + The end result is a breath-first traversal of the tree of waiters, + re-using the next_subsequent_commit pointers in place of extra stack + space in a recursive traversal. I added: The temporary list created in next_subsequent_commit is not used by the caller or any other function. @@ -6670,6 +6763,18 @@ MYSQL_BIN_LOG::queue_for_group_commit(gr { if (list->wakeup_subsequent_commits_running) { + /* + ToDo: We should not need a full lock/unlock of LOCK_wait_commit + here. All we need is a single (full) memory barrier, to ensure that + the reads of the list above are not reordered with the write of + wakeup_subsequent_commits_running, or with the writes to the list + from other threads that is allowed to happen after + wakeup_subsequent_commits_running has been set to false. + + We do not currently have explicit memory barrier primitives in the + source tree, but if we get them the below mysql_mutex_lock() could + be replaced with a full memory barrier just before the loop. + */ ok (for now). Another way would of course to only do a mutex lock for the first entry we find... No more comments. Everything looked fine! I will start coordinate with you tomorrow about things to do. Regards, Monty

1 0

[Maria-developers] Status and future of spatial extensions
by Mateusz Loskot 23 Sep '13

23 Sep '13

Hi, I haven't monitored MariaDB development closely, neither I tracked what's happening in the MySQL sandbox, so I'm hoping to catch up by asking questions here. I'm interested in the Spatial Extensions. So, I looked around the KB and the documentation and I found a few pages: general overview [1], 5.3 features [2] and page about plans [3]. I've looked into JIRA issues and recent commits too. Still, I don't see a clear big picture about the current status, roadmap and developers of the Spatial Extensions. Could anyone shed more light on the Spatial part of the MariaDB project, where it is now, where it is going, who is behind the wheel, etc.? What is (if any) the relation with geo-spatial development in MySQL? Is that correct understanding that MySQL and MariaDB follow separate paths, thus the GIS implementations go separately and in future they may seriously diverge and become no longer equivalent drop-in replacements? [1] https://mariadb.com/kb/en/gis-functionality/ [2] https://mariadb.com/kb/en/the-maria53-gis-tree-on-launchpad/ [3] https://mariadb.com/kb/en/mariadb-plans-gis/ Best regards, -- Mateusz Loskot, http://mateusz.loskot.net "Participation in this whole process is a form of torture" ~~ Szalony

1 0

Re: [Maria-developers] Rev 3818: Check for duplicate index (port from mysql)
by Sergei Golubchik 22 Sep '13

22 Sep '13

Hi, Sanja! On Sep 20, sanja(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 3818 > revision-id: sanja(a)askmonty.org-20130920193019-vr2q24c8g255aofu > parent: sanja(a)askmonty.org-20130920141533-nwk5jt07ojwmmctr > committer: sanja(a)askmonty.org > branch nick: work-maria-10.0-fts2 > timestamp: Fri 2013-09-20 22:30:19 +0300 > message: > Check for duplicate index (port from mysql) > (pre fts) ok to push Regards, Sergei

1 0

[Maria-developers] Review of MDEV-4506, parallel replication, part 2
by Michael Widenius 22 Sep '13

22 Sep '13

Hi! Part 2 of review of parallel replication === added file 'sql/rpl_parallel.cc' <cut> +pthread_handler_t +handle_rpl_parallel_thread(void *arg) +{ <cut> + if (end_of_group) + { + in_event_group= false; + Add comment: /* All events for this group has now been executed and logged (but not necessaerly synced). Inform the other event groups that are waiting for this thread that we are done. */ + rgi->commit_orderer.unregister_wait_for_prior_commit(); + thd->wait_for_commit_ptr= NULL; + + /* + Record that we have finished, so other event groups will no + longer attempt to wait for us to commit. + + We can race here with the next transactions, but that is fine, as + long as we check that we do not decrease last_committed_sub_id. If + this commit is done, then any prior commits will also have been + done and also no longer need waiting for. + */ + mysql_mutex_lock(&entry->LOCK_parallel_entry); + if (entry->last_committed_sub_id < event_gtid_sub_id) + { + entry->last_committed_sub_id= event_gtid_sub_id; + mysql_cond_broadcast(&entry->COND_parallel_entry); + } + mysql_mutex_unlock(&entry->LOCK_parallel_entry); + Add: /* Tell storage engines that they can stop waiting. See comments in sql_class.cc::wakeup_subsequent_commits2() for why we can't do this part of the wakeup in unregister_wait_for_prior_commit(). */ + rgi->commit_orderer.wakeup_subsequent_commits(); + delete rgi; + } + + events= next; + } + + mysql_mutex_lock(&rpt->LOCK_rpl_thread); + if ((events= rpt->event_queue) != NULL) + { + rpt->event_queue= rpt->last_in_queue= NULL; + mysql_mutex_unlock(&rpt->LOCK_rpl_thread); + goto more_events; + } + + if (!in_event_group) + { + rpt->current_entry= NULL; As far as I can see, rpt->free is always 0 here. - It's set to 0 at the top of this loop and only set to 1 here. Can we remove the rpt->free variable? + if (!rpt->stop && !rpt->free) + { Add comment. /* Take next group of events from the replication pool. This is faster than having to wakeup the pool manager thread to give us a new event. */ + mysql_mutex_lock(&rpt->pool->LOCK_rpl_thread_pool); + list= rpt->pool->free_list; + rpt->next= list; + rpt->pool->free_list= rpt; + if (!list) + mysql_cond_broadcast(&rpt->pool->COND_rpl_thread_pool); + mysql_mutex_unlock(&rpt->pool->LOCK_rpl_thread_pool); + rpt->free= true; + } + } + } <cut> +int +rpl_parallel_change_thread_count(rpl_parallel_thread_pool *pool, + uint32 new_count, bool skip_check) +{ + uint32 i; + rpl_parallel_thread **new_list= NULL; + rpl_parallel_thread *new_free_list= NULL; + + /* + Allocate the new list of threads up-front. + That way, if we fail half-way, we only need to free whatever we managed + to allocate, and will not be left with a half-functional thread pool. + */ + if (new_count && + !(new_list= (rpl_parallel_thread **)my_malloc(new_count*sizeof(*new_list), + MYF(MY_WME)))) + { + my_error(ER_OUTOFMEMORY, MYF(0), (int(new_count*sizeof(*new_list)))); + goto err;; + } Why not allocate new_list[] elements at the same time? Faster, less memory not allocation to check and freeing is also esier. if (!my_multi_malloc(MYF(MY_WME), &new_list, (uint) new_count*sizeof(*new_list), &list_part, new_count*sizeof(**new_list), NULL)) ... + + for (i= 0; i < new_count; ++i) + { + pthread_t th; + + if (!(new_list[i]= (rpl_parallel_thread *)my_malloc(sizeof(*(new_list[i])), + MYF(MY_WME)))) + { + my_error(ER_OUTOFMEMORY, MYF(0), sizeof(*(new_list[i]))); + goto err; + } The above would then simple be: new_list[i]= list_part++; Instead of setting everything to false and NULL, why not simple bzero() the structure? That way we can be more sure that we don't forget to initialize any new element in the future. + new_list[i]->delay_start= true; + new_list[i]->running= false; + new_list[i]->stop= false; + new_list[i]->free= false; + mysql_mutex_init(key_LOCK_rpl_thread, &new_list[i]->LOCK_rpl_thread, + MY_MUTEX_INIT_SLOW); + mysql_cond_init(key_COND_rpl_thread, &new_list[i]->COND_rpl_thread, NULL); + new_list[i]->pool= pool; + new_list[i]->current_entry= NULL; + new_list[i]->event_queue= NULL; + new_list[i]->last_in_queue= NULL; <cut> Add: /* The following loops wait until each thread is done it's work for each event group and then stops it when the thread has nothing to do. Note that this may take some time as rpl_paralell:do_event() may request threads at the same time. */ To make this faster and ensuring we don't start new request, wouldn't it be better to go through pool->threads and signal each of them? As long as rpt->stop is defined as 'stop when you are done processing one group' then this is fine (which is how it looks it's defined now). + for (i= 0; i < pool->count; ++i) + { + rpl_parallel_thread *rpt= pool->get_thread(NULL); + rpt->stop= true; + mysql_cond_signal(&rpt->COND_rpl_thread); + mysql_mutex_unlock(&rpt->LOCK_rpl_thread); + } + + for (i= 0; i < pool->count; ++i) + { + rpl_parallel_thread *rpt= pool->threads[i]; + mysql_mutex_lock(&rpt->LOCK_rpl_thread); + while (rpt->running) + mysql_cond_wait(&rpt->COND_rpl_thread, &rpt->LOCK_rpl_thread); + mysql_mutex_unlock(&rpt->LOCK_rpl_thread); + mysql_mutex_destroy(&rpt->LOCK_rpl_thread); + mysql_cond_destroy(&rpt->COND_rpl_thread); + my_free(rpt); By using my_multi_malloc() you can remove the above free + } + + my_free(pool->threads); + pool->threads= new_list; + pool->free_list= new_free_list; + pool->count= new_count; + for (i= 0; i < pool->count; ++i) + { + mysql_mutex_lock(&pool->threads[i]->LOCK_rpl_thread); + pool->threads[i]->delay_start= false; + mysql_cond_signal(&pool->threads[i]->COND_rpl_thread); + mysql_mutex_unlock(&pool->threads[i]->LOCK_rpl_thread); + } + + if (!skip_check) + { + mysql_mutex_lock(&LOCK_active_mi); + pool->changing= false; + mysql_mutex_unlock(&LOCK_active_mi); + } + return 0; + +err: + if (new_list) + { + while (new_free_list) + { + rpl_parallel_thread *next= new_free_list->next; + mysql_mutex_lock(&new_free_list->LOCK_rpl_thread); + new_free_list->delay_start= false; + new_free_list->stop= true; + mysql_cond_signal(&new_free_list->COND_rpl_thread); + while (!new_free_list->running) + mysql_cond_wait(&new_free_list->COND_rpl_thread, + &new_free_list->LOCK_rpl_thread); + while (new_free_list->running) + mysql_cond_wait(&new_free_list->COND_rpl_thread, + &new_free_list->LOCK_rpl_thread); + mysql_mutex_unlock(&new_free_list->LOCK_rpl_thread); + my_free(new_free_list); By using my_multi_malloc() you can remove the above free and even the 'next' variable + new_free_list= next; + } + my_free(new_list); + } <cut> <cut> +void +rpl_parallel::wait_for_done() +{ + struct rpl_parallel_entry *e; + uint32 i; + + for (i= 0; i < domain_hash.records; ++i) + { + e= (struct rpl_parallel_entry *)my_hash_element(&domain_hash, i); + mysql_mutex_lock(&e->LOCK_parallel_entry); + while (e->current_sub_id > e->last_committed_sub_id) + mysql_cond_wait(&e->COND_parallel_entry, &e->LOCK_parallel_entry); + mysql_mutex_unlock(&e->LOCK_parallel_entry); + } +} How to handle errors in the above case? For example when we get an error and we had to abort things? I assume we should also abort for: e->rpl_thread->thd->killed() but this is hard as thd and even rpl_thread may be deleted. One way to fix this would be to add an entry 'killed' to rpl_parell_entry that we can set. However it will be hard to keep this up to date. The other way would be to, on error, set last_committed_sub_id back to it's last value. We could also have some kind of shutdown flag that we check to avoid hang on shutdown. What do you suggest? Assuming setting the value back would be the easiest solution.. + for (i= 0; i < domain_hash.records; ++i) + { + e= (struct rpl_parallel_entry *)my_hash_element(&domain_hash, i); + mysql_mutex_lock(&e->LOCK_parallel_entry); + while (e->current_sub_id > e->last_committed_sub_id) + mysql_cond_wait(&e->COND_parallel_entry, &e->LOCK_parallel_entry); + mysql_mutex_unlock(&e->LOCK_parallel_entry); + } +bool +rpl_parallel::do_event(struct rpl_group_info *serial_rgi, Log_event *ev) +{ + rpl_parallel_entry *e; + rpl_parallel_thread *cur_thread; + rpl_parallel_thread::queued_event *qev; + struct rpl_group_info *rgi; + Relay_log_info *rli= serial_rgi->rli; + enum Log_event_type typ; + + /* ToDo: what to do with this lock?!? */ + mysql_mutex_unlock(&rli->data_lock); + + if (!(qev= (rpl_parallel_thread::queued_event *)my_malloc(sizeof(*qev), + MYF(0)))) + { + my_error(ER_OUT_OF_RESOURCES, MYF(0)); + return true; + } + qev->ev= ev; + qev->next= NULL; + + if ((typ= ev->get_type_code()) == GTID_EVENT) + { + Gtid_log_event *gtid_ev= static_cast<Gtid_log_event *>(ev); + + if (!(e= find(gtid_ev->domain_id)) || + !(rgi= new rpl_group_info(rli)) || + event_group_new_gtid(rgi, gtid_ev)) + { Here add delete of rgi. Also set rgi to 0 before. + my_error(ER_OUT_OF_RESOURCES, MYF(MY_WME)); + return true; + } + + if ((gtid_ev->flags2 & Gtid_log_event::FL_GROUP_COMMIT_ID) && + e->last_commit_id == gtid_ev->commit_id) + { + /* + We are already executing something else in this domain. But the two + event groups were committed together in the same group commit on the + master, so we can still do them in parallel here on the slave. + + However, the commit of this event must wait for the commit of the prior + event, to preserve binlog commit order and visibility across all + servers in the replication hierarchy. + */ + rpl_parallel_thread *rpt= global_rpl_thread_pool.get_thread(e); Here we could add a test if get_thread() is 0, which could happen on shutdown or when killing a replication thread. I think it's relatively safe to use get_thread() as a end-synchronize point as we should never have anything half-done when we call get_thread()). As get_thread() can take a very long time and is not called in many places, it's also sounds like a resonable thing to do. + rgi->wait_commit_sub_id= e->current_sub_id; + rgi->wait_commit_group_info= e->current_group_info; + rgi->wait_start_sub_id= e->prev_groupcommit_sub_id; + e->rpl_thread= cur_thread= rpt; + /* get_thread() returns with the LOCK_rpl_thread locked. */ Add the abvove comments also where get_thread() is defined. + } + else + { + /* Check if we already have a worker thread for this entry. */ I had a hard time to understand what the above meant. Is this new suggested comments correct? /* Get the last or current thread for the domain */ + cur_thread= e->rpl_thread; + if (cur_thread) + { /* Active thread exists. The thread may still be executing queries for this domain or it may have been rescheduled to work for another domain. */ + mysql_mutex_lock(&cur_thread->LOCK_rpl_thread); + if (cur_thread->current_entry != e) + { + /* Not ours anymore, we need to grab a new one. */ Suggested new comment: /* The thread has been reused for another domain. (e->rpl_thread was not reset or was changed during the mutex_lock) As there is no threads working on this domain, it's safe to assign a new thread to work on it */ + mysql_mutex_unlock(&cur_thread->LOCK_rpl_thread); + e->rpl_thread= cur_thread= NULL; + } + } <cut> Add here: /* Current is not set for the case where the master did not have GTID, like a MariaDB 5.5 or MySQL master. */ + else if (!Log_event::is_group_event(typ) || !current) + { + /* + Events like ROTATE and FORMAT_DESCRIPTION. Do not run in worker thread. + Same for events not preceeded by GTID (we should not see those normally, + but they might be from an old master). + */ + qev->rgi= serial_rgi; + rpt_handle_event(qev, NULL); + delete_or_keep_event_post_apply(rli, typ, qev->ev); Why is delete_or_keep_event_post_apply() not part of rpt_handle_event()? You always call these two together... + + return false; + } + else + { The following code is identical (except one line) to the code earlier in this function. Better to create a function for this to reuse code and to clarify the differences. + cur_thread= current->rpl_thread; + if (cur_thread) + { + mysql_mutex_lock(&cur_thread->LOCK_rpl_thread); + if (cur_thread->current_entry != current) + { + /* Not ours anymore, we need to grab a new one. */ + mysql_mutex_unlock(&cur_thread->LOCK_rpl_thread); + cur_thread= NULL; + } + } + if (!cur_thread) + { + cur_thread= current->rpl_thread= + global_rpl_thread_pool.get_thread(current); + } + } + qev->rgi= current->current_group_info; + + /* + Queue the event for processing. + */ + if (cur_thread->last_in_queue) + cur_thread->last_in_queue->next= qev; + else + cur_thread->event_queue= qev; + cur_thread->last_in_queue= qev; + mysql_cond_signal(&cur_thread->COND_rpl_thread); + mysql_mutex_unlock(&cur_thread->LOCK_rpl_thread); In this case it's safe to move the cond_signal outside of the mutex_unlock. (Should make things notable faster as we win a sleep-wakup in other thread). <cut> === added file 'sql/rpl_parallel.h' --- sql/rpl_parallel.h 1970-01-01 00:00:00 +0000 +++ sql/rpl_parallel.h 2013-08-29 08:18:22 +0000 @@ -0,0 +1,92 @@ +#ifndef RPL_PARALLEL_H +#define RPL_PARALLEL_H + +#include "log_event.h" + + +struct rpl_parallel; +struct rpl_parallel_entry; +struct rpl_parallel_thread_pool; + +class Relay_log_info; +struct rpl_parallel_thread { + bool delay_start; I think we can remove deley_start. Se comments from previous review. + bool running; + bool stop; We need to define 'stop' properly. Like 'stop after we have executed the current event group' (So that we don't have to do rollback). + bool free; I think we can remove free. Se comments above. Otherwise it would be good to have a good descripition for each struct in this file. Will now start reviewing your new commits to the parallel replication tree. Regards, Monty

1 0

[Maria-developers] MDEV-5019 - THREADPOOL - Create Information Schema Table for Threadpool
by Roberto Spadim 22 Sep '13

22 Sep '13

Hi guys! I openned a new MDEV https://mariadb.atlassian.net/browse/MDEV-5019 the idea is expose the threadpool group information like oracle thread pool (more information about oracle thread pool here: http://mikaelronstrom.blogspot.com.br/2011_10_01_archive.html) I can help, but i don't know how access the threadpool at scheduler.h/scheduler.c from a information schema plugin 1) lock important mutex and others mdl 2) acess variables 3) write information schema based on variables 1 and 2 i don't know how to do :( can anyone help ? :) thanks guys :) -- Roberto Spadim

2 9

[Maria-developers] Perf tuning for hot key updating with high concurrency
by Hui Liu 22 Sep '13

22 Sep '13

It's an old issue which discussed in Percona-google-group: https://groups.google.com/forum/?fromgroups#!topic/percona-discussion/z3r4-… The patch provided in that discuss group is a rough idea, based on Percona-Server, and later I almost re-wrote the patch for real-case testing. I think it's more simple to implement in the InnoDB layer, compared to the the idea post by Monty, that introduce 'Incremental lock' or 'Relative update lock' to solve this problem. I backported them to Mariadb-10.0.4, and use simple simulate test with oltp_cc.lua, which is provided as an attachment. These notes migth be helpful to understand of this patch. 1) Why InnoDB's concurrency control is BAD for high concurrency hot key updating case? It's described in the percona-disscuss-group, please refer that. 2) What's the basic logic for high concurrency hot key updating? The idea is still the same as before, but the implement is different now. With a simple case that 1000 threads updating the same record id=1 and threshold=2 to describe the basic idea for this patch. conn (1000) ... (6) -> (5) -> (4) (1) (running) (2) (suspended) (3) (suspended) That is, thread 1 is running, and thread 2 and 3 are suspended, waiting for thread 1 relase the record lock for id=1. For thread 4, it detected they are already two threads suspended, and it would be rescheduled in hash queue. conn (1000) .. (6) -> (5) (1) (running) (2) (suspended) (3) (suspended) bucket1: (4) bucket2: And for 5 and 6 ... 1000, they are also rescheduled in that queue (1) (running) (2) (suspended) (3) (suspended) bucket1: (1000)...(6) -> (5) -> (4) bucket2: After 1 is committed, 2 is waked up and running, then 4 detects it would entry now, so it enters. (1) (commit/exit) (2) (running) (3) (suspended) (4) (suspended) bucket1: (1000) ... (6) -> (5) bucket2: The same logic for 5 and 6...1000. While for the old logic, a snapshot might be: (1) (commit/exit) (2) (running) (3) (suspended) (4) (suspended) (5) (suspended) .... (1000) (suspended) Hope the demo is not too naive. You can see, in such case, there are always threshold threads running inside innodb, while not all are suspended inside. 3) Why not use the new concurrency control with AOTMIC add/sub for active conc? The result is bad for new concurrency control in 5.6/10.0, that no FIFO any more. So I changed to default (5.1/5.5) control strategy again. Note the changes in CMakeLists.txt that make -DUSE_DEFAULT_CONC=ON as default. 4) How about the concurrency counter for rescheduled thread in bucket queue? If thread number waiting for the record lock is over the configured threshold, it would be rescheduled into queue in one hash bucket, hashed by (space,space_no, heap_no) a) for the first one in the queue, that is the queue header, it would timed-wait retrying whether it would enter into innodb layer, by checking the n_ref_count for that record stored by HASH. b) for the non-head threads, they would be FAKE released the concurrent slot, that means threads in FIFO would be waked up, enter/exit innodb with faked inside and tickets. Until they are force to exit innodb, the active slot number is equal to real case without rescheduling. 5) What about the basic test result? $ ./sysbench/sysbench --test=sysbench/tests/db/oltp_cc.lua --num-threads=1000 --mysql-socket=/u01/md1004/run/mysql.sock --mysql-db=bmw --oltp-tables-count=4 --mysql-user=root --oltp-table-size=4 --max-requests=0 --max-time=120000 --oltp-read-only=off run 1000conn, oltp-cc.lua, 4x4(16 hotkey): patched (innodb_lock_wait_threshold=12 as default) : 3.1K QPS, 9.1K TPS unpatched (innodb_lock_wait_threshold=0 as disabled) : 1.2K QPS, 3.5K TPS $ ./sysbench/sysbench --test=sysbench/tests/db/oltp_cc.lua --num-threads=1000 --mysql-socket=/u01/md1004/run/mysql.sock --mysql-db=bmw --oltp-tables-count=1 --mysql-user=root --oltp-table-size=4 --max-requests=0 --max-time=120000 --oltp-read-only=off run 1000conn, oltp-cc.lua, 1x4(4 hotkey): patched (innodb_lock_wait_threshold=12 as default) : 1.3K QPS, 3.5K TPS unpatched (innodb_lock_wait_threshold=0 as disabled) : 200 QPS, 720 TPS 6) Any rest issues? The rescheduling of the waiting thread breaks the normal mechanism: each lock request must be registered in lock-table, then deadlock could be found if exists. As I am trying to reduce n in lock relaetd O(n) and O(n^2) operations, for the deadlock case, it does not work very well. For example, the update sequance: t1 updates 1, and t2 updates 1, then t3/t4/t5/t6 update 1, and then t1 updates 2, after that t2 updates 1: t1 1 2 t2 2 1 t3 1 t4 1 t5 1 t6 1 As we can see, in the new schedule model, t3/t4/t5 are suspended, and might be rescheduled in queue. Then t1 is suspended as it updates 2 which record lock is hold by t2. t2 is rescheduled as it detected too many lock waiting for rec 1. The snapshot might be: (t1[2]) (suspended) (t3) (suspended) (t4) (suspended) bucket1: (t2[1]) -> (t6) -> (t5) bucket2: The deadlock can be break until t2 enters with 1. Then for t5/t6, they would wait until timeout, and then force enter into innodb side, suspended as it should be. It's same for t2. Until t2 enters, the deadlock is found. So, the cost for deadlock detecting might be much large than it should be, depends on the timeout and retry counts. Hope Jan or Monty would have a look for this patch, or other ideas for solve this problem, if you are going.

1 0

[Maria-developers] Prep for MariaDB 5.5.33a
by Daniel Bartholomew 20 Sep '13

20 Sep '13

I've begun the prep for the special 5.5.33a release. Draft changelog and release notes are here: https://mariadb.com/kb/en/mariadb-5533a-changelog/ https://mariadb.com/kb/en/mariadb-5533a-release-notes/ As always, fixes, additions, and such are welcome. I'm currently just waiting for buildbot to finish building and then I'll package up the release so that it can get mirrored over the next few hours and we can release tomorrow after some final testing of the finished repos. Assuming things go well, my plan is to release around noon US Eastern time. Thanks. -- Daniel Bartholomew, Sr Tech Writer / Sys Admin MariaDB

2 1

Re: [Maria-developers] [Commits] Rev 3816: MDEV-4883 - Not all host_cache tests have been merged in lp:maria/10.0
by Roberto Spadim 20 Sep '13

20 Sep '13

Hi guys, what's the "MySQL variables auto-tuning code" means? i didn't saw that in mysql source and at mysql docs, any idea what this means? 2013/9/18 Sergey Vojtovich <svoj(a)mariadb.org> > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 3816 > revision-id: svoj(a)mariadb.org-20130918071716-yu2a5489uldzfuy9 > parent: sanja(a)askmonty.org-20130917190515-g3xncb1slqwy4cix > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0-mdev4883 > timestamp: Wed 2013-09-18 11:17:16 +0400 > message: > MDEV-4883 - Not all host_cache tests have been merged > > Merged main.host_cache_size_functionality test. > > Test had to be adjusted because MariaDB doesn't have > MySQL variables auto-tuning code. > === added file 'mysql-test/r/host_cache_size_functionality.result' > --- a/mysql-test/r/host_cache_size_functionality.result 1970-01-01 > 00:00:00 +0000 > +++ b/mysql-test/r/host_cache_size_functionality.result 2013-09-18 > 07:17:16 +0000 > @@ -0,0 +1,78 @@ > +'#________________________VAR_06_Host_Cache_Size__________________#' > +echo '##' > +--echo '#---------------------WL6372_VAR_6_01----------------------#' > +#################################################################### > +# Checking default value # > +#################################################################### > +SELECT COUNT(@@GLOBAL.Host_Cache_Size) > +1 Expected > +set @Default_host_cache_size=128; > +select @@global.Host_Cache_Size=@Default_host_cache_size; > +@@global.Host_Cache_Size=@Default_host_cache_size > +1 > +1 Expected > +'#---------------------WL6372_VAR_6_02----------------------#' > +# Restart server with Host_Cache_Size 1 > +SELECT @@GLOBAL.Host_Cache_Size; > +@@GLOBAL.Host_Cache_Size > +1 > +1 Expected > +set @Default_host_cache_size=128; > +SET @@GLOBAL.Host_Cache_Size=DEFAULT; > +select @@global.Host_Cache_Size=@Default_host_cache_size; > +@@global.Host_Cache_Size=@Default_host_cache_size > +1 > +1 Expected > +'#---------------------WL6372_VAR_6_03----------------------#' > +SET @@local.Host_Cache_Size=1; > +ERROR HY000: Variable 'host_cache_size' is a GLOBAL variable and should > be set with SET GLOBAL > +Expected error 'Global variable' > +SET @@session.Host_Cache_Size=1; > +ERROR HY000: Variable 'host_cache_size' is a GLOBAL variable and should > be set with SET GLOBAL > +Expected error 'Global variable' > +SET @@GLOBAL.Host_Cache_Size=1; > +SET @@GLOBAL.Host_Cache_Size=DEFAULT; > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +COUNT(@@GLOBAL.Host_Cache_Size) > +1 > +1 Expected > +select @@global.Host_Cache_Size=@Default_host_cache_size; > +@@global.Host_Cache_Size=@Default_host_cache_size > +1 > +1 Expected > +'#---------------------WL6372_VAR_6_04----------------------#' > +SELECT @@GLOBAL.Host_Cache_Size = VARIABLE_VALUE > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='Host_Cache_Size'; > +@@GLOBAL.Host_Cache_Size = VARIABLE_VALUE > +1 > +1 Expected > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +COUNT(@@GLOBAL.Host_Cache_Size) > +1 > +1 Expected > +SELECT COUNT(VARIABLE_VALUE) > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='Host_Cache_Size'; > +COUNT(VARIABLE_VALUE) > +1 > +1 Expected > +'#---------------------WL6372_VAR_6_05----------------------#' > +SELECT @@Host_Cache_Size = @@GLOBAL.Host_Cache_Size; > +@@Host_Cache_Size = @@GLOBAL.Host_Cache_Size > +1 > +1 Expected > +SELECT COUNT(@@local.Host_Cache_Size); > +ERROR HY000: Variable 'host_cache_size' is a GLOBAL variable > +Expected error 'Variable is a GLOBAL variable' > +SELECT COUNT(@@SESSION.Host_Cache_Size); > +ERROR HY000: Variable 'host_cache_size' is a GLOBAL variable > +Expected error 'Variable is a GLOBAL variable' > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +COUNT(@@GLOBAL.Host_Cache_Size) > +1 > +1 Expected > +SELECT Host_Cache_Size = @@SESSION.Host_Cache_Size; > +ERROR 42S22: Unknown column 'Host_Cache_Size' in 'field list' > +Expected error 'Unknown column Host_Cache_Size in field list' > +SET @@GLOBAL.Host_Cache_Size=DEFAULT; > > === added file 'mysql-test/t/host_cache_size_functionality.test' > --- a/mysql-test/t/host_cache_size_functionality.test 1970-01-01 > 00:00:00 +0000 > +++ b/mysql-test/t/host_cache_size_functionality.test 2013-09-18 > 07:17:16 +0000 > @@ -0,0 +1,185 @@ > > +############################################################################### > +# > # > +# Variable Name: Host_Cache_Size > # > +# Scope: Global > # > +# Access Type: Dynamic > # > +# Data Type: numeric > # > +# > # > +# > # > +# Creation Date: 2012-08-31 > # > +# Author : Tanjot Singh Uppal > # > +# > # > +# > # > +# Description:Test Cases of Dynamic System Variable Host_Cache_Size > # > +# that checks the behavior of this variable in the following > ways # > +# * Value Check > # > +# * Scope Check > # > +# * Functionality Check > # > +# * Accessability Check > # > +# > # > +# This test does not perform the crash recovery on this variable > # > +# For crash recovery test on default change please run the ibtest > # > > +############################################################################### > + > +-- source include/have_innodb.inc > +-- source include/not_embedded.inc > +-- source include/have_innodb_16k.inc > +--disable_warnings > +echo '#________________________VAR_06_Host_Cache_Size__________________#' > +echo '##' > +--echo '#---------------------WL6372_VAR_6_01----------------------#' > +#################################################################### > +# Checking default value # > +#################################################################### > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +--echo 1 Expected > + > +#set @Default_host_cache_size=(select > if(if(@@global.max_connections<500,128+@@global.max_connections,128+@ > @global.max_connections+floor((@@global.max_connections-500)/20))>2000,2000,if(@@global.max_connections<500,128+@ > @global.max_connections,128+@ > @global.max_connections+floor((@@global.max_connections-500)/20)))); > +set @Default_host_cache_size=128; > + > +select @@global.Host_Cache_Size=@Default_host_cache_size; > +--echo 1 Expected > + > + > +--echo '#---------------------WL6372_VAR_6_02----------------------#' > > +################################################################################# > +# Checking the Default value post starting the server with other value > # > > +################################################################################# > +--echo # Restart server with Host_Cache_Size 1 > + > +let $restart_file= $MYSQLTEST_VARDIR/tmp/mysqld.1.expect; > +--exec echo "wait" > $restart_file > +--shutdown_server 10 > +--source include/wait_until_disconnected.inc > +-- exec echo "restart:--host_cache_size=1 " > > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +-- enable_reconnect > +-- source include/wait_until_connected_again.inc > + > +--disable_warnings > + > +SELECT @@GLOBAL.Host_Cache_Size; > +--echo 1 Expected > + > +#set @Default_host_cache_size=(select > if(if(@@global.max_connections<500,128+@@global.max_connections,128+@ > @global.max_connections+floor((@@global.max_connections-500)/20))>2000,2000,if(@@global.max_connections<500,128+@ > @global.max_connections,128+@ > @global.max_connections+floor((@@global.max_connections-500)/20)))); > +set @Default_host_cache_size=128; > +SET @@GLOBAL.Host_Cache_Size=DEFAULT; > +select @@global.Host_Cache_Size=@Default_host_cache_size; > +--echo 1 Expected > + > + > +--echo '#---------------------WL6372_VAR_6_03----------------------#' > +#################################################################### > +# Checking Value can be set - Dynamic # > +#################################################################### > +--error ER_GLOBAL_VARIABLE > +SET @@local.Host_Cache_Size=1; > +--echo Expected error 'Global variable' > + > +--error ER_GLOBAL_VARIABLE > +SET @@session.Host_Cache_Size=1; > +--echo Expected error 'Global variable' > + > +SET @@GLOBAL.Host_Cache_Size=1; > +SET @@GLOBAL.Host_Cache_Size=DEFAULT; > + > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +--echo 1 Expected > + > +select @@global.Host_Cache_Size=@Default_host_cache_size; > +--echo 1 Expected > + > +--echo '#---------------------WL6372_VAR_6_04----------------------#' > +################################################################# > +# Check if the value in GLOBAL Table matches value in variable # > +################################################################# > +SELECT @@GLOBAL.Host_Cache_Size = VARIABLE_VALUE > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='Host_Cache_Size'; > +--echo 1 Expected > + > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +--echo 1 Expected > + > +SELECT COUNT(VARIABLE_VALUE) > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='Host_Cache_Size'; > +--echo 1 Expected > + > +--echo '#---------------------WL6372_VAR_6_05----------------------#' > > +################################################################################ > +# Checking Variable Scope > # > > +################################################################################ > +SELECT @@Host_Cache_Size = @@GLOBAL.Host_Cache_Size; > +--echo 1 Expected > + > +--Error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SELECT COUNT(@@local.Host_Cache_Size); > +--echo Expected error 'Variable is a GLOBAL variable' > + > +--Error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SELECT COUNT(@@SESSION.Host_Cache_Size); > +--echo Expected error 'Variable is a GLOBAL variable' > + > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +--echo 1 Expected > + > +--Error ER_BAD_FIELD_ERROR > +SELECT Host_Cache_Size = @@SESSION.Host_Cache_Size; > +--echo Expected error 'Unknown column Host_Cache_Size in field list' > + > +#The below check has been commented out as the IP fetch is different in a > P2P connection than BroadBand connection > +#--echo '#---------------------WL6372_VAR_6_06----------------------#' > > +############################################################################### > +# Checking the Host cahce functionality > # > > +############################################################################### > + > +#SET @@GLOBAL.Host_Cache_Size=2; > +#--disable_warnings > + > +#--perl > +#my $ip=`ifconfig | egrep "inet addr|inet" | sed -e 's/^.*inet addr://' > -e 's/^.*inet//'| sed 's/ .*\$//'|egrep -i "broadcast|bcast"|head -1|awk > '{print $1}'`; > +#open (LOGFH, ">" . $ENV{'MYSQL_TMP_DIR'} . "/bind_ip"); > +#print LOGFH "let \$bind_ip = $ip;\n"; > +#close LOGFH; > +#EOF > + > +#--source $MYSQL_TMP_DIR/bind_ip > +#--remove_file $MYSQL_TMP_DIR/bind_ip > + > +#let $restart_file= $MYSQLTEST_VARDIR/tmp/mysqld.1.expect; > +#--exec echo "wait" > $restart_file > +#--shutdown_server 10 > +#--source include/wait_until_disconnected.inc > +#-- exec echo "restart:--bind-address=$bind_ip " > > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +#-- enable_reconnect > +#-- source include/wait_until_connected_again.inc > + > +#connection default; > +#--disable_warnings > + > +#create user binduser; > +#grant all on *.* to binduser; > + > +#select count(IP) from performance_schema.host_cache; > +#--echo 0 Expected > + > +#connect (con1,$bind_ip,binduser,,); > +#select count(IP) from performance_schema.host_cache; > +#--echo 1 Expected > + > +#disconnect con1; > +#connection default; > + > +#--disable_warnings > + > +# The below check is hashed until the BUG14689561# is fixed > +# > +#flush hosts; > + > +#connect (con2,$bind_ip,binduser,,); > +#select count(IP) from performance_schema.host_cache; > +#--echo 1 Expected > + > +#disconnect con2; > + > +SET @@GLOBAL.Host_Cache_Size=DEFAULT; > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits > -- Roberto Spadim SPAEmpresarial

2 1

Re: [Maria-developers] Rev 3814: Client attributes
by Sergei Golubchik 19 Sep '13

19 Sep '13

Hi, Sanja! On Sep 19, sanja(a)askmonty.org wrote: > === modified file 'sql/sql_acl.cc' > --- a/sql/sql_acl.cc 2013-07-21 14:39:19 +0000 > +++ b/sql/sql_acl.cc 2013-09-19 10:19:36 +0000 > @@ -4615,7 +4615,8 @@ bool check_grant(THD *thd, ulong want_ac > case ACL_INTERNAL_ACCESS_GRANTED: > /* > Currently, > - - the information_schema does not subclass ACL_internal_table_access, > + - the information_schema does not subclass ACL_internal_table_access > +, Eh? What was that? > there are no per table privilege checks for I_S, > - the performance schema does use per tables checks, but at most > returns 'CHECK_GRANT', and never 'ACCESS_GRANTED'. > @@ -8222,6 +8223,43 @@ static bool find_mpvio_user(MPVIO_EXT *m > mpvio->acl_user->plugin.str)); > DBUG_RETURN(0); > } > + > +static bool > +read_client_connect_attrs(char **ptr, size_t *max_bytes_available, > + const CHARSET_INFO *from_cs) > +{ > + size_t length, length_length; > + char *ptr_save; > + /* not enough bytes to hold the length */ > + if (*max_bytes_available < 1) > + return true; > + > + /* read the length */ > + ptr_save= *ptr; > + length= net_field_length_ll((uchar **) ptr); > + length_length= *ptr - ptr_save; > + if (*max_bytes_available < length_length) > + return true; This is incorrect, it might read past the end of the buffer. it's too late to check the length *after* you've read the data. Correct code might look like if (*max_bytes_available >= 9) { ptr_save= *ptr; length= net_field_length_ll((uchar **) ptr); length_length= *ptr - ptr_save; DBUG_ASSERT(length_length <= 9); *max_bytes_available-= length_length; } else { char buf[10], *len_ptr= buf; strncpy(buf, *ptr, 9) length= net_field_length_ll((uchar **) &len_ptr); length_length= len_ptr - buf; *ptr+= length_length; if (*max_bytes_available < length_length) return true; *max_bytes_available-= length_length; } > + > + *max_bytes_available-= length_length; > + > + /* length says there're more data than can fit into the packet */ > + if (length > *max_bytes_available) > + return true; > + > + /* impose an artificial length limit of 64k */ > + if (length > 65535) > + return true; > + > +#ifdef HAVE_PSI_THREAD_INTERFACE > + if (PSI_THREAD_CALL(set_thread_connect_attrs)(*ptr, length, from_cs) && > + current_thd->variables.log_warnings) > + sql_print_warning("Connection attributes of length %lu were truncated", > + (unsigned long) length); > +#endif > + return false; > +} > + > #endif > > /* the packet format is described in send_change_user_packet() */ > @@ -8349,6 +8387,13 @@ static bool parse_com_change_user_packet > } > } > > + size_t bytes_remaining_in_packet= end - ptr; 1. This is very wrong, did you ever test it? You need a pointer to *after* the plugin name. But 'ptr' is two bytes *before* the plugin. 2. remove bytes_remaining_in_packet, pass 'end' instead to read_client_connect_attrs(). > + > + if ((mpvio->thd->client_capabilities & CLIENT_CONNECT_ATTRS) && > + read_client_connect_attrs(&ptr, &bytes_remaining_in_packet, > + mpvio->thd->charset())) > + return packet_error; > + > DBUG_PRINT("info", ("client_plugin=%s, restart", client_plugin)); > /* > Remember the data part of the packet, to present it to plugin in > @@ -8487,7 +8532,21 @@ static ulong parse_client_handshake_pack > /* strlen() can't be easily deleted without changing protocol */ > db_len= db ? strlen(db) : 0; > > - char *client_plugin= passwd + passwd_len + (db ? db_len + 1 : 0); > + char *client_plugin= end= passwd + passwd_len + (db ? db_len + 1 : 0); > + > + if ((thd->client_capabilities & CLIENT_PLUGIN_AUTH) && > + (client_plugin < (char *)net->read_pos + pkt_len)) > + { > + end+= strlen(end) + 1; > + } > + > + size_t bytes_remaining_in_packet= (((char *)net->read_pos) + pkt_len) - end; This is very-very wrong. if there're no connection attributes, then your 'end' will be after the end of the packet, it'll be after net->read_pos + pkt_len. But then you subtract, get -1 and cast it to unsigned size_t ! Don't worry, when you remove bytes_remaining_in_packet as I wrote above, this bug will go away. > + > + if (db && > + (thd->client_capabilities & CLIENT_CONNECT_ATTRS) && > + read_client_connect_attrs(&end, &bytes_remaining_in_packet, > + mpvio->thd->charset())) > + return packet_error; Put it in the same place as in parse_com_change_user_packet, that is after reading the plugin name (and fix_plugin_ptr). And rename 'end' to 'connection_attributes' > > /* Since 4.1 all database names are stored in utf8 */ > if (db) > > === modified file 'sql/sys_vars.cc' > --- a/sql/sys_vars.cc 2013-08-14 08:48:50 +0000 > +++ b/sql/sys_vars.cc 2013-09-19 10:19:36 +0000 > @@ -71,6 +71,10 @@ > #define export /* not static */ > > #ifdef WITH_PERFSCHEMA_STORAGE_ENGINE > +#ifndef EMBEDDED_LIBRARY > +#define PFS_TRAILING_PROPERTIES \ > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(NULL), ON_UPDATE(NULL), \ > + NULL, sys_var::PARSE_EARLY Remove PFS_TRAILING_PROPERTIES define (not used in MariaDB) > > static Sys_var_mybool Sys_pfs_enabled( > "performance_schema", > @@ -331,8 +335,10 @@ static Sys_var_long Sys_pfs_connect_attr > GLOBAL_VAR(pfs_param.m_session_connect_attrs_sizing), > CMD_LINE(REQUIRED_ARG), VALID_RANGE(-1, 1024 * 1024), > DEFAULT(-1), > - BLOCK_SIZE(1)); > + BLOCK_SIZE(1), > + NO_MUTEX_GUARD, NOT_IN_BINLOG); Remove this change too (sysvars are NO_MUTEX_GUARD and NOT_IN_BINLOG by default) > > +#endif /* EMBEDDED_LIBRARY */ > #endif /* WITH_PERFSCHEMA_STORAGE_ENGINE */ > > static Sys_var_ulong Sys_auto_increment_increment( > > === modified file 'libmysqld/lib_sql.cc' > --- a/libmysqld/lib_sql.cc 2013-08-01 12:09:26 +0000 > +++ b/libmysqld/lib_sql.cc 2013-09-19 10:19:36 +0000 > @@ -698,12 +698,38 @@ err: > } > > > +static void > +emb_transfer_connect_attrs(MYSQL *mysql) > +{ > +#ifdef HAVE_PSI_THREAD_INTERFACE > + if (mysql->options.extension && > + mysql->options.extension->connection_attributes_length) > + { > + uchar *buf, *ptr; > + THD *thd= (THD*)mysql->thd; > + size_t length= mysql->options.extension->connection_attributes_length; > + > + /* 9 = max length of the serialized length */ > + ptr= buf= (uchar *) my_alloca(length + 9); > + send_client_connect_attrs(mysql, buf); > + net_field_length_ll(&ptr); > + PSI_THREAD_CALL(set_thread_connect_attrs)((char *) ptr, length, thd->charset()); > + my_afree(buf); > + } > +#endif > +} > + > + > #ifdef NO_EMBEDDED_ACCESS_CHECKS > int check_embedded_connection(MYSQL *mysql, const char *db) > { > int result; > LEX_STRING db_str = { (char*)db, db ? strlen(db) : 0 }; > THD *thd= (THD*)mysql->thd; > + > + /* the server does the same as the client */ > + mysql->server_capabilities= mysql->client_flag; > + Remove the similar line in init_embedded_mysql() > thd_init_client_charset(thd, mysql->charset->number); > thd->update_charset(); > Security_context *sctx= thd->security_ctx; > @@ -728,11 +755,17 @@ int check_embedded_connection(MYSQL *mys > we emulate a COM_CHANGE_USER user here, > it's easier than to emulate the complete 3-way handshake > */ > - char buf[USERNAME_LENGTH + SCRAMBLE_LENGTH + 1 + 2*NAME_LEN + 2], *end; > + char *buf, *end; > NET *net= &mysql->net; > THD *thd= (THD*)mysql->thd; > Security_context *sctx= thd->security_ctx; > + size_t connect_attrs_len= > + (mysql->server_capabilities & CLIENT_CONNECT_ATTRS && This doesn't seem to make any sense. Why does it check for CLIENT_CONNECT_ATTRS if it's part of the server (embedded). If a client connects to the server, then the client needs to check whether the server supports CLIENT_CONNECT_ATTRS, and the server needs to check whether the client supports it. But a server doesn't need to check whether it itself support it, it knows it already :) > + mysql->options.extension) ? > + mysql->options.extension->connection_attributes_length : 0; > > + buf= my_alloca(USERNAME_LENGTH + SCRAMBLE_LENGTH + 1 + 2*NAME_LEN + 2 + > + connect_attrs_len + 2); > if (mysql->options.client_ip) > { > sctx->host= my_strdup(mysql->options.client_ip, MYF(0)); > @@ -766,6 +799,13 @@ int check_embedded_connection(MYSQL *mys > int2store(end, (ushort) mysql->charset->number); > end+= 2; > > + //end= strmake(end, "mysql_native_password", NAME_LEN) + 1; Why is it commented out? > + > + /* the server does the same as the client */ > + mysql->server_capabilities= mysql->client_flag; > + > + end= (char *) send_client_connect_attrs(mysql, (uchar *) end); > + > /* acl_authenticate() takes the data from thd->net->read_pos */ > thd->net.read_pos= (uchar*)buf; Regards, Sergei

1 1

[Maria-developers] Please review: EXPLAIN UPDATE/DELETE, post-review changes.
by Sergei Petrunia 17 Sep '13

17 Sep '13

Hello Igor, I think I'm ready to submit the EXPLAIN UPDATE/DELETE code refactoring patch for another round of review. The code is pushed to lp:~~maria-captains/maria/10.0-base-explain-update-r2/ . I am also attaching the combined patch (against 10.0-base). BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] BUG? at windows versino of mariadb 10.0.4
by Roberto Spadim 16 Sep '13

16 Sep '13

Hi guys, i was trying the mariadb with lower value of thread_pool_max_threads to 3 and after my fourth connection i got a error and server crashed, it's a new server, i downloaded the 32bits version of mariadb, unziped and run this command line: C:\Program Files\mariadb-10.0.4-win32\bin>mysqld --datadir=..\data\ --log_error=..\log --port=3306 --thread_pool_max_threads=3 the error output at log was: 130915 21:43:37 [ERROR] mysqld got exception 0x80000003 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.4-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=4 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 141750 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... mysqld.exe!os_aio_windows_handle()[os0file.cc:4786] mysqld.exe!fil_aio_wait()[fil0fil.cc:5396] mysqld.exe!io_handler_thread()[srv0start.cc:477] kernel32.dll!BaseThreadInitThunk() ntdll.dll!RtlInitializeExceptionChain() ntdll.dll!RtlInitializeExceptionChain() The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. any help is wellcome, maybe the max_threads is bigger than thread_pool_max_threads? that's could be the problem? i didn't checked the source yet -- Roberto Spadim SPAEmpresarial

2 2

Re: [Maria-developers] review of mdev-4911
by Roberto Spadim 16 Sep '13

16 Sep '13

Hi! MDEV Created 1. MDEV-5018 <https://mariadb.atlassian.net/browse/MDEV-5018> 2013/9/14 Roberto Spadim <roberto(a)spadim.com.br> > Hi sergey :) well 10.1 with kill and subquery :) nice feature :) maybe too > much, but very nice > > > Em 14/09/2013 13:58, "Sergei Golubchik" <serg(a)mariadb.org> escreveu: > > Hi, Roberto! >> >> On Sep 13, Roberto Spadim wrote: >> > >> > > My preference is to kill connection by thread_id and query by >> > > query_id, because I normally either want to stop a particular query, >> > > or stop all activity in particular connection. But it is >> > > incompatible change. >> > > >> > there's mysql work with this kind of syntax? i didn't found it at >> bug.mysql >> > at mail list, Justin at Percona, talked about patchs in others forks >> maybe >> > a single unique syntax is better than maridb only syntax, check message >> > from MDEV description (copied from mail list): >> > >> > Justin Swanhartgreenlion(a)gmail.com Percona, Inc >> > >> > KILL THREAD THREAD_ID WITH QUERY_ID QUERY_ID >> > and >> > KILL QUERY THREAD_ID WITH QUERY_ID QUERY_ID >> > and possibly >> > KILL QUERY WITH QUERY_ID >> > >> > should be supported. This is a very important and missing feature >> > which is included in other forks. >> >> It's not in MySQL 5.6 and not in Percona Server 5.5 or 5.6. >> I did not check Google patches, Facebook patches and other sets of MySQL >> patches, though. >> > i didn`t found too :/ > >> >> > my problem isn't the program allowing a kill command since it can >> restart >> > the work or stop, it's not a problem >> > the "problem" is the boring time lost at a wrong kill command, since i >> use >> > persistent connections at php, and a thread running a script can be >> used in >> > another script without changing it thread_id (can be confirmed at show >> > processlist) >> > my problem is sending a kill command to the wrong thread since i'm using >> > the thread_id to kill the connection and not the query_id, check i use >> > "kill connection xxx" not "kill query xxx" >> >> Okay... This makes sense. If you use a connection pool that, indeed, >> connection id does not correspond to a logical connection. >> >> Still, while KILL CONNECTION QUERY_ID is kind of ok, KILL QUERY QUERY_ID >> is very silly. I'd rather allow subqueries in KILL, to support >> > kill query query_id is very ugly hehehe, kill query id is ok, and kill > connection query_id or query id with space is ok too > > >> >> KILL CONNECTION (select thread_id from information_schema.processlist >> where ...) >> >> then you won't need to kill by query id or state or if_idle - you can >> have everything in the where clause. >> >> but this is a larger change that we cannot do 10.0, we simply don't have >> time for it. We could try to do it in 10.1 though, properly and >> generally, so that you can kill using as complex conditions as you like. >> Instead of creating many limited shortcut syntax variants for special >> cases. >> > yeah =) in future we can go back and do some job here > > >> >> > i'm not using the threadpool yet and i don't know how processlist is >> > reported with thread pool, is the id isn't unique in this case (using >> > threadpool)? >> >> Unique. Every connection has its own connection id, with or without >> thread pool. Internal scheduling implementation doesn't affect >> user-visible connection ids. >> > nice i will test thread pool and check how it works > > >> >> > i don't know if it's what mariadb/mysql should do inside code when using >> > threadpool, i'm using only one process per connection, and don't have >> this >> > kind of problem >> >> threadpool doesn't have this problem either. >> >> > other doubt now... when we have a daemon process (plugin) there's a >> > query id for it? >> >> No. >> >> > query_id=0? in this case we only have an "unique query id" with >> > thread_id+query_id? maybe we should avoid the KILL QUERY_ID = 0 >> >> Right, thanks. >> >> > again about sintax... maybe a WHERE could be added to KILL instead of a >> > DELETE FROM INFORMATION_SCHEMA... >> > >> > KILL [CONNECTION | QUERY] [WHERE some_fields some_operators some_values >> > and_no_subquery | <thread_id> | QUERY_ID <query_id>] >> > >> > about WHERE, we could use the same fields of show processlist: >> > >> > ID, USER, HOST, DB, COMMAND, TIME, STATE, INFO, TIME_MS, STAGE, >> > MAX_STAGE, PROGRESS, MEMORY_USED, EXAMINED_ROWS, QUERY_ID >> >> I'd rather allow subqueries there, it'll be much more natural. >> > yes maybe information schema engine, or kill subquery? > > >> >> Regards, >> Sergei >> > > bye :) > -- Roberto Spadim SPAEmpresarial

1 0

[Maria-developers] Review of patch for MDEV-4820
by Kristian Nielsen 16 Sep '13

16 Sep '13

I took a close look at your patch for MDEV-4820. I think there is a fundamental disconnect. In MariaDB GTID, I do not require or rely on monotonically increasing seqeunce numbers (monoticity is requred per-server-id, but not between different servers). Nor do I enforce or rely on absence of holes in the sequence numbers. This decision was a hard one to make and I spent considerable thought on this point quite early. It is true that this design reduces possibilities to detect some kinds of errors, like missing events and alternate futures. I can understand if this design is not optimal for what you are trying to do. However, implementing two different designs (eg. based on value of gtid_strict_mode) is not workable. I believe at least for the first version, the current design is what we have to work with. The gtid_strict_mode is about helping the user by giving an error instead of logging certain undeirable event sequences to the binlog. It should not cause different non-error behaviour ever, nor should it prevent working with binlogs that were logged when strict mode was not enabled. > diff --git a/sql/rpl_gtid.cc b/sql/rpl_gtid.cc > --- a/sql/rpl_gtid.cc > +++ b/sql/rpl_gtid.cc > @@ -993,7 +993,7 @@ rpl_binlog_state::check_strict_sequence(uint32 domain_id, uint32 server_id, > > if ((elem= (element *)my_hash_search(&hash, > (const uchar *)(&domain_id), 0)) && > - elem->last_gtid && elem->last_gtid->seq_no >= seq_no) > + elem->last_gtid && elem->last_gtid->seq_no + 1 != seq_no) If I understand this part correctly, this is about enforcing that there are no holes in sequence numbers. This goes against the GTID design. > @@ -854,8 +857,16 @@ contains_all_slave_gtid(slave_connection_state *st, Gtid_list_log_event *glev) > */ > return false; > } > - if (gtid->server_id == glev->list[i].server_id && > - gtid->seq_no <= glev->list[i].seq_no) > + if (slave_gtid_strict_mode) > + { > + if (gtid->seq_no < glev->list[i].seq_no) > + { > + has_greater_seq_no= true; > + break; > + } > + } I think here you want to depend on sequence numbers being monotonic. This does not work, as you cannot be sure that strict mode was enabled when this binlog file was written. > - if (!rpl_global_gtid_slave_state.domain_to_gtid(slave_gtid->domain_id, > - &master_replication_gtid) || > + bool master_repl_gtid_found = > + rpl_global_gtid_slave_state.domain_to_gtid(slave_gtid->domain_id, > + &master_replication_gtid); > + if (!master_repl_gtid_found || > slave_gtid->server_id != master_replication_gtid.server_id || > slave_gtid->seq_no != master_replication_gtid.seq_no) > { Right, so slave requests something in a domain that we do not have anything about in the binlog, but we do have something in the slave state (but not the correct one). So this is a bug in the current code. If slave requests to start at a position in a domain we have nothing in, either that domain is served by a different master, or slave wants to start from the very first event in that domain. Currently, we fail to check in the second case that we actually have the correct GTID as the first event. But this is the wrong way to fix it. A master should not be prevented from serving some domain D1 just because it has some old junk for another domain D2 in the slave state. Instead, when we allow to start in a domain that is missing, and then later actually encounter an event in this domain, we should check that it matches what the slave originally requested. If not, we should give an error. I'll try to find time to do this. > @@ -1191,6 +1239,16 @@ gtid_find_binlog_file(slave_connection_state *state, char *out_name, > */ > state->remove(gtid); > } > + else if (slave_gtid_strict_mode && > + gtid->seq_no == glev->list[i].seq_no) > + { > + *errormsg= "Requested slave GTID state have different server id " > + "than the one in binlog"; > + *error_gtid= *gtid; > + *good_gtid= glev->list[i]; > + error= ER_SLAVE_IS_FROM_ALTERNATE_FUTURE; > + goto end; > + } This can fail if the binlog file was written when strict mode was not enabled. > > @@ -1559,7 +1620,39 @@ send_event_to_slave(THD *thd, NET *net, String* const packet, ushort flags, > event_gtid.seq_no <= gtid->seq_no) > *gtid_skip_group = (flags2 & Gtid_log_event::FL_STANDALONE ? > GTID_SKIP_STANDALONE : GTID_SKIP_TRANSACTION); > - if (event_gtid.server_id == gtid->server_id && > + if (slave_gtid_strict_mode) > + { > + if (event_gtid.seq_no == gtid->seq_no && > + event_gtid.server_id != gtid->server_id) > + { > + my_errno= ER_SLAVE_IS_FROM_ALTERNATE_FUTURE; > + *error_gtid= *gtid; > + *good_gtid= event_gtid; > + return "Requested slave GTID state have different server id " > + "than the one in binlog"; > + } Again, what if this is an old part of the binlog where strict mode was disabled? > + else if (*first_event && event_gtid.seq_no > gtid->seq_no) > + { > + /* > + Earlier we decided that this binlog file has event that we need, > + but the first event in the file has seq_no beyond what we need. > + This can happen in two cases: > + - this file is the first binlog we have and Gtid_list event > + in it is empty; > + - there is a gap between the last event mentioned in Gtid_list > + and the first event in the binlog file. > + As the first reason is the most probable one let's issue the > + appropriate error message. > + */ > + my_errno= ER_MASTER_FATAL_ERROR_READING_BINLOG; > + *error_gtid= *gtid; > + return "Could not find GTID state requested by slave in any " > + "binlog files. Probably the slave state is too old and " > + "required binlog files have been purged"; > + } > + } This "first_event" (which I admit I do not fully understand) looks wrong - all such things usually need to be per-domain. However, I think this is the place in the code I mention above, where we get an event in a domain that the slave requested, but we did not have anything at connect time. So here I want an even stricter check, and also in non-strict mode: The first GTID in the domain must be _equal_ to what the slave requested, or we should fail. ---- Next I will take a closer look at your test cases and see what can be done to fix it in a way compatible with the basic design. I also want to re-iterate the suggestion to not delete binlogs when cloning a master to a new master. The most recent binlog file has real information (in the initial Gtid_list event) that is needed for proper operation as a master, and it really doesn't make sense to first purposely corrupt the server state and subsequently try to hack the code to fix the damage. If one really wanted not to copy any binlog files, then a better idea would be a way to restore the state, eg. a RESET MASTER TO gtid_list='xxx,yyy,...' command or something like that. Hope this helps, - Kristian.

3 23

[Maria-developers] review of mdev-4911
by Sergei Golubchik 14 Sep '13

14 Sep '13

Hi. Good work. I only have one small comment about the code, see below. And one about the syntax. > === modified file 'sql/sql_parse.cc' > --- sql/sql_parse.cc 2013-08-14 08:48:50 +0000 > +++ sql/sql_parse.cc 2013-09-11 12:29:32 +0000 > @@ -7139,24 +7139,60 @@ THD *find_thread_by_id(ulong id) > > > /** > - kill on thread. > + Find a thread by query id and return it, locking it LOCK_thd_data > + > + @param id Identifier of the query we're looking for > + > + @return NULL - not found > + pointer - thread found, and its LOCK_thd_data is locked. > +*/ > + > +static THD *find_thread_by_query_id(longlong id) why didn't you reuse find_thread_by_id()? this function is almost identical to it. > +{ > + THD *tmp; > + mysql_mutex_lock(&LOCK_thread_count); // For unlink from list > + I_List_iterator<THD> it(threads); > + while ((tmp=it++)) > + { > + if (tmp->get_command() == COM_DAEMON) > + continue; > + if (tmp->query_id == id) > + { > + mysql_mutex_lock(&tmp->LOCK_thd_data); // Lock from delete > + break; > + } > + } > + mysql_mutex_unlock(&LOCK_thread_count); > + return tmp; > +} > + > + > === modified file 'sql/sql_yacc.yy' > --- sql/sql_yacc.yy 2013-08-13 11:35:36 +0000 > +++ sql/sql_yacc.yy 2013-09-11 12:29:32 +0000 > @@ -12890,6 +12891,11 @@ kill_expr: > Lex->users_list.push_back($2); > Lex->kill_type= KILL_TYPE_USER; > } > + | ID_SYM expr > + { > + Lex->value_list.push_front($2); > + Lex->kill_type= KILL_TYPE_QUERY; > + } So, you implemented KILL [ CONNECTION | QUERY ] [ ID ] expr It allows, in particular KILL CONNECTION ID 10 which looks strange, why would that mean that 10 is a query id? I originally suggested KILL [ CONNECTION | QUERY [ ID ] ] expr to allow only KILL CONNECTION expr KILL QUERY expr KILL QUERY ID expr because in this case it's quite clear "QUERY ID" and because I thought it's a bit strange to kill a connection by query_id. If you want to allow that (as I saw in the comments, Roberto didn't thought it's strange to kill a connection by query id), may be you'd better use KILL [ CONNECTION | QUERY ] [ QUERY_ID ] expr ? Regards, Sergei

3 5

[Maria-developers] Can't sign up to JIRA
by Kouhei Sutou 13 Sep '13

13 Sep '13

Hi, I tried to sign up to JIRA(*) for submitting a bug report. (*) https://mariadb.atlassian.net/secure/Signup!default.jspa But I couldn't sign up because I couldn't get captcha image. I used "Chromium 28.0.1500.95 Debian jessie/sid (213514)" and "Iceweasel 7.0.8" on Debian GNU/Linux sid as Web browsers. I got the same result with these Web browsers. I checked captcha image response by Chromium's developer tool. https://mariadb.atlassian.net/captcha returns "500 Internal Server Error" with the following body. Could you confirm it? ---- We had some technical problems during your last operation. We deeply apologize for that :( If you need further assistance, please: Expand the section "Technical details" below Copy the content from it Send it to your JIRA Administrator Technical details Cause Referer URL: https://mariadb.atlassian.net/secure/Signup!default.jspa java.lang.NoClassDefFoundError: Could not initialize class com.atlassian.jira.servlet.NoOpImageCaptchaService java.lang.NoClassDefFoundError: Could not initialize class com.atlassian.jira.servlet.NoOpImageCaptchaService at com.atlassian.jira.servlet.NoOpCaptchaServiceImpl.getImageCaptchaService(NoOpCaptchaServiceImpl.java:14) at com.atlassian.jira.servlet.JiraCaptchaServlet.doGet(JiraCaptchaServlet.java:50) at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.jira.web.filters.steps.ChainedFilterStepRunner.doFilter(ChainedFilterStepRunner.java:87) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.jira.web.filters.XContentTypeOptionsNoSniffFilter.doFilter(XContentTypeOptionsNoSniffFilter.java:22) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.jira.tzdetect.IncludeResourcesFilter.doFilter(IncludeResourcesFilter.java:39) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.jira.baseurl.IncludeResourcesFilter.doFilter(IncludeResourcesFilter.java:38) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:25) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.mywork.client.filter.ServingRequestsFilter.doFilter(ServingRequestsFilter.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.studio.core.servlet.filter.AlacarteLicenseEnforcer.doFilter(AlacarteLicenseEnforcer.java:71) at com.atlassian.studio.jira.servlet.filter.AlacarteLicenseJiraEnforcer.doFilter(AlacarteLicenseJiraEnforcer.java:121) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.prettyurls.filter.PrettyUrlsDispatcherFilter.doFilter(PrettyUrlsDispatcherFilter.java:60) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter.doFilter(PrettyUrlsSiteMeshFilter.java:92) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.prettyurls.filter.PrettyUrlsMatcherFilter.doFilter(PrettyUrlsMatcherFilter.java:56) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.labs.botkiller.BotKillerFilter.doFilter(BotKillerFilter.java:36) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.jira.web.filters.accesslog.AccessLogFilter.executeRequest(AccessLogFilter.java:103) at com.atlassian.jira.web.filters.accesslog.AccessLogFilter.doFilter(AccessLogFilter.java:87) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.jira.security.xsrf.XsrfTokenAdditionRequestFilter.doFilter(XsrfTokenAdditionRequestFilter.java:54) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.jira.web.filters.steps.ChainedFilterStepRunner.doFilter(ChainedFilterStepRunner.java:87) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.plugin.remotable.plugin.module.permission.ApiScopingFilter.doFilter(ApiScopingFilter.java:63) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:234) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:98) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:169) at com.atlassian.jira.web.filters.JiraLoginFilter.doFilter(JiraLoginFilter.java:70) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:55) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176) at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145) at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.gzipfilter.GzipFilter.doFilterInternal(GzipFilter.java:80) at com.atlassian.gzipfilter.GzipFilter.doFilter(GzipFilter.java:51) at com.atlassian.jira.web.filters.gzip.JiraGzipFilter.doFilter(JiraGzipFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.plugin.remotable.plugin.module.oauth.OAuth2LOFilter.doFilter(OAuth2LOFilter.java:81) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.plugin.remotable.host.common.service.http.bigpipe.BigPipeRequestIdFilter.doFilter(BigPipeRequestIdFilter.java:33) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.analytics.client.filter.AnalyticsFilter.doFilter(AnalyticsFilter.java:28) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66) at com.sysbliss.jira.plugins.workflow.servlet.JWDSendRedirectFilter.doFilter(JWDSendRedirectFilter.java:25) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.jira.web.filters.steps.ChainedFilterStepRunner.doFilter(ChainedFilterStepRunner.java:87) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.core.filters.cache.AbstractCachingFilter.doFilter(AbstractCachingFilter.java:33) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.core.filters.encoding.AbstractEncodingFilter.doFilter(AbstractEncodingFilter.java:41) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at com.atlassian.jira.web.filters.PathMatchingEncodingFilter.doFilter(PathMatchingEncodingFilter.java:49) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.jira.startup.JiraStartupChecklistFilter.doFilter(JiraStartupChecklistFilter.java:78) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.jira.web.filters.steps.ChainedFilterStepRunner.doFilter(ChainedFilterStepRunner.java:87) at com.atlassian.jira.web.filters.JiraFirstFilter.doFilter(JiraFirstFilter.java:57) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:776) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:705) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) at java.lang.Thread.run(Thread.java:662) Go back to the previous page ---- Thanks, -- kou

4 8

Re: [Maria-developers] [Commits] Rev 3810: MDEV-4994: Crash in dynamic column API in file:///home/bell/maria/bzr/work-maria-10.0-MDEV-4994/
by Kristian Nielsen 13 Sep '13

13 Sep '13

sanja(a)askmonty.org writes: > At file:///home/bell/maria/bzr/work-maria-10.0-MDEV-4994/ > > ------------------------------------------------------------ > revno: 3810 > revision-id: sanja(a)askmonty.org-20130911151436-rjz6pw6fh6rtj83e > parent: svoj(a)mariadb.org-20130904072837-u3zj89fq4xvb4cnp > committer: sanja(a)askmonty.org > branch nick: work-maria-10.0-MDEV-4994 > timestamp: Wed 2013-09-11 18:14:36 +0300 > message: > MDEV-4994: Crash in dynamic column API > Dynamic columns unittest fix. > > Correct processing of empty string added to creation dynamic string procedure > List function of numeric format fixed when empty string passed. > Unit test calls changed to new ones. > Unit test added to cmake file. > === modified file 'mysys/ma_dyncol.c' > --- a/mysys/ma_dyncol.c 2013-08-07 15:40:25 +0000 > +++ b/mysys/ma_dyncol.c 2013-09-11 15:14:36 +0000 > @@ -1618,15 +1618,22 @@ dynamic_new_column_store(DYNAMIC_COLUMN > > if (!(columns_order= malloc(sizeof(void*)*column_count))) > return ER_DYNCOL_RESOURCE; > - if (new_str) > + if (new_str || str->str == 0) > { > - if (dynamic_column_init_named(str, > - fmt->fixed_hdr + > - hdr->header_size + > - hdr->nmpool_size + > - hdr->data_size + > - DYNCOL_SYZERESERVE)) > - goto err; > + if (column_count) > + { > + if (dynamic_column_init_named(str, > + fmt->fixed_hdr + > + hdr->header_size + > + hdr->nmpool_size + > + hdr->data_size + > + DYNCOL_SYZERESERVE)) > + goto err; > + } > + else > + { > + dynamic_column_initialize(str); > + } > } > else > { > @@ -2438,7 +2445,8 @@ mariadb_dyncol_list(DYNAMIC_COLUMN *str, > uint i; > enum enum_dyncol_func_result rc; > > - (*nums)= 0; /* In case of errors */ > + (*nums)= 0; (*count)= 0; /* In case of errors */ > + > if (str->length == 0) > return ER_DYNCOL_OK; /* no columns */ > > > === modified file 'unittest/mysys/CMakeLists.txt' > --- a/unittest/mysys/CMakeLists.txt 2013-07-16 17:09:54 +0000 > +++ b/unittest/mysys/CMakeLists.txt 2013-09-11 15:14:36 +0000 > @@ -20,6 +20,9 @@ INCLUDE_DIRECTORIES(${CMAKE_SOURCE_DIR}/ > MY_ADD_TESTS(bitmap base64 my_vsnprintf my_atomic my_rdtsc lf my_malloc > LINK_LIBRARIES mysys) > > +MY_ADD_TESTS(ma_dyncol > + LINK_LIBRARIES mysqlclient) > + > IF(WIN32) > MY_ADD_TESTS(my_delete LINK_LIBRARIES mysys) > ENDIF() > > === modified file 'unittest/mysys/ma_dyncol-t.c' > --- a/unittest/mysys/ma_dyncol-t.c 2012-07-31 17:29:07 +0000 > +++ b/unittest/mysys/ma_dyncol-t.c 2013-09-11 15:14:36 +0000 > @@ -35,17 +35,18 @@ > void test_value_single_null() > { > int rc= FALSE; > + uint ids[1]= {1}; > DYNAMIC_COLUMN_VALUE val, res; > DYNAMIC_COLUMN str; > /* init values */ > val.type= DYN_COL_NULL; > - dynamic_column_value_init(&res); > + mariadb_dyncol_value_init(&res); > /* create column */ > - if (dynamic_column_create(&str, 1, &val)) > + if (mariadb_dyncol_create_many(&str, 1, ids, &val, 1)) > goto err; > dynstr_append(&str, "\1"); str.length--; //check for overflow > /* read column */ > - if (dynamic_column_get(&str, 1, &res)) > + if (mariadb_dyncol_get(&str, 1, &res)) > goto err; > rc= (res.type == DYN_COL_NULL); > err: > @@ -57,18 +58,19 @@ err: > void test_value_single_uint(ulonglong num, const char *name) > { > int rc= FALSE; > + uint ids[1]= {1}; > DYNAMIC_COLUMN_VALUE val, res; > DYNAMIC_COLUMN str; > /* init values */ > val.type= DYN_COL_UINT; > val.x.ulong_value= num; > - dynamic_column_value_init(&res); > + mariadb_dyncol_value_init(&res); > /* create column */ > - if (dynamic_column_create(&str, 1, &val)) > + if (mariadb_dyncol_create_many(&str, 1, ids, &val, 1)) > goto err; > dynstr_append(&str, "\1"); str.length--; //check for overflow > /* read column */ > - if (dynamic_column_get(&str, 1, &res)) > + if (mariadb_dyncol_get(&str, 1, &res)) > goto err; > rc= (res.type == DYN_COL_UINT) && (res.x.ulong_value == num); > num= res.x.ulong_value; > @@ -81,18 +83,19 @@ err: > void test_value_single_sint(longlong num, const char *name) > { > int rc= FALSE; > + uint ids[1]= {1}; > DYNAMIC_COLUMN_VALUE val, res; > DYNAMIC_COLUMN str; > /* init values */ > val.type= DYN_COL_INT; > val.x.long_value= num; > - dynamic_column_value_init(&res); > + mariadb_dyncol_value_init(&res); > /* create column */ > - if (dynamic_column_create(&str, 1, &val)) > + if (mariadb_dyncol_create_many(&str, 1, ids, &val, 1)) > goto err; > dynstr_append(&str, "\1"); str.length--; //check for overflow > /* read column */ > - if (dynamic_column_get(&str, 1, &res)) > + if (mariadb_dyncol_get(&str, 1, &res)) > goto err; > rc= (res.type == DYN_COL_INT) && (res.x.long_value == num); > num= res.x.ulong_value; > @@ -106,18 +109,19 @@ err: > void test_value_single_double(double num, const char *name) > { > int rc= FALSE; > + uint ids[1]= {1}; > DYNAMIC_COLUMN_VALUE val, res; > DYNAMIC_COLUMN str; > /* init values */ > val.type= DYN_COL_DOUBLE; > val.x.double_value= num; > - dynamic_column_value_init(&res); > + mariadb_dyncol_value_init(&res); > /* create column */ > - if (dynamic_column_create(&str, 1, &val)) > + if (mariadb_dyncol_create_many(&str, 1, ids, &val, 1)) > goto err; > dynstr_append(&str, "\1"); str.length--; //check for overflow > /* read column */ > - if (dynamic_column_get(&str, 1, &res)) > + if (mariadb_dyncol_get(&str, 1, &res)) > goto err; > rc= (res.type == DYN_COL_DOUBLE) && (res.x.double_value == num); > num= res.x.ulong_value; > @@ -133,21 +137,22 @@ void test_value_single_decimal(const cha > char buff[80]; > int rc= FALSE; > int length= 80; > + uint ids[1]= {1}; > DYNAMIC_COLUMN_VALUE val, res; > DYNAMIC_COLUMN str; > > /* init values */ > - dynamic_column_prepare_decimal(&val); // special procedure for decimal!!! > + mariadb_dyncol_prepare_decimal(&val); // special procedure for decimal!!! > if (string2decimal(num, &val.x.decimal.value, &end) != E_DEC_OK) > goto err; > - dynamic_column_value_init(&res); > + mariadb_dyncol_value_init(&res); > > /* create column */ > - if (dynamic_column_create(&str, 1, &val)) > + if (mariadb_dyncol_create_many(&str, 1, ids, &val, 1)) > goto err; > dynstr_append(&str, "\1"); str.length--; //check for overflow > /* read column */ > - if (dynamic_column_get(&str, 1, &res)) > + if (mariadb_dyncol_get(&str, 1, &res)) > goto err; > rc= ((res.type == DYN_COL_DECIMAL) && > (decimal_cmp(&res.x.decimal.value, &val.x.decimal.value) == 0)); > @@ -206,6 +211,7 @@ void test_value_single_string(const char > CHARSET_INFO *cs) > { > int rc= FALSE; > + uint ids[1]= {1}; > DYNAMIC_COLUMN_VALUE val, res; > DYNAMIC_COLUMN str; > > @@ -214,14 +220,14 @@ void test_value_single_string(const char > val.x.string.value.str= (char*)string; > val.x.string.value.length= len; > val.x.string.charset= cs; > - dynamic_column_value_init(&res); > + mariadb_dyncol_value_init(&res); > > /* create column */ > - if (dynamic_column_create(&str, 1, &val)) > + if (mariadb_dyncol_create_many(&str, 1, ids, &val, 1)) > goto err; > dynstr_append(&str, "\1"); str.length--; //check for overflow > /* read column */ > - if (dynamic_column_get(&str, 1, &res)) > + if (mariadb_dyncol_get(&str, 1, &res)) > goto err; > rc= ((res.type == DYN_COL_STRING) && > (res.x.string.value.length == len) && > @@ -239,6 +245,7 @@ err: > void test_value_single_date(uint year, uint month, uint day, const char *name) > { > int rc= FALSE; > + uint ids[1]= {1}; > DYNAMIC_COLUMN_VALUE val, res; > DYNAMIC_COLUMN str; > /* init values */ > @@ -247,13 +254,13 @@ void test_value_single_date(uint year, u > val.x.time_value.year= year; > val.x.time_value.month= month; > val.x.time_value.day= day; > - dynamic_column_value_init(&res); > + mariadb_dyncol_value_init(&res); > /* create column */ > - if (dynamic_column_create(&str, 1, &val)) > + if (mariadb_dyncol_create_many(&str, 1, ids, &val, 1)) > goto err; > dynstr_append(&str, "\1"); str.length--; //check for overflow > /* read column */ > - if (dynamic_column_get(&str, 1, &res)) > + if (mariadb_dyncol_get(&str, 1, &res)) > goto err; > rc= ((res.type == DYN_COL_DATE) && > (res.x.time_value.time_type == MYSQL_TIMESTAMP_DATE) && > @@ -270,6 +277,7 @@ void test_value_single_time(uint neg, ui > uint mic, const char *name) > { > int rc= FALSE; > + uint ids[1]= {1}; > DYNAMIC_COLUMN_VALUE val, res; > DYNAMIC_COLUMN str; > /* init values */ > @@ -280,13 +288,13 @@ void test_value_single_time(uint neg, ui > val.x.time_value.minute= minute; > val.x.time_value.second= second; > val.x.time_value.second_part= mic; > - dynamic_column_value_init(&res); > + mariadb_dyncol_value_init(&res); > /* create column */ > - if (dynamic_column_create(&str, 1, &val)) > + if (mariadb_dyncol_create_many(&str, 1, ids, &val, 1)) > goto err; > dynstr_append(&str, "\1"); str.length--; //check for overflow > /* read column */ > - if (dynamic_column_get(&str, 1, &res)) > + if (mariadb_dyncol_get(&str, 1, &res)) > goto err; > rc= ((res.type == DYN_COL_TIME) && > (res.x.time_value.time_type == MYSQL_TIMESTAMP_TIME) && > @@ -308,6 +316,7 @@ void test_value_single_datetime(uint neg > uint mic, const char *name) > { > int rc= FALSE; > + uint ids[1]= {1}; > DYNAMIC_COLUMN_VALUE val, res; > DYNAMIC_COLUMN str; > /* init values */ > @@ -321,13 +330,13 @@ void test_value_single_datetime(uint neg > val.x.time_value.minute= minute; > val.x.time_value.second= second; > val.x.time_value.second_part= mic; > - dynamic_column_value_init(&res); > + mariadb_dyncol_value_init(&res); > /* create column */ > - if (dynamic_column_create(&str, 1, &val)) > + if (mariadb_dyncol_create_many(&str, 1, ids, &val, 1)) > goto err; > dynstr_append(&str, "\1"); str.length--; //check for overflow > /* read column */ > - if (dynamic_column_get(&str, 1, &res)) > + if (mariadb_dyncol_get(&str, 1, &res)) > goto err; > rc= ((res.type == DYN_COL_DATETIME) && > (res.x.time_value.time_type == MYSQL_TIMESTAMP_DATETIME) && > @@ -373,7 +382,7 @@ void test_value_multi(ulonglong num0, > val[1].x.long_value= num1; > val[2].type= DYN_COL_DOUBLE; > val[2].x.double_value= num2; > - dynamic_column_prepare_decimal(val + 3); // special procedure for decimal!!! > + mariadb_dyncol_prepare_decimal(val + 3); // special procedure for decimal!!! > if (string2decimal(num3, &val[3].x.decimal.value, &end3) != E_DEC_OK) > goto err; > val[4].type= DYN_COL_STRING; > @@ -404,14 +413,14 @@ void test_value_multi(ulonglong num0, > val[7].x.time_value.second_part= mic7; > val[8].type= DYN_COL_NULL; > for (i= 0; i < 9; i++) > - dynamic_column_value_init(res + i); > + mariadb_dyncol_value_init(res + i); > /* create column */ > - if (dynamic_column_create_many(&str, 9, column_numbers, val)) > + if (mariadb_dyncol_create_many(&str, 9, column_numbers, val, 1)) > goto err; > dynstr_append(&str, "\1"); str.length--; //check for overflow > /* read column */ > for (i= 0; i < 9; i++) > - if (dynamic_column_get(&str, column_numbers[i], res + i)) > + if (mariadb_dyncol_get(&str, column_numbers[i], res + i)) > goto err; > rc= ((res[0].type == DYN_COL_UINT) && > (res[0].x.ulong_value == num0) && > @@ -467,7 +476,7 @@ void test_value_multi_same_num() > for (i= 0; i < 5; i++) > val[i].type= DYN_COL_NULL; > /* create column */ > - if (!dynamic_column_create_many(&str, 5, column_numbers, val)) > + if (!mariadb_dyncol_create_many(&str, 5, column_numbers, val, 1)) > goto err; > rc= TRUE; > err: > @@ -487,68 +496,69 @@ void test_update_multi(uint *column_numb > > val.type= DYN_COL_UINT; > val.x.ulong_value= column_values[0]; > - if (dynamic_column_create(&str, column_numbers[0], &val)) > + if (mariadb_dyncol_create_many(&str, 1, column_numbers, &val, 1)) > goto err; > for (i= 1; i < all; i++) > { > val.type= (null_values[i] ? DYN_COL_NULL : DYN_COL_UINT); > val.x.ulong_value= column_values[i]; > - if (dynamic_column_update(&str, column_numbers[i], &val)) > + if (mariadb_dyncol_update_many(&str, 1, column_numbers +i, &val)) > goto err; > > /* check value(s) */ > for (j= i; j >= (i < only_add ? 0 : i); j--) > { > - if (dynamic_column_get(&str, column_numbers[j], &val)) > + if (mariadb_dyncol_get(&str, column_numbers[j], &val)) > goto err; > if (null_values[j]) > { > if (val.type != DYN_COL_NULL || > - dynamic_column_exists(&str, column_numbers[j]) == ER_DYNCOL_YES) > + mariadb_dyncol_exists(&str, column_numbers[j]) == ER_DYNCOL_YES) > goto err; > } > else > { > if (val.type != DYN_COL_UINT || > val.x.ulong_value != column_values[j] || > - dynamic_column_exists(&str, column_numbers[j]) == ER_DYNCOL_NO) > + mariadb_dyncol_exists(&str, column_numbers[j]) == ER_DYNCOL_NO) > goto err; > } > } > if (i < only_add) > { > - DYNAMIC_ARRAY num; > - if (dynamic_column_list(&str, &num)) > + uint elements, *num; > + if (mariadb_dyncol_list(&str, &elements, &num)) > + { > + my_free(num); > goto err; > + } > /* cross check arrays */ > - if ((int)num.elements != i + 1) > + if ((int)elements != i + 1) > { > - delete_dynamic(&num); > + my_free(num); > goto err; > } > for(j= 0; j { > int k; > for(k= 0; > - k - *dynamic_element(&num, k, uint*); > + k k++); > if (k >= i + 1) > { > - delete_dynamic(&num); > + my_free(num); > goto err; > } > for(k= 0; > - k - *dynamic_element(&num, j, uint*); > + k k++); > if (k >= i + 1) > { > - delete_dynamic(&num); > + my_free(num); > goto err; > } > } > - delete_dynamic(&num); > + my_free(num); > } > } > > @@ -563,32 +573,52 @@ void test_empty_string() > { > DYNAMIC_COLUMN_VALUE val, res; > DYNAMIC_COLUMN str; > - DYNAMIC_ARRAY array_of_uint; > + uint *array_of_uint; > + uint number_of_uint; > int rc; > + uint ids[1]= {1}; > + DYNAMIC_COLUMN_VALUE vals[0]; > /* empty string */ > bzero(&str, sizeof(str)); > > - rc= dynamic_column_get(&str, 1, &res); > + rc= mariadb_dyncol_get(&str, 1, &res); > ok( (rc == ER_DYNCOL_OK) && (res.type == DYN_COL_NULL), "%s", "empty get"); > > - rc= dynamic_column_delete(&str, 1); > - ok( (rc == ER_DYNCOL_OK), "%s", "empty delete"); > + vals[0].type= DYN_COL_NULL; > + rc= mariadb_dyncol_update_many(&str, 1, ids, vals); > + ok( (rc == ER_DYNCOL_OK) && (str.str == 0), "%s", "empty delete"); > > - rc= dynamic_column_exists(&str, 1); > + rc= mariadb_dyncol_exists(&str, 1); > ok( (rc == ER_DYNCOL_NO), "%s", "empty exists"); > > - rc= dynamic_column_list(&str, &array_of_uint); > - ok( (rc == ER_DYNCOL_OK) && (array_of_uint.elements == 0), > + rc= mariadb_dyncol_list(&str, &number_of_uint, &array_of_uint); > + ok( (rc == ER_DYNCOL_OK) && (number_of_uint == 0) && (str.str == 0), > "%s", "empty list"); > > val.type= DYN_COL_UINT; > val.x.ulong_value= 1212; > - rc= dynamic_column_update(&str, 1, &val); > + rc= mariadb_dyncol_update_many(&str, 1, ids, &val); > if (rc == ER_DYNCOL_OK) > - rc= dynamic_column_get(&str, 1, &res); > - ok( (rc == ER_DYNCOL_OK) && > + rc= mariadb_dyncol_get(&str, 1, &res); > + ok( (rc == ER_DYNCOL_OK) && (str.str != 0) && > (res.type == DYN_COL_UINT) && (res.x.ulong_value == val.x.ulong_value), > "%s", "empty update"); > + dynamic_column_column_free(&str); > +} > + > +void test_mdev_4994() > +{ > + DYNAMIC_COLUMN dyncol; > + LEX_STRING key= {0,0}; > + DYNAMIC_COLUMN_VALUE val; > + int rc; > + > + val.type= DYN_COL_NULL; > + > + dynamic_column_initialize(&dyncol); > + rc= mariadb_dyncol_create_many_named(&dyncol, 1, &key, &val, 0); /* crash */ > + ok( (rc == ER_DYNCOL_OK), "%s", "test_mdev_4994"); > + dynamic_column_column_free(&dyncol); > } > > > @@ -633,11 +663,11 @@ void test_update_many(uint *column_numbe > res[i].type= DYN_COL_UINT; > res[i].x.ulong_value= result_values[i]; > } > - if (dynamic_column_create_many(&str1, column_count, column_numbers, val)) > + if (mariadb_dyncol_create_many(&str1, column_count, column_numbers, val, 1)) > goto err; > - if (dynamic_column_update_many(&str1, update_count, update_numbers, upd)) > + if (mariadb_dyncol_update_many(&str1, update_count, update_numbers, upd)) > goto err; > - if (dynamic_column_create_many(&str2, result_count, result_numbers, res)) > + if (mariadb_dyncol_create_many(&str2, result_count, result_numbers, res, 1)) > goto err; > if (str1.length == str2.length && > memcmp(str1.str, str2.str, str1.length) ==0) > @@ -656,7 +686,7 @@ int main(int argc __attribute__((unused) > char *big_string= (char *)malloc(1024*1024); > > MY_INIT(argv[0]); > - plan(60); > + plan(61); > > if (!big_string) > exit(1); > @@ -664,21 +694,17 @@ int main(int argc __attribute__((unused) > big_string[i]= ('0' + (i % 10)); > test_value_single_null(); > test_value_single_uint(0, "0"); > - test_value_single_uint(ULL(0xffffffffffffffff), "0xffffffffffffffff"); > - test_value_single_uint(ULL(0xaaaaaaaaaaaaaaaa), "0xaaaaaaaaaaaaaaaa"); > - test_value_single_uint(ULL(0x5555555555555555), "0x5555555555555555"); > + test_value_single_uint(0xffffffffffffffffULL, "0xffffffffffffffff"); > + test_value_single_uint(0xaaaaaaaaaaaaaaaaULL, "0xaaaaaaaaaaaaaaaa"); > + test_value_single_uint(0x5555555555555555ULL, "0x5555555555555555"); > test_value_single_uint(27652, "27652"); > test_value_single_sint(0, "0"); > test_value_single_sint(1, "1"); > test_value_single_sint(-1, "-1"); > - test_value_single_sint(LL(0x7fffffffffffffff), > - "0x7fffffffffffffff"); > - test_value_single_sint(LL(0xaaaaaaaaaaaaaaaa), > - "0xaaaaaaaaaaaaaaaa"); > - test_value_single_sint(LL(0x5555555555555555), > - "0x5555555555555555"); > - test_value_single_sint(LL(0x8000000000000000), > - "0x8000000000000000"); > + test_value_single_sint(0x7fffffffffffffffLL, "0x7fffffffffffffff"); > + test_value_single_sint(0xaaaaaaaaaaaaaaaaLL, "0xaaaaaaaaaaaaaaaa"); > + test_value_single_sint(0x5555555555555555LL, "0x5555555555555555"); > + test_value_single_sint(0x8000000000000000LL, "0x8000000000000000"); > test_value_single_double(0.0, "0.0"); > test_value_single_double(1.0, "1.0"); > test_value_single_double(-1.0, "-1.0"); > @@ -735,7 +761,7 @@ int main(int argc __attribute__((unused) > } > { > uint column_numbers[]= {10,1,12,37,4,57,6,76,87}; > - test_value_multi(ULL(0xffffffffffffffff), LL(0x7fffffffffffffff), > + test_value_multi(0xffffffffffffffffULL, 0x7fffffffffffffffLL, > 99999999.999e120, "9999999999999999999999999999999", > big_string, 1024*1024, charset_list[0], > 9999, 12, 31, > @@ -791,5 +817,8 @@ int main(int argc __attribute__((unused) > update_numbers, update_values, update_nulls, 4, > result_numbers, result_values, 3); > } > + test_mdev_4994(); > + > + my_end(0); > return exit_status(); > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 1

[Maria-developers] Can there be a better storage engine API?
by MARK CALLAGHAN 13 Sep '13

13 Sep '13

The storage engine API has never been my favorite part of MySQL. I haven't written a storage engine but I used to maintain a custom storage engine and migrated that from 4.0 to 5.0 so I had some experience with the API. Is there a different API that would make it easier for innovation to come to MySQL/MariaDB? For example there is a lot going on in the write-optimized database space. Until recently we had nothing for MySQL open-source. Now we have open source TokuDB (great news). I haven't read their source but I have been told they use something that was derived from the BerkeleyDB API. Would the BerkeleyDB API make it easier to get new storage engines into MySQL? For example, we don't have an engine optimized for in-memory workloads. And while InnoDB will have great cache hit rates when all data is in-memory. It might write much more data to disk than required. Do we need a better API or is the current code good enough? I hope to hear from people who have done serious storage engine work. -- Mark Callaghan mdcallag(a)gmail.com

7 10