December 2013 - developers - lists.mariadb.org

[Maria-developers] RQG crashes tokudb due to unexpected storage engine API call sequence
by Rich Prohaska 05 Dec '13

05 Dec '13

We are debugging a crash induced by RQG in the tokudb storage engine. The cause of the crash is a mismatch in assumptions about the sequence of storage engine API calls. We see: tokudb::external_lock creates a txn tokudb::index_init creates a cursor on a index db with the txn tokudb::index_read use the cursor to try to read a row, and returns an error tokudb_commit commits the txn (called from sql_parse.cc:4584) tokudb::index_end tries to close the cursor and crashes since the txn has been committed. (called from the unit.cleanup() call at sql_parse.cc:4592) The current tokudb software does not expect the commit to occur before index_end is called. Is this assumption correct? Since mariadb (and i suspect mysql) does this, it appears that the storage engine should expect commit before index_end. Any comments? Thanks Rich Prohaska

2 2

[Maria-developers] MDEV-5345 - Deadlock between mysql_change_user(), SHOW VARIABLES and INSTALL PLUGIN
by Sergey Vojtovich 04 Dec '13

04 Dec '13

Hi Sergei, I'm afraid I couldn't find easy fix for this deadlock. The question is if it is worth to spend time for more complex (and possibly less stable) solution. Details below. Locking profile --------------- INSTALL PLUGIN: LOCK_plugin -> LOCK_system_variables_hash (w) SHOW VARIABLES: LOCK_system_variables_hash (r) -> LOCK_global_system_variables change_user(): LOCK_global_system_variables -> LOCK_plugin INSTALL PLUGIN -------------- Needs LOCK_plugin to register new plugin. Needs LOCK_system_variables_hash to register plugin variables. Doesn't seem to need both locks at the same time. Similar lock order is used in a few places. SHOW VARIABLES -------------- Needs LOCK_system_variables_hash to iterate system_variable_hash. Needs LOCK_global_system_variables to read variable value. Does seem to need both locks at the same time. This lock order is not used in other places. change_user() ------------- Needs LOCK_global_system_variables to read global_system_variable.table_plugin. Needs LOCK_plugin to my_plugin_lock(table_plugin). Does seem to need both locks at the same time. Similar lock order is used in a few places. "SHOW VARIABLES" looks correct. So I have only two ideas: - Fix INSTALL PLUGIN so it releases LOCK_plugin before registering variables. Sounds like the best solution, but there are a few more things to fix, e.g. UNINTALL PLUGIN. - Store default storage engine as a string, not Sys_var_plugin. Thanks, Sergey

2 3

[Maria-developers] Build fail of MariaDB10 with TokuDB enabled
by darx＠sent.com 03 Dec '13

03 Dec '13

On the (eventual) road to successfully building MariaDB10 + TokuDB, Building cd src/mariadb bzr log | head ------------------------------------------------------------ revno: 3911 committer: Alexander Barkov <bar(a)mnogosearch.org> branch nick: maria-10.0 timestamp: Tue 2013-11-26 10:53:21 +0400 message: Fixing malformed data in mysql-test/std_data/Index.xml ------------------------------------------------------------ revno: 3910 committer: Sergey Vojtovich <svoj(a)mariadb.org> on uname -a Linux dev1 3.7.10-1.16-desktop #1 SMP PREEMPT Fri May 31 20:21:23 UTC 2013 (97c14ba) x86_64 x86_64 x86_64 GNU/Linux lsb_release -ar LSB Version: core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch Distributor ID: openSUSE project Description: openSUSE 12.3 (x86_64) Release: 12.3 Codename: Dartmouth with gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib64/gcc/x86_64-suse-linux/4.8/lto-wrapper Target: x86_64-suse-linux Configured with: ../configure --prefix=/usr --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,java,ada --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.8 --enable-ssp --disable-libssp --disable-plugin --with-bugurl=http://bugs.opensuse.org/ --with-pkgversion='SUSE Linux' --disable-libgcj --disable-libmudflap --with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit --enable-libstdcxx-allocator=new --disable-libstdcxx-pch --enable-version-specific-runtime-libs --enable-linker-build-id --enable-linux-futex --program-suffix=-4.8 --without-system-libunwind --with-arch-32=i586 --with-tune=generic --build=x86_64-suse-linux --host=x86_64-suse-linux Thread model: posix gcc version 4.8.2 20131016 [gcc-4_8-branch revision 203692] (SUSE Linux) cmake --version cmake version 2.8.12.1 applying a needed src patch cd /usr/local/src/mariadb # BUG @ http://www.linuxfromscratch.org/blfs/view/svn/server/mariadb.html sed -i "s/srv_buf_size/srv_sort_buf_size/" storage/innobase/row/row0log.cc after cmake, with gcc c++11 support NOT enabled, mkdir bld cd bld cmake .. \ ... \ -DWITHOUT_TOKUDB=1 build completes successfully, make ... ldd sql/mysqld linux-vdso.so.1 (0x00007fffb9dff000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9031e86000) libaio.so.1 => /lib64/libaio.so.1 (0x00007f9031c84000) libz.so.1 => /lib64/libz.so.1 (0x00007f9031a6e000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f9031832000) libssl.so.1.0.0 => /usr/local/ssl/lib64/libssl.so.1.0.0 (0x00007f90315c6000) libcrypto.so.1.0.0 => /usr/local/ssl/lib64/libcrypto.so.1.0.0 (0x00007f90311da000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f9030fd6000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f9030cce000) libm.so.6 => /lib64/libm.so.6 (0x00007f90309d0000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f90307b9000) libc.so.6 => /lib64/libc.so.6 (0x00007f903040b000) /lib64/ld-linux-x86-64.so.2 (0x00007f90320a2000) repeating, but with TokuDB build enabled, cd bld cmake .. \ ... \ - -DWITHOUT_TOKUDB=1 + -DWITHOUT_TOKUDB=0 build now fails, make ... [ 29%] Built target perfschema Scanning dependencies of target tokuportability_static_conv [ 29%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/huge_page_detection.cc.o [ 29%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/file.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/memory.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/os_malloc.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/portability.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_assert.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_crash.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_fair_rwlock.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_path.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_pthread.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_time.cc.o Linking CXX static library libtokuportability_static_conv.a [ 30%] Built target tokuportability_static_conv [ 30%] Generating tokuportability_static_depends.cc Scanning dependencies of target tokuportability_static [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static.dir/tokuportability_static_depends.cc.o Linking CXX static library libtokuportability_static.a /usr/bin/ar: creating /usr/local/src/mariadb/bld/storage/tokudb/ft-index/portability/libtokuportability_static.a [ 30%] Built target tokuportability_static Scanning dependencies of target tokuportability [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/huge_page_detection.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/file.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/memory.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/os_malloc.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/portability.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_assert.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_crash.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_fair_rwlock.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_path.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_pthread.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_time.cc.o Linking CXX shared library libtokuportability.so [ 31%] Built target tokuportability Scanning dependencies of target logformat [ 31%] Building CXX object storage/tokudb/ft-index/ft/CMakeFiles/logformat.dir/logformat.cc.o Linking CXX executable logformat [ 31%] Built target logformat Scanning dependencies of target generate_log_code [ 31%] Generating log_code.cc, log_print.cc, log_header.h ./logformat: error while loading shared libraries: libtokuportability.so: cannot open shared object file: No such file or directory make[2]: *** [storage/tokudb/ft-index/ft/log_code.cc] Error 127 make[1]: *** [storage/tokudb/ft-index/ft/CMakeFiles/generate_log_code.dir/all] Error 2 make: *** [all] Error 2 yes, I'm aware that opensuse is not a currently "supported platform" for TokuDB. If that's a dealbreaker, I'd like to know MariaDB project's position, if any, on getting that support in place. If it's not, I'd like to keep trying to get it working, here. darx

4 15

Re: [Maria-developers] MDEV-4956 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.used_tables
by Sergei Golubchik 03 Dec '13

03 Dec '13

Hi, Sergey! On Aug 27, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 3807 > revision-id: svoj(a)mariadb.org-20130827121233-xh1uyhgfwbhedqyf > parent: jplindst(a)mariadb.org-20130823060357-pww92qxla7o8iir7 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Tue 2013-08-27 16:12:33 +0400 > message: > MDEV-4956 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.used_tables > > - tc_acquire_table and tc_release_table do not access > TABLE_SHARE::tdc.used_tables anymore > - in tc_acquire_table(): release LOCK_tdc after we relase LOCK_open > (saves a few CPU cycles in critical section) > - in tc_release_table(): if we reached table cache threshold, evict > to-be-released table without moving it to unused_tables. unused_tables > must be empty at this point (added assertion). I don't understand what you're doing here, could you elaborate? E.g. explain in a changeset comment what you've done, why you introduced a new list for tables (all_share), what is its semantics, etc. Regards, Sergei

3 12

[Maria-developers] Building TOKUDB into 5.5.34 from sources
by md＠rpzdesign.com 02 Dec '13

02 Dec '13

I am trying to build 5.5.34 from source on Linux x64 (Fedora 14). I was looking at the readme files and could not find anything about building 5.5.34 from source and including TokuDB (which has the source code in the /storage/TokuDB/ directory) into the build. I was looking for something like: -DWITH_TOKUDB_STORAGE_ENGINE=ON but did not stumble across anything. Anybody got some ideas? Thanks, Marco Here is my build script: rm -f CMakeCache.txt make clean cmake \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/mysql \ -DENABLED_PROFILING=OFF \ -DENABLE_DEBUG_SYNC=ON \ -DENABLE_GCOV=OFF \ -DINSTALL_LAYOUT=STANDALONE \ -DMYSQL_DATADIR=/mysql/data \ -DINSTALL_INCLUDEDIR=include/mysql \ -DINSTALL_UNIX_ADDRDIR=/mysql/mysql.sock \ -DUSE_ARIA_FOR_TMP_TABLES=ON \ -DWITHOUT_PBXT_STORAGE_ENGINE=ON \ -DWITHOUT_SERVER=OFF \ -DWITH_ARCHIVE_STORAGE_ENGINE=OFF \ -DWITH_EMBEDDED_SERVER=OFF \ -DWITH_FAST_MUTEXES=OFF \ -DWITH_FEDERATEDX_STORAGE_ENGINE=OFF \ -DWITH_FEEDBACK=OFF \ -DWITH_LIBWRAP=OFF \ -DWITH_PIC=ON \ -DWITH_PARTITION_STORAGE_ENGINE=OFF \ -DWITH_PERFSCHEMA_STORAGE_ENGINE=ON \ -DWITH_SPHINX_STORAGE_ENGINE=OFF \ -DWITH_SSL=OFF \ -DWITH_UNIT_TESTS=OFF \ -DWITH_VALGRIND=OFF \ -DWITH_XTRADB_STORAGE_ENGINE=ON \ -DWITHOUT_OQGRAPH=ON \ -DWITH_ZLIB=bundled make (Sorry, I posted this question on maria.com knowledge base open questions also) https://mariadb.com/kb/en/cmake-settings-to-include-tokudb-in-5534/

3 2