- developers - lists.mariadb.org

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2699)
by knielsen＠knielsen-hq.org 20 May '09

20 May '09

#At lp:maria 2699 knielsen(a)knielsen-hq.org 2009-05-11 [merge] Merge Toby Thain's Solaris fixes. modified: BUILD/SETUP.sh BUILD/compile-solaris-amd64 BUILD/compile-solaris-amd64-debug* BUILD/compile-solaris-amd64-forte BUILD/compile-solaris-amd64-forte-debug* storage/innobase/include/univ.i storage/pbxt/src/Makefile.am storage/pbxt/src/lock_xt.h storage/pbxt/src/pbms.h === modified file 'BUILD/SETUP.sh' --- a/BUILD/SETUP.sh 2009-05-06 12:03:24 +0000 +++ b/BUILD/SETUP.sh 2009-05-11 15:31:30 +0000 @@ -80,7 +80,13 @@ path=`dirname $0` . "$path/check-cpu" export AM_MAKEFLAGS -AM_MAKEFLAGS="-j 6" +# Default to a parallel build, but only if AM_MAKEFLAGS is not set. +# (So buildbots can easily disable this behaviour if required.) +if test -z "$AM_MAKEFLAGS" +then + AM_MAKEFLAGS="-j 6" +fi + # SSL library to use.--with-ssl will select our bundled yaSSL # implementation of SSL. To use openSSl you will nee too point out === modified file 'BUILD/compile-solaris-amd64' --- a/BUILD/compile-solaris-amd64 2007-04-12 11:20:38 +0000 +++ b/BUILD/compile-solaris-amd64 2009-05-09 04:01:53 +0000 @@ -1,55 +1,32 @@ -#!/usr/bin/bash +#!/bin/sh -function _find_mysql_root () ( - while [ "x$PWD" != "x/" ]; do - # Check if some directories are present - if [ -d BUILD -a -d sql -a -d mysys ]; then - echo "$PWD" - return 0 - fi - cd .. - done - return 1 -) +# Build setup for Solaris 10 + +# Make the Sun Freeware packages use the bundled Perl, instead of their own: +# ln -s `which perl` /usr/local/bin +# Your $PATH needs to include (in this order): +# /usr/local/bin:/usr/sfw/bin:/usr/ccs/bin +# (For Sun Freeware, bundled GNU utilities, Solaris ar, etc.) +# +# Required packages from http://sunfreeware.com/indexintel10.html : +# (The GNU m4 bundled with Solaris is too old.) +# automake-1.10.2-sol10-x86-local.gz +# autoconf-2.63-sol10-x86-local.gz +# m4-1.4.12-sol10-x86-local.gz +# libsigsegv-2.6-sol10-x86-local.gz +# libtool-1.5.24-sol10-x86-local.gz +# ( how to install these packages: + # wget ftp://ftp.sunfreeware.com/pub/freeware/intel/10/automake-1.10.2-sol10-x86-l… + # gunzip automake-1.10.2-sol10-x86-local.gz + # pkgadd -d automake-1.10.2-sol10-x86-local +# ) -make -k clean || true -/bin/rm -f */.deps/*.P config.cache - path=`dirname $0` -. "$path/autorun.sh" - -warning_flags="-Wimplicit -Wreturn-type -Wswitch -Wtrigraphs -Wcomment -W -Wchar-subscripts -Wformat -Wparentheses -Wsign-compare -Wwrite-strings -Wunused" -compiler_flags="-g -O3 -fno-omit-frame-pointer" +. "$path/SETUP.sh" +extra_flags="$amd64_cflags -D__sun -m64 -mtune=athlon64" +extra_configs="$amd64_configs $max_configs --with-libevent" -export CC CXX CFLAGS CXXFLAGS LDFLAGS LIBS -CC="gcc" -CXX="gcc" -CFLAGS="$warning_flags $compiler_flags" -CXXFLAGS="" -LDFLAGS="-O3 -g -static-libgcc" -LIBS=-lmtmalloc -root=$(_find_mysql_root) +LDFLAGS="-lmtmalloc -static-libgcc" +export LDFLAGS -$root/configure \ - --prefix=/usr/local/mysql \ - --localstatedir=/usr/local/mysql/data \ - --libexecdir=/usr/local/mysql/bin \ - --with-extra-charsets=complex \ - --enable-thread-safe-client \ - --enable-local-infile \ - --with-zlib-dir=bundled \ - --with-big-tables \ - --with-readline \ - --with-archive-storage-engine \ - --with-named-curses=-lcurses \ - --with-big-tables \ - --with-innodb \ - --with-berkeley-db \ - --with-example-storage-engine \ - --with-blackhole-storage-engine \ - --with-ndbcluster \ - --with-federated-storage-engine \ - --with-csv-storage-engine \ - --with-ssl \ - --with-embedded-server \ - --disable-shared +. "$path/FINISH.sh" === modified file 'BUILD/compile-solaris-amd64-debug' (properties changed: -x to +x) --- a/BUILD/compile-solaris-amd64-debug 2007-11-10 10:03:07 +0000 +++ b/BUILD/compile-solaris-amd64-debug 2009-05-09 04:01:53 +0000 @@ -1,10 +1,11 @@ -#! /bin/sh +#!/bin/sh + path=`dirname $0` . "$path/SETUP.sh" -amd64_cflags="-m64 -mtune=athlon64" -extra_flags="$amd64_cflags $debug_cflags $max_cflags" -c_warnings="$c_warnings $debug_extra_warnings" -cxx_warnings="$cxx_warnings $debug_extra_warnings" -extra_configs="$amd64_configs $debug_configs $max_configs --enable-thread-safe-client" +extra_flags="$amd64_cflags -D__sun -m64 -mtune=athlon64 $debug_cflags" +extra_configs="$amd64_configs $debug_configs $max_configs --with-libevent" + +LDFLAGS="-lmtmalloc -static-libgcc" +export LDFLAGS . "$path/FINISH.sh" === modified file 'BUILD/compile-solaris-amd64-forte' --- a/BUILD/compile-solaris-amd64-forte 2008-09-30 20:57:48 +0000 +++ b/BUILD/compile-solaris-amd64-forte 2009-05-09 04:01:53 +0000 @@ -1,53 +1,29 @@ -#! /bin/sh +#!/bin/sh + +# See file compile-solaris-amd64 for basic pre-requisites. + +# This build uses the Sun Studio compilers (cc, CC), available from: +# http://developers.sun.com/sunstudio/downloads/index.jsp +# Note that you may want to apply current patches, as the downloaded version +# is typically out of date. Download the PKG version if you intend to patch! + +# After installing, add /opt/SUNWspro/bin to your $PATH + -gmake -k clean || true -/bin/rm -f */.deps/*.P config.cache - path=`dirname $0` -. "$path/autorun.sh" +. "$path/SETUP.sh" -# For "optimal" code for this computer add -fast to EXTRA -# To compile 64 bit, add -m64 to EXTRA_64_BIT +extra_flags="-m64 -mt -D_FORTEC_ -xbuiltin=%all -xlibmil -xlibmopt -fns=no -xprefetch=auto -xprefetch_level=3" +extra_configs="$max_configs --with-libevent" -EXTRA_64_BIT="-m64" -EXTRA="-fast" +warnings="" +c_warnings="" +cxx_warnings="" +base_cxxflags="-noex" -# -# The following should not need to be touched -# - -export CC CXX CFLAGS CXXFLAGS LIBS -STD="-g -mt -D_FORTEC_ $EXTRA $EXTRA_64_BIT" -ASFLAGS="$EXTRA_64_BIT" -CC=cc-5.0 -CFLAGS="-Xa -xstrconst $STD" +CC=cc +CFLAGS="-xstrconst" CXX=CC -CXXFLAGS="-noex $STD" -LIBS=-lmtmalloc -./configure \ - --prefix=/usr/local/mysql \ - --localstatedir=/usr/local/mysql/data \ - --libexecdir=/usr/local/mysql/bin \ - --with-extra-charsets=complex \ - --enable-thread-safe-client \ - --enable-local-infile \ - --with-zlib-dir=bundled \ - --with-big-tables \ - --with-readline \ - --with-archive-storage-engine \ - --with-named-curses=-lcurses \ - --with-big-tables \ - --with-innodb \ - --with-example-storage-engine \ - --with-blackhole-storage-engine \ - --with-federated-storage-engine \ - --with-csv-storage-engine \ - --with-ssl \ - --enable-assembler - -# Not including: -# --with-ndbcluster -# --with-berkeley-db +LDFLAGS="-lmtmalloc" -gmake -j4 -test $? = 0 && make test +. "$path/FINISH.sh" === modified file 'BUILD/compile-solaris-amd64-forte-debug' (properties changed: -x to +x) --- a/BUILD/compile-solaris-amd64-forte-debug 2008-09-30 20:57:48 +0000 +++ b/BUILD/compile-solaris-amd64-forte-debug 2009-05-09 04:01:53 +0000 @@ -1,54 +1,27 @@ -#! /bin/sh +#!/bin/sh -gmake -k clean || true -/bin/rm -f */.deps/*.P config.cache - path=`dirname $0` -. "$path/autorun.sh" +. "$path/SETUP.sh" -# To compile 64 bit, add -m64 to EXTRA_64_BIT -EXTRA_64_BIT="-m64" +# Take only #define options - the others are gcc specific. +# (real fix is for SETUP.sh not to put gcc specific options in $debug_cflags) +DEFS="" +for F in $debug_cflags ; do + expr "$F" : "^-D" && DEFS="$DEFS $F" +done +debug_cflags="-O0 -g $DEFS" + +extra_flags="-m64 -mt -D_FORTEC_ -xlibmopt -fns=no $debug_cflags" +extra_configs="$max_configs --with-libevent $debug_configs" + +warnings="" +c_warnings="" +cxx_warnings="" +base_cxxflags="-noex" -# For "optimal" code for this computer add -fast to EXTRA. Note that -# this causes problem with debugging the program since -fast implies -# -xO5. -EXTRA="" - -# -# The following should not need to be touched -# - -export CC CXX CFLAGS CXXFLAGS -STD="-g -mt -D_FORTEC_ $EXTRA $EXTRA_64_BIT $debug_cflags" -ASFLAGS="$EXTRA_64_BIT" -CC=cc-5.0 -CFLAGS="-Xa -xstrconst $STD" +CC=cc +CFLAGS="-xstrconst" CXX=CC -CXXFLAGS="-noex $STD" -./configure \ - --prefix=/usr/local/mysql \ - --localstatedir=/usr/local/mysql/data \ - --libexecdir=/usr/local/mysql/bin \ - --with-extra-charsets=complex \ - --enable-thread-safe-client \ - --enable-local-infile \ - --with-zlib-dir=bundled \ - --with-big-tables \ - --with-readline \ - --with-archive-storage-engine \ - --with-named-curses=-lcurses \ - --with-big-tables \ - --with-innodb \ - --with-example-storage-engine \ - --with-blackhole-storage-engine \ - --with-federated-storage-engine \ - --with-csv-storage-engine \ - --with-ssl \ - --with-debug \ - --enable-assembler - -# Not including: -# --with-ndbcluster -# --with-berkeley-db +LDFLAGS="-lmtmalloc" -gmake -j4 +. "$path/FINISH.sh" === modified file 'storage/innobase/include/univ.i' --- a/storage/innobase/include/univ.i 2009-05-06 12:03:24 +0000 +++ b/storage/innobase/include/univ.i 2009-05-11 15:31:30 +0000 @@ -55,10 +55,10 @@ of the 32-bit x86 assembler in mutex ope # define UNIV_CAN_USE_X86_ASSEMBLER # endif -/* We only try to do explicit inlining of functions with gcc and -Microsoft Visual C++ */ +/* Enable explicit inlining of functions only for compilers known to +support it. */ -# if !defined(__GNUC__) +# if !defined(__GNUC__) && !defined(__SUNPRO_C) # undef UNIV_MUST_NOT_INLINE /* Remove compiler warning */ # define UNIV_MUST_NOT_INLINE # endif === modified file 'storage/pbxt/src/Makefile.am' --- a/storage/pbxt/src/Makefile.am 2009-04-01 06:40:11 +0000 +++ b/storage/pbxt/src/Makefile.am 2009-05-09 04:01:53 +0000 @@ -46,7 +46,7 @@ libpbxt_la_CFLAGS = $(AM_CFLAGS) -DMYSQ EXTRA_LIBRARIES = libpbxt.a noinst_LIBRARIES = libpbxt.a libpbxt_a_SOURCES = $(libpbxt_la_SOURCES) -libpbxt_a_CXXFLAGS = $(AM_CXXFLAGS) -Wno-long-long +libpbxt_a_CXXFLAGS = $(AM_CXXFLAGS) libpbxt_a_CFLAGS = $(AM_CFLAGS) -std=c99 EXTRA_DIST = CMakeLists.txt === modified file 'storage/pbxt/src/lock_xt.h' --- a/storage/pbxt/src/lock_xt.h 2009-04-02 10:03:14 +0000 +++ b/storage/pbxt/src/lock_xt.h 2009-05-09 04:01:53 +0000 @@ -208,9 +208,9 @@ inline void xt_atomic_dec2(volatile xtWo #elif defined(__GNUC__) __sync_fetch_and_sub(mptr, 1); #elif defined(XT_SPL_SOLARIS_LIB) - val1 = atomic_dec_16_nv(mptr); + atomic_dec_16_nv(mptr); #else - val1 = --(*mptr); + --(*mptr); #endif } === modified file 'storage/pbxt/src/pbms.h' --- a/storage/pbxt/src/pbms.h 2009-03-26 12:18:01 +0000 +++ b/storage/pbxt/src/pbms.h 2009-05-09 04:01:53 +0000 @@ -782,21 +782,28 @@ private: void deleteTempFiles() { - struct dirent entry; + struct dirent *entry; struct dirent *result; DIR *odir; int err; + size_t sz; char temp_file[100]; +#ifdef XT_SOLARIS + sz = sizeof(struct dirent) + pathconf("/tmp/", _PC_NAME_MAX); // Solaris, see readdir(3C) +#else + sz = sizeof(struct dirent); +#endif + entry = (struct dirent*)malloc(sz); if (!(odir = opendir("/tmp/"))) return; - err = readdir_r(odir, &entry, &result); + err = readdir_r(odir, entry, &result); while (!err && result) { const char **prefix = temp_prefix; while (*prefix) { - if (startsWith(entry.d_name, *prefix)) { - int pid = atoi(entry.d_name + strlen(*prefix)); + if (startsWith(entry->d_name, *prefix)) { + int pid = atoi(entry->d_name + strlen(*prefix)); /* If the process does not exist: */ if (kill(pid, 0) == -1 && errno == ESRCH) { @@ -807,9 +814,10 @@ private: prefix++; } - err = readdir_r(odir, &entry, &result); + err = readdir_r(odir, entry, &result); } closedir(odir); + free(entry); } }; #endif // PBMS_API

3 3

[Maria-developers] compile issue and the -lz flag
by Don Kehn 19 May '09

19 May '09

Hi all: when compiling on x64 systems (i.e. CentOS & Fedora core 10) and using the BUILD/compile-pentium64-max, I'm getting the -lz error "/usr/bin/ld: cannot find -lz" and when I simplify the ./comfigure line to a --without-zlib it works. So, after modifing the configure flags that results into: ./configure --prefix=/u02/local/mysql-5.1.34 --enable-assembler --with-extra-charsets=complex --enable-thread-safe-client --with-big-tables --with-readline --with-ssl --with-plugins=max --with-embedded-server --with-mysqld-ldflags=-all-static --with-client-ldflags=-all-static --without-zlib --enable-local-infile It still has problems with the -lz flag, so does anybody know what is bringing the -lz into the configure? Note: I've also tried --with-zlibdir=/usr/lib64, but same error. Thanks tons.

4 11

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2700)
by knielsen＠knielsen-hq.org 18 May '09

18 May '09

#At lp:maria 2700 knielsen(a)knielsen-hq.org 2009-05-18 Fix support-files/build-tags for newer bzr which do not default to --recursive on bzr ls. modified: support-files/build-tags === modified file 'support-files/build-tags' --- a/support-files/build-tags 2008-08-06 20:25:25 +0000 +++ b/support-files/build-tags 2009-05-18 07:29:14 +0000 @@ -4,7 +4,7 @@ rm -f TAGS filter='\.cc$\|\.c$\|\.h$\|\.yy$' list="find . -type f" -bzr root >/dev/null 2>/dev/null && list="bzr ls --kind=file --versioned" +bzr root >/dev/null 2>/dev/null && list="bzr ls --recursive --kind=file --versioned" $list |grep $filter |while read f; do

1 0

[Maria-developers] ptr_compare replace with memcmp and associated performance
by Stewart Smith 16 May '09

16 May '09

So i broke out mtaylor's patch that we were bumming around with at the UC to replace ptr_compare with a simple memcmp call. At the UC I benched that this patch actually caused a measurable performance regresssion. So what's the difference? (same benchmark and machine as in previous mail) MAX_FIELDS=64 with std::bitset read/write requests: 70000 (3374.50 per sec.) read/write requests: 70000 (3102.42 per sec.) read/write requests: 70000 (3113.47 per sec.) read/write requests: 70000 (3401.89 per sec.) read/write requests: 70000 (3164.61 per sec.) AVERAGE= 3231.37 With ptr_compare replaced with memcmp: read/write requests: 70000 (3090.33 per sec.) read/write requests: 70000 (3066.97 per sec.) read/write requests: 70000 (2954.93 per sec.) read/write requests: 70000 (2875.57 per sec.) read/write requests: 70000 (2953.99 per sec.) AVERAGE= 2988.35 With ptr_compare replaced with __builtin_memcmp: read/write requests: 70000 (3245.37 per sec.) read/write requests: 70000 (3001.91 per sec.) read/write requests: 70000 (3254.99 per sec.) read/write requests: 70000 (3177.13 per sec.) read/write requests: 70000 (3195.49 per sec.) AVERAGE= 3174.97 So, I thought that with just using the __builtin_memcmp automatically, we may be okay with merging this. However... I do have the following concerns: 1) I'm pretty sure that __builtin_memcmp is gcc only, and that SunStudio may have an issue (yay - wrappers!) 2) Is this perf difference going to be true on different platforms? 3) the various memcmp implementations do seem to be dependent on a few things for performance: alignment, size of data. For what we're seeing in sysbench, the parameters are things like this: "SELECT c from sbtest where id between 9942 and 10041 order by c" 0xc30780,0xc2c340,240 0xc2e560,0xc2c340,240 0xc30780,0xc2e560,240 0xc37390,0xc32f50,240 0xc35170,0xc32f50,240 0xc37390,0xc35170,240 0xc3dcc8,0xc39888,240 0xc3baa8,0xc2e560,240 0xc35170,0xc2e560,240 0xc3baa8,0xc35170,240 0xc2c340,0xc35170,240 0xc35170,0xc3dcc8,240 0xc2c618,0xc35170,240 0xc35170,0xc3d9f0,240 0xc2c8f0,0xc35170,240 0x7fbed85f10f0,0x7fbed85eccb0,240 and from the CREATE TABLE: `c` VARCHAR(120) NOT NULL COLLATE utf8_general_ci DEFAULT `` , So here it's large, and aligned. For some other micro-benchmarks I've clocked things looking much different.... so it's possibly query dependent as to what ends up being better (i.e. how much data we're comparing). (1st parameter is number of repetitions, 2nd is number of bytes to memcmp) For comparing equal values: stewart@willster:~/src/test/memcmp$ ./a.out 168435455 8 168435455 repetitions Testing memcmp ..... done, 7.432 seconds Testing builtin memcmp ....... done, 16.313 seconds Testing loop .... done, 6.144 seconds Testing loop32 .... done, 2.764 seconds Testing loop64 .... done, 2.128 seconds Testing no-op .... done, 1.488 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 16 168435455 repetitions Testing memcmp ..... done, 5.912 seconds Testing builtin memcmp ....... done, 26.418 seconds Testing loop .... done, 11.201 seconds Testing loop32 .... done, 3.980 seconds Testing loop64 .... done, 2.564 seconds Testing no-op .... done, 1.492 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 32 168435455 repetitions Testing memcmp ..... done, 6.828 seconds Testing builtin memcmp ....... done, 46.891 seconds Testing loop .... done, 21.473 seconds Testing loop32 .... done, 6.536 seconds Testing loop64 .... done, 3.804 seconds Testing no-op .... done, 1.476 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 64 168435455 repetitions Testing memcmp ..... done, 9.549 seconds Testing builtin memcmp ....... done, 87.513 seconds Testing loop .... done, 41.455 seconds Testing loop32 .... done, 11.669 seconds Testing loop64 .... done, 6.368 seconds Testing no-op .... done, 1.468 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 128 168435455 repetitions Testing memcmp ..... done, 15.081 seconds Testing builtin memcmp ....... done, 169.143 seconds Testing loop .... done, 86.397 seconds Testing loop32 .... done, 21.877 seconds Testing loop64 .... done, 11.445 seconds Testing no-op .... done, 1.488 seconds stewart@willster:~/src/test/memcmp$ gcc -O3 -fno-builtin memcmpbench.c stewart@willster:~/src/test/memcmp$ ./a.out 168435455 256 168435455 repetitions Testing memcmp ..... done, 26.134 seconds Testing loop64 .... done, 21.549 seconds Testing no-op .... done, 1.500 seconds Completely inequal values are all about the same: stewart@willster:~/src/test/memcmp$ ./a.out 168435455 8 168435455 repetitions Testing memcmp ..... done, 3.204 seconds Testing builtin memcmp ....... done, 13.945 seconds Testing loop .... done, 1.896 seconds Testing loop32 .... done, 2.092 seconds Testing loop64 .... done, 2.136 seconds Testing no-op .... done, 1.488 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 240 168435455 repetitions Testing memcmp ..... done, 5.520 seconds Testing builtin memcmp ....... done, 13.973 seconds Testing loop .... done, 1.912 seconds Testing loop32 .... done, 2.124 seconds Testing loop64 .... done, 2.104 seconds Testing no-op .... done, 1.468 seconds For half the same: stewart@willster:~/src/test/memcmp$ ./a.out 168435455 8 168435455 repetitions Testing memcmp ..... done, 7.340 seconds Testing builtin memcmp ....... done, 19.193 seconds Testing loop .... done, 4.212 seconds Testing loop32 .... done, 2.956 seconds Testing loop64 .... done, 2.112 seconds Testing no-op .... done, 1.476 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 32 168435455 repetitions Testing memcmp ..... done, 5.920 seconds Testing builtin memcmp ....... done, 34.526 seconds Testing loop .... done, 11.885 seconds Testing loop32 .... done, 4.444 seconds Testing loop64 .... done, 3.388 seconds Testing no-op .... done, 1.468 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 64 168435455 repetitions Testing memcmp ..... done, 6.948 seconds Testing builtin memcmp ....... done, 54.923 seconds Testing loop .... done, 22.081 seconds Testing loop32 .... done, 7.000 seconds Testing loop64 .... done, 4.444 seconds Testing no-op .... done, 1.488 seconds Is concurrency an issue here and not raw single threaded performance? Let's try with 64 threads (on the same 2 core box): 64 threads, __builtin_memcmp: read/write requests: 70000 (3184.20 per sec.) read/write requests: 70000 (3184.87 per sec.) read/write requests: 70000 (3166.83 per sec.) read/write requests: 70000 (3085.69 per sec.) AVERAGE=3155.39 64 threads, memcmp: read/write requests: 70000 (3218.07 per sec.) read/write requests: 70000 (3219.04 per sec.) read/write requests: 70000 (3222.48 per sec.) read/write requests: 70000 (3116.00 per sec.) AVERAGE=3193.89 64 threads, 32bit cmp loop: read/write requests: 70000 (3173.76 per sec.) read/write requests: 70000 (3156.23 per sec.) read/write requests: 70000 (3206.70 per sec.) read/write requests: 70000 (3247.61 per sec.) AVERAGE=3196.07 64 threads, 64bit cmp loop: read/write requests: 70000 (3210.02 per sec.) read/write requests: 70000 (3218.87 per sec.) read/write requests: 70000 (3225.98 per sec.) read/write requests: 70000 (3131.60 per sec.) AVERAGE= 3196.61 64 threads, baseline (using original ptr_compare): read/write requests: 70000 (3542.95 per sec.) read/write requests: 70000 (3556.15 per sec.) read/write requests: 70000 (3560.71 per sec.) read/write requests: 70000 (3476.80 per sec.) AVERAGE=3534.15 i.e. the old ptr_compare whoops the arse of any of the memcmp calls with higher concurrency. So, after this I can conclude: - memcmp is nothing if not consistent across various microbenchmarks - builtin_memcmp seems to help a bit at lower concurrency, not at all at higher and is seldom useful in micro benchmark. - the 64bit loop isn't so good at higher concurrency - the 64bit loop is favourable in microbenchmarks - memcmp is close to the 64bit loop performance (at most only 2x slower) Unrolling loops being the key? Testing loop .... done, 11.813 seconds Testing unrollloop .... done, 7.628 seconds Testing unrollloop32 .... done, 2.532 seconds Testing unrollloop64 .... done, 2.536 seconds Testing loop32 .... done, 4.432 seconds Testing loop64 .... done, 3.792 seconds Testing no-op .... done, 1.276 seconds (32 and 64 do 32,64 bit compares in an unrolled loop) Questions: - Why does my 64bit compare loop not equal performance of ptr_compare unrolled loop? (in a microbenchmark, it beats it). Concurrency again? So what should we do? Certainly a call to memcmp is easier to understand and keeps the code nice and simple, but we're talking up to a 10% speed hit here at higher concurrency (at least on my hardware). I wonder what the difference is on various other hardware setups. I'd be very interested to see what happens on sparc. I'd also like to see it micro-benchmarked - preferably something we could repeat in future (even in ./configure ? on server startup ?) I'm voting to keep the current ptr_compare code, and hope that somebody provides further explanation on top of what I've looked at here. Below is the patch, followed by the code i used for microbenchmarking (note the commented out 64bit loop i used for the above loop tests): === modified file 'drizzled/filesort.cc' --- drizzled/filesort.cc 2009-04-28 00:17:10 +0000 +++ drizzled/filesort.cc 2009-05-08 05:59:11 +0000 @@ -1135,7 +1135,7 @@ int merge_buffers(SORTPARAM *param, IO_C } else { - cmp= get_ptr_compare(sort_length); + cmp= (qsort2_cmp)ptr_compare; first_cmp_arg= (void*) &sort_length; } priority_queue<BUFFPEK *, vector<BUFFPEK *>, compare_functor > === modified file 'mysys/mf_sort.cc' --- mysys/mf_sort.cc 2009-04-17 21:01:47 +0000 +++ mysys/mf_sort.cc 2009-05-08 05:59:11 +0000 @@ -34,7 +34,7 @@ void my_string_ptr_sort(unsigned char *b { if (size && items) { - my_qsort2(base,items, sizeof(unsigned char*), get_ptr_compare(size), + my_qsort2(base,items, sizeof(unsigned char*), (qsort2_cmp)ptr_compare, (void*) &size); } } === modified file 'mysys/my_sys.h' --- mysys/my_sys.h 2009-04-27 22:05:43 +0000 +++ mysys/my_sys.h 2009-05-08 05:59:11 +0000 @@ -420,7 +420,12 @@ extern void my_qsort(void *base_ptr, siz qsort_cmp cmp); extern void my_qsort2(void *base_ptr, size_t total_elems, size_t size, qsort2_cmp cmp, void *cmp_argument); -extern qsort2_cmp get_ptr_compare(size_t); + +#if defined(__cplusplus) +extern "C" +#endif +int ptr_compare(size_t *compare_length, unsigned char **a, unsigned char **b); + void my_store_ptr(unsigned char *buff, size_t pack_length, my_off_t pos); my_off_t my_get_ptr(unsigned char *ptr, size_t pack_length); File create_temp_file(char *to, const char *dir, const char *pfx, === modified file 'mysys/ptr_cmp.cc' --- mysys/ptr_cmp.cc 2009-04-26 16:53:32 +0000 +++ mysys/ptr_cmp.cc 2009-05-08 05:59:11 +0000 @@ -21,137 +21,33 @@ #include "mysys/mysys_priv.h" #include "plugin/myisam/myisampack.h" - -static int ptr_compare(size_t *compare_length, unsigned char **a, unsigned char **b); -static int ptr_compare_0(size_t *compare_length, unsigned char **a, unsigned char **b); -static int ptr_compare_1(size_t *compare_length, unsigned char **a, unsigned char **b); -static int ptr_compare_2(size_t *compare_length, unsigned char **a, unsigned char **b); -static int ptr_compare_3(size_t *compare_length, unsigned char **a, unsigned char **b); - - /* Get a pointer to a optimal byte-compare function for a given size */ - -qsort2_cmp get_ptr_compare (size_t size) -{ - if (size < 4) - return (qsort2_cmp) ptr_compare; - switch (size & 3) { - case 0: return (qsort2_cmp) ptr_compare_0; - case 1: return (qsort2_cmp) ptr_compare_1; - case 2: return (qsort2_cmp) ptr_compare_2; - case 3: return (qsort2_cmp) ptr_compare_3; - } - return 0; /* Impossible */ -} - +#include <string.h> /* Compare to keys to see witch is smaller. - Loop unrolled to make it quick !! */ -#define cmp(N) if (first[N] != last[N]) return (int) first[N] - (int) last[N] - -static int ptr_compare(size_t *compare_length, unsigned char **a, unsigned char **b) -{ - register int length= *compare_length; - register unsigned char *first,*last; - - first= *a; last= *b; - while (--length) - { - if (*first++ != *last++) - return (int) first[-1] - (int) last[-1]; - } - return (int) first[0] - (int) last[0]; -} - - -static int ptr_compare_0(size_t *compare_length,unsigned char **a, unsigned char **b) +extern "C" +int ptr_compare(size_t *compare_length, unsigned char **a, unsigned char **b) { - register int length= *compare_length; - register unsigned char *first,*last; - - first= *a; last= *b; - loop: - cmp(0); - cmp(1); - cmp(2); - cmp(3); - if ((length-=4)) +/* size_t l= *compare_length / sizeof(uint64_t); + uint64_t *aa= (uint64_t*)*a; + uint64_t *bb= (uint64_t*)*b; + while(l--) { - first+=4; - last+=4; - goto loop; - } - return (0); -} - - -static int ptr_compare_1(size_t *compare_length,unsigned char **a, unsigned char **b) -{ - register int length= *compare_length-1; - register unsigned char *first,*last; - - first= *a+1; last= *b+1; - cmp(-1); - loop: - cmp(0); - cmp(1); - cmp(2); - cmp(3); - if ((length-=4)) - { - first+=4; - last+=4; - goto loop; - } - return (0); -} - -static int ptr_compare_2(size_t *compare_length,unsigned char **a, unsigned char **b) -{ - register int length= *compare_length-2; - register unsigned char *first,*last; - - first= *a +2 ; last= *b +2; - cmp(-2); - cmp(-1); - loop: - cmp(0); - cmp(1); - cmp(2); - cmp(3); - if ((length-=4)) - { - first+=4; - last+=4; - goto loop; + if(*aa != *bb) + { + if(*aa < *bb) + return -1; + else + return 1; + } + aa++, bb++; } - return (0); + return 0; +*/ return memcmp(*a, *b, *compare_length); } -static int ptr_compare_3(size_t *compare_length,unsigned char **a, unsigned char **b) -{ - register int length= *compare_length-3; - register unsigned char *first,*last; - - first= *a +3 ; last= *b +3; - cmp(-3); - cmp(-2); - cmp(-1); - loop: - cmp(0); - cmp(1); - cmp(2); - cmp(3); - if ((length-=4)) - { - first+=4; - last+=4; - goto loop; - } - return (0); -} void my_store_ptr(unsigned char *buff, size_t pack_length, my_off_t pos) { For those interested, i started with the code at http://gcc.gnu.org/ml/gcc/2002-10/msg01666.html and ended up with something like this: #include <sys/resource.h> #include <sys/time.h> #include <stdio.h> #include <assert.h> #include <stdlib.h> char s1[256]; char s2[256]; int cmpsize= 240; int speed_memcmp () { return memcmp (s1, s2, cmpsize); } int speed_bimemcmp () { return __builtin_memcmp (s1, s2, cmpsize); } int speed_loop () { int i=cmpsize; char *a= s1; char *b= s2; while(i--) { if(*a != *b) return -1; a++, b++; }; return 0; } int speed_loop32 () { int i=cmpsize/4; int *a= (int*)s1; int *b= (int*)s2; while(i--) { if(*a != *b) return -1; a++, b++; }; return 0; } int speed_loop64 () { int i=cmpsize/8; long long *a= (long long*)s1; long long *b= (long long*)s2; while(i--) { if(*a != *b) return -1; a++, b++; }; return 0; } int speed_null() { return 0; } double do_test (int repetitions, int (*test_function) ()) { struct rusage r1, r2; getrusage(RUSAGE_SELF, &r1); while (repetitions--) test_function (); getrusage(RUSAGE_SELF, &r2); return (r2.ru_utime.tv_sec - r1.ru_utime.tv_sec) + (r2.ru_utime.tv_usec - r1.ru_utime.tv_usec) / 1000000.0; } main(int argc, char **argv) { int repetitions = (argc == 1) ? 0x0fffffff : atoi(argv[1]); cmpsize = (argc == 2) ? 240 : atoi(argv[2]); memset(s1, 42, 256); memset(s2, 42, 256); memset(s2+(cmpsize/2), 55, cmpsize/2); printf ("%d repetitions\n\n", repetitions); printf ("Testing memcmp ....."); fflush (stdout); printf (" done, %10.3f seconds\n", do_test (repetitions, speed_memcmp)); printf ("Testing builtin memcmp ......."); fflush (stdout); printf (" done, %10.3f seconds\n", do_test (repetitions, speed_bimemcmp)); printf ("Testing loop ...."); fflush (stdout); printf (" done, %10.3f seconds\n", do_test (repetitions, speed_loop)); printf ("Testing loop32 ...."); fflush (stdout); printf (" done, %10.3f seconds\n", do_test (repetitions, speed_loop32)); printf ("Testing loop64 ...."); fflush (stdout); printf (" done, %10.3f seconds\n", do_test (repetitions, speed_loop64)); printf ("Testing no-op ...."); fflush (stdout); printf (" done, %10.3f seconds\n", do_test (repetitions, speed_null)); exit (0); } -- Stewart Smith

6 5

[Maria-developers] Cmake changes
by Stewart Smith 16 May '09

16 May '09

Antony mentioned on IRC about CMake changes that let you not use config.js and other nasty things when building on Win32 (and Win64). These are all just in the MySQL Cluster branch - namely 7.0. Who knows when this gets back into any mainline branch though... -- Stewart Smith

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2699)
by knielsen＠knielsen-hq.org 15 May '09

15 May '09

#At lp:maria 2699 knielsen(a)knielsen-hq.org 2009-05-15 test commit 1 modified: README === modified file 'README' --- a/README 2008-03-26 10:15:03 +0000 +++ b/README 2009-05-15 06:31:51 +0000 @@ -33,3 +33,4 @@ file. IMPORTANT: Bug or error reports should be sent to http://bugs.mysql.com. +

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2689)
by knielsen＠knielsen-hq.org 13 May '09

13 May '09

#At lp:maria 2689 knielsen(a)knielsen-hq.org 2009-03-31 Fix build error after last push with --with-debug=full due to SAFEMALLOC now being defined in my_config.h (as opposed to in CFLAGS before.) modified: mysys/my_malloc.c mysys/my_once.c mysys/my_realloc.c per-file messages: mysys/my_malloc.c Need to include my_global.h before messing with SAFEMALLOC, as now that macro may be re-defined in my_config.h, which is included from my_global.h mysys/my_once.c Need to include my_global.h before messing with SAFEMALLOC, as now that macro may be re-defined in my_config.h, which is included from my_global.h mysys/my_realloc.c Need to include my_global.h before messing with SAFEMALLOC, as now that macro may be re-defined in my_config.h, which is included from my_global.h === modified file 'mysys/my_malloc.c' --- a/mysys/my_malloc.c 2007-10-02 07:32:33 +0000 +++ b/mysys/my_malloc.c 2009-03-31 08:06:51 +0000 @@ -13,6 +13,9 @@ along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ +/* my_global.h may define SAFEMALLOC (through my_config.h). */ +#include <my_global.h> + #ifdef SAFEMALLOC /* We don't need SAFEMALLOC here */ #undef SAFEMALLOC #endif === modified file 'mysys/my_once.c' --- a/mysys/my_once.c 2007-05-10 09:59:39 +0000 +++ b/mysys/my_once.c 2009-03-31 08:06:51 +0000 @@ -15,6 +15,9 @@ /* Not MT-SAFE */ +/* my_global.h may define SAFEMALLOC (through my_config.h). */ +#include <my_global.h> + #ifdef SAFEMALLOC /* We don't need SAFEMALLOC here */ #undef SAFEMALLOC #endif === modified file 'mysys/my_realloc.c' --- a/mysys/my_realloc.c 2008-04-28 16:24:05 +0000 +++ b/mysys/my_realloc.c 2009-03-31 08:06:51 +0000 @@ -13,6 +13,9 @@ along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ +/* my_global.h may define SAFEMALLOC (through my_config.h). */ +#include <my_global.h> + #ifdef SAFEMALLOC /* We don't need SAFEMALLOC here */ #undef SAFEMALLOC #endif

8 29

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2698)
by knielsen＠knielsen-hq.org 10 May '09

10 May '09

#At lp:maria 2698 knielsen(a)knielsen-hq.org 2009-05-05 [merge] test merge. modified: BUILD/compile-solaris-amd64 BUILD/compile-solaris-amd64-debug* BUILD/compile-solaris-amd64-forte BUILD/compile-solaris-amd64-forte-debug* storage/innobase/include/univ.i storage/pbxt/src/Makefile.am storage/pbxt/src/lock_xt.h storage/pbxt/src/pbms.h === modified file 'BUILD/compile-solaris-amd64' --- a/BUILD/compile-solaris-amd64 2007-04-12 11:20:38 +0000 +++ b/BUILD/compile-solaris-amd64 2009-05-04 14:36:55 +0000 @@ -1,55 +1,34 @@ -#!/usr/bin/bash +#!/bin/sh -function _find_mysql_root () ( - while [ "x$PWD" != "x/" ]; do - # Check if some directories are present - if [ -d BUILD -a -d sql -a -d mysys ]; then - echo "$PWD" - return 0 - fi - cd .. - done - return 1 -) +# Pre-requisites for Solaris 10: +# +# setup: +# ln -s `which perl` /usr/local/bin +# your $PATH needs to include (in this order): +# /usr/local/bin:/usr/sfw/bin:/usr/ccs/bin +# +# packages from http://sunfreeware.com/indexintel10.html : +# automake-1.10.2-sol10-x86-local.gz +# autoconf-2.63-sol10-x86-local.gz +# m4-1.4.12-sol10-x86-local.gz +# libsigsegv-2.6-sol10-x86-local.gz +# libtool-1.5.24-sol10-x86-local.gz +# ( how to install these packages: + # wget ftp://ftp.sunfreeware.com/pub/freeware/intel/10/automake-1.10.2-sol10-x86-l… + # gunzip automake-1.10.2-sol10-x86-local.gz + # pkgadd -d automake-1.10.2-sol10-x86-local +# ) + +# +# if using --with-libevent, install that library from +# http://www.monkey.org/~provos/libevent/ -make -k clean || true -/bin/rm -f */.deps/*.P config.cache - path=`dirname $0` -. "$path/autorun.sh" - -warning_flags="-Wimplicit -Wreturn-type -Wswitch -Wtrigraphs -Wcomment -W -Wchar-subscripts -Wformat -Wparentheses -Wsign-compare -Wwrite-strings -Wunused" -compiler_flags="-g -O3 -fno-omit-frame-pointer" +. "$path/SETUP.sh" +extra_flags="$amd64_cflags -D__sun -m64 -mtune=athlon64" +extra_configs="$amd64_configs $max_configs --with-libevent" -export CC CXX CFLAGS CXXFLAGS LDFLAGS LIBS -CC="gcc" -CXX="gcc" -CFLAGS="$warning_flags $compiler_flags" -CXXFLAGS="" -LDFLAGS="-O3 -g -static-libgcc" -LIBS=-lmtmalloc -root=$(_find_mysql_root) +LDFLAGS="-lmtmalloc -static-libgcc" +export LDFLAGS -$root/configure \ - --prefix=/usr/local/mysql \ - --localstatedir=/usr/local/mysql/data \ - --libexecdir=/usr/local/mysql/bin \ - --with-extra-charsets=complex \ - --enable-thread-safe-client \ - --enable-local-infile \ - --with-zlib-dir=bundled \ - --with-big-tables \ - --with-readline \ - --with-archive-storage-engine \ - --with-named-curses=-lcurses \ - --with-big-tables \ - --with-innodb \ - --with-berkeley-db \ - --with-example-storage-engine \ - --with-blackhole-storage-engine \ - --with-ndbcluster \ - --with-federated-storage-engine \ - --with-csv-storage-engine \ - --with-ssl \ - --with-embedded-server \ - --disable-shared +. "$path/FINISH.sh" === modified file 'BUILD/compile-solaris-amd64-debug' (properties changed: -x to +x) --- a/BUILD/compile-solaris-amd64-debug 2007-11-10 10:03:07 +0000 +++ b/BUILD/compile-solaris-amd64-debug 2009-05-04 14:36:55 +0000 @@ -1,10 +1,11 @@ -#! /bin/sh +#!/bin/sh + path=`dirname $0` . "$path/SETUP.sh" -amd64_cflags="-m64 -mtune=athlon64" -extra_flags="$amd64_cflags $debug_cflags $max_cflags" -c_warnings="$c_warnings $debug_extra_warnings" -cxx_warnings="$cxx_warnings $debug_extra_warnings" -extra_configs="$amd64_configs $debug_configs $max_configs --enable-thread-safe-client" +extra_flags="$amd64_cflags -D__sun -m64 $debug_cflags" +extra_configs="$amd64_configs $debug_configs $max_configs --with-libevent" + +LDFLAGS="-lmtmalloc -static-libgcc" +export LDFLAGS . "$path/FINISH.sh" === modified file 'BUILD/compile-solaris-amd64-forte' --- a/BUILD/compile-solaris-amd64-forte 2008-09-30 20:57:48 +0000 +++ b/BUILD/compile-solaris-amd64-forte 2009-05-04 15:48:27 +0000 @@ -1,53 +1,35 @@ -#! /bin/sh +#!/bin/sh -gmake -k clean || true -/bin/rm -f */.deps/*.P config.cache - -path=`dirname $0` -. "$path/autorun.sh" +# See file compile-solaris-amd64 for basic pre-requisites. + +# This build uses the Sun Studio compilers (cc, CC). +# http://developers.sun.com/sunstudio/downloads/index.jsp +# Note that you may want to apply current patches, as the downloaded version +# is typically out of date. Download the PKG version if you intend to patch! + +# After installing, add /opt/SUNWspro/bin to your $PATH -# For "optimal" code for this computer add -fast to EXTRA -# To compile 64 bit, add -m64 to EXTRA_64_BIT -EXTRA_64_BIT="-m64" -EXTRA="-fast" +path=`dirname $0` +. "$path/SETUP.sh" + +extra_flags="-m64 -mt -D_FORTEC_ -xbuiltin=%all -xlibmil -xlibmopt -fns=no -xprefetch=auto -xprefetch_level=3" +extra_configs="$max_configs --with-libevent" -# -# The following should not need to be touched -# - -export CC CXX CFLAGS CXXFLAGS LIBS -STD="-g -mt -D_FORTEC_ $EXTRA $EXTRA_64_BIT" -ASFLAGS="$EXTRA_64_BIT" -CC=cc-5.0 -CFLAGS="-Xa -xstrconst $STD" +# I want this for my buildbot, which has limited zone resources +#export AM_MAKEFLAGS +#AM_MAKEFLAGS="" + +export warnings c_warnings cxx_warnings base_cxxflags +warnings="" +c_warnings="" +cxx_warnings="" +base_cxxflags="-noex" + +export CC CXX CFLAGS CXXFLAGS LDFLAGS +CC=cc +CFLAGS="-xstrconst" CXX=CC -CXXFLAGS="-noex $STD" -LIBS=-lmtmalloc -./configure \ - --prefix=/usr/local/mysql \ - --localstatedir=/usr/local/mysql/data \ - --libexecdir=/usr/local/mysql/bin \ - --with-extra-charsets=complex \ - --enable-thread-safe-client \ - --enable-local-infile \ - --with-zlib-dir=bundled \ - --with-big-tables \ - --with-readline \ - --with-archive-storage-engine \ - --with-named-curses=-lcurses \ - --with-big-tables \ - --with-innodb \ - --with-example-storage-engine \ - --with-blackhole-storage-engine \ - --with-federated-storage-engine \ - --with-csv-storage-engine \ - --with-ssl \ - --enable-assembler - -# Not including: -# --with-ndbcluster -# --with-berkeley-db +LDFLAGS="-lmtmalloc" -gmake -j4 -test $? = 0 && make test +. "$path/FINISH.sh" === modified file 'BUILD/compile-solaris-amd64-forte-debug' (properties changed: -x to +x) --- a/BUILD/compile-solaris-amd64-forte-debug 2008-09-30 20:57:48 +0000 +++ b/BUILD/compile-solaris-amd64-forte-debug 2009-05-04 15:48:27 +0000 @@ -1,54 +1,28 @@ -#! /bin/sh +#!/bin/sh -gmake -k clean || true -/bin/rm -f */.deps/*.P config.cache - path=`dirname $0` -. "$path/autorun.sh" +. "$path/SETUP.sh" -# To compile 64 bit, add -m64 to EXTRA_64_BIT -EXTRA_64_BIT="-m64" - -# For "optimal" code for this computer add -fast to EXTRA. Note that -# this causes problem with debugging the program since -fast implies -# -xO5. -EXTRA="" - -# -# The following should not need to be touched -# - -export CC CXX CFLAGS CXXFLAGS -STD="-g -mt -D_FORTEC_ $EXTRA $EXTRA_64_BIT $debug_cflags" -ASFLAGS="$EXTRA_64_BIT" -CC=cc-5.0 -CFLAGS="-Xa -xstrconst $STD" +# take only #define options - the others are gcc specific +DEFS="" +for F in $debug_cflags ; do + expr "$F" : "^-D" && DEFS="$DEFS $F" +done +debug_cflags="-O0 -g $DEFS" + +extra_flags="-m64 -mt -D_FORTEC_ -xlibmopt -fns=no $debug_cflags" +extra_configs="$max_configs --with-libevent $debug_configs" + +export warnings c_warnings cxx_warnings base_cxxflags debug_cflags +warnings="" +c_warnings="" +cxx_warnings="" +base_cxxflags="-noex" + +export CC CXX CFLAGS CXXFLAGS LDFLAGS +CC=cc +CFLAGS="-xstrconst" CXX=CC -CXXFLAGS="-noex $STD" -./configure \ - --prefix=/usr/local/mysql \ - --localstatedir=/usr/local/mysql/data \ - --libexecdir=/usr/local/mysql/bin \ - --with-extra-charsets=complex \ - --enable-thread-safe-client \ - --enable-local-infile \ - --with-zlib-dir=bundled \ - --with-big-tables \ - --with-readline \ - --with-archive-storage-engine \ - --with-named-curses=-lcurses \ - --with-big-tables \ - --with-innodb \ - --with-example-storage-engine \ - --with-blackhole-storage-engine \ - --with-federated-storage-engine \ - --with-csv-storage-engine \ - --with-ssl \ - --with-debug \ - --enable-assembler - -# Not including: -# --with-ndbcluster -# --with-berkeley-db +LDFLAGS="-lmtmalloc" -gmake -j4 +. "$path/FINISH.sh" === modified file 'storage/innobase/include/univ.i' --- a/storage/innobase/include/univ.i 2008-03-27 01:40:45 +0000 +++ b/storage/innobase/include/univ.i 2009-05-04 14:54:10 +0000 @@ -58,7 +58,7 @@ of the 32-bit x86 assembler in mutex ope /* We only try to do explicit inlining of functions with gcc and Microsoft Visual C++ */ -# if !defined(__GNUC__) +# if !defined(__GNUC__) && !defined(__SUNPRO_C) # undef UNIV_MUST_NOT_INLINE /* Remove compiler warning */ # define UNIV_MUST_NOT_INLINE # endif === modified file 'storage/pbxt/src/Makefile.am' --- a/storage/pbxt/src/Makefile.am 2009-04-01 06:40:11 +0000 +++ b/storage/pbxt/src/Makefile.am 2009-05-04 14:36:55 +0000 @@ -46,7 +46,7 @@ libpbxt_la_CFLAGS = $(AM_CFLAGS) -DMYSQ EXTRA_LIBRARIES = libpbxt.a noinst_LIBRARIES = libpbxt.a libpbxt_a_SOURCES = $(libpbxt_la_SOURCES) -libpbxt_a_CXXFLAGS = $(AM_CXXFLAGS) -Wno-long-long +libpbxt_a_CXXFLAGS = $(AM_CXXFLAGS) libpbxt_a_CFLAGS = $(AM_CFLAGS) -std=c99 EXTRA_DIST = CMakeLists.txt === modified file 'storage/pbxt/src/lock_xt.h' --- a/storage/pbxt/src/lock_xt.h 2009-04-02 10:03:14 +0000 +++ b/storage/pbxt/src/lock_xt.h 2009-05-04 14:36:55 +0000 @@ -208,9 +208,9 @@ inline void xt_atomic_dec2(volatile xtWo #elif defined(__GNUC__) __sync_fetch_and_sub(mptr, 1); #elif defined(XT_SPL_SOLARIS_LIB) - val1 = atomic_dec_16_nv(mptr); + atomic_dec_16_nv(mptr); #else - val1 = --(*mptr); + --(*mptr); #endif } === modified file 'storage/pbxt/src/pbms.h' --- a/storage/pbxt/src/pbms.h 2009-03-26 12:18:01 +0000 +++ b/storage/pbxt/src/pbms.h 2009-05-03 16:31:02 +0000 @@ -782,21 +782,28 @@ private: void deleteTempFiles() { - struct dirent entry; + struct dirent *entry; struct dirent *result; DIR *odir; int err; + size_t sz; char temp_file[100]; +#ifdef XT_SOLARIS + sz = sizeof(struct dirent) + pathconf("/tmp/", _PC_NAME_MAX); // Solaris, see readdir(3C) +#else + sz = sizeof(struct dirent); +#endif + entry = (struct dirent*)malloc(sz); if (!(odir = opendir("/tmp/"))) return; - err = readdir_r(odir, &entry, &result); + err = readdir_r(odir, entry, &result); while (!err && result) { const char **prefix = temp_prefix; while (*prefix) { - if (startsWith(entry.d_name, *prefix)) { - int pid = atoi(entry.d_name + strlen(*prefix)); + if (startsWith(entry->d_name, *prefix)) { + int pid = atoi(entry->d_name + strlen(*prefix)); /* If the process does not exist: */ if (kill(pid, 0) == -1 && errno == ESRCH) { @@ -807,9 +814,10 @@ private: prefix++; } - err = readdir_r(odir, &entry, &result); + err = readdir_r(odir, entry, &result); } closedir(odir); + free(entry); } }; #endif // PBMS_API

4 10

Re: [Maria-developers] SQL Parser
by Antony T Curtis 09 May '09

09 May '09

On 9 May 2009, at 10:22, Jim Idle wrote: > Antony T Curtis wrote: >> >> On 9 May 2009, at 00:35, Michael Widenius wrote: >> >>> >>> jim> Indeed. Obviously syntactical changes are just slog really and >>> jim> most of that is ensuring that there is enough coverage in >>> regression >>> jim> tests to exercise all the syntactic differences between T-SQL >>> and >>> jim> MySql, or rather, all the syntax of MySQL. A lot of >>> commonality but >>> jim> still a fair bit of work. What is the definitive language >>> reference >>> jim> for MySQL Marina? I can locate reference manuals and ANSI >>> specs of >>> jim> course, but if you have one particular document in mind, then >>> it would >>> jim> be useful to work off that so I can evaluate the syntactical >>> changes I >>> jim> would need to make. >>> >>> Unfortunately there is no other spec than the sql_yacc.yy file that >>> comes with the MySQL source code. >> >> I have a tool somewhere which can build .dot files from the .yy >> files in the same way how ANTLRworks draws its syntax diagrams.... >> Copies of the tool exist somewhere on the old MySQL internal >> repositories but I am sure I have a backup copy somewhere. > Yes, I added this ability to ANTLR C runtime too. You can see the > effect here: > > http://www.temporal-wave.com/index.php?option=com_psrrun&view=psrrun&Itemid… What I mean was... Feed in the sql_yacc.yy file and output a human understandable syntax diagram. > >> >>> >>> jim> >>> jim> I would like to do this in two steps: >>> jim> >>> jim> a) Replace the mysql_yacc.yy file, with as few changes as >>> possible in >>> jim> the rest of the MySQL source. >>> jim> >>> jim> >>> jim> ANTLR is essentially the same gestalt as bison: one or more >>> source files define the grammar and contain actions in C, >>> triggered at the parse points; These grammar definition files are >>> converted into C source by invoking an external program (in the >>> case of ANTLR it is a Java program) and the resulting .c/.h files >>> are compiled with the rest of the code. >>> jim> >>> jim> I wrote the C output to be machine independent. The idea >>> being that you can generate the .c files on one machine, check >>> that source code in and use it on any other machine without re- >>> generating. This can be useful when targeting systems without Java >>> machines (or ones that don't work very well). So, like many other >>> systems you can have a 'build completely from scratch' option and >>> a 'use pre-generated files option'. >>> jim> >>> jim> In the main then, I suspect that the bigger changes to the >>> MySQL base would be build changes. I say this based upon the >>> assumption that any new parser should build the same AST for a >>> query as the yacc based version does. This means the new parser >>> makes the same calls as the old one and thus the code external to >>> the parser would probably require little, if anything in the way >>> of change (but see b below). >>> >>> That sounds very good. >> >> Not possible to build the same AST as the yacc/bison based parser >> because there is no AST. >> The execution structures are directly built during the parsing phase. > Sure - for AST, you can read execution structures - there didn't > really seem to be any nomenclature but when I discussed this with > some people at Sun a while back, they used the terms AST and tree, > so I just copied them :-) So I think I would use the AST I already > build, to generate the execution structures there currently. Having > an AST means that there is a structure that can be later used for > optimization (should that be any better than optimization the > current execution structures). For a few of us, AST != execution structures .... mostly because the execution structures are very much not "abstract" and were dictated by historical and implementation issues. I have argued years ago that a good AST would allow better optimization later cheaply. For example, right now, there is a lot of work to try to simplify the Item trees for boolean expressions. This work would be better done with ASTs without wasting as much memory. > >> >>> Agree >> >> You will have to implement an all-new AST which then can be used to >> 'compile' the execution structures. >> (this was a minor sticking part of an earlier attempt to replace >> the parser - there were objections to building an AST and adding an >> additional stage to the execution) > Yeah - I just wrote that before I read this paragraph ;-) As I > already have this structure, it can be shown to pretty much zero > overhead compared to query optimization and execution time. A good > AST mostly simplifies rather than complicates things of course. > Whether my current AST needs shaping to optimally generate MySQL > remains to be seen, but it is easy to change if needed. ++Agree The only tricky issues is to try to generate exactly the same structures but in some instances later, there will be scope for actually creating simpler structures than what MySQL currently generates. > >> >>> n more generic structures. That should be easy to define. >>> >>> The MySQL code doesn't rely on bison structures at all (as far as >>> I know). >> >> AFAIK, MySQL does not use any Bison internals as such. > Yes - I could not see any reliance myself. >> A past project from many years ago, I put into MySQL a PCCTS (pre- >> ANTLR) LL parser, basically keeping MySQL's custom lexer but it >> generated Token objects instead suitable for the PCCTS generated >> parser. The project was never completed because integration of >> Stored Procedures became the objective for MySQL 5.0 instead of >> implementation of a new parser. >> >> I have asked Stewart Smith to see if he can help make public that >> old source repository as a Bazaar repository as people may be able >> to learn from it how to replace the parser without major pain. > Sure - any prior work obviously helps, but I think that the lexing > is already good. I might make changes to keyword detection to become > MySql friendly; T-SQL is a historically a bit unsure exactly when > keywords can be used. This would reduce the DFA sizes for the parser. >> >>> >>> Did you get time to do the analyse? >>> >>> jim> >>> jim> A few other things that may be of note: >>> jim> >>> jim> 1) The ANTLR generated code is free threading so long as any >>> action code that you place withing the parser is also free >>> threading. >>> >>> In other words, same as with bison >> >> The ANTLR runtime does, of course, require Java. > This, this would be my C runtime as Java is way too slow for this. > There is no Java required at runtime, only for developers to > generate the C. >> So developers who wish to extend the grammar do have to have the >> Java Runtime installed as well as the ANTLR jars. > For grammar yes as the ANTLR tool runs in Java. In practice this > isn't really a problem I think? Yeah. When I wrote "ANTLR runtime", I meant "ANTLR compiler". > >> >> I too am interested to help if at all possible. > One thing that this would need would be a good regression testing > suite. Right now I have 1300 .sql files that prove the lexer/parser/ > ast, but there would need to be more of these and runtime testing > and so on. There will have to be proper unit tests to show that a certain AST builds the correct required MySQL execution structures (Item trees, JOIN, TABLE_LIST etc etc). The best part of such an effort would be "At last! a formal specification of how the execution structures should look like". This would be exciting and cool if we can (at last) have a sane parser... especially as LL parsers always generate much nicer error messages for the users. Regards, Antony.

1 0

Re: [Maria-developers] SQL Parser
by Michael Widenius 09 May '09

09 May '09

Hi! I fixed the maria-developers address so that everyone else can follow and participate in the discussion. jim> jim> Monty, jim> jim> jim> jim> My T-SQL parser is now complete and a number of people are using jim> jim> the Java, C# and C version. Would you like to have a look at it? jim> jim> Monty> Yes, I would be intrested to know more about this! jim> jim> OK - I have my demo online now. If you visit http://www.temporal-wave.com jim> jim> and take the Online Demos->T-SQL menu option you can enter T-SQL jim> batches and see the parse results. Over the next day or so, I am jim> adding demos of my C# and VB.Net parsers, so if the site goes up and jim> down then just wait a few minutes and try again :-) Once I have jim> evaluated some of the things you mention below then I will let you jim> have a scan at the source code. ok. I will take a look as soon as I have got through the emails that has batched up during the last 3 weeks. jim> jim> I think that I could steal a lot of common syntax from this for a jim> new MySQL parser. There are 1100 regression tests culled from the jim> T-SQL language spec and user examples; on my Linux system the parse jim> time, including building and walking the AST is: jim> jim> jim> jim> jimi(50_64)-clean: wc -l *.sql jim> jim> ... jim> jim> 12923 total jim> jim> jim> jim> jimi(50_64)-clean: time tsqlc . >out jim> jim> 0.15user 0.19system 0:00.34elapsed 100%CPU (0avgtext+0avgdata jim> 0maxresident)k jim> jim> 0inputs+232outputs (0major+102772minor)pagefaults 0swaps jim> jim> jim> Which includes the time taken to read each of the 1100 files from jim> the file system and to print a message for each saying the parse was jim> successful of course. jim> jim> Anyway, if you want to take a look at the code, just let me know :-) jim> jim> Monty> The big question is how much work it would be to jim> replace the MySQLparser with your T-SQL parser. jim> Indeed. Obviously syntactical changes are just slog really and jim> most of that is ensuring that there is enough coverage in regression jim> tests to exercise all the syntactic differences between T-SQL and jim> MySql, or rather, all the syntax of MySQL. A lot of commonality but jim> still a fair bit of work. What is the definitive language reference jim> for MySQL Marina? I can locate reference manuals and ANSI specs of jim> course, but if you have one particular document in mind, then it would jim> be useful to work off that so I can evaluate the syntactical changes I jim> would need to make. Unfortunately there is no other spec than the sql_yacc.yy file that comes with the MySQL source code. jim> jim> I would like to do this in two steps: jim> jim> a) Replace the mysql_yacc.yy file, with as few changes as possible in jim> the rest of the MySQL source. jim> jim> jim> ANTLR is essentially the same gestalt as bison: one or more source files define the grammar and contain actions in C, triggered at the parse points; These grammar definition files are converted into C source by invoking an external program (in the case of ANTLR it is a Java program) and the resulting .c/.h files are compiled with the rest of the code. jim> jim> I wrote the C output to be machine independent. The idea being that you can generate the .c files on one machine, check that source code in and use it on any other machine without re-generating. This can be useful when targeting systems without Java machines (or ones that don't work very well). So, like many other systems you can have a 'build completely from scratch' option and a 'use pre-generated files option'. jim> jim> In the main then, I suspect that the bigger changes to the MySQL base would be build changes. I say this based upon the assumption that any new parser should build the same AST for a query as the yacc based version does. This means the new parser makes the same calls as the old one and thus the code external to the parser would probably require little, if anything in the way of change (but see b below). That sounds very good. jim> b) Optimze MySQL to be even faster, with the new parser at a base. jim> jim> Obviously there are two aspects, being the speed of the parser itself, which I don't think will be an issue, and then what kind of transformations are done on the resulting AST for optimization purposes. If we use exactly the same AST as MySQL does now, then improving MySQL performance is the same problem domain as it is now. However, perhaps there is opportunity for a new parser to provide more information about the query that would aid optimization. Possibilities here include modifications to the AST, external information structures that current code can use when walking the AST, or even a completely new AST. The latter would probably have quite an effect on the front end of the parse->AST->execution phase of course, so I would think that the first step is to try to achieve your step a with the existing AST as output. Agree. jim> Of course maintaining and changing an ANTLR based grammar is somewhat easier than changing bison/yacc, if for no other reason than ANTLR doesn't need much in the way of code assists for correct parsing, whereas Bison quite often does. jim> The a) option is quite important to be able to do easy merges with the jim> main MySQL tree. jim> jim> I have time this week to take a look at this. Mechanically I know it is easy enough, but I need to look at the calls that are made from the current yacc/bison parser and see if there is anything in there that would be awkward. The only things I could think of would be if the existing code relies on internal structures of bison rather than more generic structures. That should be easy to define. The MySQL code doesn't rely on bison structures at all (as far as I know). Did you get time to do the analyse? jim> jim> A few other things that may be of note: jim> jim> 1) The ANTLR generated code is free threading so long as any action code that you place withing the parser is also free threading. In other words, same as with bison jim> 2) The only system calls it uses is {m|c|re}alloc/free, which the C runtime component can re-direct (say to the MySQL overrides). For completely free threading, those calls also need to be free threading and so they are the only limit to threading of course. Sounds perfect. I would *really* like to see a new parser for MariaDB. Anything that we can do to help get this done ? Regards, Monty

2 1