- developers - lists.mariadb.org

[Maria-developers] Updated (by Knielsen): Using the Valgrind API in mysqld (23)
by worklog-noreply＠askmonty.org 22 May '09

22 May '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Using the Valgrind API in mysqld CREATION DATE..: Fri, 22 May 2009, 11:43 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 23 (http://askmonty.org/worklog/?tid=23) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: -=-=(Knielsen - Fri, 22 May 2009, 11:52)=-=- High-Level Specification modified. --- /tmp/wklog.23.old.17628 2009-05-22 11:52:33.000000000 +0300 +++ /tmp/wklog.23.new.17628 2009-05-22 11:52:33.000000000 +0300 @@ -1 +1,28 @@ +With custom memory allocators, using the Valgrind APIs we can tell Valgrind when +a memory block is allocated (so that data read from memory is marked as undefined +instead of being defined or not at random depending on prior use); and when a +memory block is freed (so that use after freeing can be reported as an error). +In some cases cheking for leaks may also be appropriate. + +Another possibility is to add an explicit check for whether memory is defined. + +One place this would be useful is when calling libz. Due to the design of that +library, Valgrind produces lots of false alarms about using undefined values +(I think the issue is that it runs a few bytes off of initialized memory to +reduce boundary checks in each loop iteration, then after the loop has checks to +avoid using the undefined part of the result). This means we have lots of libz +Valgrind suppressions and continue to add more as new warnings surface. So we +might easily miss a real problem in this area. This could be improved by adding +explicit checks at the call to libz functions that the passed memory is properly +defined. + +Another use is to improve debugging. It is often the case when debugging a +warning about using un-initialised memory that the detection happens long after +the real problem, the un-initialized value being passed along through the code +for a long time before being detected. This makes debugging the problem slow. + +By adding in strategic places code that asserts that a specific value must be +initialised, it is possible to detect problems earlier, speeding up debugging. +Such code can be added in more places over time as development and debugging +goes on. DESCRIPTION: Valgrind (the memcheck tool) has some very useful APIs that can be used in mysqld when testing with Valgrind to improve testing and/or debugging: file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.clientreqs file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.mempools This worklog is about adding configure checks and headers to allow to use these in a way that continues to work on machines where the Valgrind headers or functionality is missing. It also includes adding some basic Valgrind enhancements: - Adding Valgrind annotations to custom memory allocators so that Valgrind can detect leaks, use-before-init, and use-after-free problems also for these allocators. - Adding checks for definedness in appropriate places (eg. when calling libz). HIGH-LEVEL SPECIFICATION: With custom memory allocators, using the Valgrind APIs we can tell Valgrind when a memory block is allocated (so that data read from memory is marked as undefined instead of being defined or not at random depending on prior use); and when a memory block is freed (so that use after freeing can be reported as an error). In some cases cheking for leaks may also be appropriate. Another possibility is to add an explicit check for whether memory is defined. One place this would be useful is when calling libz. Due to the design of that library, Valgrind produces lots of false alarms about using undefined values (I think the issue is that it runs a few bytes off of initialized memory to reduce boundary checks in each loop iteration, then after the loop has checks to avoid using the undefined part of the result). This means we have lots of libz Valgrind suppressions and continue to add more as new warnings surface. So we might easily miss a real problem in this area. This could be improved by adding explicit checks at the call to libz functions that the passed memory is properly defined. Another use is to improve debugging. It is often the case when debugging a warning about using un-initialised memory that the detection happens long after the real problem, the un-initialized value being passed along through the code for a long time before being detected. This makes debugging the problem slow. By adding in strategic places code that asserts that a specific value must be initialised, it is possible to detect problems earlier, speeding up debugging. Such code can be added in more places over time as development and debugging goes on. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Knielsen): Using the Valgrind API in mysqld (23)
by worklog-noreply＠askmonty.org 22 May '09

22 May '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Using the Valgrind API in mysqld CREATION DATE..: Fri, 22 May 2009, 11:43 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 23 (http://askmonty.org/worklog/?tid=23) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: DESCRIPTION: Valgrind (the memcheck tool) has some very useful APIs that can be used in mysqld when testing with Valgrind to improve testing and/or debugging: file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.clientreqs file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.mempools This worklog is about adding configure checks and headers to allow to use these in a way that continues to work on machines where the Valgrind headers or functionality is missing. It also includes adding some basic Valgrind enhancements: - Adding Valgrind annotations to custom memory allocators so that Valgrind can detect leaks, use-before-init, and use-after-free problems also for these allocators. - Adding checks for definedness in appropriate places (eg. when calling libz). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Newly Added (by Knielsen): Using the Valgrind API in mysqld (23)
by worklog-noreply＠askmonty.org 22 May '09

22 May '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Using the Valgrind API in mysqld CREATION DATE..: Fri, 22 May 2009, 11:43 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 23 (http://askmonty.org/worklog/?tid=23) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: DESCRIPTION: Valgrind (the memcheck tool) has some very useful APIs that can be used in mysqld when testing with Valgrind to improve testing and/or debugging: file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.clientreqs file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.mempools This worklog is about adding configure checks and headers to allow to use these in a way that continues to work on machines where the Valgrind headers or functionality is missing. It also includes adding some basic Valgrind enhancements: - Adding Valgrind annotations to custom memory allocators so that Valgrind can detect leaks, use-before-init, and use-after-free problems also for these allocators. - Adding checks for definedness in appropriate places (eg. when calling libz). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Knielsen): Using the Valgrind API in mysqld (23)
by worklog-noreply＠askmonty.org 22 May '09

22 May '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Using the Valgrind API in mysqld CREATION DATE..: Fri, 22 May 2009, 11:43 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 23 (http://askmonty.org/worklog/?tid=23) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: DESCRIPTION: Valgrind (the memcheck tool) has some very useful APIs that can be used in mysqld when testing with Valgrind to improve testing and/or debugging: file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.clientreqs file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.mempools This worklog is about adding configure checks and headers to allow to use these in a way that continues to work on machines where the Valgrind headers or functionality is missing. It also includes adding some basic Valgrind enhancements: - Adding Valgrind annotations to custom memory allocators so that Valgrind can detect leaks, use-before-init, and use-after-free problems also for these allocators. - Adding checks for definedness in appropriate places (eg. when calling libz). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): Implement mysql-test output parser for Buildbot (22)
by worklog-noreply＠askmonty.org 21 May '09

21 May '09

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Implement mysql-test output parser for Buildbot CREATION DATE..: Thu, 21 May 2009, 22:19 SUPERVISOR.....: Knielsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Other TASK ID........: 22 (http://askmonty.org/worklog/?tid=22) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 30 (hours remain) ORIG. ESTIMATE.: 30 PROGRESS NOTES: -=-=(Knielsen - Thu, 21 May 2009, 22:33)=-=- High Level Description modified. --- /tmp/wklog.22.old.27679 2009-05-21 22:33:35.000000000 +0300 +++ /tmp/wklog.22.new.27679 2009-05-21 22:33:35.000000000 +0300 @@ -9,3 +9,6 @@ http://djmitche.github.com/buildbot/docs/0.7.10/#Writing-New-BuildSteps +Later, once we get the infrastructure to write Buildbot results into a MySQL +database, we want to extend this to also insert into the database all test +failures and the mysqltest failure output (for cross-reference search). DESCRIPTION: Like in Pushbuild at MySQL AB, we want to have buildbot parse the output of mysql-test-run for test errors so we can display on the front page the name of any tests that failed. The parser can also count the number of tests completed so far so Buildbot can provide more accurate completion estimates. Buildbot already has support for plugging in such modules. See eg. http://djmitche.github.com/buildbot/docs/0.7.10/#Writing-New-BuildSteps Later, once we get the infrastructure to write Buildbot results into a MySQL database, we want to extend this to also insert into the database all test failures and the mysqltest failure output (for cross-reference search). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2703)
by knielsen＠knielsen-hq.org 20 May '09

20 May '09

#At lp:maria 2703 knielsen(a)knielsen-hq.org 2009-05-20 Fix accessing ulong enum option as uint, failing on 64-bit big-endian. modified: mysys/my_getopt.c === modified file 'mysys/my_getopt.c' --- a/mysys/my_getopt.c 2009-04-25 10:05:32 +0000 +++ b/mysys/my_getopt.c 2009-05-20 15:34:34 +0000 @@ -647,7 +647,7 @@ static int setval(const struct my_option return EXIT_OUT_OF_MEMORY; break; case GET_ENUM: - if (((*(int*)result_pos)= find_type(argument, opts->typelib, 2) - 1) < 0) + if (((*(ulong *)result_pos)= find_type(argument, opts->typelib, 2) - 1) < 0) return EXIT_ARGUMENT_INVALID; break; case GET_SET: @@ -983,7 +983,7 @@ static void init_one_value(const struct *((int*) variable)= (int) getopt_ll_limit_value((int) value, option, NULL); break; case GET_ENUM: - *((uint*) variable)= (uint) value; + *((ulong*) variable)= (uint) value; break; case GET_UINT: *((uint*) variable)= (uint) getopt_ull_limit_value((uint) value, option, NULL); @@ -1221,7 +1221,7 @@ void my_print_variables(const struct my_ } break; case GET_ENUM: - printf("%s\n", get_type(optp->typelib, *(uint*) value)); + printf("%s\n", get_type(optp->typelib, *(ulong*) value)); break; case GET_STR: case GET_STR_ALLOC: /* fall through */

2 1

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2699)
by knielsen＠knielsen-hq.org 20 May '09

20 May '09

#At lp:maria 2699 knielsen(a)knielsen-hq.org 2009-05-11 [merge] Merge Toby Thain's Solaris fixes. modified: BUILD/SETUP.sh BUILD/compile-solaris-amd64 BUILD/compile-solaris-amd64-debug* BUILD/compile-solaris-amd64-forte BUILD/compile-solaris-amd64-forte-debug* storage/innobase/include/univ.i storage/pbxt/src/Makefile.am storage/pbxt/src/lock_xt.h storage/pbxt/src/pbms.h === modified file 'BUILD/SETUP.sh' --- a/BUILD/SETUP.sh 2009-05-06 12:03:24 +0000 +++ b/BUILD/SETUP.sh 2009-05-11 15:31:30 +0000 @@ -80,7 +80,13 @@ path=`dirname $0` . "$path/check-cpu" export AM_MAKEFLAGS -AM_MAKEFLAGS="-j 6" +# Default to a parallel build, but only if AM_MAKEFLAGS is not set. +# (So buildbots can easily disable this behaviour if required.) +if test -z "$AM_MAKEFLAGS" +then + AM_MAKEFLAGS="-j 6" +fi + # SSL library to use.--with-ssl will select our bundled yaSSL # implementation of SSL. To use openSSl you will nee too point out === modified file 'BUILD/compile-solaris-amd64' --- a/BUILD/compile-solaris-amd64 2007-04-12 11:20:38 +0000 +++ b/BUILD/compile-solaris-amd64 2009-05-09 04:01:53 +0000 @@ -1,55 +1,32 @@ -#!/usr/bin/bash +#!/bin/sh -function _find_mysql_root () ( - while [ "x$PWD" != "x/" ]; do - # Check if some directories are present - if [ -d BUILD -a -d sql -a -d mysys ]; then - echo "$PWD" - return 0 - fi - cd .. - done - return 1 -) +# Build setup for Solaris 10 + +# Make the Sun Freeware packages use the bundled Perl, instead of their own: +# ln -s `which perl` /usr/local/bin +# Your $PATH needs to include (in this order): +# /usr/local/bin:/usr/sfw/bin:/usr/ccs/bin +# (For Sun Freeware, bundled GNU utilities, Solaris ar, etc.) +# +# Required packages from http://sunfreeware.com/indexintel10.html : +# (The GNU m4 bundled with Solaris is too old.) +# automake-1.10.2-sol10-x86-local.gz +# autoconf-2.63-sol10-x86-local.gz +# m4-1.4.12-sol10-x86-local.gz +# libsigsegv-2.6-sol10-x86-local.gz +# libtool-1.5.24-sol10-x86-local.gz +# ( how to install these packages: + # wget ftp://ftp.sunfreeware.com/pub/freeware/intel/10/automake-1.10.2-sol10-x86-l… + # gunzip automake-1.10.2-sol10-x86-local.gz + # pkgadd -d automake-1.10.2-sol10-x86-local +# ) -make -k clean || true -/bin/rm -f */.deps/*.P config.cache - path=`dirname $0` -. "$path/autorun.sh" - -warning_flags="-Wimplicit -Wreturn-type -Wswitch -Wtrigraphs -Wcomment -W -Wchar-subscripts -Wformat -Wparentheses -Wsign-compare -Wwrite-strings -Wunused" -compiler_flags="-g -O3 -fno-omit-frame-pointer" +. "$path/SETUP.sh" +extra_flags="$amd64_cflags -D__sun -m64 -mtune=athlon64" +extra_configs="$amd64_configs $max_configs --with-libevent" -export CC CXX CFLAGS CXXFLAGS LDFLAGS LIBS -CC="gcc" -CXX="gcc" -CFLAGS="$warning_flags $compiler_flags" -CXXFLAGS="" -LDFLAGS="-O3 -g -static-libgcc" -LIBS=-lmtmalloc -root=$(_find_mysql_root) +LDFLAGS="-lmtmalloc -static-libgcc" +export LDFLAGS -$root/configure \ - --prefix=/usr/local/mysql \ - --localstatedir=/usr/local/mysql/data \ - --libexecdir=/usr/local/mysql/bin \ - --with-extra-charsets=complex \ - --enable-thread-safe-client \ - --enable-local-infile \ - --with-zlib-dir=bundled \ - --with-big-tables \ - --with-readline \ - --with-archive-storage-engine \ - --with-named-curses=-lcurses \ - --with-big-tables \ - --with-innodb \ - --with-berkeley-db \ - --with-example-storage-engine \ - --with-blackhole-storage-engine \ - --with-ndbcluster \ - --with-federated-storage-engine \ - --with-csv-storage-engine \ - --with-ssl \ - --with-embedded-server \ - --disable-shared +. "$path/FINISH.sh" === modified file 'BUILD/compile-solaris-amd64-debug' (properties changed: -x to +x) --- a/BUILD/compile-solaris-amd64-debug 2007-11-10 10:03:07 +0000 +++ b/BUILD/compile-solaris-amd64-debug 2009-05-09 04:01:53 +0000 @@ -1,10 +1,11 @@ -#! /bin/sh +#!/bin/sh + path=`dirname $0` . "$path/SETUP.sh" -amd64_cflags="-m64 -mtune=athlon64" -extra_flags="$amd64_cflags $debug_cflags $max_cflags" -c_warnings="$c_warnings $debug_extra_warnings" -cxx_warnings="$cxx_warnings $debug_extra_warnings" -extra_configs="$amd64_configs $debug_configs $max_configs --enable-thread-safe-client" +extra_flags="$amd64_cflags -D__sun -m64 -mtune=athlon64 $debug_cflags" +extra_configs="$amd64_configs $debug_configs $max_configs --with-libevent" + +LDFLAGS="-lmtmalloc -static-libgcc" +export LDFLAGS . "$path/FINISH.sh" === modified file 'BUILD/compile-solaris-amd64-forte' --- a/BUILD/compile-solaris-amd64-forte 2008-09-30 20:57:48 +0000 +++ b/BUILD/compile-solaris-amd64-forte 2009-05-09 04:01:53 +0000 @@ -1,53 +1,29 @@ -#! /bin/sh +#!/bin/sh + +# See file compile-solaris-amd64 for basic pre-requisites. + +# This build uses the Sun Studio compilers (cc, CC), available from: +# http://developers.sun.com/sunstudio/downloads/index.jsp +# Note that you may want to apply current patches, as the downloaded version +# is typically out of date. Download the PKG version if you intend to patch! + +# After installing, add /opt/SUNWspro/bin to your $PATH + -gmake -k clean || true -/bin/rm -f */.deps/*.P config.cache - path=`dirname $0` -. "$path/autorun.sh" +. "$path/SETUP.sh" -# For "optimal" code for this computer add -fast to EXTRA -# To compile 64 bit, add -m64 to EXTRA_64_BIT +extra_flags="-m64 -mt -D_FORTEC_ -xbuiltin=%all -xlibmil -xlibmopt -fns=no -xprefetch=auto -xprefetch_level=3" +extra_configs="$max_configs --with-libevent" -EXTRA_64_BIT="-m64" -EXTRA="-fast" +warnings="" +c_warnings="" +cxx_warnings="" +base_cxxflags="-noex" -# -# The following should not need to be touched -# - -export CC CXX CFLAGS CXXFLAGS LIBS -STD="-g -mt -D_FORTEC_ $EXTRA $EXTRA_64_BIT" -ASFLAGS="$EXTRA_64_BIT" -CC=cc-5.0 -CFLAGS="-Xa -xstrconst $STD" +CC=cc +CFLAGS="-xstrconst" CXX=CC -CXXFLAGS="-noex $STD" -LIBS=-lmtmalloc -./configure \ - --prefix=/usr/local/mysql \ - --localstatedir=/usr/local/mysql/data \ - --libexecdir=/usr/local/mysql/bin \ - --with-extra-charsets=complex \ - --enable-thread-safe-client \ - --enable-local-infile \ - --with-zlib-dir=bundled \ - --with-big-tables \ - --with-readline \ - --with-archive-storage-engine \ - --with-named-curses=-lcurses \ - --with-big-tables \ - --with-innodb \ - --with-example-storage-engine \ - --with-blackhole-storage-engine \ - --with-federated-storage-engine \ - --with-csv-storage-engine \ - --with-ssl \ - --enable-assembler - -# Not including: -# --with-ndbcluster -# --with-berkeley-db +LDFLAGS="-lmtmalloc" -gmake -j4 -test $? = 0 && make test +. "$path/FINISH.sh" === modified file 'BUILD/compile-solaris-amd64-forte-debug' (properties changed: -x to +x) --- a/BUILD/compile-solaris-amd64-forte-debug 2008-09-30 20:57:48 +0000 +++ b/BUILD/compile-solaris-amd64-forte-debug 2009-05-09 04:01:53 +0000 @@ -1,54 +1,27 @@ -#! /bin/sh +#!/bin/sh -gmake -k clean || true -/bin/rm -f */.deps/*.P config.cache - path=`dirname $0` -. "$path/autorun.sh" +. "$path/SETUP.sh" -# To compile 64 bit, add -m64 to EXTRA_64_BIT -EXTRA_64_BIT="-m64" +# Take only #define options - the others are gcc specific. +# (real fix is for SETUP.sh not to put gcc specific options in $debug_cflags) +DEFS="" +for F in $debug_cflags ; do + expr "$F" : "^-D" && DEFS="$DEFS $F" +done +debug_cflags="-O0 -g $DEFS" + +extra_flags="-m64 -mt -D_FORTEC_ -xlibmopt -fns=no $debug_cflags" +extra_configs="$max_configs --with-libevent $debug_configs" + +warnings="" +c_warnings="" +cxx_warnings="" +base_cxxflags="-noex" -# For "optimal" code for this computer add -fast to EXTRA. Note that -# this causes problem with debugging the program since -fast implies -# -xO5. -EXTRA="" - -# -# The following should not need to be touched -# - -export CC CXX CFLAGS CXXFLAGS -STD="-g -mt -D_FORTEC_ $EXTRA $EXTRA_64_BIT $debug_cflags" -ASFLAGS="$EXTRA_64_BIT" -CC=cc-5.0 -CFLAGS="-Xa -xstrconst $STD" +CC=cc +CFLAGS="-xstrconst" CXX=CC -CXXFLAGS="-noex $STD" -./configure \ - --prefix=/usr/local/mysql \ - --localstatedir=/usr/local/mysql/data \ - --libexecdir=/usr/local/mysql/bin \ - --with-extra-charsets=complex \ - --enable-thread-safe-client \ - --enable-local-infile \ - --with-zlib-dir=bundled \ - --with-big-tables \ - --with-readline \ - --with-archive-storage-engine \ - --with-named-curses=-lcurses \ - --with-big-tables \ - --with-innodb \ - --with-example-storage-engine \ - --with-blackhole-storage-engine \ - --with-federated-storage-engine \ - --with-csv-storage-engine \ - --with-ssl \ - --with-debug \ - --enable-assembler - -# Not including: -# --with-ndbcluster -# --with-berkeley-db +LDFLAGS="-lmtmalloc" -gmake -j4 +. "$path/FINISH.sh" === modified file 'storage/innobase/include/univ.i' --- a/storage/innobase/include/univ.i 2009-05-06 12:03:24 +0000 +++ b/storage/innobase/include/univ.i 2009-05-11 15:31:30 +0000 @@ -55,10 +55,10 @@ of the 32-bit x86 assembler in mutex ope # define UNIV_CAN_USE_X86_ASSEMBLER # endif -/* We only try to do explicit inlining of functions with gcc and -Microsoft Visual C++ */ +/* Enable explicit inlining of functions only for compilers known to +support it. */ -# if !defined(__GNUC__) +# if !defined(__GNUC__) && !defined(__SUNPRO_C) # undef UNIV_MUST_NOT_INLINE /* Remove compiler warning */ # define UNIV_MUST_NOT_INLINE # endif === modified file 'storage/pbxt/src/Makefile.am' --- a/storage/pbxt/src/Makefile.am 2009-04-01 06:40:11 +0000 +++ b/storage/pbxt/src/Makefile.am 2009-05-09 04:01:53 +0000 @@ -46,7 +46,7 @@ libpbxt_la_CFLAGS = $(AM_CFLAGS) -DMYSQ EXTRA_LIBRARIES = libpbxt.a noinst_LIBRARIES = libpbxt.a libpbxt_a_SOURCES = $(libpbxt_la_SOURCES) -libpbxt_a_CXXFLAGS = $(AM_CXXFLAGS) -Wno-long-long +libpbxt_a_CXXFLAGS = $(AM_CXXFLAGS) libpbxt_a_CFLAGS = $(AM_CFLAGS) -std=c99 EXTRA_DIST = CMakeLists.txt === modified file 'storage/pbxt/src/lock_xt.h' --- a/storage/pbxt/src/lock_xt.h 2009-04-02 10:03:14 +0000 +++ b/storage/pbxt/src/lock_xt.h 2009-05-09 04:01:53 +0000 @@ -208,9 +208,9 @@ inline void xt_atomic_dec2(volatile xtWo #elif defined(__GNUC__) __sync_fetch_and_sub(mptr, 1); #elif defined(XT_SPL_SOLARIS_LIB) - val1 = atomic_dec_16_nv(mptr); + atomic_dec_16_nv(mptr); #else - val1 = --(*mptr); + --(*mptr); #endif } === modified file 'storage/pbxt/src/pbms.h' --- a/storage/pbxt/src/pbms.h 2009-03-26 12:18:01 +0000 +++ b/storage/pbxt/src/pbms.h 2009-05-09 04:01:53 +0000 @@ -782,21 +782,28 @@ private: void deleteTempFiles() { - struct dirent entry; + struct dirent *entry; struct dirent *result; DIR *odir; int err; + size_t sz; char temp_file[100]; +#ifdef XT_SOLARIS + sz = sizeof(struct dirent) + pathconf("/tmp/", _PC_NAME_MAX); // Solaris, see readdir(3C) +#else + sz = sizeof(struct dirent); +#endif + entry = (struct dirent*)malloc(sz); if (!(odir = opendir("/tmp/"))) return; - err = readdir_r(odir, &entry, &result); + err = readdir_r(odir, entry, &result); while (!err && result) { const char **prefix = temp_prefix; while (*prefix) { - if (startsWith(entry.d_name, *prefix)) { - int pid = atoi(entry.d_name + strlen(*prefix)); + if (startsWith(entry->d_name, *prefix)) { + int pid = atoi(entry->d_name + strlen(*prefix)); /* If the process does not exist: */ if (kill(pid, 0) == -1 && errno == ESRCH) { @@ -807,9 +814,10 @@ private: prefix++; } - err = readdir_r(odir, &entry, &result); + err = readdir_r(odir, entry, &result); } closedir(odir); + free(entry); } }; #endif // PBMS_API

3 3

[Maria-developers] compile issue and the -lz flag
by Don Kehn 19 May '09

19 May '09

Hi all: when compiling on x64 systems (i.e. CentOS & Fedora core 10) and using the BUILD/compile-pentium64-max, I'm getting the -lz error "/usr/bin/ld: cannot find -lz" and when I simplify the ./comfigure line to a --without-zlib it works. So, after modifing the configure flags that results into: ./configure --prefix=/u02/local/mysql-5.1.34 --enable-assembler --with-extra-charsets=complex --enable-thread-safe-client --with-big-tables --with-readline --with-ssl --with-plugins=max --with-embedded-server --with-mysqld-ldflags=-all-static --with-client-ldflags=-all-static --without-zlib --enable-local-infile It still has problems with the -lz flag, so does anybody know what is bringing the -lz into the configure? Note: I've also tried --with-zlibdir=/usr/lib64, but same error. Thanks tons.

4 11

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2700)
by knielsen＠knielsen-hq.org 18 May '09

18 May '09

#At lp:maria 2700 knielsen(a)knielsen-hq.org 2009-05-18 Fix support-files/build-tags for newer bzr which do not default to --recursive on bzr ls. modified: support-files/build-tags === modified file 'support-files/build-tags' --- a/support-files/build-tags 2008-08-06 20:25:25 +0000 +++ b/support-files/build-tags 2009-05-18 07:29:14 +0000 @@ -4,7 +4,7 @@ rm -f TAGS filter='\.cc$\|\.c$\|\.h$\|\.yy$' list="find . -type f" -bzr root >/dev/null 2>/dev/null && list="bzr ls --kind=file --versioned" +bzr root >/dev/null 2>/dev/null && list="bzr ls --recursive --kind=file --versioned" $list |grep $filter |while read f; do

1 0

[Maria-developers] ptr_compare replace with memcmp and associated performance
by Stewart Smith 17 May '09

17 May '09

So i broke out mtaylor's patch that we were bumming around with at the UC to replace ptr_compare with a simple memcmp call. At the UC I benched that this patch actually caused a measurable performance regresssion. So what's the difference? (same benchmark and machine as in previous mail) MAX_FIELDS=64 with std::bitset read/write requests: 70000 (3374.50 per sec.) read/write requests: 70000 (3102.42 per sec.) read/write requests: 70000 (3113.47 per sec.) read/write requests: 70000 (3401.89 per sec.) read/write requests: 70000 (3164.61 per sec.) AVERAGE= 3231.37 With ptr_compare replaced with memcmp: read/write requests: 70000 (3090.33 per sec.) read/write requests: 70000 (3066.97 per sec.) read/write requests: 70000 (2954.93 per sec.) read/write requests: 70000 (2875.57 per sec.) read/write requests: 70000 (2953.99 per sec.) AVERAGE= 2988.35 With ptr_compare replaced with __builtin_memcmp: read/write requests: 70000 (3245.37 per sec.) read/write requests: 70000 (3001.91 per sec.) read/write requests: 70000 (3254.99 per sec.) read/write requests: 70000 (3177.13 per sec.) read/write requests: 70000 (3195.49 per sec.) AVERAGE= 3174.97 So, I thought that with just using the __builtin_memcmp automatically, we may be okay with merging this. However... I do have the following concerns: 1) I'm pretty sure that __builtin_memcmp is gcc only, and that SunStudio may have an issue (yay - wrappers!) 2) Is this perf difference going to be true on different platforms? 3) the various memcmp implementations do seem to be dependent on a few things for performance: alignment, size of data. For what we're seeing in sysbench, the parameters are things like this: "SELECT c from sbtest where id between 9942 and 10041 order by c" 0xc30780,0xc2c340,240 0xc2e560,0xc2c340,240 0xc30780,0xc2e560,240 0xc37390,0xc32f50,240 0xc35170,0xc32f50,240 0xc37390,0xc35170,240 0xc3dcc8,0xc39888,240 0xc3baa8,0xc2e560,240 0xc35170,0xc2e560,240 0xc3baa8,0xc35170,240 0xc2c340,0xc35170,240 0xc35170,0xc3dcc8,240 0xc2c618,0xc35170,240 0xc35170,0xc3d9f0,240 0xc2c8f0,0xc35170,240 0x7fbed85f10f0,0x7fbed85eccb0,240 and from the CREATE TABLE: `c` VARCHAR(120) NOT NULL COLLATE utf8_general_ci DEFAULT `` , So here it's large, and aligned. For some other micro-benchmarks I've clocked things looking much different.... so it's possibly query dependent as to what ends up being better (i.e. how much data we're comparing). (1st parameter is number of repetitions, 2nd is number of bytes to memcmp) For comparing equal values: stewart@willster:~/src/test/memcmp$ ./a.out 168435455 8 168435455 repetitions Testing memcmp ..... done, 7.432 seconds Testing builtin memcmp ....... done, 16.313 seconds Testing loop .... done, 6.144 seconds Testing loop32 .... done, 2.764 seconds Testing loop64 .... done, 2.128 seconds Testing no-op .... done, 1.488 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 16 168435455 repetitions Testing memcmp ..... done, 5.912 seconds Testing builtin memcmp ....... done, 26.418 seconds Testing loop .... done, 11.201 seconds Testing loop32 .... done, 3.980 seconds Testing loop64 .... done, 2.564 seconds Testing no-op .... done, 1.492 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 32 168435455 repetitions Testing memcmp ..... done, 6.828 seconds Testing builtin memcmp ....... done, 46.891 seconds Testing loop .... done, 21.473 seconds Testing loop32 .... done, 6.536 seconds Testing loop64 .... done, 3.804 seconds Testing no-op .... done, 1.476 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 64 168435455 repetitions Testing memcmp ..... done, 9.549 seconds Testing builtin memcmp ....... done, 87.513 seconds Testing loop .... done, 41.455 seconds Testing loop32 .... done, 11.669 seconds Testing loop64 .... done, 6.368 seconds Testing no-op .... done, 1.468 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 128 168435455 repetitions Testing memcmp ..... done, 15.081 seconds Testing builtin memcmp ....... done, 169.143 seconds Testing loop .... done, 86.397 seconds Testing loop32 .... done, 21.877 seconds Testing loop64 .... done, 11.445 seconds Testing no-op .... done, 1.488 seconds stewart@willster:~/src/test/memcmp$ gcc -O3 -fno-builtin memcmpbench.c stewart@willster:~/src/test/memcmp$ ./a.out 168435455 256 168435455 repetitions Testing memcmp ..... done, 26.134 seconds Testing loop64 .... done, 21.549 seconds Testing no-op .... done, 1.500 seconds Completely inequal values are all about the same: stewart@willster:~/src/test/memcmp$ ./a.out 168435455 8 168435455 repetitions Testing memcmp ..... done, 3.204 seconds Testing builtin memcmp ....... done, 13.945 seconds Testing loop .... done, 1.896 seconds Testing loop32 .... done, 2.092 seconds Testing loop64 .... done, 2.136 seconds Testing no-op .... done, 1.488 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 240 168435455 repetitions Testing memcmp ..... done, 5.520 seconds Testing builtin memcmp ....... done, 13.973 seconds Testing loop .... done, 1.912 seconds Testing loop32 .... done, 2.124 seconds Testing loop64 .... done, 2.104 seconds Testing no-op .... done, 1.468 seconds For half the same: stewart@willster:~/src/test/memcmp$ ./a.out 168435455 8 168435455 repetitions Testing memcmp ..... done, 7.340 seconds Testing builtin memcmp ....... done, 19.193 seconds Testing loop .... done, 4.212 seconds Testing loop32 .... done, 2.956 seconds Testing loop64 .... done, 2.112 seconds Testing no-op .... done, 1.476 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 32 168435455 repetitions Testing memcmp ..... done, 5.920 seconds Testing builtin memcmp ....... done, 34.526 seconds Testing loop .... done, 11.885 seconds Testing loop32 .... done, 4.444 seconds Testing loop64 .... done, 3.388 seconds Testing no-op .... done, 1.468 seconds stewart@willster:~/src/test/memcmp$ ./a.out 168435455 64 168435455 repetitions Testing memcmp ..... done, 6.948 seconds Testing builtin memcmp ....... done, 54.923 seconds Testing loop .... done, 22.081 seconds Testing loop32 .... done, 7.000 seconds Testing loop64 .... done, 4.444 seconds Testing no-op .... done, 1.488 seconds Is concurrency an issue here and not raw single threaded performance? Let's try with 64 threads (on the same 2 core box): 64 threads, __builtin_memcmp: read/write requests: 70000 (3184.20 per sec.) read/write requests: 70000 (3184.87 per sec.) read/write requests: 70000 (3166.83 per sec.) read/write requests: 70000 (3085.69 per sec.) AVERAGE=3155.39 64 threads, memcmp: read/write requests: 70000 (3218.07 per sec.) read/write requests: 70000 (3219.04 per sec.) read/write requests: 70000 (3222.48 per sec.) read/write requests: 70000 (3116.00 per sec.) AVERAGE=3193.89 64 threads, 32bit cmp loop: read/write requests: 70000 (3173.76 per sec.) read/write requests: 70000 (3156.23 per sec.) read/write requests: 70000 (3206.70 per sec.) read/write requests: 70000 (3247.61 per sec.) AVERAGE=3196.07 64 threads, 64bit cmp loop: read/write requests: 70000 (3210.02 per sec.) read/write requests: 70000 (3218.87 per sec.) read/write requests: 70000 (3225.98 per sec.) read/write requests: 70000 (3131.60 per sec.) AVERAGE= 3196.61 64 threads, baseline (using original ptr_compare): read/write requests: 70000 (3542.95 per sec.) read/write requests: 70000 (3556.15 per sec.) read/write requests: 70000 (3560.71 per sec.) read/write requests: 70000 (3476.80 per sec.) AVERAGE=3534.15 i.e. the old ptr_compare whoops the arse of any of the memcmp calls with higher concurrency. So, after this I can conclude: - memcmp is nothing if not consistent across various microbenchmarks - builtin_memcmp seems to help a bit at lower concurrency, not at all at higher and is seldom useful in micro benchmark. - the 64bit loop isn't so good at higher concurrency - the 64bit loop is favourable in microbenchmarks - memcmp is close to the 64bit loop performance (at most only 2x slower) Unrolling loops being the key? Testing loop .... done, 11.813 seconds Testing unrollloop .... done, 7.628 seconds Testing unrollloop32 .... done, 2.532 seconds Testing unrollloop64 .... done, 2.536 seconds Testing loop32 .... done, 4.432 seconds Testing loop64 .... done, 3.792 seconds Testing no-op .... done, 1.276 seconds (32 and 64 do 32,64 bit compares in an unrolled loop) Questions: - Why does my 64bit compare loop not equal performance of ptr_compare unrolled loop? (in a microbenchmark, it beats it). Concurrency again? So what should we do? Certainly a call to memcmp is easier to understand and keeps the code nice and simple, but we're talking up to a 10% speed hit here at higher concurrency (at least on my hardware). I wonder what the difference is on various other hardware setups. I'd be very interested to see what happens on sparc. I'd also like to see it micro-benchmarked - preferably something we could repeat in future (even in ./configure ? on server startup ?) I'm voting to keep the current ptr_compare code, and hope that somebody provides further explanation on top of what I've looked at here. Below is the patch, followed by the code i used for microbenchmarking (note the commented out 64bit loop i used for the above loop tests): === modified file 'drizzled/filesort.cc' --- drizzled/filesort.cc 2009-04-28 00:17:10 +0000 +++ drizzled/filesort.cc 2009-05-08 05:59:11 +0000 @@ -1135,7 +1135,7 @@ int merge_buffers(SORTPARAM *param, IO_C } else { - cmp= get_ptr_compare(sort_length); + cmp= (qsort2_cmp)ptr_compare; first_cmp_arg= (void*) &sort_length; } priority_queue<BUFFPEK *, vector<BUFFPEK *>, compare_functor > === modified file 'mysys/mf_sort.cc' --- mysys/mf_sort.cc 2009-04-17 21:01:47 +0000 +++ mysys/mf_sort.cc 2009-05-08 05:59:11 +0000 @@ -34,7 +34,7 @@ void my_string_ptr_sort(unsigned char *b { if (size && items) { - my_qsort2(base,items, sizeof(unsigned char*), get_ptr_compare(size), + my_qsort2(base,items, sizeof(unsigned char*), (qsort2_cmp)ptr_compare, (void*) &size); } } === modified file 'mysys/my_sys.h' --- mysys/my_sys.h 2009-04-27 22:05:43 +0000 +++ mysys/my_sys.h 2009-05-08 05:59:11 +0000 @@ -420,7 +420,12 @@ extern void my_qsort(void *base_ptr, siz qsort_cmp cmp); extern void my_qsort2(void *base_ptr, size_t total_elems, size_t size, qsort2_cmp cmp, void *cmp_argument); -extern qsort2_cmp get_ptr_compare(size_t); + +#if defined(__cplusplus) +extern "C" +#endif +int ptr_compare(size_t *compare_length, unsigned char **a, unsigned char **b); + void my_store_ptr(unsigned char *buff, size_t pack_length, my_off_t pos); my_off_t my_get_ptr(unsigned char *ptr, size_t pack_length); File create_temp_file(char *to, const char *dir, const char *pfx, === modified file 'mysys/ptr_cmp.cc' --- mysys/ptr_cmp.cc 2009-04-26 16:53:32 +0000 +++ mysys/ptr_cmp.cc 2009-05-08 05:59:11 +0000 @@ -21,137 +21,33 @@ #include "mysys/mysys_priv.h" #include "plugin/myisam/myisampack.h" - -static int ptr_compare(size_t *compare_length, unsigned char **a, unsigned char **b); -static int ptr_compare_0(size_t *compare_length, unsigned char **a, unsigned char **b); -static int ptr_compare_1(size_t *compare_length, unsigned char **a, unsigned char **b); -static int ptr_compare_2(size_t *compare_length, unsigned char **a, unsigned char **b); -static int ptr_compare_3(size_t *compare_length, unsigned char **a, unsigned char **b); - - /* Get a pointer to a optimal byte-compare function for a given size */ - -qsort2_cmp get_ptr_compare (size_t size) -{ - if (size < 4) - return (qsort2_cmp) ptr_compare; - switch (size & 3) { - case 0: return (qsort2_cmp) ptr_compare_0; - case 1: return (qsort2_cmp) ptr_compare_1; - case 2: return (qsort2_cmp) ptr_compare_2; - case 3: return (qsort2_cmp) ptr_compare_3; - } - return 0; /* Impossible */ -} - +#include <string.h> /* Compare to keys to see witch is smaller. - Loop unrolled to make it quick !! */ -#define cmp(N) if (first[N] != last[N]) return (int) first[N] - (int) last[N] - -static int ptr_compare(size_t *compare_length, unsigned char **a, unsigned char **b) -{ - register int length= *compare_length; - register unsigned char *first,*last; - - first= *a; last= *b; - while (--length) - { - if (*first++ != *last++) - return (int) first[-1] - (int) last[-1]; - } - return (int) first[0] - (int) last[0]; -} - - -static int ptr_compare_0(size_t *compare_length,unsigned char **a, unsigned char **b) +extern "C" +int ptr_compare(size_t *compare_length, unsigned char **a, unsigned char **b) { - register int length= *compare_length; - register unsigned char *first,*last; - - first= *a; last= *b; - loop: - cmp(0); - cmp(1); - cmp(2); - cmp(3); - if ((length-=4)) +/* size_t l= *compare_length / sizeof(uint64_t); + uint64_t *aa= (uint64_t*)*a; + uint64_t *bb= (uint64_t*)*b; + while(l--) { - first+=4; - last+=4; - goto loop; - } - return (0); -} - - -static int ptr_compare_1(size_t *compare_length,unsigned char **a, unsigned char **b) -{ - register int length= *compare_length-1; - register unsigned char *first,*last; - - first= *a+1; last= *b+1; - cmp(-1); - loop: - cmp(0); - cmp(1); - cmp(2); - cmp(3); - if ((length-=4)) - { - first+=4; - last+=4; - goto loop; - } - return (0); -} - -static int ptr_compare_2(size_t *compare_length,unsigned char **a, unsigned char **b) -{ - register int length= *compare_length-2; - register unsigned char *first,*last; - - first= *a +2 ; last= *b +2; - cmp(-2); - cmp(-1); - loop: - cmp(0); - cmp(1); - cmp(2); - cmp(3); - if ((length-=4)) - { - first+=4; - last+=4; - goto loop; + if(*aa != *bb) + { + if(*aa < *bb) + return -1; + else + return 1; + } + aa++, bb++; } - return (0); + return 0; +*/ return memcmp(*a, *b, *compare_length); } -static int ptr_compare_3(size_t *compare_length,unsigned char **a, unsigned char **b) -{ - register int length= *compare_length-3; - register unsigned char *first,*last; - - first= *a +3 ; last= *b +3; - cmp(-3); - cmp(-2); - cmp(-1); - loop: - cmp(0); - cmp(1); - cmp(2); - cmp(3); - if ((length-=4)) - { - first+=4; - last+=4; - goto loop; - } - return (0); -} void my_store_ptr(unsigned char *buff, size_t pack_length, my_off_t pos) { For those interested, i started with the code at http://gcc.gnu.org/ml/gcc/2002-10/msg01666.html and ended up with something like this: #include <sys/resource.h> #include <sys/time.h> #include <stdio.h> #include <assert.h> #include <stdlib.h> char s1[256]; char s2[256]; int cmpsize= 240; int speed_memcmp () { return memcmp (s1, s2, cmpsize); } int speed_bimemcmp () { return __builtin_memcmp (s1, s2, cmpsize); } int speed_loop () { int i=cmpsize; char *a= s1; char *b= s2; while(i--) { if(*a != *b) return -1; a++, b++; }; return 0; } int speed_loop32 () { int i=cmpsize/4; int *a= (int*)s1; int *b= (int*)s2; while(i--) { if(*a != *b) return -1; a++, b++; }; return 0; } int speed_loop64 () { int i=cmpsize/8; long long *a= (long long*)s1; long long *b= (long long*)s2; while(i--) { if(*a != *b) return -1; a++, b++; }; return 0; } int speed_null() { return 0; } double do_test (int repetitions, int (*test_function) ()) { struct rusage r1, r2; getrusage(RUSAGE_SELF, &r1); while (repetitions--) test_function (); getrusage(RUSAGE_SELF, &r2); return (r2.ru_utime.tv_sec - r1.ru_utime.tv_sec) + (r2.ru_utime.tv_usec - r1.ru_utime.tv_usec) / 1000000.0; } main(int argc, char **argv) { int repetitions = (argc == 1) ? 0x0fffffff : atoi(argv[1]); cmpsize = (argc == 2) ? 240 : atoi(argv[2]); memset(s1, 42, 256); memset(s2, 42, 256); memset(s2+(cmpsize/2), 55, cmpsize/2); printf ("%d repetitions\n\n", repetitions); printf ("Testing memcmp ....."); fflush (stdout); printf (" done, %10.3f seconds\n", do_test (repetitions, speed_memcmp)); printf ("Testing builtin memcmp ......."); fflush (stdout); printf (" done, %10.3f seconds\n", do_test (repetitions, speed_bimemcmp)); printf ("Testing loop ...."); fflush (stdout); printf (" done, %10.3f seconds\n", do_test (repetitions, speed_loop)); printf ("Testing loop32 ...."); fflush (stdout); printf (" done, %10.3f seconds\n", do_test (repetitions, speed_loop32)); printf ("Testing loop64 ...."); fflush (stdout); printf (" done, %10.3f seconds\n", do_test (repetitions, speed_loop64)); printf ("Testing no-op ...."); fflush (stdout); printf (" done, %10.3f seconds\n", do_test (repetitions, speed_null)); exit (0); } -- Stewart Smith

6 5