----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Benchmark suite for sysbench CREATION DATE..: Thu, 04 Mar 2010, 17:46 SUPERVISOR.....: Igor IMPLEMENTOR....: Hakan COPIES TO......: CATEGORY.......: Other TASK ID........: 100 (http://askmonty.org/worklog/?tid=100) VERSION........: Benchmarks-3.0 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 8 ESTIMATE.......: 32 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: -=-=(Hakan - Tue, 09 Mar 2010, 14:09)=-=- Low Level Design modified. --- /tmp/wklog.100.old.18803 2010-03-09 14:09:37.000000000 +0000 +++ /tmp/wklog.100.new.18803 2010-03-09 14:09:37.000000000 +0000 @@ -9,6 +9,12 @@ ** Number of concurrent clients is hardcoded ** Machine specific configuration like location of binaries and directories needed are in separate config files located at conf/<hostname>.inc + ** Set random seed of sysbench to have better comparison + ** Restart mysqld from scratch for each run and copy away + DATA_DIR of the database for faster starts. + ** Between each run, run sync and clear file system caches with + echo 3 > /proc/sys/vm/drop_caches (http://linux.die.net/man/5/proc) + ** Write out mysqld and sysbench options for reference. The main loop of run-sysbench.sh is: @@ -31,12 +37,6 @@ ** sar -u (CPU utilization) hook ** Crash detection ** Error detection - ** Set random seed of sysbench to have better comparision - ** Restart mysqld from scratch for each run and copy away - DATA_DIR of the database for faster starts. - ** Between each run, run sync and clear file system caches with - echo 3 > /proc/sys/vm/drop_caches (http://linux.die.net/man/5/proc) - ** Write out mysqld and sysbench options for reference. * Analyze numbers This is implemented in analyze-sysbench.php -=-=(Hakan - Tue, 09 Mar 2010, 14:06)=-=- Added: ** Set random seed of sysbench to have better comparison ** Restart mysqld from scratch for each run and copy away DATA_DIR of the database for faster starts. ** Between each run, run sync and clear file system caches with echo 3 > /proc/sys/vm/drop_caches (http://linux.die.net/man/5/proc) ** Write out mysqld and sysbench options for reference. Worked 8 hours and estimate 32 hours remain (original estimate unchanged). -=-=(Hakan - Mon, 08 Mar 2010, 12:27)=-=- Low Level Design modified. --- /tmp/wklog.100.old.21404 2010-03-08 12:27:08.000000000 +0000 +++ /tmp/wklog.100.new.21404 2010-03-08 12:27:08.000000000 +0000 @@ -36,6 +36,7 @@ DATA_DIR of the database for faster starts. ** Between each run, run sync and clear file system caches with echo 3 > /proc/sys/vm/drop_caches (http://linux.die.net/man/5/proc) + ** Write out mysqld and sysbench options for reference. * Analyze numbers This is implemented in analyze-sysbench.php -=-=(Guest - Thu, 04 Mar 2010, 18:15)=-=- Low Level Design modified. --- /tmp/wklog.100.old.2695 2010-03-04 18:15:46.000000000 +0000 +++ /tmp/wklog.100.new.2695 2010-03-04 18:15:46.000000000 +0000 @@ -1 +1,112 @@ +All scripts can be found at lp:mariadb-tools/sysbench +* Run sysbench tests on a machine and collect numbers +This is implemented in run-sysbench.sh. Currently it supports: + ** Optionally pull of latest source from Launchpad and compile + ** Starting the server + ** Running each sysbench test for $LOOP_COUNT times and + $RUN_TIME time. + ** Number of concurrent clients is hardcoded + ** Machine specific configuration like location of binaries and + directories needed are in separate config files located at conf/<hostname>.inc + +The main loop of run-sysbench.sh is: + +start_mysqld +for SYSBENCH_TEST in $SYSBENCH_TESTS + for THREADS in $NUM_THREADS + while [ $k -lt $LOOP_COUNT ] + drop schema sbtest + create schema sbtest + + $SYSBENCH $SYSBENCH_OPTIONS prepare + $SYSBENCH $SYSBENCH_OPTIONS run + done + done +done + +Open items: + ** OProfile hook + ** iostat hook + ** sar -u (CPU utilization) hook + ** Crash detection + ** Error detection + ** Set random seed of sysbench to have better comparision + ** Restart mysqld from scratch for each run and copy away + DATA_DIR of the database for faster starts. + ** Between each run, run sync and clear file system caches with + echo 3 > /proc/sys/vm/drop_caches (http://linux.die.net/man/5/proc) + +* Analyze numbers +This is implemented in analyze-sysbench.php + +Open items: + ** Read result files from + ${RESULT_DIR}/${TODAY}/${PRODUCT}/${SYSBENCH_TEST}/${THREADS}/results.txt + + ** Collect OProfile, iostat, cpu utilization, and machine info + ** Detect errors and crashes. + ** Generate SQL INSERT strings for presentation usage + +The layout for storing the numbers is: + CREATE TABLE sysbench_run ( + id int unsigned NOT NULL auto_increment, + host varchar(80), -- Hostname we ran the test on. + run_date date, -- The day we ran the test. + sysbench_version varchar(32), -- Version of sysbench we used. + test_name varchar(32), -- Name of the sysbench test. + run_time int unsigned, -- Run time in seconds. + runs int unsigned, -- Number of iterations of the test. + PRIMARY KEY (id), + KEY (host), + KEY (run_date + ); + + CREATE TABLE sysbench_comment ( + id int unsigned NOT NULL auto_increment, + sysbench_run_id int unsigned NOT NULL, -- FK pointing to sysbench_run. + compile_info text, -- Compile options we used. + machine_info text, -- Details about the hardware. + sysbench_options text, -- The sysbench options we used. + PRIMARY KEY (id), + KEY (sysbench_run_id) + ); + + CREATE TABLE sysbench_result ( + id int unsigned NOT NULL auto_increment, + sysbench_run_id int unsigned NOT NULL, -- FK pointing to sysbench_run. + concurrency int unsigned, -- Concurrency level we used. + result decimal(7,2), -- The actual result. + io varchar(80), -- The IO from iostat. + cpu varchar(80), -- CPU utilization. + profile text, -- Profiling information. + error text, -- Error messages and stack traces. + PRIMARY KEY (id), + KEY (sysbench_run_id + ); + +* Generate a report out of the numbers +This script will generate a HTML version for putting up on the web and a +txt version for email usage. + +Open items: + ** Generate an overview table in the form of + Number of threads + 1 4 8 16 32 64 128 +sysbench test + delete 121.52 144.77 117.70 115.15 100.48 75.39 66.56 + mean value of runs + 1 first run + 2 second run + 3 third run + STDEV + STDEV in % of mean + CPU utilization (usr/sys/wait/idle) + IO (read/write) + +For HTML version additionally generate a graph with JPGraph. + +* Get machine(s) and run the test on a weekly basis and for each release +comparing with the prior release. + +* Email weekly reports and blog about it. DESCRIPTION: Create a benchmark suite for running sysbench * Run sysbench tests on a machine and collect numbers * Analyze numbers * Generate a report out of the numbers LOW-LEVEL DESIGN: All scripts can be found at lp:mariadb-tools/sysbench * Run sysbench tests on a machine and collect numbers This is implemented in run-sysbench.sh. Currently it supports: ** Optionally pull of latest source from Launchpad and compile ** Starting the server ** Running each sysbench test for $LOOP_COUNT times and $RUN_TIME time. ** Number of concurrent clients is hardcoded ** Machine specific configuration like location of binaries and directories needed are in separate config files located at conf/<hostname>.inc ** Set random seed of sysbench to have better comparison ** Restart mysqld from scratch for each run and copy away DATA_DIR of the database for faster starts. ** Between each run, run sync and clear file system caches with echo 3 > /proc/sys/vm/drop_caches (http://linux.die.net/man/5/proc) ** Write out mysqld and sysbench options for reference. The main loop of run-sysbench.sh is: start_mysqld for SYSBENCH_TEST in $SYSBENCH_TESTS for THREADS in $NUM_THREADS while [ $k -lt $LOOP_COUNT ] drop schema sbtest create schema sbtest $SYSBENCH $SYSBENCH_OPTIONS prepare $SYSBENCH $SYSBENCH_OPTIONS run done done done Open items: ** OProfile hook ** iostat hook ** sar -u (CPU utilization) hook ** Crash detection ** Error detection * Analyze numbers This is implemented in analyze-sysbench.php Open items: ** Read result files from ${RESULT_DIR}/${TODAY}/${PRODUCT}/${SYSBENCH_TEST}/${THREADS}/results.txt ** Collect OProfile, iostat, cpu utilization, and machine info ** Detect errors and crashes. ** Generate SQL INSERT strings for presentation usage The layout for storing the numbers is: CREATE TABLE sysbench_run ( id int unsigned NOT NULL auto_increment, host varchar(80), -- Hostname we ran the test on. run_date date, -- The day we ran the test. sysbench_version varchar(32), -- Version of sysbench we used. test_name varchar(32), -- Name of the sysbench test. run_time int unsigned, -- Run time in seconds. runs int unsigned, -- Number of iterations of the test. PRIMARY KEY (id), KEY (host), KEY (run_date ); CREATE TABLE sysbench_comment ( id int unsigned NOT NULL auto_increment, sysbench_run_id int unsigned NOT NULL, -- FK pointing to sysbench_run. compile_info text, -- Compile options we used. machine_info text, -- Details about the hardware. sysbench_options text, -- The sysbench options we used. PRIMARY KEY (id), KEY (sysbench_run_id) ); CREATE TABLE sysbench_result ( id int unsigned NOT NULL auto_increment, sysbench_run_id int unsigned NOT NULL, -- FK pointing to sysbench_run. concurrency int unsigned, -- Concurrency level we used. result decimal(7,2), -- The actual result. io varchar(80), -- The IO from iostat. cpu varchar(80), -- CPU utilization. profile text, -- Profiling information. error text, -- Error messages and stack traces. PRIMARY KEY (id), KEY (sysbench_run_id ); * Generate a report out of the numbers This script will generate a HTML version for putting up on the web and a txt version for email usage. Open items: ** Generate an overview table in the form of Number of threads 1 4 8 16 32 64 128 sysbench test delete 121.52 144.77 117.70 115.15 100.48 75.39 66.56 mean value of runs 1 first run 2 second run 3 third run STDEV STDEV in % of mean CPU utilization (usr/sys/wait/idle) IO (read/write) For HTML version additionally generate a graph with JPGraph. * Get machine(s) and run the test on a weekly basis and for each release comparing with the prior release. * Email weekly reports and blog about it. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)