- developers - lists.mariadb.org

Re: [Maria-developers] [GSoC] self-tuning optimizer
by Sergei Golubchik 18 Jun '14

18 Jun '14

Hi, Anshu! Please, don't forget to cc: maria-developers@ when discussing development topics. On Jun 18, Anshu Avinash wrote: > > What I meant to say was we need N "linearly independent" equations to > solve for N variables and that might not be the case. Some of the > queries might not use any factor at all or use one factor more often. Yes, you're right, of course. > > Later we'll see what to do - perhaps we'll need to decrease the limit, > > or set it exactly to N (which means, the solution won't be "best > > fitting" anymore). But this will be not in this GSoC, not your project. > > I'm sure that in this project we won't have enough factors for memory > > usage to become an issue :) > > Yes, I'll start with a basic least squares fitting model without caring > much about the memory usage for now. Sounds good. Regards, Sergei

1 0

Re: [Maria-developers] [GSoC] self-tuning optimizer
by Sergei Golubchik 18 Jun '14

18 Jun '14

Hi, Anshu! On Jun 18, Anshu Avinash wrote: > Hi serg, > > Sorry that I had been out of touch for 1-2 days. I got caught up in some > other work. > > I have read through the comments you made on MDEV and I have just started > the implementation for the first approach, the approximate method. I have > pushed a prototype at: > https://github.com/igniting/server/commit/3bd4f5b6fd20d11575207ac36e8a3cc2c… > > Here is what I'm thinking, we add to the array of measurement data, after > each query and try to solve the system of equations because we don't know > when the system of equations would become solvable. Eh. To solve a system of linear equations with N unknowns, you need N equations. So, you wait until you get N equations and solve it. Few thoughts: 1. it might be that out of all factors that you have in your data structures (currently it's 2 + 2 * MAX_HA), only few will be actually used in queries (for example, only TIME_FOR_COMPARE and read_time for InnoDB). Then you only need two queries to solve that, not 2+2*MAX_HA. But it's an optimization, try to do the basic implementation without it first. 2. You normally store these data (for N queries) in the THD, right? It might be that the thread will disconnect before it'll do the required number of queries. To avoid throwing the useful data away, you can store one extra set of data globally and when some THD disconnects and it doesn't have enough data to solve the equation, you append this data to the global set and solve that as a whole. Again, this is an extension, only do it when the basic implementation works. > However this might be time consuming, trying to solve equations after > each query. Can we store the entire data (from all the queries) and > apply a best fitting model, instead of solving them exactly? This we > will do at disconnect. I don't have a good answer for that. It's a trade-off. Either you collect exactly N equations and solve them. Then you collect the next N equations (or you get another one and delete the first one) - and solve that. And so on. Or you collect many equations, more than N. And find a best fitting solution. This way you do less solving, but you'll need more memory. I think, for now you can do that - use a best fitting solution. We only have very few factors at the moment. Still, even now you'll need some limit to how many equations you'll collect. Later we'll see what to do - perhaps we'll need to decrease the limit, or set it exactly to N (which means, the solution won't be "best fitting" anymore). But this will be not in this GSoC, not your project. I'm sure that in this project we won't have enough factors for memory usage to become an issue :) > Should it be in ~THD() or somewhere else? Yes, ~THD is fine, just make sure you only do it for real connection THD's, where SELECT queries were issues and optimizer was run. Not for various internal THD's, like the one you create in cost_factors::init. For example, as you've noticed, I've mentioned in the last comment about using read_time() result and Nrows_estimated (the same applies to scan_time(), of course). So, when ha_read_time() is called, you need to remember (in your data set) the return value of read_time() and Nrows_estimated (that's the last argument of read_time()). This way, you only solve your equations if you have this read_time() and Nrows_estimated (or scan_time()) stored. If not - the optimizer was never run and you don't do anything. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 4257: Empty commit to check email setting in commit mails in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Roberto Spadim 18 Jun '14

18 Jun '14

working :) 2014-06-18 7:53 GMT-03:00 <knielsen(a)knielsen-hq.org>: > At http://bazaar.launchpad.net/~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4257 > revision-id: knielsen(a)knielsen-hq.org-20140618105333-44ez01iye29sljur > parent: knielsen(a)knielsen-hq.org-20140618090308-frd0tuo8rycgyi32 > committer: Kristian Nielsen <knielsen(a)knielsen-hq.org> > branch nick: work-10.0 > timestamp: Wed 2014-06-18 12:53:33 +0200 > message: > Empty commit to check email setting in commit mails > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits > -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

[Maria-developers] [MDEV-4903] Regarding range support for multi-column IN predicates
by Sergey Petrunia 18 Jun '14

18 Jun '14

Hi, So, MySQL 5.7 has got support for range optimization for multi-column IN predicates: http://optimize-this.blogspot.ru/2013/12/range-access-now-in-in-predicate-n… We've also got a user issue about it: https://mariadb.atlassian.net/browse/MDEV-4903. I'm looking at their IN handling. I don't see anything apparently wrong with it. They don't process multi-column NOT IN, but one can say that they just have a limited scope. Relevant code in opt_range.cc for single-column IN: case Item_func::IN_FUNC: { Item_func_in *func=(Item_func_in*) cond_func; if (inv) { // complicated code to handle NOT IN } else { tree= get_mm_parts(param, cond_func, field, Item_func::EQ_FUNC, func->arguments()[1], cmp_type); if (tree) { Item **arg, **end; for (arg= func->arguments()+2, end= arg+func->argument_count()-2; arg < end ; arg++) { tree= tree_or(param, tree, get_mm_parts(param, cond_func, field, Item_func::EQ_FUNC, *arg, cmp_type)); } } } mysql-5.7 extends this approach to handle multi-column case (by invoking get_mm_parts() for each compared value, then doing tree_and(), and then tree_or()). Should we just go ahead and backport it? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] problem with update and myisam
by Roberto Spadim 18 Jun '14

18 Jun '14

hi guys, could anyone help me with https://mariadb.atlassian.net/browse/MDEV-6355 thanks -- Roberto Spadim

2 5

Re: [Maria-developers] [Commits] 0eb94ca: MDEV-6137 better help for SET/ENUM sysvars
by Sergey Vojtovich 17 Jun '14

17 Jun '14

Hi Sergei, ok to push. A few minor questions inline. On Mon, Jun 16, 2014 at 06:51:53PM +0200, serg(a)mariadb.org wrote: > revision-id: 0eb94ca810500977eb0336ee30b19e9c0b769bca > parent(s): f61f36b386e8d0c6448297bb84c92e8d9b82be6a > committer: Sergei Golubchik > branch nick: maria > timestamp: 2014-06-16 18:43:26 +0200 > message: > > MDEV-6137 better help for SET/ENUM sysvars > > Auto-generate the allowed list of values for enum/set/flagset options > in --help output. But don't do that when the help text already has them. > > Also, remove lists of values from help strings of various options, where > they were simply listed without any additional information. > > --- > mysys/my_getopt.c | 81 +++++++++++++++++++++++++++------ > plugin/server_audit/server_audit.c | 2 +- > sql/log.cc | 3 +- > sql/mysqld.cc | 8 ++-- > sql/sql_plugin.cc | 2 +- > sql/sys_vars.cc | 90 ++++++++----------------------------- > storage/maria/ha_maria.cc | 14 ++---- > storage/myisam/ha_myisam.cc | 7 ++- > storage/xtradb/handler/ha_innodb.cc | 35 +++++++-------- > 9 files changed, 116 insertions(+), 126 deletions(-) Hmmm... no test case affected? > > diff --git a/mysys/my_getopt.c b/mysys/my_getopt.c > index 2e97417..3662757 100644 > --- a/mysys/my_getopt.c > +++ b/mysys/my_getopt.c > @@ -1300,6 +1300,48 @@ static uint print_name(const struct my_option *optp) > return s - optp->name; > } > > +/** prints option comment with indentation and wrapping. > + > + The comment column starts at startpos, and has width of width > + Current cursor position is curpos, returns new cursor position > + > +Note: can print one character beyond width! The above line, is it properly indented? > +*/ > +static uint print_comment(const char *comment, > + int curpos, int startpos, int width) Probably make width local to this function, it doesn't seem to be used by the caller anyway? > +{ > + const char *end= strend(comment); > + int endpos= startpos + width; > + > + for (; curpos < startpos; curpos++) > + putchar(' '); > + > + if (*comment == '.' || *comment == ',') > + { > + putchar(*comment); > + comment++; > + curpos++; > + } > + > + while (end - comment > endpos - curpos) > + { > + const char *line_end; > + for (line_end= comment + endpos - curpos; > + line_end > comment && *line_end != ' '; > + line_end--); > + for (; comment < line_end; comment++) > + putchar(*comment); I would probably replace it with fwrite(), but fine with this form too. > + while (*comment == ' ') > + comment++; /* skip the space, as a newline will take it's place now */ > + putchar('\n'); > + for (curpos= 0; curpos < startpos; curpos++) > + putchar(' '); > + } > + printf("%s", comment); > + return curpos + (end - comment); > +} > + > + > /* > function: my_print_options > > @@ -1309,12 +1351,12 @@ static uint print_name(const struct my_option *optp) > void my_print_help(const struct my_option *options) > { > uint col, name_space= 22, comment_space= 57; > - const char *line_end; > const struct my_option *optp; > DBUG_ENTER("my_print_help"); > > for (optp= options; optp->name; optp++) > { > + const char *typelib_help= 0; > if (!optp->comment) > continue; > if (optp->id && optp->id < 256) > @@ -1359,25 +1401,36 @@ void my_print_help(const struct my_option *options) > col= 0; > } > } > - for (; col < name_space; col++) > - putchar(' '); > if (optp->comment && *optp->comment) Cleanup suggestion: optp->comment can't be NULL here. > { > - const char *comment= optp->comment, *end= strend(comment); > + col= print_comment(optp->comment, col, name_space, comment_space); > > - while ((uint) (end - comment) > comment_space) > + switch (optp->var_type & GET_TYPE_MASK) { > + case GET_ENUM: > + typelib_help= ". One of: "; > + break; > + case GET_SET: > + typelib_help= ". Any combination of: "; > + break; > + case GET_FLAGSET: > + typelib_help= ". Takes a comma-separated list of option=value pairs, " > + "where value is on or off, and options are: "; > + break; > + } > + if (typelib_help && > + strstr(optp->comment, optp->typelib->type_names[0]) == NULL) Should we handle case when optp->typelib == NULL || optp->typelib->type_names[0] == NULL? Probably not. I believe this condition is fragile: comment string may mention type_names[0], but not list possible values. OTOH since this condition is for very limited data set it should cause problems only in rare cases. > { > - for (line_end= comment + comment_space; *line_end != ' '; line_end--); > - for (; comment != line_end; comment++) > - putchar(*comment); > - comment++; /* skip the space, as a newline will take it's place now */ > - putchar('\n'); > - for (col= 0; col < name_space; col++) > - putchar(' '); > + int i; > + col= print_comment(typelib_help, col, name_space, comment_space); > + col= print_comment(optp->typelib->type_names[0], col, name_space, comment_space); > + for (i= 1; i < optp->typelib->count; i++) > + { > + col= print_comment(", ", col, name_space, comment_space); > + col= print_comment(optp->typelib->type_names[i], col, name_space, comment_space); > + } > } > - printf("%s", comment); > + putchar('\n'); > } > - putchar('\n'); Does this mean if comment is empty there won't be a new line? > if ((optp->var_type & GET_TYPE_MASK) == GET_BOOL) > { > if (optp->def_value != 0) ...skip... > @@ -2785,7 +2745,7 @@ static Sys_var_mybool Sys_master_verify_checksum( > > /* These names must match RPL_SKIP_XXX #defines in slave.h. */ > static const char *replicate_events_marked_for_skip_names[]= { > - "replicate", "filter_on_slave", "filter_on_master", 0 > + "REPLICATE", "FILTER_ON_SLAVE", "FILTER_ON_MASTER", 0 > }; Does this mean there is a convention that values must be in upper case? ...skip... Regards, Sergey

2 1

[Maria-developers] WebScaleSQL: Actually fix all the array bounds warnings
by Sergey Vojtovich 16 Jun '14

16 Jun '14

Hi Sergei and Alexey, during our recent discussion re WebScaleSQL patches we agreed to review $subj: https://github.com/webscalesql/webscalesql-5.6/commit/094187c4a20e8d0c02f17… All these warnings come from strings/decimal.c. I was able to reproduce them using gcc-4.8.1, cmake -DBUILD_CONFIG=mysql_release (basically needs -Wall -O3). I simplified this function to the following: gcc -Wall -O3 -c dec.c -o dec.o static const int powers10[2 + 1]= { 1, 10, 100 }; int remove_leading_zeroes(int decimals, int var) { decimals%= 2; while (var < powers10[decimals--]) /* no-op */; return decimals; } The only allowed value for decimals is 1 and var must be > 0. Similar assertion is enforced by original function. So far I see nothing wrong with this code. I believe we should skip this patch, but still we should probably do something about these warning. What's your take? Regards, Sergey

2 2

[Maria-developers] MariaDB 10.0.12 release prep
by Daniel Bartholomew 14 Jun '14

14 Jun '14

I've begun prep work for the MariaDB 10.0.12 release. Draft Release Notes and Changelog are here: - https://mariadb.com/kb/en/mariadb-10012-release-notes/ - https://mariadb.com/kb/en/mariadb-10012-changelog/ Please add/modify/etc... as needed. Current plan is to announce the release on Mon, 16 Jun 2014. Thanks! -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

1 0

Re: [Maria-developers] Rev 3986: MDEV-5231: Per query variables from Percona Server
by Sergei Golubchik 14 Jun '14

14 Jun '14

Hi, Sanja! On May 22, sanja(a)askmonty.org wrote: > === added file 'mysql-test/suite/percona/percona_rpl_stm_per_query_variables_settings.test' > --- a/mysql-test/suite/percona/percona_rpl_stm_per_query_variables_settings.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/percona/percona_rpl_stm_per_query_variables_settings.test 2014-05-22 12:50:01 +0000 could you rename tests to have shorter names? rpl_set_statement looks good enough to me > @@ -0,0 +1,57 @@ > +--source include/master-slave.inc > +--source include/have_binlog_format_statement.inc > + > +--disable_warnings > +DROP TABLE IF EXISTS t1; > +DROP TABLE IF EXISTS t2; > +--enable_warnings > + > +call mtr.add_suppression("Unsafe statement written to the binary log*"); > +CREATE TABLE t1 (a bigint unsigned not null); > +CREATE TABLE t2 (a char(255) not null); okay, although I wouldn't bother creating *two* tables, I'd create one and hard-coded it into percona_rpl_set_statement_variable_test.inc (removing rpl_ssvt_table parameter) > + > +--echo > +--echo There are the following types of variables: > +--echo 1) variables that are NOT replicated correctly when using STATEMENT mode; > +--echo > + > +--let $rpl_ssvt_var_name=max_join_size > +--let $rpl_ssvt_var_value=2 > +--let $rpl_ssvt_table=t1 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > + > +--echo > +--echo 2) variables thar ARE replicated correctly > +--echo They must be replicated correctly with "SET STATEMENT" too. > +--echo > +--let $rpl_ssvt_var_name=auto_increment_increment > +--let $rpl_ssvt_var_value=10 > +--let $rpl_ssvt_table=t1 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > + > +--echo > +--echo 3) sql_mode which is replicated correctly exept NO_DIR_IN_CREATE value; > +--echo > +--let $rpl_ssvt_var_name=sql_mode > +--let $rpl_ssvt_var_value='ERROR_FOR_DIVISION_BY_ZERO' > +--let $rpl_ssvt_table=t2 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > +--let $rpl_ssvt_var_name=sql_mode > +--let $rpl_ssvt_var_value='NO_DIR_IN_CREATE' > +--let $rpl_ssvt_table=t2 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > + > +--echo > +--echo 4) variables that are not replicated at all: > +--echo default_storage_engine, storage_engine, max_heap_table_size > +--echo > +--let $rpl_ssvt_var_name=max_heap_table_size > +--let $rpl_ssvt_var_value=16384 > +--let $rpl_ssvt_table=t1 > +--source suite/percona/percona_rpl_set_statement_variable_test.inc > + > +connection master; > +DROP TABLE t1; > +DROP TABLE t2; > +sync_slave_with_master; > +source include/stop_slave.inc; > > === added file 'mysql-test/suite/percona/percona_statement_set.result' > --- a/mysql-test/suite/percona/percona_statement_set.result 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/percona/percona_statement_set.result 2014-05-22 12:50:01 +0000 reverse polish notation in percona someone loves. starwars too much seen he apparently. Please, rename to "set_statement.result" > @@ -0,0 +1,875 @@ > +'# SET STATEMENT ..... FOR .... TEST' > +DROP TABLE IF EXISTS t1; > +DROP FUNCTION IF EXISTS myProc; > +DROP PROCEDURE IF EXISTS p1; > +DROP PROCEDURE IF EXISTS p2; > +DROP PROCEDURE IF EXISTS p3; > +DROP PROCEDURE IF EXISTS p4; > +DROP PROCEDURE IF EXISTS p5; > +DROP TABLE IF EXISTS STATEMENT; > +'# Setup database' > +CREATE TABLE t1 (v1 INT, v2 INT); > +INSERT INTO t1 VALUES (1,2); > +INSERT INTO t1 VALUES (3,4); > +'' > +'#------------------ STATEMENT Test 1 -----------------------#' > +'# Initialize variables to known setting' > +SET SESSION sort_buffer_size=100000; > +'' > +'# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +Variable_name Value > +sort_buffer_size 100000 > +SET STATEMENT sort_buffer_size=150000 FOR SELECT * FROM t1; > +v1 v2 > +1 2 > +3 4 > +'' > +'# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +Variable_name Value > +sort_buffer_size 100000 > +'' > +'#------------------ STATEMENT Test 2 -----------------------#' > +'# Initialize variables to known setting' > +SET SESSION binlog_format=mixed; > +SET SESSION sort_buffer_size=100000; > +'# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +Variable_name Value > +sort_buffer_size 100000 > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +Variable_name Value > +binlog_format MIXED > +SET STATEMENT sort_buffer_size=150000, binlog_format=row > +FOR SELECT * FROM t1; > +v1 v2 > +1 2 > +3 4 > +'# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +Variable_name Value > +sort_buffer_size 100000 > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +Variable_name Value > +binlog_format MIXED > +'' > +'#------------------ STATEMENT Test 3 -----------------------#' > +'# set initial variable value, make prepared statement > +SET SESSION binlog_format=row; > +PREPARE stmt1 FROM 'SET STATEMENT binlog_format=row FOR SELECT * FROM t1'; > +'' > +'# Change variable setting' > +SET SESSION binlog_format=mixed; > +'' > +'# Pre-STATEMENT variable value' > +'' > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +Variable_name Value > +binlog_format MIXED > +'' > +EXECUTE stmt1; > +v1 v2 > +1 2 > +3 4 > +'' > +'# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +Variable_name Value > +binlog_format MIXED > +'' > +DEALLOCATE PREPARE stmt1; > +'#------------------ STATEMENT Test 4 -----------------------#' > +'# set initial variable value, make prepared statement > +SET SESSION myisam_sort_buffer_size=500000, myisam_repair_threads=1; > +'' > +'# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'myisam_sort_buffer_size'; > +Variable_name Value > +myisam_sort_buffer_size 500000 > +SHOW SESSION VARIABLES LIKE 'myisam_repair_threads'; > +Variable_name Value > +myisam_repair_threads 1 > +'' > +SET STATEMENT myisam_sort_buffer_size=800000, > +myisam_repair_threads=2 FOR OPTIMIZE TABLE t1; how comes that is "make prepared statement" (see comment above) ? > +Table Op Msg_type Msg_text > +test.t1 optimize status OK > +'' > +'# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'myisam_sort_buffer_size'; > +Variable_name Value > +myisam_sort_buffer_size 500000 > +SHOW SESSION VARIABLES LIKE 'myisam_repair_threads'; > +Variable_name Value > +myisam_repair_threads 1 > +'' ... > === added file 'mysql-test/suite/percona/percona_statement_set.test' > --- a/mysql-test/suite/percona/percona_statement_set.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/percona/percona_statement_set.test 2014-05-22 12:50:01 +0000 > @@ -0,0 +1,847 @@ > +--echo '# SET STATEMENT ..... FOR .... TEST' many of the tests below are pretty useless. they don't test that the value is actually set. See below: > +############################ STATEMENT_SET ############################# > +# # > +# Testing working functionality of SET STATEMENT # > +# # > +# # > +# There is important documentation within # > +# # > +# # > +# Author: Joe Lukas # > +# Creation: # > +# 2009-08-02 Implement this test as part of # > +# WL#681 Per query variable settings # > +# # > +######################################################################## > + > +--disable_warnings > +DROP TABLE IF EXISTS t1; > +DROP FUNCTION IF EXISTS myProc; > +DROP PROCEDURE IF EXISTS p1; > +DROP PROCEDURE IF EXISTS p2; > +DROP PROCEDURE IF EXISTS p3; > +DROP PROCEDURE IF EXISTS p4; > +DROP PROCEDURE IF EXISTS p5; > +DROP TABLE IF EXISTS STATEMENT; > +--enable_warnings > +#################################################################### > +#Set up current database > +#################################################################### > +--echo '# Setup database' > +CREATE TABLE t1 (v1 INT, v2 INT); > +INSERT INTO t1 VALUES (1,2); > +INSERT INTO t1 VALUES (3,4); > +--echo '' > +--echo '#------------------ STATEMENT Test 1 -----------------------#' > +#################################################################### > +# Checks with variable value type ulong # > +#################################################################### > +--echo '# Initialize variables to known setting' > +SET SESSION sort_buffer_size=100000; > +--echo '' > +--echo '# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +SET STATEMENT sort_buffer_size=150000 FOR SELECT * FROM t1; How do you know that SET STATEMENT did anything at all? > +--echo '' > +--echo '# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +--echo '' > +--echo '#------------------ STATEMENT Test 2 -----------------------#' > +#################################################################### > +# Checks for multiple set values inside STATEMENT ... FOR # > +#################################################################### > +--echo '# Initialize variables to known setting' > +SET SESSION binlog_format=mixed; > +SET SESSION sort_buffer_size=100000; > +--echo '# Pre-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +SET STATEMENT sort_buffer_size=150000, binlog_format=row > + FOR SELECT * FROM t1; > +--echo '# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'sort_buffer_size'; > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > + > +--echo '' > +--echo '#------------------ STATEMENT Test 3 -----------------------#' > +#################################################################### > +# Check current variable value is stored in using stored # > +# statements. # > +#################################################################### > +--echo '# set initial variable value, make prepared statement > +SET SESSION binlog_format=row; > +PREPARE stmt1 FROM 'SET STATEMENT binlog_format=row FOR SELECT * FROM t1'; How will you know that SET STATEMENT did anything at all? (the same applies to most tests below, I'll stop repeating it) > +--echo '' > +--echo '# Change variable setting' > +SET SESSION binlog_format=mixed; > +--echo '' > +--echo '# Pre-STATEMENT variable value' > +--echo '' > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > +--echo '' > +EXECUTE stmt1; > +--echo '' > +--echo '# Post-STATEMENT variable value' > +SHOW SESSION VARIABLES LIKE 'binlog_format'; > + > +--echo '' > +DEALLOCATE PREPARE stmt1; ... > === modified file 'sql/share/errmsg-utf8.txt' > --- a/sql/share/errmsg-utf8.txt 2014-02-11 13:21:48 +0000 > +++ b/sql/share/errmsg-utf8.txt 2014-05-22 12:50:01 +0000 > @@ -7067,3 +7067,5 @@ ER_IT_IS_A_VIEW 42S02 > eng "'%-.192s' is a view" > ER_SLAVE_SKIP_NOT_IN_GTID > eng "When using GTID, @@sql_slave_skip_counter can not be used. Instead, setting @@gtid_slave_pos explicitly can be used to skip to after a given GTID position." > +ER_SET_STATEMENT_TABLES_IS_NOT_SUPPORTED > + eng "SET STATEMENT does not support using tables in the assignment variables expressions" why not? > > === modified file 'sql/sql_parse.cc' > --- a/sql/sql_parse.cc 2014-03-29 10:33:25 +0000 > +++ b/sql/sql_parse.cc 2014-05-22 12:50:01 +0000 > @@ -2430,6 +2432,33 @@ mysql_execute_command(THD *thd) > thd->get_binlog_format(&orig_binlog_format, > &orig_current_stmt_binlog_format); > > + if (lex->set_statement && !lex->var_list.is_empty()) Hmm? No separate list of variables, but a common lex->var_list? Then the following won't work: SET STATEMENT A=1 SET B=2 I remember in the WL and in GSoC we agreed that for SET STATEMENT A=2 SET A=2 (same variable in SET STATEMENT and SET), it's acceptable to restore the original value of A, basically ignoring SET. But I doubt that the same logic applies when SET STATEMENT and SET modify *different* variables. > + { > + MEM_ROOT *save_mem_root; > + /* > + We should be sure that variables backup and assignment made on > + runtime memory > + */ > + save_mem_root= thd->mem_root; > + thd->mem_root= thd->get_execution_mem_root(); Why? > + per_query_variables_backup= copy_system_variables(&thd->variables); > + if (!per_query_variables_backup || > + (res= sql_set_variables(thd, &lex->var_list))) > + { > + thd->mem_root= save_mem_root; > + /* > + We encountered some sort of error, but no message was sent. > + Send something semi-generic here since we don't know which > + assignment in the list caused the error. > + */ > + if (!thd->is_error()) > + my_error(ER_WRONG_ARGUMENTS, MYF(0), "SET"); > + goto error; > + } > + thd->mem_root= save_mem_root; > + } > + > /* > Force statement logging for DDL commands to allow us to update > privilege, system or statistic tables directly without the updates > @@ -5144,6 +5179,30 @@ finish: > > lex->unit.cleanup(); > > + if (lex->set_statement && !lex->var_list.is_empty()) { > + List_iterator_fast<set_var_base> it(thd->lex->var_list); > + set_var *var; > + > + free_system_variables(&thd->variables); > + thd->variables= *per_query_variables_backup; > + /* per_query_variables_backup was in a mem_root */ > + per_query_variables_backup= NULL; > + /* > + When variables are restored after "SET STATEMENT ... FOR ..." statement > + execution an update callback must be invoked for the system variables > + to save special logic if it is. set_var_base class does not contain > + refference to variable as it is just an interface class. But only > + system variables are allowed to be used in "SET STATEMENT ... FOR ..." > + statement, so cast from set_var_base* to set_var* can be used here. > + */ > + while ((var=(set_var *)it++)) > + { > + var->var->stmt_update(thd); > + } I don't think it'll work. Compare set_var::update() and set_var::stmt_update(). The former calls ::session_update(), while the latter does not. Normally, session_update() simply updates the value, performs the assignment. And it is, indeed, not needed here. but in some custom classes session_update() is more complex, and it's not safe to omit it. Please 1) see all non-trivial implementations of sys_var::session_update() 2) write SET SESSION tests for these variables 3) if they'll all work - please explain why 4) if they won't - I'm afraid, this whole approach needs to be redone, then > + > + thd->lex->set_statement= false; > + } > + > if (! thd->in_sub_stmt) > { > if (thd->killed != NOT_KILLED) > > === modified file 'sql/sql_plugin.cc' > --- a/sql/sql_plugin.cc 2014-03-26 21:25:38 +0000 > +++ b/sql/sql_plugin.cc 2014-05-22 12:50:01 +0000 > @@ -4088,3 +4088,48 @@ static void restore_ptr_backup(uint n, s > (backup++)->restore(); > } > > + > +/** > + Create deep copy of system_variables instance. > +*/ > +struct system_variables * > +copy_system_variables(const struct system_variables *src) What is it doing in sql_plugin.cc? > +{ > + struct system_variables *dst; > + > + DBUG_ASSERT(src); > + > + dst= (struct system_variables *) > + sql_alloc(sizeof(struct system_variables)); > + if (!dst) > + return NULL; > + *dst= *src; > + > + if (dst->dynamic_variables_ptr) > + { > + if (!(dst->dynamic_variables_ptr= > + (char *)my_malloc(dst->dynamic_variables_size, MYF(MY_WME | MY_FAE)))) > + return NULL; > + memcpy(dst->dynamic_variables_ptr, > + src->dynamic_variables_ptr, > + src->dynamic_variables_size); > + } > + > + mysql_mutex_lock(&LOCK_plugin); > + dst->table_plugin= > + intern_plugin_lock(NULL, src->table_plugin); I'm surprised that plugin locking actually works in here, (I believe it does, otherwise you'd noticed it). What about SET STATEMENT default_engine=MEMORY ? What about SET STATEMENT default_engine=MyISAM (where MyISAM is already default)? please, try both in debug build. > + mysql_mutex_unlock(&LOCK_plugin); > + > + return dst; > +} > + > +void free_system_variables(struct system_variables *v) > +{ > + DBUG_ASSERT(v); > + > + mysql_mutex_lock(&LOCK_plugin); > + intern_plugin_unlock(NULL, v->table_plugin); > + mysql_mutex_unlock(&LOCK_plugin); > + > + my_free(v->dynamic_variables_ptr); > +} > > === modified file 'sql/sql_yacc.yy' > --- a/sql/sql_yacc.yy 2014-03-26 21:25:38 +0000 > +++ b/sql/sql_yacc.yy 2014-05-22 12:50:01 +0000 > @@ -813,8 +813,10 @@ static void sp_create_assignment_lex(THD > lex->one_shot_set= 0; > lex->autocommit= 0; > /* get_ptr() is only correct with no lookahead. */ > - DBUG_ASSERT(no_lookahead); > + if (no_lookahead) > lex->sphead->m_tmp_query= lip->get_ptr(); > + else > + lex->sphead->m_tmp_query= lip->get_tok_end(); Why? > /* Inherit from outer lex. */ > lex->option_type= old_lex->option_type; > } > @@ -955,10 +957,10 @@ bool my_yyoverflow(short **a, YYSTYPE ** > %parse-param { THD *thd } > %lex-param { THD *thd } > /* > - Currently there are 163 shift/reduce conflicts. > + Currently there are 164 shift/reduce conflicts. > We should not introduce new conflicts any more. > */ > -%expect 163 > +%expect 164 Why? > > /* > Comments for TOKENS. > @@ -14427,8 +14431,44 @@ set: > } > start_option_value_list > {} > + | SET STATEMENT_SYM > + { > + LEX *lex= Lex; > + mysql_init_select(lex); > + lex->sql_command= SQLCOM_SET_OPTION; > + /* Don't clear var_list in the case of recursive statement */ > + if (!lex->set_statement) > + lex->var_list.empty(); When var_list is not empty here (and set_statement not set)? In other words, when this statement will actually have any effect? > + lex->one_shot_set= 0; > + lex->autocommit= 0; > + lex->set_statement= true; > + sp_head *sp= lex->sphead; > + if (sp && !sp->is_invoked()) > + { > + sp->m_param_begin= ((yychar == YYEMPTY) ? YYLIP->get_ptr() : YYLIP->get_tok_start()); > + sp->m_param_end= ((yychar == YYEMPTY) ? YYLIP->get_ptr() : YYLIP->get_tok_end()); Why? > + } > + } > + set_stmt_option_value_following_option_type_list Eh? What does "set stmt option value following option type list" mean? Please, rename to set_stmt_option_value_list or even set_stmt_list > + { > + if (Lex->query_tables) > + { > + my_error(ER_SET_STATEMENT_TABLES_IS_NOT_SUPPORTED, MYF(0)); > + MYSQL_YYABORT; > + } > + } > + FOR_SYM statement > + {} > ; > > +set_stmt_option_value_following_option_type_list: > + /* > + Only system variables can be used here. If this condition is changed > + please check careful code under lex->option_type == OPT_STATEMENT > + condition on wrong type casts. > + */ > + option_value_following_option_type > + | set_stmt_option_value_following_option_type_list ',' option_value_following_option_type > > // Start of option value list > start_option_value_list: Regards, Sergei

1 0

Re: [Maria-developers] [GSoC] self-tuning optimizer
by Sergei Golubchik 13 Jun '14

13 Jun '14

Hi, Anshu! On Jun 13, Anshu Avinash wrote: > > So I have refactored Cost_factors::init() into Cost_factors::init() and > Cost_factors::re_init(). The diff is at: > https://github.com/igniting/server/commit/41e360309c067d072d788495e4e948054… Looks great. I've added one comment there. > I had started writing a test for checking whether if loading a new > storage engine works or not. I loaded 'Example' storage engine, but > was unable to test the current t1 table on it, as t1 contains a > primary key. Also I need to write test cases for 'time_for_compare'. > Any hints on that? Hm. Yes. Example engine, is pretty useless for you, as you cannot create tables in it. You can use InnoDB plugin. But create a separate test file for that. For example (I'll use the short name below, don't commit it): a.test: ============================================== if (!$HA_INNODB_SO) { skip Need InnoDB plugin; } install plugin innodb soname 'ha_innodb'; ... do your tests here uninstall plugin innodb; ============================================== you will also need a.opt file: ============================================== --ignore-builtin-innodb --loose-innodb ============================================== As for time_for_compare... Here's a universal approach, you can do it yourself. This time_for_compare is 5 by default, right? Change it to be, say, 500, and run existing tests (start from the main suite, and there from optimizer tests, e.g. from select.test). If some test will fail - here's your query, extract it, simplify, perhaps, and put in your test file. That's exactly what I just did :) And here's a simple test: create table t1 (a blob, index(a(20))); create table t2 (a blob, index(a(20))); insert into t1 values ('one'),('two'),('three'),('four'),('five'); insert into t2 values ('one'),('two'),('three'),('four'),('five'); insert into t2 values ('one'),('two'),('three'),('four'),('five'); drop table t1, t2; Perhaps you can get it down to one table - I didn't try. > Also I think I can start with the next part, i.e. collect statistics. We > can have a structure like: > > struct Cost_factor_stats > { > char *cost_factor_name; > double cost_factor_value; > double total_queries; total_queries should be ulonglong, not double. > double total_query_time; > double total_query_time_squared; > }; > > We will be updating these structures after every query per THD and write > the data into a persistent table on disconnect. Is this workflow correct? Almost. You need a Cost_factor_stats structure per cost factor (and per engine where appropriate). You don't need the name there. You have these structures per THD and globally. Basically, instead of your current double time_for_compare_val; you will have Cost_factor_stats time_for_compare_val; hmm, perhaps better to name the structure simply "Cost_factor": Cost_factor time_for_compare; And you put them all in one structure: struct Engine_cost_factors { Cost_factor read_time; Cost_factor scan_time; }; struct Global_cost_factors { Cost_factor time_for_compare; Cost_factor time_for_compare_rowid; }; class Cost_factors { Global_cost_factors global; Engine_cost_factors engine[MAX_HA]; double time_for_compare() const { return global.read_time.value; } double time_for_compare_rowid() const; double read_time(const handler*) const; double scan_time(const handler*) const; }; Now, you stick this structure in the THD, and also declare one instance globally. On connect you initialize THD::Cost_factor values from the global Cost_factor object. On disconnect you add THD::Cost_factor values back to the global Cost_factor object. There's no writing to disk so far, we'll think about it later (you don't want to write to disk on *every* disconnect). > Few more questions: > 1.What will we do if the query had used more than one cost factor? > 2.How would joins be handled? Yes, a query will certainly use more than one cost factor. In fact, this is something we might want to experiment with. I see two approaches - exact and approximate. Approximate one is described in MDEV. Say, you've counted 500 index reads, 20,000 sequential row reads (table scan), some 10,000,000 comparisons, etc. And you know how long the query took. For another query you've counted different numbers, and so on. Basically, you'll have a system of linear equations that one can solve to know how the time was split. The problem with this approach - the server is doing much more than purely index reads, table scans, and comparisons. So this unaccounted time will be split between cost factors. I don't know if it's a problem. The exact approach - we only measure times spent on the actual operation. Basically, you measure how long *each row read* took, how long *each index read* took, etc. This will show no unaccounted time. But it's more expensive, it might be too expensive to be practically useful - I don't know that. These two approaches need different data structures, the first one needs more memory, because you cannot start aggregating before you solve the system of equations. Basically, if you have four constant (time_for_compare, time_for_compare_rowid, read_time, scan_time, only one engine), you'll need four equations, that is, per THD you need to store four measurements: struct measurement { ulong time_for_compare; ulong time_for_compare_rowid; struct per_engine { ulong scan_time; ulong read_time; } per_engine[MAX_HA]; double total_time; }; and in THD you need struct measurement data[4]; Also, note that I'm writing per_engine[MAX_HA], but it's a simplification, good enough for a prototype. But in the final code we cannot allocate that much memory per THD, as most of the time we'll have only few engines, not MAX_HA. Regards, Sergei

1 0

[Maria-developers] WebScaleSQL: Adjust linking flags to not embed some libs
by Sergey Vojtovich 13 Jun '14

13 Jun '14

Hi Sergei, during our recent discussion re WebScaleSQL patches we agreed to accept $subj: https://github.com/webscalesql/webscalesql-5.6/commit/361e3b83145402945b737… I studied our binaries and it looks like we embed zlib and ssl libraries only if WITH_ZLIB=bundled/WITH_SSL=bundled. E.g. DEB/RPM packages use system libs and do not embed zlib/ssl, whereas tarball uses bundled zlib/yassl and embeds them. Not embedding yassl into the client library will make the latter almost useless. Same probably applies to zlib even though it is normally offered by OS-es. Said the above I believe our build scripts behave properly and we should skip this patch. What's your take? Regards, Sergey

2 1

[Maria-developers] Java Connector Coding till 12th June
by Puneet Dewan 13 Jun '14

13 Jun '14

Hi Everyone, This week, I was able to send the com_stmt_execute command from the *Driver * in the way it is accepted by the MariaDB server So, Now I am able to insert a row in the database using com_stmt_execute command. But from the console when I try to see the values I see *NULL* I used *Wireshark* to capture the packets,but I can see the value(Varchar /Int/..etc) is passed in the packet which wireshark captures. I am investigating the code but could not find anything till now. In Wireshark it shows* [Expert Info (warn/undecoded) :FIXME: Excute Dissector Incomplete]* Now I think I am not sending the values in a correct format.Will look and try to resolve it. Regards Puneet. On Thu, Jun 5, 2014 at 10:42 PM, Puneet Dewan <puneetd30(a)gmail.com> wrote: > Hi Everyone, > > The work done till 5th June: > 1.I started studying the connector immediately after my semester exams got > over(5th May). > 2. Understood the relationship between different classes. > 3.Read the MySql documentation for text protocol and map the commands from > documentation onto code.This thing gave a clear idea about how things are > moving from client to server. > 4.Setup the code on Ubuntu using eclipse and tested against database. > Setup the Team branch in launchpad. > 5.Debug the test classes to see how different classes are interlinked and > understood that MySQLProtocol.java is the heart of our JDBC driver. > 6.Then I started reading Client Server Protocol documentation i.e > ServerSide prepared statement and try to see how and where I have to code > it. > 7.Then I saw that there is a class MySqlServerSidePreparedStatement.java, > Initially I coded the prepareStatement() ,close(),and execute() in that > class seperately. > But after (Code review) discussion with Massimo, we donot need to use that > class. > Instead use MySqlPreparedStatement.java to do the same work.So made > changes accordingly. > 8.I had pushed some of the code which I have written to launchpad and > discussed with Massimo last week. > 9.Last week I was struck with Wireshark on windows and ubuntu both. > I was unable to see the packets going to database and seeing the response. > 10.On Monday the problem was resolved as I was using the wrong interface.I > should have used loopback.io(127.0.0.1) instead on ubuntu(thanks to > Georg). > Unfortunately the thing does not work on windows.Tried a lot many times. > 11.Now I am working on the execute method of client server protocol as I > am getting exception while using that method.Wireshark is playing a great > role in how the packets are flowing. > > From now I will do weekly blogging and keep everyone updated. > > Thanks & Regards > Puneet. >

1 0

[Maria-developers] Allow to turn off max_connect_errors
by Honza Horak 13 Jun '14

13 Jun '14

Hi guys, there are apparently some tools out there [1], that check if the server is up quite often, while not closing the connection properly. It eventually ends in 'many connection errors', because max_connect_errors is always limited now. I understand that this way of checking may be wrong, but there may be scenarios where we do not want to check for `max_connect_errors` at all. So, would it be acceptable for mariadb to change behaviour of max_connect_errors option, so that it accepts also 0 as a possible value, which would mean 'do not check connect errors at all'? I'm bringing the idea here first, but will submit a report and possibly patch if it does not seem to be undesired behaviour. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1104957 TIA and regards, Honza

4 4

Re: [Maria-developers] [Commits] Rev 4018: MDEV-4784 - merge test cases from 5.6 in lp:maria/10.0
by Sergei Golubchik 12 Jun '14

12 Jun '14

Hi, Sergey! > === added file 'mysql-test/include/have_daemon_example_plugin.inc' > --- a/mysql-test/include/have_daemon_example_plugin.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/have_daemon_example_plugin.inc 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,16 @@ > +disable_query_log; > +# > +# Check if server has support for loading plugins > +# > +if (`SELECT @@have_dynamic_loading != 'YES'`) { > + --skip daemon example plugin requires dynamic loading > +} > + > +# > +# Check if the variable DAEMONEXAMPLE is set > +# > +if (!$LIBDAEMON_EXAMPLE_SO) { > + --skip daemon_example plugin requires the environment variable \$DAEMONEXAMPLE to be set (normally done by mtr) Eh? It was in your first patch. I thought it's an additional patch, but it looks like it includes the first one? Okay, I will copy all my comments from the first patch here to have a complete review in one email. So, here I said ************************ it's always done by mtr, a better error message should be "No libdaemon_example.so was found" or simply "Need daemon_example plugin" and, btw, the first if() is redundant, if no dynamic loading was enabled no dynamic plugins will be built whatsoever. > +} > + > +enable_query_log; > > === added file 'mysql-test/include/ib_logfile_size_check.inc' > --- a/mysql-test/include/ib_logfile_size_check.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/ib_logfile_size_check.inc 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,14 @@ I'm not a great fan of obscure include files that are only used once in one single test. Perhaps it's better to inline it? > +perl; > + my $dir = $ENV{'MYSQLD_DATADIR'}; > + my $size; > + opendir(DIR, $dir) or die $!; > + while (my $file = readdir(DIR)) > + { > + > + next unless ($file =~ m/\ib_logfile.$/); > + $size = -s "$dir/$file"; > + print "The size of the ib_logfile(0/1): $size \n"; > + } > + close(DIR); > + exit(0) > +EOF > > === added file 'mysql-test/include/vardir_size_check.inc' > --- a/mysql-test/include/vardir_size_check.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/vardir_size_check.inc 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,26 @@ Same here. Although it's worse than with the previous file :) ib_logfile_size_check.inc is only used in innodb_log_file_size_functionality.test. This file - too. But in 5.6 this one is also used in query_cache_type_functionality, table_open_cache_functionality, and table_definition_cache_functionality, sort_buffer_size_functionality, query_cache_size_functionality. I suppose somebody just copied innodb_log_file_size_functionality over and edited it. I don't see why would anyone really want to check the vardir size in all these other tests. So, yes, better to inline this file too, to avoid silly mistakes like that. > +--echo Check the size of the vardir > +--echo The output size is in unit blocks > +perl; > +#!/usr/bin/perl > +use warnings; > +use strict; > +use File::Find; > +my $dir = $ENV{'MYSQLTEST_VARDIR'}; > +my $size = 0; > +find( sub { $size += -f $_ ? -s _ : 0 }, $dir ); > + > +if ( $size < 1717987000 ) > + { print "TRUE", "\n";} > +else > + { print "FALSE $size", "\n";} > + > + > +## Expected size for the VAR dir being changed for the PB2 runs > + > +##if ( $size =~ /^496[\d]{5}$/ ) > +## { > +## { print "TRUE", "\n";} > +## } > +##else > +## { print "FALSE $size", "\n";} > +EOF > > === modified file 'mysql-test/mysql-test-run.pl' > --- a/mysql-test/mysql-test-run.pl 2014-02-26 14:28:07 +0000 > +++ b/mysql-test/mysql-test-run.pl 2014-04-14 09:20:24 +0000 > @@ -5350,6 +5350,14 @@ sub mysqld_arguments ($$$) { > mtr_add_arg($args, "--gdb"); > } > > + # Enable the debug sync facility, set default wait timeout. > + # Facility stays disabled if timeout value is zero. > + mtr_add_arg($args, "--loose-debug-sync-timeout=%s", > + $opt_debug_sync_timeout) unless $opt_user_args; Why was it moved? > + > + # Options specified in .opt files should be added last so they can > + # override defaults above. > + > my $found_skip_core= 0; > foreach my $arg ( @$extra_opts ) > { > === added file 'mysql-test/suite/archive/archive_no_symlink-master.opt' > --- a/mysql-test/suite/archive/archive_no_symlink-master.opt 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/archive/archive_no_symlink-master.opt 2014-04-14 09:20:24 +0000 > @@ -0,0 +1 @@ > +--skip-symbolic-links just fyi: in MariaDB the file name can simply be archive_no_symlink.opt -master and -slave are only needed for replication tests to have different options for the master and the slave. > > === added file 'mysql-test/t/bug12969156.test' > --- a/mysql-test/t/bug12969156.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/bug12969156.test 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,18 @@ > +--source include/have_daemon_example_plugin.inc > +--source include/not_embedded.inc > +# TODO: the windows emulation of pthreads doesn't behave predictably > +--source include/not_windows.inc > + > +--echo # > +--echo # Bug #12969156 : SEGMENTATION FAULT ON UNINSTALLING > +--echo # DAEMON_EXAMPLE PLUGIN > +--echo # > + > +let $counter= 0; > +while ($counter < 10) > +{ > +--replace_result $LIBDAEMON_EXAMPLE_SO DAEMONEXAMPLE > + eval INSTALL PLUGIN daemon_example SONAME '$LIBDAEMON_EXAMPLE_SO'; in our own tests (not copied from 5.6) better use install soname 'libdaemon_example'; it's shorter, doesn't need --eval and doesn't need --replace_result > + UNINSTALL PLUGIN daemon_example; > + inc $counter; > +} > > === added file 'mysql-test/t/innodb_log_file_size_functionality.test' > --- a/mysql-test/t/innodb_log_file_size_functionality.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/innodb_log_file_size_functionality.test 2014-04-14 09:20:24 +0000 > @@ -0,0 +1,124 @@ This should be renamed to sys_vars.innodb_log_file_size_func test > +############################################################################### > +# # > +# Variable Name: innodb_log_file_size # > +# Scope: Global # > +# Access Type: Static # > +# Data Type: numeric # > +# # > +# # > +# Creation Date: 2012-08-20 # > +# Author : Tanjot Singh Uppal # > +# # > +# # > +# Description:Test Cases of Static System Variable innodb_log_file_size # > +# that checks the behavior of this variable in the following ways # > +# * Value Check # > +# * Scope Check # > +# * Functionality Check # > +# * Accessability Check # > +# # > +# This test does not perform the crash recovery on this variable # > +# For crash recovery test on default change please run the ibtest # > +############################################################################### Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 4018: MDEV-4784 - merge test cases from 5.6 in lp:maria/10.0
by Sergei Golubchik 12 Jun '14

12 Jun '14

Hi, Sergey! > === added file 'mysql-test/include/have_daemon_example_plugin.inc' > --- a/mysql-test/include/have_daemon_example_plugin.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/have_daemon_example_plugin.inc 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,16 @@ > +disable_query_log; > +# > +# Check if server has support for loading plugins > +# > +if (`SELECT @@have_dynamic_loading != 'YES'`) { > + --skip daemon example plugin requires dynamic loading > +} > + > +# > +# Check if the variable DAEMONEXAMPLE is set > +# > +if (!$LIBDAEMON_EXAMPLE_SO) { > + --skip daemon_example plugin requires the environment variable \$DAEMONEXAMPLE to be set (normally done by mtr) it's always done by mtr (not just "normally"). So a better error message should be "No libdaemon_example.so was found" or simply "Need daemon_example plugin" and, btw, the first if() is redundant, if no dynamic loading was enabled no dynamic plugins will be built whatsoever. > +} > + > +enable_query_log; > > === added file 'mysql-test/include/ib_logfile_size_check.inc' > --- a/mysql-test/include/ib_logfile_size_check.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/ib_logfile_size_check.inc 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,14 @@ I'm not a great fan of obscure include files that are only used once in one single test. Perhaps it's better to inline it? > +perl; > + my $dir = $ENV{'MYSQLD_DATADIR'}; > + my $size; > + opendir(DIR, $dir) or die $!; > + while (my $file = readdir(DIR)) > + { > + > + next unless ($file =~ m/\ib_logfile.$/); > + $size = -s "$dir/$file"; > + print "The size of the ib_logfile(0/1): $size \n"; > + } > + close(DIR); > + exit(0) > +EOF > > === added file 'mysql-test/include/vardir_size_check.inc' > --- a/mysql-test/include/vardir_size_check.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/vardir_size_check.inc 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,26 @@ Same here. Although it's worse than with the previous file :) ib_logfile_size_check.inc is only used in innodb_log_file_size_functionality.test. This file - too. But in 5.6 this one is also used in query_cache_type_functionality, table_open_cache_functionality, and table_definition_cache_functionality, sort_buffer_size_functionality, query_cache_size_functionality. I suppose somebody just copied innodb_log_file_size_functionality over and edited it without much thinking. I don't see why would anyone really want to check the vardir size in all these other tests. So, yes, better to inline this file too, to avoid silly mistakes like that. > +--echo Check the size of the vardir > +--echo The output size is in unit blocks > +perl; > +#!/usr/bin/perl > +use warnings; > +use strict; > +use File::Find; > +my $dir = $ENV{'MYSQLTEST_VARDIR'}; > +my $size = 0; > +find( sub { $size += -f $_ ? -s _ : 0 }, $dir ); > + > +if ( $size < 1717987000 ) > + { print "TRUE", "\n";} > +else > + { print "FALSE $size", "\n";} > + > + > +## Expected size for the VAR dir being changed for the PB2 runs > + > +##if ( $size =~ /^496[\d]{5}$/ ) > +## { > +## { print "TRUE", "\n";} > +## } > +##else > +## { print "FALSE $size", "\n";} > +EOF > > === added file 'mysql-test/suite/archive/archive_no_symlink-master.opt' > --- a/mysql-test/suite/archive/archive_no_symlink-master.opt 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/archive/archive_no_symlink-master.opt 2014-03-11 09:28:16 +0000 > @@ -0,0 +1 @@ just fyi: in MariaDB the file name can simply be archive_no_symlink.opt -master and -slave are only needed for replication tests to have different options for the master and the slave. > +--skip-symbolic-links > > === added file 'mysql-test/t/bug12969156.test' > --- a/mysql-test/t/bug12969156.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/bug12969156.test 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,18 @@ > +--source include/have_daemon_example_plugin.inc > +--source include/not_embedded.inc > +# TODO: the windows emulation of pthreads doesn't behave predictably > +--source include/not_windows.inc > + > +--echo # > +--echo # Bug #12969156 : SEGMENTATION FAULT ON UNINSTALLING > +--echo # DAEMON_EXAMPLE PLUGIN > +--echo # > + > +let $counter= 0; > +while ($counter < 10) > +{ > +--replace_result $LIBDAEMON_EXAMPLE_SO DAEMONEXAMPLE > + eval INSTALL PLUGIN daemon_example SONAME '$LIBDAEMON_EXAMPLE_SO'; in our own tests (not copied from 5.6) better use install soname 'libdaemon_example'; it's shorter, doesn't need --eval and doesn't need --replace_result > + UNINSTALL PLUGIN daemon_example; > + inc $counter; > +} > > === added file 'mysql-test/t/innodb_log_file_size_functionality.test' > --- a/mysql-test/t/innodb_log_file_size_functionality.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/innodb_log_file_size_functionality.test 2014-03-11 09:28:16 +0000 > @@ -0,0 +1,124 @@ This should be renamed to sys_vars.innodb_log_file_size_func test > +############################################################################### > +# # > +# Variable Name: innodb_log_file_size # > +# Scope: Global # > +# Access Type: Static # > +# Data Type: numeric # > +# # > +# # > +# Creation Date: 2012-08-20 # > +# Author : Tanjot Singh Uppal # > +# # > +# # > +# Description:Test Cases of Static System Variable innodb_log_file_size # > +# that checks the behavior of this variable in the following ways # > +# * Value Check # > +# * Scope Check # > +# * Functionality Check # > +# * Accessability Check # > +# # > +# This test does not perform the crash recovery on this variable # > +# For crash recovery test on default change please run the ibtest # > +############################################################################### > + > +-- source include/have_innodb.inc > +-- source include/not_embedded.inc > +-- source include/have_innodb_16k.inc > + > +echo '#________________________VAR_09_INNODB_LOG_FILE_SIZE__________________#' > +echo '##' > +--echo '#---------------------WL6372_VAR_9_01----------------------#' > +#################################################################### > +# Checking default value # > +#################################################################### > +SELECT COUNT(@@GLOBAL.innodb_log_file_size); > +--echo 1 Expected > + > +SELECT @@GLOBAL.innodb_log_file_size; > +--echo 5242880 Expected > + > + > + > +--echo '#---------------------WL6372_VAR_9_02----------------------#' > +#################################################################### > +# Checking Value can be set - Static # > +#################################################################### > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SET @@local.innodb_log_file_size=1; > +--echo Expected error 'Read only variable' > + > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SET @@session.innodb_log_file_size=1; > +--echo Expected error 'Read only variable' > + > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SET @@GLOBAL.innodb_log_file_size=1; > +--echo Expected error 'Read only variable' > + > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SET @@GLOBAL.innodb_log_file_size=DEFAULT; > +--echo Expected error 'Read only variable' > + > +SELECT COUNT(@@GLOBAL.innodb_log_file_size); > +--echo 1 Expected > + > + > + > +--echo '#---------------------WL6372_VAR_9_03----------------------#' > +################################################################# > +# Check if the value in GLOBAL Table matches value in variable # > +################################################################# > +SELECT @@GLOBAL.innodb_log_file_size = VARIABLE_VALUE > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='innodb_log_file_size'; > +--echo 1 Expected > + > +SELECT COUNT(@@GLOBAL.innodb_log_file_size); > +--echo 1 Expected > + > +SELECT COUNT(VARIABLE_VALUE) > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='innodb_log_file_size'; > +--echo 1 Expected > + > + > + > +--echo '#---------------------WL6372_VAR_9_04----------------------#' > +################################################################################ > +# Checking Variable Scope # > +################################################################################ > +SELECT @@innodb_log_file_size = @@GLOBAL.innodb_log_file_size; > +--echo 1 Expected > + > +--Error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SELECT COUNT(@@local.innodb_log_file_size); > +--echo Expected error 'Variable is a GLOBAL variable' > + > +--Error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SELECT COUNT(@@SESSION.innodb_log_file_size); > +--echo Expected error 'Variable is a GLOBAL variable' > + > +SELECT COUNT(@@GLOBAL.innodb_log_file_size); > +--echo 1 Expected > + > +--Error ER_BAD_FIELD_ERROR > +SELECT innodb_log_file_size = @@SESSION.innodb_log_file_size; > +--echo Expected error 'Unknown column innodb_log_file_size in field list' > + > + > + > +--echo '#---------------------WL6372_VAR_9_05----------------------#' > +############################################################################### > +# Checking the /Var directory size # > +############################################################################### > +-- source include/vardir_size_check.inc > +--echo TRUE Expected > + > +--echo '#---------------------WL6372_VAR_9_06----------------------#' > +################################################################################# > +# Checking the size of ib_logfile # > +################################################################################# > +let MYSQLD_DATADIR=`SELECT @@datadir`; > +--source include/ib_logfile_size_check.inc > +--echo 5242880 expected > + > Regards, Sergei

1 0

[Maria-developers] PERFORMANCE_SCHEMA to be disabled in 10.0 (10.0.12)
by Colin Charles 12 Jun '14

12 Jun '14

Hi all, Recently there was chat about how PERFORMANCE_SCHEMA was enabled by mistake and it should be disabled in the 10.0 series. I'm curious - how many of you are using PERFORMANCE_SCHEMA? Is it a problem to turn it on, if you use it? I'm referring to: https://mariadb.atlassian.net/browse/MDEV-6249 Looking from the webscalesql list, there are examples of performance degradation: https://groups.google.com/d/msg/webscalesql/B-4bPIb8mHI/eUNsl0lSfbMJ https://groups.google.com/d/msg/webscalesql/B-4bPIb8mHI/-YZMMCuPw3QJ quote: "Our perf testing agrees with your assessment (we see about a 5%-6% perf hit when it's included and on, and a 2%-3% hit when it's included but off)" Please discuss this, either here or on MDEV-6249 Thanks cheers, -colin -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

6 10

[Maria-developers] Extendet FederatedX storage engine to support updating Sphinx RT indexes
by Markus Lidel 12 Jun '14

12 Jun '14

Hello, the Sphinx Search engine has RT indexes, which could be updated with a subset of SQL commands, refered to as SphinxQL: http://sphinxsearch.com/docs/2.1.1/sphinxql-reference.html Because the FederatedX storage engine uses some commands, which Sphinx does not know, here is a patch, wich extends the FederatedX storage engine to support this subset of commands. Now its possible to update Sphinx RT indexes directly from MariaDB. To create a connection to the RT index in the Sphinx default RT-index: index rt { type = rt path = @CONFDIR@/data/rt rt_field = title rt_field = content rt_attr_string = title rt_attr_string = content rt_attr_uint = gid } you just have to call: CREATE TABLE `rt` ( `id` BIGINT UNSIGNED NOT NULL, `title` TEXT, `content` TEXT, `gid` INT UNSIGNED ) ENGINE=FEDERATED CONNECTION='sphinxql://root@localhost:9306/rt'; It would be great, if the attached patch could be integrated into MariaDB. If you have any suggestions, please feel free to contact me. Best regards, Markus Lidel

4 8

Re: [Maria-developers] [GSoC] Accepted student ready to work : )
by Pablo Estrada 10 Jun '14

10 Jun '14

Hello all, I'm still interested on a call, (I would make sure to write down everything as part of an email to keep it clearly organized here), but as Elena suggested, I'll try to look at Sergei's bzr repo first, and all the documentation. In any case, I already have a decent idea of how the data is arranged. I can see how it is difficult to trace test results back to code changes directly, but it might be possible. I still haven't figured out how to recognize what code changes pertain to the same 'history', or 'line' (I think that you were not considering this in your experiments... and it might turn out to not be relevant at all, but I still want to consider it, to try to make a more complete model). So I have just two questions that should be easy to answer by email: 1. In your graphs, exactly, what do you mean by Precision? What about Recall? And cutoff? 2. I looked at your launchpad profile but I can't seem to find the repo with your tests (if you even pushed it to lp at all). Is it available? That's it for today : ) Until next time. Pablo On Thu, Apr 24, 2014 at 12:28 AM, Sergei Golubchik <serg(a)skysql.com> wrote: > On April 23, 2014 10:07:23 AM CEST, Pablo Estrada wrote: > > >If it's not too inconvenient, do you think we could set up a Google > >Hangout > >or a Skype call on Monday to go over a few questions quickly? > > > Definitely. This week I'm on vacations and without a laptop, back home on > Sunday. So, Monday is fine. But please if you have specific questions - ask > them now in emails. I'll prepare answers for Monday, but it'll take time to > remember everything. That is, if you will wait with questions till the > call, I might not be able to remember the answer on the spot. > Regards, > Sergei >

4 17

Re: [Maria-developers] Rev 4234: Fixed bug mdev-6071
by Sergei Golubchik 10 Jun '14

10 Jun '14

Hi, Igor! On Jun 09, Igor Babaev wrote: > revno: 4234 > revision-id: igor(a)askmonty.org-20140610050716-0x8xaquf3itvofjc > parent: bar(a)mariadb.org-20140606062952-x2jwt4mgrgr6tpw6 > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-10.0-bugs > timestamp: Mon 2014-06-09 22:07:16 -0700 > message: > Fixed bug mdev-6071. > The method JOIN_CACHE::init may fail (return 1) if some conditions on the > used join buffer is not satisfied. For example it fails if join_buffer_size > is greater than join_buffer_space_limit. The conditions should be checked > when running the EXPLAIN command for the query. That's why the method > JOIN_CACHE::init has to be called for EXPLAIN commands as well. It's an improvement over the old behavior - as EXPLAIN shows correctly whether BKA is used. But I think there's still not enough diagnostics. In the bug report Sergey wrote that " I think it's a big gotcha that @join_buffer_size must be lower than @@join_buffer_space_limit when optimizer_space_limit optimization is off. " So, I'd believe users will be very confused that in some cases BKA is suddenly turned off, when they expect it to be used. Of course, without this bugfix they would've not even noticed that, which is even worse. So, this bugfix is good. But it would be nice to have some kind of an explanation of why BKA is not used. At the very least in the manual. Better - here as a warning, like if (for_explain) push_warning("BKA was not used because ..."; goto fail; > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2014-06-05 22:07:27 +0000 > +++ b/sql/sql_select.cc 2014-06-10 05:07:16 +0000 > @@ -10541,7 +10541,7 @@ > if (cache_level == 1) > prev_cache= 0; > if ((tab->cache= new JOIN_CACHE_BNL(join, tab, prev_cache)) && > - ((options & SELECT_DESCRIBE) || !tab->cache->init())) > + !tab->cache->init(MY_TEST(options & SELECT_DESCRIBE))) MY_TEST here (and in other similar places) is redundant, you could've simply written !tab->cache->init(options & SELECT_DESCRIBE)) > { > tab->icp_other_tables_ok= FALSE; > return (2 - MY_TEST(!prev_cache)); and while it wasn't part of your bug fix, I cannot help but noticing that MY_TEST() is redundant here too. !prev_cache is boolean, it's always 0 or 1. Regards, Sergei

1 0

Re: [Maria-developers] Google Code for Encryption of MariaDB
by Elmar Eperiesi-Beck 09 Jun '14

09 Jun '14

Hi, 1. what about an NDA between us, to give me the chance to have a pre-view. I will make sure, that your architecture and development fits into our work. Everything we can think of in an early stage does not make lot of work, but… 2. like. 3. We will post our concept next week into the MariaDB Developers community. 4. … or 1. Regards Elmar Am 09.06.2014 um 11:14 schrieb Jonas Oreland : > Hi there, > > 1) repo: as I said in my mail, the code is currently not yet published, > but the guy doing this said that he expects it to be with in a couple of weeks or so. > before this, i don't think you can see it. > > once it's published, I think you can find it at https://code.google.com, > but I'll check and let you know. > > 2) This is my personal email address, so I prefer if we keep the conversation > @maria-developers where I'm subscribed with work email address, if this is ok > by you ? > > 3) concept paper: sure > > 4) we don't really have a concept paper, but the code is complete and being rolled out. > I can answer questions about it, but I think it's easiest once it has been published. > > /Jonas > > > > On Mon, Jun 9, 2014 at 11:02 AM, Elmar Eperiesi-Beck wrote: > Jonas, > is there a repository, where I can find the actual source code of your encryption implementation? > Or do we have to sign an NDA to exchange these kind of information? > > We can have a look at your code and merge it with our existing implementation. > Additionally I would like to send you our concept paper, to give you an overview of our ideas. > > Regards > Elmar >

1 0

[Maria-developers] [GSoC] Week 3!
by sriram patil 09 Jun '14

09 Jun '14

Hi all, Here is my post for the 3rd week into GSoC project. http://goo.gl/bhl5ZW Suggestions and review are welcome! Thanks, Sriram

1 0

[Maria-developers] MariaDB 5.5.38 Release Prep
by Daniel Bartholomew 07 Jun '14

07 Jun '14

All, I've begun prep work on the MariaDB 5.5.38 release. Draft release notes and changelog are here: - https://mariadb.com/kb/en/mariadb-5538-release-notes/ - https://mariadb.com/kb/en/mariadb-5538-changelog/ Current plan is to activate and announce the release on Mon, 9 jun 2014. Thanks. -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

1 0

[Maria-developers] MDEV-5533 - Backport Threadpool improvements from Percona Server
by Sergey Vojtovich 04 Jun '14

04 Jun '14

Hi Sergei, Since MDEV-5533 is marked critical for 10.0.12 it is a good time to start this discussion. IIRC we agreed that our intention is to accept default value change for thread_pool_max_threads into 10.0 and accept priority queue throttling into 10.1. So default value for thread_pool_max_threads has bumped in percona server from 500 to 100 000. The reason is: deadlocks and uneven distribution of worker threads between thread groups in case of stalled connections. But the above is contradicting to suggestions in KB: https://mariadb.com/kb/en/threadpool-in-55/ ... so ideally, there should be a single active thread for each CPU on the machine. ... If your application constantly pegs at 500 threads, it might be a strong indicator for high contention in the application, and threadpool does not help much. ... One way to solve the problem would be to set thread_pool_max_threads to a very high value, but this might be less than ideal, and negatively affect performance. ... Said the above, do you still think it is a good idea to raise default value of thread_pool_max_threads? If so, what should KB article state wrt this variable? Thanks, Sergey

1 0

Re: [Maria-developers] MDEV-6213: Query cache is not invalidated on cascade delete when database name contains special symbols
by Sergei Golubchik 03 Jun '14

03 Jun '14

Hi, Sanja! Please add in your revision a test case for MySQL bug#13919851 http://bazaar.launchpad.net/~mysql/mysql-server/5.6/revision/4514 and make sure that your fix doesn't break it. On May 09, sanja(a)askmonty.org wrote: > revno: 4207 > revision-id: sanja(a)askmonty.org-20140509104240-ksrk10tef5dq157z > branch nick: work-maria-10.0-MDEV-6213 > timestamp: Fri 2014-05-09 13:42:40 +0300 > message: > MDEV-6213: Query cache is not invalidated on cascade delete when > database name contains special symbols. > > Innodb should de-escape db/table names when sending them to QC Regards, Sergei

1 0

[Maria-developers] Please review a patch fixing a few MDEV-6001 blockers
by Alexander Barkov 03 Jun '14

03 Jun '14

Hi Sergei, can you please review my patch fixing a few bugs that are blockers for MDEV-6001? - MDEV-4858 Wrong results for a huge unsigned value inserted into a TIME column - MDEV-6097 Inconsistent results for CAST(int,decimal,double AS DATETIME) - MDEV-6099 Bad results for DATE_ADD(.., INTERVAL 2000000000000000000.0 SECOND) - MDEV-6100 No warning on CAST(9000000 AS TIME) - MDEV-6101 Hybrid functions do not add CURRENT_DATE when converting TIME to DATETIME - MDEV-6102 Comparison between TIME and DATETIME does not use CURRENT_DATE Thanks.

2 10

[Maria-developers] Review request[major 5.3/5.5 bug]: MDEV-4511 Assertion `scale <= precision' fails on GROUP BY TIMEDIFF with incorrect types, , Agile Board, , Export
by Alexander Barkov 03 Jun '14

03 Jun '14

Hello Sergei, A few weeks ago we fixed this bug for 10.0 by changing temporal items not to have decimals=NOT_FIXED_DEC (more MySQL-5.6 compatibility), and agreed to have a separate patch for the pre-10.0 branches. This patch is for 5.3 and 5.5 only. It back-ports the mdev-4511 tests from 10.0 as is. But it does not touch the NOT_FIXED_DEC behavior of the temporal items and fixes the problem in a different way. Greetings.

1 0

[Maria-developers] POWER interest
by Colin Charles 03 Jun '14

03 Jun '14

Hi, It seems we have a POWER8 box and soon will start doing builds on this platform Stewart Smith (currently at IBM) seems to be doing some work on MySQL 5.6 and MySQL 5.7 for POWER8 systems - there doesn't seem to be much attention being paid to MariaDB 10. Ideally, our fixes that are being picked up from Stewart or focused on POWER should be targeted at 10.1 I'm thinking (unless there's customer interest for the 10.0 series -- which may very likely be the case) So far, I've noted GIS precise has an open failure, and maybe its a great idea for Holyfoot to use the POWER8 system to start fixing it: https://mariadb.atlassian.net/browse/MDEV-4153 From MySQL bugs system we see multiple examples from Stewart's blog posts - some are InnoDB specific so we will inherit them during a merge - some aren't like mysql#72715 (character set code endianness dependent on CPU type rather than endianness of CP) https://www.flamingspork.com/blog/2014/06/03/1-million-sql-queries-per-seco… https://www.flamingspork.com/blog/2014/06/03/mysql-5-7-on-power/ Relevant stuff that we may want to look at: http://bugs.mysql.com/search.php?cmd=display&status=Active&tags=PowerPC cheers, -colin -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

1 0

[Maria-developers] [GSoC] First two weeks into coding!
by sriram patil 03 Jun '14

03 Jun '14

Hi all, I have created a blog to write the details of the work that I do as part of GSoC'14. Following is the link to my first blog post explaining the details of the work that I have done during the last two weeks. http://goo.gl/PYoVZb I will be writing these blog posts every weekend from now on and will share it on the developer mailing list every Monday. Suggestions/reviews are welcome. Thanks, Sriram

2 2

[Maria-developers] Possible race condition or NULL pointer triggered by OQGraph
by Andrew McDonnell 02 Jun '14

02 Jun '14

Hi all (Cross-posted to oqgraph-developers, maria-developers) I am trying to track down a segfault apparently triggered by concurrent execution of queries through OQGraph - for context, see: https://mariadb.atlassian.net/browse/MDEV-6282 I am however a bit confused as to what is going on, because at least some of the information leads me to suspect that there could be more going on here than just a simple race condition. --Immediate Cause-- On the surface, a segfault appears to be caused in a method in sql_class.h, Statement::check_limit_rows_examined() dereferencing a NULL pointer, `lex`. The method check_limit_rows_examined() was called from sql_class.h method handler::increment_statistics() . --Analysis-- A quick inspection reveals that a NULL value for `lex` is actually perfectly valid some of the time, although I am nowhere near familiar enough with the code to really understand when or why. Specifically, `lex` is a member of the base class Statement, which is initialised to either point to THD::main_lex via a constructor , or intentionally to NULL from construction of an object Prepared_statement() (sql_prepare.cc) To my mind, if there is going to be a possibility of a pointer being NULL, then all uses of that pointer should be checking for that. I am hoping that the bug reporter can prove or otherwise my hypothesis with his particular scenario, although I am suspicious that there may be additional issues to be dealt with; it is quite possible that lex being NULL is a red herring, being caused by a race condition memory overrun elsewhere. However, I think it is still a good idea to NULL check pointers that could be set to NULL intentionally. I can submit a pull request if the following patch is agreeable: diff --git a/sql/sql_class.h b/sql/sql_class.h index 5898d9e..957acf3 100644 --- a/sql/sql_class.h +++ b/sql/sql_class.h @@ -2475,8 +2475,10 @@ public: */ void check_limit_rows_examined() { - if (++accessed_rows_and_keys > lex->limit_rows_examined_cnt) + ++accessed_rows_and_keys; + if (lex && accessed_rows_and_keys > lex->limit_rows_examined_cnt) { killed= ABORT_QUERY; + } } --Aside-- Immediately prior to the check_limit_rows_examined() call is a call to macro status_var_increment(). Comments around the macro indicate it is often likely to be called in a mutli-threaded context, but the data is such that it is possible to disable protection with the only side effect being an occasionally 'inaccurate' result. Is it possible however, that check_limit_rows_examined() is not thread-safe and should have some protection? cheers, Andrew --A

3 3

[Maria-developers] MDEV-4051 INET6_ATON() and INET6_NTOA()
by Alexander Barkov 02 Jun '14

02 Jun '14

Hello Ian, I have merged these functions from MySQL-5.6 into 10.0: INET6_ATON() INET6_NTOA() IS_IPV4() IS_IPV4_COMPAT() IS_IPV4_MAPPED() IS_IPV6() Can you please document them? The corresponding article in the MySQL manual is here: http://dev.mysql.com/doc/refman/5.6/en/miscellaneous-functions.html Thanks.

2 2

[Maria-developers] KB page on switching from bazaar to git.
by Sergey Petrunia 27 May '14

27 May '14

Hi, I was doing some stuff in git already, and so I have created a git cheatsheet with equivalents of bzr commands that were useful in MariaDB development: https://mariadb.com/kb/en/using-git/ It is not finished, any additions are welcome. Elena has agreed to look at creating a commit trigger that sends commit emails (currently, github sends emails on push (not commit), and these do not include diffs) BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] CREATE OR REPLACE DATABASE algorithm
by sriram patil 27 May '14

27 May '14

Hi Alexander and all, Following are the steps that I am following for implementing "CREATE OR REPLACE" for DATABASE "..." signifies, there exists one or more lines of code in that block of the function. mysql_create_db(...) ... ... if(db exists) { if(create_or_replace) { if(DROP_ACL access not granted) { // Error 1 DBUG_RETURN(-1) } if(mysql_rm_db(...) unsuccessful) { // Error 2 DBUG_RETURN(-1) } Reset m_status from diagnostics cause it is changed by mysql_rm_db. Proceed with normal create db. } else if(if not exists) { ... } else ... } ... ... } Is the above algorithm correct? Also, I am not yet sure which errors should be displayed at both the places "Error 1" and "Error 2". I have written and tested the above code. It is working fine. Testing involves running all the existing test cases and executing "CREATE OR REPLACE DATABASE" command on the console manually. "IF NOT EXISTS" gives error if used with "CREATE OR REPLACE". Any other test cases I need to consider? Thanks, Sriram

2 2

[Maria-developers] FOR JSON
by Alexander Barkov 26 May '14

26 May '14

Hello Monty, We discussed extracting data in JSON format today, to make it possible to return data as a single JSON value (as a result set consisting of a single row and a single column): SELECT id,name,address FROM users FOR JSON; Note, in some cases the result of a such query can obviously be very huge. So it would be nice to be able to use "FOR JSON" in subselects, like this: SELECT id, (SELECT name,address FROM users AS b WHERE a.id= b.id FOR JSON) AS RowJSON FROM users AS a; This query will return a result set with multiple rows (one row per one record) and with two columns: - the ID, as a regular number - the JSON value with "name" and "address" embedded into it. Or, another example, if one does not need "id" as a separate column: SELECT (SELECT id,name,address FROM users AS b WHERE a.id= b.id FOR JSON) AS RowJSON FROM users AS a; This query will return a result set with multiple rows (one row per one record) with a single JSON value embedding all three columns: "id", "name" and "address".

2 4

[Maria-developers] MariaDB 10.1 is now on github!
by Sergei Golubchik 26 May '14

26 May '14

Hi! I'm happy to announce that MariaDB-10.1 tree has been completely migrated to github. Since now on we'll use github for the new development in MariaDB. It's https://github.com/MariaDB/server, go on, fork it, hack around, submit pull requests. Have fun! Older trees (10.0, 5.5, 5.3, 5.2, 5.1) are not on github - we do hope to migrate them too eventually, but at the moment they are still on launchpad. If you're a maria-captain on launchpad - for you to get write access to the mariadb repository on github you need to tell me your github account name. Regards, Sergei P.S.: Don't forget that 10.1 is still *pre-alpha*. You've been warned.

4 5

Re: [Maria-developers] MariaDB 10.1 is now on github!
by Sergei Golubchik 26 May '14

26 May '14

Hi, Binod! First: please send questions like this to maria-developers(a)lists.launchpad.net it's a public mailing list dedicated to MySQL and MariaDB internals, source code, and related things. I am subscribed, so I'll see you mail there, and you may be sure I will, because it won't be accidentally catched by my spam filter, or sorted out in some obscure folder. Furthermore other subscribers will see your question and could reply if I will be not available (e.g. I could be travelling). Thank you. On May 26, Binod Bista wrote: > Hi Sergei, > > Thanks for the info. Just to interrupt, is there a page you recommend for > anyone starting out in MariaDB development? Particularly any database or > MySQL internals material that help understand the codebase. That depends on what you're trying to do. There are two books: http://shop.oreilly.com/product/9780596009571.do http://www.packtpub.com/mysql-51-plugin-development/book There's MySQL internals manual: http://dev.mysql.com/doc/internals/en/index.html There's MariaDB Knowledge Base that includes pages about MariaDB internals: http://kb.askmonty.org/ There're numerous blog posts and presentations about specific topics. Regards, Sergei

1 0

[Maria-developers] Getting started with the coding.
by sriram patil 23 May '14

23 May '14

Hi Alexander, I have started working on "CREATE OR REPLACE DATABASE" implementation. I thought this one has the least complexity among all the other commands. I am getting my hand-on debugging the source code too. I have made the required changes in sql_yacc.yy. The command is parsed properly. Handled the error where CREATE OR REPLACE and IF NOT EXISTS cannot occur at the same time. This is how I am thinking of proceeding with the implementation, In method mysql_create_db after checking the flag HA_LEX_CREATE_REPLACE. I am removing the database by calling mysql_rm_db and then the current implementation will take care of creating the new db with the same name. This seems correct? Or is there anything I need to consider? Thanks, Sriram

2 1

[Maria-developers] delete + offset
by Roberto Spadim 22 May '14

22 May '14

hi guys! i was thinking at maria-discuss, and i have a new "feature request" to mariadb DELETE statment, that's the OFFSET clausule, like sqlite my idea is... i have a table with history about for example temperature, and i will only record last 5000 records, my idea is execute a 'cleanup' for example every day instead of: select date_time from history where channel=1234 order by date_time desc limit 1 offset 5000 delete from history where channel=1234 and data_time<value from previous query i was considering something like delete from history where channel=1234 order by date_time desc limit 99999999999 offset 5000 what you think about this? -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

[Maria-developers] Regarding debugging MariaDB
by sriram patil 17 May '14

17 May '14

Hi Alexander, I wanted to run mysqld in debug mode to check the trace file and get to know workflow of SQL commands. I tried following command mysqld --user=<username> --debug Then, after connecting I executed some commands. But, the trace file is not created in /tmp folder. Is there anything I need to do while building the source code in order for the debug mode to work? Thanks, Sriram

2 1

[Maria-developers] Merging FusionIO changes to 10.0.12
by Colin Charles 16 May '14

16 May '14

Hi all, MariaDB has worked actively with Fusion-io, and we did release some improvements in a preview release a little over a month ago: https://blog.mariadb.org/significant-performance-boost-with-new-mariadb-pag… This was originally planned to be included into 10.0.10, but when we released the GA, we thought that the feature was not tested thoroughly. Now the code has proven stable we are thinking of merging this into 10.0.12. Yes, we know its already a GA, but this feature will be turned off by default, and the code is safely encapsulated. If you are not using Fusion-io or do not turn it on, there is no effect to you. To check, the changes are in the XtraDB and InnoDB that we ship (storage/xtradb & storage/innobase). Naturally, for those using Fusion-io devices, the benefits are clear - you will see significant improvements in speed and you will increase the lifetime of your cards. Think of this like adding drivers to the Linux kernel from time to time ;) What does everyone think? Package maintainers? Thanks! -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

3 2

[Maria-developers] LIMIT ROWS MATCHED
by Colin Charles 16 May '14

16 May '14

Hi! http://seonguck.blogspot.kr/2014/05/limit-rows-matched.html We have a potential contribution here for a LIMIT ROWS MATCHED feature - might be interesting for us to consider this for 10.1 ? cheers, -colin -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

2 1

[Maria-developers] tokudb database - generated "should never happen" status when mysql log partition ENOSP
by Daniel Black 15 May '14

15 May '14

1 0

[Maria-developers] New to the list & question about CONNECT
by Martijn Tonies (Upscene Productions) 13 May '14

13 May '14

Hello everyone, My name is Martijn Tonies and I’m a Windows tools developer, we have a tool called “Database Workbench”, a developer tool for several database engines. For it’s next major version, we’re planning to officially support MariaDB in the MySQL module. I’ve investigated several of the features, implemented support for Roles, Virtual Columns etc, but I’m having trouble with the CONNECT engine. Here’s what I did, using MariaDB 10.0.10 1) registered a server with CREATE SERVER: CREATE SERVER mysql55_at_local FOREIGN DATA WRAPPER mysql OPTIONS(HOST 'MT-XP-VM-MYSQL', PORT 3355, DATABASE 'test', USER 'root', PASSWORD ‘secret’); 2) created a table in the MySQL database: CREATE TABLE sfdata ( mark1 Integer(11), mark2 Integer(11), mark3 Integer(11), mark4 Integer(11), `name` VarChar(50) CHARACTER SET latin1 COLLATE latin1_swedish_ci ) ENGINE=InnoDB DEFAULT CHARACTER SET latin1 COLLATE latin1_swedish_ci; 3) created a table in the MariaDB database: CREATE TABLE `sfdata_proxy3` ( `mark1` int(11) DEFAULT NULL, `mark2` int(11) DEFAULT NULL, `mark3` int(11) DEFAULT NULL, `mark4` int(11) DEFAULT NULL, `name` varchar(50) DEFAULT NULL ) ENGINE=CONNECT DEFAULT CHARSET=utf8 CONNECTION='mysql55_at_local/sfdata' I then expected that table “sfdata_proxy3” has the rows as “sfdata” has, but I got nothing returned. Any clue where to look? With regards, Martijn Tonies Upscene Productions http://www.upscene.com Download Database Workbench for Oracle, MS SQL Server, Sybase SQL Anywhere, MySQL, InterBase, NexusDB and Firebird!

1 1

Re: [Maria-developers] Question about GTIDs in MariaBD
by Kristian Nielsen 12 May '14

12 May '14

"AL13N" <alien(a)rmail.be> writes: >> Peter Laursen <peter_laursen(a)webyog.com> writes: >> So yes, there are others that consider Oracle's design of GTID 'a dirty >> hack', >> but MariaDB 10.0 GTID is not affected by that design. > What about mariadb being a slave of a Oracle MySQL GTID? The restrictions are on the master, not on the slaves. So what do you mean? Surely, connecting a MariaDB slave to an Oracle MySQL master will not remove those restrictions. Note that when a MariaDB server connects as a slave to an Oracle MySQL master server, it is not possible to use GTID for that connection (because the GTID implementations are not compatible). Old style non-GTID replication must be used. - Kristian.

2 1

Re: [Maria-developers] Question about GTIDs in MariaBD
by Peter Laursen 12 May '14

12 May '14

With autocommit=1 every statement runs in its own transaction, so obviousluy .. yes! MyISAM is/used to be completely 'transaction-agnosstic'. But GTIDs seem to confict with non-transactional storage engines (as far as I can understand). MySQL/Oracle have "solved" (!) this by not allowing updates to tables using transactional engines and non-transactional storage engines in same transaction. However to me it is nonsense that transactions apply (have any effect at all) for non-transactional storage engines. As a consequence I consider this solution (what must have been enforced in the server layer and not the storeage engine layer) by Oracle 'a dirty hack' (unless somewone can explain me why it it necessary). -- Peter On Sun, May 11, 2014 at 4:42 PM, Roberto Spadim <roberto(a)spadim.com.br>wrote: > With autoclmmit=1 works? > > Em domingo, 11 de maio de 2014, Peter Laursen <peter_laursen(a)webyog.com> > escreveu: > > Thanks for the reply, but I am afraid I don't fully understand. >> >> Here is a simple test case with mySQL 5.6.17: >> >> 1) >> You need all those in configuration [mysqld) section >> >> log-bin >> log-slave-updates >> gtid_mode = ON >> enforce_gtid_consistency = ON >> >> >> 2) >> Then this small script will reproduce: >> >> >> DROP DATABASE IF EXISTS `gtid_test`; >> CREATE DATABASE `gtid_test`; >> USE `gtid_test`; >> >> CREATE TABLE `tab1` ( >> `a` INT DEFAULT NULL >> ) ENGINE=MYISAM ; >> >> CREATE TABLE `tab2` ( >> `b` INT >> ) ENGINE=INNODB; >> >> --start a transaction >> SET autocommit=0; >> >> INSERT INTO `tab1` VALUES (1); >> INSERT INTO `tab2` VALUES (1); >> >> -- now the first UPDATE to the MyISAM table in the transaction triggers >> this error: >> >> UPDATE `tab1` SET `a` = 5 WHERE `a` = 1; >> >> -- and you'll get: >> -- >> -- Error Code: 1785 >> -- When @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1, updates to >> non-transactional tables can only be done in either autocommitted >> statements or single-statement transactions, and never in the same >> statement as updates to transactional tables. >> >> >> 3) >> ShouldI understand that MariaDB dos not prevent this (does not raise an >> error), but it is still unsafe as binlog - and thus replication - may be >> corrupted? >> >> >> Thanks! >> >> -- Peter >> >> On Sat, May 10, 2014 at 6:09 PM, Pavel Ivanov <pivanof(a)google.com> wrote: >> >>> There is no such limitation in MariaDB as well as no limitation on >>> CREATE TABLE ... SELECT. >>> >>> Although beware that mixing statements changing MyISAM and InnoDB >>> tables in statement binlog mode you may get different results from the >>> statements on slaves. >>> >>> Pavel >>> >>> On Sat, May 10, 2014 at 3:48 AM, Peter Laursen <peter_laursen(a)webyog.com> >>> wrote: >>> > GTID implementation in MySQL 5.6 has the limitation as described here: >>> > >>> https://dev.mysql.com/doc/refman/5.6/en/replication-gtids-restrictions.html >>> > >>> > >>> > "nontransactional storage engines such as MyISAM cannot be made in the >>> same >>> > statement or transaction as updates to tables using transactional >>> storage >>> > engines such as InnoDB." >>> > >>> > If you try you will get the error: >>> > Error Code: 1785 >>> > When @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1, updates to >>> non-transactional >>> > tables can only be done in either autocommitted statements or >>> > single-statement transactions, and never in the same statement as >>> updates to >>> > transactional tables. >>> > >>> > Is there same or similar/other limitations with GTID in MariaDB 10? >>> > >>> > -- Peter Laursen >>> > -- Webyog >>> > >>> > _______________________________________________ >>> > Mailing list: https://launchpad.net/~maria-developers >>> > Post to : maria-developers(a)lists.launchpad.net >>> > Unsubscribe : https://launchpad.net/~maria-developers >>> > More help : https://help.launchpad.net/ListHelp >>> > >>> >> >> > > -- > Roberto Spadim > SPAEmpresarial > Eng. Automação e Controle > >

3 3

[Maria-developers] Question about GTIDs in MariaBD
by Peter Laursen 11 May '14

11 May '14

GTID implementation in MySQL 5.6 has the limitation as described here: https://dev.mysql.com/doc/refman/5.6/en/replication-gtids-restrictions.html "nontransactional storage engines such as MyISAM<https://dev.mysql.com/doc/refman/5.6/en/myisam-storage-engine.html> cannot be made in the same statement or transaction as updates to tables using transactional storage engines such as InnoDB<https://dev.mysql.com/doc/refman/5.6/en/innodb-storage-engine.html> ." If you try you will get the error: Error Code: 1785 When @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1, updates to non-transactional tables can only be done in either autocommitted statements or single-statement transactions, and never in the same statement as updates to transactional tables. Is there same or similar/other limitations with GTID in MariaDB 10? -- Peter Laursen -- Webyog

2 3

[Maria-developers] MariaDB 10.0.11 release prep
by Daniel Bartholomew 10 May '14

10 May '14

I've started the release prep for MariaDB 10.0.11. Draft release notes and changelog pages are here: - https://mariadb.com/kb/en/mariadb-10011-release-notes/ - https://mariadb.com/kb/en/mariadb-10011-changelog/ Please make necessary additions/modifications/etc... The release is still building at the moment, but when finished I'll upload to the mirrors later today or tomorrow. Current plan, if everything goes well, is to activate and announce the release on Monday, 12 May. Thanks! -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

1 0

[Maria-developers] 10.0 -> 10.1 merge
by Sergey Petrunia 08 May '14

08 May '14

Hello Igor, As discussed, I've done the 10.0 -> 10.1 merge. I didn't get any warnings or new test failures in the 'main' test suite, so I've pushed the merge into 10.1 tree. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] connect.alter fails
by Alexander Barkov 07 May '14

07 May '14

Hello Olivier, connect.alter fails when built without XML support complaining about unsupported table type XML. I guess it should be skipped. connect.alter also fails with libxml2 with these symptoms: --- /home/bar/maria-bzr/maria-10.0-connect/storage/connect/mysql-test/connect/r/alter.result 2014-03-23 11:32:34.519624000 +0400 +++ /home/bar/maria-bzr/maria-10.0-connect/storage/connect/mysql-test/connect/r/alter.reject 2014-03-23 11:48:43.251680097 +0400 @@ -1,3 +1,5 @@ +Warnings: +Warning 1105 No file name. Table will use t1.xml Please fix. A remainder: please check every push in BuildBot: http://buildbot.askmonty.org/buildbot/grid?branch=10.0-connect

3 4

[Maria-developers] Barkov: report for the week starting Apr 29, 2014
by Alexander Barkov 05 May '14

05 May '14

Apr 28 - May 04 =============== Apr 28 Apr 29 Apr 30 May 01 Holiday May 02 Holiday May 03 Weekend May 04 Weekend Vacation days taken: 10/35 - MDEV-5702 Incorrect results are returned with NULLIF() Pushed into 5.5 - MDEV-5459 Illegal mix of collations for datetime Pushed into 5.5 - MDEV-6146 Can't mix (latin1_swedish_ci,NUMERIC) and (utf8_unicode_ci,IMPLICIT An after-fix pushed (the original patch broke "mtr --ps fulltext" - Merging gb18030 from MySQL-5.7 In progress - Discussion the manual changes with Ian

1 0

[Maria-developers] sources won't compile
by AugustQ 03 May '14

03 May '14

Hi, I encountered a problem by compiling the sources of MariaDB 10.04. Some days ago I upgraded my system from Ubuntu 13.10 to 14.04 and now I encounter these problems: gcc used is this: gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) a minor glitch: cmake complains: Syntax Warning in cmake code at /home/august/workspace/MariaDB/mariadb-10.0.4/contrib_AQ_01/cmake/ssl.cmake:210:55 Argument not separated from preceding token by whitespace. This is the reason for this: ELSE() MESSAGE(SEND_ERROR "Wrong option for WITH_SSL. Valid values are : "${WITH_SSL_DOC}) ENDIF() I added a space between "$(WITH and this works now. But in compiling the sources this error was reported: Scanning dependencies of target innobase [ 45%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/row/row0log.cc.o /home/august/workspace/MariaDB/mariadb-10.0.4/contrib_AQ_01/storage/innobase/row/row0log.cc: In function ‘dberr_t row_log_table_apply_ops(que_thr_t*, row_merge_dup_t*)’: /home/august/workspace/MariaDB/mariadb-10.0.4/contrib_AQ_01/storage/innobase/row/row0log.cc:2201:11: error: ‘srv_buf_size’ was not declared in this scope ofs, srv_buf_size); ^ By looking into the code of row0log.cc I found that FALLOC_FL_PUNCH_HOLE seems to be defined now. This is defined in /usr/include/linux/falloc.h. So what happened here during the upgrade of Linux? Regards AugustQ PS: by looking into Ubuntu 12.04 I found that this file also existed here.

2 2

[Maria-developers] Analysing degraded performance at high concurrency in sysbench OLTP
by Kristian Nielsen 02 May '14

02 May '14

At the Barcelona meeting in January, I promised to take a look at the high-concurrency sysbench OLTP benchmarks, and now I finally had the time do do this. There was a lot of work on LOCK_open by Svoj and Serg. If I have understood correctly, the basic problem was that at high concurrency (like, 512 threads), the TPS is only a small fraction of the peak throughput at lower concurrency. Basically, the server "falls over" and starts trashing instead of doing real work, due to some kind of inter-processor communication overhead. I started from Axel' OLTP sysbench runs and scripts, using 10.0 from bzr revno:4151 (revid:svoj@mariadb.org-20140415072957-yeir4jvokyilw5hp). I compiled without performance schema and with PGO, and ran sysbench 0.5 OLTP. (I just realised that my runs are with 32 tables, while I think the benchmarks in January focused on single-table runs. Maybe I need to re-do my analysis with the single-table benchmark, or perhaps it is too artificial to matter much?). In the read-only sysbench, the server mostly does not fall over. I guess this is due to the work by Svoj on eliminating LOCK_open? But in read-write, performance drops dramatically at high concurrency. TPS drops to 2600 at 512 threads compared to a peak of around 13000 (numbers here are approximate only, they vary somewhat between different runs). So I analysed the r/w benchmark with the linux `perf` tool. It turns out two-thirds of the time is spent in a single kernel function _raw_spin_lock(): - 66.26% mysqld [kernel.kallsyms] [k] _raw_spin_lock Digging further using --call-graph, this turns out to be mostly futex waits (and futex wakeups) from inside InnoDB locking primitives. Calls like sync_array_get_and_reserve_cell() and sync_array_wait_event() stand out in particular. So this is related to the non-scalable implementation in InnoDB of locking primitives, which is a known problem. I think Mark Callaghan has written about it a couple of times. Last time I looked at the code, every single mutex wait has to take a global mutex protecting some global arrays and stuff. I even remember seeing code that at mutex release would pthread_signal_broadcast() _every_ waiter, all of them waking up, only to all (except one) go do another wait. This is a kiler for scalability. While investigating, I discovered the variable innodb_sync_array_size, which I did not know about. It seems to split the mutex for some of the synchronisation operations. So I tried to re-run the benchmark with innodb_sync_array_size set to 8 and 64. In both cases, I got significant improvement, TPS increase to 5900, twice the value with innodb_sync_array_size set to the default of 1. So it is clear that the main limitation in this benchmark was the non-scalable InnoDB synchronisation implementation. After tuning innodb_sync_array_size, time spent in _raw_spin_lock() is down to half what it was before (33% of total time): + 33.77% mysqld [kernel.kallsyms] [k] _raw_spin_lock Now investigating call-graphs show that the sync_array operations are much less visible. Instead mutex_create_func(), called from dict_mem_table_create(), is the one that turns up prominently in the profile. I am not familiar with what this part of the InnoDB code is doing, but what I saw from a quick look is that it creates a mutex - and there is another global mutex needed for this, which again limits scalability. It is a bit surprising to see mutex creation being the most significant bottleneck in the benchmark. I would have assumed that most mutexes could be created up-front and re-used? It is possible that this is a warm-up thing, maybe the code is filling up the buffer pool or some table-cache like thing inside InnoDB? Because I see TPS being rather low for the first 150 seconds of the run (around 3000), and then increasing suddenly to around 8000-9000 for the rest. This might be worth investigating further. So in summary, my investigations found that the bottleneck in this benchmark, and the likely cause of the fall-over, is a scalability problem with InnoDB locking primitives. The sync_array part seems to be mitigated to some degree by innodb_sync_array_size, the mutex creation part still needs to be investigated. I wonder if the InnoDB team @ Oracle is doing something for this in 5.7? Does anyone know? I vaguely recall reading something about it, but I am not sure. It would seem a waste to duplicate their efforts. In any case, I hope this was useful. As part of this investigation, I installed a new 3.14 kernel on the lizard2 machine and a new `perf` installation, which seems to work well to do more detailed investigations of these kind of issues. So let me know if there are other benchmarks that I should look into. One thing that could be interesting is to look for false sharing; there are some performance counters that Intel manuals describe can be used for this. As an aside: In my tests, once concurrency becomes high enough that the server falls over, the actual TPS number becomes mostly meaningless. Eg. I saw putting dummy pause loops into the code increasing TPS. If TPS stabilises at N% of peak throughput as concurrency goes to infinity, then we can compare N. But if N goes to zero as concurrency goes to infinite, I think it is meaningless to compare actual TPS numbers - we should instead focus on removing the fall-over behaviour. (Maybe this is already obvious to you, I have not followed the previous benchmark efforts that closely). Hope this helps, - Kristian.

4 4

Re: [Maria-developers] MDEV-5792 - Deadlock between SELECTs from METADATA_LOCK_INFO and another
by Sergei Golubchik 01 May '14

01 May '14

Hi, Sergey! On Apr 23, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 4155 > revision-id: svoj(a)mariadb.org-20140423073406-o20n7jvwfjinvxxr > parent: psergey(a)askmonty.org-20140418100754-v5b5sdqbavbcznh7 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0-mdev5792 > timestamp: Wed 2014-04-23 11:34:06 +0400 > message: > MDEV-5792 - Deadlock between SELECTs from METADATA_LOCK_INFO and another > I_S table > > mdl_iterate() helper function (which is used by the plugin to iterate mdl > locks) acquired mutexes in reverse order. > > Fixed by iterating MDL locks in two stages: > 1. Iterate locks hash under the protection of hash mutex, store all > lock pointers in a thread local array and increment reference counter > for the lock. > 2. Iterate local array without protection of hash mutex, handle destroyed > locks. > > It somewhat echoes hack in MDL_map_partition::move_from_hash_to_lock_mutex. Looks ok to me. Regards, Sergei

1 0

[Maria-developers] Next steps in improving single-threaded performance
by Kristian Nielsen 01 May '14

01 May '14

I have been analysing CPU bottlenecks in single-threaded sysbench read-only load. I found that icache misses is the main bottleneck, and that profile-guided compiler optimisation (PGO) with GCC gives a large speedup, 25% or more. (More details in my blog posts: http://kristiannielsen.livejournal.com/17676.html http://kristiannielsen.livejournal.com/18168.html ) Now I would like to ask for some discussions/help in how to get this implemented in practice. It involves changing the build process for our binaries: First compile with gcc --coverage, then run some profile workload, then recompile with -fprofile-use. I implemented a simple program to generate some profile load: https://github.com/knielsen/gen_profile_load It runs a bunch of simple insert/select/update/delete, with different combinations of storage engine, binlog format, and client API. It is designed to run inside the build tree and handle starting and stopping the server being tested, so it is pretty close to a working setup. These commands work to generate a binary that is faster due to PGO: mkdir bld cd bld cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" .. make tests/gen_profile_load cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" make So all the pieces really are there, it should be possible to implement it. But we need to find a good way to integrate it into our build system. The best would be to integrate it into our cmake files. The gen_profile_load.c could go into tests/, ideally we would build both a static and dynamically linked version (so we get PGO for both libmysqlclient.a and libmysqlclient.so). Anyone can help me get cmake to do that? And it would be cool if we could get the above procedure to work completely within cmake, so that the user could just do: cmake -DWITH_PGO ... ; make and cmake would itself handle first building with --coverage, then running gen_profile_load.static and gen_profile_load.dynamic, then rebuilding with -fprofile-use. Anyone know if this is possible with cmake, and if so could help implement it? But alternatively, we could integrate a double build, like the commands above, into the buildbot scripts (.deb, .rpm, bintar). Any comments? Here are some more points: - I tested that gen_profile_load gives a good speedup of sysbench read-only (around 30%, so still very significant even though it generates a different and more varied load). - As another test, I removed all SELECT from gen_profile_load, and ran the resulting PGO binary with sysbench read-only. This still gave a fair speedup, despite the PGO load being completely different from the benchmark load. This gives me confidence that the PGO should not cause performance regressions in cases not covered well by gen_profile_load - More tests would be nice, of course. Axel, would you be able to build some binaries following above procedure, and test some different random benchmarks? Anything that is easy to run could be interesting, both to test for improvement, and to check against regressions. - We probably need a recent GCC version to get good results. I used GCC version 4.7.2. Maybe we should install this GCC version in all the VMs we use to build binaries? - Should we do this in 5.5? I think we might want to. The speedup is quite significant, and it seems very safe - no code modifications are involved, only different compiler options. Any thoughts? Volunteeres for helping with the cmake or buildbot parts? - Kristian.

4 14

[Maria-developers] valgrind failure in main.statistics
by Sergey Petrunia 29 Apr '14

29 Apr '14

Hello Igor, Valgrind build in buildbot currently fails in main.statistics test: http://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/5… I have tracked it down: the failure appeared after this push by you: revno: 4157 [merge] committer: Igor Babaev <igor(a)askmonty.org> branch nick: maria-10.0 timestamp: Tue 2014-04-22 14:43:13 -0700 message: Merge. ------------------------------------------------------------ revno: 4152.1.1 committer: Igor Babaev <igor(a)askmonty.org> branch nick: maria-10.0-bugs timestamp: Tue 2014-04-22 14:39:57 -0700 message: Fixed the problem of mdev-5947. Back-ported from the mysql 5.6 code line the patch with the following comment: Fix for Bug#11757108 CHANGE IN EXECUTION PLAN FOR COUNT_DISTINCT_GROUP_ON_KEY CAUSES PEFORMANCE REGRESSION I suppose it would take you much less time than anybody else to fix this? The failure is not repeatable on every machine: it doesn't repeat on my machines, but it can be repeated on work.askmonty.org. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 10.1 planning
by Kristian Nielsen 29 Apr '14

29 Apr '14

Sergei Golubchik <serg(a)mariadb.org> writes: > See below. > What do you think about it? > > On Apr 23, Elena Stepanova wrote: >> >> There is also https://mariadb.atlassian.net/browse/MDEV-4487 >> (Replication from 5.6 with enabled GTID) to be considered. It's >> currently Minor and has 10.0.x as a target version, both of which are >> questionable: recently Colin said on IRC that it should be much more >> than a minor FR, and yet I doubt that somebody will get to it in 10.0. >> Maybe it's more realistic to raise the priority but to re-target it for >> 10.1? I think Monty had some ideas for doing this, IIRC it was discussed at the meeting in Barcelona in January. If what I remember is true, it is actually just that a 5.6 master refuses to replicate if the slave does not say it is configured for MySQL 5.6 GTID. The slave does not need to actually _use_ GTID. So I think Monty's plan was for the MariaDB slave to just lie to the master and say that it's a 5.6 slave with appropriate config. And the MariaDB slave would need to recognise and ignore the new MySQL 5.6 events, such as GTID. - Kristian.

2 2

[Maria-developers] Old autotools cruft in source tree?
by Otto Kekäläinen 28 Apr '14

28 Apr '14

Hello The Debian Lintian QA scanner yields this warning: W outdated-autotools-helper-file (http://lintian.debian.org/tags/outdated-autotools-helper-file.html) extra/jemalloc/config.guess 2012-02-10 extra/jemalloc/config.sub 2012-02-10 storage/tokudb/ft-index/third_party/xz-4.999.9beta/build-aux/config.guess 2009-04-27 storage/tokudb/ft-index/third_party/xz-4.999.9beta/build-aux/config.sub 2009-04-17 Are these still needed in MariaDB or is this simply old cruft from old autotools time that could be removed? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 1

[Maria-developers] "SELECT ... LIMIT 0" is now deterministic
by Alexander Barkov 28 Apr '14

28 Apr '14

Hello Ian, A fix for MDEV-6170 fixed another problem as a side effect: "SELECT ... LIMIT 0" is now deterministic for statement based binary log. Queries with "LIMIT 0" do not return ER_BINLOG_UNSAFE_STATEMENT any more. Can you please mention this in the manual? Some discussion is here: https://mariadb.atlassian.net/browse/MDEV-6170 Related articles in the MySQL manual: https://dev.mysql.com/doc/refman/5.6/en/replication-rbr-safe-unsafe.html https://dev.mysql.com/doc/refman/5.6/en/replication-features-limit.html Thanks.

1 0

[Maria-developers] a Query_log_event charset bug in parallel replication
by nanyi607rao 25 Apr '14

25 Apr '14

hi, If character_set in different Query_log_events changed, worker threads may apply them with wrong character_set. the codes leading this problem is in Query_log_event::do_apply_event, that is: if (charset_inited) { if (rli->cached_charset_compare(charset)) { /* Verify that we support the charsets found in the event. */ if (!(thd->variables.character_set_client= get_charset(uint2korr(charset), MYF(MY_WME))) || !(thd->variables.collation_connection= get_charset(uint2korr(charset+2), MYF(MY_WME))) || !(thd->variables.collation_server= get_charset(uint2korr(charset+4), MYF(MY_WME)))) { There is a charset[6] in rli, which cached last Query_log_event's charset in serial replication. But in parallel replication it would lead mistake, because every worker thread can read and set rli->charset[6], so rli->charset[6] isn't any worker threads' last Query_log_event charset. but rli->charset[6] can affect every worker threads' thd->variables.character_set_* setting. 2014-04-17 nanyi607rao

2 7

[Maria-developers] About the query performance issue reported on IRC (Percona 5.5 vs MariaDB 10.0)
by Elena Stepanova 25 Apr '14

25 Apr '14

Hi all, Over the last couple of days, some of you have seen or even participated in a "thread" on #maria IRC channel regarding the performance problem on query execution that james_woods observed on MariaDB 10.0 instance comparing to a Percona 5.5 instance. This email is for those who are curious about the outcome/current status. Summary ------- The original complaint sounded like this: "After switching to MariaDB 10 I see 5-10 very long running queries - which are not using indices as they do on Percona 5.5." The data in both instances is exactly the same. The HW is essentially different, but it's better on the machine with MariaDB 10. After some back-and-forth, the complaint got split into three main parts: - the query is considerably slower on MariaDB 10.0 instance than on Percona 5.5 instance; - the query plans show very different row counts, although the data is identical; - Percona uses indices when MariaDB doesn't. Sergei Petrunia investigated the alleged issues with query plans and found that there was no problem: - the row counts that Percona 5.5 reported for derived tables were smaller because EXPLAIN in Percona 5.5 had actually executed the query. - the difference in index usage was an effect of table elimination. A table was removed, hence no index use for it was shown in the explain. He also broke down the query and got a simple flat one, without subqueries, which still revealed ~4x difference in execution time. Upon his request, I ran some tests locally to see if I could reproduce such a difference between Percona 5.5 and MariaDB 10.0 and found that I couldn't (of course I don't have the HW that james_woods uses, the but the difference this big was likely to be seen anywhere). Meanwhile, james_woods installed Percona 5.5 on the machine where MariaDB 10.0 was running, and didn't get any performance gain -- that is, the difference was not between Percona 5.5 and MariaDB 10.0, but was caused by something else (HW, environment, ...). As of now, we don't know what it is. (end of summary). Sergei asked me to share the numbers I got while running the local tests. I will also put here the initial queries and explains, for the reference. It's mainly for the record, there is nothing particularly interesting in here; if you stop reading now, you won't lose much. Queries and plans ----------------- The initial offending query: SELECT SUM(visits) FROM ( SELECT IFNULL(ref.probename_unique_visits,'tr_trafficother2_uv') AS probename, SUM(x.value) AS visits, IFNULL(ref.category,'Other') AS category FROM ( SELECT aggregated_data._getSource(source, source2, source3, fb_source, request_ids) AS source, COUNT(DISTINCT u.fbuserid) AS value FROM g9_tracker.app_logins u WHERE u.created_date = '2014-04-22' GROUP BY 1 ) x LEFT JOIN aggregated_data.ref_parameter_mappings ref ON (ref.value_of_ref_parameter = x.source) GROUP BY 1,3 ) y; EXPLAINs (that's where row counts were off; note EXPLAIN execution time): MariaDB 10.0: +------+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 3394453 | | | 2 | DERIVED | <derived3> | ALL | NULL | NULL | NULL | NULL | 3394453 | Using temporary; Using filesort | | 2 | DERIVED | ref | ref | value_of_ref_parameter_UNIQUE | value_of_ref_parameter_UNIQUE | 603 | x.source | 1 | Using where | | 3 | DERIVED | u | ALL | NULL | NULL | NULL | NULL | 3394453 | Using where; Using filesort | +------+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ 4 rows in set (0,03 sec) Percona 5.5: +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 15 | | | 2 | DERIVED | <derived3> | ALL | NULL | NULL | NULL | NULL | 75 | Using temporary; Using filesort | | 2 | DERIVED | ref | ref | value_of_ref_parameter_UNIQUE | value_of_ref_parameter_UNIQUE | 603 | x.source | 1 | | | 3 | DERIVED | u | ALL | NULL | NULL | NULL | NULL | 3669609 | Using filesort | +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ 4 rows in set (1 min 40,20 sec) Example of the indices difference (caused by table elimination): Query: SELECT COUNT(*) FROM ( SELECT aggregated_data._getSource(source, source2, source3, fb_source, request_ids) AS source, COUNT(DISTINCT u.fbuserid) AS value FROM g9_tracker.app_logins u WHERE u.created_date = '2014-04-22' GROUP BY 1 ) x LEFT JOIN aggregated_data.ref_parameter_mappings ref ON (ref.value_of_ref_parameter = x.source); EXPLAINs: MariaDB 10.0: +------+-------------+------------+------+---------------+------+---------+------+---------+----------+-----------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +------+-------------+------------+------+---------------+------+---------+------+---------+----------+-----------------------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 3394453 | 100.00 | | | 2 | DERIVED | u | ALL | NULL | NULL | NULL | NULL | 3394453 | 100.00 | Using where; Using filesort | +------+-------------+------------+------+---------------+------+---------+------+---------+----------+-----------------------------+ 2 rows in set, 1 warning (0,02 sec) Percona 5.5: +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+----------+----------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+----------+----------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 75 | 100.00 | | | 1 | PRIMARY | ref | ref | value_of_ref_parameter_UNIQUE | value_of_ref_parameter_UNIQUE | 603 | x.source | 1 | 100.00 | Using index | | 2 | DERIVED | u | ALL | NULL | NULL | NULL | NULL | 3669609 | 100.00 | Using filesort | +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+----------+----------------+ 3 rows in set, 1 warning (1 min 43,62 sec) Numbers ------- The query that Sergei suggested to run in local tests: SELECT SUM(varchar_non_indexed_column_with_non_empty_different_values='foo') FROM table_with_3_mio_rows I ran tests on Percona 5.5/5.6, MariaDB 5.5/10.0, MySQL 5.5/5.6. Only the relevant results are here, I'll skip the rest. For every run the caches were cleaned, the server was started, and then the queries were executed, 3 times each. Initially servers were run with the default options (not server default, but packaging default, how they are installed on Wheezy). The difference between 5.5 and 5.6/10.0 was astonishing, but it was explained by Performance Schema. It went like this: 5.5 select sum(c2='foo') from t_innodb; 1 row in set (8.17 sec) 1 row in set (0.99 sec) 1 row in set (0.95 sec) select sum(c2='foo') from t_myisam; 1 row in set (3.04 sec) 1 row in set (0.56 sec) 1 row in set (0.55 sec) 5.6 / 10.0 select sum(c2='foo') from t_innodb; 1 row in set (24.24 sec) 1 row in set (21.65 sec) 1 row in set (23.43 sec) select sum(c2='foo') from t_myisam; 1 row in set (18.83 sec) 1 row in set (17.23 sec) 1 row in set (16.05 sec) Pretty much the same picture for MariaDB, Percona and MySQL. It would explain everything, but it turned out that james_woods already had Performance Schema turned off. (Note: we *really* should consider switching off Performance schema, at least in configs provided upon the new package installation). Then I ran another test, with the PS turned off, and otherwise default options (now really default, without any pre-installed config). There was still a difference between 5.5 and 5.6/10.0, although not that big, and for InnoDB only: Percona 5.5 select sum(c2='foo') from t_innodb; 1 row in set (5.97 sec) 1 row in set (0.79 sec) 1 row in set (0.79 sec) select sum(c2='foo') from t_myisam; 1 row in set (3.68 sec) 1 row in set (0.56 sec) 1 row in set (0.56 sec) Percona 5.6 select sum(c2='foo') from t_innodb; 1 row in set (7.33 sec) 1 row in set (2.32 sec) 1 row in set (1.48 sec) select sum(c2='foo') from t_myisam; 1 row in set (2.90 sec) 1 row in set (0.59 sec) 1 row in set (0.59 sec) MariaDB 5.5 select sum(c2='foo') from t_innodb; 1 row in set (8.17 sec) 1 row in set (0.99 sec) 1 row in set (0.95 sec) select sum(c2='foo') from t_myisam; 1 row in set (3.04 sec) 1 row in set (0.56 sec) 1 row in set (0.55 sec) MariaDB 10.0 select sum(c2='foo') from t_innodb; 1 row in set (7.34 sec) 1 row in set (2.07 sec) 1 row in set (2.07 sec) select sum(c2='foo') from t_myisam; 1 row in set (3.49 sec) 1 row in set (0.58 sec) 1 row in set (0.57 sec) MySQL 5.6 select sum(c2='foo') from t_innodb; 1 row in set (7.88 sec) 1 row in set (1.95 sec) 1 row in set (2.11 sec) select sum(c2='foo') from t_myisam; 1 row in set (4.11 sec) 1 row in set (0.55 sec) 1 row in set (0.57 sec) So, on 5.6/10.0 with default options the query from the "warm" InnoDB table was approximately 2x slower than on 5.5. Then I took non-default options from james_woods and applied them, as much as I could, considering the HW difference, to my modest VM. In fact, only innodb_buffer_pool_size mattered. After doubling it on 5.6/10.0, I got pretty much the same results as on 5.5. Percona 5.5 (default options) select sum(c2='foo') from t1; 1 row in set (5.97 sec) 1 row in set (0.79 sec) 1 row in set (0.79 sec) Percona 5.6 (default options + no PS + increased buffer pool) select sum(c2='foo') from t1; 1 row in set (8.33 sec) 1 row in set (0.83 sec) 1 row in set (0.82 sec) MariaDB 5.5 (default options) select sum(c2='foo') from t1; 1 row in set (8.17 sec) 1 row in set (0.99 sec) 1 row in set (0.95 sec) MariaDB 10.0 (default options + no PS + increased buffer pool) select sum(c2='foo') from t1; 1 row in set (7.85 sec) 1 row in set (0.86 sec) 1 row in set (0.86 sec) There are still some minor differences, but my VM is not suitable for fine-grained performance testing. If somebody wants to investigate it further, it should be done in a proper way; in any case I'm sure it can be tuned and made to perform better on 5.6/10.0, nobody who cares about differences this small actually runs the server with default options. Regards, /E

1 0

[Maria-developers] Difference between 0xHHHH and X'HHHH' hex literals
by Alexander Barkov 24 Apr '14

24 Apr '14

Hello Ian, I tried to explain in https://mariadb.atlassian.net/browse/MDEV-6092 to the complainer why it's not a bug. Can you please document the difference between the two hex notations and then close as "not a bug"? Thanks.

1 0

Re: [Maria-developers] bb01.mariadb.net
by Kristian Nielsen 23 Apr '14

23 Apr '14

Daniel Bartholomew <dbart(a)mariadb.com> writes: > Serg and Knielsen: I've got the box mostly configured I think, but > could be (very) wrong, so I wanted to give you a chance to make sure > everything looks good on it before I go ahead and add it to the It looks like it already mostly works. I noticed an error in a build though: https://buildbot.askmonty.org/buildbot/builders/kvm-tarbake-jaunty-x86/buil… The "kernel" log has this: ----------------------------------------------------------------------- /dev/sda1 contains a file system with errors, check forced. /dev/sda1: Entry 'pack-names' in /home/buildbot/.bzr/repository (3415) has an incorrect filetype (was 1, should be 2). /dev/sda1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. (i.e., without -a or -p options) ----------------------------------------------------------------------- One likely reason is a failed rsync step. This is the tarbake build; unlike most other virtual machines, this one is actually modified during a build. So it's quite possible that the rsync would go wrong if a tarbake build happened to run in the meantime. So just re-sync that particular VM, and that might solve the problem. Thanks for setting this up! - Kristian.

2 1

[Maria-developers] [GSoC] Accepted student ready to work : )
by Pablo Estrada 23 Apr '14

23 Apr '14

Hello Sergei, Elena and everyone, I'm Pablo, and I was selected as part of Google Summer of Code 2014 to work on the project "*Statistically optimize mysql-test runs by running less tests*". First of all, thank you very much for having selected me for the project. I am very excited to start reading, reviewing data and coding up. I see that Elena will be my mentor. Nice to meet you Elena : ) To begin with, I am focusing on the *Buildbot documentation*, to *(1)understand how the data is organized* inside it, and to *(2)*figure out *how to add the extra functionality* into the BuildBot. I want to get started with the data analysis, so it would be great if you could help me with the following things: 1. Sergei, you said you did a bit of research into this. Do you have your *previous results*, or *any papers* that you have read that you could suggest to me? 2. Can I get *access to the buildbot database* to look at the data? If not, can I get the E/R model, or any scheme of *how the data looks like?* - I understand that maybe the data is not just a single database. I am starting by *reading the buildbot documentation*, to understand how the stuff is organized in it. 3. *Any other information* that you consider useful for me to know? Any advice? And that's all I have for now. I'm looking forward to getting started! Regards Pablo

4 5

[Maria-developers] mdev6027 RLIKE: "." no longer matching new line (default_regex_flags)
by Alexander Barkov 23 Apr '14

23 Apr '14

Hello Serg, Please review a patch implementing a new system variable default_regex_flags, to address the remaining incompatibilities between PCRE and the old regex library. Greetings.

3 5

[Maria-developers] Proposal Acceptance - GSOC
by Puneet Dewan 22 Apr '14

22 Apr '14

Hi Everyone, I, Puneet Dewan have got accepted into Google Summer Of Code 2014. This is the first and the last time I applied for GSOC and getting accepted is *really motivating*. :) My Project is : *Support of UseServerPrepStmts to MariaDB Java Connector.* *Massimo* is my mentor. I will start working on the project after 25th April as my semester exams are going on. I am excited to be part of MariaDB and looking forward to contribute to the community. Regards Puneet.

1 0

[Maria-developers] [GSOC] Proposal accepted!
by sriram patil 22 Apr '14

22 Apr '14

Hi Sergei, Alexander and all, Thank you for accepting my proposal. This was the first time I applied for GSoC, so I was kind of nervous. I was selected to work on the project *"CREATE OR REPLACE, CREATE IF NOT EXISTS and DROP IF EXISTS" *as part of GSoC'14. My end semester exams are going on right now. I will be done with my exams and projects by 29th April. I am totally excited to get started. I have built the source code already. Have learnt a few things about the source code as you must have observed in my proposal. I will continue with the documentation and source code from 1st May. Is there anything else you want me to look into? Thanks, Sriram

1 0

Re: [Maria-developers] [Commits] Rev 4153: MDEV-5963: InnoDB: Assertion failure in file row0sel.cc line 2503 ... in file:///home/psergey/dev2/10.0-cp/
by Jan Lindström 18 Apr '14

18 Apr '14

Sergey, > === modified file 'sql/opt_range.cc' > --- a/sql/opt_range.cc 2014-04-01 16:59:51 +0000 > +++ b/sql/opt_range.cc 2014-04-18 10:07:54 +0000 > @@ -2181,7 +2181,7 @@ int QUICK_ROR_INTERSECT_SELECT::init_ror > quick->record= head->record[0]; > } > > - if (need_to_fetch_row && head->file->ha_rnd_init_with_error(1)) > + if (need_to_fetch_row && head->file->ha_rnd_init_with_error(false)) You had so nice and informative reason why this has to be like this, why not add big fat comment here ? > { > DBUG_PRINT("error", ("ROR index_merge rnd_init call failed")); > DBUG_RETURN(1); > @@ -2363,8 +2363,13 @@ int QUICK_ROR_UNION_SELECT::reset() > quick->save_last_pos(); > queue_insert(&queue, (uchar*)quick); > } > - > - if ((error= head->file->ha_rnd_init(1))) > + /* Prepare for ha_rnd_pos calls. */ > + if (head->file->inited && (error= head->file->ha_rnd_end())) > + { > + DBUG_PRINT("error", ("ROR index_merge rnd_end call failed")); > + DBUG_RETURN(error); > + } > + if ((error= head->file->ha_rnd_init(false))) > { > DBUG_PRINT("error", ("ROR index_merge rnd_init call failed")); > DBUG_RETURN(error); > > _______________________________________________ > R: -- -- Jan Lindström, Principal Engineer SkySQL - The MariaDB Company MariaDB | MaxScale | skype: jan_p_lindstrom www.skysql.com <http://www.skysql.com/> Twitter <http://twitter.com/skysql> Blog <http://www.skysql.com/blog/> Facebook <http://www.facebook.com/skysql> LinkedIn <http://www.linkedin.com/company/1214250> Google+ <https://plus.google.com/117544963211695643458/posts>

1 0

[Maria-developers] TokuDB disabled in Debian and Ubuntu - help me get it working?
by Otto Kekäläinen 18 Apr '14

18 Apr '14

Hello Rick, Last year I spent a lot of time packaging MariaDB 5.5 for Debian and finally this year it has landed in Ubuntu 14.04 and Debian testing. Unfortunately the Debian/Ubuntu version does not include TokuDB and I need your help to get it there. In 5.5.35 (I think) the TokuDB plugn was added to MariaDB but I had issues getting it build 100% correctly and I eventually dropped it (added build parameter -DWITHOUT_TOKUDB=true), as getting MariaDB in Debian at all was a bigger priority than getting it there with every possible plugin. The root cause seems to be that when Debian and Ubuntu packages are built in chroot environments (the build systems of Debian and Ubuntu use pbuilder/sbuilder systems, see https://en.wikipedia.org/wiki/Debian_build_toolchain#Isolated_build_environ…) the code that builds the plugin does not seem to correctly detect the CPU features. It seems to read the values from the build machine and not the inputted target values (in a cross-compile situation). There are two related issues that needs a solution: 1) Currenlty the code that checks what the architecture is (32-bit/64-bit) is the first lines of https://bazaar.launchpad.net/~maria-captains/maria/10.0/view/head:/storage/…. This works well for real and virtual machhines, but it does not seem to work in the pbuilder/sbuilder chroots, as CMAKE_SYSTEM_PROCESSOR always shows the chroot host CPU, not the cross-compile target CPU. Could you please investigate pbuilder/sbuilder and search for some solution that works for reliable target CPU checking? 2) When building TokuDB in Ubuntu (amd64) sbuilder environments something in crashes in the 'toku_os_get_processor_frequency' function. For this too, could you investigate the sbuilde chroot environment and figure out what goes on and how to fix it? Issue 2 has a bug report with the (a bit messy) debugging history documented: https://mariadb.atlassian.net/browse/MDEV-5618 Both of these issues requires learning a bit about sbuilder CPU things, so I assume it is most efficient if the same persons looks into both of these. Thanks! -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

3 5

[Maria-developers] Ubuntu 14.04 and MariaDB
by Otto Kekäläinen 17 Apr '14

17 Apr '14

Ubuntu 14.04 was released today. >From https://wiki.ubuntu.com/TrustyTahr/ReleaseNotes#Ubuntu_Server "MySQL Ubuntu 14.04 LTS continues to provide official support for MySQL 5.5. Three other community supported alternatives of MySQL are also included: * MariaDB 5.5 - a drop in replacement for MySQL 5.5. * Percona XtraDB Cluster 5.5 - a Galera based Active/Active MySQL solution. * MySQL 5.6 - the next release of Oracle's MySQL. Note that upgrading to MySQL 5.6 is an automatic one way process; it is possible to downgrade manually - see the upstream documentation on details of how to perform this process." I have been working on 5.5.37 today, I hope to soon get it uploaded into Debian and then via security update process into Ubuntu too. -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

1 0

Re: [Maria-developers] [GSoC] Student interested in "Optimize mysql-test runs" or adding SQL features
by Sergei Golubchik 17 Apr '14

17 Apr '14

Hi, Pablo! First: please always cc: maria-developers(a)lists.launchpad.net, don't send emails to me only. Now: On Mar 11, Pablo Estrada wrote: > Hello! > I am getting ready to submit my proposal. Is there any feedback for my > application document? Anything that would be good to add, remove, rephrase, > etc? Go on and submit it. There will be a comment form under your proposal, if I will want you to change something, I will leave my comments there. Please mention what tools/languages you think you'll be using. > Also, if I could see the data available (the history of test results), I > could write more in detail : ) Hm. Okay. Attached is the preprocessed table that I used two years ago. It lists test failures in buildbot in a given period of time. Columns are (as far as I remember): * time of the test run start * code branch that was tested * revision id that was tested - it's fake, our buildbot tables don't store revision id. I've constructed it from the set of changed files, revision number, etc. hopefully it's as unique as a real revision id * build number * platform where the tests were run * test type - basically, the set of tests to run and what protocol to use. one test may be run with different protocols * test name * test variant - or combination. certain tests may be run in many combinations Using tis file I tried to rearrange test execution order for every specific build number (using failure data from previous builds in the same table) to detect test failures as early as possible. I managed to have 90% of all test failures happening within first 10% of the tests. The goal of this project is to have something we can put back into buildbot, so that it could actually run only 10% of the tests, still having 90% probability of discovering a failure (or 20%, or 95%, whaveter). See also http://buildbot.askmonty.org/buildbot/ you can click on a link for any branch, for example, http://buildbot.askmonty.org/buildbot/grid?branch=10.0 then on any particular build, for example, http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297 and on a stdio log for a particular test run: http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297/ste… on these pages you'll see what a platform (builder) is, build number, test types, combinations, etc. Regards, Sergei

2 2

[Maria-developers] Problems with Launchpad?
by Andrew McDonnell 17 Apr '14

17 Apr '14

Hi, my mariab bzr branch on launchpad doesnt update the summary page properly, and hasn't done so for a while now, which means recent work I committed is not visible unless you drill right down into the branch change history (My related question on Launchpad Answers - https://answers.launchpad.net/launchpad/+question/246850 - hasn't been touched either) Is anyone else experiencing this? cheers, Andrew -- http://blog.oldcomputerjunk.net https://launchpad.net/~andymc73 https://github.com/andymc73 Twitter: @pastcompute GPG: http://www.andrewmcdonnell.net/gpg.html

2 1

[Maria-developers] MariaDB 5.5.37 release prep
by Daniel Bartholomew 17 Apr '14

17 Apr '14

All, I've begun release prep for MariaDB 5.5.37. Draft changelog and release notes are at: - https://mariadb.com/kb/en/mariadb-5537-release-notes/ - https://mariadb.com/kb/en/mariadb-5537-changelog/ Please let me know ASAP of any notable things that should be mentioned in the release notes. Thanks! -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

3 3

[Maria-developers] MariaDB scalability: mutexes
by Sergey Vojtovich 16 Apr '14

16 Apr '14

Hi Axel, I benchmarked various mutexes available to MariaDB. Results are here: http://svoj-db.blogspot.ru/2014/02/mariadb-mutexes-scalability.html Among other things I noticed that normal mutexes scale better than adaptive. And it looks like it gives over 5 000 TPS in my benchmarks: Original -------- 64 threads, time spent: 60s, queries executed: 11500723, qps: 191678, 1 thread qps: 2994, tps: 13691 64 threads, time spent: 60s, queries executed: 10983665, qps: 183061, 1 thread qps: 2860, tps: 13075 64 threads, time spent: 60s, queries executed: 11080325, qps: 184672, 1 thread qps: 2885, tps: 13190 slow LOCK_open -------------- 64 threads, time spent: 60s, queries executed: 12948365, qps: 215806, 1 thread qps: 3371, tps: 15414 64 threads, time spent: 60s, queries executed: 13191584, qps: 219859, 1 thread qps: 3435, tps: 15704 64 threads, time spent: 60s, queries executed: 13050647, qps: 217510, 1 thread qps: 3398, tps: 15536 slow LOCK_open + THR_LOCK::mutex -------------------------------- 64 threads, time spent: 60s, queries executed: 15863174, qps: 264386, 1 thread qps: 4131, tps: 18884 64 threads, time spent: 60s, queries executed: 15761944, qps: 262699, 1 thread qps: 4104, tps: 18764 64 threads, time spent: 60s, queries executed: 15773239, qps: 262887, 1 thread qps: 4107, tps: 18777 Could you do official benchmarks (patch attached)? It would be nice to also benchmark MySQL-5.6 with thr_lock part of the patch. Thanks, Sergey

3 8

[Maria-developers] Error on 'make' of mariadb source
by Kesha Shah 14 Apr '14

14 Apr '14

Hi all, I did some changes with in the sql_yacc.yy. During building the source , I am getting following error (attached screenshot) when I try to execute the "make" command after doing a cmake of the source. It says permission denied while trying to make jemalloc-configure. Although i am executing the make command with root user. [image: Inline image 1] [image: Inline image 2] Can somebody provide/help me with some pointers on how to proceed to resolve it ? -- Thanks and Regards, Kesha Shah. www.keshashah.com

2 3

[Maria-developers] MariaDB release and support policies
by Otto Kekäläinen 14 Apr '14

14 Apr '14

Hello! Does MariaDB promise a 8 year release cycle for 5.5 as MySQL does? Does all of the points listed at https://lists.ubuntu.com/archives/technical-board/2014-February/001812.html apply for MariaDB as well? Can we promise Ubuntu that all MariaDB minor releases fulfill the quality requirements listed above? If so, I'll then file a micro release update permission request for Ubuntu. AFAIK as I know it is the only way to get security releases to users anyway. PS. Do you people know about semver.org and try to follow the general versioning policy? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 2

[Maria-developers] Third bunch of unmerged items
by Sergey Vojtovich 14 Apr '14

14 Apr '14

Hi Sergei, here comes third bunch of items which I'm unsure about: WL#5706/Bug#58712/Bug#11746378 - Encrypt or remove passwords from slow, query, and binary logs WL#4677 - Unique Server Ids for Replication Topology (UUIDs) WL#6004 - Server extension for allowing user tables to be put into transactional SE Bug #14112682 : MYSQLTEST: PROVIDE ACCESS TO ERROR NAMES THROUGH NUMERIC CODES AND VICE VERSA Bug #11747313 31741: MYSQLTEST - DEPRECATE "SYSTEM" COMMAND WL#2739: Auditing Password Security (plugin + API) WL#6587: disconnect_on_expired_password WL#5825 Using C++ Standard Library with MySQL code WL#2111: #6 Set CLASS_ORIGIN and SUBCLASS_ORIGIN Regards, Sergey

2 1

Re: [Maria-developers] MariaDB 10.x GTID
by Kristian Nielsen 11 Apr '14

11 Apr '14

Jan Lindström <jplindst(a)mariadb.org> writes: > some background, lets assume we have two galera clusters that are > geographically distributed (lets say for example that cluster A is on Europe > and cluster B on USA). Inside a cluster all nodes are set up so that server_id > and domain_id are the same and we will use strict GTID mode (for a example on > cluster A server_id = 1 and domain_id =1 and in cluster B server_id = 2 and > domain_id = 2). In cluster we naturally use Galera replication. Additionally, > we have asynchronous replication (MySQL replication) between these clusters. > 1) We would like to have some special domain_id (i.e. GTID) for transactions > that are executed only on certain cluster e.g. on cluster A. This would mean > that there is some dynamic option to enable "local" transaction execution mode, > where transactions are executed on cluster A but not on cluster B. I know that > this kind of feature does not exists but is it even possible ? Basically, this > would mean for MySQL replication point of view a fact that if GTID is formed > like this special GTID, they are not replicated to slave, but they are written > to binlog. It sounds like what you want is some option on the slave that would cause it to ignore all GTIDs with some specific domain id, similar to how it is currently possible to filter on server id. That seems a reasonable request; such option does not exist currently though. Another option is to use the @@skip_replication variable. Set this for the sessions doing the "local" transactions. Then set replicate_events_marked_for_skip to either FILTER_ON_SLAVE or FILTER_ON_MASTER, as you like. This is specifically designed to allow to log certain transactions to the binlog but not replicate them, so it sounds like it is what you asked for. > 2) There will be some challenge in maintaining the sequence number for GTID > correctly: for global GTID purposes, we can just count commits, and skip every > local transaction commit, but, we must also initialize the sequence for every > node joining the cluster. In MySQL-5.6 this happens by copying binlog files > (only header part needed) to joining node. I wonder if same strategy works for > MGC-10, is MariaDB GTID is stored in binlog header ? Yes, the GTID is stored in the binlog header. It is also stored in a file master-bin.state. This file is read at server start to initialise the GTID sequence numbers. However, if the file is not found for any reason, the information is extracted from the last binlog file (the master-bin.state is just a cache). However, you need the whole binlog file (if master-bin.state is missing), not just the header. I am sure it must be the same for MySQL-5.6 - you need to scan to the end of the binlog file, as there may be extra GTIDs logged in the file that were not there when the header was written. Of course, just after server start, the most recent binlog file is empty, so just the header works. It sounds like you would want to just copy over the master-bin.state file. However, this whole approach sounds wrong to me. The current GTID state is a cluster-wide property, not a local property, I do not understand why you want to maintain it locally. As I understand it, Galera decides on commit order. The certification process decides for parallel transactions T1 and T2 which one will be committed before the other. The GTID sequence has to match the commit order. So the GTIDs need to be generated by Galera, at the point in the code where the commit order is decided. This Galera code then also needs to do the binlog writing, so that binlog order matches GTID order and commit order. With Galera handling GTID number allocations, I do not understand why you would need to worry about binlog headers. But maybe I have misunderstood, I do not know Galera in details. Maybe you could explain how Galera ensures matching commit order, binlog order, and GTID order? Or alternatively, explain how things work without such ensurance? > 3) Independent table (in data dictionary) or file for replication filters. I > can change the replication filters dynamically, but if the server crash and i > don?t report to my.cnf it?s lost. Therefore, there is a need to store > replication filters persistently and these should be loaded automatically when > server is restarted. Currently we do not have this, replication filters are configuration options, and those are stored in the config file, not in dynamic tables or files. The server_id filters are set by CHANGE MASTER though, IIRC, so they are stored in master.info / relay_log.info. > 4) Dynamic binlog. Currently options are read-only. This we do not have. There has been talk of allowing it under global read lock; this might work, but it would require some work to carefully check all of the code for possible issues. There are a lot of places in the code where expensive synchronisation/locking is avoided based on the knowledge that the binlog can not be enabled or disabled on-the-fly. Hope this helps, - Kristian.

1 0

[Maria-developers] debugging MDEV-5932, wrong result set for tokudb
by Rich Prohaska 10 Apr '14

10 Apr '14

Hello, TokuDB generates an empty result set for a select with sub-select, but generated a non-empty result set for the same query when creating a new table. The result set should be the same in both cases. What is going on here? InnoDB computes an empty result set for both cases. I traced the problem to the tokudb::index_read function. The tokudb::index_read does not handle the null byte key prefix. Unfortunately, I do not understand how the null byte (byte 0) in the key handed to the ::index_read function is supposed to work. Most of the time, the value of this byte is 0. Sometimes it is set to 1. How should the storage engine work WRT this? BTW, this problem exists with TokuDB in MySQL 5.5.36, MySQL 5.6.16. I have not been able to consistently see this on MariaDB 5.5.36 for some reason. In any case, I am interested in how the handler's index_read function is supposed to process the null byte. Thanks

2 1

[Maria-developers] Buildbot machines
by Rasmus Johansson 09 Apr '14

09 Apr '14

Hi, I discussed both with Dbart and Serg last week about the situation with buildbot. Kristian and Elena also had similar discussion on IRC today. To sum it up, there seems to be two aspects to it; 1) more machines, i.e. slaves could/should be added immediately and 2) some development work could/should be done on buildbot to support some other ways of building, like automatically spinning up and down platform specific virtual machines in a public cloud somewhere whenever they are needed. I looked into alternatives for the 1) immediate need of more machines and thought. I found a pretty interesting alternative from OVH.COM, who offers dedicated servers to a very reasonable price. Here is a good example of what they can offer: https://www.ovh.com/us/dedicated-servers/infra/2014-EG-32.xml . I would be prepared to give it a try. Do you think the specs are good enough to help with the load buildbot currently has? BR, Rasmus -- Rasmus Johansson, VP Engineering MariaDB | t: +358 50 499 9589 | Skype: ratzpo

3 5

Re: [Maria-developers] [Commits] Rev 3991: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 09 Apr '14

09 Apr '14

Hi, Alexey! On Apr 02, Alexey Botchkov wrote: > > Still, another question, why did you have to rename net_store_data() to > > net_store_data_cs() ? > > That's personal. > I really don't like functions with exactly same names - makes code harder to > understand and debug. > Do you belive it's better not to rename these here? That's up to you. Sometimes I create functions with the same name, sometimes I don't. For example, tdc_acquire_share() - there are few functions with the same name, because they do exactly the same. Actually, one of tdc_acquire_share() is a real function, others are inline convenience wrappers - they eventually call the "real" one. But tdc_acquire_share_shortlived() - while also only a convenience wrapper - has a different name to emphasize that it has different semantics. But anyway, that's your patch. If you'd like to rename - go ahead. Regards, Sergei

1 0

[Maria-developers] INT thread variables in mysql and mariadb (all versions as far as i know)
by Rich Prohaska 09 Apr '14

09 Apr '14

suppose that I have a THDVAR_INT variable with min = -1 and max = 1, and i set it to -1. how come select @@var returns 4294967295 instead of -1?

2 1

[Maria-developers] Error in "make install"
by sriram patil 09 Apr '14

09 Apr '14

Hello all, I have had built and installed the mariadb source code before, no hassles whatsoever. I was trying to install it again but getting the following error when trying to "make install" ....... -- Installing: /usr/local/mysql/mysql-test/././var/tmp -- Installing: /usr/local/mysql/mysql-test/././var/tmp/mysqld.2 -- Installing: /usr/local/mysql/mysql-test/././var/tmp/mysqld.1.sock CMake Error at mysql-test/cmake_install.cmake:36 (FILE): file INSTALL cannot copy file "/home/sriram/Downloads/mariadb/10.0/mysql-test/./var/tmp/mysqld.1.sock" to "/usr/local/mysql/mysql-test/././var/tmp/mysqld.1.sock". Call Stack (most recent call first): cmake_install.cmake:119 (INCLUDE) make: *** [install] Error 1 I have changed the owner and group of the folder "/usr/local/mysql", so root access is not an issue. I am using Ubuntu 12.04. Did anyone get such error while installing and was able to solve it? Thanks, Sriram

2 3

[Maria-developers] Vision for next 10.x minor releases
by Honza Horak 08 Apr '14

08 Apr '14

Hi guys, first, congratulations for the 10.0 release, well done! Even though it can look insane, I'm already wondering about next minor releases ;) Especially if you have already some idea about 10.1 and later. Specifically, I'm wondering if these releases will still be somehow tight to MySQL's release cycle (say 10.1 will be somehow following 5.7) or if you plan totally independent release schedule like slower/faster the release cadence? In other words, since MySQL changed only minor version and MariaDB bumped the major (we all know the reason), I'm wondering if something changed in thinking about major/minor versioning -- like if the difference between 10.0 and 10.1 will still be considered the same like between say 5.3 and 5.5. I don't expect you can give us any dates, but some expectations, if 10.1 can be expected in months or rather years? If that was already discussed somewhere, please, just point me to the correct place. Thanks, Honza

2 2

Re: [Maria-developers] Rev 4077: MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client
by Sergei Golubchik 07 Apr '14

07 Apr '14

Hi, Holyfoot! On Feb 22, holyfoot(a)askmonty.org wrote: > revno: 4077 > revision-id: holyfoot(a)askmonty.org-20140222173543-xkzlp5rpga41evur > parent: svoj(a)mariadb.org-20140218065405-vw3bfhvhrfjsc6hh > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-4856 > timestamp: Sat 2014-02-22 21:35:43 +0400 > message: > MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client. > The fill_schema_table() function used to call get_table_share() > for a table name in WHERE then clear the error list. That way > plugins receive the superfluous error notification if it happens > in it. Also the problem was that error handler didn't prevent > the suppressed error message from logging anyway as the logging > happens in THD::raise_condition before the handler call. > Fixed by adding the Warnings_only_error_handler before the call. > raise_condition() also fixed. The Internal_error_handler class > was exteded with virtual functions to handle storing the error > message. This looks too complex to me. Why would you extend the base Internal_error_handler class - no other error handler did that or needed that, fwiw. I'd simply push an error handler, store the first message there and access it in get_schema_tables_record() via THD::get_internal_handler(). Regards, Sergei

2 4

[Maria-developers] md5sum mariadb-10.0.10.tar.gz
by Jakub QB Dorňák 07 Apr '14

07 Apr '14

Hi guys, I am getting mariadb-10.0.10.tar.gz with md5 checksum of d41d8cd98f00b204e9800998ecf8427e, while it should be 14ce22b8197d4eae88d237776d47220f. Thank You for any help, QB

2 2

[Maria-developers] Getting started
by sriram patil 04 Apr '14

04 Apr '14

Hi Sergei, I have got myself familiarized with parsing and execution source code. I wanted to start working on the source code a little bit. But I could not find any open bugs on MariaDB launchpad. It would really help if you could give me a few open bugs or enhancements to start with. Thanks, Sriram

2 1

[Maria-developers] [GSOC 2014] CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS
by Kesha Shah 03 Apr '14

03 Apr '14

Dear Sergei Golubchik, Hope this mail finds you well. I am Kesha Shah, a third year undergraduate from DA-IICT, Gandhinagar, India. I was a Google Summer of Code student previous year as well as Google Code-In mentor with BRL-CAD open source solid modeling software. The activities related to my contributions at BRL-CAD at logged by me at - http://brlcad.org/wiki/User:KeshaSShah/GSoC13/Reports. My other projects can be found at - https://github.com/keshashah?tab=repositories. I have good programming hands on C and C++ and would like to take up and contribute at the Mariadb task of 'CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS' [MDEV-5359] as part of GSOC 2014. I have studied database systems and compiler construction during my academics. Your project would help to apply my knowledge by working at deeper level of coding database internals, thereby contributing to open source software. I had checked out the mariadb master branch from bazaar and started looking into the source code. I found the task more related to update of the existing mariadb SQL parser. The parser seems to be implemented as ascendant parser using bison parser generator (GNU version of YACC). From the current findings I could see the YACC grammar file sql_yacc.yy under maria/sql/ where I can see the rules defined for CREATE PROCEDURE_SYM, CREATE FUNCTION_SYM, CREATE TRIGGER_SYM etc. Alternatively I can see the *_SYM lexical tokens defined under the /maria/sql/lex.h file. Similarly for DROP TABLESPACE_SYM, DROP TRIGGER_SYM etc could be found. So when sql text input stream is encountered from SQL interface, it is read and categorized into lexical tokens by the lexer /maria/sql/sql_lex.cc in the lexical analysis step. I tried to figure out that we will have to define additional token 'IF' in the lex.h and 'NOT' and 'EXISTS' are already defined. I am also looking at the existing user defined actions that builds the AST nodes (sp_head being the root node for most types) when relevant grammar rule is satisfied in case of DDL statements. After semantic analysis (create of stored procs/functions), the next step would be to amend/define actions for the newly introduced grammar of handling check of existence of db artefact at the time of DDL execution. Also at this step the compiler will have to query the relevant storage engine on DB level or repository level to check if the DB object exists. This would require me to spend more time on understanding design pattern followed for the implementation of classes and behaviours of each of the underlying db artefacts. It would be interesting for me to know if the steps proposed by me above are in the right direction or not ? Also I am aware that I am late in discussing my proposal with you. But I can assure that I will be able to deliver the requirements in timely manner under your guidance. I have formulated initial draft of my proposal and would like to submit it at the earliest. Looking forward to you reply. Have a nice day! Thanks and Regards, Kesha Shah

2 2

[Maria-developers] new histogram code in 10.0.10
by Axel Schwenke 02 Apr '14

02 Apr '14

Hi Sergei, attached the results from a DBT3/SF30 run vs. 10.0.9 (as released) and 10.0-bzr rev 4128. Both runs use EITS + histograms. I see different EXPLAIN plans for queries 2, 3, 5, 7, 8, 18 and 21. Execution times only for few queries: 10.0.9 rev4128 Q2 44,44,46 44,45,45 Q3 298,298,299 236,243,243 Q5 264,302,303 293,308,308 Q7 199,200,201 204,205,206 Q8 375,375,377 264,274,276 Q18 217,218,221 178,180,182 Q21 251,259,261 258,261,262 Details in attached zip file. XL

1 0

[Maria-developers] PIC build options and MariaDB
by Otto Kekäläinen 02 Apr '14

02 Apr '14

Hello, I noticed that the MySQL packages in Debian have PIC stuff in their alternative build arguments, see http://anonscm.debian.org/gitweb/?p=pkg-mysql/mysql-5.6.git;a=blob;f=debian… http://anonscm.debian.org/gitweb/?p=pkg-mysql/mysql-5.5.git;a=blob;f=debian… Anybody here who understands deeply the build arguments who could say if having these PIC additions is desirable? Also the MySQL rules file has a section like this: # See http://blogs.innodb.com/wp/2010/04/innodb-performance-aio-linux/ 67 # Also #659565 68 ifneq (,$(findstring linux-gnu,$(DEB_HOST_GNU_TYPE))) 69 USE_LINUX_NATIVE_AIO:=-DLINUX_NATIVE_AIO=ON 70 endif Should MariaDB use this too, is it valid for XtraDB? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 1

[Maria-developers] Cassandra plugin build error: _S_empty_rep_storage' can not be used when making a shared object; recompile with -fPIC
by Otto Kekäläinen 02 Apr '14

02 Apr '14

Hello! I am testing out building MariaDB 10.0.10 on a Ubuntu 12.04 system with libthrift-dev installed (latest 0.9.0 from repo 'deb http://ppa.launchpad.net/wnoronha/thrift/ubuntu precise main'). Building fails with this error: make[4]: Siirrytään hakemistoon "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" Scanning dependencies of target cassandra make[4]: Poistutaan hakemistosta "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" make[4]: Siirrytään hakemistoon "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/ha_cassandra.cc.o [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/cassandra_se.cc.o [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/Cassandra.cpp.o [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/cassandra_types.cpp.o [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/cassandra_constants.cpp.o Linking CXX shared module ha_cassandra.so /usr/bin/ld.bfd.real: /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libthrift.a(TApplicationException.o): relocation R_X86_64_32S against `std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_empty_rep_storage' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libthrift.a: could not read symbols: Bad value collect2: ld:n paluuarvo oli 1 make[4]: *** [storage/cassandra/ha_cassandra.so] Virhe 1 Ideas? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

3 3

Re: [Maria-developers] Cassandra plugin build error: _S_empty_rep_storage' can not be used when making a shared object; recompile with -fPIC
by Otto Kekäläinen 01 Apr '14

01 Apr '14

Hello! Thanks for a quick reply! So I should add -fPIC to cmake arguments? Why isn't it in MariaDB sources by default, what are the downsides? GCC manual said "This option makes a difference on the m68k, PowerPC and SPARC." -> I am running amd64 For background, in my Debian packaging the build command looks like this: sh -c 'PATH=${MYSQL_BUILD_PATH:-"/bin:/usr/bin"} \ CC=${MYSQL_BUILD_CC:-gcc} \ CFLAGS=${MYSQL_BUILD_CFLAGS:-"-O2 -DBIG_JOINS=1 -fno-strict-aliasing "} \ CXX=${MYSQL_BUILD_CXX:-g++} \ CXXFLAGS=${MYSQL_BUILD_CXXFLAGS:-"-O3 -DBIG_JOINS=1 -felide-constructors -fno-exceptions -fno-rtti -fno-strict-aliasing "} \ cmake -DCMAKE_INSTALL_PREFIX=/usr \ \ -DCOMPILATION_COMMENT="(Ubuntu)" \ -DMYSQL_SERVER_SUFFIX="-1" \ -DSYSTEM_TYPE="debian-linux-gnu" \ -DWITHOUT_TOKUDB=true \ -DBUILD_CONFIG=mysql_release \ -DINSTALL_LIBDIR=lib/x86_64-linux-gnu \ -DINSTALL_PLUGINDIR=lib/mysql/plugin \ -DINSTALL_SUPPORTFILESDIR=share/mysql \ -DDEB=1 ..' In upstream MariaDB the build command is: sh -c 'PATH=$${MYSQL_BUILD_PATH:-"/bin:/usr/bin:/usr/local/bin"} \ CC=$${MYSQL_BUILD_CC:-gcc} \ CFLAGS=$${MYSQL_BUILD_CFLAGS:-"-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized $$(case `lsb_release -sc` in (lenny) echo -DWORKAROUND_GCC_4_3_2_BUG ;; esac)"} \ CXX=$${MYSQL_BUILD_CXX:-g++} \ CXXFLAGS=$${MYSQL_BUILD_CXXFLAGS:-"-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized"} \ cmake .. \ -DMYSQL_SERVER_SUFFIX="-$(DEBVERSION)" \ -DBUILD_CONFIG=mysql_release \ -DCOMPILATION_COMMENT="mariadb.org binary distribution" \ -DSYSTEM_TYPE="debian-linux-gnu" \ -DDEB=debian' 2014-04-01 16:21 GMT+03:00 Peter W Bowey <support(a)pbcomp.com.au>: > You require -fPIC to compile. See the info you reported = "> object; > recompile with -fPIC" > > Info is here (via GCC manual): > > -fPIC > If supported for the target machine, emit position-independent code, > suitable for dynamic linking and avoiding any limit on the size of the > global offset table. This option makes a difference on the m68k, PowerPC and > SPARC. > Position-independent code requires special support, and therefore works only > on certain machines. > > When this flag is set, the macros __pic__ and __PIC__ are defined to 2. > > In brief, the term position independent code (PIC) refers to the generated > machine code which is memory address agnostic, i.e. does not make any > assumptions about where it was loaded into RAM. Only position independent > code is supposed to be included into shared objects (SO) as they should have > an ability to dynamically change their location in RAM. > > > ----- Original Message ----- From: "Otto Kekäläinen" <otto(a)seravo.fi> > To: <maria-developers(a)lists.launchpad.net> > Cc: "Roger Meier" <roger(a)bufferoverflow.ch>; "Jake Farrell" > <jfarrell(a)apache.org> > Sent: Tuesday, April 01, 2014 11:16 PM > Subject: [Maria-developers] Cassandra plugin build error: > _S_empty_rep_storage' can not be used when making a shared object; recompile > with -fPIC > > >> Hello! >> >> I am testing out building MariaDB 10.0.10 on a Ubuntu 12.04 system >> with libthrift-dev installed (latest 0.9.0 from repo 'deb >> http://ppa.launchpad.net/wnoronha/thrift/ubuntu precise main'). >> >> Building fails with this error: >> >> make[4]: Siirrytään hakemistoon >> "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" >> Scanning dependencies of target cassandra >> make[4]: Poistutaan hakemistosta >> "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" >> make[4]: Siirrytään hakemistoon >> "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" >> [ 75%] Building CXX object >> storage/cassandra/CMakeFiles/cassandra.dir/ha_cassandra.cc.o >> [ 75%] Building CXX object >> storage/cassandra/CMakeFiles/cassandra.dir/cassandra_se.cc.o >> [ 75%] Building CXX object >> storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/Cassandra.cpp.o >> [ 75%] Building CXX object >> storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/cassandra_types.cpp.o >> [ 75%] Building CXX object >> >> storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/cassandra_constants.cpp.o >> Linking CXX shared module ha_cassandra.so >> /usr/bin/ld.bfd.real: >> >> /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libthrift.a(TApplicationException.o): >> relocation R_X86_64_32S against `std::basic_string<char, >> std::char_traits<char>, std::allocator<char> >>> >>> ::_Rep::_S_empty_rep_storage' can not be used when making a shared >> >> object; recompile with -fPIC >> /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libthrift.a: could >> not read symbols: Bad value >> collect2: ld:n paluuarvo oli 1 >> make[4]: *** [storage/cassandra/ha_cassandra.so] Virhe 1 >> >> >> Ideas? >> >> >> -- >> Check out our blog at http://seravo.fi/blog >> and follow @ottokekalainen >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~maria-developers >> Post to : maria-developers(a)lists.launchpad.net >> Unsubscribe : https://launchpad.net/~maria-developers >> More help : https://help.launchpad.net/ListHelp >> > > > --- > This email is free from viruses and malware because avast! Antivirus > protection is active. > http://www.avast.com > -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

1 0

[Maria-developers] DBT-3 benchmarks: MariaDB 10 vs MySQL 5.6
by Colin Charles 31 Mar '14

31 Mar '14

Hi XL, Nice work on posting new benchmarks in replication. Are you planning on running DBT-3 benchmarks with 10.0.10 and latest 5.6? Maybe even 5.7 DMR? Thanks -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

1 0

[Maria-developers] maria-developers@lists.launchpad.net
by Sergey Vojtovich 31 Mar '14

31 Mar '14

Hi! This is second list of questions I came up while merging test cases: - --ignore-builtin-innodb is deprecated in 5.6, do we want to deprecate it too? - LOW_PRIORITY WRITE is deprecate in 5.6, do we want to deprecate it too? The following items seem to be not merged to MariaDB. Do we want to merge them? WL#5331 Support Unicode for Windows command line client Bug#12688860 : SECURITY RECOMMENDATION: PASSWORDS ON CLI WL#5855 (EXPLAIN FORMAT=JSON) partition test WL#5980 - Allow remote tablespaces (need to check) WL5602 Providing password hashing with appropriate strength Bug#12316645: Wrong cost calculation with optimizer_join_cache_level settings WL#5602 - SHA256 password authentication Bugs#12635232: VALGRIND WARNINGS: IS_IPV6, IS_IPV4, INET6_ATON, INET6_NTOA + MULTIBYTE CHARSET Bug#14040277 UNINITIALIZED VALUE REFERENCED IN STR_TO_IPV6 And a lot of other IPV6 related stuff. Regards, Sergey

2 2

[Maria-developers] Intermediate status for test cases merge
by Sergey Vojtovich 31 Mar '14

31 Mar '14

Hi! So far I processed 431 test cases from the main suite. Most of them are up to date, a few merged without problems, a few tests fail (bugs reported), a few need extra analysis. There are a few items which I'd like to discuss: - INSERT DELAYED is deprecated in 5.6, do we want to deprecate it too? - SHOW PROFILE is deprecated in 5.6, do we want to deprecate it too? - YEAR(2) is deprecated in 5.6, do we want to deprecate it too? Monty suggests that we shouldn't deprecate it. I find it Ok too. Reasons for YEAR(2) deprecation are not obvious, relevant worklog is private. Relevant revision comment says: "YEAR(2) is a subject to deprecation since it has ill design." - EXPLAIN gives NULL in Extra column in 5.6 (empty string in 10.0), do we want behave similarly? - Do we want to merge fix for BUG#60269 - mysql should reject attempts to create system tables in incorrect engine http://bugs.mysql.com/bug.php?id=60269 - NO_ENGINE_SUBSTITUTION is default mode in 5.6 now, do we want it be default too? - 5.6 has many updates to default values, do we like to merge them too? Thanks, Sergey

2 6

Re: [Maria-developers] [Commits] Rev 3995: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 31 Mar '14

31 Mar '14

Hi, Holyfoot! On Dec 09, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3995 > revision-id: holyfoot(a)askmonty.org-20131209020845-9o3l30ao90vzf600 > parent: holyfoot(a)askmonty.org-20131208143921-3veatgg9wngyzb47 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Mon 2013-12-09 06:08:45 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > If a prepared statement calls an stored procedure, > the thd->server_status out of the SP goes up > to the PS and then to the client. So that the > client gets the SERVER_STATUS_CURSOR_EXISTS status > if the SP uses a cursor. Which makes the embedded > server fail. Why does embedded fail while a normal client-server protocol is ok with that? > Fixed by saving/restoring the upper-level server_status > in sp_head::execute(). Regards, Sergei

2 3

[Maria-developers] Engine independent statistics - naming issues?
by Sergey Petrunia 31 Mar '14

31 Mar '14

Hello, I was looking at whether column names used by EIS feature are consistent with themselves and with ther parts of the server. == Database / table / column names == There are a few naming conventions arready in use: 1. Old school: mysql.columns_priv, tables_priv Db Table_name Column_name 2. SQL STANDARD ALL CAPS HELLO FROM 1970S: information_schema.* (tables, columns, indexes, index_statistics, etc) TABLE_CATALOG (always 'def') TABLE_SCHEMA TABLE_NAME COLUMN_NAME INDEX_NAME 3. InnoDB has its own convention: mysql.innodb_table_stats, innodb_index_stats database_name table_name index_name 4. EITS feature will add: mysql.column_stats, index_stats, table_stats db_name table_name column_name index_name == Other == There are variables @@histogram_size, @@histogram_type. mysql.column_stats has columns named hist_size, hist_type. What's your opinion, should we make some renames to make things more consistent? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] [Commits] Rev 3991: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 31 Mar '14

31 Mar '14

Hi, Alexey! On Mar 26, Alexey Botchkov wrote: > Hi, Sergei. > > The Protocol::net_store_data is virtual in the embedded-server case > because we need it implemented differently for Protocol_binary. Ok, got it (finally), sorry. I'm almost always use ctags when looking for definitions, but in this case ctags didn't handle this correctly: #ifndef EMBEDDED_LIBRARY bool Protocol::net_store_data(const uchar *from, size_t length) #else bool Protocol_binary::net_store_data(const uchar *from, size_t length) #endif { ... so I never noticed that Protocol_binary actually has its own implementation of ::net_store_data() in the embedded case. Still, another question, why did you have to rename net_store_data() to net_store_data_cs() ? I've simply made it virtual and everything compiled just fine - it has a different signature so the compiler can distinguish them all right. Regards, Sergei

1 0

[Maria-developers] MariaDB 10.0.10 release prep
by Daniel Bartholomew 28 Mar '14

28 Mar '14

All, To get a jump on things, I've created the initial changelog page and a placeholder release notes page for the upcoming MariaDB 10.0.10 release. - https://mariadb.com/kb/en/mariadb-10010-changelog/ - https://mariadb.com/kb/en/mariadb-10010-release-notes/ Please let me know of things that need to be in the release notes for this release as soon as you can (or just add them yourself). The changelog page will also likely need to be updated as final pushes are made. As of right now it is up to revision 4135. FYI: I will be unresponsive on IRC or email until around mid-afternoon on Saturday due to some family events, but I will be around in the afternoon/evening on Saturday and Sunday to finish the various necessary tasks in preparation for the release. Thanks! -- Daniel Bartholomew MariaDB

1 0

[Maria-developers] A joint patch for MDEV-5969 and MDEV-5971
by Alexander Barkov 28 Mar '14

28 Mar '14

Hello, please review a joint patch fixing a crashing bug MDEV-5969, together with a non-crashing bug MDEV-5971. Thanks.

1 0

Re: [Maria-developers] parallel replication performance
by Kristian Nielsen 27 Mar '14

27 Mar '14

Axel Schwenke <axel(a)askmonty.org> writes: > The poor performance of MySQL with 1 database is expected. However MySQL is > much better with 32 databases. Even better than MariaDB :( Ok, so I looked closer at the numbers (of which there were quite a few, thanks for doing this work :-). I have some idea of what could be going on. The short story is that for this kind of configuration, it will be necessary to configure @@slave_domain_parallel_threads to be at most the value of @@slave_parallel_threads divided by the number of domain ids used. Eg. with 32 databases/domains, --slave-parallel-threads=64 --slave-domain-parallel-threads=2. Otherwise the domains will starve each other for threads, leading to poor parallelism. In fact, with such high domain parallelism @ 32 databases, I think just using --slave-domain-parallel-threads=1 could give much better performance. The longer story: MariaDB parallel replication uses a single pool of worker threads, of size @@slave_parallel_threads. GTIDs are read serially from the relay log in the SQL driver thread, and scheduled to worker threads. Each replication domain is scheduled independently from all other domains. Each GTID is allocated to a new worker thread if possible. If no worker thread is available (because all workers already have at least one GTID queued for them), then the SQL driver thread waits for one to become free; this stalls _all_ domains from getting more work queued. When your benchmark starts, the SQL driver thread will quickly read 64 events from the relay logs and queue them each to a separate worker thread. Then no more free worker threads are available, and the SQL thread will have to wait. With 64 worker threads and 32 domains, it is quite likely that some domains will not have any work to do. At the same time, the group commit size will most likely be much less than 64, so many workers will have to wait for the previous GTID in their domain without being able to do any work, especially at the "fast" configuration where commit (which can always run in parallel) does not take much time. The result will be a lot of threads waiting on each other and waking each other up, and little real work done. By setting @@slave_domain_parallel_threads, it is possible to limit how many worker threads a single domain is allowed to grab, thereby ensuring that worker threads will be available for all domains to be active doing work. Those limited number of threads will be used round-robin to queue work for; this also reduces thread sleep and wakeup, as with luck a worker thread that completes one commit will already have more work queued up for it at that point that it can process without first returning to the thread pool and going to sleep. In particular, with @@slave_domain_parallel_threads=1, we will just have 32 threads, one for each domain, each processing events independently in parallel at full speed, which is probably the best way to handle a workload as evenly distributed across domains as this one is. It would be very useful to see new benchmarks with those results. And if time permits, @@slave_domain_parallel_threads=2 could also be interesting; maybe there could be some extra performance due to more group commit in the "peer" configuration, or maybe the cost of repeatedly alternating between two worker threads within the domain will cause reduced performance. We can actually see some evidence of this theory in the numbers. Where number of worker threads is much larger than number of domains (so starvation is less of a problem), MariaDB does much better, even better than MySQL 5.6 in some cases. But performance drops as the number of domains increase, possibly because starvation kicks in. Still, even if this theory holds, we do see 5-10% slower performance of MariaDB at eg. slave@1 compared to MySQL. This suggests some higher overhead somewhere, which I need to look into at some point, though it will be a while yet before I have time for that, I am afraid. And in any case, this benchmark is highly useful as a test - as I said before, serious load testing of the domain_id parallelism has received very little testing, so it is quite possible that there is simply some bugs that cause things to work poorly. The extra testing with @@slave_domain_parallel_threads=1 (and possibly 2) will be very useful to help understand this better. > I think 32 databases is a bit too much, not only that MySQL then shines so > bright, it's also unrealistic in the real world. What do you think? I can > rerun the benchmark with 4 or 8 databases (will take 4-5 hours). Quite > certainly MySQL will have worse numbers then :) Well, I think 32 databases is not that unrealistic - shared hosting for example could have many more. The completely even spacing of the load among those databases however is probably unrealistic. But I think we should investigate and solve the issues for MariaDB @ 32 databases. More databases, and thus more parallelism, should be better for the parallel replication, if it is worse then that is a bug that should be fixed. Even if it is just the need to configure @@slave_domain_parallel_threads, maybe we could document that better. > I also notice that for MariaDB 10.0 the tuning of the commit-wait-* > variables is critical. Settings that give trx per commit group tend to slow > down the master a lot. The default settings give only few transactions per Right, this is something to be aware of. Note that for a typical web client application or similar, --commit-wait-* probably will not be much of an issue. An individual client will see slightly higher latency, which should not be much of an issue if it is in the order of a few milliseconds. Throughput is not likely to be affected, just the number of concurrent connections will increase (at least as long as system does not get choked). On the other hand, a load with fixed parallelism, like batch processing or your sysbench runs, can be hurt significantly on throughput by --commit-wait-*, as your tests show. Also, I have heard from several users that the ability to throttle a busy master in order to prevent it from running too far ahead of the slaves, is actually often seen as a benefit rather than a performance problem. Semi-sync replication also is sometimes used to achieve similar throttling. So yes, it is a tuning parameter that is unfortunately rather sensitive depending on the actual load and applications. > Please also check the "master" page of the comparison sheet. It shows the > cost of turning on the binlog. Both MySQL and MariaDB show significant > performance loss at high concurrency when the binlog is turned off. This is > completely counter-intuitive. Kristian, can this be related to MDEV-5802? It might be related. Another possibility is that this is related to InnoDB's known weaknesses at high concurrency. When the binlog is enabled, group commit will handle the commit of a lot of transactions serially, in a tight loop inside a single thread. When the binlog is disabled, all of those transactions will be committing each in their separate thread, all of those threads heavily contending each other for the hot mutexes inside InnoDB. It would need a lot of research to be sure that this is the cause, but it's a plausible guess, at least. Thanks for your efforts so far! Overall, I think the results look pretty good, actually. We see a good speedup from MariaDB parallel replication in many of the configurations. And your comparison shows that using group commit, a large percentage of the speedup can be obtained automatically from group commit, without any need to change applications and partition the data into separate databases. - Kristian.

2 2

Re: [Maria-developers] [Commits] Rev 3991: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 27 Mar '14

27 Mar '14

Hi, Holyfoot! On Nov 30, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3991 > revision-id: holyfoot(a)askmonty.org-20131130132432-etlmes8qhhqhr1iu > parent: elenst(a)wheezy-64.home-20131128160251-vn857wx5qlon6j1p > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Sat 2013-11-30 17:24:32 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > The function Protocol::net_store_data(a, b, CHARSET_A, CHARSET_B) should > be adapted to be working in the embedded server as it's done > with the Protocol::net_store_data(a, b). > That new function renamed as net_store_data_cs, so we can make it > virtual. Why does Protocol::net_store_data() have to be virtual? (if it doesn't, Protocol::net_store_data_cs doesn't have to be virtual either and doesn't have to be renamed) Regards, Sergei

2 1

[Maria-developers] Adding a storage engine - compilation
by Amith Nagarajappa 27 Mar '14

27 Mar '14

Hi, When i am creating a folder in storage/<customEngine>. I have placed the makefile from example folder and in the customEngine folder and have made required changes. However, makefile is not generated in the storage/<customEngine> folder. What change I have to make in configure or configure.in file? Right now, my custom engine code is similar to example code. I would like to build it first and then make the changes accordingly. But makefile is not generated. Appreciate your help. Thanks and Regards, Amith

3 2

Re: [Maria-developers] [Commits] Rev 3994: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 26 Mar '14

26 Mar '14

Hi, Alexey! On Mar 26, Alexey Botchkov wrote: > Hi, Sergei. > > > What do you mean by that? > > It can be seen in the sys_vars.completion_type_func test. oops. obscure feature. Okay, thanks. ok to push that. > > revno: 3994 > > message: > > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > > Thread can be disconnected internally for example after COMMIT statements. > > So we should check this for the statement execution. Regards, Sergei

1 0

[Maria-developers] Core MariaDB 10. Task suggestion.
by Сергей Дудоладов 26 Mar '14

26 Mar '14

2 1

Re: [Maria-developers] parallel replication performance
by Kristian Nielsen 25 Mar '14

25 Mar '14

Axel Schwenke <axel(a)askmonty.org> writes: > The poor performance of MySQL with 1 database is expected. However MySQL is > much better with 32 databases. Even better than MariaDB :( I haven't had time to go through all the numbers and details yet, but just a quick check: In this test, are you sure that you are running MariaDB in GTID mode? CHANGE MASTER TO master_use_gtid=slave_pos If not, you won't get any parallelism out of using 32 databases with different domain_ids. So two interesting, but different comparisons to make here: 1. MySQL multi-threaded slave compared to MariaDB GTID with different domain_id per database. If MySQL is significantly better here it could indicate some problem in MariaDB parallel replication that should be investigated; this is possible as this part has not been tested a lot. 2. MySQL multi-threaded slave with load split into multiple databases, compared to MariaDB without configuring different domains. Or just MariaDB compared without and with different domain ids. This is an "unfair" benchmark; by explicitly declaring that different databases/domains can run in parallel, more opportunity for parallellism is available. But it could still be interesting to see how much of that can be obtained from group commit parallelism, without changing applications. Both of these are interesting, but it was not clear to me which of them you were describing? - Kristian.

2 3

[Maria-developers] Patch for MDEV 5873
by Rohit Kalhans 25 Mar '14

25 Mar '14

Hello Kristian, All, As mentioned earlier i have a patch for MDEV 5873 and i am attaching here in the bundle of the patch. I have run tests on this patch and all of them have passed. Kindly review the patch. Also i am not sure which is the suitable mailing list for sending patches, so it will be great if you can point me to the same. Thanks and regards Rohit. -- Cheerio! *Rohit*

2 1

[Maria-developers] Two 10.0-related things.
by Alexey Botchkov 25 Mar '14

25 Mar '14

Hi, Sergei. I'd like to share my recently obtained knowledge :) 1. Mac OS X and sockets. As you know there on Labrador machine that mysql-test-run.pl failed if not run with --tmpdir=/tmp. The problem apparently is that the /Users directory has the access restrictions that /tmp doesn't. I still don't know what exactly was that, but it helps if we revoke all the ACLs with 'chmod -R -N /Users/mariadb'. 2. The 10.0 fails building on my machine, with linking errors in the extra/yassl/, about missing 'operator new[]' and similar. I fixed it locally with that patch: === modified file 'extra/yassl/CMakeLists.txt' --- extra/yassl/CMakeLists.txt 2014-02-26 14:28:07 +0000 +++ extra/yassl/CMakeLists.txt 2014-03-25 07:45:46 +0000 @@ -33,9 +33,10 @@ SET(YASSL_SOURCES src/buffer.cpp src/ce ADD_CONVENIENCE_LIBRARY(yassl ${YASSL_SOURCES}) RESTRICT_SYMBOL_EXPORTS(yassl) +TARGET_LINK_LIBRARIES(stdc++) + INSTALL_DEBUG_SYMBOLS(yassl) Best regards. HF

2 1

[Maria-developers] valgrind failures in ConnectSE
by Alexander Barkov 23 Mar '14

23 Mar '14

Hi Olivier, Valgrind complains about uninitialized memory read: http://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/5… It seems the problem was introduced by this change: http://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/5… Can you please fix this asap? We're going to make a release early next week. It would be nice to merge the latest ConnectSE. Thanks.

2 1

[Maria-developers] A fix for MDEV-5689 ExtractValue(xml, 'substring(/x, /y)') crashes
by Alexander Barkov 23 Mar '14

23 Mar '14

Hi Sergei, please review a fix for MDEV-5689. It also fixes MDEV-5709 ExtractValue() with XPath variable references returns wrong result. Description: 1. The main problem was that that nodeset_func->fix_fields() was called in Item_func_xml_extractvalue::val_str() and Item_func_xml_update::val_str(), which led in some cases to execution of the XPath engine *before* having a parsed XML value. Moved to Item_xml_str_func::fix_fields(). 2. Cleanup: added a new method Item_xml_str_func::fix_fields() and moved most of the code from Item_xml_str_func::fix_length_and_dec() to Item_xml_str_func::fix_fields(), to follow the usual Item layout. 3. Cleanup: a parsed XML value is useless without the raw XML value it was built from. Previously the parsed and the raw values where stored in separate String instances. It was hard to follow how they are synchronized. Added a helper class XML which contains both parsed and raw values. Makes things easier to read and modify. 4. MDEV-5709: const_item() could incorrectly return a "true" result when XPath expression contains users/SP variable references. Now nodeset_func->const_item() is also taken into account to catch such cases. 5. Minor code enhancements.

2 5

[Maria-developers] Transactions behind a failed transaction could commit in parallel replication
by nanyi607rao 21 Mar '14

21 Mar '14

hi kristian, In parallel replication if a transaction executed failed, replication should stop immediately(unconsider retry), and transactions behind the failed one should skip or rollback. you did try to do like this, there are codes like: if (unlikely(entry->stop_on_error_sub_id <= rgi->wait_commit_sub_id)) skip_event_group= true; this codes can tell latter transactions to skip but can't tell them rollback. because if a transaction started commiting before a former transaction failed (such as Lock timeout for unknown reason), the commiting transaction will not be affectd by stop_on_error_sub_id. Then the failed transaction should wakeup latter commiting transactions and tell them to rollback, unfortunately it won't. codes like if (!rgi->is_error && !skip_event_group) err= rpt_handle_event(events, rpt); else err= thd->wait_for_prior_commit(); ... ... finish_event_group(thd, err, event_gtid_sub_id, entry, rgi); if the failed transaction didn't fail at end event, err's value would come from wait_for_prior_commit, the err would be 0 if its former transaction has successed, then the failed transaction would tell latter transactions ok to commit in finish_event_group. 2014-03-19 nanyi607rao

2 2

Re: [Maria-developers] MDEV-5914
by Kristian Nielsen 21 Mar '14

21 Mar '14

Jan Lindström <jplindst(a)mariadb.org> writes: > Here is necessary changes. I have run whole mysql-test-run and I did not see > any failures, thus is is mostly safe. I say mostly, > because we really do not have much test cases that would use multimaster setup > and try to get conflicting transactions > generated from local clients against slave thread. Thanks a lot! I will try to come up with some test cases for this, with replication and external transactions that conflict and show the problem. In parallel, I tried running the parallel replication slaves with READ COMMITTED; that also seems to work and can get Elena able to continue testing. But this change is potentially more correct; I will let you know when I have some test results. - Kristian. > ================================= > > jan@jan-GE70-0NC-0ND ~/mysql/10.0-mdev5914 $ bzr diff > === modified file 'include/mysql/plugin.h' > --- include/mysql/plugin.h 2014-02-26 14:28:07 +0000 > +++ include/mysql/plugin.h 2014-03-21 09:05:58 +0000 > @@ -617,6 +617,7 @@ > int thd_in_lock_tables(const MYSQL_THD thd); > int thd_tablespace_op(const MYSQL_THD thd); > long long thd_test_options(const MYSQL_THD thd, long long test_options); > +unsigned long long thd_group_commit_id(const MYSQL_THD thd); > int thd_sql_command(const MYSQL_THD thd); > void **thd_ha_data(const MYSQL_THD thd, const struct handlerton *hton); > void thd_storage_lock_wait(MYSQL_THD thd, long long value); > > === modified file 'include/mysql/plugin_audit.h.pp' > --- include/mysql/plugin_audit.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_audit.h.pp 2014-03-21 09:06:17 +0000 > @@ -298,6 +298,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'include/mysql/plugin_auth.h.pp' > --- include/mysql/plugin_auth.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_auth.h.pp 2014-03-21 09:06:35 +0000 > @@ -298,6 +298,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'include/mysql/plugin_ftparser.h.pp' > --- include/mysql/plugin_ftparser.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_ftparser.h.pp 2014-03-21 09:06:59 +0000 > @@ -251,6 +251,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'sql/sql_class.cc' > --- sql/sql_class.cc 2014-03-20 14:36:45 +0000 > +++ sql/sql_class.cc 2014-03-21 09:07:12 +0000 > @@ -675,6 +675,12 @@ > } > > extern "C" > +unsigned long long thd_group_commit_id(const THD *thd) > +{ > + return thd->rpl_group_commit_id; > +} > + > +extern "C" > int thd_sql_command(const THD *thd) > { > return (int) thd->lex->sql_command; > > === modified file 'storage/innobase/handler/ha_innodb.cc' > --- storage/innobase/handler/ha_innodb.cc 2014-02-26 18:36:33 +0000 > +++ storage/innobase/handler/ha_innodb.cc 2014-03-20 15:42:27 +0000 > @@ -2048,6 +2048,8 @@ > trx->check_unique_secondary = !thd_test_options( > thd, OPTION_RELAXED_UNIQUE_CHECKS); > > + trx->group_commit_id = thd_group_commit_id(thd); > + > DBUG_VOID_RETURN; > } > > > === modified file 'storage/innobase/include/trx0trx.h' > --- storage/innobase/include/trx0trx.h 2014-02-26 18:36:33 +0000 > +++ storage/innobase/include/trx0trx.h 2014-03-20 15:28:19 +0000 > @@ -840,6 +840,7 @@ > or the state last time became > TRX_STATE_ACTIVE */ > trx_id_t id; /*!< transaction id */ > + ib_uint64_t group_commit_id;/*!< Group commit id or 0 */ > XID xid; /*!< X/Open XA transaction > identification to identify a > transaction branch */ > > === modified file 'storage/innobase/lock/lock0lock.cc' > --- storage/innobase/lock/lock0lock.cc 2014-02-26 18:22:48 +0000 > +++ storage/innobase/lock/lock0lock.cc 2014-03-21 09:32:54 +0000 > @@ -1020,6 +1020,23 @@ > return(FALSE); > } > > + if (trx->group_commit_id != 0 && > + lock2->trx->group_commit_id != 0 && > + trx->group_commit_id == lock2->trx->group_commit_id && > + (type_mode & LOCK_GAP || lock_rec_get_gap(lock2))) { > + /* No lock request needs to wait GAP locks if > + transactions belong to the same group commit group. > + This is because we have already verified these > + transactions on master that they do not conflict > + on record locks but as they could be executed on > + different order on slave we should not wait for > + gap locks here. Note that parallel replication > + makes sure that transaction commits are executed > + in the same order in master and slave. */ > + > + return (FALSE); > + } > + > return(TRUE); > } > > > === modified file 'storage/xtradb/handler/ha_innodb.cc' > --- storage/xtradb/handler/ha_innodb.cc 2014-02-26 18:21:23 +0000 > +++ storage/xtradb/handler/ha_innodb.cc 2014-03-20 15:42:41 +0000 > @@ -2308,6 +2308,8 @@ > > trx->fake_changes = THDVAR(thd, fake_changes); > > + trx->group_commit_id = thd_group_commit_id(thd); > + > #ifdef EXTENDED_SLOWLOG > if (thd_log_slow_verbosity(thd) & (1ULL << SLOG_V_INNODB)) { > trx->take_stats = TRUE; > > === modified file 'storage/xtradb/include/trx0trx.h' > --- storage/xtradb/include/trx0trx.h 2014-02-26 18:21:23 +0000 > +++ storage/xtradb/include/trx0trx.h 2014-03-20 15:28:20 +0000 > @@ -861,6 +861,7 @@ > or the state last time became > TRX_STATE_ACTIVE */ > trx_id_t id; /*!< transaction id */ > + ib_uint64_t group_commit_id;/*!< Group commit id or 0 */ > XID xid; /*!< X/Open XA transaction > identification to identify a > transaction branch */ > > === modified file 'storage/xtradb/lock/lock0lock.cc' > --- storage/xtradb/lock/lock0lock.cc 2013-12-22 16:06:50 +0000 > +++ storage/xtradb/lock/lock0lock.cc 2014-03-21 09:32:46 +0000 > @@ -1021,6 +1021,23 @@ > return(FALSE); > } > > + if (trx->group_commit_id != 0 && > + lock2->trx->group_commit_id != 0 && > + trx->group_commit_id == lock2->trx->group_commit_id && > + (type_mode & LOCK_GAP || lock_rec_get_gap(lock2))) { > + /* No lock request needs to wait GAP locks if > + transactions belong to the same group commit group. > + This is because we have already verified these > + transactions on master that they do not conflict > + on record locks but as they could be executed on > + different order on slave we should not wait for > + gap locks here. Note that parallel replication > + makes sure that transaction commits are executed > + in the same order in master and slave. */ > + > + return (FALSE); > + } > + > return(TRUE); > }

1 0

[Maria-developers] [GSoC 2014] CREATE OR REPLACE, CREATE IF NOT EXISTS and DROP IF EXISTS
by sriram patil 21 Mar '14

21 Mar '14

Hi Sergei, I just went through WL#3129 (thanks to James Briggs for the link). It gave me a detailed insight on what exactly is expected. I have missed 2 object types in my proposal which are, USER and SERVER. I can edit my proposal to incorporate the changes in the deliverables timeline if you want me to. Thanks, Sriram

1 0

Re: [Maria-developers] [GSOC 2014] "CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS" Project
by Sergei Golubchik 21 Mar '14

21 Mar '14

Hi, Sriram! First: please don't forget to cc: the list maria-developers(a)lists.launchpad.net Thank you. On Mar 18, sriram patil wrote: > Hi Sergei, > > I was going through the source code and was able to figure out the > flow of query execution and major functions through which a query goes > before execution. > > The query execution majorly dwells in the "sql" directory of the > source code. So, I think I will have to make changes in the following > files throughout the project, > > - sql_parse.cc (do_command, dispatch_command, mysql_execute_command) Yes. And also sql_yacc.yy - to extend the parser. And the implementation of these commands, for example, sql_acl.cc for CREATE/DROP USER, sql_trigger.cc for CREATE/DROP TRIGGER, etc. > - sql_prepare.cc (Prepared_statement::execute_loop, > Prepared_statement::execute) No, hopefully, this will work automatically. > What I am not sure about is will I have to create a handler for some of > these commands? You are not creating new commands > And will lower level storage engine code changes required? I don't think so. > > > I am interested to work on "CREATE OR REPLACE, CREATE IF NOT EXISTS, and > > > DROP IF EXISTS" project as part of GSOC 2014. I want to apply my subject > > > knowledge at a bigger level which will also give me an idea about the > > > practical issues while implementing the features. > > > Regards, Sergei

3 9

[Maria-developers] [GSoC] MDEV-371- index on BLOB
by smit hinsu 21 Mar '14

21 Mar '14

Hi Sergei and Roberto, My name is Smit Hinsu and I am final year undergrad student. I have good knowledge about database internals and good experience with C++. I find feature of creating index on BLOB quickly using MD-5 interesting. I have understood basic requirements for this feature. I have following questions in my mind currently. I think, HASH index is not supported in MySQL's version of MyISAM storage engine. Has MariaDB removed this restriction? We should support index for BLOB for BTREE index also in which we store only hash value of the value in BTREE. Is this possible? Can you suggest me some startup task related to index or storage engine which can make me familiar with the related parts? I have started to look at code. From Smit Hinsu

2 2

[Maria-developers] Engine independent index statistics - infrastructure questions
by Sergey Petrunia 20 Mar '14

20 Mar '14

Hello Igor and others, I've been playing around with engine independent statistics, and got a few questions. == mysqldump == mysqldump dumps table' data, but besides that it can dump other related objects, e.g. it will dump triggers that are attached to the table. Should it dump table statistics values, also? This question is of interest for DBAs, but also for optimizer developers. If a user has a complaint about the optimizer, we frequently ask them to upload a dataset. How do we request table/column statistics? We can't request output of SELECT * ..., because histogram data is binary and will be corrupted. We should provide users a convinent way to pass us relevant part of the EITS tables. == dropping table statistics == How does one drop table statistics? are we satisfied with needing to run three statements DELETE FROM mysql.column_stats WHERE schema_name=$db AND table_name=$table_name; DELETE FROM mysql.index_stats WHERE table_name=$db AND table_name=$table_name; DELETE FROM mysql.column_stats WHERE table_name=$db AND table_name=$table_name; or we want a simpler SQL, like DROP STATISTICS FOR t1; == Permissions on statistic tables == What should be the permissions on statistics tables? I guess we should follow pattern from other system tables - a user is typically able to read the table but not write to it (so that he can't mess with other users' SPs/triggers/etc). However, if the user is not allowed to modify the table directly, we should provide enough ways to modify the table indirectly. For example, we should provide a way to drop statistics for one's tables (which brings us back to the previous item). We might also want to allow to make other modifications to statistics of tables that one has control of? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] [GSoC 2014]
by Ivan Nikulin 20 Mar '14

20 Mar '14

Hello, I would like to take part into one of your projects on base of GSoC 2014. I am an experienced C/C++ programmer with wide knowledge of algorithms and data structures and huge interest into this sphere. I have some basic knowledge of SQL language, but unfortunatelly I don't have any database development experience. Nevertheless, gaining the latter is my essential purpose. Can I count on a chance of taking part in one of your projects? Sorry for writing to you that late. P.S. I tried writing to you a bit earlier, but the mail wouldn't appear on mailing list. -- Best regards, Ivan Nikulin

2 1

Re: [Maria-developers] [Commits] Rev 4053: MDEV-5864 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.free_tables
by Sergei Golubchik 19 Mar '14

19 Mar '14

Hi, Sergey! On Mar 15, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 4053 > revision-id: svoj(a)mariadb.org-20140315185301-ce4r6br023co80dm > parent: sanja(a)montyprogram.com-20140314073116-nklb0jkjd4t1w40n > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Sat 2014-03-15 22:53:01 +0400 > message: > MDEV-5864 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.free_tables > > Let TABLE_SHARE::tdc.free_tables, TABLE_SHARE::tdc.all_tables, > TABLE_SHARE::tdc.flushed and corresponding invariants be protected by > per-share TABLE_SHARE::tdc.LOCK_table_share instead of global LOCK_open. This looks pretty good to me! A couple of comments, see below. > === modified file 'sql/sql_plist.h' > --- a/sql/sql_plist.h 2013-11-20 11:05:39 +0000 > +++ b/sql/sql_plist.h 2014-03-15 18:53:01 +0000 > @@ -142,6 +142,14 @@ class I_P_List : public C, public I > > return result; > } > + inline T *back() > + { > + T *result= m_first; > + if (result) > + while (*B::next_ptr(result)) > + result= *B::next_ptr(result); > + return result; > + } I think this is wrong. The correct way would be to use I_P_List<> with the I_P_List_fast_push_back policy, then you could use have get_last() method to get the last element in the list. May be as inline T *back() { return *I::get_last(); } > void swap(I_P_List<T, B, C> &rhs) > { > swap_variables(T *, m_first, rhs.m_first); > > === modified file 'sql/table_cache.cc' > --- a/sql/table_cache.cc 2014-03-06 12:19:12 +0000 > +++ b/sql/table_cache.cc 2014-03-15 18:53:01 +0000 > @@ -70,13 +70,6 @@ static int32 tc_count; /**< Number of TA > > > /** > - Protects TABLE_SHARE::tdc.free_tables, TABLE_SHARE::tdc.all_tables. > -*/ > - > -mysql_mutex_t LOCK_open; So, you've now removed LOCK_open completely? Cool! > - > - > -/** > Protects unused shares list. > > TABLE_SHARE::tdc.prev > @@ -360,12 +378,15 @@ bool tc_release_table(TABLE *table) > table->in_use= 0; > /* Add table to the list of unused TABLE objects for this share. */ > table->s->tdc.free_tables.push_front(table); > - mysql_mutex_unlock(&LOCK_open); > + mysql_mutex_unlock(&table->s->tdc.LOCK_table_share); > return false; > > purge: > + /* Wait for MDL deadlock detector */ > + while (table->s->tdc.all_tables_refs) > + mysql_cond_wait(&table->s->tdc.COND_release, &table->s->tdc.LOCK_table_share); you're using it many times, I'd put it in a small static inline function, like static wait_for_mdl_deadlock_detector(TABLE *table) { mysql_mutex_assert_owner(&table->s->tdc.LOCK_table_share); while (table->s->tdc.all_tables_refs) mysql_cond_wait(&table->s->tdc.COND_release, &table->s->tdc.LOCK_table_share); } > tc_remove_table(table); > - mysql_mutex_unlock(&LOCK_open); > + mysql_mutex_unlock(&table->s->tdc.LOCK_table_share); > table->in_use= 0; > intern_close_table(table); > return true; Regards, Sergei

2 2

[Maria-developers] [GSOC 2014] GIS Enhancement
by bang honam 19 Mar '14

19 Mar '14

Hi everybody. I am Honam Bang, a researcher of Korean Institute of Science Technology and I will be a master student of UCSD, USA from this Sep. I have been working with GIS research for about one year. Therefore, I decided to work on the 'GIS Enhancement topic' because I have a lot of interest for dealing with geospatial data. My topic is 'implementing shp to sql command tool for MariaDB'. #MDEV-5813. Most shptosql tools are enable to convert 3D coordinates as I know. The shape file white paper( http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf) specifies a lot of 3D coordinates like POINT Z, MULTIPOINTZ which does not exist in MariaDB's Geometry type. Does my topic include implementing those type converting as well? Thank you Honam Bang

2 1

[Maria-developers] first results from replication benchmarks
by Axel Schwenke 18 Mar '14

18 Mar '14

Hi everybody, here are some first results from my benchmarks of replication / group commit performance. The benchmark is sysbench OLTP rw, data set fitting in memory. I tested a total of 27 configurations, 9 each for Maria-5.5.36, Maria-10.0.9 and MySQL-5.6.16. Variations were for binlog on/off, SBR vs. RBR and sync_binlog=0/1. For MariaDB I also tested XtraDB vs. InnoDB, for MySQL it was GTID on vs. GTID off. Not surprisingly turning on the binlog comes with a performance penalty. Also sync_binlog=1 adds another penalty. Quite consistently RBR was a little bit slower than SBR for all combinations (I also observed the binlogs to be bigger for RBR by a factor of nearly 2). The only real surprise was the cost of GTID in MySQL as can be seen in the attached plot. MariaDB-10.0 has GTID enabled by default (you cannot even turn it off) Right now the same benchmark is running again, this time using a single disk for binlogs (was a fast RAID before). Next step is applying binlogs to a slave with different settings for number of slave threads etc. XL

2 2

[Maria-developers] Gsoc proposal submitted for user defined event
by Savita Agrawal 18 Mar '14

18 Mar '14

Hi, I have submitted my proposal for user defined event support project. I am still working on the proposal. Please give feedback. Regards, Savita

1 0

[Maria-developers] Group commit id in mysqlbinlog output
by Kristian Nielsen 18 Mar '14

18 Mar '14

There was a question on the output of mysqlbinlog related to group commit. In 10.0, if two transactions group commit together on the master, their GTID event contains a "commit id" - a 64-bit number. This is used by parallel replication; if two transactions have the same commit id, they can be executed in parallel. In mysqlbinlog output, this looks like this: #140314 13:42:56 server id 1 end_log_pos 772 GTID 0-1-12 cid=180 ... #140314 13:42:56 server id 1 end_log_pos 1027 GTID 0-1-13 cid=180 But if a transaction commits alone on the master (no other transactions participate in the group commit), the mysqlbinlog outpus has no commit id: #140314 13:42:56 server id 1 end_log_pos 437 GTID 0-1-10 The question was why there is no cid=X in the second case, as it makes scripting/grepping the output harder. The reason is that there is no commit id in the event in the binlog in this case (to reduce the size of the binlog). So there is no valid number to put in there. One option could be to make the output something like this: #140314 13:42:56 server id 1 end_log_pos 437 GTID 0-1-10 cid=<none> This would allow to grep for "cid=" and catch everything. I do not have a strong opinion one way or the other, if there are people who find this useful, then let me know and I can change it. - Kristian.

4 6

Re: [Maria-developers] rbr triggers (all)
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, Oleksandr! Ok to push. See a couple of comments below. On Mar 18, Oleksandr Byelkin wrote: > === added file 'mysql-test/suite/rpl/t/rpl_row_triggers.test' > --- mysql-test/suite/rpl/t/rpl_row_triggers.test 1970-01-01 00:00:00 +0000 > +++ mysql-test/suite/rpl/t/rpl_row_triggers.test 2014-03-11 08:49:31 +0000 > @@ -0,0 +1,280 @@ > +-- source include/have_binlog_format_row.inc > +-- source include/have_rbr_triggers.inc > +-- source include/master-slave.inc > +-- source include/have_innodb.inc Note that master-slave.inc should always be the last in the series of includes. Everything that can fail should be before it, otherwise you can abort and skip the test after configuring replication, but without a corresponding replication cleanup. It will affect following tests. > === added file 'mysql-test/suite/rpl/t/rpl_row_triggers_sbr.test' > --- mysql-test/suite/rpl/t/rpl_row_triggers_sbr.test 1970-01-01 00:00:00 +0000 > +++ mysql-test/suite/rpl/t/rpl_row_triggers_sbr.test 2014-03-11 08:49:48 +0000 > @@ -0,0 +1,43 @@ > +--source include/master-slave.inc > +--source include/have_binlog_format_statement.inc > +--source include/have_rbr_triggers.inc Same here. > === modified file 'sql/log_event.cc' > --- sql/log_event.cc 2014-03-05 22:20:10 +0000 > +++ sql/log_event.cc 2014-03-11 18:39:51 +0000 > @@ -11121,6 +11233,13 @@ Rows_log_event::write_row(rpl_group_info > TABLE *table= m_table; // pointer to event's table > int error; > int UNINIT_VAR(keynum); > + > +#ifdef RBR_TRIGGERS > + bool invoke_triggers= > + slave_run_triggers_for_rbr && !master_had_triggers && table->triggers; > +#else > +#define invoke_triggers 0 > +#endif //RBR_TRIGGERS I think you don't need this ifdef, just declare invoke_triggers as const: const bool invoke_triggers= slave_run_triggers_for_rbr && !master_had_triggers && table->triggers; > auto_afree_ptr<char> key(NULL); > > prepare_record(table, m_width, Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3994: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, Holyfoot! On Dec 08, holyfoot(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 3994 > revision-id: holyfoot(a)askmonty.org-20131208143921-3veatgg9wngyzb47 > parent: holyfoot(a)askmonty.org-20131202104753-srqqm7hv1rtcurx4 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Sun 2013-12-08 18:39:21 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > Thread can be disconnected internally for example after COMMIT statements. What do you mean by that? > So we should check this for the statement execution. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3993: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, holyfoot! On Dec 02, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3993 > revision-id: holyfoot(a)askmonty.org-20131202104753-srqqm7hv1rtcurx4 > parent: holyfoot(a)askmonty.org-20131130142537-7j272euoaoy0svg1 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Mon 2013-12-02 14:47:53 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > mysqltest in the 'embedded-server' mode runs queries in a separate thread, > but it didn't do so for the statements - they were run in the main thread. > That leads to inconsistencies. > When a test sets SESSION 'dbug' variable like > SET SESSION debug_dbug="+d,warn_during_ha_commit_trans"; > it is run as a plain query in that separate thread, so the main thread remains unaffected. > After that the statement run in the main thread doesn't produce expected 'dbug' errors, > so the test fail. > To fix that I made statement to be run in that special thread along with the plain queries. > That makes the environment consistent. Ok to push. But sayng "statements" in the comment is a big misleading, it'd be cleared if you'd say "prepared statements". Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3992: MDEV-5138 Numerous test failures in "mtr --ps --embedded". in file:///home/hf/wmar/mdev-5138/
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, holyfoot! On Nov 30, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3992 > revision-id: holyfoot(a)askmonty.org-20131130142537-7j272euoaoy0svg1 > parent: holyfoot(a)askmonty.org-20131130132432-etlmes8qhhqhr1iu > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Sat 2013-11-30 18:25:37 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > As Davi added code like > sav_protocol= thd->protocol > thd->protocol= &thd->protocol_binary > ... > thd->protocol= sav_protocol > the fucntions like emb_store_querycache_result() cannot determine > the used protocol testing thd->protocol == &thd->protocol_binary. > Fixed by additional check thd->command == COM_STMT_EXECUTE. What about PREPARE? Regards, Sergei

1 0

Re: [Maria-developers] MaxScale as a binlog server
by Kristian Nielsen 18 Mar '14

18 Mar '14

Mark Riddoch <mark.riddoch(a)skysql.com> writes: > we have been approached by a user with a suggestion for a project that we > might do with them. They have a replication environment with a large fan out > (100 slaves per master) and are looking at doing something to cut one the > load on the master. the thought they had was to use MaxScale to effectively > cache the bin log files, so MaxScale would act as a single Slave to the > Master and would itself have a number of Slaves that read the bin log from > MaxScale. My initial response was why not just use an instance of MariaDB as > the intermediate node and connect slaves to one of those instances, in a > tree structure. They had a few reasons why they did not want to do this; > Looking in more detail I think it is feasible, we essentially write a router > module for MaxSale that acts as a slave, aching the bin log locally on the > MaxScale node. We then have another router, or maybe even the same one, that > relays that binlog to the real slaves. We are looking at doing something for > MariaDB 10, since we would want to utilise the GTID to get the best failover > semantics, I am assuming there must be some differences between the > replication stream that I can find documented for MySQL and what MariaDB 10 > does? Is this stuff anywhere on the Knowledge Base? I tried looking but Right, so I agree with what Serg said, that maybe going back to the original problem and seeing if there are better solutions possible would be better. However, I will try to answer the technical aspect here. I suppose caching and routing binlogs outside of the server should be possible, but it might be more complex than you would think at first. I am not aware of any comprehensive documentation for this. However, it should be relatively easy to see from the code. Basically, it is the function mysql_binlog_send() in sql/sql_repl.cc that handles sending binlog data to a slave. There are good comments in there describing many of the trickier points. You need to be aware that there is more logic in sending binlog data to a slave than just streaming raw binlog files. Maybe you can implement a subset and document any limitations. The old-style replication (not using GTID) is the simplest, but it still does a bit extra, like sending some extra events (FORMAT_DESCRIPTION_EVENT and fake ROTATE_EVENT). There is also the handling of old slaves. A slave sends a value in @mariadb_slave_capability, and the master will rewrite or remove any events that the slave does not understand, as appropriate. And there is the @@skip_replication flag; when the slave sets this, the master removes events that were logged with @@skip_replication set (to reduce bandwidth needs). The slave also does several SQL queries to obtain additional information. For example, the slave does a SELECT binlog_gtid_pos(file, offset) to obtain the GTID position associated with a given old-format replication position, so that it knows how to switch to GTID mode later. Maybe those queries could be handled by relaying them back to a real master server. The GTID mode is significantly more complex, due to a GTID position having multiple streams, one per replication domain. The slave sends @slave_connect_state, @slave_gtid_strict_mode, @slave_gtid_ignore_duplicates, and optionally @slave_until_gtid values. The router would need to parse these values and act accordingly. The binlog sender needs to keep track of each domain as the binlog files are scanned. Events in a domain must be skipped until the point specified in @slave_connect_state is reached. Then events are sent until the point in @slave_until_gtid is reached, at which point further events must again be skipped. The master also sends extra GTID_LIST events, which contain the state of the binlog at the point of the event, which is needed by the slave to correctly handle START SLAVE UNTIL master_gtid_pos=XXX and MASTER_GTID_WAIT(). A lot of effort was put into error handling, to support all reasonable, and even many unreasonable uses of replication, while still giving an error in cases where things are obviously wrong (to prevent silently doing the wrong thing or hanging). For example, if the slave tries to connect at a GTID that the master does not have, this is normally an error. This is important, because that GTID might be some transaction that was executed manually on the slave by mistake; if it only exists on the slave, then the master would endlessly skip events looking for the GTID that never shows up, and replication would silently hang. However, there are special cases where the slave _is_ allowed to connect at a GTID that does not exist on the master. For example, one might have a slave with --log-slave-updates=0. Then if this slave is promoted as a new master, the binlogs will not have the current position, but it is still allowed to connect to it at the point where it became the new master (but not to a point earlier than that). Another special case is if the requested GTID (but no GTIDs following that) was purged from the binary logs, as can easily happen if a replication domain is unused for long time. A _full_ implementation of a router for sending binlog to slaves would be some effort, but it might be reasonable to implement some subset. I suppose the first step would be for you to go through the code in mysql_binlog_send() and the functions it calls, and decide on which parts to implement. I will be happy to answer any questions that might pop up on the way, of course. > 2. They wanted to maintain the same group commit groups as the original > master, so as to benefit from the parallel replication. Note that this may no longer apply to MariaDB 10.0.9. The issue is that an intermediate slave that is itself a master may have less group commit than the original master, which reduces the possibility for a third-level slave to efficiently do parallel replication. But in 10.0.9, it should be possible to configure the intermediate slave with --binlog-commit-wait-* to get better group commit. > 3. They think that can get better failover semantics because they will have > n (maybe 5 or 10) MaxScales and they will all have the same bin log, so > failover between them will be easier. I am curious how it will be assured that they all have the same binlog in case of crash? But maybe you just mean that all will be a subset of the largest one. Hope this helps, - Kristian.

2 1

[Maria-developers] Java Connector Discussion
by Puneet Dewan 18 Mar '14

18 Mar '14

Hi Georg, As we missed our scheduled meeting yesterday, can you suggest a suitable time to discuss? Regards Puneet.

1 0

Re: [Maria-developers] [Maria-discuss] MariaDB vs. MySQL error codes (1900+)
by Kristian Nielsen 17 Mar '14

17 Mar '14

Felipe Gasper <felipe(a)felipegasper.com> writes: > Hi everyone, > > I notice that MariaDB is using the 1900+ range for its own > error codes; however, MySQL 5.7, at least, has a few errors in that > range as well. > > Are error codes just going to be an area of increasing > conflict between the two, or is there a plan to migrate MariaDB codes > to a different numbering scheme? Thanks for bringing this to attention. This is a pretty serious issue, and I've forwarded it to Tomas Ulin, who should be in a position to get it fixed. Tomas, please note this issue. MariaDB has used error codes from 1900 from the very start to avoid these kinds of conflicts. They can not be changed, as the codes are already in GA releases and applications may be relying on the numbers. You will need to ensure that MySQL 5.7 avoids conflicts with these before 5.7 goes GA, or there will be a conflict that will hurt all users and will be very hard to fix later. Felipe, you should probably also file a bug on bugs.mysql.com to try to get this fixed before MySQL 5.7 GA. Thanks, - Kristian.

2 1

[Maria-developers] redundant compilation of targets in mariadb 10.0
by Rohit Kalhans 17 Mar '14

17 Mar '14

hello all, I am trying to fix a few bugs but ever after small modifications in specific files, make command build all targets irrespective of whether or not they are dependent on the modified source. I am no expert in build commands but this seem off since it keeps taking exceptionally long time to build. Is there a trick/workaround that can prevent all targets from being built. Or am I missing something. -- Cheerio! *Rohit* *http://rohitkalhans.com <http://rohitkalhans.com>*

3 6

[Maria-developers] For Google Summer of Code 2014, Interested in the task of "statistically optimize mysql-test runs by running less tests"
by 胡仲义 17 Mar '14

17 Mar '14

Dear Sergei Golubchik, I am a post graduate student of Institute of Software, Chinese Academy of Sciences and my name is Zhongyi Hu. I major in computer science and my research field is data stream mining. Because I have got enough papers and works for graduation, I want to do something interesting, meaningful and valuable in the rest time as student. I have participated in two projects about database, one is main memory database and the other is database cluster. I got some experience of database system design and implementaion from them. Although I am just a beginner of this area, I really like it and expect to make it as my career. I often use Mysql in research and work, but MariaDB is not very familiar to me. I am tremendously optimistic about it's future because all of you. Well, let's come to the point. I am interested in the task of "statistically optimize mysql-test runs by running less tests". I chose this task because I have written a few tools for automatic test. I know the performance is very important if there are a large amount of data or cases to test. I read the MDEV-5776 and I think the major job is as follow. When the code is changed, the mysql-test is used to do the requisite tests. We need to integrate the information of the changes and the scenarios to predict the probability of failure for each test and get the relationships of the tests. Then decide what to test and what test cases should be used. The purpose is to optimize the efficiency of testing. All of these should be done by algorithm and program. In addition, I think that the job is in some ways like mining in data stream, such as many data need to be statistical analyzed and the hidden patterns changing over time. At last, I have two basic questions. 1) What exactly are the builder and the combination? I thought they refer to compiler and runtime environment. 2) What does the "individual tests within a big test file" mean? Maybe I am completely wrong, but I still look forward to your reply. I hope to have the opportunity to learn from you in work and discussion. Best regards Zhongyi Hu 20140314

3 5

[Maria-developers] [GSOC 2014] "CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS" Project
by sriram patil 16 Mar '14

16 Mar '14

Hi Sergei, I am Sriram Patil. I am a Masters' student at International Institute of Information Technology, Hyderabad, India. I have been working on a small DB Engine as part of my course work in C++. I have also worked on a SQLite DB abstraction library FMDB in Objective-C. Also, I am currently working on lock-free concurrent B-trees (Java), and going to evaluate it with an in-memory database to check the performance. I have also worked on libdaap and Rhythembox source code during my undergraduate project. You can check my GitHub<https://github.com/srirampatil?tab=repositories>and BitBucket <https://bitbucket.org/srirampatil> accounts which have some of my projects. I am interested to work on "CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS" project as part of GSOC 2014. I want to apply my subject knowledge at a bigger level which will also give me an idea about the practical issues while implementing the features. I have gone through the project description and understood that a generic interface needs to be implemented for "CREATE OR REPLACE" and "CREATE IF NOT EXISTS" for functions, procedures, tables, triggers, etc. But, I understand I am a pretty late to discuss these things with you. Is it okay if I submit my proposal document directly? Also, do you recommend me to go through some specific things with respect to this project before I start working on my proposal draft? Thanks, Sriram Patil IIIT Hyderabad, India

2 1

[Maria-developers] MDEV-5681- audit log not rotating
by Tan Tran 14 Mar '14

14 Mar '14

Hi maria-developers, This is my patch for MDEV-5681: " audit log will not rotate when the file size exceeds global variable setting": http://bazaar.launchpad.net/~tktran/maria/maria-fix-bug5681/revision/4102 To test, 1. set server_audit_events = 'connect' and keep default rotate size of 1000000. 2. connect with a bad password about 20 times: for i in {1..20}; do mysql -ufail -pfail; done 3. see that server_audit.log is X bytes 4. set server_audit_rotate_size = 1000 5. connect with bad password for another 20 times 6. see original server_audit.log is still X bytes (meaning log rotated immediately) and subsequent server_audit.log.* logs are ~1000 bytes or less Please review. Thanks, Tan Tran Prospective GSoC student

2 1

Re: [Maria-developers] [Commits] Rev 4066: MDEV-5089 - possible deadlocks between rwlocks and mutexes
by Sergei Golubchik 13 Mar '14

13 Mar '14

Hi, Sergey! On Feb 13, Sergey Vojtovich wrote: > revno: 4066 > revision-id: svoj(a)mariadb.org-20140213074049-70zbqa98y95sfpox > parent: sanja(a)askmonty.org-20140212150705-e1tfabmhjggqws7q > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5 > timestamp: Thu 2014-02-13 11:40:49 +0400 > message: > MDEV-5089 - possible deadlocks between rwlocks and mutexes > > Pre-MDL versions had direct relationship between LOCK_open and > LOCK_global_system_variables, e.g.: > intern_sys_var_ptr // locks LOCK_global_system_variable > mysql_sys_var_char > create_options_are_valid > ha_innobase::create > handler::ha_create > ha_create_table > rea_create_table > mysql_create_table_no_lock // locks LOCK_open > mysql_create_table > > With MDL this relationship was removed, but mutex order was still > recorded. In fact there is indirect relationship between LOCK_open > and LOCK_global_system_variables via rwlocks in reverse order. > > Removed LOCK_open and LOCK_global_system_variables order recording, > instead assert that LOCK_open is never held in intern_sys_var_ptr(). > > This solves only one of many problems detected with MDEV-5089. Looks good, thanks! ok to push! Regards, Sergei

1 0

[Maria-developers] TokuDB Performance Regression (MDEV-5777)
by Tim Callaghan 13 Mar '14

13 Mar '14

I'm very concerned about the regression I reported in https://mariadb.atlassian.net/browse/MDEV-5777, as users of the MariaDB.org supplied builds of both 5.5 and 10.0 are getting substantially less throughput than the MariaDB 5.5 build supplied by us at tokutek.com. I see that the fix version is now 5.5.37 and 10.0.10, is there a plan for how that will get done? I'll assist if possible, I just don't know how I can. -Tim Callaghan (Tokutek)

2 1

[Maria-developers] Fwd: Database/kernel community topic at Collaboration Summit 2014
by Hartmut Holzgraefe 13 Mar '14

13 Mar '14

Forwarding this as the original message doesn't seem to have made it to this list ... -------- Original Message -------- Subject: Database/kernel community topic at Collaboration Summit 2014 Date: Mon, 10 Mar 2014 10:15:37 +0000 From: Mel Gorman <mgorman(a)suse.de> To: pgsql-hackers(a)postgresql.org, mysql(a)lists.mysql.com, maria-developers(a)lists.launchpad.net, linux-kernel(a)vger.kernel.org Hi, Arrangements have been made to hold a meeting between database and kernel developers at Collaboration Summit 2014 http://sched.co/1hEBRuq on March 27th 2014. This was organised after discussions on pain points encountered by the PostgreSQL community. Originally the plan had been to just have a topic for LSF/MM there was much more interest in the topic than anticipated so the Collaboration Summit meeting will be much more open. If there are developers attending Collaboration Summit that work in the database or kernel communities, it would be great if you could come along. Previous discussions were on the PostgreSQL list and that should be expanded in case we accidentally build postgres-only features. The intent is to identify the problems encountered by databases and where relevant, test cases that can be used to demonstrate them if they exist. While the kernel community may be aware of some of the problems, they are not always widely known or understood. There is a belief that some interfaces are fine when in reality applications cannot use them properly. The ideal outcome of the meeting would be concrete proposals on kernel features that could be developed over the course of time to address any identified problem. For reference, this is a summary of the discussion that took place when the topic was proposed for LSF/MM. Thanks. ---8<--- On testing of modern kernels ---------------------------- Josh Berkus claims that most people are using Postgres with 2.6.19 and consequently there may be poor awareness of recent kernel developments. This is a disturbingly large window of opportunity for problems to have been introduced. Minimally, Postgres has concerns about IO-related stalls which may or may not exist in current kernels. There were indications that large writes starve reads. There have been variants of this style of bug in the past but it's unclear what the exact shape of this problem is and if IO-less dirty throttling affected it. It is possible that Postgres was burned in the past by data being written back from reclaim context in low memory situations. That would have looked like massive stalls with drops in IO throughput but it was fixed in relatively recent kernels. Any data on historical tests would be helpful. Alternatively, a pgbench-based reproduction test could potentially be used by people in the kernel community that track performance over time and have access to a suitable testing rig. It was mentioned that Postgres has an tool called pg_test_fsync which was mentioned in the context of testing different wal_sync_methods. Potentially it could also be used for evaluating some kernel patches. Gregory Smith highlighted the existence of a benchmark wrapper for pgbench called pgbench-tools: https://github.com/gregs1104/pgbench-tools . It can track statistics of interest to Postgres as well as report in interesting metrics such as transaction latency. He had a lot of information on testing requirements and some very interesting tuning information and it's worth reading the whole mail http://www.postgresql.org/message-id/52D99161.60305@gmail.com Postgres bug reports and LKML ----------------------------- It is claimed that LKML does not welcome bug reports but it's less clear what the basis of this claim is. Is it because the reports are ignored? A possible explanation is that they are simply getting lost in the LKML noise and there would be better luck if the bug report was cc'd to a specific subsystem list. A second possibility is the bug report is against an old kernel and unless it is reproduced on a recent kernel the bug report will be ignored. Finally it is possible that there is not enough data available to debug the problem. The worst explanation is that to date the problem has not been fixable but the details of this have been lost and are now unknown. Is is possible that some of these bug reports can be refreshed so at least there is a chance they get addressed? Apparently there were changes to the reclaim algorithms that crippled performance without any sysctls. The problem may be compounded by the introduction of adaptive replacement cache in the shape of the thrash detection patches currently being reviewed. Postgres investigated the use of ARC in the past and ultimately abandoned it. Details are in the archives (http://www.Postgres.org/search/?m=1&q=arc&l=1&d=-1&s=r) I have not read then, just noting they exist for future reference. Sysctls to control VM behaviour are not popular as such tuning parameters are often used as an excuse to not properly fix the problem. Would it be possible to describe a test case that shows 2.6.19 performing well and a modern kernel failing? That would give the VM people a concrete basis to work from to either fix the problem or identify exactly what sysctls are required to make this work. I am confident that any bug related to VM reclaim in this area has been lost. At least, I recall no instances of it being discussed on linux-mm and it has not featured on LSF/MM during the last years. IO Scheduling ------------- Kevin Grittner has stated that it is known that the DEADLINE and NOOP schedulers perform better than any alternatives for most database loads. It would be desirable to quantify this for some test case and see can the default scheduler cope in some way. The deadline scheduler makes sense to a large extent though. Postgres is sensitive to large latencies due to IO write spikes. It is at least plausible that deadline would give more deterministic behaviour for parallel reads in the presence of large writes assuming there were not ordering problems between the reads/writes and the underlying filesystem. For reference, these IO spikes can be massive. If the shared buffer is completely dirtied in a short space of time then it could be 20-25% of RAM being dirtied and writeback required in typical configurations. There have been cases where it was worked around by limiting the size of the shared buffer to a small enough size so that it can be written back quickly. There are other tuning options available such as altering when dirty background writing starts within the kernel but that will not help if the dirtying happens in a very short space of time. Dave Chinner described the considerations as follows There's no absolute rule here, but the threshold for background writeback needs to consider the amount of dirty data being generated, the rate at which it can be retired and the checkpoint period the application is configured with. i.e. it needs to be slow enough to not cause serious read IO perturbations, but still fast enough that it avoids peaks at synchronisation points. And most importantly, it needs to be fast enought that it can complete writeback of all the dirty data in a checkpoint before the next checkpoint is triggered. In general, I find that threshold to be somewhere around 2-5s worth of data writeback - enough to keep a good amount of write combining and the IO pipeline full as work is done, but no more. e.g. if your workload results in writeback rates of 500MB/s, then I'd be setting the dirty limit somewhere around 1-2GB as an initial guess. It's basically a simple trade off buffering space for writeback latency. Some applications perform well with increased buffering space (e.g. 10-20s of writeback) while others perform better with extremely low writeback latency (e.g. 0.5-1s). Some of this may have been addressed in recent changes with IO-less dirty throttling. When considering stalls related to excessive IO it will be important to check if the kernel was later than 3.2 and what the underlying filesystem was. Again, it really should be possible to demonstrate this with a test case, one driven by pgbench maybe? Workload would generate a bunch of test data, dirty a large percentage of it and try to sync. Metrics would be measuring average read-only query latency when reading in parallel to the write, average latencies from the underlying storage, IO queue lengths etc and comparing default IO scheduler with deadline or noop. NUMA Optimisations ------------------ The primary one that showed up was zone_reclaim_mode. Enabling that parameter is a disaster for many workloads and apparently Postgres is one. It might be time to revisit leaving that thing disabled by default and explicitly requiring that NUMA-aware workloads that are correctly partitioned enable it. Otherwise NUMA considerations are not that much of a concern right now. Bruce Momjian highlighted this block entry that covered zone_reclaim http://frosty-postgres.blogspot.com/2012/08/postgresql-numa-and-zone-reclai… Direct IO, buffered IO, double buffering and wishlists ------------------------------------------------------ The general position of Postgres is that the kernel knows more about storage geometries and IO scheduling that an application can or should know. It would be preferred to have interfaces that allow Postgres to give hints to the kernel about how and when data should be written back. The alternative is exposing details of the underlying storage to userspace so Postgres can implement a full IO scheduler using direct IO. It has been asserted on the kernel side that the optimal IO size and alignment is the most important detail should be all the details that are required in the majority of cases. While some database vendors have this option, the Postgres community do not have the resources to implement something of this magnitude. They also have tried direct IO in the past in the areas where it should have mattered and had mixed results. I can understand Postgres preference for using the kernel to handle these details for them. They are a cross-platform application and the kernel should not be washing its hands of the problem and hiding behind direct IO as a solution. Ted Ts'o summarises the issues as The high order bit is what's the right thing to do when database programmers come to kernel engineers saying, we want to do <FOO> and the performance sucks. Do we say, "Use O_DIRECT, dummy", not withstanding Linus's past comments on the issue? Or do we have some general design principles that we tell database engineers that they should do for better performance, and then all developers for all of the file systems can then try to optimize for a set of new API's, or recommended ways of using the existing API's? In an effort to avoid depending on direct IO there were some proposals and/or wishlist items. These are listed in order of likliehood to be implemented and usefulness to Postgres. 1. Hint to asynchronously queue writeback now in preparation for a fsync in the near future. Postgres dirties a large amount of data and asks the kernel to push it to disk over the next few minutes. Postgres still is required to fsync later but the fsync time should be minimised. vm.dirty_writeback_centisecs is unreliable for this. One possibility would be an fadvise call that queues the data for writeback by a flusher thread now and returns immediately 2. Hint that a page is a prime candidate for reclaim but only if there is reclaim pressure. This avoids a problem where fadvise(DONTNEED) discards a page only to have a read/write or WILLNEED hint immediately read it back in again. The requirements are similar to the volatile range hinting but they do not use mmap() currently and would need a file-descriptor based interface. Robert Hass had some concerns with the general concept and described them thusly This is an interesting idea but it stinks of impracticality. Essentially when the last buffer pin on a page is dropped we'd have to mark it as discardable, and then the next person wanting to pin it would have to check whether it's still there. But the system call overhead of calling vrange() every time the last pin on a page was dropped would probably hose us. Well, I guess it could be done lazily: make periodic sweeps through shared_buffers, looking for pages that haven't been touched in a while, and vrange() them. That's quite a bit of new mechanism, but in theory it could work out to a win. vrange() would have to scale well to millions of separate ranges, though. Will it? And a lot depends on whether the kernel makes the right decision about whether to chunk data from our vrange() vs. any other page it could have reclaimed. 3. Hint that a page should be dropped immediately when IO completes. There is already something like this buried in the kernel internals and sometimes called "immediate reclaim" which comes into play when pages are bgin invalidated. It should just be a case of investigating if that is visible to userspace, if not why not and do it in a semi-sensible fashion. 4. 8kB atomic write with OS support to avoid writing full page images in the WAL. This is a feature that is likely to be delivered anyway and one that Postgres is interested in. 5. Only writeback some pages if explicitly synced or dirty limits are violated. Jeff Janes states that he has problems with large temporary files that generate IO spikes when the data starts hitting the platter even though the data does not need to be preserved. Jim Nasby agreed and commented that he "also frequently see this, and it has an even larger impact if pgsql_tmp is on the same filesystem as WAL. Which *theoretically* shouldn't matter with a BBU controller, except that when the kernel suddenly +decides your *temporary* data needs to hit the media you're screwed." One proposal that may address this is Allow a process with an open fd to hint that pages managed by this inode will have dirty-sticky pages. Pages will be ignored by dirty background writing unless there is an fsync call or dirty page limits are hit. The hint is cleared when no process has the file open. 6. Only writeback pages if explicitly synced. Postgres has strict write ordering requirements. In the words of Tom Lane -- "As things currently stand, we dirty the page in our internal buffers, and we don't write it to the kernel until we've written and fsync'd the WAL data that needs to get to disk first". mmap() would avoid double buffering but it has no control about the write ordering which is a show-stopper. As Andres Freund described; Postgres' durability works by guaranteeing that our journal entries (called WAL := Write Ahead Log) are written & synced to disk before the corresponding entries of tables and indexes reach the disk. That also allows to group together many random-writes into a few contiguous writes fdatasync()ed at once. Only during a checkpointing phase the big bulk of the data is then (slowly, in the background) synced to disk. I don't see how that's doable with holding all pages in mmap()ed buffers. There are also concerns there would be an absurd number of mappings. The problem with this sort of dirty pinning interface is that it can deadlock the kernel if all dirty pages in the system cannot be written back by the kernel. James Bottomley stated No, I'm sorry, that's never going to be possible. No user space application has all the facts. If we give you an interface to force unconditional holding of dirty pages in core you'll livelock the system eventually because you made a wrong decision to hold too many dirty pages. However, it was very clearly stated that the writing ordering is critical. If the kernel breaks the requirement then the database can get trashed in the event of a power failure. This led to a discussion on write barriers which the kernel uses internally but there are scaling concerns both with the number of constraints that would exist and the requirement that Postgres use mapped buffers. There were few solid conclusions on this. It would need major reworking on all sides and it would handing control of system safety to userspace which is going to cause layering violations. This whole idea may be a bust but it is still worth recording. Greg Stark outlined the motivation best as follows; Ted T'so was concerned this would all be a massive layering violation and I have to admit that's a huge risk. It would take some clever API engineering to come with a clean set of primitives to express the kind of ordering guarantees we need without being too tied to Postgres's specific implementation. The reason I think it's more interesting though is that Postgres's journalling and checkpointing architecture is pretty bog-standard CS stuff and there are hundreds or thousands of pieces of software out there that do pretty much the same work and trying to do it efficiently with fsync or O_DIRECT is like working with both hands tied to your feet. 7. Allow userspace process to insert data into the kernel page cache without marking the page dirty. This would allow the application to request that the OS use the application copy of data as page cache if it does not have a copy already. The difficulty here is that the application has no way of knowing if something else has altered the underlying file in the meantime via something like direct IO. Granted, such activity has probably corrupted the database already but initial reactions are that this is not a safe interface and there are coherency concerns. Dave Chinner asked "why, exactly, do you even need the kernel page cache here?" when Postgres already knows how and when data should be written back to disk. The answer boiled down to "To let kernel do the job that it is good at, namely managing the write-back of dirty buffers to disk and to manage (possible) read-ahead pages". Postgres has some ordering requirements but it does not want to be responsible for all cache replacement and IO scheduling. Hannu Krosing summarised it best as Again, as said above the linux file system is doing fine. What we want is a few ways to interact with it to let it do even better when working with Postgres by telling it some stuff it otherwise would have to second guess and by sometimes giving it back some cache pages which were copied away for potential modifying but ended up clean in the end. And let the linux kernel decide if and how long to keep these pages in its cache using its superior knowledge of disk subsystem and about what else is going on in the system in general. 8. Allow copy-on-write of page-cache pages to anonymous. This would limit the double ram usage to some extent. It's not as simple as having a MAP_PRIVATE mapping of a file-backed page because presumably they want this data in a shared buffer shared between Postgres processes. The implementation details of something like this are hairy because it's mmap()-like but not mmap() as it does not have the same writeback semantics due to the write ordering requirements Postgres has for database integrity. Completely nuts and this was not mentioned on the list, but arguably you could try implementing something like this as a character device that allows MAP_SHARED with ioctls with ioctls controlling that file and offset backs pages within the mapping. A new mapping would be forced resident and read-only. A write would COW the page. It's a crazy way of doing something like this but avoids a lot of overhead. Even considering the stupid solution might make the general solution a bit more obvious. For reference, Tom Lane comprehensively described the problems with mmap at http://www.Postgres.org/message-id/17515.1389715715@sss.pgh.pa.us There were some variants of how something like this could be achieved but no finalised proposal at the time of writing. 9. Hint that a page in an anonymous buffer is a copy of a page cache page and invalidate the page cache page on COW. This limits the amount of double buffering. It's in as a low priority item as it's unclear if it's really necessary and also I suspect the implementation would be very heavy because of the amount of information we'd have to track in the kernel. It is important to note in general that Postgres has a problem with some files being written back too aggressively and other files not written back aggressively enough. Temp files for purposes such as sorting should have writeback deferred as long as possible. Data file writes that must complete before portions of the WAL can be discarded should begin writeback early so the final fsync does not stall for too long. As Dave Chinner says IOWs, there are two very different IO and caching requirements in play here and tuning the kernel for one actively degrades the performance of the other. Robert Hass categorised the IO patterns as follows - WAL files are written (and sometimes read) sequentially and fsync'd very frequently and it's always good to write the data out to disk as soon as possible - Temp files are written and read sequentially and never fsync'd. They should only be written to disk when memory pressure demands it (but are a good candidate when that situation comes up) - Data files are read and written randomly. They are fsync'd at checkpoint time; between checkpoints, it's best not to write them sooner than necessary, but when the checkpoint arrives, they all need to get out to the disk without bringing the system to a standstill No matter it was pointed out that fsync should never be able to screw the system. Robert Hass again summaried it as follows IMHO, the problem is simpler than that: no single process should be allowed to completely screw over every other process on the system. When the checkpointer process starts calling fsync(), the system begins writing out the data that needs to be fsync()'d so aggressively that service times for I/O requests from other process go through the roof. It's difficult for me to imagine that any application on any I/O scheduler is ever happy with that behavior. We shouldn't need to sprinkle of fsync() calls with special magic juju sauce that says "hey, when you do this, could you try to avoid causing the rest of the system to COMPLETELY GRIND TO A HALT?". That should be the *default* behavior, if not the *only* behavior. It is important to keep this in mind although sometimes the ordering requirements of the filesystem may make it impossible to achieve. At LSF/MM last year there was a discussion on whether userspace should hint that files are "hot" or "cold" so the underlying layers could decide to relocate some data to faster storage. I tuned out a bit during the discussion and did not track what happened with it since but I guess that any developments of that sort would be of interest to the Postgres community. Some of these wish lists still need polish but could potentially be discussed further at LSF/MM with a wider audience as well as on the lists. Then in a of unicorns and ponies it's a case of picking some of these hinting wishlists, seeing what it takes to implement it in kernel and testing it with a suitably patched version of postgres running a test case driven by something (pgbench presumably). -- Mel Gorman SUSE Labs -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/mysql

1 0

[Maria-developers] GSOC Proposal-Java Connector For MariaDB
by Puneet Dewan 12 Mar '14

12 Mar '14

Hi All, Please find attached my initial proposal for Java Connector for Maria DB. Awaiting for your comments.:) Regards Puneet.

1 0

[Maria-developers] InnoDB selected projects (corrected links)
by Jan Lindström 12 Mar '14

12 Mar '14

Hi all, After careful weighting and selection process, I have selected following two projects as a starting point to improve InnoDB (1) InnoDB file space defragmentation Description: External tool to physically delete delete marked rows from InnoDB file space and freeing unused file space. https://mariadb.atlassian.net/browse/MDEV-5834 (2) InnoDB fast bulk load Description: External tool to load CVS file directly to InnoDB file space (innodb_file_per_table = 1). https://mariadb.atlassian.net/browse/MDEV-5835 Now, I hope these projects mostly represent the most frequently requested features and are most usable to current customers and/or potential new customers. Next, I would hope some indication which one of the selected project you would like to see first. Votes can be added above links. R: -- -- Jan Lindström, Principal Engineer SkySQL - The MariaDB Company MariaDB | MaxScale | skype: jan_p_lindstrom www.skysql.com <http://www.skysql.com/> Twitter <http://twitter.com/skysql> Blog <http://www.skysql.com/blog/> Facebook <http://www.facebook.com/skysql> LinkedIn <http://www.linkedin.com/company/1214250> Google+ <https://plus.google.com/117544963211695643458/posts>

1 0

Re: [Maria-developers] [Commits] Rev 3768: MDEV-5717: Server crash with insert statement containing DEFAULT into view
by Sergei Golubchik 12 Mar '14

12 Mar '14

Hi, Sanja! On Mar 12, sanja(a)montyprogram.com wrote: > revno: 3768 > revision-id: sanja(a)montyprogram.com-20140312103416-3v6tan06crsqry28 > parent: sanja(a)montyprogram.com-20140307115707-poaet8iwv9d55azm > committer: sanja(a)montyprogram.com > branch nick: work-maria-5.3-MDEV-5717 > timestamp: Wed 2014-03-12 12:34:16 +0200 > message: > MDEV-5717: Server crash with insert statement containing DEFAULT into view > > Item_default_value::arg can be NULL so walk() should take it into consideration. > === modified file 'sql/item.h' > --- a/sql/item.h 2014-02-19 13:45:33 +0000 > +++ b/sql/item.h 2014-03-12 10:34:16 +0000 > @@ -3551,7 +3551,7 @@ public: > > bool walk(Item_processor processor, bool walk_subquery, uchar *args) > { > - return arg->walk(processor, walk_subquery, args) || > + return (arg && arg->walk(processor, walk_subquery, args)) || > (this->*processor)(args); > } Ok to push. But *please* install the diff_p bzr plugin as described here: https://mariadb.com/kb/en/how-to-get-more-out-of-bzr-when-working-on-mariad… Regards, Sergei

1 0

Re: [Maria-developers] Maria DB project aspirant for GSoC 2014 : CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS
by Sergei 11 Mar '14

11 Mar '14

Hi, Siddharth! First: please send questions like this to maria-developers(a)lists.launchpad.net it's a public mailing list dedicated to MySQL and MariaDB internals, source code, and related things. I am subscribed, so I'll see you mail there, and you may be sure I will, because it won't be accidentally catched by my spam filter, or sorted out in some obscure folder. Furthermore other subscribers will see your question and could reply if I will be not available (e.g. I could be travelling). Thank you. On Mar 11, Siddharth Dalmia wrote: > Hi Sergei, > > I am interested in working on the project. I am new to the open source > world so, can you tell how i should go forward? Do I need to work on > something and show? No, this is not required, but if you do it'll be a bonus point for you - you'll get familiar with MariaDB source code and would be able to write a better proposal. Everything you have to do and how you should move forward is written at the GSoC site. See http://www.google-melange.com/gsoc/homepage/google/gsoc2014 Regards, Sergei

1 0

[Maria-developers] GSOC 14: Discussion on "GIS enhancements to MariaDB"
by Abhishek Kumar 11 Mar '14

11 Mar '14

Hi, I am Abhishek and am interested in knowing about the gsoc project "GIS enhancements to MariaDB". I participate often in the programming contents hosted at these sites: Codeforces: abhi170893 (Division 1) TopCoder: abhi170893 ( Divsion 1) Codechef: abhi_iiit_17 ( Ranked 6 in India in DEC13 . 3 times in top 20 in country.) Spoj: abhi170893 (over 100 problems) The ideas page mentions C as the required skill and I am good at coding in C. Also on the ideas page not much is written about the project. Where can i get more details about it..? what things should i explore first before trying for this project..? the complexity level..? the competition involved..?Do i need to fix bugs first..? If yes, their links..? Thanks Abhishek Kumar CSE, IIIT Hyderabad (India)

2 1

Re: [Maria-developers] Rev 4033: MDEV-5766 - my_atomic_load does memory writes
by Sergei Golubchik 10 Mar '14

10 Mar '14

Hi, Sergey! On Mar 07, Sergey Vojtovich wrote: > revno: 4033 > revision-id: svoj(a)mariadb.org-20140307074306-i3l4fllu2rj5pouf > parent: svoj(a)mariadb.org-20140306121912-i1p8iilpsgc8m4bn > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Fri 2014-03-07 11:43:06 +0400 > message: > MDEV-5766 - my_atomic_load does memory writes > > my_atomic_load() is implemented as __sync_fetch_and_or(var, 0) which > writes or-ed value back to var. Memory writes as such have worse > performance and scalability than reads. > > gcc 4.7 and up offers better facility for atomic loads/stores. Use it > whenever it is available. Thanks. ok to push Regards, Sergei

1 0

[Maria-developers] Some questions on GSOC project User Defined Event
by Savita Agrawal 10 Mar '14

10 Mar '14

Hi, what is the difference between a trigger and user defined event? In postgres there is a clear distintion between trigger and event trigger. Triggers are activated when DML statements are executed whereas event triggers are activated when DDL statements are executed. But in soliddb (from which our post in JIRA page is inspired), it seems like user defined events can be activated for DML statements. What is it that can't be done with triggers and can be done with user defined event? Why is the restriction that events can only be posted and received by stored procedures only? Cheers, Savita

3 6

[Maria-developers] Project in GSOC 2014.
by Puneet Dewan 10 Mar '14

10 Mar '14

Hi All, I,Puneet Dewan, am participating in Google Summer of Code 2014. I am interested in contributing to Java Client Library for Maria DB. I have started investigating in Maria DB Java connector and have found one potential feature that can be added to MariaDB-Java connector. The feature is: * RowSetFactoryInterface and RowSetProviderClass as per JDBC Spec 4.1 which is not part of latest Java connector version JDBC 1.1.6* Benefit: *Any types of row sets that are supported by the JDBC Driver can be created.* *One Last Question:* I have seen the source code of Java Connector and I understood they are made in Java 6. Are there any plans to migrate the code from Java 6 to Java 7? Awaiting for the response. Regards Puneet

2 5

[Maria-developers] [Gsoc] LDAP authentication plug-in project
by Sudheera Palihakkara 10 Mar '14

10 Mar '14

Hi, I'm started to work above project. I want to know that if I have to use a local ldap server to test the feature like OpenLdapServer or are there any remote servers that I can connect to? thanks. -- *Sudheera Palihakkara.* Undergraduate Department of *Computer Science and Engineering,*Faculty of Engineering, *University of Moratuwa*, Sri Lanka.

3 5

Re: [Maria-developers] Maria DB project aspirant for GSoC 2014 : CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS
by Sergei 10 Mar '14

10 Mar '14

Hi, Siddharth! On Mar 08, Siddharth Dalmia wrote: > Hello Sergei and all, > > I am Siddharth Dalmia a second year undergraduate pursuing Computer > Science Engineering from Birla Institute Technology and Science (BITS) > Pilani, Hyderabad Campus. > > I have a keen interest in programming and learning algorithms. I have a > deep knowledge in C , C++ and sql. I have even done various projects in > C++ including graphics. I have an average knowledge of data structures, > java, Linux, html, css. I am also currently working on a project on > concurrency in databases. I have a thorough knowledge in relational > algebra and i have been using sql since 3 years (oracle and mysql). > > I have gone through various projects of your's for GSOC and the one > which interest me a most is CREATE OR REPLACE, CREATE IF NOT EXISTS, and > DROP IF EXISTS . I am very keen to be a part of this project as it is > something which i think its something which i am learning/learnt to put > to use. > > I know that i am a bit late, but it is not an end and i believe that i > can work hard with dedication to make a good and convincing proposal. It > will be very nice of you if you guide me on how to go about this project > and from where i could start and elaborate what i need to do. There's not much to add to the specifications. There is CREATE TABLE statement, CREATE TABLE IF NOT EXISTS statement, CREATE OR REPLACE TABLE statement. There is CREATE USER statetement, but no CREATE USER IF NOT EXISTS and no CREATE OR REPLACE USER. There is CREATE PROCEDURE, but neither CREATE PROCEDURE IF NOT EXISTS nor CREATE OR REPLACE PROCEDURE. And so on. We'd like to have a consistent syntax and support "IF NOT EXISTS" and "OR REPLACE" for all CREATE statements where possible. And the same for DROP IF EXISTS Regards, Sergei

1 0

[Maria-developers] Interest in the "LDAP Authentication plugin" project for GSoC14
by Marta Sequeira 09 Mar '14

09 Mar '14

Hi everyone! First of all I would like to introduce myself. My name is Marta Sequeira and I'm studying for my master's degree in Computer Science, in Instituto Superior Técnico, Portugal. I have skills in C from the academic projects I developed. I did a course in relational DBMS and I have managed a few MySQL databases. I also am currently maintaining some PHP code that uses a large LDAP directory (but not for authentication in this particular case), and I have worked with a Samba server that uses LDAP authentication, so I am somewhat familiar with LDAP. This is the reason why I'm very interested in the LDAP Authentication plugin project. I would like to apply for this project, so any guidelines on what I should include in my proposal are very welcome. I would also like to know if you consider this project idea to fullfil the workload goals of GSoC, or if I should apply for more projects from your ideas page. Since I am not yet familiar with the MariaDB's code, I have no notion of the amount of work it will require. Best regards, -- Marta Sequeira

1 0

[Maria-developers] Review to fix for mdev-5437
by Savita Agrawal 08 Mar '14

08 Mar '14

Hi, I am a student from India interested in participating gsoc. This is my first fix to mariadb. Please review it. Link is: http://bazaar.launchpad.net/~savitarit0/maria/maria-fix-mdev-5437/revision/…

2 1

[Maria-developers] [GSoC] proposal draft
by Tan Tran 08 Mar '14

08 Mar '14

Hi Colin Charles, maria-developers (cc'd), I'm looking to do my GSoC project on porting the InnoDB-memcached plugin to MariaDB, which you will be the mentor for. I asked maria-developers about it earlier, but haven't had any responses, so I've written a draft proposal (below) to elicit some responses. I would appreciate your technical corrections and suggestions. Thanks, Tan Tran tankimtran(a)gmail.com Proposal The MySQL+memcached combination is popular for high-speed, high-throughput websites that also need durable, long-term storage. By storing data mostly on disk, MySQL provides the ACID compliance and longevity of a traditional RDMS; by caching commonly accessed data solely on memory, memcached provides fast lookup for most transactions. In general, memcached can be used in two different ways: first, where the program tries to retrieve a cached copy from a separate memcached process before resorting to accessing the database; and second, where memcached is integrated into MySQL, invisibly to the client program. This is possible because MySQL versions 5.6 and up come with a daemon plugin that connects the InnoDB storage engine to memcached, bypassing the SQL layer and replicating data behind the scenes. This plugin is not in MariaDB, as it forked from MySQL before 5.6. But it would be very useful if it was. The plugin has a couple advantages over the separate process; for example, the plugin can automatically serialize binary data to store it as text in memcached, making it much less error-prone than manual efforts. As a GSoC student, I would port the InnoDB-memcached plugin from MySQL 5.6/5.7 to MariaDB 10's InnoDB and XtraDB engines. The original code is GPL-licensed, so my job would be to use the existing code but modify it to match the variety of codebase changes that have happened since the MariaDB fork. For example, upon compiling the MySQL plugin code with MariaDB instead, I got a compile error about a missing "binlog.h" in the handler API code for the InnoDB half of the plugin. Apparently, the binary logging code is different for MySQL and MariaDB, and my job would be to convert those references over. Additionally, I will write an XtraDB version of the plugin that, for example, scales well with many memcached nodes. This project would provide an alternative to MDEV-4674, where the writer requests a feature to connect MariaDB to memcached using TCP/UDP, and to MDEV-4675, where the writer requests a feature to detect and address low memory in the memcached server; instead, for example, the plugin could take parameters on a strategy to allocate memcached memory. Deliverables: MariaDB InnoDB+memcached plugin XtraDB+memcached plugin Documentation similar to MySQL's "Getting Started" guides on using the plugin

1 0

[Maria-developers] [GSoC] Student interested in "Optimize mysql-test runs" or adding SQL features
by Pablo Estrada 08 Mar '14

08 Mar '14

Hello! I'm Pablo, from Mexico. Before going into detail, I wanna give a bit of background: I am a master's student, pursuing a degree on Scientific Computing in Korea. My program is 3 years long, and I'm at the middle of the first one. The reason why I think I have skills that will be useful for MariaDB is that before being a master's student, I was working for Oracle during 2 years, committing into the Oracle executable, so I have experience testing, coding and working inside a database executable. Specifically, I was working on the DBMS_SCHEDULER package. Although I am interested in software development, I am more interested in data analysis. That's why I switched from computer science to scientific computing, and that's why the project that interests me the most is the one called "Statistically optimize mysql-test runs by running less tests". I am very excited by the idea of using data to try to test more smartly. If it was not possible to participate in the latter project, I would also be interested to work on adding support for "OR REPLACE, CREATE IF NOT EXISTS and DROP IF EXISTS". Depending on your feedback, I can get to work on a proposal for either one of the two projects. I'd be very happy to be able to participate in GSoC with MariaDB. Regards Pablo

2 2

Re: [Maria-developers] Rev 4004: RBR triggers compiled-out with ifdefs in 10.0
by Sergei Golubchik 07 Mar '14

07 Mar '14

Hi, Sanja! On Feb 12, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-10.0-MDEV-5095/ > > ------------------------------------------------------------ > revno: 4004 > revision-id: sanja(a)askmonty.org-20140212204600-n89hns6zvrut1nl7 > parent: sanja(a)askmonty.org-20140212190858-38n5izb9m3pwz7wc > committer: sanja(a)askmonty.org > branch nick: work-maria-10.0-MDEV-5095 > timestamp: Wed 2014-02-12 22:46:00 +0200 > message: > RBR triggers compiled-out with ifdefs in 10.0 Here're my comments, see below: Note, that I didn't review the implementation of RBR triggers, I looked at ifdefs only. > === modified file 'mysql-test/suite/rpl/disabled.def' > --- mysql-test/suite/rpl/disabled.def 2013-09-27 12:58:49 +0000 > +++ mysql-test/suite/rpl/disabled.def 2014-02-28 10:56:24 +0000 > @@ -14,3 +14,5 @@ rpl_row_create_table : Bug#11759274 > rpl_spec_variables : BUG#11755836 2009-10-27 jasonh rpl_spec_variables fails on PB2 hpux > rpl_get_master_version_and_clock : Bug#11766137 Jan 05 2011 joro Valgrind warnings rpl_get_master_version_and_clock > rpl_partition_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table > +rpl_row_triggers : IFDEF-ed in 10.0 (RBR_TRIGGERS) > +rpl_row_triggers_sbr : IFDEF-ed in 10.0 (RBR_TRIGGERS) better than disabled.def would be to have a file, say --source include/have_rbr_triggers.inc that would do, like if (`select count(*) = 0 from information_schema.session_variables where variable_name = 'slave_run_triggers_for_rbr'`) { skip RBR triggers are not available; } > === modified file 'sql/log_event.cc' > --- sql/log_event.cc 2013-11-13 22:03:48 +0000 > +++ sql/log_event.cc 2014-02-28 10:56:24 +0000 > @@ -9282,10 +9284,31 @@ int Rows_log_event::do_add_row_data(ucha > } > #endif > > -#if !defined(MYSQL_CLIENT) && defined(HAVE_REPLICATION) > +#if defined(MYSQL_SERVER) && defined(HAVE_REPLICATION) why is this change? > + > + > +#ifdef RBR_TRIGGERS > +/** > + Restores empty table list as it was before trigger processing. > + > + @note We have a lot of ASSERTS that check the lists when we close tables. > + There was the same problem with MERGE MYISAM tables and so here we try to > + go the same way. > +*/ > +static void restore_empty_query_table_list(LEX *lex) > +{ > + if (lex->first_not_own_table()) > + (*lex->first_not_own_table()->prev_global)= NULL; > + lex->query_tables= NULL; > + lex->query_tables_last= &lex->query_tables; > +} > +#endif //RBR_TRIGGERS > + > + > int Rows_log_event::do_apply_event(rpl_group_info *rgi) > { > Relay_log_info const *rli= rgi->rli; > + TABLE* table; > DBUG_ENTER("Rows_log_event::do_apply_event(Relay_log_info*)"); > int error= 0; > /* > @@ -9632,9 +9691,15 @@ int Rows_log_event::do_apply_event(rpl_g > */ > thd->reset_current_stmt_binlog_format_row(); > thd->is_slave_error= 1; > - DBUG_RETURN(error); > + /* remove trigger's tables */ > + goto err; hmm, here you've added rgi->slave_close_thread_tables(thd); > } > > +#ifdef RBR_TRIGGERS > + /* remove trigger's tables */ > + if (slave_run_triggers_for_rbr) > + restore_empty_query_table_list(thd->lex); > +#endif //RBR_TRIGGERS > if (get_flags(STMT_END_F) && (error= rows_event_stmt_cleanup(rgi, thd))) > slave_rows_error_report(ERROR_LEVEL, > thd->is_error() ? 0 : error, > @@ -10132,6 +10206,9 @@ Table_map_log_event::Table_map_log_event > DBUG_ASSERT(static_cast<size_t>(cbuf_end - cbuf) <= sizeof(cbuf)); > m_data_size+= (cbuf_end - cbuf) + m_colcnt; // COLCNT and column types > > + if (tbl->triggers) > + m_flags|= TM_BIT_HAS_TRIGGERS_F; why didn't you ifdef this? > + > /* If malloc fails, caught in is_valid() */ > if ((m_memory= (uchar*) my_malloc(m_colcnt, MYF(MY_WME)))) > { > @@ -10538,7 +10616,11 @@ int Table_map_log_event::do_apply_event( > table_list->table_id= DBUG_EVALUATE_IF("inject_tblmap_same_id_maps_diff_table", 0, m_table_id); > table_list->updating= 1; > table_list->required_type= FRMTYPE_TABLE; > - DBUG_PRINT("debug", ("table: %s is mapped to %u", table_list->table_name, table_list->table_id)); > + table_list->master_had_triggers= ((m_flags & TM_BIT_HAS_TRIGGERS_F) ? 1 : 0); why didn't you ifdef this? > + DBUG_PRINT("debug", ("table: %s is mapped to %u%s", > + table_list->table_name, table_list->table_id, > + (table_list->master_had_triggers ? > + " (master had triggers)" : ""))); > enum_tbl_map_status tblmap_status= check_table_map(rgi, table_list); > if (tblmap_status == OK_TO_PROCESS) > { > @@ -10801,6 +10887,10 @@ Write_rows_log_event::do_before_row_oper > from the start. > */ > } > +#ifdef RBR_TRIGGERS > + if (slave_run_triggers_for_rbr && !master_had_triggers && m_table->triggers) > + m_table->prepare_triggers_for_insert_stmt_or_event(); > +#endif //RBR_TRIGGERS But you don't need to ifdef this and many blocks in Rows_log_event::do_apply_event. If you only do #define slave_run_triggers_for_rbr 0 the compiler will remove the whole if() block anyway. > > /* Honor next number column if present */ > m_table->next_number_field= m_table->found_next_number_field; > @@ -10928,6 +11039,13 @@ Rows_log_event::write_row(rpl_group_info > TABLE *table= m_table; // pointer to event's table > int error; > int UNINIT_VAR(keynum); > + > +#ifdef RBR_TRIGGERS > + bool invoke_triggers= > + slave_run_triggers_for_rbr && !master_had_triggers && table->triggers; > +#else > +#define invoke_triggers 0 > +#endif if you have slave_run_triggers_for_rbr defined to 0 then you don't need this idef either. although I'd still declare invoke_triggers as const, to make it easier for the compiler. > auto_afree_ptr<char> key(NULL); > > prepare_record(table, m_width, > @@ -10953,6 +11075,14 @@ Rows_log_event::write_row(rpl_group_info > DBUG_PRINT_BITSET("debug", "read_set = %s", table->read_set); > #endif > > +#ifdef RBR_TRIGGERS > + if (invoke_triggers && > + process_triggers(TRG_EVENT_INSERT, TRG_ACTION_BEFORE, TRUE)) > + { > + DBUG_RETURN(HA_ERR_GENERIC); // in case if error is not set yet > + } > +#endif //RBR_TRIGGERS why ifdef if invoke_triggers is 0 anyway? > + > /* > Try to write record. If a corresponding record already exists in the table, > we try to change it using ha_update_row() if possible. Otherwise we delete > @@ -11079,38 +11209,71 @@ Rows_log_event::write_row(rpl_group_info > !table->file->referenced_by_foreign_key()) > { > DBUG_PRINT("info",("Updating row using ha_update_row()")); > - error=table->file->ha_update_row(table->record[1], > - table->record[0]); > - switch (error) { > - > - case HA_ERR_RECORD_IS_THE_SAME: > - DBUG_PRINT("info",("ignoring HA_ERR_RECORD_IS_THE_SAME error from" > - " ha_update_row()")); > - error= 0; > - > - case 0: > - break; > - > - default: > - DBUG_PRINT("info",("ha_update_row() returns error %d",error)); > - table->file->print_error(error, MYF(0)); > +#ifdef RBR_TRIGGERS why ifdef if invoke_triggers is 0 anyway? > + if (invoke_triggers && > + process_triggers(TRG_EVENT_UPDATE, TRG_ACTION_BEFORE, FALSE)) > + error= HA_ERR_GENERIC; // in case if error is not set yet > + else > +#endif > + { > + error= table->file->ha_update_row(table->record[1], > + table->record[0]); > + switch (error) { > + > + case HA_ERR_RECORD_IS_THE_SAME: > + DBUG_PRINT("info",("ignoring HA_ERR_RECORD_IS_THE_SAME error from" > + " ha_update_row()")); > + error= 0; > + > + case 0: > + break; > + > + default: > + DBUG_PRINT("info",("ha_update_row() returns error %d",error)); > + table->file->print_error(error, MYF(0)); > + } > +#ifdef RBR_TRIGGERS > + if (invoke_triggers && !error && > + (process_triggers(TRG_EVENT_UPDATE, TRG_ACTION_AFTER, TRUE) || > + process_triggers(TRG_EVENT_INSERT, TRG_ACTION_AFTER, TRUE))) > + error= HA_ERR_GENERIC; // in case if error is not set yet > +#endif //RBR_TRIGGERS > } > - > + > DBUG_RETURN(error); > } > else > { > DBUG_PRINT("info",("Deleting offending row and trying to write new one again")); > - if ((error= table->file->ha_delete_row(table->record[1]))) > +#ifdef RBR_TRIGGERS > + if (invoke_triggers && > + process_triggers(TRG_EVENT_DELETE, TRG_ACTION_BEFORE, TRUE)) > + error= HA_ERR_GENERIC; // in case if error is not set yet > + else > +#endif //RBR_TRIGGERS > { > - DBUG_PRINT("info",("ha_delete_row() returns error %d",error)); > - table->file->print_error(error, MYF(0)); > - DBUG_RETURN(error); > + if ((error= table->file->ha_delete_row(table->record[1]))) > + { > + DBUG_PRINT("info",("ha_delete_row() returns error %d",error)); > + table->file->print_error(error, MYF(0)); > + DBUG_RETURN(error); > + } > +#ifdef RBR_TRIGGERS > + if (invoke_triggers && > + process_triggers(TRG_EVENT_DELETE, TRG_ACTION_AFTER, TRUE)) > + DBUG_RETURN(HA_ERR_GENERIC); // in case if error is not set yet > +#endif //RBR_TRIGGERS > } > /* Will retry ha_write_row() with the offending row removed. */ > } > } > > +#ifdef RBR_TRIGGERS > + if (invoke_triggers && > + process_triggers(TRG_EVENT_INSERT, TRG_ACTION_AFTER, TRUE)) > + error= HA_ERR_GENERIC; // in case if error is not set yet > +#endif //RBR_TRIGGERS > + > DBUG_RETURN(error); > } > > === modified file 'sql/sys_vars.cc' > --- sql/sys_vars.cc 2013-11-25 14:49:40 +0000 > +++ sql/sys_vars.cc 2014-02-28 10:56:24 +0000 > @@ -2601,6 +2601,22 @@ static Sys_var_enum Slave_exec_mode( > GLOBAL_VAR(slave_exec_mode_options), CMD_LINE(REQUIRED_ARG), > slave_exec_mode_names, DEFAULT(SLAVE_EXEC_MODE_STRICT)); > > +#ifdef RBR_TRIGGERS > +static const char *slave_run_triggers_for_rbr_names[]= > + {"NO", "YES", "LOGGING", 0}; > +static Sys_var_enum Slave_run_triggers_for_rbr( > + "slave_run_triggers_for_rbr", > + "Modes for how triggers in row-base replication on slave side will be " > + "executed. Legal values are NO (default), YES and LOGGING. NO means " > + "that trigger for RBR will not be running on slave. YES and LOGGING " > + "means that triggers will be running on slave, if there was not " > + "triggers running on the master for the statement. LOGGING also means " > + "results of that the executed triggers work will be written to " > + "the binlog.", > + GLOBAL_VAR(slave_run_triggers_for_rbr), CMD_LINE(REQUIRED_ARG), > + slave_run_triggers_for_rbr_names, > + DEFAULT(SLAVE_RUN_TRIGGERS_FOR_RBR_NO)); > +#endif //RBR_TRIGGERS empty line between variables would be nice > static const char *slave_type_conversions_name[]= {"ALL_LOSSY", "ALL_NON_LOSSY", 0}; > static Sys_var_set Slave_type_conversions( > "slave_type_conversions", > > === modified file 'sql/table.cc' > --- sql/table.cc 2013-11-11 18:46:14 +0000 > +++ sql/table.cc 2014-02-27 19:10:57 +0000 > @@ -6660,6 +6660,81 @@ int TABLE::update_default_fields() > > > /* > + Prepare triggers for INSERT-like statement. > + > + SYNOPSIS > + prepare_triggers_for_insert_stmt_or_event() > + > + NOTE > + Prepare triggers for INSERT-like statement by marking fields > + used by triggers and inform handlers that batching of UPDATE/DELETE > + cannot be done if there are BEFORE UPDATE/DELETE triggers. > +*/ > + > +void TABLE::prepare_triggers_for_insert_stmt_or_event() > +{ > + if (triggers) > + { > + if (triggers->has_triggers(TRG_EVENT_DELETE, > + TRG_ACTION_AFTER)) > + { > + /* > + The table has AFTER DELETE triggers that might access to > + subject table and therefore might need delete to be done > + immediately. So we turn-off the batching. > + */ > + (void) file->extra(HA_EXTRA_DELETE_CANNOT_BATCH); > + } > + if (triggers->has_triggers(TRG_EVENT_UPDATE, > + TRG_ACTION_AFTER)) > + { > + /* > + The table has AFTER UPDATE triggers that might access to subject > + table and therefore might need update to be done immediately. > + So we turn-off the batching. > + */ > + (void) file->extra(HA_EXTRA_UPDATE_CANNOT_BATCH); > + } > + } > +} > + > + > +bool TABLE::prepare_triggers_for_delete_stmt_or_event() > +{ > + if (triggers && > + triggers->has_triggers(TRG_EVENT_DELETE, > + TRG_ACTION_AFTER)) > + { > + /* > + The table has AFTER DELETE triggers that might access to subject table > + and therefore might need delete to be done immediately. So we turn-off > + the batching. > + */ > + (void) file->extra(HA_EXTRA_DELETE_CANNOT_BATCH); > + return TRUE; > + } > + return FALSE; > +} why didn't you put this in the .h file? it's something that should rather be inlined > + > + > +bool TABLE::prepare_triggers_for_update_stmt_or_event() > +{ > + if (triggers && > + triggers->has_triggers(TRG_EVENT_UPDATE, > + TRG_ACTION_AFTER)) > + { > + /* > + The table has AFTER UPDATE triggers that might access to subject > + table and therefore might need update to be done immediately. > + So we turn-off the batching. > + */ > + (void) file->extra(HA_EXTRA_UPDATE_CANNOT_BATCH); > + return TRUE; > + } > + return FALSE; > +} > + > +/* > @brief Reset const_table flag > > @detail > Regards, Sergei

1 0

[Maria-developers] two places maybe need adjusted in rpl_parallel.cc
by nanyi607rao 07 Mar '14

07 Mar '14

hi Kistian, I afraid there are two palces need adjusted in rpl_parallel.cc First is in rpl_parallel_thread::get_rgi() adjusted like this ... else { if(!(rgi= new rpl_group_info(rli))) { my_error(ER_OUTOFMEMORY, MYF(0), (int)sizeof(*rgi)); return NULL; } rgi->is_parallel_exec = true; - if ((rgi->deferred_events_collecting= rli->mi->rpl_filter->is_on())) _ rgi->deferred_events= new Deferred_log_events(rli); } + if ((rgi->deferred_events_collecting= rli->mi->rpl_filter->is_on())) + rgi->deferred_events= new Deferred_log_events(rli); because rgi->deferred_events was freed in rpl_parallel_thread::free_rgi() if it wasn't NULL Second is in rpl_parallel_change_thread_count() I think it need add mysql_cond_broadcast(&pool->COND_rpl_thread_pool) after all new threads are ok, this is for the case that skip_slave_start=OFF,then SQL thread would start automaticlly, it could wait forever in rpl_parallel_thread_pool::get_thread(), because SQL thread may reach this function before another thread reach the rpl_parallel_change_thread_count(). 2014-03-04 nanyi607rao

2 2

Re: [Maria-developers] Replication consistancy in MaxScale
by Kristian Nielsen 07 Mar '14

07 Mar '14

Mark Riddoch <mark.riddoch(a)skysql.com> writes: > We have been looking for a cheap way to find this out and came up with an > idea for MariaDB 10 that might be good to discuss. If we use the global > transaction ID we can get the current master ID and the current GTID form > each slave and use them to compare and discover if a slave is lagging based > on a single connection from a client or the entire server. Clearly we can > get this by jus asking in a polling monitor loop, but what would be really > good would be to have a function in MariaDB that we could call that would > wait for the GTID id to change and return the new GTID. THen we could have a > monitor that makes this blocking call and get's to know the new GTID very > quickly compered to having to have some form of poling loop, which would > limit to the rate of change that could be detected to the polling rate of > the monitor. See here: http://kristiannielsen.livejournal.com/18308.html It's similar to what you describe. MASTER_GTID_WAIT() is already in 10.0 (I can't remember if it is in 10.0.8, else it will be in 10.0.9). Note that if you really want to wait for any change in GTID on a slave, you can just take the current @@gtid_slave_pos, add one to every sequence number, and wait for that. - Kristian.

2 1

[Maria-developers] MariaDB 10.0.9 Release Prep
by Daniel Bartholomew 07 Mar '14

07 Mar '14

All, I've begun preparing for the MariaDB 10.0.9 release. The current plan is to upload the files to the mirrors sometime tomorrow (7 Mar, after they are finished building). I'll probably hold off on actually announcing the release until Monday. Draft changelog and release notes pages are at: - https://mariadb.com/kb/en/mariadb-1009-release-notes/ - https://mariadb.com/kb/en/mariadb-1009-changelog/ Thanks. -- Daniel Bartholomew MariaDB

1 0

Re: [Maria-developers] Rev 4009: MDEV-5675 - Performance: my_hash_sort_bin is called too often in lp:maria/10.0
by Sergei Golubchik 06 Mar '14

06 Mar '14

Hi, Sergey! On Mar 06, Sergey Vojtovich wrote: > revno: 4009 > revision-id: svoj(a)mariadb.org-20140306121912-2a2soan1oc7u1hrt > parent: svoj(a)mariadb.org-20140226085528-80vzk8f2c4b4ugxm > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Thu 2014-03-06 16:19:12 +0400 > message: > MDEV-5675 - Performance: my_hash_sort_bin is called too often > > Reduced number of my_hash_sort_bin() calls from 4 to 1 per query. > Reduced number of memory accesses done by my_hash_sort_bin(). Looks great! Ok to push Regards, Sergei

1 0

[Maria-developers] Standard way for time->datetime cast
by Alexander Barkov 06 Mar '14

06 Mar '14

Hi Sergei, (my mailer failed during the previous attempt. writing again, sorry if you get this twice) Please review the patch implementing the standard (and MySQL-5.6) compatible way of time to datetime conversion. Thanks.

2 3

[Maria-developers] Parallel replication benchmarks
by Kristian Nielsen 06 Mar '14

06 Mar '14

I've done a set of benchmarks for parallel replication on the same machine I used previously for my group commit benchmarks, http://kristiannielsen.livejournal.com/16382.html The code tested is the newest code in the bzr repository and what will be in 10.0.9 (this is significantly improved from what is in 10.0.8). I plan to write up a blog post about it in a couple of days with nice graphs, but meanwhile Axel asked me to summarise in this mail. I tested with sysbench 0.5, using oltp.lua (medium-sized transactions) and update_index.lua (minimal transactions with just a single primary-key update per transaction). I used 10M rows, 16GB buffer pool and 2 * 1.9 GB redo logs. This is with a single table. I tested simply by preparing the binlog on the master, then setting up an already prepared slave and doing START SLAVE UNTIL the end of the log. The error log then shows the time spent for the slave to catch up. I tested everyting in GTID mode, as that is the recommended mode for parallel replication (though my guess is that old-style replication will be much the same, there isn't much difference in the code between what is done to actually execute events). Node that these tests are for in-order parallel replication. All commits on all slaves happen in the same order as on the master; the use of parallel replication is invisible to applications. This is in contrast to eg. MySQL 5.6 multi-threaded slave, which requires the application to partition its data into independent schemas. Here are the prelimiary results, in number of seconds for the slave to catch up (lower is better) versus number of threads (--slave-parallel-threads, 0 means not using parallel replication): For oltp.lua. 48 threads used to generate the load on the master, and --binlog-commit-wait-count=12 --binlog-commit-wait-usec=10000 to allow the server to delay a commit by up to 10 milliseconds in order to get more group commit and thus more opportunities for parallel apply on the slave: A: --log-slave-updates --sync-binlog=1 --innodb-flush-log-at-trx-commit=1 B: --skip-log-slave-updates --innodb-flush-log-at-trx-commit=1 C: --skip-log-bin --innodb-flush-log-at-trx-commit=2 D: --log-bin=master-bin --sync-binlog=0 --innodb-flush-log-at-trx-commit=0 #thr A B C D 0 1065 869 193 202 2 361 432 147 161 4 221 264 118 121 8 135 177 103 107 12 114 153 104 105 16 109 140 104 107 24 111 139 107 105 32 111 136 99 109 48 111 126 108 109 64 111 121 99 111 We see here a 2-10 times speedup from parallel replication. The master has around 12 transactions in every group commit, which provides good opportunities for parallelism on the slave. Note that parallel replication is especially effective when the binlog is enabled and crash-safe (--sync-binlog=1 --innodb-flush-log-at-trx-commit=1). This is because parallel replication can run the commit of one transaction in parallel with any other transaction, even if the two transactions would otherwise conflict. This makes group commit especially effective. In fact, this manages to more or less completely eliminate any penalty for enabling crash-safe binlog on the slave, which is quite nice. Note also that disabling the binlog actually tends to make things _slower_, not faster, when using parallel replication. I believe this is due to MDEV-5802, which may be worth fixing for 10.0. Here are results for update_simple.lua with 48 threads on the master. This produced around 13 transactions per group commit on the master, with no --binlog-commit-wait-count to delay commits: A: --log-slave-updates --sync-binlog=1 --innodb-flush-log-at-trx-commit=1 B: --skip-log-slave-updates --innodb-flush-log-at-trx-commit=1 C: --skip-log-bin --innodb-flush-log-at-trx-commit=2 #thr A B C 0 931 899 271 2 546 653 258 4 365 494 176 8 261 365 203 12 247 350 197 16 233 336 207 24 242 316 209 32 237 292 194 48 235 270 208 64 228 249 195 Again we get a good speedup from parallel replication, even though with such small transactions, there is less opportunity for improvement, as the actual work for transactions is rather small compared to the overhead for managing the replication of each event. And again, the ability to utilise group commit effectively provides the biggest benefit. Finally, I tried a test of update_index.lua where I ran the load on the master single-threadedly. This creates a binlog with _no_ opportunities for parallelism from group commits - each transaction needs to be executed on its own by the slave, as we do not know for sure that they will not conflict on row locks. However, due to the possibility to run the commits in parallel (and hence get group commit on the slave), we still see some speedup even here when --sync-binlog=1 and --innodb-flush-log-at-trx-commit=1. When binlog and innodb sync is disabled, parallel replication makes things slower due to the overhead of thread communication: A: --log-slave-updates --sync-binlog=1 --innodb-flush-log-at-trx-commit=1 B: --skip-log-slave-updates --innodb-flush-log-at-trx-commit=1 C: --skip-log-bin --innodb-flush-log-at-trx-commit=2 #thr A B C 0 1075 949 270 2 597 673 319 4 443 623 334 8 407 588 349 12 393 544 336 16 391 536 352 24 - 492 336 32 389 472 358 48 389 419 344 64 391 399 354 So overall, results look very good, especially for slaves with binlog enabled. (And binlog disabled could turn out better if MDEV-5802 is fixed). Let me know if there are any questions, and I'll be happy to answer them. - Kristian.

1 0

[Maria-developers] A problem of DDL apply in parallel replication
by nanyi607rao 06 Mar '14

06 Mar '14

hi kristian , Maybe you have encountered this problem and fixed it, that is a DDL statement Query event has a Gtid event ahead, but has no end_event (like xid_event, commit or rollback) behind. so in the latest codes of parallel replication, count_queued_event_groups will increase but count_committing_event_groups will not for DDL. it would lead next event_group wait forever I guess. 2014-03-05 nanyi607rao

2 2

Re: [Maria-developers] gsoc replication tasks
by Kristian Nielsen 06 Mar '14

06 Mar '14

Sergei Golubchik <serg(a)mariadb.org> writes: > I'd like to discuss what replication tasks we might want to be > implemented in this year's google summer of code. I don't really have the bandwidth for handling any GSoC at the moment. But here are a bunch of replication tasks that should be suitable for a newcomer, in approximate increasing order of difficulty: MDEV-4776 FR: Consider showing GTID-based 'until' value in slave status MDEV-5119 Events from a GTID-incapable master inherit domain ID of the slave MDEV-4987 Sort by domain_id when list of GTIDs are output MDEV-5289 master server starts slave parallel threads MDEV-4989 Support for GTID in mysqlbinlog MDEV-4990 Implement simple failover script using GTID MDEV-5262 Missing retry after temp error in parallel replication I would be happy to assist anyone that wants to work seriously on one of these, GSoC or not. I'm available on IRC now, and most of the day if you want to discuss more... - Kristian.

1 0

Re: [Maria-developers] [JIRA] (MDEV-5497) Test suite test connect.fix fails
by Alexander Barkov 05 Mar '14

05 Mar '14

Hello Olivier, You changed status to "fixed", but I cannot see any fixes in 10.0-connect. Are you going to push a fix soon? Thanks. On 03/05/2014 03:36 PM, Olivier Bertrand (JIRA) wrote: > > [ https://mariadb.atlassian.net/browse/MDEV-5497?page=com.atlassian.jira.plug… ] > > Olivier Bertrand commented on MDEV-5497: > ---------------------------------------- > > Sorry Elena, I did not read your comment entirely before answering. Your analyze was quite good, just missing the reason why this was happening. > >> Test suite test connect.fix fails >> --------------------------------- >> >> Key: MDEV-5497 >> URL: https://mariadb.atlassian.net/browse/MDEV-5497 >> Project: MariaDB Development >> Issue Type: Bug >> Affects Versions: 10.0.7, 10.0.8 >> Environment: Slackware Linux 14.1 + kernel 3.12.6 >> Reporter: Duncan Roe >> Assignee: Olivier Bertrand >> Priority: Minor >> Fix For: 10.0.9 >> >> Time Spent: 1 hour >> Remaining Estimate: 0 minutes >> >> {noformat} >> cd /usr/mysql-test; ./mtr connect.fix >> {noformat} >> (as root) gives me an error report as below. I am keeping the mysql-test directory, so please let me know if you'd like anything else from it. >> {noformat} >> Logging: ./mtr connect.fix >> vardir: /usr/mysql-test/var >> Checking leftover processes... >> Removing old var directory... >> Creating var directory '/usr/mysql-test/var'... >> Checking supported features... >> MariaDB Version 10.0.7-MariaDB >> - SSL connections supported >> Collecting tests... >> Installing system database... >> Using server port 43317 >> ============================================================================== >> TEST RESULT TIME (ms) or COMMENT >> -------------------------------------------------------------------------- >> worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 >> worker[1] mysql-test-run: WARNING: running this script as _root_ will cause some tests to be skipped >> connect.fix [ fail ] >> Test ended at 2014-01-01 19:32:13 >> CURRENT_TEST: connect.fix >> --- /usr/mysql-test/plugin/connect/connect/r/fix.result 2013-12-23 20:35:03.000000000 +1100 >> +++ /usr/mysql-test/plugin/connect/connect/r/fix.reject 2014-01-01 19:32:13.000000000 +1100 >> @@ -80,7 +80,7 @@ >> name city birth hired >> John Boston 2 1986-01-05 2010-06-02 >> Henry Boston 0 1987-06-07 2008-04-01 >> -George San Jose 1 1981-08-01 2010-06-02 >> +George San Jose 1 1981-07-31 2010-06-02 >> Sam Chicago 2 1979-11-02 2007-10-10 >> James Dallas 1 1992-05-03 2009-12-14 >> Bill Boston 1 1986-09-01 2008-02-10 >> @@ -96,7 +96,7 @@ >> name city birth hired >> John Boston 2 1986-01-05 2010-06-02 >> Henry Boston 0 1987-06-07 2008-04-01 >> -George San Jose 1 1981-08-01 2010-06-02 >> +George San Jose 1 1981-07-31 2010-06-02 >> Sam Chicago 2 1979-11-02 2007-10-10 >> James Dallas 1 1992-05-03 2009-12-14 >> Bill Boston 1 1986-09-01 2008-02-10 >> @@ -112,7 +112,7 @@ >> name city birth hired >> John Boston 2 1986-01-05 2010-06-02 >> Henry Boston 0 1987-06-07 2008-04-01 >> -George San Jose 1 1981-08-01 2010-06-02 >> +George San Jose 1 1981-07-31 2010-06-02 >> Sam Chicago 2 1979-11-02 2007-10-10 >> James Dallas 1 1992-05-03 2009-12-14 >> Bill Boston 1 1986-09-01 2008-02-10 >> @@ -128,7 +128,7 @@ >> name city birth hired >> John Boston 2 1986-01-05 2010-06-02 >> Henry Boston 0 1987-06-07 2008-04-01 >> -George San Jose 1 1981-08-01 2010-06-02 >> +George San Jose 1 1981-07-31 2010-06-02 >> Sam Chicago 2 1979-11-02 2007-10-10 >> James Dallas 1 1992-05-03 2009-12-14 >> Bill Boston 1 1986-09-01 2008-02-10 >> mysqltest: Result content mismatch >> - saving '/usr/mysql-test/var/log/connect.fix/' to '/usr/mysql-test/var/log/connect.fix/' >> -------------------------------------------------------------------------- >> The servers were restarted 0 times >> Spent 0.000 of 6 seconds executing testcases >> Failure: Failed 1/1 tests, 0.00% were successful. >> Failing test(s): connect.fix >> The log files in var/log may give you some hint of what went wrong. >> If you want to report this error, please read first the documentation >> at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html >> mysql-test-run: *** ERROR: there were failing test cases >> {noformat} > > > > -- > This message was sent by Atlassian JIRA > (v6.2-OD-10-004-WN#6253) >

2 1

Re: [Maria-developers] Active Directory authentication plugin - GSoC 2014
by Sergei Golubchik 05 Mar '14

05 Mar '14

Hi, Talina! On Feb 28, talinashrotriya(a)gmail.com wrote: > Hello Sir, > > My name is Talina, presently studying Computer Science from India. > > I am interested to work on the “Active Directory authentication plugin”project. I am good at C/C++ , MYSQL and also have good knowledge about Active directory. Could you suggest me a few links to get started with? > Eager to start work. > > Thanks! We don't have this project on the https://mariadb.com/kb/en/google-summer-of-code-2014/ anymore. I will add it back when we have a mentor for it. Regards, Sergei

1 0

[Maria-developers] MariaDB & MySQL community event 3rd of April 2014 in Santa Clara
by Michael Widenius 05 Mar '14

05 Mar '14

Hi! I am happy to announce that the MariaDB Foundation is organising a MariaDB & MySQL community event in Santa Clara on Thursday the 3rd of April. The venue is the Hilton Santa Clara hotel, a short walk from the Percona Live 2014 event. You can find more information on my blog at: http://monty-says.blogspot.fi/2014/03/mariadb-mysql-community-event-2014-in… We already got some very interesting talks and speakers, but I would like to see even more talk proposals. It's ok to propose a talk that it's also hold at Percona Live. As this event is free, I expect a totally different group of attendees, so that should be fine! When we have got a nice list of proposed talks, we will post them here so that everyone has a chance of commenting of which talks should be accepted. (Sorry, but this time we don't have time to create a proper selection committee so we have to do this a bit rough) Personally I will talk about MariaDB 10.0, which should be declared stable by then. Hope to see many MariaDB developers and users in Santa Clara! You can sign up for the event and/or dinner at: http://www.eventbrite.com/e/mariadb-mysql-community-event-2014-tickets-1080… Regards, Monty PS: Feel free to discuss the event on this email list! We want to have as many as possible participating in making this the best possible MariaDB and MySQL event.

1 0

[Maria-developers] regarding the project in gsoc 2014
by nitin goel 05 Mar '14

05 Mar '14

Sir/Ma'am, I would like to bring to your notice that i used 'mysql' to drop and create tables to understand project better but would like to know more specifications for the project 'create or replace, create if not exists, and drop if exists'. I have been working in C/C++ since 2010 and would like to learn more from you. Please also mention how to contact the mentor for the project and proceed further on project. Thanks Nitin Goel

2 1

[Maria-developers] MariaDB slave_parallel_threads and MySQL slave_parallel_workers
by Rasmus Johansson 05 Mar '14

05 Mar '14

Hi, I received a question that I wanted to share here to get some insight on. MariaDB 10.0 has a global setting named "slave_parallel_threads" and MySQL 5.7 has the same parameter named "slave_parallel_workers". Should slave_parallel_workers be an alias to slave_parallel_threads in MariaDB 10.0 to be more compatible with MySQL? BR, Rasmus -- Rasmus Johansson, VP Engineering MariaDB | t: +358 50 499 9589 | Skype: ratzpo

2 1

[Maria-developers] [GSoC] memcached priorities
by Tan Tran 04 Mar '14

04 Mar '14

Hi all, I'm looking to do my GSoC project with MariaDB, specifically the suggestion to "Port the InnoDB memcached interface to MariaDB" on the page https://mariadb.com/kb/en/google-summer-of-code-2014/ . But the project description links to a JIRA bug at https://mariadb.atlassian.net/browse/MDEV-4674 , where the author suggests that we support memcached through TCP/UDP instead, with no mention of the InnoDB plugin. I figure that the InnoDB plugin is more important, since it's a MySQL 5.6 feature, it's widespread in industry, and the lack of a memcache interface is one of only a handful X's for MariaDB on http://www.skysql.com/blogs/max-mether/mysql-56-vs-mariadb-100 . Additionally, even without TCP/UDP support, one could just code the memcache logic manually anyway, whereas the InnoDB integration is more sophisticated and feature-rich and would be a good selling point for SkySQL. So I'd like to start working on a proposal. Can any devs weigh in on which aspect of memcache is more desirable for MariaDB? If it sounds likes a good, straightforward GSoC project that nobody else has claimed yet? Thanks, Tan Tran

1 0

Re: [Maria-developers] Rev 4006: MDEV-5674 - Performance: my_hash_sort_bin is called too often
by Sergei Golubchik 04 Mar '14

04 Mar '14

Hi, Sergey! On Feb 14, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 4006 > revision-id: svoj(a)mariadb.org-20140214122541-tenvcllk760deai8 > parent: svoj(a)mariadb.org-20140213071355-psk0jpsarx2u7myc > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Fri 2014-02-14 16:25:41 +0400 > message: > MDEV-5674 - Performance: my_hash_sort_bin is called too often > > - reduced number of my_hash_sort_bin() calls from 4 to 2 per query. > Let MDL subsystem use pre-calculated hash value for hash > inserts and deletes. Is this necessary? One of the reasons why I didn't review this your first patch, because I believed that the second one obsoletes it. Why do you need my_hash_insert_using_hash_value() when you achieve the same effect with your calc_hash callback in the second patch? > - reduced number of memory accesses done by my_hash_sort_bin() I'd expect the compiler to handle it automatically, but feel free to change it, if it doesn't. Regards, Sergei

2 1

Re: [Maria-developers] MDEV-5674 - Performance: my_hash_sort_bin is called too often
by Sergei Golubchik 04 Mar '14

04 Mar '14

Hi, Sergey! On Feb 20, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 4007 > revision-id: svoj(a)mariadb.org-20140220072533-ko14y1714ouvd2f7 > parent: svoj(a)mariadb.org-20140214122541-tenvcllk760deai8 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Thu 2014-02-20 11:25:33 +0400 > message: > MDEV-5674 - Performance: my_hash_sort_bin is called too often It's MDEV-5675, not MDEV-5674 > Reuse MDL hash value in table cache: > - MDL namespace is excluded from hash value calculation, so that > hash value can be used by table cache as is > - hash value for MDL is calculated as resulting hash value + MDL > namespace > - extended hash implementation to accept user defined hash function This looks ok. I have a couple of comments, but they are mostly a question of taste. Up to you, you can push as is too. > === modified file 'sql/mdl.cc' > --- a/sql/mdl.cc 2014-02-14 12:25:41 +0000 > +++ b/sql/mdl.cc 2014-02-20 07:25:33 +0000 > @@ -760,13 +760,22 @@ void MDL_map::init() > } > > > +my_hash_value_type mdl_hash_function(const CHARSET_INFO *cs, > + const uchar *key, size_t length) > +{ > + return *((my_hash_value_type*) (key - offsetof(MDL_key, m_ptr) + > + offsetof(MDL_key, m_hash_value))) + > + key[0]; I'd prefer to avoid pointer and offset juggling. Because you use a custom hash function anyway, you can directly pass MDL_key as your "key", then here you can use ((MDL_key*)key)->hash_value() > +} > + > /** Initialize the partition in the container with all MDL locks. */ > > === modified file 'sql/mdl.h' > --- a/sql/mdl.h 2014-02-14 12:25:41 +0000 > +++ b/sql/mdl.h 2014-02-20 07:25:33 +0000 > @@ -397,7 +396,11 @@ class MDL_key > { > return & m_namespace_to_wait_state_name[(int)mdl_namespace()]; > } > - ulong hash_value() const > + my_hash_value_type hash_value() const > + { > + return m_hash_value + m_ptr[0]; I'm not sure it's a good idea, + is not a very good hash function :) On the other hand, I don't see anything immediately wrong with it in this particular case. Ok, let it be. Anyway, could you please use mdl_namespace() instead of m_ptr[0] where possible? > @@ -405,12 +408,14 @@ class MDL_key > private: > MDL_key(const MDL_key &); /* not implemented */ > MDL_key &operator=(const MDL_key &); /* not implemented */ > + friend my_hash_value_type mdl_hash_function(const CHARSET_INFO *, > + const uchar *, size_t); You probably wouldn't need this friend, if you'd change mdl_hash_function() to use MDL_key::hash_value() Regards, Sergei

2 5

[Maria-developers] Suggestion ,a new member
by anupama vijjapu 04 Mar '14

04 Mar '14

Hello fellow developers, I joined MariaDB community today and need few suggestions/projects/bugs that i can work on. I do not have any knowledge of MariaDB source, but I will start looking into it now. I have hands on experience in C/C++/Java/Python and MySQL Database. Any suggestions/recommendation are highly appreciated.

2 1

Re: [Maria-developers] Rev 4029: MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client
by Sergei Golubchik 03 Mar '14

03 Mar '14

Hi, Holyfoot! On Jan 31, holyfoot(a)askmonty.org wrote: > revno: 4029 > revision-id: holyfoot(a)askmonty.org-20140131103303-1rx7ielo83f8iahe > parent: holyfoot(a)askmonty.org-20140124020722-dd5twtwlcc8o1xiy > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-4856 > timestamp: Fri 2014-01-31 14:33:03 +0400 > message: > MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client. > The fill_schema_table() function used to call get_table_share() for > a table name in WHERE then clear the error list. That way plugins > receive the superfluous error notification if it happens in it. Also > the problem was that error handler didn't prevent the suppressed > error message from logging anyway as the logging happens in > THD::raise_condition before the handler call. > Fixed by adding the No_table_error_handler before the call. > raise_condition() also fixed. > === modified file 'sql/sql_class.cc' > --- a/sql/sql_class.cc 2014-01-23 18:21:02 +0000 > +++ b/sql/sql_class.cc 2014-01-31 10:33:03 +0000 > @@ -1145,7 +1145,6 @@ MYSQL_ERROR* THD::raise_condition(uint s > got_warning= 1; > break; > case MYSQL_ERROR::WARN_LEVEL_ERROR: > - mysql_audit_general(this, MYSQL_AUDIT_GENERAL_ERROR, sql_errno, msg); > break; > default: > DBUG_ASSERT(FALSE); > @@ -1156,6 +1155,8 @@ MYSQL_ERROR* THD::raise_condition(uint s > > if (level == MYSQL_ERROR::WARN_LEVEL_ERROR) > { > + mysql_audit_general(this, MYSQL_AUDIT_GENERAL_ERROR, sql_errno, msg); > + This is good. > === modified file 'sql/sql_show.cc' > --- a/sql/sql_show.cc 2013-11-19 12:16:25 +0000 > +++ b/sql/sql_show.cc 2014-01-31 10:33:03 +0000 > @@ -4199,8 +4199,13 @@ static int fill_schema_table_from_frm(TH > key_length= create_table_def_key(thd, key, &table_list, 0); > hash_value= my_calc_hash(&table_def_cache, (uchar*) key, key_length); > mysql_mutex_lock(&LOCK_open); > - share= get_table_share(thd, &table_list, key, > - key_length, OPEN_VIEW, &not_used, hash_value); > + { > + No_such_table_error_handler no_such_table_handler; > + thd->push_internal_handler(&no_such_table_handler); > + share= get_table_share(thd, &table_list, key, > + key_length, OPEN_VIEW, &not_used, hash_value); > + thd->pop_internal_handler(); > + } But I don't understand that. Do you mean, old code did *not* suppress errors here? How could selects from I_S tables work without it? Regards, Sergei

2 5

[Maria-developers] New member, Need suggestion
by Neil Shah 03 Mar '14

03 Mar '14

Hello fellow developers, I joined MariaDB community today and need few suggestions/projects/bugs that i can work on. I do not have any knowledge of MariaDB source, but I will start looking into it now. I have hands on experience in C/C++/Java/Python and MySQL Database. Any suggestions/recommendation are highly appreciated. -- Thanks & Regards Neil Shah

2 3

[Maria-developers] slave_ddl_exec_mode and incompatible change in MariaDB 10.0.8
by Pavel Ivanov 03 Mar '14

03 Mar '14

Hi, I've discovered a change in MariaDB 10.0.8 that I don't quite understand motivation for and which looks really surprising to me. So I would appreciate if you could tell me what's the motivation and why you think it's appropriate to do that. The change in question is http://bazaar.launchpad.net/~maria-captains/maria/10.0/revision/3961.1.7. I see several issues with it: 1. It introduces a pretty significant variable slave_ddl_exec_mode which wasn't announced in the Release Notes. 2. The default variable value was set to IDEMPOTENT which changes MariaDB slave behavior in an incompatible way. Not everyone will want to have this variable set to IDEMPOTENT (I'd actually argue that no one should set it to IDEMPOTENT), but without any mentioning of this variable in the Release Notes people basically have the only chance to get knowledge about this change (let me stress out: _incompatible_ change) from production incidents or from accidental code inspection. 3. When slave_ddl_exec_mode is set to IDEMPOTENT every "DROP TABLE" event in master's binlog is converted into "DROP TABLE IF EXISTS" in slave's binlog, which I believe is a major no-no for replication. Binlogs on master and slave should be identical, otherwise after failover even if one would set slave_ddl_exec_mode = STRICT it will still execute "DROP TABLE IF EXISTS" because new master has that in binlogs. 4. Why this change introduces OPTION_GTID_BEGIN in the code? Why OPTION_BEGIN wasn't enough? From reading the code the only thing I understand is that with broken binlogs and sequence of events "GTID BEGIN; CREATE TABLE; CREATE TABLE" MariaDB will try to execute both CREATE TABLE statements in the same transaction without auto-committing at the end. This sounds pretty crazy to me. And given the fact that in normal situation such sequence of events on the master is not possible it seems that existence of both OPTION_BEGIN and OPTION_GTID_BEGIN only introduces confusion (MariaDB have GTID always on, right? So it should always execute GTID_BEGIN and never BEGIN without GTID, right?). So what is your word on this? Can you at least set default value for slave_ddl_exec_mode to STRICT so that slave behavior didn't change with the version upgrade? Thank you, Pavel

2 7

Re: [Maria-developers] MariaDB Master Replication To MySQL
by Kristian Nielsen 03 Mar '14

03 Mar '14

> In this particular case, we have someone that wants to use MariaDB 10.0 master with a MySQL 5.6 slave. But they are running into a problem with GTID. Generally, the recommended setup is to only replicate from old version on master to new (or same) version on slave. Replicating from new master to old slave in general is hard to support 100%, as there may be eg. new SQL syntax on the master that is impossible for the old version slave to execute. That being said, the intention is that replication from new master to old slave (like MariaDB 10.0 -> MySQL 5.6) should work in most reasonable cases. So this sounds like a bug which might be worth filing. > 2.) Is it possible to disable GTID in 10.0? It seems like it would break a few other things like parallel replication, but they're only running MariaDB on the master. GTID is disabled automatically when the slave does not support it. The code rewrites the binlog stream on the fly to remove events that are not supported on the old slave, and replace them with something appropriate for that slave version. Does the same problem occur when replicating from MariaDB 10.0 master to MariaDB 5.5 slave? What about when replicating from MySQL 5.5 master to MySQL 5.6 slave? - Kristian.

1 0

Re: [Maria-developers] MDEV-5755 - Scalability: let LOCK_open and THR_LOCK::mutex be normal mutexes
by Sergei Golubchik 03 Mar '14

03 Mar '14

Hi, Sergey! On Feb 27, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 4011 > revision-id: svoj(a)mariadb.org-20140227141521-fgj5b0h3qbijyojh > parent: svoj(a)mariadb.org-20140220072533-l3au3y7mrsyclfpb > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Thu 2014-02-27 18:15:21 +0400 > message: > MDEV-5755 - Scalability: let LOCK_open and THR_LOCK::mutex be normal mutexes > > Normal mutexes scale better than adaptive at high thread count. > At low thread count they behave similarly. To summarize Axel results: at low concurency normal can be ~1% slower. at high concurrency r/o - normal are up to 25% faster. at high concurrency r/w - normal are up to 10% slower. That last result bothers me a litte. Regards, Sergei

2 1

[Maria-developers] MDEV-5081 - Simple performance improvement for MariaDB
by Sergey Vojtovich 28 Feb '14

28 Feb '14

Hi Sergei, do you know if we ship any binaries with fast mutex (-DMY_PTHREAD_FASTMUTEX) enabled or officially suggest it? What was it's intention and what loads it was supposed to make faster? I'm trying to understand if fastmutex is worth fixing. It looks like duplication of PTHREAD_MUTEX_ADAPTIVE_NP, so we have double spinning when they're enabled. According to my recent benchmarks it shown worst performance. Thanks, Sergey

3 5

[Maria-developers] Keystone authentication plugin GSoc 14
by Abhishek Solanki 28 Feb '14

28 Feb '14

Hello MariaDB members , I am interested to work on "Keystone authentication plugin" project , i have a good knowledge of C/C++ and OpenStack. So can anyone help me how to get started with this project, so that i cold dig something before applying for this project. -- Thanks, Abhishek

2 1

[Maria-developers] Debian build / multi-architecture related bugs filed
by Otto Kekäläinen 27 Feb '14

27 Feb '14

Hello! The Debian build status can be seen at https://buildd.debian.org/status/package.php?p=mariadb-5.5 (and for MySQL at https://buildd.debian.org/status/package.php?p=mysql-5.5) The sparc build issue already got solved but for the others I filed the following bugs (as each failure seems to have different cause): https://mariadb.atlassian.net/browse/MDEV-5708 https://mariadb.atlassian.net/browse/MDEV-5707 https://mariadb.atlassian.net/browse/MDEV-5706 Could somebody please look into these? I assume that some of them might be easy if MySQL already has them solved, then all that needs to be done is to copy a patch from MySQL upstream or from MySQL patches in Debian (see http://anonscm.debian.org/gitweb/?p=pkg-mysql/mysql-5.5.git;a=tree;f=debian…) (MariaDB Debian builds are pulled from http://anonscm.debian.org/gitweb/?p=pkg-mysql/mariadb-5.5.git and mirrored at https://github.com/ottok/mariadb-5.5) Also, here are two QA issues that are related. I am not sure what to do with them: http://qa.debian.org/bls/packages/m/mariadb-5.5.html

1 1

[Maria-developers] Problem with debug_sync, THD::enter_cond(), and kill ...
by Kristian Nielsen 26 Feb '14

26 Feb '14

I got a crash in my parallel replication test case, and this one turned out to be caused by a fundamental problem in debug_sync. I am not sure how to solve it, so I wanted to explain the issue in case others have some ideas. The crash happens in THD::awake() when a thread is killed: mysql_mutex_lock(&mysys_var->mutex); ... int ret= mysql_mutex_trylock(mysys_var->current_mutex); mysql_cond_broadcast(mysys_var->current_cond); if (!ret) mysql_mutex_unlock(mysys_var->current_mutex); The problem is that mysys_var->current_mutex changed between the trylock() and the unlock(), so we unlock a different mutex than the one we locked - ouch! The mutex changed because of this code in debug_sync.cc: if (thd->mysys_var) { old_mutex= thd->mysys_var->current_mutex; old_cond= thd->mysys_var->current_cond; thd->mysys_var->current_mutex= &debug_sync_global.ds_mutex; thd->mysys_var->current_cond= &debug_sync_global.ds_cond; } There is no mutex protection here. So this means that it is not safe to use debug_sync inside enter_cond()/exit_cond() if kill will be used on the thread at that point. Since I have a lot of tests for parallel replication where I test exactly for correct error handling when killing threads at various points in the parallel processing, it is not too surprising that I would eventually be hit by this :) I am going to avoid using debug_sync inside enter_cond()/exit_cond() for now. But any ideas for how to solve this properly? This is particularly nasty to be hit by, as it is not at all obvious that this use of debug_sync should be a problem. And it is not exactly easy to guess from the failure what the real problem is - assuming one can even reproduce the failure, the window of opportunity in the race is after all rather quite small. - Kristian.

3 4

[Maria-developers] How handle_rpl_parallel_thread rollback partial transaction when STOP SLAVE In parallel replication
by nanyi607rao 26 Feb '14

26 Feb '14

hi Kristian, I have a quesion about parallel replication. In handle_rpl_parallel_thread(), a worker thread has got a whole transaction events, but it only apply partial events. this thread wouldl skip left events when do STOP SLAVE(is that rigth?) ,because sql_worker_killed() return true. but it seems that partial transaction won't be rollbacked for wait_for_prior_commit() always return false. do me wrong? or how would it rollback that partial transaction. 2014-02-25 nanyi607rao

2 3

[Maria-developers] GSOC 2014 participation
by Suyash Loiwal 26 Feb '14

26 Feb '14

Hello I am Suyash Loiwal from India, currently pursuing my B.Tech from Manipal institute of Technology, Manipal. I would like to take a project under your organization. Can someone please guide me how to start? Thanks Suyash

2 1

[Maria-developers] MDEV-4309: Review for 201402181003_sanja_copy_calls_patch.diff
by Sergey Petrunia 25 Feb '14

25 Feb '14

Hi Sanja, My input for 201402181003_sanja_copy_calls_patch.diff: * memcpy_field_possible() should have a function comment mentioning that it checks whether memcpy_field_value() can be used. * field_conv_incompatible() should have a function comment mentioning that it is a function that is used instead of field_conv() when we know in advance that memcpy_field_possible(...)==false. Ok to push after this is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB 5.5.36 stable and MariaDB Galera 10.0.7 alpha release prep
by Daniel Bartholomew 25 Feb '14

25 Feb '14

All, I've completed most of the prep work for the MariaDB 5.5.36 (stable) and MariaDB Galera Cluster 10.0.7 (alpha) releases. Current plan is to activate and announce these releases together on Monday, 24 Feb before noon US Eastern Time (barring any last-minute issues). Draft changelogs and release notes are at the following locations: MariaDB 5.5.36 stable - https://mariadb.com/kb/en/mariadb-5536-release-notes/ - https://mariadb.com/kb/en/mariadb-5536-changelog/ MariaDB Galera Cluster 10.0.7 alpha - https://mariadb.com/kb/en/mariadb-galera-cluster-1007-release-notes/ - https://mariadb.com/kb/en/mariadb-galera-cluster-1007-changelog/ Please let me know ASAP of any additions, changes, typos, etc.... The 5.5.36 release notes in particular are very generic at the moment, so I'm very interested in additions to that page (please jump in and add!). The files for these releases are gradually making their way to MariaDB mirrors world wide. The primary mirror has them now if you want to get a jump on the competition: http://ftp.osuosl.org/pub/mariadb/mariadb-galera-10.0.7 http://ftp.osuosl.org/pub/mariadb/mariadb-5.5.36 Obilgatory note #1: The plan is for these files to be the officially released files without any changes, but that may change if something unexpected comes up (and such things have happened in the past). For those that would like to test the MariaDB Galera Cluster 10.0.7-alpha yum and apt repositories prior to the actual release, because that's just how you roll, the following config entries should get you going (you'll need to copy/paste/edit for whatever distro you are actually using/testing/playing-around-with): # - - Ubuntu - - - - - - - - - - - - - - - - # MariaDB 10.0.8 Ubuntu test mariadb.list w/MGC 10.0.7-alpha deb http://ftp.osuosl.org/pub/mariadb/mariadb-10.0.8/repo/ubuntu saucy main # - - Debian - - - - - - - - - - - - - - - - - # MariaDB 10.0.8 Debian test mariadb.list w/MGC 10.0.7-alpha deb http://ftp.osuosl.org/pub/mariadb/mariadb-10.0.8/repo/debian wheezy main # - - Red Hat - - - - - - - - - - - - - - - - # MariaDB 10.0.8 RedHat MariaDB-test.repo w/ MGC 10.0.7-alpha [mariadb] name = MariaDB baseurl = http://yum.mariadb.org/10.0.8-upd/rhel6-amd64 gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB gpgcheck=1 # - - Fedora - - - - - - - - - - - - - - - - - # MariaDB 10.0.8 Fedora MariaDB-test.repo w/ MGC 10.0.7-alpha [mariadb] name = MariaDB baseurl = http://yum.mariadb.org/10.0.8-upd/fedora20-amd64 gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB gpgcheck=1 Obligatory note #2: These test alpha repos, if you use them, should be switched to the official ones (as produced by the repository configuration generator) after the release is officially out. Obligatory disclaimer #1: These test alpha repos are _completely_ untested. They may not work at all. Obligatory disclaimer #2: MariaDB Galera Cluster 10.0.7 is an ALPHA release. You have been warned. You're on your own. Good luck and Godspeed. :) Thanks. -- Daniel Bartholomew MariaDB

4 4

[Maria-developers] MySQL Optimizer Trace Questions
by Christian Rober 25 Feb '14

25 Feb '14

Hello All, I work for TokuTek as a Storage Engine developer and I have been using the, relatively, new optimizer trace functionality. I have found the results very helpful when combined with analyzing the source code, but there a few questions I want to ask. Here is some output from the trace for a recent SELECT query on an InnoDB table: At the end of the trace, it shows the index chosen by, what I presume, is the optimizer: ... "considered_execution_plans": [ { "plan_prefix": [], "table": "`reports` `a`", "best_access_path": { "considered_access_paths": [ { "access_type": "ref", "index": "reports_group_id", "rows": 1.55e6, "cost": 411057, "chosen": true }, ... ] }, "cost_for_plan": 411057, "rows_for_plan": 1.55e6, "chosen": true } ] }, ... Here are the cost results for some rejected alternative indexes, (note: the row count for two of them is similar to the chosen index): ... "analyzing_range_alternatives": { "range_scan_alternatives" ... "rows": 2377102, "cost": 2.81e6, ... "rows": 1554164, "cost": 1.84e6, ... "rows": 2346664, "cost": 2.78e6, ... "rows": 1554164, "cost": 1.84e6, ... Here is the cost analysis for a table scan, from the beginning of the trace (still not as good as the above chosen index, but close on cost): ... "table": "`reports` `a`", "range_analysis": { "table_scan": { "rows": 3108329, "cost": 655076 }, ... Related to the fact that the table scan analysis is dominated by the large row count, and probably rejected as a valid plan compared to the chosen index for that very reason: 1. How is the row count more than merely informational to the optimizer? I had the impression the row count did not directly contribute to the optimizer's decision as much as the estimated cost, or that the row count was already accounted for/factored into in the final cost measurement. 2. Related to that question, what is cost exactly? Is there a unit of measurement (ex: Time, Disk Seeks, etc.)? 3. Is cost more important than row count or equally important? 4. What is the genesis of the cost variable displayed in the trace? I assume it is just a guess on the respective storage engine's part of how expensive (in terms of disk access, processing, time, etc.) getting the rows will be for the query. thanks in advance, Christian

2 1

[Maria-developers] We are participating in Google Summer of Code 2014
by Colin Charles 25 Feb '14

25 Feb '14

Hi, It is with great pleasure that I can inform you that we have been accepted as an organisation to participate in Google Summer of Code 2014! It would be great to see more ideas pop up here: https://mariadb.com/kb/en/google-summer-of-code-2014/ More mentors are always wanted! I presume we will see a lot more student discussions here as well. If you have any questions that aren't answered on the lists, please send me a personal note Here's to more features & contributors in 2014. cheers, -colin -- Colin Charles, Chief Evangelist MariaDB | t: +6-012-204-3201 | Skype: colincharles

1 0

[Maria-developers] MariaDB C-API and specific P_S tables
by Peter Laursen 24 Feb '14

24 Feb '14

THis is probably a question that Gerog Richter (or possibly Wlad) could reply to. I came across this bug report: http://bugs.mysql.com/bug.php?id=71801 .. files by former MySQL/Sun/Oracle supporter Veleriuy Kravtchuk We have a 64 bit C-application compiled with MariaDB C-API (on and for Windows 64 bit). I connect to MySQL 5.6.15 and execute SELECT * FROM `session_connect_attrs`; /* PROCESSLIST_ID ATTR_NAME ATTR_VALUE ORDINAL_POSITION -------------- --------------- ---------- ------------------ 2 _os Windows 0 2 _client_name libmariadb 1 2 _pid 5012 2 2 _thread 4640 3 2 _platform x86 4 2 _client_version 1.1.2 5 */ With the MySQL CLI 5.6.15, you will get /* PROCESSLIST_ID ATTR_NAME ATTR_VALUE ORDINAL_POSITION -------------- --------------- ---------- ------------------ 4 _os Win64 0 4 _client_name libmysql 1 4 _pid 4888 2 4 _thread 4320 3 4 _platform x86_64 4 4 program_name mysql 5 4 _client_version 5.6.15 6 */ The differences I notice are * Our application populates P_S with "Windows" - not "Win64" as the 5.6.15 CLI * Our application populates 'P_S with "x86" and not "x86_64". * Our application does not populate P_S with a "program_name". But it has a name of course! MySQL 5.6 docs at http://dev.mysql.com/doc/refman/5.6/en/performance-schema-connection-attrib… (as far as I can understand) that the client should use mysql_options() and mysql_options4() C API functions to populate P_S with all information. I have two questions: 1) Will the first 2 differences listed require a patch in our code or in libmariadb? 2) Obviosuly the lacking program name in the P_S tables is something that we will neeed to take care of ourselves. However I think that mysql_options4() was added in MySQL 5.6 and the MariaDB API does not supprot it currently (it is targeted to be compatible with Oracle 5.5 API, I think). And to populate the program name we may need to use mysql_options4(). Am I right?

2 4

[Maria-developers] Using MariaDB SQL for other storage engine
by Amith Nagarajappa 24 Feb '14

24 Feb '14

Hello, I am a database guy and want to plug mariaDB sql engine for other storage engine. If I want to plugin a storage engine with mariadb to use mariadb's SQL should i use handler interface in the storage/examples directory. I am not sure how to start with this interface implementation for MariaDB. Any help or docs or weblink on this would be helpful. Thanks and Regards, Amith

3 4

[Maria-developers] GROUP BY with non-grouped column returns NULL in some cases?
by Pavel Ivanov 24 Feb '14

24 Feb '14

Hi, I've noticed a difference in MariaDB's behavior compared to its documentation and compared to MySQL 5.1 (didn't check with MySQL 5.5 or 5.6 yet), and I wanted to know what's your opinion -- is it a bug or a conscious difference and documentation is just lagging. This doc https://mariadb.com/kb/en/select/#group-by says: "If you select a non-grouped column or a value computed from a non-grouped column, it is undefined which row the returned value is taken from". I believe this implies that selected value must be returned from a row group, not just something random. But I have a scenario when NULL is returned instead of real values. I understand that this is completely undefined behavior in SQL standard, but MySQL and MariaDB seem to have a documented support for this, that's why I'm asking. Here's the mysql session log: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 5 Server version: 10.0.8-MariaDB-log Source distribution Copyright (c) 2000, 2013, Oracle, Monty Program Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> create database test; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> use test; Database changed MariaDB [test]> CREATE TABLE t (id bigint(20), name varchar(30), parent bigint(20), primary key (id)) ENGINE=InnoDB; Query OK, 0 rows affected (0.27 sec) MariaDB [test]> INSERT INTO t values (1, 'a', NULL), (4, 'b', 1), (5, 'c', 4), (10, 'd', 1); Query OK, 4 rows affected (0.04 sec) Records: 4 Duplicates: 0 Warnings: 0 MariaDB [test]> SELECT t1.id, t1.name, t2.id, t2.name FROM t t1 LEFT OUTER JOIN t t2 ON (t1.parent = t2.id and t2.name <> 'a') GROUP BY t1.id; +----+------+------+------+ | id | name | id | name | +----+------+------+------+ | 1 | a | NULL | NULL | | 4 | b | NULL | NULL | | 5 | c | NULL | NULL | | 10 | d | NULL | NULL | +----+------+------+------+ 4 rows in set (0.00 sec) MariaDB [test]> SELECT t1.id, t1.name, t2.id, t2.name FROM t t1 LEFT OUTER JOIN t t2 ON (t1.parent = t2.id and t2.name <> 'a'); +----+------+------+------+ | id | name | id | name | +----+------+------+------+ | 1 | a | NULL | NULL | | 4 | b | NULL | NULL | | 5 | c | 4 | b | | 10 | d | NULL | NULL | +----+------+------+------+ 4 rows in set (0.00 sec) MariaDB [test]> SELECT t1.id, t1.name, t2.id, t2.name FROM t t1 LEFT OUTER JOIN t t2 ON (t1.parent = t2.id and t2.name <> 'a') GROUP BY t1.id, t1.name; +----+------+------+------+ | id | name | id | name | +----+------+------+------+ | 1 | a | NULL | NULL | | 4 | b | NULL | NULL | | 5 | c | 4 | b | | 10 | d | NULL | NULL | +----+------+------+------+ 4 rows in set (0.00 sec) MariaDB [test]> SELECT t1.id, t1.name, t2.id, t2.name FROM t t1 LEFT OUTER JOIN t t2 ON (t1.parent = t2.id) GROUP BY t1.id; +----+------+------+------+ | id | name | id | name | +----+------+------+------+ | 1 | a | NULL | NULL | | 4 | b | 1 | a | | 5 | c | 4 | b | | 10 | d | 1 | a | +----+------+------+------+ 4 rows in set (0.00 sec) As you see the first query returns NULLs instead of real values which can be seen in second query without GROUP BY. And if I add another column to GROUP BY or remove one condition from JOIN I can see real values again. I've also tested with the table without PRIMARY KEY and it also returned real values in the first query. So is this a bug? Thank you, Pavel

2 2

[Maria-developers] Interested in GSOC
by Gopichand Paturi 24 Feb '14

24 Feb '14

Dear Sir/Madam, I am a student of Computer Science at BITS PILANI, India. I am interested in working as a developer for MariaDB in 2014. Can someone please let me know the ideas and proposed projects this year? Someone please guide me what to familiarize myself to be helpful and contribute for MariaDB. I am C++ developer....with good knowledge in Data Structures,Algorithms,Database Systems. Thanks& Regards, Gopichand.

2 1

[Maria-developers] Proposal for: MDEV-5244: Make extended_keys=ON by default in 10.0
by Sergey Petrunia 22 Feb '14

22 Feb '14

Hello, I was looking at how to make extended_keys=ON by default in 10.0. There haven't been any bugs in the feature for quite some time, but we still have a question of how to be with the testsuite (do we run it with extended_keys=OFF, the old way, or with extended_keys=ON, the new way? both approaches have their advantages) So, I've just changed the default, and ran the default set of test suites. A total 12 tests fail, of those only 3 tests fail in the way that indicates that extended keys feature had an impact on the testcase (e.g. it got a different execution plan and probably no longer tests what it used to test). For these, I intend to place set opimizer_switch='extended_keys=off' <testcase> set opimizer_switch='extended_keys=on' around the affected testcases. The cases are: range_vs_index_merge_innodb - switch @@optimizer_switch (~3 testcases) index_merge_innodb - switch @@optimizer_switch (1 testcase) innodb.innodb_mysql -switch @@optimizer_switch (1 testcase) Any comments/input on this? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] [svoj@mariadb.org: [Commits] Rev 4006: MDEV-5674 - Performance: my_hash_sort_bin is called too often in lp:maria/10.0]
by Sergey Vojtovich 20 Feb '14

20 Feb '14

Hi! JFYI: we discussed this during the Barcelona meeting. I will submit table cache part separately (not to overcomplicate this patch). Reviewers are very welcome. Regards, Sergey ----- Forwarded message from Sergey Vojtovich <svoj(a)mariadb.org> ----- Date: Fri, 14 Feb 2014 12:25:50 +0000 (UTC) From: Sergey Vojtovich <svoj(a)mariadb.org> To: commits(a)mariadb.org Subject: [Commits] Rev 4006: MDEV-5674 - Performance: my_hash_sort_bin is called too often in lp:maria/10.0 At lp:maria/10.0 ------------------------------------------------------------ revno: 4006 revision-id: svoj(a)mariadb.org-20140214122541-tenvcllk760deai8 parent: svoj(a)mariadb.org-20140213071355-psk0jpsarx2u7myc committer: Sergey Vojtovich <svoj(a)mariadb.org> branch nick: 10.0 timestamp: Fri 2014-02-14 16:25:41 +0400 message: MDEV-5674 - Performance: my_hash_sort_bin is called too often - reduced number of my_hash_sort_bin() calls from 4 to 2 per query. Let MDL subsystem use pre-calculated hash value for hash inserts and deletes. - reduced number of memory accesses done by my_hash_sort_bin() === modified file 'include/hash.h' --- a/include/hash.h 2013-09-14 01:09:36 +0000 +++ b/include/hash.h 2014-02-14 12:25:41 +0000 @@ -86,7 +86,13 @@ uchar *my_hash_first_from_hash_value(con uchar *my_hash_next(const HASH *info, const uchar *key, size_t length, HASH_SEARCH_STATE *state); my_bool my_hash_insert(HASH *info, const uchar *data); +my_bool my_hash_insert_using_hash_value(HASH *info, + my_hash_value_type hash_value, + const uchar *record); my_bool my_hash_delete(HASH *hash, uchar *record); +my_bool my_hash_delete_using_hash_value(HASH *hash, + my_hash_value_type hash_value, + uchar *record); my_bool my_hash_update(HASH *hash, uchar *record, uchar *old_key, size_t old_key_length); void my_hash_replace(HASH *hash, HASH_SEARCH_STATE *state, uchar *new_row); === modified file 'mysys/hash.c' --- a/mysys/hash.c 2013-09-14 01:09:36 +0000 +++ b/mysys/hash.c 2014-02-14 12:25:41 +0000 @@ -380,6 +380,15 @@ static int hashcmp(const HASH *hash, HAS my_bool my_hash_insert(HASH *info, const uchar *record) { + return my_hash_insert_using_hash_value(info, rec_hashnr(info, record), + record); +} + + +my_bool my_hash_insert_using_hash_value(HASH *info, + my_hash_value_type hash_value, + const uchar *record) +{ int flag; size_t idx,halfbuff,first_index; my_hash_value_type hash_nr; @@ -480,7 +489,7 @@ my_bool my_hash_insert(HASH *info, const } /* Check if we are at the empty position */ - idx= my_hash_mask(rec_hashnr(info, record), info->blength, info->records + 1); + idx= my_hash_mask(hash_value, info->blength, info->records + 1); pos=data+idx; if (pos == empty) { @@ -528,6 +537,15 @@ my_bool my_hash_insert(HASH *info, const my_bool my_hash_delete(HASH *hash, uchar *record) { + return my_hash_delete_using_hash_value(hash, rec_hashnr(hash, record), + record); +} + + +my_bool my_hash_delete_using_hash_value(HASH *hash, + my_hash_value_type hash_value, + uchar *record) +{ uint pos2,idx,empty_index; my_hash_value_type pos_hashnr, lastpos_hashnr; size_t blength; @@ -539,7 +557,7 @@ my_bool my_hash_delete(HASH *hash, uchar blength=hash->blength; data=dynamic_element(&hash->array,0,HASH_LINK*); /* Search after record with key */ - pos= data + my_hash_mask(rec_hashnr(hash, record), blength, hash->records); + pos= data + my_hash_mask(hash_value, blength, hash->records); gpos = 0; while (pos->data != record) === modified file 'sql/mdl.cc' --- a/sql/mdl.cc 2013-12-10 15:31:04 +0000 +++ b/sql/mdl.cc 2014-02-14 12:25:41 +0000 @@ -124,15 +124,9 @@ class MDL_map_partition public: MDL_map_partition(); ~MDL_map_partition(); - inline MDL_lock *find_or_insert(const MDL_key *mdl_key, - my_hash_value_type hash_value); - unsigned long get_lock_owner(const MDL_key *key, - my_hash_value_type hash_value); + inline MDL_lock *find_or_insert(const MDL_key *mdl_key); + unsigned long get_lock_owner(const MDL_key *key); inline void remove(MDL_lock *lock); - my_hash_value_type get_key_hash(const MDL_key *mdl_key) const - { - return my_calc_hash(&m_locks, mdl_key->ptr(), mdl_key->length()); - } private: bool move_from_hash_to_lock_mutex(MDL_lock *lock); /** A partition of all acquired locks in the server. */ @@ -846,11 +840,10 @@ MDL_lock* MDL_map::find_or_insert(const return lock; } - my_hash_value_type hash_value= m_partitions.at(0)->get_key_hash(mdl_key); - uint part_id= hash_value % mdl_locks_hash_partitions; + uint part_id= mdl_key->hash_value() % mdl_locks_hash_partitions; MDL_map_partition *part= m_partitions.at(part_id); - return part->find_or_insert(mdl_key, hash_value); + return part->find_or_insert(mdl_key); } @@ -863,15 +856,14 @@ MDL_lock* MDL_map::find_or_insert(const @retval NULL - Failure (OOM). */ -MDL_lock* MDL_map_partition::find_or_insert(const MDL_key *mdl_key, - my_hash_value_type hash_value) +MDL_lock* MDL_map_partition::find_or_insert(const MDL_key *mdl_key) { MDL_lock *lock; retry: mysql_mutex_lock(&m_mutex); if (!(lock= (MDL_lock*) my_hash_search_using_hash_value(&m_locks, - hash_value, + mdl_key->hash_value(), mdl_key->ptr(), mdl_key->length()))) { @@ -902,7 +894,9 @@ MDL_lock* MDL_map_partition::find_or_ins lock= MDL_lock::create(mdl_key, this); } - if (!lock || my_hash_insert(&m_locks, (uchar*)lock)) + if (!lock || my_hash_insert_using_hash_value(&m_locks, + mdl_key->hash_value(), + (uchar*) lock)) { if (unused_lock) { @@ -1023,10 +1017,9 @@ MDL_map::get_lock_owner(const MDL_key *m } else { - my_hash_value_type hash_value= m_partitions.at(0)->get_key_hash(mdl_key); - uint part_id= hash_value % mdl_locks_hash_partitions; + uint part_id= mdl_key->hash_value() % mdl_locks_hash_partitions; MDL_map_partition *part= m_partitions.at(part_id); - res= part->get_lock_owner(mdl_key, hash_value); + res= part->get_lock_owner(mdl_key); } return res; } @@ -1034,15 +1027,14 @@ MDL_map::get_lock_owner(const MDL_key *m unsigned long -MDL_map_partition::get_lock_owner(const MDL_key *mdl_key, - my_hash_value_type hash_value) +MDL_map_partition::get_lock_owner(const MDL_key *mdl_key) { MDL_lock *lock; unsigned long res= 0; mysql_mutex_lock(&m_mutex); lock= (MDL_lock*) my_hash_search_using_hash_value(&m_locks, - hash_value, + mdl_key->hash_value(), mdl_key->ptr(), mdl_key->length()); if (lock) @@ -1085,7 +1077,8 @@ void MDL_map::remove(MDL_lock *lock) void MDL_map_partition::remove(MDL_lock *lock) { mysql_mutex_lock(&m_mutex); - my_hash_delete(&m_locks, (uchar*) lock); + my_hash_delete_using_hash_value(&m_locks, lock->key.hash_value(), + (uchar*) lock); /* To let threads holding references to the MDL_lock object know that it was moved to the list of unused objects or destroyed, we increment the version === modified file 'sql/mdl.h' --- a/sql/mdl.h 2013-12-16 08:26:20 +0000 +++ b/sql/mdl.h 2014-02-14 12:25:41 +0000 @@ -347,12 +347,17 @@ class MDL_key m_ptr - 1); m_length= static_cast<uint16>(strmake(m_ptr + m_db_name_length + 2, name, NAME_LEN) - m_ptr + 1); + ulong nr2= 4; + m_hash_value= 1; + my_charset_bin.coll->hash_sort(&my_charset_bin, (uchar*) m_ptr, m_length, + &m_hash_value, &nr2); } void mdl_key_init(const MDL_key *rhs) { memcpy(m_ptr, rhs->m_ptr, rhs->m_length); m_length= rhs->m_length; m_db_name_length= rhs->m_db_name_length; + m_hash_value= rhs->m_hash_value; } bool is_equal(const MDL_key *rhs) const { @@ -392,10 +397,15 @@ class MDL_key { return & m_namespace_to_wait_state_name[(int)mdl_namespace()]; } + ulong hash_value() const + { + return m_hash_value; + } private: uint16 m_length; uint16 m_db_name_length; + ulong m_hash_value; char m_ptr[MAX_MDLKEY_LENGTH]; static PSI_stage_info m_namespace_to_wait_state_name[NAMESPACE_END]; private: === modified file 'strings/ctype-bin.c' --- a/strings/ctype-bin.c 2013-11-20 11:05:39 +0000 +++ b/strings/ctype-bin.c 2014-02-14 12:25:41 +0000 @@ -277,7 +277,9 @@ void my_hash_sort_8bit_bin(CHARSET_INFO ulong *nr1, ulong *nr2) { const uchar *pos = key; - + ulong tmp1= *nr1; + ulong tmp2= *nr2; + /* Remove trailing spaces. We have to do this to be able to compare 'A ' and 'A' as identical @@ -286,10 +288,13 @@ void my_hash_sort_8bit_bin(CHARSET_INFO for (; pos < (uchar*) key ; pos++) { - nr1[0]^=(ulong) ((((uint) nr1[0] & 63)+nr2[0]) * - ((uint)*pos)) + (nr1[0] << 8); - nr2[0]+=3; + tmp1^= (ulong) ((((uint) tmp1 & 63) + tmp2) * + ((uint) *pos)) + (tmp1 << 8); + tmp2+= 3; } + + *nr1= tmp1; + *nr2= tmp2; } @@ -297,15 +302,20 @@ void my_hash_sort_bin(CHARSET_INFO *cs _ const uchar *key, size_t len,ulong *nr1, ulong *nr2) { const uchar *pos = key; - + ulong tmp1= *nr1; + ulong tmp2= *nr2; + key+= len; - + for (; pos < (uchar*) key ; pos++) { - nr1[0]^=(ulong) ((((uint) nr1[0] & 63)+nr2[0]) * - ((uint)*pos)) + (nr1[0] << 8); - nr2[0]+=3; + tmp1^= (ulong) ((((uint) tmp1 & 63) + tmp2) * + ((uint) *pos)) + (tmp1 << 8); + tmp2+= 3; } + + *nr1= tmp1; + *nr2= tmp2; } _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message -----

1 1

[Maria-developers] Buildbot: 10.0-mroonga and 10.0-spider
by Alexander Barkov 18 Feb '14

18 Feb '14

Hello Daniel, Kristian, Sergei, Can you please add 10.0-mroonga and 10.0-spider into BuildBot? Thanks.

1 1

[Maria-developers] Fwd: [debian-mysql] mariadb-5.5_5.5.35-1_amd64.changes ACCEPTED into unstable
by Otto Kekäläinen 18 Feb '14

18 Feb '14

FYI, my 5.5.35 just got uploaded to Debian by James Page. Next time .36 is released I'll try to get the upload done in 1-2 days after release. This of course if you help me a little bit by setting up the discussed build bots so that Debian is part of your QA and if you accept to merge my debian/* upstream so that there will be less "delta" per release to look at. ---------- Forwarded message ---------- From: Debian FTP Masters <ftpmaster(a)ftp-master.debian.org> Date: 2014-02-18 11:21 GMT+02:00 Subject: [debian-mysql] mariadb-5.5_5.5.35-1_amd64.changes ACCEPTED into unstable To: Debian MySQL Maintainers <pkg-mysql-maint(a)lists.alioth.debian.org>, James Page <jamespage(a)debian.org> Accepted: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Format: 1.8 Date: Mon, 17 Feb 2014 16:51:52 +0000 Source: mariadb-5.5 Binary: libmariadbclient18 libmariadbd-dev libmariadbclient-dev mariadb-common mariadb-client-core-5.5 mariadb-client-5.5 mariadb-server-core-5.5 mariadb-test-5.5 mariadb-server-5.5 mariadb-server mariadb-client mariadb-test Architecture: source all amd64 Version: 5.5.35-1 Distribution: unstable Urgency: low Maintainer: Debian MySQL Maintainers <pkg-mysql-maint(a)lists.alioth.debian.org> Changed-By: James Page <jamespage(a)debian.org> Description: libmariadbclient-dev - MariaDB database development files libmariadbclient18 - MariaDB database client library libmariadbd-dev - MariaDB embedded database development files mariadb-client - MariaDB database client (metapackage depending on the latest vers mariadb-client-5.5 - MariaDB database client binaries mariadb-client-core-5.5 - MariaDB database core client binaries mariadb-common - MariaDB common metapackage mariadb-server - MariaDB database server (metapackage depending on the latest vers mariadb-server-5.5 - MariaDB database server binaries mariadb-server-core-5.5 - MariaDB database core server files mariadb-test - MariaDB database regression test suite (metapackage for the lates mariadb-test-5.5 - MariaDB database regression test suite Closes: 732967 733002 734426 735284 736480 737597 Changes: mariadb-5.5 (5.5.35-1) unstable; urgency=low . [ Otto Kekäläinen ] * New upstream release, fixing the following security issues: - Buffer overflow in client/mysql.cc (Closes: #737597). - CVE-2014-0001 - http://www.oracle.com/technetwork/topics/security/cpujan2014-1972949.html - CVE-2013-5891 - CVE-2013-5908 - CVE-2014-0386 - CVE-2014-0393 - CVE-2014-0401 - CVE-2014-0402 - CVE-2014-0412 - CVE-2014-0420 - CVE-2014-0437 * Upstream https://mariadb.atlassian.net/browse/MDEV-4902 fixes compatibility with Bison 3.0 (Closes: #733002) * Updated Russian debconf translation (Closes: #734426) * Updated Japanese debconf translation (Closes: #735284) * Updated French debconf translation (Closes: #736480) * Renamed SONAME properly (Closes: #732967) -- Otto Kekäläinen +358 44 566 2204 http://seravo.fi/

1 0

[Maria-developers] Plugin init()/deinit() call convention
by Sergey Vojtovich 17 Feb '14

17 Feb '14

Hi Sergei, IIRC during early days of plugins framework there was a strict convention when plugin deinit() function must be called. Specifically only after successful call to init() function. I'm now fixing a bug in related code (5.5) and noticed that: - deinit() may (or may not) be called without prior call to init() - deinit() may (or may not) be called with prior failed call to init() Do you think the above is correct? Thanks, Sergey

2 1

[Maria-developers] system variable value
by Rich Prohaska 17 Feb '14

17 Feb '14

I have a ULONGLONG system variable in my storage engine, and I set its value to '-1' on the mysqld command line. Its value gets set to all ones without any warnings or errors. The number conversion functions just cast between longlong and ulonglong types without regard to the sign of the number. Is this correct behavior?

2 1

[Maria-developers] Problems installing mariadb published debian packages when using tmpfs for /tmp
by Andrew McDonnell 17 Feb '14

17 Feb '14

Hiya, I followed the instructions here: https://downloads.mariadb.org/mariadb/repositories/#mirror=aarnet_pty_ltd&d… cluminating with: apt-get install mariadb-server mariadb-client -t wheezy and eventually after almost everything appears to have worked, I get a fail starting mysqld. (Log follows after) To summarise the log below, my interpetation of the errors is: * mysql doesnt like tmpfs * I am using tmpfs for /tmp. But the ibdata files getcreated in /var/lib/mysql which is not on tmpfs. So I dont understand what is happening here... * I am running zfs. Could be part of the problem? So I set innodb_data_home_dir = /path/to/something/on/ext4, then chown mysql:mysql /path/to/something/on/ext4 And fail with "Cannot create ./ib_logfile101" So I reverted my change to the location and instead tried using innodb_flush_method = O_DSYNC (which seems to default to O_DIRECT) This time things moved a little further, failing instead at 'Setting log file ./ib_logfile1 size to 48 MB 2014-02-16 21:51:39 7f03b252b780 InnoDB: Operating system error number 22 in a file operation.' Now I have managed to build mariadb from source and run without these problems. So there seems to be something tricky needed mariadb 10 to run in Debian? This on a wheezy 7.2 amd64 system. ---- Long trace follows ---- So I noted that mysql-common remained in the system, so I apt-get purged it (not just removed) and tried again. Purging this took out mariadb which obviously seems to have mysql-common as a dependency, so they all got a reinstalling. Which drags back in mysql-common, but this time as a 10.0, the one that got purged was 5.5.x This time I had the same errors, plus the following as well: 140216 21:22:08 [ERROR] InnoDB: Failed to set O_DIRECT on file ./ibdata1: OPEN: Invalid argument, continuing anyway 140216 21:22:08 [ERROR] InnoDB: O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662 I noticed then that /var/log/mysql.err was empty, oddly. So if I then manually run mysqld-safe --syslog: Feb 16 21:31:05 atlantis4 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [Note] InnoDB: The InnoDB memory heap is disabled Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [Not e] InnoDB: Compressed tables use zlib 1.2.7 Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [Note] InnoDB: Using Linux native AIO Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [Note] InnoDB: Not using CPU crc32 instructions Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [Note] InnoDB: Initializing buffer pool, size = 256.0M Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [Note] InnoDB: Completed initialization of buffer pool Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [ERROR] InnoDB: Failed to set O_DIRECT on file ./ibdata1: OPEN: Invalid argument, continuing anyway Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [ERROR] InnoDB: O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662 Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [ERROR] InnoDB: auto-extending data file ./ibdata1 is of a different size 640 pages (rounded down to MB) than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero) pages! Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data! Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [ERROR] Plugin 'InnoDB' init function returned error. Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [Note] Plugin 'FEEDBACK' is disabled. Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [ERROR] Unknown/unsupported storage engine: InnoDB Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [ERROR] Aborting Feb 16 21:31:05 atlantis4 mysqld: Feb 16 21:31:05 atlantis4 mysqld: 140216 21:31:05 [Note] /usr/sbin/mysqld: Shutdown complete Feb 16 21:31:05 atlantis4 mysqld: Feb 16 21:31:05 atlantis4 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended (END) So I deleted /var/log/mysql and reran mysql_installdb. # mysql_install_db Installing MariaDB/MySQL system tables in '/var/lib/mysql' ... 140216 21:33:12 [Note] InnoDB: The InnoDB memory heap is disabled 140216 21:33:12 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140216 21:33:12 [Note] InnoDB: Compressed tables use zlib 1.2.7 140216 21:33:12 [Note] InnoDB: Using Linux native AIO 140216 21:33:12 [Note] InnoDB: Not using CPU crc32 instructions 140216 21:33:12 [Note] InnoDB: Initializing buffer pool, size = 256.0M 140216 21:33:12 [Note] InnoDB: Completed initialization of buffer pool 140216 21:33:12 [ERROR] InnoDB: Failed to set O_DIRECT on file ./ibdata1: CREATE: Invalid argument, continuing anyway 140216 21:33:12 [ERROR] InnoDB: O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662 140216 21:33:12 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created! 140216 21:33:12 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB 140216 21:33:12 [Note] InnoDB: Database physically writes the file full: wait... 140216 21:33:12 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB 140216 21:33:12 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB 140216 21:33:12 [ERROR] InnoDB: Failed to set O_DIRECT on file ./ibdata1: OPEN: Invalid argument, continuing anyway 140216 21:33:12 [ERROR] InnoDB: O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662 2014-02-16 21:33:12 7f66ea899780 InnoDB: Operating system error number 22 in a file operation. InnoDB: Error number 22 means 'Invalid argument'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 140216 21:33:12 [ERROR] InnoDB: File ./ib_logfile101: 'aio write' returned OS error 122. Cannot continue operation Installation of system tables failed! Examine the logs in /var/lib/mysql for more information. <snipped> Errors after setting innodb_data_home_dir = /scratch/mysql # chown -R mysql:mysql /scratch/mysql/ root@atlantis4:/scratch# mysql_install_db Installing MariaDB/MySQL system tables in '/var/lib/mysql' ... 140216 21:47:44 [Note] InnoDB: The InnoDB memory heap is disabled 140216 21:47:44 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140216 21:47:44 [Note] InnoDB: Compressed tables use zlib 1.2.7 140216 21:47:44 [Note] InnoDB: Using Linux native AIO 140216 21:47:44 [Note] InnoDB: Not using CPU crc32 instructions 140216 21:47:44 [Note] InnoDB: Initializing buffer pool, size = 256.0M 140216 21:47:44 [Note] InnoDB: Completed initialization of buffer pool 140216 21:47:44 [Note] InnoDB: The first specified data file /scratch/mysql/ibdata1 did not exist: a new database to be created! 140216 21:47:44 [Note] InnoDB: Setting file /scratch/mysql/ibdata1 size to 12 MB 140216 21:47:44 [Note] InnoDB: Database physically writes the file full: wait... 140216 21:47:44 [ERROR] InnoDB: Cannot create ./ib_logfile101 140216 21:47:44 [ERROR] Plugin 'InnoDB' init function returned error. 140216 21:47:44 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 140216 21:47:44 [ERROR] Unknown/unsupported storage engine: InnoDB 140216 21:47:44 [ERROR] Aborting --- Log trace of first failed install follows --- Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libmariadbclient18 libmysqlclient18 mariadb-client-10.0 mariadb-client-core-10.0 mariadb-common mariadb-server-10.0 mariadb-server-core-10.0 Suggested packages: tinyca mariadb-test The following packages will be REMOVED: mysql-client-5.5 mysql-server-core-5.5Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libmariadbclient18 libmysqlclient18 mariadb-client-10.0 mariadb-client-core-10.0 mariadb-common mariadb-server-10.0 mariadb-server-core-10.0 Suggested packages: tinyca mariadb-test The following packages will be REMOVED: mysql-client-5.5 mysql-server-core-5.5 The following NEW packages will be installed: libmariadbclient18 mariadb-client mariadb-client-10.0 mariadb-client-core-10.0 mariadb-common mariadb-server mariadb-server-10.0 mariadb-server-core-10.0 The following packages will be upgraded: libmysqlclient18 1 upgraded, 8 newly installed, 2 to remove and 147 not upgraded. Need to get 36.2 MB of archives. After this operation, 65.4 MB of additional disk space will be used. Do you want to continue [Y/n]? Get:1 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-common all 10.0.8+maria-1~wheezy [3,310 B] Get:2 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main libmysqlclient18 amd64 10.0.8+maria-1~wheezy [2,966 B] Get:3 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main libmariadbclient18 amd64 10.0.8+maria-1~wheezy [941 kB] Get:4 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-client-core-10.0 amd64 10.0.8+maria-1~wheezy [1,993 kB] Get:5 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-client-10.0 amd64 10.0.8+maria-1~wheezy [5,593 kB] Get:6 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-server-core-10.0 amd64 10.0.8+maria-1~wheezy [6,096 kB] Get:7 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-server-10.0 amd64 10.0.8+maria-1~wheezy [21.6 MB] Get:8 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-client all 10.0.8+maria-1~wheezy [2,874 B] Get:9 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-server all 10.0.8+maria-1~wheezy [3,004 B] Fetched 36.2 MB in 2min 8s (281 kB/s) Preconfiguring packages ... (Reading database ... 358195 files and directories currently installed.) Removing mysql-client-5.5 ... Removing mysql-server-core-5.5 ... Processing triggers for man-db ... Selecting previously unselected package mariadb-common. (Reading database ... 358053 files and directories currently installed.) Unpacking mariadb-common (from .../mariadb-common_10.0.8+maria-1~wheezy_all.deb) ... Preparing to replace libmysqlclient18:amd64 5.5.31+dfsg-0+wheezy1 (using .../libmysqlclient18_10.0.8+maria-1~wheezy_amd64.deb) ... Unpacking replacement libmysqlclient18 ... Selecting previously unselected package libmariadbclient18. Unpacking libmariadbclient18 (from .../libmariadbclient18_10.0.8+maria-1~wheezy_amd64.deb) ... Selecting previously unselected package mariadb-client-core-10.0. Unpacking mariadb-client-core-10.0 (from .../mariadb-client-core-10.0_10.0.8+maria-1~wheezy_amd64.deb) ... Selecting previously unselected package mariadb-client-10.0. Unpacking mariadb-client-10.0 (from .../mariadb-client-10.0_10.0.8+maria-1~wheezy_amd64.deb) ... Selecting previously unselected package mariadb-server-core-10.0. Unpacking mariadb-server-core-10.0 (from .../mariadb-server-core-10.0_10.0.8+maria-1~wheezy_amd64.deb) ... Processing triggers for man-db ... Processing triggers for menu ... Setting up mariadb-common (10.0.8+maria-1~wheezy) ... Selecting previously unselected package mariadb-server-10.0. (Reading database ... 358190 files and directories currently installed.) Unpacking mariadb-server-10.0 (from .../mariadb-server-10.0_10.0.8+maria-1~wheezy_amd64.deb) ... Selecting previously unselected package mariadb-client. Unpacking mariadb-client (from .../mariadb-client_10.0.8+maria-1~wheezy_all.deb) ... Selecting previously unselected package mariadb-server. Unpacking mariadb-server (from .../mariadb-server_10.0.8+maria-1~wheezy_all.deb) ... Processing triggers for man-db ... Setting up libmysqlclient18 (10.0.8+maria-1~wheezy) ... Setting up libmariadbclient18 (10.0.8+maria-1~wheezy) ... Setting up mariadb-client-core-10.0 (10.0.8+maria-1~wheezy) ... Setting up mariadb-client-10.0 (10.0.8+maria-1~wheezy) ... Setting up mariadb-server-core-10.0 (10.0.8+maria-1~wheezy) ... Setting up mariadb-server-10.0 (10.0.8+maria-1~wheezy) ... [ ok ] Stopping MariaDB database server: mysqld. 140216 21:14:37 [Note] InnoDB: The InnoDB memory heap is disabled 140216 21:14:37 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140216 21:14:37 [Note] InnoDB: Compressed tables use zlib 1.2.7 140216 21:14:37 [Note] InnoDB: Using Linux native AIO 140216 21:14:37 [Note] InnoDB: Not using CPU crc32 instructions 140216 21:14:37 [Note] InnoDB: Initializing buffer pool, size = 128.0M 140216 21:14:37 [Note] InnoDB: Completed initialization of buffer pool 140216 21:14:37 [ERROR] InnoDB: auto-extending data file ./ibdata1 is of a different size 640 pages (rounded down to MB) than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero) pages! 140216 21:14:37 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data! 140216 21:14:37 [ERROR] Plugin 'InnoDB' init function returned error. 140216 21:14:37 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 140216 21:14:37 [Note] Plugin 'FEEDBACK' is disabled. 140216 21:14:37 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it [FAIL] Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed! invoke-rc.d: initscript mysql, action "start" failed. dpkg: error processing mariadb-server-10.0 (--configure): subprocess installed post-installation script returned error exit status 1 Setting up mariadb-client (10.0.8+maria-1~wheezy) ... dpkg: dependency problems prevent configuration of mariadb-server: mariadb-server depends on mariadb-server-10.0 (= 10.0.8+maria-1~wheezy); however: Package mariadb-server-10.0 is not configured yet. dpkg: error processing mariadb-server (--configure): dependency problems - leaving unconfigured Processing triggers for menu ... Errors were encountered while processing: mariadb-server-10.0 mariadb-server localepurge: Disk space freed in /usr/share/locale: 0 KiB localepurge: Disk space freed in /usr/share/man: 0 KiB localepurge: Disk space freed in /usr/share/gnome/help: 0 KiB localepurge: Disk space freed in /usr/share/omf: 0 KiB localepurge: Disk space freed in /usr/share/doc/kde/HTML: 0 KiB Total disk space freed by localepurge: 0 KiB E: Sub-process /usr/bin/dpkg returned an error code (1) The following NEW packages will be installed: libmariadbclient18 mariadb-client mariadb-client-10.0 mariadb-client-core-10.0 mariadb-common mariadb-server mariadb-server-10.0 mariadb-server-core-10.0 The following packages will be upgraded: libmysqlclient18 1 upgraded, 8 newly installed, 2 to remove and 147 not upgraded. Need to get 36.2 MB of archives. After this operation, 65.4 MB of additional disk space will be used. Do you want to continue [Y/n]? Get:1 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-common all 10.0.8+maria-1~wheezy [3,310 B] Get:2 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main libmysqlclient18 amd64 10.0.8+maria-1~wheezy [2,966 B] Get:3 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main libmariadbclient18 amd64 10.0.8+maria-1~wheezy [941 kB] Get:4 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-client-core-10.0 amd64 10.0.8+maria-1~wheezy [1,993 kB] Get:5 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-client-10.0 amd64 10.0.8+maria-1~wheezy [5,593 kB] Get:6 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-server-core-10.0 amd64 10.0.8+maria-1~wheezy [6,096 kB] Get:7 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-server-10.0 amd64 10.0.8+maria-1~wheezy [21.6 MB] Get:8 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-client all 10.0.8+maria-1~wheezy [2,874 B] Get:9 http://mirror.aarnet.edu.au/pub/MariaDB/repo/10.0/debian/ wheezy/main mariadb-server all 10.0.8+maria-1~wheezy [3,004 B] Fetched 36.2 MB in 2min 8s (281 kB/s) Preconfiguring packages ... (Reading database ... 358195 files and directories currently installed.) Removing mysql-client-5.5 ... Removing mysql-server-core-5.5 ... Processing triggers for man-db ... Selecting previously unselected package mariadb-common. (Reading database ... 358053 files and directories currently installed.) Unpacking mariadb-common (from .../mariadb-common_10.0.8+maria-1~wheezy_all.deb) ... Preparing to replace libmysqlclient18:amd64 5.5.31+dfsg-0+wheezy1 (using .../libmysqlclient18_10.0.8+maria-1~wheezy_amd64.deb) ... Unpacking replacement libmysqlclient18 ... Selecting previously unselected package libmariadbclient18. Unpacking libmariadbclient18 (from .../libmariadbclient18_10.0.8+maria-1~wheezy_amd64.deb) ... Selecting previously unselected package mariadb-client-core-10.0. Unpacking mariadb-client-core-10.0 (from .../mariadb-client-core-10.0_10.0.8+maria-1~wheezy_amd64.deb) ... Selecting previously unselected package mariadb-client-10.0. Unpacking mariadb-client-10.0 (from .../mariadb-client-10.0_10.0.8+maria-1~wheezy_amd64.deb) ... Selecting previously unselected package mariadb-server-core-10.0. Unpacking mariadb-server-core-10.0 (from .../mariadb-server-core-10.0_10.0.8+maria-1~wheezy_amd64.deb) ... Processing triggers for man-db ... Processing triggers for menu ... Setting up mariadb-common (10.0.8+maria-1~wheezy) ... Selecting previously unselected package mariadb-server-10.0. (Reading database ... 358190 files and directories currently installed.) Unpacking mariadb-server-10.0 (from .../mariadb-server-10.0_10.0.8+maria-1~wheezy_amd64.deb) ... Selecting previously unselected package mariadb-client. Unpacking mariadb-client (from .../mariadb-client_10.0.8+maria-1~wheezy_all.deb) ... Selecting previously unselected package mariadb-server. Unpacking mariadb-server (from .../mariadb-server_10.0.8+maria-1~wheezy_all.deb) ... Processing triggers for man-db ... Setting up libmysqlclient18 (10.0.8+maria-1~wheezy) ... Setting up libmariadbclient18 (10.0.8+maria-1~wheezy) ... Setting up mariadb-client-core-10.0 (10.0.8+maria-1~wheezy) ... Setting up mariadb-client-10.0 (10.0.8+maria-1~wheezy) ... Setting up mariadb-server-core-10.0 (10.0.8+maria-1~wheezy) ... Setting up mariadb-server-10.0 (10.0.8+maria-1~wheezy) ... [ ok ] Stopping MariaDB database server: mysqld. 140216 21:14:37 [Note] InnoDB: The InnoDB memory heap is disabled 140216 21:14:37 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140216 21:14:37 [Note] InnoDB: Compressed tables use zlib 1.2.7 140216 21:14:37 [Note] InnoDB: Using Linux native AIO 140216 21:14:37 [Note] InnoDB: Not using CPU crc32 instructions 140216 21:14:37 [Note] InnoDB: Initializing buffer pool, size = 128.0M 140216 21:14:37 [Note] InnoDB: Completed initialization of buffer pool 140216 21:14:37 [ERROR] InnoDB: auto-extending data file ./ibdata1 is of a different size 640 pages (rounded down to MB) than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero) pages! 140216 21:14:37 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data! 140216 21:14:37 [ERROR] Plugin 'InnoDB' init function returned error. 140216 21:14:37 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 140216 21:14:37 [Note] Plugin 'FEEDBACK' is disabled. 140216 21:14:37 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it insserv: warning: script 'K01fcron' missing LSB tags and overrides insserv: warning: script 'fcron' missing LSB tags and overrides [FAIL] Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed! invoke-rc.d: initscript mysql, action "start" failed. dpkg: error processing mariadb-server-10.0 (--configure): subprocess installed post-installation script returned error exit status 1 Setting up mariadb-client (10.0.8+maria-1~wheezy) ... dpkg: dependency problems prevent configuration of mariadb-server: mariadb-server depends on mariadb-server-10.0 (= 10.0.8+maria-1~wheezy); however: Package mariadb-server-10.0 is not configured yet. dpkg: error processing mariadb-server (--configure): dependency problems - leaving unconfigured Processing triggers for menu ... Errors were encountered while processing: mariadb-server-10.0 mariadb-server localepurge: Disk space freed in /usr/share/locale: 0 KiB localepurge: Disk space freed in /usr/share/man: 0 KiB localepurge: Disk space freed in /usr/share/gnome/help: 0 KiB localepurge: Disk space freed in /usr/share/omf: 0 KiB localepurge: Disk space freed in /usr/share/doc/kde/HTML: 0 KiB Total disk space freed by localepurge: 0 KiB E: Sub-process /usr/bin/dpkg returned an error code (1)

3 4

[Maria-developers] Open Knowledge Base questions
by Ian Gilfillan 16 Feb '14

16 Feb '14

There are a few open questions on the Knowledge Base - some have been around a while, and it would be great if anyone can answer them: 1) Licensing question about MariaDB and dot net Connector: https://mariadb.com/kb/en/licencing-of-mariadb-and-mysql-dot-net-connector/ 2) Query on whether the MariaDB Java Client can support Java 1.5 https://mariadb.com/kb/en/java-15-and-mariadb-connector/ The answer's presumably no, but a 1-liner on why not would be nice to have 3) Another Java client query: https://mariadb.com/kb/en/client-for-mysql-zerodatetimebehaviorconverttonul… 4) A couple of Galera queries: https://mariadb.com/kb/en/asynchronous-replication-with-galera-cluster/ https://mariadb.com/kb/en/simple-cluster-two-nodes-error/ https://mariadb.com/kb/en/strange-ssl-messages-in-error-log/ thanks, ian

1 0

Re: [Maria-developers] uint6korr optimization
by Michael Widenius 16 Feb '14

16 Feb '14

Hi! >>>>> "Alexey" == Alexey Botchkov <holyfoot(a)askmonty.org> writes: Alexey> Hi, Monty. Alexey> I've looked at that part of the code, and tried to make the improved Alexey> versions of uint6korr and mi_uint6korr. Alexey> The small program attached that benchmarks these. Alexey> Depending on enabled macros it loops uint6korr, hf_uint6korr, Alexey> mi_uint6korr and hf_mi_uint6korr respectively. It performs 2 loops on Alexey> each functions first runs it once, the second - twice, so we can Alexey> calculate how much time was spent on the operation itself. Alexey> The results i got so far are: Alexey> elapsed 103 seconds on korr6-1 Alexey> elapsed 190 seconds on korr6-2 Alexey> elapsed 50 seconds on hf_korr6-1 Alexey> elapsed 79 seconds on hf_korr6-2 Alexey> elapsed 106 seconds on mi6-1 Alexey> elapsed 195 seconds on mi6-2 Alexey> elapsed 56 seconds on hf_mi6B-1 Alexey> elapsed 88 seconds on hf_mi6-2 Alexey> So the Alexey> hf_uint6korr is 3 times faster than uint6korr. Alexey> hf_mi_uint6korr is 2.8 times faster than mi_uint6korr. Alexey> You're welcome to check the code out. Thanks. What is important is to get fast versions of mi_uint3korr (Used a lot in ma_dynrec.c and gis) mi_uint4korr mi_uint5korr mi_uint6korr mi_uint7korr mi_uint8korr (Used for some variables) uint5korr uint6korr (Used a lot in Aria) uint8korr (I am including the full test so that anyone can comment upon this) ---------------------------------------------------------------------- #include <stdio.h> #include <time.h> #include <malloc.h> #define TEST_KORR6 #define TEST_HF_KORR6 #define TEST_MI6 #define TEST_HF_MI6 #define uint6korr(A) ((ulonglong)(((uint32) ((uchar) (A)[0])) + \ (((uint32) ((uchar) (A)[1])) << 8) + \ (((uint32) ((uchar) (A)[2])) << 16) + \ (((uint32) ((uchar) (A)[3])) << 24)) + \ (((ulonglong) ((uchar) (A)[4])) << 32) + \ (((ulonglong) ((uchar) (A)[5])) << 40)) #define mi_uint6korr(A) ((ulonglong)(((uint32) (((const uchar*) (A))[5])) +\ (((uint32) (((const uchar*) (A))[4])) << 8) +\ (((uint32) (((const uchar*) (A))[3])) << 16) +\ (((uint32) (((const uchar*) (A))[2])) << 24)) +\ (((ulonglong) (((uint32) (((const uchar*) (A))[1])) +\ (((uint32) (((const uchar*) (A))[0]) << 8)))) <<\ 32)) #define hf_uint6korr(A) (((ulonglong) ((uint32 *) (A))[0]) + (((ulonglong) ((uint16 *) (A))[2]) << 32)) #define hf_mi_uint6korr(src, dest) \ __asm__ ( \ "bswapq %1;" \ "mov %1, %0;" \ :"=r"(dest) \ :"r"(hf_uint6korr(src)<<16) \ : \ ) typedef unsigned long long int ulonglong; typedef unsigned int uint32; typedef unsigned char uchar; typedef unsigned short uint16; time_t t0, t1; ulonglong i; #define GM 10000000000LL #define BAS 2000000 int main() { ulonglong *pb, *pb2; char *art, *art2; ulonglong ci; art= malloc(6*BAS); pb= malloc(sizeof(ulonglong)*BAS); for (i=0; i<6*BAS; i++) art[i]= (char)i; art2= malloc(6*BAS); pb2= malloc(sizeof(ulonglong)*BAS); for (i=0; i<6*BAS; i++) art2[i]= (char)i; #ifdef TEST_KORR6 ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; pb[ci]= uint6korr(art+ci*6); ci++; } t1= time(0); printf("elapsed %d seconds on korr6-1\n", t1 - t0); ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; pb[ci]= uint6korr(art+ci*6); pb2[ci]= uint6korr(art2+ci*6); ci++; } t1= time(0); printf("elapsed %d seconds on korr6-2\n", t1 - t0); #endif /*KORR6*/ #ifdef TEST_HF_KORR6 ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; pb[ci]= hf_uint6korr(art+ci*6); ci++; } t1= time(0); printf("elapsed %d seconds on hf_korr6-1\n", t1 - t0); ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; pb[ci]= hf_uint6korr(art+ci*6); pb2[ci]= hf_uint6korr(art2+ci*6); ci++; } t1= time(0); printf("elapsed %d seconds on hf_korr6-2\n", t1 - t0); #endif /*HF_KORR6*/ #ifdef TEST_MI6 ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; pb[ci]= mi_uint6korr(art+ci*6); ci++; } t1= time(0); printf("elapsed %d seconds on mi6-1\n", t1 - t0); ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; pb[ci]= mi_uint6korr(art+ci*6); pb2[ci]= mi_uint6korr(art2+ci*6); ci++; } t1= time(0); printf("elapsed %d seconds on mi6-2\n", t1 - t0); #endif /*MI6*/ #ifdef TEST_HF_MI6 ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; hf_mi_uint6korr(art+ci*6, pb[ci]); ci++; } t1= time(0); printf("elapsed %d seconds on hf_mi6B-1\n", t1 - t0); ci= 0; t0= time(0); for (i=0; i<GM; i++) { if (ci >= BAS) ci= 0; hf_mi_uint6korr(art+ci*6, pb[ci]); hf_mi_uint6korr(art2+ci*6, pb2[ci]); ci++; } t1= time(0); printf("elapsed %d seconds on hf_mi6-2\n", t1 - t0); #endif /*HF_MI6*/ return 0; } Looks ok. Did you try with: #define hf_uint6korr(A) (*((uint64 *) A) & 0xffffffffffffLL) That may be faster, with the extra cost problem of reading 2 bytes extra, so this only safe to use if one knows that there is a few extra bytes in the buffer. Can you do a patch to my_global.h and myisampack and add the new functions there. What to do: - All improvements for now only for x64 architecture - Replace uint6korr(A) and any other function that can be improved without reading extra bytes with a faster version. - Improve with faster variants: mi_uint3korr mi_uint4korr mi_uint5korr mi_uint6korr mi_uint7korr mi_uint8korr uint5korr uint6korr uint8korr If the above suggested version of uint6korr() that reads extra bytes is notable faster, add a function: uint6korr_unsafe() Which of course should map to uint6korr for other architectures. ok? Regards, Monty

4 5

[Maria-developers] Potential participation in Google Summer of Code 2014
by Colin Charles 15 Feb '14

15 Feb '14

Hi! We have applied to be part of the Google Summer of Code 2014 project, and hope to gain some new contributors. If you are on this list and think you can be a mentor for a project, please consider writing a note at: https://mariadb.com/kb/en/google-summer-of-code-2014/ I think the aim for the GSoC students code should be for 10.1 inclusion If you have any queries, don't hesitate to ask me about them, I'm happy to help. Typically mentors need about 4 hours or so per week to assist the student in seeing their project become a reality I encourage many to participate as its a rewarding experience. In 2013, we did participate, and got 3 features out of it: SQL ROLES, PCRE regular expressions, and a Kerberos authentication plugin Looking forward to seeing more happen in 2014! (here's hoping we get accepted into GSoC 2014 of course!) cheers, -colin -- Colin Charles, Chief Evangelist MariaDB | t: +6-012-204-3201 | Skype: colincharles

1 0

Re: [Maria-developers] Rev 4056: MDEV-5616 - Deadlock between CREATE/DROP FUNCTION and SELECT from view
by Sergei Golubchik 13 Feb '14

13 Feb '14

Hi, Sergey! Ok to push. A couple of comments below. On Feb 07, Sergey Vojtovich wrote: > revno: 4056 > revision-id: svoj(a)mariadb.org-20140207100451-8pnzn85pjjwvwfs4 > parent: elenst(a)wheezy-64.home-20140205102537-7ern5gfca6a6ojg3 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5-mdev5616 > timestamp: Fri 2014-02-07 14:04:51 +0400 > message: > MDEV-5616 - Deadlock between CREATE/DROP FUNCTION and SELECT from view > > Deadlock happened due to mixed lock order. > CREATE/DROP function: wrlock(THR_LOCK_udf) -> lock(LOCK_open) > SELECT from view: lock(LOCK_open) -> rdlock(THR_LOCK_udf) > > Fixed CREATE/DROP function so that LOCK_open does not intersect with > wrlock(THR_LOCK_udf). > > 10.0 is not affected: it doesn't hold LOCK_open while opening view. I'm pretty sure it'll auto-merge and nobody will remember this warning at that time. So this patch will be in 10.0. Perhaps you should add #if MYSQL_VERSION_ID > 50600 #error don't merge into 10.0 #endif ? Or, may be, it's not a big deal to have it in 10.0... > === modified file 'sql/sql_udf.cc' > --- a/sql/sql_udf.cc 2012-03-27 23:04:46 +0000 > +++ b/sql/sql_udf.cc 2014-02-07 10:04:51 +0000 > @@ -466,7 +467,11 @@ int mysql_create_function(THD *thd,udf_f > if ((save_binlog_row_based= thd->is_current_stmt_binlog_format_row())) > thd->clear_current_stmt_binlog_format_row(); > > + tables.init_one_table("mysql", 5, "func", 4, "func", TL_WRITE); I know it's a old line that you've moved and it's a matter of taste anyway, so up to you, but generally I prefer to see STRING_WITH_LEN here. It 1) makes a certain class of typos impossible; 2) documents the code, one doesn't need to think what these 4 and 5 stand for. Regards, Sergei

2 1

Re: [Maria-developers] Rev 3967: MDEV-5597 - Reduce usage of LOCK_open: LOCK_flush
by Sergei Golubchik 12 Feb '14

12 Feb '14

Hi, Sergey! On Jan 31, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 3967 > revision-id: svoj(a)mariadb.org-20140131141943-fjyyih3h92lek62n > parent: svoj(a)mariadb.org-20140130124905-vy28zbs9swlpg8uq > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Fri 2014-01-31 18:19:43 +0400 > message: > MDEV-5597 - Reduce usage of LOCK_open: LOCK_flush > > Replaced LOCK_flush with per-share conditional variable. > === modified file 'sql/table_cache.cc' > --- a/sql/table_cache.cc 2014-01-30 12:49:05 +0000 > +++ b/sql/table_cache.cc 2014-01-31 14:19:43 +0000 > @@ -945,6 +949,7 @@ bool tdc_remove_table(THD *thd, enum_tdc > if ((share= tdc_delete_share(db, table_name))) > { > I_P_List <TABLE, TABLE_share> purge_tables; > + uint my_refs= 1; > > mysql_mutex_lock(&LOCK_open); > /* > @@ -969,28 +974,48 @@ bool tdc_remove_table(THD *thd, enum_tdc > if (kill_delayed_threads) > kill_delayed_threads_for_table(share); > > -#ifndef DBUG_OFF > - if (remove_type == TDC_RT_REMOVE_NOT_OWN) > + if (remove_type == TDC_RT_REMOVE_NOT_OWN || > + remove_type == TDC_RT_REMOVE_NOT_OWN_KEEP_SHARE) > { > TABLE_SHARE::All_share_tables_list::Iterator it(share->tdc.all_tables); > while ((table= it++)) > + { > + my_refs++; > DBUG_ASSERT(table->in_use == thd); > + } > } > -#endif > - > - mysql_rwlock_rdlock(&LOCK_flush); > + DBUG_ASSERT(share->tdc.all_tables.is_empty() || remove_type != TDC_RT_REMOVE_ALL); > mysql_mutex_unlock(&LOCK_open); > > while ((table= purge_tables.pop_front())) > intern_close_table(table); > - mysql_rwlock_unlock(&LOCK_flush); > > - DBUG_ASSERT(share->tdc.all_tables.is_empty() || remove_type != TDC_RT_REMOVE_ALL); > - tdc_release_share(share); > + if (remove_type != TDC_RT_REMOVE_UNUSED) > + { > + /* > + Even though current thread holds exclusive metadata lock on this share > + (asserted above), concurrent FLUSH TABLES threads may be in process of > + closing unused table instances belonging to this share. E.g.: > + thr1 (FLUSH TABLES): table= share->tdc.free_tables.pop_front(); > + thr1 (FLUSH TABLES): share->tdc.all_tables.remove(table); > + thr2 (ALTER TABLE): tdc_remove_table(); > + thr1 (FLUSH TABLES): intern_close_table(table); > + > + Current remove type assumes that all table instances (except for those > + that are owned by current thread) must be closed before > + thd_remove_table() returns. Wait for such tables here. > + > + Unfortunately TABLE_SHARE::wait_for_old_version() cannot be used here > + because it waits for all table instances, whereas we have to wait only > + for those that are not owned by current thread. > + */ > + mysql_mutex_lock(&share->tdc.LOCK_table_share); > + while (share->tdc.ref_count > my_refs) > + mysql_cond_wait(&share->tdc.COND_release, &share->tdc.LOCK_table_share); > + mysql_mutex_unlock(&share->tdc.LOCK_table_share); I'm sorry, I don't understand that at all. Why my_refs is only incremented in debug builds? In non-debug builds you wait until all tables but one are closed, in debug builds you may keep more than one table opened. You wait here on a condition, how do you know there's FLUSH TABLES (or anything at all) running that will signal this condition? > + } > > - /* Wait for concurrent threads to free unused objects. */ > - mysql_rwlock_wrlock(&LOCK_flush); > - mysql_rwlock_unlock(&LOCK_flush); > + tdc_release_share(share); > > found= true; > } Regards, Sergei

2 4

[Maria-developers] MariaDB config for Akonadi - suggestions?
by Sergey Petrunia 12 Feb '14

12 Feb '14

Hi, Daniel Vratil from Akonadi (a PIM manager for KDE) has asked me at FOSDEM about optimal MariaDB configuration for Akonadi. Some specific questions: * innodb_flush_log_at_trx_commit. Currently, Akonadi uses innodb_flush_log_at_trx_commit=2 which as far as I understand can cause up to 1 second of changes to be lost. Using value of 1 (always flush on commit) should be safer, but maybe it would put an unnecessary load on the system? Akonadi's use case has a bit different idea of what load is - there is only one person making changes, but the system is likely to be a laptop where one cares about battery life, etc. * innodb_buffer_pool_size=80M. The dataset size can vary from very small to up to 1-1.5Gb. Some users have very little data and complain that 80M is too much. Does anybody have an experience with using very small innodb_buffer_pool_size? Please find the full config file below: === Akonadi configuration file === # These settings can be adjusted using $HOME/.config/akonadi/mysql-local.conf # # Based on advice by Kris Köhntopp <kris(a)mysql.com> # [mysqld] # strict query parsing/interpretation # TODO: make Akonadi work with those settings enabled # sql_mode=strict_trans_tables,strict_all_tables,strict_error_for_division_by_zero,no_auto_create_user,no_auto_value_on_zero,no_engine_substitution,no_zero_date,no_zero_in_date,only_full_group_by,pipes_as_concat # sql_mode=strict_trans_tables # DEBUGGING: # log all queries, useful for debugging but generates an enormous amount of data # log=mysql.full # log queries slower than n seconds, log file name relative to datadir (for debugging only) # log_slow_queries=mysql.slow # long_query_time=1 # log queries not using indices, debug only, disable for production use # log_queries_not_using_indexes=1 # # mesure database size and adjust innodb_buffer_pool_size # SELECT sum(data_length) as bla, sum(index_length) as blub FROM information_schema.tables WHERE table_schema not in ("mysql", "information_schema"); # NOTES: # Keep Innob_log_waits and keep Innodb_buffer_pool_wait_free small (see show global status like "inno%", show global variables) #expire_logs_days=3 #sync_bin_log=0 # Use UTF-8 encoding for tables character_set_server=utf8 collation_server=utf8_general_ci # use InnoDB for transactions and better crash recovery default_storage_engine=innodb # memory pool InnoDB uses to store data dictionary information and other internal data structures (default:1M) # Deprecated in MySQL >= 5.6.3 innodb_additional_mem_pool_size=1M # memory buffer InnoDB uses to cache data and indexes of its tables (default:128M) # Larger values means less I/O innodb_buffer_pool_size=80M # Create a .ibd file for each table (default:0) innodb_file_per_table=1 # Write out the log buffer to the log file at each commit (default:1) innodb_flush_log_at_trx_commit=2 # Buffer size used to write to the log files on disk (default:1M for builtin, 8M for plugin) # larger values means less I/O innodb_log_buffer_size=1M # Size of each log file in a log group (default:5M) larger means less I/O but more time for recovery. innodb_log_file_size=64M # # error log file name, relative to datadir (default:hostname.err) log_error=mysql.err # print warnings and connection errors (default:1) log_warnings=2 # Convert table named to lowercase lower_case_table_names=1 # Maximum size of one packet or any generated/intermediate string. (default:1M) max_allowed_packet=32M # Maximum simultaneous connections allowed (default:100) max_connections=256 # The two options below make no sense with prepared statements and/or transactions # (make sense when having the same query multiple times) # Memory allocated for caching query results (default:0 (disabled)) query_cache_size=0 # Do not cache results (default:1) query_cache_type=0 # Do not use the privileges mechanisms skip_grant_tables # Do not listen for TCP/IP connections at all skip_networking # The number of open tables for all threads. (default:64) table_open_cache=200 # How many threads the server should cache for reuse (default:0) thread_cache_size=3 # wait 365d before dropping the DB connection (default:8h) wait_timeout=31536000 # We use InnoDB, so don't let MyISAM eat up memory key_buffer_size=16K [client] default-character-set=utf8 BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Minor glitch when doing out of tree cmake build
by Andrew McDonnell 10 Feb '14

10 Feb '14

Hi, I like to build MariaDB in a separate directory to keep the source tree clean. This works for me as follows: cd path/to/maria/source mkdir mybuildir cd mybuildir cmake .. "-DWITH_PLUGIN_ARIA=1 (etc...)" make mysql-test/mysql-test-run --suite oqgraph (etc) However when I want to use this build to create a non-MTR database as follows: MARIA_HOME=path/to/maria/source MARIA_BUILD=path/to/maria/source/mybuilddir sh $MARIA_BUILD/scripts/mysql_install_db \ --srcdir=$MARIA_HOME \ --builddir=$MARIA_BUILD \ --datadir=path/to/data --defaults-file=... This fails with FATAL ERROR: Could not find maria/source/sql/share/english/errmsg.sys Different combinations of --srcdir and --builddir produce similar but different errors. I tracked down where the files are set to to lines 263..303 of scripts/mysql_install_db.sh >From my reading of the help: --builddir=path If using --srcdir with out-of-directory builds, you will need to set this to the location of the build directory where built files reside. --srcdir=path The path to the MariaDB source directory. This option uses the compiled binaries and support files within the source tree, useful for if you don't want to install MariaDB yet and just want to create the system tables. I was able to make the install script work with the following patch === modified file 'scripts/mysql_install_db.sh' --- scripts/mysql_install_db.sh 2014-02-03 14:22:39 +0000 +++ scripts/mysql_install_db.sh 2014-02-08 11:52:52 +0000 @@ -267,7 +267,7 @@ bindir="$basedir/client" extra_bindir="$basedir/extra" mysqld="$basedir/sql/mysqld" - langdir="$srcdir/sql/share/english" + langdir="$basedir/sql/share/english" pkgdatadir="$srcdir/scripts" scriptdir="$srcdir/scripts" elif test -n "$basedir" What I dont know enough about, is whether this breaks other scenarios? any advice appreciated, thanks, Andrew -- http://blog.oldcomputerjunk.net https://launchpad.net/~andymc73 https://github.com/andymc73 Twitter: @pastcompute GPG: http://www.andrewmcdonnell.net/gpg.html

2 3

Re: [Maria-developers] Rev 4056: MDEV-5607: Query cache destroys uninitialized rwlock
by Sergei Golubchik 10 Feb '14

10 Feb '14

Hi, Sanja! On Feb 08, sanja(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 4056 > revision-id: sanja(a)askmonty.org-20140208132448-qdkkpdncnya0tr6j > parent: elenst(a)wheezy-64.home-20140205102537-7ern5gfca6a6ojg3 > committer: sanja(a)askmonty.org > branch nick: work-maria-5.5-MDEV-5607 > timestamp: Sat 2014-02-08 15:24:48 +0200 > message: > MDEV-5607: Query cache destroys uninitialized rwlock > > - Resize destroyed rw_lock twice, now it is prevented. > - Trash destroyed rw lock. > === modified file 'include/mysql/psi/mysql_thread.h' > --- a/include/mysql/psi/mysql_thread.h 2013-01-11 00:03:43 +0000 > +++ b/include/mysql/psi/mysql_thread.h 2014-02-08 13:24:48 +0000 > @@ -16,6 +16,8 @@ > #ifndef MYSQL_THREAD_H > #define MYSQL_THREAD_H > > +#include <my_valgrind.h> This is not ok. Files under include/mysql/ are Plugin API files, they should be clean from internal includes. You cannot include my_valgrind.h from there. Do it in my_pthread.h Regards, Sergei

2 1