- developers - lists.mariadb.org

Re: [Maria-developers] [Commits] 3d1b65b: MDEV-7943 - pthread_getspecific() takes 0.76% in OLTP RO
by Sergei Golubchik 18 Jun '15

18 Jun '15

Hi, Sergey! On Apr 28, svoj(a)mariadb.org wrote: > revision-id: 3d1b65b4923a49f9303501dd546af02982608b2b > parent(s): 8a0f3310275f7b4fa445f907140f677910e18999 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-04-28 15:20:48 +0400 > message: > > MDEV-7943 - pthread_getspecific() takes 0.76% in OLTP RO > > Pass THD to find_all_keys() and Item_equal::Item_equal(). > In MRR use table->in_use instead of current_thd. > > This reduces number of pthread_getspecific() calls from 354 to 320. ok to push. although for find_all_keys() it doesn't make any sense, it's a big, complex, and expensive function. One current_thd doesn't change anything. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] 2e08ec2: MDEV-7956 - handler::rebind_psi() takes 0.07% in OLTP RO
by Sergei Golubchik 18 Jun '15

18 Jun '15

Hi, Sergey! On Jun 17, svoj(a)mariadb.org wrote: > revision-id: 2e08ec24623e0b3d3db02333e16658dcfaa0535c > parent(s): b776f8d3a8afff55d5f8208da7bdfce9aa33eb39 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-17 13:09:22 +0400 > message: > > MDEV-7956 - handler::rebind_psi() takes 0.07% in OLTP RO > > Do not call handler::rebind_psi() and handler::unbind_psi() when performance > schema is compiled out. > > Overhead change: > handler::rebind_psi 0.04% -> out of radar > handler::unbind_psi 0.03% -> out of radar > open_table 0.21% -> 0.18% > close_thread_table 0.05% -> 0.05% Really? I don't believe it. handler::rebind_psi() and handler::unbind_psi() are empty methods, not doing anything. There's only a cost a virtual function call. And you're saying that it took 15% of the open_table() execution time? Regards, Sergei

2 1

[Maria-developers] [GSoC] MXS-2: Week 2 and 3
by sriram patil 17 Jun '15

17 Jun '15

Hi All, Please find link to my first blog post of second and third week below. <http://srirampatil.tumblr.com/post/121594591925/mxs-2-week-2-and-3> MXS-2: Week 2 and 3 <http://srirampatil.tumblr.com/post/121594591925/mxs-2-week-2-and-3> Feel free to point out mistakes, if any. Also, suggestions ans/or reviews are welcome! Thanks, Sriram

2 2

[Maria-developers] GSoC Progress Report [MDEV-5199]
by Blessing Atie 16 Jun '15

16 Jun '15

Hi Sergei, colin! This past week I have studied the sequence engine to understand how it generates tables on the fly. I also looked at how table functions work in other databases. I am currently working on how to implement the simplest table function from our new plugin API(Still to be completed) and which will do exacly what the sequence engine does but this time in a function call like in <select * from udf_seq(1,10)>. Exams are fast approaching in my University. I will acceletrate work as soon as I finish my Exams which will run from 22nd May to 6th June and after which I will be available to the fullest capacity to put enough time on my Work. In the mean while for this week I plan on completing our plugin declaration structure and also look for other candidate functions that can be implemented. Regards, Blessing Atie.

2 1

[Maria-developers] Sprint 10.0: MDEV-8205 timediff returns null when comparing decimal time to time string value
by Alexander Barkov 16 Jun '15

16 Jun '15

Hi Sergei, I'm not sure what to do with this: > MariaDB [test]> select > -> timediff('2014-01-01 00:00:00' , '2014-01-01 01:00:00' ), > -> timediff(20140101000000.000 , 20140101010000.000 ), > -> timediff(20140101000000.000 , '2014-01-01 01:00:00' ), > -> datediff('2014-01-01 00:00:00' , '2014-01-02 01:00:00' ), > -> datediff(20140101000000.000 , 20140102010000.000 ), > -> datediff(20140101000000.000 , '2014-01-02 01:00:00' )\G > *************************** 1. row *************************** > timediff('2014-01-01 00:00:00' , '2014-01-01 01:00:00' ): -01:00:00 > timediff(20140101000000.000 , 20140101010000.000 ): -01:00:00.000 > timediff(20140101000000.000 , '2014-01-01 01:00:00' ): NULL > datediff('2014-01-01 00:00:00' , '2014-01-02 01:00:00' ): -1 > datediff(20140101000000.000 , 20140102010000.000 ): -1 > datediff(20140101000000.000 , '2014-01-02 01:00:00' ): -1 > 1 row in set (0.03 sec) Notice, it returns NULL in one case. The problem is in this piece of the code: > if (args[0]->get_time(&l_time1) || > args[1]->get_time(&l_time2) || > l_time1.time_type != l_time2.time_type) > return (null_value= 1); get_time() forces decimal-to-time conversion to truncate the date part and return MYSQL_TIMESTAMP_TIME. get_time() does not force string-to-time conversion to truncate the date part. It still returns the full MYSQL_TIMESTAMP_DATETIME. I'm afraid a lot of the code rely in this behavior. If we fix string-to-time conversion, mtr will start to fail in other cases. Any ideas what to do?

2 4

[Maria-developers] MDEV-7502 Automatic provisioning of slave – retrieval of data from master
by Martin Kaluznik 16 Jun '15

16 Jun '15

Hello Kristian, after master-slave connection part, I proceeded to retrieval of data from master. There are two types of data which provisioning master will have to send, meta-data and row data. Meta-data For this I found only one method, pass output of SHOW CREATE * to slave in 'Query_log_event'. When I was looking for alternatives, I have found MySQL online backup work log where they are describing similar/same problem under High Level Architecture – Decision ( http://dev.mysql.com/worklog/task/?id=3574 ). Because of this, I find it unlikely, that there currently is better way. Because meta-data retrieval will take only small fraction of provisioning time, I think, that it could be implemented using 'Ed_connection' class, it executes query in string form and returns result set from within replication thread. Disadvantage of this solution is that query has to go through parser and all steps of generic execution, but on the other hand resulting code will be more independent from exact implementations, their pre-conditions and won’t contain any logical mistakes (locking, etc). Row data For this, same method with 'Ed_connection' could work, but it would be probably too big overhead – parser, result processing. So I created low level loop which goes through records of table and fills them in 'Write_rows_log_event'. Could you please check it, if it looks acceptable and I will then rewrite it to generic form, it is provisioning_send_info::send_table_data() function. https://github.com/f4rnham/server/blob/3d36cbfc20dff73d92ce61168dcc9805d68d… Right now it is working as expected on small test case. It goes through whole table at once and creates multiple row events when one would be enough. The same file also contains test of 'Ed_connection' and test of slave being able to receive and process 'Query_log_event'. Thanks for advice, Martin.

2 1

[Maria-developers] Getting the columns from LEX or THD
by sriram patil 16 Jun '15

16 Jun '15

Hi all, I wanted to retrieve a tables columns i.e its schema definition when a query is executed. I figured out LEX and THD structures store these in different variables for different queries. I wanted to get the same when a simple INSERT query is executed, e.g INSERT INTO <table> VALUES(<values>). Where can I find it? Is there any mechanism to get a tables schema that will work for any kind of query? Thanks, Sriram

2 2

Re: [Maria-developers] 8456b23: MDEV-5309 - RENAME TABLE does not check for existence of the table's engine
by Sergei Golubchik 16 Jun '15

16 Jun '15

Hi, Sergey! On Jun 15, svoj(a)mariadb.org wrote: > revision-id: 8456b231e9079be26c77da2a294ea1dac64d9d76 > parent(s): aad8667f8a4e2a26652494b29ee6c12824865ac8 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-15 15:49:00 +0400 > message: > > MDEV-5309 - RENAME TABLE does not check for existence of the table's engine > > When RENAME TABLE is executed, it apparently does not check whether the engine > is available (unlike ALTER TABLE .. RENAME, which does). It means that if the > engine in question was not loaded on some reason, the table might become > unusable, since the engine won't know about the change. > > With this patch RENAME TABLE fails if storage engine is not available. Ok to push. I'm not completely happy with the "table does not exist" error in this case, but I suspect that doing something reasonable might be not very easy. And it's a corner case anyway. Regards, Sergei

2 1

[Maria-developers] unsubscribe
by Abdul Hakeem 14 Jun '15

14 Jun '15

2 1

Re: [Maria-developers] 186e8e9: MDEV-5171: Add support for --innodb-optimize-keys to mysqldump.
by Sergei Golubchik 14 Jun '15

14 Jun '15

Hi, Jan! On May 04, Jan Lindström wrote: > revision-id: 186e8e91f3ce797258389f6d8729082f7d3ea164 > parent(s): aa5095627e2619bdad7916d33d1016802a84a9e1 > committer: Jan Lindström > branch nick: 10.0-git > timestamp: 2015-05-04 10:23:50 +0300 > message: > > MDEV-5171: Add support for --innodb-optimize-keys to mysqldump. > > Merged revision 93 from bzr+ssh://bazaar.launchpad.net/+branch/percona-server/5.6/ > authored by Alexey Kopytov. > > This patch expands the applicability of InnoDB fast index creation to > mysqldump, ALTER TABLE and OPTIMIZE TABLE as follows: > > 1. mysqldump has now a new option, --innodb-optimize-keys, which changes > the way InnoDB tables are dumped so that secondary and foreign keys are > created after loading the data thus taking advantage of fast index > creation. > > This part of the patch is an implementation of the feature request > reported as MySQL bug #49120. > > More specifically: > > - KEY, UNIQUE KEY and CONSTRAINT specifications are omitted from CREATE > TABLE corresponding to InnoDB tables. > > - an additional ALTER TABLE is issued after dumping the data to create > the previously omitted keys. > > Delaying foreign key creation does not introduce any additional risks as > mysqldump always prepends its output with SET FOREIGN_KEY_CHECKS=0 anyway. > > 2. When ALTER TABLE requires a table copy, secondary keys are now dropped > and recreated later after copying the data. The following restrictions > apply: > > - only non-unique keys can be involved in this optimization > > - if the table contains foreign keys, or a foreign key is being added as > a part of the current ALTER TABLE statement, the optimization is > disabled for all keys. > > This part of the patch is an implementation of the feature request > reported as MySQL bug #57583. > > 3. As OPTIMIZE TABLE is mapped to ALTER TABLE ... ENGINE=InnoDB for > InnoDB tables, it now also benefits from fast index creation with the > same restrictions as for ALTER TABLE. > diff --git a/client/client_priv.h b/client/client_priv.h > index 67a6e82..1846926 100644 > --- a/client/client_priv.h > +++ b/client/client_priv.h > @@ -94,6 +94,7 @@ enum options_client > OPT_SSL_CRL, OPT_SSL_CRLPATH, > OPT_USE_GTID, > OPT_GALERA_SST_MODE, > + OPT_INNODB_OPTIMIZE_KEYS, /* mysqldump */ Not needed, please, remove > OPT_MAX_CLIENT_OPTION /* should be always the last */ > }; > > diff --git a/client/mysqldump.c b/client/mysqldump.c > index e118198..0970082 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -375,6 +386,11 @@ static struct my_option my_long_options[] = > "in dump produced with --dump-slave.", &opt_include_master_host_port, > &opt_include_master_host_port, 0, GET_BOOL, NO_ARG, > 0, 0, 0, 0, 0, 0}, > + {"innodb-optimize-keys", OPT_INNODB_OPTIMIZE_KEYS, Here use 0 instead of OPT_INNODB_OPTIMIZE_KEYS > + "Use InnoDB fast index creation by creating secondary indexes after " > + "dumping the data.", > + &opt_innodb_optimize_keys, &opt_innodb_optimize_keys, 0, GET_BOOL, NO_ARG, > + 0, 0, 0, 0, 0, 0}, > {"insert-ignore", OPT_INSERT_IGNORE, "Insert rows with INSERT IGNORE.", > &opt_ignore, &opt_ignore, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, > 0, 0}, > @@ -2581,6 +2597,359 @@ static inline my_bool general_log_or_slow_log_tables(const char *db, > } > > /* > + Find the first occurrence of a quoted identifier in a given string. Returns > + the pointer to the opening quote, and stores the pointer to the closing quote > + to the memory location pointed to by the 'end' argument, > + > + If no quoted identifiers are found, returns NULL (and the value pointed to by > + 'end' is undefined in this case). > +*/ > + > +static const char *parse_quoted_identifier(const char *str, > + const char **end) > +{ > + const char *from; > + const char *to; > + > + if (!(from= strchr(str, '`'))) > + return NULL; This is wrong, see below. Identifiers ar enot necessarily quoted, so this function should parse both quoted and not quoted identifiers (and renamed to, say, parse_next_identifier) > + > + to= from; > + > + while ((to= strchr(to + 1, '`'))) { > + /* > + Double backticks represent a backtick in identifier, rather than a quote > + character. > + */ > + if (to[1] == '`') > + { > + to++; > + continue; > + } > + > + break; > + } > + > + if (to <= from + 1) > + return NULL; /* Empty identifier */ > + > + *end= to; > + > + return from; > +} > + > +/* > + Parse the specified key definition string and check if the key contains an > + AUTO_INCREMENT column as the first key part. We only check for the first key > + part, because unlike MyISAM, InnoDB does not allow the AUTO_INCREMENT column > + as a secondary key column, i.e. the AUTO_INCREMENT column would not be > + considered indexed for such key specification. > +*/ > +static my_bool contains_autoinc_column(const char *autoinc_column, > + const char *keydef, > + key_type_t type) > +{ > + const char *from, *to; > + uint idnum; > + > + DBUG_ASSERT(type != KEY_TYPE_NONE); > + > + if (autoinc_column == NULL) > + return FALSE; > + > + idnum= 0; > + > + /* > + There is only 1 iteration of the following loop for type == KEY_TYPE_PRIMARY > + and 2 iterations for type == KEY_TYPE_UNIQUE / KEY_TYPE_NON_UNIQUE. > + */ > + while ((from= parse_quoted_identifier(keydef, &to))) > + { > + idnum++; > + > + /* > + Skip the check if it's the first identifier and we are processing a > + secondary key. > + */ > + if ((type == KEY_TYPE_PRIMARY || idnum != 1) && > + !strncmp(autoinc_column, from + 1, to - from - 1)) > + return TRUE; > + > + /* > + Check only the first (for PRIMARY KEY) or the second (for secondary keys) > + quoted identifier. > + */ > + if ((idnum == 1 + MY_TEST(type != KEY_TYPE_PRIMARY))) > + break; > + > + keydef= to + 1; > + } > + > + return FALSE; > +} > + > +/* > + Find a node in the skipped keys list whose name matches a quoted > + identifier specified as 'id_from' and 'id_to' arguments. > +*/ > + > +static LIST *find_matching_skipped_key(const char *id_from, > + const char *id_to) > +{ > + LIST *list; > + size_t id_len; > + > + id_len= id_to - id_from + 1; > + DBUG_ASSERT(id_len > 2); > + > + for (list= skipped_keys_list; list; list= list_rest(list)) > + { > + const char *keydef; > + const char *keyname_from; > + const char *keyname_to; > + size_t keyname_len; > + > + keydef= list->data; > + > + if ((keyname_from= parse_quoted_identifier(keydef, &keyname_to))) > + { > + keyname_len= keyname_to - keyname_from + 1; > + > + if (id_len == keyname_len && > + !strncmp(keyname_from, id_from, id_len)) > + return list; > + } > + } > + > + return NULL; > +} > + > +/* > + Remove secondary/foreign key definitions from a given SHOW CREATE TABLE string > + and store them into a temporary list to be used later. > + > + SYNOPSIS > + skip_secondary_keys() > + create_str SHOW CREATE TABLE output > + has_pk TRUE, if the table has PRIMARY KEY > + (or UNIQUE key on non-nullable columns) > + > + > + DESCRIPTION > + > + Stores all lines starting with "KEY" or "UNIQUE KEY" > + into skipped_keys_list and removes them from the input string. > + Ignoring FOREIGN KEYS constraints when creating the table is ok, because > + mysqldump sets foreign_key_checks to 0 anyway. > +*/ > + > +static void skip_secondary_keys(char *create_str, my_bool has_pk) > +{ > + char *ptr, *strend; > + char *last_comma= NULL; > + my_bool pk_processed= FALSE; > + char *autoinc_column= NULL; > + my_bool has_autoinc= FALSE; > + key_type_t type; > + const char *constr_from; > + const char *constr_to; > + LIST *keydef_node; > + my_bool keys_processed= FALSE; > + > + strend= create_str + strlen(create_str); > + > + ptr= create_str; > + while (*ptr && !keys_processed) > + { please, fix the coding style (here and everywhere in the patch) to follow the rest of the file. That is while { char *tmp, *orig_ptr, c; ... > + char *tmp, *orig_ptr, c; > + > + orig_ptr= ptr; > + /* Skip leading whitespace */ > + while (*ptr && my_isspace(charset_info, *ptr)) > + ptr++; > + > + /* Read the next line */ > + for (tmp= ptr; *tmp != '\n' && *tmp != '\0'; tmp++); > + > + c= *tmp; > + *tmp= '\0'; /* so strstr() only processes the current line */ > + > + if (!strncmp(ptr, "CONSTRAINT ", sizeof("CONSTRAINT ") - 1) && > + (constr_from= parse_quoted_identifier(ptr, &constr_to)) && yuck! identifiers do not have to be quoted. this code needs to be fixed to accept non-quoted identifiers too. > + (keydef_node= find_matching_skipped_key(constr_from, constr_to))) > + { > + char *keydef; > + size_t keydef_len; > + > + /* > + There's a skipped key with the same name as the constraint name. Let's > + put it back before the current constraint definition and remove from the > + skipped keys list. 1. fix the formatting, please 2. why not to create the FK constraint afterwards? > + */ > + keydef= keydef_node->data; > + keydef_len= strlen(keydef) + 5; /* ", \n " */ > + > + memmove(orig_ptr + keydef_len, orig_ptr, strend - orig_ptr + 1); > + memcpy(ptr, keydef, keydef_len - 5); > + memcpy(ptr + keydef_len - 5, ", \n ", 5); > + > + skipped_keys_list= list_delete(skipped_keys_list, keydef_node); > + my_free(keydef); > + my_free(keydef_node); > + > + strend+= keydef_len; > + orig_ptr+= keydef_len; > + ptr+= keydef_len; > + tmp+= keydef_len; > + > + type= KEY_TYPE_NONE; > + } > + else if (!strncmp(ptr, "UNIQUE KEY ", sizeof("UNIQUE KEY ") - 1)) > + type= KEY_TYPE_UNIQUE; > + else if (!strncmp(ptr, "KEY ", sizeof("KEY ") - 1)) > + type= KEY_TYPE_NON_UNIQUE; > + else if (!strncmp(ptr, "PRIMARY KEY ", sizeof("PRIMARY KEY ") - 1)) > + type= KEY_TYPE_PRIMARY; > + else > + type= KEY_TYPE_NONE; > + > + has_autoinc= (type != KEY_TYPE_NONE) ? > + contains_autoinc_column(autoinc_column, ptr, type) : FALSE; > + > + /* Is it a secondary index definition? */ > + if (c == '\n' && > + ((type == KEY_TYPE_UNIQUE && (pk_processed || !has_pk)) || > + type == KEY_TYPE_NON_UNIQUE) && !has_autoinc) > + { > + char *data, *end= tmp - 1; > + > + /* Remove the trailing comma */ > + if (*end == ',') > + end--; > + data= my_strndup(ptr, end - ptr + 1, MYF(MY_FAE)); > + skipped_keys_list= list_cons(data, skipped_keys_list); > + > + memmove(orig_ptr, tmp + 1, strend - tmp); > + ptr= orig_ptr; > + strend-= tmp + 1 - ptr; > + > + /* Remove the comma on the previos line */ > + if (last_comma != NULL) > + { > + *last_comma= ' '; > + } > + } > + else > + { > + char *end; > + > + if (last_comma != NULL && *ptr == ')') > + { > + keys_processed= TRUE; > + } > + else if (last_comma != NULL && !keys_processed) > + { > + /* > + It's not the last line of CREATE TABLE, so we have skipped a key > + definition. We have to restore the last removed comma. > + */ > + *last_comma= ','; > + } > + > + /* > + If we are skipping a key which indexes an AUTO_INCREMENT column, it is > + safe to optimize all subsequent keys, i.e. we should not be checking for > + that column anymore. > + */ > + if (type != KEY_TYPE_NONE && has_autoinc) > + { > + DBUG_ASSERT(autoinc_column != NULL); > + > + my_free(autoinc_column); > + autoinc_column= NULL; > + } > + > + if ((has_pk && type == KEY_TYPE_UNIQUE && !pk_processed) || > + type == KEY_TYPE_PRIMARY) > + pk_processed= TRUE; > + > + if (strstr(ptr, "AUTO_INCREMENT") && *ptr == '`') > + { > + /* > + The first secondary key defined on this column later cannot be > + skipped, as CREATE TABLE would fail on import. Unless there is a > + PRIMARY KEY and it indexes that column. > + */ > + for (end= ptr + 1; > + /* Skip double backticks as they are a part of identifier */ > + *end != '\0' && (*end != '`' || end[1] == '`'); > + end++) > + /* empty */; > + > + if (*end == '`' && end > ptr + 1) > + { > + DBUG_ASSERT(autoinc_column == NULL); > + > + autoinc_column= my_strndup(ptr + 1, end - ptr - 1, MYF(MY_FAE)); Eh? Why does it need to strdup? > + } > + } > + > + *tmp= c; > + > + if (tmp[-1] == ',') > + last_comma= tmp - 1; > + ptr= (*tmp == '\0') ? tmp : tmp + 1; > + } > + } > + > + my_free(autoinc_column); > +} > + > +/* > + Check if the table has a primary key defined either explicitly or > + implicitly (i.e. a unique key on non-nullable columns). > + > + SYNOPSIS > + my_bool has_primary_key(const char *table_name) > + > + table_name quoted table name > + > + RETURNS TRUE if the table has a primary key > + > + DESCRIPTION > +*/ > + > +static my_bool has_primary_key(const char *table_name) > +{ > + MYSQL_RES *res= NULL; > + MYSQL_ROW row; > + char query_buff[QUERY_LENGTH]; > + my_bool has_pk= TRUE; > + > + my_snprintf(query_buff, sizeof(query_buff), > + "SELECT COUNT(*) FROM INFORMATION_SCHEMA.COLUMNS WHERE " > + "TABLE_SCHEMA=DATABASE() AND TABLE_NAME='%s' AND " > + "COLUMN_KEY='PRI'", table_name); > + if (mysql_query(mysql, query_buff) || !(res= mysql_store_result(mysql)) || > + !(row= mysql_fetch_row(res))) > + { > + fprintf(stderr, "Warning: Couldn't determine if table %s has a " > + "primary key (%s). " > + "--innodb-optimize-keys may work inefficiently.\n", > + table_name, mysql_error(mysql)); > + goto cleanup; > + } > + > + has_pk= atoi(row[0]) > 0; > + > + cleanup: > + if (res) > + mysql_free_result(res); > + > + return has_pk; > +} > + > +/* > get_table_structure -- retrievs database structure, prints out corresponding > CREATE statement and fills out insert_pat if the table is the type we will > be dumping. > @@ -2659,6 +3029,9 @@ static uint get_table_structure(char *table, char *db, char *table_type, > result_table= quote_name(table, table_buff, 1); > opt_quoted_table= quote_name(table, table_buff2, 0); > > + if (opt_innodb_optimize_keys && !strcmp(table_type, "InnoDB")) > + has_pk= has_primary_key(table); > + see below > if (opt_order_by_primary) > order_by= primary_key_fields(result_table); > > @@ -2838,6 +3211,9 @@ static uint get_table_structure(char *table, char *db, char *table_type, > > row= mysql_fetch_row(result); > > + if (opt_innodb_optimize_keys && !strcmp(table_type, "InnoDB")) > + skip_secondary_keys(row[1], has_pk); > + Not that it makes a lot of difference, but I'd remove the previous hunk, the variable has_pk, and put everything here, as if (opt_innodb_optimize_keys && !strcmp(table_type, "InnoDB")) skip_secondary_keys(row[1], has_primary_key(table)); this removes one if(), one strcmp(), and doesn't invoke has_primary_key() (meaning, one SQL query less) unless all other conditions for skip_secondary_keys() are met. even better - don't do has_primary_key() at all, remove the function altogether, and look for "PRIMARY KEY" in the create string inside skip_secondary_keys(). > is_log_table= general_log_or_slow_log_tables(db, table); > if (is_log_table) > row[1]+= 13; /* strlen("CREATE TABLE ")= 13 */ Regards, Sergei

1 0

Re: [Maria-developers] Features Wish-list in MariaDB 10.1.
by Kristian Nielsen 12 Jun '15

12 Jun '15

So my opinion on these below two points was requested, let's keep it on the public mailing list: > * Crash-Safe replication with filename and position (not depending on > GTIDs) > Something similar as what is described in the following link would > be good: > http://blog.booking.com/better_crash_safe_replication_for_mysql.html > * Have a way to avoid "SELECT binlog_gtid_pos" when a slave connect to > the master in MASTER_USE_GTID = no So my personal opinion on this is that I prefer to focus on improving the MariaDB GTID implementation. Considerable effort was invested into this, and the idea is that future development will focus on GTID mode, keeping non-GTID mode only as a backward compatibility. Significant effort was made to ensure that MariaDB GTID is usable for all existing use-cases of replication. For the first point, GTID replication is already crash-safe. For the second point, for GTID we in any case need a solution for locating GTID position quickly, otherwise GTID replication can have performance problems in case of frequent slave connects (and Jonas Oreland has promised a patch for this any day now ;-). And once this solution is in, the second point should be moot anyway. Of course, others may have a different opinion. For the second case, I suppose the easiest thing is just to remove the code that does SELECT binlog_gtid_pos() from the slave, if no-one cares about that functionality anyway... Hope this helps, - Kristian.

2 2

[Maria-developers] 10.0 sprint: Please advise re MDEV-8109 unexpected CAST result
by Alexander Barkov 11 Jun '15

11 Jun '15

Hi Sergei, I'm thinking about what to do with MDEV-8109 unexpected CAST result. Observations: 1. An empty string in VALUES SET sql_mode='STRICT_ALL_TABLES'; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (a DECIMAL); INSERT INTO t1 VALUES (''); ERROR 1292 (22007): Truncated incorrect DECIMAL value: '' This is OK. In strict mode INSERT with a bad value correctly returns the error. 2. CAST('' AS DECIMAL) SET sql_mode='STRICT_ALL_TABLES'; SELECT CAST('' AS DECIMAL); +---------------------+ | CAST('' AS DECIMAL) | +---------------------+ | 0 | +---------------------+ 1 row in set, 1 warning (0.00 sec) This is also OK. There are no tables involved, and CAST itself does not convert warnings to errors. 3. CAST('' AS DECIMAL) in VALUES: INSERT INTO t1 VALUES(CAST('' AS DECIMAL)); ERROR 1292 (22007): Truncated incorrect DECIMAL value: '' I think this is not OK. I wrote an explicit CAST, so I expect: - CAST to return 0 with a warning, like in #2 - INSERT to write 0 into the table normally, as I'm actually inserting the CAST result, which is 0. 4. COLUMN_GET(AS DECIMAL) in VALUES: SET @aaa = COLUMN_CREATE('price', ''); INSERT INTO t1 VALUES(COLUMN_GET(@aaa, 'price' AS DECIMAL)); ERROR 1918 (22007): Encountered illegal value '' when converting to DECIMAL This is also not OK. I wrote explicit column conversion to DECIMAL. I expect: - COLUMN_GET() to return 0 with a warning, like in #2 - INSERT to write 0 into the table normally, as I'm actually inserting the COLUMN_GET result, which is 0. Do you agree? Thanks.

2 4

Re: [Maria-developers] MariaDB replication event generators?
by Kristian Nielsen 11 Jun '15

11 Jun '15

James Cheng <jcheng(a)tivo.com> writes: > The binlog stream has all the row data, but doesn't have > everything. It doesn't have column names, for example, or primary > keys, or character sets. Using the binlog for replication seems to > assume that you have a copy of the schema beforehand on the receiving > slave. It doesn't seem possible to replicate to a slave using only the > information in the binlogs. It is a common request to be able to do 3rd-party replication from mysqld. However, the replication protocol between master and slave servers was never designed as a public API, it is based on internal data formats and so on. So unfortunately, there does not exist currently a public API for doing external replication, at least not that I am aware of. The binlog stream _does_ have everything of course, eg. in the form of CREATE TABLE statements and so on. However, being internal to the server, they are not very easy to use by an external replication mechanism. - Kristian.

1 0

Re: [Maria-developers] [Commits] 6ff2c0a: MDEV-5309 - RENAME TABLE does not check for existence of the table's engine
by Sergei Golubchik 10 Jun '15

10 Jun '15

Hi, Sergey! On Feb 09, svoj(a)mariadb.org wrote: > revision-id: 6ff2c0a05155f985484e23bd8611eb95f045907f > parent(s): f13939061d468a47985dee0268652a6fe2db7862 > committer: Sergey Vojtovich > branch nick: 10.1 > timestamp: 2015-02-09 16:12:00 +0400 > message: > > MDEV-5309 - RENAME TABLE does not check for existence of the table's engine > > When RENAME TABLE is executed, it apparently does not check whether the engine > is available (unlike ALTER TABLE .. RENAME, which does). It means that if the > engine in question was not loaded on some reason, the table might become > unusable, since the engine won't know about the change. > > With this patch RENAME TABLE fails if storage engine is not available. .... > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index e9a1ae9..f1096da 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -5172,18 +5172,19 @@ bool mysql_create_table(THD *thd, TABLE_LIST *create_table, > error= my_errno; > (void) file->ha_create_partitioning_metadata(to, from, CHF_RENAME_FLAG); > } > - else if (!file || !(error=file->ha_rename_table(from_base, to_base))) > + else if (!file) > { > + error= ER_UNKNOWN_STORAGE_ENGINE; > + } > + else if (!(error= file->ha_rename_table(from_base, to_base))) > + { > if (!(flags & NO_FRM_RENAME) && rename_file_ext(from,to,reg_ext)) > { > error= my_errno; > - if (file) > - { > if (error == ENOENT) > error= 0; // this is ok if file->ha_rename_table() succeeded > else > file->ha_rename_table(to_base, from_base); // Restore old file name > - } > } > } > delete file; May be you'd rather fix it in the caller? Let's state that the first argument of mysql_rename_table() can never be NULL. While mysql_rename_table() is called in many places, the only place where it can actually get a NULL hton is sql_rename.cc, if ha_table_exists() returns hton=NULL. I'd add the check here and simplified mysql_rename_table(). Regards, Sergei

1 0

[Maria-developers] Progress Report on [MDEV5199]: UDF Table Functions.
by Blessing Atie 10 Jun '15

10 Jun '15

Hi Serg, Colin! The following is a rundown on my progress with the above task. I started this project by getting the source of 10.1 from Github, setting my working environment by building and running the tests on ubuntu 14 then proceeded with an analysis by studying plugins and how they interact with the server.(Especially storage engines plugins). With analysis and sources investigations being done, I realized with help from assigned mentor(Serg) that our task is about implementing MYSQL_UDF_PLUGIN with the functionalities of the CONNECT engine which is similar in functionality to table functions in a way. This past week, I worked on the interface for our plugin by declaring a structure with version and some useful init/deinit function pointers to work with our UDFs(table UDF only for now). I am currently studying the Connect engine to see how I can implement its functionalities as a UDF plugin. For this week, I intend to look at other databases for example table functions that will be good candidates that we can implement with our plugin(This was not mentioned in my proposal but I intend to implement a couple of them using our plugin even if we are short of time after the final evaluation). So far, I have had difficulties in communication due to unscheduled powerouts in my city. This has really been an issue to me because my mentor was beginning to loose hope and on the other side, I don't have the chance to ask questions when im faced with a powerout. I really hope my mentors understand with me how difficult it is to code from Cameroon. Please feel free to request for any important details i might have not mentioned which you will want to know. Best Wishes, Blessing Atie

2 1

Re: [Maria-developers] MariaDB replication event generators?
by Kristian Nielsen 10 Jun '15

10 Jun '15

James Cheng <jcheng(a)tivo.com> writes: > On Jun 7, 2015, at 12:05 AM, Kristian Nielsen <knielsen(a)knielsen-hq.org> wrote: > >> James Cheng <jcheng(a)tivo.com> writes: >> >>> I recently read your old blog post from 2010 >>> (http://kristiannielsen.livejournal.com/14305.html) about some ideas >>> for MariaDB replication. In particular, your ideas for replication >>> event generators that would allow plugins to be written to receive >>> replication events. >>> >>> Did this work ever happen in MariaDB? Or anything close/equivalent? If >>> so, could you point me to something so that I can learn more about it? >> >> Unfortunately, nothing like this ever happened. > That's too bad. Are there any interesting details about why it never > happened? Was it simply due to not enough time, not high enough > priority? Was there any technical or organizational resistance to the > actual ideas or functionality? If someone were to build this today and > contribute it back to the community, do you think it would be > accepted? I think there was not enough interest and/or time/priority. For example, Galera cluster would be a prime example for something using this kind of API. But the developers did not prioritise spending time working on such API, rather they hacked in their functionality all over the server. It is the same with other parties in the community, everyone is focused on solving their own narrow problem, not in working together. And MySQL@Oracle, of course, is not interested in anything that involves working with anyone else on the code. So I don't see much chance in the current state of the community of any significant effort being spent neither on developing such an API, nor on using it. - Kristian.

1 0

Re: [Maria-developers] [Commits] 865bbbf: MDEV-3870 - Valgrind warnings on OPTIMIZE MyISAM or Aria TABLE with disabled
by Sergei Golubchik 09 Jun '15

09 Jun '15

Hi, Sergey! On Jun 09, svoj(a)mariadb.org wrote: > revision-id: 865bbbf14d1a8ed500eeeadabe0a7fdfa90420df > parent(s): b8e26b04d4ac22e29a78bca68e101176b6ef507a > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-09 16:08:09 +0400 > message: > > MDEV-3870 - Valgrind warnings on OPTIMIZE MyISAM or Aria TABLE with disabled > keys > > Fixed that OPTIMIZE TABLE against MyISAM/Aria table may write uninitialized key > root position for disabled keys. ok to push, thanks! Regards, Sergei

1 0

Re: [Maria-developers] [Commits] 7948cc3: MDEV-8260 : Issues related to concurrent CTAS
by Sergei Golubchik 09 Jun '15

09 Jun '15

Hi, Nirbhay! On Jun 08, Nirbhay Choubey wrote: > revision-id: 7948cc32b6c12c9696e8b497ae90f123cafcd234 > parent(s): 6d5b723bdc3e04978619b9673fca266e0426916f > committer: Nirbhay Choubey > branch nick: 10.0-galera.ctas > timestamp: 2015-06-08 22:50:26 -0400 > message: > > MDEV-8260 : Issues related to concurrent CTAS > > * Wait for aborted thd (victim) to release MDL locks > * Skip aborting an already aborted thd > * Defer setting OK status in case of CTAS > * Minor cosmetic changes > * Added a test case ... > +bool select_insert::send_eof() > +{ > + DBUG_ENTER("select_insert::send_eof"); > + DBUG_RETURN(prepare_eof() || send_ok_packet()); > } Please, don't do that, don't put function calls in DBUG_RETURN. This will result in dbug traces like > select_insert::send_eof < select_insert::send_eof > select_insert::prepare_eof < select_insert::prepare_eof > select_insert::send_ok_packet < select_insert::send_ok_packet while the correct trace should've be > select_insert::send_eof | > select_insert::prepare_eof | < select_insert::prepare_eof | > select_insert::send_ok_packet | < select_insert::send_ok_packet < select_insert::send_eof So, either write like bool res= prepare_eof() || send_ok_packet(); DBUG_RETURN(res); or fix DBUG_RETURN() macro. Whatever you prefer :) Otherwise the patch is ok. Regards, Sergei

2 1

[Maria-developers] Review requested for MDEV-7818
by Kristian Nielsen 09 Jun '15

09 Jun '15

I need someone to review my below fix for MDEV-7818. This involves the FLUSH TABLES WITH READ LOCK command and associated metadata locking, something that I know little about. Monty suggested the basic semantics - that all currently running parallel replication transactions would be allowed to complete before granting the global read lock. But it really needs a review of the actual implementation, by someone who has an understanding of the FTWRL semantics and implementation. Thanks, - Kristian. Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > revision-id: 0e609daea4796fafca94f7adf796addfc506168b > parent(s): 22b6c297dc0de6b176238ae820bddb5e70f2c8ab > committer: Kristian Nielsen > branch nick: mariadb > timestamp: 2015-06-09 16:01:53 +0200 > message: > > MDEV-7818: Deadlock occurring with parallel replication and FTWRL > > Problem is that FLUSH TABLES WITH READ LOCK first blocks threads from > starting new commits, then waits for running commits to complete. But > in-order parallel replication needs commits to happen in a particular > order, so this can easily deadlock. > > To fix this problem, this patch introduces a way to temporarily pause > the parallel replication worker threads. Before starting FTWRL, we let > all worker threads complete in-progress transactions, and then > wait. Then we proceed to take the global read lock. Once the lock is > obtained, we unpause the worker threads. Now commits are blocked from > starting by the global read lock, so the deadlock will no longer occur. > > --- > sql/mysqld.cc | 3 + > sql/mysqld.h | 3 + > sql/rpl_parallel.cc | 230 ++++++++++++++++++++++++++++++++++++++++++++++++++-- > sql/rpl_parallel.h | 27 ++++-- > sql/sql_parse.cc | 13 +++ > 5 files changed, 262 insertions(+), 14 deletions(-) > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index e05c0b6..af8ae2c 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -9514,6 +9514,9 @@ PSI_stage_info stage_waiting_for_prior_transaction_to_commit= { 0, "Waiting for > PSI_stage_info stage_waiting_for_prior_transaction_to_start_commit= { 0, "Waiting for prior transaction to start commit before starting next transaction", 0}; > PSI_stage_info stage_waiting_for_room_in_worker_thread= { 0, "Waiting for room in worker thread event queue", 0}; > PSI_stage_info stage_waiting_for_workers_idle= { 0, "Waiting for worker threads to be idle", 0}; > +PSI_stage_info stage_waiting_for_ftwrl= { 0, "Waiting due to global read lock", 0}; > +PSI_stage_info stage_waiting_for_ftwrl_threads_to_pause= { 0, "Waiting for worker threads to pause for global read lock", 0}; > +PSI_stage_info stage_waiting_for_rpl_thread_pool= { 0, "Waiting while replication worker thread pool is busy", 0}; > PSI_stage_info stage_master_gtid_wait_primary= { 0, "Waiting in MASTER_GTID_WAIT() (primary waiter)", 0}; > PSI_stage_info stage_master_gtid_wait= { 0, "Waiting in MASTER_GTID_WAIT()", 0}; > PSI_stage_info stage_gtid_wait_other_connection= { 0, "Waiting for other master connection to process GTID received on multiple master connections", 0}; > diff --git a/sql/mysqld.h b/sql/mysqld.h > index 156e7f9..8c953c1 100644 > --- a/sql/mysqld.h > +++ b/sql/mysqld.h > @@ -454,6 +454,9 @@ extern PSI_stage_info stage_waiting_for_prior_transaction_to_commit; > extern PSI_stage_info stage_waiting_for_prior_transaction_to_start_commit; > extern PSI_stage_info stage_waiting_for_room_in_worker_thread; > extern PSI_stage_info stage_waiting_for_workers_idle; > +extern PSI_stage_info stage_waiting_for_ftwrl; > +extern PSI_stage_info stage_waiting_for_ftwrl_threads_to_pause; > +extern PSI_stage_info stage_waiting_for_rpl_thread_pool; > extern PSI_stage_info stage_master_gtid_wait_primary; > extern PSI_stage_info stage_master_gtid_wait; > extern PSI_stage_info stage_gtid_wait_other_connection; > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 738d0e5..d90507a 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -280,6 +280,8 @@ do_gco_wait(rpl_group_info *rgi, group_commit_orderer *gco, > rpl_parallel_entry *entry= rgi->parallel_entry; > uint64 wait_count; > > + mysql_mutex_assert_owner(&entry->LOCK_parallel_entry); > + > if (!gco->installed) > { > group_commit_orderer *prev_gco= gco->prev_gco; > @@ -336,6 +338,159 @@ do_gco_wait(rpl_group_info *rgi, group_commit_orderer *gco, > } > > > +static void > +do_ftwrl_wait(rpl_group_info *rgi, > + bool *did_enter_cond, PSI_stage_info *old_stage) > +{ > + THD *thd= rgi->thd; > + rpl_parallel_entry *entry= rgi->parallel_entry; > + uint64 sub_id= rgi->gtid_sub_id; > + > + mysql_mutex_assert_owner(&entry->LOCK_parallel_entry); > + > + if (unlikely(entry->pause_sub_id > 0) && sub_id > entry->pause_sub_id) > + { > + thd->ENTER_COND(&entry->COND_parallel_entry, &entry->LOCK_parallel_entry, > + &stage_waiting_for_ftwrl, old_stage); > + *did_enter_cond= true; > + do > + { > + if (entry->force_abort || rgi->worker_error) > + break; > + if (thd->check_killed()) > + { > + thd->send_kill_message(); > + slave_output_error_info(rgi, thd); > + signal_error_to_sql_driver_thread(thd, rgi, 1); > + break; > + } > + mysql_cond_wait(&entry->COND_parallel_entry, &entry->LOCK_parallel_entry); > + } while (entry->pause_sub_id > 0 && sub_id > entry->pause_sub_id); > + } > + > + if (sub_id > entry->largest_started_sub_id) > + entry->largest_started_sub_id= sub_id; > +} > + > + > +static void > +pool_mark_busy(rpl_parallel_thread_pool *pool) > +{ > + mysql_mutex_assert_owner(&pool->LOCK_rpl_thread_pool); > + DBUG_ASSERT(!pool->busy); > + pool->busy= true; > +} > + > + > +static void > +pool_mark_not_busy(rpl_parallel_thread_pool *pool) > +{ > + mysql_mutex_assert_owner(&pool->LOCK_rpl_thread_pool); > + DBUG_ASSERT(pool->busy); > + pool->busy= false; > + mysql_cond_broadcast(&pool->COND_rpl_thread_pool); > +} > + > + > +void > +rpl_unpause_for_ftwrl(THD *thd) > +{ > + uint32 i; > + rpl_parallel_thread_pool *pool= &global_rpl_thread_pool; > + > + DBUG_ASSERT(pool->busy); > + > + for (i= 0; i < pool->count; ++i) > + { > + rpl_parallel_entry *e; > + rpl_parallel_thread *rpt= pool->threads[i]; > + > + mysql_mutex_lock(&rpt->LOCK_rpl_thread); > + if (!rpt->current_owner) > + { > + mysql_mutex_unlock(&rpt->LOCK_rpl_thread); > + continue; > + } > + e= rpt->current_entry; > + mysql_mutex_lock(&e->LOCK_parallel_entry); > + mysql_mutex_unlock(&rpt->LOCK_rpl_thread); > + e->pause_sub_id= 0; > + mysql_cond_broadcast(&e->COND_parallel_entry); > + } > + > + mysql_mutex_lock(&pool->LOCK_rpl_thread_pool); > + pool_mark_not_busy(pool); > + mysql_cond_broadcast(&pool->COND_rpl_thread_pool); > + mysql_mutex_unlock(&pool->LOCK_rpl_thread_pool); > +} > + > + > +/* > + . > + > + Note: in case of error return, unpause_for_ftwrl() must _not_ be called. > +*/ > +int > +rpl_pause_for_ftwrl(THD *thd) > +{ > + uint32 i; > + rpl_parallel_thread_pool *pool= &global_rpl_thread_pool; > + int err= 0; > + > + /* > + While the count_pending_pause_for_ftwrl counter is non-zero, the pool > + cannot be shutdown/resized, so threads are guaranteed to not disappear. > + > + This is required to safely be able to access the individual threads below. > + (We cannot lock an individual thread while holding LOCK_rpl_thread_pool, > + as this can deadlock against release_thread()). > + */ > + mysql_mutex_lock(&pool->LOCK_rpl_thread_pool); > + pool_mark_busy(pool); > + mysql_mutex_unlock(&pool->LOCK_rpl_thread_pool); > + > + for (i= 0; i < pool->count; ++i) > + { > + PSI_stage_info old_stage; > + rpl_parallel_entry *e; > + rpl_parallel_thread *rpt= pool->threads[i]; > + > + mysql_mutex_lock(&rpt->LOCK_rpl_thread); > + if (!rpt->current_owner) > + { > + mysql_mutex_unlock(&rpt->LOCK_rpl_thread); > + continue; > + } > + e= rpt->current_entry; > + mysql_mutex_lock(&e->LOCK_parallel_entry); > + mysql_mutex_unlock(&rpt->LOCK_rpl_thread); > + ++e->need_sub_id_signal; > + if (!e->pause_sub_id) > + e->pause_sub_id= e->largest_started_sub_id; > + thd->ENTER_COND(&e->COND_parallel_entry, &e->LOCK_parallel_entry, > + &stage_waiting_for_ftwrl_threads_to_pause, &old_stage); > + while (e->last_committed_sub_id < e->pause_sub_id && !err) > + { > + if (thd->check_killed()) > + { > + thd->send_kill_message(); > + err= 1; > + break; > + } > + mysql_cond_wait(&e->COND_parallel_entry, &e->LOCK_parallel_entry); > + }; > + --e->need_sub_id_signal; > + thd->EXIT_COND(&old_stage); > + if (err) > + break; > + } > + > + if (err) > + rpl_unpause_for_ftwrl(thd); > + return err; > +} > + > + > #ifndef DBUG_OFF > static int > dbug_simulate_tmp_error(rpl_group_info *rgi, THD *thd) > @@ -856,6 +1011,9 @@ handle_rpl_parallel_thread(void *arg) > > if (unlikely(entry->stop_on_error_sub_id <= rgi->wait_commit_sub_id)) > skip_event_group= true; > + if (likely(!skip_event_group)) > + do_ftwrl_wait(rgi, &did_enter_cond, &old_stage); > + > register_wait_for_prior_event_group_commit(rgi, entry); > > unlock_or_exit_cond(thd, &entry->LOCK_parallel_entry, > @@ -1058,6 +1216,47 @@ rpl_parallel_change_thread_count(rpl_parallel_thread_pool *pool, > rpl_parallel_thread **new_list= NULL; > rpl_parallel_thread *new_free_list= NULL; > rpl_parallel_thread *rpt_array= NULL; > + THD *thd; > + PSI_stage_info old_stage; > + int res; > + > + /* > + Wait here while the queue is busy. This is done to make FLUSH TABLES WITH > + READ LOCK work correctly, without incuring extra locking penalties in > + normal operation. FLUSH TABLES WITH READ LOCK needs to lock threads in the > + thread pool, and for this we need to make sure the pool will not go away > + during the operation. The LOCK_rpl_thread_pool is not suitable for > + this. It is taken by release_thread() while holding LOCK_rpl_thread; so it > + must be released before locking any LOCK_rpl_thread lock, or a deadlock > + can occur. > + > + So we protect the infrequent operations of FLUSH TABLES WITH READ LOCK and > + pool size changes with this condition wait. > + */ > + thd= current_thd; > + res= 0; > + mysql_mutex_lock(&pool->LOCK_rpl_thread_pool); > + if (thd) > + thd->ENTER_COND(&pool->COND_rpl_thread_pool, &pool->LOCK_rpl_thread_pool, > + &stage_waiting_for_rpl_thread_pool, &old_stage); > + while (pool->busy) > + { > + if (thd->check_killed()) > + { > + thd->send_kill_message(); > + res= 1; > + break; > + } > + mysql_cond_wait(&pool->COND_rpl_thread_pool, &pool->LOCK_rpl_thread_pool); > + } > + if (!res) > + pool_mark_busy(pool); > + if (thd) > + thd->EXIT_COND(&old_stage); > + else > + mysql_mutex_unlock(&pool->LOCK_rpl_thread_pool); > + if (res) > + return res; > > /* > Allocate the new list of threads up-front. > @@ -1106,7 +1305,14 @@ rpl_parallel_change_thread_count(rpl_parallel_thread_pool *pool, > */ > for (i= 0; i < pool->count; ++i) > { > - rpl_parallel_thread *rpt= pool->get_thread(NULL, NULL); > + rpl_parallel_thread *rpt; > + > + mysql_mutex_lock(&pool->LOCK_rpl_thread_pool); > + while ((rpt= pool->free_list) == NULL) > + mysql_cond_wait(&pool->COND_rpl_thread_pool, &pool->LOCK_rpl_thread_pool); > + pool->free_list= rpt->next; > + mysql_mutex_unlock(&pool->LOCK_rpl_thread_pool); > + mysql_mutex_lock(&rpt->LOCK_rpl_thread); > rpt->stop= true; > mysql_cond_signal(&rpt->COND_rpl_thread); > mysql_mutex_unlock(&rpt->LOCK_rpl_thread); > @@ -1157,7 +1363,7 @@ rpl_parallel_change_thread_count(rpl_parallel_thread_pool *pool, > } > > mysql_mutex_lock(&pool->LOCK_rpl_thread_pool); > - mysql_cond_broadcast(&pool->COND_rpl_thread_pool); > + pool_mark_not_busy(pool); > mysql_mutex_unlock(&pool->LOCK_rpl_thread_pool); > > return 0; > @@ -1182,6 +1388,9 @@ rpl_parallel_change_thread_count(rpl_parallel_thread_pool *pool, > } > my_free(new_list); > } > + mysql_mutex_lock(&pool->LOCK_rpl_thread_pool); > + pool_mark_not_busy(pool); > + mysql_mutex_unlock(&pool->LOCK_rpl_thread_pool); > return 1; > } > > @@ -1445,7 +1654,7 @@ rpl_parallel_thread::loc_free_gco(group_commit_orderer *gco) > > > rpl_parallel_thread_pool::rpl_parallel_thread_pool() > - : count(0), threads(0), free_list(0), inited(false) > + : threads(0), free_list(0), count(0), inited(false), busy(false) > { > } > > @@ -1453,9 +1662,10 @@ rpl_parallel_thread_pool::rpl_parallel_thread_pool() > int > rpl_parallel_thread_pool::init(uint32 size) > { > - count= 0; > threads= NULL; > free_list= NULL; > + count= 0; > + busy= false; > > mysql_mutex_init(key_LOCK_rpl_thread_pool, &LOCK_rpl_thread_pool, > MY_MUTEX_INIT_SLOW); > @@ -1496,8 +1706,14 @@ rpl_parallel_thread_pool::get_thread(rpl_parallel_thread **owner, > rpl_parallel_thread *rpt; > > mysql_mutex_lock(&LOCK_rpl_thread_pool); > - while ((rpt= free_list) == NULL) > + for (;;) > + { > + while (unlikely(busy)) > + mysql_cond_wait(&COND_rpl_thread_pool, &LOCK_rpl_thread_pool); > + if ((rpt= free_list) != NULL) > + break; > mysql_cond_wait(&COND_rpl_thread_pool, &LOCK_rpl_thread_pool); > + } > free_list= rpt->next; > mysql_mutex_unlock(&LOCK_rpl_thread_pool); > mysql_mutex_lock(&rpt->LOCK_rpl_thread); > @@ -1908,7 +2124,7 @@ rpl_parallel::wait_for_workers_idle(THD *thd) > > e= (struct rpl_parallel_entry *)my_hash_element(&domain_hash, i); > mysql_mutex_lock(&e->LOCK_parallel_entry); > - e->need_sub_id_signal= true; > + ++e->need_sub_id_signal; > thd->ENTER_COND(&e->COND_parallel_entry, &e->LOCK_parallel_entry, > &stage_waiting_for_workers_idle, &old_stage); > while (e->current_sub_id > e->last_committed_sub_id) > @@ -1921,7 +2137,7 @@ rpl_parallel::wait_for_workers_idle(THD *thd) > } > mysql_cond_wait(&e->COND_parallel_entry, &e->LOCK_parallel_entry); > } > - e->need_sub_id_signal= false; > + --e->need_sub_id_signal; > thd->EXIT_COND(&old_stage); > if (err) > return err; > diff --git a/sql/rpl_parallel.h b/sql/rpl_parallel.h > index 04f8706..77a4005 100644 > --- a/sql/rpl_parallel.h > +++ b/sql/rpl_parallel.h > @@ -199,12 +199,18 @@ struct rpl_parallel_thread { > > > struct rpl_parallel_thread_pool { > - uint32 count; > struct rpl_parallel_thread **threads; > struct rpl_parallel_thread *free_list; > mysql_mutex_t LOCK_rpl_thread_pool; > mysql_cond_t COND_rpl_thread_pool; > + uint32 count; > bool inited; > + /* > + While FTWRL runs, this counter is incremented to make SQL thread or > + STOP/START slave not try to start new activity while that operation > + is in progress. > + */ > + bool busy; > > rpl_parallel_thread_pool(); > int init(uint32 size); > @@ -219,6 +225,12 @@ struct rpl_parallel_entry { > mysql_mutex_t LOCK_parallel_entry; > mysql_cond_t COND_parallel_entry; > uint32 domain_id; > + /* > + Incremented by wait_for_workers_idle() and rpl_pause_for_ftwrl() to show > + that they are waiting, so that finish_event_group knows to signal them > + when last_committed_sub_id is increased. > + */ > + uint32 need_sub_id_signal; > uint64 last_commit_id; > bool active; > /* > @@ -228,12 +240,6 @@ struct rpl_parallel_entry { > */ > bool force_abort; > /* > - Set in wait_for_workers_idle() to show that it is waiting, so that > - finish_event_group knows to signal it when last_committed_sub_id is > - increased. > - */ > - bool need_sub_id_signal; > - /* > At STOP SLAVE (force_abort=true), we do not want to process all events in > the queue (which could unnecessarily delay stop, if a lot of events happen > to be queued). The stop_count provides a safe point at which to stop, so > @@ -291,6 +297,11 @@ struct rpl_parallel_entry { > The value is ULONGLONG_MAX when no error occured. > */ > uint64 stop_on_error_sub_id; > + /* > + During FLUSH TABLES WITH READ LOCK, transactions with sub_id larger than > + this value must not start, but wait until the global read lock is released. > + */ > + uint64 pause_sub_id; > /* Total count of event groups queued so far. */ > uint64 count_queued_event_groups; > /* > @@ -331,5 +342,7 @@ extern struct rpl_parallel_thread_pool global_rpl_thread_pool; > extern int rpl_parallel_activate_pool(rpl_parallel_thread_pool *pool); > extern int rpl_parallel_inactivate_pool(rpl_parallel_thread_pool *pool); > extern bool process_gtid_for_restart_pos(Relay_log_info *rli, rpl_gtid *gtid); > +extern int rpl_pause_for_ftwrl(THD *thd); > +extern void rpl_unpause_for_ftwrl(THD *thd); > > #endif /* RPL_PARALLEL_H */ > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 5635e9a..ca6b11e 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -4271,6 +4271,17 @@ case SQLCOM_PREPARE: > break; > } > > + if (lex->type & REFRESH_READ_LOCK) > + { > + /* > + We need to pause any parallel replication slave workers during FLUSH > + TABLES WITH READ LOCK. Otherwise we might cause a deadlock, as > + worker threads eun run in arbitrary order but need to commit in a > + specific given order. > + */ > + if (rpl_pause_for_ftwrl(thd)) > + goto error; > + } > /* > reload_acl_and_cache() will tell us if we are allowed to write to the > binlog or not. > @@ -4301,6 +4312,8 @@ case SQLCOM_PREPARE: > if (!res) > my_ok(thd); > } > + if (lex->type & REFRESH_READ_LOCK) > + rpl_unpause_for_ftwrl(thd); > > break; > } > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] b8e26b0: MDEV-363 - Server crashes in intern_plugin_lock on concurrent installing
by Sergei Golubchik 09 Jun '15

09 Jun '15

Hi, Sergey! On Jun 09, svoj(a)mariadb.org wrote: > revision-id: b8e26b04d4ac22e29a78bca68e101176b6ef507a > parent(s): 08b2e1ae73f6fe055391bf8437b6dff1a889c49c > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-09 13:50:43 +0400 > message: > > MDEV-363 - Server crashes in intern_plugin_lock on concurrent installing > semisync plugin and setting rpl_semi_sync_master_enabled > > There was race condition between INSTALL PLUGIN and SET. It was caused by a > gap in INSTALL PLUGIN when plugin variables were registered but not fully > initialized. Accessing such variables concurrently may reference uninitialized > memory, specifically sys_var_pluginvar::plugin. > > Fixed by initializing sys_var_pluginvar::plugin early, before variable is > registered. ok to push! Regards, Sergei

1 0

[Maria-developers] MDEV-7268 Column of table cannot be converted from type 'decimal(0, ?)' to type ' 'decimal(10, 7)'
by Alexander Barkov 09 Jun '15

09 Jun '15

Hi Sergei, Please review a patch for mdev-7268. Thanks.

2 1

Re: [Maria-developers] 1ed4683: MDEV-8260 : Issues related to concurrent CTAS
by Sergei Golubchik 09 Jun '15

09 Jun '15

Hi, Nirbhay! On Jun 02, Nirbhay Choubey wrote: > revision-id: 1ed46831ed3a1c842b45f7819edd48a233080ce4 > parent(s): 6d5b723bdc3e04978619b9673fca266e0426916f > committer: Nirbhay Choubey > branch nick: 10.0-galera.ctas > timestamp: 2015-06-02 17:59:32 -0400 > message: > > MDEV-8260 : Issues related to concurrent CTAS > > * Wait for aborted thd (victim) to release MDL locks > * Skip aborting an already aborted thd > * Defer setting OK status in case of CTAS > * Minor cosmetic changes > * Added a test case So, I'm only reviewing non-wsrep part, ok? > @@ -3735,24 +3757,16 @@ bool select_insert::send_eof() > table->file->print_error(error,MYF(0)); > DBUG_RETURN(1); > } > - char buff[160]; > - if (info.ignore) > - sprintf(buff, ER(ER_INSERT_INFO), (ulong) info.records, > - (ulong) (info.records - info.copied), > - (long) thd->get_stmt_da()->current_statement_warn_count()); > - else > - sprintf(buff, ER(ER_INSERT_INFO), (ulong) info.records, > - (ulong) (info.deleted+info.updated), > - (long) thd->get_stmt_da()->current_statement_warn_count()); > - row_count= info.copied + info.deleted + > - ((thd->client_capabilities & CLIENT_FOUND_ROWS) ? > - info.touched : info.updated); > - id= (thd->first_successful_insert_id_in_cur_stmt > 0) ? > - thd->first_successful_insert_id_in_cur_stmt : > - (thd->arg_of_last_insert_id_function ? > - thd->first_successful_insert_id_in_prev_stmt : > - (info.copied ? autoinc_value_of_last_inserted_row : 0)); > - ::my_ok(thd, row_count, id, buff); > + > + /* > + Avoid setting OK status for CREATE .. SELECT as it may fail during commit. > + */ > + if (!((thd->lex->sql_command == SQLCOM_CREATE_TABLE) && > + ((&thd->lex->select_lex)->item_list.elements))) { > + prepare_ok_info(thd); > + ::my_ok(thd, ok.row_count, ok.id, ok.message); > + } > + > DBUG_RETURN(0); > } I think we've discussed a slightly different fix, where the parent class (select_insert) wouldn't need to check thd->lex->sql_command. Like, create two methods select_insert::prepare_eof select_insert::send_ok_packet the first method will have everything of the current select_insert::send_eof but without ::my_ok part. Then select_insert::send_eof() will be just bool select_insert::send_eof() { return prepare_eof() || send_ok_packet(); } and select_create::send_eof() will do prepare_eof(), commit and other stuff, send_ok_packet(). In this case you won't need to test for thd->lex->sql_command in the parent class. And you won't need prepare_ok_info(), it'll be inside send_ok_packet(). Regards, Sergei

2 1

Re: [Maria-developers] 87088b9: MDEV-8211 plugins.server_audit fails sporadically in buildbot.
by Sergei Golubchik 08 Jun '15

08 Jun '15

Hi, Holyfoot! On Jun 08, holyfoot(a)askmonty.org wrote: > revision-id: 87088b91f7dfa49a1946454d347d823dab643569 > parent(s): 96b37035e4045a27f64810c2d946bad2d1453a80 > committer: Alexey Botchkov > branch nick: server > timestamp: 2015-06-08 21:44:13 +0500 > message: > > MDEV-8211 plugins.server_audit fails sporadically in buildbot. > This test also should be fixed - delay added so the connection > event doesn't happen before the query. > > --- > mysql-test/suite/plugins/t/thread_pool_server_audit.test | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/mysql-test/suite/plugins/t/thread_pool_server_audit.test b/mysql-test/suite/plugins/t/thread_pool_server_audit.test > index 1a2d729..2c749e2 100644 > --- a/mysql-test/suite/plugins/t/thread_pool_server_audit.test > +++ b/mysql-test/suite/plugins/t/thread_pool_server_audit.test > @@ -13,6 +13,7 @@ set global server_audit_incl_users=null; > set global server_audit_file_path='server_audit.log'; > set global server_audit_output_type=file; > set global server_audit_logging=on; > +--sleep 2 > connect (con1,localhost,root,,mysql); > connection default; > disconnect con1; This is not a good idea. First, delays make mysql-test slower. And most often - unnecessary slower, as the test succeeds without a delay too. But also delays are unreliable, on a slow box two seconds may be not enough. Don't use delays, please, in this case you can wait for the query to complete before opening a new connection. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] 5ccbfa0: MDEV-7445: Server crash with Signal 6
by Sergey Petrunia 08 Jun '15

08 Jun '15

On Thu, Apr 23, 2015 at 07:11:09PM +0200, sanja(a)mariadb.com wrote: > revision-id: 5ccbfa02d4da91c3f9080dc1b1627d19cf40eccd > parent(s): c7efc0315b51f214b6423ae4dcb4ddeb1b14e91a > committer: Oleksandr Byelkin > branch nick: server > timestamp: 2015-04-23 19:11:06 +0200 > message: > > MDEV-7445: Server crash with Signal 6 > > Problem was in rewriting left expression which had 2 references on it. Solved with making subselect reference main. > > Item_in_optimized can have not Item_in_subselect reference in left part so type casting with no check is dangerous. > > Item::cols() should be checked after Item::fix_fields(). > > --- ... > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index 998cb1c..51a38f0 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -1442,9 +1442,25 @@ bool Item_in_optimizer::eval_not_null_tables(uchar *opt_arg) > bool Item_in_optimizer::fix_left(THD *thd, Item **ref) > { > DBUG_ENTER("Item_in_optimizer::fix_left"); > - if ((!args[0]->fixed && args[0]->fix_fields(thd, args)) || > - (!cache && !(cache= Item_cache::get_cache(args[0])))) > + Item **ref0= args; > + if (args[1]->type() == Item::SUBSELECT_ITEM && > + ((Item_subselect *)args[1])->is_in_predicate()) > + { > + /* > + left_expr->fix_fields() may cause left_expr to be substituted for > + another item. (e.g. an Item_field may be changed into Item_ref). This > + transformation is undone at the end of statement execution (e.g. the > + Item_ref is deleted). However, Item_in_optimizer::args[0] may keep > + the pointer to the post-transformation item. Because of that, on the > + next execution we need to copy args[1]->left_expr again. > + */ > + ref0= &(((Item_in_subselect *)args[1])->left_expr); > + args[0]= ref0[0]; > + } > + if ((!args[0]->fixed && args[0]->fix_fields(thd, ref0)) || > + (!cache && !(cache= Item_cache::get_cache(ref0[0])))) > DBUG_RETURN(1); > + args[0]= ref0[0]; > DBUG_PRINT("info", ("actual fix fields")); > > cache->setup(args[0]); > @@ -1500,6 +1516,16 @@ bool Item_in_optimizer::fix_left(THD *thd, Item **ref) > bool Item_in_optimizer::fix_fields(THD *thd, Item **ref) > { > DBUG_ASSERT(fixed == 0); > + Item_subselect *sub= 0; > + uint col; > + > + /* > + MAX/MIN optimization can convert the subquery into > + expr + Item_singlerow_subselect > + */ (*) > + if (args[1]->type() == Item::SUBSELECT_ITEM) > + sub= (Item_subselect *)args[1]; > + I was looking for the case where the above if statement is not taken. Eventually I have applied this patch, the patch after it (e540d023e2ec6f37efc9ab695ccdfd4a6744ad64) to get the test cases, then added this at line (*): DBUG_ASSERT(args[1]->type() == Item::SUBSELECT_ITEM); and ran the entire main test suite, with and without --ps-protocol. It worked. Can you add test coverage for this case? > if (fix_left(thd, ref)) > return TRUE; > if (args[0]->maybe_null) > @@ -1507,10 +1533,10 @@ bool Item_in_optimizer::fix_fields(THD *thd, Item **ref) > > if (!args[1]->fixed && args[1]->fix_fields(thd, args+1)) > return TRUE; > - Item_in_subselect * sub= (Item_in_subselect *)args[1]; > - if (args[0]->cols() != sub->engine->cols()) > + if ((sub && ((col= args[0]->cols()) != sub->engine->cols())) || > + (!sub && (args[1]->cols() != (col= 1)))) > { > - my_error(ER_OPERAND_COLUMNS, MYF(0), args[0]->cols()); > + my_error(ER_OPERAND_COLUMNS, MYF(0), col); > return TRUE; > } > if (args[1]->maybe_null) ... BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] [Commits] 9f8a458: MDEV-7846: Server crashes in Item_subselect::fix_fields or fails with Thread stack overrun
by Sergey Petrunia 08 Jun '15

08 Jun '15

Hi Sanja, It is very difficult to understand this patch. What is the exact meaning of left_expr_orig? It is declared but has no comments. It is interesting that convert_subq_to_sj has got new code to handle scalar comparison case ( foo IN (SELECT bar ...)), while there is no handling for tuple comparison ( (foo1,foo2) IN (SELECT bar1,bar2 ..)). Is this intentional? On Thu, Apr 23, 2015 at 08:09:00PM +0200, sanja(a)mariadb.com wrote: > revision-id: 9f8a458fb2d07298810bb5d9824ce728017bfb32 > parent(s): e540d023e2ec6f37efc9ab695ccdfd4a6744ad64 > committer: Oleksandr Byelkin > branch nick: server > timestamp: 2015-04-23 20:08:57 +0200 > message: > > MDEV-7846: Server crashes in Item_subselect::fix_fields or fails with Thread stack overrun > > Substitute into transformed subselects original left expression and than register its change in case it was substituted. > > --- > mysql-test/r/subselect.result | 28 +++++++++++++++++++++++++ > mysql-test/r/subselect_no_mat.result | 28 +++++++++++++++++++++++++ > mysql-test/r/subselect_no_opts.result | 28 +++++++++++++++++++++++++ > mysql-test/r/subselect_no_scache.result | 28 +++++++++++++++++++++++++ > mysql-test/r/subselect_no_semijoin.result | 28 +++++++++++++++++++++++++ > mysql-test/t/subselect.test | 34 +++++++++++++++++++++++++++++++ > sql/item_cmpfunc.cc | 8 ++++---- > sql/item_subselect.cc | 8 +++----- > sql/item_subselect.h | 1 + > sql/opt_subselect.cc | 4 +++- > 10 files changed, 185 insertions(+), 10 deletions(-) > > diff --git a/mysql-test/r/subselect.result b/mysql-test/r/subselect.result > index 0ab2d48..cf52ba2 100644 > --- a/mysql-test/r/subselect.result > +++ b/mysql-test/r/subselect.result > @@ -7053,3 +7053,31 @@ WHERE h.host in (SELECT host FROM mysql.user) > ) AS sq > FROM mysql.host h GROUP BY h.host; > sq > +# > +# MDEV-7846:Server crashes in Item_subselect::fix > +#_fields or fails with Thread stack overrun > +# > +CREATE TABLE t1 (column1 INT) ENGINE=MyISAM; > +INSERT INTO t1 VALUES (3),(9); > +CREATE TABLE t2 (column2 INT) ENGINE=MyISAM; > +INSERT INTO t2 VALUES (1),(4); > +CREATE TABLE t3 (column3 INT) ENGINE=MyISAM; > +INSERT INTO t3 VALUES (6),(8); > +CREATE TABLE t4 (column4 INT) ENGINE=MyISAM; > +INSERT INTO t4 VALUES (2),(5); > +PREPARE stmt FROM " > +SELECT ( > + SELECT MAX( table1.column1 ) AS field1 > + FROM t1 AS table1 > + WHERE table3.column3 IN ( SELECT table2.column2 AS field2 FROM t2 AS table2 ) > +) AS sq > +FROM t3 AS table3, t4 AS table4 GROUP BY sq > +"; > +EXECUTE stmt; > +sq > +NULL > +EXECUTE stmt; > +sq > +NULL > +deallocate prepare stmt; > +drop table t1,t2,t3,t4; > diff --git a/mysql-test/r/subselect_no_mat.result b/mysql-test/r/subselect_no_mat.result > index 5780351..73a69e9 100644 > --- a/mysql-test/r/subselect_no_mat.result > +++ b/mysql-test/r/subselect_no_mat.result > @@ -7050,6 +7050,34 @@ WHERE h.host in (SELECT host FROM mysql.user) > ) AS sq > FROM mysql.host h GROUP BY h.host; > sq > +# > +# MDEV-7846:Server crashes in Item_subselect::fix > +#_fields or fails with Thread stack overrun > +# > +CREATE TABLE t1 (column1 INT) ENGINE=MyISAM; > +INSERT INTO t1 VALUES (3),(9); > +CREATE TABLE t2 (column2 INT) ENGINE=MyISAM; > +INSERT INTO t2 VALUES (1),(4); > +CREATE TABLE t3 (column3 INT) ENGINE=MyISAM; > +INSERT INTO t3 VALUES (6),(8); > +CREATE TABLE t4 (column4 INT) ENGINE=MyISAM; > +INSERT INTO t4 VALUES (2),(5); > +PREPARE stmt FROM " > +SELECT ( > + SELECT MAX( table1.column1 ) AS field1 > + FROM t1 AS table1 > + WHERE table3.column3 IN ( SELECT table2.column2 AS field2 FROM t2 AS table2 ) > +) AS sq > +FROM t3 AS table3, t4 AS table4 GROUP BY sq > +"; > +EXECUTE stmt; > +sq > +NULL > +EXECUTE stmt; > +sq > +NULL > +deallocate prepare stmt; > +drop table t1,t2,t3,t4; > set optimizer_switch=default; > select @@optimizer_switch like '%materialization=on%'; > @@optimizer_switch like '%materialization=on%' > diff --git a/mysql-test/r/subselect_no_opts.result b/mysql-test/r/subselect_no_opts.result > index 4bea029..1512e39 100644 > --- a/mysql-test/r/subselect_no_opts.result > +++ b/mysql-test/r/subselect_no_opts.result > @@ -7048,4 +7048,32 @@ WHERE h.host in (SELECT host FROM mysql.user) > ) AS sq > FROM mysql.host h GROUP BY h.host; > sq > +# > +# MDEV-7846:Server crashes in Item_subselect::fix > +#_fields or fails with Thread stack overrun > +# > +CREATE TABLE t1 (column1 INT) ENGINE=MyISAM; > +INSERT INTO t1 VALUES (3),(9); > +CREATE TABLE t2 (column2 INT) ENGINE=MyISAM; > +INSERT INTO t2 VALUES (1),(4); > +CREATE TABLE t3 (column3 INT) ENGINE=MyISAM; > +INSERT INTO t3 VALUES (6),(8); > +CREATE TABLE t4 (column4 INT) ENGINE=MyISAM; > +INSERT INTO t4 VALUES (2),(5); > +PREPARE stmt FROM " > +SELECT ( > + SELECT MAX( table1.column1 ) AS field1 > + FROM t1 AS table1 > + WHERE table3.column3 IN ( SELECT table2.column2 AS field2 FROM t2 AS table2 ) > +) AS sq > +FROM t3 AS table3, t4 AS table4 GROUP BY sq > +"; > +EXECUTE stmt; > +sq > +NULL > +EXECUTE stmt; > +sq > +NULL > +deallocate prepare stmt; > +drop table t1,t2,t3,t4; > set @optimizer_switch_for_subselect_test=null; > diff --git a/mysql-test/r/subselect_no_scache.result b/mysql-test/r/subselect_no_scache.result > index fdb3b12..26cea1f 100644 > --- a/mysql-test/r/subselect_no_scache.result > +++ b/mysql-test/r/subselect_no_scache.result > @@ -7059,6 +7059,34 @@ WHERE h.host in (SELECT host FROM mysql.user) > ) AS sq > FROM mysql.host h GROUP BY h.host; > sq > +# > +# MDEV-7846:Server crashes in Item_subselect::fix > +#_fields or fails with Thread stack overrun > +# > +CREATE TABLE t1 (column1 INT) ENGINE=MyISAM; > +INSERT INTO t1 VALUES (3),(9); > +CREATE TABLE t2 (column2 INT) ENGINE=MyISAM; > +INSERT INTO t2 VALUES (1),(4); > +CREATE TABLE t3 (column3 INT) ENGINE=MyISAM; > +INSERT INTO t3 VALUES (6),(8); > +CREATE TABLE t4 (column4 INT) ENGINE=MyISAM; > +INSERT INTO t4 VALUES (2),(5); > +PREPARE stmt FROM " > +SELECT ( > + SELECT MAX( table1.column1 ) AS field1 > + FROM t1 AS table1 > + WHERE table3.column3 IN ( SELECT table2.column2 AS field2 FROM t2 AS table2 ) > +) AS sq > +FROM t3 AS table3, t4 AS table4 GROUP BY sq > +"; > +EXECUTE stmt; > +sq > +NULL > +EXECUTE stmt; > +sq > +NULL > +deallocate prepare stmt; > +drop table t1,t2,t3,t4; > set optimizer_switch=default; > select @@optimizer_switch like '%subquery_cache=on%'; > @@optimizer_switch like '%subquery_cache=on%' > diff --git a/mysql-test/r/subselect_no_semijoin.result b/mysql-test/r/subselect_no_semijoin.result > index cb6d35d..4c6f037 100644 > --- a/mysql-test/r/subselect_no_semijoin.result > +++ b/mysql-test/r/subselect_no_semijoin.result > @@ -7048,5 +7048,33 @@ WHERE h.host in (SELECT host FROM mysql.user) > ) AS sq > FROM mysql.host h GROUP BY h.host; > sq > +# > +# MDEV-7846:Server crashes in Item_subselect::fix > +#_fields or fails with Thread stack overrun > +# > +CREATE TABLE t1 (column1 INT) ENGINE=MyISAM; > +INSERT INTO t1 VALUES (3),(9); > +CREATE TABLE t2 (column2 INT) ENGINE=MyISAM; > +INSERT INTO t2 VALUES (1),(4); > +CREATE TABLE t3 (column3 INT) ENGINE=MyISAM; > +INSERT INTO t3 VALUES (6),(8); > +CREATE TABLE t4 (column4 INT) ENGINE=MyISAM; > +INSERT INTO t4 VALUES (2),(5); > +PREPARE stmt FROM " > +SELECT ( > + SELECT MAX( table1.column1 ) AS field1 > + FROM t1 AS table1 > + WHERE table3.column3 IN ( SELECT table2.column2 AS field2 FROM t2 AS table2 ) > +) AS sq > +FROM t3 AS table3, t4 AS table4 GROUP BY sq > +"; > +EXECUTE stmt; > +sq > +NULL > +EXECUTE stmt; > +sq > +NULL > +deallocate prepare stmt; > +drop table t1,t2,t3,t4; > set @optimizer_switch_for_subselect_test=null; > set @join_cache_level_for_subselect_test=NULL; > diff --git a/mysql-test/t/subselect.test b/mysql-test/t/subselect.test > index 3eb056d..00ab48b 100644 > --- a/mysql-test/t/subselect.test > +++ b/mysql-test/t/subselect.test > @@ -5930,3 +5930,37 @@ SELECT > WHERE h.host in (SELECT host FROM mysql.user) > ) AS sq > FROM mysql.host h GROUP BY h.host; > + > + > +--echo # > +--echo # MDEV-7846:Server crashes in Item_subselect::fix > +--echo #_fields or fails with Thread stack overrun > +--echo # > +CREATE TABLE t1 (column1 INT) ENGINE=MyISAM; > +INSERT INTO t1 VALUES (3),(9); > + > +CREATE TABLE t2 (column2 INT) ENGINE=MyISAM; > + > +INSERT INTO t2 VALUES (1),(4); > + > +CREATE TABLE t3 (column3 INT) ENGINE=MyISAM; > +INSERT INTO t3 VALUES (6),(8); > + > +CREATE TABLE t4 (column4 INT) ENGINE=MyISAM; > +INSERT INTO t4 VALUES (2),(5); > + > + > +PREPARE stmt FROM " > +SELECT ( > + SELECT MAX( table1.column1 ) AS field1 > + FROM t1 AS table1 > + WHERE table3.column3 IN ( SELECT table2.column2 AS field2 FROM t2 AS table2 ) > +) AS sq > +FROM t3 AS table3, t4 AS table4 GROUP BY sq > +"; > + > +EXECUTE stmt; > +EXECUTE stmt; > + > +deallocate prepare stmt; > +drop table t1,t2,t3,t4; > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index 51a38f0..29fff8e 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -1455,12 +1455,12 @@ bool Item_in_optimizer::fix_left(THD *thd, Item **ref) > next execution we need to copy args[1]->left_expr again. > */ > ref0= &(((Item_in_subselect *)args[1])->left_expr); > - args[0]= ref0[0]; > } > - if ((!args[0]->fixed && args[0]->fix_fields(thd, ref0)) || > - (!cache && !(cache= Item_cache::get_cache(ref0[0])))) > + if ((!(*ref0)->fixed && (*ref0)->fix_fields(thd, ref0)) || > + (!cache && !(cache= Item_cache::get_cache(*ref0)))) > DBUG_RETURN(1); > - args[0]= ref0[0]; > + if (args[0] != (*ref0)) > + current_thd->change_item_tree(args, (*ref0)); > DBUG_PRINT("info", ("actual fix fields")); > > cache->setup(args[0]); > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index 4c11808..e04f5ff 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -1360,7 +1360,7 @@ Item_in_subselect::Item_in_subselect(Item * left_exp, > upper_item(0) > { > DBUG_ENTER("Item_in_subselect::Item_in_subselect"); > - left_expr= left_exp; > + left_expr_orig= left_expr= left_exp; > func= &eq_creator; > init(select_lex, new select_exists_subselect(this)); > max_columns= UINT_MAX; > @@ -1384,7 +1384,7 @@ Item_allany_subselect::Item_allany_subselect(Item * left_exp, > :Item_in_subselect(), func_creator(fc), all(all_arg) > { > DBUG_ENTER("Item_allany_subselect::Item_allany_subselect"); > - left_expr= left_exp; > + left_expr_orig= left_expr= left_exp; > func= func_creator(all_arg); > init(select_lex, new select_exists_subselect(this)); > max_columns= 1; > @@ -2584,15 +2584,13 @@ Item_in_subselect::select_in_like_transformer(JOIN *join) > arena= thd->activate_stmt_arena_if_needed(&backup); > if (!optimizer) > { > - result= (!(optimizer= new Item_in_optimizer(left_expr, this))); > + result= (!(optimizer= new Item_in_optimizer(left_expr_orig, this))); > if (result) > goto out; > } > > thd->lex->current_select= current->return_after_parsing(); > result= optimizer->fix_left(thd, optimizer->arguments()); > - /* fix_fields can change reference to left_expr, we need reassign it */ > - left_expr= optimizer->arguments()[0]; > thd->lex->current_select= current; > > if (changed) > diff --git a/sql/item_subselect.h b/sql/item_subselect.h > index 592e771..930bd66 100644 > --- a/sql/item_subselect.h > +++ b/sql/item_subselect.h > @@ -449,6 +449,7 @@ class Item_in_subselect :public Item_exists_subselect > Item **having_item); > public: > Item *left_expr; > + Item *left_expr_orig; > /* Priority of this predicate in the convert-to-semi-join-nest process. */ > int sj_convert_priority; > /* > diff --git a/sql/opt_subselect.cc b/sql/opt_subselect.cc > index 5b1a7f2..1363be0 100644 > --- a/sql/opt_subselect.cc > +++ b/sql/opt_subselect.cc > @@ -1593,7 +1593,9 @@ static bool convert_subq_to_sj(JOIN *parent_join, Item_in_subselect *subq_pred) > { > nested_join->sj_outer_expr_list.push_back(subq_pred->left_expr); > Item_func_eq *item_eq= > - new Item_func_eq(subq_pred->left_expr, subq_lex->ref_pointer_array[0]); > + new Item_func_eq(subq_pred->left_expr_orig, subq_lex->ref_pointer_array[0]); > + if (subq_pred->left_expr_orig != subq_pred->left_expr) > + thd->change_item_tree(item_eq->arguments(), subq_pred->left_expr); > item_eq->in_equality_no= 0; > sj_nest->sj_on_expr= and_items(sj_nest->sj_on_expr, item_eq); > } > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] [GSoC] MXS-2
by sriram patil 08 Jun '15

08 Jun '15

Hi All, There is no blog post this week. Could not conclude the task at hand. Will be writing a blog post for 2nd and 3rd week together. Thanks, Sriram

1 0

[Maria-developers] 5.5.44 Sprint: MDEV-8067 correct fix for MySQL Bug # 19699237: UNINITIALIZED VARIABLE IN ITEM_FIELD::STR_RESULT
by Alexander Barkov 08 Jun '15

08 Jun '15

Hi Sergei, Please review my patch for MDEV-8067. Some subselect_(mat|sj) test chanted "EXPLAIN" output. Perhaps the new results are correct. Please advise. Thanks.

1 0

[Maria-developers] Please review MDEV-7269 mysqlbinlog Don't know how to handle column type=0 meta=0 (0000)#
by Alexander Barkov 05 Jun '15

05 Jun '15

Hi Sergei, Please review a joint patch for: MDEV-7269 mysqlbinlog Don't know how to handle column type=0 meta=0 (0000)# MDEV-8267 Add /*old*/ comment into I_S.COLUMN_TYPE for old DECIMAL Thanks.

2 4

[Maria-developers] [maria-developers] MMC Snapin and WMI Provider
by Rolain Djeumen 04 Jun '15

04 Jun '15

Hello to all , I would like to work on the MariaDB MMC Snapin and WMI Provider. Is there anyone in charge of the said feature ?

2 1

Re: [Maria-developers] 91259f9: MDEV-7505 - Too large scale in DECIMAL dynamic column getter crashes mysqld
by Sergei Golubchik 04 Jun '15

04 Jun '15

Hi, svoj! On Jun 04, svoj(a)mariadb.org wrote: > revision-id: 91259f903c2ffac7c7c8fc72385a572eace4fa89 > parent(s): 9bb9bde42dbf6f936faadeaa34cde838bf6b63dd > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-04 16:04:05 +0400 > message: > > MDEV-7505 - Too large scale in DECIMAL dynamic column getter crashes mysqld > > Server may crash if sanity checks of COLUMN_GET() fail. > > COLUMN_GET() description generator expects parent CAST item, which may not have > been created due to failure of sanity checks. Then further attempt to report > an error may crash the server. > > Fixed COLUMN_GET() description generator to handle such case. > > --- > mysql-test/r/dyncol.result | 6 ++++++ > mysql-test/t/AAA.test | 1 + Mistakenly committed file ^^^^ > mysql-test/t/dyncol.test | 6 ++++++ > sql/item_strfunc.cc | 7 +++++++ > 4 files changed, 20 insertions(+) > > diff --git a/mysql-test/t/AAA.test b/mysql-test/t/AAA.test > new file mode 100644 > index 0000000..371db62 > --- /dev/null > +++ b/mysql-test/t/AAA.test > @@ -0,0 +1 @@ > +SELECT COLUMN_GET(`x`, 'y' AS DECIMAL(5,34)); > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index eedf149..7ad4c06 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -4467,6 +4467,13 @@ bool Item_dyncol_get::get_date(MYSQL_TIME *ltime, ulonglong fuzzy_date) > > void Item_dyncol_get::print(String *str, enum_query_type query_type) How do you get to Item_dyncol_get::print if cast couldn't be created? The comment below says "see create_func_dyncol_get". This function looks like if (!(res= new (thd->mem_root) Item_dyncol_get(str, num))) return res; // Return NULL return create_func_cast(thd, res, cast_type, c_len, c_dec, cs); and to get Item_dyncol_get without a parent, one needs create_func_cast to return 'res' without wrapping it in a CAST(...). But it never does that. > { > + /* Parent cast doesn't exist yet. Only print dynamic column name. */ > + if (!str->length()) Not safe, if you wrap your COLUMN_GET() in, say, CONCAT(): CONCAT(COLUMN_GET(`x`, 'y' AS DECIMAL(5,34))) then str->length() won't be 0, despite the error in CAST. > + { > + args[1]->print(str, query_type); > + return; > + } > + > /* see create_func_dyncol_get */ > DBUG_ASSERT(str->length() >= 5); > DBUG_ASSERT(strncmp(str->ptr() + str->length() - 5, "cast(", 5) == 0); Regards, Sergei

2 2

[Maria-developers] Probably in a bug in simple_pred()
by Alexander Barkov 04 Jun '15

04 Jun '15

Hi Sergey, I probably found a bug in simple_pred() in opt_sum.cc. In this piece of code: > case 3: > /* field BETWEEN const AND const */ > item= func_item->arguments()[0]->real_item(); > if (item->type() == Item::FIELD_ITEM) > { > args[0]= item; > for (int i= 1 ; i <= 2; i++) > { > item= func_item->arguments()[i]->real_item(); > if (!item->const_item()) > return 0; > args[i]= item; > if (check_item1_shorter_item2(args[0], args[1])) Should't it be if (check_item1_shorter_item2(args[0], args[i])) ? > return 0; > } > } If this is a bug, can you please help to make an SQL scrip which would be affected by this bug? Thanks.

2 3

[Maria-developers] Work on MariaDB management console snapin and WMI Provider
by Rolain Djeumen 04 Jun '15

04 Jun '15

Hello to All, I am new here , and i will like to work on the MariaDB management console snapin and WMI Provider. Who do i need to contact for that ? Thanks,

1 0

Re: [Maria-developers] [Commits] 9bb9bde: MDEV-6236 - [PATCH] mysql_tzinfo_to_sql may produce invalid SQL
by Sergei Golubchik 03 Jun '15

03 Jun '15

Hi, Sergey! On Jun 03, svoj(a)mariadb.org wrote: > revision-id: 9bb9bde42dbf6f936faadeaa34cde838bf6b63dd > parent(s): 54977bee6705b9f3ebe2e534245defdb18ac31be > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-03 14:30:09 +0400 > message: > > MDEV-6236 - [PATCH] mysql_tzinfo_to_sql may produce invalid SQL > > Factory timezone is supposed "For companies who don't want to put time > zone specification in their installation procedures. When users run > date, they'll get the message. Also useful for the "comp.sources" > version." > > This "message" is exposed as timezone abbreviation, which is supposed > to be short and thus may cause generated INSERT statements to fail. > > Do not attempt to load Factory timezone. Are you usure Factory is the only one with a message instead of a abbreviation? Perhaps it'd be safer to filter out all timezones with extra-long "abbreviations"? Regards, Sergei

2 2

Re: [Maria-developers] [Commits] 54977be: MDEV-7207 - ALTER VIEW does not change ALGORITM
by Sergei Golubchik 03 Jun '15

03 Jun '15

Hi, Sergey! On Jun 03, svoj(a)mariadb.org wrote: > revision-id: 54977bee6705b9f3ebe2e534245defdb18ac31be > parent(s): 6bd76f8b7e086ab7cf0249a05277ff552e564554 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-03 13:59:58 +0400 > message: > > MDEV-7207 - ALTER VIEW does not change ALGORITM > > Fixed that ALTER VIEW ALGORITHM=UNDEFINED behaved as if algorithm was not > specified. ok to push, thanks! Regards, Sergei

1 0

[Maria-developers] which version MDEV-6991 GROUP_MIN_MAX optimization is erroneously applied in some cases
by Alexander Barkov 02 Jun '15

02 Jun '15

Hi Sergei, I suggest to move MDEV-6691 to 10.1: Reasons: 1. It will be easier to fix it on top of: MDEV-6991 GROUP_MIN_MAX optimization is erroneously applied in some cases So fixing before 10.0 is not desirable 2. The relevant code is different in 10.1. So it's not desirable to fix before 10.1. 3. The bug is not much important Thanks.

2 3

[Maria-developers] bugs MDEV-8167 and MDEV-6973
by Alexander Barkov 02 Jun '15

02 Jun '15

Hi Sergei, Can we move these bugs from 10.0 to 10.1: (MDEV-8167) XOR returns bad results for an indexed column (MDEV-6973) XOR aggregates argument collations ? I earlier sent you a joint patch that fixes both. It moves Item_func_xor from Item_bool_func2 to Item_bool_func. But the problem is that in 10.1 I had to move Arg_comparator from Item_bool_func2 to Item_bool_rowready_func2, to fix some other bugs in LIKE handling. So in 10.1 it's now Ok for Item_func_xor to stay in Item_bool_func2. and the patch will be different! The bugs are not quite important. So I suggest not to fix them in 10.0 to avoid merge problems and confusion about Item_func_xor jumping in the hierarchy. Thanks.

2 1

[Maria-developers] Patch for bug in libmysql
by Matthew Fagan 02 Jun '15

02 Jun '15

Hi, I'm new to this list. I have a bug and a patch for it for libmysql There is a bug in the file libmariadb.c (on line 1761 in latest source). The line is as follows: vio_wait_or_timeout(net->vio, FALSE, mysql->options.connect_timeout * 100) < 1) The bug is that FALSE should be TRUE. The bug means that the mysql_real_connect does not handle connection timeout correctly. It also means that the async connect call cannot be used with an edge-triggered event system (such as WSAEventSelect in Windows).

2 1

Re: [Maria-developers] [Commits] ab7640f: my_b_fill, inline my_b_* functions instead of hairy macros
by Sergey Vojtovich 02 Jun '15

02 Jun '15

Hi Sergei, ok to push. Should we always prefer "static inline" over macroses? Regards, Sergey On Tue, May 26, 2015 at 03:38:02PM +0200, serg(a)mariadb.org wrote: > revision-id: ab7640f4d30f5a160d3a1c77efe9d2c459a3bda6 > parent(s): d916a1dc7705ae29a916e7ec91f0c76bceab708c > committer: Sergei Golubchik > branch nick: maria > timestamp: 2015-05-26 15:28:23 +0200 > message: > > my_b_fill, inline my_b_* functions instead of hairy macros > > --- > include/my_sys.h | 141 ++++++++++++++++++++++++++++++++++++---------------- > include/thr_lock.h | 1 - > mysys/mf_iocache2.c | 55 -------------------- > sql/log.cc | 1 - > sql/wsrep_binlog.cc | 4 -- > 5 files changed, 97 insertions(+), 105 deletions(-) > > diff --git a/include/my_sys.h b/include/my_sys.h > index 3d930d3..8e0e34f 100644 > --- a/include/my_sys.h > +++ b/include/my_sys.h > @@ -19,6 +19,8 @@ > > #include "my_global.h" /* C_MODE_START, C_MODE_END */ > > +#include <m_string.h> > + > C_MODE_START > > #ifdef HAVE_AIOWAIT > @@ -507,52 +509,108 @@ typedef void (*my_error_reporter)(enum loglevel level, const char *format, ...) > > extern my_error_reporter my_charset_error_reporter; > > - /* defines for mf_iocache */ > +/* inline functions for mf_iocache */ > + > +extern int my_b_flush_io_cache(IO_CACHE *info, int need_append_buffer_lock); > +extern int _my_b_get(IO_CACHE *info); > +extern int _my_b_read(IO_CACHE *info,uchar *Buffer,size_t Count); > +extern int _my_b_write(IO_CACHE *info,const uchar *Buffer,size_t Count); > > - /* Test if buffer is inited */ > -#define my_b_clear(info) (info)->buffer=0 > -#define my_b_inited(info) (info)->buffer > +/* Test if buffer is inited */ > +static inline void my_b_clear(IO_CACHE *info) { info->buffer= 0; } > +static inline int my_b_inited(IO_CACHE *info) { return MY_TEST(info->buffer); } > #define my_b_EOF INT_MIN > > -#define my_b_read(info,Buffer,Count) \ > - ((info)->read_pos + (Count) <= (info)->read_end ?\ > - (memcpy(Buffer,(info)->read_pos,(size_t) (Count)), \ > - ((info)->read_pos+=(Count)), 0) : _my_b_read((info), (Buffer), (Count))) > - > -#define my_b_write(info,Buffer,Count) \ > - ((info)->write_pos + (Count) <=(info)->write_end ?\ > - (memcpy((info)->write_pos, (Buffer), (size_t)(Count)),\ > - ((info)->write_pos+=(Count)), 0) : _my_b_write((info), (Buffer), (Count))) > - > -#define my_b_get(info) \ > - ((info)->read_pos != (info)->read_end ?\ > - ((info)->read_pos++, (int) (uchar) (info)->read_pos[-1]) :\ > - _my_b_get(info)) > - > - /* my_b_write_byte dosn't have any err-check */ > -#define my_b_write_byte(info,chr) \ > - (((info)->write_pos < (info)->write_end) ?\ > - ((*(info)->write_pos++)=(chr)) :\ > - ((my_b_flush_io_cache(info, 1)), ((*(info)->write_pos++)=(chr)))) > - > -#define my_b_tell(info) ((info)->pos_in_file + \ > - (size_t) (*(info)->current_pos - (info)->request_pos)) > -#define my_b_write_tell(info) ((info)->pos_in_file + \ > - ((info)->write_pos - (info)->write_buffer)) > - > -#define my_b_get_buffer_start(info) (info)->request_pos > -#define my_b_get_bytes_in_buffer(info) (char*) (info)->read_end - \ > - (char*) my_b_get_buffer_start(info) > -#define my_b_get_pos_in_file(info) (info)->pos_in_file > - > -/* tell write offset in the SEQ_APPEND cache */ > +static inline int my_b_read(IO_CACHE *info, uchar *Buffer, size_t Count) > +{ > + if (info->read_pos + Count <= info->read_end) > + { > + memcpy(Buffer, info->read_pos, Count); > + info->read_pos+= Count; > + return 0; > + } > + return _my_b_read(info, Buffer, Count); > +} > + > +static inline int my_b_write(IO_CACHE *info, const uchar *Buffer, size_t Count) > +{ > + if (info->write_pos + Count <= info->write_end) > + { > + memcpy(info->write_pos, Buffer, Count); > + info->write_pos+= Count; > + return 0; > + } > + return _my_b_write(info, Buffer, Count); > +} > + > +static inline int my_b_get(IO_CACHE *info) > +{ > + if (info->read_pos != info->read_end) > + { > + info->read_pos++; > + return info->read_pos[-1]; > + } > + return _my_b_get(info); > +} > + > +/* my_b_write_byte dosn't have any err-check */ > +static inline void my_b_write_byte(IO_CACHE *info, uchar chr) > +{ > + if (info->write_pos >= info->write_end) > + my_b_flush_io_cache(info, 1); > + *info->write_pos++= chr; > +} > + > +/** > + Fill buffer of the cache. > + > + @note It assumes that you have already used all characters in the CACHE, > + independent of the read_pos value! > + > + @returns > + 0 On error or EOF (info->error = -1 on error) > + # Number of characters > +*/ > +static inline size_t my_b_fill(IO_CACHE *info) > +{ > + info->read_pos= info->read_end; > + return _my_b_read(info,0,0) ? 0 : info->read_end - info->read_pos; > +} > + > +static inline my_off_t my_b_tell(const IO_CACHE *info) > +{ > + return info->pos_in_file + (*info->current_pos - info->request_pos); > +} > + > +static inline my_off_t my_b_write_tell(const IO_CACHE *info) > +{ > + return info->pos_in_file + (info->write_pos - info->write_buffer); > +} > + > +static inline uchar* my_b_get_buffer_start(const IO_CACHE *info) > +{ > + return info->request_pos; > +} > + > +static inline size_t my_b_get_bytes_in_buffer(const IO_CACHE *info) > +{ > + return info->read_end - info->request_pos; > +} > + > +static inline my_off_t my_b_get_pos_in_file(const IO_CACHE *info) > +{ > + return info->pos_in_file; > +} > + > +static inline size_t my_b_bytes_in_cache(const IO_CACHE *info) > +{ > + return *info->current_end - *info->current_pos; > +} > + > int my_b_copy_to_file(IO_CACHE *cache, FILE *file); > my_off_t my_b_append_tell(IO_CACHE* info); > my_off_t my_b_safe_tell(IO_CACHE* info); /* picks the correct tell() */ > > -#define my_b_bytes_in_cache(info) (size_t) (*(info)->current_end - \ > - *(info)->current_pos) > - > typedef uint32 ha_checksum; > extern ulong my_crc_dbug_check; > > @@ -741,24 +799,19 @@ extern my_bool reinit_io_cache(IO_CACHE *info,enum cache_type type, > my_off_t seek_offset, my_bool use_async_io, > my_bool clear_cache); > extern void setup_io_cache(IO_CACHE* info); > -extern int _my_b_read(IO_CACHE *info,uchar *Buffer,size_t Count); > extern void init_io_cache_share(IO_CACHE *read_cache, IO_CACHE_SHARE *cshare, > IO_CACHE *write_cache, uint num_threads); > extern void remove_io_thread(IO_CACHE *info); > -extern int _my_b_get(IO_CACHE *info); > extern int _my_b_async_read(IO_CACHE *info,uchar *Buffer,size_t Count); > -extern int _my_b_write(IO_CACHE *info,const uchar *Buffer,size_t Count); > extern int my_b_append(IO_CACHE *info,const uchar *Buffer,size_t Count); > extern int my_b_safe_write(IO_CACHE *info,const uchar *Buffer,size_t Count); > > extern int my_block_write(IO_CACHE *info, const uchar *Buffer, > size_t Count, my_off_t pos); > -extern int my_b_flush_io_cache(IO_CACHE *info, int need_append_buffer_lock); > > #define flush_io_cache(info) my_b_flush_io_cache((info),1) > > extern int end_io_cache(IO_CACHE *info); > -extern size_t my_b_fill(IO_CACHE *info); > extern void my_b_seek(IO_CACHE *info,my_off_t pos); > extern size_t my_b_gets(IO_CACHE *info, char *to, size_t max_length); > extern my_off_t my_b_filelength(IO_CACHE *info); > diff --git a/include/thr_lock.h b/include/thr_lock.h > index 2561709..bc916b8 100644 > --- a/include/thr_lock.h > +++ b/include/thr_lock.h > @@ -166,7 +166,6 @@ void thr_set_lock_wait_callback(void (*before_wait)(void), > void (*after_wait)(void)); > > #ifdef WITH_WSREP > -#include <my_sys.h> > typedef my_bool (* wsrep_thd_is_brute_force_fun)(void *, my_bool); > typedef int (* wsrep_abort_thd_fun)(void *, void *, my_bool); > typedef int (* wsrep_on_fun)(void *); > diff --git a/mysys/mf_iocache2.c b/mysys/mf_iocache2.c > index 06dfc9f..a642e11 100644 > --- a/mysys/mf_iocache2.c > +++ b/mysys/mf_iocache2.c > @@ -61,7 +61,6 @@ my_b_copy_to_file(IO_CACHE *cache, FILE *file) > if (my_fwrite(file, cache->read_pos, bytes_in_cache, > MYF(MY_WME | MY_NABP)) == (size_t) -1) > DBUG_RETURN(1); > - cache->read_pos= cache->read_end; > } while ((bytes_in_cache= my_b_fill(cache))); > if(cache->error == -1) > DBUG_RETURN(1); > @@ -183,60 +182,6 @@ void my_b_seek(IO_CACHE *info,my_off_t pos) > > > /* > - Fill buffer of the cache. > - > - NOTES > - This assumes that you have already used all characters in the CACHE, > - independent of the read_pos value! > - > - RETURN > - 0 On error or EOF (info->error = -1 on error) > - # Number of characters > -*/ > - > - > -size_t my_b_fill(IO_CACHE *info) > -{ > - my_off_t pos_in_file=(info->pos_in_file+ > - (size_t) (info->read_end - info->buffer)); > - size_t diff_length, length, max_length; > - > - if (info->seek_not_done) > - { /* File touched, do seek */ > - if (mysql_file_seek(info->file, pos_in_file, MY_SEEK_SET, MYF(0)) == > - MY_FILEPOS_ERROR) > - { > - info->error= 0; > - return 0; > - } > - info->seek_not_done=0; > - } > - diff_length=(size_t) (pos_in_file & (IO_SIZE-1)); > - max_length=(info->read_length-diff_length); > - if (max_length >= (info->end_of_file - pos_in_file)) > - max_length= (size_t) (info->end_of_file - pos_in_file); > - > - if (!max_length) > - { > - info->error= 0; > - return 0; /* EOF */ > - } > - DBUG_EXECUTE_IF ("simulate_my_b_fill_error", > - {DBUG_SET("+d,simulate_file_read_error");}); > - if ((length= mysql_file_read(info->file, info->buffer, max_length, > - info->myflags)) == (size_t) -1) > - { > - info->error= -1; > - return 0; > - } > - info->read_pos=info->buffer; > - info->read_end=info->buffer+length; > - info->pos_in_file=pos_in_file; > - return length; > -} > - > - > -/* > Read a string ended by '\n' into a buffer of 'max_length' size. > Returns number of characters read, 0 on error. > last byte is set to '\0' > diff --git a/sql/log.cc b/sql/log.cc > index 826a4d1..5d71732 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -6694,7 +6694,6 @@ int MYSQL_BIN_LOG::write_cache(THD *thd, IO_CACHE *cache) > return ER_ERROR_ON_WRITE; > status_var_add(thd->status_var.binlog_bytes_written, length); > > - cache->read_pos=cache->read_end; // Mark buffer used up > } while ((length= my_b_fill(cache))); > > DBUG_ASSERT(carry == 0); > diff --git a/sql/wsrep_binlog.cc b/sql/wsrep_binlog.cc > index b71da70..3788a6c 100644 > --- a/sql/wsrep_binlog.cc > +++ b/sql/wsrep_binlog.cc > @@ -71,7 +71,6 @@ int wsrep_write_cache_buf(IO_CACHE *cache, uchar **buf, size_t *buf_len) > > memcpy(*buf + *buf_len, cache->read_pos, length); > *buf_len = total_length; > - cache->read_pos = cache->read_end; > } while ((cache->file >= 0) && (length = my_b_fill(cache))); > > if (reinit_io_cache(cache, WRITE_CACHE, saved_pos, 0, 0)) > @@ -200,7 +199,6 @@ static int wsrep_write_cache_once(wsrep_t* const wsrep, > > memcpy(buf + used, cache->read_pos, length); > used = total_length; > - cache->read_pos = cache->read_end; > } while ((cache->file >= 0) && (length = my_b_fill(cache))); > > if (used > 0) > @@ -270,7 +268,6 @@ static int wsrep_write_cache_inc(wsrep_t* const wsrep, > cache->read_pos, length))) > goto cleanup; > > - cache->read_pos = cache->read_end; > } while ((cache->file >= 0) && (length = my_b_fill(cache))); > > if (WSREP_OK == err) *len = total_length; > @@ -397,7 +394,6 @@ void wsrep_dump_rbr_direct(THD* thd, IO_CACHE* cache) > WSREP_ERROR("Failed to write file '%s'", filename); > goto cleanup; > } > - cache->read_pos= cache->read_end; > } while ((cache->file >= 0) && (bytes_in_cache= my_b_fill(cache))); > if(cache->error == -1) > { > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 1

Re: [Maria-developers] [Commits] 68b5057: always use my_b_pread() instead of mysql_file_pread()
by Sergey Vojtovich 02 Jun '15

02 Jun '15

Hi Sergei, ok to push. Regards, Sergey On Tue, May 26, 2015 at 03:38:11PM +0200, serg(a)mariadb.org wrote: > revision-id: 68b5057695165b88db38d5090f45ebe3a5149995 > parent(s): ab7640f4d30f5a160d3a1c77efe9d2c459a3bda6 > committer: Sergei Golubchik > branch nick: maria > timestamp: 2015-05-26 15:28:24 +0200 > message: > > always use my_b_pread() instead of mysql_file_pread() > > when working with IO_CACHE's, don't access IO_CACHE::file directly

1 0

[Maria-developers] MDEV-7502 Automatic provisioning of slave
by Martin Kaluznik 02 Jun '15

02 Jun '15

Provisioning has to be triggered by a command from slave, so my implementation started there. After performing initial checks, command handler starts IO and SQL threads, like during regular replication. I was trying to write separate code for provisioning IO thread. Main benefit would be clean code to start with – no backward compatibility / bug workaround code – old master will not be able to provide provisioning functionality anyway. On the other hand there would be too much duplicate code, and having to do future changes on two different places is not acceptable. I have decided to use regular replication IO thread as Kristian Nielsen suggested before and add special handling where necessary. Now when the IO thread requests binlog dump from master, it sends packet with following structure master position | binlog flags | server id | log name Master needs to receive information, that slaves IO thread is expecting provisioning data along with current binlog. I have created new BINLOG_PROVISIONING_MODE flag, but I believe that the name is incorrect and it doesn’t belong to binlog flags field. My question is, what is better for compatibility, change binlog flags to any flags and use it for sending information about provisioning request to master (as in current implementation) or add another byte to packet after log name containing provisioning mode flag and possibly more flags later. My current implementation progress can be seen at https://github.com/f4rnham/server/commits/MDEV-7502 diff of all commits at once https://github.com/MariaDB/server/compare/10.1...f4rnham:MDEV-7502 There is one more decision I am not sure about. What should result of provisioning on non-empty slave be? For example, slave contains table with same name in same database as master, but with different columns. Should slave detect already existing tables and fail with error, or silently drop everything what could cause conflicts (add optional FORCE part of provisioning start command?). Which one from these solutions should be default? Or maybe there is third one and better. Thanks for comments, Martin

2 1

[Maria-developers] [GSoC] MXS-2: Week 1
by sriram patil 01 Jun '15

01 Jun '15

Hi All, Please find links to my first blog posts of this GSoC season below. MaxScale Installation and Configuration <http://srirampatil.tumblr.com/post/120346691880/maxscale-installation-and-c…> MXS-2: Week 1 <http://srirampatil.tumblr.com/post/120349951405/mxs-2-week-1> Feel fre to point out mistakes, if any. Also, suggestions ans/or reviews are welcome! Thanks, Sriram

1 0

Re: [Maria-developers] [Commits] 52c3687: MDEV-8241: Debug build on Windows is broken: error LNK2019: unresolved external symbol pthread_detach referenced in function ma_checkpoint_init
by Sergei Golubchik 29 May '15

29 May '15

Hi, Sanja! On May 28, sanja(a)mariadb.com wrote: > revision-id: 52c36874e251c3ac5ff0f925c8133074166d481d > parent(s): 979c5049ef6fdd55c29fa5170a96c62d2a8a6823 > committer: Oleksandr Byelkin > branch nick: server > timestamp: 2015-05-28 07:59:57 +0200 > message: > > MDEV-8241: Debug build on Windows is broken: error LNK2019: unresolved external symbol pthread_detach referenced in function ma_checkpoint_init > > pthread_detach() replaced with pthread_detach_this_thread() > > pthread_detach_this_thread() definition fixed > > diff --git a/storage/maria/ma_checkpoint.c b/storage/maria/ma_checkpoint.c > index 2268ca6..36b729e 100644 > --- a/storage/maria/ma_checkpoint.c > +++ b/storage/maria/ma_checkpoint.c > @@ -340,7 +340,6 @@ int ma_checkpoint_init(ulong interval) > { > /* thread lives, will have to be killed */ > checkpoint_control.status= THREAD_RUNNING; > - pthread_detach(th); > } > } > DBUG_RETURN(res); > @@ -574,6 +573,8 @@ pthread_handler_t ma_checkpoint_background(void *arg) > sleeps= 1; > pages_to_flush_before_next_checkpoint= 0; > > + pthread_detach_this_thread(); > + > for(;;) /* iterations of checkpoints and dirty page flushing */ > { How does that help? What was wrong with pthread_detach()? Regards, Sergei

2 1

Re: [Maria-developers] [Commits] 2b17bfc: MDEV-7484: Log Time not consistent with InnoDB errors nor with MySQL error log time format
by Sergei Golubchik 29 May '15

29 May '15

Hi, Jan! On May 22, Jan Lindström wrote: > revision-id: 2b17bfc6e14283aefeff87d959f3ecfc48af5c2c > parent(s): 8635c4b4e66712558330d9534491af0c9025e4dc > committer: Jan Lindström > branch nick: 10.1-encrypt > timestamp: 2015-05-22 08:24:59 +0300 > message: > > MDEV-7484: Log Time not consistent with InnoDB errors nor with MySQL error log time format Ok to push Regards, Sergei

1 0

[Maria-developers] Please review: MDEV-8222 "string_field LIKE int_const" returns a wrong result in case of UCS2
by Alexander Barkov 29 May '15

29 May '15

Hi Sergey, please review my patch for MDEV-8222. It also moves Arg_compatator from Item_bool_func2 to Item_bool_rowready_func2. Thanks.

1 0

Re: [Maria-developers] [Commits] 7819ddb: MDEV-8192 - THD::set_command() takes 0.05% in OLTP RO
by Sergei Golubchik 29 May '15

29 May '15

Hi, Sergey! On May 20, svoj(a)mariadb.org wrote: > revision-id: 7819ddb99be5dff26c8a2109ba70a54289a950ad > parent(s): fcb48ee418959ea2c05d528266bb2dcad5bce46b > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-05-20 16:55:20 +0400 > message: > > MDEV-8192 - THD::set_command() takes 0.05% in OLTP RO > > Moved THD::set_command() definition to sql_class.h so that it can be inlined. ok > Call PSI_server::set_thread_command() conditionally: it is cheaper than calling > dummy function when PFS is off (though it adds some overhead when PFS is on). I'm not sure about it. P_S will be used more and more over time, let's not intentionally penalize it. Regards, Sergei

2 2

Re: [Maria-developers] [Commits] 6efaf34: MDEV-7011: MAX_STATEMENT_TIME has no effect in a procedure after a previous successful statement
by Sergei Golubchik 29 May '15

29 May '15

Hi, Sanja! On May 21, sanja(a)mariadb.com wrote: > revision-id: 6efaf349adec30115bdb94b9a9927f7c5103ed99 > parent(s): 3e55ef26d49a900782d2c2bb2c03470faed6ec9d > committer: Oleksandr Byelkin > branch nick: server > timestamp: 2015-05-21 16:31:24 +0200 > message: > > MDEV-7011: MAX_STATEMENT_TIME has no effect in a procedure after a previous successful statement > > Do not reset timer inside stored procedures and functions. > > diff --git a/sql/sql_class.h b/sql/sql_class.h > index ca532ae..ebb76df 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3966,6 +3966,8 @@ class THD :public Statement, > void reset_query_timer() > { > #ifndef EMBEDDED_LIBRARY > + if (spcont || in_sub_stmt) > + return; > if (!query_timer.expired) > thr_timer_end(&query_timer); > #endif I'd expect reset_query_timer() to use the same condition that set_query_timer() does: void set_query_timer() { if (!variables.max_statement_time || spcont || in_sub_stmt || slave_thread || query_timer.expired == 0) return; thr_timer_settime(&query_timer, variables.max_statement_time); } So, I'd think that reset_query_timer() could look like void reset_query_timer() { if (spcont || in_sub_stmt || slave_thread) return; if (!query_timer.expired) thr_timer_end(&query_timer); } Regards, Sergei

2 1

[Maria-developers] [GSoC] And it begins
by sriram patil 29 May '15

29 May '15

Hi All, Today is the official day (at least in my timezone) to start working on the GSoC'15 projects. I am totally excited! I will be working on MaxScale, MXS-2, for next 3 months. I will be posting a blog link every Monday starting from next week which will give an idea about what I did in the week previous to that and the tasks lined up for that week. I hope you all follow my blog. It will be great to get some inputs from the community as well. Also, all the best to all the other students who will be working with MariaDB this summer. Thanks, Sriram

2 2

Re: [Maria-developers] 8928497: MDEV-7067: Server outputs Galera (WSREP) information, even if Galera is disabled
by Sergei Golubchik 29 May '15

29 May '15

Hi, Nirbhay! On May 09, Nirbhay Choubey wrote: > revision-id: 8928497acbe3e17a105897f405d5e5138d57a249 > parent(s): 3674c363a7e77e534c3e0d28659dc614c53fbcbc > committer: Nirbhay Choubey > branch nick: 10.1-wsrep > timestamp: 2015-05-09 16:11:59 -0400 > message: > > MDEV-7067: Server outputs Galera (WSREP) information, even if Galera is disabled > > * mysqld_safe: Since wsrep_on variable is mandatory in 10.1, skip wsrep > position recovery if its OFF. > * mysqld: Remove "-wsrep" from server version > * mysqld: Remove wsrep patch version from @@version_comment > * mysqld: Introduce @@wsrep_patch_version Looks ok, I just didn't like the wsrep.cmake change. See below. > diff --git a/cmake/wsrep.cmake b/cmake/wsrep.cmake > index c37cf74..305ab27 100644 > --- a/cmake/wsrep.cmake > +++ b/cmake/wsrep.cmake > @@ -28,29 +28,22 @@ OPTION(WITH_WSREP "WSREP replication API (to use, e.g. Galera Replication librar > # Set the patch version > SET(WSREP_PATCH_VERSION "10") > > -# MariaDB addition: Revision number of the last revision merged from > -# codership branch visible in @@version_comment. > -# Branch : codership-mysql/5.6 > -SET(WSREP_PATCH_REVNO "4144") # Should be updated on every merge. > - > -# MariaDB addition: Revision number of the last revision merged from > -# Branch : lp:maria/maria-10.0-galera > -SET(WSREP_PATCH_REVNO2 "3919") # Should be updated on every merge. > - > -# MariaDB: Obtain patch revision number: > -# Update WSREP_PATCH_REVNO if WSREP_REV environment variable is set. > -IF (DEFINED ENV{WSREP_REV}) > - SET(WSREP_PATCH_REVNO $ENV{WSREP_REV}) > -ENDIF() > > -SET(WSREP_INTERFACE_VERSION 25) > +# Obtain wsrep API version > +EXECUTE_PROCESS( > + COMMAND sh -c "grep WSREP_INTERFACE_VERSION ${MySQL_SOURCE_DIR}/wsrep/wsrep_api.h | cut -d '\"' -f 2" > + OUTPUT_VARIABLE WSREP_API_VERSION > + RESULT_VARIABLE RESULT > +) I don't like calling grep|cut from cmake files. If you want to extract the version from wsrep_api.h, use cmake commands, e.g. FILE(STRINGS "${MySQL_SOURCE_DIR}/wsrep/wsrep_api.h" WSREP_API_H LIMIT_COUNT 1 REGEX "WSREP_INTERFACE_VERSION") STRING(REGEX MATCH .... > +STRING(REGEX REPLACE "(\r?\n)+$" "" WSREP_API_VERSION "${WSREP_API_VERSION}") > > -SET(WSREP_VERSION > - "${WSREP_INTERFACE_VERSION}.${WSREP_PATCH_VERSION}.r${WSREP_PATCH_REVNO}") > +SET(WSREP_VERSION "${WSREP_API_VERSION}.${WSREP_PATCH_VERSION}" > + CACHE STRING "WSREP version") make it INTERNAL, not STRING > SET(WSREP_PROC_INFO ${WITH_WSREP}) > > IF(WITH_WSREP) > - SET(COMPILATION_COMMENT "${COMPILATION_COMMENT}, wsrep_${WSREP_VERSION}") > + SET(WSREP_PATCH_VERSION "wsrep_${WSREP_VERSION}") > ENDIF() > > diff --git a/mysql-test/suite/sys_vars/r/sysvars_wsrep.result b/mysql-test/suite/sys_vars/r/sysvars_wsrep.result > index 9c392b1..1763078 100644 > --- a/mysql-test/suite/sys_vars/r/sysvars_wsrep.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_wsrep.result > @@ -365,6 +365,20 @@ NUMERIC_BLOCK_SIZE NULL > ENUM_VALUE_LIST TOI,RSU > READ_ONLY NO > COMMAND_LINE_ARGUMENT OPTIONAL > +VARIABLE_NAME WSREP_PATCH_VERSION > +SESSION_VALUE NULL > +GLOBAL_VALUE wsrep_25.10 > +GLOBAL_VALUE_ORIGIN COMPILE-TIME > +DEFAULT_VALUE NULL > +VARIABLE_SCOPE GLOBAL > +VARIABLE_TYPE VARCHAR > +VARIABLE_COMMENT wsrep patch version > +NUMERIC_MIN_VALUE NULL > +NUMERIC_MAX_VALUE NULL > +NUMERIC_BLOCK_SIZE NULL > +ENUM_VALUE_LIST NULL > +READ_ONLY YES > +COMMAND_LINE_ARGUMENT NULL > VARIABLE_NAME WSREP_PROVIDER > SESSION_VALUE NULL > GLOBAL_VALUE none > diff --git a/mysql-test/suite/sys_vars/r/wsrep_patch_version_basic.result b/mysql-test/suite/sys_vars/r/wsrep_patch_version_basic.result > new file mode 100644 > index 0000000..80f29e6 > --- /dev/null > +++ b/mysql-test/suite/sys_vars/r/wsrep_patch_version_basic.result I've actually stopped creating sys_var tests for every new variable (even though I was the one who introduced sys_vars.all_vars test in the first place :). With the new I_S.SYSTEM_VARIABLES table we have all variables listed there anyway, with all properties. So there's no need to copy these boilerplate tests over and over. > diff --git a/mysql-test/suite/wsrep/suite.pm b/mysql-test/suite/wsrep/suite.pm > index 02dff24..33744a6 100644 > --- a/mysql-test/suite/wsrep/suite.pm > +++ b/mysql-test/suite/wsrep/suite.pm > @@ -27,6 +27,7 @@ push @::global_suppressions, > qr(WSREP: Could not open saved state file for reading: ), > qr(WSREP: option --wsrep-casual-reads is deprecated), > qr(WSREP: --wsrep-casual-reads=ON takes precedence over --wsrep-sync-wait=0), > + qr|WSREP: access file$gvwstate.dat$ failed$No such file or directory$|, why? > ); > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 14d904b..49a639b 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -1902,13 +1902,15 @@ static void __cdecl kill_server(int sig_ptr) > } > #endif > > - if (WSREP_ON) > + /* Stop wsrep threads in case they are running. */ > wsrep_stop_replication(NULL); Does this print any messages? Regards, Sergei

2 1

Re: [Maria-developers] [Commits] 6f03ea7: MDEV-8202 - st_select_lex::master_unit() takes 0.17% in OLTP RO
by Sergei Golubchik 28 May '15

28 May '15

Hi, Sergey! On May 21, svoj(a)mariadb.org wrote: > revision-id: 6f03ea78aa4a70b9a9f2f096f1d7a6c282ca9575 > parent(s): 79372cf58e426eb60acb9f0abf862a1f0fe0e4e3 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-05-21 19:35:49 +0400 > message: > > MDEV-8202 - st_select_lex::master_unit() takes 0.17% in OLTP RO > > To make st_select_lex::master_unit() inlinable: > - moved it's definition to sql_lex.h > - removed base class virtual master_unit() declaration since this method is > specific to st_select_lex ok to push Regards, Sergei

1 0

[Maria-developers] MDEV-7950 Item_func::type() takes 0.26% in OLTP RO
by Alexander Barkov 28 May '15

28 May '15

Hello Sergey, Please review the next step for MDEV-7950. It does the following: - Adds specific implementations of virtual method get_mm_tree() into Item_func_like, Item_bool_rowready_func2, Item_func_spatial_rel - Gets rid of two virtual calls: select_optimize() and have_rev_func(). - Removes virtual methods Item_func::select_optimize(), Item_func::have_rev_func(), as well as enum Item_func::optimize_type, because they are not needed any more. Thanks.

2 1

Re: [Maria-developers] [Commits] 8a595a9: cleanup: LOAD DATA replication support in IO_CACHE
by Sergey Vojtovich 28 May '15

28 May '15

Hi Sergei, a few thoughts/questions inline. On Tue, May 26, 2015 at 03:37:36PM +0200, serg(a)mariadb.org wrote: > revision-id: 8a595a9825de286ad073f6f043db4c0dce88a2ea > parent(s): 3e55ef26d49a900782d2c2bb2c03470faed6ec9d > committer: Sergei Golubchik > branch nick: maria > timestamp: 2015-05-26 15:09:25 +0200 > message: > > cleanup: LOAD DATA replication support in IO_CACHE > > remove some 14-year old code that added support for > LOAD DATA replication to IO_CACHE: > * three callbacks, of which only two were actually used and that > were only needed for LOAD DATA replication but were > tested in every IO_CACHE instance > * an additional opaque void * argument in IO_CACHE, also only > used for LOAD DATA replication, but present everywhere > * the code to close IO_CACHE prematurely in LOAD DATA to have > these callbacks called in the correct order and a long > comment explaining what will happen if IO_CACHE is not > closed prematurely > * a variable to track whether IO_CACHE was closed prematurely > (to avoid double-closing it) > > this also fixes a bug with replication of LOAD DATA LOCAL > (IO_CACHE of READ_NET type was not calling these callbacks at all) I couldn't track this down. That is GET/my_b_get/pre_read was called in both cases as well as end_io_cache/pre_close. Could you elaborate? ...skip... > diff --git a/sql/sql_load.cc b/sql/sql_load.cc > index 18982c5..62b6e17 100644 > --- a/sql/sql_load.cc > +++ b/sql/sql_load.cc > @@ -1410,20 +1372,15 @@ READ_INFO::READ_INFO(File file_par, uint tot_length, CHARSET_INFO *cs, > } > else > { > - /* > - init_io_cache() will not initialize read_function member > - if the cache is READ_NET. So we work around the problem with a > - manual assignment > - */ > - need_end_io_cache = 1; > - > #ifndef EMBEDDED_LIBRARY > if (get_it_from_net) > cache.read_function = _my_b_net_read; > > if (mysql_bin_log.is_open()) > - cache.pre_read = cache.pre_close = > - (IO_CACHE_CALLBACK) log_loaded_block; > + { > + cache.real_read_function= cache.read_function; > + cache.read_function= log_loaded_block; > + } > #endif > } > } Will this compile with replication disabled? OTOH I guess it didn't compile even before: log_loaded_block() seem to be unavailable when replication is disabled. IOCACHE::read_function may change, e.g. on error in _my_b_async_read(). Shouldn't we handle this? > @@ -1432,8 +1389,7 @@ READ_INFO::READ_INFO(File file_par, uint tot_length, CHARSET_INFO *cs, > > READ_INFO::~READ_INFO() > { > - if (need_end_io_cache) > - ::end_io_cache(&cache); > + ::end_io_cache(&cache); > my_free(buffer); > List_iterator<XML_TAG> xmlit(taglist); > XML_TAG *t; Hmm... I may miss something, but shouldn't we call log_loaded_block() for last successfully read block? Thanks, Sergey

2 2

Re: [Maria-developers] [Commits] 1cd9319: myisam/aria: don't mess with IO_CACHE::file
by Sergei Golubchik 28 May '15

28 May '15

Hi, Sergey! On May 27, Sergey Vojtovich wrote: > Hi Sergei, > > I'm lost now. In Aria you did change very similar code and kept > init_io_cache(-1) while init_io_cache(info->dfile) was moved down > and replaced with init_io_cache(new_file). This sounds reasonable. > FWICS MyISAM has very same code. > > Isn't "temp" in fact info->dfile and not -1? Ok, the original problem. init_io_cache(-1) means that IO_CACHE uses a temporary file as a backing store. That is, when in-memory buffer is full, IO_CACHE creates a temporary file to keep the data. MyISAM in a couple of places was creating IO_CACHE with init_io_cache(-1) and later was replacing IO_CACHE::file. That is, IO_CACHE was created as if it was a temporary storage, but later changed to use a real file. Which is not a good idea now, as I (in a later changeset) detect temporary IO_CACHE's in init_io_cache. So, I've fixed MyISAM/Aria not to access internal IO_CACHE fields and create IO_CACHE to have the correct file descriptior from the start. Regards, Sergei

2 1

Re: [Maria-developers] [Commits] 1cd9319: myisam/aria: don't mess with IO_CACHE::file
by Sergei Golubchik 27 May '15

27 May '15

Hi, Sergey! On May 27, Sergey Vojtovich wrote: > Hi Sergei, > > looks good. Ok to push, a few minor suggestions inline. > > > diff --git a/storage/maria/ma_check.c b/storage/maria/ma_check.c > > index 8c012d2..d50528e 100644 > > --- a/storage/maria/ma_check.c > > +++ b/storage/maria/ma_check.c > > @@ -4267,20 +4267,22 @@ int maria_repair_parallel(HA_CHECK *param, register MARIA_HA *info, > > > > if (!(sort_info.key_block= > > alloc_key_blocks(param, (uint) param->sort_key_blocks, > > - share->base.max_key_block_length)) || > > - init_io_cache(&param->read_cache, info->dfile.file, > > + share->base.max_key_block_length))) > > + goto err; > > + > > + if (init_io_cache(&param->read_cache, info->dfile.file, > > (uint) param->read_buffer_length, > > - READ_CACHE, share->pack.header_length, 1, MYF(MY_WME)) || > > - (!rep_quick && > > - (init_io_cache(&info->rec_cache, info->dfile.file, > > - (uint) param->write_buffer_length, > > - WRITE_CACHE, new_header_length, 1, > > - MYF(MY_WME | MY_WAIT_IF_FULL) & param->myf_rw) || > > - init_io_cache(&new_data_cache, -1, > > - (uint) param->write_buffer_length, > > - READ_CACHE, new_header_length, 1, > > - MYF(MY_WME | MY_DONT_CHECK_FILESIZE))))) > > + READ_CACHE, share->pack.header_length, 1, MYF(MY_WME))) > > goto err; > > + > > + if (!rep_quick) > > + { > > + if (init_io_cache(&new_data_cache, -1, > > + (uint) param->write_buffer_length, > > + READ_CACHE, new_header_length, 1, > > + MYF(MY_WME | MY_DONT_CHECK_FILESIZE))) > > + goto err; > > + } > Why not to move new_data_cache initialization down along with rec_cache? I've simply kept it where it was, the goal was to avoid io_cache.file=new_file_descriptor; for example, there was info->rec_cache.file=new_file; and I moved info->rec_cache initialization down where new_file was known and could be passed as an argument to init_io_cache(). There was no such problem with new_data_cache. Regards, Sergei

2 2

[Maria-developers] step7 (cleanup) MDEV-7950 Item_func::type() takes 0.26% in OLTP RO
by Alexander Barkov 27 May '15

27 May '15

Hi Sergey, please review the next step for MDEV-7950. It breaks get_mm_parts() into a virtual method in Item, so - replaces one virtual call item->type() to another virtual call, item->get_mm_tree(). - and also removes this virtual call for functype(), which used to distinguish between COND_AND_FUNC and COND_OR_FUNC: >- if (((Item_cond*) cond)->functype() == Item_func::COND_AND_FUNC) The next patches will: 1. Split the code in Item::get_mm_tree() responsible for functions into methods in Item_func and its descendants: Item_func Item_func_like, to get rid of the select_optimize() call Item_func_between Item_func_in Item_equal 2. Remove virtual method Item_func::select_optimize(), as well as enum Item_func::optimize_type, because they won't be needed any more. Thanks.

2 3

[Maria-developers] Memory barrier problem in InnoDB/XtraDB mutex implementation causing hangs
by Kristian Nielsen 27 May '15

27 May '15

I have been debugging some test failures seen in Buildbot: https://mariadb.atlassian.net/browse/MDEV-7026 After some debugging, this is what I think is going on. I will refer to MariaDB 5.5 sources, though the problem is also in 10.0. InnoDB/XtraDB mutexes are taken in mutex_enter_func() and unlocked in mutex_exit_func(). Simplifying for clarity, here are the relevant parts of the logic: ENTER: # First spin wait, hoping to get the mutex without yielding. os_compare_and_swap_ulint(&mutex->waiters, 0, 1); if (os_atomic_test_and_set_byte(&mutex->lock_word, 1)) sync_array_wait_event() RELEASE: os_atomic_lock_release_byte(&mutex->lock_word); os_rmb; waiters = *(volatile *)&mutex->waiters; if (waiters) mutex_signal_object(mutex); The interesting point here is how to ensure that we do not lose the wakeup of a waiting thread when the holding thread releases the mutex. The idea is that the waiting thread first sets mutex->waiters=1. Then it checks if the mutex is free - if not, it goes to sleep. The thread that releases the lock first marks the mutex free. Then it checks if there are any waiters, and if so wakes them up. For this to work, we need to ensure that the setting of waiters=1 has completed before the load of lock_word can start - a full memory barrier. Likewise, we need the setting of lock_word to 0 to complete before the load of mutex->waiters - again a full memory barrier. Otherwise we can get something like this: ENTER RELEASE Start store to mutex->waiters Start store to lock_word Load mutex->lock_word Store to lock_word becomes visible Load mutex->waiters Store to waiters becomes visible Check lock word==1 Check waiters==0, skip wakeup Sleep (forever) Or this: ENTER RELEASE Start store to lock_word Start store to mutex->waiters Load mutex->waiters Store to waiters becomes visible Load mutex->lock_word Store to lock_word becomes visible Check lock word==1 Sleep (forever) Check waiters==0, skip wakeup (So both full barriers are needed, the one in ENTER to prevent the first example, the one in RELEASE to prevent the second). It looks like for x86, the full barriers were broken in MariaDB 5.5.40 and 10.0.13. os_compare_and_swap_ulint() is defined as either __sync_bool_compare_and_swap() or atomic_cas_ulong(), both of which implies a full barrier. So ENTER looks ok. os_atomic_lock_release_byte() however is now defined as __sync_lock_release(). This is _not_ a full barrier, just a release barrier. So the RELEASE case seems to be incorrect, allowing the second deadlock example to occur. Previously, RELEASE used os_atomic_test_and_set_byte(), which is __sync_lock_test_and_set(). This is defined in GCC docs as an "aquire" barrier only, which is also wrong, obviously. But it happens to generate an XCHG instruction on x86, which is a full barrier. This is probably why it works on x86 (and probably why it did not work on PowerPC). Here is a small test program, and the assembly from GCC 5.7 on amd64: void test(unsigned *p, unsigned *q) { while (__sync_lock_test_and_set(q, 1)) { } *p++; __sync_lock_release(q); } test: movl $1, %edx .L2: movl %edx, %eax xchgl (%rsi), %eax testl %eax, %eax jne .L2 movl $0, (%rsi) ret As can be seen, the __sync_lock_test_and_set() generates a full barrier, but the __sync_lock_release() generates nothing (as this is sufficient for release semantics for the rather strong memory order on x86). So the InnoDB mutexes look completely broken from MariaDB 5.5.40 and 10.0.13 :-( This is consistent with the observations in MDEV-7026. A thread is waiting for wakeup on a mutex that is already marked free. Apparently this is not a new problem. This comment in mutex_exit_func() makes me want to cry: /* A problem: we assume that mutex_reset_lock word is a memory barrier, that is when we read the waiters field next, the read must be serialized in memory after the reset. A speculative processor might perform the read first, which could leave a waiting thread hanging indefinitely. Our current solution call every second sync_arr_wake_threads_if_sema_free() to wake up possible hanging threads if they are missed in mutex_signal_object. */ So someone found the mutex implementation broken, but instead of fixing it, they implemented a horrible workaround to try and break any deadlocks that might occur. (The reason that the deadlocks are not broken in the MDEV-7026 seems to be that it occurs during startup, before starting the srv_error_monitor_thread() that does the sync_arr_wake_threads_if_sema_free()). Now, what is also interesting is that this seems to explain very nicely the many server lockups we have seen for users that upgrade to recent MariaDB versions. I was doing a very detailed investigation of one such case where the user was able to supply unusually detailed information. I found a thread that was stuck waiting on a mutex, but no other threads seemed to be holding that mutex. This is exactly what would be seen in case of a lost wakeup due to this issue. Furthermore, the sync_arr_wake_threads_if_sema_free() mechanism, as well as the normal InnoDB "long semaphore wait", was itself blocked on that same mutex wait, which explains why they were not effective for the user in that particular case. So it looks like we need to revert the incorrect patch and release new 5.5.40a and 10.0.15 releases ASAP, to solve these hangs for users. And do a proper patch for PowerPC with proper full memory barriers. (Incidentally, the os_rmb in RELEASE seems to also be part of some PowerPC patch, it also looks completely wrong). However, more importantly, this issue clearly shows some fundamental problems in our development procedures that we need to fix ASAP. Just look at it. We have users (and customers!) that suffer from strange hangs. This is their production systems that are down. Support and developers are unable to help them due to the difficulty of debugging rare race conditions in production systems. And all the time the bug was staring us in the face, easily reproducable in our own Buildbot! But we deliberately chose to ignore those failures. The only thing that matters is "when can you push", and getting more releases out. Never mind what kind of crap is then pushed out to users. I am deeply ashamed of treating our users like this. And extremely unhappy. I even wrote about this (the importance of not ignoring Buildbot failures) just a few weeks ago. This was completely ignored, I got zero replies, and the weekly calls continued with mindless juggling of Jira numbers and buggy releases. As a result, our users suffer. We need to stop this. Please? - Kristian.

4 16

Re: [Maria-developers] Rotating from encry
by Jonas Oreland 22 May '15

22 May '15

8043 I tried using a search engine and googled for "jira mariadb innodb encryption jonas" :) /Jonas On Thu, May 21, 2015 at 4:36 PM, Jan Lindström <jan.lindstrom(a)mariadb.com> wrote: > Hi, > > Can you give the MDEV, I do not seem to be able to find it. > > R: Jan > > On Thu, May 21, 2015 at 2:51 PM, Jonas Oreland <jonaso(a)google.com> wrote: > >> i made a JIRA entry with latest innodb encryption... >> >> /Jonas >> >> On Thu, May 21, 2015 at 12:30 PM, Jan Lindström < >> jan.lindstrom(a)mariadb.com> wrote: >> >>> Hi, >>> >>> On Thu, May 21, 2015 at 11:33 AM, Jonas Oreland <jonaso(a)google.com> >>> wrote: >>> >>>> Hi Jan, >>>> >>>> 1) rotating encrypted => unencrypted is definitely supported, >>>> in my latest version the TODO is removed...(hope you have a recent >>>> enough version) >>>> >>> >>> Based on this, I do not have the latest version. Where, I could get >>> that, it does not seem to be on branch? >>> >>>> >>>> >>>> 2) Thanks! for testcase, it did indeed reveal a bug with the "encrypted >>>> => unencrypted => encrypted" sequence >>>> >>>> attaching fix. >>>> >>>> Let me know it fixes your version of testcase (i've used a modified >>>> version since e.g we don't have per table settings...) >>>> >>> >>> Yes, this fix is good. Thanks. >>> >>> R: Jan >>> >> >> >

1 0

[Maria-developers] How do I read information_schema.global_status not using query.
by kyung nam Kim 22 May '15

22 May '15

Hi, I hava a trouble with getting a data from informaition_schema.global_status table not using query. I wrote below code and it's possible to get data from some table in performance_schema . ** reader.cc =============================================================================================== my_thread_init(); ... MEM_ROOT mem_field; READ_RECORD read_record_info; tables.init_one_table(STRING_WITH_LEN("performance_schema"), STRING_WITH_LEN("events_statements_current"), bool result = open_and_lock_tables(ctx->thd, &tables, FALSE, MYSQL_LOCK_IGNORE_TIMEOUT); if (result) { return false; } if(table = tables.table) /* if only table exists */ { init_sql_alloc(&mem_field, 1024, 0, MYF(0)); ... while (!(error= read_record_info.read_record(&read_record_info))) { char* value = get_field(&mem_field, table->field[0]); // It's OK. .... } } ... ==================================================================================== It's possible to read data from performance_schema tables using low-level function . But I tried to read data from information_schema.global_status table as same way(just changed the name of database and table) , an error occurred in open_and_lock_tables(). Is there a way to read data from information_schema.global_status not using query? Regards, Kim kyungnam.

1 0

Re: [Maria-developers] Rotating from encry
by Jonas Oreland 21 May '15

21 May '15

i made a JIRA entry with latest innodb encryption... /Jonas On Thu, May 21, 2015 at 12:30 PM, Jan Lindström <jan.lindstrom(a)mariadb.com> wrote: > Hi, > > On Thu, May 21, 2015 at 11:33 AM, Jonas Oreland <jonaso(a)google.com> wrote: > >> Hi Jan, >> >> 1) rotating encrypted => unencrypted is definitely supported, >> in my latest version the TODO is removed...(hope you have a recent enough >> version) >> > > Based on this, I do not have the latest version. Where, I could get that, > it does not seem to be on branch? > >> >> >> 2) Thanks! for testcase, it did indeed reveal a bug with the "encrypted >> => unencrypted => encrypted" sequence >> >> attaching fix. >> >> Let me know it fixes your version of testcase (i've used a modified >> version since e.g we don't have per table settings...) >> > > Yes, this fix is good. Thanks. > > R: Jan >

1 0

Re: [Maria-developers] Rotating from encry
by Jonas Oreland 21 May '15

21 May '15

Hi Jan, 1) rotating encrypted => unencrypted is definitely supported, in my latest version the TODO is removed...(hope you have a recent enough version) 2) Thanks! for testcase, it did indeed reveal a bug with the "encrypted => unencrypted => encrypted" sequence attaching fix. Let me know it fixes your version of testcase (i've used a modified version since e.g we don't have per table settings...) /Jonas On Fri, May 15, 2015 at 1:23 PM, Jan Lindström <jan.lindstrom(a)mariadb.com> wrote: > Hi, > > At fil0crypt.cc there is > > fil_crypt_needs_rotation(uint key_version, const key_state_t *key_state) > { > // TODO(jonaso): Add support for rotating encrypted => unencrypted > > if (key_version == 0 && key_state->key_version != 0) { > /* this is rotation unencrypted => encrypted > * ignore rotate_key_age */ > return true; > } > > Thus to me it is not clear is the support for rotating encrypted => > unencrypted really missing or not and furthermore, see attached test case > for this, > > encrypted + insert + grep : ok > encrypted => unencrypted + grep: ok > unencrypted => encrypted + grep: not ok > > R: Jan >

1 0

Re: [Maria-developers] [Commits] b2db8e8: MDEV-7949: Item_field::used_tables() takes 0.29% in OLTP RO
by Sergey Vojtovich 21 May '15

21 May '15

Hi Sanja, this patch looks Ok, just one minor suggestion inline. Though effect for given workload is quite moderate: Item_field::used_tables 0.21% -> 0.20% Item_basic_constant::used_tables 0.11% -> 0.11% Item_func::used_tables 0.05% -> 0.05% Item_sum::used_tables 0.00% -> 0.00% On Wed, May 13, 2015 at 04:17:26PM +0200, sanja(a)mariadb.com wrote: > revision-id: b2db8e85422a455a10cd28ef7e44977182528f70 > parent(s): b22959903b89e798f8804ec9a815c88f75915cd9 > committer: Oleksandr Byelkin > branch nick: server > timestamp: 2015-05-13 16:17:22 +0200 > message: > > MDEV-7949: Item_field::used_tables() takes 0.29% in OLTP RO > > small sixes of used_tables() usage > ...skip... > diff --git a/sql/item_func.h b/sql/item_func.h > index 0d57c2b..c48cd84 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -1337,7 +1337,7 @@ class Item_func_sleep :public Item_int_func > const char *func_name() const { return "sleep"; } > table_map used_tables() const > { > - return Item_int_func::used_tables() | RAND_TABLE_BIT; > + return used_tables_cache | RAND_TABLE_BIT; > } > bool is_expensive() { return 1; } > longlong val_int(); > @@ -1591,7 +1591,7 @@ class Item_func_get_lock :public Item_int_func > void fix_length_and_dec() { max_length=1; maybe_null=1;} > table_map used_tables() const > { > - return Item_int_func::used_tables() | RAND_TABLE_BIT; > + return used_tables_cache | RAND_TABLE_BIT; > } > bool const_item() const { return 0; } > bool is_expensive() { return 1; } > @@ -1611,7 +1611,7 @@ class Item_func_release_lock :public Item_int_func > void fix_length_and_dec() { max_length= 1; maybe_null= 1;} > table_map used_tables() const > { > - return Item_int_func::used_tables() | RAND_TABLE_BIT; > + return used_tables_cache | RAND_TABLE_BIT; > } > bool const_item() const { return 0; } > bool is_expensive() { return 1; } > @@ -1724,7 +1724,7 @@ class Item_func_set_user_var :public Item_func > void fix_length_and_dec(); > table_map used_tables() const > { > - return Item_func::used_tables() | RAND_TABLE_BIT; > + return used_tables_cache | RAND_TABLE_BIT; > } > bool const_item() const { return 0; } > bool is_expensive() { return 1; } Was there any reason for this change? Compiler should be clever enough to inline used_tables() of base class here. Regards, Sergey

3 2

[Maria-developers] Misnamed libmariadb-dev packages on aptitude
by pcrews 19 May '15

19 May '15

Hello, Today, I was experimenting with building mariadb images and encountered this issue: E: Unable to locate package libmariadb-dev pcrews@erlking-dev:~/git/dibtest$ aptitude search libmariadb p libmariadbd-dev - MariaDB embedded database development files p libmariadbd-dev:i386 - MariaDB embedded database development files Note how these are listed as libmariadbd vs. libmariadb. I am writing to see if this can be fixed or if this might be some change in naming (which would require some updating of other tools' references, etc). Thank you, Patrick Crews

1 0

[Maria-developers] Step#6: MDEV-7950 Item_func::type() takes 0.26% in OLTP RO
by Alexander Barkov 19 May '15

19 May '15

Hi Sergey, Please review the next iteration for MDEV-7950. This one splits the function add_key_fields() into a method in Item. This change removes about 3 virtual calls item->type(), as well as some virtual calls item_func->functype(), and adds one virtual call item->add_key_fields() instead. Thanks.

2 2

Re: [Maria-developers] [Commits] 8b30c0f: MDEV-7949: Item_field::used_tables() takes 0.29% in OLTP RO
by Sergey Vojtovich 19 May '15

19 May '15

Hi Sanja, There're 3 bug fixes and 4 revisions concerning these lines: https://bugs.mysql.com/bug.php?id=16716 git: 1991a87d831bb48b31989c622e35acda11df7fef git: ca22a81b1c84ce81e1e9ea2c3ace7be1848027d8 (source for all used_tables()) https://bugs.mysql.com/bug.php?id=36488 git: 66367aeea83a0cd7f583a0194651a235b63975a9 https://bugs.mysql.com/bug.php?id=46815 git: 10406ae65871de074e807e626f9ede686e9321d4 If we don't have these bugs, why not to revert all the code introduced by these patches? Or we're not affected by the need to use used_tables() as an indication of prepared statement? If so, why? Thanks, Sergey On Sun, May 17, 2015 at 03:10:48PM +0200, sanja(a)mariadb.com wrote: > revision-id: 8b30c0f26c7b35fc73429f85aa35eb3614d60bd8 > parent(s): b2db8e85422a455a10cd28ef7e44977182528f70 > committer: Oleksandr Byelkin > branch nick: server > timestamp: 2015-05-17 15:10:45 +0200 > message: > > MDEV-7949: Item_field::used_tables() takes 0.29% in OLTP RO > > Part 2: removed hack workaround for bug we do not have. > > --- > sql/item_strfunc.cc | 7 +++---- > 1 file changed, 3 insertions(+), 4 deletions(-) > > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 4bf8dd5..7b9169e 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -626,8 +626,7 @@ String *Item_func_concat::val_str(String *str) > if (!(res=args[0]->val_str(str))) > goto null; > use_as_buff= &tmp_value; > - /* Item_subselect in --ps-protocol mode will state it as a non-const */ > - is_const= args[0]->const_item() || !args[0]->used_tables(); > + is_const= args[0]->const_item(); > for (i=1 ; i < arg_count ; i++) > { > if (res->length() == 0) > @@ -639,7 +638,7 @@ String *Item_func_concat::val_str(String *str) > non-empty argument. Because of this we need is_const to be > evaluated only for it. > */ > - is_const= args[i]->const_item() || !args[i]->used_tables(); > + is_const= args[i]->const_item(); > } > else > { > @@ -975,7 +974,7 @@ String *Item_func_concat_ws::val_str(String *str) > for (i=1; i < arg_count; i++) > if ((res= args[i]->val_str(str))) > { > - is_const= args[i]->const_item() || !args[i]->used_tables(); > + is_const= args[i]->const_item(); > break; > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] MDEV-7937: Enforce SSL when --ssl client option is used
by Vicențiu Ciorbaru 18 May '15

18 May '15

Hi Sergei! I've done some work on this issue. I've read MySQL's implementation of this and have looked at our implementation. They have done a bit of refactoring, introducing an enforce_ssl flag, as well as changing the C interface a bit, to allow setting this flag programatically. I've created a patch here that changes the minimum amount possible, in order to implement what MDEV-7937 requires. That being said, I agree with (most of) MySQL's refactoring in this case. They've moved all the SSL related connection code into its own separate function, before actually calling send_client_reply_packet. I can work towards implementing things the way MySQL does, but since I saw that you've actually done most of the work in that area of the code, I figured I'd ask for your input on it. There are two more things that I'm not sure of: 1. Specifying --ssl as a command line parameter to the mysql client is not enough to enforce ssl and the client's code in this case just ignores the option. We need to provide at least one of the additional ones like --ssl-key or --ssl-ca. My patch will not cause the client to report an error in this case. Is this acceptable behaviour or not? 2. Do we want mysql's enforce_ssl feature? Regards, Vicențiu

2 1

Re: [Maria-developers] MDEV-8112: PATCH: no slave left behind
by Kristian Nielsen 18 May '15

18 May '15

Hi Jonas, Thanks for the patch! I read through the patch. I am not that familiar with the semi-sync plugin code, but it looks reasonable. But the patch causes a test failure in sys_vars.all_vars. This is because not all newly introduced variables have corresponding test cases in mysql-test/suite/sys_vars/t/*_basic.test. Also, do you have some documentation for the feature? (There is a bit in the Jira entry, but for example the new variable rpl_semi_sync_master_slave_lag_heartbeat_frequency_us seems to be not described at all... If you can fix the above two points (and if Buildbot does not point out more test suite issues), then I will push it into the MariaDB tree. Thanks, - Kristian. > patch has been ported to 10.1 > you can use it under new (also called "3-clause") BSD license > i have done internal benchmarks...but you are most welcome to do some too. > > DESCRIPTION: > no slave left behind > > this patch implements master throttling based on slave lag, > aka no slave left behind. the core feature works as follows > 1) the semi-sync-reply is ammended to also report back SQL-thread > position (aka exec position) > 2) transactions are not removed from the "active-transaction-list" > in the semi-sync-master plugin until atleast one slave has reported > that it has executed this transaction. the slave lag can then > be estimated by calculating how long the oldest transaction has been > lingering in the active-transaction-list. > 3) client-threads are forced to wait before commit until slave lag > has decreased to acceptable value. > > the following variables are introduced on master: > > rpl_semi_sync_master_max_slave_lag (global) > rpl_semi_sync_master_slave_lag_wait_timeout (session) > > the following status variables are introduced on master: > > rpl_semi_sync_master_slave_lag_wait_sessions > rpl_semi_sync_master_estimated_slave_lag > rpl_semi_sync_master_trx_slave_lag_wait_time > rpl_semi_sync_master_trx_slave_lag_wait_num > rpl_semi_sync_master_avg_trx_slave_lag_wait_time > > the following variables are introduced on slave: > > rpl_semi_sync_slave_lag_enabled (global) > > in addition to this, 2 optimizations that decreases overhead of semi-sync > is introduced. > 1) the idea of this is that if when a slave should send and transaction, > it checks if it should be semi-synced, but rather > than semi-sync:ing each transaction (which is done currently) the code > will skip semi-syncing transaction if there already is newer transactions > committed. But, since this can mean that semi-syncing is delayed indefinitely > a cap is set using 2 new master variables: > > rpl_semi_sync_master_max_unacked_event_bytes (global) > rpl_semi_sync_master_max_unacked_event_count (global) > 2) rpl_semi_sync_master_group_commit which makes the semi-sync > plugin only semi-sync the last transaction in a group commit. > diff --git a/mysql-test/suite/rpl/r/rpl_semi_sync_slave_lag.result b/mysql-test/suite/rpl/r/rpl_semi_sync_slave_lag.result > new file mode 100644 > index 0000000..f3545a4 > --- /dev/null > +++ b/mysql-test/suite/rpl/r/rpl_semi_sync_slave_lag.result > @@ -0,0 +1,150 @@ > +include/master-slave.inc > +[connection master] > +set global rpl_semi_sync_master_enabled = 1; > +set global rpl_semi_sync_master_max_slave_lag = 10; > +show variables like 'rpl_semi_sync_master_%slave_lag%'; > +Variable_name Value > +rpl_semi_sync_master_max_slave_lag 10 > +rpl_semi_sync_master_slave_lag_heartbeat_frequency_us 500000 > +rpl_semi_sync_master_slave_lag_wait_timeout 50 > +include/stop_slave.inc > +set global rpl_semi_sync_slave_enabled = 1; > +set global rpl_semi_sync_slave_lag_enabled = 1; > +include/start_slave.inc > +# create non-root user for testing READ_ONLY > +grant SELECT, INSERT on *.* to test@localhost; > +CREATE TABLE t1 (i INT NOT NULL AUTO_INCREMENT PRIMARY KEY, f varchar(8)) > +ENGINE=innodb; > +# > +# Check basic behaviour > +# > +INSERT INTO t1 (f) VALUES ('1'),('2'),('3'); > +# Now wait for slave lag to decrease to 0 > +# [ on slave ] > +STOP SLAVE SQL_THREAD; > +# [ on master ] > +INSERT INTO t1 (f) VALUES ('4'),('5'),('6'); > +# Now wait for slave lag to increase to > 0 > +# [ on slave ] > +START SLAVE SQL_THREAD; > +# [ on master ] > +# Now wait for slave lag to decrease to 0 > +# [ on slave ] > +STOP SLAVE SQL_THREAD; > +# [ on master ] > +set session rpl_semi_sync_master_slave_lag_wait_timeout = 5; > +# First transaction should succeed. slave_lag is zero when it commits > +INSERT INTO t1 (f) VALUES ('7'),('8'),('9'); > +# Now wait for slave lag to increase to > 10s > +# Check that estimated_slave_lag is > 10s > +SELECT VARIABLE_VALUE > 10000000 as should_be_1 > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_estimated_slave_lag'; > +should_be_1 > +1 > +# Second transaction should now fail. slave_lag is >10s when it commits > +INSERT INTO t1 (f) VALUES ('a'),('b'),('c'); > +ERROR HY000: Slave-lag timeout > +# [ on slave ] > +START SLAVE SQL_THREAD; > +# [ on master ] > +# Now wait for slave lag to decrease to < 10s > +# And now it should succeed again > +INSERT INTO t1 (f) VALUES ('d'),('e'),('f'); > +SELECT * > +FROM t1 > +ORDER BY 1; > +i f > +1 1 > +2 2 > +3 3 > +4 4 > +5 5 > +6 6 > +7 7 > +8 8 > +9 9 > +13 d > +14 e > +15 f > +# [ on slave ] > +SELECT * > +FROM t1 > +ORDER BY 1; > +i f > +1 1 > +2 2 > +3 3 > +4 4 > +5 5 > +6 6 > +7 7 > +8 8 > +9 9 > +13 d > +14 e > +15 f > +# > +# Test interaction with READ_ONLY > +# > +# [ on slave ] > +STOP SLAVE SQL_THREAD; > +# [ on master ] > +INSERT INTO t1 (f) VALUES ('g'),('h'),('i'); > +# Now wait for slave lag to increase to > 10s > +# [ on con1 ] > +BEGIN; > +INSERT INTO t1 (f) VALUES ('g'),('h'),('i'); > +# [ on master ] > +set global read_only = 1; > +# [ on con1 ] > +set session rpl_semi_sync_master_slave_lag_wait_timeout = 5; > +# read-only is check *before* slave lag > +COMMIT; > +ERROR HY000: The MariaDB server is running with the --read-only option so it cannot execute this statement > +# [ on slave ] > +START SLAVE SQL_THREAD; > +# [ on master ] > +# Now wait for slave lag to decrease to 0 > +set global read_only = 0; > +# > +# check slave_lag > 0 but less than rpl_semi_sync_master_max_slave_lag > +# > +# [ on slave ] > +STOP SLAVE SQL_THREAD; > +# [ on master ] > +INSERT INTO t1 (f) VALUES ('j'),('k'),('l'); > +# Now wait for slave lag to increase to > 0 > +# Capture rpl_semi_sync_master_tx_slave_lag_waits before transaction > +select @count_before := VARIABLE_VALUE > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_tx_slave_lag_waits'; > +# Set maximum allowed slave lag to 24h > +set global rpl_semi_sync_master_max_slave_lag = 86400; > +INSERT INTO t1 (f) VALUES ('m'),('n'),('o'); > +# Capture rpl_semi_sync_master_tx_slave_lag_waits after transaction > +select @count_after := VARIABLE_VALUE > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_tx_slave_lag_waits'; > +# There should have been no wait, since maximum allowed is very high > +select @count_before = @count_after as should_be_1; > +should_be_1 > +1 > +# [ on slave ] > +START SLAVE SQL_THREAD; > +# [ on master ] > +# Now wait for slave lag to decrease to 0 > +# > +# Clean up > +# > +# [ on master ] > +set global rpl_semi_sync_master_max_slave_lag = default; > +set session rpl_semi_sync_master_slave_lag_wait_timeout = default; > +DROP USER test@localhost; > +include/stop_slave.inc > +set global rpl_semi_sync_slave_enabled = 0; > +set global rpl_semi_sync_slave_lag_enabled = default; > +set global rpl_semi_sync_master_enabled = 0; > +include/start_slave.inc > +DROP TABLE t1; > +include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_slave_lag.test b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_lag.test > new file mode 100644 > index 0000000..f9bdbd9 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_lag.test > @@ -0,0 +1,252 @@ > +source include/have_semisync.inc; > +source include/not_embedded.inc; > +source include/have_innodb.inc; > +source include/master-slave.inc; > + > +connection master; > +set global rpl_semi_sync_master_enabled = 1; > +set global rpl_semi_sync_master_max_slave_lag = 10; > + > +show variables like 'rpl_semi_sync_master_%slave_lag%'; > + > +connection slave; > +source include/stop_slave.inc; > +set global rpl_semi_sync_slave_enabled = 1; > +set global rpl_semi_sync_slave_lag_enabled = 1; > + > +source include/start_slave.inc; > + > +connection master; > + > +--echo # create non-root user for testing READ_ONLY > +grant SELECT, INSERT on *.* to test@localhost; > +connect (con1,localhost,test,,test); > + > +CREATE TABLE t1 (i INT NOT NULL AUTO_INCREMENT PRIMARY KEY, f varchar(8)) > +ENGINE=innodb; > + > +--echo # > +--echo # Check basic behaviour > +--echo # > +INSERT INTO t1 (f) VALUES ('1'),('2'),('3'); > + > +--echo # Now wait for slave lag to decrease to 0 > +let $wait_condition= SELECT VARIABLE_VALUE = 0 > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_estimated_slave_lag'; > +--source include/wait_condition.inc > + > +--echo # [ on slave ] > +connection slave; > +STOP SLAVE SQL_THREAD; > + > +--echo # [ on master ] > +connection master; > +INSERT INTO t1 (f) VALUES ('4'),('5'),('6'); > + > +--echo # Now wait for slave lag to increase to > 0 > +let $wait_condition= SELECT VARIABLE_VALUE > 0 > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_estimated_slave_lag'; > +--source include/wait_condition.inc > + > +--echo # [ on slave ] > +connection slave; > +START SLAVE SQL_THREAD; > + > +--echo # [ on master ] > +connection master; > + > +--echo # Now wait for slave lag to decrease to 0 > +let $wait_condition= SELECT VARIABLE_VALUE = 0 > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_estimated_slave_lag'; > +--source include/wait_condition.inc > + > +--echo # [ on slave ] > +connection slave; > +STOP SLAVE SQL_THREAD; > + > +--echo # [ on master ] > +connection master; > + > +set session rpl_semi_sync_master_slave_lag_wait_timeout = 5; > + > +--echo # First transaction should succeed. slave_lag is zero when it commits > +INSERT INTO t1 (f) VALUES ('7'),('8'),('9'); > + > +--echo # Now wait for slave lag to increase to > 10s > +let $wait_condition= SELECT VARIABLE_VALUE > 10000000 > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_estimated_slave_lag'; > +--source include/wait_condition.inc > + > +--echo # Check that estimated_slave_lag is > 10s > +SELECT VARIABLE_VALUE > 10000000 as should_be_1 > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_estimated_slave_lag'; > + > +--echo # Second transaction should now fail. slave_lag is >10s when it commits > +--error ER_ERROR_DURING_COMMIT > +INSERT INTO t1 (f) VALUES ('a'),('b'),('c'); > + > +--echo # [ on slave ] > +connection slave; > +START SLAVE SQL_THREAD; > + > +--echo # [ on master ] > +connection master; > + > +--echo # Now wait for slave lag to decrease to < 10s > +let $wait_condition= SELECT VARIABLE_VALUE < 10000000 > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_estimated_slave_lag'; > +--source include/wait_condition.inc > + > +--echo # And now it should succeed again > +INSERT INTO t1 (f) VALUES ('d'),('e'),('f'); > + > +SELECT * > +FROM t1 > +ORDER BY 1; > + > +sync_slave_with_master; > + > +--echo # [ on slave ] > +connection slave; > + > +SELECT * > +FROM t1 > +ORDER BY 1; > + > +--echo # > +--echo # Test interaction with READ_ONLY > +--echo # > + > +--echo # [ on slave ] > +connection slave; > +STOP SLAVE SQL_THREAD; > + > +--echo # [ on master ] > +connection master; > + > +INSERT INTO t1 (f) VALUES ('g'),('h'),('i'); > + > +--echo # Now wait for slave lag to increase to > 10s > +let $wait_condition= SELECT VARIABLE_VALUE > 10000000 > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_estimated_slave_lag'; > +--source include/wait_condition.inc > + > +connection con1; > +--echo # [ on con1 ] > +BEGIN; > +INSERT INTO t1 (f) VALUES ('g'),('h'),('i'); > + > +connection master; > +--echo # [ on master ] > +set global read_only = 1; > + > +connection con1; > +--echo # [ on con1 ] > +set session rpl_semi_sync_master_slave_lag_wait_timeout = 5; > + > +--echo # read-only is check *before* slave lag > +--error ER_OPTION_PREVENTS_STATEMENT > +COMMIT; > + > +disconnect con1; > + > +--echo # [ on slave ] > +connection slave; > +START SLAVE SQL_THREAD; > + > +connection master; > +--echo # [ on master ] > + > +--echo # Now wait for slave lag to decrease to 0 > +let $wait_condition= SELECT VARIABLE_VALUE = 0 > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_estimated_slave_lag'; > +--source include/wait_condition.inc > + > +set global read_only = 0; > + > +--echo # > +--echo # check slave_lag > 0 but less than rpl_semi_sync_master_max_slave_lag > +--echo # > + > +--echo # [ on slave ] > +connection slave; > +STOP SLAVE SQL_THREAD; > + > +connection master; > +--echo # [ on master ] > +INSERT INTO t1 (f) VALUES ('j'),('k'),('l'); > + > +--echo # Now wait for slave lag to increase to > 0 > +let $wait_condition= SELECT VARIABLE_VALUE > 0 > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_estimated_slave_lag'; > +--source include/wait_condition.inc > + > +--echo # Capture rpl_semi_sync_master_tx_slave_lag_waits before transaction > +--disable_result_log > +select @count_before := VARIABLE_VALUE > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_tx_slave_lag_waits'; > +--enable_result_log > + > +--echo # Set maximum allowed slave lag to 24h > +set global rpl_semi_sync_master_max_slave_lag = 86400; # 24h > + > +INSERT INTO t1 (f) VALUES ('m'),('n'),('o'); > + > +--echo # Capture rpl_semi_sync_master_tx_slave_lag_waits after transaction > +--disable_result_log > +select @count_after := VARIABLE_VALUE > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_tx_slave_lag_waits'; > +--enable_result_log > + > +--echo # There should have been no wait, since maximum allowed is very high > +select @count_before = @count_after as should_be_1; > + > +--echo # [ on slave ] > +connection slave; > +START SLAVE SQL_THREAD; > + > +connection master; > +--echo # [ on master ] > + > +--echo # Now wait for slave lag to decrease to 0 > +let $wait_condition= SELECT VARIABLE_VALUE = 0 > +FROM INFORMATION_SCHEMA.GLOBAL_STATUS > +WHERE VARIABLE_NAME = 'rpl_semi_sync_master_estimated_slave_lag'; > +--source include/wait_condition.inc > + > +--echo # > +--echo # Clean up > +--echo # > +connection master; > +--echo # [ on master ] > +set global rpl_semi_sync_master_max_slave_lag = default; > +set session rpl_semi_sync_master_slave_lag_wait_timeout = default; > +DROP USER test@localhost; > + > +connection slave; > +source include/stop_slave.inc; > +set global rpl_semi_sync_slave_enabled = 0; > +set global rpl_semi_sync_slave_lag_enabled = default; > + > +connection master; > +set global rpl_semi_sync_master_enabled = 0; > + > +connection slave; > +source include/start_slave.inc; > + > +connection master; > + > +DROP TABLE t1; > +sync_slave_with_master; > +--source include/rpl_end.inc > diff --git a/plugin/semisync/semisync.cc b/plugin/semisync/semisync.cc > index 4a80360..fc02b9d 100644 > --- a/plugin/semisync/semisync.cc > +++ b/plugin/semisync/semisync.cc > @@ -20,6 +20,7 @@ > > const unsigned char ReplSemiSyncBase::kPacketMagicNum = 0xef; > const unsigned char ReplSemiSyncBase::kPacketFlagSync = 0x01; > +const unsigned char ReplSemiSyncBase::kPacketFlagSyncAndReport = 0x02; > > > const unsigned long Trace::kTraceGeneral = 0x0001; > @@ -29,3 +30,6 @@ const unsigned long Trace::kTraceFunction = 0x0040; > > const unsigned char ReplSemiSyncBase::kSyncHeader[2] = > {ReplSemiSyncBase::kPacketMagicNum, 0}; > + > +const char* const ReplSemiSyncBase::kRplSemiSyncSlaveReportExec = > + "rpl_semi_sync_slave_report_exec"; > diff --git a/plugin/semisync/semisync.h b/plugin/semisync/semisync.h > index 2857729..78faba9 100644 > --- a/plugin/semisync/semisync.h > +++ b/plugin/semisync/semisync.h > @@ -75,13 +75,23 @@ class ReplSemiSyncBase > > /* Constants in network packet header. */ > static const unsigned char kPacketMagicNum; > + /* this event should be semisync acked */ > static const unsigned char kPacketFlagSync; > + /* this event should be semisync acked including the current SQL position */ > + static const unsigned char kPacketFlagSyncAndReport; > + > + /* user variable for enabling exec-pos reporting */ > + static const char* const kRplSemiSyncSlaveReportExec; > }; > > /* The layout of a semisync slave reply packet: > 1 byte for the magic num > 8 bytes for the binlog positon > - n bytes for the binlog filename, terminated with a '\0' > + n bytes for the binlog filename, NOT terminated with a '\0' > + [ optionally ] > + 1 byte == 0 > + 8 bytes for the sql-thread position > + n bytes for the sql-thread filename, terminated with a '\0' > */ > #define REPLY_MAGIC_NUM_LEN 1 > #define REPLY_BINLOG_POS_LEN 8 > diff --git a/plugin/semisync/semisync_master.cc b/plugin/semisync/semisync_master.cc > index c88c162..3ee6e26 100644 > --- a/plugin/semisync/semisync_master.cc > +++ b/plugin/semisync/semisync_master.cc > @@ -22,6 +22,9 @@ > #define TIME_MILLION 1000000 > #define TIME_BILLION 1000000000 > > +/* thd_key for per slave thread state */ > +static MYSQL_THD_KEY_T thd_key; > + > /* This indicates whether semi-synchronous replication is enabled. */ > char rpl_semi_sync_master_enabled; > unsigned long rpl_semi_sync_master_wait_point = > @@ -45,6 +48,18 @@ unsigned long long rpl_semi_sync_master_net_wait_time = 0; > unsigned long long rpl_semi_sync_master_trx_wait_time = 0; > char rpl_semi_sync_master_wait_no_slave = 1; > > +unsigned long rpl_semi_sync_master_max_unacked_event_count = 0; > +unsigned long rpl_semi_sync_master_max_unacked_event_bytes = 4096; > + > +unsigned long rpl_semi_sync_master_slave_lag_clients = 0; > +unsigned long long rpl_semi_sync_master_estimated_slave_lag = 0; > +unsigned long rpl_semi_sync_master_slave_lag_heartbeat_frequency_us = 500000; > +unsigned long rpl_semi_sync_master_max_slave_lag = 0; > +unsigned long rpl_semi_sync_master_slave_lag_wait_sessions = 0; > + > +unsigned long rpl_semi_sync_master_avg_trx_slave_lag_wait_time = 0; > +unsigned long long rpl_semi_sync_master_trx_slave_lag_wait_num = 0; > +unsigned long long rpl_semi_sync_master_trx_slave_lag_wait_time = 0; > > static int getWaitTime(const struct timespec& start_ts); > > @@ -150,6 +165,15 @@ int ActiveTranx::insert_tranx_node(const char *log_file_name, > ins_node->log_name_[FN_REFLEN-1] = 0; /* make sure it ends properly */ > ins_node->log_pos_ = log_file_pos; > > + { > + /** > + * set trans commit time > + * this is called when writing into binlog, which is not > + * exactly right, but close enough for our purposes > + */ > + ins_node->tranx_commit_time_us = my_hrtime().val; > + } > + > if (!trx_front_) > { > /* The list is empty. */ > @@ -193,12 +217,11 @@ int ActiveTranx::insert_tranx_node(const char *log_file_name, > return function_exit(kWho, result); > } > > -bool ActiveTranx::is_tranx_end_pos(const char *log_file_name, > - my_off_t log_file_pos) > +TranxNode* ActiveTranx::lookup_tranx_end_pos(const char *log_file_name, > + my_off_t log_file_pos) > { > - const char *kWho = "ActiveTranx::is_tranx_end_pos"; > + const char *kWho = "ActiveTranx::lookup_tranx_end_pos"; > function_enter(kWho); > - > unsigned int hash_val = get_hash_value(log_file_name, log_file_pos); > TranxNode *entry = trx_htb_[hash_val]; > > @@ -211,38 +234,24 @@ bool ActiveTranx::is_tranx_end_pos(const char *log_file_name, > } > > if (trace_level_ & kTraceDetail) > - sql_print_information("%s: probe (%s, %lu) in entry(%u)", kWho, > - log_file_name, (unsigned long)log_file_pos, hash_val); > + sql_print_information("%s: probe (%s, %lu)", kWho, > + log_file_name, (unsigned long)log_file_pos); > > function_exit(kWho, (entry != NULL)); > - return (entry != NULL); > + return entry; > } > > -int ActiveTranx::clear_active_tranx_nodes(const char *log_file_name, > - my_off_t log_file_pos) > +int ActiveTranx::clear_active_tranx_nodes() > { > - const char *kWho = "ActiveTranx::::clear_active_tranx_nodes"; > - TranxNode *new_front; > + set_new_front(NULL); > + return 0; > +} > > +void ActiveTranx::set_new_front(TranxNode *new_front) > +{ > + const char *kWho = "ActiveTranx::set_new_front"; > function_enter(kWho); > > - if (log_file_name != NULL) > - { > - new_front = trx_front_; > - > - while (new_front) > - { > - if (compare(new_front, log_file_name, log_file_pos) > 0) > - break; > - new_front = new_front->next_; > - } > - } > - else > - { > - /* If log_file_name is NULL, clear everything. */ > - new_front = NULL; > - } > - > if (new_front == NULL) > { > /* No active transaction nodes after the call. */ > @@ -257,7 +266,6 @@ int ActiveTranx::clear_active_tranx_nodes(const char *log_file_name, > trx_front_ = NULL; > trx_rear_ = NULL; > } > - > if (trace_level_ & kTraceDetail) > sql_print_information("%s: cleared all nodes", kWho); > } > @@ -291,14 +299,40 @@ int ActiveTranx::clear_active_tranx_nodes(const char *log_file_name, > > trx_front_ = new_front; > allocator_.free_nodes_before(trx_front_); > - > if (trace_level_ & kTraceDetail) > sql_print_information("%s: cleared %d nodes back until pos (%s, %lu)", > kWho, n_frees, > trx_front_->log_name_, (unsigned long)trx_front_->log_pos_); > } > + function_exit(kWho, 0); > +} > > - return function_exit(kWho, 0); > +bool ActiveTranx::prune_active_tranx_nodes( > + LogPosPtr pos, > + ulonglong *oldest_tranx_commit_time_us) > +{ > + TranxNode *old_front = trx_front_; > + TranxNode *new_front; > + > + new_front = trx_front_; > + while (new_front) > + { > + if (compare(new_front, pos.file_name, pos.file_pos) > 0) > + break; > + new_front = new_front->next_; > + } > + > + set_new_front(new_front); > + > + if (oldest_tranx_commit_time_us) > + { > + if (trx_front_ == NULL) > + *oldest_tranx_commit_time_us = 0; > + else > + *oldest_tranx_commit_time_us = trx_front_->tranx_commit_time_us; > + } > + > + return ! (old_front == trx_front_); > } > > > @@ -334,7 +368,8 @@ ReplSemiSyncMaster::ReplSemiSyncMaster() > wait_file_pos_(0), > master_enabled_(false), > wait_timeout_(0L), > - state_(0) > + state_(0), > + oldest_unapplied_tranx_commit_time_us_(0) > { > strcpy(reply_file_name_, ""); > strcpy(wait_file_name_, ""); > @@ -362,11 +397,19 @@ int ReplSemiSyncMaster::initObject() > mysql_cond_init(key_ss_cond_COND_binlog_send_, > &COND_binlog_send_, NULL); > > + /* Mutex initialization can only be done after MY_INIT(). */ > + mysql_mutex_init(key_ss_mutex_LOCK_slave_lag_, > + &LOCK_slave_lag_, MY_MUTEX_INIT_FAST); > + mysql_cond_init(key_ss_cond_COND_slave_lag_, > + &COND_slave_lag_, NULL); > + > if (rpl_semi_sync_master_enabled) > result = enableMaster(); > else > result = disableMaster(); > > + thd_key_create(&thd_key); > + > return result; > } > > @@ -437,6 +480,8 @@ void ReplSemiSyncMaster::cleanup() > { > mysql_mutex_destroy(&LOCK_binlog_); > mysql_cond_destroy(&COND_binlog_send_); > + mysql_mutex_destroy(&LOCK_slave_lag_); > + mysql_cond_destroy(&COND_slave_lag_); > init_done_= 0; > } > > @@ -473,7 +518,34 @@ void ReplSemiSyncMaster::add_slave() > { > lock(); > rpl_semi_sync_master_clients++; > + if (has_semi_sync_slave_lag()) > + rpl_semi_sync_master_slave_lag_clients++; > unlock(); > + > + if (has_semi_sync_slave_lag()) > + { > + int null_val = 0; > + longlong new_val = > + rpl_semi_sync_master_slave_lag_heartbeat_frequency_us * 1000; > + longlong old_val = new_val + 1; > + > + get_user_var_int("master_heartbeat_period", &old_val, &null_val); > + if (old_val > new_val || null_val) > + { > + /* if there no old value or it's bigger than what we want */ > + int res = set_user_var_int("master_heartbeat_period",new_val, &old_val); > + if (res == -1) > + { > + sql_print_error( > + "Repl_semi_sync::failed to set master_heartbeat_period"); > + } > + } > + } > + > + /** > + * create per slave-state and store it in thread-local-storage */ > + ReplSemiSyncMasterPerSlaveState *state = new ReplSemiSyncMasterPerSlaveState; > + thd_setspecific(current_thd, thd_key, state); > } > > void ReplSemiSyncMaster::remove_slave() > @@ -492,7 +564,31 @@ void ReplSemiSyncMaster::remove_slave() > rpl_semi_sync_master_clients == 0) > switch_off(); > } > + > + bool no_slave_lag_clients = false; > + if (has_semi_sync_slave_lag()) > + { > + if (--rpl_semi_sync_master_slave_lag_clients == 0) > + { > + no_slave_lag_clients = true; > + } > + } > + > unlock(); > + > + ReplSemiSyncMasterPerSlaveState *state = > + (ReplSemiSyncMasterPerSlaveState*)thd_getspecific(current_thd, thd_key); > + thd_setspecific(current_thd, thd_key, NULL); > + > + if (state != NULL) > + { > + delete state; > + } > + > + if (no_slave_lag_clients) > + { > + wake_slave_lag_waiters(0); > + } > } > > bool ReplSemiSyncMaster::is_semi_sync_slave() > @@ -503,14 +599,115 @@ bool ReplSemiSyncMaster::is_semi_sync_slave() > return val; > } > > +bool ReplSemiSyncMaster::has_semi_sync_slave_lag() > +{ > + int null_value; > + long long val= 0; > + get_user_var_int(kRplSemiSyncSlaveReportExec, &val, &null_value); > + return val; > +} > + > +int ReplSemiSyncMaster::checkSyncReq(const LogPosPtr *log_pos) > +{ > + if (log_pos == NULL) > + { > + /* heartbeat events does not have logpos (since they are not actually > + * stored in the binlog). > + */ > + if (!has_semi_sync_slave_lag()) > + { > + /* don't semi-sync them if we haven't enabled slave-lag handling */ > + return 0; > + } > + else > + { > + /* else ask for both IO and exec position */ > + return 2; > + } > + } > + > + /** > + * check if this log-pos is a candidate for semi-syncing event > + */ > + TranxNode *entry = active_tranxs_->lookup_tranx_end_pos(log_pos->file_name, > + log_pos->file_pos); > + > + if (entry == NULL) > + return 0; > + > + ReplSemiSyncMasterPerSlaveState *state = > + (ReplSemiSyncMasterPerSlaveState*)thd_getspecific(current_thd, > + thd_key); > + do > + { > + state->unacked_event_count_++; > + > + if (active_tranxs_->is_rear(entry)) > + { > + /* always ask for ack on last event in tranx list */ > + break; > + } > + > + if (state->unacked_event_count_ >= > + rpl_semi_sync_master_max_unacked_event_count) > + { > + /* enough events passed that it's time for another ack */ > + break; > + } > + > + if (!state->sync_req_pos_.IsInited()) > + { > + /* first event => time for ack */ > + break; > + } > + > + if (strcmp(log_pos->file_name, state->sync_req_pos_.file_name) != 0) > + { > + /* new file => time for ack */ > + break; > + } > + > + if (log_pos->file_pos >= (state->sync_req_pos_.file_pos + > + rpl_semi_sync_master_max_unacked_event_bytes)) > + { > + /* enough bytes => time for ack */ > + break; > + } > + > + /* we skip asking for semi-sync ack on this event */ > + return 0; > + > + } while (0); > + > + /* keep track on when we last asked for semi-sync-ack */ > + state->unacked_event_count_ = 0; > + state->sync_req_pos_.Assign(log_pos); > + > + /** > + * check if this slave can report back exec position > + */ > + if (!has_semi_sync_slave_lag()) > + { > + /* slave can't report back SQL position */ > + return 1; > + } > + > + /* ask for both IO and SQL position */ > + return 2; > +} > + > int ReplSemiSyncMaster::reportReplyBinlog(uint32 server_id, > const char *log_file_name, > - my_off_t log_file_pos) > + my_off_t log_file_pos, > + const LogPos *exec_pos) > { > const char *kWho = "ReplSemiSyncMaster::reportReplyBinlog"; > int cmp; > bool can_release_threads = false; > bool need_copy_send_pos = true; > + bool pruned_trx_list = false; > + ulonglong oldest_tranx_commit_time_us = 0; > + > > if (!(getMasterEnabled())) > return 0; > @@ -559,15 +756,29 @@ int ReplSemiSyncMaster::reportReplyBinlog(uint32 server_id, > reply_file_pos_ = log_file_pos; > reply_file_name_inited_ = true; > > - /* Remove all active transaction nodes before this point. */ > - assert(active_tranxs_ != NULL); > - active_tranxs_->clear_active_tranx_nodes(log_file_name, log_file_pos); > - > if (trace_level_ & kTraceDetail) > sql_print_information("%s: Got reply at (%s, %lu)", kWho, > log_file_name, (unsigned long)log_file_pos); > } > > + assert(active_tranxs_ != NULL); > + if (exec_pos != NULL) > + { > + /* prune using exec_pos */ > + LogPosPtr ptr(*exec_pos); > + pruned_trx_list = active_tranxs_->prune_active_tranx_nodes( > + ptr, &oldest_tranx_commit_time_us); > + } > + else if (rpl_semi_sync_master_slave_lag_clients == 0 && need_copy_send_pos) > + { > + /** > + * if we don't have any slaves that can do exec_pos reporting, > + * prune by IO position as "plain old semi sync" > + */ > + LogPosPtr ptr(log_file_name, log_file_pos); > + active_tranxs_->prune_active_tranx_nodes(ptr, NULL); > + } > + > if (rpl_semi_sync_master_wait_sessions > 0) > { > /* Let us check if some of the waiting threads doing a trx > @@ -596,6 +807,15 @@ int ReplSemiSyncMaster::reportReplyBinlog(uint32 server_id, > cond_broadcast(); > } > > + if (pruned_trx_list) > + { > + /** > + * if we did prune trx list, it might be that we should wake up > + * threads waiting for slave-lag to decrease > + */ > + wake_slave_lag_waiters(oldest_tranx_commit_time_us); > + } > + > return function_exit(kWho, 0); > } > > @@ -743,14 +963,6 @@ int ReplSemiSyncMaster::commitTrx(const char* trx_wait_binlog_name, > } > } > > - /* > - At this point, the binlog file and position of this transaction > - must have been removed from ActiveTranx. > - */ > - assert(thd_killed(NULL) || > - !active_tranxs_->is_tranx_end_pos(trx_wait_binlog_name, > - trx_wait_binlog_pos)); > - > l_end: > /* Update the status counter. */ > if (is_on()) > @@ -794,7 +1006,7 @@ int ReplSemiSyncMaster::switch_off() > > /* Clear the active transaction list. */ > assert(active_tranxs_ != NULL); > - result = active_tranxs_->clear_active_tranx_nodes(NULL, 0); > + result = active_tranxs_->clear_active_tranx_nodes(); > > rpl_semi_sync_master_off_times++; > wait_file_name_inited_ = false; > @@ -884,7 +1096,7 @@ int ReplSemiSyncMaster::updateSyncHeader(unsigned char *packet, > { > const char *kWho = "ReplSemiSyncMaster::updateSyncHeader"; > int cmp = 0; > - bool sync = false; > + int sync = 0; > > /* If the semi-sync master is not enabled, or the slave is not a semi-sync > * target, do not request replies from the slave. > @@ -905,6 +1117,13 @@ int ReplSemiSyncMaster::updateSyncHeader(unsigned char *packet, > /* semi-sync is ON */ > /* sync= false; No sync unless a transaction is involved. */ > > + if (log_file_name == NULL) > + { > + /* this is heartbeat, request io_pos and exec_pos */ > + sync = checkSyncReq(0); > + goto l_end; > + } > + > if (reply_file_name_inited_) > { > cmp = ActiveTranx::compare(log_file_name, log_file_pos, > @@ -933,12 +1152,12 @@ int ReplSemiSyncMaster::updateSyncHeader(unsigned char *packet, > */ > if (cmp >= 0) > { > - /* > + /* > * We only wait if the event is a transaction's ending event. > */ > assert(active_tranxs_ != NULL); > - sync = active_tranxs_->is_tranx_end_pos(log_file_name, > - log_file_pos); > + LogPosPtr pos(log_file_name, log_file_pos); > + sync = checkSyncReq(&pos); > } > } > else > @@ -951,7 +1170,7 @@ int ReplSemiSyncMaster::updateSyncHeader(unsigned char *packet, > } > else > { > - sync = true; > + sync = 1; > } > } > > @@ -966,10 +1185,14 @@ int ReplSemiSyncMaster::updateSyncHeader(unsigned char *packet, > /* We do not need to clear sync flag because we set it to 0 when we > * reserve the packet header. > */ > - if (sync) > + if (sync == 1) > { > (packet)[2] = kPacketFlagSync; > } > + else if (sync == 2) > + { > + (packet)[2] = kPacketFlagSyncAndReport; > + } > > return function_exit(kWho, 0); > } > @@ -1018,7 +1241,8 @@ int ReplSemiSyncMaster::writeTranxInBinlog(const char* log_file_name, > if (is_on()) > { > assert(active_tranxs_ != NULL); > - if(active_tranxs_->insert_tranx_node(log_file_name, log_file_pos)) > + bool empty = active_tranxs_->is_empty(); > + if (active_tranxs_->insert_tranx_node(log_file_name, log_file_pos)) > { > /* > if insert tranx_node failed, print a warning message > @@ -1028,6 +1252,14 @@ int ReplSemiSyncMaster::writeTranxInBinlog(const char* log_file_name, > log_file_name, (ulong)log_file_pos); > switch_off(); > } > + else if (empty && rpl_semi_sync_master_slave_lag_clients > 0) > + { > + /* if the list of transactions was empty, > + * we need to init the oldest_tranx_commit_time_us > + */ > + oldest_unapplied_tranx_commit_time_us_ = > + active_tranxs_->get_oldest_tranx_commit_time_us(); > + } > } > > l_end: > @@ -1037,10 +1269,10 @@ int ReplSemiSyncMaster::writeTranxInBinlog(const char* log_file_name, > } > > int ReplSemiSyncMaster::readSlaveReply(NET *net, uint32 server_id, > - const char *event_buf) > + const char *event_buf_) > { > const char *kWho = "ReplSemiSyncMaster::readSlaveReply"; > - const unsigned char *packet; > + const unsigned char *packet, *packet_start; > char log_file_name[FN_REFLEN]; > my_off_t log_file_pos; > ulong log_file_len = 0; > @@ -1048,12 +1280,15 @@ int ReplSemiSyncMaster::readSlaveReply(NET *net, uint32 server_id, > int result = -1; > struct timespec start_ts; > ulong trc_level = trace_level_; > + const unsigned char *event_buf = (const unsigned char*)event_buf_; > + bool exec_pos_present = false; // is SQL exec pos present in reply > + LogPos exec_pos; // position of SQL thread > LINT_INIT_STRUCT(start_ts); > > function_enter(kWho); > > - assert((unsigned char)event_buf[1] == kPacketMagicNum); > - if ((unsigned char)event_buf[2] != kPacketFlagSync) > + assert(event_buf[1] == kPacketMagicNum); > + if ((event_buf[2] & (kPacketFlagSync | kPacketFlagSyncAndReport)) == 0) > { > /* current event does not require reply */ > result = 0; > @@ -1111,28 +1346,60 @@ int ReplSemiSyncMaster::readSlaveReply(NET *net, uint32 server_id, > goto l_end; > } > > - packet = net->read_pos; > + packet_start = packet = net->read_pos; > if (packet[REPLY_MAGIC_NUM_OFFSET] != ReplSemiSyncMaster::kPacketMagicNum) > { > sql_print_error("Read semi-sync reply magic number error"); > goto l_end; > } > > + /* we determine if this semisync ack contains a sql-thread exec-pos > + * by checking if last byte == 0, since the packet then contains > + * \0-terminated filenames */ > + exec_pos_present = packet[packet_len - 1] == 0; > + > log_file_pos = uint8korr(packet + REPLY_BINLOG_POS_OFFSET); > - log_file_len = packet_len - REPLY_BINLOG_NAME_OFFSET; > + if (exec_pos_present == false) > + { > + log_file_len = packet_len - REPLY_BINLOG_NAME_OFFSET; > + } > + else > + { > + log_file_len = strnlen((char*)packet + REPLY_BINLOG_NAME_OFFSET, > + MY_MIN((ulong)FN_REFLEN, > + packet_len - REPLY_BINLOG_NAME_OFFSET)); > + } > if (log_file_len >= FN_REFLEN) > { > sql_print_error("Read semi-sync reply binlog file length too large"); > goto l_end; > } > - strncpy(log_file_name, (const char*)packet + REPLY_BINLOG_NAME_OFFSET, log_file_len); > + packet+= REPLY_BINLOG_NAME_OFFSET; > + > + strncpy(log_file_name, (const char*)packet, log_file_len); > log_file_name[log_file_len] = 0; > > + if (exec_pos_present) > + { > + packet += log_file_len + 1; > + if (packet + 8 + 1 >= (packet_start + packet_len)) > + { > + sql_print_error("Read semi-sync reply binlog. " > + "Packet to short to contain exec-position!"); > + goto l_end; > + } > + exec_pos.file_pos = uint8korr(packet); > + packet += 8; > + strncpy(exec_pos.file_name, (char*)packet, > + (packet_start + packet_len) - packet); > + } > + > if (trc_level & kTraceDetail) > sql_print_information("%s: Got reply (%s, %lu)", > kWho, log_file_name, (ulong)log_file_pos); > > - result = reportReplyBinlog(server_id, log_file_name, log_file_pos); > + result = reportReplyBinlog(server_id, log_file_name, log_file_pos, > + exec_pos_present ? &exec_pos : NULL); > > l_end: > return function_exit(kWho, result); > @@ -1154,6 +1421,16 @@ int ReplSemiSyncMaster::resetMaster() > wait_file_name_inited_ = false; > reply_file_name_inited_ = false; > commit_file_name_inited_ = false; > + if (active_tranxs_ != NULL) > + { > + /** > + * make sure to empty transaction hash/list > + * with slave-lag reporting this container does > + * not have to be empty even if no transaction is > + * currently running > + */ > + active_tranxs_->clear_active_tranx_nodes(); > + } > > rpl_semi_sync_master_yes_transactions = 0; > rpl_semi_sync_master_no_transactions = 0; > @@ -1168,6 +1445,13 @@ int ReplSemiSyncMaster::resetMaster() > > unlock(); > > + mysql_mutex_lock(&LOCK_slave_lag_); > + rpl_semi_sync_master_slave_lag_wait_sessions = 0; > + oldest_unapplied_tranx_commit_time_us_ = 0; > + rpl_semi_sync_master_trx_slave_lag_wait_num = 0; > + rpl_semi_sync_master_trx_slave_lag_wait_time = 0; > + mysql_mutex_unlock(&LOCK_slave_lag_); > + > return function_exit(kWho, result); > } > > @@ -1186,6 +1470,29 @@ void ReplSemiSyncMaster::setExportStats() > ((double)rpl_semi_sync_master_net_wait_num)) : 0); > > unlock(); > + > + if (oldest_unapplied_tranx_commit_time_us_ != 0) > + { > + rpl_semi_sync_master_estimated_slave_lag = my_hrtime().val - > + oldest_unapplied_tranx_commit_time_us_; > + } > + else > + { > + rpl_semi_sync_master_estimated_slave_lag = 0; > + } > + > + mysql_mutex_lock(&LOCK_slave_lag_); > + if (rpl_semi_sync_master_trx_slave_lag_wait_num) > + { > + rpl_semi_sync_master_avg_trx_slave_lag_wait_time = > + (unsigned long)((double)rpl_semi_sync_master_trx_slave_lag_wait_time / > + (double)rpl_semi_sync_master_trx_slave_lag_wait_num); > + } > + else > + { > + rpl_semi_sync_master_avg_trx_slave_lag_wait_time = 0; > + } > + mysql_mutex_unlock(&LOCK_slave_lag_); > } > > /* Get the waiting time given the wait's staring time. > @@ -1213,3 +1520,117 @@ static int getWaitTime(const struct timespec& start_ts) > > return (int)(end_usecs - start_usecs); > } > + > +void ReplSemiSyncMaster::wake_slave_lag_waiters( > + ulonglong oldest_unapplied_tranx_commit_time_us) > +{ > + mysql_mutex_lock(&LOCK_slave_lag_); > + oldest_unapplied_tranx_commit_time_us_ = > + oldest_unapplied_tranx_commit_time_us; > + > + if (rpl_semi_sync_master_slave_lag_wait_sessions > 0) > + { > + mysql_cond_broadcast(&COND_slave_lag_); > + } > + mysql_mutex_unlock(&LOCK_slave_lag_); > +} > + > +int ReplSemiSyncMaster::wait_slave_lag(ulong timeout_sec) > +{ > + int error = 0; > + PSI_stage_info old_stage; > + > + /* slave lag waiting not enabled, return directly */ > + if (rpl_semi_sync_master_max_slave_lag == 0) > + return 0; > + > + /* there is no slave that can report slave lag, return directly */ > + if (rpl_semi_sync_master_slave_lag_clients == 0) > + return 0; > + > + /* compute start_time and end_time */ > + struct timespec end_time; > + set_timespec(end_time, 0); > + ulonglong start_time_us = timespec_to_usec(&end_time); > + end_time.tv_sec += timeout_sec; > + > + mysql_mutex_lock(&LOCK_slave_lag_); > + > + if (oldest_unapplied_tranx_commit_time_us_ == 0) > + { > + /* no slave lag, atleast one slave is up to date */ > + mysql_mutex_unlock(&LOCK_slave_lag_); > + return 0; > + } > + > + if (rpl_semi_sync_master_max_slave_lag == 0) > + { > + /* slave lag waiting not enabled */ > + mysql_mutex_unlock(&LOCK_slave_lag_); > + return 0; > + } > + > + /* This must be called after acquired the lock */ > + THD_ENTER_COND(NULL, &COND_slave_lag_, &LOCK_slave_lag_, > + &stage_waiting_for_semi_sync_slave_lag, > + &old_stage); > + > + bool waited = false; > + ulonglong lag = 0; > + ulonglong max_lag = 0; > + while (oldest_unapplied_tranx_commit_time_us_ != 0) > + { > + /* check kill_level after THD_ENTER_COND but *before* cond_wait > + * to avoid missing kills */ > + if (! (getMasterEnabled() && is_on() && > + thd_kill_level(NULL) == THD_IS_NOT_KILLED)) > + break; > + > + lag = start_time_us - oldest_unapplied_tranx_commit_time_us_; > + max_lag = 1000000 * rpl_semi_sync_master_max_slave_lag; > + if (lag <= max_lag) > + break; > + > + waited = true; > + rpl_semi_sync_master_slave_lag_wait_sessions++; > + int wait_result = mysql_cond_timedwait(&COND_slave_lag_, &LOCK_slave_lag_, > + &end_time); > + rpl_semi_sync_master_slave_lag_wait_sessions--; > + > + bool thd_was_killed = thd_kill_level(NULL) != THD_IS_NOT_KILLED; > + if (wait_result != 0 || thd_was_killed) > + { > + break; > + } > + } > + > + if (thd_kill_level(NULL) != THD_IS_NOT_KILLED) > + { > + /* Return error to client. */ > + error = 1; > + my_printf_error(ER_ERROR_DURING_COMMIT, > + "Killed while waiting for replication semi-sync slave-lag.", > + MYF(0)); > + } > + else if (lag > max_lag) > + { > + /* Return error to client. */ > + error = 1; > + my_printf_error(ER_ERROR_DURING_COMMIT, > + "Slave-lag timeout", > + MYF(0)); > + } > + > + if (waited) > + { > + rpl_semi_sync_master_trx_slave_lag_wait_num++; > + rpl_semi_sync_master_trx_slave_lag_wait_time += > + (my_hrtime().val - start_time_us); > + } > + > + /* The lock held will be released by thd_exit_cond, so no need to > + call unlock() here */ > + THD_EXIT_COND(NULL, & old_stage); > + > + return error; > +} > diff --git a/plugin/semisync/semisync_master.h b/plugin/semisync/semisync_master.h > index d9dc4ce..e5c0de6 100644 > --- a/plugin/semisync/semisync_master.h > +++ b/plugin/semisync/semisync_master.h > @@ -24,17 +24,101 @@ > #ifdef HAVE_PSI_INTERFACE > extern PSI_mutex_key key_ss_mutex_LOCK_binlog_; > extern PSI_cond_key key_ss_cond_COND_binlog_send_; > + > +extern PSI_mutex_key key_ss_mutex_LOCK_slave_lag_; > +extern PSI_cond_key key_ss_cond_COND_slave_lag_; > #endif > > extern PSI_stage_info stage_waiting_for_semi_sync_ack_from_slave; > +extern PSI_stage_info stage_waiting_for_semi_sync_slave_lag; > > struct TranxNode { > char log_name_[FN_REFLEN]; > - my_off_t log_pos_; > + my_off_t log_pos_; > + ulonglong tranx_commit_time_us; > struct TranxNode *next_; /* the next node in the sorted list */ > struct TranxNode *hash_next_; /* the next node during hash collision */ > }; > > +struct LogPos; > + > +/* This represent a log position */ > +struct LogPosPtr { > + LogPosPtr() { Uninit();} > + LogPosPtr(const char *name, my_off_t pos) : file_name(name), file_pos(pos){} > + explicit LogPosPtr(const LogPos& pos) { Assign(&pos); } > + > + const char *file_name; > + my_off_t file_pos; > + > + LogPosPtr& Assign(const LogPosPtr *src) { > + file_name = src->file_name; > + file_pos = src->file_pos; > + return *this; > + } > + > + LogPosPtr& Assign(const LogPos *src); > + > + void Uninit() { file_name = NULL;} > + bool IsInited() const { return file_name != NULL; } > +}; > + > +struct LogPos { > + char file_name[FN_REFLEN]; > + my_off_t file_pos; > + > + LogPos() { Uninit(); } > + > + LogPosPtr ToLogPosPtr() const { > + if (IsInited()){ > + LogPosPtr p(file_name, file_pos); > + return p; > + } else { > + LogPosPtr p; > + return p; > + } > + } > + > + LogPos& Assign(const LogPosPtr *src) { > + if (src->IsInited()) { > + strcpy(file_name, src->file_name); > + file_pos = src->file_pos; > + } else { > + Uninit(); > + } > + return *this; > + } > + > + LogPos& Assign(const LogPos* src) { > + LogPosPtr p = src->ToLogPosPtr(); > + Assign(&p); > + return *this; > + } > + > + void Uninit() { file_name[0] = 0; } > + bool IsInited() const { return file_name[0] != 0; } > +}; > + > +inline LogPosPtr& LogPosPtr::Assign(const LogPos* src) { > + LogPosPtr p = src->ToLogPosPtr(); > + Assign(&p); > + return *this; > +} > + > +inline int CompareLogPos(const LogPosPtr *pos1, const LogPosPtr *pos2) { > + int cmp = strcmp(pos1->file_name, pos2->file_name); > + > + if (cmp != 0) > + return cmp; > + > + if (pos1->file_pos > pos2->file_pos) > + return 1; > + else if (pos1->file_pos < pos2->file_pos) > + return -1; > + else > + return 0; > +} > + > /** > @class TranxNodeAllocator > > @@ -329,10 +413,14 @@ class ActiveTranx > node2->log_name_, node2->log_pos_); > } > > + void set_new_front(TranxNode* new_front); > + > public: > ActiveTranx(mysql_mutex_t *lock, unsigned long trace_level); > ~ActiveTranx(); > > + bool is_empty() const { return trx_front_ == NULL; } > + > /* Insert an active transaction node with the specified position. > * > * Return: > @@ -340,21 +428,42 @@ class ActiveTranx > */ > int insert_tranx_node(const char *log_file_name, my_off_t log_file_pos); > > - /* Clear the active transaction nodes until(inclusive) the specified > - * position. > - * If log_file_name is NULL, everything will be cleared: the sorted > + /* Clear the active transaction > + * Everything will be cleared: the sorted > * list and the hash table will be reset to empty. > - * > + * > * Return: > - * 0: success; non-zero: error > + * 0 success; non-zero: error > + */ > + int clear_active_tranx_nodes(); > + > + /* Prune the active transaction nodes until the specified > + * position (inclusive). > + * > + * Return: > + * true if any transaction was removed > + * false if list was left unchanged > */ > - int clear_active_tranx_nodes(const char *log_file_name, > - my_off_t log_file_pos); > + bool prune_active_tranx_nodes(LogPosPtr logpos, > + ulonglong *oldest_tranx_commit_time_us); > > - /* Given a position, check to see whether the position is an active > - * transaction's ending position by probing the hash table. > + /* Lookup a transaction's ending position by probing the hash table. > + * > + * return entry if found or NULL otherwise > */ > - bool is_tranx_end_pos(const char *log_file_name, my_off_t log_file_pos); > + TranxNode* lookup_tranx_end_pos(const char *log_file_name, > + my_off_t log_file_pos); > + > + > + /* Check if an entry is rear (i.e last) */ > + bool is_rear(TranxNode* entry) const { return entry == trx_rear_; } > + > + /** > + * return timestamp of oldest transaction in list > + */ > + ulonglong get_oldest_tranx_commit_time_us() const { > + return trx_front_->tranx_commit_time_us; > + } > > /* Given two binlog positions, compare which one is bigger based on > * (file_name, file_position). > @@ -365,6 +474,24 @@ class ActiveTranx > }; > > /** > + * State that semisync master keeps per slave > + */ > +struct ReplSemiSyncMasterPerSlaveState > +{ > + ReplSemiSyncMasterPerSlaveState() : unacked_event_count_(0) {} > + > + /** > + * No of events that has not been semi-sync acked > + */ > + unsigned unacked_event_count_; > + > + /** > + * Position of last event that was semisync'ed > + */ > + LogPos sync_req_pos_; > +}; > + > +/** > The extension class for the master of semi-synchronous replication > */ > class ReplSemiSyncMaster > @@ -432,6 +559,16 @@ class ReplSemiSyncMaster > > bool state_; /* whether semi-sync is switched */ > > + /* This cond variable is signaled when slave lag has decreased */ > + mysql_cond_t COND_slave_lag_; > + > + /* Mutex that protects oldest_unapplied_tranx_commit_time_us */ > + mysql_mutex_t LOCK_slave_lag_; > + > + /* this is commit time of oldest transaction that has not been applied > + * on any slave */ > + ulonglong oldest_unapplied_tranx_commit_time_us_; > + > void lock(); > void unlock(); > void cond_broadcast(); > @@ -493,6 +630,9 @@ class ReplSemiSyncMaster > /* Is the slave servered by the thread requested semi-sync */ > bool is_semi_sync_slave(); > > + /* Does this slave have slave lag reporting capabilities */ > + bool has_semi_sync_slave_lag(); > + > /* In semi-sync replication, reports up to which binlog position we have > * received replies from the slave indicating that it already get the events. > * > @@ -501,13 +641,15 @@ class ReplSemiSyncMaster > * log_file_name - (IN) binlog file name > * end_offset - (IN) the offset in the binlog file up to which we have > * the replies from the slave > + * exec_position - (IN) position of SQL thread or NULL if not present > * > * Return: > * 0: success; non-zero: error > */ > int reportReplyBinlog(uint32 server_id, > const char* log_file_name, > - my_off_t end_offset); > + my_off_t end_offset, > + const LogPos *exec_position); > > /* Commit a transaction in the final step. This function is called from > * InnoDB before returning from the low commit. If semi-sync is switch on, > @@ -540,6 +682,16 @@ class ReplSemiSyncMaster > */ > int reserveSyncHeader(unsigned char *header, unsigned long size); > > + /* > + * check if an event should be semi synced and optionally > + * if it should report back position of SQL thread on slave > + * > + * return 0 - no semi sync > + * 1 - semi sync > + * 2 - semi sync and report exec position > + */ > + int checkSyncReq(const LogPosPtr *log_pos); > + > /* Update the sync bit in the packet header to indicate to the slave whether > * the master will wait for the reply of the event. If semi-sync is switched > * off and we detect that the slave is catching up, we switch semi-sync on. > @@ -592,6 +744,21 @@ class ReplSemiSyncMaster > * go off for that. > */ > int resetMaster(); > + > + /** > + * wake potential slave-lag waiters > + * called by binlog dump-thread(s) > + */ > + void wake_slave_lag_waiters(ulonglong oldest_unapplied_tranx_commit_time_us); > + > + /** > + * wait for slave lag to get below threshold > + * called by user-thread(s) > + * > + * return 0 - success > + * 1 - timeout > + */ > + int wait_slave_lag(ulong max_wait_time_sec); > }; > > enum rpl_semi_sync_master_wait_point_t { > @@ -621,6 +788,17 @@ extern unsigned long long rpl_semi_sync_master_trx_wait_num; > extern unsigned long long rpl_semi_sync_master_net_wait_time; > extern unsigned long long rpl_semi_sync_master_trx_wait_time; > > +extern unsigned long rpl_semi_sync_master_max_unacked_event_count; > +extern unsigned long rpl_semi_sync_master_max_unacked_event_bytes; > +extern unsigned long rpl_semi_sync_master_max_slave_lag; > +extern unsigned long rpl_semi_sync_master_slave_lag_heartbeat_frequency_us; > +extern unsigned long rpl_semi_sync_master_slave_lag_wait_sessions; > +extern unsigned long long rpl_semi_sync_master_estimated_slave_lag; > + > +extern unsigned long rpl_semi_sync_master_avg_trx_slave_lag_wait_time; > +extern unsigned long long rpl_semi_sync_master_trx_slave_lag_wait_num; > +extern unsigned long long rpl_semi_sync_master_trx_slave_lag_wait_time; > + > /* > This indicates whether we should keep waiting if no semi-sync slave > is available. > diff --git a/plugin/semisync/semisync_master_plugin.cc b/plugin/semisync/semisync_master_plugin.cc > index 7bb0eea..3282158 100644 > --- a/plugin/semisync/semisync_master_plugin.cc > +++ b/plugin/semisync/semisync_master_plugin.cc > @@ -21,6 +21,11 @@ > > static ReplSemiSyncMaster repl_semisync; > > +// forward declaration > +static inline ulong get_slave_lag_wait_timeout(THD* thd); > + > +static char rpl_semi_sync_master_group_commit = 0; > + > C_MODE_START > > int repl_semi_report_binlog_update(Binlog_storage_param *param, > @@ -31,6 +36,13 @@ int repl_semi_report_binlog_update(Binlog_storage_param *param, > > if (repl_semisync.getMasterEnabled()) > { > + if (rpl_semi_sync_master_group_commit && > + ((flags & BINLOG_GROUP_COMMIT_TRAILER) == 0)) > + { > + /** there are transactions more coming... */ > + return 0; > + } > + > /* > Let us store the binlog file name and the position, so that > we know how long to wait for the binlog to the replicated to > @@ -43,8 +55,11 @@ int repl_semi_report_binlog_update(Binlog_storage_param *param, > return error; > } > > -int repl_semi_request_commit(Trans_param *param) > +int repl_semi_before_commit(Trans_param *param, int *error) > { > + *error = repl_semisync.wait_slave_lag( > + get_slave_lag_wait_timeout(current_thd)); > + > return 0; > } > > @@ -53,6 +68,14 @@ int repl_semi_report_binlog_sync(Binlog_storage_param *param, > my_off_t log_pos, uint32 flags) > { > int error= 0; > + > + if (rpl_semi_sync_master_group_commit && > + ((flags & BINLOG_GROUP_COMMIT_TRAILER) == 0)) > + { > + /** there are transactions more coming... */ > + return 0; > + } > + > if (rpl_semi_sync_master_wait_point == > SEMI_SYNC_MASTER_WAIT_POINT_AFTER_BINLOG_SYNC) > { > @@ -100,7 +123,7 @@ int repl_semi_binlog_dump_start(Binlog_transmit_param *param, > Let's assume this semi-sync slave has already received all > binlog events before the filename and position it requests. > */ > - repl_semisync.reportReplyBinlog(param->server_id, log_file, log_pos); > + repl_semisync.reportReplyBinlog(param->server_id, log_file, log_pos, NULL); > } > sql_print_information("Start %s binlog_dump to slave (server_id: %d), pos(%s, %lu)", > semi_sync_slave ? "semi-sync" : "asynchronous", > @@ -242,15 +265,72 @@ static MYSQL_SYSVAR_ULONG(trace_level, rpl_semi_sync_master_trace_level, > &fix_rpl_semi_sync_master_trace_level, // update > 32, 0, ~0UL, 1); > > +static MYSQL_SYSVAR_ULONG(max_unacked_event_count, > + rpl_semi_sync_master_max_unacked_event_count, > + PLUGIN_VAR_OPCMDARG, > + "Maximum unacked replication events", > + NULL, // check > + NULL, // update > + rpl_semi_sync_master_max_unacked_event_count, 0, ~0UL, 1); > + > +static MYSQL_SYSVAR_ULONG(max_unacked_event_bytes, > + rpl_semi_sync_master_max_unacked_event_bytes, > + PLUGIN_VAR_OPCMDARG, > + "Maximum unacked replication bytes", > + NULL, // check > + NULL, // update > + rpl_semi_sync_master_max_unacked_event_bytes, 0, ~0UL, 1); > + > +static MYSQL_SYSVAR_ULONG(max_slave_lag, rpl_semi_sync_master_max_slave_lag, > + PLUGIN_VAR_OPCMDARG, > + "Maximum allowed lag of fastest semi-sync slave (in seconds), " > + "checked before commit.", > + NULL, // check > + NULL, // update > + rpl_semi_sync_master_max_slave_lag, 0, ~0UL, 1); > + > +static MYSQL_THDVAR_ULONG(slave_lag_wait_timeout, > + PLUGIN_VAR_RQCMDARG, > + "Timeout in seconds a rw-transaction may wait for max slave lag before " > + "being rolled back.", > + NULL, NULL, 50, 1, 1024 * 1024 * 1024, 0); > + > +static MYSQL_SYSVAR_ULONG( > + slave_lag_heartbeat_frequency_us, > + rpl_semi_sync_master_slave_lag_heartbeat_frequency_us, > + PLUGIN_VAR_RQCMDARG, > + "Heartbeat frequency when slave-lag is enabled (in microseconds).", > + NULL, // check > + NULL, // update > + 500000, /* 500 ms */ > + 1, ~0UL, 1); > + > +static MYSQL_SYSVAR_BOOL(group_commit, rpl_semi_sync_master_group_commit, > + PLUGIN_VAR_OPCMDARG, > + "Group commit for semi sync", > + NULL, // check > + NULL, > + 0); > + > static SYS_VAR* semi_sync_master_system_vars[]= { > MYSQL_SYSVAR(enabled), > MYSQL_SYSVAR(wait_point), > MYSQL_SYSVAR(timeout), > MYSQL_SYSVAR(wait_no_slave), > MYSQL_SYSVAR(trace_level), > + MYSQL_SYSVAR(max_unacked_event_count), > + MYSQL_SYSVAR(max_unacked_event_bytes), > + MYSQL_SYSVAR(max_slave_lag), > + MYSQL_SYSVAR(slave_lag_wait_timeout), > + MYSQL_SYSVAR(slave_lag_heartbeat_frequency_us), > + MYSQL_SYSVAR(group_commit), > NULL, > }; > > +static inline ulong get_slave_lag_wait_timeout(THD* thd) > +{ > + return THDVAR(thd, slave_lag_wait_timeout); > +} > > static void fix_rpl_semi_sync_master_timeout(MYSQL_THD thd, > SYS_VAR *var, > @@ -297,6 +377,7 @@ Trans_observer trans_observer = { > > repl_semi_report_commit, // after_commit > repl_semi_report_rollback, // after_rollback > + repl_semi_before_commit, // before commit > }; > > Binlog_storage_observer storage_observer = { > @@ -339,7 +420,11 @@ DEF_SHOW_FUNC(net_wait_time, SHOW_LONGLONG) > DEF_SHOW_FUNC(net_wait_num, SHOW_LONGLONG) > DEF_SHOW_FUNC(avg_net_wait_time, SHOW_LONG) > DEF_SHOW_FUNC(avg_trx_wait_time, SHOW_LONG) > - > +DEF_SHOW_FUNC(slave_lag_wait_sessions, SHOW_LONG) > +DEF_SHOW_FUNC(estimated_slave_lag, SHOW_LONGLONG) > +DEF_SHOW_FUNC(trx_slave_lag_wait_time, SHOW_LONGLONG) > +DEF_SHOW_FUNC(trx_slave_lag_wait_num, SHOW_LONGLONG) > +DEF_SHOW_FUNC(avg_trx_slave_lag_wait_time, SHOW_LONG) > > /* plugin status variables */ > static SHOW_VAR semi_sync_master_status_vars[]= { > @@ -385,32 +470,55 @@ static SHOW_VAR semi_sync_master_status_vars[]= { > {"Rpl_semi_sync_master_net_avg_wait_time", > (char*) &SHOW_FNAME(avg_net_wait_time), > SHOW_SIMPLE_FUNC}, > + {"Rpl_semi_sync_master_slave_lag_wait_sessions", > + (char*) &SHOW_FNAME(slave_lag_wait_sessions), > + SHOW_SIMPLE_FUNC}, > + {"Rpl_semi_sync_master_estimated_slave_lag", > + (char*) &SHOW_FNAME(estimated_slave_lag), > + SHOW_SIMPLE_FUNC}, > + {"Rpl_semi_sync_master_tx_slave_lag_wait_time", > + (char*) &SHOW_FNAME(trx_slave_lag_wait_time), > + SHOW_SIMPLE_FUNC}, > + {"Rpl_semi_sync_master_tx_slave_lag_waits", > + (char*) &SHOW_FNAME(trx_slave_lag_wait_num), > + SHOW_SIMPLE_FUNC}, > + {"Rpl_semi_sync_master_tx_avg_slave_lag_wait_time", > + (char*) &SHOW_FNAME(avg_trx_slave_lag_wait_time), > + SHOW_SIMPLE_FUNC}, > {NULL, NULL, SHOW_LONG}, > }; > > #ifdef HAVE_PSI_INTERFACE > PSI_mutex_key key_ss_mutex_LOCK_binlog_; > +PSI_mutex_key key_ss_mutex_LOCK_slave_lag_; > > static PSI_mutex_info all_semisync_mutexes[]= > { > - { &key_ss_mutex_LOCK_binlog_, "LOCK_binlog_", 0} > + { &key_ss_mutex_LOCK_binlog_, "LOCK_binlog_", 0 }, > + { &key_ss_mutex_LOCK_slave_lag_, "LOCK_slave_lag_", 0 } > }; > > PSI_cond_key key_ss_cond_COND_binlog_send_; > +PSI_cond_key key_ss_cond_COND_slave_lag_; > > static PSI_cond_info all_semisync_conds[]= > { > - { &key_ss_cond_COND_binlog_send_, "COND_binlog_send_", 0} > + { &key_ss_cond_COND_binlog_send_, "COND_binlog_send_", 0 }, > + { &key_ss_cond_COND_slave_lag_, "COND_slave_lag_", 0 } > }; > #endif /* HAVE_PSI_INTERFACE */ > > PSI_stage_info stage_waiting_for_semi_sync_ack_from_slave= > { 0, "Waiting for semi-sync ACK from slave", 0}; > > +PSI_stage_info stage_waiting_for_semi_sync_slave_lag= > +{ 0, "Waiting for semi-sync slave lag", 0}; > + > #ifdef HAVE_PSI_INTERFACE > PSI_stage_info *all_semisync_stages[]= > { > - & stage_waiting_for_semi_sync_ack_from_slave > + & stage_waiting_for_semi_sync_ack_from_slave, > + & stage_waiting_for_semi_sync_slave_lag > }; > > static void init_semisync_psi_keys(void) > @@ -492,4 +600,3 @@ maria_declare_plugin(semisync_master) > MariaDB_PLUGIN_MATURITY_GAMMA > } > maria_declare_plugin_end; > - > diff --git a/plugin/semisync/semisync_slave.cc b/plugin/semisync/semisync_slave.cc > index 5f98472..839e0cc 100644 > --- a/plugin/semisync/semisync_slave.cc > +++ b/plugin/semisync/semisync_slave.cc > @@ -20,6 +20,7 @@ > char rpl_semi_sync_slave_enabled; > char rpl_semi_sync_slave_status= 0; > unsigned long rpl_semi_sync_slave_trace_level; > +char rpl_semi_sync_slave_lag_enabled= 0; > > int ReplSemiSyncSlave::initObject() > { > @@ -42,7 +43,7 @@ int ReplSemiSyncSlave::initObject() > > int ReplSemiSyncSlave::slaveReadSyncHeader(const char *header, > unsigned long total_len, > - bool *need_reply, > + unsigned char *need_reply, > const char **payload, > unsigned long *payload_len) > { > @@ -52,7 +53,7 @@ int ReplSemiSyncSlave::slaveReadSyncHeader(const char *header, > > if ((unsigned char)(header[0]) == kPacketMagicNum) > { > - *need_reply = (header[1] & kPacketFlagSync); > + *need_reply = (header[1] & (kPacketFlagSync | kPacketFlagSyncAndReport)); > *payload_len = total_len - 2; > *payload = header + 2; > > @@ -95,16 +96,20 @@ int ReplSemiSyncSlave::slaveStop(Binlog_relay_IO_param *param) > return 0; > } > > -int ReplSemiSyncSlave::slaveReply(MYSQL *mysql, > - const char *binlog_filename, > - my_off_t binlog_filepos) > +int ReplSemiSyncSlave::slaveReply(unsigned char header_byte, > + MYSQL *mysql, > + const char *binlog_filename, > + my_off_t binlog_filepos, > + Master_info * mi) > { > const char *kWho = "ReplSemiSyncSlave::slaveReply"; > NET *net= &mysql->net; > - uchar reply_buffer[REPLY_MAGIC_NUM_LEN > - + REPLY_BINLOG_POS_LEN > - + REPLY_BINLOG_NAME_LEN]; > + uchar reply_buffer[REPLY_MAGIC_NUM_LEN + > + 2 * ( REPLY_BINLOG_POS_LEN + > + REPLY_BINLOG_NAME_LEN + > + /* '\0' */ 1) ]; > int reply_res, name_len = strlen(binlog_filename); > + int msg_len = name_len + REPLY_BINLOG_NAME_OFFSET; > > function_enter(kWho); > > @@ -119,10 +124,29 @@ int ReplSemiSyncSlave::slaveReply(MYSQL *mysql, > sql_print_information("%s: reply (%s, %lu)", kWho, > binlog_filename, (ulong)binlog_filepos); > > + if (header_byte & kPacketFlagSyncAndReport) > + { > + /** > + * master requests that we also report back SQL-thread position > + */ > + > + // where to store sql filename/position > + char *bufptr = (char*)reply_buffer + msg_len; > + bufptr[0] = 0; // '\0' terminate previous filename > + bufptr++; > + > + my_off_t sql_file_pos; > + // get file/position and store the filename directly info bufptr+8 > + size_t name_len2 = get_master_log_pos(mi, bufptr + 8, &sql_file_pos); > + int8store(bufptr, sql_file_pos); // store position > + > + msg_len += /* '\0' */ 1 + /* position */ 8 + name_len2 + /* '\0' */ 1; > + } > + > net_clear(net, 0); > /* Send the reply. */ > - reply_res = my_net_write(net, reply_buffer, > - name_len + REPLY_BINLOG_NAME_OFFSET); > + reply_res = my_net_write(net, reply_buffer, msg_len); > + > if (!reply_res) > { > reply_res = net_flush(net); > diff --git a/plugin/semisync/semisync_slave.h b/plugin/semisync/semisync_slave.h > index 1bf8cf3..c91847d 100644 > --- a/plugin/semisync/semisync_slave.h > +++ b/plugin/semisync/semisync_slave.h > @@ -60,23 +60,30 @@ class ReplSemiSyncSlave > * Return: > * 0: success; non-zero: error > */ > - int slaveReadSyncHeader(const char *header, unsigned long total_len, bool *need_reply, > + int slaveReadSyncHeader(const char *header, unsigned long total_len, > + unsigned char *need_reply_byte, > const char **payload, unsigned long *payload_len); > > /* A slave replies to the master indicating its replication process. It > * indicates that the slave has received all events before the specified > * binlog position. > - * > + * > * Input: > + * need_reply_byte - (IN) the header byte > * mysql - (IN) the mysql network connection > * binlog_filename - (IN) the reply point's binlog file name > * binlog_filepos - (IN) the reply point's binlog file offset > + * master_info - (IN) the master info struct so that we can get more > + * info if needed > * > * Return: > * 0: success; non-zero: error > */ > - int slaveReply(MYSQL *mysql, const char *binlog_filename, > - my_off_t binlog_filepos); > + int slaveReply(unsigned char need_reply_byte, > + MYSQL *mysql, > + const char *binlog_filename, > + my_off_t binlog_filepos, > + Master_info* master_info); > > int slaveStart(Binlog_relay_IO_param *param); > int slaveStop(Binlog_relay_IO_param *param); > @@ -93,5 +100,6 @@ class ReplSemiSyncSlave > extern char rpl_semi_sync_slave_enabled; > extern unsigned long rpl_semi_sync_slave_trace_level; > extern char rpl_semi_sync_slave_status; > +extern char rpl_semi_sync_slave_lag_enabled; > > #endif /* SEMISYNC_SLAVE_H */ > diff --git a/plugin/semisync/semisync_slave_plugin.cc b/plugin/semisync/semisync_slave_plugin.cc > index 572ead2..0bf03be 100644 > --- a/plugin/semisync/semisync_slave_plugin.cc > +++ b/plugin/semisync/semisync_slave_plugin.cc > @@ -28,7 +28,7 @@ static ReplSemiSyncSlave repl_semisync; > event read is the last event of a transaction. And the value is > checked in repl_semi_slave_queue_event. > */ > -bool semi_sync_need_reply= false; > +unsigned char semi_sync_need_reply= 0; > > C_MODE_START > > @@ -81,6 +81,23 @@ int repl_semi_slave_request_dump(Binlog_relay_IO_param *param, > return 1; > } > mysql_free_result(mysql_store_result(mysql)); > + > + if (rpl_semi_sync_slave_lag_enabled) > + { > + char buf[100]; > + /* > + Tell master that we can do exec-position reporting > + */ > + snprintf(buf, sizeof(buf), "SET @%s= 1", > + ReplSemiSyncBase::kRplSemiSyncSlaveReportExec); > + if (mysql_real_query(mysql, buf, strlen(buf))) > + { > + sql_print_error("query: %s on master failed", buf); > + return 1; > + } > + mysql_free_result(mysql_store_result(mysql)); > + } > + > rpl_semi_sync_slave_status= 1; > return 0; > } > @@ -110,9 +127,11 @@ int repl_semi_slave_queue_event(Binlog_relay_IO_param *param, > should not cause the slave IO thread to stop, and the error > messages are already reported. > */ > - (void) repl_semisync.slaveReply(param->mysql, > + (void) repl_semisync.slaveReply(semi_sync_need_reply, > + param->mysql, > param->master_log_name, > - param->master_log_pos); > + param->master_log_pos, > + param->mi); > } > return 0; > } > @@ -164,9 +183,17 @@ static MYSQL_SYSVAR_ULONG(trace_level, rpl_semi_sync_slave_trace_level, > &fix_rpl_semi_sync_trace_level, // update > 32, 0, ~0UL, 1); > > +static MYSQL_SYSVAR_BOOL(lag_enabled, rpl_semi_sync_slave_lag_enabled, > + PLUGIN_VAR_OPCMDARG, > + "Enable semi-synchronous replication slave lag reporting. ", > + NULL, // check > + NULL, // update > + 0); > + > static SYS_VAR* semi_sync_slave_system_vars[]= { > MYSQL_SYSVAR(enabled), > MYSQL_SYSVAR(trace_level), > + MYSQL_SYSVAR(lag_enabled), > NULL, > }; > > @@ -230,4 +257,3 @@ maria_declare_plugin(semisync_slave) > MariaDB_PLUGIN_MATURITY_GAMMA > } > maria_declare_plugin_end; > - > diff --git a/sql/handler.cc b/sql/handler.cc > index 3ca9ec3..3e6cd65 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1364,12 +1364,30 @@ int ha_commit_trans(THD *thd, bool all) > uint rw_ha_count= ha_check_and_coalesce_trx_read_only(thd, ha_info, all); > /* rw_trans is TRUE when we in a transaction changing data */ > bool rw_trans= is_real_trans && (rw_ha_count > 0); > + bool mdl_request_initialized= false; > MDL_request mdl_request; > DBUG_PRINT("info", ("is_real_trans: %d rw_trans: %d rw_ha_count: %d", > is_real_trans, rw_trans, rw_ha_count)); > > if (rw_trans) > { > + /* check READ-ONLY just before before_commit hook to decrease likelihood > + * of having threads hanging waiting for slave-lag only to be aborted > + * due to read-only. > + */ > + if (opt_readonly && > + !(thd->security_ctx->master_access & SUPER_ACL) && > + !thd->slave_thread) > + { > + my_error(ER_OPTION_PREVENTS_STATEMENT, MYF(0), "--read-only"); > + goto err; > + } > + > + if (RUN_HOOK(transaction, before_commit, (thd))) > + { > + goto err; > + } > + > /* > Acquire a metadata lock which will ensure that COMMIT is blocked > by an active FLUSH TABLES WITH READ LOCK (and vice versa: > @@ -1378,6 +1396,7 @@ int ha_commit_trans(THD *thd, bool all) > We allow the owner of FTWRL to COMMIT; we assume that it knows > what it does. > */ > + mdl_request_initialized= true; > mdl_request.init(MDL_key::COMMIT, "", "", MDL_INTENTION_EXCLUSIVE, > MDL_EXPLICIT); > > @@ -1486,7 +1505,7 @@ int ha_commit_trans(THD *thd, bool all) > ha_rollback_trans(thd, all); > > end: > - if (rw_trans && mdl_request.ticket) > + if (rw_trans && mdl_request_initialized && mdl_request.ticket) > { > /* > We do not always immediately release transactional locks > diff --git a/sql/replication.h b/sql/replication.h > index 4731c22..309bdb4 100644 > --- a/sql/replication.h > +++ b/sql/replication.h > @@ -110,6 +110,21 @@ typedef struct Trans_observer { > @retval 1 Failure > */ > int (*after_rollback)(Trans_param *param); > + > + /** > + This callback is called before transaction commit > + If function does not return *error == 0 transaction will > + not be committed but error code will be returned to client > + > + @note *error!=0 and return code 0 shall be used by plugin to signal > + that transaction should be aborted. > + If returning non-zero transaction will also be aborted and an error > + will be printed to error log. > + > + @retval 0 Sucess > + @retval non-zero error > + */ > + int (*before_commit)(Trans_param *param, int *error); > } Trans_observer; > > /** > @@ -294,6 +309,8 @@ enum Binlog_relay_IO_flags { > }; > > > +class Master_info; > + > /** > Replication binlog relay IO observer parameter > */ > @@ -309,8 +326,20 @@ typedef struct Binlog_relay_IO_param { > my_off_t master_log_pos; > > MYSQL *mysql; /* the connection to master */ > + > + Master_info * mi; /* master info handle */ > } Binlog_relay_IO_param; > > + > +/* get the master log given a Master_info > + * and store it in filename_buf/filepos > + * return length of filename (excluding \0) > + * > + * note: filename_buf should be a minimum FN_REFLEN > + */ > +size_t get_master_log_pos(const Master_info *mi, > + char *filename_buf, my_off_t *filepos); > + > /** > Observes and extends the service of slave IO thread. > */ > @@ -561,7 +590,21 @@ int get_user_var_str(const char *name, > char *value, unsigned long len, > unsigned int precision, int *null_value); > > - > + > +/** > + Set or replace the value of user variable as to an ulonglong > + > + @param name user variable name > + @param value the value > + @param old_value pointer to where old value will be stored (or NULL) > + > + @retval 0 Success, no prior value found > + @retval 1 Success, old_value populated > + @retval -1 Fail > +*/ > +int set_user_var_int(const char *name, > + long long int value, > + long long int *old_value); > > #ifdef __cplusplus > } > diff --git a/sql/rpl_handler.cc b/sql/rpl_handler.cc > index 3962600..209a809 100644 > --- a/sql/rpl_handler.cc > +++ b/sql/rpl_handler.cc > @@ -23,6 +23,7 @@ > #include "rpl_filter.h" > #include <my_dir.h> > #include "rpl_handler.h" > +#include "sql_prepare.h" > > Trans_delegate *transaction_delegate; > Binlog_storage_delegate *binlog_storage_delegate; > @@ -88,6 +89,42 @@ int get_user_var_str(const char *name, char *value, > return 0; > } > > +int set_user_var_int(const char *name, > + long long int value, > + long long int *old_value) > +{ > + THD* thd= current_thd; > + bool null_val; > + user_var_entry *entry= > + (user_var_entry*) my_hash_search(&thd->user_vars, > + (uchar*) name, strlen(name)); > + if (entry != NULL) > + { > + if (old_value != NULL) > + *old_value= entry->val_int(&null_val); > + } > + > + Ed_connection con(thd); > + > + char buf[256]; > + int res= snprintf(buf, sizeof(buf), "SET @%s=%lld", name, value); > + if (/* error */ res < 0 || > + /* truncated */ res >= sizeof(buf)) > + { > + return -1; > + } > + > + LEX_STRING str; > + lex_string_set(&str, buf); > + > + if (con.execute_direct(str)) > + { > + return -1; > + } > + > + return entry == NULL ? 0 : 1; > +} > + > int delegates_init() > { > static my_aligned_storage<sizeof(Trans_delegate), MY_ALIGNOF(long)> trans_mem; > @@ -249,6 +286,17 @@ int Trans_delegate::after_rollback(THD *thd, bool all) > return ret; > } > > +int Trans_delegate::before_commit(THD *thd) > +{ > + int ret= 0, error= 0; > + Trans_param param; > + param.flags= 0; > + param.log_file= 0; > + param.log_pos= 0; > + FOREACH_OBSERVER(ret, before_commit, thd, (&param, &error)); > + return error; > +} > + > int Binlog_storage_delegate::after_flush(THD *thd, > const char *log_file, > my_off_t log_pos, > @@ -374,17 +422,19 @@ int Binlog_transmit_delegate::reserve_header(THD *thd, ushort flags, > > int Binlog_transmit_delegate::before_send_event(THD *thd, ushort flags, > String *packet, > - const char *log_file, > + const char *log_file_path, > my_off_t log_pos) > { > Binlog_transmit_param param; > param.flags= flags; > > int ret= 0; > + const char* log_file_name= log_file_path != NULL ? > + log_file_path + dirname_length(log_file_path) : NULL; > FOREACH_OBSERVER(ret, before_send_event, false, > (&param, (uchar *)packet->c_ptr(), > packet->length(), > - log_file+dirname_length(log_file), log_pos)); > + log_file_name, log_pos)); > return ret; > } > > @@ -414,6 +464,7 @@ int Binlog_transmit_delegate::after_reset_master(THD *thd, ushort flags) > void Binlog_relay_IO_delegate::init_param(Binlog_relay_IO_param *param, > Master_info *mi) > { > + param->mi = mi; > param->mysql= mi->mysql; > param->user= mi->user; > param->host= mi->host; > @@ -540,6 +591,20 @@ int unregister_binlog_relay_io_observer(Binlog_relay_IO_observer *observer, void > { > return binlog_relay_io_delegate->remove_observer(observer, (st_plugin_int *)p); > } > + > +/* get master log pos for a Master_info struct */ > +size_t get_master_log_pos(const Master_info* mi, > + char *filename_buf, my_off_t *filepos) > +{ > + mysql_mutex_t *mutex= &mi->rli.data_lock; > + > + mysql_mutex_lock(mutex); > + *filepos= mi->rli.group_master_log_pos; > + strncpy(filename_buf, mi->rli.group_master_log_name, FN_REFLEN); > + mysql_mutex_unlock(mutex); > + return strnlen(filename_buf, FN_REFLEN); > +} > + > #else > int register_binlog_transmit_observer(Binlog_transmit_observer *observer, void *p) > { > @@ -560,4 +625,13 @@ int unregister_binlog_relay_io_observer(Binlog_relay_IO_observer *observer, void > { > return 0; > } > + > +size_t get_master_log_pos(const Master_info* mi, > + char *filename_buf, my_off_t *filepos) > +{ > + *filepos= 0; > + filename_buf[0]= 0; > + return 0; > +} > + > #endif /* HAVE_REPLICATION */ > diff --git a/sql/rpl_handler.h b/sql/rpl_handler.h > index afcfd9d..5119ee4 100644 > --- a/sql/rpl_handler.h > +++ b/sql/rpl_handler.h > @@ -142,7 +142,7 @@ class Trans_delegate > :public Delegate { > public: > typedef Trans_observer Observer; > - int before_commit(THD *thd, bool all); > + int before_commit(THD *thd); > int before_rollback(THD *thd, bool all); > int after_commit(THD *thd, bool all); > int after_rollback(THD *thd, bool all); > diff --git a/sql/rpl_rli.h b/sql/rpl_rli.h > index 56300c6..d60a122 100644 > --- a/sql/rpl_rli.h > +++ b/sql/rpl_rli.h > @@ -154,7 +154,9 @@ class Relay_log_info : public Slave_reporting_capability > standard lock acquisition order to avoid deadlocks: > run_lock, data_lock, relay_log.LOCK_log, relay_log.LOCK_index > */ > - mysql_mutex_t data_lock, run_lock; > + mutable mysql_mutex_t data_lock; > + mysql_mutex_t run_lock; > + > /* > start_cond is broadcast when SQL thread is started > stop_cond - when stopped > diff --git a/sql/sql_repl.cc b/sql/sql_repl.cc > index d9ae6ca..9662058 100644 > --- a/sql/sql_repl.cc > +++ b/sql/sql_repl.cc > @@ -818,6 +818,13 @@ static int send_heartbeat_event(binlog_send_info *info, > packet->append(b, sizeof(b)); > } > > + if (RUN_HOOK(binlog_transmit, before_send_event, > + (info->thd, info->flags, packet, 0, 0))) > + { > + info->error= ER_UNKNOWN_ERROR; > + DBUG_RETURN(-1); > + } > + > if (my_net_write(net, (uchar*) packet->ptr(), packet->length()) || > net_flush(net)) > { > @@ -825,6 +832,13 @@ static int send_heartbeat_event(binlog_send_info *info, > DBUG_RETURN(-1); > } > > + if (RUN_HOOK(binlog_transmit, after_send_event, > + (info->thd, info->flags, packet))) > + { > + info->error= ER_UNKNOWN_ERROR; > + DBUG_RETURN(-1); > + } > + > DBUG_RETURN(0); > } >

1 0

Re: [Maria-developers] [Commits] 51926a1: Fix assertion failure.
by Sergei Golubchik 16 May '15

16 May '15

Hi, Jan! On May 16, Jan Lindström wrote: > revision-id: 51926a1d2ed756902257fe802b1a01abb89a8fa3 > parent(s): 8d2117c0fd0a2211f1ec55f1d61fc5ae822c0ccf > committer: Jan Lindström > branch nick: 10.1-encrypt > timestamp: 2015-05-16 08:58:03 +0300 > message: > > Fix assertion failure. > > --- > mysql-test/suite/encryption/r/encrypt_and_grep.result | 6 +----- > mysql-test/suite/encryption/t/encrypt_and_grep.test | 10 +--------- > storage/innobase/fil/fil0crypt.cc | 9 +++++++-- > storage/xtradb/fil/fil0crypt.cc | 9 +++++++-- > 4 files changed, 16 insertions(+), 18 deletions(-) > > diff --git a/mysql-test/suite/encryption/t/encrypt_and_grep.test b/mysql-test/suite/encryption/t/encrypt_and_grep.test > index 1968b71..0301aa2 100644 > --- a/mysql-test/suite/encryption/t/encrypt_and_grep.test > +++ b/mysql-test/suite/encryption/t/encrypt_and_grep.test > @@ -52,7 +52,7 @@ insert t3 values (repeat('dummy', 42)); > SET GLOBAL innodb_encrypt_tables = off; > SET GLOBAL innodb_encryption_threads = 4; > > ---echo # Wait max 10 min for key encryption threads to decrypt all space > +--echo # Wait max 10 min for key encryption threads to decrypt all spacew A typo? Did you mean "spaces" ? > --let $wait_timeout= 600 > --let $wait_condition=SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION WHERE MIN_KEY_VERSION <> 0; > --source include/wait_condition.inc > @@ -69,14 +69,6 @@ SET GLOBAL innodb_encryption_threads = 4; > --echo # t3 no on expecting FOUND > -- let SEARCH_FILE=$t3_IBD > -- source include/search_pattern_in_file.inc > ---let SEARCH_PATTERN=foobar > ---echo # ibdata1 expecting NOT FOUND > --- let SEARCH_FILE=$ib1_IBD > --- source include/search_pattern_in_file.inc > ---let SEARCH_PATTERN=temp > ---echo # ibdata1 expecting NOT FOUND > --- let SEARCH_FILE=$ib1_IBD > --- source include/search_pattern_in_file.inc Why? > --echo # Now turn on encryption and wait for threads to encrypt all spaces > SET GLOBAL innodb_encryption_threads = 4; > diff --git a/storage/innobase/fil/fil0crypt.cc b/storage/innobase/fil/fil0crypt.cc > index 65be860..0c2280f 100644 > --- a/storage/innobase/fil/fil0crypt.cc > +++ b/storage/innobase/fil/fil0crypt.cc > @@ -2128,8 +2128,13 @@ fil_crypt_complete_rotate_space( > crypt_data->rotate_state.end_lsn = state->end_lsn; > } > > - ut_a(crypt_data->rotate_state.active_threads > 0); > - crypt_data->rotate_state.active_threads--; > + /* Remember that we can try to rotate tablespaces with > + FIL_SPACE_ENCRYPTION_OFF and we might want to scrub them > + also. */ Do you mean that when rotating FIL_SPACE_ENCRYPTION_OFF spaces crypt_data->rotate_state.active_threads == 0? > + if (crypt_data->rotate_state.active_threads > 0) { > + crypt_data->rotate_state.active_threads--; > + } > + > bool last = crypt_data->rotate_state.active_threads == 0; Regards, Sergei

1 0

[Maria-developers] Need help with review/suggestions for MDEV-8075 (was Re: [Commits] c325a34: MDEV-8075 ...)
by Kristian Nielsen 15 May '15

15 May '15

Hi Serg, Svoj, I need some help with MDEV-8075, as it involves code around transaction start/end which I am not familiar with. I hope one of you know enough about that part of the server to help me? I have a partial patch (below), but it still has some problems remaining and I am not sure how to solve it properly. I tried to base the patch on some corresponding MySQL code, which was only partially merged to MariaDB 10.1. The basic problem is to remember during a transaction whether DDL was used, and then at the end of the transaction in binlogging, to set a flag in the GTID event if DDL is used. The concrete problem is with this kind of SQL: BEGIN; INSERT INTO innodb_table VALUES (1); ALTER TABLE temporary_table ADD b INT; In this case the ALTER TABLE does not get the "ddl" mark in my patch. The reason is that during mysql_alter_table(), THD_TRANS::reset() gets called, which clears the flag DID_DDL that was set in mysql_execute_command(). Call stack: #0 THD_TRANS::reset #1 trans_commit_implicit #2 ha_enable_transaction #3 mysql_trans_commit_alter_copy_data #4 copy_data_between_tables #5 mysql_alter_table #6 Sql_cmd_alter_table::execute #7 mysql_execute_command (An earlier version of the patch looked directly at thd->lex->sql_command during binlogging to see if it was a DDL. That caused another problem: the INSERT INTO got wrongly marked as DDL, because it is implicitly committed by an ALTER TABLE statement). But the more general problem is that I am very unsure about the proper places to put in code to clear the required flags at the start of a transaction, and to copy relevant flags from "stmt" to "all" transaction. I tried using what I found in MySQL code, and to look for where the existing MariaDB code copies the existing THD_TRANS::modified_non_trans_table flag. But I am very unsure that I choose the right places, the existing code seems rather ad-hoc (I'm wondering if it is even correct). I also replaced some add-hoc code that clears THD_TRANS::modified_non_trans_table directly with calls to THD_TRANS::reset(); that seems more appropriate when more flags are affected, but it does more things (notably clearing no_2pc), so again I'm very unsure if it is correct... Any suggestions welcomed. The underlying bug makes optimistic parallel replication quite broken with temporary tables and statement-based binlogging in 10.1 (can probably even crash the server), so it is of some importance to get fixed.... - Kristian. ----------------------------------------------------------------------- revision-id: c325a34d93b931d1c927255c18df94e1560af6eb parent(s): 53382ac128d9571b5f779bf625567fc3ea0b475e committer: Kristian Nielsen branch nick: mariadb timestamp: 2015-05-15 14:56:54 +0200 message: MDEV-8075: DROP TEMPORARY TABLE not marked as ddl, causing optimistic parallel replication to fail CREATE/DROP TEMPORARY TABLE are not safe to optimistically replicate in parallel with other transactions, so they need to be marked as "ddl" in the binlog. This was already done for stand-alone CREATE/DROP TEMPORARY. But temporary tables can also be created and dropped inside a BEGIN...END transaction, and such transactions were not marked as ddl. Nor was the DROP TEMPORARY TABLE statement emitted implicitly when a client connection is closed. So this patch adds such ddl mark for the missing cases. --- .../include/binlog_parallel_replication_marks.test | 83 ++++++++++++++++++ .../r/binlog_parallel_replication_marks_row.result | 98 ++++++++++++++++++++++ ...inlog_parallel_replication_marks_stm_mix.result | 95 +++++++++++++++++++++ .../t/binlog_parallel_replication_marks_row.test | 3 + .../binlog_parallel_replication_marks_stm_mix.test | 3 + .../suite/rpl/r/rpl_parallel_optimistic.result | 27 ++++++ .../suite/rpl/t/rpl_parallel_optimistic.test | 25 ++++++ sql/handler.h | 14 ++++ sql/log_event.cc | 6 +- sql/sql_base.cc | 1 + sql/sql_class.h | 10 +++ sql/sql_insert.cc | 8 ++ sql/sql_parse.cc | 4 +- sql/sql_table.cc | 22 +++-- sql/transaction.cc | 25 +++--- 15 files changed, 400 insertions(+), 24 deletions(-) diff --git a/mysql-test/include/binlog_parallel_replication_marks.test b/mysql-test/include/binlog_parallel_replication_marks.test new file mode 100644 index 0000000..4775e30 --- /dev/null +++ b/mysql-test/include/binlog_parallel_replication_marks.test @@ -0,0 +1,83 @@ +# Test the markings on GTID events (ddl, waited, trans, +# @@skip_parallel_replication) that are used to control parallel +# replication on the slave. + +--source include/have_innodb.inc + +RESET MASTER; +--source include/wait_for_binlog_checkpoint.inc + +--let $stable_stamp= `SELECT UNIX_TIMESTAMP("2020-01-21 15:32:22")` +eval set timestamp=$stable_stamp; + +CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; +--let $binlog_pos1=query_get_value(SHOW MASTER STATUS, Position, 1) +/* GTID */ INSERT INTO t1 VALUES (1,0); +/* GTID */ BEGIN; +/* GTID */ INSERT INTO t1 VALUES (2,0); +/* GTID */ ALTER TABLE t1 ADD c INT; +/* GTID */ INSERT INTO t1 VALUES (3,0,0); +/* GTID */ COMMIT; +/* GTID */ BEGIN; +/* GTID */ UPDATE t1 SET b=1, c=1 WHERE a=2; +/* GTID */ CREATE TEMPORARY TABLE t2 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; +/* GTID */ INSERT INTO t2 VALUES (4,10), (5,20); +/* GTID */ INSERT INTO t1 SELECT a, 2, b FROM t2; +/* GTID */ DROP TEMPORARY TABLE t2; +/* GTID */ INSERT INTO t1 VALUES (6, 3, 0); +/* GTID */ COMMIT; +/* GTID */ CREATE TEMPORARY TABLE t3 (a INT PRIMARY KEY) ENGINE=InnoDB; +/* GTID */ BEGIN; +/* GTID */ DELETE FROM t1 WHERE a=5; +/* GTID */ INSERT INTO t3 VALUES (7); +/* GTID */ INSERT INTO t1 SELECT a, 4, 0 FROM t3; +/* GTID */ UPDATE t1 SET c=1 WHERE a=7; +/* GTID */ DROP TEMPORARY TABLE t3; +/* GTID */ COMMIT; +/* GTID */ CREATE TEMPORARY TABLE t4 (a INT PRIMARY KEY) ENGINE=InnoDB; +/* GTID */ BEGIN; +/* GTID */ INSERT INTO t1 VALUES (8, 5, 0); +/* GTID */ ALTER TABLE t4 ADD b INT; +/* GTID */ INSERT INTO t1 VALUES (9, 5, 1); +/* GTID */ COMMIT; +connect (tmp_con,localhost,root,,); +eval set timestamp=$stable_stamp; +/* GTID */ INSERT INTO t1 VALUES (10, 6, 0); +/* GTID */ BEGIN; +/* GTID */ CREATE TEMPORARY TABLE t5 (a INT PRIMARY KEY) ENGINE=InnoDB; +/* GTID */ INSERT INTO t1 VALUES (11, 7, 0); +/* GTID */ COMMIT; +--let $before_drop_pos=query_get_value(SHOW MASTER STATUS, Position, 1) +disconnect tmp_con; +connection default; + +# We need to wait for the implicit DROP TEMPORARY TABLE to be logged after +# tmp_con disconnect, otherwise we get sporadic test failures. +--let $wait_condition= SELECT variable_value > $before_drop_pos FROM information_schema.global_status WHERE variable_name = 'binlog_snapshot_position' +--source include/wait_condition.inc + +--let $binlog_pos2=query_get_value(SHOW MASTER STATUS, Position, 1) + +--let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1) +FLUSH LOGS; + +--let $MYSQLD_DATADIR= `select @@datadir` +--let $file= $MYSQLTEST_VARDIR/tmp/binlog_parallel_replication_marks.out +--let OUTPUT_FILE=$file +exec $MYSQL_BINLOG --start_position=$binlog_pos1 --stop_position=$binlog_pos2 $MYSQLD_DATADIR/$binlog_file > $file; + +perl; +my $file= $ENV{'OUTPUT_FILE'}; +open F, "<", $file + or die "Unable to open file '$file': $!\n"; +while (<F>) { + s/GTID \d+-\d+-\d+/GTID #-#-#/; + s/end_log_pos \d+/end_log_pos #/; + s/table id \d+/table id #/; + s/mapped to number \d+/mapped to number #/; + print if /GTID|BEGIN|COMMIT|Table_map|Write_rows|Update_rows|Delete_rows|generated by server|40005 TEMPORARY/; +} +close F; +EOF + +DROP TABLE t1; diff --git a/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_row.result b/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_row.result new file mode 100644 index 0000000..b9c8666 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_row.result @@ -0,0 +1,98 @@ +RESET MASTER; +set timestamp=1579617142; +CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; +/* GTID */ INSERT INTO t1 VALUES (1,0); +/* GTID */ BEGIN; +/* GTID */ INSERT INTO t1 VALUES (2,0); +/* GTID */ ALTER TABLE t1 ADD c INT; +/* GTID */ INSERT INTO t1 VALUES (3,0,0); +/* GTID */ COMMIT; +/* GTID */ BEGIN; +/* GTID */ UPDATE t1 SET b=1, c=1 WHERE a=2; +/* GTID */ CREATE TEMPORARY TABLE t2 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; +/* GTID */ INSERT INTO t2 VALUES (4,10), (5,20); +/* GTID */ INSERT INTO t1 SELECT a, 2, b FROM t2; +/* GTID */ DROP TEMPORARY TABLE t2; +/* GTID */ INSERT INTO t1 VALUES (6, 3, 0); +/* GTID */ COMMIT; +/* GTID */ CREATE TEMPORARY TABLE t3 (a INT PRIMARY KEY) ENGINE=InnoDB; +/* GTID */ BEGIN; +/* GTID */ DELETE FROM t1 WHERE a=5; +/* GTID */ INSERT INTO t3 VALUES (7); +/* GTID */ INSERT INTO t1 SELECT a, 4, 0 FROM t3; +/* GTID */ UPDATE t1 SET c=1 WHERE a=7; +/* GTID */ DROP TEMPORARY TABLE t3; +/* GTID */ COMMIT; +/* GTID */ CREATE TEMPORARY TABLE t4 (a INT PRIMARY KEY) ENGINE=InnoDB; +/* GTID */ BEGIN; +/* GTID */ INSERT INTO t1 VALUES (8, 5, 0); +/* GTID */ ALTER TABLE t4 ADD b INT; +/* GTID */ INSERT INTO t1 VALUES (9, 5, 1); +/* GTID */ COMMIT; +set timestamp=1579617142; +/* GTID */ INSERT INTO t1 VALUES (10, 6, 0); +/* GTID */ BEGIN; +/* GTID */ CREATE TEMPORARY TABLE t5 (a INT PRIMARY KEY) ENGINE=InnoDB; +/* GTID */ INSERT INTO t1 VALUES (11, 7, 0); +/* GTID */ COMMIT; +FLUSH LOGS; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# trans +BEGIN +#200121 15:32:22 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +#200121 15:32:22 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# trans +BEGIN +#200121 15:32:22 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +#200121 15:32:22 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# ddl +/* GTID */ ALTER TABLE t1 ADD c INT +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# trans +BEGIN +#200121 15:32:22 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +#200121 15:32:22 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# ddl +BEGIN +#200121 15:32:22 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +#200121 15:32:22 server id 1 end_log_pos # Update_rows: table id # flags: STMT_END_F +#200121 15:32:22 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +#200121 15:32:22 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +DROP TEMPORARY TABLE IF EXISTS `test`.`t2` /* generated by server */ +#200121 15:32:22 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +#200121 15:32:22 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# ddl +BEGIN +#200121 15:32:22 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +#200121 15:32:22 server id 1 end_log_pos # Delete_rows: table id # flags: STMT_END_F +#200121 15:32:22 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +#200121 15:32:22 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +#200121 15:32:22 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +#200121 15:32:22 server id 1 end_log_pos # Update_rows: table id # flags: STMT_END_F +DROP TEMPORARY TABLE IF EXISTS `test`.`t3` /* generated by server */ +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# trans +BEGIN +#200121 15:32:22 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +#200121 15:32:22 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# trans +BEGIN +#200121 15:32:22 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +#200121 15:32:22 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# trans +BEGIN +#200121 15:32:22 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +#200121 15:32:22 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# trans +BEGIN +#200121 15:32:22 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +#200121 15:32:22 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# ddl +DROP /*!40005 TEMPORARY */ TABLE IF EXISTS `t5` +DROP TABLE t1; diff --git a/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_stm_mix.result b/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_stm_mix.result new file mode 100644 index 0000000..c3dc00d --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_stm_mix.result @@ -0,0 +1,95 @@ +RESET MASTER; +set timestamp=1579617142; +CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; +/* GTID */ INSERT INTO t1 VALUES (1,0); +/* GTID */ BEGIN; +/* GTID */ INSERT INTO t1 VALUES (2,0); +/* GTID */ ALTER TABLE t1 ADD c INT; +/* GTID */ INSERT INTO t1 VALUES (3,0,0); +/* GTID */ COMMIT; +/* GTID */ BEGIN; +/* GTID */ UPDATE t1 SET b=1, c=1 WHERE a=2; +/* GTID */ CREATE TEMPORARY TABLE t2 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; +/* GTID */ INSERT INTO t2 VALUES (4,10), (5,20); +/* GTID */ INSERT INTO t1 SELECT a, 2, b FROM t2; +/* GTID */ DROP TEMPORARY TABLE t2; +/* GTID */ INSERT INTO t1 VALUES (6, 3, 0); +/* GTID */ COMMIT; +/* GTID */ CREATE TEMPORARY TABLE t3 (a INT PRIMARY KEY) ENGINE=InnoDB; +/* GTID */ BEGIN; +/* GTID */ DELETE FROM t1 WHERE a=5; +/* GTID */ INSERT INTO t3 VALUES (7); +/* GTID */ INSERT INTO t1 SELECT a, 4, 0 FROM t3; +/* GTID */ UPDATE t1 SET c=1 WHERE a=7; +/* GTID */ DROP TEMPORARY TABLE t3; +/* GTID */ COMMIT; +/* GTID */ CREATE TEMPORARY TABLE t4 (a INT PRIMARY KEY) ENGINE=InnoDB; +/* GTID */ BEGIN; +/* GTID */ INSERT INTO t1 VALUES (8, 5, 0); +/* GTID */ ALTER TABLE t4 ADD b INT; +/* GTID */ INSERT INTO t1 VALUES (9, 5, 1); +/* GTID */ COMMIT; +set timestamp=1579617142; +/* GTID */ INSERT INTO t1 VALUES (10, 6, 0); +/* GTID */ BEGIN; +/* GTID */ CREATE TEMPORARY TABLE t5 (a INT PRIMARY KEY) ENGINE=InnoDB; +/* GTID */ INSERT INTO t1 VALUES (11, 7, 0); +/* GTID */ COMMIT; +FLUSH LOGS; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# trans +BEGIN +/* GTID */ INSERT INTO t1 VALUES (1,0) +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# trans +BEGIN +/* GTID */ INSERT INTO t1 VALUES (2,0) +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# ddl +/* GTID */ ALTER TABLE t1 ADD c INT +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# trans +BEGIN +/* GTID */ INSERT INTO t1 VALUES (3,0,0) +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# ddl +BEGIN +/* GTID */ UPDATE t1 SET b=1, c=1 WHERE a=2 +/* GTID */ CREATE TEMPORARY TABLE t2 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB +/* GTID */ INSERT INTO t2 VALUES (4,10), (5,20) +/* GTID */ INSERT INTO t1 SELECT a, 2, b FROM t2 +DROP TEMPORARY TABLE `t2` /* generated by server */ +/* GTID */ INSERT INTO t1 VALUES (6, 3, 0) +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# ddl +/* GTID */ CREATE TEMPORARY TABLE t3 (a INT PRIMARY KEY) ENGINE=InnoDB +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# ddl +BEGIN +/* GTID */ DELETE FROM t1 WHERE a=5 +/* GTID */ INSERT INTO t3 VALUES (7) +/* GTID */ INSERT INTO t1 SELECT a, 4, 0 FROM t3 +/* GTID */ UPDATE t1 SET c=1 WHERE a=7 +DROP TEMPORARY TABLE `t3` /* generated by server */ +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# ddl +/* GTID */ CREATE TEMPORARY TABLE t4 (a INT PRIMARY KEY) ENGINE=InnoDB +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# trans +BEGIN +/* GTID */ INSERT INTO t1 VALUES (8, 5, 0) +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# ddl +/* GTID */ ALTER TABLE t4 ADD b INT +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# trans +BEGIN +/* GTID */ INSERT INTO t1 VALUES (9, 5, 1) +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# trans +BEGIN +/* GTID */ INSERT INTO t1 VALUES (10, 6, 0) +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# ddl +BEGIN +/* GTID */ CREATE TEMPORARY TABLE t5 (a INT PRIMARY KEY) ENGINE=InnoDB +/* GTID */ INSERT INTO t1 VALUES (11, 7, 0) +COMMIT/*!*/; +#200121 15:32:22 server id 1 end_log_pos # GTID #-#-# ddl +DROP /*!40005 TEMPORARY */ TABLE IF EXISTS `t5` +DROP TABLE t1; diff --git a/mysql-test/suite/binlog/t/binlog_parallel_replication_marks_row.test b/mysql-test/suite/binlog/t/binlog_parallel_replication_marks_row.test new file mode 100644 index 0000000..8289848 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_parallel_replication_marks_row.test @@ -0,0 +1,3 @@ +--source include/have_log_bin.inc +--source include/have_binlog_format_row.inc +--source include/binlog_parallel_replication_marks.test diff --git a/mysql-test/suite/binlog/t/binlog_parallel_replication_marks_stm_mix.test b/mysql-test/suite/binlog/t/binlog_parallel_replication_marks_stm_mix.test new file mode 100644 index 0000000..15042b3 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_parallel_replication_marks_stm_mix.test @@ -0,0 +1,3 @@ +--source include/have_log_bin.inc +--source include/have_binlog_format_mixed_or_statement.inc +--source include/binlog_parallel_replication_marks.test diff --git a/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result b/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result index 7593b49..2da7830 100644 --- a/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result +++ b/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result @@ -455,6 +455,33 @@ a b include/stop_slave.inc SET GLOBAL debug_dbug= @old_debug; include/start_slave.inc +*** MDEV-8075: DROP TEMPORARY TABLE not marked as ddl, causing optimistic parallel replication to fail *** +include/stop_slave.inc +INSERT INTO t1 VALUES (40, 10); +CREATE TEMPORARY TABLE t1 (a INT PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (41); +BEGIN; +INSERT INTO t2 SELECT a, 20 FROM t1; +DROP TEMPORARY TABLE t1; +COMMIT; +INSERT INTO t1 VALUES (42, 10); +include/save_master_gtid.inc +SELECT * FROM t1 WHERE a >= 40 ORDER BY a; +a b +40 10 +42 10 +SELECT * FROM t2 WHERE a >= 40 ORDER BY a; +a b +41 20 +include/start_slave.inc +include/sync_with_master_gtid.inc +SELECT * FROM t1 WHERE a >= 40 ORDER BY a; +a b +40 10 +42 10 +SELECT * FROM t2 WHERE a >= 40 ORDER BY a; +a b +41 20 include/stop_slave.inc SET GLOBAL slave_parallel_mode=@old_parallel_mode; SET GLOBAL slave_parallel_threads=@old_parallel_threads; diff --git a/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test b/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test index e7d4f18..376c8d6 100644 --- a/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test +++ b/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test @@ -429,6 +429,31 @@ SET GLOBAL debug_dbug= @old_debug; --source include/start_slave.inc +--echo *** MDEV-8075: DROP TEMPORARY TABLE not marked as ddl, causing optimistic parallel replication to fail *** + +--connection server_2 +--source include/stop_slave.inc + +--connection server_1 +INSERT INTO t1 VALUES (40, 10); +CREATE TEMPORARY TABLE t1 (a INT PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (41); +BEGIN; +INSERT INTO t2 SELECT a, 20 FROM t1; +DROP TEMPORARY TABLE t1; +COMMIT; +INSERT INTO t1 VALUES (42, 10); +--source include/save_master_gtid.inc +SELECT * FROM t1 WHERE a >= 40 ORDER BY a; +SELECT * FROM t2 WHERE a >= 40 ORDER BY a; + +--connection server_2 +--source include/start_slave.inc +--source include/sync_with_master_gtid.inc +SELECT * FROM t1 WHERE a >= 40 ORDER BY a; +SELECT * FROM t2 WHERE a >= 40 ORDER BY a; + + # Clean up. --connection server_2 diff --git a/sql/handler.h b/sql/handler.h index 279bfaf..8faec7f 100644 --- a/sql/handler.h +++ b/sql/handler.h @@ -1444,16 +1444,30 @@ struct THD_TRANS static unsigned int const CREATED_TEMP_TABLE= 0x02; static unsigned int const DROPPED_TEMP_TABLE= 0x04; static unsigned int const DID_WAIT= 0x08; + static unsigned int const DID_DDL= 0x10; void mark_created_temp_table() { DBUG_PRINT("debug", ("mark_created_temp_table")); m_unsafe_rollback_flags|= CREATED_TEMP_TABLE; } + void mark_dropped_temp_table() + { + DBUG_PRINT("debug", ("mark_dropped_temp_table")); + m_unsafe_rollback_flags|= DROPPED_TEMP_TABLE; + } + bool has_created_dropped_temp_table() const { + return + (m_unsafe_rollback_flags & (CREATED_TEMP_TABLE|DROPPED_TEMP_TABLE)) != 0; + } void mark_trans_did_wait() { m_unsafe_rollback_flags|= DID_WAIT; } bool trans_did_wait() const { return (m_unsafe_rollback_flags & DID_WAIT) != 0; } + void mark_trans_did_ddl() { m_unsafe_rollback_flags|= DID_DDL; } + bool trans_did_ddl() const { + return (m_unsafe_rollback_flags & DID_DDL) != 0; + } }; diff --git a/sql/log_event.cc b/sql/log_event.cc index aa34996..98cc4b1 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -6405,8 +6405,10 @@ Gtid_log_event::Gtid_log_event(THD *thd_arg, uint64 seq_no_arg, if (thd_arg->transaction.stmt.trans_did_wait() || thd_arg->transaction.all.trans_did_wait()) flags2|= FL_WAITED; - if (sql_command_flags[thd->lex->sql_command] & - (CF_DISALLOW_IN_RO_TRANS | CF_AUTO_COMMIT_TRANS)) + if (thd_arg->transaction.stmt.trans_did_ddl() || + thd_arg->transaction.stmt.has_created_dropped_temp_table() || + thd_arg->transaction.all.trans_did_ddl() || + thd_arg->transaction.all.has_created_dropped_temp_table()) flags2|= FL_DDL; else if (is_transactional) flags2|= FL_TRANSACTIONAL; diff --git a/sql/sql_base.cc b/sql/sql_base.cc index c400598..6ef46ca 100644 --- a/sql/sql_base.cc +++ b/sql/sql_base.cc @@ -1253,6 +1253,7 @@ bool close_temporary_tables(THD *thd) thd->variables.character_set_client= cs_save; thd->get_stmt_da()->set_overwrite_status(true); + thd->transaction.stmt.mark_dropped_temp_table(); if ((error= (mysql_bin_log.write(&qinfo) || error))) { /* diff --git a/sql/sql_class.h b/sql/sql_class.h index a39c8cd..11eab4e 100644 --- a/sql/sql_class.h +++ b/sql/sql_class.h @@ -3953,6 +3953,16 @@ class THD :public Statement, { main_lex.restore_set_statement_var(); } + /* Copy relevant `stmt` transaction flags to `all` transaction. */ + void merge_unsafe_rollback_flags() + { + if (transaction.stmt.modified_non_trans_table) + transaction.all.modified_non_trans_table= TRUE; + transaction.all.m_unsafe_rollback_flags|= + (transaction.stmt.m_unsafe_rollback_flags & + (THD_TRANS::DID_WAIT | THD_TRANS::CREATED_TEMP_TABLE | + THD_TRANS::DROPPED_TEMP_TABLE | THD_TRANS::DID_DDL)); + } }; diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc index 24d75d9..90e68d5 100644 --- a/sql/sql_insert.cc +++ b/sql/sql_insert.cc @@ -4226,6 +4226,14 @@ void select_create::store_values(List<Item> &values) bool select_create::send_eof() { + /* + The routine that writes the statement in the binary log + is in select_insert::send_eof(). For that reason, we + mark the flag at this point. + */ + if (create_info->options & HA_LEX_CREATE_TMP_TABLE) + thd->transaction.stmt.mark_created_temp_table(); + if (select_insert::send_eof()) { abort_result_set(); diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc index 5eca972..142596f 100644 --- a/sql/sql_parse.cc +++ b/sql/sql_parse.cc @@ -2798,6 +2798,7 @@ mysql_execute_command(THD *thd) goto error; } } + thd->transaction.stmt.mark_trans_did_ddl(); } #ifndef DBUG_OFF @@ -6781,8 +6782,7 @@ void THD::reset_for_next_command() if (!thd->in_multi_stmt_transaction_mode()) { thd->variables.option_bits&= ~OPTION_KEEP_LOG; - thd->transaction.all.modified_non_trans_table= FALSE; - thd->transaction.all.m_unsafe_rollback_flags&= ~THD_TRANS::DID_WAIT; + thd->transaction.all.reset(); } DBUG_ASSERT(thd->security_ctx== &thd->main_security_ctx); thd->thread_specific_used= FALSE; diff --git a/sql/sql_table.cc b/sql/sql_table.cc index 1f96793..9abe012 100644 --- a/sql/sql_table.cc +++ b/sql/sql_table.cc @@ -2559,6 +2559,9 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST *tables, bool if_exists, if (non_trans_tmp_table_deleted || trans_tmp_table_deleted || non_tmp_table_deleted) { + if (non_trans_tmp_table_deleted || trans_tmp_table_deleted) + thd->transaction.stmt.mark_dropped_temp_table(); + query_cache_invalidate3(thd, tables, 0); if (!dont_log_query && mysql_bin_log.is_open()) { @@ -5014,6 +5017,9 @@ bool mysql_create_table(THD *thd, TABLE_LIST *create_table, if (thd->is_current_stmt_binlog_format_row() && create_info->tmp_table()) DBUG_RETURN(result); + if (create_info->options & HA_LEX_CREATE_TMP_TABLE) + thd->transaction.stmt.mark_created_temp_table(); + /* Write log if no error or if we already deleted a table */ if (!result || thd->log_current_statement) { @@ -5491,13 +5497,17 @@ bool mysql_create_like_table(THD* thd, TABLE_LIST* table, DBUG_PRINT("info", ("res: %d tmp_table: %d create_info->table: %p", res, create_info->tmp_table(), local_create_info.table)); - if (!res && create_info->tmp_table() && local_create_info.table) + if (create_info->tmp_table()) { - /* - Remember that tmp table creation was logged so that we know if - we should log a delete of it. - */ - local_create_info.table->s->table_creation_was_logged= 1; + thd->transaction.stmt.mark_created_temp_table(); + if (!res && local_create_info.table) + { + /* + Remember that tmp table creation was logged so that we know if + we should log a delete of it. + */ + local_create_info.table->s->table_creation_was_logged= 1; + } } do_logging= TRUE; } diff --git a/sql/transaction.cc b/sql/transaction.cc index f84c4e5..fa6d338 100644 --- a/sql/transaction.cc +++ b/sql/transaction.cc @@ -154,8 +154,7 @@ bool trans_begin(THD *thd, uint flags) The following set should not be needed as the flag should always be 0 when we come here. We should at some point change this to an assert. */ - thd->transaction.all.modified_non_trans_table= FALSE; - thd->transaction.all.m_unsafe_rollback_flags&= ~THD_TRANS::DID_WAIT; + thd->transaction.all.reset(); thd->has_waiter= false; thd->waiting_on_group_commit= false; @@ -251,8 +250,7 @@ bool trans_commit(THD *thd) else (void) RUN_HOOK(transaction, after_commit, (thd, FALSE)); thd->variables.option_bits&= ~(OPTION_BEGIN | OPTION_KEEP_LOG); - thd->transaction.all.modified_non_trans_table= FALSE; - thd->transaction.all.m_unsafe_rollback_flags&= ~THD_TRANS::DID_WAIT; + thd->transaction.all.reset(); thd->lex->start_transaction_opt= 0; DBUG_RETURN(MY_TEST(res)); @@ -299,8 +297,7 @@ bool trans_commit_implicit(THD *thd) } thd->variables.option_bits&= ~(OPTION_BEGIN | OPTION_KEEP_LOG); - thd->transaction.all.modified_non_trans_table= FALSE; - thd->transaction.all.m_unsafe_rollback_flags&= ~THD_TRANS::DID_WAIT; + thd->transaction.all.reset(); /* Upon implicit commit, reset the current transaction @@ -345,8 +342,7 @@ bool trans_rollback(THD *thd) thd->variables.option_bits&= ~(OPTION_BEGIN | OPTION_KEEP_LOG); /* Reset the binlog transaction marker */ thd->variables.option_bits&= ~OPTION_GTID_BEGIN; - thd->transaction.all.modified_non_trans_table= FALSE; - thd->transaction.all.m_unsafe_rollback_flags&= ~THD_TRANS::DID_WAIT; + thd->transaction.all.reset(); thd->lex->start_transaction_opt= 0; DBUG_RETURN(MY_TEST(res)); @@ -390,8 +386,7 @@ bool trans_rollback_implicit(THD *thd) preserve backward compatibility. */ thd->variables.option_bits&= ~(OPTION_KEEP_LOG); - thd->transaction.all.modified_non_trans_table= false; - thd->transaction.all.m_unsafe_rollback_flags&= ~THD_TRANS::DID_WAIT; + thd->transaction.all.reset(); /* Rollback should clear transaction_rollback_request flag. */ DBUG_ASSERT(! thd->transaction_rollback_request); @@ -427,6 +422,8 @@ bool trans_commit_stmt(THD *thd) */ DBUG_ASSERT(! thd->in_sub_stmt); + thd->merge_unsafe_rollback_flags(); + if (thd->transaction.stmt.ha_list) { if (WSREP_ON) @@ -481,6 +478,8 @@ bool trans_rollback_stmt(THD *thd) */ DBUG_ASSERT(! thd->in_sub_stmt); + thd->merge_unsafe_rollback_flags(); + if (thd->transaction.stmt.ha_list) { if (WSREP_ON) @@ -913,8 +912,7 @@ bool trans_xa_commit(THD *thd) } thd->variables.option_bits&= ~(OPTION_BEGIN | OPTION_KEEP_LOG); - thd->transaction.all.modified_non_trans_table= FALSE; - thd->transaction.all.m_unsafe_rollback_flags&= ~THD_TRANS::DID_WAIT; + thd->transaction.all.reset(); thd->server_status&= ~(SERVER_STATUS_IN_TRANS | SERVER_STATUS_IN_TRANS_READONLY); DBUG_PRINT("info", ("clearing SERVER_STATUS_IN_TRANS")); @@ -969,8 +967,7 @@ bool trans_xa_rollback(THD *thd) res= xa_trans_force_rollback(thd); thd->variables.option_bits&= ~(OPTION_BEGIN | OPTION_KEEP_LOG); - thd->transaction.all.modified_non_trans_table= FALSE; - thd->transaction.all.m_unsafe_rollback_flags&= ~THD_TRANS::DID_WAIT; + thd->transaction.all.reset(); thd->server_status&= ~(SERVER_STATUS_IN_TRANS | SERVER_STATUS_IN_TRANS_READONLY); DBUG_PRINT("info", ("clearing SERVER_STATUS_IN_TRANS")); _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Step#5: MDEV-7950, , Item_func::type() takes 0.26% in OLTP RO
by Alexander Barkov 14 May '15

14 May '15

Hi Sergey, please review the next step for MDEV-7950. This step changes the function remove_eq_conds() into a method in Item. It removes 6 virtual calls for Item_func::type(), and adds only 2 virtual calls for Item***::remove_eq_conds(). Thanks.

1 0

Re: [Maria-developers] 8a0f331: MDEV-7943 - pthread_getspecific() takes 0.76% in OLTP RO
by Sergei Golubchik 13 May '15

13 May '15

Hi, Sergey! On Apr 22, svoj(a)mariadb.org wrote: > revision-id: 8a0f3310275f7b4fa445f907140f677910e18999 > parent(s): 7feee74dd30c96bd50d1c90e4ce3b06a656b17a5 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-04-22 14:18:51 +0400 > message: > > MDEV-7943 - pthread_getspecific() takes 0.76% in OLTP RO > > Avoid calling current_thd from thd_kill_level(). This reduces number of > pthread_getspecific() calls from 776 to 354. > > Also thd_kill_level(NULL) is not permitted anymore: this saves one condition. > > --- > plugin/semisync/semisync_master.cc | 4 ++-- > sql/sql_class.cc | 20 +++++++++----------- > 2 files changed, 11 insertions(+), 13 deletions(-) > > diff --git a/plugin/semisync/semisync_master.cc b/plugin/semisync/semisync_master.cc > index c88c162..b1f7fbd 100644 > --- a/plugin/semisync/semisync_master.cc > +++ b/plugin/semisync/semisync_master.cc > @@ -635,7 +635,7 @@ int ReplSemiSyncMaster::commitTrx(const char* trx_wait_binlog_name, > (int)is_on()); > } > > - while (is_on() && !thd_killed(NULL)) > + while (is_on() && !thd_killed(current_thd)) thd_killed() is a function of the kill_statement service (see include/mysql/service_kill_statement.h). It is created for plugins to use. But current_thd is not. Plugins generally have no access to it. > { > if (reply_file_name_inited_) > { > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index f273974..d32b14d 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -4208,20 +4208,18 @@ void THD::restore_backup_open_tables_state(Open_tables_backup *backup) > */ > extern "C" enum thd_kill_levels thd_kill_level(const MYSQL_THD thd) > { > - THD* current= current_thd; > - > - if (!thd) > - thd= current; > - > - if (thd == current) I wonder if there's a cheaper way to check "if THD is current_thd". E.g. THD stores the "end of stack" value. If it'll also store "beginning of stack", one can easily check whether a local variable is within these limits, which would mean the THD is current. > - { > - Apc_target *apc_target= (Apc_target*)&thd->apc_target; > - if (apc_target->have_apc_requests()) > - apc_target->process_apc_requests(); > - } > + DBUG_ASSERT(thd); > > if (likely(thd->killed == NOT_KILLED)) > + { > + Apc_target *apc_target= (Apc_target*) &thd->apc_target; > + if (unlikely(apc_target->have_apc_requests())) > + { > + if (thd == current_thd) > + apc_target->process_apc_requests(); > + } > return THD_IS_NOT_KILLED; > + } Why did you put process_apc_requests() under thd->killed == NOT_KILLED ? > return thd->killed & KILL_HARD_BIT ? THD_ABORT_ASAP : THD_ABORT_SOFTLY; > } Regards, Sergei

2 4

[Maria-developers] Step#4: MDEV-7950 Item_func::type() takes 0.26% in OLTP RO
by Alexander Barkov 13 May '15

13 May '15

Hi Sergey, Please review a patch for the next step for MDEV-7950 (one small thing at a time, to avoid huge unclear patches) Thanks.

2 4

Re: [Maria-developers] 7feee74: MDEV-7943 - pthread_getspecific() takes 0.76% in OLTP RO
by Sergei Golubchik 12 May '15

12 May '15

Hi, Sergey! On Apr 22, svoj(a)mariadb.org wrote: > revision-id: 7feee74dd30c96bd50d1c90e4ce3b06a656b17a5 > parent(s): 696188fe2f0756a98914df5b95ba302f7e5c55df > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-04-22 13:29:56 +0400 > message: > > MDEV-7943 - pthread_getspecific() takes 0.76% in OLTP RO > > Added THD argument to select_result and all derivative classes. This > reduces number of pthread_getspecific calls from 796 to 776 per OLTP > RO transaction. Lots of changes for a little gain :) Ok to push. An idea. Create a variant of Sql_alloc (say, Sql_alloc_explicit, a parent of Sql_alloc) that does not use current_thd but requires MEM_ROOT to be specified explicitly. And use it for classes where you want it. Regards, Sergei

2 1

[Maria-developers] MariaDB Debian deprecation policy
by Daniel Bartholomew 11 May '15

11 May '15

Looking for input on this. On the MariaDB Deprecation Policy page we list planned deprecation dates for most of the platforms we build on. https://mariadb.com/kb/en/mariadb/deprecation-policy/ The various Debian releases we build for are listed, but there are no deprecation dates. According to https://www.debian.org/releases/ with the release of Debian 8 "Jessie" both Squeeze and Wheezy are now "obsolete". Should we deprecate both of them and stop providing builds? Or should we just deprecate Squeeze? Thanks. -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

3 3

Re: [Maria-developers] 0520a00: MDEV-7964 - delete_dynamic() takes 0.12% in OLTP RO
by Sergei Golubchik 10 May '15

10 May '15

Hi, Sergey! On Apr 16, svoj(a)mariadb.org wrote: > revision-id: 0520a009c7139c940ab92345054bf0dba7808b11 > parent(s): eb059cd41e5553597f81fd90ad55c07d76d317ce > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-04-16 18:38:26 +0400 > message: > > MDEV-7964 - delete_dynamic() takes 0.12% in OLTP RO ... > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 0e9f668..d36de7f 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -4240,9 +4233,17 @@ thr_lock_type read_lock_type_for_table(THD *thd, > DBUG_RETURN(true); > } > > - if (! (flags & MYSQL_OPEN_SKIP_SCOPED_MDL_LOCK) && > - schema_set.insert(table)) > - DBUG_RETURN(TRUE); > + /* Scoped locks: Take intention exclusive locks on all involved schemas. */ > + if (!(flags & MYSQL_OPEN_SKIP_SCOPED_MDL_LOCK)) > + { > + MDL_request *schema_request= new (thd->mem_root) MDL_request; > + if (schema_request == NULL) > + DBUG_RETURN(TRUE); > + schema_request->init(MDL_key::SCHEMA, table->db, "", > + MDL_INTENTION_EXCLUSIVE, > + MDL_TRANSACTION); > + mdl_requests.push_front(schema_request); > + } As far as I understand, the Hash_set was used here to make only one MDL request per schema. Now if you'll have 10 tables all in one schema, you'll make push 10 MDL requests instead of one. Which may be fine, if MDL subsystem places only one lock in this case (iirc, it does). But what is cheaper - collapse identical schemas into one mdl request here (with Hash_set) or let mdl sort it out later? > mdl_requests.push_front(&table->mdl_request); > } Regards, Sergei

2 4

Re: [Maria-developers] [Commits] 845edb6: MDEV-7067: Server outputs Galera (WSREP) information, even if Galera is disabled
by Sergei Golubchik 09 May '15

09 May '15

Hi, Nirbhay! On Apr 08, Nirbhay Choubey wrote: > revision-id: 845edb6259559bed39eb7365860aa5d588990775 > parent(s): 3674c363a7e77e534c3e0d28659dc614c53fbcbc > committer: Nirbhay Choubey > branch nick: 10.1-wsrep > timestamp: 2015-04-08 16:49:57 -0400 > message: > > MDEV-7067: Server outputs Galera (WSREP) information, even if Galera is disabled > > - mysqld_safe: Skip wsrep position recovery if wsrep_on variable > (mandatory since 10.1) is not specified ok > - Do not load "dummy" wsrep provider if wsrep_on is not ON see below > - Remove "-wsrep" from server version ok I'd also move "wsrep_25.10.r4144" from version_comment to a separate variable, like, wsrep_version. > diff --git a/sql/wsrep_var.cc b/sql/wsrep_var.cc > index ca921b7..1cd0a54 100644 > --- a/sql/wsrep_var.cc > +++ b/sql/wsrep_var.cc > @@ -295,6 +295,8 @@ bool wsrep_provider_options_check(sys_var *self, THD* thd, set_var* var) > > bool wsrep_provider_options_update(sys_var *self, THD* thd, enum_var_type type) > { > + if (!wsrep) return false; > + What about that comment by Alexey Yurchenko? The one where he says that checking for wsrep being not NULL inside wsrep functions is never needed, because wsrep functions should be never even called if wsrep is not enabled. Regards, Sergei

2 1

Re: [Maria-developers] ceae9ba: MDEV-7999 - PROFILING routines take 0.2% when profiling disabled
by Sergei Golubchik 09 May '15

09 May '15

Hi, Sergey! On Apr 15, svoj(a)mariadb.org wrote: > revision-id: ceae9ba7c8872d6558b331ef1ed1ebef42ecdc19 > parent(s): 367f043248112394bd1dc05c72d2844d217a4d55 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-04-15 18:12:23 +0400 > message: > > MDEV-7999 - PROFILING routines take 0.2% when profiling disabled Ok to push. One question: > diff --git a/sql/sql_profile.cc b/sql/sql_profile.cc > index 2eaaea4..9fa4d64 100644 > --- a/sql/sql_profile.cc > +++ b/sql/sql_profile.cc > @@ -720,4 +665,10 @@ int PROFILING::fill_statistics_info(THD *thd_arg, TABLE_LIST *tables, Item *cond > DBUG_RETURN(0); > } > + > + > +void PROFILING::reset() > +{ > + enabled= thd->variables.option_bits & OPTION_PROFILING; > +} Why is this not inlined? Regards, Sergei

2 1

Re: [Maria-developers] eb059cd: MDEV-8001 - mysql_reset_thd_for_next_command() takes 0.04% in OLTP RO
by Sergei Golubchik 09 May '15

09 May '15

Hi, Sergey! On Apr 15, svoj(a)mariadb.org wrote: > revision-id: eb059cd41e5553597f81fd90ad55c07d76d317ce > parent(s): ceae9ba7c8872d6558b331ef1ed1ebef42ecdc19 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-04-15 18:32:34 +0400 > message: > > MDEV-8001 - mysql_reset_thd_for_next_command() takes 0.04% in OLTP RO ok to push > Removed mysql_reset_thd_for_next_command(). Call THD::reset_for_next_command() > directly instead. > > mysql_reset_thd_for_next_command() overhead dropped 0.04% -> out of radar. > THD::reset_for_next_command() overhead didn't increase. Sure, why would it increase? Regards, Sergei

2 1

[Maria-developers] what could be better than %1 OLTP test gains?
by Daniel Black 08 May '15

08 May '15

Appreciate all the effort recently to save <1% on OLTP tests. MDEV-6735 / MDEV-7786 has a 3 line patch in progress that saves a 6700% (0.1 seconds to 11 min 10.44 sec) by changing optimizer decision based on something more meaningful than an absolute number of rows on one side of a join (a specific tuning for some test at some time?). Sure it might not make someone's blog (though it should), but it will make it possible to move one customer of mine beyond 5.5.32 which is near where a regression happened and for all those Magento users who've chosen MariaDB where performance has just broken to unusable when 100 categories was exceeded. -- -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.

1 0

Re: [Maria-developers] 696188f: MDEV-7951 - sql_alloc() takes 0.25% in OLTP RO
by Sergei Golubchik 08 May '15

08 May '15

Hi, Sergey! On Apr 17, svoj(a)mariadb.org wrote: > revision-id: 696188fe2f0756a98914df5b95ba302f7e5c55df > parent(s): 0520a009c7139c940ab92345054bf0dba7808b11 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-04-17 14:30:15 +0400 > message: > > MDEV-7951 - sql_alloc() takes 0.25% in OLTP RO > > sql_alloc() has additional costs compared to direct mem_root allocation: > - function call: it is defined in a separate translation unit and can't be > inlined > - it needs to call pthread_getspecific() to get THD::mem_root > > It is called dozens of times implicitely at least by: > - List<>::push_back() > - List<>::push_front() > - new (for Sql_alloc derived classes) > - sql_memdup() > > Replaced lots of implicit sql_alloc() calls with direct mem_root allocation, > passing through THD pointer whenever it is needed. > > Number of sql_alloc() calls reduced 345 -> 41 per OLTP RO transaction. > pthread_getspecific() overhead dropped 0.76 -> 0.59 > sql_alloc() overhed dropped 0.25 -> 0.06 ok to push Regards, Sergei

1 0

Re: [Maria-developers] d116e02: MDEV-7945 - THD::enter_stage() takes 0.48% in OLTP RO
by Sergei Golubchik 08 May '15

08 May '15

Hi, Sergey! On Apr 14, svoj(a)mariadb.org wrote: > revision-id: d116e028a9545af8bd9eeb03f2ce0e56a9c455f9 > parent(s): 201c985be56f375677df9275d38f18a20cf41029 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-04-14 17:52:08 +0400 > message: > > MDEV-7945 - THD::enter_stage() takes 0.48% in OLTP RO > > THD::enter_stage() optimizations: > - stage backup code moved to THD::backup_stage(), saves one condition > - moved check for "new_stage" out to callers that actually need it > - remnants of enter_stage() moved to sql_class.h so it can be inlined > > THD::enter_stage() overhead dropped 0.48% -> 0.07%. > > PROFILING::status_change() optimizations: > - no need to check "status_arg": it is checked by caller anyway > - no need to check "enabled": !enabled && current is impossible > - remnants of status_change() moved to sql_profile.h so it can be inlined > > PROFILING::status_change() overhead dropped 0.1% -> out of radar. ok to push Regards, Sergei

1 0

Re: [Maria-developers] 334e7c9: MDEV-7895 - HANDLER READ doesn't upgrade metadata lock from S to SR
by Sergei Golubchik 08 May '15

08 May '15

Hi, Sergey! On May 07, Sergey Vojtovich wrote: > revision-id: 334e7c9bccbc6892efbb35f5930f80ccf1d770a8 > parent(s): cbc5157feb9801310e458f7ed10983ad478c881e > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-04-01 14:07:02 +0400 > message: > > MDEV-7895 - HANDLER READ doesn't upgrade metadata lock from S to SR > > Change code for HANDLER READ statements to upgrade S metadata lock to > SR metadata lock for the duration of read. This allows us properly > isolate HANDLER READ from LOCK TABLES WRITE and makes metadata locking > for these statements consistent with locking for other DML. > > HANDLER-related tests had to be adjusted to take into account that > HANDLER READ will wait for and acquire SR lock. code-wise it's ok, but I'm not sure I like the resulting behavior. Here, the very first test: con1: handler t1 open; con2: drop table t1; -- waits con1: handler t1 read; -- deadlock as a user I'd expect that after handler open, I can use the handler. I don't care about locks, but supposedly handler open should take them all or handler read should solve it somehow, why I, a user, should bother? So, indeed, one option would be to acquire SR lock on handler open. Also, note the comment for MDL_SHARED: /* A shared metadata lock. To be used in cases when we are interested in object metadata only and there is no intention to access object data (e.g. for stored routines or during preparing prepared statements). We also mis-use this type of lock for open HANDLERs, since lock acquired by this statement has to be compatible with lock acquired by LOCK TABLES ... WRITE statement, i.e. SNRW (We can't get by by acquiring S lock at HANDLER ... OPEN time and upgrading it to SR lock for HANDLER ... READ as it doesn't solve problem with need to abort DML statements which wait on table level lock while having open HANDLER in the same connection). To avoid deadlock which may occur when SNRW lock is being upgraded to X lock for table on which there is an active S lock which is owned by thread which waits in its turn for table-level lock owned by thread performing upgrade we have to use thr_abort_locks_for_thread() facility in such situation. This problem does not arise for locks on stored routines as we don't use SNRW locks for them. It also does not arise when S locks are used during PREPARE calls as table-level locks are not acquired in this case. */ MDL_SHARED, Which begs more questions: - why open handler lock has to be compatible with SNRW? - why using S and upgrading to SR "didn't solve the problem" and now it suddenly does? Regards, Sergei

2 1

[Maria-developers] Windows line endings in source files (Re: [Maria-discuss] MariaDB 10.0.18 now available)
by Otto Kekäläinen 08 May '15

08 May '15

While preparing the Debian updates I noticed git complaning: The file will have its original line endings in your working directory. warning: CRLF will be replaced by LF in storage/spider/mysql-test/spider/t/have_partition.inc. The file will have its original line endings in your working directory. warning: CRLF will be replaced by LF in storage/spider/mysql-test/spider/t/have_trigger.inc. The file will have its original line endings in your working directory. warning: CRLF will be replaced by LF in storage/spider/mysql-test/spider/t/spider3_fixes.test. The file will have its original line endings in your working directory. warning: CRLF will be replaced by LF in storage/spider/mysql-test/spider/t/spider3_fixes_part.test. Turns out a lot of source files in this release has Windows line endings: mariadb-10.0/mysql-test/r/func_regexp_pcre.result mariadb-10.0/mysql-test/r/loadxml.result mariadb-10.0/mysql-test/r/mysql_binary_mode.result mariadb-10.0/mysql-test/r/perror-win.result mariadb-10.0/mysql-test/std_data/loaddata7.dat mariadb-10.0/mysql-test/std_data/loadxml2.dat mariadb-10.0/packaging/WiX/CPackWixConfig.cmake mariadb-10.0/packaging/WiX/create_msi.cmake.in mariadb-10.0/packaging/WiX/custom_ui.wxs mariadb-10.0/packaging/WiX/extra.wxs.in mariadb-10.0/packaging/WiX/mysql_server.wxs.in mariadb-10.0/pcre/RunTest.bat mariadb-10.0/pcre/testdata/greppatN4 mariadb-10.0/sql/message.rc mariadb-10.0/sql/winservice.h mariadb-10.0/storage/connect/colblk.cpp mariadb-10.0/storage/connect/mysql-test/connect/my.cnf mariadb-10.0/storage/connect/mysql-test/connect/std_data/biblio.json mariadb-10.0/storage/connect/mysql-test/connect/std_data/bookstore.xml mariadb-10.0/storage/connect/mysql-test/connect/std_data/boyswin.txt mariadb-10.0/storage/connect/mysql-test/connect/std_data/emp.txt mariadb-10.0/storage/connect/mysql-test/connect/std_data/expense.json mariadb-10.0/storage/connect/mysql-test/connect/std_data/expenses.txt mariadb-10.0/storage/connect/mysql-test/connect/std_data/mulexp3.json mariadb-10.0/storage/connect/mysql-test/connect/std_data/mulexp4.json mariadb-10.0/storage/connect/mysql-test/connect/std_data/mulexp5.json mariadb-10.0/storage/connect/mysql-test/connect/std_data/sexe.csv mariadb-10.0/storage/connect/mysql-test/connect/std_data/sitmat.csv mariadb-10.0/storage/connect/mysql-test/connect/t/alter.test mariadb-10.0/storage/connect/mysql-test/connect/t/alter_xml.test mariadb-10.0/storage/connect/mysql-test/connect/t/bin.test mariadb-10.0/storage/connect/mysql-test/connect/t/datest.test mariadb-10.0/storage/connect/mysql-test/connect/t/fmt.test mariadb-10.0/storage/connect/mysql-test/connect/t/general.test mariadb-10.0/storage/connect/mysql-test/connect/t/json.test mariadb-10.0/storage/connect/mysql-test/connect/t/json_udf.test mariadb-10.0/storage/connect/mysql-test/connect/t/mrr.test mariadb-10.0/storage/connect/mysql-test/connect/t/mul.test mariadb-10.0/storage/connect/mysql-test/connect/t/myconn.inc mariadb-10.0/storage/connect/mysql-test/connect/t/myconn_cleanup.inc mariadb-10.0/storage/connect/mysql-test/connect/t/mysql.test mariadb-10.0/storage/connect/mysql-test/connect/t/mysql_discovery.test mariadb-10.0/storage/connect/mysql-test/connect/t/mysql_exec.test mariadb-10.0/storage/connect/mysql-test/connect/t/mysql_grant.test mariadb-10.0/storage/connect/mysql-test/connect/t/mysql_new.test mariadb-10.0/storage/connect/mysql-test/connect/t/null.test mariadb-10.0/storage/connect/mysql-test/connect/t/occur.test mariadb-10.0/storage/connect/mysql-test/connect/t/odbc_sqlite3.test mariadb-10.0/storage/connect/mysql-test/connect/t/part_file.test mariadb-10.0/storage/connect/mysql-test/connect/t/part_table.test mariadb-10.0/storage/connect/mysql-test/connect/t/pivot.test mariadb-10.0/storage/connect/mysql-test/connect/t/tbl.test mariadb-10.0/storage/connect/mysql-test/connect/t/unsigned.test mariadb-10.0/storage/connect/mysql-test/connect/t/upd.test mariadb-10.0/storage/connect/mysql-test/connect/t/updelx.test mariadb-10.0/storage/connect/mysql-test/connect/t/updelx2.test mariadb-10.0/storage/connect/mysql-test/connect/t/xcol.test mariadb-10.0/storage/connect/mysql-test/connect/t/xml_mdev5261.test mariadb-10.0/storage/connect/rcmsg.c mariadb-10.0/storage/connect/tabsys.h mariadb-10.0/storage/federatedx/README.windows mariadb-10.0/storage/spider/mysql-test/spider/bg/include/ha_deinit_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/ha_deinit_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/ha_deinit_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/ha_init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/ha_init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/ha_init_child2_3.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/ha_init_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/ha_init_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/ha_init_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/hs_init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/hs_init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/init_child2_3.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/include/init_slave1_1.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/t/direct_aggregate.test mariadb-10.0/storage/spider/mysql-test/spider/bg/t/direct_aggregate_part.test mariadb-10.0/storage/spider/mysql-test/spider/bg/t/have_partition.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/t/have_trigger.inc mariadb-10.0/storage/spider/mysql-test/spider/bg/t/spider3_fixes.test mariadb-10.0/storage/spider/mysql-test/spider/bg/t/spider3_fixes_part.test mariadb-10.0/storage/spider/mysql-test/spider/handler/include/ha_deinit_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/ha_deinit_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/ha_deinit_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/ha_init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/ha_init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/ha_init_child2_3.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/ha_init_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/ha_init_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/ha_init_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/hs_init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/hs_init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/init_child2_3.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/include/init_slave1_1.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/t/direct_aggregate.test mariadb-10.0/storage/spider/mysql-test/spider/handler/t/direct_aggregate_part.test mariadb-10.0/storage/spider/mysql-test/spider/handler/t/have_partition.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/t/have_trigger.inc mariadb-10.0/storage/spider/mysql-test/spider/handler/t/spider3_fixes.test mariadb-10.0/storage/spider/mysql-test/spider/handler/t/spider3_fixes_part.test mariadb-10.0/storage/spider/mysql-test/spider/include/deinit_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/include/deinit_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/include/deinit_child2_3.inc mariadb-10.0/storage/spider/mysql-test/spider/include/deinit_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/include/deinit_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/include/deinit_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/include/deinit_engine.inc mariadb-10.0/storage/spider/mysql-test/spider/include/ha_deinit_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/include/ha_deinit_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/include/ha_deinit_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/include/ha_init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/include/ha_init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/include/ha_init_child2_3.inc mariadb-10.0/storage/spider/mysql-test/spider/include/ha_init_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/include/ha_init_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/include/ha_init_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/include/hs_init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/include/hs_init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/include/init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/include/init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/include/init_child2_3.inc mariadb-10.0/storage/spider/mysql-test/spider/include/init_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/include/init_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/include/init_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/include/init_slave1_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/deinit_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/deinit_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/deinit_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/ha_deinit_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/ha_deinit_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/ha_deinit_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/ha_init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/ha_init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/ha_init_child2_3.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/ha_init_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/ha_init_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/ha_init_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/hs_init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/hs_init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/init_child2_3.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/init_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/init_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/init_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/include/init_slave1_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/t/direct_aggregate.test mariadb-10.0/storage/spider/mysql-test/spider/oracle/t/direct_aggregate_part.test mariadb-10.0/storage/spider/mysql-test/spider/oracle/t/have_partition.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/t/have_trigger.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle/t/spider3_fixes.test mariadb-10.0/storage/spider/mysql-test/spider/oracle/t/spider3_fixes_part.test mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/deinit_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/deinit_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/deinit_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/ha_deinit_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/ha_deinit_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/ha_deinit_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/ha_init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/ha_init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/ha_init_child2_3.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/ha_init_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/ha_init_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/ha_init_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/hs_init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/hs_init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/init_child2_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/init_child2_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/init_child2_3.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/init_child3_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/init_child3_2.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/init_child3_3.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/include/init_slave1_1.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/t/direct_aggregate.test mariadb-10.0/storage/spider/mysql-test/spider/oracle2/t/direct_aggregate_part.test mariadb-10.0/storage/spider/mysql-test/spider/oracle2/t/have_partition.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/t/have_trigger.inc mariadb-10.0/storage/spider/mysql-test/spider/oracle2/t/spider3_fixes.test mariadb-10.0/storage/spider/mysql-test/spider/oracle2/t/spider3_fixes_part.test mariadb-10.0/storage/spider/mysql-test/spider/t/direct_aggregate.test mariadb-10.0/storage/spider/mysql-test/spider/t/direct_aggregate_part.test mariadb-10.0/storage/spider/mysql-test/spider/t/have_partition.inc mariadb-10.0/storage/spider/mysql-test/spider/t/have_trigger.inc mariadb-10.0/storage/spider/mysql-test/spider/t/spider3_fixes.test mariadb-10.0/storage/spider/mysql-test/spider/t/spider3_fixes_part.test mariadb-10.0/win/packaging/COPYING.rtf mariadb-10.0/win/packaging/CPackWixConfig.cmake mariadb-10.0/win/packaging/ca/CustomAction.cpp mariadb-10.0/win/packaging/custom_ui.wxs mariadb-10.0/win/packaging/extra.wxs.in mariadb-10.0/win/packaging/heidisql.cmake mariadb-10.0/win/packaging/heidisql.wxi.in mariadb-10.0/win/packaging/heidisql_feature.wxi.in mariadb-10.0/win/packaging/mysql_server.wxs.in mariadb-10.0/win/upgrade_wizard/stdafx.h mariadb-10.0/win/upgrade_wizard/targetver.h mariadb-10.0/win/upgrade_wizard/upgrade.cpp mariadb-10.0/win/upgrade_wizard/upgrade.h mariadb-10.0/win/upgrade_wizard/upgrade.rc mariadb-10.0/win/upgrade_wizard/upgradeDlg.h mariadb-10.0/win/upgrade_wizard/upgrade_wizard.exe.manifest This is the first time in two years I've found CRLF line endings in MariaDB source files. I hope I won't see them again..

3 2

[Maria-developers] Proposal accepted for GSoc'15
by sriram patil 08 May '15

08 May '15

Hi All, I am Sriram from IIIT Hyderabad, India. I am a Masters' student here. I have worked with MariaDB in GSoC'14 with Alexander Barkov. I had great time hacking on MariaDB last year, so, here I am again. :) This year I will be working on MaxScale filters with Markus Makela. The project title is "MaxScale filter to capture incoming operations for consumption in external sources". (MXS-2) I am excited to get started. Hoping to have a great summer this year as well. :) Thanks, Sriram Patil

2 2

Re: [Maria-developers] [Commits] 313a970: MDEV-7922 - ERROR 1939 (HY000): Engine PERFORMANCE_SCHEMA failed to discover
by Sergei Golubchik 07 May '15

07 May '15

Hi, Sergey! On Apr 08, svoj(a)mariadb.org wrote: > revision-id: 313a970cbe9ac523b0584d293e40bff66d4fbafc > parent(s): 7d9e94e2e6c32b906d1d8469e58bc04b8da1c121 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-04-08 10:55:51 +0400 > message: > > MDEV-7922 - ERROR 1939 (HY000): Engine PERFORMANCE_SCHEMA failed to discover > table > > Performance schema discovery fails if connection has no active database set. > > This happened due to restriction in SQL parser: table name with no database name > is ambiguous in such case. > > Fixed by temporary substitution of default database with being discovered table > database. ok to push Regards, Sergei

1 0

[Maria-developers] MDEV-7807 information_schema.processlist truncates queries with binary strings
by Alexander Barkov 07 May '15

07 May '15

Hi Sergei, Please review the patch for: MDEV-7807 information_schema.processlist truncates queries with binary strings The patch is for 10.1, as agreed with the reporter. Previously in 10.1 we fixed binary->string conversion behaviour not to truncate on bad bytes, and replace bad bytes to question marks instead. That fixed half of the problem reported. This patch adds a new new column "PROCESSLIST.INFO_BINARY blob". Thanks.

2 1

Re: [Maria-developers] [Commits] 93e746a: MDEV-7793 - Race condition between XA COMMIT/ROLLBACK and disconnect
by Sergei Golubchik 07 May '15

07 May '15

Hi, Sergey! On Mar 17, svoj(a)mariadb.org wrote: > revision-id: 93e746afb2d1a23cb933291ad736a20858b5ac3e > parent(s): ccc7297fe94af1129c717f91d31fa075d54a0371 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-03-17 19:49:04 +0400 > message: > > MDEV-7793 - Race condition between XA COMMIT/ROLLBACK and disconnect > > XA COMMIT/ROLLBACK of XA transaction owned by different thread may access > freed memory if that thread disconnects at the same time. > > Also concurrent XA COMMIT/ROLLBACK of recovered XA transaction were not > serialized properly. > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 3071ba6..f18231d 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5355,7 +5378,7 @@ bool xid_cache_insert(THD *thd, XID_STATE *xid_state) > switch (res) > { > case 0: > - xid_state->xid_cache_element->mark_initialized(); > + xid_state->xid_cache_element->set(XID_cache_element::ACQUIRED); Why not mark_acquired() here? Can you be sure that no other thread has already acquired this xid_cache_element after you've inserted it? > break; > case 1: > my_error(ER_XAER_DUPID, MYF(0)); Regards, Sergei

2 2

Re: [Maria-developers] [Commits] 54e9df4: MDEV-7985: MySQL Users Break when Migrating to MariaDB, part 2
by Sergei Golubchik 05 May '15

05 May '15

Hi, Vicentiu! Two comments to the patch - see below. One general comment - you've done quite a bit of refactoring and this refactoring is *not* in 10.1 style. How should it be merged into 10.1? Should your changes be ignored (and not merged)? Should they be merged as is? Aren't they mostly unnecessary in 10.1? On May 05, vicentiu(a)mariadb.org wrote: > revision-id: 54e9df42d1f5837ae0df51e511ba48d3682be20d > parent(s): ae18a28500974351cf42fa3cac67c83e0647d510 > committer: Vicențiu Ciorbaru > branch nick: server > timestamp: 2015-05-05 21:19:53 +0300 > message: > > MDEV-7985: MySQL Users Break when Migrating to MariaDB, part 2 > > Gave priority to password field when using a native authentication > plugin. > > Also, prevented a user from setting an invalid auth_string, when using > native authentication. > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -862,6 +862,17 @@ static char *fix_plugin_ptr(char *name) > return name; > } > > +static bool plugin_is_native(const char* plugin) > +{ > + if (!plugin) > + return false; > + if ((my_strcasecmp(system_charset_info, plugin, > + native_password_plugin_name.str) == 0) || > + (my_strcasecmp(system_charset_info, plugin, > + old_password_plugin_name.str) == 0)) > + return true; > + return false; > +} I tried to compare plugin names almost always by pointers, not by comparing strings. That is, the string (as coming from the table or from the user) is strcmp'ed *once* and then replaced with a pointer to a static plugin name string. So I'd prefer it if you'd remove plugin_is_native() and instead used fix_user_plugin_ptr(). And then you remove fix_user_plugin_ptr() from acl_update_user() and acl_insert_user(). > /** > Fix ACL::plugin pointer to point to a hard-coded string, if appropriate > > @@ -1973,6 +2020,10 @@ static void acl_update_user(const char *user, const char *host, > acl_user->auth_string.length= auth->length; > if (fix_user_plugin_ptr(acl_user)) > acl_user->plugin.str= strmake_root(&acl_memroot, plugin->str, plugin->length); > + else > + set_user_salt(acl_user, acl_user->auth_string.str, > + acl_user->auth_string.length); what if auth_string is empty, how does the password take precedence in this case? > + > } > else > if (password[0]) Regards, Sergei

1 0

[Maria-developers] Please review: MDEV-7778 impossible create copy of table, if table contain default value for timestamp field
by Alexander Barkov 04 May '15

04 May '15

Hi Sergei, Please review a patch for MDEV-7778. As agreed on IRC, this is a merge from the latest MySQL, which has already fixed the problem. Greetings.

2 1

[Maria-developers] connect on IBM s390
by Alexander Barkov 04 May '15

04 May '15

Hi Olivier, MariaDB fails with some Connect tests on IBM s390. Please have a look into Debian build bot results: https://buildd.debian.org/status/fetch.php?pkg=mariadb-10.0&arch=s390x&ver=… I currently have a temporary access to an IMB s390 machine. It would be nice to fix the failures. Please find the mtr log attached. This is what I got on this test machine. The failures are very similar to what Debian build bot has. If you don't mind, I'd like to start fixing failures with "bin" and "dbf". 1. dbf.bin fails because Connect writes the DBF file header using the current platform endiannes, while from my understanding it should always write the header using little endianness. This should be fixed in the relevant DBF code in Connect. Can you please do this? 2. connect.bin fails because Connect treats columns in the current machine byte order, while Testbal.dat was created on a little endian machine. I can see a few options to fix this test failure: a. Disable on big endian machines. Bad idea. b. Make it possible to specify endiannes of a data file (or even per column), so we can tell in the tests that Testbal.dat (or its relevant columns) are in little endian format. Perhaps it's already possible in the latest Connect versions. Is it? If it's not, how difficult would be to implement this? c. Have two pre-created files Testbal_le.dat and Testbal_be.dat and load the one that suites the current machine endiannes. I'm thinking of going "c" now, with a hope to have both "b" and "c" in the future. If you agree, we'll need a big-endian version of Testbal.dat. 3. It seems that some other tests fail for the same reasons with #2. Having a big-endian version of the test data files, or having the endian flag in CREATE TABLE, would fix some of them. I suggest we fix "bin" first, and make sure it works fine an all platforms. Then do the same thing for the other tests. Thanks.

1 0

Re: [Maria-developers] [Commits] 4bff8b0: MDEV-7004 - Merge scalability fixes from 10.0-power
by jocelyn fournier 02 May '15

02 May '15

Hi Sergey, Just to be sure, currently MariaDB doesn't implement any Hardware Transactional Memory Built-in Functions ? If so, is there any plan to evaluate the gain they could offer to MariaDB ? Thanks and regards, Jocelyn Fournier Le 04/12/2014 14:45, svoj(a)mariadb.org a écrit : > revision-id: 4bff8b0b03ba82cd0ae6817b54de1780e89047ca > parent(s): 3526fe1b6000b2558290007e5cc3a1018850787c > committer: Sergey Vojtovich > branch nick: 10.1 > timestamp: 2014-12-04 17:44:46 +0400 > message: > > MDEV-7004 - Merge scalability fixes from 10.0-power > > Preallocate dynamic array and bitmap on mem_root to avoid expensive malloc. > This reduces number of allocations from 39 to 31 per OLTP RO transaction. > > --- > sql/opt_range.cc | 12 ++++++------ > 1 file changed, 6 insertions(+), 6 deletions(-) > > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index 1a9ae3d..813508c 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -1838,8 +1838,6 @@ void SQL_SELECT::cleanup() > index= key_nr; > head= table; > key_part_info= head->key_info[index].key_part; > - my_init_dynamic_array(&ranges, sizeof(QUICK_RANGE*), 16, 16, > - MYF(MY_THREAD_SPECIFIC)); > > /* 'thd' is not accessible in QUICK_RANGE_SELECT::reset(). */ > mrr_buf_size= thd->variables.mrr_buff_size; > @@ -1857,9 +1855,12 @@ void SQL_SELECT::cleanup() > file= head->file; > record= head->record[0]; > > - /* Allocate a bitmap for used columns (Q: why not on MEM_ROOT?) */ > - if (!(bitmap= (my_bitmap_map*) my_malloc(head->s->column_bitmap_size, > - MYF(MY_WME | MY_THREAD_SPECIFIC)))) > + my_init_dynamic_array2(&ranges, sizeof(QUICK_RANGE*), > + thd->alloc(sizeof(QUICK_RANGE*) * 16), 16, 16, > + MYF(MY_THREAD_SPECIFIC)); > + > + /* Allocate a bitmap for used columns */ > + if (!(bitmap= (my_bitmap_map*) thd->alloc(head->s->column_bitmap_size))) > { > column_bitmap.bitmap= 0; > *create_error= 1; > @@ -1923,7 +1924,6 @@ void QUICK_RANGE_SELECT::range_end() > } > delete_dynamic(&ranges); /* ranges are allocated in alloc */ > free_root(&alloc,MYF(0)); > - my_free(column_bitmap.bitmap); > } > my_free(mrr_buf_desc); > DBUG_VOID_RETURN; > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 1

[Maria-developers] [GSoC 2015] Accepted Student- Self Introduction
by Delveri chick 30 Apr '15

30 Apr '15

Hello everyone ! I am Delveri Chick, a junior software engineering student from Cameroon. I have been accepted into GSoC 2015 to hack on project titled : Improved Temporary Tables. Here <http://www.google-melange.com/gsoc/proposal/review/student/google/gsoc2015/…> is my GSoC proposal. Sergei Golubchik will be mentoring me for this project. I am really excited to spend my summer hacking on MariaDB, and hoping to learn a lot of things from the developer community. Thanks. Delveri

2 1

Re: [Maria-developers] [Commits] 13ecde8: Fix compiler warnings.
by Vicențiu Ciorbaru 30 Apr '15

30 Apr '15

Hi Jan! I have one concern with this patch. See below: On Thu, 30 Apr 2015 at 11:45 Jan Lindström <jan.lindstrom(a)mariadb.com> wrote: > revision-id: 13ecde851f08a7952f774707b19734f351bb6598 > parent(s): 5027e0b035f96e2583eff2d1ac71c7c44a093ac8 > committer: Jan Lindström > branch nick: 10.0-git > timestamp: 2015-04-30 11:44:34 +0300 > message: > > Fix compiler warnings. > > --- > storage/innobase/sync/sync0sync.cc | 27 +++++++++++++-------------- > storage/xtradb/sync/sync0sync.cc | 27 +++++++++++++-------------- > 2 files changed, 26 insertions(+), 28 deletions(-) > > diff --git a/storage/innobase/sync/sync0sync.cc > b/storage/innobase/sync/sync0sync.cc > index 121f8e9..78d86f3 100644 > --- a/storage/innobase/sync/sync0sync.cc > +++ b/storage/innobase/sync/sync0sync.cc > @@ -862,10 +862,10 @@ sync_thread_levels_g( > { > ulint i; > > - for (i = 0; i < arr->n_elems; i++) { > + for (i = 0; i < arr->size(); i++) { > const sync_level_t* slot; > > I think this here has to be arr->at(i), or &(*arr)[i]. Doing &arr[i] will actually get the i'th std::vector<> which does not exist, since arr is a pointer to only one std::vector<>. > - slot = &arr->elems[i]; > + slot = (const sync_level_t*)&arr[i]; > > if (slot->latch != NULL && slot->level <= limit) { > if (warn) { > @@ -897,10 +897,10 @@ sync_thread_levels_contain( > { > ulint i; > > - for (i = 0; i < arr->n_elems; i++) { > + for (i = 0; i < arr->size(); i++) { > const sync_level_t* slot; > > Same here. > - slot = &arr->elems[i]; + slot = (const sync_level_t*)&arr[i]; > > if (slot->latch != NULL && slot->level == level) { > > @@ -944,10 +944,10 @@ sync_thread_levels_contains( > > arr = thread_slot->levels; > > - for (i = 0; i < arr->n_elems; i++) { > + for (i = 0; i < arr->size(); i++) { > sync_level_t* slot; > Same here. > - slot = &arr->elems[i]; > + slot = (sync_level_t*)&arr[i]; > > if (slot->latch != NULL && slot->level == level) { > > @@ -993,10 +993,10 @@ sync_thread_levels_nonempty_gen( > > arr = thread_slot->levels; > > - for (i = 0; i < arr->n_elems; ++i) { > + for (i = 0; i < arr->size(); ++i) { > const sync_level_t* slot; > Same here. > - slot = &arr->elems[i]; > + slot = (const sync_level_t*)&arr[i]; > > if (slot->latch != NULL > && (!dict_mutex_allowed > @@ -1053,10 +1053,10 @@ sync_thread_levels_nonempty_trx( > > arr = thread_slot->levels; > > - for (i = 0; i < arr->n_elems; ++i) { > + for (i = 0; i < arr->size(); ++i) { > const sync_level_t* slot; > Same here. > - slot = &arr->elems[i]; > + slot = (const sync_level_t*)&arr[i]; > > if (slot->latch != NULL > && (!has_search_latch > @@ -1331,7 +1331,7 @@ sync_thread_add_level( > > sync_level.latch = latch; > sync_level.level = level; > - array->elems.push_back(sync_level); > + array->push_back(sync_level); > > mutex_exit(&sync_thread_mutex); > } > @@ -1377,14 +1377,14 @@ sync_thread_reset_level( > > array = thread_slot->levels; > - for (std::vector<sync_level_t>::iterator it = array->elems.begin(); > it != array->elems.end(); ++it) { > + for (std::vector<sync_level_t>::iterator it = array->begin(); it > != array->end(); ++it) { > sync_level_t level = *it; > > if (level.latch != latch) { > continue; > } > > - array->elems.erase(it); > + array->erase(it); > mutex_exit(&sync_thread_mutex); > return(TRUE); > } > @@ -1471,7 +1471,6 @@ sync_thread_level_arrays_free(void) > > /* If this slot was allocated then free the slot memory > too. */ > if (slot->levels != NULL) { > - > slot->levels->elems.erase(slot->levels->elems.begin(),slot->levels->elems.end()); > delete slot->levels; > } > } > diff --git a/storage/xtradb/sync/sync0sync.cc > b/storage/xtradb/sync/sync0sync.cc > index d0e3b71..9dcdd9c 100644 > --- a/storage/xtradb/sync/sync0sync.cc > +++ b/storage/xtradb/sync/sync0sync.cc > @@ -970,10 +970,10 @@ sync_thread_levels_g( > { > ulint i; > > - for (i = 0; i < arr->n_elems; i++) { > + for (i = 0; i < arr->size(); i++) { > const sync_level_t* slot; > > Same here. > - slot = &arr->elems[i]; > + slot = (const sync_level_t*)&arr[i]; > > if (slot->latch != NULL && slot->level <= limit) { > if (warn) { > @@ -1005,10 +1005,10 @@ sync_thread_levels_contain( > { > ulint i; > > - for (i = 0; i < arr->n_elems; i++) { > + for (i = 0; i < arr->size(); i++) { > const sync_level_t* slot; > Same here. > - slot = &arr->elems[i]; > + slot = (const sync_level_t*)&arr[i]; > > if (slot->latch != NULL && slot->level == level) { > > @@ -1052,10 +1052,10 @@ sync_thread_levels_contains( > > arr = thread_slot->levels; > > - for (i = 0; i < arr->n_elems; i++) { > + for (i = 0; i < arr->size(); i++) { > sync_level_t* slot; > > Same here. > - slot = &arr->elems[i]; > + slot = (sync_level_t*)&arr[i]; > > if (slot->latch != NULL && slot->level == level) { > > @@ -1101,10 +1101,10 @@ sync_thread_levels_nonempty_gen( > > arr = thread_slot->levels; > > - for (i = 0; i < arr->n_elems; ++i) { > + for (i = 0; i < arr->size(); ++i) { > const sync_level_t* slot; > > Same here. > - slot = &arr->elems[i]; > + slot = (const sync_level_t*)&arr[i]; > > if (slot->latch != NULL > && (!dict_mutex_allowed > @@ -1161,10 +1161,10 @@ sync_thread_levels_nonempty_trx( > > arr = thread_slot->levels; > > - for (i = 0; i < arr->n_elems; ++i) { > + for (i = 0; i < arr->size(); ++i) { > const sync_level_t* slot; > > Same here. > - slot = &arr->elems[i]; > + slot = (const sync_level_t*)&arr[i]; > > if (slot->latch != NULL > && (!has_search_latch > @@ -1456,7 +1456,7 @@ sync_thread_add_level( > > sync_level.latch = latch; > sync_level.level = level; > - array->elems.push_back(sync_level); > + array->push_back(sync_level); > > mutex_exit(&sync_thread_mutex); > } > @@ -1502,14 +1502,14 @@ sync_thread_reset_level( > > array = thread_slot->levels; > > - for (std::vector<sync_level_t>::iterator it = > array->elems.begin(); it != array->elems.end(); ++it) { > + for (std::vector<sync_level_t>::iterator it = array->begin(); it > != array->end(); ++it) { > sync_level_t level = *it; > > if (level.latch != latch) { > continue; > } > > - array->elems.erase(it); > + array->erase(it); > mutex_exit(&sync_thread_mutex); > return(TRUE); > } > @@ -1597,7 +1597,6 @@ sync_thread_level_arrays_free(void) > > /* If this slot was allocated then free the slot memory > too. */ > if (slot->levels != NULL) { > - > slot->levels->elems.erase(slot->levels->elems.begin(),slot->levels->elems.end()); > delete slot->levels; > } > } >

1 0

Re: [Maria-developers] [Commits] 8a18bbc: MDEV-8079: Crash when running MariaDB Debug with InnoDB on Windows
by Vicențiu Ciorbaru 30 Apr '15

30 Apr '15

Here is a patch that fixes all compilation related issues. Unfortunately it's based on 10.1. It might need a bit of tweaking to merge directly into 10.0

1 0

[Maria-developers] GSoC 2015 student
by Blessing Atie 29 Apr '15

29 Apr '15

Hello everyone, My surname is Blessing and I am a student at University of Buea. I am very happy to be selected by MariaDb to part take in GSoC 2015. I will be working on Improving table Functions mentored Sergei Golubchik and I look forward to know him soon on #maria as well as the other students. I equally look forward on using this as an opportunity to learn more about open source and MariaDB community. Can't wait to get hands dirty on MariaDB source. Thanks. Blessing

2 1

Re: [Maria-developers] MDEV-7774: Crash when dropping user within rebuild_role_grants
by Sergei Golubchik 29 Apr '15

29 Apr '15

Hi, Vicentiu! On Mar 13, vicentiu(a)mariadb.org wrote: > revision-id: 22884d35e39ba078dddf5c15b42dd7e2ebb39c07 > parent(s): 184f718fef0101a7559364cb97e22ee568e64c12 > committer: Vicențiu Ciorbaru > branch nick: server > timestamp: 2015-03-13 20:12:22 +0200 > message: > > MDEV-7774: Crash when dropping user within rebuild_role_grants > > The issue comes from not taking all possibilities to match an entry > within the roles_mapping HASH, when updating the data structure. ok to push Regards, Sergei

1 0

[Maria-developers] Likely a dead code in build_equal_items_for_cond()
by Alexander Barkov 29 Apr '15

29 Apr '15

Hi Sanja and Igor, Can you please help to understand this pieces of code in sql_select.cc in 10.0: > else if (cond->type() == Item::FUNC_ITEM || > cond->real_item()->type() == Item::FIELD_ITEM) > { It seems that the "cond->real_item()->type() == Item::FIELD_ITEM" part of this condition is a dead code. I added a DBUG_ASSERT in the very beginning of this block, like this: > else if (cond->type() == Item::FUNC_ITEM || > cond->real_item()->type() == Item::FIELD_ITEM) > { > DBUG_ASSERT(cond->real_item()->type() != Item::FIELD_ITEM); and run all tests. Nothing crashed. I suspect that the following scenario could happen: 1. In April 2011 Igor fixed bugs #717577, #724942 with this commit: > commit 8d9dd21d85e257051b45b2f779dcd9bf696bf9e1 > Author: Igor Babaev <igor(a)askmonty.org> Wed Apr 27 06:58:41 2011 > Committer: Igor Babaev <igor(a)askmonty.org> Wed Apr 27 06:58:41 2011 > > Fixed LP bugs #717577, #724942. and this line was probably to handle conditions like "WHERE field". 2. In July 2011 Sanja introduced a new function normalize_cond(), with this commit: > commit 20a2e1d0ac46e375ee7246576b541624c6f8f028 > Author: unknown <sanja(a)askmonty.org> Thu Jul 21 11:20:55 2011 > unknown <sanja(a)askmonty.org> Thu Jul 21 11:20:55 2011 so all conditions "WHERE/HAVING field" are now replaced to "WHERE/HAVING field<>0". So now, by the time of a build_equal_items_for_cond() call all conditions are already replaced and no FIELD_ITEM are actually possible. Can you please confirm this? If my guess is correct then this part can now be safely removed: > || > cond->real_item()->type() == Item::FIELD_ITEM Thanks.

1 0

[Maria-developers] MDEV-7802 adding status variables - backport to 10.0 request
by Daniel Black 29 Apr '15

29 Apr '15

Hi, I'm requesting a backport of MDEV-7802 https://github.com/MariaDB/server/pull/30/files to 10.0 It added status variables which where important to determining what operational values to set binlog_commit_wait_usec / binlog_commit_wait_count variables to. With parallel replication efficiency and general group commit IO reduction dependent on the tradeoff of these settings with the commit delay, the status variables binlog_group_commit_trigger_count, binlog_group_commit_trigger_timeout and binlog_group_commit_trigger_lock_wait that this patch adds are operationally important. The changes added a few variables and didn't affect the flow of the program at all. In the binary log execution path, no locks where added and only a few increments of static variables where added. There was an additional lock usage of LOCK_prepare_ordered in the show global status for the period to copy the added variables into the global status structures. I'm quite aware of the dangers of show global status holding locks wrongly (MDEV-7126) I don't suspect this does. On the balance of risk/benefit can I request that this feature be merged to 10.0. -- -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.

4 4

[Maria-developers] GSoC 2015 Accepted Student
by arlene seuleu 28 Apr '15

28 Apr '15

Hi everyone, I am Arlene Flavie a computer engineering student at the university of Buea and I just got confirmation mail that I was selected to participate in the Google summer of code 2015 under MariaDB. I wish to use this opportunity to thank the mentors for giving me this opportunity to contribute to opensource. I wish to learn much in this experience as I will be Mentored by Sanda Byelkin on the project Import and Export data formats into and from dynamic columns. I look forward to do a similar introduction of myself on #maria. Thanks Arlene

1 0

[Maria-developers] MDEV-7950 Item_func::type() takes 0.26% in OLTP RO
by Alexander Barkov 28 Apr '15

28 Apr '15

Hi Sergey, Can you please review the "step 2" part of MDEV-7950? This is rather a mechanical cut-and-paste change. It does the following things: 1. Removes the function build_equal_items_for_cond() and introduces a new method Item::build_equal_items() instead, with specific implementations in the following Items: Item (the default implementation) Item_ident_or_func_or_sum Item_cond Item_cond_and 2. Adds a new abstract class Item_ident_or_func_or_sum, a common parent for Item_ident and Item_func_or_sum, as they have exactly the same build_equal_items(). 3. Renames Item_cond_and::cond_equal to Item_cond_and::m_cond_equal, to avoid confusion between the member and local variables named "cond_equal". The main purpose of this patch is to prepare to the next step in MDEV-7950. But it already improves performance per se: it gets rid of some virtual calls: - The former call for "((Item_cond*) cond)->functype()" is not needed any more, because Item_cond and Item_cond_and now have different implementations of the method build_equal_items(). - This condition: "else if (cond->type() == Item::FUNC_ITEM || cond->real_item()->type() == Item::FIELD_ITEM)" (which used to cover Item_func, Item_ident and Item_ref) consisted of three virtual calls. It's now replaced by a single virtual call for Item_ident_or_func_or_sum::build_equal_items(), which covers all three item types. Thanks.

2 2

[Maria-developers] Mariadb java client issue.
by 김경남 28 Apr '15

28 Apr '15

Hi, I hava a issue of using mariadb java client. I can't get column alias from query result set. My query as below. select col1 as column_alias1 from table1; reuslt: --------------------- col1 ------------------------ xxx ... In case of mysql connectorJ, there was a same issue. so I added a option like "useOldAliasMetadataBehavior=true" to connection string and that issue solved. but I cant't solve this issue in case of using mariadb java client. Is there a correct way? Regards, Kyungnam kim 김경남 부책임연구원 솔루션사업본부 정보기술 연구원 신기술개발팀 02-2099-6177 / 010-2208-3006 _____ 상기 메시지와 첨부화일 내에는 비밀정보가 포함되어 있을 수 있으며, 지정된 수신자에 한하여 조회 및 사용될 수 있습니다. 만약 송신자의 실수로 인하여 상기 메시지를 수신하였다면, 송신자에게 메시지를 반송해 주시고, 원본 메시지와 모든 사본을 폐기해 주시기 바랍니다. 상기 메시지의 전체 또는 일부에 대해 무단 열람, 사용, 공개, 배포하는 것은 금지되어 있습니다.(주)LG CNS. This message and its attachments may contain confidential information, and they are intended to be viewed or used by only the individuals specified in the message. If you have received this message in an error from the sender, please contact the sender immediately to notify the error and delete all of the message and its copies. It is prohibited to view, use, make public and/or distribute part or whole of this message without written permission.

1 0

[Maria-developers] MariaDB / Galera BUG
by Andrew W Elble 27 Apr '15

27 Apr '15

Environment: MariaDB-10.0.16 Galera-3-25.3.9 This statement does nasty things to a MariaDB+Galera cluster: CREATE TABLE u91rw_zoo_version (`version` varchar(255) NOT NULL) ENGINE=innoDB DEFAULT CHARSET=utf8 SELECT '3.3.3' as version; Statement fails to replicate to the other nodes because the target table doesn't exist. Cluster then disassembles itself. This patch makes the problem disappear. Please evaluate? (note: patch based against github.com/MariaDB/server.git) ============= diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc index be57be9b223a..d0521ed896b8 100644 --- a/sql/sql_parse.cc +++ b/sql/sql_parse.cc @@ -3325,6 +3325,11 @@ mysql_execute_command(THD *thd) /* Store reference to table in case of LOCK TABLES */ create_info.table= create_table->table; + if (WSREP(thd) && (!thd->is_current_stmt_binlog_format_row() || + !create_info.tmp_table())) + { + WSREP_TO_ISOLATION_BEGIN(create_table->db, create_table->table_name, NULL) + } /* select_create is currently not re-execution friendly and needs to be created for every execution of a PS/SP. ============= We have a full environment setup in which to test. debug/trace available on request. Will open bug if required for your workflow. Thanks, Andy -- Andrew W. Elble aweits(a)discipline.rit.edu Infrastructure Engineer, Communications Technical Lead Rochester Institute of Technology PGP: BFAD 8461 4CCF DC95 DA2C B0EB 965B 082E 863E C912

2 6