November 2014 - developers - lists.mariadb.org

[Maria-developers] add a rotate event at begin of relay log make greate help
by nanyi607rao 28 Nov '14

28 Nov '14

Hi kristian, currently, a relaylog is immediately purged after it be completely replayed (relay_log_purge=1). and if IO thread crashes, we should delete all relay logs and fetch master binlog again from exec_master_log_pos/file (relay_log_purge=ON). I think that make a greap waste of disk IO especially to cloud disk. The critical problem is that there is no method to know the original binlog filename on master of a event in relaylog. Actually we can know a relaylog event's original binlog offset on master, but we can't know the original binlog filename. so why not forcely add a rotate event or any other new type event which contained the original master binlog filename at begin of a relay log. as I know two different master binlogs' events wouldn't contained in one relay log, so the original binlog filename of a relaylog's events is only one. If we want to know a relaylog event's original binlog filename/position, we can get original filename from the begin of relaylog, and get original binlog offset from event's "end log pos". oppositely, if we know a event's binlog filename/end positon on master, wo also can quickly find it in slave's relay log. That can make many benefits, firstly no need to delete all relay logs and fetch master binlogs again when IO thread crashed, because we can get exact last read_master_log_filename/position from last relay log. secondly, there is no need to use GTID in 1 vs n replication failover Scenario (Gtid must set log_slave_updates=ON in mysql 5.6, which increase disk IO load), if master crashes, other slaves can get lost events for the newest slave's relay log, as long as relay log don't be purged, then promotes the newest slave to master. Thanks. 2014-11-26 nanyi607rao

3 3

[Maria-developers] MDEV-6431 review
by Sergey Vojtovich 27 Nov '14

27 Nov '14

Hi Sergei, overall it looks good. I want to double check the clean-up stuff, since I didn't understand everything at the first glance. Some comments inline. > diff --git a/include/mysql/plugin_password_validation.h b/include/mysql/plugin_password_validation.h > new file mode 100644 > index 0000000..0bbb8f4 > --- /dev/null > +++ b/include/mysql/plugin_password_validation.h > @@ -0,0 +1,45 @@ > +/** > + Password validation plugin descriptor > +*/ > +struct st_mysql_password_validation > +{ > + int interface_version; /**< version plugin uses */ > + /** > + Function provided by the plugin which should perform password validation > + and return 0 if the password has passed the validation. > + */ > + int (*validate_password)(MYSQL_LEX_STRING *username, > + MYSQL_LEX_STRING *password); > +}; > +#endif > + This differs from MySQL API, so we can't reuse their plugins. Also they have this cool get_password_strength method, but that's something we can expose via UDF. OTOH we can compare password against username, and that's quite common password misuse. And what I always hated in this validation stuff is that you'll never come up with a password that makes it happy. Of course unless you know requirements. In our case requirements are exposed via system variables. This is more or less acceptable. But I believe it would be nice if this stuff would be able to generate strong passwords. Again, that's something that can be exposed as UDF. > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 529a795..cdf8c7b 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -1051,7 +1055,8 @@ static bool plugin_add(MEM_ROOT *tmp_root, > continue; // invalid plugin type > > if (plugin->type == MYSQL_UDF_PLUGIN || > - (plugin->type == 8 && tmp.plugin_dl->mariaversion == 0)) > + (plugin->type == MariaDB_PASSWORD_VALIDATION_INTERFACE_VERSION && > + tmp.plugin_dl->mariaversion == 0)) > continue; // unsupported plugin type > > if (name->str && my_strnncoll(system_charset_info, You compare type against version here. Why? Regards, Sergey

2 1

Re: [Maria-developers] [Commits] Rev 4369: MDEV-7148 - Recurring: InnoDB: Failing assertion: !lock->recursive in lp:maria/5.5
by Kristian Nielsen 27 Nov '14

27 Nov '14

Sergey Vojtovich <svoj(a)mariadb.org> writes: > revno: 4369 > revision-id: svoj(a)mariadb.org-20141120094823-fozdsrm1kzv46kxi > parent: jplindst(a)mariadb.org-20141119182734-cwbaed0ka90ocj5e > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5 > timestamp: Thu 2014-11-20 13:48:23 +0400 > message: > MDEV-7148 - Recurring: InnoDB: Failing assertion: !lock->recursive > > On PPC64 high-loaded server may crash due to assertion failure in InnoDB > rwlocks code. > > This happened because load order between "recursive" and "writer_thread" > wasn't properly enforced. Looks ok. I failed to understand why the patch would fix the assertion mentioned in the bug report. But the patch itself looks correct. - Kristian.

3 3

Re: [Maria-developers] [Commits] Rev 3902: MDEV-7123: MariaDB 10.0.14 Galera node shutdown with signal 11 in lp:~maria-captains/maria/maria-10.0-galera
by Kristian Nielsen 26 Nov '14

26 Nov '14

Nirbhay Choubey <nirbhay(a)mariadb.com> writes: > revno: 3902 > revision-id: nirbhay(a)mariadb.com-20141123163533-9ov1yirj26cur301 > parent: nirbhay(a)mariadb.com-20141119183337-n5spskf54beqbhif > committer: Nirbhay Choubey <nirbhay(a)mariadb.com> > branch nick: maria-10.0-galera > timestamp: Sun 2014-11-23 11:35:33 -0500 > message: > MDEV-7123: MariaDB 10.0.14 Galera node shutdown with signal 11 > > wsrep-patch uses same connection name for constructing Master_info > objects. As a result all existing wsrep Master_info objects refer > to same rpl_filter object. This could lead to race when multiple > threads try to destruct/delete Master_info object, as they would all > try to delete the same relay_log object. > Fixed by adding a mutex to protect the critical section. The patch is ok. But please consider the following points, and act on them or not as you see fit: 1. You are introducing a mutex lock/unlock in a general data structure used in a couple of places (looks like keycaches and rpl_filters?). Did you consider the performance impact of this? When will the additional locking/unlocking take place? If it only happens in less often used operations (like re-configuring replication filters or key caches?), then extra locking is not an issue. If it happens often, like once per query or something like that, it would probably be best to find a solution with less performance impact. (mutex lock and unlock are quite expensive even if not contended). 2. Consider using explicit mysql_mutex_lock() and mysql_mutex_unlock() calls rather than introducing extra lock()/unlock() methods. The extra methods are needless abtractions that make it harder to understand exactly what is happening wrt. locking. 3. Did you consider if a test case could be made for the originally reported bug? We used to have a rule that any bug fix must include a test case that reproduces the bug fixed (ie. the test case must fail before the patch). Not a hard rule (a few bugs can be extremely hard to write mtr testcases for), but a good rule nevertheless. But maybe this rule has been dropped, now that the focus is on quantity of Jira tickes closed rather than quality. - Kristian.

1 0

[Maria-developers] Windows: git config autocrlf
by Alexander Barkov 26 Nov '14

26 Nov '14

Hi, What would be the correct value for the autocrlf setting in git on Windows? Git suggests autocrlf=true by default at installation time. But then mysql-test do not work with this value. What is a better choice between the other two: "false" or "input"? Thanks.

1 0

[Maria-developers] doubt about read uncommited
by Roberto Spadim 25 Nov '14

25 Nov '14

hi guys, i use read uncommited sometimes with innodb, that's nice and work but now, i'm using a myisam table, and a aria table.... does read uncommited work with this kind of engine? i tested and table stay 'waiting table lock' while a looooong update occurs i don't know what's the internal diference but is possible to "easily" implement read uncommited to myisam/aria? -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

4 10

[Maria-developers] Please review: MDEV-7199 Introduce the concept of data type handlers
by Alexander Barkov 25 Nov '14

25 Nov '14

Hi Sergei, while working on addressing your review suggestions for "MDEV-4912 Add a plugin to field types (column types)", I decided to create a separate tasks for a piece of additional changes: https://mariadb.atlassian.net/browse/MDEV-7199 MDEV-7199 Introduce the concept of data type handlers Please review. Thanks.

1 0

Re: [Maria-developers] [Commits] Rev 4507: MDEV-7157 plugins.server_audit fails sporadically in buildbot. in file:///home/hf/wmar/mdev-7157/
by Kristian Nielsen 24 Nov '14

24 Nov '14

<holyfoot(a)askmonty.org> writes: > ------------------------------------------------------------ > revno: 4507 > revision-id: holyfoot(a)askmonty.org-20141123225345-p2xd7yrklzee10u1 > parent: sergii(a)pisem.net-20141121192039-d0lv6cj96kg5pw02 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-7157 > timestamp: Mon 2014-11-24 02:53:45 +0400 > message: > MDEV-7157 plugins.server_audit fails sporadically in buildbot. > Records can get to the different place in the log when multiple thread > are logged. So the delay added to let the record be saved on the same > place. > === modified file 'mysql-test/suite/plugins/t/server_audit.test' > --- a/mysql-test/suite/plugins/t/server_audit.test 2014-03-26 19:58:27 +0000 > +++ b/mysql-test/suite/plugins/t/server_audit.test 2014-11-23 22:53:45 +0000 > @@ -43,6 +43,7 @@ show variables like 'server_audit%'; > set global server_audit_mode=1; > set global server_audit_events=''; > create database sa_db; > +--sleep 2 The correct way to do this is to use DEBUG_SYNC... - Kristian.

1 0

Re: [Maria-developers] [Commits] Rev 3879: MDEV-6593 : domain_id based replication filters in lp:~maria-captains/maria/maria-10.0-galera
by Kristian Nielsen 24 Nov '14

24 Nov '14

Nirbhay Choubey <nirbhay(a)skysql.com> writes: > ------------------------------------------------------------ > revno: 3879 > revision-id: nirbhay(a)skysql.com-20140912181622-s33e7r0vtgpju95q > parent: nirbhay(a)skysql.com-20140814224304-rtkeal7p9lk06li1 > committer: Nirbhay Choubey <nirbhay(a)skysql.com> > branch nick: maria-10.0-galera_6593 > timestamp: Fri 2014-09-12 14:16:22 -0400 > message: > MDEV-6593 : domain_id based replication filters > > Added support for IGNORE_DOMAIN_IDS and DO_DOMAIN_IDS. Here is my review. Some general comments, followed by some detailed comments in-line in the patch: 1. Did you prepare any documentation? You could either add it to the knowledgebase directly, or write up a text file and ask Daniel to add it in relevant places. 2. Did you run the full test suite? I would have expected at least some tests in the multi_source suite to need modification. The chance touches enough things that you should test it in a feature tree in Buildbot to ensure that it is green before pushing it to a main tree. 3. I think some extra test cases are needed: - You should test (and document) what happens when the same domain_id is in both the DO and the IGNORE list. Is it actually ever sensible to have both DO_DOMAIN_IDS and IGNORE_DOMAIN_IDS ? If not, I think it should be an error to try. - You should also add a test using non-GTID mode. - You should also add a test where parallel replication is used (--slave-parallel-threads > 0). - Add a test for the following scenario when using GTID mode: Some domain id is ignored, some events are received from the master and ignored. Then the slave is stopped, and the ignore rule is removed. What happens when the slave is restarted and reconnects? I think the previously ignored domain id events will now be re-fetched and replicated. I guess that's ok, but it should be documented. Also test what happens when binlog files on the master have been rotated and purged, so that the events are no longer available on the master (an error should occur). - Add a similar test for when not using GTID mode, the last event replicated by the slave is ignored, the slave is stopped, the ignore rule removed and the slave restarted; will the last event now be re-fetched or not (I think not). - Add a test that checks that seconds_behind_master is properly updated to zero even when the last event is ignored due to DO_DOMAIN_IDS/IGNORE_DOMAIN_IDS. It can perhaps be a bit tricky to test, but I think you can use this existing test as a basis: mysql-test/suite/rpl/t/rpl_parallel2.test - It would also be good to check that the code works when used on an old master (which does not send GTID events). You can do this by creating a binlog file with a 5.5 server and storing it in mysql-test/std_data/, and then installing it in a master; there should be several other test cases that already do this. 4. Also see detailed comments for some possible problems with the implementation. The most serious is probably to ensure that events are not skipped after the end of the group, we need a couple of tests for this, see below. Also, there is a small memory leak, see below. > +DO_DOMAIN_IDS (before) : 1 > +IGNORE_DOMAIN_IDS (before) : 2 > +CHANGE MASTER TO DO_DOMAIN_IDS=(4,4,5,1,7,7,7,1,1,2,6,8,1,4,5,5,9,3), MASTER_USE_GTID=current_pos; So if a domain_id is specified multiple times, the redundant ones will be ignored. It would seem preferable to give an error... but as far as I can see the behaviour is the same for IGNORE_SERVER_IDS, so better to be consistent as you did in the code. Just be sure to add this to the documentation. > === added file 'mysql-test/suite/rpl/t/rpl_domain_id_filter.test' > --- a/mysql-test/suite/rpl/t/rpl_domain_id_filter.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/rpl/t/rpl_domain_id_filter.test 2014-09-12 18:16:22 +0000 > +SET @gtid_domain_id_saved = @@session.gtid_domain_id; This is not needed, nor do you use it anywhere, you can remove it. > +##### Case 3: When both DO_DOMAIN_IDS and IGNORE_DOMAIN_IDS are non-empty > + > +source include/stop_slave.inc; > +let $do_domain_ids_before= query_get_value(SHOW SLAVE STATUS, Replicate_Do_Domain_Ids, 1); > +let $ignore_domain_ids_before= query_get_value(SHOW SLAVE STATUS, Replicate_Ignore_Domain_Ids, 1); > +--echo DO_DOMAIN_IDS (before) : $do_domain_ids_before > +--echo IGNORE_DOMAIN_IDS (before) : $ignore_domain_ids_before > + > +# Replicate events belonging to "domain_id 1". (DO_DOMAIN_IDS should win!) > +CHANGE MASTER TO DO_DOMAIN_IDS=(1), IGNORE_DOMAIN_IDS=(2), MASTER_USE_GTID=current_pos; Did you intend here to test overlapping ids in DO_DOMAIN_IDS and IGNORE_DOMAIN_IDS (due to the comment "DO_DOMAIN_IDS should win") ? But if I understand correctly, IGNORE_DOMAIN_IDS will be ignored if DO_DOMAIN_IDS is non-empty. So I think it should be an error to try to set both - do you agree? > +--source include/rpl_end.inc Please add additional testcases, as suggested at the start of review. > === modified file 'sql/rpl_mi.cc' > --- a/sql/rpl_mi.cc 2014-02-19 10:05:15 +0000 > +++ b/sql/rpl_mi.cc 2014-09-12 18:16:22 +0000 > @@ -513,6 +513,25 @@ > else > mi->using_gtid= Master_info::USE_GTID_NO; > } > + > + // DO_DOMAIN_IDS > + if (0 == strncmp(buf, STRING_WITH_LEN("do_domain_ids=")) && > + mi->domain_id_filter.init_ids(buf + sizeof("do_domain_ids"), > + Domain_id_filter::DO_DOMAIN_IDS)) > + { > + sql_print_error("Failed to initialize master info do_domain_ids"); > + goto errwithmsg; > + } > + > + // IGNORE_DOMAIN_IDS > + if (0 == strncmp(buf, STRING_WITH_LEN("ignore_domain_ids=")) && > + mi->domain_id_filter.init_ids(buf + sizeof("ignore_domain_ids"), > + Domain_id_filter::IGNORE_DOMAIN_IDS)) > + { > + sql_print_error("Failed to initialize master info " > + "ignore_domain_ids"); > + goto errwithmsg; > + } So here you use the same style as adjacent code (that I wrote). However, I have since learned that the style (0 == strncmp(...)) is not used in the server. So please change this, as well as my existing code, to use either strncmp(...) == 0 or !strncmp(), as you prefer. Another thing from my old code is the sizeof("do_domain_ids"), which highly confused me for a bit before I realised that while the '=' is not included in the sizeof(), the terminating '\0' _is_ included. So since even I myself is confused by this code, better clarify it ;-) Perhaps it would be clearer if changed to (sizeof("do_domain_ids=")-1) (both in my old code and in your new code). (Normally it's good to follow the style of existing code as you did, I just noticed this in the review and realised that my own "existing code" would benefit from these small improvements). > @@ -634,6 +653,16 @@ > } > } > > + char* do_domain_ids_buf= > + mi->domain_id_filter.as_string(Domain_id_filter::DO_DOMAIN_IDS); > + if (do_domain_ids_buf == NULL) > + DBUG_RETURN(1); > + > + char* ignore_domain_ids_buf= > + mi->domain_id_filter.as_string(Domain_id_filter::IGNORE_DOMAIN_IDS); > + if (ignore_domain_ids_buf == NULL) > + DBUG_RETURN(1); You have a memory leak here, do_domain_ids_buf will not be freed if ignore_domain_ids_buf fails due to out-of-memory. > @@ -1348,4 +1382,218 @@ > DBUG_RETURN(result); > } > > +/* ctor */ > +Domain_id_filter::Domain_id_filter() : m_filter(false) > +{ > + for (int i= DO_DOMAIN_IDS; i <= IGNORE_DOMAIN_IDS; i ++) > + { > + my_init_dynamic_array(&m_domain_ids[i], sizeof(uint32), 16, 16, MYF(0)); > + } > +} > + > +/* dtor */ > +Domain_id_filter::~Domain_id_filter() I don't think comments like /* ctor */ and /* dtor */ are useful, better just omit them. > +/** > + Update the domain id list of the given type with elements from the new list. > + > + @param new_ids [IN] new list of ids > + @param type [IN] domain id list type to update > + @param changed [IN] has the domain id list changed in the > + last CHANGE MASTER command? > + > + @retval void > +*/ > +void Domain_id_filter::update_ids(DYNAMIC_ARRAY *new_ids, enum_list_type type, > + bool changed) > +{ > + DYNAMIC_ARRAY *ids= &m_domain_ids[type]; > + > + if (changed == true) > + reset_dynamic(ids); > + > + /* bsearch requires an ordered list. */ > + sort_dynamic(new_ids, change_master_domain_id_cmp); > + > + for (uint i= 0; i < new_ids->elements; i++) > + { > + uint32 domain_id; > + get_dynamic(new_ids, (void *) &domain_id, i); > + > + if (bsearch((const uint32 *) &domain_id, ids->buffer, ids->elements, > + sizeof(uint32), change_master_domain_id_cmp) == NULL) > + { > + insert_dynamic(ids, (uint32*) &domain_id); > + } > + } > + return; > +} Ok, so I understand that this code is modelled after the existing code for IGNORED_SERVER_IDS. But I found it rather confusing, because it looks like there is a bug: if changed==false, then the code does not ensure that the resulting array will still be sorted. I guess the bug is not possible to trigger, because the list of stuff to add will only be non-empty if changed==true? But that is rather confusing. Please simplify the logic; maybe you only need to call this function when changed==true, so the parameter can be omitted and the array always reset? Also, consider if you can rewrite the existing code for IGNORE_SERVER_IDS so that the same common code can be used as for IGNORE_DOMAIN_IDS and DO_DOMAIN_IDS, that would simplify the code and reduce risk of bugs or inconsistencies. > +/** > + Initialize the given domain id list with the numbers from the specified > + buffer. > + > + @param buf [IN] buffer > + @param type [IN] domain id list type > + > + @retval false Success > + true Error > +*/ > +bool Domain_id_filter::init_ids(char *buf, enum_list_type type) > +{ > + char *token, *last; > + uint num_items; > + DYNAMIC_ARRAY *ids= &m_domain_ids[type]; > + > + token= strtok_r(buf, " ", &last); > + if (token == NULL) > + { > + return true; /* error */ > + } > + > + num_items= atoi(token); > + for (uint i= 0; i < num_items; i++) > + { > + token= strtok_r(NULL, " ", &last); > + if (token == NULL) > + { > + return true; /* error */ > + } > + else > + { > + uint32 val= atoi(token); > + insert_dynamic(ids, (uchar *) &val); > + } > + } > + return false; > +} Again, did you consider using the same code for this and for IGNORE_SERVER_IDS? (I think you did, because you added a comment to init_dynarray_intvar_from_file ;-). Maybe consider it again; it is fine to rewrite/remove the existing code for IGNORE_SERVER_IDS and replace it with your new code. Again, it would reduce risk for bugs or inconsistencies. > + /* getter */ > + bool get_filter() { return m_filter; } > + > + /* setter */ > + void set_filter(uint32 domain_id); These are not really getter and setters, I think, at least I find those names confusing. Maybe something like do_filter(uint32 domain_id) and is_group_filtered() ? And expand the comments, so that they explain that these functions respectively check a GTID for needing to be filtered, and return whether the current group is being filtered. Also, personally I dislike setter/getter, and would prefer just to access the m_filter as a public member. But I suppose that's style and up to you. > + /* > + Initialize the given domain id list with the numbers from the specified > + buffer. > + > + @param buf [IN] buffer > + @param type [IN] domain id list type > + > + @retval false Success > + true Error > + */ > + bool init_ids(char *buf, enum_list_type type); Clarify in the comment that the list is initialized from a string of space-separated numbers, the first of which is the number of following entries. > @@ -182,6 +279,7 @@ > bool flush_relay_log_cache, > bool need_lock_relay_log); > int change_master_server_id_cmp(ulong *id1, ulong *id2); > +int change_master_domain_id_cmp(uint32 *id1, uint32 *id2); I think change_master_domain_id_cmp() is only used in rpl_mi.cc, so should be static there, and not refered here in rpl_mi.h > === modified file 'sql/slave.cc' > --- a/sql/slave.cc 2014-08-12 03:55:41 +0000 > +++ b/sql/slave.cc 2014-09-12 18:16:22 +0000 > @@ -1259,6 +1259,7 @@ > going to ignore (to not log them in the relay log). > Items being read are supposed to be decimal output of values of a > type shorter or equal of @c long and separated by the single space. > + It also used to restore DO_DOMAIN_IDS & IGNORE_DOMAIN_IDS lists. This comment suggests you wanted to share code, then later changed your mind. If you end up keeping the code separate, you still need to delete this comment. (And there is a typo: "It also" -> "It is also", but if you remove it anyway...). > @@ -5601,6 +5610,10 @@ > mi->last_queued_gtid= event_gtid; > mi->last_queued_gtid_standalone= > (gtid_flag & Gtid_log_event::FL_STANDALONE) != 0; > + > + /* Should filter all the subsequent events in the current GTID group? */ > + mi->domain_id_filter.set_filter(event_gtid.domain_id); So, here we will set m_filter to true, if we want to skip the following group. But I don't see any code that will set it to false at the end of the group? That seems a problem, as we could then end up wrongly skipping all kinds of other events, such as Gtid_list, Incident, ... Also note that it is possible for a GTID group to not be terminated by COMMIT/XID. This can happen at least if the master crashes in the middle of writing a group; the slave will receive only the first part of it, followed by a master restart Format_description event. That case needs to be handled, and you should add a test case for it. Also it is possible for eg. a master downgrade to introduce following event groups that do not start with GTID - seems these would also be wrongly skipped? So we also need a test case for this... Hope this helps, - Kristian.

2 8

[Maria-developers] The variable "table_definition_cache" is autosized by default on MySQL 5.6.8+ but not on MariaDB 10
by Jean Weisbuch 20 Nov '14

20 Nov '14

The value of the variable "table_definition_cache" is by default autosized (based on the table_open_cache value) on MySQL >=5.6.8 while it is still static (400) on the latest MariaDB 10.0.14 and 10.1.?, is that by choice or this feature hasn't been backported and should be? From the MySQL doc http://dev.mysql.com/doc/refman/5.6/en/server-system-variables.html#sysvar_… : *Permitted Values (>= 5.6.8)* *Type* |numeric| *Default* |-1 (autosized)| *Min Value* |400| *Max Value* |524288| The default value is based on the following formula, capped to a limit of 2000: 400 + (table_open_cache / 2) Before MySQL 5.6.8, the default is 400.

1 0