March 2014 - developers - lists.mariadb.org

[Maria-developers] Group commit id in mysqlbinlog output
by Kristian Nielsen 18 Mar '14

18 Mar '14

There was a question on the output of mysqlbinlog related to group commit. In 10.0, if two transactions group commit together on the master, their GTID event contains a "commit id" - a 64-bit number. This is used by parallel replication; if two transactions have the same commit id, they can be executed in parallel. In mysqlbinlog output, this looks like this: #140314 13:42:56 server id 1 end_log_pos 772 GTID 0-1-12 cid=180 ... #140314 13:42:56 server id 1 end_log_pos 1027 GTID 0-1-13 cid=180 But if a transaction commits alone on the master (no other transactions participate in the group commit), the mysqlbinlog outpus has no commit id: #140314 13:42:56 server id 1 end_log_pos 437 GTID 0-1-10 The question was why there is no cid=X in the second case, as it makes scripting/grepping the output harder. The reason is that there is no commit id in the event in the binlog in this case (to reduce the size of the binlog). So there is no valid number to put in there. One option could be to make the output something like this: #140314 13:42:56 server id 1 end_log_pos 437 GTID 0-1-10 cid=<none> This would allow to grep for "cid=" and catch everything. I do not have a strong opinion one way or the other, if there are people who find this useful, then let me know and I can change it. - Kristian.

4 6

Re: [Maria-developers] rbr triggers (all)
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, Oleksandr! Ok to push. See a couple of comments below. On Mar 18, Oleksandr Byelkin wrote: > === added file 'mysql-test/suite/rpl/t/rpl_row_triggers.test' > --- mysql-test/suite/rpl/t/rpl_row_triggers.test 1970-01-01 00:00:00 +0000 > +++ mysql-test/suite/rpl/t/rpl_row_triggers.test 2014-03-11 08:49:31 +0000 > @@ -0,0 +1,280 @@ > +-- source include/have_binlog_format_row.inc > +-- source include/have_rbr_triggers.inc > +-- source include/master-slave.inc > +-- source include/have_innodb.inc Note that master-slave.inc should always be the last in the series of includes. Everything that can fail should be before it, otherwise you can abort and skip the test after configuring replication, but without a corresponding replication cleanup. It will affect following tests. > === added file 'mysql-test/suite/rpl/t/rpl_row_triggers_sbr.test' > --- mysql-test/suite/rpl/t/rpl_row_triggers_sbr.test 1970-01-01 00:00:00 +0000 > +++ mysql-test/suite/rpl/t/rpl_row_triggers_sbr.test 2014-03-11 08:49:48 +0000 > @@ -0,0 +1,43 @@ > +--source include/master-slave.inc > +--source include/have_binlog_format_statement.inc > +--source include/have_rbr_triggers.inc Same here. > === modified file 'sql/log_event.cc' > --- sql/log_event.cc 2014-03-05 22:20:10 +0000 > +++ sql/log_event.cc 2014-03-11 18:39:51 +0000 > @@ -11121,6 +11233,13 @@ Rows_log_event::write_row(rpl_group_info > TABLE *table= m_table; // pointer to event's table > int error; > int UNINIT_VAR(keynum); > + > +#ifdef RBR_TRIGGERS > + bool invoke_triggers= > + slave_run_triggers_for_rbr && !master_had_triggers && table->triggers; > +#else > +#define invoke_triggers 0 > +#endif //RBR_TRIGGERS I think you don't need this ifdef, just declare invoke_triggers as const: const bool invoke_triggers= slave_run_triggers_for_rbr && !master_had_triggers && table->triggers; > auto_afree_ptr<char> key(NULL); > > prepare_record(table, m_width, Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3994: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, Holyfoot! On Dec 08, holyfoot(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 3994 > revision-id: holyfoot(a)askmonty.org-20131208143921-3veatgg9wngyzb47 > parent: holyfoot(a)askmonty.org-20131202104753-srqqm7hv1rtcurx4 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Sun 2013-12-08 18:39:21 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > Thread can be disconnected internally for example after COMMIT statements. What do you mean by that? > So we should check this for the statement execution. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3993: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, holyfoot! On Dec 02, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3993 > revision-id: holyfoot(a)askmonty.org-20131202104753-srqqm7hv1rtcurx4 > parent: holyfoot(a)askmonty.org-20131130142537-7j272euoaoy0svg1 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Mon 2013-12-02 14:47:53 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > mysqltest in the 'embedded-server' mode runs queries in a separate thread, > but it didn't do so for the statements - they were run in the main thread. > That leads to inconsistencies. > When a test sets SESSION 'dbug' variable like > SET SESSION debug_dbug="+d,warn_during_ha_commit_trans"; > it is run as a plain query in that separate thread, so the main thread remains unaffected. > After that the statement run in the main thread doesn't produce expected 'dbug' errors, > so the test fail. > To fix that I made statement to be run in that special thread along with the plain queries. > That makes the environment consistent. Ok to push. But sayng "statements" in the comment is a big misleading, it'd be cleared if you'd say "prepared statements". Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3992: MDEV-5138 Numerous test failures in "mtr --ps --embedded". in file:///home/hf/wmar/mdev-5138/
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, holyfoot! On Nov 30, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3992 > revision-id: holyfoot(a)askmonty.org-20131130142537-7j272euoaoy0svg1 > parent: holyfoot(a)askmonty.org-20131130132432-etlmes8qhhqhr1iu > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Sat 2013-11-30 18:25:37 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > As Davi added code like > sav_protocol= thd->protocol > thd->protocol= &thd->protocol_binary > ... > thd->protocol= sav_protocol > the fucntions like emb_store_querycache_result() cannot determine > the used protocol testing thd->protocol == &thd->protocol_binary. > Fixed by additional check thd->command == COM_STMT_EXECUTE. What about PREPARE? Regards, Sergei

1 0

Re: [Maria-developers] MaxScale as a binlog server
by Kristian Nielsen 18 Mar '14

18 Mar '14

Mark Riddoch <mark.riddoch(a)skysql.com> writes: > we have been approached by a user with a suggestion for a project that we > might do with them. They have a replication environment with a large fan out > (100 slaves per master) and are looking at doing something to cut one the > load on the master. the thought they had was to use MaxScale to effectively > cache the bin log files, so MaxScale would act as a single Slave to the > Master and would itself have a number of Slaves that read the bin log from > MaxScale. My initial response was why not just use an instance of MariaDB as > the intermediate node and connect slaves to one of those instances, in a > tree structure. They had a few reasons why they did not want to do this; > Looking in more detail I think it is feasible, we essentially write a router > module for MaxSale that acts as a slave, aching the bin log locally on the > MaxScale node. We then have another router, or maybe even the same one, that > relays that binlog to the real slaves. We are looking at doing something for > MariaDB 10, since we would want to utilise the GTID to get the best failover > semantics, I am assuming there must be some differences between the > replication stream that I can find documented for MySQL and what MariaDB 10 > does? Is this stuff anywhere on the Knowledge Base? I tried looking but Right, so I agree with what Serg said, that maybe going back to the original problem and seeing if there are better solutions possible would be better. However, I will try to answer the technical aspect here. I suppose caching and routing binlogs outside of the server should be possible, but it might be more complex than you would think at first. I am not aware of any comprehensive documentation for this. However, it should be relatively easy to see from the code. Basically, it is the function mysql_binlog_send() in sql/sql_repl.cc that handles sending binlog data to a slave. There are good comments in there describing many of the trickier points. You need to be aware that there is more logic in sending binlog data to a slave than just streaming raw binlog files. Maybe you can implement a subset and document any limitations. The old-style replication (not using GTID) is the simplest, but it still does a bit extra, like sending some extra events (FORMAT_DESCRIPTION_EVENT and fake ROTATE_EVENT). There is also the handling of old slaves. A slave sends a value in @mariadb_slave_capability, and the master will rewrite or remove any events that the slave does not understand, as appropriate. And there is the @@skip_replication flag; when the slave sets this, the master removes events that were logged with @@skip_replication set (to reduce bandwidth needs). The slave also does several SQL queries to obtain additional information. For example, the slave does a SELECT binlog_gtid_pos(file, offset) to obtain the GTID position associated with a given old-format replication position, so that it knows how to switch to GTID mode later. Maybe those queries could be handled by relaying them back to a real master server. The GTID mode is significantly more complex, due to a GTID position having multiple streams, one per replication domain. The slave sends @slave_connect_state, @slave_gtid_strict_mode, @slave_gtid_ignore_duplicates, and optionally @slave_until_gtid values. The router would need to parse these values and act accordingly. The binlog sender needs to keep track of each domain as the binlog files are scanned. Events in a domain must be skipped until the point specified in @slave_connect_state is reached. Then events are sent until the point in @slave_until_gtid is reached, at which point further events must again be skipped. The master also sends extra GTID_LIST events, which contain the state of the binlog at the point of the event, which is needed by the slave to correctly handle START SLAVE UNTIL master_gtid_pos=XXX and MASTER_GTID_WAIT(). A lot of effort was put into error handling, to support all reasonable, and even many unreasonable uses of replication, while still giving an error in cases where things are obviously wrong (to prevent silently doing the wrong thing or hanging). For example, if the slave tries to connect at a GTID that the master does not have, this is normally an error. This is important, because that GTID might be some transaction that was executed manually on the slave by mistake; if it only exists on the slave, then the master would endlessly skip events looking for the GTID that never shows up, and replication would silently hang. However, there are special cases where the slave _is_ allowed to connect at a GTID that does not exist on the master. For example, one might have a slave with --log-slave-updates=0. Then if this slave is promoted as a new master, the binlogs will not have the current position, but it is still allowed to connect to it at the point where it became the new master (but not to a point earlier than that). Another special case is if the requested GTID (but no GTIDs following that) was purged from the binary logs, as can easily happen if a replication domain is unused for long time. A _full_ implementation of a router for sending binlog to slaves would be some effort, but it might be reasonable to implement some subset. I suppose the first step would be for you to go through the code in mysql_binlog_send() and the functions it calls, and decide on which parts to implement. I will be happy to answer any questions that might pop up on the way, of course. > 2. They wanted to maintain the same group commit groups as the original > master, so as to benefit from the parallel replication. Note that this may no longer apply to MariaDB 10.0.9. The issue is that an intermediate slave that is itself a master may have less group commit than the original master, which reduces the possibility for a third-level slave to efficiently do parallel replication. But in 10.0.9, it should be possible to configure the intermediate slave with --binlog-commit-wait-* to get better group commit. > 3. They think that can get better failover semantics because they will have > n (maybe 5 or 10) MaxScales and they will all have the same bin log, so > failover between them will be easier. I am curious how it will be assured that they all have the same binlog in case of crash? But maybe you just mean that all will be a subset of the largest one. Hope this helps, - Kristian.

2 1

[Maria-developers] Java Connector Discussion
by Puneet Dewan 18 Mar '14

18 Mar '14

Hi Georg, As we missed our scheduled meeting yesterday, can you suggest a suitable time to discuss? Regards Puneet.

1 0

Re: [Maria-developers] [Maria-discuss] MariaDB vs. MySQL error codes (1900+)
by Kristian Nielsen 17 Mar '14

17 Mar '14

Felipe Gasper <felipe(a)felipegasper.com> writes: > Hi everyone, > > I notice that MariaDB is using the 1900+ range for its own > error codes; however, MySQL 5.7, at least, has a few errors in that > range as well. > > Are error codes just going to be an area of increasing > conflict between the two, or is there a plan to migrate MariaDB codes > to a different numbering scheme? Thanks for bringing this to attention. This is a pretty serious issue, and I've forwarded it to Tomas Ulin, who should be in a position to get it fixed. Tomas, please note this issue. MariaDB has used error codes from 1900 from the very start to avoid these kinds of conflicts. They can not be changed, as the codes are already in GA releases and applications may be relying on the numbers. You will need to ensure that MySQL 5.7 avoids conflicts with these before 5.7 goes GA, or there will be a conflict that will hurt all users and will be very hard to fix later. Felipe, you should probably also file a bug on bugs.mysql.com to try to get this fixed before MySQL 5.7 GA. Thanks, - Kristian.

2 1

[Maria-developers] redundant compilation of targets in mariadb 10.0
by Rohit Kalhans 17 Mar '14

17 Mar '14

hello all, I am trying to fix a few bugs but ever after small modifications in specific files, make command build all targets irrespective of whether or not they are dependent on the modified source. I am no expert in build commands but this seem off since it keeps taking exceptionally long time to build. Is there a trick/workaround that can prevent all targets from being built. Or am I missing something. -- Cheerio! *Rohit* *http://rohitkalhans.com <http://rohitkalhans.com>*

3 6

[Maria-developers] For Google Summer of Code 2014, Interested in the task of "statistically optimize mysql-test runs by running less tests"
by 胡仲义 17 Mar '14

17 Mar '14

Dear Sergei Golubchik, I am a post graduate student of Institute of Software, Chinese Academy of Sciences and my name is Zhongyi Hu. I major in computer science and my research field is data stream mining. Because I have got enough papers and works for graduation, I want to do something interesting, meaningful and valuable in the rest time as student. I have participated in two projects about database, one is main memory database and the other is database cluster. I got some experience of database system design and implementaion from them. Although I am just a beginner of this area, I really like it and expect to make it as my career. I often use Mysql in research and work, but MariaDB is not very familiar to me. I am tremendously optimistic about it's future because all of you. Well, let's come to the point. I am interested in the task of "statistically optimize mysql-test runs by running less tests". I chose this task because I have written a few tools for automatic test. I know the performance is very important if there are a large amount of data or cases to test. I read the MDEV-5776 and I think the major job is as follow. When the code is changed, the mysql-test is used to do the requisite tests. We need to integrate the information of the changes and the scenarios to predict the probability of failure for each test and get the relationships of the tests. Then decide what to test and what test cases should be used. The purpose is to optimize the efficiency of testing. All of these should be done by algorithm and program. In addition, I think that the job is in some ways like mining in data stream, such as many data need to be statistical analyzed and the hidden patterns changing over time. At last, I have two basic questions. 1) What exactly are the builder and the combination? I thought they refer to compiler and runtime environment. 2) What does the "individual tests within a big test file" mean? Maybe I am completely wrong, but I still look forward to your reply. I hope to have the opportunity to learn from you in work and discussion. Best regards Zhongyi Hu 20140314

3 5