March 2014 - developers - lists.mariadb.org

[Maria-developers] two places maybe need adjusted in rpl_parallel.cc
by nanyi607rao 07 Mar '14

07 Mar '14

hi Kistian, I afraid there are two palces need adjusted in rpl_parallel.cc First is in rpl_parallel_thread::get_rgi() adjusted like this ... else { if(!(rgi= new rpl_group_info(rli))) { my_error(ER_OUTOFMEMORY, MYF(0), (int)sizeof(*rgi)); return NULL; } rgi->is_parallel_exec = true; - if ((rgi->deferred_events_collecting= rli->mi->rpl_filter->is_on())) _ rgi->deferred_events= new Deferred_log_events(rli); } + if ((rgi->deferred_events_collecting= rli->mi->rpl_filter->is_on())) + rgi->deferred_events= new Deferred_log_events(rli); because rgi->deferred_events was freed in rpl_parallel_thread::free_rgi() if it wasn't NULL Second is in rpl_parallel_change_thread_count() I think it need add mysql_cond_broadcast(&pool->COND_rpl_thread_pool) after all new threads are ok, this is for the case that skip_slave_start=OFF,then SQL thread would start automaticlly, it could wait forever in rpl_parallel_thread_pool::get_thread(), because SQL thread may reach this function before another thread reach the rpl_parallel_change_thread_count(). 2014-03-04 nanyi607rao

2 2

Re: [Maria-developers] Replication consistancy in MaxScale
by Kristian Nielsen 07 Mar '14

07 Mar '14

Mark Riddoch <mark.riddoch(a)skysql.com> writes: > We have been looking for a cheap way to find this out and came up with an > idea for MariaDB 10 that might be good to discuss. If we use the global > transaction ID we can get the current master ID and the current GTID form > each slave and use them to compare and discover if a slave is lagging based > on a single connection from a client or the entire server. Clearly we can > get this by jus asking in a polling monitor loop, but what would be really > good would be to have a function in MariaDB that we could call that would > wait for the GTID id to change and return the new GTID. THen we could have a > monitor that makes this blocking call and get's to know the new GTID very > quickly compered to having to have some form of poling loop, which would > limit to the rate of change that could be detected to the polling rate of > the monitor. See here: http://kristiannielsen.livejournal.com/18308.html It's similar to what you describe. MASTER_GTID_WAIT() is already in 10.0 (I can't remember if it is in 10.0.8, else it will be in 10.0.9). Note that if you really want to wait for any change in GTID on a slave, you can just take the current @@gtid_slave_pos, add one to every sequence number, and wait for that. - Kristian.

2 1

[Maria-developers] MariaDB 10.0.9 Release Prep
by Daniel Bartholomew 07 Mar '14

07 Mar '14

All, I've begun preparing for the MariaDB 10.0.9 release. The current plan is to upload the files to the mirrors sometime tomorrow (7 Mar, after they are finished building). I'll probably hold off on actually announcing the release until Monday. Draft changelog and release notes pages are at: - https://mariadb.com/kb/en/mariadb-1009-release-notes/ - https://mariadb.com/kb/en/mariadb-1009-changelog/ Thanks. -- Daniel Bartholomew MariaDB

1 0

Re: [Maria-developers] Rev 4009: MDEV-5675 - Performance: my_hash_sort_bin is called too often in lp:maria/10.0
by Sergei Golubchik 06 Mar '14

06 Mar '14

Hi, Sergey! On Mar 06, Sergey Vojtovich wrote: > revno: 4009 > revision-id: svoj(a)mariadb.org-20140306121912-2a2soan1oc7u1hrt > parent: svoj(a)mariadb.org-20140226085528-80vzk8f2c4b4ugxm > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Thu 2014-03-06 16:19:12 +0400 > message: > MDEV-5675 - Performance: my_hash_sort_bin is called too often > > Reduced number of my_hash_sort_bin() calls from 4 to 1 per query. > Reduced number of memory accesses done by my_hash_sort_bin(). Looks great! Ok to push Regards, Sergei

1 0

[Maria-developers] Standard way for time->datetime cast
by Alexander Barkov 06 Mar '14

06 Mar '14

Hi Sergei, (my mailer failed during the previous attempt. writing again, sorry if you get this twice) Please review the patch implementing the standard (and MySQL-5.6) compatible way of time to datetime conversion. Thanks.

2 3

[Maria-developers] Parallel replication benchmarks
by Kristian Nielsen 06 Mar '14

06 Mar '14

I've done a set of benchmarks for parallel replication on the same machine I used previously for my group commit benchmarks, http://kristiannielsen.livejournal.com/16382.html The code tested is the newest code in the bzr repository and what will be in 10.0.9 (this is significantly improved from what is in 10.0.8). I plan to write up a blog post about it in a couple of days with nice graphs, but meanwhile Axel asked me to summarise in this mail. I tested with sysbench 0.5, using oltp.lua (medium-sized transactions) and update_index.lua (minimal transactions with just a single primary-key update per transaction). I used 10M rows, 16GB buffer pool and 2 * 1.9 GB redo logs. This is with a single table. I tested simply by preparing the binlog on the master, then setting up an already prepared slave and doing START SLAVE UNTIL the end of the log. The error log then shows the time spent for the slave to catch up. I tested everyting in GTID mode, as that is the recommended mode for parallel replication (though my guess is that old-style replication will be much the same, there isn't much difference in the code between what is done to actually execute events). Node that these tests are for in-order parallel replication. All commits on all slaves happen in the same order as on the master; the use of parallel replication is invisible to applications. This is in contrast to eg. MySQL 5.6 multi-threaded slave, which requires the application to partition its data into independent schemas. Here are the prelimiary results, in number of seconds for the slave to catch up (lower is better) versus number of threads (--slave-parallel-threads, 0 means not using parallel replication): For oltp.lua. 48 threads used to generate the load on the master, and --binlog-commit-wait-count=12 --binlog-commit-wait-usec=10000 to allow the server to delay a commit by up to 10 milliseconds in order to get more group commit and thus more opportunities for parallel apply on the slave: A: --log-slave-updates --sync-binlog=1 --innodb-flush-log-at-trx-commit=1 B: --skip-log-slave-updates --innodb-flush-log-at-trx-commit=1 C: --skip-log-bin --innodb-flush-log-at-trx-commit=2 D: --log-bin=master-bin --sync-binlog=0 --innodb-flush-log-at-trx-commit=0 #thr A B C D 0 1065 869 193 202 2 361 432 147 161 4 221 264 118 121 8 135 177 103 107 12 114 153 104 105 16 109 140 104 107 24 111 139 107 105 32 111 136 99 109 48 111 126 108 109 64 111 121 99 111 We see here a 2-10 times speedup from parallel replication. The master has around 12 transactions in every group commit, which provides good opportunities for parallelism on the slave. Note that parallel replication is especially effective when the binlog is enabled and crash-safe (--sync-binlog=1 --innodb-flush-log-at-trx-commit=1). This is because parallel replication can run the commit of one transaction in parallel with any other transaction, even if the two transactions would otherwise conflict. This makes group commit especially effective. In fact, this manages to more or less completely eliminate any penalty for enabling crash-safe binlog on the slave, which is quite nice. Note also that disabling the binlog actually tends to make things _slower_, not faster, when using parallel replication. I believe this is due to MDEV-5802, which may be worth fixing for 10.0. Here are results for update_simple.lua with 48 threads on the master. This produced around 13 transactions per group commit on the master, with no --binlog-commit-wait-count to delay commits: A: --log-slave-updates --sync-binlog=1 --innodb-flush-log-at-trx-commit=1 B: --skip-log-slave-updates --innodb-flush-log-at-trx-commit=1 C: --skip-log-bin --innodb-flush-log-at-trx-commit=2 #thr A B C 0 931 899 271 2 546 653 258 4 365 494 176 8 261 365 203 12 247 350 197 16 233 336 207 24 242 316 209 32 237 292 194 48 235 270 208 64 228 249 195 Again we get a good speedup from parallel replication, even though with such small transactions, there is less opportunity for improvement, as the actual work for transactions is rather small compared to the overhead for managing the replication of each event. And again, the ability to utilise group commit effectively provides the biggest benefit. Finally, I tried a test of update_index.lua where I ran the load on the master single-threadedly. This creates a binlog with _no_ opportunities for parallelism from group commits - each transaction needs to be executed on its own by the slave, as we do not know for sure that they will not conflict on row locks. However, due to the possibility to run the commits in parallel (and hence get group commit on the slave), we still see some speedup even here when --sync-binlog=1 and --innodb-flush-log-at-trx-commit=1. When binlog and innodb sync is disabled, parallel replication makes things slower due to the overhead of thread communication: A: --log-slave-updates --sync-binlog=1 --innodb-flush-log-at-trx-commit=1 B: --skip-log-slave-updates --innodb-flush-log-at-trx-commit=1 C: --skip-log-bin --innodb-flush-log-at-trx-commit=2 #thr A B C 0 1075 949 270 2 597 673 319 4 443 623 334 8 407 588 349 12 393 544 336 16 391 536 352 24 - 492 336 32 389 472 358 48 389 419 344 64 391 399 354 So overall, results look very good, especially for slaves with binlog enabled. (And binlog disabled could turn out better if MDEV-5802 is fixed). Let me know if there are any questions, and I'll be happy to answer them. - Kristian.

1 0

[Maria-developers] A problem of DDL apply in parallel replication
by nanyi607rao 06 Mar '14

06 Mar '14

hi kristian , Maybe you have encountered this problem and fixed it, that is a DDL statement Query event has a Gtid event ahead, but has no end_event (like xid_event, commit or rollback) behind. so in the latest codes of parallel replication, count_queued_event_groups will increase but count_committing_event_groups will not for DDL. it would lead next event_group wait forever I guess. 2014-03-05 nanyi607rao

2 2

Re: [Maria-developers] gsoc replication tasks
by Kristian Nielsen 06 Mar '14

06 Mar '14

Sergei Golubchik <serg(a)mariadb.org> writes: > I'd like to discuss what replication tasks we might want to be > implemented in this year's google summer of code. I don't really have the bandwidth for handling any GSoC at the moment. But here are a bunch of replication tasks that should be suitable for a newcomer, in approximate increasing order of difficulty: MDEV-4776 FR: Consider showing GTID-based 'until' value in slave status MDEV-5119 Events from a GTID-incapable master inherit domain ID of the slave MDEV-4987 Sort by domain_id when list of GTIDs are output MDEV-5289 master server starts slave parallel threads MDEV-4989 Support for GTID in mysqlbinlog MDEV-4990 Implement simple failover script using GTID MDEV-5262 Missing retry after temp error in parallel replication I would be happy to assist anyone that wants to work seriously on one of these, GSoC or not. I'm available on IRC now, and most of the day if you want to discuss more... - Kristian.

1 0

Re: [Maria-developers] [JIRA] (MDEV-5497) Test suite test connect.fix fails
by Alexander Barkov 05 Mar '14

05 Mar '14

Hello Olivier, You changed status to "fixed", but I cannot see any fixes in 10.0-connect. Are you going to push a fix soon? Thanks. On 03/05/2014 03:36 PM, Olivier Bertrand (JIRA) wrote: > > [ https://mariadb.atlassian.net/browse/MDEV-5497?page=com.atlassian.jira.plug… ] > > Olivier Bertrand commented on MDEV-5497: > ---------------------------------------- > > Sorry Elena, I did not read your comment entirely before answering. Your analyze was quite good, just missing the reason why this was happening. > >> Test suite test connect.fix fails >> --------------------------------- >> >> Key: MDEV-5497 >> URL: https://mariadb.atlassian.net/browse/MDEV-5497 >> Project: MariaDB Development >> Issue Type: Bug >> Affects Versions: 10.0.7, 10.0.8 >> Environment: Slackware Linux 14.1 + kernel 3.12.6 >> Reporter: Duncan Roe >> Assignee: Olivier Bertrand >> Priority: Minor >> Fix For: 10.0.9 >> >> Time Spent: 1 hour >> Remaining Estimate: 0 minutes >> >> {noformat} >> cd /usr/mysql-test; ./mtr connect.fix >> {noformat} >> (as root) gives me an error report as below. I am keeping the mysql-test directory, so please let me know if you'd like anything else from it. >> {noformat} >> Logging: ./mtr connect.fix >> vardir: /usr/mysql-test/var >> Checking leftover processes... >> Removing old var directory... >> Creating var directory '/usr/mysql-test/var'... >> Checking supported features... >> MariaDB Version 10.0.7-MariaDB >> - SSL connections supported >> Collecting tests... >> Installing system database... >> Using server port 43317 >> ============================================================================== >> TEST RESULT TIME (ms) or COMMENT >> -------------------------------------------------------------------------- >> worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 >> worker[1] mysql-test-run: WARNING: running this script as _root_ will cause some tests to be skipped >> connect.fix [ fail ] >> Test ended at 2014-01-01 19:32:13 >> CURRENT_TEST: connect.fix >> --- /usr/mysql-test/plugin/connect/connect/r/fix.result 2013-12-23 20:35:03.000000000 +1100 >> +++ /usr/mysql-test/plugin/connect/connect/r/fix.reject 2014-01-01 19:32:13.000000000 +1100 >> @@ -80,7 +80,7 @@ >> name city birth hired >> John Boston 2 1986-01-05 2010-06-02 >> Henry Boston 0 1987-06-07 2008-04-01 >> -George San Jose 1 1981-08-01 2010-06-02 >> +George San Jose 1 1981-07-31 2010-06-02 >> Sam Chicago 2 1979-11-02 2007-10-10 >> James Dallas 1 1992-05-03 2009-12-14 >> Bill Boston 1 1986-09-01 2008-02-10 >> @@ -96,7 +96,7 @@ >> name city birth hired >> John Boston 2 1986-01-05 2010-06-02 >> Henry Boston 0 1987-06-07 2008-04-01 >> -George San Jose 1 1981-08-01 2010-06-02 >> +George San Jose 1 1981-07-31 2010-06-02 >> Sam Chicago 2 1979-11-02 2007-10-10 >> James Dallas 1 1992-05-03 2009-12-14 >> Bill Boston 1 1986-09-01 2008-02-10 >> @@ -112,7 +112,7 @@ >> name city birth hired >> John Boston 2 1986-01-05 2010-06-02 >> Henry Boston 0 1987-06-07 2008-04-01 >> -George San Jose 1 1981-08-01 2010-06-02 >> +George San Jose 1 1981-07-31 2010-06-02 >> Sam Chicago 2 1979-11-02 2007-10-10 >> James Dallas 1 1992-05-03 2009-12-14 >> Bill Boston 1 1986-09-01 2008-02-10 >> @@ -128,7 +128,7 @@ >> name city birth hired >> John Boston 2 1986-01-05 2010-06-02 >> Henry Boston 0 1987-06-07 2008-04-01 >> -George San Jose 1 1981-08-01 2010-06-02 >> +George San Jose 1 1981-07-31 2010-06-02 >> Sam Chicago 2 1979-11-02 2007-10-10 >> James Dallas 1 1992-05-03 2009-12-14 >> Bill Boston 1 1986-09-01 2008-02-10 >> mysqltest: Result content mismatch >> - saving '/usr/mysql-test/var/log/connect.fix/' to '/usr/mysql-test/var/log/connect.fix/' >> -------------------------------------------------------------------------- >> The servers were restarted 0 times >> Spent 0.000 of 6 seconds executing testcases >> Failure: Failed 1/1 tests, 0.00% were successful. >> Failing test(s): connect.fix >> The log files in var/log may give you some hint of what went wrong. >> If you want to report this error, please read first the documentation >> at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html >> mysql-test-run: *** ERROR: there were failing test cases >> {noformat} > > > > -- > This message was sent by Atlassian JIRA > (v6.2-OD-10-004-WN#6253) >

2 1

Re: [Maria-developers] Active Directory authentication plugin - GSoC 2014
by Sergei Golubchik 05 Mar '14

05 Mar '14

Hi, Talina! On Feb 28, talinashrotriya(a)gmail.com wrote: > Hello Sir, > > My name is Talina, presently studying Computer Science from India. > > I am interested to work on the “Active Directory authentication plugin”project. I am good at C/C++ , MYSQL and also have good knowledge about Active directory. Could you suggest me a few links to get started with? > Eager to start work. > > Thanks! We don't have this project on the https://mariadb.com/kb/en/google-summer-of-code-2014/ anymore. I will add it back when we have a mentor for it. Regards, Sergei

1 0