- developers - lists.mariadb.org

[Maria-developers] Patch for atomics on MIPS in mroonga
by Kristian Nielsen 22 Oct '16

22 Oct '16

Who should be contacted about issues in the mroonga storage engine? The attached patch is from Debian Bug#838914 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838914 Apparently, libatomic is needed on this platform to support 64-bit atomic operations. The patch looks reasonable and should probably be upstreamed. But I am not sure how the mroonga storage engine is maintained - should this go directly into MariaDB? If there is an upstream maintained mroonga storage engine, probably it should preferably go there first? - Kristian.

2 1

[Maria-developers] 161M for storage/tokudb isn't friendly
by MARK CALLAGHAN 21 Oct '16

21 Oct '16

Filed https://jira.mariadb.org/browse/MDEV-11101 but I have no idea who will fix this. All other engines use less than 20M while tokudb uses 161M including one file in mysql-test that is more than 30M. Lots of time, network and disk space is being wasted by this. -- Mark Callaghan mdcallag(a)gmail.com

1 0

Re: [Maria-developers] [MariaDB/server] MDEV-11064 - Restrict the speed of reading binlog from Master (#246)
by Kristian Nielsen 21 Oct '16

21 Oct '16

vinchen <notifications(a)github.com> writes: > cli_safe_read_reallen() and and my_net_read_packet_reallen() is a good > way to fix the ABI problem. I will fix it like this. > > And the minimum precision is second in slave_sleep(), and it also a > the mutex. I think it is too heavy in most case. (It will wait > millisecond seconds usually) > So I will use my_sleep when wait less than 1 second, otherwise use slave_sleep. > > Is it ok? Yes, it looks fine thanks! And thanks for the explanation, yes now I see that usually the sleep will be quite small, unless there is a huge event. So I agree with using my_sleep() in the common case. I'll merge it like this after checking that everything compiles and tests ok. Thanks! - Kristian.

1 0

Re: [Maria-developers] [MariaDB/server] Restrict the speed of reading binlog from Master (#246)
by Kristian Nielsen 19 Oct '16

19 Oct '16

GCSAdmin <notifications(a)github.com> writes: > In some case, the speed of reading binlog from master is high, especially when doing a new replica. > It would bring the high traffic in master. > So We introduce a new variable "read_binlog_speed_limit" to control the binlog reading rate for IO thread to solve the problem. > It can work when slave_compressed_protocol is on. > But it maybe doesn't work well when the binlog event is very big. > You can view, comment on, or merge this pull request online at: > > https://github.com/MariaDB/server/pull/246 > -- Patch Links -- > > https://github.com/MariaDB/server/pull/246.patch > https://github.com/MariaDB/server/pull/246.diff Overall this looks clean and simple. There is one problem. The patch adds a field real_network_read_len to the NET structure. This will break the client library ABI, because NET is a part of MYSQL. So this would cause client programs to crash if they are linked with a different version of the client library. So this needs to be changed (if I understand correctly). One option might be to introduce new functions like cli_safe_read_reallen() and my_net_read_packet_reallen(), which return in addition the actual amount of bytes read from the server. The old cli_safe_read() and my_net_read_packet() could then become simple wrapper functions around those. And cli_safe_read_reallen() can be used in read_event() in sql/slave.cc. A smaller issue is that in case of a large packet, a large my_sleep() may be invoked, which will cause STOP SLAVE to hang. I think this can be solved simply by calling slave_sleep() instead, it handles terminating the wait early if interrupted by STOP SLAVE. Detailed comments on the patch below. I rebased the series against latest 10.2 to get a clean diff (the pull request includes a couple merges against the main 10.2 tree, these changes are unrelated to the patch). The rebase is in https://github.com/knielsen/server/tree/GCSAdmin-10.2-binlog-speed-limit-2 > diff --git a/include/mysql.h.pp b/include/mysql.h.pp > index 857f5b9..1da038c 100644 > --- a/include/mysql.h.pp > +++ b/include/mysql.h.pp > @@ -35,6 +35,7 @@ typedef struct st_net { > my_bool thread_specific_malloc; > unsigned char compress; > my_bool unused3; > + unsigned long real_network_read_len; As explained above, I believe this would break the ABI (that's the purpose of mysql.h.pp, to catch such problems). > diff --git a/sql/slave.cc b/sql/slave.cc > index 20bf68e..52bb668 100644 > --- a/sql/slave.cc > +++ b/sql/slave.cc > @@ -3307,13 +3308,14 @@ static int request_dump(THD *thd, MYSQL* mysql, Master_info* mi, > try a reconnect. We do not want to print anything to > the error log in this case because this a anormal > event in an idle server. > + network_read_len get the real network read length in VIO, especially using compressed protocol > > RETURN VALUES > 'packet_error' Error > number Length of packet > */ > > -static ulong read_event(MYSQL* mysql, Master_info *mi, bool* suppress_warnings) > +static ulong read_event(MYSQL* mysql, Master_info *mi, bool* suppress_warnings, ulong* network_read_len) Generally, lines longer than 80 characters should be avoided (coding style). > @@ -4473,6 +4479,34 @@ Stopping slave I/O thread due to out-of-memory error from master"); > goto err; > } > > + /* Control the binlog read speed of master when read_binlog_speed_limit is non-zero > + */ > + ulonglong read_binlog_speed_limit_in_bytes = opt_read_binlog_speed_limit * 1024; > + if (read_binlog_speed_limit_in_bytes) > + { > + /* prevent the tokenamount become a large value, > + for example, the IO thread doesn't work for a long time > + */ > + if (tokenamount > read_binlog_speed_limit_in_bytes * 2) > + { > + lastchecktime = my_hrtime().val; > + tokenamount = read_binlog_speed_limit_in_bytes * 2; > + } > + > + do > + { > + ulonglong currenttime = my_hrtime().val; > + tokenamount += (currenttime - lastchecktime) * read_binlog_speed_limit_in_bytes / (1000*1000); > + lastchecktime = currenttime; > + if(tokenamount < network_read_len) > + { > + ulonglong micro_sleeptime = 1000*1000 * (network_read_len - tokenamount) / read_binlog_speed_limit_in_bytes ; > + my_sleep(micro_sleeptime > 1000 ? micro_sleeptime : 1000); // at least sleep 1000 micro second > + } > + }while(tokenamount < network_read_len); > + tokenamount -= network_read_len; > + } > + As explained above, probably better to use slave_sleep() here to allow STOP SLAVE to interrupt a long sleep. Would it make sense to do this wait after calling queue_event()? This way the SQL thread can start applying the event immediately, reducing slave lag. What do you think? Thanks, - Kristian.

1 0

[Maria-developers] Consolidated patch for mdev-8646 introduced a bug MDEV-11081
by Alexander Barkov 18 Oct '16

18 Oct '16

Hello Igor, This patch: commit 2cfc450bf78c2d951729d1a0e8f731c0d987b1d5 Author: Igor Babaev <igor(a)askmonty.org> Date: Tue Feb 9 12:35:59 2016 -0800 seems to have introduced this bug: MDEV-11081 Cursor fetches NULL for aggregate functions Can you please have a look? Thanks!

1 0

[Maria-developers] Article on git rebasing - follow up from Developer Meetup
by Vicențiu Ciorbaru 17 Oct '16

17 Oct '16

Hi everyone! During the Developer Meetup in Amsterdam, we've had quite some productive discussions. One of these was on how to make use of git rebase to clean up our history before pushing final changes. The presentation on it was a bit rushed and perhaps some did not get the full gist out of it. I've decided to write an article on this and go into more details. The content should be generally useful and should give a primer to anyone not familiar with how to use git rebase. You can find the article at: http://vicentiu.ciorbaru.io/keeping-development-history-clean-using-git-reb… Hope it is of use. Regards, Vicențiu

1 0

[Maria-developers] Problem with parallel replication in 10.2
by Michael Widenius 17 Oct '16

17 Oct '16

Hi! I was trying to run a test that fails in the upcoming bb-10.2-jan on the normal 10.2 tree, when I noticed this strange issue: - Test fails with timeout when running with --debug - When looking at the trace file, I notice that we get a duplicate key error for the table gtid_slave_post (MyISAM table). Is this something normal ? To repeat: Store the included test in suite/rpl and run it with: mysql-test-run --debug --record rpl_skr Issue 2: bb-10.2-jan tree is a working tree for a merge of MariaDB 10.2 and MySQL 5.7 When running rpl_skr in 10.2 it takes 2 seconds When running it in the bb-10.2-jan tree it takes either a long time or we get a timeout. This is probably because of the new lock code in lock0lock.cc and lock0wait.cc which doesn't break conflicting transaction but instead waits for a timeout Would appreciate any help with this! Note that rpl_skr is a test that was originally part of rpl_parallel.test, but I have made it separate to be able to more easily test for this issue. Before testing bb-10.2-jan, please apply this patch that fixes one critical issue in this tree related to group-commit: diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index b69468d..0e9caed 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -4976,16 +4953,15 @@ innobase_commit( thd_wakeup_subsequent_commits(thd, 0); /* Now do a write + flush of logs. */ - if (!read_only) { - trx_commit_complete_for_mysql(trx); - } + trx_commit_complete_for_mysql(trx); trx_deregister_from_2pc(trx); } else { /* We just mark the SQL statement ended and do not do a transaction commit */ + DBUG_PRINT("info", ("Just mark SQL statement")); /* If we had reserved the auto-inc lock for some table in this SQL statement we release it now */ Regards, Monty

3 9

Re: [Maria-developers] [MariaDB/server] MDEV-11039 - Add new scheduling algorithm for reducing tail latencies (#245)
by Kristian Nielsen 17 Oct '16

17 Oct '16

Jiamin Huang <notifications(a)github.com> writes: > @sensssz pushed 1 commit. > > 5dc7ad8 Reduce conflict during in-order replication. Cool, that looks nice and simple. Thanks! - Kristian.

1 0

Re: [Maria-developers] MDEV-7145
by Kristian Nielsen 17 Oct '16

17 Oct '16

Phil Sweeney <launchpad(a)sweeneymail.com> writes: > Re: https://jira.mariadb.org/browse/MDEV-7145 > Delayed replication (a feature shipped in MySQL 5.6) has been a MariaDB > feature request since 2014, and has been listed as 'major/red' priority in > Can anyone comment on whether this is still likely to make it into 10.2. I am planning to look into this. - Kristian.

3 4

Re: [Maria-developers] [MariaDB/server] MDEV-11064 - Restrict the speed of reading binlog from Master (#246)
by Kristian Nielsen 15 Oct '16

15 Oct '16

Sergey Vojtovich <notifications(a)github.com> writes: > Similar to other open source projects, the MariaDB Foundation needs to > have shared ownership of all code that is included in the MariaDB > distribution. The easiest way to achieve this is by submitting your > code under the BSD-new license. This is a deliberate lie. There is no such requirement. In fact, most of the code in MariaDB Server is contributed only under the GPLv2, including code from the MariaDB Corporation, Oracle, Galera, Tokutek, ... You already indicated that your code is contributed under the GPLv2 by publishing it on a public github branch here: https://github.com/GCSAdmin/MariaDB/tree/10.2-binlog-speed-limit Please consider keeping your contribution under the GPLv2. Like other open source projects, the MariaDB server needs the strong protection that the GPL gives. You are of course free to use whatever license you choose for your own work. But I cannot be seen as supporting this continued abuse from managers at the MariaDB Corporation, people that will not even stand forward publically. So If you choose to support the BSD/MCA, I regret that I will not be able to look into your pull request. Otherwise, I will be happy to look into it, probably sometimes during the coming week. Thanks, - Kristian.

1 0