November 2015 - developers - lists.mariadb.org

[Maria-developers] On the issue of Seconds_behind_master and Parallel Replication
by Kristian Nielsen 16 Nov '15

16 Nov '15

It was brought to my attention an issue with parallel replication and the Seconds_Behind_Master field of SHOW SLAVE STATUS. I have a possible patch for this, but I wanted to discuss it on the list, as it changes semantics compared to the non-parallel case. Each binlog event contains a timestamp (**) of when the event was created on the master. Whenever the slave SQL thread reads an event from the relay log, it updates the value of Seconds_Behind_Master to the difference between the slave's current time and the event's timestamp. Now in parallel replication, the SQL thread can read a large number of events from the relay log and queue them in-memory for the worker threads. So a small value of Seconds_Behind_Master means only that recent events have been queued - it might still be a long time before the worker threads have had time to actually execute all the queued events. Apparently the problem is (justified) user confusion about this queuing delay not being reflected in Seconds_Behind_Master. The same problem actually exists in the non-parallel case. In case of a large transaction, the Seconds_Behind_Master can be small even though there is still a large amount of execution time remaining for the transaction to complete on the slave. However, in the non-parallel case, at most one transaction can be involved. In the parallel case, the problem is amplified by the potential of thousands of queued transactions awaiting execution. So how to solve it? Attached is a patch that implements one possible solution: the Seconds_Behind_Master is only updated after a transaction commits, with the timestamp of the commit events. This seems more intuitive anyway. But it does introduce a semantic difference between the non-parallel and parallel behaviour for Seconds_Behind_Master. The value will in general be larger on a parallel slave than on a non-parallel slave, for the same actual slave lag. Monty suggested changing the behaviour also for non-parallel mode - letting Seconds_Behind_Master reflect only events actually committed, not just read from the relay log. This would introduce an incompatible behaviour for Seconds_Behind_Master, but could perhaps be done for 10.1, if desired. Doing it in stable 10.0 would be more drastic. So any opinions on this? - Should Seconds_Behind_Master be changed as per above in parallel replication (from 10.0 on)? - If not, any suggestion for another semantics for Seconds_Behind_Master in parallel replication? - If so, should the change to Seconds_Behind_Master also be done in the non-parallel case in 10.1? What about 10.0? - Any comments on the patch? Here is a test case that shows the problem. It simulates a slave catching up with a bunch of INSERT(SLEEP(1)). Without my patch, Seconds_Behind_Master=2 just after starting the parallel slave, while the non-parallel slave shows Seconds_Behind_Master=9. With my patch, the parallel slave shows Seconds_Behind_Master=11. ----------------------------------------------------------------------- --source include/have_innodb.inc --source include/have_binlog_format_statement.inc --let $rpl_topology=1->2 --source include/rpl_init.inc # Test various aspects of parallel replication. --connection server_1 ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; CALL mtr.add_suppression("Unsafe statement written to the binary log using statement format"); CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; --save_master_pos --connection server_2 --sync_with_master --source include/stop_slave.inc SET GLOBAL slave_parallel_threads=1; --connection server_1 --disable_warnings INSERT INTO t1 VALUES (1, SLEEP(1)); INSERT INTO t1 VALUES (2, SLEEP(1)); INSERT INTO t1 VALUES (3, SLEEP(1)); INSERT INTO t1 VALUES (4, SLEEP(1)); INSERT INTO t1 VALUES (5, SLEEP(1)); INSERT INTO t1 VALUES (6, SLEEP(1)); INSERT INTO t1 VALUES (7, SLEEP(1)); INSERT INTO t1 VALUES (8, SLEEP(1)); INSERT INTO t1 VALUES (9, SLEEP(1)); INSERT INTO t1 VALUES (10, SLEEP(1)); --disable_warnings --save_master_pos --connection server_2 --source include/start_slave.inc SELECT SLEEP(1); --let $status_items= Seconds_Behind_Master --source include/show_slave_status.inc --sync_with_master --let $status_items= Seconds_Behind_Master --source include/show_slave_status.inc # Clean up. --connection server_2 --source include/stop_slave.inc SET GLOBAL slave_parallel_threads=DEFAULT; --source include/start_slave.inc --connection server_1 DROP TABLE t1; --source include/rpl_end.inc ----------------------------------------------------------------------- Thanks, - Kristian. ** The full detail is a bit more complex. Some events actually contains the time when the corresponding query _started_ executing, and another value which is the duration of execution. These are added by the slave to get the timestamp for the creation of the event (at the end of query exection). Also, the slave adjusts the timestamp with any time difference between the master and slave clock (eg. time zone for example).

6 10

[Maria-developers] MDEV-8109 unexpected CAST result
by Alexander Barkov 11 Nov '15

11 Nov '15

Hi Serg, Please review my patch for MDEV-8109. Thanks!

2 1

[Maria-developers] Please review MDEV-8140 patch
by chengxiaoz 10 Nov '15

10 Nov '15

Hi kentoku and Sergey, Please review mypatch for MDEV-8140. https://github.com/MariaDB/server/pull/61 Thanks.

3 3

[Maria-developers] Please review MDEV-8140 patch
by chengxiaoz＠189.cn 08 Nov '15

08 Nov '15

1 0

Re: [Maria-developers] [Commits] 7931ca4: MDEV-9024 Build fails with VS2015
by Sergei Golubchik 05 Nov '15

05 Nov '15

Hi, Wlad! Mostly ok, please see below On Nov 03, wlad(a)mariadb.com wrote: > revision-id: 7931ca4f89ad5cd38ea3dc54f1363b7de2fd3414 (mariadb-10.0.22-5-g7931ca4) > parent(s): fa1438cbf4307731a54ea4137d5f7d4b744cdfbc > committer: Vladislav Vaintroub > timestamp: 2015-11-03 11:32:00 +0100 > message: > > MDEV-9024 Build fails with VS2015 > > Fix build failures caused by new C runtime library > - isnan, snprintf, struct timespec are now defined, attempt to redefine them leads > - P_tmpdir, tzname are no more defined > - lfind() and lsearch() in lf_hash.c had to be renamed, declaration > conflicts with some C runtime functions with the same name declared > in a header included by stdlib.h > > Also fix couple of annoying warnings : > - remove #define NOMINMAX from config.h to avoid "redefined" compiler > warnings(NOMINMAX is already in compile flags) > > - disable incremental linker in Debug as well (feature not used much > and compiler crashes often) > > > Also simplify package building with Wix, require Wix 3.9 or later > (VS2015 is not compatible with old Wix 3.5/3.6) > > diff --git a/cmake/os/Windows.cmake b/cmake/os/Windows.cmake > index 4c01bab..fd6e739 100644 > --- a/cmake/os/Windows.cmake > +++ b/cmake/os/Windows.cmake > @@ -181,14 +184,13 @@ CHECK_SYMBOL_REPLACEMENT(S_IROTH _S_IREAD sys/stat.h) > CHECK_SYMBOL_REPLACEMENT(S_IFIFO _S_IFIFO sys/stat.h) > CHECK_SYMBOL_REPLACEMENT(SIGQUIT SIGTERM signal.h) > CHECK_SYMBOL_REPLACEMENT(SIGPIPE SIGINT signal.h) > -CHECK_SYMBOL_REPLACEMENT(isnan _isnan float.h) How can this be wrong? It checks whetherone should use _isnan instead of isnan. That seems like the kind of check that works both with old and new VS. > CHECK_SYMBOL_REPLACEMENT(finite _finite float.h) > diff --git a/config.h.cmake b/config.h.cmake > index 46eed79..a1d505f 100644 > --- a/config.h.cmake > +++ b/config.h.cmake > > -/* We don't want the min/max macros */ > -#ifdef __WIN__ > -#define NOMINMAX > -#endif I'd generally prefer to define symbols in headers, instead of putting everything into ever-growing compiler command line Besides, the minimal fix for the redefinition warning is -#define NOMINMAX +#define NOMINMAX 1 because -DNOMINMAX defines NOMINMAX to be 1, not empty. > /* > MySQL features > */ > diff --git a/mysys/lf_hash.c b/mysys/lf_hash.c > index aa96ca9..fa92971 100644 > --- a/mysys/lf_hash.c > +++ b/mysys/lf_hash.c > @@ -43,7 +43,7 @@ const int LF_HASH_OVERHEAD= sizeof(LF_SLIST); > > /* > a structure to pass the context (pointers two the three successive elements > - in a list) from lfind to linsert/ldelete > + in a list) from my_lfind to my_linsert/my_ldelete > */ > typedef struct { > intptr volatile *prev; > @@ -70,7 +70,7 @@ typedef struct { > cursor is positioned in either case > pins[0..2] are used, they are NOT removed on return > */ > -static int lfind(LF_SLIST * volatile *head, CHARSET_INFO *cs, uint32 hashnr, > +static int my_lfind(LF_SLIST * volatile *head, CHARSET_INFO *cs, uint32 hashnr, er... I don't like that. lfind/lsearch are internal static functions in lf_hash.c, they shouldn't need or use my_ prefix. Two alternatives: 1. #define lfind remap_lfind_for_win #include <my_global.h> #include <my_sys.h> #undef lfind 2. -static int lfind(LF_SLIST * volatile *head, CHARSET_INFO *cs, uint32 hashnr, +static int l_find(LF_SLIST * volatile *head, CHARSET_INFO *cs, uint32 hashnr, > const uchar *key, uint keylen, CURSOR *cursor, LF_PINS *pins) > { > uint32 cur_hashnr; > diff --git a/win/packaging/CMakeLists.txt b/win/packaging/CMakeLists.txt > index 3b9b72f..0535a48 100644 > --- a/win/packaging/CMakeLists.txt > +++ b/win/packaging/CMakeLists.txt > SET(CPACK_WIX_UPGRADE_CODE "49EB7A6A-1CEF-4A1E-9E89-B9A4993963E3") > - SET(CPACK_WIX_PACKAGE_NAME "MariaDB @MAJOR_VERSION@.@MINOR_VERSION@") > + SET(CPACK_WIX_PACKAGE_NAME "MariaDB ${MAJOR_VERSION}.${MINOR_VERSION}") > ELSE() > SET(CPACK_WIX_UPGRADE_CODE "2331E7BD-EE58-431B-9E18-B2B918BCEB1B") > - SET(CPACK_WIX_PACKAGE_NAME "MariaDB @MAJOR_VERSION@.@MINOR_VERSION@ (x64)") > + SET(CPACK_WIX_PACKAGE_NAME "MariaDB ${MAJOR_VERSION}.${MINOR_VERSION} (x64)") Weird. I was sure I've fixed this months ago :-[] Regards, Sergei

1 0

Re: [Maria-developers] [Commits] 01bd373: MDEV-8437 - plugin variables conflict with bootstrap
by Sergei Golubchik 05 Nov '15

05 Nov '15

Hi, Sergey! On Nov 03, Sergey Vojtovich wrote: > revision-id: 01bd3737777f8f0cf2e1737d01a48033a7615b2a (mariadb-10.1.8-23-g01bd373) > parent(s): dc40d737d4af3cb0e272c58507ad54e4eb24be28 > committer: Sergey Vojtovich > timestamp: 2015-11-03 14:02:49 +0400 > message: > > MDEV-8437 - plugin variables conflict with bootstrap ok to push > There is no strong need to change password column: the only side effect is > that 4.0 -> 10.1 upgrades may get root/debian-sys-maint passwords stored in old > format. This should be perfectly acceptable, since all password at this point > are stored in old format. Regards, Sergei

2 3

Re: [Maria-developers] [Commits] 178cddd: MDEV-9082 - Debian: mysql_install_db is called on upgrade
by Sergei Golubchik 05 Nov '15

05 Nov '15

Hi, Sergey! On Nov 05, Sergey Vojtovich wrote: > revision-id: 178cddd664cb4ce2c6ccbe76c33e4fd2c1fc1b3d (mariadb-10.1.8-25-g178cddd) > parent(s): 2ef562d2c74297b0b92c4322697dad98767cc873 > committer: Sergey Vojtovich > timestamp: 2015-11-05 13:57:24 +0400 > message: > > MDEV-9082 - Debian: mysql_install_db is called on upgrade looks ok to me Regards, Sergei

1 0

Re: [Maria-developers] [Commits] 2ef562d: MDEV-9080 - Debian: incorrect empty password check in postinst
by Sergei Golubchik 05 Nov '15

05 Nov '15

Hi, Sergey! ok to push On Nov 04, Sergey Vojtovich wrote: > revision-id: 2ef562d2c74297b0b92c4322697dad98767cc873 (mariadb-10.1.8-24-g2ef562d) > parent(s): 01bd3737777f8f0cf2e1737d01a48033a7615b2a > committer: Sergey Vojtovich > timestamp: 2015-11-04 20:37:16 +0400 > message: > > MDEV-9080 - Debian: incorrect empty password check in postinst > > There was code that was supposed to "catch upgrades from previous versions where > the root password wasn't set". But it is wrong in many regards: > - it is supposed to be executed against running server, but at this point server > should be down, which makes this code no-op > - if the above is fixed, root password will be requested twice (initial root > password request + this one) > - it asks for a password only once, while "initial root password request" asks > twice (password + password verification) > - it may give false positive if unix socket based authentication is in effect > > Removed this code since it didn't work for quite a while (at least since > mysql-5.1) and nobody cared about it. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] dc40d73: MDEV-8437 - plugin variables conflict with bootstrap
by Sergei Golubchik 05 Nov '15

05 Nov '15

Hi, Sergey! On Nov 03, Sergey Vojtovich wrote: > revision-id: dc40d737d4af3cb0e272c58507ad54e4eb24be28 (mariadb-10.1.8-22-gdc40d73) > parent(s): a574407444fc3ea93ca88afa93dc18154251bf74 > committer: Sergey Vojtovich > timestamp: 2015-11-03 12:08:23 +0400 > message: > > MDEV-8437 - plugin variables conflict with bootstrap ok to push > Removed redundant attempt to create mysql.plugin table: > - original code was supposed to INSTALL some plugins: > INSERT INTO plugin VALUES ('innodb', 'ha_innodb.so'), > ('federated', 'ha_federated.so'), ('blackhole', 'ha_blackhole.so'), > ('archive', 'ha_archive.so'); I didn't see any INSERT's in the postinst script. > - original code was supposed to fail if mysql.plugin exists: > The query sequence is supposed to be aborted if the CREATE TABLE fails due > to an already existent table in which case the admin might already have > chosen to remove one or more plugins. > - mysql.plugin must've been created by preceeding mysql_install_db anyway > Regards, Sergei

2 1

Re: [Maria-developers] [Commits] 4685c90: mariadb@.service.in should be configured for all systemd platforms.
by Sergey Vojtovich 05 Nov '15

05 Nov '15

Hi Nirbhay, taking into account that CPackRPM won't be able to install it, what's the purpose of having this configured for all systemd platforms? Regards, Sergey On Tue, Nov 03, 2015 at 05:38:12PM -0500, Nirbhay Choubey wrote: > revision-id: 4685c903bc822596bcf12cbfdb5bcc4460ed0105 > parent(s): 95289e5b665d6a3e37f938d0db978d0d0166e493 > committer: Nirbhay Choubey > branch nick: 10.1 > timestamp: 2015-11-03 17:38:11 -0500 > message: > > mariadb@.service.in should be configured for all systemd platforms. > > --- > support-files/CMakeLists.txt | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/support-files/CMakeLists.txt b/support-files/CMakeLists.txt > index 2ab4350..2929db1 100644 > --- a/support-files/CMakeLists.txt > +++ b/support-files/CMakeLists.txt > @@ -79,9 +79,9 @@ IF(UNIX) > IF(HAVE_SYSTEMD) > CONFIGURE_FILE(mariadb.service.in > ${CMAKE_CURRENT_BINARY_DIR}/mariadb.service @ONLY) > + CONFIGURE_FILE(mariadb@.service.in > + ${CMAKE_CURRENT_BINARY_DIR}/mariadb@.service @ONLY) > IF(NOT RPM) > - CONFIGURE_FILE(mariadb@.service.in > - ${CMAKE_CURRENT_BINARY_DIR}/mariadb@.service @ONLY) > INSTALL(FILES use_galera_new_cluster.conf > ${CMAKE_CURRENT_BINARY_DIR}/mariadb.service > ${CMAKE_CURRENT_BINARY_DIR}/mariadb@.service > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 2