- commits - lists.mariadb.org

[PATCH] MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction
by Kristian Nielsen 02 Sep '24

02 Sep '24

From: Libing Song <anders.slb(a)alibaba-inc.com> Description =========== When a transaction commits, it copies the binlog events from binlog cache to binlog file. Very large transactions (eg. gigabytes) can stall other transactions for a long time because the data is copied while holding LOCK_log, which blocks other commits from binlogging. The solution in this patch is to rename the binlog cache file to a binlog file instead of copy, if the commiting transaction has large binlog cache. Rename is a very fast operation, it doesn't block other transactions a long time. The feature is called `Binlog Free Flush`, The term will be used in the design and code. Design ====== * binlog_free_flush_threshold type: ulonglong scope: global dynamic: yes default: 128MB Only the binlog cache temporary files large than 256MB are renamed to binlog file. * #binlog_cache_files directory To support rename, all binlog cache temporary files are managed as normal files now. `#binlog_cache_files` directory is in the same directory with binlog files. It is created at server startup if it doesn't exist. Otherwise, all files in the directory is deleted at startup. The temporary files are named with ML_ prefix and the memorary address of the binlog_cache_data object which guarantees it is unique. * Reserve space To supprot rename feature, It must reserve enough space at the begin of the binlog cache file. The space is required for Format description, Gtid list, checkpoint and Gtid events when renaming it to a binlog file. Since binlog_cache_data's cache_log is directly accessed by binlog log, online alter and wsrep. It is not easy to update all the code. Thus binlog cache will not reserve space if it is not session binlog cache or wsrep session is enabled. - m_file_reserved_bytes Stores the bytes reserved at the begin of the cache file. It is initialized in write_prepare() and cleared by reset(). The reserved file header is hide to callers. Thus there is no change for callers. E.g. - get_byte_position() still get the length of binlog data written to the cache, but not the file length. - truncate(0) will truncate the file to m_file_reserved_bytes but not 0. - write_prepare() write_prepare() is called everytime when anything is being written into the cache. It will call init_file_reserved_bytes() to create the cache file (if it doesn't exist) and reserve suitable space if the data written exceeds buffer's size. * Binlog_free_flush It is used to encapsulate the code for remaing a binlog cache tempoary file to binlog file. - should_free_flush() it is called by write_transaction_to_binlog_events() to check if a binlog cache should be rename to a binlog file. - commit() That is the entry to rename a binlog cache and commit the transaction. Both rename and commit are protected by LOCK_log, Thus not other transactions can write anything into the renamed binlog before it. Rename happens in a rotation. After the new binlog file is generated, replace_binlog_file() is called to: - copy data from binlog file to binlog cache file. - write gtid event. - rename the binlog cache file to binlog file. After that the rotation will continue to succeed. Then the transaction is committed. The transaction will be committed in a seperated group itself. Its cache file will be detached and cache log will be reset before calling trx_group_commit_leader(). Thus only Xid event be written. --- libmysqld/CMakeLists.txt | 2 +- mysql-test/main/mysqld--help.result | 5 + mysql-test/main/tmp_space_usage.result | 9 +- mysql-test/main/tmp_space_usage.test | 10 +- .../binlog/r/binlog_free_flush_atomic.result | 68 +++ .../binlog/t/binlog_free_flush_atomic.test | 110 +++++ .../encryption/r/binlog_cache_encrypt.result | 18 + .../t/binlog_cache_encrypt-master.opt | 1 + .../encryption/t/binlog_cache_encrypt.test | 19 + .../suite/rpl/r/rpl_binlog_free_flush.result | 117 +++++ .../suite/rpl/t/rpl_binlog_free_flush.test | 217 +++++++++ .../sys_vars/r/sysvars_server_embedded.result | 10 + .../r/sysvars_server_notembedded.result | 10 + sql/CMakeLists.txt | 2 +- sql/log.cc | 432 ++++++++++++++++-- sql/log.h | 21 +- sql/log_cache.cc | 122 +++++ sql/log_cache.h | 116 ++++- sql/log_event.h | 13 + sql/log_event_server.cc | 32 +- sql/mysqld.cc | 4 +- sql/sys_vars.cc | 10 + 22 files changed, 1288 insertions(+), 60 deletions(-) create mode 100644 mysql-test/suite/binlog/r/binlog_free_flush_atomic.result create mode 100644 mysql-test/suite/binlog/t/binlog_free_flush_atomic.test create mode 100644 mysql-test/suite/encryption/r/binlog_cache_encrypt.result create mode 100644 mysql-test/suite/encryption/t/binlog_cache_encrypt-master.opt create mode 100644 mysql-test/suite/encryption/t/binlog_cache_encrypt.test create mode 100644 mysql-test/suite/rpl/r/rpl_binlog_free_flush.result create mode 100644 mysql-test/suite/rpl/t/rpl_binlog_free_flush.test create mode 100644 sql/log_cache.cc diff --git a/libmysqld/CMakeLists.txt b/libmysqld/CMakeLists.txt index 12bcc7ce1aa..f0d837470e2 100644 --- a/libmysqld/CMakeLists.txt +++ b/libmysqld/CMakeLists.txt @@ -67,7 +67,7 @@ SET(SQL_EMBEDDED_SOURCES emb_qcache.cc libmysqld.c lib_sql.cc ../sql/item_subselect.cc ../sql/item_sum.cc ../sql/item_timefunc.cc ../sql/item_xmlfunc.cc ../sql/item_jsonfunc.cc ../sql/json_schema.cc ../sql/json_schema_helper.cc - ../sql/key.cc ../sql/lock.cc ../sql/log.cc + ../sql/key.cc ../sql/lock.cc ../sql/log.cc ../sql/log_cache.cc ../sql/log_event.cc ../sql/log_event_server.cc ../sql/mf_iocache.cc ../sql/my_decimal.cc ../sql/net_serv.cc ../sql/opt_range.cc diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result index a1752a356d9..862382b844d 100644 --- a/mysql-test/main/mysqld--help.result +++ b/mysql-test/main/mysqld--help.result @@ -95,6 +95,10 @@ The following specify which files/extra groups are read (specified before remain statement-based binary logging (smaller binary logs), MIXED for statement-based binary logging when it's safe with fall back to row-based otherwise + --binlog-free-flush-threshold=# + Try to rename the binlog cache temporary file of the + commiting transaction to a binlog file when its binlog + cache size is bigger than the value of this variable --binlog-gtid-index Enable the creation of a GTID index for every binlog file, and the use of such index for speeding up GTID lookup in the binlog @@ -1620,6 +1624,7 @@ binlog-direct-non-transactional-updates FALSE binlog-expire-logs-seconds 0 binlog-file-cache-size 16384 binlog-format MIXED +binlog-free-flush-threshold 134217728 binlog-gtid-index TRUE binlog-gtid-index-page-size 4096 binlog-gtid-index-span-min 65536 diff --git a/mysql-test/main/tmp_space_usage.result b/mysql-test/main/tmp_space_usage.result index ff2f58ab437..242227e0783 100644 --- a/mysql-test/main/tmp_space_usage.result +++ b/mysql-test/main/tmp_space_usage.result @@ -160,16 +160,17 @@ ERROR HY000: Global temporary space limit reached # set @save_max_tmp_total_space_usage=@@global.max_tmp_total_space_usage; set @@global.max_tmp_total_space_usage=64*1024*1024; -set @@max_tmp_session_space_usage=1179648; +set @@max_tmp_session_space_usage=1179648+65536; select @@max_tmp_session_space_usage; @@max_tmp_session_space_usage -1179648 +1245184 set @save_aria_repair_threads=@@aria_repair_threads; set @@aria_repair_threads=2; set @save_max_heap_table_size=@@max_heap_table_size; set @@max_heap_table_size=16777216; CREATE TABLE t1 (a CHAR(255),b INT,INDEX (b)); INSERT INTO t1 SELECT SEQ,SEQ FROM seq_1_to_100000; +set @@max_tmp_session_space_usage=1179648; SELECT * FROM t1 UNION SELECT * FROM t1; ERROR HY000: Local temporary space limit reached DROP TABLE t1; @@ -205,11 +206,13 @@ ERROR HY000: Local temporary space limit reached # connect c1, localhost, root,,; set @@binlog_format=row; -CREATE OR REPLACE TABLE t1 (a DATETIME) ENGINE=MyISAM; +CREATE OR REPLACE TABLE t1 (a DATETIME) ENGINE=InnoDB; +BEGIN; INSERT INTO t1 SELECT NOW() FROM seq_1_to_6000; SET max_tmp_session_space_usage = 64*1024; SELECT * FROM information_schema.ALL_PLUGINS LIMIT 2; ERROR HY000: Local temporary space limit reached +ROLLBACK; drop table t1; connection default; disconnect c1; diff --git a/mysql-test/main/tmp_space_usage.test b/mysql-test/main/tmp_space_usage.test index af7b295f343..1685dbbc450 100644 --- a/mysql-test/main/tmp_space_usage.test +++ b/mysql-test/main/tmp_space_usage.test @@ -215,7 +215,8 @@ select count(distinct concat(seq,repeat('x',1000))) from seq_1_to_1000; set @save_max_tmp_total_space_usage=@@global.max_tmp_total_space_usage; set @@global.max_tmp_total_space_usage=64*1024*1024; -set @@max_tmp_session_space_usage=1179648; +# Binlog cache reserve 4096 bytes at the begin of the temporary file. +set @@max_tmp_session_space_usage=1179648+65536; select @@max_tmp_session_space_usage; set @save_aria_repair_threads=@@aria_repair_threads; set @@aria_repair_threads=2; @@ -224,6 +225,7 @@ set @@max_heap_table_size=16777216; CREATE TABLE t1 (a CHAR(255),b INT,INDEX (b)); INSERT INTO t1 SELECT SEQ,SEQ FROM seq_1_to_100000; +set @@max_tmp_session_space_usage=1179648; --error 200 SELECT * FROM t1 UNION SELECT * FROM t1; DROP TABLE t1; @@ -266,11 +268,15 @@ SELECT MIN(VARIABLE_VALUE) OVER (), NTILE(1) OVER (), MAX(VARIABLE_NAME) OVER () connect(c1, localhost, root,,); set @@binlog_format=row; -CREATE OR REPLACE TABLE t1 (a DATETIME) ENGINE=MyISAM; +CREATE OR REPLACE TABLE t1 (a DATETIME) ENGINE=InnoDB; +# Use the transaction to keep binlog cache temporary file large enough +BEGIN; INSERT INTO t1 SELECT NOW() FROM seq_1_to_6000; + SET max_tmp_session_space_usage = 64*1024; --error 200 SELECT * FROM information_schema.ALL_PLUGINS LIMIT 2; +ROLLBACK; drop table t1; connection default; disconnect c1; diff --git a/mysql-test/suite/binlog/r/binlog_free_flush_atomic.result b/mysql-test/suite/binlog/r/binlog_free_flush_atomic.result new file mode 100644 index 00000000000..31175d32581 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_free_flush_atomic.result @@ -0,0 +1,68 @@ +RESET MASTER; +# +# binlog cache file is created in #binlog_cache_files directory +# and it is deleted at disconnect +# +connect con1,localhost,root,,; +CREATE TABLE t1 (c1 LONGTEXT) ENGINE = InnoDB; +# list binlog_cache_files/ +INSERT INTO t1 values(repeat("1", 5242880)); +INSERT INTO t1 values(repeat("1", 5242880)); +FLUSH BINARY LOGS; +# list #binlog_cache_files/ +ML_BINLOG_CACHE_FILE +SET debug_sync = "thread_end SIGNAL signal.thread_end"; +disconnect con1; +connection default; +SET debug_sync = "now WAIT_FOR signal.thread_end"; +# binlog cache file is deleted at disconnection +# list #binlog_cache_files/ +# +# Reserved space is not big enough, rename will not happen. But rotate +# will succeed. +# +SET GLOBAL binlog_free_flush_threshold = 10 * 1024 * 1024; +SET debug = 'd,simulate_required_size_too_big'; +UPDATE t1 SET c1 = repeat('2', 5242880); +include/assert.inc [Binlog is rotated, but free flush is not executed.] +# +# Error happens when renaming binlog cache to binlog file, rename will +# not happen. Since the original binlog is delete, the rotate will failed +# too. binlog will be closed. +# +SET debug = 'd,simulate_rename_binlog_cache_to_binlog_error'; +UPDATE t1 SET c1 = repeat('3', 5242880); +ERROR HY000: Can't open file: './master-bin.000004' (errno: 1 "Operation not permitted") +SELECT count(*) FROM t1 WHERE c1 like "3%"; +count(*) +2 +# Binlog is closed +show master status; +File Position Binlog_Do_DB Binlog_Ignore_DB +# restart +show master status; +File Position Binlog_Do_DB Binlog_Ignore_DB +master-bin.000004 # <Binlog_Do_DB> <Binlog_Ignore_DB> +# +# Crash happens before rename the file +# +SET GLOBAL binlog_free_flush_threshold = 10 * 1024 * 1024; +SET debug = 'd,binlog_free_flush_crash_before_rename'; +UPDATE t1 SET c1 = repeat('4', 5242880); +ERROR HY000: Lost connection to server during query +# One cache file left afte crash +# list #binlog_cache_files/ +ML_BINLOG_CACHE_FILE +non_binlog_cache +# restart +# The cache file is deleted at startup. +# list #binlog_cache_files/ +non_binlog_cache +include/assert_grep.inc [warning: non_binlog_cache file is in #binlog_cache_files/] +include/show_binlog_events.inc +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000005 # Format_desc # # SERVER_VERSION, BINLOG_VERSION +master-bin.000005 # Gtid_list # # [#-#-#] +call mtr.add_suppression(".*Turning logging off for the whole duration.*"); +call mtr.add_suppression(".*non_binlog_cache is in #binlog_cache_files/.*"); +DROP TABLE t1; diff --git a/mysql-test/suite/binlog/t/binlog_free_flush_atomic.test b/mysql-test/suite/binlog/t/binlog_free_flush_atomic.test new file mode 100644 index 00000000000..05b4792c314 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_free_flush_atomic.test @@ -0,0 +1,110 @@ +################################################################################ +# MDEV-32014 Rename binlog cache to binlog file +# +# It verifies that the rename logic is handled correct if error happens. +################################################################################ +--source include/have_binlog_format_row.inc +--source include/have_innodb.inc +--source include/have_debug.inc +RESET MASTER; + +--echo # +--echo # binlog cache file is created in #binlog_cache_files directory +--echo # and it is deleted at disconnect +--echo # +--connect(con1,localhost,root,,) +CREATE TABLE t1 (c1 LONGTEXT) ENGINE = InnoDB; + +--echo # list binlog_cache_files/ +--let $datadir= `SELECT @@datadir` +--list_files $datadir/#binlog_cache_files + +INSERT INTO t1 values(repeat("1", 5242880)); +INSERT INTO t1 values(repeat("1", 5242880)); +FLUSH BINARY LOGS; + +--echo # list #binlog_cache_files/ +--replace_regex /ML_[0-9]+/ML_BINLOG_CACHE_FILE/ +--list_files $datadir/#binlog_cache_files + +SET debug_sync = "thread_end SIGNAL signal.thread_end"; +--disconnect con1 +--connection default +# Wait until the connection is closed completely. +SET debug_sync = "now WAIT_FOR signal.thread_end"; + +--echo # binlog cache file is deleted at disconnection +--echo # list #binlog_cache_files/ +--list_files $datadir/#binlog_cache_files + +--echo # +--echo # Reserved space is not big enough, rename will not happen. But rotate +--echo # will succeed. +--echo # +SET GLOBAL binlog_free_flush_threshold = 10 * 1024 * 1024; +SET debug = 'd,simulate_required_size_too_big'; +UPDATE t1 SET c1 = repeat('2', 5242880); + +--let $gtid_end_pos= query_get_value(SHOW BINLOG EVENTS IN 'master-bin.000002' LIMIT 4, End_log_pos, 4) +--let $assert_cond= $gtid_end_pos < 4096 +--let $assert_text= Binlog is rotated, but free flush is not executed. +--source include/assert.inc + +--echo # +--echo # Error happens when renaming binlog cache to binlog file, rename will +--echo # not happen. Since the original binlog is delete, the rotate will failed +--echo # too. binlog will be closed. +--echo # +SET debug = 'd,simulate_rename_binlog_cache_to_binlog_error'; +--error ER_CANT_OPEN_FILE +UPDATE t1 SET c1 = repeat('3', 5242880); +SELECT count(*) FROM t1 WHERE c1 like "3%"; + +--echo # Binlog is closed +--source include/show_master_status.inc + +--source include/restart_mysqld.inc +--source include/show_master_status.inc + +--echo # +--echo # Crash happens before rename the file +--echo # +SET GLOBAL binlog_free_flush_threshold = 10 * 1024 * 1024; + +SET debug = 'd,binlog_free_flush_crash_before_rename'; +--source include/expect_crash.inc +--error 2013 +UPDATE t1 SET c1 = repeat('4', 5242880); + +--write_file $datadir/#binlog_cache_files/non_binlog_cache +It is not a binlog cache file +EOF + +--echo # One cache file left afte crash +--echo # list #binlog_cache_files/ +--replace_regex /ML_[0-9]+/ML_BINLOG_CACHE_FILE/ +--list_files $datadir/#binlog_cache_files + +--source include/start_mysqld.inc +--echo # The cache file is deleted at startup. +--echo # list #binlog_cache_files/ +--list_files $datadir/#binlog_cache_files + +--let $assert_text= warning: non_binlog_cache file is in #binlog_cache_files/ +--let $assert_file= $MYSQLTEST_VARDIR/log/mysqld.1.err +--let $assert_select= non_binlog_cache.*#binlog_cache_files/ +--let $assert_count= 1 +--let $assert_only_after= CURRENT_TEST: binlog.binlog_free_flush_atomic +--source include/assert_grep.inc + +--remove_file $datadir/#binlog_cache_files/non_binlog_cache + +--let $binlog_file= LAST +--let $binlog_start= 4 +--let $skip_checkpoint_events= 1 +--source include/show_binlog_events.inc + +call mtr.add_suppression(".*Turning logging off for the whole duration.*"); +call mtr.add_suppression(".*non_binlog_cache is in #binlog_cache_files/.*"); +DROP TABLE t1; + diff --git a/mysql-test/suite/encryption/r/binlog_cache_encrypt.result b/mysql-test/suite/encryption/r/binlog_cache_encrypt.result new file mode 100644 index 00000000000..a479a39cbcf --- /dev/null +++ b/mysql-test/suite/encryption/r/binlog_cache_encrypt.result @@ -0,0 +1,18 @@ +RESET MASTER; +CREATE TABLE t1 (c1 LONGTEXT) ENGINE = InnoDB; +INSERT INTO t1 values(repeat("1", 5242880)); +INSERT INTO t1 values(repeat("1", 5242880)); +FLUSH BINARY LOGS; +SET @saved_threshold= @@GLOBAL.binlog_free_flush_threshold; +SET GLOBAL binlog_free_flush_threshold = 10 * 1024 * 1024; +UPDATE t1 SET c1 = repeat('2', 5242880); +include/show_binlog_events.inc +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000002 # Gtid # # BEGIN GTID #-#-# +master-bin.000002 # Annotate_rows # # UPDATE t1 SET c1 = repeat('2', 5242880) +master-bin.000002 # Table_map # # table_id: # (test.t1) +master-bin.000002 # Update_rows_v1 # # table_id: # +master-bin.000002 # Update_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000002 # Xid # # COMMIT /* XID */ +SET GLOBAL binlog_free_flush_threshold = @saved_threshold; +DROP TABLE t1; diff --git a/mysql-test/suite/encryption/t/binlog_cache_encrypt-master.opt b/mysql-test/suite/encryption/t/binlog_cache_encrypt-master.opt new file mode 100644 index 00000000000..469148de64a --- /dev/null +++ b/mysql-test/suite/encryption/t/binlog_cache_encrypt-master.opt @@ -0,0 +1 @@ +--encrypt-tmp-files=on diff --git a/mysql-test/suite/encryption/t/binlog_cache_encrypt.test b/mysql-test/suite/encryption/t/binlog_cache_encrypt.test new file mode 100644 index 00000000000..93725ce653e --- /dev/null +++ b/mysql-test/suite/encryption/t/binlog_cache_encrypt.test @@ -0,0 +1,19 @@ +--source include/have_file_key_management_plugin.inc +--source include/have_binlog_format_row.inc +--source include/have_innodb.inc +RESET MASTER; +CREATE TABLE t1 (c1 LONGTEXT) ENGINE = InnoDB; +INSERT INTO t1 values(repeat("1", 5242880)); +INSERT INTO t1 values(repeat("1", 5242880)); +FLUSH BINARY LOGS; + +SET @saved_threshold= @@GLOBAL.binlog_free_flush_threshold; +SET GLOBAL binlog_free_flush_threshold = 10 * 1024 * 1024; +UPDATE t1 SET c1 = repeat('2', 5242880); + +--let $binlog_file= LAST +--let $skip_checkpoint_events=1 +--source include/show_binlog_events.inc + +SET GLOBAL binlog_free_flush_threshold = @saved_threshold; +DROP TABLE t1; diff --git a/mysql-test/suite/rpl/r/rpl_binlog_free_flush.result b/mysql-test/suite/rpl/r/rpl_binlog_free_flush.result new file mode 100644 index 00000000000..a216e252dd8 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_binlog_free_flush.result @@ -0,0 +1,117 @@ +include/master-slave.inc +[connection master] +# Prepare +SET @saved_threshold= @@GLOBAL.binlog_free_flush_threshold; +SET @saved_checksum= @@GLOBAL.binlog_checksum; +SET GLOBAL binlog_checksum = "NONE"; +CREATE TABLE t1 (c1 LONGTEXT) ENGINE = InnoDB; +CREATE TABLE t2 (c1 LONGTEXT) ENGINE = MyISAM; +INSERT INTO t1 values(repeat("1", 5242880)); +INSERT INTO t1 values(repeat("1", 5242880)); +INSERT INTO t2 values(repeat("1", 5242880)); +INSERT INTO t2 values(repeat("1", 5242880)); +FLUSH BINARY LOGS; +# Not renamed to binlog, since the binlog cache is not larger than the +# threshold. And it should works well after ROLLBACK TO SAVEPOINT +BEGIN; +SAVEPOINT s1; +UPDATE t1 SET c1 = repeat('1', 5242880); +ROLLBACK TO SAVEPOINT s1; +UPDATE t1 SET c1 = repeat('2', 5242880); +SAVEPOINT s2; +UPDATE t1 SET c1 = repeat('3', 5242880); +UPDATE t1 SET c1 = repeat('4', 5242880); +ROLLBACK TO SAVEPOINT s2; +COMMIT; +include/assert.inc [Binlog is not rotated] +# +# Test binlog cache rename to binlog file with checksum off +# +SET GLOBAL binlog_free_flush_threshold = 10 * 1024 * 1024; +# Transaction cache can be renamed and works well with ROLLBACK TO SAVEPOINT +BEGIN; +SAVEPOINT s1; +UPDATE t1 SET c1 = repeat('2', 5242880); +ROLLBACK TO s1; +UPDATE t1 SET c1 = repeat('3', 5242880); +SAVEPOINT s2; +UPDATE t1 SET c1 = repeat('4', 5242880); +UPDATE t1 SET c1 = repeat('5', 5242880); +UPDATE t1 SET c1 = repeat('6', 5242880); +ROLLBACK TO SAVEPOINT s2; +COMMIT; +INSERT INTO t1 VALUES("after_update"); +include/assert.inc [Free flush is executed.] +# statement cache can be renamed +BEGIN; +UPDATE t2 SET c1 = repeat('4', 5242880); +INSERT INTO t1 VALUES("after_update"); +COMMIT; +include/assert.inc [Free flush is executed.] +# CREATE SELECT works well +CREATE TABLE t3 SELECT * FROM t1; +include/assert.inc [Free flush is executed.] +CREATE TABLE t4 SELECT * FROM t2; +include/assert.inc [Free flush is executed.] +# XA statement works well +XA START "test-a-long-xid========================================"; +UPDATE t1 SET c1 = repeat('1', 5242880); +XA END "test-a-long-xid========================================"; +XA PREPARE "test-a-long-xid========================================"; +XA COMMIT "test-a-long-xid========================================"; +include/assert.inc [Free flush is executed.] +XA START "test-xid"; +UPDATE t1 SET c1 = repeat('2', 5242880); +XA END "test-xid"; +XA COMMIT "test-xid" ONE PHASE; +include/assert.inc [Free flush is executed.] +# +# It works well in the situation that binlog header is larger than +# IO_SIZE and binlog file's buffer. +# +FLUSH BINARY LOGS; +SET SESSION server_id = 1; +UPDATE t1 SET c1 = repeat('3', 5242880); +include/assert.inc [Free flush is executed.] +# +# RESET MASTER should work well. It also verifies binlog checksum mechanism. +# +include/rpl_reset.inc +# +# Test binlog cache rename to binlog file with checksum on +# +SET GLOBAL binlog_checksum = "CRC32"; +# It will not rename the cache to file, since the cache's checksum was +# initialized when reset the cache at the end of previous transaction. +UPDATE t1 SET c1 = repeat('5', 5242880); +include/assert.inc [Binlog is not rotated] +# +# Not rename to binlog file If the cache's checksum is not same +# to binlog_checksum +# +BEGIN; +UPDATE t1 SET c1 = repeat('6', 5242880); +SET GLOBAL binlog_checksum = "NONE"; +COMMIT; +include/assert.inc [Binlog is not rotated] +BEGIN; +UPDATE t1 SET c1 = repeat('7', 5242880); +SET GLOBAL binlog_checksum = "CRC32"; +COMMIT; +include/assert.inc [Binlog is not rotated] +# +# Not rename to binlog file If both stmt and trx cache are not empty +# +UPDATE t1, t2 SET t1.c1 = repeat('8', 5242880), t2.c1 = repeat('7', 5242880); +include/assert.inc [Binlog is not rotated] +# +# Not rename to binlog file If binlog_legacy_event_pos is on +# +SET GLOBAL binlog_legacy_event_pos = ON; +UPDATE t1 SET c1 = repeat('9', 5242880); +SET GLOBAL binlog_legacy_event_pos = OFF; +include/assert.inc [Binlog is not rotated] +DROP TABLE t1, t2, t3, t4; +SET GLOBAL binlog_free_flush_threshold = @saved_threshold; +SET GLOBAL binlog_checksum = @saved_checksum; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_binlog_free_flush.test b/mysql-test/suite/rpl/t/rpl_binlog_free_flush.test new file mode 100644 index 00000000000..5152779dd48 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_binlog_free_flush.test @@ -0,0 +1,217 @@ +################################################################################ +# MDEV-32014 Rename binlog cache to binlog file +# +# It verifies that the binlog caches which are larger +# than binlog_free_flush_threshold can be move to a binlog file +# successfully. With a successful rename, +# - it rotates the binlog and the cache is renamed to the new binlog file +# - an ignorable event is generated just after the Gtid_log_event of the +# transaction to take the reserved spaces which is unused. +# +# It also verifies that rename is not supported in below cases +# though the cache is larger than the threshold +# - both statement and transaction cache should be flushed. +# - the cache's checksum option is not same to binlog_checksum +# - binlog_legacy_event_pos is enabled. +################################################################################ +--source include/master-slave.inc +--source include/have_binlog_format_row.inc +--source include/have_innodb.inc + +--echo # Prepare +SET @saved_threshold= @@GLOBAL.binlog_free_flush_threshold; +SET @saved_checksum= @@GLOBAL.binlog_checksum; + +SET GLOBAL binlog_checksum = "NONE"; + +CREATE TABLE t1 (c1 LONGTEXT) ENGINE = InnoDB; +CREATE TABLE t2 (c1 LONGTEXT) ENGINE = MyISAM; + +INSERT INTO t1 values(repeat("1", 5242880)); +INSERT INTO t1 values(repeat("1", 5242880)); +INSERT INTO t2 values(repeat("1", 5242880)); +INSERT INTO t2 values(repeat("1", 5242880)); + +FLUSH BINARY LOGS; + +--echo # Not renamed to binlog, since the binlog cache is not larger than the +--echo # threshold. And it should works well after ROLLBACK TO SAVEPOINT +BEGIN; +SAVEPOINT s1; +UPDATE t1 SET c1 = repeat('1', 5242880); +ROLLBACK TO SAVEPOINT s1; +UPDATE t1 SET c1 = repeat('2', 5242880); +SAVEPOINT s2; +UPDATE t1 SET c1 = repeat('3', 5242880); +UPDATE t1 SET c1 = repeat('4', 5242880); +ROLLBACK TO SAVEPOINT s2; +COMMIT; + +--let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1) +--let $assert_cond= "$binlog_file" = "master-bin.000003" +--let $assert_text= Binlog is not rotated +--source include/assert.inc + +--echo # +--echo # Test binlog cache rename to binlog file with checksum off +--echo # +SET GLOBAL binlog_free_flush_threshold = 10 * 1024 * 1024; + +--echo # Transaction cache can be renamed and works well with ROLLBACK TO SAVEPOINT +BEGIN; +SAVEPOINT s1; +UPDATE t1 SET c1 = repeat('2', 5242880); +ROLLBACK TO s1; +UPDATE t1 SET c1 = repeat('3', 5242880); +SAVEPOINT s2; +UPDATE t1 SET c1 = repeat('4', 5242880); +UPDATE t1 SET c1 = repeat('5', 5242880); +UPDATE t1 SET c1 = repeat('6', 5242880); +ROLLBACK TO SAVEPOINT s2; +COMMIT; +INSERT INTO t1 VALUES("after_update"); + +--let $gtid_end_pos= query_get_value(SHOW BINLOG EVENTS IN 'master-bin.000004' LIMIT 4, End_log_pos, 4) +--let $assert_cond= $gtid_end_pos = 4096 +--let $assert_text= Free flush is executed. +--source include/assert.inc + +--echo # statement cache can be renamed +BEGIN; +UPDATE t2 SET c1 = repeat('4', 5242880); +INSERT INTO t1 VALUES("after_update"); +COMMIT; +--let $gtid_end_pos= query_get_value(SHOW BINLOG EVENTS IN 'master-bin.000005' LIMIT 4, End_log_pos, 4) +--let $assert_cond= $gtid_end_pos = 4096 +--let $assert_text= Free flush is executed. +--source include/assert.inc + +--echo # CREATE SELECT works well +CREATE TABLE t3 SELECT * FROM t1; +--let $gtid_end_pos= query_get_value(SHOW BINLOG EVENTS IN 'master-bin.000006' LIMIT 4, End_log_pos, 4) +--let $assert_cond= $gtid_end_pos = 4096 +--let $assert_text= Free flush is executed. +--source include/assert.inc + +CREATE TABLE t4 SELECT * FROM t2; +--let $gtid_end_pos= query_get_value(SHOW BINLOG EVENTS IN 'master-bin.000007' LIMIT 4, End_log_pos, 4) +--let $assert_cond= $gtid_end_pos = 4096 +--let $assert_text= Free flush is executed. +--source include/assert.inc + +--echo # XA statement works well +XA START "test-a-long-xid========================================"; +UPDATE t1 SET c1 = repeat('1', 5242880); +XA END "test-a-long-xid========================================"; +XA PREPARE "test-a-long-xid========================================"; +XA COMMIT "test-a-long-xid========================================"; +--let $gtid_end_pos= query_get_value(SHOW BINLOG EVENTS IN 'master-bin.000008' LIMIT 4, End_log_pos, 4) +--let $assert_cond= $gtid_end_pos = 4096 +--let $assert_text= Free flush is executed. +--source include/assert.inc + +XA START "test-xid"; +UPDATE t1 SET c1 = repeat('2', 5242880); +XA END "test-xid"; +XA COMMIT "test-xid" ONE PHASE; +--let $gtid_end_pos= query_get_value(SHOW BINLOG EVENTS IN 'master-bin.000009' LIMIT 4, End_log_pos, 4) +--let $assert_cond= $gtid_end_pos = 4096 +--let $assert_text= Free flush is executed. +--source include/assert.inc + +--echo # +--echo # It works well in the situation that binlog header is larger than +--echo # IO_SIZE and binlog file's buffer. +--echo # +--disable_query_log + +# make Gtid_list_event larger than 64K(binlog file's buffer) +--let $server_id= 100000 +while ($server_id < 104096) +{ + eval SET SESSION server_id = $server_id; + eval UPDATE t1 SET c1 = "$server_id" LIMIT 1; + --inc $server_id +} + +--enable_query_log + +# After flush, reserved space should be updated. +FLUSH BINARY LOGS; + +SET SESSION server_id = 1; +UPDATE t1 SET c1 = repeat('3', 5242880); + +--let $gtid_end_pos= query_get_value(SHOW BINLOG EVENTS IN 'master-bin.000011' LIMIT 4, End_log_pos, 4) +# 69632 is 65K which is larger, binlog's buffer is 64K +--let $assert_cond= $gtid_end_pos = 69632 +--let $assert_text= Free flush is executed. +--source include/assert.inc + +--echo # +--echo # RESET MASTER should work well. It also verifies binlog checksum mechanism. +--echo # +--source include/rpl_reset.inc + +--echo # +--echo # Test binlog cache rename to binlog file with checksum on +--echo # +SET GLOBAL binlog_checksum = "CRC32"; + +--echo # It will not rename the cache to file, since the cache's checksum was +--echo # initialized when reset the cache at the end of previous transaction. +UPDATE t1 SET c1 = repeat('5', 5242880); +--let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1) +--let $assert_cond= "$binlog_file" = "master-bin.000002" +--let $assert_text= Binlog is not rotated +--source include/assert.inc + +--echo # +--echo # Not rename to binlog file If the cache's checksum is not same +--echo # to binlog_checksum +--echo # +BEGIN; +UPDATE t1 SET c1 = repeat('6', 5242880); +SET GLOBAL binlog_checksum = "NONE"; +COMMIT; +--let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1) +--let $assert_cond= "$binlog_file" = "master-bin.000003" +--let $assert_text= Binlog is not rotated +--source include/assert.inc + +BEGIN; +UPDATE t1 SET c1 = repeat('7', 5242880); +SET GLOBAL binlog_checksum = "CRC32"; +COMMIT; +--let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1) +--let $assert_cond= "$binlog_file" = "master-bin.000004" +--let $assert_text= Binlog is not rotated +--source include/assert.inc + +--echo # +--echo # Not rename to binlog file If both stmt and trx cache are not empty +--echo # +UPDATE t1, t2 SET t1.c1 = repeat('8', 5242880), t2.c1 = repeat('7', 5242880); +--let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1) +--let $assert_cond= "$binlog_file" = "master-bin.000004" +--let $assert_text= Binlog is not rotated +--source include/assert.inc + +--echo # +--echo # Not rename to binlog file If binlog_legacy_event_pos is on +--echo # +SET GLOBAL binlog_legacy_event_pos = ON; +UPDATE t1 SET c1 = repeat('9', 5242880); +SET GLOBAL binlog_legacy_event_pos = OFF; +--let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1) +--let $assert_cond= "$binlog_file" = "master-bin.000004" +--let $assert_text= Binlog is not rotated +--source include/assert.inc + +# cleanup +DROP TABLE t1, t2, t3, t4; +SET GLOBAL binlog_free_flush_threshold = @saved_threshold; +SET GLOBAL binlog_checksum = @saved_checksum; +--let $binlog_file= +--let $skip_checkpoint_events=0 +--source include/rpl_end.inc diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result index 7a113a02b02..8771e4ac87f 100644 --- a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result +++ b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result @@ -432,6 +432,16 @@ NUMERIC_BLOCK_SIZE NULL ENUM_VALUE_LIST MIXED,STATEMENT,ROW READ_ONLY NO COMMAND_LINE_ARGUMENT REQUIRED +VARIABLE_NAME BINLOG_FREE_FLUSH_THRESHOLD +VARIABLE_SCOPE GLOBAL +VARIABLE_TYPE BIGINT UNSIGNED +VARIABLE_COMMENT Try to rename the binlog cache temporary file of the commiting transaction to a binlog file when its binlog cache size is bigger than the value of this variable +NUMERIC_MIN_VALUE 10485760 +NUMERIC_MAX_VALUE 18446744073709551615 +NUMERIC_BLOCK_SIZE 1 +ENUM_VALUE_LIST NULL +READ_ONLY NO +COMMAND_LINE_ARGUMENT REQUIRED VARIABLE_NAME BINLOG_GTID_INDEX VARIABLE_SCOPE GLOBAL VARIABLE_TYPE BOOLEAN diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result index 16d77d397d4..ceb141cfc7a 100644 --- a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result +++ b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result @@ -452,6 +452,16 @@ NUMERIC_BLOCK_SIZE NULL ENUM_VALUE_LIST MIXED,STATEMENT,ROW READ_ONLY NO COMMAND_LINE_ARGUMENT REQUIRED +VARIABLE_NAME BINLOG_FREE_FLUSH_THRESHOLD +VARIABLE_SCOPE GLOBAL +VARIABLE_TYPE BIGINT UNSIGNED +VARIABLE_COMMENT Try to rename the binlog cache temporary file of the commiting transaction to a binlog file when its binlog cache size is bigger than the value of this variable +NUMERIC_MIN_VALUE 10485760 +NUMERIC_MAX_VALUE 18446744073709551615 +NUMERIC_BLOCK_SIZE 1 +ENUM_VALUE_LIST NULL +READ_ONLY NO +COMMAND_LINE_ARGUMENT REQUIRED VARIABLE_NAME BINLOG_GTID_INDEX VARIABLE_SCOPE GLOBAL VARIABLE_TYPE BOOLEAN diff --git a/sql/CMakeLists.txt b/sql/CMakeLists.txt index 0195555efaf..050cd54d98c 100644 --- a/sql/CMakeLists.txt +++ b/sql/CMakeLists.txt @@ -107,7 +107,7 @@ SET (SQL_SOURCE hostname.cc init.cc item.cc item_buff.cc item_cmpfunc.cc item_create.cc item_func.cc item_geofunc.cc item_row.cc item_strfunc.cc item_subselect.cc item_sum.cc item_timefunc.cc - key.cc log.cc lock.cc + key.cc log.cc log_cache.cc lock.cc log_event.cc log_event_server.cc rpl_record.cc rpl_reporting.cc mf_iocache.cc my_decimal.cc diff --git a/sql/log.cc b/sql/log.cc index 34f9ad745fc..3dc57b21c05 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -163,6 +163,111 @@ static SHOW_VAR binlog_status_vars_detail[]= {NullS, NullS, SHOW_LONG} }; +/** + This class implementes the feature to rename a binlog cache temporary file to + a binlog file. It is used to avoid holding LOCK_log long time when writting a + huge binlog cache to binlog file. + + With this feature, temporary files of binlog caches will be created in + BINLOG_CACHE_DIR which is created in the same directory to binlog files + at server startup. +*/ +class Binlog_free_flush +{ +public: + Binlog_free_flush() {} + + /** + Check whether free flush should be executed on the cache_data. + + @param group_commit_entry object of current transaction + + @retval true it should do free flush + @retval false it should do normal commit. + */ + bool should_free_flush(const MYSQL_BIN_LOG::group_commit_entry *entry) const; + + /** + This function is the entry function to do free flush. It first, + rotate the binlog, then rename the temporary file of the + binlog cache to new binlog file, after that it commits the transaction. + + @param entry, group_commit_entry object of current transaction. + + @retval true free flush succeeds. + @retval false free flush fails, it should go to normal commit process. + */ + bool commit(MYSQL_BIN_LOG::group_commit_entry *entry); + + /** + After rotate has created the new binlog file, it copies the content + of the new binlog file to the binlog cache, delete the new binlog file + and then rename the binlog cache to the new binlog file. + + @retval true Succeeds to replace the binlog file. + @retval false Failed to replace the binlog file. It only return + true if some error happened after the new binlog file + is deleted. In this situation rotate process will fail. + */ + bool replace_binlog_file(); + /** + The space left is more than a gtid event required, thus the extra + space is padded into the gtid event as 0. This function is used + to calculate the real gtid size with pad. + */ + size_t get_gtid_event_pad_size(); + + /** + The space required for session binlog caches to reserve. It is calculated + from the length of current binlog file when it is generated and aligned + to IO_SIZE; + + @param header_len header length of current binlog file. + */ + void set_reserved_bytes(uint32 header_len) + { + // Gtid event length + header_len+= LOG_EVENT_HEADER_LEN + Gtid_log_event::max_data_length + + BINLOG_CHECKSUM_LEN; + header_len= header_len - (header_len % IO_SIZE) + IO_SIZE; + if (header_len != m_reserved_bytes) + m_reserved_bytes= header_len; + } + + /** + Return reserved space required for binlog cache. It is NOT defined as + an atomic variable, while it is get and set in parallel. Synchronizing + between set and get is not really necessary. + */ + uint32 get_reserved_size() + { + return m_reserved_bytes; + } +private: + Binlog_free_flush &operator=(const Binlog_free_flush &); + Binlog_free_flush(const Binlog_free_flush &); + + char m_cache_dir[FN_REFLEN]; + + /** The commit entry of current transaction which is doing free flush. */ + MYSQL_BIN_LOG::group_commit_entry *m_entry{nullptr}; + + /** The cache_data which will be renamed to binlog. */ + binlog_cache_data *m_cache_data{nullptr}; + + /** It will be set to true if rename operation succeeds */ + bool m_replaced{false}; + + uint32 m_reserved_bytes {IO_SIZE}; +}; +static Binlog_free_flush binlog_free_flush; +ulonglong opt_binlog_free_flush_threshold= 10 * 1024 * 1024; + +uint32 binlog_cache_reserved_size() +{ + return binlog_free_flush.get_reserved_size(); +} + /* Variables for the binlog background thread. Protected by the MYSQL_BIN_LOG::LOCK_binlog_background_thread mutex. @@ -3761,7 +3866,8 @@ bool MYSQL_BIN_LOG::open(const char *log_name, enum cache_type io_cache_type_arg, ulong max_size_arg, bool null_created_arg, - bool need_mutex) + bool need_mutex, + bool is_free_flush) { xid_count_per_binlog *new_xid_list_entry= NULL, *b; DBUG_ENTER("MYSQL_BIN_LOG::open"); @@ -4027,14 +4133,20 @@ bool MYSQL_BIN_LOG::open(const char *log_name, goto err; bytes_written+= description_event_for_queue->data_written; } + + // offset must be saved before replace_binlog_file(), it will update the pos + my_off_t offset= my_b_tell(&log_file); + + if (is_free_flush && binlog_free_flush.replace_binlog_file()) + goto err; + if (flush_io_cache(&log_file) || mysql_file_sync(log_file.file, MYF(MY_WME))) goto err; - my_off_t offset= my_b_tell(&log_file); - if (!is_relay_log) { + binlog_free_flush.set_reserved_bytes((uint32)offset); /* update binlog_end_pos so that it can be read by after sync hook */ reset_binlog_end_pos(log_file_name, offset); @@ -4126,8 +4238,7 @@ bool MYSQL_BIN_LOG::open(const char *log_name, /* Notify the io thread that binlog is rotated to a new file */ if (is_relay_log) signal_relay_log_update(); - else - update_binlog_end_pos(); + DBUG_RETURN(0); err: @@ -5717,9 +5828,9 @@ int MYSQL_BIN_LOG::new_file() @retval nonzero - error */ -int MYSQL_BIN_LOG::new_file_without_locking() +int MYSQL_BIN_LOG::new_file_without_locking(bool is_free_flush) { - return new_file_impl(); + return new_file_impl(is_free_flush); } @@ -5734,7 +5845,7 @@ int MYSQL_BIN_LOG::new_file_without_locking() binlog_space_total will be updated if binlog_space_limit is set */ -int MYSQL_BIN_LOG::new_file_impl() +int MYSQL_BIN_LOG::new_file_impl(bool is_free_flush) { int error= 0, close_on_error= FALSE; char new_name[FN_REFLEN], *new_name_ptr, *old_name, *file_to_open; @@ -5856,7 +5967,8 @@ int MYSQL_BIN_LOG::new_file_impl() { /* reopen the binary log file. */ file_to_open= new_name_ptr; - error= open(old_name, new_name_ptr, 0, io_cache_type, max_size, 1, FALSE); + error= open(old_name, new_name_ptr, 0, io_cache_type, max_size, 1, FALSE, + is_free_flush); } /* handle reopening errors */ @@ -6207,11 +6319,11 @@ static binlog_cache_mngr *binlog_setup_cache_mngr(THD *thd) sizeof(binlog_cache_mngr), MYF(MY_ZEROFILL)); if (!cache_mngr || - open_cached_file(&cache_mngr->stmt_cache.cache_log, mysql_tmpdir, - LOG_PREFIX, (size_t)binlog_stmt_cache_size, + open_cached_file(&cache_mngr->stmt_cache.cache_log, binlog_cache_dir, + LOG_PREFIX, (size_t) binlog_stmt_cache_size, MYF(MY_WME | MY_TRACK_WITH_LIMIT)) || - open_cached_file(&cache_mngr->trx_cache.cache_log, mysql_tmpdir, - LOG_PREFIX, (size_t)binlog_cache_size, + open_cached_file(&cache_mngr->trx_cache.cache_log, binlog_cache_dir, + LOG_PREFIX, (size_t) binlog_cache_size, MYF(MY_WME | MY_TRACK_WITH_LIMIT))) { my_free(cache_mngr); @@ -6866,7 +6978,8 @@ Event_log::prepare_pending_rows_event(THD *thd, TABLE* table, bool MYSQL_BIN_LOG::write_gtid_event(THD *thd, bool standalone, bool is_transactional, uint64 commit_id, - bool has_xid, bool is_ro_1pc) + bool has_xid, bool is_ro_1pc, + bool is_free_flush) { rpl_gtid gtid; uint32 domain_id; @@ -6934,6 +7047,9 @@ MYSQL_BIN_LOG::write_gtid_event(THD *thd, bool standalone, } #endif + if (unlikely(is_free_flush)) + gtid_event.pad_to_size= binlog_free_flush.get_gtid_event_pad_size(); + if (write_event(&gtid_event)) DBUG_RETURN(true); status_var_add(thd->status_var.binlog_bytes_written, gtid_event.data_written); @@ -7623,7 +7739,8 @@ MYSQL_BIN_LOG::do_checkpoint_request(ulong binlog_id) @retval nonzero - error in rotating routine. */ -int MYSQL_BIN_LOG::rotate(bool force_rotate, bool* check_purge) +int MYSQL_BIN_LOG::rotate(bool force_rotate, bool *check_purge, + bool is_free_flush) { int error= 0; ulonglong binlog_pos; @@ -7664,7 +7781,7 @@ int MYSQL_BIN_LOG::rotate(bool force_rotate, bool* check_purge) */ mark_xids_active(binlog_id, 1); - if (unlikely((error= new_file_without_locking()))) + if (unlikely((error= new_file_without_locking(is_free_flush)))) { /** Be conservative... There are possible lost events (eg, @@ -7965,12 +8082,14 @@ int Event_log::write_cache_raw(THD *thd, IO_CACHE *cache) int Event_log::write_cache(THD *thd, binlog_cache_data *cache_data) { - int res; IO_CACHE *cache= &cache_data->cache_log; DBUG_ENTER("Event_log::write_cache"); mysql_mutex_assert_owner(&LOCK_log); + if (cache_data->init_for_read()) + DBUG_RETURN(ER_ERROR_ON_WRITE); + /* If possible, just copy the cache over byte-by-byte with pre-computed checksums. @@ -7979,14 +8098,12 @@ int Event_log::write_cache(THD *thd, binlog_cache_data *cache_data) likely(!crypto.scheme) && likely(!opt_binlog_legacy_event_pos)) { - int res= my_b_copy_all_to_cache(cache, &log_file); + int res= + my_b_copy_to_cache(cache, &log_file, cache_data->length_for_read()); status_var_add(thd->status_var.binlog_bytes_written, my_b_tell(cache)); DBUG_RETURN(res ? ER_ERROR_ON_WRITE : 0); } - if ((res= reinit_io_cache(cache, READ_CACHE, 0, 0, 0))) - DBUG_RETURN(ER_ERROR_ON_WRITE); - /* Amount of remaining bytes in the IO_CACHE read buffer. */ size_t log_file_pos; uchar header_buf[LOG_EVENT_HEADER_LEN]; @@ -8301,6 +8418,7 @@ MYSQL_BIN_LOG::write_transaction_to_binlog(THD *thd, DBUG_RETURN(0); } + entry.next= nullptr; entry.thd= thd; entry.cache_mngr= cache_mngr; entry.error= 0; @@ -8703,7 +8821,16 @@ MYSQL_BIN_LOG::queue_for_group_commit(group_commit_entry *orig_entry) bool MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) { - int is_leader= queue_for_group_commit(entry); + int is_leader; + + if (binlog_free_flush.should_free_flush(entry) && + binlog_free_flush.commit(entry)) + { + is_leader= 1; + goto commit; + } + + is_leader= queue_for_group_commit(entry); #ifdef WITH_WSREP /* commit order was released in queue_for_group_commit() call, here we check if wsrep_commit_ordered() failed or if we are leader */ @@ -8754,6 +8881,7 @@ MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) */ } +commit: if (!opt_optimize_thread_scheduling) { /* For the leader, trx_group_commit_leader() already took the lock. */ @@ -8852,7 +8980,8 @@ MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) */ void -MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) +MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader, + bool is_free_flush) { uint xid_count= 0; my_off_t UNINIT_VAR(commit_offset); @@ -8863,6 +8992,16 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) uint64 commit_id; DBUG_ENTER("MYSQL_BIN_LOG::trx_group_commit_leader"); + /* + When move a binlog cache to a binlog file, the transaction itself is + a group. + */ + if (unlikely(is_free_flush)) + { + last_in_queue= leader; + queue= leader; + } + else { #ifdef ENABLED_DEBUG_SYNC DBUG_EXECUTE_IF("inject_binlog_commit_before_get_LOCK_log", @@ -8888,7 +9027,6 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) current= group_commit_queue; group_commit_queue= NULL; mysql_mutex_unlock(&LOCK_prepare_ordered); - binlog_id= current_binlog_id; /* As the queue is in reverse order of entering, reverse it. */ last_in_queue= current; @@ -8908,8 +9046,9 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) /* Now we have in queue the list of transactions to be committed in order. */ } - - DBUG_ASSERT(is_open()); + + binlog_id= current_binlog_id; + if (likely(is_open())) // Should always be true { commit_id= (last_in_queue == leader ? 0 : (uint64)leader->thd->query_id); @@ -8944,10 +9083,11 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) */ DBUG_ASSERT(!cache_mngr->stmt_cache.empty() || !cache_mngr->trx_cache.empty() || - current->thd->transaction->xid_state.is_explicit_XA()); + current->thd->transaction->xid_state.is_explicit_XA() || + is_free_flush); - if (unlikely((current->error= write_transaction_or_stmt(current, - commit_id)))) + if (unlikely((current->error= write_transaction_or_stmt( + current, commit_id, is_free_flush)))) current->commit_errno= errno; strmake_buf(cache_mngr->last_commit_pos_file, log_file_name); @@ -9203,7 +9343,7 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) int MYSQL_BIN_LOG::write_transaction_or_stmt(group_commit_entry *entry, - uint64 commit_id) + uint64 commit_id, bool is_free_flush) { binlog_cache_mngr *mngr= entry->cache_mngr; bool has_xid= entry->end_event->get_type_code() == XID_EVENT; @@ -9224,10 +9364,17 @@ MYSQL_BIN_LOG::write_transaction_or_stmt(group_commit_entry *entry, DBUG_ASSERT(!(entry->using_stmt_cache && !mngr->stmt_cache.empty() && mngr->get_binlog_cache_log(FALSE)->error)); - if (write_gtid_event(entry->thd, is_prepared_xa(entry->thd), - entry->using_trx_cache, commit_id, - has_xid, entry->ro_1pc)) - DBUG_RETURN(ER_ERROR_ON_WRITE); + /* + gtid will be written when renaming the binlog cache to binlog file, + if is_free_flush is true. Thus skip write_gtid_event here. + */ + if (likely(!is_free_flush)) + { + if (write_gtid_event(entry->thd, is_prepared_xa(entry->thd), + entry->using_trx_cache, commit_id, has_xid, + entry->ro_1pc)) + DBUG_RETURN(ER_ERROR_ON_WRITE); + } if (entry->using_stmt_cache && !mngr->stmt_cache.empty() && write_cache(entry->thd, mngr->get_binlog_cache_data(FALSE))) @@ -12933,3 +13080,220 @@ void wsrep_register_binlog_handler(THD *thd, bool trx) } #endif /* WITH_WSREP */ + +inline bool Binlog_free_flush::should_free_flush( + const MYSQL_BIN_LOG::group_commit_entry *entry) const +{ + binlog_cache_data *trx_cache= entry->cache_mngr->get_binlog_cache_data(true); + binlog_cache_data *stmt_cache= + entry->cache_mngr->get_binlog_cache_data(false); + + /* + The binlog cache file is not encrypted in the same way with binlog, so it + cannot be renamed to binlog file. It is not supported to rename both + statement cache and transaction cache to binlog files at the same time. + */ + if (unlikely(encrypt_binlog || + (entry->using_stmt_cache && entry->using_trx_cache && + !stmt_cache->empty() && !trx_cache->empty()))) + return false; + + binlog_cache_data *cache_data; + if (unlikely(entry->using_stmt_cache && !stmt_cache->empty())) + cache_data= stmt_cache; + else + cache_data= trx_cache; + + /* Do not free flush if total_bytes smaller than limit size. */ + if (likely(cache_data->get_byte_position() <= + opt_binlog_free_flush_threshold)) + return false; + + /* Do not free flush if reserve space equal to zero. */ + if (cache_data->file_reserved_bytes() == 0) + return false; + + /* + Do not free flush if no tmp file writes, happened when threshold is + smaller than binlog cache size. + */ + if (unlikely(cache_data->cache_log.disk_writes == 0)) + return false; + + return true; +} + +bool Binlog_free_flush::commit(MYSQL_BIN_LOG::group_commit_entry *entry) +{ + bool check_purge= false; + THD *thd= entry->thd; + binlog_cache_mngr *cache_mngr= entry->cache_mngr; + binlog_cache_data *cache_data= cache_mngr->get_binlog_cache_data(true); + if (unlikely(!entry->using_trx_cache || cache_data->empty())) + cache_data= cache_mngr->get_binlog_cache_data(false); + + /* Sync the temp file before enter log_lock to avoid holding the lock long */ + if (cache_data->sync_temp_file()) + return false; + + thd->wait_for_prior_commit(); + + // It will be released by trx_group_commit_leader + mysql_mutex_lock(&mysql_bin_log.LOCK_log); + + enum enum_binlog_checksum_alg expected_alg= + mysql_bin_log.checksum_alg_reset != BINLOG_CHECKSUM_ALG_UNDEF + ? mysql_bin_log.checksum_alg_reset + : (enum_binlog_checksum_alg) binlog_checksum_options; + + /* + In legacy mode, all events should has a valid position this done by + updating log_pos field when writing events from binlog cache to binlog + file. Thus rename binlog cache to binlog file is not supported in legacy + mode. + + if the cache's checksum alg is not same to the binlog's checksum, it needs + to recalculate the checksum. Thus rename binlog cache to binlog file is + not supported. + */ + if (!mysql_bin_log.is_open() || opt_binlog_legacy_event_pos || + (expected_alg != cache_data->checksum_opt)) + { + mysql_mutex_unlock(&mysql_bin_log.LOCK_log); + return false; + } + + m_entry= entry; + m_replaced= false; + m_cache_data= cache_data; + ulong prev_binlog_id= mysql_bin_log.current_binlog_id; + + /* + It will call replace_binlog_file() to rename the transaction's binlog cache + to the new binlog file. + */ + if (mysql_bin_log.rotate(true, &check_purge, true /* is_free_flush */)) + { + DBUG_ASSERT(!m_replaced); + DBUG_ASSERT(!mysql_bin_log.is_open()); + } + + if (!m_replaced) + { + mysql_mutex_unlock(&mysql_bin_log.LOCK_log); + if (check_purge) + mysql_bin_log.checkpoint_and_purge(prev_binlog_id); + return false; + } + + /* Seek binlog file to the end */ + reinit_io_cache(&mysql_bin_log.log_file, WRITE_CACHE, + cache_data->temp_file_length(), false, true); + status_var_add(m_entry->thd->status_var.binlog_bytes_written, + cache_data->get_byte_position()); + m_cache_data->detach_temp_file(); + + mysql_bin_log.trx_group_commit_leader(entry, true /* is_free_flush */); + + if (check_purge) + mysql_bin_log.checkpoint_and_purge(prev_binlog_id); + return true; +} + +bool Binlog_free_flush::replace_binlog_file() +{ + size_t binlog_size= my_b_tell(&mysql_bin_log.log_file); + size_t required_size= binlog_size; + // space for Gtid_log_event + required_size+= LOG_EVENT_HEADER_LEN + Gtid_log_event::max_data_length + + BINLOG_CHECKSUM_LEN; + + DBUG_EXECUTE_IF("simulate_required_size_too_big", required_size= 10000;); + if (required_size > m_cache_data->file_reserved_bytes()) + { + sql_print_information("Could not rename binlog cache to binlog, " + "require %llu bytes but only %llu bytes reserved.", + required_size, m_cache_data->file_reserved_bytes()); + return false; + } + + File new_log_fd= -1; + bool ret= false; + + /* Create fd for the cache file as a new binlog file fd */ + new_log_fd= mysql_file_open(key_file_binlog, m_cache_data->temp_file_name(), + O_BINARY | O_CLOEXEC | O_WRONLY, MYF(MY_WME)); + if (new_log_fd == -1) + return false; + + /* Copy the part which has been flushed to binlog file to binlog cache */ + if (mysql_bin_log.log_file.pos_in_file > 0) + { + size_t copy_len= 0; + uchar buf[IO_SIZE]; + + int read_fd= + mysql_file_open(key_file_binlog, mysql_bin_log.get_log_fname(), + O_RDONLY | O_BINARY | O_SHARE, MYF(MY_WME)); + if (read_fd == -1) + goto err; + + while (copy_len < mysql_bin_log.log_file.pos_in_file) + { + int read_len= (int) mysql_file_read(read_fd, buf, IO_SIZE, MYF(MY_WME)); + if (read_len < 0 || + mysql_file_write(new_log_fd, buf, read_len, + MYF(MY_WME | MY_NABP | MY_WAIT_IF_FULL))) + { + mysql_file_close(read_fd, MYF(MY_WME)); + goto err; + } + copy_len+= read_len; + } + + mysql_file_close(read_fd, MYF(MY_WME)); + } + + // Set the cache file as binlog file. + mysql_file_close(mysql_bin_log.log_file.file, MYF(MY_WME)); + mysql_bin_log.log_file.file= new_log_fd; + new_log_fd= -1; + my_delete(mysql_bin_log.get_log_fname(), MYF(0)); + + /* Any error happens after the file is deleted should return true. */ + ret= true; + + if (mysql_bin_log.write_gtid_event( + m_entry->thd, is_prepared_xa(m_entry->thd), m_entry->using_trx_cache, + 0 /* commit_id */, m_entry->end_event->get_type_code() == XID_EVENT, + m_entry->ro_1pc, true /* is_free_flush */)) + goto err; + + DBUG_EXECUTE_IF("binlog_free_flush_crash_before_rename", DBUG_SUICIDE();); + + if (DBUG_IF("simulate_rename_binlog_cache_to_binlog_error") || + my_rename(m_cache_data->temp_file_name(), mysql_bin_log.get_log_fname(), + MYF(MY_WME))) + goto err; + + sql_print_information("Renamed binlog cache to binlog %s", + mysql_bin_log.get_log_fname()); + m_replaced= true; + return false; +err: + if (new_log_fd != -1) + mysql_file_close(new_log_fd, MYF(MY_WME)); + return ret; +} + +size_t Binlog_free_flush::get_gtid_event_pad_size() +{ + size_t begin_pos= my_b_tell(&mysql_bin_log.log_file); + size_t pad_to_size= + m_cache_data->file_reserved_bytes() - begin_pos - LOG_EVENT_HEADER_LEN; + + if (binlog_checksum_options != BINLOG_CHECKSUM_ALG_OFF) + pad_to_size-= BINLOG_CHECKSUM_LEN; + + return pad_to_size; +} diff --git a/sql/log.h b/sql/log.h index 3ee06e17264..117f7638f2b 100644 --- a/sql/log.h +++ b/sql/log.h @@ -600,9 +600,12 @@ class binlog_cache_mngr; class binlog_cache_data; struct rpl_gtid; struct wait_for_commit; +class Binlog_free_flush; class MYSQL_BIN_LOG: public TC_LOG, private Event_log { + friend Binlog_free_flush; + #ifdef HAVE_PSI_INTERFACE /** The instrumentation key to use for @ LOCK_index. */ PSI_mutex_key m_key_LOCK_index; @@ -756,18 +759,20 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log new_file() is locking. new_file_without_locking() does not acquire LOCK_log. */ - int new_file_impl(); + int new_file_impl(bool is_free_flush= false); void do_checkpoint_request(ulong binlog_id); - int write_transaction_or_stmt(group_commit_entry *entry, uint64 commit_id); + int write_transaction_or_stmt(group_commit_entry *entry, uint64 commit_id, + bool is_free_flush= false); int queue_for_group_commit(group_commit_entry *entry); bool write_transaction_to_binlog_events(group_commit_entry *entry); - void trx_group_commit_leader(group_commit_entry *leader); + void trx_group_commit_leader(group_commit_entry *leader, + bool is_free_flush= false); bool is_xidlist_idle_nolock(); void update_gtid_index(uint32 offset, rpl_gtid gtid); public: void purge(bool all); - int new_file_without_locking(); + int new_file_without_locking(bool is_free_flush= false); /* A list of struct xid_count_per_binlog is used to keep track of how many XIDs are in prepared, but not committed, state in each binlog. And how @@ -997,7 +1002,8 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log enum cache_type io_cache_type_arg, ulong max_size, bool null_created, - bool need_mutex); + bool need_mutex, + bool is_free_flush = false); bool open_index_file(const char *index_file_name_arg, const char *log_name, bool need_mutex); /* Use this to start writing a new log file */ @@ -1037,7 +1043,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log bool is_active(const char* log_file_name); bool can_purge_log(const char *log_file_name, bool interactive); int update_log_index(LOG_INFO* linfo, bool need_update_threads); - int rotate(bool force_rotate, bool* check_purge); + int rotate(bool force_rotate, bool* check_purge, bool is_free_flush= false); void checkpoint_and_purge(ulong binlog_id); int rotate_and_purge(bool force_rotate, DYNAMIC_ARRAY* drop_gtid_domain= NULL); /** @@ -1117,7 +1123,8 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log bool is_xidlist_idle(); bool write_gtid_event(THD *thd, bool standalone, bool is_transactional, uint64 commit_id, - bool has_xid= false, bool ro_1pc= false); + bool has_xid= false, bool ro_1pc= false, + bool is_free_flush= false); int read_state_from_file(); int write_state_to_file(); int get_most_recent_gtid_list(rpl_gtid **list, uint32 *size); diff --git a/sql/log_cache.cc b/sql/log_cache.cc new file mode 100644 index 00000000000..014e76611cf --- /dev/null +++ b/sql/log_cache.cc @@ -0,0 +1,122 @@ +#include "my_global.h" +#include "log_cache.h" +#include "handler.h" +#include "my_sys.h" +#include "mysql/psi/mysql_file.h" +#include "mysql/service_wsrep.h" + +const char *BINLOG_CACHE_DIR= "#binlog_cache_files"; +char binlog_cache_dir[FN_REFLEN]; +extern uint32 binlog_cache_reserved_size(); + +bool binlog_cache_data::init_file_reserved_bytes() +{ + // Session's cache file is not created, so created here. + if (cache_log.file == -1) + { + char name[FN_REFLEN]; + + /* Cache file is named with PREFIX + binlog_cache_data object's address */ + snprintf(name, FN_REFLEN, "%s/%s_%llu", cache_log.dir, cache_log.prefix, + (ulonglong) this); + + if ((cache_log.file= + mysql_file_open(0, name, O_CREAT | O_RDWR, MYF(MY_WME))) < 0) + { + sql_print_error("Failed to open binlog cache temporary file %s", name); + cache_log.error= -1; + return true; + } + } + +#ifdef WITH_WSREP + /* + WSREP code accesses cache_log directly, so don't reserve space if WSREP is + on. + */ + if (unlikely(wsrep_on(current_thd))) + return false; +#endif + + m_file_reserved_bytes= binlog_cache_reserved_size(); + cache_log.pos_in_file= m_file_reserved_bytes; + cache_log.seek_not_done= 1; + return false; +} + +void binlog_cache_data::detach_temp_file() +{ + /* + If there was a rollback_to_savepoint happened before, the real length of + tmp file can be greater than the file_end_pos. Truncate the cache tmp + file to file_end_pos of this cache. + */ + my_chsize(cache_log.file, my_b_tell(&cache_log), 0, MYF(MY_WME)); + + mysql_file_close(cache_log.file, MYF(0)); + cache_log.file= -1; + reset(); +} + +extern void ignore_db_dirs_append(const char *dirname_arg); + +bool init_binlog_cache_dir() +{ + size_t length; + uint max_tmp_file_name_len= + 2 /* prefix */ + 10 /* max len of thread_id */ + 1 /* underline */; + + ignore_db_dirs_append(BINLOG_CACHE_DIR); + + dirname_part(binlog_cache_dir, log_bin_basename, &length); + /* + Must ensure the full name of the tmp file is shorter than FN_REFLEN, to + avoid overflowing the name buffer in write and commit. + */ + if (length + strlen(BINLOG_CACHE_DIR) + max_tmp_file_name_len >= FN_REFLEN) + { + sql_print_error("Could not create binlog cache dir %s%s. It is too long.", + binlog_cache_dir, BINLOG_CACHE_DIR); + return true; + } + + memcpy(binlog_cache_dir + length, BINLOG_CACHE_DIR, + strlen(BINLOG_CACHE_DIR)); + binlog_cache_dir[length + strlen(BINLOG_CACHE_DIR)]= 0; + + MY_DIR *dir_info= my_dir(binlog_cache_dir, MYF(0)); + + if (!dir_info) + { + /* Make a dir for binlog cache temp files if not exist. */ + if (my_mkdir(binlog_cache_dir, 0777, MYF(0)) < 0) + { + sql_print_error("Could not create binlog cache dir %s.", + binlog_cache_dir); + return true; + } + return false; + } + + /* Try to delete all cache files in the directory. */ + for (uint i= 0; i < dir_info->number_of_files; i++) + { + FILEINFO *file= dir_info->dir_entry + i; + + if (strncmp(file->name, LOG_PREFIX, strlen(LOG_PREFIX))) + { + sql_print_warning("%s is in %s/, but it is not a binlog cache file", + file->name, BINLOG_CACHE_DIR); + continue; + } + + char file_path[FN_REFLEN]; + fn_format(file_path, file->name, binlog_cache_dir, "", + MYF(MY_REPLACE_DIR)); + + my_delete(file_path, MYF(0)); + } + + my_dirend(dir_info); + return false; +} diff --git a/sql/log_cache.h b/sql/log_cache.h index 79a9b94d8bc..a16e85b4b73 100644 --- a/sql/log_cache.h +++ b/sql/log_cache.h @@ -22,6 +22,16 @@ static constexpr my_off_t MY_OFF_T_UNDEF= ~0ULL; /** Truncate cache log files bigger than this */ static constexpr my_off_t CACHE_FILE_TRUNC_SIZE = 65536; +/** + Create binlog cache directory if it doesn't exist, otherwise delete all + files existing in the directory. + + @retval false Succeeds to initialize the directory. + @retval true Failed to initialize the directory. +*/ +bool init_binlog_cache_dir(); + +extern char binlog_cache_dir[FN_REFLEN]; /* Helper classes to store non-transactional and transactional data @@ -35,7 +45,7 @@ class binlog_cache_data before_stmt_pos(MY_OFF_T_UNDEF), m_pending(0), status(0), incident(FALSE), precompute_checksums(precompute_checksums), saved_max_binlog_cache_size(0), ptr_binlog_cache_use(0), - ptr_binlog_cache_disk_use(0) + ptr_binlog_cache_disk_use(0), m_file_reserved_bytes(0) { /* Read the current checksum setting. We will use this setting to decide @@ -47,9 +57,13 @@ class binlog_cache_data (enum_binlog_checksum_alg)binlog_checksum_options; } - ~binlog_cache_data() + virtual ~binlog_cache_data() { DBUG_ASSERT(empty()); + + if (cache_log.file != -1 && !encrypt_tmp_files) + unlink(my_filename(cache_log.file)); + close_cached_file(&cache_log); } @@ -67,7 +81,7 @@ class binlog_cache_data bool empty() const { return (pending() == NULL && - (my_b_write_tell(&cache_log) == 0 || + (my_b_write_tell(&cache_log) - m_file_reserved_bytes == 0 || ((status & (LOGGED_ROW_EVENT | LOGGED_CRITICAL)) == 0))); } @@ -97,6 +111,8 @@ class binlog_cache_data bool truncate_file= (cache_log.file != -1 && my_b_write_tell(&cache_log) > MY_MIN(CACHE_FILE_TRUNC_SIZE, binlog_stmt_cache_size)); + // m_file_reserved_bytes must be reset to 0, before truncate. + m_file_reserved_bytes= 0; truncate(0,1); // Forget what's in cache checksum_opt= !precompute_checksums ? BINLOG_CHECKSUM_ALG_OFF : (enum_binlog_checksum_alg)binlog_checksum_options; @@ -112,7 +128,8 @@ class binlog_cache_data my_off_t get_byte_position() const { - return my_b_tell(&cache_log); + DBUG_ASSERT(cache_log.type == WRITE_CACHE); + return my_b_tell(&cache_log) - m_file_reserved_bytes; } my_off_t get_prev_position() const @@ -172,6 +189,81 @@ class binlog_cache_data status|= status_arg; } + /** + This function is called everytime when anything is being written into the + cache_log. To support rename binlog cache to binlog file, the cache_log + should be initialized with reserved space. + */ + bool write_prepare(size_t write_length) + { + /* Data will exceed the buffer size in this write */ + if (unlikely(cache_log.write_pos + write_length > cache_log.write_end && + cache_log.pos_in_file == 0)) + { + /* Only session's binlog cache need to reserve space. */ + if (cache_log.dir == binlog_cache_dir && !encrypt_tmp_files) + return init_file_reserved_bytes(); + } + return false; + } + + /** + For session's binlog cache, it have to call this function to skip the + reserved before reading the cache file. + */ + bool init_for_read() + { + return reinit_io_cache(&cache_log, READ_CACHE, m_file_reserved_bytes, 0, 0); + } + + /** + For session's binlog cache, it have to call this function to get the + actual data length. + */ + my_off_t length_for_read() const + { + DBUG_ASSERT(cache_log.type == READ_CACHE); + return cache_log.end_of_file - m_file_reserved_bytes; + } + + /** + It function returns the cache file's actual length which includes the + reserved space. + */ + my_off_t temp_file_length() + { + return my_b_tell(&cache_log); + } + + uint32 file_reserved_bytes() { return m_file_reserved_bytes; } + + /** + Flush and sync the data of the file into storage. + + @retval true Error happens + @retval false Succeeds + */ + bool sync_temp_file() + { + DBUG_ASSERT(cache_log.file != -1); + + if (my_b_flush_io_cache(&cache_log, 1) || + mysql_file_sync(cache_log.file, MYF(MY_WME))) + return true; + return false; + } + + /** + Copy the name of the cache file to the argument name. + */ + const char *temp_file_name() { return my_filename(cache_log.file); } + + /** + It is called after renaming the cache file to a binlog file. The file + now is a binlog file, so detach it from the binlog cache. + */ + void detach_temp_file(); + /* Cache to store data before copying it to the binary log. */ @@ -253,6 +345,12 @@ class binlog_cache_data */ ulong *ptr_binlog_cache_disk_use; + /* + Stores the bytes reserved at the begin of the cache file. It could be + 0 for cases that reserved space are not supported. see write_prepare(). + */ + uint32 m_file_reserved_bytes {0}; + /* It truncates the cache to a certain position. This includes deleting the pending event. @@ -266,12 +364,18 @@ class binlog_cache_data delete pending(); set_pending(0); } - my_bool res __attribute__((unused))= - reinit_io_cache(&cache_log, WRITE_CACHE, pos, 0, reset_cache); + my_bool res __attribute__((unused))= reinit_io_cache( + &cache_log, WRITE_CACHE, pos + m_file_reserved_bytes, 0, reset_cache); DBUG_ASSERT(res == 0); cache_log.end_of_file= saved_max_binlog_cache_size; } + /** + Reserve required space at the begin of the tempoary file. It will create + the temporary file if it doesn't exist. + */ + bool init_file_reserved_bytes(); + binlog_cache_data& operator=(const binlog_cache_data& info); binlog_cache_data(const binlog_cache_data& info); }; diff --git a/sql/log_event.h b/sql/log_event.h index fdbd46f8d0d..8c1edfe2c0b 100644 --- a/sql/log_event.h +++ b/sql/log_event.h @@ -3340,6 +3340,14 @@ class Gtid_log_event: public Log_event uint64 sa_seq_no; // start alter identifier for CA/RA #ifdef MYSQL_SERVER event_xid_t xid; + /* + Pad the event to this size if it is not zero. It is only used for renaming + a binlog cache to binlog file. There is some reserved space for gtid event + and the events at the begin of the binlog file. There must be some space + left after the events are filled. Thus the left space is padded into the + gtid event with 0. + */ + uint64 pad_to_size; #else event_mysql_xid_t xid; #endif @@ -3404,6 +3412,11 @@ class Gtid_log_event: public Log_event static const uchar FL_EXTRA_THREAD_ID= 16; // thread_id like in BEGIN Query #ifdef MYSQL_SERVER + static const uint max_data_length= GTID_HEADER_LEN + 2 + sizeof(XID) + + 1 /* flags_extra: */ + + 4 /* Extra Engines */ + + 4 /* FL_EXTRA_THREAD_ID */; + Gtid_log_event(THD *thd_arg, uint64 seq_no, uint32 domain_id, bool standalone, uint16 flags, bool is_transactional, uint64 commit_id, bool has_xid= false, bool is_ro_1pc= false); diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc index 9b179836651..44cec8b15fd 100644 --- a/sql/log_event_server.cc +++ b/sql/log_event_server.cc @@ -29,6 +29,7 @@ #include "unireg.h" #include "log_event.h" +#include "log_cache.h" #include "sql_base.h" // close_thread_tables #include "sql_cache.h" // QUERY_CACHE_FLAGS_SIZE #include "sql_locale.h" // MY_LOCALE, my_locale_by_number, my_locale_en_US @@ -690,6 +691,9 @@ void Log_event::init_show_field_list(THD *thd, List<Item>* field_list) int Log_event_writer::write_internal(const uchar *pos, size_t len) { DBUG_ASSERT(!ctx || encrypt_or_write == &Log_event_writer::encrypt_and_write); + if (cache_data && cache_data->write_prepare(len)) + return 1; + if (my_b_safe_write(file, pos, len)) { DBUG_PRINT("error", ("write to log failed: %d", my_errno)); @@ -2839,7 +2843,7 @@ Gtid_log_event::Gtid_log_event(THD *thd_arg, uint64 seq_no_arg, bool ro_1pc) : Log_event(thd_arg, flags_arg, is_transactional), seq_no(seq_no_arg), commit_id(commit_id_arg), domain_id(domain_id_arg), - flags2((standalone ? FL_STANDALONE : 0) | + pad_to_size(0), flags2((standalone ? FL_STANDALONE : 0) | (commit_id_arg ? FL_GROUP_COMMIT_ID : 0)), flags_extra(0), extra_engines(0), thread_id(thd_arg->variables.pseudo_thread_id) @@ -2959,10 +2963,7 @@ Gtid_log_event::peek(const uchar *event_start, size_t event_len, bool Gtid_log_event::write(Log_event_writer *writer) { - uchar buf[GTID_HEADER_LEN + 2 + sizeof(XID) - + 1 /* flags_extra: */ - + 4 /* Extra Engines */ - + 4 /* FL_EXTRA_THREAD_ID */]; + uchar buf[max_data_length]; size_t write_len= 13; int8store(buf, seq_no); @@ -3042,6 +3043,27 @@ Gtid_log_event::write(Log_event_writer *writer) bzero(buf+write_len, GTID_HEADER_LEN-write_len); write_len= GTID_HEADER_LEN; } + + if (unlikely(pad_to_size > write_len)) + { + if (write_header(writer, pad_to_size) || + write_data(writer, buf, write_len)) + return true; + + pad_to_size-= write_len; + + char pad_buf[IO_SIZE]; + bzero(pad_buf, pad_to_size); + while (pad_to_size) + { + uint64 size= pad_to_size >= IO_SIZE ? IO_SIZE : pad_to_size; + if (write_data(writer, pad_buf, size)) + return true; + pad_to_size-= size; + } + return write_footer(writer); + } + return write_header(writer, write_len) || write_data(writer, buf, write_len) || write_footer(writer); diff --git a/sql/mysqld.cc b/sql/mysqld.cc index e938e8f6cfa..a2ef34a28db 100644 --- a/sql/mysqld.cc +++ b/sql/mysqld.cc @@ -120,7 +120,7 @@ #include "sp_cache.h" #include "sql_reload.h" // reload_acl_and_cache #include "sp_head.h" // init_sp_psi_keys - +#include "log_cache.h" #include <mysqld_default_groups.h> #ifdef HAVE_POLL_H @@ -5609,6 +5609,8 @@ static int init_server_components() mysql_mutex_unlock(log_lock); if (unlikely(error)) unireg_abort(1); + if (unlikely(init_binlog_cache_dir())) + unireg_abort(1); } #ifdef HAVE_REPLICATION diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc index d4997793428..cd7bae8ab12 100644 --- a/sql/sys_vars.cc +++ b/sql/sys_vars.cc @@ -7380,3 +7380,13 @@ static Sys_var_enum Sys_block_encryption_mode( "AES_ENCRYPT() and AES_DECRYPT() functions", SESSION_VAR(block_encryption_mode), CMD_LINE(REQUIRED_ARG), block_encryption_mode_values, DEFAULT(0)); + +extern ulonglong opt_binlog_free_flush_threshold; +static Sys_var_ulonglong Sys_binlog_free_flush_threshold( + "binlog_free_flush_threshold", + "Try to rename the binlog cache temporary file of the commiting " + "transaction to a binlog file when its binlog cache size " + "is bigger than the value of this variable", + GLOBAL_VAR(opt_binlog_free_flush_threshold), + CMD_LINE(REQUIRED_ARG), VALID_RANGE(10 * 1024 * 1024, ULLONG_MAX), + DEFAULT(128 * 1024 * 1024), BLOCK_SIZE(1)); -- 2.39.2

1 0

[PATCH 0/6] Testcase fixes
by Kristian Nielsen 22 Aug '24

22 Aug '24

A handful of fixes for test failures in buildot. Kristian Nielsen (6): MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates Restore skiping rpl.rpl_mdev6020 under Valgrind Fix sporadic test failure in rpl.rpl_create_drop_event Fix sporadic failure of test case rpl.rpl_old_master Skip mariabackup.slave_provision_nolock in --valgrind, it uses a lot of CPU Fix sporadic failure of test case rpl.rpl_start_stop_slave .../mariabackup/slave_provision_nolock.test | 2 ++ mysql-test/suite/rpl/r/rpl_old_master.result | 3 --- .../suite/rpl/t/rpl_create_drop_event.test | 6 ++++++ mysql-test/suite/rpl/t/rpl_mdev6020.test | 2 ++ mysql-test/suite/rpl/t/rpl_old_master.test | 7 ------- .../suite/rpl/t/rpl_start_stop_slave.test | 12 ++++++++++- sql/rpl_parallel.cc | 20 +++++++++++++++---- sql/rpl_rli.cc | 17 ++++++++++++++++ 8 files changed, 54 insertions(+), 15 deletions(-) -- 2.39.2

2 7

[PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 31 Jul '24

31 Jul '24

From: Andrei <andrei.elkin(a)mariadb.com> This work partly implements a ROW binlog format MDEV-33455 part that is makes non-unique-index-only table XA replication safe in RBR. Existence of at least one non-null unique index has always guaranteed safety (no hang to error). Two transaction that update a non-unique-only index table could not be isolated on slave when on slave they used different non-unique indexes than on master. Unsolvable hang could be seen in case the first of the two is a prepared XA --connection slave_worker_1 xa start 'xid'; /* ... lock here ... */ ; xa prepare 'xid' while the 2nd being of any kind including of normal type --connection slave_worker_2 begin; /* ... get lock ... => wait/hang...error out */ was unable to wait up for the conflicting lock, even though the XA transaction did not really lock target records of the 2nd. This type of hang was caused by a chosen method the 2nd transaction employs to reach the target record, which is the index scan. The scanning orthodoxically just could not step over a record in the way that was locked by the XA. However as the in-the-way record can not be targeted by the 2nd transaction, otherwise the transactions would have sensed the conflict back on master *and* the other possibility of collecting extra locks by the 'xid' on non-modified records is tacked by MDEV-33454/MDEV-34466, the non-unique index scanning server/slave-applier layer must not panic at seeing a timeout error from the engine. Instead the scanning would just proceed to next possibly free index records of the same key value and ultimately must reach the target one. More generally, on the way to its target all busy records belonging to earlier (binlog order) prepared XA transactions need not be tried locking by the current non-unique index scanning transaction. This patch implements the plan for Innodb. The server layer expects the engine to mark an attempt to wait for a conflicting lock that belongs to a transaction in prepared state. The engine won't exercise, need not to, the timeout wait. When marking is done the timeout error is ignored by the server and next index record is tried out. An mtr test checks a scenario in sequential and parallel modes. --- sql/log_event_server.cc | 37 +++++++++++++++++++++++++----- sql/rpl_rli.cc | 4 +++- sql/rpl_rli.h | 14 +++++++++++ sql/sql_class.cc | 30 +++++++++++++++++++----- storage/innobase/lock/lock0lock.cc | 30 +++++++++++++++++------- 5 files changed, 94 insertions(+), 21 deletions(-) diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc index f3d01b7d9ed..166295ed263 100644 --- a/sql/log_event_server.cc +++ b/sql/log_event_server.cc @@ -8105,6 +8105,11 @@ int Rows_log_event::find_row(rpl_group_info *rgi) if (m_key_info) { + bool skip_first_compare= false; + bool is_index_unique= + (table->key_info->flags & HA_NOSAME) && + !(table->key_info->flags & (HA_NULL_PART_KEY)); + DBUG_PRINT("info",("locating record using key #%u [%s] (index_read)", m_key_nr, m_key_info->name.str)); /* We use this to test that the correct key is used in test cases. */ @@ -8149,11 +8154,20 @@ int Rows_log_event::find_row(rpl_group_info *rgi) HA_READ_KEY_EXACT)))) { DBUG_PRINT("info",("no record matching the key found in the table")); - if (error == HA_ERR_KEY_NOT_FOUND) - error= row_not_found_error(rgi); - table->file->print_error(error, MYF(0)); - table->file->ha_index_end(); - goto end; + if (error == HA_ERR_LOCK_WAIT_TIMEOUT && !is_index_unique && + (rgi->exec_flags & (1 << rpl_group_info::HIT_BUSY_INDEX))) + { + rgi->exec_flags &= ~(1 << rpl_group_info::HIT_BUSY_INDEX); + skip_first_compare= true; + } + else + { + if (error == HA_ERR_KEY_NOT_FOUND) + error= row_not_found_error(rgi); + table->file->print_error(error, MYF(0)); + table->file->ha_index_end(); + goto end; + } } /* @@ -8222,17 +8236,28 @@ int Rows_log_event::find_row(rpl_group_info *rgi) /* We use this to test that the correct key is used in test cases. */ DBUG_EXECUTE_IF("slave_crash_if_index_scan", abort();); - while (record_compare(table, m_vers_from_plain)) + while (skip_first_compare || record_compare(table, m_vers_from_plain)) { + if (skip_first_compare) + skip_first_compare= false; while ((error= table->file->ha_index_next(table->record[0]))) { DBUG_PRINT("info",("no record matching the given row found")); if (error == HA_ERR_END_OF_FILE) error= end_of_file_error(rgi); + else if (error == HA_ERR_LOCK_WAIT_TIMEOUT && !is_index_unique && + (rgi->exec_flags & (1 << rpl_group_info::HIT_BUSY_INDEX))) + { + rgi->exec_flags &= ~(1 << rpl_group_info::HIT_BUSY_INDEX); + continue; + } + DBUG_ASSERT(!(rgi->exec_flags & (1 << rpl_group_info::HIT_BUSY_INDEX))); + table->file->print_error(error, MYF(0)); table->file->ha_index_end(); goto end; } + DBUG_ASSERT(!(rgi->exec_flags & (1 << rpl_group_info::HIT_BUSY_INDEX))); } } else diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc index 3b036cdaa79..2839b6e68b0 100644 --- a/sql/rpl_rli.cc +++ b/sql/rpl_rli.cc @@ -2154,12 +2154,14 @@ rpl_group_info::reinit(Relay_log_info *rli) gtid_ignore_duplicate_state= GTID_DUPLICATE_NULL; speculation= SPECULATE_NO; commit_orderer.reinit(); + exec_flags= 0; } rpl_group_info::rpl_group_info(Relay_log_info *rli) : thd(0), wait_commit_sub_id(0), wait_commit_group_info(0), parallel_entry(0), - deferred_events(NULL), m_annotate_event(0), is_parallel_exec(false) + deferred_events(NULL), m_annotate_event(0), is_parallel_exec(false), + exec_flags(0) { reinit(rli); bzero(&current_gtid, sizeof(current_gtid)); diff --git a/sql/rpl_rli.h b/sql/rpl_rli.h index 3f24481839b..a9ec44b58cb 100644 --- a/sql/rpl_rli.h +++ b/sql/rpl_rli.h @@ -833,6 +833,15 @@ struct rpl_group_info RETRY_KILL_KILLED }; uchar killed_for_retry; + enum enum_exec_flags { + HIT_BUSY_INDEX= 0 + }; + /* + Being executed replication event's context. It could contain + notification from engine such as attempts to lock already acquired + index record. + */ + uint32 exec_flags; rpl_group_info(Relay_log_info *rli_); ~rpl_group_info(); @@ -960,6 +969,11 @@ struct rpl_group_info if (!is_parallel_exec) rli->event_relay_log_pos= future_event_relay_log_pos; } + + bool is_row_event_execution() + { + return m_table_map.count() > 0; + } }; diff --git a/sql/sql_class.cc b/sql/sql_class.cc index 6419a58fbe4..b2321cfd865 100644 --- a/sql/sql_class.cc +++ b/sql/sql_class.cc @@ -5418,7 +5418,9 @@ thd_need_wait_reports(const MYSQL_THD thd) /* Used by storage engines (currently InnoDB) to report that one transaction THD is about to go to wait for a transactional lock held by - another transactions OTHER_THD. + another transactions OTHER_THD. Also in case of a non-unique index locking + the slave applier is informed on a prepared XA transaction holding + an offending lock. This is used for parallel replication, where transactions are required to commit in the same order on the slave as they did on the master. If the @@ -5432,6 +5434,9 @@ thd_need_wait_reports(const MYSQL_THD thd) slave in the first place. However, it is possible in case when the optimizer chooses a different plan on the slave than on the master (eg. table scan instead of index scan). + When the latter takes places and the conflict on a non-unique index involves + the slave applier the latter needs to know whether the lock onwer is + a prepared XA and this is provided. Storage engines report lock waits using this call. If a lock wait causes a deadlock with the pre-determined commit order, we kill the later @@ -5444,20 +5449,33 @@ thd_need_wait_reports(const MYSQL_THD thd) transaction. */ extern "C" int -thd_rpl_deadlock_check(MYSQL_THD thd, MYSQL_THD other_thd) +thd_rpl_deadlock_check(MYSQL_THD thd, MYSQL_THD other_thd, + bool is_other_prepared, bool is_index_unique, + uint *flagged) { - rpl_group_info *rgi; - rpl_group_info *other_rgi; + rpl_group_info *rgi= thd ? thd->rgi_slave : NULL; + rpl_group_info *other_rgi= other_thd ? other_thd->rgi_slave : NULL; if (!thd) return 0; DEBUG_SYNC(thd, "thd_report_wait_for"); thd->transaction->stmt.mark_trans_did_wait(); + + /* + Return to the caller the fact of the non-unique index wait lock + conflicts with one of a prepared state transaction. + */ + if (!is_index_unique && rgi && rgi->is_row_event_execution()) { + if (is_other_prepared && !other_thd) + { + rgi->exec_flags |= 1 << rpl_group_info::HIT_BUSY_INDEX; + (*flagged)++; + } + } + if (!other_thd) return 0; binlog_report_wait_for(thd, other_thd); - rgi= thd->rgi_slave; - other_rgi= other_thd->rgi_slave; if (!rgi || !other_rgi) return 0; if (!rgi->is_parallel_exec) diff --git a/storage/innobase/lock/lock0lock.cc b/storage/innobase/lock/lock0lock.cc index 7b7e7ef3e46..2aeb4774631 100644 --- a/storage/innobase/lock/lock0lock.cc +++ b/storage/innobase/lock/lock0lock.cc @@ -61,7 +61,10 @@ my_bool innodb_deadlock_detect; ulong innodb_deadlock_report; #ifdef HAVE_REPLICATION -extern "C" void thd_rpl_deadlock_check(MYSQL_THD thd, MYSQL_THD other_thd); +extern "C" void thd_rpl_deadlock_check(MYSQL_THD thd, MYSQL_THD other_thd, + bool is_prepared= false, + bool is_index_uniq= false, + uint *flagged= NULL); extern "C" int thd_need_wait_reports(const MYSQL_THD thd); extern "C" int thd_need_ordering_with(const MYSQL_THD thd, const MYSQL_THD other_thd); extern "C" int thd_deadlock_victim_preference(const MYSQL_THD thd1, const MYSQL_THD thd2); @@ -1874,10 +1877,10 @@ ATTRIBUTE_NOINLINE MY_ATTRIBUTE((nonnull, warn_unused_result)) trx->error_state= DB_DEADLOCK if trx->lock.was_chosen_as_deadlock_victim was set when lock_sys.wait_mutex was unlocked. @param trx transaction that may be waiting for a lock -@param wait_lock lock that is being waited for +@param wait_timeout either remains or can be zeroed @return lock being waited for (may have been replaced by an equivalent one) @retval nullptr if no lock is being waited for */ -static lock_t *lock_wait_rpl_report(trx_t *trx) +static lock_t *lock_wait_rpl_report(trx_t *trx, ulong& wait_timeout) { mysql_mutex_assert_owner(&lock_sys.wait_mutex); ut_ad(trx->state == TRX_STATE_ACTIVE); @@ -1886,6 +1889,7 @@ static lock_t *lock_wait_rpl_report(trx_t *trx) lock_t *wait_lock= trx->lock.wait_lock; if (!wait_lock) return nullptr; + uint count_non_unique_index_hit= 0; /* This would likely be too large to attempt to use a memory transaction, even for wait_lock->is_table(). */ const bool nowait= lock_sys.wr_lock_try(); @@ -1903,6 +1907,8 @@ static lock_t *lock_wait_rpl_report(trx_t *trx) lock_sys.wait_mutex was unlocked, let's check it. */ if (!nowait && trx->lock.was_chosen_as_deadlock_victim) trx->error_state= DB_DEADLOCK; + if (count_non_unique_index_hit > 0) + wait_timeout= 0; return wait_lock; } ut_ad(wait_lock->is_waiting()); @@ -1940,7 +1946,11 @@ static lock_t *lock_wait_rpl_report(trx_t *trx) lock= lock_rec_get_next(heap_no, lock); do if (lock->trx->mysql_thd != thd) - thd_rpl_deadlock_check(thd, lock->trx->mysql_thd); + thd_rpl_deadlock_check(thd, lock->trx->mysql_thd, + lock->trx->state == TRX_STATE_PREPARED, + (lock->index->is_unique() && + !lock->index->n_nullable), + &count_non_unique_index_hit); while ((lock= lock_rec_get_next(heap_no, lock))); } } @@ -1971,7 +1981,7 @@ dberr_t lock_wait(que_thr_t *thr) /* InnoDB system transactions may use the global value of innodb_lock_wait_timeout, because trx->mysql_thd == NULL. */ - const ulong innodb_lock_wait_timeout= trx_lock_wait_timeout_get(trx); + ulong innodb_lock_wait_timeout= trx_lock_wait_timeout_get(trx); const my_hrtime_t suspend_time= my_hrtime_coarse(); ut_ad(!trx->dict_operation_lock_mode); @@ -2034,7 +2044,6 @@ dberr_t lock_wait(que_thr_t *thr) const bool row_lock_wait= thr->lock_state == QUE_THR_LOCK_ROW; timespec abstime; set_timespec_time_nsec(abstime, suspend_time.val * 1000); - abstime.MY_tv_sec+= innodb_lock_wait_timeout; /* Dictionary transactions must wait be immune to lock wait timeouts for locks on data dictionary tables. Here we check only for SYS_TABLES, SYS_COLUMNS, SYS_INDEXES, SYS_FIELDS. Locks on further @@ -2094,10 +2103,15 @@ dberr_t lock_wait(que_thr_t *thr) lock_sys.wait_start(); #ifdef HAVE_REPLICATION + /* + innodb_lock_wait_timeout can be zeroed in a specific case of non-unique + index locking by slave applier. When that happens no actual waiting for + wait_lock is done before the server layer has received the timeout error. + */ if (rpl) - wait_lock= lock_wait_rpl_report(trx); + wait_lock= lock_wait_rpl_report(trx, innodb_lock_wait_timeout); #endif - + abstime.MY_tv_sec+= innodb_lock_wait_timeout; switch (trx->error_state) { case DB_SUCCESS: break; -- 2.39.2

1 0

[PATCH] MDEV-33856 New definition for Seconds_Behind_Master
by Kristian Nielsen 22 Jul '24

22 Jul '24

From: Monty <monty(a)mariadb.org> This commit adds 3 new status variables to 'show all slaves status': - Master_last_event_time ; timestamp of the last event read from the master by the IO thread. - Slave_last_event_time ; Master timestamp of the last event committed on the slave. - Master_Slave_time_diff: The difference of the above two timestamps. All the above variables are NULL until the slave has started and the slave has read one query event from the master that changes data. - Added information_schema.slave_status, which allows us to remove: - show_master_info(), show_master_info_get_fields(), send_show_master_info_data(), show_all_master_info() - class Sql_cmd_show_slave_status. - Protocol::store(I_List<i_string_pair>* str_list) as it is not used anymore. - Changed old SHOW SLAVE STATUS and SHOW ALL SLAVES STATUS to use the SELECT code path, as all other SHOW ... STATUS commands. Other things: - Xid_log_time is set to time of commit to allow slave that reads the binary log to calculate Master_last_event_time and Slave_last_event_time. This is needed as there is not 'exec_time' for row events. - Fixed that Load_log_event calculates exec_time identically to Query_event. - Updated RESET SLAVE to reset Master/Slave_last_event_time - Updated SQL thread's update on first transaction read-in to only update Slave_last_event_time on group events. - Fixed possible (unlikely) bugs in sql_show.cc ...old_format() functions if allocation of 'field' would fail. --- mysql-test/include/show_slave_status.inc | 2 +- mysql-test/main/grant_slave_monitor.result | 2 + mysql-test/main/grant_slave_monitor.test | 2 + mysql-test/main/information_schema-big.result | 2 + mysql-test/main/information_schema-big.test | 1 + .../information_schema-big_embedded.result | 133 +++ .../main/information_schema-big_embedded.test | 43 + mysql-test/main/information_schema.result | 7 +- .../information_schema_all_engines.result | 12 +- mysql-test/main/ps_change_master.test | 4 +- .../suite/funcs_1/r/is_columns_is.result | 131 +++ .../suite/funcs_1/r/is_tables_is.result | 50 + .../suite/multi_source/info_logs.result | 12 +- .../multi_source_slave_alias_replica.result | 6 + mysql-test/suite/multi_source/simple.result | 18 + mysql-test/suite/multi_source/syntax.result | 2 +- mysql-test/suite/perfschema/r/relaylog.result | 4 +- mysql-test/suite/perfschema/t/relaylog.test | 4 +- .../rpl/r/master_last_event_time_row.result | 203 ++++ .../rpl/r/master_last_event_time_stmt.result | 71 ++ .../suite/rpl/r/rpl_parallel_sbm.result | 8 + .../suite/rpl/t/master_last_event_time.inc | 58 ++ .../rpl/t/master_last_event_time_row.cnf | 16 + .../rpl/t/master_last_event_time_row.test | 168 ++++ .../rpl/t/master_last_event_time_stmt.cnf | 15 + .../rpl/t/master_last_event_time_stmt.test | 84 ++ mysql-test/suite/rpl/t/rpl_parallel_sbm.test | 42 + sql/handler.h | 3 + sql/item.h | 15 +- sql/log.cc | 18 +- sql/log_event.cc | 37 +- sql/log_event.h | 15 +- sql/log_event_server.cc | 21 +- sql/protocol.cc | 27 - sql/protocol.h | 2 +- sql/rpl_mi.cc | 53 +- sql/rpl_mi.h | 5 +- sql/rpl_parallel.cc | 3 + sql/rpl_rli.cc | 9 +- sql/rpl_rli.h | 16 +- sql/slave.cc | 874 ++++++------------ sql/slave.h | 4 +- sql/sql_class.h | 1 + sql/sql_cmd.h | 20 - sql/sql_lex.h | 2 + sql/sql_parse.cc | 2 +- sql/sql_prepare.cc | 40 +- sql/sql_repl.cc | 2 + sql/sql_show.cc | 245 ++++- sql/sql_type.cc | 20 +- sql/sql_yacc.yy | 14 +- sql/table.h | 13 +- 52 files changed, 1784 insertions(+), 777 deletions(-) create mode 100644 mysql-test/main/information_schema-big_embedded.result create mode 100644 mysql-test/main/information_schema-big_embedded.test create mode 100644 mysql-test/suite/rpl/r/master_last_event_time_row.result create mode 100644 mysql-test/suite/rpl/r/master_last_event_time_stmt.result create mode 100644 mysql-test/suite/rpl/t/master_last_event_time.inc create mode 100644 mysql-test/suite/rpl/t/master_last_event_time_row.cnf create mode 100644 mysql-test/suite/rpl/t/master_last_event_time_row.test create mode 100644 mysql-test/suite/rpl/t/master_last_event_time_stmt.cnf create mode 100644 mysql-test/suite/rpl/t/master_last_event_time_stmt.test diff --git a/mysql-test/include/show_slave_status.inc b/mysql-test/include/show_slave_status.inc index 429ed8a5abd..c91e76f1ec5 100644 --- a/mysql-test/include/show_slave_status.inc +++ b/mysql-test/include/show_slave_status.inc @@ -97,7 +97,7 @@ if ($all_slaves_status) { --die Bug in test case: Both $all_slaves_status and $slave_name are set. } - --let $_show_query=SHOW ALL SLAVES STATUS + --let $_show_query=SELECT * from information_schema.slave_status } if ($slave_name) { diff --git a/mysql-test/main/grant_slave_monitor.result b/mysql-test/main/grant_slave_monitor.result index 61c4ca308cc..75fb07fe317 100644 --- a/mysql-test/main/grant_slave_monitor.result +++ b/mysql-test/main/grant_slave_monitor.result @@ -13,6 +13,8 @@ GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, F # SHOW SLAVE STATUS; ERROR 42000: Access denied; you need (at least one of) the SLAVE MONITOR privilege(s) for this operation +SELECT * from information_schema.slave_status; +ERROR 42000: Access denied; you need (at least one of) the SLAVE MONITOR privilege(s) for this operation # # Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW RELAYLOG EVENTS # Expected error: Access denied; you need (at least one of) the REPLICA MONITOR diff --git a/mysql-test/main/grant_slave_monitor.test b/mysql-test/main/grant_slave_monitor.test index b5e65ef2cb4..1be9b7e944d 100644 --- a/mysql-test/main/grant_slave_monitor.test +++ b/mysql-test/main/grant_slave_monitor.test @@ -38,6 +38,8 @@ SHOW GRANTS; --echo # --error ER_SPECIFIC_ACCESS_DENIED_ERROR SHOW SLAVE STATUS; +--error ER_SPECIFIC_ACCESS_DENIED_ERROR +SELECT * from information_schema.slave_status; --echo # --echo # Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW RELAYLOG EVENTS diff --git a/mysql-test/main/information_schema-big.result b/mysql-test/main/information_schema-big.result index 8c93cc79f64..cad4078deb5 100644 --- a/mysql-test/main/information_schema-big.result +++ b/mysql-test/main/information_schema-big.result @@ -52,6 +52,7 @@ SCHEMA_PRIVILEGES TABLE_SCHEMA SEQUENCES SEQUENCE_SCHEMA SESSION_STATUS VARIABLE_NAME SESSION_VARIABLES VARIABLE_NAME +SLAVE_STATUS Connection_name SPATIAL_REF_SYS SRID SQL_FUNCTIONS FUNCTION STATISTICS TABLE_SCHEMA @@ -117,6 +118,7 @@ SCHEMA_PRIVILEGES TABLE_SCHEMA SEQUENCES SEQUENCE_SCHEMA SESSION_STATUS VARIABLE_NAME SESSION_VARIABLES VARIABLE_NAME +SLAVE_STATUS Connection_name SPATIAL_REF_SYS SRID SQL_FUNCTIONS FUNCTION STATISTICS TABLE_SCHEMA diff --git a/mysql-test/main/information_schema-big.test b/mysql-test/main/information_schema-big.test index 89d7d877fb2..8baaaab6e0c 100644 --- a/mysql-test/main/information_schema-big.test +++ b/mysql-test/main/information_schema-big.test @@ -1,4 +1,5 @@ -- source include/have_innodb.inc +-- source include/not_embedded.inc # check that CSV engine was compiled in, as the result of the test depends # on the presence of the log tables (which are CSV-based). diff --git a/mysql-test/main/information_schema-big_embedded.result b/mysql-test/main/information_schema-big_embedded.result new file mode 100644 index 00000000000..8c93cc79f64 --- /dev/null +++ b/mysql-test/main/information_schema-big_embedded.result @@ -0,0 +1,133 @@ +# +# Bug#18925: subqueries with MIN/MAX functions on INFORMATION_SCHEMA +# +SELECT t.table_name, c1.column_name +FROM information_schema.tables t +INNER JOIN +information_schema.columns c1 +ON t.table_schema = c1.table_schema AND +t.table_name = c1.table_name +WHERE t.table_schema = 'information_schema' AND +c1.ordinal_position = +( SELECT COALESCE(MIN(c2.ordinal_position),1) +FROM information_schema.columns c2 +WHERE c2.table_schema = t.table_schema AND +c2.table_name = t.table_name AND +c2.column_name LIKE '%SCHEMA%' + ) +AND t.table_name NOT LIKE 'innodb%' and t.table_name NOT LIKE "OPTIMIZER_TRACE%"; +table_name column_name +ALL_PLUGINS PLUGIN_NAME +APPLICABLE_ROLES GRANTEE +CHARACTER_SETS CHARACTER_SET_NAME +CHECK_CONSTRAINTS CONSTRAINT_SCHEMA +CLIENT_STATISTICS CLIENT +COLLATIONS COLLATION_NAME +COLLATION_CHARACTER_SET_APPLICABILITY COLLATION_NAME +COLUMNS TABLE_SCHEMA +COLUMN_PRIVILEGES TABLE_SCHEMA +ENABLED_ROLES ROLE_NAME +ENGINES ENGINE +EVENTS EVENT_SCHEMA +FILES TABLE_SCHEMA +GEOMETRY_COLUMNS F_TABLE_SCHEMA +GLOBAL_STATUS VARIABLE_NAME +GLOBAL_VARIABLES VARIABLE_NAME +INDEX_STATISTICS TABLE_SCHEMA +KEYWORDS WORD +KEY_CACHES KEY_CACHE_NAME +KEY_COLUMN_USAGE CONSTRAINT_SCHEMA +KEY_PERIOD_USAGE CONSTRAINT_SCHEMA +OPTIMIZER_COSTS ENGINE +PARAMETERS SPECIFIC_SCHEMA +PARTITIONS TABLE_SCHEMA +PERIODS TABLE_SCHEMA +PLUGINS PLUGIN_NAME +PROCESSLIST ID +PROFILING QUERY_ID +REFERENTIAL_CONSTRAINTS CONSTRAINT_SCHEMA +ROUTINES ROUTINE_SCHEMA +SCHEMATA SCHEMA_NAME +SCHEMA_PRIVILEGES TABLE_SCHEMA +SEQUENCES SEQUENCE_SCHEMA +SESSION_STATUS VARIABLE_NAME +SESSION_VARIABLES VARIABLE_NAME +SPATIAL_REF_SYS SRID +SQL_FUNCTIONS FUNCTION +STATISTICS TABLE_SCHEMA +SYSTEM_VARIABLES VARIABLE_NAME +TABLES TABLE_SCHEMA +TABLESPACES TABLESPACE_NAME +TABLE_CONSTRAINTS CONSTRAINT_SCHEMA +TABLE_PRIVILEGES TABLE_SCHEMA +TABLE_STATISTICS TABLE_SCHEMA +TRIGGERS TRIGGER_SCHEMA +USERS USER +USER_PRIVILEGES GRANTEE +USER_STATISTICS USER +VIEWS TABLE_SCHEMA +SELECT t.table_name, c1.column_name +FROM information_schema.tables t +INNER JOIN +information_schema.columns c1 +ON t.table_schema = c1.table_schema AND +t.table_name = c1.table_name +WHERE t.table_schema = 'information_schema' AND +c1.ordinal_position = +( SELECT COALESCE(MIN(c2.ordinal_position),1) +FROM information_schema.columns c2 +WHERE c2.table_schema = 'information_schema' AND +c2.table_name = t.table_name AND +c2.column_name LIKE '%SCHEMA%' + ) +AND t.table_name NOT LIKE 'innodb%' and t.table_name NOT LIKE "OPTIMIZER_TRACE%"; +table_name column_name +ALL_PLUGINS PLUGIN_NAME +APPLICABLE_ROLES GRANTEE +CHARACTER_SETS CHARACTER_SET_NAME +CHECK_CONSTRAINTS CONSTRAINT_SCHEMA +CLIENT_STATISTICS CLIENT +COLLATIONS COLLATION_NAME +COLLATION_CHARACTER_SET_APPLICABILITY COLLATION_NAME +COLUMNS TABLE_SCHEMA +COLUMN_PRIVILEGES TABLE_SCHEMA +ENABLED_ROLES ROLE_NAME +ENGINES ENGINE +EVENTS EVENT_SCHEMA +FILES TABLE_SCHEMA +GEOMETRY_COLUMNS F_TABLE_SCHEMA +GLOBAL_STATUS VARIABLE_NAME +GLOBAL_VARIABLES VARIABLE_NAME +INDEX_STATISTICS TABLE_SCHEMA +KEYWORDS WORD +KEY_CACHES KEY_CACHE_NAME +KEY_COLUMN_USAGE CONSTRAINT_SCHEMA +KEY_PERIOD_USAGE CONSTRAINT_SCHEMA +OPTIMIZER_COSTS ENGINE +PARAMETERS SPECIFIC_SCHEMA +PARTITIONS TABLE_SCHEMA +PERIODS TABLE_SCHEMA +PLUGINS PLUGIN_NAME +PROCESSLIST ID +PROFILING QUERY_ID +REFERENTIAL_CONSTRAINTS CONSTRAINT_SCHEMA +ROUTINES ROUTINE_SCHEMA +SCHEMATA SCHEMA_NAME +SCHEMA_PRIVILEGES TABLE_SCHEMA +SEQUENCES SEQUENCE_SCHEMA +SESSION_STATUS VARIABLE_NAME +SESSION_VARIABLES VARIABLE_NAME +SPATIAL_REF_SYS SRID +SQL_FUNCTIONS FUNCTION +STATISTICS TABLE_SCHEMA +SYSTEM_VARIABLES VARIABLE_NAME +TABLES TABLE_SCHEMA +TABLESPACES TABLESPACE_NAME +TABLE_CONSTRAINTS CONSTRAINT_SCHEMA +TABLE_PRIVILEGES TABLE_SCHEMA +TABLE_STATISTICS TABLE_SCHEMA +TRIGGERS TRIGGER_SCHEMA +USERS USER +USER_PRIVILEGES GRANTEE +USER_STATISTICS USER +VIEWS TABLE_SCHEMA diff --git a/mysql-test/main/information_schema-big_embedded.test b/mysql-test/main/information_schema-big_embedded.test new file mode 100644 index 00000000000..0817a6e2be7 --- /dev/null +++ b/mysql-test/main/information_schema-big_embedded.test @@ -0,0 +1,43 @@ +-- source include/have_innodb.inc +-- source include/is_embedded.inc + +# check that CSV engine was compiled in, as the result of the test depends +# on the presence of the log tables (which are CSV-based). +--source include/have_csv.inc + +--echo # +--echo # Bug#18925: subqueries with MIN/MAX functions on INFORMATION_SCHEMA +--echo # + +--sorted_result +SELECT t.table_name, c1.column_name + FROM information_schema.tables t + INNER JOIN + information_schema.columns c1 + ON t.table_schema = c1.table_schema AND + t.table_name = c1.table_name + WHERE t.table_schema = 'information_schema' AND + c1.ordinal_position = + ( SELECT COALESCE(MIN(c2.ordinal_position),1) + FROM information_schema.columns c2 + WHERE c2.table_schema = t.table_schema AND + c2.table_name = t.table_name AND + c2.column_name LIKE '%SCHEMA%' + ) + AND t.table_name NOT LIKE 'innodb%' and t.table_name NOT LIKE "OPTIMIZER_TRACE%"; +--sorted_result +SELECT t.table_name, c1.column_name + FROM information_schema.tables t + INNER JOIN + information_schema.columns c1 + ON t.table_schema = c1.table_schema AND + t.table_name = c1.table_name + WHERE t.table_schema = 'information_schema' AND + c1.ordinal_position = + ( SELECT COALESCE(MIN(c2.ordinal_position),1) + FROM information_schema.columns c2 + WHERE c2.table_schema = 'information_schema' AND + c2.table_name = t.table_name AND + c2.column_name LIKE '%SCHEMA%' + ) + AND t.table_name NOT LIKE 'innodb%' and t.table_name NOT LIKE "OPTIMIZER_TRACE%"; diff --git a/mysql-test/main/information_schema.result b/mysql-test/main/information_schema.result index e0bee79ee3c..7c939064410 100644 --- a/mysql-test/main/information_schema.result +++ b/mysql-test/main/information_schema.result @@ -87,6 +87,7 @@ SCHEMA_PRIVILEGES SEQUENCES SESSION_STATUS SESSION_VARIABLES +SLAVE_STATUS SPATIAL_REF_SYS SQL_FUNCTIONS STATISTICS @@ -960,6 +961,8 @@ PARTITIONS UPDATE_TIME datetime PARTITIONS CHECK_TIME datetime ROUTINES CREATED datetime ROUTINES LAST_ALTERED datetime +SLAVE_STATUS Master_last_event_time datetime +SLAVE_STATUS Slave_last_event_time datetime TABLES CREATE_TIME datetime TABLES UPDATE_TIME datetime TABLES CHECK_TIME datetime @@ -1336,8 +1339,8 @@ table_schema='information_schema' and or column_type = 'varchar(27)') group by column_type order by num; column_type group_concat(table_schema, '.', table_name) num -varchar(7) information_schema.ROUTINES,information_schema.VIEWS 2 -varchar(20) information_schema.ALL_PLUGINS,information_schema.ALL_PLUGINS,information_schema.ALL_PLUGINS,information_schema.FILES,information_schema.FILES,information_schema.PLUGINS,information_schema.PLUGINS,information_schema.PLUGINS,information_schema.PROFILING 9 +varchar(7) information_schema.ROUTINES,information_schema.VIEWS,information_schema.SLAVE_STATUS 3 +varchar(20) information_schema.ALL_PLUGINS,information_schema.ALL_PLUGINS,information_schema.ALL_PLUGINS,information_schema.FILES,information_schema.FILES,information_schema.PLUGINS,information_schema.PLUGINS,information_schema.PLUGINS,information_schema.PROFILING,information_schema.SLAVE_STATUS 10 create table t1(f1 char(1) not null, f2 char(9) not null) default character set utf8; select CHARACTER_MAXIMUM_LENGTH, CHARACTER_OCTET_LENGTH from diff --git a/mysql-test/main/information_schema_all_engines.result b/mysql-test/main/information_schema_all_engines.result index b413c63b59d..745d03234e8 100644 --- a/mysql-test/main/information_schema_all_engines.result +++ b/mysql-test/main/information_schema_all_engines.result @@ -58,6 +58,7 @@ SCHEMA_PRIVILEGES SEQUENCES SESSION_STATUS SESSION_VARIABLES +SLAVE_STATUS SPATIAL_REF_SYS SQL_FUNCTIONS STATISTICS @@ -144,6 +145,7 @@ SCHEMA_PRIVILEGES TABLE_SCHEMA SEQUENCES SEQUENCE_SCHEMA SESSION_STATUS VARIABLE_NAME SESSION_VARIABLES VARIABLE_NAME +SLAVE_STATUS Connection_name SPATIAL_REF_SYS SRID SQL_FUNCTIONS FUNCTION STATISTICS TABLE_SCHEMA @@ -230,6 +232,7 @@ SCHEMA_PRIVILEGES TABLE_SCHEMA SEQUENCES SEQUENCE_SCHEMA SESSION_STATUS VARIABLE_NAME SESSION_VARIABLES VARIABLE_NAME +SLAVE_STATUS Connection_name SPATIAL_REF_SYS SRID SQL_FUNCTIONS FUNCTION STATISTICS TABLE_SCHEMA @@ -278,8 +281,6 @@ ENGINES information_schema.ENGINES 1 EVENTS information_schema.EVENTS 1 FILES information_schema.FILES 1 GEOMETRY_COLUMNS information_schema.GEOMETRY_COLUMNS 1 -GLOBAL_STATUS information_schema.GLOBAL_STATUS 1 -GLOBAL_VARIABLES information_schema.GLOBAL_VARIABLES 1 INDEX_STATISTICS information_schema.INDEX_STATISTICS 1 INNODB_BUFFER_PAGE information_schema.INNODB_BUFFER_PAGE 1 INNODB_BUFFER_PAGE_LRU information_schema.INNODB_BUFFER_PAGE_LRU 1 @@ -318,8 +319,7 @@ ROUTINES information_schema.ROUTINES 1 SCHEMATA information_schema.SCHEMATA 1 SCHEMA_PRIVILEGES information_schema.SCHEMA_PRIVILEGES 1 SEQUENCES information_schema.SEQUENCES 1 -SESSION_STATUS information_schema.SESSION_STATUS 1 -SESSION_VARIABLES information_schema.SESSION_VARIABLES 1 +SLAVE_STATUS information_schema.SLAVE_STATUS 1 SPATIAL_REF_SYS information_schema.SPATIAL_REF_SYS 1 STATISTICS information_schema.STATISTICS 1 SYSTEM_VARIABLES information_schema.SYSTEM_VARIABLES 1 @@ -395,6 +395,7 @@ Database: information_schema | SEQUENCES | | SESSION_STATUS | | SESSION_VARIABLES | +| SLAVE_STATUS | | SPATIAL_REF_SYS | | SQL_FUNCTIONS | | STATISTICS | @@ -471,6 +472,7 @@ Database: INFORMATION_SCHEMA | SEQUENCES | | SESSION_STATUS | | SESSION_VARIABLES | +| SLAVE_STATUS | | SPATIAL_REF_SYS | | SQL_FUNCTIONS | | STATISTICS | @@ -493,5 +495,5 @@ Wildcard: inf_rmation_schema | information_schema | SELECT table_schema, count(*) FROM information_schema.TABLES WHERE table_schema IN ('mysql', 'INFORMATION_SCHEMA', 'test', 'mysqltest') GROUP BY TABLE_SCHEMA; table_schema count(*) -information_schema 71 +information_schema 72 mysql 31 diff --git a/mysql-test/main/ps_change_master.test b/mysql-test/main/ps_change_master.test index 19040917bde..677ad46a787 100644 --- a/mysql-test/main/ps_change_master.test +++ b/mysql-test/main/ps_change_master.test @@ -32,8 +32,8 @@ EXECUTE stmt; EXECUTE stmt; DEALLOCATE PREPARE stmt; -let $master_host= query_get_value(SHOW SLAVE STATUS, Master_Host, 1); -let $master_user= query_get_value(SHOW SLAVE STATUS, Master_User, 1); +let $master_host= `select Master_Host from information_schema.slave_status`; +let $master_user= `select Master_user from information_schema.slave_status`; --echo # Master_Host : $master_host --echo # Master_User : $master_user diff --git a/mysql-test/suite/funcs_1/r/is_columns_is.result b/mysql-test/suite/funcs_1/r/is_columns_is.result index 287ea2d905a..85a02d6a6ef 100644 --- a/mysql-test/suite/funcs_1/r/is_columns_is.result +++ b/mysql-test/suite/funcs_1/r/is_columns_is.result @@ -388,6 +388,71 @@ def information_schema SESSION_STATUS VARIABLE_NAME 1 NULL NO varchar 64 192 NUL def information_schema SESSION_STATUS VARIABLE_VALUE 2 NULL NO varchar 4096 12288 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(4096) select NEVER NULL NO NO def information_schema SESSION_VARIABLES VARIABLE_NAME 1 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL NO NO def information_schema SESSION_VARIABLES VARIABLE_VALUE 2 NULL NO varchar 4096 12288 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(4096) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Connection_name 1 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Connect_Retry 7 NULL NO int NULL NULL 10 0 NULL NULL NULL int(10) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Executed_log_entries 59 NULL NO int NULL NULL 10 0 NULL NULL NULL int(10) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Exec_Master_Log_Pos 24 NULL NO bigint NULL NULL 20 0 NULL NULL NULL bigint(10) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Gtid_IO_Pos 46 NULL NO varchar 1024 3072 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(1024) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Gtid_Slave_Pos 62 NULL NO varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Last_Errno 21 NULL NO int NULL NULL 10 0 NULL NULL NULL int(4) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Last_Error 22 NULL YES varchar 20 60 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(20) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Last_IO_Errno 37 NULL NO int NULL NULL 10 0 NULL NULL NULL int(4) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Last_IO_Error 38 NULL YES varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Last_SQL_Errno 39 NULL NO int NULL NULL 10 0 NULL NULL NULL int(4) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Last_SQL_Error 40 NULL YES varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_Host 4 NULL YES varchar 255 765 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(255) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_last_event_time 63 NULL YES datetime NULL NULL NULL NULL 0 NULL NULL datetime select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_Log_File 8 NULL NO varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_Port 6 NULL NO int NULL NULL 10 0 NULL NULL NULL int(7) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_Server_Id 42 NULL NO int NULL NULL 10 0 NULL NULL NULL int(10) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_Slave_time_diff 65 NULL YES bigint NULL NULL 19 0 NULL NULL NULL bigint(10) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_SSL_Allowed 29 NULL YES varchar 7 21 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(7) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_SSL_CA_File 30 NULL YES varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_SSL_CA_Path 31 NULL YES varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_SSL_Cert 32 NULL YES varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_SSL_Cipher 33 NULL YES varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_SSL_Crl 43 NULL YES varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_SSL_Crlpath 44 NULL YES varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_SSL_Key 34 NULL YES varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_SSL_Verify_Server_Cert 36 NULL NO varchar 3 9 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(3) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Master_User 5 NULL YES varchar 384 1152 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(384) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Max_relay_log_size 58 NULL NO bigint NULL NULL 20 0 NULL NULL NULL bigint(10) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Parallel_Mode 49 NULL NO varchar 15 45 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(15) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Read_Master_Log_Pos 9 NULL NO bigint NULL NULL 20 0 NULL NULL NULL bigint(10) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Relay_Log_File 10 NULL NO varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Relay_Log_Pos 11 NULL NO bigint NULL NULL 20 0 NULL NULL NULL bigint(10) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Relay_Log_Space 25 NULL NO bigint NULL NULL 20 0 NULL NULL NULL bigint(10) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Relay_Master_Log_File 12 NULL NO varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Replicate_Do_DB 15 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Replicate_Do_Domain_Ids 47 NULL NO varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Replicate_Do_Table 17 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Replicate_Ignore_DB 16 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Replicate_Ignore_Domain_Ids 48 NULL NO varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Replicate_Ignore_Server_Ids 41 NULL NO varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Replicate_Ignore_Table 18 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Replicate_Rewrite_DB 56 NULL NO varchar 1024 3072 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(1024) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Replicate_Wild_Do_Table 19 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Replicate_Wild_Ignore_Table 20 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Retried_transactions 57 NULL NO int NULL NULL 10 0 NULL NULL NULL int(10) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Seconds_Behind_Master 35 NULL YES bigint NULL NULL 19 0 NULL NULL NULL bigint(10) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Skip_Counter 23 NULL NO int NULL NULL 10 0 NULL NULL NULL int(10) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Slave_DDL_Groups 53 NULL NO bigint NULL NULL 20 0 NULL NULL NULL bigint(20) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Slave_heartbeat_period 61 NULL NO float NULL NULL 9 3 NULL NULL NULL float(9,3) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Slave_IO_Running 13 NULL NO varchar 10 30 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(10) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Slave_IO_State 3 NULL YES varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Slave_last_event_time 64 NULL YES datetime NULL NULL NULL NULL 0 NULL NULL datetime select NEVER NULL NO NO +def information_schema SLAVE_STATUS Slave_Non_Transactional_Groups 54 NULL NO bigint NULL NULL 20 0 NULL NULL NULL bigint(20) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Slave_received_heartbeats 60 NULL NO int NULL NULL 10 0 NULL NULL NULL int(10) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Slave_SQL_Running 14 NULL NO varchar 3 9 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(3) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Slave_SQL_Running_State 52 NULL YES varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Slave_SQL_State 2 NULL YES varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Slave_Transactional_Groups 55 NULL NO bigint NULL NULL 20 0 NULL NULL NULL bigint(20) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS SQL_Delay 50 NULL NO int NULL NULL 10 0 NULL NULL NULL int(10) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS SQL_Remaining_Delay 51 NULL YES int NULL NULL 10 0 NULL NULL NULL int(10) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Until_Condition 26 NULL NO varchar 6 18 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(6) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Until_Log_File 27 NULL YES varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO +def information_schema SLAVE_STATUS Until_Log_Pos 28 NULL NO bigint NULL NULL 20 0 NULL NULL NULL bigint(10) unsigned select NEVER NULL NO NO +def information_schema SLAVE_STATUS Using_Gtid 45 NULL YES varchar 15 45 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(15) select NEVER NULL NO NO def information_schema SPATIAL_REF_SYS AUTH_NAME 2 NULL NO varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL NO NO def information_schema SPATIAL_REF_SYS AUTH_SRID 3 NULL NO int NULL NULL 10 0 NULL NULL NULL int(5) select NEVER NULL NO NO def information_schema SPATIAL_REF_SYS SRID 1 NULL NO smallint NULL NULL 5 0 NULL NULL NULL smallint(5) select NEVER NULL NO NO @@ -589,6 +654,7 @@ NULL bigint NULL NULL NULL datetime NULL NULL NULL decimal NULL NULL NULL double NULL NULL +NULL float NULL NULL NULL int NULL NULL NULL smallint NULL NULL NULL tinyint NULL NULL @@ -994,6 +1060,71 @@ NULL information_schema SEQUENCES INCREMENT bigint NULL NULL NULL NULL bigint(21 3.0000 information_schema SESSION_STATUS VARIABLE_VALUE varchar 4096 12288 utf8mb3 utf8mb3_general_ci varchar(4096) 3.0000 information_schema SESSION_VARIABLES VARIABLE_NAME varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) 3.0000 information_schema SESSION_VARIABLES VARIABLE_VALUE varchar 4096 12288 utf8mb3 utf8mb3_general_ci varchar(4096) +3.0000 information_schema SLAVE_STATUS Connection_name varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) +3.0000 information_schema SLAVE_STATUS Slave_SQL_State varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) +3.0000 information_schema SLAVE_STATUS Slave_IO_State varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) +3.0000 information_schema SLAVE_STATUS Master_Host varchar 255 765 utf8mb3 utf8mb3_general_ci varchar(255) +3.0000 information_schema SLAVE_STATUS Master_User varchar 384 1152 utf8mb3 utf8mb3_general_ci varchar(384) +NULL information_schema SLAVE_STATUS Master_Port int NULL NULL NULL NULL int(7) unsigned +NULL information_schema SLAVE_STATUS Connect_Retry int NULL NULL NULL NULL int(10) +3.0000 information_schema SLAVE_STATUS Master_Log_File varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +NULL information_schema SLAVE_STATUS Read_Master_Log_Pos bigint NULL NULL NULL NULL bigint(10) unsigned +3.0000 information_schema SLAVE_STATUS Relay_Log_File varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +NULL information_schema SLAVE_STATUS Relay_Log_Pos bigint NULL NULL NULL NULL bigint(10) unsigned +3.0000 information_schema SLAVE_STATUS Relay_Master_Log_File varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +3.0000 information_schema SLAVE_STATUS Slave_IO_Running varchar 10 30 utf8mb3 utf8mb3_general_ci varchar(10) +3.0000 information_schema SLAVE_STATUS Slave_SQL_Running varchar 3 9 utf8mb3 utf8mb3_general_ci varchar(3) +3.0000 information_schema SLAVE_STATUS Replicate_Do_DB varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) +3.0000 information_schema SLAVE_STATUS Replicate_Ignore_DB varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) +3.0000 information_schema SLAVE_STATUS Replicate_Do_Table varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) +3.0000 information_schema SLAVE_STATUS Replicate_Ignore_Table varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) +3.0000 information_schema SLAVE_STATUS Replicate_Wild_Do_Table varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) +3.0000 information_schema SLAVE_STATUS Replicate_Wild_Ignore_Table varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) +NULL information_schema SLAVE_STATUS Last_Errno int NULL NULL NULL NULL int(4) +3.0000 information_schema SLAVE_STATUS Last_Error varchar 20 60 utf8mb3 utf8mb3_general_ci varchar(20) +NULL information_schema SLAVE_STATUS Skip_Counter int NULL NULL NULL NULL int(10) unsigned +NULL information_schema SLAVE_STATUS Exec_Master_Log_Pos bigint NULL NULL NULL NULL bigint(10) unsigned +NULL information_schema SLAVE_STATUS Relay_Log_Space bigint NULL NULL NULL NULL bigint(10) unsigned +3.0000 information_schema SLAVE_STATUS Until_Condition varchar 6 18 utf8mb3 utf8mb3_general_ci varchar(6) +3.0000 information_schema SLAVE_STATUS Until_Log_File varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +NULL information_schema SLAVE_STATUS Until_Log_Pos bigint NULL NULL NULL NULL bigint(10) unsigned +3.0000 information_schema SLAVE_STATUS Master_SSL_Allowed varchar 7 21 utf8mb3 utf8mb3_general_ci varchar(7) +3.0000 information_schema SLAVE_STATUS Master_SSL_CA_File varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +3.0000 information_schema SLAVE_STATUS Master_SSL_CA_Path varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +3.0000 information_schema SLAVE_STATUS Master_SSL_Cert varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +3.0000 information_schema SLAVE_STATUS Master_SSL_Cipher varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +3.0000 information_schema SLAVE_STATUS Master_SSL_Key varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +NULL information_schema SLAVE_STATUS Seconds_Behind_Master bigint NULL NULL NULL NULL bigint(10) +3.0000 information_schema SLAVE_STATUS Master_SSL_Verify_Server_Cert varchar 3 9 utf8mb3 utf8mb3_general_ci varchar(3) +NULL information_schema SLAVE_STATUS Last_IO_Errno int NULL NULL NULL NULL int(4) +3.0000 information_schema SLAVE_STATUS Last_IO_Error varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +NULL information_schema SLAVE_STATUS Last_SQL_Errno int NULL NULL NULL NULL int(4) +3.0000 information_schema SLAVE_STATUS Last_SQL_Error varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +3.0000 information_schema SLAVE_STATUS Replicate_Ignore_Server_Ids varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +NULL information_schema SLAVE_STATUS Master_Server_Id int NULL NULL NULL NULL int(10) unsigned +3.0000 information_schema SLAVE_STATUS Master_SSL_Crl varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +3.0000 information_schema SLAVE_STATUS Master_SSL_Crlpath varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +3.0000 information_schema SLAVE_STATUS Using_Gtid varchar 15 45 utf8mb3 utf8mb3_general_ci varchar(15) +3.0000 information_schema SLAVE_STATUS Gtid_IO_Pos varchar 1024 3072 utf8mb3 utf8mb3_general_ci varchar(1024) +3.0000 information_schema SLAVE_STATUS Replicate_Do_Domain_Ids varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +3.0000 information_schema SLAVE_STATUS Replicate_Ignore_Domain_Ids varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +3.0000 information_schema SLAVE_STATUS Parallel_Mode varchar 15 45 utf8mb3 utf8mb3_general_ci varchar(15) +NULL information_schema SLAVE_STATUS SQL_Delay int NULL NULL NULL NULL int(10) unsigned +NULL information_schema SLAVE_STATUS SQL_Remaining_Delay int NULL NULL NULL NULL int(10) unsigned +3.0000 information_schema SLAVE_STATUS Slave_SQL_Running_State varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) +NULL information_schema SLAVE_STATUS Slave_DDL_Groups bigint NULL NULL NULL NULL bigint(20) unsigned +NULL information_schema SLAVE_STATUS Slave_Non_Transactional_Groups bigint NULL NULL NULL NULL bigint(20) unsigned +NULL information_schema SLAVE_STATUS Slave_Transactional_Groups bigint NULL NULL NULL NULL bigint(20) unsigned +3.0000 information_schema SLAVE_STATUS Replicate_Rewrite_DB varchar 1024 3072 utf8mb3 utf8mb3_general_ci varchar(1024) +NULL information_schema SLAVE_STATUS Retried_transactions int NULL NULL NULL NULL int(10) unsigned +NULL information_schema SLAVE_STATUS Max_relay_log_size bigint NULL NULL NULL NULL bigint(10) unsigned +NULL information_schema SLAVE_STATUS Executed_log_entries int NULL NULL NULL NULL int(10) unsigned +NULL information_schema SLAVE_STATUS Slave_received_heartbeats int NULL NULL NULL NULL int(10) unsigned +NULL information_schema SLAVE_STATUS Slave_heartbeat_period float NULL NULL NULL NULL float(9,3) +3.0000 information_schema SLAVE_STATUS Gtid_Slave_Pos varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) +NULL information_schema SLAVE_STATUS Master_last_event_time datetime NULL NULL NULL NULL datetime +NULL information_schema SLAVE_STATUS Slave_last_event_time datetime NULL NULL NULL NULL datetime +NULL information_schema SLAVE_STATUS Master_Slave_time_diff bigint NULL NULL NULL NULL bigint(10) NULL information_schema SPATIAL_REF_SYS SRID smallint NULL NULL NULL NULL smallint(5) 3.0000 information_schema SPATIAL_REF_SYS AUTH_NAME varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) NULL information_schema SPATIAL_REF_SYS AUTH_SRID int NULL NULL NULL NULL int(5) diff --git a/mysql-test/suite/funcs_1/r/is_tables_is.result b/mysql-test/suite/funcs_1/r/is_tables_is.result index c3c53e149b4..21ab57f7595 100644 --- a/mysql-test/suite/funcs_1/r/is_tables_is.result +++ b/mysql-test/suite/funcs_1/r/is_tables_is.result @@ -889,6 +889,31 @@ user_comment Separator ----------------------------------------------------- TABLE_CATALOG def TABLE_SCHEMA information_schema +TABLE_NAME SLAVE_STATUS +TABLE_TYPE SYSTEM VIEW +ENGINE MEMORY +VERSION 11 +ROW_FORMAT Fixed +TABLE_ROWS #TBLR# +AVG_ROW_LENGTH #ARL# +DATA_LENGTH #DL# +MAX_DATA_LENGTH #MDL# +INDEX_LENGTH #IL# +DATA_FREE #DF# +AUTO_INCREMENT NULL +CREATE_TIME #CRT# +UPDATE_TIME #UT# +CHECK_TIME #CT# +TABLE_COLLATION utf8mb3_general_ci +CHECKSUM NULL +CREATE_OPTIONS #CO# +TABLE_COMMENT #TC# +MAX_INDEX_LENGTH #MIL# +TEMPORARY Y +user_comment +Separator ----------------------------------------------------- +TABLE_CATALOG def +TABLE_SCHEMA information_schema TABLE_NAME SPATIAL_REF_SYS TABLE_TYPE SYSTEM VIEW ENGINE MEMORY @@ -2130,6 +2155,31 @@ user_comment Separator ----------------------------------------------------- TABLE_CATALOG def TABLE_SCHEMA information_schema +TABLE_NAME SLAVE_STATUS +TABLE_TYPE SYSTEM VIEW +ENGINE MEMORY +VERSION 11 +ROW_FORMAT Fixed +TABLE_ROWS #TBLR# +AVG_ROW_LENGTH #ARL# +DATA_LENGTH #DL# +MAX_DATA_LENGTH #MDL# +INDEX_LENGTH #IL# +DATA_FREE #DF# +AUTO_INCREMENT NULL +CREATE_TIME #CRT# +UPDATE_TIME #UT# +CHECK_TIME #CT# +TABLE_COLLATION utf8mb3_general_ci +CHECKSUM NULL +CREATE_OPTIONS #CO# +TABLE_COMMENT #TC# +MAX_INDEX_LENGTH #MIL# +TEMPORARY Y +user_comment +Separator ----------------------------------------------------- +TABLE_CATALOG def +TABLE_SCHEMA information_schema TABLE_NAME SPATIAL_REF_SYS TABLE_TYPE SYSTEM VIEW ENGINE MEMORY diff --git a/mysql-test/suite/multi_source/info_logs.result b/mysql-test/suite/multi_source/info_logs.result index 36065116784..4dca25b8fc7 100644 --- a/mysql-test/suite/multi_source/info_logs.result +++ b/mysql-test/suite/multi_source/info_logs.result @@ -94,17 +94,17 @@ MASTER 2.2 # EOF # show all slaves status; -Connection_name Slave_SQL_State Slave_IO_State Master_Host Master_User Master_Port Connect_Retry Master_Log_File Read_Master_Log_Pos Relay_Log_File Relay_Log_Pos Relay_Master_Log_File Slave_IO_Running Slave_SQL_Running Replicate_Do_DB Replicate_Ignore_DB Replicate_Do_Table Replicate_Ignore_Table Replicate_Wild_Do_Table Replicate_Wild_Ignore_Table Last_Errno Last_Error Skip_Counter Exec_Master_Log_Pos Relay_Log_Space Until_Condition Until_Log_File Until_Log_Pos Master_SSL_Allowed Master_SSL_CA_File Master_SSL_CA_Path Master_SSL_Cert Master_SSL_Cipher Master_SSL_Key Seconds_Behind_Master Master_SSL_Verify_Server_Cert Last_IO_Errno Last_IO_Error Last_SQL_Errno Last_SQL_Error Replicate_Ignore_Server_Ids Master_Server_Id Master_SSL_Crl Master_SSL_Crlpath Using_Gtid Gtid_IO_Pos Replicate_Do_Domain_Ids Replicate_Ignore_Domain_Ids Parallel_Mode SQL_Delay SQL_Remaining_Delay Slave_SQL_Running_State Slave_DDL_Groups Slave_Non_Transactional_Groups Slave_Transactional_Groups Replicate_Rewrite_DB Retried_transactions Max_relay_log_size Executed_log_entries Slave_received_heartbeats Slave_heartbeat_period Gtid_Slave_Pos - Slave has read all relay log; waiting for more updates Waiting for master to send event 127.0.0.1 root MYPORT_1 60 master-bin.000001 <read_master_log_pos> relay.000002 <relay_log_pos> master-bin.000001 Yes Yes 0 0 <read_master_log_pos> <relay_log_space1> None 0 Yes 0 No 0 0 1 No optimistic 0 NULL Slave has read all relay log; waiting for more updates 0 0 0 0 1073741824 7 0 60.000 -MASTER 2.2 Slave has read all relay log; waiting for more updates Waiting for master to send event 127.0.0.1 root MYPORT_2 60 master-bin.000001 <read_master_log_pos> relay-master@00202@002e2.000002 <relay_log_pos> master-bin.000001 Yes Yes 0 0 <read_master_log_pos> <relay_log_space2> None 0 Yes 0 No 0 0 2 No optimistic 0 NULL Slave has read all relay log; waiting for more updates 0 0 0 0 1073741824 7 0 60.000 +Connection_name Slave_SQL_State Slave_IO_State Master_Host Master_User Master_Port Connect_Retry Master_Log_File Read_Master_Log_Pos Relay_Log_File Relay_Log_Pos Relay_Master_Log_File Slave_IO_Running Slave_SQL_Running Replicate_Do_DB Replicate_Ignore_DB Replicate_Do_Table Replicate_Ignore_Table Replicate_Wild_Do_Table Replicate_Wild_Ignore_Table Last_Errno Last_Error Skip_Counter Exec_Master_Log_Pos Relay_Log_Space Until_Condition Until_Log_File Until_Log_Pos Master_SSL_Allowed Master_SSL_CA_File Master_SSL_CA_Path Master_SSL_Cert Master_SSL_Cipher Master_SSL_Key Seconds_Behind_Master Master_SSL_Verify_Server_Cert Last_IO_Errno Last_IO_Error Last_SQL_Errno Last_SQL_Error Replicate_Ignore_Server_Ids Master_Server_Id Master_SSL_Crl Master_SSL_Crlpath Using_Gtid Gtid_IO_Pos Replicate_Do_Domain_Ids Replicate_Ignore_Domain_Ids Parallel_Mode SQL_Delay SQL_Remaining_Delay Slave_SQL_Running_State Slave_DDL_Groups Slave_Non_Transactional_Groups Slave_Transactional_Groups Replicate_Rewrite_DB Retried_transactions Max_relay_log_size Executed_log_entries Slave_received_heartbeats Slave_heartbeat_period Gtid_Slave_Pos Master_last_event_time Slave_last_event_time Master_Slave_time_diff + Slave has read all relay log; waiting for more updates Waiting for master to send event 127.0.0.1 root MYPORT_1 60 master-bin.000001 <read_master_log_pos> relay.000002 <relay_log_pos> master-bin.000001 Yes Yes 0 0 <read_master_log_pos> <relay_log_space1> None 0 Yes 0 No 0 0 1 No optimistic 0 NULL Slave has read all relay log; waiting for more updates 0 0 0 0 1073741824 7 0 60.000 NULL NULL NULL +MASTER 2.2 Slave has read all relay log; waiting for more updates Waiting for master to send event 127.0.0.1 root MYPORT_2 60 master-bin.000001 <read_master_log_pos> relay-master@00202@002e2.000002 <relay_log_pos> master-bin.000001 Yes Yes 0 0 <read_master_log_pos> <relay_log_space2> None 0 Yes 0 No 0 0 2 No optimistic 0 NULL Slave has read all relay log; waiting for more updates 0 0 0 0 1073741824 7 0 60.000 NULL NULL NULL include/wait_for_slave_to_start.inc set default_master_connection = 'MASTER 2.2'; include/wait_for_slave_to_start.inc set default_master_connection = ''; show all slaves status; -Connection_name Slave_SQL_State Slave_IO_State Master_Host Master_User Master_Port Connect_Retry Master_Log_File Read_Master_Log_Pos Relay_Log_File Relay_Log_Pos Relay_Master_Log_File Slave_IO_Running Slave_SQL_Running Replicate_Do_DB Replicate_Ignore_DB Replicate_Do_Table Replicate_Ignore_Table Replicate_Wild_Do_Table Replicate_Wild_Ignore_Table Last_Errno Last_Error Skip_Counter Exec_Master_Log_Pos Relay_Log_Space Until_Condition Until_Log_File Until_Log_Pos Master_SSL_Allowed Master_SSL_CA_File Master_SSL_CA_Path Master_SSL_Cert Master_SSL_Cipher Master_SSL_Key Seconds_Behind_Master Master_SSL_Verify_Server_Cert Last_IO_Errno Last_IO_Error Last_SQL_Errno Last_SQL_Error Replicate_Ignore_Server_Ids Master_Server_Id Master_SSL_Crl Master_SSL_Crlpath Using_Gtid Gtid_IO_Pos Replicate_Do_Domain_Ids Replicate_Ignore_Domain_Ids Parallel_Mode SQL_Delay SQL_Remaining_Delay Slave_SQL_Running_State Slave_DDL_Groups Slave_Non_Transactional_Groups Slave_Transactional_Groups Replicate_Rewrite_DB Retried_transactions Max_relay_log_size Executed_log_entries Slave_received_heartbeats Slave_heartbeat_period Gtid_Slave_Pos - Slave has read all relay log; waiting for more updates Waiting for master to send event 127.0.0.1 root MYPORT_1 60 master-bin.000001 <read_master_log_pos> relay.000004 <relay_log_pos> master-bin.000001 Yes Yes 0 0 <read_master_log_pos> <relay_log_space1> None 0 Yes 0 No 0 0 1 No optimistic 0 NULL Slave has read all relay log; waiting for more updates 0 0 0 0 1073741824 6 0 60.000 -MASTER 2.2 Slave has read all relay log; waiting for more updates Waiting for master to send event 127.0.0.1 root MYPORT_2 60 master-bin.000001 <read_master_log_pos> relay-master@00202@002e2.000004 <relay_log_pos> master-bin.000001 Yes Yes 0 0 <read_master_log_pos> <relay_log_space2> None 0 Yes 0 No 0 0 2 No optimistic 0 NULL Slave has read all relay log; waiting for more updates 0 0 0 0 1073741824 6 0 60.000 +Connection_name Slave_SQL_State Slave_IO_State Master_Host Master_User Master_Port Connect_Retry Master_Log_File Read_Master_Log_Pos Relay_Log_File Relay_Log_Pos Relay_Master_Log_File Slave_IO_Running Slave_SQL_Running Replicate_Do_DB Replicate_Ignore_DB Replicate_Do_Table Replicate_Ignore_Table Replicate_Wild_Do_Table Replicate_Wild_Ignore_Table Last_Errno Last_Error Skip_Counter Exec_Master_Log_Pos Relay_Log_Space Until_Condition Until_Log_File Until_Log_Pos Master_SSL_Allowed Master_SSL_CA_File Master_SSL_CA_Path Master_SSL_Cert Master_SSL_Cipher Master_SSL_Key Seconds_Behind_Master Master_SSL_Verify_Server_Cert Last_IO_Errno Last_IO_Error Last_SQL_Errno Last_SQL_Error Replicate_Ignore_Server_Ids Master_Server_Id Master_SSL_Crl Master_SSL_Crlpath Using_Gtid Gtid_IO_Pos Replicate_Do_Domain_Ids Replicate_Ignore_Domain_Ids Parallel_Mode SQL_Delay SQL_Remaining_Delay Slave_SQL_Running_State Slave_DDL_Groups Slave_Non_Transactional_Groups Slave_Transactional_Groups Replicate_Rewrite_DB Retried_transactions Max_relay_log_size Executed_log_entries Slave_received_heartbeats Slave_heartbeat_period Gtid_Slave_Pos Master_last_event_time Slave_last_event_time Master_Slave_time_diff + Slave has read all relay log; waiting for more updates Waiting for master to send event 127.0.0.1 root MYPORT_1 60 master-bin.000001 <read_master_log_pos> relay.000004 <relay_log_pos> master-bin.000001 Yes Yes 0 0 <read_master_log_pos> <relay_log_space1> None 0 Yes 0 No 0 0 1 No optimistic 0 NULL Slave has read all relay log; waiting for more updates 0 0 0 0 1073741824 6 0 60.000 NULL NULL NULL +MASTER 2.2 Slave has read all relay log; waiting for more updates Waiting for master to send event 127.0.0.1 root MYPORT_2 60 master-bin.000001 <read_master_log_pos> relay-master@00202@002e2.000004 <relay_log_pos> master-bin.000001 Yes Yes 0 0 <read_master_log_pos> <relay_log_space2> None 0 Yes 0 No 0 0 2 No optimistic 0 NULL Slave has read all relay log; waiting for more updates 0 0 0 0 1073741824 6 0 60.000 NULL NULL NULL # # List of files matching '*info*' pattern # after slave server restart diff --git a/mysql-test/suite/multi_source/multi_source_slave_alias_replica.result b/mysql-test/suite/multi_source/multi_source_slave_alias_replica.result index 4c721a17428..182d9687cb9 100644 --- a/mysql-test/suite/multi_source/multi_source_slave_alias_replica.result +++ b/mysql-test/suite/multi_source/multi_source_slave_alias_replica.result @@ -82,6 +82,9 @@ Executed_log_entries 7 Slave_received_heartbeats 0 Slave_heartbeat_period 60.000 Gtid_Slave_Pos +Master_last_event_time NULL +Slave_last_event_time NULL +Master_Slave_time_diff NULL Connection_name slave2 Slave_SQL_State Slave has read all relay log; waiting for more updates Slave_IO_State Waiting for master to send event @@ -144,6 +147,9 @@ Executed_log_entries 7 Slave_received_heartbeats 0 Slave_heartbeat_period 60.000 Gtid_Slave_Pos +Master_last_event_time NULL +Slave_last_event_time NULL +Master_Slave_time_diff NULL "Command: STOP ALL SLAVES --> STOP ALL REPLICAS" STOP ALL REPLICAS; Warnings: diff --git a/mysql-test/suite/multi_source/simple.result b/mysql-test/suite/multi_source/simple.result index f9f43d44ca7..006d0aacfbd 100644 --- a/mysql-test/suite/multi_source/simple.result +++ b/mysql-test/suite/multi_source/simple.result @@ -80,6 +80,9 @@ Executed_log_entries 7 Slave_received_heartbeats 0 Slave_heartbeat_period 60.000 Gtid_Slave_Pos +Master_last_event_time NULL +Slave_last_event_time NULL +Master_Slave_time_diff NULL Connection_name slave2 Slave_SQL_State Slave has read all relay log; waiting for more updates Slave_IO_State Waiting for master to send event @@ -142,6 +145,9 @@ Executed_log_entries 7 Slave_received_heartbeats 0 Slave_heartbeat_period 60.000 Gtid_Slave_Pos +Master_last_event_time NULL +Slave_last_event_time NULL +Master_Slave_time_diff NULL # # MDEV:16437: merge 5.7 P_S replication instrumentation and tables # @@ -327,6 +333,9 @@ Executed_log_entries 7 Slave_received_heartbeats 0 Slave_heartbeat_period 60.000 Gtid_Slave_Pos +Master_last_event_time NULL +Slave_last_event_time NULL +Master_Slave_time_diff NULL Connection_name slave2 Slave_SQL_State Slave has read all relay log; waiting for more updates Slave_IO_State Waiting for master to send event @@ -389,6 +398,9 @@ Executed_log_entries 7 Slave_received_heartbeats 0 Slave_heartbeat_period 60.000 Gtid_Slave_Pos +Master_last_event_time NULL +Slave_last_event_time NULL +Master_Slave_time_diff NULL reset slave 'slave1' all; show all slaves status; Connection_name slave2 @@ -453,6 +465,9 @@ Executed_log_entries 7 Slave_received_heartbeats 0 Slave_heartbeat_period 60.000 Gtid_Slave_Pos +Master_last_event_time NULL +Slave_last_event_time NULL +Master_Slave_time_diff NULL stop all slaves; Warnings: Note 1938 SLAVE 'slave2' stopped @@ -519,6 +534,9 @@ Executed_log_entries 7 Slave_received_heartbeats 0 Slave_heartbeat_period 60.000 Gtid_Slave_Pos +Master_last_event_time NULL +Slave_last_event_time NULL +Master_Slave_time_diff NULL stop all slaves; include/reset_master_slave.inc disconnect slave; diff --git a/mysql-test/suite/multi_source/syntax.result b/mysql-test/suite/multi_source/syntax.result index ce60f48c831..7475ff76e58 100644 --- a/mysql-test/suite/multi_source/syntax.result +++ b/mysql-test/suite/multi_source/syntax.result @@ -5,7 +5,7 @@ Slave_IO_State Master_Host Master_User Master_Port Connect_Retry Master_Log_File show slave '' status; Slave_IO_State Master_Host Master_User Master_Port Connect_Retry Master_Log_File Read_Master_Log_Pos Relay_Log_File Relay_Log_Pos Relay_Master_Log_File Slave_IO_Running Slave_SQL_Running Replicate_Do_DB Replicate_Ignore_DB Replicate_Do_Table Replicate_Ignore_Table Replicate_Wild_Do_Table Replicate_Wild_Ignore_Table Last_Errno Last_Error Skip_Counter Exec_Master_Log_Pos Relay_Log_Space Until_Condition Until_Log_File Until_Log_Pos Master_SSL_Allowed Master_SSL_CA_File Master_SSL_CA_Path Master_SSL_Cert Master_SSL_Cipher Master_SSL_Key Seconds_Behind_Master Master_SSL_Verify_Server_Cert Last_IO_Errno Last_IO_Error Last_SQL_Errno Last_SQL_Error Replicate_Ignore_Server_Ids Master_Server_Id Master_SSL_Crl Master_SSL_Crlpath Using_Gtid Gtid_IO_Pos Replicate_Do_Domain_Ids Replicate_Ignore_Domain_Ids Parallel_Mode SQL_Delay SQL_Remaining_Delay Slave_SQL_Running_State Slave_DDL_Groups Slave_Non_Transactional_Groups Slave_Transactional_Groups Replicate_Rewrite_DB show all slaves status; -Connection_name Slave_SQL_State Slave_IO_State Master_Host Master_User Master_Port Connect_Retry Master_Log_File Read_Master_Log_Pos Relay_Log_File Relay_Log_Pos Relay_Master_Log_File Slave_IO_Running Slave_SQL_Running Replicate_Do_DB Replicate_Ignore_DB Replicate_Do_Table Replicate_Ignore_Table Replicate_Wild_Do_Table Replicate_Wild_Ignore_Table Last_Errno Last_Error Skip_Counter Exec_Master_Log_Pos Relay_Log_Space Until_Condition Until_Log_File Until_Log_Pos Master_SSL_Allowed Master_SSL_CA_File Master_SSL_CA_Path Master_SSL_Cert Master_SSL_Cipher Master_SSL_Key Seconds_Behind_Master Master_SSL_Verify_Server_Cert Last_IO_Errno Last_IO_Error Last_SQL_Errno Last_SQL_Error Replicate_Ignore_Server_Ids Master_Server_Id Master_SSL_Crl Master_SSL_Crlpath Using_Gtid Gtid_IO_Pos Replicate_Do_Domain_Ids Replicate_Ignore_Domain_Ids Parallel_Mode SQL_Delay SQL_Remaining_Delay Slave_SQL_Running_State Slave_DDL_Groups Slave_Non_Transactional_Groups Slave_Transactional_Groups Replicate_Rewrite_DB Retried_transactions Max_relay_log_size Executed_log_entries Slave_received_heartbeats Slave_heartbeat_period Gtid_Slave_Pos +Connection_name Slave_SQL_State Slave_IO_State Master_Host Master_User Master_Port Connect_Retry Master_Log_File Read_Master_Log_Pos Relay_Log_File Relay_Log_Pos Relay_Master_Log_File Slave_IO_Running Slave_SQL_Running Replicate_Do_DB Replicate_Ignore_DB Replicate_Do_Table Replicate_Ignore_Table Replicate_Wild_Do_Table Replicate_Wild_Ignore_Table Last_Errno Last_Error Skip_Counter Exec_Master_Log_Pos Relay_Log_Space Until_Condition Until_Log_File Until_Log_Pos Master_SSL_Allowed Master_SSL_CA_File Master_SSL_CA_Path Master_SSL_Cert Master_SSL_Cipher Master_SSL_Key Seconds_Behind_Master Master_SSL_Verify_Server_Cert Last_IO_Errno Last_IO_Error Last_SQL_Errno Last_SQL_Error Replicate_Ignore_Server_Ids Master_Server_Id Master_SSL_Crl Master_SSL_Crlpath Using_Gtid Gtid_IO_Pos Replicate_Do_Domain_Ids Replicate_Ignore_Domain_Ids Parallel_Mode SQL_Delay SQL_Remaining_Delay Slave_SQL_Running_State Slave_DDL_Groups Slave_Non_Transactional_Groups Slave_Transactional_Groups Replicate_Rewrite_DB Retried_transactions Max_relay_log_size Executed_log_entries Slave_received_heartbeats Slave_heartbeat_period Gtid_Slave_Pos Master_last_event_time Slave_last_event_time Master_Slave_time_diff # # Check error handling # diff --git a/mysql-test/suite/perfschema/r/relaylog.result b/mysql-test/suite/perfschema/r/relaylog.result index 7cc87530770..dc611b59b27 100644 --- a/mysql-test/suite/perfschema/r/relaylog.result +++ b/mysql-test/suite/perfschema/r/relaylog.result @@ -20,7 +20,7 @@ if (count_write > 0,"MANY", "NONE") as COUNT_WRITE, if (sum_number_of_bytes_read > 0, "MANY", "NONE") as SUM_NUMBER_OF_BYTES_READ, if (sum_number_of_bytes_write > 0, "MANY", "NONE") as SUM_NUMBER_OF_BYTES_WRITE from performance_schema.file_summary_by_instance -where file_name like "%master-%" order by file_name; +where file_name like "%master-%" and file_name not like "%sql/share%" order by file_name; FILE_NAME EVENT_NAME COUNT_READ COUNT_WRITE SUM_NUMBER_OF_BYTES_READ SUM_NUMBER_OF_BYTES_WRITE master-bin.000001 wait/io/file/sql/binlog MANY MANY MANY MANY master-bin.000001.idx wait/io/file/sql/gtid_index NONE MANY NONE MANY @@ -96,7 +96,7 @@ wait/synch/mutex/sql/MYSQL_RELAY_LOG::LOCK_index 0 0 0 0 0 connection slave; "============ Performance schema on slave ============" select * from performance_schema.file_summary_by_instance -where file_name like "%master-%" order by file_name; +where file_name like "%master-%" and file_name not like "%sql/share%" order by file_name; FILE_NAME EVENT_NAME OBJECT_INSTANCE_BEGIN COUNT_STAR SUM_TIMER_WAIT MIN_TIMER_WAIT AVG_TIMER_WAIT MAX_TIMER_WAIT COUNT_READ SUM_TIMER_READ MIN_TIMER_READ AVG_TIMER_READ MAX_TIMER_READ SUM_NUMBER_OF_BYTES_READ COUNT_WRITE SUM_TIMER_WRITE MIN_TIMER_WRITE AVG_TIMER_WRITE MAX_TIMER_WRITE SUM_NUMBER_OF_BYTES_WRITE COUNT_MISC SUM_TIMER_MISC MIN_TIMER_MISC AVG_TIMER_MISC MAX_TIMER_MISC select substring(file_name, locate("slave-", file_name)) as FILE_NAME, diff --git a/mysql-test/suite/perfschema/t/relaylog.test b/mysql-test/suite/perfschema/t/relaylog.test index 12fc96a8b27..16f72efae98 100644 --- a/mysql-test/suite/perfschema/t/relaylog.test +++ b/mysql-test/suite/perfschema/t/relaylog.test @@ -43,7 +43,7 @@ select if (sum_number_of_bytes_read > 0, "MANY", "NONE") as SUM_NUMBER_OF_BYTES_READ, if (sum_number_of_bytes_write > 0, "MANY", "NONE") as SUM_NUMBER_OF_BYTES_WRITE from performance_schema.file_summary_by_instance - where file_name like "%master-%" order by file_name; + where file_name like "%master-%" and file_name not like "%sql/share%" order by file_name; select * from performance_schema.file_summary_by_instance where file_name like "%slave-%" order by file_name; @@ -96,7 +96,7 @@ sync_slave_with_master; -- echo "============ Performance schema on slave ============" select * from performance_schema.file_summary_by_instance - where file_name like "%master-%" order by file_name; + where file_name like "%master-%" and file_name not like "%sql/share%" order by file_name; select substring(file_name, locate("slave-", file_name)) as FILE_NAME, diff --git a/mysql-test/suite/rpl/r/master_last_event_time_row.result b/mysql-test/suite/rpl/r/master_last_event_time_row.result new file mode 100644 index 00000000000..2dd332007e8 --- /dev/null +++ b/mysql-test/suite/rpl/r/master_last_event_time_row.result @@ -0,0 +1,203 @@ +include/rpl_init.inc [topology=1->2->3] +# +# Initialize test data +# Ensure that all slaves has master_last_event_time == NULL +# +connection server_1; +SET STATEMENT sql_log_bin=0 FOR create table t1 (a int) engine=innodb; +include/save_master_pos.inc +connection server_2; +SET STATEMENT sql_log_bin=0 FOR create table t1 (a int) engine=innodb; +connection server_3; +SET STATEMENT sql_log_bin=0 FOR create table t1 (a int) engine=innodb; +# +# Test *_last_event_time is updated at proper place. +# Master_last_event_time should be updated by the IO thread when reading +# in a new transaction from the primary. +# Slave_last_event_time should be updated by the SQL thread +# 1) immediately upon seeing the first transaction if the replica is +# starting fresh, or +# 2) after committing a transaction. +# +connection server_2; +start slave io_thread; +include/sync_io_with_master.inc +master_time: (should be empty) +slave_time: (should be empty) +# Sleep 2s to create a time gap between the header events (i.e. +# Format_description and Gtid_list) and the transaction event to allow +# proving that header events should not update +# (Master|Slave)_last_event_time +connect server_1_1,127.0.0.1,root,,test,$SERVER_MYPORT_1,; +set @@timestamp= TIMESTAMP; +insert into t1 values (0); +include/save_master_pos.inc +connection server_2; +include/sync_io_with_master.inc +# For the first event, at execution start, Slave_last_event_time should +# be updated to be 1 second prior to the time that the first transaction +# was binlogged on the primary. This is to represent that the slave is +# otherwise up-to-date. Note the table is locked to prevent the +# transaction from committing (and thereby progressing +# Slave_last_event_time to represent commit-time). +connect server_2_2,127.0.0.1,root,,test,$SERVER_MYPORT_2,; +lock tables t1 write; +connection server_2; +start slave sql_thread; +# Waiting for replica to start executing the transaction (yet get stuck on the table lock) +connection server_2_2; +unlock tables; +connection server_2; +include/wait_for_slave_param.inc [Relay_Master_Log_File] +include/wait_for_slave_param.inc [Exec_Master_Log_Pos] +# +# Test that changes are properly applied by server_2 and server_3 +# +connection server_1; +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +# Show that the server_2 received the insert from master +select * from t1; +a +0 +master <> NULL; Should be 1 +1 +master_time == slave_time ; Should be 1 +1 +connection server_3; +include/start_slave.inc +include/sync_with_master_gtid.inc +# Show that the server_3 received the insert from master +select * from t1; +a +0 +master <> NULL; Should be 1 +1 +master_time == slave_time ; Should be 1 +1 +include/stop_slave.inc +connection server_1; +# +# Test simple insert +# +connection server_1; +insert into t1 values (1+sleep(3)); +# +# Test that changes are properly applied by server_2 and server_3 +# +connection server_1; +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +# Show that the server_2 received the insert from master +select * from t1; +a +0 +1 +master <> NULL; Should be 1 +1 +master_time == slave_time ; Should be 1 +1 +connection server_3; +include/start_slave.inc +include/sync_with_master_gtid.inc +# Show that the server_3 received the insert from master +select * from t1; +a +0 +1 +master <> NULL; Should be 1 +1 +master_time == slave_time ; Should be 1 +1 +include/stop_slave.inc +connection server_1; +# +# Test insert with forced time +# +SET TIMESTAMP=unix_timestamp("2000-01-01"); +insert into t1 values (2+sleep(3)); +SET TIMESTAMP=DEFAULT; +# +# Test that changes are properly applied by server_2 and server_3 +# +connection server_1; +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +# Show that the server_2 received the insert from master +select * from t1; +a +0 +1 +2 +master <> NULL; Should be 1 +1 +master_time == slave_time ; Should be 1 +1 +connection server_3; +include/start_slave.inc +include/sync_with_master_gtid.inc +# Show that the server_3 received the insert from master +select * from t1; +a +0 +1 +2 +master <> NULL; Should be 1 +1 +master_time == slave_time ; Should be 1 +1 +include/stop_slave.inc +connection server_1; +# +# Test multi-transaction +# +begin; +insert into t1 values (3+sleep(3)); +insert into t1 values (4+sleep(3)); +commit; +# +# Test that changes are properly applied by server_2 and server_3 +# +connection server_1; +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +# Show that the server_2 received the insert from master +select * from t1; +a +0 +1 +2 +3 +4 +master <> NULL; Should be 1 +1 +master_time == slave_time ; Should be 1 +1 +connection server_3; +include/start_slave.inc +include/sync_with_master_gtid.inc +# Show that the server_3 received the insert from master +select * from t1; +a +0 +1 +2 +3 +4 +master <> NULL; Should be 1 +1 +master_time == slave_time ; Should be 1 +1 +include/stop_slave.inc +connection server_1; +# cleanup +connection server_3; +include/start_slave.inc +connection server_1; +drop table t1; +include/rpl_end.inc +# End of master_last_event_time_row diff --git a/mysql-test/suite/rpl/r/master_last_event_time_stmt.result b/mysql-test/suite/rpl/r/master_last_event_time_stmt.result new file mode 100644 index 00000000000..4b1e41a7704 --- /dev/null +++ b/mysql-test/suite/rpl/r/master_last_event_time_stmt.result @@ -0,0 +1,71 @@ +include/rpl_init.inc [topology=1->2->3] +connection server_3; +include/stop_slave.inc +connection server_1; +call mtr.add_suppression("Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT"); +alter table mysql.gtid_slave_pos engine=innodb; +create table t1 (a int) engine=aria; +create table t2 (a int) engine=innodb; +include/save_master_gtid.inc +# +# Ensure that the slave doesn't overwrite exec_time when binlogging +# +connection server_2; +include/sync_with_master_gtid.inc +include/stop_slave.inc +change master to master_delay=SLAVE_DELAY; +include/start_slave.inc +connection server_1; +# Sleep 2 to ensure DDL and DML have different binlog timestamps +connection server_2; +flush logs; +connection server_1; +set @@timestamp= TIMESTAMP; +insert into t1 values (sleep(2)); +include/save_master_gtid.inc +# Waiting for slave to delay and commit transaction.. +connection server_2; +include/sync_with_master_gtid.inc +# MYSQL_BINLOG slave_local_binlog > slave_outfile +include/assert_grep.inc [Ensure slave doesn't overwrite exec_time in the binlog event] +# +# Test that changes are properly applied by server_2 and server_3 +# +connection server_1; +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +# Show that the server_2 received the insert from master +select * from t1; +a +0 +master <> NULL; Should be 1 +1 +master_time == slave_time ; Should be 1 +1 +connection server_3; +include/start_slave.inc +include/sync_with_master_gtid.inc +# Show that the server_3 received the insert from master +select * from t1; +a +0 +master <> NULL; Should be 1 +1 +master_time == slave_time ; Should be 1 +1 +include/stop_slave.inc +connection server_1; +connection server_2; +include/stop_slave.inc +change master to master_delay=0; +include/start_slave.inc +connection server_3; +include/start_slave.inc +# +# Cleanup +connection server_1; +drop table t1; +drop table t2; +include/rpl_end.inc +# End of master_last_event_time_stmt diff --git a/mysql-test/suite/rpl/r/rpl_parallel_sbm.result b/mysql-test/suite/rpl/r/rpl_parallel_sbm.result index 75012c93f3b..c41de974588 100644 --- a/mysql-test/suite/rpl/r/rpl_parallel_sbm.result +++ b/mysql-test/suite/rpl/r/rpl_parallel_sbm.result @@ -23,6 +23,11 @@ connection slave; # Waiting for transaction to arrive on slave and begin SQL Delay.. # Validating SBM is updated on event arrival.. # ..done +# MDEV-33856: New definition for Seconds_Behind_Master +# Validating Master_last_event_time is updated on event arrival.. +# ..done +# Validating Slave_last_event_time is still from the last transaction.. +# ..done # MDEV-32265. At time of STOP SLAVE, if the SQL Thread is currently # delaying a transaction; then when the reciprocal START SLAVE occurs, # if the event is still to be delayed, SBM should resume accordingly @@ -41,6 +46,9 @@ connection slave; connection server_2; UNLOCK TABLES; include/sync_with_master_gtid.inc +# MDEV-33856: New definition for Seconds_Behind_Master +# Ensuring Slave_last_event_time is now up-to-date once event is executed +# ..done # # Pt 2) If the worker threads have not entered an idle state, ensure # following events do not update SBM diff --git a/mysql-test/suite/rpl/t/master_last_event_time.inc b/mysql-test/suite/rpl/t/master_last_event_time.inc new file mode 100644 index 00000000000..0790d557a7b --- /dev/null +++ b/mysql-test/suite/rpl/t/master_last_event_time.inc @@ -0,0 +1,58 @@ +--echo # +--echo # Test that changes are properly applied by server_2 and server_3 +--echo # + +--connection server_1 +--source include/save_master_gtid.inc + +--connection server_2 +--source include/sync_with_master_gtid.inc + +--echo # Show that the server_2 received the insert from master +select * from t1; + +--let $master_time= query_get_value(SHOW ALL SLAVES STATUS, Master_last_event_time, 1) +--let $slave_time= query_get_value(SHOW ALL SLAVES STATUS, Slave_last_event_time, 1) +--disable_query_log +--eval select "$master_time" <> "NULL" as "master <> NULL; Should be 1" + +if (`SELECT NOT "$master_time" <> "NULL"`) +{ + --echo MASTER: $master_time +} + +--eval select "$master_time" = "$slave_time" as "master_time == slave_time ; Should be 1" +if (`SELECT NOT "$master_time" = "$slave_time"`) +{ + --echo MASTER: $master_time SLAVE: $slave_time +} +--enable_query_log + +--connection server_3 +--sleep 1 +--source include/start_slave.inc +--source include/sync_with_master_gtid.inc + +--echo # Show that the server_3 received the insert from master +select * from t1; + +--let $master_time= query_get_value(SHOW ALL SLAVES STATUS, Master_last_event_time, 1) +--let $slave_time= query_get_value(SHOW ALL SLAVES STATUS, Slave_last_event_time, 1) +--disable_query_log + +--eval select "$master_time" <> "NULL" as "master <> NULL; Should be 1" +if (`SELECT NOT "$master_time" <> "NULL"`) +{ + --echo MASTER: $master_time +} + +--eval select "$master_time" = "$slave_time" as "master_time == slave_time ; Should be 1" +if (`SELECT NOT "$master_time" = "$slave_time"`) +{ + --echo MASTER: $master_time SLAVE: $slave_time +} +--enable_query_log + +# Reset things for next test +--source include/stop_slave.inc +--connection server_1 diff --git a/mysql-test/suite/rpl/t/master_last_event_time_row.cnf b/mysql-test/suite/rpl/t/master_last_event_time_row.cnf new file mode 100644 index 00000000000..65a4396edf3 --- /dev/null +++ b/mysql-test/suite/rpl/t/master_last_event_time_row.cnf @@ -0,0 +1,16 @@ +!include suite/rpl/my.cnf + +[mysqld.1] + +[mysqld.2] +log-slave-updates +binlog-checksum=CRC32 + +[mysqld.3] +log-slave-updates +binlog-checksum=CRC32 + +[ENV] +SERVER_MYPORT_3= @mysqld.3.port +SERVER_MYSOCK_3= @mysqld.3.socket + diff --git a/mysql-test/suite/rpl/t/master_last_event_time_row.test b/mysql-test/suite/rpl/t/master_last_event_time_row.test new file mode 100644 index 00000000000..92abe00d407 --- /dev/null +++ b/mysql-test/suite/rpl/t/master_last_event_time_row.test @@ -0,0 +1,168 @@ +# +# Row specific tests for master_last_event_time +# +--source include/have_binlog_format_row.inc +--source include/have_innodb.inc +--let $rpl_skip_start_slave=1 +--let $rpl_topology=1->2->3 +--source include/rpl_init.inc + +--echo # +--echo # Initialize test data +--echo # Ensure that all slaves has master_last_event_time == NULL +--echo # + +--connection server_1 +SET STATEMENT sql_log_bin=0 FOR create table t1 (a int) engine=innodb; +--source include/save_master_pos.inc +--connection server_2 +SET STATEMENT sql_log_bin=0 FOR create table t1 (a int) engine=innodb; +--connection server_3 +SET STATEMENT sql_log_bin=0 FOR create table t1 (a int) engine=innodb; + +--echo # +--echo # Test *_last_event_time is updated at proper place. +--echo # Master_last_event_time should be updated by the IO thread when reading +--echo # in a new transaction from the primary. +--echo # Slave_last_event_time should be updated by the SQL thread +--echo # 1) immediately upon seeing the first transaction if the replica is +--echo # starting fresh, or +--echo # 2) after committing a transaction. +--echo # + +--connection server_2 +start slave io_thread; +--source include/sync_io_with_master.inc + +# Ensure Master_last_event_time and Slave_last_event_time are not yet set +--let $master_time= `select Master_last_event_time from information_schema.slave_status` +--echo master_time: $master_time (should be empty) +--let $slave_time=`select Slave_last_event_time from information_schema.slave_status` +--echo slave_time: $slave_time (should be empty) + +--echo # Sleep 2s to create a time gap between the header events (i.e. +--echo # Format_description and Gtid_list) and the transaction event to allow +--echo # proving that header events should not update +--echo # (Master|Slave)_last_event_time +--sleep 2 + +--connect (server_1_1,127.0.0.1,root,,test,$SERVER_MYPORT_1,) +--let $t1_time_begin= `select truncate(@@timestamp,0)` +--replace_result $t1_time_begin TIMESTAMP +--eval set @@timestamp= $t1_time_begin +insert into t1 values (0); + +--source include/save_master_pos.inc + +--connection server_2 +--source include/sync_io_with_master.inc + +--let $mle_time= `select Master_last_event_time from information_schema.slave_status` +--let $mle_time_unix= `select truncate(unix_timestamp("$mle_time"),0)` +if (`SELECT ($mle_time_unix < $t1_time_begin)`) +{ + --echo # Expected timestamp (master binlog time): $t1_time_begin + --echo # Reported Master_last_event_time: $mle_time_unix ($mle_time) + --die Master_last_event_time did not correspond to time that the transaction was binlogged on primary +} + +--let $slave_time= `select Slave_last_event_time from information_schema.slave_status` +if (`select strcmp("$slave_time", "") != 0`) +{ + --echo # Slave_last_event_time: $slave_time + --die SQL thread was never started, Slave_last_event_time should be NULL +} + +# Check that we also get the values from show all slaves status +--let $time_diff= query_get_value(SHOW ALL SLAVES STATUS, Master_Slave_time_diff, 1) +if (`select strcmp("$time_diff", "NULL") != 0`) +{ + --echo # Master_Slave_time_diff: $time_diff + --die SQL thread was never started, Master_Slave_time_diff should be NULL +} + +--echo # For the first event, at execution start, Slave_last_event_time should +--echo # be updated to be 1 second prior to the time that the first transaction +--echo # was binlogged on the primary. This is to represent that the slave is +--echo # otherwise up-to-date. Note the table is locked to prevent the +--echo # transaction from committing (and thereby progressing +--echo # Slave_last_event_time to represent commit-time). + +--connect (server_2_2,127.0.0.1,root,,test,$SERVER_MYPORT_2,) +lock tables t1 write; + +--connection server_2 +start slave sql_thread; + +--echo # Waiting for replica to start executing the transaction (yet get stuck on the table lock) +--let $wait_condition= SELECT count(*) FROM information_schema.processlist WHERE state LIKE 'Waiting for table metadata lock'; +--source include/wait_condition.inc + +--let $slave_time= query_get_value(SHOW ALL SLAVES STATUS, Slave_last_event_time, 1) +--let $slave_time_unix= `select truncate(unix_timestamp("$slave_time"),0)` +--let $expected_slave_time= `select ($t1_time_begin - 1)` +if ($slave_time_unix != $expected_slave_time) +{ + --echo # Master_last_event_time: $mle_time_unix ($mle_time) + --echo # Slave_last_event_time: $slave_time_unix ($slave_time) + --echo # Expected value: $expected_slave_time + --die SQL thread has not yet committed its first transaction, Slave_last_event_time should be 1s before that transaction +} + +# Master_Slave_time_diff isn't guaranteed to be 1 second, despite the +# hard-coded logic to subtract 1s from the first non-group event that comes in. +# This is because the Gtid and Xid events can be logged with different +# timestamps, and Slave_last_event_time is updated using the Gtid log event, +# and Master_last_event_time is updated using the Xid log event. So to ensure +# that Master_Slave_time_diff is updated correctly for the first transaction, +# it must also take into account the difference in timestamps of these events +--let $xid_gtid_time_diff=`SELECT $mle_time_unix - $t1_time_begin` +--let $time_diff= query_get_value(SHOW ALL SLAVES STATUS, Master_Slave_time_diff, 1) +if (`SELECT $time_diff != 1 + $xid_gtid_time_diff`) +{ + --echo # Master_Slave_time_diff: $time_diff + --echo # Xid Gtid Timestamp Difference: $xid_gtid_time_diff + --die SQL thread has not yet committed its first transaction, Master_Slave_time_diff should be updated to look up-to-date prior to this trx +} + +--connection server_2_2 +unlock tables; + +--connection server_2 +--source include/sync_with_master.inc +--source master_last_event_time.inc + +--echo # +--echo # Test simple insert +--echo # + +--connection server_1 +insert into t1 values (1+sleep(3)); +--source master_last_event_time.inc + +--echo # +--echo # Test insert with forced time +--echo # + +SET TIMESTAMP=unix_timestamp("2000-01-01"); +insert into t1 values (2+sleep(3)); +SET TIMESTAMP=DEFAULT; +--source master_last_event_time.inc + +--echo # +--echo # Test multi-transaction +--echo # + +begin; +insert into t1 values (3+sleep(3)); +insert into t1 values (4+sleep(3)); +commit; +--source master_last_event_time.inc + +--echo # cleanup +--connection server_3 +--source include/start_slave.inc +--connection server_1 +drop table t1; +--source include/rpl_end.inc +--echo # End of master_last_event_time_row diff --git a/mysql-test/suite/rpl/t/master_last_event_time_stmt.cnf b/mysql-test/suite/rpl/t/master_last_event_time_stmt.cnf new file mode 100644 index 00000000000..35b68f7906f --- /dev/null +++ b/mysql-test/suite/rpl/t/master_last_event_time_stmt.cnf @@ -0,0 +1,15 @@ +!include suite/rpl/my.cnf + +[mysqld.1] + +[mysqld.2] +log-slave-updates +binlog-checksum=CRC32 + +[mysqld.3] +log-slave-updates +binlog-checksum=CRC32 + +[ENV] +SERVER_MYPORT_3= @mysqld.3.port +SERVER_MYSOCK_3= @mysqld.3.socket diff --git a/mysql-test/suite/rpl/t/master_last_event_time_stmt.test b/mysql-test/suite/rpl/t/master_last_event_time_stmt.test new file mode 100644 index 00000000000..fe0f16a3db3 --- /dev/null +++ b/mysql-test/suite/rpl/t/master_last_event_time_stmt.test @@ -0,0 +1,84 @@ +# +# Statement specific tests for master_last_event_time +# +--source include/have_binlog_format_statement.inc +--source include/have_innodb.inc +--let $rpl_topology=1->2->3 +--source include/rpl_init.inc + +# Server_3 state is maintained by master_last_event_time.inc +--connection server_3 +--source include/stop_slave.inc + +--connection server_1 +call mtr.add_suppression("Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT"); +alter table mysql.gtid_slave_pos engine=innodb; +create table t1 (a int) engine=aria; +create table t2 (a int) engine=innodb; +--source include/save_master_gtid.inc + +--echo # +--echo # Ensure that the slave doesn't overwrite exec_time when binlogging +--echo # + +--let $slave_delay= 3 + +--connection server_2 +--source include/sync_with_master_gtid.inc +--source include/stop_slave.inc +--replace_result $slave_delay SLAVE_DELAY +--eval change master to master_delay=$slave_delay +--source include/start_slave.inc + +--connection server_1 +--echo # Sleep 2 to ensure DDL and DML have different binlog timestamps +--sleep 2 +--let $t1_time_begin= `select truncate(@@timestamp,0)` +--connection server_2 +flush logs; +--connection server_1 +--replace_result $t1_time_begin TIMESTAMP +--eval set @@timestamp= $t1_time_begin +--disable_warnings +insert into t1 values (sleep(2)); +--source include/save_master_gtid.inc +--enable_warnings + +--echo # Waiting for slave to delay and commit transaction.. +--connection server_2 +--source include/sync_with_master_gtid.inc + +# Slave will replicate the above with a row event which will be very fast +# compared to the master event. +# Check the exec time is not 0 (which is typical for very fast row events) + +--let $datadir= `select @@datadir` +--let $filename= query_get_value(SHOW MASTER STATUS, File, 1) +--let $slave_local_binlog=$datadir/$filename +--let $slave_outfile=$MYSQLTEST_VARDIR/tmp/slave_binlog.sql +--echo # MYSQL_BINLOG slave_local_binlog > slave_outfile +--exec $MYSQL_BINLOG $slave_local_binlog > $slave_outfile +--let $assert_count=0 +--let $assert_text= Ensure slave doesn't overwrite exec_time in the binlog event +--let $assert_select=exec_time=0 +--let $assert_file= $slave_outfile +--source include/assert_grep.inc + +--source master_last_event_time.inc + +--connection server_2 +--source include/stop_slave.inc +change master to master_delay=0; +--source include/start_slave.inc +--connection server_3 +--source include/start_slave.inc + +--echo # +--echo # Cleanup +--connection server_1 +drop table t1; +drop table t2; + +--source include/rpl_end.inc +--remove_file $slave_outfile +--echo # End of master_last_event_time_stmt diff --git a/mysql-test/suite/rpl/t/rpl_parallel_sbm.test b/mysql-test/suite/rpl/t/rpl_parallel_sbm.test index 90753caf143..2c9b4882c16 100644 --- a/mysql-test/suite/rpl/t/rpl_parallel_sbm.test +++ b/mysql-test/suite/rpl/t/rpl_parallel_sbm.test @@ -61,6 +61,30 @@ if (`SELECT $sbm_trx1_arrive > ($seconds_since_idling + 1)`) } --echo # ..done +--echo # MDEV-33856: New definition for Seconds_Behind_Master +--echo # Validating Master_last_event_time is updated on event arrival.. +--let $mle_time_trx1_arrive= query_get_value(SHOW ALL SLAVES STATUS, Master_last_event_time, 1) +--let $mle_time_trx1_arrive_unix= `SELECT truncate(unix_timestamp("$mle_time_trx1_arrive"), 0)` +if (`SELECT $mle_time_trx1_arrive_unix < ($ts_trx_before_ins - 1)`) +{ + --echo # Master_last_event_time: $mle_time_trx1_arrive_unix ($mle_time_trx1_arrive) + --die Master_last_event_time was not updated for delayed replica at event arrival time +} +--echo # ..done + +--echo # Validating Slave_last_event_time is still from the last transaction.. +# Note we infer Slave_last_event_time via Master_Slave_time_diff +--let $time_diff_trx1_arrive= query_get_value(SHOW ALL SLAVES STATUS, Master_Slave_time_diff, 1) +if ($time_diff_trx1_arrive < 5) +{ + --let $slave_time_trx1_arrive= query_get_value(SHOW ALL SLAVES STATUS, Slave_last_event_time, 1) + --let $slave_time_trx1_arrive_unix= `SELECT truncate(unix_timestamp("$slave_time_trx1_arrive"), 0)` + --echo # Slave_last_event_time: $slave_time_trx1_arrive_unix ($slave_time_trx1_arrive) + --echo # Master_Slave_time_diff: $time_diff_trx1_arrive + --die Slave_last_event_time is too recent, should not be less than 5 seconds, ie. 3 from delay + 2 from sleep +} +--echo # ..done + --echo # MDEV-32265. At time of STOP SLAVE, if the SQL Thread is currently --echo # delaying a transaction; then when the reciprocal START SLAVE occurs, @@ -98,6 +122,24 @@ UNLOCK TABLES; --source include/sync_with_master_gtid.inc +--echo # MDEV-33856: New definition for Seconds_Behind_Master +--echo # Ensuring Slave_last_event_time is now up-to-date once event is executed +--let $slave_time_trx1_commit= query_get_value(SHOW ALL SLAVES STATUS, Slave_last_event_time, 1) +--let $slave_time_trx1_commit_unix= `SELECT truncate(unix_timestamp("$slave_time_trx1_commit"),0)` +if ($slave_time_trx1_commit_unix != $mle_time_trx1_arrive_unix) +{ + --echo # Slave_last_event_time: $slave_time_trx1_commit_unix ($slave_time_trx1_commit) + --echo # Master_last_event_time: $mle_time_trx1_arrive_unix ($mle_time_trx1_arrive) + --die Slave_last_event_time is not equal to Master_last_event_time despite being up-to-date +} +--let $time_diff_trx1_commit= query_get_value(SHOW ALL SLAVES STATUS, Master_Slave_time_diff, 1) +if ($time_diff_trx1_commit != 0) +{ + --echo # Master_Slave_time_diff: $time_diff_trx1_commit + --die Master_Slave_time_diff should be 0, as slave is up-to-date +} +--echo # ..done + --echo # --echo # Pt 2) If the worker threads have not entered an idle state, ensure --echo # following events do not update SBM diff --git a/sql/handler.h b/sql/handler.h index 37b403c942c..86862210fab 100644 --- a/sql/handler.h +++ b/sql/handler.h @@ -1113,6 +1113,9 @@ enum enum_schema_tables SCH_USERS, SCH_USER_PRIVILEGES, SCH_VIEWS, +#ifdef HAVE_REPLICATION + SCH_SLAVE_STATUS, +#endif SCH_ENUM_SIZE }; diff --git a/sql/item.h b/sql/item.h index 6c69f33aecb..28014a1e135 100644 --- a/sql/item.h +++ b/sql/item.h @@ -4902,8 +4902,8 @@ class Item_partition_func_safe_string: public Item_string /** Item_empty_string -- is a utility class to put an item into List<Item> - which is then used in protocol.send_result_set_metadata() when sending SHOW output to - the client. + which is then used in protocol.send_result_set_metadata() when sending SHOW + output to the client. */ class Item_empty_string :public Item_partition_func_safe_string @@ -5284,6 +5284,17 @@ class Item_datetime_literal: public Item_temporal_literal set_maybe_null(cached_time.check_date(TIME_NO_ZERO_DATE | TIME_NO_ZERO_IN_DATE)); } + Item_datetime_literal(THD *thd, const char *name_arg, + decimal_digits_t dec_arg): + Item_temporal_literal(thd, dec_arg), + cached_time(Datetime::zero()) + { + max_length= MAX_DATETIME_WIDTH + (decimals ? decimals + 1 : 0); + set_maybe_null(true); + // Set the name (see also a similar code in Item_int): + name.str= name_arg; + name.length= strlen(name.str); + } const Type_handler *type_handler() const override { return &type_handler_datetime2; } void print(String *str, enum_query_type query_type) override; diff --git a/sql/log.cc b/sql/log.cc index 1cc0fa77f08..63d9c756911 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -1898,10 +1898,20 @@ static inline int binlog_commit_flush_xid_caches(THD *thd, binlog_cache_mngr *cache_mngr, bool all, my_xid xid) { + DBUG_ENTER("binlog_commit_flush_xid_caches"); DBUG_ASSERT(xid); // replaced former treatment of ONE-PHASE XA Xid_log_event end_evt(thd, xid, TRUE); - return (binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); + if (!thd->slave_thread && ! thd->user_time.val) + { + /* + Ensure that on the master the event time is the time of commit, + not the start of statement time. + */ + my_hrtime_t hrtime= my_hrtime(); + end_evt.when= hrtime_to_my_time(hrtime); + } + DBUG_RETURN(binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); } /** @@ -8224,6 +8234,7 @@ MYSQL_BIN_LOG::write_transaction_to_binlog(THD *thd, { group_commit_entry entry; Ha_trx_info *ha_info; + bool has_xid; DBUG_ENTER("MYSQL_BIN_LOG::write_transaction_to_binlog"); /* @@ -8251,13 +8262,16 @@ MYSQL_BIN_LOG::write_transaction_to_binlog(THD *thd, ha_info= all ? thd->transaction->all.ha_list : thd->transaction->stmt.ha_list; entry.ro_1pc= is_ro_1pc; entry.end_event= end_ev; - auto has_xid= entry.end_event->get_type_code() == XID_EVENT; + has_xid= entry.end_event->get_type_code() == XID_EVENT; for (; has_xid && !entry.need_unlog && ha_info; ha_info= ha_info->next()) { if (ha_info->is_started() && ha_info->ht() != binlog_hton && !ha_info->ht()->commit_checkpoint_request) + { entry.need_unlog= true; + break; + } } if (cache_mngr->stmt_cache.has_incident() || diff --git a/sql/log_event.cc b/sql/log_event.cc index 3802f76efe0..706a4903eb0 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -964,6 +964,18 @@ Log_event* Log_event::read_log_event(IO_CACHE* file, DBUG_RETURN(res); } +/* + Update thd->orig_exec_time +*/ + +inline void set_orig_exec_time_in_thd(my_time_t exec_time) +{ +#if !defined(MYSQL_CLIENT) && defined(HAVE_REPLICATION) + THD *thd= current_thd; + if (likely(thd)) + thd->orig_exec_time= exec_time; +#endif +} /** Binlog format tolerance is in (buf, event_len, fdle) @@ -1104,10 +1116,12 @@ Log_event* Log_event::read_log_event(const uchar *buf, uint event_len, switch(event_type) { case QUERY_EVENT: ev= new Query_log_event(buf, event_len, fdle, QUERY_EVENT); + set_orig_exec_time_in_thd(((Query_log_event*) ev)->exec_time); break; case QUERY_COMPRESSED_EVENT: ev= new Query_compressed_log_event(buf, event_len, fdle, - QUERY_COMPRESSED_EVENT); + QUERY_COMPRESSED_EVENT); + set_orig_exec_time_in_thd(((Query_compressed_log_event*) ev)->exec_time); break; case ROTATE_EVENT: ev= new Rotate_log_event(buf, event_len, fdle); @@ -1194,6 +1208,7 @@ Log_event* Log_event::read_log_event(const uchar *buf, uint event_len, break; case EXECUTE_LOAD_QUERY_EVENT: ev= new Execute_load_query_log_event(buf, event_len, fdle); + set_orig_exec_time_in_thd(((Query_log_event*) ev)->exec_time); break; case INCIDENT_EVENT: ev= new Incident_log_event(buf, event_len, fdle); @@ -1747,6 +1762,26 @@ Query_log_event::Query_log_event(const uchar *buf, uint event_len, DBUG_VOID_RETURN; } + +/* + Get the time when the event had been executed on the master. + This works for both query events and load data events. +*/ + +#if Q_EXEC_TIME_OFFSET != L_EXEC_TIME_OFFSET +#error "Q_EXEC_TIME_OFFSET is not same as L_EXEC_TIME_OFFSET" +#endif + +time_t query_event_get_time(const uchar *buf, + const Format_description_log_event + *description_event) +{ + time_t when= uint4korr(buf); + buf+= description_event->common_header_len; + return when + uint4korr(buf + Q_EXEC_TIME_OFFSET); +} + + Query_compressed_log_event::Query_compressed_log_event(const uchar *buf, uint event_len, const Format_description_log_event diff --git a/sql/log_event.h b/sql/log_event.h index 3f707ee25c0..69ba7cde440 100644 --- a/sql/log_event.h +++ b/sql/log_event.h @@ -809,6 +809,10 @@ static inline bool LOG_EVENT_IS_ROW_V2(enum Log_event_type type) (type >= WRITE_ROWS_COMPRESSED_EVENT && type <= DELETE_ROWS_COMPRESSED_EVENT); } +static inline bool LOG_EVENT_IS_LOAD_DATA(enum Log_event_type type) +{ + return type == LOAD_EVENT || type == NEW_LOAD_EVENT; +} /* The number of types we handle in Format_description_log_event (UNKNOWN_EVENT @@ -1293,7 +1297,7 @@ class Log_event my_time_t when; ulong when_sec_part; /* The number of seconds the query took to run on the master. */ - ulong exec_time; + my_time_t exec_time; /* Number of bytes written by write() function */ size_t data_written; @@ -1414,8 +1418,8 @@ class Log_event log; used by SHOW BINLOG EVENTS, the binlog_dump thread on the master (reads master's binlog), the slave IO thread (reads the event sent by binlog_dump), the slave SQL thread (reads the event - from the relay log). If mutex is 0, the read will proceed without - mutex. We need the description_event to be able to parse the + from the relay log). + We need the description_event to be able to parse the event (to know the post-header's size); in fact in read_log_event we detect the event's type, then call the specific event's constructor and pass description_event as an argument. @@ -1438,8 +1442,6 @@ class Log_event Reads an event from a binlog or relay log. Used by the dump thread this method reads the event into a raw buffer without parsing it. - @Note If mutex is 0, the read will proceed without mutex. - @Note If a log name is given than the method will check if the given binlog is still active. @@ -5504,6 +5506,9 @@ int row_log_event_uncompress(const Format_description_log_event const uchar *src, ulong src_len, uchar* buf, ulong buf_size, bool *is_malloc, uchar **dst, ulong *newlen); +time_t query_event_get_time(const uchar *buf, + const Format_description_log_event + *description_event); bool is_parallel_retry_error(rpl_group_info *rgi, int err); diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc index e2ab6a659cd..2699ec773f8 100644 --- a/sql/log_event_server.cc +++ b/sql/log_event_server.cc @@ -1328,8 +1328,6 @@ Query_log_event::Query_log_event(THD* thd_arg, const char* query_arg, { /* status_vars_len is set just before writing the event */ - time_t end_time; - #ifdef WITH_WSREP /* If Query_log_event will contain non trans keyword (not BEGIN, COMMIT, @@ -1348,8 +1346,14 @@ Query_log_event::Query_log_event(THD* thd_arg, const char* query_arg, memset(&host, 0, sizeof(host)); error_code= errcode; - end_time= my_time(0); - exec_time = (ulong) (end_time - thd_arg->start_time); + /* + For slave threads, remember the original master exec time. + This is needed to be able to calculate the master commit time. + */ + exec_time= ((thd->slave_thread) ? + thd->orig_exec_time : + (my_time(0) - thd_arg->start_time)); + /** @todo this means that if we have no catalog, then it is replicated as an existing catalog of length zero. is that safe? /sven @@ -3682,6 +3686,15 @@ int Xid_apply_log_event::do_apply_event(rpl_group_info *rgi) general_log_print(thd, COM_QUERY, get_query()); thd->variables.option_bits&= ~OPTION_GTID_BEGIN; + /* + Use the time from the current Xid_log_event for the generated + Xid_log_event in binlog_commit_flush_xid_caches(). + This ensures that the time for Xid_log_events does not change + and allows slaves to give a consistent value for + Slave_last_event_time. + */ + thd->start_time= when; + res= do_commit(); if (!res && rgi->gtid_pending) { diff --git a/sql/protocol.cc b/sql/protocol.cc index 1e27c996687..ca7803248c6 100644 --- a/sql/protocol.cc +++ b/sql/protocol.cc @@ -1417,33 +1417,6 @@ bool Protocol::store(I_List<i_string>* str_list) return store((char*) tmp.ptr(), tmp.length(), tmp.charset()); } - -/** - Send a set of strings as a string of key-value pairs with ',' in between. -*/ - -bool Protocol::store(I_List<i_string_pair>* str_list) -{ - char buf[256]; - const char *delimiter= ","; - String tmp(buf, sizeof(buf), &my_charset_bin); - size_t delim_len= 0; - I_List_iterator<i_string_pair> it(*str_list); - i_string_pair* s; - - tmp.length(0); - while ((s=it++)) - { - tmp.append(delimiter, delim_len); - tmp.append(s->key, strlen(s->key)); - tmp.append(STRING_WITH_LEN("->")); - tmp.append(s->val, strlen(s->val)); - delim_len= 1; - } - return store((char*) tmp.ptr(), tmp.length(), tmp.charset()); -} - - /**************************************************************************** Functions to handle the simple (default) protocol where everything is This protocol is the one that is used by default between the MySQL server diff --git a/sql/protocol.h b/sql/protocol.h index d1096b041af..0a55d1b1331 100644 --- a/sql/protocol.h +++ b/sql/protocol.h @@ -105,7 +105,7 @@ class Protocol bool send_result_set_row(List<Item> *row_items); bool store(I_List<i_string> *str_list); - bool store(I_List<i_string_pair> *str_list); + bool store_string_or_null(const char *from, CHARSET_INFO *cs); bool store_warning(const char *from, size_t length); String *storage_packet() { return packet; } diff --git a/sql/rpl_mi.cc b/sql/rpl_mi.cc index 4f28f1f8edf..224f1c3ea16 100644 --- a/sql/rpl_mi.cc +++ b/sql/rpl_mi.cc @@ -1913,6 +1913,14 @@ void Domain_id_filter::store_ids(THD *thd) } } +void Domain_id_filter::store_ids(Field ***field) +{ + for (int i= DO_DOMAIN_IDS; i <= IGNORE_DOMAIN_IDS; i ++) + { + field_store_ids(*((*field)++), &m_domain_ids[i]); + } +} + /** Initialize the given domain_id list (DYNAMIC_ARRAY) with the space-separated list of numbers from the specified IO_CACHE where @@ -1987,20 +1995,10 @@ void update_change_master_ids(DYNAMIC_ARRAY *new_ids, DYNAMIC_ARRAY *old_ids) return; } -/** - Serialize and store the ids from the given ids DYNAMIC_ARRAY into the thd's - protocol buffer. - - @param thd [IN] thread handler - @param ids [IN] ids list - - @retval void -*/ - -void prot_store_ids(THD *thd, DYNAMIC_ARRAY *ids) +static size_t store_ids(DYNAMIC_ARRAY *ids, char *buff, size_t buff_len) { - char buff[FN_REFLEN]; - uint i, cur_len; + uint i; + size_t cur_len; for (i= 0, buff[0]= 0, cur_len= 0; i < ids->elements; i++) { @@ -2008,7 +2006,7 @@ void prot_store_ids(THD *thd, DYNAMIC_ARRAY *ids) char dbuff[FN_REFLEN]; get_dynamic(ids, (void *) &id, i); len= sprintf(dbuff, (i == 0 ? "%lu" : ", %lu"), id); - if (cur_len + len + 4 > FN_REFLEN) + if (cur_len + len + 4 > buff_len) { /* break the loop whenever remained space could not fit @@ -2019,8 +2017,33 @@ void prot_store_ids(THD *thd, DYNAMIC_ARRAY *ids) } cur_len+= sprintf(buff + cur_len, "%s", dbuff); } + return cur_len; +} + + +/** + Serialize and store the ids from the given ids DYNAMIC_ARRAY into the thd's + protocol buffer. + + @param thd [IN] thread handler + @param ids [IN] ids list + + @retval void +*/ + +void prot_store_ids(THD *thd, DYNAMIC_ARRAY *ids) +{ + char buff[FN_REFLEN]; + size_t cur_len= store_ids(ids, buff, sizeof(buff)); thd->protocol->store(buff, cur_len, &my_charset_bin); - return; +} + + +void field_store_ids(Field *field, DYNAMIC_ARRAY *ids) +{ + char buff[FN_REFLEN]; + size_t cur_len= store_ids(ids, buff, sizeof(buff)); + field->store(buff, cur_len, &my_charset_bin); } diff --git a/sql/rpl_mi.h b/sql/rpl_mi.h index a0f17a7dd01..37a5fc30099 100644 --- a/sql/rpl_mi.h +++ b/sql/rpl_mi.h @@ -105,7 +105,8 @@ class Domain_id_filter @retval void */ void store_ids(THD *thd); - + /* Same as above, but store the id's into a group of fields */ + void store_ids(Field ***field); /* Initialize the given domain id list (DYNAMIC_ARRAY) with the space-separated list of numbers from the specified IO_CACHE where @@ -430,7 +431,7 @@ int flush_master_info(Master_info* mi, void copy_filter_setting(Rpl_filter* dst_filter, Rpl_filter* src_filter); void update_change_master_ids(DYNAMIC_ARRAY *new_ids, DYNAMIC_ARRAY *old_ids); void prot_store_ids(THD *thd, DYNAMIC_ARRAY *ids); - +void field_store_ids(Field *field, DYNAMIC_ARRAY *ids); /* Multi master are handled trough this struct. Changes to this needs to be protected by LOCK_active_mi; diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc index 51ca3b183ec..d279761767a 100644 --- a/sql/rpl_parallel.cc +++ b/sql/rpl_parallel.cc @@ -60,7 +60,10 @@ rpt_handle_event(rpl_parallel_thread::queued_event *qev, rgi->future_event_relay_log_pos= qev->future_event_relay_log_pos; strcpy(rgi->future_event_master_log_name, qev->future_event_master_log_name); if (event_can_update_last_master_timestamp(ev)) + { rgi->last_master_timestamp= ev->when + ev->exec_time; + thd->orig_exec_time= ev->exec_time; + } err= apply_event_and_update_pos_for_parallel(ev, thd, rgi); rli->executed_entries++; diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc index f2247638c43..461540c0d3f 100644 --- a/sql/rpl_rli.cc +++ b/sql/rpl_rli.cc @@ -54,7 +54,9 @@ Relay_log_info::Relay_log_info(bool is_slave_recovery, const char* thread_name) cur_log_old_open_count(0), error_on_rli_init_info(false), group_relay_log_pos(0), event_relay_log_pos(0), group_master_log_pos(0), log_space_total(0), ignore_log_space_limit(0), - last_master_timestamp(0), sql_thread_caught_up(true), slave_skip_counter(0), + sql_thread_caught_up(true), + last_master_timestamp(0), newest_master_timestamp(0), slave_timestamp(0), + slave_skip_counter(0), abort_pos_wait(0), slave_run_id(0), sql_driver_thd(), gtid_skip_flag(GTID_SKIP_NOT), inited(0), abort_slave(0), stop_for_until(0), slave_running(MYSQL_SLAVE_NOT_RUN), until_condition(UNTIL_NONE), @@ -1028,9 +1030,7 @@ void Relay_log_info::inc_group_relay_log_pos(ulonglong log_pos, potentially thousands of events are still queued up for worker threads waiting for execution. */ - if (rgi->last_master_timestamp && - rgi->last_master_timestamp > last_master_timestamp) - last_master_timestamp= rgi->last_master_timestamp; + set_if_bigger(last_master_timestamp, rgi->last_master_timestamp); } else { @@ -1041,6 +1041,7 @@ void Relay_log_info::inc_group_relay_log_pos(ulonglong log_pos, if (log_pos) // not 3.23 binlogs (no log_pos there) and not Stop_log_event group_master_log_pos= log_pos; } + set_if_bigger(slave_timestamp, rgi->last_master_timestamp); /* If the slave does not support transactions and replicates a transaction, diff --git a/sql/rpl_rli.h b/sql/rpl_rli.h index d20b0eb2ab4..9097e032de5 100644 --- a/sql/rpl_rli.h +++ b/sql/rpl_rli.h @@ -251,7 +251,6 @@ class Relay_log_info : public Slave_reporting_capability */ bool sql_force_rotate_relay; - my_time_t last_master_timestamp; /* The SQL driver thread sets this true while it is waiting at the end of the relay log for more events to arrive. SHOW SLAVE STATUS uses this to report @@ -259,6 +258,19 @@ class Relay_log_info : public Slave_reporting_capability */ bool sql_thread_caught_up; + /* Last executed timestamp */ + my_time_t last_master_timestamp; + /* + Latest when + exec_time read from the master (by io_thread). + 0 if there has been no new update events since the slave started. + */ + time_t newest_master_timestamp; + /* + When + exec_time of the last committed event on the slave. + In case of delayed slave and slave_timestamp is not set + then set to when + exec_time -1 of the first seen event. + */ + time_t slave_timestamp; void clear_until_condition(); /** Reset the delay. @@ -395,7 +407,7 @@ class Relay_log_info : public Slave_reporting_capability /* Invalidate cached until_log_name and group_relay_log_name comparison - result. Should be called after any update of group_realy_log_name if + result. Should be called after any update of group_relay_log_name if there chances that sql_thread is running. */ inline void notify_group_relay_log_name_update() diff --git a/sql/slave.cc b/sql/slave.cc index 4d8b475aca6..61e2d0d731b 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -172,7 +172,7 @@ static int queue_event(Master_info *mi,const uchar *buf, ulong event_len); static int terminate_slave_thread(THD *, mysql_mutex_t *, mysql_cond_t *, volatile uint *, bool); static bool check_io_slave_killed(Master_info *mi, const char *info); -static bool send_show_master_info_data(THD *, Master_info *, bool, String *); + /* Function to set the slave's max_allowed_packet based on the value of slave_max_allowed_packet. @@ -1477,40 +1477,6 @@ bool net_request_file(NET* net, const char* fname) #endif /* HAVE_REPLICATION */ -bool Sql_cmd_show_slave_status::execute(THD *thd) -{ -#ifndef HAVE_REPLICATION - my_ok(thd); - return false; -#else - DBUG_ENTER("Sql_cmd_show_slave_status::execute"); - bool res= true; - - /* Accept one of two privileges */ - if (check_global_access(thd, PRIV_STMT_SHOW_SLAVE_STATUS)) - goto error; - if (is_show_all_slaves_stat()) - { - mysql_mutex_lock(&LOCK_active_mi); - res= show_all_master_info(thd); - mysql_mutex_unlock(&LOCK_active_mi); - } - else - { - LEX_MASTER_INFO *lex_mi= &thd->lex->mi; - Master_info *mi; - if ((mi= get_master_info(&lex_mi->connection_name, - Sql_condition::WARN_LEVEL_ERROR))) - { - res= show_master_info(thd, mi, 0); - mi->release(); - } - } -error: - DBUG_RETURN(res); -#endif -} - int init_strvar_from_file(char *var, int max_size, IO_CACHE *f, const char *default_val) { @@ -2874,287 +2840,6 @@ int register_slave_on_master(MYSQL* mysql, Master_info *mi, } -/** - Execute a SHOW SLAVE STATUS statement. - - @param thd Pointer to THD object for the client thread executing the - statement. - - @param mi Pointer to Master_info object for the IO thread. - - @retval FALSE success - @retval TRUE failure -*/ - -bool show_master_info(THD *thd, Master_info *mi, bool full) -{ - DBUG_ENTER("show_master_info"); - String gtid_pos; - List<Item> field_list; - - if (full && rpl_global_gtid_slave_state->tostring(&gtid_pos, NULL, 0)) - DBUG_RETURN(TRUE); - show_master_info_get_fields(thd, &field_list, full, gtid_pos.length()); - if (thd->protocol->send_result_set_metadata(&field_list, - Protocol::SEND_NUM_ROWS | Protocol::SEND_EOF)) - DBUG_RETURN(TRUE); - if (send_show_master_info_data(thd, mi, full, &gtid_pos)) - DBUG_RETURN(TRUE); - my_eof(thd); - DBUG_RETURN(FALSE); -} - -void show_master_info_get_fields(THD *thd, List<Item> *field_list, - bool full, size_t gtid_pos_length) -{ - Master_info *mi; - MEM_ROOT *mem_root= thd->mem_root; - DBUG_ENTER("show_master_info_get_fields"); - - if (full) - { - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Connection_name", - MAX_CONNECTION_NAME), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Slave_SQL_State", 30), - mem_root); - } - - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Slave_IO_State", 30), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Master_Host", sizeof(mi->host)), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Master_User", sizeof(mi->user)), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Master_Port", 7, MYSQL_TYPE_LONG), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Connect_Retry", 10, - MYSQL_TYPE_LONG), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Master_Log_File", FN_REFLEN), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Read_Master_Log_Pos", 10, - MYSQL_TYPE_LONGLONG), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Relay_Log_File", FN_REFLEN), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Relay_Log_Pos", 10, - MYSQL_TYPE_LONGLONG), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Relay_Master_Log_File", - FN_REFLEN), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Slave_IO_Running", 3), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Slave_SQL_Running", 3), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Replicate_Do_DB", 20), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Replicate_Ignore_DB", 20), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Replicate_Do_Table", 20), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Replicate_Ignore_Table", 23), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Replicate_Wild_Do_Table", 24), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Replicate_Wild_Ignore_Table", - 28), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Last_Errno", 4, MYSQL_TYPE_LONG), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Last_Error", 20), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Skip_Counter", 10, - MYSQL_TYPE_LONG), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Exec_Master_Log_Pos", 10, - MYSQL_TYPE_LONGLONG), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Relay_Log_Space", 10, - MYSQL_TYPE_LONGLONG), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Until_Condition", 6), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Until_Log_File", FN_REFLEN), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Until_Log_Pos", 10, - MYSQL_TYPE_LONGLONG), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Master_SSL_Allowed", 7), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Master_SSL_CA_File", - sizeof(mi->ssl_ca)), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Master_SSL_CA_Path", - sizeof(mi->ssl_capath)), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Master_SSL_Cert", - sizeof(mi->ssl_cert)), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Master_SSL_Cipher", - sizeof(mi->ssl_cipher)), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Master_SSL_Key", - sizeof(mi->ssl_key)), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Seconds_Behind_Master", 10, - MYSQL_TYPE_LONGLONG), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Master_SSL_Verify_Server_Cert", - 3), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Last_IO_Errno", 4, - MYSQL_TYPE_LONG), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Last_IO_Error", 20), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Last_SQL_Errno", 4, - MYSQL_TYPE_LONG), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Last_SQL_Error", 20), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Replicate_Ignore_Server_Ids", - FN_REFLEN), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Master_Server_Id", sizeof(ulong), - MYSQL_TYPE_LONG), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Master_SSL_Crl", - sizeof(mi->ssl_crl)), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Master_SSL_Crlpath", - sizeof(mi->ssl_crlpath)), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Using_Gtid", - sizeof("Current_Pos")-1), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Gtid_IO_Pos", 30), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Replicate_Do_Domain_Ids", - FN_REFLEN), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Replicate_Ignore_Domain_Ids", - FN_REFLEN), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Parallel_Mode", - sizeof("conservative")-1), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "SQL_Delay", 10, - MYSQL_TYPE_LONG)); - field_list->push_back(new (mem_root) - Item_return_int(thd, "SQL_Remaining_Delay", 8, - MYSQL_TYPE_LONG)); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Slave_SQL_Running_State", - 20)); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Slave_DDL_Groups", 20, - MYSQL_TYPE_LONGLONG), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Slave_Non_Transactional_Groups", 20, - MYSQL_TYPE_LONGLONG), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Slave_Transactional_Groups", 20, - MYSQL_TYPE_LONGLONG), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Replicate_Rewrite_DB", 23), - mem_root); - - /* - Note, we must never, _ever_, add extra rows to this output of SHOW SLAVE - STATUS, except here at the end before the extra rows of SHOW ALL SLAVES - STATUS. Otherwise, we break backwards compatibility with applications or - scripts that parse the output! - - This also means that we cannot add _any_ new rows in a GA version if a - different row was already added in a later MariaDB version, as this would - make it impossible to merge the change up while preserving the order of - rows. - */ - - if (full) - { - field_list->push_back(new (mem_root) - Item_return_int(thd, "Retried_transactions", 10, - MYSQL_TYPE_LONG), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Max_relay_log_size", 10, - MYSQL_TYPE_LONGLONG), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Executed_log_entries", 10, - MYSQL_TYPE_LONG), - mem_root); - field_list->push_back(new (mem_root) - Item_return_int(thd, "Slave_received_heartbeats", 10, - MYSQL_TYPE_LONG), - mem_root); - field_list->push_back(new (mem_root) - Item_float(thd, "Slave_heartbeat_period", 0.0, 3, 10), - mem_root); - field_list->push_back(new (mem_root) - Item_empty_string(thd, "Gtid_Slave_Pos", - (uint)gtid_pos_length), - mem_root); - } - DBUG_VOID_RETURN; -} - /* Text for Slave_IO_Running */ static const LEX_CSTRING slave_running[]= { @@ -3171,146 +2856,170 @@ static const LEX_CSTRING msg_ignored= { STRING_WITH_LEN("Ignored") }; #endif -static bool send_show_master_info_data(THD *thd, Master_info *mi, bool full, - String *gtid_pos) +inline void store_string_or_null(Field **field, const char *str) +{ + if (str) + (*field)->store(str, strlen(str), &my_charset_bin); + else + (*field)->set_null(); +} + +inline void store_string(Field **field, const char *str) +{ + (*field)->store(str, strlen(str), &my_charset_bin); +} + +inline void store_string(Field **field, const LEX_CSTRING *str) { - DBUG_ENTER("send_show_master_info_data"); + (*field)->store(str->str, str->length, &my_charset_bin); +} + - if (mi->host[0]) +void store_list(Field **field, I_List<i_string>* str_list) +{ + char buf[256]; + String tmp(buf, sizeof(buf), &my_charset_bin); + uint32 len; + I_List_iterator<i_string> it(*str_list); + i_string* s; + + tmp.length(0); + while ((s=it++)) { - DBUG_PRINT("info",("host is set: '%s'", mi->host)); - String *packet= &thd->packet; - Protocol *protocol= thd->protocol; - Rpl_filter *rpl_filter= mi->rpl_filter; - StringBuffer<256> tmp; - const char *msg; + tmp.append(s->ptr, strlen(s->ptr)); + tmp.append(','); + } + if ((len= tmp.length())) + len--; // Remove last ',' + (*field)->store((char*) tmp.ptr(), len, tmp.charset()); +} - protocol->prepare_for_resend(); - /* - slave_running can be accessed without run_lock but not other - non-volotile members like mi->io_thd, which is guarded by the mutex. - */ - if (full) - protocol->store(mi->connection_name.str, mi->connection_name.length, - &my_charset_bin); - mysql_mutex_lock(&mi->run_lock); - if (full) - { - /* - Show what the sql driver replication thread is doing - This is only meaningful if there is only one slave thread. - */ - msg= (mi->rli.sql_driver_thd ? - mi->rli.sql_driver_thd->get_proc_info() : ""); - protocol->store_string_or_null(msg, &my_charset_bin); - } - msg= mi->io_thd ? mi->io_thd->get_proc_info() : ""; - protocol->store_string_or_null(msg, &my_charset_bin); +/* + Store master info for information_schema_tables +*/ - mysql_mutex_unlock(&mi->run_lock); +void store_master_info(THD *thd, Master_info *mi, TABLE *table, + String *gtid_pos) +{ + Field **field= table->field; + const char *msg; + Rpl_filter *rpl_filter= mi->rpl_filter; + StringBuffer<256> tmp; + time_t master_timestamp, slave_timestamp; + DBUG_ENTER("store_master_info_data"); - mysql_mutex_lock(&mi->data_lock); - mysql_mutex_lock(&mi->rli.data_lock); - /* err_lock is to protect mi->last_error() */ - mysql_mutex_lock(&mi->err_lock); - /* err_lock is to protect mi->rli.last_error() */ - mysql_mutex_lock(&mi->rli.err_lock); - - DBUG_EXECUTE_IF("hold_sss_with_err_lock", { - DBUG_ASSERT(!debug_sync_set_action( - thd, STRING_WITH_LEN("now SIGNAL sss_got_err_lock " - "WAIT_FOR sss_continue"))); + table->clear_null_bits(); + + (*field++)->store(mi->connection_name.str, mi->connection_name.length, + &my_charset_bin); + + mysql_mutex_lock(&mi->run_lock); + msg= (mi->rli.sql_driver_thd ? + mi->rli.sql_driver_thd->get_proc_info() : ""); + store_string_or_null(field++, msg); + msg= mi->io_thd ? mi->io_thd->get_proc_info() : ""; + store_string_or_null(field++, msg); + mysql_mutex_unlock(&mi->run_lock); + + mysql_mutex_lock(&mi->data_lock); + mysql_mutex_lock(&mi->rli.data_lock); + /* err_lock is to protect mi->last_error() */ + mysql_mutex_lock(&mi->err_lock); + /* err_lock is to protect mi->rli.last_error() */ + mysql_mutex_lock(&mi->rli.err_lock); + + DBUG_EXECUTE_IF("hold_sss_with_err_lock", { + DBUG_ASSERT(!debug_sync_set_action(thd, + STRING_WITH_LEN("now SIGNAL sss_got_err_lock " + "WAIT_FOR sss_continue"))); DBUG_SET("-d,hold_sss_with_err_lock"); }); - protocol->store_string_or_null(mi->host, &my_charset_bin); - protocol->store_string_or_null(mi->user, &my_charset_bin); - protocol->store((uint32) mi->port); - protocol->store((uint32) mi->connect_retry); - protocol->store(mi->master_log_name, strlen(mi->master_log_name), + store_string_or_null(field++, mi->host); + store_string_or_null(field++, mi->user); + (*field++)->store((uint32) mi->port); + (*field++)->store((uint32) mi->connect_retry); + (*field++)->store(mi->master_log_name, strlen(mi->master_log_name), &my_charset_bin); - protocol->store((ulonglong) mi->master_log_pos); - msg= (mi->rli.group_relay_log_name + - dirname_length(mi->rli.group_relay_log_name)); - protocol->store(msg, strlen(msg), &my_charset_bin); - protocol->store((ulonglong) mi->rli.group_relay_log_pos); - protocol->store(mi->rli.group_master_log_name, - strlen(mi->rli.group_master_log_name), - &my_charset_bin); - protocol->store(&slave_running[mi->slave_running], &my_charset_bin); - protocol->store(mi->rli.slave_running ? &msg_yes : &msg_no, &my_charset_bin); - protocol->store(rpl_filter->get_do_db()); - protocol->store(rpl_filter->get_ignore_db()); - - rpl_filter->get_do_table(&tmp); - protocol->store(&tmp); - rpl_filter->get_ignore_table(&tmp); - protocol->store(&tmp); - rpl_filter->get_wild_do_table(&tmp); - protocol->store(&tmp); - rpl_filter->get_wild_ignore_table(&tmp); - protocol->store(&tmp); - - protocol->store(mi->rli.last_error().number); - protocol->store_string_or_null(mi->rli.last_error().message, - &my_charset_bin); - protocol->store((uint32) mi->rli.slave_skip_counter); - protocol->store((ulonglong) mi->rli.group_master_log_pos); - protocol->store((ulonglong) mi->rli.log_space_total); - - msg= (mi->rli.until_condition==Relay_log_info::UNTIL_NONE ? "None" : - (mi->rli.until_condition==Relay_log_info::UNTIL_MASTER_POS? "Master": - (mi->rli.until_condition==Relay_log_info::UNTIL_RELAY_POS? "Relay": - "Gtid"))); - protocol->store(msg, strlen(msg), &my_charset_bin); - protocol->store_string_or_null(mi->rli.until_log_name, &my_charset_bin); - protocol->store((ulonglong) mi->rli.until_log_pos); + (*field++)->store((ulonglong) mi->master_log_pos, true); + msg= (mi->rli.group_relay_log_name + + dirname_length(mi->rli.group_relay_log_name)); + store_string(field++, msg); + (*field++)->store((ulonglong) mi->rli.group_relay_log_pos, true); + store_string(field++, mi->rli.group_master_log_name); + store_string(field++, &slave_running[mi->slave_running]); + store_string(field++, mi->rli.slave_running ? &msg_yes : &msg_no); + store_list(field++, rpl_filter->get_do_db()); + store_list(field++, rpl_filter->get_ignore_db()); + + rpl_filter->get_do_table(&tmp); + (*field++)->store(tmp.ptr(), tmp.length(), &my_charset_bin); + rpl_filter->get_ignore_table(&tmp); + (*field++)->store(tmp.ptr(), tmp.length(), &my_charset_bin); + rpl_filter->get_wild_do_table(&tmp); + (*field++)->store(tmp.ptr(), tmp.length(), &my_charset_bin); + rpl_filter->get_wild_ignore_table(&tmp); + (*field++)->store(tmp.ptr(), tmp.length(), &my_charset_bin); + + (*field++)->store(mi->rli.last_error().number); + store_string_or_null(field++, mi->rli.last_error().message); + (*field++)->store((uint32) mi->rli.slave_skip_counter); + (*field++)->store((ulonglong) mi->rli.group_master_log_pos, true); + (*field++)->store((ulonglong) mi->rli.log_space_total, true); + + msg= (mi->rli.until_condition==Relay_log_info::UNTIL_NONE ? "None" : + (mi->rli.until_condition==Relay_log_info::UNTIL_MASTER_POS? "Master": + (mi->rli.until_condition==Relay_log_info::UNTIL_RELAY_POS? "Relay": + "Gtid"))); + (*field++)->store(msg, strlen(msg), &my_charset_bin); + store_string_or_null(field++, mi->rli.until_log_name); + (*field++)->store((ulonglong) mi->rli.until_log_pos, true); #ifdef HAVE_OPENSSL - protocol->store(mi->ssl ? &msg_yes : &msg_no, &my_charset_bin); + (*field++)->store(mi->ssl ? &msg_yes : &msg_no, &my_charset_bin); #else - protocol->store(mi->ssl ? &msg_ignored: &msg_no, &my_charset_bin); + (*field++)->store(mi->ssl ? &msg_ignored: &msg_no, &my_charset_bin); #endif - protocol->store_string_or_null(mi->ssl_ca, &my_charset_bin); - protocol->store_string_or_null(mi->ssl_capath, &my_charset_bin); - protocol->store_string_or_null(mi->ssl_cert, &my_charset_bin); - protocol->store_string_or_null(mi->ssl_cipher, &my_charset_bin); - protocol->store_string_or_null(mi->ssl_key, &my_charset_bin); + store_string_or_null(field++, mi->ssl_ca); + store_string_or_null(field++, mi->ssl_capath); + store_string_or_null(field++, mi->ssl_cert); + store_string_or_null(field++, mi->ssl_cipher); + store_string_or_null(field++, mi->ssl_key); - /* - Seconds_Behind_Master: if SQL thread is running and I/O thread is - connected, we can compute it otherwise show NULL (i.e. unknown). - */ - if ((mi->slave_running == MYSQL_SLAVE_RUN_READING) && - mi->rli.slave_running) - { - long time_diff; - bool idle; - time_t stamp= mi->rli.last_master_timestamp; + /* + Seconds_Behind_Master: if SQL thread is running and I/O thread is + connected, we can compute it otherwise show NULL (i.e. unknown). + */ + if ((mi->slave_running == MYSQL_SLAVE_RUN_READING) && + mi->rli.slave_running) + { + long time_diff; + bool idle; + time_t stamp= mi->rli.last_master_timestamp; - if (!stamp) - idle= true; - else - { - idle= mi->rli.sql_thread_caught_up; + if (!stamp) + idle= true; + else + { + idle= mi->rli.sql_thread_caught_up; - /* - The idleness of the SQL thread is needed for the parallel slave - because events can be ignored before distribution to a worker thread. - That is, Seconds_Behind_Master should still be calculated and visible - while the slave is processing ignored events, such as those skipped - due to slave_skip_counter. - */ - if (mi->using_parallel() && idle && - !rpl_parallel::workers_idle(&mi->rli)) - idle= false; - } - if (idle) - time_diff= 0; - else - { - time_diff= ((long)(time(0) - stamp) - mi->clock_diff_with_master); + /* + The idleness of the SQL thread is needed for the parallel slave + because events can be ignored before distribution to a worker thread. + That is, Seconds_Behind_Master should still be calculated and visible + while the slave is processing ignored events, such as those skipped + due to slave_skip_counter. + */ + if (mi->using_parallel() && idle && + !rpl_parallel::workers_idle(&mi->rli)) + idle= false; + } + if (idle) + time_diff= 0; + else + { + time_diff= ((long)(time(0) - stamp) - mi->clock_diff_with_master); /* Apparently on some systems time_diff can be <0. Here are possible reasons related to MySQL: @@ -3331,102 +3040,121 @@ static bool send_show_master_info_data(THD *thd, Master_info *mi, bool full, last_master_timestamp == 0 (an "impossible" timestamp 1970) is a special marker to say "consider we have caught up". */ - if (time_diff < 0) - time_diff= 0; - } - protocol->store((longlong)time_diff); - } - else - { - protocol->store_null(); + if (time_diff < 0) + time_diff= 0; } - protocol->store(mi->ssl_verify_server_cert? &msg_yes : &msg_no, + (*field++)->store((longlong)time_diff, true); + } + else + (*field++)->set_null(); + + (*field++)->store(mi->ssl_verify_server_cert? &msg_yes : &msg_no, &my_charset_bin); - // Last_IO_Errno - protocol->store(mi->last_error().number); - // Last_IO_Error - protocol->store_string_or_null(mi->last_error().message, &my_charset_bin); - // Last_SQL_Errno - protocol->store(mi->rli.last_error().number); - // Last_SQL_Error - protocol->store_string_or_null(mi->rli.last_error().message, - &my_charset_bin); - // Replicate_Ignore_Server_Ids - prot_store_ids(thd, &mi->ignore_server_ids); - // Master_Server_id - protocol->store((uint32) mi->master_id); - // SQL_Delay - // Master_Ssl_Crl - protocol->store_string_or_null(mi->ssl_crl, &my_charset_bin); - // Master_Ssl_Crlpath - protocol->store_string_or_null(mi->ssl_crlpath, &my_charset_bin); - // Using_Gtid - protocol->store_string_or_null(mi->using_gtid_astext(mi->using_gtid), - &my_charset_bin); - // Gtid_IO_Pos - { - mi->gtid_current_pos.to_string(&tmp); - protocol->store(tmp.ptr(), tmp.length(), &my_charset_bin); - } + // Last_IO_Errno + (*field++)->store(mi->last_error().number); + // Last_IO_Error + store_string_or_null(field++, mi->last_error().message); + // Last_SQL_Errno + (*field++)->store(mi->rli.last_error().number); + // Last_SQL_Error + store_string_or_null(field++, mi->rli.last_error().message); + // Replicate_Ignore_Server_Ids + field_store_ids((*field++), &mi->ignore_server_ids); + // Master_Server_id + (*field++)->store((uint32) mi->master_id); + // SQL_Delay + // Master_Ssl_Crl + store_string_or_null(field++, mi->ssl_crl); + // Master_Ssl_Crlpath + store_string_or_null(field++, mi->ssl_crlpath); + // Using_Gtid + store_string_or_null(field++, mi->using_gtid_astext(mi->using_gtid)); + // Gtid_IO_Pos + { + mi->gtid_current_pos.to_string(&tmp); + (*field++)->store(tmp.ptr(), tmp.length(), &my_charset_bin); + } - // Replicate_Do_Domain_Ids & Replicate_Ignore_Domain_Ids - mi->domain_id_filter.store_ids(thd); + // Replicate_Do_Domain_Ids & Replicate_Ignore_Domain_Ids + mi->domain_id_filter.store_ids(&field); - // Parallel_Mode - { - const char *mode_name= get_type(&slave_parallel_mode_typelib, - mi->parallel_mode); - protocol->store(mode_name, strlen(mode_name), &my_charset_bin); - } + // Parallel_Mode + { + const char *mode_name= get_type(&slave_parallel_mode_typelib, + mi->parallel_mode); + (*field++)->store(mode_name, strlen(mode_name), &my_charset_bin); + } + + (*field++)->store((uint32) mi->rli.get_sql_delay()); + // SQL_Remaining_Delay + // THD::proc_info is not protected by any lock, so we read it once + // to ensure that we use the same value throughout this function. + const char *slave_sql_running_state= + mi->rli.sql_driver_thd ? mi->rli.sql_driver_thd->proc_info : ""; + if (slave_sql_running_state == stage_sql_thd_waiting_until_delay.m_name) + { + time_t t= my_time(0), sql_delay_end= mi->rli.get_sql_delay_end(); + (*field++)->store((uint32)(t < sql_delay_end ? sql_delay_end - t : 0)); + } + else + (*field++)->set_null(); + // Slave_SQL_Running_State + store_string_or_null(field++, slave_sql_running_state); + + (*field++)->store(mi->total_ddl_groups, true); + (*field++)->store(mi->total_non_trans_groups, true); + (*field++)->store(mi->total_trans_groups, true); + rpl_filter->get_rewrite_db(&tmp); + (*field++)->store(tmp.ptr(), tmp.length(), &my_charset_bin); + + (*field++)->store((uint32) mi->rli.retried_trans, true); + (*field++)->store((ulonglong) mi->rli.max_relay_log_size, true); + (*field++)->store(mi->rli.executed_entries, true); + (*field++)->store((uint) mi->received_heartbeats, true); + (*field++)->store((double) mi->heartbeat_period); + (*field++)->store(gtid_pos->ptr(), gtid_pos->length(), &my_charset_bin); - protocol->store((uint32) mi->rli.get_sql_delay()); - // SQL_Remaining_Delay - // THD::proc_info is not protected by any lock, so we read it once - // to ensure that we use the same value throughout this function. - const char *slave_sql_running_state= - mi->rli.sql_driver_thd ? mi->rli.sql_driver_thd->proc_info : ""; - if (slave_sql_running_state == stage_sql_thd_waiting_until_delay.m_name) + /* + newest_master_timestamp is a guard for both newest_master_timestamp and + slave_timestamp. This is needed as newest_master_timestamp is only + updated when a commit is read while slave_timestamp is updated at + first event read from the relay log, which can happen before + newest_master_timestamp is read. + The below code also protects against a concurrent reset_slave(). + */ + if ((master_timestamp= mi->rli.newest_master_timestamp)) + { + (*field++)->store_timestamp((my_time_t) master_timestamp, 0); + if ((slave_timestamp= mi->rli.slave_timestamp)) { - time_t t= my_time(0), sql_delay_end= mi->rli.get_sql_delay_end(); - protocol->store((uint32)(t < sql_delay_end ? sql_delay_end - t : 0)); + (*field++)->store_timestamp((my_time_t) slave_timestamp, 0); + (*field++)->store((uint) (master_timestamp - slave_timestamp), true); } else - protocol->store_null(); - // Slave_SQL_Running_State - protocol->store_string_or_null(slave_sql_running_state, &my_charset_bin); - - protocol->store(mi->total_ddl_groups); - protocol->store(mi->total_non_trans_groups); - protocol->store(mi->total_trans_groups); - protocol->store(rpl_filter->get_rewrite_db()); - - if (full) { - protocol->store((uint32) mi->rli.retried_trans); - protocol->store((ulonglong) mi->rli.max_relay_log_size); - protocol->store(mi->rli.executed_entries); - protocol->store((uint32) mi->received_heartbeats); - protocol->store_double(mi->heartbeat_period, 3); - protocol->store(gtid_pos->ptr(), gtid_pos->length(), &my_charset_bin); + (*field++)->set_null(); + (*field++)->set_null(); } - - mysql_mutex_unlock(&mi->rli.err_lock); - mysql_mutex_unlock(&mi->err_lock); - mysql_mutex_unlock(&mi->rli.data_lock); - mysql_mutex_unlock(&mi->data_lock); - - if (my_net_write(&thd->net, (uchar*) thd->packet.ptr(), packet->length())) - DBUG_RETURN(TRUE); } - DBUG_RETURN(FALSE); + else + { + (*field++)->set_null(); + (*field++)->set_null(); + (*field++)->set_null(); + } + mysql_mutex_unlock(&mi->rli.err_lock); + mysql_mutex_unlock(&mi->err_lock); + mysql_mutex_unlock(&mi->rli.data_lock); + mysql_mutex_unlock(&mi->data_lock); + DBUG_VOID_RETURN; } /* Used to sort connections by name */ -static int cmp_mi_by_name(const Master_info **arg1, - const Master_info **arg2) +int cmp_mi_by_name(const Master_info **arg1, + const Master_info **arg2) { return Lex_ident_master_info::charset_info()->strnncoll( (*arg1)->connection_name, @@ -3434,72 +3162,6 @@ static int cmp_mi_by_name(const Master_info **arg1, } -/** - Execute a SHOW FULL SLAVE STATUS statement. - - @param thd Pointer to THD object for the client thread executing the - statement. - - Elements are sorted according to the original connection_name. - - @retval FALSE success - @retval TRUE failure - - @note - master_info_index is protected by LOCK_active_mi. -*/ - -bool show_all_master_info(THD* thd) -{ - uint i, elements; - String gtid_pos; - Master_info **tmp; - List<Item> field_list; - DBUG_ENTER("show_all_master_info"); - mysql_mutex_assert_owner(&LOCK_active_mi); - - gtid_pos.length(0); - if (rpl_append_gtid_state(&gtid_pos, true)) - { - my_error(ER_OUT_OF_RESOURCES, MYF(0)); - DBUG_RETURN(TRUE); - } - - show_master_info_get_fields(thd, &field_list, 1, gtid_pos.length()); - if (thd->protocol->send_result_set_metadata(&field_list, - Protocol::SEND_NUM_ROWS | Protocol::SEND_EOF)) - DBUG_RETURN(TRUE); - - if (!master_info_index || - !(elements= master_info_index->master_info_hash.records)) - goto end; - - /* - Sort lines to get them into a predicted order - (needed for test cases and to not confuse users) - */ - if (!(tmp= (Master_info**) thd->alloc(sizeof(Master_info*) * elements))) - DBUG_RETURN(TRUE); - - for (i= 0; i < elements; i++) - { - tmp[i]= (Master_info *) my_hash_element(&master_info_index-> - master_info_hash, i); - } - my_qsort(tmp, elements, sizeof(Master_info*), (qsort_cmp) cmp_mi_by_name); - - for (i= 0; i < elements; i++) - { - if (send_show_master_info_data(thd, tmp[i], 1, &gtid_pos)) - DBUG_RETURN(TRUE); - } - -end: - my_eof(thd); - DBUG_RETURN(FALSE); -} - - void set_slave_thread_options(THD* thd) { DBUG_ENTER("set_slave_thread_options"); @@ -4353,13 +4015,32 @@ static int exec_relay_log_event(THD* thd, Relay_log_info* rli, the user might be surprised to see a claim that the slave is up to date long before those queued events are actually executed. */ - if ((!rli->mi->using_parallel()) && - event_can_update_last_master_timestamp(ev)) + if (event_can_update_last_master_timestamp(ev)) { - rli->last_master_timestamp= ev->when + ev->exec_time; - rli->sql_thread_caught_up= false; - } + if ((!rli->mi->using_parallel())) + { + rli->last_master_timestamp= ev->when + (time_t) ev->exec_time; + rli->sql_thread_caught_up= false; + /* + For slave_timestamp, we update slave_timestamp at the end of the + transaction, so we follow the pattern of the parallel slave and + cache the timestamp of the last-event of the transaction within the + RGI, and then use it to update slave_timestamp at commit-time. + */ + if (Log_event::is_group_event(typ)) + serial_rgi->last_master_timestamp= rli->last_master_timestamp; + } + + if (unlikely(!rli->slave_timestamp) && Log_event::is_group_event(typ)) + { + /* + First event for this slave. Assume that all the slave was up to date + with the master just before the current event. + */ + rli->slave_timestamp= (time_t) ev->when + (time_t) ev->exec_time-1; + } + } /* This tests if the position of the beginning of the current event hits the UNTIL barrier. @@ -7027,6 +6708,23 @@ static int queue_event(Master_info* mi, const uchar *buf, ulong event_len) } else { + /* + replay_log.description_event_for_exec can be null if the slave thread + is getting killed + */ + if (LOG_EVENT_IS_QUERY((Log_event_type) buf[EVENT_TYPE_OFFSET]) || + LOG_EVENT_IS_LOAD_DATA((Log_event_type) buf[EVENT_TYPE_OFFSET])) + { + time_t exec_time= query_event_get_time(buf, rli->relay_log. + description_event_for_queue); + set_if_bigger(rli->newest_master_timestamp, exec_time); + } + else if (((Log_event_type) buf[EVENT_TYPE_OFFSET]) == XID_EVENT) + { + /* XID_EVENT is used for COMMIT */ + time_t commit_time= uint4korr(buf); + set_if_bigger(rli->newest_master_timestamp, commit_time); + } if (mi->do_accept_own_server_id) { int2store(const_cast<uchar*>(buf + FLAGS_OFFSET), diff --git a/sql/slave.h b/sql/slave.h index 02de9135c2a..5dd7d6e262b 100644 --- a/sql/slave.h +++ b/sql/slave.h @@ -279,7 +279,9 @@ void slave_background_kill_request(THD *to_kill); void slave_background_gtid_pos_create_request (rpl_slave_state::gtid_pos_table *table_entry); void slave_background_gtid_pending_delete_request(void); - +void store_master_info(THD *thd, Master_info *mi, TABLE *table, + String *gtid_pos); +int cmp_mi_by_name(const Master_info **arg1, const Master_info **arg2); extern Master_info *active_mi; /* active_mi for multi-master */ extern Master_info *default_master_info; /* To replace active_mi */ extern Master_info_index *master_info_index; diff --git a/sql/sql_class.h b/sql/sql_class.h index 2d671139a6c..05c9c4e8012 100644 --- a/sql/sql_class.h +++ b/sql/sql_class.h @@ -3168,6 +3168,7 @@ class THD: public THD_count, /* this must be first */ ulonglong start_utime, utime_after_lock, utime_after_query; /* This can be used by handlers to send signals to the SQL level */ ulonglong replication_flags; + my_time_t orig_exec_time; // Exec time for last read binlog event // Process indicator struct { /* diff --git a/sql/sql_cmd.h b/sql/sql_cmd.h index bedd7e118f0..c9681c25d9f 100644 --- a/sql/sql_cmd.h +++ b/sql/sql_cmd.h @@ -397,26 +397,6 @@ class Sql_cmd_dml : public Sql_cmd }; -class Sql_cmd_show_slave_status: public Sql_cmd -{ -protected: - bool show_all_slaves_status; -public: - Sql_cmd_show_slave_status() - :show_all_slaves_status(false) - {} - - Sql_cmd_show_slave_status(bool status_all) - :show_all_slaves_status(status_all) - {} - - enum_sql_command sql_command_code() const { return SQLCOM_SHOW_SLAVE_STAT; } - - bool execute(THD *thd); - bool is_show_all_slaves_stat() { return show_all_slaves_status; } -}; - - class Sql_cmd_create_table_like: public Sql_cmd, public Storage_engine_name { diff --git a/sql/sql_lex.h b/sql/sql_lex.h index 93756d8d80d..0b1f3cf8e15 100644 --- a/sql/sql_lex.h +++ b/sql/sql_lex.h @@ -499,6 +499,8 @@ struct LEX_MASTER_INFO int sql_delay; bool is_demotion_opt; bool is_until_before_gtids; + bool show_all_slaves; + /* Enum is used for making it possible to detect if the user changed variable or if it should be left at old value diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc index 507a068b731..01fb010e974 100644 --- a/sql/sql_parse.cc +++ b/sql/sql_parse.cc @@ -3916,6 +3916,7 @@ mysql_execute_command(THD *thd, bool is_called_from_prepared_stmt) case SQLCOM_SHOW_COLLATIONS: case SQLCOM_SHOW_STORAGE_ENGINES: case SQLCOM_SHOW_PROFILE: + case SQLCOM_SHOW_SLAVE_STAT: case SQLCOM_SELECT: { #ifdef WITH_WSREP @@ -5829,7 +5830,6 @@ mysql_execute_command(THD *thd, bool is_called_from_prepared_stmt) DBUG_ASSERT(first_table == all_tables && first_table != 0); /* fall through */ case SQLCOM_ALTER_SEQUENCE: - case SQLCOM_SHOW_SLAVE_STAT: case SQLCOM_SIGNAL: case SQLCOM_RESIGNAL: case SQLCOM_GET_DIAGNOSTICS: diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc index 6261ba46cac..4a0f55562d7 100644 --- a/sql/sql_prepare.cc +++ b/sql/sql_prepare.cc @@ -1861,30 +1861,6 @@ static int mysql_test_show_grants(Prepared_statement *stmt) #ifndef EMBEDDED_LIBRARY -/** - Validate and prepare for execution SHOW SLAVE STATUS statement. - - @param stmt prepared statement - - @retval - FALSE success - @retval - TRUE error, error message is set in THD -*/ - -static int mysql_test_show_slave_status(Prepared_statement *stmt, - bool show_all_slaves_stat) -{ - DBUG_ENTER("mysql_test_show_slave_status"); - THD *thd= stmt->thd; - List<Item> fields; - - show_master_info_get_fields(thd, &fields, show_all_slaves_stat, 0); - - DBUG_RETURN(send_stmt_metadata(thd, stmt, &fields)); -} - - /** Validate and prepare for execution SHOW BINLOG STATUS statement. @@ -2309,6 +2285,7 @@ static bool check_prepared_statement(Prepared_statement *stmt) case SQLCOM_SHOW_STATUS_FUNC: case SQLCOM_SHOW_STATUS_PACKAGE: case SQLCOM_SHOW_STATUS_PACKAGE_BODY: + case SQLCOM_SHOW_SLAVE_STAT: case SQLCOM_SELECT: res= mysql_test_select(stmt, tables); if (res == 2) @@ -2345,21 +2322,6 @@ static bool check_prepared_statement(Prepared_statement *stmt) break; #endif /* NO_EMBEDDED_ACCESS_CHECKS */ #ifndef EMBEDDED_LIBRARY - case SQLCOM_SHOW_SLAVE_STAT: - { - DBUG_ASSERT(thd->lex->m_sql_cmd); - Sql_cmd_show_slave_status *cmd; - cmd= dynamic_cast<Sql_cmd_show_slave_status*>(thd->lex->m_sql_cmd); - DBUG_ASSERT(cmd); - if ((res= mysql_test_show_slave_status(stmt, - cmd->is_show_all_slaves_stat())) - == 2) - { - /* Statement and field info has already been sent */ - DBUG_RETURN(FALSE); - } - break; - } case SQLCOM_SHOW_BINLOG_STAT: if ((res= mysql_test_show_binlog_status(stmt)) == 2) { diff --git a/sql/sql_repl.cc b/sql/sql_repl.cc index 8ca252c37c6..ba8266d46e9 100644 --- a/sql/sql_repl.cc +++ b/sql/sql_repl.cc @@ -3700,6 +3700,8 @@ int reset_slave(THD *thd, Master_info* mi) mi->rli.clear_until_condition(); mi->rli.clear_sql_delay(); mi->rli.slave_skip_counter= 0; + mi->rli.newest_master_timestamp= 0; + mi->rli.slave_timestamp= 0; // close master_info_file, relay_log_info_file, set mi->inited=rli->inited=0 end_master_info(mi); diff --git a/sql/sql_show.cc b/sql/sql_show.cc index aa494b90d36..d7d798eb36f 100644 --- a/sql/sql_show.cc +++ b/sql/sql_show.cc @@ -47,6 +47,8 @@ #include "sql_derived.h" #include "sql_statistics.h" #include "sql_connect.h" +#include "sql_repl.h" // rpl_load_gtid_state +#include "rpl_mi.h" // master_info_index #include "authors.h" #include "contributors.h" #include "sql_partition.h" @@ -8836,6 +8838,90 @@ TABLE *create_schema_table(THD *thd, TABLE_LIST *table_list) } +#ifdef HAVE_REPLICATION +int fill_slave_status(THD *thd, TABLE_LIST *tables, COND *cond) +{ + String gtid_pos; + Master_info **tmp; + TABLE *table= tables->table; + uint elements, i; + bool single_slave= (thd->lex->sql_command == SQLCOM_SHOW_SLAVE_STAT && + !thd->lex->mi.show_all_slaves); + DBUG_ENTER("fill_slave_status"); + + if (check_global_access(thd, PRIV_STMT_SHOW_SLAVE_STATUS)) + DBUG_RETURN(TRUE); + + gtid_pos.length(0); + if (rpl_append_gtid_state(&gtid_pos, true)) + { + my_error(ER_OUT_OF_RESOURCES, MYF(0)); + DBUG_RETURN(TRUE); + } + + if (!master_info_index || + !(elements= master_info_index->master_info_hash.records)) + { + /* No registered slaves */ + return 0; + } + + /* + Sort lines to get them into a predicted order + (needed for test cases and to not confuse users) + */ + if (!(tmp= (Master_info**) thd->alloc(sizeof(Master_info*) * elements))) + goto error; + + if (single_slave) + { + LEX_MASTER_INFO *lex_mi= &thd->lex->mi; + Master_info *mi; + if ((mi= get_master_info(&lex_mi->connection_name, + Sql_condition::WARN_LEVEL_ERROR))) + { + bool res= 0; + if (mi->host[0]) + { + store_master_info(thd, mi, table, &gtid_pos); + res= schema_table_store_record(thd, table); + } + mi->release(); + if (res) + goto error; + } + } + else + { + mysql_mutex_lock(&LOCK_active_mi); + for (i= 0; i < elements; i++) + { + tmp[i]= (Master_info *) my_hash_element(&master_info_index-> + master_info_hash, i); + } + my_qsort(tmp, elements, sizeof(Master_info*), (qsort_cmp) cmp_mi_by_name); + + for (i= 0; i < elements; i++) + { + if (tmp[i]->host[0]) + { + store_master_info(thd, tmp[i], table, &gtid_pos); + if (schema_table_store_record(thd, table)) + { + mysql_mutex_unlock(&LOCK_active_mi); + goto error; + } + } + } + mysql_mutex_unlock(&LOCK_active_mi); + } + DBUG_RETURN(0); + +error: + DBUG_RETURN(1); +} +#endif + /* For old SHOW compatibility. It is used when old SHOW doesn't have generated column names @@ -8862,12 +8948,11 @@ static int make_old_format(THD *thd, ST_SCHEMA_TABLE *schema_table) LEX_CSTRING field_name= field_info->name(); Item_field *field= new (thd->mem_root) Item_field(thd, context, field_name); - if (field) - { - field->set_name(thd, field_info->old_name()); - if (add_item_to_list(thd, field)) - return 1; - } + if (!field) + return 1; + field->set_name(thd, field_info->old_name()); + if (add_item_to_list(thd, field)) + return 1; } } return 0; @@ -8922,14 +9007,14 @@ int make_table_names_old_format(THD *thd, ST_SCHEMA_TABLE *schema_table) buffer.append(')'); } Item_field *field= new (thd->mem_root) Item_field(thd, context, field_name); - if (add_item_to_list(thd, field)) + if (!field || add_item_to_list(thd, field)) return 1; field->set_name(thd, &buffer); if (thd->lex->verbose) { field_info= &schema_table->fields_info[3]; field= new (thd->mem_root) Item_field(thd, context, field_info->name()); - if (add_item_to_list(thd, field)) + if (! field || add_item_to_list(thd, field)) return 1; field->set_name(thd, field_info->old_name()); } @@ -8953,12 +9038,11 @@ int make_columns_old_format(THD *thd, ST_SCHEMA_TABLE *schema_table) continue; Item_field *field= new (thd->mem_root) Item_field(thd, context, field_info->name()); - if (field) - { - field->set_name(thd, field_info->old_name()); - if (add_item_to_list(thd, field)) - return 1; - } + if (!field) + return 1; + field->set_name(thd, field_info->old_name()); + if (add_item_to_list(thd, field)) + return 1; } return 0; } @@ -8976,17 +9060,15 @@ int make_character_sets_old_format(THD *thd, ST_SCHEMA_TABLE *schema_table) field_info= &schema_table->fields_info[*field_num]; Item_field *field= new (thd->mem_root) Item_field(thd, context, field_info->name()); - if (field) - { - field->set_name(thd, field_info->old_name()); - if (add_item_to_list(thd, field)) - return 1; - } + if (!field) + return 1; + field->set_name(thd, field_info->old_name()); + if (add_item_to_list(thd, field)) + return 1; } return 0; } - int make_proc_old_format(THD *thd, ST_SCHEMA_TABLE *schema_table) { int fields_arr[]= {2, 3, 4, 27, 24, 23, 22, 26, 28, 29, 30, -1}; @@ -8999,10 +9081,44 @@ int make_proc_old_format(THD *thd, ST_SCHEMA_TABLE *schema_table) field_info= &schema_table->fields_info[*field_num]; Item_field *field= new (thd->mem_root) Item_field(thd, context, field_info->name()); - if (field) + if (!field) + return 1; + field->set_name(thd, field_info->old_name()); + if (add_item_to_list(thd, field)) + return 1; + } + return 0; +} + + +static int make_slave_status_old_format(THD *thd, ST_SCHEMA_TABLE *schema_table) +{ + ST_FIELD_INFO *field_info= schema_table->fields_info; + Name_resolution_context *context= &thd->lex->first_select_lex()->context; + DBUG_ASSERT(thd->lex->sql_command == SQLCOM_SHOW_SLAVE_STAT); + bool all_slaves= thd->lex->mi.show_all_slaves; + ulonglong used_fields= ~0ULL; + + if (!all_slaves) + { + /* Remove 2 first fields and all fields above and including field 55 */ + used_fields&= ~((1ULL << 0) | (1ULL << 1)); + used_fields&= ((1ULL << 56)-1); + } + + for (uint i=0; !field_info->end_marker(); field_info++, i++) + { + /* + We have all_slaves here to take into account that we some day may have + more than 64 fields in the list. If all_slaves is set we should show + all fields. + */ + if (all_slaves || (used_fields & ((1ULL << i)))) { - field->set_name(thd, field_info->old_name()); - if (add_item_to_list(thd, field)) + LEX_CSTRING field_name= field_info->name(); + Item_field *field= new (thd->mem_root) + Item_field(thd, context, field_name); + if (!field || add_item_to_list(thd, field)) return 1; } } @@ -10249,10 +10365,7 @@ ST_FIELD_INFO files_fields_info[]= void init_fill_schema_files_row(TABLE* table) { - int i; - for(i=0; !Show::files_fields_info[i].end_marker(); i++) - table->field[i]->set_null(); - + table->set_null_bits(); table->field[IS_FILES_STATUS]->set_notnull(); table->field[IS_FILES_STATUS]->store("NORMAL", 6, system_charset_info); } @@ -10414,8 +10527,78 @@ ST_FIELD_INFO check_constraints_fields_info[]= CEnd() }; -}; // namespace Show +ST_FIELD_INFO slave_status_info[]= +{ + Column("Connection_name", Name(), NOT_NULL), + Column("Slave_SQL_State", Varchar(64), NULLABLE), + Column("Slave_IO_State", Varchar(64), NULLABLE), + Column("Master_Host", Varchar(HOSTNAME_LENGTH), NULLABLE), + Column("Master_User", Varchar(USERNAME_LENGTH), NULLABLE), + Column("Master_Port", ULong(7), NOT_NULL), + Column("Connect_Retry", SLong(10), NOT_NULL), + Column("Master_Log_File", Varchar(FN_REFLEN), NOT_NULL), + Column("Read_Master_Log_Pos", ULonglong(10), NOT_NULL), + Column("Relay_Log_File", Varchar(FN_REFLEN), NOT_NULL), + Column("Relay_Log_Pos", ULonglong(10), NOT_NULL), + Column("Relay_Master_Log_File", Varchar(FN_REFLEN), NOT_NULL), + Column("Slave_IO_Running", Varchar(10), NOT_NULL), + Column("Slave_SQL_Running", Varchar(3), NOT_NULL), + Column("Replicate_Do_DB", Name(), NOT_NULL), + Column("Replicate_Ignore_DB", Name(), NOT_NULL), + Column("Replicate_Do_Table", Name(), NOT_NULL), + Column("Replicate_Ignore_Table", Name(), NOT_NULL), + Column("Replicate_Wild_Do_Table", Name(), NOT_NULL), + Column("Replicate_Wild_Ignore_Table", Name(), NOT_NULL), + Column("Last_Errno", SLong(4), NOT_NULL), + Column("Last_Error", Varchar(20), NULLABLE), + Column("Skip_Counter", ULong(10), NOT_NULL), + Column("Exec_Master_Log_Pos", ULonglong(10), NOT_NULL), + Column("Relay_Log_Space", ULonglong(10), NOT_NULL), + Column("Until_Condition", Varchar(6), NOT_NULL), + Column("Until_Log_File", Varchar(FN_REFLEN), NULLABLE), + Column("Until_Log_Pos", ULonglong(10), NOT_NULL), + Column("Master_SSL_Allowed", Varchar(7), NULLABLE), + Column("Master_SSL_CA_File", Varchar(FN_REFLEN), NULLABLE), + Column("Master_SSL_CA_Path", Varchar(FN_REFLEN), NULLABLE), + Column("Master_SSL_Cert", Varchar(FN_REFLEN), NULLABLE), + Column("Master_SSL_Cipher", Varchar(FN_REFLEN), NULLABLE), + Column("Master_SSL_Key", Varchar(FN_REFLEN), NULLABLE), + Column("Seconds_Behind_Master", SLonglong(10), NULLABLE), + Column("Master_SSL_Verify_Server_Cert", Varchar(3), NOT_NULL), + Column("Last_IO_Errno", SLong(4), NOT_NULL), + Column("Last_IO_Error", Varchar(MYSQL_ERRMSG_SIZE), NULLABLE), + Column("Last_SQL_Errno", SLong(4), NOT_NULL), + Column("Last_SQL_Error", Varchar(MYSQL_ERRMSG_SIZE), NULLABLE), + Column("Replicate_Ignore_Server_Ids", Varchar(FN_REFLEN), NOT_NULL), + Column("Master_Server_Id", ULong(10), NOT_NULL), + Column("Master_SSL_Crl", Varchar(FN_REFLEN), NULLABLE), + Column("Master_SSL_Crlpath", Varchar(FN_REFLEN), NULLABLE), + Column("Using_Gtid", Varchar(15), NULLABLE), + Column("Gtid_IO_Pos", Varchar(1024), NOT_NULL), + Column("Replicate_Do_Domain_Ids", Varchar(FN_REFLEN), NOT_NULL), + Column("Replicate_Ignore_Domain_Ids", Varchar(FN_REFLEN), NOT_NULL), + Column("Parallel_Mode", Varchar(15), NOT_NULL), + Column("SQL_Delay", ULong(10), NOT_NULL), + Column("SQL_Remaining_Delay", ULong(10), NULLABLE), + Column("Slave_SQL_Running_State", Varchar(64), NULLABLE), + Column("Slave_DDL_Groups", ULonglong(20), NOT_NULL), + Column("Slave_Non_Transactional_Groups", ULonglong(20), NOT_NULL), + Column("Slave_Transactional_Groups", ULonglong(20), NOT_NULL), + Column("Replicate_Rewrite_DB",Varchar(1024), NOT_NULL), + Column("Retried_transactions", ULong(10), NOT_NULL), + Column("Max_relay_log_size", ULonglong(10), NOT_NULL), + Column("Executed_log_entries", ULong(10), NOT_NULL), + Column("Slave_received_heartbeats", ULong(10), NOT_NULL), + Column("Slave_heartbeat_period", Float(703), NOT_NULL), // 3 decimals + Column("Gtid_Slave_Pos", Varchar(FN_REFLEN), NOT_NULL), + Column("Master_last_event_time", Datetime(0), NULLABLE), + Column("Slave_last_event_time", Datetime(0), NULLABLE), + Column("Master_Slave_time_diff", SLonglong(10), NULLABLE), + CEnd() +}; + +}; // namespace Show namespace Show { @@ -10565,6 +10748,10 @@ ST_SCHEMA_TABLE schema_tables[]= {"VIEWS"_Lex_ident_i_s_table, Show::view_fields_info, 0, get_all_tables, 0, get_schema_views_record, 1, 2, 0, OPEN_VIEW_ONLY|OPTIMIZE_I_S_TABLE|I_S_EXTENDED_ERROR_HANDLING}, +#ifdef HAVE_REPLICATION + {"SLAVE_STATUS"_Lex_ident_i_s_table, Show::slave_status_info, 0, + fill_slave_status, make_slave_status_old_format, 0, 1, 0, 0, 0 }, +#endif {Lex_ident_i_s_table(), 0, 0, 0, 0, 0, 0, 0, 0, 0} }; diff --git a/sql/sql_type.cc b/sql/sql_type.cc index 0c505c29503..dc5daac0341 100644 --- a/sql/sql_type.cc +++ b/sql/sql_type.cc @@ -3971,12 +3971,22 @@ Field *Type_handler_float::make_schema_field(MEM_ROOT *root, TABLE *table, const ST_FIELD_INFO &def) const { LEX_CSTRING name= def.name(); + uint32 len= def.char_length(); + uint dec= NOT_FIXED_DEC; + if (len >= 100) + { + dec= def.decimal_scale(); + uint prec= def.decimal_precision(); + /* Field defined in sql_show.cc with decimals */ + len= my_decimal_precision_to_length(prec, dec, 0); + } + return new (root) - Field_float(addr.ptr(), def.char_length(), - addr.null_ptr(), addr.null_bit(), - Field::NONE, &name, - (uint8) NOT_FIXED_DEC, - 0/*zerofill*/, def.unsigned_flag()); + Field_float(addr.ptr(), len, + addr.null_ptr(), addr.null_bit(), + Field::NONE, &name, + dec, + 0/*zerofill*/, def.unsigned_flag()); } diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy index 0d76b57d6c0..f17c6b23c93 100644 --- a/sql/sql_yacc.yy +++ b/sql/sql_yacc.yy @@ -14225,17 +14225,19 @@ show_param: } | ALL SLAVES STATUS_SYM { - if (!(Lex->m_sql_cmd= new (thd->mem_root) - Sql_cmd_show_slave_status(true))) + LEX *lex= Lex; + lex->sql_command= SQLCOM_SHOW_SLAVE_STAT; + lex->mi.show_all_slaves= 1; + if (prepare_schema_table(thd, lex, 0, SCH_SLAVE_STATUS)) MYSQL_YYABORT; - Lex->sql_command = SQLCOM_SHOW_SLAVE_STAT; } | SLAVE optional_connection_name STATUS_SYM optional_for_channel { - if (!(Lex->m_sql_cmd= new (thd->mem_root) - Sql_cmd_show_slave_status())) + LEX *lex= Lex; + lex->sql_command= SQLCOM_SHOW_SLAVE_STAT; + lex->mi.show_all_slaves= 0; + if (prepare_schema_table(thd, lex, 0, SCH_SLAVE_STATUS)) MYSQL_YYABORT; - Lex->sql_command = SQLCOM_SHOW_SLAVE_STAT; } | CREATE PROCEDURE_SYM sp_name { diff --git a/sql/table.h b/sql/table.h index bd5c11d9430..5648edd2667 100644 --- a/sql/table.h +++ b/sql/table.h @@ -1646,6 +1646,16 @@ struct TABLE bool fill_item_list(List<Item> *item_list) const; void reset_item_list(List<Item> *item_list, uint skip) const; void clear_column_bitmaps(void); + inline void clear_null_bits() + { + if (s->null_bytes) + bzero(null_flags, s->null_bytes); + } + inline void set_null_bits() + { + if (s->null_bytes) + bfill(null_flags, s->null_bytes, 255); + } void prepare_for_position(void); MY_BITMAP *prepare_for_keyread(uint index, MY_BITMAP *map); MY_BITMAP *prepare_for_keyread(uint index) @@ -3494,8 +3504,7 @@ inline void mark_as_null_row(TABLE *table) { table->null_row=1; table->status|=STATUS_NULL_ROW; - if (table->s->null_bytes) - bfill(table->null_flags,table->s->null_bytes,255); + table->set_null_bits(); } /* -- 2.39.2

1 0

[PATCH] MDEV-15393 part 2. Fixes to 'gtid_slave_pos duplicate key errors after mysqldump restore' MDEV-34615 mysqldump wipes off pre-existing gtid slave state
by Kristian Nielsen 17 Jul '24

17 Jul '24

From: Andrei <andrei.elkin(a)mariadb.com> When mysqldump run on mysql system database it generates inserts sql commands into mysql.gtid_slave_pos. After running the backup script those inserts did not produce an expected gtid state on slave. In particular the maximum of mysql.gtid_slave_pos.sub_id did not make into rpl_global_gtid_slave_state.last_sub_id a memory object that the table persists. And that was regardless of whether --gtid option in or out. Later when the backup recipient server starts as slave in *non-gtid* mode this desychronization may lead to a duplicate key error. This effect is corrected for --gtid mode mysqldump/mariadb-dump only as the following. The fixes ensure the insert block of the dump script is followed with a "summing-up" SET @global.gtid_slave_pos assignment. To address MDEV-34615 the assignment makes sure a possibly pre-existing gtid state (e.g reflecting another domain) remains as a sub-state upon the dump installation. A new test provides a how-to-reproduce mode as well. For the implemenation part, note a deferred print-out of SET-gtid_slave_pos and associated comments is prefered over relocating of the entire blocks if (opt_master,slave_data && do_show_master,slave_status) ... because of compatiblity concern. Namely an error inside do_show_*() is handled in the new code the same way, as early, as before. --- client/mysqldump.c | 53 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 12 deletions(-) diff --git a/client/mysqldump.c b/client/mysqldump.c index 1c9d92d6e09..0be890e2528 100644 --- a/client/mysqldump.c +++ b/client/mysqldump.c @@ -5985,8 +5985,13 @@ static int dump_selected_tables(char *db, char **table_names, int tables) } /* dump_selected_tables */ +const char fmt_gtid_pos[] = "%sSET GLOBAL gtid_slave_pos=concat('%s', " + "if(length(@@global.gtid_slave_pos) = 0, '', ','), " + "@@global.gtid_slave_pos);\n"; + static int do_show_master_status(MYSQL *mysql_con, int consistent_binlog_pos, - int have_mariadb_gtid, int use_gtid) + int have_mariadb_gtid, int use_gtid, + char *set_gtid_pos) { MYSQL_ROW row; MYSQL_RES *UNINIT_VAR(master); @@ -6054,8 +6059,11 @@ static int do_show_master_status(MYSQL *mysql_con, int consistent_binlog_pos, fprintf(md_result_file, "%sCHANGE MASTER TO MASTER_USE_GTID=slave_pos;\n", comment_prefix); - fprintf(md_result_file, - "%sSET GLOBAL gtid_slave_pos='%s';\n", + /* + Defer print of SET gtid_slave_pos until after its placeholder + table is guaranteed to have been dumped. + */ + sprintf(set_gtid_pos, fmt_gtid_pos, (!use_gtid ? "-- " : comment_prefix), gtid_pos); } @@ -6140,8 +6148,8 @@ static int add_slave_statements(void) return(0); } -static int do_show_slave_status(MYSQL *mysql_con, int use_gtid, - int have_mariadb_gtid) +static int do_show_slave_status(MYSQL *mysql_con, int have_mariadb_gtid, + int use_gtid, char* set_gtid_pos) { MYSQL_RES *UNINIT_VAR(slave); MYSQL_ROW row; @@ -6181,10 +6189,8 @@ static int do_show_slave_status(MYSQL *mysql_con, int use_gtid, mysql_free_result(slave); return 1; } - print_comment(md_result_file, 0, - "-- GTID position to start replication:\n"); - fprintf(md_result_file, "%sSET GLOBAL gtid_slave_pos='%s';\n", - gtid_comment_prefix, gtid_pos); + /* similarly to do_show_master_status */ + sprintf(set_gtid_pos, fmt_gtid_pos, gtid_comment_prefix, gtid_pos); } if (use_gtid) print_comment(md_result_file, 0, @@ -6905,6 +6911,19 @@ static void dynstr_realloc_checked(DYNAMIC_STRING *str, ulong additional_size) die(EX_MYSQLERR, DYNAMIC_STR_ERROR_MSG); } +/** + Print earlier prepared SET to @@global.gtid_slave_pos. + + @param set_gtid_pos[in] formatted sql set statement +**/ +static void do_print_set_gtid_slave_pos(const char *set_gtid_pos) +{ + DBUG_ASSERT(opt_master_data || opt_slave_data); + if (opt_slave_data) + print_comment(md_result_file, 0, + "-- GTID position to start replication:\n"); + fprintf(md_result_file, "%s", set_gtid_pos); +} int main(int argc, char **argv) { @@ -6913,6 +6932,11 @@ int main(int argc, char **argv) int exit_code; int consistent_binlog_pos= 0; int have_mariadb_gtid= 0; + /* + to hold SET @@global.gtid_slave_pos which is deferred to print + until the function epilogue. + */ + char set_gtid_pos[3 + sizeof(fmt_gtid_pos) + MAX_GTID_LENGTH]= {0}; MY_INIT(argv[0]); sf_leaking_memory=1; /* don't report memory leaks on early exits */ @@ -7016,10 +7040,12 @@ int main(int argc, char **argv) goto err; if (opt_master_data && do_show_master_status(mysql, consistent_binlog_pos, - have_mariadb_gtid, opt_use_gtid)) + have_mariadb_gtid, + opt_use_gtid, set_gtid_pos)) goto err; - if (opt_slave_data && do_show_slave_status(mysql, opt_use_gtid, - have_mariadb_gtid)) + if (opt_slave_data && do_show_slave_status(mysql, + have_mariadb_gtid, + opt_use_gtid, set_gtid_pos)) goto err; if (opt_single_transaction && do_unlock_tables(mysql)) /* unlock but no commit! */ goto err; @@ -7087,6 +7113,9 @@ int main(int argc, char **argv) if (opt_system & OPT_SYSTEM_TIMEZONES) dump_all_timezones(); + if ((opt_master_data || opt_slave_data) && set_gtid_pos[0]) + do_print_set_gtid_slave_pos(set_gtid_pos); + /* add 'START SLAVE' to end of dump */ if (opt_slave_apply && add_slave_statements()) goto err; -- 2.39.2

1 0

[PATCH] MDEV-34504 PURGE BINARY LOGS not working anymore
by Kristian Nielsen 09 Jul '24

09 Jul '24

From: Monty <monty(a)mariadb.org> PURGE BINARY LOGS did not always purge binary logs. This commit fixes some of the issues and adds notifications if a binary log cannot be purged. User visible changes: - 'PURGE BINARY LOG TO log_name' and 'PURGE BINARY LOGS BEFORE date' worked differently. 'TO' ignored 'slave_connections_needed_for_purge' while 'BEFORE' did not. Now both versions ignores the 'slave_connections_needed_for_purge variable'. - 'PURGE BINARY LOG..' commands now returns 'note' if a binary log cannot be deleted like Note 1375 Binary log 'master-bin.000004' is not purged because it is the current active binlog - Automatic binary log purges, based on date or size, will write a note to the error log if a binary log matching the size or date cannot yet be deleted. - If 'slave_connections_needed_for_purge' is set from a config or command line, it is set to 0 if Galera is enabled and 1 otherwise (old default). This ensures that automatic binary log purge works with Galera as before the addition of 'slave_connections_needed_for_purge'. If the variable is changed to 0, a warning will be printed to the error log. Code changes: - Added THD argument to several purge_logs related functions that needed THD. - Added 'interactive' options to purge_logs functions. This allowed me to remove testing of sql_command == SQLCOM_PURGE. - Changed purge_logs_before_date() to first check if log is applicable before calling can_purge_logs(). This ensures we do not get a notification for logs that does not match the remove criteria. - MYSQL_BIN_LOG::can_purge_log() will write notifications to the user or error log if a log cannot yet be removed. - log_in_use() will return reason why a binary log cannot be removed. - Moved checking of binlog_format for Galera to be after Galera is initialized (The old check never worked). If Galera is enabled we now change the binlog_format to ROW, with a warning, instead of aborting the server. If this change happens, the binlog_format variable will be marked with AUTO or FORCED, for information_schema.system_variables, and a warning will be printed to the error log. - Print also a warning if FLASHBACK changes the binlog_format to ROW. Before this was done silently. --- mysql-test/main/mysqld--help.result | 1 + .../binlog_flush_binlogs_delete_domain.result | 3 + mysql-test/suite/binlog/r/binlog_index.result | 1 + .../suite/binlog/r/binlog_xa_recover.result | 2 + .../t/binlog_flush_binlogs_delete_domain.test | 2 + .../binlog_encryption/binlog_index.result | 1 + .../binlog_xa_recover.result | 2 + .../oracle/r/binlog_ptr_mysqlbinlog.result | 1 + .../oracle/t/binlog_ptr_mysqlbinlog.test | 2 + mysql-test/suite/galera/r/basic.result | 6 ++ mysql-test/suite/galera/t/basic.test | 3 + mysql-test/suite/rpl/r/purge_binlog.result | 52 ++++++++++ mysql-test/suite/rpl/r/rpl_rotate_logs.result | 2 + mysql-test/suite/rpl/t/purge_binlog.test | 54 +++++++++++ .../r/sysvars_server_notembedded.result | 2 +- sql/log.cc | 95 ++++++++++++++----- sql/log.h | 8 +- sql/mysqld.cc | 48 +++++++--- sql/mysqld.h | 1 + sql/set_var.cc | 3 +- sql/set_var.h | 3 +- sql/sql_repl.cc | 26 +++-- sql/sql_repl.h | 2 +- sql/sys_vars.cc | 21 +++- sql/sys_vars.h | 17 ++++ 25 files changed, 303 insertions(+), 55 deletions(-) create mode 100644 mysql-test/suite/rpl/r/purge_binlog.result create mode 100644 mysql-test/suite/rpl/t/purge_binlog.test create mode 100644 sql/sys_vars.h diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result index ee75f586970..026f135ebd5 100644 --- a/mysql-test/main/mysqld--help.result +++ b/mysql-test/main/mysqld--help.result @@ -1308,6 +1308,7 @@ The following specify which files/extra groups are read (specified before remain Minimum number of connected slaves required for automatic binary log purge with max_binlog_total_size, binlog_expire_logs_seconds or binlog_expire_logs_days. + Default is 0 when Galera is enabled and 1 otherwise. --slave-ddl-exec-mode=name How replication events should be executed. Legal values are STRICT and IDEMPOTENT (default). In IDEMPOTENT mode, diff --git a/mysql-test/suite/binlog/r/binlog_flush_binlogs_delete_domain.result b/mysql-test/suite/binlog/r/binlog_flush_binlogs_delete_domain.result index 1c11191802f..f967a367898 100644 --- a/mysql-test/suite/binlog/r/binlog_flush_binlogs_delete_domain.result +++ b/mysql-test/suite/binlog/r/binlog_flush_binlogs_delete_domain.result @@ -49,6 +49,9 @@ ERROR HY000: Could not delete gtid domain. Reason: binlog files may contain gtid MDEV-31140: Missing error from DELETE_DOMAIN_ID when gtid_binlog_state partially matches GTID_LIST. FLUSH BINARY LOGS; PURGE BINARY LOGS TO 'master-bin.000005'; +SHOW BINARY LOGS; +Log_name File_size +master-bin.000005 # SET @@SESSION.gtid_domain_id=8; SET @@SESSION.server_id=10*8 + 1; INSERT INTO t SELECT 1+MAX(a) FROM t; diff --git a/mysql-test/suite/binlog/r/binlog_index.result b/mysql-test/suite/binlog/r/binlog_index.result index 9dfda71f9a7..2d2363a7fec 100644 --- a/mysql-test/suite/binlog/r/binlog_index.result +++ b/mysql-test/suite/binlog/r/binlog_index.result @@ -30,6 +30,7 @@ flush logs; flush logs; *** must be a warning master-bin.000001 was not found *** Warnings: +Note 1375 Binary log 'master-bin.000004' is not purged because it is the current active binlog Warning 1612 Being purged log master-bin.000001 was not found *** must show one record, of the active binlog, left in the index file after PURGE *** show binary logs; diff --git a/mysql-test/suite/binlog/r/binlog_xa_recover.result b/mysql-test/suite/binlog/r/binlog_xa_recover.result index f5060fd5160..da8dff2dfd1 100644 --- a/mysql-test/suite/binlog/r/binlog_xa_recover.result +++ b/mysql-test/suite/binlog/r/binlog_xa_recover.result @@ -89,6 +89,8 @@ master-bin.000006 # Format_desc # # SERVER_VERSION, BINLOG_VERSION master-bin.000006 # Gtid_list # # [#-#-#] master-bin.000006 # Binlog_checkpoint # # master-bin.000004 PURGE BINARY LOGS TO "master-bin.000006"; +Warnings: +Note 1375 Binary log 'master-bin.000004' is not purged because it is in use by an active XID transaction show binary logs; Log_name File_size master-bin.000004 # diff --git a/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test b/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test index 1643ecff72d..4a974c642c0 100644 --- a/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test +++ b/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test @@ -91,6 +91,8 @@ while ($domain_cnt) FLUSH BINARY LOGS; --let $purge_to_binlog= query_get_value(SHOW MASTER STATUS, File, 1) --eval PURGE BINARY LOGS TO '$purge_to_binlog' +--replace_column 2 # +SHOW BINARY LOGS; --eval SET @@SESSION.gtid_domain_id=$err_domain_id --eval SET @@SESSION.server_id=10*$err_domain_id + $err_server_id eval INSERT INTO t SELECT 1+MAX(a) FROM t; diff --git a/mysql-test/suite/binlog_encryption/binlog_index.result b/mysql-test/suite/binlog_encryption/binlog_index.result index 9dfda71f9a7..2d2363a7fec 100644 --- a/mysql-test/suite/binlog_encryption/binlog_index.result +++ b/mysql-test/suite/binlog_encryption/binlog_index.result @@ -30,6 +30,7 @@ flush logs; flush logs; *** must be a warning master-bin.000001 was not found *** Warnings: +Note 1375 Binary log 'master-bin.000004' is not purged because it is the current active binlog Warning 1612 Being purged log master-bin.000001 was not found *** must show one record, of the active binlog, left in the index file after PURGE *** show binary logs; diff --git a/mysql-test/suite/binlog_encryption/binlog_xa_recover.result b/mysql-test/suite/binlog_encryption/binlog_xa_recover.result index 3e4ed42cf7c..7aacd0e034a 100644 --- a/mysql-test/suite/binlog_encryption/binlog_xa_recover.result +++ b/mysql-test/suite/binlog_encryption/binlog_xa_recover.result @@ -93,6 +93,8 @@ master-bin.000006 # Start_encryption # # master-bin.000006 # Gtid_list # # [#-#-#] master-bin.000006 # Binlog_checkpoint # # master-bin.000004 PURGE BINARY LOGS TO "master-bin.000006"; +Warnings: +Note 1375 Binary log 'master-bin.000004' is not purged because it is in use by an active XID transaction show binary logs; Log_name File_size master-bin.000004 # diff --git a/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result b/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result index 0656a685976..543ec5336d8 100644 --- a/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result +++ b/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result @@ -1,3 +1,4 @@ +call mtr.add_suppression("Binlog_format changed to.*flashback"); SET @@SQL_MODE = 'ORACLE'; ########################################################################## # Test verifies Gtid_log_event/Xid_log_event specific print # diff --git a/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test index bda32af5d4e..165f2cc0afe 100644 --- a/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test +++ b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test @@ -18,6 +18,8 @@ --source include/have_log_bin.inc --source include/have_innodb.inc +call mtr.add_suppression("Binlog_format changed to.*flashback"); + let $MYSQLD_DATADIR= `select @@datadir`; SET @@SQL_MODE = 'ORACLE'; diff --git a/mysql-test/suite/galera/r/basic.result b/mysql-test/suite/galera/r/basic.result index 10f180e7a94..7b4cbd93c7b 100644 --- a/mysql-test/suite/galera/r/basic.result +++ b/mysql-test/suite/galera/r/basic.result @@ -1,5 +1,11 @@ connection node_2; connection node_1; +select @@slave_connections_needed_for_purge; +@@slave_connections_needed_for_purge +0 +select VARIABLE_NAME, GLOBAL_VALUE, GLOBAL_VALUE_ORIGIN from information_schema.system_variables where variable_name="slave_connections_needed_for_purge"; +VARIABLE_NAME GLOBAL_VALUE GLOBAL_VALUE_ORIGIN +SLAVE_CONNECTIONS_NEEDED_FOR_PURGE 0 AUTO USE test; CREATE TABLE t1(c1 INT PRIMARY KEY) ENGINE=INNODB; INSERT INTO t1 VALUES (1), (2), (3), (4), (5); diff --git a/mysql-test/suite/galera/t/basic.test b/mysql-test/suite/galera/t/basic.test index 8fc6eee3b3b..a70ee962bc0 100644 --- a/mysql-test/suite/galera/t/basic.test +++ b/mysql-test/suite/galera/t/basic.test @@ -1,6 +1,9 @@ --source include/galera_cluster.inc --source include/have_innodb.inc +select @@slave_connections_needed_for_purge; +select VARIABLE_NAME, GLOBAL_VALUE, GLOBAL_VALUE_ORIGIN from information_schema.system_variables where variable_name="slave_connections_needed_for_purge"; + USE test; CREATE TABLE t1(c1 INT PRIMARY KEY) ENGINE=INNODB; INSERT INTO t1 VALUES (1), (2), (3), (4), (5); diff --git a/mysql-test/suite/rpl/r/purge_binlog.result b/mysql-test/suite/rpl/r/purge_binlog.result new file mode 100644 index 00000000000..6be060e5774 --- /dev/null +++ b/mysql-test/suite/rpl/r/purge_binlog.result @@ -0,0 +1,52 @@ +include/master-slave.inc +[connection master] +# +# MDEV-34504 PURGE BINARY LOGS not working anymore +# +select @@slave_connections_needed_for_purge; +@@slave_connections_needed_for_purge +0 +set @old_dbug= @@global.debug_dbug; +create table t1 (a int, b varchar(32768)); +insert into t1 values(1,repeat("a",32768)); +connection slave; +select a from t1; +a +1 +set @@global.debug_dbug= "+d,pause_before_io_read_event"; +connection master; +insert into t1 values(2,repeat("b",32768)); +insert into t1 values(3,repeat("c",32768)); +connection slave; +set debug_sync='now wait_for io_thread_at_read_event'; +select a from t1; +a +1 +connection master; +FLUSH BINARY LOGS; +SHOW BINARY LOGS; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +PURGE BINARY LOGS TO 'master-bin.000002'; +Warnings: +Note 1375 Binary log XXX is not purged because it is in use by a slave thread +SHOW BINARY LOGS; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +connection slave; +set @@global.debug_dbug= @old_dbug; +set debug_sync='now signal io_thread_continue_read_event'; +connection master; +connection slave; +select count(*) from t1; +count(*) +103 +connection master; +PURGE BINARY LOGS TO 'master-bin.000002'; +SHOW BINARY LOGS; +Log_name File_size +master-bin.000002 # +drop table t1; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_rotate_logs.result b/mysql-test/suite/rpl/r/rpl_rotate_logs.result index 4b7c1642a11..8e8f0026c93 100644 --- a/mysql-test/suite/rpl/r/rpl_rotate_logs.result +++ b/mysql-test/suite/rpl/r/rpl_rotate_logs.result @@ -70,6 +70,8 @@ master-bin.000002 # master-bin.000003 # SELECT @time_for_purge:=DATE_ADD('tmpval', INTERVAL 1 SECOND); purge master logs before (@time_for_purge); +Warnings: +Note 1375 Binary log 'master-bin.000003' is not purged because it is the current active binlog show binary logs; Log_name File_size master-bin.000003 # diff --git a/mysql-test/suite/rpl/t/purge_binlog.test b/mysql-test/suite/rpl/t/purge_binlog.test new file mode 100644 index 00000000000..5c804ca22bb --- /dev/null +++ b/mysql-test/suite/rpl/t/purge_binlog.test @@ -0,0 +1,54 @@ +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/master-slave.inc +--source include/have_binlog_format_row.inc + +--echo # +--echo # MDEV-34504 PURGE BINARY LOGS not working anymore +--echo # + +select @@slave_connections_needed_for_purge; +set @old_dbug= @@global.debug_dbug; + +create table t1 (a int, b varchar(32768)); +insert into t1 values(1,repeat("a",32768)); +--sync_slave_with_master +select a from t1; +set @@global.debug_dbug= "+d,pause_before_io_read_event"; +--connection master +insert into t1 values(2,repeat("b",32768)); +insert into t1 values(3,repeat("c",32768)); +--connection slave +set debug_sync='now wait_for io_thread_at_read_event'; +select a from t1; +--connection master +--disable_query_log +let $i=100; +while ($i) +{ +--eval insert into t1 values($i+4,repeat(char(64+$i),32768)); +--dec $i +} +--enable_query_log + +FLUSH BINARY LOGS; +--replace_column 2 # +SHOW BINARY LOGS; +--let $purge_to_binlog= query_get_value(SHOW MASTER STATUS, File, 1) +--replace_regex /Binary log.*is not/Binary log XXX is not/ +--eval PURGE BINARY LOGS TO '$purge_to_binlog' +--replace_column 2 # +SHOW BINARY LOGS; +--connection slave +set @@global.debug_dbug= @old_dbug; +set debug_sync='now signal io_thread_continue_read_event'; +--connection master +--sync_slave_with_master +select count(*) from t1; +--connection master +--eval PURGE BINARY LOGS TO '$purge_to_binlog' +--replace_column 2 # +SHOW BINARY LOGS; +drop table t1; + +--source include/rpl_end.inc diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result index 2740319dd05..0b2feb675b4 100644 --- a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result +++ b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result @@ -3995,7 +3995,7 @@ COMMAND_LINE_ARGUMENT OPTIONAL VARIABLE_NAME SLAVE_CONNECTIONS_NEEDED_FOR_PURGE VARIABLE_SCOPE GLOBAL VARIABLE_TYPE INT UNSIGNED -VARIABLE_COMMENT Minimum number of connected slaves required for automatic binary log purge with max_binlog_total_size, binlog_expire_logs_seconds or binlog_expire_logs_days. +VARIABLE_COMMENT Minimum number of connected slaves required for automatic binary log purge with max_binlog_total_size, binlog_expire_logs_seconds or binlog_expire_logs_days. Default is 0 when Galera is enabled and 1 otherwise. NUMERIC_MIN_VALUE 0 NUMERIC_MAX_VALUE 4294967295 NUMERIC_BLOCK_SIZE 1 diff --git a/sql/log.cc b/sql/log.cc index c27c4f3353b..1384fb0b3e7 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -4791,8 +4791,8 @@ int MYSQL_BIN_LOG::purge_first_log(Relay_log_info* rli, bool included) DBUG_EXECUTE_IF("crash_before_purge_logs", DBUG_SUICIDE();); - rli->relay_log.purge_logs(to_purge_if_included, included, - 0, 0, &log_space_reclaimed); + rli->relay_log.purge_logs(current_thd, to_purge_if_included, included, + 0, 0, 0, &log_space_reclaimed); mysql_mutex_lock(&rli->log_space_lock); rli->log_space_total-= log_space_reclaimed; @@ -4859,16 +4859,17 @@ int MYSQL_BIN_LOG::update_log_index(LOG_INFO* log_info, bool need_update_threads mysql_file_stat() or mysql_file_delete() */ -int MYSQL_BIN_LOG::purge_logs(const char *to_log, +int MYSQL_BIN_LOG::purge_logs(THD *thd, + const char *to_log, bool included, bool need_mutex, - bool need_update_threads, + bool need_update_threads, + bool interactive, ulonglong *reclaimed_space) { int error= 0; bool exit_loop= 0; LOG_INFO log_info; - THD *thd= current_thd; DBUG_ENTER("purge_logs"); DBUG_PRINT("info",("to_log= %s",to_log)); @@ -4894,7 +4895,7 @@ int MYSQL_BIN_LOG::purge_logs(const char *to_log, if (unlikely((error=find_log_pos(&log_info, NullS, 0 /*no mutex*/)))) goto err; while ((strcmp(to_log,log_info.log_file_name) || (exit_loop=included)) && - can_purge_log(log_info.log_file_name)) + can_purge_log(thd, log_info.log_file_name, interactive)) { if (unlikely((error= register_purge_index_entry(log_info.log_file_name)))) { @@ -4902,7 +4903,6 @@ int MYSQL_BIN_LOG::purge_logs(const char *to_log, log_info.log_file_name); goto err; } - if (find_next_log(&log_info, 0) || exit_loop) break; } @@ -5243,13 +5243,13 @@ int MYSQL_BIN_LOG::purge_index_entry(THD *thd, ulonglong *reclaimed_space, mysql_file_stat() or mysql_file_delete() */ -int MYSQL_BIN_LOG::purge_logs_before_date(time_t purge_time) +int MYSQL_BIN_LOG::purge_logs_before_date(THD *thd, time_t purge_time, + bool interactive) { int error; char to_log[FN_REFLEN]; LOG_INFO log_info; MY_STAT stat_area; - THD *thd= current_thd; DBUG_ENTER("purge_logs_before_date"); mysql_mutex_lock(&LOCK_index); @@ -5258,7 +5258,7 @@ int MYSQL_BIN_LOG::purge_logs_before_date(time_t purge_time) if (unlikely((error=find_log_pos(&log_info, NullS, 0 /*no mutex*/)))) goto err; - while (can_purge_log(log_info.log_file_name)) + for (;;) { if (!mysql_file_stat(m_key_file_log, log_info.log_file_name, &stat_area, MYF(0))) @@ -5296,7 +5296,8 @@ int MYSQL_BIN_LOG::purge_logs_before_date(time_t purge_time) } else { - if (stat_area.st_mtime >= purge_time) + if (stat_area.st_mtime >= purge_time || + !can_purge_log(thd, log_info.log_file_name, interactive)) break; strmake_buf(to_log, log_info.log_file_name); } @@ -5307,7 +5308,7 @@ int MYSQL_BIN_LOG::purge_logs_before_date(time_t purge_time) if (to_log[0]) { ulonglong reclaimed_space= 0; - error= purge_logs(to_log, 1, 0, 1, &reclaimed_space); + error= purge_logs(thd, to_log, 1, 0, 1, interactive, &reclaimed_space); binlog_space_total-= reclaimed_space; } @@ -5338,6 +5339,7 @@ int MYSQL_BIN_LOG::real_purge_logs_by_size(ulonglong binlog_pos) MY_STAT stat_area; char to_log[FN_REFLEN]; ulonglong found_space= 0; + THD *thd= current_thd; DBUG_ENTER("real_purge_logs_by_size"); mysql_mutex_lock(&LOCK_index); @@ -5351,7 +5353,7 @@ int MYSQL_BIN_LOG::real_purge_logs_by_size(ulonglong binlog_pos) goto err; to_log[0] = 0; - while (can_purge_log(log_info.log_file_name)) + while (can_purge_log(thd, log_info.log_file_name, 0)) { if (!mysql_file_stat(m_key_file_log, log_info.log_file_name, &stat_area, MYF(0))) @@ -5394,8 +5396,9 @@ int MYSQL_BIN_LOG::real_purge_logs_by_size(ulonglong binlog_pos) if (found_space) { ulonglong reclaimed_space= 0; - purge_logs(to_log, true, false /*need_lock_index=false*/, + purge_logs(thd, to_log, true, false /*need_lock_index=false*/, true /*need_update_threads=true*/, + false /* not interactive */, &reclaimed_space); DBUG_ASSERT(reclaimed_space == found_space); binlog_space_total-= reclaimed_space; @@ -5414,6 +5417,10 @@ int MYSQL_BIN_LOG::real_purge_logs_by_size(ulonglong binlog_pos) } /* + @param THD thd, may be null at startup + @param log_file_name_arg Name of log file to check + @param interactive True if called by a PURGE BINLOG command. + The following variables are here to allows us to quickly check if the can_purge_log(log_file_name_arg) name will fail in the 'log_in_use' call. @@ -5428,18 +5435,23 @@ int MYSQL_BIN_LOG::real_purge_logs_by_size(ulonglong binlog_pos) static bool waiting_for_slave_to_change_binlog= 0; static ulonglong purge_sending_new_binlog_file= 0; static char purge_binlog_name[FN_REFLEN]; +static bool purge_warning_given= 0; bool -MYSQL_BIN_LOG::can_purge_log(const char *log_file_name_arg) +MYSQL_BIN_LOG::can_purge_log(THD *thd, const char *log_file_name_arg, + bool interactive) { - THD *thd= current_thd; // May be NULL at startup - bool res; + int res; + const char *reason; if (is_active(log_file_name_arg) || (!is_relay_log && waiting_for_slave_to_change_binlog && purge_sending_new_binlog_file == sending_new_binlog_file && !strcmp(log_file_name_arg, purge_binlog_name))) - return false; + { + reason= "it is the current active binlog"; + goto error; + } DBUG_ASSERT(!is_relay_log || binlog_xid_count_list.is_empty()); if (!is_relay_log) @@ -5455,7 +5467,10 @@ MYSQL_BIN_LOG::can_purge_log(const char *log_file_name_arg) } mysql_mutex_unlock(&LOCK_xid_list); if (b) - return false; + { + reason= "it is in use by an active XID transaction"; + goto error; + } } if (!is_relay_log) @@ -5464,8 +5479,7 @@ MYSQL_BIN_LOG::can_purge_log(const char *log_file_name_arg) purge_sending_new_binlog_file= sending_new_binlog_file; } if ((res= log_in_use(log_file_name_arg, - (is_relay_log || - (thd && thd->lex->sql_command == SQLCOM_PURGE)) ? + (is_relay_log || interactive) ? 0 : slave_connections_needed_for_purge))) { if (!is_relay_log) @@ -5473,9 +5487,39 @@ MYSQL_BIN_LOG::can_purge_log(const char *log_file_name_arg) waiting_for_slave_to_change_binlog= 1; strmake(purge_binlog_name, log_file_name_arg, sizeof(purge_binlog_name)-1); + if (res == 1) + reason= "it is in use by a slave thread"; + else + reason= "less than 'slave_connections_needed_for_purge' slaves has " + "processed it"; + goto error; } } - return !res; + /* We can purge this file, reset for next failure */ + purge_warning_given= 0; + return 1; + +error: + if (!is_relay_log && (interactive || !purge_warning_given)) + { + /* Remove directory (to keep things shorter and compatible */ + log_file_name_arg+= dirname_length(log_file_name_arg); + + /* purge_warning_given is reset after next sucessful purge */ + purge_warning_given= 1; + if (interactive) + { + my_printf_error(ER_BINLOG_PURGE_PROHIBITED, + "Binary log '%s' is not purged because %s", + MYF(ME_NOTE), log_file_name_arg, reason); + } + else + { + sql_print_information("Binary log '%s' is not purged because %s", + log_file_name_arg, reason); + } + } + return 0; } #endif /* HAVE_REPLICATION */ @@ -7611,16 +7655,17 @@ void MYSQL_BIN_LOG::purge(bool all) { mysql_mutex_assert_not_owner(&LOCK_log); #ifdef HAVE_REPLICATION + THD *thd= current_thd; if (binlog_expire_logs_seconds) { - DEBUG_SYNC(current_thd, "at_purge_logs_before_date"); + DEBUG_SYNC(thd, "at_purge_logs_before_date"); time_t purge_time= my_time(0) - binlog_expire_logs_seconds; DBUG_EXECUTE_IF("expire_logs_always", { purge_time = my_time(0); }); if (purge_time >= 0) { - purge_logs_before_date(purge_time); + purge_logs_before_date(thd, purge_time, 0); } - DEBUG_SYNC(current_thd, "after_purge_logs_before_date"); + DEBUG_SYNC(thd, "after_purge_logs_before_date"); } if (all && binlog_space_limit) { diff --git a/sql/log.h b/sql/log.h index 23513c75ac5..c8a8eb72b37 100644 --- a/sql/log.h +++ b/sql/log.h @@ -1035,7 +1035,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log void mark_xid_done(ulong cookie, bool write_checkpoint); void make_log_name(char* buf, const char* log_ident); bool is_active(const char* log_file_name); - bool can_purge_log(const char *log_file_name); + bool can_purge_log(THD *thd, const char *log_file_name, bool interactive); int update_log_index(LOG_INFO* linfo, bool need_update_threads); int rotate(bool force_rotate, bool* check_purge); void checkpoint_and_purge(ulong binlog_id); @@ -1054,10 +1054,10 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log @retval other Failure */ bool flush_and_sync(bool *synced); - int purge_logs(const char *to_log, bool included, - bool need_mutex, bool need_update_threads, + int purge_logs(THD *thd, const char *to_log, bool included, + bool need_mutex, bool need_update_threads, bool interactive, ulonglong *decrease_log_space); - int purge_logs_before_date(time_t purge_time); + int purge_logs_before_date(THD *thd, time_t purge_time, bool interactive); int purge_first_log(Relay_log_info* rli, bool included); int count_binlog_space(); void count_binlog_space_with_mutex() diff --git a/sql/mysqld.cc b/sql/mysqld.cc index 2b12001de9e..604eaadcc92 100644 --- a/sql/mysqld.cc +++ b/sql/mysqld.cc @@ -52,6 +52,7 @@ #include "sql_manager.h" // stop_handle_manager, start_handle_manager #include "sql_expression_cache.h" // subquery_cache_miss, subquery_cache_hit #include "sys_vars_shared.h" +#include "sys_vars.h" #include "ddl_log.h" #include "optimizer_defaults.h" @@ -519,6 +520,8 @@ uint default_password_lifetime; my_bool disconnect_on_expired_password; bool max_user_connections_checking=0; +bool slave_connections_needed_for_purge_option_used= 0; + /** Limit of the total number of prepared statements in the server. Is necessary to protect the server against out-of-memory attacks. @@ -5847,7 +5850,28 @@ int mysqld_main(int argc, char **argv) #ifdef WITH_WSREP wsrep_set_wsrep_on(nullptr); if (WSREP_ON && wsrep_check_opts()) unireg_abort(1); -#endif + + if (!opt_bootstrap && WSREP_PROVIDER_EXISTS && WSREP_ON) + { + if (global_system_variables.binlog_format != BINLOG_FORMAT_ROW) + { + sql_print_warning("Binlog_format changed to \"ROW\" because of Galera"); + global_system_variables.binlog_format= BINLOG_FORMAT_ROW; + mark_binlog_format_used(binlog_format_used); + } + binlog_format_used= 1; + if (!slave_connections_needed_for_purge_option_used) + { + slave_connections_needed_for_purge= + internal_slave_connections_needed_for_purge= 0; + mark_slave_connections_needed_for_purge_as_auto(); + sql_print_information( + "slave_connections_needed_for_purge changed to 0 because " + "of Galera. Change it to 1 or higher if this Galera node " + "is also Master in a normal replication setup"); + } + } +#endif /* WITH_WSREP */ #ifdef _WIN32 /* @@ -8219,7 +8243,9 @@ mysqld_get_one_option(const struct my_option *opt, const char *argument, ((enum_slave_parallel_mode)opt_slave_parallel_mode); break; } - + case (int) OPT_SLAVE_CONNECTIONS_NEEDED_FOR_PURGE: + slave_connections_needed_for_purge_option_used= 1; + break; case (int)OPT_BINLOG_IGNORE_DB: { binlog_filter->add_ignore_db(argument); @@ -8714,18 +8740,14 @@ static int get_options(int *argc_ptr, char ***argv_ptr) opt_bin_log= opt_bin_log_used= 1; /* Force format to row */ + if (global_system_variables.binlog_format != BINLOG_FORMAT_ROW) + { + sql_print_warning("Binlog_format changed to \"ROW\" because of " + "flashback"); + global_system_variables.binlog_format= BINLOG_FORMAT_ROW; + mark_binlog_format_used(binlog_format_used); + } binlog_format_used= 1; - global_system_variables.binlog_format= BINLOG_FORMAT_ROW; - } - - if (!opt_bootstrap && WSREP_PROVIDER_EXISTS && WSREP_ON && - global_system_variables.binlog_format != BINLOG_FORMAT_ROW) - { - - WSREP_ERROR ("Only binlog_format = 'ROW' is currently supported. " - "Configured value: '%s'. Please adjust your configuration.", - binlog_format_names[global_system_variables.binlog_format]); - return 1; } // Synchronize @@global.autocommit on --autocommit diff --git a/sql/mysqld.h b/sql/mysqld.h index 33492ad7286..61313079f3f 100644 --- a/sql/mysqld.h +++ b/sql/mysqld.h @@ -852,6 +852,7 @@ enum options_mysqld OPT_SILENT, OPT_SKIP_HOST_CACHE, OPT_SLAVE_PARALLEL_MODE, + OPT_SLAVE_CONNECTIONS_NEEDED_FOR_PURGE, OPT_SSL_CA, OPT_SSL_CAPATH, OPT_SSL_CERT, diff --git a/sql/set_var.cc b/sql/set_var.cc index 61528b45de6..de537174825 100644 --- a/sql/set_var.cc +++ b/sql/set_var.cc @@ -1119,7 +1119,8 @@ int fill_sysvars(THD *thd, TABLE_LIST *tables, COND *cond) { STRING_WITH_LEN("AUTO") }, { STRING_WITH_LEN("SQL") }, { STRING_WITH_LEN("COMPILE-TIME") }, - { STRING_WITH_LEN("ENVIRONMENT") } + { STRING_WITH_LEN("ENVIRONMENT") }, + { STRING_WITH_LEN("FORCED") }, }; const LEX_CSTRING *origin= origins + var->value_origin; fields[3]->store(origin->str, origin->length, scs); diff --git a/sql/set_var.h b/sql/set_var.h index aed4955ef62..94ddef6a127 100644 --- a/sql/set_var.h +++ b/sql/set_var.h @@ -66,7 +66,8 @@ class sys_var: protected Value_source // for double_from_string_with_check READONLY=1024, ALLOCATED=2048, PARSE_EARLY=4096, NO_SET_STATEMENT=8192, AUTO_SET=16384}; enum { NO_GETOPT=-1, GETOPT_ONLY_HELP=-2 }; - enum where { CONFIG, COMMAND_LINE, AUTO, SQL, COMPILE_TIME, ENV }; + /* If where is changed, change also GLOBAL_VALUE_ORIGIN in set_var.h */ + enum where { CONFIG, COMMAND_LINE, AUTO, SQL, COMPILE_TIME, ENV, FORCED }; /** Enumeration type to indicate for a system variable whether diff --git a/sql/sql_repl.cc b/sql/sql_repl.cc index 895ff090da6..22bc1befa6e 100644 --- a/sql/sql_repl.cc +++ b/sql/sql_repl.cc @@ -612,14 +612,26 @@ static my_bool log_in_use_callback(THD *thd, st_log_in_use *arg) } -bool log_in_use(const char* log_name, uint min_connected) +/* + Check if a log is in use. + + @return 0 Not used + @return 1 A slave is reading from the log + @return 2 There are less than 'min_connected' slaves that + has recived the log. +*/ + +int log_in_use(const char* log_name, uint min_connected) { st_log_in_use arg; arg.log_name= log_name; arg.connected_slaves= 0; - return ((server_threads.iterate(log_in_use_callback, &arg) || - arg.connected_slaves < min_connected)); + if (server_threads.iterate(log_in_use_callback, &arg)) + return 1; + if (arg.connected_slaves < min_connected) + return 2; + return 0; } @@ -659,8 +671,8 @@ bool purge_master_logs(THD* thd, const char* to_log) mysql_bin_log.make_log_name(search_file_name, to_log); return purge_error_message(thd, - mysql_bin_log.purge_logs(search_file_name, 0, 1, - 1, NULL)); + mysql_bin_log.purge_logs(thd, search_file_name, + 0, 1, 1, 1, NULL)); } @@ -683,7 +695,9 @@ bool purge_master_logs_before_date(THD* thd, time_t purge_time) return 0; } return purge_error_message(thd, - mysql_bin_log.purge_logs_before_date(purge_time)); + mysql_bin_log.purge_logs_before_date(thd, + purge_time, + 1)); } void set_read_error(binlog_send_info *info, int error) diff --git a/sql/sql_repl.h b/sql/sql_repl.h index 51b6a599d5f..c03384aa5a3 100644 --- a/sql/sql_repl.h +++ b/sql/sql_repl.h @@ -38,7 +38,7 @@ int reset_master(THD* thd, rpl_gtid *init_state, uint32 init_state_len, ulong next_log_number); bool purge_master_logs(THD* thd, const char* to_log); bool purge_master_logs_before_date(THD* thd, time_t purge_time); -bool log_in_use(const char* log_name, uint min_connections); +int log_in_use(const char* log_name, uint min_connections); void adjust_linfo_offsets(my_off_t purge_offset); void show_binlogs_get_fields(THD *thd, List<Item> *field_list); bool show_binlogs(THD* thd); diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc index a2705990736..35a29c29ef8 100644 --- a/sql/sys_vars.cc +++ b/sql/sys_vars.cc @@ -35,6 +35,7 @@ #include "sql_priv.h" #include "sql_class.h" // set_var.h: THD #include "sys_vars.inl" +#include "sys_vars.h" #include "my_sys.h" #include "events.h" @@ -704,6 +705,13 @@ Sys_binlog_format( NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(binlog_format_check), ON_UPDATE(fix_binlog_format_after_update)); + +void mark_binlog_format_used(bool forced) +{ + Sys_binlog_format.value_origin= forced ? sys_var::FORCED : sys_var::AUTO; +} + + static bool binlog_direct_check(sys_var *self, THD *thd, set_var *var) { if (var->type == OPT_GLOBAL) @@ -1265,7 +1273,7 @@ static bool update_binlog_space_limit(sys_var *, THD *, mysql_bin_log.count_binlog_space(); /* Inform can_purge_log() that it should do a recheck of log_in_use() */ sending_new_binlog_file++; - mysql_bin_log.unlock_index(); + mysql_bin_log.unlock_index(); mysql_bin_log.purge(1); return 0; } @@ -1274,6 +1282,7 @@ static bool update_binlog_space_limit(sys_var *, THD *, return 0; } + static Sys_var_on_access_global<Sys_var_ulonglong, PRIV_SET_SYSTEM_GLOBAL_VAR_MAX_BINLOG_CACHE_SIZE> Sys_max_binlog_total_size( @@ -1303,13 +1312,19 @@ Sys_slave_connections_needed_for_purge( "slave_connections_needed_for_purge", "Minimum number of connected slaves required for automatic binary " "log purge with max_binlog_total_size, binlog_expire_logs_seconds " - "or binlog_expire_logs_days.", + "or binlog_expire_logs_days. Default is 0 when Galera is enabled and 1 " + "otherwise.", GLOBAL_VAR(internal_slave_connections_needed_for_purge), - CMD_LINE(REQUIRED_ARG), + CMD_LINE(REQUIRED_ARG, OPT_SLAVE_CONNECTIONS_NEEDED_FOR_PURGE), VALID_RANGE(0, UINT_MAX), DEFAULT(1), BLOCK_SIZE(1), NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0), ON_UPDATE(update_binlog_space_limit)); +void mark_slave_connections_needed_for_purge_as_auto() +{ + Sys_slave_connections_needed_for_purge.value_origin= sys_var::AUTO; +} + static Sys_var_mybool Sys_flush( "flush", "Flush MyISAM tables to disk between SQL commands", GLOBAL_VAR(myisam_flush), diff --git a/sql/sys_vars.h b/sql/sys_vars.h new file mode 100644 index 00000000000..8f4eac38cd0 --- /dev/null +++ b/sql/sys_vars.h @@ -0,0 +1,17 @@ +/* Copyright (c) 2024, MariaDB Corporation. + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; version 2 of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */ + +extern void mark_binlog_format_used(bool forced); +extern void mark_slave_connections_needed_for_purge_as_auto(); -- 2.30.2

1 0

[PATCH] MDEV-33487 Feature non-blocking binlog
by Kristian Nielsen 08 Jul '24

08 Jul '24

From: poempeng <poempeng(a)tencent.com> During the commit stage of a large transaction, it may cost too much time to write binlog cache and block all subsequent transactions for a long time. One possible solution of this issue is to write the binlog of the large transaction to a temporary file and then rename it to the next new binlog file. We need to reserve some space for format description event, gtid list event, binlog checkpoint event and gtid event. After writing the binlog cache, we lock LOCK_log and write the aforementioned events to the reserved space. The default size of reserved space is 10MB, any unusable space is filled with a skip event(Ignorable_log_event). If any error occurs before acquiring LOCK_log, we fallback to the original commit path. However, if an error occurs after acquiring LOCK_log, we abort the server. New variable: non_blocking_binlog_threshold, if the size of binlog cache exceeds this value, try using non-blocking binlog. Limitations: Binlog encryption, wsrep transaction, stmt cache, incident event and slave thread are not supported. --- mysql-test/main/mysqld--help.result | 5 + .../binlog/r/non_blocking_binlog_basic.result | 42 +++ .../binlog/r/non_blocking_binlog_error.result | 36 ++ .../binlog/t/non_blocking_binlog_basic.test | 31 ++ .../binlog/t/non_blocking_binlog_error.test | 49 +++ .../perfschema/r/dml_setup_instruments.result | 2 +- .../rpl/r/rpl_non_blocking_binlog.result | 22 ++ .../suite/rpl/t/rpl_non_blocking_binlog.test | 31 ++ .../r/sysvars_server_notembedded.result | 10 + sql/log.cc | 354 +++++++++++++++++- sql/log.h | 30 +- sql/mysqld.cc | 9 +- sql/mysqld.h | 4 +- sql/sys_vars.cc | 8 + 14 files changed, 615 insertions(+), 18 deletions(-) create mode 100644 mysql-test/suite/binlog/r/non_blocking_binlog_basic.result create mode 100644 mysql-test/suite/binlog/r/non_blocking_binlog_error.result create mode 100644 mysql-test/suite/binlog/t/non_blocking_binlog_basic.test create mode 100644 mysql-test/suite/binlog/t/non_blocking_binlog_error.test create mode 100644 mysql-test/suite/rpl/r/rpl_non_blocking_binlog.result create mode 100644 mysql-test/suite/rpl/t/rpl_non_blocking_binlog.test diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result index 431c37c25ed..d652eacadb0 100644 --- a/mysql-test/main/mysqld--help.result +++ b/mysql-test/main/mysqld--help.result @@ -735,6 +735,10 @@ The following specify which files/extra groups are read (specified before remain --net-write-timeout=# Number of seconds to wait for a block to be written to a connection before aborting the write + --non-blocking-binlog-threshold[=#] + If the binlog size of a transaction exceeds this value, + we write it to a new temporary file and rename it to the + next binlog file. --note-verbosity=name Verbosity level for note-warnings given to the user. See also @@sql_notes.. Any combination of: basic, @@ -1797,6 +1801,7 @@ net-buffer-length 16384 net-read-timeout 30 net-retry-count 10 net-write-timeout 60 +non-blocking-binlog-threshold 18446744073709551615 note-verbosity basic,explain old FALSE old-mode UTF8_IS_UTF8MB3 diff --git a/mysql-test/suite/binlog/r/non_blocking_binlog_basic.result b/mysql-test/suite/binlog/r/non_blocking_binlog_basic.result new file mode 100644 index 00000000000..dfbf6721f89 --- /dev/null +++ b/mysql-test/suite/binlog/r/non_blocking_binlog_basic.result @@ -0,0 +1,42 @@ +set global non_blocking_binlog_threshold=1; +Warnings: +Warning 1292 Truncated incorrect non_blocking_binlog_threshold value: '1' +show variables like 'non_blocking_binlog_threshold'; +Variable_name Value +non_blocking_binlog_threshold 268435456 +set global non_blocking_binlog_threshold=256*1024*1024; +show variables like 'non_blocking_binlog_threshold'; +Variable_name Value +non_blocking_binlog_threshold 268435456 +set global non_blocking_binlog_threshold=default; +show variables like 'non_blocking_binlog_threshold'; +Variable_name Value +non_blocking_binlog_threshold 18446744073709551615 +reset master; +create table t1(id int, c1 int) engine=innodb; +insert into t1 values(1,1); +set debug_dbug='+d,non_blocking_binlog_ignore_cache_size'; +update t1 set c1=2; +set debug_dbug='-d,non_blocking_binlog_ignore_cache_size'; +insert into t1 values(3,3); +show master status; +File Position Binlog_Do_DB Binlog_Ignore_DB +master-bin.000002 # +show binlog events in 'master-bin.000002'; +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000002 # Format_desc # # # +master-bin.000002 # Gtid_list # # # +master-bin.000002 # Binlog_checkpoint # # # +master-bin.000002 # Gtid # # # +master-bin.000002 # Ignorable log event # # # +master-bin.000002 # Annotate_rows # # # +master-bin.000002 # Table_map # # # +master-bin.000002 # Update_rows_v1 # # # +master-bin.000002 # Xid # # # +master-bin.000002 # Gtid # # # +master-bin.000002 # Annotate_rows # # # +master-bin.000002 # Table_map # # # +master-bin.000002 # Write_rows_v1 # # # +master-bin.000002 # Xid # # # +drop table t1; +include/assert_grep.inc [write non-blocking binlog succeed should be logged] diff --git a/mysql-test/suite/binlog/r/non_blocking_binlog_error.result b/mysql-test/suite/binlog/r/non_blocking_binlog_error.result new file mode 100644 index 00000000000..536c80e04b4 --- /dev/null +++ b/mysql-test/suite/binlog/r/non_blocking_binlog_error.result @@ -0,0 +1,36 @@ +call mtr.add_suppression(".*"); +reset master; +create table t1(id int, c1 int) engine=innodb; +insert into t1 values(1,1); +set debug_dbug='+d,non_blocking_binlog_ignore_cache_size'; +set debug_dbug='+d,non_blocking_binlog_open_file_failed'; +update t1 set c1=2; +set debug_dbug='-d,non_blocking_binlog_ignore_cache_size'; +set debug_dbug='-d,non_blocking_binlog_open_file_failed'; +insert into t1 values(3,3); +show master status; +File Position Binlog_Do_DB Binlog_Ignore_DB +master-bin.000001 # +include/assert_grep.inc [Open file failed should be logged] +set debug_dbug='+d,non_blocking_binlog_ignore_cache_size'; +set debug_dbug='+d,non_blocking_binlog_write_cache_failed'; +update t1 set c1=3; +set debug_dbug='-d,non_blocking_binlog_ignore_cache_size'; +set debug_dbug='-d,non_blocking_binlog_write_cache_failed'; +insert into t1 values(4,4); +show master status; +File Position Binlog_Do_DB Binlog_Ignore_DB +master-bin.000001 # +set debug_dbug='+d,non_blocking_binlog_ignore_cache_size'; +set debug_dbug='+d,non_blocking_binlog_write_gtid_failed'; +update t1 set c1=5; +ERROR HY000: Lost connection to server during query +show master status; +File Position Binlog_Do_DB Binlog_Ignore_DB +master-bin.000002 # +select * from t1; +id c1 +1 3 +3 3 +4 4 +drop table t1; diff --git a/mysql-test/suite/binlog/t/non_blocking_binlog_basic.test b/mysql-test/suite/binlog/t/non_blocking_binlog_basic.test new file mode 100644 index 00000000000..2964f921698 --- /dev/null +++ b/mysql-test/suite/binlog/t/non_blocking_binlog_basic.test @@ -0,0 +1,31 @@ +--source include/have_debug.inc +--source include/have_innodb.inc +--source include/have_binlog_format_row.inc +--source include/not_embedded.inc + +set global non_blocking_binlog_threshold=1; +show variables like 'non_blocking_binlog_threshold'; +set global non_blocking_binlog_threshold=256*1024*1024; +show variables like 'non_blocking_binlog_threshold'; +set global non_blocking_binlog_threshold=default; +show variables like 'non_blocking_binlog_threshold'; + +reset master; +create table t1(id int, c1 int) engine=innodb; +insert into t1 values(1,1); +set debug_dbug='+d,non_blocking_binlog_ignore_cache_size'; +update t1 set c1=2; +set debug_dbug='-d,non_blocking_binlog_ignore_cache_size'; +insert into t1 values(3,3); +--replace_column 2 # +show master status; +--replace_column 2 # 4 # 5 # 6 # +show binlog events in 'master-bin.000002'; +drop table t1; + +--let $assert_count = 1 +--let $assert_file = $MYSQLTEST_VARDIR/log/mysqld.1.err +--let $assert_text = write non-blocking binlog succeed should be logged +--let $assert_select = write non-blocking binlog succeed +--let $assert_only_after = CURRENT_TEST +--source include/assert_grep.inc diff --git a/mysql-test/suite/binlog/t/non_blocking_binlog_error.test b/mysql-test/suite/binlog/t/non_blocking_binlog_error.test new file mode 100644 index 00000000000..6699006bffb --- /dev/null +++ b/mysql-test/suite/binlog/t/non_blocking_binlog_error.test @@ -0,0 +1,49 @@ +--source include/have_debug.inc +--source include/have_innodb.inc +--source include/have_binlog_format_row.inc +--source include/not_valgrind.inc +--source include/not_embedded.inc + +call mtr.add_suppression(".*"); + +reset master; +create table t1(id int, c1 int) engine=innodb; +insert into t1 values(1,1); +set debug_dbug='+d,non_blocking_binlog_ignore_cache_size'; +set debug_dbug='+d,non_blocking_binlog_open_file_failed'; +update t1 set c1=2; +set debug_dbug='-d,non_blocking_binlog_ignore_cache_size'; +set debug_dbug='-d,non_blocking_binlog_open_file_failed'; +insert into t1 values(3,3); +--replace_column 2 # +show master status; + +--let $assert_count = 1 +--let $assert_file = $MYSQLTEST_VARDIR/log/mysqld.1.err +--let $assert_text = Open file failed should be logged +--let $assert_select = Open file .* failed +--let $assert_only_after = CURRENT_TEST +--source include/assert_grep.inc + +set debug_dbug='+d,non_blocking_binlog_ignore_cache_size'; +set debug_dbug='+d,non_blocking_binlog_write_cache_failed'; +update t1 set c1=3; +set debug_dbug='-d,non_blocking_binlog_ignore_cache_size'; +set debug_dbug='-d,non_blocking_binlog_write_cache_failed'; +insert into t1 values(4,4); +--replace_column 2 # +show master status; + +set debug_dbug='+d,non_blocking_binlog_ignore_cache_size'; +set debug_dbug='+d,non_blocking_binlog_write_gtid_failed'; +--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +--error 2013 +update t1 set c1=5; +--enable_reconnect +--source include/wait_until_connected_again.inc +--replace_column 2 # +show master status; +# c1 should not be 5 +select * from t1; + +drop table t1; diff --git a/mysql-test/suite/perfschema/r/dml_setup_instruments.result b/mysql-test/suite/perfschema/r/dml_setup_instruments.result index ff000a09312..95d8bfb882a 100644 --- a/mysql-test/suite/perfschema/r/dml_setup_instruments.result +++ b/mysql-test/suite/perfschema/r/dml_setup_instruments.result @@ -21,6 +21,7 @@ where name like 'Wait/Synch/Rwlock/sql/%' 'wait/synch/rwlock/sql/LOCK_named_pipe_full_access_group') order by name limit 10; NAME ENABLED TIMED +wait/synch/rwlock/sql/binlog_checksum_rwlock YES YES wait/synch/rwlock/sql/LOCK_all_status_vars YES YES wait/synch/rwlock/sql/LOCK_dbnames YES YES wait/synch/rwlock/sql/LOCK_dboptions YES YES @@ -30,7 +31,6 @@ wait/synch/rwlock/sql/LOCK_SEQUENCE YES YES wait/synch/rwlock/sql/LOCK_ssl_refresh YES YES wait/synch/rwlock/sql/LOCK_system_variables_hash YES YES wait/synch/rwlock/sql/LOCK_sys_init_connect YES YES -wait/synch/rwlock/sql/LOCK_sys_init_slave YES YES select * from performance_schema.setup_instruments where name like 'Wait/Synch/Cond/sql/%' and name not in ( diff --git a/mysql-test/suite/rpl/r/rpl_non_blocking_binlog.result b/mysql-test/suite/rpl/r/rpl_non_blocking_binlog.result new file mode 100644 index 00000000000..c130848bfe9 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_non_blocking_binlog.result @@ -0,0 +1,22 @@ +include/master-slave.inc +[connection master] +connection master; +create table t1(id int, c1 int) engine=innodb; +insert into t1 values(1,1); +set debug_dbug='+d,non_blocking_binlog_ignore_cache_size'; +update t1 set c1=2; +set debug_dbug='-d,non_blocking_binlog_ignore_cache_size'; +include/assert_grep.inc [write non-blocking binlog succeed should be logged] +insert into t1 values(3,3); +checksum table t1; +Table Checksum +test.t1 4217795292 +include/sync_slave_sql_with_master.inc +connection slave; +checksum table t1; +Table Checksum +test.t1 4217795292 +connection master; +drop table t1; +include/sync_slave_sql_with_master.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_non_blocking_binlog.test b/mysql-test/suite/rpl/t/rpl_non_blocking_binlog.test new file mode 100644 index 00000000000..558ca63c5ec --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_non_blocking_binlog.test @@ -0,0 +1,31 @@ +--source include/have_debug.inc +--source include/have_innodb.inc +--source include/master-slave.inc +--source include/not_embedded.inc + +connection master; +create table t1(id int, c1 int) engine=innodb; +insert into t1 values(1,1); +set debug_dbug='+d,non_blocking_binlog_ignore_cache_size'; +update t1 set c1=2; +set debug_dbug='-d,non_blocking_binlog_ignore_cache_size'; + +--let $assert_count = 1 +--let $assert_file = $MYSQLTEST_VARDIR/log/mysqld.1.err +--let $assert_text = write non-blocking binlog succeed should be logged +--let $assert_select = write non-blocking binlog succeed +--let $assert_only_after = CURRENT_TEST +--source include/assert_grep.inc + +insert into t1 values(3,3); +checksum table t1; + +--source include/sync_slave_sql_with_master.inc +connection slave; +checksum table t1; + +connection master; +drop table t1; +--source include/sync_slave_sql_with_master.inc + +--source include/rpl_end.inc diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result index 2740319dd05..0ad02a33089 100644 --- a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result +++ b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result @@ -2512,6 +2512,16 @@ NUMERIC_BLOCK_SIZE 1 ENUM_VALUE_LIST NULL READ_ONLY NO COMMAND_LINE_ARGUMENT REQUIRED +VARIABLE_NAME NON_BLOCKING_BINLOG_THRESHOLD +VARIABLE_SCOPE GLOBAL +VARIABLE_TYPE BIGINT UNSIGNED +VARIABLE_COMMENT If the binlog size of a transaction exceeds this value, we write it to a new temporary file and rename it to the next binlog file. +NUMERIC_MIN_VALUE 268435456 +NUMERIC_MAX_VALUE 18446744073709551615 +NUMERIC_BLOCK_SIZE 1 +ENUM_VALUE_LIST NULL +READ_ONLY NO +COMMAND_LINE_ARGUMENT OPTIONAL VARIABLE_NAME NOTE_VERBOSITY VARIABLE_SCOPE SESSION VARIABLE_TYPE SET diff --git a/sql/log.cc b/sql/log.cc index 460cefea47b..9de1c6118cd 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -74,6 +74,9 @@ #include <utility> // pair #endif +#include <atomic> +#include <chrono> + /* max size of the log message */ #define MAX_LOG_BUFFER_SIZE 1024 #define MAX_TIME_SIZE 32 @@ -3727,7 +3730,10 @@ bool MYSQL_BIN_LOG::open(const char *log_name, enum cache_type io_cache_type_arg, ulong max_size_arg, bool null_created_arg, - bool need_mutex) + bool need_mutex, + const char *file_to_rename, + my_off_t file_size, + group_commit_entry *entry) { xid_count_per_binlog *new_xid_list_entry= NULL, *b; DBUG_ENTER("MYSQL_BIN_LOG::open"); @@ -3792,6 +3798,18 @@ bool MYSQL_BIN_LOG::open(const char *log_name, write_error= 0; + if (file_to_rename) + { + DBUG_ASSERT(!is_relay_log && entry); + if (mysql_file_rename(PSI_NOT_INSTRUMENTED, file_to_rename, log_file_name, + MY_WME | MY_NABP | MY_WAIT_IF_FULL) != 0) + { + sql_print_error("rename from '%s' to '%s' failed!", file_to_rename, + log_file_name); + DBUG_RETURN(1); + } + } + /* open the main log file */ if (MYSQL_LOG::open( #ifdef HAVE_PSI_INTERFACE @@ -3816,7 +3834,7 @@ bool MYSQL_BIN_LOG::open(const char *log_name, { bool write_file_name_to_index_file=0; - if (!my_b_filelength(&log_file)) + if (!my_b_filelength(&log_file) || file_to_rename) { /* The binary log file was empty (probably newly created) @@ -3993,6 +4011,12 @@ bool MYSQL_BIN_LOG::open(const char *log_name, goto err; bytes_written+= description_event_for_queue->data_written; } + if (file_to_rename) + { + if (write_gtid_and_skip_event(file_size, entry)) + goto err; + } + if (flush_io_cache(&log_file) || mysql_file_sync(log_file.file, MYF(MY_WME))) goto err; @@ -6826,7 +6850,6 @@ MYSQL_BIN_LOG::write_gtid_event(THD *thd, bool standalone, thd->variables.server_id= global_system_variables.server_id; } #endif - if (write_event(&gtid_event)) DBUG_RETURN(true); status_var_add(thd->status_var.binlog_bytes_written, gtid_event.data_written); @@ -7849,11 +7872,18 @@ int Event_log::write_cache_raw(THD *thd, IO_CACHE *cache) events prior to fill in the binlog cache. */ -int Event_log::write_cache(THD *thd, binlog_cache_data *cache_data) +int Event_log::write_cache(THD *thd, binlog_cache_data *cache_data, + IO_CACHE *f) { DBUG_ENTER("Event_log::write_cache"); IO_CACHE *cache= &cache_data->cache_log; - mysql_mutex_assert_owner(&LOCK_log); + + IO_CACHE *out_file= f; + if (likely(f == nullptr)) + { + mysql_mutex_assert_owner(&LOCK_log); + out_file= get_log_file(); + } /* If possible, just copy the cache over byte-by-byte with pre-computed @@ -7863,7 +7893,7 @@ int Event_log::write_cache(THD *thd, binlog_cache_data *cache_data) likely(!crypto.scheme) && likely(!opt_binlog_legacy_event_pos)) { - int res= my_b_copy_all_to_cache(cache, &log_file); + int res= my_b_copy_all_to_cache(cache, out_file); status_var_add(thd->status_var.binlog_bytes_written, my_b_tell(cache)); DBUG_RETURN(res ? ER_ERROR_ON_WRITE : 0); } @@ -7873,7 +7903,7 @@ int Event_log::write_cache(THD *thd, binlog_cache_data *cache_data) /* Amount of remaining bytes in the IO_CACHE read buffer. */ size_t log_file_pos; uchar header_buf[LOG_EVENT_HEADER_LEN]; - Log_event_writer writer(get_log_file(), 0, + Log_event_writer writer(out_file, 0, (enum_binlog_checksum_alg)binlog_checksum_options, &crypto); uint checksum_len= writer.checksum_len; @@ -7904,7 +7934,7 @@ int Event_log::write_cache(THD *thd, binlog_cache_data *cache_data) split. */ - log_file_pos= (size_t)my_b_tell(get_log_file()); + log_file_pos= (size_t)my_b_tell(out_file); for (;;) { /* @@ -8194,6 +8224,7 @@ MYSQL_BIN_LOG::write_transaction_to_binlog(THD *thd, ha_info= all ? thd->transaction->all.ha_list : thd->transaction->stmt.ha_list; entry.ro_1pc= is_ro_1pc; entry.end_event= end_ev; + entry.non_blocking_log= false; auto has_xid= entry.end_event->get_type_code() == XID_EVENT; for (; has_xid && !entry.need_unlog && ha_info; ha_info= ha_info->next()) @@ -8581,7 +8612,23 @@ MYSQL_BIN_LOG::queue_for_group_commit(group_commit_entry *orig_entry) bool MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) { - int is_leader= queue_for_group_commit(entry); + int is_leader; + if (unlikely(entry->cache_mngr->trx_cache.get_byte_position() >= + non_blocking_binlog_threshold) || + DBUG_IF("non_blocking_binlog_ignore_cache_size")) + { + if (!can_use_non_blocking_binlog(entry) || + write_non_blocking_binlog(entry)) + goto original_commit_path; + /* thread using non-blocking binlog is treated as a single group */ + is_leader= 1; + entry->non_blocking_log= true; + entry->next= nullptr; + goto group_commit_leader; + } +original_commit_path: + + is_leader= queue_for_group_commit(entry); #ifdef WITH_WSREP /* commit order was released in queue_for_group_commit() call, here we check if wsrep_commit_ordered() failed or if we are leader */ @@ -8616,7 +8663,10 @@ MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) if (is_leader < 0) return true; /* Error */ else if (is_leader) + { +group_commit_leader: trx_group_commit_leader(entry); + } else if (!entry->queued_by_other) { DEBUG_SYNC(entry->thd, "after_semisync_queue"); @@ -8748,6 +8798,12 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) ("commit_before_get_LOCK_log SIGNAL waiting WAIT_FOR cont TIMEOUT 1"))); ); #endif + if (unlikely(leader->non_blocking_log)) + { + mysql_mutex_assert_owner(&LOCK_log); + current= leader; + goto after_fetching_group; + } /* Lock the LOCK_log(), and once we get it, collect any additional writes that queued up while we were waiting. @@ -8766,6 +8822,7 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) current= group_commit_queue; group_commit_queue= NULL; mysql_mutex_unlock(&LOCK_prepare_ordered); +after_fetching_group: binlog_id= current_binlog_id; /* As the queue is in reverse order of entering, reverse it. */ @@ -8824,9 +8881,12 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) !cache_mngr->trx_cache.empty() || current->thd->transaction->xid_state.is_explicit_XA()); - if (unlikely((current->error= write_transaction_or_stmt(current, - commit_id)))) - current->commit_errno= errno; + if (likely(leader->non_blocking_log == false)) + { + if (unlikely((current->error= + write_transaction_or_stmt(current, commit_id)))) + current->commit_errno= errno; + } strmake_buf(cache_mngr->last_commit_pos_file, log_file_name); commit_offset= my_b_write_tell(&log_file); @@ -12459,6 +12519,7 @@ mysql_bin_log_commit_pos(THD *thd, ulonglong *out_pos, const char **out_file) } #endif /* INNODB_COMPATIBILITY_HOOKS */ +mysql_rwlock_t binlog_checksum_rwlock; static void binlog_checksum_update(MYSQL_THD thd, struct st_mysql_sys_var *var, @@ -12468,6 +12529,7 @@ binlog_checksum_update(MYSQL_THD thd, struct st_mysql_sys_var *var, bool check_purge= false; ulong UNINIT_VAR(prev_binlog_id); + mysql_rwlock_wrlock(&binlog_checksum_rwlock); mysql_mutex_lock(mysql_bin_log.get_log_lock()); if(mysql_bin_log.is_open()) { @@ -12484,6 +12546,7 @@ binlog_checksum_update(MYSQL_THD thd, struct st_mysql_sys_var *var, DBUG_ASSERT(binlog_checksum_options == value); mysql_bin_log.checksum_alg_reset= BINLOG_CHECKSUM_ALG_UNDEF; mysql_mutex_unlock(mysql_bin_log.get_log_lock()); + mysql_rwlock_unlock(&binlog_checksum_rwlock); if (check_purge) mysql_bin_log.checkpoint_and_purge(prev_binlog_id); } @@ -12781,3 +12844,270 @@ void wsrep_register_binlog_handler(THD *thd, bool trx) } #endif /* WITH_WSREP */ + +bool MYSQL_BIN_LOG::can_use_non_blocking_binlog(group_commit_entry *entry) +{ + DBUG_ASSERT(entry->cache_mngr->trx_cache.get_byte_position() >= + non_blocking_binlog_threshold || + DBUG_IF("non_blocking_binlog_ignore_cache_size")); + THD *thd= entry->thd; + if (unlikely(!is_open()) || encrypt_binlog || + !entry->cache_mngr->stmt_cache.empty() || + entry->cache_mngr->trx_cache.has_incident() || thd->slave_thread || + thd->wait_for_commit_ptr) + return false; +#ifdef WITH_WSREP + if (wsrep_is_active(thd)) + return false; +#endif /* WITH_WSREP */ + return true; +} + +std::string MYSQL_BIN_LOG::generate_random_file_name() +{ + static std::atomic<uint64_t> temp_bin_counter{0}; + static std::atomic<bool> binlog_dir_inited{false}; + static std::string binlog_dir; + std::string temp_file_name; + if (unlikely(!binlog_dir_inited.load(std::memory_order_acquire))) + { + char dev[FN_REFLEN]; + size_t dev_length; + mysql_mutex_lock(&LOCK_log); + if (!binlog_dir_inited.load(std::memory_order_relaxed) && name != nullptr) + { + /* + see MYSQL_BIN_LOG::generate_new_name and fn_format for more + information + */ + size_t length= dirname_part(dev, name, &dev_length); + if (length == 0) + { + // log_bin is not set, use mysql_data_home + convert_dirname(dev, mysql_data_home, NullS); + } + unpack_dirname(dev, dev); // Replace ~/.. with dir + binlog_dir.assign(dev); + binlog_dir_inited.store(true); + } + mysql_mutex_unlock(&LOCK_log); + } + + temp_file_name.reserve(FN_REFLEN); + auto now_in_sys= std::chrono::system_clock::now().time_since_epoch(); + auto now_in_ms= + std::chrono::duration_cast<std::chrono::milliseconds>(now_in_sys) + .count(); + auto count= temp_bin_counter.fetch_add(1); + temp_file_name.append(binlog_dir); + temp_file_name.append("_temp_bin_"); + temp_file_name.append(std::to_string(now_in_ms)); + temp_file_name.push_back('_'); + temp_file_name.append(std::to_string(count)); + + return temp_file_name; +} + +void MYSQL_BIN_LOG::generate_skip_event(uchar *buf, size_t buf_len, + my_off_t pos, THD *thd) +{ + uchar *header= buf; + my_time_t now= thd->start_time; + int4store(header, now); + header[EVENT_TYPE_OFFSET]= IGNORABLE_LOG_EVENT; + int4store(header + SERVER_ID_OFFSET, server_id); + int4store(header + EVENT_LEN_OFFSET, buf_len); + int4store(header + LOG_POS_OFFSET, pos + buf_len); + int2store(header + FLAGS_OFFSET, LOG_EVENT_IGNORABLE_F); + if (binlog_checksum_options != BINLOG_CHECKSUM_ALG_OFF && + binlog_checksum_options != BINLOG_CHECKSUM_ALG_UNDEF) + { + ha_checksum crc= my_checksum(0, header, buf_len - BINLOG_CHECKSUM_LEN); + int4store(header + buf_len - BINLOG_CHECKSUM_LEN, crc); + } +} + +template <typename F= std::function<void()>> class Scoped_guard +{ +public: + Scoped_guard(F f); + ~Scoped_guard(); + +private: + F func; +}; + +template <typename F> Scoped_guard<F>::Scoped_guard(F f) : func(f) {} + +template <typename F> Scoped_guard<F>::~Scoped_guard() +{ + if (func) + func(); +} + +bool MYSQL_BIN_LOG::write_gtid_and_skip_event(my_off_t file_size, + group_commit_entry *entry) +{ + mysql_mutex_assert_owner(&LOCK_log); + DBUG_EXECUTE_IF("non_blocking_binlog_write_gtid_failed", { return true; }); + uchar *skip_event_buf= nullptr; + Scoped_guard<> scoped_guard{[&]() { + if (skip_event_buf) + my_free(skip_event_buf); + }}; + bool has_xid= entry->end_event->get_type_code() == XID_EVENT; + if (write_gtid_event(entry->thd, is_prepared_xa(entry->thd), + entry->using_trx_cache, 0, has_xid, entry->ro_1pc)) + return true; + my_off_t skip_event_start= my_b_safe_tell(&log_file); + size_t skip_event_len= non_blocking_binlog_reserved_size - skip_event_start; + skip_event_buf= + (uchar *) my_malloc(PSI_INSTRUMENT_ME, skip_event_len, MYF(MY_ZEROFILL)); + generate_skip_event(skip_event_buf, skip_event_len, skip_event_start, + entry->thd); + if (my_b_write(&log_file, skip_event_buf, skip_event_len) != 0) + return true; + DBUG_ASSERT(non_blocking_binlog_reserved_size == my_b_safe_tell(&log_file)); + if (flush_io_cache(&log_file) != 0) + return true; + if (reinit_io_cache(&log_file, io_cache_type, file_size, FALSE, TRUE)) + return true; + return false; +} + +bool MYSQL_BIN_LOG::write_non_blocking_binlog(group_commit_entry *entry) +{ + /* generate random file name */ + std::string temp_file_name(generate_random_file_name()); + if (temp_file_name.empty()) + { + sql_print_warning("Generate random file name failed!"); + return true; + } + if (temp_file_name.size() >= FN_REFLEN) + { + sql_print_warning("The name of temp file '%s' is too long!", + temp_file_name.c_str()); + return true; + } + THD *thd= entry->thd; + IO_CACHE temp_log_file; + File file= -1; + bool io_cache_inited= false; + bool delete_file= true; + bool binlog_checksum_locked= false; + // ignore all errors in this function + Dummy_error_handler dummy_error_handler; + thd->push_internal_handler(&dummy_error_handler); + Scoped_guard<> scoped_guard{[&]() { + if (io_cache_inited) + end_io_cache(&temp_log_file); + if (file > 0) + { + mysql_file_close(file, MYF(MY_WME)); + if (delete_file) + { + mysql_file_delete(PSI_NOT_INSTRUMENTED, temp_file_name.c_str(), + MYF(MY_WME)); + } + } + if (binlog_checksum_locked) + mysql_rwlock_unlock(&binlog_checksum_rwlock); + thd->pop_internal_handler(); + }}; + + // open temporary file + int flags= O_WRONLY | O_CREAT; + DBUG_EXECUTE_IF("non_blocking_binlog_open_file_failed", + { flags= O_WRONLY; }); + if ((file= mysql_file_open(PSI_NOT_INSTRUMENTED, temp_file_name.c_str(), + flags, MYF(MY_WME))) < 0) + { + sql_print_error("Open file '%s' failed!", temp_file_name.c_str()); + return true; + } + if (init_io_cache(&temp_log_file, file, LOG_BIN_IO_SIZE, WRITE_CACHE, + non_blocking_binlog_reserved_size, 0, + MYF(MY_WME | MY_NABP | MY_WAIT_IF_FULL))) + { + sql_print_error("Init io cache for file '%s' failed!", + temp_file_name.c_str()); + return true; + } + io_cache_inited= true; + // prevent any changes to binlog_checksum_options while writing cache + mysql_rwlock_rdlock(&binlog_checksum_rwlock); + binlog_checksum_locked= true; + enum_binlog_checksum_alg alg= + (enum_binlog_checksum_alg) binlog_checksum_options; + // write cache + { + binlog_cache_mngr *mngr= entry->cache_mngr; + binlog_cache_data *cache_data= mngr->get_binlog_cache_data(TRUE); + IO_CACHE *cache= &cache_data->cache_log; + if (reinit_io_cache(cache, READ_CACHE, 0, 0, 0)) + return true; + if (write_cache(entry->thd, cache_data, &temp_log_file) || + DBUG_IF("non_blocking_binlog_write_cache_failed")) + return true; + if (write_event(entry->end_event, alg, nullptr, &temp_log_file)) + return true; + status_var_add(entry->thd->status_var.binlog_bytes_written, + entry->end_event->data_written); + if (unlikely(cache->error)) + return true; + } + if (flush_io_cache(&temp_log_file) != 0) + return true; + mysql_mutex_lock(&LOCK_log); + mysql_mutex_lock(&LOCK_index); + // rename temporary file to next binlog file, abort if any error occurs + { + char new_name[FN_REFLEN], *new_name_ptr, *old_name; + File UNINIT_VAR(old_file); + // Reuse old name if not binlog and not update log + new_name_ptr= name; + if (unlikely(generate_new_name(new_name, name, 0))) + abort(); + new_name_ptr= new_name; + /* + We log the whole file name for log file as the user may decide + to change base names at some point. + */ + Rotate_log_event r(new_name + dirname_length(new_name), 0, + LOG_EVENT_OFFSET, 0); + if (unlikely(write_event(&r, alg))) + abort(); + if (unlikely(flush_io_cache(&log_file) != 0)) + abort(); + update_binlog_end_pos(); + old_name= name; + name= 0; // Don't free name + old_file= log_file.file; + uint close_flag= + LOG_CLOSE_TO_BE_OPENED | LOG_CLOSE_INDEX | LOG_CLOSE_DELAYED_CLOSE; + /* + We need to keep the old binlog file open (and marked as in-use) until + the new one is fully created and synced to disk and index. Otherwise we + leave a window where if we crash, there is no binlog file marked as + crashed for server restart to detect the need for recovery. + */ + close(close_flag); + if (unlikely(open_index_file(index_file_name, 0, FALSE))) + abort(); + if (unlikely(open(old_name, new_name_ptr, 0, io_cache_type, max_size, 1, + FALSE, temp_file_name.c_str(), + my_b_safe_tell(&temp_log_file), entry))) + abort(); + my_free(old_name); + delete_file= false; + clear_inuse_flag_when_closing(old_file); + mysql_file_close(old_file, MYF(MY_WME)); + } + mysql_mutex_unlock(&LOCK_index); + // unlock lock_log in trx_group_commit_leader +#ifndef NDEBUG + sql_print_information("write non-blocking binlog succeed!"); +#endif /* ! NDEBUG */ + return false; +} diff --git a/sql/log.h b/sql/log.h index fc5209d1922..af2b56da802 100644 --- a/sql/log.h +++ b/sql/log.h @@ -19,6 +19,7 @@ #include "handler.h" /* my_xid */ #include "rpl_constants.h" +#include <string> class Relay_log_info; class Gtid_index_writer; @@ -406,7 +407,8 @@ class Event_log: public MYSQL_LOG void set_write_error(THD *thd, bool is_transactional); static bool check_write_error(THD *thd); static bool check_cache_error(THD *thd, binlog_cache_data *cache_data); - int write_cache(THD *thd, binlog_cache_data *cache_data); + int write_cache(THD *thd, binlog_cache_data *cache_data, + IO_CACHE *f= nullptr); int write_cache_raw(THD *thd, IO_CACHE *cache); char* get_name() { return name; } void cleanup() @@ -650,6 +652,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log bool queued_by_other; ulong binlog_id; bool ro_1pc; // passes the binlog_cache_mngr::ro_1pc value to Gtid ctor + bool non_blocking_log; }; /* @@ -986,7 +989,10 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log enum cache_type io_cache_type_arg, ulong max_size, bool null_created, - bool need_mutex); + bool need_mutex, + const char *file_to_rename= nullptr, + my_off_t file_size= 0, + group_commit_entry *entry= nullptr); bool open_index_file(const char *index_file_name_arg, const char *log_name, bool need_mutex); /* Use this to start writing a new log file */ @@ -1177,6 +1183,24 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log */ my_off_t binlog_end_pos; char binlog_end_pos_file[FN_REFLEN]; + +private: + // reserved size for format_description_log_event, gtid_list_log_event ... + static const my_off_t non_blocking_binlog_reserved_size= 10 * 1024 * 1024; + bool can_use_non_blocking_binlog(group_commit_entry *entry); + std::string generate_random_file_name(); + void generate_skip_event(uchar *buf, size_t buf_len, my_off_t pos, THD *thd); + /* + write gtid_log_event and ignorable_log_event(skip event) to binlog + @returns if a problem occurs, false otherwise + */ + bool write_gtid_and_skip_event(my_off_t file_size, + group_commit_entry *entry); + /* + write non-blocking binlog for big transaction + @returns true if a problem occurs, false otherwise + */ + bool write_non_blocking_binlog(group_commit_entry *entry); }; class Log_event_handler @@ -1476,4 +1500,6 @@ int binlog_commit_by_xid(handlerton *hton, XID *xid); int binlog_rollback_by_xid(handlerton *hton, XID *xid); bool write_bin_log_start_alter(THD *thd, bool& partial_alter, uint64 start_alter_id, bool log_if_exists); + +extern mysql_rwlock_t binlog_checksum_rwlock; #endif /* LOG_H */ diff --git a/sql/mysqld.cc b/sql/mysqld.cc index dc4a955aa79..8f9bdb02599 100644 --- a/sql/mysqld.cc +++ b/sql/mysqld.cc @@ -557,6 +557,7 @@ ulong opt_binlog_commit_wait_usec= 0; ulong opt_slave_parallel_max_queued= 131072; my_bool opt_gtid_ignore_duplicates= FALSE; uint opt_gtid_cleanup_batch_size= 64; +ulonglong non_blocking_binlog_threshold= ULONGLONG_MAX; const double log_10[] = { 1e000, 1e001, 1e002, 1e003, 1e004, 1e005, 1e006, 1e007, 1e008, 1e009, @@ -1059,7 +1060,8 @@ PSI_rwlock_key key_rwlock_LOCK_grant, key_rwlock_LOCK_logger, key_rwlock_LOCK_vers_stats, key_rwlock_LOCK_stat_serial, key_rwlock_LOCK_ssl_refresh, key_rwlock_THD_list, - key_rwlock_LOCK_all_status_vars; + key_rwlock_LOCK_all_status_vars, + key_rwlock_binlog_checksum; static PSI_rwlock_info all_server_rwlocks[]= { @@ -1077,7 +1079,8 @@ static PSI_rwlock_info all_server_rwlocks[]= { &key_rwlock_LOCK_stat_serial, "TABLE_SHARE::LOCK_stat_serial", 0}, { &key_rwlock_LOCK_ssl_refresh, "LOCK_ssl_refresh", PSI_FLAG_GLOBAL }, { &key_rwlock_THD_list, "THD_list::lock", PSI_FLAG_GLOBAL }, - { &key_rwlock_LOCK_all_status_vars, "LOCK_all_status_vars", PSI_FLAG_GLOBAL } + { &key_rwlock_LOCK_all_status_vars, "LOCK_all_status_vars", PSI_FLAG_GLOBAL }, + { &key_rwlock_binlog_checksum, "binlog_checksum_rwlock", PSI_FLAG_GLOBAL } }; #ifdef HAVE_MMAP @@ -2105,6 +2108,7 @@ static void clean_up_mutexes() mysql_mutex_destroy(&LOCK_start_thread); mysql_mutex_destroy(&LOCK_status); mysql_rwlock_destroy(&LOCK_all_status_vars); + mysql_rwlock_destroy(&binlog_checksum_rwlock); mysql_mutex_destroy(&LOCK_delayed_insert); mysql_mutex_destroy(&LOCK_delayed_status); mysql_mutex_destroy(&LOCK_delayed_create); @@ -4428,6 +4432,7 @@ static int init_thread_environment() mysql_rwlock_init(key_rwlock_LOCK_ssl_refresh, &LOCK_ssl_refresh); mysql_rwlock_init(key_rwlock_LOCK_grant, &LOCK_grant); mysql_rwlock_init(key_rwlock_LOCK_all_status_vars, &LOCK_all_status_vars); + mysql_rwlock_init(key_rwlock_binlog_checksum, &binlog_checksum_rwlock); mysql_cond_init(key_COND_start_thread, &COND_start_thread, NULL); #ifdef HAVE_REPLICATION mysql_mutex_init(key_LOCK_rpl_status, &LOCK_rpl_status, MY_MUTEX_INIT_FAST); diff --git a/sql/mysqld.h b/sql/mysqld.h index 4548a247ac1..4f9dfe72b4c 100644 --- a/sql/mysqld.h +++ b/sql/mysqld.h @@ -317,6 +317,7 @@ extern const char *encryption_algorithm_names[]; extern long opt_secure_timestamp; extern uint default_password_lifetime; extern my_bool disconnect_on_expired_password; +extern ulonglong non_blocking_binlog_threshold; enum secure_timestamp { SECTIME_NO, SECTIME_SUPER, SECTIME_REPL, SECTIME_YES }; bool is_set_timestamp_forbidden(THD *thd); @@ -371,7 +372,8 @@ extern PSI_rwlock_key key_rwlock_LOCK_grant, key_rwlock_LOCK_logger, key_rwlock_LOCK_system_variables_hash, key_rwlock_query_cache_query_lock, key_LOCK_SEQUENCE, key_rwlock_LOCK_vers_stats, key_rwlock_LOCK_stat_serial, - key_rwlock_THD_list; + key_rwlock_THD_list, + key_rwlock_binlog_checksum; #ifdef HAVE_MMAP extern PSI_cond_key key_PAGE_cond, key_COND_active, key_COND_pool; diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc index 47e31291e6b..74e6fe9bb23 100644 --- a/sql/sys_vars.cc +++ b/sql/sys_vars.cc @@ -7327,3 +7327,11 @@ static Sys_var_enum Sys_block_encryption_mode( "AES_ENCRYPT() and AES_DECRYPT() functions", SESSION_VAR(block_encryption_mode), CMD_LINE(REQUIRED_ARG), block_encryption_mode_values, DEFAULT(0)); + +static Sys_var_ulonglong Sys_var_non_blocking_binlog_threshold( + "non_blocking_binlog_threshold", + "If the binlog size of a transaction exceeds this value, we write it to a " + "new temporary file and rename it to the next binlog file.", + GLOBAL_VAR(non_blocking_binlog_threshold), CMD_LINE(OPT_ARG), + VALID_RANGE(256 * 1024 * 1024, ULONGLONG_MAX), DEFAULT(ULONGLONG_MAX), + BLOCK_SIZE(1)); -- 2.30.2

1 0

[PATCH] MDEV-21322: Report slave progress to the master
by Kristian Nielsen 31 May '24

31 May '24

From: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> This patch extends the command `SHOW REPLICA HOSTS` with three columns: 1) Gtid_State_Sent. This represents that latest GTIDs sent to the replica in each domain. It will always be populated, regardless of the semi-sync status (i.e. asynchronous connections will still update this column with the latest GTID state sent to the replica). 2) Gtid_State_Ack. For semi-synchronous connections (only), this column represents the last GTID in each domain that the replica has acknowledged. 3) Sync_Status. This value represents the synchronization status of the replica, and is used to help determine how to interpret the Gtid_State_Ack column. There are four possible values: 3.1) Initializing. This means the binlog dump thread is still initializing, and has not yet determined the synchronization status of the replica. 3.2) Asynchronous: This means the replica is not configured for semi-sync replication, and thereby, Gtid_State_Ack should always be empty. 3.3) Semi-sync Stale: This means the replica is configured for semi-sync replication, however, connected using an old state, and is not readily able to send ACKs for new transactions. Functionally, this means that the primary will try to catch the replica up-to-date by sending transactions which will not be ACKed. Additionally, the value shown by Gtid_State_Ack will be empty until the replica catches up and ACKs its first transaction. 3.4) Semi-sync Active: This means the replica is configured for semi-sync replication, and is readily sending ACKs for new transactions it receives. It is possible for Gtid_State_Ack to be empty while Sync_Status is "Semi-sync Active" if no new transactions have been executed on the primary since the replica has connected. Additionally, this patch creates a new semantic for the configuration rpl_semi_sync_master_timeout=0. That is, now when 0, 1) new transactions will not attempt to wait for an ACK before completing, and 2) the primary will still request ACKs from the replica for new transactions. This means that Gtid_State_Ack will be updated for each ACK from the replica and Sync_Status will read as "Semi-sync Active". Effectively, this creates a mode to mimic the asynchronous connection behavior, while allowing one to monitor the progress at which the primary is sending transactions to the replica via the new columns Gtid_State_Sent and Gtid_State_Ack. Also note that a new error message was added to account for the case that Gtid_State_(Sent/Ack) represents a binary log file that was purged/cannot be found. The overall implementation is rather simple. It leverages the existing semi-sync framework, where the replica uses binlog file:pos to ACK transactions, in order to infer GTID state by performing a binlog lookup at the time `SHOW REPLICA HOSTS` is executed. In particular, the Slave_info struct is extended to store 1) the binlog file:pos pair of the transaction which was last sent to the replica, 2) the binlog file:pos pair that was last ACKed by the replica, and 3) and enum to represent the Sync_Status. This patch was initially started by @JackSlateur in PR#1427, where it was then transferred to @an3l who buffed it out in PR#2374, and final touches were put on by @bnestere. Reviewed By: ============ <TODO> --- mysql-test/main/grant_master_admin.result | 2 +- .../suite/rpl/r/rpl_fail_register.result | 2 +- .../suite/rpl/r/rpl_mixed_ddl_dml.result | 4 +- .../suite/rpl/r/rpl_show_slave_hosts.result | 516 ++++++++++- .../suite/rpl/t/rpl_show_slave_hosts.cnf | 9 +- .../suite/rpl/t/rpl_show_slave_hosts.test | 808 +++++++++++++++++- sql/repl_failsafe.cc | 81 +- sql/semisync_master.cc | 58 +- sql/semisync_master.h | 93 +- sql/semisync_master_ack_receiver.cc | 16 +- sql/share/errmsg-utf8.txt | 2 + sql/slave.cc | 47 +- sql/sql_repl.cc | 66 +- sql/sql_repl.h | 3 +- 14 files changed, 1616 insertions(+), 91 deletions(-) diff --git a/mysql-test/main/grant_master_admin.result b/mysql-test/main/grant_master_admin.result index bd08ade940c..97a7b4d0024 100644 --- a/mysql-test/main/grant_master_admin.result +++ b/mysql-test/main/grant_master_admin.result @@ -28,7 +28,7 @@ GRANT REPLICATION MASTER ADMIN ON *.* TO `user1`@`localhost` connect con1,localhost,user1,,; connection con1; SHOW SLAVE HOSTS; -Server_id Host Port Master_id +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status connection default; DROP USER user1@localhost; # diff --git a/mysql-test/suite/rpl/r/rpl_fail_register.result b/mysql-test/suite/rpl/r/rpl_fail_register.result index 0398220c4d0..7af07b335b3 100644 --- a/mysql-test/suite/rpl/r/rpl_fail_register.result +++ b/mysql-test/suite/rpl/r/rpl_fail_register.result @@ -14,7 +14,7 @@ set global debug_dbug=@old_dbug; connection master; kill DUMP_THREAD; show slave hosts; -Server_id Host Port Master_id +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status connection slave; start slave; include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result b/mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result index 0cee79434ee..ec1b8d46824 100644 --- a/mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result +++ b/mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result @@ -11,8 +11,8 @@ n 2002 connection master; show slave hosts; -Server_id Host Port Master_id -2 127.0.0.1 9999 1 +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +2 127.0.0.1 9999 1 0-1-2 Asynchronous drop table t1; connection slave; stop slave; diff --git a/mysql-test/suite/rpl/r/rpl_show_slave_hosts.result b/mysql-test/suite/rpl/r/rpl_show_slave_hosts.result index 0c8903378a7..34a5450ed82 100644 --- a/mysql-test/suite/rpl/r/rpl_show_slave_hosts.result +++ b/mysql-test/suite/rpl/r/rpl_show_slave_hosts.result @@ -1,20 +1,504 @@ -include/master-slave.inc -[connection master] -connect slave2,127.0.0.1,root,,test,$SLAVE_MYPORT2,; -connection slave2; -RESET SLAVE; -CHANGE MASTER TO master_host='127.0.0.1',master_port=MASTER_PORT,master_user='root', master_ssl_verify_server_cert=0; -START SLAVE IO_THREAD; -include/wait_for_slave_io_to_start.inc -connection master; +include/rpl_init.inc [topology=1->2,1->3] +connection server_1; SHOW SLAVE HOSTS; -Server_id Host Port Master_id -3 slave2 SLAVE_PORT 1 -2 localhost SLAVE_PORT 1 -connection slave2; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +2 localhost SLAVE_PORT 1 Asynchronous +3 slave2 SLAVE2_PORT 1 Asynchronous +connection server_3; include/stop_slave_io.inc -connection master; +connection server_1; SHOW SLAVE HOSTS; -Server_id Host Port Master_id -2 localhost SLAVE_PORT 1 +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +2 localhost SLAVE_PORT 1 Asynchronous +# +# MDEV-21322: report slave progress to the primary +# +# +# 21322.0: Test case set-up +# +connection server_1; +set sql_log_bin=0; +call mtr.add_suppression("Got an error reading communication packets"); +call mtr.add_suppression("Semi-sync master failed on net_flush"); +call mtr.add_suppression("Could not read packet:.* vio_errno: 1158"); +call mtr.add_suppression("Could not write packet:.* vio_errno: 1160"); +set sql_log_bin=1; +set @save_primary_dbug= @@global.debug_dbug; +set @save_semisync_timeout= @@global.rpl_semi_sync_master_timeout; +set @save_semisync_master_enabled= @@global.rpl_semi_sync_master_enabled; +create table t1 (a int); +connection server_2; +set @save_s2_debug= @@GLOBAL.debug_dbug; +set @save_semisync_server_2_enabled= @@global.rpl_semi_sync_master_enabled; +set sql_log_bin=0; +call mtr.add_suppression('Slave I/O: Relay log write failure: could not queue event from master.*'); +call mtr.add_suppression('Slave I/O: Replication event checksum verification failed while reading from network.*'); +call mtr.add_suppression('Replication event checksum verification failed'); +call mtr.add_suppression("Timeout waiting for reply of binlog*"); +call mtr.add_suppression('Found invalid event in binary log'); +call mtr.add_suppression('event read from binlog did not pass crc check'); +call mtr.add_suppression('Event crc check failed! Most likely there is event corruption'); +call mtr.add_suppression('Slave SQL: Error initializing relay log position: I/O error reading event at position .*, error.* 1593'); +call mtr.add_suppression("Semi-sync slave .* reply"); +set sql_log_bin=1; +connection server_3; +set @save_s3_debug= @@GLOBAL.debug_dbug; +set @save_semisync_server_3_enabled= @@global.rpl_semi_sync_master_enabled; +set sql_log_bin=0; +call mtr.add_suppression('Slave I/O: Relay log write failure: could not queue event from master.*'); +call mtr.add_suppression('Slave I/O: Replication event checksum verification failed while reading from network.*'); +call mtr.add_suppression('Replication event checksum verification failed'); +call mtr.add_suppression("Timeout waiting for reply of binlog*"); +call mtr.add_suppression('Found invalid event in binary log'); +call mtr.add_suppression('event read from binlog did not pass crc check'); +call mtr.add_suppression('Event crc check failed! Most likely there is event corruption'); +call mtr.add_suppression('Slave SQL: Error initializing relay log position: I/O error reading event at position .*, error.* 1593'); +call mtr.add_suppression("Semi-sync slave .* reply"); +set sql_log_bin=1; +connection server_1; +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +# +# 21322.1: In a fresh replication state with semi-sync disabled, +# the Sync_Status column should reflect an asynchronous replication +# state, and Gtid_State_Sent/Ack should start, and only Gtid_State_Sent +# should update with new transaction. Note only server_2 is currently +# connected. +# +connection server_1; +SHOW SLAVE HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +2 localhost SLAVE_PORT 1 0-1-1 Asynchronous +insert into t1 values (1); +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +connection server_1; +# Gtid_State_Sent should be updated for new transaction +SHOW SLAVE HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +2 localhost SLAVE_PORT 1 0-1-2 Asynchronous +# +# 21322.2: When only the primary enables semi-sync, Sync_Status should +# still be asynchronous +# +connection server_1; +set global rpl_semi_sync_master_enabled= 1; +show variables like 'rpl_semi_sync_master_enabled'; +Variable_name Value +rpl_semi_sync_master_enabled ON +SHOW SLAVE HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +2 localhost SLAVE_PORT 1 0-1-2 Asynchronous +# +# 21322.3: Finalizing the semi-sync connection on server_2 (i.e. by +# enabling it on the slave) should update Sync_Status to semi-sync +# active, as the slave is up-to-date. +# +connection server_2; +include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +include/start_slave.inc +connection server_1; +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +2 localhost SLAVE_PORT 1 Semi-sync Active +# +# 21322.4: After new semi-sync transactions are ACKed, +# Gtid_State_Sent/Ack should match gtid_binlog_pos, and Sync_Status +# should read that semi-sync is active +# +connection server_1; +insert into t1 values (2); +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +connection server_1; +# Ensuring master gtid_binlog_pos matches Gtid_State_Sent +# Ensuring master gtid_binlog_pos matches Gtid_State_Ack +# Ensuring Sync_Status is semi-sync active +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +2 localhost SLAVE_PORT 1 0-1-3 0-1-3 Semi-sync Active +# +# 21322.5: When connecting a new slave (server_id 3) which initially has +# semi-sync disabled, SHOW SLAVE HOSTS on the master should show its +# Sync_Status as asynchronous (while server_id 2 is still semi-sync +# active). +# +connection server_3; +include/start_slave.inc +include/sync_with_master_gtid.inc +connection server_1; +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +3 slave2 SLAVE2_PORT 1 0-1-3 Asynchronous +2 localhost SLAVE_PORT 1 0-1-3 0-1-3 Semi-sync Active +# +# 21322.6: Reconnecting server_3 as a semi-sync enabled replica should +# result in a Sync_Status reflecting active semi-sync +# +connection server_3; +include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +include/start_slave.inc +connection server_1; +show status like 'Rpl_semi_sync_master_clients'; +Variable_name Value +Rpl_semi_sync_master_clients 2 +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +3 slave2 SLAVE2_PORT 1 Semi-sync Active +2 localhost SLAVE_PORT 1 0-1-3 0-1-3 Semi-sync Active +# +# 21322.7: New transactions on the master should update Gtid_State_Sent +# when sent to the slave, and Gtid_State_Ack once receiving an ACK +# +connection server_2; +connection server_1; +SET @@GLOBAL.debug_dbug="+d,pause_ack_thread_on_next_ack"; +connection default; +insert into t1 values (3); +connection server_1; +# waiting for pause_ack_reply_to_binlog +SET debug_sync='now WAIT_FOR pause_ack_reply_to_binlog'; +# Waiting for Gtid_State_Sent to reflect latest transaction for all replicas.. +# Ensuring Gtid_State_Ack is not yet updated (as ACK thread is paused) +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +3 slave2 SLAVE2_PORT 1 0-1-4 Semi-sync Active +2 localhost SLAVE_PORT 1 0-1-4 0-1-3 Semi-sync Active +connection server_1; +SET debug_sync='now SIGNAL resume_ack_thread'; +connection default; +connection server_1; +# Waiting for Gtid_State_Ack to reflect latest transaction for all replicas.. +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +3 slave2 SLAVE2_PORT 1 0-1-4 0-1-4 Semi-sync Active +2 localhost SLAVE_PORT 1 0-1-4 0-1-4 Semi-sync Active +# Reset debug state +SET @@GLOBAL.debug_dbug= @save_primary_dbug; +SET debug_sync='RESET'; +# +# 21322.8: Holding one replica's ACK (server_2) should result in +# Gtid_State_Ack of server_3 updating to the most recent GTID, while +# server_id 2 has the old GTID. Note that we need to use debug_sync to +# synchronize the ACKs of both server_2 and server_3, so server_3 can't +# ACK the transaction before server_2's binlog dump thread sends the +# transaction (which would negate the need for server_2 to ACK at all, +# resulting in MTR hanging on its expected debug_sync WAIT_FOR point.) +# +connection server_2; +include/stop_slave.inc +set @@GLOBAL.debug_dbug="+d,synchronize_semisync_slave_reply"; +include/start_slave.inc +connection server_3; +include/stop_slave.inc +set @@GLOBAL.debug_dbug="+d,synchronize_semisync_slave_reply"; +include/start_slave.inc +connection server_1; +# Waiting for master to recognize slave restart.. +insert into t1 values (4); +connection server_2; +set debug_sync= "now WAIT_FOR at_slave_reply"; +connection server_3; +set debug_sync= "now WAIT_FOR at_slave_reply"; +connection default; +# Ensure Gtid_State_Sent reflects latest transaction (0-1-5) for all replicas.. +connection server_3; +set debug_sync= "now SIGNAL reply_ack_to_master"; +connection server_1; +# Wait for Gtid_State_Ack to show the latest transaction for server_3.. +# Only server_3 should have ACKed the new GTID, server_2 should not due to debug_sync holding off the ACK +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +3 slave2 SLAVE2_PORT 1 0-1-5 0-1-5 Semi-sync Active +2 localhost SLAVE_PORT 1 0-1-5 Semi-sync Active +connection server_2; +# Resume slave so it can ACK the transaction +set debug_sync= "now SIGNAL reply_ack_to_master"; +# Waiting for Gtid_State_Ack to reflect latest transaction for all replicas.. +connection server_1; +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +3 slave2 SLAVE2_PORT 1 0-1-5 0-1-5 Semi-sync Active +2 localhost SLAVE_PORT 1 0-1-5 0-1-5 Semi-sync Active +connection server_2; +include/stop_slave.inc +set @@GLOBAL.debug_dbug= @save_s2_debug; +SET debug_sync='RESET'; +include/start_slave.inc +connection server_3; +include/stop_slave.inc +set @@GLOBAL.debug_dbug= @save_s3_debug; +SET debug_sync='RESET'; +include/start_slave.inc +connection server_1; +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +connection server_3; +include/sync_with_master_gtid.inc +# +# 21322.9: If a server is behind when connecting to a primary (i.e. the +# primary has newer transactions), the replica Sync_Status should +# present as semi-sync stale and the Gtid_State_Ack should not populate +# until it has reached Semi-Sync Active +# +connection server_2; +include/stop_slave.inc +connection server_1; +insert into t1 values (5); +include/save_master_gtid.inc +# Pause dump_thread of server_2 (server_3 won't be affected as it has +# already successfully ACKed the new transaction) +set @@global.debug_dbug= "+d,pause_dump_thread_after_sending_next_full_trx"; +connection server_2; +include/start_slave.inc +connection server_1; +set debug_sync= 'now WAIT_FOR dump_thread_paused'; +# Ensure Gtid_State_Sent is updated to represent new transaction has +# been sent to both replicas +# Ensure Sync_Status is Semi-sync Stale for the debug_sync held dump +# thread, as it hasn't yet got "up-to-date" +set debug_sync= 'now SIGNAL dump_thread_continue'; +# Ensure Sync_Status will automatically update to Semi-sync Active +# once the last stale transaction has finished sending +set @@global.debug_dbug= @save_primary_dbug; +SET debug_sync='RESET'; +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +connection server_3; +include/sync_with_master_gtid.inc +# +# 21322.10a: If one replica errors (i.e. server_2 by injecting +# corrupt_gtid_event with debug_dbug), it shouldn't send its ACK, +# and server_3 does ACK; then Gtid_State_Sent should still reflect the +# new transaction for each replica, but Gtid_State_Ack should only be +# updated by the successful transaction. When the errored replica +# reconnects, it shouldn't ACK the transaction, so its on-reconnect +# Gtid_State_Ack value should be empty, but still have a Sync_Status +# of "Semi-sync Active" when it receives the latest transaction, as it +# will be ready to ACK new transactions going forward. +# +connection server_2; +include/stop_slave.inc +SET @@GLOBAL.debug_dbug= "+d,corrupt_gtid_event"; +include/start_slave.inc +# Set-up server_3 for 10b (so we don't have to restart the slave) +connection server_3; +include/stop_slave.inc +set @@GLOBAL.debug_dbug="+d,synchronize_semisync_slave_reply"; +include/start_slave.inc +connection server_1; +# Waiting for master to recognize slave restarts.. +connection server_1; +insert into t1 values (6); +# Debug_sync is irrelevant to this testcase (10a) but we must do it to +# allow server_3 to ACK now (Note debug_sync is needed for 10b) +connection server_3; +set debug_sync= "now WAIT_FOR at_slave_reply"; +set debug_sync= "now SIGNAL reply_ack_to_master"; +connection server_1; +include/save_master_gtid.inc +# Waiting for Gtid_State_Sent to reflect latest transaction for all replicas.. +connection server_2; +include/wait_for_slave_io_error.inc [errno=1595] +set @@GLOBAL.debug_dbug= @save_s2_debug; +connection server_1; +# Only server_3 should ACKed have the new GTID, server_2 should not due to corrupt_queue_event +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +3 slave2 SLAVE2_PORT 1 0-1-7 0-1-7 Semi-sync Active +2 localhost SLAVE_PORT 1 0-1-7 Semi-sync Active +connection server_2; +include/start_slave.inc +include/sync_with_master_gtid.inc +connection server_1; +# With replica restarted/synced, its Gtid_State_Ack should be empty with Sync_Status semi-sync active +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +2 localhost SLAVE_PORT 1 0-1-7 Semi-sync Active +3 slave2 SLAVE2_PORT 1 0-1-7 0-1-7 Semi-sync Active +# +# 21322.10b: Succeeding the previous 10a test, if server_3 now stalls +# (i.e. using debug_sync), then the previously errored server_2 should +# receive and ACK new transactions as a "lone" replica, and update its +# Gtid_State_* columns appropriately, whereas server_3's Gtid_State_Ack +# column should not be updated. +# +insert into t1 values (7); +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +# Waiting for Gtid_State_Sent to reflect latest transaction for all replicas.. +connection server_3; +set debug_sync= "now WAIT_FOR at_slave_reply"; +connection server_1; +# Only server_2 should have ACKed the new GTID ACKed, server_3 should not due to stall +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +2 localhost SLAVE_PORT 1 0-1-8 0-1-8 Semi-sync Active +3 slave2 SLAVE2_PORT 1 0-1-8 0-1-7 Semi-sync Active +# Resume server_3 +connection server_3; +set debug_sync= "now SIGNAL reply_ack_to_master"; +include/sync_with_master_gtid.inc +SET debug_sync='RESET'; +include/stop_slave.inc +set @@GLOBAL.debug_dbug= @save_s3_debug; +SET debug_sync='RESET'; +include/start_slave.inc +# +# 21322.11: Configuration rpl_semi_sync_master_timeout of 0 should +# have transaction behavior match asynchronous behavior (i.e. trxs don't +# need to wait for ACKs), yet the slave should still send ACKs as a +# normal semi-sync replica, and Gtid_State_Ack should still be updated +# accordingly. +# +# The actual behavior tested in this case is as follows: +# a) Transactions won't await ACKs to complete +# b) Semi-sync remains ON when a transaction completes without an ACK +# c) Gtid_State_Ack is updated accordingly for each replica's ACK (even +# when it is behind). Here, we hold both replicas using DEBUG_SYNC to +# not send their ACKs, meanwhile, we continue creating transactions +# on the primary. +# d) If a "very lagged" replica sends an ACK for a transaction from a +# purged binlog, the Gtid_State_Ack value should be cleared, and +# issue a warning to the user with the slave's last ACKed binlog +# coordinate (i.e. filename and position). +# +connection server_1; +set global rpl_semi_sync_master_timeout=0; +connection server_2; +include/stop_slave.inc +SET @@GLOBAL.DEBUG_DBUG="+d,synchronize_semisync_slave_reply"; +include/start_slave.inc +connection server_3; +include/stop_slave.inc +SET @@GLOBAL.DEBUG_DBUG="+d,synchronize_semisync_slave_reply"; +include/start_slave.inc +connection server_1; +# Waiting for master to recognize slave restarts.. +# +# 21322.11.a +connection server_1; +insert into t1 values (8); +# Waiting for Gtid_State_Sent to reflect latest transaction for all replicas.. +connection server_2; +SET debug_sync='now WAIT_FOR at_slave_reply'; +connection server_3; +SET debug_sync='now WAIT_FOR at_slave_reply'; +# Gtid_State_Ack should be empty for both replicas (as they were restarted).. +# +# 21322.11.b +connection server_1; +# Ensuring semi-sync status on primary is correct.. +# +# 21322.11.c +connection server_1; +insert into t1 values (9); +include/save_master_gtid.inc +# Waiting for Gtid_State_Sent to reflect latest transaction for all replicas.. +connection server_1; +# server_2 and 3 should both show an empty ACK state +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +3 slave2 SLAVE2_PORT 1 TRX2_GTID Semi-sync Active +2 localhost SLAVE_PORT 1 TRX2_GTID Semi-sync Active +# Let server_2 ACK just the first transaction +connection server_2; +set debug_sync= "now SIGNAL reply_ack_to_master"; +set debug_sync= "now WAIT_FOR at_slave_reply"; +connection server_1; +# Waiting for server_2 Gtid_State_Ack to reflect first transaction +# Let server_2 ACK the second transaction +connection server_2; +set debug_sync= "now SIGNAL reply_ack_to_master"; +include/sync_with_master_gtid.inc +connection server_1; +# Waiting for server_2 Gtid_State_Ack to reflect second transaction +# Let server_3 now ACK the first transaction +connection server_3; +set debug_sync= "now SIGNAL reply_ack_to_master"; +set debug_sync= "now WAIT_FOR at_slave_reply"; +connection server_1; +# Waiting for server_3 Gtid_State_Ack to reflect first transaction +# Let server_3 ACK the second transaction +connection server_3; +set debug_sync= "now SIGNAL reply_ack_to_master"; +include/sync_with_master_gtid.inc +connection server_1; +# Waiting for Gtid_State_Ack to reflect second transaction for both servers +# +# 21322.11.d +connection server_1; +FLUSH LOGS; +insert into t1 values (10); +include/save_master_gtid.inc +# Waiting for Gtid_State_Sent to reflect latest transaction for all replicas.. +connection server_2; +set debug_sync= "now WAIT_FOR at_slave_reply"; +connection server_3; +set debug_sync= "now WAIT_FOR at_slave_reply"; +connection server_1; +# server_2 and 3 should both show ACKed TRX2 (with TRX3 Sent) +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +3 slave2 SLAVE2_PORT 1 TRX3_GTID TRX2_GTID Semi-sync Active +2 localhost SLAVE_PORT 1 TRX3_GTID TRX2_GTID Semi-sync Active +include/wait_for_purge.inc "master-bin.000002" +# Master should warn that the binary log which contains the last ACKed +# binlog coordinates has been purged, and clear Gtid_State_Ack +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +3 slave2 SLAVE2_PORT 1 TRX3_GTID Semi-sync Active +2 localhost SLAVE_PORT 1 TRX3_GTID Semi-sync Active +Warnings: +Warning 4200 Error constructing GTID state for binlog position TRX2_BINLOG_POS in file 'TRX2_BINLOG_FILE': Could not find binary log file. Probably the slave state is too old and required binlog files have been purged. +Warning 4200 Error constructing GTID state for binlog position TRX2_BINLOG_POS in file 'TRX2_BINLOG_FILE': Could not find binary log file. Probably the slave state is too old and required binlog files have been purged. +# Let servers ACK new transaction +connection server_2; +set debug_sync= "now SIGNAL reply_ack_to_master"; +connection server_3; +set debug_sync= "now SIGNAL reply_ack_to_master"; +connection server_1; +# Waiting for Gtid_State_Ack to reflect latest transaction for all replicas.. +# Gtid_State_Ack should now show the latest transaction GTID +SHOW REPLICA HOSTS; +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status +3 slave2 SLAVE2_PORT 1 TRX3_GTID TRX3_GTID Semi-sync Active +2 localhost SLAVE_PORT 1 TRX3_GTID TRX3_GTID Semi-sync Active +connection server_2; +include/sync_with_master_gtid.inc +connection server_3; +include/sync_with_master_gtid.inc +# +# MDEV-21322 Cleanup +connection server_1; +set @@global.debug_dbug= @save_primary_dbug; +set @@global.rpl_semi_sync_master_timeout= @save_semisync_timeout; +set @@global.rpl_semi_sync_master_enabled= @save_semisync_master_enabled; +drop table t1; +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +include/stop_slave.inc +set @@global.rpl_semi_sync_slave_enabled= @save_semisync_server_2_enabled; +SET @@GLOBAL.debug_dbug= ""; +SET debug_sync='RESET'; +include/start_slave.inc +connection server_3; +include/sync_with_master_gtid.inc +include/stop_slave.inc +set @@global.rpl_semi_sync_slave_enabled= @save_semisync_server_3_enabled; +SET @@GLOBAL.debug_dbug= ""; +SET debug_sync='RESET'; +include/start_slave.inc +# +# End of MDEV-21322 tests +# include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_show_slave_hosts.cnf b/mysql-test/suite/rpl/t/rpl_show_slave_hosts.cnf index 288f0132fba..43f9f1a294d 100644 --- a/mysql-test/suite/rpl/t/rpl_show_slave_hosts.cnf +++ b/mysql-test/suite/rpl/t/rpl_show_slave_hosts.cnf @@ -2,19 +2,24 @@ [mysqld.1] server_id=1 +log_warnings=9 [mysqld.2] server_id=2 report-host= report-user= +log_slave_updates=1 [mysqld.3] server_id=3 report-host=slave2 slave-net-timeout=5 +log_slave_updates=1 +log_bin=slave2 [ENV] -SLAVE_MYPORT2= @mysqld.3.port -SLAVE_MYSOCK2= @mysqld.3.socket +SERVER_MYPORT_1= @mysqld.1.port +SERVER_MYPORT_2= @mysqld.2.port +SERVER_MYPORT_3= @mysqld.3.port diff --git a/mysql-test/suite/rpl/t/rpl_show_slave_hosts.test b/mysql-test/suite/rpl/t/rpl_show_slave_hosts.test index afac298495f..a63c1662b81 100644 --- a/mysql-test/suite/rpl/t/rpl_show_slave_hosts.test +++ b/mysql-test/suite/rpl/t/rpl_show_slave_hosts.test @@ -9,17 +9,10 @@ # Remove the "Rpl_recovery_rank" column from SHOW SLAVE HOSTS, It is not # implemented. ####################################################################### -source include/master-slave.inc; -connect (slave2,127.0.0.1,root,,test,$SLAVE_MYPORT2,); +--let $rpl_topology= 1->2,1->3 +--source include/rpl_init.inc -connection slave2; -RESET SLAVE; ---replace_result $MASTER_MYPORT MASTER_PORT ---eval CHANGE MASTER TO master_host='127.0.0.1',master_port=$MASTER_MYPORT,master_user='root', master_ssl_verify_server_cert=0 -START SLAVE IO_THREAD; -source include/wait_for_slave_io_to_start.inc; - -connection master; +connection server_1; let $show_statement= SHOW SLAVE HOSTS; let $field= Server_id; # 3 is server_id of slave2. @@ -30,14 +23,13 @@ source include/wait_show_condition.inc; # HOSTS, when that slave is much slower to register due to thread scheduling. let $condition= ='2'; source include/wait_show_condition.inc; ---replace_column 3 'SLAVE_PORT' ---replace_result $SLAVE_MYPORT SLAVE_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT SHOW SLAVE HOSTS; -connection slave2; +connection server_3; --source include/stop_slave_io.inc -connection master; +connection server_1; let $show_statement= SHOW SLAVE HOSTS; let $field= Server_id; # 3 is server_id of slave2. @@ -49,4 +41,792 @@ source include/wait_show_condition.inc; --replace_result $SLAVE_MYPORT SLAVE_PORT SHOW SLAVE HOSTS; + +--echo # +--echo # MDEV-21322: report slave progress to the primary +--echo # + +--echo # +--echo # 21322.0: Test case set-up +--echo # +connection server_1; +set sql_log_bin=0; +call mtr.add_suppression("Got an error reading communication packets"); +call mtr.add_suppression("Semi-sync master failed on net_flush"); +call mtr.add_suppression("Could not read packet:.* vio_errno: 1158"); +call mtr.add_suppression("Could not write packet:.* vio_errno: 1160"); +set sql_log_bin=1; +set @save_primary_dbug= @@global.debug_dbug; +set @save_semisync_timeout= @@global.rpl_semi_sync_master_timeout; +set @save_semisync_master_enabled= @@global.rpl_semi_sync_master_enabled; +create table t1 (a int); + +connection server_2; +set @save_s2_debug= @@GLOBAL.debug_dbug; +set @save_semisync_server_2_enabled= @@global.rpl_semi_sync_master_enabled; +set sql_log_bin=0; +call mtr.add_suppression('Slave I/O: Relay log write failure: could not queue event from master.*'); +call mtr.add_suppression('Slave I/O: Replication event checksum verification failed while reading from network.*'); +call mtr.add_suppression('Replication event checksum verification failed'); +call mtr.add_suppression("Timeout waiting for reply of binlog*"); +call mtr.add_suppression('Found invalid event in binary log'); +call mtr.add_suppression('event read from binlog did not pass crc check'); +call mtr.add_suppression('Event crc check failed! Most likely there is event corruption'); +call mtr.add_suppression('Slave SQL: Error initializing relay log position: I/O error reading event at position .*, error.* 1593'); +call mtr.add_suppression("Semi-sync slave .* reply"); +set sql_log_bin=1; + +connection server_3; +set @save_s3_debug= @@GLOBAL.debug_dbug; +set @save_semisync_server_3_enabled= @@global.rpl_semi_sync_master_enabled; +set sql_log_bin=0; +call mtr.add_suppression('Slave I/O: Relay log write failure: could not queue event from master.*'); +call mtr.add_suppression('Slave I/O: Replication event checksum verification failed while reading from network.*'); +call mtr.add_suppression('Replication event checksum verification failed'); +call mtr.add_suppression("Timeout waiting for reply of binlog*"); +call mtr.add_suppression('Found invalid event in binary log'); +call mtr.add_suppression('event read from binlog did not pass crc check'); +call mtr.add_suppression('Event crc check failed! Most likely there is event corruption'); +call mtr.add_suppression('Slave SQL: Error initializing relay log position: I/O error reading event at position .*, error.* 1593'); +call mtr.add_suppression("Semi-sync slave .* reply"); +set sql_log_bin=1; + +--connection server_1 +--source include/save_master_gtid.inc +--connection server_2 +--source include/sync_with_master_gtid.inc + + +--echo # +--echo # 21322.1: In a fresh replication state with semi-sync disabled, +--echo # the Sync_Status column should reflect an asynchronous replication +--echo # state, and Gtid_State_Sent/Ack should start, and only Gtid_State_Sent +--echo # should update with new transaction. Note only server_2 is currently +--echo # connected. +--echo # +--connection server_1 +--replace_result $SLAVE_MYPORT SLAVE_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW SLAVE HOSTS; +insert into t1 values (1); +--source include/save_master_gtid.inc +--connection server_2 +--source include/sync_with_master_gtid.inc + +--connection server_1 +--echo # Gtid_State_Sent should be updated for new transaction +--replace_result $SLAVE_MYPORT SLAVE_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW SLAVE HOSTS; +--let $master_gtid= `SELECT @@gtid_binlog_pos` +--let $gtid_sent= query_get_value(show slave hosts, Gtid_State_Sent, 1) +if (`SELECT strcmp("$master_gtid","$gtid_sent") != 0`) +{ + --echo # Master gtid_binlog_pos: $master_gtid + --echo # Gtid_State_Sent: $gtid_sent + --die Master did not update Gtid_State_Sent for asynchronous replica +} + + +--echo # +--echo # 21322.2: When only the primary enables semi-sync, Sync_Status should +--echo # still be asynchronous +--echo # +--connection server_1 +set global rpl_semi_sync_master_enabled= 1; +show variables like 'rpl_semi_sync_master_enabled'; +--replace_result $SLAVE_MYPORT SLAVE_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW SLAVE HOSTS; + + +--echo # +--echo # 21322.3: Finalizing the semi-sync connection on server_2 (i.e. by +--echo # enabling it on the slave) should update Sync_Status to semi-sync +--echo # active, as the slave is up-to-date. +--echo # + +--connection server_2 +--source include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +--source include/start_slave.inc + +--connection server_1 +let $status_var= Rpl_semi_sync_master_clients; +let $status_var_value= 1; +source include/wait_for_status_var.inc; + +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW REPLICA HOSTS; + + +--echo # +--echo # 21322.4: After new semi-sync transactions are ACKed, +--echo # Gtid_State_Sent/Ack should match gtid_binlog_pos, and Sync_Status +--echo # should read that semi-sync is active +--echo # +--connection server_1 +--let $nextval= `SELECT max(a)+1 from t1` +--eval insert into t1 values ($nextval) +--source include/save_master_gtid.inc +--connection server_2 +--source include/sync_with_master_gtid.inc + +--connection server_1 +--let $master_gtid= `SELECT @@gtid_binlog_pos` + +--echo # Ensuring master gtid_binlog_pos matches Gtid_State_Sent +--let $gtid_sent= query_get_value(show slave hosts, Gtid_State_Sent, 1) +if (`SELECT strcmp("$master_gtid","$gtid_sent") != 0`) +{ + --echo # Master gtid_binlog_pos: $master_gtid + --echo # Gtid_State_Sent: $gtid_sent + --die Master's gtid_binlog_pos should match Gtid_State_Sent, but doesn't +} + +--echo # Ensuring master gtid_binlog_pos matches Gtid_State_Ack +--let $gtid_ack= query_get_value(show slave hosts, Gtid_State_Ack, 1) +if (`SELECT strcmp("$master_gtid","$gtid_ack") != 0`) +{ + --echo # Master gtid_binlog_pos: $master_gtid + --echo # Gtid_State_Ack: $gtid_ack + --die Master's gtid_binlog_pos should match Gtid_State_Ack, but doesn't +} + +--echo # Ensuring Sync_Status is semi-sync active +--let $sync_status= query_get_value(show slave hosts, Sync_Status, 1) +if (`SELECT strcmp("$sync_status","semi-sync active") != 0`) +{ + --echo # Sync_Status: $sync_status + --die Incorrect value for Sync_Status, should be "semi-sync active" +} + +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW REPLICA HOSTS; + + +--echo # +--echo # 21322.5: When connecting a new slave (server_id 3) which initially has +--echo # semi-sync disabled, SHOW SLAVE HOSTS on the master should show its +--echo # Sync_Status as asynchronous (while server_id 2 is still semi-sync +--echo # active). +--echo # +# Iniital replication state on server_3 is off +connection server_3; +--source include/start_slave.inc +--source include/sync_with_master_gtid.inc + +connection server_1; +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW REPLICA HOSTS; + +--echo # +--echo # 21322.6: Reconnecting server_3 as a semi-sync enabled replica should +--echo # result in a Sync_Status reflecting active semi-sync +--echo # + +connection server_3; +--source include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +--source include/start_slave.inc + +connection server_1; +let $status_var= Rpl_semi_sync_master_clients; +let $status_var_value= 2; +source include/wait_for_status_var.inc; +show status like 'Rpl_semi_sync_master_clients'; +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW REPLICA HOSTS; + + +--echo # +--echo # 21322.7: New transactions on the master should update Gtid_State_Sent +--echo # when sent to the slave, and Gtid_State_Ack once receiving an ACK +--echo # + +--connection server_2 +let $server_2_sent_ack= query_get_value(show status like 'Rpl_semi_sync_slave_send_ack', Value, 1); + +--connection server_1 +SET @@GLOBAL.debug_dbug="+d,pause_ack_thread_on_next_ack"; + +# Write the new event +--connection default +--let $old_binlog_gtid= `SELECT @@gtid_binlog_pos` +--let $nextval= `SELECT max(a)+1 from t1` +--send_eval insert into t1 values ($nextval) + +--connection server_1 +--echo # waiting for pause_ack_reply_to_binlog +SET debug_sync='now WAIT_FOR pause_ack_reply_to_binlog'; +--let $new_binlog_gtid= `SELECT @@gtid_binlog_pos` + +--echo # Waiting for Gtid_State_Sent to reflect latest transaction for all replicas.. +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Sent; +let $condition= LIKE '$new_binlog_gtid'; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + +--echo # Ensuring Gtid_State_Ack is not yet updated (as ACK thread is paused) +--let $gtid_ack= query_get_value(show slave hosts, Gtid_State_Ack, 1) +if (`SELECT strcmp("$old_master_gtid","$gtid_ack") != 0`) +{ + --echo # Master gtid_binlog_pos: $master_gtid + --echo # Gtid_State_Ack: $gtid_ack + --die Gtid_State_Ack should not yet reflect the GTID of the new transaction +} + +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW REPLICA HOSTS; + +--connection server_1 +SET debug_sync='now SIGNAL resume_ack_thread'; + +--connection default +--reap +--connection server_1 + +--echo # Waiting for Gtid_State_Ack to reflect latest transaction for all replicas.. +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Ack; +let $condition= LIKE '$new_binlog_gtid'; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW REPLICA HOSTS; + +--echo # Reset debug state +SET @@GLOBAL.debug_dbug= @save_primary_dbug; +SET debug_sync='RESET'; + + +--echo # +--echo # 21322.8: Holding one replica's ACK (server_2) should result in +--echo # Gtid_State_Ack of server_3 updating to the most recent GTID, while +--echo # server_id 2 has the old GTID. Note that we need to use debug_sync to +--echo # synchronize the ACKs of both server_2 and server_3, so server_3 can't +--echo # ACK the transaction before server_2's binlog dump thread sends the +--echo # transaction (which would negate the need for server_2 to ACK at all, +--echo # resulting in MTR hanging on its expected debug_sync WAIT_FOR point.) +--echo # + +--connection server_2 +--source include/stop_slave.inc +set @@GLOBAL.debug_dbug="+d,synchronize_semisync_slave_reply"; +--source include/start_slave.inc +--connection server_3 +--source include/stop_slave.inc +set @@GLOBAL.debug_dbug="+d,synchronize_semisync_slave_reply"; +--source include/start_slave.inc + +--connection server_1 +--echo # Waiting for master to recognize slave restart.. +let $status_var= Rpl_semi_sync_master_clients; +let $status_var_value= 2; +source include/wait_for_status_var.inc; + +--let $nextval= `SELECT max(a)+1 from t1` +--send_eval insert into t1 values ($nextval) + +--connection server_2 +set debug_sync= "now WAIT_FOR at_slave_reply"; +--connection server_3 +set debug_sync= "now WAIT_FOR at_slave_reply"; + +# New transaction is binlogged at this point, so we can query gtid_binlog_pos +--connection default +--let $binlog_gtid= `SELECT @@gtid_binlog_pos` + +--echo # Ensure Gtid_State_Sent reflects latest transaction ($binlog_gtid) for all replicas.. +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Sent; +let $condition= LIKE '$binlog_gtid'; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + +--connection server_3 +set debug_sync= "now SIGNAL reply_ack_to_master"; + +--connection server_1 +--reap + +--echo # Wait for Gtid_State_Ack to show the latest transaction for server_3.. +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Ack; +let $condition= LIKE '$binlog_gtid'; +let $wait_for_all= 0; +source include/wait_show_condition.inc; + +--echo # Only server_3 should have ACKed the new GTID, server_2 should not due to debug_sync holding off the ACK +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW REPLICA HOSTS; + +--connection server_2 +--echo # Resume slave so it can ACK the transaction +set debug_sync= "now SIGNAL reply_ack_to_master"; + +--echo # Waiting for Gtid_State_Ack to reflect latest transaction for all replicas.. +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Ack; +let $condition= LIKE '$binlog_gtid'; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + +--connection server_1 +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW REPLICA HOSTS; + +--connection server_2 +--source include/stop_slave.inc +set @@GLOBAL.debug_dbug= @save_s2_debug; +SET debug_sync='RESET'; +--source include/start_slave.inc + +--connection server_3 +--source include/stop_slave.inc +set @@GLOBAL.debug_dbug= @save_s3_debug; +SET debug_sync='RESET'; +--source include/start_slave.inc + +--connection server_1 +--source include/save_master_gtid.inc +--connection server_2 +--source include/sync_with_master_gtid.inc +--connection server_3 +--source include/sync_with_master_gtid.inc + + +--echo # +--echo # 21322.9: If a server is behind when connecting to a primary (i.e. the +--echo # primary has newer transactions), the replica Sync_Status should +--echo # present as semi-sync stale and the Gtid_State_Ack should not populate +--echo # until it has reached Semi-Sync Active +--echo # + +--connection server_2 +--source include/stop_slave.inc + +--connection server_1 +--let $nextval= `SELECT max(a)+1 from t1` +--eval insert into t1 values ($nextval) +--source include/save_master_gtid.inc +--let $binlog_gtid= `SELECT @@gtid_binlog_pos` +--echo # Pause dump_thread of server_2 (server_3 won't be affected as it has +--echo # already successfully ACKed the new transaction) +set @@global.debug_dbug= "+d,pause_dump_thread_after_sending_next_full_trx"; + +--connection server_2 +--source include/start_slave.inc + +--connection server_1 +set debug_sync= 'now WAIT_FOR dump_thread_paused'; + +--echo # Ensure Gtid_State_Sent is updated to represent new transaction has +--echo # been sent to both replicas +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Sent; +let $condition= LIKE '$binlog_gtid'; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + +--echo # Ensure Sync_Status is Semi-sync Stale for the debug_sync held dump +--echo # thread, as it hasn't yet got "up-to-date" +let $show_statement= SHOW REPLICA HOSTS; +let $field= Sync_Status; +let $condition= LIKE 'Semi-sync Stale'; +let $wait_for_all= 0; +source include/wait_show_condition.inc; + +set debug_sync= 'now SIGNAL dump_thread_continue'; + +--echo # Ensure Sync_Status will automatically update to Semi-sync Active +--echo # once the last stale transaction has finished sending +let $show_statement= SHOW REPLICA HOSTS; +let $field= Sync_Status; +let $condition= LIKE 'Semi-sync Active'; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + +set @@global.debug_dbug= @save_primary_dbug; +SET debug_sync='RESET'; + +--source include/save_master_gtid.inc +--connection server_2 +--source include/sync_with_master_gtid.inc +--connection server_3 +--source include/sync_with_master_gtid.inc + + +--echo # +--echo # 21322.10a: If one replica errors (i.e. server_2 by injecting +--echo # corrupt_gtid_event with debug_dbug), it shouldn't send its ACK, +--echo # and server_3 does ACK; then Gtid_State_Sent should still reflect the +--echo # new transaction for each replica, but Gtid_State_Ack should only be +--echo # updated by the successful transaction. When the errored replica +--echo # reconnects, it shouldn't ACK the transaction, so its on-reconnect +--echo # Gtid_State_Ack value should be empty, but still have a Sync_Status +--echo # of "Semi-sync Active" when it receives the latest transaction, as it +--echo # will be ready to ACK new transactions going forward. +--echo # +--connection server_2 +--source include/stop_slave.inc +SET @@GLOBAL.debug_dbug= "+d,corrupt_gtid_event"; +--source include/start_slave.inc + +--echo # Set-up server_3 for 10b (so we don't have to restart the slave) +--connection server_3 +--source include/stop_slave.inc +set @@GLOBAL.debug_dbug="+d,synchronize_semisync_slave_reply"; +--source include/start_slave.inc + +--connection server_1 +--echo # Waiting for master to recognize slave restarts.. +let $status_var= Rpl_semi_sync_master_clients; +let $status_var_value= 2; +source include/wait_for_status_var.inc; + +--connection server_1 +--let $nextval= `SELECT max(a)+1 from t1` +--send_eval insert into t1 values ($nextval) + +--echo # Debug_sync is irrelevant to this testcase (10a) but we must do it to +--echo # allow server_3 to ACK now (Note debug_sync is needed for 10b) +--connection server_3 +set debug_sync= "now WAIT_FOR at_slave_reply"; +set debug_sync= "now SIGNAL reply_ack_to_master"; + +--connection server_1 +--reap +--let $binlog_gtid= `SELECT @@gtid_binlog_pos` +--source include/save_master_gtid.inc + +--echo # Waiting for Gtid_State_Sent to reflect latest transaction for all replicas.. +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Sent; +let $condition= LIKE '$binlog_gtid'; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + +--connection server_2 +--let $slave_io_errno= 1595 +--source include/wait_for_slave_io_error.inc +set @@GLOBAL.debug_dbug= @save_s2_debug; + +--connection server_1 +--echo # Only server_3 should ACKed have the new GTID, server_2 should not due to corrupt_queue_event +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW REPLICA HOSTS; + +--connection server_2 +--source include/start_slave.inc +--source include/sync_with_master_gtid.inc + +--connection server_1 +--echo # With replica restarted/synced, its Gtid_State_Ack should be empty with Sync_Status semi-sync active +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW REPLICA HOSTS; + + +--echo # +--echo # 21322.10b: Succeeding the previous 10a test, if server_3 now stalls +--echo # (i.e. using debug_sync), then the previously errored server_2 should +--echo # receive and ACK new transactions as a "lone" replica, and update its +--echo # Gtid_State_* columns appropriately, whereas server_3's Gtid_State_Ack +--echo # column should not be updated. +--echo # + +--let $old_binlog_gtid= `SELECT @@gtid_binlog_pos` +--let $nextval= `SELECT max(a)+1 from t1` +--eval insert into t1 values ($nextval) +--let $new_binlog_gtid= `SELECT @@gtid_binlog_pos` +--source include/save_master_gtid.inc + +--connection server_2 +--source include/sync_with_master_gtid.inc + +--echo # Waiting for Gtid_State_Sent to reflect latest transaction for all replicas.. +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Sent; +let $condition= LIKE '$new_binlog_gtid'; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + +--connection server_3 +set debug_sync= "now WAIT_FOR at_slave_reply"; + +--connection server_1 +--echo # Only server_2 should have ACKed the new GTID ACKed, server_3 should not due to stall +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT +SHOW REPLICA HOSTS; + +--echo # Resume server_3 +--connection server_3 +set debug_sync= "now SIGNAL reply_ack_to_master"; +--source include/sync_with_master_gtid.inc +SET debug_sync='RESET'; + +--source include/stop_slave.inc +set @@GLOBAL.debug_dbug= @save_s3_debug; +SET debug_sync='RESET'; +--source include/start_slave.inc + + +--echo # +--echo # 21322.11: Configuration rpl_semi_sync_master_timeout of 0 should +--echo # have transaction behavior match asynchronous behavior (i.e. trxs don't +--echo # need to wait for ACKs), yet the slave should still send ACKs as a +--echo # normal semi-sync replica, and Gtid_State_Ack should still be updated +--echo # accordingly. +--echo # +--echo # The actual behavior tested in this case is as follows: +--echo # a) Transactions won't await ACKs to complete +--echo # b) Semi-sync remains ON when a transaction completes without an ACK +--echo # c) Gtid_State_Ack is updated accordingly for each replica's ACK (even +--echo # when it is behind). Here, we hold both replicas using DEBUG_SYNC to +--echo # not send their ACKs, meanwhile, we continue creating transactions +--echo # on the primary. +--echo # d) If a "very lagged" replica sends an ACK for a transaction from a +--echo # purged binlog, the Gtid_State_Ack value should be cleared, and +--echo # issue a warning to the user with the slave's last ACKed binlog +--echo # coordinate (i.e. filename and position). +--echo # + +--connection server_1 +set global rpl_semi_sync_master_timeout=0; + +--connection server_2 +--source include/stop_slave.inc +SET @@GLOBAL.DEBUG_DBUG="+d,synchronize_semisync_slave_reply"; +--source include/start_slave.inc +--connection server_3 +--source include/stop_slave.inc +SET @@GLOBAL.DEBUG_DBUG="+d,synchronize_semisync_slave_reply"; +--source include/start_slave.inc + +--connection server_1 +--echo # Waiting for master to recognize slave restarts.. +let $status_var= Rpl_semi_sync_master_clients; +let $status_var_value= 2; +source include/wait_for_status_var.inc; + +--echo # +--echo # 21322.11.a +connection server_1; +--let $yes_tx_orig= query_get_value(show status like 'Rpl_semi_sync_master_yes_tx', Value, 1) +--let $no_tx_orig= query_get_value(show status like 'Rpl_semi_sync_master_no_tx', Value, 1) +--let $nextval= `SELECT max(a)+1 from t1` +--eval insert into t1 values ($nextval) +--let $trx1_gtid= `SELECT @@gtid_binlog_pos` + +--echo # Waiting for Gtid_State_Sent to reflect latest transaction for all replicas.. +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Sent; +let $condition= LIKE '$trx1_gtid'; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + +--connection server_2 +SET debug_sync='now WAIT_FOR at_slave_reply'; +--connection server_3 +SET debug_sync='now WAIT_FOR at_slave_reply'; + +--echo # Gtid_State_Ack should be empty for both replicas (as they were restarted).. +--let $binlog_gtid= `SELECT @@gtid_binlog_pos` +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Ack; +let $condition= = ''; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + +--echo # +--echo # 21322.11.b +--connection server_1 +--echo # Ensuring semi-sync status on primary is correct.. +--let $yes_tx_post_commit= query_get_value(show status like 'Rpl_semi_sync_master_yes_tx', Value, 1) +if ($yes_tx_orig != $yes_tx_post_commit) +{ + --echo # yes_tx original: $yes_tx_orig + --echo # yes_tx after commit: $yes_tx_post_commit + --die Rpl_semi_sync_master_yes_tx should not have changed when Rpl_semi_sync_master_timeout is 0 +} +--let $no_tx_post_commit= query_get_value(show status like 'Rpl_semi_sync_master_no_tx', Value, 1) +if ($no_tx_orig != $no_tx_post_commit) +{ + --echo # no_tx original: $no_tx_orig + --echo # no_tx after commit: $no_tx_post_commit + --die Rpl_semi_sync_master_no_tx should not have changed when Rpl_semi_sync_master_timeout is 0 +} +--let $master_semisync_status= query_get_value(show status like 'Rpl_semi_sync_master_status', Value, 1) +if (`SELECT strcmp('$master_semisync_status','ON') != 0`) +{ + --die Master semi-sync status was disabled after transaction +} + +--echo # +--echo # 21322.11.c +--connection server_1 +--let $nextval= `SELECT max(a)+1 from t1` +--eval insert into t1 values ($nextval) +--let $trx2_gtid= `SELECT @@gtid_binlog_pos` +--source include/save_master_gtid.inc + +--echo # Waiting for Gtid_State_Sent to reflect latest transaction for all replicas.. +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Sent; +let $condition= LIKE '$trx2_gtid'; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + +--connection server_1 +--echo # server_2 and 3 should both show an empty ACK state +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT $trx1_gtid TRX1_GTID $trx2_gtid TRX2_GTID +SHOW REPLICA HOSTS; + +--echo # Let server_2 ACK just the first transaction +--connection server_2 +set debug_sync= "now SIGNAL reply_ack_to_master"; +set debug_sync= "now WAIT_FOR at_slave_reply"; + +--connection server_1 +--echo # Waiting for server_2 Gtid_State_Ack to reflect first transaction +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Ack; +let $condition= LIKE '$trx1_gtid'; +let $wait_for_all= 0; +source include/wait_show_condition.inc; + +--echo # Let server_2 ACK the second transaction +--connection server_2 +set debug_sync= "now SIGNAL reply_ack_to_master"; +--source include/sync_with_master_gtid.inc + +--connection server_1 +--echo # Waiting for server_2 Gtid_State_Ack to reflect second transaction +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Ack; +let $condition= LIKE '$trx2_gtid'; +let $wait_for_all= 0; +source include/wait_show_condition.inc; + +--echo # Let server_3 now ACK the first transaction +--connection server_3 +set debug_sync= "now SIGNAL reply_ack_to_master"; +set debug_sync= "now WAIT_FOR at_slave_reply"; + +--connection server_1 +--echo # Waiting for server_3 Gtid_State_Ack to reflect first transaction +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Ack; +let $condition= LIKE '$trx1_gtid'; +let $wait_for_all= 0; +source include/wait_show_condition.inc; + +--echo # Let server_3 ACK the second transaction +--connection server_3 +set debug_sync= "now SIGNAL reply_ack_to_master"; +--source include/sync_with_master_gtid.inc + +--connection server_1 +--echo # Waiting for Gtid_State_Ack to reflect second transaction for both servers +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Ack; +let $condition= LIKE '$trx2_gtid'; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + + +--echo # +--echo # 21322.11.d +--connection server_1 +--let $trx2_binlog_file= query_get_value(SHOW BINARY LOGS, Log_name, 1) +--let $trx2_binlog_pos= query_get_value(SHOW BINARY LOGS, File_size, 1) +FLUSH LOGS; +--let $nextval= `SELECT max(a)+1 from t1` +--eval insert into t1 values ($nextval) +--let $trx3_gtid= `SELECT @@gtid_binlog_pos` +--source include/save_master_gtid.inc + +--echo # Waiting for Gtid_State_Sent to reflect latest transaction for all replicas.. +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Sent; +let $condition= LIKE '$trx3_gtid'; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + +--connection server_2 +set debug_sync= "now WAIT_FOR at_slave_reply"; +--connection server_3 +set debug_sync= "now WAIT_FOR at_slave_reply"; + +--connection server_1 +--echo # server_2 and 3 should both show ACKed TRX2 (with TRX3 Sent) +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT $trx2_gtid TRX2_GTID $trx3_gtid TRX3_GTID +SHOW REPLICA HOSTS; + +--let $purge_to_binlog= query_get_value(SHOW MASTER STATUS, File, 1) +--let $purge_binlogs_to=$purge_to_binlog +--source include/wait_for_purge.inc + +--echo # Master should warn that the binary log which contains the last ACKed +--echo # binlog coordinates has been purged, and clear Gtid_State_Ack +--enable_warnings +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT $trx2_gtid TRX2_GTID $trx3_gtid TRX3_GTID $trx2_binlog_file TRX2_BINLOG_FILE $trx2_binlog_pos TRX2_BINLOG_POS +SHOW REPLICA HOSTS; + +--echo # Let servers ACK new transaction +--connection server_2 +set debug_sync= "now SIGNAL reply_ack_to_master"; +--connection server_3 +set debug_sync= "now SIGNAL reply_ack_to_master"; + +--connection server_1 +--echo # Waiting for Gtid_State_Ack to reflect latest transaction for all replicas.. +let $show_statement= SHOW REPLICA HOSTS; +let $field= Gtid_State_Ack; +let $condition= LIKE '$trx3_gtid'; +let $wait_for_all= 1; +source include/wait_show_condition.inc; + +--echo # Gtid_State_Ack should now show the latest transaction GTID +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT $trx2_gtid TRX2_GTID $trx3_gtid TRX3_GTID +SHOW REPLICA HOSTS; +--disable_warnings + +--connection server_2 +--source include/sync_with_master_gtid.inc +--connection server_3 +--source include/sync_with_master_gtid.inc + + +--echo # +--echo # MDEV-21322 Cleanup +connection server_1; +set @@global.debug_dbug= @save_primary_dbug; +set @@global.rpl_semi_sync_master_timeout= @save_semisync_timeout; +set @@global.rpl_semi_sync_master_enabled= @save_semisync_master_enabled; + +drop table t1; +--source include/save_master_gtid.inc +--save_master_pos + +connection server_2; +--source include/sync_with_master_gtid.inc +--source include/stop_slave.inc +set @@global.rpl_semi_sync_slave_enabled= @save_semisync_server_2_enabled; +--eval SET @@GLOBAL.debug_dbug= "$save_server_2_dbug" +SET debug_sync='RESET'; +--source include/start_slave.inc + +connection server_3; +--source include/sync_with_master_gtid.inc +--source include/stop_slave.inc +set @@global.rpl_semi_sync_slave_enabled= @save_semisync_server_3_enabled; +--eval SET @@GLOBAL.debug_dbug= "$save_server_3_dbug" +SET debug_sync='RESET'; +--source include/start_slave.inc + +--echo # +--echo # End of MDEV-21322 tests +--echo # + +# End of rpl_show_slave_hosts.test --source include/rpl_end.inc diff --git a/sql/repl_failsafe.cc b/sql/repl_failsafe.cc index d0285b54928..67a12cee80d 100644 --- a/sql/repl_failsafe.cc +++ b/sql/repl_failsafe.cc @@ -37,18 +37,7 @@ #include "rpl_filter.h" #include "log_event.h" #include <mysql.h> - - -struct Slave_info -{ - uint32 server_id; - uint32 master_id; - char host[HOSTNAME_LENGTH*SYSTEM_CHARSET_MBMAXLEN+1]; - char user[USERNAME_LENGTH+1]; - char password[MAX_PASSWORD_LENGTH*SYSTEM_CHARSET_MBMAXLEN+1]; - uint16 port; -}; - +#include "semisync_master.h" Atomic_counter<uint32_t> binlog_dump_thread_count; ulong rpl_status=RPL_NULL; @@ -125,8 +114,11 @@ int THD::register_slave(uchar *packet, size_t packet_length) if (check_access(this, PRIV_COM_REGISTER_SLAVE, any_db.str, NULL,NULL,0,0)) return 1; if (!(si= (Slave_info*)my_malloc(key_memory_SLAVE_INFO, sizeof(Slave_info), - MYF(MY_WME)))) + MYF(MY_WME|MY_ZEROFILL)))) return 1; + memset(si->gtid_state_sent.log_file, '\0', FN_REFLEN); + memset(si->gtid_state_ack.log_file, '\0', FN_REFLEN); + si->sync_status= Slave_info::SYNC_STATE_INITIALIZING; variables.server_id= si->server_id= uint4korr(p); p+= 4; @@ -179,7 +171,10 @@ static my_bool show_slave_hosts_callback(THD *thd, Protocol *protocol) { my_bool res= FALSE; mysql_mutex_lock(&thd->LOCK_thd_data); - if (auto si= thd->slave_info) + String gtid_sent, gtid_ack; + const char *sync_str; + const char *err_msg= NULL; + if (const Slave_info *si= thd->slave_info) { protocol->prepare_for_resend(); protocol->store(si->server_id); @@ -191,6 +186,50 @@ static my_bool show_slave_hosts_callback(THD *thd, Protocol *protocol) } protocol->store((uint32) si->port); protocol->store(si->master_id); + + if (gtid_state_from_binlog_pos(si->gtid_state_sent.log_file, + (uint32) si->gtid_state_sent.log_pos, + &gtid_sent, &err_msg)) + { + gtid_sent.length(0); + DBUG_ASSERT(err_msg); + if (global_system_variables.log_warnings >= 2) + push_warning_printf( + current_thd, Sql_condition::WARN_LEVEL_WARN, + ER_MASTER_CANNOT_RECONSTRUCT_GTID_STATE_FOR_BINLOG_POS, + ER_THD(current_thd, + ER_MASTER_CANNOT_RECONSTRUCT_GTID_STATE_FOR_BINLOG_POS), + si->gtid_state_sent.log_pos, si->gtid_state_sent.log_file, + err_msg); + } + protocol->store(&gtid_sent); + + if (rpl_semi_sync_master_enabled && thd->semi_sync_slave) + { + if (gtid_state_from_binlog_pos(si->gtid_state_ack.log_file, + (uint32) si->gtid_state_ack.log_pos, + &gtid_ack, &err_msg)) + { + gtid_ack.length(0); + DBUG_ASSERT(err_msg); + + if (global_system_variables.log_warnings >= 2) + { + push_warning_printf( + current_thd, Sql_condition::WARN_LEVEL_WARN, + ER_MASTER_CANNOT_RECONSTRUCT_GTID_STATE_FOR_BINLOG_POS, + ER_THD(current_thd, + ER_MASTER_CANNOT_RECONSTRUCT_GTID_STATE_FOR_BINLOG_POS), + si->gtid_state_ack.log_pos, si->gtid_state_ack.log_file, + err_msg); + } + } + } + protocol->store(&gtid_ack); + + sync_str= si->get_sync_status_str(); + protocol->store(sync_str, safe_strlen(sync_str), &my_charset_bin); + res= protocol->write(); } mysql_mutex_unlock(&thd->LOCK_thd_data); @@ -235,6 +274,20 @@ bool show_slave_hosts(THD* thd) Item_return_int(thd, "Master_id", 10, MYSQL_TYPE_LONG), thd->mem_root); + /* Length matches GTID_IO_Pos of SHOW SLAVE STATUS on slave */ + field_list.push_back(new (mem_root) + Item_empty_string(thd, "Gtid_State_Sent", 30), + thd->mem_root); + + field_list.push_back(new (mem_root) + Item_empty_string(thd, "Gtid_State_Ack", 30), + thd->mem_root); + + /* For the length, use the size of the longest possible value */ + field_list.push_back(new (mem_root) Item_empty_string( + thd, "Sync_Status", sizeof("Semi-sync Active")), + thd->mem_root); + if (protocol->send_result_set_metadata(&field_list, Protocol::SEND_NUM_ROWS | Protocol::SEND_EOF)) DBUG_RETURN(TRUE); diff --git a/sql/semisync_master.cc b/sql/semisync_master.cc index fdf2cf21cf1..8b07c25040e 100644 --- a/sql/semisync_master.cc +++ b/sql/semisync_master.cc @@ -53,13 +53,6 @@ ulonglong rpl_semi_sync_master_trx_wait_time = 0; Repl_semi_sync_master repl_semisync_master; Ack_receiver ack_receiver; -/* - structure to save transaction log filename and position -*/ -typedef struct Trans_binlog_info { - my_off_t log_pos; - char log_file[FN_REFLEN]; -} Trans_binlog_info; static int get_wait_time(const struct timespec& start_ts); @@ -591,7 +584,7 @@ void Repl_semi_sync_master::remove_slave() @retval -1 Slave is going down (ok) */ -int Repl_semi_sync_master::report_reply_packet(uint32 server_id, +int Repl_semi_sync_master::report_reply_packet(Slave_info *slave_info, const uchar *packet, ulong packet_len) { @@ -635,12 +628,12 @@ int Repl_semi_sync_master::report_reply_packet(uint32 server_id, DBUG_ASSERT(dirname_length(log_file_name) == 0); - DBUG_PRINT("semisync", ("%s: Got reply(%s, %lu) from server %u", - "Repl_semi_sync_master::report_reply_packet", - log_file_name, (ulong)log_file_pos, server_id)); - + DBUG_PRINT("semisync", + ("%s: Got reply(%s, %lu) from server %u", + "Repl_semi_sync_master::report_reply_packet", log_file_name, + (ulong) log_file_pos, slave_info->server_id)); rpl_semi_sync_master_get_ack++; - report_reply_binlog(server_id, log_file_name, log_file_pos); + report_reply_binlog(slave_info, log_file_name, log_file_pos); DBUG_RETURN(0); l_end: @@ -649,13 +642,13 @@ int Repl_semi_sync_master::report_reply_packet(uint32 server_id, octet2hex(buf, (const unsigned char*) packet, MY_MIN(sizeof(buf)-1, (size_t) packet_len)); sql_print_information("First bytes of the packet from semisync slave " - "server-id %d: %s", server_id, buf); + "server-id %d: %s", slave_info->server_id, buf); } DBUG_RETURN(result); } -int Repl_semi_sync_master::report_reply_binlog(uint32 server_id, +int Repl_semi_sync_master::report_reply_binlog(Slave_info *slave_info, const char *log_file_name, my_off_t log_file_pos) { @@ -675,7 +668,7 @@ int Repl_semi_sync_master::report_reply_binlog(uint32 server_id, if (!is_on()) /* We check to see whether we can switch semi-sync ON. */ - try_switch_on(server_id, log_file_name, log_file_pos); + try_switch_on(slave_info->server_id, log_file_name, log_file_pos); /* The position should increase monotonically, if there is only one * thread sending the binlog to the slave. @@ -719,6 +712,19 @@ int Repl_semi_sync_master::report_reply_binlog(uint32 server_id, DBUG_PRINT("semisync", ("%s: Got reply at (%s, %lu)", "Repl_semi_sync_master::report_reply_binlog", log_file_name, (ulong)log_file_pos)); + goto update_gtid_state_ack; + } + else if (rpl_semi_sync_master_clients > 1 && + Active_tranx::compare(slave_info->gtid_state_ack.log_file, + slave_info->gtid_state_ack.log_pos, + m_reply_file_name, m_reply_file_pos)) + { +update_gtid_state_ack: + /* + Each slave should still maintain its Gtid_state_ack + */ + strncpy(slave_info->gtid_state_ack.log_file, log_file_name, strlen(log_file_name)); + slave_info->gtid_state_ack.log_pos= log_file_pos; } @@ -828,7 +834,7 @@ int Repl_semi_sync_master::dump_start(THD* thd, } add_slave(); - report_reply_binlog(thd->variables.server_id, + report_reply_binlog(thd->slave_info, log_file + dirname_length(log_file), log_pos); sql_print_information("Start semi-sync binlog_dump to slave " "(server_id: %ld), pos(%s, %lu)", @@ -858,6 +864,16 @@ int Repl_semi_sync_master::commit_trx(const char *trx_wait_binlog_name, bool success= 0; DBUG_ENTER("Repl_semi_sync_master::commit_trx"); + /* + If the semi-sync timeout is set to 0, we effectively are configured for + asynchronous replication; except we still want to request/receive ACKs from + slaves so we can monitor replication status via SHOW SLAVE HOSTS columns + Gtid_State_Sent and Gtid_State_Ack. Thereby, we should quit now before + updating rpl_semi_sync_master_(no/yes)_transactions. + */ + if (!m_wait_timeout) + DBUG_RETURN(0); + if (!rpl_semi_sync_master_clients && !rpl_semi_sync_master_wait_no_slave) { rpl_semi_sync_master_no_transactions++; @@ -1235,6 +1251,14 @@ int Repl_semi_sync_master::update_sync_header(THD* thd, unsigned char *packet, *need_sync= sync; l_end: + if (is_on()) + { + thd->slave_info->sync_status= + sync ? thd->slave_info->sync_status= + Slave_info::SYNC_STATE_SEMI_SYNC_ACTIVE + : thd->slave_info->sync_status= + Slave_info::SYNC_STATE_SEMI_SYNC_STALE; + } unlock(); /* diff --git a/sql/semisync_master.h b/sql/semisync_master.h index 3978d21a61d..cfb5e3d150b 100644 --- a/sql/semisync_master.h +++ b/sql/semisync_master.h @@ -28,6 +28,91 @@ extern PSI_mutex_key key_LOCK_binlog; extern PSI_cond_key key_COND_binlog_send; #endif + +/* + structure to save transaction log filename and position +*/ +typedef struct Trans_binlog_info { + my_off_t log_pos; + char log_file[FN_REFLEN]; +} Trans_binlog_info; + + +struct Slave_info +{ +public: + enum synchronization_status { + /* + Binlog dump thread is initializing, we don't yet know the synchronization + status + */ + SYNC_STATE_INITIALIZING, + + /* + Slave is asynchronous, so Gtid_State_Ack will not be updated + */ + SYNC_STATE_ASYNCHRONOUS, + + /* + Slave is configured for semi-sync, but connected with an old state, and + is catching up now + */ + SYNC_STATE_SEMI_SYNC_STALE, + + /* + Slave is configured for semi-sync, and is readily ACKing new transactions + */ + SYNC_STATE_SEMI_SYNC_ACTIVE + }; + + uint32 server_id; + uint32 master_id; + char host[HOSTNAME_LENGTH*SYSTEM_CHARSET_MBMAXLEN+1]; + char user[USERNAME_LENGTH+1]; + char password[MAX_PASSWORD_LENGTH*SYSTEM_CHARSET_MBMAXLEN+1]; + uint16 port; + + /* + Binlog file:pos of the last transaction sent to this replica. Used to infer + Gtid_State_Sent in SHOW REPLICA HOSTS. Used for both asynchronous and + semi-sync connections. + */ + Trans_binlog_info gtid_state_sent; + + /* + If replica is configured for semi-sync, the binlog file:pos of the last + transaction ACKed by this replica. Used to infer Gtid_State_Ack in + SHOW REPLICA HOSTS. + */ + Trans_binlog_info gtid_state_ack; + + /* + Sync_Status of SHOW REPLICA HOSTS. + */ + synchronization_status sync_status; + + const char *get_sync_status_str() const + { + const char *ret; + switch (sync_status) + { + case SYNC_STATE_INITIALIZING: + ret= "Initializing"; + break; + case SYNC_STATE_ASYNCHRONOUS: + ret= "Asynchronous"; + break; + case SYNC_STATE_SEMI_SYNC_STALE: + ret= "Semi-sync Stale"; + break; + default: + ret= "Semi-sync Active"; + } + return ret; + } +}; + + struct Tranx_node { char log_name[FN_REFLEN]; my_off_t log_pos; @@ -561,14 +646,14 @@ class Repl_semi_sync_master void remove_slave(); /* It parses a reply packet and call report_reply_binlog to handle it. */ - int report_reply_packet(uint32 server_id, const uchar *packet, - ulong packet_len); + int report_reply_packet(Slave_info *slave_info, const uchar *packet, + ulong packet_len); /* In semi-sync replication, reports up to which binlog position we have * received replies from the slave indicating that it already get the events. * * Input: - * server_id - (IN) master server id number + * slave_info - (IN) info of the slave which sent the ACK * log_file_name - (IN) binlog file name * end_offset - (IN) the offset in the binlog file up to which we have * the replies from the slave @@ -576,7 +661,7 @@ class Repl_semi_sync_master * Return: * 0: success; non-zero: error */ - int report_reply_binlog(uint32 server_id, + int report_reply_binlog(Slave_info *slave_info, const char* log_file_name, my_off_t end_offset); diff --git a/sql/semisync_master_ack_receiver.cc b/sql/semisync_master_ack_receiver.cc index 29fa5fd5328..ba2ea0c9762 100644 --- a/sql/semisync_master_ack_receiver.cc +++ b/sql/semisync_master_ack_receiver.cc @@ -16,6 +16,7 @@ #include <my_global.h> #include "semisync_master.h" #include "semisync_master_ack_receiver.h" +#include "debug_sync.h" #ifdef HAVE_PSI_MUTEX_INTERFACE extern PSI_mutex_key key_LOCK_ack_receiver; @@ -353,7 +354,20 @@ void Ack_receiver::run() if (likely(len != packet_error)) { int res; - res= repl_semisync_master.report_reply_packet(slave->server_id(), +#ifdef ENABLED_DEBUG_SYNC + /* + A (+d,pause_ack_thread_on_next_ack)-test is supposed to + be run to check `Gtid_state_ack` in show replica hosts + for cases where there are multiple active replicas. + */ + DBUG_EXECUTE_IF("pause_ack_thread_on_next_ack", + { + const char act[]= "now SIGNAL pause_ack_reply_to_binlog WAIT_FOR resume_ack_thread"; + DBUG_ASSERT(!debug_sync_set_action(thd, STRING_WITH_LEN(act))); + DBUG_SET("-d,pause_ack_thread_on_next_ack"); + };); +#endif + res= repl_semisync_master.report_reply_packet(slave->thd->slave_info, net.read_pos, len); if (unlikely(res < 0)) { diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt index 22d0f4d3392..a428c1c63f8 100644 --- a/sql/share/errmsg-utf8.txt +++ b/sql/share/errmsg-utf8.txt @@ -12280,3 +12280,5 @@ ER_SEQUENCE_TABLE_CANNOT_HAVE_ANY_CONSTRAINTS eng "Sequence tables cannot have any constraints" ER_SEQUENCE_TABLE_ORDER_BY eng "ORDER BY" +ER_MASTER_CANNOT_RECONSTRUCT_GTID_STATE_FOR_BINLOG_POS + eng "Error constructing GTID state for binlog position %u in file '%s': %s" \ No newline at end of file diff --git a/sql/slave.cc b/sql/slave.cc index 54e70bc7385..8b6d09d28ac 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -5121,6 +5121,13 @@ Stopping slave I/O thread due to out-of-memory error from master"); { DBUG_EXECUTE_IF("simulate_delay_semisync_slave_reply", my_sleep(800000);); +#ifdef ENABLED_DEBUG_SYNC + DBUG_EXECUTE_IF("synchronize_semisync_slave_reply", + { + const char act[]= "now SIGNAL at_slave_reply WAIT_FOR reply_ack_to_master"; + DBUG_ASSERT(!debug_sync_set_action(thd, STRING_WITH_LEN(act))); + };); +#endif if (repl_semisync_slave.slave_reply(mi)) { /* @@ -6120,19 +6127,33 @@ static int queue_event(Master_info* mi, const uchar *buf, ulong event_len) // will have to refine the clause. DBUG_ASSERT(mi->rli.relay_log.relay_log_checksum_alg != BINLOG_CHECKSUM_ALG_UNDEF); - - // Emulate the network corruption - DBUG_EXECUTE_IF("corrupt_queue_event", - if (buf[EVENT_TYPE_OFFSET] != FORMAT_DESCRIPTION_EVENT) - { - uchar *debug_event_buf_c= const_cast<uchar*>(buf); - int debug_cor_pos = rand() % (event_len - BINLOG_CHECKSUM_LEN); - debug_event_buf_c[debug_cor_pos] =~ debug_event_buf_c[debug_cor_pos]; - DBUG_PRINT("info", ("Corrupt the event at queue_event: byte on position %d", debug_cor_pos)); - DBUG_SET("-d,corrupt_queue_event"); - } - ); - + +#ifndef DBUG_OFF + { + const char *dbug_unset; + // Emulate the network corruption + DBUG_EXECUTE_IF( + "corrupt_gtid_event", + if (buf[EVENT_TYPE_OFFSET] == GTID_EVENT) { + dbug_unset= "-d,corrupt_gtid_event"; + goto corrupt_event; + }); + DBUG_EXECUTE_IF( + "corrupt_queue_event", + if (buf[EVENT_TYPE_OFFSET] != FORMAT_DESCRIPTION_EVENT) { + dbug_unset= "-d,corrupt_queue_event"; + corrupt_event: + uchar *debug_event_buf_c= const_cast<uchar *>(buf); + int debug_cor_pos= rand() % (event_len - BINLOG_CHECKSUM_LEN); + debug_event_buf_c[debug_cor_pos]= ~debug_event_buf_c[debug_cor_pos]; + DBUG_PRINT("info", + ("Corrupt the event at queue_event: byte on position %d", + debug_cor_pos)); + DBUG_SET(dbug_unset); + }); + } +#endif + if (event_checksum_test((uchar*) buf, event_len, checksum_alg)) { error= ER_NETWORK_READ_EVENT_CHECKSUM_FAILURE; diff --git a/sql/sql_repl.cc b/sql/sql_repl.cc index 8ca252c37c6..18a52dc07ad 100644 --- a/sql/sql_repl.cc +++ b/sql/sql_repl.cc @@ -1830,18 +1830,26 @@ gtid_state_from_pos(const char *name, uint32 offset, return errormsg; } - -int -gtid_state_from_binlog_pos(const char *in_name, uint32 pos, String *out_str) +int gtid_state_from_binlog_pos(const char *in_name, uint32 pos, + String *out_str, const char **out_err) { slave_connection_state gtid_state; const char *lookup_name; char name_buf[FN_REFLEN]; LOG_INFO linfo; + const char *dummy_err; + const char **err_save; + int find_err= 0; + + if (out_err) + err_save= out_err; + else + err_save= &dummy_err; if (!mysql_bin_log.is_open()) { my_error(ER_NO_BINARY_LOGGING, MYF(0)); + *err_save= "Binary logging is disabled."; return 1; } @@ -1853,15 +1861,27 @@ gtid_state_from_binlog_pos(const char *in_name, uint32 pos, String *out_str) else lookup_name= NULL; linfo.index_file_offset= 0; - if (mysql_bin_log.find_log_pos(&linfo, lookup_name, 1)) + if ((find_err= mysql_bin_log.find_log_pos(&linfo, lookup_name, 1))) + { + if (find_err == LOG_INFO_EOF) + *err_save= "Could not find binary log file. Probably the slave state is " + "too old and required binlog files have been purged."; + else + *err_save= "Error reading index file."; return 1; + } if (pos < 4) pos= 4; - if (gtid_state_from_pos(linfo.log_file_name, pos, &gtid_state) || + if ((*err_save= + gtid_state_from_pos(linfo.log_file_name, pos, &gtid_state)) || gtid_state.to_string(out_str)) + { + if (!*err_save) + *err_save= "Failed converting GTID state to string representation."; return 1; + } return 0; } @@ -2272,6 +2292,30 @@ send_event_to_slave(binlog_send_info *info, Log_event_type event_type, return "Failed to run hook 'after_send_event'"; } + if (info->thd->slave_info) + { + strncpy(info->thd->slave_info->gtid_state_sent.log_file, + info->log_file_name + info->dirlen, + strlen(info->log_file_name) - info->dirlen); + info->thd->slave_info->gtid_state_sent.log_pos= pos; + } + +#ifdef ENABLED_DEBUG_SYNC + DBUG_EXECUTE_IF("pause_dump_thread_after_sending_next_full_trx", { + if (event_type == XID_EVENT || + (event_type == QUERY_EVENT && + Query_log_event::peek_is_commit_rollback( + (uchar *) packet->ptr() + ev_offset, len - ev_offset, + current_checksum_alg))) + { + DBUG_ASSERT(!debug_sync_set_action( + info->thd, STRING_WITH_LEN("now SIGNAL dump_thread_paused " + "WAIT_FOR dump_thread_continue"))); + DBUG_SET("-d,pause_dump_thread_after_sending_next_full_trx"); + } + }); +#endif + return NULL; /* Success */ } @@ -2753,6 +2797,10 @@ static int wait_new_events(binlog_send_info *info, /* in */ break; } + if (info->thd->semi_sync_slave) + info->thd->slave_info->sync_status= + Slave_info::SYNC_STATE_SEMI_SYNC_ACTIVE; + if (info->heartbeat_period) { struct timespec ts; @@ -3087,6 +3135,14 @@ void mysql_binlog_send(THD* thd, char* log_ident, my_off_t pos, /* Check if the dump thread is created by a slave with semisync enabled. */ thd->semi_sync_slave = is_semi_sync_slave(); + /* + If the slave is not set up for a semi-sync connection, we can tag it + immediately as asynchronous. Otherwise, we need to wait and see if the + replica is up-to-date or not to mark semi-sync active vs stale. + */ + if (thd->slave_info && !thd->semi_sync_slave) + thd->slave_info->sync_status= Slave_info::SYNC_STATE_ASYNCHRONOUS; + DBUG_ASSERT(pos == linfo.pos); if (repl_semisync_master.dump_start(thd, linfo.log_file_name, linfo.pos)) diff --git a/sql/sql_repl.h b/sql/sql_repl.h index 51b6a599d5f..5e3ccc9df45 100644 --- a/sql/sql_repl.h +++ b/sql/sql_repl.h @@ -66,7 +66,8 @@ void rpl_init_gtid_slave_state(); void rpl_deinit_gtid_slave_state(); void rpl_init_gtid_waiting(); void rpl_deinit_gtid_waiting(); -int gtid_state_from_binlog_pos(const char *name, uint32 pos, String *out_str); +int gtid_state_from_binlog_pos(const char *name, uint32 pos, String *out_str, + const char **out_err= NULL); int rpl_append_gtid_state(String *dest, bool use_binlog); int rpl_load_gtid_state(slave_connection_state *state, bool use_binlog); bool rpl_gtid_pos_check(THD *thd, char *str, size_t len); -- 2.30.2

1 0

[PATCH 1/2] MDEV-33798: ROW base optimistic deadlock with concurrent writes on same table
by Kristian Nielsen 02 May '24

02 May '24

One case is conflicting transactions T1 and T2 with different domain id, in optimistic parallel replication in non-GTID mode. Then T2 will wait_for_prior_commit on T1; and if T1 got a row lock wait on T2 it would hang, as different domains caused the deadlock kill to be skipped in thd_rpl_deadlock_check(). More generally, if we have transactions T1 and T2 in one domain/master connection, and independently transactions U1 and U2 in another, then we can still deadlock like this: T1 row low wait on U2 U2 wait_for_prior_commit on U1 U1 row lock wait on T2 T2 wait_for_prior_commit on T1 This commit enforces the deadlock kill in these cases. If the waited-for transaction is speculatively applied, then it will be deadlock killed in case of a conflict, even if the two transactions are in different domains or master connections. Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> --- mysql-test/suite/rpl/r/rpl_mdev33798.result | 143 +++++++++++++++ mysql-test/suite/rpl/t/rpl_mdev33798.cnf | 17 ++ mysql-test/suite/rpl/t/rpl_mdev33798.test | 182 ++++++++++++++++++++ sql/sql_class.cc | 42 ++++- 4 files changed, 376 insertions(+), 8 deletions(-) create mode 100644 mysql-test/suite/rpl/r/rpl_mdev33798.result create mode 100644 mysql-test/suite/rpl/t/rpl_mdev33798.cnf create mode 100644 mysql-test/suite/rpl/t/rpl_mdev33798.test diff --git a/mysql-test/suite/rpl/r/rpl_mdev33798.result b/mysql-test/suite/rpl/r/rpl_mdev33798.result new file mode 100644 index 00000000000..8796e948546 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_mdev33798.result @@ -0,0 +1,143 @@ +include/rpl_init.inc [topology=1->2,1->3] +connect server_2b,127.0.0.1,root,,,$SERVER_MYPORT_2; +connection server_2; +SET @old_parallel_threads=@@GLOBAL.slave_parallel_threads; +SET @old_parallel_mode= @@GLOBAL.slave_parallel_mode; +SET @old_timeout= @@GLOBAL.lock_wait_timeout; +SET @old_innodb_timeout= @@GLOBAL.innodb_lock_wait_timeout; +include/stop_slave.inc +SET GLOBAL slave_parallel_threads=5; +set global slave_parallel_mode= aggressive; +SET GLOBAL lock_wait_timeout= 86400; +SET GLOBAL innodb_lock_wait_timeout= 86400; +SET STATEMENT sql_log_bin=0 FOR ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +include/start_slave.inc +connection server_1; +CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 0), (2, 0), (3, 0), (4, 0), (5, 0), (6, 0), (7, 0), (8, 0); +connection server_2; +include/stop_slave.inc +connection server_2b; +BEGIN; +SELECT * FROM t1 WHERE a=1 FOR UPDATE; +a b +1 0 +SELECT * FROM t1 WHERE a=5 FOR UPDATE; +a b +5 0 +connection server_1; +SET SESSION gtid_domain_id= 1; +BEGIN; +UPDATE t1 SET b=1 WHERE a=1; +UPDATE t1 SET b=1 WHERE a=7; +COMMIT; +UPDATE t1 SET b=2 WHERE a=3; +SET SESSION gtid_domain_id=2; +BEGIN; +UPDATE t1 SET b=3 WHERE a=5; +UPDATE t1 SET b=3 WHERE a=3; +COMMIT; +UPDATE t1 SET b=4 WHERE a=7; +SET SESSION gtid_domain_id= 0; +include/save_master_gtid.inc +connection server_2; +include/start_slave.inc +connection server_2b; +ROLLBACK; +connection server_2; +include/sync_with_master_gtid.inc +SELECT a, ( +(a=1 AND b=1) OR +(a=3 AND (b=2 OR b=3)) OR +(a=5 AND b=3) OR +(a=7 AND (b=1 OR b=4)) OR +((a MOD 2)=0 AND b=0)) AS `ok` + FROM t1 +ORDER BY a; +a ok +1 1 +2 1 +3 1 +4 1 +5 1 +6 1 +7 1 +8 1 +connection server_3; +include/sync_with_master_gtid.inc +include/stop_slave.inc +connection server_2; +include/stop_slave.inc +CHANGE MASTER 'm2' to master_port=MYPORT_3 , master_host='127.0.0.1', master_user='root', master_use_gtid=slave_pos; +connection server_1; +SET SESSION gtid_domain_id= 1; +BEGIN; +UPDATE t1 SET b=11 WHERE a=1; +UPDATE t1 SET b=11 WHERE a=7; +COMMIT; +UPDATE t1 SET b=12 WHERE a=3; +SET SESSION gtid_domain_id= 1; +connection server_3; +SET SESSION gtid_domain_id=3; +BEGIN; +UPDATE t1 SET b=13 WHERE a=5; +UPDATE t1 SET b=13 WHERE a=3; +COMMIT; +UPDATE t1 SET b=14 WHERE a=7; +include/save_master_gtid.inc +connection server_2b; +BEGIN; +SELECT * FROM t1 WHERE a=1 FOR UPDATE; +a b +1 1 +SELECT * FROM t1 WHERE a=5 FOR UPDATE; +a b +5 3 +START ALL SLAVES; +Warnings: +Note 1937 SLAVE 'm2' started +Note 1937 SLAVE '' started +connection server_2b; +ROLLBACK; +connection server_1; +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +connection server_3; +include/save_master_gtid.inc +connection server_2; +include/sync_with_master_gtid.inc +SELECT a, ( +(a=1 AND b=11) OR +(a=3 AND (b=12 OR b=13)) OR +(a=5 AND b=13) OR +(a=7 AND (b=11 OR b=14)) OR +((a MOD 2)=0 AND b=0)) AS `ok` + FROM t1 +ORDER BY a; +a ok +1 1 +2 1 +3 1 +4 1 +5 1 +6 1 +7 1 +8 1 +SET default_master_connection = 'm2'; +include/stop_slave.inc +RESET SLAVE 'm2' ALL; +SET default_master_connection = ''; +connection server_3; +include/start_slave.inc +disconnect server_2b; +connection server_1; +DROP TABLE t1; +connection server_2; +include/stop_slave.inc +SET GLOBAL slave_parallel_threads=@old_parallel_threads; +set global slave_parallel_mode= @old_parallel_mode; +SET GLOBAL lock_wait_timeout= @old_timeout; +SET GLOBAL innodb_lock_wait_timeout= @old_innodb_timeout; +include/start_slave.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_mdev33798.cnf b/mysql-test/suite/rpl/t/rpl_mdev33798.cnf new file mode 100644 index 00000000000..8e5125ea6ca --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_mdev33798.cnf @@ -0,0 +1,17 @@ +!include suite/rpl/my.cnf + +[mysqld.1] +log-slave-updates +loose-innodb + +[mysqld.2] +log-slave-updates +loose-innodb + +[mysqld.3] +log-slave-updates +loose-innodb + +[ENV] +SERVER_MYPORT_3= @mysqld.3.port +SERVER_MYSOCK_3= @mysqld.3.socket diff --git a/mysql-test/suite/rpl/t/rpl_mdev33798.test b/mysql-test/suite/rpl/t/rpl_mdev33798.test new file mode 100644 index 00000000000..1448ed91133 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_mdev33798.test @@ -0,0 +1,182 @@ +--source include/have_innodb.inc +--source include/have_log_bin.inc +--let $rpl_topology=1->2,1->3 +--source include/rpl_init.inc +--connect (server_2b,127.0.0.1,root,,,$SERVER_MYPORT_2) + +--connection server_2 +SET @old_parallel_threads=@@GLOBAL.slave_parallel_threads; +SET @old_parallel_mode= @@GLOBAL.slave_parallel_mode; +SET @old_timeout= @@GLOBAL.lock_wait_timeout; +SET @old_innodb_timeout= @@GLOBAL.innodb_lock_wait_timeout; +--source include/stop_slave.inc +SET GLOBAL slave_parallel_threads=5; +set global slave_parallel_mode= aggressive; +# High timeout so we get replication sync error and test failure if the +# conflict handling is insufficient and lock wait timeout occurs. +SET GLOBAL lock_wait_timeout= 86400; +SET GLOBAL innodb_lock_wait_timeout= 86400; +SET STATEMENT sql_log_bin=0 FOR ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +--source include/start_slave.inc + +--connection server_1 +CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 0), (2, 0), (3, 0), (4, 0), (5, 0), (6, 0), (7, 0), (8, 0); +--save_master_pos + +--connection server_2 +--sync_with_master +--source include/stop_slave.inc + +# Test the following scenario: +# +# Transactions T1, T2 in domain 1, U1, U2 in domain 2. +# Wait cycle T1->U2->U1->T2->T1 as follows: +# T1 row lock wait on U2 +# U2 wait_for_prior_commit on U1 +# U1 row lock wait on T2 +# T2 wait_for_prior_commit on T1 +# +# Test that the wait cycle is broken correctly with deadlock kill. + +--connection server_2b +# Temporarily block T1 and U1. +BEGIN; +SELECT * FROM t1 WHERE a=1 FOR UPDATE; +SELECT * FROM t1 WHERE a=5 FOR UPDATE; + +--connection server_1 + +SET SESSION gtid_domain_id= 1; +# T1 in domain 1 +BEGIN; +UPDATE t1 SET b=1 WHERE a=1; +UPDATE t1 SET b=1 WHERE a=7; +COMMIT; +# T2 in domain 1 +UPDATE t1 SET b=2 WHERE a=3; + +SET SESSION gtid_domain_id=2; +# U1 in domain 2 +BEGIN; +UPDATE t1 SET b=3 WHERE a=5; +UPDATE t1 SET b=3 WHERE a=3; +COMMIT; +# U2 in domain 2 +UPDATE t1 SET b=4 WHERE a=7; +SET SESSION gtid_domain_id= 0; +--source include/save_master_gtid.inc + +--connection server_2 +--source include/start_slave.inc +# Wait until T2, U2 are holding the row locks. +--let $wait_condition= SELECT COUNT(*)=2 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE state LIKE '%Waiting for prior transaction to commit%' +--source include/wait_condition.inc + +# Then let T1, U1 continue to conflict on the row locks, and check that +# replication correctly handles the conflict. +--connection server_2b +ROLLBACK; + +--connection server_2 +--source include/sync_with_master_gtid.inc + +# Allow either domain to "win" on the conflicting updates. +SELECT a, ( + (a=1 AND b=1) OR + (a=3 AND (b=2 OR b=3)) OR + (a=5 AND b=3) OR + (a=7 AND (b=1 OR b=4)) OR + ((a MOD 2)=0 AND b=0)) AS `ok` + FROM t1 + ORDER BY a; + +# Now try the same thing with multi-source replication. + +# Make server_3 a second master +--connection server_3 +--source include/sync_with_master_gtid.inc +--source include/stop_slave.inc + +--connection server_2 +--source include/stop_slave.inc +--replace_result $SERVER_MYPORT_3 MYPORT_3 +eval CHANGE MASTER 'm2' to master_port=$SERVER_MYPORT_3 , master_host='127.0.0.1', master_user='root', master_use_gtid=slave_pos; + +--connection server_1 + +SET SESSION gtid_domain_id= 1; +# T1 in domain 1 +BEGIN; +UPDATE t1 SET b=11 WHERE a=1; +UPDATE t1 SET b=11 WHERE a=7; +COMMIT; +# T2 in domain 1 +UPDATE t1 SET b=12 WHERE a=3; +SET SESSION gtid_domain_id= 1; + +--connection server_3 +SET SESSION gtid_domain_id=3; +# U1 in domain 3 +BEGIN; +UPDATE t1 SET b=13 WHERE a=5; +UPDATE t1 SET b=13 WHERE a=3; +COMMIT; +# U2 in domain 3 +UPDATE t1 SET b=14 WHERE a=7; +--source include/save_master_gtid.inc + +--connection server_2b +# Temporarily block T1 and U1. +BEGIN; +SELECT * FROM t1 WHERE a=1 FOR UPDATE; +SELECT * FROM t1 WHERE a=5 FOR UPDATE; + +START ALL SLAVES; +# Wait until T2, U2 are holding the row locks. +--let $wait_condition= SELECT COUNT(*)=2 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE state LIKE '%Waiting for prior transaction to commit%' +--source include/wait_condition.inc + +--connection server_2b +ROLLBACK; + +--connection server_1 +--source include/save_master_gtid.inc +--connection server_2 +--source include/sync_with_master_gtid.inc +--connection server_3 +--source include/save_master_gtid.inc +--connection server_2 +--source include/sync_with_master_gtid.inc + +SELECT a, ( + (a=1 AND b=11) OR + (a=3 AND (b=12 OR b=13)) OR + (a=5 AND b=13) OR + (a=7 AND (b=11 OR b=14)) OR + ((a MOD 2)=0 AND b=0)) AS `ok` + FROM t1 + ORDER BY a; + +SET default_master_connection = 'm2'; +--source include/stop_slave.inc +RESET SLAVE 'm2' ALL; +SET default_master_connection = ''; + +--connection server_3 +--source include/start_slave.inc + +# Cleanup + +--disconnect server_2b +--connection server_1 +DROP TABLE t1; +--connection server_2 +--source include/stop_slave.inc +SET GLOBAL slave_parallel_threads=@old_parallel_threads; +set global slave_parallel_mode= @old_parallel_mode; +SET GLOBAL lock_wait_timeout= @old_timeout; +SET GLOBAL innodb_lock_wait_timeout= @old_innodb_timeout; +--source include/start_slave.inc + +--source include/rpl_end.inc diff --git a/sql/sql_class.cc b/sql/sql_class.cc index 0d39f359a09..6026e1e13bb 100644 --- a/sql/sql_class.cc +++ b/sql/sql_class.cc @@ -5384,14 +5384,40 @@ thd_rpl_deadlock_check(MYSQL_THD thd, MYSQL_THD other_thd) return 0; if (!rgi->is_parallel_exec) return 0; - if (rgi->rli != other_rgi->rli) - return 0; - if (!rgi->gtid_sub_id || !other_rgi->gtid_sub_id) - return 0; - if (rgi->current_gtid.domain_id != other_rgi->current_gtid.domain_id) - return 0; - if (rgi->gtid_sub_id > other_rgi->gtid_sub_id) - return 0; + if (rgi->rli == other_rgi->rli) + { + /* + Within the same master connection, we can compare transaction order on + the GTID sub_id, and rollback the later transaction to allow the earlier + transaction to commit first. + */ + if (!rgi->gtid_sub_id || !other_rgi->gtid_sub_id) + return 0; + if (rgi->current_gtid.domain_id != other_rgi->current_gtid.domain_id && + other_rgi->speculation != rpl_group_info::SPECULATE_OPTIMISTIC) + return 0; + if (rgi->gtid_sub_id > other_rgi->gtid_sub_id) + return 0; + } + else + { + /* + Lock conflicts between different master connection should usually not + occur, but could still happen if user is running some special setup that + tolerates conflicting updates (or in case of user error). We do not have a + pre-defined ordering of transactions in this case, but we still need to + handle conflicts in _some_ way to avoid undetected deadlocks and hangs. + + We do this by rolling back and retrying any transaction that is being + _optimistically_ applied. This can be overly conservative in some cases, + but should be fine as conflicts between different master connections are + not expected to be common. And it ensures that we won't end up in a + deadlock and hang due to a transaction doing wait_for_prior_commit while + holding locks that block something in another master connection. + */ + if (other_rgi->speculation != rpl_group_info::SPECULATE_OPTIMISTIC) + return 0; + } if (rgi->finish_event_group_called || other_rgi->finish_event_group_called) { /* -- 2.30.2

1 0

[PATCH] MDEV-33602: Sporadic test failure in rpl.rpl_gtid_stop_start
by Kristian Nielsen 23 Apr '24

23 Apr '24

The test could fail with a duplicate key error because switching to non-GTID mode could start at the wrong old-style position. The position could be wrong when the previous GTID connect was stopped before receiving the fake GTID list event which gives the old-style position corresponding to the GTID connected position. Work-around by injecting an extra event and syncing the slave before switching to non-GTID mode. Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> --- .../suite/rpl/r/rpl_gtid_stop_start.result | 4 ++++ mysql-test/suite/rpl/t/rpl_gtid_stop_start.test | 16 ++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result b/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result index ae0050c353a..e8633cd45bb 100644 --- a/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result +++ b/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result @@ -111,6 +111,10 @@ a 6 7 *** MDEV-4486: Allow to start old-style replication even if mysql.gtid_slave_pos is unavailable +connection server_1; +INSERT INTO t1 VALUES (8); +DELETE FROM t1 WHERE a=8; +connection server_2; connection server_2; include/stop_slave.inc CHANGE MASTER TO master_use_gtid= no; diff --git a/mysql-test/suite/rpl/t/rpl_gtid_stop_start.test b/mysql-test/suite/rpl/t/rpl_gtid_stop_start.test index b5ff294908b..032ebb77d1e 100644 --- a/mysql-test/suite/rpl/t/rpl_gtid_stop_start.test +++ b/mysql-test/suite/rpl/t/rpl_gtid_stop_start.test @@ -173,6 +173,22 @@ SELECT * FROM t1 ORDER BY a; --echo *** MDEV-4486: Allow to start old-style replication even if mysql.gtid_slave_pos is unavailable +# In GTID mode, the old-style replication position is also updated. But during +# GTID connect, the old-style position is not known until receiving the fake +# GTID list event, which contains the required position value. If we happened +# to stop the slave above before this fake GTID list event, the test could fail +# with duplicate key errors due to switching to non-GTID mode at a wrong +# position too far back in the binlog. +# +# Work-around this by injecting an extra dummt event and syncing the slave to +# it, ensuring the old-style position will be updated. +--connection server_1 +INSERT INTO t1 VALUES (8); +DELETE FROM t1 WHERE a=8; +--save_master_pos +--connection server_2 +--sync_with_master + --connection server_2 --source include/stop_slave.inc CHANGE MASTER TO master_use_gtid= no; -- 2.30.2

1 0