September 2023 - commits - lists.mariadb.org

[PATCH 0/4] MDEV-31273, Pre-compute binlog checksums outside of LOCK_log
by Kristian Nielsen 18 Oct '23

18 Oct '23

Hi Monty, Here's the implementation of MDEV-31273, pre-compute binlog checksums. The main patch to review is the last one, number 4. This is the actual implementation of binlog checksum pre-computation, and the only patch that changes the behaviour of the code. Most of the work (and most of the changes) are cleanups of the old checksum code that don't change the functionality but removes a lot of complex and hard-to-modify logic (and I think actually fixes a bug or two). I have kept this cleanup separate in the first 3 patches to make it easier to review and not get mixed up with the actual implementation of the new functionality. With this patch series, calculation of binlog checksum will happen when writing events into the stmt/trx caches. Later, when writing the binlog file under LOCK_log, only a direct copy of the bytes is done, which should improve binlog scalability with checksums enabled. The patch series is also available on github: https://github.com/MariaDB/server/commits/knielsen_mdev31273 - Kristian. Kristian Nielsen (4): MDEV-31273: Replace Log_event::writer with function parameter MDEV-31273: Eliminate Log_event::checksum_alg MDEV-31273: Refactor MYSQL_BIN_LOG::write_cache() MDEV-31273: Precompute binlog checksums include/my_atomic.h | 41 +- include/my_sys.h | 2 + .../main/mysqlbinlog_row_compressed.result | 48 +- .../main/mysqlbinlog_row_minimal.result | 48 +- .../main/mysqlbinlog_stmt_compressed.result | 16 +- mysql-test/main/mysqld--help.result | 7 + .../suite/binlog/include/binlog_ioerr.inc | 3 + mysql-test/suite/binlog/r/binlog_ioerr.result | 2 + .../r/binlog_mysqlbinlog_raw_flush.result | 1 + mysql-test/suite/binlog/t/binlog_killed.test | 2 +- .../t/binlog_mysqlbinlog_raw_flush.test | 2 + .../t/binlog_table_map_optional_metadata.test | 4 +- .../binlog_encryption/binlog_ioerr.result | 2 + .../suite/rpl/r/rpl_checksum_cache.result | 43 +- .../suite/rpl/t/rpl_checksum_cache.test | 98 +++- .../r/sysvars_server_notembedded.result | 10 + mysys/mf_iocache2.c | 34 ++ sql/log.cc | 468 ++++++++++-------- sql/log.h | 14 +- sql/log_event.cc | 27 +- sql/log_event.h | 158 +++--- sql/log_event_client.cc | 22 +- sql/log_event_old.cc | 14 +- sql/log_event_old.h | 4 +- sql/log_event_server.cc | 395 ++++++--------- sql/mysqld.cc | 1 + sql/mysqld.h | 1 + sql/privilege.h | 3 + sql/slave.cc | 60 +-- sql/sql_repl.cc | 2 +- sql/sys_vars.cc | 13 + sql/wsrep_binlog.cc | 6 +- sql/wsrep_mysqld.cc | 12 +- 33 files changed, 922 insertions(+), 641 deletions(-) -- 2.30.2

3 7

[PATCH] MDEV-31872: Add --old_mode=COMPAT_DISCOURAGED
by Kristian Nielsen 12 Sep '23

12 Sep '23

This mode can be used to disable warnings for functions whose use is strongly discouraged because of inherent limitations in the functionality they provide. But the use of which can be needed for backwards compatibility with legacy applications, and which cannot easily be replaced by the user. Currently used for weak encryption functions ENCODE()/DECODE(), to help users who need these functions to access existing data encoded with this non-standard algorithm. Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> --- mysql-test/main/func_str.result | 4 ++++ mysql-test/main/func_str.test | 2 ++ mysql-test/main/mysqld--help.result | 2 +- mysql-test/suite/sys_vars/r/old_mode_basic.result | 4 ++-- .../sys_vars/r/sysvars_server_notembedded.result | 2 +- mysql-test/suite/sys_vars/t/old_mode_basic.test | 2 +- sql/item_strfunc.cc | 11 ++++++++++- sql/sql_class.h | 1 + sql/sys_vars.cc | 3 ++- 9 files changed, 24 insertions(+), 7 deletions(-) diff --git a/mysql-test/main/func_str.result b/mysql-test/main/func_str.result index 8c09d19cc44..725ce2b7c46 100644 --- a/mysql-test/main/func_str.result +++ b/mysql-test/main/func_str.result @@ -323,6 +323,10 @@ decode(encode("abcdef","monty"),"monty")="abcdef" Warnings: Warning 1287 'encode' is deprecated and will be removed in a future release Warning 1287 'decode' is deprecated and will be removed in a future release +SET STATEMENT old_mode= 'COMPAT_DISCOURAGED' + FOR select decode(encode("abcdef","monty"),"monty")="abcdef"; +decode(encode("abcdef","monty"),"monty")="abcdef" +1 select quote('\'\"\\test'); quote('\'\"\\test') '\'"\\test' diff --git a/mysql-test/main/func_str.test b/mysql-test/main/func_str.test index feb3057ec41..e4b572a7f3f 100644 --- a/mysql-test/main/func_str.test +++ b/mysql-test/main/func_str.test @@ -142,6 +142,8 @@ select least(1,2,3) | greatest(16,32,8), least(5,4)*1,greatest(-1.0,1.0)*1,least select decode(encode(repeat("a",100000),"monty"),"monty")=repeat("a",100000); --enable_view_protocol select decode(encode("abcdef","monty"),"monty")="abcdef"; +SET STATEMENT old_mode= 'COMPAT_DISCOURAGED' + FOR select decode(encode("abcdef","monty"),"monty")="abcdef"; select quote('\'\"\\test'); select quote(concat('abc\'', '\\cba')); diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result index 6dc8543f225..0d5d2daade5 100644 --- a/mysql-test/main/mysqld--help.result +++ b/mysql-test/main/mysqld--help.result @@ -706,7 +706,7 @@ The following specify which files/extra groups are read (specified before remain NO_DUP_KEY_WARNINGS_WITH_IGNORE, NO_PROGRESS_INFO, ZERO_DATE_TIME_CAST, UTF8_IS_UTF8MB3, IGNORE_INDEX_ONLY_FOR_JOIN, COMPAT_5_1_CHECKSUM, - LOCK_ALTER_TABLE_COPY + LOCK_ALTER_TABLE_COPY, COMPAT_DISCOURAGED --old-passwords Use old password encryption method (needed for 4.0 and older clients) --old-style-user-limits diff --git a/mysql-test/suite/sys_vars/r/old_mode_basic.result b/mysql-test/suite/sys_vars/r/old_mode_basic.result index 7701bc6bbfa..3032a82581b 100644 --- a/mysql-test/suite/sys_vars/r/old_mode_basic.result +++ b/mysql-test/suite/sys_vars/r/old_mode_basic.result @@ -132,8 +132,8 @@ Warning 1287 'ZERO_DATE_TIME_CAST' is deprecated and will be removed in a future SELECT @@global.old_mode; @@global.old_mode ZERO_DATE_TIME_CAST -SET @@global.old_mode = 128; -ERROR 42000: Variable 'old_mode' can't be set to the value of '128' +SET @@global.old_mode = 256; +ERROR 42000: Variable 'old_mode' can't be set to the value of '256' SELECT @@global.old_mode; @@global.old_mode ZERO_DATE_TIME_CAST diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result index 0f4c54bb217..a48dd2dafe1 100644 --- a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result +++ b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result @@ -2459,7 +2459,7 @@ VARIABLE_COMMENT Used to emulate old behavior from earlier MariaDB or MySQL vers NUMERIC_MIN_VALUE NULL NUMERIC_MAX_VALUE NULL NUMERIC_BLOCK_SIZE NULL -ENUM_VALUE_LIST NO_DUP_KEY_WARNINGS_WITH_IGNORE,NO_PROGRESS_INFO,ZERO_DATE_TIME_CAST,UTF8_IS_UTF8MB3,IGNORE_INDEX_ONLY_FOR_JOIN,COMPAT_5_1_CHECKSUM,LOCK_ALTER_TABLE_COPY +ENUM_VALUE_LIST NO_DUP_KEY_WARNINGS_WITH_IGNORE,NO_PROGRESS_INFO,ZERO_DATE_TIME_CAST,UTF8_IS_UTF8MB3,IGNORE_INDEX_ONLY_FOR_JOIN,COMPAT_5_1_CHECKSUM,LOCK_ALTER_TABLE_COPY,COMPAT_DISCOURAGED READ_ONLY NO COMMAND_LINE_ARGUMENT REQUIRED VARIABLE_NAME OLD_PASSWORDS diff --git a/mysql-test/suite/sys_vars/t/old_mode_basic.test b/mysql-test/suite/sys_vars/t/old_mode_basic.test index cb18796729e..c3fc4c57189 100644 --- a/mysql-test/suite/sys_vars/t/old_mode_basic.test +++ b/mysql-test/suite/sys_vars/t/old_mode_basic.test @@ -172,7 +172,7 @@ SET @@global.old_mode = 4; SELECT @@global.old_mode; --Error ER_WRONG_VALUE_FOR_VAR -SET @@global.old_mode = 128; +SET @@global.old_mode = 256; SELECT @@global.old_mode; # use of decimal values diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc index 4b96271c331..339430a2794 100644 --- a/sql/item_strfunc.cc +++ b/sql/item_strfunc.cc @@ -2720,7 +2720,16 @@ bool Item_func_encode::seed() hash_password(rand_nr, key->ptr(), key->length()); sql_crypt.init(rand_nr); - warn_deprecated<1103>(current_thd, func_name_cstring().str); + if (!(current_thd->variables.old_behavior & OLD_MODE_COMPAT_DISCOURAGED)) + { + /* + This function should not be removed despite the deprecation warning. + That would cause problems for users who can then no longer access + columns in their database that might have been inserted as + ENCODE(str,pass_str). + */ + warn_deprecated<1103>(current_thd, func_name_cstring().str); + } return FALSE; } diff --git a/sql/sql_class.h b/sql/sql_class.h index 28c2831cfee..ff730ec2297 100644 --- a/sql/sql_class.h +++ b/sql/sql_class.h @@ -203,6 +203,7 @@ enum enum_binlog_row_image { #define OLD_MODE_IGNORE_INDEX_ONLY_FOR_JOIN (1 << 4) #define OLD_MODE_COMPAT_5_1_CHECKSUM (1 << 5) #define OLD_MODE_LOCK_ALTER_TABLE_COPY (1 << 6) +#define OLD_MODE_COMPAT_DISCOURAGED (1 << 7) #define OLD_MODE_DEFAULT_VALUE OLD_MODE_UTF8_IS_UTF8MB3 diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc index e25d293bbb7..18d08996aa8 100644 --- a/sql/sys_vars.cc +++ b/sql/sys_vars.cc @@ -3863,12 +3863,13 @@ static const char *old_mode_names[]= "IGNORE_INDEX_ONLY_FOR_JOIN", // deprecated since 11.3 "COMPAT_5_1_CHECKSUM", // deprecated since 11.3 "LOCK_ALTER_TABLE_COPY", // deprecated since 11.3 + "COMPAT_DISCOURAGED", 0 }; void old_mode_deprecated_warnings(THD *thd, ulonglong v) { - v &= ~OLD_MODE_DEFAULT_VALUE; + v &= ~(OLD_MODE_DEFAULT_VALUE | OLD_MODE_COMPAT_DISCOURAGED); for (uint i=0; old_mode_names[i]; i++) if ((1ULL<<i) & v) { -- 2.30.2

1 0

[PATCH] MDEV-10356: rpl.rpl_parallel_temptable failure due to incorrect commit optimization of temptables
by Kristian Nielsen 07 Sep '23

07 Sep '23

The problem was that parallel replication of temporary tables using statement-based binlogging could overlap the COMMIT in one thread with a DML or DROP TEMPORARY TABLE in another thread using the same temporary table. Temporary tables are not safe for concurrent access, so this caused reference to freed memory and possibly other nastiness. The fix is to disable the optimisation with overlapping commits of one transaction with the start of a later transaction, when temporary tables are in use. Then the following event groups will be blocked from starting until the one using temporary tables is completed. This also fixes occasional test failures of rpl.rpl_parallel_temptable seen in Buildbot. Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> --- sql/rpl_parallel.cc | 18 +++++++++++++++++- sql/sql_class.cc | 4 ++++ sql/sql_class.h | 13 +++++++++++++ 3 files changed, 34 insertions(+), 1 deletion(-) diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc index 3bd27c73932..fc4434b75de 100644 --- a/sql/rpl_parallel.cc +++ b/sql/rpl_parallel.cc @@ -218,6 +218,7 @@ finish_event_group(rpl_parallel_thread *rpt, uint64 sub_id, waiting for this). In most cases (normal DML), it will be a no-op. */ rgi->mark_start_commit_no_lock(); + rgi->commit_orderer.wakeup_blocked= false; if (entry->last_committed_sub_id < sub_id) { @@ -1425,7 +1426,22 @@ handle_rpl_parallel_thread(void *arg) if (!thd->killed) { DEBUG_SYNC(thd, "rpl_parallel_before_mark_start_commit"); - rgi->mark_start_commit(); + if (thd->lex->stmt_accessed_temp_table()) + { + /* + Temporary tables are special, they require strict + single-threaded use as they have no locks protecting concurrent + access. Therefore, we cannot safely use the optimization of + overlapping the commit of this transaction with the start of the + following. + So we skip the early mark_start_commit() and also block any + wakeup_subsequent_commits() until this event group is fully + done, inside finish_event_group(). + */ + rgi->commit_orderer.wakeup_blocked= true; + } + else + rgi->mark_start_commit(); DEBUG_SYNC(thd, "rpl_parallel_after_mark_start_commit"); } } diff --git a/sql/sql_class.cc b/sql/sql_class.cc index e7e27401d61..17feb006e21 100644 --- a/sql/sql_class.cc +++ b/sql/sql_class.cc @@ -7536,6 +7536,7 @@ wait_for_commit::reinit() wakeup_error= 0; wakeup_subsequent_commits_running= false; commit_started= false; + wakeup_blocked= false; #ifdef SAFE_MUTEX /* When using SAFE_MUTEX, the ordering between taking the LOCK_wait_commit @@ -7808,6 +7809,9 @@ wait_for_commit::wakeup_subsequent_commits2(int wakeup_error) { wait_for_commit *waiter; + if (unlikely(wakeup_blocked)) + return; + mysql_mutex_lock(&LOCK_wait_commit); wakeup_subsequent_commits_running= true; waiter= subsequent_commits_list; diff --git a/sql/sql_class.h b/sql/sql_class.h index 4487a67c76d..4c172ba8e2a 100644 --- a/sql/sql_class.h +++ b/sql/sql_class.h @@ -2142,6 +2142,19 @@ struct wait_for_commit group commit as T1. */ bool commit_started; + /* + Set to temporarily ignore calls to wakeup_subsequent_commits(). The + caller must arrange that another wakeup_subsequent_commits() gets called + later after wakeup_blocked has been set back to false. + + This is used for parallel replication with temporary tables. + Temporary tables require strict single-threaded operation. The normal + optimization, of doing wakeup_subsequent_commits early and overlapping + part of the commit with the following transaction, is not safe. Thus + when temporary tables are replicated, wakeup is blocked until the + event group is fully done. + */ + bool wakeup_blocked; void register_wait_for_prior_commit(wait_for_commit *waitee); int wait_for_prior_commit(THD *thd, bool allow_kill=true) -- 2.30.2

1 0