- developers - lists.mariadb.org

Re: [PATCH] MDEV-33551: Semi-sync Wait Point AFTER_COMMIT Slow on Workloads with Heavy Concurrency
by Kristian Nielsen 18 Mar '24

18 Mar '24

> From: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > > When using semi-sync replication with > rpl_semi_sync_master_wait_point=AFTER_COMMIT, the performance of the > primary can significantly reduce compared to AFTER_SYNC's > performance for workloads with many concurrent users executing > transactions. This is because all connections on the primary share Hi Brandon, Here is my second round review of the MDEV-33551 patch. I think these comments of mine from first round still need to be addressed: > > A bit of care must be taken about lifetime of Tranx_node and THD > > respectively (in case one goes away but not the other). But the whole > > > +void Repl_semi_sync_master::await_all_slave_replies() > > > + wait_result= cond_timewait(front.thd, &abstime); > > > > I'm wondering, is it safe here to wait on front.thd->COND_wakeup_ready? > > Isn't there a possibility that this might go away during the wait? I tried a simple testcase (attached) which disconnects the session thread while await_all_slave_replies() is running during shutdown. As I expected, I could see the COND_wakeup_ready of the session thread being de-allocated while await_all_slave_replies() is still waking up from that condition. That cannot be correct. Similarly, it seems clearly possible that a session thread is killed and exits while its entry is still present in the m_active_tranxs list, which will similarly lead to signalling a de-allocated COND_wakeup_ready. When a thread aborts the wait before the entry is removed, I think it should simply look up its entry (can use the m_trx_htb hash table, or even just iterate the list) and set its THD to NULL (or remove the entry, whatever is simpler). For the await_all_slave_replies() problem, I think the whole wait-at-shutdown is much too complex as it is now. The whole is_thd_awaiting_semisync_ack(), n_threads_awaiting_ack, special-case not killing threads waiting for semi-sync-ack, and now introducing unmark_thd_awaiting_ack. And it still has the bug waiting on a COND_wakeup_ready of another THD that can disappear during the wait. I think all of that should just go away, the wait in await_all_slave_replies() should be done much simpler with a dedicated condition variable to wait for. Eg. await_all_slave_replies() just does a mysql_cond_wait(COND_binlog_send) under LOCK_binlog until the last binlog position is ack'ed, and this condition is then signalled from report_reply_binlog() (either always, or await_all_slave_replies() can set a flag to request signal). Or have the shutdown thread enqueue its THD at the end of the m_active_tranxs list. Let's avoid all this complex interaction between SHUTDOWN WAIT FOR ALL SLAVES and session threads. BTW, COND_binlog_send is currently unused in your patch. Below detailed comments on individual parts of the patch: > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -1750,18 +1750,12 @@ static void close_connections(void) > - if (shutdown_wait_for_slaves && repl_semisync_master.get_master_enabled()) > + if (shutdown_wait_for_slaves && repl_semisync_master.get_master_enabled() && > + repl_semisync_master.sync_get_master_wait_sessions()) Do you need to check sync_get_master_wait_sessions() > 0 here? Why not wait unconditionally for the last binlog position sent? > --- a/sql/semisync_master.cc > +++ b/sql/semisync_master.cc > @@ -68,6 +68,14 @@ static ulonglong timespec_to_usec(const struct timespec *ts) > return (ulonglong) ts->tv_sec * TIME_MILLION + ts->tv_nsec / TIME_THOUSAND; > } > > +int signal_waiting_transaction(THD *waiting_thd, const char *binlog_file, > + my_off_t binlog_pos) > +{ > + if (waiting_thd->is_awaiting_semisync_ack) > + mysql_cond_signal(&waiting_thd->COND_wakeup_ready); The check on is_awaiting_semi_sync_ack seems wrong here. Normally, the THD will be waiting, otherwise it would not be in the list in the first place, so faster with an occasional redundant signal than a conditional always. And if the THD is not waiting, it's not safe to access it anyway. Instead, as mentioned above, have a check for thd != NULL here, and let a non-waiting thread clear the THD in its entry before it releases the LOCK_binlog. > -int Repl_semi_sync_master::cond_timewait(struct timespec *wait_time) > +int Repl_semi_sync_master::cond_timewait(THD *thd, struct timespec *_wait_time) > + if (_wait_time == NULL) > + { > + create_timeout(&gen_wait_time, NULL); > + real_wait_time= &gen_wait_time; > + } > + else > + { > + real_wait_time= _wait_time; > + } Don't pass in booleans that are always hard-coded true or false in the caller. Instead create two different functions, one that takes a time_spec, and one that does not (the latter creating it and calling the former). > @@ -533,7 +559,8 @@ void Repl_semi_sync_master::remove_slave() > Signal transactions waiting in commit_trx() that they do not have to > wait anymore. > */ > - cond_broadcast(); > + m_active_tranxs->clear_active_tranx_nodes(NULL, NULL, > + signal_waiting_transaction); This does not compile on my machine (the second NULL is passed to my_off_t position), is that a last-minute change that was never tested? Or NULL is defined differently on your machine, so it doesn't fail to compile? BTW, question about the old code: Why didn't the old code clear the m_active_tranxs list? Was it a bug? Or was it cleared elsewhere before (and if so where)? > -void Repl_semi_sync_master::await_slave_reply() > +void Repl_semi_sync_master::await_all_slave_replies() > + The lock is taken per iteration rather than over the whole loop to allow > + more opportunity for the user threads to handle their ACKs. That sounds like a mis-understanding. The lock is always released by mysql_cond_timedwait() in each iteration of the loop, no need to release/take it again for just a couple instructions. > + while (TRUE) > + { > + lock(); > + if (m_active_tranxs->is_empty() || !get_master_enabled() || !is_on()) > + { > + unlock(); > + break; > + } > + wait_result= m_active_tranxs->for_front( > + [](THD *thd, const char *binlog_file, my_off_t binlog_pos) -> int { > + return (thd->is_awaiting_semisync_ack) > + ? repl_semisync_master.cond_timewait(thd, NULL) > + : 0; > + }); As I wrote at the top, it is not safe to wait on the COND_wakeup_ready of another THD, since that THD might go away during the wait. Instead, let's do a specific wait for the end position here, either on COND_binlog or by enqueing an extra THD at the end. (And btw, in your code, why wait for the _front_ of the list and not the _rear_?) > diff --git a/sql/semisync_master.h b/sql/semisync_master.h > index 99f46869354..1eb690b572b 100644 > --- a/sql/semisync_master.h > +++ b/sql/semisync_master.h > > + * The pre_delete_hook parameter is a function pointer that will be invoked > + * for each Active_tranx node, in order, from m_trx_front to log_file_name, > + * e.g. to signal their wakeup condition. Repl_semi_sync_binlog::LOCK_binlog > + * is held while this is invoked. "from m_trx_front to log_file_name", I don't understand? (Typo s/log_file_name/m_trx_rear/ ?) > +/* > + Element in Repl_semi_sync_master::wait_queue to preserve the state of a > + transaction waiting for an ACK. > +*/ > +typedef struct _semisync_wait_trx { > + const char *binlog_name; > + my_off_t binlog_pos; > + THD *thd; > +} semisync_wait_trx_t; > + This is not longer used (should be removed). > * Input: (the transaction events' ending binlog position) > * trx_wait_binlog_name - (IN) ending position's file name > * trx_wait_binlog_pos - (IN) ending position's file offset > + * unmark_thd_awaiting_ack - (IN) Whether or not to unmark > + * thd->is_awaiting_semisync_ack after receiving > + * an ACK from the replica. If using wait_point > + * AFTER_SYNC with binlog_group_commit, only the > + * last transaction written to binlog in the > + * group should negate the boolean, because the > + * same thread (i.e. leader thread) will wait for > + * all transaction ACKs. Negating it too early > + * would break SHUTDOWN WAIT FOR ALL SLAVES, as > + * that is the condition tested to bypass killing > + * threads in phase 1. In all other situations, > + * the boolean should be negated immediately. That seems unnecessarily convoluted. I think the real problem here is that write_tranx_in_binlog() sets the is_awaiting_semi_sync_ack flag on the wrong THD, on the waiter_thd instead of the trx_thd. But once we fix the await_all_slave_replies(), do we even need the is_awaiting_semi_sync_ack flag any more? If so, why? If not, then this point becomes moot anyway. - Kristian. --source include/have_innodb.inc --source include/have_debug.inc --source include/have_binlog_format_mixed.inc --source include/master-slave.inc --connection master CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; INSERT INTO t1 VALUES (1,0); --sync_slave_with_master source include/stop_slave.inc; SET @old_semi_enabled= @@GLOBAL.rpl_semi_sync_slave_enabled; SET GLOBAL rpl_semi_sync_slave_enabled = 1; --source include/start_slave.inc --connection master SET GLOBAL rpl_semi_sync_master_enabled= 1; SET GLOBAL rpl_semi_sync_master_timeout= 5000; let $status_var= Rpl_semi_sync_master_clients; let $status_var_value= 1; source include/wait_for_status_var.inc; show status like 'Rpl_semi_sync_master_status'; show status like 'Rpl_semi_sync_master_clients'; --connection slave SET @old_dbug= @@GLOBAL.debug_dbug; SET GLOBAL debug_dbug="+d,simulate_delay_semisync_slave_reply"; --connect(con1, localhost, root,,) --connect(con2, localhost, root,,) --connection con1 send INSERT INTO t1 VALUES (2,0); --sleep 0.3 --disconnect con1 --connection default --write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect wait EOF #let $status_var= Rpl_semi_sync_master_wait_sessions; #let $status_var_value= 1; #source include/wait_for_status_var.inc; --sleep 0.1 --connection con2 SHUTDOWN WAIT FOR ALL SLAVES; --disconnect con2 --connection default --source include/wait_until_disconnected.inc --connection master --source include/wait_until_disconnected.inc --connection server_1 --source include/wait_until_disconnected.inc --connection default --append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect restart EOF --enable_reconnect --source include/wait_until_connected_again.inc --connection master --enable_reconnect --source include/wait_until_connected_again.inc --connection server_1 --enable_reconnect --source include/wait_until_connected_again.inc --save_master_pos --connection slave SET GLOBAL debug_dbug= @old_dbug; SET GLOBAL rpl_semi_sync_slave_enabled= @old_semi_enabled; --source include/stop_slave.inc --source include/start_slave.inc --sync_with_master --connection default DROP TABLE t1; --source include/rpl_end.inc

2 3

Re: e6cacca0b39: MDEV-23729 MDEV-32218 INFORMATION_SCHEMA table for user data
by Sergei Golubchik 15 Mar '24

15 Mar '24

Hi, Nikita, no test for FLUSH PRIVILEGES no test for a user with manually expired password doesn't compile in embedded hostname 'neo' is replaced, but not everywhere I'll try to fix it myself, but no guarantees On Mar 15, Nikita Malyavin wrote: > revision-id: e6cacca0b39 (mariadb-11.4.1-10-ge6cacca0b39) > parent(s): 929c2e06aae > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-03-15 15:32:45 +0100 > message: > > MDEV-23729 MDEV-32218 INFORMATION_SCHEMA table for user data > > * A new table INFORMATION_SCHEMA.USERS is introduced. > * It stores auxiliary user data > * An unprivileged user can access their own data, and that is the main > difference with what mysql.global_priv provides > * The fields are currently: USER, PASSWORD_ERRORS, PASSWORD_EXPIRATION_TIME > * If password_errors is ignored for the user, PASSWORD_ERRORS is NULL > * PASSWORD_EXPIRATION_TIME is a timestamp with exact point in time, calculated > from password_last_changed and password_lifetime (i.e. days) stored for the user > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 14450a5a610..a87b04f3d39 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc ... > +int fill_users_schema_table(THD *thd, TABLE_LIST *tables, COND *cond) > +{ > + int res= 0; > +#ifndef NO_EMBEDDED_ACCESS_CHECKS > + bool see_whole_table= check_access(thd, SELECT_ACL, "mysql", NULL, NULL, > + true, true) == 0; > + TABLE *table= tables->table; > + > + if (!see_whole_table) > + { > + mysql_mutex_lock(&acl_cache->lock); > + ACL_USER *cur_user= find_user_exact(thd->security_ctx->priv_host, > + thd->security_ctx->priv_user); > + if (!cur_user) > + { > + mysql_mutex_unlock(&acl_cache->lock); > + my_error(ER_INVALID_CURRENT_USER, MYF(0)); Hmm. Questionable. I'd just return an empty result set here, I_S tables normally don't throw errors. But this is an exceptional situation and I cannot say that an error is wrong here, let's keep it your way. > + return 1; > + } > + > + res= fill_users_schema_record(thd, table, cur_user); > + mysql_mutex_unlock(&acl_cache->lock); > + return res; > + } > + > + mysql_mutex_lock(&acl_cache->lock); > + for (size_t i= 0; res == 0 && i < acl_users.elements; i++) > + { > + ACL_USER *user= dynamic_element(&acl_users, i, ACL_USER*); > + res= fill_users_schema_record(thd, table, user); > + } > + mysql_mutex_unlock(&acl_cache->lock); > +#endif > + return res; > +} > + Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: [PATCH 0/2] Suggestion for smaller fix to XA parallel replication performance, MDEV-31949
by Kristian Nielsen 13 Mar '24

13 Mar '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > which makes a worker with dependent pieces of job be *effectively* placed at the > tail of the available worker ('fifo') queue. Indeed. And this BTW is why a simple sliding window of size 2*workers is not enough, because in principle a worker can sit in the fifo for many transactions without being scheduled, if there are many dependency restrictions. I coded the thing with "generations" as a quick way to handle this and have a way to clean out old dependency information. This could of course be improved in many ways. Maybe remember the two last XID for every worker and have a hash table mapping each XID to its worker, or something. But this is just a PoC. > No you are not. Thanks for the catch! To excuse myself it was supposed > to be a poc that I committed hastedly. Yes yes, no problem, I understand, it's good to do poc. I was just momentarily confused, but the intent is clear, thanks. - Kristian.

1 0

Re: a473db9e4fe: MDEV-25829 Change default collation to utf8mb4_1400_ai_ci
by Sergei Golubchik 13 Mar '24

13 Mar '24

Hi, Alexander, On Mar 08, Alexander Barkov wrote: > commit a473db9e4fe > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Thu Nov 2 14:16:09 2023 +0400 > > MDEV-25829 Change default collation to utf8mb4_1400_ai_ci "utf8mb4_uca1400_ai_ci". It's still not too late to rename the MDEV and change commits to match. I'm not reviewing InnoDB changes, I hope Marko did > > Step#3 The main patch > > diff --git a/mysql-test/include/ctype_utf8mb4.inc b/mysql-test/include/ctype_utf8mb4.inc > index 436b0f2782f..73011cef81d 100644 > --- a/mysql-test/include/ctype_utf8mb4.inc > +++ b/mysql-test/include/ctype_utf8mb4.inc > @@ -58,11 +58,11 @@ select CONVERT(_koi8r' > # "a\0" < "a" > # "a\0" < "a " > > -SELECT 'a' = 'a '; > -SELECT 'a\0' < 'a'; > -SELECT 'a\0' < 'a '; > -SELECT 'a\t' < 'a'; > -SELECT 'a\t' < 'a '; > +SELECT 'a' = 'a ' collate utf8mb4_general_ci; > +SELECT 'a\0' < 'a' collate utf8mb4_general_ci; > +SELECT 'a\0' < 'a ' collate utf8mb4_general_ci; > +SELECT 'a\t' < 'a' collate utf8mb4_general_ci; > +SELECT 'a\t' < 'a ' collate utf8mb4_general_ci; 1. what will happen in utf8mb4_uca1400_ai_ci? 2. if you specifically want this file to test utf8mb4_general_ci, you can do SET NAMES, cannot you? Once for the whole file. Or set character_set_collations > > # > # The same for binary collation > diff --git a/mysql-test/main/ctype_euckr.result b/mysql-test/main/ctype_euckr.result > index 9e030f9cc6d..8de90f89a26 100644 > --- a/mysql-test/main/ctype_euckr.result > +++ b/mysql-test/main/ctype_euckr.result > @@ -2244,7 +2244,7 @@ FE80 > DELETE FROM t2 WHERE a='?'; > ALTER TABLE t2 ADD u VARCHAR(1) CHARACTER SET utf8, ADD a2 VARCHAR(1) CHARACTER SET euckr; > UPDATE IGNORE t2 SET u=a, a2=u; > -SELECT s as unassigned_code FROM t2 WHERE u='?'; > +SELECT s as unassigned_code FROM t2 WHERE u=binary'?'; why? > unassigned_code > A2E8 > A2E9 > diff --git a/mysql-test/main/ctype_like_range.result b/mysql-test/main/ctype_like_range.result > index e5e5e61126b..fa694b0c250 100644 > --- a/mysql-test/main/ctype_like_range.result > +++ b/mysql-test/main/ctype_like_range.result > @@ -284,7 +284,7 @@ id name val > 32 mn 63616161616161616161616161616161 > 32 mx 63616161616161616161616161616161 > 32 sp -------------------------------- > -ALTER TABLE t1 MODIFY a VARCHAR(32) CHARACTER SET utf8; > +ALTER TABLE t1 MODIFY a VARCHAR(32) CHARACTER SET utf8 COLLATE utf8_general_ci; do you have tests for ctype_like_range with utf8mb3_uca1400_ai_ci ? > INSERT INTO t1 (a) VALUES (_ucs2 0x0425),(_ucs2 0x045F); > INSERT INTO t1 (a) VALUES (_ucs2 0x2525),(_ucs2 0x5F5F); > SELECT * FROM v1; > diff --git a/mysql-test/main/ctype_utf8.test b/mysql-test/main/ctype_utf8.test > index a875fe51f3a..2f56d10bccf 100644 > --- a/mysql-test/main/ctype_utf8.test > +++ b/mysql-test/main/ctype_utf8.test > @@ -24,7 +24,7 @@ drop database if exists mysqltest; > --disable_warnings > drop table if exists t1,t2; > --enable_warnings > -set names utf8; > +set names utf8 collate utf8_general_ci; why not to change the test to work with utf8mb3_uca1400_ai_ci? > > select left(_utf8 0xD0B0D0B1D0B2,1); > select right(_utf8 0xD0B0D0B2D0B2,1); > diff --git a/mysql-test/main/dyncol.test b/mysql-test/main/dyncol.test > index 493e9b3842d..ba302861e03 100644 > --- a/mysql-test/main/dyncol.test > +++ b/mysql-test/main/dyncol.test > @@ -10,10 +10,10 @@ > --echo # column create > --echo # > select hex(COLUMN_CREATE(1, NULL AS char character set utf8)); > -select hex(COLUMN_CREATE(1, "afaf" AS char character set utf8)); > -select hex(COLUMN_CREATE(1, 1212 AS char character set utf8)); > -select hex(COLUMN_CREATE(1, 12.12 AS char character set utf8)); > -select hex(COLUMN_CREATE(1, 99999999999999999999999999999 AS char character set utf8)); > +select hex(COLUMN_CREATE(1, "afaf" AS char character set utf8 collate utf8_general_ci)); > +select hex(COLUMN_CREATE(1, 1212 AS char character set utf8 collate utf8_general_ci)); > +select hex(COLUMN_CREATE(1, 12.12 AS char character set utf8 collate utf8_general_ci)); > +select hex(COLUMN_CREATE(1, 99999999999999999999999999999 AS char character set utf8 collate utf8_general_ci)); why? > select hex(COLUMN_CREATE(1, NULL AS unsigned int)); > select hex(COLUMN_CREATE(1, 1212 AS unsigned int)); > select hex(COLUMN_CREATE(1, 7 AS unsigned int)); > diff --git a/mysql-test/main/func_json.result b/mysql-test/main/func_json.result > index ab35822a6ce..fce2caa2842 100644 > --- a/mysql-test/main/func_json.result > +++ b/mysql-test/main/func_json.result > @@ -414,6 +414,13 @@ select json_object('foo', json_unquote(json_object('bar', c)),'qux', c) as fld f > fld > {"foo": "{\"bar\": \"abc\"}", "qux": "abc"} > {"foo": "{\"bar\": \"def\"}", "qux": "def"} > +create table t2 as select json_object('foo', json_unquote(json_object('bar', c)),'qux', c) as fld from t1 limit 0; > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `fld` varchar(39) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci > +drop table t2; why do you test here? > drop table t1; > select json_object("a", json_object("b", "abcd")); > json_object("a", json_object("b", "abcd")) > diff --git a/mysql-test/main/func_regexp_pcre.test b/mysql-test/main/func_regexp_pcre.test > index 77f5af6e0ff..7cf9e8007de 100644 > --- a/mysql-test/main/func_regexp_pcre.test > +++ b/mysql-test/main/func_regexp_pcre.test > @@ -23,17 +23,17 @@ SELECT 'Ã ' RLIKE '\\x{00C0}' COLLATE utf8_bin; > SELECT 'Ã' RLIKE '\\x{00C0}' COLLATE utf8_bin; > > # Checking how (?i) and (?-i) affect case sensitivity > -CREATE TABLE t1 (s VARCHAR(10) CHARACTER SET utf8); > +CREATE TABLE t1 (s VARCHAR(10) CHARACTER SET utf8 COLLATE utf8_general_ci); why? > INSERT INTO t1 VALUES ('a'),('A'); > -CREATE TABLE t2 (p VARCHAR(10) CHARACTER SET utf8); > +CREATE TABLE t2 (p VARCHAR(10) CHARACTER SET utf8 COLLATE utf8_general_ci); > INSERT INTO t2 VALUES ('a'),('(?i)a'),('(?-i)a'),('A'),('(?i)A'),('(?-i)A'); > SELECT s,p,s RLIKE p, s COLLATE utf8_bin RLIKE p FROM t1,t2 ORDER BY BINARY s, BINARY p; > DROP TABLE t1,t2; > > > # Checking Unicode character classes > -CREATE TABLE t1 (ch VARCHAR(22)) CHARACTER SET utf8; > -CREATE TABLE t2 (class VARCHAR(32)) CHARACTER SET utf8; > +CREATE TABLE t1 (ch VARCHAR(22)) CHARACTER SET utf8 COLLATE utf8_general_ci; > +CREATE TABLE t2 (class VARCHAR(32)) CHARACTER SET utf8 COLLATE utf8_general_ci; and here > INSERT INTO t1 VALUES ('Ð¯'),('Î£'),('A'),('Ã'); > INSERT INTO t1 VALUES ('Ñ'),('Ï'),('a'),('Ã '); > INSERT INTO t1 VALUES ('ã'),('ê°·'),('à¶´'); > diff --git a/mysql-test/main/mrr_icp_extra.result b/mysql-test/main/mrr_icp_extra.result > index 8f6ee88acc6..d7cde103288 100644 > --- a/mysql-test/main/mrr_icp_extra.result > +++ b/mysql-test/main/mrr_icp_extra.result > @@ -754,7 +754,7 @@ t1 CREATE TABLE `t1` ( > KEY `t` (`t`(5)) > ) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci > drop table t1; > -create table t1 (v char(10) character set utf8); > +create table t1 (v char(10) character set utf8 collate utf8_general_ci); why here? > show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > diff --git a/mysql-test/main/myisam.test b/mysql-test/main/myisam.test > index 1a20f97a54f..4a283737122 100644 > --- a/mysql-test/main/myisam.test > +++ b/mysql-test/main/myisam.test > @@ -1039,7 +1039,7 @@ create table t1 (v varchar(65536)); > show create table t1; > drop table t1; > set statement sql_mode = 'NO_ENGINE_SUBSTITUTION' for > -create table t1 (v varchar(65530) character set utf8); > +create table t1 (v varchar(65530) character set utf8 collate utf8_general_ci); why? collation doesn't matter here, does it? > show create table t1; > drop table t1; > > diff --git a/mysql-test/main/order_by.result b/mysql-test/main/order_by.result > index 274f29e34dc..85059fb3370 100644 > --- a/mysql-test/main/order_by.result > +++ b/mysql-test/main/order_by.result > @@ -4142,7 +4142,7 @@ drop table t1; > # > # MDEV-21922: Allow packing addon fields even if they don't honour max_length_for_sort_data > # > -create table t1 (a varchar(200) character set utf8, b int); > +create table t1 (a varchar(200) character set utf8 collate utf8_general_ci, b int); why? I think I see too many forced utf8_general_ci in tests and it's difficult to believe that the collation actually matters in all these cases. > insert into t1 select seq, seq from seq_1_to_10; > select * from t1 order by a; > a b > diff --git a/sql/item_jsonfunc.cc b/sql/item_jsonfunc.cc > index 508ea9f644e..5731ad1f04b 100644 > --- a/sql/item_jsonfunc.cc > +++ b/sql/item_jsonfunc.cc > @@ -842,7 +842,7 @@ String *Item_func_json_quote::val_str(String *str) > bool Item_func_json_unquote::fix_length_and_dec(THD *thd) > { > collation.set(&my_charset_utf8mb3_general_ci, > - DERIVATION_COERCIBLE, MY_REPERTOIRE_ASCII); > + DERIVATION_CAST, MY_REPERTOIRE_ASCII); why? I guess that's why you have json_unquote test above. > max_length= args[0]->max_length; > set_maybe_null(); > return FALSE; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: b657bc75f90: MDEV-25829 Change default collation to utf8mb4_1400_ai_ci
by Sergei Golubchik 13 Mar '24

13 Mar '24

Hi, Alexander, On Mar 08, Alexander Barkov wrote: > revision-id: b657bc75f90 (mariadb-11.0.1-262-gb657bc75f90) > parent(s): 8e5f41b3453 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-11-15 10:58:03 +0400 > message: > > MDEV-25829 Change default collation to utf8mb4_1400_ai_ci > > Step#2 - Adding a new collation derivation level for CAST and CONVERT. > > Now character string cast functions: > - CAST(string_expr AS CHAR) > - CONVERT(expr USING charset_name) > > have a new collation derivation level between: > > - string literals > - utf8 metadata functions, e.g. user() and database() > > Before the change these cast functions had collation derivation equal > to table columns, which caused more illegal mix of collation conflicts. > > Note, binary string cast functions: > - CAST(string_expr AS BINARY) > - CONVERT(expr USING binary) > did not change their collation derivation. Why not? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 8e5f41b3453: MDEV-25829 Change default collation to utf8mb4_1400_ai_ci
by Sergei Golubchik 13 Mar '24

13 Mar '24

Hi, Alexander, On Mar 08, Alexander Barkov wrote: > revision-id: 8e5f41b3453 (mariadb-11.0.1-261-g8e5f41b3453) > parent(s): 9e457cbe501 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-11-15 09:37:26 +0400 > message: > > MDEV-25829 Change default collation to utf8mb4_1400_ai_ci > > Step#1 - Changing collation derivation for string user variables > from EXPLICIT to COERCIBLE. from IMPLICIT, oherwise ok to push. and I still think user variable could remember the derivation. It's definitely not a part of MDEV-25829, but if you agree that it should, let's create an MDEV for that. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: [PATCH 0/2] Suggestion for smaller fix to XA parallel replication performance, MDEV-31949
by Kristian Nielsen 12 Mar '24

12 Mar '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > I reviewed your branch to agree with the idea of xid conflicts > handling and its implementation. > > I however could not understand the need of the refactoring part. > In order to track the xid dependency couple functions and and > rpl_parallel_entry::maybe_active_xid > a sort of a sliding window - that the 2nd commit of your branch > introduced must be sufficient to my analysis. > Of course it needs to hold worker indexes of active xid:s. > > This observation led me to create a review branch > > origin/review__knielsen_xa_sched_minimal_fix > > [where origin git@github.com:MariaDB/server.git] > which is a somewhat light elaboration over the 2nd commit of your branch. The reason for the refactoring part is to allow to distribute the transactions as evenly as possible over the worker threads, given the scheduling constraints imposed by XA dependencies. Let's say we have 5 worker threads, and transactions T1..T15. If there are no scheduling constraints, we can schedule evenly like this: W1: T1 T6 T11 W2: T2 T7 T12 W3: T3 T8 T13 W4: T4 T9 T14 W5: T5 T10 T15 Now suppose we need to schedule T6 on the same worker as T4, and T8 on the same worker as T3. The most even way to distribute the transactions is now this: W1: T1 T7 T12 W2: T2 T9 T14 W3: T3 T8 T13 W4: T4 T6 T11 W5: T5 T10 T15 You see, we try to always have 5 consecutive transactions T_i, T_{i+1}, ..., T+{i+4} scheduled on 5 different worker threads, except when this is not possible due to dependency restrictions. But this most-even scheduling requires to change the scheduling order of threads from the original sequential 1,2,3,4,5. You see, the last transactions T11..T15 are scheduled on worker threads in order: W4, W1, W3, W2, W5. The refactoring patch introduces thread_sched_fifo to hold the current scheduling order of worker threads. Whenever we have to schedule out-of-order due to a scheduling dependency, we put the scheduled worker at the end of this fifo, to preserve even scheduling for the next N transactions. In the original scheduling code, we never scheduled workers out-of-order, so the scheduling order was always cyclic W1, W2, W3, W4, W5, W1, ..., and a simple incrementing counter was sufficient. But this is no longer sufficient when out-of-order scheduling occurs. In the example, if we use a simple counter and try to schedule on worker ((i+1) mod N) after worker (i), then we get the following uneven scheduling: W1: T1 T11 W2: T2 T12 W3: T3 T8 T13 W4: T4 T6 T9 T14 W5: T5 T7 T10 T15 Because T6 and T8 have particular scheduling requirements, they cause the scheduling to skip workers. The result is that W4 and W5 get too many transactions, while W1 and W2 get too few, and parallelism is reduced. I hope this explain the reasoning. > Could you please have a look at it? If I understand your patch correctly, it schedules using a simple incrementing counter when there are no dependency requirements, and so would suffer from the uneven scheduling in the above example. It also looks like there's a mistake in the code: > + if ((idx= check_xa_xid_dependency(&gtid_ev->xid)) < (uint32) -1) > + { > + /* > + A previously scheduled event group with the same XID might still be > + active in a worker, so schedule this event group in the same worker > + to avoid a conflict. > + */ > + } > + else > + { > + /* Record this XID now active. */ > + xid_active_generation *a= > + (xid_active_generation *)alloc_dynamic(&maybe_active_xid); > + if (!a) > + return NULL; > + ++idx; > + if (idx >= rpl_thread_max) > + idx= 0; If check_xa_xid_dependency() returns "no dependency", then idx is set to (uint32)-1 and the else {...} branch is executed. This does idx++, which increments the (uint32)-1 to 0. So it looks to me that _all_ transactions are scheduled on worker 0, or am I missing something? This is a simple mistake, it's an easy fix to not assign idx when check_xa_xid_dependency() returns "no dependency". But the code would still suffer from uneven scheduling as in the example, IIUC. So there's a good reason to introduce the thread_sched_fifo as done in my refactoring patch. This though still leaves the limitation in my minimal patch that XA PREPARE xid1 cannot group commit with XA COMMIT xid1. The impact of this will depend on how many transactions on average appear between an XA PREPARE and its associated XA COMMIT, as well as on how expensive fsync() is relative to the cost of each transaction. - Kristian.

1 0

Re: 9e5d4dfc49b: MDEV-23729 MDEV-32218 INFORMATION_SCHEMA table for user login data
by Sergei Golubchik 11 Mar '24

11 Mar '24

Hi, Nikita, On Mar 10, Nikita Malyavin wrote: > revision-id: 9e5d4dfc49b (mariadb-11.4.1-10-g9e5d4dfc49b) > parent(s): 929c2e06aae > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-02-29 17:24:27 +0100 > message: > > MDEV-23729 MDEV-32218 INFORMATION_SCHEMA table for user login data > > * A new table INFORMATION_SCHEMA.LOGON is introduced. There were many cases where we lack an INFORMATION_SCHEMA with all user accounts. Forcing users to select from mysql.user is a shame. Let's use the chance and create a table USERS. It doesn't need more columns now than what you've created, but we'll likely create more in the future. > * Upon idea, it stores auxiliary user data related to login/security/resources > * An unprivileged user can access their own data, and that is the main > difference with what mysql.global_priv provides exactly! > * The fields are currently: USER, WRONG_PASSWORD_ATTEMPTS, EXPIRATION_TIME I wanted to write about splitting USER into USER and HOST, but indeed it's just USER everywhere else in INFORMATION_SCHEMA (e.g. DEFINER and GRANTEE columns). And functions like USER() and CURRENT_USER() don't split either. So, you're right, let's consistently use user@host everywhere. > diff --git a/mysql-test/main/information_schema_stats.result b/mysql-test/main/information_schema_stats.result > index 352bcbab823..e38788872fa 100644 > --- a/mysql-test/main/information_schema_stats.result > +++ b/mysql-test/main/information_schema_stats.result ... > +connect(localhost,naughty_user,wrong_passwd,test,16000,/home/nik/mariadb/bld/mysql-test/var/tmp/mysqld.1.sock); here and below - you forgot to replace the path with MASTER_MYSOCK > diff --git a/mysql-test/main/information_schema_stats.test b/mysql-test/main/information_schema_stats.test > index fd5171c3fb4..dbcabe45965 100644 > --- a/mysql-test/main/information_schema_stats.test > +++ b/mysql-test/main/information_schema_stats.test > @@ -47,3 +47,69 @@ select * from information_schema.index_statistics where table_schema='test' and > select * from information_schema.table_statistics where table_schema='test' and table_name='just_a_test'; > set global userstat=@save_userstat; > --enable_ps2_protocol > + > +--echo # > +--echo # MDEV-23729 INFORMATION_SCHEMA Table info. about user locked due to > +--echo # max_password_errors > +--echo # > +--echo # MDEV-32218 message to notify end-user N-days prior the password get > +--echo # expired > +--echo # I don't see how you test "info about user locked due to max_password_errors". This is the reason for implementing this new table, it has to be tested. At least add select * from information_schema.logon where wrong_password_attemps >= @max_password_errors; and show it it's empty at first and then not empty. > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 14450a5a610..4d199bf0e61 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -12956,6 +12956,87 @@ int fill_schema_column_privileges(THD *thd, TABLE_LIST *tables, COND *cond) > #endif > } > > +namespace Show > +{ > + ST_FIELD_INFO users_fields_info[] = > + { > + Column("USER", Userhost(), NOT_NULL), > + Column("WRONG_PASSWORD_ATTEMPTS", SLonglong(), NULLABLE), @@max_password_errors is documented as "If there is more than this number of failed connect attempts due to invalid password, user will be blocked from further connections until FLUSH_PRIVILEGES" Let's use call incorrect password consistently everywhere. It could be "bad password", "wrong password", "incorrect password", "invalid password", but it should be the same everywhere. I personally think that "invalid" implies some kind if validity check, e.g. not shorter than 10 characters, not equal to the username, etc and a password that doesn't satisfy these validity rules is "invalid". Trying to login with an incorrect password is a different kind of error, a password can be valid but wrong. May be, ask Ian and/or Daniel about it? And then don't forget to update @@max_password_errors help text to match. Also, you don't have a test case for FLUSH PRIVILEGES. > + Column("EXPIRATION_TIME", SLonglong(), NULLABLE), PASSWORD_EXPIRATION_TIME, the user account itself does not expire. > + CEnd() > + }; > +}; > + > +static bool ignore_max_password_errors(const ACL_USER *acl_user); > + > +static int fill_logon_schema_record(THD *thd, TABLE * table, ACL_USER *user) > +{ > + ulonglong lifetime= user->password_lifetime < 0 > + ? default_password_lifetime > + : user->password_lifetime; > + > + bool ignore_password_errors= ignore_max_password_errors(user); why? I think it's still useful to show how many wrong password attempts were there for an account even if it doesn't get blocked. > + bool ignore_expiration_date= lifetime == 0; > + > + /* Skip user if nothing to show */ > + if (ignore_password_errors && ignore_expiration_date) > + return 0; > + > + Grantee_str grantee(user->user.str, safe_str(user->host.hostname)); > + table->field[0]->store(grantee, strlen(grantee), system_charset_info); > + if (ignore_password_errors) > + { > + table->field[1]->set_null(); > + } > + else > + { > + table->field[1]->set_notnull(); > + table->field[1]->store(user->password_errors); > + } > + if (ignore_expiration_date) > + { > + table->field[2]->set_null(); > + } > + else > + { > + table->field[2]->set_notnull(); > + table->field[2]->store(user->password_last_changed > + + user->password_lifetime * 3600 * 24, true); I think it'd be more generally useful to show password_last_changed and expiration period separately. > + } > + > + return schema_table_store_record(thd, table); > +} > + > +int fill_logon_schema_table(THD *thd, TABLE_LIST *tables, COND *cond) > +{ > + int res= 0; > +#ifndef NO_EMBEDDED_ACCESS_CHECKS > + bool see_whole_table= check_global_access(thd, PROCESS_ACL, true) == 0; I don't think PROCESS_ACL is a very logical choice here. And there's nothing better as far as I can see. May be let's just do if (check_access(thd, SELECT_ACL, "mysql", NULL, NULL, 1, 1)) ? There are many checks like that in e.g. sql_parse.cc > + > + TABLE *table= tables->table; > + > + if (!see_whole_table) > + { > + mysql_mutex_lock(&acl_cache->lock); > + ACL_USER *cur_user= find_user_exact(thd->security_ctx->priv_host, > + thd->security_ctx->priv_user); 1. cur_user can be NULL if someone dropped it while there was an active connection for this user 2. add a test for it > + > + res= fill_logon_schema_record(thd, table, cur_user); > + mysql_mutex_unlock(&acl_cache->lock); > + return res; > + } > + > + mysql_mutex_lock(&acl_cache->lock); > + for (size_t i= 0; res == 0 && i < acl_users.elements; i++) > + { > + ACL_USER *user= dynamic_element(&acl_users, i, ACL_USER*); > + res= fill_logon_schema_record(thd, table, user); > + } > + mysql_mutex_unlock(&acl_cache->lock); > +#endif > + return res; > +} > + Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Backup MariaDB Databases for Suprema BioStar 2 Door Access System
by Turritopsis Dohrnii Teo En Ming 09 Mar '24

09 Mar '24

Subject: Backup MariaDB Databases for Suprema BioStar 2 Door Access System Good day from Singapore, On 7 March 2024 Thursday, when I was installing new self-signed SSL certificate for the door access system for a law firm in Singapore, I notice that Suprema BioStar 2 also uses the open source MariaDB database server. If I did not remember wrongly, there are 3 databases. However, the entire platform is a Windows environment, not Linux. How can I backup all the databases in the MariaDB database server? Thank you. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore Blogs: https://tdtemcerts.blogspot.com https://tdtemcerts.wordpress.com GIMP also stands for Government-Induced Medical Problems.

1 0

Re: be66e975ecf: MDEV-31340 Remove MY_COLLATION_HANDLER::strcasecmp()
by Sergei Golubchik 09 Mar '24

09 Mar '24

Hi, Alexander, Ok to push, thanks for the great refactoring! I've looked at the diff of this and previous commit excluding InnoDB. On Mar 08, Alexander Barkov wrote: > revision-id: be66e975ecf (mariadb-11.0.1-282-gbe66e975ecf) > parent(s): c0c1c80346b > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-01-19 11:04:02 +0400 > message: > > MDEV-31340 Remove MY_COLLATION_HANDLER::strcasecmp() > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0